I'm not sure right off, to be honest. The only thing I can think of, is when you are accessing an element in the PoolByteArray from within an Array, it is not setting it because of the indirect reference. That said, I do not think that is the issue.
Maybe try this:
var a : PoolByteArray = PoolByteArray()
a.resize(5)
t.append(a)
print("Before modify array : ", t[0].hex_encode())
var a_from_t = t[0]
a_from_t.set(4, 128)
# test to see if it was set in a_from_t
print("a-from-t after modify array : ", a_from_t.hex_encode())
# test if a in the t array has changed without any additional modification
print("After modify array : ", t[0].hex_encode())
# finally, see if applying a_from_t to t fixes the issue
t[0] = a_from_t
print("After applying modify a-from-t array : ", t[0].hex_encode())
Outside of that, I'm not sure right off. I haven't really used PoolByteArray that much myself. If you have not already, it might not be a bad idea to also ask on other Godot channels, as there might be users there who have used PoolByteArray that can chime in with solutions.