I don’t recall now how the slots are numbered in my G5, but I do remember the pairs need tobe physically located from inside out or outside in…so your config should be…
256
256
512
512
256
256
or
512
256
256
256
256
512
starting from the innermost slots and moving outward.
If your RAM is in a row as you’ve indicated, this certainly could manifest in some strange behavior…
Just my thoughts…
Chris