I think it needs to be:
Mac Pro
1- GT120
2 -Cubix expansion
3-Decklink
4-HBA
Cubix
1-FX4800
2-FX4800
3-small tree 10g
4-open
the configuration guide never lists the GT120 in conjunction with a Cubix, but since it’s supported, i can’t imagine it being a problem.
One other thought.. are the FX4800’s single wide or double wide? you may not be able to fit that small tree card in there as well..