Christopher,
We have been editing large amounts of XDCAM50 at our facility for the past 2 years, and the render times are exactly what you are getting for keying green screen with the Primatte Keyer. It is due to the XDCAM codec being long GOP instead of i-Frame with DVCProHD. In our workflow, we went from DVCProHD to XDCAM50, and were shocked at the difference in overall render times (roughly 4 to 5x slower). Best of luck with your project; sounds like your setup is correct.
Jason