[Alex Lindblom]

Well for CS6 it’s very well explained in this link. The good stuff is at the end.
https://tv.adobe.com/watch/digital-video-cs6/how-to-optimize-after-effects-cs6-for-high-performance/

I use to same set up on 32 gig system and it works fine.

6 GB for other applications

4 real cpus, in your case it will be 6 cpus he explains why in the video.

3 GB per core and you can afford 4 GB since you have more memory.

[Alex Lindblom]

I managed to pull an okay, fast key with only using…
Screen gain — 130
Screen Balance — 50
Then
Clip White — 8
Screen softness — 5
But since you have the original footage you will probably get away with less softness.
And now the most important part.
Replace Method — Source

It’s the soft color choice that introduces the artifacts you see on your key. So I would say in 99% of the cases use the “Source” option for Replace Method.