I’m thinking you are running into anti-aliasing issues. The jaggies are being caused by your magic wand cutting off hard instead of feathering in the shades of gray.
If John’s response meets your needs then ignore the rest of this.
Ideally, I think you would want to copy your gray scale image and paste it into a layer mask. Unfortunately, I could not figure out a way to do that directly (which is interesting), so try this:
– Go to your Channels tab.
– Ctrl-click on any of the layers so that a new selection is made.
– Go back to your Layers tab.
– Create a new layer (this should be 100% transparent)
– Click on the add layer mask button (this should use your selection to create a layer mask).
– Use the paint bucket to fill your new layer with black (there should be no active selection at this time, so the entire layer will be black, but the mask will show your original shape).
– This is inverted, which is what it seemed like you wanted. If you need to, then click on the mask and invert the mask by hitting Ctrl-I.
Pauley