I’m guessing you are having a problem for one or both of the following reasons:
1) you need to be working in a 16 bit or floating point project. Many times an 8 bit displacement map does provide enough information to prevent aliasing (or what you call pixelized)
2) AE’s displacement mapping does not provide any form of filtering. It just does a simple pixel “look up”. This is a know limition of AE’s displacement map filter. You should try our RE:Map product ( https://www.revisionfx.com/rmap.htm ). RE:Map has a displacement map filter that does the proper filtering. Like I suggested with AE’s displacement map filter, you’ll probably want to work in a 16 bit project.
RE:Map also has a corner pinner that provides proper filtering (along with many other features that AE’s does not), a UV map filter (takes a rendered UV map from a 3D program and renders with any texture you provide) and an auto-distort plugin that distorts images based on their features.
Cheers,
Pete LItwinowicz
https://www.revisionfx.com