It seems to me like it’s done by using a displacement map obtained from the luminance of a duplicate of that layer; in fact, as you can see, the brightest the region of the image is, the more it’s being displaced. If the trails you get are not as smooth as you would like them to be, try pre-composing the reference layer and inside the pre-comp use a fast blur set to a very low amount, 2-5px sould do the job; then maybe try setting the whole composition to 16 or 32 bits per channel (though I’m not sure wheteher the Displacement Map effect can actually work up to 16 bits).
Let me know if you have any more questions.
Leopy