[Mort Richardion]

You can get something similar to Roughen Edges in photoshop using the following steps:

1. Ctrl click on your target layer to make a selection

2. Go to Select> Modify> Contract, and shrink the selection by a few pixels

3. Alt click on the Create Layer Mask icon to mask your selection, and Ctrl+I to invert the new mask

4. Go to Filter> Distort> Ripple to distort the mask

5. Then try Filter> Brush Strokes> Accented Edges to distress the mask. Just be sure to make the Edge Width the same value as the selection contraction from step two.

(And yes, I’m aware this question is from five years ago…but it’s the first thing that came up in google for Roughen Edges in Photoshop)