To make them look like they’re flying past the camera, you should make them 3D layers, not 2D layers that are scaling. Scaling 2D layers will make it look like they are slowing down as the scaling goes higher, whereas real motion towards camera in 3D is exponential, not linear in terms of scale. Although with 3D you will have to deal with the layers not being 0 scale in the beginning by fading them in.
Several ways of making a random end point with a common beginning point. It’s not a big deal to do it manually either, as you can first have the layers starting at the same time and keyframe the beginning positions to be the same, then at another point in time keyframe them to be different and finally offset the layers.
If you want to do it with expressions, you could use this for random X/Y position, (doesn’t affect Z, so you’d have to animate it or modify this accordingly):
seedRandom(index, true);
end_offset_x = random(250)-random(250);
end_offset_y = random(250)-random(250);
xy_offset=(time-startTime)*[end_offset_x,end_offset_y,0];
value+xy_offset
250 may not be the right value for you, you’ll have to see what works in your comp.