The way I understand collapse tranformations is like this, If in one comp I scale a layer down to 10% of its original size and then I scale a comp containing that comp up 10 times, collapse transformation joins the math for displaying the 2nd comp into one operation like this scale = 0.1*10 which equals 1 so the layer in the nested comp is displayed at its original size without being pixelated. When collapse transformations is not selected, the scaled down layer is scaled up as if it were a small raster image and you get distortion. An example of when you need to have collapse transformations turned off is when you are using AE’s motion blur.
Pete
roguekeyframe.com