All of your children layers must be set as 3D layers if you want them to transform with the 3D layer they are parented to. This should work without any issues.
Otherwise you could make a pre-comp of your entire animation and then just scale up the pre-comp to have your letters zoom out at the end.