To create a proper 3D tearing effect you would need to go down the route of soft body dynamics in a serious 3D application however there should be a way of cheating it in AE.
I have done this very effect many years ago before 3D existed in AE and back then I simply filmed real ripped paper against black and composited it into my work.
If not using a 3D app or filming real paper tears you might want to have a look at all the distortion effects that AE has to offer and design your tearing effect within Photoshop with reference photography to build up a PSD file full of layers containing torn edges that can be manipulated and distorted in AE to create the animated tear you desire!
I’m not sure how Illustrator’s bevelling or extruding can create 3D-ness in AE which compares to the shatter effect however it might be possible to use shatter with some carefully made shatter maps and the right motion for the bits which fly off to achieve some sort of ripped hole appearing in a layer.
To flex or bend the layer like paper would if being punched through might be achievable with Invigorator or freeform (which comes with AE CS5. see: https://digieffects.com/product/freeform_ae)
Graham Macfarlane
3D animator and VFX specialist
London UK