It is now owned by Adobe (it was previously Macromedia) so it plays well with AE and their other products.
There would be a variety of ways to approach what you want to do based on how you want to deliver it (DVD, browser, standalone app, etc.) and how much you want to get into actionscripting.
Flashkit is a good resource https://www.flashkit.com/
I’m not saying this would be the best method, but this is probably what I would do:
Create the various comps in AE and render them.
Then create the interactivity in Flash and have the videos load in a swf.
This would give you more versatility than Encore unless you are burning to disk.
I haven’t seen the commercials you are referring to, but you should be able to create that effect by converting your layers into 3D layers. Move your anchor to the bottom of the image and animate it rotating on the appropriate axis.
Try fooling around with and reading up on blending modes, circular masks, and track mattes.
That’s pretty much all you need to know for the technical aspects of this sort of effect.
To make it look nice, you’ll need a good sense of design also.
I found a solution. (You probably already discovered this)
There is an easier way in CS3 that was unavailable in prior versions.
Create a shape layer and modify the roundness under the rectangle.
Voila, instantly the shape you want.
It has something to do with a change instilled in CS3 for sure.
I tried it out in CS3 and 7. It worked fine in 7, but I got the same problem as you in CS3.
You need to click the stopwatch to remove all of your keyframes.
That way, AE uses the present frame for all frames.
What is happening in your video is AE is animating the mask change between the 2 keyframes.