Combustion is a video compositing app made by Discreet:
https://www4.discreet.com/combustion/
It is After Effects main competitor.
You said the exported swf quality was horrible. How so? Pixelated, jerky, washed out…? Was the initial QT export out of Flash bad (try playing it in QT before going to AE) or just after you got it into/out of AE?
I would suggest skipping Flash altogether for this. Go straight to Combustion or AE. Maybe even Motion would do what you want? Then, depending on what your expected output is to be (DVD, Film, Broadcast, Web) you might be able to get by with Final Cut Express. This limits you in a few ways (see here for a comparison between iMovie, FCE and FCP: https://www.apple.com/finalcut/) but ‘most’ are mitigated if you have AE or Combustion. Of course, going to film is a different story, you’ll need FCP for that.
As for the animation stuff… Why Flash? For the vectors maybe? AFAIK, vectors aren’t of much value unless you’re intending to stay on the web. Plus, through Flash, I doubt you’ll be able to export an alpha channel, which is what you’d need to do the compositing you mentioned. Is it just the animation aspect?
If so, take a look at Bauhaus Mirage:
https://www.bauhaussoftware.com/home_LP.php
This would allow the animations AND compositing in one app. In fact, if what you’re doing is short enough, say, less than 5 minutes, Mirage alone might do it without FCP or FCE.