Rush charges are fair if you have to work overtime to make the deadline, over weekends and holidays and the like, or if you have to over-extend yourself with a subcontractor or vendor, being on the hook for costs the client should have already handled, and settle accounts after the dust dies down. Never a good place to be, even with long trusted clients.
For me the question is along the lines of: does taking on the job under the special circumstances force me into a special hardship or does it prevent me from other work that could be bringing in money? Did I have to bump another client and was that a huge deal to do? Did I have to miss my kid’s school play debut AGAIN for this guy? Put off the long overdue dinner out with the wife? Time to charge a little more for the added annoyance and opportunity cost.
The huge mistake is to ever bill flat fixed price, IMO. You are always going to be way over or way under what you predicted for the time you put in a fixed fee job, especially with clients like this that hold needed elements until the last minute or are otherwise unprepared or unorganized. I would bill hourly for everything as much as possible. That way, you reward them for having their act together because it makes the work go faster and thus cost less each time they improve.
If you work so fast that you think you’re losing money billing by the hour, I think you probably need to re-evaluate your rates. You must be pretty darn good, and good is worth paying more for.
Alternately, slow down after the first version is done and re-evaluate the work, maybe take a breath and play around with alternate versions or added graphics or sound tweaking, etc. because maybe in all the rush you missed an opportunity to add some art, taking it to another level.