“Abnormal condition” is probably the error you received.
Although deleting unused layers appeared to fix your problem, I very much doubt that the number of layers or the presence of layer comps had anything to do with the error you received. I’ve worked with menus with far more layers, and also with layer comps and have not gotten any error in Encore.
If you could post one of your menus someplace, I’d like to look at it, though.