Ok I actually found a solution, simple really. I had every layer on a flat z space in 3D. Layers would hide behind the background I had when zooming out. I put the background layer further back and the issue was solved.
Working off a server might be the problem. I always work locally, especially dealing with footage. If I work with footage off a server, it gets pretty slow. I would work locally first, then save everything back to your server when you are done.