I have reproduced the first method exactly as you described it. Unfortunately it made no difference, I don’t think the issue is related to cache.
However, with your PS I feel like you have pointed me into the right direction!
I opened a project, selected all the audio from all the layers and did the following: right click – Remove Attributes – Effects (I always use Hard Limiter, Denoise, Lowpass) and clicked on Ok. It works perfectly after doing this. No playback delay.
Obviously, this is not ideal, I do need these effects. I tried narrowing it down to one, but it seems like it’s not a particular sound effect that’s causing the issue, but the quantity of them.
Overall this is still pretty weird, because it doesn’t matter how many sound effects do I use, if I skip copying the project to another PC and back, it has no impact on performance.
I could do a ‘Render and replace’ with all the audio selected, but than all of my fade transitions would be ruined and needed to be redone. Also not very ideal.