Actually I’ve just made a discovery which seems to work for me. In Preferences/Audio Hardware I set my default device to System Default Input/Output, since then everything is stable.
Doesn’t really make much sense as I had my settings matching my system settings anyway but I guess there may have been some hidden conflict somewhere.
Hope that helps you too.