Thanks for all of you advice.
Now this is a bit confusing: things seem to work right now.
Same system, same GPU, no newer driver.
I did this:
I switched back from the Quadro to the GTX first.
My plan was to make a backup and then upgrading the system hoping the new system and driver would solve the issue.
Now I wanted to give it a shot and render my previous sequence in order to see if the machine would run out of RAM before crashing.
Before I checked the drivers again and there was a cuda driver available dating from January 2016. I guess this had to be reinstalled after switching GPUs. I am 150% sure that I was up to date after putting in the GTX in March, I even double checked through the preference pane when I had the events of crashing.
Now I started rendering and nothing of the expected happened again.
RAM in use during rendering was always below 4GB, the program is set to “use 18GB” and “leave 6GB” to other programs.
Rendering 30min of 1080p in ProResHQ material to H.264 in vbr 10mbit took roughly 2h. A few tracks, some color correction, some denoising.
I am glad it worked but still a bit worried not to know why.
Best B.