Essentially this is down to the picture in picture plugin being kinda primitive. It lacks anti-aliasing.
I did find a fix for this though – add “Border” before “Picture In Picture” and set it to Blurred. For me size 0,005 was enough. This should fix the jagged borders.
Note that the preview quality is also important here. Setting it to Good (Half) vs Best (Full) – in Best (Full) the edges will be more detailed by default, so keep that in mind. Your project is likely set to render with the Best settings, not just Good.