Perhaps create a Mask image in Photoshop. White parts represent Opaque and black areas for transparent. Make it the size of your Sequence, such as 1920×1080 pixels. So you would create a white octagon on black background.
Put a background image or video on V1, put overlay video (that you want the cutout from) on V2 and apply Track Matte Key to it, and put Mask image on V3. In effects setup, set Matte to Video3, and Composite using LUMA.(See image)
If set up correctly, this should put an octagonal video over whatever background you have in place on V1.
In my sample image, I combined red and yellow using the matte. Note that areas of gray will have varying levels of transparency, providing a blend of V1 and V2 images.
Note that if the V2 clip is animated, that of course will move your masking for interesting results. Or change opacity over time, many options.

Thanks
Jeff Pulera
Safe Harbor Computers