You have many options. One easy one is to duplicate the track, size it up to fill the screen, and Gaussian blur the heck out of it. Then float your clean version over this.
What’s nice about this is that the colors in the blur will mostly match the colors at the edges of the clear picture. I tried this just this past week, when I had a 16×9 clip that had to play inside a 4:3 program. There, I sized the 16:9 to touch the side edges and the blur top and bottom worked great, a differnrt look and feel than black letterbox.
Sometimes just a simple gradient works great to fill the dead space; try sampling the two main color areas in each source shot to build the gradient with, and the look then becomes more unified and harmonious.
Other options include making several tracks with the same element sized and cropped in diferent ways, then float your clean track above this comp with some drop shadow, and it looks intentional.
Really, your creativity is the only limit here.