Tracking a 360 footage – Oculus Rift

[Philip Bowser]

This is a great question! And not one exclusive to the Oculus Rift. I had previously pondered about how one would go about match moving panoramic footage for an interactive short film I was going to develop. In all of my research it doesn’t seem as though there are any tools built specifically for tracking this type of footage.

There were two potential solutions I came up with that I never put into practice, but here they are anyways:

Solution 1:
Track all cameras individually/unstitched in a tracking software (I use PFTrack). You can use the first camera track’s position/rotation data as a “hint” for the second camera track, and then use the second camera’s data as a hint for the third and so forth. You’re going to want to align each of the cameras to each other in virtual space as they were set up in real world space. There are tools for this in PF Track but I’m not sure what software you would use. You then export all of your cameras to your 3D app or AE. You animate whatever magic you’re trying to have happen and render it out separately from each 3D camera view. Then you stitch each of those renders back together using the same technique that you stitched the original footage together and if all works out perfectly for some reason it should line up! And Composite away!

Solution 2:
If your individual graphic elements will only be localized to a certain area in your panoramic footage, you may be able to get away with rendering out those portions of your panoramic video as regular HD footage (essentially cropping in to the 16:9 dimension you desire) and then do a camera track on just that portion, add your effects, render out, and then bring that render with your effects back into the panoramic composition and line it up to that original portion you rendered out.

Because this is such an unusual request, I doubt there are any easy tools out there for this kind of problem. And without any camera data or scene surveying recorded on set to make the tracking easier, it could prove to be extremely challenging to get anything usable. However if you find a solution please post about it, I would love to know how you end up solving this!!

Cheers,

Philip. Bowser

[Clyde Desouza]

Interesting question for sure.
The clue must lie in the Lens distortion map of the camera. Maybe shooting a distortion grid and then aligning the Graphic element to this grid could be the workflow?

Check this post for a possible clue:
https://cgi.tutsplus.com/tutorials/correcting-barrel-distortion-and-compositing-3d-elements-using-pftrack-maya-and-after-effects–cg-14068

Please let us know how this goes.

—————————-
Clyde DeSouza
twitter: @cly3d
Author: THINK in 3D: Food For Thought for Directors and Cinematographers.

now available on Amazon:
https://www.amazon.com/Think-in-3D-ebook/dp/B007DK92J0

https://www.realvision.ae/blog
—————————–

[Dylan Saliba]

Hello,
I just wanted to say I’ve been using the cow for YEARS as a rock solid reference to any A/V limitations / issues and usually the answer is found here! This is my first post, but it’s a good one.
The solution I have found to this problem you list is a small script for after effects from Mettle called SkyBox. It basically sets up a 360 rig with AE cameras and then precomps to a nice and reasonably simple setup. The demo allows for 3 “skyboxs” to be created so you can totally check it out and see if it works for you before purchasing. It’s only $65 too, completely worth every penny! With this script / plugin you can also have interactive HDRi plates…but that is a whole other can of worms. This 360 stuff is cRaZy!
I’m sure this will help,
D

[Jesus Diaz]

Hi Simone, Just wondering how you were able to pull off the track motion for the 360 footage you were asking about? Did you end up using sky box?