Every year at Christmas time, the folks at Temple Square in Salt Lake City Utah put on a brilliant display of Christmas lights. No one knows exactly how many lights there are, but some people estimate it is just under a million. In addition to the lights, there is a giant nativity scene, a bunch of smaller nativity scenes from around the world, and various groups singing Christmas songs.
I figured this would be the perfect type of event to create a Walkabout virtual walk through for. This Walkabout consists of over 130 full 360 degree panoramas linked together so that you can "walk" almost anywhere you want over the 10 acres of Temple Square.
Click on the link below to start your Walkabout through this wonderful display of lights.
You can click and drag the images to look in any direction you want, including up or down. Little green spheres will appear to show which ways you can walk
If you are a mobile device look for this icon:
If you click on this, you will go into compass mode. This will allow you to look around just by moving your phone in whatever direction you want to look.
If you own a Google Cardboard virtual reality headset, look for the Google Cardboard icon. Click it to go into virtual reality mode and have a truly immersion viewing experience
I have a vision that one day I will be able to take a 360 camera and quickly walk through a room and then with no additionally input from me be able to take that footage and have it automatically converted into a virtual reconstruction of that room that you could walk around in virtual reality. I want to make the process so easy that anyone even the most untech savy person could do it.
So today's experiment is a first pass at that. The way I figured I approach this is that if I can assess my position in the room at all times while walking through a room with a 360 video camera going, then I could map frames from the video to floor coordinates and then as the user "walked" around a room I could show the appropriate frame according to the users floor coordinates.
Ideally, I would have some sort of high tech positional tracking like the Vive Lighthouse or The Oculus positional tracker. Although requiring those would probably contradict my goal of making this accessible to anyone who is not tech savy. So since I don't have either of those, instead I just coded up a rudimentary color based object tracker and using some geometry and the orientation of the camera I made a python script that automatically calculates my position in the room at each frame. So then the process becomes this:
Walk through room with 360 video camera
Load video into my software and in the first frame mark some object that can be seen everywhere in the room
Let software run and reconstruct floor positions
Done
The red is the window I am tracking to calculate my position
Follow the link below to see the end result. Use the arrow keys to look around and the W key to walk forward. (or if you are on mobile, just swipe around and tap on the screen to walk forward)
As you can see it worked okay but not perfect. There are still a number of issues to be solved. Like for example it let's you pass right through the couch. And sometimes the position estimation is a little off so you get some weird jumps. And you will also notice that you can't go into the dining room. I tried to do this but the hanging light blocked the marker which messed up the positional tracking. You may notice the one of the window is colored red. This window is what I set as the object to be tracked.
There are a lot of things I could do to make the tracking more robust and with some more work I am confident I could make it so the user doesn't even have to mark an initial tracking object at the beginning. I am experimenting with that now using OpenCV and feature detection. Hopefully my next blog update with have some positive results from that. Also in the future I hope to be able to take all this positional information and all this still frames from various positions and use them to automatically generate a 3d model.
After my last experiment with creating bidirectional 360 video, I decided I should create a virtual path of something that might actually be interesting to see. Below you can take a walk down a mountain trail and take a look at the Fall colors. Or if you prefer I also made it so you could run. I really wish Youtube had a way in its javascript api to be able to tell which way you are looking. That way I could just make you run or walk in whichever way you are currently looking instead of having separate buttons for each direction. Eventually I will create an app in Unity that will allow you to use the virtual reality headset of you choice, then it will be easier to tie the direction you are looking to which way you move. Until then give my latest experiment a try: (just hold down the run and walk icons to go up or down the trail, drag on the video or use the circle controller to look around)
Bidirectional 360 video is video that can be played smoothly in either forward or backwards. Due to how compression works for video, usually it cannot be played smoothly in reverse.
Why would bidirectional 360 video be useful?
Image that you create a 360 video while walking down a path. If you could play it both forward and backwards, then you instantly have a little virtual reality path that you can walk back and forth in. String a bunch of this bidirectional videos together and you could construct a whole virtual world you could walk around in. Below is a little example of a single path. Use the white arrows to walk back and forth. The example will work best on the desktop, though it should work on mobile as well.
Walk back and forth and look around and see if you can find the Tardis:
How to make bidirectional video:
You can make bidirectional with a few easy steps.
First take your video clip and reverse it.
Concatenate the reversed clip with the original clip
Then make a controller that knows where to jump to when you want to change directions
The logic for the controller is fairly simple. because the first half of the video will going forward and the second half of the video is going backwards, whenever you want to change directions, you can simple do something like this:
var time = player.getDuration() - player.getCurrentTime();
player.seekTo(time,true);
This will jump to the exact place in the video that corresponds to going in the opposite direction of where you are currently in the video.
Walkabout worlds have come a long way since I first showed some friends my first demos at the start of the year. In general I have am pretty happy with the the state they are at (see here), but I found that while static photospheres were great for building a virtual world for something like a house, they did not quite capture what it was like to be at great outdoor location. There wasn't the sounds or the movement of leaves and water that make you feel like you are there.
As I thought about it, I realized that it would only take a small tweak to make a Walkabout use a 360 video instead of a 360 panorama. So I went and grabbed my new Ricoh Theta S, took some video, and ran it through the walkabout editor and 10 minutes later I had my first video based Walkabout. Right now it only works on Google Chrome, but I am sure with a little tweaking I can get it working on IE and Firefox. I have had other 360 videos work using WebGL on those other browsers, so I think I just need to use a different codec or something.
Also the video quality is not great. The Theta S video quality is improved over the Theta M15 but it is still not pro level and the Theta doesn't let you play with color or ISO settings in video mode, so you will notice as you move around that the color changes quite a bit.
But still, as a test of concept, I really like these first results. Check it out:
