Warping Fisheye Images for Spherical Mirror Projection… Or… How to make your front room into an immersive cinema.
This post doesn’t go into any of the complex mathematics involved with spherical mirror projection. For that information I recommend using the invaluable resources at http://paulbourke.net . This post explains a practical solution for projecting onto complex geometry using spherical mirrors, 3ds Max and 123D Catch.
Okay, so really I’m just trying to figure out a good way of previewing my fulldome work at home, thus speeding up the creative process. The idea is to be able to sit in a standard space, such as the corner of a room, and watch an immersive video.
This is a video showing the physical setup and some immersive video.
The main elements to consider are these:
- Spherical Mirror
- Computer/Robotic Thinking Machine
- Viewing Position
The area I’m projecting onto is mainly white walls, which are appropriate for the brightness of projector I have. The corner of the room should give me a good amount of enclosure to create a panoramic perspective. The area could be almost any structure, although shadows may be cast from the position of the mirror which will be visible if the viewing position differs greatly. More complex surfaces and structures will take longer to produce and require more precise measurements. These are all the elements in situ. I’m projecting a flat grid here, just to show the raw distortion from the mirror. It’s also possible to see just how much light falloff there is as the projection surface gets further away from the mirror. It would have been better to place the mirror at an equal distance away from the walls as the ceiling, but hey, I know for next time.
I’m using a terrible projector for the purpose, it’s a 4:3 800×600 with no lens shift and a dim lamp by today’s standards. Nevertheless, it’s all I have and it’s good enough for working things out. Really a 1080p projector would be a much better choice. Also, a tilt shift would help maximise the resolution and positioning of the elements to achieve optimal projection. Blah blah blah.
The spherical mirror is an old prototype rig, gratefully borrowed off Richard Lake from Polestar Planetarium. A spherical mirror like this can be purchased from Amazon for around £70, but will have a plastic outer coating, which will cause refractive artefacts in the projection. Both of the mirrors I’m using have an external reflective coating, which is best for a high fidelity image, but extremely fragile. A secondary mirror is useful to save space but not a requirement, in fact this will actually reduce the brightness of the projected image slightly.
The laptop is a thousand year old Macbook Pro running Leopard. Nothing fancy is needed… Just lights and clockwork.
The viewing position is important. I positioned a seat just behind the mirror to allow comfortable viewing of the immersive environment. The system will only really work from a one person perspective, but that perspective should be possible to move wherever.
The Virtual Setup
The main elements needed for the virtual setup are:
- Room Geometry
- Spherical Mirror
- Fisheye Projection Light
The virtual setup is basically an exact copy of the real-world setup with the addition of a fisheye projecting light and replacing the projector with a camera.
The room will need to be created with accurate measurements or things aren’t going to match up. To make this process easier I’m using 123D Catch to create a rough version of the room and then built the room with simple geometric shapes that aligned to the 123D Catch version. This process could be replaced by measuring the room and positions of all the important elements within it.
123D Catch works best with textured surfaces, plain white walls aren’t so good. So I projected some Kandinsky onto it to help with the catch.
Here’s the 123D Catch geometry. It’s pretty rough in places but only took 10 minutes and will give me the position I need for all the elements in the scene.
After exporting an fbx file and importing it into 3ds Max, it’s possible to overlay simple geometry over the catch geometry to recreate the scene more accurately.
The geometry will need to be a solid white standard material with no specular.
For the mirror to work correctly it will need to be rendered using physically accurate ray-trace reflections. I’m using the built in Mental Ray renderer.
For the mirror I added a full sphere with Mental Ray Arch+Design chrome material on it. This must be accurately aligned to the mirror in the real world using the 123D Catch geometry as a guide. It’s worth pointing out that the mirror may not be a halved hemisphere, but rather a section of a hemisphere, so the length of the back of the mirror won’t necessarily be the diameter.
The mirror will need a dense smooth mesh for best results. I rotated the sphere so the highest density of the mesh was towards the camera, which I found did make a difference.
The camera should be positioned in the same place as the projector lens. I decided to miss the secondary mirror out in the virtual setup and position the camera as it would appear without it. This may be a little confusing, and it does require an additional step to flip the image ready for reflection, so in hindsight I would probably have set it up with the secondary mirror and kept the virtual setup and real-world setup exactly the same.
The camera FOV should match that of the projector.
The projector I’m using doesn’t have a lens shift on it but it should be possible to match the lens shift using a skew modifier on the camera and rotating the gizmo to create vertical skew.
Fisheye Projection Light
The fisheye light will project a fisheye image onto the geometry in the scene, much the same way a fisheye projector would in the real world. The light will need projection map turning on and a bitmap adding to the map slot. The light will also need specular turning off to not cause highlights in the mirror. The map will need to be an angular fisheye to create an immersive environment when viewed from the correct position. The mapping needs to be set to Shrink-wrap environment and the tiling set to U:-2 V:2 and turn off tiling. The W angle needs to be set to 90. The light can then be positioned in the desired viewing position. Basically, place the light where you can easily stand or sit down. Keep in mind the position of the mirror and the maximum area of projection. If you place the light too far behind the mirror you will lose a large portion of the fisheye image from the projection area. I’ve placed the light at sitting height just behind the mirror to miximise the panoramic potential of the space.
The render output should be the same aspect ration of the projectors.
Indirect illumination such as final gathering should be avoided for standard immersive projection. This will actually occur when it’s projected anyway, the next step is to figure out how to reduce this.
Before rendering any fisheye imagery it’s a good idea to test the setup. To do this I used a dome alignment grid as the light projection map. Then render the view from the camera.
This render must be flipped if you are missing the secondary mirror stage. The image can then be displayed full screen on the projector and it should look like a circular grid with even spacing between the lines as you move your head around. I actually added the two coloured squares onto the geometry of the room so I could check they were positioned in the same place in the room.
Here you can see the green and pink panels matching their position in the virtual space. You can also see that the grid is heavily distorted from this perspective.
You can see in the following image from the set viewing position that the curved lines of the grid continue as they should, even over the corner of the room. Looking at this for the first time was a pretty good feeling. Beyond the awful resolution and slight misalignment, it actually worked.
Once you know your setup works, you can render all kinds of fisheye imagery or video to view in an immersive space.
Here’s a few stills from Alien Action by Ralph Heinsohn.
Some city images
What really felt good was watching some fisheye home movies that I shot several years ago in Canada and down the canal at Swarkestone.
Dome Sweet Dome. The current setup of this is pretty much as rough as it gets, so being able to enjoy this suggests that there’s much more to come. A better projector is top of the list, a bright one, with lens shifting and lots of pixels. I think I’ll set the mirror back a bit further and avoid projecting onto those blinds.
Here’s an example of the conversion from fisheye to warped image and finally the undistorted (still quite rough) projected image.
Here’s VORTEX projected into the front room:
This method can also be applied to dome projection. There are a number of off the shelf solutions to warp fisheye imagery for projecting onto domes using spherical mirrors such as WarpPlayer for Mac and Domeview for Windows. These and many of the others are capable of warping video in realtime.
It is possible to create a standard fulldome viewing environment by simply exchanging the room geometry in the above example for a dome. To create a standard fisheye projection, place the fisheye projecting light in the centre of the dome. It is possible to change the viewing position by moving the light to the desired position and angle.
In this example I’m using 3ds Max but the principles of this method should be applicable to any ray tracing 3d software package.
A zdepth pass could be used to balance the brightness of the projection across all surfaces regardless of their distance from the projector. This also goes for brightness variation due to differences in the angle of the surface from the viewing position.
Darker surface materials could be used with a brighter projection to improve contrast and brightness.