Warping Fisheye Images for Spherical Mirror Projection

Posted by on Mar 14, 2013 in Fulldome | 22 Comments

Warping Fisheye Images for Spherical Mirror Projection… Or… How to make your front room into an immersive cinema.

This post doesn’t go into any of the complex mathematics involved with spherical mirror projection. For that information I recommend using the invaluable resources at http://paulbourke.net . This post explains a practical solution for projecting onto complex geometry using spherical mirrors, 3ds Max and 123D Catch.

Okay, so really I’m just trying to figure out a good way of previewing my fulldome work at home, thus speeding up the creative process. The idea is to be able to sit in a standard space, such as the corner of a room, and watch an immersive video.

This is a video showing the physical setup and some immersive video.

Real-World Setup

The main elements to consider are these:

  1. Room
  2. Projector
  3. Spherical Mirror
  4. Computer/Robotic Thinking Machine
  5. Viewing Position

Room

The area I’m projecting onto is mainly white walls, which are appropriate for the brightness of projector I have. The corner of the room should give me a good amount of enclosure to create a panoramic perspective. The area could be almost any structure, although shadows may be cast from the position of the mirror which will be visible if the viewing position differs greatly. More complex surfaces and structures will take longer to produce and require more precise measurements. These are all the elements in situ. I’m projecting a flat grid here, just to show the raw distortion from the mirror. It’s also possible to see just how much light falloff there is as the projection surface gets further away from the mirror. It would have been better to place the mirror at an equal distance away from the walls as the ceiling, but hey, I know for next time.

Projector

I’m using a terrible projector for the purpose, it’s a 4:3 800×600 with no lens shift and a dim lamp by today’s standards. Nevertheless, it’s all I have and it’s good enough for working things out. Really a 1080p projector would be a much better choice. Also, a tilt shift would help maximise the resolution and positioning of the elements to achieve optimal projection. Blah blah blah.

Spherical Mirror

The spherical mirror is an old prototype rig, gratefully borrowed off Richard Lake from Polestar Planetarium. A spherical mirror like this can be purchased from Amazon for around £70, but will have a plastic outer coating, which will cause refractive artefacts in the projection. Both of the mirrors I’m using have an external reflective coating, which is best for a high fidelity image, but extremely fragile. A secondary mirror is useful to save space but not a requirement, in fact this will actually reduce the brightness of the projected image slightly.

Computer

The laptop is a thousand year old Macbook Pro running Leopard. Nothing fancy is needed… Just lights and clockwork.

Viewing Position

The viewing position is important. I positioned a seat just behind the mirror to allow comfortable viewing of the immersive environment. The system will only really work from a one person perspective, but that perspective should be possible to move wherever.

The Virtual Setup

The main elements needed for the virtual setup are:

  1. Room Geometry
  2. Spherical Mirror
  3. Camera
  4. Fisheye Projection Light

The virtual setup is basically an exact copy of the real-world setup with the addition of a fisheye projecting light and replacing the projector with a camera.

Room Geometry

The room will need to be created with accurate measurements or things aren’t going to match up. To make this process easier I’m using 123D Catch to create a rough version of the room and then built the room with simple geometric shapes that aligned to the 123D Catch version. This process could be replaced by measuring the room and positions of all the important elements within it.

123D Catch works best with textured surfaces, plain white walls aren’t so good. So I projected some Kandinsky onto it to help with the catch.

Here’s the 123D Catch geometry. It’s pretty rough in places but only took 10 minutes and will give me the position I need for all the elements in the scene.

After exporting an fbx file and importing it into 3ds Max, it’s possible to overlay simple geometry over the catch geometry to recreate the scene more accurately.

The geometry will need to be a solid white standard material with no specular.

Spherical Mirror

For the mirror to work correctly it will need to be rendered using physically accurate ray-trace reflections. I’m using the built in Mental Ray renderer.

For the mirror I added a full sphere with Mental Ray Arch+Design chrome material on it. This must be accurately aligned to the mirror in the real world using the 123D Catch geometry as a guide. It’s worth pointing out that the mirror may not be a halved hemisphere, but rather a section of a hemisphere, so the length of the back of the mirror won’t necessarily be the diameter.

The mirror will need a dense smooth mesh for best results. I rotated the sphere so the highest density of the mesh was towards the camera, which I found did make a difference.

Camera

The camera should be positioned in the same place as the projector lens. I decided to miss the secondary mirror out in the virtual setup and position the camera as it would appear without it. This may be a little confusing, and it does require an additional step to flip the image ready for reflection, so in hindsight I would probably have set it up with the secondary mirror and kept the virtual setup and real-world setup exactly the same.

The camera FOV should match that of the projector.

The projector I’m using doesn’t have a lens shift on it but it should be possible to match the lens shift using a skew modifier on the camera and rotating the gizmo to create vertical skew.

Fisheye Projection Light

The fisheye light will project a fisheye image onto the geometry in the scene, much the same way a fisheye projector would in the real world. The light will need projection map turning on and a bitmap adding to the map slot. The light will also need specular turning off to not cause highlights in the mirror.  The map will need to be an angular fisheye to create an immersive environment when viewed from the correct position. The mapping needs to be set to Shrink-wrap environment and the tiling set to U:-2 V:2 and turn off tiling. The W angle needs to be set to 90. The light can then be positioned in the desired viewing position. Basically, place the light where you can easily stand or sit down. Keep in mind the position of the mirror and the maximum area of projection. If you place the light too far behind the mirror you will lose a large portion of the fisheye image from the projection area. I’ve placed the light at sitting height just behind the mirror to miximise the panoramic potential of the space.

Rendering

The render output should be the same aspect ration of the projectors.

Indirect illumination such as final gathering should be avoided for standard immersive projection. This will actually occur when it’s projected anyway, the next step is to figure out how to reduce this.

Before rendering any fisheye imagery it’s a good idea to test the setup. To do this I used a dome alignment grid as the light projection map. Then render the view from the camera.

This render must be flipped if you are missing the secondary mirror stage. The image can then be displayed full screen on the projector and it should look like a circular grid with even spacing between the lines as you move your head around. I actually added the two coloured squares onto the geometry of the room so I could check they were positioned in the same place in the room.

Here you can see the green and pink panels matching their position in the virtual space. You can also see that the grid is heavily distorted from this perspective.

You can see in the following image from the set viewing position that the curved lines of the grid continue as they should, even over the corner of the room. Looking at this for the first time was a pretty good feeling. Beyond the awful resolution and slight misalignment, it actually worked.

Once you know your setup works, you can render all kinds of fisheye imagery or video to view in an immersive space.

Here’s a few stills from Alien Action by Ralph Heinsohn.

Some city images

What really felt good was watching some fisheye home movies that I shot several years ago in Canada and down the canal at Swarkestone.

Dome Sweet Dome. The current setup of this is pretty much as rough as it gets, so being able to enjoy this suggests that there’s much more to come. A better projector is top of the list, a bright one, with lens shifting and lots of pixels. I think I’ll set the mirror back a bit further and avoid projecting onto those blinds.

Here’s an example of the conversion from fisheye to warped image and finally the undistorted (still quite rough) projected image.

Here’s VORTEX projected into the front room:

 

Dome Projection

This method can also be applied to dome projection. There are a number of off the shelf solutions to warp fisheye imagery for projecting onto domes using spherical mirrors such as WarpPlayer for Mac and Domeview for Windows. These and many of the others are capable of warping video in realtime.

It is possible to create a standard fulldome viewing environment by simply exchanging the room geometry in the above example for a dome. To create a standard fisheye projection, place the fisheye projecting light in the centre of the dome. It is possible to change the viewing position by moving the light to the desired position and angle.

Potential Developments

In this example I’m using 3ds Max but the principles of this method should be applicable to any ray tracing 3d software package.

A zdepth pass could be used to balance the brightness of the projection across all surfaces regardless of their distance from the projector. This also goes for brightness variation due to differences in the angle of the surface from the viewing position.

Darker surface materials could be used with a brighter projection to improve contrast and brightness.

22 Comments

  1. siddharth Prabhakar
    March 20, 2013

    hi

    really impressed with ur work…pls guide me how i can create a planetarium like 3d projection for school students in India with minimum cost using ur reflection thoery….pls eagerly waiting for ur reply.

  2. Aaron
    March 20, 2013

    Hi Siddharth

    A planetarium style dome can be quite an expensive thing to setup. There are many companies that specialise in producing cheap planetarium solutions for schools, so I would recommend searching for a company near to where you are based.

    A couple of tips:
    1. A good resolution is important for viewing astronomical imagery. Stars are small, so I’d say 1920×1080 is a minimum requirement.
    2. Most modern laptops should be able to handle playing HD movies without any issues, so don’t aim for the top specification if that’s all you require.
    3. If you want to explore some real-time software, give Microsoft World Wide Telescope a go. It’s free and is easy to set up for a mirror system: http://www.worldwidetelescope.org/Home.aspx
    4. Check out Paul Bourke’s website: http://paulbourke.net/dome/faq.html The man behind the mirror system.

  3. siddharth Prabhakar
    March 21, 2013

    thank you for the information.really useful….your promptitude is really appreciable.would need more guidance from you in the future..thanks again and keep up the good work.my friend are also very impressed with ur work

  4. @cpm5280 (chris maytag)
    March 21, 2013

    This is great stuff – thanks for posting. I’m curious enough about this that I may attempt a stab of my own in our planetarium lobby at some point, using a ‘dead’ corner.

  5. Aaron
    March 21, 2013

    Hi Chris,

    Yeah, sounds like a good idea. A curious corner. Let me know if you have any issues and I might be able to help out.

  6. Shall
    March 23, 2013

    Hello Mr. siddharth ,
    Will you guide us for full dome projection using fish eye lanse . We are confuse with diffrent manufacturers product setup and image-video warping software.
    Kindly reply on email as well.
    Regards
    Shall
    Mumbai
    India

  7. jonathan
    April 4, 2013

    Hey Aaron, this is a great approach. I wondered, did you still model a sphere for your mirror (which I presume is not entirely spherical)?

  8. Alex
    April 9, 2013

    impressive

  9. Jason Fletcher
    April 9, 2013

    Fascinating work! Thanks for writing this up and sharing it.

  10. Aaron
    April 9, 2013

    Hi Jonathan,

    Thanks. The mirror used is spherical. The physical mirror is a section of a hemisphere, which looks oddly not spherical, but it actually is.

  11. Aaron
    April 9, 2013

    Thanks Jason

  12. jonathan
    April 10, 2013

    Im going to give this a try with my dome. Is your camera orthographic?

  13. Luniere » DIY Dome – First Light
    August 30, 2013

    […] from using it as a test dome. I have to render everything twice using the method I developed for projecting into the corner of my room, but it’s pretty fast. It means I can render several camera passes and test them in one […]

  14. KC Lai
    January 8, 2014

    Hi siddharth,

    fantastic article. We need some help in ray tracing a spherical mirror project, shall be easy for you but quite steep for me. Do you interested in getting a freelance job? Please PM me for details, thanks.

  15. Luniere » v.02 Breakdown – How it all Works and That
    February 11, 2014

    […] experience a perfectly mapped immersive environment. I’ve written a post about how to this here. The big development I made during this project was a way to retain the alpha from the creation […]

  16. Jonathan
    April 23, 2015

    Hi Aaron, I’m very impressed with your work here, and have me inspired to try and replicate a similar set up. I am currently trying to project onto a flat wall. A couple questions:

    1) Did you use the meshmapper for the warpplayer to create any .cfg or .data files, to project on the corner or flat wall?

    2) How would I apply the model projections in 3dsMax into the warpplayer?

    Thanks, and looking forward to your reply!

  17. Aaron
    April 23, 2015

    Hi Jonathan,

    Unfortunately I didn’t use meshmapper for this. I have used the warpplayer before but I haven’t generated my own config files. I’m not sure how easy it would be to generate warp meshes for use in warpplayer from 3ds max. Maybe there is a more manual way of doing this, but it sounds time consuming. I’d love to hear if you do come up with a method though.

  18. Dave
    August 4, 2015

    Hi
    This is fantastic
    I have built a 737 home simulator and this would work great for my outside view.
    I would be very grateful if you could give my some more info on the build process
    Thanks
    Dave

  19. Aaron
    August 17, 2015

    Hi Dave,

    Is there a specific part of the build process you need some more info on? For the rendering side of things this post may be of use: http://www.luniere.com/2014/08/17/stereoscopic-workflow-for-immersive-cinema/

    And this may be of use for the mirror setup:
    http://www.luniere.com/2013/08/30/diy-dome-first-light/

    And ther may be something of use in these:
    http://www.luniere.com/2014/01/19/v-02-breakdown-how-it-all-works-and-that/
    http://www.luniere.com/2013/07/24/diy-dome-building-a-geodesic-monodome/

  20. Carl lark
    September 29, 2015

    Hi Aaron

    Don’t know if you can help me but I’m looking to try and use this system to project large back drop images for our school production.
    The area I want to project onto is flat and 7m x 4m. It doesn’t matter if it bleeds onto the floor of the stage, I just want it to fill the back wall so the audience can see it.
    Could I make this work?

    Thanks

    Carl

  21. Amr
    March 1, 2016

    Hey Aaron ,

    thanks God , i have my Inflatable dome now a 7M dome. i need to decide which projector will buy , i have the option of ( Sony VPL-DX127 – 2,600 Lumens XGA Desktop Projector ) is it a good one ? with 60cm Spherical mirror

  22. Emily Jane
    February 27, 2018

    Thank you so much for this tutorial! I am an artist working with immersive video projection within galleries which tend not to be domes and this is the first time I’ll be trying something with a convex mirror.

Leave a Reply