Some time ago Paul Bourke sent me some images he captured with the Red Scarlet and a 4.5mm lens. The result is really impressive. He can get a recording in crystal clear 4K at 30fps. Below you can see one of his images:

Red Scarlet sample photo – credits Paul Bourke + synthetic elements by yours truly

Wait, what is Suzanne doing there?

Ok, that’s not really his original capture. I wanted to explore how would be to insert virtual elements in a fisheye image. It shouldn’t be much different than integrating synthetic elements in a panorama (topic of some previous blog entries and a paper waiting for approval 😉 – more on that later this year ). And as it turned out, it’s ‘straightforward’-ish enough.

First take a look at the original image:

Red Scarlet sample photo – credits Paul Bourke

This is a cropped image, expanded vertically to fill the 180 FOV (field of view). This arrange of camera+lens+4k doesn’t give you a full frame fisheye nor a circular fisheye. As a curiosity, the Red Scarlet can get a complete 180 fisheye circle if the photo is made in 5k. However you can’t get a 30fps movie capture at that resolution.

In order to use the IBL Toolkit for the scene reconstruction I first generated a full panorama (360×180) out of the original fisheye photo. I used the open source tool Hugin for that.

Be aware that Hugin has a bug in the calculation of the  focal length multiplier for equisolid fisheye lens (basically it’s using the full frame fisheye calculation for all its fisheye modes). Actually if you know someone involved in Hugin/Panotools project, I would send her/him over this patch. As far as I can tell the fix is along these lines. I couldn’t manage to compile Hugin though, so I don’t feel like sending a not-working patch for their tracker.

Back on topic … this is the image I got from Hugin (using 4.5 as lens and 2.35 as scale factor for equisolid – 2.35 was eyeballed because I couldn’t find in the internet the sensor size of the 4K capture for the Red Scarlet, and remember, once they fix the software the input would have to be different):

360×180 fullpanorama

 

Once I got the full panorama the rest of a piece of cake. This scene is perfect for the IBL Toolkit (this square in the front plane is practically screaming “Calibrate with me !!11!!”).

Blender IBL Toolkit in Action

And a render from a different angle (where glitches are expected). I used the Project UV option of IBL Toolkit to project the corresponding UV in the panorama to the subdivided meshes.

Extra ‘render’ – more a behind the scenes shot instead

 

Final considerations:

  • I really wish Blender had a shadow-only shader to help integrate support meshes, synthetic elements and a background plate.
  • I’m pretty sure Blender Institute crew worked nicely around that for the Tears of Steel project. I’m still waiting for them to finish the movie and release the files though.
  • The lighting is indeed bad here because the original plate was a LDR, not an HDR, so I didn’t have the lighting of the scene (and didn’t want to bother recreating it – thus you see no shadow in the original scene support elements).
  • If I had the HDR I would use Luxrender (AR Luxrender actually) for the render 🙂
  • IBL Toolkit should be called Pano something instead, anyways 😉
  • I forgot to say that the final render was only possible due to the Fisheye Lens in Cycles, a patch that I wrote on top of Brecht’s original full panorama code and is already on trunk (and will be available in Blender 2.64).
  • In fact I’m sure I could have fisheye implemented as an input option for the IBL Toolkit (discarding the need of Hugin). That would help to output the content in the exactly same position as the original camera (if you put them side-by-side you can see they have a slightly different orintation).

I’m planning to present a complete framework for working with panoramas and virtual elements in the Blender Conference this year. Even though this is based of my work with Aldo Zang (using Luxrender and not Blender) I think it can help to inspire possible solutions for Blender. So finger crossed for the presentation to be accepted and I hope we can make it interesting. The original paper (submitted to CLEI 2012 goes by the name:

Production framework for full panoramic scenes with photo-realistic augmented reality

 

So stay tuned, (and enjoy the Summer, Vancouver is finally sunny o/)
Dalai

(thanks Paul Bourke for authorizing the re-use of his image, consider giving his website a visit, its one of these corners of the internet that will keep you busy for a long time)