Blog‎ > ‎

Making a panorama picture

posted Sep 24, 2012, 7:38 PM by Unknown user   [ updated Sep 24, 2012, 7:42 PM ]
Continuing with my project detailed on the last post, I've spent several time on what's become a personal quest: to find out how to make a panorama picture.

There are many possible options when representing a series of images into a single panorama. Just to mention a few:

Cylindrical projection
  • Pros
    • Easy to blend if you maintain tilt
  • Cons
    • Limited vertical field of view, to just one row of pictures
Spherical or Equirectangular projection
  • Pros
    • Extended field of view, covering the whole scene
  • Cons
    • More complex to blend the pictures
Cubic projection
  • Pros
    • Covers the whole scene
    • Less picture size
  • Cons
    • Very complex to build

So I decided to go for the Spherical projection, which consists on projecting the scene (as a sphere) into a 2D plane.

In my research I found this may be achieved through different ways.

Stitcher class from OpenCV library [1]

This was the more promising option, for it would be the easiest one to use.
  • Pros
    • Extremely simple. Build panorama in just one command.
    • It performs from finding features through blending in the selected projection by itself.
  • Cons
    • Uses SURF algorithm (patented, not allowed for commercial use).
    • Slow at searching for features, running on a smartphone
It was hard to accept, but just the first con was enough to discard this magical method.

BoofCV [2]

Then I found another library, which stated to be faster than OpenCV.
  • Pros
    • Still easy to use.
    • Java native.
  • Cons
    • It blends without warping into the spherical projection.
    • Still uses SURF.
So this was discarded too because of obvious reasons.

"How hard could it be? I'll make it myself".

Many of you sure have reached to this point in any project. I guess it's part of the well known first step of a crisis: denial. The last breath of hope before dropping out. But I think this part shouldn't be avoided, ever.

In the end, I didn't have to make it all by myself. Instead, this approach served to rethink the problem and look at it from a different point.

The first thing I did on this was to read a few chapters from the book Learning OpenCV [3], since I was sure it would help me to understand how images are represented on memory, and how to access and manipulate them directly, using the library. Then I found a very cool method for warping images called cvRemap. For input, it takes an image, and two 2D matrices. The first matrix represents, for each pixel (x, y) the X offset, and the second one, the same but for the Y offset.

While I was searching for spherical projection equations, I found an interest paper from Microsoft Research [4] which stated: 

Unfortunately for them, but fortunately for me, because the phone sensors give me the possibility to know what the actual tilt of the camera is when the picture is taken, and it applies not just for cylindrical, but also for spherical coordinates.

Now the scheme for my stitching method looks like this:

The same paper also shows the equations for converting spherical coordinates.

The understanding of this formulas took me a deep graphical analysis of the projection of the sphere on the 2D plane.

By implementing this equations I was able to merge two pictures, just by overlapping (simple translating transformation).

Finally lets list the pros and cons for the selected stitching method:

  • Pros
    • Simple blending by overlapping.
    • Does not require match feature, which considerably reduces processing time.
    • Free to use (score!).
  • Cons
    • It depends on the precision of the phone orientation sensors.

Where's your SURF now? :P