Why camera calibration is so important in computer vision

https://www.opencv.ai/blog/camera-calibration

by zubkovnn • 2 years ago

99 86 2 years ago

zubkovnn• 2 years ago

he cameras are used to capture the world around us. But frames are just representations of the world, and the actual relationship between the flat image and the real thing is not always obvious. In order to extract meaningful insights about the real objects from the images, developers and users of computer vision systems need to understand camera characteristics — with the help of the calibration solutions

dheera• 2 years ago

Hot take: Offline camera calibration is largely unneeded anymore. As long as you have an approximate starting point, collect about 5 minutes of data from all sensors and you have everything you need to reverse solve for the calibration.

Not saying it's easy, but solving it in software is O(1) and solving it by calibrating each device is O(N), and allows you to be resilient to things like temperature deformations and other nasty things that can happen in the field.

jiggawatts• 2 years ago

There have been papers published about making robots and autopilot-like systems robust to all sorts of "system degradation" such as obstructed optics, missing signals, noisy signals, etc...

I can't find the reference any more, but one particularly impressive result was a four-wheeled robot that could tolerate the loss of a wheel, distortions of the entire chassis, and multiple faulty sensors all at the same time!

midjji28• 2 years ago

Well, its good that it works, but its intended for correcting for hardware defects, not hardware engineer defects.

ryandamm8• 2 years ago

I agree, and I don't think this is a particularly hot take; if you've been working with camera systems, you've seen calibration without explicit calibration routines get better and better over the past few years.

For critical imaging, you still want to calibrate, but the gap between explicit calibration and "implicit calibration" (just using your subject matter) is definitely getting smaller, and quickly.

midjji28• 2 years ago

I disagree strongly, getting a coarse calibration which could be used for some stuff is one thing, getting a accurate one is another.

doctorpangloss• 2 years ago

Nobody is going to sit around and wait for your app or website to collect 5 minutes of data for calibration.

The browser doesn't let you collect all this sensor data.

Mobile Safari doesn't tell you the known camera calibration.

The Apple Vision Pro doesn't give you any data at all.

Nonetheless, 8thwall's XR8 takes seconds to calibrate absolute scale. It has to do device lookup for approximate calibrations for your iPhone model.

jfim2• 2 years ago

Depends on the use case. A car that requires five minutes of driving to calibrate its cameras once after exiting the factory would be completely acceptable. A drone that has a pre flight calibration when one first takes it out of the box would be okay.

tomp2• 2 years ago

Do you know what does 8thwall use for SLAM? is it OrbSLAM or something else?

midjji28• 2 years ago

Calibrate absolute scale to a margin where people can no longer tell most likely. Its a huge difference between that and you can do stereo at half a kilometer.

midjji28• 2 years ago

I guess you havent actually tried to do this, or you would emphasis the crucial important of, you still need a good initial guess, you still need to know the sync, and you still need data from the right environment with the right motions, and in the end, it will be worse.

ryandamm8• 2 years ago

As a camera nerd, I have to weigh in with some minor annoyances: there are errors in this article that reflect common misconceptions or elisions that computer vision engineers often believe in. Let me debunk a few...

"Cameras, by design, also introduce a level of distortion": distortion is an artifact that is usually minimized in lens design if you are trying to create a rectilinear lens, though you are typically forced to trade it off versus other design criteria (like resolving power, or optical complexity). If you're trying to design a rectilinear lens, you explicitly try to remove any distortion; for "normal" focal lengths (focal length ~= image circle diagonal) you can often get vanishingly low distortion, even with relatively simple lenses. (There was a 19th century lens design that achieved "zero" distortion with just a few elements, and a precisely-chosen aperture position; the name escapes me at the moment.) If you are not designing a rectilinear lens, there are other lens mappings, in which case it's not really proper to describe the effect as distortion, though in technical literature it's often still described this way. (For further reading, check out F-theta vs. F-tan-theta lenses: https://en.wikipedia.org/wiki/Fisheye_lens#Focal_length; https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10....)

"... the proportions of objects closer to the edges of the frame are distorted." This is technically incorrect; objects subtend the same number of pixels in an f*theta fisheye image whether they're at the center of the frame or at the edge, for a given camera-subject distance. The apparent distortion exists because you're viewing it further away than the focal length implies, so the viewer at a "normal" viewing distance is applying a transform to the image that is not distance-preserving. If you get extremely close to the image, and if it were displayed on a curved surface (so the image's angular extent to the viewer matched the camera's FOV), you'd see no distortion. Also, we think of arranging subject matter for, e.g., a portrait, along a plane that's orthogonal to the camera's axis -- but then people closer to the edge are farther from the camera, and would naturally appear smaller. It's a weird artifact of rectilinear lenses that they're actually magnified in the resulting image so people standing in a line are all equal size; to get the same effect with a fisheye lens, you just have people arranged in an arc centered on the camera. (Most portraits are not shot on fisheye lenses though, for very good reasons, so this is more of an edge case.)

Related: "there are no straight lines" -- if you're using a camera to make an image, you're operating in projective space. Whether a line is straight in 3D space, and is then straight in the projective space, is a function of your projection. Rectilinear lenses -- for which the relation f*tan(theta) applies -- straight lines remain straight. But for most (all?) other mappings they do not; for example, f*theta lenses preserve angular extent: an object that subtends a particular angle from the camera's perspective will occupy the same number of pixels regardless of where they appear in the image. This is arguably superior for many computer vision tasks.

... I need to stop here, because this is already getting tedious. For anyone interested in developing a deep intuition for lenses and imaging, I would recommend playing around with a view camera, understanding the Scheimpflug relationship (https://en.wikipedia.org/wiki/Scheimpflug_principle), and really thinking through imaging from first principles if you want to understand these things more deeply.

Or just, you know, use Colmap and don't look back. That works too.

dcanelhas• 2 years ago

Mathematically, the camera most often used in computer vision is a pinhole camera. When we talk about "distortions" I think it's usually with regards to how the real device systematically deviates from that model.

Calibration in this context is essentially the task of finding the optimal parameters of some (usually nonlinear) function (u,v)=f(x,y) that remaps positions in the original image frame to a rectified frame, where all straight lines in the world appear straight in the image. Technically, a skewed and squashed image would also fulfill those requirements, too. But this is a customer-oriented blog post, to give someone enough of an understanding to convince them of the importance of calibration, it's not a rigorous technical paper, so I actually think it's fine to skip/simplify some details.

ryandamm8• 2 years ago

You are, of course, absolutely right. I nonetheless think it may have value for some people to understand where the deviations from that simplifying assumption lie, and at least understand the stakes.

Now, I may be biased, because I work in imaging-for-humans, and I've had many a conversations with engineers about why a particular simplification doesn't work for, e.g., filmmakers, but I think that even for purely technical disciplines, understanding the assumptions that go into the pinhole model can be useful. At the margins. Which sometimes matter.

ryandamm8• 2 years ago

A specific example of where it's useful to know the underlying mechanics: for very wide angle lenses, you will typically get brightness falloff at the edges due to a phenomenon called the cos^4 phenomenon (https://nvlpubs.nist.gov/nistpubs/jres/39/jresv39n3p213_A1b....).

This is often elided by camera systems, that apply a gain to peripheral pixels to correct for this phenomenon. If you understand imaging, you will expect that, and understand why, for example, your wide angle lens displays a lower signal-to-noise ratio for a given illumination value than you might otherwise expect at the edges of the image.

This is a really specific example, but there are dozens. Imaging is its own deep, technical field that is abstracted, and occasionally obscured, by the pinhole model.

tripletao• 2 years ago

The "rectilinear" or "f*tan(theta)" projection mentioned in the grandparent comment is equivalent to that pinhole camera. The former name is because the pinhole camera preserves straight lines, as noted. The latter is because if theta is the angle between the incoming ray of light and the lens's optical axis, and f is the focal length, then the pixel illuminated by that ray of light is at a distance f*tan(theta) from the center of the imager.

That rectilinear projection is indeed the most popular choice, but all projections involve tradeoffs as the FOV gets bigger, in the same way that all planar cartographic projections involve tradeoffs as the depicted region gets bigger. For example, the magnification of objects at the edges of a rectilinear projection gets extreme as the FOV approaches 180 degrees, and the projection stops existing entirely at or beyond that. That magnification is sometimes called "perspective distortion", even though it's inherent to the rectilinear projection.

Wide-angle lenses or multi-lens arrays will often deliberately choose f*theta instead, to avoid that "perspective distortion" or support FOV >= 180 degrees. Other projections (e.g. equirectangular) are also used, especially for stuff like panoramas and VR. The concept of distortion is meaningful only with respect to a desired baseline, which is often but not always rectilinear.

backes3• 2 years ago

When we talk about "distortions" I think it's usually with regards to how the real device systematically deviates from that model.

IMO this is the correct definition of distortion. However, as the parent comment said:

If you are not designing a rectilinear lens, there are other lens mappings, in which case it's not really proper to describe the effect as distortion, though in technical literature it's often still described this way.

I think many people confuse mapping and distortion. When a fisheye lens is used, it's often seen as "heavy distortion". But a more accurate way should be to say that it's a different mapping/projection, and the _distortion_ a calibration measures is the difference of this ideal projection (ftheta (e.g. Kannala Brandt) rather than fthan*theta (pinhole) ), and the actual image. This can be a minuscule amount. This means that, "undistorting" a fisheye image doesn't give you a rectilinear image, but still a fisheye image. You can of course decide to map the undistorted fisheye image to a rectilinear one, but that's conceptually a different operation than (un)distortion.

midjji28• 2 years ago

Thank you for trying, I thought about it myself, after all someone has to teach people so I dont get another, yeah we collected the dataset with a calibrated camera rig and here it is. What do you mean what was the calibration?

ec109685• 2 years ago

This seems like a temporary problem as AI models improve. If we can figure out how to extract meaning from distorted images, AI will be able to as well.

OCR for distorted text is a solved problem. Relatively simple math based on normalizing text size and shape can be used to calculate how to reverse distortion.

ryandamm8• 2 years ago

If calibration is a temporary problem, it's because the people building those AI models understand imaging really deeply, and engage with some of the ideas in the linked post. To put it another way: if you want to build AI models that understand the world in 3D via 2D images, you should probably understand projective geometry, camera calibration, etc, extremely well.

lelag• 2 years ago

Or does it ? Drawing on a parallel from my experience in NLP during the early 2000s, the field was largely dominated by linguists trying to grok language and structuring rules manually. However, the most significant advancements came when we shifted towards using massive datasets to train models, without requiring explicit, deep linguistic knowledge.

Similarly, maybe the next frontier in 3D vision is employing large datasets to train a black-box AI without us having to understand and get the math right.

midjji28• 2 years ago

Thank you for keeping us old school computer vision guys in buisness. If deep learning folks actually used good calibration, and didnt constantly argue eh an offset of a few pixels does not matter, they might actually be competetive with classic methods by now. Well competetive accuracy wise, but not performance wise at least. Most VO systems localize a camera before the images have been uploaded from ram to the gpu, and deep learning will have a hard time getting around that kind of performance limit.

KaiserPro• 2 years ago

OCR for distorted text is a solved problem

I mean yeah, but thats because text is a fairly predictable, simple thing to extract. It has sharp edges, limited number of lines, and is neatly spaced relative to each glyph. Its also mostly high contrast.

Structure from motion however has none of those luxuries. for "AI" to overcome that, it would require a model that has an innate understanding of the scale, shape and orientation of most objects in the world. Not only that, it'd also need to have an innate understanding of lens distortion to work out if the object is bent because of the image, or the object, or some other effect.

I look forward to you releasing your model that does all this.

alexcnwy• 2 years ago

i've done loads of computer vision projects and they live or die by 2 things:

1. camera calibration 2. lighting

Lighting is the easy part

imtringued• 2 years ago

I once messed up and input the RGB values in the wrong order. This lost me 2mIoU in evaluation. If anything it was a pain in the ass to figure out, because the error was so subtle that I thought maybe I was imagining it. So I don't believe that camera calibration is important anymore. What you need is to have a dataset taken by a variety of cameras.

midjji28• 2 years ago

Heh yeah that most deep learning stuff degrades massively if you switch from one camera to another model is telling. But its still important for most things that arent trivial dl.

dima554• 2 years ago

Calibrating cameras is still important; it's only "mostly irrelevant in 2024" if you don't care about accuracy. Incidentally, tools like opencv and their ilk are also what you use if you don't care about accuracy. Modern tools like mrcal (https://mrcal.secretsauce.net/tour.html) are essential if you're trying to do long-range stereo or use wide lenses or have calibration instability or any number of other ever-present issues.

seventytwo• 2 years ago

What makes this better than OpenCV?

dima554• 2 years ago

Uncertainty propagation. Richer models that fit better. Lots of feedback and metrics and visualization to evaluate the quality of the solve. Flexibility of the tool. Documentation.

why_only_15• 2 years ago

what is "mostly irrelevant in 2024" quoting from?

usui• 2 years ago

I was wondering this too, haha. Does someone believe this, and if so, who are you quoting?

ryandamm8• 2 years ago

There is a series of comments further down in the discussion thread.

midjji28• 2 years ago

Opencv is fine for basic models like the most popular one. This one is also sufficient in most cases, I have used it for subpixel 400m stereo with a 40cm base, visual odometry, high end 3d reconstruction etc, it also has fully functional variants for wide and fisheye. You just have to select the right model, calibrate without observability issues, verify the calibration, understand the camera, etc. The better tools help a little bit, but they do not eliminate the need to understand what you are doing in the slightest. The better tools are easier to use, but generally not more accurate. The spline field variants are more accurate if you know what you are doing and the right conditions apply, but much less robust if you dont.

Opencv calibration is hard to get right if you dont know what you are doing, and at first glance it will look right despite being irrecoverably bad. I had to edit the docs and tutorials because the examples they had were of FAILED calibrations. Visualizing the resulting distortion field is critical to understand if the calibration succeeded, and that s what the better tools provide. That said, if you dont know you are looking at, they likely dont help, and the golden rule is if someone calibrated something and said it was easy, they didnt. If someone has a pdh in camera calibration said they calibratied them but that they havent used them for sfm but they plan to soon, odds are 20% its right.

jvanderbot7• 2 years ago

Mrcal was written and verified for more difficult stereo tracking than that, approximately by a 10-15x factor (in, say, fisher information sense). It's a shame it doesn't have a better data set for verification but I expect there are reasons for that as well. Not to say that other tools are completely inappropriate for this domain, obviously, or denigrate anything or anyone.

backes3• 2 years ago

if you don't mind, I have two questions about your comment. Why is sfm relevant to assess the quality of your calibration? What does the distortion field tell you besides using the wrong distortion model or having an unbalanced data set? (e.g. not having enough samples at the borders of the image)

jvanderbot7• 2 years ago

For those curious, keep in mind that the use case for open cv has been almost exclusively indoor, close range, or otherwise loose constraints. Graph models for localization famously make use of quite primitive features but tons of observations over time to produce excellent estimates and loop closures.

There are applications that require pulling almost magical levels of signal from very small numbers of pixels, and for those extremely accurate results you really need to drill down on the camera model in particular. Even modelling the environment or atmosphere to get accurate results.

For those types of applications opencv is absolutely not the right choice. It just so happens that dima has an accessible software kit that is more for the latter case. But all the better because it's perfectly fine to use this tool to calibrate any camera and then go use it with any other CV pipeline. His pain your gain.

midjji28• 2 years ago

Nonsense. Opencv has been used widely for outdoor localisation too, in particular since it is much easier to do outdoor than indoor.

Such algorithms always create a graph over the images, but if you mean the graphslam graph filter methods, those are substantially subpar compared to classic feature based methods as well as the more modern, dense and semidense methods.

moandcompany2• 2 years ago

Concur with the article's premise.

I've been working in AI/ML and some CV projects over the last decade.

I've seen far too many cases where the "algorithms" and modeling teams had no concern or even a concept for how the input systems for data that would be used to train models, and later inference, mattered to the quality of outcomes.

In CV and computational photography cases, there was little concern or understanding for how photography and imaging actually works, nor consideration for how variances in hardware, configuration, or the capture pipeline in general might affect those models when doing training data capture in parallel. Adding another layer, consideration for how variances between the capture pipeline and actual inference pipelines might need to be accounted for when designing the overall system and how to approach data collection and curation for training + evaluation, as well as to build-in robustness. (similar thoughts apply to concepts like bias and fairness in models)

Example: training vision models using one set of imaging hardware and configurations while applying those models on very different imaging hardware with different characteristics.

To summarize the above, not caring about calibration in CV is like not caring about how variances in feature extraction/embedding generation will affect the overall quality of your results.

jvanderbot7• 2 years ago

I've noticed a really fun trend at the companies I've worked with. 10-15 years ago stereo vision was pretty reliable for feature tracking or dense navigation. Pose graph methods were stellar. A new grad student could make a pretty decent pipeline themselves. Then detectors became AI/CNN driven and stereo tracking became really good. Like radar good. Again, couple knowledgeable grad students, hard studiers, or PhD could do magic.

But then it all crashed. Somewhere along the line we lost something. My last job (not Shield), I watched the estimates for a truck jump in and out of existence at 50m range with 2m baseline. Its orientation flickered 30degrees or more. But because they had used an EKF somewhere, they claimed it was the best they could do and anything else was inventing signal. I haven't dug into state estimation in the last five years but it's not gotten better if this is what new grads believe.

midjji28• 2 years ago

Very few know the old methods. Few were trained, and deep learning started working just 5 years after classics got good and easy enough that a few people could use it for a product. So there was never the needed wave of non deep learning based computer vision students, they all went to deep learning.

Many of the libraries are not well maintained. I needed a basic homography estimation recently and asked a minion to try opencv for it before anything more advanced. He got it working, but the best inlier ratio for every feature he tried with default parameters was 3%. The images were offset by 30 pixels left right, and less than a pixel in warp and taken with different exposure time. So he used sift...

He argued he got it working and that was as good as keypoint matching got... If I hadnt happened to be the guy who needed it one step up, it could have just propagated, someone adding a shitty ekf to make it smooth, then buried in layers of heuristics and api.

moandcompany2• 2 years ago

Part of this is a corruption in the field/industry that results from greed and chasing a gold rush.

Many "AI/ML" teams have been doing more alchemy than science.

One memorable team "revelation" was when they figured out the hard way that many objects and human subjects, in particular, register completely differently based on the imaging spectrum used (e.g. human visible light, versus near infrared wavelengths).

People familiar with the "old ways" probably would have considered this to be obvious, but it seems that the "new ways" are often correlated with an absence of domain knowledge related to what the "new ways" are being applied to. This isn't a new story, and history seems to repeat itself.

anArbitraryOne• 2 years ago

Yeah wow who would have thought that calibrating a sensor is important for the algorithms that process its output. Edit: typo

horsellama• 2 years ago

Is there a good resource I can read about the last point in the article?

We don't know where the camera will be. For example, we ask players to use the cameras of their mobile devices to explore the size and proportions of a room — to work with the augmented reality helmet. We don't know anything about the rooms, and we can't ask users to use a pattern.

Are there automatic techniques to tackle this problem?

midjji28• 2 years ago

Yeah, you can do live simultaneous calibrations and reconstructions and localizations. Its a difficult problem, and a slightly different one for each phone, but there are plenty of variants which work for single phone models. The imu is key to making it possible though.

What makes it harder is that older phones have insufficient processing power, whereas newer phones have too many cameras. The phones also do post processing in undocumented ways that are impossible to disable. There is also rolling shutter. This in turn can be simplified down again by just subsampling the images to 480p and estimating camera parameters per image.

midjji28• 2 years ago

I do not think there is an off the shelf solution aside from e.g. arkit.

In general a good starting point for such a problem would be to implement https://grail.cs.washington.edu/rome/ on your own.

Its not particularily hard, and there is no difference between reconstructing from uncalibrated cameras and reconstructing from many images from one camera with unknown calibration. Then you can check if the calibration is unchanged over time, and if it is, add that as a constraint. So far it will still take half an hour to run, but now you can add in tracking exploiting motion prediction and imu, which in turn will give you the real time speed.

ryandamm8• 2 years ago

This is just describing SLAM (https://en.wikipedia.org/wiki/Simultaneous_localization_and_... it's a deep field and there are many techniques and tools.

Extrinsic calibration (where is my camera in 3D space, and how is it oriented?) being different from intrinsic calibration (how do the pixels in my image map to ray directions relative to the front nodal point of the lens?).

whiplash4513• 2 years ago

You can also do some navigation guidance without calibration. Old work but still relevant today (ICCV’09)

https://www.semanticscholar.org/paper/Body-relative-navigati...

jvanderbot7• 2 years ago

can.

Auto calibration/calibration in the loop is not really "no calibration".

whiplash4513• 2 years ago

Indeed. The most interesting bit here is loose calibration, not no-calibration.

zokier• 2 years ago

For example, this is how they calibrate the camera on the robotic arm of the Mars rover.

I'm pretty confident that all cameras used in space exploration missions have been thoroughly characterized on Earth labs before launch.

For example here is paper describing the calibration activities for Curiosity rover: https://agupubs.onlinelibrary.wiley.com/doi/10.1002/2016EA00...

rurban• 2 years ago

Pre-calibration on earth is totally irrelevant if the conditions in space are totally different then. You need to calibrate in space, for different temperatures, pressure, heavy mechanical stress from the liftoff, a.s.o

midjji28• 2 years ago

This is correct. They were calibrated in advance.

midjji28• 2 years ago

Camera calibration is almost always underestimated and a source of both failure and algorithm degradation. Shit in, shit out. Keep the model simple, but actually understand it. Make sure its observable. Visualize and verify.

bmitc• 2 years ago

For the same reason that calibration is so important for any sensor.

wdfx• 2 years ago

Weird that the mars rover has a coin embedded as a scale reference instead of a printed ruler scale?

jvanderbot7• 2 years ago

Maybe coins are easier to extract, and bonus might provide color information.

But I didn't know a coin was used either.

defrost• 2 years ago

A linear scale lacks aspect ratio, they'd want at least a right angled scale.

That said, there's a long tradition of using circles and checkerboards that goes back to early BBC TV broadcasts transmitting a tuning pattern.

The coin used is a circle, likely has "interesting" patterns, and perhaps even has raised and gouged features that the stereo paired cameras can height resolve (and calibrate against for when examining rocks).

midjji28• 2 years ago

Its just for fun.

midjji28• 2 years ago

Looking through the comments People are confusing the lens models, with the nl estimator used, with the framework for calibrating cameras. There is no such thing as the OPENCV lens model. There is such a thing as the brown lense model as implemented in opencv.

The lense model is the parametrized approximation of the lense function. Picking the right one matters for ideal accuracy.

The estimator takes observations which constrain the parameters of the lens function, and find the parameters.

The framework helps you create those observations.

Why camera calibration is so important in computer vision

Related Stories