DxOMark-Sensor: Compare DSLR's Performance Six Ways to Sunday

February 4, 2009 12:07 PM

Objective measurements of RAW images are an essential basis for any analysis of digital cameras, but such measurements were neither possible nor available until now. DxO Labs has developed a new scale for digital camera image quality performance, called DxOMark Sensor, to serve as an additional tool to help photographers rank and compare digital cameras. This scale is based on three underlying metrics, Color Depth, Dynamic Range and Low-Light ISO, each one tied to a real-life photographic scenario: landscape, studio & portrait, and photojournalism & sport. (This application requires Flash™ as it uses FusionCharts.) Hours of fun sorting the data by the various metrics, including $$$.

Note that Canon models may require some translation:
300D = EOS Digital Rebel = Kiss Digital (Japan)
350D = EOS Digital Rebel XT = Kiss Digital N
400D = EOS Digital Rebel XTi = Kiss Digital X
450D = EOS Digital Rebel XSi = Kiss Digital X2
1000D = EOS Digital Rebel XS = Kiss Digital F

You may also find some errors. I appears that they have the Pentax K10D and the K20D identifications transposed, for example.

posted by spock (39 comments total)

DxO Labs has developed a new scale for digital camera image quality performance, called DxOMark...

pronounced DxOMark...
posted by shmegegge at 12:14 PM on February 4

This is a step in the right direction. I think resolution is the red herring in the digital camera world.
posted by rageagainsttherobots at 12:22 PM on February 4

You can't really boil down a concept like image quality into three numbers, though, no matter how much thought you put into crafting them. For example, they imply that dynamic range is the only important things for shooting landscapes, and on that basis they suggest the Nikon D3X. That's a fine camera, and you could certainly make fine landscapes with it, but if I want to make big Ansel Adams-style prints, there's no way it beats out a medium-format back like the Phase One P45 (listed fourth best).

I don't mean to come across too snarky, as I'm sure this will be useful to many, and it does show some interesting data all in one place. But I'd hardly call it the be-all and end-all of camera metrics.
posted by echo target at 12:25 PM on February 4

Yes, it's silly to only use those three, since in one of their scenarios (photojournalism & sport), the speed at which you can take pictures is very important. So you might be better off with a 12Mpixel camera that can shoot at 10fps than a 24Mpx camera that only does 3fps.

Still, it's interesting to compare cameras like the Nikon D80 (2 yr old) , D90 (recent), and D3X. The really big changes are in sensitivity. Color depth and dynamic range less so.
posted by CheeseDigestsAll at 12:40 PM on February 4

You've just changed the subject CheeseDigestsAll. This isn't a "which camera to buy" decision-maker. It is a way of looking at a comparative performance based upon RAW image quality looked at in three main ways: Color Depth, Dynamic Range and Low-Light ISO. Obviously there are other ways of deciding which is important to you, but given two cameras with the same FPS speed, you would presumably want the one with the best image quality (or low-light performance), for example.
posted by spock at 12:45 PM on February 4

This is also handy for people wondering if the move from an APS-C sensor to a full-frame sensor (or a 4/3 sensor) is worth the money. Or how much of a "hit" you take on noise since your camera manufacturer is now shoving more megapixel output from the same size of image sensor.
posted by spock at 12:48 PM on February 4

The really big changes are in sensitivity. Color depth and dynamic range less so.

I'd imagine that's because no one wants to buy cameras that take washed out photos, but not everyone will utilize low-light settings.
posted by filthy light thief at 12:48 PM on February 4

I wouldn't call that a change of subject but just a difference of interpretation.
posted by bz at 12:48 PM on February 4

It would be nice to see some kind of sharpness statistic in there too. Not megapixels, which are increasingly useless as a measure of anything other than how fast it will fill up your card, but actual pixel-for-pixel sharpness.

Years ago Kodak made a black-and-white-only camera with no antialiasing filter. Only 6 megapixels, but so incredibly sharp that Luminous Landscape said it could compete with 24 megapixel color arrays. How I'd love to see one of those come back.
posted by echo target at 12:56 PM on February 4

Oh goodie, another thing for photo tech nerds, wannabes, and other assorted people who care more about jerking off over these kinds of numbers than taking pictures to argue about.
posted by bradbane at 12:59 PM on February 4

Oh goodie, a thread-shitter! (But there is some truth to what you say)
posted by spock at 1:02 PM on February 4

For what it's worth, DxOMark has been roundly criticized amongst the pixel peepers out there for a variety of issues. Most notably:

- The global sensor scores arbitrarily weight each of their categories.

- Sharpness, detail, and megapixels are not considered directly.

- The appropriateness of applying their metrics to medium format cameras as they have just done by adding the Leafs, Phase One, and Hasseys into the mix recently has been scrutinized soundly. Most bring up this Luminous Landscape critique. The big issue appears to be a fundamental difference between how these cameras handle pattern noise and anti-aliasing in RAW output.

That said, if you understand it's flaws and where comparisons may fall short, it remains a very cool resource. It also blasts the hype that surrounds claims of dramatic IQ improvements that PR flacks put out with each generation of cameras from the big manufacturers.
posted by drpynchon at 1:28 PM on February 4

I don't believe any stat I can't recreate myself with common equipment. We don't need a site claiming their methodology is king, we need a digital version of Adams' "The Negative" and "Photographic Materials and Processes."

In the absence of anything more rigorous, here's their Test Protocol

It looks like they're using software to measure images taken of a garden-variety Macbeth color-checker, and a home-grown test target made with ND filters in a circular pattern against a light table. A Minolta color meter to calibrate for the studio lighting(because all flash tubes are =not= the same), and away you go.

I'd give it until next Wednesday before there's an open-source Python script that runs against these targets for your very own rating of your very own sensor. That sensor will still be influenced by the glass you slap in front of it, but that's why you, the photographer, and not some famous lab, need to be able to run your own rig through the gauntlet.

And, in response to bradbane - You can take pictures and call yourself a photographer, or you can create photographs and be a photographer. A good artist understands their tools and materials, not to compare dick-length, but to improve their craft.
posted by Slap*Happy at 1:29 PM on February 4

- The global sensor scores arbitrarily weight each of their categories.

Imagine if I'm going to create a "global boyfriend/girlfriend" score. I'm going to apply five metrics: looks, personality, intelligence, sense-of-humor, and mental health. To come up with a global score, I'm going to probably weigh those somewhat differently in approaching my overall score.

Secondly, if you distrust the global score, then IGNORE IT. Look at how cameras compare according to the individual metrics that are important to you. (Duh)

- Sharpness, detail, and megapixels are not considered directly.

This is also silly, because image sharpness and detail are related to other things (like the optics attached to the front of the camera). How are you gonna normalize THAT? As far as megapixels are concerned, you can compare cameras in the same megapixel range - which is quite useful (or you can compare based upon the sensor-size itself).

Related to that you may enjoy: Contrary to conventional wisdom, higher resolution actually compensates for noise.
posted by spock at 1:55 PM on February 4

Wow, that digital B&W looks pretty damn cool.
posted by klangklangston at 1:57 PM on February 4

sharpness and detail are related to other things (like the optics attached to the front of the camera)

This is true, but the sensor also plays a role, and if you don't consider it you're not getting the full picture. The standard way to normalize it is to use a high-quality 50mm prime. There's a good one for every lens mount, and they outresolve most sensors when stopped down properly. It's not perfect, but it's good enough for DPReview.

I'll have to read the article about resolution compensating for noise, but I'm not sure I buy it. DPReview's opinion is that it the theory doesn't live up to practice.
posted by echo target at 2:04 PM on February 4

Uh spock, tone it down. My goal wasn't to start a flame war (Duh!). The limitations are what they are. Of course one can ignore the global scores, and of course picture sharpness and detail are affected by glass (but the sensor plays a role in this). As I said, DxOmark has its uses. Also, contrary to contrary to conventional wisdom, abstract notions of how resolution might factor into signal-to-noise may not hold up in real world use.
posted by drpynchon at 2:07 PM on February 4

The standard way to normalize it is to use a high-quality 50mm prime.

Is there one that will work across all camera manufacturers' mounts and their various focal plane dimension differences? It would let you compare Canons to Canons (for instance) but not Canon to Pentax.
posted by spock at 2:09 PM on February 4

And, in response to bradbane - You can take pictures and call yourself a photographer, or you can create photographs and be a photographer. A good artist understands their tools and materials, not to compare dick-length, but to improve their craft.

I agree with you, sorry i wasn't trying to shit on the thread. I have read the Adam's trilogy (and unlike most people I've done all the tests, got out the densitometer, graphed the characteristic curve of my film and camera, etc.) so I understand the importance of knowing your tools and craft. This phenomenon of endless pixel peeping however just makes no sense to me and it is largely perpetrated by people who seem to be obsessed with everything about photography except actually making images. Sorry if I'm a little skeptical of this new way to contribute to the endless digital argument, but a scorecard for image quality just seems like more dick waving to me.
posted by bradbane at 2:16 PM on February 4

Blame the dick, not the ruler.
posted by pokermonk at 2:18 PM on February 4

bradbane..if your 'art' involves highly detailed, incredibly sharp reproductions of nature, or just whatever your material is, and you don't want to pay the increasingly high costs of staying with analog, then you are going to need to care about the technology of sharpness and RAW image quality, period. It isn't pixel peeping if you are forced to live with blurry images at huge scales because digital cameras can't record the kind of sharpness a medium or large format analog camera can. Also, people giving a shit about 'pixel peeping' is what forces the continual innovation that will get over this problem. I for one would rather to continue to shoot on film, but I get rather tired of paying the exorbitant costs of developing and printing. On the other hand, when I have to shell out crazy dollars and pull out what little hair I have left to make sure my printed image corresponds to the image I see on my monitor in photoshop, I wonder if it's a worthwhile trade off.
posted by spicynuts at 2:23 PM on February 4 [1 favorite]

Is there [a high-quality 50mm prime] that will work across all camera manufacturers' mounts?

Not necessary. Just use the camera manufacturer's high-end 50mm. If it sucks, too bad for them.
posted by ryanrs at 2:31 PM on February 4

You've missed the entire point, but thanks anyway.
posted by spock at 2:36 PM on February 4

Years ago Kodak made a black-and-white-only camera with no antialiasing filter. Only 6 megapixels, but so incredibly sharp that Luminous Landscape said it could compete with 24 megapixel color arrays. How I'd love to see one of those come back.

When you shoot RAW, there is no image processing, so there's no antialiasing filter applied (or contrast adjustment, sharpening, etc.) RAW images tend to be less sharp than the JPEGs that come out of the same camera.
posted by bugmuncher at 3:03 PM on February 4

This is also very useful for thinking about how to better use the camera you already own. I didn't know until visiting this site that the Manufacturer's ISO rating is based on the resulting image from the in-camera image processor rather than the actual ISO sensitivity of the sensor. That's important to know if your workflow is entirely in RAW, like mine.

It's eye-opening to know that with the Canon 30D (at least according to DxO), when I shoot RAW at ISO 1600, I'm actually shooting somewhere closer to ISO 1200. It explains why I don't like the built-in exposure metering at the higher ISOs, and why I've been assuming the meter was off, and compensating. Knowing that it's about a half-stop under at ISO 1600, I can let in a little more light to compensate. It matches up with what I've been doing to shoot basketball, hockey, and other indoor sports.

It's also interesting to note that (according to DxO) the ISO 50 and ISO 100 in one of the more expensive Canon cameras (1D Mark III?) are both ISO 100 in RAW. That would make the full-stop difference between the two something done entirely in the camera image processor.
posted by bugmuncher at 3:33 PM on February 4

When you shoot RAW, there is no image processing, so there's no antialiasing filter applied

Wrong and wrong. Raw (not RAW) images are processed. They have various tricks applied to them, most commonly black point clipping. It's not serious, but they are definitely massaged before they ever hit your raw interpolation algorithm.

You're wrong about the antialiasing filter as well. In front of the bayer color filter array, which is in front of (bonded to) the sensor itself, there is an optical antialiasing filter, also known as an OLPF (Optical Low Pass Filter). The purpose of the OLPF is to prevent aliasing at the sampling frequency of the CFA.

JPEG images may look sharper because they've had perceptual sharpening (eg Unsharp Mask) applied to them, along with a host of other tricks. This does not mean more absolute resolution, it's a perceptual thing which relies on opponent processes in our visual system.

I have spoken with the DxO people at several conferences and they are *top notch* imaging scientists and engineers. They know more about sensors and how they work than almost any other lab I've spoken with or read papers from. Most importantly they are deeply interested in and aware of the signal processing tricks that happen inside cameras to improve "image quality". You may not know this, but DxO technology is in almost every mobile phone out there, correcting for cruddy lenses and tiny pixels.

The standard way to normalize it is to use a high-quality 50mm prime.

You have no idea what you are talking about.

Every "high-quality" 50mm prime from different manufacturers has a different modulation transfer function and would directly affect the results.

A good artist understands their tools and materials, not to compare dick-length, but to improve their craft.

A-fucking-men.
posted by fake at 3:40 PM on February 4 [5 favorites]

I'd like to see some of the Foveon sensors in there, or some other tech which does not use a Bayer filter (or its descendants).
posted by adipocere at 3:45 PM on February 4 [1 favorite]

Also: this site isn't so much about comparing the performance of a camera so much as it is comparing the performance of the camera's sensor. If you don't shoot in RAW, most of this benchmarking information won't be of much use to you, since they're not putting any numbers on the output of the image processors built into the cameras.

(DxO Image Science sells RAW image conversion software called DxO Optics Pro. They're probably hoping that this new resource will get people that use RAW more interested in their product.)
posted by bugmuncher at 3:45 PM on February 4

I'd like to see some of the Foveon sensors in there, or some other tech which does not use a Bayer filter (or its descendants).


I agree, but I'd be interested to see how they present it, because the stacked Foveon design makes it really hard to compare to Bayer CFAs. The color channels are no longer neatly spatially independent and I've heard that it can be an interesting problem to separate them.
posted by fake at 3:46 PM on February 4

Fake, thanks for pointing me in the right direction. I had called it RAW because that's what DxO called it, and I always thought it was just a dump of what the sensor recorded. It also hadn't occurred to me that the OLPF would do antialiasing.

I didn't suggest that JPEGs had better resolution. Just that they'd been processed (and sharpened) in-camera. So if a reader plans to shoot RAW (or Raw, or raw), they should also plan to learn how to sharpen manually.
posted by bugmuncher at 3:53 PM on February 4

I didn't suggest that JPEGs had better resolution
Yep, sorry, didn't mean to imply that you did.

I have spent a lot of time trying to calibrate cameras for natural image statistics research, and you'd be shocked the kind of shit they do to the data. The popular perception that raw is a record of the photons that landed on the sensor is most often not true, even if it only means they crushed the blacks.

Raw means, basically, no color interpolation. (Sorry about the pedantry -- RAW is not an acronym (like MAC is not an acronym) and occasionally I can't help but let the pedantic BS slip.
posted by fake at 3:58 PM on February 4

The standard way to normalize it is to use a high-quality 50mm prime.

Is there one that will work across all camera manufacturers' mounts and their various focal plane dimension differences? It would let you compare Canons to Canons (for instance) but not Canon to Pentax.

You can basically do this with a T2 mount lens+adapter, from what I've read. Wiki
M42 is a similar type of mount, but apparently has issues when hooking up to Nikon bodies.

Now as for the quality of the glass that you're going to find in that type of mount, that's not something I'm incredibly knowledgeable about.
posted by agress at 4:07 PM on February 4

As a style rule, RAW (all-caps) is pretty much de facto in the industry. Canon, Nikon, Hasselblad, and DXO all use it, just to name a few. Even professional-oriented publications use the rule. The one real exception is Adobe (and I'm not sure their motive is altruistic since they would love nothing more than to have RAW be DNG). Admittedly, I'm not sure on the scientific texts.

Even though RAW is the only exception at the moment, maybe the rule is clearer as all-caps for file formats and extensions (without periods ahead of them).

RAW is definitely a strange thing, but it's what it is.

If you really need it to be a legalistic acronym, think of it as the shortened term for "raw file format" etc.
posted by pokermonk at 6:29 PM on February 4

This DxOMark stuff is a bunch of bullshit that you can safely ignore. Anyone who cites a DxOMark score in a discussion of cameras should also be ignored.
posted by gen at 7:13 PM on February 4

Thanks for ignoring this thread, gen! You're clearly a "do as I say, not as I do kinda guy"!
:P
posted by spock at 7:30 PM on February 4

Is there one that will work across all camera manufacturers' mounts and their various focal plane dimension differences? It would let you compare Canons to Canons (for instance) but not Canon to Pentax.

It's a costly option, but Zeiss makes a 50/1.4 lens with mounts for Canon (ZE), Nikon (ZF), Sony (ZA), Pentax (ZK), and M42 mount (ZS).
posted by heeeraldo at 12:11 AM on February 5

DxO Image Science sells RAW image conversion software called DxO Optics Pro. They're probably hoping that this new resource will get people that use RAW more interested in their product.

Could be. But DxO Optics Pro is actually much more than a Raw (or RAW) converter (in fact, I think previous versions didn't do Raw conversion). Its main claim to fame is correcting for aberrations introduced by lenses and sensors. It corrects distortion, vignetting, and sharpness issues introduced by the specific lens and camera combination you're using (it reads information for those from the EXIF data in the image). I've tried it, and the results can be pretty impressive (and not reproducible in Photoshop), though it only works if you're using a lens and camera combination that they've profiled. I don't want to sound like a shill or anything, but based on the results I've seen from their software, I'd say that they know what they're doing. Which doesn't mean that their DxOMark ratings aren't open to dispute, but I'd guess that the data they're based on is probably good.
posted by klausness at 3:07 AM on February 5

It's pretty pointless to test the sensor without the lenses you're going to use with it.

This isn't like testing film, where you can declare ProviaF is the best use of silver humanity has invented (except for killing werewolves), and then merrily use it in your Lomo and your Leica.

Once you pick a sensor, you're stuck with the lenses designed for the camera body that houses that sensor, as they are not interchangeable from one camera to another. This means different DSLR types by the same manufacturer! Lenses designed for full-frame sensors have a different set of design criteria than lenses designed for APS-C than lenses designed for 4/3rds.

So, you will need one "canonical" lens for each sensor size, which means, automatically, it's an apples-to-oranges comparison.

Apples-to-apples is testing the whole rig as a single unit - camera and lens.
posted by Slap*Happy at 5:45 AM on February 5

Apples-to-apples is testing the whole rig as a single unit - camera and lens.

Yes, but then you have to test every camera-lens combination. That would give you a huge mess of data that'd you'd have a hard time extracting actual information from when you're trying to evaluate a camera. What it looks like they're trying to do with DxOMark is factor out the effect of the optics to just evaluate the camera sensor. Since they produce software that evaluates the optics of lenses, I assume that they're using the knowledge from that to come up with a sensor evaluation that's independent of the lens used. Of course you're dependent on the available lenses for your camera body once you've chosen a sensor, so you need to also evaluate those when choosing a camera. But I think they're providing useful information that you can't get elsewhere (and that appears to be much more sophisticated than what you'd get by slapping a 50mm Zeiss lens on every camera and taking some pictures).
posted by klausness at 3:45 AM on February 6