 |
| I compare a lot of lenses. They aren’t all exactly the same. |
In today’s article we’ll look at variation versus bad copies a bit differently to last time. Plus, I’ll explain how people get three ‘bad copies’ of a lens in a row.
Variation versus bad copy frequency
Imatest type graphs are easier to visualize so I’m going to use those today. These graphs allow us to visualize center resolution (toward the top on the y-axis of the graph) and overall resolution (toward the right on the x-axis), with individual lenses plotted as dots. Don’t worry about the numbers on the X and Y axes, all you need to know is that the sharpest lenses are plotted up and to the right, and the softest are lower and to the left.
The graph below shows plots from multiple copies of two prime lenses. Let’s call them ‘Red’ and ‘Green’. The Green lens is a fairly expensive, pro-grade optic. The Red lens is a cheaper, consumer-level prime. You’ll see that there’s one copy of each in roughly the middle of this graph, away from the main cluster at upper-right. I’d return both of these samples to the manufacturer. So would you – they’re awful.
 |
| Multiple copies of two lenses, the ‘Red’ lens and the ‘Green’ lens, plotted by center and overall sharpness. Two bad copies of each are obvious at the lower left. |
But could you tell the difference between the best and the worst of the other copies, in that big cluster at upper-right? That would depend on the resolution of your camera, how carefully you pixel-peeped, which lens we are talking about, and honestly, how much you cared.
The Green lens shows less variation, which is about what we expect (but don’t always get) from a fairly expensive, high-quality lens. A perfectionist with a high resolution camera, some testing skill and enough time could tell the top third from the bottom third, but it would take effort.
The Red lens has more variation, which is typical for a consumer-grade lens. A reasonably picky photographer could tell the difference from the top third and the bottom third. None of the bottom third are awful; they’re a little fuzzier, a little more tilted, not quite as good when viewed at 100% magnification, and you might see issues if you made a large print.
With more variation, you get more ‘not as good’ lenses, but they’re still not ‘bad copies’
If you look carefully, though, the top third of the Green and Red samples are about the same. With more variation, you get more ‘not as good’ lenses, but they’re still clearly not ‘bad copies’; they’re just ‘not quite as good’ copies.
So why would we argue about these two lenses on the Internet? Because based on a graph like this, a lot of testing sites might say “Red is as good as Green and costs a lot less.” The truth is simply that the Red lens has more variation. Sure – a good copy of the Red lens might match a good copy of the Green lens. But you’re not guaranteed to get one.
A word about that yellow line and worse variation
There’s obviously a point when large variation means the lower end of the ‘acceptable group’ is unacceptable. Where that line lies is of course arbitrary, so I put an arbitrary yellow line in the graph above, to illustrate the point. Where the yellow line is for you depends on your expectations and your requirements.
The Subjective Quality Factor can theoretically decide when the low end of variation is not OK, and it can be used as a guide to where to place the yellow line. The key words, though, are ‘subjective quality’. Things like print size, camera resolution, even subject matter are variables when it comes to deciding when SQF is not OK. For example, the SQF needed for online display or 4K video is a lot lower than for a 24″ print of a detailed landscape taken with a 40 megapixel camera.
Every one of us has our own SQF; call it your PQF (Personal Quality Factor) and your yellow line might be higher or lower than the one in the graph above. Manufacturers have a Manufacturer’s Quality Factor (MQF) for each of their lenses, which is the famous ‘in spec’.
When your PQF is higher than the MQF, those lower lenses are not OK for you. They might be fine for someone else. Wherever a person’s yellow line is, that’s their demarkation line. These days, if they get a lens below the line, they go on an Internet rant. So now, as promised, I have explained the cause of 8.2% of Ranting On Online Forums (ROOFing). It’s the difference between MQF and PQF.
Put another way, it’s the difference between expectations and reality.
 |
| If you test a set of $ 5,000 lenses carefully enough, you may find some differences in image quality. The technical term for this phenomenon is ‘reality’. |
It should be pretty obvious that people could screen three or four copies of the Red lens and end up with a copy that’s as good as any Green lens. I don’t find it worth my time, but I’m not judging; testing lenses is what I do.
Unfortunately, though, people don’t post online “I was willing to spend a lot of time to save some money, so I spent 20 hours comparing three copies and got a really good Red lens.” They say “I went through three bad copies before I got a good one.”
The frequency of bad copies and variation
Just so we get it out of the way, the actual, genuine ‘bad copy’ rate is way lower than I showed in the graph above. For high-quality lenses it’s about 1% out-of-the-box. This explains why I roll my eyes every time I hear “I’ve owned 14 Wonderbar lenses and they’re all perfect.” Statistics suggest you’d need to buy over 50 lenses to get a single bad one. The worst lenses we’ve ever seen have a bad copy rate of maybe 3% so even then, the chances are good you wouldn’t get a bad one out of 14.
Most of these ‘those lenses suck / I’ve never had a bad copy’ arguments are just a different way of saying ‘I have different standards than you’
What about the forum warrior ROOFing about getting several bad copies in a row? He’s probably screening his way through sample variation looking for a better than average copy. If he exchanges it, there’s a good chance he won’t get a better one, but after two or three, he’ll get a good one. So he’s really saying “I had to try three copies to find one that was better than average.” Or close to average. Something like that.
Semantics are important. Most of these “those lenses suck / I’ve never had a bad copy” arguments are just a different way of saying “I have different standards than you”. I get asked all the time what happens to the two lenses John Doe returned when he kept the third? Well, they got re-sold, and the new owners are probably happy with them.
Why are there actual bad copies?
In short – inadequate testing. Most photographers greatly overestimate the amount and quality of testing that’s actually done at the factory, particularly at the end of the assembly line.
Many companies use a test target of thick bars to set AF and give a cursory pass-fail evaluation. A target of thick bars is low-resolution; equivalent to the 10 lp/mm on an MTF bench. Some use a 20 lp/mm target to test, and 20 is higher than 10, so that’s good. The trouble is that most modern sensors with a good lens can resolve 50 lp/mm easily. This is what I mean when I say (as I do often) that you and your camera are testing to a higher standard than most manufacturers.
Why is there high variation?
Usually, it’s the manufacturer’s choice, and usually for cost reasons. Occasionally it’s because the manufacturer is living on the cutting edge of technology. I know of a couple cases where a lens had high variation because the manufacturer wanted it to be spectacularly good. They designed-in tolerances that turned out to be too tight to practically produce, but convinced themselves they could produce it. Lenses like this tend to deliver amazing test results, but then attract a whole lot of complaints from some owners and a whole lot of love from others.
What’s that? You want some examples?
 |
| This is not the bookcase mentioned below; that one is under nondisclosure. This is my bookcase. My bookcase has better optical books. |
Service center testing
Years ago, we had in our possession a $ 4,000 lens that was simply optically bad. It went to the service center twice with no improvement. Finally, the manufacturer insisted I send ‘my’ camera overseas with it for adjusting. The lens and camera came back six weeks later. The lens was no better, but the camera contained a memory card with 27 pictures on it. Those pictures were of a bookshelf full of books, and each image was slightly different as the technician took test shots while they optically adjusted the lens.
This, my friends, is why we decided to start adjusting lenses ourselves. And yes – after offering to share those bookshelf images – I was eventually sent a replacement lens.
Non-adjustable lenses
Many lenses have no optical adjustments. They’re assembled, and then what you get is what you get. If in-factory QC detects a really bad one, it might be disassembled and the parts reused, in the hope that random reassortment gives a better result next time. Or it may just get thrown away; the cost of disassembling and reassembling may be greater than the saved parts.
A common type of non-adjustable lens called a stacked lens; ‘element – spacer – element – spacer, etc’ with a front and rear retaining ring holding everything together. The usual method of correcting it is to loosen the retaining rings, bang the lens on a table a few times, and tighten it back up. That probably sounds ridiculously crude, but it sometimes works.
Many fully manual lenses (not those made by Zeiss or Leica) are non-adjustable, as are some less expensive manufacturer and third-party lenses.
Minimally-adjustable lenses
A number of prime lenses have only one or two adjustable elements. This is not necessarily a bad thing; adjusting one or two elements is a lot easier than adjusting six, so the technician is more likely to get things right.
One of my favorite lenses, both to shoot with and to adjust, is the venerable Zeiss 21mm F2.8 Distagon / Milvus. The front element of this lens is adjustable for centering and we’ve done hundreds of these adjustments over the years. The fun part is doing this adjustment lets you choose what type of lens you want. You can have razor sharp in the center with soft corners or you can let the center be a little softer and the corners much sharper. It’s a great example of adjustment being a trade-off, even for relatively simple adjustments.
 |
| MTF graphs of a Zeiss 21mm F2.8 Distagon, adjusted for best center sharpness (above), and optimal edge sharpness (below). |
Consumer-grade zoom lenses (manufacturer or third-party) and prime lenses with apertures smaller than F1.4 tend to be minimally or non-adjustable. A fair number of better zooms and primes are minimally adjustable, too.
Lenses with many adjustable elements
More adjustments means less variation, at least in theory. It also, however, means when something is wrong it’s far more complex and time consuming to get the adjustments right. Time, as they say, is money and complex lenses can be rather hard to adjust.
I think the most we’ve seen is nine adjustable elements. These are usually top-of the line zooms, but we’ve seen six adjustable elements in some top-end primes. That’s something we never saw even five or six years ago.
So, what’s the key takeaway?
Let’s start with my definitions. A bad copy of a lens has one or more elements so out of adjustment that its images are obviously bad at a glance. Such a lens (assuming it is optically adjustable) can usually be made as good as the rest.
Variance, on the other hand, means some lenses aren’t as good as others, usually as a result of a number of small imperfections. A simple optical adjustment isn’t likely to make them as good as average. All lenses have a little variance. Some have more. A few have a lot. How much is too much depends on the photographer who’s shooting with them.
 |
| The Canon 70-200mm F2.8 RF has (give or take one, I’m not certain I recall all of them) 8 or 9 different adjustable elements. |
Reducing variation costs money. The reality is the manufacturers are doing what works for them (or at least they think they are). There is a place for $ 500 lenses with higher variation and good image quality, just like there’s a market for $ 2,000 lenses with better image quality and less variation.
Roger
Roger Cicala is the founder of Lensrentals.com. He started by writing about the history of photography a decade ago, but now mostly writes about the testing, construction and repair of lenses and cameras. He follows Josh Billings’ philosophy: “It’s better to know nothing than to know what ain’t so.”
Articles: Digital Photography Review (dpreview.com)
