Recently, on the excellent onlinephotographer.typepad.com blog, Mike Johnson summed up something I have been skirting around for awhile now. Simply put, when it comes to lens design needs and expectations, we are there. He (,I and others) actually lament the lost days of actual rendering characteristic variables and a more relevant pecking order. All of these lenses were graded by eye. Funny thing that, using your eyes to look at actual images, who knew. Even testing lenses came down to printing visible "line pairs" of information on paper and studying them.
The lens makers lot is a tough one. On one hand they will have a pre determined path laid down for them. Criteria that will set minimum and maximum requirements and limitations such as the widest f stop, size/weight/price limits, image quality expectations and sometimes deeper requirements such as special attention paid to it's intended role (close focus, smooth bokeh being common ones at the moment). Once the parameters have been set, the designer must push and pull the laws of physics where possible to get the desired lens. Many of these physical limits are set in stone, but modern science is always offering up solutions to some problems such as better glass or more exotic replacements.
Lens carefully designed, lens ever more carefully made and then released into the wild.
Then it starts. Some lenses cannot even get to their release date before the critics effectively kill them off. Funny thing is though people still buy them, even those who read these negative reviews and they find that the lens is actually capable of great, even spectacular results (Fuji 18mm f2, Olympus 17mm f1.8, Canon 28mm f1.8, Fuji 23 f2, Fuji 16-50 kit......). This can really be a case of ignorance is bliss.
If your maximum quality requirement is a large, fine art grade print, then a well used kit lens on a medium to low resolution camera can provide, but that will seldom satisfy the pixel peeping and hair splitting crowd in the reviewasphere. We are effectively talking about two ways of measuring lenses. The mathematical way and the practical/realistic way and the math is winning at the moment.
Looking at the futility of the testing exercise; If the difference between two lenses is measured on a micro scale and the tester has picked settings to "even the field", usually medium settings (not the best for the specific sensor/lens combination) and has ignored the huge variety of post processing options available, then the test is flawed. A lens resolution test (if it is that important to you) should be measured in two criteria. (1) What is the best it can possibly produce in real world situations and (2) how consistently does it do it, I.E. when "stressed", does the lens still perform.
A real bug bear of mine is the often unintentional misinterpretation of the provided data. Some tests will show you the top part of a graph, demonstrating what looks to be a large difference between the good and the not so good. If I show you only the top 20% of a graph (numbers such as 1750 vs 1890 measured from 0-2000), the differences between two candidates may look pretty horrifying, but if you see the whole graph from top to bottom, the differences are put into much better perspective.
Some of the test bench reviewers will, to their credit, point out that the measured differences will seldom be visible to the eye and are never relevant without direct comparison. They may also point out that these conditions are "ideal".
Another inconsistency is in the differences inherent in the camera's sensors, both in how they measure and in their native resolution. When I was working in a camera shop, a customer felt he needed some "visual" proof in the form of graphs etc when deciding between two lenses, a known, premium wide angle and it's cheaper and more practical sibling. Against my better judgement, I showed him a site that had pretty clear graphs showing the centre and edge performance of two wide angle lenses. The differences looked to be huge (top 10% of the graph syndrome), but then, as the customer dove head first into the world of lens bisection, he was dismayed to find a cheap camera from another brand with a kit lens showing higher figures than his expensive full frame SLR and even more expensive wide angle lens on the same site! Be careful what you look for.
In the end we (....wait for it), shot some test images of the building across the road from the shop (a bank!) with both lenses and looked at them on screen. The test data was pretty much on the money, but meant nothing compared to actually seeing the images. The customer decided to take the "lesser" lens as the difference did not seem that great in real life, especially when he was shown that even mild post processing closed the difference significantly.
The third short coming of some testing sites is that they seem to be unable to find a lens that fully satisfies. Obsessing over colour fringing, edge softness, subjectively poor Bokeh etc., they cannot settle the readers nerves with their base line negativity and often completely overlook a whole swathe of other characteristics the lens may offer.
The final issue that I find is the inconsistent nature of different testing procedures and results. Sometimes, but not always you will find contradicting results on different sites. I am guilty of looking most at the sites that tell me what I want to hear and avoiding the ones I don't!
Image Resource has an excellent "handkerchief" graph that looks to defy sensor resolution as it shows real results of blurring, not mathematical measures of resolution and has often come very close to what my own eyes see with the same lens, except with long telephoto lenses that often look better to the eye than the testing would suggest.
Frustrating huh?
A test bench blitz and quick walk around the block taking "real life", but ordinary images, matched with (far too) close scrutiny and test bench comparisons, is not fair on any lens, but what is the point of reviews if they do not find (ever decreasing) differences. The only fair measure is to own and use a lens over a period of time and use it to take the special images you are looking for and a few, more realistic and practical reviewers have switched to the extended field test with no charts and tables model which is closer to the true way of testing.
If it becomes less about the lens and more about the pictures it takes, then there is no issue.
Why do they even try?
Lens tests are not useless, but they must be taken in context. The numbers, carefully measured are an indication of some characteristics the lens shows, but this is similar to asking the salesman what the top speed of a car is, with no interest shown in other features.
Recently I wanted to show how good/bad the Olympus kit 14-42 was with some comparison images (vs 12-40 and 17mm prime). I had to stress the lens so much to find a noticeable difference at normal viewing/printing sizes, that it became pointless and impractical.
A couple of things have become evident lately.
The first is that the average lens is so good, that better lenses are having to grow ever bigger and more expensive in order to matter. As an example of this look at the recent "improved" offerings from Canon (35 f1.4, all their zooms with f2.8 apertures), They are all sporting an increase in bulk, price and filter size.
The second is, how many pro photographers are using "lesser" lenses happily.
If I told you this image was taken with a cheap kit lens or a premium fast prime, would it change your reaction to it?
There are a LOT of things that make up a lenses personality. Some can be measured, but many not. Most can be fixed if flawed (CA, sharpness) and some have no right or wrong (bokeh, vignetting). It is only by switching off the annoying little voice in our heads and actually using our gear that we will find out if it works for us or not.
Every lens will have some strengths and weaknesses. It is actually more fulfilling to find the hidden excellence in an average lens than to find the flaw in an expensive super lens.
A lens should only be discarded if it;
It is too difficult to use, making the process harder, not easier (Panasonic 20mm AF/MF pain, Fuji 60MM macro for focus speed issues).
It is actually poor quality when viewed at a normal viewing distance and on the medium it is intended, which is rare these days, but not impossible, especially if you are reacting to a character in the rendering and not an actual design fault.
It is just the wrong lens for the purpose intended (why did you buy it?!). Lens speed, focal length and close focus are often sighted by dissatisfied users, but none are a surprise.
As an aside, lenses take better photos when you like them, true story.