How do you perform your tests?

I run all lenses through a set of identical tests. The so called 'lab' tests are performed by photographing a standardized test target and then using analytical software to calculate various optical properties. The setup consists of:

• Two standardized test targets (see next section)
• Giottos GB-5200 tripod
  
• Kirk BH-1 ballhead
• Smith-Victor K6RC Tungsten Lightning Kit and two Lowel Tota-Light light kits
• Analytical software - Imatest and MITRE libraries (see next section)
• Adobe Photoshop CS2

When taking test shots, I use each lens in manual and if the lens supports it in AF modes. In manual focus mode, I take series of focus-bracketed shots, which I then run through analytical software to discover series that produce best results. For both AF and MF modes, I take series of shots with different aperture settings - from the widest supported by the lens, to f/11. Furthermore, I take these series of shots at varying distances to target, typically in increments of ~1m.

How do you measure MTF?

I measure MTF using 2 software packages: Imatest and adopted MITRE mtf/sfr libraries. Imatest is my primary tool, since it's fully automated, works in a batch mode and also includes additional test modules such as chromatic aberration measurement, distortion measurement, light falloff measurement etc. I use MITRE package primarily as a backup to validate results. I use two targets when measuring MTF. My first target is the standard ISO-12233 Enhanced Digital Camera Resolution Chart (from Sine Patterns)

My second test target is a self-assembled chart that contains elements designed to test various camera properties. All slanted edge charts you see on the test target come from Imates itselft. The focusing test chart comes from Tim Jackson's Nikon D70 site. And the circular patterns chart is an emulation of The Putora Sharpness Indicator which I designed myself while waiting for the official chart from ZGC. I did get the official chart some time ago, but am currently not using it in my tests.

To measure the actual MTF, I run each lens through series of test steps outlined above to capture the test targets under different aperture and focal settings. As mentioned, all images are captured in RAW format and then are converted to maximum resolution (uncompressed) JPEG using Adobe Photoshop RAW conversion. Sharpening, color and gamma adjustments are always turned off. This typically results in dozens and sometimes hundreds of test images. For every focal length setting and every aperture setting, I then run Imatest in a batch mode to find the best (max MTF) images for each individual setting, separately for each test target. Imatest is set to analyze multiple regions of interest for every image - 4 areas for image center and 2 groups of 4 areas for borders (lower right and upper left corners). Images with the highest MTFs are then marked as test images and I run them manually through MITRE libraries to validate the overall results. Finally, I compare results from the two test targets - results are accepted if the variance between MTFs produced from the two targets fall within 5%, otherwise the entire process is repeated.

How do you report the results?

Results are reported in two sections: Field Results and Lab Results. Field Results discuss how the lens handles in the field. This is a subjective evaluation of the lens. I typically conduct a number of general tests to compile this section, starting with a round of shots of a well lit area that carries sharp color transitions. This test, primarily conducted with a wide open aperture, can quickly uncover a number of flaws with a given lens, such as color fringing, axial CA and flare. I recently started testing lenses with f/2.8 or faster apertures for bokeh. Since bokeh is a completely subjective evaluation, I base my comments only on a limited number of points. Specifically, I look at the shape and rendition of out-of-focus highlights, contrast transitions and double edging around around foreground and background objects. Finally, I evaluate any given lens for sharpness and general color representation by taking pictures of generic scenery at different aperture settings. Keep in mind here that the evaluation at this stage is still purely visual.

On the other hand, Lab tests are conducted using normalized charts and software described in sections above. I measure MTF sharpness, chromatic aberration and distortion, and include quantified results typically in the form of bar chart followed by sample crops which were used in testing. 

You have different recommendations for lenses that show similar performance. Why?

Even when two lenses have identical performance numbers (i.e. MTF50 results), it does not mean they should be treated equally. In general, I split all lenses into subcategories and then base my recommendation on how well the lens performed compared to other lenses in the same subcategory. For example, everyone should expect some difference in performance between a zoom and a prime, a wide angle prime and a telephoto prime, an ultra wide zoom and a standard zoom. I set the bar much higher for a standard angle prime lens then for a wide angle prime. Why? Because the wider the lens is, the harder it is to make it perform well - it is harder to design a well performing 14mm lens then a well performing 28mm lens then a well performing 50mm then a well performing 85mm lens. Same applies to the maximum supported aperture - the faster the lens, the harder it is to make... You get the point...

How accurate are your tests?

As accurate as one can produce without building a multi-million dollar optical testing lab. Seriously, I have no means, nor any desire to perform professional lens testing. As such, I am using equipment and testing products that are readily available off the shelf and am obviously bound by the limitations of these products as well as my personal home where I conduct these tests.

How comprehensive are your tests?

Currently I run each lens through the following tests:

• Resolution & Sharpness (MTF50)
• Chromatic Aberration
• Vignetting
• Distortion
  

I am also considering adding tests for

• Colorcheck
• Veiling glare
  

Are the results of tests comparable?

It depends. The results of tests are comparable only for lenses that were tested on the same platform. Why? Because different cameras will have different sensors (i.e. different resolution) different image processing algorithms etc.. So to be able to compare tests of lenses on different camera platforms, all results have to be normalized across cameras. I currently don't have means, nor frankly speaking desire to normalize results across camera platforms. Thus for example, you can compare results of not only EF and EF-S lenses, but also a number of Carl Zeiss Contax and Leica R lenses that I tested using the same camera system.

Are your reviews biased?

Well, depends what you mean by biased. All reviews reflect my opinions. I'm not being paid by, nor I am affiliated with any photographic equipment manufacturer, so a positive review represents my impressions of the particular photo equipment, not the requirement to write a nice thing because I'm being paid by the manufacturer. It's also true that a negative review will represent my experience/impressions, not the fact that somebody did not pay me enough money to write a nice review.

I disagree with your review of [name your own] lens...

Great. The purpose of this site is not to enforce an opinion but rather bring some additional (and hopefully useful) information to you, which can help you in making a good decision. You should always keep in mind that as with any other product on the market, lenses even from the same manufacturer will always have some variance in quality and performance. It might as well be that the lens I picked for testing is an outlier and does not accurately represent particular sample group. Ultimately however, picking a lens is a matter of your preferences, style, taste etc. lens sharpness, vignetting, distortion, aberration are not the only characteristics that need to be considered when picking a lens - so read, test and choose wisely!

Why do you perform reviews of alternative gear only on Canon cameras?

The main reason for that is when I started these tests, Canon offered the most flexible and adaptable platform, with the smallest distance to the focusing flange. Plus Canon (up until recently at least) was the only company that offered a plethora of reasonably priced full frame digital SLRs. Things obviously changed since then - Nikon added a full frame sensor to its lineup, as did Sony. Olympus came out with a Micro 4/3 format, which allows adapting even rangefinder lenses. Still, Canon remains the most popular platform out there and this means that I will keep using Canon as the primary alternative platform. As a rule of thumb, I will keep 

• Use Canon platform to test native EF/EF-S lenses, as well as all alternative, non-zoom lenses that can be adapted to Canon platform
• Use native mount cameras to test lenses designed for that camera mount (i.e. Nikon lenses will be tested on Nikon cameras, and fixed focal Nikons will also be tested on Canon cameras)
• Use Micro 4/3 mount camera to test alternative gear that cannot be adapted to Canon platform (i.e. Voigtlander/Leica/Contax rangefinder lenses will be tested on a Micro 4/3 camera)

All kind. I used pretty much every 35mm film and digital SLR and rangefinder out there. Canon, Nikon, Olympus, Leica, Kiev, FED, Zorky, Contax, Minolta... The list goes on. And while I've tried many different 35mm cams, I have yet to try a medium or large format.