[Kozuch]

Hi there,



I look at your LW/PH (MTF50) lens resolution tests. I am disappointed they are made against a concrete image sensor (or camera), because like this you limit the test results (by the resolution of a camera sensor). The tests as you do them are useful for a concrete camera models, but because camera sensor technology evolves much faster than lens technology (at least given the resolution of sensors so far), this is pitty, because the tests say nothing about performance with better sensors that tested (say 15 Mpix vs 18 Mpix). Only true optical tests can show real resolution limits of a lens - for example projecting against a distant wall and then manually counting the lines...



What more, as I got to know [url="http://www.cambridgeincolour.com/tutorials/diffraction-photography-2.htm"]here[/url], due to the Bayer array the resolution of each imaging sensor might nicely be 1.3-1.5 times less than the actual horizontal/vertical pixel count - I guess this fact also lowers the measured resolution of a lens when measured against a sensor... When doing true optical tests, your tests (expecially for third party lenses) would be much more useful, because they could be used accross camera models and sensor resolutions.

[Klaus]

Ok, we'll do that. I will send you the details of my bank account. A purely optical MTF testing lab costs just a quarter to half a million EUR. :-) In the real life we've to observe our costs, of course.



If we followed your suggestion the reverse question would occur - whether our pure lab test results would actual apply to the results obtained from an image sensor.



As you may have noticed we are updating our test cameras. We've upgraded from 8mp to 15mp for Canon APS-C tests and we'll do the same for Pentax and Nikon very soon.



The test procedure is not custom-made but an out-of-the-box solution which is also used by http://www.imatest.com/customers (you may spot Canon in there for instance).

If you check the documentation you may also spot a section where the manufacturer explains how they can look beyond Nyquist (the characteristic of test target is known).

[Kozuch]

Well, of course a profi lab would be very expensive... but as a Czech I try to think about an affordable workaround of this issue... <img src='http://forum.photozone.de/public/style_emoticons/<#EMO_DIR#>/biggrin.gif' class='bbc_emoticon' alt='' /> I can imagine it can not be that difficult to create a simple true optical resolution test at home?



Probably, the only thing that would need some more effort is how to evaluate the results, maybe one could capture the projection of reference image with another digital camera (glue more photos together so we are sure this action does not limit the resolution test) and then have a software count these lines, as I can imagine no one wants to count 2000 lines by hand...



I am sure this set-up is doable and could produce nearly proffesional results. It is only a question of will. I really appreciate your current tests, but this is just an idea on how to make it better.



Regarding the usability of lab results for concrete sensor - I did not really understand what you meant by this. Do you mean maybe the ability of a sensor to reach a certain resolution?

[Klaus]

[quote name='Kozuch' timestamp='1280403014' post='1380']

Well, of course a profi lab would be very expensive... but as a Czech I try to think about an affordable workaround of this issue... <img src='http://forum.photozone.de/public/style_emoticons/<#EMO_DIR#>/biggrin.gif' class='bbc_emoticon' alt='' /> I can imagine it can not be that difficult to create a simple true optical resolution test at home?



Probably, the only thing that would need some more effort is how to evaluate the results, maybe one could capture the projection of reference image with another digital camera (glue more photos together so we are sure this action does not limit the resolution test) and then have a software count these lines, as I can imagine no one wants to count 2000 lines by hand...



I am sure this set-up is doable and could produce nearly proffesional results. It is only a question of will. I really appreciate your current tests, but this is just an idea on how to make it better.



Regarding the usability of lab results for concrete sensor - I did not really understand what you meant by this. Do you mean maybe the ability of a sensor to reach a certain resolution?

[/quote]



Oh, it's certainly possible to assemble an optical MTF lab "by hand". However, reliability and speed of analysis have to be taken into account.

If we ever consider moving this way we'd certainly go for a Zeiss lab and ... this simply costs a fortune. Regarding the current site revenues this is simply prohibitive.



Regarding the sensor as base for testing - have you noticed in our tests that very few lenses are actually capable of outresolving today's sensors at image portions other than the center ? Moving to an optical MTF test seems like looking for a problem that does not exist.

Beyond 10mp (APS-C) and 20mp (full format) an camera upgrade has more to do with "political correctness" rather than sense.



There're a few dependencies of an image sensor. Vignetting comes into mind first of course. Purple fringing is another one. AA filter side effects.



And if you think that an optical MTF test is the way to go ... well, go for it .. :-)

[Brightcolours]

[quote name='Kozuch' timestamp='1280403014' post='1380']

Well, of course a profi lab would be very expensive... but as a Czech I try to think about an affordable workaround of this issue... <img src='http://forum.photozone.de/public/style_emoticons/<#EMO_DIR#>/biggrin.gif' class='bbc_emoticon' alt='' /> I can imagine it can not be that difficult to create a simple true optical resolution test at home?



Probably, the only thing that would need some more effort is how to evaluate the results, maybe one could capture the projection of reference image with another digital camera (glue more photos together so we are sure this action does not limit the resolution test) and then have a software count these lines, as I can imagine no one wants to count 2000 lines by hand...



I am sure this set-up is doable and could produce nearly proffesional results. It is only a question of will. I really appreciate your current tests, but this is just an idea on how to make it better.



Regarding the usability of lab results for concrete sensor - I did not really understand what you meant by this. Do you mean maybe the ability of a sensor to reach a certain resolution?

[/quote]

It does NOT matter with which sensor Klaus/photozone.de tests the lenses. If lens A shows larger vignetting, more CA and less resolution than lens B on a 10mp Canon EOS 40D, the same will apply when you put lens A and B on a 18mp Canon EOS 7D.

Of course, the resolution figures will both be higher, but resolution figures on their own are rather useless, you compare them with other lenses.



You can verify this with the Canon ESO 350D (8mp) tests and the later Canon EOS 50D (15mp) tests.



So, no need for photozone to not use normal DSLRs to test how lenses compare... Less sharp lenses will be less sharp, no matter which DSLR model you use.



If you have trouble figuring out if a Canon 70-200 f4 L IS USM is sharp on your 550D (18mp) while Klaus showed it to be exceptionally sharp on a 350D (8mp) and 50D (15mp), then how will results from a fictive sensor help you determine that?



I basically think you are trying to find a solution to a problem that does not exist..

[mst]

[quote name='Kozuch' timestamp='1280403014' post='1380']

Regarding the usability of lab results for concrete sensor - I did not really understand what you meant by this. Do you mean maybe the ability of a sensor to reach a certain resolution?

[/quote]



No. If we published pure resoultion data of a lens measured with an optical lab, the question really is: how well does this lens behave on a given sensor? Will a lens with "average" resolution still be usable on a current DSLR with high MP count?



As brightcolours correctly put it: the essence of our reviews is to have comparable data, not absolute data. And unless someone of the PZ team wins the lottery we need to have a measurement workflow that is affordable, reproducable as well as fast. A lab just doesn't work for us, especially not a home made one. We would not only have to count 2000 lines once, but once for every measurement spot (center, border, corner), every aperture and focal length measured and any of the x shots taken for a given aperture and focal lenght (x usually in the range of 20 to 40, since we use careful manual focus bracketing to find the maximum resolution). No way <img src='http://forum.photozone.de/public/style_emoticons/<#EMO_DIR#>/wink.gif' class='bbc_emoticon' alt='' />



-- Markus

[Guest]

Thank godness Klaus you are testing the way you are testing. When I look at the lens tests of some German photo magazines like colorfoto or fotomagazine with all their MTF machines and all those stupid curves and data, I am glad that your tests include SAMPLE PICTURES.



Yes, sample pictures are the best way to judge a lens. If I were to recommend a new testing procedure I would suggest a more coherent approach to sample pictures. It would go a long way if you just shot every lens on the same wall (or some other flat, real life subeject)with wide opne aperture and two stops down (for a three focal length in case of zooms). To that I'd add one portrait (equal composition is mandatory here for every lens) and one or two landscape shots at differnt apertures ( again same subject for every lens). One more pic would be required if a lens showed a curved focus field, to show that by correctly setting the focus one can get the corners into focus. With these sample pictures everyody could judge for himself if the lens is good enough for one's needs. Now dont say that there are other websites which provide samples. No there arent, at least none which are reliable. A lot of people on the inertnet post out of focus shots or hand held high iso shots for judgeing lens quality which is really useless. (hardly any picture on pixelpeeper.com is free from handshake for example) What I sometimes miss in your sample shots are wide open shots, which help a lot in judging image quality.

Thanks and keep up the good work

[Klaus]

It's tricky to create the same sample image testing condition throughout the seasons and in-room sample images are just soso from a real world perspective.

I think it can only be an add-on.

[Guest]

Well I recognize that views differ here. If you could just take one or two wide open pics of each lens (espacially with the border region in focus)your tests would be even more valubale to me. But thanks anyway, I know that testing lenses and writing everything up is a tremendous amount of work.

[wim]

First of all, I do think the test here on Photozone are great, in the first place because they are unbiased, and often have enough sample pictures to judge lens quality in real life.



As tor an optical bench for testing: it tells you nothing. Any fast (large aperture) quality prime lens will be limited in resolution by diffraction by F/4. Any slightly cheaper or lesser quality lens, or zoom lens for that matter, will reach that point at F/5.6 or F/8, some a little later, and some of the really lesser lenses will only reach that point where diffraction limits the sensor resolution, e.g. F/13 on APS-C and F/18 on FF, or mayeb even not at all.



In the past, in analog days, lenses were often tested with a specific film, which makes life a lot easier, because provided you always use excatly the same techniques and settings for shooting and developing, the only unknown factor is lens resolution and lens aberrations at a specific aperture. However, films used always were high contrast films with very little tonal gradation and developed to get the highest acuity and largest contrast difference possible, hence not a scenario used for normal photography at all.



Each new generation and different size sensor and signal processing assembly behaves differently, due to the AA-filter assembly applied, due to pixel spacing, lenslet placement, distance between well sites, signal processing algorithms, etc. So from that POV it does make sense to use an optical bench. However, because of the different sensors, it is of little use, because in the end we want to know how a lens performs in combination with its recording medium, film in the old days, and sensors in the modern days, as that is the only way to obtain an end result.



In short, if you'd want to do a theoretical analysis of the performance of a lens, an optical bench is fine, provided BTW that there is nothing that intervenes with the actual analysis, i.e., automated analysis by the software analyzing the air image rendered, not an image projected on some wall, or a human counting lines). This will allow to compare lenses of different brands against each other (an i am sure th eoptical companies do just that).



However, for real life photography it makes no sense at all, because this is all theoretical stuff, and there is much more to rendering an image that, e.g., just resolution or optical aberrations. It really is a package deal, which one person will like, and the other won't.



You'll find that there actually are some photographers who specifically use a certain lens to get a certain type of rendering, for a specific occasion, and for another shoot, they'll use another brand lens with the same FL and aperture because they need a different type of rendering. Certainly with large format film this was very common. Nowadays it is less so, but there are still photographers who do, and who have a selection of lenses to choose from just for a single FL, for specific rendering purposes.



Just my 2c.



Kind regards, Wim