Why I never recommend the best products
Why I never recommend the best products
Why our reviews won’t do your thinking for you
The collection of detailed reviews and articles on the Northlight Images site has been built up over several years and covers a range of equipment and software.
One thing they don’t ever contain is any ‘best buy’ recommendations or test scores.
Keith believes such ratings and awards rarely answer any meaningful questions, and he’s put together this short article which explores how we address the issues and why.
As you’d expect, the longer answer is not always that simple…
Many of the camera equipment testing web sites and almost all magazines find it obligatory to end a review with a set of product ratings and test scores.
You know the style – Camera A scored 73% whilst Camera B scored 76%
The better review sites have information about how these scores were arrived at, but it is rarely in any great detail and for anyone with the slightest scientific training, usually has methodological holes you could drive a truck through.
The photo shows my 2003 Canon 1Ds and 2007 Canon 1Ds mk3 – surely an easy comparison?
Even when broken down into smaller components, the apparent precision is largely meaningless.
Take these categories (from a camera review site):
- Build quality
- Ergonomics & handling
- Metering & focus accuracy
- Image quality (raw)
- Image quality (JPEG)
- Low light / high ISO performance
- Viewfinder / screen rating
- Movie / video mode
Look carefully at the list and just for a moment, consider how you would measure the features for your favourite camera.
Please take a moment and go through all of them and give it some thought.
Now, think of a similar camera from a different manufacturer and think how you’d go about a similar process.
Some are measurable with a suitably sophisticated setup, but remember that more precision in measurement does not make the measurements any more relevant or meaningful in themselves.
I could make a similar list for printers or any other equipment/software I’ve reviewed, but the list would likely be different in varying degrees for every single item.
I’m sure I could make the lists and charts look very nice, with animated graphics comparing the numbers and allowing you to customise the comparisons.
Yes, this makes it easier for some visitors to decide what matters, but it’s all too easy to be swayed by the spurious precision of the charts and graphs.
Simple lists of feature can be of help (I always try to include detailed specs at the foot of reviews) but I’m always wary of this ever since a manufacturer confided in me that they added ‘features’ to lists, just to make it easier for lazy reviewers, who might not want to read the entire press release.
If comparing different makes is tricky, I’d just note that looking for meaningful changes in product updates can be equally difficult.
We’re often asked about which product is ‘best’ for someone looking for a new camera or printer and have to say why there is often no simple answer.
People want simple answers, and I could easily tell someone who emails me, that printer XXX is the one that will produce the best prints.
There, I’ve helped them – problem is that I do actually care about if I’ve genuinely helped them.
If it’s a product I’ve reviewed in the last year or so, then I can remember enough to ask a few questions about what they want to do with it. Even if it’s one I looked at a while ago I usually know what the current range of products look like and whether they have changed much.
The thing is, I need to know why they are asking – never are spurious percentage scores of less use to me. Quite often, I can tell from the tone (written or spoken) that they have already decided and just want someone to confirm their decisions.
I don’t mind this, since my reviews are usually written to show features and how I used them. I’m writing (hopefully) from the position of someone comfortable with using the item. I just hope that the reviews display a level of expertise, and that it will be felt as relevant.
Our reviews are thorough – these are some of the test prints from just one printer review.
If I go into some stores, I know that asking an assistant will get advice about products. I also know that stores promote some products over others for purely commercial reasons. The sales assistant (clue is in the name) is an employee of the store not your friend.
Photography magazines are critically dependent on advertising revenue, and in the past there have been some pretty close relationships between the advertising departments and manufacturers, with advertising departments having effects on editorial and features.
I’m sure I’m not the only one to notice that the ‘product awards’, so beloved by manufacturers’ press departments, seem to be given out almost on a rotation basis, so that there are enough categories that everyone is a winner every year.
Look around this page – yes, we have advertisers. I use the ads (placed dynamically using Google AdSense) to help fund Karen’s work with Northlight and pay to run the site. The ads are not picked by us and will vary, depending on where you are reading this from.
We also have direct sales affiliate links, such as the box at the right – mainly since if you decide to buy any product, then going via one of our links gets a small commission from the sale (it costs you no extra), and I’d really prefer it went to us… [Thanks to those who do!]
I was recently asked if I thought this affected our impartiality?
Hopefully not, since I like to make the links clear, and I try to make sure that I have products reviewed from a number of competing manufacturers.
I’m always looking to expand what we have, as long as it’s in an area I have the relevant expertise (one reason we have no video related reviews).
If you are curious, we do have a more formal Article and Reviews policy for the site.
OK, we do have some…
First up, we’re Mac based, so Windows only products are not overly likely to get looked at. I know there are virtual PC solutions, but I’d still have to get a copy of Windows and the software to run it, possible, but in the practical world unlikely.
Canon cameras – this one is purely by dint of my use of Canon cameras for my commercial work since 2003 (Canon 1Ds), and that for quite some time, has been because of the lenses (although the 50MP 5Ds helped).
Same goes for lens info. and reviews – they need to be Canon fit (even if I make the lens myself).
I’m happy to look at other makes, such as my recent test of the pre-release Laowa 12mm ultra wide angle lens.
The photo to the right shows the kind of ‘real-world’ examples I prefer to an over reliance on test charts.
Why no other camera systems?
Well, I’ve asked many times, but not being that wealthy I’m reliant on loans from manufacturers, and apart from a direct contact from Nikon UK in 2007 (just before the crash) I get silence.
There are many cameras and lenses I’d like to experiment with – if you’re reading this and work in a camera company…
One aspect of our independence that I value is that we do pre-release product testing for a number of competing manufacturers under NDA.
Several have said that elements of our brand neutrality make our reviews all the more relevant. Their trust in our abilities and values is appreciated.
I’ve liked using all the new printers from Epson and Canon over the last year or so. Don’t ask which is best – it depends ;-)
If HP would update the z3200ps that I looked at in 2009, I’d add HP to the mix as well.
I sometimes look at user reviews of products I’m looking to buy on sites like Amazon, but not so much for anything important – more to see if they raise an issue or problem I hadn’t thought of.
I also want to know more about the credentials of the supposed comment makers, since I’ve also seen too many examples of on-line reviews being padded and promoted. Despite attempts to curb it, it is endemic. If fraud is easy to do and carries little risk, then it will occur.
In a way, our reviews are extended ‘user reviews’, but I hope that the depth and history of the site gives a bit more credence to my conclusions?
No, I was serious… I’m not going to pick any.
Email me with specific questions about a review and I’m happy to expand on what I’ve written, or use the comments section of the review and I’ll happily clarify stuff.
Also, if you think there is something I should look at, let me know? I can’t guarantee looking at things (the photo business is my day job) but all suggestions are welcome (I’m still not getting a Win PC though)
Yes, the ads below are the sorts of ads and links that clutter up some of our pages. I’d rather have them than go for any type of subscription model for the site. I’ve a strong belief that our content should remain 100% free to access.
Enjoyed this article?
All the latest articles and photo news items appear on Keith's Photo blog
We've a whole section of the site devoted to Digital Black and White photography and printing. It covers all of Keith's specialist articles and reviews.
Articles below by Keith (Google's picks for matching this page)
If you start your buying of anything whatsoever from Amazon (not just what's listed) via one of our links below, it helps myself and Karen to keep the site going. We really do appreciate this - thanks