The Leopard Report: Slower than Windows XP?
April 14th, 2008The other day, I read an article in Daniel Knight’s usually excellent LowEndMac site, in which he interviews musician and artist Scott Hansen, a devoted Mac user who is willing to overlook alleged performance limitations with Mac OS X because he still thinks it’s far, far better.
So what performance limitations are we talking about? Well, according to Knight, Hansen has benchmarked Photoshop on a fully decked out system running Windows XP and Mac OS X, and found the former is measurably faster. However, he still reports that Mac OS X runs far more reliably.
According to Knight, Hansen’s “conclusions are sure to disappoint some Mac partisans.”
I’m disappointed, all right, but not in the subject’s conclusions, such as they are. I’m more upset by the lack of journalism Knight exhibits in writing this article. Whenever you publish something about benchmarks, it’s always a good idea to be as specific about the methodology as possible, particularly when the test results seem at odds with other published reports.
Take, for example, this one, in which Photoshop CS3 is evaluated in four ways in a new MacBook. Here, Boot Camp and virtual machines are used to run Windows XP. Even in Boot Camp, where the Mac runs Windows as fast as any dedicated PC of comparable specs, Mac OS X runs Photoshop CS3 faster, and a fair amount faster as a matter of fact.
Now of course different benchmarking techniques may yield differing results, so it’s fair to say there may be some trials that give Windows XP the advantage, and I’ll grant that.
Unfortunately, Knight doesn’t tell us anything about Hansen’s tests that so impress him. And before you ask, yes, I sent two emails over the weekend asking for an explanation and further details. One even included a link to the tests that revealed a decidedly different result. There were no responses.
One extremely troubling aspect of this story is the fact that Hansen apparently ran these trials on his own custom-configured overclocked 4.2GHz Intel Core 2 Duo Quad-Core Extreme system. In other words, the test computer’s innards weren’t all built by Apple, but were heavily-altered in ways that may certainly yield unpredictable results.
In describing the process, Hansen concedes, “I have no idea what I’m talking about,” and I would agree wholeheartedly. Worse, severe overclocking can heavily compromise the longevity of the system, resulting in overheating and shortened component life. That to me is not an optimized Photoshop computer, but a time bomb waiting to explode.
Besides, Apple already has a computer that’s optimized for high-end content creation. It’s called the Mac Pro, and it doesn’t require tricked up processors or other silliness to deliver excellent performance.
Even forgetting the questionable choice of a production system, when you run benchmarks of this sort, you carefully select the test methodology to provide heavy-duty rendering functions, say, with a large photograph. That way, you wouldn’t be dealing with possible differences of tenths of a second that might challenge your stopwatch or whatever you use to time the runs.
Additionally, it’s a good idea to restart the computers after each trial, to make sure that everything is as pristine as possible. Running other applications at the same time is a no-no. What’s more, timed runs need to be repeated at least two or three times to remove the possibility of anomalous results. You then average the figures to get as accurate a measurement as possible.
Well at least he seems to have been running Photoshop CS3, to his credit, though it’s not at all clear which version of Mac OS X was used. I’ll grant, however, that Tiger and Leopard would probably yield fairly similar results regardless.
Of course, what I’m writing is just plain common sense. Even then, there are reasons why one might criticize the resulting benchmarks, perhaps suggesting that different combinations of Photoshop filters might favor one platform or the other. I accept that as well.
In fact, all things being equal, I would actually have expected Mac OS X and Windows XP to be fairly close in terms of ultimate performance potential. Windows Vista is another story, and almost every test I’ve seen shows it runs applications slower. No wonder Microsoft is having one hell of a time selling upgrades, and tens of thousands of Windows users are begging them to keep XP alive longer, and not just for a handful of cheap note-books.
However, I can see where this poorly researched article is going to appeal to Windows diehards who are absolutely begging for evidence their chosen platform is superior to the Mac. They have certainly lost the argument about Macs costing more, when identically equipped, although that fiction is still being repeated to the level of total exhaustion. And, no, I’m not going to argue the point here. I’ve done that so often that I’m tired of the discussion, at least for now.
Now perhaps Knight was looking for a fast headline to generate some hits, though I always considered him to be a pretty careful reporter regardless. I also hope that he and Hansen will explain themselves a little further so we can see just what’s going on here.
Meantime, dear reader, I welcome your benchmarks too, with as much detail as you’re able to provide.
Print This Post
“a time bomb ready to explode” isn’t that just a touch dramatic a description for an overclocked computer?
Considering the large amount of overclocking involved, I’d have serious concerns about the stability of that customized box.
Peace,
Gene
“…Besides, Apple already has a computer that’s optimized for high-end content creation. It’s called the Mac Pro, and it doesn’t require tricked up processors or other silliness to deliver excellent performance…”
Yes, but there’s a thing called price/performance, and the PC he has configured is quite inexpensive for the power it offers. The Mac Pro is overkill oftentimes (plus its optimizations are debatable, GPU and other subsystems-wise).
I think you protest too much.
My main protest is the questionable test result. I am, however, also concerned about the extent of overclocking and how it would impact the long-term reliability of that computer.
And one more thing: In what way do you feel the Mac Pro’s “optimizations are debatable”?
Peace,
Gene
Are the versions of Photoshop CS3 for Windows XP and Mac OS X the same? Has Adobe heavily optimized Photoshop CS3 for Windows XP, whereas the Mac OS X version is the PowerPC version using Rosetta?
The link to the site describing the souped-up Mac mentions Photoshop CS3, so I would hope that’s the version used. But lacking specifics, we can only hope that this common sense practice was followed.
Peace,
Gene
Well, with all benchmarks, YMMV (Your Mileage May Vary).
I can build a PC which will perform tasks that I do in Photoshop faster than an off-the-shelf Mac for an equivalent price would. And if that’s what I want to do, more power to me. To drag out the dreaded car analogy, I can modify my Toyota so that I can do 0-60 in under 6 seconds. Of course, I have to remove the seats, much of the body panels, use different tires, etc.
So I have no doubts that his overclocked PC beats a Mac running Photoshop because he has built it to do just that. Ultimately, that’s what he wanted it to do and it does so. Huzzah.
But in the Toyota analogy above, saying that “Toyotas are as fast as Porsches” would be inaccurate.
I’d also point out that, in the article, he states he is running Mac OS X on the same hardware–his Windows PC. So, again, I have no doubts that this is true but it certainly isn’t a reasonable comparison. While Mac OS X may run on his Windows PC, it may not have optimized display drivers which would slow it down.
And, from an operating system standpoint, I have no doubts that Windows XP runs faster than Mac OS X Leopard any more than I have no doubts that Mac OS 9 runs faster than Mac OS X simply because it’s not doing as much. Heck, we could get rid of these fancy graphical hoo-haws and just use the command line and things would run a whole lot faster! We humans might not be as productive, but the computer would be more productive.
So I wouldn’t get too offended over the fact that someone built a superfast PC running a stripped down operating system manages to get better benchmarks than a Mac. Bully for them. But for the rest of us in the world who want to go buy a decent machine to run Photoshop, the Mac wins.
Gene, nice article. I remember reading that same article and thinking the same thing. Every other benchmark I’ve seen puts OS X equal to or above Windows XP much less Vista. There are a few exceptions with games using DirectX, etc. of course.
I was also put off by the lack of results much less details of the alleged benchmark. I agree with your assessment – it was lousy journalism. I also took issue with the tone of the article: “Look, I’m a Mac guy and therefore you Mac folks have to accept what I’m saying as true”.
Finally, I’m not aware of how the overclocking is accomplished. Is it done through software/firmware? Is it possible that the PC side accepted the overclocking but the Mac side didn’t? If so, that would explain why the PC side was faster but much less stable. Once again, we are left with more questions than answers with that poorly written article. Seeing as though you didn’t receive a response, to your inquiries, one has to question the validity of the article in general. Really, what is there to hide?
Hi Gene,
I read Dan Knights article a couple of days ago, and this was the comment that I e-mailed to him:
Just wanted to briefly comment on your article about how “Windows XP Is Faster” (http://lowendmac.com/musings/08mm/mac-so-x-vs-windows-xp.html).
In the recent past there have been articles on the Web complaining about how Adobe has dropped the ball (purposely?) in optimizing the Windows version of Photoshop for that operating system, while the Mac version runs slower on the same processors. In earlier times, Adobe applications ran noticeably faster on the Mac OS (on slower Motorola processors) that they did on Windows PCs. Remember Apple’s snail & bunny-man ads in the nineties? It was only when Apple started becoming a competitor to Adobe by coming out with applications like Aperture and Final Cut, that Adobe began to devote more resources to optimizing Photoshop for Windows than for the Mac.
It should be made clear that this speed difference is a problem with the Mac version of Photoshop, and not a problem with the Mac OS. It has absolutely nothing to do with “OS X has more overhead – more tasks running in the background”, which is not true at all.
There are many applications that run faster on a Mac than on Windows. These include Apple’s own applications like Safari and iTunes, as well as applications from third-party developers.
Your article should have made this distinction, rather than painting Windows XP as “faster” only because Photoshop runs faster than on a Mac. In fact, Mac OS X (the operating system) has been found to run faster than Windows in various tests. This becomes very clear when you try start ups and shut downs of each OS and the same hardware, using Boot Camp.
And, even though Photoshop is an application that currently runs slower on the Mac, Scott Hansen points out another very important factor:
His second point is that Mac OS X is much more stable than Windows when using Photoshop. He complains that working with Photoshop “in Windows is wrought with errors, bugs, crashes, and lost work.” And it’s not just one particular machine – this has been his constant experience over the years with Windows XP. By comparison, he says that Photoshop on the Mac has never crashed or misbehaved.
Despite Photoshop being milliseconds slower at some functions on the Mac, it saves minutes in lost productivity compared to Photoshop running on Windows. So, the reality is that running Photoshop on the Mac is much faster (and less frustrating) than on Windows.
These points should have been made in your article, rather than leaving readers with the false impression that “Windows XP Is Faster” than Mac OS X.
Once again, the claim that Photoshop runs faster under Windows XP is not settled, and not even proven. A few of the comments posted so far imply otherwise, but that’s just not so.
Now I have heard from Dan Knight since writing the article, and he said that Hansen’s observations were subjective, based on experience and not measurements. That only increases my doubts about the accuracy of this performance claim, particularly since a hacked Mac with a processor not sold by Apple and severely overclocked was used.
Whenever Apple releases a new Mac, they provide a version of the operating system optimized for that model. So if you install a different Intel processor, even if plugin compatible, there’s no guarantee Leopard will perform at its best. Let’s not forget that.
Peace,
Gene
I have never used XP on the same Mac that I use Leopard on, but having used several computers at one time or another with XP or Leopard, my subjective reality is that Leopard is snappier. And often, with XP, a bit of something will appear, like a dialog box and then get filled in.
And there are a ton of other things that slow it down too. If you’ve just installed the OS, XP is snappy but a few months along the line and it’s slowed down. It also seems to freeze and crash for no reason. Not often, it’s true, but when it does, the results can be far worse than on a Mac, adding time to the whole thing. Installing or reinstalling the OS can also take time. How long does this take on a Mac? Half an hour? An hour?
XP is not integrated like the Mac is. It’s easy to access iPhoto from ComicLife, drag or drop from anything to anything, hover the mouse over a word to get a definition, operate from the keyboard with the same (more or less) set of keyboard shortcuts for EVERYTHING. Spotlight – brilliant. How do you find stuff on XP? Remember where you left it?
XP has to be slower.
Yours sincerely,
John Davis
Hansen’s article was written by someone who uses Photoshop regularly to manipulate large images. His findings aren’t based on artificial benchmarks but his real experience working with his own projects. He said that on the same hardware, Photoshop under Windows XP was faster than Photoshop for the Mac. As he’s running very up-to-date hardware (even if it is a “hackintosh”), I’m guessing he’s using Leopard and the latest version of Photoshop, although he doesn’t state that and didn’t answer that question when I interviewed him.
The gist of the article is that while Windows computers may get the job done a little bit faster, the Mac OS is both friendlier and more stable, allowing you to get more work done because you don’t have to deal with as many crashes. I can buy that without needing to know exactly which version of OS X and Photoshop he’s using, let alone the exact time differences between his tests. As Hansen needs to be productive in Photoshop, I have to trust that he’s making as fair an assessment as he can – it directly impacts his productivity.
Dan Knight, LowEndMac.com
Thanks for writing, Dan.
My argument, however, remains unanswered. What he perceives doesn’t reflect anything that you can quantify and measure. Maybe he runs Photoshop on Windows XP with fewer plugins and fonts, so it launches faster, but that has no impact on the time it takes to get work done. So many questions, so few answers.
To sum it up, your article makes a good argument for Macs being more productive in the real world. It does not, however, demonstrate in any way that Photoshop runs faster on Windows XP. Hansen’s subjective perceptions are far too general to evaluate in any meaningful way.
Peace,
Gene