Should we hate the player? Or hate the game? Both, probably.
With today’s shocking revelation that it’s possible to cheat Android benchmark apps, it’s worth a quick reminder that we’ve always been able to manipulate Android benchmarks. And, frankly, it shouldn’t come as any surprise that manufacturers do everything they can to score as high as possible.
I quite fondly recall sitting in a Hell’s Kitchen bar in 2011 2010, just hours after the launch event for the original Samsung Galaxy S. I had a brand new Motorola Droid X in my pocket — announced just a week prior — and (now former) Samsung engineers asking to see how it benchmarked. Not knowing better at the time, I happily obliged.
Here’s the thing: We have only ourselves to blame for this nonsense.
Android sites — ourselves included for some time — have long made a big deal about benchmarks. And here’s why: They’re easy to do. Fire up an app and a video camera, and watch the fun begin. “It benchmarks 9,000!!!” is a joke we still make today, long after Android Central stopped worrying about benchmarks.
Anyhoo. Or own Jerry Hildenbrand back in May 2011 walked us through just how easy it is to manipulate benchmark apps through software — never mind what Samsung’s doing now, optimizing for specific apps. Can you really blame Samsung, though? Fans and bloggers make a big deal about benchmarks. Why wouldn’t the manufacturers do everything they can to score as high as possible?
Where it gets shady, of course, is that Samsung is controlling the CPU for specific apps, and not exactly in a transparent manner.
“They should have a checkbox in the (operating system level) settings that says ‘Run in full throttle,’ ” said Marcus Adoflsson, founder and CEO of Mobile Nations.
I’m inclined to agree. But these aren’t Samsung’s apps. It’d behoove the app developers to perhaps point out just how the hardware is interacting with the software. Using more cores than with other apps? Tell me.
But then again, if someone’s going to game the system we helped create, we shouldn’t act so surprised.