RSSHow do you define performance?
By Carmi Levy | Published October 15, 2009, 4:26 PM
System performance is an interesting concept; everyone seems to define it differently. To some, it involves chewing through a complex spreadsheet. To others, it's how fast a 3D video sequence can be rendered, or how easily Web pages are served up.
Call me a rebel, but after years of living off of a BlackBerry, my thinking has evolved. As much as I focused on megahertz and gigahertz for much of my computing life, the most important criteria for me these days are how fast the thing turns on, and how long it stays on before I have to recharge it.
It goes against everything we grew up with, of course, as the PC industry has always been obsessed with the electronic era equivalent of brute force horsepower, rather than eco-friendly efficiency metrics. But when I'm sitting in a deserted airport terminal with an hour's worth of work to power through and only 45 minutes of remaining battery life, the last thing I'm thinking about is processor performance.
Machines (and uses) have changed
However we choose to define performance, we've always assumed that performance measurement benchmarks were run on machines that had already been powered on and booted up. We also assumed that these machines were plugged into a power outlet. These days, these assumptions are laughable, because outside of cubicle-dwellers sweating over dusty desktops for their corporate overlords, few of us leave our computers on 24/7/365. We take them with us to places like Starbucks, where finding a power outlet is about as challenging as locating the lost city of Atlantis, and we don't have the luxury of time to sit and wait while our machines boot up.
This shift in performance metric priority is being driven by two major changes in attitude and technology:
- Green matters. Conserving power didn't matter much when 30-pound desktops ruled the planet and no one paid attention to how much all that electricity cost. Now that we've mobilized, disconnected, and learned the value of power-performance curves, the cost and availability of power are suddenly much more important to us. It isn't feasible to leave a laptop or other mobile device perpetually powered on. Even if we didn't run the risk of frying our laptops inside their padded cases, their three-hour battery life would leave us high and dry before lunchtime.
- Software deployment has evolved. Corporate software distribution methods that required conventional desktop machines to be powered on at night, are yesterday's news. Today, commodity chipsets, firmware, and monitoring software allow machines to be remotely -- and automatically -- woken up, updated, and shut down. Web-based remote management solutions keep mobile machines in top form as well. In many cases, cloud-based software infrastructure is largely obviating the need to update client code in the first place. One reason that everyday users neither know nor care when Google Docs adds a new feature is because the software no longer makes a big deal of it. Windows tends to announce when it's been patched or updated, but Web apps typically don't.
Beyond eco-friendliness, better, smarter client hardware, network control, and an increasingly thin software stack, lies another simple truth: Today's mobile machines lead very different lives than more sedentary desktop hardware we all grew up with. We shut them down and hibernate them before shlepping them from one place to another, and when we get to our destination and open the lid, we expect things to happen quickly. The more dynamic nature of the modern workplace, coupled with the transition of mobile handsets from basic voice devices to full-featured computing devices, has radically altered our expectations of how these devices -- mobile or not, pocketable or not -- should work.
Thankfully, the industry is tentatively taking notice. The turtle-slow speed with which the average computer wakes itself up, freshens its desktop, and presents its happy face to its owner, is finally getting some attention from hardware vendors. HP's QuickWeb and Dell's Latitude ON, for example, boot into reduced-scale environments for quick access to simple apps like e-mail and contact management. Beyond launch performance, once these machines are on, the amount of time they actually remain on is also finally becoming a key concern for hardware makers.
I'd like to ask them what took so long, but that would be cliché. What matters is that this real-world performance metric is no longer being ignored. And as operating systems initially developed for handsets such as Google Android start showing up on netbooks and other lower-end machines, end users will increasingly expect these larger devices to offer the same kind of instant-on, battery-friendly experience provided by more traditional handhelds.
Not so fast
As heartening as it is to see vendors finally wake up to power-on and battery life performance, we're not quite up to speed yet in terms of standardizing and normalizing the messages they send to the broad base of consumer and enterprise buyers.
For example, Nokia claims its new Booklet 3G will run for 12 hours before it needs to be recharged. In the absence of standardized power-based performance metrics, however, it's difficult to take Nokia at its word. Is it 12 hours of watching a movie? Is wireless on or off? If so, is it Wi-Fi, 3G or both? Since vendors have no established baseline for these performance claims, they're free to say pretty much whatever they want.
As a result, they'll continue to use rather useless metrics (like how many cells their batteries have) to flog their machines. But computer vendors making a big deal out of six- or nine-cell batteries aren't much removed from automakers hawking the size of a given vehicle's gas tank. In both cases, we still don't know how far the thing will go. In the real world, we continue to lack industry-accepted parameters akin to the EPA's fuel economy standards that allow computer buyers to make apples-to-apples comparisons.
Change often comes to an industry when consumers demand it. And as we spend ever greater stretches of time using battery-powered devices far from home, the onus is on us to challenge vendors to get straight with realistic methods of measuring such real-world performance parameters like battery life and power-on availability. I'm sure I'm not the only one who looks forward to evaluating comparable machines based on performance figures that weren't first invented by the marketing department.
Carmi Levy is a Canadian-based independent technology analyst and journalist still trying to live down his past life leading help desks and managing projects for large financial services organizations. He comments extensively in a wide range of media, and works closely with clients to help them leverage technology and social media tools and processes to drive their business.
Performance to me means two things.
On the desktop side i have my gaming rig. loaded to the gills with parts that makes everything fly. software matters cause a good many antivirus apps slow any pc down way to much. so i select what runs on my pc carefully. I need my gaming pc for the best possible gaming experience, so everything needs to be powerfull or lite on resources.
For my laptop i need decent speed with decent battery life. I create power profiles that allow the cpu and other things to crank down when not needed. and crank down alot. Same with the desktop, software that doesnt chew thru the cpu is a must. To give you a idea my brand new laptop when i first turned it on was slow with all the preinstalled crap which no one needs. so i stripped that all away and put on what i wanted and created my power profile. My power profile gets me arround 60 percent more battery life then i would get otherwise.
And i do my downloading(torrents), on my laptop most times. Cause i dont want or need my desktop running 24/7.
Score: 0
|Hardware is the solution to faster bootups, not software. Put your money into hardware. In particular, SSD's really make a difference, and are well worth the current price premium.
Score: -2
|The solution....
Gotta love it.
Score: 0
|You still need software to do the boot and it would be better to have efficient code than sloppy code.
Score: 0
|Well, we're still waiting on the audio industry to give us real and consistent measurements that mean something in the real world, so I won't hold my breath on the computer industry.
In my earliest days as a computer science student through each project at work, the performance measurement was throughput. It didn't matter how quickly the processor ran, if the programme was input-output bound, such as waiting on disk or tape or human input.
If an operating system/GUI gets in your way on the desktop, it constrains your ability to be productive. If the system is constantly asking you questions, you may not finish anything, but you'll certainly be annoyed and want to switch to a system that does. Too many electronic devices (and software, in general) work in a way that was the easiest to programme and don't work in an intuitive way for the user, lowering his/her productivity. The device/software may perform well at 600 MHz right up until the time it hits the wall and becomes inoperative.
Score: 0
|Excellent. I whole heartedly agree with this. I gave the desktop to the kids, downsized to a netbook, added an SSD and moved to Linux. Drastic? Nope - boot to desktop times of the order of 15 seconds, portability and longer time on the move without a powerblock. Great. Key thing is MY productivity - and this was a big improvement for me.
Score: 0
|Carmi,
I had serious doubts as to whether or not I'd ever be able to say this again, but:
Excellent article!
I honestly wish I could remember the name of the article you did early on that I liked so much (thus likely giving me unrealistic expectations), but that's not a problem. This one exceeds those expectations.
I've never quite looked at it performance/power management this way. I suppose it is probably somewhat analogous to price/performance...now we just get to add a third factor (lucky us).
Score: 0
|I'd say I personally define performance as how fast programs execute tasks and how well the machine multitasks.
I care more for the machine that boots in 40 seconds and runs 50 programs and wickedly demanding games without breaking a sweat than a machine that boots in 20 seconds but chokes on a few programs or games. This translates to my phone as well. First of all, my phone is rarely ever in an off-state...but failing that, I'd rather my phone take 30 seconds to power up but then be able to run several apps, or run smoothly while it IS on (ie: not laggy, no constant freezes, etc) than to power up in 5 and then just be terrible to use. Granted, I don't want a phone that drains to nothingness in 30 2 minute calls on bluetooth but I don't consider battery power a huge part of performance in a phone either. Because I have no problem charging it while I drive or in my home. I don't want one that dies in a an hour or 2 standby, of course....BUT I also don't need one with 10 hours of talk time, especially if it's crappy to use otherwise.
I'm not sure when people started getting caught up on battery life. I think it's because of cell phones becoming ubiquitous. Granted, I think battery life is of some importance, but I wouldn't use it to determine "performance". I guess coming from a place like Miami, FL where I was never much into taking the bus or public transport...this has never phased me. I always, always have an outlet nearby whether it was in college or if I needed to use it on the road for extended periods. Worse come to worse I plug it into my car. I suppose the time will come when I need 8 or 9 hour battery life...a flight to some far away place perhaps? Since I don't do this daily though, I'm not affected.
Score: 0
|Yap. Battery life isn't that important to me either. Eventually some breakthrough will yield 10+ hours on average cell phones and laptops, but until then, I definitely prefer a device that takes very long to boot up, but once up it runs beautifully. As long as talktime/laptop-worktime is 2 hours and standby is 24 hours, I'm a happy camper. For the rare occasions I need more than that, I just buy an extra battery or two off eBay, shipping directly from the Great Republic of China. Takes less than 2 mins to HIBERNATE a laptop, switch the battery, and get back into action.
Since there are few standalone laptop/cellphone battery chargers, I've considered buying a busted cell/laptop of same model to be used as chargers.
Anyway, 99% of the time there WILL be a power outlet where I need it, or I'll be in some car with my DC-AC converter powering my laptop/cell/etc.
And to prepare for emergencies, I'll get this human powered cell & laptop charger when it becomes available:
http://greenlivingideas....ed-cell-laptop-chargers
Score: 0
|Exactly. So in the end I think performance is a bit different for everyone. Some prefer battery time while others, like us prefer the computer's ability to work. Also, I don't quite agree with the battery vs. gas thing. I would gladly take the car that gets more mpg, because at the end of that tank I'm paying directly out of my pocket to refill that tank - it's very different for your laptop. With the laptop, not being required to charge up every say 3hrs or so is a matter of convenience, not value per se [unless you look at it as directly affecting the contribution of pollutants and/or use of our energy sources].
Anyway, I digress. In the end, I'm still old school I suppose - preferring horsepower to eco-friendliness and battery life. In the end, I agree that batteries will have to catch up and only the most demanding of hardware will drain them as quickly as they do/are today.
My netbook won't need 9hrs of battery life....it WILL however need to be able to run a couple tabs and play music without any skips.
Score: 0
|Well, when anyone says "performance", it's universally assumed that he's talking "speed at executing tasks". Let's settle that. And for me it's ok.
However, your concern and personal interest seems to be battery life and the consequent independence it gives to the user, and it's a valid one but rather in the mobile area. At home, it is not really that important except for extreme cases like very powerful PCs for gaming e.g.
There is no word for the "whole scene" performance but you could try naming the concept. It should be some kind of ratio of speed/battery life i guess.
Consider anyway that battery life has become another standard measure in electronics' reviews some years ago. No one misses it anymore.
Score: 0
|The generic term performance is definitely a topic that can no longer be clearly defined by the amount of ram, speed of your CPU, or even battery life. Specifically with computers, weighing the performance of any one element of a computer is just not enough. If we're talking about computers in general we need to even go beyond the hardware and even look at the operating system. MacOS has obvious benefits to Windows, I'm hard pressed to say vice versa, but I'm sure Windows has it's merits in performance as well. Once the entire system is taken in, and a benchmark is formed (which PC magazines do), we have the true system performance, increasing a piece of the puzzle just isn't enough. However, I do definitely agree it's about time that batteries stepped forward and made some advancements.
Score: 0
|