The National Resources Defense Council's flawed game console study

The SciAm blog from November 28th uncritically highlights a claim by the National Resources Defense Council that

video game systems are huge energy wasters, mostly because people (read: kids) tend to leave them on even when they're not using them.

[...]

Sony PlayStation 3 (which uses 150 Watts of energy) and Microsoft Xbox 360 (which uses 119 Watts) are the biggest offenders, while the Nintendo Wii draws less than 20 Watts, according to the NRDC report. The PlayStation 3 and Xbox 360 each if left on all the time [emphasis mine], consume more than 1,000 kilowatt-hours each year—equal to the annual energy use of two new refrigerators. The PlayStation 3, which can also be used as a high-definition video player, uses five times the power of a stand-alone Sony Blu-ray player to show the same movie.

There are a number of things about this report (PDF), or at least the popular reporting of it, that I consider flawed.

1. As we've discussed twice before now [1, 2], the energy used to produce an electronic device is a non-negligible fraction of its total lifetime energy usage (the study I linked to before claims that production of a personal computer [which game consoles effectively are] accounts for ~80% of the total energy usage over its entire lifetime). Focusing solely on energy efficiency in use is a flawed metric to assess the net cost to the climate.
   
2. The report chooses to highlight the energy cost associated with leaving the devices on all the time, with no power saving options selected. This is a maximum cost, but is it at all a realistic scenario? Their figure 1 shows that, according to their own calculations, the net energy use by users who switch their consoles off after use is typically ~10% of leaving it on all the time. Any Xbox 360 user knows that heat associated with use can lead to the Red Ring of Death - I certainly don't leave my Xbox on when I'm not using it. Their summary and the news reports all focus on the worst case scenario, without mentioning the equally plausible or more plausible scenario. Of course, one can't make a big fuss about game consoles being bad for the environment under the conscientious user scenario, can we?
   
3. Any technologically savvy computer user knows that there is a difference between peak energy usage and idle energy usage. The report quotes power uses when active and idle in Table 3 (e.g. an average of 119 W for an Xbox 360 when active, 118 W when Idle). Unfortunately they redefine "Idle" when gaming to mean a game is running, but the user is not touching the controller - this is simply not realistic. Even if we assume people leave their consoles on all the time they certainly don't have games loaded and paused 24/7 365 days a year - I certainly don't. Any Xbox 360 user also is aware of the annoying whine associated with a spinning disk, which no one would put up with 24/7. Yet the NRDC study deliberately rejects manufacturers Idle power usage numbers (i.e. no disk inserted, no game running, what I consider to be a realistic scenario and akin to standard personal computer Idle power ratings) and states on p26 that "Some video game console manufacturers define Idle mode as a state during which there is no game disc inserted in the console. We believe that users are more likely to leave game discs in their consoles when they are left in Idle mode (a user who takes the time to eject the game disc would more likely just power down the console completely rather than leave it running) and have defined this mode accordingly."
   
4. They did not themselves assess usage patterns - in Chapter 5 they quote a Nielsen Group study that finds that "on average, users who account for close to 75 percent of all playing time have their consoles on for an average of 5 hours and 45 minutes per day." This implies that the per console there is 7 hours 40 minutes play time per day. The NRDC study has footnote (p26) associated with the previous quote questioning the Nielsen numbers: "Nielsen’s statistics can be difficult to interpret because the time in Active mode reflects an average across only the days when the console was turned on, rather than a true daily average reflecting use across the entire time metered. It is likely, however, that many heavy users often have the console on every day. For all of these reasons, we built upon the information available and the following assumptions to complete the energy analysis. ... Many users are assumed to leave their video game consoles on when they are finished with a game, even when they go to sleep at night." [emphasis mine]. On p27 they absurdly claim that a Lawrence Berkely National Laboratory study of 60% of their computers being on overnight and over weekends somehow justifies their assumptions about game consoles. This is idiotic - a national laboratory where many computers are workstations that effectively have to be on 24/7 has nothing to do with home entertainment usage by children and young adults. My work machines are on 24/7 365, as they need to be, for example either they're running programs or I need access to the disks from home. My Xbox 360 is only on when I'm using it. Hidden in Endote 1 (p28) is the nugget of information that "Due to the absence of any studies, we based our calculations on the assumption that 50 percent of users leave their device on when they are finished playing a game or watching a movie."
   
5. Items 2 and 3 raise very serious concerns in my mind about how the NRDC assessed (a) the fraction of users who leave their consoles on all the time, and (b) the net energy usage. It is obvious that if either of these aspects of the study is flawed then its conclusions are totally untrustworthy. In fact, both aspects are deeply flawed.
   

Item 4 suggests that the NRDC did not use the Nielsen numbers (I must say I'd consider the Nielsen numbers to be suspiciously high, personally) at all, but simply assumed that 50% of all users leave their consoles on all the time. Note the phrasing of the quote in Item 4: first they assume that "heavy users" leave their consoles on all the time. Then they move from that to assuming all users leave their consoles on. As far as I can tell they did not measure actual power usage. Rather they took their (inaccurately high) "Idle" power numbers and multiplied up by the hours in a year and the number of consoles, and then multiplied by their made-up 50% figure. Assumption after assumption, and all unwarranted.

In summary I think its pretty clear that this study is fundamentally flawed. All assumptions appear to have been made to in order to achieve the result they want, and evidence to the contrary (realistic Idle power ratings using, estimated usage times from Nielsen) has been rejected without justification. The Endnote mentioned above is telling: although this is supposedly a study of the energy cost associated with console use they have no data of actual console usage, and rather than actually trying to find out they simply assume 50% of all consoles are left on all the time. If you're going to do a study why not actually try to find out as accurately as you can the numbers you need? Never mind the issue of the end-to-end energy usage (of personal computers, at least) being dominated by manufacture, not actual use.

I'd suspect that who ever wrote this "study" (deliberate use of sarcastic quotation marks) does not own a game console of their own, and has no training in hypothesis testing or scientific quantitative analysis. The lack of analytical skill or scientific rigor in this study astounds me.

This report reflects very badly on the NRDC and the Ecos consulting group who performed it. This feeds into the "computer games are bad and are making our children bad" narrative our sensationalist media loves to feed to us. Environmental groups should not resort to bogus studies in order to make the case for conserving - doing so weakens the cause and gives ammunition to the denialists and polluters. It also pisses off your potential allies, who don't want to associate with bumblers.

Furthermore, it ignores the big picture. Even if we accepted this flawed study, how significant is this energy waste compared to TVs, lights or other electronics being left on, or cars being idled in the morning to warm up? A decent study would place the numbers in context. This study would never pass any form of peer review - its terrible. I'm also annoyed at SciAm adopting the parroting-the-press-release school of journalism.

Finally the focus on consoles is blatantly a PR gimmick from the NRDC, timed for the holidays so they can get some media attention while picking on a group (gamers) that it is socially acceptable to pick on.

Energy conservation and reducing resource waste is not about consoles, it has to extend to all areas of life. Water, electricity, gas, food stuffs. If you haven't taught your child to turn off the light when he leaves the room then why expect him to turn off the TV, computer, or game console. Its about personal responsibility.
Without that a million studies, flawed or accurate, won't make an iota of a difference.