Action-oriented Benchmarking

Energy benchmarking is a powerful way to educate and inspire occupants and other decision makers seeking to improve energy performance and ensure persistence over time. By enhancing the transparency of the energy management process, benchmarking plays an important role in reducing real and perceived risks in market transactions that depend on the comparative valuation of energy use and savings. Buildings energy disclosure ordinances are now requiring benchmarking in some markets (Figure 1), and voluntary efforts such as the Open Compute Project are taking hold in other parts of the technology sector.

Figure 1. There is a trend towards mandatory energy benchmarking and disclosure.

Applications

Energy benchmarking has come to be recognized as an integral part of the process of managing energy use of a facility’s entire lifecycle. Benchmarking the performance of average and exemplary buildings can be used to inform design intent with respect to aspirational energy savings. Best practice technologies and operational procedures can be identified by delving into the design choices and performance outcomes in other facilities. Once design has commenced, model-based benchmarking can be used to compare predicted performance of the subject building to peer groups or specific targets. These benchmarks can continue to be used once a facility is operational (recurring, longitudinal benchmarking over time) to verify attainment of design objectives, to diagnose performance problems, and to track progress towards performance improvements over time and the persistence of savings achieved by physical or operational changes intended to save energy. For all use cases, outliers can be studied to identify best practices as well as critical causes of energy inefficiencies. The University of California Merced Campus has made a substantial investment in building a site-wide benchmarking platform and integrating the information into operations, performance tracking, and decisionmaking (Mercado and Elliott 2012).

A caveat in benchmarking actual measured energy use is that user “behavior” or other operational choices influence the patterns observed as much as do physical attributes of the subject facility. Other confounding factors include variations in weather. Such “operational” variations are one reason that energy intensity is not equivalent to intrinsic efficiency. An alternative to such ratings is to benchmark performance associated only with the facility’s fixed elements (HVAC equipment, envelope, etc.). This is referred to as an “asset” rating, and must be performed using simulation. To use the vehicle analogy, official fuel-economy ratings are derived using a highly standardized test procedure, while the actual performance of a given car often varies significantly from the standard value depending on vehicle loading, how the car is driven, driving conditions, maintenance, etc. Both approaches have value, and it must be kept in mind that, while eliminating various forms of noise from their assessments, asset-based techniques by definition do not capture the very real effects of building operations and management.

At one end of the buildings benchmarking spectrum lies whole-building benchmarking (e.g., ENERGY STAR’s Portfolio Manager), where all forms of energy and all end uses are aggregated into a single metric and compared against loosely similar types of buildings. EPA reports that about 40% of the building stock by floor area has been assessed with Portfolio Manager, and 25,000 buildings have received EnergyStar ratings (as of year-end 2014). The appeal of this approach is that it is conceptually simple and takes less time than other approaches. The limitation is that less actionable information is yielded. While the relative performance of a given building may broadly suggest a potential to save energy, the specific pathways for doing so remain unclear. Leveraging Portfolio Manager's API, some utilities offer "automated benchmarking" using customer data.

At the other end of the spectrum, the most rigorous pathway to identifying applicable energy efficiency measures is through in-depth energy audits and intensive simulation modeling. However, this is a costly proposition and requires considerable engineering expertise. Midway between these extremes is the “action-oriented” benchmarking approach in which specific fuels and end uses are analyzed and logic applied in order to identify candidate energy efficiency recommendations (Mills et al., 2008) (Figure 2). EnergyIQ is one such tool, including application programming interfaces to allow the underlying data and benchmarking engine to be used in any website. . EnergyIQ takes advantage of "features benchmarking" (presence/absence of specific features and their efficiency levels) to inform an opportunity assessment process and recommend specific actions. Examples include lighting ballast upgrades where non-electronic ballasts are existing, hot water water pipe and water-heater tank insulation where none is present, HVAC efficiency upgrades, constant- to variable-air-volume systems, etc.

Figure 2. Action-oriented benchmarking is nested along the spectrum between light-touch,

whole-building analysis and investment-grade audits.

Many building energy benchmarking procedures have been suggested, spanning a range of analytical techniques and data requirements (Li et al., 2014). One review identified 47 protocols for benchmarking non-residential buildings and 31 that applied to residences (Glazer 2006). The diversity of protocols reflects in part the evolving nature of the space and a degree of fragmentation in efforts, as well as a diversity of facility types and use cases that call for varying approaches. For example, specialized tools are available for datacenters and laboratories

Benchmarking Mechanics

Of central importance in the benchmarking process is the assembly of a comparison peer group of facilities. Poor peer group specification can easily lead to a distorted view of how the subject building is performing. More subtle considerations also come into play, for example the operating hours, geography, building size, or vintage. Peer-group definition is an ongoing area of research (Gao and Malkawi 2014), as are methods for ensuring quality data, particularly when disparate sources are combined (Brown et al., 2014).

An example of the influence these considerations can have on how a given subject building is “rated” against given peer groups is shown in a case study of the California Energy Commission’s headquarters (Figure 3). While the peer group sample size necessarily falls as stricter filters are applied, as the peer group becomes more aligned with the subject building the results become more meaningful. In this case, results for the most loosely defined peer group suggest that the subject building was not a particularly good performer. However, upon improving the filtering, relative performance improved considerably.

Figure 3. Illustration of how relative benchmarking outcomes can shift as the peer group dataset is refined.

Peer groups can be derived from statistical surveys of a given building stock, or managers of real estate portfolios can also benchmark within their enterprise, rather than to a broader more varied population of buildings. Individual buildings can be “self-benchmarked” over time in order to track actual changes in performance. The determination of an appropriate peer group is context-sensitive. Most benchmarking tools suffice for common peer groups such as office buildings. In some cases specialized building types (or particular regions) are less well represented.

Once a reasonable peer group is defined, and one or more filters applied to account for characteristics such as location, a benchmarking metric is then computed. The metric’s numerator could be energy or some other quantity of interest such as cost or greenhouse-gas emissions. The denominator is important for normalization. While floor area is widely used, other factors may better characterize activities that occur in the building, such as number of employees or meals served for a restaurant.

The choice of metrics is important, and are sometimes highly specialized. In the case of datacenters, the ratio of total facility energy to the IT-related subset, known as the PUE (Power Usage Effectiveness), is widely used. A self-benchmarking protocol is available for data centers [Greenberg et al. 2006]. Benchmarking can even be performed at the equipment level. For example, special-purpose benchmarks can be computed for high-performance computers such as those used for gaming and special effects. These can be extended even to sub-components such as graphics-processor watts per unit of rendering performance, e.g., watts per frame-per-second (Mills and Mills 2015). Metrics are beginning to be applied to other green attributes, as illustrated by Facebook's benchmarking of "water usage effectiveness" (liters/kWh) in their datacenters.

Improved peer-group data sets are critical. The advent of public-domain “big data” and its application to the buildings energy arena through projects such as the Buildings Performance Database (Brown et al., 2014) is yielding new sources of peer-group data and larger datasets that promise to enable more fine-grain filtering than is currently possible.

An exciting frontier is occupancy-based dynamic load-shape benchmarking, using, for example, mobile phone data to track occupancy loads within a building or campus. Mathieu et al. (2011) illustrate a variety of ways to use 15-minute-interval load data, primarily in the context of demand response decisionmaking.

Institutional Requirements and Capacity

To achieve its full value, benchmarking should inform action. Whole-building benchmarks are highly constrained in this respect because they do not disclose the reasons for particular energy outcomes. A layered approach, however, differentiating types of energy sources by end use, together with a profile of building characteristics and modes of operation begins to form the basis of analyses that can inform the identification of energy efficiency opportunities.

While improved information does not in and of itself achieve energy efficiency improvements, it is critically enabling. A more nuanced view is that benchmarking enables the identification and ranking of opportunities, creates awareness and attention, and provides intelligence that enables building operators to remain vigilant and ensure that intended performance targets are met and persist over time. Benchmarking also builds confidence about performance levels and savings claims, hence managing investment risks. For maximum value, benchmarking should be continuous and the resulting data streams tightly integrated into broader business practices. Disparate audiences—from financial to technical—must be engaged and find value.

The integration of benchmarking with energy data acquisition, visualization, and building management systems has been pursued for some time. Much more can be done to integrate benchmarking into the process of operating buildings and diagnosing deficiencies that lead to energy waste. To be more broadly adopted, benchmarking user interfaces (and associated data visualizations) must be designed with target users in mind, with an emphasis on usability and application to diagnosing and correcting deficiencies.

In parallel with a benchmarking system’s analytical underpinnings is its user interface through which users conduct the benchmarking process. More than a decade ago, Orlov et al., (2003) reviewed the state of the art, including surveys of 22 early-adopter companies that were using computer based information dashboards (for a variety of purposes, outside of the energy domain). They found that these systems were often “tentative and not linked to business processes” and contained “passive displays meant for executive eyes only.” If dashboards aren’t connected to the people who “own” the processes they are evaluating, then the information does not become actionable. A metric that does not fit the need is of little value, and can even be counterproductive.

References

Brown, R.E., T. Walter, L.N. Dunn, C.Y. Custodio, and P.A. Mathew. 2014. “Getting Real with Energy Data: Using the Buildings Performance Database to Support Data-Driven Analyses and Decision-Making.” Proceedings of the ACEEE Summer Study on Energy Efficiency in Buildings, American Council for an Energy Efficient Economy: Washington, DC, pp. 11-49 – 11-60.

http://energy.gov/eere/buildings/building-performance-database

Glazer, J. 2006. “Evaluation of Building Energy Performance Rating Protocols.” Prepared for ASHRAE Technical Committee 7.6 – Systems Energy Utilization. December 6, 255pp.

Greenberg, S., W. Tschudi, and J. Weale. 2006. LBNL "Self-Benchmarking Guide for Data Center Energy Performance (Version 1.0)" [PDF]

Mathieu, J. L., P. N. Price, S. Kiliccote, and M.A. Piette. 2011. "Quantifying Changes in Building Electricity Use, with Application to Demand Response." IEEE Transactions on Smart Grid, 2(3)507-518. http://eetd.lbl.gov/publications/quantifying-changes-building-electricity-use-application-demand-response

Mercado, A., and J. Elliott. 2012. "Energy Performance Platform: Revealing and Maintaining Efficiency with a Customized Energy Information System." Proceedings of the 2012 ACEEE Summer Study on Energy Efficiency in Buildings, pp. 12.166-12.178.

http://aceee.org/files/proceedings/2012/data/papers/0193-000369.pdf

Mills, N. and E. Mills. 2015. “Taming the Energy Use of Gaming Computers.” In review.

Mills, E., P. Mathew, N. Bourassa, M. Brook, and M.A. Piette. 2008. "Action-Oriented Benchmarking: Concepts and Tools." Energy Engineering, 105(4):21-40. LBNL-358E [PDF]

Go To Summary Matrix