Commercial Building Energy Baseline Modeling Software: Performance Metrics and Method Testing with Open Source Models and Implications for Proprietary Software Testing

A number of Energy Management and Information System (EMIS) software products designed for commercial buildings offer advanced functionality whereby energy savings is estimated by comparing pre-and post-installation energy use baselines determined through empirical modeling using interval meter and other types of data, with little or no need for on-site data collection.

The overarching goal of this work is to advance the capabilities of technology evaluators in evaluating the building-level baseline modeling capabilities of Energy Management and Information System (EMIS) software. Through their customer engagement platforms and products, EMIS software products have the potential to produce whole-building energy savings through multiple strategies: building system operation improvements, equipment efficiency upgrades and replacements, and inducement of behavioral change among the occupants and operations personnel. Some offerings may also automate the quantification of whole-building energy savings, relative to a baseline period, using empirical models that relate energy consumption to key influencing parameters, such as ambient weather conditions and building operation schedule. These automated baseline models can be used to streamline the whole-building measurement and verification (M&V) process, and therefore are of critical importance in the context of multi-measure whole-building focused utility efficiency programs.

There are two reports delivered as a part of this PG&E Emerging Technologies Technology Development Support Project (ET12PGE5312): "Commercial Building Energy Baseline Modeling Software: Performance Metrics and Method Testing with Open Source Models and Implications for Proprietary Software Testing" (referred to as the "Model Analysis Report") and "Functional Testing Protocols for Commercial Building Efficiency Baseline Modeling Software" (referred to as the "Test Protocol Report"). The Model Analysis Report documents the findings of a study that was conducted to begin answering critical questions regarding quantification of savings at the whole-building level, and the use of automated and commercial software tools. To evaluate the modeling capabilities of EMIS software particular to the use case of whole-building savings estimation, four research questions were addressed:

1.       What is a general methodology that can be used to evaluate baseline model performance, both in terms of a) overall robustness, and b) relative to other models?

2.       How can that general methodology be applied to evaluate proprietary models that are embedded in commercial EMIS tools? How might one handle practical issues associated with data security, intellectual property, appropriate testing ‘blinds’, and large data sets?

3.       How can buildings be pre-screened to identify those that are the most model-predictable, and therefore those whose savings can be calculated with least error?

4.       What is the state of public domain models, that is, how well do they perform, and what are the associated implications for whole-building measurement and verification (M&V)?

The Test Protocol Report describes procedures for testing and validating proprietary baseline energy modeling software accuracy in predicting energy use over the period of interest, such as a month or a year. The procedures are designed according to the methodology used for public domain baselining software in the Model Analysis Report. The test procedure focuses on the quality of the software’s predictions rather than on the specific algorithms used to predict energy use. In this way the software vendor is not required to divulge or share proprietary information about how their software works, while enabling stakeholders to assess its performance.

An independent technical review conducted by SAM Associates of these two reports (Model Analysis and Test Protocol) and of a related report titled "Energy Management and Information System Software Technology Assessment: Considerations for Evaluating Baselining and Savings Estimation Functionality" (ET12PGE1311) is also published below ("ET12PGE1311_ET12PGE5312_TechnicalReview").