This mini series of posts will explore the differences between our “cheaper, faster, and better” climate forecasts and the competing fossil fuel emissions – based Global Circulation Models (GCMs) which are widely distributed and relied upon throughout the world today.
There are several underlying principles which guide MWA models that don’t appear to apply to GCM practices. Those principles form part of the reasons that our forecasts are accurate and theirs are not.
The first principle is Reproducibility. This principal involves the ability to reproduce scientific calculations for each step back to the underlying data. Reproducibility was once a hallmark of the scientific method, and we try to carry on the tradition.
Reproducibility has several essential characteristics and drivers. One driver is Accountability. Accountability is roughly a commitment to the Customer and/or the Constituent that Best Management Practices (BMPs) are applied in the development of the Forecast. BMPs themselves form an umbrella for several other practices, such as Best Scientific and/or Engineering Practices, and Best Accounting Practices.
A complement to Reproducibility is Transparency of Performance Skill (TPS). Transparency involves making one’s methods and processes as clear as possible. The more clearly the performance results are revealed to the Customer, the more Transparent the process and accordingly, the more Reproducible the methodology.
For example, our performance disclosures are more transparent than those by taxpayer funded GCM practices. The featured image is a comparison of our (training) forecast performance for annual flows of the Gila River near Gila, NM (in the United States) to a surrogate conventional method (Auto Correlation, AC) and to a GCM based method. Note for comparison, each sim output limited to period from 1987 to 2016. GCM values are adapted from an ensemble of model runs covering a period from 1950 to 2100
The featured image and the remaining comparisons included in this post highlight that across the board, no matter what metric is used, the GCM based method produces the worst estimates of future hydro – climate, and the MWA method produces the best estimates.
Yet this was not straightforward to document for the sample GCM based product because of its poor Transparency. The forecast product for that method did not include any of these critical features:
A. Cost (After several requests, it was finally determined that the official taxpayer costs were $60,000 US).
B. Root Mean Square Error (RMSE). This once was a customary item reported with any predictive hydrologic model. Yet for GCM model outputs, the Customer or another must manually digitize the results in order to derive that information.
C. Goodness of Fit. I used the Chi-Squared method, again based on manual digitization because the GCM report did not include that. In fact, the GCM product did not even include a simple graph for one to visually compare the observed flow time series to the simulated flow. I have once again utilized the manually digitized extract of their ensemble model result in order to produce that comparison product as well.
The metrics that MWA routinely utilizes to give Customers and Constituents a candid accounting for our performance are included in this post as self explanatory charts. As noted, we have included the GCM equivalents because the GCM products have omitted them. Perhaps GCM practitioners have valid reasons for not disclosing the performance of their forecast products, but I have not been informed of any such reasons to date. Future posts will explore more details of the GCM methods and practices in comparison to our own.
We encourage those who pay for services such as water (for irrigation, electricity generation, municipal etc.) to inquire from their providers if underlying water availability forecast methods are as reproducible as those of MWA. If answers are not satisfactory, please consider requesting that the service provider reach out to us. As the final tables show, we may be able to improve performance and save many ratepayers a great deal of money.
|Gila near Gila, NM, avg 1 yta flow is 179 CFS||RMSE||difference cfs||difference acre ft||value at $700/af/year||value at $1,000/af/year|
|MWA vs AC||4.53||3282||$ 2,297,272||$ 3,281,817|
|MWA vs GCM||20.30||14707||$ 10,294,617||$ 14,706,595|
|AC vs GCM||15.8||11425||$ 7,997,345||$ 11,424,779|
|5 year average|
|Gila near Gila, NM, avg 5 yta flow is 183 CFS||RMSE||difference cfs||difference acre ft||value at $700/af/year for 5 yrs||value at $1,000/af/year for 5 yrs|
|MWA vs AC||7.80||5651||$ 19,777,835||$ 28,254,050|
|MWA vs GCM||38.53||27914||$ 97,697,432||$139,567,760|
|AC vs GCM||30.7||22263||$ 77,919,598||$111,313,711|
Copyright 2016, 2017, Michael Wallace, www.abeqas.com
Disclaimer: This is a draft and preliminary post. Original Source for GCM product: Gutzler, D.S. 2013, “Streamflow Projections for the upper Gila River” UNM Contract No. 37675. Submitted to the New Mexico Interstate Stream Commission for Deliverables 2 and 3.