NCA producing the world's most inaccurate climate models at a breakneck pace.

From a multidecadal base of experience in modeling under Best Management Practices (BMPs), I’ve been exploring the hydrological products authored by the National Climate Assessment (NCA) [1].   BMPs have been universally adopted by many industries.  This is because their bottom lines require that models can be relied upon for specific purposes.  In this context, what is best for industry should also be best for the public.  The category tests below relate to BMPs and so cover the general notion that a model will be as traceable, reproducible, and as accurate as reasonably achievable.  Without adhering to those goals, one can have no confidence in the model results, and a resulting inaccurate model can obviously be costly and risky for many public enterprises.

I review the models of others from the perspective of BMPs as a part of my own forecast business line, largely because there are no other existing independent model evaluation resources.  I try to adhere to objectivity if only because my own credibility depends on that.  For example, although my business can provide a form of cyclonic forecasting, I have favorably reviewed hurricane related forecasts by others. Any are free to contact this site if they require further clarification of the scores I assign.

The National Climate Assessment has been reviewed within this framework, with a specific focus on their hindcasts of Gila River flows in New Mexico.   Readers who may be stakeholders in the NCA results (covering many streams in the US) are welcome to alert the NCA authors and practitioners of this review so that future revisions might benefit.  Or perhaps if my review is not acceptable to them, that they may communicate with me to allow us to arrive at common standards together.  For now the NCA score is considered poor based on these ten metric assessment summaries.

  1. Poor or impossible reproducibility.  (for example, see item 3).
  2. Poor or nonexistent transparency (for example, see item 3).
  3. High bias in model results (only found after time consuming work on my part because of bullets 1 and 2 above).
  4. Poor accuracy.  Based on the many models I’ve reviewed and the figure below, the NCA models appear to rank among the most inaccurate hydrology models of any I have yet reviewed.
  5. High cost.  It appears that scores of western hydrology model studies, involving thousands of model simulations, have been conducted at a taxpayer funded cost of millions of dollars per study, if not more.  However given the poor transparency, none of the major reports appear to disclose the taxpayer costs.  Searches of “Westwide Climate Assessment” plus “Bureau of Reclamation” plus “cost” or “fund” or “$” come up empty.  A search of a closely related study for the Santa Fe Basin found 34 mentions of the word “funding” but no actual dollar values were published.
  6. High visibility.   For example, this recent revision of the NCA has been featured by over 1000 media outlets.
  7. Consistent emphasis upon alarming outcomes.  Readers are encouraged to visit the link in item 6 to easily verify.
  8. Consistent disregard of actual data and alternate conceptual models.  Among the more notable examples are the NCA products’ widespread lack of comparisons of their CMIP and VIC based* model projections to actual hydrologic data.  That is also a characteristic of item 1 and 2 above.
  9. Extensive claims of attention to details, without the actual attention to details.  I elaborate regarding a “bias-correction” term shortly in this post.
  10. Extensive claims of certitude, without the justification.  In the BMP world it is typically understood that even well calibrated models are still associated with high uncertainties.  Accordingly any statements of certainties, especially those based on poorly calibrated models are topics of concern.  One of many examples: ” Reduced U.S. snowfall accumulations in much warmer future climates are virtually certain as frozen precipitation is replaced by rain regardless of the projected changes in total precipitation amounts ” [2].

Over recent years, I’ve given several presentations on my research to peers and others in scientific venues.  I’ve sometimes used those opportunities to point out a number of the NCA’s deficiencies.  To date at those venues (and outside of them) my concerns on these issues have not been addressed by  NCA practitioners who were present.  As I wait for the growing army of newly minted hydrologic climate change expert peers to return to BMPs, I can only work to ensure my own products score high according to the ten metrics above.

About the Featured Image and this companion image: 

The monthly values profiled in the featured image and the annual values profiled below are exemplary of each of the 10 concerns I’ve detailed above.  Not only are the errors much higher than those from conventional methods, but the bias in the model simulations is also notable.  This can be easily confirmed by the fact that none of the model results capture the lower end of the actual Gila flows.  This is of natural concern given the ultimate source of the simulations [3] which includes the term “bias-corrected” in their title.  In identifying this peculiar combination of claiming something is not biased in a title for a document that is replete with readily quantified bias, I was motivated to insert the third and ninth concerns above.

GilaNCAPoorModelSkillExample39

As a convenient resource to additionally document these eleven NCA deficiencies I share a few links to past posts below:

MW&A continues to outperform Federal and UN IPCC climate projections

 

Update on MW&A Climate Forecasting Accuracy in 2016

 

 

Better Forecast Accuracy Through Reproducibility

Preliminary evaluation of West-Wide Climate Risk Assessment Forecasts

 

References

[1] USGCRP, 2017: Climate Science Special Report: Fourth National Climate Assessment, Volume I[Wuebbles, D.J., D.W. Fahey, K.A. Hibbard, D.J. Dokken, B.C. Stewart, and T.K. Maycock (eds.)]. U.S. Global Change Research Program, Washington, DC, USA, 470 pp, doi: 10.7930/J0J964J6.

[2] Wehner, M.F., J.R. Arnold, T. Knutson, K.E. Kunkel, and A.N. LeGrande, 2017: Droughts, floods, and wildfires. In: Climate Science Special Report: Fourth National Climate Assessment, Volume I [Wuebbles, D.J., D.W. Fahey, K.A. Hibbard, D.J. Dokken, B.C. Stewart, and T.K. Maycock (eds.)]. U.S. Global Change Research Program, Washington, DC, USA, pp. 231-256, doi: 10.7930/J0CJ8BNN.

[3] reference for actual observations:  # USGS 09430500 GILA RIVER NEAR GILA, NM

[4] US Bureau of Reclamation, 2011, West-Wide Climate Risk Assessments: Bias-Corrected and Spatially Downscaled Surface Water Projections  Technical Memorandum No. 86-68210-2011-01 

[5] Gutzler, D.S.  2013.   Streamflow Projections for the upper Gila River.  New Mexico Interstate Stream Commission.  UNM Contract No. 37675

 

* CMIP and VIC are acronyms associated with greenhouse gas emissions based numerical climate model products

This is a work in progress and includes opinions.