Questions Executives Should Ask Model Providers

By Ming Lee

Over their nearly 25-year history, catastrophe models have evolved into pervasive risk analysis tools. For companies exposed to natural and man-made hazards, catastrophe model output provides critical input for many significant business decisions. However, the rush to deploy results has sometimes outpaced careful evaluation of the models themselves, the data being input to the software, the workflow generating those input files, and the way users take actions based on the output.

One problem is that many busy executives are overwhelmed by the quantitative and scientific details and therefore either defer model evaluation to those with technical training or avoid it entirely. Regard­less of whether you have experts on staff, keeping a few questions in mind and asking them often can go a long way toward performing the basic due diligence necessary to ascertain whether your organization is viewing models with an unbiased, even skeptical, eye. A good place to begin your evaluation of the effects of models on business performance is by vetting the model and its provider. Here are some useful questions executives should ask model providers:

What is your approach to model validation?
All components of a catastrophe model should be independently developed and validated to the greatest extent possible, but appropriate oversight is necessary to make sure the component parts produce a coherent whole. Model providers should be able to demonstrate that their internal processes ensure final model output is consistent with basic physical expectations of the underlying hazard and unbiased when tested against both historical and real-time information.

How has your view of the risk evolved?
Models change as new science is vetted, new data becomes available, and the user marketplace demands solutions to new problems. Change itself is not a sign of weakness, but neither is it a sign of strength. Change should be thoroughly justified and competently managed, with an emphasis on communication and understanding of the business consequences. In the end, the numbers need to make sense.

What are your processes for helping your clients manage the impact of model updates and changes?
Business consequences are often determined not only by what comes out of the models but by the degree of stability of model results. Before implementing changes, model providers should carefully consider how such changes will affect results. When updates are made, a client-specific service infrastructure should be in place accompanied by thorough and transparent communication of component changes, underlying reasons, estimated impacts, and suggestions for transition.

Why should I trust your scientific, engineering, and actuarial expertise? What distinguishes you from other firms?
Model providers all employ experts, but they should be prepared to explain their capital investments, human and otherwise, and demonstrate whether those investments have resulted in a robust, credible analysis of natural hazard risk. If the model provider outsources components, what quality controls are in place to ensure that final model output makes sense?

Have independent experts reviewed your model?
As in all scientific endeavors, independent peer review is the gold standard of credibility. The model provider should be able to provide you with names and credentials of reviewers who are independent of the enterprise and recognized as experts in their respective fields.

What's unique about your model architecture?
Despite certain common underpinnings in science, engineering, and historical observations, all models are not built the same way. Architecture affects everything from the ability to accept refined inputs to the credibility and versatility of outputs. Model providers should be able to honestly assess both their strengths and limitations.

What are your models leaving out? What other information should I be using to understand risk?
Model providers freely admit that models are tools, not oracles. A comprehensive understanding of your business is necessary to integrate other sources of quantitative and qualitative information into any corporate evaluation of disaster risk. Model providers should provide transparency by clearly stating what's included in a particular model (e.g., subperils, lines of business, coverages) — and what's not. Model providers should also be able to offer expertise and consulting services to assist in filling in the gaps.

As chief executive of a company driven by science and technology as well as human capital, I've learned that repeatedly posing the right questions to decision makers both inside and outside the firm is often the most productive action to ensure robust processes and credible results. Insurance leaders who manage the complex decision analytics of catastrophe modeling should ask questions and insist that the answers reflect the best practices needed to provide confidence that the tools are being used properly.

Ming Lee is president and CEO of AIR Worldwide. AIR offers analytical tools and software systems to help customers estimate and manage the risks associated with natural and man-made catastrophes, weather, and climate.