Better risk rating models

How CIBC developed a model performance monitoring system

Banks use statistical risk rating models extensively to guide a wide range of decision processes associated with loan origination, account management and portfolio performance analysis. The increased sophistication of modeling techniques and the broader application of models, however, introduce new risk management challenges. In particular: How do we know that our credit risk models are working as intended?

At CIBC, the bank risk management team adjusts cutoff-score criteria based on an assessment of market conditions as well as on the observed default rate. Standard risk scorecards, however, are not flexible enough to incorporate changes in underlying economic conditions.

Rating a model’s performance depends on many factors. For example, a model’s performance is likely to be better in stable economic environments. Given the economic implications associated with a model’s accuracy and effectiveness, issues concerning model validation and performance over time are of obvious concern to the banking industry. Because erroneous or underperforming models can lead to lost revenues through poor customer selection or collections management, it is important to evaluate constantly whether the risk rating models are continuing to rank order well over time.

Size and scale considerations increase the importance of monitoring and validating a model’s performance. Carefully and continually monitoring the performance of risk rating models becomes more important as our retail portfolios become larger and more complex. In all areas of the bank, credit scoring is more and more embedded in decision processes. Critically, gaining results from the bank’s rating system depends not only on improved risk ratings accuracy, but also on how much confidence the bank and its senior managers have in their rating system when making important decisions.

Basel II compliance and business benefits
Basel II requires banks to assess the soundness and appropriateness of internal credit risk measurement and management systems, compelling banks to improve their risk rating systems. Basel II compliance should not be seen as a compliance hurdle but rather as a source of potential business benefits.

In fact, improving a bank’s risk rating system offers immediate business gains. Some recent studies show that moderately improving a credit scoring system can generate more than $5 million in profits for a midsized US bank. That number reflects the profit just from improved loan screening during origination. The figure would be even higher if gains from improved risk-based pricing, better capital allocation and other competitive benefits were included.

Basel II intends to link the regulatory capital of financial institutions with the dynamic risk profile of their business. Risk ranking and forecasting are linked by requiring a portfolio to be segmented into homogeneous pools of risk. At CIBC, we estimate risk parameters, namely probability of default, loss given default and exposure at default, for the regulatory capital calculation. These risk parameters determine the minimum capital requirements for that pool. The capital requirements can then be added across pools to get the total capital requirement. Credit model scores are key inputs for pooling retail portfolios and estimating the risk components.

Consequently, monitoring the performance of the underlying rating systems is a key component of the supervisory review process. Regulators expect that a Basel II implementation leads to better risk management and, ultimately, produces tangible benefits for the business.

Reporting requirements
Across the retail risk management group at CIBC, there are more than 50 credit scoring models in use. Some scorecards are purchased from vendors who provide a “black box” solution. Generic scorecards, such as the Beacon score, are also frequently used on a variety of portfolios. Since there are so many scoring models in use, the challenge is how to manage model performance, validation and monitoring tasks continuously in an effective and efficient manner.

Credit rating systems and scorecards typically have a limited useful life span of two to three years before they need to be replaced or recalibrated. New loan products and new scorecards will certainly be added over time. Moreover, credit model performance monitoring and validation techniques and processes will continue to evolve, as will the credit rating systems to which these methodologies are applied. The risk rating system needs to anticipate such developments to the extent possible. It must be flexible enough to accommodate future changes, incorporate new scorecards easily and efficiently remove retired scorecards.

For the bank, different levels of reports are needed for different audiences:

• Detailed, granular reports for analysts.
• Summary reports for business units.
• Highlighted executive summary reports of all models for line-of-business executives.
• Partial or whole reproductions of reports are required for internal audit, vetting and other regulators. 

Preparation must be in place to produce any of these reports in a timely manner or on short notice.

To handle all of these requirements to mitigate model risk, it is necessary to develop a scalable, flexible and truly integrated monitoring reporting system.

Modular integrated reporting system
In the past, the first version of the model monitoring system was a “quick and dirty” job, and the report for each scorecard was built in a one-off process. There was no scalability. After a new risk rating model was commissioned, hard coding processes were needed to accommodate any new models. The previous system used different software and platforms. Heterogeneous and convoluted software languages, methods and different platforms created many unwanted effects, including performance issues with large data sets, maintenance issues and hard-to-accommodate changes.

The Product Risk Analytic Team of Retail Risk Management at CIBC has developed the Credit Model Performance Monitoring and Validation system, which is a centralized, enterprisewide monitoring system. The new reporting program uses SAS as its main programming language.

The reporting process is mostly automated, standardized and scalable: pulling data from the database, performing a series of statistical calculations and derivations and producing a set of performance monitoring reports. The new system is designed around a modular architect offering many benefits and flexibilities, including the ability to change or add scorecards, change analytical methods, conduct ad hoc analysis and run the process on a scheduled timetable in a production environment. These are done in a straightforward manner that involves no hard coding. This new system integrates seamlessly into the bank’s current SAS infrastructure.

With continued investment in risk management, we believe the bank’s investment in the Credit Model Performance Monitoring and Validation system will be well placed to support future requirements.

With the new reporting process in place, monitoring and improving retail risk rating models has become routine, quick, easy and cost-effective throughout the model life cycle. In today’s environment of high volatility and uncertain credit risk, the importance of consistently monitoring and determining the effectiveness of risk rating scorecards has never been more important. Managers need to be confident that the risk models are methodologically sound, perform appropriately and consistently, and meet the industry’s best practices standards.

Bio: Myint Zaw, a Senior Quantitative Analyst in Product Risk Analytics of Card Products Risk Management at CIBC, is assessing current research, analytical concepts and methodologies for applicability and value at CIBC.

Myint Zaw, CIBC