SS3/18 – Model risk management principles for stress testing

Principle 1 – Banks have an established definition of a model and maintain a model inventory

P1.1 Definition of a model: Banks should establish their own definition of a model. When identifying models banks are expected to take into consideration:

1. (a) Calculation methods or systems that are based on statistical, financial or economic assumptions (eg impairment models, income models).
2. (b) Calculation mechanisms used to transform a set of parameters or values into a quantitative measure (eg scenario expansion models, probability of default models).
3. (c) Frameworks or systems where qualitative judgement is applied to generate quantitative results (eg where adjustments are made to address known model limitations).
4. (d) Calculation mechanisms where outputs of other models are used to calculate financial/risk measures (eg expected loss which uses the output of probability of default, loss given default and exposure at default models).

In cases where calculation mechanisms are not classified as models, banks should ensure the risks associated with the implementation and use of such calculations are adequately understood, controlled, and documented as part of an established management control process.

P1.2 Model inventory: Banks should maintain a comprehensive set of information on models ‘implemented for use’, ‘under development’, or ‘recently retired’. The information should clearly identify model owners and users, and should also include all model uses and direct or material dependencies, ie models that depend or use the output of other models. A designated internal party⁴ should be responsible for maintaining the bank-wide inventory of all models. Any variation of a model which requires separate validation and approval should be classified as a separate model.

Principle 2 – Banks have implemented an effective governance framework, policies, procedures and controls to manage their model risk

P2.1 Board of directors and senior management responsibility:⁵ The board of directors should establish a framework for the management of model risk and this should be adequately documented. Senior management is responsible for the execution and maintenance of the framework and should designate the roles and responsibilities for the framework to model owners, model users, and control and compliance functions.⁶ The board of directors and senior management are expected to provide challenge to model outputs and understand model capabilities, the model limitations, and the potential impact of model uncertainty for the most material models and the aggregate outputs.

P2.2 Model risk management policies: These should cover all aspects of model risk management, including model definitions; model development standards; model change; implementation; use; validation; review; and management sign-off. The policies should set out appropriate governance and challenge frameworks, and the roles and responsibilities of model owners, model users, and control and compliance functions. The prioritisation, scope and frequency of validation, review, and monitoring activities should also be set out in the policies.

P2.3 Model developers, owners, users and control functions: Model developers should be responsible for the development, evaluation and documentation of models and may be involved in model monitoring and a reassessment of already implemented models. Model owners should have accountability for model use and performance. Model owners should be responsible for ensuring that models are appropriately developed, conceptually sound, implemented, used as intended, have undergone appropriate validation and approval, and are recorded and maintained in the model inventory. Model users should ensure that models are used consistently with the model’s intended purpose, and that any model limitations are understood and taken into consideration when using the output of the model. Control functions should have the authority to restrict the use of models and monitor any limits on model use.

P2.4 Role of Internal Audit (IA): IA should assess the overall effectiveness of the model risk management framework. IA should evaluate and independently verify whether model risk management practices are comprehensive, rigorous, and effective.

P2.5 Use of external resources: If external resources are used for any model development, validation, or review activities, banks should be able to verify that these are conducted in accordance with their model risk management standards. Designated internal staff should be responsible for the work delivered by the external party, and should be able to address any issues identified either with model development or as a result of model validation.

Principle 3 – Banks have implemented a robust model development and implementation process and ensure appropriate use of models

P3.1 Model purpose and design: The purpose, design, choice of parameters, mathematical theory, and underlying assumptions of a model should be appropriately documented and conceptually sound (appropriate for the intended purpose), and supported by published research, where available, and generally accepted industry practice where appropriate. Particular emphasis should be placed on model limitations and, where appropriate, model results should be supported by a comparison with alternative theories/approaches, or by assessing the sensitivities of changes in model inputs.

P3.2 Use of data: The data used to develop a model should be assessed for quality and relevance. Where adjustments are made, proxies are used, or where the data are not representative of the bank’s portfolio or asset mix, the impact should be justified and documented so that users are aware of the potential model limitations.

P3.3 Testing: Appropriate testing of models should be conducted to take into account potential limitations, assess their robustness and stability over time, and across a variety of economic and market conditions, in particular those relating to periods of stress. Testing activities should be appropriately documented.

P3.4 Documentation: Banks should have sufficiently detailed model documentation so that an independent third party with relevant expertise is able to understand how the model operates, identify its key assumptions and limitations, and replicate any parameter estimation and model results. Where a bank uses vendor models, it should have appropriate documentation on the approach to be able to validate the model.

P3.5 Use of judgement: Any judgements or model overlays that are used to modify the parameters, inputs and/or outputs of a model due to known model limitations should form a part of the development process, should be appropriately understood and documented, and should be subject to review and challenge by independent parties.

P3.6 Supporting systems: Model calculations should be implemented in information systems or environments which have been thoroughly tested for this purpose. The findings of any system and/or implementation tests should be documented.

P3.7 Business involvement: Business experts should play an integral part in the design and testing of models and should challenge the methods, the underlying assumptions, and the output of the models – both at inception and on an ongoing basis.

P3.8 Model uncertainty:⁷ Banks should implement a model risk appetite framework and demonstrate that model uncertainties are adequately understood, managed, monitored and reported, both on an individual level as well as in aggregate. Model uncertainty should be accounted for in the results and where conservatism is used to mitigate model uncertainty, banks should justify and document any such adjustments and demonstrate that the adjustments are intuitive from a business and economic perspective.

P3.9 Monitoring: Banks should perform periodic monitoring of model performance to ensure parameter estimates and model constructs remain fit for purpose and use when sufficient new observations are available and to ensure model assumptions remain valid. The frequency of model monitoring should be commensurate with the nature and materiality of the models and risks, with due consideration given to model complexity.

Principle 4 – Banks undertake appropriate model validation and independent review activities to ensure sound model performance and greater understanding of model uncertainties

P4.1 Scope of validation and review: All model components (inputs, calculations and reporting outputs) should be subject to independent validation for both in-house developed models and vendor models. Any validation work undertaken by model developers and users as well as any material changes to already validated models or overlays should be subject to review by an independent party. The nature and extent of validation and independent review should be appropriate with the overall use, complexity, and materiality of the models, model components, adjustments to model results or changes to a model.

P4.2 Independence: The people performing model reviews should be independent of the model development process to be able to provide a robust and objective view, evidenced by the quality of issues identified and the actions taken to address them.

P4.3 Competence and influence: Banks should consider whether people performing validation have: the necessary knowledge, skills, and expertise to perform model validations; an adequate degree of familiarity with the business, product, risk, and intended use of the model; and sufficient influence and stature within the bank to ensure that issues and deficiencies are escalated and addressed in a timely manner.

P4.4 Treatment of model issues and/or deficiencies: When significant model deficiencies and/or errors are identified during the validation process, banks should consider whether the use of models should either be prohibited or only be permitted under strict controls and mitigants. The process of managing identified model issues should include the tracking of the outstanding issues and should be adequately documented.

P4.5 Frequency of model validation: Banks should undertake regular revalidation of models to track known limitations and to identify potential new issues. Periodic reviews should be carried out with a frequency and level of rigour commensurate with the overall use, complexity, and materiality of the models.