Complex and unreliable code, manual processes, and extensive back-and-forth with development teams made it difficult for a Top 20 bank to implement models required for the Comprehensive Capital Analysis and Review (CCAR), a federal stress-testing exercise. The existing environment resulted in delays, data issues and model revisions that put the bank at risk for non-compliance in submitting on time.
Facing multiple pain points with that process, the bank tapped into our expertise and experience to develop a stress-testing framework for implementing forecasting models for its $140 billion loan portfolio and meeting federal requirements efficiently and effectively.
The model implementation team at a Top 20 bank deployed models to be used in its annual Comprehensive Capital Analysis and Review (CCAR) stress-testing exercise. The team implemented code that had been created by a separate model development group for the purpose of modeling the bank’s more than $140 billion loan portfolio and associated balance sheet items. But tedious manual processes, a lack of coding standards, and extensive back-and-forth with the development team resulted in delays, issues with updating key variables during production runs, and costly model rewrites. These challenges put the bank at risk of not complying with the regulatory requirement for timely submission of CCAR results to the Federal Reserve.
Enhance the stress-testing framework to satisfy requirements for timely CCAR execution while implementing models more efficiently and effectively.
“We also developed a framework for reusable utilities and code modules to significantly ease handoffs and make the flow of execution clearer to all teams and stakeholders while simultaneously reducing errors in implementation.”
We began by interviewing participants and stakeholders involved in the implementation process, which uncovered pain points about the handoffs between teams. To betterunderstand these problems, we simulated several end-to- end model implementations and reviewed the complete deployment process. We noted several key challenges – including a lack of formalized procedures when the code was handed over to production, incomplete standard operating procedures for execution, and disjointed coding standards and practices across the development teams.
Although leadership and Risk Management consistently reviewed and approved implementation code as producing desired outcomes, our review of the entire system noted previously unidentified errors. Our testing flagged “hard-coded” variables, buried in the code, that had never been updated, as well as critical gaps in model execution controls that led to inconsistent and unexpected results. Each model implementation required line-by-line code reviews by an implementation analyst to identify and accurately update variables to ensure models would execute properly in the future. Even when the team spotted and corrected issues, the resulting rework and lengthy re-approval process delayed implementation. In other cases, errors were missed entirely and the code was placed into production, with potentially critical impacts to CCAR results.
We instituted enhanced controls in the handover process, to include standard operating procedures and standardized coding practices across all modeling teams. We also developed a framework for reusable utilities and code modules to significantly ease handoffs and make the flow of execution clearer to all teams and stakeholders while simultaneously reducing errors in implementation.
To facilitate ease of use and limit operational risk, we added steps in the procedures for identifying variables clearly and established parameters for updating and documenting variables with each execution. We also automated key handover controls and model run logs, which allowed model risk teams to audit the entire process flow and more easily demonstrate the integrity of the process to regulators.
These frameworks and workflows significantly reduced model implementation turnaround, reduced future implementation errors, and provided transparency on key inputs during production, giving leadership better confidence in how CCAR is executed and governed while meeting key submission deadlines.