Investments in AI-Fueled Innovation May Generate Significant Returns - But Executive Management and the Board Must Pay Heed to the Rising Risks and Costs of an Expanding Algorithmic Infrastructure
The recent non-stop headlines, consultancy pitches, and LinkedIn posts trumpeting AI-fueled business transformation might have you thinking that we just entered the AI age, but AI technologies have been permeating organizations - particularly banks - for several years now. What is different today, however, is the broadening of AI deployment to ever more strategic applications. What began with very focused functionally-concentrated use cases leveraging primarily traditional (i.e., numeric) data - such as managing fraud and money laundering risks, automating certain operational processes, optimizing customer revenues, and managing credit risks - has now exploded into a panoply of new, potentially-transformational enterprise-wide use cases built upon new deep learning and Generative AI tools (collectively, "AI") involving text, audio, images, and (very soon) video.
The potential transformative nature of these technologies cannot be denied, and many financial institutions ("FIs") are investing significant sums in AI experimentation, development, testing, and deployment. In fact, according to PwC:
"Seventy-three percent of US companies have already adopted AI in at least some areas of their business ... and generative AI (GenAI) is leading the way. One year after ChatGPT hit the market, more than half of the companies we surveyed (54%) have implemented GenAI in some areas of their business."
Indeed, FIs without an AI transformation strategy risk the market's wrath - with potentially lower market valuations fueled by fears of customer attrition and revenue erosion from AI-powered competitors, as well as subpar operating margins from "old school" processes lacking the optimizations made possible through AI-fueled innovations. For these reasons, AI has now reached executive management and Board levels as an integral part of corporate strategy.
But for all the potential that AI transformation has, there is an important downside that receives scant attention, yet jeopardizes the realized returns on this strategic investment - the growth in organizational risks and costs accompanying an unfettered expansion in an FI's "algorithmic infrastructure".
Let's explore this.
AI Transformation Is Somewhat Unique Relative to Other Technology Cycles in the Sheer Volume and Diversity of Important and Powerful AI Development and Deployment Tools Easily Accessible to Developers
With other technology cycles, the early stages were characterized by a proliferation of market entrants attempting to capture market share through differentiated commercial offerings. And that is no different for AI. Indeed, as I have written elsewhere and as illustrated in Figure 1 below, algorithmic developers have access to a vast and growing “superstore” of AI development and deployment tools and technologies they can leverage to effectuate the FI's digital transformation - including various AI training algorithms, technical/operations platforms supporting these algorithms, various structured and unstructured database products, and numerous AI development tools.
What is distinctly different in this cycle, however, is the sheer volume and diversity of powerful, publicly-available, open-source code, curated datasets, and pre-trained models to accelerate algorithmic development and deployment. And developers love to experiment with these tools because ...
AI Transformation is Inherently a Dynamic, Innovative, and Experimental Process Dependent on Human Ingenuity and Intellectual Curiosity
As FIs build out their data science teams, it is fairly common for new hires to have proficiency in only a subset of these commercial and open-source tools based on their prior experiences - for example, PyTorch instead of Tensorflow for deep learning, or Llama 2 rather than ChatGPT for Large Language Models ("LLMs"). In many cases, the applicant's experience with tools that are different from the FI's current toolset is viewed as a plus for the applicant's candidacy - allowing the team to expand its algorithmic toolbox. And the existing team isn't just standing still either - they are also looking for opportunities to expand their knowledgebase and skills by experimenting with new algorithms / algorithmic architectures such as neural networks, transformers, and ensemble algorithms to see if they can address a new use case, eke out additional performance gains on existing use cases, or speed up model training or deployment.
The natural inquisitiveness of data science talent and AI engineers - particularly about the newest algorithmic research, the newest / "coolest" open-source development tools and codebases, and new open-source datasets permitting algorithmic training and/or testing for new AI-powered solutions - is exactly the mindset needed for FIs to "stay in the game" against competitors who are also heavily investing in AI transformation initiatives. However, these positive behaviors can also inadvertently lead to the growth of an algorithmic infrastructure that creates systemic - yet under-the-radar - organizational risks and costs that may jeopardize the realized returns from the FI's AI transformation initiatives over time. In effect, they perpetuate a type of technical debt ("algorithmic sprawl") that becomes increasingly difficult for the FI to unwind without large-scale disruption. And these systemic risks are more likely in large FIs with sizable data science teams spread out across a web of decentralized analytical business units.
So what are these risks and costs?
The Pitfalls of Algorithmic Sprawl
There are several - including the following:
Pitfall 1: Insufficient Vetting of New Algorithmic Methods
It is very common for code packages implementing the newest algorithmic methodologies, algorithmic architectures, and pre-trained models to become freely available from GitHub shortly after (if not upon) their initial public release. This fairly rapid and easy accessibility to the newest algorithmic research allows developers to experiment with cutting-edge algorithmic solutions for a whole host of FI use cases - potentially solving a key transformation roadblock or increasing materially the return on an existing use case solution.
However, in most cases, these algorithms and their implementation artifacts have not yet passed rigorous peer-reviewed vetting. Accordingly, without sufficient due diligence by the FI into their theoretical soundness, limiting assumptions, use case appropriateness, and technical validity, the use of these algorithms can create various operational risks that - at a minimum - could adversely impact an important FI use case and - at worst - could have a larger, systemic adverse impact should their deployment become widespread across a business unit or the enterprise.
Pitfall 2: Excess Algorithmic Complexity
AI algorithms are diverse when it comes to their technical complexity - with some being relatively easy to understand, explain, and evaluate, and others being incredibly complex, opaque, and difficult - if not impossible - to fathom. Accordingly, when a developer deploys an algorithm for use cases where the algorithm's complexity is not justified by a sufficient increase in the net benefits it produces (i.e., relative to a simpler algorithm), the FI is likely sub-optimizing its AI investment returns.
This occurs because complexity isn't costless - whether due to less transparency, impaired explainability, greater training and inference costs, greater operational risks, or greater governance costs from specialized first, second, and third line of defense resources needed to maintain and test the algorithm. For example, if a developer chooses to use a complex neural network or ensemble of models for a specific use case when a simpler regression model or decision tree produces practically equivalent performance, the risk-adjusted return on this algorithmic choice is likely sub-optimal since the benefits (i.e., the predictive performance) are about the same as generated by the simpler algorithms, but the risks and costs associated with the increased complexity are much higher.
Anyone who has worked with data science professionals knows that, in general, they tend to be intellectually curious and naturally gravitate to new, "cool" algorithmic solutions to expand their knowledge and skills, to be at the cutting-edge of the field, and to potentially produce publishable research related to these new algorithms. And such innovation-oriented behaviors are important. However, such behaviors can also create a tension when selecting an "appropriate" algorithm to address a given use case - perhaps giving an edge to the more complex solution even though the added complexity may not be justified based on a purely "net benefits"-based assessment as described above.
Pitfall 3: Tension with Climate and ESG Goals
An FI's AI experimentation, development, testing, and deployment have both direct and indirect costs - some of which are not fully transparent, appropriately allocated, and properly considered when evaluating AI investment returns. For example, the incremental costs associated with large-scale structured and unstructured data storage and expensive GPU computational resources can be significant for large-scale, highly complex AI models - particularly those relying on neural networks and GenAI. And beyond the direct costs of these resources, unfettered AI experimentation, the training and deployment of large-scale GenAI models, and even the growing inventory of more traditional AI production algorithms can collectively generate material carbon emissions through their associated data center and CPU/GPU server activity - thereby compromising the FI's ability to meet its specific ESG goals related to net zero emissions and carbon footprint.
Pitfall 4: Algorithmic Key Person Dependencies
AI, in general, is very complex and the pool of specialized, appropriately-trained resources who can competently build and maintain such algorithms is currently relatively small. Accordingly, an FI's use of novel and/or highly complex algorithms whose underlying theoretical and technical foundations, implementing code packages, complexity, risks, and limitations are only known well by a small group of internal data scientists, external vendors, or outside consultants exposes the FI to potentially material key person dependencies.
Pitfall 5: Non-Strategic Algorithmic Infrastructure Growth
In most organizations, investment dollars to fund AI development are initially limited as management awaits a compelling proof-of-concept of such investments. This naturally leads to the development and use of more ad hoc AI development and deployment systems - many based on open-source components such as those contained in Figure 1 above.
For those organizations whose developers are decentralized across business units (or functional areas), one may also observe the organic evolution of different AI infrastructures across such units (or sometimes even within some units) built upon different MLOps platforms (e.g., Amazon Sagemaker, MLFlow, or DataBricks), different analytical programming languages (e.g., Python, SAS, or R), different AI code bases (e.g., PyTorch or TensorFlow), different development tools (e.g., Weights & Biases or Comet), and different large foundation models (e.g., ChatGPT or Llama 2). This relatively ad hoc organizational infrastructure growth and fragmentation can create excessive costs (e.g., paying for multiple MLOps platforms), key person dependencies, technical debt, and greater second and third line of defense governance costs.
Pitfall 6: Open-Source Risks
As mentioned previously, the AI technology cycle is arguably unique in terms of the sheer volume of open-source datasets, code bases, and tools readily available to develop or deploy AI-powered solutions. However, without sufficient due diligence, FIs can be exposed to significant risks related to their reliability, appropriateness, provenance, legal restrictions, security, etc. For example, open-source datasets may have been created in violation of applicable consumer privacy laws and regulations, the code packages may have been created in violation of others' intellectual property rights, the FI's intended usage may be in violation of the open-source license terms, and modifications of the code packages for the FI's specific use may inadvertently require public sharing of such code as a derivative work.[1] More practically, open-source code - particularly for newer algorithms and tools - may be updated more frequently than commercial code bases, thereby causing potential system integration issues, or the persistence of errors if the FI doesn't systematically track and install timely critical updates.
Overall, while passive technological agnosticism in the building of an AI infrastructure can certainly benefit an FI via accelerated innovation, increased development and deployment productivity, lower AI technology costs, and higher analytical team morale, such a passive, ad hoc AI infrastructure "strategy" will eventually become incongruent with enterprise-wide risk, compliance, and financial management objectives - thereby jeopardizing the realization of expected returns on the FI's AI transformation investments. Additionally, the longer the period over which these infrastructures are used, and the more systemic an FI's AI transformation becomes, the more difficult and expensive it becomes to unwind these legacy infrastructures in favor of more strategic technology roadmaps.
But Doesn't the FI's Model Risk Management Group Address These Pitfalls?
Partially, but not fully.
FIs with a well-designed and well-functioning Model Risk Management ("MRM") program may mitigate certain algorithmic-specific risks during model validation activities - such as algorithmic methods lacking sound bases or implemented using unvetted open-source code packages. However, even in these instances, how MRM defines and enforces the criteria of "sound basis" and "unvetted" can vary depending on the function's leadership personnel and whether the program has explicit criteria for evaluating the theoretical and technical soundness of new/novel algorithms.[2]
In general, though, many MRM programs are not currently designed to identify, mitigate, or track the types of organizational risks and costs described in the previous section. For example,
They are likely not performing a "net benefits test" to evaluate potential excess algorithmic complexity as part of their model validation procedures. And, even if they were, they are unlikely to reject an unnecessarily complex algorithm if it otherwise poses an acceptable risk - primarily due to the likely long redevelopment time required to replace it with a simpler algorithm. At worst, such excess complexity would likely yield a recommendation for developers to research simpler algorithmic solutions during future redevelopment efforts.
They are unlikely to delve into the types of inquiries and analyses necessary to ascertain the presence of key person dependencies - other than an evaluation of whether developer documentation meets MRM's requirements. However, having sufficient documentation and having personnel who understand the information contained in this documentation are not the same thing.
Analyzing the developer's AI infrastructure for organizational redundancies or inefficiencies - or whether the infrastructure is based on a thoughtful, strategic design - is not a typical component of most MRM programs.
Assessing climate / ESG impacts of first line AI experimentation, development, and deployment activities is typically out of scope for most MRM programs.
While many FIs have policies governing the use of open-source software, it is more typical for first-line management, IT, and Internal Audit to test and enforce developers' compliance with these policies - not MRM. However, even in these situations, there are important nuances of these policies and their enforcement that may still permit developers to leverage many open-source tools.
Overall, an FI's Board and Executive Management cannot take solace that their existing governance structures - including Model Risk Management - are sufficient to mitigate the pitfalls described above.
So What Can FIs Do To Mitigate These Pitfalls?
To maximize the returns on its AI transformation investments, FIs should be proactive in designing, growing, and managing their algorithmic infrastructures. This should be done with input from key stakeholders to ensure an appropriate balance between the standards, policies, and controls necessary to achieve risk, financial, and compliance objectives while - at the same time - providing developers with the tools and environments necessary to experiment, develop, and deploy effective AI-driven solutions to prioritized use cases.
In larger, more decentralized FIs, the design of such a governance and control framework can be set at the overall enterprise-wide level. However, individual business units could be provided with responsibility and accountability for the implementation of this framework in the manner that is most effective for the specifics of their units. This ensures that overall enterprise-wide objectives apply equally across the decentralized infrastructures, yet gives appropriate implementation flexibility to prevent the suppression of developer ingenuity, experimentation, and development/adoption of high-value AI-driven solutions.
* * *
ENDNOTES:
[1] This is a complex legal area. Consult appropriate legal counsel for specific legal advice.
[2] For example, many new algorithmic methods are introduced via pre-print articles. However, such articles are not the same as peer-reviewed journal articles that have gone through rounds of academic review and challenge. If MRM deems the information in a pre-print article to be sufficient for establishing the algorithm's theoretical and technical soundness, then residual risks still remain.
© Pace Analytics Consulting LLC, 2024.
Comments