Unreadable Rules: Hidden Risk of GenAI

Trusted Outcomes

We’ve witnessed extraordinary breakthroughs with generative AI. From customer service voice agents that speak with stunning fluency, to supply chain optimizations that crunch massive datasets in seconds, GenAI has become a core driver of innovation. But as these applications scale across industries, a new and underappreciated risk has emerged: the rules guiding AI systems are becoming unreadable—not just to developers, but to the very functional experts who should be governing their use. We touched briefly on this ‘real time risk’ in our last post.

In legacy systems, business rules were buried deep in the source code. This was far from ideal, but at least there was a clear structure: developers worked in tandem with functional experts to review, test, and validate that the software met business requirements. The shift to GenAI hasn’t eliminated that need—it’s simply changed the location of the rules.

Today, ai business logic often resides in the prompts and tool definitions that guide large language models. These prompts can span thousands of lines, combining semi-structured instruction, conditionals, edge-case clauses, and informal tone. The instructions may be buried in text files, fine-tuned systems, or opaque orchestration layers, and connected to tool sets which the model uses to complete complex transactions. What’s more, these prompts are not easy to decipher in terms of correlation between instruction sets and expected model behavior. QA teams struggle to predict failure rates in a traditional testing process.

The result? We’ve traded one black box for another.

The danger isn’t theoretical. In a landmark paper, [The Fallacy of AI Functionality] ( https://arxiv.org/abs/2206.09511) (Raji et al., 2022), the authors documented how AI systems often don’t work—and when they fail, they do so at real human cost. Michigan’s MIDAS system falsely flagged over 20,000 unemployment cases as fraudulent. RealPage’s tenant screening tool pushed families into homelessness based on inaccurate data. These weren’t failures of ethics, they were failures of basic functionality. And too often, no one could pinpoint which hidden rule or prompt drove the AI’s behavior.

Here’s the crux: functional experts have lost visibility into the logic. In the old world, rules were code. In the new world, they are instructions consumed by large language models trained on their own set of data. The instructions are often unstructured text blended into paragraphs, conditionals, system messages, or tool schemas. And unlike code, these rules don’t have breakpoints, log files, or static analyzers. They're effectively invisible.

As we reflected on this challenge, we recalled the classic HBR article Staple Yourself to an Order (Shapiro, 1992), which emphasized that each step in an operational process affects the customer. Every time an order is handled, the customer is handled. Every time an order sits unattended, the customer sits unattended. In today’s AI-infused workflows, a similar principle applies: “Staple yourself to the prompt.” If you don’t understand how the model reaches its output, you can’t assure it aligns with business intent.

So why are the rules unreadable?

Because they’re no longer written in structured logic. They’re composed as narratives, injected into prompt chains, tool handlers, and orchestration logic and consumed by opaque langauge models. It’s hard to isolate which instruction triggered which response. It's harder still to test exhaustively. And unlike traditional code, prompts evolve rapidly - modified by prompt engineers, model updates, or auto-rewriting tools.

This isn’t just a design challenge — it’s a governance imperative which demands executive attention. And given the awesome capabilities of ai models, it is a challenge that is worth solving.

To mitigate the risk, we need new visibility tools. Prompt documentation standards. Functional test suites that treat prompts as first-class artifacts. Change management processes that include prompt reviews by business stakeholders. And, critically, a cultural shift: prompts are business rules. They should be governed as such. This is an area that we are aggressively working on for our clients and intend to collaborate with the broader community to address effectively.

The era of GenAI is here. But if we can’t read the rules, we can’t trust the outcomes.