Quality, Cost, Skill, Accountability, or Scope: What Matters Most to Your Build-or-Buy AI Decision?

How can Internal Auditors develop a sound AI strategy when AI technologies keep changing?

You develop a strategy that provides direction without locking you into today’s technology choices. That means isolating the problems worth solving, anchoring decision-making around your strategic objective, and weighing evergreen considerations around the five key decision dimensions of quality, cost, accountability, skill, and scope.

AI technologies will obviously keep changing. What won’t change are the AI tool definitions, core principles, and decision-making framework that David Coulombe and I included in How to Develop Internal Audit’s Build-or-Buy AI Strategy: A Decision-Making Guide. 

During David’s presentation at APEX, everyone broke into groups based on which decision dimension they saw as most important in their team’s build-or-buy decision. Each made a compelling case:

• The largest group, QUALITY, insisted that Cost shouldn’t be primary, calling investments in autonomous decision-making the wrong place to “cut corners” and suggesting that Skill and Accountability would develop organically with a Quality focus. “But if you don’t have Quality, you can’t have a program.”
• COST — one of the smaller groups — countered that they were focused on ROI, not dollars: “We want to see it prove itself out. So if the Quality is there and there’s a good business case around the investment, we can justify it. It’s not that we’re not willing to pay the cost. We just want to see that the value is high.”
• SKILL emphasized the practical reality that small teams may lack the skills to build OR the resources to buy. Their focus was on how teams will develop and preserve both audit and AI skills: “If we widely implement AI, how do we make sure our junior auditors know what they’re doing when we’re gone?”
• SCOPE suggested that everything comes back to Scope. “We’re all auditors, so we start with the Scope. What do we want to tackle both long and short term? If we don’t know what we’re doing, how can we justify the Cost and show Quality, Accountability, and Skill?” 
• ACCOUNTABILITY also pointed to core audit principles as they explained their choice. “Governance is the thing we always talk about. If you don’t have evidence  of ownership and accountability over the whole process, where does the buck stop?” 

No big surprise here, but there’s really no wrong answer. A robust due diligence approach should include thorough consideration of all five. 

To help your team weigh every dimension of its build-or-buy AI decision, download the full eBook. For a preview, read on for David’s detailed breakdown of each decision dimension, excerpted from “Section Three: AI Build-or-Buy Decision Framework.” 

Weighting the Five Decision Dimensions

The AI decision dimensions form the practical, tangible layer that helps you select “build” or “buy.” The graphic below provides an overview of the five AI strategy decision dimensions. You’ll assign your own weights to the five dimensions based on your stated strategic objective and priorities. 

Five AI Strategy Decision Dimensions

Note that weighting doesn’t mean every dimension is fully tradeable. For work that will be relied upon as audit evidence, your team should establish minimum standards for the Quality and Accountability dimensions before considering cost or efficiency trade-offs. For example, a team whose objective is redeploying capacity may weight Cost and Scope more heavily, while a team whose objective is precision and trust may weight Quality and Accountability more heavily. 

Score whichever use case candidates survived the strategic gate against these weighted dimensions using tangible, measurable criteria — either the vendor selection categories for a “buy” candidate or the build success metrics for a “build” candidate (pages 37-38 in the eBook). This produces a comparable basis for making your decision. As a result, instead of a generic build-or-buy checklist, what follows for each dimension is what should practically count when the candidate is specifically an AI system.

Cost

Definition: The true cost of a solution over its full lifecycle, not just the upfront price.

In an AI context, the Cost dimension looks at the:

• UPFRONT COST OF ACQUIRING the solution: 
  
  • For “buy,” this includes subscription or license fees, implementation and configuration fees, integration work, and any professional services needed to go live. 
  • For “build,” it includes the entirety of the development effort itself: internal staff time, contractors or new engineering hires, platform and tooling licenses, and the time audit subject matter experts spend defining requirements and building test sets. Internal time is easy to leave out because it is already budgeted. But it’s a real cost, and it often comes from the same people who would otherwise be doing audit work.
• ONGOING COST OF OWNING the solution after go-live: 
  
  • For “buy,” this includes renewals and price escalators, usage tiers or overages, and add-on modules that turn out to be required. 
  • For “build,” it includes the team that maintains, retests, and extends the solution. For a complex agentic system, this team is a standing engineering cost, not a one-time project cost. 
  • “Build” and “buy” options should be compared over the same multi-year horizon (e.g., three years), because building front-loads cost while buying spreads it out.
• Usage-based costs that scale with volume, inference, or API calls; tokens consumed; and for any self-hosted models (which behave differently from fixed-software licenses), compute or hosting.
• Cost of continued tuning, such as prompts, configurations, or fine-tuning that need periodic rework as models change, evidence patterns shift, or the audit population evolves.
• Human review time, which doesn’t disappear when AI produces a first pass — rather, it shifts from performing the test to reviewing the AI's conclusion, and it may not shrink until trust in the tool is established.
• Cost of “getting it wrong,” such as rework, missed exceptions, or the cost of a false pass, which a pure licensing or engineering-hours view of cost misses.
• Monitoring and drift-detection costs, including ongoing infrastructure to confirm that the tool still performs as it did at go-live — not the one-time cost paid at implementation.

Quality

Definition: Whether the solution's outputs can meet audit standards.

For assurance work, quality should be treated as a threshold, not simply another trade-off against cost or efficiency. Teams should consider any early signs that quality cannot be demonstrated — or that quality isn’t treated as a primary design objective by the vendor or internal team building the solution — as significant warning signs. The most consequential failure is the false pass (see below). 

In an AI context, the Quality dimension looks at:

• Leniency and false passes: whether the tool ever concludes that a control operated effectively when the evidence shows an exception.
• This is the most serious quality failure in assurance work, because a false pass is invisible. A false fail gets investigated and costs time; a false pass buries a finding inside a clean result that reviewers have little reason to reopen. 
• Models can be lenient in subtle ways (e.g., accepting a plausible explanation at face value, treating a missing approval as implied, rounding a near-miss up to a pass). Accordingly, your team should measure the false-pass rate on its own, using test sets seeded with known exceptions, instead of folding everything into an overall accuracy figure that may look strong while missing exceptions that matter.
• Hallucination and fabrication risk: whether the tool ever invents a finding, cites evidence that does not exist, or asserts a conclusion with unwarranted confidence, a failure mode largely specific to generative AI.
• Handling of insufficient evidence: whether the tool correctly flags insufficient evidence rather than defaulting to a pass when the population or documentation is incomplete.
• Consistency and reproducibility: whether the same inputs produce the same conclusion each time, since non-determinism in model outputs is a real quality risk that traditional deterministic software does not have.
• Performance across messy, real-world evidence (e.g., scans, screenshots, emails, and embedded attachments), not only across clean, structured, or vendor-curated samples.
• Drift over time, because quality measured once at go-live can degrade as the underlying model updates or the use case expands; accordingly, quality requires a monitoring cadence, not a single test.

Accountability (aka Governance Execution)

Definition: After a path has been chosen, how well the AI solution handles ownership of outputs in practice.

This is a distinct question from ownership intent at the strategic gate. Ownership intent decides, once, whether Internal Audit or a vendor should own a given activity at all. Accountability as a decision dimension is scored afterward on whichever path was chosen. It measures how well ownership is actually executed — not who should hold it.

Distribution of accountability differs by path. Internal Audit remains fully accountable for its conclusions in both “build” and “buy” use cases; that cannot be delegated. What differs is the responsibility for the tool that produces the conclusions.

When BUILDING, the buck stops entirely with Internal Audit. Your team is accountable not only for the conclusions, but also for every layer underneath them: whether the model performs, how changes are tested, whether data is secured, and how the system is documented for an External Auditor. There is no one else to escalate to, no contract to enforce, and no independent assurance report to rely on.

When BUYING, accountability for the conclusions stays with Internal Audit, but responsibility for the tool is shared with the vendor. The vendor commits contractually to performance, security, model change management, and remediation of defects, backed by SLAs and independent attestations (e.g., SOC 2 report). Internal Audit’s role shifts to overseeing that the vendor meets those commitments, similar to how it would oversee any other service provider the team relies on.

In this context, the Accountability dimension looks at:

• A defined process for when the tool is found to have been wrong, including who detects it, who fixes it, and how prior conclusions that relied on it are identified and reassessed. In a “buy,” this should be a contractual obligation on the vendor to notify; in a “build,” it falls entirely on Internal Audit.
• Model transparency, which entails knowing which model and version produced a given conclusion, and being notified before it changes, whether through the vendor in a “buy” or the model provider’s enterprise terms for a “build.”
• Data governance specific to AI, such as assurance that audit evidence is not used to train shared models and clarity on where data physically flows. In a “buy,” this is secured contractually with the vendor; in a “build,” it is secured through the model provider’s terms and internal data-handling controls.
• A genuine human-in-the-loop step, proportionate to risk, so that no AI conclusion becomes final without review. This should include a working mechanism that enables a reviewer to challenge or override a conclusion and have that override captured, not lost.
• An audit trail specific to AI actions, including what evidence the tool consulted, what it concluded, and who approved it. The audit trail must be defensible to an External Auditor or a regulator, and not simply a generic system log.
• Registration in a central AI inventory, so that oversight of what AI is in use does not depend on any one team remembering to mention it.

Skill

Definition: The capabilities required to make the approach work, as well as the capabilities that the approach builds within the team.

Skill is the one dimension where the decision is itself an investment. The Internal Auditors who define the profession over the next decade won’t be the auditors who are replaced by AI; rather, they’ll be the auditors who know how to direct it, question it, and stand behind what it produces. So the real risk to an Internal Audit team is not that AI takes its work, but that it will fall behind other teams (and individual auditors) who have learned to work with AI. Decision-makers should therefore weigh not only whether the team has the skills an approach requires, but whether the approach will grow the team’s skills. Any option that delivers efficiency without building team capability foregoes a significant part of any AI solution’s value.

Junior auditors traditionally learn judgment by performing tests themselves. If AI performs those tests, teams must create a deliberate path for developing that judgment in the next generation of reviewers (e.g., having junior staff review AI conclusions vs. evidence).

In an AI context, the Skill decision dimension looks at:

• Capability built within Internal Audit, including whether the approach (a) develops auditors who can direct, configure, and challenge AI as part of their everyday work, or (b) leaves them as passive recipients of AI’s output. “Build” options tend to develop this most deeply, but a well-chosen “buy” can do the same if auditors configure the tool to their own methodology and review its reasoning rather than only its results. A built or bought black-box tool least develops team capability.
• Trustworthiness, as in the ability to tell when a model’s output can be trusted and when it cannot or recognize scenarios prone to hallucination or misapplied judgment — a genuinely new competency for most Internal Auditors.
• Configuration or prompting capability: whether the team can actually tune the tool to its own methodology and re-run it without depending on a vendor or a single internal expert in every instance.
• A shift in the reviewer’s own job, from performing a test to critically evaluating a machine's conclusion — a different skill than performing the test itself.
• Key-person risk for anything built internally: how many people understand the model, prompts, or pipeline well enough to maintain or extend it.
• A learning curve that does not end at go-live, because unlike static software, keeping pace with a fast-moving model landscape is an ongoing skill commitment — not a one-time training cost.

Scope

Definition: What the approach can realistically cover.

In an AI context, the Scope dimension looks at:

• Breadth across control types: whether the same capability extends across business-process controls and ITGCs, design, and operating- effectiveness testing, or it was built or bought for one narrow use case.
• Where it sits on the automation ceiling: whether it only handles the mechanical parts (e.g., data pulls, formatting) or the full judgment chain through to a conclusion, and how realistic it is to expect that ceiling to move.
• Portability to a similar but not identical use case with reasonable effort — a real test of whether this is a reusable capability or a one-off script.
• Coverage of full populations, not only samples, and of the range of evidence formats actually encountered, not just the cleanest cases.

THE LAST WORD: Don’t Waste Time on Unproductive Building or Buying Paths

Your build-or-buy AI strategy is critical — but your time and resources are limited. 

That’s why we created this eBook. 

Internal Auditors need objective guidance to help them see past the hype and oversized promises. What matters to your team? What problems do you want to solve, and how do considerations of quality, cost, accountability, skill, and scope change the equation? 

The eBook shares a four-layer decision-making framework, of which the five decision dimensions are layer three. Each layer serves as a gate and gut check to the next, so that you won’t waste time building or buying AI solutions that won’t get you where you want to go. 

It’s a framework by auditors for auditors, designed to help you stay focused on what actually matters. And it’s complemented by case studies and lessons learned from 16+ Internal Audit leaders real-world build-or-buy experiences. Download the full eBook, start running your use cases through the framework, and get on the path to building or buying the AI solutions that actually solve your problems. 

‍