In my previous post on the Five Levels of AI Automation, I introduced benchmark numbers for the minimum quality AI needs to achieve at each level — from simple assistants to fully autonomous systems. These “quality bars” represent how reliable AI must be to outperform manual execution.
But where do those numbers come from?
This post explains the method behind those thresholds. We’ll explore the logic, walk through the time-based model used to derive them, and look at how complexity and automation level affect the acceptable margin for error. Understanding this helps leaders and engineers make smart, measurable decisions about where to deploy AI — and where to hold back.
What’s a Quality Bar and Why It Matters
A quality bar is the minimum probability (expressed as a percentage) that an AI system produces an acceptable result in a single elementary operation. Since tasks vary in complexity, we standardize measurement at this smallest unit — then scale it up to estimate how well the AI performs across an entire task composed of multiple steps.
This allows us to compare different AI systems and tasks based on a common framework, and to define typical quality bar ranges for each level of automation. Let’s dive into the model that makes this possible.
The Math Behind Trusting AI
To assess whether automation is worthwhile, we compare the time it takes a human to perform a task manually with the expected time it would take an AI system to complete the same task, factoring in preparation, evaluation, and retries.
We assume that a task consists of N elementary operations, each with its own independent chance of success. If an operation has a probability Po of succeeding, the probability that the entire task succeeds on one attempt is:
- Pt = Poⁿ
The expected number of attempts required for success, based on Bernoulli trials, is:
- A = 1 / Pt
Now we calculate the time required to complete a task using AI. There are several time components to consider:
- Tp: Time to prepare the AI request
- Tx: Time for the AI to generate the result
- Te: Time to evaluate whether the result is acceptable
- Tc: Time to correct or restart if the result is inadequate
The first attempt includes preparation, execution, and evaluation:
- T₁ = Tp + Tx + Te
Each subsequent attempt adds execution, evaluation, and correction time:
- Tₛ = Tx + Te + Tc
The total expected AI time is:
- Tai = T₁ + (A — 1) × Tₛ
We then compare this to the manual time for completing the same task, Tm. If:
- Tai < Tm,
then using AI is more efficient. To ensure this inequality holds, Po (the AI’s per-operation success rate) must exceed a certain threshold — this is your quality bar.
How Automation Level Changes the Game
As we move up the automation spectrum, AI takes on more complex responsibilities. Each higher level of automation involves tasks with more elementary operations — often decisions, context handling, or compound logic. These tasks are no longer trivial. The AI must reason across steps, manage its own output structure, and maintain relevance across components.
At the same time, preparation time for the user tends to decrease, since humans are giving up control and letting AI operate more independently. However, evaluation and correction time often increase because the outputs are more detailed and nuanced. Validating the correctness of an entire AI-generated proposal takes longer than approving a one-line suggestion.
And with more complexity comes more risk: failures become more costly, retries more expensive, and the required quality bar higher.
Level 1 — AI as Assistant (50–60%)
At this level, the human remains fully in charge of the process and delegates isolated, elementary operations to the AI. These operations are simple and easy to verify, such as rewriting a sentence, formatting content, or running a one-step transformation. The human drives the overall workflow and makes all decisions.
- Task Complexity: 1 operation
- Manual Time (Tm): ~10–15 minutes
- Preparation Time (Tp): ≈ 5 min
- Execution Time (Tx): ≈ 0.5 min
- Evaluation Time (Te): ≈ 2–3 min
- Correction Time (Tc): ≈ 1–2 min
Because preparation and correction are fast and the impact of failure is low, a 50–60% success rate at the operation level can still yield time savings.
Level 2 — AI with Approval (70–80%)
At this level, the AI takes over execution of the task. It decides how to approach the problem, generates the necessary context, and produces the output. The human does not guide it step-by-step but instead reviews the final result and either approves it or provides corrections for improvement.
- Task Complexity: 3 operations
- Manual Time: ~15–20 minutes
- Preparation Time (Tp): ≈ 3–4 min
- Execution Time (Tx): ≈ 1 min
- Evaluation Time (Te): ≈ 3–4 min
- Correction Time (Tc): ≈ 2–3 min
Because multiple dependent steps must succeed, the required quality bar increases to around 70–80%.
Level 3 — AI with Supervision (85–95%)
AI is now responsible for a full, structured workflow of significant complexity. It performs a wide range of actions and decisions without step-by-step guidance. The human oversees the process, checking progress periodically, correcting errors when necessary, or validating results after the fact. This is suitable for workflows like automated quality assurance, content transformation pipelines, or multi-step document generation.
- Task Complexity: 10 operations
- Manual Time: ~20–25 minutes
- Preparation Time (Tp): ≈ 2–3 min
- Execution Time (Tx): ≈ 2–3 min
- Evaluation Time (Te): ≈ 3–5 min
- Correction Time (Tc): ≈ 3–4 min
Because the process has more moving parts, and correction becomes more expensive, the quality bar rises sharply — typically in the 85–95% range.
Level 4 — Fully Autonomous AI (99–100%)
At the highest level, AI operates independently with no human intervention during execution. Humans may check the result after delivery or only become involved if something goes wrong. In such cases, two distinct operational modes exist.
- Escalation-Based: If something goes wrong, the AI escalates to a human. But since the person isn’t behind the screen, response time could be 30 minutes, an hour, or more. The cost of a failure becomes time delay, missed deadlines, or customer impact.
- Error-Tolerant: If the AI must run without escalation, then the system must be nearly perfect. The accepted failure rate may be less than 1%, depending on the domain.
In both cases, the quality bar must exceed 99%, as the consequences of error are too high for anything lower.
Define Your Own Quality Bar
While these ranges offer a helpful reference, your own environment may demand higher or lower thresholds. Each use case brings unique risks, expectations, and retry costs. The good news? With this model, you don’t need to guess.
Use the formulas, define your task parameters, and calculate the quality bar that makes sense for you.
Subscribe to our blog for more deep dives on AI automation, performance thresholds, and building intelligent systems that scale.
