Defining Correctness: The Key to Quality AI Systems

Most of us struggle to define what good quality work looks like for our AI systems, and this is a significant issue. I'm not just talking about corporate AI systems; this challenge affects all of us. The ability to define what "good" looks like in a prompt is crucial. It turns out that defining quality is one of the most powerful insights in AI. This clarity cuts through the vagueness that often plagues both our business and personal lives.

As humans, we typically prioritize social cohesion over accuracy, which has worked for a long time. However, it doesn't work when dealing with AI systems. This isn't just for developers or data scientists; everyone needs to understand what quality means if we want to be effective in our interactions with AI. So, let's dig into why defining correctness is critical.

The Importance of Correctness in AI Projects

Correctness is fundamental to everything in AI. Most AI projects fail not because the model is incapable but because no one can answer the straightforward question: What does "correct" even mean in this context? If you can't define correctness, you can't measure it. If you can't measure it, improvement becomes impossible. Everything from how we set up our AI architecture to how we choose models relies on a clearly defined standard of correctness.

It's essential to acknowledge that our definitions of quality often change mid-project. I've seen countless instances where product priorities shift during a quarter, disrupting the entire process. It's not about reaching a point where we can freeze our definitions of correctness and never change them; it's about recognizing the importance of having a solid foundation of what good looks like.

Building AI Systems with Quality at the Core

When we think about normal software, correctness seems straightforward: does it pass the tests or not? But AI is different. Correctness isn't binary; it's a complex mix of requirements we often fail to discuss upfront. These can include truthfulness, completeness, tone, compliance with policies, cost, and speed.

When I'm asked about the architecture for an AI system, I always rewind to the beginning: What is the expected output? How do we know what good looks like? This is the first-order decision that shapes everything else.

OpenAI emphasizes that evaluations should measure outputs against specified quality criteria. Without a clear definition of what to measure, we risk building elaborate systems on a foundation of vagueness.

Common Pitfalls: Measurement and Human Behavior

One of the biggest challenges we face is that measurement can distort behavior. Goodhart's Law tells us that when a measure becomes a target, it loses its value. In the context of AI, this means if you select a proxy metric for correctness, the system will optimize for that metric, even if it strays from what you genuinely need.

"When a measure becomes a target, it ceases to be a good measure. AI systems will optimize for whatever metric you choose, even if it diverges from your true goals."

For example, if you tell an AI model it must always provide an answer, you risk it generating responses even when it's uncertain, leading to issues like hallucinations. The key is not just to pick metrics but to foster a culture of correctness that resists manipulation.

Actionable Steps for Defining Quality

To build effective AI systems, consider the following:

These steps will help you break down the problem of defining correctness into manageable parts.

The Bigger Picture: It's About Us

At the end of the day, understanding correctness is not just a technical issue; it's a human one. AI reflects our own ambiguity about quality. If we don't clearly define what we want, we'll be left with subpar outputs.

I encourage anyone involved with AI to think deeply about what quality means in their context. Whether you're an engineer or someone simply using AI tools, defining what good looks like is essential.