Arehsoft
You have a prototype. It works — in the demo, on your machine, with the inputs you picked. The question keeping you up is the honest one: would I bet a customer on this?
Here’s a checklist to answer it without kidding yourself. Score one point for each you can answer “yes, and I can prove it” — not “yes, probably.” Be strict. The gap between “probably” and “provably” is where production incidents live.
Inputs and data
1. It rejects inputs it can’t handle. Feed it a malformed file, an empty field, a wrong-language document, a hostile prompt. Does it fail cleanly and say so, or does it produce confident nonsense? A system that knows its limits beats one that guesses past them.
2. It’s been tested on real data, not sample data. Sample data is the data you wish you had. Real data is messy, inconsistent, and full of edge cases you didn’t invent. If it’s only ever seen the clean set, you haven’t tested it — you’ve rehearsed it.
3. Sensitive data is handled deliberately. You know what PII, customer data, or regulated data flows through it, where it goes, and whether it’s being sent to a third-party model provider. If you can’t say this in one sentence, that’s a finding.
Behavior when it’s wrong
4. You’ve defined what “wrong” looks like. There’s an actual definition of an unacceptable output — not a vibe. Without it, you can’t measure quality, and you can’t tell whether a change made things better or worse.
5. There’s a fallback for low confidence. When the model isn’t sure, something specific happens: it escalates to a human, asks for clarification, or declines. “It just answers anyway” is not a fallback.
6. A human can intervene without a code change. For the decisions that matter, a person can review, override, or stop an action through the product — not by paging an engineer at midnight.
Operations and trust
7. Every action is logged and explainable. For any output, you can reconstruct the input, the model version, and the decision path. This is your debugging tool, your audit evidence, and your defense when a customer disputes an outcome. All three.
8. You know the cost per action. Not the monthly bill — the unit cost. Cost per ticket triaged, per document processed, per query answered. Without it, scale is a financial surprise waiting to happen.
9. Latency holds up under load. It’s been run at something like real concurrency, not one call at a time in a quiet meeting. The version that feels instant for one user can crawl for a hundred.
10. It’s monitored on output quality, not just uptime. “The server is up” tells you nothing about whether the AI is still doing its job. You’re tracking quality signals — override rates, rejection rates, drift — and you’d know within hours if it degraded, not weeks.
Ownership
11. It’s integrated with the systems the work actually lives in. It reads from and writes to the real tools — CRM, tickets, database, internal APIs — with proper error handling, not a sample file and a screen. This is usually most of the remaining work.
12. Someone can own it after launch. It’s documented well enough that a team who didn’t build it can run, debug, and extend it. If the only person who understands it is the person who wrote it, you don’t have production software — you have a dependency on one human.
Scoring it
- 10–12: You’re genuinely close. The remaining gaps are worth closing deliberately before launch, but the foundation is real.
- 6–9: You have a strong prototype and a real distance to production. The missing points are exactly the ones that cause incidents. Prioritize them before you scale usage, not after.
- 0–5: You have a promising demo. That’s a good thing to have — just don’t mistake it for a product yet. The good news: none of these are model problems, so none of them require starting over.
Every “no” on this list is a specific, buildable thing — validation, fallbacks, logging, monitoring, integration, docs. That’s the unglamorous half of AI work, and it’s the half that decides whether the demo becomes something your team depends on.
If you scored lower than you’d like and want a second opinion on what to fix first, send us the prototype. You’ll leave with a clear read on feasibility and effort — whether or not you hire us.