From 20,000 Validated Cases to an AI-Assisted Validation Engine

Executive snapshot

1. Executive Summary

In validation, a performer receives a degree certificate and a passport and enters 15 to 20 fields into the internal case system. What goes in each field is not simply what is printed on the document. It is what the client’s specification says to put there — and that specification is more than 100 rules long, conditional, and revised continuously. Every case arrives from a different university, in a different country and language, on a differently laid out document.

Before I started, those rules had never been written down as rules. They existed as know-how, and as documents spread across a site, per-field PDFs and update mails. Performers could miss an update, spend a long time looking for the right instruction, and still apply the wrong one, because a general instruction such as “enter the name as printed” is wrong on a meaningful share of cases.

I treated the missing rules, not the typing, as the problem. I took about 20,000 previously validated and correct cases as the evidence base, grouped them by the scenario each one represented, and worked backwards from the correct answers to the decisions behind them. That took more than two months and produced a rule book of more than 100 rules written as explicit conditions: a passport’s fields are read from fixed positions in the machine-readable zone; a qualification is handled by a rule written for the country that issued the degree; a name is entered in Arabic and English or in English alone depending on the language of the document; and where a certificate carries several dates, the rule names the one that governs the field.

On that foundation I built three tools in sequence, each answering a limitation observed in the one before it. A Google Sites hub, organised by system field, with one PDF per field. A NotebookLM guide that answers a question and links back to the full rule. And finally a Google AI Studio assistant that takes an uploaded degree and passport and returns the values for the system fields.

The assistant is in daily use by a team of about 20 performers, working through a daily volume of 25 to 30 cases. In a random sample of 100 cases it handled, 95 carried no field error. One of the five is instructive: the assistant chose the wrong qualification date on a certificate carrying several dates, so I audited the case, traced the logic back to the rule book and corrected the rule. The change I care most about is behavioural: a performer no longer retypes the fields, they audit prepared values against the source document.

What I have not done is measure the time this saves, and I make no claim that it does. The agent that would remove the manual step entirely — opening the case, writing the fields, checking sufficiency and routing the case onward — I designed and costed but did not build, because it needed organisational approval, a budget and integration with the case system. Section 4.7 sets out that design.

2. Business Context and Problem Definition

2.1 How the work actually happens

The operation verifies education documents for a government ministry client whose decisions determine who may travel or study abroad. A case arrives with the applicant’s documents. A performer reads the degree certificate and the passport, decides what each of the 15 to 20 system fields should contain, and types those values into the internal system. On some cases more fields are required than on others, depending on the documents presented and on the applicant’s situation.

2.2 The rules are conditions, not answers

The client’s specification is more than 100 rules long, and the important property of it is that it is not a list of answers. It is a list of conditions. A generic instruction is wrong on a meaningful share of cases, which is why the specification is difficult to apply and difficult to hold in one person’s head.

• Passport. The machine-readable zone encodes every field in a fixed position, so the rules specify which position carries which value, with additional handling written for particular countries.
• Qualification. The value entered depends on the country that issued the degree, because degrees issued within a country follow a consistent layout while degrees issued elsewhere do not. The rules are therefore written per country, with edge cases written as their own scenarios.
• Name. The format depends on the language of the document: an Arabic-language applicant is recorded in Arabic and English, an English-only applicant in English alone.
• Dates. A certificate can carry several dates, and the rules specify which of them governs a given field.

2.3 What made it hard

• The rules were never written down as rules. They were tacit know-how. Nobody could look up a rule that had never been stated, and every new performer had to acquire them through experience.
• They were scattered. Guidance lived across a site, a set of per-field PDFs and update mails, and the client issued updates continuously. Updates were missed, and searching for the right instruction consumed time.
• They are conditional. A general instruction cannot be made safe by being general, so guidance had to be restructured before it could be shared, let alone applied.
• The cost of the work sat in transcription. Every value was retyped by hand, so time lost finding a rule was time lost checking the case.

2.4 One constraint shaped the whole design

The company permits Google tooling only, so the solution had to be built with Google Sites, Google Docs, NotebookLM and Google AI Studio. That limit did not reduce the ambition. It shaped the sequence, because each of those products answers a different part of the problem: one stores the rules, one finds them, and one applies them.

3. Data Discovery and Root Cause Analysis

3.1 The evidence base that already existed

This project did not begin with an analysis of operating data, and I do not claim a Six Sigma method, a control plan or a statistical technique. It began with an evidence base that was already in place: a reference sheet of 20,000 cases that had been validated and were known to be correct, each carrying the qualification and certificate details behind it. I used that corpus as 20,000 known-good answers — which is a far stronger starting point than an opinion about where the errors come from.

3.2 How the rules were derived

The method was deliberately unglamorous, and it is the part of this project I would most want to be judged on. I worked backwards from correct outcomes to the decisions behind them:

• Group the cases by scenario — the country of the university, the type and language of the document, the shape of the certificate.
• Ask what decision a person had to make to get the recorded value right.
• Write that decision as a condition — “when the issuing country is X, enter Y”, never “the value is usually Y”.
• Collect the edge cases that break the condition, and write each of them as its own scenario.
• Attach a worked example from the corpus, so the rule can be checked against a case already known to be right.
• Keep the book open. Every scenario met in live work was added as a rule.

Working through the corpus this way took more than two months. It is the largest single investment in the project, and it is the reason the tools themselves could stay small: the difficulty was moved into the rules, where it can be reviewed, argued about and corrected, rather than left inside an instruction that has to guess.

3.3 Root causes

3.4 What I did not establish

Two gaps are worth stating plainly, because they bound the claims in this document. I never measured the time a performer spent searching for a rule or typing a field, so I cannot put a number on the time that was lost or on the time that was saved. And I did not separate rule retrieval from rule application as causes; the two were addressed by the same design, and isolating them would have needed a measurement I did not take.

4. Solution Design and AI-Directed Development

4.1 The rule book — two months before any tool

The first deliverable was not software. It was the rule book: more than 100 rules, each stating a condition, the case it governs, and a worked example taken from the 20,000-case corpus. Writing it exposed the parts of the specification that had never been stated at all, including the cases where a general instruction is not safe.

4.2 Stage 1 — a knowledge base organised by the field you are filling

The first tool was a Google Sites hub. The organising decision was to structure it by system field rather than by subject, and to give every field its own PDF holding the rules and logic that apply to it. That turned a search problem into a lookup problem: a performer filling a field opens that field’s PDF and reads exactly the rules that govern it. It also gave client updates a single landing point.

4.3 Stage 2 — asking instead of searching

Even organised this way, performers were still locating and reading. So I consolidated the whole rule set into a single written document and loaded it into NotebookLM, which answered questions grounded in that document. The detail that made it trustworthy was that every answer came back with a link to the field-specific PDF, so the performer could verify the full rule rather than rely on a short answer alone. The remaining weakness was that the answer still had to be read, understood and applied by hand.

4.4 Stage 3 — the assistant that applies the rules

So I moved the rules into Google AI Studio and built the assistant as a chat: the performer types a question or asks by voice, or uploads the degree certificate and the passport as a PDF or an image, and the assistant returns the values for the system fields, applying the rule set. Figure 4 shows the interaction as it is designed to work, and it shows a partial view of the fields, as the real interface scrolls through them.

4.5 The directed-development loop, on a real example

The assistant was wrong about the qualification date on a certificate that carried several dates. The correction is the part worth describing, because it shows how the system was improved rather than merely built:

The principle underneath this is the one I would carry to any similar problem: a wrong output is treated as a defect in the rules, and fixing a rule once improves every future case that meets the same condition. Correcting the performer would have fixed one case; correcting the rule fixed the class.

4.6 Where the human stays

The assistant prepares; the performer decides. Nothing enters a case without a person comparing the prepared values against the source document. That is not a workaround for a limitation — it is the control that makes the whole approach defensible, because a field that affects an applicant’s outcome is read by a person before it counts.

4.7 The next design — the assistant as the performer

The natural end point of this work is an assistant that performs the whole field-entry task: the interface behaves as if the performer is speaking to a manager who knows the rules, the assistant opens the case, writes the fields, checks whether the documents are sufficient, and submits the case onward. The team’s role changes with it — from producing the fields to auditing the work and maintaining the rules.

I designed this and costed it, and I did not build it. It required organisational approval, a budget and integration with the operational system, and I had none of the three. Figure 6 shows the design as I left it.

5. Project Management and Implementation

I ran this as an operational improvement project with a hard constraint on tooling and a clear rule about what I would and would not build. The phases were sequential because each one was built on the artefact of the previous one, and each was closed by a decision rather than a date.

5.1 The two scope decisions that shaped the outcome

I stopped at the point where the organisation took the work forward. The end-to-end agent needed an operational decision that I could not make alone: it would write directly into the case system and move cases between stages, which makes it an operational change and not only a tool. It also needed a budget and an integration that only the organisation could provide. When the organisation subsequently took a similar initiative forward through its IT team, I stopped developing my own version rather than run two competing efforts inside the same operation, and I kept the assistant live as my team’s support tool — which is where it still is, used and maintained by me.

The tool was optional, and that is the point. I set no adoption target and measured no adoption rate. Performers adopted it because it saved them time and improved the quality of the case — that is their judgement, not a time study I have run. If it had not, the correct outcome would have been that it was not used. I would rather report a tool that people chose than one that was mandated and quietly abandoned.

5.2 Risks, and how they were handled

6. Preventive Improvement

I did not run a formal Six Sigma project and I claim no belt. The preventive thinking, however, is the backbone of the design: every failure mode I could identify was answered with a control that stops it recurring, rather than with more checking or more training.

What is not yet in place, stated honestly: there is no standing evaluation set that runs the rule families against known-correct cases on a schedule, there is no automatic re-run of affected historical cases when a rule changes, and errors are corrected individually rather than through a maintained error taxonomy. Those are the next controls, and they are specified in Section 8 rather than claimed here.

7. Testing and Validation

Testing happened at three levels during the build, and one measurement was taken after it was in use. They deserve different confidence, so I describe them separately.

The measurement is worth being precise about. I drew 100 cases at random from those the assistant had handled, reviewed each one by hand against the source documents, and counted a case as an error if any field in it was wrong. Five cases carried an error, which is 95 per cent correct at the level of the case. It was a manual review, with no automated tracker behind it.

8. Business Impact and Performance Measurement

The measurable result is a correctness result, and the most important result is a change in what the work consists of. Both are set out below, with the evidence status attached to each one.

8.1 The behavioural change, and why I treat it as the real result

Before, the performer’s job was production: read the document, decide the value, type it. Now, for performers who use the assistant, the job is judgement: read what the assistant prepared, check it against the source document, correct it if it is wrong, and let it stand if it is right. That is a different job, and it is a better one to be responsible for — but it only works because a human still reads every value that enters a case. I would rather report that honestly than present a number that suggests the human step was unnecessary.

8.2 Validation plan for the measurements that do not yet exist

The following turns the estimates into measurements. None of these has been performed, and none of the targets below is presented as a result that has already been achieved.

9. Lessons Learned and Future Roadmap

9.1 What the project taught me

• The hardest asset in an operation is often knowledge, not labour. The typing was the visible work; the rules were the actual work, and they were not written down anywhere.
• A rule has to carry its condition. A rule that only shows what the value usually looks like is not applicable — it is wrong whenever the case is not the usual one. Writing the condition is what makes a rule shareable, and it is what made it applicable by a machine.
• Derive rules from cases that are already known to be right. Nineteen thousand correct cases answered questions that no amount of opinion would have answered reliably.
• Fix the rule, not the case. A corrected rule improves every future case that meets its condition; a corrected case improves one.
• Keep the human on judgement, and keep them there deliberately. The audit step is what makes the approach defensible, and removing it would be a change in kind rather than an improvement in speed.
• Let the tool be optional. Adoption that people chose is worth more than adoption that was imposed.

9.2 What comes next

10. Skills Demonstrated

The competencies below are the ones this project actually evidences. They are separated from qualifications, which are recorded elsewhere and are not claimed here.