Four questions to discuss before deciding what to support in an AI pilot.
Fery Yundara Putera · Senior Software Engineer
I build software for internal processes at Solve Education!, including a shared automation foundation.
If a team proposes using artificial intelligence (AI), software that can perform tasks such as drafting text or writing code from instructions, what would help you decide whether to try it? A demonstration can show an answer appearing on screen. It leaves a harder question: would that answer be useful in the team’s actual work?
Solve Education! helps youth turn learning into livelihoods through behavioural science and technology. I work on software for its internal processes. When assessing a proposed tool, the difficulty is deciding what should count as success before time and money go into building it. An output might still need enough checking and correction to outweigh the help it provides.
Using AI in my development work has made me more careful about what I accept as a finished result. These are four questions I would bring to a discussion about an AI pilot: a limited trial used to learn whether an idea is worth continuing.
01 Whose problem will this solve?
Ask the team to describe one task as it happens today. Who does it? Where do they run into difficulty? What would they want to do differently? That gives everyone something concrete to discuss before choosing a tool.
Consider an illustrative example: a programme team receives repeated questions, and a proposed AI tool would draft responses for staff to check. The need might be to reduce time spent writing similar answers. The useful result would be an accurate, helpful response that staff can send after review.
Ask why AI is a suitable option. A clearer information page or a set of reusable answers might also address the problem. A proposal should explain what AI adds and what work people would still need to do.
02 What would count as a useful result?
Before the trial, agree on what the team will examine. In the response-drafting example, I would ask them to check whether answers are accurate, address the question and take less staff effort after corrections are included.
Compare a sample with the current way of working. Agree who will review it, how it will be selected and when you will discuss the findings. Include difficult cases, so the decision does not depend entirely on the easiest examples.
Also ask what evidence would lead you to continue, change the approach or stop. A faster response is useful only if it meets the agreed standard. If the proposal claims better learning or employment outcomes, those need evidence beyond the quality or speed of an AI-generated answer.
03 Who checks the work when AI gets it wrong?
Ask for a named owner of the review process and an explanation of what happens when an output is unsuitable. In our illustrative example, staff would need to check drafts before sending them and have a way to handle questions the tool cannot answer reliably.
In my own coding setup, an AI agent, meaning software carrying out an assigned task with AI, can report that it has finished. My system also runs automated checks, which test whether the work meets requirements defined for the project. If those checks fail, the task stays unfinished. I still review what the checks cover before accepting the work.
That rule is one example of keeping a result open to examination. Ask where the proposed project has a similar check: who examines the output, and who takes responsibility for using it?
04 What will it take to keep this running?
Ask the team to include the work around the tool in its plan. In the drafting example, staff still spend time reviewing answers. Someone also needs to keep the information current, maintain the software and manage tool costs.
A pilot should help reveal how much of that work is needed. Ask who will own it after the trial and what happens if the person who built the tool leaves. If reviewing and correcting the output takes as much effort as the original task, that should influence the decision to continue.
I would request an estimate of ongoing costs alongside the setup budget, with assumptions that can be checked during the trial.
Put the answers in the pilot brief
Before agreeing to a pilot, ask the team to record the problem, evidence they will collect, review owner and ongoing costs in a short brief. Set a date to revisit those answers together. This gives both sides a basis for deciding what to do next.
For projects supporting Solve Education!’s purpose, it matters to distinguish a working tool from evidence of progress in learning, skills people can use to earn an income, or earning opportunities. The pilot brief should make clear which result is being tested and what remains to be demonstrated.