Field note, week 3 after launch. Customers asked about claim rejections. The assistant answered confidently — sometimes with the wrong policy clause. Nobody had written down what "accurate" meant for 200 real questions.
Field note, after the layoff. Meera realised the missing piece was hers: an evaluation set. A list of real inputs, the answer a good assistant would give, and the answers it must never give. That's an AI product analyst's core work, and it's acceptance criteria at scale.
Criteria-to-Eval Converter
Paste one old-style acceptance criterion ("the system shall respond accurately"). The converter splits it into testable rows: input, good answer looks like, must not, grade (correct / partly / wrong / unsafe). One vague line becomes five or six concrete test rows you fill in. It's a template generator — no AI call, nothing stored.
Criterion converter
Before and after: the same BA week
| Before (classic BA) | After (AI product analyst) |
|---|---|
| Write user stories with Given/When/Then | Write feature specs with example inputs and graded outputs |
| UAT: does the button work? | Eval: is the answer right, partly right, wrong or unsafe? |
| Sign-off once | Re-test after every prompt or model change |
| Defect log | Failure taxonomy (wrong fact, missed refusal, wrong tone, made-up source) |
| Requirements traceability | Eval results traced to user feedback ("thumbs down") |
| Stakeholder demo | Weekly quality report: what got better, what got worse |
The left column is still useful. The right column is where it's going.
Six cards to flip
Tap a card to flip it.
The proof project: one feature, forty examples
- Pick an AI feature you can use freely — a public chatbot on a bank, telecom or e-commerce site, or a free AI assistant answering questions in your old domain.
- Write a one-page behaviour spec: purpose, users, must-do, must-never-do, tone, when to hand over to a human.
- Write 40 test inputs: 20 normal, 10 tricky (ambiguous, multi-part), 10 that should be refused or escalated.
- Write the good answer (or the good behaviour) for each.
- Run them and grade using four grades. Keep it in a spreadsheet, or log traces in Langfuse.
- Write a one-page finding: top three failure types, examples, and what you'd change in the spec.
Don't put personal data or real customer details into any chatbot.
Tools for this move (checked 24 Oct 2026)
- Google Sheets / Excel — enough for the first eval set.
- Langfuse — open-source (MIT) LLM (large language model) tracing and evaluation, with a free cloud tier (GitHub).
- PostHog — product analytics with a free tier; code is MIT-licensed (PostHog pricing).
- draw.io — for the escalation flow diagram.
What a reviewer looks for in your findings page
Clear grades with examples, not just percentages. One failure you found that the team would not have spotted in classic UAT. A sentence on what you'd re-test after the next prompt change. And honesty about limits: "40 examples is a starting sample, not a full evaluation." That last line shows judgement, which is what this role is hired for.
Keyword reality
"Ai business analyst" gets about 880 searches a month in India and 320 in the US; "ai product manager" about 4,400 and 2,900; "product owner jobs" about 880 and 2,900 (OpenSEO, 24 Oct 2026). The title "AI product analyst" itself is rarely searched, so look for listings titled product analyst, AI product owner, AI quality analyst or conversational AI analyst too.
Product-side AI pay
We haven't verified pay data for AI product analyst or product owner roles. Compare live listings for your city and experience level rather than trusting any figure quoted in a course ad.
Your realistic 90 days
You wrote user stories with acceptance criteria and sat in sprint reviews. AI product analysts do this for AI features, where 'correct' is fuzzy, so they define evals, track usage and spot failure patterns. You'll rewrite your user stories for an AI feature.
Assumes 2–3 hours a day alongside job applications, a laptop and a phone, and free plans only. Every step has a smaller version and a way to pause. Use them. It runs in three blocks of 30 days, each ending with a signal that you're on track. A signal is not a promise of a job or income.
What AI changed in your old tasks
| You used to… | Now you… |
|---|---|
| Wrote acceptance criteria as yes/no | Write criteria as rubrics with thresholds |
| Tracked defects in Jira | Track AI failure categories from real conversations |
| Measured feature adoption | Measure AI feature usage and drop-off with PostHog |
Your first 7 days
| Day | Task | Done when |
|---|---|---|
| 1 | Pick a product you used daily (food delivery, bank app) and imagine an AI feature | Feature idea |
| 2 | Write 5 user stories with rubric-style acceptance criteria | 5 stories |
| 3 | Create a 20-item test set and score a free AI tool against it | Scores |
| 4 | Group failures into 3 categories | Categories |
| 5 | Draft a metrics plan (usage, success, complaints) | Plan |
| 6 | Write a 1-page PRD | PRD |
| 7 | Save 10 product analyst postings | 10 links |
Days 1–30: An AI feature spec with evals
| Step | Time | Done when | Smaller version | If you need to pause |
|---|---|---|---|---|
| Publish PRD, stories, test set and scores | about 10 hours over 3 weeks | PDF + Sheet | PRD only. | Save or commit what works now and stop adding. A finished small version beats an unfinished big one. |
| Share with 5 PMs | about 2 hours | 5 messages | Share with 2 PMs. | Optional. If reaching out feels like too much this week, skip it and put the time into your proof. |
Signal by day 30: A PM asks how you'd measure it.
Days 31–60: Event tracking
| Step | Time | Done when | Smaller version | If you need to pause |
|---|---|---|---|---|
| Add PostHog (free tier) to a tiny demo and show a funnel | about 8 hours | funnel screenshot | Mock funnel. | Stop at the end of the step you're on and write two lines about where you stopped. Pick it up from those notes later. |
| Apply to 8 AI product analyst roles | about 4 hours | 8 logged | Apply to 4 roles. | Stop sending for now. Keep a list of who you've contacted so you can follow up later without doubling up. |
Signal by day 60: A screening call.
Days 61–90: Interviews for AI product analyst roles
| Step | Time | Done when | Smaller version | If you need to pause |
|---|---|---|---|---|
| Practise 2 product cases on AI features | about 3 hours | 2 done | Just one instead of 2. | Optional. Skip it if the week is full; nothing later depends on it. |
| Follow up with PRD | about 2 hours | done | Top 3. | Optional. If reaching out feels like too much this week, skip it and put the time into your proof. |
Signal by day 90: A product case round.
The two worries we hear most about this move
Will companies hire a BA as a product analyst? Some will, especially if you show you can work with data. Your proof project should include a SQL query or a simple analysis of user feedback, not only documents. That answers the doubt before it's asked. I don't know how AI products are measured. Few people do yet. Start with ordinary product metrics (usage, completion, complaints) and add AI-specific ones such as wrong-answer rate and hand-offs to humans. Writing that down clearly is already rare.
Other worries, such as age, a career gap, no degree, English, family pressure and savings, are covered on the route page and the role page.
One search that fits your background
The full set of search strings for this move is on the route page. They work on LinkedIn, Naukri and foundit; Instahyre and Cutshort are worth a weekly look too. Add your old job title to any of them with AND, so recruiters who want both skills find you first.
Job-hunt moves that matter for this route
- Search 'product analyst', 'AI product', 'associate product manager' on LinkedIn and Wellfound.
- Startups value a BA who measures.
- Evals in a PRD stand out.
Your 30/60/90 quests
XP: learning 20 · proof 60 · outreach 40. Ranks: Wayfinder → Builder → Pathmaker. Progress stays in your browser only.
"Rate the chatbot" task scams
Some fake "AI product testing" jobs pay a few small tasks then ask for a deposit to unlock bigger ones. Real employers don't charge you to work. Never share OTPs or UPI PINs with anyone. Use official domains only. Report in India at cybercrime.gov.in or call 1930; in the US at ReportFraud.ftc.gov.
Where this bridge leads
Quick quiz: does this path fit?
Answer each, then press Check. Every answer explains itself.
Make your Path Card
A shareable reminder of where you're heading. The day count starts the first time you open this page; quest XP comes from your log above.
Scam check before you apply anywhere
- Real job and gig platforms are free to join. Never pay a "joining" or "training" fee.
- Never share an OTP or UPI PIN — no genuine employer needs them.
- Use official domains only; type the address yourself.
- India: report at cybercrime.gov.in or call 1930.
- US: report at ReportFraud.ftc.gov.
Small steps, every day.
Each quest has a smaller version for hard weeks, and a pause button. Nothing is lost.
Keep exploring
Sources
Checked 24 Oct 2026. We opened every link below ourselves. Search volumes on this site come from OpenSEO keyword research (Oct 2026).
The other facts here are shared with the route page, where each is listed in full with its link.
| What we checked | Where |
|---|---|
| NIST AI RMF (voluntary; GenAI profile NIST-AI-600-1) | nist.gov |