1,000 searches in the US, 720 in India — and very few people who do it well

• Do you enjoy writing tests? → Yes: keep going. No: try data products.
• Are you comfortable judging whether an answer is correct and explaining why? → Yes: keep going. No: start with code review practice.
• Have you called an LLM (large language model) API (application programming interface)? → Yes: go straight to the proof project. No: spend week one on that first.
• Do you like writing clear reports for non-specialists? → Yes: this route fits well. Not yet: it's learnable; the proof will help.

• Building golden datasets: real-looking inputs with hand-checked expected outputs.
• Choosing metrics: correctness, faithfulness to sources, relevance, refusal when appropriate, format.
• Running LLM-as-judge scoring — and checking the judge with human spot-checks.
• Wiring regression checks into CI so a prompt or model change can't silently break things.
• Writing reports that explain what changed, what got better and what got worse.

• DeepEval — open-source Python framework with pytest-style tests for LLM output.
• promptfoo — open-source CLI for comparing prompts and models across test cases; check its licence and current features on GitHub.
• Ragas — open-source metrics focused on retrieval-augmented generation.

All are free to start. We don't endorse a specific tool.

Grade the Grader

The page shows a question, an AI answer, and an "AI judge" score of 9/10. Your job: grade the grader. Is the score fair? Five rounds — in two of them the judge is confidently wrong (one rewards a fluent but incorrect answer; one punishes a correct but short one). At the end you see how often you agreed with the judge, and a note on why humans spot-check. Fixed examples; nothing leaves your device.

• Called an LLM API from Python
• Wrote 20 golden examples
• Ran one eval framework
• Spot-checked a judge
• Added an eval to CI
• Wrote a one-page eval report

Six ticks means you can talk through a real eval with confidence.

Build or borrow a small question-answering bot over public documents (for example, a government FAQ). Write 30–50 golden questions, including some that should be refused because the answer isn't in the documents. Measure correctness and faithfulness with an eval framework, spot-check 10 judge scores by hand, then change one thing — the prompt, chunk size or model — and show the before/after table. Add the eval to GitHub Actions so it runs on every push.

Monday: a product team changes the system prompt; you rerun the suite and find three new failures in refund questions. Tuesday: you add eight golden examples for a new feature. Wednesday: you spot-check the judge and discover it rewards long answers, so you adjust the rubric. Thursday: you write a short report. Friday: you tidy the dataset and its notes.

• Only testing easy questions.
• Trusting judge scores without spot-checks.
• Reporting a single average with no examples.
• Using private or customer data without permission.

Cover the happy path, edge cases, questions that should be refused, and tricky phrasings. Keep each expected answer short and checkable. Note where an answer came from. Version the dataset in git so changes are traceable. Fifty good examples beat five hundred careless ones.

Lead with the decision: "Change B is safer to ship." Then a small table of metrics, three example wins, three example failures, and what you'd test next. Avoid jargon; product managers and leaders are often the readers.

AI code trainer uses similar judgement skills on code. AI agent engineer builds the systems you'd evaluate.

• Faithfulness: does the answer stick to the provided sources, or make things up?
• Answer relevance: does it answer the question asked?
• Context recall: did retrieval find the passages needed?
• Refusal rate: does it decline when it should, and only then?
• Regression: a case that used to pass and now fails.

Learn these five and you can follow most evaluation conversations.

India: AI product start-ups, IT services firms building GenAI pilots for clients, and quality teams inside larger companies. Titles vary: "AI QA", "GenAI test engineer", "evaluation engineer". Global: AI labs and product companies; contract evaluation work exists but varies by region and platform — check terms carefully.

You've written pytest suites, fixtures and CI checks. LLM evaluation engineers write tests for systems whose outputs change every run — scoring answers for accuracy, grounding and safety. Your testing habits transfer directly; the scoring methods are new.

Assumes 2–3 hours a day alongside job applications, a laptop and a phone, and free plans only. Every step has a smaller version and a way to pause. Use them. It runs in three blocks of 30 days, each ending with a signal that you're on track. A signal is not a promise of a job or income.

What AI changed in your old tasks

Your first 7 days

Days 1–30: An eval harness with a finding

Signal by day 30: You can explain a case where the metric and your judgement disagreed.

Days 31–60: Variance and safety

Signal by day 60: A screening call.

Days 61–90: Interview depth for LLM eval engineer roles

Signal by day 90: A technical round.

Job-hunt moves that matter for this route

• Search 'LLM evaluation', 'AI quality engineer', 'evals' on LinkedIn and Wellfound.
• Startups building on LLMs need evals. Message their engineering leads with your table.
• Honest variance reporting impresses more than a single good score.

One search that fits your background

The full set of search strings for this move is on the route page. They work on LinkedIn, Naukri and foundit; Instahyre and Cutshort are worth a weekly look too. Add your old job title to any of them with AND, so recruiters who want both skills find you first.

The two worries we hear most about this move

Is evaluation real engineering? Yes. You build test sets, scoring code, regression checks and reports for systems whose outputs vary. It's testing and data work combined, and teams shipping AI features need it. Do I need research experience? No. Product evaluation is practical: does this change make answers better or worse for our users? Clear, repeatable experiments matter more than papers.

Other worries, such as age, a career gap, no degree, English, family pressure and savings, are covered on the route page and the role page.

We haven't verified pay data for this route.

Fake "AI rater" jobs asking for registration fees or your bank login. Real platforms don't charge you to work. Never share OTPs or UPI PINs. Official domains only. Report: cybercrime.gov.in / 1930 (India), ReportFraud.ftc.gov (US).

Do I need a maths degree? No. Clear thinking, testing habits and careful writing matter most. Will automation replace evaluators? Judges get better, but someone still has to decide what "good" means and check the checkers.

Take the quiz · Python developer · LLM evaluation route · Document AI