DataPacts

AI Raters and Evaluation Data: What Raters Do, Who Hires Them and How Labs Buy Evaluation Data

Checked

At a glance

AI raters are people paid to score, rank or check the answers of AI models, and their judgments become evaluation data that AI labs use to measure and improve those models.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Six platforms list this work on their own pages as of 2026-11-16, each through a named contracting entity (or none stated), and the pay claims they publish for rating roles include Handshake's "AI Evaluation Specialist" listing at up to USD 17 per hour, an Alignerr evaluator card at USD 15 to 35 per hour and DataAnnotation's general projects from USD 25 to 50 per hour.

Rating work is one route by which data is acquired for AI training, and within AI training data licensing it is the route where the data is made by people on commission and not found.

Labs obtain evaluation data in five ways on record: managed rating services, commissioned benchmarks, marketplaces, participant panels and adversarial evaluation. Prolific's published fee of usually 42.8 % for corporate researchers is the one buyer-side price structure on record, and no source states what a lab pays per rating.

The DataPacts Editorial Team checked each pay claim, filing share and company description below against its primary source, following how we source, date and check every figure.

What is an AI rater?

An AI rater is a contributor who judges model output against instructions or a rubric, usually as an independent contractor engaged through a human data platform rather than by the AI lab itself.

Outlier lists "response evaluation and ranking" among its work types, and Handshake uses the role title "AI Evaluation Specialist" on its AI Fellowship page, both as checked on 2026-11-09.

Contractor status is the common arrangement. In McKinney v. Scale AI (San Francisco Superior Court, CGC-24-620481), the class notice states that contributor payments are reported on Form 1099, not as W-2 wages. The phrase "AI rater" also names consumer apps that score selfies and faces; this page does not cover them.

Job search is a separate question from what the work is. The platform records here describe the work, and job listings, pay across platforms and the trainer-versus-rater difference sit with AI training jobs and platforms compared.

What raters do: the rating tasks platforms list

Platforms describe rating work in five forms: scoring one response, ranking several, grading against a rubric, reviewing an expert's reasoning, and rating voice or image output. The table below sets each task against the platforms whose own pages list it, as of 2026-11-16 (platform pages checked 2026-11-09).

Task What the rater produces Platforms that list it Source
Response evaluation a score or judgment of one model answer DataAnnotation ("general AI response evaluation"); Outlier; Prolific ("AI evaluation tasks") dataannotation.tech; outlier.ai; researcher-help.prolific.com
Ranking an order of several answers to one prompt Outlier ("response evaluation and ranking") outlier.ai
Rubric grading a grade against written criteria Alignerr ("rubric grading") alignerr.com
Expert reasoning review a domain expert's check of a model's reasoning Alignerr ("expert reasoning review"); Handshake AI Fellowship ("expert evaluation") alignerr.com; joinhandshake.com/ai
Voice and image rating ratings of audio or image output Outlier ("voice/image rating (Aether)") outlier.ai
Rating within mixed task pools ratings beside annotation and data collection TELUS Digital AI Community ("annotation, rating, data collection tasks") telusdigital.com

DataAnnotation states support for 43 languages on its general response evaluation work, so the same task type spans many language pools.

Outlier lists voice and image rating under the name Aether beside prompt writing and domain expert tasks, and TELUS Digital describes rating as one task among annotation and data collection, so at several platforms rating is one task type in a wider pool.

The tasks differ by what the rater must know: a response score needs judgment of fluency and accuracy, while rubric grading and expert reasoning review need the subject knowledge the criteria assume.

Adversarial testing, where a person tries to make a model fail, is its own kind of work with its own record, covered under red-teaming data and adversarial testing work. Gray Swan's adversarial evaluations appear in the routes table below only as a route by which labs obtain evaluation data.

When a rating trains a model and when it measures one

The same rating becomes training data when a lab optimizes a model on it and evaluation data when the lab only measures the model with it. The InstructGPT paper (arXiv 2203.02155, March 2022) shows the first case: about 40 contractors, hired on Upwork and through Scale AI, produced ranked comparisons that trained a reward model.

The second case leaves the model unchanged. A lab scores a finished model on a fixed set of tasks to learn what it can do before release.

Anthropic's S-1 of 2026-10-26 states that it suspended higher-risk RL environments after incidents in 2026 in pre-release cybersecurity evaluations, which shows evaluation work being run on models before they ship.

A lab can also commission evaluation without training on it, and the balance of demand is hard to read from outside.

Epoch AI (2026-01-12) reports one startup employee saying the company received 10 to 20 times more RL requests than benchmarking requests, a single interviewee's statement and not a market measure. The data that trains models through ranked comparisons is covered in full under what reinforcement learning from human feedback needs.

Who hires AI raters: platforms, contracting entities and stated pay

Six platforms list rating or evaluation work on their own pages, and each engages raters through a named contracting entity on its own terms. The table below lists them alphabetically, as of 2026-11-16 (platform checks 2026-11-09, re-checks 2026-11-16 where marked), with the pay each platform states for rating or evaluation roles only.

Platform Contracting entity on record Rating or evaluation work it lists Stated pay for that work (claim, check date) Source
Alignerr ALIGNERR LLC ("powered by Labelbox") expert reasoning review, rubric grading, coding challenges, voice and audio data listing card "Italian Workspace Evaluator" USD 15 to 35/hr (2026-11-16); all site listings, not only rating, USD 40 to 120/hr (2026-11-09) alignerr.com
DataAnnotation Surge Labs, Inc. general AI response evaluation; coding, STEM, medical, legal, finance; 43 languages general projects "start at $25-$50 per hour" (FAQ, 2026-11-09, same on 2026-11-16) dataannotation.tech
Handshake AI Fellowship Stryder Corp. (d/b/a Handshake); Handshake AI Solutions, LLC expert evaluation, software engineering, domain tasks; US-based fellows "AI Evaluation Specialist" up to USD 17/hr (2026-11-09, same on 2026-11-16); page meta "Earn up to $100/hr" joinhandshake.com/ai
Outlier Smart Ecosystem, Inc. (dba Outlier AI) with Scale AI, Inc. response evaluation and ranking; voice and image rating (Aether); prompt writing; domain expert tasks no rate published; pays "either hourly or per task", weekly (2026-11-09) outlier.ai
Prolific entity not stated on its pages AI evaluation tasks, human feedback, preference data, studies researchers set pay: minimum GBP 6 / USD 8 per hour, recommended GBP 9 / USD 12 (help page dated 2026-07-17) prolific.com
TELUS Digital AI Community TELUS International (Cda) Inc. annotation, rating, data collection tasks not published on the pages read (2026-11-09) telusdigital.com

Four terms in the table carry a fixed meaning, listed here.

  • "States" means the platform's own wording on the check date, quoted where quotation marks appear.
  • "Claim" means pay as published, never what a contributor earned.
  • "Entity not stated" means no platform page names the contracting company.
  • Mindrift and Appen's CrowdGen are not rows, because no record shows rating work on their pages.

Handshake's two figures stay side by side because they sit on the same platform: as checked on 2026-11-09, the "AI Evaluation Specialist" listing reads up to USD 17 per hour and the page description reads "Earn up to $100/hr".

Rates for every field and platform, including coding and medical work, are collected under pay for AI training work. Eligibility differs by platform: Outlier lists 117 allowed working locations (Flexible Working Guidelines effective 2026-07-15), and Handshake states that fellows are US-based.

The labs behind the platforms: what filings and company pages show

Rating platforms rarely name their clients, so the clearest record of who the work is done for comes from securities filings and the evaluation companies' own customer statements.

TELUS Digital's 10-K for fiscal 2024 states that Google accounted for 14.3 % of revenue, against 13.1 % in 2023 and 10.9 % in 2022, and describes the TELUS AI Community as one million contributors in more than 118 countries, adding that TELUS accounted for 24.7 % of revenue and that the company had more than 650 clients.

TELUS Digital states these as its own figures, and TELUS Digital has been wholly owned by TELUS Corporation since 2025-10-31.

Innodata's 10-Q for the quarter ended 2026-09-30 states that its two largest customers were approximately 43 % and 31 % of revenue, and Innodata states it is engaged by five of the "Magnificent Seven".

Gray Swan states that it has more than 20 customers and names Anthropic, OpenAI and Meta as users (Series A announcement, 2026-05-28). Anthropic describes Andon Labs as "an AI safety evaluation company" in its Project Vend research post.

These shares are of total company revenue. No filing splits rating revenue from other services, so none of them shows what any lab pays for rating, and the customer names are each company's own statement.

How labs buy evaluation data: five routes on record

Labs obtain evaluation data in five ways on record: managed rating and evaluation services, commissioned benchmarks and evaluation environments, marketplace purchases, participant panels, and adversarial evaluation. The table below groups twelve companies by route, as of 2026-11-16 (checks 2026-11-09 unless marked), and each "states" cell is the company's own description.

Route Company What it states it supplies Source
Managed rating and evaluation Deccan AI post-training data, evaluation and RL environments through an expert network deccan.ai
Managed rating and evaluation iMerit annotation, evaluation and RL services; owned by EXL since 2026-07-31 imerit.ai; sec.gov
Managed rating and evaluation Innodata training and post-training data and evaluation for AI labs and Big Tech innodata.com; sec.gov
Managed rating and evaluation micro1 expert contributors, RL environments and evaluations micro1.ai
Commissioned benchmarks and evaluation environments Andon Labs long-horizon agent benchmarks (Vending-Bench, Blueprint-Bench, Drone-Bench); described by Anthropic as "an AI safety evaluation company" andonlabs.com; anthropic.com
Commissioned benchmarks and evaluation environments Datacurve RL environments, datasets and benchmarks (DeepSWE), agent trajectories, SFT data datacurve.ai
Commissioned benchmarks and evaluation environments Mechanize "environments and evals for frontier coding agents" mechanize.work
Commissioned benchmarks and evaluation environments Proximal post-training data and coding RL environments; FrontierSWE v2 "34 tasks, 13 frontier models, 20-hour budgets" (2026-11-16) proximal.ai
Commissioned benchmarks and evaluation environments Sepal AI training data, evaluation benchmarks and RL environments; acquired by Mercor.io Corporation (announcement 2026-02-06, price undisclosed) orrick.com
Commissioned benchmarks and evaluation environments Taste Labs post-training data and evaluation environments for design from an expert-creative network tastelabs.com
Marketplace DataVendor (HUD) marketplace "where AI labs buy training and evaluation data"; 20 % fee on vendor earnings at withdrawal hud.ai; docs.datavendor.ai
Adversarial evaluation Gray Swan AI adversarial evaluations for frontier labs; Arena with more than 15,000 participants and more than 1,000,000 attack trajectories (2026-05-28) grayswan.ai

Benchmark builders often also build the training environments, which is why the same companies appear on both sides; the wider field is covered under RL environments, often built by the same benchmark makers.

The companies in the first two routes sell finished services or benchmarks to labs, the marketplace route lists data for labs to buy on DataVendor's terms, and the adversarial route supplies attack data from a large participant pool.

The fifth route, participant panels, is the Prolific row of the platform table above: researchers set what participants are paid, and Prolific states a platform fee of usually 42.8 % for corporate researchers and 33.3 % for academic or non-profit ones (pricing page, checked 2026-11-09).

DataVendor's 20 % fee falls on the vendor's earnings at withdrawal, the opposite side of the transaction.

The record has no price for a commissioned benchmark or a rating item, and iMerit, which EXL completed acquiring on 2026-07-31, shows that managed rating suppliers change owners. The Proximal row shows what a benchmark release states about itself, and the Sepal AI row shows that evaluation suppliers are being acquired, with the price undisclosed.

Rating work and evaluation data: benefits, risks, downsides and alternatives

Rating work pays some contributors and evaluation licensing pays some data holders, and both hand over material that is hard to take back.

The audiences differ: contributors take project work on platform terms, while data holders license records that evaluation builders may use. Regulation (EU) 2025/327 (EHDS), Article 53(1), names "training, testing and evaluation of algorithms" among permitted secondary-use purposes, with Chapter IV applying from 2029-03-26.

Benefits

  • Paid project work with stated rates: the DataAnnotation and Handshake claims in the platform table are examples.
  • Participant panels where researchers, not the platform, set pay, with a stated floor of GBP 6 / USD 8 per hour on Prolific (help page dated 2026-07-17).
  • Data holders are also paid when Turing builds evaluations on licensed records; Turing states a revenue share and does not publish the split (checked 2026-11-09).

Risks

  • Outlier's terms of use (effective 2026-05-11) state that the contributor irrevocably assigns work product (s.4(b)), that confidentiality survives termination (s.5), and that disputes go to individual arbitration with a class waiver and a 30-day opt-out for US residents (s.10).
  • noyb and three groups filed five GDPR complaints on 2026-10-27 against Scale AI, the company behind Outlier, over trackers, biometric identity checks and unanswered access requests, in France, Italy, the Netherlands and Austria; these are complaints, not findings.
  • Rates differ between a platform's own pages, as Handshake's USD 17 listing and "up to $100/hr" description show.

Downsides

  • Work ends with the project, and no platform publishes hours available to a rater.
  • No platform publishes an applicant pass rate, so the odds of qualifying are not on record.
  • Pay disputes reach court: in McKinney v. Scale AI the gross settlement is USD 12,500,000 as of 2026-11-16, covered with the other contributor lawsuits against human data platforms.

Alternatives

  • Other contributor work, such as expert tasks priced by field, rather than rating.
  • Licensing records instead of labor, where the holder owns the data.
  • Neither rating nor licensing, which is a valid outcome for a person or company.

The choice between licensing and keeping data is its own question, treated under whether to sell or license data to AI companies at all.

What the public record does not show about rating work

The record leaves four things open: what a lab pays per rating, how many raters any lab uses, how many applicants pass, and which vendor rates for which lab today. Filings give customer shares of total revenue, and platform pages give rates for roles, but neither answers these four questions.

  • Price per rating or per evaluation item: no rate card or filing states one.
  • Rater headcount per lab: no lab or platform publishes it.
  • Applicant pass rates: no platform publishes them.
  • Vendor-to-lab pairings: platforms name few clients, and the public customer statements above are each company's own.

Where rating work sits among human data work

This page covers rating work and evaluation data; the pages below cover the wider human data market, expert fields, assessments, preference data and how this site is funded.

Frequently asked questions

How much do AI raters get paid?

Platforms publish different claims for rating roles. As checked on 2026-11-09 and 2026-11-16, Handshake lists "AI Evaluation Specialist" at up to USD 17 per hour, DataAnnotation states that general projects start at USD 25 to 50 per hour, and an Alignerr card read USD 15 to 35 per hour, each a claim with the date shown in the platform table. The pay page linked above holds the full rates.

Can AI raters work from outside the United States?

It depends on the platform. Handshake's AI Fellowship is US-based, Outlier lists 117 allowed working locations (effective 2026-07-15), and TELUS Digital states a community of contributors in more than 118 countries. Language-specific work is collected under AI training work by language.

Are AI raters employees or contractors?

Contractors on platform terms in the records read. In McKinney v. Scale AI the class notice reports contributor payments on Form 1099, not W-2. The legal tests are set out under how AI training contributors are classified.

Is AI rater work legit?

The checks are a named contracting entity, written terms and a stated payment method. The platform table lists the first for five of six platforms, and the legal pages of each platform hold the terms. The wider question is covered under is AI training work legit.