The one published record that splits the two, Meta's Llama 2 paper of 2023-07-19, counts 2,919,326 comparisons, of which 1,418,091 were collected by Meta and 1,501,235 (about 51 %) came from open-source sets.
This page treats human judgments as one kind of data in AI training data licensing. Four companies state on their own pages that they run ranking, preference, feedback or rating work, TELUS Digital reports Google at 14.3 % of its FY2024 revenue, and no company publishes what a lab pays per comparison.
Stated pay for ranking and rating roles runs from USD 8 per hour (a participant panel's minimum, help page of 2026-07-17) to USD 50 per hour for general projects.
What is preference data in AI? Pairwise choices, ranked lists and ratings
Preference data in AI is a set of prompts, each with two or more model outputs and a person's judgment of which output is better, stored so that a model can be trained or checked against it. It takes three forms, compared in the table as of 2026-11-17 with the record that names each.
| Form | What a person records | Where the record names it | Source |
|---|---|---|---|
| Pairwise choice | which of two outputs to one prompt is better | Llama 2 counts its preference data in comparisons (1,418,091 collected by Meta, 2023-07-19) | arxiv.org/abs/2307.09288 |
| Ranked list | an order of several outputs to one prompt | InstructGPT: 4 to 9 outputs ranked per prompt (2022-03); Outlier "response evaluation and ranking" (2026-11-09) | arxiv.org/abs/2203.02155; outlier.ai |
| Rating | a score for one output against instructions or a rubric | TELUS Digital AI Community "rating" tasks; Outlier voice and image rating (Aether) (2026-11-09) | telusdigital.com; outlier.ai |
A ranked list of several outputs holds more than one pairwise preference, which is why published records count comparisons rather than prompts. Ratings serve both training and model checks, so the same score can appear in either use.
What labs use human comparisons for: reward models, direct optimization and model checks
Labs use human comparisons in three ways: to train a reward model that scores outputs, to optimize a model on the comparisons directly, and to check a finished model without training on the judgments.
OpenAI's InstructGPT paper (arXiv 2203.02155, 2022-03) trains a reward model on rankings, and the direct optimization paper (arXiv 2305.18290, submitted 2023-05-29) trains on ranked comparisons without a separate reward model. The full pipeline and the comparison of methods sit under what reinforcement learning from human feedback needs.
The third use is evaluation: the judgments score a model that is already trained, and they are not fed back into it. Rating tasks and the routes by which labs buy evaluation data belong to that side of the work and are not repeated here.
Commissioned or public: where the preference data on record came from
The published record shows labs both commissioning preference data from paid labelers and drawing on public preference sets, and the one record that splits the two puts them at about half each. The table lists three records by date, as of 2026-11-17.
| Record | Date | What it shows about where preference data came from | Source |
|---|---|---|---|
| InstructGPT paper (OpenAI) | 2022-03 | rankings of 4 to 9 outputs per prompt by about 40 contractors hired on Upwork and through ScaleAI | arxiv.org/abs/2203.02155 |
| Llama 2 paper (Meta) | 2023-07-19 | 2,919,326 comparisons: 1,418,091 collected by Meta, 1,501,235 (about 51 %) from open-source sets | arxiv.org/abs/2307.09288 (Table 6) |
| Anthropic, PBC Form S-1 | 2026-10-26 | training data sources include datasets licensed or commissioned from third parties; no share per source | sec.gov (S-1) |
The open-source share is arithmetic on the paper's two figures: 2,919,326 minus 1,418,091 leaves 1,501,235, which is 51.4 % of the total. InstructGPT's rankings came from a hired team, so its record is commissioned work only, and the Anthropic filing below names both routes without a split.
Public sets exist alongside commissioned ones. Anthropic's HH-RLHF paper was submitted on 2022-04-12 (arXiv 2204.05862). Sizes and licenses of public preference sets are not stated here, and the licenses of major AI training datasets and their licenses are covered on the datasets page.
Anthropic's S-1 of 2026-10-26 lists datasets "licensed or commissioned from third parties" among its training data sources, without a share per source.
Companies whose own pages name ranking, preference or feedback work
Four companies state on their own pages or in filings that they run ranking, preference, feedback or rating work for AI developers, and each engages contributors or sells the work through a named entity. The table is alphabetical, as of 2026-11-17, with page checks of 2026-11-09.
- "States" means the company's own wording on the check date.
- Inclusion means the page names ranking, preference data, human feedback, feedback or rating; the order carries no ranking.
- "Not stated" means that no page read gives it.
| Company | Contracting or legal entity on record | What its own pages state (its wording) | Source |
|---|---|---|---|
| Outlier (Scale AI) | Smart Ecosystem, Inc. (dba Outlier AI) with Scale AI, Inc. | "response evaluation and ranking", prompt writing, voice and image rating (Aether), domain expert tasks | outlier.ai |
| Prolific | entity not stated on its pages | "human feedback, preference data, evaluations, SFT data" from participants; researchers set the reward | prolific.com/pricing |
| TaskUs | TaskUs, Inc. (SEC CIK 0001829864) | AI Safety and Alignment; "Data Feedback & Evaluations" | taskus.com/services/ai-services; sec.gov |
| TELUS Digital | TELUS International (Cda) Inc.; wholly owned by TELUS Corporation since 2025-10-31 | AI Community: "annotation, rating, data collection tasks" | telusdigital.com/solutions/data-for-ai-training; sec.gov |
Outlier's and Prolific's wording is the closest to preference data. Prolific's pages name preference data outright, but they do not state a legal entity for the company. By contrast, TaskUs and TELUS Digital name feedback and rating inside broader service lists.
Platforms that describe evaluation or grading work without naming ranking, such as DataAnnotation, Alignerr and the Handshake AI Fellowship, appear in the pay table below. Their tasks are compared as raters under AI raters and evaluation data.
What filings show about who pays for rating and ranking work
Securities filings show the buyer side only in aggregate: suppliers of rating and feedback work report a few large customers, and one names Google.
TELUS Digital's 10-K for FY2024 states Google at 14.3 % of revenue (13.1 % in 2023 and 10.9 % in 2022) and its parent TELUS at 24.7 %. In the same filing TELUS Digital states more than 650 clients and an AI Community of one million contributors across more than 118 countries.
Innodata's 10-Q for the period ending 2026-09-30 states that its two largest customers were about 43 % and 31 % of Q3 2026 revenue, and TaskUs files with the SEC under CIK 0001829864.
These are shares of total revenue, not of preference or ranking work, and the filings do not say that any customer buys preference data.
What ranking and rating work pays, as the platforms state it
No company publishes what a lab pays per comparison, so the figures on record are what platforms state they pay for ranking and rating work and what one participant panel charges requesters on top. The table is alphabetical, as of 2026-11-17, and every value is the company's own claim.
| Company and role | Work named | Pay as stated | Who pays | Stated on | Source |
|---|---|---|---|---|---|
| DataAnnotation (Surge Labs, Inc.), general projects | general AI response evaluation | "start at $25-$50 per hour" | the platform | 2026-11-09 | dataannotation.tech/faq |
| Handshake AI Fellowship (Stryder Corp.), AI Evaluation Specialist | expert evaluation | "Up to $17/hr"; page meta headline "Earn up to $100/hr" | the platform | 2026-11-09 | joinhandshake.com/ai |
| Outlier (Smart Ecosystem, Inc.) | response evaluation and ranking | "either hourly or per task", paid weekly; no rate published | the platform | 2026-11-09 | outlier.ai |
| Prolific, participant studies | human feedback, preference data, evaluations | minimum GBP 6 / USD 8 per hour; recommended GBP 9 / USD 12 | the researcher or AI developer | help page dated 2026-07-17 | researcher-help.prolific.com |
| Prolific, requester fee | platform fee on top of participant pay | usually 42.8 % corporate; 33.3 % academic or non-profit | the researcher or AI developer pays Prolific | 2026-11-09 | prolific.com/pricing |
| TaskUs | data feedback and evaluations | not in the record | n/a | 2026-11-09 | taskus.com |
| TELUS Digital AI Community | annotation, rating, data collection | not published on its pages | n/a | 2026-11-09 | telusdigital.com |
Two rows name evaluation roles because no company publishes a rate for ranking alone. Rates change by platform and field, and the dated figures sit under pay for AI training work by platform.
Ranking work and preference data: benefits, risks, downsides and alternatives
Ranking work pays contributors per hour or task, and the rankings belong to the platform once they are submitted. The lists set out the position of a contributor or a data holder, and every figure is a claim or a record as dated.
Benefits
- Paid project work with low entry requirements at the stated rates (DataAnnotation, Handshake, checked 2026-11-09).
- Requester-set pay with a published floor on a participant panel (Prolific, 2026-07-17).
- For data holders, public or licensed records can substitute for commissioned work, as half of the Llama 2 comparisons did (2023-07-19).
Risks
- Outlier's terms of use (effective 2026-05-11) have the contributor irrevocably assign all rights worldwide in work product (s.4(b)), with individual arbitration and a 30-day opt-out for US residents (s.10).
- noyb and three groups filed five GDPR complaints against Scale AI over Outlier in France, Italy, the Netherlands and Austria on 2026-10-27.
- Classification disputes: McKinney v. Scale AI (San Francisco Superior Court, CGC-24-620481) has a gross settlement of USD 12,500,000 with final approval filed 2026-10-30 (as of 2026-11-09), and contributor payments are reported on Form 1099.
Downsides
- No lab price per comparison is public, so the value of the work is hard to judge.
- Supplier revenue is concentrated in a few customers (TELUS Digital 2024-12-31, Innodata 2026-09-30).
- Human judgments disagree: InstructGPT reports inter-labeler agreement of 72.6 % among training labelers and 77.3 % among held-out labelers (2022-03).
Alternatives
- Other human data work, such as writing demonstrations.
- Licensing company records through a program instead of contributing labor.
- Not ranking and not licensing at all.
The alternatives are weighed in the general guide on whether to sell or license data at all.
What the public record does not show about preference data
The record leaves four things open: what a lab pays per comparison, which supplier ranks for which lab today, how many comparisons labs have bought since the Llama 2 paper, and the size and license of each public preference set.
- No price per comparison or per preference item is stated by a lab or a supplier.
- No page or filing links a supplier to a lab for ranking work.
- No comparison volume is published after the Llama 2 paper of 2023-07-19.
- No size or license is stated here for public preference sets.
Where preference data sits among human feedback, evaluation and demonstrations
This page covers preference and ranking data; the pages below cover the wider human data market, written demonstrations, AI training jobs, contributor lawsuits, synthetic data, published prices and how this site is funded.
- Human data for AI: the companies, their platforms and their court record
- Expert demonstrations and SFT data: the written side of human feedback
- AI training jobs and platforms compared: for contributors
- Contributor lawsuits against human data platforms
- Synthetic data for AI training, including machine-made preferences
- AI training data prices stated by named parties
- How DataPacts is funded
- Red-teaming data and adversarial testing work
This page is information about published records, not a recommendation to take or avoid any work.
Frequently asked questions
What is a preference model?
A preference model is a model trained on human comparisons to score outputs, called a reward model in the InstructGPT pipeline (2022-03). It learns from rankings people made of several outputs per prompt. The pipeline is described on the RLHF page linked above.
Does DPO need human preference data?
Yes, direct preference optimization still trains on ranked comparisons, but without a separate reward model (arXiv 2305.18290, submitted 2023-05-29). People still make the comparisons, or a lab reuses an existing set. The RLHF page above compares the methods.
Can a company buy a public preference dataset instead of commissioning one?
Labs use public sets alongside commissioned ones: Llama 2 drew about half of its comparisons from open-source sets (2023-07-19). Licenses vary by set and are covered on the datasets page above, and sizes and licenses are not stated here.
Who owns the rankings a contributor makes?
On Outlier, the contributor irrevocably assigns all rights worldwide in work product to the platform (terms effective 2026-05-11, s.4(b)). Prolific's pages read do not state license terms. Each platform's own terms govern its contributors.