DataPacts

Preference and Ranking Data for AI

Checked

At a glance

Preference data is the record of which model output people judged better, collected as pairwise choices, ranked lists or ratings, and AI labs obtain it by commissioning ranking work from human data companies or by using public preference sets.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

The one published record that splits the two, Meta's Llama 2 paper of 2023-07-19, counts 2,919,326 comparisons, of which 1,418,091 were collected by Meta and 1,501,235 (about 51 %) came from open-source sets.

This page treats human judgments as one kind of data in AI training data licensing. Four companies state on their own pages that they run ranking, preference, feedback or rating work, TELUS Digital reports Google at 14.3 % of its FY2024 revenue, and no company publishes what a lab pays per comparison.

Stated pay for ranking and rating roles runs from USD 8 per hour (a participant panel's minimum, help page of 2026-07-17) to USD 50 per hour for general projects.

What is preference data in AI? Pairwise choices, ranked lists and ratings

Preference data in AI is a set of prompts, each with two or more model outputs and a person's judgment of which output is better, stored so that a model can be trained or checked against it. It takes three forms, compared in the table as of 2026-11-17 with the record that names each.

Form What a person records Where the record names it Source
Pairwise choice which of two outputs to one prompt is better Llama 2 counts its preference data in comparisons (1,418,091 collected by Meta, 2023-07-19) arxiv.org/abs/2307.09288
Ranked list an order of several outputs to one prompt InstructGPT: 4 to 9 outputs ranked per prompt (2022-03); Outlier "response evaluation and ranking" (2026-11-09) arxiv.org/abs/2203.02155; outlier.ai
Rating a score for one output against instructions or a rubric TELUS Digital AI Community "rating" tasks; Outlier voice and image rating (Aether) (2026-11-09) telusdigital.com; outlier.ai

A ranked list of several outputs holds more than one pairwise preference, which is why published records count comparisons rather than prompts. Ratings serve both training and model checks, so the same score can appear in either use.

What labs use human comparisons for: reward models, direct optimization and model checks

Labs use human comparisons in three ways: to train a reward model that scores outputs, to optimize a model on the comparisons directly, and to check a finished model without training on the judgments.

OpenAI's InstructGPT paper (arXiv 2203.02155, 2022-03) trains a reward model on rankings, and the direct optimization paper (arXiv 2305.18290, submitted 2023-05-29) trains on ranked comparisons without a separate reward model. The full pipeline and the comparison of methods sit under what reinforcement learning from human feedback needs.

The third use is evaluation: the judgments score a model that is already trained, and they are not fed back into it. Rating tasks and the routes by which labs buy evaluation data belong to that side of the work and are not repeated here.

Commissioned or public: where the preference data on record came from

The published record shows labs both commissioning preference data from paid labelers and drawing on public preference sets, and the one record that splits the two puts them at about half each. The table lists three records by date, as of 2026-11-17.

Record Date What it shows about where preference data came from Source
InstructGPT paper (OpenAI) 2022-03 rankings of 4 to 9 outputs per prompt by about 40 contractors hired on Upwork and through ScaleAI arxiv.org/abs/2203.02155
Llama 2 paper (Meta) 2023-07-19 2,919,326 comparisons: 1,418,091 collected by Meta, 1,501,235 (about 51 %) from open-source sets arxiv.org/abs/2307.09288 (Table 6)
Anthropic, PBC Form S-1 2026-10-26 training data sources include datasets licensed or commissioned from third parties; no share per source sec.gov (S-1)

The open-source share is arithmetic on the paper's two figures: 2,919,326 minus 1,418,091 leaves 1,501,235, which is 51.4 % of the total. InstructGPT's rankings came from a hired team, so its record is commissioned work only, and the Anthropic filing below names both routes without a split.

Public sets exist alongside commissioned ones. Anthropic's HH-RLHF paper was submitted on 2022-04-12 (arXiv 2204.05862). Sizes and licenses of public preference sets are not stated here, and the licenses of major AI training datasets and their licenses are covered on the datasets page.

Anthropic's S-1 of 2026-10-26 lists datasets "licensed or commissioned from third parties" among its training data sources, without a share per source.

Companies whose own pages name ranking, preference or feedback work

Four companies state on their own pages or in filings that they run ranking, preference, feedback or rating work for AI developers, and each engages contributors or sells the work through a named entity. The table is alphabetical, as of 2026-11-17, with page checks of 2026-11-09.

  • "States" means the company's own wording on the check date.
  • Inclusion means the page names ranking, preference data, human feedback, feedback or rating; the order carries no ranking.
  • "Not stated" means that no page read gives it.
Company Contracting or legal entity on record What its own pages state (its wording) Source
Outlier (Scale AI) Smart Ecosystem, Inc. (dba Outlier AI) with Scale AI, Inc. "response evaluation and ranking", prompt writing, voice and image rating (Aether), domain expert tasks outlier.ai
Prolific entity not stated on its pages "human feedback, preference data, evaluations, SFT data" from participants; researchers set the reward prolific.com/pricing
TaskUs TaskUs, Inc. (SEC CIK 0001829864) AI Safety and Alignment; "Data Feedback & Evaluations" taskus.com/services/ai-services; sec.gov
TELUS Digital TELUS International (Cda) Inc.; wholly owned by TELUS Corporation since 2025-10-31 AI Community: "annotation, rating, data collection tasks" telusdigital.com/solutions/data-for-ai-training; sec.gov

Outlier's and Prolific's wording is the closest to preference data. Prolific's pages name preference data outright, but they do not state a legal entity for the company. By contrast, TaskUs and TELUS Digital name feedback and rating inside broader service lists.

Platforms that describe evaluation or grading work without naming ranking, such as DataAnnotation, Alignerr and the Handshake AI Fellowship, appear in the pay table below. Their tasks are compared as raters under AI raters and evaluation data.

What filings show about who pays for rating and ranking work

Securities filings show the buyer side only in aggregate: suppliers of rating and feedback work report a few large customers, and one names Google.

TELUS Digital's 10-K for FY2024 states Google at 14.3 % of revenue (13.1 % in 2023 and 10.9 % in 2022) and its parent TELUS at 24.7 %. In the same filing TELUS Digital states more than 650 clients and an AI Community of one million contributors across more than 118 countries.

Innodata's 10-Q for the period ending 2026-09-30 states that its two largest customers were about 43 % and 31 % of Q3 2026 revenue, and TaskUs files with the SEC under CIK 0001829864.

These are shares of total revenue, not of preference or ranking work, and the filings do not say that any customer buys preference data.

What ranking and rating work pays, as the platforms state it

No company publishes what a lab pays per comparison, so the figures on record are what platforms state they pay for ranking and rating work and what one participant panel charges requesters on top. The table is alphabetical, as of 2026-11-17, and every value is the company's own claim.

Company and role Work named Pay as stated Who pays Stated on Source
DataAnnotation (Surge Labs, Inc.), general projects general AI response evaluation "start at $25-$50 per hour" the platform 2026-11-09 dataannotation.tech/faq
Handshake AI Fellowship (Stryder Corp.), AI Evaluation Specialist expert evaluation "Up to $17/hr"; page meta headline "Earn up to $100/hr" the platform 2026-11-09 joinhandshake.com/ai
Outlier (Smart Ecosystem, Inc.) response evaluation and ranking "either hourly or per task", paid weekly; no rate published the platform 2026-11-09 outlier.ai
Prolific, participant studies human feedback, preference data, evaluations minimum GBP 6 / USD 8 per hour; recommended GBP 9 / USD 12 the researcher or AI developer help page dated 2026-07-17 researcher-help.prolific.com
Prolific, requester fee platform fee on top of participant pay usually 42.8 % corporate; 33.3 % academic or non-profit the researcher or AI developer pays Prolific 2026-11-09 prolific.com/pricing
TaskUs data feedback and evaluations not in the record n/a 2026-11-09 taskus.com
TELUS Digital AI Community annotation, rating, data collection not published on its pages n/a 2026-11-09 telusdigital.com

Two rows name evaluation roles because no company publishes a rate for ranking alone. Rates change by platform and field, and the dated figures sit under pay for AI training work by platform.

Ranking work and preference data: benefits, risks, downsides and alternatives

Ranking work pays contributors per hour or task, and the rankings belong to the platform once they are submitted. The lists set out the position of a contributor or a data holder, and every figure is a claim or a record as dated.

Benefits

  • Paid project work with low entry requirements at the stated rates (DataAnnotation, Handshake, checked 2026-11-09).
  • Requester-set pay with a published floor on a participant panel (Prolific, 2026-07-17).
  • For data holders, public or licensed records can substitute for commissioned work, as half of the Llama 2 comparisons did (2023-07-19).

Risks

  • Outlier's terms of use (effective 2026-05-11) have the contributor irrevocably assign all rights worldwide in work product (s.4(b)), with individual arbitration and a 30-day opt-out for US residents (s.10).
  • noyb and three groups filed five GDPR complaints against Scale AI over Outlier in France, Italy, the Netherlands and Austria on 2026-10-27.
  • Classification disputes: McKinney v. Scale AI (San Francisco Superior Court, CGC-24-620481) has a gross settlement of USD 12,500,000 with final approval filed 2026-10-30 (as of 2026-11-09), and contributor payments are reported on Form 1099.

Downsides

  • No lab price per comparison is public, so the value of the work is hard to judge.
  • Supplier revenue is concentrated in a few customers (TELUS Digital 2024-12-31, Innodata 2026-09-30).
  • Human judgments disagree: InstructGPT reports inter-labeler agreement of 72.6 % among training labelers and 77.3 % among held-out labelers (2022-03).

Alternatives

  • Other human data work, such as writing demonstrations.
  • Licensing company records through a program instead of contributing labor.
  • Not ranking and not licensing at all.

The alternatives are weighed in the general guide on whether to sell or license data at all.

What the public record does not show about preference data

The record leaves four things open: what a lab pays per comparison, which supplier ranks for which lab today, how many comparisons labs have bought since the Llama 2 paper, and the size and license of each public preference set.

  • No price per comparison or per preference item is stated by a lab or a supplier.
  • No page or filing links a supplier to a lab for ranking work.
  • No comparison volume is published after the Llama 2 paper of 2023-07-19.
  • No size or license is stated here for public preference sets.

Where preference data sits among human feedback, evaluation and demonstrations

This page covers preference and ranking data; the pages below cover the wider human data market, written demonstrations, AI training jobs, contributor lawsuits, synthetic data, published prices and how this site is funded.

This page is information about published records, not a recommendation to take or avoid any work.

Frequently asked questions

What is a preference model?

A preference model is a model trained on human comparisons to score outputs, called a reward model in the InstructGPT pipeline (2022-03). It learns from rankings people made of several outputs per prompt. The pipeline is described on the RLHF page linked above.

Does DPO need human preference data?

Yes, direct preference optimization still trains on ranked comparisons, but without a separate reward model (arXiv 2305.18290, submitted 2023-05-29). People still make the comparisons, or a lab reuses an existing set. The RLHF page above compares the methods.

Can a company buy a public preference dataset instead of commissioning one?

Labs use public sets alongside commissioned ones: Llama 2 drew about half of its comparisons from open-source sets (2023-07-19). Licenses vary by set and are covered on the datasets page above, and sizes and licenses are not stated here.

Who owns the rankings a contributor makes?

On Outlier, the contributor irrevocably assigns all rights worldwide in work product to the platform (terms effective 2026-05-11, s.4(b)). Prolific's pages read do not state license terms. Each platform's own terms govern its contributors.