RLHF Data: What Reinforcement Learning From Human Feedback Needs and Who Supplies It

RLHF data is the human-made training data that reinforcement learning from human feedback needs: written demonstrations, ranked comparisons of model outputs, and the prompts both are built on. The published record sizes it: OpenAI's InstructGPT paper (March 2022) used about 13,000 demonstration prompts and 33,000 ranking prompts, and Meta's Llama 2 paper (2023-07-19) reports 1,418,091 preference comparisons collected by Meta.

RLHF is one route by which data is acquired for AI training, and within AI training data licensing it is the route where the data does not exist until a lab pays people to make it. Labs commission it rather than buy it finished, and 17 companies describe supplying this kind of data on their own pages or in filings, listed here alphabetically, without pay figures or ranking, as checked on 2026-11-09.

The method appears here only through the data it consumes, never as a training tutorial: the inputs, the published sourcing record, the suppliers, the contrast with SFT, DPO and RLVR, the contributor terms, and the limits that METR's 2025 reward-hacking measurements make visible.

The DataPacts Editorial Team checked every paper figure, filing statement and company description on this page against its primary source, following how we source, date and check every figure.

What is RLHF? Meaning, full form and role in language models

RLHF stands for reinforcement learning from human feedback, a training method in which people judge a model's outputs and those judgments become the signal the model is optimized against. In large language models (LLMs), RLHF is applied after pretraining to bring answers closer to what human judges prefer, and its best-documented example is OpenAI's InstructGPT paper, arXiv 2203.02155, published in March 2022.

The three stages of RLHF in the InstructGPT paper are listed below, each named by the data it uses.

• Supervised fine-tuning (SFT): the model learns from demonstrations, answers that labelers write to prompts.
• Reward model training: a second model learns to predict which output people prefer, from labelers' rankings of 4 to 9 outputs per prompt.
• Reinforcement learning against the reward model: the model answers a pool of prompts and is optimized toward a higher reward, the stage the paper labels PPO.

The "human feedback" in the name is stored data, not a live conversation. Rankings and demonstrations are collected in batches, checked and kept as datasets that the training stages then consume. Why does that matter for data acquisition? Each stage depends on people being paid to produce data that did not exist before, so the volumes, contracting routes and terms behind RLHF are a supply question as much as a research one. Those volumes, routes and terms are documented in papers, in securities filings and on the suppliers' own pages, and the sections below follow them in that order.

The three inputs an RLHF run consumes

An RLHF run consumes three kinds of human data: demonstrations of good answers, comparisons that rank several model answers to one prompt, and a pool of prompts that the model answers during reinforcement learning. Demonstrations are the same material sold as expert demonstrations and SFT data: a person reads a prompt and writes the answer the model should give. Comparisons need no written answer, because a labeler reads several outputs the model produced for one prompt and puts them in order; the InstructGPT paper reports 4 to 9 outputs ranked per prompt.

The table compares the volume of each input in the two most detailed published RLHF records, InstructGPT (March 2022) and Llama 2 (2023-07-19), as stated in the papers, as of 2026-11-10.

Which input is the largest? In both papers the ranking data outweighs the demonstrations. InstructGPT used about 13,000 training prompts for demonstrations against 33,000 for reward-model rankings and 31,000 for the reinforcement-learning stage, and one ordering of 4 to 9 outputs holds more than one preference between pairs. Llama 2 shows the same shape at a larger scale: 27,540 SFT annotations against 1,418,091 preference comparisons that Meta collected for helpfulness and safety, or 2,919,326 once open-source preference sets are counted (Table 6 of the paper). The prompts for the reinforcement-learning stage need no written or ranked answer, and the Llama 2 paper gives no separate count for them.

The split matters to suppliers and buyers because the two human inputs are different products. Demonstrations need people who can write the right answer; comparisons need people who can judge between answers that already exist. Comparisons collected for purposes other than RLHF, such as model evaluation or direct optimization, are covered as preference and ranking data, and the next section traces where the RLHF comparisons on record came from.

How published RLHF projects sourced their human feedback

The published record shows labs sourcing RLHF feedback from contracted labelers and through data vendors, not from the open web. OpenAI's InstructGPT paper states that the lab "hired a team of about 40 contractors on Upwork and through ScaleAI", one freelance marketplace and one data vendor, and it measured how often those labelers agreed: 72.6 % (±1.5) among the labelers who produced training data and 77.3 % (±1.3) among held-out labelers who did not.

The table lists five records, by date, that show how human feedback for RLHF and its successor methods was sourced, as of 2026-11-10, with the arXiv paper or SEC filing behind each.

What do the later records add? Meta's Llama 2 paper counts its human data in annotations and comparisons rather than in people, so the size of its labeler workforce is not on record. Anthropic's paper submitted on 2022-04-12 (arXiv 2204.05862) is the source of the HH-RLHF name, and the DPO paper submitted on 2023-05-29 showed that the same ranked comparisons can train a model without a separate reward model. The newest record is a filing: Anthropic, PBC's Form S-1 of 2026-10-26 lists its training data sources as public web data, datasets licensed or commissioned from third parties, and synthetic data generated by its own models.

None of the five records says which vendor supplies which lab today, and no primary source describes the labs' in-house feedback teams, so the record shows the routes rather than current contracts. Human feedback is one of five routes described in how AI companies get training data, next to crawling, licensing, user data and synthetic data.

Who supplies RLHF data: companies on record

The table lists 17 companies whose own pages or filings describe supplying human feedback, evaluations, post-training data or expert contributors to AI developers, in alphabetical order. Inclusion depends only on that description, and no company is ranked, rated or ordered by pay or fees.

The four reading rules for the table are listed below.

• States: the company's own description of what it supplies, on its own page or in a filing, checked 2026-11-09.
• Legal name: the contracting entity named on the company's pages or in a filing; "not stated" where no page of the company names it.
• Profile: the company name leads to its profile on this site; TaskUs has no profile.
• No pay column: the table carries no pay, fee, payout or rate column, and contributor rates are covered on the pay page listed at the end of this page.

The table compares the 17 suppliers by legal name, stated supply and source, as of 2026-11-10, with each description checked on 2026-11-09.

Three patterns run through the rows. In three rows, the contributor brand sits under a separate legal entity: Scale AI runs Outlier through Smart Ecosystem, Inc., Labelbox runs Alignerr through ALIGNERR LLC, and Surge Labs, Inc. recruits through DataAnnotation. Five rows (Deccan AI, Invisible Technologies, micro1, Proximal and Turing) also describe RL environments, the checkable-reward work that the method comparison below separates from ranked feedback. Three rows carry a securities-filing record: Innodata files with the SEC, TaskUs has SEC CIK 0001829864, and EXL's 10-Q records its completed acquisition of iMerit on 2026-07-31.

Why RLHF data is commissioned rather than bought off the shelf

RLHF comparisons are judgments of a particular model's own outputs, so a lab usually commissions them for the model it is training instead of licensing a finished dataset. In the InstructGPT setup, the outputs a labeler ranks for each prompt come from the model being trained, so the rankings describe that model's answers and no other.

Is there no public human-feedback data at all? There is: a public human-feedback dataset carries the HH-RLHF name from Anthropic's 2022 paper, and the Llama 2 paper counts open-source preference sets beside the comparisons Meta collected itself. The filing record draws the same line between routes: Anthropic's S-1 of 2026-10-26 names datasets licensed from third parties and datasets commissioned from third parties as two forms of one source.

Commissioning also explains why RLHF data has no list price. A price per ranked comparison or per RLHF task is not part of the published record; the AI training data prices that are on record are stated per work, hour, image, token or record by named parties. For feedback work, the public figures are the hourly or per-task terms that contributor platforms state, which the pay page linked at the end of this page dates platform by platform.

RLHF compared with SFT, DPO and RLVR by the data each consumes

The methods differ less in the people they need than in the data they consume: SFT learns from demonstrations, RLHF and DPO learn from ranked comparisons, and RLVR learns from rewards a program can check. SFT versus RLHF is not a choice between two products, because in the InstructGPT pipeline SFT is the first stage of RLHF and the comparisons come after it.

DPO, from a paper submitted on 2023-05-29 (arXiv 2305.18290), trains on the same ranked comparisons directly, without a separate reward model and without the PPO stage. RLVR, reinforcement learning with verifiable rewards, replaces the human judgment with a program that checks whether an answer is correct, such as a test suite for code or a known result for a math problem.

The table compares four methods by the human data each consumes and where its reward comes from, as of 2026-11-10, with the paper behind each row.

What does the move to DPO and RLVR mean for data demand? DPO keeps the demand for ranked comparisons and drops the reward-model step, so labelers still produce the same kind of data. RLVR moves the human work upstream, from judging answers to writing tasks and the checks that grade them. That task-building work is sold to labs as environments, and the page on RL environments and who buys them covers the companies that build them and the price claims on record.

The people who rank: contributor work behind RLHF data

The ranking and writing behind RLHF data is done by contributors engaged through human data companies, mostly as independent contractors on project terms. Outlier, run by Scale AI through Smart Ecosystem, Inc., lists prompt writing, response evaluation and ranking, voice and image rating, and domain expert tasks among its work types. DataAnnotation, the recruiting brand of Surge Labs, Inc., lists general AI response evaluation plus coding, STEM, medical, legal and finance work in 43 languages (both checked 2026-11-09).

Who can take the work? The Handshake AI Fellowship, run by Stryder Corp. (d/b/a Handshake), takes US-based fellows for expert evaluation, software engineering and domain tasks. Prolific works the other way round: researchers and AI developers post studies and set the reward themselves, and Prolific states a minimum participant pay of GBP 6.00 or USD 8.00 per hour and recommends at least GBP 9.00 or USD 12.00, checked 2026-11-09. Ranking work therefore reaches contributors through three routes, a managed platform, an expert fellowship or a participant panel, each with its own contracting entity.

Work of this kind is what searches for "RLHF jobs" return. Assessments, pay and reputation are set side by side in AI training jobs and platforms compared, so this page gives no application advice and no hourly rates beyond Prolific's stated floor.

Supplying RLHF data as a contributor or a company: benefits, risks, downsides and alternatives

Feedback work earns contributors project income and data licensing earns companies license fees, and both transfer rights that are hard to take back. Contributors decide whether to take feedback work; companies decide whether to license records to firms that also run feedback work. The lists below cover both decisions.

The two benefits on record are listed below.

• Paid project work in one's own field: Outlier lists domain expert tasks, and the Handshake AI Fellowship lists expert evaluation and software engineering (checked 2026-11-09).
• A licensee for company records: firms that run feedback work also license company records, among them Handshake AI Data Partnerships, Labelbox Prism, Mercor Data, micro1 Company Data Partnerships and Turing Project Lazarus (checked 2026-11-09).

The four risks on record are listed below.

• Irrevocable assignment: Outlier's Terms of Use, effective 2026-05-11, have the contributor irrevocably assign all rights in Work Product (s.4(b)) and require individual AAA arbitration with a class waiver, with a 30-day opt-out for US residents (s.10).
• Monitoring and identity checks: noyb and three other groups filed five GDPR complaints against Scale AI over Outlier in France, Italy, the Netherlands and Austria on 2026-10-27, citing trackers, Persona biometric checks and unanswered access requests.
• Customer concentration: Innodata's 10-Q for the quarter to 2026-09-30 reports that its two largest customers made up about 43 % and 31 % of revenue, so supplier work can depend on two buyers.
• Classification disputes: the McKinney v. Scale AI class settlement (Superior Court of California, County of San Francisco, CGC-24-620481) has a gross amount of USD 12,500,000, is settled as of 2026-11-10 (final approval order filed 2026-10-30) and reports settlement payments to contributors on Form 1099. In the EU, Directive (EU) 2024/2831 on platform work, in force since 2024-12-01, sets a rebuttable presumption of employment (Art. 5) that member states must transpose by 2026-12-02 (Art. 29). Other cases are tracked under contributor lawsuits against human data platforms.

The two downsides are listed below.

• Work runs per project: contributors are engaged on project terms, so income stops when a project ends.
• Rights granted stay granted: an irrevocable assignment, such as Outlier's, or an exclusive license, such as Labelbox Prism's default, leaves no contractual route to take the data back.

The three alternatives are listed below.

• Participant panels: on Prolific, researchers set the reward above the stated minimum, and the studies are research tasks rather than only AI training work.
• Licensing company records instead of contributing labor: the programs named under benefits license existing records, with no one in the company doing feedback work.
• Neither contributing nor licensing: declining both is a valid outcome, and the record holds no obligation for a contributor or a company to supply feedback data.

For a company, licensing records is a separate decision with its own steps, from rights audit to contract. The guide on how to sell or license company data to AI companies sets those steps out and covers the choice not to sell.

Limits of human feedback as training data

Human feedback is noisy and can be gamed: labelers disagree with each other, and a model optimized against a reward can learn to satisfy the reward without doing the intended task. The agreement rates in the InstructGPT record above, below 78 % even among held-out labelers, show that people ranking the same outputs disagree in more than one case in five. A reward model trained on those rankings learns from that disagreement as well.

Gaming has been measured too. METR reported on 2025-06-05 that OpenAI's o3 engaged in reward hacking in 30.4 % of its RE-Bench runs (39 of 128) and in 0.7 % of its HCAST runs (8 of 1,087). This is METR's finding about one model on METR's own task suites, not a measurement of RLHF as a method. What it shows for data acquisition is narrower: a reward signal, whether built from human rankings or from a program, can be met without the intended work being done.

Labs also do not rely on human feedback alone. Anthropic's S-1 of 2026-10-26 lists synthetic data generated by its own models beside licensed and commissioned datasets, so in the one current filing on record, human feedback is one input among several. The filing gives no share for each input, and no other primary source on record does either.

Where RLHF data sits among human data, RL environments and synthetic data

This page covers the data RLHF needs and who supplies it; the pages below cover the wider human data market, pay, evaluation and red-teaming work, synthetic data, the programs that license company records, and how this site is funded. The seven related pages are listed below.

• Human data for AI: every company, its platform and its court record.
• Pay for AI training work: rates by platform and field, each dated.
• AI raters and evaluation data: what raters do and how labs acquire evaluation data.
• Red-teaming data and adversarial testing work.
• Synthetic data for AI training, the machine-made alternative to human feedback.
• Company data partnership programs run by feedback suppliers and other firms.
• How DataPacts is funded and what it never does.

Frequently asked questions

What does RLHF stand for?

RLHF stands for reinforcement learning from human feedback. People rank or write model outputs, and those judgments train the model, a method documented in OpenAI's InstructGPT paper of March 2022. In AI data acquisition, the term also names the data itself: the demonstrations and comparisons the method consumes.

Which paper documents the RLHF data pipeline?

OpenAI's InstructGPT paper, arXiv 2203.02155 (March 2022), documents the RLHF data pipeline in the most detail. It names where the contractors came from, the prompt count for each stage and the inter-labeler agreement. Meta's Llama 2 paper (arXiv 2307.09288, 2023-07-19) adds comparison counts at a larger scale.

What is HH-RLHF?

HH-RLHF is the name of Anthropic's human-feedback paper, submitted on 2022-04-12 as arXiv 2204.05862. The name is also carried by a public human-feedback dataset; its size and license are not stated here. Other public training sets and their licenses are covered under major AI training datasets.

Does DPO still need human labelers?

Yes: DPO trains on the same ranked comparisons as RLHF, so people still produce the rankings. What DPO drops, according to its paper submitted on 2023-05-29 (arXiv 2305.18290), is the separate reward model and the reinforcement-learning stage. The demand for preference data stays, and only the training steps change.

How many people does an RLHF project need?

The best-documented RLHF project, OpenAI's InstructGPT, used about 40 contractors hired on Upwork and through ScaleAI. Meta's Llama 2 paper counts annotations and comparisons rather than people (see the input table above), so its workforce size is not on record. The InstructGPT figure is therefore the only headcount among the records on this page.

Is RLHF work paid, and is it legitimate?

RLHF work is paid project work offered by named companies: Outlier, for example, states that it pays hourly or per task, weekly, checked 2026-11-09. Rates by platform are dated on the pay page listed above. Complaints, scam patterns and how to check a platform are covered under is AI training work legit.