DataPacts

Privacy Risk to People in Licensed Data

Checked

At a glance

People whose records sit in a licensed dataset can be exposed in more ways than the delivered copy suggests: the copy can be linked back to them, the trained model can carry their data, and a trained model cannot give it back.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

One buyer program states that models already trained on licensed data cannot be unlearned (checked November 9, 2026), and European regulators test the model itself, not only the dataset.

This page takes the side of the people in the data: the routes of exposure, what regulators have said with dates, the case that the risk is contained and the case that it is not, and the mitigations on record.

The people in the data are one party affected by AI training data licensing, and the seller's own steps and the legal tests sit on other pages. Not licensing records about people is a valid outcome.

DataPacts Editorial Team dated and checked every rule, case and buyer term on this page against its primary source, and the page on how we source, date and check every figure sets out the method.

Six Ways People in a Licensed Dataset Can Be Exposed

Exposure can happen at six points: in the delivered copy, in the trained model, through inference, in the buyer's systems, through onward transfer, and after training, when the data can no longer be recalled. The table lists each path with one example on record, as of 2026-11-17.

Path What can happen to a person Example on record Source
Delivered copy linked back to them through remaining attributes 15 demographic attributes re-identify 99.98 % of Americans (2019-07-23) nature.com
Trained model personal data extracted from the model or obtained through queries EDPB anonymity test, 2024-12-17 edpb.europa.eu
Inference shown to be in the source data (membership) EDPB Guidelines 02/2026 s.3.4, consultation draft of 2026-07-07 edpb.europa.eu
Buyer's systems exposed in a breach at the buyer Mercor incident, breach dates 2026-03-24 and 2026-03-30 oag.ca.gov
Onward transfer passed to further parties micro1 license sublicensable through multiple tiers (s.1.2, last updated 2026-11-12); 28 CFR Part 202 reaches anonymized or de-identified bulk data micro1.ai; ecfr.gov
After training cannot be recalled from a model Labelbox: "cannot be unlearned" (checked 2026-11-09); micro1 s.4.4 prism.labelbox.com; micro1.ai

The paths add up, so a copy that passes one test can still fail another.

Re-identification: Why Removing Names Leaves People Findable

A record without a name can still point to one person, because a handful of ordinary attributes is often unique.

Rocher, Hendrickx and de Montjoye estimated in Nature Communications 10:3069 (published July 23, 2019) that 15 demographic attributes correctly re-identify 99.98 % of Americans in any dataset, which means a person can be found again in a file that holds no name at all.

The legal standards accept a residual risk. Under 45 CFR 164.514(b)(1), an expert determination must find the risk "very small" that the information could be used, alone or in combination, by an anticipated recipient to identify an individual, with the methods documented, while Safe Harbor under (b)(2) removes 18 identifier types.

GDPR Recital 26 treats information as anonymous only when a person is not identifiable by all means reasonably likely to be used.

The law answers with promises and penalties. Cal. Civ. Code 1798.140(m) asks for reasonable measures against association, a public commitment not to reidentify and contractual obligations on recipients.

Quebec's Law 25 (S.Q. 2021, c. 25) makes re-identifying de-identified or anonymized information an offence under P-39.1 s.91, as replaced by s.160, with fines for an organization of CAD 15,000 to 25,000,000 or 4 % of worldwide turnover, per the Act's text published by the Quebec access commission (CAI).

For education records, 34 CFR 99.31(b)(1) requires a reasonable determination that a person cannot be identified across single or multiple releases.

The seller's identifiers, methods and pre-delivery test sit on the page about PII removal and de-identification before licensing, and this page does not repeat them.

Memorization and Extraction: When the Trained Model Carries Personal Data

A model trained on personal records can keep parts of them, which is why European regulators test the model itself and not only the dataset. The EDPB adopted Opinion 28/2024 on December 17, 2024 under Article 64(2) GDPR at the Irish authority's request.

It says a model is anonymous only if the likelihood of direct extraction of personal data, including probabilistic extraction, and of obtaining it through queries is insignificant, taking into account all means reasonably likely to be used by the controller or another person.

The opinion names two routes in plain terms: personal data extracted from the model's parameters, and personal data obtained from what the model outputs. It does not say that every model is personal data, and it does not say that no model can be anonymous.

Memorization is no longer only a research claim in court. The Munich Regional Court (42 O 14139/24, November 11, 2025) held that song lyrics memorized in a model's parameters are a reproduction. This is a copyright ruling and it is not final. It matters here only because it is a judicial finding that memorization happens.

Inference is the third route. The EDPB's Guidelines 02/2026 on anonymisation (adopted for consultation on July 7, 2026, with the consultation open until October 30, 2026, and not final) treat membership inference, learning that a person's record was in the source data, as an inference that "would, in particular, be the case for AI models".

Being known to be in a dataset can itself reveal something, for example that a person is on a clinic's patient list.

Opinion 28/2024 (para. 114) lists erasure of the dataset and/or the model among the corrective measures. The full test is set out on the page about when an AI model is anonymous under EDPB Opinion 28/2024.

What Regulators Have Said About Risk to People in the Data, Dated

Regulators have addressed each risk path at least once since 2021, mostly through orders against the company that trained or collected, and rarely through a statement about a license. The table runs in date order, as of 2026-11-17.

Date Regulator Statement or action Risk path Source
2021-05-07 FTC Everalbum final order: delete "the models and algorithms it developed by using the photos and videos uploaded by its users" after training ftc.gov
2024-01-09 FTC Office of Technology staff prior orders required deleting models and algorithms built on unlawfully obtained data after training ftc.gov
2024-02-13 FTC Office of Technology staff quietly adopting more permissive data practices may be unfair or deceptive (staff blog, not a rule) delivered copy (purpose) ftc.gov
2024-11-02 Garante (Italy) EUR 15 million fine on OpenAI (legal basis, transparency, unnotified March 2023 breach, age checks); opposition upheld by the Tribunale di Roma, 4153/2026, published 2026-03-18 trained model; buyer's systems garanteprivacy.it
2024-12-17 EDPB Opinion 28/2024: model anonymous only if extraction and query risks are insignificant trained model edpb.europa.eu
2025-04-22 FTC COPPA Rule amendments published (compliance 2026-04-22): disclosing a child's data to train AI needs separate verifiable parental consent delivered copy federalregister.gov
2025-04-28 ICO (UK) with OPC Canada joint letter to the Acting US Trustee in the 23andMe bankruptcy (content not stated here) onward transfer priv.gc.ca
2026-05-06 OPC Canada with provincial offices Findings #2026-002: licensed datasets require that individuals were given a choice delivered copy priv.gc.ca
2026-07-07 EDPB Guidelines 02/2026 (consultation draft): membership inference can breach the no-inference criterion inference edpb.europa.eu
2026-07-13 Autoriteit Persoonsgegevens (Netherlands) the model builder, as buyer of the data, must check that personal data was lawfully obtained and may be used for the new purpose onward transfer autoriteitpersoonsgegevens.nl

The pattern is recall and model deletion in the United States, model anonymity and buyer checks in the European Union, and consent for licensed datasets in Canada. The CNIL's non-binding recommendations (English summary dated January 5, 2026) add a sheet on analysing the status of an AI model under the GDPR.

The ICO appears in this record only as co-signer of the 23andMe letter, and no regulator statement in this record names the re-identification of a licensed AI training set as a finding. The rules behind these statements are compared in the page on privacy law and AI training data by jurisdiction.

Is the Risk Contained or Understated? The Case Each Side Makes

Both sides cite primary sources: one points to standards that accept a small residual risk and to sales approved with safeguards, the other to how few attributes identify a person and to models that cannot forget.

The case that the risk is contained:

  • HIPAA's expert determination accepts a "very small", documented risk (45 CFR 164.514(b)(1)).
  • Opinion 28/2024 allows that a model can be anonymous when extraction and query risks are insignificant.
  • The Spirit Airlines ombudsman's supplemental report (Dkt. 1684, October 5, 2026) found that approval of the de-identified data sale "would be appropriate" from a privacy perspective after passenger databases were excluded and Tonic.ai was hired as de-identification agent; the sale was not approved as of November 9, 2026.
  • The Tribunale di Roma (4153/2026, published March 18, 2026) upheld OpenAI's opposition to the Garante's fine of November 2, 2024; the grounds are not stated here.

The case that the risk is understated:

  • Fifteen demographic attributes suffice for re-identification in the 2019 estimate.
  • AfterQuery states that it runs a re-identification test before delivery and that "no process removes all risk on its own" (checked November 9, 2026).
  • Labelbox states "Models already trained on the data cannot be unlearned" (checked November 9, 2026), and micro1's terms (s.4.4, last updated November 12, 2026) impose no duty to delete or retrain.
  • Mercor reported a security incident with breach dates of March 24 and March 30, 2026 to the California Attorney General on June 25, 2026, and the Spirit ombudsman asked for further research into it if the court considers Mercor as successful bidder. In the consolidated class action (lead case 3:26-cv-02821-WHO) the parties stipulated on July 21, 2026 to stay proceedings pending mediation, with no merits ruling as of November 9, 2026.

Neither side claims zero risk; the disagreement is about how small is small enough and who carries the rest. Breaches at buyers in general are tracked on the page about data buyer incidents, breaches and complaints.

Practical Mitigations Before, During and After Delivery

The mitigations on record work at three moments: before delivery by leaving people out, during delivery by limiting what the buyer can do, and after delivery by keeping a route to deletion and notice. The table groups them by moment, as of 2026-11-17.

Moment Mitigation What it reduces Example or rule Source
Before leave children's data out unless separate parental consent exists exposure of children COPPA Rule, 90 FR 16950 federalregister.gov
Before exclude whole systems that hold the most sensitive records linkage and breach exposure Spirit passenger databases excluded (Dkt. 1684) courtlistener.com
Before de-identify to a named standard linkage CCPA 1798.140(m); 45 CFR 164.514(b) cppa.ca.gov; ecfr.gov
Before run a re-identification test linkage AfterQuery states it tests before delivery (checked 2026-11-09) afterquery.com
During ban re-identification in the contract linkage by the buyer CCPA contractual obligations; Human Archive states its recordings are licensed under written agreements prohibiting re-identification (checked 2026-11-17) cppa.ca.gov; humanarchive.ai
During require encryption and fast breach notice buyer breach micro1 s.2.4 and s.2.5: notice within 2 business days (updated 2026-11-12) micro1.ai
During check the buyer's security record buyer breach Spirit ombudsman on Mercor (Dkt. 1684) courtlistener.com
After keep a deletion trigger onward use Labelbox kill clause: deletion within 30 days (re-read 2026-11-17) prism.labelbox.com
After tell people and give a choice purpose and consent OPC 2026-05-06; FTC staff 2024-02-13 priv.gc.ca; ftc.gov

No mitigation reaches a model that is already trained, which is why the first two moments carry most of the weight. Clause drafting for the third is covered in the page on deletion, return and machine unlearning clauses.

Licensing Records About People: Benefits, Risks, Downsides and Alternatives

Records about people can make a dataset more useful to a buyer, and every gain comes with a risk to those people that the license cannot fully remove.

Benefits:

  • Records with context are what buyers ask for, and an exclusion or de-identification standard written into the license gives the people in it a defined protection.
  • A court-supervised sale adds an ombudsman's review.

Risks:

  • Re-identification of the copy.
  • Personal data retained in the model or obtained from it.
  • A breach at the buyer.

Downsides:

  • No recall after training, as Labelbox and micro1 state.
  • The seller usually indemnifies privacy claims: micro1's terms (s.8.1) have the licensor indemnify privacy and data-protection claims.

Alternatives:

  • License only systems that hold no personal data.
  • Exclude whole systems, as the Spirit sale did with passenger databases.
  • De-identify to a named standard and test the result.
  • Do not license records about people at all.

The full comparison is on the page about alternatives to selling data, including not selling at all.

How Privacy Risk Fits the Ethics of Selling Data

Privacy risk is one of four ethics questions about selling data for AI training, next to the employee view, fair treatment of creators and reputational risk to the company. The parent overview covers ethics and risk of selling data for AI training in full.

What This Page Does Not Cover

This page covers people inside data that a company licenses for AI training, not apps training on their own users, chatbot privacy settings or what people type into AI tools. The privacy of contributors paid to produce training data is a separate topic. The page is information, not legal advice.

Frequently Asked Questions

Can an AI model leak personal data from its training data?

It can, which is why the EDPB treats a model as anonymous only when extracting personal data from it, or obtaining it through queries, is insignificant. The Munich Regional Court found in November 2025 that lyrics memorized in a model's parameters are a reproduction, a copyright finding that shows memorization happens. See GEMA v. OpenAI, where the Munich court found memorized lyrics in the model.

Can de-identified data be re-identified?

Yes, when enough attributes remain: one study estimated that 15 demographic attributes correctly re-identify 99.98 % of Americans in any dataset. California's standard therefore adds a public commitment not to reidentify and contractual obligations, and Quebec's Law 25 makes re-identification an offence.

Who carries the risk if people in licensed data are identified?

The people in the data carry the harm, while the law and the contract split responsibility between seller and buyer. The Dutch authority said on July 13, 2026 that the model builder, as buyer, must check that personal data was lawfully obtained, micro1's terms have the licensor indemnify privacy claims, and Opinion 28/2024 lists erasure of the model among the corrective measures.