One buyer program states that models already trained on licensed data cannot be unlearned (checked November 9, 2026), and European regulators test the model itself, not only the dataset.
This page takes the side of the people in the data: the routes of exposure, what regulators have said with dates, the case that the risk is contained and the case that it is not, and the mitigations on record.
The people in the data are one party affected by AI training data licensing, and the seller's own steps and the legal tests sit on other pages. Not licensing records about people is a valid outcome.
DataPacts Editorial Team dated and checked every rule, case and buyer term on this page against its primary source, and the page on how we source, date and check every figure sets out the method.
Six Ways People in a Licensed Dataset Can Be Exposed
Exposure can happen at six points: in the delivered copy, in the trained model, through inference, in the buyer's systems, through onward transfer, and after training, when the data can no longer be recalled. The table lists each path with one example on record, as of 2026-11-17.
| Path | What can happen to a person | Example on record | Source |
|---|---|---|---|
| Delivered copy | linked back to them through remaining attributes | 15 demographic attributes re-identify 99.98 % of Americans (2019-07-23) | nature.com |
| Trained model | personal data extracted from the model or obtained through queries | EDPB anonymity test, 2024-12-17 | edpb.europa.eu |
| Inference | shown to be in the source data (membership) | EDPB Guidelines 02/2026 s.3.4, consultation draft of 2026-07-07 | edpb.europa.eu |
| Buyer's systems | exposed in a breach at the buyer | Mercor incident, breach dates 2026-03-24 and 2026-03-30 | oag.ca.gov |
| Onward transfer | passed to further parties | micro1 license sublicensable through multiple tiers (s.1.2, last updated 2026-11-12); 28 CFR Part 202 reaches anonymized or de-identified bulk data | micro1.ai; ecfr.gov |
| After training | cannot be recalled from a model | Labelbox: "cannot be unlearned" (checked 2026-11-09); micro1 s.4.4 | prism.labelbox.com; micro1.ai |
The paths add up, so a copy that passes one test can still fail another.
Re-identification: Why Removing Names Leaves People Findable
A record without a name can still point to one person, because a handful of ordinary attributes is often unique.
Rocher, Hendrickx and de Montjoye estimated in Nature Communications 10:3069 (published July 23, 2019) that 15 demographic attributes correctly re-identify 99.98 % of Americans in any dataset, which means a person can be found again in a file that holds no name at all.
The legal standards accept a residual risk. Under 45 CFR 164.514(b)(1), an expert determination must find the risk "very small" that the information could be used, alone or in combination, by an anticipated recipient to identify an individual, with the methods documented, while Safe Harbor under (b)(2) removes 18 identifier types.
GDPR Recital 26 treats information as anonymous only when a person is not identifiable by all means reasonably likely to be used.
The law answers with promises and penalties. Cal. Civ. Code 1798.140(m) asks for reasonable measures against association, a public commitment not to reidentify and contractual obligations on recipients.
Quebec's Law 25 (S.Q. 2021, c. 25) makes re-identifying de-identified or anonymized information an offence under P-39.1 s.91, as replaced by s.160, with fines for an organization of CAD 15,000 to 25,000,000 or 4 % of worldwide turnover, per the Act's text published by the Quebec access commission (CAI).
For education records, 34 CFR 99.31(b)(1) requires a reasonable determination that a person cannot be identified across single or multiple releases.
The seller's identifiers, methods and pre-delivery test sit on the page about PII removal and de-identification before licensing, and this page does not repeat them.
Memorization and Extraction: When the Trained Model Carries Personal Data
A model trained on personal records can keep parts of them, which is why European regulators test the model itself and not only the dataset. The EDPB adopted Opinion 28/2024 on December 17, 2024 under Article 64(2) GDPR at the Irish authority's request.
It says a model is anonymous only if the likelihood of direct extraction of personal data, including probabilistic extraction, and of obtaining it through queries is insignificant, taking into account all means reasonably likely to be used by the controller or another person.
The opinion names two routes in plain terms: personal data extracted from the model's parameters, and personal data obtained from what the model outputs. It does not say that every model is personal data, and it does not say that no model can be anonymous.
Memorization is no longer only a research claim in court. The Munich Regional Court (42 O 14139/24, November 11, 2025) held that song lyrics memorized in a model's parameters are a reproduction. This is a copyright ruling and it is not final. It matters here only because it is a judicial finding that memorization happens.
Inference is the third route. The EDPB's Guidelines 02/2026 on anonymisation (adopted for consultation on July 7, 2026, with the consultation open until October 30, 2026, and not final) treat membership inference, learning that a person's record was in the source data, as an inference that "would, in particular, be the case for AI models".
Being known to be in a dataset can itself reveal something, for example that a person is on a clinic's patient list.
Opinion 28/2024 (para. 114) lists erasure of the dataset and/or the model among the corrective measures. The full test is set out on the page about when an AI model is anonymous under EDPB Opinion 28/2024.
What Regulators Have Said About Risk to People in the Data, Dated
Regulators have addressed each risk path at least once since 2021, mostly through orders against the company that trained or collected, and rarely through a statement about a license. The table runs in date order, as of 2026-11-17.
| Date | Regulator | Statement or action | Risk path | Source |
|---|---|---|---|---|
| 2021-05-07 | FTC | Everalbum final order: delete "the models and algorithms it developed by using the photos and videos uploaded by its users" | after training | ftc.gov |
| 2024-01-09 | FTC Office of Technology staff | prior orders required deleting models and algorithms built on unlawfully obtained data | after training | ftc.gov |
| 2024-02-13 | FTC Office of Technology staff | quietly adopting more permissive data practices may be unfair or deceptive (staff blog, not a rule) | delivered copy (purpose) | ftc.gov |
| 2024-11-02 | Garante (Italy) | EUR 15 million fine on OpenAI (legal basis, transparency, unnotified March 2023 breach, age checks); opposition upheld by the Tribunale di Roma, 4153/2026, published 2026-03-18 | trained model; buyer's systems | garanteprivacy.it |
| 2024-12-17 | EDPB | Opinion 28/2024: model anonymous only if extraction and query risks are insignificant | trained model | edpb.europa.eu |
| 2025-04-22 | FTC | COPPA Rule amendments published (compliance 2026-04-22): disclosing a child's data to train AI needs separate verifiable parental consent | delivered copy | federalregister.gov |
| 2025-04-28 | ICO (UK) with OPC Canada | joint letter to the Acting US Trustee in the 23andMe bankruptcy (content not stated here) | onward transfer | priv.gc.ca |
| 2026-05-06 | OPC Canada with provincial offices | Findings #2026-002: licensed datasets require that individuals were given a choice | delivered copy | priv.gc.ca |
| 2026-07-07 | EDPB | Guidelines 02/2026 (consultation draft): membership inference can breach the no-inference criterion | inference | edpb.europa.eu |
| 2026-07-13 | Autoriteit Persoonsgegevens (Netherlands) | the model builder, as buyer of the data, must check that personal data was lawfully obtained and may be used for the new purpose | onward transfer | autoriteitpersoonsgegevens.nl |
The pattern is recall and model deletion in the United States, model anonymity and buyer checks in the European Union, and consent for licensed datasets in Canada. The CNIL's non-binding recommendations (English summary dated January 5, 2026) add a sheet on analysing the status of an AI model under the GDPR.
The ICO appears in this record only as co-signer of the 23andMe letter, and no regulator statement in this record names the re-identification of a licensed AI training set as a finding. The rules behind these statements are compared in the page on privacy law and AI training data by jurisdiction.
Is the Risk Contained or Understated? The Case Each Side Makes
Both sides cite primary sources: one points to standards that accept a small residual risk and to sales approved with safeguards, the other to how few attributes identify a person and to models that cannot forget.
The case that the risk is contained:
- HIPAA's expert determination accepts a "very small", documented risk (45 CFR 164.514(b)(1)).
- Opinion 28/2024 allows that a model can be anonymous when extraction and query risks are insignificant.
- The Spirit Airlines ombudsman's supplemental report (Dkt. 1684, October 5, 2026) found that approval of the de-identified data sale "would be appropriate" from a privacy perspective after passenger databases were excluded and Tonic.ai was hired as de-identification agent; the sale was not approved as of November 9, 2026.
- The Tribunale di Roma (4153/2026, published March 18, 2026) upheld OpenAI's opposition to the Garante's fine of November 2, 2024; the grounds are not stated here.
The case that the risk is understated:
- Fifteen demographic attributes suffice for re-identification in the 2019 estimate.
- AfterQuery states that it runs a re-identification test before delivery and that "no process removes all risk on its own" (checked November 9, 2026).
- Labelbox states "Models already trained on the data cannot be unlearned" (checked November 9, 2026), and micro1's terms (s.4.4, last updated November 12, 2026) impose no duty to delete or retrain.
- Mercor reported a security incident with breach dates of March 24 and March 30, 2026 to the California Attorney General on June 25, 2026, and the Spirit ombudsman asked for further research into it if the court considers Mercor as successful bidder. In the consolidated class action (lead case 3:26-cv-02821-WHO) the parties stipulated on July 21, 2026 to stay proceedings pending mediation, with no merits ruling as of November 9, 2026.
Neither side claims zero risk; the disagreement is about how small is small enough and who carries the rest. Breaches at buyers in general are tracked on the page about data buyer incidents, breaches and complaints.
Practical Mitigations Before, During and After Delivery
The mitigations on record work at three moments: before delivery by leaving people out, during delivery by limiting what the buyer can do, and after delivery by keeping a route to deletion and notice. The table groups them by moment, as of 2026-11-17.
| Moment | Mitigation | What it reduces | Example or rule | Source |
|---|---|---|---|---|
| Before | leave children's data out unless separate parental consent exists | exposure of children | COPPA Rule, 90 FR 16950 | federalregister.gov |
| Before | exclude whole systems that hold the most sensitive records | linkage and breach exposure | Spirit passenger databases excluded (Dkt. 1684) | courtlistener.com |
| Before | de-identify to a named standard | linkage | CCPA 1798.140(m); 45 CFR 164.514(b) | cppa.ca.gov; ecfr.gov |
| Before | run a re-identification test | linkage | AfterQuery states it tests before delivery (checked 2026-11-09) | afterquery.com |
| During | ban re-identification in the contract | linkage by the buyer | CCPA contractual obligations; Human Archive states its recordings are licensed under written agreements prohibiting re-identification (checked 2026-11-17) | cppa.ca.gov; humanarchive.ai |
| During | require encryption and fast breach notice | buyer breach | micro1 s.2.4 and s.2.5: notice within 2 business days (updated 2026-11-12) | micro1.ai |
| During | check the buyer's security record | buyer breach | Spirit ombudsman on Mercor (Dkt. 1684) | courtlistener.com |
| After | keep a deletion trigger | onward use | Labelbox kill clause: deletion within 30 days (re-read 2026-11-17) | prism.labelbox.com |
| After | tell people and give a choice | purpose and consent | OPC 2026-05-06; FTC staff 2024-02-13 | priv.gc.ca; ftc.gov |
No mitigation reaches a model that is already trained, which is why the first two moments carry most of the weight. Clause drafting for the third is covered in the page on deletion, return and machine unlearning clauses.
Licensing Records About People: Benefits, Risks, Downsides and Alternatives
Records about people can make a dataset more useful to a buyer, and every gain comes with a risk to those people that the license cannot fully remove.
Benefits:
- Records with context are what buyers ask for, and an exclusion or de-identification standard written into the license gives the people in it a defined protection.
- A court-supervised sale adds an ombudsman's review.
Risks:
- Re-identification of the copy.
- Personal data retained in the model or obtained from it.
- A breach at the buyer.
Downsides:
- No recall after training, as Labelbox and micro1 state.
- The seller usually indemnifies privacy claims: micro1's terms (s.8.1) have the licensor indemnify privacy and data-protection claims.
Alternatives:
- License only systems that hold no personal data.
- Exclude whole systems, as the Spirit sale did with passenger databases.
- De-identify to a named standard and test the result.
- Do not license records about people at all.
The full comparison is on the page about alternatives to selling data, including not selling at all.
How Privacy Risk Fits the Ethics of Selling Data
Privacy risk is one of four ethics questions about selling data for AI training, next to the employee view, fair treatment of creators and reputational risk to the company. The parent overview covers ethics and risk of selling data for AI training in full.
- Employees: staff inside the archive
- Creators: creators whose works are in the data
- Company: reputational risk of licensing data
- Legal status: what each legal status achieves; membership inference against synthetic sets
- Children and sensitive records: the parental-consent rule in full; health, biometric and other sensitive records; health records and the expert determination standard
- Regulators and courts: model deletion orders and policy changes; the Garante's fine and the court's judgment; Quebec Law 25 and AI training data; Spirit Airlines' data archive sale
What This Page Does Not Cover
This page covers people inside data that a company licenses for AI training, not apps training on their own users, chatbot privacy settings or what people type into AI tools. The privacy of contributors paid to produce training data is a separate topic. The page is information, not legal advice.
Frequently Asked Questions
Can an AI model leak personal data from its training data?
It can, which is why the EDPB treats a model as anonymous only when extracting personal data from it, or obtaining it through queries, is insignificant. The Munich Regional Court found in November 2025 that lyrics memorized in a model's parameters are a reproduction, a copyright finding that shows memorization happens. See GEMA v. OpenAI, where the Munich court found memorized lyrics in the model.
Can de-identified data be re-identified?
Yes, when enough attributes remain: one study estimated that 15 demographic attributes correctly re-identify 99.98 % of Americans in any dataset. California's standard therefore adds a public commitment not to reidentify and contractual obligations, and Quebec's Law 25 makes re-identification an offence.
Who carries the risk if people in licensed data are identified?
The people in the data carry the harm, while the law and the contract split responsibility between seller and buyer. The Dutch authority said on July 13, 2026 that the model builder, as buyer, must check that personal data was lawfully obtained, micro1's terms have the licensor indemnify privacy claims, and Opinion 28/2024 lists erasure of the model among the corrective measures.