DataPacts

Seller track · Lesson 4 of 9 · 9:33

De-identification without destroying the value

What you'll learn

  • Where identifiers hide in your records
  • Five ways to remove them
  • Which standards a license can name
  • How to test before it ships

Free lesson

Enter your email to watch this lesson and the rest of the course.

Transcript

Researchers estimated in 2019 that ninety-nine point nine eight percent of Americans can be correctly re-identified in any dataset, using just fifteen demographic attributes. No names needed. Dates, places and job titles are enough. So taking out the names doesn't make a table anonymous.

De-identification means taking names, contact details and other identifiers out of company records, so the copy you license can't reasonably be linked to a person. In AI deals it happens before the data reaches the lab. It's one step. It narrows what you sell. It doesn't replace the rights audit, the consent review, or the choice not to license at all.

First, what counts as an identifier.

A direct identifier names the person on its own: a name, an email address, an account number. An indirect identifier, also called a quasi-identifier, only identifies someone in combination with others. Think job titles, dates and locations. That's why removing names alone doesn't suffice.

Redaction, masking and pseudonymization are techniques. Anonymization is a legal result, and it depends on the jurisdiction. One buyer program uses de-identified and anonymized for the same step. So before you sign, ask which standard is meant.

Next, where identifiers actually hide.

In email, chat and shared documents, identifiers sit in headers, signatures, mentions and file metadata, as well as in the text. And one inbox holds staff, customer and outside parties at once. One buyer program says it removes or pseudonymizes identifiers and masks customer identities.

Staff messages count too. California's exemption for employee and business-to-business data has been inoperative since January first, twenty twenty-three, so employee messages are personal information. Removing names doesn't answer whether staff were told. You still need employee notice.

Source code carries personal data in commit authors, email addresses and comments. It also carries secrets, like keys and passwords, which have to come out for security as well as privacy. Pseudonymized authors keep the commit history usable. A leaked key harms the company whatever the privacy standard says.

Health data needs a health-data standard first. One buyer program says clinical records must be de-identifiable to HIPAA Safe Harbor or Expert Determination. Images must be fully de-identified, including text burned into the pixels. Audio and transcripts need the same, with consent that covers secondary use.

Client files in law, accounting and advisory firms hold the client's people and confidential business. So buyers ask for client consent, or for files that can be fully anonymized. And anonymizing doesn't lift every bar. A tax preparer may not sell or exchange statistical compilations of return information for value, except with the sale of the business.

Some material never goes in at all. One buyer program says it never uses payroll, privileged legal material or credentials. Take those out of your inventory first, so nobody spends time redacting what can't be licensed anyway.

Now, five ways to remove them.

Redaction deletes or blacks out the identifier. Masking swaps it for a fixed placeholder. Pseudonymization replaces it with a consistent token. Generalization coarsens values into ranges. And synthetic replacement substitutes realistic fake values.

The trade is what the buyer keeps. Redaction leaves context without the value, and the risk is low for the removed field. Pseudonymization keeps the links between records, so the data stays useful. But the risk is higher, because the key or patterns can relink it. That's why the key stays with the seller.

Generalization shows how precise a rule can be. HIPAA's Safe Harbor method removes eighteen identifier types. ZIP codes are cut to three digits, and only where the area holds more than twenty thousand people. And ages over eighty-nine become ninety or older.

Then, which standard your license should name.

A standard is the test your licensed copy must pass, and the license should name one. Take California's. Data counts as deidentified only if the business takes reasonable measures against re-association, publicly commits not to re-identify, and binds every recipient by contract.

For health data held by a covered entity, there are two routes. Safe Harbor removes the eighteen identifier types. Expert Determination relies on a qualified expert. A limited data set is different. It keeps some dates and locations, so it stays protected health information, and licensing it for a fee is a sale that needs authorization.

In the EU, the question is whether data is anonymous under the GDPR. The Court of Justice held in September twenty twenty-five that pseudonymized data isn't necessarily personal data for every recipient. The EU privacy board's draft guidelines test isolation, linkage and inference. Neither text is final as of November twenty twenty-six, so talk to a lawyer.

Now, who does the work, and who pays.

Most buyer programs say they de-identify the data themselves. Where they do it differs. One program says it anonymizes inside the holder's environment, before any data is extracted. Another de-identifies after transfer. That's a real difference in when raw data moves.

Who pays? It depends. Two programs say they cover the cost. In one bankruptcy sale, the sale document put it on the buyer. Another program doesn't say. And no program or vendor publishes a price, so you can learn who pays, and still not learn how much.

A bankrupt US airline's sale shows a court-supervised choice. The estate hired an independent agent to de-identify the data, instead of letting the buyer do it. The sale document requires the California standard, plus the HIPAA rule for health data. And whole systems stayed out, deidentified or otherwise.

One more detail from that sale. The ombudsman's report says that from a privacy perspective, approving the sale of the deidentified data would be appropriate. Yet its footnote says risks to employees and third parties were not investigated. De-identified doesn't mean fully examined.

Then, test the result before it ships.

A seller's own testing has three parts. First, review a sample of the redacted output before transfer. Second, run a re-identification test on the delivered copy, including combinations of indirect identifiers. Third, keep a written record of the standard you applied and the result.

Why bother? One data program states that models already trained on the data cannot be unlearned. An error you find after delivery can't be reversed in a trained model. Tools help, but none of them guarantees complete removal. One open-source toolkit says so itself: there is no guarantee it will find all sensitive information.

Last, what de-identification can't fix.

Some rules apply even to anonymized data. A US Justice Department rule, 28 CFR Part 202, bars bulk US sensitive personal data from buyers in or linked to six named countries, even if the data is de-identified. The thresholds are more than ten thousand US persons for personal health data, and more than a hundred thousand for covered personal identifiers. Talk to a lawyer about it.

A standard only holds if the license carries it. Four terms do that. A commitment not to re-identify. Flow-down of the same commitment to later recipients. The named standard, and who applies it. And who carries the privacy risk. In one program's master terms, that's the seller, through an indemnity.

Now the title question: keeping the value. Lost detail lowers how useful the data is to the buyer. So choose methods that keep what buyers need, like the links between records. Or license a narrower subset. Or leave whole systems out. Or do not license at all, which is a valid outcome.

Where does this step sit in the whole process? After the inventory, the rights audit and the consent review. Before valuation, the data manifest, and any sample a buyer sees. Skip the order, and you risk cleaning data you can't sell.

So, in one line. Find where identifiers hide, pick a method that keeps what the buyer needs, name a standard, test before delivery, and remember that saying no is a valid outcome.

Next up is lesson A5, Valuing your data and reading an offer.

Read and watch the full lesson free. Enter your email to unlock every lesson of the academy.

Sources on DataPacts