DataPacts

Institutional Books (Harvard Library and IDI): Contents, Access Terms, Legal Status

Checked

At a glance

Institutional Books 1.0 is a text dataset of 983,004 public domain volumes from Harvard Library's collection, digitized in the Google Books project and released on June 12, 2025 by the Institutional Data Initiative (IDI) at Harvard Law School Library. The technical report counts 242 billion o200k_base tokens in the released set (arXiv 2506.08300, 2025-06-10).

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Access runs through Hugging Face and requires accepting IDI's noncommercial Terms of Use for Early-Access, and commercial use, including AI model training, requires contacting IDI first.

This page reads the set from the side of commercial users and rights holders, as a library-released alternative to licensed or scraped book text within AI training data licensing. Every figure is dated as of 2026-11-17 unless another date is given.

Who Publishes Institutional Books: IDI, Harvard Library and the Technical Report

Institutional Books is published by the Institutional Data Initiative (IDI), which describes itself as a research initiative at Harvard Law School Library, founded there and born from the Library Innovation Lab. The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Publisher Institutional Data Initiative, "a research initiative at Harvard Law School Library"; no separate legal entity stated institutional.org
Source material Harvard Library, scanned through its participation in the Google Books project, beginning in 2006 arxiv.org/abs/2506.08300; huggingface.co
Released open-sourced June 12, 2025 institutional.org/institutional-books
Technical report "Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability", arXiv 2506.08300, v1 2025-06-10 arxiv.org
Host Hugging Face, repository institutional/institutional-books-hl, created 2025-05-24, last modified 2026-08-19, gated huggingface.co
Funding stated unrestricted gifts from Microsoft and OpenAI; amounts not stated arXiv report; institutional.org
Terms IDI Terms of Use for Early-Access, noncommercial huggingface.co

No separate legal entity is stated, and the terms define "we" to include the President and Fellows of Harvard College. The source material is contributed by Harvard Library, as the attribution text puts it.

The report has 11 authors from IDI, the Library Innovation Lab, Harvard Library, Harvard Law School Library and Harvard Law School, and it states the aim of a "community-led process" and an "institutional commons" for library data.

Hugging Face, Inc. hosts the files and did not create the set. Institutional Books is one of the corpora compared on the page about major AI training datasets.

Who Funded Institutional Books, and What Google Books and Hugging Face Contributed

The technical report states that the work "was supported by unrestricted gifts from Microsoft and OpenAI", and IDI's launch post of December 12, 2024 says IDI's launch is supported by gifts from the same two companies. The post adds that IDI is building "a mosaic of philanthropic and industry supporters". Neither source states an amount.

The report's acknowledgments name help in kind. Hugging Face provided "compute and storage credits", HathiTrust gave "guidance", the Library Innovation Lab incubated the project, and the Google Books team "assisted us in retrieving volumes and metadata and cleared the way for their release".

The launch post also quotes Google as saying it is proud to support IDI's work, but it does not list Google as a funder.

A gift is not a license. Neither source says that Microsoft or OpenAI received access, terms or data in return, and this page states nothing beyond the quotes. Both companies acquire training data elsewhere, as the profile of OpenAI as a data buyer shows.

What Institutional Books 1.0 Contains: Volumes, Pages, Tokens and Dates

Institutional Books 1.0 holds 983,004 volumes, 242 billion o200k_base tokens and, per its card, 386 million pages, published largely in the 19th and 20th centuries. The table sets the released set beside the whole Harvard collection as of 2026-11-17.

Measure Released set Whole Harvard collection Source
Volumes 983,004 (IDI page: "about 983 thousand") 1,075,899 listed; 1,004,977 retrieved arXiv 2506.08300; institutional.org
Tokens (o200k_base) 242 billion 248,299,000,580 arXiv, Table 1 and Section 5.2
Pages 386 million (card) 394,338,216 scanned huggingface.co; arXiv
Dates "largely in the 19th and 20th centuries" (card) 60.55 % from 1820 to 1920; 32.19 % undated huggingface.co; arXiv
Languages n/a 241 (metadata), 254 (detected per volume), 379 (detected in text) arXiv
Likely near-duplicates flagged, not removed 41,366 volumes (3.84 %) arXiv

The full Harvard collection listed 1,075,899 volumes. Of these, 1,004,977 were retrieved through Google's GRIN interface and 70,922 could not be, and the report says the reasons are not fully clear. The collection-wide analysis covers all 1,075,899 records, so whole-collection figures and release figures differ, and the two columns are not combined here.

On dates, 729,604 records carry a valid date, and 650,979 of them (60.55 % of the collection) fall between 1820 and 1920. A fine-tuned model assigned 20 Library of Congress classes, with language and literature, law, philosophy, psychology and religion, and science largest, and the labels are not reviewed by librarians at volume level.

IDI's page says "about 983 thousand" and its report says 983,004, and both are kept. How book text is licensed for a fee is covered on the page about books as AI training data.

Languages in Institutional Books: Why 241, 254 and 379 Are All Stated

Institutional Books is mostly English and other Western European languages, and its three language counts measure different things: 241 languages in the library metadata, 254 detected as a volume's main language, and 379 detected anywhere in the text.

IDI's release page shows 254 in its summary and 379 in its language section, and the report explains the difference between volume-level and text-level detection.

English is about 47 % of volumes. At text level, English holds 105,918,942,360 tokens (43.83 %), German 17.30 % and French 14.01 % (report Table 2). Seventeen languages exceed 1 billion tokens and 230 exceed 1 million.

The report says detection can never be fully accurate, with ancient against modern Greek as an example. Language data as a market of its own is covered on the page about multilingual and low-resource language data.

How IDI Processed the Scans: OCR Clean-up, Near-Duplicates and Topic Labels

IDI kept the Google OCR text and added a post-processed version, quality scores, generated metadata and duplicate flags, without deleting volumes for quality. The processing has four parts.

  • OCR post-processing. A line-type classifier and heuristics reassemble paragraphs and separate running heads and page numbers. The report calls this "minimally but partially destructive", so both texts are released.
  • OCR quality. The Google-provided score averages 88.38 and the OCRoscope score averages 88.16.
  • Near-duplicates. 41,366 volumes (3.84 %) are flagged as likely near-duplicates and listed, not removed, and the last manual check found 97 % accuracy.
  • Fields. Each volume has 24 fields, with generated values marked _gen, including language distribution, topic, OCR score, duplicate barcodes, HathiTrust data and text by page.

At launch, the metadata is published in full for anyone, the text sits behind a click-through, and raw scan images are withheld but shared "liberally with researchers and libraries".

The report warns of historical harmful language in text and metadata. How such release choices are recorded is covered on the page about dataset documentation and data cards.

How IDI Decided Which Volumes Carry No Known Copyright

IDI based its rights screening on HathiTrust data: it matched each Harvard barcode to the HathiTrust Digital Library and released only volumes HathiTrust marks public domain, public domain in the United States, or CC-Zero. The table gives HathiTrust's statuses across the full collection as of 2026-11-17, from the report's Table 7 and Appendix J.

HathiTrust status Volumes % of collection Source
pd (public domain) 786,964 73.14 % arXiv 2506.08300, Appendix J
pdus (public domain in the US) 196,171 18.23 % same
cc-zero 375 0.03 % same
Subtotal pd, pdus, cc-zero 983,510 91.41 % arXiv, Table 7
Known copyright 16,902 1.57 % arXiv, Table 7
Unknown status (incl. no HathiTrust match) 75,353 7.00 % arXiv, Table 7
Other statuses 134 0.01 % arXiv, Table 7
Released after removing textless volumes 983,004 n/a arXiv, Section 5.2

In all, 1,004,497 records matched HathiTrust (93.36 %), 983,510 carried pd, pdus or cc-zero, and 983,004 remained after textless volumes were dropped, with 242 billion tokens. The 16,902 volumes with known copyright and the 75,353 with unknown status stayed out.

The report's rights statement says a volume "may be in the public domain in the United States but still subject to copyright or other rights protections in other jurisdictions". Status is given "on a good-faith basis", and users are "solely responsible for making independent legal assessments".

In the EU the term runs for the author's life plus 70 years from 1 January of the following year (Directive 2006/116/EC, Arts. 1(1), 8), which is why the 196,171 pdus volumes can differ abroad, though no count of affected works exists.

Term rules by country are covered on the page about public domain and openly licensed material in AI training.

The Early-Access Terms of Use: Noncommercial, No Redistribution, Contact First for AI Training

Institutional Books carries no open license: access on Hugging Face requires sharing contact details and accepting IDI's noncommercial Terms of Use for Early-Access, and commercial users, including for AI model training, must contact IDI first. The card metadata names no license.

The report calls the launch terms "a noncommercial license" with a "click-through", and the gate checkbox reads "I accept the noncommercial IDI Terms of Use for Early-Access and will not redistribute the dataset". The table summarizes the ten clauses in the order published, as of 2026-11-17.

Clause What it says Source
1 Noncommercial use only noncommercial purposes only; commercial affiliation or purpose "(including AI model training)" requires contacting IDI first huggingface.co card gate
2 No redistribution no sharing or redistribution in whole or in part, including through public repositories or aggregators same
3 Derivative works allowed for noncommercial use; works that substantially reproduce the dataset may not be made available; only significantly transformed outputs, such as evaluations, summary statistics or visualizations, may be shared with attribution same
4 Attribution public-facing work credits "Institutional Books provided by the Institutional Data Initiative with source material contributed by Harvard Library" same
5 Provisional terms apply only to the Early-Access release and may change same
6 Disclaimer of warranty provided "as is", including no warranty of noninfringement same
7 Limitation of liability "we" includes the President and Fellows of Harvard College same
8 Indemnification user indemnifies against claims from use in breach of the terms or of third-party rights same
9 Governing law Commonwealth of Massachusetts; courts in Middlesex or Suffolk County same
10 Whole agreement entire agreement "between you and Harvard"; amendable by posting same

The terms say they "apply only to this Early-Access release and may change". IDI publishes no commercial terms, price or approval criteria, and this page does not say what IDI answers to a commercial request.

Open-source and public-use projects "unaffiliated with commercial actors or intent" are welcome under the noncommercial clause. How custom terms differ from Creative Commons and ODC licenses is covered on the page about dataset licenses for AI training.

No court record and no published training-content summary on file names Institutional Books as training data, as of 2026-11-17. This page does not claim that any lab trained on the set, including the two funders.

The report says IDI plans to work with other AI research labs to evaluate how the dataset affects model outputs, and that it is in conversation with other Google Books partner libraries. These are plans, not results.

IDI's post of 2026-09-08 describes later releases built on the set, Enriched Text and Visual Elements, with 22.6 million visual elements. The post does not state the access terms of those sets.

Rights holders who believe a work "is misidentified or improperly included" are invited to contact IDI, which says it "will promptly review any concerns", and the stated address is contact@institutional.org. Removal routes across datasets are covered on the page about removing content from AI training datasets.

Using Institutional Books Commercially: Benefits, Risks, Downsides and Alternatives

For a company that wants book text for commercial training, Institutional Books offers documented provenance but no commercial right until IDI agrees.

Benefits

  • Per-volume rights status, dates, languages and duplicate flags travel with the data.
  • A library source chain runs from Harvard Library to the HathiTrust status.

Risks

  • A pdus volume may be protected outside the United States, in IDI's own words.
  • The terms are provisional and may change.
  • Claims and disputes fall under Massachusetts law in Middlesex or Suffolk County courts.

Downsides

  • No commercial terms or timeline are published.
  • Redistribution is barred and derivative datasets are limited.

Alternatives

  • Other openly licensed corpora, listed below.
  • Licensing book text directly from rights holders, as covered on the page about licensed vs scraped training data.
  • Contacting IDI for its commercial route, with no claim made here about its answer.
  • Not using the set at all.

Library-side choices are covered in the border list below. Not using or not licensing is a valid outcome.

This page covers Institutional Books 1.0; other corpora, library releases, funders' profiles and copyright rules each have their own page.

Frequently Asked Questions About Institutional Books

What is Institutional Books 1.0?

Institutional Books 1.0 is a set of 983,004 public domain volumes from Harvard Library's Google Books scans, released by IDI on June 12, 2025. The card describes it as a growing corpus.

How big is Institutional Books?

The released set has 983,004 volumes and 242 billion o200k_base tokens, and its card states 386 million pages. The full Harvard collection listed 1,075,899 volumes.

Who funds the Institutional Data Initiative?

The technical report names unrestricted gifts from Microsoft and OpenAI. Amounts are not published.

Is everything in Institutional Books in the public domain?

Every volume carries a HathiTrust status of public domain, public domain in the US, or CC-Zero. IDI says US status may not hold in other jurisdictions. This page is information, not legal advice.