An AI company has “destroyed” millions of books as part of a “controversial project” to train its models, said City A.M.
Tech giants want to “feed an insatiable demand for training data” but their methods could have serious ramifications for the publishing industry and the quality of future books.
Suspicious orders
The alarm was first raised when antiquarian booksellers in the UK and across Europe started to receive curious requests for thousands of obscure titles from anonymous buyers.
A London-based seller of rare books received two “unusually large” enquiries for specialist books from anonymous email addresses, with “no names or company details attached”, said City A.M. The buyers asked for additional photographs of the books before placing sizeable orders.
When the bookseller asked the purchasers “who they were and where the books were going, neither replied”. The retailer suspects the orders were placed by AI companies.
According to recent court documents, Anthropic, the developer behind Claude, launched an internal programme known as “Project Panama”, which concluded that books were “essential” for training advanced AI models.
One senior executive at the AI giant said that books taught systems “how to write well”, as opposed to using “low-quality internet speak”. The company bought millions of second-hand books, “sliced off their spines” using “industrial cutting machines”, scanned every page and recycled the rest.
Internal planning documents at Anthropic also showed bosses wanted to keep the programme secret, said Futurism. One executive said: “We don’t want it to be known that we are pursuing this project,” according to those recent court documents.
Orders have also come from Canadian company Zoom Books, according to AA, but the company has denied participating in the digitisation and destruction of books for AI development.
‘Legal loopholes’
This trend has “raised fresh concerns about copyright and the future of physical books”, said AA. But dealing with the issue could be quite another matter; copyright violations are “difficult to prove in court” because of “legal loopholes”.
Thomas Koch, of the German Publishers and Booksellers Association, said it “appears to be yet another example of AI companies using vast quantities of copyright-protected works to train their language models, without consent and without payment”.
The absence of disclosure obligations makes it “nearly impossible” for authors and publishers to determine whether their works have been incorporated into AI training datasets. Also, the legal standing “differs significantly across jurisdictions”.
AI systems have been “trained on copyrighted books” and can now produce new books “almost instantly”, leading to an “influx” of “low-cost, AI-generated titles” on digital marketplaces. Publishers are calling for stronger regulation from national governments and the EU.