Hi! I'm one of the programmers at Gutenberg.
We've been improving the site a lot over the past few months (and more is coming!).
If you haven't visited the page recently, it's worth checking out again: https://www.gutenberg.org/
Perhaps you can find the information you are looking for there.
However if you plan on scraping or otherwise hitting them with a ton of traffic, consider at least to donate a good amount for the traffic you cause them. It ain't free after all.
> All Project Gutenberg metadata are available digitally in the XML/RDF format. This is updated daily (other than the legacy format mentioned below). Please use one of these files as input to a database or other tools you may be developing, instead of crawling or roboting the website.
When I thought about Project Gutenberg I remembered that original brutalist non-design. The current site has been very tastefully updated but looks like it's still very accessible if you turn styles off. Great job!
I like the design but liked the previous design as well, it was unique and Craigslistish, you knew what website you were visiting just by looking at it.
I uploaded a PDF to archive.org that auto-OCRs with plenty of mistakes. I have found no way of updating the entire stack of documents produced. I wonder if Project Gutenberg is similar
OCR has improved a lot since then, but OCR is just step 1 of reading in text. They make a lot of errors (even now, especially on old worn out paper pages) and even if they didn't, one has to format the book, deal with footnotes, sidenotes, illustrations, etc. DP is very active, we will welcome you back with open arms :)
Thanks for the free work! Project Gutenberg is nice to have :).
On the site I noticed the library boxes have roughly a single extra line causing a scrollbar to appear and the last line to be chopped off https://i.imgur.com/PQ8T0qc.png is there an issues/bug portal to properly submit these kinds of things?
Huh that's interesting: 4.5 seconds for the TCP handshake and an additional 9.2 seconds for the TLS handshake. Is this some kind of captcha, since most bots would disconnect before that, so if you complete it once then it knows you're good? (Until the bots catch on of course, but so long as it works it's relatively unintrusive and not discriminatory against uncommon client software (that is, non-Chrome/ium).) The rest of the requests were lightning fast
Edit: welcome to your first comment after 9 years on HN btw, nice to have you here!
I think their site is just slow, potentially because more people than they are used to are trying to view it.
I was unable to load it initially (got an error from firefox) and had to re-attempt. Still slow if one forces a reload (shift-r, etc, to not use local cache).
we are having occasional lows in page speed performance due to LARGE amounts of bot traffic. full disclosure - we've not really been able to resolve this fully/well. Let us know if you have a good idea for how to deal with it
I'm only a small-scale sysadmin but the way that I understand the internet is that you send abuse notifications to the IP address block owner and, if it doesn't get resolved, you block. The whois/rdap database reveals which IPs all belong to the same hosting provider or ISP, so you can summarize that all to one list of IP addrs + timestamps per some time period
The ISP actually knows which subscriber is on that line, can send them notices, block them, terminate them... loads of things that you simply cannot do because you have no relation to this person. And frankly I wouldn't want to need to have a personal relation with every website that I visit; my ISP can reach me if there is anything relevant to continued use of the internet. From personal experience, when I was a teenager, the ISP cutting our household off after an abuse report was an effective way of stopping what I was doing
It’s effective against teenagers maybe. Not so much against Amazon, Meta or wherever botnet/crawler is coming out of China these days from up-and-coming AI companies.
wouldn't help, much of the traffic we've observed look closer to ddos patterns - IPs from all over the world, many different networks, each IP makes one request only, doesn't come back. highly distributed, no form of blocking would be effective except maybe captcha or proof of work.
The problem with this approach is that modern scrapers use hordes of residential proxies and quickly rotate through IP addresses which belong to ASes you get a lot of real traffic from. There's nothing you can do if the ISP won't take any action against the customer.
Worse than that - even if they would take action, you can't possibly orchestrate filing all of the complaints. It's a drown-in-quicksand problem, you can't fight quicksand one grain at a time.
anubis only works against lazy scrapers, and at a cost to your users. I'd prefer people not use it.
Bot traffic comes from machines that usually have a lot of idle cpu (since they're largely blocked on network IO as they scrape a bunch of sites in parallel), so they can trivially solve the anubis "proof of work" challenge, save the cookie, and then not solve it again for that site.
The only reason scrapers don't solve it is if the developers were too lazy to implement it... and modern scrapers also do, codeberg stopped using anubis because modern scrapers were updated to solve it.
The "proof of work" has to be easy or else people on old cell phones couldn't access your site (since an old android phone would start to overheat and throttle trying to solve a challenge that would take a modern server even several seconds), and it also consumes your cell-phone user's batteries, which is a really precious resource for them compared to the idle cpu on a server.
Please no. I'm a non-bot who gets stopped and turned away all the time by that menace. Anubis doesn't work without JS.
One of the things I give duckduckgo a lot of credit for is that while they're quick to interrupt me for a bot check (sometimes multiple times in a span of minutes) they'll let me identify ducks even on the most locked down browsers I use.
Occasionally, you misclassify a real user as a bot, and then your reputation is ruined forever.
The official Polish train schedules website did this recently, feeding incorrect departure and arrival times to IP addresses known for aggressive scraping, without taking CGNAT into account. People... have noticed[1].
How do you currently host everything? Your main web server should not be responsible for hosting content. All books should be hosted on mirrors, and clicking download should automatically select a mirror to download it from.
Furthermore:
* Make sure that all books are downloadable in bulk as torrents.
* Every day, generate a CSV file of all available books and their metadata. Distribute this so that bots and user clients can run queries locally, instead of using your search engine.
Have you considered having a detailed version history for each book (etext)? The process of submitting fixes to typos etc in books involves sending an email (https://www.gutenberg.org/help/errata.html) and although the last time I did this (2011) the fixes did get applied reasonably quickly (couple of days), it all felt a bit opaque. The version history could also include the project (usually PGDP correct?) the etext originated from; that way one would be able to compare against the actual page scans.
I have very mixed feelings about Standard Ebooks and would much prefer being able to use Project Gutenberg directly, but one good thing Standard Ebooks does is that every book has an associated git repository (on GitHub), so it's (in principle) possible to see a history of fixes to the text over time.
We're using git repos internally to keep history for each book. They existed on github for a while, but our implementation was awkward, and too big of project for the volunteer dev team. But it's likely that we'll evolve towards that.
It splits the community and number of possible volunteer hours for one. It also splits the canon into different versions. More projects fight for the attention attention (and possibly donations) of the audience.
There are lots of reasons it could be preferable to centralize. OTOH their mission is limited and some competition is healthy, if only to explore alternative ways to do things.
PG focuses on an accurate digital translation of the source material, sometimes hosting multiple different versions of the same text, and doing things like putting work into recreating the adverts at the back of some novels.
SE focuses less of preservation and more on making readers’ versions of the texts, like other publishing imprints. So there’s typography standardisation, a light-touch moderinisation of hyphenation and soundalike spelling, and things like author-wide collections of short fiction and poetry even if it didn’t previously exist.
Both are valuable, but they serve different segments.
Not the GP, but I also have mixed feelings about Standard Ebooks. They modernise texts for American readers. This means changing the punctuation, merging some words, altering the syntax, etc.
When I read an old novel, written two centuries ago in England, the little differences to modern English are part of the charm, and I certainly don't want any Americanism mixed in. For one of my favorite novels, The Forsyte saga, the author deliberately used some rare forms of words, which SE replaced with the mainstream forms.
You may already be aware, but SE marks all commits making those kinds of changes as '[Editorial]', so it is generally trivial to use their tooling to build your own high-quality ebook without any of the editorial changes.
SE editor in chief here. What you describe is incorrect. The only thing we do is very light sound-alike spelling modernization, like "to-night" -> "tonight". We do not do things like change from en-GB to en-US, replace old words with different modern words, or change text for "American readers", whatever that means. I have no idea where you got that impression.
I personally worked on the Forsyte saga. If you think something was done in error, please let us know and we'll be happy to fix it.
As long as you're taking suggestions, since many of the books are quite old, adding a publication date or date range to the search functionality might be nice. I personally would find it very useful since I have a tendency to look for things that are older than year _x_ when researching various things.
Hi for the past 20 years I have known about Project Gutenberg and I used to read a lot from it. One of the obstacle that I face is that there is no way to arrange the books in the order of their original publication.
Do you know of any such way.
Surely we can arrange the books by their release date on Gutenberg but it has long baffled me as it feels to me the most useless way of sorting the books.
Thank you for Project Gutenberg.
Great project. Are many of the books in a format that can easily be converted into audio? Is there a way to search for them, and information on what software your readers find useful for this purpose?
(Note: A lot of print media these days has switched to far-to-small font-sizes. Less of a problem for (zoomable) digital media, but for many that's still a barrier.)
A while back I attempted to extract the FF reader code to make it a front end to various non-web clients (email with pine key bindings etc)
I got it to a prototype level but then shelved it after having difficulty getting good results with various test datasets. Probably would make a fantastic ereader though
The ebook editions are very good for this. Most of the e-reader software provides all the amenities (bookmarks, highlighting, notes, control of margins, etc).
FWIW I absolutely love how 'no-frills' PG is compared to so much of the bloated, over-engineered, script-riddled web these days. Please don't ever change that!
While PG has probably gotten a lot of use and growth with the growth/maintreaming of the Internet since the 1990s, (TIL) it started back in 1971:
> Michael S. Hart began Project Gutenberg in 1971 with the digitization of the United States Declaration of Independence.[5] Hart, a student at the University of Illinois, obtained access to a Xerox Sigma V mainframe computer in the university's Materials Research Lab. […] This computer was one of the 15 nodes on ARPANET, the computer network that would become the Internet. Hart believed one day the general public would be able to access computers and decided to make works of literature available in electronic form for free. […]
In what way? And from what sources? (Wikipedia as a tertiary source is supposed to be a summary of information present in reliable secondary sources — see for instance https://en.wikipedia.org/wiki/Wikipedia:Based_upon. So if the information on the Wikipedia article is incomplete or out of date, where is the correct information available?)
"Project Gutenberg began in 1971 when Michael Hart was given an operator’s account with $100,000,000 of computer time in it by the operators of the Xerox Sigma V mainframe at the Materials Research Lab at the University of Illinois."
I'm surprised no eBook Reader vendor has a Project Gutenberg "Store." Where you can just browse Gutenberg, find a book, and just grab it down to the reader. Instead, they either are actively hostile (Kindle), or require the use of Calibre (which itself is good, it is just the friction).
Personally I find the formatting used by the Gutenberg one to be a lot nicer/easier to read, despite (or perhaps because of) being simpler, more plain.
At least for the first few pages of content that I looked at on both versions.
To be precise, the vast majority of SE is from Gutenberg, but we also source from Faded Page, Gutenberg Australia, Wikisource and occasionally do our own transcriptions.
e-book app Gutebooks (in addition to their audio app), but it seems to have been deprecated (I'm no longer able to connect to the server on my copy (which I only got 'cause there was an in-app purchase to fund Project Librivox).
FWIW, Barnes & Noble has been plundering the public domain using a book composition/keying house in the Philippines to make their public domain books which they make available in their stores --- Amazon apparently has a similar setup for the Kindle Store:
the way I see it PG is a labor of love. Bit odd if Barnes & Noble or whoever piggyback off it. But in the end - the more people read the books, the better.
It is a public good, and it would be appropos if corporations would support it directly rather than work at cross-purposes to it.
If Amazon is going to sell public domain texts, then it would make sense to source them from PG, and fund some money from those sales to the non-profit, similarly, they could then funnel reports of typos to PG for review and correction (it was a bit of a struggle the last time I tried to get a text corrected, and the project founder/director actually stepped in on my behalf).
>Barnes & Noble has been plundering the public domain using a book composition/keying house in the Philippines to make their public domain books which they make available in their stores
Why is it 'plundering' for B&N to print physical books, transport them to their brick-and-mortar stores to sell? There are real costs associated to doing so. It would not have zero cost for me to print and bind a copy myself at home.
In a similar vein to cobbzilla, I have a couple of family members (and to a lesser extent myself too) who would be keen on such an app for their iOS devices if you ever need some testers for that :)
(iPhones 15 Pro, 11 Pro, SE-2nd; and an iPad of some kind)
If you don’t strip the Project Gutenberg license from the book text (leaving only the book text, which no-one disputes is public domain and freely distributable), you are required to give “pay a royalty fee of 20% of the gross profits you derive from the use of Project Gutenberg-tm works calculated using the method you already use to calculate your applicable taxes”
[Way back in the early days of the iPhone, I sold a book reading app which was backed directly by Project Gutenberg texts, called “Eucalyptus”. I sent 20% of the gross profits to PG - which was never less than very supportive of the app - and felt good about doing so.]
I've found that the larger open-weight AI models do a great job of explaining the old non-fiction content on PG, particularly magazine articles which are a good size for the AI to handle. It breaks down the long wall-of-text paragraphs for you and explains all the historically relevant background that would've been assumed to be known back in the day.
If you ask it to assess the relevance of the text in the present day it will also do that very nicely, highlighting the places where the text shows old-fashioned viewpoints that would be sharply criticized today.
so maybe Karpathy has a point that LLM-assisted reading should be a thing. Would be cool if that worked on E-Reader screens as well. Maybe when the browsers on E-Readers become good enough ...
Project Gutenberg had (has?) a tendency toward plaintext that always put me off. (And it has been over a decade I'm sure since I explored the site—so I am no doubt now misinformed.)
I like a styled formatted book—would prefer PDFs. (I know, not a popular format apparently.)
I like the idea of Project Gutenberg but guess I found book scans on archive.org my preference.
My go-to example is Lewis Carroll's "Through the Looking Glass" with the fantastic art of John Tenniel and Carroll's sometimes creative formatting of the prose…
I see they (Project Gutenberg) have ePub now, which can be good if well done.
(If not well done it can be a kind of mess. Re-flowable "HTML", paginated… Anyone ever try to print a long web page and did you enjoy the result? Perhaps that is as much on the ePub reader though.)
As others here have mentioned, https://standardebooks.org/ is excellent and my understanding is that they use Gutenberg books as a source for theirs but done up much nicer.
We're supporting EPUB3 for the vast majority of books! At the same time we also have a "Plain Text" version for each as in a sense it's the most robust. PdFs are in the works!
That's cool. I'll have to read up on EPUB3—I'm not familiar with it.
(I worked on iBooks for the Mac like 15 years ago—it's where I got to dive into the ePub format. A lot has changed in the standard since I am sure.)
EDIT: looks like EPUB3 has a "paginated" mode as well as more sophisticated layout tags.
Also appears to have support for ruby and vertical writing modes. This was not yet supported in WebKit when I worked on iBooks. Somehow, this white guy from Kansas (who knows no language other than English) got tapped to implement the vertical TOC for Asian languages. Also tasked with annotating the ePUB pages to display (also vertical) ruby text…
I love, love, looove the fact that I can have a book's html version on project gutenberg bookmarked and continue to read across devices without ever having to login. I use the browser's inbuilt capability extensively to enhance my reading experience (fonts, backgrounds, text to speech, print formatting, share snippets). None of this is a good experience with pdf, epub or any other format.
I've read more (meaningful) text on PG than any other digital platform. Huge fan. Thanks for all the work and for keeping it clean and free
As a Kindle user, I still miss the old version of the site. The new one looks great on normal desktop, but the old one was simple enough to load and directly download books on the device's built-in browser.
• On the one hand, E Ink devices have a fairly known set of limitations, and it would be ridiculous for me to expect them to render the whole web well.
• On the other hand, it's good for website designs to consider the kind of devices employed by their users. Using a Kindle to access Gutenberg is likely less of an edge case than it would be for other sites, so it's worth the extra design work.
(Keep in mind that -- given my sibling comment -- this is all theoretical. The latest iteration of Gutenberg's site is much better than the previous version)
Every day you'll get much more than you're bargaining for, right into your feed or inbox. Easy download books you're interested in and put them on your Kindle.
I'm slightly curious how PG handles heavily illustrated books. I've downloaded some years ago, and the quality of the illustrations was always pretty poor. Has it been improved lately? What's the QA like for illustrations?
Nowadays we depend on scans from Internet Archive, Hathitrust, and other sources. Some scans are better than others. Bear in mind that our illustrations need to be in the public domain and usually from the same edition as the text. https://www.gutenberg.org/help/errata.html
Nice to see so much appreciation for what we do. (I'm the new-ish executive director.) Any wikipedians reading this, the article about PG is... aging. Last I looked, it said we offered Plucker files. @Jseiko has done some nice work.
I remember printing out project Gutenberg books in the mid-90s, four regular pages to an A4 page, double-sided on my inkjet. I had a background in typography, so I made it work.
Any yes, the text needed a lot of processing to make it right.
Now, in my early fifties and with declining eyesight, that's out of reach now.
that's cool! one of my "pet-ideas" is actually to make an AI-agent that does all that typographical work for any PG book to make it nicely printable without any manual labor whatsoever. Maybe that's doable now ...
That is doable. Most of my work was regexp and repetitive stuff. And the typograhpy stuff is achievable with the current state of the art models. Not that I remember what I did, it was 30 years ago.
good question. first though - maybe some bot has downloaded it often for whatever reasons and our systems didn't detect it as bot traffic. just a guess.
Looks like the top downloaded book yesterday[0] was Concrete Construction: Methods and Costs by Gillette and Hill.[1] Beat out Moby Dick, Count of Monte Cristo, Frankenstien, Romeo and Juliet, and others.
> 23644 downloads in the last 30 days.
I wonder if this is bot behavior? 23k downloads feels like a lot?
For context, here is the first paragraph of the book's preface:
How best to perform construction work and what it will cost for materials, labor, plant and general expenses are matters of vital interest to engineers and contractors. This book is a treatise on the methods and cost of concrete construction. No attempt has been made to present the subject of cement testing which is already covered by Mr. W. Purves Taylor's excellent book, nor to discuss the physical properties of cements and concrete, as they are discussed by Falk and by Sabin, nor to consider reinforced concrete design as do Turneaure and Maurer or Buel and Hill, nor to present a general treatise on cements, mortars and concrete construction like that of Reid or of Taylor and Thompson. On the contrary, the authors have handled the subject of concrete construction solely from the viewpoint of the builder of concrete structures. By doing this they have been able to crowd a great amount of detailed information on methods and costs of concrete construction into a volume of moderate size.
just heard back that the server provider has been doing a security update. Maybe you were one of the users that got unlucky as a result... maybe try later if still interested
Not a recommendation per se but I used to use Amphetype on Gutenberg texts to practise touch-typing. There's something about writing out a book that hits differently to reading it. You skip less, odd parts stick with you.
I think the last one I tried was The Island of Dr Moreau.
Worth mentioning the Project Gutenberg ZIMs. You can download the entire ENglish Gutenberg corpus for about 60GB (English Wikipedia ZIM complete with images is ~120GB):
Project Gutenberg is a treasure trove, though many technical details defy automatic typesetting of its books. Standard Ebooks takes consistency to an unbelievable level. My post compares various sources of public domain books with an eye on typesetting:
From Italy, https://www.gutenberg.org/ gives a 404 error and https://gutenberg.org/ opens a very official-looking page stating "police notice. This site is under judicial seizure" and references a sentence number: "criminal proceedings 52127/20 R.N.R.I. tribunal of Rome"
Any idea what's happening?
I thought PG published public domain books...
PG works based on US copyright law. And as I understand it that's also 70 years after author/translator death.
My gut feeling is that if anyone tried hard enough this ban could probably get lifted
I asked Claude to research the background story:
"In May 2020, the Court of Rome ordered Italian ISPs to seize/block a list of domains as part of a criminal case (the 52127/20 R.N.R. you're seeing) targeting sites and Telegram channels distributing pirated newspapers and magazines. 28 domains were on the list, and Project Gutenberg got thrown in alongside the actual pirate sites."
apparently this situation hasn't been resolved yet
The best thing I ever did for my father was to buy him a kindle and an access point and show him how to use Project Gutenberg to get books. He loved the old writings (he being a GED holder who was in the Navy during Korea yet had read the entire Harvard Classics). He had a special rolled up towel he used to prop it on his lap in his favorite chair and he read and read and read. When he passed he was reading "Legends of the Jews" from 1931.
I had some small e-correspondence with Michael S. Hart back in the 90's as well, and made a few modest contributions to the project, which made my English major undergraduate heart swell with pride and joy.
I guess this is only to say that PG is special to me for these reasons, and I am glad to see it still thriving. <3
this is so great to hear! Distributed proofreaders (the org that actually does transcriptions) is still looking for volunteer should you feel the urge/inclination :)
https://www.pgdp.net
If you like Project Gutenberg, the closest analog for music is IMSLP, the Petrucci Music Library (imslp.org) — over 855,000 public-domain scores maintained by volunteers, with the same labor-of-love energy and the same perpetual scan-quality and copyright-jurisdiction headaches. Same ethos of "the works belong to humanity, not a storefront." Worth a bookmark for the musicians on HN.
Well, mainly in the fact that Anna's has several orders of magnitude more books, and includes research publications and more, ah, contemporary materials to boot.
Project Guttenburg was my first introduction to the foss ethos. Well I suppose there was Wikipedia, but project Guttenburg really spoke to me. This was probably around 2003? So I'm glad to see it still going strong.
All the books should be there. I understand that current
society has restrictions, what with near infinite copyright
and other shenanigans - but I don't see any of these as
reason to hide information from mankind. Eventually we'll
free all the information. Remuneration will have to occur
in other ways than the current status quo.
[OP] JSeiko | 26 days ago
Falimonda | 26 days ago
Keep up the good work!
[OP] JSeiko | 26 days ago
Falimonda | 26 days ago
I've since discontinued hosting it, but happy to add you all and merge into an official PG offering: https://www.reddit.com/r/SideProject/s/VtYKxjrMme
Falimonda | 26 days ago
https://x.com/abal_ai
samcollins | 26 days ago
(I can’t quite tell if that’s an egregious abuse of the site or you’re perfectly fine to share without human eye balls hitting your www?)
kay_o | 26 days ago
Don't hit the site with agent. The section furtherst bottom machine readable.
[OP] JSeiko | 26 days ago
ancientcatz | 26 days ago
gluejar | 26 days ago
jzs | 26 days ago
https://www.gutenberg.org/ebooks/offline_catalogs.html
Perhaps you can find the information you are looking for there.
However if you plan on scraping or otherwise hitting them with a ton of traffic, consider at least to donate a good amount for the traffic you cause them. It ain't free after all.
[OP] JSeiko | 26 days ago
jimnotgym | 26 days ago
samcollins | 26 days ago
> All Project Gutenberg metadata are available digitally in the XML/RDF format. This is updated daily (other than the legacy format mentioned below). Please use one of these files as input to a database or other tools you may be developing, instead of crawling or roboting the website.
And strongly consider a donation! (My addition)
https://www.gutenberg.org/ebooks/offline_catalogs.html#the-p...
gluejar | 26 days ago
dredmorbius | 26 days ago
xrd | 26 days ago
ExtremisAndy | 26 days ago
[OP] JSeiko | 26 days ago
excitednumber | 26 days ago
shuvrojit | 26 days ago
[OP] JSeiko | 26 days ago
autocat3 and gutenbergsite are repos responsible for generating gutenberg.org
smallnix | 26 days ago
[OP] JSeiko | 26 days ago
[OP] JSeiko | 26 days ago
smallnix | 25 days ago
BiraIgnacio | 26 days ago
TimorousBestie | 26 days ago
[OP] JSeiko | 26 days ago
TimorousBestie | 26 days ago
jefurii | 26 days ago
[OP] JSeiko | 26 days ago
Wistar | 26 days ago
ok_dad | 26 days ago
Less than three is a classic!
agys | 26 days ago
s2 < <3
ricardonunez | 26 days ago
freedomben | 26 days ago
lapetitejort | 26 days ago
gluejar | 26 days ago
jfengel | 26 days ago
tangledhelix | 25 days ago
zamadatix | 26 days ago
On the site I noticed the library boxes have roughly a single extra line causing a scrollbar to appear and the last line to be chopped off https://i.imgur.com/PQ8T0qc.png is there an issues/bug portal to properly submit these kinds of things?
[OP] JSeiko | 26 days ago
lucb1e | 26 days ago
Edit: welcome to your first comment after 9 years on HN btw, nice to have you here!
codys | 26 days ago
I was unable to load it initially (got an error from firefox) and had to re-attempt. Still slow if one forces a reload (shift-r, etc, to not use local cache).
[OP] JSeiko | 26 days ago
TurdF3rguson | 26 days ago
lucb1e | 26 days ago
The ISP actually knows which subscriber is on that line, can send them notices, block them, terminate them... loads of things that you simply cannot do because you have no relation to this person. And frankly I wouldn't want to need to have a personal relation with every website that I visit; my ISP can reach me if there is anything relevant to continued use of the internet. From personal experience, when I was a teenager, the ISP cutting our household off after an abuse report was an effective way of stopping what I was doing
Jolter | 26 days ago
tonetegeatinst | 26 days ago
tangledhelix | 26 days ago
miki123211 | 25 days ago
tangledhelix | 25 days ago
gropo | 26 days ago
I have about 50k of the books, I would have used a torrent of just the txt files if it was prominent.
dimava | 26 days ago
You could have seen it on some websites already
https://anubis.techaro.lol/
TheDong | 26 days ago
Bot traffic comes from machines that usually have a lot of idle cpu (since they're largely blocked on network IO as they scrape a bunch of sites in parallel), so they can trivially solve the anubis "proof of work" challenge, save the cookie, and then not solve it again for that site.
The only reason scrapers don't solve it is if the developers were too lazy to implement it... and modern scrapers also do, codeberg stopped using anubis because modern scrapers were updated to solve it.
The "proof of work" has to be easy or else people on old cell phones couldn't access your site (since an old android phone would start to overheat and throttle trying to solve a challenge that would take a modern server even several seconds), and it also consumes your cell-phone user's batteries, which is a really precious resource for them compared to the idle cpu on a server.
autoexec | 26 days ago
One of the things I give duckduckgo a lot of credit for is that while they're quick to interrupt me for a bot check (sometimes multiple times in a span of minutes) they'll let me identify ducks even on the most locked down browsers I use.
jimnotgym | 26 days ago
miki123211 | 25 days ago
Occasionally, you misclassify a real user as a bot, and then your reputation is ruined forever.
The official Polish train schedules website did this recently, feeding incorrect departure and arrival times to IP addresses known for aggressive scraping, without taking CGNAT into account. People... have noticed[1].
[1] (Polish) https://zaufanatrzeciastrona.pl/post/kto-i-dlaczego-losuje-w...
uyzstvqs | 25 days ago
Furthermore:
* Make sure that all books are downloadable in bulk as torrents.
* Every day, generate a CSV file of all available books and their metadata. Distribute this so that bots and user clients can run queries locally, instead of using your search engine.
svat | 26 days ago
I have very mixed feelings about Standard Ebooks and would much prefer being able to use Project Gutenberg directly, but one good thing Standard Ebooks does is that every book has an associated git repository (on GitHub), so it's (in principle) possible to see a history of fixes to the text over time.
[OP] JSeiko | 26 days ago
[OP] JSeiko | 26 days ago
gluejar | 26 days ago
marcprux | 26 days ago
Why?
a2800276 | 26 days ago
There are lots of reasons it could be preferable to centralize. OTOH their mission is limited and some competition is healthy, if only to explore alternative ways to do things.
robin_reala | 26 days ago
PG focuses on an accurate digital translation of the source material, sometimes hosting multiple different versions of the same text, and doing things like putting work into recreating the adverts at the back of some novels.
SE focuses less of preservation and more on making readers’ versions of the texts, like other publishing imprints. So there’s typography standardisation, a light-touch moderinisation of hyphenation and soundalike spelling, and things like author-wide collections of short fiction and poetry even if it didn’t previously exist.
Both are valuable, but they serve different segments.
idoubtit | 26 days ago
When I read an old novel, written two centuries ago in England, the little differences to modern English are part of the charm, and I certainly don't want any Americanism mixed in. For one of my favorite novels, The Forsyte saga, the author deliberately used some rare forms of words, which SE replaced with the mainstream forms.
AdamN | 26 days ago
phaedrix | 25 days ago
The only thing they are is truly, truly wonderful.
jcurtis | 26 days ago
acabal | 25 days ago
I personally worked on the Forsyte saga. If you think something was done in error, please let us know and we'll be happy to fix it.
natex | 25 days ago
Curious. Why even bother?
0x0203 | 26 days ago
Thanks for all the effort put into the site!
Guestmodinfo | 26 days ago
8bitsrule | 26 days ago
(Note: A lot of print media these days has switched to far-to-small font-sizes. Less of a problem for (zoomable) digital media, but for many that's still a barrier.)
OfflineSergio | 25 days ago
tangledhelix | 25 days ago
human-read: https://www.gutenberg.org/browse/categories/1
computer-generated: https://www.gutenberg.org/browse/categories/2
IIRC many of the human-generated ones come from LibriVox, many of the computer-generated ones came from a collaboration with Microsoft.
samwho | 26 days ago
shevy-java | 25 days ago
Also by the way I just searched for 3d printing and found nothing. Either there are no books, or the search query makes things too complicated, IMO.
eulerpoolapi | 25 days ago
SwampertX | 25 days ago
drzaiusx11 | 25 days ago
I got it to a prototype level but then shelved it after having difficulty getting good results with various test datasets. Probably would make a fantastic ereader though
tangledhelix | 25 days ago
windowliker | 25 days ago
taubek | 26 days ago
seizethecheese | 26 days ago
[OP] JSeiko | 26 days ago
throw0101c | 26 days ago
> Michael S. Hart began Project Gutenberg in 1971 with the digitization of the United States Declaration of Independence.[5] Hart, a student at the University of Illinois, obtained access to a Xerox Sigma V mainframe computer in the university's Materials Research Lab. […] This computer was one of the 15 nodes on ARPANET, the computer network that would become the Internet. Hart believed one day the general public would be able to access computers and decided to make works of literature available in electronic form for free. […]
* https://en.wikipedia.org/wiki/Project_Gutenberg
gluejar | 26 days ago
svat | 26 days ago
[OP] JSeiko | 26 days ago
mcdonje | 26 days ago
aksss | 26 days ago
https://www.gutenberg.org/about/background/history_and_philo...
solarity_studio | 26 days ago
mowmiatlas | 26 days ago
[OP] JSeiko | 26 days ago
OfflineSergio | 25 days ago
Someone1234 | 26 days ago
[OP] JSeiko | 26 days ago
but yes, generally I agree with your point. Library of 75k books seems pretty valuable to have direct access to.
GaryBluto | 26 days ago
[OP] JSeiko | 26 days ago
WillAdams | 26 days ago
https://play.google.com/store/apps/details?id=biz.bookdesign...
should ~~be~~ EDIT have been ENDEDIT opensource --- it does at least work to support Project Librivox (or at least that's my understanding)
Seems to no longer be available (see below)
[OP] JSeiko | 26 days ago
WillAdams | 26 days ago
dgellow | 26 days ago
If you mean epub reader software Calibre and a bunch of others exist since pretty much the beginning of epub
horsawlarway | 26 days ago
Technically, I can also just directly pull the epub from Project Gutenberg, but sometimes the formatting leaves a lot to be desired.
Once you get an e-reader that runs a semi-capable OS (ex - stock android, even an older version), it's hard to go back to something like a kindle.
[OP] JSeiko | 26 days ago
everybodyknows | 26 days ago
https://www.gutenberg.org/cache/epub/1513/pg1513-images.html
https://standardebooks.org/ebooks/william-shakespeare/romeo-...
Each has its particular advantages relative to the other ...
svat | 26 days ago
Also one should probably compare the former to the single-page version on standardebooks: https://standardebooks.org/ebooks/william-shakespeare/romeo-...
swores | 26 days ago
At least for the first few pages of content that I looked at on both versions.
robin_reala | 26 days ago
cstever | 26 days ago
WillAdams | 26 days ago
https://librivox.org/
e-book app Gutebooks (in addition to their audio app), but it seems to have been deprecated (I'm no longer able to connect to the server on my copy (which I only got 'cause there was an in-app purchase to fund Project Librivox).
FWIW, Barnes & Noble has been plundering the public domain using a book composition/keying house in the Philippines to make their public domain books which they make available in their stores --- Amazon apparently has a similar setup for the Kindle Store:
https://www.amazon.com/Public-Domain-Books-Kindle-Store/s?k=...
Rather a shame that PG didn't monetize by putting their books up there pre-emptively.
[OP] JSeiko | 26 days ago
WillAdams | 26 days ago
If Amazon is going to sell public domain texts, then it would make sense to source them from PG, and fund some money from those sales to the non-profit, similarly, they could then funnel reports of typos to PG for review and correction (it was a bit of a struggle the last time I tried to get a text corrected, and the project founder/director actually stepped in on my behalf).
[OP] JSeiko | 26 days ago
WillAdams | 26 days ago
dessimus | 26 days ago
Why is it 'plundering' for B&N to print physical books, transport them to their brick-and-mortar stores to sell? There are real costs associated to doing so. It would not have zero cost for me to print and bind a copy myself at home.
gman83 | 26 days ago
cobbzilla | 26 days ago
swores | 26 days ago
(iPhones 15 Pro, 11 Pro, SE-2nd; and an iPad of some kind)
daveoc64 | 26 days ago
moichael | 25 days ago
jrmg | 26 days ago
https://www.gutenberg.org/policy/license.html
[Way back in the early days of the iPhone, I sold a book reading app which was backed directly by Project Gutenberg texts, called “Eucalyptus”. I sent 20% of the gross profits to PG - which was never less than very supportive of the app - and felt good about doing so.]
BHSPitMonkey | 26 days ago
https://koreader.rocks/
brcmthrowaway | 26 days ago
[OP] JSeiko | 26 days ago
could be a trick to ease that fear :D
zozbot234 | 26 days ago
If you ask it to assess the relevance of the text in the present day it will also do that very nicely, highlighting the places where the text shows old-fashioned viewpoints that would be sharply criticized today.
[OP] JSeiko | 26 days ago
aronhegedus | 26 days ago
[OP] JSeiko | 26 days ago
JKCalhoun | 26 days ago
I like a styled formatted book—would prefer PDFs. (I know, not a popular format apparently.)
I like the idea of Project Gutenberg but guess I found book scans on archive.org my preference.
My go-to example is Lewis Carroll's "Through the Looking Glass" with the fantastic art of John Tenniel and Carroll's sometimes creative formatting of the prose…
I see they (Project Gutenberg) have ePub now, which can be good if well done.
(If not well done it can be a kind of mess. Re-flowable "HTML", paginated… Anyone ever try to print a long web page and did you enjoy the result? Perhaps that is as much on the ePub reader though.)
graemep | 26 days ago
RattlesnakeJake | 26 days ago
JLO64 | 26 days ago
dempedempe | 26 days ago
everybodyknows | 26 days ago
jiffygist | 26 days ago
[OP] JSeiko | 26 days ago
JKCalhoun | 26 days ago
(I worked on iBooks for the Mac like 15 years ago—it's where I got to dive into the ePub format. A lot has changed in the standard since I am sure.)
EDIT: looks like EPUB3 has a "paginated" mode as well as more sophisticated layout tags.
Also appears to have support for ruby and vertical writing modes. This was not yet supported in WebKit when I worked on iBooks. Somehow, this white guy from Kansas (who knows no language other than English) got tapped to implement the vertical TOC for Asian languages. Also tasked with annotating the ePUB pages to display (also vertical) ruby text…
the_af | 26 days ago
[OP] JSeiko | 26 days ago
lxgr | 26 days ago
As an output format it’s ok, but as an input format, it’s almost as bad as PDF.
gluejar | 26 days ago
skrtskrt | 26 days ago
gofreddygo | 26 days ago
I've read more (meaningful) text on PG than any other digital platform. Huge fan. Thanks for all the work and for keeping it clean and free
[OP] JSeiko | 25 days ago
iberator | 26 days ago
RattlesnakeJake | 26 days ago
[OP] JSeiko | 26 days ago
RattlesnakeJake | 26 days ago
The previous version of the site had two major flaws:
1. The search bar had been removed from the top of the page, and hidden behind a "Click here to search" (or similar) link partway down the page
2. Once you opened that page, the coloring of the site was so washed out on e-ink that the text input was hard to find.
Thanks for fixing it!
[OP] JSeiko | 26 days ago
bitigchi | 26 days ago
graemep | 26 days ago
You can download books in most browsers. I know Amazon have done things to make life difficult for other stores in the past.
RattlesnakeJake | 26 days ago
• On the one hand, E Ink devices have a fairly known set of limitations, and it would be ridiculous for me to expect them to render the whole web well.
• On the other hand, it's good for website designs to consider the kind of devices employed by their users. Using a Kindle to access Gutenberg is likely less of an edge case than it would be for other sites, so it's worth the extra design work.
(Keep in mind that -- given my sibling comment -- this is all theoretical. The latest iteration of Gutenberg's site is much better than the previous version)
carlosjobim | 26 days ago
https://www.gutenberg.org/ebooks/feeds.html
Every day you'll get much more than you're bargaining for, right into your feed or inbox. Easy download books you're interested in and put them on your Kindle.
WillAdams | 26 days ago
https://onlinebooks.library.upenn.edu/new.html
ndr42 | 26 days ago
[OP] JSeiko | 26 days ago
debo_ | 26 days ago
AndrewStephens | 26 days ago
[OP] JSeiko | 26 days ago
oidar | 26 days ago
gluejar | 26 days ago
gluejar | 26 days ago
kreyenborgi | 26 days ago
https://www.fadedpage.com/ from Canada I think
https://runeberg.org/ from Sweden
Arcorann | 26 days ago
bryankaplan | 26 days ago
[OP] JSeiko | 26 days ago
[OP] JSeiko | 26 days ago
smilespray | 26 days ago
Any yes, the text needed a lot of processing to make it right.
Now, in my early fifties and with declining eyesight, that's out of reach now.
Thanks for sticking with the project!
[OP] JSeiko | 26 days ago
smilespray | 26 days ago
[OP] JSeiko | 26 days ago
kgwxd | 26 days ago
[OP] JSeiko | 26 days ago
ssgodderidge | 26 days ago
> 23644 downloads in the last 30 days.
I wonder if this is bot behavior? 23k downloads feels like a lot?
[0] https://www.gutenberg.org/browse/scores/top [1] https://www.gutenberg.org/ebooks/24855
[OP] JSeiko | 26 days ago
sovietswag | 26 days ago
why_at | 26 days ago
tmoertel | 26 days ago
How best to perform construction work and what it will cost for materials, labor, plant and general expenses are matters of vital interest to engineers and contractors. This book is a treatise on the methods and cost of concrete construction. No attempt has been made to present the subject of cement testing which is already covered by Mr. W. Purves Taylor's excellent book, nor to discuss the physical properties of cements and concrete, as they are discussed by Falk and by Sabin, nor to consider reinforced concrete design as do Turneaure and Maurer or Buel and Hill, nor to present a general treatise on cements, mortars and concrete construction like that of Reid or of Taylor and Thompson. On the contrary, the authors have handled the subject of concrete construction solely from the viewpoint of the builder of concrete structures. By doing this they have been able to crowd a great amount of detailed information on methods and costs of concrete construction into a volume of moderate size.
[OP] JSeiko | 25 days ago
autoexec | 26 days ago
monegator | 26 days ago
[OP] JSeiko | 26 days ago
[OP] JSeiko | 26 days ago
jwpapi | 26 days ago
klondike_klive | 26 days ago
jwpapi | 26 days ago
[OP] JSeiko | 26 days ago
I've heard good things. Also - Sherlock Holmes :)
BaseBaal | 26 days ago
https://www.gutenberg.org/ebooks/78684
fmajid | 26 days ago
https://ebookfoundation.org/openzim.html
cxr | 25 days ago
Timixx | 26 days ago
cold_tom | 26 days ago
tomjen3 | 26 days ago
I was visting the ruins of a monestary the other day, and one of the texts listed that it had a library of 320ish books.
I chucked because I have almost 200 books in my personal Kindle library, but I was wrong. I actually have 75000+ books, thanks to Project Gutenberg.
I just haven't downloaded them all yet.
thangalin | 26 days ago
https://dave.autonoma.ca/blog/2020/04/11/project-gutenberg-p...
elias1233 | 26 days ago
Myzel394 | 26 days ago
cosmos0072 | 26 days ago
Any idea what's happening? I thought PG published public domain books...
cosmos0072 | 26 days ago
Full story (in Italian) at https://www.wired.it/internet/web/2020/06/30/progetto-gutenb...
charonn0 | 26 days ago
amelius | 25 days ago
johndough | 26 days ago
[OP] JSeiko | 26 days ago
[OP] JSeiko | 26 days ago
apparently this situation hasn't been resolved yet
dgellow | 26 days ago
tangledhelix | 26 days ago
drummojg | 26 days ago
I had some small e-correspondence with Michael S. Hart back in the 90's as well, and made a few modest contributions to the project, which made my English major undergraduate heart swell with pride and joy.
I guess this is only to say that PG is special to me for these reasons, and I am glad to see it still thriving. <3
[OP] JSeiko | 26 days ago
j_bum | 26 days ago
mentalgear | 26 days ago
derekhdawson | 26 days ago
greenie_beans | 26 days ago
[OP] JSeiko | 26 days ago
gwerbret | 26 days ago
[OP] JSeiko | 26 days ago
gwerbret | 26 days ago
oxag3n | 26 days ago
[OP] JSeiko | 26 days ago
jdthedisciple | 26 days ago
vwkd | 26 days ago
https://projekt-gutenberg.org/authors/johann-wolfgang-von-go...
timonoko | 26 days ago
https://www-gutenberg-org.translate.goog/cache/epub/64099/pg...
benj111 | 25 days ago
I just looked at the history (https://www.gutenberg.org/cache/epub/60600/pg60600-images.ht...) and it dates back to the 70s. There was me thinking it was some new fangled web thing.
shevy-java | 25 days ago
[OP] JSeiko | 25 days ago
aymenfurter | 25 days ago
GeorgeTirebiter | 25 days ago
1vuio0pswjnm7 | 25 days ago
Good job