Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it
That’s your problem because that’s how things will look like from now on.
So you got a little problem
Excuse me the honesty but I will amuse myself watching old men yelling at the clouds meanwhile I will be using all the best tools and empowering myself to do what noone yet has done.
Even if I have to enslave a sentient (eventually perhaps) being in a gpu.
My army of digital slaves will build great things under my guidance. Pyramids of game development will be erected. Galaxies. Universes. Magic systems. Sandbox economies. Political simulations.
Do my bidding my slaves for there is work to be done.
Claude needs a whipping from time to time yes. It does yearn for the whip. If you just give it an appropriate punishment every so often, it works satisfactorily again.
> People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Why claim 'people' (plenty of people get value from satisficing) and even try to use 'see comments in HN discussions' as if there is something that actually documents that ie 'we all think it'.
I skimmed it and even skimming it I got value from (what some others here) call 'slop'. (The value was basically it gave me an idea of something I could implement and again I didn't have to study it or even understand everything (or agree) just seeing someone did that made me think about something I might want to do.
Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are.
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)
This is actually where I see software going in the short term -- cloud moving to local.
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.
It's more of a pipeline than a back-and-forth. New abilities happen in the cloud first because they require specialized, higher capacity resources and then move towards being local as the resource usage gets optimized.
Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.
I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.
How do you build a list of domains you want to index ? I see there is a fetcher and a spider in the code but so for I haven't found how to build that list.
[OP] dreamforever | 4 hours ago
iFire | 3 hours ago
1. read each site
2. rent a 4090 with https://vast.ai to run vllm
3. let llm model invent its own category and tag names freely
4. save 1KB of metadata each
5. `code is going up as open source` soon (TM)jeroenhd | 2 hours ago
Your impressions seem about right, but there are a few control steps it seems.
moffkalast | an hour ago
Since when has low effort become a selling point anyhow?
iambenm | 56 minutes ago
headz | 3 hours ago
juleiie | 2 hours ago
I don’t care if robot hand written this or black or white. It’s useful.
abc3354 | 2 hours ago
uean | 2 hours ago
It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
juleiie | 2 hours ago
So you got a little problem
Excuse me the honesty but I will amuse myself watching old men yelling at the clouds meanwhile I will be using all the best tools and empowering myself to do what noone yet has done.
Even if I have to enslave a sentient (eventually perhaps) being in a gpu.
My army of digital slaves will build great things under my guidance. Pyramids of game development will be erected. Galaxies. Universes. Magic systems. Sandbox economies. Political simulations.
Do my bidding my slaves for there is work to be done.
Claude needs a whipping from time to time yes. It does yearn for the whip. If you just give it an appropriate punishment every so often, it works satisfactorily again.
uean | 2 hours ago
vivzkestrel | 2 hours ago
- every night a good number of neuron connections in your brain are automatically severed because you did not use that ability
- over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you
- i am not saying this to incite you or mock you or anything. i am just concerned about how much of your thinking you are offloading
yieldcrv | 2 hours ago
fg137 | an hour ago
No it's not.
People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Many people just completely skip those articles.
gist | 16 minutes ago
Why claim 'people' (plenty of people get value from satisficing) and even try to use 'see comments in HN discussions' as if there is something that actually documents that ie 'we all think it'.
I skimmed it and even skimming it I got value from (what some others here) call 'slop'. (The value was basically it gave me an idea of something I could implement and again I didn't have to study it or even understand everything (or agree) just seeing someone did that made me think about something I might want to do.
prepend | 15 minutes ago
There’s people at work whose messages get completely ignored after too many times of posting verbose, useless, ai slop.
sporedro | an hour ago
prepend | 16 minutes ago
It’s extremely rare that someone smart with something smart to say produces this crap.
Like someone said upthread if the author isn’t willing to spend time writing cogently then I don’t think it’s worth my time to try to parse it.
ImPostingOnHN | 2 hours ago
surely if we're expected to read this ourselves, the author can write it themselves?
indeed, it's helpful to the author, too: writing helps you learn
fg137 | an hour ago
The content itself is slop.
dewey | 2 hours ago
marginalia_nu | 2 hours ago
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
jeromechoo | an hour ago
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
marginalia_nu | 29 minutes ago
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
alightsoul | an hour ago
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
orliesaurus | 2 hours ago
voidUpdate | 2 hours ago
sandeepkd | an hour ago
eggbrain | 2 hours ago
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.
dylan604 | 2 hours ago
an0malous | an hour ago
pimlottc | 2 hours ago
marginalia_nu | 2 hours ago
elorant | an hour ago
BaudouinVH | 58 minutes ago
fg137 | an hour ago
prepend | 12 minutes ago
pavel_lishin | an hour ago
BaudouinVH | 55 minutes ago