I second this. If you haven't watched 3blu1brown series on this it's excellent. These ideas are old and looking at LLMs like this is also relatively old. I'm happy to see people having these 'Aha' moments discovering this on their own!
The idea itself has been, of course, popularised well before the current wave of language models: https://en.wikipedia.org/wiki/Hutter_Prize for a version with money behind the promotion
To add, the Hutter prize was originally set in hopes that progress in compression will lead to improvements in AI. Ironically, it worked out the other way round -- the best modern text compression algorithms (measuring output size only, ignoring performance) are based on LLMs.
One could argue, however, whether some of the steps in ML progress were basically training models to compress/decompress text with not too much (prediction) error. Cross-feeding between the two tasks seems (to me) quite aligned with the idea of Hutter prize.
vrthra | 18 hours ago
Of course, and there is this awesome project (not mine) that actually uses gzip as a language model.
eBPF | 14 hours ago
Youtuber 3blue1brown has a good series on this (still in-progress at time of writing). The first video in the series: https://www.youtube.com/watch?v=l6DKRf-fAAM
mempko | an hour ago
I second this. If you haven't watched 3blu1brown series on this it's excellent. These ideas are old and looking at LLMs like this is also relatively old. I'm happy to see people having these 'Aha' moments discovering this on their own!
Here is Claude Shannon's (founder of information theory) paper about combining the idea of entropy and prediction. https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
Apparently he used his wife as a sort of LLM.
Here is a great paper called Language Modeling Is Compression (2023) https://arxiv.org/abs/2309.10668 by Grégoire Delétan
Repo from DeepMind https://github.com/google-deepmind/language_modeling_is_compression
k749gtnc9l3w | 18 hours ago
The idea itself has been, of course, popularised well before the current wave of language models: https://en.wikipedia.org/wiki/Hutter_Prize for a version with money behind the promotion
purplesyringa | 18 hours ago
To add, the Hutter prize was originally set in hopes that progress in compression will lead to improvements in AI. Ironically, it worked out the other way round -- the best modern text compression algorithms (measuring output size only, ignoring performance) are based on LLMs.
k749gtnc9l3w | 17 hours ago
One could argue, however, whether some of the steps in ML progress were basically training models to compress/decompress text with not too much (prediction) error. Cross-feeding between the two tasks seems (to me) quite aligned with the idea of Hutter prize.
silentbicycle | 4 hours ago
Of course. And "AI hallucinations" are artifacts from lossy compression, like JPEG artifacts confidently mis-modeling reality with statistical noise.
carlana | an hour ago
https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web