This is border level insane (in a good way). I loved the super Mario Bros 3 example, unconventional but just what I needed to keep reading and I genuinely learned something about Nix that I had failed to grasp in many of my previous attempts at learning it...
Sometimes I'm at a shell, interactively transforming a dataset like curl $URL | transform1 | transform2 | filter, and debugging takes a long time since curl or transform1 takes a long time. So I cache it, and I'm left with:
cat intermediate.csv | transform2 | filter
After a few steps I have many such intermediate files, and a confusing shell history.
I wish my shell did smarter caching here, so I could just do:
curl $URL | transform1 | transform3 | filter
and have my shell know that it can reuse outputs.
Has anyone tried Nix, or similar, to get this transparent caching? Similar to this supermario example.
I know this isn't trivial, since $URL might change, and transform1 might be non-deterministic ,or have side-effects. Nix operates with the same constraints.
After all, my issue is ultimately a build problem.
\\ is a command that interprets \\ as pipe-operator, caches all intermediate output and replaces commands with cache when one is found.
The script is <100 lines of Python. Really the tricky bit is designing cache invalidation (I've keyed the cache by shell session) and just remembering to use it since it's not fully replacing bash's own pipe operator.
Apart from the non-determinism, the file size also makes a difference. The curl result could be a hundred megabytes and the filter could cut it down to just one line. You likely wouldn't want to cache that by default.
Maybe some cool "cached pipe" operator could work?
If I were to think about your idea:
It would be a shell that transforms every command into a Nix derivation.
If you used pipes, those are inputs to the derivation so they change.
The command itself and the argv (maybe the environment variables) are the inputs to calculate the hash? A neat idea I think worth exploring.
loige | a day ago
This is border level insane (in a good way). I loved the super Mario Bros 3 example, unconventional but just what I needed to keep reading and I genuinely learned something about Nix that I had failed to grasp in many of my previous attempts at learning it...
tuxes | a day ago
Sometimes I'm at a shell, interactively transforming a dataset like
curl $URL | transform1 | transform2 | filter, and debugging takes a long time sincecurlortransform1takes a long time. So I cache it, and I'm left with:cat intermediate.csv | transform2 | filterAfter a few steps I have many such intermediate files, and a confusing shell history.
I wish my shell did smarter caching here, so I could just do:
curl $URL | transform1 | transform3 | filterand have my shell know that it can reuse outputs.
Has anyone tried Nix, or similar, to get this transparent caching? Similar to this supermario example.
I know this isn't trivial, since $URL might change, and
transform1might be non-deterministic ,or have side-effects. Nix operates with the same constraints.After all, my issue is ultimately a build problem.
untitaker | 17 hours ago
I've had a script in my dotfiles for this purpose for quite a while, usage like this:
\\is a command that interprets\\as pipe-operator, caches all intermediate output and replaces commands with cache when one is found.The script is <100 lines of Python. Really the tricky bit is designing cache invalidation (I've keyed the cache by shell session) and just remembering to use it since it's not fully replacing bash's own pipe operator.
T6 | 15 hours ago
In case anyone else is curious, the script in question: https://codeberg.org/untitaker/dotfiles/src/branch/master/home/.scripts/%5C
fzakaria | 13 hours ago
Can I write about this maybe?
T6 | 11 hours ago
(Not mine, I just found it on their profile :).)
viraptor | 22 hours ago
Apart from the non-determinism, the file size also makes a difference. The curl result could be a hundred megabytes and the filter could cut it down to just one line. You likely wouldn't want to cache that by default.
Maybe some cool "cached pipe" operator could work?
vagos | 17 hours ago
You might be interested in our project, Incr!
It provides incremental execution for unmodified shell programs and correctly handles side-effects and most common forms of non-determinism.
fzakaria | a day ago
If I were to think about your idea: It would be a shell that transforms every command into a Nix derivation. If you used pipes, those are inputs to the derivation so they change.
The command itself and the argv (maybe the environment variables) are the inputs to calculate the hash? A neat idea I think worth exploring.