QotNews
Hacker News, Reddit, Lobsters, and Tildes articles rendered in reader mode.

Oils - Reviewing Our NLnet Grants After 4 Years

Source: oils.pub
42 points by andyc 21 days ago on lobsters | 3 comments

| blog | oils.pub

2026-06-01

We received our first NLnet grant in June 2022, and we've been awarded three more grants since then. Four years and four grants later, it's worth looking back to see what we did.

So I made some plots for this post!

Unfortunately, we won't finish the fourth grant. I wrapped it up early because of the family issues mentioned in March: A Pause in the Project. What happened is that more contributions led to a backlog of pull requests, I haven't been able to review them all.

I also share some high-level thoughts and feelings on the state of the project.

New Plots

In March 2022, just before we received the first grant, I published Oil Is Being Implemented "Middle Out".

It has a plot of our spec test metrics.

I wrote that the gray rectangle showed a "fallow period" in the development of OSH. OSH wasn't making progress as fast as it had, because I started working on YSH (then called the Oil language).

In other words, it was clear the project got too big, and I needed help! So what happened after we were funded, and more people helped?

OSH - Speed and Compatibility

I revived the code to generate that 2022 plot (shell, Python, and R scripts!). Here's the update:

Progress on OSH

OK cool, I wasn't expecting the story to look this clean!

  • The blue line and red line converged in early 2023, which means that the Python implementation and C++ implementation pass all the same tests. In other words, we finished the Python-to-C++ translation.

    • That means we can "just write Python", and we get a shell in fast, native code, with no dependencies. This is just like shells written in C.
    • We also finished the garbage collector in January 2023: Pictures of a Working Garbage Collector. This was a big achievement that doesn't appear on the plot. We now have something better than shells written in C: garbage collection means that our code is memory safe!

  • The slope of the converged lines increased. That means that we made faster progress on OSH once the translation was done.

    • The fallow period I mentioned is the part where the blue line was flatter — from mid-2020 to 2023.

So the funding from NLnet really helped OSH.

YSH - A Graceful Upgrade, and a New Language

But that's not all! After the first grant, contributors were excited about YSH, and I applied for another grant. (I also renamed the project in early 2023.)

Here's a similar plot of YSH progress:

Progress on YSH

Notice a few things:

  • YSH didn't really exist until after the NLnet funding. There was no red line at all, and I did not recommend using the pure Python version.
  • YSH was fully translated to native code in early 2024 — i.e. the red and blue line converged then, about a year after they converged for OSH.
  • The increased slope of the converged line shows that we made a lot of progress on YSH from mid-2023 to 2025. That's when most of the language was developed.

So these plots paint a clear and positive picture. The funding helped! Before making the plots, I wasn't sure how they would look.

Alpine Package Disagreements - 139 to 11

OK now let's look at what happened in the last year or so. The goal of the fourth grant was to stress test OSH, by running it as the only shell on a complete Linux system (Alpine Linux). I described that work in December:

Contributors wrote blog posts about some of the fixes, which I linked to in March:

We kept track of the work with this dashboard (made with Markdown and shell scripts):

Here's a visualization of that data (limited to the Alpine main repo):

Progress on regtest/aports in 2025

So we went from 139 disagreements building with OSH in August, to 11 in early December. (There was at least 1 disagreement fixed after the last measurement of 12.)

This was fantastic momentum, but there is a caveat. I merged many pull requests, but the backlog of unreviewed pull requests also grew.

And, in the same month, most of my free time evaporated.

Evaluations

Honestly, I was feeling a bit down about the project before writing this post. I sent more than one e-mail to NLnet explaining the early end of the grant, which were perhaps overly pessimistic.

But now I feel pretty good about what we did.

The way I framed it is that we met our goals for the grants, but that doesn't necessarily mean the overall project has been a success. So let's talk about the grants first, and then the project.

Did We Meet Our Goals for the Grants?

Yes, the plots basically show that, but they don't tell the whole story. I think the real story is that we did many hard things that could have failed, but ended up working.

  • We could have failed to speed up the code 30x-50x by translating typed Python to C++. But this did happen, in the first grant.
  • We could have failed to write a garbage collector. But we got it done, and it's been solid for a few years now.
  • The design of YSH could have failed, e.g. due to being constrained by compatibility. But that didn't happen:
    • When I started the project, many people thought upgrading shell was impossible! But now, multiple people written thousands of lines of YSH, including in professional contexts! Thanks to Will Clardy and many others for their code and feedback.
  • It could have been that nobody wanted to work on Oils for 50 euros an hour. But in fact I got an overwhelming number of responses. And the contributors we chose were very skilled — more skilled than me in significant respects.

So one reason I've been working on this project for 10 years is that things keep working. There have been several "wrong turns" and some backtracking, but we haven't gotten stuck on anything for long.

And I think there's basically no risk that we don't close the gap on the remaining 11 disagreements, as long as we have time to do it right.

Did the Project Meet Its Goals?

But even though we met these smaller goals, I think it's time to re-evaluate the project goals.

(It's important to say that my available time colors this whole discussion. I hope that situation will change, but it may not change soon.)

So let's go back to the slogan:

Our upgrade path from bash to a better language and runtime

We did in fact do what many people thought was impossible: implement a POSIX- and bash-compatible shell in a principled way; and then upgrade that to a new language YSH (with some caveats).

But, I am not sure that this is actually an upgrade path in practice. What I underestimated is that this strategy requires a lot of work for distros. And the foundations of most distros don't change a lot, for obvious reasons. Many foundational shell scripts are in maintenance mode, and the maintainers only understand parts of them.

On the other hand, I would say that e.g. systemd is evidence that the foundations can be changed, in a pretty radical way. And distros pulled systemd, apparently because it saved them a lot of work. But I can see that, right now, there is not a killer reason for distros to pull Oils. It's more like we would be pushing, and I think that's somewhat fundamental, without some big changes.

Even Bigger Goals

So I may no longer believe in the "upgrade path from bash" framing. Right now, it doesn't seem realistic.

But I still believe in the even more ambitious goals of the project:

  • "Compressing" many domain-specific languages into a general purpose shell language: Sketches of YSH Features (June 2023)

    • We made some progress on YSH as a host for DSLs, but not enough.
    • Why is this still interesting? When writing code becomes cheap, reading code becomes more valuable. And so far I haven't found anyone who likes to read a mish-mash of shell, awk, make, CMake, YAML — that is, Unix sludge and cloud sludge.

  • A distributed shell

    • We're running into this problem ourselves, e.g. with regtest/aports, our Soil CI, and making a release in the cloud. See the TODO list below.
    • The complement to a distributed shell is a distributed file system. As with J8 Notation, it helps to rethink things from the perspective of data rather than computation.
    • The most popular post on this blog was circulated as Kubernetes Is Our Generation's Multics (5 years ago, and 5 years after the project started!)

  • Headless shell / GUIs interfaces - Screenshots and Wiki

    • LLM web interfaces like Claude and Gemini look a lot like a shell to me. This fact hasn't been lost on many people, e.g. yosh by Filip Pizlo, and the Warp terminal. LLMs are the biggest thing to hit programming in a decade, and they are intricately entangled with Unix shell.

But I simply don't know how to make progress on those goals, with the time I have, and the resources we have. But I honestly have a lot of fun pursuing them.

Other Things I Wrote to NLnet

I made most of the points in this post in my mails to NLnet. Here are some other thoughts from those mails.

Project Management

We didn't find enough people to work on YSH itself. Aidan implemented essential parts of YSH, and provided crucial design feedback on features like closures, but we could have used more help.

One way you can see this is that we never solved the problem of making progress on OSH and YSH in parallel. For example, the fourth grant was all about OSH, and we didn't make much progress on YSH during that time.

The plots do show increased progress, so the grants did help. But I think the scope of the project was always too large.

Design Bottleneck / Decentralized FOSS Model

I also think I was a bit naive about what you can do in the FOSS model. I didn't pay enough attention to the project's organization. Prior to NLnet funding, I worked on the project myself for six years! There were other contributors, but nobody approaching full time.

Another way to say this is that, historically, I would be interested in working on something like Bell Labs Unix, more so than GNU/Linux. I'm interested in new system design ideas, just as much as principled implementations of existing systems.

Here's a related comment:

I think Linux has an enormous design debt to Bell Labs [2] ...

I actually wonder if LLM-generated code could be a sign of a breakdown. i.e. we have this big mess of open source code, which in theory should be very valuable, but some large proportion of new programmers can only understand it by throwing LLMs at it ...

In other words, I'd say that our collective software infrastructure was not in great shape, prior to the introduction of LLMs.

Conclusion

Since we are ending the 4th grant early, I thought it would be appropriate to publish my thoughts. We made a ton of progress over the last 4 years, but the project is "paused" for now. There will still be updates and releases — see the TODO list below — but it will be slow and on-demand.

That is, I hope people will show up on Zulip to chat, but I won't "push" the project for quite awhile.

Thank You

Thank you to everyone who contributed! Not just people who worked under the grant, but everyone else as well.

  • Jesse Hughes, Melvin Walls, CoffeeTableEspresso
  • Chris Watkins, Aidan Olsen, Samuel Hierholzer
  • Justin Pombrio, Koichi Murase
  • Andriy Sultanov, Bram Tertoolen, Gabriel Stang, Adejumo David Adewale

And thank you to NLnet!

  • Michiel Leenaars, Bob Goudriaan, Lwenn Bussière, Gerben, Layla Otter, and to everybody else who helped behind the scenes.

Please feel to ask questions or give feedback in the comments on Zulip!

Appendices

TODO List

Here's what I'm thinking about right now.

  • Address issues in the design of YSH try, from Will Clardy and Samuel Hierholzer.
  • I think we can make the Oils parser run under Python 3, for the resholve use case (Travis Everett).
  • I want to merge some of the pull requests, most of which are related to OSH. Changes with new tests should get priority.
  • Make another release.
  • Respond to some Github issues

Future Posts

Here are some topics I cut out of this post.

LLMs helped me with crappy old languages: VimScript and raw SQL

In early 2025, I wrote a YSH syntax definition in VimScript, which resulted in the doc Three Algorithms for YSH Syntax Highlighting.

Based on my previous experience with VimScript around 2008 or so, I can say that this wouldn't have happened without dozens of queries to Claude. (But you can see from the doc that it's not vibe-coded. Instead, I learned the language by writing a program, with tests.)

LLMs also helped me learn more SQL. In the past, I've used the R language for table manipulation — for example, the plots above were generated with ggplot2 (the grammar of graphics).

But for the table manipulation in regtest/aports, I used sqlite. I remember starting a similar project with raw SQL in 2014, and it was hard for me. So again I used dozens of queries to Claude to learn SQL. (I like the analysis in Against SQL.)

Stories About Personal Agency

Here's another excerpt from my comment above:

Maybe future programmers will actually want some control back - to build systems that are understandable, reliable and secure, use hardware in a reasonable way, etc.

So that comment is about personal agency, and that's basically what the Oils project is about:

  • I like the shell interface because I tell the computer what to do! Not the other way around. I don't get pop-up notifications and dark patterns while using shell.
  • I also like to have the source code of the software I run. And turning source into machine code invariably evolves thousands of lines of shell scripts (and related glue, aka Unix sludge).

So shell still very much aligns with my interests, and it aligns the future of computing. I hope I'll again have time to work on these ideas — perhaps from a different angle, and putting to use some of the things I learned about both engineering and project management.