How AST-grep Rewrote Tree-sitter in Rust and Made It 30% Faster

91 points by herrington_d a day ago on hackernews | 15 comments

Reading the article is like climbing a load-bearing wall to quote their last phrase.

Pretty disappointing when you hit ⌘A, right-click, "Check for AI Content" and Pangram comes back with 100% AI... and on every section.

rayiner | 20 hours ago

I've seen "load bearing" so many times in AI now I can't remember whether we always used "load bearing wall" to refer to a structural interior wall, or whether it's a recent AI-ism.

hidd | 20 hours ago

Jerry, these are load-bearing walls! [1]

[1]: https://youtu.be/IbVmxkVC5kc?t=83

: thejazzman | 17 hours ago
It’s a load bearing poster.
https://amphetamem.es/meme/?id=the-simpsons_08_08_248&text=I...

trickypr | 22 hours ago

> incremental old-tree reuse were removed

Maybe I am missing what they meant here, but isn’t the entire point of tree-sitter that you can reuse old trees to make edit updates faster? It is seems quite probable that all the performance gains came from optimising for fixed files with no error recovery, at the cost of how ts is actually used.

: ivanjermakov | 22 hours ago
Made a tree-sitter powered text editor. Implementing incremental parsing improved performance from 5ms to 400us for small edits in a ~100LOC file. For many applications tree-sitter without incremental parsing makes no sense, but I guess it's not a problem for ast-grep.
: sweetgiorni | 21 hours ago
> isn’t the entire point of tree-sitter that you can reuse old trees to make edit updates faster?
I've used tree-sitter for a few projects, but I never touched the tree-reuse stuff. The value of tree-sitter came from the fact that grammars compile to a .so and can be easily distributed through package managers. Adding support for a new grammar in ast-grep might just mean pointing to the .so at runtime (idk, I haven't checked). So this optimization might make sense for this project.
: gumby | 21 hours ago
The article explicitly says incremental parsing was not needed for this application (ingesting whole files for AI analysis).
: jibal | 17 hours ago
> isn’t the entire point of tree-sitter that you can reuse old trees to make edit updates faster?
No, not for ast-grep.

gumby | 21 hours ago

Even this AI generated article hs a hard on for rust. If all references to switching to rust were eliminated the article would be the same.

senderista | 20 hours ago

I’m always so disoriented by wondering how to interpret the pronoun “I” in these AI-generated articles.

nullbio | 19 hours ago

ast-grep is a great tool. Really helps with agentic work. I wish the LLMs were trained to use it by default. Would make them incredibly powerful.

why_only_15 | 17 hours ago

I really don't like this. Don't post your AI-generated articles here. https://www.pangram.com/history/fce67fb4-4f6a-4eb4-a323-5048...

: anitil | 17 hours ago
I like ast-grep and I think the changes are interesting, but it isn't fun reading an ai-generated article.
It seems like a lot of the speedup comes from using an arena, using realloc with offsets rather than raw pointers and some change to the graph that enabled this (though I don't want to re-read it to confirm).