The fact that astra had access to tools doesn’t really reduce the impact of this. The question is what can it do if you load it up with tools and enough time to try to improve itself.
How can we be sure what this really uses under the hood? Neural nets are black boxes in general, commercial LLM-based providers are even blacker boxes.
This means none provided by the harness, not no tools ever. Several of the experiments provided tools (python and other things), and the logs of both directions including what astra reported as reasoning summaries are linked on the blog post. internal python calculator use would likely tend to show up in the logs if that happened.
I’m kind of spooked. I wonder what astra could do with more tools and a prompt to brainstorm research directions and tooling to generate new architectures then transfer weights in.
I am also spooked that this might allow to build Vantablack boxes instead of black boxes, and we know people don't evaluate things seriously before deploying them unsupervised…
This definitely makes it seem like recursive self improvement is within reach, and requires really only putting the resources at the disposal of an llm-driven setup, with a level of cleverness that definitely exists in the frontier labs.
joshka | 14 hours ago
The is actually nuts.
hwayne | 2 hours ago
Might not be as big a deal as claimed
Student | an hour ago
The fact that astra had access to tools doesn’t really reduce the impact of this. The question is what can it do if you load it up with tools and enough time to try to improve itself.
icefox | 2 hours ago
Obvious question is, if an LLM can take apart the weights of another (simpler) LLM, can we make tools that do it for us in more rigorous ways?
Edit: Next question is, how well does this actually work on sample sizes greater than 4?
k749gtnc9l3w | 2 hours ago
Kind of? Needs more research to see how close it is to the previous work…
nicoco | 9 hours ago
Pretty crazy, BUT...
How can we be sure what this really uses under the hood? Neural nets are black boxes in general, commercial LLM-based providers are even blacker boxes.
joshka | 7 hours ago
This means none provided by the harness, not no tools ever. Several of the experiments provided tools (python and other things), and the logs of both directions including what astra reported as reasoning summaries are linked on the blog post. internal python calculator use would likely tend to show up in the logs if that happened.
Student | 5 hours ago
I’m kind of spooked. I wonder what astra could do with more tools and a prompt to brainstorm research directions and tooling to generate new architectures then transfer weights in.
k749gtnc9l3w | 2 hours ago
I am also spooked that this might allow to build Vantablack boxes instead of black boxes, and we know people don't evaluate things seriously before deploying them unsupervised…
Student | an hour ago
This definitely makes it seem like recursive self improvement is within reach, and requires really only putting the resources at the disposal of an llm-driven setup, with a level of cleverness that definitely exists in the frontier labs.