Oof. It's hard to pick one! I like the subtle fixes like swapping sleeps in tests with DB readiness checks or pinpointing a single long running test causing a shard to run slow.
Our primary difference is self-improving PRs from continuously analyzing the telemetry from your CI. Most folks just sell fast hardware and are happy to charge you for jobs that don't saturate the host.
In our examples with open-source repos above you'll see that most of speedups that lead to a 10X speedup come from code changes or optimizations.
We use Mastra to run the agents in our CI harness. It's been great for being able to swap out models easily. More details here https://starsling.dev/blog/agent-loop.
> 3 -- How many times you have to run that critical path. Most orgs run it 2-3x
This is the one we went after. Our agents have to run tsc, lint and the unit tests before they push (it's in a hook), and CI was then doing all three again from scratch. Now the agent records what it ran as a git trailer on the commit and the PR build skips those steps [1]. Anything headed to main still runs everything, so the trailer only has to be trusted on the intermediate pushes.
Builds went from ~10 minutes to 1.5 over a few weeks, and a good third/half of that was this change. It tends to compound too - getting the build time down led to lower contention, warmer caches, faster throughput. All the usual queuing theory stuff.
I experimented with something similar, but running the actual gha jobs locally. Were you doing any verification to ensure the agent actually run all tests? do you sitll run remote CI? how do you deal with side effects of work that integrates with other PRs in some way? Were you sandboxing them?
I love this. I'm thinking we could make a git trailer of the evidence oid we create -- would make the UX much better.
I maintain witness.dev (CNCF) and cilock.dev (Apache 2.0 open core) that builds provenance around SDLC (or ADLC steps). We are experimenting using this to just let the agents land code after they prove their testing. Right now we are expecting a 2x increase in throughput at the same amount of compute
Congrats! I spent alot of time scaling really large and complex CI and integration testing builds so its great seeing a product that can help devs with this.
As you note, we can do those optimizations ourselves. The kind of person looking for CI runners is likely doing that already. What I value is speed, cost, scheduling delay, reliability, observability, local reproducibility, not using YAML... There is a lot of value to provide but workflow optimization is not where I would look.
That's fair. At a certain point, once you value all of those things enough, you'll just move off of YAML entirely (we think of this as Level 4 of CI https://starsling.dev/blog/levels-of-ci#level-4). But I think there are a lot of people still trying to get the biggest gains with the least amount of effort. That's who we're going after first.
I think YAML used to matter more when humans were writing much of the code, but they matter alot less now, especially because you can already write much of your YAML workflows in high level code. It's still matters, but alot less than it used to or enough to be a pain. I agree with the rest of the concerns though.
Yeah YAML written by agents seems like it's gonna be around for a while. GitHub Actions has 90%+ market share for Claude recommendations: https://amplifying.ai/research/claude-code-picks/report#rank.... The problem is, nobody is going back to clean it up until it's a real bottleneck.
Off topic but every time I read a name like "Star Sling" I get excited to see another space startup tackling a difficult challenge; and so when it turns out to be CI for agents, it's a letdown and an annoyance. Idk, this project seems cool, but maybe pick a less pretentious name.
We chose the name because our original inspiration was an agentic developer portal. So we thought Stargate + Dr. Strange's sling ring that opens portals was a cool mashup. Sorry to disappoint :(
Bereket from Better Auth - we've been on StarSling for about a year now, it's been such a hard a problem for us ot solve CI to the amount of PRs we get and continues need of optimization and faster runners, im so bullish!
So the agent compares the changed workflow with the existing one and judges whether the speedup is significant. But a change that re-shards or skips tests would always look like a win on speed alone, wouldn't it? How do you check that such changes still catch the bugs the old workflow caught? I had the same worry when I trimmed our tests because CI was too slow.
Good question! In terms of the what counts as a win for our agents: speed alone isn't enough. A change only counts if the patched run and the unchanged run executed the same tests (same test IDs, same coverage).
Re-sharding is allowed because it only changes where tests run, not what runs. It has the same sort of parity check as other patches but it also has a cost check to make sure the change doesn't increase runner time by more than 5% by default. We added the cost check after seeing the agents crush wall clock while 10x'ing costs on some test :)
Judging whether or not a test should be trimmed isn't something our agents do yet, because as you alluded to, it's hard. But we're working on it! If you have any ideas or an approach that's worked for you, would love to hear it.
When I trimmed my tests, I looked for ones that could be merged. Many unit tests shared the same setup and teardown, so when I merged their middle parts, that setup and teardown only had to run once. Even after merging, the tests still had to stay independent of execution order, so I checked that. In another case, some infrastructure checks were repeated across tests in one folder. For example, “the app needs environment variable A at startup” is already a test of its own, but every application-layer test also asserted it at the start. I kept that check in one place instead.
The parity check on test IDs plus the cost cap is a good deterministic checker. For trimming tests, where judgment comes in, I'd have a separate reviewer agent decide it and not the one that proposed the change.
Congrats on the launch! In the past pre-AI I did an improvement to the github actions pipeline from 20 minutes down to 6, it's a relief when you improve things like this and improve the team throughput!
Yeah it accelerates the whole team! If you remember, what was the biggest lever? We're always looking to expand the patterns in our /speed-up skill. :)
Curious how the inference cost of the agent loop compares to the CI minutes it saves. On a smaller repo I could see it eating most of the savings. Is there a repo size where it starts to clearly pay off?
gkoberger | 23 hours ago
What are some of the most interesting things the AI has caught + fixed so far?
dbworku | 23 hours ago
[OP] yonasb | 18 hours ago
https://github.com/starsling-demo-org/benchmark-gogs/blob/sh...
dotmike | 23 hours ago
dbworku | 22 hours ago
In our examples with open-source repos above you'll see that most of speedups that lead to a 10X speedup come from code changes or optimizations.
d-sanoj | 22 hours ago
[OP] yonasb | 22 hours ago
jstrasner | 22 hours ago
[OP] yonasb | 22 hours ago
kinfish | 22 hours ago
[OP] yonasb | 20 hours ago
colek42 | 22 hours ago
Three things matter.
1 -- How many non-conflicting changes can actually be made at one time (Fan Out)
2 -- How long a critical path takes
3 -- How many times you have to run that critical path. Most orgs run it 2-3x, once before the PR in the agent loop, once in CI, and another in the MQ
This is why most orgs cant get past about 12 PRs an hour.
dbworku | 22 hours ago
jwilliams | 18 hours ago
This is the one we went after. Our agents have to run tsc, lint and the unit tests before they push (it's in a hook), and CI was then doing all three again from scratch. Now the agent records what it ran as a git trailer on the commit and the PR build skips those steps [1]. Anything headed to main still runs everything, so the trailer only has to be trusted on the intermediate pushes.
Builds went from ~10 minutes to 1.5 over a few weeks, and a good third/half of that was this change. It tends to compound too - getting the build time down led to lower contention, warmer caches, faster throughput. All the usual queuing theory stuff.
[1] https://jonathannen.com/dont-run-it-twice/
dbworku | 17 hours ago
We love Buck2 and Bazel, from first principles. Unfortunately, the DX hurdles of making a codebase hermetic have prevented widespread adoption IMO.
jwilliams | 16 hours ago
Bnjoroge | 14 hours ago
jwilliams | 13 hours ago
colek42 | 13 hours ago
I maintain witness.dev (CNCF) and cilock.dev (Apache 2.0 open core) that builds provenance around SDLC (or ADLC steps). We are experimenting using this to just let the agents land code after they prove their testing. Right now we are expecting a 2x increase in throughput at the same amount of compute
buildwrangler | 22 hours ago
dbworku | 21 hours ago
[OP] yonasb | 21 hours ago
esafak | 21 hours ago
[OP] yonasb | 20 hours ago
Bnjoroge | 19 hours ago
[OP] yonasb | 18 hours ago
sochs | 20 hours ago
[OP] yonasb | 20 hours ago
bekacru | 19 hours ago
[OP] yonasb | 19 hours ago
emscb | 18 hours ago
[OP] yonasb | 17 hours ago
Re-sharding is allowed because it only changes where tests run, not what runs. It has the same sort of parity check as other patches but it also has a cost check to make sure the change doesn't increase runner time by more than 5% by default. We added the cost check after seeing the agents crush wall clock while 10x'ing costs on some test :)
Judging whether or not a test should be trimmed isn't something our agents do yet, because as you alluded to, it's hard. But we're working on it! If you have any ideas or an approach that's worked for you, would love to hear it.
emscb | 16 hours ago
slaser79 | 6 hours ago
brunoarueira | 18 hours ago
dbworku | 17 hours ago
tristenharr | 18 hours ago
[OP] yonasb | 17 hours ago
jbqueries | 9 hours ago
Actual problem tho, congrats on the launch!
poxlr | 9 hours ago
MisterMunchkin | 2 hours ago