So frustrating. You can only experience so many of these outages before you seriously have to start questioning whether it makes sense to continue to use GitHub (and many have already long since reached this point)
Same here. Gitea has been pleasant to use for my business, and I've had 0 downtime issues compared to near-daily problems with GitHub during the first half of the year.
Edit: Sorry, thought you wrote Gitea rather than GitLab. I've also had a nice experience with self-hosted GitLab years ago, and I'd wager that pretty much any self-hosted option would be better than GitHub at this point.
Yup am running too for 2 years now and love it. It’s my favorite service. Uptime is basically 100% since it’s only down when I’m doing server maintenance (apt get update and reboot) and then I’m not pushing anyway.
Isn't this an example of how everyone wants to believe that vibe-coding software is enough, but no one actually wants to depend on vibe-coded software (like GitHub)?
I switched to Forgejo and it's such a breath of fresh air. You can feel the solidity, snappiness and dependability of a good old carefully coded app.
Except Github outages are not related to vibe coding Github itself, and the current situation is arguably only tenable at all because they're able to rapidly add optimizations and adjust their architecture.
If you've been paying attention to any news related to this [0], they've been drowning with the rise of agentic development. By September this year, they were up to six times the monthly activity YoY. Even after moving to Azure, they're struggling to keep up with the sheer amount of traffic being thrown their way. But they have so far been adamant on not gutting features like free actions pipelines for public projects.
> Even after moving to Azure, they're struggling …
More like _because_ they moved to Azure. Azure has been nothing but crapulent. GitHub needed more capacity, but they chose the wrong place to add that capacity.
How do we know that it's not related to vibe coding GitHub?
There can be multiple reasons. The flood of agentic development repos is real. But showing another reason doesn't show that vibe coding GitHub isn't a reason itself.
Six times! Oh the tragedy! The only thing I remember being down more was Twitter circa 2009-2010, but they grew 15x in tweet volume on a hilariously inadequate RoR platform.
This was back when the 10k challenge was still a real thing for web servers.
GitHub’s UI is slow and clunky. And that has nothing to do with how many vibecoded lines of code are being pushed.
> And that has nothing to do with how many vibecoded lines of code are being pushed.
I think it has to do with the grandfathered code they had to replace to make things more efficient. And it's way more faster than 5 years ago, come on!
I feel like this is a very unserious response. Github is enormous and were at capacity in April at 2x YoY, then they tripled their traffic again within a few months. They have added an additional 4 million CPUs and are still struggling to keep up with an insatiable, exponentially rising demand.
As of September, they handle 2.8k commits and 1.4k actions a second - each which involve spinning up docker, installing hundreds of MBs of dependencies, and then running things like CPU-heavy linters.
If a blog with 100 people goes up 6x, that's nothing; a service with 150,000,000 active accounts going up 6x is a big deal. And I know you know that.
> GitHub’s UI is slow and clunky.
Can you read TFA or just not participate in the discussion, please? This has literally nothing to do with the UI.
That time has long passed IMO. This time last year GitHub started having these outages (before the explosion in agentic use, and shortly after the announcement they were moving everything to azure), and by February/March it was clearly untenable IMO.
I've believed, since jenkins was the the new hotness, that you should ALWAYS have a path to prod outside your CI. You can put whatever restrictions you want on it, but it must exist and be tested from time to time.
I don't know them, and am not an investor, but I'm trying Blacksmith as a GHA alternative because I'm losing my mind over the regular downtime. Apparently all the cool kids already use it, so... Obviously this doesn't help at all with Webhooks.
update: first jobs are green already. migration took 5 minutes. so far so good.
Yes Blacksmith is great. We use them for all our CI jobs. Make sure to enable the Docker build cache if you use it.
In our benchmark, Blacksmith was the fastest runner among the providers we tested (Warp was another one) and then the build cache adds additional time savings.
I'm not saying bots/LLMs aren't making GitHub's job harder but I saw an uptime graphic with a pretty stark and obvious change pre/post acquisition by Microsoft.
There has been degradation in their copilot cloud agent in the last few weeks (sessions erroring/disappearing, connections not working, etc) that is not on a status dashboard.
We self-host actions and this still impacted us. Incident resolved 30 minutes ago, but all our builds are still queued/pending. Maybe GitHub needs to replay some webhooks?
The more these incidents happen, the more I think about I talking to investors about investing in my GitHub alternative; an old computer in the corner running Gitea.
I have been moving everything away to a self hosted Forgejo, and I am so happy.
But now I wonder about Netlify. Their dashboard is jammed full of prompt boxes for agents. Could more AI psychosis there turn that into a house of cards as well?
I was setting up a project for CI/CD for the first time in a much more complex way than before and this issue had me scratch my head a lot as I was questioning my sanity. Then I remembered it's GitHub and that it's just another outage, just not reported on their dashboard yet.
After the GitHub post about their new git storage architecture, I wonder how other parts of the system, like issues, actions and webhooks, are being improved to handle current scaling problems, if at all.
Apparently the git layer is not the only thing that needs a little attention.
For those using GitHub Actions who want to be out of the blast radius of these outages, we at Depot built our own CI engine that speaks a GitHub Actions syntax but is our own control plane and runner [0]. Connected to all of the other things we do to accelerate CI, builds, tests, etc.
Happy to answer any questions or hook anyone up with an extended trial that wants to move things over to it.
dgritsko | 16 hours ago
x86a | 15 hours ago
Aaronstotle | 15 hours ago
trevinhofmann | 14 hours ago
Edit: Sorry, thought you wrote Gitea rather than GitLab. I've also had a nice experience with self-hosted GitLab years ago, and I'd wager that pretty much any self-hosted option would be better than GitHub at this point.
Riemer | 15 hours ago
sandeepkd | 15 hours ago
betaby | 15 hours ago
nvme0n1p1 | 14 hours ago
So you switched last week?
xyst | 15 hours ago
huijzer | 15 hours ago
colechristensen | 15 hours ago
The features I care about work as I want them to, the features I don't care about are kinda janky.
kilroy123 | 15 hours ago
adamddev1 | 14 hours ago
adamddev1 | 14 hours ago
I switched to Forgejo and it's such a breath of fresh air. You can feel the solidity, snappiness and dependability of a good old carefully coded app.
Tadpole9181 | 14 hours ago
If you've been paying attention to any news related to this [0], they've been drowning with the rise of agentic development. By September this year, they were up to six times the monthly activity YoY. Even after moving to Azure, they're struggling to keep up with the sheer amount of traffic being thrown their way. But they have so far been adamant on not gutting features like free actions pipelines for public projects.
[0] https://github.blog/news-insights/octoverse/octoverse-commit...
onionisafruit | 14 hours ago
More like _because_ they moved to Azure. Azure has been nothing but crapulent. GitHub needed more capacity, but they chose the wrong place to add that capacity.
1991g | 14 hours ago
onionisafruit | 14 hours ago
Tadpole9181 | 13 hours ago
adamddev1 | 14 hours ago
There can be multiple reasons. The flood of agentic development repos is real. But showing another reason doesn't show that vibe coding GitHub isn't a reason itself.
cromka | 14 hours ago
Let's just assume Occam's razor and give them benefit of a doubt over the six-fold usage growth figures.
slau | 14 hours ago
This was back when the 10k challenge was still a real thing for web servers.
GitHub’s UI is slow and clunky. And that has nothing to do with how many vibecoded lines of code are being pushed.
cromka | 13 hours ago
I think it has to do with the grandfathered code they had to replace to make things more efficient. And it's way more faster than 5 years ago, come on!
Tadpole9181 | 12 hours ago
As of September, they handle 2.8k commits and 1.4k actions a second - each which involve spinning up docker, installing hundreds of MBs of dependencies, and then running things like CPU-heavy linters.
If a blog with 100 people goes up 6x, that's nothing; a service with 150,000,000 active accounts going up 6x is a big deal. And I know you know that.
> GitHub’s UI is slow and clunky.
Can you read TFA or just not participate in the discussion, please? This has literally nothing to do with the UI.
sharts | 6 hours ago
eluru | 5 hours ago
herpdyderp | 14 hours ago
nerdypepper | 14 hours ago
maccard | 14 hours ago
OptionOfT | 15 hours ago
saejox | 15 hours ago
tosser811 | 15 hours ago
vrosas | 14 hours ago
mtremsal | 15 hours ago
update: first jobs are green already. migration took 5 minutes. so far so good.
rocho | 14 hours ago
In our benchmark, Blacksmith was the fastest runner among the providers we tested (Warp was another one) and then the build cache adds additional time savings.
mococa | 15 hours ago
Riemer | 15 hours ago
hrpnk | 15 hours ago
Not being able to isolate OSS and Enterprise workloads properly is also quite the downer.
deadbunny | 15 hours ago
colechristensen | 15 hours ago
MYEUHD | 14 hours ago
https://hstockplus.com/accounts/twitter-x-accounts?sortBy=pr...
bpiroman | 15 hours ago
plasma | 15 hours ago
Vosporos | 15 hours ago
herpdyderp | 14 hours ago
cyberpunk | 14 hours ago
godzillafarts | 15 hours ago
joquarky | 7 hours ago
fishgoesblub | 14 hours ago
adamddev1 | 14 hours ago
But now I wonder about Netlify. Their dashboard is jammed full of prompt boxes for agents. Could more AI psychosis there turn that into a house of cards as well?
herpdyderp | 14 hours ago
rvz | 14 hours ago
6 year prediction and counting. [1]
[0] https://news.ycombinator.com/item?id=50307843
[1] https://news.ycombinator.com/item?id=22868406
time0ut | 14 hours ago
antonvs | 14 hours ago
spartanatreyu | 13 hours ago
And for the e-ink display hanging on the wall: https://www.dayswithoutgithubincident.com/
throw03172019 | 14 hours ago
voganmother42 | 14 hours ago
unconscionable | 14 hours ago
cromka | 14 hours ago
danielmartins | 13 hours ago
Apparently the git layer is not the only thing that needs a little attention.
kylegalbraith | 12 hours ago
Happy to answer any questions or hook anyone up with an extended trial that wants to move things over to it.
[0]: https://depot.dev/docs/ci/quickstart