Codeberg's domain suddenly disappeared from their account, site is unreachable

152 points by mort 9 hours ago on lobsters | 43 comments

Update: We have received the confirmation that our domain was suspended and our case got forwarded to the abuse team. We're still waiting for a final decision.

https://social.anoxinon.de/@codebergstatus/117636595327014447

Apparently hosting source code gets you treated like you're the pirate bay these days.

[OP] mort | 9 hours ago

I just don't get why supposedly-reputable Internet infrastructure companies keep doing this. If a long-term paying customer with a history of good behavior gets flagged by some automated abuse detection, the next step seems to just be to terminate service immediately? We keep hearing stories like this from AWS and Google and other hosting providers too, I don't get it

WilhelmVonWeiner | 9 hours ago

I've been of the opinion the entire DNS system is not fit for purpose for a while now.

Well that's just clear hyperbole. While the DNS has its problems, it absolutely succeeds at its general purpose of resolving hierarchical names into data. It's remarkably resilient, scalable, and (by distributed systems standards) simple. It's the nerve center of modern digital communication.

If DNS is not fit for purpose (what purpose?), then what is? Bear in mind the inherent political nature of naming, and Zooko's Triangle.

xyproto | 3 hours ago

One of the purposes is that sites like codeberg.org should not be made inaccessible outside of their control.

JulianSildenLanglo | 22 minutes ago

But some sites should be made inaccessible outside of their control. Therefore there has to be a mechanism for shutting down service

Yeah, it's frustrating. App developers getting their apps censored or removed for completely arbitrary reasons etc. The importance of these platforms have outgrown their technical design, the governance structures and the regulations around them, but at the same time the companies that control them have gotten so powerful politically that improving the situation has also become almost impossible it seems.

mikedorf | 3 hours ago

completely arbitrary reasons

One thing that bugs me about this discourse is that most of the 'censorship' is not for arbitrary reasons at all. Developers often feel like they bear no responsibility for complying with laws/regulation/norms and get mad when that worldview is incompatible with society. Sometimes things slip through the cracks but it's annoying how many times spammers, bots, and others hosting malicious content complain about platforms doing the right thing and cutting them off.

It's not worth speculating about codeberg in particular though. Who knows what happened.

The arbitrariness I am talking about is not about Apple or Google stopping spam or malicious apps. That's fine, and I wish they were better at this than they are, way too much crap gets through the review process today. The arbitrary reasons is something every app developer can attest to: You submit a legitimate app for review and get a denial without any clear reason or way to address it, or it gets denied because Apple intends to release an app that does the same thing but hasn't told anyone. But ultimately, it's arbitrary because the only app stores in existence are controlled by two US corporations acting as such with a very US-centric view of what is acceptable or not.

The EU is trying to address this but is completely and utterly ineffective, which is annoying and sad. But it's a problem for the entire world.

mikedorf | 2 hours ago

I don't know, imo seizing a domain is a pretty different thing than blocking an app from a walled garden. The relationship is completely different.

Like I'd compare this to problems I had at a job where our hosting provider was getting IP blocked by other hosts/services (by being a bad citizen). It's annoying but understandable and it keeps me from recommending that hosting provider, because your services become guilty by association.

The similarity is in the experience of being subject to the whims of some distant and untouchable corporation for something that ought to be a common good. That they can just decide to stop providing service on a whim without explanation, and provide little or no support. The ownership and level of regulation does not match the role it plays in society, I think.

jamesgecko | 2 hours ago

And yet, it’s funny how the nuclear approach is deployed against smaller hosts and never Github.

JulianSildenLanglo | 15 minutes ago

I suspect that Github pays good money to Mark Monitor for their "Corporate Domain Management" services.

It's not worth speculating about codeberg in particular though. Who knows what happened.

edit: I reread the comment in context and get what you are saying a bit better. I think this part still makes sense though:

The reasonable response for any of the scenarios you outline would have been for the registrar to contact Codeberg and allow them to address the issue, not to silently delete the domain name from their account.

This is a strange take. What are "norms"? Whose norms, exactly, should developers adhere to? The whole case for the rule of law is that laws are created through a process which provides, in theory, some level of accountability; in contrast norms are defined by their arbitrariness. If they weren't arbitrary they'd be laws.

In any case, what's really arbitrary here is who these norms get applied to and who they don't. Remember when, through a democratic process, the US government decided to ban TikTok? and then all the tech companies decided, you know what, we make a lot of money off these guys, we're just going to ignore the law. It's only the small fish who are ever the targets of these kinds of actions. You can use Facebook to run a genocide and nobody is going to take down Facebook.

viraptor | 8 hours ago

It even becomes a special feature sometimes. There are exclusive registrars that charge premium, but deal with the big names. I think Twitter used them. Like MarkMonitor and CSC Corporate.

Unfortunately if you're running something the scale of codeberg, you probably want to pay for it.

mitsuhiko | 2 hours ago

If a long-term paying customer with a history of good behavior gets flagged

The unfortunate reality is that sites with user generated content are just tough to service for everyone involved. I used to run a pastebin and I shut it down because people used it for really, really bad stuff and that just makes you not want to deal with it.

icefox | an hour ago

Oversight on these companies is not strict enough. Which is to say, as far as I can tell there really isn't any.

viccuad | 8 hours ago

I think it's fair to name shame the DNS registrar here, Namesilo. Took me a click to find it. Once this is resolved, I suppose they will migrate from it. Because who in their sane mind would ever trust again a registrar like that.

pgeorgi | 6 hours ago

I suppose that depends on how they handle it. If it's sufficiently and believably "oops, we're sorry" with some reassurances that the account is moved to the "manual review" tier, it might be better to stay - the devil you know, and all that…

strongoose | 8 hours ago

Bogus DMCA takedown do we think?

radex | 8 hours ago

I (naively) hope that the registrar either fixes the problem ASAP and issues a good apology and post-mortem detailing how this happened and what they will do to prevent it from happening again… OR get an absolutely brutal treatment by their customers.

Losing a domain is devastating and any registrar that causes that to happen (without an extremely good explanation) cannot be trusted.

[OP] mort | 8 hours ago

Kinda feeling like time is running out for option #1; it has been 2 hours already, the registrar's support staff has responded so they've been aware of the issue for hours, yet it's not resolved. That's just too long

eduard | 8 hours ago

Losing a domain is devastating and any registrar that causes that to happen (without an extremely good explanation) cannot be trusted.

I totally agree, does anyone know which registrar caused this?

This whole situation is really appalling.

[OP] mort | 8 hours ago

Seems they use NameSilo, but let's be careful blaming them until we know more of what's going on. For all we know, it's something outside of their control too; it could be the fault of a service provider they use, or it could be something Public Interest Registry (the company behind .org) did, or something completely different. It seems likely that it's NameSilo's fault but I'll wait until things are a bit more clear before I bring out the pitchforks.

chrismorgan | 7 hours ago

The EPP status clientHold indicates that it’s the registrar (NameSilo) that did it. If it was the Registry Operator (PIR), it’d be serverHold.

chwilson | 8 hours ago

[OP] mort | 9 hours ago

They apparently host a status page on a different domain that's still reachable: https://status.codeberg.eu/status/codeberg

Psentee | 7 hours ago

The domain is slowly recovering. Many users are able to connect to our systems again. Some might need to wait for DNS propagation.

Thank you so much for your patience. We're sorry for the serious disruption.

We haven't yet received an official response from Namesilo's abuse department about the incident.

https://social.anoxinon.de/@codebergstatus/117637187711177698

dethos | 7 hours ago

This begs the question: should a critical service (for FOSS) like Codeberg also provide access through an onion service?

This would allow users to access it and keep working in cases like this one.

mariusor | 2 hours ago

With all the love for codeberg, but if they are a "critical service" for your ops, you're probably doing something wrong. You should have a failsafe somewhere for running your deployment pipeline locally, not to mention source control, which most definitely shouldn't be load bearing.

JulianSildenLanglo | 12 minutes ago

A backup domain in a separate jurisdiction could be prudent.

What a nightmare! I don’t have anything on Codeberg, but it must surely suck to experience such a rug-pull. Hope everything works out for them.

gasche | 4 hours ago

I use codeberg and in fact a few hours ago I was giving a talk at work with a URL pointing to a project hosted there, so I definitely got remarks that it was down. I'm sure it was stressful and unpleasant for codeberg sysadmin people and I wish them best.

And/but I also see this as part of codeberg's value proposition: I think of it as kind of "amateurish in some aspects, cooperative infrastructure" as opposed to "rock-solid, commercial infrastructure". I think that some codeberg people agree with this view that they are not aiming for absolute stability/reliability. In that respect I think that it's not bad to have breakage from time to time, so that everyone is aware that this is something that can happen and plan/design their things accordingly. If codeberg was up all the time, I would start to assume that it will always be up in the future; with a yearly or montly reminder of a few hours of downtime I will plan for it properly.

chrismorgan | 8 hours ago

Text, since the website isn’t rendering for me for uncertain reasons and I’m probably not the only one:

codeberg.org is unfortunately down right now due to a yet undefined issue with our domain. It was not expired and not transferred, the domain is completely gone from our account and we did not receive any email notifications, and it is currently in the status clientHold. We are currently trying to contact our registrar to get more information right now & trying to resolve this issue, but this is probably going to take some time.

We are sorry for the inconveniences & are working on a solution.

(I wish Mastodon would render proper HTML bodies in their pages. They already put the complete text in three places in the head markup, so it’s not like it’s a pure client-side-rendering stub. Even for the most rudimentary of support, they could have put the text inside their .app-holder element very easily so there’s something if it breaks, as such things are like to.)

mariusor | 2 hours ago

Yeah, when wanting to be Twitter is taken too far...

toastal | 8 hours ago

Great reason to always have at least 2 mirrors of your projects (ideally 1 self-hosted where you are in control). You might not have bug tracker up, but at least folks can fetch the source from a fallback for these situations.

For instance if you use Git, downstream folks that are fetching your Nix overlay, they can get the tarball from Codeberg or self-hosted cgit with pkgs.fetchzip { urls = [ /* … */ ]; } & not be interrupted by downtimes (no, not all source code is in cache.nixos.org).

chrismorgan | 7 hours ago

It’s not cut and dried. One domain makes authenticity easy to track, multiple encourages phishing and is incompatible with unphishable 2FA methods that are tied to the origin (passkeys, hardware tokens, not TOTP).

terinjokes | 5 hours ago

I don't think OP was arguing for Codeberg to have muliple domains in active use, but you as the user should mirror to multiple platforms.

toastal | 2 hours ago

Yes. Readonly mirrors are fine—& can be automated on push/pull to stay in sync.

ginkogruen | 5 hours ago

I wasn't aware you could specify multiple urls. Thanks for the tip!

toastal | 2 hours ago

pkgs.fetchurl also supports it.

I don’t like urls since it doesn’t express the empty case correctly. I added pkgs.fetchdarcs { url; mirrors; } & pkgs.fetchpijul { url; mirrors; } to Darcs & Pijul’s fetchers which does express the that you need a non-empty list of places to fetch from but 🤷. This is also why Nixtamal (self-plug) supports mirrors—since it support the build-time fetchers + Darcs/Pijul (whose prefetchers I got merged upstream into Nixpkgs).

drrlvn | 6 hours ago

Seems to be recovering:

The domain is slowly recovering. Many users are able to connect to our systems again. Some might need to wait for DNS propagation.

Thank you so much for your patience. We're sorry for the serious disruption.

We haven't yet received an official response from Namesilo's abuse department about the incident.

https://social.anoxinon.de/@codebergstatus/117637187711177698