Bunny's scriptable DNS is something I geek out so hard over. Bunny's DNS products in general, actually, are highly underrated. Kevin, my offer still stands to come work on it! :P
Hasn't stuff like this been solved with anycast and typical routing algorithms of old etc? Or is this solving a different problem? (This doesn't affect me but have been considering moving some stuff from aws/cloudflare to bunny)
Very few providers offer anycast. It's also kind of simpler to take servers from multiple providers and have DNS be the integration layer between all of those.
What I meant to say is that if you're going to build your own CDN, or edge network or whatever, very few providers offer VMs or dedicated servers with anycast.
Some hobbyists do anycast on their own, but this is more complex. See https://tylerobrien.dev/posts/2022/06/28/building-an-anycast... for example. This also require some advanced traffic engineering: what looks close for BGP may be on the other side of the planet. Relying on DNS only is more accessible.
I worked for a CDN that ran using anycast for 10 years, and there are a LOT of gotchas and trade offs that make it complicated to use.
For one, your ability to control traffic flow is limited to very coarse knobs, since where your traffic lands is dependent entirely on the BGP routes you advertise and who you advertise them to.
The basic idea of BGP is that you advertise the same IP blocks to multiple peers, and then every other network will choose the destination that is the least number of ‘hops’ away. However, hops are not always a perfect representation of the fastest, or even closest, destination. Sometimes a hop travels a far distance, or is over a slow or congested link. Sure, one connection might be 2 hops instead of 3, but if those two hops are slow the route with “more” hops might actually be faster.
You can solve this with a lot of different tricks, such as adding extra fake hops to your announcements in certain places, but you still have to be constantly monitoring the route traffic is taking to make sure your traffic continues to take the best route.
Secondly, there is no easy way to distribute traffic from a single network to multiple destinations for load balancing purposes. Every request coming from a particular network is going to make the same anycast calculation, so it will send all requests to the same destination. If you want to split that up into two different destinations, you have to go back to using DNS tricks or using different destinations IPs to split traffic up.
We would do this sort of load management by dividing traffic onto different IP blocks, and then dynamically advertising or not advertising those blocks from datacenters depending on load. If one datacenter is becoming overloaded, you pull some blocks from the advertisements for that datacenter, and that traffic stops being routed there.
There are a ton of other factors that go into it, but the bottom line is that anycast is not a simple fix to this problem.
I guess I'm just confused on who the actual target for this is (running your blog on 7 different servers around the world is uh...something...but I wouldn't call it practical/realistic). I wouldn't think that this type of thing is the first thing you reach for if your situation is actually calling for geobalancing (what level of the stack should this problem be solved, is maybe closer to what I'm asking).
You are right, this is literally the exact use case for a CDN; serving static content around the world with low latency. You only need one origin server, or two for redundancy.
Of course, my setup is overkill. It's mostly something to play with.
But, I also serve HTML pages "myself." CDN is only used for assets. I want to control the integrity of the content. CSS and JavaScript files served by the CDN are protected by an integrity hash (SRI). If I could, I would also include images in this, but this is not possible (and not something that looks planned).
I actually went and looked at the linked Terraform blog post and I realized something.
It mentions that you used CDKTF because you prefer its expressiveness. The actual TF is generated from CDKTF. Having struggled with understanding and writing TF in the past, now, I don't really have to care with agents. I let them rip, then just read the plan output.
I am sure we will find more such examples where we can rip out an intermediate step completely because of agents.
Yes, same experience here. Was also a breeze to go from CDKTF TypeScript to TF HCL and shed a substantial number of dependencies as a result. Very happy with where it's at today, but also happy I came in through CDKTF.
The "cloud provider rates" for this are a bit crazy. It's great bunny.net is offering a (more flexible!) product basically for free.
The NTP Pool does ~1m dns queries a second on the hourly peak with a small-ish (old) Go program (called `geodns`) running on a bunch of mostly small VMs around the world. The average is much less, but paying $0.70/million for ~11 billion queries a day would be a lot of money for a "random volunteers on the internet" project at "cloud provider rates".
uneekname | 22 hours ago
Wow!
arm32 | 22 hours ago
arm32 | 22 hours ago
vbernat | 21 hours ago
[OP] sshah03 | 21 hours ago
vbernat | 20 hours ago
arm32 | 19 hours ago
temp0826 | 22 hours ago
PaoloBarbolini | 22 hours ago
dotwaffle | 19 hours ago
Huh? Most of them offer anycast... Unless I'm misunderstanding what you're trying to say with this?
PaoloBarbolini | 18 hours ago
vbernat | 22 hours ago
cortesoft | 20 hours ago
For one, your ability to control traffic flow is limited to very coarse knobs, since where your traffic lands is dependent entirely on the BGP routes you advertise and who you advertise them to.
The basic idea of BGP is that you advertise the same IP blocks to multiple peers, and then every other network will choose the destination that is the least number of ‘hops’ away. However, hops are not always a perfect representation of the fastest, or even closest, destination. Sometimes a hop travels a far distance, or is over a slow or congested link. Sure, one connection might be 2 hops instead of 3, but if those two hops are slow the route with “more” hops might actually be faster.
You can solve this with a lot of different tricks, such as adding extra fake hops to your announcements in certain places, but you still have to be constantly monitoring the route traffic is taking to make sure your traffic continues to take the best route.
Secondly, there is no easy way to distribute traffic from a single network to multiple destinations for load balancing purposes. Every request coming from a particular network is going to make the same anycast calculation, so it will send all requests to the same destination. If you want to split that up into two different destinations, you have to go back to using DNS tricks or using different destinations IPs to split traffic up.
We would do this sort of load management by dividing traffic onto different IP blocks, and then dynamically advertising or not advertising those blocks from datacenters depending on load. If one datacenter is becoming overloaded, you pull some blocks from the advertisements for that datacenter, and that traffic stops being routed there.
There are a ton of other factors that go into it, but the bottom line is that anycast is not a simple fix to this problem.
temp0826 | 20 hours ago
cortesoft | 10 hours ago
vbernat | 7 hours ago
But, I also serve HTML pages "myself." CDN is only used for assets. I want to control the integrity of the content. CSS and JavaScript files served by the CDN are protected by an integrity hash (SRI). If I could, I would also include images in this, but this is not possible (and not something that looks planned).
whinvik | 22 hours ago
It mentions that you used CDKTF because you prefer its expressiveness. The actual TF is generated from CDKTF. Having struggled with understanding and writing TF in the past, now, I don't really have to care with agents. I let them rip, then just read the plan output.
I am sure we will find more such examples where we can rip out an intermediate step completely because of agents.
miyuru | 21 hours ago
I liked it but, it was a hassle to get it running.
tootubular | 21 hours ago
vbernat | 21 hours ago
askbjoernhansen | 22 hours ago
The NTP Pool does ~1m dns queries a second on the hourly peak with a small-ish (old) Go program (called `geodns`) running on a bunch of mostly small VMs around the world. The average is much less, but paying $0.70/million for ~11 billion queries a day would be a lot of money for a "random volunteers on the internet" project at "cloud provider rates".
matja | 19 hours ago
jedisct1 | 18 hours ago