All I can think is software is such an unserious field that we keep repeating these mistakes.
I habitually use the systemd random jitter setting on my timer units to prevent this exact problem, even when it’s not really clearly necessary, and you’re telling me the people building these grid scale battery systems didn’t think of this before deployment?
Yes, That's exactly what we are telling you. Those of us who have been burned before are aware of issues like this. But not everyone is. I'm guessing that the people responsible for manufacturing batteries just hadn't experienced the problem. And it wouldn't show up with testing -- it's only when you have a huge number of these on the grid that there is any noticeable effect.
On the topic of avoiding thundering herds, one of my friends has a neat way to randomize the timing of his cron jobs: while testing he sets the job to fire in the next minute, and once he has got it working OK, he leaves the minute at the current time.
(I tend to use numerology, which is more entertaining but less principled: eg, I set an hourly DNS rebuild to trigger at XX:53. I didn’t come up with that wheeze until after I was doing email; back then the hourly update was at XX:15 not XX:25. Memorable numbers are sometimes helpful if other people might need to know when an update happens, and the update can’t be fast for whatever reason.)
But how do all these devices synchronize their consumption?
This question wasn't fully answered by the article, at least not explicitly. I think the implication is that the timers themselves must be synchronized to the same clock somehow? (NTP, GPS, etc.) In other words, it's not sleep for 60 seconds, check power, repeat. It's sleep until the start of the next minute, and the devices all agree when the minute ticks over.
Systems that wouldn't cause that synchronization:
A simple microcontroller that sleeps for 60 seconds: I imagine clock drift would act as its own natural jitter.
Keeping time by counting AC cycles: No drift, but devices would be spread out based on when they were powered on.
Random jitter is superior I’d imagine in every dimension, right? Even cheap random bytes that aren’t suitable for cryptography would be better than clock drift or cycle count.
I wonder if either the designers didn’t think of it or if they didn’t know they needed to spread out their load. It’s harder to assume the latter, so it must be the former, right? 🤔
Random jitter is superior I’d imagine in every dimension, right?
For sure, intentionally breaking synchronization by adding jitter is definitely better than assuming systems will naturally drift out of sync. But that's not what I'm asking. What I am asking is this: given that these systems are currently displaying synchronized behavior, what's keeping them from drifting?
To me, it feels like the news article stops short:
There's a mystery spike in power demand at the start of every minute.
Why the spike? Because lots of different devices are causing smaller spikes, and the spikes are all synchronized so they add up.
Why are the spikes synchronized? Because they're tied to a clock.
Wait, why are their clocks synchronized? (Too late, that's enough questions.)
Even if the nominal timestamp on the grid differs from "real time" (i.e ~23s right now https://gridradar.net/en/net-time) all devices on the grid can have the same internal reference time.
Something in such systems need to talk to the power grid accounting, and the current system setup default is to connect to the Internet everything in a subnet once any machine of the subnet is connected — and to run NTP on every Internet-connected machine.
Allowing the drift would presumably use every cycle of the grid, random jitter has a risk of people implementing it with one-second resolution (which is better than per-minute spikes, but can be further improved)
Oh, the research facility that poisoned the wells, featuring some hard to attribute but suspicious cluster of leukemia in its neighborhood and running a "hopefully in 40 years we'll have the tech to clean up our mess by the end of the century" strategy, still operates.
rkaw92 | 13 hours ago
A thundering herd problem, but for power? Jitter saves the day as usual.
hailey | 13 hours ago
All I can think is software is such an unserious field that we keep repeating these mistakes.
I habitually use the systemd random jitter setting on my timer units to prevent this exact problem, even when it’s not really clearly necessary, and you’re telling me the people building these grid scale battery systems didn’t think of this before deployment?
mcherm | 11 hours ago
Yes, That's exactly what we are telling you. Those of us who have been burned before are aware of issues like this. But not everyone is. I'm guessing that the people responsible for manufacturing batteries just hadn't experienced the problem. And it wouldn't show up with testing -- it's only when you have a huge number of these on the grid that there is any noticeable effect.
k749gtnc9l3w | 11 hours ago
If only they treated time-keeping as unseriously, the problem would be avoided though! Uneven unseriousness is the worst.
fanf | 11 hours ago
On the topic of avoiding thundering herds, one of my friends has a neat way to randomize the timing of his cron jobs: while testing he sets the job to fire in the next minute, and once he has got it working OK, he leaves the minute at the current time.
(I tend to use numerology, which is more entertaining but less principled: eg, I set an hourly DNS rebuild to trigger at XX:53. I didn’t come up with that wheeze until after I was doing email; back then the hourly update was at XX:15 not XX:25. Memorable numbers are sometimes helpful if other people might need to know when an update happens, and the update can’t be fast for whatever reason.)
moltonel | 11 hours ago
The simplest way to add jitter to a system is often to replace "do this every 60s" by "do this, wait 60s, repeat".
fanf | 10 hours ago
Yeah, if only Vixie Cron were able to do something like that :-)
It has the added advantage of avoiding problems if a periodic job unexpectedly takes longer than its period to run.
bitshift | 11 hours ago
This question wasn't fully answered by the article, at least not explicitly. I think the implication is that the timers themselves must be synchronized to the same clock somehow? (NTP, GPS, etc.) In other words, it's not sleep for 60 seconds, check power, repeat. It's sleep until the start of the next minute, and the devices all agree when the minute ticks over.
Systems that wouldn't cause that synchronization:
wonk | 3 hours ago
Random jitter is superior I’d imagine in every dimension, right? Even cheap random bytes that aren’t suitable for cryptography would be better than clock drift or cycle count.
I wonder if either the designers didn’t think of it or if they didn’t know they needed to spread out their load. It’s harder to assume the latter, so it must be the former, right? 🤔
bitshift | 3 hours ago
For sure, intentionally breaking synchronization by adding jitter is definitely better than assuming systems will naturally drift out of sync. But that's not what I'm asking. What I am asking is this: given that these systems are currently displaying synchronized behavior, what's keeping them from drifting?
To me, it feels like the news article stops short:
gerikson | 52 minutes ago
Because the European grid operates at 50Hz, it's not unfeasible to treat it as a reference frequency for a clock. See https://en.wikipedia.org/wiki/Electric_clock#Synchronous_electric_clock
Even if the nominal timestamp on the grid differs from "real time" (i.e ~23s right now https://gridradar.net/en/net-time) all devices on the grid can have the same internal reference time.
k749gtnc9l3w | 6 minutes ago
Something in such systems need to talk to the power grid accounting, and the current system setup default is to connect to the Internet everything in a subnet once any machine of the subnet is connected — and to run NTP on every Internet-connected machine.
k749gtnc9l3w | 8 minutes ago
Allowing the drift would presumably use every cycle of the grid, random jitter has a risk of people implementing it with one-second resolution (which is better than per-minute spikes, but can be further improved)
pgeorgi | 11 hours ago
Oh, the research facility that poisoned the wells, featuring some hard to attribute but suspicious cluster of leukemia in its neighborhood and running a "hopefully in 40 years we'll have the tech to clean up our mess by the end of the century" strategy, still operates.