Today I learned that I'm a manager. (Melting emoji face.)
No leadership in the office during this nightmare. We manage our tasks through Outlook. Apparently I'm the only one who knows how to find all of our work outside of outlook, so I ended up creating task lists and parceling them out to people so we could keep going (we were already behind and short staffed before today).
I need to remember to put a comment down for my performance review about how I exceeded expectations today.
I need to remember to put a comment down for my performance review about how I exceeded expectations today
I’ve got a little folder where I keep things like this, and stuff like positive feedback from executives etc, so that when the reviews come around, I have it all in one place. It’s just in a little email folder in my Outlook inbox... oh.
I’ve been tempted by Obsidian in my personal life for some time, but in the workplace our machines are very locked down, I don’t think I’d be able to install it. Does Obsidian have a runs-in-browser version or something similar?
My office not only uses Outlook to schedule meetings, but the meeting room bookings and telepresence systems are integrated with it...so tomorrow could be interesting. (Fortunately I didn't have meetings later in the day today.)
Was just about to post an article! I don't use any Microsoft programs, so I was surprised to hear it's still ongoing. It's apparently impacting anything that communicates with Exchange Online and any functions related to it, which turns out to be a lot of things.
Worse: they apparently haven't identified the root cause yet, which is crazy given it's gone on so long.
Second most recent status at the time of this post from the cloud status page:
Current status: As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.
We will continue to share more information as soon as we have it, and for now we can confirm that there is no indication this is the result of malicious activity. We’ll continue to apply learnings towards remediation actions as we confirm effectiveness mitigation actions.
Which as far as I can tell means "we have no clue what caused this and are just trying anything we can, but we don't think it was a hacker or breach." Not sure if that makes this better or worse. On the one hand, yay security... On the other hand, that means MS holds all the blame and the company's competence is now in question.
Then while writing this comment a new status update dropped:
Current status: We’re continuing our efforts to improve server stability to improve traffic flow. This includes adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures.
We're observing partial recovery across regions in which we've applied these adjustments; however, these actions aren't resulting in sustained recovery, so we're reviewing what additional actions we can take to improve overall infrastructure stability.
Which I think means: "we're trying to get servers online by messing with traffic, but it only works until it gets traffic and then it goes down again. [Probably because the root problem causing this mess is still there and unidentified.]"
Last part is my addition. Feels like they might be trying to just treat the symptoms now...
By the way, if anyone wants a laugh, check out the r/sysadmin subreddit. Went there to check for updates and translations of the gobbledygook from Microsoft. (Typically translates to "still no clue, have some word salad so it's less obvious.") Lots of late-night snark and some hopes to enjoy a nice three-day weekend. (Australia may have already gotten it.)
And one person made the point that companies will have a bunch of emails queued up today to send as soon as the servers are back online, citing 10k from one company of unknown size. So even when they do fix it, the servers are going to be dealing with an onslaught of emails. (Which might explain why the servers crash whenever they get them back online...)
Also for some reason I'm now getting flashbacks to email storm sagas. Given this is outlook, would be hilarious if this is somehow the result of an email hurricane.
Also for some reason I'm now getting flashbacks to email storm sagas. Given this is outlook, would be hilarious if this is somehow the result of an email hurricane.
I love one of the names for that - 'Reply Allpocalypse'.
This is the company that also has a GitHub outage practically every day. They don't even have two nines of service, unless 89.9% counts. They're a joke.
My org's self-hosted Forgejo that runs on a single Hetzner node with high memory pressure causing it to OOM all of the time has better uptime than Github.
As a former employee of Microsoft Microslop (and even part of the org that contains Outlook) - the only thing that shocks me is that this doesn't happen every other month.
The management there is so astoundingly bad at letting engineers do their job. This is only going to make it worse for them too. Hopefully other companies take note that this is what happens when you lay off thousands of talented engineers because you think AI can replace them.
Had a fun talk with my boss today. He wanted to discuss an email he sent me today about a problem with one of our older apps running TLS 1.0 and how it's going to get fully blocked by the corporate firewall changes in less than a week.
I had to remind him I hadn't seen the email because of the international 365 outage.
He also mentioned how he felt like he was going through withdrawal because Copilot wasn't working.
So the company I work for got acquired last month, and we've all been jittery about the pending impacts of migration of our domain and Microsoft 365/Azure instances. Far too much of our workflow, operations, and institutional knowledge relies on that single provider (and we're going to lose Slack in favor of Teams, which is just :facepalm).
When the outage started yesterday, most of us thought it was the new mothership flubbing configuration or the old owners cutting off services prematurely, until the new e-mail addresses stopped responding as well. I was the first to try and check the Microsoft 365 status page and found it loading a blank screen, then hit the r/sysadmin thread... The gibberish status posts from Microsoft suggest things that should never happen with well-managed regionalized cloud infrastructure.
Well done, M$.
And it's extra special that this is the busiest season of the year, as we try to wrap up projects before healthcare customers (nearly all of whom have also gotten overly Microsoft-dependent) enter holiday change freezes. A quiet Inbox is not what I'm looking for with major go-lives and cutovers happening next week.
This has been one of the most peaceful Fridays I’ve had in a long time, and it was so satisfying to be left alone to actually get everything from my backlog knocked down before the weekend.
Walking in on Monday is going to be so crazy by comparison!
According to this series of Substack posts from last year (saw linked in /r/sysadmin), Azure has been mismanaged for a while. I did not attempt to do any sort of vetting of this, so could be complete hogwash.
Microsoft is struggling to fix a major global outage of its service that is reportedly affecting tens of thousands of users.
Services including Teams, Outlook and Microsoft's artificial intelligence software Copilot have been affected by "connectivity problems".
[...]
Reports of issues with Outlook peaked globally at nearly 140,000 around 18:00 GMT on Thursday, according to monitoring website Downdetector. The BBC understands there is no indication the outage was caused by a cyber-attack.
Between this and Nvidia, these last couple days sure have been interesting.
Saw this elsewhere: “the fact that the world is cheering the outlook outage as a national holiday tells more about the state of burnout in the workplace than any HR survey ever will”
Not seeing the logic there. Of course people might welcome a surprise "holiday". It's not like our parents loved their jobs so much they wouldn't have enjoyed an unexpected break just as much.
After he was retired for a few years, I asked my dad if he missed anything about working. He said "nothing." Even after pressing him he insisted that he hated it the whole time, and it was just something he did to support the family. Of course, he never complained to me about it a single time. I think it's just become more normalized to complain about your job.
My job today was for a ~700 person organisation that had no email and no teams for the entire day. Thankfully I mostly build physical things for them and it made basically no difference.
My partner works for an organisation of more like 7000 and they had no email and no files (sharepoint). For the whole day. The $ involved are insane
Yesterday afternoon (US eastern time), I was packing up for the day when one of our Tier 1 guys came over as he was getting ready to leave and said, "You guys hear there's a Microsoft outage?" I responded with, "Oh really? Huh. Well, I'm going home." So I went home, as did everyone else.
While I and my team manage our Microsoft tenant, we only use Azure, Intune, and Entra. We do use 365, but that's an entirely different tenant and an entirely different team/department within our agency who runs all that. Confusing, but there are reasons for this. Either way, it'll be interesting going into work today to see if we have access to 365. That said, we're a government agency, so maybe we're not as affected? I didn't know anything was amiss yesterday before I left, until my coworker said something.
† "365" does not refer to uptime or availability. In fact, 365 refers to nothing, similar to 'KFC.' It's simply a number we pulled out of our ass. It does not and never did refer to the number of days in a year. A totally random number with no meaning.
<noise>
Well, this is going to be an interesting Friday. It's 8 AM here, I have 4 Teams meetings scheduled before noon and about 2 dozen e-mails I wanted to work through in between those...
crialpaca | 18 hours ago
Today I learned that I'm a manager. (Melting emoji face.)
No leadership in the office during this nightmare. We manage our tasks through Outlook. Apparently I'm the only one who knows how to find all of our work outside of outlook, so I ended up creating task lists and parceling them out to people so we could keep going (we were already behind and short staffed before today).
I need to remember to put a comment down for my performance review about how I exceeded expectations today.
ThrowdoBaggins | 17 hours ago
I’ve got a little folder where I keep things like this, and stuff like positive feedback from executives etc, so that when the reviews come around, I have it all in one place. It’s just in a little email folder in my Outlook inbox... oh.
hobbes64 | 6 hours ago
It’s a good reason to use Obsidian for notes instead of a Microsoft product or anything that uses cloud storage or a proprietary format.
ThrowdoBaggins | 13 minutes ago
I’ve been tempted by Obsidian in my personal life for some time, but in the workplace our machines are very locked down, I don’t think I’d be able to install it. Does Obsidian have a runs-in-browser version or something similar?
redwall_hp | 18 hours ago
My office not only uses Outlook to schedule meetings, but the meeting room bookings and telepresence systems are integrated with it...so tomorrow could be interesting. (Fortunately I didn't have meetings later in the day today.)
CannibalisticApple | 19 hours ago
Was just about to post an article! I don't use any Microsoft programs, so I was surprised to hear it's still ongoing. It's apparently impacting anything that communicates with Exchange Online and any functions related to it, which turns out to be a lot of things.
Worse: they apparently haven't identified the root cause yet, which is crazy given it's gone on so long.
Second most recent status at the time of this post from the cloud status page:
Which as far as I can tell means "we have no clue what caused this and are just trying anything we can, but we don't think it was a hacker or breach." Not sure if that makes this better or worse. On the one hand, yay security... On the other hand, that means MS holds all the blame and the company's competence is now in question.
Then while writing this comment a new status update dropped:
Which I think means: "we're trying to get servers online by messing with traffic, but it only works until it gets traffic and then it goes down again. [Probably because the root problem causing this mess is still there and unidentified.]"
Last part is my addition. Feels like they might be trying to just treat the symptoms now...
CannibalisticApple | 18 hours ago
By the way, if anyone wants a laugh, check out the r/sysadmin subreddit. Went there to check for updates and translations of the gobbledygook from Microsoft. (Typically translates to "still no clue, have some word salad so it's less obvious.") Lots of late-night snark and some hopes to enjoy a nice three-day weekend. (Australia may have already gotten it.)
And one person made the point that companies will have a bunch of emails queued up today to send as soon as the servers are back online, citing 10k from one company of unknown size. So even when they do fix it, the servers are going to be dealing with an onslaught of emails. (Which might explain why the servers crash whenever they get them back online...)
Also for some reason I'm now getting flashbacks to email storm sagas. Given this is outlook, would be hilarious if this is somehow the result of an email hurricane.
l_one | 17 hours ago
I love one of the names for that - 'Reply Allpocalypse'.
teaearlgraycold | 5 hours ago
Pretty common issue, it even has a name: Thundering Herd Problem.
redwall_hp | 18 hours ago
This is the company that also has a GitHub outage practically every day. They don't even have two nines of service, unless 89.9% counts. They're a joke.
https://mrshu.github.io/github-statuses/
teaearlgraycold | 5 hours ago
My org's self-hosted Forgejo that runs on a single Hetzner node with high memory pressure causing it to OOM all of the time has better uptime than Github.
derekiscool | 17 hours ago
As a former employee of
MicrosoftMicroslop (and even part of the org that contains Outlook) - the only thing that shocks me is that this doesn't happen every other month.The management there is so astoundingly bad at letting engineers do their job. This is only going to make it worse for them too. Hopefully other companies take note that this is what happens when you lay off thousands of talented engineers because you think AI can replace them.
Knockout_Mouse | 19 hours ago
Had a fun talk with my boss today. He wanted to discuss an email he sent me today about a problem with one of our older apps running TLS 1.0 and how it's going to get fully blocked by the corporate firewall changes in less than a week.
I had to remind him I hadn't seen the email because of the international 365 outage.
He also mentioned how he felt like he was going through withdrawal because Copilot wasn't working.
Strange times.
patience_limited | 16 hours ago
So the company I work for got acquired last month, and we've all been jittery about the pending impacts of migration of our domain and Microsoft 365/Azure instances. Far too much of our workflow, operations, and institutional knowledge relies on that single provider (and we're going to lose Slack in favor of Teams, which is just :facepalm).
When the outage started yesterday, most of us thought it was the new mothership flubbing configuration or the old owners cutting off services prematurely, until the new e-mail addresses stopped responding as well. I was the first to try and check the Microsoft 365 status page and found it loading a blank screen, then hit the r/sysadmin thread... The gibberish status posts from Microsoft suggest things that should never happen with well-managed regionalized cloud infrastructure.
Well done, M$.
And it's extra special that this is the busiest season of the year, as we try to wrap up projects before healthcare customers (nearly all of whom have also gotten overly Microsoft-dependent) enter holiday change freezes. A quiet Inbox is not what I'm looking for with major go-lives and cutovers happening next week.
kingofsnake | 9 hours ago
I'm all for bringing back M$ as shorthand.
ThrowdoBaggins | 18 hours ago
This has been one of the most peaceful Fridays I’ve had in a long time, and it was so satisfying to be left alone to actually get everything from my backlog knocked down before the weekend.
Walking in on Monday is going to be so crazy by comparison!
hobbes64 | 6 hours ago
Sadly, Ms Teams wasn’t also down so a lot of people are interrupted through other channels
jredd23 | 6 hours ago
My phone is on mute.
SamusAu | 18 hours ago
MS updates read like an AI summary of a longer AI slop report. I really wonder how much of this has been (mis)managed by AI.
Zorind | 18 hours ago
According to this series of Substack posts from last year (saw linked in /r/sysadmin), Azure has been mismanaged for a while. I did not attempt to do any sort of vetting of this, so could be complete hogwash.
https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion
[OP] LukeZaz | 19 hours ago
From the article:
[...]
Between this and Nvidia, these last couple days sure have been interesting.
dhcrazy333 | 19 hours ago
Our work place has its entire email down basically the entire day. Teams still worked though…
Mopeybloke | 12 hours ago
Has any move away from Microsoft been considered aloud?
vord | 9 hours ago
I mean, there were jokes from the CIO about mirroring everything to Gmail, but that probably becomes less of a joke every outage.
I followed up with an IRC/postfix/LDAP/KRB server, and got a "Let's put that on an LTO."
Maybe all these constant outages will be what finally causes big shops to reconsider why on-prem was a thing to begin with.
PraiseTheSoup | 19 hours ago
I was at work all day and our outlook went out around 11am and was still out when I left at 8pm. Pretty embarrassing for a company like Microsoft.
preposterous | 16 hours ago
Just ask copilot to fix it. Oh wait, it’s down and it’s probably what created the outage in the first place.
preposterous | 4 hours ago
Saw this elsewhere: “the fact that the world is cheering the outlook outage as a national holiday tells more about the state of burnout in the workplace than any HR survey ever will”
cdb | 2 hours ago
Not seeing the logic there. Of course people might welcome a surprise "holiday". It's not like our parents loved their jobs so much they wouldn't have enjoyed an unexpected break just as much.
After he was retired for a few years, I asked my dad if he missed anything about working. He said "nothing." Even after pressing him he insisted that he hated it the whole time, and it was just something he did to support the family. Of course, he never complained to me about it a single time. I think it's just become more normalized to complain about your job.
kingofsnake | 22 minutes ago
While I'm working in the field I studied for, I like everybody else at a desk didn't imagine that they'd be just one more sucker in an "Email job".
But her, we also didn't think that email jobs would become all jobs.
lostwax | 14 hours ago
My job today was for a ~700 person organisation that had no email and no teams for the entire day. Thankfully I mostly build physical things for them and it made basically no difference.
My partner works for an organisation of more like 7000 and they had no email and no files (sharepoint). For the whole day. The $ involved are insane
JCPhoenix | 13 hours ago
Yesterday afternoon (US eastern time), I was packing up for the day when one of our Tier 1 guys came over as he was getting ready to leave and said, "You guys hear there's a Microsoft outage?" I responded with, "Oh really? Huh. Well, I'm going home." So I went home, as did everyone else.
While I and my team manage our Microsoft tenant, we only use Azure, Intune, and Entra. We do use 365, but that's an entirely different tenant and an entirely different team/department within our agency who runs all that. Confusing, but there are reasons for this. Either way, it'll be interesting going into work today to see if we have access to 365. That said, we're a government agency, so maybe we're not as affected? I didn't know anything was amiss yesterday before I left, until my coworker said something.
vord | 9 hours ago
Good news! Intune is affected, so you don't need to feel left out!
JCPhoenix | 8 hours ago
Huzzah! We belong!
But actually, we haven't seen nor heard of any issues affecting users or devices. So it's just another Friday for us. Thank god.
hobbes64 | 6 hours ago
I hope they rename the product to 364 or 363 to avoid false advertising
JCPhoenix | 5 hours ago
They'll just toss it in the fine print:
† "365" does not refer to uptime or availability. In fact, 365 refers to nothing, similar to 'KFC.' It's simply a number we pulled out of our ass. It does not and never did refer to the number of days in a year. A totally random number with no meaning.
snowgoon | 12 hours ago
We are getting some emails now. One of my colleagues just got an invite to a meeting that ended 30 minutes ago.
NonoAdomo | 8 hours ago
I got one that was for a meeting that was held yesterday at 8:30 this morning. There's quite the backlog in the system right now.
Landhund | 16 hours ago