Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition. customers may experience issues accessing the Azure Portal. Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation.
We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:57 UTC on 29 October 2025
---
Update: 16:35 UTC:
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:35 UTC on 29 October 2025
---
Azure Portal Access Issues
We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.
This message was last updated at 16:18 UTC on 29 October 2025
Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.
We have failed the portal away from Azure Front Door (AFD) to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.
We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.
This message was last updated at 17:17 UTC on 29 October 2025
"We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes."
"This message was last updated at 18:11 UTC on 29 October 2025"
At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.
This message was last updated at 19:57 UTC on 29 October 2025
I think we weren't paying for support and it was standard Business Support they were pitching. At the time we were having pretty fundamental problems with Azure Single Server Postgres which was really just a terribly engineered solution which they admitted had some nasty issues (there was some bug that would cause the storage IO threads to deadlock causing Postgres to crash)
in many cases: no service health alerts, no status page updates and no confirmations from the support team in tickets.
still we can confirm these issues from different customers accross europe. Mostly the issues are regional dependent.
Where do these alerts supposedly come from? I started having issues around 4PM (GMT), couldn't access portal, and couldn't make AKV requests from the CLI, and initially asked our Ops guys but with no info and a vague "There may be issues with Portal" on their status page, that was me done for the day.
This is the single most frustrating thing about these incidents. As you're harmstrung on what you can do or how you can react until Microsoft officially acknowledges a problem. Took nearly 90mins both today and when it happened on 9th October.
It's pretty unlikely. AWS published a public 'RCA' https://aws.amazon.com/message/101925/. A race condition in a DNS 'record allocator' causing all DNS records for DDB to be wiped out.
I'm simplifying a bit, but I don't think it's likely that Azure has a similar race condition wiping out DNS records on _one_ system than then propagates to all others. The similarity might just end at "it was DNS".
That RCA was fun. A distributed system with members that don't know about each other, don't bother with leader elections, and basically all stomp all over each other updating the records. It "worked fine" until one of the members had slightly increased latency and everything cascade-failed down from there. I'm sure there was missing (internal) context but it did not sound like a well-architected system at all.
THIS is the real deal. Some say it's always DNS but many times it's some routing fuckup with BGP. two most cursed 3 letter acronym technologies out there
Whilst the status message acknowledge's the issue with Front Door (AFD), it seems as though the rest of the actions are about how to get Portal/internal services working without relying on AFD. For those of us using Front Door does that mean we're in for a long haul?
They briefly had a statement about using Traffic Manager to work with your AFD to work around this issue, with a link to learn.microsoft.com/...traffic-manager, and the link didn't work. Due to the same issue affecting everyone right now.
They quickly updated the message to REMOVE the link. Comical at this point.
Yeah, I am guessing it's just a placeholder till they get more info. I thought I saw somewhere that internally within Microsoft it's seen as a "Sev 1" with "all hands on deck" - Annoyingly I can't remember where I saw it, so if someone spots it before I do, please credit that person :D
It's a Sev 0 actually (as one would expect - this isn't a big secret). I was on the engineering bridge call earlier for a bit.
The Azure service I work on was minimally impacted (our customer facing dashboard could not load, but APIs and data layer were not impacted) but we found a workaround.
yea I saw that, but im not sure on how accurate that is. a few large apps/companies I know to be 100% on AWS in us-east-1 are cranking along just fine.
We already had to do it for large files served from Blob Storage since they would cap out at 2MB/s when not in cache of the nearest PoP. If you’ve ever experienced slow Windows Store or Xbox downloads it’s probably the same problem.
I had a support ticket open for months about this and in the end the agent said “this is to be expected and we don’t plan on doing anything about it”.
We’ve moved to Cloudflare and not only is the performance great, but it costs less.
Only thing I need to move off Front Door is a static website for our docs served from Blob Storage, this incident will make us do it sooner rather than later.
we are considering the same but because our website uses APEX domain we would need to move all DNS resolver to cloudfront right ? Does it have as a nice "rule set builder" as azure ?
Unless you pay for CloudFlare’s Enterpise plan, you’re required to have them host your DNS zone, you can use a different registrar as long as you just point your NS records to Cloudflare.
Be aware that if you’re using Azure as your registrar, it’s (probably still) impossible to change your NS records to point to CloudFlare’s DNS server, at least it was for me about 6 months ago.
This also makes it impossible to transfer your domain to them either, as CloudFlare’s domain transfer flow requires you set your NS records to point to them before their interface shows a transfer option.
In our case we had to transfer to a different registrar, we used Namecheap.
However, transferring a domain from Azure was also a nightmare. Their UI doesn’t have any kind of transfer option, I eventually found an obscure document (not on their Learn website) which had an az command which would let you get a transfer code which I could give to Namecheap.
Then I had to wait over a week for the transfer timeout to occur because there is no way on Azure side that I could find to accept the transfer immediately.
I found CloudFlare’s way of building rules quite easy to use, different from Front Door but I’m not doing anything more complex than some redirects and reverse proxying.
I will say that Cloudflare’s UI is super fast, with Front Door I always found it painfully slow when trying to do any kind of configuration.
Cloudflare also doesn’t have the problem that Front Door has where it requires a manual process every 6 months or so to renew the APEX certificate.
Thanks :). We don't use Azure as our registrar. It seems I'll have to plan for this then, we also had another issue, AFD has a hard 500ms tls handshake timeout (doesn't matter how much you put on the origin timeout settings) which means if our server was slow for some reason we would get 504 origin timeout.
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:35 UTC on 29 October 2025
----
Azure Portal Access Issues
We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.
This message was last updated at 16:18 UTC on 29 October 2025
I'd say, people that need it. Which could be the same for all the other models out there.
To create one model that is great at everything is probably a pipedream. Much like creating a multi-tool that can do everything- but can it? I wouldn't trust a multi-tool to take a wheel nut off a wheel, but I would find it useful if I suddenly needed a cross-head screw taken out of something.
But then I also have a specific crosshead screwdriver that is good at just taking out cross-head screws.
Use the right tool for the right reason. In this case, there maybe a legal reason why someone might need to use it. It might be that this version of a model can create something better that another model can't. It might be that for cost reasons you are within AWS, that it makes sense to use a model at the cheaper cost than say something else.
So yeah, I am sure it will be great for some people, and terrible for others... just the way things go!
> Do you still have to quit and restart the entire application after you give the permission?
The funny thing about this, even on Sonoma - I could click the button to allow it, when it said "restart app" I closed the box (or clicked cancel), and it worked anyway. Specifically, I noticed it more on things like Teams/Zoom where I was doing a screen share, it just "worked" - no need to restart the entire application.
Refreshed before typing this because I realised someone might have beaten me to it! - But that's a big difference here - even though the service is gone, you got the refund and still a usable device as a controller out of it...
Spotify has taken something that could be used generically too, and just decided to brick it.
Insert something about product and consumers and how its all just some big money game or something somewhere :D
Uniqlo has something similar to this - they use RFID's on their stock so you can dump it into a bucket at the self-checkout, and it scans everything immediately [1]. You still have someone pop over to check if you are OK, but it is a lot quicker than self-scan or usually waiting for someone.
Oddly enough, I am in the UK - and I do have it, but it was already turned off when I went there. I wonder if things have changed, or there are some canary releases of the box... or am I just completely unaware my account isn't considered a UK-based account?
I did a few things in my younger days - I used to like playing MUDs, and one day a few of my college friends wanted to create our own. So we created a fairly unknown MUD called "Faereal" which still happens to be used as my domain name for my personal stuff!
I was lucky enough to have a good friend and neighbour down the road who ran ExNet [1], who provided me with space to host my first server, and oh boy looking back, I am surprised I didn't blow everything up! [2] - Windows 98 connected directly into the internet, with a fairly terrible firewall and some random remote control software I found!.
Eventually, though another MUD, we were donated a more up-to-date box, which ran Linux, and we hosted that MUD and the Faereal MUD for a while, eventually adding in my own DNS server, website hosting (PHP), and that is how I ended up hosting friends websites.
That turned into a hobby where I started to write my own PHP, started helping firstly helping out on a game called "PhaseOne" which was essentially a copy of a game we were all playing at the time called "Planetarion" [3] -- (OMG As I looked for this, its still running!). Part of this code I created a "Team based chat area", which eventually became the primary base for something that has taken over nearly 20 years of my life.
The code became the custom-written forum code behind DDR:UK, a Dance Dance Revolution fan website for the UK, which through the founders we created the "official" Sim Packs for DDR simulators such as DWI [4] and Stepmania [5]. This eventually moved into us working at events such as the London MCM ComicCon [6], where we bought in actual DDR arcade machines, including a Stepmania run DDR Machine that used to sit in the Namco Station in Central London on the South Bank. (I would love to say it was a world first, but there was one group in the US that had a temporary setup... I would like to hope we are the world first permanent money-making one :D)
That got me into running a Japanese Culture Festival called Tokonatsu [7] which got me into learning AWS. This festival has now been running for 20 years!
So all in all, how did this help:
* Interviews, it's a great story to tell, and I always get a lot of fun looks!
* Experience, from hardware, to networking, to early days of internet, software, hosting etc etc. I went thought a LOT of sleepless nights when I was younger sorting this out, gave me a whole bunch of experience that I would never would have had.
* Networking, still talk to a lot of people today, and these people are key for where I am.
Honestly, the owner of ExNet, I couldn't have done any of this, if he hadn't of started me on the right path.
EDIT: Totally forgot to explain where I am now! So with all this, through support tech, manager of of datacentres, through lead engineers, etc etc... I am now the AWS Practice Lead for my company, a Principle Consultant, and I am writing this in the airport on the way back from AWS Re:Invent 2023 :D
So yeah, that is my story! Hope someone does eventually read it :D
It's currently Re:Invent 2023 [1], where AWS usually "store up" announcements for the week. Meaning a load of product announcements are being released in quick succession. That would explain a higher than usual number of articles and links being put up.
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition. customers may experience issues accessing the Azure Portal. Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation.
We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:57 UTC on 29 October 2025
---
Update: 16:35 UTC:
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:35 UTC on 29 October 2025
---
Azure Portal Access Issues
We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.
This message was last updated at 16:18 UTC on 29 October 2025
---
Message from the Azure Status Page: https://azure.status.microsoft/en-gb/status