Incident - Unable to access GXP Manager

Incident Report for IRIS Software Systems Ltd

Resolved

We have continued to monitor our service over the past 12 hours and there have been no further issues. Microsoft, have also provided a PIR for the documented outage yesterday, for more information please see https://azure.status.microsoft/en-us/status/history/

Preliminary Post Incident Review (PIR) - Azure Front Door - Connectivity issues across multiple regions
Tracking ID: YKYN-BWZ

We would like to take this opportunity to apologise for any inconvenience that this may have caused. If you are continuing to experience difficulties, please contact the support team at support@iris.net or +44 203 322 5690 for assistance.
Posted Oct 30, 2025 - 10:55 GMT

Update

We are continuing to monitor for any further issues.
Posted Oct 30, 2025 - 08:57 GMT

Update

GXP, IRIS Manager and guest facing services are now fully operation again, including images and we are now seeing orders moving through the service again.

Microsoft are continuing to work on a full solution to the issue, and recovery in progress, IRIS will continue to monitor our services while they continue to keep the issue open

https://azure.status.microsoft/en-us/status

Azure Front Door - Connectivity issues - Observing recovery


Starting at approximately 16:00 UTC on 29 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Affected Azure services include, but are not limited to: App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID, Microsoft Purview, Microsoft Sentinel, Video Indexer, and Virtual Desktop.

Current status: We initiated the deployment of our ‘last known good’ configuration, which has now successfully completed. We are currently recovering nodes and re-routing traffic through healthy nodes.

As recovery progresses, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability until more nodes are fully restored. This recovery effort involves reloading configurations and rebalancing traffic across a large volume of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover. We expect continued improvement across affected regions. This means we expect recovery to happen by 23:20 UTC on 29 October 2025

Customer configuration changes remain temporarily blocked to prevent new deployments that could interfere with recovery. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

We will provide another update on our progress within two hours, or sooner if warranted. Although we are seeing signs of recovery, customers may also consider implementing existing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure. https://learn.microsoft.com/azure/frontdoor/routing-methods

This message was last updated at 21:04 UTC on 29 October 2025
Posted Oct 29, 2025 - 21:10 GMT

Monitoring

We are now seeing that GXP, IRIS Manager and guest facing services are now operational again. However, we are still not able to access images on all front end services. We will continue to monitor as service begin to return to normal.

Microsoft continue to work on a resolution, with the latest update. https://azure.status.microsoft/en-us/status

Azure Front Door - Connectivity issues - Observing recovery


Starting at approximately 16:00 UTC on 29 October, 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Current status:

We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

Although we are seeing signs of recovery and have an estimated timeline, customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure.

Learn more about Azure Front Door failover strategies for AFD: https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview

This message was last updated at 19:22 UTC on 29 October 2025
Posted Oct 29, 2025 - 19:45 GMT

Identified

The issue has been identified and confirmed to be related to our vendor - Microsoft. Please refer to their notice for further information.

https://azure.status.microsoft/en-us/status

Azure Front Door - Connectivity issues


Starting at approximately 16:00 UTC, customers and Microsoft services that leverage Azure Front Door (AFD) may have experienced issues resulting in latencies, timeouts and errors. We have confirmed that an inadvertent configuration change as the trigger event for this issue.

We are taking several concurrent actions: Firstly where we are blocking all changes to the AFD services, this includes customer configuration changes as well. At the same time, we are rolling back our AFD configuration to our last known good state. As we rollback we want to ensure that the problematic configuration doesn't re-initiate upon recovery.

Customers may have experienced problems accessing the Azure management portal. We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly, while all portal extensions are working correctly there may be a small number of endpoints that might have a problem loading (i.e. Marketplace).

We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.

While we dont have an ETA yet. customers can consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview

This message was last updated at 17:51 UTC on 29 October 2025
Posted Oct 29, 2025 - 17:58 GMT

Update

We are continuing to investigate this issue.
Posted Oct 29, 2025 - 16:44 GMT

Investigating

We are currently investigating an issue with users not being able to access GXP Manager, or front end Services. It appears that our services have been affected by one of our vendor Microsoft, with the current notice

Azure Portal Access Issues

Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
https://azure.status.microsoft/en-us/status

Our engineering team are investigating, and we expect an update in the next hour.

Services affected: GXP, Guest Web, Guest Mobile, guest Room

Workaround: There is no current workaround

Next update: 17:00 GMT
Posted Oct 29, 2025 - 16:43 GMT
This incident affected: Manager, Manager API, Public API, API Management, CMS API (Europe, Nort America East, South East Asia), Notifications (SMS, Voice, Email), and IRIS Guest (Web, Mobile, Room, Table, Kiosk).