GXP Platform Outage
Incident Report for iRiS Software Systems Ltd
Postmortem

During the evening of the 22nd of October one of the storage services that we depend on entered an inconsistent state. In order to ensure the integrity of the stored data the storage service was brought offline.

A number of our services that depend on the underlying storage failed as a result and this prevented guest messaging, orders and service requests from being processed.

Our guest facing applications delivered content throughout with reduced performance.

Once the storage inconsistencies were resolved and storage was brought online, all other services recovered.

We have investigated the reliance on the storage services and how this is implemented throughout GXP and we will make adjustments that will mean that future potential storage failures will not affect the product to this extent. These changes will be delivered as part of our normal product update process.

We would like to take this opportunity to apologise to our customers and partners for any inconvenience that this outage may have caused.

Please contact your account manager should you have further questions or require more information.

Regards,

iRiS Systems Engineering

Posted Oct 24, 2019 - 16:10 UTC

Resolved
This incident is now resolved.

A postmortem is expected in the next 48 hours.
Posted Oct 23, 2019 - 00:51 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 23, 2019 - 00:26 UTC
Update
We have isolated additional services as being affected by this storage problem and we have updated the affected components list.

We are working with our storage supplier to rectify this fault.
Posted Oct 22, 2019 - 23:08 UTC
Identified
We have isolated the fault to a problem with storage. Storage operations are currently failing, we will update here once we have further information.
Posted Oct 22, 2019 - 22:31 UTC
Investigating
Our monitoring system has alerted us to the fact that orders are not being processed and manager may not respond.

We are investigating.
Posted Oct 22, 2019 - 22:03 UTC
This incident affected: iRiS Guest (Web, Mobile, Room, Table, Kiosk), CMS API (Europe, Nort America East, South East Asia), Integrations (PMS, POS, SOS), and Manager, Manager API, Public API.