GXP Platform Outage

Incident Report for iRiS Software Systems Ltd

Postmortem

During the evening of the 22nd of October one of the storage services that we depend on entered an inconsistent state. In order to ensure the integrity of the stored data the storage service was brought offline.

A number of our services that depend on the underlying storage failed as a result and this prevented guest messaging, orders and service requests from being processed.

Our guest facing applications delivered content throughout with reduced performance.

Once the storage inconsistencies were resolved and storage was brought online, all other services recovered.

We have investigated the reliance on the storage services and how this is implemented throughout GXP and we will make adjustments that will mean that future potential storage failures will not affect the product to this extent. These changes will be delivered as part of our normal product update process.

We would like to take this opportunity to apologise to our customers and partners for any inconvenience that this outage may have caused.

Please contact your account manager should you have further questions or require more information.

Regards,

iRiS Systems Engineering

Posted Oct 24, 2019 - 17:10 BST

Resolved

This incident is now resolved.

A postmortem is expected in the next 48 hours.

Posted Oct 23, 2019 - 01:51 BST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Oct 23, 2019 - 01:26 BST

Update

We have isolated additional services as being affected by this storage problem and we have updated the affected components list.

We are working with our storage supplier to rectify this fault.

Posted Oct 23, 2019 - 00:08 BST

Identified

We have isolated the fault to a problem with storage. Storage operations are currently failing, we will update here once we have further information.

Posted Oct 22, 2019 - 23:31 BST

Investigating

Our monitoring system has alerted us to the fact that orders are not being processed and manager may not respond.

We are investigating.

Posted Oct 22, 2019 - 23:03 BST

This incident affected: Manager, Manager API, Public API, CMS API (Europe, Nort America East, South East Asia), Integrations (PMS, POS, SOS), and IRIS Guest (Web, Mobile, Room, Table, Kiosk).