Performance problem - GXP Manager
Incident Report for iRiS Software Systems Ltd
Postmortem

GXP Manager Performance Degradation

We recently experienced a degradation in peformance affecting GXP Manager. During this event users of GXP Manager may have experienced slow loading times when using the product.

Cause

GXP usage varies throughout an average day and sometimes load can increase or decrease in an unpredictable manner. We continuosly monitor the performance of the plaform, and in cases where our tools detect an increase in load, we automatically provision more resources on a component by component basis.

The thresholds configured to trigger these scale events were in this case too conservative, this in turn caused scaling to be triggered too late. This caused a gap of about 30 minutes where response times was no longer acceptable.

This is indicated by a 30 minute window where the APDEX of GXP Manager dropped from 0.98 (Satisfied) to 0.87 (Tolerated).

Customers could still use the product but some operations could take considerably longer than normal.

Resolution

After investigating we found that in some scenarios the threshold used to trigger a scale event was too conservative. We have altered this value to ensure that scaling happens sooner.

Scaling in itself can cause a temporary performance degradation. This is due to requests being temporarily held back while the scaling takes place. It is therefore a balancing act to ensure that scaling is only performed when needed.

We believe that the new setting will reduce the risk of another event of this kind.

We apologise to any customers that were affected by this event.

Feel free to reach out to support@iris.net should you have any questions or concerns.

Kind regards,

iRiS Systems Engineering

Posted 6 months ago. Jan 25, 2019 - 17:27 UTC

Resolved
This incident has been resolved.
Posted 6 months ago. Jan 24, 2019 - 10:45 UTC
Monitoring
This was identified as an issue with auto scaling, manual scaling have been performed while we work with our supplier to rectify the underlying issues. Response times have now returned to normal values.
Posted 6 months ago. Jan 23, 2019 - 15:48 UTC
Investigating
We are currently experiencing slower than normal response times from GXP Manager.

We are investigating in order to find the cause of the problem. Updates will be posted as information becomes available.
Posted 6 months ago. Jan 23, 2019 - 15:34 UTC
This incident affected: Manager.