AUSTRALIA: ASX Trade Outage Report

Friday September 23 2016

On Monday 19 September 2016, a hardware failure in the main database used to operate the ASX market triggered a number of events that meant the ASX market did not open at 10am. The market was also closed early.

The faulty hardware was replaced that evening. The ASX market opened as usual the next day and has since functioned normally.

Overview

ASX maintains a full replication of its systems in a backup datacentre (the disaster recovery site – DRS), and has automated processes (failover) that enable faulty hardware within the primary datacentre to be seamlessly replaced. ASX regularly tests this failover and also conducts business continuity tests with industry participants and includes testing failover processes. This was last performed and successfully completed on 27 July 2016.

On Monday an unprecedented hardware malfunction triggered a complete database failover to DRS and not all parts of ASX Trade successfully connected to the DRS database. As a result, ASX could not open the market until 11.30am and took action to close the market at 2.05pm.

ASX Actions

Overnight, the failed hardware was replaced, ASX Trade was reset to its standard operating configuration, and the system was validated as being ready for market open. The market opened as normal at 10.00am on Tuesday 20 September and has operated without incident since.

Detail of Events

From 6.57am there was a degradation, and ultimately failure, of the physical database that supports ASX Trade.

As designed, the backup database at the DRS successfully became the new primary database. The sub-systems that comprise ASX Trade and that utilise the database are designed to automatically reconnect to the new primary database. This reconnection process was only partially successful. A number of sub-systems incorrectly maintained connection to the failed database. This malfunction created a series of technical issues. Participants could not log onto the system. Similarly, ASX Trading Operations was not able to log-on and perform certain market functions, although alternative control mechanisms were available. The market did not open as scheduled at 10.00am.

• At 9.45am, ASX announced that the market would not open on schedule and was placed into Enquire.
• At 10:54am, ASX announced that the market would be placed in Pre-Open at 11.10am with a staggered opening commencing at 11.30am

The market operated until 1.42pm when there was a reference data inconsistency, and as a result invalid order acknowledgments were disseminated on Partition 2. ASX subsequently placed the market into Enquire, and recovery and validation procedures commenced.

• At 2.05pm, ASX placed the entire market into Enquire.
• At 3.35pm, ASX announced that the market would not re-open and that there would be no Closing Single Price Auction (CSPA).
• At 5.20pm, ASX announced that closing prices were the last prices traded on the ASX market.

At 11.10am Groups 1-3 (securities A-M) transitioned into Pre-Open and Open as expected.

The opening of Group 4 (securities N-R) and Group 5 (securities S-Z) were impacted due to an operational error.

Group 4 incorrectly opened at 11.10am and remained open until 11.11am during which time a number of trades occurred.

Group 5 did not open as advised and was rescheduled to Pre-Open at 11.40am and Open at 12.00 noon.

Market Actions

• Group 4 was placed into Enquire at 11.11am, and was placed in Pre-Open at 1.00pm and Open at 1.20pm.
• Group 5 was rescheduled to Pre-Open at 11.40am and Open at 12.00 noon.

On Tuesday 20 September, any acknowledgements market participants received for orders in Partition 2 securities on 19 September after matching engine timestamp 13:42:42.3679418 and up until the market was placed into Enquire (2.05pm) were invalid. ASX communicated with all participants impacted.

At 12.47pm on Tuesday 20 September, ASX published notice 1114.16.09 confirming that the invalid transactions were cancelled.

Click on the link above for further details