Router failure in Farnborough

Minor incident Gen4 Gen3 Public Cloud
14 days, 5 hours, 30 minutes

Updates

Resolved

We’ve seen no issues following this work, and are confident this issue is now resolved. Apologies for any disruption you’ve suffered during the incident and associated remedial work.

If you would like to discuss this incident with us please contact your account manager on 01252 560565.

January 9, 2020 · 18:13 BST
Monitoring

This work has been completed. We will monitor closely overnight.

January 8, 2020 · 18:44 BST
Update

This work is commencing now.

January 8, 2020 · 18:02 BST
Update

Due to some complications this evening we’ve decided to hold on this evenings second maintenance window, and instead we are re-scheduling that to tomorrow.

The new window for work is:

  • Wednesday 8th Jan 2020, 6pm till 7pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

January 7, 2020 · 18:29 BST
Investigating

We will be replacing the faulty router with a new one tonight. Work will take place in two windows:

  • 4pm till 6pm, no expected customer impact
  • 7pm till 8pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

This post will be updated once work is complete.

January 7, 2020 · 14:12 BST
Investigating

This same router caused us issues again on Sunday, so we have now taken it out of service completely.

There were two periods of impact on Sunday, approximately 4 minutes on each occurrence:

  • 17:18 until 17:22
  • 18:48 until 18:52

A replacement router will be installed later this week, and this post will be updated with times when we know more.

January 6, 2020 · 16:49 BST
Resolved

The router is now back in service. There was a brief (less than 30 seconds) blip of packet loss during the switch over. We will monitor the service closely, however we believe this issue is now resolved.

December 26, 2019 · 22:09 BST
Update

We are about to bring the impacted router back in to service. There is a possibility of extremely minor disruption of less than a minute while the BGP routes converge. This should take no longer than 20 minutes in total, so there will be another update here in the next 30 minutes.

December 26, 2019 · 21:25 BST
Investigating

Customers with firewalls and VMs in separate sites who are mid-migration should now be back online.

December 26, 2019 · 13:05 BST
Investigating

For some customers who are in between migration between data centres as part of the move out of Byfleet this is impacting service. We are attempting a work around for that now and will have an update shortly.

December 26, 2019 · 12:52 BST
Investigating

A core router has failed in Farnborough which has reduced redundancy (as it is a high availability pair of routers at that site). It will have also caused packet loss for a short period of time while traffic re-routed.

We are investigating remotely, and an engineer is en route to the data centre to investigate in person.

December 26, 2019 · 12:46 BST