Router failure in Farnborough

Minor Incident Gen4 Gen3 Public Cloud
14 days, 5 hours, 30 minutes

Update

Resolved

We’ve seen no issues following this work, and are confident this issue is now resolved. Apologies for any disruption you’ve suffered during the incident and associated remedial work.

If you would like to discuss this incident with us please contact your account manager on 01252 560565.

Jan 9th, 2020 18:13 UTC
Monitoring

This work has been completed. We will monitor closely overnight.

Jan 8th, 2020 18:44 UTC
Update

This work is commencing now.

Jan 8th, 2020 18:02 UTC
Update

Due to some complications this evening we’ve decided to hold on this evenings second maintenance window, and instead we are re-scheduling that to tomorrow.

The new window for work is:

  • Wednesday 8th Jan 2020, 6pm till 7pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

Jan 7th, 2020 18:29 UTC
Investigating

We will be replacing the faulty router with a new one tonight. Work will take place in two windows:

  • 4pm till 6pm, no expected customer impact
  • 7pm till 8pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

This post will be updated once work is complete.

Jan 7th, 2020 14:12 UTC
Investigating

This same router caused us issues again on Sunday, so we have now taken it out of service completely.

There were two periods of impact on Sunday, approximately 4 minutes on each occurrence:

  • 17:18 until 17:22
  • 18:48 until 18:52

A replacement router will be installed later this week, and this post will be updated with times when we know more.

Jan 6th, 2020 16:49 UTC
Resolved

The router is now back in service. There was a brief (less than 30 seconds) blip of packet loss during the switch over. We will monitor the service closely, however we believe this issue is now resolved.

Dec 26th, 2019 22:09 UTC
Update

We are about to bring the impacted router back in to service. There is a possibility of extremely minor disruption of less than a minute while the BGP routes converge. This should take no longer than 20 minutes in total, so there will be another update here in the next 30 minutes.

Dec 26th, 2019 21:25 UTC
Investigating

Customers with firewalls and VMs in separate sites who are mid-migration should now be back online.

Dec 26th, 2019 13:05 UTC
Investigating

For some customers who are in between migration between data centres as part of the move out of Byfleet this is impacting service. We are attempting a work around for that now and will have an update shortly.

Dec 26th, 2019 12:52 UTC
Investigating

A core router has failed in Farnborough which has reduced redundancy (as it is a high availability pair of routers at that site). It will have also caused packet loss for a short period of time while traffic re-routed.

We are investigating remotely, and an engineer is en route to the data centre to investigate in person.

Dec 26th, 2019 12:46 UTC