Router failure in Farnborough

We’ve seen no issues following this work, and are confident this issue is now resolved. Apologies for any disruption you’ve suffered during the incident and associated remedial work.

If you would like to discuss this incident with us please contact your account manager on 01252 560565.

This work has been completed. We will monitor closely overnight.

This work is commencing now.

Due to some complications this evening we’ve decided to hold on this evenings second maintenance window, and instead we are re-scheduling that to tomorrow.

The new window for work is:

• Wednesday 8th Jan 2020, 6pm till 7pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

We will be replacing the faulty router with a new one tonight. Work will take place in two windows:

• 4pm till 6pm, no expected customer impact
• 7pm till 8pm, 10 mins of expected disruption

During the 10 mins of disruption most will not notice anything, however you may see a short burst of packet loss and/or monitoring alerts while traffic changes path.

This post will be updated once work is complete.

This same router caused us issues again on Sunday, so we have now taken it out of service completely.

There were two periods of impact on Sunday, approximately 4 minutes on each occurrence:

• 17:18 until 17:22
• 18:48 until 18:52

A replacement router will be installed later this week, and this post will be updated with times when we know more.

The router is now back in service. There was a brief (less than 30 seconds) blip of packet loss during the switch over. We will monitor the service closely, however we believe this issue is now resolved.

We are about to bring the impacted router back in to service. There is a possibility of extremely minor disruption of less than a minute while the BGP routes converge. This should take no longer than 20 minutes in total, so there will be another update here in the next 30 minutes.

Customers with firewalls and VMs in separate sites who are mid-migration should now be back online.

For some customers who are in between migration between data centres as part of the move out of Byfleet this is impacting service. We are attempting a work around for that now and will have an update shortly.

A core router has failed in Farnborough which has reduced redundancy (as it is a high availability pair of routers at that site). It will have also caused packet loss for a short period of time while traffic re-routed.

We are investigating remotely, and an engineer is en route to the data centre to investigate in person.