On Wednesday 29 April at approximately 1:06 PM AEST, one of our upstream carriers experienced a network outage that impacted some of our voice and internet services.
The issue was caused by a configuration error on one of their core routers, which took a key piece of hardware offline. The fault cascaded into wider routing disruptions across their network.
Remote recovery wasn't possible, so an engineer was dispatched to the datacentre. Services were fully restored at 2:08 PM same-day, and have remained stable since.
The vendor has identified the root cause as human error during a configuration change. They are tightening their pre-deployment validation and change control processes, accelerating the migration of legacy systems off the affected site, and redesigning their core routing architecture to remove single points of failure.
We apologise for any disruption this caused and appreciate your patience while the issue was resolved. If you have any questions or believe you're still experiencing related issues, please get in touch.