Postmortem -
Read details
May 1, 17:04 AEST
Resolved -
On Wednesday 29 April at approximately 1:06 PM AEST, one of our upstream carriers experienced a network outage that impacted some of our voice and internet services.
The issue was caused by a configuration error on one of their core routers, which took a key piece of hardware offline. The fault cascaded into wider routing disruptions across their network.
Remote recovery wasn't possible, so an engineer was dispatched to the datacentre. Services were fully restored at 2:08 PM same-day, and have remained stable since.
The vendor has identified the root cause as human error during a configuration change. They are tightening their pre-deployment validation and change control processes, accelerating the migration of legacy systems off the affected site, and redesigning their core routing architecture to remove single points of failure.
We apologise for any disruption this caused and appreciate your patience while the issue was resolved. If you have any questions or believe you're still experiencing related issues, please get in touch.
May 1, 17:04 AEST
Monitoring -
A fix has been implemented, and we are seeing services coming back online. Please contact our helpdesk on 1300 724 817 if you cannot access the internet, or are having problems making phone calls. Further information will be provided once we receive an incident post-mortem from upstream vendor(s).
Apr 29, 14:19 AEST
Update -
The fault appears to be a network or routing problem with one of our upstream carriers. As such, in addition to internet outages, some voice calls may not connect. We are working with the vendor to troubleshoot and identify the fault.
Apr 29, 13:59 AEST
Investigating -
We are investigating reports of an NBN internet outage affecting multiple customers. More information will follow.
Apr 29, 13:45 AEST