Facebook Inc has cited faulty configuration changes on its routers as the root cause of the nearly six-hour outage that prevented the company’s 3.5 billion users from accessing its social media and messaging services.
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Santosh Janardhan, VP engineering and infrastructure wrote in a Facebook blog.
“To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms”, Janardhan wrote.
“We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem.
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication.
“This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.
“Our services are now back online and we’re actively working to fully return them to regular operations.
“We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.
“People and businesses around the world rely on us everyday to stay connected.
“We understand the impact outages like these have on people’s lives, and our responsibility to keep people informed about disruptions to our services.
“We apologize to all those affected, and we’re working to understand more about what happened today so we can continue to make our infrastructure more resilient”.
