Xbox explains what caused its almost day-long network outage

"This is an unacceptable situation," admits chief technology officer.

Xbox chief technology officer Scott Van Vliet has gone to great lengths to detail what caused an almost day-long network outage earlier in the week, admitting that “this is an unacceptable situation, and we must do better to support you.”

What caused the Xbox network outage

“Late last night, a licensing service that sits outside of Xbox, but which Xbox depends upon, began failing,” Van Vliet wrote on Twitter.

“This caused some sign-in scenarios to fail, and it caused many scenarios that require an entitlement check to also fail, such as listing out your Full Library and launching games that you own,” he continued. “It also affected several of our publishing and store partners who depend on those same systems which is why some of you saw issues in specific games and not others.”

As we noted, the Xbox network began to fail at around 2.00 pm AEST on 27 July, and wasn’t fully back online until around 9.30 am on 28 July.

Van Vliet thanked players for their patience, adding, “nonetheless, this is an unacceptable situation, and we must do better to support you.”

What Xbox did, and what’s next

Van Vliet’s post then detailed what his teams did to try to correct the problem, and what will happen now that the situation has been contained.

“Our on-call teams caught the first signals overnight through automated monitoring and declared a major incident,” the CTO wrote. “Once we isolated the failing infrastructure, we shifted traffic onto the healthy parts of the system while the team continued to work towards a root cause. Service came back unevenly across regions which is why some of you recovered hours before others.”

Moving forward, Van Vliet has promised a “full post-incident review” which will focus on “why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can’t ruin your night again.”