Remember that monster outage in June when Fastly managed to take down huge chunks of the internet? The CEO of the US-based cloud computing services provider has admitted some customers still do.
The blackout happened on June 8 and the following day, in a post incident report, Fastly explained it was due to an undiscovered software bug triggered by a customer’s “valid configuration change,” which caused 85 per cent of its network to return an error message.
CEO Joshua Bixby said yesterday the bug was detected within one minute of it tripping swathes of the network, and engineers “returned 95 per cent of our network to normal within 49 minutes,” he told analysts on a calendar Q2 earnings call.
“That being said, our customers were negatively impacted. As a result, we saw traffic volumes decrease and subsequently issued credits to select customers following the incident.
“Given the usage-based nature of our business model this resulted in impact to our Q2 results. And we expect to see some downstream impact on revenue from the outage in the near to medium-term, as we work with our customers to bring back their traffic to normal levels,” he added.
A “couple of customers”, including one in the “top ten,” “have not yet returned traffic back to the platform post outage,” Bixby confirmed.
Separately, “We also have some additional uncertainty with the timing of several customers ramping additional traffic onto the platform in the second half of the year”. He said this was not related to the June outage: “these are timelines that are based on situations that are either regulatory or licensing, for example”.
“Those are delayed for things unrelated to the work that we’re doing internally and unrelated to the outage for the most part,” he added.
Akamai Edge DNS goes down, takes a chunk of the internet with it
What knocked out Brit cloud slinger Memset for the night? A busted fibre cable upstream of its data centre, apparently
Excuse me, what just happened? Resilience is tough when your failure is due to a ‘sequence of events that was almost impossible to foresee’
Fastly ‘fesses up to breaking the internet with an ‘an undiscovered software bug’ triggered by a customer
Bixby said the company expects this traffic to come onto the network this year. He told analysts, “our confidence remains high” and customer churn in general was still “very low”.
“We have implemented and will continue to implement significant measures to ensure increased resiliency for our customers and their users.”
Fastly didn’t quantify the financial impact of the outage on its Q2 results for the three months ended 30 June, which showed 14 per cent year-on-year revenue growth to $85m. Total customer count increased to 2,581 from 2,458 in Q1.
“There’s sort of two impacts,” said CFO Adriel Lares, “One, which is the impact of the outage and those customers at the time that had not yet ramped or brought that traffic… So you could imagine that the impact for lost revenue, lost traffic occurred in June. The other impact… the actual sort of credits that we actually ended up giving out.”
She said the business’ financials were still within its guided range, “despite the credits” payment, which she then described as “relatively immaterial”, and was thankful it “wasn’t longer”
Enterprise customers grew to 408 from 395 and their average spend was $702k compared to $705k. It reported a loss before tax of $58.45m, up from a pre-tax loss of $14.43m.
“Aside from the outage, the business performed as expected,” said Lares. She added: “the outage along with the uncertainty related to the timing of returning traffic, new initiatives and new customers ramping traffic have had an impact on our third quarter and full year guidance.”
Revenue guidance for Q3 is in the range of $82m to $85m.
Outages do happen, but it is the way that companies handle them which sorts the adults from the children. Fastly was relatively transparent in its dealings, explaining what went wrong the day after the outage occurred. And while that won’t have appeased all customers, it is a facet.