Salesforce engineers roll back change after breaking own cloud for hours today

Updated If you noticed something funky going on with Salesforce and its software-as-a-service empire today, it's not you: it's recovering from an hours-long outage.

As of writing, Salesforce said at 1829 UTC (1130 PT) in a status update that all of its clouds are returning to normal after suffering about a four-hour downtime.

Well, all except for Salesforce's Tableau and Mulesoft, which remain down or affected to some extent.

The IT breakdown started at 1448 UTC, and we're told by the enterprise software giant this hit "customers across multiple clouds including, Commerce Cloud, Mulesoft, Tableau, Core, Marketing Cloud Account Engagement, Marketing Cloud Intelligence, and Omni Channel."

As a result, "users are unable to log into Salesforce or access any of their services," the biz admitted.

Two hours later, the team said its ClickSoftware, Trailblazer, and Data Cloud products were also affected. At first a third-party cloud provider was thought to be to blame for intermittent networking issues causing Salesforce services to fail.

Then the tech titan said, actually, a cloud provider wasn't at fault, and Salesforce engineers "successfully executed a rollback to mitigate the issue." By 1752 UTC, the biz said its cloud systems were getting back on track, and customers should be able to log in and use their applications as usual, as long as those apps weren't Tableau or Mulesoft.

• Azure SQL Database takes Saturday off on US east coast following network power failure
• Microsoft admits slim staff and broken automation contributed to Azure outage
• Square blames last week's outage on DNS screw-up
• Gartner: Stop worrying and love the cloud, with all its outages and lock-in

Us vultures, as Slack users, noticed a blip in connectivity with that Salesforce-owned chat app, for what it's worth, though Slack is not reporting any substantial outage. It's possible Slack was slightly affected as a knock-on from the outage or efforts to recover from it.

For those relying on the analytics platform Tableau: although Tableau Public has apparently been fully restored, Tableau Cloud is still screwed, we're told, some four hours after a problem was first detected. A status update two hours ago stated:

Meanwhile, 20 minutes ago, automation toolkit Mulesoft said its services are recovering after a fix was applied.

We think it's safe to say Salesforce broke part of its back-end systems with an update, causing services and products relying on that infrastructure to fail, and customers unable to log in and use the stuff as expected. Backing out that change has helped fix things. We'll keep you updated. ®

Updated to add at 1920 UTC

It appears Mulesoft has joined the rest of the Salesforce collective in recovering from the earlier outage though Tableau is awaiting the wider deployment of an update to right its services. In an update, the SaaS giant said:

We've also asked Salesforce for further info, such as how exactly it broke its clouds and what steps are being taken to prevent that from happening again.