Want some Facebook schadenfreude?

[herding cats]

The gist, as I understand it, is they did this to their own selves. It's in essence a configuration change which broke the entire routing system. Which, apparently due to poor planing on their part, cannot be remotely managed and has to be done in person.

Inside Facebook, workers scrambled because their internal systems also stopped functioning. The company’s global security team “was notified of a system outage affecting all Facebook internal systems and tools,” according to an internal memo sent to employees. Those tools included security systems, an internal calendar and scheduling tools, the memo said.

Employees said they had trouble making calls from work-issued cellphones and receiving emails from people outside the company. Facebook’s internal communications platform, Workplace, was also taken out, leaving many unable to do their jobs. Some turned to other platforms to communicate, including LinkedIn and Zoom as well as Discord chat rooms.

Some Facebook employees who had returned to working in the office were also unable to enter buildings and conference rooms because their digital badges stopped working. Security engineers said they were hampered from assessing the outage because they could not get to server areas.

Facebook’s global security operations center determined the outage was “a HIGH risk to the People, MODERATE risk to Assets and a HIGH risk to the Reputation of Facebook,” the company memo said.

A small team of employees was soon dispatched to Facebook’s Santa Clara, Calif., data center to try a “manual reset” of the company’s servers, according to an internal memo.

https://www.nytimes.com/2021/10/04/technology/facebook-down.html

And a moment of silence is in order for ramenporn... they had to delete their 6 year 30k karma Reddit account for talking about this publicly. 😢

Link to tweet

Archive of his last moments here: https://archive.ph/sMgCi

[Rorey]

1. A company as huge as FB didn't have a backup plan

Absolutely absurd.

[Comfortably_Numb]

2. There's no ad revenue in a backup plan. Algorithms for hate generate cash.

That’s FB top, bottom, and middle line deal.

[rurallib]

3. but they saved M O N E Y!

[herding cats]

4. Right?

The part about being locked out of the buildings is hilarious!

[Rorey]

11. I thought so too

And that they resorted to using Zoom and LinkedIn. Truly embarrassing for them.

[herding cats]

12. I loved that, too!

And how they're having to use Twitter to try and reach out to their disgruntled users. 😂

[Gidney N Cloyd]

5. At least a failover for the critical internal systems.

[uponit7771]

10. I worked at a place bigger than FB (at the time) that not only did NOT have a backout plan they ...

... denied that there was anything wrong for nearly 2 weeks while the other huge IT component team appeared on site with chief engineers.

[Ocelot II]

6. Well, that was a pretty big oopsie, wasn't it?

[JohnQFunk]

7. 👇🏽

[FakeNoose]

8. Hasn't Mark Zuckerberg made enough money already this year? This decade?

He had to fire all the coders who were the only ones that knew how to fix this. Am I right?

[lagomorph777]

9. Hahahahahahahaaaaaaaaaaaaaa!

I hope this helps wean a few people away from the Fascist teat.

[Nictuku]

13. Wired Magazine article

https://www.wired.com/story/why-facebook-instagram-whatsapp-went-down-outage/?fbclid=IwAR0P3sLjWzIKu4aMshfXltqpv_diGHVez_O4S495YBvuiW5xNVmeil2yaz4