UPDATE: Wednesday, Mar 15, 2023 · 6:43:11 PM +00:00
Sorry folks, commenting isn’t working for non-staff yet. Devs are restricting site writing by IP in order to keep things flowing at a trickle, to make sure the code changes hold up. Assuming the site holds up with staff stories, they can start opening it up further.
Holy crap, this has been brutal, and it’s not over. But progress is ongoing.
The tl;dr version of what happened is that Amazon Web Services, our web hosting service, crashed. This affected wide swaths of the web, like Reddit. But while all those other sites were able to come back online, the Amazon crash exposed bad or corrupted code on our end. The result was this weird situation in which stories created during the times of Amazon’s server issues crashed the site if they were updated. So periodically the site would come back online, only to have someone, somewhere, update a story and bring it all crashing back down again.
Our engineering team has been working for over a day, with barely any sleep, to diagnose and fix the problem. The solution, which requires changing settings on our database and reconfiguring tables (or something like that, I’m not an engineer), has already begun.
The first phase you’ve already seen—we took the site down for 30 minutes and cleaned up a subset of tables. Staff can now post stories, but the community cannot. Our devs want posting slowed to a trickle so they can monitor performance and ensure the fix holds. Commenting should be possible, but don’t be surprised if it’s buggy.
Assuming all goes well, the site will be taken down again tonight, during our night traffic lull, and the rest of the database will be cleaned up. That should take a couple of hours. When we’re back up, everything will—knock on wood—be back to normal.
In some ways, we were lucky. We weren’t hit with big news. Yes, Abbreviated Pundit Roundups and Ukraine Updates never came, and there were plenty of Republican shenanigans left unreported. But no one important died, or gave birth, or did something to change the course of history. Could you imagine this happening on Election Day? Or the day when Donald Trump is indicted? (I’m going to manifest that into reality, I promise you!)
But it was still … lonely. A reminder that even at our most obnoxious, the alternative is much, much worse. Give me pie wars, all the flavors, even the “religion” one. I don’t ever want to live in a world without them. Give me all the Marjorie Taylor Greene updates, even if they’re stupid. Inform me of Trump’s latest idiocy, even if I want to pretend that I’ve moved on. Is Mitch McConnell still in the hospital for a “concussion protocol” that clearly doesn’t exist, because no one stays in a hospital for a week for a regular concussion? Please let me know! Also, how many more woke banks are going to fail for having too many women and brown people in their employ? This is important stuff!
And yeah, Twitter exists, and I can get that information there. But our analysis and contextualization is better, not to mention the community. It’s not even close. So I miss what the crew had to say about all those topics and more. And I missed what you guys had to add to the conversation.
So yeah, it was too quiet, and too lonely, and I didn’t like it. And while I wish we were fully back up, I am glad we are on our way.
Let’s not do this ever again, okay?
Comments are closed on this story.