Red Dwarf

As a lot of y’all probably noticed, AO3 had some extended downtime this past weekend.

They upgraded their database software on Saturday. The first crash happened with Sunday-evening traffic. AD&T (the committee of OTW volunteers who manage the AO3 software) painstakingly nursed the servers back onto their feet. Then they crashed again under Monday-evening traffic. (I’m describing this in US time zones, because it correlated with “the spike when USians get out of school/work and start opening fic en masse.”)

Official AO3 social-media posts haven’t named the software. I assumed because it’s a third-party vendor, and they (very reasonably!) don’t want the place getting deluged with angry emails from AO3’s wankiest users. But at this point, there’s other public confirmation that it’s MariaDB.

AO3’s public Jira board had two new tickets created while they were dealing with the downtime. There’s a helpful breakdown of the tech implications by siropsalot on Bluesky. In short:

“Audits cleanup job” – AO3 has been storing certain logs in a single giant table that updates forever and never gets archived or cleared, which is fine if you’re a small or low-traffic project, but bad if you’re one of the top 100 highest-traffic sites on the internet. (This is part of a long pattern of AO3 being, ah, poorly-designed for the scale of traffic it gets.) This ticket is to create a regular clearing-up process.

“Patch Devise to prevent excessive audits” – One specific user has a buggy older browser, which generated over 2 million entries in the giant table just for them. This ticket is to patch against that specific edge case.

(Denise was pretty alarmed by the first one, because the proposed fix might delete data the OTW is legally required to keep. That’s a tangent, this is mostly a post about the tech problems, I’m just throwing it in because it seems worth knowing.)

MariaDB also has a public Jira board. Which documents this bug in the version of the software that AO3 just upgraded to: ““Local temporary space limit reached” on not so rare occasions.”

AD&T brought the site back up on Tuesday. It’s been safely up ever since. My understanding is, it stayed up after tech support from MariaDB helped them troubleshoot and implement a workaround for that issue.

Disclaimer that I am not a programmer! Someone more technical might come along and correct me on this! AD&T is working on an official postmortem — hopefully after they catch up on some well-deserved sleep — which will be way more illuminating than anything I can figure out in the meantime.

But my impression right now is that “AO3 software has problems with huge poorly-managed piles of data” ran headlong into “MariaDB upgrade has problems with not allocating certain operations enough space,” and it went about as well as a 12′ truck trying to drive under the 11’8″ bridge.

(Except in this case it’s a more normal bridge, where safely-loaded trucks usually pass under it with no problem, while AO3 is…I guess a truck with an extra 5 feet of clearance, caused by a wobbly pile of stuff held on top of it with a precarious set of bungee cords?) (It’s not a perfect analogy, okay. But you get the point.)

Have you heard about the “feed this document to an LLM, it’ll generate a podcast episode discussing it” gimmick?

Somebody fed it a document that’s just the phrase “poopoo peepee” over and over. And then took the discussion of that document, and fed it back into the LLM to discuss that. And then did it again. That link is to a 10-episode playlist. (…They’re only between 6 and 15 minutes long, so it’s not as much content as you’d think.)

It’s as stupid as it sounds — which makes it the perfect demonstration of “how LLMs will take anything, even the dumbest nonsense, and generate a response that has the vibe of something meaningful.” The recursion reveals even more levels of this. I swear every single video has included some version of “it reminds me of the absurdity of Dadaism”…and none of them acknowledge “the video we’re analyzing said it reminded them of Dadaism”…and they’re always repeating the same 2 basic facts about Dadaism.

(Bonus: the software always pronounces it “Day-day-ism” or “Daddy-ism”.) (Bonus 2: At least one of the videos inserts a fake ad break.)

There’s some hallucination, too. The TTS voices start referring to things like “funny sound effects” and “dramatic zooms”, which do not exist at all in the video they’re “analyzing.” One video says “Do we even really know what the original document said anymore?” (Yes. Yes, we do.) There are at least three variations of “it makes me think of apophenia, have you ever heard of that?” / “No, what does it mean?” when they’re supposedly “discussing” a video where apophenia was brought up and defined.

This is almost as good as the Chatbot Chess Championships 2026. (Which had a lot fewer random nonsense moves than the last one…but in one of the videos GothamChess mentions that he’s been “reminding the bots where their pieces are.” Boo.)

–

The rest of this is just a roundup of bots faceplanting in non-chess fields:

“I decided to do an experiment/torture myself with the default image model you get when you open Gemini. The prompt: “create a grid showing the flags of european countries in alphabetical order. there should be labels below them stating their name. the one for Liechtenstein should have a note below it saying “doubly-landlocked”.”“

“What really happened was that someone who is fully equipped to know better was surprised when her AI agent — a class of software that does not work reliably and cannot work reliably — messed up. […] This is not a misfortune befalling some random person — this is the director of AI alignment at Meta.”

“Amazon is absolutely clear who’s to blame for all this — this 13-hour outage caused by their own bot turning something off and on again is officially user error!“

“Press Start Gaming is almost certainly a tool for making money off of ads and sponsored posts, and posts like the Phantasy Star Fukkokuban misinformation exist mostly to give the site more juice of looking like a real website. If someone goes out and buys a copy of Fukkokuban expecting a new and improved Phantasy Star with better graphics and new sidequests, what do they care? The article wasn’t really meant to provide information.”

“It finally became clear to me and the COYOTE team that we’d been bamboozled. Someone had fabricated an identity and put our call for pitches into a large language model like ChatGPT, just to make a fake story that we’d feasibly pay for. When I started asking too many questions, that someone evaporated.“

“Even entire passwords repeat: In the above 50 attempts, there are actually only 30 unique passwords. The most common password was G7$kL9#mQ2&xP4!w, which repeated 18 times, giving this specific password a 36% probability in our test set.”

The Onion: “HmmAI is at the bleeding edge of artificial intelligence, responding to prompts about recipe ideas, ancient history, or even advanced nuclear physics with the word ‘huh’ in just a fraction of a second.”