Search
Return to News

The Observatory

Scar-imus week three: the phantom database

October 20, 2020

Nights seem to be creeping up earlier and earlier every evening, and before you know it, we’ll be jumping an hour back in time on Nov. 1 and losing another hour to the dark. Don’t fret, we still have approximately two weeks until then, and more importantly, we still have Halloween and more Scar-imus content to look forward to.

Tales of monsters and ghosts and goblins did its job in scaring us as kids, but as we grew up, we realized real life can sometimes be just a scary as supernatural creatures

For the month of October, the Carimus crew wants to share some of our unpredictable, “nightmare” situations that occurred to us during our time in the professional field. We’re calling this a month of “Scar-imus.”

Get your fall scented candles burning and a bowl of candy corn at the ready, because we’ve got a haunting tale in store for you all in this week’s Scar-imus.

The phantom database

We’ve all heard of and seen The Walking Dead. In the first episode, our protagonist, Rick, wakes from a coma all alone in the hospital after being out for who knows how long. He wanders around searching for any sign of life, but there’s nothing.

Everything is gone.

This was the horrifying reality for technical manager, Kristen Katona. Well, not the zombies, but the dreadful “everything is gone and I’m trying to figure out what happened” feeling. 

It was a normal Friday. There was nothing out of the ordinary. Kristen was finishing up her last assignment for the day, which was setting up a test server and running some script for a client who wanted the project done ASAP. After this, she would be free to enjoy her weekend.

However, this project seemed to have other plans in mind for her.

“It was a very lackadaisical script at this job,” Kristen said. “It was supposed to run one database, but they rushed me and they were yelling at me, “Hurry up. Why haven’t you run this script yet?” And so I went and ran it…and I ran it on the wrong database.”

In the blink of an eye, millions of records were gone and dropped out of the database. 

This truly is a nightmare that managed to break through the dream world and make itself become a reality. 

Thankfully, this story does not have a tragic ending. 

Kristen had been working on the staging site, and not the production site. So, pooling her resources and getting the help from a coworker, they called the customer and got the backups of the database back.

The situation was thankfully resolved, and hopefully nothing like this would happen again.

Right?

The ghost of the database’s past

It was the witching hour on the night of a full moon, and there’s no telling what creatures were playing beneath the pale light. It’s believed to be the time where the supernatural appears and are the most powerful. 

Little did Kristen and senior software consultant, Rob Downs, the ghosts of the past would emerge from the shadows and pay them a visit. 

Rob received a phone call from a client in the early hours of the morning and was given the news that a ton of production tables had gone missing.

The two were up until 7 a.m. working with the employee who had called them trying to figure out what went wrong. 

They asked if the employee had run some script as the wrong database user, and they swore they hadn’t.

Turns out, they had.

“We lost seven or eight hours in the middle of the night trying to troubleshoot what the person knew they already did, which was blown away a bunch of production tables,” Rob said.

The lesson learned here is to a) make sure you always have backups of your data and b) if you messed up, just own up to it. It saves everyone a lot of trouble in the end. Also, beware of the witching hour, you never know what trouble ghosts and ghouls may cook up for you.

Written by Taylor Sexton
Interested in reading more from the Carimus team?Here are some related stories

Next.js on AWS

Dynamic Experiences, Statically Generated

Contact Us

Carimus Icon