Pull on your favorite pair of spooky socks and grab a glass of warm mulled cider, because you’re going to want a drink after reading this week’s horrifying Scar-imus stories.
Sure, Freddy Krueger with his gloved razor hands and ability to lurk in his victims dreams can be pretty scary, but we all know that’s fiction. Real life has the ability to be just as terrifying, though not in the same way as movie monsters.
For the month of Scar-imus, we want to share with you some of the Carimus crew’s “nightmare” situations that occurred during our time in the professional field.
The tale of a crumbling warehouse and a drone race
Among a concrete slab sits an abandoned, dilapidated warehouse. When looking at it from the outside, you’ll find yourself buried in the shadows the towering building casts. And if you look at the shattered windows long enough, you may just even see a pale face staring back at you.
When entering the long-forgotten warehouse, you can hear the creaks and groans as the building settles further into the earth. Usually, the only objects occupying the space are fallen bricks from the buildings structure, but on this fateful day, the warehouse will be home to over a hundred people and a drone race.
An event in an abandoned, slightly creepy place; I mean, what could possibly go wrong?
Well, CEO Tony Pease, software architect Dan Shugan and Brandon Phillips are about to experience the same fear teenagers experience in horror movies once they realize they are stranded in the middle of the woods with a psycho killer coming for them.
GoodBookey is an app that puts a spin on placing wagers on sporting events by allowing users to place their bets on a team, but if they lose, their money goes to the charity of the winner’s choice.
Taking place in Detroit, GoodBookey snagged their first, big opportunity with the Drone Racing League (DRL).
The plan was the crowd would be able to do intermittent betting on which drone they believe will win the race
Drone Racing Leagues are often post produced, mainly due to the fact that while racing, many of the drones will get busted up and the courses succumb to the chaos and need to be reset up. No one really wants to sit and wait for 10 to 15 minutes for the course to be set back up.
The moment the three men heard their footsteps echo off the empty walls and saw the bricks piled in the middle of the floor, they knew they were in for an interesting day.
“For the event itself, we had no idea what we were getting into. We had literally built a custom version of the app just in the two weeks leading up to the event so we could change things on the fly. And boy did we ever,” Dan said.
The DRL said there would be wifi while the trio were on their way to the event, however, tragedy struck. Alas, there was no wifi, so the three were left to rely on and make updates through their cellular hotspot.
Oh, but the day is only getting started.
Nestled in a damp, dark corner and surrounded by rubble, Dan and Brandon are hard at work on finalizing updates for the app.
“I remember hiding in the back frantically working on changes all while still wearing a coat because of all the broken windows and it was a cold day in Detroit,” Dan said.
After hyping up the crowd for the event and the GoodBookey app, the final update goes through and the crowd has their phones out, waiting for the app to load.
Then, tragedy strikes.
“I’m not able to get it to show up,” pipes one voice from the crowd.
“I’m not able to get it to show up either,” says another.
Soon, the whole crowd is chiming in and the nightmare is becoming a reality; the app isn’t working. After all of this work, after getting the crowd excited and ready to go, it’s a bust.
Turns out that since there was no wifi, and no one else was on wifi, it takes 24 hours for cell phones to register the DNS cellular update.
“It’s the biggest under the lights failure we’ve ever had as individuals. In my 20 years of software, that was the worst, worst demo experience ever. So it really was like a nightmare,” Tony said.
All issues aside, Dan said this event served as a learning experience in many ways.
“Managing expectations, planning (and testing!) contingencies, working in strange locations, realizing when too much is going on and obviously dealing with setbacks. I mean, we did basically try to build a second-screen, in-person experience on the fly. What’s the expression? Maybe our eyes were a little bigger than our appetite,” he said.
The lesson learned here? There are two reasons to stay out of abandoned warehouses; the first being because it could be haunted and pick off members of your group one by one, and two (the most realistic reason) is because there’s no wifi.
A few years later…
It was an early morning, and the fog was hanging thickly in the air. Creatures of the night returned to their homes, waiting for their time to return to the earth’s surface. Tony, Bryan Martin and Nico Gonzalez sat in theChive’s headquarters, finishing up the last few details for a custom application which would be be used in just mere hours for theChive’s spelling bee charity event.
The day before the event, the team was asked if a small change could be made to a dashboard that was going to be presented on the live feed. This dashboard was the main point of interaction between the host, the contestants and the live audience of the event. It was a core piece to the experience of the event.
The room was buzzing with excitement as everyone prepared for the event. Hopes were high and the day seemed promising. However, the ghosts from the past were determined to haunt the GoodBookey crew and make them relive their worst nightmare.
They’re 45 minutes out from the live event and so far everything is going smoothly. The crew is confident this will launch and be a success.
All it took was one small adjustment to create a major regression.
“We quickly realized one of those changes made to fix the dashboard for the live feed had introduced a regression to the api that broke parts of the app itself and now none of the assets were loading correctly,” Nico said.
It was happening again; another under the lights failure.
All hope was lost, but Nico, only a few months on the job, was rushing to fix it and fix the regressions.
“I said, roll it back, Nico. Let it go. Or we decided to roll it back and let it go,” Tony said.
The team worked to fix the problems this ‘minor’ change had created; trying to patch up the leaderboard issue and rollback the changes on the api to fix the regression of the app.
“In the 5 minutes before the event started, we triggered a rollback, killed all the nodes with the old code and we waited until all traffic was redirected to the nodes with the new (old) stable version,” Nico said.
It was a miracle, but after fixing some issues on the back end of the app, they were able to fix the app and the event ended up raising almost $20,000.
With their demons defeated, the crew celebrated their happy ending by going to the afterparty hosted at theChive and having themselves a well-deserved drink.
Stay tuned during the month of October to hear more Scar-imus stories, and make sure to check out last week’s story here!
Written by Taylor Sexton
