About Production Mistakes

Or yes, it can happen to everyone

Written in Development by Elena Maroto — October 19, 2020

This is kind of a thinking out loud post.

I just "broke" production. Yes, my friends, I decided to make a dump of the Empresaula DB at peak time (not knowing that it was peak time) and it was too much to manage for our system.

Yes, Empresaula users, it was my fault.

A desing of Puss in Boots (the Shrek character) with sad eyes, with a legend that says forgiveness please

But I want to break a lance in favor of this kind of mistake.

Of course, nobody wants to break production, nor did I intend to do it. But it happened.

Luckily, my senior and lead devs came to the rescue and saved the situation, bringing everything back up again in a few minutes.

This unintended issue has also served to discover that we might have a resources issue in the application, so we might need to think about it.

Turns out an unintended mistake can help us improve our application.

Arnau, my senior dev, insisted on telling me that it wasn't my fault, that it could have happened to him... but I was not even concerned for a second. Even if it was me, these things happen.

I had a boss who used to say that only those who never do, never make mistakes. And it is completely true.

Let me tell you a short story while we are at it.

A colorful desing of a frog with glasses holding a book with a message at the right saying 'it's story time'

It was back in... many years ago, maybe 15 or so? Never mind 👵🏼. I was a developer back then working in SAP projects. I was working in a Finance project. If you've done that, especially in the SAP world, you must know that most of those projects go live the 1st of January (way to spend nice Christmas holidays, you'd think...).

This time was no different, and it was the 24th of December.

Imagine a poor developer with 4 or 5 people around waiting for her to do the changes. That can never end well!!!

A picture of a girl keying in a computer and one woman and two men watching her work

...and it didn't. The moment I hit enter I realized I shouldn't have done that 😱.

I'll spare you the details, but it was the kind of thing experienced developers know, cause at some point they have done the same mistake. I knew it, but with the pressure I had in that moment I forgot about it... and hit enter.

Cutting to the chase, my German DevOps colleagues at the time weren't really keen on me for a while... they had to work on the 25th of December to recover the most recent backup as there was no way to undo the change other than restore.

Yeap. That I did. Oh, and I forgot to mention that it was a system used by at least 10 countries if I recall correctly. So, not like the little mistake that nobody sees, no.

A desing with three wooden direction signs stick to a pole, the upper one points to the left and says 'go big', the sencond one points to the right and says 'or' and the third one points to the left and says 'go home'.

So, yeah, since then, I sweat these kind of issues a little less.

Things happen. Errors happen. We're all human and, if you're a developer, chances are you've done something similar at one point or another in your career.

And you're still alive and reading this, aren't you? And if you haven't done it yet, just give it time... you will.

... just try not to make one of these mistakes that often, will you?

An image with a sentence saying 'till next time...'

View all posts tagged as