Or How I Learned to Stop Worrying and Love my Errors
This is a story about errors, random incomprehensible and frustrating errors. But it’s also a story about culture and how to use everything you have at hand (even errors) to foster a great working environment.
About 18 years ago (February 2000) I started working at the company that’s now GyF Soluciones. I already wrote about my experience at the best company I ever worked for, well GyF Soluciones was the place where I learned many of those things I later promoted at Inaka myself. My job there was the foundation of my career.
By the time I joined GyF, we mainly focused on the development of financial systems in VB6. These systems were big and complex. VB6 was not a great language to develop big and complex systems. But with passion and creativity, we tamed that beast.
One of the many challenges presented by VB6 was the sheer amount of duplicated code needed to run a system with multiple components. To give you an example, we had one system whose main function was to read and write files in a configurable fashion. Managing that configuration required an important number of tables, each one with its own CRUD. Each one of those CRUDs was a VB6 program, almost identical to all the others but not exactly equal.
It didn’t take much time to realize that writing and maintaining all those programs by hand was not the smartest solution. So, we devised a two-fold solution:
- First, we wrote a DLL called Marco. Marco was a library that allowed us to share common behavior among different programs. That DLL took care of everything except the UI, which was either impossible or extremely painful to include in a DLL.
- Then, we created Arturito (not R2D2, Arturito). Arturito was a code-generation software that wrote CRUD programs based on the structure of SQL Server tables using Marco, of course.
Arturito, Marco and a couple of other tools greatly streamlined the development of our systems, and made them very homogeneous. With them in place, all of our programs looked similar and (more relevant to this post), they all behaved in a similar way.
One of those shared behaviors was the way we used to display errors. Since error management was embedded in Marco, we had a truly generic way to process them. Marco’s error handling method grabbed information from different sources: some data was passed in from the VB program while other information was directly taken from the running execution context.
With all that data, Marco emitted a message box.
Marco used some data as the title, and other information to build the message. And that message had a particular format:
$ERROR_DESC - $ERROR_CODE / $SYS_ERROR_DESC / $EXTRA_INFO
When all that data was present, error messages looked like:
Could not open file - 53 / File not Found / thefile.dat
Really helpful and clear, isn’t it? Not the best error handling in the world, but for a super-generic error displaying component it was great…
…as long as you had all that data. Because of, well… VB6, and DLLs and code-generating tools… You know… Sometimes parts of that data were missing. For instance, it might happen that we were unable to gather system information and therefore the error looked like:
A date in the future is required - 0 / / 2001-12-03
In other occasions, the system was unprepared to handle/understand the error so it bubbled up to Marco and it had to be displayed with whatever information we had at hand, like:
- 312 / Corrupted Memory /
You see where this is going: For some errors Marco had absolutely no information. According to VB6 everything was OK and the actual running program didn’t know what happened or how to explain it, but the error handling code was being executed so there had to be an error. In those situations, Marco displayed the MessageBox with the following message:
That error (actually, that kind of error) followed us everywhere. It appeared in every single system we developed with Arturito and/or Marco. And it always popped up out of the blue, regardless of how many times or how thoroughly we had tested our systems.
Every once in a while we found instances of
-0// that we could actually reproduce. We celebrated. We added tests for them, we reviewed them and we either made sure to emit proper error messages or just remove the bugs.
Sometimes, those errors were generic (i.e. they applied to all systems) and we could catch them and deal with them directly in Marco. Each time, we improved Marco like that, we hoped that was the last instance of
-0// in our lifetime. Just to find a new one months later in the same or a different system.
If you can’t beat them…
After months or years of
-0// hunting, we finally recognized that we were not going to get rid of it anytime soon. And at that time, we officially welcomed him into our family.
We had a constantly-running bug-hunt where we all celebrated the discovery of a new root cause for
-0// errors (particularly the generic ones).
We celebrated the baptism of every new developer who faced their first
We had banners and pictures around the office with it, too.
We explained our clients the whole situation the best we could and we made sure they didn’t panic when they were confronted with those dreadful things.
-0// we learned a lot about debugging, tracing and benchmarking production systems.
Years later, when most of our systems were migrated to .NET, we deeply missed our beloved friend
At some point, I left the company and Facu Villanueva joined the board of directors. A great friend of mine and another proud son of the company, his story within GyF deserves another blog post (or many), but he will have to write those on his own. Anyway, the company expanded and he was commissioned with the decoration of the new office. He added a big sign that read GyF to a wall: the one you can see in the picture above. It was built with a cloud of small words that represented the company. Among them, as you might have guessed, we can find…
It was an error, it was something that could’ve brought shame to us particularly when it popped up in the most thoroughly tested systems. We could’ve hated it. But instead, we recognized that it did represent the company and its culture entirely. It was the symbol of how we approached our errors and how we learned from them with a fun attitude instead of a blaming one. It was undoubtedly part of us!
A while ago I submitted a version of this story to TheDailyWTF, so if you see it popping up there… you know it’s us! :)
OffTopic Shameless Plug
I’ll try to make these brief today, because there are many things coming up:
New York, New York
Next weekend I’ll be in New York City for the first time in my life. I’ll be talking at 2 events:
- BeamCamp @ OpenCamps (November 17–20), talking about Opaque Data Structures like I did at the EFLBAthis year (but in English, of course).
- ErlangNYC @ PypeStream Offices, having fun with Erlang Oddities :)
Don’t be a stranger, come talk to me if you’re around ;)