How I Invented Databases
Learning how to code as a child with no internet connection; you are forced to invent everything from scratch. I have many stories about this but the one that I keep coming back to is how I invented databases.
In 1999, I was around 12 years old and had been programming in some capacity for half of my life. Mostly games and small hacks. Now, I was ready to build something useful. Something that could solve a real problem and would hopefully make money. At the time I was obsessed with LAN gaming centers and saw that they were in desperate need for software to manage their stores. They kept track of user accounts using pen and paper. When someone paid to play for a certain amount of time the store manager would write that down and then keep looking at their watch and their notebook to see if someone ran out of time. After the time is up, they would physically go tap them on the shoulder and ask them to logout. I thought this was the perfect problem waiting to be solved. Everyone already used computers to play the games then why not use the same computer to manage the entire thing?
I had a big piece of the puzzle figured out. It had to be a client/server architecture and I knew how to do networked applications using WinSock (I used it in my various escapades into the world of hacking remote computers which I’ll talk about in later stories). The big thing that was missing is how the hell do you save the data on disk and fetch it the next time the program starts?
At the time I used Visual Basic 4 and as you can tell from the name it was a GUI-centric development tool. It was either not supported or incredibly hard to write code that wasn’t tied to a UI element (they made it easier in later versions). All the applications that I built in the past kept all their state and data in the UI and I never needed to persist data between program runs.
So the problem was two parts:
- How do you keep data in your program that isn’t currently used on the screen? (Also known as data in memory)
- How do you save and fetch the data between program runs?
For the first problem — I decided that a good enough solution was to mark the UI elements that kept the data as hidden. So in effect if I had hundreds or even thousands of user accounts they needed to live in textboxes on the screen but marked as hidden (sounds terrible, and it is but in my defense this is not very different from how the web development community — until very recently — stored all their data in the DOM).
Every row was an array of textboxes with the following naming scheme: `accounts_{row number}_{field name}`. So to get the username for the first row: all I had to do was something like`Form1.accounts_1_username.text`. I also had all sorts of math tricks to get information about my data quickly. E.g. If I wanted to get the row count I would count the text boxes and then divide by the number of fields.
I also made up the concept of indices to make certain lookups faster. For example looking up a user record by username is one of the most common operation so it needed to be fast. So I had a giant textbox that was basically a comma separated `username: row_number` string. So I would use `InStr` to get the index of the username and then get the row number to lookup the entire record.
Now came the time to save it and loaded up from disk. This, I had to cheat for. Since I had the vocabulary to search for this I went to the internet cafe and looked up how do I save the contents of a textbox to a text file and how I can loaded back in again. Turns out it’s a pretty common use case and there is tons of code examples about this. So I named the text files exactly after the text boxes and the rest was easy. The challenge came when I was testing and figured out that when I get to hundreds of rows the program starts taking up minutes to boot. So I also invented lazy loading. I would load a big chunk of the data upfront and then use that until I hit a case where I couldn’t find a record. Then I would load another chunk and so on until I get all my data in memory (and by memory I mean ‘on screen’).
This hack unblocked me and allowed me to iterate on my product. I built the client side which talked to the server to persist data from counting down the time in — ahem — textboxes. And also figured out a security model around getting windows to boot up to my client application and nothing else. But luckily before I went out to start selling my program I came up on a discovery that would change my life: Microsoft Access. This delayed shipping my product for a year but in the end I did ship in 2002 and made a lot of money for a 15 year old.
