Developing Systems for a Benchtop Manufacturing Company
For the past 4 years I have been working for a company in New Zealand called the O’Brien Group. They specialise in making benchtops (such as kitchen benchtops) and are very good at it. When I first joined I never expected this medium sized family owned business to be so remarkable, but over the last few years we have achieved so much together and come so far. What has stood out as perhaps the most shocking thing of all is that we have continued to work so well together for so long without me having any desire or need to move to a different company, a first in my career.
Recruitment
I was initially approached whilst working for a different business by a recruiter. I was interviewed by both the owners and the outgoing IT Manager. The two interviews were about an hour long and mostly non technical (as they should be for a management role). The only things that stood out was that they mentioned that they had a Microsoft Access database and had been working on a website where customers could order benchtops online and would like me to continue that. I said that I would need to hire at least one employee to work with me to fill in the gaps in my skills if we were to achieve what we wanted to. They agreed that would be fine and after a few days the recruiter told me that the owners had liked me and would make an offer.
I waited for nearly a month for an offer and was starting to worry that one wasn’t coming but eventually they sent me a contract and I started 4 weeks after that. The reason the contract took a long time to come is because, as is often the case for manufacturing businesses, everyone gets so caught up doing their day to day tasks that things like paperwork often come second.
A Rough Introduction
The first week I started was the last week for the outgoing IT Manager. He was supposed to fill me in on the projects and give access to the systems. This was where the problems started.
The IT Manager was the kind of person who loved to talk but not answer questions. He was set on telling me how I should do things once he left, without actually telling me what had been done or explaining anything. Every time I tried to ask him a question or ask for clarification, he would stop me and say that he would explain that later but just keep on going and never answer questions. He would ramble on and on and was mostly incoherent, and it was very apparent that he had very little IT knowledge outside of programming and even that was hard to follow. It was near impossible to figure out what he was talking about, something which wasn’t helped by the fact that they seemed to have made up their own terminology. To give an example, the previously mentioned Access Database was actually a program written in Access that connected to a MySQL database (even after years of correcting everyone it still comes up sometimes).
The Sysadmin from hell
The other surprise was that, it turns out, there were two IT staff in the company. The other staff member, which they had conveniently forgotten to mention, was a grumpy old man who handled all of the Sysadmin work. I quickly found out that he had created a culture of fear where no one dared to ask him for anything as he would be angry and rude. He’d berate anyone who dared ask for anything and would take weeks to only half solve anything that was asked of him, no matter how simple.
Throughout my “training” with the IT Manager, any time I would ask for access to anything I would always be told that the Sysadmin would give me access. But every time I would ask for access to a system the Sysadmin would say that he had already given it to me, so I would push back until he would actually create the credentials. Then I would go ahead and try them and they almost never worked (he loved inventing new passwords for me and not telling me what they were), most often because they were not set up correctly or did not have the proper permissions. He also would not give me any information on how to connect to the systems so I spent a lot of time reverse engineering them and trying to break in to get the access I needed.
Legacy systems
I used a combination of techniques to figure out what systems and computers were running. Some port scanning, a lot of looking at the configuration of the systems I did have access to, and even sometimes I resorted to finding vulnerabilities in the encryption of some legacy algorithms to find passwords. Over time I built up a large database of every password I had found and which systems they accessed. The whole discovery process took about 3 months, with many disasters along the way.
The whole place was running on Windows XP, Windows 98, Windows Server 2000 and even one computer was running IBM OS/2 (connected to the network through a Windows XP middleware desktop). There were also many linux servers running little known proprietary distros, and he had set-up some linux desktops for a few users. For these I only learnt the admin password about 3 years later.
For the networking, the switches in the server room looked like they were projectile vomiting a giant stream of spaghetti. I wish I had taken a before and after photo, but once I had finished removing the unused cables that either did absolutely nothing or were looping to the same switches many times over I had created a small mountain of ethernet cables in the server room.
Trying to get me fired
After about two weeks the Sysadmin called a meeting with the company owners about me. The company owners were fed up with him and wanted to be open about what was happening so they invited me to the meeting. The Sysadmin spent an hour explaining how I was horrible and incompetent and how they needed to fire me. He said I didn’t have the credentials to fix the database (Access) problems as anyone would have to have had 20 years of experience with the database (Access) to be able to handle it. After an hour I had had enough so I excused myself, leaving the owners to figure out how to get him to stop complaining.
For the next few months he made it his mission to get rid of me. He continued to not give me access to systems or to show me how to access them, then used that as evidence to demonstrate my incompetence to the owners. On multiple occasions he broke key systems and when people tried to contact him he was unresponsive, then pinned the blame on me for having broken them because I “changed things”. Each time he would do this I would take the opportunity to get a bit more access to another system as I was fixing them. Throughout this period I made sure to do my best to keep the peace and not lock him out of the systems or aggravate him.
On one of the many occasions when he called yet another meeting to try and get me fired he claimed that he had given me access to a linux system (he had, but my user did not have admin rights so I couldn’t actually do much). He also said he didn’t know what the root password was to this system. However I had been squirrelling away evidence to cover my back in case things went poorly for me and one of the things I had managed to get access to was his personal credentials for the system (but not the root password, sudo was not installed). Using his credentials I had copied out his users’ command history on 2 separate dates and was able to show what commands he had written in the time since he had first claimed he didn’t know the admin password. The command history clearly showed him navigating through the file system and typing the “su root” command (switch user to root) and then not having done any commands until he exited. And I could also see that he had made a root only change to the system during that time. I explained what this meant to the owners, explaining each command to them and the implications. They already knew who they were dealing with so were unfazed, however the Sysadmin lost his cool and started shouting about how we were accusing him of being a liar. He shouted at everyone for 5 minutes, said he didn’t have to take this and stormed off back to his house slamming the meeting room door on the way out.
Grasping this opportunity, over the next few months I would regularly call him to come and meet with me over the petty problems he was creating, ask him to solve the problems and when he said he couldn’t I would show him evidence demonstrating that he could and just didn’t want to. I never called him a liar or accused him of anything, but continuously showed him evidence that he was lying and asked him to explain to me what that meant which would cause him to lose his cool and storm off.
After about 3 months of no progress on him giving me credentials to the last systems I hadn’t been able to break or hack my way into, I called a sit down with the owners and him to make him give me access while I tested each set of credentials in front of the owners. He sat there saying I already had access so I made him try and login with the credentials he had given me and of course he was not able to. After doing this a few times he started shouting and insulting me again and looked like he was going to punch me. He stormed off and I told the owners that I had been patient enough and he clearly was never going to cooperate. I said it was time for us to lock him out of the systems and deny him access to our premises. They agreed and I never had to deal with him again. The systems that I didn’t get access to I ended up replacing, reinstalling from scratch or just ditching because they were old and irrelevant.
The server room
The server room is located on the upper floor of the main factory. When I first stepped foot in it, it was a gross hoarder hell hole. The windows had been covered up with old yellowing polystyrene, there was junk and old computer parts piled high everywhere and everything was covered in about 20 years of dust. And 20 years of dust in a factory that saws wood all day is a LOT of dust.
Once I had gotten rid of the old Sysadmin I was able to get to work on making this room a place I was comfortable setting foot in. I found many boxes and started organising everything that was still useful and throwing away the rest. I lost count of the number of boxes of rubbish I ended up throwing away but it would have been multiple tons worth. I bought big rolls of velcro cable ties to wrap and store all the cables that could still be used. I got one of the industrial vacuum cleaners we use in the factory for sucking up saw dust and proceeded to meticulously vacuum every surface of the server room. There were leather chairs that were completely brown that revealed themselves to have secretly been black or burgundy underneath.
Networking
Once I could step foot in the room without having to take a shower afterwards, I really started to look at the networking in detail. Since I had started I had noticed that the network speed would fluctuate between 10–40MB/s depending on the time of day, keeping in mind that most of the networking gear was fairly new Gigabit speed stuff. The switches were mostly Ubiquiti, so fairly standard, however the Router was some weird linux device running an OS called ClearOS (to give you an idea of how bad it is, their own website barely loads and their fonts are hard to read). This is a truly awful product, it is supposed to make linux networking easy but it is so limited that it can barely even do the most basic of tasks. Their software is not quite as capable as the free router your cheap local ISP would give you in 2001. Also it was on dedicated hardware that was horribly slow. It was being used as our router, active directory (but just for the mail users), DNS server and DHCP server. A local DNS lookup would take about 125ms.
Unfortunately the network setup was quite complicated and there was some kind of LAN that had been setup with the ISP so that several of our factories around the country were all on different subnets of our network so they could all RDP into our Windows servers. The performance was obviously horrible, it would break all the time and the drives would run out of space constantly freezing up the whole thing.
My first goal was to completely redo the networking. I figured that without good networking nothing else would work properly. The plan was that over the Christmas period while the factory was shut down, instead of spending Christmas with my family I would work all the way through redoing the networking. I did as much preparation work as I could, coming in at night and spending many long evenings there rewiring everything.
I decided to get a MikroTik router to replace ClearOS considering that there was quite a complicated network setup and I was planning on hosting multiple servers on a pool of IP’s. I didn’t want any technical limitations imposed on me and wanted an extremely fast router so I never had to worry about speed again.
I prepared by pre-configuring the router and planning out all the settings and ports that needed configuring. I double checked with the ISP to make sure that I had the right configuration to talk to all of their end points and was ready to migrate.
As soon as the holidays started I was there everyday. I had hoped it would only take a couple of days at most even with unforeseen problems. However I could not get some parts of our router to communicate with the ISP’s router. I searched and searched for answers trying every possible iteration of settings I could think of, I tried talking to the ISP but they were mostly on holiday and refused to help, just saying that I was incompetent and didn’t know what I was doing. After about three weeks of effort I got rather angry at their on-call support guy (a Russian guy called Alex) who would always blow me off. He finally actually looked and realised he had given me the incorrect IPs for some of the ports on their devices, so once I had put in the correct IP’s for their ports everything started to work. However it had taken about 3 weeks of me being very stressed and working everyday trying to come up with some solution. But at least we now had stable gigabit speed internet all across the network and DNS lookups would resolve in under 20 milliseconds.
The new and unused servers
The previous IT Manager and Sysadmin had decided they wanted to upgrade Active Directory and Remote Desktop Services to a recent version. They had spent a fortune buying servers from an MSP who had done some of the setup and installed the servers into the rack. However they had no idea how to actually use it or set anything up.
This left me with a bit of a conundrum. As far as I could tell Microsoft had completely stopped caring about Windows Server in about 2008 so other than for Group Policies and Active Directory it was probably not a great idea to seriously commit to using it. On the other hand, many tens of thousands of dollars had already been spent, including buying dozens of thin clients that would go to waste if I didn’t use them.
So I opted to bite the bullet and get everything working. I had actually started on this from the moment I had arrived and it took about 3 months to finish. I hate working on Windows Server, it has a way of making you feel like an imposter in IT. Things that should work just don’t and give you no reason or errors for it, it really is an unmaintained mess. But eventually I did get everything going and started migrating some users to it, reducing the load on the legacy servers.
However the legacy servers were still needed because the Access program was still running on the 1999 version of Access with many dependencies from that era. Fixing and updating this software would be a major undertaking.
Backups
The old sysadmin swore that he had backups but all I ever found was one shoddy old desktop on site that had been setup to run backups. Once I eventually figured out the password I found that most of the backup tasks had not been running for years if they had ever worked at all. There had been many attempts to backup various pieces of data but in most cases it looked like they had been set up, set to run and never once checked to see if they actually worked. I could see years of logs of them having failed every time the tasks were run.
One of my first things to do was to start backing everything up. I set up both a server on site in a safe location and also got a cloud provider that I could dump data onto. We now have daily backups in three separate locations for all our data.
After 6 months of being there I needed to travel to one of the branches to fix some networking issues and as is often the case when I travel, one of the critical pieces of infrastructure failed while I was away. In this case it was the mail server, caused by a series of consecutive power outages a few minutes apart and an incorrectly setup UPS. I was kicking myself for not having taken the time to check all the power wiring sooner.
After some investigation I realised that I was not going to be able to fix it quickly as the data on the drive had been badly corrupted, so I spent my trip in a hotel room on my laptop setting up a cloud email service for the whole company and getting everyone migrated.
Once I got back to the office my newly employed colleague and I spent a few weeks trying to fix the server and eventually we managed to reverse engineer the proprietary system for storing emails. We wrote a program to upload all the lost email data to our new email provider using their API. I was going to change to a cloud email provider (Gmail) anyway but I guess instead of a calm, planned, well thought out migration I got the high stress version instead.
Microsoft Access
I was vaguely familiar with Access before I started working at the O’Brien Group but had never used it extensively. However, I knew that I was not a good enough programmer to take on this problem by myself. I needed help and had been continuously asking if I could hire that person the owners had agreed to in the interview. They had been dragging their feet, always saying not yet, which was starting to seriously hold up my ability to progress with projects. However they eventually gave me the go ahead and so I put out an ad for a “Site Reliability Engineer”.
I guess I had aimed too high with the ad as no serious candidates applied so I took a different approach. I put out an ad for a support person and mentioned in the description that some programming would be required but no prior knowledge of the languages was required. I didn’t mention the languages we would be using at all. Eventually someone was recommended to me, who has turned out to be a fantastic partner for all the projects we have taken on over the years.
Before we started the Access project, the big question we had to answer was would it make more sense to rewrite the whole program in a new language or should we just try and fix all the issues and bugs until it ran on a modern version of Windows. Eventually, due to the enormous size of the program we decided to take the “fix her up” route.
In retrospect this turned out to be a mistake. The program was so massive and poorly written it took about a year to get through it all and we ended up making huge changes to every form, report and query in the program. All the while grappling with the many terrible bugs and design choices that are the Access platform. It would have been faster and better to just re-write the whole thing in a new language, but at least now we know for next time.
When we had finished this is how it looked:
If anyone is asking, Access is a horrible language with no redeeming qualities. Avoid it like the plague.
Developing new systems
Now that we had gotten the legacy systems up to date and the environment was stable and reliable, it was time to get to work on making the factory high tech and adding new features. The first projects we did in no particular order were:
Factory tracking
Adding tracking to every machine in the factory, so that as benchtop panels moved through the factory and had their barcodes’ scanned, the time and location were logged to the database.
Security cameras
We set up 4k security cameras all around the factories so that when there was an issue with a panel or a job we could look at the times they were tracked and follow them through the factory to see where the issue occurred so it could be fixed. We have over 30 cameras storing nearly 100 Terabytes of footage at any given time.
Information monitors
At certain machines throughout the factory we set up screens that would show information relevant to that machine about what was coming down the line towards them so they could prepare.
Power BI
To replace the reports that Access used to make, we set up Power BI which provides us with details about all the stats of the company’s production which we use to make business decisions about all aspects of the company. Designing these reports has been an ongoing process that we keep on adding to and tweaking as we get more ideas about what metrics are needed.
Big new projects
Dispatch
The first really big new project we worked on was our dispatch system. We actually did this at the same time as we redeveloped the Access program as they needed to work in tandem.
We wanted to build a truly novel and innovative system for handling all our outgoing jobs as they leave the factory. We chose to develop this system in Svelte and loved it, so have continued to use Svelte for all our web projects since then. Also we use sockets to send messages back and forth with the server which allows us to have all the displays sync their data in real time, so a change on one instantly displays on every other screen as well.
The program’s job is to know where the panels are stored or sent once they have finished being made in the factory. The display below is a touch screen with a barcode scanner attached. The dispatch team simply specifies what type of pallet or type of delivery is going to be processed and then scan the barcode of the panel as they are loading it onto the pallet, truck or storage rack.
The team can see all the panels they have scanned onto the pallets and can press the “Checklist” button to view a list of the panels from the same jobs that are missing. They can also print out packing slips to attach to the pallets. There are also handy barcode's they can scan that are quick shortcuts to change modes or pallets without having to walk back to the screen.
Online quoting and website landing pages
The reason I was so willing to do all of the tedious, painful work of dealing with 20 year old unmaintained systems was for the promise of being able to design and develop a new website and online quoting system for the company. They had sadly been taken advantage of by the other IT people before me who had promised many things but had ultimately just taken an easy pay check and not delivered, so I was determined to knock this one out of the park.
Fortunately, improving on what had come before me was going to be a low bar to pass. These were the two previous website designs:
The second website was certainly a big improvement in terms of design but still lacked class. Also it was made in PHP with all of the content being pulled from a database that lived on the same server as the website. However the entire website was static, so could very well have been written in HTML and CSS with no Javascript and would not have lost any functionality. The way it was written was ridiculously over complicated which meant no one had had the knowledge to make any changes to it since it was first put into production.
Landing pages
Because of how monumental the task before us was, I decided I might be able to save some time by outsourcing the design and building of the landing pages to a local web design company. Ultimately, this proved to not be the case as they asked so many questions and required me to write all the content so it would have been faster if I had just done it myself. Especially once you also factor in that we had to convert their code to Svelte and they had styled it with bootstrap and their own custom CSS dump file that they seem to reuse for all their projects (11,000 lines of unused code) which took a lot of time and effort to clean up. In saying that, I really like their design and there is no guarantee that I would have come up with something that looked as nice.
Factory tour video
Something I was dead set on was a cool video that would be the banner that you first saw when you loaded the home page. They put me in touch with a local videographer called Clive Copeman who really blew me away with how he took charge and got the whole thing done. I just needed to show him around the factory, explain how things worked and he started building a storyboard and ideas which all I had to do was sign off on and make a few suggestions to.
I organised some days for filming (where it was understood that we would heavily impact the flow of production in the factory) along with a benchtop that we were to follow through the factory on its construction journey. Filming took three days and was loads of fun.
Once all the shots had been put together this was the end result:
Quoting website
To date this is still the biggest single new project I have ever undertaken. I started working on this site shortly after I first started this job, doing planning for the user interface and user experience. Designing the look, feel and functionality of each page in Slides. Two years after I had started, we finally got to work on building it.
I started by building the interface, doing the design work, CSS, responsiveness, networking and SEO while my colleague Cameron was still finishing up the programming work on the Access project. Once I had a pretty interface that did nothing, Cameron swooped in and started making everything work. I helped as much as I could but he is a much better backend programmer than me. He is also fantastic at finding things that will improve the user experience for the user and implementing them, things that you only think of when you are working on the gritty details of how each element should behave.
Ultimately I think that it is both of us having the freedom to implement our own ideas on how to improve every little detail of the user experience which has managed to turn the incredibly complicated process of designing a benchtop into such a fluid experience for our users. It has been so successful that we have had technologically averse, retired people who have never worked in joinery before being able to design their dream benchtop perfectly, every time and with no guidance.
It took a whole year of work to get this ready for release. Initially the scope of the project was just so that the simplest of rectangle designs could be quoted online to what has now become a fully fledged system that now covers 95% of use cases.
Timeclock
Even before I started at the O’Brien Group I had realised that many businesses needed to clock their employees in and out and that every solution out there was seriously lacking in some way or another. A timeclock is such a basic staple of so many businesses and despite there being at least 50 commercial products out there none of them are any good.
As a side project I had started building the timeclock that I had long envisioned with my business partner Cameron Hill. Ironically a few months after we had begun work on this, one of the owners of the company came to me and asked if we could make something like that for the O’Brien Group. At first I was hesitant to mix my side project and my main job but decided that openness was the best policy (after all they had always been open and honest to me) and explained the situation. I explained that I couldn’t realistically design the same thing in two different ways and keep the projects separate, leaving us with only two choices. Either I didn’t take on this particular project or they agreed to use my side project in the company.
They agreed to use my side project within the business and even to let us spend some office hours developing it so it would become production ready sooner. Luckily we were at least 80% of the way through development at this point so it allowed us to clear the last hurdles quickly and get it rolled out.
Nowadays we use it all throughout the O’Brien Group in many of the factories and it is great seeing it being used by the employees everyday. They also are in the privileged position of being able to request features that we prioritise before anyone else's.
Conclusion
Even though my time with the O’Brien Group is far from over, I felt it was the right moment for me to write about my experience here. It has been a job like no other. In every other role I have ever held, after two years I have always felt the need to move on, either because the work was being hamstrung by bad management or because the people I was working with were unpleasant.
However, the O’Brien’s have it all figured out. They know how to ask for what they need then give me the freedom to just get on with the job. Sometimes it can feel like a lot of pressure, with just two IT people in such a large company (we have 9 factories!) and so much responsibility on our shoulders, but somehow we have managed to keep on delivering project after project and the people here appreciate what we do and treat us kindly.
We still have so many things we want to make, and certainly will not be running out of things to work on in the near future.
