20 years of engineering at Redgate
Redgate is celebrating its 20th anniversary this year. How has the way we write code changed over these years? In this post, I’ll try and pick out some of the way’s things have changed from an engineering perspective.
The Early Days
Our first version of SQL Compare was written in C# 1.0. Our version history does go back that far for some of our products, and it’s quite cool digging through the history (software archaeology for the win!) and seeing some of Neil’s (co-founder of the company) code that formed the basis of SQL Compare.
Early versions of our code were written in an object-oriented language, but written in a procedural style, characterized by large methods, many “out” parameters and little decomposition.
Back then we didn’t use unit testing tools (unit test runners of the xUnit framework hadn’t really come to the fore yet). Our release schedule was based on the idea of a big-bang release once a year and we gained confidence in our software by relentless manual testing by a dedicated test engineers.
We were early adopters of continuous integration — any code change caused at least a built to run. We used CruiseControl (a CI system still available, though the .NET version seems to have disappeared).
The first test I could find built a little test runner for SQL Compare. It allowed you to write code that described what SQL Compare should do (in this case, restore a couple of databases and ensure that after applying a comparison they are both the same). This isn’t a unit test (it tests the whole product), but it’s automated and part of the CI pipeline. This probably put Redgate ahead of its time.
RestoreDatabase(
SQL7SERVER, “SQLDataCompare2”,
BACKUP_LOCATION+”\\SQL7\\SQLDataCompare2.bak”, SQL7PASSWORD);RestoreDatabase(
SQL2000SERVER, “SQLDataCompare1”,
BACKUP_LOCATION+”\\SQL2000\\SQLDataCompare1.bak”, SQL2000PASSWORD);SynchronizeDatabase(
SQL2000SERVER, SQL2000PASSWORD,
“SQLDataCompare1”, SQL7SERVER,
SQL7PASSWORD, “SQLDataCompare2”, 1);
Pretty neat! (even if I did find hard-coded passwords to test databases in the source 😊).
For version control, we used Subversion (apparently we might have used Vault or SourceSafe at some point too, but when I ask about that people just refer to “the dark times” and run away screaming). A side effect of this is that commits tended to be fairly large and infrequent. There was also none of that rewriting history controversy which meant that the number of times “hopefully” accompanies commits is high!).
C# starts maturing
C# gained generic support in version 2 (around 2005).
At this point, we only used generics for new code; we didn’t go back and remove usages of non-generic collection classes. Was this a mistake? It’s difficult to judge; Redgate’s products were in a growth phase — if we’d have rolled out usage of generics instead of homebrew strongly-typed collection classes, we’d have gone slower for a while. On the other hand, we could have deleted thousands of lines of code.
At this phase, our homebrew test runner was phased out and we adopted NUnit as our testing framework. Having a third-party testing framework meant that Redgate no longer had to spend time writing both the framework AND the tests and could focus on writing the valuable things (the tests!). If I look through our code at the time, I see a notable increase in the number of automated tests that are being written. This is a great example of affordance — using NUnit created an environment in which writing tests was easy, therefore tests were written!
In late 2007, C# made a major change with the introduction of lambda expressions, expression trees and extension methods. C# moved from being a purely object-oriented language into a hybrid language.
It took some considerable time (until 2012!) for SQL Compare to adopt .NET Framework 3.5 and hence LINQ. This was roughly the time I joined Redgate and I remember there being plenty of feisty debate over what “good” looked like.
I suspect most agree that code of the form:
a.Apples.Where(x => x.Ripeness > 10)Is more readable than it’s for-loop equivalent, but where do you draw the line? Here’s the type signature for Aggregate.
public static TResult Aggregate<TSource,TAccumulate,TResult> (this System.Collections.Generic.IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate,TSource,TAccumulate> func, Func<TAccumulate,TResult> resultSelector);Aggregate’s type signature is complicated! I think there’s a spectrum of “functionalness” from for-loops (i.e. no at all!) all the way through to aggregate (and probably further — think monads, and dependent types). Around this time, I think if you used Aggregate in code it probably wouldn’t pass code review; it was considered a little too complicated.
Smaller, Faster
As Redgate moved forward, so did Microsoft and new versions of SQL Server were released. Our approach to quality was to test everything on every system. The led to a combinatorial explosion — SQL Compare was tested on all versions of SQL Server and different versions of Windows.
By the time it got to 2014, our comprehensive test suite was taking 12+ hours to run. Even when the tests did run they were unreliable due to heavy reliance on the external environment. On any given test run there could be 500 random failures, each of which had to be manually reviewed to see whether it was a genuine regression of simply a transient fault.
We had many options at this point. We could continue as we are and just suck it up. We could incrementally try and make things better over time. We could bin the code and start again. Or we could find an elegant hack. You can read the full story (slaying the beast), but long story short we created a seam for the SQL Server communication, recorded the traffic and the “record/replay” tests were born. Tests that took 12 hours+ now took 10 minutes and the number of transient failures dropped.
This (and the horror stories borne from it!) led to us favouring writing unit tests over integration tests. Topics like the SOLID principles and test-driven development were discussed and shared in lightning talks, workshops and training and gradually used across the teams. Over time this led to a stylistic change in the code we write, with smaller methods, better defined responsibilities and more unit tests.
During this period, we moved to Git (hurrah!) and the pace of releases has increased dramatically (during this time I guess we started releasing at least monthly, with some brave teams releasing each sprint). Due to affordance, we make more frequent commits and most work is carried out on a branch.
Today
As we move into the third decade of Redgate (how many tech companies can say that?!) our style is starting to gradually morph again.
The pace of releases has jumped again. Almost all products have a frequent release schedule (every couple of weeks). In order to keep up with this (or to drive it?), our CI system is rapidly maturing — we no longer rely on physical machines being in a certain state, almost everything is automated via Vagrant. We’ve got our eye on the four key metrics and are committed to further improving this in the future.
Microsoft’s development pace has also rapidly increased, and we’re keeping up! We’ve seen big changes in our world, with the release of SQL Server on Linux and the introduction of new platforms such as Azure Data Studio. This has driven change within Redgate and this year we’ve been on a journey of achieving Microsoft .NET Standard compatibility. This has culminated in our first release on a different operating system, SQL Compare on Linux which runs inside a Docker container.
There’s thriving communities around functional programming (and F#) who are bringing ideas back to C#, such as property-based testing or monads. This is starting to have an effect and we’ve started to see some teams experiment with the Maybe monad (a way of avoiding that billion dollar mistake). Time will tell whether we start using F# in production, or whether C# continues its hybrid makeover.
We’ve also got a larger number of times, and that CI dashboard that we were so proud of isn’t visible by everyone. We’ve started to move over to individual build alarms for each team.
We’re at the beginning of our next change; the Platform. As we start to build more complicated systems, we want a common foundation to build upon. It’s early days, but I’m super excited about the next 20 years and we’ll see more usages of containers, Linux and maybe even a bit of Kubernetes.
Here’s to the next 20 years of learning!
