Selecting the Proper Stack for a Project

Wm. Klos

With every new project comes the requirement of selecting the most appropriate set of technologies to help it succeed. This post will detail some of the decision making that goes into the process of determining the best approach for selecting a client’s technology posture, namely, a programming language.

Requirements of Most New Projects

Most of the projects with which I am involved, are centered around IoT, distributed architectures, services, some sort of web UI and possibly some cool mobile angle. For most of those projects, these are the architectural considerations that any language/stack candidate must fulfill:

  1. Support Web Services, potentially Micro-services Design
  2. Easy Asynchronous Processing
  3. Ability to Horizontally Scale at the Component Level
  4. Distributed Design of Smaller Applications
  5. Ability to Use Containers
  6. Availability to Connect to External Services (databases, queues, external vendors)

Fulfilling these requirements is only the beginning for the stack candidates. There are still a slew of other avenues to consider. For the development team, this is not a set of decisions we take lightly because, in addition to the client having to live with our solution for years, we will be on the front lines of that decision for the next several months of development as well as on the follow-on support tasks.

Client Readiness

One of the most important considerations outside of “Can it technically do the job?” is the client’s ability to take over once we’ve gone. In most cases, the client already has a fully matured IT function (which, to be honest, can actually be a hindrance to the project), and they’ve got approved stacks, trained people, a ton of infrastructure and sometimes even a blacklist of things that are not allowed to be even talked about much less considered for deployment. If that’s the case, then our selection process fits neatly within the box with maybe a few “pokes” to see if they’re open for things like Docker, or git, or the cloud, or even TCP/IP (I had a client once that was a staunch Token Ring/SNA proponent).

I define technology baggage as the set of commandments an IT shop has to follow because they were mandated at some point in time. These commandments take the form of “Thou shalt always do…because we always have” instead of “Thou shalt not…” . If the client has no technology baggage, or does, but the project in question is not to be part of it, — or if it is a startup, then we are off to the races! We can then work with them to see what kind of technology shop they want to build and begin laying the foundations for their future.

If the client has no technology baggage, but a country’s worth of technical debt, like for example business processes that appear to be congressionally gerrymandered in order to fit the constraints of their system (or vice versa), then the old “detonate and innovate“ approach works best as long as we can do the same overhaul to how the business is expected to interact with the system.

Language Viability

With many of the assessment and improvement projects I get involved with, there typically is some sort of vendor selection which invariably includes a vendor viability judgment. Vendor viability speaks to the ability of the vendor to be able to provide services to meet the requirements as well as their ability to service the client by being able to stay in business or not discontinue whatever product or service we are attempting to engage them with. As you can imagine, picking the wrong vendor can be disastrous for a project.

Determining the viability of a programming language is a bit different. There is no profit/loss sheet to look at since languages, for the most part, are free to download and use. You can’t really call for references because every client has an environment designed to suit their own needs. They’re solving different problems with different variables which probably don’t apply to you. In the end, the decision on the most appropriate language can come down to a few basic considerations:

  • Ability to solve the problem at hand
  • Ability to find or build resources to work in the language
  • Availability of training and education resources for that language
  • Support from either the community or another entity dedicated to furthering the languages’s evolution to meet future needs

Now most of these points are self-evident, if a language cannot solve the problem, or nobody knows how to build applications in it, or I can’t Google any examples — I’m most likely not going to select it for my project (nor my client’s).

Support is a little trickier though. Back-in-the-day (Wednesday, 1990), language compilers and interpreters were just another piece of software you bought, Turbo Pascal, QuickC, even BASIC were shrink-wrapped software that cam with manuals, floppy disks and honest-to-goodness telephone support. Today though, languages are offered for free downloads backed by no one other than the individuals who dreamed it up. Sometimes there are companies backing these languages (e.g. Google for Go, Dart; Ericcson for erlang, Oracle/Sun for Java) — but sometimes it is a foundation or loose collaborations of non-profits that oversee the distribution, updates, bug fixes and future of the language. This arrangement has to be factored in to your decision about adopting a language for your stack.

As an example of future support, here is an announcement regarding Python 2.7 from the Python Foundation:

The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020. This decision was made to clarify the status of Python 2.7 and relieve worries for those users who cannot yet migrate to Python 3. See also PEP 466.

This declaration does not guarantee that bugfix releases will be made on a regular basis, but it should enable volunteers who want to contribute bugfixes for Python 2.7 and it should satisfy vendors who still have to support Python 2 for years to come.

There will be no Python 2.8.

Aside from Python 2.7 being a popular language than is generally suited for for a wide variety of problems, I cannot count on it being available long enough to support the lifecycle of most new development efforts. Thus Python 3 is forced to stand in its place, and based on this, it would need to be evaluated sans its history.

But once we’ve determined the general viability of a language, what next?

Language Maturity

For a programming language, maturity is a double-edged sword. If it is too young, businesses don’t take it seriously — in part due to its instability and in part due to its lack of a track record or critical mass. Conversely, if it is considered too old, it may start to atrophy due to lack of new features and developer support.

All languages have a lifecycle, and considerations of that lifecycle are made for any project. We always consider Java and C# for their apparent ubiquity. We consider Node.js and Ruby as well for ease of development and availability of tools and expertise. I, personally, always consider Google’s Go because it has the perfect blend of speed, concurrent capabilities, ease of use, operations versatility, commercial viability and available libraries — if I was starting a company, all of its applications would be in Go. But ultimately, it doesn’t pass the programmer availability test — yet. Some languages, like /perl/ and /awk/, have historic significance and are still in play — but enjoy more of a niche role as a utility play. And some languages that started off in one primary role, have evolved to a new primary role. To me, Python is a good example of that — moving from a language of the web to a language of data analytics.

Almost always, the languages we consider for a project are all on the upper right corner of the chart below, indicating the amount of popularity they enjoy in 2 of the largest online developer communities and is also reflected in aggregated job listings over the last several years. /Now, of course, we also get called for more specialized assignments like mobile where our decisions are constrained by the platforms (Swift or Objective-C for iOS, Java for Android, Xamarin for cross-platform). Those discussions are easier because they are relatively closed-ended./

All of the “orange boxed” languages are the typical candidates for most projects, and they have strengths and weaknesses but for the most part could do any job we anticipate that needs to be done. But different tools are made to solve different problems. Each of the languages we typically consider are classified as “general purpose” meaning that they fit well in a variety of situations — but each has their “sweet spot”. Primary considerations are usually given to the languages below [Language, Version, Primary Backers]. First the two obvious candidates:

* Java 8: 1995 (Sun/Oracle)
* C# 6.0: 2000 (Microsoft)

These platforms continue to be two of the most popular choices for software development in business today. Both products enjoy mature ecospheres, available 3rd party resources, large pool of developers, and a high demand for those developers. They are also considered “enterprise strength” which means that, conceivably, they can solve any development problem an enterprise may have. The issue with being “enterprise” is that it comes with a lot functionality and capability that is not needed for a lot of development projects so it can be cumbersome to iterate quickly and deploy easily.

One reason we usually would not select C# or Java is because of the sheer weight of those technologies both from a “coming up-to-speed” perspective, as well as building, deploying and operating those environments. Because our standard system architecture requires at least a partially distributed system, the ability to deploy, destroy, and deploy again components is a critical process in an active environment. Since there may be functions that are being configured on the fly and running with several instances simultaneously, the need to be able to build, ramp up/down these components becomes a normal part of the DevOps process. The solution needs to have a very small footprint and being able to modify the functionality on the fly and standup new instances on a seconds’ notice is a valuable asset. C# and Java can be unwieldy, just in the setting up of environments, much less their automation.

Add to that the potential licensing considerations of Windows, a distributed environment could add complexity in areas where it is not wanted. In the specific case of C#, being part of a Microsoft platform does not lend itself to being containerized easily, which adversely impacts our goals in the distributed design, deployment, and execution of the project. Now, that said, if the client already has a heavy investment in those areas and is adamant about about keeping that direction intact, we will typically change the architecture to suit their desires and to accentuate the positives of those platforms.

A stack that has worked very well for us on many projects has been one based on Node.js and Javascript:

  • JavaScript ECMAScript 2016: 1995 (Mozilla Foundation, 501c(3); Ecma International Standards Organization, founded 1961)
  • Node.js 7.20: 2009 (Joyent/Samsung) — asynchronous runtime environment, web server for Javascript
  • V8 5.5: 2008 (The Chromium Project, Google) — JIT compiler used for front-end performance on Google Chrome & Opera and for back-end performance on Node.js, Couchbase, MongoDB.

The Node.js stack has been powerful for us in part because it allows the team to avoid “mind shifting” when moving from the front-end to the back-end since we use JavaScript throughout. Additionally, JavaScript developers are hard to find like any developer is, but JavaScript developers easier to make than for many other established languages. Add to that, ease of deployment, availability of reference resources, vendor support, and size of community — JavaScript/Node.js is always a solid contender.

Other languages that we like to consider but are usually dismissed for various reasons:

  • Go 1.7: 2009 (Google)very capable, very well supported, harder to find resources (no critical mass, but rising)
  • Ruby 2.3.3: 1995 (Yukihiro Matsumoto, Heroku) — large developer base, first-class citizen of the Internet, high development productivity, some performance concerns but can be mitigated. Would be recommended after JavaScript/Node.js.
  • Erlang 19.1: 1986 (Ericsson) — one of the fastest solutions, highly fault-tolerant, and industry tested (used for phone switches), more esoteric, quick development once functional programming learning curve overcome. Rising programmer resource availability.

Because of my role in the company, I’m able to do a lot of pilots, proofs-of-concept, and one-off prototypes. Because those usually don’t come with a lot of long-term baggage regarding supportability, availability of resources and other non-technical concerns, I have chosen Go over the last 3 years for those situations because of its flexibility, speed (both execution and development) and ease of use. But I also use it as an opportunity to try new things so this year I’ll be selecting a new personal stack. But even when I’m a development team of one, I have to worry about availability of external libraries to help me complete my goal.

Library Proliferation

Choosing the right libraries necessary to complete a project is an exercise every project needs to make regardless of the language being used. Ideally, the bulk of a project can be completed using native libraries but as soon as a data source is required, or socket communications, or testing — external libraries become a necessary choice.

For any language that has been around for a while like say, C#, Python or Javascript, there are sure to be many choices for each type of need. Fragmentation increases as the number of developers jump on to the bandwagon and have their own twists on how to solve common problems. Here is where experience in dealing with open source type applications come in to play. Identifying which 3rd party libraries are most likely to bring a positive impact to the project is a function of popularity/community, ease of integration/use, availability of examples or track record, support by the primary developer, and “fit” with regards to the philosophy of the project.

One indicator of critical mass of a language is when the large vendors start supporting it either as a main offering or as an “official” access language to their own products. As an example, when developing various cloud platform services, JavaScript running on Node.js is typically one of the first, if not the first, development environment supported. Some examples of this include:

  • Google’s Cloud Functions
  • Amazon Lambda & ElasticBeanstalk
  • Heroku
  • Cloud Foundry
  • IBM BlueMix
  • Facebook’s Messaging Platform (examples are in Node.js)

Given that companies typically treat Node.js as a first-class citizen, vendor support for connection libraries (i.e. databases, queues, mail provider) will either directly provide libraries or provide copious amounts of instruction in how to make their products work with Node.js. This makes it easy for us to recommend for client usage.

But, in actual practice, there are very few critical libraries that are usually required on a project. A connection to the primary and secondary databases, a cache server, a services framework, and a queue engine all have major vendor support for all the languages listed above. If there are critical needs to your application, like say a math library around matrices that your target language does not have, nor does it have a viable 3rd party library for it, then you might have selected the wrong language.

Availability of Developer Resources

I typically don’t put much credence in job surveys because it doesn’t take into consideration the type of work being requested. As an example, R, listed below, is in the 8th spot but is not considered a general purpose language — but rather a language for data processing/analysis. C & C++ are typically required in more scientific, hardware-related, and other areas where speed and complete control over the hardware environment are paramount concerns. Similarly, PHP, Scheme, & Perl each have different strengths that would not be viable for overall application foundations — but may be used for spot needs as a supporting utility language. Thus the popularity of 6 of the top 10 languages (in this survey) are not commonly relevant to our situation most of the time because most business applications do not contain the types of problems for which those languages were designed. So when determining language popularity and developer demand, make sure considerations are being given to those choices that contextually relevant. The point being that some of languages you do not see in the top 10 can be more capable of solving your problem than those that are.

Additionally, these types of surveys only look at the /demand/ side of the equation and do not speak to the actual supply side. The reality is that there is a skills gap across the board. For languages like C# and Java, it’s typically more cost effective to search and buy (or steal) resources, but for many of the other languages (including both Python and Javascript) it can be more productive to build them.

Do you think the general trending downwards is indicative of programming language proliferation or specialization?

Nevertheless, these are the top 10 “in-demand” technologies according to site point based on the frequency of technologies appearing in job postings. see: Best Programming Languages For Job Demand and Salaries, 2015

  1. Java — featured in 18% of advertisements
  2. JavaScript — 17%
  3. C# — 16%
  4. C — 9%
  5. C++ — 9%
  6. PHP — 7%
  7. Python — 5.5%
  8. R — 3%
  9. Scheme — 3%
  10. Perl — 3%

Additionally, for more detailed information about various different metrics and how they are determined, this link has several different metrics of language popularity: What’s the Best Programming Language to Learn in 2015? or also here.

But, the most appropriate pull quote from this article, and the sub-text of many others is this:

Those who pick a language based on survey data or monetary prospects will fail.

This is true when selecting a stack for a project based solely on graphs and charts as well as when looking for a job.

Summary

We select our stacks based on experience with the these technologies solving similar types of problems to those being presented by the project at hand. Because of the way we are usually designing systems, “swapping” out smaller applications will negate the need for performing any type of wholesale overhaul in order to upgrade specific functionality. No monoliths. The new application is meant to be scaleable and agile in both development and operation so as to avoid the need to “have to live” with decisions we make now over the next decade because it is too painful to maintain or replace.

In the end, it comes down to the problems we are trying to solve. Because I’m a polyglot proponent, I come down on the side of the “right tool for the right job”. Since most of my recent projects involve many small sets of standalone services, the stack selection exercise can be performed over-and-over as new modules are brought on line. A stack suitable for one module, might be completely ill-suited for the next.

Wm. Klos

Written by

Wm. Klos

A cloud architect, attracted strongly to the bright & shiny, thus my addictions, while intense, are short-lived. You can find me on twitter @williamklos.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade