Netflix Engineering — Pilfering Innovation

Plagiarism and Populism Replaces Invention and Innovation

This week Netflix Engineering posted an article, Performance Under Load, in which they claim the following software innovation — adaptive concurrency:

https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581

Anyone with some prior knowledge of my R&D work in 2012 and 2013 would immediately recognize and equate the above description as a rehash of what was called then by myself as Adaptive Control Valves, which I bundled under the product feature umbrella theme of Adaptive Control in Execution (ACE).

https://web.archive.org/web/20130920082923/http://www.jinspired.com/research/adaptive-control-in-execution

All of the design and development work for the first Adaptive Control Valve metering extensions was done in 2012 with the product released in 2013. It was built on and extended another of my innovations — QoS for Applications.

https://web.archive.org/web/20130215051003/http://www.jinspired.com:80/site/jxinsight-opencore-6-4

There is no realistic way to prevent software engineers, knowingly or otherwise, from copying inventions and approaches of others, but to claim such as innovation is astonishing considering the evidence trail and the history of copying my other Metrics work. This from the Netflix Github:

https://github.com/Netflix/Hystrix/issues/131

Below are the points I called out to Ben, the then lead for Hystrix, in my article. Most of the items relate specifically to a particular adaptive control valve metering extension as Sentris includes many different ones.

Adaptive (Safety) Control Valves versus Circuit Breakers

  1. Dynamic vs Manual Instrumentation: Control valves are dynamically added into code at runtime. It is not necessary to explicitly call into an API and introduce a compile time dependency, though OpenCore does offer a Probes Open API for contextual activity/task execution metering, which is used by the svalveextension to hook in adaptive control into the runtime. No command framework or actor programming model with software circuit breaker capabilities needs to be adopted pervasively (and perversely). Safety valves can be automatically introduced into any method that is metered. Valves can be specified at deployment time via externalized configuration in comparison circuit breaker APIs require hardwiring at development time.
  2. Preventive vs Reactive: Control valves are preemptive and preventive whereas circuit breakers are reactive. Valves attempt to detect problems before a failure is indeed observed. Once triggered, valves mitigate further risk by temporarily suspending processing and then slowly reintroducing workload (valve flow). Circuit breakers react post failure and don’t attempt to manage the software execution other than to abort abruptly.
  3. Protective vs Fail-fast: Control valves are protective, providing a window in which the (predicted) outcome can be altered via self-healing and risk mitigation strategies. Circuit breakers merely fail-fast, replacing one failure type with another, which can increase risk elsewhere in the caller chain and further choke performance especially when the underlying predicate is incorrect, the costs in exception handling and propagation are high, and the resulting system-wide dynamics not fully understood.
  4. Complete vs Partial Visibility: Control valves observe both good (normal) and bad (abnormal) behavior. Control valves can learn to predict transitions from one to another. Valves are able to assess the effectiveness of their predictions and abort intervention early if invalidated. Circuit breakers generally only receive post-failure notifications.
  5. Entry vs Exit Control Points: Control valves can be applied at the entry point into a web application or service ensuring resource consumption only occurs when it is likely the request will succeed. Being at the entry point also means the adaptive control spans multiple backend resources and external services. Circuit breakers are instead applied at the point of interaction with each backend resource or external service, resulting in request processing blindly proceeding only to fail as soon as it hits a triggered circuit breaker resulting in a forceful termination by way of a thrown exception.
  6. Immediate vs Delayed Response: Control valves can be applied to safe guard the application from its own problematic workload patterns that cause excessive resource contention or garbage collection cycles. A key trait of resilient systems is immediate and effective action in the event of anticipated (predicted) failure. Whilst our tvalve and rvalve adaptive control extensions are perfect at finding optimal levels of concurrency under normal operating conditions, their hill climbing (and descent) algorithms means there is a time lag in adjustment as each new level is assessed. The svalveadaptive control can be used to compensate by immediately dropping down through all levels. Circuit breakers are not designed for such usage in not modeling and managing concurrency.

I recently started giving a workshop style meetup talk on my research work in 2012–13 around imbuing software systems with self-regulation capabilities.

https://medium.com/@autoletics/slides-controlling-software-with-thought-9dca3b5bcfad

The software engineering profession seems so completely messed up when you consider that the Netflix post received 650 claps and the following a few.

I was curious to know what was the trigger for this new project after 5 years of pretty much nothing in the way of Hystrix so I took a look at the first commit.

https://github.com/Netflix/concurrency-limits/commit/6780828763f43c347bb0c71f2516ebb8ad4f7101

What a coincidence on the very same day of this initial commit I tweeted a link to my annotated talk as a follow-up to an article posted by Adrian Coyler.

https://twitter.com/autoletics/status/940172314297208832

Note: The reason I quoted 2011 is that adaptive control valve technology first appeared as feature within the QoS for Applications, qos, metering extension.

Innovation and Invention

If you look up the definition of Innovation you will see “new” and “novel” listed in the accompanying descriptive text. For Invention we have “productive” and “imagination”. Invention transforms a novel concept from inception into something far more substantial and concrete such as a well-defined idea or an actual realization in the form of a machine, device, algorithm, or process. Some have argued that Innovation is the exploitation of an Invention. Exploitation seems appropriate here but I would argue that for many of us Innovation signifies an authorship of some newness within a contextual scope. Invention is the big bang of some new idea or field and Innovation is the application of this newness and the incremental refinement of such.

Unfortunately in our age of “don’t have me think…just let me tweet” we get:

Yes, whatever you are doing at this moment is possibly innovative. All you need do is limit the scope of a claim to some microcosm. Give yourself a clap!