As the news filters in from Charlottesville, VA, what emerges is a day that will likely be immortalized in history books for the next several decades, and not in a good way. Throughout the winter and spring of 2017, protest marches have become a regular occurrence, as Donald Trump’s draconian agenda has brought opposition from women, minorities, scientists, environmentalists, the LGBT community, native Americans, the elderly … it’s really hard to find a group that has not protested his administration.

In every single one of these, the condemnation of these marches by Trump was guaranteed, a tweet at 3 am claiming these people who were exercising their constitutional rights were traitors or worse. In only one incident, a protest at Berkeley, was there any property damage — a cart was set afire — or supposed violence — an alt-right spokesman was punched in what looked suspiciously like a staged incident. The right wing media ate it up, claiming that the leftist fringe of the US was becoming violent, proving what they had been screaming without evidence for years.

Yet one thing has been true since before Trump became president.

No one has died. Until now.

The call to arms went out to alt-right groups, to the Ku Klux Klan, to white supremacists, to everyone who felt that all those “snowflakes” were the real enemy and should have been stopped. The gathering at Charlotte was meant to be a show of support for James Damore, a Google engineer who wrote a loaded and controversial memo that asserted that Google should dismantle its diversity hiring program, making veiled references that women in particular were not up to the task of being programmers. He was fired several days later, his firing instantly becoming a cause célèbre among the far right.

When Google CEO Sundar Pichai announced a company-wide meeting, the response from this group was to send death threats to everyone at Google, forcing Pichai to cancel the planned teleconference for fear that someone would carry these threats out and Google employees would be hurt or killed. This has been a common tactic of late among the alt-right — a recent set of talks at Evergreen State College in Olympia, Washington was canceled due to similar bombardments of threats, and the list of people who have cancelled public appearances because they talked about these controversial people is now numbers in the dozens.

So the alt-right gathered, and the counter protesters gathered. Then one person, James Alex Fields Jr., decided to use that opportunity to show what he really thought of those “snowflakes”, taking a tactic out of terrorist attacks in Europe. He rammed his car into the counter protesters at high speed. Witnesses reported seeing people flying from the impact. More than a dozen people ended up being sent to the hospital even as police captured Fields.

One of those people died, en route.

Fields came from Ohio specifically for the event. He was young, only twenty, but old enough to know right from wrong. What he did was unequivocally an act of terrorism against Americans. It was not a foolish lark. He did it for the cause, and may have been encouraged by others around him. Two police officers, Lieutenant H. Jay Cullen, 48 and Trooper-Pilot Berke M.M. Bates, died when the helicopter they were in crashed while trying to control the mobs.

Donald Trump, upon hearing of the incident, praised the alt-right group, claiming they were true patriots, while not even mentioning the fact that this same group that he has been pushing and promoting at rallies and elsewhere was responsible for the death of a person. He later came out condemning the hate on all sides (ambiguously), and extended his condolences to the family of the victims, but again refused to single out any one group, even after being pressed multiple times by reporters.

He wants this. Trump has made no secret that he wants to encourage this violence, wants people to be afraid of his home grown army of nazis, klansmen and thugs. Perhaps he sees it as a way of silencing his opponents, perhaps a way to distract from the mounting investigations into his actions before and during the election. Perhaps he truly is just as much of a bigot as the people he supports.

This is not a question of Left vs. Right any more. It has become a question of a terrorist group fomenting a campaign of violence against all Americans.

The right to protest is enshrined in the first amendment. The right to murder is not.

Kurt Cagle is the editor of The Cagle Report.

This was a Lynch Mob, Not a Protest was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you are a web developer, chances are pretty good that you’ve become adept with React, Facebook’s view layer technology that has turned JavaScript and Node into one of the hottest combinations to tempt a VC’s pocketbook. For those of you not familiar with React, you can think of it as a way to build components that react to changes in data state in web applications, part of what is commonly called a model-view-controller design pattern. The model is the data and its associated structures, the view is how that data gets represented, and the controller is the layer that mediates between the two.

I’m going to go on record taking what I suspect will be an unpopular stance. I don’t like React. I’ve been working with it for a few years now, and yet I find that I haven’t hit that Aha! moment when suddenly the language crystallizes for me, where I realize that there is an elegant simplicity that I simply had to train my brain right to find. For a while I saw React as a necessary evil, because it did solve certain problems that were very much issues even as much as a couple of years ago, but I think that React (and quite a few other node-based frameworks) will in the very near future start fading away. Ultimately, the vehicle for that will be a little sleeper technology that’s just now beginning to make its debut: Web Components.

The Rise of W3C Web Components

The idea behind web components has been floating around in one form or another since the late 1990s, and is really quite simple. A web component is simply an html-like tag in a web page (or similar markup) that displays content in certain ways and performs certain actions, but is not defined as part of the standard markup language. What makes such components so exciting is that the “framework” necessary to make this happen should become native to most browsers within the next couple of years, which means that rather than building complex server side compile applications, such web-component based forms have tiny footprints, stunning performance and ultimately cross-browser compatibility — all of which were reasons that so many people moved to React and similar tools in the first place.

To give an idea about why this is such a big deal, consider an application I wrote recently. The need behind it was simple — I wanted to create a text box that would let me enter tags using the hash symbol (a.k.a. hash tags) that would then become live buttons for bringing up an external web service, here a dictionary entry. This ultimately ended up requiring three components — one that let people enter the text and generate the tags as output, one that displayed a dictionary entry, and a third that retrieved a list of alternatives if no definition was found for that term. The back end database that powered this was provided by Merriam Webster.

This example can also be seen working (for Chrome users only) at https://codepen.io/kurt_cagle/pen/KqLagb.

The beauty of web components can be seen in the markup that describes the above application:

<link rel="import" name="kurt-app" href="kurt-app.html"></link>

<kurt-app>
  <kurt-headline level="1">Tagger</kurt-headline>
  <kurt-appcontainer>
     <kurt-tagger id="tag1">
       This is an #active #web #component. You can type into the 
       #text_area (this #pane), and every #hashtag (#) will become a 
       #button that returns a #dictionary entry.
     </kurt-tagger>
     <kurt-defs id="defs1"></kurt-defs>
     <kurt-suggest id="suggest1"></kurt-suggest>
  </kurt-appcontainer>
</kurt-app>

This is what a web component applications can look like. It is “nominally” in HTML, if you look at the web component foundation as being an extension of the HTML space. The elements are labeled with some kind of identifier (here ‘kurt-’) that serve to identify a library of components in my particular domain (what in XML terms would be considered a namespace prefix). The first line — the link import statement — brings the app into the current page, and at each linked page related links make up sub-components. For instance, the component that displays a definition is a separate file (kurt-devs.html) that contains both the script definition and the style and structural form of the component.

<script> 

Element `kurt-defs` 

(class Defs extends HTMLElement 
 {
   constructor(){
     super();  
     const t = kurtDefsTemplate;
     var clone = document.importNode(t.content, true);
     this.appendChild(clone);
     this.termsTemplate =  Template `term`;
     this.defsTemplate = Template `defs`;
     this.style.display='none'; 
    }
   
   load(obj){
      this.base=obj;
      var entries = this.base.entry_list.entry;
     if (!this.base.entry_list.suggestion){
      var entries = Array.isArray(entries)?entries:[entries]
     var name = entries[0].ew;
     var pos = "["+entries[0].fl+"]";
     var pronounce =entries[0].pr;      
     var et =entries[0].et+""; this.termsTemplate.bind({
       name:name,
       pos:pos,
       pr:pronounce,
       et:et
     });
       
     var defs = Array.from((""+entries[0].def.dt)
         .split(/[:]/).slice(1))
         .filter((item)=>(item.length>5))
         .map((item)=>{return {def:item}});
      
      this.defsTemplate.bind(defs);                                       this.style.display='block'; 
                                                    
     }
     else {this.style.display="none"}
    }
   });

</script>

<template id="kurtDefsTemplate">
  <div class="defs-container">
     <template name="term">
       <div>
            <span class="term">{name}</span> 
            <span class="pos">{pos}<span> 
            <span class="pr">{pr}</span>
       </div>
       <div class="et">{et}</div>
      </div>
    </template>
    <ol class="def">
       <template name="defs">
           <li>{def}</li>
       </template>
    </ol>  
  </div>  
  <style>
    .defs-container {
      border:inset 2px gray;
      width:400px;
      min-height:200px;
      padding:5px;
    }
    .def {}
    .term {
      font-weight:bold;
      text-transform: capitalize;
    }
    .et {font-style:italic;
      margin-top:5px;
      margin-left:5px;}
  </style>   
</template>

These component are modular — stylesheets, for instance, only apply to the components where they are defined. Additionally, templates can be used to promote data binding, associating both getters and setters in the associated class with variables, and making it possible to bind Javascript objects (and object arrays) with specific sub-templates. This can be seen specifally in this part:

<ol class="def">
       <template name="def">
           <li>{def}</li>
       </template>
    </ol>  

<script>

this.defsTemplate = Template `defs`;

var defs = Array.from((""+entries[0].def.dt)
         .split(/[:]/).slice(1))
         .filter((item)=>(item.length>5))
         .map((item)=>{return {def:item}});
      
      this.defsTemplate.bind(defs);

</script>

The defs array does some cleanup from the data source (loaded in earlier) and has the form:

defs = [{def:"This is definition 1"},{def:"This is definition 2.",...]

The statement this.defsTemplate,bind(defs) then iterates through the definition array, looking for the def property and passing the value into the {def} element in the template.

Web Components vs. React

There are many other aspects to web components that are worth digging deeper into, and will likely be fodder for future articles. However, in this post, I’d like to consider why I think this will have such a big impact upon web development vs. React, looking at what many consider the strengths of React.

Separation of Concerns

React works by focusing specifically on the View aspect of the MVC paradigm — you have a single overarching data model, then when something changes in the model, React acts as the controller to modify the view. However, it also places the onus of managing state into the hands of the developer, and despite a design philosophy to the contrary, the reality is that much of this state management could and should be more readily be handled within an encapsulated environment.

Javascript has also been evolving to better handle asynchronous events well, especially with the advent of both Promises and async variables. The Web Components (WC) approach lets you create composable applications without the need for heavy state management. React requires MobX or similar state management APIs. A WC approach, on the other hand, lets the component manage it’s own state, hence only needing to be written once.

This also works better when dealing with immutable data structures — if you essentially treat what data comes in from the outside as static, then any output (created by using the immutable streams to build new structures) will always be both declarative and consistent — there will be no unexpected side effects.

In practice this means that a web developer can create a component (with the assistance of a UX designer), and make it available as a library, then a web author can just use that component, without needing to know its inner structure or working mechanisms.

Ease of Development

React (with an add-on Javascript Extension (JSX) library) can be used to quickly prototype components. It uses an HTML-like language to bind attribute properties, and state changes can occur when you modify the properties of the state object, and React builds a specialized style language for setting changes in appearance

Web components don’t need an HTML-like language to build templates, because they use HTML itself. Web Components are easy to write. After a couple of years, I still stumble over some arcane aspect of React that I “should have known”, in part because there are so many different state management libraries that each impose their own requirements, in part because there are many places where the HTML-like language is not in fact all that HTML-like.

With web components, I was writing basic components in a few hours and reasonably complex ones (such as the one above) in a few days. These components are fun to write, and code reuse is high — I’m already building web applications that I’ve been struggling with in React in days rather than weeks. What’s better — I don’t have to spend so much trying to figure out various support libraries that have poor documentation, or spend all my time compiling.

Performance

React’s biggest claim is it’s ability to create a separate shadow DOM that is used to make changes, then only modify those things in the live DOM that have changed in the shadow DOM. This results in significant performance improvements, even despite the cost of managing the shadow DOM.

In practice, in Web Components a template is an inert DOM, but can be used at various levels within a given application. The Template literal operator lets developers create a duplicate of the template (using one of the slickest bits of TemplateLiteral programming I’ve seen to date), make changes to that, then either insert or replace live nodes in the document when everything is ready. Mostly, this process is very transparent — and very, very fast.

Web Components are something that will be built into the browser. Currently for Chrome and Firefox there are small polyfills because not everything is supported yet, but much of it is. Microsoft is also heavily engaged in working with web components, and has been an active participant in the W3C Web Components (W3C-WC) group. This means that the amount of information that needs to be sent over the wire is tiny — in some cases only a single TCP/IP packet.

In a number of benchmarks I’ve seen, a W3C-WC version of a web application is in and rendered before the React based version has completed loading, and once loaded, both can make use of caching to reduce transmission latency. React components run through node can take a while to compile, and while the React library itself . W3C-WC components, on the other hand, piggyback on the browsers’ support for debugging.

It is worth noting as well that Web Components can be served up by any server, not just nodejs, and in theory can also be precompiled into the javascript package — it just doesn’t have to be. Since the bulk of web component code is built into the browser, such compiled web components are usually very lean and have much less likelihood of collision, especially with larger projects.

Browser Support

Because React is precompiled, multiple versions that facilitate different browsers can be produced without additional code on the part of developers. It was in fact this need for optimization of browser specific code that justified an HTML-like layer in the first place. Given the rapid change in the Javascript framework in the last few years, this was an unavoidable requirement.

However, there are distinct signs that the previous rapid pace of Javascript evolution is slowing. Most of the “to-do” items on the Javascript roadmap from 2015 have been accomplished, and what is left are very much edge case issues with respect to the browser (the introduction of fixed byte-width data types being the most significant).

This in turn has focused attention to the web components layer. At this stage, Microsoft has been engaged in a significant rewrite of their underlying DOM Architecture in preparation for this (most especially in areas such as the Shadow DOM and CSS Custom Properties) and Mozilla, Google and Apple have been working closely on keeping both the specification consistent and building this component architecture into their respective browsers.

Google Chrome is perhaps the farthest publicly along in this regard. Mozilla Firefox has also been working heavily on this specification, though for the moment these are hidden behind a custom flag (see https://developer.mozilla.org/en-US/docs/Web/Web_Components):

The snuggi library provides a thin up-to-date support for Web Component features in Chrome specifically, and was used for the application discussed earlier:

<script src=”https://unpkg.com/snuggsi">

This last point means that in this one crucial area, Web Components are not yet ready for prime time, though that should change by late 2018 or early 2019. The primary blocker to date is support for shadow DOM, which is where most development efforts are now going. This makes it a good time for web developers to start becoming proficient with the tech, especially in the Chrome ecosystem.

Implications of Web Components

As web components become more heavily adopted, one effect of this will be a shift towards the development of WC component libraries. Currently this exists on the back end for Javascript by itself, but the W3C-WC spec opens up adoption of an import style to make requests for specific library components within a web browser. Because such products are self contained (and perhaps self-documenting) they can be downloaded independently of one another, which reduces potential code version problems.

Another area where I believe Web Components will represent a huge win will be in the domains of code security and licensing. Because these are self-contained units, such components can be certified and have an encrypted hash associated with them through certification boards (most likely private). Similarly this makes monetization of component libraries easier, in which basic capabilities may be freely available but advanced capabilities may be opened for a licensing fee.

Overall, though, I believe that web components represent the final evolutionary stage for web client development. They can be developed inline, can be imported, can even be compiled. They can be combined with one another, communicate with one another, and can be responsibly forked and versioned. It’s a compelling version for the web, and one that I think we are close to achieving.

Kurt Cagle is a writer, blogger and information architect, who has been writing on web related technologies for more than thirty years. He lives in Issaquah, WA, with his family and cat Bright Eyes.

#javascript #react #json #webComponents #w3C #TheCagleReport

Web Components: Moving Beyond React was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.

We live in the age of data. Around us, databases contain exobytes about us and the world that we live in. There is a sense, especially among managers and decision makers, that all of that data should be able to tell us something, especially after nearly a decade where “Big Data” as a meme has become one of the most marketed phrases on the planet.

Yet for all of that, the millions upon millions of dollars spent on big Hadoop projects, enough data lakes to turn into a data sea, the rise of the Data Scientist as rock star (okay, I’ll admit, I didn’t see that one coming), the reality has been … underwhelming. That ten million dollar investment in high volume systems resulted in a one million dollar total (not net) return over the course of a couple of years. That data lake sits underutilized, because developers continue working with mySQL databases. They’re faster. NoSQL has become, “well sort of kind of SQL”. We’ve built multiple social media platforms that nobody is using because they’ve used all the top spots on their phone.

Those data scientists? They spend most of their time dealing cleaning up the same crap data that they did before they became rock stars, just at a higher salary. Moreover, while they can tell you things like how data clusters and the fact that you have a 46% chance of making a 1.5% profit this quarter, a 38% chance of making a 0.5% profit this quarter, and a 16% chance that you’ll take a 3% loss. Yet when you take that 3% loss, they get fired, and the remaining data scientists very quietly readjust their models accordingly.

This is not the fault of programmers or data scientists, and arguably not completely the fault of system architects (though they shoulder some of the blame). They are in fact doing what they are being asked to do. A bigger part of the responsibility for this is directly due to business managers who do not understand what should be the central concept of databases … the notion of context.

Here There Be Dragons

Context is a remarkably subtle idea, one that is perhaps so subtle that it bypasses people who work with information every day. Context is the set of assumptions and relationships that information has based upon the environment around it. It determines uniqueness, categorization, what relationships are important and what aren’t, what metrics are used and so forth.

A good example of a context problem was the Y2K problem, or the fast-approaching Y2.038K problem. Many database designers from the 1960s to the mid-1990s made a context assumption — that they were in the twentieth century, and so didn’t have to rely upon the first two digits of a year being anything but “19”. Of course, as the twenty-first century loomed, someone realized that if something wasn’t done, a lot of people were going to suddenly find themselves having been born in the future. This made COBOL programmers very rich at the end of the twentieth century.

A similar problem will happen in 2038, by the way. On most computer systems (and the databases written for them), dates are calculated as the number of seconds from 1970. As it turns out, many databases until a few years ago allocated a long integer, which was 31 bits long (the remaining bit was used to set the sign of that integer). Two to the thirty first power is about 2.147 billion seconds, which implies that in that period (about sixty eight years), dates will start going negative if the maximum size of a date stored in a database is 32 bits. 1970 + 68 is, you guessed it, 2038, so in about twenty years, any legacy databases are about to go very wonky.

The context here is subtle — the representation of the data in a specific field was only valid in a certain range. This is why contextual problems are so hard to resolve — they are all too often at a deep or subtle enough level that even realizing that it is a problem in the first place can be tricky. In retrospection, anyone could have seen it, but in practice, most people do not challenge their assumptions this deeply. They lack either time, expertise, or desire to do so, assuming that these kinds of problems are “edge-cases”. Indeed, an edge-case is simply a context assumption that the problem won’t bite the programmer in the butt while they still have a job. After that, it’s someone else’s problem.

A similar problem comes about due to identifiers. Most relational databases (and even many NoSQL databases), make use of numeric keys to identify specific resources. Early on, databases administrators or programmers used sixteen bit integers for those keys, meaning that they were good so long as you didn’t have 65,539 rows for a given table you were golden. After that, the indexer would roll back to 1, and suddenly you begin getting duplicate entries to certain queries. So these were switched to long integers, because after all you’d never have two billion entries in a given database, right? Twitter produces that many entries in a day. Now, developers have finally begun working with UUIDs, which have two to the 128th power possible values, or roughly 340,000,000,000,000,000,000,000,000,000,000,000,000 potential values.

Now, the chance for collision here (where two different resources are assigned the same GUID) is not 3.4x10³⁸. Many GUIDs are based upon specific randomizer algorithms, and creating a truly random seed is actually one of the hardest problems in computer science. However, the possibility that it will come up is small enough that for all but the most specialized of cases, the numbers will be unique.

Again, this discussion problem is contextual. A numeric identifier is only unique within a given database, the number that I assign to that book will be different in a publisher’s database will be different (most likely) than the one that may be in a bookseller’s database. The number is meaningful only if the context is fully qualified: publisherX:1953302 is different from bookseller:1953302 but may be the same as bookseller:3295322. Master data management works upon the assumption that two books can be deemed the same if they happen to share an agreed upon third identifier (say an ISBN number), but may be the same if additional contextual evidence (a book’s publisher is the same, the author(s) are the same, the title is the same and the version is the same).

Now, even the first condition above is not a guarantee — someone may have mis-keyed an entry. It happens. A lot. This means that in order to identify that two books are “probably” the same, you need to do forensic analysis — analysis of information afterthe facts.

Now forensic analysis is hard in traditional databases, because in order to make the comparison, you need BOTH databases to be instantiated at the same time in the same server, then have to make sure that the properties that each table has are equivalent. In the real world, that never happens. Ever. In very rare cases, standards are set up (and rigorously enforced) that identify common schemas, but even there, the likelihood that such schemas are enforced and consistently validated and constrained is virtually nil … and that’s a best case scenario that I, in thirty plus years of working with both structured SQL data and semistructured XML and JSON data, have never actually seen.

Transmogrification

Most data scientists would love to spend all their time doing statistics and generating cool looking reports. In reality, most data scientists spend the bulk of their time (90% +) attempting to unify data from disparate sources, clean it up, map it with some kind of transformation, then save it out into a different data format that matches their working tools, a process known in the database community as Extract/Transform/Load (or ETL). Of that, the extraction and loading process is straightforward. The transformation process is … not.

There are many reasons for this. Some are structural — one modeler may represent a user name as a single field, where others may represent it in two or three. One may represent the name first then last, another may represent it as “last, first”. These require knowing how to work with a computer language to do the mapping. Others may have revenue projections by month rather than by quarter — and it may not be obvious that this is the case. Others may define revenue as gross revenue, a second as net revenue. One may use FASB standards, others may use Basel III. These differences are contextual, and often may not even be known by the programmer at the time (most programmers have only a very limited understanding of accounting unless they spend a huge amount of time in the financial sector).

Teasing out the metadata that represents the contextual underpinnings takes time, and frequently involves injecting new data into the system that better establishes this context. Thus, even if the same entity is identified in two different databases, the likelihood that the data that each row contains will be very different.

This is one reason why open data, usually governmental based, is so important. That data doesn’t (necessarily) give you the context, but if you work from that data it guarantees that you are sharing the same context, whether it is stated or not. Unfortunately, due to one of those larger contractions that occur at the end of large scale periods of growth, countries are becoming increasingly nationalistic and disinclined to share that data, which means that the easy solution of having common data sources is disappearing. This means that companies will need to focus more on building contextualization than they have in the past.

Semantic data stores represent one potential solution. These are a form of database that allows you to encode not only the data, but the rules that identify the logical model of that data in a consistent fashion. It makes it easier to do master data management (as multiple “graphs” can be stored and worked on simultaneously), and it can also store additional information such as properties of relationships, and common controlled vocabularies. They make cross-graph comparisons much easier to do, and they make it possible to store new kinds of information without the need of restructuring the existing database to do so.

What’s more, once stored in this fashion, it can be queried using either a common programming language called SPARQL or, in many cases, can be queried in a more SQL like manner through a bridge (which increasingly is built into the store). Because SPARQL data is (mostly) normalized, it can map easily to SQL databases and can also map to denormalized structures such as JSON or XML. This is expecially true as JSON works towards exposing a schema language called SWAGGER even as RDF (the generic framework for triple store databases) has both OWL and the recently established SHACL language, to match the XML Schema Language (XSD).

This is a critical development, because data instances are notoriously difficult to reverse engineer to a centralized schema. In JSON, for instance, a database using object keys is difficult to discern from a generic object, while an array of objects may have no context information about what those objects are. By establishing schematic languages across all three formats, it makes interchange (and mapping) a MUCH simpler problem. I started working with interchange issues for Microsoft’s BizTalk server in 1997, and it has remained a major thorn for the majority of my professional working life.

The remaining challenge, in that respect, is, ironically SQL. SQL is challenging because it has only limited ability to be self-referential, it cannot handle inheritance well, and it generally cannot, within a query, reference information about its own schema — which means that SQL context usually only exists within the heads of a limited number of application developers. SQL also does not handle distributed data well. There are signs that serialization standards especially are emerging, but there’s a huge installed SQL/RDBMS base for which serialization is the responsibility of the programmer, not the data store.

Serialization is the flip-side of context. When you serialize information, you are usually taking a subset of that data, filtering it by some matching algorithm, and often renormalizing data (folding hierarchies). Yet the reality of most data is that it has multiple axes of connectivity, or mechanisms for categorization. Context is almost always what mathematicians refer to as a graph. When you analyse the resources described by that graph, especially when you incorporate categorization into the mix, you are choosing a bias on what connections to break, what categories (and how they are defined) are important.

There’s a Hole in the Bucket

Ontologists, taxonomists, machine learning specialists and data scientists spend a huge amount of their time determining how data is categorized, because it is usually much easier to work with discrete groups than it is a continuum computationally. It is how the brain handles so much of its own processing.

A book or movie can be modeled on the basis of its genre — that is, as a cluster of plot elements that are frequently found in general proximity which appeal to a certain market demographic. Does it have robots in it? It’s science fiction. Does it have unicorns in it? It’s fantasy. The problem comes when it contains both robots and unicorns in it (e.g., Blade Runner). Or vampires and detectives, or any other myriad hybridizations. Ultimately, one of the biggest challenges found in any modeling is that the real world is very seldom as orthogonally reducible as we’d like to believe, especially when dealing with conceptual rather than physical characteristics.

Context plays a part here as well — our choice of categorization are often modeled upon our own preconceptions, regardless of the real world. Gender, employment, race, education, industry, generational cohort, social class, all of these are arbitrary and defined largely by the bias of one organization or another, often to the detriment of those who fall through the cracks. Is a writer who gets paid through royalties employed? What about someone who’s independently wealthy and writes books to influence others? Both are doing the same activity, all that has changed is the nature of the monetary transaction.

Semantic systems are generally more ideal for capturing these nuances, because categorization can be defined inferentially. Machine learning can also be effective in this regard because it has (little) fundamental bias in choosing clustering, so can identify when resources cluster, but it still becomes necessary to identify why those clusters occur. If you feed an image recognition systems with pictures of red squirrels, and then ask it to identify whether a gray squirrel is a squirrel, it may answer negatively. The system needs training, which in turn means that the selection of the training set is again a contextual constraint that introduces bias in the categorization.

Semantic (contextual) systems make it possible to capture the context of such categorization, to examine the rules (which may themselves be inferred) that determine why a given category exists, and that in turn can create subtle variations of classifications that more accurately represent the information about resources.

I like to think of context as a map with a marker saying “you are here”. The marker by itself has no meaning, it is only when emplaced within the map that the marker has meaning. You need both marker and map to know where you are. Or as Buckaroo Bonzai once said (in turn quoting Confucius) “No matter where you go … there you are.”

Kurt Cagle (kurt.cagle@gmail.com) is a writer and ontologist who has been puzzling over the notion of context for years now.

#TheCagleReport

The Importance of Context was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.

How do you know what’s true? This seemingly simple question underpins a surprisingly large number of issues that we as information producers and consumers face every day. From the political sphere, memes and alt-facts have become commonplace, threatening to engulf us all in a morass of disinformation. Data necessary for finance and business analytics goes through dozens or even hundreds of points, and the attempt to mine data from social media introduces biases and questions about how valid such data really is. Even in very carefully controlled environments, the question of where information comes from should be central to anyone working with data, though all too often isn’t.

Provenance is a term that in data circles means the specific history of a given piece of information. In the age before computers, such provenance was typically relegated to academic works. A bibliographic citation in a book would indicate where a particular assertion, table or quotation came from. If you retrieved the particular work being referenced, then you could look up the source of information and the context in which it was used. Such a cited work itself may have its own citations, pointing to earlier works. In effect, citations created a web of trust — at any point you could locate all of the previous works, and as importantly, could evaluate the source of these works for their validity.

As such, a provenance trail established the degree of authority that a work had. It didn’t necessarily guarantee truthfulness — an author is simply someone who creates a given work — but because of the complexities and time consuming nature of publishing, it was more likely that there had been a peer review process to insure that the work bears some reasonable semblance to what really happened. The rise of authority is critical, because in general we are reliant upon others to provide information that we could not ourselves obtain — we weren’t there at the time, we weren’t witnesses. An untrustworthy witnesses throws into doubt the whole provenance chain.

The English legal system is built upon authoritative testimony, attesting both to the significance of circumstantial evidence, the perceptions of witnesses at the time and character assessments on the part of those who were involved with one or the other of the plaintiffs or defendants. Ledgers developed as a way of recording transactions, not just for managing current accounts but also to provide an authoritative record of previous transactions. It is one reason that accountants are held to such high levels of trust (and are penalized so heavily if they fail in their duties to properly record such transactions) — they are authoritative witnesses.

News organizations, before they became entertainment, were judged on their reliability and their adherence to the truth. A yellow newspaper (a term that originated from the fact that the cheap newsprint used for publication would yellow more quickly because of the high acid content in the paper) was one that made up stories that had no authoritativeness, no legitimacy, and were often sensational, salacious and quite frequently not only crossed the line into libel, but routinely romped and stamped on that line to blur it out of all recognizability.

The rise of television produced a new type of news. A newspaper had a comparatively high bandwith — any single issue may have dozens of stories and was served daily, but any such news was always, implicitly a day old, and was only open to those who were proficient readers. Radio (and later television) news, on the other hand had a much lower bandwidth — you could get perhaps a dozen articles tops in a news program. However, human beings are narrative creatures. A person speaking a story was compelling in ways that print could never be, especially if that person was also seen speaking. Radio and television news activated our hindbrains, the part of the brain that bypassed the filter of rational thought and went straight into memories.

There were giants in the radio and television field — Edward R. Murrow and Walter Cronkite are two that come to mind — who championed the need for provenance in their work, but there were also others who saw the potential for social engineering on a massive scale by bypassing this provenance trail through the new media. It is perhaps not surprising that in the current culture wars, those most likely to hold beliefs without substantiation are also those that were most heavily reliant upon radio and television for their news.

The Internet changes that equation. The key aspect of the Internet is the hyperlink. A hyperlink — the underlined blue text you see in a browser that lets you “click” to a different “page” — is in fact a form of citation. This makes it possible to better determine the provenance of an article, though it doesn’t necessarily insure the truthiness of an article’s source. However, what it does enable you to do is to aggregate sources and determine, based upon contextual records, how likely that domain reflects known facts. It also becomes possible to crowd-source the process of determining the legitimacy of a domain as an authority. Snopes and Politifact have built their business models on providing ratings about the trustworthiness of a given assertion or political record, and other are emerging as the market for legitimacy gains steam.

What such rating systems do is provide a mechanism for quantifying trust. This mechanism requires a combination of several factors: crowd-sourcing reliability metrics so that biases can be more readily identified and factored out, establishing a base-line history of news providers to determine the degree to which they change over time (due to changing editorial policies or ownership), and analyzing this in light of where the consensus is for a given piece of information.

This is not “news” in the traditional sense, more along the lines of “pre-news”, determining the authority of a given source to better provide that information in a relatively unbiased manner. Not surprisingly, the modern day analogs of yellow journalism do everything they can to discredit these authoritative metric organizations. The next wave of contemporary journalism will almost certainly be a battle between these two forces, just as these are played out in other areas such as political polling. Most campaigns maintain two polling campaigns, one to provide (or in some cases create) sentiment data that shows their candidate doing well, the other working with the internal ‘books’ that show how the campaign is really doing.

The hottest technology in the fintech space, blockchain, deserves a mention here as well. Blockchain is intriguing because while it is usually seen as a measure of financial transactions, blockchains also provide mechanisms for identifying provenance information — who makes a claim, when was the claim made, what was the basis of that claim and so forth. Because blockchains are distributed, they can in effect record when changes are made into a set of transactions and can be used to ascertain when spurious data is introduced.

Again this does not guarantee that the transactions themselves — the assertion of certain acts — are true, only indicates when and where the assertions were made. However, even that is a huge step forward, especially if some measure of trustworthiness can also be made about the transaction. People do not like to lie when such a lie can be recorded, because it makes it much harder for them to deny they made the lie in the first place. There are very real legal and financial penalties for those lies, including fines and incarceration.

Semantics, on the other hand, makes it possible to make assertions about assertions. If blockchain is the ledge, semantic data provides the contextual metadata for those assertions, in effect creating tools to link the assertions to other assertions about the things being recorded.

This has significant impacts upon business as a consequence. As we enter farther into the reputation economy, a reputation for honesty and factual reliability will become a distinct competitive advantage, especially in an increasingly virtualized world. Indeed, in many respects, such an authoritative score will very likely factor into everything from company valuations and stock prices to available to customer bases.

Already, an informal word of mouth system can have a major impact upon the viability of companies (cf. the recent scandal that engulfed United for forceably ejecting a paying customer because of a logistics error, costing the company billions of dollars in market valuation). Expect as provenance becomes more readily determinable that it will play a major part in the reputation economy.

Provenance is crucial. I hope, in a subsequent article, to look at the mechanics and modeling of provenance, but the idea behind it is already becoming a major factor in the way that business is done in the twenty first century.

Kurt Cagle is the editor of The Cagle Report on Linked In and the Metaphorical Web.

Provenance Is Not A Region In France was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.

Programmers love a fight. Ask a roomful of programmers the question “EMACS or VI”, leave the room for coffee, come back half an hour later and there will be blood on the floor. For data storage buffs, the same question, at least lately, comes down to SQL vs. NoSQL. Each has its defenders, its book of holy writ, its benchmarks and success stories … and each has its dark secrets and stunning failures as well.

Yet the question of whether data content is SQL vs. NoSQL misses the bigger picture issue, which can be summarized most readily as “How structured is your data?”. Put another way, to what extent is your data normalizeable.

Not that long ago, the distinction between data was pretty clear cut — you had “data” — rows and columns and linked references — then you had documents where the information was contained in a single streaming narrative. You knew where you stood in those days (about thirty years ago). You simply couldn’t put document content into a database. It made no sense to.

In the late 1960s, as computers began to be used in more and more interesting ways, a document made use of markup to provide signals to the typesetter (either human or electronic) about how a given sequence of text was to be rendered. A number of different systems emerged, most proprietary, but one or two that were agreed upon as open and standardized. The most heavily utilized of these was a language originally called the Generalized Markup Language (GML), but would eventually becoming the Standardized General Markup Language (SGML) as different proposals made their way through both US and later ISO standards processes.

Tim Berners-Lee’s great genius lay in creating a dialect of SGML called the Hypertext Markup Language (HTML) and tying it into a linking protocol (HTTP). SGML was a set of rules for creating languages, and HTML was one such language. It was not especially rich when it was first laid out, as its primary purpose was in the construction of citational abstracts, but over time it became obvious that you could extend this language in any number of different ways.

A parallel track was occurring as the SQL database took form. Ted Codd of IBM had been looking at different data systems through the early 1970s, and realized that you could take complex objects and decompose them into linked tables, using these to navigate across complex structures by joining these tables at the link points. He introduced the idea of normal forms that in essence handled the various kinds of cardinality relationships that data can have.

Cardinality has always been a problem with data — indeed, it is perhaps the CENTRAL problem. Most data ultimately comes down to four types of relationships:

• Optional data (minOccurs = 0, maxOccurs = 1) where content for a given property could be absent or present. Optional relationships are complicated because you still have to include a field for data in a relational table, but also have to incorporate the notion of NULL — a property has no associated value in a particular case.
• Required data (minOccurs = 1, maxOccurs = 1) where for a given table property there is one and only one value (this is called a functional relationship, by the way).
• Unbounded (minOccurs = 0, maxOccurs = infinity) where a property may or may not be present, and if it is present then there are be any number of links.
• One or more (minOccurs = 1, maxOccurs = infinity) where there may arbitrarily many connections, but at least one exists.

Ordering also plays a huge part. The moment that you shift from maxOccurs = 1 to maxOccurs = infinity you introduce a bias towards a particular ordered sequence of items that is largely machine dependent. Additionally, you have to deal with the problem inherent with functional tables — you cannot put more than one value into a given field for a row in a table without some kind of encoding process, such as converting all entries to strings and then using a delimiter such as a comma, tab or pipe to indicate separation.

Instead, you have to normalize the relationship, by creating a primary key identifier for each row in the “subject” table, and then creating a foreign key reference to that primary key. This way you preserve a functional (1 to 1) relationship between rows of each table. You can think of this as a directed reference from the subordinate row to its parent, where the subordinate row “owns” that reference (bookmark this concept of ownership — it’s VERY important). Who owns references is fundamental to modeling.

The moment that you create a link to a separate table, you are in effect creating an object. Indeed, even atomic values are instead links to objects that have scalar values and dimensional attributes, event markers and attribution, but in SQL relational model, most of these are subsumed in schema in order to keep computation manageable.

Most relational DBAs tend to focus on normal forms, though in general, these effectively describe which objects own which vectors. First normal form indicates that a generic object also maintains its own data, while in a second normal form, each row of a join table points to a different row in different tables. Second normal forms typically describe many to many relationships, such as the relationship between a book and an author (an author may have more than one book that they have contributed to, a book may have more than one author contributing to it). Third normal forms and so forth then describe more specific structures.

One of the major realizations made in the early 1990s was that you could in fact describe a data structure using markup. To do this, it was deemed necessary to simplify the SGML structures into a more limited dialect which was eventually named XML. This had two major effects. First, it meant that you could in fact begin to decompose a document into a tree structure (or a forest of tree structures) using a document object model (DOM). The second was that you could incorporate metadata into this model that made it possible to schematically describe this structure.

XML originally was designed upon the assumption that order was not a part of the the structure (that there was no guarantee that nodes should return in document order), but in point of fact sequential order was implicitly assumed shortly after the language was adopted. The XML language also assumed the existence of invisible key/key reference pairs that held the structure together. This meant that a DOM collapsed certain information that were explicit in a SQL model, making it possible to create complex structured resources.

Indeed, in many respects, an XML or JSON is not so much “semi-structured” but “super-structured”, in that it had more structure (less entropy) than SQL tables — an XML structure can always be reduced to to a set of table rows, but in doing so, more information needed to be generated (such as ordering priorities and key/ref pairs) to do so.

This actually brings up one of the bigger conundrums of translation between XML and JSON. XML has a concept called a sequence, which can hold element nodes, text or related nodes such as comments or processing instructions, and atomic values such as strings, dates or numbers of various sorts. If you insert a sequence into a sequence the inserted sequence is “dissolved” and the items are added at the point of insertion directly. JSON has no such structure, but it does have an array. When you insert an array, the inserted array itself becomes a distinct object.

In general, these subtle differences can be worked around, but they make it clear why working with document content in particular is still such a challenge. Most documents do have clear structures at the macro-level, but the issue of dealing with tagged content content at the micro level (such as a mixed bag of paragraphs, headers and images) make for encoding this information at the micro-level difficult.

This is another area where semantics (and RDF in general) can serve as an important bridge. For instance, sequences and lists can be surprisingly complex structures. RDF can describe arrays as well as linked lists. A linked list is known in RDF as a collection, and consists of a “backbone” of blank nodes:

book:StormCrow  book:chapters _:list.
_:list rdf:first     chapter:StormCrow_Intro.
_:list rdf:rest      _:b1.
_:b1   rdf:first     chapter:StormCrow_Ch1.
_:b1   rdf:rest      _:b2.
_:b2   rdf:first     chapter:StormCrow_Ch2.
_:b2   rdf:rest      _:b3.
_:b3   rdf:first     chapter:StormCrow_Ch2.
_:b3   rdf:rest      _:b4.

...

_:b21  rdf:first     chapter:StormCrow_Ch20.
_:b21  rdf:rest      _:b22.
_:b22  rdf:first     chapter:StormCrow_Epilog.
_:b22  rdf:rest      rdf:nil

In this form of list, if you have the starting address, then you have a link to each chapter in order as well as a pointer to the next “vertebrae” in the spine. The downside to this approach is that you can get all of the items from such a list easily enough in SPARQL, but you have no guarantee of order:

select ?chapter ?baseNode ?nextNode where {
     ?book book:chapters ?chapterList.
     ?chapterList rdf:rest*/rdf:first ?chapter.
     ?baseNode rdf:first ?chapter.
     ?baseNode rdf:rest ?nextNode.
}, {book:iri(book:StormCrow)}

=>
     chapter           baseNode     nextNode
chapter:StormCrow_Ch2    _:b2         _:b3
chapter:StormCrow_Ch7    _:b7         _:b8
chapter:StormCrow_Ch13   _:b13        _:b14
chapter:StormCrow_Ch10   _:b10        _:b11

This doesn’t mean that you can’t get the list ordered outside of SPARQL (it’s a pretty easy script to do so in Java or Javascript), but it does mean that linked lists are not obviously equivalent to arrays.

An alternative approach is to use indexes that specify the order as a weight:

book:StormCrow  book:chapters _:list.
_:list :hasMembers
     [:resource chapter:StormCrow_Intro;
      :index "1"^^xsd:long],
     [:resource chapter:StormCrow_Ch1;
      :index "2"^^xsd:long],
     [:resource chapter:StormCrow_Ch2;
      :index "3"^^xsd:long],
     
     ...,

     [:resource chapter:StormCrow_Ch20;
      :index "21"^^xsd:long],
     [:resource chapter:StormCrow_Epilog;
      :index "22"^^xsd:long]
.

There are several advantages to this approach. First, it corresponds far more closely to the concept of an array. Getting a listing in order (in SPARQL) becomes a matter of specifying the index as a sort key:

select ?chapter where {
    ?book book:chapters ?chapterList.
    ?chapterList :hasMember ?member.
    ?member :resource ?chapter.
    ?member :index ?index.
    } order by ?index,
{book:iri(book:StormCrow)}

Additionally, retrieving a given indexed entry becomes trivial — it uses the same query, but fixes the index to a specific value (here, the fifth chapter in the book).

select ?chapter where {
    ?${book} book:chapters ?chapterList.
    ?chapterList :hasMember ?member.
    ?member :resource ?chapter.
    ?member :index ?index.
    } order by ?index,
{book:iri(book:StormCrow),index:5}

(Here, I’m using ?${book} to indicate passing the identifier key for the book, something akin to how template literals work in Javascript.)

If a chapter is contained in more than one book, you can determine all the books that contain the chapter without having to walk a chain:

select ?book where {
    ?book book:chapters ?chapterList.
    ?chapterList :hasMember ?member.
    ?member :resource ?${chapter}.
    ?member :index ?index.
    } order by ?index,
{chapter:iri(chapter:StormCrow_Intro)}

=>

    ?book
book:StormCrow   # As the first 'chapter' in the book
book:Magi        # As a teaser chapter at the end of the book for the sequel

Indeed, what’s most notable here is that the body of the query in all three cases is the same — it’s only the item selected and the constraints that change. If we extend this with a title and a publication date then it is possible to create one query that can respond to a large number of potential services:

select ?${selections} where {
    ?book     book:chapters ?list.
    ?list     list:hasMember ?member.
    ?member   list:resource ?chapter.
    ?member   list:index ?index.
    ?book     book:publicationDate ?pubDate.
    ?book     book:title ?title.
    ?chapter  chapter:title ?chapterTitle.
    } order by ?${sort},
{selections:"?book ?pubDate",sort:"asc(?pubDate)"}

=>

    book            pubDate
book:Mage            2017
book:StormCrow       2019

It’s also worth noting here that ordering is extrinsic in certain types of lists (such as where a chapter occurs in a given book list), but not in others (you can sort by date or alphanumeric code, or even some combination thereof).

RDF consequently can be seen from a data standpoint as something that straddles the distinction between document and data of the various other formats. You could create an ordering structure for decorator elements in a paragraph (such as <b>, <u>, <a> and so forth), in much the same way, although admittedly this gets cumbersome:

[] a html:P;
    il:hasIndexedList _:list.
_:list :hasMembers
     [il:resource [html:literal "This"; a html:B];
      il:index "1"^^xsd:long],
     [il:resource [html:literal "is"; a html:U];
      il:index "2"^^xsd:long],
     [il:resource [html:literal "a"; a html:I];
      il:index "3"^^xsd:long],
     [il:resource [html:literal "test"; 
         html:href "http://www,mytest.com"; 
         a html:A];
      il:index "4"^^xsd:long],
     [il:resource [html:literal:"."; a html:text];
      il:index "5"^^xsd:long]
.

I’ve put this particular indexed list into the “il” namespace (for now, the namespace itself doesn’t matter) primarily to indicate what is in which domain. The indices here are part of the generic member entities, not the specific resources. In this respect the blank node

[] il:resource owl:Thing;
   il:index "1"^^xsd:long;
   # a class:indexedListMember
.

is an indexed list member, as shown by the commented statement above.

This use of intermediate “helper object” such as an indexed list or a linked list is important because they shift the management of structures away from the semantics of the primary ontology (html here) and towards a separate data structure space that can be used in a large number of contexts.

Given the advantages of an indexed list, it’s odd that the RDF specification doesn’t have an explicit structure for indexed lists (it sort of does, if you look at the definition of rdf:Seq, but the solution is weak). The reasons are largely historical:

1. Linked lists are faster to work with in OWL
2. Inserting an item into the middle of a sequence is easier to do with linked lists.
3. Most stack operations are faster on linked lists.

Indeed, this highlights the distinction between the two in a way that gets lost in languages like Javascript (and is completely obscured in XML). An indexed list is a different structure than a stack, which is a linked list. If all of my operations involve pushing and popping things onto a stack, a linked list is a great thing: new items are pushed onto the head of the stack, and the pointer to the head then gets changed accordingly to the new item, a classic Turing action. Popping does the inverse action, setting the head to the next item in the stack and removing the previous head. These are easy actions to do in SPARQL.

With an indexed list, on the other hand, iteration over the stack is a bit slower (you have to calculate the next index in the stack first, then search for the member which has the corresponding index), but random access is generally faster, and you can get a guaranteed ordered list from SPARQL queries, which you can’t do with transitive closure. What also happens here (which I believe is an important distinction) is that each member is directly linked to the list object, rather than indirectly linked as is the case with an unindexed list. This is the ownership principal at work — in a linked list, any given node in the backbone is owned by a previous node, rather than a specific list object. Transitive closure is a useful property — walking the chain — is a useful property, but it makes modeling difficult.

The broader takeaway from this is that modeling frequently relies upon identifying (and utilizing) those helper structures that are semantically neutral but that hold semantically sensitive data constructs in place. Narrative structures imply explicit ordering and indexed lists. Saving state usually involves a stack, or linked list. One to many relationships should always be inbound connections. If a resource “points” to more than one object of the same type using the same property, then you’re probably doing something wrong. These rules may seem simple, but they can usually help to disentangle complex data models in the most efficient manner possible.

By understanding the importance of cardinality and ordering in data modeling (and that these are usually not as arbitrary as they may seem on the surface) you can make your data models work better regardless of how they are stored. This in turn improves serialization, and, with intelligent query design, can also simplify creating data services. They are all related.

Kurt Cagle is the founder of Semantical LLC and is editor of The Cagle Report. He writes regularly on semantics, data modeling and data science issues. #TheDataReport

How Structured Is Your Data? was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.

Frederick Taylor has a lot to answer for. The father of “efficiency management”, Taylor made a name for himself in the early twentieth century by consulting with companies, offering his services to improve a new metric that he had devised called “productivity”. He would go onto factory floors with stopwatches and clipboards and would record exactly how long it took a given laborer to accomplish a given task. He would then take the best of these measurements and declare that this was where his factory workers should be producing at, proceeding to remove all “unnecessary” breaks or downtime for those workers.

The factory owners loved him, because it gave them the leverage to demand higher efficiency from their workers based upon the “scientific” methods of Taylor. It also reinforced the notion that workers, when not otherwise kept in line with the whip, would become lazy and attempt to steal all the hard-earned money that these factory owners made by “wasting time”.

Ironically, while the factories did improve productivity in the short term, the amount of accidents, turnover and anger fueled labor riots through the 1920s and 30s. By then, Taylor had turned is his efficiency management practice into one of the worlds first successful management consultancies, and it was only in hindsight that the damage that he wrought really became obvious.

Unions emerged that enabled collective bargaining, that negotiated not only for better wages but saner working conditions. They also made it harder to fire employees without cause and with no compensation. However, what made unions vulnerable was that they could only do so by enforcing strikes and work actions, and this in turn meant that union dues became a point of attraction for both organized crime and company owners who found themselves paying more for labor.

Since the 1930s, unions have been under continuous attack, and the percentage of Americans that belong to a union have dropped from north of 50% in the 1950s to about 6% in the intervening decades, to the extent that unions have ceased becoming a major factor in the economic (and hence political) landscape.

Automation has played a part in this extinction, but it is worth understanding that for decades now the ideal form of labor for most businesses is one in which employers pay only for the most productive part of an employee’s time. In this view, workers would have only one employer, could be called in at any time to work for however long was necessary, and could be paid as minimal a wage as possible. This Just-In-Time (JIT) workforce vision is frequently called “Work For Hire”.

Yet there are fundamental problems with work for hire, problems that are increasing as this mode of work — the most prevalent form of employment in the gig economy — becomes routine. The first is exclusivity. Anyone who has consulted for a living knows this problem. Work for hire often places limits on being able to work for more than one employer at a time, and even where this isn’t the case, most employers expect that they can call on a contractor at any time without regard to availability. This makes scheduling multiple jobs (a complex task at the best of times) almost impossible. This blocking problem generally means that working a second or third job becomes harder, so even if there is no specific legal bar on having multiple contracts, in practice few people can pull it off without some kind of support system.

Another primary difference between a gig job and consulting is that in the latter there is some ability to control both when and where the work gets done. A writer, for instance, has more flexibility in their schedule because this kind of work can be done remotely, but most “freelancers” are bound to both a workplace and a work schedule. Because that work schedule is all too often set arbitrarily with little time to react, such work creates dead zones in people’s schedules that are either utilized by transportation to and from different work or are effectively unproductive for the employee.

A similar mechanism where the gig economy represents a transfer of power from worker to employer is the shift towards the expectation that equipment owned by the gig employee be used to perform the services involved. The gig economy is in essence the rise of brokered dispatch services, where the company places the onus of ownership of cars, houses, cleaning supplies and tools, uniforms, computers and so forth on the employee, while charging a significant overage for getting the job in the first place. As the key benefit that the dispatchers have is in providing lower cost services, this often translates into net losses for the employees, but because the dispatchers control the market, the ability of the employee to recoup their own investment into “their” business is minimal.

The primary benefit of full time work is its reliability — you know how much money you will make each week, know how much you can budget, and as a consequence have much easier access. People who have a full time job can maintain a good credit rating. Once you are in the gig economy, your credit rating will drop dramatically, because you no longer have the reliability of payment. Even if you make high five or low six figure incomes, it will come in fits and starts, and you will have periods where you have no money coming in for months on end.

This volatile cash flow model has become the standard mode of working for most people, and because you don’t have the same access to the credit markets as you would when working “full time” to smooth out those variations, you have less real buying power because you are paying a penalty for that variability of income.

Health insurance has also become a stealth weapon against gig workers. Agencies may provide a health insurance plan, but it is typically minimal, and it is contingent upon working a set number of hours. One reason there is such a massive pushback on universal health insurance by business is because controlled, affordable health care increases worker mobility, and that threatens the JIT model of a captive workforce. When workers are interchangeable and the investment in them by business is low, then providing health care at all is “bad business” because the workers are easily replaced, but as the investment in individuals increase (because of the need for specialized, high demand skills), so too does the need to lock in employees so they find it harder to leave.

Automation does play a factor in the gig economy as well, but it should be noted that, despite the rising hysteria of robots replacing workers the reality is that the impact of automation has largely been the furtherance of a just-in-time labor philosophy. The reality has been that most of the major disruptions brought about by the computer/networking revolution have already taken place, and there is now rising push-back by consumers on attempts to further that agenda.

Cashiers should have been replaced by automated checkout stands by now, yet the reality is that most self-cashout registers are underutilized — people prefer going to a person to make sure their accounts are settled. Automating wait staff is a non-starter for most restaurants, even where the technology exists to do so. It is my prediction that self-driving taxis will actually take decades to replace taxi-drivers, because people want a human being in control in unfamiliar situations (a taxi driver is also tour guide and human conduit if things go wrong). Similarly, truck drivers will not be replaced by autonomous driving systems, because drivers also handle security, fueling, emergency repairs and other operations.

I see the gig economy overall going through a stage shortly where those who both provide the labor and those who consume the products of that labor begin to fight back. Ironically, I think that one of the main effects that the current anti-immigrant pushback is going to do is dry up the supply of those people willing to accept the minimal conditions that the worst offenders of the gig economy are taking advantage of.

It will be harder to find farm workers, hotel cleaning staffs and fast food restaurant workers — areas that traditionally have been claimed by immigrants because Americans generally didn’t want to take them, and this will push wages upward. Already Uber and others like them are souring the appetite for “entrepreneurial” gigs as the reality of poor wages, punitive working conditions and so forth become obvious, and this in turn is leading towards a repricing of the value of the dispatchers in such a scheme, as this often tends to be one of the easier parts of a distributed services business to recreate.

The fact that the Republican Congress is fighting such a massive backlash on dismantling the Affordable Care Act points to a similar pushback. A universal health care system, coupled with a single payer model that better regulates overall costs, is becoming politically feasible. It’s noteworthy that most countries that now have a universal system went through exactly the same stage where they had separate privatized care that eventually became too greedy and caused the system to collapse, as premiums become unaffordable to businesses and individuals alike. There are signs that this is happening in the United States now, despite considerable propaganda to the opposite. Once such a system becomes established, one of the main reasons that people have for staying with overbearing and abusive companies falls away.

Similarly, predatory credit lending regulation (currently under attack) makes it harder for lenders to create unreasonable terms of repayment on loans (including the business loans that most smaller businesses need to better control cashflow).

Even in areas where the gig economy makes sense — such as for writers, artists, musicians and programmers — there are signs that pricing power is beginning to come back into play. Both journalism and the book writing sector all but collapsed between 1990 and 2010, as existing publishers sought out free content — but the reality was that value is dependent upon both quantity and quality. Many of those publishers went out of business, while others figured out that their roles had changed and refactored their business accordingly. There is value in good writing, in editorial curation, in copy editing, and there is always value in reliability, and the more successful new writers, artists, etc. are now learning how to price their work accordingly.

In many respects I think that its worth looking at what has happened in the last forty years as being something analogous to the rise of printing in the sixteenth century in Europe. Printing was a huge innovation, one that brought massive social change with ramifications that were felt for centuries. Yet as printing presses spread rapidly through Europe, their introduction caused long-standing institutions to collapse, and forced a rethinking about how we determine value.

The same processes are underway now, though I think we’re closer to the end of the most volatile period, rather than the beginning. The barriers to entry for people to make a living have dropped, which caused a surge of people to enter into various fields and created a temporary glut of labor in many fields, sending down wages. Yet employment has always been a winnowing process, where people discover whether they can be competitive or not in certain fields, and as people find their niche or abandon it in favor of others, it becomes easier to start raising your own rates in response as you provide more value.

I should probably bring up the topic of Universal Basic Income (UBI) here. I expect it to eventually emerge, but it’s likely going to be a contentious topic in the United States for decades before it happens (I see it happening in Canada far sooner). It is highly stimulating to the economy (far more so than massive tax cuts for the wealthy which have historically accelerated economic decline), but culturally it requires us to move past a Calvinist view of the government. Short of an outright split between the Red and Blue states, I just don’t see that happening. I’ll have more to say about that in a separate post.

At the same time, as the percentage of people in the gig economy rise relative to those that are already ensconced in the corporate world, so too does the political clout of those people. This is an incredibly important point. Businesses exist to provide value to its customers … not its shareholders. Too many large businesses today have forgotten that point, have become wedded to the notion that the shareholder has a prior right even ahead of the customer, which increasingly is seen by these behemoths as being inconveniences.

Yet the reduced cost of entry that the last forty years has brought due to the innovations from the web, from cleaner energy, from biomedical advances, from a profound advance in our understandings of materials engineering, all of these are making it easier to topple the existing institutions, but they are also forcing us to redefine how we account for (and compensate for) value in our economy. I see what’s happening in the United States today as being the last hurrah of a dying economic regime. The elections of 2018 should prove … interesting.

Kurt Cagle is the producer of The Cagle Report on Linked In, and blogs regularly on Medium. He lives in Issaquah, Washington, and likes rain.

#TheCagleReport

The Gig Economy: License to Exploit was originally published in The Cagle Report on Medium, where people are continuing the conversation by highlighting and responding to this story.