Physical Surveillance is Quickly Becoming Digital, Automated, and Crowd-Sourced
Much attention has been paid to how companies and governments watch private people’s activities online and through the electronic traces of their smartphones, beacons, and laptops. The tech industry has been doing an increasingly effective job collecting, analyzing, and monetizing everyone’s digital detritus from socializing, texting, browsing, entertaining, and, of course, shopping. Some companies have built huge empires based upon people’s digital activities. Today, these same techniques are moving to the physical world.
Less understood is how private people are increasingly being watched by computers as they move about the physical world.
While the conspiracists would say that there is some grand scheme, it is more likely that the early adopters are realizing that the age-old habit of people-watching can be digitized wholesale and analyzed in whole new ways. Life is changing in increasingly apparent ways and we are likely headed towards a completely transparent world.
The Crowd is Watching the Sky
On a completely normal, boring, sunny day I was surprised to see my personal flight — yes, I fly — tracked on FlightAware despite not having had the flight “followed” by the FAA or filed an instrument flight plan. I had started the airplane, spoke with the ground controller, taxied over to the runway, and received permission from the tower to take off. I didn’t talk to, or need to talk to, any other controllers until I reached my destination. I then received permission to land, taxied to the parking space, and put the plane to bed.
A moment later I received a notification from FlightAware that my airplane had landed. Huh? How did they know?
It was a little like the local newspaper welcoming you back from your trip to the supermarket.
Formal flights involving weather, airlines, high-altitudes, large international airports, etc. are always filed with the FAA or other national aviation agency. One needs permission to go through clouds, carry lots of passengers, land at LAX, or fly above 18,000 feet. These are always tracked and reported, sometimes to the annoyance of celebrities and intelligence agencies.

I decided that I would find out what was going on. After all, flying is one of the few times in these days when one feels completely disconnected from smartphones, road traffic, and life’s daily annoyances.

A couple of clicks later shows that FlightAware has crowd-sourced airplane surveillance. They have enlisted more than 12,000 people to deploy transceivers which “talk” to the airplanes flying overhead. Each time a transceiver speaks it sends the registered serial number for the airplane, thereby telling the world who they are. My airplane was tracked by 8–10 people, not the FAA, throughout its journey. As designed, the airplane reported its speed, direction, altitude, serial number, etc. to anyone who asked.

The plane’s transponder, designed to help Air Traffic Control manage flights and to keep airplanes from running into each other, is now being used for private party surveillance and as a medium for advertising.
So much for the privacy of the skies.
But what about the privacy of the commons, even the home? Are we being surveilled by the people around us? It turns out that, yes, very similar tracking activities are happening closer to regular life. The crowd is quietly being enlisted to collect data on people, their things, vehicles, and their actions.
Watching the Streets
A PI from the the old movies would be amazed at the tools available today to watch the streets. No more sleeping in cars waiting for the wayward spouse to appear. The fantastic movie Chinatown would make for a poor story in 2017.
There is a quiet revolution happening; the Internet of Things (IoT) — the cloud, computer vision, edge compute, radios, and sensors — are creating a whole new type of Neighborhood Watch. The flood of digital goodies are enabling companies (and governments?) to enlist private people to watch other people.
Perhaps the most benign and famous crowd-watching system is Google’s Waze. More than 70 million people use Waze to figure out how to get around other people (traffic) using an algorithm based upon people reporting congestion. In Los Angeles this app is used by perhaps 40% of the drivers and is an essential tool for getting around city streets. The City uses the app to deliver safety messages to drivers and to look for illicit street construction, confirming potholes and trash heaps, and so forth.

While Waze users report congestion caused by people, they don’t actually watch individuals (other than perhaps the police, regretfully) and can’t report someone’s activities.
Waze is something of a crowd-sourced crowd-watcher for people who want to avoid crowded streets.
A rather more creepy street-watcher is the combination of a simple video camera and various types of computer vision. These are being sold to property and vehicle owners as an additional security element, but the implications go far beyond that. While few video surveillance cameras are actually watched (some of the most boring TV ever), the addition of computer vision to them makes them into the unblinking eye registering the comings and goings of whomever wanders by.
The simplest example is the personal automatic license plate reader (ALPR or just LPR). Formerly only used by police departments, the combination of a camera with some optical character recognition, perhaps with a little more advanced computer vision and machine learning, is now being made and sold for anyone to go use. For kicks I built one in a day using an Intel Edison, a camera, and the classic OpenCV computer vision library. There is even an open-source devkit out there with a streaming service for the tech crowd. There are numerous commercial products out there for the DIY surveillance crowd.

Today’s version of the repo man is a bit more high-tech; they’re driving around using LPR to find their targets. Apartment owners are able to keep an eye on their residents by putting a camera at the entrance. It is likely that we will see LPR built into the ubiquitous dash-cams and into the cameras on tomorrow’s connected autonomous vehicles (“self-driving cars”).
Unlike government use of LPRs, primarily around crime-fighting, the examples above are just a hint of things to come in the private world. Every request to recognize a license plate is an opportunity to record where and when that car was recognized. Looking to see when your employees leave work? Put an LPR at the entrance of the parking garage. Want to know when your neighbors are home? Put a camera up. Want to case a house? Put a camera up. Want to get some customer intel on your retail competitor? Set up a camera.
So much for the privacy of driving.
The idea that your daily habits driving around will remain opaque to the community through traditional anonymity is looking unlikely.
Copious amounts of data and metadata about the vehicles driving by are being collected with some proposals, but essentially no regulation. Assuming that LPR is considered photography, then in the US it is basically legal to photography anyone and anything anywhere and at any time.
Per http://www.krages.com/ThePhotographersRight.pdf:
The general rule in the United States is that anyone may take photographs of whatever they want when they are in a public place or places where they have permission to take photographs. Absent a specific legal prohibition such as a statute or ordinance, you are legally entitled to take photographs. Examples of places that are traditionally considered public are streets, sidewalks, and public parks. Property owners may legally prohibit photography on their premises but have no right to prohibit others from photographing their property from other locations. Whether you need permission from property owners to take photographs while on their premises depends on the circumstances. In most places, you may reasonably assume that taking photographs is allowed and that you do not need explicit permission. However, this is a judgment call and you should request permission when the circumstances suggest that the owner is likely to object. In any case, when a property owner tells you not to take photographs while on the premises, you are legally obligated to honor the request.
The ramifications are clear: it is a matter of minutes before someone comes up with a business model to incentivize people to put LPR on their properties in much the same way that FlightAware has done so with aviation transponders.
The Crowd is Watching You
If reading transponder codes and license plates identifies airplanes and cars then recognizing faces identifies you. There are many examples of governments doing facial recognition for various reasons, especially with border control. The US government is even doing it at various airports on people leaving the country.
The private world isn’t far behind. Some airlines are trying out facial recognition (“stand in front of the camera”) as an alternative to boarding passes, though, as a frequent traveler, it is a bit of a mystery as to why this would be better for the travelers themselves. One still has to look up their seat, etc. On the other hand, the benefits to the airlines are obvious. They can use computers to let them literally recognize their passengers on sight. Putting up a few more cameras here and there will allow them to track their customers as they go through airports and beyond. The use of facial recognition further ensures that passengers can be tracked, controlled, and “monetized.”

Being able to provide personal service is considered the holy grail of retailing, after all. It feels better when your barista, barber, stylist, or hairdresser recognizes you and gives you what you’re looking for with a minimum of fuss. The airlines, department stores, and banks believe that facial recognition will do this on a mass scale.
But, wait, you say! Isn’t this totally creepy?
Yes it is today, but the norms of behavior are changing. We are increasingly allowing companies and governments to monitor us in an increasingly wider range of activities. We have quickly moved from shock at Scott McNealy’s famous statement in 1999 that “privacy was dead!” to discussing the rate that privacy is disappearing.
It is likely only a matter of time before we see widespread facial recognition. Some other technologies such as augmented reality virtually require facial recognition for their evolution. TVs, monitors, and videogame machines have been flirting with using facial recognition for a long time to recognize the user(s) and deliver their personal games, shows, and apps.
Tying Physical Tracking to Digital Tracking
The ultimate goal of all of this stuff is, of course, to build the complete picture of the person’s activities with an eye towards understanding how to influence them, perhaps control them, and to predict their future actions. Whether for public safety (“is this person a security consideration?”) or for retail (“can I persuade this person to buy my product?”), the combination of physical and digital activity data is undeniably valuable. To put it another way, companies and others want to know who’s in their store, airport, stadium, etc. and what they’re doing.
Many sensors exist today to recognize presence. Sensors are being built into fixtures installed in the buildings designed to understand the presence, movement, and activities of the people in view. When coupled with the sensors in an individual’s smartphone, the amount of understanding dramatically increases. For example, motion and course motion classifiers are used to understand when the user is walking, driving, sitting, etc.
The question then becomes one of how to tie the information collected by environmental sensors (computer vision, transponders, etc.) with the information collected by the individual’s smartphone sensors.
We are on the cusp of this happening. Today there are numerous apps running on smartphones from companies with physical locations. Think of the airline apps, the shopping apps, the mobility apps, the fitness apps, entertainment apps, and, of course, the social apps. A certain TNC app, for example, says that they need to access your location even when you’re not hailing or riding in a car. Do they really need to know where you are otherwise? Similarly, the airlines are asking for persistent location. They will eventually be able to confirm the identity of each smartphone user through facial and location information.

Even outside of having an app on the phone there are ways to tie together smartphones and their humans. For those watching the cars go by using LPR it is a small step to also capture the signatures (MAC addresses) of the Wi-Fi and Bluetooth radios in the cars. As an outsider looking into people’s homes and businesses, Google collected MAC addresses of the Wi-Fi radios that their Street View cars went by. For property owners and managers it is trivial to collect smartphones’ unique networking MAC address through Bluetooth and Wi-Fi. Many Wi-Fi providers such as public ones, airlines, retailers, sporting venues, and so forth routinely collect the serialized MAC address from every passing smartphone.
There are an increasing number of services emerging to recognize users within geographies and what they’re doing.
Can We Hide From the Crowd?
It has been commonly said that there are almost no tools by which individuals can control the surveillance and further distribution of their activity data. While some laws exist to keep cellular-derived activities and health-related information from being distributed without prior permission, the actuality is that today’s laws are almost no measure of protection for individuals. Other laws against wire-tapping are in controversy right now as to whether they apply to collecting MAC addresses, etc. The Wiretap Act, says it doesn’t apply “to [the] intercept or access an electronic communication made through an electronic communication system that is configured so that such electronic communication is readily accessible to the general public." To be sure, though, I’m not a lawyer.
Leaving aside whether your digital footprints can be obscured, the various types of protection from automated physical surveillance such as wearing masks, removing license plates, or turning off the transponder are either impractical or illegal.
Put another way, would wearing a mask in public add to your anonymity or decrease it?
The future is coming. It will be interesting to live in a world that is nearly completely transparent.
