Data Privacy: Securing JSON

Although my full-time job these days is not data privacy, I am on my way to the annual Ponemon Fellows and Responsible Information Management conference next month and got to thinking about the state of data privacy and JSON document stores like MongoDB. Other than the JSON Web Encryption standard (JWE), remarkably little information of a generic nature is available, and most relates to specific data stores. For JWE, there are few end-to-end tutorials, it says nothing about how to pinpoint what to protect,and it can be rather complex. With respect to data stores, e.g. Redis, many do not have much in the way of security at all (albeit by design). Others, implemented as SAAS services, simply take an all or nothing approach to encrypting data. This article demonstrates via code a simple, flexible security approach that could be used with almost any JSON document store in the browser or on a server.

Future articles will cover creating a full-text searchable but encrypted index, sharding data for compliance and security, approaches to authorization, and records retention/deletion.

Case Description

Let’s assume that Joe is creating JavaScript objects on a client machine and wishes to transfer them to a server or another client machine while protecting different aspects on the objects in different ways and also allowing them to be indexed and searched. Below is a part of one of Joe’s objects:

const joe = {
messages: [
{
security:"confidential",
summary:"confidential executive summary",
message:"joe's confidential info"
},
{
security:"secret",
summary:"secret executive summary",
message:"joe's secret info"
},
{
security:"top secret executive summary",
summary:"executive summary",
message:"joe's top secret info"
}
],
privateKey: "joe's private key",
name: "joe",
ssn: "555-55-5555"
};

Joe would like to treat the data as follows:

  • A SSN should be masked to show just the last 4 and not recoverable on another machine.
  • Confidential info can be searched based on its summary, but the full text should be encrypted.
  • Secret info should have its summary and full text encrypted.
  • Top Secret info should never leave the originating machine.
  • Eyes Only data, e.g. a private key, can leave a machine if encrypted. And, since Joe does not want anyone to know what kind of info it is, the key name also needs to be encrypted.

So, Joe needs to write code to walk through his object and mask, encrypt, or delete data. Not too hard, except Joe knows that over time the nature of the object will change and if he writes highly specific code it will rapidly become unmanageable. There are a number of libraries for walking document graphs. MongoDB has a great one, but you have to be using Mongo. GraphQL is great for extraction, but not ideal for transformation. And, I have actually written a couple of libraries like this myself, one is called assentials.

Assentials provides a router that can be used for object transformation. A router you ask? Yes, a router. Routers typically take an input and modify it or produce side effects based on matching conditions and chainable actions. Some routers are highly specialized and support only URLs, others like the one with ExpressJS could really handle just about any object, although by convention they are used for HTTP requests and responses. The assentials router does not care what kind of data you route. Think of it like physical package routing through many handlers who may look at or modify the contents of the package, but at the end of the day must deliver it to the next recipient in the chain before it ultimately gets delivered to the requestor, unless a valid substitution is provided (So much for secrecy! But, Joe can use this to his advantage before releasing his objects to the world.)

The assentials router is particularly powerful in that it supports literal, functional, regular expression, and object matching via destructive assignment or nested literal comparisons. As a result, it is possible to hand an object to an assentials router and use the routes to match on parts of the object and then make modifications to the object. This means each security constraint can be its own separate case. Easy to read, easy to maintain. This being said, there are sure to be other options out there to which the general approach below could be applied.

Securing The JavaScript Object

So, back to Joe, here is what he could write. Note the effective use of destructuring and partial object literals to get at just the object properties desired:

const router = assentials.router,
route = assentials.route;
const secure = async (object,parentKey,parentObject) => {

const transform = router(

// mask SSN
route(/^\d{3}-?\d{2}-?\d{4}$/,
(value) => {
parentObject[parentKey] = "***-**-" + value.substring(7);
}),
    // insert more literal rules here
    // return anything that is not an object
// assentials uses the Iterator paradigm to exit routing
route((value) => !value || typeof(value)!=="object",
(value) => {
return {value,done:true};
}),
    // walk down objects
route((value) => value && typeof(value)==="object",
async (object) => {
// not using forEach, async needed
for(const key in object) {
// pass parentKey and parentObject on recursion
await secure(object[key],key,object);
}
}),
    // encrypt confidential message
route({security:"confidential"},
async (object) => {
object.message = await encrypt(object.message);
}),
    // encrypt secret message and summary
route({security:"secret"},
async (object) => {
object.summary = await encrypt(object.summary);
object.message = await encrypt(object.message);
}),
    // delete top secret message
route({security:"top secret"},
// Note, parentKey and parentObject were passed as
// last arguments key and object to recursion above
(object) => {
delete parentObject[parentKey];
}),
    // encrypt key name and value for eyes only private key
route(({privateKey}) => privateKey!==undefined,
async (object) => {
object[await encrypt("privateKey")] =
await encrypt(object.privateKey);
delete object.privateKey;
})
    // insert more object property rules here
)

await transform(object);
  return object;
}

That’s it, except we don’t have the encrypt function. Joe can just dummy it up for now so that visual inspection for debugging is easier. At the end of the article, real encryption will be introduced.

const encrypt = async (value) {
// reverse and mark the string returned so we know
// it is something encrypted, or we won't
// know what to decrypt!
return `!e!${value.split("").reverse().join("")}!e!`;
}

In case you are wondering about all the async code above, it is there for two reasons:

  1. assentials functions are designed to be asynchronous.
  2. Many encryption libraries are asynchronous and our dummy function needs to be ready for transformation into the real thing.

Calling and awaitingsecure with the document above will return:

{
"messages": [
{
"security": "confidential",
"summary": "confidential executive summary",
"message": "!e!ofni laitnedifnoc s'eoj!e!"
},
{
"security": "secret",
"summary": "!e!yrammus evitucexe terces!e!",
"message": "!e!ofni terces s'eoj!e!"
},
null
],
"name": "joe",
"ssn": "***-**-5555",
"!e!yeKetavirp!e!": "!e!yek etavirp s'eoj!e!"
}

If you can read backward, Joe has not fooled you, but he can debug the code. Of course, now Joe has to write code so the data can be unencrypted at some point!

De-encrypting The JavaScript Object

De-encryption is somewhat simpler.

const decrypt = async (value) => {
// if we are dealing with an encrypted string
if(typeof(value)==="string"
&& value.startsWith("!e!")
&& value.endsWith("!e!")) {
// reverse the encryption process
return value
.substring(3)
.split("")
.reverse()
.join("")
.substring(3);
}
  // otherwise just return value
return value;
}
const unsecure = async (object,parentKey,parentObject) => {
const transform = router(
// treat all values like they are encrypted
// if they are not, no change will be made
route((value) => !value || typeof(value)!=="object",
async (value) => {
parentObject[parentKey] = await decrypt(value);
}),
    route((value) => value && typeof(value)==="object",
async (object) => {
// treat all keys like they are encrypted
// if they are not, no change will be made
for(const key in object) {
const value = object[key],
decrypted = await decrypt(key);
// if the key changed it was encrypted
// so, update object
if(key!==decrypted) {
object[decrypted] = value;
delete object[key];
}
}
}),
    // walk down objects
route((value) => value && typeof(value)==="object",
async (object) => {
for(const key in object) {
object[key] =
await unsecure(await decrypt(object[key]),key,object);
}
})
);
  await transform(object);
  return object;
}

Calling and awaiting unsecure with the encrypted object above will return:

{
"messages": [
{
"security": "confidential",
"summary": "confidential executive summary",
"message": "joe's confidential info"
},
{
"security": "secret",
"summary": "secret executive summary",
"message": "joe's secret info"
},
null
],
"name": "joe",
"ssn": "***-**-5555",
"privateKey": "joe's private key"
}

Real Encryption

With a code base that is known to work, Joe is ready to add real encryption. The cryptozoa library makes this easy.

const encrypt = async (value) {
const {data} = await
cryptozoa.symmetric.encrypt(value,"mypassword");
return `!e!${data}!e!`;
}
const decrypt = async (value) => {
if(typeof(value)==="string"
&& value.startsWith("!e!")
&& value.endsWith("!e!")) {
const {data} = cryptozoa.symmetric.decrypt(
value.substring(3).substring(0,-3),
"mypassword");
return data;
}

Obviously, Joe should not include his symmetric encryption keys in production source code and should only transfer and store them encrypted themselves with asymmetric encryption. But, that is a topic for another time.

Below are the encrypted and unencrypted versions of Joe’s data:

{
"messages": [
{
"security": "confidential",
"summary": "confidential executive summary",
"message": "!e!/qbBRmBi/V69SZaQqvLzjzvFQfUVq1kkyhXbPelVOdk=!e!"
},
{
"security": "secret",
"summary": "!e!leCB4/9PGQs7fiUIs+TwK/Qgddzop/846Ejiyc4nvdA=!e!",
"message": "!e!SwmsCQoyKQ593clsR3XlMmxahX8glq1gzqnX9lzHbT4=!e!"
},
null
],
"name": "joe",
"ssn": "***-**-5555",
"!e!u8sO9HKsYlpEtq4W1PdTfw==!e!": "!e!1l5srmvJMF2629Iu0MUwVnhNcHZoM/12UJMQSaPJXDo=!e!"
}
{
"messages": [
{
"security": "confidential",
"summary": "confidential executive summary"
},
{
"security": "secret"
},
null
],
"name": "joe",
"ssn": "***-**-5555"
}

Joe is done for now, but he should consider how to apply JWE to his needs since it is part of an an industry standard with additional capabilities around signing and the sharing of keys. The easiest to grasp documentation is available on BitBucket. Perhaps adding JWE processing on a node by node basis in the routes or using it to enhance the encrypt and decrypt functions would work.

Clap if you found working through this process with Joe useful. Be safe out there!