JSON Structure and Schema

I am sure you are familiar with JSON Structure — JSON stands for Javascript Object Notation. Some people say JSON notation but even though I don’t think it is wrong, the double notation doesn’t look good -. You see it everywhere. I am not so sure you are familiar with JSON Schema. This post meant to be read by everyone, not just beginners. I write it to people who are ready to feel like beginners.

Why do we need to learn JSON?

Maybe 15 years ago, when the ASP programmers were ubiquitous, XML (eXtensible Markup Language) responses were popular. XML is still popular of course, but I see JSON a lot more probably because I examine Javascript and Python systems.

Because of this familiarity, like the words you learned from people but not from the dictionary, you may think you know JSON, or maybe you are a beginner and you know that you don’t know JSON.

I realized I don’t know JSON in a programming assessment for a job application. The question was “What is the proper content type for JSON from the below answers?”. I was very confident and looking for application/json but couldn't see it. Instead, I saw application/vnd.api+json(won't be covered in this post but next) and I told myself "Obviously that is the answer but what(the f) is that?" and saw there is a whole new world of JSON out there.

JSON Structure

Let’s start at the very beginning. As the name implies, JSON is object notation in Javascript but actually, there are slight differences. The root of the JSON Response or Request is wrapped either in curly brackets or square brackets. Curly brackets

{
  // Empty JSON Object
}

indicate it is an object or square brackets

[{
  // An array of empty object
}]

indicate it is in an array of objects. An example to a JSON Object:

{
  "id": 1,
  "first_name": "Neal",
  "last_name": "Sonley",
  "email": "nsonley0@businesswire.com",
  "gender": "Male",
  "ip_address": "102.251.232.235"
}

And an example of a JSON Array of Objects:

[{
  "id": 1,
  "first_name": "Darryl",
  "last_name": "Gerling",
  "email": "dgerling0@shinystat.com",
  "gender": "Female",
  "ip_address": "67.227.241.208"
}, {
  "id": 2,
  "first_name": "Vivi",
  "last_name": "Dosdill",
  "email": "vdosdill1@google.com",
  "gender": "Female",
  "ip_address": "237.238.137.105"
}]

I hope you didn’t understand anything. If you do, please jump to “JSON Schema” section and don’t kill your motivation by reading the same things you know, I need your motivation.

JSON Structure in-depth

Showing different possibilities opens up one’s horizon. In this section, I will show you different possibilities of JSON Structures and define each property type with their possible values.

What Properties in JSON Structure Consist of

JSON structure consists of key and value pairs. And a single item of this key and value pair is called the “property” of its root object(You saw an example of an object above).

{
  "key": "value",
  "anotherKey": "anotherValue"
}

Different from Javascript objects, JSON properties’ keys have to be wrapped in double-quotes.

If a value a is a String, it has to be wrapped in double-quotes.

What is a String in JSON Structure?

As you know by definition a string is an object which consists of little parts positioned in tandem. In Javascript, every text is a decomposable array of letters, that’s why it is called a String.

The key and the value of a string property must be double-quoted. If it is not, you will get an error.

{
  "aStringKey": "Some str1ng valu3"
}

There is a comma after the first property in the previous JSON Object example with two properties. If you have properties more than one, you separate them with commas.

{
  "firstPropertyKey": "firstPropertyValue",
  "secondPropertyKey": "secondPropertyValue"
}

I think capital letters at the beginning of the words took your attention. This way of writing, starting every word with a capital letter except the first word, is called camelCase.

What is a Number?

In different languages, there are usually integer data structures. There also some integer data structures in Javascript too but the most generic usage which you will use in your JSON Document will have a Number data structure.

A Number’s key is double-quoted but not its value.

{
  "aString": "Some text",
  "aNumber": 10
}

A number can have a floating-point number.

{
  "aFloatingPointNumber": 3.1415,
  "anExponentional": 8.15e6
}

But an integer can’t(There is no distinct integer type other than a number in JSON Structure, there is in JSON Schema). Also, numbers in double-quotes are interpreted as strings, not numbers.

What is an Object?

An Object in JSON Structure, as we mentioned before, a map of key and value pairs. Every other property type can exist as a property in an Object.

Nested Objects

Nested objects are objects as values of a key in an upper-level object. Here is one:

{
  "id": 456,
  "name": "Joe",
  "additionalInformation": {
    "playsGuitar": true,
    "playsPiano": false
  }
}

additionalInformation property in this example is a nested object.

But what are those true and false values?

What is a Boolean?

A boolean is a binary data structure. It’s either true or false. You can use it for your binary expressions.

{
  "role": "writer",
  "isHandsome": true
}

What is an Array?

An Array is a list, ordered in your preference. There are native Array methods in Javascript to work on this list-like data structure more easily.

{
  "name": "Billy",
  "pets": ["dog", "cat", "alligator"]
}

An Array can contain different types of data structures:

{
  "aMixedArray": [12, "Hello", 5.16, "/^world$/", null, {"name": "Alex"}]
}

What is null?

Null is simply the absence of a value.

We Learned JSON

Now go back to the first populated JSON object and array examples and enjoy understanding them. I created examples with Mockaroo, you can create any amount of any JSON Structure with mock data you want with it. The free plan is limited to 1000 records and there are also many other types like XML and CSV if you are C# or a Python developer.

I want to show you an example of JSON usage in a CRUD(Create Read/Retrieve Update Delete) application. I think going so theoretical won’t burn this topic into your memory.

Our Node.js server application will get some data in JSON format from the client/browser and respond with processed data. It will be a letterbox. It will receive, keep and show letters sent to it.

I will use the curl application for making requests from the Linux Subsystem Shell in Windows. Our JSON request to the server will look like this:

{
  "username": "some_guy144",
  "message": "I am a letter from some guy"
}

Let’s create our express.js server to accept requests. First, install the required packages.

npm install express body-parser --save

Then create server.js

const express = require("express");
const http = require("http");
const app = express();
let letters = [];
const bodyParser = require("body-parser");
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.post("/", function(req, res, next) {
  const newRecord = {
    ...req.body,
    id: getLastId(letters) + 1,
    createdAt: new Date().toDateString()
  };
  letters.push(newRecord);
  res.end(JSON.stringify(newRecord));
  console.log("--------------------------------")
  console.log("-------------LETTERS------------")
  console.log("--------------------------------")
  console.log(letters);
});
http.createServer(app).listen(3003);
function getLastId(anArray) {
  return anArray.length ? anArray[anArray.length - 1].id : 0;
}

Run this file with node and it will listen for new requests until you terminate the console application. It listens on localhost port 3003, we will send two different JSON requests there with curl.

We should save these JSON objects to files for convenient console usage.

letter1.json

{
  "username": "John",
  "message": "I am a benign letter"
}

letter2.json

{
  "username": "Mia",
  "message": "I am another benign letter"
}

You can send letter1 with the command below in your terminal:

curl --header "Content-Type: application/json" --request POST --data @letter1.json http://localhost:3003

If everything goes successful, you should receive a response like this:

{
  id: 1,
  username: 'John',
  message: 'I am a benign letter',
  createdAt: 'Tue Jul 16 2019'
}

Send the other letter with editing the curl command we used. You should see something like this in your console:

--------------------------------
-------------LETTERS------------
--------------------------------
[
  {
    id: 1,
    username: 'John',
    message: 'I am a benign letter',
    createdAt: 'Tue Jul 16 2019'
  },
  {
    id: 2,
    username: 'Mia',
    message: 'I am another benign letter',
    createdAt: 'Tue Jul 16 2019'
  }
]

But will everyone be good-willed like you? What if we put a number to username property in our JSON request? JSON requests and responses must be validated, not just for security but also for having a standard. Let’s learn about how to annotate structures for our JSON documents.

JSON Schema

JSON Schema, by its own words, is a media type for validating and annotating JSON documents. We saw the issue of validating in our Javascript CRUD example and will see in other projects in the future too, validation is something you should always do. Annotation lays down the template for documents, providing a good understanding of them. A good understanding of templates of the documents is required for obeying to a standard. Obeying to a standard is important for consistency in code and preventing conflicts in usage in the future. Also, our code has to be self-documented so annotation plays a part in it too.

We will explore the JSON Schema in this section. I won’t cover hyper-media now. We will cover it and other API specifications in another article. I want to give you a glance at how to construct schemas. Otherwise, JSON Schema has great and easy documentation so it would be copying it to the blog post.

Remember this link for learning further about JSON Schema. I also recommend core specification and validation specification for detailed knowledge. I will give you an example JSON document for making its schema together. Here is the JSON document:

{
  "first_name": "Lauryn",
  "last_name": "McGillacoell",
  "email": "lmcgillacoell0@w3.org",
  "gender": "Female",
  "ip_address": "18.254.152.131",
  "pets": 
  [
    {
      "species": "sheep",
      "leg_count": 4
    },
    {
      "species": "monkey",
      "leg_count": 2
    }
  ],
  "car": {
    "make": "Dodge",
    "model": "Challenger",
    "year": 2012
  }
}

First, identify what data structures we have. We have integers at pet[].leg_count and car.year properties. first_name, last_name, car.make and car.model are ordinary string properties(make and model options are much so we will approach them as ordinary strings). email is a string property with a pattern, as well as ip_address. gender is a string with a limited number of value options. pets is an array of objects with a template consists of species and leg_count properties.

Our first empty schema should look like this:

{
  "$id": "https://mehmetegemen.com/imaginary-json-schemas/person.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object"
}

We set an id for later usage of schema, indicated that this JSON is a schema by setting $schema, put a descriptive title and set the type. There are several values type property can take, we will discover them in our steps. Let's fill our JSON schema.

What do we do for ordinary strings?

We have ordinary strings — by ordinary I mean strings don’t have patterns — in root object and car nested object. we need to add these lines to the root object:

"properties": {
  "first_name": {
    "type": "string",
    "description": "First name of the person",
    "minLength": 2,
    "maxLength": 128
  },
  "last_name": {
    "type": "string",
    "description": "Last name of the person",
    "minLength": 2,
    "maxLength": 128
  }
}

properties object is a property of the root object. So the whole document is something like

{
  "$id": "https://mehmetegemen.com/imaginary-json-schemas/person.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "first_name": {
      "type": "string",
      "description": "First name of the person"
    },
    "last_name": {
      "type": "string",
      "description": "Last name of the person"
    }
  }
}

The document is very self-explanatory. Properties’ types are string, schema looks for minimum of 2 characters, maximum of 128 characters in those properties.

Patterns in Strings

Strings can be checked whether they match a pattern with regular expressions. An example:

"properties": {
  ...,
  "email": {
    "type": "string",
    "pattern": "^[^@]+@[^\\.]+\\..+$"
  },
  ...
}

With pattern property, validators use our schema will check the email property in the request that it is an email or not. Of course, this pattern is a little bit wrong because of its extreme simplicity. If you want a complete solution you can check RFC 822 and email regex. There is also a possibility of doing this according to RFC 5322, section 3.4.1:

"properties": {
  ...,
  "email": {
    "type": "string",
    "format": "email"
  },
  ...
}

Adding other properties with patterns:

"gender": {
  "type": "string",
  "pattern": "^(Female|Male|Lgbti)$"
},
"ip_address": {
  "type": "string",
  "pattern": "^(?=.*[^\\.]$)((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.?){4}$"
}

gender’s pattern is a way of pre-defining values. Without patterns:

"gender": {
  "type": "string",
  "enum": ["Female", "Male", "Lgbti"] 
},
"ip_address": {
  "type": "string",
  "format": "ipv4"
}

There are pre-defined formats you can use, check them from validation specification.

Nested Object

It is very easy, the declaration is almost the same as the root object.

"properties": {
  ...,
  "car": {
    "type": "object",
    "properties": {
      "make": {
        "type": "string"
      },
      "model": {
        "type": "string"
      },
      "year": {
        "type": "integer",
        "minimum": 1900
      }
    },
    ...
  }
}

Defining an integer is as simple as you see.

Definitions

pet array is an array of objects with a template. We can define the items of the array and import them from the definition from somewhere else.

"properties": {
  ...,
  "pets": {
    "type": "array",
    "items": { "$ref": "#/definitions/pet" }
  }
},
"required": ["email"],
"definitions": {
  "pet": {
    "type": "object",
    "properties": {
      "species": {
        "type": "string"
      },
      "leg_count": {
        "type": "integer"
        "minimum": 2
      }
    }
  }
}

Email is required because it is the only unique property, it is not a must, just an arbitrary choice to show you that functionality. We defined the pets array's items in the same document but we could also migrate definition to a different file and import it from there.

End Result

Our final schema looks like this:

{
  "$id": "https://mehmetegemen.com/imaginary-json-schemas/person.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "first_name": {
      "type": "string",
      "description": "First name of the person",
      "minLength": 2,
      "maxLength": 128
    },
    "last_name": {
      "type": "string",
      "description": "Last name of the person",
      "minLength": 2,
      "maxLength": 128
    },
    "email": {
      "type": "string",
      "format": "email"
    },
    "gender": {
      "type": "string",
      "enum": ["Female", "Male", "Lgbti"]
    },
    "ip_address": {
      "type": "string",
      "format": "ipv4"
    },
    "car": {
      "type": "object",
      "properties": {
        "make": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "year": {
          "type": "integer",
          "minimum": 1900
        }
      }
    },
    "pets": {
      "type": "array",
      "items": { "$ref": "#/definitions/pet" }
    }
  },
  "required": ["email"],
  "definitions": {
    "pet": {
      "type": "object",
      "properties": {
        "species": {
          "type": "string"
        },
        "leg_count": {
          "type": "integer",
          "minimum": 2
        }
      }
    }
  }
}

I strongly recommend you to extend your knowledge by reading the provided links at the beginning of the section, I share again, especially validation specification.

Javascript library which uses JSON Schema for validation is Ajv. And it is very easy to use. First, install with

npm install ajv --save

Then run something like this:

const Ajv = require("ajv");
const schema = {
  $id: "https://mehmetegemen.com/imaginary-json-schemas/person.json",
  $schema: "http://json-schema.org/draft-07/schema#",
  title: "Person",
  type: "object",
  ...
};
const exampleRequest = {
  first_name: "Lauryn",
  last_name: "McGillacoell",
  ...
};
const ajv = new Ajv();
console.log(ajv.validate(schema, exampleRequest));

In our case:

const Ajv = require("ajv");
const schema = {
  $id: "https://mehmetegemen.com/imaginary-json-schemas/person.json",
  $schema: "http://json-schema.org/draft-07/schema#",
  title: "Person",
  type: "object",
  properties: {
    first_name: {
      type: "string",
      description: "First name of the person",
      minLength: 2,
      maxLength: 128
    },
    last_name: {
      type: "string",
      description: "Last name of the person",
      minLength: 2,
      maxLength: 128
    },
    email: { type: "string", format: "email" },
    gender: { type: "string", enum: ["Female", "Male", "Lgbti"] },
    ip_address: { type: "string", format: "ipv4" },
    car: {
      type: "object",
      properties: {
        make: { type: "string" },
        model: { type: "string" },
        year: { type: "integer", minimum: 1900 }
      }
    },
    pets: { type: "array", items: { $ref: "#/definitions/pet" } }
  },
  required: ["email"],
  definitions: {
    pet: {
      type: "object",
      properties: {
        species: { type: "string" },
        leg_count: { type: "integer", minimum: 2 }
      }
    }
  }
};
const exampleRequest = {
  first_name: "Lauryn",
  last_name: "McGillacoell",
  email: "lmcgillacoell0@w3.org",
  gender: "Female",
  ip_address: "18.254.152.131",
  pets: [
    { species: "sheep", leg_count: 4 },
    { species: "monkey", leg_count: 2 }
  ],
  car: { make: "Dodge", model: "Challenger", year: 2012 }
};
const ajv = new Ajv();
console.log(ajv.validate(schema, exampleRequest));

I kept it simple. Schema can get complex, I can make another mini-post if you would like to. The next post will be about API specifications like HAL, JSON:API and JSON Schema’s hyper-media. If you want to ask something or just say your good wishes, commenting is a good way. You can share the post with others if you think they will benefit. I wish you a fun day.
— — — — —
Originally from https://mehmetegemen.com/json-structure-and-schema/