At some point in your application’s lifecycle, there might come a time when you need to start scaling your data storage. If you are storing media files or other blobs that have no relations between them, you can easily add storage capacity to solve the problem. For (semi-)structured data in a database however, scaling is a whole different story. Simply adding database instances is not enough. You will need to reconsider the usage patterns and decide what solution solves the problem you have. If your database is hitting resource limits because it is accessed very frequently, adding an asynchronous read replica might be the way to go. If the size of the data is the issue and lookups become very slow, you might consider sharding your database.

In this post, we will look at how Table Store can help us shard our data automatically. Table Store is a NoSQL database service in Alibaba Cloud. It is available in all regions of Alibaba Cloud. Because of its architecture, Table Store can store an unlimited amount of data and still keep read/write throughput at a very high level. It also offers secondary indices, search options and data streaming capabilities. For now, we will focus on basic operations to learn how data is stored.

Sharding in a nutshell

When your data size is becoming too large to efficiently query, you will often end up sharding your data. A shard is a partition of a full table that contains only a subset of the records. If you have two shards for a single table, each shard will contain half of the rows. Every record gets assigned to a shard when it is created. The target shard is determined by a hash of the primary key of the record. When the record needs to be retrieved from the table, the same hash is calculated. You then only have to traverse the target shard where the record lives and can skip all other shards.

A real-life analogy would be a closet of small drawers with patient information in it. When you need to store a patient’s information, you determine which drawer (or shard) the information card goes. Your sharding strategy is likely based on the first letter of the last name. Then you look up the drawer that contains cards with that letter and put it in. When someone requests a patient’s information, you look up the drawer again for his or her last name. Then you flick all cards and find the correct one.

Choosing the right sharding strategy

As you might see now, database sharding is very similar. The only real difference is the sharding strategy. In the real world, you would shard the data by the first letter of the last name. In a database however, you calculate a hash from one of the fields in the data and let the computer decide which shard it should fall in. The field that you use to shard the data is often called the partition key (or sometimes hash key). It is part of the primary key of a record.

But what do you do if your colleague John has the bad habit of putting the cards into a drawer randomly? You might have to flick all of the cards in a certain drawer to find the correct one. It might be smart to ask your colleague to sort the cards in the drawer. That will help you look up the right one even faster. A good choice would be to sort the cards by last name. You can split the stack of cards in half, see if the card you picked is lower or higher in alphabetical order than the card you need, split the remaining stack again and keep going until you find the right one. This is a binary search algorithm and makes it really efficient to look up a single value. This method only works because the cards are sorted, so you will have to make sure John adheres to the strategy.

In a digital database, the values in a single shard are also stored in a certain order. As long as the records are sorted, the computer can perform binary search. You have to decide on the sort order yourself, because the database can not decide what usage patterns you have. To make a decision you need to consider the usage patterns of your application. For key-value access patterns, where you always want to read a single value with a known primary key, you will probably sort by the partition key that was also used for sharding. If you need more complex queries, like range queries or filtering, you should add the fields that support those queries to the primary key. For example, if you need to query records by a date range, it would be most efficient to have the records sorted by that date. A final note is that every record should be uniquely identifiable in the end, which means that the combination of fields for the primary key have to be unique.

Looking at Table Store

Let’s see how Table Store helps us solve the intricacies of database sharding described above. I assume you have created an account and are logged in to the Alibaba Cloud Management Console. Open the ‘Products’ drawer and selected ‘Table Store’. Switch to the region closest to you before proceeding.

Before we can start creating sharded tables, we need to create a Table Store instance. An instance is a container around one or more tables. It is similar to a database in traditional database systems. Click the blue ‘Create Instance’ button in the top-right corner to create an instance. Fill in a name, choose a type (Capacity or High-performance) and fill in a description. The difference between the two types is not important here, but if you want you can read more about it here. Click ‘OK’ to create the instance.

When the instance is created, click on its name to go to the detail page. You will see some access URLs and a list of tables. To start creating your first table, click ‘Create Table’ in the top-right corner.

First you should decide on a name for the table. If you want to store user profiles, Profiles might be a good pick. Leave the advanced settings disabled for now and start building up the primary key. As you can see, the first field in the primary key will become the partition key. You might recall that this is the data point that is hashed to determine which partition the record is put in. If you want to store user profiles, the userId might be a good choice. That will allow us to access the user profiles as a key-value store. Fill in userId in the field and selected String as the data type. Using a UUID for user profiles is often a good idea, so that will be a string. Click ‘OK’ to create the table.

That is it. We’ve created a sharded table in Table Store that can contain an extreme amount of data. We don’t have to worry about scaling compute resources, database instances or read replicas. All of that is handled behind the scenes so we can focus on building our application.

To insert some data, click on the table name after you’ve created the table. Then click ‘Data Editor’ in the left pane. A newly created table is empty of course, so let’s input some data. Click the blue ‘Insert’ button to insert a record. We are asked to fill in the primary key fields, which for us is only userId , so generate a UUID here. Copy the UUID and put it in the userId column. Because Table Store is a NoSQL database, you can store any other fields you want in a record without specifying a schema. Records in a NoSQL database are sometimes also referred to as documents.

For our user profiles, we want to store the user’s first name, last name and age. To add the fields, click the ‘Add Column’ button. Fill in firstName, choose the String type and fill in your first name. Do the same for the lastName and age fields. Make sure the agefield is of type Integer. If everything looks like the image below, click ‘Insert’.

You’ve just created your first record in a Table Store table! After the record is inserted, you will go back to the data editor to see what is in the table. Try to insert another record with the same fields. Or remove one of the records. You can also try adding a record with different fields to see what happens. Then delete that record again. The data editor will always reflect the records as a table, with rows and columns, but it is important to remember that you are working with a NoSQL database. That means that there is no overarching schema for all the records in a table, only the primary key is defined up front.

Managing data programmatically

In your application, it is clearly not feasible to manage all the data by hand with the data editor. Luckily, Table Store has SDKs for several languages so you can access your data in code. Among the supported languages is Python. I will show you how you can create and update records in Python easily using the SDK. I have set up a repository with a simple Flask application that should make it easy to get started.

To get started in your Python project, you need to install the SDK:

pip install tablestore

Then, you can instantiate the OTSClient that allows you to interface with all the Table Store APIs:

You have to import the `OTSClient` from the tablestore package. Then you can instantiate it with an endpoint, access key ID, access key secret and the instance name. The endpoint and instance name can be found in the Table Store dashboard of the instance you created. Make sure to use the internet endpoint if you want to talk to Table Store from your local machine. An access key can be created on the Security Management dashboard. It is used to programmatically access the resources in your Alibaba Cloud account.

In the example above, the Table Store information and credentials are loaded from the environment. You can replace it with the actual values or make sure to set them before running the application.

Now let’s create an endpoint for fetching a profile:

In the /profile/<user_id>route, we define the user_id as a primary key field and call client.get_row() to fetch a row from the Profile table. As we would like to return JSON from our Flask endpoint, we need to interpret the resulting row and transform it in to a Python dictionary. To do this, we add all primary keys and attribute columns (like firstName , lastName and age ) into the dictionary. That dictionary is then transformed into JSON and returned from the API.

The process of storing a record in the table is very similar:

Here we create a POST endpoint with the same URL as the endpoint used to fetch records. A Row object is then created from the primary key and the attribute columns. The primary key is just the user_id , while a list comprehension is used to get all other columns as a list of tuples. That is then set as the attribute columns for the Row . Finally, we call client.put_row() to store the record. Note that this is an idempotent store operation. If a record already exists with the same ID, the current contents are overwritten.

Table Store has many more options like versioning of records and fields, secondary indices that help you look up data with different access patterns and a full-fledged search index to search all the data. If you want to learn more about those, please refer to the documentation.

Wrapping it up

We have seen how the basics of database sharding work, how Table Store enables us to easily shard data and how you can use the Table Store SDKs to access your data programmatically. The code repository contains a working example of the Flask app that you can boot locally as follows.

• Clone the repository
• Create a .env file in the cloned directory with the following contents:

OTS_ENDPOINT=endpoint
OTS_INSTANCE=instance-name
ACCESS_KEY_ID=access-key
ACCESS_KEY_SECRET=access-key-secret

Replace the values with the Table Store endpoint, instance name and access key information relevant to you.

• Use Pipenv to create a virtual environment and install the dependencies:

pipenv install

• Run the application:

FLASK_DEBUG=1 FLASK_APP=src/app.py pipenv run flask run

This uses the Pipenv virtual environment to start up the Flask app. It will expose the /profile/<user_id> GET and POST endpoints to fetch and store data. Make sure your Table Store instance contains a Profile table with the corresponding primary key (a single userId field that is a String ).

Lastly, if you are familiar with Terraform, you can use the Terraform template in the repository to provision a Table Store instance and table. It also specifies the key structure of the table so you should be good to go after the template is deployed.

Happy sharding!

Automatic database sharding with Alibaba Cloud Table Store was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I will introduce you to the concepts of auto-scaling and how it can help you deploy highly available applications on your cloud infrastructure. We will also look at Alibaba Cloud and see how auto-scaling is done there. By means of an experiment with Terraform, used to practise infrastructure as code, we will get acquainted with the moving parts in a typical auto-scaling setup.

What is auto-scaling?

Auto-scaling an application means that the nodes that are used to run that application can adapt to changes in load automatically. You can do horizontal scaling by adding new nodes with stateless copies of the application as needed. In most settings you could also scale vertically if you wanted to. Vertical scaling is the process of upgrading a single node with more CPU, memory or disk space. As vertical scaling is not common and often not the right solution to the problem, we will focus the discussion on scaling horizontally. Adding more identical nodes to your cluster is also referred to as scaling out, while removing nodes is called scaling in. Oftentimes the scaled nodes are grouped behind a load balancer, that will make sure all nodes receive an evenly distributed amount of traffic. As load balancing is a whole new topic in itself, I will not focus on load-balancing an application in this post.

The statelessness of the copies of the application on every node is very important. If a node stores state, for example user sessions in web applications, the state is lost when the node goes down or is removed from the auto-scaling group. This could mean that users are suddenly logged out of the application. The same goes for user-uploaded data or any other data that needs to be shared between the nodes. Normally you would use a database for user and session data or store files in network attached storage that is shared between nodes.

So what options are there for automating the scaling operations of your cluster? Most cloud providers have two choices: scheduled or demand-based. Scheduled scaling is done based on a daily, weekly or monthly schedule that defines when to scale out and when to scale in. This is applicable when you know that most users work with your application at a certain time of day. You could also use it to make sure you have enough capacity right before a new marketing campaign is launched. Demand-based scaling looks at metrics provided by the operating system of running nodes and makes decisions based on that. You could scale out when the average CPU usage is above a certain threshold, and scale in again when the CPU usage is low enough. Often you can also define custom metrics to base the scaling activity on.

Before we dive into setting up auto-scaling on Alibaba Cloud, it is important to understand what benefits auto-scaling might give you. Apart from providing the right user experience at the right time by scaling out when demand increases, scaling in appropriately can save you a lot of money on the monthly cloud bill. Although this kind of cost-optimisation is probably not the thing to start with for your new project, it can become very important as applications evolve. In any case, thinking about how to scale your application upfront can really prevent headaches when the time to cut costs arises.

Alibaba Cloud

As one of the lesser used cloud providers in Europe and the US, the Chinese Alibaba Cloud might not be your first choice for deploying a new project. But despite its relative anonymity, its feature set is pretty complete and has included all tools necessary to do auto-scaling since 2015. These tools are gathered under the name Auto Scaling in the services overview. With data centers in London, Frankfurt, Virginia and Silicon Valley amongst others, there should always be a region close enough to you so you can test the waters.

Implementing auto-scaling with Terraform

To follow along with the implementation of Auto Scaling in Alibaba Cloud, you will need to register an account here. We will be practicing Infrastructure as Code using Terraform, so you should make sure that you have Terraform installed on your machine. Terraform helps us describe the resources we need in the cloud declaratively in configuration files. It can then apply any needed changes for you, so you don’t have to click through the management console yourself to update your resources. This gives you the valuable ability to plan resources, repeat deployments and destroy complete environments with simple command-line calls. Alibaba Cloud has an official open-source Terraform provider that makes it easy to create these repeatable deployments of resources in your Alibaba account. Auto Scaling is also among the supported services.

The full source code is contained in a directory in this repository. I will walk you through the most important parts of the templates and the deployment so you should have a running auto-scaling experiment in just a few minutes.

Setting up the experiment

To start the experiment, we need to clone the repository, move into the relevant directory and initialize Terraform so the plugins are installed:

You should see a message like Terraform has been successfully initialized! that confirms that everything works.

If you look in the directory, you will see a bunch of .tf files and one user data file:

.
├── _alicloud.tf
├── _variables.tf
├── template.tf
├── user-data.conf
└── vpc.tf

0 directories, 5 files

The moving parts involved in auto-scaling are all in the template.tf file. We will focus our discussion on that file.

Scaling group & scaling configuration

The template.tf file contains the following Terraform resources at the top:

A scaling group is the wrapper around all auto-scaling resources. It defines a minimum and maximum number of Elastic Compute Service (ECS) instances that should be in the group. If also defines which instances should be removed first in the removal policy. We also reference two VSwitches where our instances will be placed. A VSwitch is a virtual switch in an Alibaba Cloud VPC that defines a single subnet. We want to be highly available so we created two VSwitches (subnets), each in a separate availability zone. If you point to these in the scaling group then auto-scaling will make sure that instances are balanced across the specified VSwitches. The VPC and VSwitches are defined in vpc.tf , should you be curious about their definition. Also note the cooldown property, which defines how long auto-scaling should wait before triggering the next auto-scaling event. This makes sure that the scaling group will not go crazy on a short spike of traffic but the scaling is spread out more evenly.

The scaling configuration defined below the scaling group defines the template that is used to launch new instances. If you look closely, you can see that we reference the Ubuntu 18.04 image, so our instances will run that Linux distribution. The instance type is a small t5.nano to make sure our bill stays low for this experiment. Most other properties speak for themselves. Another important part is the user_data property, that points to a configuration file that is run on the machine during its boot process. If you look in the file user-data.conf you can see we install NGINX, fire it up and then start a stress test on the machine to generate some load:

The CPU load eats up a whole core for ten minutes, which will come in handy for the rest of the experiment.

Scaling rules & alarms

We have now defined a scaling group and a scaling configuration, but have not defined how to scale our instances. Those definitions are called scaling rules and you can see two of them in template.tf :

The definition is straightforward. We create a rule called add-instance and another one remove-instance . These add and remove one instance respectively.

In itself, a scaling rule does nothing. It only gives a name to a certain type of scaling activity with a set amount. To actually perform the scaling, a scaling rule has to be triggered. We spoke earlier of the way auto-scaling can be performed in most public clouds, and this is where Alibaba Cloud is no different. You can trigger a scaling rule manually, on a schedule or after a customisable alarm goes off. We will choose the last option for this experiment:

In the last part of our resource definitions, we define the alarms that will trigger our scaling rules. We have one alarm that adds instances when the average CPU usage of our instances goes above 70% in a 60-second period and that value is found two times in a row. Every metric is evaluated every minute, so if we have two consecutive minutes of 70%+ average CPU usage the alarm will trigger. It will then add an instance (see the alarm_actions property).

As you might recall from the scaling configuration user data, we stress the single-core CPU for a full 10 minutes after it booted. The scaling group starts with 1 instance and goes up to a maximum of 3. So, when we start the experiment, a single instance will be booted. But after two minutes, a second instance will be started as the average CPU is above 70%. Then, again after two minutes, the third will be launched. A scaling group never goes above its maximum number of instances, so the experiment is capped at 3 instances.

After some time, the stress on the CPU will stop. Our average will drop again. If you look in the second alarm defined above, you can see that we will start removing instances as soon as the average CPU usage will go below 10% two times in a row. So, the alarm will trigger and start removing the oldest instance. After some time, it will also remove the second instance and leave us with the instance that was started up last. The scaling group will also never go below its minimum number of instances, so we will keep running a single instance. This concludes the experiment.

Run the experiment

Now that we have an expectation of what will happen with our resources, let’s start the experiment:

export ALICLOUD_ACCESS_KEY="YOUR_ACCESS_KEY"
export ALICLOUD_SECRET_KEY="YOUR_SECRET_KEY"
export ALICLOUD_REGION="eu-central-1"

terraform apply

First we set environment variables with the access key, secret access key and region. The keys can be found in your account. I’ve set the region to eu-central-1 which is Frankfurt. Then use terraform apply to create an execution plan for the changes. Type yes to approve the changes and create the resources in your account.

You can go to the ECS and Auto-Scaling consoles to see the running experiment. It will take a few minutes before the first instance is booted and the second instance will be added. There are several buttons to see the monitoring for the scaling group, the instances and the alarms we set up. There’s also a Terraform query in place to get the running instances in the scaling group:

terraform refresh -target="data.alicloud_instances.scaled-instances"

This will output the currently running ECS instances in our scaling group. If you look up the public IP addresses of the instances in the output and browse to them, you should see the default NGINX landing page.

Conclusion

This concludes the auto-scaling experiment on Alibaba Cloud. You should now have a good idea of why we need auto-scaling for making sure our application is highly available and our costs are kept to a minimum. You’ve also created a scaling group, scaling configuration, scaling rules and alarms in Alibaba Cloud using Terraform. The next step is to put the instances behind a load-balancer. This will create a single entry point for your application and distribute all the traffic among your instances. The documentation for Server Load Balancer (SLB) resources should point you in the right direction!

Auto-Scaling on Alibaba Cloud was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pub-Sub (publish-subscribe) messaging is an asynchronous, decoupled style of messaging. The side where the message originates has no notion of any of the recipients, while the recipients do not know where the message came from. Pub-sub can help you to build a more resilient application, as you can change either the producing or subscribing side without any impact to the message flow. You could also scale both sides independently.

The AWS messaging candy shop

As our platform is run on AWS, we’ve been trying to use more and more of the services you get (almost) for free out of the box. One of these is Kinesis, which served as an inter-service message bus where messages would be published and consumed by any interested party. Using Kinesis saved us from having to run and manage our own Kafka cluster and did not force us to choose a specific language or framework like some others do. We were on AWS anyway, so that buy-in had already been taken. After a while though, it showed that it lacked some of the properties that we needed for our applications. Most annoyingly, it is hard to debug or replay a single message. In Kinesis, you can replay messages by setting the tracker to a specific message ID. While this gets the job done, it also replays any of the messages that were sent after the one that you wanted to replay. Naturally, the application should be able to deduplicate any of the messages but sometimes it can be tough or costly to make it fully resilient to double delivery. So we looked for alternatives.

We had been using Amazon’s Simple Queue Service (SQS) for the last few weeks, to drive some of the more 1-to-1 asynchronous communication between several services. Unlike the streams of Kinesis or Kafka, where the messages are sharded, a queue is a simpler abstraction that can form a natural ‘inbox’ for a service. Especially for low priority messaging or processes where the user is not waiting for a response, the queue is at its best. Using SQS queues makes it possible to configure a dead letter queue, which is where the messages will end up if they could not be consumed for whatever reason. That is the hook you can use to debug and replay messages for only the consumers that failed. The down side of plain queues is that producers of messages need to know where to route them, so you lose a lot of the advantages of decoupling mentioned earlier.

In comes the AWS Simple Notification Service (SNS), our hero of the day. As the name suggests, it is used to send notifications. As the name does not suggest, at first glance, is that these notifications can also be messages on SQS queues. Do you see the pattern emerging? Decoupled messaging through SNS topics that are linked to the inboxes of the services that need those specific messages! Feels a bit like the best of both worlds.

SNS + SQS = 🚀

Alright, let’s try to set it up. I’ve created a repository (available here) with the CloudFormation templates to set up an SNS topic that is connected to one or more SQS queues that will also be created. That connection is called a subscription. To make the stack a bit more manageable and because CloudFormation does not support looping natively, I’ve used Sceptre’s Jinja2 templating in the `templates/sns-pubsub.j2` file. You don’t need to make any changes there. If you do not want to deploy in eu-west-1, you need to change `config/messaging/config.yaml` accordingly. Also, let’s have a look at `config/messaging/sns-pubsub.yaml`:

When you’re done editing, you can deploy the stack to AWS (this assumes you have Sceptre installed and valid credentials):

This will create the SNS topic and the SQS queues. It also sets up a queue policy so the SNS topic gets the rights to send messages to the queues. If you decide that you need more or less queues at a later point in time, you can just add them to or delete them from the array and use the same command to update the stack.

Let’s send a message on the SNS topic:

Replace YOUR_ARN_HERE with the ARN of the created topic. You can find it in the SNS service in the AWS management console. This should return a simple JSON response with a message ID, if it succeeded.

If you now log in to the AWS management console and go to SQS, you should see the queues you created and that there is one message available in every queue. If you click a queue, you can select View/Delete Messages under Queue Actions. Click the blue Start Polling for Messages and your message should appear. My payload was as follows:

Awesome! As a final side note, when you want to use this in real life, make sure that the producing side has the right roles or permissions to send notifications on the topic. Similarly, make sure the consuming side has the permissions to read from the queue.

Originally published at xebia.com.

Pub-Sub messaging with AWS SNS and SQS was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the recent upsurge of Siri, Google Assistant and Amazon’s Alexa, speech recognition and synthesis have become an increasingly important tool in the developer’s toolbox. Working with speech data can not only improve the accessibility of your application. It can also increase conversion in your webshop, especially when customers shop on their mobile phones. Native apps have a large advantage in this space, as Apple’s SiriKit and Google’s Assistant SDK can get you up and running in a few minutes to hours.

For browsers, the story is different. If we disregard native apps for a second, plain old websites (or progressive web apps) seem to severely lag behind. How many websites can you mention from the top of your head that offer you the option to search with your voice? I can only think of a handful. Is it because people are scared to talk to their laptops, but are comfortable telling stories to their smartphones? Probably not. The biggest problem is that the easy solutions provided by libraries like SiriKit are not there for the web. Or are they?

To do speech recognition and synthesis you need a massive amount of training data for all kinds of languages in all kinds of settings. As a lone wolf or small startup, accruing that amount of data is near-impossible. So, you go to the companies that have already managed to train their neural networks: the Googles and Amazons. For a small amount of money per transcribed audio second, you can send your audio files to Google’s Cloud Speech-to-Text API or Amazon Transcribe. They will transcribe the audio within sub-second response times. Although the technology is cool, setting it up can be a hassle, especially if you want to use the audio recorder provided in the browser. The recording format of the browser does not always match the prescriptions of the cloud transcribers. This requires you to hit another backend for transcoding files into something that works. Yes I know, ugh.

When I recently had to implement a proof of concept voice search with some of my colleagues, we came across a 2012 (!) specification for speech recognition called the Web Speech API Specification. In it we found several interfaces like SpeechRecognition and SpeechSynthesis which seemed to do exactly what we were looking for. Another round on Google brought us to the following MDN page on the Web Speech API. To our big surprise, the browser support table showed the following:

You don’t say! Chrome has had support since around 2013, when Chrome 25 was released. Coming to think of it, it is even weirder now that so little websites seem to make use of the native API’s that are supported in Chrome, Firefox and Edge. Disregarding the part of the user base that still uses Internet Explorer, we continued on our journey.

We based our implementation on a repository that accompanies the MDN page (available here). After downsizing it a little bit, our implementation looked something like the following:

And it worked! This piece of code will ask for your permission to use the microphone after 3 seconds. If you click ‘allow’, it will then listen how well you do on pronouncing Bahasa Indonesia. Finally, it logs the transcribed results in your console. If your Bahasa is rusty, you can of course change the language code on line 6 to any of the supported languages shown in this demo. Note that these are the languages available in Chrome, other browsers might have a different list.

The Web Speech API seems to do a pretty good job at transcribing user commands in various languages. By default, the recording will stop as soon as the user stops speaking. If you want to transcribe longer pieces of audio, you can also set recognition.continuous to true . In production environments, you would first want to check support for the SpeechRecognition interface and handle its absence. Also, it might be wise to only start recording after the user clicks a button instead of triggering it with a window.setTimeout() . But I’ll leave that as an exercise to you.

Speech recognition is a breeze to implement using the Web Speech API. Speech synthesis is even easier to implement, of which you can see the proof here. Enhancing your website with speech recognition can really enhance the user experience when it comes to searching or shopping. Inexperienced customers might like to express their intent in natural language, describing what they want to do or find. In traditional UIs, this is often not possible. Most search inputs force you to type in a condensed kind of language that does not even closely resemble a grammatically correct sentence or action, something which speech recognition might solve. Speech synthesis can in turn help in the realm of chatbots, notifications or messaging, amongst others. In short, smart assistant technologies can not only help us in our homes or on our phones, we can also use them anywhere on the web where natural language is a good transport for communication with your users. So, how are you going to use it in your app?

This blog post was originally published on the Xebia Blog, available here.

Speech Recognition and Synthesis in the Browser was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

After the previous two more theoretical posts, it is time to get our hands dirty. As you might recall from part 2 of this series, companies can register an official account (OA) on WeChat to get access to all the APIs that the platform provides. An OA shows up in a user’s chat list, either between the regular chats or in a dedicated folder, and can send and receive messages to users or provide other useful on-demand services in the chat. Today, we will build an endpoint that follows the WeChat standards and can be tested on WeChat using the debug sandbox. It will echo any message a user sends by replying with the contents of the message reversed. Although this is not a really compelling example, it sets you up with the right tools to build more enticing user experiences on WeChat (and it forces you to register a WeChat account ;)).

We will build the endpoint using Express, a library for creating web applications on Node.js, but it is important to understand that any type of HTTP server can be used. If you have an existing backend that needs to integrate with WeChat, you could of course implement it on the existing stack. As long as it fulfils the (simple) requirements set forth by WeChat, it will suffice. Let’s get started. The final code is on GitHub.

Step 1: Setup

1. Make sure you have the latest (or a pretty recent) Node.js and NPM installed on your machine.
2. Install ngrok. It will be used as a secure tunnel to your machine that WeChat can access, more on that later.
3. Create a new directory on your filesystem and initialize a new project (this guide assumes you are on a macOS machine, Linux or Windows users should change the commands and directories accordingly):

Install Express through NPM:

Open the code editor of your choice and point it at the directory you just created (it should contain a package.json, package-lock.json and node_modules/ directory).

Step 2: Get access to the WeChat sandbox

1. Install WeChat through the App Store or Google Play if you haven’t already and register an account. If you’re already on WeChat, you can skip this step.
2. Browse to the sandbox (unfortunately, the authentication in the English edition seems to be broken, so we will use the Chinese edition).
3. Log in by clicking the green button (登录 ‘log in’) and scanning the QR code (you can find the scanner by tapping the ‘+’ on the chat overview).
4. Authorize the sandbox by confirming the login on your phone.
5. You should be taken to a page that shows your appID and appsecret.

Step 3: Initialize the Express app

Create a file index.js and set up a simple Express app:

This should be enough to test your setup. Go back to the terminal and run Node on the current directory (or index.js):

You should see the message WeChat endpoint listening on port 3000… logged to the terminal. Open a browser and browse to http://localhost:3000. It should show the simple message ‘WeChat endpoint’.

Step 4: Connect your app to WeChat

We will set up the routes that are specific to WeChat in a new file. Create a new file wechat.js, in the same directory as index.js, and set up the Express router in that file:

The first time you set up your server as the endpoint of an Official Account, it needs to respond with an echo string to verify that everything works correctly. WeChat will send a signature, nonce, timestamp and echo string (echostr) to your app with which you can verify that it is indeed WeChat that is connecting. If the signature is correct, you should respond with the echostr. To make this blog post a bit shorter, we will short-circuit this and always return the echostr (in production, you want to make sure that everything is correct though):

This code will check if an echostr is set as a query parameter and outputs it if it is set. Otherwise, it returns a 404 status code and error message. The logging statements will be shown in the terminal session you used to start up the Node application.

Before we can test whether our setup is working, we need to tell Express to also load the route in the wechat.js file when starting up in index.js. You can do that by adding the following line to index.js:

Put this below the app.get('/') call we created earlier, above the port definition.

You are now ready to check if WeChat can reach your application through an ngrok tunnel. If your Node application is still running, kill it with Ctrl-C and then restart it with node .. Make sure there are no errors while starting up and then open a second terminal window. Run ngrok and have it create a tunnel to port 3000 with the following command:

After a few seconds you should see the output from ngrok. Copy-paste the second ‘Forwarding’ URL (the https:// one) and paste it in your browser. This should connect without problems. You can open the ngrok web interface if you want (mentioned under the ‘Web Interface’ entry).

Now go back to the WeChat sandbox page, and log in if necessary. There should be two fields named ‘URL’ and ‘Token’ in the second panel. Append /wechat to the https:// URL of ngrok and paste it in the ‘URL’ field. Then type a random string in the ‘Token’ field (like 123456, the contents don’t matter because we are not checking the signature).

Click the green button (提交 ‘submit’). You probably get an error. Click the button again. You should now see a 200 OK request coming in on the ngrok dashboard, and the URL and token should be saved in the sandbox. Your account has been set up!

Step 5: Process incoming messages

Whenever a user starts following your official accounts, sends a message to your account or unfollows it, you will receive a POST request with some XML describing the message. To handle these messages, we need to add another endpoint to our wechat.js file, but instead of calling app.get() to create a handler for GET requests, we will call app.post() to register a handler for POST requests. As we don’t want to write a lot of XML-parsing boilerplate code ourselves, we will install an XML body parser for Express that can handle this for us. Let’s start with that.

Install body-parser and body-parser-xml with the following command in your terminal:

Let’s set up the body parser. In index.js, change the two top lines (the ones that set up the Express app) so they reflect the following:

This again sets up an Express app but now also installs the body-parser XML middleware which will help us parse the incoming request bodies. Let’s write the POST request handler, below all other code in wechat.js:

This piece of code sets up a POST handler on the router (line 6), takes the parsed XML out of the body and switches over the message type (line 9, the XML parser puts single text nodes into an array so that’s why we need to access it with index 0). The function then forwards the response object and parsed XML to one of three handlers (line 11, 13 and 15), for handling new events, text messages and any messages of unknown type respectively. The last step will involve implementing these handlers and returning an XML response that will be sent to the user.

Step 6: Write message handlers

This is the final step of implementing our WeChat echo service! As WeChat requires us to respond with XML, we will use template strings to easily create an XML string that can be written to the response object. WeChat then sends our message to the user’s device, so it feels as if they are talking to an actual person.

The function for setting up the template is as follows:

This will inject the receiver, sender and text content into an XML string that can be sent in the handlers. You should add this function above the handlers, by replacing TODO 3.

Next, we’ll set up our handlers. The most complex one is the handler for event type messages, as there can be multiple types of events that are triggered. We are only interested in the subscribe event, that is triggered when a user scans your QR code and taps ‘Follow’. This is the handler:

The handleEvent function extracts the event type from the XML data and checks if it is a subscribe event. It then creates an XML response by flipping the sender and receiver from the XML data and adding a string message. This will welcome new followers with the ‘Welcome to our Official Account!’ message.

While we’re at it, let’s implement the other message handlers:

In the handleText function, the string that is sent back is the reversed text message sent by the user. In the handleUnknown function, we just send back a message telling the user that our echo service does not (yet) understand what they’re saying.

And that’s it! After all the hard work it is time to see what happens when we start to follow our own Official Account and send some messages. To start following yourself, scan the QR code that is shown on the WeChat sandbox page, halfway down. On your phone, tab the ‘+’ at the top-right corner in the chat overview, then tap ‘Scan QR code’. After scanning the code, you should see a simple page with a large ‘Follow’ button. If you tap that, WeChat should open a new chat with the echo service. It should welcome you with the message we added earlier. If you send a text message, the echo service should reply with your message reversed. If you send anything else (a picture maybe, or your location) the echo service will tell you that it doesn’t understand that message type. If this all works, good job!

Wrap up

You’ve now seen how you can build a simple application on WeChat using the Official Account debug sandbox. As an exercise, you could try changing the code so the service greets the user with a menu, and based on the user’s response trigger an action like sending a video or image. The technical documentation of WeChat will help you in figuring out how to handle incoming media files and how to respond to them. Luckily, the ideas are similar to what you have seen so far.

Also, you could host your Node application somewhere in the cloud so it keeps reacting to new messages. Right now, if you kill your ngrok tunnel and/or Node app, WeChat will show an error to users that sent anything to your sandbox account. You can look into tools like Zeit Now to easily deploy your Node app in a more persistent way, or try the Amazon and Google clouds for easy hosting. After setting up your application elsewhere, don’t forget to change the URL on the sandbox page.

I hope the implementation of a simple echo service helped you assess whether or not WeChat is the right platform for your business and whether or not it can help you get more traction within the Chinese market. If you decide to go with WeChat, the first steps in this blog post should help you get up and running as quickly as possible. Should you inadvertently need help with the strategic or technical implementation of WeChat for your proposition on the Chinese market, please don’t hesitate to contact us!

This blog post was originally published on the Xebia Blog, available here.

Léon Rodenburg is a full stack development consultant at Xebia. He has a background in Computer Science and Sinology and is always on the lookout for the crossroads between the two. Having lived and studied in China for quite some time, he has put his knowledge of the Chinese language into practice by experiencing on- and offline daily life in Beijing like a local.

Some useful links:

• WeChat sandbox (Chinese, English version here but might be broken)
• WeChat technical documentation (English)
• WeChat on the Dutch iOS App Store
• WeChat on Google Play

Why Your Business Should Be on the WeChat Platform (Part III): Implementing an echo service was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part I compares WeChat to WhatsApp and explains why a presence on WeChat matters for any Chinese-facing business. Part II provides an overview of WeChat’s most essential features, describing how it’s typically used day-to-day. It also explains how companies can use WeChat to improve their business in the Chinese market.

WeChat, as the name implies, was first put out on the market as a messaging application, but has rapidly outgrown that label in recent years. In addition to messaging, WeChat boasts three other feature categories: payments, social, and official accounts (OA). All of these features integrate seamlessly into a single platform. And, with the help the APIs WeChat provides, these features can be intertwined to create complete user experiences within the app. If you are interested in creating a WeChat presence for your brand or company, the OA section of this article is most relevant.

Disclaimer: Some of the functionality discussed here is only available in the Chinese edition of WeChat, so it won’t show up if you install the English version. For this reason, some screenshot examples in this article inevitably contain Chinese, but the subtitles included should help you place the content in context. Depending on your platform, you will get the Chinese version when you have your phone set to Chinese or connect to a Chinese app marketplace. But to quickly start using WeChat, I recommend that you install the English version.

Messaging

Chat — WeChat has everything you might expect from a messaging application when it comes to chat functionality. It shows you a list of the most recent (group) chats, an item for collecting all your subscriptions in one place (more on that later), and any companies that you follow that have registered a service account.

In Figure 1, the fourth item on the list comes from Wumart (物美超市), a chain of supermarkets that implemented its loyalty program on WeChat. The fifth item is from Watsons (屈臣氏), a Hong Kong-based health and beauty care store, that did the same. Users can pull up their membership cards through these entries, more on that later.

Within a chat, you can send text, audio, video, pictures, emoji, animated GIFs (sometimes called “stickers”), location data, and other contacts (Figure 2). You can also send money to contacts or even to groups. Chinese people are much more likely to send audio messages than European texters, as these are more private and information dense. It’s also not not uncommon to see Chinese people hold their mobile phones up to their ears vertically to listen and then horizontally to their mouths to respond, like a walkie-talkie.

To add new contacts, users usually scan QR codes (Figure 2). You can pull up your QR code through your profile menu (the fourth item in the bottom navigation of Figure 1). The other party can then use the + button (in the top-right corner of Figure 1) to scan your QR “bread slice” (Figure 3), or another image of your choice. It’s also possible to search by name.

Audio and video calls — Audio and video calls are comparable to the features offered by WhatsApp, Skype and FaceTime. You can call a contact through the chat or the contact list, which you can reach by tapping the second item in the bottom navigation of Figure 1.

Payments

Store payments — You can reach your WeChat wallet (Figure 4) through the profile menu. The wallet prominently shows your balance, a button to pay in a physical store, and a list of linked credit/debit cards. Underneath these, there’s a list of Tencent services and banking-related apps that let you do things like top up your cell phone account or pay your utility bills. You can also invest your money or buy insurances. At the very bottom, there’s a list of other services, like hotels, taxis, bicycles and food, mostly from third-party vendors that have partnered directly with Tencent. Adding an app here is and won’t be covered in this post.

If you are at a physical store, you can pull up a barcode to pay using the first button on the top left (Figure 4). The cashier scans your code (Figure 5), and within one or two seconds, the payment succeeds (unless your account is overdrawn, of course). You leave the store without ever taking out your wallet, swiping a card, or filling in a code. This type of payment requires the store to obtain additional capabilities for WeChat Pay and WeChat-branded hardware to scan these codes.

Transfer money — As Figure 6 shows, you can transfer money to other people individually or in a group within chats. The group transfer is mostly used during Chinese New Year when people send the infamous “red envelopes” (红包) around. You can set an amount, and it will be divided equally among the group’s members or based on a first-come-highest-amount rate.

You can also create a QR code that others can scan to transfer money to you without adding you to their contact list. A lot of smaller vendors use this option because it doesn’t require a special WeChat account, plus it’s free. When you want to use WeChat to pay for something in a local shop, these vendors can provide a QR code, either printed or on their phones, for you to scan. Then, you just fill out the amount of money to transfer, authorize the payment and take off with your newly bought goodies.

Social

When it comes to social media functionality, WeChat feels like a hybrid of Facebook and Instagram with a slimmed down feature set. Called “Moments” in English, this aspect of WeChat translates literally as “friend circle.” You can post text, pictures, and video here. You can also like posts and place comments just as you would with the familiar social media platforms I just mentioned (Figure 7). However, there is one striking difference: the messages and comments you post are only visible to people in your contact list. This potential drawback often creates confusion, since you can’t read the whole thread of comments. For example, a friend may appear to respond to himself when he’s having a conversation with someone not in your contact list. Generally speaking though, it also saves you from the useless spam sometimes seen on other platforms. Plus, the moments you share only reach the audience they were intended for, your friends.

Official Accounts (OA)

An official account (公众号) is a special type of account on the WeChat platform. Companies use these accounts to get in touch with their customers, both actively and passively. The workflow for these accounts is mostly the same: users can physically scan your QR code from your advertisement, search for your company within WeChat, or get redirected to your account from somewhere else. Your OA shows up in the user’s WeChat as soon as she clicks “follow.” Simultaneously, you receive all the user’s details (Figure 8). There are four types of official accounts, each with its own specifications. Choosing the right type is critical because it cannot be changed once it’s registered.

Service accounts (服务号)
We’ve already looked at two examples of official accounts in the messaging section of this article. Both Wumart and Watsons use their OAs to update followers on discounts, pull people into their mobile websites, and identify customers in their loyalty programs.

Both examples are service accounts used mostly to offer customer support and build chat-like applications with custom menus on the WeChat platform. The menus often link to the companies’ web applications, which are loaded into a browser view within WeChat. Service accounts have access to advanced features like payments, geolocation and authorization APIs, but can only push messages to users four times a month. An example of these messages is shown in Figure 9, which contains the store’s latest discounts and offers. You can also use these messages to target specific customer groups, such as by age, location, or gender.

There is a JS SDK available that makes it easy to enhance the loaded web pages with native-like mobile features, like taking pictures or getting a user’s location. That only works if the page is loaded through the in-app browser view provided by WeChat. In the case of Wumart and Watsons, the WeChat APIs are used to authenticate so you can show your membership card to the cashier right from within their service accounts (Figure 10). You can host these pages yourself and link your domain name to the official account, so WeChat knows the page is served from the right location.

Behind the scenes, you have a server in place that receives the messages users send in the chat. You have a 48-hour window to (automatically) respond to the message, if you want to avoid spam. You can also query your own database and return results, or send images, video or coupon codes. You could even build a full-blown customer support dashboard on top of this, in which an actual operator can respond to incoming messages. Part III of this blog series describes how to build your own echo backend with a custom menu that will send all the messages a user sends back to them.

Subscription accounts (订阅号) — Subscription accounts are available for individuals and companies. These accounts look a lot like service accounts, but they have a few downsides. For one, they don’t provide access to the more advanced APIs, like payments. Also, your chat appears in a dedicated folder for subscriptions, not at the top of the chat. Subscription accounts do have one advantage over service accounts, however; they can send push messages once per day, instead of four times a month. For individuals that blog a lot or companies that generate content as their main product (news agencies, for example), a subscription account works well for this reason. Otherwise, I don’t recommend them.

Mini Programs (小程序) — In January 2017, Tencent released Mini Programs (MP, 小程序). MPs operate like an app by using WeChat’s SDKs to build a native experience. But they don’t require the user to install an app. An MP can be shared with other contacts and accessed through a QR code. So, it’s a suitable option if you need a heavy user intergace, but don’t want to go the whole nine yards on (cross-platform) native app development. There is a library of UI components you can use in your app to build on so you can get started quickly.

You can access an MP in two ways. When you pull up all your chats, a list of the most recently accessed MPs appears, along with a button to access the other MPs. The Discover (发现) page, item three in the bottom navigation of Figure 15, also pulls up the other MPs.

Enterprise accounts (企业号) — Chinese organizations use this type of account internally to make communication and file sharing easier. Only authorized employees can access the data, so it is not suitable for consumer-facing applications. I won’t discuss this account type any further here, but you can read up on enterprise accounts on the WeChat website.

Those are WeChat’s most essential features. I’ve left out a lot of others, like meeting people that are close to you, or winning prizes on television by shaking your phone, because those functionalities are not immediately relevant for companies. There are also some features that are most relevant if you have a physical store in China, like granting WeChat users access to your WiFi access point (and consequently have them follow your OA). This article gives you an overview of the most critical features and how you can use them to expand customer support and broaden the customer base of your products.

In the third and final issue of this blog series, I will explain how developers can start building a service account on WeChat through a sandbox account.

This article was originally published on Xebia Articles, available here.

Léon Rodenburg is a full stack development consultant at Xebia. He has a background in Computer Science and Sinology and is always on the lookout for the crossroads between the two. He lived and studied in China for quite some time, where he put his knowledge of the Chinese language into practice by investigating on- and offline daily life in Beijing, especially regarding technologies like WeChat.

Some useful links (in English):

• WeChat on the Dutch iOS App Store
• WeChat on Google Play

Sources

https://mp.weixin.qq.com/ (in Chinese)
http://admin.wechat.com/wiki/index.php?title=JS_SDK_DOCUMENT
https://mp.weixin.qq.com/debug/wxadoc/design/index.html
https://mp.weixin.qq.com/debug/wxadoc/dev/demo.html?t=1477656485442 (scan code to access the MP)
https://work.weixin.qq.com/?from=mp_home (in Chinese)

Why Your Business Should Be on the WeChat Platform (Part II): WeChat Feature Overview was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, the absence of WhatsApp wouldn’t change the way you get in touch with the companies you buy products from because this new customer support functionality isn’t widespread yet. And it wouldn’t change the way you do your daily banking because it’s still just a chat app, and it certainly wouldn’t change the way you prove your identity to, say, the government. If WhatsApp went away, you would probably just use an alternative chat application to talk to and message your friends, like Facebook Messenger or Telegram. WhatsApp is only a small part of our daily life, and we need a multitude of other services to get everything else done. Removing one these services doesn’t change everything, we just swap it for an alternative and go back to our daily business.

You can probably also imagine a world without WeChat (微信), because you most likely haven’t heard much about it. Tencent (腾讯), the Chinese internet giant that’s worth more than Google, released this multi-purpose social media mobile application software back in 2011. Today, it has over 980 million monthly active users (902 million daily active users), and has been called China’s “App for Everything” and the world’s first “Super App.”

But besides comparing it to messaging applications like WhatsApp and Facebook Messenger, most media coverage on WeChat doesn’t go much deeper than that. Only a handful of news items, blog posts or articles discuss its particular features, and even fewer discuss how to use WeChat in sales and marketing strategies for business. On the implementation side, an English version of the documentation is available, but it is outdated in some areas and completely lacking in others.

Clearly, WeChat is not much of a thing here, yet, but Tencent’s app did feature in the last Technology Radar by ThoughtWorks. The article puts WeChat in the ‘Trial’ category, which means that it’s worth checking out to determine if it could help your business, but hasn’t gained much traction. In the future, that might change.

To see why, let’s go back to our previous discussion and switch things around a bit. Would the daily life of a Chinese consumer change if we took WeChat out of it? You bet. WeChat has penetrated so many aspects of daily life that the average Chinese consumer is lost without it. From store payments, hotel reservations and online purchases to gym membership cards, frequent flyer statuses, and supermarket discounts, everything is contained in “the one platform to rule them all.”

Almost all WeChat users are also active on the Instagram-like social media functions that allow you to share pictures with your friends and respond to their funny cat videos. What’s more, it can even be used to officially prove your identity, as it is replacing paper identification in Guangzhou and soon throughout China. More and more Chinese businesses are moving abroad and taking WeChat along with them as their primary marketing platform. So, it is not surprising that WeChat has finally started popping up on the Technology Radar. If we look at the pervasiveness of WeChat in the Chinese market and what it might do in ours, do you think the comparison to WhatsApp is still fair?

Definitely not. WeChat has surpassed WhatsApp in so many ways that it is hard to see why Facebook didn’t manage to improve their platform earlier. Tencent’s WeChat Open Platform (微信开放平台) already opened its doors for innovative solutions to everyday problems years ago, not just messaging and communication.

Anyone can step up and open a sales or marketing channel on WeChat. The commercial users range from small street vendors who want to accept payments through WeChat to large retailers that implement a full-blown e-commerce and customer support platform that replaces their webshop. WeChat APIs take away the complexity of authentication, payments and messaging by providing these out-of-the-box, yet with enough flexibility to connect with a customized backend of any larger enterprise. With over 900 million daily active users, it is not hard to see how an effective presence on WeChat can easily double or triple your user base overnight.

From a marketing or sales perspective, the pervasiveness of WeChat is precisely its beauty. If you want to increase your visibility on the Chinese market or reach more of the Chinese consumers in your own country, it is imperative that you have a presence on Tencent’s app, because these 900 million users are already accustomed to contacting companies through WeChat.

The downside is, it can be hard to get started. Especially when it comes to the more advanced features, like payments and location services. If you don’t read Chinese, you are mostly out of luck. Paired with a rather lengthy application process for companies and the fact that you need a locally registered business entity in China should you want to target mainland Chinese users, publishing anything even remotely useful on WeChat can be a daunting task.

At Xebia, we’ve experienced these difficulties first hand. Over the last few months, we’ve been investigating WeChat’s possibilities and built several applications on top of its APIs. We’re very enthusiastic about the potential and would like to help you get up and running with WeChat quickly. Over the next few weeks, we’ll follow up on this article as a three-part series. Part two will present the most important features of WeChat, and part three will provide a step-by-step guide on how to build your first application on the platform. This series should help you assess whether or not WeChat has something to offer your business. In the meantime, feel free to contact me or Xebia directly about WeChat if you’re interested. Let’s make WeChat a thing here!

This article was originally published on Xebia Articles, available here.

Download WeChat from Google Play
Download WeChat from Apple App Store

Some useful links (in English):

• WeChat showcases
• WeChat documentation
• WeChat Official Account Admin Platform

Sources

Constine, Josh. WhatsApp Announces Free Business App, will Charge Big Enterprises. TechCrunch. September 5, 2017.

“How WeChat Became China’s App For Everything”. Fast Company. January 2, 2017.

Kasteleijn, Nando. Chatten en betalen we straks in één app? In China gebeurt het al. NOS. April 10, 2016. (in Dutch)

Rutherford, Sam. 5 Things to Know About Tencent; The Chinese Internet Giant That’s Worth More Than Facebook Now.Gizmodo. November 27,2017.

Technology Radar. WeChat. ThoughtWorks. Last accessed on January 14, 2018.

WeChat Open Platform, documentation

Wikipedia. WeChat. Last accessed on January 14, 2018.

Zhao, Christina. WeChat is About to Become the World’s Most Popular Form of ID, but You’ve Probably Never Heard of It. Newsweek. December 28, 2017.

Why Your Business Should Be on the WeChat Platform was originally published in Xebia on Medium, where people are continuing the conversation by highlighting and responding to this story.