Develop a mind-reading Twitter client with Azure Cognitive Services

Published in

Windows Developer

9 min readMar 12, 2018

Adding machine learning, artificial intelligence and cognitive abilities to your Windows applications is easier than ever, thanks to Microsoft Azure services. Imagine a Twitter client that practically checks the user’s state of mind before tweeting, potentially saving them an embarrassing tweet made in anger. With the Emotion and Text Analytics APIs from Azure Cognitive Services, not only is this possible, it’s easy to do, analyzing both the text of a tweet and the user’s facial expressions, before the send button is clicked.

Getting Started with Azure Cognitive Services

Azure Cognitive Services includes a portfolio of machine learning powered APIs that fall into five primary areas:

· Vision — Image/video processing and classification services, etc.

· Speech — Speech-to-text, voice verification, etc.

· Language — Natural language processing including translations, linguistics, spell check, etc.

· Knowledge — Recommendation systems, academic knowledge, etc.

· Search — Bing Search API that covers web, media, news, etc.

To access these services, API keys are necessary:

1. Navigate to the Cognitive Services API Console.

2. Login or create a free account.

3. Select the Vision tab.

4. Locate the row for Emotion API and click Get API Key button.

5. Select the Language tab.

6. Locate the row for Text Analytics API and click the Get API Key button.

7. Make a note of the endpoint and API keys.

Creating the Safe Tweet App

To get started creating the Safe Tweet app, follow these simple steps to set up your project:

· Open Visual Studio 2017

· Choose File -> New -> Project from the main menu

· Select Windows Universal from the left navigation pane

· Select Blank App (Universal Windows) from the right pane

· Enter the name of your app in the Name field (we’re using “SafeTweet”)

· Enter a Location, or keep the default

· Select the OK button

After creating the initial project, Visual Studio will prompt you to select the target and minimum platform versions.

· Ensure the Target version is set to Windows 10 Fall Creators Update (10.0; Build 16299)

· Set the Minimum version to Windows 10 Fall Creators Update (10.0; Build 16299)

We’re setting both to the Windows 10 Fall Creators Update because some of the packages we’re using have a dependency on .Net Standard 2.0.

Implementing the Text Analytics API

The Text Analytics API is a great algorithm for determining the overall sentiment of a block of text to determine its positivity or negativity.

Microsoft provides a package via NuGet that simplifies integrating the Text Analytics API into a UWP app. Install the NuGet package from the NuGet console:

Install-Package Microsoft.ProjectOxford.Text -Version 1.1.7

With this package installed into the project, a piece of text can be analyzed with the following steps:

1. Create an instance of the SentimentClient class using the API key previously obtained for the Text Analytics API in the Cognitive Service Console.

2. Create a list of SentimentDocument objects containing the text to be analyzed.

3. Call the GetSentimentAsync() method providing the list of SentimentDocument objects as a parameter.

4. Process the collection of Document objects returned from GetSentimentAsync().

Note, SentimentDocument contains an Id property that correlates a request to the corresponding response. In the sample Twitter client app, the Id happens to be unnecessary since only one line of text is being sent for analysis.

private async Task<float> RequestTextSentimentAnalysisAsync(string text){var client = new SentimentClient(this.Settings.Text.Key1){// NOTE: use the URL endpoint provided by the Cognitive Services API console.Url = this.Settings.Text.EndPoint};var document = new SentimentDocument{Id = Guid.NewGuid().ToString(),Text = text,Language = “en”};var request = new SentimentRequest{Documents = new List<IDocument> { document }};var response = await client.GetSentimentAsync(request);// Only one document was sent, therefore only one result should be returned.var result = response.Documents.FirstOrDefault();if (result == null){throw new ApplicationException(“Text Analysis Failed.”);}return result.Score;}

Implementing the Emotion API

The Emotion API uses facial expression recognition to detect eight different emotions: anger, contempt, disgust, fear, happiness, neutral, sadness and surprise. This algorithm can detect multiple faces in a single image and provide confidence measurements for each emotion for each face.

Microsoft provides a package via NuGet that simplifies integrating the Emotion API in UWP Apps. Install the NuGet package from the NuGet console:

Install-Package Microsoft.ProjectOxford.Emotion -Version 2.0.40

After installing this package, analyze the emotions in an image with the following steps:

1. Create an instance of the EmotionServiceClient class using the API key previously obtained for the Emotion API in the Cognitive Service Console.

2. Call the RecognizeAsync() method passing in the image itself as a Stream, or as a String that represents a Url to the image.

3. Process the collection of Emotion objects returned from RecognizeAsync().

The following code demonstrates these steps:

private async Task<IEnumerable<Emotion>> RequestEmotionAnalysisAsync(Stream image){// NOTE: use the URL endpoint provided by the Cognitive Services API console.using (var client = new EmotionServiceClient(this.Settings.Emotion.Key1,this.Settings.Emotion.EndPoint)){try{var results = await client.RecognizeAsync(image);return results;}catch (ClientException ce){Debug.WriteLine(ce.Error);}}return null;}

Note: The EmotionServiceClient class implements IDisposable, and therefore the instance needs to be disposed. This is different behavior than the SentimentClient class from the Text Analytics API.

Processing the collection of the returned Emotion objects is also straightforward. For every face detected in the image, the collection will contain a corresponding Emotion object. Each Emotion object corresponds to a face recognized in the image and includes:

· A Rectangle object with the coordinates and size of the face found

· An EmotionScores object, with a numeric score ranging from 0 to 1 for each emotion detected (the sum of which will equal 1).

The ToRankedList() helper method from the EmotionScores object can be used to rank the scores for easier consumption.

The following code demonstrates consuming the Emotions API and determines the primary emotion:

var emotion = await this.RequestEmotionAnalysisAsync(imageStream);var face = emotion.FirstOrDefault();if (face == null){this.State = “Couldn’t detect a face or there was another issue with the Emotion API.”;return;}var primary = face.Scores.ToRankedList().First();

Taking a Snap Shot

The above Emotion API sample uses a Stream of the image to analyze. There are several methods in the UWP to obtain an image — reading it from disk or directly from a web cam. For the Safe Tweet app, reading directly from the web cam would create a seamless user experience. A good starting place to learn more about using the web cam in a UWP app can be found here.

Since UWP are sandboxed and have restricted rights, developers need to request access to specific capabilities:

· In Visual Studio, open the project manifest and select the Capabilities tab.

· Select the Webcam capability.

The MediaCapture class is a mechanism used to capture video and photos from a webcam and provides a more seamless approach than using the CameraCaptureUI class (which simply uses the Windows built-in camera app).

Before using the MediaCapture class, it must be initialized, as follows:

_mediaCapture = new MediaCapture();var settings = new MediaCaptureInitializationSettings{StreamingCaptureMode = StreamingCaptureMode.Video};await _mediaCapture.InitializeAsync(settings);

Note, failure to set the capture mode to StreamingCaptureMode.Video value, MediaCapture will default to both video and audio and will require adding the Microphone capability in the project manifest.

Once MediaCapture is initialized it can be used to take a photo as follows:

private async Task<Stream> TakeSnapshotAsync(){var encoding = ImageEncodingProperties.CreateUncompressed(MediaPixelFormat.Bgra8);var capture = await _mediaCapture.PrepareLowLagPhotoCaptureAsync(encoding);var photo = await capture.CaptureAsync();var frame = photo.Frame;await capture.FinishAsync();return await this.WriteToStreamAsync(frame);

Converting the captured photo to a Stream that the Emotion API can use is a little more involved:

private async Task<Stream> WriteToStreamAsync(CapturedFrame frame){using (var outputStream = new InMemoryRandomAccessStream()){var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, outputStream);encoder.IsThumbnailGenerated = false;encoder.SetSoftwareBitmap(frame.SoftwareBitmap);try{await encoder.FlushAsync();}catch (Exception e){Debug.WriteLine(e.Message);}await outputStream.FlushAsync();var ms = new MemoryStream();await outputStream.AsStream().CopyToAsync(ms);ms.Position = 0;return ms;}}

Posting to Twitter using the UWP Community Toolkit

While not part of the Cognitive Services API, the sample app wouldn’t be complete without demonstrating how to connect to Twitter and post the tweet.

Start by visiting https://apps.twitter.com/app/ to create and register the app.

Note: The callback URL is optional and does not need to be provided.

After submitting the above form, click the Create my access token button and select the level of access needed (read only, read and write, or read, write and access direct messages). For this app, select Read and Write.

Once completed, the Twitter registration page will present a summary of the app’s setting including the consumer key, the consumer secret as well as the access token and access token secret.

This consumer key and consumer secret will be needed to connect the UWP app to the Twitter infrastructure (note: the consumer secret should never be human-readable in your app or checked into source code repositories).

Back in Visual Studio, install the UWP Community Toolkit Services NuGet package.

Install-Package Microsoft.Toolkit.Uwp.Services -Version 2.1.1

This package not only allows developers to easily integrate with Twitter, but many other services including Bing, Facebook, LinkedIn, OneDrive, and more.

The Twitter Service provides a rich API allowing consuming apps to tweet, retrieve the user profile or timeline, or live-stream tweets. The following snippet shows how to connect to the Twitter service and send a text-based tweet.

public Task SendTweet(string message){// The consumer key, secret, and callback Url entered when creating the Twitter appTwitterService.Instance.Initialize(consumerKey, consumerSecret, callbackUrl);// Login to Twitter returns true for a successful loginif (!await TwitterService.Instance.LoginAsync()){// Unable to loginreturn;}// See docs for several other optional parameters defined in Twitter “update” APIvar status = new TwitterStatus{Message = txtTweet.Text,};// Post a tweet with the status and the picture streamawait TwitterService.Instance.TweetStatusAsync(status, stream);}

Cognitive Capabilities

There may come a day when most apps will perform cognitive tasks previously thought only possible by humans. Saving us from ourselves may just be a small component of the larger set of capabilities available with a few lines of code and Azure Cognitive Services.

We’ve only scratched the surface of what is possible in a UWP app consuming Azure Cognitive Services and leveraging the UWP Community Toolkit. Check out the links below to explore more opportunities.