ikcomponeer blog part 5 — deserializing compositions in an Azure Function
INTRODUCTION
In the context of the migration / rebuild of the ikcomponeer.nl music composition platform for children, I am hoping that maybe some day someone else may also be enabled to work on the platform. Therefore I am documenting my work via these blog posts (and maybe it can just be fun to read or you will learn something). In the last post (click here) we looked at the ikcomponeer music composition environment basic data structure. This blog post we will work towards migrating those data to the Azure platform, especially to CosmosDB. It will be a long read!
LOGICAL DESIGN
We want the old platform to function side by side with the new platform, enabling forward but not backward compatibility. That is, when a user stores a composition on the old platform he should be able to continue with the newer version, but once the new version is used, it will not necessarily be possible to open the same data with the old version.
In the old database we will make a flag “migrated” and we will have a job run periodically to retrieve and migrate the least recent composition without the “migrated” flag. When successful this job should call an API method on the old platform to set the “migrated” flag to “true”.
MAKING SURE THE MIGRATED PROPERTY WILL BE ZERO WHEN A COMPOSITION IS SAVED
Starting in MySql we will give each composition a field ‘migrated’ with a default value of zero. Every time a user stores a new composition via the old platform, it will have a value of migrated = 0.
But what will happen when the user changes an existing composition? To make sure the composition will be re-migrated (updated to the target Azure platform) we find all places where a composition gets saved in the PHP code.
Because we are using a model-view-controller design, all the cases where a composition is stored are in the models/mcomposition.php file. Because we use the RedBeanPHP framework, we have to look for the R::store(composition) method.
Because it is so long ago that this code was written I really have no idea anymore which methods may consist of dead code, or why it looks like a cabinet of horrors…. just add the migrated = 0 assignment to the compositions where ever they are stored.
SUPPORTING THE EXTRACTION OF UNMIGRATED COMPOSITIONS
To create an API method in the old PHP platform we first need to add a controller method. By adding a controller method we define the functionality as it is exposed to the outside world. In any case, without a controller method you cannot call any functionality.
Let’s see what the method when called via Postman returns. Hopefully it returns “hello there”.
Let’s call a method in the data model which has access to the database. Also there are some reusable methods present there, all having to do with the composition and how it is represented in the MySQL database.
Also, let’s make a minimal implementation just to test the wiring.
And test again in postman. We see an array where something named “name” has a value of “value” and also, you are nice!
Let’s create an associative array that will have some null fields for the data objects that we are going to retrieve.
The composition will contain the composition itself, and the environment will contain all the data for all other elements in context of the composition, such as the containers where the user drags the music blocks from, which he then uses to create the composition.
Let’s load the least recent composition that was not yet migrated. We can use the mtime field which I happen to know is updated every time a composition is modified. It is a unix timestamp — a number expressing the seconds since January first 1970 so we can sort on it, and we will get the unmigrated one that is the oldest.
Before we will dive into the data itself, let’s start with the id, just to see it at work.
Just to be sure if it is correct, let’s write a good old SQL query to check.
….
Yaiks!! Apparently it is not working … looks like the RedBeanPHP silently ignored our order by clause. Well, let’s resort to SQL to find the ID then.
In my solution here, we do it in two steps … we bother the DB twice. Once for the selection and the second time for the loading. It is not beautiful but we’ll get over it.
The result is correct now.
Let’s see how the code will respond when we run out of unmigrated compositions. It should exit earlier and return our associative array with null values.
And voila
Now let’s use the built in getProperties method of the redbeanPHP. It will export the composition properties. It is used in other read methods for compositions like this as well (I have forgotten why) so let’s do it here, too.
It results in the data of the composition populating our return array.
It is immediately apparent that there is some strangely formatted XML in there. I happen to know it is the actual composition data: the tracks with the blocks, their positions in the tracks and the references to their MP3 contents. We will deal with it later. In order to show you a neatly formatted and comprehensible bunch of composition data I’ve replaced it below and used another tool to add some spacing.
{
"composition": {
"id": "76",
"created": "1281944819",
"content": "<HERE COMES THE XML\/>",
"published": "0",
"hasmp3": "1",
"name": "Beethovenssssssss",
"description": null,
"pubtime": "1281944819",
"user_id": "4",
"mtime": null,
"template_id": "6",
"styleicon": null,
"project_id": "1",
"mixtime": "0",
"limited": "0",
"empty": "0",
"agent_id": null,
"migrated": "0",
"user": {
}
},
"environment": null
}Looks like it is coming from prehistoric times when the mtime field was not even implemented :-)
Now we will add some code to retrieve the so-called environment for the composition. This mainly consists of the context of a static set of music blocks, subdivided in orchestral groups such as woodwind or brass, as well as some information about tempo, the grid on which the blocks can be placed, etc.
We use functionality already present in the PHP file that defines the model for this entity, named “Musicbase”. This functionality is already used by the front end when loading a composition. Only thing is, it does not contain an ID. We would like to use an ID like that, to determine equality as to not have to store the same thing a number of times. So we added an identifier field. The identifying property here is a “template” because a template has a music set (consisting of instrumental groups, with the phrases in them).
The result now contains these data as well:
{
"composition": {
"id": "76",
"created": "1281944819",
"content": "<COMPOSITION CONTENT AS XML GOES HERE\/>",
"published": "0",
"hasmp3": "1",
"name": "Beethovenssssssss",
"description": null,
"pubtime": "1281944819",
"user_id": "4",
"mtime": null,
"template_id": "6",
"styleicon": null,
"project_id": "1",
"mixtime": "0",
"limited": "0",
"empty": "0",
"agent_id": null,
"migrated": "0",
"user": {
}
},
"environment": {
"projectRules": [
{
"id": "1",
"type": "gridQuantizeNumerator",
"description": "uit hoeveel delen bestaat het deel",
"value": "8"
},
{
"id": "2",
"type": "gridQuantizeDenominator",
"description": "hoeveel delen zijn samen 1",
"value": "1"
},
{
"id": "3",
"type": "measureNumerator",
"description": "uit hoeveel beats bestaat een maat",
"value": "4"
},
{
"id": "4",
...
...
...Above you see the “projectRules” which determine the tempo and the grid.
Then below that, you see the groups of clips. You can see that the clips have some visual properties as well.
...
...
...
"projectMaterial": {
"id": "6",
"name": "Beethoven",
"textSlogan": "Dit is de eerste muziek die undefined<br\/>het Metropole voor de cmoponeertool opgenomen heeft.",
"background": "http:\/\/ikcomponeer.nl\/data\/clients\/3\/images\/template_slogans\/beethoven.png",
"type": "template",
"parts": {
"1": {
"id": "1",
"name": "hout",
"marginx": "2",
"marginy": "2",
"order": "0",
"width": "190",
"clips": {
"1": {
"id": "231",
"exitpoint": "12000",
"entrypoint": "0",
"name": "01fluit",
"tag": "a1",
"icon": "images\/clip_icons\/iconen-fluit.svg",
"ficon": null,
"ficonroll": null,
"order": "0",
"color": "#FF007B",
"width": null,
"height": null
},
"2": {
"id": "232",
"exitpoint": "12000",
"entrypoint": "0",
...
...
...And there are quite a few. When scrolling down, it ends with the strings and then there is the aforementioned identifier field, and there ends the definition of the environment:
"79": {
"id": "309",
"exitpoint": "12000",
"entrypoint": "0",
"name": "12strijkers",
"tag": "b8",
"icon": "images\/clip_icons\/iconen-viool.svg",
"ficon": null,
"ficonroll": null,
"order": "11",
"color": "#910D00",
"width": null,
"height": null
},
"80": {
"id": "310",
"exitpoint": "12000",
"entrypoint": "0",
"name": "13strijkers",
"tag": "eind",
"icon": "images\/clip_icons\/iconen-viool.svg",
"ficon": null,
"ficonroll": null,
"order": "12",
"color": "#870300",
"width": null,
"height": null
}
}
}
}
}
},
"environment_identifier": "6"
}The next piece of PHP code adds a list of so-called scroll items. These are other music puzzles, music sets, environments or templates (whichever term suits you best I guess) which the user can choose from. The scroll items presented to the user can be different per portal where the composition tool is hosted (this is called a “project” in the old database).
After adding this code, the data has the scroll items as well. They are actually a list of references to other compositions. Also they seem to have lots of fields that are of no importance to the front end.
"scrollitems": [
{
"id": "6592",
"created": "1371381407",
"content": "",
"published": "0",
"hasmp3": "1",
"name": "Buckley's Kronkels",
"description": null,
"pubtime": "0",
"user_id": "1126",
"mtime": "1387328393",
"template_id": "15",
"styleicon": null,
"project_id": "1",
"mixtime": "0",
"limited": "0",
"empty": "0",
"agent_id": null,
"migrated": "0",
"order": "3"
},
{
"id": "6901",
"created": "1373919291",
"content": "",
"published": "0",
"hasmp3": "1",
"name": "Mozart's Blaaskwintet",
"description": null,
"pubtime": "0",
"user_id": "15290",
"mtime": "1387317157",
"template_id": "21",
"styleicon": null,
"project_id": null,
"mixtime": "0",
"limited": "0",
"empty": "0",
"agent_id": null,
"migrated": "0",
"order": "7"
},
...
...
...
],
"scrollitems_identifier": "17"
}The last thing we need to add are special configuration settings that may change the behavior of a music environment. Also some behavioral changes can be applied on the level of the project. Both these collections are merged into one format and therefore we will need to compose the identifier also from the project and the template id. This looks as follows in code.
The data for the conf override array looks like this, for example. The license text looks a bit worrying … this is however a detail that we will deal with later.
"conf_identifier": "6-17",
"conf_override": {
"conf_helpVidDialAtStartUp": "0",
"conf_topLeftButtonText": "MUZIEK LIJST",
"conf_topLeftButtonNote": "open session module",
"conf_topLeftButtonMode": "button",
"conf_saveButtonVisible": "1",
"conf_defaultTemplateCompositionId": "6028",
"conf_licenseText": "Gelicenseerd aan: undefined<br\/>Metropole Orkestundefined<br\/>"
}Okay, we’re done with returning all the data that the front end needs, in order to enable the functionality for the user.
SUPPORTING THE MANIPULATION OF THE MIGRATED FLAG
After (and only after) the migrating process has transformed and stored the composition to its new destination, will it mark the composition as migrated. You may notice by the way, that this setup is not suitable for parallel processing. In this case we won’t need to do parallel processing. Also when there would be a race condition, the worst thing that could happen is the same version of the same composition is read, transformed and stored twice, which isn’t a problem at all.
So let’s make a method that could mark the composition as migrated. The migration process will derive the ID from the data. We will add it as an argument to the exposed method in the controller.
Now let’s call the model, create a method there and test if the wiring was okay.
result in postman
Now let’s store it to the database…
And in the database we should see the flag having the value 1.
And the getLeastRecentUnmigratedComposition method should now return the next one … yes there you go
And does the database also think it’s 1779?
Yes indeed! Okay, let’s just reset number 64 to migrated = 0 because it was only a test but of course we did not migrate it yet … that will be the next part.
SETTING UP THE FUNCTIONS TO EXTRACT, TRANSFORM AND LOAD THE COMPOSITIONS
We will use Azure Functions to make a timer triggered function to migrate the compositions one at a time, each so-many seconds (for example each five seconds). Let’s see how that goes. I will make a new function app so that i can easily throw it away when all storage is on the new platform, when the old platform and thus the migration flow will not be used any longer.
From the home page of the Azure portal we will go to Resource Groups
Here we find the resource group for the composition environment.
Then we click the add button to add a Function App to the resource group.
search for adding a function app, find Function App and click it, read all about it and then click Create.
The following is kind of important. We will go through it one by one.
The name will make sure that you recognize it. Take some time to think of a good name. The hosting plan is worth mentioning. The consumption plan will make sure that only for the amount of seconds that functions are actually executing, there is money to be payed. The data center will make for less transport costs as all the other resources are or will be also in the same data center. It is recommended to create a new storage so that you can delete it together with the function app. Accumulating such resources will make sure you will at some point never know anymore, which to keep and which to throw away … this will cost you. Application insights I must say I never used with Functions, let’s turn it on because we might run into some use cases and find out how to do it.
After pressing create it takes a few moments for the platform to set everything up.
After it was finished it shows a button to go to the resource. This works nicely.
Here we are … but there are no functions yet. This is what we will change. We will make a function that triggers each five seconds. When hoovering over the functions list …
there appears an “Add” button
CLICK!
We will of course choose a timer triggered function for C#.
We see some basic code pre filled in
Also when clicking the log and observing it for some time we see indeed the function periodically runs, triggered by a timer.
However it doesn’t run nearly as often as we want it to so we will click the integrate tab to adjust this.
We now see an expression with six segments.
The seconds are zeroed so it will run on the 0th second once in five minutes, every hour, day, month and day of the week. Instead we want it to run every five seconds, like this.
After pressing save and returning to the log, we can see it worked.
We will use this function later to run the code that we will now develop … to make it easier and more controlled to develop, we will do that in a manually triggered function. First let’s turn the timer triggered function off.
Now add a new one by clicking the “+”
As we are apparently now considered advanced users we get a lot of options in the form of tiles. There is a manually triggered C# option so let’s do that.
It is quickly created by the platform and we can trigger it with the run button.
The log shows indeed it ran.
But wait … there is some test data. It is entered via the test tab on the right. Our function will have no arguments actually, so let’s remove it.
Now we have an empty string as input. Let’s just remove the function parameter as well.
Oh-oh the function won’t compile now — it says “missing a trigger named input”
Looks like for a manually triggered function there should be an argument named input in the function profile. Let’s re-add it, although we will not use it.
Looks like now everything is happy.
RETRIEVING AN UNMIGRATED COMPOSITION
We’re using the System.Net.Http HttpClient library with its async methods. We are awaiting their Results so the code looks somewhat strange.
When we run it we can indeed see that the logger has received the bunch of json data with the XML data wrapped in it.
While we’re logging this amount of data I am wondering what happens to the log when I am not looking. In the coding window I am connected to the log streaming interface, but something tells me this log is written to the storage container that was created together with the function. Let’s use storage explorer and have a look. There is a storage container with a name that looks a bit like my function app. This is by default, we were given the option to choose our own name (see the image above from when we created our function app).
The Storage Explorer is a product that sits nicely in my task bar because I use it a lot.
I can navigate all my storage resources like this.
I have to explore quite a long path but then finally I discover a log file for each day which indeed contains the entire log:
Then exploring further I also see another log in tabular format.
It also contains the invocation data as well as the log output, but this time it is ordered per invocation. I am beginning to understand that at high volume the storage account may become a cost burden and I am wondering if there is some form of cycling it to eternity available. Some googling reveals there are logic app templates for these sort of things: deleting blobs (and probably also tables) after a period of time. We will revisit it later, naturally. For now, let’s move on. And let’s not log ridiculously large amounts of data anymore!
CREATING A NUGET DESERIALIZATION PACKAGE
Deserialization is the process by which you can load data packaged as text (in the string data type) into a strongly typed object (in a class) that you can program more easily on. As you can see the composition data come to us as a large text and therefore we will need to deserialize it. Because this involves some class definitions we prefer to package these structures as library projects and not mention all those classes in the function code. That would be ugly and messy. Also it would make those data structures less reusable. We will see how easy it is to use the platform and publish a library. Solet’s create a project in order to do that…
In order to use the project contents as a library in an Azure function we must choose the .NET Framework project type.
Also you have to google which version of the .NET Framework to use with Azure Functions. If you are not sure, no problem. The Function App will give you an error that will direct you to the right version and then you can change it later. In my case I will choose the .NET Framework 4.6 — it is part of the project properties.
Now that we have the project ready, let’s change the name of the standard class file to something more good looking.
Let’s delete the class definition from the file contents. We will use a special trick to generate the data structure from the sample we have already.
Now we take a copy of the data that were retrieved … we can use the stored log file to take the copy (by the way I have replaced the ugly XML with something more civilized because else I expect complications … we will deal with the XML shortly).
Now here comes the trick … with your cursor still in the .cs file where we removed the empty class definition, under Edit go to Paste Special and choose Paste JSON As Classes.
This gives an enormous amount of structures and I don’t immediately find it all making such a lot of immediate sense like you hoped for …
So let’s also paste the data in a JSON file to have an example and decide what to do.
Now we have the example itself, looking nice in Visual Studio.
Let’s use the collapse buttons and take it top down.
We see seven fields of which the three identifiers are easily classified as a string (text) type.
In the cs file, when we look from top down, we see also those properties.
Those were the easy parts. The others are special structures. Starting from above we first see the composition type. Uncollapsed in the json example it looks like this:
Except for the content XML (yukh) it looks all very simple. Let’s see what “paste special” has made of it.
Okay, this needs some tidying up. We can see that there is an empty object “user” in the example. But the classification assumes a User class on that field. But how will it define a user then?
Well that’s not much of a class definition is it? Let’s remove it, and the “user” field of type User as well. Because we also have the field user_id which can hold the id of the user as a reference to it. We can then always load the user when we need it.
From my knowledge of the data structure I see some boolean types. I simply know that these can be only 0 or 1 and are cast in PHP to boolean type (FALSE or TRUE) and I know that C# will do the same.
So I change the types of these from string to boolean.
Now look at the field agent_id. This is a leftover from an earlier version of the mixdown agent with a design where the id of the mixdown agent was stored to the composition. This does not exist anymore and rightly so, so we remove that property from the class definition. Removing a property will not remove it from the text, but it will prevent the property from being deserialized. When we will deserialize the data and use this structure to store it in a tabular format, this field will not be there any more.
The next field is the mixtime. This field is the time that the composition was mixed down to an MP3 file, expressed as the number of seconds since January 1st 1970. I believe it is not used anymore today. Let’s look at the latest compositions in the playlist.
When we look at these compositions in the MySQL database we see the field is not used anymore.
Yes, gotcha! This is also a deprecated field. Let’s wipe it out from our class definition ass well then.
Then we have the project id. This we can be sure because of the SQL typing that it is in all cases an integer and no need for a string type.
Also the style icon is an integer type, but I think this field ended up here by accident. Let’s see if it is ever used.
This looks indeed like a bug. We will remove the field.
Then we get to the template_id. It is really important in the current database, but in the new database it will be replaced by the environment identifier that containt the template. So we will drop it in our deserialization data structure as well.
Then we have the mtime.
This means the time of modification, in other words the time the composition was most recently changed. It is expressed in unix time so an int type will do.
Then we have the user id
The user id refers to a table which I am not going to show here, but is has something to do with this:
The composition environment allows you to log in, but after a while it was discovered that having to create an account in order to retrieve your song was not the way to go, for the target group, which is children in the age between five and ten. We resorted to having no user accounts at all and remembering the name of your song. You can always change your song, but when the browser closes you have to search for it by name in the next session. Then when opening it, you can not overwrite it, but you will create a copy of that song. This appeared to work much better. Therefore I will not migrate the table with the users and I might as well drop the user_id field.
The next field is “pubtime”
This does not denote the time (seconds since January first, 1970) that you will go to the pub. It denoted the time that the composition was most recently published (in contrast with mtime which denotes the time the composition was most recently modified).
Since we are now working with a more simplified model (see above) we will also skip the field “pubtime”.
Then we have the field “description” and I will now also do a query to see how it is used because I think this is a bug as well.
Well, it is not a bug, but it is deprecated. Nowadays we have these descriptions in another field. We will find it in the environment. There it is now the textSlogan of the projectMaterial.
This means we will skip the description field as well …
Now we get to some fields that are quite important:
The content contains the actual composition (in XML, let’s keep it string for now). The published bool denotes if the user wants an MP3 for the playlist rendered as a result. The hasMP3 boolean will tell us if the MP3 was already created and confirmed by the mixdown agent and the name property is entered by the children themselves. The name enables them to retrieve the composition.
Then we have the “created” field … the moment that the composition was created. This field is actually not used so we will remove it. Then last but not least, the ID property … very practical to retrieve the composition by its number (or ID or Primary Key as they call it in database-land).
So here is our end result, it became a lot smaller! We could even simplify it more, but that would take us to lengthy discussions so let’s move on :-)
Below the “composition” property we have the “environment” which consists of “project rules” and “project material”
The projectRules consists of an array with repeatedly the same data structure visible. These data are used to construct the visual grid regarding its rendering and timing behavior related to concepts such as tempo and how this relates to the spacing of blocks over the pixels on the screen.
The class for this data structure covers it nicely, but I am hereby deciding to drop the id property (because it will have no use in how we will store it … these things are always tied to a specific environment and it makes no sense to store them in separate records … we will never make a selection in the database or relate the same concrete projectRule to another music set.
Also the description can go … it is more a kind of documentation but functionally it is not used in the app. So this one, we simplified as well.
Apart from the project rules, the environment also consists of project material. The project material starts with some straight forward properties that can be expressed as an integer (id) and some strings (name, textSlogan, background and type), but then an entire and somewhat strange tree unfolds. It is an associative array named “parts” with numerical values indexing objects. For example the key “1” points to an object (which also happens to have id “1”).
The algorithm does not recognize that this is a dictionary where an integer is the index of an object. Let’s see what the Visual Studio Special Paste Magic makes of it.
Then there are these classes named _1, _21, _31 and so forth. In them we find the properties that should be classified as a “part”. And then inside we find the clips, also associative arrays with numbers as an index. And because the algorithm does not recognize, it sees entirely different objects. It thinks the numbers are property names of classes it will name after these numbers. So that’s why you get this kind of stuff…
And all the individual clips (the music blocks) are regarded a class as well:
On the sunny side, it’s actually really simple. You have an environment that consists of parts (such as brass, wood, percussion) each consisting of clips (music phrases looking like blocks).
So let’s first clean up this mess. I will write by hand the classification of the parts dictionary.
And the part itself is also very simple … it contains a Dictionary where numbers point to clips.
Now let’s add some explanation to the class. The id is the identifier (the index number) of the part. Then the name is obviously printed above the part. The order determines which one is printed first from left to right. The width determines the horizontal space on the screen that the part can take, and the marginx and marginy determine how much space will go inbetween the clips. We only have to change the type of some of the values to number instead of string.
Now let’s look at the clip. The id is again the index or the identifier or primary key. The exitpoint is something special. It is a number used as the time in milliseconds, counted from the start of the MP3, where the MP3 will stop showing on the screen visually. The entrypoint is something similar. This enables for overlap from attack or reverb sounds. A good listener who will check out all the clips, will notice that sometimes you hear the clip sooner than the playhead is displayed over it, or you may hear it still although visually it is already displayed as if it would have stopped. This enables for a playful jigsaw puzzle effect that still sounds well.
The name is there, but it is more for moderators as it is not displayed on the screen. The tag IS displayed … you see it in the image, for example B1. This allows for making the puzzle sound well without having to guess.
Then you have the icon property, this points to an image that is displayed on the block (note that in the example picture it is the same for all clips in one part, but it need not be).
There is also an ficon property. When this property is present, this is the icon that shows when the clip is idle below the composition. And there is the ficonroll, this is activated when rolling over and deactivated when rolling out.
The order obviously gives an ordering to the clips within the group. The color determines the color of the block (note on the picture the subtle variations from a distribution of color intensity that is projected over the order, but this has happened server side so that the color value will be slightly different for each clip in this case.
The width and the height have some special properties … their usage is too complex to explain here.
With the types optimized it looks like this:
Then on to the next item, the config settings. This is simply a bunch of key-value pairs.
Again Visual Studio guesses each of these things is a property…
Let’s remove the imaginary class Conf_Override … and go to the top level to help out.
Then we have only one thing left and that’s the scroll items. It is actually a list of compositions but it contains way too many data. I happen to know only some of the properties are used. So we can go from this:
to this:
TESTING IF THE TEXT CAN BE PARSED TO AN OBJECT USING THE DESERIALIZATION DEFINITION
To do that, we create another project … a test project.
We get a special type of project, a Unit Test Project
The class already contains an empty test. We can write a simple test and run it.
The runner will actually run the code and present the successes and failures.
Let’s also see what happens when we have a failure.
Okay … now let’s see if we can load our extracted data.
It contains the data exactly like we get them in our Azure Function (remember?)
We will load this file into a string. Then we have the situation like we have in our Function. We can open a file and use the unit test runner to run the code in debug mode. It will soon hit our breakpoint and we can step through the code.
We can then see that the text variable actually contains our raw data.
To deserialize them let’s use a famous library that also works in Azure Functions: Newtonsoft JSON library. We have to right click our project, manage nuget packages and get the package.
Then we have to import the functionality on top of our unit test file.
And then we can use it like this … we have to tell it which kind of thing (which Class) we expect in the text …
Only thing it does not know what we mean with RootObject. We have to lay a reference to our definition project from our unit test project.
Click okay and the code is ready for testing. When we place the breakpoint, we can now view in our IDE if the right values landed in our object. But unfortunately we still get an error. Apparently we cannot convert “0” or “1” in string form to bool.
We should then try to convert it to int and map it to boolean later.
Let’s compile the code and try again …
WOW!!! See how beautiful these errors are!!!
So we have to go to position 17187 and we will find the clip 1499 of part 103 of the project material of the environment …. which is null and the code is not ready for that.
OKAY it is correct
Now this is okay … it can happen that this property is not set … this means that the icon on a clip, should inherit the height and the width exactly and unchanged, from the value of the clip it is projected on.
The solution here is to make it a nullable type. We can do it with a question mark.
Okay, this time we came through it without any errors …
However we should be aware that such errors may still occur and then we have to do something with it … we have to catch those errors and store the data that could not be processed …
Now let’s look at the data.
We can see that the ugly XML data are indeed where we expect them … we will have to deal with that still.
The rest of the fields look okay …
… well not entirely. The ID below here should not be a string but an int. This can be corrected immediately…
And after that I find another one where the id is classified as a string … my bad, but again we can correct it on the fly.
Also we see that the value of a project rule is always numeric so let’s implement that too.
And we find that the scroll order and id can be int instead of string.
One last run to see if everything now looks good …
yes! Let’s get to the next chapter …
TRANSFORMING THE XML CONTENTS OF THE COMPOSITION TO JSON
Now thanks to http://mightykudu.azurewebsites.net/ we can find the website http://xmltocsharp.azurewebsites.net/ where we can just paste our XML and then we get some code …
OOPS!!!
We probably have to remove all those newline things. Notepad++ to the rescue…
It looks like Notepad++ did it again!
But we still have invalid XML… probably because of all the escaped slashes and quotes. We can use notepad again to replace them as well.
This time it worked but let’s again do a check. First … create a new class file.
… and paste the output of the fantastic web site in it.
Now let’s paste our XML data in Visual Studio and check it side by side with the class file.
It’s ugly!
But we can select all and then do Edit -> Advanced -> Format selection …
And abra cadabra hokus pokus!
It is suddenly something we may try to understand. We can even use the minuses to make it more comprehensible.
We can see now that the composition consists of tracks.
This kind of matches reality. There are six rows of blocks.
Also the numbers of blocks match with the numEvents property. This seems a redundant attribute … but maybe it was a lazy way of programming to serialize the XML including some variables that were cached in the object in the RAM operated on by the program (so it won’t need to count all the blocks everytime it somehow would need this information). Still it stinks. However … when we uncollapse a track, what do we get?
We see the startTime and endTime but this time it looks to be not milliseconds but samples. There are 44,100 samples per second in this case (depending on the sample rate, but here it happens to be 44kHz).
Then there are some other properties … I wonder if they will all be necessary but I must admit I can’t remember this part because it is kind of complicated.
Anyhow … this is what it is. Tracks consisting of events that have some reference to a source (probably that’s a reference to the MP3 where the sound comes from for that block).
When we look at the class we can see it starts at the bottom and ends at the top.
Uncollapsing the Composition we see a list of tracks.
That matches exactly with what we see in the sample data.
When opening a track we can see it must have an id (numeric), a numEvents (numeric) and a list of Events.
The class indeed has that too.
We only have to turn the string types to int (integer number)
Then when we open the Event type we see what properties it has. It has some numeric properties that all have the string type which we will correct now … and it has a Source property of type Source.
This totally harmonizes with the way the data look.
The last class to inspect is obviously the Source class. It should have a property id.
Again we will make an int from the string …
and then we’re done.
Let’s now add it to our test project. To deserialize the XML data we can use the XML deserializer class. In order to support it, we have to add a reference to the right libraries. Visual Studio helps with that…
Then we have to wrap our string in a string reader. This may look overly complex but the deserializer uses a kind of streaming thing called a StreamReader and the StringReader is one of the implementations of it, that you can use for a string.
When we now run the test we should get an object that contains the deserialized composition. Let’s inspect it with care.
It looks like everything is there!
CREATE A NUGET PACKAGE FOR USAGE IN A FUNCTION APP
We have seen our deserialization at work in a test project. As explained before, we don’t want all this code in our Azure Function because we want to reuse it. So let’s create a NuGet package which will make for reusable code. We may then import it to our function.
To create the nuget package it will be very easy to work with a build server and a package feed on VSTS and host the code on VSTS as well.
First let’s create a new repository.
In short … first you create a project, then go to the code section of that project, drop down the repositories list and create a new repository.
You better take the Git option, make a nice name, add a README and choose the VisualStudio .gitignore (it makes sure all user specific workspace files won’t go into source control).
Then you will see the repository and you can Clone it using classic Git tools … copying the https link with the small copy button next to the repository URL or using the SSH url
Or what I am prefering now is, Clone in Visual Studio
Visual Studio will open and make sure you will log in
After it authorized you it will suggest a local path where the repository will be stored. You can change it and when you are happy you can click Clone (I am lazy so I won’t change it).
You will land in the folder view of the solution explorer … because there is no solution yet.
Let’s adapt the readme.
You can now test source control already. After you save the file you can go to the Team Explorer in Visual Studio and go to “Changes”. I will describe my changes with a Commit Message and then click Commit.
My change is now added to a locally kept ledger of changes but still has to be pushed to the remote repository that I want to use to build from.
After pressing “Sync” I can “push” the changes to the remote server.
The Visual Studio gives me a success message: successfully pushed to origin/master. But what I really would like to see is, that the code actually arrived at the repository hosted under VSTS. And of course …
Let’s go on and create a Visual Studio Solution to compile the code and publish it as a NuGet package. First we go to the folder where we stored the repository locally. In Visual Studio I will find it in the Solution Explorer with a handy right click to open the folder in Windows File Explorer.
You can see there are some hidden files. I like to make a setting under Folder Options so that I can see all files. Without doing that, the operating system will not show files and directories that start with a dot. All the Git recording of the different versions, the commit messages and many more things go in the .git directory. Also there are Visual Studio settings related to the project which go in the .vs folder. Then you have .gitignore where you can add files and directories that you want to have locally but not in source control.
Our next step is to copy the location of this directory.
And we go to new Project in Visual Studio.
And we will create a blank solution (Other Project Types / Visual Studio Solution)
Now our Solution Explorer shows the solution.
The team explorer / Changes shows us what happened on the file system.
Now I will copy the project we created earlier and paste it next to the solution file.
In order to view the project in the solution I can use the Add Existing Project menu option.
The solution starts to look like a dedicated build for the NuGet package.
I can build the solution and run the unit test. Of course the whole thing behaves as before. We can commit the changes and push them to our source control.
I can see that the changes ended on the source control VSTS server …. Also I see a curious button on the top right of the page!
In this case we will start with an Empty Process.
The build has some standard settings. Let’s just press Save and Queue and see what is the basic behavior.
Just go with the defaults, find that a build has been queued and click on it.
Beautiful! It just downloaded the sources and did nothing. We will add the following steps here.
- build the whole solution (the library and the test project)
- run the tests from the test project
- package the library as a NuGet package
- generate a version number for the NuGet package
- push the NuGet package to a NuGet server where we can download it from for usage in our Azure Function
But first I want to make sure that this will automatically happen every time I push new changes from Visual Studio. I will then grab a coffee and come back and find a new version of my library tested and ready for use in my projects.
To do that, we go to the settings of the build. In order to get there I have to go to my build definition and press Edit.
I can then go to the Triggers tab and click “enable continuous integration”
Now I click Save (but not Save and Queue because I want to trigger a build task by pushing code).
After pushing some code (adding some whitespace) and pushing it to the remote repository from Visual Studio, I can see that a build is triggered. To see this I have to go to the main page under the “Build” menu item.
Then I can see the build automatically start, after pushing new code. I can see there is a build of type “ikcomponeer-CI” in progress and the screen will after a while show a success mark. As it stands now, nothing can go wrong actually. Let’s take some more risk and go on with our plans.
The plan was to modify the build such that it will
- build the whole solution (the library and the test project)
- run the tests from the test project
- package the library as a NuGet package
- generate a version number for the NuGet package
- push the NuGet package to a NuGet server where we can download it from for usage in our Azure Function
To have it build our solution we can use a predefined step. First go to the build definition and use the Edit button to open it. Then you can add a step.
Now I am going to add a Visual Studio build step first … but I do recommend you read what is on the information button because there changes a lot. What I understood today is, for a solution, use the Visual Studio build step.
I just went with all the defaults, pressed Save and Queue and now I am looking at the running of the build.
WOW what a nice and human friendly error message.
There is a NuGet package referenced so we should add a task to do a nuget package restore before building.
Be sure to drag the added NuGet task to the first position so it will run before the build.
There are a number of things you can do but in this case we just need to Restore. For the rest let’s go with the defaults (of course I always read and try to understand what these defaults will do …)
After pressing “save and queue” this time it’s all green and happiness.
When we click the name of this build run, we see that there are lots of possible results yet not present.
There are No changes. No work. No test runs. No build code coverage. No deployments and also No artifacts. The artifacts are the actual results from the build, things that you could install somewhere. But we don’t want that … we want a package to be pushed to a location where we can download it into other software solutions so that we can use it.
Now let’s have the build server run our test project. There is a step for that too, turns out this is the Visual Studio Test step. As my project is named *.test, the defaults will work.
It looks like we now have some test result. Zero tests failed and one tests passed. This is not bad for a first time. You can strive for a huge coverage but on the other hand, a strategy more efficient can be, to add tests as you experience and reflect on how the code tends to break.
The next thing we want to do is, package the library as a NuGet package. I have a feeling we need the same type of NuGet task for that, but configure it differently.
We can choose the pack command this time.
We want to pack a specific project so we click the navigation button
and then we navigate to our library’s csproj file.
Also I chose to go with the semantic versioning option and add a symbols file for debugging purposes. If I wanted to shift to a major version I would have to do that manually but that’s the whole purpose of semantic versioning.
By doing this the NuGet package will be added to the drop folder I suspect. But let’s save and queue it and then see.
We ran another successful build but let’s click the nuget pack step from the logs, and see what it did.
Okay, this looks okay. It created a nuget package.
Now we should probably push it to a package feed, with the same build task, configured for the push command or something. So we go back to the build definition, click Edit and add another NuGet task at the end.
The defaults look good, it’s easy to push to my package feed and I have to do next to nothing to configure it.
However I would like to push to NuGet.org so that the availability of the package won’t be depending on me having to pay a Visual Studio license. There is an upload link on the NuGet site. Let’s try …
I am directed to a login where I can use my Microsoft account.
Now I am taken through a very concise registration process …
After the usual confirmation email I am logged into the site. It looks very good. But also I see an exclamation mark in a worrying color …
… it is two factor authentication. After having enabled it, I am good.
Now I can manually browse a package file but that’s not the kind of thing I can continue doing for the rest of the time.
Its not immediately clear what to do. Let’s go back to the configuration of our NuGet push task and try from there.
I can click “external nuget server” and then I will see a “Manage” link.
Clicking the “manage” link takes me to a section of the VSTS site where I can connect with services. There are some templates already. Nuget is one of the options under “New Service Endpoint”.
The text under the info button in the dialog tells me that I am going the right direction.
Clicking the “lean more” I learn that the connection name is just something independent for me to refer to this configuration … probably this text will be shown in dropdown menus on the platform whenever such an interface is an option. So I think of something nice and now I only need to fill the API key.
It makes me think of a menu option I saw on NuGet.org.
This looks like the thing we need!!! Of course the process is not too hard to follow (but I won’t show the key itself because as explained it needs to be a secret or else anyone can push any code to this trusted place).
This key probably needs to be filled in to the service definition in VSTS that we were working on. Let’s do it!
After pressing OK, we see it appear in a list of endpoints.
Going back to the Build Step Configuration for NuGet Push we must now refresh our service endpoints dropdown menu.
And we can now choose our NuGet service endpoint configuration!
I have a feeling that we’re done. Let’s save and queue and see what happens.
WOW it succeeded!! I am going to drink beer now :-)
Oh no not yet …. we actually wanted to deserialize our raw composition data loaded as text, in our Azure Function App … let’s do that too!
First inspect our result on NuGet.org
OKAY here it is!!
Really cool!!! And notice that I had to do next to nothing … It is now very easy to make reusable code. Nice!!
Look mum, I am on Nuget.org!
I hope my mama can see it too. But when opening an anonymous window I can see that the package is available to anyone. Great.
Besides I also want to expose the source code in my repository … let’s see if we can do that, too?
… turns out according to this post, it is only in a very limited preview as of now.
https://blogs.msdn.microsoft.com/devops/2018/04/27/vsts-public-projects-limited-preview/
Will the next code project be a GitHub hosted code base or Visual Studio??
Anyway let’s move on. In order to use this library in our Function App we have to go to the function where we want to use it.
If you remember we have to go to the main level of the function app.
There we find platform features.
And we find Kudu there.
There you have to open the debug console command shell (CMD or PowerShell what you like in this case)
You will now get a command line into your azure function host, in your browser. There you navigate to the directory where your function is installed, especially the one that you want to add the library packages to
Here you can click a plus sign and add a file named project.json
There is a pencil icon to edit the file.
I always have to google “include nuget packages in azure functions” to get to the same stackoverflow page and copy paste the example.
Only thing is we need to replace the example by our very own package.
After pressing Save we go to the function and we may see in the log stream how the package restore is now automatically downloading the package to our function app…
Beautiful!! But will it work?
We can now go to our test project where we have made it work on our own machine, and see if we can do the same thing to the text we earlier pulled from the ikcomponeer.php website.
copy — paste:
Upon pressing save we get some errors.
The first one tells us we need to install the NuGet package for JsonConvert as well. Let’s look in Visual Studio which one it was.
This can be added to the project.json file as well. After that, I had to do another few using statements in the beginning of the file so that the file understands which libraries we are using … only making them available in the packaging is not enough, you have to add the using directives to the start of the code file, the compiler needs to be sure what you are talking (coding) about.
The only error we have left:
This is the last thing we have to do and now the object should download from the server and deserialize in the Azure Function.
Now we can press the RUN button ….
Oh noes!!! There appeared an error!
It is even too long to take a picture of … but this is the full error.
2018-05-16T20:49:23.049 [Error] Exception while executing function: Functions.ManualTriggerCSharp1. mscorlib: Exception has been thrown by the target of an invocation. Newtonsoft.Json: Error converting value {null} to type 'System.Int32'. Path 'composition.mtime', line 1, position 10291. mscorlib: Null object cannot be converted to a value type.Oh how I love the total completeness of the clarity and information in this error. Hallelujah. Heaven be praised. Thanks God, you are so nice to give us such a good JSON deserialization library!!!
Apparently we have to make the mtime nullable as well, because in this composition (other than in our example) it is null. Let’s do that immediately.
Now we make a commit and a push to source control. Then we grab a drink while our build will run, it will push a new version of the package to source control and from there we will have to pull the latest version of the package, recompile the function and retry the code.
… now that’s what I call a successful build!
On NuGet.org the new version is there ….
We now have to copy the number over to our project.json file
… and then wait (this can take half a minute apparently) for the package to update, but then sure enough it happens.
And retry!
Okay, the function started and it completed. We can assume that the data were deserialized well.
Now that we have reached this milestone we can start to actually do something with the data. Also we have to think of a way to deal with errors. Let’s do that in the next blog post, hopefully next week!!
As a closing remark, I hope you have had fun in reading this, learned something that you can use, or maybe you are reading this as a form of documentation for the composition environment, which I would love because the goal is to have children play with music and when you can help with that, this would be fantastic.
Have a nice day.
