1. Introduction to Deep Learning

2. Types of Neural Networks in Audio

3. State of the Art

Introduction to Deep Learning

Machine Learning V.S. Deep Learning:

Have you ever wondered about the differences between deep learning, machine learning, and artificial intelligence? In fact, they are not the same technology, they are both subsets of artificial intelligence. And Deep learning is also a subset of machine learning.

Machine learning is all about teaching systems to learn from experience rather than being explicitly programmed. These systems use algorithms to sift through massive amounts of data, finding patterns and connections to make smart decisions and predictions. It’s like the more data they see, the better they get at what they do.

Deep learning takes things a step further. It’s a type of machine learning but with a twist. Deep learning uses neural networks that have multiple layers. These layers help in handling complex tasks by reducing the need for extensive data pre-processing. This technique is popular in the audio field because it can analyse and process audio signals more effectively.

Deep Learning Neural Network Architecture:

As mentioned above, deep learning is a type of machine learning, which is structured as a neural network. The word “deep” refers to the multi-layered architecture of neural networks, which attempts to mimic the workings of our own brains.

Neural networks consist of interconnected artificial “neurons”, each of which plays a role in processing and analysing information. These neurons are organised into layers, each of which has a specific function. The first layer is known as the input layer. It is where the data enters the network. Next, the data goes into one or more hidden layers. These layers are like the “thinking” part of the network. Each neuron in the hidden layer receives inputs from the previous layer and uses mathematical operations to process them. As the data flows through the hidden layers, each subsequent layer builds upon the previous, extracting increasingly intricate features and patterns. Ultimately, the output layer presents the final result.

Types of Neural Network

CNN:

CNN stands for Convolutional Neural Network and is inspired by the visual processing mechanisms of the human brain. A convolutional neural network consists of a three-dimensional arrangement of neurons. The first layer is called the convolutional layer. Each neuron in the convolutional layer processes information from only a small part of the visual field. Input features are acquired in batches like a filter. Like the human brain, the network completes image processing by understanding the decomposed image and integrating the results of multiple calculations.

The propagation in a CNN is unidirectional, meaning information flows through the network in a sequential manner. One or more convolutional layers and fully connected layers are included to perform various tasks. Max-pooling and subsampling are applied in each convolutional layer in order to reduce the computational complexity. Finally, the fully connected layer connects these extracted features to the output layer, enabling classification or prediction based on the learned features.

In audio-related applications, the sound is usually transformed into a spectrogram as input to the CNN and analysed as an image. CNNs are often used in audio event detection and classification.

DNN:

DNNs refer to Deep Neural Networks and are characterised by their depth, indicating that they possess a greater number of hidden layers compared to CNNs.In the architecture of a DNN, each node is connected to all neurons in the subsequent layer, resulting in a fully connected neural network. These networks typically consist of multiple hidden layers between the input and output layers, typically containing three or more layers in total. It has both forward and backward propagation.

Since DNNs have more hidden layers and fully connected neural layers, this feature is able to capture higher-level audio features. Speech recognition, speech enhancement, and noise reduction are often implemented using DNNs. Like CNNs, DNNs also use spectrograms as input data for neural networks. When the objective involves modifying audio, the output of the DNN is often transformed back to the time domain to obtain the modified sound.

RNN:

RNNs refer to Recurrent Neural Networks and are specifically designed to process sequential data or data with a temporal component, such as time-series data, speech and text. The key feature of RNNs is the presence of recursive connections, which allow information to flow from one step to the next in a sequence. This recursive structure allows the network to maintain some form of memory, enabling it to retain information from earlier inputs and incorporate it into the processing of subsequent inputs.

The basic building block of an RNN is the recurrent neuron, also known as an RNN unit. It accepts input at each time step and combines it with the internal state of the previous time step to produce an output and update its internal state. This feedback loop forms the basis for capturing sequential dependencies.

There are different RNN variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), which address the gradient disappearance problem often encountered when training traditional RNNs. These variants introduce additional mechanisms to selectively retain and update the information in recursive connections.

The recursive connectivity of RNNs allows them to model time-varying dependencies and makes them suitable for tasks involving sequences, such as analysing time-varying effects: delay and reverb.

State of the Art

Wave-U-Net:

Wave-U-Net¹ is a deep learning architecture specifically designed for audio source separation, a task that involves separating a single audio source from a mixture of multiple sources. It was published in an article entitled “Wave-U-Net: a multiscale neural network for end-to-end audio source separation” in 2018.

Wave-U-Net is a one-dimensional version of the U-Net, a convolutional neural network architecture commonly used for image segmentation tasks. The “U” represents the symmetrical shape of the neural network structure.

The structure of the Wave-U-Net network consists of an encoder-decoder structure. The input signal is downsampled through the encoder until the data becomes the smallest and deepest dimension, then restored and separated from the source by upsampling through the decoder.

In addition, the network incorporates concatenations between the corresponding encoder and decoder layers, which combine the results of each encoder layer with data from the decoder layer to obtain both the high resolution of the encoder layer and the multiscale characteristics of the decoder layer.

Hybrid Decmus:

Hybrid Demucs² is a neural network model developed by Meta AI, designed for music source separation. Its primary objective is to separate individual sound sources, such as vocals, drums, bass, and other instruments, from a musical mixture.

The architecture of Hybrid Demucs builds upon the underlying structure of Wave-U-Net¹, employing symmetric coders and decoders as the fundamental framework. To enhance its source separation capabilities, additional frequency domain networks have been incorporated. These networks selectively attenuate or remove specific frequency components associated with different sound sources. Also, an LSTM(Long Short-Term Memory) network is added to the middle of the neural network to process longer audio sequences.

To achieve the isolation of sources from the mixture, Hybrid Demucs employs a technique known as time-frequency masks. This involves modifying time-domain waveforms by capturing features in the frequency domain. These masks aid in isolating the desired source from the mixture. Once the masked spectrogram is obtained, it is converted back to the time domain, resulting in a separate audio signal.

As of 2023, it has attained the highest rating in MUSDB18³, a prominent music dataset dedicated to source separation research.

GANSynth:

GANsynth⁴ is a generative model for audio synthesis proposed by Google that is built upon the powerful framework of Generative Adversarial Network (GAN). GAN has proven to be immensely successful in generating high-quality images, and GANsynth harnesses this methodology to generate audio waveforms of remarkable quality.

The core architecture of GANsynth consists of two neural networks: the Generative Model and the Discriminative Model. The Generative Model takes random noise as input and produces synthetic audio waveforms, while the Discriminative Model aims to identify between the generated audio and real audio samples. By learning to identify the disparities between the two, the Discriminative Model provides invaluable feedback to the Generative Model, enabling it to refine and enhance the quality and authenticity of synthesised audio.

During the training process, GANsynth employs an adversarial approach to optimize the generative and discriminative networks. The generative model strives to create synthetic audio that can deceive the discriminative model, while the discriminative model tries to accurately differentiate between real and synthetic audio. This adversarial training process encourages continuous improvement in the synthesis capabilities of the generative model.

A key technique employed by GANsynth is the utilisation of a two-level hierarchical structure. At the lower level, the generator operates in the time domain, synthesising audio waveforms. At the higher level, the generator operates within a learned frequency domain representation called the “time-frequency (TF) domain.” This TF domain representation significantly enhances the efficiency of audio signal generation. The research paper states that GANsynth achieves a staggering 50,000 times improvement in sound synthesis efficiency compared to the traditional WaveNet model.

Reference

1. Stoller, D., Ewert, S. and Dixon, S., 2018. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. arXiv preprint arXiv:1806.03185.

2. Rouard, S., Massa, F. & D ́efossez, A. (2023), Hybrid transformers for music source separation, in ‘ICASSP 23’.

3. Rafii, Z., Liutkus, A., St ̈oter, F.-R., Mimilakis, S. I. & Bittner, R. (2017), ‘The MUSDB18 corpus for music separation’.

4. Engel, J., Agrawal, K.K., Chen, S., Gulrajani, I., Donahue, C. and Roberts, A., 2019. Gansynth: Adversarial neural audio synthesis. arXiv preprint arXiv:1902.08710.

Guitar Mix Retrieval is my Master’s research project at the University of York. If you are interested in this topic, feel free to follow me or leave your comments below!

Outline

1. What is Mix Retrieval?

2. Related Research

3. Model Structure

4. Potential Applications

What is Mix Retrieval?

When it comes to music production, electric guitars don’t just rely on amplifiers only. Various effects are often applied to the guitar signal. Like those cool sounds that make the guitar go from smooth and mellow to rocking and intense. The guitar sound you hear on the album is mostly different from the raw signal that comes out of the guitar itself.

Mix retrieval is a technique for analysing mixed signals, and identifying all the processes during mixing. Guitar mix retrieval is all about digging deep into the signal chain that shapes a guitar’s tone. Imagine analysing the player’s pedal board, the post-production effects, and everything in between. Specifically, giving an AI model a post-production file will output the original signal and a list of effects. It is like converting a dish into ingredients and recipes.

Related Research

As mentioned above, the mixed signal contains a large amount of information. However, extracting all of this information using just one algorithm is a tough nut to crack. Related studies have divided Mix Retrieval into three main directions: effect recognition, parameter extraction, and de-mixing.

Effect Recognition:

Effect Recognition is all about identifying the different effects used in the guitar signal chain. Effects are usually categorised into non-linear processing effects, modulation effects, and ambience effects, and then further broken down into specific effects. In a German study¹ from 2010, a support vector machine (SVM) model was trained to recognize and classify 10 different effects with an impressive average accuracy of 90%. Similarly, a UK study² from 2020 explored the analysis of 13 classical guitar overdrive pedals using a convolutional neural network (CNN) model, achieving an average accuracy of 85%.

Parameter Extraction:

Parameter Extraction focuses on extracting the settings or parameters of the effects. These parameters have a significant impact on the sound produced. Once the effects are recognised, the next step is to extract their specific settings. In a German research project³ conducted in 2020, a CNN model was utilised to predict parameters such as distortion gain, tremolo frequency, and slapback delay time with a high confidence interval of 95%.

De-Mixing:

De-Mixing comes into play when we want to recover the original dry signal from the mix. This allows for remixing or further analysis of the guitar sound. The majority of research in this area has focused on de-reverberation and distortion recovery. In a study by Sony⁴, a U-net-based neural network model proved to be effective in converting distorted guitar sounds back into clean dry signals. Demonstration Site

Model Structure

As mentioned earlier, a single algorithm or AI model cannot fully achieve true Mix Retrieval. I propose a model structure to be validated by integrating the above research.

The features of the input signal are used as input data for the effects classifier and parameter extractor. A set of default effects chains is used as a reference to determine the closest mix to the input signal.

On the other hand, the input signal is simultaneously fed into a de-mixer to recover the dry signal. The output signal passes through a pre-defined chain of effects, applying parameters through an effects classifier and a parameter extractor, and then back to the input of the de-mixer as a reference signal for the model.

Potential Applications

Customised Auto-Mixing:

The output information from the effect classifier and parameter extractor can be considered as a mixing style. By inputting specific mixed signals, the output of the model is gradually averaged into a customised mixing style, determined by the input dataset. This result can be used in further auto-mixing algorithms.

DAW project file recovery:

The loss of DAW project files is one of the greatest concerns of the sound engineer. When files are lost, it is super hard to recreate either the mixes or the recorded tracks. Mix retrieval can go some way to “rescuing” lost files by importing completed mixed audio.

Dataset Generation:

Nowadays, most of the studies related to music technology, in particular to Music Information Retrieval (MIR), are difficult to develop due to the lack of datasets. Pure dry signals are relatively easy for MIR algorithms to analyse. If dry signals could be generated in large quantities by mix retrieval, researchers would not need to run between the studio and the laboratory and could concentrate on the algorithms.

Reference

1. Stein, M., Abeßer, J., Dittmar, C. and Schuller, G., 2010, May. Automatic detection of audio effects in guitar and bass recordings. In Audio Engineering Society Convention 128. Audio Engineering Society.

2. Comunità, M., Stowell, D. and Reiss, J.D., 2020. Guitar effects recognition and parameter estimation with convolutional neural networks. arXiv preprint arXiv:2012.03216.

3. Hinrichs, R., Gerkens, K. and Ostermann, J., 2022, April. Classification of Guitar Effects and Extraction of Their Parameter Settings from Instrument Mixes Using Convolutional Neural Networks. In Artificial Intelligence in Music, Sound, Art and Design: 11th International Conference, EvoMUSART 2022, Held as Part of EvoStar 2022, Madrid, Spain, April 20–22, 2022, Proceedings (pp. 101–116). Cham: Springer International Publishing.

4. Imort, J., Fabbro, G., Martínez-Ramírez, M.A., Uhlich, S., Koyama, Y. and Mitsufuji, Y., 2022, February. Distortion Audio Effects: Learning How to Recover the Clean Signal. In 23rd International Society for Music Information Retrieval Conference (ISMIR).

Outline

1. The History of Guitar Learning Tools

2. Music Information Retrieval

3. Guitar Learning Tools Using Machine Learning

The History of Guitar Learning Tools

Guitar learning tools have come a long way, evolving alongside technological advancements. Let’s take a look at how guitarists learned at different times:

60s-80s:
The 60s saw the rise of rock ’n’ roll, and the electric guitar became popular as a result of the genre. Musicians from this era often learned by imitating their favourite guitarists by listening to records or catching their performances on the radio.

80s-00s:
Advancements in technology during the late 20th century greatly impacted guitar learning. The invention of cassette tapes, followed by CDs and DVDs, gave guitarists access to instructional materials, backing tracks, and video lessons. Instructional books and magazines for guitarists show up. Popular publications like “Guitar Player” and “Guitar World” provided a wealth of tips, tutorials, and sheet music, including demonstration CDs or tapes, enriching the learning experience. Guitarists used these magazines to learn new techniques and expand their repertoire.

2000–2010:
The turn of the millennium brought the Internet revolution, leading to the emergence of online platforms, websites, and forums dedicated to guitar instruction. YouTube became a popular tool for guitarists, as they could access video lessons and performances from various guitar players. Additionally, guitar sheet music software like “Guitar Pro” gained popularity, providing tablature and allowing users to play, edit, and assign virtual instruments to each track.

2010-present:
With the widespread use of smartphones, guitar learning tools made their way onto mobile app platforms. Leading guitarists and educational media began offering online lessons, presenting comprehensive learning programs for beginners and advanced players.

Accompaniment and sheet music tools also advanced, enabling users to load songs from the cloud or their devices and access analysed notes, chords, and even backing tracks. These groundbreaking applications utilise Music Information Retrieval (MIR) technology.

Music Information Retrieval

Music Information Retrieval is a type of multimedia retrieval that focuses on retrieving music-related information, and it has all sorts of applications. Typical applications include song recognition, humming/singing retrieval, cover song retrieval, genre classification, music sentiment calculation, music recommendation, karaoke applications, backing track generation, automatic music scoring, music content annotation, artist identification, performance evaluation, song synthesis and conversion, intelligent composition, digital musical instruments, audio/music editing and production, etc. It is now a core technology in the field of music education tools.

Early MIR technology was studied with symbolic music such as MIDI. Due to its accurate pitch, time and other information, it soon developed more maturely. Subsequent research soon shifted to the study of audio signals, which became dramatically more difficult until machine learning techniques became popular. Today’s MIR techniques rely heavily on machine learning training to turn to sound into a database of learning tools by capturing pitch, rhythm, timbre and other information in audio.

Guitar Learning Tools Using Machine Learning

Backing track generator:

Unlike traditional generators that rely on a database of sheet music to produce accompaniment, deep learning for music generation takes a different approach. It uses real commercially available music as training data and puts it through a whole host of processes such as feature extraction, tagging, beat tracking, timbre analysis and instrument separation. The synthesised music sounds more realistic as if it were played by real musicians. Best of all, you can tweak it as much as you like. It can be customised in terms of duration, genre, mood, tempo, instruments and other parameters. You can adjust all these interesting parameters to make the music fit exactly what you are looking for.

The generator is even more powerful when combined with Natural Language Processing(NLP) technology. OpenAI and Google have launched their own AI music generator in 2022. Users can simply “chat” an ideal backing track to the AI model in the same way as they did with ChatGPT!

Chords and tabs generator:

Using machine learning to generate music sheets follows a similar process to traditional Music Information Retrieval (MIR) algorithms. Both involve feature extraction to analyze frequencies and notes. However, AI models seem to have an edge when it comes to accurately identifying the connections between notes and translating them into scores that guitarists can use for practice. Let’s take the Chord AI app as an example. This nifty app can take any audio file or web link and transform it into a music sheet that’s easy to read. The app comes equipped with some powerful features, including chord recognition, beat tracking, a chord dictionary, instrument separation, MIDI file exportation, and much more. The best thing is, all you need is a mobile device!

AI Guitar Lesson:

When ChatGPT was first introduced, users began to try it out to design guitar lessons. But just a few months later, AI guitar lesson apps popped up on the scene. They combine chatbots and AI MIR models to create a series of tailor-made guitar lessons for users. For instance, AI Music Lessons asks users to connect their own guitar to the sound card during the lesson in order to track users playing. Then, the AI models kick in and provide live feedback and progress tracking. It’s like having a virtual tutor right by your side, giving you tips and advice in real time!

1. The History of Amp Modelling

2. Types of Amp Modelling

3. Amp Modelling with Machine Learning

4. Future Trends in Amp Modelling

The History of Amp Modelling

The earliest amplifiers were basically just cranking up the electric signal from a guitar so it could be heard through a speaker. Guitarists quickly discovered that turning up the volume not only made it louder but also gave a cool overdrive effect. And they loved it, so they started cranking their amps for that distorted sound.

To keep the overdrive effect without blasting everyone’s eardrums, amp designers came up with a clever solution. They divided the amplifier into two parts: a pre-amp to control the tone and a power amp to boost the volume to a playable level. This way, players could get that awesome distorted sound without making the neighbours go crazy. But traditional tube amps were pricey and a pain to lug around, which got manufacturers thinking. They started exploring amp modelling as a solution for different performance and recording scenarios. Enter amp simulators, the first products to embrace amp modelling. They were way more affordable and practical than buying multiple guitar amps, giving players access to a wide range of amp tones all in one package. No wonder they became so popular!

But there’s more to achieving that regular guitar tone than just the amplifier itself. When it comes to shaping your guitar tone, it’s not just about the amp. There’s a whole chain of elements involved:

Guitar — Guitar Amplifier — Guitar Cab — Microphone — Playback System

Any of the above processes can greatly affect the final tone that reaches the listener’s ears. That’s why modern amp simulators are stepping up their game. They’re not just focused on simulating the guitar amplifier anymore. They’re starting to incorporate all the different components that contribute to the overall tone. They want to capture the essence of the entire signal chain, ensuring that you get the most authentic and satisfying tone possible. So it’s not just about the amp anymore — it’s about recreating the complete tone experience.

Types of Amp Modelling

Amp modelling comes in two flavours: White-box modelling and Black-box modelling.

White-box modelling involves studying the amp’s internal circuitry and replicating the changes in the electronic signal at each stage. In other words, it tries to mimic the amp’s behaviour in detail. White-box simulation requires all the details about the amp’s circuitry. White box modelling can be done in both analogue circuits and digital algorithms.

On the other hand, Black-box modelling focuses on reproducing the input signal to the actual output signal of the amp. It does this by comparing the changes in the input and output signals algorithmically. The beauty of Black-box simulation is not necessary to know the exact structure of the real circuit, only the result of the signal passing through the amp. Black box models are mostly made up of AI technology.

Pros And Cons:

White-box modelling is all about simulating a real circuit. This means it can mimic the dynamics of a genuine amplifier, including how each knob affects the sound. So when you tweak those knobs, the simulated amp responds accordingly. However, it might not capture every tiny nuance of the dynamic changes as accurately as the real thing. It’s pretty close, but not spot on.

On the flip side, the Black-box modelling approach perfectly captures the sound of the amp at specific settings. It’s like freezing the amp’s sound in time, giving you an accurate representation of its static state. But here’s the catch: when you start twisting those knobs and making adjustments, the accuracy starts to waver. It won’t quite match the intricacies of the real amp when you’re dialling things in.

Amp Modelling with Machine Learning

Amp modelling using machine learning is what we call black box modelling, as mentioned above. These models train neural networks by analysing a bunch of dry and amped signals.

Picture the neural network structure like a web of connections in the human brain, with layers upon layers of neurons. During the model training process, it’s like the network is on a quest to find patterns and relationships between the incoming and outgoing signals. They crunch through a massive dataset, establishing correlations between the different neurons. And the end result? A magical black box that takes your input signal and transforms it into an amped version.

Common AI amp model products on the market offer a pre-trained model for users to replicate the desired tone. These models have already been trained to a certain level. All users need to do is provide both the dry signal and the amped signal. The pre-trained model learns the ins and outs of the desired tone, capturing all the nuances and effects that the amp has on the original signal.

One of the most iconic AI amp modelling products in the market is the Kemper-Amp. The folks over at Kemper refer to their training process as “profiling.” Basically, Kemper-Amp turns trained models into profiles that represent different amp tones. These profiles are stored in the device, allowing users to quickly switch between various amp profiles. Since every user has different gear setup, the reproduced tones can vary greatly. Kemper-Amp profiles are widely shared and even sold online.

The technical details of how the Kemper-Amp works aren’t publicly disclosed. We can only make educated guesses based on the profiling process. However, other companies are catching on, and some have already released products with similar profiling capabilities. In fact, there are some GitHub repositories that have developed similar features.

GitHub repositories : neural-amp-modeler / PedalNetRT / AIDA-X

Future Trends in Amp Modelling

AI amp modelling has been around for quite a while now, ever since the Kemper shook things up over a decade ago. And it’s not just Kemper, other brands like Mooer and Neural DSP have added AI amp modelling features to their own products as well. In the future, we might see even more brands follow suit, especially those who have hit the limits of their original digital amp simulations.

But here’s the thing: once manufacturers invest in this technology, they need to consider factors like cost, pricing, and target audience. Do people really need all these fancy features? Are they carrying thousands of amp profiles wherever they go? These are questions that manufacturers will have to ponder.

Let’s take a closer look at the AI amp modelling products available today. The best and most popular ones are sold as hardware units with built-in amp modelling functionality. This has inadvertently created a bit of a monopoly in the market. Studio folks who are used to using software plugins for their sound now have to invest in hardware to get the advantages of amp profiles online community. It’s a bit of a bummer for those who prefer the convenience of plugins.

As more developers get their hands dirty with AI amp modelling, the trend is likely to move towards standardizing file formats. Just like virtual sound libraries and MIDI formats have become more unified, we might see a similar progression with AI amp profiles. This would make it easier for users to share and use different profiles across various platforms and devices.

And here’s an exciting thought: AI modelling technology can go beyond just amps. It can be extended to other types of audio processors like compressors, delays, overdrives, and more. Imagine replicating all these common audio processors using AI models. It’s like a whole new world of possibilities opening up. Plus, digitizing effects in this way could potentially make hi-fi mixing much more affordable.

However, it’s important to note that AI modelling still relies on the signals of physical equipment to train itself. So, if manufacturers stop releasing new amps due to price wars or other factors, it begs the question: Will we eventually run out of new sounds to play with?

To export the EXS24 library and load it into AudioKit, the following steps are required:

1. Choose a sample library from Logic pro’s Sampler plugin

If you would like to make your own sample library from Sampler, please check out the official AudioKit tutorial below.

2. Create a “Sounds” folder in your Xcode project

3. Export a copy of the sample library to the Sounds folder you’ve just created

Note that the AudioKit loader only accepts sound libraries under “/Sounds/”, “/Sampler Files/” and “/Apple Loops/” paths. You can find more details in the AudioKit lower layer AUSampler documentation.

4. Make an instance of AppleSampler class and load the exs24 library via the path

import AudioKit
var instrument = AppleSampler()

try? instrument.loadEXS24("Sounds/myLibrary")

5. Design MIDI events

• Assign a note directly to the sample library

instrument.play(noteNumber: 60, velocity: 127, channel: 0)

instrument.stop(noteNumber: 60, channel: 0)

or

• Create a sequencer to trigger the sample library

import AudioKit

class mySequencer{
    var instrument = AppleSampler()
    var sequencer = AppleSequencer()
    var midiCallback = MIDICallbackInstrument()
    init() {
        midiCallback.callback = { status, note, velocity in
            if status == 144 { //Note On
                self.instrument.play(noteNumber: note, velocity: velocity, channel: 0)
            }
            else if status == 128 { //Note Off
                self.instrument.stop(noteNumber: note, channel: 0)
            }
        }
        try? instrument.loadEXS24("Sounds/myLibrary")
        track1 = sequencer.newTrack()
        track1.setMIDIOutput(midiCallback.midiIn)
        sequencer.setLength(Duration(beats: 4))
        sequencer.enableLooping()
    }

  open func play(){
    sequencer.play()
  }
  
  open func stop(){
    sequencer.stop()
  }

  open func addNote(note: Int8, velocity: Int, position: Double, length: Double){
    sequencer.tracks.first?.add(
      noteNumber: note,
      velocity: velocity,
      position: Duration(beats: position)),
      duration: Duration(beats: length))
  }
}

1. Introduction of Micro:bit

2. Why Micro:bit ?

3. Desktop dev environment set-up

4. Build C++ in Micro:bit

Introduction of Micro:bit

Micro:bit is a programming education kit developed by the BBC in 2016¹. Built-in motion sensors and a 5x5 LED matrix are the most impressive features. Like the Raspberry Pi, Micro:bit can be found in various educational institutions.

The second version was released in 2020 and focused on upgrading the processor from the nRF51822 (M0) to the nRF52833(M4). The processor clock, Flash memory and RAM size have all changed significantly. My only complaint is that a product released in 2020 is still using the micro USB port.

Whatever the version, most users choose to develop their firmware on the MakeCode platform developed by Microsoft.

If you are entirely new to embedded development, I recommend you start by browsing the Tutorials for this platform. But if you would like to explore more of Micro:bit’s possibilities, read on

Why Micro:bit ?

For the new embedded developers, the most painful part of development is reading the lengthy peripheral and middleware configuration documentation. Any misunderstanding of the functions may lead to unexpected errors at runtime and is difficult to debug. I experienced similar problems when training software engineers to learn embedded systems in the industry.

In order to lower the barrier of usage for developers and students, the University of Lancaster has created a framework called CODAL² (Component-Oriented Device Abstraction Layer).

CODAL pre-compiled all the processor drivers into a bin file. With the online compiler provided by MakeCode, the logic written by users can be mapped to the static bin and re-generated to a completed bin file to upload. And MakeCode uses the Typescript language to lock all the data types, avoiding the problem of incorrect types in run time. This excellent framework created an easy-to-develop environment for beginners.

Desktop dev environment set-up

For experienced software developers, the graphical interface in MakeCode may not be the best choice for development.

To build the Microbit dev environment on the desktop, the following steps are required:

1. Install Node.js

• windows

winget install OpenJS.NodeJS.LTS

• macOS

curl "https://nodejs.org/dist/latest/node-${VERSION:-$(wget -qO- https://nodejs.org/dist/latest/ | sed -nE 's|.*>node-(.*)\.pkg</a>.*|\1|p')}.pkg" > "$HOME/Downloads/node-latest.pkg" && sudo installer -store -pkg "$HOME/Downloads/node-latest.pkg" -target "/"2.

2. Install the MakeCode command line tool (add sudo on macOS)

npm install -g pxt

3. Add a workspace and initialise the project

# create a new folder for your project
mkdir workspace
cd workspace

# start the project set-up
pxt target microbit
pxt init
pxt install

4. Build your code on main.ts

5. Commands

Run pxt help for the list of all commands. The following list of links contains more info on specific commands.

• build, builds the current project
• deploy, builds and deploy the current project
• console, monitors console.log output (ex: serial.write)
• bump, publish your code, increment the version and push to GitHub
• serve, run local server
• login, store a GitHub token
• pyconv, convert MicroPython code into Static TypeScript.

Build C++ in Micro:bit

If you are not happy with MakeCode’s built-in functions, you can extend Micro:bit functionality by rebuilding static libraries. (such as the standard library for C, or if you want to make it run faster, you can write in Assembly).

In addition to importing the function that you would like to add, you also need to give it a corresponding Typescript mapping (this is equivalent to adding a new block to the MakeCode interface).

CODAL provides Yotta (only available in V1) and docker environment to recompile the base bin.

Reference

1. Wikipedia contributors, “Micro Bit,” Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/w/index.php?title=Micro_Bit&oldid=1126166946

2. J. Devine, J. Finney, P. de Halleux, M. Moskal, T. Ball, and S. Hodges, “Makecode and codal: intuitive and efficient embedded systems programming for education,” ACM SIGPLAN Notices, vol. 53, no. 6, pp. 19–30, 2018.