Human-Robot Guitar Duo

Guitar test with the hand

Introduction

During the 1st semester in M1 EdTech studies in the class of Learning by doing at CRI-Centre de Recherche Interdisciplinaire in Paris i started working on a project that aims to make a Human-Robot guitar duo.

One of the subjects that i found very interesting during the course and decided to explore was synesthesia projects.

I decided to work on transferring human mechanical movement to electronic mechanical movement, in order to create a kind of dialog between the movements.

In the class i discovered many projects dealing with synesthesia and was interested to work on a project that could explore this through robotics.

I will show here the First phase of the project and continue until the finalisation of the project in other posts.

Concerning the Idea

The Second left hand project concerns the idea of creating two musical partitions which are embedded into one.

Arduino,controller,and battery setup

The hands full setup

The idea came from a compostional point of view, where we would have one two several compositions that are embedded within each other a kind of “inceptive orchestration”.

With this idea we have the ability to play a musical piece and generate the accompaniment in more than one instrument via robots.

The purpose would serve musical education and as a symbolic example of the mechanical process of learning music and would enable closer specific study of the movement aspects in a performance.

How does it work?

One set of musical material generates another one as its accompaniment.

I write the first musical material in paper for guitar solo that the human guitarist plays, and the robotic “second left hand” responds with different musical material that is the actual coded accompaniment of the first musical material.

First stave — Human guitarist, Second stave — The corresponding robot hand

In the guitars neck there will be a controller which controls the movement of the robotic hand that is mounted in another guitar.

In this way when the guitarist plays in the first guitar the “second left hand”(that is the robotic hand) responds, and moves.

The movement is programed in a way that when the guitarist plays , the controllers signal to the hand gives patterns that are different for playing other notes than he is playing , so that the accompaniment of the robotic hand is different in sound,rhythm,dynamic etc.

There is also a set of plucks mounted in the right side where the strings are plucked and they pluck the strings in corresponding order to the notes that are being pressed in the left hand.

The project consists several phases of construction, programming and testing.

I direct and monitor the process while working in different time periods with 3 separate teams of programming students from Epita and Descartes.

The phases consist of :

First phase — Finished

EPITA team that worked on the first phase of the project

• the 3d printing of the robotic hand with all its pieces
• Testing the hand movement via midi signal from the computer

Testing the Hand in the Lab with the EPITA team

Second phase — In Process

• Creating the specific midi controller that is mounted in the neck
• Coding the specific movement based on the controller
• Creating the plucking system
• Coding the movement of the plucking system in line with the left hand

Model finger to help analyse the movement possibilities of the finger

For the link between the hand and the controller which is to be placed in the neck of the guitar we will use a midi interface with a specific matrix that is mounted inside the neck and is of a flat surface.

Then we proceed by coding each button on the controller with a specific movement of the hand that results to a specific note in the other guitar.

Obviously the idea of the robotic hand accompanying the guitar player with different musical material could have been achieved by installing only one button which would trigger a whole set of movements in the guitar that the robotic hand would play from the beginning to end, but the idea here is to create and also add a performance aspect which is more natural where the guitarist is not following the robots pattern but is actually playing with it together.

This means that the time between each button on the guitar would be left to the moment and would be flexible which would give the piece the idea of a more natural harmonic performance.

We are currently in the 2nd phase and will soon have the finished project where we can show a working prototype.