Control your Windows system remotely via voice
SEPIA tutorial: Connecting to a Mesh-Node using the client app and Teach-UI
This is a tutorial for SEPIA the open-source, digital voice-assistant. If you want to learn more about SEPIA please visit: https://sepia-framework.github.io/
Goal of the article is to explain how the SEPIA Mesh-Node server can be used to connect different devices to your local network and how to control these devices from your SEPIA app via custom voice-commands. Ultimately this is supposed to inspire you to experiment with the tools and build your own little voice-project :-).
The presented examples make use of the so called ‘RuntimePlugin’ integrated in the SEPIA Mesh-Node to run Windows shell commands that will open a website and start a mp3 playlist.
Requirements
- SEPIA-Home v2.2.0 or newer (6min installation guide)
- Windows PC with Java 8 (or higher)
- IP addresses or hostnames of SEPIA-Home and Windows PC. If you’re running everything on the same machine you can use ‘localhost’.
Get started
Before you continue please make sure that you can access your SEPIA-Home folder and your SEPIA client app. Login to your app (as described in the installation guide above).
1) Copy and run the SEPIA Mesh-Node
First thing we’ll do is to make a copy of the Mesh-Node server located in your SEPIA-Home folder. The SEPIA Mesh-Node will create a secure, remote access-point for your SEPIA core server (and thus your app) and is able to run programs on the host machine (Windows in our example).
Open the SEPIA-Home folder and look for another folder called sepia-mesh-nodes. Copy this folder to any location on your Windows system, e.g. the desktop (you can rename it if you like).
Open the folder (in Windows) and use the run.bat script to start the Mesh-Node server. Don’t close this window during the tutorial (it will stop the node). If you want to know in detail what your node is doing and if there where any complications during start-up (fingers crossed) open the log.out file in the same folder. If you see an error please double-check if Java is installed correctly, e.g. by using the Windows terminal: java -version. You can change the settings of the server as well by editing Settings/node.custom.properties (default properties-file used by run.bat). This might be handy if you need to use a different server port. By default your Mesh-Node will be password protected and run on port 20780.
2) Create your first custom voice-command
Next thing we’ll do is to use the Teach-UI integrated into your SEPIA app to create a voice command that will access the SEPIA Mesh-Node and trigger a shell command.
Inside your SEPIA app open the shortcuts menu (bottom left) and tap on the ‘teach’ button to open the Teach-UI. In the drop-down menu of ‘the assistant does …’ choose Call SEPIA Mesh-Node plugin and define a sentence that will trigger your command (‘When I say …’).
The next part might look a little bit more complicated (working on it ^^) but you can simply copy-paste everything to your window with minor changes. The field ‘Mesh-Node URL’ defines the location of your node. Remember the IP of your Windows system? Then enter it here like this: http://[my-win-IP]:20780 (assuming that you did not change the default port of your node in the properties file).
The field ‘Name of plugin’ refers to a plugin installed on your Mesh-Node. Currently the RuntimePlugin is the only pre-installed one (in SEPIA-Home v2.2.0). It is an interface to the system’s command-line and thus a very powerful tool that can basically run any program of the host computer (click here to check-out the code).
Because the plugin is very flexible we need to send an additional configuration string to tell it what to do. On the bottom of the Teach-UI tap ‘MORE OPTIONS’ to show the ‘JSON string sent to plugin’ field and copy-paste this string: {"command":["start", "", "http://example.com"]} . This will tell windows to start the default browser and open the given URL.
That’s it! I’ll leave it up to you to experiment with the ‘Success answer’ and ‘Fail answer’ fields but you’ll get some generic default answers even if you leave them blank. Press the “TEACH”-button at the bottom and you are ready to try your new custom voice-command. Just speak or type the sentence you used in the chat window.
In case you get an error or fail-answer check the log.out files in your Mesh-Node folder and your SEPIA-Home sepia-assist-server folder to get an idea what went wrong.
3) Call a Windows shortcut via voice-command
In this section we’ll create an additional voice-command to demonstrate the potential of the ‘RuntimePlugin’.
To call a Windows shortcut we’ll first create a new sub-folder in our Mesh-Node folder called Shortcuts. Now you can copy any shortcut you like to this folder. I’ll use a shortcut to an mp3-playlist called ‘DaftPunk’.
Use the following command in the Teach-UI to start the shortcut: {"command":["start", "", "Shortcuts/DaftPunk.lnk"]} . The path of the shortcut is relative to your Mesh-Node folder (note the .lnk ending! Every shortcut has it but usually you won’t see it). Fill out the remaining fields as seen in the previous example then try your new command. Windows will use the default program to start your shortcut, in my case the VLC media player will be used to open the list and play the songs :-).
Since you can create shortcuts for almost anything in Windows the possibilities are endless here ;-).
If you need more than just a set-and-forget command you can call complex programs as well, e.g. via batch-files and even use the returned result in your answer, but it might be more useful to write your own Mesh-Node plugin in this case or even think about a little Smart-Service using the SEPIA SDK.
More tutorials will be released soon. Stay up-to-date by following the SEPIA twitter feed and if you have any questions please click this link.
Have fun building voice-commands, cu around! :-)
