A Practical Guide for Language Server Protocol

Behind the screens of a programming language — part 2

Overview

The Language Server Protocol(LSP) developed by Microsoft, is a protocol that standardizes the communication between language tooling and development tools. This article is a continuation of the practical aspects of my previous blog in which I have discussed the basics of the LSP.

The following sections showcase how to implement a Language Server(LS) and an LS client in the simplest form.

I have chosen the Ballerina programming language to develop the Language Server and LS Client for. Find more about ballerina on ballerina.io

Please find the source code of the following guide on https://github.com/malinthar/lsp-tutorial.

Developing the Ballerina Language Server

The language Server(LS) is the entity in the LSP which provides a set of agreed services/procedures to LS Clients. To accommodate the LSP communication, the LS has to receive and send messages through a particular transport. The messages should be properly read and responses should be created adhering to the LSP. Implementing the communication layer by yourself can be cumbersome for a developer. Therefore, multiple open-source reusable plugins/packages which seamlessly handle the above complexities of the communication layer were introduced by vendors. If an LS developer uses the aforementioned package he just has to write the application logic (application layer) instead of focusing on the communication and protocol specifics. However, there can be cases where you need your own LSP implementation. In that case, you just have to follow the LSP specification in the implementation.

We are going to use the Language Server Protocol For Java(LSP4J) by Eclipse for implementing the Language Server.

First, we need the following setup.

• JDK 11 or above : https://www.oracle.com/java/technologies/downloads/
• Maven 3.5.4 or above: https://maven.apache.org/download.cgi

The lsp4j library

We should have a proper knowledge of the following interfaces before implementing the LS.

LSP4J interfaces

LSP4J provides us with a nice structure to implement the language server. It provides an interface called LanguageServer that should be realized when implementing a concrete LS. According to LSP4J, a Language Server instance must have two services named TextDocumentService and WorkspaceService. TextDocumentService represents LS capabilities relevant to a particular text document(source code files such as .bal, .java, etc. file). In our case, the text document is .bal representing Ballerina text documents. The Workspace capabilities are relevant to the workspace/project that the user is currently working on. It contains capabilities related to file renaming, deleting, folder structure changes, etc. As depicted in the above diagram, under these two services we have a set of methods that must be implemented or overridden depending on the set of capabilities that we are going to support.

LanguageClientAware interface should be realized by the concrete language server if it should have the knowledge(be aware of) the Language Client it is communicating with. LanguageClient interface is a representation of the LS client or a remote proxy through which LS can communicate with the LS client. Essentially, the LS can access the Language Client instance and invoke its methods using this interface. Now, Let’s move to the interesting part.

Implementation

First, we need to create a Maven project using an IDE such as IntelliJ IDEA. Next, add the LSP4J dependency to the pom.xml file.

pom.xml

Refer to https://mvnrepository.com/artifact/org.eclipse.lsp4j/org.eclipse.lsp4j. I’m going with the latest version(0.12.0) of LSP4J. This version of LSP4J supports Language Server Protocol Version 3.16. You can find the LSP and LSP4J compatibility table here.

TextDocumentService

As depicted above Text Document Service is an interface that we should implement as follows.

TextDocumentService implementation

In our concrete class, we must implement didOpen, didSave, didClose and didChange methods as I have discussed in my previous article. Though I have implemented them, I have not added any logic in these methods. Instead, just acknowledge the reception of the procedure invocation.

I have overridden the logic for completion service which is invoked when a client sends an auto-completion request to which the LS should respond with a list of completion items.

WorkspaceService

WorkspaceService implementation

LanguageServer

Ballerina Language Server

The Language Server Launcher

This is a very important step that should be understood before implementing an LS. The Language Server is transport protocol agnostic. That means the LS strictly adheres to the LSP and does not know about the transport layer. Nevertheless, there should be some component that is aware of the communication layer. This is where the LS Launcher comes into the picture.

Due to this transport-independent nature, an LS can be running locally or in the cloud, opening up a new set of capabilities for language tooling. Though we usually run a language server in the local machine as a separate process spawned by the LS client, it’s also possible to have the LS in the cloud while you develop on your laptop. Refer to github.dev and vscode web versions as examples.

As discussed above, the Language Server should be launched on-demand. Usually, the Launcher does this task.

Take a look at the LSPLauncher provided by lsp4j.

LSPLauncher in lsp4j

As depicted in the code snippet above, we can create a Language Server Launcher using the above method which takes in a input stream(in) and output stream(out) as arguments.

What are input and output streams?

These are the message transports used for communication. The LS listens to incoming messages via Input stream(in), and sends the outgoing messages via output stream(out).

There is a multitude of options that we can use for the in/out streams. The choice of transport depends on the ability of the client to use it. You can host the LS in a remote server and connect via WebSocket or run LS locally and connect via stdin/stdout or web sockets. Take a look at the transports supported by the vscode-languageclient.

Note: In java, System.in and System.out stands for Standard I/O.

We are going to use Standard I/O as the input/output streams for LSP communication in our case. The LS Client will write to the standard input of the java process that is spawned by it.

BallerinaLanguageServerLauncher

However, there is one downside of using standard I/O. We can not use regular logging mechanisms such as System.out.print(), as those would write to Standard output compromising the flawless communication between LS and LS Client. As a solution, we can have a separate log stream to create logs at the LS client side. There is a further explanation in the Language Client section of this article regarding this. An additional solution would be to use a different transport such as TCP sockets. Take a look at the TCP launcher implementation here.

Now, Let’s assemble our application into an executable as follows. The language client will use a command to start the java process using this executable. Therefore, I have packed the dependencies into an uber jar and placed it in a directory called language_server_lib in the base directory.

pom.xml

Developing the Ballerina Language Server Client

In the previous section, we developed the Language Server and created an executable jar that can be executed by the LS client. In this section, we are going to develop the Language Client using the VSCode extension API and vscode-languageclient npm package.

Following are required to get started with the development.

1. VSCode : https://code.visualstudio.com/
2. Node : nodejs.org
3. JDK 8 or above : https://www.oracle.com/java/technologies/downloads/
4. Maven 3.5.4 or above: https://maven.apache.org/download.cgi

You can use your preferred development tool for developing the language server. I’m going to use IntelliJ IDEA.

Writing the language client is the easy part. We need to create an extension first. Then we’ll make it an LS Client. We can use the Yeoman scaffolding tool to create the extension. Please refer to my previous article for a detailed guide on how to create a VSCode extension. I’m just going to put the steps that I followed here for your reference.

1. Install Yeoman scaffolding tool

npm install -g yo generator-code

2. Generate a project

yo code

Generating a VSCode extension

3. Configuring the package.json

We need to configure the contributes section package.json generated by the tool as follows for the extension to activate when ballerina source files are opened on the editor.

Contributes section in package.json

You can quickly test whether the extension works by debugging the extension using the provided debug config. This config launches the extension development host with our extension.

Debugging the extension

Let’s run the command we have in the contributes section to check of the extension works. First create a file with .bal extension(eg : sample.bal). Then, get the the command palette (hit cmd +shift + p) and search for a command titled Hello world. Then, run the command and check if you get a Hello world message at the bottom of the screen.

Testing the extension

4. Make the extension a LS client

extension.ts file in the src directory

The above file is the main source of the extension. When a file of .bal extension is focused/opened in the VSCode editor, the Extension API of VSCode invokes the activate function depicted above. This is the entry point of our extension. Now that we have created our extension, it is time to make it an LS client. The code structure and the implementation that I follow here might not be the ideal(best practice) as I’m trying to simplify things as much as possible.

1. Add the vscode-languageclient npm dependency using the following command

npm install vscode-languageclient --save

2. Create a directory named core in the src directory. This directory consists of the LS client core. In this directory add a file named extension.ts and create the BallerinaExtension class as follows.

src/core/extension.ts file

In the second step above, we have used the APIs provided by the vscode-languageclient package in order to create a Language Client instance and bind and initialize a Language Server process. The init method is supposed to be called when the extension is activated. The serverOptions object specifies how to initialize the server using a command. In our case, the Language Server is a java executable. Therefore, I have specified how to initialize the LS using the relevant command and command options. You can replace the value of the project home constant according to your cloned directory. In order to run this command you have to have JAVA_HOME set as well.

After specifying the serverOptions, I have specified the client options. The first property, documentSelector specifies the file scheme and language id as specified in the package.json of the project. Next, the outputChannel property specifies the log stream to which the Language Server can log messages. And the final property, specifies the log level. Further, you can set client capabilities as well. For this tutorial I’m going to skip that and use the default capabilities for the client. In a future article I will discuss about the capabilities in deep. Let’s keep it simple for this article.

Ultimately, I have initialized the language client and pushed the disposable into the extension context.

3. Change the activate function in the extension.ts file as follows to use the above extension instance in the activate function as follows.

src/extension.ts file

Now try to debug the extension as we previously did. If everything is fne, you will get the following message popping at the bottom of the editor when you open a Ballerina file.

Initialization notification

Further you can see the logs by the Language Server in the Ballerina output by selecting the Ballerina output channel in the output console. Try to do a couple of text edits and see how the logs are printed on the console.

Output console

Let’s check whether we get the completion item we added in the LS. Try cmd + space anywhere in the file and check if you get our test completion item as follows.

Completion test

If you want to look at the messages in between the server and the client, you can use the following option in the settings.json file of the editor to enable the verbose mode. By enabling this, you can see the content part of the JSON RPC messages communicated, as depicted in the following image.

"ballerina-vscode-lsclient.trace.server": "verbose"

To open the settings.json file use cmd + shift + p and search for settings.

Initialize request from the client to server

4. Packaging the extension

Refer to my previous article for a more detailed explanation on how to package and publish your extension.

What’s next?

This article provides a comprehensive yet gentle practical guide for getting started with the LSP. The next step is to discuss the application logic in the Language Server and develop a more advanced Language Server. Further, improving the Language Server Client is another avenue to focus. Please refer to the following sources for additional details.

Source code for the above tutorial:

https://github.com/malinthar/lsp-tutorial

A bit more comprehensive implementation:

https://github.com/lsp-and-implementation/language-server

Ballerina Language Server open source project

https://github.com/ballerina-platform/ballerina-lang/tree/master/language-server

A good read on Language Servers.

https://blogs.itemis.com/en/a-birds-view-on-language-servers

Special Thanks to Imesha Sudasingha & Nadeeshaan Gunasinghe.

Thank you for reading!