Implementing a Language Server…How Hard Can It Be?? — Part 1 (Introduction)
Well, it is really not that hard … :D
Okay… so let me get you into what I’m talking about here. If you have ever worked with an IDE for programming languages like Java, C# you have probably experience that how easy it is to write program with auto-completion feature ( some might know it as intellisense ) provided by the IDE which help you to type code faster and select suitable functions to complete your task, or find the usage of the particular function, or go to the implementation of a function just by clicking on it or rename a method safely without having to change everywhere it is used, by yourself. As most of the IDEs support multiple languages these language features are injected into the IDE via the language plugin which is written by the language developers or IDE developers (or whoever love that language).
So where is this so-called language server fit in?
If you are a developer (Obviously) you might have a favorite IDE that you always use. And it is really easy if you can develop software from different languages on top of the same IDE that you like. So various developers choose IDEs that they like and stick to it. Because of that language developers always trying to add language feature support for all most all the IDEs and that is hard because every IDE has their own way (Architecturally) of supporting languages features. So it is very difficult for the language developers to support every IDE because they need a big effort to develop language plugin for each. It will be really easy if they have a protocol for supporting language features which are supported by every IDE. So Microsoft came up with an idea of Language Server Protocol(LSP) when they developing the Visual Studio Code. Well for your information Language server is not a new concept it was there for a long time only it was not standardized. You can find more details about the language server in Protocol History. So the purpose of language server protocol is to save the language developer from the misery of writing different implementations to support the same language in various IDEs by developing just one implementation to support a language for all IDEs.
Now with this protocol more old and new languages have language servers that anyone can get and integrated into their own plugin for an IDE or an advanced text editor to give support to a language.
Now the question is how we implement this protocol to support a language … well, let me walk you through ways we can implement this protocol to support our own language.
How to implement language features of LSP?
So before getting to know how to implement let's look into how it works.
So there are three main components involved in the process of language server. Below picture (which I took from Microsoft's site for LSP) will explain how the process of language server works.
- Client — the development tool
- Transport/Communication — the JSON RPC
- Server — the Language server
So the client is the development tool, the IDE, which has the client to request and consume data from the Language Server which transported via JSON-RPC. So the language server protocol implementation is bound with JSON-RPC.
So in the image, you can see that there are processes of two types. Notifications and Request/Response. Notifications are the once send to perform an action that IDE handles. request/ response is used normally for performing language feature operations. So when the user opens a file which has the extension to activate your plugin IDE language server client will send a “textDocument/didOpen” notification to the Language server with the file information to perform an action that you implemented when opening a file. After that as to the actions you perform on your file, it will send a notification to fetch Diagnostics (Compilation errors in your code and etc) and will send a request to Completions, Hover information, go to definition and references and so on. Likewise, there are more language features and document management features available in LSP which you need to manage the complete development editor environment. You can find those features in LSP specification.
So how can you implement LSP for your own project?
well, there are two ways that I know of.
- You can implement it your self by looking at the LSP and JSON-RPC 2.0 specification.
- OR you can simply use LSP4J which is an implementation of LSP with the JSON-RPC 2.0 implementation by Eclipse. This is a Java implementation of the LSP.
The 1st DIY method is a bit hard because you have to do all by your self. So what I recommend is the second one. Because we have tried both ways when trying to implement Ballerina Language Server for the Ballerina Language and we learned that using LSP4J is much easier and productive than implementing our own.
How we implemented the Ballerina Language Server using LSP4J?
I will not be going into more technical level details because there will be a Part 2 of this article which will be going into the architecture level details of Ballerina Language server.
So LSP4J has the interfaces for LSP. What we simply have to do is implement them to support our language. There are three main classes. Language Server, TextDocumentService, Workspace Service. Language server is the initial class which initializes the language server for TextDocumentService which provides the language feature interface and Workspace service which provides the workspace feature interface. We have implemented above three classes for Ballerina Language server and you can find the implementations if you follow below-given links.
Ballerina Language Server — https://github.com/ballerina-platform/ballerina-lang/blob/master/language-server/modules/langserver-core/src/main/java/org/ballerinalang/langserver/BallerinaLanguageServer.java
Ballerina Text Document Service — https://github.com/ballerina-platform/ballerina-lang/blob/master/language-server/modules/langserver-core/src/main/java/org/ballerinalang/langserver/BallerinaTextDocumentService.java
Ballerina Workspace Service — https://github.com/ballerina-platform/ballerina-lang/blob/master/language-server/modules/langserver-core/src/main/java/org/ballerinalang/langserver/BallerinaWorkspaceService.java
There is one other class which is Language Server Client. This is used if you are to implement a client to consume the language server. Let's ignore this for now since we are targeting the implementation for the editors which support LSP.
The other main component is the communication. There are few concepts you need to learn when using LSP4J which are related to the JSON-RPC implementation. You can read about those core concepts here to understand more about how communication is handled in LSP4J. Also Related to the Language Server implementation in LSP4J there is a concept called Launcher. The launcher is what allows the client to connect to the language server. As to the LSP4J Launcher is written to get input and output stream as arguments so the IDE can write the request into the input stream and Language server can write the result into the output stream. They have used streams because then the language server can push notification to the client. The following link will take you to the Launcher implementation of Ballerina Language server.
How we connect language server to a plugin?
To connect language server to a plugin first you have to import the language server launcher or create a launcher inside your plugin and then pass two streams to the launcher and pass the launcher to the language client which will initialize the server.
Please follow the following URLs to see how language server is integrated into the VSCode plugin implementation for the Ballerina Language.
Launcher for language server — https://github.com/ballerina-platform/ballerina-lang/tree/master/distribution/zip/jballerina-tools/resources
Starting LS for the plugin — https://github.com/ballerina-platform/ballerina-lang/blob/master/tool-plugins/vscode/src/server/server.ts
Attach server to the client — https://github.com/ballerina-platform/ballerina-lang/blob/master/tool-plugins/vscode/src/core/extension.ts#L97
In Conclusion…
Language server protocol is a standardize way of for providing language features initialized by Microsoft which enable language developers to develop only one language server per language to provide language features such as code completion, hover provider, go to definition, find all references, and code refactoring which can be used across IDEs when developing plugin for the particular language. Also, we discussed how the language server works by understanding three main components, the client(IDE), Communication(JSON-RPC) and Server(Language server). Also, we discussed the ways to implement the LSP and found LSP4J which is an implementation of LSP from eclipse which made it easy to implement our own language server. So we discussed how to implement the three main classes of language server using the example of Ballerina language server implementation and how to integrate language server with an IDE which supports LSP using the Launcher we implemented using LSP4J.
So as you can see with the help of LSP4J it is pretty easy to implement your language server.
In part 2 of this article, I will be going into architectural and more implementation level details about how we implemented the Ballerina language server.
