Textual and graphical languages for the cloud era

TypeScript has become the de-facto standard for cloud-based developer tools. The Eclipse Cloud DevTools Working Group features several projects that are largely based on TypeScript, such as the web IDE framework Eclipse Theia and the diagramming framework Eclipse Sprotty. The obvious advantage of this technology stack is the direct support for running in a web browser, providing immediate responsiveness and relatively low network traffic. Even when the target is a desktop app, TypeScript-based tools can be packaged with the Electron platform to build a self-contained application. Apart from these fundamental benefits, the TypeScript language itself has a very powerful type system and has proven to work well for large-scale software solutions.

Xtext and Sprotty: a suboptimal relation

In a presentation at EclipseCon 2018, Jan Köhnlein and I showed an architecture for a domain-specific language (DSL) with automatically generated graphical views in a web IDE (see the recording on YouTube). The example featured a state machine DSL with rich text editor support, side-by-side with a graphical representation of the states and transitions that are specified in text.

The underlying frameworks in that architectural solution are Eclipse Xtext for the DSL, Sprotty for the diagrams, and Theia for the web IDE. These technologies work together very well and are still a good choice today, even four years after the aforementioned presentation. However, there is a flaw that puts the whole solution at risk: the language server is based on Java, while the rest of the application is written in TypeScript. While the resulting software components can communicate via the JSON-based Language Server Protocol, the development environments and execution runtimes are totally different.

For example, the Xtext language server requires a Java Development Toolkit as well as Maven or Gradle as build tools. It uses the Maven dependency system and it is bound to the Eclipse IDE (Xtext does not support other IDEs for development). In contrast, the TypeScript-based parts depend on Node.js, Yarn, Webpack, React and other development tools from the web toolstack. Unsurprisingly, the best development experience for TypeScript is delivered by VS Code (both projects are led by Microsoft). So we are left with two incompatible development environments, and the burden is carried by the involved engineers. The result is a potentially high cost of development and maintenance of the application.

Langium: a new language toolkit

Motivated by the need for a coherent technology stack, our team at TypeFox initiated a new toolkit for textual languages named Langium. The idea was to keep the concepts and features that made Xtext successful, and to lift them onto a new technological foundation by relying completely on TypeScript. The conceptual similarity makes it easy for those who are familiar with Xtext to migrate to Langium:

• A grammar language is provided to specify the syntax and structure of your DSL.
• Cross-references have first-class support with built-in scoping and indexing.
• Dependency injection is used to wire everything together and customize on a fine-grained level.

Overall, we aimed to simplify areas where we saw potential for reducing complexity. Most importantly, Langium does not depend on any modeling framework, but relies solely on plain TypeScript types. This means that the grammar rules of your language are mapped to JavaScript objects and accompanied with generated TypeScript interface declarations. This might be seen as a restriction for those who hoped to integrate directly with existing modeling tools, but it’s a great simplification for the majority of use cases.

A brief example of a Langium grammar declaration is shown below. It contains three grammar rules Statemachine, State and Transition. The first rule is marked with the keyword entry to indicate that the parser starts processing input documents with this rule. The rule definitions consist of keywords like 'state', property assignments like name=ID, and cross-references like [State].

grammar States

entry Statemachine:
    'statemachine' name=ID
    states+=State*;

State:
    'state' name=ID
        transitions+=Transition*;

Transition:
    '=>' state=[State];

For a given text document, Langium creates a data structure called an Abstract Syntax Tree (AST). When written like the example above, every grammar rule invocation leads to a corresponding node (a JavaScript object) in the AST, and Langium generates a TypeScript interface for every rule to provide static typing for these nodes. The interface properties that are inferred from the assignments are found in their respective rules. This approach is great for rapid prototyping, where the focus is on designing the syntax of a new language. However, more mature language projects should declare their AST types explicitly to avoid accidental changes, as large parts of the language-specific code will depend on these types (validation checks, code generator, etc.). In that case, types can be declared within the grammar language using a syntax that is very similar to TypeScript. As an example, the listing below shows type declarations for the “States” grammar. The only difference to the TypeScript syntax is the reference notation @State, which means a cross-reference to a State.

interface Statemachine {
    name: string
    states: State[]
}

interface State {
    name: string
    transitions: Transition[]
}

interface Transition {
    state: @State
}

Langium and Sprotty: made for the cloud era

By combining Langium and Sprotty in a VS Code extension, you get a powerful text editor for your DSL with an accompanying graphical view. You can embed this extension in a web IDE based on Theia or VS Code, or publish it to the Open VSX Registry so others can benefit from it. The whole project is built with TypeScript, so you get a consistent code base that can be compiled, tested and packaged with a single set of tools. The result is a smooth onboarding experience for developers joining the project, as well as a secured long-term maintainability.

The diagram models rendered in the Sprotty frontend are backed by a diagram server. For the integration with Langium, the diagram server is included within the language server implementation, so it can be regarded as a “graphics extension” of the language server. The Language Server Protocol that connects the editor with the language server is based on the extensible JSON-RPC protocol. Messages between the diagram server and the graphical view in the frontend are transported via that protocol using additional JSON-RPC methods.

While Theia generally offers a deep API for customizing every part of your application, its @theia/languages package was deprecated in version 1.4.0 (July 2020), which means that language servers can no longer be integrated via Theia’s API, but only within a VS Code extension. As a consequence, the Sprotty-based diagrams must be embedded using the Webview API of VS Code. A Webview is an isolated web application embedded in its host IDE using an iframe (a page embedding concept in HTML). From the perspective of a tool developer, this means higher complexity due to the isolation of the graphical views and the involved communication channels, but the benefit for end users is higher security, making it feasible to install a VS Code extension without the risk of interfering with the rest of your IDE.

The sprotty-vscode integration, which is part of the Sprotty project, mitigates the complexity of Webviews by taking care of many technical details. The repository also contains the state machine example, of which an excerpt is shown in this article. The example is currently available with two different language server implementations: one based on Xtext and the other on Langium. This demonstrates the functional equivalence of the two solutions. Aside from the already discussed technological advantage, Langium exhibits better language server start-up performance in this example: approximately measured as one second for Langium and four seconds for Xtext.

To go further

The web page of Langium features a Getting Started guide as well as more in-depth documentation. Another example of a Langium DSL combined with Sprotty diagrams has been presented by a student at Kiel University.

Full-automatic creation of diagrams usually also involves a graph layout algorithm. Sprotty has a built-in integration with Eclipse Layout Kernel (ELK), which delivers highly configurable layouts of graph-like structures. While ELK is implemented in Java, a variant transpiled to JavaScript is also available so it can be integrated directly with a TypeScript language server. Alternatively, the layout engine can run in a background process and communicate with the language server via JSON. The latter approach can yield better performance for large graphs.

This article covered the approach where a textual language is used as the primary input format and corresponding diagrams are a secondary notation derived from the text documents. In case you want a graphics-first solution, Eclipse GLSP might be useful. It builds on top of Sprotty and adds diagram editing and other features.