How and why use Xtext without the IDE

Xtext is a language development framework that is best known for the rich tool support it gives you for your programming languages. But even if you don’t need editing capabilities, Xtext has much more to offer than a simple parser generator like Antlr. In this post I will first describe the aspects and features that Xtext offers on the runtime side, before I briefly explain how to use them in any vanilla Java process.

Why use Xtext even without IDE support

Antlr is an excellent parser generator, but a language needs more than just a parser. With Xtext you get callbacks, support and even full implementations for the following non-tooling aspects of a language:

• lexer & parser (through Antlr)
• typed abstract syntax tree (AST)
• parse tree
• unparser (AST -> text)
• lazy linking
• scoping framework
• static analysis / validation
• code generation
• interpreter
• EMF compatibility
• Incremental compiler support
• Maven and Gradle plugins

For the simple cases you would only throw a grammar and a code generator template at Xtext and you have a fully featured language compiler (or transpiler). However, the architecture of Xtext allows you to customize every aspect of your language in a very clean non-invasive manner. On top of that, there are standard plugins for Maven and Gradle that let you include your Xtext compiler within your builds.

How can I use the runtime part?

No matter what kind of Xtext project you have, the structure already separates tooling-related things from the runtime part. The main project, which contains the *.xtext grammar file, includes everything you need to parse, validate and process files of your language. Since version 2.9 you can even create new projects without any IDE support. To do so you simply need to deselect all the tooling-related options in the wizard, pick whether you want to build your project with Gradle or Maven and your are done. The following screenshot shows that wizard page.

If you want to use the runtime part of your language in e.g. a mavenized Java project, you simply need to add a dependency to your language’s pom. Next up you can for instance use the EMF API to load files of your language like this:

// do this only once per application
Injector injector = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration();

// obtain a resourceset from the injector
XtextResourceSet resourceSet = injector.getInstance(XtextResourceSet.class);

// load a resource by URI, in this case from the file system
Resource resource = resourceSet.getResource(URI.createFileURI("./mymodel.mydsl"), true);

If you want to load a bunch of files that have references to each other, you should add them all to the resourceset at this point, by calling resourceSet.getResource(URI.createFileURI("./anothermodel.mydsl"), true); It is a good idea to check the validity of the model before processing it, so better call validate next:

// Validation
IResourceValidator validator = ((XtextResource)resource).getResourceServiceProvider().getResourceValidator();
List<Issue> issues = validator.validate(resource, CheckMode.ALL, CancelIndicator.NullImpl);
for (Issue issue : issues) {
  System.out.println(issue.getMessage());
}

The Issue objects contain all the information needed, here we only print out the error message. If you now also want to run the code generator you can do so by asking the injector for the GeneratorDelegate (since 2.9, use IGenerator for earlier versions) like in the following example:

// Code Generator
GeneratorDelegate generator = injector.getInstance(GeneratorDelegate.class);
InMemoryFileSystemAccess fsa = new InMemoryFileSystemAccess();
generator.doGenerate(resource, fsa);
for (Entry<String, CharSequence> file : fsa.getTextFiles().entrySet()) {
  System.out.println("Generated file path : "+file.getKey());
  System.out.println("Generated file contents : "+file.getValue());
}

In this example we pass an in-memory file system to the generator; of course there is also one that delegates to java.io.File, i.e. directly writes to disk.

Using the new Standalone Builder programmatically

Another option is to use the incremental builder, which takes care of all the lifecycles and indexing. For a set of files that it maintains and builds it will automatically detect the effective changes for subsequent smaller changes. The new gradle deamon as well as the new IntelliJ IDEA plugin use this component. If you want to learn how to use that these unit tests should give you an idea.

When not to use Xtext?

There are only a few scenarios where I wouldn’t use Xtext. Mostly this would be because I need to process huge amount of data, where I simply cannot afford translating to an in-memory AST before processing. So basically the cases where you would prefer a SAX parser over a DOM when processing XML.

Platform independent tool support

Maybe you are interested in tool support for your language, but you simply don’t want to have it for Eclipse, IntelliJ IDEA or Orion/Ace/CodeMirror which Xtext covers out-of-the-box. Since 2.9 there is an additional platform independent IDE project that can be used in any Java program and gives you the basic infrastructure for things like content assist and syntax coloring. This code can be used from wherever you want. People have already built support for JavaFX and Atom with it.

Summary

Xtext has a lot to offer even if you don’t need an editor for your language. The grammar language is a convenient way to describe syntax and map it to a typed AST that can be further processed with Java or Xtend. The linking, validation and code generation hooks are all in place and battle-proven. Furthermore Xtext provides a serializer (aka. unparser), which allows you to modify the AST programmatically and write the changes back to the concrete textual syntax.