Benchmark your language servers with LSBench

Ever found yourself performing the same actions manually in VS Code to check something in a language server? Possibly to reproduce a reported bug? To stress-test some functionality, or sanity-check performance after a major change? We certainly have, more times than we can count.

These kinds of checks are important for maintaining quality and integrity while developing DSLs, but also tedious and time consuming (not to mention easy to forget). So, we whipped up a nice tool to automate performing those kinds of checks.

We call it LSBench, short for “Language Server Benchmark”.

What is LSBench?

LSBench is a tool for benchmarking language servers, simple as that.

It accepts action scripts (written in TypeScript) that programmatically invoke LSP methods in a target workspace. This includes such things as opening a file, requesting document symbols, invoking go-to-definition requests, reading diagnostics, and more. LSBench can perform most of what you would initiate by hand via a language client, whether in a VS Code extension or otherwise.

By automating invocation of LSP methods that would otherwise be done manually, we can produce some really helpful analyses. For example, we’ve used LSBench for checking the performance of existing language servers and doing comparative analysis between language servers (for the same language). We’ve also used LSBench for measuring the before and after impacts of significant changes, such as introducing caching to speed up various validation checks.

Why did we make it?

Originally, LSBench was created to solve a single problem: to automate comparing two language servers for the same language (one older and one newer). In order to do that, we needed a tool that could automate calling LSP methods, warm up the server, aggregate over a number of runs, etc. Not only this, but it needed to be repeatable while respecting all of these constraints for both language servers.

Initially we looked around for tooling to do this in early 2026, and we found quite a few nice benchmarking tools. However, what we didn’t find was a scriptable one where we could have fine control over the full execution of the benchmark. Since the need was there, and the solution didn’t exist yet, we decided to build our own.

How does it work?

It’s pretty simple. If you have a language server that can communicate via stdio, then you’re good to go. Overall, what you need is:

• One or more language servers and a command to fire them up (e.g. node ./out/language-server.js --stdio)
• An LSBench action script listing a series of LSP method calls with timings for your benchmark

That’s it, not much more to it.

The following example shows an action script that calls some LSP methods, makes an edit, and awaits feedback in each case from the language server:

import type { BenchContext } from 'lsbench';
export default async function(ctx: BenchContext) {
    const prog = 'gallery.logo';
    await ctx.openDocument(prog);
    await ctx.waitForDiagnostics(prog, 10_000);

    // invoke some methods via line + col
    await ctx.completion(prog, 99, 4);
    await ctx.hover(prog, 29, 4);
    await ctx.definition(prog, 67, 12);
    await ctx.references(prog, 64, 4);

    // edit & await diagnostics
    await ctx.edit(prog, {
        range: {
            start: { line: 140, character: 0 },
            end: { line: 140, character: 0 },
        },
        text: 'someNewText(1,2,3)\n',
    });
    await ctx.waitForDiagnostics(prog, 10_000);
    await ctx.closeDocument(prog);
}

And it can be run like so:

lsbench "node ./out/language-server/main.js --stdio" \
    --workspace examples \
    --script action-script.ts \
    --iterations 50 \
    --warmup 2 \
    --output results.json

Giving results like these:

  Running iteration 48/50... 105ms
  Running iteration 49/50... 103ms
  Running iteration 50/50... 109ms

╔══════════════════════════════════════════════════════════════╗
║                       lsbench results                        ║
╚══════════════════════════════════════════════════════════════╝

  Server:      node ./out/language-server/main.js --stdio
  Workspace:   examples
  Iterations:  50 (+ 2 warmup)
  Total time:  6.3s
  Restart:     no

  Iteration totals:
    avg=112.464ms  median=109.558ms  p95=130.179ms  min=103.486ms  max=133.472ms

  Method                          Avg    Median       P95       P99       Min       Max    Stddev   Fail%
  ───────────────────────────────────────────────────────────────────────────────────────────────────────
  textDocument/completion       3.264     2.925     4.011    13.153     2.729    13.153     1.449    0.0%
  textDocument/hover            0.333     0.316      0.55     0.769     0.214     0.769     0.105    0.0%
  textDocument/definition       0.275      0.25     0.407     0.419     0.193     0.419     0.068    0.0%
  textDocument/references       0.687     0.654     0.861     1.166     0.554     1.166      0.11    0.0%

This is a quick overview, but expressive enough to demonstrate the kinds of steps you can sequence together. Also, since it’s TypeScript, you can thoroughly customize your benchmarks.

What’s great is that Langium language servers support stdio communication by default (go try it out if you’re curious). This extends to Fastbelt and Xtext based language servers too. In fact, most language servers support stdio communication. That made it easy to target arbitrary language servers, regardless of their implementation language or framework.

Where can I get it?

You can get LSBench on npm via the lsbench package, or go check out LSBench on GitHub. As it stands, LSBench is intended to be consumed as a CLI tool to run its own action scripts. But we’re thinking about exposing the API more thoroughly as well, so it can be imported into an existing project if needed.

Wrapping up

With LSBench, you can automate checking corner cases, complicated setups, and other circumstances that would otherwise take time and mechanical effort. We think that open tooling is fantastic, and so we’re happy to make this an MIT-licensed open source contribution for others to use and build upon as well. Go check it out, let us know what you think (or contribute), and happy benching!