DIY Zeebe Applications

Sour grapes…, so let’s grow our own.

Zeebe is a workflow and process automation engine developed by Camunda, which powers their current main product Camunda Platform 8. While its new architecture enables Camunda to provide turnkey cloud solutions for their customers, open source users have been left in dust. There is no Open Source Community Edition version of “Camunda Platform 8”. Many supporting libraries are still being released with open source licenses, and Zeebe License mostly only limits use of Zeebe in competing solutions, but the official end-user applications and their APIs are for licensed users only.

So, the rules are clear, and we have been freed to look out of the box for open source solutions!

Sorting out the pieces

Unlike Camunda Platform 7, Zeebe based Camunda Platform 8 offerings do not include usable out-of-the-box experience without a paid license. Yet, Zeebe engine itself, with its semi-open license, is good and usable enough that filling the gaps should still be worth of the effort.

The core pieces available to be used with free or open source licenses are

• Zeebe engine (and its cluster gateway) itself (license does not including the right to provide a Commercial Process Automation Service)

• zbctl command-line client for interacting with the engine even without custom job worker

• a collection Zeebe exporters to make the engine state available for custom end-user application

• all BPMN.io components for building user-interfaces with BPMN diagrams, DMN tables and Camunda Forms.

The remaining pieces required to make a usable Zeebe installation include

• optional custom Zeebe interceptors for e.g. authenticating Zeebe gRPC API requests

• end-user application specific databases and Zeebe events consumer implementations for consuming the exported Zeebe event stream with required detail into application databases

• implementations of the required end-user application specific APIs and the applications themselves.

To be fair, Camunda Community Hub does include simple monitor application, simple tasklist application and even a GraphQL API implementation. Yet, it is hard to recommend any of them to be used in production without good understanding of Zeebe architecture, and skills to further customize and develop them when required.

Simply put, to use Zeebe “for free” requires technical understanding of how it works. And the best way to understand it, is to put your hands into mud and really try it out…

KISS with Redis and SQLite

Zeebe is being advertised with its ability to scale, but most of my problems are relatively small. I’d be happy to have Zeebe itself as the the sole memory hungry enterprise Java app in my stack. Needless to say, the release of Zeebe Redis Exporter got me excited. It inspired me to try it with an another light weight solution: SQLite.

SQLite is a serverless relational open-source database system, which is designed to be efficient and easy to use and is widely used in small to medium-sized applications. It has a small memory footprint and is optimized for read-heavy workloads. That sounds pretty much like the requirements for a simple Zeebe end-user application database:

• only the Zeebe export stream consumer really needs to write to the database

• when the application needs to interact with the processes, it writes directly to Zeebe using its gRPC API, not touching its own database

• and, therefore, the application really needs only read access to the exported database.

This task could not get much simpler than with Redis and SQLite, but writing my first Zeebe export stream consumer was still substantial amount of work. It took a little more than 600 lines of Python code to start with, but I probably missed some events, and in practice, the importer would grow over time.

Datasette.io for API and monitoring

SQLite ecosystem includes a real open-source gem: https://datasette.io/.

Datasette is a tool for exploring and publishing data of any shape from SQLite databases. Out-of-the box it provides powerful and easy to use database browser for the database built from the exported Zeebe events:

Also, with almost no additional configuration at all, Datasette GraphQL plugin makes the same database accessible with GraphQL. Of course, properly connecting tables by their foreign key relationships:

And if that was not enough, Datasette dashboard plugin adds support for monitoring dashboards with declarative JSON or YAML configuration and Vega based charts:

That’s a lot of features with almost no configuration at all.

Process applications with Notebooks

The final task was to create an end-user application for interacting with the Zeebe orchestrated processes. I’ve been maintaining simple BPMN and DMN rendering plugins for JupyterLab for some time already. I realized, I was only missing a plugin for Camunda Forms to make Jupyter notebook my process application platform.

I also had some luck with timing. Jupyter Voilà just got new alpha releases, which made it work properly with my BPMN, DMN ja form-js plugins. Voilà is the latest product from Project Jupyter, which can turn Jupyter notebooks into standalone web applications.

In my use case, Voilà turns my Jupyter notebook into a simple interactive process application – with my JupyterLab plugins, using data from Datasette GraphQL API, and interacting with Zeebe using just zbctl.

Obviously, all this gets nowhere near the feature set of the official Camunda Platform applications. But this is a start, and open source usually gets better over time…