Action Engine

Action Engine is a cross-language library for building interactive applications that need streaming, multimodal, interactive capabilities. Think of:

• GenAI prototypes and fully served applications that need different media modalities and interactivity—image generation and editing, speech to text, etc.
• LLM agents that can interact with users and tools
• Native applications that need to interact with LLMs or just combine API calls asynchronously in a flexible way

The core library is written in C++, and there are bindings for Python with almost complete feature coverage. A TypeScript library is also available for building UIs that interact with the core library.

Quickstart in Python

If there is a wheel already built for your platform, you can install Action Engine from PyPI (note that the package is still under development, so the API is not stable yet):

pip install action-engine
python -c 'import actionengine; print(actionengine.to_chunk("Hello, world!"))'

Otherwise, you can install Action Engine directly from the GitHub repository. First, make sure you have a recent C++20-compatible clang installed, then run:

pip install git+https://github.com/google-deepmind/actionengine -v
python -c 'import actionengine; print(actionengine.to_chunk("Hello, world!"))'

You can then explore the examples in the examples folder, such as 007-python-generative-media, which showcases how to build generative media applications with LLMs and text-to-image models.

Action Engine requires Python 3.11 or higher.

Features

• Flexible unopinionated architecture based on two core concepts:
  • async nodes, which are not more and not less than asynchronous data channels, with a bring-your-own storage system in mind,
  • actions, which are computations that transform data from input nodes to output nodes—and can produce side effects. This makes it easy to build complex data flows and interactions.
• Agnostic to wire formats and connection contexts: The core library does not care about how data is serialized or transmitted. This makes it easy to integrate with different transport layers, such as WebSockets, gRPC, or custom protocols.
  • WebSockets and WebRTC WireStreams are supported out of the box.
  • msgpack serialisation is implemented by default, with Protobuf definitions available for Action Engine data structures, and customisation points for application-specific data types.
• Streaming-first: Everything is designed to be streamed, from LLM outputs to images and audio. This enables low-latency, interactive applications.
• Multimodal and extensible: Supports text, images, audio, and can be easily extended to support other modalities. No architecture components are mandatory; you can pick and choose what you need, combining and extending as you see fit.
• Cross-language: Core library in C++ with bindings for Python and a TypeScript UI-centered library. Enables building applications in different environments: server backends, native apps, web UIs, peer-to-peer apps, etc.

Getting started

Core library and Python bindings

This library is under development, so the API is not stable yet, hence no packages to install from PyPI or similar. To get started, clone this repo and follow these steps:

1. Make sure you have CMake (version 3.22 or higher), clang with C++20 support, and ninja-build installed on your system.

2. Create and activate a Python virtual environment (optional but highly recommended).

3. Compile and install the PyBind11 bindings and the Python package:

   ./scripts/configure.sh
   ./scripts/build_python.sh

4. At this point, you should have an editable installation of the Python package actionengine in your virtual environment. This command should work:

   python -c "import actionengine; print('Action Engine imports successfully!')"

Run a trivial C++ example

All examples are in the examples folder and can be built with CMake. For instance, to build and run the 000-actions example:

cd build
cmake --build . --target 000-actions
chmod +x examples/000-actions/000-actions
./examples/000-actions/000-actions

Use the TypeScript library

The TypeScript library lives under the js folder. To get started, follow these steps:

1. Make sure you have Node.js (version 23 or higher) and npm installed on your system.

2. Build the bundle:

   cd js
   npm install
   npm run build:bundle

3. Link the package for local development:

   npm link  # (or yarn link if you use yarn, for example)

4. Now you can use the package in your TypeScript projects by linking it:

   npm link actionengine  # (or yarn link actionengine)

Examples that use the TypeScript library to build UIs will be added soon.

Demos and examples

At examples/007-python-generative-media, there is backend code for several demo applications that showcase the capabilities of Action Engine. TBD: insert images for text to image, deep research and stateful LLM conversations.

Development log / to do list

• Set up CI and publishing pipelines for PyPI and docs
• Cover the essentials with tests
• Ensure no fatal errors and no hangs in basic session usage, including in Python bindings
• Propagate action status to all output nodes
• Make ChunkStoreReader and ChunkStoreWriter run tasks on a fiber pool, not own fibers
• Extend core PyBind11 bindings by more Pythonic interfaces more comprehensively; revise object ownership model