Plexus-Flow Documentation
plexus-flow is a workflow, DAG, and Saga orchestration engine packaged as a single Go binary.
It coordinates multi-step workflows across in-process or remote gRPC workers using two embedded Raft consensus groups (FlowStore and ActivityStore) backed by BoltDB.
Quick Start
Build the binary and start a single-node server:
go build -o bin/plexus-flow ./cmd/plexus-flow
./bin/plexus-flow \
--id node-1 \
--http-addr :8080 \
--grpc-addr :9090 \
--raft-addr 127.0.0.1:7001 \
--act-raft-addr 127.0.0.1:7002 \
--data-dir ./data/node-1
Runnable Examples
The repository includes two standalone examples in ./examples:
# Order Fulfillment Saga (happy path and simulated failure rollback)
go run ./examples/ordersaga
go run ./examples/ordersaga --simulate-failure --fail-step ship-order
# Keep the web UI running on http://localhost:8080:
go run ./examples/ordersaga --serve --http-addr :8080
# ML Canary Release Pipeline (Fork-Join DAG, conditional branching, signal gate, cron)
go run ./examples/mlrelease
Dual-Raft Architecture
- Upstream (
FlowStore): Maintains workflow definitions, workflow execution state, DAG dependency resolution, conditional predicates, timers, signals, cron schedules, and the in-memory search index. - Downstream (
ActivityStore): Manages queued activity tasks, worker leases, heartbeats, retry scheduling, and task completion records. - Idempotent Dispatch Key (
UpstreamID:Term:Index:Seq): Every task emitted when an upstream Raft log entry commits carries its Raft log coordinates. If a leader fails and a new leader re-evaluates the transition,ActivityStorededuplicates the emission using this key.
Workflow & Step Definition Schema
Workflow definitions can be registered via HTTP (POST /api/v1/definitions), loaded at startup with --flows=./flows/*.json, or submitted through the Go and Python clients.
{
"name": "data-ingestion-pipeline",
"steps": [
{
"name": "extract-parquet",
"activity": "s3.extract_parquet",
"compensating_action": "s3.cleanup_staging",
"retry_policy": {
"max_attempts": 4,
"backoff": 1000000000,
"non_retryable_errors": ["ERR_CORRUPT_SCHEMA"]
}
},
{
"name": "cleanse-records",
"activity": "spark.cleanse",
"depends_on": ["extract-parquet"]
},
{
"name": "embed-vectors",
"activity": "gpu.embed_chunks",
"depends_on": ["extract-parquet"],
"when": {
"source": "input",
"field": "enable_embeddings",
"op": "eq",
"value": true
}
},
{
"name": "cooldown",
"sleep": "5s",
"depends_on": ["cleanse-records", "embed-vectors"]
},
{
"name": "await-review",
"wait_for_signal": "steward-approval",
"depends_on": ["cooldown"]
},
{
"name": "commit-snapshot",
"activity": "iceberg.commit_snapshot",
"depends_on": ["await-review"],
"when": {
"source": "await-review",
"field": "approved",
"op": "eq",
"value": true
}
}
]
}
Fork-Join DAGs & the Claim-Check Pattern (depends_on)
Steps with no depends_on entries start as soon as the workflow starts. Multiple steps depending on the same parent run in parallel (fan-out), and a step listing multiple dependencies waits until all listed dependencies reach COMPLETED or SKIPPED (fan-in).
ctx.dependency_outputs (Python) or info.DependencyOutputs (Go) to read and write large files or database tables directly.
# Fan-out step (depends_on: ["extract-parquet"]): fetches S3 file by key from Step 1, writes cleaned Parquet to S3
@worker.activity("spark.cleanse")
async def cleanse_records(ctx: TaskContext, payload: dict) -> dict:
extracted = ctx.dependency_outputs["extract-parquet"]
obj = s3.get_object(Bucket=extracted["bucket"], Key=extracted["raw_key"])
table = pq.read_table(io.BytesIO(obj["Body"].read())).drop_null()
clean_key = f"curated/{ctx.workflow_id}.parquet"
buf = io.BytesIO()
pq.write_table(table, buf)
s3.put_object(Bucket=extracted["bucket"], Key=clean_key, Body=buf.getvalue())
return {"bucket": extracted["bucket"], "clean_key": clean_key, "rows": table.num_rows}
# Fan-in join step (depends_on: ["cleanse-records", "embed-vectors"]): registers outputs of both branches in Postgres
@worker.activity("iceberg.commit_snapshot")
async def commit_snapshot(ctx: TaskContext, payload: dict) -> dict:
cleansed = ctx.dependency_outputs["cleanse-records"]
embedded = ctx.dependency_outputs.get("embed-vectors") # None if skipped by a "when" condition
vector_count = embedded["indexed_vectors"] if embedded else 0
db.execute(
"INSERT INTO catalog_snapshots (workflow_id, s3_key, rows, vectors) VALUES (%s, %s, %s, %s)",
(ctx.workflow_id, cleansed["clean_key"], cleansed["rows"], vector_count),
)
db.commit()
return {"published_key": cleansed["clean_key"], "rows": cleansed["rows"], "vectors": vector_count}
Conditional Branching (when)
When all dependencies of a step are satisfied, the FSM evaluates its optional when condition. If false, the step is marked SKIPPED without dispatching a task.
| Field | Supported Values | Description |
|---|---|---|
source |
"input" (default), "metadata", or "<step-name>" |
Selects the workflow input JSON, workflow metadata map, or a completed dependency step's output JSON. |
field |
Dot-separated JSON path (e.g. "metrics.f1") or "$status" |
Extracts a nested field, or checks whether the dependency finished as "COMPLETED" or "SKIPPED". |
op |
eq, ne, gt, gte, lt, lte, in, exists, not_exists |
Comparison operator evaluated against value. |
LIFO Saga Compensation
If a step fails after exhausting its retry policy, the workflow transitions to COMPENSATING. Any previously completed steps that define a compensating_action are executed in reverse topological order until they reach ROLLED_BACK and the workflow reaches COMPENSATED.
Durable Timers (sleep) & Signal Gates (wait_for_signal)
- Sleep Timers: Setting
"sleep": "10m"records a deterministicWakeAttimestamp in the Raft FSM. The timer survives node restarts and completes once the cluster clock passesWakeAt. - Signal Gates: Setting
"wait_for_signal": "steward-approval"places the step inWAITING_SIGNAL. When a signal arrives viaPOST /api/v1/workflows/{id}/signal, its JSON payload becomes the step's output. Signals sent early are buffered in the workflow state.
Durable Cron Schedules
Schedules are stored in the Raft FSM and support standard 5-field cron expressions, IANA timezones, and overlap policies (SKIP, ALLOW, or CANCEL_PREVIOUS):
curl -X POST http://localhost:8080/api/v1/schedules \
-H "Content-Type: application/json" \
-d '{
"id": "hourly-compaction",
"workflow_name": "data-ingestion-pipeline",
"cron": "0 * * * *",
"timezone": "UTC",
"overlap_policy": "SKIP",
"input": {"enable_embeddings": true},
"metadata": {"env": "prod"}
}'
Data Pipeline Patterns
1. Progress Checkpointing (HeartbeatWithProgress & LastProgress)
Long-running activities can include a JSON progress payload when sending heartbeats. The payload is persisted on the step state in Raft (`StepState.LastProgress`). If a worker disconnects or a leader fails over, the retried task receives `LastProgress` so it can continue from the last checkpoint.
@worker.activity("backfill-partition")
async def backfill_partition(ctx: TaskContext, job: dict) -> dict:
state = ctx.last_progress or {"offset": 0, "rows": 0}
offset = state["offset"]
rows = state["rows"]
while offset < job["total_rows"]:
batch = read_rows(job["table"], offset=offset, limit=50_000)
write_rows(batch)
offset += len(batch)
rows += len(batch)
await ctx.report_progress({"offset": offset, "rows": rows})
return {"rows_written": rows}
2. Targeted Step Replay (/replay)
When a downstream step fails after expensive upstream steps have already completed, you can reset only the failed step (and its transitive dependents) back to PENDING while preserving upstream outputs:
curl -X POST http://localhost:8080/api/v1/workflows/etl-run-101/replay \
-H "Content-Type: application/json" \
-d '{"step": "cleanse-records"}'
3. Selective Retries & Error Codes
Steps can configure max_attempts, initial backoff (exponentially doubled per attempt), and non_retryable_errors. If an activity returns a matching error code, retries are skipped immediately:
// Return a non-retryable coded error in Go:
return Output{}, worker.NewCodedError("ERR_CORRUPT_SCHEMA", errors.New("missing column: timestamp"))
# Raise a non-retryable coded error in Python:
from plexus_flow import ActivityFailure
raise ActivityFailure("missing column: timestamp", code="ERR_CORRUPT_SCHEMA", retryable=False)
4. Faceted Search
Workflows can be queried by status, definition name, and arbitrary metadata key-value pairs via the embedded inverted index:
curl "http://localhost:8080/api/v1/workflows?status=FAILED&name=data-ingestion-pipeline&meta.env=prod&limit=50"
Go SDK Reference
Use worker.RegisterTyped to register strongly typed activity handlers, workerclient.TaskFromContext to inspect dependency outputs and checkpoints, and workerclient.New to connect to a single server or a multi-node cluster.
package main
import (
"context"
"encoding/json"
"log"
"github.com/avivklas/plexus-flow/pkg/worker"
"github.com/avivklas/plexus-flow/pkg/workerclient"
)
type CleanseInput struct {
DatasetURI string `json:"dataset_uri"`
}
type CleanseOutput struct {
RowsWritten int `json:"rows_written"`
OutputURI string `json:"output_uri"`
}
func main() {
reg := worker.NewRegistry()
worker.RegisterTyped(reg, "spark.cleanse", func(ctx context.Context, in CleanseInput) (CleanseOutput, error) {
info, _ := workerclient.TaskFromContext(ctx)
_ = info.LastProgress // Checkpoint from prior attempt, if any
_ = info.DependencyOutputs["extract-parquet"] // Upstream dependency output JSON
_ = workerclient.ReportProgress(ctx, json.RawMessage(`{"pct": 50}`))
return CleanseOutput{
RowsWritten: 100000,
OutputURI: "s3://curated/clean.parquet",
}, nil
})
// Comma-separated addresses enable automatic failover if the current leader stops
w := workerclient.New("10.0.0.1:9090,10.0.0.2:9090,10.0.0.3:9090", workerclient.Options{
WorkerID: "go-worker-1",
MaxConcurrency: 16,
Registry: reg,
})
if err := w.Run(context.Background()); err != nil {
log.Fatal(err)
}
}
Python SDK Reference
Install the package from sdk/python (pip install -e sdk/python):
import asyncio
from plexus_flow import Worker, TaskContext, ActivityFailure, WorkflowClient
worker = Worker("localhost:9090", worker_id="py-worker-1", max_concurrency=4)
@worker.activity("gpu.embed_chunks")
async def embed_chunks(ctx: TaskContext, payload: dict) -> dict:
if "dataset_uri" not in payload:
raise ActivityFailure("dataset_uri is required", code="ERR_INVALID_INPUT", retryable=False)
extracted = ctx.dependency_outputs.get("extract-parquet", {})
await ctx.report_progress({"completed_batches": 5, "source": extracted.get("uri")})
return {"vectors": 50000}
# Start a workflow and stream updates over SSE:
client = WorkflowClient("http://localhost:8080")
wf = client.start_workflow(
"data-ingestion-pipeline",
{"dataset_uri": "s3://raw/events.parquet", "enable_embeddings": True},
workflow_id="ingest-42",
metadata={"tenant": "acme"}
)
for state in client.watch("ingest-42", timeout=60.0):
if state["status"] in ("COMPLETED", "FAILED", "COMPENSATED"):
print("Terminal status:", state["status"])
break
HTTP & SSE API Reference
Mutating requests sent to a follower node are forwarded to the current Raft leader.
| Endpoint | Description |
|---|---|
POST /api/v1/definitions |
Register or update a workflow definition. |
GET /api/v1/definitions |
List all registered workflow definitions. |
POST /api/v1/workflows |
Start a workflow instance (id, workflow_name, input, metadata). |
GET /api/v1/workflows |
Search workflows (?status=...&name=...&meta.key=val&offset=0&limit=50). |
GET /api/v1/workflows/{id} |
Get workflow instance details (falls back to cold archive if compacted). |
POST /api/v1/workflows/{id}/signal |
Send a named signal ({"signal": "...", "payload": {...}}) to a workflow. |
POST /api/v1/workflows/{id}/replay |
Reset a failed step ({"step": "..."}) and its downstream dependents to PENDING. |
POST /api/v1/workflows/{id}/cancel |
Cancel a running workflow. |
GET /api/v1/workflows/{id}/events |
Server-Sent Events stream for a single workflow instance. |
GET /api/v1/events |
Server-Sent Events stream for all workflow transitions. |
POST /api/v1/schedules |
Create or update a cron schedule. |
GET /api/v1/cluster/status |
Return Raft leadership status, connected workers, and active leases. |
POST /api/v1/admin/compact |
Compact terminal workflows older than a specified duration. |
GET /healthz & GET /readyz |
Liveness and Raft quorum readiness endpoints. |
Multi-Node Raft Cluster Setup
To run a 3-node cluster, start node-1 as the bootstrap node and join node-2 and node-3 using --join (for FlowStore) and --act-join (for ActivityStore):
# Node 1 (bootstrap)
plexus-flow \
--id node-1 \
--http-addr 0.0.0.0:8081 \
--grpc-addr 0.0.0.0:9091 \
--raft-addr 127.0.0.1:7001 \
--act-raft-addr 127.0.0.1:7002 \
--data-dir ./data/node-1
# Node 2 (joins Node 1)
plexus-flow \
--id node-2 \
--http-addr 0.0.0.0:8082 \
--grpc-addr 0.0.0.0:9092 \
--raft-addr 127.0.0.1:7003 \
--act-raft-addr 127.0.0.1:7004 \
--join 127.0.0.1:7001 \
--act-join 127.0.0.1:7002 \
--data-dir ./data/node-2
# Node 3 (joins Node 1)
plexus-flow \
--id node-3 \
--http-addr 0.0.0.0:8083 \
--grpc-addr 0.0.0.0:9093 \
--raft-addr 127.0.0.1:7005 \
--act-raft-addr 127.0.0.1:7006 \
--join 127.0.0.1:7001 \
--act-join 127.0.0.1:7002 \
--data-dir ./data/node-3
CLI Flags & Environment Variables
| Flag | Environment Variable | Default | Description |
|---|---|---|---|
--id |
PLEXUS_FLOW_NODE_ID |
node-1 |
Unique node ID. |
--http-addr |
PLEXUS_FLOW_HTTP_ADDR |
:8080 |
HTTP API and web UI listen address. |
--grpc-addr |
PLEXUS_FLOW_GRPC_ADDR |
:9090 |
gRPC worker broker listen address. |
--raft-addr |
PLEXUS_FLOW_RAFT_ADDR |
127.0.0.1:7001 |
Upstream (FlowStore) Raft bind address. |
--act-raft-addr |
PLEXUS_FLOW_ACT_RAFT_ADDR |
127.0.0.1:7002 |
Downstream (ActivityStore) Raft bind address. |
--advertise-raft-addr |
PLEXUS_FLOW_ADVERTISE_RAFT_ADDR |
(same as --raft-addr) |
Advertised address for FlowStore Raft peers. |
--advertise-act-raft-addr |
PLEXUS_FLOW_ADVERTISE_ACT_RAFT_ADDR |
(same as --act-raft-addr) |
Advertised address for ActivityStore Raft peers. |
--join / --act-join |
PLEXUS_FLOW_JOIN / PLEXUS_FLOW_ACT_JOIN |
"" |
Existing cluster addresses to join as a voter. |
--data-dir |
PLEXUS_FLOW_DATA_DIR |
./data |
Directory for BoltDB Raft logs and snapshots. |
--flows |
PLEXUS_FLOW_FLOWS |
"" |
Comma-separated paths/globs of JSON workflow definitions to load on startup. |
--retention-ttl |
PLEXUS_FLOW_RETENTION_TTL |
0 |
TTL after which terminal workflows are compacted. |
--max-completed-workflows |
PLEXUS_FLOW_MAX_COMPLETED_WORKFLOWS |
0 |
Maximum completed workflows kept in hot memory. |
Retention & Cold Archival
Configuring --retention-ttl or --max-completed-workflows runs a periodic compaction loop that removes older terminal workflows from the in-memory Raft FSM. When a ColdArchive implementation is configured on the coordinator, compacted workflows are archived prior to deletion and transparently fetched on GET /api/v1/workflows/{id}.
Running the Test Suite
# Run all unit and integration tests with the race detector
go test -race ./...
# Run the cluster failover and concurrency tests
go test -v -race -run TestBattle ./test