Plexus-Flow Documentation

plexus-flow is a workflow, DAG, and Saga orchestration engine packaged as a single Go binary. It coordinates multi-step workflows across in-process or remote gRPC workers using two embedded Raft consensus groups (FlowStore and ActivityStore) backed by BoltDB.

Quick Start

Build the binary and start a single-node server:

bash
go build -o bin/plexus-flow ./cmd/plexus-flow

./bin/plexus-flow \
  --id node-1 \
  --http-addr :8080 \
  --grpc-addr :9090 \
  --raft-addr 127.0.0.1:7001 \
  --act-raft-addr 127.0.0.1:7002 \
  --data-dir ./data/node-1

Runnable Examples

The repository includes two standalone examples in ./examples:

bash
# Order Fulfillment Saga (happy path and simulated failure rollback)
go run ./examples/ordersaga
go run ./examples/ordersaga --simulate-failure --fail-step ship-order

# Keep the web UI running on http://localhost:8080:
go run ./examples/ordersaga --serve --http-addr :8080

# ML Canary Release Pipeline (Fork-Join DAG, conditional branching, signal gate, cron)
go run ./examples/mlrelease

Dual-Raft Architecture

  • Upstream (FlowStore): Maintains workflow definitions, workflow execution state, DAG dependency resolution, conditional predicates, timers, signals, cron schedules, and the in-memory search index.
  • Downstream (ActivityStore): Manages queued activity tasks, worker leases, heartbeats, retry scheduling, and task completion records.
  • Idempotent Dispatch Key (UpstreamID:Term:Index:Seq): Every task emitted when an upstream Raft log entry commits carries its Raft log coordinates. If a leader fails and a new leader re-evaluates the transition, ActivityStore deduplicates the emission using this key.

Workflow & Step Definition Schema

Workflow definitions can be registered via HTTP (POST /api/v1/definitions), loaded at startup with --flows=./flows/*.json, or submitted through the Go and Python clients.

json
{
  "name": "data-ingestion-pipeline",
  "steps": [
    {
      "name": "extract-parquet",
      "activity": "s3.extract_parquet",
      "compensating_action": "s3.cleanup_staging",
      "retry_policy": {
        "max_attempts": 4,
        "backoff": 1000000000,
        "non_retryable_errors": ["ERR_CORRUPT_SCHEMA"]
      }
    },
    {
      "name": "cleanse-records",
      "activity": "spark.cleanse",
      "depends_on": ["extract-parquet"]
    },
    {
      "name": "embed-vectors",
      "activity": "gpu.embed_chunks",
      "depends_on": ["extract-parquet"],
      "when": {
        "source": "input",
        "field": "enable_embeddings",
        "op": "eq",
        "value": true
      }
    },
    {
      "name": "cooldown",
      "sleep": "5s",
      "depends_on": ["cleanse-records", "embed-vectors"]
    },
    {
      "name": "await-review",
      "wait_for_signal": "steward-approval",
      "depends_on": ["cooldown"]
    },
    {
      "name": "commit-snapshot",
      "activity": "iceberg.commit_snapshot",
      "depends_on": ["await-review"],
      "when": {
        "source": "await-review",
        "field": "approved",
        "op": "eq",
        "value": true
      }
    }
  ]
}

Fork-Join DAGs & the Claim-Check Pattern (depends_on)

Steps with no depends_on entries start as soon as the workflow starts. Multiple steps depending on the same parent run in parallel (fan-out), and a step listing multiple dependencies waits until all listed dependencies reach COMPLETED or SKIPPED (fan-in).

Control Plane vs. Data Plane: Keep payloads stored in the Raft state machine small (S3 object keys, database row IDs, row counts, and checkpoint cursors). Activities use those keys from ctx.dependency_outputs (Python) or info.DependencyOutputs (Go) to read and write large files or database tables directly.
python — S3 & Postgres I/O Across Fan-Out and Fan-In Steps
# Fan-out step (depends_on: ["extract-parquet"]): fetches S3 file by key from Step 1, writes cleaned Parquet to S3
@worker.activity("spark.cleanse")
async def cleanse_records(ctx: TaskContext, payload: dict) -> dict:
    extracted = ctx.dependency_outputs["extract-parquet"]
    obj = s3.get_object(Bucket=extracted["bucket"], Key=extracted["raw_key"])
    table = pq.read_table(io.BytesIO(obj["Body"].read())).drop_null()

    clean_key = f"curated/{ctx.workflow_id}.parquet"
    buf = io.BytesIO()
    pq.write_table(table, buf)
    s3.put_object(Bucket=extracted["bucket"], Key=clean_key, Body=buf.getvalue())
    return {"bucket": extracted["bucket"], "clean_key": clean_key, "rows": table.num_rows}

# Fan-in join step (depends_on: ["cleanse-records", "embed-vectors"]): registers outputs of both branches in Postgres
@worker.activity("iceberg.commit_snapshot")
async def commit_snapshot(ctx: TaskContext, payload: dict) -> dict:
    cleansed = ctx.dependency_outputs["cleanse-records"]
    embedded = ctx.dependency_outputs.get("embed-vectors")  # None if skipped by a "when" condition
    vector_count = embedded["indexed_vectors"] if embedded else 0

    db.execute(
        "INSERT INTO catalog_snapshots (workflow_id, s3_key, rows, vectors) VALUES (%s, %s, %s, %s)",
        (ctx.workflow_id, cleansed["clean_key"], cleansed["rows"], vector_count),
    )
    db.commit()
    return {"published_key": cleansed["clean_key"], "rows": cleansed["rows"], "vectors": vector_count}

Conditional Branching (when)

When all dependencies of a step are satisfied, the FSM evaluates its optional when condition. If false, the step is marked SKIPPED without dispatching a task.

Field Supported Values Description
source "input" (default), "metadata", or "<step-name>" Selects the workflow input JSON, workflow metadata map, or a completed dependency step's output JSON.
field Dot-separated JSON path (e.g. "metrics.f1") or "$status" Extracts a nested field, or checks whether the dependency finished as "COMPLETED" or "SKIPPED".
op eq, ne, gt, gte, lt, lte, in, exists, not_exists Comparison operator evaluated against value.

LIFO Saga Compensation

If a step fails after exhausting its retry policy, the workflow transitions to COMPENSATING. Any previously completed steps that define a compensating_action are executed in reverse topological order until they reach ROLLED_BACK and the workflow reaches COMPENSATED.

Durable Timers (sleep) & Signal Gates (wait_for_signal)

  • Sleep Timers: Setting "sleep": "10m" records a deterministic WakeAt timestamp in the Raft FSM. The timer survives node restarts and completes once the cluster clock passes WakeAt.
  • Signal Gates: Setting "wait_for_signal": "steward-approval" places the step in WAITING_SIGNAL. When a signal arrives via POST /api/v1/workflows/{id}/signal, its JSON payload becomes the step's output. Signals sent early are buffered in the workflow state.

Durable Cron Schedules

Schedules are stored in the Raft FSM and support standard 5-field cron expressions, IANA timezones, and overlap policies (SKIP, ALLOW, or CANCEL_PREVIOUS):

bash
curl -X POST http://localhost:8080/api/v1/schedules \
  -H "Content-Type: application/json" \
  -d '{
    "id": "hourly-compaction",
    "workflow_name": "data-ingestion-pipeline",
    "cron": "0 * * * *",
    "timezone": "UTC",
    "overlap_policy": "SKIP",
    "input": {"enable_embeddings": true},
    "metadata": {"env": "prod"}
  }'

Data Pipeline Patterns

1. Progress Checkpointing (HeartbeatWithProgress & LastProgress)

Long-running activities can include a JSON progress payload when sending heartbeats. The payload is persisted on the step state in Raft (`StepState.LastProgress`). If a worker disconnects or a leader fails over, the retried task receives `LastProgress` so it can continue from the last checkpoint.

python
@worker.activity("backfill-partition")
async def backfill_partition(ctx: TaskContext, job: dict) -> dict:
    state = ctx.last_progress or {"offset": 0, "rows": 0}
    offset = state["offset"]
    rows = state["rows"]

    while offset < job["total_rows"]:
        batch = read_rows(job["table"], offset=offset, limit=50_000)
        write_rows(batch)
        offset += len(batch)
        rows += len(batch)
        await ctx.report_progress({"offset": offset, "rows": rows})

    return {"rows_written": rows}

2. Targeted Step Replay (/replay)

When a downstream step fails after expensive upstream steps have already completed, you can reset only the failed step (and its transitive dependents) back to PENDING while preserving upstream outputs:

bash
curl -X POST http://localhost:8080/api/v1/workflows/etl-run-101/replay \
  -H "Content-Type: application/json" \
  -d '{"step": "cleanse-records"}'

3. Selective Retries & Error Codes

Steps can configure max_attempts, initial backoff (exponentially doubled per attempt), and non_retryable_errors. If an activity returns a matching error code, retries are skipped immediately:

go
// Return a non-retryable coded error in Go:
return Output{}, worker.NewCodedError("ERR_CORRUPT_SCHEMA", errors.New("missing column: timestamp"))
python
# Raise a non-retryable coded error in Python:
from plexus_flow import ActivityFailure
raise ActivityFailure("missing column: timestamp", code="ERR_CORRUPT_SCHEMA", retryable=False)

Workflows can be queried by status, definition name, and arbitrary metadata key-value pairs via the embedded inverted index:

bash
curl "http://localhost:8080/api/v1/workflows?status=FAILED&name=data-ingestion-pipeline&meta.env=prod&limit=50"

Go SDK Reference

Use worker.RegisterTyped to register strongly typed activity handlers, workerclient.TaskFromContext to inspect dependency outputs and checkpoints, and workerclient.New to connect to a single server or a multi-node cluster.

go
package main

import (
    "context"
    "encoding/json"
    "log"
    "github.com/avivklas/plexus-flow/pkg/worker"
    "github.com/avivklas/plexus-flow/pkg/workerclient"
)

type CleanseInput struct {
    DatasetURI string `json:"dataset_uri"`
}

type CleanseOutput struct {
    RowsWritten int    `json:"rows_written"`
    OutputURI   string `json:"output_uri"`
}

func main() {
    reg := worker.NewRegistry()

    worker.RegisterTyped(reg, "spark.cleanse", func(ctx context.Context, in CleanseInput) (CleanseOutput, error) {
        info, _ := workerclient.TaskFromContext(ctx)
        _ = info.LastProgress                          // Checkpoint from prior attempt, if any
        _ = info.DependencyOutputs["extract-parquet"]  // Upstream dependency output JSON

        _ = workerclient.ReportProgress(ctx, json.RawMessage(`{"pct": 50}`))

        return CleanseOutput{
            RowsWritten: 100000,
            OutputURI:   "s3://curated/clean.parquet",
        }, nil
    })

    // Comma-separated addresses enable automatic failover if the current leader stops
    w := workerclient.New("10.0.0.1:9090,10.0.0.2:9090,10.0.0.3:9090", workerclient.Options{
        WorkerID:       "go-worker-1",
        MaxConcurrency: 16,
        Registry:       reg,
    })

    if err := w.Run(context.Background()); err != nil {
        log.Fatal(err)
    }
}

Python SDK Reference

Install the package from sdk/python (pip install -e sdk/python):

python
import asyncio
from plexus_flow import Worker, TaskContext, ActivityFailure, WorkflowClient

worker = Worker("localhost:9090", worker_id="py-worker-1", max_concurrency=4)

@worker.activity("gpu.embed_chunks")
async def embed_chunks(ctx: TaskContext, payload: dict) -> dict:
    if "dataset_uri" not in payload:
        raise ActivityFailure("dataset_uri is required", code="ERR_INVALID_INPUT", retryable=False)

    extracted = ctx.dependency_outputs.get("extract-parquet", {})
    await ctx.report_progress({"completed_batches": 5, "source": extracted.get("uri")})
    return {"vectors": 50000}

# Start a workflow and stream updates over SSE:
client = WorkflowClient("http://localhost:8080")
wf = client.start_workflow(
    "data-ingestion-pipeline",
    {"dataset_uri": "s3://raw/events.parquet", "enable_embeddings": True},
    workflow_id="ingest-42",
    metadata={"tenant": "acme"}
)

for state in client.watch("ingest-42", timeout=60.0):
    if state["status"] in ("COMPLETED", "FAILED", "COMPENSATED"):
        print("Terminal status:", state["status"])
        break

HTTP & SSE API Reference

Mutating requests sent to a follower node are forwarded to the current Raft leader.

Endpoint Description
POST /api/v1/definitions Register or update a workflow definition.
GET /api/v1/definitions List all registered workflow definitions.
POST /api/v1/workflows Start a workflow instance (id, workflow_name, input, metadata).
GET /api/v1/workflows Search workflows (?status=...&name=...&meta.key=val&offset=0&limit=50).
GET /api/v1/workflows/{id} Get workflow instance details (falls back to cold archive if compacted).
POST /api/v1/workflows/{id}/signal Send a named signal ({"signal": "...", "payload": {...}}) to a workflow.
POST /api/v1/workflows/{id}/replay Reset a failed step ({"step": "..."}) and its downstream dependents to PENDING.
POST /api/v1/workflows/{id}/cancel Cancel a running workflow.
GET /api/v1/workflows/{id}/events Server-Sent Events stream for a single workflow instance.
GET /api/v1/events Server-Sent Events stream for all workflow transitions.
POST /api/v1/schedules Create or update a cron schedule.
GET /api/v1/cluster/status Return Raft leadership status, connected workers, and active leases.
POST /api/v1/admin/compact Compact terminal workflows older than a specified duration.
GET /healthz & GET /readyz Liveness and Raft quorum readiness endpoints.

Multi-Node Raft Cluster Setup

To run a 3-node cluster, start node-1 as the bootstrap node and join node-2 and node-3 using --join (for FlowStore) and --act-join (for ActivityStore):

bash
# Node 1 (bootstrap)
plexus-flow \
  --id node-1 \
  --http-addr 0.0.0.0:8081 \
  --grpc-addr 0.0.0.0:9091 \
  --raft-addr 127.0.0.1:7001 \
  --act-raft-addr 127.0.0.1:7002 \
  --data-dir ./data/node-1

# Node 2 (joins Node 1)
plexus-flow \
  --id node-2 \
  --http-addr 0.0.0.0:8082 \
  --grpc-addr 0.0.0.0:9092 \
  --raft-addr 127.0.0.1:7003 \
  --act-raft-addr 127.0.0.1:7004 \
  --join 127.0.0.1:7001 \
  --act-join 127.0.0.1:7002 \
  --data-dir ./data/node-2

# Node 3 (joins Node 1)
plexus-flow \
  --id node-3 \
  --http-addr 0.0.0.0:8083 \
  --grpc-addr 0.0.0.0:9093 \
  --raft-addr 127.0.0.1:7005 \
  --act-raft-addr 127.0.0.1:7006 \
  --join 127.0.0.1:7001 \
  --act-join 127.0.0.1:7002 \
  --data-dir ./data/node-3

CLI Flags & Environment Variables

Flag Environment Variable Default Description
--id PLEXUS_FLOW_NODE_ID node-1 Unique node ID.
--http-addr PLEXUS_FLOW_HTTP_ADDR :8080 HTTP API and web UI listen address.
--grpc-addr PLEXUS_FLOW_GRPC_ADDR :9090 gRPC worker broker listen address.
--raft-addr PLEXUS_FLOW_RAFT_ADDR 127.0.0.1:7001 Upstream (FlowStore) Raft bind address.
--act-raft-addr PLEXUS_FLOW_ACT_RAFT_ADDR 127.0.0.1:7002 Downstream (ActivityStore) Raft bind address.
--advertise-raft-addr PLEXUS_FLOW_ADVERTISE_RAFT_ADDR (same as --raft-addr) Advertised address for FlowStore Raft peers.
--advertise-act-raft-addr PLEXUS_FLOW_ADVERTISE_ACT_RAFT_ADDR (same as --act-raft-addr) Advertised address for ActivityStore Raft peers.
--join / --act-join PLEXUS_FLOW_JOIN / PLEXUS_FLOW_ACT_JOIN "" Existing cluster addresses to join as a voter.
--data-dir PLEXUS_FLOW_DATA_DIR ./data Directory for BoltDB Raft logs and snapshots.
--flows PLEXUS_FLOW_FLOWS "" Comma-separated paths/globs of JSON workflow definitions to load on startup.
--retention-ttl PLEXUS_FLOW_RETENTION_TTL 0 TTL after which terminal workflows are compacted.
--max-completed-workflows PLEXUS_FLOW_MAX_COMPLETED_WORKFLOWS 0 Maximum completed workflows kept in hot memory.

Retention & Cold Archival

Configuring --retention-ttl or --max-completed-workflows runs a periodic compaction loop that removes older terminal workflows from the in-memory Raft FSM. When a ColdArchive implementation is configured on the coordinator, compacted workflows are archived prior to deletion and transparently fetched on GET /api/v1/workflows/{id}.

Running the Test Suite

bash
# Run all unit and integration tests with the race detector
go test -race ./...

# Run the cluster failover and concurrency tests
go test -v -race -run TestBattle ./test