Declarative data pipelines, DAGs, and Sagas in a single binary.

plexus-flow is a workflow orchestration engine built on HashiCorp Raft and BoltDB. It lets you define multi-stage data pipelines, fan-out/fan-in DAGs, and compensating Sagas in JSON, and execute steps with standard Go or Python workers—without managing external databases, message queues, or replay determinism rules.

bash — Start a single-node server
go run ./cmd/plexus-flow \
  --id node-1 \
  --http-addr :8080 \
  --grpc-addr :9090 \
  --data-dir ./data
Self-Contained Binary
Embeds Raft consensus, BoltDB storage, gRPC worker broker, inverted search index, and web UI.
No Determinism Constraints
State transitions are tracked in the Raft FSM; workers run ordinary Go and Python functions.
Stateful Checkpointing
Long-running batch steps report progress via heartbeats and resume from LastProgress on failover.
Targeted Step Replay
Re-run a failed downstream transformation step via /replay while keeping upstream outputs intact.

Interactive Pipeline Walkthrough

In plexus-flow, the Raft state machine carries lightweight pointers (S3 keys, database IDs, checkpoint cursors) while workers perform the actual I/O—reading files from object storage, transforming tables, indexing vectors, or calling external APIs.

Raft Cluster (3 Voters)
node-1 (leader) node-2 (voter) node-3 (voter)
Select an action above to step through the workflow state machine.

Primitives for Data-Intensive Platforms

Developing data pipelines often requires gluing together separate systems for scheduling, batch checkpointing, retries, and metadata search. plexus-flow provides these primitives directly in the state machine.

1. Progress Checkpointing

Workers processing large datasets can attach JSON progress payloads to periodic heartbeats. Checkpoints are replicated via Raft (last_progress), allowing a replacement worker to resume from the last committed offset after a crash.

python
state = ctx.last_progress or {"offset": 0}
for batch in read_batches(start=state["offset"]):
    process(batch)
    await ctx.report_progress({"offset": batch.next_offset})

2. Targeted Step Replay

If a downstream step fails due to a bug or schema mismatch, you can deploy a fix and call /replay on that specific step. Upstream steps remain COMPLETED while the target step and its dependents are reset to PENDING.

bash
curl -X POST http://localhost:8080/api/v1/workflows/etl-42/replay \
  -H "Content-Type: application/json" \
  -d '{"step": "build-vector-index"}'

3. Standard Worker Code

Because orchestration state is stored explicitly in the Raft FSM rather than reconstructed by re-running workflow code, worker activities have no determinism restrictions and can use any standard library or native driver.

go
worker.RegisterTyped(reg, "embed-chunks",
    func(ctx context.Context, in EmbedReq) (EmbedRes, error) {
        return runEmbeddingModel(ctx, in.DatasetURI)
    })

4. Fork-Join DAGs & Conditions

Steps declare upstream dependencies with depends_on for parallel fan-out and join synchronization. Optional when predicates inspect workflow inputs, metadata, or upstream outputs to skip unneeded branches.

json
{
  "name": "publish-table",
  "activity": "iceberg.commit",
  "depends_on": ["validate-schema", "redact-pii"],
  "when": {
    "source": "validate-schema",
    "field": "valid",
    "op": "eq",
    "value": true
  }
}

5. LIFO Saga Compensation

When a workflow spans external resources that lack two-phase commit, each step can specify a compensating_action. On terminal step failure, completed steps are compensated in reverse topological order.

json
{
  "name": "reserve-inventory",
  "activity": "inventory.reserve",
  "compensating_action": "inventory.release"
}

6. In-Memory Faceted Search

Workflows can be tagged with key-value metadata. An embedded inverted index is updated synchronously as Raft log entries apply, supporting filtered queries without external search infrastructure.

bash
curl "http://localhost:8080/api/v1/workflows?status=RUNNING&meta.tenant=acme"

Architecture & Design Trade-Offs

plexus-flow pairs an upstream workflow state machine (FlowStore) with a downstream task lease queue (ActivityStore) over a deterministic Raft Graph pipe.

Aspect Plexus-Flow Approach Operational Trade-Off
Deployment Footprint Single Go binary with embedded Raft & BoltDB No external database or broker to operate; requires local disk for Raft logs and snapshots.
Workflow Model Declarative JSON DAGs + stateless Go/Python workers Simple to reason about and free of replay determinism bugs, though less flexible than arbitrary Turing-complete workflow code.
Dispatch Guarantees Deterministic Raft Graph key (UpstreamID:Term:Index:Seq) + lease attempts Prevents duplicate task creation across leader failovers; external side effects should still use the provided task key or checkpoint token for idempotency.
State Storage Active workflows in memory (backed by BoltDB); terminal workflows compacted to cold archive Provides sub-millisecond reads and search for active runs, bounded by configuring --retention-ttl and --max-completed-workflows.

Cluster & Concurrency Verification

The repository includes automated multi-node Raft failover, worker disconnect, and race-detector test suites in test/battletest_test.go.

TestBattle_ThreeNodeCluster_QuorumAndFollowerMutations

Verifies 3-node cluster formation, follower-to-leader HTTP/gRPC command forwarding, and continued operation after a node shutdown.

TestBattle_ThreeNodeCluster_LeaderKillMidFlightAndCheckpointFailover

Kills the Raft leader while a remote worker is streaming progress checkpoints and verifies the new leader resumes the task from LastProgress.

TestBattle_ThreeNodeCluster_LeaderCrashDuringSagaCompensation

Stops the leader mid-rollback and confirms the surviving quorum finishes the remaining LIFO compensation steps.

TestBattle_WorkerChaos_MassDisconnectsAndZombieFencing

Disconnects half of the worker pool during execution and injects delayed completions from superseded attempts to verify lease fencing.

TestBattle_HighConcurrency_ParallelDAGAndSagaStorm

Executes 80 concurrent branching workflows and Sagas across a 4-shard ShardedCoordinator with concurrent search and SSE subscribers under -race.

TestBattle_Deduplication_IdempotencyUnderReplayAndSnapshotRestore

Verifies that duplicate (UpstreamID, Term, Index, Seq) task emissions are ignored across FSM snapshot restores and BoltDB restarts.

bash — Run the cluster and concurrency test suite
go test -v -race -run TestBattle ./test

Embedded Web Dashboard

Every node serves a web interface on its HTTP port for inspecting workflow topologies, step payloads, and execution timelines over Server-Sent Events.

Workflow DAG topology view

Flow Topology View

Displays step dependencies, conditional branches, compensating actions, and current execution status.

Execution trace timeline view

Execution Trace & Step Inspector

Shows per-step timing, retry counts, JSON inputs/outputs, and error messages in real time.