Operations — RingLoom

This guide covers the operational pieces you need to run RingLoom brokers and services outside the test harness: configuration, metadata storage, monitoring, capacity planning, and troubleshooting.

Broker configuration

The broker loads Java-style properties. Use --config <path> with ringloom-broker or zig build run -- --config <path>. If no path is passed, the loader can fall back to RINGLOOM_CONFIG_FILE and then broker.properties in the current directory.

Minimal single-node config:

broker.node.id=1
broker.local.host.port=127.0.0.1:19001
broker.group.name=ringloom
broker.storage.path=/dev/shm
broker.control.buffer.size=65536
broker.messages.buffer.size=1048576
broker.threading.mode=dedicated
broker.idle.strategy=backoff

Minimal two-node config for node 1:

broker.node.id=1
broker.local.host.port=10.0.0.1:9100
broker.member.host.ports=[email protected]:9100
broker.group.name=ringloom
broker.storage.path=/dev/shm
broker.control.buffer.size=65536
broker.messages.buffer.size=1048576

Node 2 uses the inverse peer list:

broker.node.id=2
broker.local.host.port=10.0.0.2:9100
broker.member.host.ports=[email protected]:9100

Important broker properties

Property	Purpose
`broker.node.id`	Unique broker node ID. Also stored in service metadata and message headers.
`broker.local.host.port`	TCP listen address for broker-to-broker traffic.
`broker.member.host.ports`	Comma-separated peers in `nodeId@host:port` form. Omit for a single-node cluster.
`broker.group.name`	Directory namespace under the storage path. Use one group per isolated deployment.
`broker.storage.path`	Base directory for broker and service metadata files. `/dev/shm` is the normal Linux hot path.
`broker.control.buffer.size`	Broker control ring capacity in bytes.
`broker.messages.buffer.size`	Broker send ring capacity in bytes.
`broker.max.frame.length`	Maximum TCP frame size accepted by the broker.
`broker.peer.write.queue.capacity`	Per-peer outbound write queue capacity.
`broker.threading.mode`	`dedicated`, `shared_network`, or `shared`.
`broker.idle.strategy`	`busy_spin`, `yielding`, `sleeping`, `backoff`, or `blocking`.
`broker.counter.values.buffer.size`	Shared counter values region size.
`broker.error.log.buffer.size`	Shared error-log region size.
`broker.max.services`	Maximum services tracked by the broker.
`broker.max.peers`	Maximum peer brokers tracked by the broker.
`broker.flow.control.enabled`	Enables flow-control metadata regions used by service clients.
`broker.flow.control.peer.send.counters.enabled`	Enables per-peer send-state counters.
`broker.benchmark.latency.tracing.enabled`	Enables additional latency trace timestamps used by benchmarks.

Several numeric sizes are aligned to powers of two during validation. Control and message buffers must be at least 4096 bytes.

Environment overrides

Most broker properties can be overridden with environment variables. Convert the property name to upper case, replace dots with underscores, and prefix RINGLOOM_.

Examples:

RINGLOOM_BROKER_NODE_ID=1
RINGLOOM_BROKER_LOCAL_HOST_PORT=127.0.0.1:19001
RINGLOOM_BROKER_STORAGE_PATH=/dev/shm

Use environment overrides for deployment-time values and keep topology defaults in checked-in properties files.

Storage layout

RingLoom creates metadata under:

<storage_path>/<group>/services/

Common choices:

Storage path	Use case
`/dev/shm`	Normal Linux runtime path for shared-memory IPC.
`/tmp/ringloom-.../storage`	Local tests, e2e harnesses, and throwaway sample runs.
A controlled tmpfs mount	Production deployments that want explicit sizing, ownership, and cleanup policy.

Metadata files are runtime state. Do not put them on a network filesystem. On restart, either reuse known service files only when the owning PID is dead or clean the workspace before starting a fresh experiment.

Service startup and shutdown

A service should start after its local broker metadata file exists. RingLoomEngine.start opens the broker metadata, creates the service metadata, sends registration, waits for the broker response, starts heartbeat updates, and starts the control/message consumer agents.

On graceful shutdown:

stop inbound message processing;
stop the control agent;
send unregister to the broker;
close mapped metadata files;
release cached buffer providers.

When orchestrating multiple processes, stop producers first, then consumers, then brokers. For the order-management sample, the script shuts down in reverse dependency order.

Monitoring with ringloom-stat

ringloom-stat reads metadata files out-of-process and prints broker/service state:

zig build stat -- --storage-path /dev/shm --group ringloom
zig build stat -- --storage-path /dev/shm --group ringloom --broker-node-id 1
zig-out/bin/ringloom-stat /dev/shm/ringloom/services/broker_1.dat

Use it to inspect:

metadata headers and owner PIDs;
heartbeat age;
ring capacity, used bytes, free bytes, and usage ratio;
producer and consumer positions;
allocated counters and error-log state.

Prometheus exporter

The repository includes ringloom-observability, a standalone exporter that maps metadata files read-only and serves Prometheus text exposition.

Build and run:

zig build observability
zig build run-observability -- \
  --storage-path /dev/shm \
  --group ringloom \
  --listen 127.0.0.1:9464

Endpoints:

Endpoint	Purpose
`/metrics`	Prometheus metrics.
`/healthz`	HTTP server liveness.
`/readyz`	Ready after the first successful metadata scan.

The exporter is intentionally out-of-process so broker and service hot paths only update mapped counters and positions.

Capacity planning

Start with the default 64 KiB control rings and 1 MiB message rings for development. Increase capacities when:

discovery or registration traffic bursts fill control rings;
producers observe SendBufferFull on same-host sends;
cross-node traffic fills the broker send ring;
TCP peer queues grow during remote sends;
large payloads approach broker.max.frame.length.

Ring buffers store record headers and alignment padding, so usable payload capacity is less than the raw capacity. As a rule, keep individual messages well below one eighth of the ring size. For large logical payloads, rely on fragmentation-aware paths and test them with the e2e fragmentation scenarios.

Idle strategy and threading choices

Setting	Behavior
`dedicated` threading	Separate broker loops. Best isolation and easiest tuning.
`shared_network`	Shared network loop where appropriate. Useful when conserving threads.
`shared`	Fewer runtime threads, lower isolation.
`busy_spin` idle	Lowest wakeup latency, highest CPU usage.
`backoff` idle	Good default balance for development and most deployments.
`sleeping` or `blocking` idle	Lower CPU usage, higher wakeup latency.

For latency work, isolate CPU-heavy processes, prefer ReleaseFast, and benchmark with the provided scripts rather than browser or IDE workloads running in the background.

Failure handling

Symptom	What to check
Broker exits with config error	Required `broker.node.id` or `broker.local.host.port` missing; invalid peer format; node ID conflicts with a peer.
Service fails to start	Broker metadata file is missing; storage path or group mismatch; broker node ID mismatch.
Service is discovered but sends fail	Target ring is full, no current instance exists, peer is disconnected, or flow-control says the target is congested.
Stale services appear in tools	Check heartbeat age and process liveness. Remove the workspace if this is a local test.
Cross-node traffic does not arrive	Verify both brokers list each other with correct `nodeId@host:port`, ports are reachable, and broker logs show successful peer handshakes.
High ring usage	Increase ring capacity, reduce producer rate, add consumers, or investigate slow handlers.

Production checklist

Before using RingLoom for a long-running deployment:

assign stable, unique broker node IDs;
use a dedicated group name per environment;
size the metadata storage filesystem explicitly;
keep broker and service configs under version control;
expose ringloom-stat or the Prometheus exporter in your operational workflow;
add alerts for heartbeat age, ring usage ratio, process liveness, send failures, and peer connection state;
run zig build test and zig build e2e before deploying changes;
run relevant performance benchmarks before and after latency-sensitive changes;
keep message handlers allocation-free and avoid per-message logging.