Heartbeat monitoring with automatic restart, exponential backoff, and escalation.

SelfHealer

The SelfHealer monitors registered processes via heartbeats and error rates. When a process degrades or fails, it automatically diagnoses the issue, restarts the process with exponential backoff, and escalates to alerts when restart attempts are exhausted.

import { SelfHealer } from '@codmir/cortex';
import type { RestartFn } from '@codmir/cortex';

Constructor

new SelfHealer(config?: Partial<SelfHealerConfig>)

SelfHealerConfig

interface SelfHealerConfig {
  heartbeatExpectedMs: number;              // default: 15000
  heartbeatDriftThresholdMs: number;        // default: 5000
  maxConsecutiveFailures: number;           // default: 3
  restartBackoffMs: number;                // default: 5000
  maxRestartAttempts: number;              // default: 5
  degradedErrorRateThreshold: number;      // default: 0.1
  unhealthyErrorRateThreshold: number;     // default: 0.5
  checkIntervalMs: number;                // default: 10000
}
PropertyDefaultDescription
heartbeatExpectedMs15000Expected interval between heartbeats
heartbeatDriftThresholdMs5000Drift above expected that triggers degraded
maxConsecutiveFailures3Consecutive failures before unhealthy
restartBackoffMs5000Base backoff for restart attempts
maxRestartAttempts5Max restarts before escalation to alert
degradedErrorRateThreshold0.1Error rate (0-1) threshold for degraded
unhealthyErrorRateThreshold0.5Error rate threshold for unhealthy
checkIntervalMs10000Default healing check interval

Methods

register()

Registers a process for monitoring. Initializes it as healthy with zero drift and no failures.

register(pid: number): void
healer.register(1001);
healer.register(1002);

unregister()

Removes a process from monitoring.

unregister(pid: number): void

heartbeat()

Records a heartbeat for a process, resetting its consecutive failure count and updating the last heartbeat timestamp.

heartbeat(pid: number): void
// Call periodically from each worker
healer.heartbeat(1001);

recordRequest()

Records a request outcome for a process, updating the EMA response time and error rate.

recordRequest(pid: number, durationMs: number, success: boolean): void
ParameterTypeDescription
pidnumberProcess identifier
durationMsnumberRequest duration in milliseconds
successbooleanWhether the request succeeded
healer.recordRequest(1001, 250, true);
healer.recordRequest(1001, 0, false); // failure

onRestart()

Registers the restart function called when the healer decides to restart a process.

onRestart(fn: RestartFn): void
type RestartFn = (pid: number) => Promise<boolean>;
healer.onRestart(async (pid) => {
  await killProcess(pid);
  await spawnProcess(pid);
  return true; // return false if restart failed
});

check()

Runs a full healing cycle. For each registered process, diagnoses its health status. If the status changed, takes the appropriate action (restart or alert) and returns the resulting healing events.

async check(): Promise<HealingEvent[]>

Restart uses exponential backoff: restartBackoffMs * 2^(restartCount - 1), capped at 30 seconds.

const events = await healer.check();
for (const event of events) {
  console.log(`[${event.pid}] ${event.previousStatus} -> ${event.newStatus}: ${event.action} (${event.reason})`);
}

getHealth()

Returns the health status for a single process.

getHealth(pid: number): ProcessHealth | undefined

getAllHealth()

Returns the health status for all registered processes.

getAllHealth(): ProcessHealth[]

getEventLog()

Returns the most recent healing events.

getEventLog(limit?: number): HealingEvent[]
ParameterTypeDescription
limitnumber (optional)Max events to return (default: 50)

flush()

Clears all registered processes and the event log.

flush(): void

Types

HealthStatus

type HealthStatus = 'healthy' | 'degraded' | 'unhealthy' | 'dead';

Diagnosis rules:

ConditionStatus
Heartbeat drift > heartbeatExpectedMs * 3dead
Consecutive failures >= maxConsecutiveFailuresunhealthy
Error rate > unhealthyErrorRateThresholdunhealthy
Heartbeat drift > expected + threshold, or error rate > degradedErrorRateThresholddegraded
Otherwisehealthy

HealingAction

type HealingAction = 'none' | 'restart' | 'migrate' | 'scale_down' | 'alert';

Action selection:

StatusRestarts RemainingAction
healthyanynone
degradedanynone
unhealthyyesrestart
unhealthynoalert
deadyesrestart
deadnoalert

ProcessHealth

interface ProcessHealth {
  pid: number;
  status: HealthStatus;
  heartbeatDriftMs: number;
  errorRate: number;
  avgResponseMs: number;
  consecutiveFailures: number;
  lastAction: HealingAction;
  lastActionAt: number;
}

HealingEvent

interface HealingEvent {
  pid: number;
  action: HealingAction;
  reason: string;
  previousStatus: HealthStatus;
  newStatus: HealthStatus;
  timestamp: number;
  success: boolean;
}

Example

import { SelfHealer } from '@codmir/cortex';

const healer = new SelfHealer({
  heartbeatExpectedMs: 10_000,
  maxConsecutiveFailures: 2,
  maxRestartAttempts: 3,
  restartBackoffMs: 2000,
});

// Register restart handler
healer.onRestart(async (pid) => {
  console.log(`Restarting process ${pid}...`);
  // Restart logic here
  return true;
});

// Register processes
healer.register(1);
healer.register(2);

// Simulate healthy heartbeats
healer.heartbeat(1);
healer.heartbeat(2);

// Simulate failures on process 2
healer.recordRequest(2, 0, false);
healer.recordRequest(2, 0, false);

// Run healing cycle
const events = await healer.check();
for (const e of events) {
  console.log(`PID ${e.pid}: ${e.action} — ${e.reason}`);
  // PID 2: restart — 2 consecutive failures
}

// Inspect health
const health = healer.getAllHealth();
for (const h of health) {
  console.log(`PID ${h.pid}: ${h.status}, error rate: ${(h.errorRate * 100).toFixed(1)}%`);
}

// View event log
const log = healer.getEventLog(10);
console.log(`${log.length} healing events recorded`);