Skip to main content

Locks, deadlines, retries, and circuit breakers

Reliability settings do not hide failures. They define when work may repeat and where it must stop.

Distributed lease​

The action concurrency field limits concurrent writes to the same resource:

concurrency: {
  key: (_context, input) => `guild-settings:${input.guildId}`,
  ttlMs: 8_000,
  renewEveryMs: 2_000,
  waitMs: 250,
}

When LockStore.acquireLease() is available, the kernel supplies a fencing token. A failed renewal is treated as lost ownership.

Why fencing is required​

A worker may pause long enough for its lease to expire. A second worker can finish a write under a newer lease before the first worker resumes. If the domain table stores a fencing token and accepts only a greater token, the stale write is rejected.

UPDATE guild_settings
SET payload = $1, fencing_token = $2
WHERE guild_id = $3
  AND fencing_token < $2;

Deadline​

timeoutMs: 5_000

In 1.1.0, the timeout result is returned at the configured boundary even when the underlying promise ignores AbortSignal. The settlement promise on DeadlineExceededError reports how that work eventually finishes.

Application code should still honor the signal:

async execute(context, input) {
  context.signal.throwIfAborted();
  return repository.update(input, { signal: context.signal });
}

Late-settlement bound​

reliability: {
  maximumLateSettlementMs: 5 * 60_000,
}

This value limits how long idempotency reservation and lease ownership may be retained after the HTTP deadline. Do not use an unbounded value. The idempotency TTL should be longer than the action deadline plus the late-settlement window.

When late work commits, the result is stored for idempotent replay so a later identical request can receive it.

Retry​

retry: {
  attempts: 3,
  baseDelayMs: 50,
  maximumDelayMs: 500,
}

Keep the retryable error set narrow. Validation, authorization, revision conflict, and reservation loss should not be retried.

Before enabling retries, answer:

  • Is the domain write under the same idempotency key?
  • Does the external API accept its own idempotency key?
  • Does this worker still own the lease?
  • Does the next attempt fit inside the total deadline?

Circuit breaker​

circuitBreaker: {
  name: "discord.guild.update",
  failureThreshold: 5,
  resetAfterMs: 30_000,
}

While open, the breaker rejects a new call immediately. This prevents an unavailable dependency from creating an unbounded queue. Do not create breaker keys with unnecessary tenant- or endpoint-level cardinality.

Cache stampede​

CacheManager can use a renewable lease for cache misses. A fallback loader may run while the lock is busy. Preserve the lease and fencing policy when writing the fallback result; do not start uncontrolled parallel refreshes for one key.

Observability​

Measure:

  • timeout count and actual settlement duration
  • retry count
  • breaker state changes
  • lease acquire, renew, and release failures
  • reservation ownership loss
  • rejected stale fencing writes

Return a stable error code to the client while retaining requestId and action name in server logs.

Tests​

  • an operation ignores the signal but the response returns at the deadline
  • a late commit is available through replay
  • ownership is released after maximumLateSettlementMs
  • an old reservation cannot complete a newer record
  • lease-renewal loss stops the action
  • SQL rejects a stale fencing token
  • the breaker performs a controlled call after reset
Last updated on