---
title: "Reliable execution and recovery"
description: "Create runs idempotently, classify retries, understand leases, and recover without repeating unknown side effects blindly."
category: "Operate Pipelogiq"
---

Pipelogiq stores orchestration state in PostgreSQL and delivers work through RabbitMQ. Delivery and recovery are at-least-once oriented. Execution leases and attempt identifiers prevent conflicting ownership and stale results from advancing the current run. They do not make an external payment, message, or API call exactly once.

## Make run creation safe to repeat

Choose one stable idempotency key for the business operation. Reuse it after a timeout or disconnected response. Use `POST /pipelines/idempotent`, not the legacy create route.

Assume `PIPELOGIQ_API_KEY` contains an application key and your application worker registers `ValidateInvoiceHandler`:

```bash
curl --fail-with-body -i http://localhost:8081/pipelines/idempotent \
  -H "X-API-Key: $PIPELOGIQ_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON'
{
  "idempotencyKey": "invoice-42:validate:v1",
  "name": "validate-invoice",
  "stages": [
    { "stageName": "validate", "stageHandlerName": "ValidateInvoiceHandler", "input": "{\"invoiceId\":\"invoice-42\"}" }
  ]
}
JSON
```

| Response | Meaning |
| --- | --- |
| 201 | A new run was created. Save its pipeline ID. |
| 200 | An equivalent request already created the run; the response identifies it. |
| 409 | The same application already used this key for different creation intent. |

The key is scoped to an application and retained with the pipeline. A differently ordered list of stages changes the request. Trace metadata may change without creating a conflict. Retention/deletion also removes the stored pipeline identity; do not use pipeline storage as your only permanent business deduplication ledger.

After an unknown HTTP outcome, look up the same key:

```bash
curl --fail-with-body http://localhost:8081/pipelines/by-idempotency-key \
  -H "X-API-Key: $PIPELOGIQ_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"idempotencyKey":"invoice-42:validate:v1"}'
```

## Classify retries

Configure retry options on stages that can fail transiently:

```json
{
  "stageName": "send",
  "stageHandlerName": "SendInvoiceHandler",
  "options": {
    "maxRetries": 3,
    "retryInterval": 10,
    "retryOnErrorCodes": ["TIMEOUT", "UPSTREAM_ERROR", "RATE_LIMIT_EXCEEDED", "TRANSPORT_UNAVAILABLE"],
    "backoff": "exponential",
    "maxRetryInterval": 120,
    "jitter": true,
    "timeOut": 60
  }
}
```

Stage intervals and timeout use seconds. `maxRetries` counts retries after the first execution. Both a positive maximum and positive interval are needed for the stage-options fallback. Fixed, linear, and exponential backoff are supported.

The worker must report an appropriate `errorCode`. A nonempty allowlist retries only matching codes. `retryable:false` suppresses automatic retry. Business rejection, invalid state, validation failure, and missing required data are terminal classifications even if included in a retry list. Correct the business input before retrying them.

A matched action retry policy takes precedence over stage retry options. A policy whose filter does not match does not fall back to the stage configuration. Inspect [effective policies](/docs/policies) when behavior differs from the stage options.

## Leases and result delivery

A custom HTTP worker should follow this order:

1. Bootstrap with the application's API key and register handler names.
2. Pull a job from the application/handler queue.
3. Acquire the lease for the job's `executionId`; execute only when `acquired` is true.
4. Renew the lease while doing long-running work.
5. Submit a result containing the job's stage ID, execution ID, and attempt.
6. Acknowledge the delivery only after result submission returns 202.

The 202 response means the broker confirmed result acceptance. The orchestration worker applies it asynchronously, so the dashboard may update shortly afterward. A 503 means the result was not confirmed; retry delivery of the same result with backoff rather than executing the business action again.

Recovery can replace an expired execution. A handler may still be running when its lease expires; stop cooperatively when ownership is lost. Use an upstream idempotency key or check the external system's outcome before repeating a side effect.

## Pause, rerun, skip, and cancel

**Pause** stops further dispatch from a nonterminal pipeline; in-flight handlers can finish. **Resume** makes a paused pipeline eligible again. Neither resets completed work.

**Rerun stage** queues that stage again and clears its stored output and retry state. The dashboard uses a single-stage rerun; it does not rewind later stages. The internal API additionally supports resetting subsequent stages. Treat that operation as replaying side effects, not as automatic compensation.

**Skip stage** records that work was intentionally omitted and may let dependent work continue. Use it only when the business process can safely proceed without that result.

**Cancel** is an SDK/external API operation. It makes the pipeline and unfinished stages terminal and fences late results. It cannot undo an external side effect already started. A cancelled run is not resumed with the ordinary Resume operation.

## Approval waits

A waiting result parks the stage without completing the run. Your application submits the decision to `POST /stages/{stageId}/resume` with an application key:

```json
{ "approved": true }
```

For rejection:

```json
{ "approved": false, "rejectionReason": "The requested operation was declined." }
```

An identical repeated decision and reason is accepted; a conflicting decision returns 409. Generic approval completes the stage, while generic rejection fails it. Integrate the decision with your own authorization and business audit requirements. There is no built-in dashboard approval inbox or Slack approval workflow in this guide.

## Before an operator retries

Read the error and attempt history, check whether the external operation already succeeded, resolve the cause, and confirm that the handler is idempotent or can reconcile the previous outcome. Then use the failed-stage action in the [dashboard](/docs/dashboard). Persist business operation identifiers in context so the next operator can follow the same evidence.