Workflow runs stay Pending: automation is off, its queue is down or processing has stopped
Rules keep saving and events keep being recorded, but nothing runs while automation is switched off on the server, the queue that carries changes to it is missing or down, or processing has stopped; the Health tab shows which, the NirmaanOne team fixes it, and the waiting work is then processed.
Messages this explains
- “Automation is switched off: rules are saved and changes are kept, but runs stay Pending until it is fixed.”
- “The queue that carries changes to the automation is not set up: rules are saved and changes are kept, but runs stay Pending until it is fixed.”
- “The queue that carries changes to the automation is not answering: rules are saved and changes are kept, but runs stay Pending until it is fixed.”
- “Waiting to be picked up.”
- “Backlog is stuck”
- “The engine is working on it. You can close this and check run history later.”
- “Disabled”
- “Not configured”
- “Unreachable”
- “Unknown”
- “Backlog is stuck”
- “Nothing dispatched yet”
- “Quiet for a while”
- “not processed after <n> dispatches”
- “No schedule yet.”
- “Only failed events can be requeued.”
Workflow runs stay Pending: automation is off, its queue is down or processing has stopped
Rules are switched on, but nothing happens: runs sit at Pending, events pile up on the outbox, a workflow started from a record never finishes, or automatic demand or payout runs never appear.
What you see
The engine warning. People with the Edit permission on Workflow Rules see a yellow bar above the Rules, Event types, Message templates, Runs and Outbox tabs of Workflows, ending in Open Health:
- "Automation is switched off: rules are saved and changes are kept, but runs stay Pending until it is fixed."
- "The queue that carries changes to the automation is not set up: rules are saved and changes are kept, but runs stay Pending until it is fixed."
- "The queue that carries changes to the automation is not answering: rules are saved and changes are kept, but runs stay Pending until it is fixed."
Nobody else sees the warning; to them, runs simply stay Pending.
On Health. The Health tab shows the cause in red or amber: Automation "Off"; Queue "Not set up" or "Not answering"; Processing "Unknown" (it can't be checked while the queue is not answering), "Backlog is stuck" (the oldest change has waited more than two minutes although the queue answers), "Nothing picked up yet" or "Quiet for a while". Oldest pending turns red after two minutes. Stuck processing shows only here, without the yellow bar.
On Runs and Outbox. On Runs, runs you started by hand (or retried) stay Pending, and the run panel says "Waiting to be picked up." Runs from events and schedules don't appear at all yet: their events wait on the Outbox as Pending or Dispatched. An event that was picked up but not processed within 30 minutes is handed over again; on the fifth attempt it turns Failed with "not processed after <n> dispatches". Requeue on an event that has already moved on answers "Only failed events can be requeued."
On a record. After you start a rule from the Workflows menu, the progress dialog keeps saying "The engine is working on it. You can close this and check run history later."
On the run lists. The Automatic demand runs card on Demand runs, or Automatic payout runs on Payout runs, reads "No schedule yet." when no scheduled rule creates those runs.
Why it happens
Workflows run in the background, apart from the screens you work on, so that a slow email or a call to another system never holds up saving a booking or a payment. Three parts must work together:
- The automation switch. The NirmaanOne team can switch automation off on the server. Rules still save and events are still recorded, but nothing is processed.
- The queue, which carries each event and run to the automation. If none is set up, or it doesn't answer, events wait in the outbox.
- Processing and the scheduler, the background part that carries out runs, checks scheduled rules every minute and raises the daily due-date events at 08:00. If it stops, the backlog grows even though the queue answers.
Nothing is lost while any of these is down: the events stay on the outbox and the runs stay Pending, and they are processed in order once the problem is fixed. Only an event that failed five times needs a nudge.
"No schedule yet." is different: nothing is broken, but no scheduled rule creates those runs, so demand or payout runs are only made when someone starts one. A scheduled rule also stops by itself when its Until date has passed, and an inactive rule never runs.
How to fix it
- Check Health. Open Health (Account menu > Settings > Workflows > Health, or Open Health on the warning). You need the Edit permission on Workflow Rules; in the default roles only the Administrator has it. If you don't, ask your administrator to look.
- Open a case with the NirmaanOne team, quoting the red or amber card. Its detail line says whether automation is off, the queue is not set up or not answering, or processing has stopped; the team puts it right on the server. There is nothing to change in the app itself.
- Wait for the backlog to clear. Once Automation reads On, Queue Connected and Processing Idle or Processing, waiting events are picked up within seconds and Pending runs are picked up (a run started by hand is picked up about a minute after it was queued). Click Refresh on Runs or Outbox to follow.
- Requeue failed events. On Outbox, filter Status to Failed, open each event and press Requeue (the Edit permission on Workflow Rules). You see "Event requeued; it is picked up within a few seconds."
- Retry errored runs that failed while the engine was struggling: open them on Runs and press Retry (the Edit permission on Workflow Runs).
- For "No schedule yet.", click Schedule automatic demand runs (or Schedule automatic payout runs) on the card to build a scheduled rule from a ready-made starting point, or switch on "Daily demand runs (system)" or "Daily payout runs (system)" on Rules. Both need the Create or Edit permission on Workflow Rules, and switching on also needs the permissions the run's actions use (for demand runs the Create and Issue permissions on Demands). If a scheduled rule is listed but shows Inactive, open it and check its Until date before switching it back on.
Related
Related
- HealthScreen
Whether the automation engine can process what the rules produce: the engine flag, its message queue, its worker, the outbox and run counts, and the scheduler's heartbeat.
- RunsScreen
Every time a rule ran: what started it, which record it worked on, how each action went, and the panel from which an errored run is retried.
- OutboxScreen
The events recorded when records change, on their way to the rules that listen to them, with the runs each event produced and the panel from which a failed event is requeued.
- WorkflowsScreen
The Workflows area of Settings, where you manage automation rules, event types and message templates, and check the runs, the outbox and the engine's health.
- RulesScreen
Every automation rule in your organization, system and custom, with filters, the switch that turns each rule on or off, and shortcuts to edit, test, clone, see the run history of or delete a rule.
- Demand runsScreen
Every demand run of the chosen books, the dialog that previews and drafts a run billing everything due on a project, and the scheduled rules that create runs automatically.
- Payout runsScreen
Every payout run of the chosen books, the dialog that drafts a new run of member benefits, commissions or refunds, and the scheduled rules that draft runs automatically.
- A run's action errored: webhook refused, empty role, empty reason, unpublished template or the chain ran too deepTroubleshooting
A workflow run stops as Errored when one of its actions fails: the webhook address is internal or the other system refused the call, the role to assign to has no active members, a cancellation or lost reason came out blank, or the document template is not published; a run is Blocked when a chain of rules emitting events runs deeper than the engine allows.
- Run errored or blocked with "permission_revoked": the rule's owner no longer holds a permission its actions needTroubleshooting
A rule's automatic runs act as the person who last saved or switched the rule, and manual runs as the person who pressed the button; each action is checked against that person's permissions when it runs, so a role change or a deactivated user makes the run fail until the permission is restored or someone who holds it takes the rule over.
- Workflow runs and the outboxConcept
How a change becomes automation: the outbox event written with the change, the pick-up that hands it to the engine, one run per listening rule with a recorded result per action, the statuses each passes through, and how Errored runs are retried and Failed events requeued.
- Triggers and event typesConcept
The five ways a workflow rule starts (On Create, On Update, On Event, Scheduled, Manual), the catalogue of system event types the app raises, the daily due-date events for instalments and overdue demands, how recurrences are counted, and custom event types that chain rules.
- System rules and starter templatesConcept
The 53 automation rules, six email templates and six draft document templates every organization starts with: which ship switched on, why the customer-facing ones ship off, what is locked, and how to switch, clone or customise them.