Skip to content
Thijn de Haas CTO @ Wux
RSS feed
en nl

thijn@wux:~/blog$ cat laravel-apex-de-queue-worker-supervisor.md

[note] 5 min read

Laravel Apex: the queue worker supervisor

Horizon is the right answer for one app with steady traffic. We needed something else: many installations on one server, most of them idle most of the time.

Contents
  1. Where Horizon stops fitting
  2. What Apex does differently
  3. Choices that look strange
  4. How Symphoria configures it
  5. Lessons in the config
  6. When to use it

Apex is a Symphoria package, extracted from the supervisor that runs Symphoria's own installations. For a single Laravel application with steady traffic, Horizon is the easy answer: first-party, a dashboard, and everyone already knows it. Our situation is different: many installations sharing one machine, most of them idle most of the time.

Where Horizon stops fitting

Horizon's minimum is one process per queue, and it has no notion of the machine as a whole. Twenty idle installations cost twenty times a worker with nothing to do. And when they wake up, they all boot at once.

That last part is the real problem. A queue going from 0 to 8 workers means eight simultaneous framework boots. Several installations doing that together will flatten a server.

What Apex does differently

Apex is a supervisor, not a queue backend. Jobs stay on Laravel's redis or database driver; Apex reads the backlog from the queue and decides how many workers each queue needs.

  • The floor is zero. A queue with min_processes: 0 runs no workers when nothing is queued. What stays is one master process per installation.
  • A warm pool while people are around. min_processes_when_active keeps workers ready while there is human traffic, so users never wait for a cold start, and nights and weekends are free. Dashboard polling does not count as activity, so the monitoring screen cannot create the load it displays.
  • Boots are throttled globally. At most max_concurrent_boots workers boot at the same time, spaced by boot_spawn_interval_ms.
  • Up now, down slowly. Scaling up is immediate; scaling down waits out a debounce, so a short dip cannot start a kill/spawn loop.
  • Bursting yields to the CPU. A queue that stays busy may go past its maximum, one worker per tick, unless the CPU is already above cpu_max_percent.

Choices that look strange

  • No dashboard. Apex ships a JSON API and no screen. The package owns the data and the control actions; the application owns the UI.
  • No file store. That would mean writing our own locking, which fails differently on Windows and silently on network shares. For "one file, no daemon", use SQLite with WAL.
  • One master per installation. A second apex:start refuses and says which host and pid hold the lock. Two masters cannot see each other's workers and would scale the same queues past every maximum.
  • Migrations are loaded, not published. Publishing makes you the owner of the schema, and later column additions would never reach you.

How Symphoria configures it

Symphoria's own config follows one rule: one queue per kind of waiting, not per module. Two facts shape it. A worker costs a full framework boot, so a pool is only kept warm where a person notices the boot. And a job that waits for another job must live on a different queue than the job it waits for.

  • Chat stays warm. chat keeps one worker even when nobody is around, plus a standby while people are active: someone is watching the answer arrive.
  • Tool calls get their own queue. A chat run fans out tool calls and waits for them. On chat the waiting run would block the very worker its tool needs, so tools is separate and keeps one worker permanently warm.
  • Batches burst, carefully. Importing a folder queues hundreds of jobs at once and then nothing for hours. ingestion runs at most two workers, and only adds helpers once the backlog has held for 15 seconds; those helpers are CPU-gated, low priority and short-lived.
  • Watchdogs never queue behind the work they rescue. They get their own small queue; on default a watchdog would arrive after the damage.
  • Housekeeping scales to zero. Background queues run no workers at all when nothing is queued.
'tools' => [
    'profile' => 'latency-critical',
    'group' => 'chat',
    'min_processes' => 1,
    'min_processes_when_active' => 2,
    'max_processes' => 8,
    'use_activity' => true,
    'timeout_seconds' => 60,
    'burst_threshold_depth' => 4,
    'burst_threshold_sustained_ms' => 1000,
    'burst_max_extra' => 4,
],

Lessons in the config

  • Timeouts are ordered. Every queue's timeout_seconds is at or above the longest $timeout of its jobs, and retry_after in config/queue.php sits above all of them. Otherwise a slow job is handed out twice.
  • Deploys wait for answers. A restart stops workers gracefully and kills whatever is still busy after a window. That window has to cover the longest interactive job, or every deploy kills answers in flight. The master exits as soon as the last worker is gone, so an idle installation still restarts within a second.
  • Idle means quiet. Polling four times a second with no workers, no backlog and no users dominated Redis traffic across all installations. Now an idle master checks every five seconds and returns to the normal cadence on the next tick once anything changes.
  • No OPcache in workers by default. It saves about 170 ms of boot, but CLI OPcache is not shared between processes: each worker pays 30 to 45 MB extra, and with validate_timestamps=0 a fresh worker can run stale code after a deploy.

When to use it

One app with steady traffic: stay on Horizon. With Apex you give up its dashboard and first-party support. Apex is for the case where many installations share a server and idle workers are the main cost.

Symphoria publishes the code on GitHub under the MIT license: composer require symphoria/laravel-apex.

Welcome! Have a look around.