Farfalla Queues
Reference of the SQS queues Farfalla dispatches to, what each queue is for, and how queue names are derived per environment. Use this doc when wiring a new job, debugging a stuck pipeline in Horizon, or reading the SQS console.
Manage live queues from the AWS SQS console.
Naming Convention
Queue names follow the pattern {QueueName}-{APP_NAME}-{APP_ENV}. The suffix is built from the SQS_SUFFIX environment variable defined in the queue connection config; it expands to -{APP_NAME}-{APP_ENV} automatically (for example EpubProcessing-farfalla-production). Two Farfalla deployments share the same SQS account: the regular farfalla-* workers and a separate bulk-processing-* worker pool that runs the same supervisors against higher-throughput infrastructure for ONIX imports.
Queue Catalog
The canonical list of base queue names lives in the QueueName enum and the local Horizon supervisor config. Production queues are these names plus the environment suffix.
| Base queue | Purpose | Typical dispatchers |
|---|---|---|
default | Catch-all for jobs that do not opt into a specific queue. | Generic application work. |
highPriority | Latency-sensitive work that must run ahead of normal processing. | User-facing flows that cannot wait behind a long content pipeline. |
lowPriority | Background cleanup that can wait minutes or hours without user impact. | File deletion after issue removal (DeleteIssueFiles), other janitorial work. |
EpubProcessing | EPUB ingestion and conversion. | Single-issue EPUB upload pipeline. |
PdfExtractInfo | First PDF stage: extract metadata and counts. | Single-issue PDF upload pipeline. |
PdfOptimize | Second PDF stage: optimize the source file. | Single-issue PDF upload pipeline. |
PdfProcessPages | Third PDF stage: render each page. | Single-issue PDF upload pipeline. |
PdfSplitInPages | Fourth PDF stage: split the PDF into per-page files. | Single-issue PDF upload pipeline. |
AudioProcessing | Audiobook ingestion and conversion. | Audio upload pipeline. |
BulkImportIssuesProcessing | Bulk import orchestration for ONIX and Excel-driven imports. | ONIX feeds, Excel uploads, Medusa automations. |
The bulk import pool exposes parallel queues for each PDF/EPUB stage prefixed with BulkImportIssuesProcessing-* (for example BulkImportIssuesProcessing-EpubProcessing-bulk-processing-production); the same job class is dispatched to either pool depending on whether the source is a single upload or a bulk feed.
archiving-coniglio-production appears in the SQS console but is owned by the Coniglio analytics service, not the Farfalla supervisors. Farfalla does not dispatch to it; ignore it when triaging Farfalla pipelines.