Background Jobs with Task Queues

For work that must be durable, retried or scheduled — emails, reports, webhooks, image processing — enqueue jobs to a broker (Redis, RabbitMQ, SQS) processed by separate workers: Celery (mature, feature-rich), ARQ (asyncio + Redis), Dramatiq, or Postgres-backed queues. Enqueue after the database commit (or via an outbox) so workers see the data, make tasks idempotent, and monitor queue depth.

# ARQ example (asyncio-native)
from arq import create_pool
from arq.connections import RedisSettings


async def generate_report(ctx, report_id: int) -> None:
    async with SessionLocal() as db:
        report = await db.get(Report, report_id)
        if report.status == "done":          # idempotent
            return
        report.file_key = await build_and_upload(report)
        report.status = "done"
        await db.commit()


class WorkerSettings:
    functions = [generate_report]
    redis_settings = RedisSettings(host="redis")
    max_tries = 5


@app.post("/reports", status_code=202)
async def request_report(body: ReportIn, db: DB, request: Request):
    report = await reports.create(db, body)
    await db.commit()
    await request.app.state.arq.enqueue_job("generate_report", report.id)
    return {"id": report.id, "status": "queued"}

Common pitfalls

• Enqueuing before commit, so the worker can't find the row.
• Non-idempotent tasks that double-charge or double-send on retry.
• No visibility into failed jobs.

Key points

• Queues for durable work
• Enqueue after commit; idempotent tasks
• Return 202 with a status resource

Further reading: ARQ documentation