Queue State and Completion Semantics

QueueFS determines completion from one backend status snapshot: a queue is empty only when both pending and processing are zero. An empty pending queue alone does not mean processing has finished.

An empty queue means no messages are currently unacknowledged. It does not prove every operation succeeded or prevent new work from arriving. To verify an import or session commit, inspect its task, including the terminal status and error.

Problem

The previous implementation inferred completion from two independent sources:

• QueueFS /size, which reports messages still pending for dequeue.
• Python NamedQueue._in_progress, which reports work observed by the current process.

These values cannot be read or updated atomically. After QueueFS moves a message from pending to processing, another thread, event loop, or process can observe size == 0 while the _in_progress value it reads is zero. It may therefore report the queue as complete before the message is acknowledged.

Moving the local counter update before or after task creation only narrows some timing windows. It cannot make the backend state transition and local counter update one atomic operation, so it does not strictly solve the problem across threads, event loops, or processes.

ACK Lifecycle

SQLite and cache-backed queues implement an acknowledgement lifecycle:

enqueue -> pending -> dequeue -> processing -> ack -> removed

The memory backend also maintains a processing collection. Dequeue moves messages into it; ACK removes them. It uses the same completion condition as the other backends, but does not preserve queued work across process restarts.

State Model

Queue length is not one scalar. QueueFS maintains these current-state gauges:

Field	Meaning
pending	Messages available for dequeue
processing	Messages dequeued but not yet acknowledged
unacked	pending + processing

A queue is complete only when:

pending == 0 && processing == 0

Equivalently, unacked == 0.

An unacknowledged message remains incomplete when its handler fails, its ACK fails, or its worker exits. It becomes pending again through recovery or leaves the queue through a successful ACK.

Ownership

QueueFS is the owner of queue lifecycle state. Each backend must update pending and processing together with enqueue, dequeue, ACK, clear, and recovery operations.

Python must not combine backend pending with a process-local counter to decide completion. A local worker counter may still be exposed as runtime telemetry, but it is not queue length and is not authoritative.

The following cumulative counters are also not queue length:

• processed
• requeue_count
• error_count

They describe processing outcomes rather than current queue occupancy. They remain in the processing or metrics layer. Moving them into QueueFS would require an explicit backend outcome protocol because the backend cannot infer a handler result from dequeue or ACK alone.

Backend Contract

QueueFS /status returns counts read atomically:

{
  "pending": 3,
  "processing": 2
}

unacked is derived as pending + processing. The existing /size operation can continue to mean pending for compatibility and worker scheduling. QueueFS control names are reserved path components: a queue name cannot end in enqueue, dequeue, peek, size, status, messages, clear, or ack.

Backend requirements:

• SQLite: read both counts from one database snapshot.
• Cache: return LLEN(pending) and ZCARD(processing) from one Lua script.
• Memory: maintain a processing collection; dequeue moves a message into it and ACK removes the message from it.

NamedQueue.get_status() consumes this backend snapshot. wait_complete() and is_all_complete() use only the backend-owned pending and processing values.