Enable Cross-Worker Queue Testing Locally

[StefanStuehrmann]

Proposal: SQLite-based Queue Implementation for Local Development

Background

Currently, wrangler's local queue implementation uses an in-memory array for storing queue messages, which prevents queue usage across different workers during local development. This limitation makes it difficult to test distributed worker architectures locally.

Proposal

Extend the current queue implementation to optionally use SQLite for message storage (which is already used for D1).
This would enable:

• Shared queue access across different workers
• Persistent queue state between restarts
• Better debugging capabilities
• More production-like behavior for local development

Implementation Details

Configuration

# wrangler.toml
[queues]
storagetType = "sqlite"  # Optional, defaults to "memory" for backwards compatibility
persistTo = ".wrangler/queues.db" # needs to be introduced for queues but would be consistent to e.g. D1, R2
retentionPeriod = 86400  # Optional, defaults to 24 hours

You could think about if a queue state should always reside with the consumer worker as queues can only have one consumer but multiple producers.

Key Features

1. Backward Compatible: Maintains existing memory-based implementation as default
2. Concurrent Access: Uses SQLite transactions for safe concurrent queue operations
3. Message Lifecycle: Proper handling of message states, retries, and dead-letter queues
4. Performance: Indexed queries and batch operations for efficient message processing
5. Debugging: Messages persist between runs and can be inspected using SQLite tools

Benefits

• Test distributed worker architectures locally, which is the main purpose of introducing queues
• Debug queue-related issues more effectively
• More accurate representation of production behavior
• Simplified testing of retry and error handling logic

Implementation Impact

This change would be isolated to the local development environment and wouldn't affect production queues. The implementation would live entirely within the miniflare package and tests and wouldn't require changes to the core Workers runtime.

Looking forward to hearing the queue teams thoughts on this approach.
The effort estimation for the implementation seems to be relatively small (few days).
So in case it does not make it to the queues team backlog I would be willing to contribute once we agree on details and it will make it into the main branch.

[longas]

Leaving this comment here to highlight this as a major annoyance in an otherwise fantastic DX with Cloudflare.

I'm surprised this has so few upvotes, how are others getting around this? The only solutions I can think of are deploying to staging or, as suggested in the docs, duplicating all the logic from my consumer's queue() handler and putting it in the same Worker as the producer, neither of which are particularly ideal.

@StefanStuehrmann in this thread you mention something about wiring handlers of different workers, could you give an example of what you mean here? In any case, I hope the Cloudflare people will support you in implementing your proposal.

[emily-shen]

We've got an experimental feature to run multiple workers in one instance by providing multiple configs, it may work with queues?
Something like this: npx wrangler dev -c ./wrangler.toml -c ../worker2/wrangler.toml

Thanks for the feedback though, passing it on to the queues team
cc @maheshwarip

[StefanStuehrmann]

I started implementing a Durable Object based persistency for local queues in Miniflare.
In general the feature itself was fairly easy to implement as it follows a similar approach as other services / plugins.

Bigger problem are the edge cases and some implementation details:

• right now there is no real consumer implementation but the consumer is always triggered by a producer
• therefore spawning consumers and polling is needed if you want to run producer / consumers across multiple instances -> this implementation was still manageable)
• where it got hairy was in nitty gritty details like how to configure the behavior in miniflare as "miniflare-only" config can't be set via wrangler config (and should probably not)
• ideally if producer and consumer are located in the same instance you keep it simple and don't use a more heavyweight polling approach but if they are distributed you poll (requires additional configuration - just for local)
• some of the queue logic right now is outside of the queue plugin and in miniflare core itself (e.g. to validate single consumer, etc.), ideally with changes in the consumer handling you would move this into the plugin and perform the validation during runtime -> bigger change in miniflare where I'm not even sure I would get this change merged
• lot's of different scenarios (consumer and producer in same instance single worker, consumer and producer in same instance multiple workers -> multiple configs, consumer and producer in different instances, multiple consumer per worker for different queues ...)

After spending quite some time on an implementation that was working for me but still with quite some shortcomings I decided to go for the single instance approach.

Below you can see how I start my workers (using NX and a startup script in my api gateway worker to start all).
wrangler dev -c apps/worker_1/wrangler.toml -c apps/worker_2/wrangler.toml -c apps/worker_3/wrangler.toml --persist-to .wrangler/state' --port 8780

I had to persist services to / level .wrangler/state so that other dependencies like D1 were properly working with multiple config support but it is working ok.

Biggest disadvantage is that I need to run those services now with a single command instead of being able to have my own session for each service to follow (which is minor).

[StefanStuehrmann]

@longas what I did initially is that I basically just imported the queue function into the producer worker and then "wired" it in via wrangler config.

e.g.
producer worker wrangler.toml

local

queues.consumers = [
{ queue = "queue1-dev"},
]
queues.producers = [
{ queue = "queue1-dev, binding = "QUEUE1" },
]

prod

queues.producers = [
{ queue = "queue1-production, binding = "QUEUE1" },
]

consumer worker wrangler.toml

local (not really needed) just for some test scenarios

queues.consumers = [
{ queue = "queue1-dev",
]

prod

queues.consumers = [
{ queue = "queue1-production"},
]

But this is super dirty and basically requiring you in your producer worker to have something like this, where the queue part is only executed locally because of the queue wirering.

import honoApp from './hono-app';
import narabiApp from '../apps/consumer-app/narabi-app';

export default {
fetch: honoApp.fetch,
queue: narabiApp.queue,
};

The multiple config otpion that was suggested before is what I'm also using since some time and which is working much better.

[longas]

Thanks to both of you for your help! Using multiple configs with Wrangler worked perfectly in my monorepo with two workers, one as
producer and the other as consumer. I hadn't even tried it before because the docs make it seem like only one cfg can be used with wrangler. Also good to know that --persist-to will be necessary when I start using D1.