Redis
Run Runwick on one durable Redis primary with append-only fsync on every write, and know the limits that come with it.
On this page
The Redis store keeps Runwick's data on one Redis primary that writes every change to its append-only file and syncs it to disk before it replies. Producers and workers can run on many hosts. Job behaviour is the same as on PostgreSQL: the same states, retries, receipts, uniqueness and rate rules.
Redis is only safe for jobs with the right server settings, so Runwick checks them before it starts and keeps checking while it runs.
Requirements
Server configuration
Start Redis with these settings. Runwick refuses to run with anything else:
Size maxmemory with room to spare. With noeviction, a full server refuses writes, and dispatches fail until you free space.
Keep CONFIG GET readable
Runwick reads the settings with CONFIG GET and MODULE LIST. If an ACL blocks either, the value counts as unverifiable and the store refuses to run. Restrict CONFIG SET with ACLs if you need to, but leave CONFIG GET and MODULE LIST allowed for Runwick's user. Do not remove CONFIG with rename-command. Managed Redis services that hide CONFIG are not supported.
Install and migrate
Write the store's identity once before the first start:
Running it again does nothing if the store is already installed. Without it, the app refuses to start because the store is not installed.
Register the store
The configuration string is a StackExchange.Redis string. Runwick uses only the endpoint, credentials, TLS settings, defaultDatabase and connectTimeout from it, and never logs it. Several endpoints, or a Sentinel service name, are refused at startup with topology_unsupported.
What Runwick checks
At startup, on every reconnect and every ProfileRecheckInterval, Runwick checks on a separate connection:
- the five settings above, through
CONFIG GET; - through
INFO:role:master,redis_mode:standalone,cluster_enabled:0, AOF enabled, the last AOF write succeeded, and the version is at least 8.2; - through
MODULE LIST: the loaded modules, which are reported but allowed.
Replicas attached to the primary are allowed and reported. But a replica promoted to primary is a different history, and Runwick does not follow a failover.
History checks
Each process remembers the store's identity and the highest revision it has seen for the application. If Redis comes back empty (for example after FLUSHALL), with another store's identity, or with a lower revision, Runwick stops admitting work and treats every lease it held as lost. It does not resume until you restart the process against an intact store. This check catches a history that went backwards. It cannot prove that no write was lost.
One application, one hash slot
Every key of an application starts with rw:{<application>}:, so all of them live in one hash slot. That is what lets one Lua script change a job's records together. Several applications can share one Redis database.
The cost: one application's throughput is bounded by one Redis server thread, and one very large command (for example a big batch) delays every other command of that application while it runs. If one application needs more, split the work into separate applications.
No transactions with your data
Redis cannot join your business database's transaction. There is no StageAsync and no transactional completion on Redis. To create a job only when your business data commits, write it to an outbox in your business database; the relay delivers it to the Redis store.
Polling
There are no wake-up notifications between processes. Workers find new jobs by polling. Within one process, a commit wakes that process's own workers.
What Redis does not promise
- No high availability. Failover, Sentinel promotion and Cluster are not supported. A promoted replica may have lost acknowledged jobs.
- Durability is the primary's disk and
appendfsync always. Restoring an AOF or RDB file is a break in history. - Running Redis without
appendfsync alwaysis not a supported configuration, even for benchmarks. - Results on Redis say nothing about speed compared with PostgreSQL.