When someone says what is Redis, we also presume it to be a cache. But when you dig deep into their site, you realize it is an ecosystem of storage solutions. In addition to simple K-V store, it supports a wide range of different data types. It has persistance which is highly configurable. Also there are other data paradigms supported such as Pub/Sub, Streams etc.

Redis as a cache

Pre-requisite: Caching strategies

Before we go further, here is a primer on different caching strategies:

1. Write-behind cache

2. Write-through cache

3. Cache-aside

4. Read-through cache

Cache-aside

Most popular caching strategy where cache is a fast storage layer and application server takes read/write to storage based on cache response.

1. If Cache Hit: Read from cache

• If Cache Miss

  2. Read from database

• 3. Write to cache

Write-through cache

Here the cache is in middle of application and database. And all write happen through cache to database at exact time

Write-behind cache

Similar to Write-through cache, cache is in between application and database. But the key difference is that cache batches up writes coming from application and flushes to database at a later time.

Read-through cache

Cache comes in between application and database, but the reads are the operation that occurs here which are driven from database.

1. Read from database and store in cache

2. Serve the data to application from the cache This is useful when the database has a lot of data that needs to be read from the application which do not change over time.

Alternates to Redis

• Memcached

• Hazelcast

• Infinispan

Redis data structures

Core

• Strings

> SET RedisStringType "This is Redis String" "OK" 
> GET RedisStringType "This is Redis String"

• Lists

> LPUSH RedisListType 322 (integer) 1 
> LPUSH RedisListType 341 (integer) 2
> LLEN RedisListType (integer) 2
> LRANGE RedisListType 0 2 
 1) "341" 
 2) "322" 
> LPOP RedisListType "341" 
> LRANGE RedisListType 0 2 
 1) "322"

• Sets

> SADD RedisSetType 
  545 (integer) 1 
> SADD RedisSetType 545 (integer) 0 
> SADD RedisSetType 544 (integer) 1 
> SCARD RedisSetType (integer) 2
> SISMEMBER RedisSetType 544 (integer) 0 
> SISMEMBER RedisSetType 544 (integer) 1

• Sorted Sets

> ZADD RedisSortSet 13 "Rob" 54 "Rohit" (integer) 2 
> ZRANGE RedisSortSet 0 -1 
 1) "Rob" 
 2) "Rohit" 
> ZADD RedisSortSet10 "Maya"
> ZRANGE RedisSortSet 0 -1 
1) "Maya" 
2) "Rob" 
3) "Rohit"

• Hashes

> HSET RedisHashType HashKey "Welcome to Redis" (integer) 1 
> HGET RedisHashType HashKey "Welcome to Redis"

• Streams

> XADD RedisStreamType * user James 54 Australia MALE "12Feb2024" "1708786153271-0" 
> XADD RedisStreamType * user Donal 34 Germany MALE "13Feb2024" "1708786187391-0" 
> XRANGE RedisStreamType 1708786153271-0 + COUNT 2 
1) 1) "1708786153271-0" 
   1) 1) "user" 
      2) "James" 
      3) "54" 
      4) "Australia" 
      5) "MALE" 
      6) "12Feb2024" 
2) 1) "1708786187391-0" 
   1) 1) "user" 
      2) "Donal" 
      3) "34" 
      4) "Germany" 
      5) "MALE" 
      6) "13Feb2024"

• Bitmaps Set as Bits. Only 0 or 1 allowed

> SETBIT RedisBitMap 323 1 (integer) 
> GETBIT RedisButMap 323 (integer) 
> GETBIT RedisBitMap 323 (integer) 1

• Bitfields

> BITFIELD RedisBitFields:stats SET u32 #0 1000
1) (integer) 0
> BITFIELD RedisBitFields:stats INCRBY u32 #0 -50 INCRBY u32 #1 1
1) (integer) 950
2) (integer) 1
> BITFIELD RedisBitFields:stats INCRBY u32 #0 500 INCRBY u32 #1 1
1) (integer) 1450
2) (integer) 2

• HyperLogLog

> PFADD RedisCardinalityData Marvel DC Sony Universal
(integer) 1
> PFCOUNT bikes
(integer) 4

• Geospatial Indexes

Redis persistance

We are all aware of Redis as a in-memory store which is a volatile memory(gets erased when server is shutdown); but there are persistence mechanisms by which you can store data on disk. This is similar to Write-ahead logs found in traditional databases where the series of transactions are stored as a log file in various formats (UNDO, REDO etc) In Redis, there are 3 ways of persisting data to durable storage

1. Redis Database (RDB)

2. Append-only file (AOF)

3. RDB+AOF

Pre-requisite: Copy-on-write on Linux

RDB

This takes a snapshots of the Redis instance at sampling intervals.

AOF

AOF persists every write operation that has occurred in Redis. As it has every operation detail, this file offers the advantage to refreshing the Redis to the current state if the server shuts down. But this comes at the expense of large AOF size as there ain't no free lunch

Comparison matrix of RDB and AOF

Combining RDB and AOF

Redis allows to combine AOF and RDB to have bost of both worlds. RDB brings compact file and AOF brings durability, hence you don't loose speed or data.

Redis as a Database: ACID and other things

Redis is an in-memory but can act as a database by providing ACID compliance

• Partial ACID: As it is single-threaded it guarantees consistency and isolation

• Full ACID if AOF appendfsync always is used. As AOF is written to disk; Atomicity and Durability are guaranteed as AOF can be used to power up Redis to the stable state before the server crash occurred

Atomicity

All transaction should happen or not atomically

Redis provides ATOMICITY if AOF appendfsync always is used.

Consistency

How the data is consistent. What data has been persisted should be read consistently

Redis provides this out-of-the-box as it is single-threaded. In Multi-node Redis setup; Redis provides consistency via Master-Slave architecture. In Active-Active (aka Master-Master) setup, Redis is able to achieve by using CRDT (Link).

Isolation

How isolated is the row when concurrently upsert/delete commands occur

Redis provides this out-of-the-box as it is single-threaded (except for I/O in Redis 6)

Durability

How durable is database in event of failure like machine crashes etc without loosing write

Redis provides ATOMICITY if AOF appendfsync always is used. Database provides Durability via Write-ahead-logs whilst Redis does this by AOF or RDB as discussed before.

Availability

The database transaction (read, write) is returned by database and is available to consumer

Concurrency control

How concurrent transactions are controlled. For example, a user David data is being updated by 2 transactions. We need optimistic or pessimistic concurrency control

Redis has WATCH command

Redis as a Streaming

Redis Streams

Inspired by Event Sourcing systems with Append-only logs, Redis provides a data structure Redis Streams to facilitate event-driven architecture.

Similar to Kafka, Flink; Redis also has concept of Message Broker with Consumer groups. Consuming groups and consumer

An example of creating a Redis Stream, Consumer Group and Read for a Consumer

// Create Redis Stream `Redis_Stream_Example` with data
> XADD Redis_Stream_Example * name "AymanPatel" age 54 gender "MALE" sessionId 433
"1708792289107-0"

// Create Consumer Group `redis_consumer-group`
> XGROUP CREATE Redis_Stream_Example redis_consumer_group $
"OK"


// Add data `Selena`
> XADD Redis_Stream_Example * name "Selena" age 14 gender "FEMALE" sessionId 431 
"1708792778677-0"

// Create a reader which reads new data `Selena`
> XREADGROUP GROUP redis_consumer_group consumer1 STREAMS Redis_Stream_Example >
1) 1) "Redis_Stream_Example"
   2) 1) 1) "1708792778677-0"
         2) 1) "name"
            2) "AymanPatel"
            3) "age"
            4) "54"
            5) "gender"
            6) "MALE"
            7) "sessionId"
            8) "433"

1. Add data to Redis Stream

   

2. Create consumer group

1. Add data to Redis Stream and Read from Consumer

   

2. Read the Consumer has received. They can ACK or CLAIM too!

   

Redis Pub/Sub

Similar to radio broadcast, a Broadcaster can PUBLISH a message to Redis, whilst subscribers can SUBSCRIBE to messages

Can check via Redis Insights

Redis Search

RedisSearch is a Secondary Index & Full-text Search

Full-text Search

Alternative to Apache Solr (OSS), Algolia (SaaS) to provide text-search capability over data stored in Redis

Redis's documentation site is powered by RedisStack which includes Redis's full text search.

Secondary Index

Database has ability to index non-primary keys for faster retrieval using the Secondary Index. Oracle documentation on Secondary index

Redis provides a plethora of commands for working via FT commands.

• Link to RedisSearch methods

Redis Search also has other building blocks for creating full-text systems such as

• Aggregations such as GROUPBY, FILTER, SORT etc

• Stop words: Discarding stop-words (a, an, the etc) to improve search performance

• Tokenization: Destructure sentence into words for individual storage

• Stemming: Coagulating same-meaning words as one. Example: Study can be studying, studied etc and can be store as study only

• and much more

Document Database with RedisJSON

Mongo had become popular to store data as JSON BLOBs in the 2010s. This was useful in many e-commerce applications to store database as a JSON store which can be served directly via an API without need to do database JOINs or serializing/deserializing while serving. Note that this use-case became less popular as ACID is not necessarily guaranteed.

Redis as a Vector Database

What is a Vector Database by Pinecone

In the age of ChatGPT and LLMs, a new type of database has emerged called Vector databases. These databases store any type of data in vector(numeric) format which can lead to semantic meaning of correlated data nearer to each other which can be used by LLM to gain Long-term memory and increasing its context window. Redis can be used as a vector embedding to store vectors via Redis server and RedisJSON. Read more about Redis Vector setup here

Note that this is a hacky approach. For better Vector databases look into Chroma, Pinecone etc

Extending Redis

1. Similar to User-defined functions in traditional database, Redis 7 has introduced to programmatically run Lua scripts inside Redis. Read more here

2. Redis Modules API

   • Redis TimeSeries(Community)

   • Redis Bloom(Community)

   • redis-cell (Community and Rate limiting library)

Conclusion

When I started digging into Redis, I realised how diverse is it in its toolset. It may still be a cache; but making it work with other design paradigms just shows how well the architecture and abstraction of Redis is.

Also, one more thing. Cache is just a catch-all term that requires nuance; so keep that in mind :)

Resources

1. Can Redis be used as a Primary database? - Hussein Nasser Youtube Link

2. I've been using Redis wrong this whole time...- Dreams of Code Youtube Link

3. Coding with Raphael De Lio - Youtube playlist

4. Redis Docs

5. Redis Insight