Caching Module (Redis)

Complete guide to deploying and managing Redis caching with high performance, persistence, and monitoring capabilities.

🏗️ Overview

The Redis module provides high-performance in-memory data structure store used as a database, cache, and message broker. It offers exceptional performance, data persistence, and clustering capabilities for modern applications.

Architecture Components

Redis Deployment:
├── Redis Master Pod
├── Redis Replica Pods (Optional)
├── Persistent Storage (Optional)
├── Memory-Optimized Configuration
├── Health Monitoring
└── Performance Metrics

🚀 Features

High Performance Caching

In-Memory Storage: Ultra-fast data access with sub-millisecond response times
Data Structures: Support for strings, hashes, lists, sets, sorted sets, and streams
Atomic Operations: Thread-safe operations for concurrent access
Pipelining: Batch multiple commands for improved throughput

Data Persistence

RDB Snapshots: Point-in-time snapshots for backup and recovery
AOF Logging: Append-only file for durability and crash recovery
Hybrid Persistence: Combine RDB and AOF for optimal performance and safety
Configurable Persistence: Flexible persistence strategies

Advanced Features

Pub/Sub Messaging: Real-time messaging capabilities
Lua Scripting: Server-side script execution
Transactions: Multi-key atomic operations
Streams: Log-like data structure for time-series data

📦 Deployment Configuration

Helmfile Configuration

The Redis deployment uses Helmfile for environment management:

repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: redis
    namespace: default
    createNamespace: true
    chart: bitnami/redis
    values:
      - values.yaml

Core Configuration Values

architecture: standalone

cluster:
  enabled: false
  slaveCount: 0

usePassword: false

auth:
  enabled: false
  password: ""

replica:
  replicaCount: 0

persistence:
  enabled: false
  size: 8Gi

master:
  disableCommands: []
  persistence:
    enabled: false
  resources:
    requests:
      cpu: 2000m
      memory: 1Gi
    limits:
      cpu: 4000m
      memory: 8Gi
  configuration: |-
    appendonly no
    save ""
    maxmemory 8000mb
    maxmemory-policy volatile-lru

resources:
  requests:
    memory: 1Gi
    cpu: 2000m
  limits:
    memory: 8Gi
    cpu: 4000m

Advanced Configuration Options

# High-availability configuration
architecture: replication
replica:
  replicaCount: 2

# Performance tuning
master:
  configuration: |-
    maxmemory 6gb
    maxmemory-policy allkeys-lru
    appendonly yes
    appendfsync everysec
    save 900 1
    save 300 10
    save 60 10000

# Security configuration
auth:
  enabled: true
  password: "secure-password"
  sentinel: true

# Monitoring
metrics:
  enabled: true
  serviceMonitor:
    enabled: true

🔧 Redis Configuration

Memory Management

# Memory configuration
configuration: |-
  # Memory limits
  maxmemory 6gb
  maxmemory-policy allkeys-lru

  # Persistence settings
  appendonly yes
  appendfsync everysec

  # Snapshot settings
  save 900 1
  save 300 10
  save 60 10000

  # Performance tuning
  tcp-keepalive 300
  timeout 0
  tcp-backlog 511

Eviction Policies

# Available eviction policies
maxmemory-policy:
  - noeviction # Never evict keys
  - allkeys-lru # Evict any key by LRU
  - volatile-lru # Evict expired keys by LRU
  - allkeys-random # Evict any key randomly
  - volatile-random # Evict expired keys randomly
  - volatile-ttl # Evict expired keys by TTL

Persistence Strategies

# RDB (Snapshot) configuration
save:
  - "900 1" # Save if at least 1 key changed in 900 seconds
  - "300 10" # Save if at least 10 keys changed in 300 seconds
  - "60 10000" # Save if at least 10000 keys changed in 60 seconds

# AOF (Append-Only File) configuration
appendonly: yes
appendfsync:
  - always # Sync on every write (safest, slowest)
  - everysec # Sync once per second (balanced)
  - no # Let OS handle syncing (fastest, least safe)

📊 Monitoring & Metrics

Health Checks

# Check Redis service status
kubectl get pods -l app.kubernetes.io/name=redis

# Check service endpoints
kubectl get endpoints -l app.kubernetes.io/name=redis

# Test Redis connectivity
kubectl exec -it redis-master-0 -- redis-cli ping

Performance Monitoring

# Check resource usage
kubectl top pods -l app.kubernetes.io/name=redis

# Monitor Redis memory usage
kubectl exec -it redis-master-0 -- redis-cli info memory

# Check Redis statistics
kubectl exec -it redis-master-0 -- redis-cli info stats

# Monitor slow queries
kubectl exec -it redis-master-0 -- redis-cli slowlog get 10

Key Metrics

# Memory usage
kubectl exec -it redis-master-0 -- redis-cli info memory | grep used_memory_human

# Hit rate
kubectl exec -it redis-master-0 -- redis-cli info stats | grep keyspace_hits

# Connected clients
kubectl exec -it redis-master-0 -- redis-cli info clients | grep connected_clients

# Database size
kubectl exec -it redis-master-0 -- redis-cli dbsize

Log Analysis

# View Redis logs
kubectl logs -l app.kubernetes.io/name=redis

# Follow logs in real-time
kubectl logs -f deployment/redis-master

# Check for errors
kubectl logs -l app.kubernetes.io/name=redis | grep ERROR

🚀 Deployment

Deploy Redis Module

# Navigate to module directory
cd iac/modules/redis

# Deploy using Helmfile
helmfile apply

# Verify deployment
kubectl get pods -l app.kubernetes.io/name=redis
kubectl get services -l app.kubernetes.io/name=redis

Verify Deployment

# Check pod status
kubectl get pods -l app.kubernetes.io/name=redis

# Test Redis connection
kubectl exec -it redis-master-0 -- redis-cli ping

# Check Redis info
kubectl exec -it redis-master-0 -- redis-cli info server

Post-Deployment Setup

# Port forward for local access
kubectl port-forward svc/redis-master 6379:6379

# Test with redis-cli
redis-cli -h localhost -p 6379 ping

# Set and get a test value
redis-cli -h localhost -p 6379 set test "Hello Redis"
redis-cli -h localhost -p 6379 get test

🔧 Maintenance Operations

Backup Operations

# Create RDB backup
kubectl exec -it redis-master-0 -- redis-cli save

# Copy RDB file
kubectl cp redis-master-0:/data/dump.rdb ./redis-backup-$(date +%Y%m%d).rdb

# Restore from backup
kubectl cp ./redis-backup-20240110.rdb redis-master-0:/data/dump.rdb
kubectl exec -it redis-master-0 -- redis-cli flushall
kubectl exec -it redis-master-0 -- redis-cli debug reload

Scaling Operations

# Scale Redis replicas (if using replication)
kubectl scale statefulset redis-replicas --replicas=3

# Update memory limits
kubectl patch deployment redis-master -p '{"spec":{"template":{"spec":{"containers":[{"name":"redis","resources":{"limits":{"memory":"12Gi"}}}]}}}}'

Update Operations

# Update Redis version
helmfile apply

# Monitor update progress
kubectl rollout status deployment/redis-master

# Rollback if needed
kubectl rollout undo deployment/redis-master

🚨 Troubleshooting

Common Issues

1. Connection Problems

# Check service connectivity
kubectl get services -l app.kubernetes.io/name=redis

# Test network connectivity
kubectl exec -it redis-master-0 -- nc -zv redis-master 6379

# Verify DNS resolution
kubectl exec -it redis-master-0 -- nslookup redis-master

2. Memory Issues

# Check memory usage
kubectl exec -it redis-master-0 -- redis-cli info memory

# Check eviction statistics
kubectl exec -it redis-master-0 -- redis-cli info stats | grep evicted

# Monitor memory fragmentation
kubectl exec -it redis-master-0 -- redis-cli info memory | grep mem_fragmentation

3. Performance Issues

# Check slow queries
kubectl exec -it redis-master-0 -- redis-cli slowlog get 10

# Monitor command statistics
kubectl exec -it redis-master-0 -- redis-cli info commandstats

# Check for blocking operations
kubectl exec -it redis-master-0 -- redis-cli client list | grep -i block

4. Persistence Issues

# Check AOF status
kubectl exec -it redis-master-0 -- redis-cli info persistence

# Check RDB status
kubectl exec -it redis-master-0 -- redis-cli info persistence | grep rdb

# Verify persistence files
kubectl exec -it redis-master-0 -- ls -la /data/

Recovery Procedures

Emergency Recovery

# Force delete stuck pods
kubectl delete pod redis-master-0 --grace-period=0 --force

# Restore from backup
kubectl cp ./redis-backup.rdb redis-master-0:/data/dump.rdb
kubectl exec -it redis-master-0 -- redis-cli debug reload

# Verify data integrity
kubectl exec -it redis-master-0 -- redis-cli dbsize

🔒 Security Configuration

Authentication

# Enable authentication
auth:
  enabled: true
  password: "secure-password"
  sentinel: true

# Network security
networkPolicy:
  enabled: true
  allowExternal: false
  ingressRules:
    primaryAccessOnlyFrom:
      enabled: true
      namespaceSelector:
        matchLabels:
          name: production
      podSelector:
        matchLabels:
          app.kubernetes.io/name: backend

SSL/TLS Configuration

# TLS configuration
tls:
  enabled: true
  secretName: redis-tls

# Certificate configuration
certificatesSecret: "redis-certs"

📝 Configuration Examples

High-Performance Configuration

# High-performance values
resources:
  requests:
    memory: 8Gi
    cpu: 4000m
  limits:
    memory: 16Gi
    cpu: 8000m

# Performance tuning
master:
  configuration: |-
    maxmemory 12gb
    maxmemory-policy allkeys-lru
    appendonly yes
    appendfsync everysec
    save 900 1
    save 300 10
    save 60 10000
    tcp-keepalive 300
    timeout 0
    tcp-backlog 511

Memory-Optimized Configuration

# Memory optimization
master:
  configuration: |-
    maxmemory 4gb
    maxmemory-policy volatile-lru
    appendonly no
    save ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64

Persistence-Optimized Configuration

# Persistence optimization
master:
  configuration: |-
    appendonly yes
    appendfsync everysec
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes

🔄 Maintenance Schedule

Daily Tasks

Monitor memory usage
Check persistence status
Review slow queries
Verify service health

Weekly Tasks

Analyze performance metrics
Review memory fragmentation
Update statistics
Check for updates

Monthly Tasks

Capacity planning review
Performance optimization
Security audit
Disaster recovery testing

📋 Operational Checklist

Pre-Deployment

Post-Deployment

Regular Maintenance

Backend Application - Redis integration
Configuration Guide - Cache configuration
Security Guide - Cache security
Performance Guide - Cache optimization

The Redis caching module provides high-performance in-memory data storage with persistence, monitoring, and comprehensive management capabilities.