Message Queue Module (RabbitMQ)

Complete guide to deploying and managing RabbitMQ message broker with high availability, clustering, and monitoring capabilities.

🏗️ Overview

The RabbitMQ module provides a robust and scalable message broker that implements the Advanced Message Queuing Protocol (AMQP). It enables reliable message delivery, supports multiple messaging patterns, and provides comprehensive monitoring and management tools.

Architecture Components

RabbitMQ Deployment:
├── RabbitMQ Server Pods (Cluster)
├── Persistent Storage
├── Management UI
├── AMQP Protocol Support
├── Health Monitoring
└── Performance Metrics

🚀 Features

Message Broker Capabilities

AMQP Protocol: Full support for Advanced Message Queuing Protocol
Multiple Exchange Types: Direct, topic, fanout, and headers exchanges
Queue Management: Durable, auto-delete, and priority queues
Message Routing: Flexible routing with bindings and routing keys

High Availability

Clustering: Multi-node cluster for fault tolerance
Mirrored Queues: Queue replication across cluster nodes
Automatic Failover: Seamless failover in case of node failure
Load Balancing: Distributed message processing

Advanced Features

Dead Letter Queues: Handle failed message processing
Message TTL: Time-to-live for messages
Priority Queues: Message prioritization
Delayed Messages: Scheduled message delivery
Publisher Confirms: Reliable message publishing

📦 Deployment Configuration

Helmfile Configuration

The RabbitMQ deployment uses Helmfile for environment management:

repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: rabbitmq
    namespace: queue
    createNamespace: true
    chart: bitnami/rabbitmq
    values:
      - values.yaml

Core Configuration Values

auth:
  password: CHANGEME

# Basic configuration
replicaCount: 1

persistence:
  enabled: true
  size: 8Gi
  storageClass: "local-path"

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

service:
  type: ClusterIP

ingress:
  enabled: false

Advanced Configuration Options

# High-availability configuration
replicaCount: 3

# Clustering configuration
clustering:
  enabled: true
  rebalance: true

# Performance tuning
resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "2Gi"
    cpu: "1000m"

# Security configuration
auth:
  enabled: true
  password: "secure-password"
  erlangCookie: "SWQOKODSQALRPCLNMEQG"

# Monitoring
metrics:
  enabled: true
  serviceMonitor:
    enabled: true

# Management UI
management:
  enabled: true
  ingress:
    enabled: true
    hostname: rabbitmq.theratap.de
    tls: true

🔧 RabbitMQ Configuration

Cluster Configuration

# Cluster settings
clustering:
  enabled: true
  rebalance: true
  partitionHandlingStrategy: "autoheal"

# Erlang cookie for cluster communication
erlangCookie: "SWQOKODSQALRPCLNMEQG"

# Node naming
nodeName: "rabbit@$(HOSTNAME).$(NAMESPACE).pod.cluster.local"

Queue Configuration

# Default queue settings
defaultQueueSettings:
  durable: true
  autoDelete: false
  arguments:
    x-message-ttl: 86400000 # 24 hours
    x-max-length: 10000
    x-overflow: drop-head

# Dead letter configuration
deadLetterExchange: "dlx"
deadLetterRoutingKey: "dlq"

Performance Tuning

# Memory and disk thresholds
vmMemoryHighWatermark:
  relative: 0.6 # 60% of available memory

diskFreeLimit:
  relative: 2.0 # 2GB free disk space

# Connection limits
maxConnections: 1000
maxChannels: 2000

# Message size limits
maxMessageSize: 134217728 # 128MB

📊 Monitoring & Metrics

Health Checks

# Check RabbitMQ service status
kubectl get pods -n queue -l app.kubernetes.io/name=rabbitmq

# Check service endpoints
kubectl get endpoints -n queue -l app.kubernetes.io/name=rabbitmq

# Test RabbitMQ connectivity
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl status

Performance Monitoring

# Check resource usage
kubectl top pods -n queue -l app.kubernetes.io/name=rabbitmq

# Monitor queue statistics
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name messages consumers

# Check connection status
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_connections

# Monitor channel usage
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_channels

Key Metrics

# Queue depth
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name messages

# Message rates
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name message_stats.publish_details.rate

# Consumer count
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name consumers

# Memory usage
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl status | grep memory

Management UI

# Port forward management UI
kubectl port-forward -n queue svc/rabbitmq 15672:15672

# Access management UI
open http://localhost:15672

# Default credentials
# Username: admin
# Password: (from auth.password in values.yaml)

Log Analysis

# View RabbitMQ logs
kubectl logs -n queue -l app.kubernetes.io/name=rabbitmq

# Follow logs in real-time
kubectl logs -f -n queue deployment/rabbitmq

# Check for errors
kubectl logs -n queue -l app.kubernetes.io/name=rabbitmq | grep ERROR

🚀 Deployment

Deploy RabbitMQ Module

# Navigate to module directory
cd iac/modules/rabbitmq

# Deploy using Helmfile
helmfile apply

# Verify deployment
kubectl get pods -n queue
kubectl get services -n queue

Verify Deployment

# Check pod status
kubectl get pods -n queue -l app.kubernetes.io/name=rabbitmq

# Test RabbitMQ connection
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl status

# Check cluster status
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl cluster_status

Post-Deployment Setup

# Port forward for local access
kubectl port-forward -n queue svc/rabbitmq 5672:5672

# Test with amqp-tools
amqp-declare-queue --url=amqp://admin:password@localhost:5672 -q test-queue

# Publish a test message
amqp-publish --url=amqp://admin:password@localhost:5672 -r test-queue -p "Hello RabbitMQ"

# Consume messages
amqp-consume --url=amqp://admin:password@localhost:5672 -q test-queue

🔧 Maintenance Operations

Queue Management

# Create a queue
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl declare_queue name=my-queue durable=true

# List all queues
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues

# Purge a queue
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl purge_queue my-queue

# Delete a queue
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl delete_queue my-queue

Exchange Management

# Create an exchange
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl declare_exchange name=my-exchange type=direct durable=true

# List all exchanges
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_exchanges

# Delete an exchange
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl delete_exchange my-exchange

User Management

# Create a user
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl add_user appuser apppassword

# Set user permissions
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl set_permissions -p / appuser ".*" ".*" ".*"

# List users
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_users

# Delete a user
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl delete_user appuser

Scaling Operations

# Scale RabbitMQ replicas
kubectl scale statefulset rabbitmq -n queue --replicas=3

# Verify cluster formation
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl cluster_status

# Rebalance cluster
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl rebalance

Update Operations

# Update RabbitMQ version
helmfile apply

# Monitor update progress
kubectl rollout status statefulset/rabbitmq -n queue

# Rollback if needed
kubectl rollout undo statefulset/rabbitmq -n queue

🚨 Troubleshooting

Common Issues

1. Connection Problems

# Check service connectivity
kubectl get services -n queue

# Test network connectivity
kubectl exec -it rabbitmq-0 -n queue -- nc -zv rabbitmq 5672

# Verify DNS resolution
kubectl exec -it rabbitmq-0 -n queue -- nslookup rabbitmq

2. Cluster Issues

# Check cluster status
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl cluster_status

# Check node health
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl status

# Verify erlang cookie
kubectl exec -it rabbitmq-0 -n queue -- cat /var/lib/rabbitmq/.erlang.cookie

3. Memory Issues

# Check memory usage
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl status | grep memory

# Check memory alarms
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_alarms

# Monitor disk usage
kubectl exec -it rabbitmq-0 -n queue -- df -h

4. Queue Issues

# Check queue status
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name messages consumers

# Check for unacknowledged messages
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name messages_unacknowledged

# Monitor message rates
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl list_queues name message_stats

Recovery Procedures

Emergency Recovery

# Force delete stuck pods
kubectl delete pod rabbitmq-0 -n queue --grace-period=0 --force

# Reset cluster state
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl stop_app
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl reset
kubectl exec -it rabbitmq-0 -n queue -- rabbitmqctl start_app

# Rejoin cluster
kubectl exec -it rabbitmq-1 -n queue -- rabbitmqctl join_cluster [email protected]

🔒 Security Configuration

Authentication

# Enable authentication
auth:
  enabled: true
  password: "secure-password"
  erlangCookie: "SWQOKODSQALRPCLNMEQG"

# Network security
networkPolicy:
  enabled: true
  allowExternal: false
  ingressRules:
    primaryAccessOnlyFrom:
      enabled: true
      namespaceSelector:
        matchLabels:
          name: production
      podSelector:
        matchLabels:
          app.kubernetes.io/name: backend

SSL/TLS Configuration

# TLS configuration
tls:
  enabled: true
  secretName: rabbitmq-tls

# Certificate configuration
certificatesSecret: "rabbitmq-certs"

Access Control

# User permissions
permissions:
  - user: appuser
    vhost: /
    configure: ".*"
    write: ".*"
    read: ".*"

📝 Configuration Examples

High-Performance Configuration

# High-performance values
resources:
  requests:
    memory: 2Gi
    cpu: 1000m
  limits:
    memory: 4Gi
    cpu: 2000m

# Performance tuning
configuration: |-
  vm_memory_high_watermark.relative = 0.6
  disk_free_limit.relative = 2.0
  max_connections = 1000
  max_channels = 2000
  max_message_size = 134217728

High-Availability Configuration

# High-availability setup
replicaCount: 3

clustering:
  enabled: true
  rebalance: true
  partitionHandlingStrategy: "autoheal"

# Mirrored queues
defaultQueueSettings:
  arguments:
    x-ha-policy: "all"

Monitoring-Optimized Configuration

# Enhanced monitoring
metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s

management:
  enabled: true
  loadDefinitions: true
  definitions:
    queues:
      - name: "monitoring-queue"
        durable: true
        arguments:
          x-message-ttl: 3600000

🔄 Maintenance Schedule

Daily Tasks

Monitor queue depths
Check connection counts
Review error logs
Verify cluster health

Weekly Tasks

Analyze performance metrics
Review queue statistics
Update user permissions
Check for updates

Monthly Tasks

Capacity planning review
Performance optimization
Security audit
Disaster recovery testing

📋 Operational Checklist

Pre-Deployment

Post-Deployment

Regular Maintenance

Backend Application - RabbitMQ integration
Configuration Guide - Queue configuration
Security Guide - Message queue security
Performance Guide - Queue optimization

The RabbitMQ message queue module provides reliable message brokering with clustering, monitoring, and comprehensive management capabilities.