Scenario: RabbitMQ receives 1 million messages rapidly

Without smart memory management:
  Time 0s:    1 million messages → All in RAM
  Time 1s:    RAM fills up completely
  Time 2s:    No more RAM available
  Time 3s:    RabbitMQ crashes (out of memory)
  
With smart memory management:
  Time 0s:    1 million messages arrive
  Time 1s:    First 100k messages in RAM, rest on disk
  Time 2s:    Consumers process RAM messages
  Time 3s:    As RAM frees, RabbitMQ pages messages from disk
  Time 4s:    System keeps running smoothly

Message Data = {
  Metadata:      routing_key, timestamp, correlation_id, etc. (~100 bytes)
  Content:       actual message body (1KB - 1GB each)
  State:         is_acked, consumer_info, priority, etc. (~50 bytes)
}

Queue in RAM = [Message1, Message2, Message3, ... MessageN]
               ↑
            Consumers pull from here (fast!)

Queue (spilled to disk) = [...PagedMessages...]
                          ↑
                     When RAM fills up, messages move here

Message arrives → Queue receives it
                    ↓
              Check: Is RAM usage < threshold?
                    ↓
        ┌───────────┴─────────────┐
        ↓                         ↓
      YES                        NO
        ↓                         ↓
   ┌─────────────┐      ┌──────────────────┐
   │ Store in    │      │ Write to disk    │
   │ RAM Queue   │      │ (page store)     │
   └─────────────┘      └──────────────────┘
        ↓                         ↓
   Store metadata         Store metadata
   in memory index        in memory index
        ↓
   Ready for fast access

When memory pressure increases:

1. Page out oldest messages first
   (Messages that arrived earliest)

2. Keep newest messages in RAM
   (Recently arrived = more likely to be consumed soon)

3. Consumers prefer messages from RAM
   (If message is on disk, pull it in, evict older RAM message)

Page messages until memory usage < threshold:
  For each queue (in some order):
    While queue has paged messages:
      Take oldest message
      Write to disk (page store)
      Remove from RAM
      Update index

Message Lifecycle:

1. ARRIVAL (RAM)
   └─ Producer publishes → Message in RAM queue
   
2. WAITING (RAM or DISK)
   └─ If memory pressure: → Page to disk
   
3. DELIVERY (RAM)
   └─ Consumer requests → Load from disk if needed
   
4. ACKNOWLEDGMENT (FREED)
   └─ Consumer acks → Remove from queue entirely

TIME    EVENT                          LOCATION      MEMORY
────────────────────────────────────────────────────────────

T0      Producer publishes message
        ("hello world", 11 bytes)      

T1      RabbitMQ receives             RAM           +11 bytes
        Message stored in queue
        Index entry created
        
T2      No consumer waiting           RAM           (same)
        Message sits in queue

T3      Memory at 50% usage          RAM           (same)
        No paging triggered

T4      Another 100k messages         RAM           +100MB
        arrive rapidly

T5      Memory now at 95% (paging     RAM/DISK      (same RAM,
        threshold exceeded!)                         +disk space)

T6      Paging starts
        Our message: "hello world"    DISK          -11 bytes RAM
        → Written to page store                     +11 bytes DISK

T7      Another consumer request      RAM           (same RAM,
        arrives                                     loads from DISK)

T8      Message loaded from disk      RAM           +11 bytes RAM
        into consumer's buffer                      (evict another
                                                     message)

T9      Consumer processes message    RAM           (processing)

T10     Consumer sends ACK            RAM           (nothing)

T11     Message removed from queue    -             -11 bytes RAM
        Acknowledgment recorded
        Message can be garbage
        collected

TIME    EVENT                          LOCATION      STORAGE
────────────────────────────────────────────────────────────

T0      Producer publishes            
        Durable=true

T1      RabbitMQ receives             RAM+DISK      +11 bytes RAM
        Message → RAM queue                         +11 bytes DISK
        + Write to persistent store                 (wal log)

T2      Confirmation sent to          RAM+DISK      (same)
        producer only after
        disk write completes

T3-5    (same as non-durable)         RAM+DISK      (same)

T6      Paging occurs                 RAM+DISK      -11 bytes RAM
        Message already on disk                     (same disk -
                                                     was already there)

T7      Message paged from             DISK only    (already on disk,
        RAM (no redundant write)                    so faster)

T8      Message loaded from disk      RAM+DISK      +11 bytes RAM
        into consumer buffer

T9-11   (same as non-durable)         RAM+DISK      -11 bytes RAM
                                                    -11 bytes DISK
        Both removed when acked

% In /etc/rabbitmq/rabbitmq.conf

% Method 1: Memory threshold (relative)
vm_memory_high_watermark.relative = 0.6
% Meaning: Page when RAM usage exceeds 60% of available

% Method 2: Memory threshold (absolute)
vm_memory_high_watermark.absolute = 2GB
% Meaning: Page when RAM usage exceeds 2GB

% Method 3: Multiple thresholds for gradual paging
vm_memory_high_watermark.relative = 0.6
% 60% = start paging
% 50% = moderate paging
% 40% = aggressive paging (if memory keeps rising)

100% ┌────────────────────────────────────
     │ CRITICAL (node starts blocking new messages)
 80% ├────────────────────────────────────
     │ ALARM (aggressive paging, connections suspended)
 60% ├────────────────────────────────────
     │ WARNING (paging active, but still accepting)
 40% ├────────────────────────────────────
     │ NORMAL (no paging)
  0% └────────────────────────────────────

Monitor this with:
rabbitmqctl eval 'erlang:memory().'

# System has 16GB total RAM
# RabbitMQ can use max: ~14GB (leaving 2GB for OS)

# Configuration:
vm_memory_high_watermark.relative = 0.4
# 40% of 14GB = 5.6GB
# When RabbitMQ uses > 5.6GB RAM: START PAGING

# Real-world impact:

Case 1: Light load (2GB in use)
  Status: NORMAL
  Paging: OFF
  Performance: ✅ Excellent (all messages in RAM)

Case 2: Moderate load (5.5GB in use)
  Status: NORMAL → approaching WARNING
  Paging: OFF → Starting
  Performance: ✅ Good (most messages in RAM)

Case 3: Heavy load (8GB in use)
  Status: WARNING
  Paging: ACTIVE
  Performance: ⚠️  Degraded (frequent disk I/O)

Case 4: Overload (13GB in use)
  Status: ALARM → CRITICAL
  Paging: AGGRESSIVE
  Performance: ❌ Bad (most messages on disk, frequent seeks)
  New producers: BLOCKED (can't send more messages)

const amqp = require('amqplib');

async function setupQueueWithMemoryPolicy() {
  const conn = await amqp.connect('amqp://localhost');
  const ch = await conn.createChannel();
  
  // Queue with memory limit
  await ch.assertQueue('high-priority-queue', {
    durable: true,
    arguments: {
      // Max 100MB for this queue only
      'x-max-length-bytes': 100 * 1024 * 1024,
      
      // OR max 10,000 messages
      'x-max-length': 10000,
      
      // What to do when limit exceeded?
      // 'drop-head': Remove oldest (default)
      // 'reject-publish': Reject new messages
      'x-overflow': 'drop-head'
    }
  });
  
  console.log('Queue with memory limit created');
}

Queue: "orders"
Memory limit: 100MB
Current size: 90MB

Message arrives (11 bytes):
  Step 1: Check if adding message exceeds limit
          90MB + 11 bytes = 90MB + 0.000011MB < 100MB ✓
  
  Step 2: Add message
          Queue size: 90.000011MB

[Later...]

Queue: "orders"
Memory limit: 100MB
Current size: 99.999MB

Message arrives (10MB):
  Step 1: Check if adding message exceeds limit
          99.999MB + 10MB > 100MB ✗ LIMIT EXCEEDED
  
  Step 2: x-overflow = 'drop-head'
          Remove oldest message (was 5MB)
  
  Step 3: Queue size: 99.999 - 5 + 10 = 104.999MB
          Wait, still over! Remove another message (8MB)
  
  Step 4: Queue size: 96.999MB < 100MB ✓
  
  Step 5: Add new message
          Queue size: 106.999MB
          
Wait, that exceeds 100MB!
Actually, RabbitMQ removes UNTIL new message fits, allowing overflow:
  99.999 - 5 - 8 + 10 = 96.999MB
  But some implementations allow the overflow

If queue at limit: reject the new message immediately
Producer gets error: 406 PRECONDITION_FAILED
Queue stays at/under limit

await ch.assertQueue('transient-queue', {
  durable: false  // ← Not persistent
});

const msg = { data: 'important work' };
ch.sendToQueue('transient-queue', Buffer.from(JSON.stringify(msg)));

Message only exists in RAM queue
No disk writes
No WAL (write-ahead log) entries
Memory freed immediately when acked

If RabbitMQ crashes:
  Message is GONE forever
  
Fast but risky:
  ✓ Fast publish (no disk I/O)
  ✓ Low memory overhead (no persistent store)
  ✗ Data loss on crash

await ch.assertQueue('persistent-queue', {
  durable: true  // ← Persistent
});

const msg = { data: 'important transaction' };
ch.sendToQueue('persistent-queue', Buffer.from(JSON.stringify(msg)), {
  persistent: true  // ← Both queue AND message
});

Message written to:
  1. RAM queue (fast access)
  2. Disk (Write-Ahead Log / persistent store)

Actually two copies exist temporarily:
  Memory: ~11 bytes/message
  Disk: ~11 bytes/message
  Total: ~22 bytes per message

RabbitMQ only confirms to producer after disk write.
Slower but safer.

If RabbitMQ crashes:
  Messages recovered from disk
  No data loss
  
When RabbitMQ restarts:
  Loads durable queues from disk
  Can take minutes if millions of messages

Message Durability:      Memory Usage:           Disk Usage:
─────────────────────────────────────────────────────────────

Non-durable              Minimal                 None
  Message in RAM only    (just message)          (messages lost
                                                  on crash)

Durable                  Doubled initially       Full message
  Message in RAM+Disk    (RAM + disk while      stored to disk
                         writing, then just
                         RAM after page)

Example: 1 million messages, 1KB each
─────────────────────────────────────────────────────────────
Non-durable: ~1GB RAM max
Durable: ~1GB RAM + ~1GB Disk writes + paging

Producer publishes durable message:

1. Message received
2. Before keeping in RAM queue...
3. Write to disk: "queue-x, message-y, [data]" ← WAL
4. Flush to disk (fsync)
5. NOW add to RAM queue
6. Confirm to producer: "message persisted"

/var/lib/rabbitmq/mnesia/[node-name]/queues/

# Directory structure:
├── queue-1-seg-1  # Segment 1 of queue-1 WAL
├── queue-1-seg-2  # Segment 2 of queue-1 WAL
├── queue-1.idx    # Index file (metadata)
├── queue-2-seg-1
├── queue-2-seg-2
└── ...

[Header] [Entry1] [Entry2] [Entry3] ... [Entry1000]

Entry format:
  ┌──────────────────────────────┐
  │ Timestamp   │ Message data   │
  │ (8 bytes)   │ (variable)     │
  │             │                │
  │ Checksum    │ Flags          │
  │ (4 bytes)   │ (1 byte)       │
  └──────────────────────────────┘

const ch = await conn.createChannel();

// Non-durable: Just RAM
console.time('non-durable publish');
for (let i = 0; i < 10000; i++) {
  ch.sendToQueue('temp-queue', Buffer.from('msg'));
}
console.timeEnd('non-durable publish');
// Output: ~50ms (super fast, no disk I/O)

// Durable: RAM + Disk WAL
console.time('durable publish');
for (let i = 0; i < 10000; i++) {
  ch.sendToQueue('durable-queue', Buffer.from('msg'), 
    { persistent: true });
}
console.timeEnd('durable publish');
// Output: ~5000ms (100x slower! Disk I/O overhead)

Write 10 messages to WAL:
  Old way: 10 separate fsync calls = ~70ms
  Batch way: 1 fsync call for all 10 = ~7ms
  
RabbitMQ batches by default, so actual: ~500ms
(Still 10x slower than non-durable, but acceptable)

# Get memory statistics
rabbitmqctl eval 'erlang:memory().'

# Output:
# {total,4729034},
#  {processes,1840101},
#  {processes_used,1840101},
#  {system,2888933},
#  {atom,523421},
#  {atom_used,507451},
#  {binary,123456},
#  {code,7891234},
#  {ets,456789}

# Interpret:
# total = 4.7MB total memory
# processes = Erlang processes
# system = System allocations
# code = Compiled code in memory
# ets = Erlang Term Storage (queues, etc.)

# Get node memory details
curl -s http://localhost:15672/api/nodes \
  -u guest:guest | jq '.[] | {name, memory_used, memory_limit}'

# Output:
# {
#   "name": "rabbit@localhost",
#   "memory_used": 8453234,     # 8.4MB
#   "memory_limit": 14532424,   # 14.5MB (40% of 16GB)
#   "memory_alarm": false
# }

# Get queue memory usage
curl -s http://localhost:15672/api/queues \
  -u guest:guest | jq '.[] | {name, memory, messages}'

# Output:
# [
#   {
#     "name": "orders",
#     "memory": 1234567,        # 1.2MB
#     "messages": 5000          # 5000 messages in queue
#   },
#   {
#     "name": "notifications",
#     "memory": 567890,         # 567KB
#     "messages": 1000
#   }
# ]

# Get queue stats
QUEUE_NAME="orders"

MEMORY=$(curl -s "http://localhost:15672/api/queues/%2F/${QUEUE_NAME}" \
  -u guest:guest | jq '.memory')

MESSAGES=$(curl -s "http://localhost:15672/api/queues/%2F/${QUEUE_NAME}" \
  -u guest:guest | jq '.messages')

BYTES_PER_MSG=$((MEMORY / MESSAGES))

echo "Queue: $QUEUE_NAME"
echo "Total Memory: $MEMORY bytes"
echo "Message Count: $MESSAGES"
echo "Bytes per Message: $BYTES_PER_MSG"

# Example output:
# Queue: orders
# Total Memory: 1234567 bytes
# Message Count: 5000
# Bytes per Message: 247  (includes overhead!)

# Check if memory alarm is active
rabbitmqctl status | grep memory_alarm

# If memory_alarm is true:
# RabbitMQ has hit the high watermark
# Paging is active
# New publishers may be blocked

# Check memory threshold
rabbitmqctl eval 'rabbit:vm_memory_monitor_get_watermark().'
# Returns: {relative, 0.4} or similar

# Real-time monitoring
watch -n 1 "rabbitmqctl eval 'erlang:memory().' | grep total"

# /etc/rabbitmq/rabbitmq.conf

# More aggressive paging (start earlier)
vm_memory_high_watermark.relative = 0.3
# Start paging at 30% instead of 40%
# Pro: Prevents memory exhaustion
# Con: More disk I/O, slower

# Less aggressive paging (start later)
vm_memory_high_watermark.relative = 0.5
# Start paging at 50%
# Pro: Keep more in RAM, faster
# Con: Risk of OOM if traffic spikes

// For temporary work (don't need durability)
await ch.assertQueue('temp-processing', { durable: false });

// For critical data (need durability)
await ch.assertQueue('orders', { durable: true });

// Prevent single queue from consuming all memory
await ch.assertQueue('user-events', {
  durable: true,
  arguments: {
    'x-max-length-bytes': 500 * 1024 * 1024,  // 500MB max
    'x-overflow': 'drop-head'
  }
});

// ❌ Large message (5KB)
const msg = {
  userId: 123,
  userName: 'John Doe',
  email: 'john@example.com',
  fullProfile: {...large object...},  // Lots of unnecessary data
  timestamp: Date.now()
};
ch.sendToQueue('users', Buffer.from(JSON.stringify(msg)));

// ✅ Optimized message (500 bytes)
const msg = {
  userId: 123,
  timestamp: Date.now()
};
ch.sendToQueue('users', Buffer.from(JSON.stringify(msg)));
// Consumer can fetch full profile from DB if needed

const ch = await conn.createChannel();

// ❌ Slow consumer (bottleneck)
ch.prefetch(1);  // Only 1 message at a time
ch.consume('orders', async (msg) => {
  await slowDatabaseInsert(msg);  // Takes 1 second per message
  ch.ack(msg);
});
// Processing 1 message/sec
// 1 million messages = 11.5 days!

// ✅ Fast consumer (keeps up)
ch.prefetch(100);  // 100 messages at a time
ch.consume('orders', async (msg) => {
  // Same slow DB insert, but...
  fastBatchInsert([messages]);  // Batch inserts (faster)
  ch.ack(msg);
});
// Processing 100 messages/sec
// 1 million messages = ~2.8 hours

// Classic queue (eager) - keeps messages in RAM
await ch.assertQueue('eager-queue', {
  durable: true,
  arguments: {
    'x-queue-type': 'classic'
  }
});

// Lazy queue - keeps messages on disk until accessed
await ch.assertQueue('lazy-queue', {
  durable: true,
  arguments: {
    'x-queue-type': 'lazy'
  }
});

Classic (Eager):
  Message arrives → Stored in RAM → Paged to disk if memory pressure

Lazy:
  Message arrives → Stored on disk immediately → Loaded to RAM when needed

Lazy queue memory: 95% lower than classic!
Classic queue latency: 95% lower than lazy!

Trade-off: Pick based on your priority
  Critical low-latency: Use classic queues
  High-volume buffering: Use lazy queues

# Symptoms:
# - Memory grows every hour
# - Paging increases
# - Eventually runs out of memory

# Investigation:
curl -s http://localhost:15672/api/queues \
  -u guest:guest | jq 'sort_by(.memory) | reverse | .[:5]'

# Output: Top 5 queues by memory usage
# Check if one queue is growing endlessly

# Diagnosis:
# Consumers not acknowledging messages
# Messages accumulate in queue
# Memory keeps growing

# Fix:
# 1. Check consumer code for missing ch.ack()
# 2. Check for long-running consumer (blocked on I/O)
# 3. Add consumer timeout

Normal traffic: 100 messages/sec
Suddenly: 10,000 messages/sec (spike)

Without memory tuning:
  T0: 100MB/sec queued
  T1: RAM fills
  T2: Paging starts (slow)
  T3: Memory alarm triggered
  T4: New publishers blocked
  T5: Cascade failure

With memory tuning:
  T0: Pre-configured memory limit: 500MB per queue
  T1: Excess messages dropped (x-overflow: drop-head)
  T2: System stays responsive
  T3: Alerting triggered, engineers notified
  T4: Auto-scaling adds RabbitMQ nodes
  T5: Spike absorbed, no cascade failure

# Monitor over 24 hours

# Hour 0
rabbitmqctl eval 'erlang:memory(total).'
# {total, 1073741824}  # 1GB

# Hour 6
rabbitmqctl eval 'erlang:memory(total).'
# {total, 2147483648}  # 2GB (doubles!)

# Hour 12
# {total, 3221225472}  # 3GB

# Hour 24
# {total, 5368709120}  # 5GB

# Problem: Memory grows 200MB per hour
# Likely causes:
# 1. Connection leaks (new connections not closing)
# 2. Channel leaks (channels not closing)
# 3. Queue accumulation (messages not being consumed)
# 4. Internal memory fragmentation (Erlang GC issue)

# Investigation:
rabbitmqctl list_connections | wc -l    # Check connection count
rabbitmqctl list_channels | wc -l         # Check channel count
curl -s http://localhost:15672/api/queues -u guest:guest | \
  jq '[.[] | select(.messages > 0)] | length'  # Non-empty queues

# /etc/rabbitmq/rabbitmq.conf

# Memory threshold (start paging)
vm_memory_high_watermark.relative = 0.4

# Enable flow control (block publishers when memory high)
credit_flow_default_credit = 400

# Page cache size
vm_memory_monitor_interval = 5000  # milliseconds

# Lazy queue default (keep on disk)
queue_master_locator = min-masters

# Paging segment size (page store files)
# Larger = fewer files, less overhead
# Smaller = more granular paging
disk_free_limit.relative = 0.1

# Maximum file descriptor limit
nofile = 65535

# How often does RabbitMQ check memory?
vm_memory_monitor_interval = 5000  # milliseconds

# If memory spikes to 100% in < 5 seconds:
# May not catch it in time!

# For bursty traffic, reduce interval:
vm_memory_monitor_interval = 1000  # Check every 1 second