home/Projects/kompose/docs/content/5.stacks/data.md

title, description, navigation

title	description	navigation
Data - The Memory Palace of Your Infrastructure	In data we trust... and backup, and replicate, and backup again	icon i-lucide-database

"In data we trust... and backup, and replicate, and backup again" - Every DBA Ever

What's This All About?

This is the beating heart of your infrastructure - where all the data lives, breathes, and occasionally takes a nap (cache). Think of it as the library, post office, and filing cabinet all rolled into one extremely organized digital space!

The Data Dream Team

:icon{name="simple-icons:postgresql"} PostgreSQL

Container: data_postgres
Image: postgres:latest
Port: 5432
Volume: pgdata

The elephant in the room (literally, look at the logo!). PostgreSQL is your rock-solid relational database:

• :icon{name="lucide:dumbbell"} ACID Compliance: Your data stays consistent, always
• :icon{name="lucide:lock"} Rock Solid: Banks trust it, you should too
• :icon{name="lucide:bar-chart"} Advanced Features: JSON, full-text search, geospatial data
• :icon{name="lucide:rocket"} Performance: Handles millions of rows like a champ
• :icon{name="lucide:refresh-cw"} Extensible: PostGIS, TimescaleDB, and more

Who Uses It:

• auth → Keycloak database
• news → Newsletter/Letterspace database
• auto → Semaphore database
• sexy → Directus CMS database
• track → Umami analytics database
• Basically, everyone! :icon{name="lucide:party-popper"}

:icon{name="lucide:zap"} Redis

Container: data_redis
Image: redis:latest
Port: 6379

The speed demon of data storage! Redis is your in-memory cache:

• :icon{name="lucide:car"} Lightning Fast: Sub-millisecond response times
• :icon{name="lucide:hard-drive"} In-Memory: Data lives in RAM for max speed
• :icon{name="lucide:key"} Key-Value Store: Simple and effective
• :icon{name="lucide:package"} Pub/Sub: Real-time messaging support
• :icon{name="lucide:clock"} Expiration: Auto-delete old data

Who Uses It:

• sexy → Directus cache for faster API responses
• Perfect for session storage, rate limiting, queues

:icon{name="lucide:ethernet-port"} pgAdmin 4

Container: pgadmin4_container
Image: dpage/pgadmin4
Port: 8088
Home: http://localhost:8088

Your graphical database management interface:

• :icon{name="lucide:mouse"} Visual Interface: No SQL required (but you can if you want!)
• :icon{name="lucide:bar-chart"} Query Tool: Run queries and see pretty results
• :icon{name="lucide:search"} Database Explorer: Browse tables, views, functions
• :icon{name="lucide:trending-up"} Monitoring: Check performance and connections
• :icon{name="lucide:hammer"} Management: Create, modify, backup databases

Architecture Overview

Your Applications
    ├── PostgreSQL (Persistent Data)
    │   ├── auth/keycloak DB
    │   ├── news/letterspace DB
    │   ├── auto/semaphore DB
    │   ├── sexy/directus DB
    │   └── track/umami DB
    │
    └── Redis (Cache & Speed)
        └── sexy/directus cache

pgAdmin 4 (You) → PostgreSQL (Manage everything)

Configuration Breakdown

PostgreSQL Setup

User & Password: Configured in root .env file

DB_USER=your_db_user
DB_PASSWORD=super_secret_password

Health Check:

pg_isready -d postgres

Runs every 30 seconds to ensure the database is accepting connections.

Redis Setup

Health Check:

redis-cli ping
# Response: PONG (if healthy)

Checks every 10 seconds because Redis is speedy like that!

pgAdmin Setup

Credentials: From root .env

ADMIN_EMAIL=your@email.com
ADMIN_PASSWORD=your_password

Data Persistence: pgadmin-data volume stores your server configurations

First Time Setup :icon

Postgres

1. Create a new database:

   docker exec -it data_postgres psql -U your_db_user
   

   CREATE DATABASE myapp;
   \q
   

2. Or use pgAdmin (easier for beginners):

   • Access http://localhost:8088
   • Login with admin credentials
   • Right-click "Databases" → Create → Database

Redis cache

Redis works out of the box! No setup needed. Just start using it:

docker exec -it data_redis redis-cli
> SET mykey "Hello Redis!"
> GET mykey
"Hello Redis!"
> EXIT

pgAdmin

1. First Login:

   URL: http://localhost:8088
   Email: Your ADMIN_EMAIL
   Password: Your ADMIN_PASSWORD
   

2. Add PostgreSQL Server:

   • Right-click "Servers" → Register → Server
   • General Tab:
     • Name: "Kompose PostgreSQL"
   • Connection Tab:
     • Host: postgres (container name)
     • Port: 5432
     • Database: postgres
     • Username: Your DB_USER
     • Password: Your DB_PASSWORD
   • Save!

Common Database Tasks

Create a New Database

Via pgAdmin:

1. Right-click "Databases"
2. Create → Database
3. Name it, set owner, click Save

Via Command Line:

docker exec data_postgres createdb -U your_db_user myapp

Backup a Database

# Backup to file
docker exec data_postgres pg_dump -U your_db_user myapp > backup.sql

# Or use pg_dumpall for everything
docker exec data_postgres pg_dumpall -U your_db_user > all_databases.sql

Restore a Database

# Restore from backup
docker exec -i data_postgres psql -U your_db_user myapp < backup.sql

Connect from Application

// Node.js example
const { Pool } = require('pg')
const pool = new Pool({
  host: 'postgres',  // Container name
  port: 5432,
  database: 'myapp',
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD
})

Monitor Active Connections

SELECT * FROM pg_stat_activity;

Check Database Sizes

SELECT 
    datname AS database,
    pg_size_pretty(pg_database_size(datname)) AS size
FROM pg_database
ORDER BY pg_database_size(datname) DESC;

Redis Common Tasks

Check Redis Stats

docker exec data_redis redis-cli INFO stats

Monitor Commands in Real-Time

docker exec -it data_redis redis-cli MONITOR

Flush All Data (:icon{name="lucide:alert-triangle"} DANGER!)

docker exec data_redis redis-cli FLUSHALL
# Only do this if you know what you're doing!

Check Memory Usage

docker exec data_redis redis-cli INFO memory

Ports & Networking

Service	Internal Port	External Port	Access
PostgreSQL	5432	5432	Direct + kompose network
Redis	6379	6379	Direct + kompose network
pgAdmin	80	8088	http://localhost:8088

Volumes & Persistence :icon

pgdata

PostgreSQL database files live here. DON'T DELETE THIS unless you enjoy pain!

pgadmin-data

Your pgAdmin settings and configurations.

Security Best Practices :icon

1. Strong Passwords: Use long, random passwords
2. Network Isolation: Only expose ports you need
3. Regular Backups: Automate them!
4. User Permissions: Don't use superuser for applications
5. SSL Connections: Consider enabling for production
6. Update Regularly: Keep images up to date

Performance Tips :icon

PostgresSQL Server

1. Indexes: Create them on frequently queried columns

   CREATE INDEX idx_user_email ON users(email);
   

2. Analyze Queries:

   EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com';
   

3. Vacuum Regularly:

   VACUUM ANALYZE;
   

Redis Service

1. Use Appropriate Data Structures: Lists, Sets, Hashes, etc.
2. Set Expiration: Don't let cache grow forever

   SET key value EX 3600  # Expires in 1 hour
   

3. Monitor Memory: Keep an eye on RAM usage

Troubleshooting

Q: PostgreSQL won't start?

# Check logs
docker logs data_postgres

# Check if port is in use
lsof -i :5432

Q: Can't connect to database?

• Verify credentials in .env
• Check if container is healthy: docker ps
• Ensure network is correct: docker network ls

Q: Redis out of memory?

# Check memory
docker exec data_redis redis-cli INFO memory

# Configure max memory (if needed)
docker exec data_redis redis-cli CONFIG SET maxmemory 256mb

Q: pgAdmin can't connect to PostgreSQL?

• Use container name postgres, not localhost
• Check if both containers are on same network
• Verify credentials match

Advanced: Connection Pooling

For high-traffic apps, use connection pooling:

PgBouncer (PostgreSQL):

# Could add to this stack
pgbouncer:
  image: pgbouncer/pgbouncer
  environment:
    DATABASES_HOST: postgres
    DATABASES_PORT: 5432

Monitoring & Metrics

PostgreSQL Server

• pg_stat_statements: Track slow queries
• pg_stat_user_tables: Table statistics
• pg_stat_database: Database-level stats

Redis System

• INFO command: Comprehensive stats
• SLOWLOG: Track slow commands
• CLIENT LIST: Active connections

When Things Go Wrong :icon

Database Corruption

1. Stop all applications
2. Restore from latest backup
3. Investigate what caused corruption

Out of Disk Space

1. Check volume sizes: docker system df -v
2. Clean old backups
3. Archive old data
4. Consider larger disk

Connection Pool Exhaustion

1. Check active connections
2. Kill long-running queries
3. Increase max_connections (PostgreSQL)
4. Implement connection pooling

Fun Database Facts :icon

• PostgreSQL started in 1986 at UC Berkeley (older than some developers!)
• Redis stands for "REmote DIctionary Server"
• PostgreSQL supports storing emojis (:icon{name="simple-icons:postgresql"}💖)
• Redis can process millions of operations per second
• pgAdmin is used by database admins worldwide

Resources

• PostgreSQL Documentation
• Redis Documentation
• pgAdmin Documentation
• PostgreSQL Tutorial

---

"Data is the new oil, but unlike oil, you can actually back it up." - Modern DevOps Proverb :icon{name="lucide:hard-drive"}:icon{name="lucide:sparkles"}