Skip to content

Production Testing Strategy

Real-World Validation of Tonight's Features

Overview

Our synthetic tests verify that endpoints exist and return expected responses, but production testing validates that features actually improve performance, reliability, and user experience under real-world conditions.

🗄️ Database Connection + Supabase Caching (Supavisor)

Production Testing Approach

Real Performance Testing

# Load test with real database queries
k6 run --vus 50 --duration 5m database-load-test.js

# Compare direct vs pooler performance
artillery run database-performance-comparison.yml

Connection Pool Testing

  • Test Scenario: Simulate 100+ concurrent users making database queries
  • Metrics to Monitor:
  • Connection pool utilization
  • Query response times (p50, p95, p99)
  • Connection establishment time
  • Pool exhaustion scenarios

Failover Testing

# Test what happens when Supavisor is unavailable
# Should gracefully fall back to direct connection
curl -X POST http://localhost:8000/api/v1/test/simulate-pooler-failure

Production Monitoring

# Supabase Dashboard Metrics
- Connection pool usage
- Query performance (before/after Supavisor)
- Error rates and timeouts
- Cache hit rates

# Custom Application Metrics
- Database response times by endpoint
- Connection pool health
- Fallback activation frequency

🚀 HTTP Cache Headers

Production Testing Approach

CDN Integration Testing

# Test with real CDN (Cloudflare/AWS CloudFront)
curl -H "Cache-Control: max-age=0" https://your-domain.com/api/v1/test/cache-test
curl -I https://your-domain.com/api/v1/test/cache-test  # Check cache headers

Cache Performance Validation

  • Test Scenario: Measure actual cache hit rates in production
  • Tools:
  • WebPageTest for real-world performance
  • Lighthouse for Core Web Vitals
  • CDN analytics dashboards

Cache Invalidation Testing

// Test cache invalidation strategies
const testCacheInvalidation = async () => {
  // 1. Make request, verify cached
  // 2. Update data
  // 3. Verify cache invalidated
  // 4. Verify new data served
};

Production Monitoring

# CDN Metrics
- Cache hit ratio (target: >90%)
- Cache miss reasons
- Origin server load reduction
- Time to first byte (TTFB)

# Application Metrics  
- ETag generation performance
- Cache header consistency
- Cache-related error rates

⚙️ Background Processing (Supabase Queues)

Production Testing Approach

Real Task Processing

# Submit actual production tasks
async def test_real_background_tasks():
    tasks = [
        {"type": "send_notification", "user_id": 123},
        {"type": "process_sensor_data", "data": sensor_readings},
        {"type": "generate_report", "farm_id": 456}
    ]

    for task in tasks:
        result = await background_service.submit_task(task)
        # Monitor actual processing time and success rate

Queue Performance Testing

  • Test Scenario: Submit 1000+ tasks and monitor processing
  • Metrics:
  • Queue depth over time
  • Task processing latency
  • Throughput (tasks/second)
  • Error and retry rates

Failure Scenario Testing

# Test dead letter queue behavior
async def test_task_failures():
    # Submit task that will fail
    # Verify retry mechanism
    # Verify dead letter queue handling
    # Test manual task recovery

Production Monitoring

# Supabase Queue Metrics
- Queue depth by priority
- Processing time percentiles
- Error rates by task type
- Dead letter queue size

# Application Metrics
- Task success/failure rates
- Processing time by task type
- Queue backlog alerts
- Worker health status

🏠 Home Assistant Error Handling

Production Testing Approach

Real Device Integration

# Connect to actual Home Assistant instance
async def test_real_ha_integration():
    # Test with real devices (lights, sensors, switches)
    # Verify state synchronization
    # Test device control commands
    # Monitor WebSocket stability

Network Resilience Testing

# Simulate network issues
# Use tools like tc (traffic control) or toxiproxy
tc qdisc add dev eth0 root netem delay 100ms loss 5%

Authentication & Security Testing

  • Test token expiration and renewal
  • Test invalid credentials handling
  • Test rate limiting behavior
  • Test SSL/TLS certificate validation

Production Monitoring

# Home Assistant Integration Metrics
- Connection uptime percentage
- WebSocket reconnection frequency
- Device state sync accuracy
- Command success rates

# Error Handling Metrics
- Graceful degradation activation
- Error recovery time
- User experience during outages

🔧 Production Testing Tools & Infrastructure

Load Testing Tools

# k6 for API load testing
k6 run --vus 100 --duration 10m production-load-test.js

# Artillery for complex scenarios
artillery run production-scenarios.yml

# Apache Bench for simple tests
ab -n 1000 -c 10 http://localhost:8000/api/v1/health

Monitoring & Observability

# Application Performance Monitoring
- Datadog / New Relic / Supabase Analytics
- Custom metrics dashboards
- Real-time alerting

# Infrastructure Monitoring
- Server resource usage
- Database performance
- Network latency and errors
- CDN performance

Chaos Engineering

# Chaos Monkey-style testing
async def chaos_tests():
    # Kill database connections randomly
    # Introduce network latency
    # Simulate service outages
    # Test circuit breaker behavior

📊 Production Success Metrics

Database & Caching

  • Target: 40x performance improvement (200ms → <5ms for cached queries)
  • SLA: 99.9% database availability
  • Alert: Query time >100ms for cached endpoints

Background Processing

  • Target: <30 second task processing time
  • SLA: 99.5% task success rate
  • Alert: Queue depth >100 tasks

Home Assistant Integration

  • Target: 99% device command success rate
  • SLA: <5 second device state updates
  • Alert: WebSocket disconnection >1 minute

Overall System Health

  • Target: <2 second page load times
  • SLA: 99.9% API uptime
  • Alert: Error rate >1%

🚀 Gradual Rollout Strategy

Phase 1: Canary Deployment (5% traffic)

  • Deploy to small subset of users
  • Monitor all metrics closely
  • Validate performance improvements

Phase 2: Blue-Green Testing (50% traffic)

  • A/B test new features vs old implementation
  • Compare user experience metrics
  • Validate business impact

Phase 3: Full Production (100% traffic)

  • Complete rollout with monitoring
  • Establish baseline metrics
  • Set up automated alerts

🔍 Continuous Production Testing

Automated Health Checks

# Run every 5 minutes
./production-health-check.sh

# Includes:
# - Real database query performance
# - Cache hit rate validation  
# - Background task processing
# - Home Assistant connectivity

Weekly Performance Reviews

  • Analyze performance trends
  • Review error rates and patterns
  • Optimize based on real usage data
  • Plan capacity scaling

Monthly Chaos Engineering

  • Scheduled failure testing
  • Disaster recovery validation
  • Performance regression testing
  • Security penetration testing

🎯 Key Differences: Synthetic vs Production Testing

Aspect Synthetic Tests Production Tests
Data Mock/fake data Real user data
Load Single requests Realistic traffic patterns
Environment Development Production infrastructure
Metrics HTTP status codes Business impact metrics
Duration Seconds Hours/days/weeks
Scope Individual features End-to-end workflows
Risk Low (dev environment) High (real users affected)
Value Smoke testing User experience validation

The goal shifts from "Does it work?" to "Does it improve the real user experience and business outcomes?"