Building Robust Health Checks for Microservices 🚀

Introduction
Health checks are an integral part of modern service architectures, ensuring systems are running smoothly and identifying issues early. This article introduces the concept of a /health_check endpoint, a standard monitoring tool that every service should provide. It serves both automated systems and human operators to check the health of a service, ensuring everything is functioning as expected and allowing quick diagnosis when things go wrong.

🎯 Motivation

Automated tools, like Kubernetes, use liveness and readiness probes to monitor service health and take corrective actions. By implementing a reliable health check, we improve service resilience and provide clear insights into system performance.

🔍 Key Probes and Their Purpose

1. Liveness Probe
   Ensures a service is still alive and functioning. Services can sometimes end up in broken states that can only be recovered by restarting them. Liveness probes catch these issues and allow the system to automatically restart the service.

   livenessProbe:
     httpGet:
       path: /health_check
       port: 8000
     initialDelaySeconds: 10
     periodSeconds: 120
     successThreshold: 1
     failureThreshold: 3
     timeoutSeconds: 5
   

   How It Works:
   Kubernetes periodically hits the /health_check endpoint to confirm the service is responding. If it fails (e.g., returns a 503 status), it retries and, upon continuous failure, triggers a restart.

   For humans: Check if the service is operational by reviewing the response:

   • 200 OK: All good!
   • 503 Service Unavailable: Service needs a restart.

2. Readiness Probe
   Verifies if a service is ready to accept traffic. Sometimes, a service might be alive but not ready to handle requests (e.g., it’s still loading configuration). The readiness probe prevents sending traffic to an unprepared service.

   readinessProbe:
     httpGet:
       path: /health_check?check_dependencies=true
       port: 8000
     initialDelaySeconds: 5
     periodSeconds: 300
     successThreshold: 1
     failureThreshold: 1
     timeoutSeconds: 5
   

   How It Works:
   Checks if a service can handle requests by verifying its dependencies, like databases or caches.

   For humans: Get a detailed look at service readiness by using:

   curl -i "http://localhost:8080/health_check?check_dependencies=true&full=true"
   

   • pass: System and dependencies are healthy.
   • degraded: Service can handle traffic but with some limitations.
   • fail: Service is not ready.

🏁 Startup Probe

Ensures that the service has started correctly. This probe is crucial for slow-starting containers, which might otherwise fail liveness checks before they're fully operational.

startupProbe:
  httpGet:
    path: /health_check?check_dependencies=true&strict=true
    port: 8000
  failureThreshold: 30
  periodSeconds: 10

This probe delays liveness and readiness probes until the service is confirmed to have started, avoiding unnecessary restarts.

📊 System State Representation

The /health_check endpoint offers insights into the system's state. A JSON object is returned, detailing the health of the service and its dependencies:

{
  "status": "degraded",
  "version": "0.0.1-dev",
  "dependencies": {
    "configuration": "pass",
    "database": "pass",
    "blob_storage": "pass",
    "redis": "fail"
  }
}

• pass: Service and dependencies are fully operational.
• fail: Service or a crucial dependency has failed.
• degraded: A non-critical dependency is down, but the service remains functional.

💡 Design Guidelines for Health Checks

Query Parameters

Health checks support flexible queries for more granular control:

Example endpoints:

• Basic check: /health_check
• Dependency check: /health_check?check_dependencies=true
• Strict check: /health_check?check_dependencies=true&strict=true

Unauthenticated Endpoint

Most cases recommend leaving the /health_check endpoint unauthenticated to allow easy integration with monitoring tools and infrastructure systems.

🔐 Security Considerations

For security reasons, the /health_check endpoint should only reveal high-level status like pass or fail for dependencies. Sensitive details such as endpoint URLs or credentials should be logged but not included in the public response.

Example Response:

{
  "status": "fail",
  "version": "0.0.1-dev",
  "dependencies": {
    "redis": "fail",
    "database": "pass",
    "blob_storage": "pass"
  }
}

🔧 Monitoring and Alerting

Automated monitoring systems like Datadog, Prometheus, or Statuspage frequently query the /health_check endpoint. Monitoring tools rely on HTTP status codes to trigger alerts or corrective actions:

• 200 OK: Service is healthy.
• 503 Service Unavailable: Service or dependency is down, prompting alert.

Tip: Be mindful of the frequency of checks to prevent overloading the service. Adjust the interval for more relaxed monitoring if necessary.

📈 Scaling and Resource Usage

Health checks can affect resource usage, especially when executed frequently. Ensure checks are optimized to avoid impacting service performance while ensuring they don’t monopolize resources.

🌐 Kubernetes Configuration Example

In Kubernetes, both readiness and liveness probes are specified within the container's deployment configuration:

readinessProbe:
  httpGet:
    path: /health_check?check_dependencies=true
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 300

livenessProbe:
  httpGet:
    path: /health_check
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 120

This setup ensures services remain both stable and performant while automatically recovering from failure states.

Conclusion 🎉

Health checks are a foundational element for building resilient services. They provide essential insights into the operational state of a service and its dependencies, helping systems self-heal and alert operators of potential issues. Whether you are running Kubernetes or another platform, a well-implemented /health_check endpoint is a must-have for any service in today's dynamic infrastructure world.

By adopting the standard design principles and configurations outlined here, your services will be healthier, more reliable, and easier to maintain.