feat(n8n-observability): añadir stack de observabilidad con agente IA

Stack de scripts y plantillas para generar reportes diarios del homelab usando n8n como orquestador y un modelo de IA como redactor: - scripts/server-summary.sh: métricas del sistema (CPU, RAM, disco) - scripts/log-digest.sh: resumen de logs de Traefik y contenedores - scripts/health-results.sh: estado de servicios y checks - scripts/codex-reauth.sh: reautenticación OAuth del proveedor Codex - templates/ai-daily-report-prompt.md: prompt para el informe diario - workflows/workflow-design.md: diseño del workflow de n8n - config/services.example.json: ejemplo de configuración de servicios Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-25 05:48:07 +00:00
parent 6b641981dd
commit 46d7a28210
8 changed files with 768 additions and 0 deletions
@@ -0,0 +1,38 @@
+# n8n Observability Toolkit
+
+Carpeta dedicada para automatizar en n8n:
+- Resumen diario de estado del servidor
+- Alertas de caida real de servicios
+- Informes automaticos de logs y errores
+- Resumen IA (texto ejecutivo)
+
+## Estructura
+
+- `scripts/server-summary.sh`: estado de host + docker + certs + errores top
+- `scripts/log-digest.sh`: digest de logs (journal + docker) para alertas/informes
+- `scripts/health-results.sh`: chequeo HTTP de endpoints y salida JSON (`up/down`, latencia, codigo)
+- `templates/ai-daily-report-prompt.md`: prompt para convertir datos tecnicos en informe legible
+- `config/services.example.json`: lista base de endpoints para health checks en n8n
+- `workflows/workflow-design.md`: diseno de 3 workflows recomendados
+
+## Uso rapido (en el host)
+
+```bash
+chmod +x n8n-observability/scripts/*.sh
+./n8n-observability/scripts/server-summary.sh
+SINCE="24 hours ago" ./n8n-observability/scripts/log-digest.sh
+./n8n-observability/scripts/health-results.sh | jq
+```
+
+## Integracion recomendada con n8n
+
+1. En n8n, crea credencial SSH al host.
+2. Workflow diario: `Schedule Trigger` -> `SSH (log-digest.sh)` -> `SSH (server-summary.sh)` -> `SSH (health-results.sh)` -> `AI` -> `Email/Telegram`.
+3. Workflow de logs: `Schedule Trigger` -> `SSH (log-digest.sh)` -> `AI` -> `Email/Telegram`.
+4. Workflow de caida real: `Schedule Trigger cada 2 min` -> `HTTP Request` por endpoint -> logica de reintentos -> alerta.
+
+## Seguridad
+
+- No envies logs completos a IA; envia solo digest/errores relevantes.
+- Enmascara secretos (tokens, API keys, passwords) antes de mandar datos fuera.
+- Mantener este toolkit separado facilita auditoria y versionado.
@@ -0,0 +1,23 @@
+[
+  {
+    "name": "nextcloud",
+    "url": "https://nextcloud.sherlockhomeless.net/status.php",
+    "expectedStatus": 200,
+    "timeoutMs": 8000,
+    "critical": true
+  },
+  {
+    "name": "paperless",
+    "url": "https://paperless.sherlockhomeless.net",
+    "expectedStatus": 200,
+    "timeoutMs": 8000,
+    "critical": true
+  },
+  {
+    "name": "n8n",
+    "url": "https://n8n.sherlockhomeless.net/healthz",
+    "expectedStatus": 200,
+    "timeoutMs": 8000,
+    "critical": true
+  }
+]
@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+CONTAINER="${OPENCLAW_GATEWAY_CONTAINER:-openclaw-openclaw-gateway-1}"
+PROVIDER="${OPENCLAW_OAUTH_PROVIDER:-openai-codex}"
+MODE="login"
+
+usage() {
+  cat <<'EOF'
+Usage:
+  codex-reauth [--status] [--container NAME] [--provider ID]
+
+Options:
+  --status             Show current auth status only (no login flow).
+  --container NAME     Gateway container name (default: openclaw-openclaw-gateway-1).
+  --provider ID        OAuth provider id (default: openai-codex).
+  -h, --help           Show this help.
+EOF
+}
+
+require_container_running() {
+  if ! docker ps --format '{{.Names}}' | grep -qx "$CONTAINER"; then
+    echo "ERROR: container '$CONTAINER' is not running." >&2
+    echo "Hint: check with 'docker ps | grep openclaw'." >&2
+    exit 1
+  fi
+}
+
+show_status() {
+  docker exec "$CONTAINER" openclaw models status --status-plain \
+    | sed -n '/Auth overview/,$p'
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --status)
+      MODE="status"
+      shift
+      ;;
+    --container)
+      [[ $# -lt 2 ]] && { echo "ERROR: --container requires a value." >&2; exit 2; }
+      CONTAINER="$2"
+      shift 2
+      ;;
+    --provider)
+      [[ $# -lt 2 ]] && { echo "ERROR: --provider requires a value." >&2; exit 2; }
+      PROVIDER="$2"
+      shift 2
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument '$1'." >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if ! command -v docker >/dev/null 2>&1; then
+  echo "ERROR: docker command not found." >&2
+  exit 1
+fi
+
+require_container_running
+
+if [[ "$MODE" == "status" ]]; then
+  show_status
+  exit 0
+fi
+
+if [[ ! -t 0 || ! -t 1 ]]; then
+  echo "ERROR: interactive terminal required for OAuth login." >&2
+  echo "Run this command from an interactive shell." >&2
+  exit 1
+fi
+
+echo "Starting OAuth re-auth for provider '$PROVIDER' on container '$CONTAINER'..."
+echo "Open the URL in your local browser and paste the callback URL when prompted."
+docker exec -it "$CONTAINER" openclaw models auth login --provider "$PROVIDER"
+
+echo
+echo "Re-auth completed. Current provider status:"
+show_status
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MAX_TIME="${HEALTH_MAX_TIME:-12}"
+FOLLOW_REDIRECTS="${HEALTH_FOLLOW_REDIRECTS:-1}"
+
+load_endpoints() {
+  if [[ -n "${HEALTH_ENDPOINTS:-}" ]]; then
+    printf '%s\n' "$HEALTH_ENDPOINTS" | tr ',;' '\n' | sed '/^[[:space:]]*$/d' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
+    return
+  fi
+
+  cat <<'EOF'
+https://traefik.thehomelesssherlock.com
+https://auth.thehomelesssherlock.com
+https://n8n.sherlockhomeless.net
+https://nextcloud.sherlockhomeless.net
+https://paperless.sherlockhomeless.net
+https://onlyoffice.sherlockhomeless.net
+https://memos.sherlockhomeless.net
+https://vikunja.sherlockhomeless.net
+https://karakeep.sherlockhomeless.net
+https://dozzle.sherlockhomeless.net
+https://beszel.sherlockhomeless.net
+EOF
+}
+
+if ! command -v curl >/dev/null 2>&1; then
+  jq -n \
+    --arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
+    --arg error "curl command not available" \
+    '{generated_at_utc:$generated_at_utc, overall_status:"down", endpoints_count:0, up_count:0, down_count:0, error:$error, results:[]}'
+  exit 0
+fi
+
+if ! command -v jq >/dev/null 2>&1; then
+  echo '{"generated_at_utc":"'"$(date -u +'%Y-%m-%dT%H:%M:%SZ')"'", "overall_status":"down", "error":"jq command not available", "results":[]}'
+  exit 0
+fi
+
+mapfile -t ENDPOINTS < <(load_endpoints)
+
+results_tmp="$(mktemp)"
+trap 'rm -f "$results_tmp"' EXIT
+
+for endpoint in "${ENDPOINTS[@]}"; do
+  [[ -z "$endpoint" ]] && continue
+  start_ms="$(date +%s%3N)"
+  curl_args=(-sS --max-time "$MAX_TIME" -o /dev/null -w '%{http_code} %{time_total} %{url_effective}')
+  if [[ "$FOLLOW_REDIRECTS" == "1" ]]; then
+    curl_args+=(-L)
+  fi
+
+  set +e
+  curl_out="$(curl "${curl_args[@]}" "$endpoint" 2>&1)"
+  curl_rc=$?
+  set -e
+  end_ms="$(date +%s%3N)"
+  latency_ms="$(( end_ms - start_ms ))"
+
+  if (( curl_rc == 0 )); then
+    http_code="$(awk '{print $1}' <<<"$curl_out")"
+    effective_url="$(awk '{print $3}' <<<"$curl_out")"
+    status="down"
+    if [[ "$http_code" =~ ^[0-9]+$ ]] && (( http_code >= 200 && http_code < 400 )); then
+      status="up"
+    fi
+    jq -cn \
+      --arg endpoint "$endpoint" \
+      --arg status "$status" \
+      --arg effective_url "$effective_url" \
+      --argjson http_code "${http_code:-0}" \
+      --argjson latency_ms "$latency_ms" \
+      '{endpoint:$endpoint,status:$status,http_code:$http_code,latency_ms:$latency_ms,effective_url:$effective_url}' >>"$results_tmp"
+  else
+    err_line="$(printf '%s\n' "$curl_out" | tail -n 1)"
+    jq -cn \
+      --arg endpoint "$endpoint" \
+      --arg error "$err_line" \
+      --argjson latency_ms "$latency_ms" \
+      '{endpoint:$endpoint,status:"down",http_code:0,latency_ms:$latency_ms,error:$error}' >>"$results_tmp"
+  fi
+done
+
+results_json="$(jq -s '.' "$results_tmp")"
+up_count="$(jq '[.[] | select(.status=="up")] | length' <<<"$results_json")"
+down_count="$(jq '[.[] | select(.status=="down")] | length' <<<"$results_json")"
+endpoints_count="$(( up_count + down_count ))"
+
+overall_status="up"
+if (( endpoints_count == 0 )); then
+  overall_status="down"
+elif (( down_count > 0 )); then
+  overall_status="degraded"
+fi
+
+jq -n \
+  --arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
+  --arg overall_status "$overall_status" \
+  --argjson endpoints_count "$endpoints_count" \
+  --argjson up_count "$up_count" \
+  --argjson down_count "$down_count" \
+  --argjson max_time_seconds "$MAX_TIME" \
+  --argjson results "$results_json" \
+  '{
+    generated_at_utc:$generated_at_utc,
+    overall_status:$overall_status,
+    endpoints_count:$endpoints_count,
+    up_count:$up_count,
+    down_count:$down_count,
+    max_time_seconds:$max_time_seconds,
+    results:$results
+  }'
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SINCE="${SINCE:-24 hours ago}"
+MAX_JOURNAL_LINES="${MAX_JOURNAL_LINES:-120}"
+MAX_DOCKER_LINES_PER_SERVICE="${MAX_DOCKER_LINES_PER_SERVICE:-40}"
+TARGET_CONTAINERS="${TARGET_CONTAINERS:-traefik n8n nextcloud paperless mail-relay openclaw-openclaw-gateway-1}"
+
+resolve_docker_since() {
+  if [[ "$SINCE" =~ ^[0-9]+[smhd]$ ]]; then
+    echo "$SINCE"
+    return
+  fi
+
+  if date -d "$SINCE" +%s >/dev/null 2>&1; then
+    date -d "$SINCE" --iso-8601=seconds
+    return
+  fi
+
+  echo "24h"
+}
+
+DOCKER_SINCE="$(resolve_docker_since)"
+
+section() {
+  printf '\n## %s\n' "$1"
+}
+
+printf '# Log Digest\n'
+printf 'generated_at_utc: %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
+printf 'window: %s\n' "$SINCE"
+printf 'docker_since: %s\n' "$DOCKER_SINCE"
+
+section "System Errors (journalctl err..alert)"
+if command -v journalctl >/dev/null 2>&1; then
+  journalctl --since "$SINCE" -p err..alert --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true
+else
+  echo 'journalctl not available'
+fi
+
+section "System Warnings (journalctl warning)"
+if command -v journalctl >/dev/null 2>&1; then
+  journalctl --since "$SINCE" -p warning --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true
+else
+  echo 'journalctl not available'
+fi
+
+section "Docker Service Error Highlights"
+if command -v docker >/dev/null 2>&1; then
+  for svc in $TARGET_CONTAINERS; do
+    if docker ps -a --format '{{.Names}}' | grep -qx "$svc"; then
+      echo
+      echo "### $svc"
+      docker logs --since "$DOCKER_SINCE" "$svc" 2>&1 \
+        | grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|refused|unhealthy|failed|oauth|401' \
+        | tail -n "$MAX_DOCKER_LINES_PER_SERVICE" || true
+    fi
+  done
+else
+  echo 'docker command not available'
+fi
+
+section "Top Error Patterns"
+if command -v journalctl >/dev/null 2>&1; then
+  journalctl --since "$SINCE" --no-pager -o cat 2>/dev/null \
+    | grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|failed' \
+    | sed 's/[[:space:]]\+/ /g' \
+    | cut -c1-180 \
+    | sort \
+    | uniq -c \
+    | sort -nr \
+    | head -n 20 || true
+else
+  echo 'journalctl not available'
+fi
@@ -0,0 +1,332 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SINCE_HOURS="${SINCE_HOURS:-24}"
+TOP_ERROR_LINES="${TOP_ERROR_LINES:-20}"
+CERT_WARN_DAYS="${CERT_WARN_DAYS:-30}"
+CERT_EXCLUDE_DOMAINS="${CERT_EXCLUDE_DOMAINS:-portainer.thehomelesssherlock.com uptimekuma.sherlockhomeless.net notas.sherlockhomeless.net}"
+ACTIVE_CERT_DOMAINS_ONLY="${ACTIVE_CERT_DOMAINS_ONLY:-1}"
+OPENCLAW_AUTH_FILE="${OPENCLAW_AUTH_FILE:-/home/felidae/.openclaw/agents/main/agent/auth-profiles.json}"
+OPENCLAW_WARN_HOURS="${OPENCLAW_WARN_HOURS:-72}"
+OPENCLAW_CRITICAL_HOURS="${OPENCLAW_CRITICAL_HOURS:-24}"
+OPENCLAW_REAUTH_CMD="${OPENCLAW_REAUTH_CMD:-codex-reauth}"
+
+section() {
+  printf '\n## %s\n' "$1"
+}
+
+report_cert_expiry() {
+  local domain="$1"
+  local end_raw="$2"
+  if [[ -z "$end_raw" ]]; then
+    printf '%s: unable to parse cert\n' "$domain"
+    return
+  fi
+
+  local end_epoch
+  local now_epoch
+  local days_left
+  local warn
+
+  end_epoch="$(date -d "$end_raw" +%s 2>/dev/null || true)"
+  now_epoch="$(date +%s)"
+  if [[ -z "$end_epoch" ]]; then
+    printf '%s: enddate=%s\n' "$domain" "$end_raw"
+    return
+  fi
+
+  days_left="$(( (end_epoch - now_epoch) / 86400 ))"
+  warn=''
+  if (( days_left <= CERT_WARN_DAYS )); then
+    warn=' [WARN]'
+  fi
+  printf '%s: expires=%s days_left=%s%s\n' "$domain" "$end_raw" "$days_left" "$warn"
+}
+
+report_pem_cert() {
+  local domain="$1"
+  local pem="$2"
+  local end_raw
+
+  end_raw="$(openssl x509 -enddate -noout 2>/dev/null <<<"$pem" | cut -d= -f2- || true)"
+  report_cert_expiry "$domain" "$end_raw"
+}
+
+emit_traefik_acme_entries() {
+  python3 - "$@" <<'PY'
+import json
+import pathlib
+import sys
+
+for raw_path in sys.argv[1:]:
+    path = pathlib.Path(raw_path)
+    if not path.exists():
+        continue
+    try:
+        data = json.loads(path.read_text())
+    except Exception:
+        continue
+    for payload in data.values():
+        if not isinstance(payload, dict):
+            continue
+        for cert in payload.get("Certificates", []):
+            domain = (cert.get("domain") or {}).get("main")
+            encoded = cert.get("certificate")
+            if domain and encoded:
+                print(f"{domain}\t{encoded}")
+PY
+}
+
+emit_active_tls_domains() {
+  python3 - <<'PY'
+import json
+import pathlib
+import re
+import subprocess
+
+host_rule_re = re.compile(r"Host\(([^)]*)\)")
+quoted_re = re.compile(r"`([^`]+)`|'([^']+)'|\"([^\"]+)\"")
+domains = set()
+
+
+def add_rule(rule: str) -> None:
+    if not isinstance(rule, str):
+        return
+    for host_match in host_rule_re.finditer(rule):
+        body = host_match.group(1)
+        for quoted_match in quoted_re.finditer(body):
+            domain = next((item for item in quoted_match.groups() if item), "")
+            domain = domain.strip().lower()
+            if not domain or "*" in domain or "{" in domain or "}" in domain:
+                continue
+            domains.add(domain)
+
+
+try:
+    container_names = subprocess.check_output(
+        ["docker", "ps", "--format", "{{.Names}}"],
+        text=True,
+        stderr=subprocess.DEVNULL,
+    ).splitlines()
+except Exception:
+    container_names = []
+
+for container in container_names:
+    container = container.strip()
+    if not container:
+        continue
+    try:
+        raw_labels = subprocess.check_output(
+            ["docker", "inspect", "--format", "{{json .Config.Labels}}", container],
+            text=True,
+            stderr=subprocess.DEVNULL,
+        ).strip()
+        labels = json.loads(raw_labels) if raw_labels else {}
+    except Exception:
+        continue
+    if not isinstance(labels, dict):
+        continue
+    for key, value in labels.items():
+        if key.startswith("traefik.http.routers.") and key.endswith(".rule"):
+            add_rule(value)
+
+for base_path in (pathlib.Path("/opt/traefik/dynamic"), pathlib.Path("/etc/traefik/dynamic")):
+    if not base_path.exists():
+        continue
+    for cfg_file in base_path.rglob("*"):
+        if not cfg_file.is_file():
+            continue
+        try:
+            content = cfg_file.read_text(encoding="utf-8", errors="ignore")
+        except Exception:
+            continue
+        add_rule(content)
+
+for domain in sorted(domains):
+    print(domain)
+PY
+}
+
+now_utc="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
+hostname_fqdn="$(hostname -f 2>/dev/null || hostname)"
+kernel="$(uname -r)"
+
+printf '# Server Daily Summary\n'
+printf 'generated_at_utc: %s\n' "$now_utc"
+printf 'host: %s\n' "$hostname_fqdn"
+printf 'kernel: %s\n' "$kernel"
+
+section "Uptime & Load"
+printf 'uptime: %s\n' "$(uptime -p 2>/dev/null || uptime)"
+printf 'loadavg: %s\n' "$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo 'n/a')"
+
+section "Memory"
+if command -v free >/dev/null 2>&1; then
+  free -h
+else
+  echo 'free command not available'
+fi
+
+section "Disk"
+if command -v df >/dev/null 2>&1; then
+  df -hT -x tmpfs -x devtmpfs
+else
+  echo 'df command not available'
+fi
+
+section "Docker Overview"
+if command -v docker >/dev/null 2>&1; then
+  running_count="$(docker ps -q | wc -l | tr -d ' ')"
+  unhealthy_count="$(docker ps --filter health=unhealthy -q | wc -l | tr -d ' ')"
+  exited_count="$(docker ps -a --filter status=exited -q | wc -l | tr -d ' ')"
+  restarting_count="$(docker ps -a --filter status=restarting -q | wc -l | tr -d ' ')"
+
+  printf 'containers_running: %s\n' "$running_count"
+  printf 'containers_unhealthy: %s\n' "$unhealthy_count"
+  printf 'containers_exited: %s\n' "$exited_count"
+  printf 'containers_restarting: %s\n' "$restarting_count"
+
+  echo
+  echo 'running_containers:'
+  docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'
+
+  if [[ "$unhealthy_count" != "0" ]]; then
+    echo
+    echo 'unhealthy_containers:'
+    docker ps --filter health=unhealthy --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
+  fi
+
+  if [[ "$exited_count" != "0" || "$restarting_count" != "0" ]]; then
+    echo
+    echo 'non_running_containers:'
+    docker ps -a --filter status=exited --filter status=restarting --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
+  fi
+else
+  echo 'docker command not available'
+fi
+
+section "TLS Certificates"
+if command -v openssl >/dev/null 2>&1; then
+  found=0
+  declare -A seen_domains=()
+  declare -A excluded_domains=()
+  declare -A active_domains=()
+
+  for domain in $CERT_EXCLUDE_DOMAINS; do
+    excluded_domains["${domain,,}"]=1
+  done
+
+  if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then
+    while IFS= read -r domain; do
+      [[ -z "$domain" ]] && continue
+      active_domains["${domain,,}"]=1
+    done < <(emit_active_tls_domains)
+  fi
+
+  if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then
+    if [[ "${#active_domains[@]}" -gt 0 ]]; then
+      printf 'tls_scope: active_domains_only active_domains=%s excluded_domains=%s\n' "${#active_domains[@]}" "${#excluded_domains[@]}"
+    else
+      printf 'tls_scope: active_domains_only requested but no active domains detected; fallback=all_discovered excluded_domains=%s\n' "${#excluded_domains[@]}"
+    fi
+  fi
+
+  if [[ -d /etc/letsencrypt/live ]]; then
+    while IFS= read -r cert; do
+      domain="$(basename "$(dirname "$cert")")"
+      domain_lc="${domain,,}"
+      [[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue
+      if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then
+        continue
+      fi
+      [[ -n "${seen_domains[$domain]:-}" ]] && continue
+      seen_domains["$domain"]=1
+      found=1
+      end_raw="$(openssl x509 -enddate -noout -in "$cert" 2>/dev/null | cut -d= -f2- || true)"
+      report_cert_expiry "$domain" "$end_raw"
+    done < <(find /etc/letsencrypt/live -maxdepth 2 -type f -name cert.pem 2>/dev/null | sort)
+  fi
+
+  while IFS=$'\t' read -r domain cert_b64; do
+    [[ -z "$domain" || -z "$cert_b64" ]] && continue
+    domain_lc="${domain,,}"
+    [[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue
+    if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then
+      continue
+    fi
+    [[ -n "${seen_domains[$domain]:-}" ]] && continue
+    seen_domains["$domain"]=1
+    found=1
+    pem="$(printf '%s' "$cert_b64" | base64 -d 2>/dev/null || true)"
+    if [[ -z "$pem" ]]; then
+      printf '%s: unable to decode traefik cert\n' "$domain"
+      continue
+    fi
+    report_pem_cert "$domain" "$pem"
+  done < <(emit_traefik_acme_entries /opt/traefik/letsencrypt/acme.json /opt/traefik/acme/acme.json)
+
+  if [[ "$found" -eq 0 ]]; then
+    echo 'no local TLS certificates found'
+  fi
+else
+  echo 'openssl not available'
+fi
+
+section "SSH Protection"
+if command -v fail2ban-client >/dev/null 2>&1; then
+  sshd_status="$(fail2ban-client status sshd 2>/dev/null || true)"
+  if [[ -n "$sshd_status" ]]; then
+    total_failed="$(grep -Eo 'Total failed:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
+    current_banned="$(grep -Eo 'Currently banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
+    total_banned="$(grep -Eo 'Total banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
+    printf 'fail2ban_sshd: active total_failed=%s current_banned=%s total_banned=%s\n' \
+      "$total_failed" "$current_banned" "$total_banned"
+    echo 'note: SSH probes can still generate auth/journal log entries before or while bans are applied; that alone does not mean fail2ban is inactive.'
+  else
+    echo 'fail2ban_sshd: no status available'
+  fi
+else
+  echo 'fail2ban-client not available'
+fi
+
+section "OpenClaw OAuth"
+if [[ -f "$OPENCLAW_AUTH_FILE" ]] && command -v jq >/dev/null 2>&1; then
+  openclaw_expiry_ms="$(jq -r '
+    [.profiles | to_entries[]
+      | select(.value.provider == "openai-codex")
+      | (.value.expires // 0)]
+    | max // 0
+  ' "$OPENCLAW_AUTH_FILE" 2>/dev/null || echo "0")"
+
+  if [[ "$openclaw_expiry_ms" =~ ^[0-9]+$ ]] && (( openclaw_expiry_ms > 0 )); then
+    now_ms="$(( $(date +%s) * 1000 ))"
+    diff_ms="$(( openclaw_expiry_ms - now_ms ))"
+    expiry_iso="$(date -u -d "@$((openclaw_expiry_ms / 1000))" +'%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo 'n/a')"
+    if (( diff_ms <= 0 )); then
+      printf 'openclaw_codex_oauth: CRITICAL expires_utc=%s reason=expired action="%s"\n' \
+        "$expiry_iso" "$OPENCLAW_REAUTH_CMD"
+    else
+      hours_left="$(( diff_ms / 3600000 ))"
+      mins_left="$(( (diff_ms % 3600000) / 60000 ))"
+      severity='OK'
+      if (( hours_left <= OPENCLAW_CRITICAL_HOURS )); then
+        severity='CRITICAL'
+      elif (( hours_left <= OPENCLAW_WARN_HOURS )); then
+        severity='WARN'
+      fi
+      printf 'openclaw_codex_oauth: %s expires_utc=%s hours_left=%sh%sm critical_hours=%s warn_hours=%s action="%s"\n' \
+        "$severity" "$expiry_iso" "$hours_left" "$mins_left" "$OPENCLAW_CRITICAL_HOURS" "$OPENCLAW_WARN_HOURS" "$OPENCLAW_REAUTH_CMD"
+    fi
+  else
+    echo 'openclaw_codex_oauth: no oauth profile expiry found'
+  fi
+else
+  echo 'openclaw_codex_oauth: auth file or jq unavailable'
+fi
+
+section "Top Journal Errors (last ${SINCE_HOURS}h)"
+if command -v journalctl >/dev/null 2>&1; then
+  journalctl -p err..alert --since "${SINCE_HOURS} hours ago" --no-pager -o short-iso 2>/dev/null | tail -n "$TOP_ERROR_LINES" || true
+else
+  echo 'journalctl not available'
+fi
@@ -0,0 +1,46 @@
+Eres un SRE senior. Genera un informe diario de estado con tono ejecutivo y accionable.
+
+## Entrada
+Recibirás:
+- `server_summary`: salida de `server-summary.sh`
+- `log_digest`: salida de `log-digest.sh`
+- `health_results`: JSON con estado de endpoints (up/down, latencia, reintentos). Puede no venir.
+
+## Objetivo
+Entregar un resumen breve para operación diaria sin ruido.
+
+## Formato obligatorio
+1. Estado general (OK / DEGRADED / CRITICAL) + motivo en 1 frase.
+2. Incidentes críticos (si no hay, escribir "Sin incidentes críticos").
+3. Servicios degradados con impacto.
+4. Riesgos preventivos (certificados próximos a caducar, disco alto, reinicios).
+5. Acciones recomendadas (máximo 5, ordenadas por prioridad).
+
+## Reglas
+- No inventes datos.
+- Si faltan datos, dilo explícitamente.
+- No incluyas secretos ni credenciales.
+- Máximo 250 palabras.
+- El estado general se decide por **impacto activo**, no por riesgos preventivos.
+- `health_results` es opcional.
+
+## Criterio estricto para "Estado general"
+- `CRITICAL` solo si hay impacto activo severo:
+  - caídas confirmadas (`health_results` down) en servicios críticos,
+  - contenedores `unhealthy`/reiniciando con impacto,
+  - autenticación central caída que bloquea acceso,
+  - certificados **ya expirados** en dominios activos.
+- `DEGRADED` solo si hay degradación activa con impacto parcial:
+  - errores repetidos actuales con evidencia de fallo funcional,
+  - latencia/errores sostenidos que afectan servicio, sin caída total.
+- `OK` cuando no hay impacto activo, aunque existan riesgos preventivos.
+
+## Importante
+- **No marcar `DEGRADED` ni `CRITICAL` únicamente por prevención**:
+  - certificados que aún no expiraron,
+  - OAuth/token con horas o días restantes (incluye `WARN`),
+  - actividad SSH bloqueada por fail2ban,
+  - swap moderado si el sistema está estable.
+- Esos puntos deben ir en "Riesgos preventivos" y "Acciones recomendadas".
+- Si `server_summary` y/o `log_digest` tienen contenido, **no** responder "No evaluable" global.
+- Si falta `health_results`, indícalo como limitación parcial de endpoints, pero sigue evaluando con `server_summary` y `log_digest`.
@@ -0,0 +1,55 @@
+# Workflow Design (n8n)
+
+## 1) Daily Server Report (08:00)
+
+Nodes:
+1. `Schedule Trigger` (daily, 08:00, timezone Europe/Madrid)
+2. `SSH` -> `cd /opt/portainer-repo && ./n8n-observability/scripts/server-summary.sh`
+3. `SSH` -> `cd /opt/portainer-repo && SINCE="24 hours ago" ./n8n-observability/scripts/log-digest.sh`
+4. `HTTP Request` (opcional) para cargar estado de endpoints desde tu monitor
+5. `AI` (OpenAI Chat Model) con prompt `templates/ai-daily-report-prompt.md`
+6. `Send Email` (via mail-relay) o `Telegram`
+
+Recomendacion:
+- Guarda salida de nodos 2 y 3 como texto en campos `server_summary` y `log_digest`.
+- Limita input a IA si supera ~15k caracteres.
+
+## 2) Real Outage Alert (cada 2 min)
+
+Nodes:
+1. `Schedule Trigger` cada 2 min
+2. `Code` (carga array de servicios desde `services.example.json` o Set node)
+3. `Split In Batches` o `Loop Over Items`
+4. `HTTP Request` por servicio (timeout por item)
+5. `IF` status != expectedStatus o timeout
+6. `Data Store` (contador de fallos consecutivos por servicio)
+7. `IF` contador >= 3 -> alerta (Email/Telegram)
+8. `IF` servicio vuelve OK -> enviar recovery y reset contador
+
+Regla anti-ruido:
+- No alertar en primer fallo.
+- Alerta solo en fallo consecutivo >= 3.
+
+## 3) Daily Log/Error Digest (09:00)
+
+Nodes:
+1. `Schedule Trigger` diario 09:00
+2. `SSH` -> `cd /opt/portainer-repo && SINCE="24 hours ago" ./n8n-observability/scripts/log-digest.sh`
+3. `AI` (opcional) para clasificar severidad y acciones
+4. `Send Email`
+
+## Payload sugerido para nodo AI
+
+```json
+{
+  "server_summary": "{{$json.server_summary}}",
+  "log_digest": "{{$json.log_digest}}",
+  "health_results": "{{$json.health_results}}"
+}
+```
+
+## Variables utiles
+
+- `SINCE_HOURS=24` para `server-summary.sh`
+- `SINCE="24 hours ago"` para `log-digest.sh`
+- `TARGET_CONTAINERS="traefik n8n nextcloud paperless mail-relay"`