diff --git a/n8n-observability/README.md b/n8n-observability/README.md index 1338be4..6563ad6 100644 --- a/n8n-observability/README.md +++ b/n8n-observability/README.md @@ -8,9 +8,10 @@ Carpeta dedicada para automatizar en n8n: ## Estructura -- `scripts/server-summary.sh`: estado de host + docker + certs + errores top -- `scripts/log-digest.sh`: digest de logs (journal + docker) para alertas/informes -- `scripts/health-results.sh`: chequeo HTTP de endpoints y salida JSON (`up/down`, latencia, codigo) +- `scripts/ops-report`: motor Python unificado para `summary`, `logs`, `health` y `all` +- `scripts/server-summary.sh`: wrapper compatible para estado de host + docker + certs + errores top +- `scripts/log-digest.sh`: wrapper compatible para digest de logs (journal + docker) para alertas/informes +- `scripts/health-results.sh`: wrapper compatible para chequeo HTTP de endpoints y salida JSON (`up/down`, latencia, codigo) - `templates/ai-daily-report-prompt.md`: prompt para convertir datos tecnicos en informe legible - `config/services.example.json`: lista base de endpoints para health checks en n8n - `workflows/workflow-design.md`: diseno de 3 workflows recomendados @@ -18,10 +19,11 @@ Carpeta dedicada para automatizar en n8n: ## Uso rapido (en el host) ```bash -chmod +x n8n-observability/scripts/*.sh +chmod +x n8n-observability/scripts/*.sh n8n-observability/scripts/ops-report ./n8n-observability/scripts/server-summary.sh SINCE="24 hours ago" ./n8n-observability/scripts/log-digest.sh ./n8n-observability/scripts/health-results.sh | jq +./n8n-observability/scripts/ops-report all ``` ## Integracion recomendada con n8n diff --git a/n8n-observability/scripts/health-results.sh b/n8n-observability/scripts/health-results.sh index c70475f..326176c 100755 --- a/n8n-observability/scripts/health-results.sh +++ b/n8n-observability/scripts/health-results.sh @@ -1,113 +1,6 @@ #!/usr/bin/env bash set -euo pipefail -MAX_TIME="${HEALTH_MAX_TIME:-12}" -FOLLOW_REDIRECTS="${HEALTH_FOLLOW_REDIRECTS:-1}" - -load_endpoints() { - if [[ -n "${HEALTH_ENDPOINTS:-}" ]]; then - printf '%s\n' "$HEALTH_ENDPOINTS" | tr ',;' '\n' | sed '/^[[:space:]]*$/d' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' - return - fi - - cat <<'EOF' -https://traefik.thehomelesssherlock.com -https://auth.thehomelesssherlock.com -https://n8n.sherlockhomeless.net -https://nextcloud.sherlockhomeless.net -https://paperless.sherlockhomeless.net -https://onlyoffice.sherlockhomeless.net -https://memos.sherlockhomeless.net -https://vikunja.sherlockhomeless.net -https://karakeep.sherlockhomeless.net -https://dozzle.sherlockhomeless.net -https://beszel.sherlockhomeless.net -EOF -} - -if ! command -v curl >/dev/null 2>&1; then - jq -n \ - --arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ - --arg error "curl command not available" \ - '{generated_at_utc:$generated_at_utc, overall_status:"down", endpoints_count:0, up_count:0, down_count:0, error:$error, results:[]}' - exit 0 -fi - -if ! command -v jq >/dev/null 2>&1; then - echo '{"generated_at_utc":"'"$(date -u +'%Y-%m-%dT%H:%M:%SZ')"'", "overall_status":"down", "error":"jq command not available", "results":[]}' - exit 0 -fi - -mapfile -t ENDPOINTS < <(load_endpoints) - -results_tmp="$(mktemp)" -trap 'rm -f "$results_tmp"' EXIT - -for endpoint in "${ENDPOINTS[@]}"; do - [[ -z "$endpoint" ]] && continue - start_ms="$(date +%s%3N)" - curl_args=(-sS --max-time "$MAX_TIME" -o /dev/null -w '%{http_code} %{time_total} %{url_effective}') - if [[ "$FOLLOW_REDIRECTS" == "1" ]]; then - curl_args+=(-L) - fi - - set +e - curl_out="$(curl "${curl_args[@]}" "$endpoint" 2>&1)" - curl_rc=$? - set -e - end_ms="$(date +%s%3N)" - latency_ms="$(( end_ms - start_ms ))" - - if (( curl_rc == 0 )); then - http_code="$(awk '{print $1}' <<<"$curl_out")" - effective_url="$(awk '{print $3}' <<<"$curl_out")" - status="down" - if [[ "$http_code" =~ ^[0-9]+$ ]] && (( http_code >= 200 && http_code < 400 )); then - status="up" - fi - jq -cn \ - --arg endpoint "$endpoint" \ - --arg status "$status" \ - --arg effective_url "$effective_url" \ - --argjson http_code "${http_code:-0}" \ - --argjson latency_ms "$latency_ms" \ - '{endpoint:$endpoint,status:$status,http_code:$http_code,latency_ms:$latency_ms,effective_url:$effective_url}' >>"$results_tmp" - else - err_line="$(printf '%s\n' "$curl_out" | tail -n 1)" - jq -cn \ - --arg endpoint "$endpoint" \ - --arg error "$err_line" \ - --argjson latency_ms "$latency_ms" \ - '{endpoint:$endpoint,status:"down",http_code:0,latency_ms:$latency_ms,error:$error}' >>"$results_tmp" - fi -done - -results_json="$(jq -s '.' "$results_tmp")" -up_count="$(jq '[.[] | select(.status=="up")] | length' <<<"$results_json")" -down_count="$(jq '[.[] | select(.status=="down")] | length' <<<"$results_json")" -endpoints_count="$(( up_count + down_count ))" - -overall_status="up" -if (( endpoints_count == 0 )); then - overall_status="down" -elif (( down_count > 0 )); then - overall_status="degraded" -fi - -jq -n \ - --arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \ - --arg overall_status "$overall_status" \ - --argjson endpoints_count "$endpoints_count" \ - --argjson up_count "$up_count" \ - --argjson down_count "$down_count" \ - --argjson max_time_seconds "$MAX_TIME" \ - --argjson results "$results_json" \ - '{ - generated_at_utc:$generated_at_utc, - overall_status:$overall_status, - endpoints_count:$endpoints_count, - up_count:$up_count, - down_count:$down_count, - max_time_seconds:$max_time_seconds, - results:$results - }' +SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")" +SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)" +exec "$SCRIPT_DIR/ops-report" health "$@" diff --git a/n8n-observability/scripts/log-digest.sh b/n8n-observability/scripts/log-digest.sh index 8958936..5dc9368 100755 --- a/n8n-observability/scripts/log-digest.sh +++ b/n8n-observability/scripts/log-digest.sh @@ -1,75 +1,6 @@ #!/usr/bin/env bash set -euo pipefail -SINCE="${SINCE:-24 hours ago}" -MAX_JOURNAL_LINES="${MAX_JOURNAL_LINES:-120}" -MAX_DOCKER_LINES_PER_SERVICE="${MAX_DOCKER_LINES_PER_SERVICE:-40}" -TARGET_CONTAINERS="${TARGET_CONTAINERS:-traefik n8n nextcloud paperless mail-relay openclaw-openclaw-gateway-1}" - -resolve_docker_since() { - if [[ "$SINCE" =~ ^[0-9]+[smhd]$ ]]; then - echo "$SINCE" - return - fi - - if date -d "$SINCE" +%s >/dev/null 2>&1; then - date -d "$SINCE" --iso-8601=seconds - return - fi - - echo "24h" -} - -DOCKER_SINCE="$(resolve_docker_since)" - -section() { - printf '\n## %s\n' "$1" -} - -printf '# Log Digest\n' -printf 'generated_at_utc: %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" -printf 'window: %s\n' "$SINCE" -printf 'docker_since: %s\n' "$DOCKER_SINCE" - -section "System Errors (journalctl err..alert)" -if command -v journalctl >/dev/null 2>&1; then - journalctl --since "$SINCE" -p err..alert --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true -else - echo 'journalctl not available' -fi - -section "System Warnings (journalctl warning)" -if command -v journalctl >/dev/null 2>&1; then - journalctl --since "$SINCE" -p warning --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true -else - echo 'journalctl not available' -fi - -section "Docker Service Error Highlights" -if command -v docker >/dev/null 2>&1; then - for svc in $TARGET_CONTAINERS; do - if docker ps -a --format '{{.Names}}' | grep -qx "$svc"; then - echo - echo "### $svc" - docker logs --since "$DOCKER_SINCE" "$svc" 2>&1 \ - | grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|refused|unhealthy|failed|oauth|401' \ - | tail -n "$MAX_DOCKER_LINES_PER_SERVICE" || true - fi - done -else - echo 'docker command not available' -fi - -section "Top Error Patterns" -if command -v journalctl >/dev/null 2>&1; then - journalctl --since "$SINCE" --no-pager -o cat 2>/dev/null \ - | grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|failed' \ - | sed 's/[[:space:]]\+/ /g' \ - | cut -c1-180 \ - | sort \ - | uniq -c \ - | sort -nr \ - | head -n 20 || true -else - echo 'journalctl not available' -fi +SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")" +SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)" +exec "$SCRIPT_DIR/ops-report" logs "$@" diff --git a/n8n-observability/scripts/ops-report b/n8n-observability/scripts/ops-report new file mode 100755 index 0000000..b05360f --- /dev/null +++ b/n8n-observability/scripts/ops-report @@ -0,0 +1,686 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import datetime as dt +import http.client +import json +import os +import pathlib +import re +import ssl +import subprocess +import sys +import time +import urllib.parse +from collections import Counter + + +DEFAULT_ENDPOINTS = [ + "https://traefik.sherlockhomeless.net", + "https://auth.sherlockhomeless.net", + "https://n8n.sherlockhomeless.net/healthz", + "https://nextcloud.sherlockhomeless.net", + "https://paperless.sherlockhomeless.net", + "https://onlyoffice.sherlockhomeless.net", + "https://memos.sherlockhomeless.net", + "https://vikunja.sherlockhomeless.net", + "https://karakeep.sherlockhomeless.net", + "https://ots.sherlockhomeless.net", + "https://dozzle.sherlockhomeless.net", + "https://beszel.sherlockhomeless.net", +] + +DEFAULT_TARGET_CONTAINERS = [ + "traefik", + "n8n", + "nextcloud", + "paperless", + "mail-relay", + "openclaw-openclaw-gateway-1", +] + +ERROR_PATTERN = re.compile( + r"error|exception|fatal|panic|critical|denied|timeout|refused|unhealthy|failed|oauth|401", + re.IGNORECASE, +) +HOST_RULE_RE = re.compile(r"Host\(([^)]*)\)") +QUOTED_RE = re.compile(r"`([^`]+)`|'([^']+)'|\"([^\"]+)\"") +SABLIER_RE = re.compile(r"sablier-([A-Za-z0-9_.-]+)@file") + + +def env_int(name: str, default: int, minimum: int = 0) -> int: + raw = os.environ.get(name) + try: + value = int(raw) if raw is not None else default + except ValueError: + return default + return value if value >= minimum else default + + +def env_bool(name: str, default: bool) -> bool: + raw = os.environ.get(name) + if raw is None: + return default + return raw.strip().lower() in {"1", "true", "yes", "on"} + + +def now_utc() -> str: + return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def section(title: str) -> None: + print(f"\n## {title}") + + +def run(cmd, *, timeout=30, check=False, input_text=None): + try: + return subprocess.run( + cmd, + text=True, + input=input_text, + capture_output=True, + timeout=timeout, + check=check, + ) + except Exception as exc: + completed = subprocess.CompletedProcess(cmd, 1, "", str(exc)) + return completed + + +def command_exists(name: str) -> bool: + return run(["sh", "-lc", f"command -v {name} >/dev/null 2>&1"]).returncode == 0 + + +def print_command(cmd, *, timeout=30) -> None: + proc = run(cmd, timeout=timeout) + out = proc.stdout.rstrip("\n") + err = proc.stderr.rstrip("\n") + if out: + print(out) + if err: + print(err) + + +def parse_list_env(raw: str | None, default: list[str]) -> list[str]: + if not raw: + return default + items = [] + for part in re.split(r"[,;\n]", raw): + part = part.strip() + if part: + items.append(part) + return items + + +def resolve_docker_since(since: str) -> str: + if re.match(r"^\d+[smhd]$", since): + return since + proc = run(["date", "-d", since, "--iso-8601=seconds"]) + if proc.returncode == 0 and proc.stdout.strip(): + return proc.stdout.strip() + return "24h" + + +def docker_names(all_containers=False) -> list[str]: + cmd = ["docker", "ps", "--format", "{{.Names}}"] + if all_containers: + cmd.insert(2, "-a") + proc = run(cmd) + if proc.returncode != 0: + return [] + return [line.strip() for line in proc.stdout.splitlines() if line.strip()] + + +def docker_inspect_labels(container: str) -> dict: + proc = run(["docker", "inspect", "--format", "{{json .Config.Labels}}", container]) + if proc.returncode != 0 or not proc.stdout.strip(): + return {} + try: + return json.loads(proc.stdout) + except json.JSONDecodeError: + return {} + + +def add_host_rule(rule: str, domains: set[str]) -> None: + if not isinstance(rule, str): + return + for host_match in HOST_RULE_RE.finditer(rule): + body = host_match.group(1) + for quoted_match in QUOTED_RE.finditer(body): + domain = next((item for item in quoted_match.groups() if item), "") + domain = domain.strip().lower() + if domain and "*" not in domain and "{" not in domain and "}" not in domain: + domains.add(domain) + + +def dynamic_files(): + for base in (pathlib.Path("/opt/traefik/dynamic"), pathlib.Path("/etc/traefik/dynamic")): + if not base.exists(): + continue + for path in base.rglob("*"): + if path.is_file() and path.suffix.lower() in {".yml", ".yaml", ".toml"}: + yield path + + +def active_tls_domains() -> set[str]: + domains: set[str] = set() + for container in docker_names(): + labels = docker_inspect_labels(container) + for key, value in labels.items(): + if key.startswith("traefik.http.routers.") and key.endswith(".rule"): + add_host_rule(value, domains) + for path in dynamic_files(): + try: + add_host_rule(path.read_text(encoding="utf-8", errors="ignore"), domains) + except Exception: + continue + return domains + + +def sablier_managed_containers() -> set[str]: + names = set() + for path in dynamic_files(): + try: + content = path.read_text(encoding="utf-8", errors="ignore") + except Exception: + continue + for match in SABLIER_RE.finditer(content): + names.add(match.group(1)) + return names + + +def health_request(url: str, max_time: int, follow_redirects: bool, max_redirects=8) -> dict: + current_url = url + redirects = 0 + start = time.monotonic() + last_code = 0 + context = ssl.create_default_context() + + while True: + parsed = urllib.parse.urlparse(current_url) + scheme = parsed.scheme or "https" + host = parsed.hostname + if not host: + raise ValueError(f"invalid URL: {current_url}") + port = parsed.port or (443 if scheme == "https" else 80) + path = urllib.parse.urlunparse(("", "", parsed.path or "/", parsed.params, parsed.query, "")) + conn_cls = http.client.HTTPSConnection if scheme == "https" else http.client.HTTPConnection + kwargs = {"timeout": max_time} + if scheme == "https": + kwargs["context"] = context + conn = conn_cls(host, port, **kwargs) + try: + conn.request("GET", path, headers={"User-Agent": "ops-report/1.0"}) + response = conn.getresponse() + last_code = response.status + response.read() + if follow_redirects and last_code in {301, 302, 303, 307, 308} and redirects < max_redirects: + location = response.getheader("Location") + if location: + current_url = urllib.parse.urljoin(current_url, location) + redirects += 1 + continue + elapsed = time.monotonic() - start + return { + "http_code": last_code, + "curl_time_total_seconds": f"{elapsed:.6f}", + "effective_url": current_url, + } + finally: + conn.close() + + +def cmd_health(_args) -> int: + max_time = env_int("HEALTH_MAX_TIME", 12, 1) + retries = env_int("HEALTH_RETRIES", 2, 1) + retry_delay = env_int("HEALTH_RETRY_DELAY", 8, 0) + follow_redirects = env_bool("HEALTH_FOLLOW_REDIRECTS", True) + endpoints = parse_list_env(os.environ.get("HEALTH_ENDPOINTS"), DEFAULT_ENDPOINTS) + + results = [] + for endpoint in endpoints: + start_ms = int(time.time() * 1000) + status = "down" + http_code = 0 + effective_url = "" + time_total = "" + error = "" + attempts = 0 + for attempt in range(1, retries + 1): + attempts = attempt + try: + response = health_request(endpoint, max_time, follow_redirects) + http_code = int(response["http_code"]) + effective_url = response["effective_url"] + time_total = response["curl_time_total_seconds"] + if 200 <= http_code < 400: + status = "up" + error = "" + break + error = f"HTTP {http_code}" + except Exception as exc: + http_code = 0 + effective_url = "" + time_total = "" + error = str(exc) + if attempt < retries: + time.sleep(retry_delay) + latency_ms = int(time.time() * 1000) - start_ms + item = { + "endpoint": endpoint, + "status": status, + "http_code": http_code, + "latency_ms": latency_ms, + "attempts": attempts, + } + if time_total: + item["curl_time_total_seconds"] = time_total + if effective_url: + item["effective_url"] = effective_url + if status == "down": + item["error"] = error + results.append(item) + + up_count = sum(1 for item in results if item["status"] == "up") + down_count = sum(1 for item in results if item["status"] == "down") + endpoints_count = up_count + down_count + overall = "up" + if endpoints_count == 0: + overall = "down" + elif down_count: + overall = "degraded" + print(json.dumps({ + "generated_at_utc": now_utc(), + "overall_status": overall, + "endpoints_count": endpoints_count, + "up_count": up_count, + "down_count": down_count, + "max_time_seconds": max_time, + "retries": retries, + "retry_delay_seconds": retry_delay, + "results": results, + }, indent=2)) + return 0 + + +def proc_text(path: pathlib.Path) -> str: + try: + return path.read_text(errors="ignore") + except Exception: + return "" + + +def top_swap_processes(limit: int) -> list[tuple[int, str, str]]: + rows = [] + if limit <= 0: + return rows + for status_file in pathlib.Path("/proc").glob("[0-9]*/status"): + data = proc_text(status_file) + if not data: + continue + name = "" + pid = status_file.parent.name + swap = 0 + for line in data.splitlines(): + if line.startswith("Name:"): + name = line.split(None, 1)[1] + elif line.startswith("Pid:"): + pid = line.split(None, 1)[1] + elif line.startswith("VmSwap:"): + parts = line.split() + if len(parts) >= 2: + try: + swap = int(parts[1]) + except ValueError: + swap = 0 + if swap > 0: + rows.append((swap, name, pid)) + rows.sort(reverse=True) + return rows[:limit] + + +def cert_end_date_from_file(path: pathlib.Path) -> str: + proc = run(["openssl", "x509", "-enddate", "-noout", "-in", str(path)]) + if proc.returncode != 0: + return "" + return proc.stdout.strip().split("=", 1)[-1] + + +def cert_end_date_from_pem(pem: str) -> str: + proc = run(["openssl", "x509", "-enddate", "-noout"], input_text=pem) + if proc.returncode != 0: + return "" + return proc.stdout.strip().split("=", 1)[-1] + + +def parse_cert_epoch(end_raw: str) -> int | None: + if not end_raw: + return None + proc = run(["date", "-d", end_raw, "+%s"]) + if proc.returncode != 0: + return None + try: + return int(proc.stdout.strip()) + except ValueError: + return None + + +def print_cert_line(domain: str, end_raw: str, warn_days: int) -> None: + if not end_raw: + print(f"{domain}: unable to parse cert") + return + epoch = parse_cert_epoch(end_raw) + if epoch is None: + print(f"{domain}: enddate={end_raw}") + return + days_left = int((epoch - time.time()) / 86400) + warn = " [WARN]" if days_left <= warn_days else "" + print(f"{domain}: expires={end_raw} days_left={days_left}{warn}") + + +def acme_entries(paths: list[str]) -> list[tuple[str, str]]: + entries = [] + for raw_path in paths: + path = pathlib.Path(raw_path) + if not path.exists(): + continue + try: + data = json.loads(path.read_text()) + except Exception: + continue + for payload in data.values(): + if not isinstance(payload, dict): + continue + for cert in payload.get("Certificates", []): + domain = (cert.get("domain") or {}).get("main") + encoded = cert.get("certificate") + if domain and encoded: + entries.append((domain, encoded)) + return entries + + +def cmd_summary(_args) -> int: + since_hours = env_int("SINCE_HOURS", 24, 1) + top_error_lines = env_int("TOP_ERROR_LINES", 20, 0) + cert_warn_days = env_int("CERT_WARN_DAYS", 30, 0) + top_swap = env_int("TOP_SWAP_PROCESSES", 10, 0) + show_overlays = env_bool("SHOW_DF_OVERLAYS", False) + show_running = env_bool("SHOW_RUNNING_CONTAINERS", False) + active_only = env_bool("ACTIVE_CERT_DOMAINS_ONLY", True) + exclude_domains = {d.lower() for d in parse_list_env( + os.environ.get("CERT_EXCLUDE_DOMAINS", "portainer.thehomelesssherlock.com uptimekuma.sherlockhomeless.net notas.sherlockhomeless.net").replace(" ", "\n"), + [], + )} + + print("# Server Daily Summary") + print(f"generated_at_utc: {now_utc()}") + hostname = run(["hostname", "-f"]).stdout.strip() or run(["hostname"]).stdout.strip() + print(f"host: {hostname}") + print(f"kernel: {run(['uname', '-r']).stdout.strip()}") + + section("Uptime & Load") + uptime = run(["uptime", "-p"]).stdout.strip() or run(["uptime"]).stdout.strip() + print(f"uptime: {uptime}") + loadavg = pathlib.Path("/proc/loadavg").read_text().split()[:3] + print(f"loadavg: {' '.join(loadavg)}") + + section("Memory") + print_command(["free", "-h"]) + if top_swap > 0: + print() + print("top_swap_processes:") + print("SWAP_KIB\tCOMMAND\tPID") + for swap, name, pid in top_swap_processes(top_swap): + print(f"{swap}\t{name}\t{pid}") + + section("Disk") + df_cmd = ["df", "-hT", "-x", "tmpfs", "-x", "devtmpfs"] + if not show_overlays: + df_cmd += ["-x", "overlay"] + print_command(df_cmd) + if not show_overlays: + print("overlay_mounts: omitted set SHOW_DF_OVERLAYS=1 to include") + + section("Docker Overview") + if not command_exists("docker"): + print("docker command not available") + else: + sablier = sablier_managed_containers() + running = docker_names() + unhealthy = run(["docker", "ps", "--filter", "health=unhealthy", "-q"]).stdout.splitlines() + exited = run(["docker", "ps", "-a", "--filter", "status=exited", "-q"]).stdout.splitlines() + restarting = run(["docker", "ps", "-a", "--filter", "status=restarting", "-q"]).stdout.splitlines() + print(f"containers_running: {len(running)}") + print(f"containers_unhealthy: {len(unhealthy)}") + print(f"containers_exited: {len(exited)}") + print(f"containers_restarting: {len(restarting)}") + print() + if show_running: + print("running_containers:") + print_command(["docker", "ps", "--format", "table {{.Names}}\t{{.Status}}\t{{.Networks}}"]) + else: + print(f"running_containers: omitted count={len(running)} set SHOW_RUNNING_CONTAINERS=1 to include") + if unhealthy: + print() + print("unhealthy_containers:") + print_command(["docker", "ps", "--filter", "health=unhealthy", "--format", "table {{.Names}}\t{{.Status}}\t{{.Image}}"]) + if exited or restarting: + print() + print("non_running_containers:") + non_running = run(["docker", "ps", "-a", "--filter", "status=exited", "--filter", "status=restarting", "--format", "{{.Names}}\t{{.Status}}\t{{.Image}}"]).stdout.splitlines() + print("NAMES\tSTATUS\tIMAGE") + parsed = [] + for line in non_running: + parts = line.split("\t") + if len(parts) == 3: + parsed.append(tuple(parts)) + print(line) + if sablier: + print() + print("non_running_sablier_managed_containers:") + print("NAMES\tSTATUS\tIMAGE") + for name, status, image in parsed: + if name in sablier: + print(f"{name}\t{status}\t{image}") + print() + print("non_running_other_containers:") + print("NAMES\tSTATUS\tIMAGE") + for name, status, image in parsed: + if name not in sablier: + print(f"{name}\t{status}\t{image}") + + section("TLS Certificates") + if not command_exists("openssl"): + print("openssl not available") + else: + found = False + seen = set() + active = active_tls_domains() if active_only else set() + if active_only: + if active: + print(f"tls_scope: active_domains_only active_domains={len(active)} excluded_domains={len(exclude_domains)}") + else: + print(f"tls_scope: active_domains_only requested but no active domains detected; fallback=all_discovered excluded_domains={len(exclude_domains)}") + live = pathlib.Path("/etc/letsencrypt/live") + if live.exists(): + for cert in sorted(live.glob("*/cert.pem")): + domain = cert.parent.name + domain_lc = domain.lower() + if domain_lc in exclude_domains: + continue + if active_only and active and domain_lc not in active: + continue + if domain in seen: + continue + seen.add(domain) + found = True + print_cert_line(domain, cert_end_date_from_file(cert), cert_warn_days) + for domain, encoded in acme_entries(["/opt/traefik/letsencrypt/acme.json", "/opt/traefik/acme/acme.json"]): + domain_lc = domain.lower() + if domain_lc in exclude_domains: + continue + if active_only and active and domain_lc not in active: + continue + if domain in seen: + continue + seen.add(domain) + found = True + try: + pem = base64.b64decode(encoded).decode() + except Exception: + print(f"{domain}: unable to decode traefik cert") + continue + print_cert_line(domain, cert_end_date_from_pem(pem), cert_warn_days) + if not found: + print("no local TLS certificates found") + + section("SSH Protection") + if not command_exists("fail2ban-client"): + print("fail2ban-client not available") + else: + proc = run(["fail2ban-client", "status", "sshd"]) + text = proc.stdout + if text: + def extract(label): + m = re.search(rf"{re.escape(label)}:\s*(\d+)", text) + return m.group(1) if m else "0" + print(f"fail2ban_sshd: active total_failed={extract('Total failed')} current_banned={extract('Currently banned')} total_banned={extract('Total banned')}") + print("note: SSH probes can still generate auth/journal log entries before or while bans are applied; that alone does not mean fail2ban is inactive.") + else: + print("fail2ban_sshd: no status available") + + section("OpenClaw OAuth") + report_openclaw_oauth() + + section(f"Top Journal Errors (last {since_hours}h)") + if command_exists("journalctl") and top_error_lines > 0: + proc = run(["journalctl", "-p", "err..alert", "--since", f"{since_hours} hours ago", "--no-pager", "-o", "short-iso"], timeout=30) + for line in proc.stdout.splitlines()[-top_error_lines:]: + print(line) + elif not command_exists("journalctl"): + print("journalctl not available") + return 0 + + +def report_openclaw_oauth() -> None: + auth_file = pathlib.Path(os.environ.get("OPENCLAW_AUTH_FILE", "/home/felidae/.openclaw/agents/main/agent/auth-profiles.json")) + warn_hours = env_int("OPENCLAW_WARN_HOURS", 72, 0) + critical_hours = env_int("OPENCLAW_CRITICAL_HOURS", 24, 0) + action = os.environ.get("OPENCLAW_REAUTH_CMD", "codex-reauth") + if not auth_file.exists(): + print("openclaw_codex_oauth: auth file or jq unavailable") + return + try: + data = json.loads(auth_file.read_text()) + profiles = data.get("profiles", {}) + expiries = [ + int(value.get("expires") or 0) + for value in profiles.values() + if isinstance(value, dict) and value.get("provider") == "openai-codex" + ] + expiry_ms = max(expiries) if expiries else 0 + except Exception: + expiry_ms = 0 + if expiry_ms <= 0: + print("openclaw_codex_oauth: no oauth profile expiry found") + return + now_ms = int(time.time() * 1000) + diff_ms = expiry_ms - now_ms + expiry_iso = dt.datetime.fromtimestamp(expiry_ms / 1000, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + if diff_ms <= 0: + print(f'openclaw_codex_oauth: CRITICAL expires_utc={expiry_iso} reason=expired action="{action}"') + return + hours_left = diff_ms // 3_600_000 + mins_left = (diff_ms % 3_600_000) // 60_000 + severity = "OK" + if hours_left <= critical_hours: + severity = "CRITICAL" + elif hours_left <= warn_hours: + severity = "WARN" + print(f'openclaw_codex_oauth: {severity} expires_utc={expiry_iso} hours_left={hours_left}h{mins_left}m critical_hours={critical_hours} warn_hours={warn_hours} action="{action}"') + + +def cmd_logs(_args) -> int: + since = os.environ.get("SINCE", "24 hours ago") + max_journal = env_int("MAX_JOURNAL_LINES", 120, 0) + max_docker = env_int("MAX_DOCKER_LINES_PER_SERVICE", 40, 0) + targets = parse_list_env(os.environ.get("TARGET_CONTAINERS"), DEFAULT_TARGET_CONTAINERS) + docker_since = resolve_docker_since(since) + + print("# Log Digest") + print(f"generated_at_utc: {now_utc()}") + print(f"window: {since}") + print(f"docker_since: {docker_since}") + + section("System Errors (journalctl err..alert)") + if command_exists("journalctl"): + proc = run(["journalctl", "--since", since, "-p", "err..alert", "--no-pager", "-o", "short-iso"], timeout=30) + for line in proc.stdout.splitlines()[-max_journal:]: + print(line) + else: + print("journalctl not available") + + section("System Warnings (journalctl warning)") + if command_exists("journalctl"): + proc = run(["journalctl", "--since", since, "-p", "warning", "--no-pager", "-o", "short-iso"], timeout=30) + for line in proc.stdout.splitlines()[-max_journal:]: + print(line) + else: + print("journalctl not available") + + section("Docker Service Error Highlights") + if command_exists("docker"): + all_names = set(docker_names(all_containers=True)) + for svc in targets: + if svc not in all_names: + continue + print() + print(f"### {svc}") + proc = run(["docker", "logs", "--since", docker_since, svc], timeout=30) + lines = (proc.stdout + proc.stderr).splitlines() + for line in [line for line in lines if ERROR_PATTERN.search(line)][-max_docker:]: + print(line) + else: + print("docker command not available") + + section("Top Error Patterns") + if command_exists("journalctl"): + proc = run(["journalctl", "--since", since, "--no-pager", "-o", "cat"], timeout=30) + counts = Counter() + for line in proc.stdout.splitlines(): + if re.search(r"error|exception|fatal|panic|critical|denied|timeout|failed", line, re.IGNORECASE): + normalized = re.sub(r"\s+", " ", line.strip())[:180] + counts[normalized] += 1 + for line, count in counts.most_common(20): + print(f"{count:7d} {line}") + else: + print("journalctl not available") + return 0 + + +def cmd_all(_args) -> int: + print("__N8N_LOG_DIGEST_BEGIN__") + rc_logs = cmd_logs(_args) + print("\n__N8N_SERVER_SUMMARY_BEGIN__") + rc_summary = cmd_summary(_args) + print("\n__N8N_HEALTH_RESULTS_BEGIN__") + rc_health = cmd_health(_args) + print(f"\n__N8N_STATUS__ log={rc_logs} summary={rc_summary} health={rc_health}") + return 0 if rc_logs == rc_summary == rc_health == 0 else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description="Personal ops report helper") + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("health").set_defaults(func=cmd_health) + sub.add_parser("summary").set_defaults(func=cmd_summary) + sub.add_parser("logs").set_defaults(func=cmd_logs) + sub.add_parser("all").set_defaults(func=cmd_all) + args = parser.parse_args() + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/n8n-observability/scripts/server-summary.sh b/n8n-observability/scripts/server-summary.sh index b40c438..6afb92f 100755 --- a/n8n-observability/scripts/server-summary.sh +++ b/n8n-observability/scripts/server-summary.sh @@ -1,332 +1,6 @@ #!/usr/bin/env bash set -euo pipefail -SINCE_HOURS="${SINCE_HOURS:-24}" -TOP_ERROR_LINES="${TOP_ERROR_LINES:-20}" -CERT_WARN_DAYS="${CERT_WARN_DAYS:-30}" -CERT_EXCLUDE_DOMAINS="${CERT_EXCLUDE_DOMAINS:-portainer.thehomelesssherlock.com uptimekuma.sherlockhomeless.net notas.sherlockhomeless.net}" -ACTIVE_CERT_DOMAINS_ONLY="${ACTIVE_CERT_DOMAINS_ONLY:-1}" -OPENCLAW_AUTH_FILE="${OPENCLAW_AUTH_FILE:-/home/felidae/.openclaw/agents/main/agent/auth-profiles.json}" -OPENCLAW_WARN_HOURS="${OPENCLAW_WARN_HOURS:-72}" -OPENCLAW_CRITICAL_HOURS="${OPENCLAW_CRITICAL_HOURS:-24}" -OPENCLAW_REAUTH_CMD="${OPENCLAW_REAUTH_CMD:-codex-reauth}" - -section() { - printf '\n## %s\n' "$1" -} - -report_cert_expiry() { - local domain="$1" - local end_raw="$2" - if [[ -z "$end_raw" ]]; then - printf '%s: unable to parse cert\n' "$domain" - return - fi - - local end_epoch - local now_epoch - local days_left - local warn - - end_epoch="$(date -d "$end_raw" +%s 2>/dev/null || true)" - now_epoch="$(date +%s)" - if [[ -z "$end_epoch" ]]; then - printf '%s: enddate=%s\n' "$domain" "$end_raw" - return - fi - - days_left="$(( (end_epoch - now_epoch) / 86400 ))" - warn='' - if (( days_left <= CERT_WARN_DAYS )); then - warn=' [WARN]' - fi - printf '%s: expires=%s days_left=%s%s\n' "$domain" "$end_raw" "$days_left" "$warn" -} - -report_pem_cert() { - local domain="$1" - local pem="$2" - local end_raw - - end_raw="$(openssl x509 -enddate -noout 2>/dev/null <<<"$pem" | cut -d= -f2- || true)" - report_cert_expiry "$domain" "$end_raw" -} - -emit_traefik_acme_entries() { - python3 - "$@" <<'PY' -import json -import pathlib -import sys - -for raw_path in sys.argv[1:]: - path = pathlib.Path(raw_path) - if not path.exists(): - continue - try: - data = json.loads(path.read_text()) - except Exception: - continue - for payload in data.values(): - if not isinstance(payload, dict): - continue - for cert in payload.get("Certificates", []): - domain = (cert.get("domain") or {}).get("main") - encoded = cert.get("certificate") - if domain and encoded: - print(f"{domain}\t{encoded}") -PY -} - -emit_active_tls_domains() { - python3 - <<'PY' -import json -import pathlib -import re -import subprocess - -host_rule_re = re.compile(r"Host\(([^)]*)\)") -quoted_re = re.compile(r"`([^`]+)`|'([^']+)'|\"([^\"]+)\"") -domains = set() - - -def add_rule(rule: str) -> None: - if not isinstance(rule, str): - return - for host_match in host_rule_re.finditer(rule): - body = host_match.group(1) - for quoted_match in quoted_re.finditer(body): - domain = next((item for item in quoted_match.groups() if item), "") - domain = domain.strip().lower() - if not domain or "*" in domain or "{" in domain or "}" in domain: - continue - domains.add(domain) - - -try: - container_names = subprocess.check_output( - ["docker", "ps", "--format", "{{.Names}}"], - text=True, - stderr=subprocess.DEVNULL, - ).splitlines() -except Exception: - container_names = [] - -for container in container_names: - container = container.strip() - if not container: - continue - try: - raw_labels = subprocess.check_output( - ["docker", "inspect", "--format", "{{json .Config.Labels}}", container], - text=True, - stderr=subprocess.DEVNULL, - ).strip() - labels = json.loads(raw_labels) if raw_labels else {} - except Exception: - continue - if not isinstance(labels, dict): - continue - for key, value in labels.items(): - if key.startswith("traefik.http.routers.") and key.endswith(".rule"): - add_rule(value) - -for base_path in (pathlib.Path("/opt/traefik/dynamic"), pathlib.Path("/etc/traefik/dynamic")): - if not base_path.exists(): - continue - for cfg_file in base_path.rglob("*"): - if not cfg_file.is_file(): - continue - try: - content = cfg_file.read_text(encoding="utf-8", errors="ignore") - except Exception: - continue - add_rule(content) - -for domain in sorted(domains): - print(domain) -PY -} - -now_utc="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" -hostname_fqdn="$(hostname -f 2>/dev/null || hostname)" -kernel="$(uname -r)" - -printf '# Server Daily Summary\n' -printf 'generated_at_utc: %s\n' "$now_utc" -printf 'host: %s\n' "$hostname_fqdn" -printf 'kernel: %s\n' "$kernel" - -section "Uptime & Load" -printf 'uptime: %s\n' "$(uptime -p 2>/dev/null || uptime)" -printf 'loadavg: %s\n' "$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo 'n/a')" - -section "Memory" -if command -v free >/dev/null 2>&1; then - free -h -else - echo 'free command not available' -fi - -section "Disk" -if command -v df >/dev/null 2>&1; then - df -hT -x tmpfs -x devtmpfs -else - echo 'df command not available' -fi - -section "Docker Overview" -if command -v docker >/dev/null 2>&1; then - running_count="$(docker ps -q | wc -l | tr -d ' ')" - unhealthy_count="$(docker ps --filter health=unhealthy -q | wc -l | tr -d ' ')" - exited_count="$(docker ps -a --filter status=exited -q | wc -l | tr -d ' ')" - restarting_count="$(docker ps -a --filter status=restarting -q | wc -l | tr -d ' ')" - - printf 'containers_running: %s\n' "$running_count" - printf 'containers_unhealthy: %s\n' "$unhealthy_count" - printf 'containers_exited: %s\n' "$exited_count" - printf 'containers_restarting: %s\n' "$restarting_count" - - echo - echo 'running_containers:' - docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}' - - if [[ "$unhealthy_count" != "0" ]]; then - echo - echo 'unhealthy_containers:' - docker ps --filter health=unhealthy --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' - fi - - if [[ "$exited_count" != "0" || "$restarting_count" != "0" ]]; then - echo - echo 'non_running_containers:' - docker ps -a --filter status=exited --filter status=restarting --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' - fi -else - echo 'docker command not available' -fi - -section "TLS Certificates" -if command -v openssl >/dev/null 2>&1; then - found=0 - declare -A seen_domains=() - declare -A excluded_domains=() - declare -A active_domains=() - - for domain in $CERT_EXCLUDE_DOMAINS; do - excluded_domains["${domain,,}"]=1 - done - - if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then - while IFS= read -r domain; do - [[ -z "$domain" ]] && continue - active_domains["${domain,,}"]=1 - done < <(emit_active_tls_domains) - fi - - if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then - if [[ "${#active_domains[@]}" -gt 0 ]]; then - printf 'tls_scope: active_domains_only active_domains=%s excluded_domains=%s\n' "${#active_domains[@]}" "${#excluded_domains[@]}" - else - printf 'tls_scope: active_domains_only requested but no active domains detected; fallback=all_discovered excluded_domains=%s\n' "${#excluded_domains[@]}" - fi - fi - - if [[ -d /etc/letsencrypt/live ]]; then - while IFS= read -r cert; do - domain="$(basename "$(dirname "$cert")")" - domain_lc="${domain,,}" - [[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue - if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then - continue - fi - [[ -n "${seen_domains[$domain]:-}" ]] && continue - seen_domains["$domain"]=1 - found=1 - end_raw="$(openssl x509 -enddate -noout -in "$cert" 2>/dev/null | cut -d= -f2- || true)" - report_cert_expiry "$domain" "$end_raw" - done < <(find /etc/letsencrypt/live -maxdepth 2 -type f -name cert.pem 2>/dev/null | sort) - fi - - while IFS=$'\t' read -r domain cert_b64; do - [[ -z "$domain" || -z "$cert_b64" ]] && continue - domain_lc="${domain,,}" - [[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue - if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then - continue - fi - [[ -n "${seen_domains[$domain]:-}" ]] && continue - seen_domains["$domain"]=1 - found=1 - pem="$(printf '%s' "$cert_b64" | base64 -d 2>/dev/null || true)" - if [[ -z "$pem" ]]; then - printf '%s: unable to decode traefik cert\n' "$domain" - continue - fi - report_pem_cert "$domain" "$pem" - done < <(emit_traefik_acme_entries /opt/traefik/letsencrypt/acme.json /opt/traefik/acme/acme.json) - - if [[ "$found" -eq 0 ]]; then - echo 'no local TLS certificates found' - fi -else - echo 'openssl not available' -fi - -section "SSH Protection" -if command -v fail2ban-client >/dev/null 2>&1; then - sshd_status="$(fail2ban-client status sshd 2>/dev/null || true)" - if [[ -n "$sshd_status" ]]; then - total_failed="$(grep -Eo 'Total failed:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')" - current_banned="$(grep -Eo 'Currently banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')" - total_banned="$(grep -Eo 'Total banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')" - printf 'fail2ban_sshd: active total_failed=%s current_banned=%s total_banned=%s\n' \ - "$total_failed" "$current_banned" "$total_banned" - echo 'note: SSH probes can still generate auth/journal log entries before or while bans are applied; that alone does not mean fail2ban is inactive.' - else - echo 'fail2ban_sshd: no status available' - fi -else - echo 'fail2ban-client not available' -fi - -section "OpenClaw OAuth" -if [[ -f "$OPENCLAW_AUTH_FILE" ]] && command -v jq >/dev/null 2>&1; then - openclaw_expiry_ms="$(jq -r ' - [.profiles | to_entries[] - | select(.value.provider == "openai-codex") - | (.value.expires // 0)] - | max // 0 - ' "$OPENCLAW_AUTH_FILE" 2>/dev/null || echo "0")" - - if [[ "$openclaw_expiry_ms" =~ ^[0-9]+$ ]] && (( openclaw_expiry_ms > 0 )); then - now_ms="$(( $(date +%s) * 1000 ))" - diff_ms="$(( openclaw_expiry_ms - now_ms ))" - expiry_iso="$(date -u -d "@$((openclaw_expiry_ms / 1000))" +'%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo 'n/a')" - if (( diff_ms <= 0 )); then - printf 'openclaw_codex_oauth: CRITICAL expires_utc=%s reason=expired action="%s"\n' \ - "$expiry_iso" "$OPENCLAW_REAUTH_CMD" - else - hours_left="$(( diff_ms / 3600000 ))" - mins_left="$(( (diff_ms % 3600000) / 60000 ))" - severity='OK' - if (( hours_left <= OPENCLAW_CRITICAL_HOURS )); then - severity='CRITICAL' - elif (( hours_left <= OPENCLAW_WARN_HOURS )); then - severity='WARN' - fi - printf 'openclaw_codex_oauth: %s expires_utc=%s hours_left=%sh%sm critical_hours=%s warn_hours=%s action="%s"\n' \ - "$severity" "$expiry_iso" "$hours_left" "$mins_left" "$OPENCLAW_CRITICAL_HOURS" "$OPENCLAW_WARN_HOURS" "$OPENCLAW_REAUTH_CMD" - fi - else - echo 'openclaw_codex_oauth: no oauth profile expiry found' - fi -else - echo 'openclaw_codex_oauth: auth file or jq unavailable' -fi - -section "Top Journal Errors (last ${SINCE_HOURS}h)" -if command -v journalctl >/dev/null 2>&1; then - journalctl -p err..alert --since "${SINCE_HOURS} hours ago" --no-pager -o short-iso 2>/dev/null | tail -n "$TOP_ERROR_LINES" || true -else - echo 'journalctl not available' -fi +SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")" +SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)" +exec "$SCRIPT_DIR/ops-report" summary "$@" diff --git a/n8n-observability/templates/ai-daily-report-prompt.md b/n8n-observability/templates/ai-daily-report-prompt.md index 95b3808..be1a36b 100644 --- a/n8n-observability/templates/ai-daily-report-prompt.md +++ b/n8n-observability/templates/ai-daily-report-prompt.md @@ -23,6 +23,12 @@ Entregar un resumen breve para operación diaria sin ruido. - Máximo 250 palabras. - El estado general se decide por **impacto activo**, no por riesgos preventivos. - `health_results` es opcional. +- Si `health_results` está presente, úsalo como fuente principal para disponibilidad actual. +- No declares un endpoint caído si `health_results` lo marca `up`, aunque haya errores antiguos en logs. +- Los errores de logs de una ventana amplia (por ejemplo 24h) son evidencia histórica; solo son incidente activo si aparecen como repetidos cerca de `generated_at_utc` o están correlacionados con `health_results` down/unhealthy/restarting. +- Si hay contradicción entre logs antiguos y health actual, reporta el punto como "resuelto/ruido histórico" o riesgo preventivo, no como incidente crítico. +- Si `log_digest` cubre una ventana amplia (por ejemplo 24h), no uses esos logs por sí solos para declarar "Servicios degradados con impacto". +- Para listar un servicio como degradado debe haber evidencia actual: `health_results` down, contenedor unhealthy/restarting, o una señal explícita actual en `server_summary`. ## Criterio estricto para "Estado general" - `CRITICAL` solo si hay impacto activo severo: @@ -35,6 +41,15 @@ Entregar un resumen breve para operación diaria sin ruido. - latencia/errores sostenidos que afectan servicio, sin caída total. - `OK` cuando no hay impacto activo, aunque existan riesgos preventivos. +## Reglas específicas de interpretación +- `302` hacia Authentik en servicios protegidos por SSO cuenta como disponible si `health_results` lo marca `up`. +- Un `404` en un dominio legacy o no listado por `health_results` no implica caída del servicio activo. +- Errores ACME antiguos no degradan si los dominios activos tienen certificados válidos y `health_results` está `up`. +- OAuth expirado en OpenClaw sí puede degradar OpenClaw, pero no debe atribuirse a Traefik/Auth. +- Si `server_summary` indica `openclaw_codex_oauth: OK`, no pidas reautenticación ni marques OpenClaw como degradado por errores OAuth antiguos del `log_digest`. +- Si `health_results` marca Traefik/Auth `up` y no hay errores actuales correlacionados, no marques Traefik/Auth como degradados por errores ACME/middleware antiguos. +- Si `server_summary` lista `non_running_sablier_managed_containers`, interprétalos como servicios bajo arranque bajo demanda. No los marques como incidente si `health_results` marca su endpoint asociado como `up`; si el endpoint está `down`, entonces sí trátalo como degradación activa. + ## Importante - **No marcar `DEGRADED` ni `CRITICAL` únicamente por prevención**: - certificados que aún no expiraron,