TheHomelessSherlock/Portainer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): consolidate ops reports in python

Browse Source
This commit is contained in:
Eduardo David Paredes Vara
2026-07-25 11:22:37 +00:00
parent 10d2a301e0
commit b9291b9c99
6 changed files with 716 additions and 515 deletions
Download Patch File Download Diff File Expand all files Collapse all files
n8n-observability/README.md
+6 -4
View File
@@ -8,9 +8,10 @@ Carpeta dedicada para automatizar en n8n:
## Estructura
- `scripts/server-summary.sh`: estado de host + docker + certs + errores top
- `scripts/log-digest.sh`: digest de logs (journal + docker) para alertas/informes
- `scripts/health-results.sh`: chequeo HTTP de endpoints y salida JSON (`up/down`, latencia, codigo)
- `scripts/ops-report`: motor Python unificado para `summary`, `logs`, `health` y `all`
- `scripts/server-summary.sh`: wrapper compatible para estado de host + docker + certs + errores top
- `scripts/log-digest.sh`: wrapper compatible para digest de logs (journal + docker) para alertas/informes
- `scripts/health-results.sh`: wrapper compatible para chequeo HTTP de endpoints y salida JSON (`up/down`, latencia, codigo)
- `templates/ai-daily-report-prompt.md`: prompt para convertir datos tecnicos en informe legible
- `config/services.example.json`: lista base de endpoints para health checks en n8n
- `workflows/workflow-design.md`: diseno de 3 workflows recomendados
@@ -18,10 +19,11 @@ Carpeta dedicada para automatizar en n8n:
## Uso rapido (en el host)
```bash
chmod +x n8n-observability/scripts/*.sh
chmod +x n8n-observability/scripts/*.sh n8n-observability/scripts/ops-report
./n8n-observability/scripts/server-summary.sh
SINCE="24 hours ago" ./n8n-observability/scripts/log-digest.sh
./n8n-observability/scripts/health-results.sh | jq
./n8n-observability/scripts/ops-report all
```
## Integracion recomendada con n8n
n8n-observability/scripts/health-results.sh
+3 -110
View File
@@ -1,113 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
MAX_TIME="${HEALTH_MAX_TIME:-12}"
FOLLOW_REDIRECTS="${HEALTH_FOLLOW_REDIRECTS:-1}"
load_endpoints() {
if [[ -n "${HEALTH_ENDPOINTS:-}" ]]; then
printf '%s\n' "$HEALTH_ENDPOINTS" | tr ',;' '\n' | sed '/^[[:space:]]*$/d' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
return
fi
cat <<'EOF'
https://traefik.thehomelesssherlock.com
https://auth.thehomelesssherlock.com
https://n8n.sherlockhomeless.net
https://nextcloud.sherlockhomeless.net
https://paperless.sherlockhomeless.net
https://onlyoffice.sherlockhomeless.net
https://memos.sherlockhomeless.net
https://vikunja.sherlockhomeless.net
https://karakeep.sherlockhomeless.net
https://dozzle.sherlockhomeless.net
https://beszel.sherlockhomeless.net
EOF
}
if ! command -v curl >/dev/null 2>&1; then
jq -n \
--arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
--arg error "curl command not available" \
'{generated_at_utc:$generated_at_utc, overall_status:"down", endpoints_count:0, up_count:0, down_count:0, error:$error, results:[]}'
exit 0
fi
if ! command -v jq >/dev/null 2>&1; then
echo '{"generated_at_utc":"'"$(date -u +'%Y-%m-%dT%H:%M:%SZ')"'", "overall_status":"down", "error":"jq command not available", "results":[]}'
exit 0
fi
mapfile -t ENDPOINTS < <(load_endpoints)
results_tmp="$(mktemp)"
trap 'rm -f "$results_tmp"' EXIT
for endpoint in "${ENDPOINTS[@]}"; do
[[ -z "$endpoint" ]] && continue
start_ms="$(date +%s%3N)"
curl_args=(-sS --max-time "$MAX_TIME" -o /dev/null -w '%{http_code} %{time_total} %{url_effective}')
if [[ "$FOLLOW_REDIRECTS" == "1" ]]; then
curl_args+=(-L)
fi
set +e
curl_out="$(curl "${curl_args[@]}" "$endpoint" 2>&1)"
curl_rc=$?
set -e
end_ms="$(date +%s%3N)"
latency_ms="$(( end_ms - start_ms ))"
if (( curl_rc == 0 )); then
http_code="$(awk '{print $1}' <<<"$curl_out")"
effective_url="$(awk '{print $3}' <<<"$curl_out")"
status="down"
if [[ "$http_code" =~ ^[0-9]+$ ]] && (( http_code >= 200 && http_code < 400 )); then
status="up"
fi
jq -cn \
--arg endpoint "$endpoint" \
--arg status "$status" \
--arg effective_url "$effective_url" \
--argjson http_code "${http_code:-0}" \
--argjson latency_ms "$latency_ms" \
'{endpoint:$endpoint,status:$status,http_code:$http_code,latency_ms:$latency_ms,effective_url:$effective_url}' >>"$results_tmp"
else
err_line="$(printf '%s\n' "$curl_out" | tail -n 1)"
jq -cn \
--arg endpoint "$endpoint" \
--arg error "$err_line" \
--argjson latency_ms "$latency_ms" \
'{endpoint:$endpoint,status:"down",http_code:0,latency_ms:$latency_ms,error:$error}' >>"$results_tmp"
fi
done
results_json="$(jq -s '.' "$results_tmp")"
up_count="$(jq '[.[] | select(.status=="up")] | length' <<<"$results_json")"
down_count="$(jq '[.[] | select(.status=="down")] | length' <<<"$results_json")"
endpoints_count="$(( up_count + down_count ))"
overall_status="up"
if (( endpoints_count == 0 )); then
overall_status="down"
elif (( down_count > 0 )); then
overall_status="degraded"
fi
jq -n \
--arg generated_at_utc "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
--arg overall_status "$overall_status" \
--argjson endpoints_count "$endpoints_count" \
--argjson up_count "$up_count" \
--argjson down_count "$down_count" \
--argjson max_time_seconds "$MAX_TIME" \
--argjson results "$results_json" \
'{
generated_at_utc:$generated_at_utc,
overall_status:$overall_status,
endpoints_count:$endpoints_count,
up_count:$up_count,
down_count:$down_count,
max_time_seconds:$max_time_seconds,
results:$results
}'
SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)"
exec "$SCRIPT_DIR/ops-report" health "$@"
n8n-observability/scripts/log-digest.sh
+3 -72
View File
@@ -1,75 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
SINCE="${SINCE:-24 hours ago}"
MAX_JOURNAL_LINES="${MAX_JOURNAL_LINES:-120}"
MAX_DOCKER_LINES_PER_SERVICE="${MAX_DOCKER_LINES_PER_SERVICE:-40}"
TARGET_CONTAINERS="${TARGET_CONTAINERS:-traefik n8n nextcloud paperless mail-relay openclaw-openclaw-gateway-1}"
resolve_docker_since() {
if [[ "$SINCE" =~ ^[0-9]+[smhd]$ ]]; then
echo "$SINCE"
return
fi
if date -d "$SINCE" +%s >/dev/null 2>&1; then
date -d "$SINCE" --iso-8601=seconds
return
fi
echo "24h"
}
DOCKER_SINCE="$(resolve_docker_since)"
section() {
printf '\n## %s\n' "$1"
}
printf '# Log Digest\n'
printf 'generated_at_utc: %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
printf 'window: %s\n' "$SINCE"
printf 'docker_since: %s\n' "$DOCKER_SINCE"
section "System Errors (journalctl err..alert)"
if command -v journalctl >/dev/null 2>&1; then
journalctl --since "$SINCE" -p err..alert --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true
else
echo 'journalctl not available'
fi
section "System Warnings (journalctl warning)"
if command -v journalctl >/dev/null 2>&1; then
journalctl --since "$SINCE" -p warning --no-pager -o short-iso 2>/dev/null | tail -n "$MAX_JOURNAL_LINES" || true
else
echo 'journalctl not available'
fi
section "Docker Service Error Highlights"
if command -v docker >/dev/null 2>&1; then
for svc in $TARGET_CONTAINERS; do
if docker ps -a --format '{{.Names}}' | grep -qx "$svc"; then
echo
echo "### $svc"
docker logs --since "$DOCKER_SINCE" "$svc" 2>&1 \
| grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|refused|unhealthy|failed|oauth|401' \
| tail -n "$MAX_DOCKER_LINES_PER_SERVICE" || true
fi
done
else
echo 'docker command not available'
fi
section "Top Error Patterns"
if command -v journalctl >/dev/null 2>&1; then
journalctl --since "$SINCE" --no-pager -o cat 2>/dev/null \
| grep -Ei 'error|exception|fatal|panic|critical|denied|timeout|failed' \
| sed 's/[[:space:]]\+/ /g' \
| cut -c1-180 \
| sort \
| uniq -c \
| sort -nr \
| head -n 20 || true
else
echo 'journalctl not available'
fi
SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)"
exec "$SCRIPT_DIR/ops-report" logs "$@"
n8n-observability/scripts/ops-report
Executable
+686
View File
@@ -0,0 +1,686 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import base64
import datetime as dt
import http.client
import json
import os
import pathlib
import re
import ssl
import subprocess
import sys
import time
import urllib.parse
from collections import Counter
DEFAULT_ENDPOINTS = [
"https://traefik.sherlockhomeless.net",
"https://auth.sherlockhomeless.net",
"https://n8n.sherlockhomeless.net/healthz",
"https://nextcloud.sherlockhomeless.net",
"https://paperless.sherlockhomeless.net",
"https://onlyoffice.sherlockhomeless.net",
"https://memos.sherlockhomeless.net",
"https://vikunja.sherlockhomeless.net",
"https://karakeep.sherlockhomeless.net",
"https://ots.sherlockhomeless.net",
"https://dozzle.sherlockhomeless.net",
"https://beszel.sherlockhomeless.net",
]
DEFAULT_TARGET_CONTAINERS = [
"traefik",
"n8n",
"nextcloud",
"paperless",
"mail-relay",
"openclaw-openclaw-gateway-1",
]
ERROR_PATTERN = re.compile(
r"error|exception|fatal|panic|critical|denied|timeout|refused|unhealthy|failed|oauth|401",
re.IGNORECASE,
)
HOST_RULE_RE = re.compile(r"Host\(([^)]*)\)")
QUOTED_RE = re.compile(r"`([^`]+)`|'([^']+)'|\"([^\"]+)\"")
SABLIER_RE = re.compile(r"sablier-([A-Za-z0-9_.-]+)@file")
def env_int(name: str, default: int, minimum: int = 0) -> int:
raw = os.environ.get(name)
try:
value = int(raw) if raw is not None else default
except ValueError:
return default
return value if value >= minimum else default
def env_bool(name: str, default: bool) -> bool:
raw = os.environ.get(name)
if raw is None:
return default
return raw.strip().lower() in {"1", "true", "yes", "on"}
def now_utc() -> str:
return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def section(title: str) -> None:
print(f"\n## {title}")
def run(cmd, *, timeout=30, check=False, input_text=None):
try:
return subprocess.run(
cmd,
text=True,
input=input_text,
capture_output=True,
timeout=timeout,
check=check,
)
except Exception as exc:
completed = subprocess.CompletedProcess(cmd, 1, "", str(exc))
return completed
def command_exists(name: str) -> bool:
return run(["sh", "-lc", f"command -v {name} >/dev/null 2>&1"]).returncode == 0
def print_command(cmd, *, timeout=30) -> None:
proc = run(cmd, timeout=timeout)
out = proc.stdout.rstrip("\n")
err = proc.stderr.rstrip("\n")
if out:
print(out)
if err:
print(err)
def parse_list_env(raw: str | None, default: list[str]) -> list[str]:
if not raw:
return default
items = []
for part in re.split(r"[,;\n]", raw):
part = part.strip()
if part:
items.append(part)
return items
def resolve_docker_since(since: str) -> str:
if re.match(r"^\d+[smhd]$", since):
return since
proc = run(["date", "-d", since, "--iso-8601=seconds"])
if proc.returncode == 0 and proc.stdout.strip():
return proc.stdout.strip()
return "24h"
def docker_names(all_containers=False) -> list[str]:
cmd = ["docker", "ps", "--format", "{{.Names}}"]
if all_containers:
cmd.insert(2, "-a")
proc = run(cmd)
if proc.returncode != 0:
return []
return [line.strip() for line in proc.stdout.splitlines() if line.strip()]
def docker_inspect_labels(container: str) -> dict:
proc = run(["docker", "inspect", "--format", "{{json .Config.Labels}}", container])
if proc.returncode != 0 or not proc.stdout.strip():
return {}
try:
return json.loads(proc.stdout)
except json.JSONDecodeError:
return {}
def add_host_rule(rule: str, domains: set[str]) -> None:
if not isinstance(rule, str):
return
for host_match in HOST_RULE_RE.finditer(rule):
body = host_match.group(1)
for quoted_match in QUOTED_RE.finditer(body):
domain = next((item for item in quoted_match.groups() if item), "")
domain = domain.strip().lower()
if domain and "*" not in domain and "{" not in domain and "}" not in domain:
domains.add(domain)
def dynamic_files():
for base in (pathlib.Path("/opt/traefik/dynamic"), pathlib.Path("/etc/traefik/dynamic")):
if not base.exists():
continue
for path in base.rglob("*"):
if path.is_file() and path.suffix.lower() in {".yml", ".yaml", ".toml"}:
yield path
def active_tls_domains() -> set[str]:
domains: set[str] = set()
for container in docker_names():
labels = docker_inspect_labels(container)
for key, value in labels.items():
if key.startswith("traefik.http.routers.") and key.endswith(".rule"):
add_host_rule(value, domains)
for path in dynamic_files():
try:
add_host_rule(path.read_text(encoding="utf-8", errors="ignore"), domains)
except Exception:
continue
return domains
def sablier_managed_containers() -> set[str]:
names = set()
for path in dynamic_files():
try:
content = path.read_text(encoding="utf-8", errors="ignore")
except Exception:
continue
for match in SABLIER_RE.finditer(content):
names.add(match.group(1))
return names
def health_request(url: str, max_time: int, follow_redirects: bool, max_redirects=8) -> dict:
current_url = url
redirects = 0
start = time.monotonic()
last_code = 0
context = ssl.create_default_context()
while True:
parsed = urllib.parse.urlparse(current_url)
scheme = parsed.scheme or "https"
host = parsed.hostname
if not host:
raise ValueError(f"invalid URL: {current_url}")
port = parsed.port or (443 if scheme == "https" else 80)
path = urllib.parse.urlunparse(("", "", parsed.path or "/", parsed.params, parsed.query, ""))
conn_cls = http.client.HTTPSConnection if scheme == "https" else http.client.HTTPConnection
kwargs = {"timeout": max_time}
if scheme == "https":
kwargs["context"] = context
conn = conn_cls(host, port, **kwargs)
try:
conn.request("GET", path, headers={"User-Agent": "ops-report/1.0"})
response = conn.getresponse()
last_code = response.status
response.read()
if follow_redirects and last_code in {301, 302, 303, 307, 308} and redirects < max_redirects:
location = response.getheader("Location")
if location:
current_url = urllib.parse.urljoin(current_url, location)
redirects += 1
continue
elapsed = time.monotonic() - start
return {
"http_code": last_code,
"curl_time_total_seconds": f"{elapsed:.6f}",
"effective_url": current_url,
}
finally:
conn.close()
def cmd_health(_args) -> int:
max_time = env_int("HEALTH_MAX_TIME", 12, 1)
retries = env_int("HEALTH_RETRIES", 2, 1)
retry_delay = env_int("HEALTH_RETRY_DELAY", 8, 0)
follow_redirects = env_bool("HEALTH_FOLLOW_REDIRECTS", True)
endpoints = parse_list_env(os.environ.get("HEALTH_ENDPOINTS"), DEFAULT_ENDPOINTS)
results = []
for endpoint in endpoints:
start_ms = int(time.time() * 1000)
status = "down"
http_code = 0
effective_url = ""
time_total = ""
error = ""
attempts = 0
for attempt in range(1, retries + 1):
attempts = attempt
try:
response = health_request(endpoint, max_time, follow_redirects)
http_code = int(response["http_code"])
effective_url = response["effective_url"]
time_total = response["curl_time_total_seconds"]
if 200 <= http_code < 400:
status = "up"
error = ""
break
error = f"HTTP {http_code}"
except Exception as exc:
http_code = 0
effective_url = ""
time_total = ""
error = str(exc)
if attempt < retries:
time.sleep(retry_delay)
latency_ms = int(time.time() * 1000) - start_ms
item = {
"endpoint": endpoint,
"status": status,
"http_code": http_code,
"latency_ms": latency_ms,
"attempts": attempts,
}
if time_total:
item["curl_time_total_seconds"] = time_total
if effective_url:
item["effective_url"] = effective_url
if status == "down":
item["error"] = error
results.append(item)
up_count = sum(1 for item in results if item["status"] == "up")
down_count = sum(1 for item in results if item["status"] == "down")
endpoints_count = up_count + down_count
overall = "up"
if endpoints_count == 0:
overall = "down"
elif down_count:
overall = "degraded"
print(json.dumps({
"generated_at_utc": now_utc(),
"overall_status": overall,
"endpoints_count": endpoints_count,
"up_count": up_count,
"down_count": down_count,
"max_time_seconds": max_time,
"retries": retries,
"retry_delay_seconds": retry_delay,
"results": results,
}, indent=2))
return 0
def proc_text(path: pathlib.Path) -> str:
try:
return path.read_text(errors="ignore")
except Exception:
return ""
def top_swap_processes(limit: int) -> list[tuple[int, str, str]]:
rows = []
if limit <= 0:
return rows
for status_file in pathlib.Path("/proc").glob("[0-9]*/status"):
data = proc_text(status_file)
if not data:
continue
name = ""
pid = status_file.parent.name
swap = 0
for line in data.splitlines():
if line.startswith("Name:"):
name = line.split(None, 1)[1]
elif line.startswith("Pid:"):
pid = line.split(None, 1)[1]
elif line.startswith("VmSwap:"):
parts = line.split()
if len(parts) >= 2:
try:
swap = int(parts[1])
except ValueError:
swap = 0
if swap > 0:
rows.append((swap, name, pid))
rows.sort(reverse=True)
return rows[:limit]
def cert_end_date_from_file(path: pathlib.Path) -> str:
proc = run(["openssl", "x509", "-enddate", "-noout", "-in", str(path)])
if proc.returncode != 0:
return ""
return proc.stdout.strip().split("=", 1)[-1]
def cert_end_date_from_pem(pem: str) -> str:
proc = run(["openssl", "x509", "-enddate", "-noout"], input_text=pem)
if proc.returncode != 0:
return ""
return proc.stdout.strip().split("=", 1)[-1]
def parse_cert_epoch(end_raw: str) -> int | None:
if not end_raw:
return None
proc = run(["date", "-d", end_raw, "+%s"])
if proc.returncode != 0:
return None
try:
return int(proc.stdout.strip())
except ValueError:
return None
def print_cert_line(domain: str, end_raw: str, warn_days: int) -> None:
if not end_raw:
print(f"{domain}: unable to parse cert")
return
epoch = parse_cert_epoch(end_raw)
if epoch is None:
print(f"{domain}: enddate={end_raw}")
return
days_left = int((epoch - time.time()) / 86400)
warn = " [WARN]" if days_left <= warn_days else ""
print(f"{domain}: expires={end_raw} days_left={days_left}{warn}")
def acme_entries(paths: list[str]) -> list[tuple[str, str]]:
entries = []
for raw_path in paths:
path = pathlib.Path(raw_path)
if not path.exists():
continue
try:
data = json.loads(path.read_text())
except Exception:
continue
for payload in data.values():
if not isinstance(payload, dict):
continue
for cert in payload.get("Certificates", []):
domain = (cert.get("domain") or {}).get("main")
encoded = cert.get("certificate")
if domain and encoded:
entries.append((domain, encoded))
return entries
def cmd_summary(_args) -> int:
since_hours = env_int("SINCE_HOURS", 24, 1)
top_error_lines = env_int("TOP_ERROR_LINES", 20, 0)
cert_warn_days = env_int("CERT_WARN_DAYS", 30, 0)
top_swap = env_int("TOP_SWAP_PROCESSES", 10, 0)
show_overlays = env_bool("SHOW_DF_OVERLAYS", False)
show_running = env_bool("SHOW_RUNNING_CONTAINERS", False)
active_only = env_bool("ACTIVE_CERT_DOMAINS_ONLY", True)
exclude_domains = {d.lower() for d in parse_list_env(
os.environ.get("CERT_EXCLUDE_DOMAINS", "portainer.thehomelesssherlock.com uptimekuma.sherlockhomeless.net notas.sherlockhomeless.net").replace(" ", "\n"),
[],
)}
print("# Server Daily Summary")
print(f"generated_at_utc: {now_utc()}")
hostname = run(["hostname", "-f"]).stdout.strip() or run(["hostname"]).stdout.strip()
print(f"host: {hostname}")
print(f"kernel: {run(['uname', '-r']).stdout.strip()}")
section("Uptime & Load")
uptime = run(["uptime", "-p"]).stdout.strip() or run(["uptime"]).stdout.strip()
print(f"uptime: {uptime}")
loadavg = pathlib.Path("/proc/loadavg").read_text().split()[:3]
print(f"loadavg: {' '.join(loadavg)}")
section("Memory")
print_command(["free", "-h"])
if top_swap > 0:
print()
print("top_swap_processes:")
print("SWAP_KIB\tCOMMAND\tPID")
for swap, name, pid in top_swap_processes(top_swap):
print(f"{swap}\t{name}\t{pid}")
section("Disk")
df_cmd = ["df", "-hT", "-x", "tmpfs", "-x", "devtmpfs"]
if not show_overlays:
df_cmd += ["-x", "overlay"]
print_command(df_cmd)
if not show_overlays:
print("overlay_mounts: omitted set SHOW_DF_OVERLAYS=1 to include")
section("Docker Overview")
if not command_exists("docker"):
print("docker command not available")
else:
sablier = sablier_managed_containers()
running = docker_names()
unhealthy = run(["docker", "ps", "--filter", "health=unhealthy", "-q"]).stdout.splitlines()
exited = run(["docker", "ps", "-a", "--filter", "status=exited", "-q"]).stdout.splitlines()
restarting = run(["docker", "ps", "-a", "--filter", "status=restarting", "-q"]).stdout.splitlines()
print(f"containers_running: {len(running)}")
print(f"containers_unhealthy: {len(unhealthy)}")
print(f"containers_exited: {len(exited)}")
print(f"containers_restarting: {len(restarting)}")
print()
if show_running:
print("running_containers:")
print_command(["docker", "ps", "--format", "table {{.Names}}\t{{.Status}}\t{{.Networks}}"])
else:
print(f"running_containers: omitted count={len(running)} set SHOW_RUNNING_CONTAINERS=1 to include")
if unhealthy:
print()
print("unhealthy_containers:")
print_command(["docker", "ps", "--filter", "health=unhealthy", "--format", "table {{.Names}}\t{{.Status}}\t{{.Image}}"])
if exited or restarting:
print()
print("non_running_containers:")
non_running = run(["docker", "ps", "-a", "--filter", "status=exited", "--filter", "status=restarting", "--format", "{{.Names}}\t{{.Status}}\t{{.Image}}"]).stdout.splitlines()
print("NAMES\tSTATUS\tIMAGE")
parsed = []
for line in non_running:
parts = line.split("\t")
if len(parts) == 3:
parsed.append(tuple(parts))
print(line)
if sablier:
print()
print("non_running_sablier_managed_containers:")
print("NAMES\tSTATUS\tIMAGE")
for name, status, image in parsed:
if name in sablier:
print(f"{name}\t{status}\t{image}")
print()
print("non_running_other_containers:")
print("NAMES\tSTATUS\tIMAGE")
for name, status, image in parsed:
if name not in sablier:
print(f"{name}\t{status}\t{image}")
section("TLS Certificates")
if not command_exists("openssl"):
print("openssl not available")
else:
found = False
seen = set()
active = active_tls_domains() if active_only else set()
if active_only:
if active:
print(f"tls_scope: active_domains_only active_domains={len(active)} excluded_domains={len(exclude_domains)}")
else:
print(f"tls_scope: active_domains_only requested but no active domains detected; fallback=all_discovered excluded_domains={len(exclude_domains)}")
live = pathlib.Path("/etc/letsencrypt/live")
if live.exists():
for cert in sorted(live.glob("*/cert.pem")):
domain = cert.parent.name
domain_lc = domain.lower()
if domain_lc in exclude_domains:
continue
if active_only and active and domain_lc not in active:
continue
if domain in seen:
continue
seen.add(domain)
found = True
print_cert_line(domain, cert_end_date_from_file(cert), cert_warn_days)
for domain, encoded in acme_entries(["/opt/traefik/letsencrypt/acme.json", "/opt/traefik/acme/acme.json"]):
domain_lc = domain.lower()
if domain_lc in exclude_domains:
continue
if active_only and active and domain_lc not in active:
continue
if domain in seen:
continue
seen.add(domain)
found = True
try:
pem = base64.b64decode(encoded).decode()
except Exception:
print(f"{domain}: unable to decode traefik cert")
continue
print_cert_line(domain, cert_end_date_from_pem(pem), cert_warn_days)
if not found:
print("no local TLS certificates found")
section("SSH Protection")
if not command_exists("fail2ban-client"):
print("fail2ban-client not available")
else:
proc = run(["fail2ban-client", "status", "sshd"])
text = proc.stdout
if text:
def extract(label):
m = re.search(rf"{re.escape(label)}:\s*(\d+)", text)
return m.group(1) if m else "0"
print(f"fail2ban_sshd: active total_failed={extract('Total failed')} current_banned={extract('Currently banned')} total_banned={extract('Total banned')}")
print("note: SSH probes can still generate auth/journal log entries before or while bans are applied; that alone does not mean fail2ban is inactive.")
else:
print("fail2ban_sshd: no status available")
section("OpenClaw OAuth")
report_openclaw_oauth()
section(f"Top Journal Errors (last {since_hours}h)")
if command_exists("journalctl") and top_error_lines > 0:
proc = run(["journalctl", "-p", "err..alert", "--since", f"{since_hours} hours ago", "--no-pager", "-o", "short-iso"], timeout=30)
for line in proc.stdout.splitlines()[-top_error_lines:]:
print(line)
elif not command_exists("journalctl"):
print("journalctl not available")
return 0
def report_openclaw_oauth() -> None:
auth_file = pathlib.Path(os.environ.get("OPENCLAW_AUTH_FILE", "/home/felidae/.openclaw/agents/main/agent/auth-profiles.json"))
warn_hours = env_int("OPENCLAW_WARN_HOURS", 72, 0)
critical_hours = env_int("OPENCLAW_CRITICAL_HOURS", 24, 0)
action = os.environ.get("OPENCLAW_REAUTH_CMD", "codex-reauth")
if not auth_file.exists():
print("openclaw_codex_oauth: auth file or jq unavailable")
return
try:
data = json.loads(auth_file.read_text())
profiles = data.get("profiles", {})
expiries = [
int(value.get("expires") or 0)
for value in profiles.values()
if isinstance(value, dict) and value.get("provider") == "openai-codex"
]
expiry_ms = max(expiries) if expiries else 0
except Exception:
expiry_ms = 0
if expiry_ms <= 0:
print("openclaw_codex_oauth: no oauth profile expiry found")
return
now_ms = int(time.time() * 1000)
diff_ms = expiry_ms - now_ms
expiry_iso = dt.datetime.fromtimestamp(expiry_ms / 1000, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
if diff_ms <= 0:
print(f'openclaw_codex_oauth: CRITICAL expires_utc={expiry_iso} reason=expired action="{action}"')
return
hours_left = diff_ms // 3_600_000
mins_left = (diff_ms % 3_600_000) // 60_000
severity = "OK"
if hours_left <= critical_hours:
severity = "CRITICAL"
elif hours_left <= warn_hours:
severity = "WARN"
print(f'openclaw_codex_oauth: {severity} expires_utc={expiry_iso} hours_left={hours_left}h{mins_left}m critical_hours={critical_hours} warn_hours={warn_hours} action="{action}"')
def cmd_logs(_args) -> int:
since = os.environ.get("SINCE", "24 hours ago")
max_journal = env_int("MAX_JOURNAL_LINES", 120, 0)
max_docker = env_int("MAX_DOCKER_LINES_PER_SERVICE", 40, 0)
targets = parse_list_env(os.environ.get("TARGET_CONTAINERS"), DEFAULT_TARGET_CONTAINERS)
docker_since = resolve_docker_since(since)
print("# Log Digest")
print(f"generated_at_utc: {now_utc()}")
print(f"window: {since}")
print(f"docker_since: {docker_since}")
section("System Errors (journalctl err..alert)")
if command_exists("journalctl"):
proc = run(["journalctl", "--since", since, "-p", "err..alert", "--no-pager", "-o", "short-iso"], timeout=30)
for line in proc.stdout.splitlines()[-max_journal:]:
print(line)
else:
print("journalctl not available")
section("System Warnings (journalctl warning)")
if command_exists("journalctl"):
proc = run(["journalctl", "--since", since, "-p", "warning", "--no-pager", "-o", "short-iso"], timeout=30)
for line in proc.stdout.splitlines()[-max_journal:]:
print(line)
else:
print("journalctl not available")
section("Docker Service Error Highlights")
if command_exists("docker"):
all_names = set(docker_names(all_containers=True))
for svc in targets:
if svc not in all_names:
continue
print()
print(f"### {svc}")
proc = run(["docker", "logs", "--since", docker_since, svc], timeout=30)
lines = (proc.stdout + proc.stderr).splitlines()
for line in [line for line in lines if ERROR_PATTERN.search(line)][-max_docker:]:
print(line)
else:
print("docker command not available")
section("Top Error Patterns")
if command_exists("journalctl"):
proc = run(["journalctl", "--since", since, "--no-pager", "-o", "cat"], timeout=30)
counts = Counter()
for line in proc.stdout.splitlines():
if re.search(r"error|exception|fatal|panic|critical|denied|timeout|failed", line, re.IGNORECASE):
normalized = re.sub(r"\s+", " ", line.strip())[:180]
counts[normalized] += 1
for line, count in counts.most_common(20):
print(f"{count:7d} {line}")
else:
print("journalctl not available")
return 0
def cmd_all(_args) -> int:
print("__N8N_LOG_DIGEST_BEGIN__")
rc_logs = cmd_logs(_args)
print("\n__N8N_SERVER_SUMMARY_BEGIN__")
rc_summary = cmd_summary(_args)
print("\n__N8N_HEALTH_RESULTS_BEGIN__")
rc_health = cmd_health(_args)
print(f"\n__N8N_STATUS__ log={rc_logs} summary={rc_summary} health={rc_health}")
return 0 if rc_logs == rc_summary == rc_health == 0 else 1
def main() -> int:
parser = argparse.ArgumentParser(description="Personal ops report helper")
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("health").set_defaults(func=cmd_health)
sub.add_parser("summary").set_defaults(func=cmd_summary)
sub.add_parser("logs").set_defaults(func=cmd_logs)
sub.add_parser("all").set_defaults(func=cmd_all)
args = parser.parse_args()
return args.func(args)
if __name__ == "__main__":
sys.exit(main())
n8n-observability/scripts/server-summary.sh
+3 -329
View File
@@ -1,332 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
SINCE_HOURS="${SINCE_HOURS:-24}"
TOP_ERROR_LINES="${TOP_ERROR_LINES:-20}"
CERT_WARN_DAYS="${CERT_WARN_DAYS:-30}"
CERT_EXCLUDE_DOMAINS="${CERT_EXCLUDE_DOMAINS:-portainer.thehomelesssherlock.com uptimekuma.sherlockhomeless.net notas.sherlockhomeless.net}"
ACTIVE_CERT_DOMAINS_ONLY="${ACTIVE_CERT_DOMAINS_ONLY:-1}"
OPENCLAW_AUTH_FILE="${OPENCLAW_AUTH_FILE:-/home/felidae/.openclaw/agents/main/agent/auth-profiles.json}"
OPENCLAW_WARN_HOURS="${OPENCLAW_WARN_HOURS:-72}"
OPENCLAW_CRITICAL_HOURS="${OPENCLAW_CRITICAL_HOURS:-24}"
OPENCLAW_REAUTH_CMD="${OPENCLAW_REAUTH_CMD:-codex-reauth}"
section() {
printf '\n## %s\n' "$1"
}
report_cert_expiry() {
local domain="$1"
local end_raw="$2"
if [[ -z "$end_raw" ]]; then
printf '%s: unable to parse cert\n' "$domain"
return
fi
local end_epoch
local now_epoch
local days_left
local warn
end_epoch="$(date -d "$end_raw" +%s 2>/dev/null || true)"
now_epoch="$(date +%s)"
if [[ -z "$end_epoch" ]]; then
printf '%s: enddate=%s\n' "$domain" "$end_raw"
return
fi
days_left="$(( (end_epoch - now_epoch) / 86400 ))"
warn=''
if (( days_left <= CERT_WARN_DAYS )); then
warn=' [WARN]'
fi
printf '%s: expires=%s days_left=%s%s\n' "$domain" "$end_raw" "$days_left" "$warn"
}
report_pem_cert() {
local domain="$1"
local pem="$2"
local end_raw
end_raw="$(openssl x509 -enddate -noout 2>/dev/null <<<"$pem" | cut -d= -f2- || true)"
report_cert_expiry "$domain" "$end_raw"
}
emit_traefik_acme_entries() {
python3 - "$@" <<'PY'
import json
import pathlib
import sys
for raw_path in sys.argv[1:]:
path = pathlib.Path(raw_path)
if not path.exists():
continue
try:
data = json.loads(path.read_text())
except Exception:
continue
for payload in data.values():
if not isinstance(payload, dict):
continue
for cert in payload.get("Certificates", []):
domain = (cert.get("domain") or {}).get("main")
encoded = cert.get("certificate")
if domain and encoded:
print(f"{domain}\t{encoded}")
PY
}
emit_active_tls_domains() {
python3 - <<'PY'
import json
import pathlib
import re
import subprocess
host_rule_re = re.compile(r"Host\(([^)]*)\)")
quoted_re = re.compile(r"`([^`]+)`|'([^']+)'|\"([^\"]+)\"")
domains = set()
def add_rule(rule: str) -> None:
if not isinstance(rule, str):
return
for host_match in host_rule_re.finditer(rule):
body = host_match.group(1)
for quoted_match in quoted_re.finditer(body):
domain = next((item for item in quoted_match.groups() if item), "")
domain = domain.strip().lower()
if not domain or "*" in domain or "{" in domain or "}" in domain:
continue
domains.add(domain)
try:
container_names = subprocess.check_output(
["docker", "ps", "--format", "{{.Names}}"],
text=True,
stderr=subprocess.DEVNULL,
).splitlines()
except Exception:
container_names = []
for container in container_names:
container = container.strip()
if not container:
continue
try:
raw_labels = subprocess.check_output(
["docker", "inspect", "--format", "{{json .Config.Labels}}", container],
text=True,
stderr=subprocess.DEVNULL,
).strip()
labels = json.loads(raw_labels) if raw_labels else {}
except Exception:
continue
if not isinstance(labels, dict):
continue
for key, value in labels.items():
if key.startswith("traefik.http.routers.") and key.endswith(".rule"):
add_rule(value)
for base_path in (pathlib.Path("/opt/traefik/dynamic"), pathlib.Path("/etc/traefik/dynamic")):
if not base_path.exists():
continue
for cfg_file in base_path.rglob("*"):
if not cfg_file.is_file():
continue
try:
content = cfg_file.read_text(encoding="utf-8", errors="ignore")
except Exception:
continue
add_rule(content)
for domain in sorted(domains):
print(domain)
PY
}
now_utc="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
hostname_fqdn="$(hostname -f 2>/dev/null || hostname)"
kernel="$(uname -r)"
printf '# Server Daily Summary\n'
printf 'generated_at_utc: %s\n' "$now_utc"
printf 'host: %s\n' "$hostname_fqdn"
printf 'kernel: %s\n' "$kernel"
section "Uptime & Load"
printf 'uptime: %s\n' "$(uptime -p 2>/dev/null || uptime)"
printf 'loadavg: %s\n' "$(cut -d' ' -f1-3 /proc/loadavg 2>/dev/null || echo 'n/a')"
section "Memory"
if command -v free >/dev/null 2>&1; then
free -h
else
echo 'free command not available'
fi
section "Disk"
if command -v df >/dev/null 2>&1; then
df -hT -x tmpfs -x devtmpfs
else
echo 'df command not available'
fi
section "Docker Overview"
if command -v docker >/dev/null 2>&1; then
running_count="$(docker ps -q | wc -l | tr -d ' ')"
unhealthy_count="$(docker ps --filter health=unhealthy -q | wc -l | tr -d ' ')"
exited_count="$(docker ps -a --filter status=exited -q | wc -l | tr -d ' ')"
restarting_count="$(docker ps -a --filter status=restarting -q | wc -l | tr -d ' ')"
printf 'containers_running: %s\n' "$running_count"
printf 'containers_unhealthy: %s\n' "$unhealthy_count"
printf 'containers_exited: %s\n' "$exited_count"
printf 'containers_restarting: %s\n' "$restarting_count"
echo
echo 'running_containers:'
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}'
if [[ "$unhealthy_count" != "0" ]]; then
echo
echo 'unhealthy_containers:'
docker ps --filter health=unhealthy --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
fi
if [[ "$exited_count" != "0" || "$restarting_count" != "0" ]]; then
echo
echo 'non_running_containers:'
docker ps -a --filter status=exited --filter status=restarting --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
fi
else
echo 'docker command not available'
fi
section "TLS Certificates"
if command -v openssl >/dev/null 2>&1; then
found=0
declare -A seen_domains=()
declare -A excluded_domains=()
declare -A active_domains=()
for domain in $CERT_EXCLUDE_DOMAINS; do
excluded_domains["${domain,,}"]=1
done
if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then
while IFS= read -r domain; do
[[ -z "$domain" ]] && continue
active_domains["${domain,,}"]=1
done < <(emit_active_tls_domains)
fi
if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" ]]; then
if [[ "${#active_domains[@]}" -gt 0 ]]; then
printf 'tls_scope: active_domains_only active_domains=%s excluded_domains=%s\n' "${#active_domains[@]}" "${#excluded_domains[@]}"
else
printf 'tls_scope: active_domains_only requested but no active domains detected; fallback=all_discovered excluded_domains=%s\n' "${#excluded_domains[@]}"
fi
fi
if [[ -d /etc/letsencrypt/live ]]; then
while IFS= read -r cert; do
domain="$(basename "$(dirname "$cert")")"
domain_lc="${domain,,}"
[[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue
if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then
continue
fi
[[ -n "${seen_domains[$domain]:-}" ]] && continue
seen_domains["$domain"]=1
found=1
end_raw="$(openssl x509 -enddate -noout -in "$cert" 2>/dev/null | cut -d= -f2- || true)"
report_cert_expiry "$domain" "$end_raw"
done < <(find /etc/letsencrypt/live -maxdepth 2 -type f -name cert.pem 2>/dev/null | sort)
fi
while IFS=$'\t' read -r domain cert_b64; do
[[ -z "$domain" || -z "$cert_b64" ]] && continue
domain_lc="${domain,,}"
[[ -n "${excluded_domains[$domain_lc]:-}" ]] && continue
if [[ "$ACTIVE_CERT_DOMAINS_ONLY" != "0" && "${#active_domains[@]}" -gt 0 && -z "${active_domains[$domain_lc]:-}" ]]; then
continue
fi
[[ -n "${seen_domains[$domain]:-}" ]] && continue
seen_domains["$domain"]=1
found=1
pem="$(printf '%s' "$cert_b64" | base64 -d 2>/dev/null || true)"
if [[ -z "$pem" ]]; then
printf '%s: unable to decode traefik cert\n' "$domain"
continue
fi
report_pem_cert "$domain" "$pem"
done < <(emit_traefik_acme_entries /opt/traefik/letsencrypt/acme.json /opt/traefik/acme/acme.json)
if [[ "$found" -eq 0 ]]; then
echo 'no local TLS certificates found'
fi
else
echo 'openssl not available'
fi
section "SSH Protection"
if command -v fail2ban-client >/dev/null 2>&1; then
sshd_status="$(fail2ban-client status sshd 2>/dev/null || true)"
if [[ -n "$sshd_status" ]]; then
total_failed="$(grep -Eo 'Total failed:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
current_banned="$(grep -Eo 'Currently banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
total_banned="$(grep -Eo 'Total banned:[[:space:]]*[0-9]+' <<<"$sshd_status" | grep -Eo '[0-9]+' || echo '0')"
printf 'fail2ban_sshd: active total_failed=%s current_banned=%s total_banned=%s\n' \
"$total_failed" "$current_banned" "$total_banned"
echo 'note: SSH probes can still generate auth/journal log entries before or while bans are applied; that alone does not mean fail2ban is inactive.'
else
echo 'fail2ban_sshd: no status available'
fi
else
echo 'fail2ban-client not available'
fi
section "OpenClaw OAuth"
if [[ -f "$OPENCLAW_AUTH_FILE" ]] && command -v jq >/dev/null 2>&1; then
openclaw_expiry_ms="$(jq -r '
[.profiles | to_entries[]
| select(.value.provider == "openai-codex")
| (.value.expires // 0)]
| max // 0
' "$OPENCLAW_AUTH_FILE" 2>/dev/null || echo "0")"
if [[ "$openclaw_expiry_ms" =~ ^[0-9]+$ ]] && (( openclaw_expiry_ms > 0 )); then
now_ms="$(( $(date +%s) * 1000 ))"
diff_ms="$(( openclaw_expiry_ms - now_ms ))"
expiry_iso="$(date -u -d "@$((openclaw_expiry_ms / 1000))" +'%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo 'n/a')"
if (( diff_ms <= 0 )); then
printf 'openclaw_codex_oauth: CRITICAL expires_utc=%s reason=expired action="%s"\n' \
"$expiry_iso" "$OPENCLAW_REAUTH_CMD"
else
hours_left="$(( diff_ms / 3600000 ))"
mins_left="$(( (diff_ms % 3600000) / 60000 ))"
severity='OK'
if (( hours_left <= OPENCLAW_CRITICAL_HOURS )); then
severity='CRITICAL'
elif (( hours_left <= OPENCLAW_WARN_HOURS )); then
severity='WARN'
fi
printf 'openclaw_codex_oauth: %s expires_utc=%s hours_left=%sh%sm critical_hours=%s warn_hours=%s action="%s"\n' \
"$severity" "$expiry_iso" "$hours_left" "$mins_left" "$OPENCLAW_CRITICAL_HOURS" "$OPENCLAW_WARN_HOURS" "$OPENCLAW_REAUTH_CMD"
fi
else
echo 'openclaw_codex_oauth: no oauth profile expiry found'
fi
else
echo 'openclaw_codex_oauth: auth file or jq unavailable'
fi
section "Top Journal Errors (last ${SINCE_HOURS}h)"
if command -v journalctl >/dev/null 2>&1; then
journalctl -p err..alert --since "${SINCE_HOURS} hours ago" --no-pager -o short-iso 2>/dev/null | tail -n "$TOP_ERROR_LINES" || true
else
echo 'journalctl not available'
fi
SCRIPT_PATH="$(readlink -f -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd -- "$(dirname -- "$SCRIPT_PATH")" && pwd)"
exec "$SCRIPT_DIR/ops-report" summary "$@"
n8n-observability/templates/ai-daily-report-prompt.md
+15
View File
@@ -23,6 +23,12 @@ Entregar un resumen breve para operación diaria sin ruido.
- Máximo 250 palabras.
- El estado general se decide por **impacto activo**, no por riesgos preventivos.
- `health_results` es opcional.
- Si `health_results` está presente, úsalo como fuente principal para disponibilidad actual.
- No declares un endpoint caído si `health_results` lo marca `up`, aunque haya errores antiguos en logs.
- Los errores de logs de una ventana amplia (por ejemplo 24h) son evidencia histórica; solo son incidente activo si aparecen como repetidos cerca de `generated_at_utc` o están correlacionados con `health_results` down/unhealthy/restarting.
- Si hay contradicción entre logs antiguos y health actual, reporta el punto como "resuelto/ruido histórico" o riesgo preventivo, no como incidente crítico.
- Si `log_digest` cubre una ventana amplia (por ejemplo 24h), no uses esos logs por sí solos para declarar "Servicios degradados con impacto".
- Para listar un servicio como degradado debe haber evidencia actual: `health_results` down, contenedor unhealthy/restarting, o una señal explícita actual en `server_summary`.
## Criterio estricto para "Estado general"
- `CRITICAL` solo si hay impacto activo severo:
@@ -35,6 +41,15 @@ Entregar un resumen breve para operación diaria sin ruido.
- latencia/errores sostenidos que afectan servicio, sin caída total.
- `OK` cuando no hay impacto activo, aunque existan riesgos preventivos.
## Reglas específicas de interpretación
- `302` hacia Authentik en servicios protegidos por SSO cuenta como disponible si `health_results` lo marca `up`.
- Un `404` en un dominio legacy o no listado por `health_results` no implica caída del servicio activo.
- Errores ACME antiguos no degradan si los dominios activos tienen certificados válidos y `health_results` está `up`.
- OAuth expirado en OpenClaw sí puede degradar OpenClaw, pero no debe atribuirse a Traefik/Auth.
- Si `server_summary` indica `openclaw_codex_oauth: OK`, no pidas reautenticación ni marques OpenClaw como degradado por errores OAuth antiguos del `log_digest`.
- Si `health_results` marca Traefik/Auth `up` y no hay errores actuales correlacionados, no marques Traefik/Auth como degradados por errores ACME/middleware antiguos.
- Si `server_summary` lista `non_running_sablier_managed_containers`, interprétalos como servicios bajo arranque bajo demanda. No los marques como incidente si `health_results` marca su endpoint asociado como `up`; si el endpoint está `down`, entonces sí trátalo como degradación activa.
## Importante
- **No marcar `DEGRADED` ni `CRITICAL` únicamente por prevención**:
- certificados que aún no expiraron,