/loki-error-triage

Pull error-level lines from Loki for the portal namespaces, cluster them into defect signatures, dedupe against open GitHub issues, and file what's real — with counts, samples, and a fix direction.

• Instructions: # /loki-error-triage — errors in Loki become triaged GitHub issues The cluster runs Loki + Grafana + Prometheus + Alertmanager in the `monitoring` namespace (Loki at `http://loki.monitoring:3100`, in-cluster only — the cluster is private, so every query runs from inside via `az aks command invoke`). ## 1. Query (escape-safe — this exact shape is validated) LogQL through four quoting layers mangles backslashes; use character classes instead: ``` kubectl run lokiq --image=curlimages/curl --restart=Never -q --command -- sh -c ' start=\((( (\)(date +%s) - 21600) * 1000000000 )); for ns in memex memex-cloud; do echo "###NS $ns"; curl -sG http://loki.monitoring:3100/loki/api/v1/query\_range \ --data-urlencode "query={namespace=\"$ns\"} |~ \"[[]Error[]]|fail: |Unhandled exception\"" \ --data-urlencode start=\(start --data-urlencode limit=120 --data-urlencode direction=backward; done' kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/lokiq --timeout=90s; kubectl logs lokiq; kubectl delete pod lokiq --now \`\`\` 🚨 \`start\` is a \*\*Unix nanosecond\*\* timestamp and it must be COMPUTED in the pod's shell — an undefined variable makes \`\)((now-21600))` expand to `-21600`, which Loki accepts and answers from the beginning of retention. The window then silently is not the one you named, and every rate you derive from it is wrong. Echo the value back and sanity-check its magnitude (19 digits today). ## 2. Cluster to signatures Group by logger category + first message line, with ids/hashes/numbers normalised out (`[0-9a-f]{8,}` → `#`). The .NET console format puts the MESSAGE on the line after `fail: Category[0]` — Loki ships lines separately, so for message text and TRUE counts go to the pod logs: 🚨 **`logs deploy/…` reads ONE pod.** These deployments run 2–8 replicas, so a single-pod read undercounts the namespace by whatever the scale is — and this step's whole output is labelled a TRUE count. Read EVERY replica: ``` sel=$(kubectl -n {ns} get deploy meme…
• Auto mount: yes
• Launches sub thread: no