AI-Powered Infrastructure Monitoring

A Saturday Afternoon Conversation

Martin Dobberstein
AI-Assisted Coding Community · March 2026

It Started With This

That's it. That was the entire brief.

What came back: 7 Architectural Decision Records, a full implementation plan, and a working system — in one afternoon.

The Process

7 Decisions — All Reviewed

The AI wrote 7 Architectural Decision Records before writing any code. I reviewed every one.

ADR #4: Hybrid Alerting

The most important decision

ADR #7: Access — When "Secure" Breaks the Use Case

The Architecture

    flowchart TB
        subgraph Internet[" "]
            User["👤 Browser"]
            AI["🤖 OpenClaw AI"]
        end
        
        subgraph Ingress["☁️ Ingress — TLS + Basic Auth"]
            GrafanaURL["grafana.gutsch.it"]
            PromURL["prometheus.gutsch.it"]
        end
        
        subgraph Cluster["⎈ Kubernetes Cluster"]
            subgraph Monitoring["Monitoring Stack"]
                Prometheus[("📊 Prometheus")]
                Loki[("📜 Loki")]
                Grafana["📈 Grafana"]
                AlertManager["🚨 AlertManager"]
            end
            
            subgraph Collection["Collection Layer — DaemonSets"]
                Promtail["Promtail"]
                NodeExp["node-exporter"]
                KSM["kube-state-metrics"]
            end
            
            subgraph Apps["Applications"]
                CB["circleback-webhook\n/metrics"]
            end
        end
        
        User --> GrafanaURL --> Grafana
        AI --> PromURL --> Prometheus
        
        Prometheus -.->|scrape| NodeExp
        Prometheus -.->|scrape| KSM
        Prometheus -.->|scrape| CB
        Prometheus -->|rules| AlertManager
        
        Promtail -->|push| Loki
        Grafana -->|query| Prometheus
        Grafana -->|query| Loki
    

What Actually Got Built

10+ components wired together. ~45 min of my time. ~€6/month ongoing.

The Two-Model Insight

Use the right model for the job. Cheap for routine, smart for analysis.

What It Cost

Compare to: Datadog, New Relic, or your own time writing YAML.

It Actually Works

First automated check ran at 19:00, one hour after setup:

System monitoring itself. No human in the loop.

Fast Forward: Two Weeks Later

That was the build. But the real question is:

Does it actually help?

🔴 Live Demo

The Setup

🔴 Demo: Node Goes Down

Let's kill a worker node. For real — via Hetzner API:

🔴 The Alert

What arrives on Telegram

Compare this to a PagerDuty alert: FIRING: NodeDown worker-node-2 [critical]

Dashboard vs. AI Summary

Which one do you want at 3 AM?

What This Could Mean at Work

Try This Yourself

What made it work:

1. Real problem, real constraints — budget, preferences, existing infra
2. Review and challenge — don't just accept; push back
3. Use the pauses — AI thinking time is your time
4. Trust the boring parts — YAML, configs, boilerplate
5. Own the decisions — architecture is still your job

What's Next

For this project:

• Tune thresholds based on real alerts
• Add more applications to monitoring
• Log analysis — AI reads Loki, not just Prometheus

For you to explore:

• OpenClaw — the framework used here (open source)
• PromEx — Elixir metrics made easy
• Kustomize — explicit k8s deployments
• The ADR pattern — document decisions, not just code

The tools are ready. The patterns work. Go build something.

Questions?

Session log: ~/clawd/projects/infra-monitoring/SESSION-LOG.md
Live system: https://grafana.gutsch.it
This deck: https://openclaw-presentation.gutsch.it/observability

Thank You

Built during a Saturday afternoon of chores
with Claude (via OpenClaw)

Martin Dobberstein
@gutschilla