> ## Documentation Index
> Fetch the complete documentation index at: https://blog.elevate.do/llms.txt
> Use this file to discover all available pages before exploring further.
# Troubleshooting Kubernetes Autonomously with k8x
> How GenAI agents can help with multi-step cluster diagnostics on Kubernetes
> This article was co-authored with [Shankar Ganeshan](https://www.linkedin.com/in/shankar-ganesan-653226a/).
When you're setting up a Kubernetes cluster, DevOps and Platform engineers feel like they're navigating a maze. Take a simple service deployment: you might have to check the deployment (`kubectl get deployment`), the service (`kubectl get service`), inspect events (`kubectl describe pod`), the logs (`kubectl logs pod-name`), and even Ingress rules (`kubectl get ingress`). Each step requires context, copy-pasting names and ids from one place into another. A single typo can leave you scratching your head. Worse, incomplete information can lead you down the wrong path and waste hours. The complexity of multi-step diagnostics, especially when things go wrong, can be overwhelming.
## Why Kubernetes troubleshooting still hurts and why we built `k8x`.
Even veteran SREs know the familiar pain cycle:
| Symptom | Usual manual steps |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pods in `CrashLoopBackOff` | • `kubectl get pods -A` → copy failing names
• `kubectl logs …` (repeat per pod & container)
• grep for hints, cross-check ConfigMaps |
| 503s on an Ingress | • Check Service endpoints → kube-proxy rules → NetworkPolicy
• Compare readiness probes & resource pressure |
| "`myservice` works on *most* nodes" | • Inspect taints/labels, daemonsets, CNI logs
• Describe nodes for kernel versions & allocatable |
Each scenario is *multi-step*, context-heavy, and needs you to track context across multiple kubernetes commands.
You need to keep a note of resource ids to use them in other resource descriptions and logs to know what's going on.
## The agentic leap: from *suggestion* to *orchestrated review*
Recent agents like **GitHub Copilot Chat** (in VS Code w/ terminal access), **Claude Code** (terminal-native edits) and Goose showed a new pattern: the LLM *drives* an interactive loop—executes safe commands autonomously, then narrates the findings.
General-purpose LLM helpers (ChatGPT, Claude, Copilot Chat) can go beyond **suggesting** commands - they can copy-paste, re-run, and stitch results together.
*k8x* applies this agentic idea to Kubernetes:
* **Natural-language prompts** → e.g. "Find pods that aren't ready and tell me why."
* The agent **plans** a sequence: `kubectl get …`, `kubectl describe …`, maybe `kubectl top …`.
* It **executes** those *read-only* commands, parses output, and **reasons** about root causes.
* Results appear as an **explanation first**, with raw command logs one keystroke away.
Unlike code-centric tools, *k8x* is infra-native. It understands resource kinds, status fields, events, and failure taxonomies (image pull, scheduling, OOMKilled).
## Design choices that matter to operators
`k8x` works in your console with the your current `kubectl` configuration, to perform autonomous, multi-step workflows to detect and troubleshoot kubernetes issues with your credentials.
| Principle | Experience | Why it builds trust |
| ---------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------- |
| **Read-only by default (v0.1)** | Zero risk of deletions; mirrors commands you could type | Safely trial AI before granting mutate rights (\[GitHub]\[3]) |
| **`Plain kubectl under the hood`** | Familiar audit trail; works anywhere your kubeconfig does | No proprietary sidecars or admission webhooks |
| **Multi-LLM back-end** | Select OpenAI, Claude, or Gemini at `k8x configure` | Avoid vendor lock-in; keep traffic in-house |
| **Command history & undo** | `k8x history list` shows past sessions | Auditors see exactly what ran; SREs replay in staging |
There's more to come, including write permissions, parallelism, etc. Conrtributions are welcome.
## A day in the life with *k8x*
```bash theme={null}
# 1️⃣ Something is off
default$ k8x -c "my checkout service is returning 502s"
# 2️⃣ Agent plan (condensed)
• Check Ingress status
• Verify Service endpoints
• Scan pod readiness & logs
• Examine recent HPA events
# 3️⃣ Summary
❌ 2/5 endpoints unhealthy
↳ Pods stuck in Init:CrashLoopBackOff (db-migrations)
↳ Migration container fails on `ALTER TABLE …` (lock timeout)
Suggested fix: run `kubectl exec` into db-migration-pod or scale replicas to 0/1 to release lock
```
In \~30 seconds, you get an actionable story instead of fifteen manual commands.
## How a *multi-step* review actually works
1. **Intent parsing** - Translates English prompts into an internal diagnostic goal.
2. **Planning** - LLM selects a **safe** chain of read-only `kubectl` queries.
3. **Adaptive execution** - After each command, it decides if deeper queries are needed.
4. **Reasoning & templated explanations** - Maps results to known issue patterns for a deterministic, auditable summary.
Only redacted command output reaches the LLM—no full application logs—another trust measure.
## Where *k8x* stands in the AI-ops landscape
| Tool | Domain | Autonomy | Local-first? | Write access |
| ----------------------- | --------------------- | ---------------------------------- | ------------ | ------------------- |
| **GitHub Copilot Chat** | Code / CI | Suggests fixes, runs queries in UI | No | Optional PR commits |
| **Claude Code** | Code & CLI automation | Plans & edits files | Yes | Yes (file edits) |
| **Goose** | Multi-agent dev tasks | Runs terminal commands | Yes | Yes |
| **k8x (v0.1)** | Kubernetes operations | Plans & executes `kubectl` reads | Yes | No (read-only) |
*k8x* fills the gap for platform and DevOps engineers looking for Copilot-level assistance after deployment, not just in CI/CD.
## Getting started in 60 seconds
```bash theme={null}
brew tap aihero/k8x
brew install k8x # installs v0.1.1
k8x configure # choose LLM & set API key
k8x -c "Are all pods running?"
```
You'll never look at a 3-screen tmux layout the same way again.
## Open Source
*k8x* is Apache 2.0-licensed and available on [GitHub](https://github.com/shankarg87/k8x).
We're looking for contributors to help build out the next features, including:
* **v0.2 - Declarative fixes**
* Generate a patch plan (`kubectl diff`) and let humans `--approve`.
* **Support ArgoCD and other k8s tools**
* Integrate with ArgoCD for GitOps workflows.
* Use `kubectl apply` to update resources based on agent suggestions.
* **Terraform & cloud-CLI mode**
* Run `terraform plan` or `aws eks update-kubeconfig` as sub-steps.
* **Cluster runbooks as code**
* Store successful sessions as YAML recipes to auto-trigger on alerts.
## Final thoughts
Generative-AI agents are moving from IDEs into production infrastructure. By combining **LLM planning**, **policy-guarded execution**, and **domain-specific reasoning**, *k8x* transforms Kubernetes troubleshooting from a scavenger hunt into a guided review. Start with read-only diagnostics today; when you're ready, the agent will apply fixes—one audited pull request at a time.
Share this article with your network
Tweet this article to your followers
Let's discuss AI and potential opportunities
Learn more about our work in AI infrastructure and workplace transformation
Connect with me for more insights on AI
Follow for real-time thoughts on AI agents and infrastructure