Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add service debugging scenario #714

Merged
merged 7 commits into from
Sep 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 177 additions & 0 deletions examples/service-debugging/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# BotKube Use Case: Collaboratively debug Kubernetes resources via Slack

This example showcases debugging a failing Pod with a network issue. You will learn:

- how BotKube notifies you about errors
- how to execute `kubectl` commands via `@Botkube`
- how the `kubectl` permission restriction works

## Prerequisites

Install the following applications:

- [k3d](https://k3d.io/v5.4.6/)
- [Helm](https://helm.sh/)
- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)

### Set up local Kubernetes cluster

Run the command:

```bash
k3d cluster create svc-debug
```

### Deploy BotKube

1. Install [BotKube in your Slack workspace](https://botkube.io/installation/slack/#install-botkube-slack-app-to-your-slack-workspace).

2. Export required environment variables:

```bash
export SLACK_BOT_TOKEN="{token}"
export TEAM_SLACK_CHANNEL="{channel}" # e.g. gophers
export ADMIN_SLACK_CHANNEL="{channel}" # e.g. admin
```

> **Note**
> The channels must exist and the BotKube bot must be added to them.

3. Add BotKube Helm chart:

```bash
helm repo add botkube https://charts.botkube.io
helm repo update botkube
```

4. Deploy BotKube:

```bash
helm install botkube --version v0.13.0 --namespace botkube --create-namespace \
-f ./examples/service-debugging/botkube-values.yaml \
--set communications.default-group.slack.token=${SLACK_BOT_TOKEN} \
--set communications.default-group.slack.channels.default.name=${TEAM_SLACK_CHANNEL} \
--set communications.default-group.slack.channels.admin.name=${ADMIN_SLACK_CHANNEL} \
--wait \
botkube/botkube
```

### Deploy example app

Run the command:

```bash
kubectl apply -f ./examples/service-debugging/deploy
```

## Scenario

In this scenario, we will learn how to react to the error event sent on Slack channel.

1. Open the team Slack channel.

2. You should see new events sent by BotKube about the created service:

![](assets/create-events.png)

3. After a minute, you should see an error event:

![](assets/error-event.png)

Let's start the investigation! 🕵️

4. For now, we don't know too much about the error itself. To learn more, let's check the `meme` Pod logs:

```
@Botkube logs -l app=meme
```

5. From the logs, we learned that the `meme` Pod cannot call the `quote` Pod using the defined Service URL. To be able to call the `quote` Pod, we definitely need to have the `quote` Service defined. To get all Services, run:

```
@Botkube get services
```

6. We can see that the `quote` Service is there, so we need to dig deeper into the configuration. Let's describe the Service to check if there are any endpoints:

```
@Botkube describe svc quote
```

Now it gets interesting: there are no endpoints, which means there isn't a single Pod that is matched by the Service selectors.

7. We need to check whether the `quote` Pod is up and running:

```
@Botkube get pods
```

💡 The `quote` Pod is up and running, so it might be a problem with incorrect labels.

8. There is a handy `--show-lables` flag which allows us to check that easily:

```
@Botkube get po {quote_pod_name} --show-labels
```

🎉 We got it! The bug was found. The problem is the incorrect labels. The `quote` Service only matches Pods with the `app=quote` label, this is missing from our Pod!

9. Add the missing label to the `quote` Pod:

```
@Botkube label pod {quote_pod_name} app=quote
```

If you execute that command in the team channel, you will get an error. This is yet another BotKube feature, it allows you to define executor permissions per channel.

10. To be able to run the label command, you must switch to the admin channel.

11. Once again, try to add a label to the `quote` Pod:

```
@Botkube label pod {quote_pod_name} app=quote
```

12. Restart the `meme` Pod:

```
@Botkube delete po -l app=meme
```
> **Note**
> The `quote` Pod is managed by Deployment, so once we delete the Pod, it will be automatically recreated.

13. Run `logs` to confirm that `http://quote/quote` is reachable now:

```
@Botkube logs -l app=meme
```

14. **(Optional)** From your terminal, forward the `meme` Service to your localhost:

```bash
kubectl port-forward svc/meme 9090:80
```

Navigate to [http://localhost:9090/meme](http://localhost:9090/meme).

Hurray! We are up and running again! 🥳

### Summary

During the short demo, you'll notice that:

- You don't need to install and configure any tools locally
- You don't need to repeat commands that were already executed by others
- and you don't need to discover the same thing by your own when it was already discussed by your teammates
- You don't need to switch context - between Slack and your terminal
- You can define different `kubectl` permissions per channel

If you are very observant, you probably noticed that events were sent only to the team channel and not to the admin one. This is possible with a single BotKube instance. To learn more, check [Source and Executors Bindings](https://botkube.io/configuration/communication/#source-and-executor-bindings).

## Cleanup

Remove the `svc-debug` cluster:

```bash
k3d cluster delete svc-debug
```
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
103 changes: 103 additions & 0 deletions examples/service-debugging/botkube-values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
sources:
'k8s-events':
kubernetes:
recommendations:
pod:
noLatestImageTag: false
labelsSet: false
ingress:
backendServiceValid: false
tlsSecretValid: false

resources:
- name: v1/pods
namespaces:
include:
- ".*"
events:
- create
- delete
- error
- name: v1/services
namespaces:
include:
- ".*"
events:
- create
- delete
- error

executors:
'kubectl-read-only':
kubectl:
namespaces:
include:
- ".*"
enabled: true
commands:
verbs: ["describe", "get", "logs"]
resources: ["pods","services"]
defaultNamespace: default
restrictAccess: false
'kubectl-admin':
kubectl:
namespaces:
include:
- ".*"
enabled: true
commands:
verbs: ["label", "delete"]
resources: ["deployments", "pods"]
defaultNamespace: default
restrictAccess: false

communications:
'default-group':
slack:
enabled: true
channels:
'team':
name: 'gophers'
bindings:
executors:
- kubectl-read-only
sources:
- k8s-events
'admin':
name: 'admin'
bindings:
executors:
- kubectl-read-only
- kubectl-admin

token: 'SLACK_API_TOKEN'
notification:
type: short

settings:
clusterName: gke-stage
configWatcher: false
upgradeNotifier: false

# -- Role Based Access for BotKube Pod.
# [Ref doc](https://kubernetes.io/docs/admin/authorization/rbac/).
rbac:
create: true
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["get", "watch", "list"]

# to be able to run `kubectl label pod`
- apiGroups: [""]
resources: ["pods"]
verbs: ["patch"]

# to be able to run `kubectl delete pod`
- apiGroups: [""]
resources: ["pods"]
verbs: ["delete"]

serviceAccount:
create: true

38 changes: 38 additions & 0 deletions examples/service-debugging/deploy-final/meme.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: meme
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: meme
template:
metadata:
labels:
app: meme
spec:
containers:
- name: meme
image: ghcr.io/kubeshop/botkube/examples/meme-service:v1.1.0
ports:
- containerPort: 9090
env:
- name: QUOTE_URL
value: "http://quote/"
---
kind: Service
apiVersion: v1
metadata:
name: meme
labels:
app: meme
spec:
selector:
app: meme
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 9090
36 changes: 36 additions & 0 deletions examples/service-debugging/deploy-final/quote.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: quote
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: quote
template:
metadata:
labels:
app: quote
spec:
containers:
- name: quote
image: ghcr.io/kubeshop/botkube/examples/quote-service:v1.1.0
ports:
- containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
name: quote
labels:
app: quote
spec:
selector:
app: quote
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
38 changes: 38 additions & 0 deletions examples/service-debugging/deploy/meme.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: meme
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: meme
template:
metadata:
labels:
app: meme
spec:
containers:
- name: meme
image: ghcr.io/kubeshop/botkube/examples/meme-service:v1.1.0
ports:
- containerPort: 9090
env:
- name: QUOTE_URL
value: "http://quote/"
---
kind: Service
apiVersion: v1
metadata:
name: meme
labels:
app: meme
spec:
selector:
app: meme
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 9090
Loading