今天我想分享一個(gè)有趣的項(xiàng)目叫做 “K8s ChatGPT Bot[1]”。該項(xiàng)目的目的是為 K8s 集群部署一個(gè) ChatGPT 機(jī)器人。我們可以詢問 ChatGPT 幫助我們解決 Prometheus 的警報(bào)，可以得到簡潔的答復(fù)，不再需要一人在黑暗中獨(dú)自解決警報(bào)問題！

我們需要用到 Robusta[2]，如果你還沒有 Robusta，可以參考《K8s — Robusta, K8s Troubleshooting Platform[3]》搭建一個(gè) Robusta 平臺(tái)。

下圖是 Robusta 平臺(tái)如何工作的截圖：

你可以在此處查看完整的演示視頻：

https://www.loom.com/share/964cd8735a874287a9155c77320bdcdb

運(yùn)行 K8s ChatGPT 機(jī)器人項(xiàng)目

該機(jī)器人項(xiàng)目是基于 Robusta.dev[4] 實(shí)現(xiàn)的，Robusta.dev 是一個(gè)用于響應(yīng) K8s 警報(bào)的開源平臺(tái)。其工作流程大致如下：

• Prometheus 使用 Webhook 接收器將警報(bào)轉(zhuǎn)發(fā)給 Robusta.dev 。
• Robusta.dev 詢問 ChatGPT 如何修復(fù) Prometheus 警報(bào)。

基于 Spring Boot + MyBatis Plus + Vue & Element 實(shí)現(xiàn)的后臺(tái)管理系統(tǒng) + 用戶小程序，支持 RBAC 動(dòng)態(tài)權(quán)限、多租戶、數(shù)據(jù)權(quán)限、工作流、三方登錄、支付、短信、商城等功能

• 項(xiàng)目地址：https://github.com/YunaiV/ruoyi-vue-pro
• 視頻教程：https://doc.iocoder.cn/video/

先決條件

• Slack
• Kubernetes 集群
• Python 3.7 及以上

基于 Spring Cloud Alibaba + Gateway + Nacos + RocketMQ + Vue & Element 實(shí)現(xiàn)的后臺(tái)管理系統(tǒng) + 用戶小程序，支持 RBAC 動(dòng)態(tài)權(quán)限、多租戶、數(shù)據(jù)權(quán)限、工作流、三方登錄、支付、短信、商城等功能

• 項(xiàng)目地址：https://github.com/YunaiV/yudao-cloud
• 視頻教程：https://doc.iocoder.cn/video/

如何安裝 Robusta

生成 Robusta 配置文件

為 Robusta 準(zhǔn)備 Python 虛擬環(huán)境。

$python3.10-mvenvrobusta
$sourcerobusta/bin/activate
(robusta)$pipinstall-Urobusta-cli--no-cache
Collectingrobusta-cli
Downloadingrobusta_cli-0.10.10-py3-none-any.whl(223kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━223.8/223.8kB30.0MB/seta0:00:00
Collectingpymsteams<0.2.0,>=0.1.16
Downloadingpymsteams-0.1.16.tar.gz(7.6kB)
Preparingmetadata(setup.py)...done
...
SuccessfullyinstalledPyJWT-2.4.0appdirs-1.4.4autopep8-2.0.1black-21.5b2
cachetools-5.2.1certifi-2022.12.7cffi-1.15.1charset-normalizer-3.0.1
...
ruamel.yaml.clib-0.2.7six-1.16.0slack-sdk-3.19.5tenacity-8.1.0
toml-0.10.2tomli-2.0.1typer-0.4.2typing-extensions-4.4.0urllib3-1.26.14
watchgod-0.7webexteamssdk-1.6.1websocket-client-1.3.3

使用 robusta 生成一個(gè)配置文件：

$robustagen-config
Robustareportsitsfindingstoexternaldestinations(wecallthem"sinks").
We'lldefinesomeofthemnow.

ConfigureSlackintegration?ThisisHIGHLYrecommended.[Y/n]:Y
Ifyourbrowserdoesnotautomaticallylaunch,openthebelowurl:
https://api.robusta.dev/integrations/slack?id=xxxx

配置 Slack 集成

使用瀏覽器打開網(wǎng)頁：https://api.robusta.dev/integrations/slack?id=xxxx

更新權(quán)限：

恭喜你配置 Slack 集成成功。

現(xiàn)在回到我們的 Terminal 終端，我們可以看到以下內(nèi)容，說明操作成功：

$robustagen-config
Robustareportsitsfindingstoexternaldestinations(wecallthem"sinks").
We'lldefinesomeofthemnow.

ConfigureSlackintegration?ThisisHIGHLYrecommended.[Y/n]:Y
Ifyourbrowserdoesnotautomaticallylaunch,openthebelowurl:
https://api.robusta.dev/integrations/slack?id=xxxx
You'vejustconnectedRobustatotheSlackof:devopsfans
WhichslackchannelshouldIsendnotificationsto?#k8s-chatgpt-bot
ConfigureRobustaUIsink?ThisisHIGHLYrecommended.[Y/n]:Y
EnteryourGmail/Googleaddress.Thiswillbeusedtologin:xxx@gmail.com
Chooseyouraccountname(e.gyourorganizationname):devopsfans
Successfullyregistered.

RobustacanusePrometheusasanalertsource.
Ifyouhaven'tinstalledityet,Robustacaninstalla
pre-configuredPrometheus.
Wouldyouliketodoso?[y/N]:y
PleasereadandapproveourEndUserLicenseAgreement:
https://api.robusta.dev/eula.html
DoyouacceptourEndUserLicenseAgreement?[y/N]:y
Lastquestion!WouldyouliketohelpusimproveRobustabysendingexceptionreports?[y/N]:N
Savedconfigurationto./generated_values.yaml-savethisfileforfutureuse!
FinishinstallingwithHelm(seetheRobustadocs).
ThenlogintoRobustaUIathttps://platform.robusta.dev

Bytheway,we'llsendyousomemessageslatertogetfeedback.
(Wedon'tstoreyourAPIkey,sowescheduledfuturemessagesusingSlack's
API)

在 slack channel 中，我們還可以看到：

使用 Helm3 安裝 Robusta

安裝和更新 robusta 倉庫。

$helmrepoaddrobustahttps://robusta-charts.storage.googleapis.com&&helmrepoupdate
"robusta"hasbeenaddedtoyourrepositories
Hangtightwhilewegrabthelatestfromyourchartrepositories...
...Successfullygotanupdatefromthe"kedacore"chartrepository
...Successfullygotanupdatefromthe"robusta"chartrepository
...Successfullygotanupdatefromthe"grafana"chartrepository
...Successfullygotanupdatefromthe"prometheus-community"chartrepository
...Successfullygotanupdatefromthe"stable"chartrepository
UpdateComplete.?HappyHelming!?

更新 generated_values.yaml 文件

使用以下內(nèi)容更新 generated_values.yaml 文件：

playbookRepos:
chatgpt_robusta_actions:
url:"https://github.com/robusta-dev/kubernetes-chatgpt-bot.git"

customPlaybooks:
#Addthe'AskChatGPT'buttontoallPrometheusalerts
-triggers:
-on_prometheus_alert:{}
actions:
-chat_gpt_enricher:{}

globalConfig:
chat_gpt_token:YOURKEYGOESHERE

將 Robusta 部署到 K8s

$helminstallrobustarobusta/robusta-f./generated_values.yaml
--setclusterName=dev-cluster

驗(yàn)證兩個(gè) Robusta pod 正常運(yùn)行且在日志中沒有發(fā)現(xiàn)錯(cuò)誤日志：

$kubectlgetpods-A|greprobusta
defaultalertmanager-robusta-kube-prometheus-st-alertmanager-02/2Running1(4m19sago)9m25s
defaultprometheus-robusta-kube-prometheus-st-prometheus-02/2Running09m25s
defaultrobusta-forwarder-6b7d8d9d88-2rv9d1/1Running09m29s
defaultrobusta-grafana-64944bfcdc-v97xh3/3Running09m29s
defaultrobusta-kube-prometheus-st-admission-patch-6zj4b0/1Completed09m28s
defaultrobusta-kube-prometheus-st-operator-7b985d7fb-c9f9t1/1Running09m29s
defaultrobusta-kube-state-metrics-688d794968-ll6gf1/1Running09m29s
defaultrobusta-prometheus-node-exporter-2k5f71/1Running05m24s
defaultrobusta-prometheus-node-exporter-zxsrg1/1Running09m29s
defaultrobusta-runner-5868b494d6-m62921/1Running09m29s

$robustalogs
settingupcoloredlogging
2023-01-1422:57:01.428INFOloggerinitializedusingINFOloglevel
2023-01-1422:57:01.429INFOCreatinghikarumonkeypatches
2023-01-1422:57:01.429INFOCreatingyamlmonkeypatch
2023-01-1422:57:01.429INFOCreatingkubernetesContainerImagemonkeypatch
2023-01-1422:57:01.430INFOwatchingdir/etc/robusta/playbooks/forcustomplaybookschanges
2023-01-1422:57:01.431INFOwatchingdir/etc/robusta/config/active_playbooks.yamlforcustomplaybookschanges
2023-01-1422:57:01.431INFOReloadingplaybookpackagesduetochangeoninitialization
2023-01-1422:57:01.431INFOloadingconfig/etc/robusta/config/active_playbooks.yaml
2023-01-1422:57:01.467INFONocustomplaybooksdefinedat/etc/robusta/playbooks/storage
2023-01-1422:57:01.468INFOCloninggitrepohttps://github.com/robusta-dev/kubernetes-chatgpt-bot.git.reponamekubernetes-chatgpt-bot
...
2023-01-1422:57:07.364INFOconnectingtoserverasaccount_id=8302df56-c554-4129-8b95-d143d1f2e3a2;cluster_name=dev-cluster
2023-01-1422:57:07.977INFOInitializingservicescache
2023-01-1422:57:08.203INFOInitializingnodescache
2023-01-1422:57:08.395INFOInitializingjobscache
2023-01-1422:57:08.603INFOGettingeventshistory
2023-01-1422:57:10.403INFOClusterhistoricaldatasent.
2023-01-1423:04:43.681INFOclusterstatus{'account_id':'8302df56-c554-4129-8b95-d143d1f2e3a2','cluster_id':'dev-cluster','version':'0.10.10','last_alert_at':'2023-01-142318.959377','light_actions':['related_pods','prometheus_enricher','add_silence','delete_pod','delete_silence','get_silences','logs_enricher','pod_events_enricher','deployment_events_enricher','job_events_enricher','job_pod_enricher','get_resource_yaml','node_cpu_enricher','node_disk_analyzer','node_running_pods_enricher','node_allocatable_resources_enricher','node_status_enricher','node_graph_enricher','oomkilled_container_graph_enricher','pod_oom_killer_enricher','oom_killer_enricher','volume_analysis','python_profiler','pod_ps','python_memory','debugger_stack_trace','python_process_inspector','prometheus_alert','create_pvc_snapshot'],'updated_at':'now()'}

使用 Robusta

現(xiàn)在，我們終于可以使用 Robusta 了！默認(rèn)情況下，Robusta 在當(dāng) K8s Pod 崩潰時(shí)發(fā)送通知。

因此，讓我們創(chuàng)建一個(gè)崩潰的 Pod：

$kubectlapply-fhttps://gist.githubusercontent.com/robusta-lab/283609047306dc1f05cf59806ade30b6/raw
deployment.apps/crashpodcreated

$kubectlgetpods-A|grepcrash
defaultcrashpod-64db77b594-cgz4s0/1CrashLoopBackOff2(21sago)36s

一旦 Pod 達(dá)到兩次重啟，在 Slack channel 我們就將收到有關(guān) Pod 崩潰的消息，如下所示：

與 ChatGPT 互動(dòng)

經(jīng)過我們的實(shí)驗(yàn)，我們已經(jīng)確認(rèn) Robusta 已與我們的 Slack 和 K8s 集群集成。接下來讓我們與 ChatGPT 機(jī)器人進(jìn)行交互！

立即觸發(fā) Prometheus 警報(bào)，跳過正常延遲：

$robustaplaybookstriggerprometheus_alertalert_name=KubePodCrashLoopingnamespace=defaultpod_name=example-pod
======================================================================
Triggeringaction...
======================================================================
runningcmd:curl-XPOSThttp://localhost:5000/api/trigger-H'Content-Type:application/json'-d
'{"action_name":"prometheus_alert","action_params":
{"alert_name":"KubePodCrashLooping","namespace":"default",
"pod_name":"example-pod"}}'
{"success":true}

======================================================================
Fetchinglogs...
======================================================================
2023-01-1423:14:33.463INFOErrorloadingkubernetespoddefault/example-pod.reason:NotFoundstatus:404
2023-01-1423:14:33.481INFOErrorloadingkubernetespoddefault/example-pod.reason:NotFoundstatus:404
2023-01-1423:14:33.503INFOErrorloadingkubernetespoddefault/example-pod.reason:NotFoundstatus:404
2023-01-1423:14:33.505ERRORcannotrunpod_events_enricheronalertwithnopodobject:PrometheusKubernetesAlert(sink_findings=defaultdict(<class'list'>,{'main_slack_sink':[0x7fab53074e20>],'main_ms_teams_sink':[0x7fab53074700>],'robusta_ui_sink':[0x7fab40773a30>]}),named_sinks=['main_slack_sink','main_ms_teams_sink','robusta_ui_sink'],response={'success':True},stop_processing=False,_scheduler=0x7fab4088e0a0>,_context=ExecutionContext(account_id='8302df56-c554-4129-8b95-d143d1f2e3a2',cluster_name='dev-cluster'),obj=None,alert=PrometheusAlert(endsAt=datetime.datetime(2023,1,14,23,14,33,430401),generatorURL='',startsAt=datetime.datetime(2023,1,14,23,14,33,430406),fingerprint='',status='firing',labels={'severity':'error','namespace':'default','alertname':'KubePodCrashLooping','pod':'example-pod'},annotations={}),alert_name='KubePodCrashLooping',alert_severity='error',label_namespace='default',node=None,pod=None,deployment=None,job=None,daemonset=None,statefulset=None)
2023-01-1423:14:33.524INFOErrorloadingkubernetespoddefault/example-pod.reason:NotFoundstatus:404
2023-01-1423:14:33.696ERRORCallbackBlocknotsupportedformsteams
2023-01-1423:14:33.697ERRORerrorsendingmessagetomsteams
e=InvalidURL'False':Noschemasupplied.Perhapsyoumeanthttp://False?

======================================================================
Done!
======================================================================

現(xiàn)在切換到 Slack，我們將看到一個(gè)新警報(bào)，并且這次帶有 “Ask ChatGPT” 的按鈕！

就是這樣！恭喜，我們剛剛成功安裝了我們的第一個(gè) K8s ChatGPT 機(jī)器人！

示例 2：節(jié)點(diǎn)容量為 100%

下面是一個(gè)節(jié)點(diǎn)達(dá)到 100% 容量的例子：

Robusta UI

Robusta 有一個(gè)用于集成的 UI，也有一個(gè)預(yù)配置的 Promethus 系統(tǒng)，如果你還沒有自己的 K8s 集群，并且想嘗試一下這個(gè) ChatGPT 機(jī)器人，你可以使用 Robusta 現(xiàn)有的！

結(jié)論

我們花了一些時(shí)間終于把 K8s + ChatGPT 平臺(tái)搭建起來了。這是一個(gè)由愛好者們共同組建的項(xiàng)目，并且此項(xiàng)目具有很大的潛力。我希望你喜歡這篇文章。

如果你還沒有自己的 K8s 集群和 Prometheus 監(jiān)控系統(tǒng)，你可以使用 Robusta 預(yù)配置的 Promethus 監(jiān)控系統(tǒng)。

審核編輯 ：李倩