今天偶然間發現了一個 kubectl 插件 kubectl foreach? ，可以在多個集群（contexts?）上執行 kubectl 命令。比如 kubectl foreach cluster-1 cluster-2 -- get po -n kube-system 。
上周在寫 K8s 多集群的流量調度 的 demo 部分時需要不停地在多個集群中安裝組件、部署應用，或者執行各種命令。當時是通過 Linux shell 腳本并通過工具 kubectx 進行集群的切換，像這樣：
或者這樣：
操作繁瑣，很是痛苦。
今天偶然間發現了一個 kubectl 插件 kubectl foreach? ，可以在多個集群（contexts?）上執行 kubectl 命令。比如 kubectl foreach cluster-1 cluster-2 -- get po -n kube-system 。
插件安裝和使用很簡單，通過 krew 進行安裝：
kubectl krew install foreach
使用也很簡單：
kubectl foreach -h
Usage:
kubectl foreach [OPTIONS] [PATTERN]... -- [KUBECTL_ARGS...]
Patterns can be used to match context names from kubeconfig:
(empty): matches all contexts
NAME: matches context with exact name
/PATTERN/: matches context with regular expression
^NAME: remove context with exact name from the matched results
^/PATTERN/: remove contexts matching the regular expression from the results
Options:
-c=NUM Limit parallel executions (default: 0, unlimited)
-I=VAL Replace VAL occurring in KUBECTL_ARGS with context name
-q Disable and accept confirmation prompts ($KUBECTL_FOREACH_DISABLE_PROMPTS)
-h/--help Print help
Examples:
# get nodes on contexts named a b c
kubectl foreach a b c -- get nodes
# get nodes on all contexts named c0..9 except c1 (note the escaping)
kubectl foreach '/^c[0-9]/' ^c1 -- get nodes
# get nodes on all contexts that has "prod" but not "foo"
kubectl foreach /prod/ ^/foo/ -- get nodes
# use 'kubectl tail' plugin to follow logs of pods in contexts named *test*
kubectl foreach -I _ /test/ -- tail --cnotallow=_ -l app=foo
接下來測試下，使用 k3d 創建 3 個集群 （k3d 貌似不支持同時創建多個集群，還是需要 for 腳本來操作）：
for CLUSTER_NAME in cluster-1 cluster-2 cluster-3
do
k3d cluster create ${CLUSTER_NAME} \
--image docker.io/rancher/k3s:v1.23.8-k3s2 \
--servers-memory 4g \
--k3s-arg "--disable=traefik@server:0" \
--no-lb \
--timeout 120s \
--wait
done
集群安裝完成：
k3d cluster list
NAME SERVERS AGENTS LOADBALANCER
cluster-1 1/1 0/0 false
cluster-2 1/1 0/0 false
cluster-3 1/1 0/0 false
注意，k3d 安裝的集群的 context 都帶有前綴 k3d-? ，在使用 kubectl foreach 的時候要注意：
kubectx
k3d-cluster-1
k3d-cluster-2
k3d-cluster-3
比如查看各個集群中的 kube-sysmte 下的 pod：
kubectl foreach -q k3d-cluster-1 k3d-cluster-2 k3d-cluster-3 -- get po -n kube-system
或者試試創建 deployment，這次我們不列出完整的 context name，而是使用正則 /cluster/：
kubectl foreach -q /cluster/ -- create deploy pipy --image flomesh/pipy -n default
Will run command in context(s):
- k3d-cluster-1
- k3d-cluster-2
- k3d-cluster-3
k3d-cluster-1 | deployment.apps/pipy created
k3d-cluster-3 | deployment.apps/pipy created
k3d-cluster-2 | deployment.apps/pipy created
然后查看下 pod：
kubectl foreach -q /cluster/ -- get pod -n default
Will run command in context(s):
- k3d-cluster-1
- k3d-cluster-2
- k3d-cluster-3
k3d-cluster-1 | NAME READY STATUS RESTARTS AGE
k3d-cluster-1 | pipy-df659b55f-bnr27 1/1 Running 0 25s
k3d-cluster-3 | NAME READY STATUS RESTARTS AGE
k3d-cluster-3 | pipy-df659b55f-p9j49 1/1 Running 0 25s
k3d-cluster-2 | NAME READY STATUS RESTARTS AGE
k3d-cluster-2 | pipy-df659b55f-9bjgf 1/1 Running 0 25s
查看日志：
kubectl foreach -q /cluster/ -- logs -l app=pipy -n default --tail 3
Will run command in context(s):
- k3d-cluster-1
- k3d-cluster-2
- k3d-cluster-3
k3d-cluster-2 | 2022-11-30 10:40:56.520 [INF] [listener] Listening on TCP port 8080 at 0.0.0.0
k3d-cluster-2 | 2022-11-30 10:40:56.520 [INF] [listener] Listening on TCP port 8081 at 0.0.0.0
k3d-cluster-2 | 2022-11-30 10:40:56.520 [INF] [listener] Listening on TCP port 8082 at 0.0.0.0
k3d-cluster-1 | 2022-11-30 10:40:56.551 [INF] [listener] Listening on TCP port 8080 at 0.0.0.0
k3d-cluster-1 | 2022-11-30 10:40:56.551 [INF] [listener] Listening on TCP port 8081 at 0.0.0.0
k3d-cluster-1 | 2022-11-30 10:40:56.551 [INF] [listener] Listening on TCP port 8082 at 0.0.0.0
k3d-cluster-3 | 2022-11-30 10:40:55.813 [INF] [listener] Listening on TCP port 8080 at 0.0.0.0
k3d-cluster-3 | 2022-11-30 10:40:55.813 [INF] [listener] Listening on TCP port 8081 at 0.0.0.0
k3d-cluster-3 | 2022-11-30 10:40:55.813 [INF] [listener] Listening on TCP port 8082 at 0.0.0.0
注意，多集群的操作要謹慎，尤其是使用正則來匹配 context name；還有 ?-q 參數會跳過要操作的集群提醒，直接執行命令。?
