misterli's Blog.

k8s集群中pod镜像版本检查

字数统计: 1.7k阅读时长: 9 min
2020/11/17

概述

version-checker用于观察k8s集群中运行的镜像的当前版本以及上游可用的最新版本。这些检查作为Prometheus指标公开,可以在gara fana上查看。

该工具目前处于实验阶段。

version-checker支持以下仓库:

  • ACR
  • Docker Hub
  • ECR
  • GCR (inc gcr facades such as k8s.gcr.io)
  • Quay
  • 自托管(符合Docker V2 API的registry,例如harbor,artifactory等)。可以一次配置多个自托管registry。

这些镜像仓库支持身份验证。

项目地址:https://github.com/jetstack/version-checker.git

安装

可以使用官方提供的yaml文件或者helm文件安装, 均在项目下的deploy目录下

yaml安装

1
$ kubectl apply -k ./deploy/yaml

helm文件安装

1
2
$ cd ./deploy/charts/version-checker && kubectl create namespace version-checker
$ helm install version-checker . -n version-checker

这里我使用yaml安装,安装后如下

1
2
3
4
5
6
7
[root@master-01 version-check]# kubectl get pod,svc -n version-checker  -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/version-checker-8cfbf9f69-4htlv 1/1 Running 0 35m 100.67.79.188 node-01 <none> <none>

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/version-checker ClusterIP 10.104.135.133 <none> 8080/TCP 81m app=version-checker

使用

Version-checker 暴露prometheus指标的地址为 http://podip:8080/metrics

1
2
3
4
[root@master-01 version-check]# curl http://100.67.79.188:8080/metrics
# HELP version_checker_is_latest_version Where the container in use is using the latest upstream registry version
# TYPE version_checker_is_latest_version gauge
version_checker_is_latest_version{container="version-checker",current_version="v0.2.1",image="quay.io/jetstack/version-checker",latest_version="v0.2.1",namespace="version-checker",pod="version-checker-8cfbf9f69-4htlv"}

version-checker 默认只会检查包含注解 enable.version-checker.io/my-container-name: "true" 的pod。 所以上面只显示了 version-checker-8cfbf9f69-4htlv这个pod,可以通过启动时添加选项 -a, –test-all-containers 来检查所有pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: version-checker
name: version-checker
namespace: version-checker
spec:
replicas: 1
selector:
matchLabels:
app: version-checker
template:
metadata:
labels:
app: version-checker
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
enable.version-checker.io/version-checker: "true"
spec:
serviceAccountName: version-checker
containers:
- image: quay.io/jetstack/version-checker:v0.2.1
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
name: version-checker
command: ["version-checker","-a"]

我们此时在访问metrics 如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@master-01 version-check]# curl 100.67.79.156:8080/metrics
# HELP version_checker_is_latest_version Where the container in use is using the latest upstream registry version
# TYPE version_checker_is_latest_version gauge
version_checker_is_latest_version{container="alertmanager",current_version="v0.21.0",image="quay.io/prometheus/alertmanager",latest_version="v0.21.0",namespace="monitoring",pod="alertmanager-main-0"} 1
version_checker_is_latest_version{container="alertmanager-proxy",current_version="6.0.1",image="keycloak/keycloak-gatekeeper",latest_version="7.0.0",namespace="dex",pod="alertmanager-proxy-c4c8c5754-vh2tz"} 0
version_checker_is_latest_version{container="alertmanager-proxy",current_version="6.0.1",image="keycloak/keycloak-gatekeeper",latest_version="7.0.0",namespace="dex",pod="kuboard-proxy-765848d698-mdsdj"} 0
version_checker_is_latest_version{container="alertmanager-proxy",current_version="6.0.1",image="keycloak/keycloak-gatekeeper",latest_version="7.0.0",namespace="dex",pod="longhorn-proxy-f7cbfb745-8kvct"} 0
version_checker_is_latest_version{container="alertmanager-proxy",current_version="6.0.1",image="keycloak/keycloak-gatekeeper",latest_version="7.0.0",namespace="dex",pod="traefik-proxy-5c744bb4dc-t5t4t"} 0
version_checker_is_latest_version{container="argocd-application-controller",current_version="v1.7.5",image="argoproj/argocd",latest_version="v1.7.8",namespace="argocd",pod="argocd-application-controller-6c5dfc6fc5-spvvq"} 0
version_checker_is_latest_version{container="argocd-repo-server",current_version="v1.7.5",image="argoproj/argocd",latest_version="v1.7.8",namespace="argocd",pod="argocd-repo-server-67f8db4f7c-7524w"} 0
version_checker_is_latest_version{container="argocd-server",current_version="v1.7.5",image="argoproj/argocd",latest_version="v1.7.8",namespace="argocd",pod="argocd-server-646767ff75-2gld6"} 0
version_checker_is_latest_version{container="blackbox",current_version="v0.16.0",image="prom/blackbox-exporter",latest_version="v0.18.0",namespace="monitoring",pod="blackbox-549bfdd9dc-g679w"} 0
version_checker_is_latest_version{container="calico-node",current_version="v3.8.2",image="calico/node",latest_version="9512289",namespace="kube-system",pod="calico-node-jz8pj"} 0
version_checker_is_latest_version{container="calico-node",current_version="v3.8.2",image="calico/node",latest_version="9512289",namespace="kube-system",pod="calico-node-p8nv4"} 0
version_checker_is_latest_version{container="chartmuseum",current_version="v2.1.1",image="goharbor/chartmuseum-photon",latest_version="v2.1.1",namespace="harbor",pod="harbor-harbor-chartmuseum-78d9dcf76-9wrnt"} 1
version_checker_is_latest_version{container="check-ecs-price",current_version="v0.4",image="misterli/checkecsprice",latest_version="v0.4",namespace="default",pod="check-ecs-price-5b74cbf8dc-ghzf7"} 1
version_checker_is_latest_version{container="kubernetes-dashboard-proxy",current_version="6.0.1",image="keycloak/keycloak-gatekeeper",latest_version="7.0.0",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-proxy-5bcf658b56-k5p97"} 0
version_checker_is_latest_version{container="kuboard",current_version="v2.0.6-beta.1",image="eipwork/kuboard",latest_version="v2.0.5",namespace="kube-system",pod="kuboard-5b5b5859f7-87flj"} 1
version_checker_is_latest_version{container="loki",current_version="2.0.0@sha256:77e138f81a8e253f1d0ea5d2dc329a02212ecab3247c87f85f1f2182a0160ccd",image="grafana/loki",latest_version="2.0.0@sha256:91b0a08eb482c677304a3ab09e3e71eb10a9e78b05309cc178b07be83c0b238e",namespace="monitoring",pod="loki-0"} 1
version_checker_is_latest_version{container="longhorn-csi-plugin",current_version="v1.0.2",image="longhornio/longhorn-manager",latest_version="v1.0.2",namespace="longhorn-system",pod="longhorn-csi-plugin-2gfls"} 1
version_checker_is_latest_version{container="longhorn-driver-deployer",current_version="v1.0.2",image="longhornio/longhorn-manager",latest_version="v1.0.2",namespace="longhorn-system",pod="longhorn-driver-deployer-7d957dcd9-pc74k"} 1
version_checker_is_latest_version{container="longhorn-manager",current_version="v1.0.2",image="longhornio/longhorn-manager",latest_version="v1.0.2",namespace="longhorn-system",pod="longhorn-manager-9qdcs"} 1
version_checker_is_latest_version{container="longhorn-ui",current_version="v1.0.2",image="longhornio/longhorn-ui",latest_version="v1.0.2",namespace="longhorn-system",pod="longhorn-ui-65d76ddf9b-f9vkw"} 1
.......略

version-checker还支持如下注释:

  • pin-major.version-checker.io/my-container: 4。将主版本固定为4(v4.0.0)。
  • pin-minor.version-checker.io/my-container: 3。将次要版本固定为3(v0.3.0)。
  • pin-patch.version-checker.io/my-container: 23。将补丁程序的版本固定为23(v0.0.23)。
  • use-metadata.version-checker.io/my-container: "true"。将允许搜索包含字符串第一部分之后的信息的镜像标签。例如,这可以是预发布或生成元数据(v1.2.4-alpha.0v1.2.3-debian-r3)。
  • use-sha.version-checker.io/my-container: “true”`。将检查可用的最新SHA标签。如果未设置镜像标签或“latest”镜像标签,则将其静默设置为true。不能与任何其他选项一起使用。
  • match-regex.version-checker.io/my-container: ^v\d+\.\d+\.\d+-debian-。仅用于与匹配正则表达式集的镜像标签进行比较。例如,上面的注释将仅检查格式为的镜像标签v1.3.4-debian-r30use-metadata.version-checker.io设置时此项时不需要。设置此选项后,除URL覆盖以外的所有其他选项都将被忽略。
  • override-url.version-checker.io/my-container: docker.io/bitnami/etcd。用于更改URL,以查找最新图像版本所在的位置。在此示例中,my-container将把当前版本与docker.io/bitnami/etcd仓库中的镜像版本进行比较。

监控展示

我们还可以用prometheus和gtafana 进行监控展示

注意:通过helm安装会自动创建ServiceMonitor,这里我们需要手动创建ServiceMonitor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: version-checker
labels:
release: prometheus-operator
namespace: monitoring
spec:
selector:
matchLabels:
app: version-checker
endpoints:
- port: web
path: /metrics
jobLabel: version-checker
namespaceSelector:
matchNames:
- version-checker

稍等片刻我们可以到prometheus查看到相应target

image-20201117133709228

grafana模板地址: https://grafana.com/grafana/dashboards/12833

效果如下

image-20201117133817364

CATALOG
  1. 1. 概述
  2. 2. 安装
  3. 3. 使用
  4. 4. 监控展示