packages/iSulad
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
iSulad/0045-add-cri-1.29-update-design-doc.patch

313 lines
228 KiB
Diff
Raw Normal View History

upgrade from upstream
2024-04-20 10:00:53 +08:00
From a1f8641f7b440362a615e72246c92654008f61e9 Mon Sep 17 00:00:00 2001
From: jikai <jikai11@huawei.com>
Date: Thu, 22 Feb 2024 11:09:50 +0800
Subject: [PATCH 45/69] add cri 1.29 update design doc
Signed-off-by: jikai <jikai11@huawei.com>
---
README.md | 2 +-
docs/design/README_zh.md | 3 +-
.../detailed/CRI/CRI_1.29_update_design.md | 237 ++++++++++++++++++
.../cri_api_1.29_GetContainerEvents.svg | 8 +
docs/images/cri_api_1.29_overview.svg | 5 +
5 files changed, 253 insertions(+), 2 deletions(-)
create mode 100644 docs/design/detailed/CRI/CRI_1.29_update_design.md
create mode 100644 docs/images/cri_api_1.29_GetContainerEvents.svg
create mode 100644 docs/images/cri_api_1.29_overview.svg
diff --git a/README.md b/README.md
index 970b6e72..694ddbc2 100644
--- a/README.md
+++ b/README.md
@@ -240,4 +240,4 @@ iSulad Version | Kubernetes Version | CRI Version
--- | --- | ---
v2.0.0+ | v1.13-v1.18 | v1alpha2
v2.0.8+ | v1.19-v1.22 | v1alpha2
-v2.1.4+ | v1.23-v1.26 | v1, v1alpha2
+v2.1.4+ | v1.23-v1.29 | v1, v1alpha2
diff --git a/docs/design/README_zh.md b/docs/design/README_zh.md
index b7ec3ddb..3382bfbe 100644
--- a/docs/design/README_zh.md
+++ b/docs/design/README_zh.md
@@ -69,4 +69,5 @@
- 查看 Sandbox 模块的设计文档： [sandbox_design_zh](./detailed/Sandbox/sandbox_design_zh.md) 。
- 查看 Controller 模块的设计文档： [controller_design_zh](./detailed/Sandbox/controller_design_zh.md) 。
-- 查看 CRI V1 模块的设计文档： [podsandbox_cri_interface_design](./detailed/Sandbox/podsandbox_cri_interface_design.md) 。
\ No newline at end of file
+- 查看 CRI V1 模块的设计文档： [podsandbox_cri_interface_design](./detailed/Sandbox/podsandbox_cri_interface_design.md) 。
+- 查看 CRI 1.29 更新模块的设计文档： [cri_1.29_update_design](./detailed/CRI/CRI_1.29_update_design.md) 。
diff --git a/docs/design/detailed/CRI/CRI_1.29_update_design.md b/docs/design/detailed/CRI/CRI_1.29_update_design.md
new file mode 100644
index 00000000..0a0c860b
--- /dev/null
+++ b/docs/design/detailed/CRI/CRI_1.29_update_design.md
@@ -0,0 +1,237 @@
+| Author | 吉凯 |
+| ------ | ----------------------|
+| Date | 2024-02-20 |
+| Email | jikai11@huawei.com |
+
+### 方案目标
+
+当前iSulad CRI接口版本采用的1.25版本, 预期升级至1.29版本，iSulad CRI接口需要对升级后的新增CRI字段进行补充。
+
+该需求需要iSulad对K8S新版本CRI接口1.29提供支持，在K8S 1.25及之前，K8S存在CRI V1alpha2 和 CRI V1两种版本的CRI接口，
+但从1.26开始，K8S开始仅提供对于新版本CRI V1的支持，对1.25及之前的CRI接口，此次升级中，iSulad引用CRI V1，
+对于1.25及之前的CRI接口，V1alpha2和V1功能保持一致，1.26及之后新增的特性仅在CRI V1中提供支持。
+
+CRI V1使能：
+
+1、编译iSulad时开启ENABLE_CRI_API_V1编译选项
+
+```bash
+cmake ../ -D ENABLE_CRI_API_V1=ON
+```
+
+2、daemon.json中enable-cri-v1设置为true
+
+```json
+{
+ "group": "isula",
+ "default-runtime": "runc",
+ "graph": "/var/lib/isulad",
+ "state": "/var/run/isulad",
+ "log-level": "ERROR",
+ "pidfile": "/var/run/isulad.pid",
+ ...
+ "enable-cri-v1": true,
+}
+```
+
+### 总体设计
+
+#### 变更文档
+
+<https://gitee.com/openeuler/iSulad/blob/master/docs/design/detailed/CRI/CRI_1.29_interface_change.md>
+
+#### 总体设计图
+
+![](../../../images/cri_api_1.29_overview.svg)
+
+#### 此次更新包含以下变更
+
+##### [kubelet: get cgroup driver config from CRI](https://github.com/kubernetes/kubernetes/pull/118770)
+
+`RuntimeConfig` 获取cgroup驱动配置 cgroupfs 或 systemd-cgroup
+
+##### [Add Support for Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111384)
+
+`GetContainerEents`，提供对pod生命周期相关事件流
+
+`PodSandboxStatus`有相应调整
+
+##### [Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource)](https://github.com/kubernetes/kubernetes/pull/118865)
+
+`ContainerStats`新增虚拟内存使用情况信息： `SwapUsage`
+
+##### 变更排查
+
+1、[Propose comment to standardize the reason field](https://github.com/kubernetes/kubernetes/pull/112977)
+
+Must be set to "OOMKilled" for containers terminated by cgroup-based Out-of-Memory killer.
+
+container status中reason字段在cgroup out-of-memory时应该设置为OOMKilled
+
+2、[Improve the API description of `PodSecurityContext.SupplementalGroups` to clarify its unfamiliar behavior](https://github.com/kubernetes/kubernetes/pull/113047)
+
+描述修改，优化`PodSecurityContext.SupplementalGroups`的注释，明确容器镜像定义的主UID不在该列表下的行为
+
+- Clarified the expected behavior of `SupplementalGroups` field of `PodSecurityContext`
+
+```txt
+ // List of groups applied to the first process run in the sandbox, in
+ // addition to the sandbox's primary GID, and group memberships defined
+ // in the container image for the sandbox's primary UID of the container process.
+ // If the list is empty, no additional groups are added to any container.
+ // Note that group memberships defined in the container image for the sandbox's primary UID
+ // of the container process are still effective, even if they are not included in this list.
+```
+
+3、[cri-api: document expectation of 16 MB limit](https://github.com/kubernetes/kubernetes/pull/110435)
+
+```txt
+ // Captured command stdout output.
+ // The runtime should cap the output of this response to 16MB.
+ // If the stdout of the command produces more than 16MB, the remaining output
+ // should be discarded, and the command should proceed with no error.
+ // See CVE-2022-1708 and CVE-2022-31030 for more information.
+```
+
+##### 其它不涉及
+
+1、windows相关
+
+[CRI: Add Windows Podsandbox Stats](https://github.com/kubernetes/kubernetes/pull/110754)
+
+[Windows hostnetwork alpha](https://github.com/kubernetes/kubernetes/pull/112961)
+
+[Expose commit memory used in WindowsMemoryUsage struct](https://github.com/kubernetes/kubernetes/pull/119238)
+
+2、iSulad 底层暂不支持
+
+[Add mappings for volumes](https://github.com/kubernetes/kubernetes/pull/116377)
+
+[CRI: Add CDI device info for containers](https://github.com/kubernetes/kubernetes/pull/115891/)
+
+3、K8S KEP 初期，其它容器引擎未支持，iSulad暂不支持
+
+[kubelet: add support for broadcasting metrics from CRI](https://github.com/kubernetes/kubernetes/pull/113609)
+
+4、其它
+
+[Add user specified image to CRI ContainerConfig](https://github.com/kubernetes/kubernetes/pull/118652)
+
+[Add runtime handler field to ImageSpec struct](https://github.com/kubernetes/kubernetes/pull/121121)
+
+[cri-api: fix comment lines about PROPAGATION_PRIVATE](https://github.com/kubernetes/kubernetes/pull/115704)
+
+[Kubelet disk api cri update](https://github.com/kubernetes/kubernetes/pull/120914)
+
+### 详细设计
+
+#### Add swap stats to ContainerStats
+
+`ContainerStats`新增虚拟内存使用情况信息： `SwapUsage`
+
+```mermaid
+sequenceDiagram
+ participant kubelet
+ participant isulad
+ participant runc
+ participant cgroup
+
+ kubelet->>isulad:CRI request(ContainerStats)isulad->>isulad:container stats request
+ isulad->>runc:container create request(w/wo --sytemd-cgroup)
+ runc->>cgroup:read cgroup file
+ runc->>isulad:container stats info
+ isulad ->> kubelet:response
+```
+
+#### define reason to ContainerStatus
+
+container status中reason字段在cgroup out-of-memory时应该设置为OOMKilled
+
+```mermaid
+sequenceDiagram
+ participant kubelet
+ participant isulad
+ participant runc
+ participant cgroup
+
+ kubelet->>isulad:CRI request
+ alt CreateContainer&StartContainer
+ isulad->>runc:container create request(w memory limit set)
+ runc ->> cgroup:create cgroup path
+ isulad->>cgroup:monitor cgroup oom event
+ cgroup->>isulad:oom triggered
+ else ContainerStatus
+ isulad->>isulad:get container status
+ end
+ isulad ->> kubelet:response
+```
+
+#### kubelet: get cgroup driver config from CRI
+
+KEP：<https://github.com/kubernetes/enhancements/issues/4033>
+
+##### RuntimeConfig CRI API修改
+
+新增rpc，获取cgroup驱动配置: cgroupfs 或者 systemd-cgroup
+
+##### iSulad RuntimeConfig 设计
+
+仅提供对runc的支持
+
+配置文件daemon.json: systemd-cgroup: true表示使用systemd-cgroup否则使用cgroupfs
+
+在iSulad配置cgroup driver为systemd-cgroup时，底层runc采用对应的systemd-cgroup
+
+```mermaid
+sequenceDiagram
+ participant kubelet
+ participant isulad
+ participant runc
+ participant cgroup
+
+ kubelet->>isulad:CRI request
+ alt RuntimeConfig
+ isulad->>isulad:get cgroup driver from conf
+ else RunPodSandbox/CreateContainer
+ isulad->>isulad:get cgroup driver from conf
+ isulad->>runc:container create request(w/wo --sytemd-cgroup)
+ runc ->> cgroup:create cgroup path
+ end
+ isulad ->> kubelet:response
+```
+
+#### Add Support for Evented PLEG
+
+##### GetContainerEvents CRI API修改
+
+`GetContainerEvents` rpc接口此前尚未支持，此次提供对该接口的支持
+
+##### iSulad GetContainerEvents设计
+
+CRIContainerEvents 模块
+
+增加接口mailbox_publish
+
+事件包括沙箱和容器以下几种类型生命周期事件
+
+```proto
+enum ContainerEventType {
+ // Container created
+ CONTAINER_CREATED_EVENT = 0;
+
+ // Container started
+ CONTAINER_STARTED_EVENT = 1;
+
+ // Container stopped
+ CONTAINER_STOPPED_EVENT = 2;
+
+ // Container deleted
+ CONTAINER_DELETED_EVENT = 3;
+}
+```
+
+![](../../../images/cri_api_1.29_GetContainerEvents.svg)
+
+### 使用限制
+
+以上特性仅保证容器运行时设置为runc时支持。
diff --git a/docs/images/cri_api_1.29_GetContainerEvents.svg b/docs/images/cri_api_1.29_GetContainerEvents.svg
new file mode 100644
index 00000000..be855b5b
--- /dev/null
+++ b/docs/images/cri_api_1.29_GetContainerEvents.svg
@@ -0,0 +1,8 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 711.9182835520442 573.5964413809206" width="711.9182835520442" height="573.5964413809206" filter="invert(93%) hue-rotate(180deg)" class="excalidraw-svg">
+ <!-- svg-source:excalidraw -->
+
+ <defs>
+ <style class="style-fonts"> @font-face {font-display: swap;font-family: "Virgil";src: url("data:application/font-woff;charset=utf-8;base64,d09GMk9UVE8AAO9AAAkAAAABO1AAAO73AAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAADYTJHQZgAIluATYCJAOQWAQGBY9lByBbpzpxQ1QZOitEWvl1ExHg7NqqxsS6RlCwSkciyuhqMfv//89KOsZwUAeGmJXV32EeijNbRUcYuVUTmWNvG5FWRBfkAW4YWVueXU6tq9d0pZufMcACcUzc7AmXTyBwYmw0WojBScItqLq853lRPMu/i21F/1VrveHFeyj+LdHE3AfsFiWacmNotmSIhpJesVGSFIFWJDgsMxNmRIsQQzGF4rvOX33MPtY+LeOF490dgR0G/l0zmGbP8ku9beGBOf2+EcKFm8Md59ESM5GZKYtFok/F14jmMBPlbsKEWTmjIcDMVCgM0RQv1VD4IZddn+x6g2Ff91p6BriTOHYqRgBPP/Z8O3Pn/WXBXUTBKoZoSbZg6N3qwPP/937uvc+bD5huHSziTSgio5mQNpAsr235cGq9P8/r5vc1H/KatrSSHj8fGwzDhavuia24cMyBvutkjYQtKFtwTcS1wVWcODY7KMSFOM69OS/+4fm59f7vRf5FsY0aMXLQQgsGrRwggmTYYCJVEmVBm4V5Z0ThidFYPeyrtRKEoeme3dvtgT6iOeJnlC8MsAKKT0UoFD469oV+l7DD8pn6lvkVc++0kSEfkCM0Vauqh+RnHNszHtYIqREKELpb3S0eAhMvIXmB7+tkb9/wpNO+5fOjmT/aU+6YYwu6wVlib1xrbGxcwNgduqmiNNOESimjUWmogISEAOOOe086XjvNccGlpjQnb8gjOY3Gyr1sLlmavf8vKSIJwtUoBiEBfY1jpaqrq1ET/z2/7f+/4/hmD/yU4cZxDvyseBGJ7/W9rVwMSlAkJUsk6hzqQCsqJVbd9NbifX/4r6pJ4LDfsD5vl0a3Nl1TSqk09SUxpviFT9KCpj3jFmScMz4t9f8VWXkadd4bSR35QqxNH33Ai5BewBI5hQAbQsZFe2nQZ84A787uLKHXduwYMHUawjI9wuP+7R+jZdW/WpL3hdof92mkcoac8YBwj3gJB7CB3aayqzgrsSIikYrAZYK2u3l6AJfgdm+OybOvB8+JTFAi4hU026JOz7fHCxQujUCxVVutY8Wgje+dK4CA+afpvl+9Nyecky85rLBnngvwxHEpBa9ATSI+MxnxjMcbadI6LaWIPXHlOeusTrrDBkAFCAzMc6UTOFEFKTY1Nm4GuAPasdAah4+DcVmNg/pfU5Nao+e7QykouG0KYR4ewkKQVmtP1n91ike6Kvt60/WySWeBpeOKr7QCg0hoKAwgPPOvqrl+UPI78CqVirROp/RhKm2YMy0C6CJQdAHdBNBOBOmKyGuC5AZe6X1LKQUflBN+2M7DB2VHoBs/KOcRvCZQfn6i0lsdM5Ypw5otoyCn0UlftixbMiaTk2XKy5Q1Y17G2zJOB7nKUm0/PkgM7J3K/X5azIxlS8eShkmeIYrIIuvN47j+8Xk5xmbzq/fd0AYlB/aifvZu/3or31eH2eCosZNM23gKUvMyo25K7fx8x2ZRqLM4/xxLbOST4x/7aOuym35y8olHctP4b8V/TX/YCP2p/1mult99+Gj0u488j59Q7/DTaWN4EkPx7itzRBhgghZ7XPAiQJAIcTLMtFCJKku1WqHHWlvsst9xLnKDYR4xwicQZ7tG+knSE5NnLF69LLqrR0X89vtHVudoGtiw59gDkletTr9hov04IcNsWAhCFGnUMYNrWMA6cqigBQoCbHTgQRhxDOMAjuIkzuIyzuAybuEuHmMEbxC773VEWIigqlrWqT4NbHCjmtiM5nd+y1vXlq5ob9d1c/f3eC93qA/7pk6NEihFn0Rmmnk61thin7O8wilOc4krXGeGedbYJEaSKhN6GOUQD/IKz/ACr/IW7/ERn/Il3/ITf/B1//9y0DjoHH5x+M2hIRCCxhEpTKlFmUZcYipGDGMmFmBNbIBtYYfYXewxq9h9hvn6/vr3thQeZ+KUArHScmqorznNakXbKqkpSqIM+SLlVlB9GtJ+HdUZXdZZXdGQ7uuZ3uizfnlBkASHEFk57pwrbnvKU17wunOuGjNrxbYj2+1zj/u92yM+4fO+5gu+7jt+6Od+40+uMZH/QTH0f6P/F2WjGegMdD6aieagxWg5WoU2oG0oH5WhWgfrEB2GI+UoOOqOGcc5+Rat6/9k/8+mUCCeJiIC5sZapHx9QEX2IMLTPuIbQjZoyoY9XgDCc1CSUcEfzRVFJcPJ4YfarTTXnEEp9Fll4bmCyF6huXD0c2NkyBrUclqFDS0fCcldmnBxq5ZzUdjQ0hBhra3noZkdrY2QIXdtwDXbzuJpombNK3hcpdld/aaHaGW4yOQ0Uq2KuSpqqHgkIlXMSxG5hZWvipLO5Yja+dGe2Ec0Xl7hJVeLriry+vKRHMnapHSSWh3Na/r0xosnKF8vBiOnHHm02DmrmKh7JKa/6Kx4A1EGB5L5gPZja3EC36Px8gVmpEMfpVo7eKal3dCDQk3OQWrtEfywxsP7kXKdc+xllnS10CSorh6UNHRFJNThKpI3Uj12NefH5EBdrYWSRruZkfa+i7mL+aR7b+irdu+/Svdr602Xpz3AJtUl38juYpi/pk0HYWKZtRHy//WcNGHEROmgcepGC9vOyzGRY1+qfJxX8Ka4G2gxbNKvLqrqBjSlQRMYHTrDDGsYBjAs71XrIXW0LYetKxhHqGHoNTTSyi1NsvSW9pYPZAY4gkxWtrU0z9vbRMU08fOJQxc+Syy+gtk9G54gA+xj+zrZX5Y0rjZrdBmGy68X/sBnzCF5AdMtKp4mqPbyQez0lZGifaq47Tf7SIZExoiy9kkO2JjFGs8VnQMdK9nMR0/nKEUbGADCEcjocG6DUGjz+CoAhQFwBBIDEwuFxo4bN+++CUBhAAhHIDEwsVBobBxhCwAAwIcDAomBiYVCY+OISUAQBH0SBIEgCIIgCIKgcvQdCMIRSAy0MCC+M6Y5BcVI/gkYkMn8sRZzoj6jjymGesj9+I2a22/ocz8NNPA2C+fgZtOvbe7ZbLKKuW1NchzK6mJ1Pf/1IF9PGNzuWk+fWr5AJLsGbXx6WqXtz7Sw5sK/2TRigdQEnKWCSwHOJDBxwFWcy/MuPpBExwGTydS++B38ygHYhoswSGJKao0cwRU+cxa87AMKDql645JDKYO+5m4QVzcJ3y7stzuYTaZTHAbNHXHpPfKmuHugSHKiUk/ZPC/TXrC0xqt/dsGQ57sgXd3hs/6nDbLRY/DknG/yxVziZVz+ldwy0Iy6cNfv9o1UVPeO7SRI/Eej3M5tWU/0ydP7mv8W+vb7QPcr3ytNdXF9bro22Y7bj7Zf+/5e7C/73n67/xrB1Iy9URj6uDPOjzvj1fgeZsnDRCrKMR3LsR2VaAcbarhhDXcEIhLDcSROx6U4E5fjZgzHoxiJt3PJ9GZwpV151l4c+73B+fGXExniT6JM1AELXkAM6XgMP5YTZEIWvN+8x96yP86/HmkfeR30BDeCq6FruBPGwn1qRw26oBG6Ql8N9qqpBupSDaqaqtbOeqAPtE9H9a6ua4yag2bZtBiT0RqTSZtKYzeKbdmu2Fors26rtyFbtJX2mzudd3CO+7zBVdzBKraziKWsZzf7OcUPuIzfccG8beSYgWdMypiWsTBjeUZ2xraMHRl1GW0ZwgxVhjlDyrAzchn1jJlUWRX48WpiZWxuhR6qmw57kBDolGzsgvY89rhb7XvJzwQfY4+20SX4OtWreKpuSFaXlECpiOVamcv4JA4/x8nKhM2qKCxtMjn4oyulp5+2+DzPxYtYzPSoPf9Yvlvd4vcuYU3142xsRkmAfbwfK6ODt6039/4JpW8VEqlHIZIunBL+6JmzfHyE3o3mF33fTPtyd+kt8NrwT5yY3tXcECqX0RZ
+
+ </defs>
+ <rect x="0" y="0" width="711.9182835520442" height="573.5964413809206" fill="#ffffff"/><g stroke-linecap="round" transform="translate(117.27154915703676 238.02839846037386) rotate(0 125.6132919736506 62.51589262187065)"><path d="M31.26 0 C93.8 0.58, 154.08 0.67, 219.97 0 M31.26 0 C84.04 -1.13, 134.76 -1.21, 219.97 0 M219.97 0 C242.41 1.71, 250.55 10.05, 251.23 31.26 M219.97 0 C242.9 -0.4, 252.85 11.5, 251.23 31.26 M251.23 31.26 C252.2 53.7, 250.18 76.91, 251.23 93.77 M251.23 31.26 C251.63 54.31, 250.34 79.61, 251.23 93.77 M251.23 93.77 C249.84 113.22, 242.8 126.81, 219.97 125.03 M251.23 93.77 C252.52 116.18, 239.79 123.63, 219.97 125.03 M219.97 125.03 C156.21 125.09, 94.35 124.88, 31.26 125.03 M219.97 125.03 C166.39 125.29, 111.61 125.6, 31.26 125.03 M31.26 125.03 C9.54 123.71, 0.89 114.79, 0 93.77 M31.26 125.03 C10.68 127.23, -0.19 114.27, 0 93.77 M0 93.77 C-0.41 73.25, 1.57 55.04, 0 31.26 M0 93.77 C-0.32 73.57, 0.04 52.78, 0 31.26 M0 31.26 C1.85 9.36, 11.18 0.6, 31.26 0 M0 31.26 C0.96 10.27, 9.51 1.34, 31.26 0" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g stroke-linecap="round" transform="translate(186.70726707100698 10) rotate(0 67.83414177602211 24.5)"><path d="M12.25 0 C40.5 0.38, 68.72 0.57, 123.42 0 M12.25 0 C55.6 0.03, 100.91 -0.01, 123.42 0 M123.42 0 C130.91 1.57, 136.88 5.9, 135.67 12.25 M123.42 0 C132.06 -1.24, 137.56 3.82, 135.67 12.25 M135.67 12.25 C135.51 17.2, 134.08 24.91, 135.67 36.75 M135.67 12.25 C134.61 19.88, 134.58 27.19, 135.67 36.75 M135.67 36.75 C137.48 46.76, 131.69 48.23, 123.42 49 M135.67 36.75 C137.06 43.82, 129.39 48.33, 123.42 49 M123.42 49 C98.21 47.23, 77.19 50.26, 12.25 49 M123.42 49 C91.01 48.47, 56.89 48.69, 12.25 49 M12.25 49 C3.53 48.6, 0.28 46.55, 0 36.75 M12.25 49 C5.16 50.53, -1.41 43.61, 0 36.75 M0 36.75 C-2.07 30.49, -0.06 21.95, 0 12.25 M0 36.75 C-0.15 28.95, -0.46 18.52, 0 12.25 M0 12.25 C-1.08 4.97, 3.28 -1.96, 12.25 0 M0 12.25 C-0.18 2.58, 5.05 2.17, 12.25 0" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(236.42063553159937 27.10232178526644) rotate(0 18.120773315429688 7.39767821473356)"><text x="18.120773315429688" y="10.368585785770504" font-family="Virgil, Segoe UI Emoji" font-size="11.836285143573637px" fill="#1e1e1e" text-anchor="middle" style="white-space: pre;" direction="ltr" dominant-baseline="alphabetic">Kublet</text></g><g stroke-linecap="round"><g transform="translate(248.10775797750728 63.908976948636564) rotate(0 0.33648643876290407 114.86137924595312)"><path d="M0 0 C0.11 38.29, 0.56 191.44, 0.67 229.72 M0 0 C0.11 38.29, 0.56 191.44, 0.67 229.72" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(248.10775797750728 63.908976948636564) rotate(0 0.33648643876290407 114.86137924595312)"><path d="M-7.95 206.26 C-5.24 213.62, -2.53 220.99, 0.67 229.72 M-7.95 206.26 C-5.83 212.03, -3.71 217.8, 0.67 229.72" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(248.10775797750728 63.908976948636564) rotate(0 0.33648643876290407 114.86137924595312)"><path d="M9.15 206.21 C6.49 213.59, 3.83 220.97, 0.67 229.72 M9.15 206.21 C7.07 211.99, 4.98 217.78, 0.67 229.72" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g></g><mask/><g stroke-linecap="round" transform="translate(10 144.1616587722251) rotate(0 345.9591417760221 209.71739130434776)"><path d="M32 0 C227.91 -2.73, 425.18 -2.08, 659.92 0 M659.92 0 C680.58 1.93, 690.18 10.06, 691.92 32 M691.92 32 C693.39 120.38, 693.37 207.8, 691.92 387.43 M691.92 387.43 C691.29 409.26, 680.31 419.89, 659.92 419.43 M659.92 419.43 C427.48 418.55, 196.22 418.73, 32 419.43 M32 419.43 C10.97 419.21, 1.58 409.18, 0 387.43 M0 387.43 C-0.16 309.85, 0.43 229.56, 0 32 M0 32 C1.75 12.18, 10.36 -1.82, 32 0" stroke="#1e1e1e" stroke-width="1" fill="none" stroke-dasharray="8 8.5"/></g><g stroke-opacity="0.7" fill-opacity="0.7" transform="translate(261.64178340975064 78.9663359790759) rotate(0 78.24016571044922 20)"><text x="0" y="14.016" font-family="Virgil, Segoe UI Emoji" font-size="16px" fill="#1e1e1e" text-anchor="start" style="white-space: pre;" dir
\ No newline at end of file
diff --git a/docs/images/cri_api_1.29_overview.svg b/docs/images/cri_api_1.29_overview.svg
new file mode 100644
index 00000000..94911913
--- /dev/null
+++ b/docs/images/cri_api_1.29_overview.svg
@@ -0,0 +1,5 @@
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1209.6444540126743 730.1247574550649" width="1209.6444540126743" height="730.1247574550649" filter="invert(93%) hue-rotate(180deg)" class="excalidraw-svg">
+ <!-- svg-source:excalidraw -->
+
+ <defs><style> @font-face {font-display: swap;font-family: "Virgil";src: url("data:application/font-woff;charset=utf-8;base64,d09GMk9UVE8AAO9AAAkAAAABO1AAAO73AAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAADYTJHQZgAIluATYCJAOQWAQGBY9lByBbpzpxQ1QZOitEWvl1ExHg7NqqxsS6RlCwSkciyuhqMfv//89KOsZwUAeGmJXV32EeijNbRUcYuVUTmWNvG5FWRBfkAW4YWVueXU6tq9d0pZufMcACcUzc7AmXTyBwYmw0WojBScItqLq853lRPMu/i21F/1VrveHFeyj+LdHE3AfsFiWacmNotmSIhpJesVGSFIFWJDgsMxNmRIsQQzGF4rvOX33MPtY+LeOF490dgR0G/l0zmGbP8ku9beGBOf2+EcKFm8Md59ESM5GZKYtFok/F14jmMBPlbsKEWTmjIcDMVCgM0RQv1VD4IZddn+x6g2Ff91p6BriTOHYqRgBPP/Z8O3Pn/WXBXUTBKoZoSbZg6N3qwPP/937uvc+bD5huHSziTSgio5mQNpAsr235cGq9P8/r5vc1H/KatrSSHj8fGwzDhavuia24cMyBvutkjYQtKFtwTcS1wVWcODY7KMSFOM69OS/+4fm59f7vRf5FsY0aMXLQQgsGrRwggmTYYCJVEmVBm4V5Z0ThidFYPeyrtRKEoeme3dvtgT6iOeJnlC8MsAKKT0UoFD469oV+l7DD8pn6lvkVc++0kSEfkCM0Vauqh+RnHNszHtYIqREKELpb3S0eAhMvIXmB7+tkb9/wpNO+5fOjmT/aU+6YYwu6wVlib1xrbGxcwNgduqmiNNOESimjUWmogISEAOOOe086XjvNccGlpjQnb8gjOY3Gyr1sLlmavf8vKSIJwtUoBiEBfY1jpaqrq1ET/z2/7f+/4/hmD/yU4cZxDvyseBGJ7/W9rVwMSlAkJUsk6hzqQCsqJVbd9NbifX/4r6pJ4LDfsD5vl0a3Nl1TSqk09SUxpviFT9KCpj3jFmScMz4t9f8VWXkadd4bSR35QqxNH33Ai5BewBI5hQAbQsZFe2nQZ84A787uLKHXduwYMHUawjI9wuP+7R+jZdW/WpL3hdof92mkcoac8YBwj3gJB7CB3aayqzgrsSIikYrAZYK2u3l6AJfgdm+OybOvB8+JTFAi4hU026JOz7fHCxQujUCxVVutY8Wgje+dK4CA+afpvl+9Nyecky85rLBnngvwxHEpBa9ATSI+MxnxjMcbadI6LaWIPXHlOeusTrrDBkAFCAzMc6UTOFEFKTY1Nm4GuAPasdAah4+DcVmNg/pfU5Nao+e7QykouG0KYR4ewkKQVmtP1n91ike6Kvt60/WySWeBpeOKr7QCg0hoKAwgPPOvqrl+UPI78CqVirROp/RhKm2YMy0C6CJQdAHdBNBOBOmKyGuC5AZe6X1LKQUflBN+2M7DB2VHoBs/KOcRvCZQfn6i0lsdM5Ypw5otoyCn0UlftixbMiaTk2XKy5Q1Y17G2zJOB7nKUm0/PkgM7J3K/X5azIxlS8eShkmeIYrIIuvN47j+8Xk5xmbzq/fd0AYlB/aifvZu/3or31eH2eCosZNM23gKUvMyo25K7fx8x2ZRqLM4/xxLbOST4x/7aOuym35y8olHctP4b8V/TX/YCP2p/1mult99+Gj0u488j59Q7/DTaWN4EkPx7itzRBhgghZ7XPAiQJAIcTLMtFCJKku1WqHHWlvsst9xLnKDYR4xwicQZ7tG+knSE5NnLF69LLqrR0X89vtHVudoGtiw59gDkletTr9hov04IcNsWAhCFGnUMYNrWMA6cqigBQoCbHTgQRhxDOMAjuIkzuIyzuAybuEuHmMEbxC773VEWIigqlrWqT4NbHCjmtiM5nd+y1vXlq5ob9d1c/f3eC93qA/7pk6NEihFn0Rmmnk61thin7O8wilOc4krXGeGedbYJEaSKhN6GOUQD/IKz/ACr/IW7/ERn/Il3/ITf/B1//9y0DjoHH5x+M2hIRCCxhEpTKlFmUZcYipGDGMmFmBNbIBtYYfYXewxq9h9hvn6/vr3thQeZ+KUArHScmqorznNakXbKqkpSqIM+SLlVlB9GtJ+HdUZXdZZXdGQ7uuZ3uizfnlBkASHEFk57pwrbnvKU17wunOuGjNrxbYj2+1zj/u92yM+4fO+5gu+7jt+6Od+40+uMZH/QTH0f6P/F2WjGegMdD6aieagxWg5WoU2oG0oH5WhWgfrEB2GI+UoOOqOGcc5+Rat6/9k/8+mUCCeJiIC5sZapHx9QEX2IMLTPuIbQjZoyoY9XgDCc1CSUcEfzRVFJcPJ4YfarTTXnEEp9Fll4bmCyF6huXD0c2NkyBrUclqFDS0fCcldmnBxq5ZzUdjQ0hBhra3noZkdrY2QIXdtwDXbzuJpombNK3hcpdld/aaHaGW4yOQ0Uq2KuSpqqHgkIlXMSxG5hZWvipLO5Yja+dGe2Ec0Xl7hJVeLriry+vKRHMnapHSSWh3Na/r0xosnKF8vBiOnHHm02DmrmKh7JKa/6Kx4A1EGB5L5gPZja3EC36Px8gVmpEMfpVo7eKal3dCDQk3OQWrtEfywxsP7kXKdc+xllnS10CSorh6UNHRFJNThKpI3Uj12NefH5EBdrYWSRruZkfa+i7mL+aR7b+irdu+/Svdr602Xpz3AJtUl38juYpi/pk0HYWKZtRHy//WcNGHEROmgcepGC9vOyzGRY1+qfJxX8Ka4G2gxbNKvLqrqBjSlQRMYHTrDDGsYBjAs71XrIXW0LYetKxhHqGHoNTTSyi1NsvSW9pYPZAY4gkxWtrU0z9vbRMU08fOJQxc+Syy+gtk9G54gA+xj+zrZX5Y0rjZrdBmGy68X/sBnzCF5AdMtKp4mqPbyQez0lZGifaq47Tf7SIZExoiy9kkO2JjFGs8VnQMdK9nMR0/nKEUbGADCEcjocG6DUGjz+CoAhQFwBBIDEwuFxo4bN+++CUBhAAhHIDEwsVBobBxhCwAAwIcDAomBiYVCY+OISUAQBH0SBIEgCIIgCIKgcvQdCMIRSAy0MCC+M6Y5BcVI/gkYkMn8sRZzoj6jjymGesj9+I2a22/ocz8NNPA2C+fgZtOvbe7ZbLKKuW1NchzK6mJ1Pf/1IF9PGNzuWk+fWr5AJLsGbXx6WqXtz7Sw5sK/2TRigdQEnKWCSwHOJDBxwFWcy/MuPpBExwGTydS++B38ygHYhoswSGJKao0cwRU+cxa87AMKDql645JDKYO+5m4QVzcJ3y7stzuYTaZTHAbNHXHpPfKmuHugSHKiUk/ZPC/TXrC0xqt/dsGQ57sgXd3hs/6nDbLRY/DknG/yxVziZVz+ldwy0Iy6cNfv9o1UVPeO7SRI/Eej3M5tWU/0ydP7mv8W+vb7QPcr3ytNdXF9bro22Y7bj7Zf+/5e7C/73n67/xrB1Iy9URj6uDPOjzvj1fgeZsnDRCrKMR3LsR2VaAcbarhhDXcEIhLDcSROx6U4E5fjZgzHoxiJt3PJ9GZwpV151l4c+73B+fGXExniT6JM1AELXkAM6XgMP5YTZEIWvN+8x96yP86/HmkfeR30BDeCq6FruBPGwn1qRw26oBG6Ql8N9qqpBupSDaqaqtbOeqAPtE9H9a6ua4yag2bZtBiT0RqTSZtKYzeKbdmu2Fors26rtyFbtJX2mzudd3CO+7zBVdzBKraziKWsZzf7OcUPuIzfccG8beSYgWdMypiWsTBjeUZ2xraMHRl1GW0ZwgxVhjlDyrAzchn1jJlUWRX48WpiZWxuhR6qmw57kBDolGzsgvY89rhb7XvJzwQfY4+20SX4OtWreKpuSFaXlECpiOVamcv4JA4/x8nKhM2qKCxtMjn4oyulp5+2+DzPxYtYzPSoPf9Yvlvd4vcuYU3142xsRkmAfbwfK6ODt6039/4JpW8VEqlHIZIunBL+6JmzfHyE3o3mF33fTPtyd+kt8NrwT5yY3tXcECqX0RZRjsN416y3bBO8IjD
+ <rect x="0" y="0" width="1209.6444540126743" height="730.1247574550649" fill="#ffffff"/><g stroke-linecap="round" transform="translate(449.046845608731 593.6612861565363) rotate(0 67.83414177602211 24.5)"><path d="M12.25 0 C38.27 1.54, 69.71 0.86, 123.42 0 M12.25 0 C57.08 1.25, 100.41 1.21, 123.42 0 M123.42 0 C130.05 0.87, 137.62 3.02, 135.67 12.25 M123.42 0 C129.41 -0.25, 135.23 3.57, 135.67 12.25 M135.67 12.25 C137.2 17.84, 137.08 24.32, 135.67 36.75 M135.67 12.25 C135.63 18.87, 135.53 27.61, 135.67 36.75 M135.67 36.75 C135.96 45.56, 130.18 48.49, 123.42 49 M135.67 36.75 C134.67 43.8, 132.19 48.93, 123.42 49 M123.42 49 C95.98 49.68, 66.49 49.35, 12.25 49 M123.42 49 C90.86 49.22, 59.02 49.84, 12.25 49 M12.25 49 C2.25 49.89, 1.02 46.81, 0 36.75 M12.25 49 C3.81 48.52, 1.25 44.93, 0 36.75 M0 36.75 C0.91 27.08, 1.15 18.34, 0 12.25 M0 36.75 C0.47 29.41, 0.25 22.37, 0 12.25 M0 12.25 C-0.6 2.28, 3.59 0.5, 12.25 0 M0 12.25 C-0.87 4.69, 5.95 0.6, 12.25 0" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(454.23300001903044 610.7636079418028) rotate(0 62.647987365722656 7.397678214733503)"><text x="62.647987365722656" y="0" font-family="Virgil, Segoe UI Emoji" font-size="11.836285143573637px" fill="#1e1e1e" text-anchor="middle" style="white-space: pre;" direction="ltr" dominant-baseline="text-before-edge">ListPodSandboxStats</text></g><g stroke-linecap="round" transform="translate(447.35828363198516 671.1247574550649) rotate(0 67.83414177602211 24.5)"><path d="M12.25 0 C34.97 0.97, 59.23 -1.52, 123.42 0 M12.25 0 C40.77 0.34, 68.84 -0.06, 123.42 0 M123.42 0 C132.18 -0.16, 134.7 5.15, 135.67 12.25 M123.42 0 C133.44 1.32, 136.16 6.03, 135.67 12.25 M135.67 12.25 C135.14 19.88, 133.97 29.85, 135.67 36.75 M135.67 12.25 C135.08 21.23, 135.94 30.75, 135.67 36.75 M135.67 36.75 C136.32 45.43, 130.5 50.16, 123.42 49 M135.67 36.75 C136.02 43.05, 130.87 48.21, 123.42 49 M123.42 49 C83.36 47.82, 40.33 49.98, 12.25 49 M123.42 49 C88.35 49.03, 52.52 49.67, 12.25 49 M12.25 49 C5.04 50.61, 0.69 45.22, 0 36.75 M12.25 49 C4 50.41, -1.68 44.04, 0 36.75 M0 36.75 C-0.96 27.04, 0.16 18.04, 0 12.25 M0 36.75 C-0.03 30.79, -0.54 24.97, 0 12.25 M0 12.25 C1.26 5.34, 4.4 -1.81, 12.25 0 M0 12.25 C1.86 5.23, 3.09 0.52, 12.25 0" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(463.93497911894474 688.2270792403314) rotate(0 51.2574462890625 7.397678214733503)"><text x="51.2574462890625" y="0" font-family="Virgil, Segoe UI Emoji" font-size="11.836285143573637px" fill="#1e1e1e" text-anchor="middle" style="white-space: pre;" direction="ltr" dominant-baseline="text-before-edge">PodSandboxStats</text></g><g stroke-linecap="round" transform="translate(737.1843705885069 631.1247574550649) rotate(0 67.83414177602211 24.5)"><path d="M12.25 0 C37.96 -0.25, 58.44 0.26, 123.42 0 M123.42 0 C130.73 1.63, 137.21 3.69, 135.67 12.25 M135.67 12.25 C134.46 18.36, 134.06 28.96, 135.67 36.75 M135.67 36.75 C136.93 45.64, 130.11 49.46, 123.42 49 M123.42 49 C101.97 49.01, 78.66 48.81, 12.25 49 M12.25 49 C2.69 49.52, 1.03 43.94, 0 36.75 M0 36.75 C-1.43 30.68, -0.02 19.27, 0 12.25 M0 12.25 C1.09 3.03, 5.16 1.26, 12.25 0" stroke="#1e1e1e" stroke-width="1" fill="none" stroke-dasharray="8 8.5"/></g><g transform="translate(753.7610660754665 648.2270792403314) rotate(0 51.2574462890625 7.397678214733503)"><text x="51.2574462890625" y="0" font-family="Virgil, Segoe UI Emoji" font-size="11.836285143573637px" fill="#1e1e1e" text-anchor="middle" style="white-space: pre;" direction="ltr" dominant-baseline="text-before-edge">PodSandboxStats</text></g><g stroke-linecap="round"><g transform="translate(588.1924254080072 612.3090108639277) rotate(0 72.91304347826085 19.572020984666608)"><path d="M0 0 C24.3 6.52, 121.52 32.62, 145.83 39.14 M0 0 C24.3 6.52, 121.52 32.62, 145.83 39.14" stroke="#1e1e1e" stroke-width="0.5" fill="none"/></g><g transform="translate(588.1924254080072 612.3090108639277) rotate(0 72.91304347826085 19.572020984666608)"><path d="M120.92 41.31 C130.67 40.46, 140.41 39.62, 145.83 39.14 M120.92 41.31 C126.71 40.81, 132.49 40.3
\ No newline at end of file
--
2.34.1
Reference in New Issue Copy Permalink