packages/iSulad
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

upgrade from upstream

Browse Source 
Signed-off-by: liuxu <liuxu156@huawei.com>
This commit is contained in:
liuxu 2024-04-20 19:31:28 +08:00
parent 02ceb7b87c
commit 60efcd7887
4 changed files with 481 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
0071-Remove-sandboxer-ut-if-sandboxer-is-not-enabled.patch Normal file
View File

@ -0,0 +1,28 @@
From 7c0c79e8ad4680f97651dd52721344961c803c15 Mon Sep 17 00:00:00 2001
From: xuxuepeng <xuxuepeng1@huawei.com>
Date: Sat, 20 Apr 2024 11:48:56 +0800
Subject: [PATCH 71/73] Remove sandboxer ut if sandboxer is not enabled
Signed-off-by: xuxuepeng <xuxuepeng1@huawei.com>
---
test/sandbox/CMakeLists.txt | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/test/sandbox/CMakeLists.txt b/test/sandbox/CMakeLists.txt
index 68397477..38d7ccb9 100644
--- a/test/sandbox/CMakeLists.txt
+++ b/test/sandbox/CMakeLists.txt
@@ -1,5 +1,7 @@
project(iSulad_UT)
-add_subdirectory(controller)
-add_subdirectory(sandbox_manager)
-add_subdirectory(sandbox)
+if (ENABLE_SANDBOXER)
+ add_subdirectory(controller)
+ add_subdirectory(sandbox_manager)
+ add_subdirectory(sandbox)
+endif()
--
2.34.1

391
0072-cdi-design-doc.patch Normal file
View File

@ -0,0 +1,391 @@
From 72246a9e83ada3af7560817534013a93f40d5bae Mon Sep 17 00:00:00 2001
From: liuxu <liuxu156@huawei.com>
Date: Sat, 20 Apr 2024 11:46:11 +0800
Subject: [PATCH 72/73] cdi:design doc
Signed-off-by: liuxu <liuxu156@huawei.com>
---
docs/design/README.md | 3 +
docs/design/README_zh.md | 3 +
docs/design/detailed/CDI/cdi_design_zh.md | 341 ++++++++++++++++++++++
3 files changed, 347 insertions(+)
create mode 100644 docs/design/detailed/CDI/cdi_design_zh.md
diff --git a/docs/design/README.md b/docs/design/README.md
index cf29c0a1..d2a3702d 100644
--- a/docs/design/README.md
+++ b/docs/design/README.md
@@ -18,6 +18,9 @@ This section contains some design documents for users who want to learn more abo
- You can see how the CRI startup process is refactored in [cri_cni_refactor](./detailed/CRI/cri_cni_refactor.md).
+## CDI
+- You can see how the CDI is refactored in [cdi_design](./detailed/CDI/cdi_design_zh.md.md)。
+
## Events
- You can see how the events modules are designed in [events_design](./detailed/Events/events_design.md).
diff --git a/docs/design/README_zh.md b/docs/design/README_zh.md
index 3382bfbe..c6172b6f 100644
--- a/docs/design/README_zh.md
+++ b/docs/design/README_zh.md
@@ -24,6 +24,9 @@
- 查看 CRI的启动程序的重构文档： [cri_cni_refactor](./detailed/CRI/cri_cni_refactor_zh.md) 。
+## CDI
+- 查看 CDI 的设计文档： [cdi_design](./detailed/CDI/cdi_design_zh.md.md)。
+
## Events
- 查看 events 模块的设计文档： [events_design](./detailed/Events/events_design_zh.md) 。
diff --git a/docs/design/detailed/CDI/cdi_design_zh.md b/docs/design/detailed/CDI/cdi_design_zh.md
new file mode 100644
index 00000000..15c88f93
--- /dev/null
+++ b/docs/design/detailed/CDI/cdi_design_zh.md
@@ -0,0 +1,341 @@
+| Author | liuxu |
+| ------ | --------------------- |
+| Date | 2024-02-27 |
+| Email | liuxu156@huawei.com |
+
+# 背景介绍
+## What is CDI?
+CDI（容器设备接口）是容器运行时的一种规范，用于支持第三方设备。
+
+它引入了device作为资源的抽象概念。device由完全限定的名称唯一指定，该名称由vendor ID、一个device class和在每个vendor ID-device class对中唯一的name构成。
+```
+vendor.com/class=unique_name
+```
+vendor ID和device class （上例中的vendor.com/class）的组合称为device kind.
+
+CDI只关心使容器能够感知设备。CDI明确地忽略了诸如资源管理之类的领域（并期望由编排器处理）。由于这个原因，CDI规范实现起来很简单，并且为运行时和编排器提供了极大的灵活性。
+
+注：CDI模型基于容器网络接口（CNI）模型和规范。
+
+## Why is CDI needed?
+在Linux上，使容器具有设备感知能力过去只需在该容器中暴露一个设备节点。但是，随着设备和软件变得越来越复杂，供应商希望执行更多的操作，例如：
+- 向容器公开设备可能需要公开多个设备节点、从运行时命名空间挂载文件或隐藏procfs条目。
+- 执行容器和设备之间的兼容性检查（例如：检查容器是否可以在指定设备上运行）。
+- 执行特定于运行时的操作（例如：虚拟机与基于Linux容器的运行时）。
+- 执行特定于设备的操作（例如：清理GPU的内存或重新配置FPGA）。
+
+在缺乏第三方设备标准的情况下，供应商通常不得不为不同的运行时编写和维护多个插件，甚至直接在运行时中贡献特定于供应商的代码。此外，运行时不统一地暴露插件系统（甚至根本不暴露插件系统），导致在更高级别的抽象（例如Kubernetes设备插件）中重复功能。
+
+## How does CDI work?
+要使CDI正常工作，需要完成以下操作：
+- JSON格式的CDI文件应位于CDI规范目录中，它的作用是更新OCI spec。默认目录为/etc/cdi和/var/run/cdi
+- 应使用CRI的annotations与CDI_devices（本次特性支持）将唯一的设备名称传递给运行时
+- 容器运行时应该能够通过设备名称找到CDI文件在内存中对应的缓存信息，并使用缓存的内容更新容器配置。
+
+## How to configure CDI?
+### iSulad
+daemon.json中开启cri-v1和cdi配置：
+
+当 cdi-spec-dirs 不指定时，默认为"/etc/cdi", "/var/run/cdi"
+```json
+"enable-cri-v1": true,
+"cdi-spec-dirs": ["/etc/cdi", "/var/run/cdi"], # 指定CDI规范所在目录
+"enable-cdi": true # 打开CDI特性
+```
+
+在CRI创建容器的参数中使用CDI，以下两种方式均可：
+1. annotations中指定设备
+```json
+{
+ ... ...
+ "annotations": [
+ ... ...
+ // key值格式要求含有cdi.k8s.io作为前缀，后面跟随pluginName
+ {"cdi.k8s.io/test": "vendor.com/device=myDevice"},
+ ... ...
+ ]
+ ... ...
+}
+```
+2. CDI_Devices中指定设备
+```json
+{
+ ... ...
+ "CDI_Devices": [
+ ... ...
+ {"Name": "vendor.com/device=myDevice"},
+ ... ...
+ ]
+ ... ...
+}
+```
+
+# 方案目标
+## 概述
+容器设备接口（Container Device Interface，简称CDI）描述了容器运行时创建能够与第三方设备交互的容器的机制。
+
+对于第三方设备，与这些设备进行交互通常需要容器运行时公开多个设备节点。例如，第三方设备可能需要加载内核模块、装载主机库、暴露/屏蔽特定procfs路径。
+
+容器设备接口描述了一种允许第三方供应商执行这些操作的机制，从而不需要更改容器运行时。
+
+使用的机制是一个JSON文件（类似于容器网络接口(CNI）)，它允许供应商描述容器运行时应该对容器的OCI规范执行的操作。
+
+CDI 支持以下两个流程：
+
+A.设备安装
+1. 用户在机器上安装第三方设备驱动程序（和第三方设备/被测试的设备）。
+2. 设备驱动程序安装软件会在一个已知路径（/etc/cdi/vendor.json）上写入一个 JSON 文件。
+
+B.容器运行时
+1. 用户在创建容器时，CRI中指定设备名称
+2. 容器运行时会读取 JSON 文件。
+3. 容器运行时会验证 JSON 文件中是否描述了设备。
+4. 容器运行时会根据 JSON 文件中的指令转换 OCI 规范，并插入OCI Spec中
+# 总体设计
+**iSulad支持CDI功能目前仅支持CRI方式调用，CLI方式暂不支持**
+## 整体结构
+
+```mermaid
+flowchart TD
+classDef unFinish fill:#c19,stroke:#216,stroke-width:2px,color:#fff,stroke-dasharray: 5 5;
+subgraph isulad
+ subgraph CDIOperate
+ RegistryOps
+ AnnotationsOps
+ ......
+ end
+
+ subgraph CDI
+ D[ContainerEdits]
+ E[Device]
+ F[Registry]
+ G[Cache]
+ H[Spec]
+ end
+ CDIOperate --> CDI
+
+ OA[isula module] --> |call| CDIOperate
+ OB[CRI module] --> |call| CDIOperate
+ CDIOperate:::unFinish
+ CDI:::unFinish
+end
+ PA[CLI] --> OA
+ PB[CRI] --> OB
+```
+
+- CDIOperate封装了CDI模块，对外提供更合理的CDI功能的相关接口。
+- CDI负责实现CDI Specs的读取、校验、解析、devices注入OCI Spec等具体功能。
+## 时序设计
+在isulad启动后，以创建一个容器为例。
+
+图中isulad启动后，拉起一个新的线程isulad-cdi_watcher，负责监控cdi-spec-dirs，当cdi-spec-dirs中的cdi Spec文件发生修改、删除等动作时，重新扫描cdi-spec-dirs中的cdi Spec文件。
+
+```mermaid
+sequenceDiagram
+ participant CRI
+ participant isulad
+ participant cdi_watcher
+ isulad ->> cdi_watcher: init cdi
+ par
+ loop
+ cdi_watcher ->> cdi_watcher:wait for inotify event
+ cdi_watcher ->> cdi_watcher:refresh cdi cache
+ end
+ and
+ opt CRI
+ CRI ->> isulad: create container
+ isulad ->> isulad:parse cdi annotations/CDI_devices
+ isulad ->> isulad:generates an OCI specification with cdi info
+ end
+ end
+```
+
+# 接口描述
+## 3.1 结构体和常量说明
+
+```c
+// 实现CDI Spec规范中的json字段，参考 https://github1s.com/containerd/containerd/blob/main/vendor/tags.cncf.io/container-device-interface/specs-go/config.go#L9
+typedef struct {} cdi_spec;
+typedef struct {} cdi_spec_device;
+typedef struct {} cdi_spec_container_edits;
+typedef struct {} cdi_spec_device_node;
+typedef struct {} cdi_spec_mount;
+typedef struct {} cdi_spec_hook;
+
+struct cdi_cache_device {
+ const cdi_device *raw_device;
+ const struct cdi_cache_spec *cache_spec;
+};
+
+struct cdi_cache_spec {
+ cdi_spec *raw_spec;
+ char *vendor;
+ char *class;
+ char *path;
+ int priority;
+ map_t *devices; // MAP_STR_PTR devices[cdi_device.name] = cdi_cache_device*
+};
+
+struct cdi_cache_ops {
+ // injecting CDI devices into an OCI Spec.
+ // Resolver
+ int (*inject_devices)(struct cdi_cache *c, oci_runtime_spec *spec, string_array *devices);
+
+ // refreshing the cache of CDI Specs and devices.
+ // Refresher
+ int (*configure)(struct cdi_cache *c, string_array *spec_dirs);
+ int (*refresh)(struct cdi_cache *c);
+};
+
+struct cdi_watch {
+ int watcher_fd; // inotify fd
+ map_t *tracked; // MAP_STR_BOOL tracked[spec_dirs[i]] = bool
+ map_t *wd_dirs; // MAP_INT_STR wd_dirs[wd] = spec_dirs[i]
+};
+
+// Cache stores CDI Specs loaded from Spec directories.
+struct cdi_cache {
+ pthread_mutex_t mutex;
+ string_array *spec_dirs; // cdi-spec-dirs will scan for CDI Spec files
+ map_t *specs; // MAP_STR_PTR specs[vendor] = common_array of cdi_cache_spec*
+ // This map holding the reference to cdi device, the devices will not released when the map is freed.
+ map_t *devices; // MAP_STR_PTR devices[cdi_device.name] = cdi_cache_device*
+ bool refresh_error_flag;
+ bool auto_refresh;
+ struct cdi_watch *watch;
+};
+
+struct cdi_registry {
+ struct cdi_cache *cdi_cache;
+ struct cdi_cache_ops *ops;
+};
+```
+涉及修改现有的结构体：
+```c
+// 用于从CRI向executor传递devices数据
+typedef struct {
+ ... ...
+ char **cdi_requested_devices;
+ size_t cdi_requested_devices_len;
+} host_config;
+
+// isulad的daemon.json增加cdi相关的基本配置
+typedef struct {
+ ... ...
+ char **cdi_spec_dirs;
+ size_t cdi_spec_dirs_len;
+ bool enable_cdi;
+} isulad_daemon_configs;
+```
+
+## 3.2 接口说明
+CDIOperate的设计目标：使得CDI规范有关的内容不过多的对外暴露，降低CDI模块和外部的耦合。
+### RegistryOps
+```c
+int cdi_operate_registry_init(char **specs_dirs, size_t specs_dirs_len);
+
+int cdi_operate_refresh(void);
+
+int cdi_operate_inject_devices(oci_runtime_spec *spec, string_array *devices);
+
+int cdi_operate_parse_annotations(json_map_string_string *annotations, string_array **keys,
+ string_array **devices, char **error);
+```
+
+# 详细设计
+## daemon.json
+daemon.json中增加cdi配置：
+```json
+"cdi-spec-dirs": ["/etc/cdi", "/var/run/cdi"],
+"enable-cdi": true
+```
+## CDIOperate被调用点
+### CreateContainer
+创建新的Container时需要将devices插入OCI Spec。
+本次开发暂不涉及isula支持CDI。
+
+为什么已经有isulad-cdi_watcher线程了，在create的时候还需要refresh?
+1. isulad-cdi_watcher在触发inotify event后，如果为Rename、Remove、Write事件，会直接执行refresh，将重新扫描cdi-spec-dirs中的CDI Specs到内存。
+2. 而在create container时，cdi_refresh先检查tracked，tracked标记了是否所有目录都已被跟踪，如果不是才会触发refresh。tracked在cache生成时初始化为未跟踪。
+
+```mermaid
+flowchart TD
+O((begin))
+A(ContainerManagerService::CreateContainer <br> CRI创建容器)
+B(ContainerManagerService::CDIParseDevices <br> 解析CDI有关的字段)
+C(container_cb::container_create_cb <br> 创建容器的前置准备)
+D(spec::merge_all_specs <br> 准备生成OCI Spec)
+E(spec::spec_inject_cdi_devices <br> 解析需要的设备名)
+F(CDIOperate::cdi_refresh <br> 刷新cdi的缓存)
+G(CDIOperate::cdi_inject_devices <br> 将容器需要的设备的相关信息插入OCI Spec)
+Z((create end))
+O --> A --> B --> C --> D --> E --> F --> G --> Z
+```
+
+ContainerManagerService::CDIParseDevices 需要支持两种CDI devices解析方式：
+1. CDIOperate::cdi_parse_annotations 解析ContainerConfig.annotations()
+2. 解析ContainerConfig.CDI_devices()
+解析后放入container_create_request
+### main
+isulad 启动时，读取所有CDI Specs，初始化cache
+```mermaid
+flowchart TD
+O((begin))
+A(main)
+B(isulad_server_init_common <br> isulad服务端初始化)
+C(cdi::cdi_registry_init <br> 读取现有的CDI Specs <br> 并拉起新的线程监控CDI Specs的变化)
+Z(end)
+O --> A --> B --> C --> Z
+```
+
+## CDI 模块设计
+```mermaid
+flowchart TD
+ subgraph CDIoperate
+ operate
+ end
+
+ subgraph CDI
+ annotations
+ registry
+ cache
+ subgraph behavior
+ spec-dirs
+ spec
+ device
+ container-edits
+ version
+ parser
+ end
+ end
+
+ operate --> registry
+ operate --> annotations
+ registry --> cache
+ cache --> spec-dirs
+ cache --> device
+ cache --> spec
+ cache --> container-edits
+ device <--> spec
+ device --> parser
+ device --> container-edits
+ device --> parser
+ spec --> container-edits
+ spec --> version
+ version --> parser
+```
+
+### new cache
+```mermaid
+flowchart TD
+O((begin))
+A(ContainerManagerService::CreateContainer <br> CRI创建容器)
+B(ContainerManagerService::CDIParseDevices <br> 解析CDI有关的字段)
+C(container_cb::container_create_cb <br> 创建容器的前置准备)
+D(spec::merge_all_specs <br> 准备生成OCI Spec)
+E(spec::spec_inject_cdi_devices <br> 解析需要的设备名)
+F(CDIOperate::cdi_refresh <br> 刷新cdi的缓存)
+G(CDIOperate::cdi_inject_devices <br> 将容器需要的设备的相关信息插入OCI Spec)
+Z((create end))
+O --> A --> B --> C --> D --> E --> F --> G --> Z
+```
--
2.34.1

52
0073-bugfix-cdi-version-check.patch Normal file
View File

@ -0,0 +1,52 @@
From 4d8a89e0a3700253db044f6641d91ab10ad6ce10 Mon Sep 17 00:00:00 2001
From: liuxu <liuxu156@huawei.com>
Date: Sat, 20 Apr 2024 19:36:55 +0800
Subject: [PATCH 73/73] bugfix: cdi version check
Signed-off-by: liuxu <liuxu156@huawei.com>
---
src/daemon/modules/device/cdi/behavior/cdi_spec.c | 2 +-
src/daemon/modules/device/cdi/behavior/cdi_version.c | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/daemon/modules/device/cdi/behavior/cdi_spec.c b/src/daemon/modules/device/cdi/behavior/cdi_spec.c
index 235b1863..8783debc 100644
--- a/src/daemon/modules/device/cdi/behavior/cdi_spec.c
+++ b/src/daemon/modules/device/cdi/behavior/cdi_spec.c
@@ -199,7 +199,7 @@ static int cdi_spec_init(struct cdi_cache_spec *s)
return -1;
}
if (version_result) {
- ERROR("The spec version must be at least v%s", min_version);
+ ERROR("The %s spec version must be at least v%s", s->path, min_version);
return -1;
}
diff --git a/src/daemon/modules/device/cdi/behavior/cdi_version.c b/src/daemon/modules/device/cdi/behavior/cdi_version.c
index 882a965e..550f3107 100644
--- a/src/daemon/modules/device/cdi/behavior/cdi_version.c
+++ b/src/daemon/modules/device/cdi/behavior/cdi_version.c
@@ -138,9 +138,9 @@ static struct required_version_map g_valid_spec_versions[VALID_SPEC_VERSIONS_LEN
{CDI_V010, NULL},
{CDI_V020, NULL},
{CDI_V030, NULL},
- {CDI_V040, requires_v060},
+ {CDI_V040, requires_v040},
{CDI_V050, requires_v050},
- {CDI_V060, requires_v040}
+ {CDI_V060, requires_v060}
};
const char *cdi_minimum_required_version(cdi_spec *spec)
@@ -166,7 +166,7 @@ const char *cdi_minimum_required_version(cdi_spec *spec)
min_version = g_valid_spec_versions[i].version;
}
}
- if (strcmp(min_version, CDI_CURRENT_VERSION)) {
+ if (strcmp(min_version, CDI_CURRENT_VERSION) == 0) {
break;
}
}
--
2.34.1

11
iSulad.spec
View File

@ -1,5 +1,5 @@
%global _version 2.1.5
%global _release 5
%global _release 6
%global is_systemd 1
%global enable_criv1 1
%global enable_cdi 1
@ -86,6 +86,9 @@ Patch0067: 0067-cdi-support-modules-container_edits-parser.patch
Patch0068: 0068-cdi-invoke-cdi-operate-when-init-isulad-and-create-c.patch
Patch0069: 0069-bugfix-fix-cni_operate_ut-ut.patch
Patch0070: 0070-isolate-sandboxer-code-by-using-macro.patch
Patch0071: 0071-Remove-sandboxer-ut-if-sandboxer-is-not-enabled.patch
Patch0072: 0072-cdi-design-doc.patch
Patch0073: 0073-bugfix-cdi-version-check.patch
%ifarch x86_64 aarch64
Provides: libhttpclient.so()(64bit)
@ -342,6 +345,12 @@ fi
%endif
%changelog
* Sat Apr 20 2024 liuxu <liuxu156@huawei.com> - 2.1.5-6
- Type: update
- ID: NA
- SUG: NA
- DESC: upgrade from upstream
* Sat Apr 20 2024 liuxu <liuxu156@huawei.com> - 2.1.5-5
- Type: update
- ID: NA