upgrade to A-Tune v0.2
Signed-off-by: Zhipeng Xie <xiezhipeng1@huawei.com>
This commit is contained in:
Zhipeng Xie
parent
4335408875
commit
fdf64cfe54
3
.gitignore
vendored
3
.gitignore
vendored
@ -1,3 +0,0 @@
|
||||
/pkg/
|
||||
*.pyc
|
||||
database/atuned.db
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,56 +0,0 @@
|
||||
#
|
||||
# example of atuned profile configuration
|
||||
#
|
||||
|
||||
[main]
|
||||
# list it's parent profile
|
||||
|
||||
|
||||
[tip]
|
||||
# the recommended optimization, which should be performed manunaly
|
||||
|
||||
|
||||
[check]
|
||||
# check the environment
|
||||
|
||||
|
||||
[affinity.irq]
|
||||
# to change the affinity of irqs
|
||||
|
||||
|
||||
[affinity.task]
|
||||
# to change the affinity of tasks
|
||||
|
||||
|
||||
[bios]
|
||||
# to change the bios config
|
||||
|
||||
|
||||
[bootloader.grub2]
|
||||
# to change the grub2 config
|
||||
|
||||
|
||||
[kernel_config]
|
||||
# to change the kernel config
|
||||
|
||||
|
||||
[script]
|
||||
# the script extention of cpi
|
||||
|
||||
|
||||
[sysctl]
|
||||
# to change the /proc/sys/* config
|
||||
|
||||
|
||||
[sysfs]
|
||||
# to change the /sys/* config
|
||||
|
||||
|
||||
[systemctl]
|
||||
# to change the system service config
|
||||
|
||||
|
||||
[ulimit]
|
||||
# to change the resources limit of user
|
||||
|
||||
|
||||
121
License/LICENSE
121
License/LICENSE
@ -1,121 +0,0 @@
|
||||
木兰宽松许可证, 第1版
|
||||
|
||||
木兰宽松许可证， 第1版
|
||||
2019年8月 http://license.coscl.org.cn/MulanPSL
|
||||
|
||||
您对“软件”的复制、使用、修改及分发受木兰宽松许可证，第1版（“本许可证”）的如下条款的约束：
|
||||
|
||||
0. 定义
|
||||
|
||||
“软件”是指由“贡献”构成的许可在“本许可证”下的程序和相关文档的集合。
|
||||
|
||||
“贡献者”是指将受版权法保护的作品许可在“本许可证”下的自然人或“法人实体”。
|
||||
|
||||
“法人实体”是指提交贡献的机构及其“关联实体”。
|
||||
|
||||
“关联实体”是指，对“本许可证”下的一方而言，控制、受控制或与其共同受控制的机构，此处的控制是指有受控方或共同受控方至少50%直接或间接的投票权、资金或其他有价证券。
|
||||
|
||||
“贡献”是指由任一“贡献者”许可在“本许可证”下的受版权法保护的作品。
|
||||
|
||||
1. 授予版权许可
|
||||
|
||||
每个“贡献者”根据“本许可证”授予您永久性的、全球性的、免费的、非独占的、不可撤销的版权许可，您可以复制、使用、修改、分发其“贡献”，不论修改与否。
|
||||
|
||||
2. 授予专利许可
|
||||
|
||||
每个“贡献者”根据“本许可证”授予您永久性的、全球性的、免费的、非独占的、不可撤销的（根据本条规定撤销除外）专利许可，供您制造、委托制造、使用、许诺销售、销售、进口其“贡献”或以其他方式转移其“贡献”。前述专利许可仅限于“贡献者”现在或将来拥有或控制的其“贡献”本身或其“贡献”与许可“贡献”时的“软件”结合而将必然会侵犯的专利权利要求，不包括仅因您或他人修改“贡献”或其他结合而将必然会侵犯到的专利权利要求。如您或您的“关联实体”直接或间接地（包括通过代理、专利被许可人或受让人），就“软件”或其中的“贡献”对任何人发起专利侵权诉讼（包括反诉或交叉诉讼）或其他专利维权行动，指控其侵犯专利权，则“本许可证”授予您对“软件”的专利许可自您提起诉讼或发起维权行动之日终止。
|
||||
|
||||
3. 无商标许可
|
||||
|
||||
“本许可证”不提供对“贡献者”的商品名称、商标、服务标志或产品名称的商标许可，但您为满足第4条规定的声明义务而必须使用除外。
|
||||
|
||||
4. 分发限制
|
||||
|
||||
您可以在任何媒介中将“软件”以源程序形式或可执行形式重新分发，不论修改与否，但您必须向接收者提供“本许可证”的副本，并保留“软件”中的版权、商标、专利及免责声明。
|
||||
|
||||
5. 免责声明与责任限制
|
||||
|
||||
“软件”及其中的“贡献”在提供时不带任何明示或默示的担保。在任何情况下，“贡献者”或版权所有者不对任何人因使用“软件”或其中的“贡献”而引发的任何直接或间接损失承担责任，不论因何种原因导致或者基于何种法律理论,即使其曾被建议有此种损失的可能性。
|
||||
|
||||
条款结束。
|
||||
|
||||
如何将木兰宽松许可证，第1版，应用到您的软件
|
||||
|
||||
如果您希望将木兰宽松许可证，第1版，应用到您的新软件，为了方便接收者查阅，建议您完成如下三步：
|
||||
|
||||
1， 请您补充如下声明中的空白，包括软件名、软件的首次发表年份以及您作为版权人的名字；
|
||||
|
||||
2， 请您在软件包的一级目录下创建以“LICENSE”为名的文件，将整个许可证文本放入该文件中；
|
||||
|
||||
3， 请将如下声明文本放入每个源文件的头部注释中。
|
||||
|
||||
Copyright (c) [2019] [name of copyright holder]
|
||||
[Software Name] is licensed under the Mulan PSL v1.
|
||||
You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
You may obtain a copy of Mulan PSL v1 at:
|
||||
http://license.coscl.org.cn/MulanPSL
|
||||
THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
PURPOSE.
|
||||
See the Mulan PSL v1 for more details.
|
||||
|
||||
|
||||
Mulan Permissive Software License，Version 1
|
||||
|
||||
Mulan Permissive Software License，Version 1 (Mulan PSL v1)
|
||||
August 2019 http://license.coscl.org.cn/MulanPSL
|
||||
|
||||
Your reproduction, use, modification and distribution of the Software shall be subject to Mulan PSL v1 (this License) with following terms and conditions:
|
||||
|
||||
0. Definition
|
||||
|
||||
Software means the program and related documents which are comprised of those Contribution and licensed under this License.
|
||||
|
||||
Contributor means the Individual or Legal Entity who licenses its copyrightable work under this License.
|
||||
|
||||
Legal Entity means the entity making a Contribution and all its Affiliates.
|
||||
|
||||
Affiliates means entities that control, or are controlled by, or are under common control with a party to this License, ‘control’ means direct or indirect ownership of at least fifty percent (50%) of the voting power, capital or other securities of controlled or commonly controlled entity.
|
||||
|
||||
Contribution means the copyrightable work licensed by a particular Contributor under this License.
|
||||
|
||||
1. Grant of Copyright License
|
||||
|
||||
Subject to the terms and conditions of this License, each Contributor hereby grants to you a perpetual, worldwide, royalty-free, non-exclusive, irrevocable copyright license to reproduce, use, modify, or distribute its Contribution, with modification or not.
|
||||
|
||||
2. Grant of Patent License
|
||||
|
||||
Subject to the terms and conditions of this License, each Contributor hereby grants to you a perpetual, worldwide, royalty-free, non-exclusive, irrevocable (except for revocation under this Section) patent license to make, have made, use, offer for sale, sell, import or otherwise transfer its Contribution where such patent license is only limited to the patent claims owned or controlled by such Contributor now or in future which will be necessarily infringed by its Contribution alone, or by combination of the Contribution with the Software to which the Contribution was contributed, excluding of any patent claims solely be infringed by your or others’ modification or other combinations. If you or your Affiliates directly or indirectly (including through an agent, patent licensee or assignee）, institute patent litigation (including a cross claim or counterclaim in a litigation) or other patent enforcement activities against any individual or entity by alleging that the Software or any Contribution in it infringes patents, then any patent license granted to you under this License for the Software shall terminate as of the date such litigation or activity is filed or taken.
|
||||
|
||||
3. No Trademark License
|
||||
|
||||
No trademark license is granted to use the trade names, trademarks, service marks, or product names of Contributor, except as required to fulfill notice requirements in section 4.
|
||||
|
||||
4. Distribution Restriction
|
||||
|
||||
You may distribute the Software in any medium with or without modification, whether in source or executable forms, provided that you provide recipients with a copy of this License and retain copyright, patent, trademark and disclaimer statements in the Software.
|
||||
|
||||
5. Disclaimer of Warranty and Limitation of Liability
|
||||
|
||||
The Software and Contribution in it are provided without warranties of any kind, either express or implied. In no event shall any Contributor or copyright holder be liable to you for any damages, including, but not limited to any direct, or indirect, special or consequential damages arising from your use or inability to use the Software or the Contribution in it, no matter how it’s caused or based on which legal theory, even if advised of the possibility of such damages.
|
||||
|
||||
End of the Terms and Conditions
|
||||
|
||||
How to apply the Mulan Permissive Software License，Version 1 (Mulan PSL v1) to your software
|
||||
|
||||
To apply the Mulan PSL v1 to your work, for easy identification by recipients, you are suggested to complete following three steps:
|
||||
|
||||
i. Fill in the blanks in following statement, including insert your software name, the year of the first publication of your software, and your name identified as the copyright owner;
|
||||
ii. Create a file named “LICENSE” which contains the whole context of this License in the first directory of your software package;
|
||||
iii. Attach the statement to the appropriate annotated syntax at the beginning of each source file.
|
||||
|
||||
Copyright (c) [2019] [name of copyright holder]
|
||||
[Software Name] is licensed under the Mulan PSL v1.
|
||||
You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
You may obtain a copy of Mulan PSL v1 at:
|
||||
http://license.coscl.org.cn/MulanPSL
|
||||
THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
PURPOSE.
|
||||
|
||||
See the Mulan PSL v1 for more details.
|
||||
53
Makefile
53
Makefile
@ -1,53 +0,0 @@
|
||||
.PHONY: all clean modules
|
||||
|
||||
PKGPATH=pkg
|
||||
CURDIR=$(shell pwd)
|
||||
PREFIX ?= /usr
|
||||
LIBEXEC ?= libexec
|
||||
BINDIR = $(DESTDIR)$(PREFIX)/bin
|
||||
SYSTEMDDIR = $(DESTDIR)$(PREFIX)/lib/systemd/system
|
||||
|
||||
all: modules atune-adm atuned db
|
||||
|
||||
atune-adm:
|
||||
export GOPATH=`cd ../../;pwd` && go build -v -o $(PKGPATH)/atune-adm cmd/atune-adm/*.go
|
||||
|
||||
atuned:
|
||||
export GOPATH=`cd ../../;pwd` && go build -v -o $(PKGPATH)/atuned cmd/atuned/*.go
|
||||
|
||||
modules:
|
||||
export GOPATH=`cd ../../;pwd` && cd ${CURDIR}/modules/server/profile/ && go build -buildmode=plugin -o ${CURDIR}/pkg/daemon_profile_server.so *.go
|
||||
|
||||
clean:
|
||||
rm -rf $(PKGPATH)/*
|
||||
|
||||
db:
|
||||
sqlite3 database/atuned.db ".read database/init.sql"
|
||||
|
||||
install:
|
||||
@echo "BEGIN INSTALL A-Tune"
|
||||
mkdir -p $(BINDIR)
|
||||
mkdir -p $(SYSTEMDDIR)
|
||||
mkdir -p $(DESTDIR)/etc/atuned
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/lib/atuned/modules
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/share/atuned
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/scripts
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/analysis
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/collector
|
||||
mkdir -p $(DESTDIR)/var/lib/atuned
|
||||
mkdir -p $(DESTDIR)$(PREFIX)/share/bash-completion/completions
|
||||
install -m 640 pkg/daemon_profile_server.so $(DESTDIR)$(PREFIX)/lib/atuned/modules
|
||||
install -m 750 pkg/atune-adm $(BINDIR)
|
||||
install -m 750 pkg/atuned $(BINDIR)
|
||||
install -m 640 misc/atuned.service $(SYSTEMDDIR)
|
||||
install -m 640 misc/atuned.cnf $(DESTDIR)/etc/atuned/
|
||||
install -m 640 database/atuned.db $(DESTDIR)/var/lib/atuned/
|
||||
install -m 640 misc/atune-adm $(DESTDIR)$(PREFIX)/share/bash-completion/completions/
|
||||
install -m 640 misc/atune.logo $(DESTDIR)$(PREFIX)/share/atuned
|
||||
\cp -rf scripts/* $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/scripts/
|
||||
chmod -R 750 $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/scripts/
|
||||
\cp -rf analysis/* $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/analysis/
|
||||
chmod -R 750 $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/analysis/
|
||||
\cp -rf collection/* $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/collector/
|
||||
chmod -R 750 $(DESTDIR)$(PREFIX)/$(LIBEXEC)/atuned/collector/
|
||||
@echo "END INSTALL A-Tune"
|
||||
67
README.md
67
README.md
@ -1,67 +0,0 @@
|
||||
# A-Tune介绍
|
||||
|
||||
A-Tune是一款系统自优化的系统基础软件，该软件能够自动识别系统的负载类型，实现业务模型到资源模型的动态调度，充分发挥鲲鹏服务器的计算能力。
|
||||
|
||||
## 编译环境准备
|
||||
|
||||
第一步：**下载Go发行版**
|
||||
|
||||
下载链接：https://golang.org/
|
||||
|
||||
第二步：**现有目标环境清理**
|
||||
|
||||
这步，如果原有环境已经安装过go版本，要先卸载，卸载方法如下：
|
||||
|
||||
卸载就是清理
|
||||
|
||||
[root@localhost atune]# whereis go
|
||||
|
||||
[root@localhost atune]# whereis golang
|
||||
|
||||
[root@localhost atune]# whereis gocode #如果需要的话
|
||||
|
||||
//找到后删除
|
||||
|
||||
rm -rf xxx
|
||||
|
||||
第三步：**安装Go发行版**
|
||||
tar -C /usr/local -xzf go-xxxxxxxxx.tar.gz
|
||||
|
||||
解压后在目录 /usr/local/go中
|
||||
|
||||
第四步：**设置Go环境**
|
||||
|
||||
设置GOPATH 目录
|
||||
|
||||
mkdir -p /home/gocode
|
||||
|
||||
编辑环境
|
||||
vim /etc/profile
|
||||
在最后一行加入 安i插入
|
||||
|
||||
export GOROOT=/usr/local/go #设置为go安装的路径
|
||||
|
||||
export GOPATH=/home/gocode #默认安装包的路径
|
||||
|
||||
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
|
||||
|
||||
执行下面生效配置
|
||||
source /etc/profile
|
||||
|
||||
验证是否生效
|
||||
go version
|
||||
输出 go version go1.11 linux/amd64
|
||||
|
||||
|
||||
第五步：**解决protoc和protoc-gen-go三方包依赖**
|
||||
|
||||
1、下载protobuf源码：https://github.com/golang/protobuf
|
||||
|
||||
2、执行make install
|
||||
|
||||
3、把生成的protoc和proto-gen-go的二进制所在路径设置到系统PATH上：export PATH=$PATH:/XX/
|
||||
|
||||
第六步：**编译atune源码**
|
||||
|
||||
执行make all即可
|
||||
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
@ -1,75 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Application initialization, including log configuration, restful api registration.
|
||||
"""
|
||||
|
||||
from flask import Flask
|
||||
from flask_restful import Api
|
||||
import os
|
||||
import sys
|
||||
from resources import configurator
|
||||
from resources import monitor
|
||||
from resources import optimizer
|
||||
from resources import collector
|
||||
from resources import classification
|
||||
from resources import profile
|
||||
from resources import train
|
||||
|
||||
from configparser import ConfigParser
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler, SysLogHandler
|
||||
log = logging.getLogger('werkzeug')
|
||||
log.setLevel(logging.ERROR)
|
||||
|
||||
|
||||
app = Flask(__name__)
|
||||
api = Api(app)
|
||||
|
||||
api.add_resource(configurator.Configurator, '/v1/setting' ,'/setting')
|
||||
api.add_resource(monitor.Monitor, '/v1/monitor' ,'/monitor')
|
||||
api.add_resource(optimizer.Optimizer, '/v1/optimizer' ,'/v1/optimizer/<string:task_id>')
|
||||
api.add_resource(collector.Collector, '/v1/collector','/v1/collector')
|
||||
api.add_resource(classification.Classification, '/v1/classification','/v1/classification')
|
||||
api.add_resource(profile.Profile, '/v1/profile','/v1/profile')
|
||||
api.add_resource(train.Training, '/v1/training','/v1/training')
|
||||
|
||||
|
||||
def configLog(level):
|
||||
loggingFormat = logging.Formatter('atuned: %(asctime)s [%(levelname)s] %(name)s : %(message)s')
|
||||
syslogHandler = SysLogHandler(address="/dev/log", facility=SysLogHandler.LOG_LOCAL0)
|
||||
syslogHandler.setFormatter(loggingFormat)
|
||||
syslogHandler.setLevel(level)
|
||||
|
||||
rootLogger = logging.getLogger()
|
||||
rootLogger.addHandler(syslogHandler)
|
||||
|
||||
|
||||
def main(filename):
|
||||
if not os.path.exists(filename):
|
||||
print("conf file is not exist")
|
||||
return
|
||||
config = ConfigParser()
|
||||
config.read(filename)
|
||||
|
||||
level = logging.getLevelName(config.get("log", "level").upper())
|
||||
configLog(level)
|
||||
app.run(host="localhost", port=config.get("server", "rest_port"))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) != 2:
|
||||
print("lack of conf file parameter")
|
||||
sys.exit(-1)
|
||||
main(sys.argv[1])
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,12 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
@ -1,93 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
This class is used to find optimal settings and generate optimized profile.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
from multiprocessing import Process
|
||||
from skopt.optimizer import gp_minimize
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Optimizer(Process):
|
||||
def __init__(self, name, params, child_conn, engine = "bayes", max_eval=50):
|
||||
super(Optimizer, self).__init__(name=name)
|
||||
self.knobs = params
|
||||
self.child_conn = child_conn
|
||||
self.engine = engine
|
||||
self.max_eval = int(max_eval)
|
||||
|
||||
def build_space(self):
|
||||
objective_params_list = []
|
||||
for p in self.knobs:
|
||||
if p['type'] == 'discrete':
|
||||
if p['dtype'] == 'int':
|
||||
items = p['items']
|
||||
r = p['range']
|
||||
s = 1
|
||||
if 'step' in p.keys():
|
||||
s = p['step']
|
||||
if r is not None:
|
||||
for i in range(0, len(r), 2):
|
||||
items.extend(list(np.arange(r[i], r[i + 1], step=s)))
|
||||
objective_params_list.append(items)
|
||||
if p['dtype'] == 'string':
|
||||
items = p['options']
|
||||
keys = []
|
||||
for i in range(len(items)):
|
||||
keys.append(i)
|
||||
objective_params_list.append(keys)
|
||||
elif p['type'] == 'continuous':
|
||||
r = p['range']
|
||||
objective_params_list.append((r[0], r[1]))
|
||||
return objective_params_list
|
||||
|
||||
def run(self):
|
||||
def objective(var):
|
||||
for i, knob in enumerate(self.knobs):
|
||||
params[knob['name']] = var[i]
|
||||
print(params)
|
||||
self.child_conn.send(params)
|
||||
result = self.child_conn.recv()
|
||||
x = 0.0
|
||||
evalList = result.split(',')
|
||||
for value in evalList:
|
||||
num = float(value)
|
||||
x = x + num
|
||||
return x
|
||||
|
||||
ref = []
|
||||
params = {}
|
||||
for knob in self.knobs:
|
||||
ref.append(int(knob['ref']))
|
||||
try:
|
||||
logger.info("Running performance evaluation.......")
|
||||
ret = gp_minimize(objective, self.build_space(), n_calls=self.max_eval)
|
||||
logger.info("Minimization procedure has been completed.")
|
||||
except ValueError as v:
|
||||
logger.error('Value Error:', v)
|
||||
|
||||
for i, knob in enumerate(self.knobs):
|
||||
params[knob['name']] = ret.x[i]
|
||||
self.child_conn.send(params)
|
||||
logger.info("Optimized result: %s" % (params))
|
||||
logger.info("The optimized profile has been generated.")
|
||||
return params
|
||||
|
||||
def stopProcess(self):
|
||||
self.child_conn.close()
|
||||
self.terminate()
|
||||
@ -1,400 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
This class is used to train models and characterize system workload.
|
||||
"""
|
||||
|
||||
import os
|
||||
import glob
|
||||
import hashlib
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from sklearn import svm
|
||||
from collections import Counter
|
||||
from xgboost import XGBClassifier
|
||||
from sklearn.cluster import KMeans
|
||||
from sklearn.externals import joblib
|
||||
from sklearn.metrics import accuracy_score
|
||||
from sklearn.ensemble import AdaBoostClassifier
|
||||
from sklearn.model_selection import train_test_split as tts
|
||||
from sklearn.preprocessing import StandardScaler, LabelEncoder
|
||||
|
||||
class WorkloadCharacterization(object):
|
||||
def __init__(self, model_path):
|
||||
self.model_path = model_path
|
||||
self.scaler = StandardScaler()
|
||||
self.encoder = LabelEncoder()
|
||||
self.dataset = None
|
||||
self.X = None
|
||||
self.y = None
|
||||
|
||||
def parsing(self, data_path):
|
||||
df = [ ]
|
||||
print (data_path)
|
||||
csvfiles = glob.glob(data_path)
|
||||
for csv in csvfiles:
|
||||
print ("file:", csv)
|
||||
data = pd.read_csv(csv, index_col=None, header=None)
|
||||
df.append(data)
|
||||
dataset = pd.concat(df)
|
||||
self.dataset = dataset
|
||||
return dataset
|
||||
|
||||
def clustering(self, data_path):
|
||||
complete_path = os.path.join(data_path,"*.csv")
|
||||
self.parsing(complete_path)
|
||||
dataset = self.dataset.drop([28,41,42,43,44,57,58], axis=1)
|
||||
cpu_util = pd.concat([dataset.iloc[:,8], dataset.iloc[:,45]], axis=1, join='inner')
|
||||
io_util = pd.concat([dataset.iloc[:,3], dataset.iloc[:,19]], axis=1, join='inner')
|
||||
net_util = dataset.iloc[:,25]
|
||||
mem_total = dataset.iloc[:,29]
|
||||
|
||||
cpu_path = os.path.join(self.model_path,'cpu_clu.m')
|
||||
io_path = os.path.join(self.model_path,'io_clu.m')
|
||||
net_path = os.path.join(self.model_path,'net_clu.m')
|
||||
mem_path = os.path.join(self.model_path,'mem_clu.m')
|
||||
|
||||
cpu = KMeans(n_clusters=2)
|
||||
s = cpu.fit(cpu_util)
|
||||
joblib.dump(cpu, cpu_path)
|
||||
c_pred = cpu.predict(cpu_util)
|
||||
|
||||
io = KMeans(n_clusters=2)
|
||||
s = io.fit(io_util)
|
||||
joblib.dump(io, io_path)
|
||||
i_pred = io.predict(io_util)
|
||||
|
||||
net = KMeans(n_clusters=2)
|
||||
x= [[x] for x in net_util]
|
||||
s = net.fit(x)
|
||||
joblib.dump(net, net_path)
|
||||
n_pred = net.predict(x)
|
||||
|
||||
mem = KMeans(n_clusters=2)
|
||||
x= [[x] for x in mem_total]
|
||||
s = mem.fit(x)
|
||||
joblib.dump(mem, mem_path)
|
||||
m_pred = mem.predict(x)
|
||||
|
||||
result = pd.concat([pd.DataFrame(c_pred), pd.DataFrame(i_pred), pd.DataFrame(n_pred), pd.DataFrame(m_pred)], axis=1)
|
||||
cpu_result = result.iloc[:,0].map(lambda x:str(x))
|
||||
io_result = result.iloc[:,1].map(lambda x:str(x))
|
||||
net_result = result.iloc[:,2].map(lambda x:str(x))
|
||||
mem_result = result.iloc[:,3].map(lambda x:str(x))
|
||||
label = cpu_result.str.cat(io_result)
|
||||
label = label.str.cat(net_result)
|
||||
label = label.str.cat(mem_result)
|
||||
dataset.insert(0, 'label', label)
|
||||
dataset.to_csv('clusteringresult.csv')
|
||||
|
||||
def labelling(self, data):
|
||||
cpu_path = os.path.join(self.model_path,'cpu_clu.m')
|
||||
io_path = os.path.join(self.model_path,'io_clu.m')
|
||||
net_path = os.path.join(self.model_path,'net_clu.m')
|
||||
mem_path = os.path.join(self.model_path,'mem_clu.m')
|
||||
|
||||
cpu = joblib.load(cpu_path)
|
||||
io = joblib.load(io_path)
|
||||
net = joblib.load(net_path)
|
||||
mem = joblib.load(mem_path)
|
||||
|
||||
X = pd.DataFrame(data)
|
||||
|
||||
cpu_util = pd.concat([X.iloc[:,8], X.iloc[:,45]], axis=1, join='inner')
|
||||
io_util = pd.concat([X.iloc[:,3], X.iloc[:,19]], axis=1, join='inner')
|
||||
net_util = [[x] for x in X.iloc[:, 25]]
|
||||
mem_total = [[x] for x in X.iloc[:, 29]]
|
||||
|
||||
ctype = cpu.predict(cpu_util)
|
||||
itype = io.predict(io_util)
|
||||
ntype = net.predict(net_util)
|
||||
mtype = mem.predict(mem_total)
|
||||
cpred = Counter(ctype).most_common(1)[0]
|
||||
ipred = Counter(itype).most_common(1)[0]
|
||||
npred = Counter(ntype).most_common(1)[0]
|
||||
mpred = Counter(mtype).most_common(1)[0]
|
||||
|
||||
print ("The clustering result is", cpred[0], ipred[0], npred[0], mpred[0])
|
||||
print ("The accuracy of the label prediction are ", cpred[1]/len(data),
|
||||
ipred[1]/len(data), npred[1]/len(data), mpred[1]/len(data))
|
||||
return [cpred[0], ipred[0], npred[0], mpred[0]]
|
||||
|
||||
def svm_clf(self, clfpath, kernel):
|
||||
X_train, X_test, y_train, y_test = tts(self.X, self.y, test_size=0.3)
|
||||
|
||||
rbf = svm.SVC(kernel= kernel, C=200)
|
||||
rbf.fit(X_train, y_train)
|
||||
rbf_score = rbf.score(X_test, y_test)
|
||||
print ("the accuracy of rbf classification is %f" %rbf_score)
|
||||
joblib.dump(rbf, clfpath)
|
||||
|
||||
def ada_clf(self, clfpath):
|
||||
X_train, X_test, y_train, y_test = tts(self.X, self.y, test_size=0.3)
|
||||
adaboost = AdaBoostClassifier(n_estimators=1000, learning_rate=0.05)
|
||||
adaboost.fit(X_train, y_train)
|
||||
adaboost_score = adaboost.score(X_test, y_test)
|
||||
print ("the accuracy of adaboost is %f" %adaboost_score)
|
||||
print ("the feature importances of this classifier are", adaboost.feature_importances_)
|
||||
joblib.dump(adaboost, clfpath)
|
||||
|
||||
def xgb_clf(self, clfpath):
|
||||
X_train, X_test, y_train, y_test = tts(self.X, self.y, test_size=0.3)
|
||||
model = XGBClassifier(learning_rate=0.1, n_estimators=400, max_depth=6, min_child_weight = 0.5, gamma=0.01,
|
||||
subsample=1, colsample_btree=1, scale_pos_weight=1, random_state=27, slient = 0, alpha=100)
|
||||
model.fit(X_train,y_train)
|
||||
y_pred = model.predict(X_test)
|
||||
print ("the accuracy of xgboost is %f" %accuracy_score(y_test,y_pred))
|
||||
print ("the feature importances of this classifier are", model.feature_importances_)
|
||||
joblib.dump(model, clfpath)
|
||||
|
||||
def train(self, data_path):
|
||||
encoder_path = os.path.join(self.model_path,"encoder.pkl")
|
||||
complete_path = os.path.join(data_path,"*.csv")
|
||||
|
||||
io_path = os.path.join(data_path,"*0100.csv")
|
||||
net_path = os.path.join(data_path,"*0010.csv")
|
||||
cpu_path = os.path.join(data_path,"*1000.csv")
|
||||
cpumem_path = os.path.join(data_path,"*1001.csv")
|
||||
cpuio_path = os.path.join(data_path,"*1100.csv")
|
||||
cpunet_path = os.path.join(data_path,"*1010.csv")
|
||||
cpuiomem_path = os.path.join(data_path,"*1101.csv")
|
||||
cpunetmem_path = os.path.join(data_path,"*1011.csv")
|
||||
|
||||
io_clf = os.path.join(self.model_path,"io_clf.m")
|
||||
net_clf = os.path.join(self.model_path,"net_clf.m")
|
||||
cpu_clf = os.path.join(self.model_path,"cpu_clf.m")
|
||||
cpumem_clf = os.path.join(self.model_path,"cpumem_clf.m")
|
||||
cpuio_clf = os.path.join(self.model_path,"cpuio_clf.m")
|
||||
cpuiomem_clf = os.path.join(self.model_path,"cpuiomem_clf.m")
|
||||
cpunet_clf = os.path.join(self.model_path,"cpunet_clf.m")
|
||||
cpunetmem_clf = os.path.join(self.model_path,"cpunetmem_clf.m")
|
||||
|
||||
io_scl = os.path.join(self.model_path,"io_scaler.pkl")
|
||||
net_scl = os.path.join(self.model_path,"net_scaler.pkl")
|
||||
cpu_scl = os.path.join(self.model_path,"cpu_scaler.pkl")
|
||||
cpumem_scl = os.path.join(self.model_path,"cpumem_scaler.pkl")
|
||||
cpuio_scl = os.path.join(self.model_path,"cpuio_scaler.pkl")
|
||||
cpuiomem_scl = os.path.join(self.model_path,"cpuiomem_scaler.pkl")
|
||||
cpunet_scl = os.path.join(self.model_path,"cpunet_scaler.pkl")
|
||||
cpunetmem_scl = os.path.join(self.model_path,"cpunetmem_scaler.pkl")
|
||||
|
||||
self.parsing(complete_path)
|
||||
self.y = self.encoder.fit_transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.encoder, encoder_path)
|
||||
|
||||
self.parsing(io_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, io_scl)
|
||||
self.svm_clf(io_clf, 'sigmoid')
|
||||
print ("The classifier for io-intensive workload has been generated.")
|
||||
|
||||
self.parsing(net_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, net_scl)
|
||||
self.svm_clf(net_clf, 'sigmoid')
|
||||
print ("The classifier for net-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpu_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpu_scl)
|
||||
self.xgb_clf(cpu_clf)
|
||||
print ("The classifier for cpu-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpumem_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpumem_scl)
|
||||
self.svm_clf(cpumem_clf, 'rbf')
|
||||
print ("The classifier for cpumem-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpuio_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpuio_scl)
|
||||
self.svm_clf(cpuio_clf, 'rbf')
|
||||
print ("The classifier for cpuio-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpunet_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpunet_scl)
|
||||
self.svm_clf(cpunet_clf, 'rbf')
|
||||
print ("The classifier for cpunet-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpuiomem_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpuiomem_scl)
|
||||
self.svm_clf(cpuiomem_clf, 'rbf')
|
||||
print ("The classifier for cpuiomem-intensive workload has been generated.")
|
||||
|
||||
self.parsing(cpunetmem_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:,:-1])
|
||||
self.y = self.encoder.transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, cpunetmem_scl)
|
||||
self.svm_clf(cpunetmem_clf, 'rbf')
|
||||
print ("The classifier for cpunetmem-intensive workload has been generated.")
|
||||
|
||||
def identify(self, x):
|
||||
x = pd.DataFrame(x)
|
||||
x = x.drop([28,41,42,43,44,57,58], axis=1)
|
||||
l = self.labelling(x)
|
||||
|
||||
encoder_path = os.path.join(self.model_path,'encoder.pkl')
|
||||
|
||||
io_scaler = os.path.join(self.model_path,'io_scaler.pkl')
|
||||
net_scaler = os.path.join(self.model_path,'net_scaler.pkl')
|
||||
cpu_scaler = os.path.join(self.model_path,'cpu_scaler.pkl')
|
||||
cpuio_scaler = os.path.join(self.model_path,'cpuio_scaler.pkl')
|
||||
cpuiomem_scaler = os.path.join(self.model_path,'cpuiomem_scaler.pkl')
|
||||
cpunet_scaler = os.path.join(self.model_path,'cpunet_scaler.pkl')
|
||||
cpumem_scaler = os.path.join(self.model_path,'cpumem_scaler.pkl')
|
||||
cpunetmem_scaler = os.path.join(self.model_path,'cpunetmem_scaler.pkl')
|
||||
|
||||
io_clf = os.path.join(self.model_path,'io_clf.m')
|
||||
net_clf = os.path.join(self.model_path,'net_clf.m')
|
||||
cpu_clf = os.path.join(self.model_path,'cpu_clf.m')
|
||||
cpumem_clf = os.path.join(self.model_path,'cpumem_clf.m')
|
||||
cpuio_clf = os.path.join(self.model_path,'cpuio_clf.m')
|
||||
cpunet_clf = os.path.join(self.model_path,'cpunet_clf.m')
|
||||
cpuiomem_clf = os.path.join(self.model_path,'cpuiomem_clf.m')
|
||||
cpunetmem_clf = os.path.join(self.model_path,'cpunetmem_clf.m')
|
||||
|
||||
if l == [0, 0, 0, 0]:
|
||||
resourcelimit = " "
|
||||
print ("The workload has been identified as non-intensive type")
|
||||
return resourcelimit, "default", 1
|
||||
elif l == [0, 0, 0, 1] :
|
||||
resourcelimit = "io"
|
||||
print ("The workload has been identified as mem-intensive type")
|
||||
return resourcelimit, "in-memory_computing", 1
|
||||
elif l == [0, 0, 1, 0] :
|
||||
resourcelimit = "network"
|
||||
scaler = joblib.load(net_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as network-intensive type")
|
||||
model = joblib.load(net_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [0, 1, 0, 0] :
|
||||
resourcelimit = "io"
|
||||
scaler = joblib.load(io_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as io-intensive type")
|
||||
model = joblib.load(io_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 0, 0, 0]:
|
||||
resourcelimit = "cpu"
|
||||
scaler = joblib.load(cpu_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-intensive type")
|
||||
model = joblib.load(cpu_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 0, 0, 1] :
|
||||
resourcelimit = "cpu,memory"
|
||||
scaler = joblib.load(cpumem_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-mem-intensive type")
|
||||
model = joblib.load(cpumem_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 0, 1, 0] :
|
||||
resourcelimit = "cpu,io"
|
||||
scaler = joblib.load(cpunet_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-net-intensive type")
|
||||
model = joblib.load(cpunet_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 0, 1, 1] :
|
||||
resourcelimit = "cpu,network,memory"
|
||||
scaler = joblib.load(cpunetmem_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-net-mem-intensive type")
|
||||
model = joblib.load(cpunetmem_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 1, 0, 0]:
|
||||
resourcelimit = "cpu,io"
|
||||
scaler = joblib.load(cpuio_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-io-intensive type")
|
||||
model = joblib.load(cpuio_clf)
|
||||
result = model.predict(x)
|
||||
elif l == [1, 1, 0, 1] :
|
||||
resourcelimit = "cpu,io,memory"
|
||||
scaler = joblib.load(cpuiomem_scaler)
|
||||
x = scaler.transform(x)
|
||||
print ("The workload has been identified as cpu-io-mem-intensive type")
|
||||
model = joblib.load(cpuiomem_clf)
|
||||
result = model.predict(x)
|
||||
elif l ==[1, 1, 1, 1]:
|
||||
resourcelimit = "cpu,io,network,memory"
|
||||
print ("The workload has been identified as cpu-io-net-mem-intensive type")
|
||||
return resourcelimit, "big_database", 1
|
||||
else:
|
||||
resourcelimit = ""
|
||||
print ("Error: Unsupported classifier type")
|
||||
return resourcelimit, "default", 0
|
||||
|
||||
encoder = joblib.load(encoder_path)
|
||||
result = encoder.inverse_transform(result)
|
||||
print (result)
|
||||
prediction = Counter(result).most_common(1)[0]
|
||||
confidence = prediction[1]/len(result)
|
||||
if confidence > 0.5:
|
||||
return resourcelimit, prediction[0], confidence
|
||||
else:
|
||||
return resourcelimit, "default", confidence
|
||||
|
||||
def retrain(self, data_path, custom_path):
|
||||
custom_path = os.path.abspath(custom_path)
|
||||
(dirname, filename) = os.path.split(custom_path)
|
||||
(modelname, extension) = os.path.splitext(filename)
|
||||
scalername = modelname + '_scaler.pkl'
|
||||
encodername = modelname + '_encoder.pkl'
|
||||
|
||||
data_path = os.path.join(data_path,"*.csv")
|
||||
self.parsing(data_path)
|
||||
self.X = self.scaler.fit_transform(self.dataset.iloc[:, :-1])
|
||||
self.y = self.encoder.fit_transform(self.dataset.iloc[:, -1])
|
||||
joblib.dump(self.scaler, os.path.join(dirname, scalername))
|
||||
joblib.dump(self.encoder, os.path.join(dirname, encodername))
|
||||
|
||||
class_num = len(self.encoder.classes_)
|
||||
if class_num == 1:
|
||||
self.y[-1] = self.y[-1] + 1
|
||||
self.svm_clf(custom_path)
|
||||
|
||||
def reidentify(self, x, custom_path):
|
||||
custom_path = os.path.abspath(custom_path)
|
||||
(dirname, filename) = os.path.split(custom_path)
|
||||
(modelname, extension) = os.path.splitext(filename)
|
||||
scalername = modelname + '_scaler.pkl'
|
||||
encodername = modelname + '_encoder.pkl'
|
||||
|
||||
scaler = joblib.load(os.path.join(dirname, scalername))
|
||||
encoder = joblib.load(os.path.join(dirname, encodername))
|
||||
model = joblib.load(custom_path)
|
||||
|
||||
x = scaler.transform(x)
|
||||
result = model.predict(x)
|
||||
result = encoder.inverse_transform(result)
|
||||
print (result)
|
||||
|
||||
prediction = Counter(result).most_common(1)[0]
|
||||
confidence = prediction[1]/len(result)
|
||||
if confidence > 0.5:
|
||||
return prediction[0], confidence
|
||||
else:
|
||||
return "default", confidence
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
@ -1,30 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = [
|
||||
"affinity",
|
||||
"bios",
|
||||
"bootloader",
|
||||
"kernel_config",
|
||||
"script",
|
||||
"sysctl",
|
||||
"sysfs",
|
||||
"systemctl",
|
||||
"ulimit",
|
||||
"common"]
|
||||
|
||||
from . import *
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["irq", "task"]
|
||||
|
||||
from . import *
|
||||
@ -1,117 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the affinity of irqs.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import os
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class IrqAffinity(Configurator):
|
||||
"""To change the affinity of irqs"""
|
||||
_module = "AFFINITY"
|
||||
_submod = "IRQ"
|
||||
_option = "/proc/irq"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
self._set.__func__.__doc__ = Configurator._set.__doc__ % (
|
||||
'irq or irq_name', 'cpumask in hex, no "0x" prefix, "," is permitted')
|
||||
|
||||
def __get_irq_id(self, key):
|
||||
if key.isdecimal():
|
||||
id = key
|
||||
name = None
|
||||
else:
|
||||
id = None
|
||||
name = key
|
||||
|
||||
if id is None:
|
||||
irqs = sorted(os.listdir("/sys/kernel/irq/"), key=lambda x: int(x))
|
||||
for irq in irqs:
|
||||
with open("/sys/kernel/irq/{}/actions".format(irq), 'r') as f:
|
||||
action = f.read().replace("\n", "")
|
||||
if action == name:
|
||||
id = irq
|
||||
break
|
||||
return id
|
||||
|
||||
def _get(self, key):
|
||||
id = self.__get_irq_id(key)
|
||||
if id is None:
|
||||
err = LookupError("Fail to find irq {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
f = open("{opt}/{id}/smp_affinity".format(opt=self._option,
|
||||
id=id),
|
||||
mode='r',
|
||||
buffering=-1,
|
||||
encoding=None,
|
||||
errors=None,
|
||||
newline=None,
|
||||
closefd=True)
|
||||
ret = f.read().replace(",", "")
|
||||
f.close()
|
||||
return ret
|
||||
|
||||
def _set(self, key, value):
|
||||
id = self.__get_irq_id(key)
|
||||
if id is None:
|
||||
err = LookupError("Fail to find irq {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
mask = value.replace(",", "")
|
||||
f = open("{opt}/{id}/smp_affinity".format(opt=self._option,
|
||||
id=id),
|
||||
mode='w',
|
||||
buffering=-1,
|
||||
encoding=None,
|
||||
errors=None,
|
||||
newline=None,
|
||||
closefd=True)
|
||||
f.write(mask)
|
||||
f.close()
|
||||
return 0
|
||||
|
||||
def _check(self, config1, config2):
|
||||
config1 = config1.replace(",", "")
|
||||
config2 = config2.replace(",", "")
|
||||
return int(config1, base=16) == int(config2, base=16)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = IrqAffinity("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,125 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the affinity of tasks.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TaskAffinity(Configurator):
|
||||
"""To change the affinity of tasks"""
|
||||
_module = "AFFINITY"
|
||||
_submod = "TASK"
|
||||
_option = "taskset -p"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
self._set.__func__.__doc__ = Configurator._set.__doc__ % (
|
||||
'pid or task_name', 'cpumask in hex, no "0x" prefix, "," is permitted')
|
||||
|
||||
def __get_task_id(self, key):
|
||||
if key.isdecimal():
|
||||
id = key
|
||||
name = None
|
||||
else:
|
||||
id = None
|
||||
name = key.replace("(", "\(")
|
||||
name = name.replace(")", "\)")
|
||||
|
||||
if id is None:
|
||||
output = subprocess.check_output("ps -e".split(),
|
||||
stderr=subprocess.STDOUT)
|
||||
pattern = "^\ *(\d.*?)\ +(.*?)\ +(.*?)\ +" + name
|
||||
searchObj = re.search(
|
||||
pattern,
|
||||
output.decode(),
|
||||
re.ASCII | re.MULTILINE)
|
||||
if searchObj is not None:
|
||||
id = searchObj.group(1)
|
||||
return id
|
||||
|
||||
def _get(self, key):
|
||||
id = self.__get_task_id(key)
|
||||
if id is None:
|
||||
err = LookupError("Fail to find task {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{opt} {pid}".format(
|
||||
opt=self._option,
|
||||
pid=id).split(),
|
||||
stderr=subprocess.STDOUT)
|
||||
pattern = "^pid.*?current affinity mask:\ (.+)"
|
||||
searchObj = re.search(
|
||||
pattern,
|
||||
output.decode(),
|
||||
re.ASCII | re.MULTILINE)
|
||||
if searchObj is None:
|
||||
err = GetConfigError("Fail to find {} affinity".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return searchObj.group(1)
|
||||
|
||||
def _set(self, key, value):
|
||||
id = self.__get_task_id(key)
|
||||
if id is None:
|
||||
err = LookupError("Fail to find task {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
mask = value.replace(",", "")
|
||||
return subprocess.call(
|
||||
"{opt} {mask} {pid}".format(
|
||||
opt=self._option,
|
||||
mask=mask,
|
||||
pid=id).split(),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL)
|
||||
|
||||
def _check(self, config1, config2):
|
||||
config1 = config1.replace(",", "")
|
||||
config2 = config2.replace(",", "")
|
||||
return int(config1, base=16) == int(config2, base=16)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = TaskAffinity("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["bios"]
|
||||
|
||||
from . import *
|
||||
@ -1,81 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the bios config.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Bios(Configurator):
|
||||
"""To change the bios config"""
|
||||
_module = "BIOS"
|
||||
_submod = "BIOS"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
|
||||
def _set(self, key, value):
|
||||
raise NeedConfigWarning(
|
||||
"Please change the BIOS configuration {key} to {val}.".format(
|
||||
key=key, val=value))
|
||||
|
||||
def _get(self, key):
|
||||
if key.lower() == "version":
|
||||
output = subprocess.check_output(
|
||||
"dmidecode -t bios | grep -Po '(?<=Version:)(.*)' | sed 's/^ *//g' | sed 's/ *$//g'",
|
||||
shell=True)
|
||||
return output.decode()
|
||||
elif key.lower() == "hpre_support":
|
||||
ret = subprocess.call(
|
||||
"lspci | grep HPRE",
|
||||
shell=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL)
|
||||
if ret == 0:
|
||||
return "yes"
|
||||
else:
|
||||
return "no"
|
||||
else:
|
||||
err = NotImplementedError(
|
||||
"{} can not get {}".format(
|
||||
self._module, key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
return ""
|
||||
|
||||
def _resume(self, key, value):
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = Bios("UT")
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,23 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(__file__))
|
||||
__all__ = ["grub2", "cmdline"]
|
||||
|
||||
from . import *
|
||||
@ -1,43 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Utils class.
|
||||
"""
|
||||
|
||||
|
||||
class Utils:
|
||||
"""Utils class"""
|
||||
|
||||
@staticmethod
|
||||
def get_keypos(str, key):
|
||||
keys = [" " + key + "=", " " + key + " ", " " + key + "\n"]
|
||||
ret = []
|
||||
for k in keys:
|
||||
pos = str.rfind(k)
|
||||
if pos != -1:
|
||||
pos += 1
|
||||
ret.append(pos)
|
||||
return max(ret)
|
||||
|
||||
@staticmethod
|
||||
def get_value(key):
|
||||
with open("/proc/cmdline", 'r') as f:
|
||||
active_cmd = f.read()
|
||||
keypos = Utils.get_keypos(active_cmd, key)
|
||||
active = None
|
||||
if keypos != -1:
|
||||
config = active_cmd[keypos:].split()[0]
|
||||
if config.find("=") != -1:
|
||||
active = config.split("=")[1]
|
||||
return active
|
||||
@ -1,40 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to collect the cmdline info.
|
||||
"""
|
||||
import sys
|
||||
from configurator.common import *
|
||||
from bootutils import Utils
|
||||
|
||||
|
||||
class Cmdline(Configurator):
|
||||
"""To collect the cmdline info"""
|
||||
_module = "BOOTLOADER"
|
||||
_submod = "CMDLINE"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
|
||||
def _get(self, key):
|
||||
return Utils.get_value(key)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key[=value]')
|
||||
sys.exit(-1)
|
||||
ct = Cmdline("UT1")
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,12 +0,0 @@
|
||||
{
|
||||
"hugepagesz": [
|
||||
"64k",
|
||||
"64K",
|
||||
"2m",
|
||||
"2M",
|
||||
"32m",
|
||||
"32M",
|
||||
"1g",
|
||||
"1G"
|
||||
]
|
||||
}
|
||||
@ -1,145 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the grub2 config.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import random
|
||||
import shutil
|
||||
import re
|
||||
from bootutils import Utils
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Grub2(Configurator):
|
||||
"""To change the grub2 config"""
|
||||
_module = "BOOTLOADER"
|
||||
_submod = "GRUB2"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
__cfg_files = ["/etc/grub2.cfg", "/etc/grub2-efi.cfg"]
|
||||
for f in __cfg_files:
|
||||
if os.path.isfile(f):
|
||||
self.__cfg_file = f
|
||||
break
|
||||
self.__kernel_ver = subprocess.check_output("uname -r".split()).decode(
|
||||
).replace("\n", "")
|
||||
|
||||
def __get_cfg_entry(self, version):
|
||||
with open(self.__cfg_file, 'r') as f:
|
||||
ctx = f.read()
|
||||
pattern = re.compile(
|
||||
"\n(menuentry)[^\{\}]*?\{[^\{\}]*?\n\s+(linux)[^\{\}]*?" +
|
||||
version.replace(
|
||||
".",
|
||||
"\.") +
|
||||
"[^\{\}]*?(\})",
|
||||
re.ASCII | re.DOTALL)
|
||||
searchObj = pattern.search(ctx)
|
||||
start = searchObj.span(1)
|
||||
cmd = searchObj.span(2)
|
||||
end = searchObj.span(3)
|
||||
return {"START": start[0], "CMD": cmd[0], "END": end[1]}
|
||||
|
||||
def _get(self, key):
|
||||
entry = self.__get_cfg_entry(self.__kernel_ver)
|
||||
with open(self.__cfg_file, 'r') as f:
|
||||
f.seek(entry["CMD"])
|
||||
cmd = f.readline()
|
||||
|
||||
keypos = Utils.get_keypos(cmd, key)
|
||||
if keypos == -1:
|
||||
err = GetConfigError("Fail to find {} config".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
config = cmd[keypos:].split()[0]
|
||||
if config.find("=") != -1:
|
||||
return config.split("=")[1]
|
||||
else:
|
||||
return None
|
||||
|
||||
@pre_check(
|
||||
Configurator._precheck,
|
||||
os.path.split(
|
||||
os.path.realpath(__file__))[0] +
|
||||
"/grub2.json")
|
||||
def _set(self, key, value):
|
||||
entry = self.__get_cfg_entry(self.__kernel_ver)
|
||||
if value is None:
|
||||
new = key
|
||||
else:
|
||||
new = "{key}={val}".format(key=key, val=value)
|
||||
|
||||
with open(self.__cfg_file, 'r+') as f:
|
||||
f.seek(entry["CMD"])
|
||||
cmd = f.readline()
|
||||
|
||||
keypos = Utils.get_keypos(cmd, key)
|
||||
if keypos != -1:
|
||||
old = cmd[keypos:].split("\n")[0].split()[0]
|
||||
file_modify(f, entry["CMD"] + keypos,
|
||||
entry["CMD"] + keypos + len(old) - 1, new)
|
||||
else:
|
||||
file_modify(f, entry["CMD"] + len(cmd) - 1, -1, " " + new)
|
||||
|
||||
active = Utils.get_value(key)
|
||||
if value == active:
|
||||
return 0
|
||||
raise NeedRebootWarning(
|
||||
"Need reboot to make the config change of grub2 effect.")
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
name = os.path.basename(self.__cfg_file)
|
||||
bak_file = "{path}/{file}{ver}".format(path=rollback_info, file=name,
|
||||
ver=random.random())
|
||||
shutil.copy(self.__cfg_file, bak_file)
|
||||
return "CPI_ROLLBACK_INFO = {}".format(bak_file)
|
||||
|
||||
def _resume(self, key, value):
|
||||
if key != "CPI_ROLLBACK_INFO":
|
||||
err = ValueError("unsupported resume type: {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
shutil.copy(value, self.__cfg_file)
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key[=value]')
|
||||
sys.exit(-1)
|
||||
ct = Grub2("UT1")
|
||||
save = ct.backup(sys.argv[1], "./1/")
|
||||
print(save)
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
print(ct.resume(save))
|
||||
@ -1,380 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The base class of the configuration, used to set the value, get the value of the given key,
|
||||
backup from the given config and resume from the saved config info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import json
|
||||
from public import *
|
||||
from functools import wraps
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Configurator():
|
||||
"""Base class for configurators"""
|
||||
|
||||
# sub class should init these
|
||||
_module = "UNKNOWN"
|
||||
_submod = "UNKNOWN"
|
||||
|
||||
# for inner options usage
|
||||
_option = ""
|
||||
|
||||
def __init__(self, user=None):
|
||||
"""
|
||||
Initialize.
|
||||
|
||||
:param user(optional): "UT" for unit test, others are ignored
|
||||
:returns: None
|
||||
:raises: None
|
||||
"""
|
||||
self._user = user
|
||||
|
||||
def module(self):
|
||||
"""
|
||||
Get the the module of this configurator.
|
||||
|
||||
:param: None
|
||||
:returns: The module of this configurator
|
||||
:raises: None
|
||||
"""
|
||||
return self._module
|
||||
|
||||
def submod(self):
|
||||
"""
|
||||
Get the the sub module of this configurator.
|
||||
|
||||
:param: None
|
||||
:returns: The sub module of this configurator
|
||||
:raises: None
|
||||
"""
|
||||
return self._submod
|
||||
|
||||
def _getopt(self):
|
||||
"""
|
||||
Get the the inner option of this configurator.
|
||||
Multi-options should be splited by ";".
|
||||
|
||||
:param: None
|
||||
:returns: The iterator for get all inner options
|
||||
:raises: None
|
||||
"""
|
||||
return iter(self._option.split(";"))
|
||||
|
||||
def set(self, config):
|
||||
"""
|
||||
Set the given config.
|
||||
|
||||
:param config: The config to be setted, string like "key = value"
|
||||
:returns None: Success
|
||||
:returns NeedRebootWarning: Success, but need reboot
|
||||
:returns SetConfigError: Fail, fail in _set()
|
||||
:returns Exceptions: Fail, error in _set()
|
||||
:raises Exceptions: Error, unexpected errors
|
||||
"""
|
||||
cfg = self._getcfg(config)
|
||||
try:
|
||||
ret = self._set(cfg[0], cfg[1])
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
|
||||
if (0 == ret) and self._check(cfg[1], self.get(cfg[0])):
|
||||
err = None
|
||||
elif cfg[1] is None:
|
||||
err = SetConfigError(
|
||||
"Fail to set {mod}.{sub} config: {key}".format(
|
||||
mod=self.module(), sub=self.submod(), key=cfg[0]))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
else:
|
||||
err = SetConfigError("Fail to set {mod}.{sub} config: {key}={val}".format(
|
||||
mod=self.module(), sub=self.submod(), key=cfg[0], val=cfg[1]))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
|
||||
def _precheck(self, key, value, file):
|
||||
"""
|
||||
The common method to precheck config.
|
||||
|
||||
:param key: The config key
|
||||
:param value: The config value
|
||||
:param file: The check file
|
||||
:returns None: Success
|
||||
:raises TypeError: Fail, invalid rule
|
||||
:raises ValueError: Fail, invalid value
|
||||
:raises KeyError: Fail, invalid key
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
with open(file, 'r') as f:
|
||||
ctx = f.read()
|
||||
check_rules = json.loads(ctx)
|
||||
|
||||
if (type(check_rules).__name__ != 'dict'):
|
||||
raise TypeError("Invalid rule file")
|
||||
|
||||
rule = check_rules.get(key)
|
||||
if (type(rule).__name__ == 'NoneType'):
|
||||
raise KeyError('Invalid key "{}"'.format(key))
|
||||
elif (type(rule).__name__ == 'list'):
|
||||
if value in rule:
|
||||
return None
|
||||
elif (type(rule).__name__ == 'dict'):
|
||||
start = rule.get("start")
|
||||
end = rule.get("end")
|
||||
step = rule.get("step")
|
||||
try:
|
||||
ranges = list(range(start, end, step))
|
||||
except BaseException:
|
||||
raise TypeError("Invalid rule")
|
||||
if int(value) in ranges:
|
||||
return None
|
||||
else:
|
||||
raise TypeError(
|
||||
"Invalid rule type: {}".format(
|
||||
type(rule).__name__))
|
||||
raise ValueError('Invalid value "{}" for key "{}"'.format(value, key))
|
||||
|
||||
def _set(self, key, value):
|
||||
"""
|
||||
The inner method to set config.
|
||||
The sub class should implement this method.
|
||||
|
||||
:param key: The config key,
|
||||
[%s]
|
||||
:param value: The config value,
|
||||
[%s]
|
||||
:returns 0: Success
|
||||
:returns errno: Fail
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
err = NotImplementedError("_set method is not implemented")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def get(self, key):
|
||||
"""
|
||||
Get the given config.
|
||||
|
||||
:param key: The config to be getted, string like "key"
|
||||
:returns None: Success
|
||||
:returns value: Success, config value string
|
||||
:returns Exceptions: Fail, error in _get()
|
||||
:raises: None
|
||||
"""
|
||||
try:
|
||||
ret = self._get(key)
|
||||
if ret is not None:
|
||||
ret = ret.replace('\n', ' ').strip()
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
return ret
|
||||
|
||||
def _get(self, key):
|
||||
"""
|
||||
The inner method to get config.
|
||||
The sub class should implement this method.
|
||||
|
||||
:param key: The config key
|
||||
:returns None: Success
|
||||
:returns value: Success, config value string
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
err = NotImplementedError("_get method is not implemented")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def _getcfg(self, para):
|
||||
"""
|
||||
Get the the key and value from the config string.
|
||||
|
||||
:param para: The config string
|
||||
:returns list: Success, e.g. ["key", "value" or None]
|
||||
:raises Exceptions: Error, unexpected errors
|
||||
"""
|
||||
cfg = para.split("=", 1)
|
||||
for i in range(0, len(cfg)):
|
||||
cfg[i] = cfg[i].strip()
|
||||
if len(cfg) == 1:
|
||||
cfg.append(None)
|
||||
return cfg
|
||||
|
||||
def _check(self, config1, config2):
|
||||
"""
|
||||
Check whether the given configs are the same.
|
||||
|
||||
:param config1: The 1st config value string
|
||||
:param config2: The 2nd config value string
|
||||
:returns True: Same
|
||||
:returns False: Different
|
||||
:raises: None
|
||||
"""
|
||||
return config1 == config2
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
"""
|
||||
The inner method to backup config.
|
||||
The sub class should implement this method if needed.
|
||||
|
||||
:param key: The config key
|
||||
:param rollback_info: The additional info for rollback, mostly a path
|
||||
:returns value: Success, config info
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
val = self._get(key).replace('\n', ' ').strip()
|
||||
return "{} = {}".format(key, val)
|
||||
|
||||
def backup(self, config, rollback_info):
|
||||
"""
|
||||
Backup from the given config.
|
||||
|
||||
:param config: The config to be setted, string like "key = value"
|
||||
:param rollback_info: The additional info for rollback, mostly a path
|
||||
:returns Exceptions: Fail, error in _backup()
|
||||
:returns value: Success, config info
|
||||
:raises: None
|
||||
"""
|
||||
cfg = self._getcfg(config)
|
||||
try:
|
||||
ret = self._backup(cfg[0], rollback_info)
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
return ret
|
||||
|
||||
def _resume(self, key, value):
|
||||
"""
|
||||
The inner method to resume config.
|
||||
The sub class should implement this method if needed.
|
||||
|
||||
:param key: The config key
|
||||
:param value: The config value
|
||||
:returns None: Success
|
||||
:returns Exceptions: Fail, error in _set()
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
if key == "CPI_ROLLBACK_INFO":
|
||||
err = NotImplementedError("_resume method is not implemented")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return self.set("{} = {}".format(key, value))
|
||||
|
||||
def resume(self, config_info):
|
||||
"""
|
||||
Resume from the saved config info.
|
||||
|
||||
:param config_info: The config info to be resumed
|
||||
:returns None: Success
|
||||
:returns NeedRebootWarning: Success, but need reboot
|
||||
:returns SetConfigError: Fail, fail in _resume()
|
||||
:returns Exceptions: Fail, error in _resume()
|
||||
:raises: None
|
||||
"""
|
||||
cfg = self._getcfg(config_info)
|
||||
try:
|
||||
ret = self._resume(cfg[0], cfg[1])
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
return ret
|
||||
|
||||
|
||||
def file_modify(file, start, end, str):
|
||||
file.seek(0)
|
||||
content = file.read()
|
||||
if end < start:
|
||||
content = content[:start] + str + content[start:]
|
||||
else:
|
||||
content = content[:start] + str + content[end + 1:]
|
||||
file.seek(0)
|
||||
file.truncate()
|
||||
file.write(content)
|
||||
|
||||
|
||||
def pre_check(checker=None, file=None, strict=False):
|
||||
def wrapper(set):
|
||||
@wraps(set)
|
||||
def prechecked_set(self, key, value):
|
||||
if checker is not None:
|
||||
try:
|
||||
if file is not None:
|
||||
checker(self, key, value, file)
|
||||
else:
|
||||
checker(self, key, value)
|
||||
except TypeError as err:
|
||||
if strict is True:
|
||||
raise err
|
||||
except KeyError as err:
|
||||
if strict is True:
|
||||
raise err
|
||||
except ValueError as err:
|
||||
raise err
|
||||
except Exception as err:
|
||||
if strict is True:
|
||||
raise err
|
||||
return set(self, key, value)
|
||||
return prechecked_set
|
||||
return wrapper
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["kconfig"]
|
||||
|
||||
from . import *
|
||||
@ -1,87 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the kernel config.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class KernelConfig(Configurator):
|
||||
"""To change the kernel config"""
|
||||
_module = "KERNEL_CONFIG"
|
||||
_submod = "KERNEL_CONFIG"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
cfg_file = "/proc/config.gz"
|
||||
if os.path.isfile(cfg_file):
|
||||
self.__cfg_file = cfg_file
|
||||
else:
|
||||
self.__kernel_ver = subprocess.check_output(
|
||||
"uname -r".split()).decode().replace("\n", "")
|
||||
self.__cfg_file = "/boot/config-" + self.__kernel_ver
|
||||
|
||||
def _set(self, key, value):
|
||||
if self.get(key) == value:
|
||||
return 0
|
||||
else:
|
||||
raise NeedConfigWarning(
|
||||
"Please change the kernel configuration {key} to {val}.".format(
|
||||
key=key, val=value))
|
||||
|
||||
def _get(self, key):
|
||||
file_type = os.path.splitext(self.__cfg_file)[-1]
|
||||
if file_type == ".gz":
|
||||
with gzip.open(self.__cfg_file, 'rt') as f:
|
||||
cfgs = f.read()
|
||||
else:
|
||||
with open(self.__cfg_file, 'r') as f:
|
||||
cfgs = f.read()
|
||||
|
||||
pattern = re.compile("^" + key + "=(.+)", re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(cfgs)
|
||||
if len(searchObj) > 1:
|
||||
err = LookupError("find more than one " + key)
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return searchObj[0]
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
return ""
|
||||
|
||||
def _resume(self, key, value):
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = KernelConfig("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["script"]
|
||||
|
||||
from . import *
|
||||
@ -1,135 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to extend script for CPI.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import random
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Script(Configurator):
|
||||
"""The script extention of CPI"""
|
||||
_module = "SCRIPT"
|
||||
_submod = "SCRIPT"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
|
||||
def _set(self, key, value):
|
||||
name = os.path.basename(key)
|
||||
script = "{}/set.sh".format(key)
|
||||
output = subprocess.run(
|
||||
"bash {script} {val}".format(
|
||||
script=script,
|
||||
val=value).split(),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
check=True)
|
||||
if len(output.stderr) != 0:
|
||||
err = UserWarning(name + ": " + output.stderr.decode())
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return 0
|
||||
|
||||
def _get(self, key):
|
||||
name = os.path.basename(key)
|
||||
script = "{}/get.sh".format(key)
|
||||
output = subprocess.run(
|
||||
"bash {script}".format(
|
||||
script=script).split(),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=True)
|
||||
if len(output.stderr) != 0:
|
||||
err = UserWarning(name + ": " + output.stderr.decode())
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return output.stdout.decode()
|
||||
|
||||
def _check(self, config1, config2):
|
||||
return True
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
name = os.path.basename(key)
|
||||
script = "{}/backup.sh".format(key)
|
||||
if os.path.isfile(script):
|
||||
output = subprocess.run(
|
||||
"bash {script} {rb_info} {ver}".format(
|
||||
script=script,
|
||||
rb_info=rollback_info,
|
||||
ver=random.random()).split(),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=True)
|
||||
if len(output.stderr) != 0:
|
||||
err = UserWarning(name + ": " + output.stderr.decode())
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return output.stdout.decode()
|
||||
else:
|
||||
return Configurator._backup(self, key, rollback_info)
|
||||
|
||||
def _resume(self, key, value):
|
||||
name = os.path.basename(key)
|
||||
script = "{}/resume.sh".format(key)
|
||||
if os.path.isfile(script):
|
||||
output = subprocess.run(
|
||||
"bash {script} {val}".format(
|
||||
script=script,
|
||||
val=value).split(),
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
check=True)
|
||||
if len(output.stderr) != 0:
|
||||
err = UserWarning(name + ": " + output.stderr.decode())
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return 0
|
||||
else:
|
||||
return Configurator._resume(self, key, value)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' script_file=value')
|
||||
sys.exit(-1)
|
||||
ct = Script("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["sysctl"]
|
||||
|
||||
from . import *
|
||||
@ -1,64 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the /proc/sys/* config.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Sysctl(Configurator):
|
||||
"""To change the /proc/sys/* config"""
|
||||
_module = "SYSCTL"
|
||||
_submod = "SYSCTL"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
self.__cmd = "sysctl"
|
||||
|
||||
def _get(self, key):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} -n {key}".format(
|
||||
cmd=self.__cmd, key=key).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def _set(self, key, value):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
return subprocess.call('{cmd} -w {key}="{val}"'.format(
|
||||
cmd=self.__cmd, key=key, val=value), shell=True,
|
||||
stdout=no_print, stderr=no_print)
|
||||
|
||||
def _check(self, config1, config2):
|
||||
config1 = re.sub("\s{1,}", " ", config1)
|
||||
config2 = re.sub("\s{1,}", " ", config2)
|
||||
return config1 == config2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = Sysctl("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["sysfs"]
|
||||
|
||||
from . import *
|
||||
@ -1,68 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the /sys/* config.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Sysfs(Configurator):
|
||||
"""To change the /sys/* config"""
|
||||
_module = "SYSFS"
|
||||
_submod = "SYSFS"
|
||||
_option = "/sys"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
|
||||
def _get(self, key):
|
||||
f = open("{opt}/{key}".format(opt=self._option, key=key), mode='r',
|
||||
buffering=-1, encoding=None, errors=None, newline=None, closefd=True)
|
||||
ret = f.read()
|
||||
f.close()
|
||||
|
||||
pattern = ".*\[(.*)\].*"
|
||||
searchObj = re.search(pattern, ret, re.ASCII | re.MULTILINE)
|
||||
if searchObj is not None:
|
||||
return searchObj.group(1)
|
||||
else:
|
||||
return ret
|
||||
|
||||
def _set(self, key, value):
|
||||
f = open("{opt}/{key}".format(opt=self._option, key=key), mode='w',
|
||||
buffering=-1, encoding=None, errors=None, newline=None, closefd=True)
|
||||
f.write(value)
|
||||
f.close()
|
||||
return 0
|
||||
|
||||
def _check(self, config1, config2):
|
||||
return True
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = Sysfs("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["systemctl"]
|
||||
|
||||
from . import *
|
||||
@ -1,80 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the system service config.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Systemctl(Configurator):
|
||||
"""To change the system service config"""
|
||||
_module = "SYSTEMCTL"
|
||||
_submod = "SYSTEMCTL"
|
||||
_option = "is-active"
|
||||
_path = "/usr/lib/systemd/system/"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
self.__cmd = "systemctl"
|
||||
|
||||
def _get(self, service):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt} {service}; exit 0".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option,
|
||||
service=service),
|
||||
stderr=no_print,
|
||||
shell=True)
|
||||
return output.decode().replace('\n', ' ').strip()
|
||||
|
||||
def _set(self, service, oper):
|
||||
if not os.path.exists(self._path + service + ".service"):
|
||||
return 0
|
||||
status = self._get(service)
|
||||
if status == "active" and oper == "start" or status == "inactive" and oper == "stop":
|
||||
return 0
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
return subprocess.call("{cmd} {oper} {service}".format(
|
||||
cmd=self.__cmd, service=service, oper=oper).split(),
|
||||
stdout=no_print, stderr=no_print)
|
||||
|
||||
def _check(self, config1, config2):
|
||||
return True
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
val = self._get(key)
|
||||
if val == "active" or val == "activating":
|
||||
val = "start"
|
||||
else:
|
||||
val = "stop"
|
||||
return "{} = {}".format(key, val)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' service=oper')
|
||||
sys.exit(-1)
|
||||
ct = Systemctl("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["ulimit"]
|
||||
|
||||
from . import *
|
||||
@ -1,121 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the Configurator, used to change the resources limit of user.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import random
|
||||
import shutil
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from configurator.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Ulimit(Configurator):
|
||||
"""To change the resources limit of user"""
|
||||
_module = "ULIMIT"
|
||||
_submod = "ULIMIT"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Configurator.__init__(self, user)
|
||||
self.__cfg_file = "/etc/security/limits.conf"
|
||||
|
||||
def _set(self, key, value):
|
||||
f = open(
|
||||
self.__cfg_file,
|
||||
mode='r+',
|
||||
buffering=-1,
|
||||
encoding=None,
|
||||
errors=None,
|
||||
newline=None,
|
||||
closefd=True)
|
||||
info = f.read()
|
||||
keyword = key.split(".")
|
||||
pattern = re.compile(
|
||||
"^\s*?(?!#)" +
|
||||
keyword[0] +
|
||||
"\s+?" +
|
||||
keyword[1] +
|
||||
"\s+?" +
|
||||
keyword[2] +
|
||||
"\s+(\w+)\s*?",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.search(info)
|
||||
if searchObj is not None:
|
||||
offset = searchObj.span(1)
|
||||
file_modify(f, offset[0], offset[1] - 1, value)
|
||||
else:
|
||||
file_modify(f, len(info), -1, "\n{domain}\t{type}\t{item}\t{value}".format(
|
||||
domain=keyword[0], type=keyword[1], item=keyword[2], value=value))
|
||||
f.close()
|
||||
return 0
|
||||
|
||||
def _get(self, key):
|
||||
with open(self.__cfg_file, 'r') as f:
|
||||
info = f.read()
|
||||
keyword = key.split(".")
|
||||
pattern = re.compile(
|
||||
"^\s*?(?!#)" +
|
||||
keyword[0] +
|
||||
"\s+?" +
|
||||
keyword[1] +
|
||||
"\s+?" +
|
||||
keyword[2] +
|
||||
"\s+(\w+)\s*?",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = GetConfigError("Fail to find {} config".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
return searchObj[-1]
|
||||
|
||||
def _backup(self, key, rollback_info):
|
||||
name = os.path.basename(self.__cfg_file)
|
||||
bak_file = "{path}/{file}{ver}".format(path=rollback_info, file=name,
|
||||
ver=random.random())
|
||||
shutil.copy(self.__cfg_file, bak_file)
|
||||
return "CPI_ROLLBACK_INFO = {}".format(bak_file)
|
||||
|
||||
def _resume(self, key, value):
|
||||
if key != "CPI_ROLLBACK_INFO":
|
||||
err = ValueError("unsupported resume type: {}".format(key))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
shutil.copy(value, self.__cfg_file)
|
||||
return None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
print('usage: ' + sys.argv[0] + ' key=value')
|
||||
sys.exit(-1)
|
||||
ct = Ulimit("UT")
|
||||
print(ct.set(sys.argv[1]))
|
||||
print(ct.get(ct._getcfg(sys.argv[1])[0]))
|
||||
@ -1,27 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = [
|
||||
"memory",
|
||||
"network",
|
||||
"performance",
|
||||
"processor",
|
||||
"storage",
|
||||
"common",
|
||||
"system"]
|
||||
|
||||
from . import *
|
||||
@ -1,266 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The base class of the monitor, used to report the given config, get the collected info,
|
||||
decode the collected info, format the collected info and output collected info to file.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
from public import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Monitor():
|
||||
"""Base class for monitors"""
|
||||
|
||||
# sub class should init these
|
||||
_module = "UNKNOWN"
|
||||
_purpose = "UNKNOWN"
|
||||
|
||||
# for inner options usage
|
||||
_option = ""
|
||||
|
||||
def __init__(self, user=None):
|
||||
"""
|
||||
Initialize.
|
||||
|
||||
:param user(optional): "UT" for unit test, others are ignored
|
||||
:returns: None
|
||||
:raises: None
|
||||
"""
|
||||
self._user = user
|
||||
|
||||
def module(self):
|
||||
"""
|
||||
Get the the module of this monitor.
|
||||
|
||||
:param: None
|
||||
:returns: The module of this monitor
|
||||
:raises: None
|
||||
"""
|
||||
return self._module
|
||||
|
||||
def purpose(self):
|
||||
"""
|
||||
Get the the purpose of this monitor.
|
||||
|
||||
:param: None
|
||||
:returns: The purpose of this monitor
|
||||
:raises: None
|
||||
"""
|
||||
return self._purpose
|
||||
|
||||
def _getopt(self):
|
||||
"""
|
||||
Get the the inner option of this monitor.
|
||||
Multi-options should be splited by ";".
|
||||
|
||||
:param: None
|
||||
:returns: The iterator for get all inner options
|
||||
:raises: None
|
||||
"""
|
||||
return iter(self._option.split(";"))
|
||||
|
||||
def _getpara(self, paras):
|
||||
"""
|
||||
Get all the configs from one string.
|
||||
|
||||
:param paras: The configs string splited by ";"
|
||||
:returns None: No parameter
|
||||
:returns parameter: next parameter
|
||||
:raises: None
|
||||
"""
|
||||
if paras is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
nextp = next(paras)
|
||||
except StopIteration:
|
||||
return None
|
||||
|
||||
if nextp == "":
|
||||
nextp = None
|
||||
return nextp
|
||||
|
||||
def report(self, fmt, path, para=None):
|
||||
"""
|
||||
Report the given config.
|
||||
|
||||
:param fmt: The option for format(fmt)
|
||||
:param path: The path to output, None for pass through
|
||||
:param para: Multi-options for get(para) and decode(para), should be splited by ";"
|
||||
:returns None: Success
|
||||
:returns info: Success, output info
|
||||
:returns Exceptions: Fail, with info
|
||||
:raises: None
|
||||
"""
|
||||
try:
|
||||
if para is None:
|
||||
paras = None
|
||||
else:
|
||||
paras = iter(para.split(";"))
|
||||
info = self._get(self._getpara(paras))
|
||||
decoded_info = self.decode(info, self._getpara(paras))
|
||||
fmted_info = self.format(decoded_info, fmt)
|
||||
return self.output(fmted_info, path)
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
|
||||
def _get(self, para):
|
||||
"""
|
||||
The inner method to get collected info.
|
||||
The sub class should implement this method.
|
||||
|
||||
:param para: The option for get,
|
||||
[%s]:
|
||||
"--interval=" to specify period of time
|
||||
"--cpu=" to select which cpu
|
||||
"--event=" to select which event
|
||||
:returns value: Success, collected info string
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
err = NotImplementedError("_get method is not implemented")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def get(self, para=None):
|
||||
"""
|
||||
Get the collected info.
|
||||
|
||||
:param para(optional): The option for get
|
||||
:returns info: Success, collected info string
|
||||
:returns Exceptions: Fail, error in _get()
|
||||
:raises: None
|
||||
"""
|
||||
try:
|
||||
ret = self._get(para)
|
||||
except Exception as err:
|
||||
if self._user == "UT":
|
||||
raise err
|
||||
else:
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
return err
|
||||
return ret
|
||||
|
||||
def decode(self, info, para):
|
||||
"""
|
||||
The inner method to decode collected info.
|
||||
|
||||
:param info: The collected info string
|
||||
:param para: The option for decode,
|
||||
[%s]:
|
||||
"--fields=" to select which data
|
||||
"--cpu=" to select which cpu
|
||||
"--nic=" to select which net interface
|
||||
"--device=" to select which device
|
||||
:returns info: Success, decoded info
|
||||
:raises NotImplementedError: Error, not supported
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
if para is None:
|
||||
return info
|
||||
else:
|
||||
err = NotImplementedError("Not supported decode: {}".format(para))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def format(self, info, fmt):
|
||||
"""
|
||||
The inner method to format collected info.
|
||||
|
||||
:param info: The decoded info
|
||||
:param fmt: The option for format,
|
||||
[raw, data, %s]:
|
||||
"raw" for original string
|
||||
"xml" for xml string
|
||||
"json" for json string
|
||||
"data" for list of decoded data string
|
||||
"table" for pretty table string
|
||||
:returns info: Success, formatted info
|
||||
:raises NotImplementedError: Error, not supported
|
||||
:raises Exceptions: Fail, with info
|
||||
"""
|
||||
if (fmt == "raw"):
|
||||
return info
|
||||
elif (fmt == "data"):
|
||||
return info.split()
|
||||
else:
|
||||
err = NotImplementedError("Not supported format: {}".format(fmt))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
def output(self, info, path):
|
||||
"""
|
||||
The method to output collected info to file.
|
||||
|
||||
:param info: The formatted info
|
||||
:param path: The path to output, None for pass through
|
||||
:returns None: Success
|
||||
:returns info: Success, output info
|
||||
:raises: None
|
||||
"""
|
||||
if path is None:
|
||||
return info
|
||||
|
||||
f = open(path, mode='w', buffering=-1, encoding=None,
|
||||
errors=None, newline=None)
|
||||
f.write(info)
|
||||
f.close()
|
||||
return None
|
||||
|
||||
|
||||
def walk_class_type(father, class_type, desc, datas):
|
||||
if "class" in father and \
|
||||
father["class"] == class_type:
|
||||
if "description" in father and \
|
||||
(desc is None or father["description"] == desc):
|
||||
datas.append(father)
|
||||
return
|
||||
if "children" in father:
|
||||
for i in father["children"]:
|
||||
walk_class_type(i, class_type, desc, datas)
|
||||
|
||||
|
||||
def get_class_type(all, class_type, desc=None):
|
||||
datas = []
|
||||
walk_class_type(all, class_type, desc, datas)
|
||||
dict_datas = {}
|
||||
dict_datas[class_type + "s"] = datas
|
||||
return dict_datas
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["numainfo", "topo", "bandwidth", "vmstat", "utilstat"]
|
||||
|
||||
from . import *
|
||||
@ -1,236 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect memory bandwidth stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
import json
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
from monitor.memory import topo
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemBandwidth(Monitor):
|
||||
"""To collect memory bandwidth stat info"""
|
||||
_module = "MEM"
|
||||
_purpose = "BANDWIDTH"
|
||||
_option = "-a -e {events} --interval-print {int} --interval-count 1"
|
||||
|
||||
__evs1620 = {
|
||||
"c0d0c0_r": "hisi_sccl1_ddrc0/flux_rd/",
|
||||
"c0d0c1_r": "hisi_sccl1_ddrc1/flux_rd/",
|
||||
"c0d0c2_r": "hisi_sccl1_ddrc2/flux_rd/",
|
||||
"c0d0c3_r": "hisi_sccl1_ddrc3/flux_rd/",
|
||||
"c0d0c0_w": "hisi_sccl1_ddrc0/flux_wr/",
|
||||
"c0d0c1_w": "hisi_sccl1_ddrc1/flux_wr/",
|
||||
"c0d0c2_w": "hisi_sccl1_ddrc2/flux_wr/",
|
||||
"c0d0c3_w": "hisi_sccl1_ddrc3/flux_wr/",
|
||||
"c0d1c0_r": "hisi_sccl3_ddrc0/flux_rd/",
|
||||
"c0d1c1_r": "hisi_sccl3_ddrc1/flux_rd/",
|
||||
"c0d1c2_r": "hisi_sccl3_ddrc2/flux_rd/",
|
||||
"c0d1c3_r": "hisi_sccl3_ddrc3/flux_rd/",
|
||||
"c0d1c0_w": "hisi_sccl3_ddrc0/flux_wr/",
|
||||
"c0d1c1_w": "hisi_sccl3_ddrc1/flux_wr/",
|
||||
"c0d1c2_w": "hisi_sccl3_ddrc2/flux_wr/",
|
||||
"c0d1c3_w": "hisi_sccl3_ddrc3/flux_wr/",
|
||||
"c1d0c0_r": "hisi_sccl5_ddrc0/flux_rd/",
|
||||
"c1d0c1_r": "hisi_sccl5_ddrc1/flux_rd/",
|
||||
"c1d0c2_r": "hisi_sccl5_ddrc2/flux_rd/",
|
||||
"c1d0c3_r": "hisi_sccl5_ddrc3/flux_rd/",
|
||||
"c1d0c0_w": "hisi_sccl5_ddrc0/flux_wr/",
|
||||
"c1d0c1_w": "hisi_sccl5_ddrc1/flux_wr/",
|
||||
"c1d0c2_w": "hisi_sccl5_ddrc2/flux_wr/",
|
||||
"c1d0c3_w": "hisi_sccl5_ddrc3/flux_wr/",
|
||||
"c1d1c0_r": "hisi_sccl7_ddrc0/flux_rd/",
|
||||
"c1d1c1_r": "hisi_sccl7_ddrc1/flux_rd/",
|
||||
"c1d1c2_r": "hisi_sccl7_ddrc2/flux_rd/",
|
||||
"c1d1c3_r": "hisi_sccl7_ddrc3/flux_rd/",
|
||||
"c1d1c0_w": "hisi_sccl7_ddrc0/flux_wr/",
|
||||
"c1d1c1_w": "hisi_sccl7_ddrc1/flux_wr/",
|
||||
"c1d1c2_w": "hisi_sccl7_ddrc2/flux_wr/",
|
||||
"c1d1c3_w": "hisi_sccl7_ddrc3/flux_wr/"}
|
||||
|
||||
__cnt1620 = {
|
||||
"Total": 0,
|
||||
"CPU0": 0,
|
||||
"CPU1": 0,
|
||||
"CPU0_Die0": 0,
|
||||
"CPU0_Die1": 0,
|
||||
"CPU1_Die0": 0,
|
||||
"CPU1_Die1": 0,
|
||||
"CPU0_Die0_R": 0,
|
||||
"CPU0_Die1_R": 0,
|
||||
"CPU1_Die0_R": 0,
|
||||
"CPU1_Die1_R": 0,
|
||||
"CPU0_Die0_W": 0,
|
||||
"CPU0_Die1_W": 0,
|
||||
"CPU1_Die0_W": 0,
|
||||
"CPU1_Die1_W": 0,
|
||||
"Total_Max": 0,
|
||||
"CPU0_Max": 0,
|
||||
"CPU1_Max": 0,
|
||||
"Total_Util": 0}
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "perf stat"
|
||||
self.__interval = 1000
|
||||
|
||||
self.__evs = self.__evs1620
|
||||
self.__cnt = self.__cnt1620
|
||||
self.__cnt["CPU0_Max"] = self.__get_theory_bandwidth(0) / 1024 / 1024
|
||||
self.__cnt["CPU1_Max"] = self.__get_theory_bandwidth(1) / 1024 / 1024
|
||||
|
||||
self.__events = ""
|
||||
for e in self.__evs:
|
||||
self.__events = self.__events + self.__evs[e] + ","
|
||||
self.__events = self.__events.strip(",")
|
||||
|
||||
help_info = "--fields="
|
||||
for c in self.__cnt:
|
||||
help_info = help_info + c + "/"
|
||||
help_info = help_info.strip("/")
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (help_info)
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val) * 1000
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval,
|
||||
events=self.__events)).split(),
|
||||
stderr=subprocess.STDOUT)
|
||||
return output.decode()
|
||||
|
||||
def __get_theory_bandwidth(self, socket):
|
||||
memtopo = topo.MemTopo()
|
||||
info_json = memtopo.report("json", None)
|
||||
info = json.loads(info_json)
|
||||
|
||||
width = 0
|
||||
dimms = [[0 for i in range(8)] for i in range(8)]
|
||||
for dimm in info["memorys"][0]["children"]:
|
||||
if dimm.get("size") is None:
|
||||
continue
|
||||
locator = memtopo._table_get_locator(dimm["slot"])
|
||||
if dimms[locator[0]][locator[1]] == 0:
|
||||
dimms[locator[0]][locator[1]] = dimm["width"] * \
|
||||
memtopo._table_get_freq(dimm["description"]) / 8
|
||||
ret = 0
|
||||
for channel in dimms[socket]:
|
||||
ret += channel
|
||||
return ret
|
||||
|
||||
def __read_counters(self, c_evs):
|
||||
self.__cnt["CPU0_Die0_R"] = ((int(c_evs["c0d0c0_r"]) + int(c_evs["c0d0c1_r"]) + int(
|
||||
c_evs["c0d0c2_r"]) + int(c_evs["c0d0c3_r"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU0_Die1_R"] = ((int(c_evs["c0d1c0_r"]) + int(c_evs["c0d1c1_r"]) + int(
|
||||
c_evs["c0d1c2_r"]) + int(c_evs["c0d1c3_r"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU1_Die0_R"] = ((int(c_evs["c1d0c0_r"]) + int(c_evs["c1d0c1_r"]) + int(
|
||||
c_evs["c1d0c2_r"]) + int(c_evs["c1d0c3_r"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU1_Die1_R"] = ((int(c_evs["c1d1c0_r"]) + int(c_evs["c1d1c1_r"]) + int(
|
||||
c_evs["c1d1c2_r"]) + int(c_evs["c1d1c3_r"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU0_Die0_W"] = ((int(c_evs["c0d0c0_w"]) + int(c_evs["c0d0c1_w"]) + int(
|
||||
c_evs["c0d0c2_w"]) + int(c_evs["c0d0c3_w"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU0_Die1_W"] = ((int(c_evs["c0d1c0_w"]) + int(c_evs["c0d1c1_w"]) + int(
|
||||
c_evs["c0d1c2_w"]) + int(c_evs["c0d1c3_w"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU1_Die0_W"] = ((int(c_evs["c1d0c0_w"]) + int(c_evs["c1d0c1_w"]) + int(
|
||||
c_evs["c1d0c2_w"]) + int(c_evs["c1d0c3_w"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU1_Die1_W"] = ((int(c_evs["c1d1c0_w"]) + int(c_evs["c1d1c1_w"]) + int(
|
||||
c_evs["c1d1c2_w"]) + int(c_evs["c1d1c3_w"])) * 32 / 1024 / 1024) * 1000 / self.__interval
|
||||
self.__cnt["CPU0_Die0"] = self.__cnt["CPU0_Die0_R"] + \
|
||||
self.__cnt["CPU0_Die0_W"]
|
||||
self.__cnt["CPU0_Die1"] = self.__cnt["CPU0_Die1_R"] + \
|
||||
self.__cnt["CPU0_Die1_W"]
|
||||
self.__cnt["CPU1_Die0"] = self.__cnt["CPU1_Die0_R"] + \
|
||||
self.__cnt["CPU1_Die0_W"]
|
||||
self.__cnt["CPU1_Die1"] = self.__cnt["CPU1_Die1_R"] + \
|
||||
self.__cnt["CPU1_Die1_W"]
|
||||
self.__cnt["CPU0"] = self.__cnt["CPU0_Die0_R"] + self.__cnt["CPU0_Die0_W"] + \
|
||||
self.__cnt["CPU0_Die1_R"] + self.__cnt["CPU0_Die1_W"]
|
||||
self.__cnt["CPU1"] = self.__cnt["CPU1_Die0_R"] + self.__cnt["CPU1_Die0_W"] + \
|
||||
self.__cnt["CPU1_Die1_R"] + self.__cnt["CPU1_Die1_W"]
|
||||
self.__cnt["Total"] = self.__cnt["CPU0"] + self.__cnt["CPU1"]
|
||||
self.__cnt["Total_Max"] = self.__cnt["CPU0_Max"] + self.__cnt["CPU1_Max"]
|
||||
self.__cnt["Total_Util"] = self.__cnt["Total"] / self.__cnt["Total_Max"] * 100
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
c_evs = self.__evs.copy()
|
||||
for e in self.__evs:
|
||||
pattern = "^\ {2,}(\d.*?)\ {2,}(\d.*?)\ {2,}(" + \
|
||||
self.__evs[e] + ").*?"
|
||||
searchObj = re.search(pattern, info, re.ASCII | re.MULTILINE)
|
||||
if searchObj is not None:
|
||||
c_evs[e] = searchObj.group(2).replace(",", "")
|
||||
else:
|
||||
err = LookupError("Fail to find {}".format(self.__evs[e]))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
self.__read_counters(c_evs)
|
||||
fields = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
fields.append(val)
|
||||
continue
|
||||
|
||||
for f in fields:
|
||||
ret = ret + " {:.2f}".format(self.__cnt[f])
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = MemBandwidth("UT")
|
||||
# help(ct)
|
||||
ct.report(sys.argv[1], sys.argv[2], "--interval=5;--fields=Total --fields=CPU0 --fields=CPU1 --fields=Total_Max --fields=CPU0_Max --fields=CPU1_Max --fields=Total_Util")
|
||||
@ -1,50 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the memory numa info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemNuma(Monitor):
|
||||
"""To collect the memory numa info"""
|
||||
_module = "MEM"
|
||||
_purpose = "NUMA"
|
||||
_option = "-H"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "numactl"
|
||||
|
||||
def _get(self, para=None):
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split())
|
||||
return output.decode()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = MemNuma("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,157 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the memory topo.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import re
|
||||
import json
|
||||
import prettytable
|
||||
import dict2xml
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemTopo(Monitor):
|
||||
"""To collect the memory topo"""
|
||||
_module = "MEM"
|
||||
_purpose = "TOPO"
|
||||
_option = "-c memory"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "lshw"
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("json, table")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def __table_init(self, column):
|
||||
self.__columns = column
|
||||
data = ['Socket', 'Channel']
|
||||
for i in range(self.__columns):
|
||||
data.append("Slot {}".format(i))
|
||||
return prettytable.PrettyTable(data)
|
||||
|
||||
def _table_get_locator(self, bank):
|
||||
pattern = re.compile(
|
||||
"DIMM.*?(\d)(\d)(\d)\s.*",
|
||||
re.ASCII | re.MULTILINE)
|
||||
scd = pattern.findall(bank)
|
||||
if len(scd) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
ret = []
|
||||
for i in scd[0]:
|
||||
ret.append(int(i))
|
||||
return ret
|
||||
|
||||
def _table_get_freq(self, desc):
|
||||
pattern = re.compile(
|
||||
".*?(\d+)\s([KMGT]?)Hz.*?",
|
||||
re.ASCII | re.MULTILINE)
|
||||
freq = pattern.findall(desc)
|
||||
if len(freq) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
if freq[-1][1] == "K":
|
||||
ret = int(freq[-1][0]) * 1000
|
||||
elif freq[-1][1] == "M":
|
||||
ret = int(freq[-1][0]) * 1000000
|
||||
elif freq[-1][1] == "G":
|
||||
ret = int(freq[-1][0]) * 1000000000
|
||||
elif freq[-1][1] == "T":
|
||||
ret = int(freq[-1][0]) * 1000000000000
|
||||
else:
|
||||
ret = int(freq[-1][0])
|
||||
return ret
|
||||
|
||||
def __table_add_banks(self, table, datas, i, j):
|
||||
data = [i, j]
|
||||
for k in range(self.__columns):
|
||||
data.append(datas[i][j][k])
|
||||
table.add_row(data)
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json") or (fmt == "table") or (fmt == "xml"):
|
||||
o_json = subprocess.check_output("{cmd} -json".format(
|
||||
cmd=self.__cmd).split(), stderr=subprocess.DEVNULL)
|
||||
info = o_json.decode()
|
||||
all = json.loads(info)
|
||||
|
||||
dict_datas = get_class_type(all, "memory", "System Memory")
|
||||
if (fmt == "json"):
|
||||
return json.dumps(dict_datas, indent=2)
|
||||
elif (fmt == "xml"):
|
||||
return dict2xml.dict2xml(dict_datas, "topology")
|
||||
|
||||
dimms = [[["NO DIMM" for i in range(8)]
|
||||
for i in range(8)] for i in range(8)]
|
||||
max_socket = 0
|
||||
max_slot = 0
|
||||
max_channel = 0
|
||||
datas = dict_datas["memorys"]
|
||||
for data in datas:
|
||||
for bank in data["children"]:
|
||||
locator = self._table_get_locator(bank["slot"])
|
||||
if locator[0] > max_socket:
|
||||
max_socket = locator[0]
|
||||
if locator[1] > max_channel:
|
||||
max_channel = locator[1]
|
||||
if locator[2] > max_slot:
|
||||
max_slot = locator[2]
|
||||
if "size" in bank.keys():
|
||||
dimms[locator[0]][locator[1]][locator[2]] = "{} {} {} {}".format(
|
||||
bank["vendor"], bank["description"], bank["size"], bank["units"])
|
||||
|
||||
tables = ""
|
||||
for socket in range(max_socket + 1):
|
||||
table = self.__table_init(max_slot + 1)
|
||||
for channel in range(max_channel + 1):
|
||||
self.__table_add_banks(table, dimms, socket, channel)
|
||||
tables += table.get_string(hrules=prettytable.ALL) + "\n\n"
|
||||
return tables
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = MemTopo("UT")
|
||||
print(ct.report(sys.argv[1], sys.argv[2]))
|
||||
@ -1,122 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the mem util stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemUtilStat(Monitor):
|
||||
"""To collect the mem util stat info"""
|
||||
_module = "MEM"
|
||||
_purpose = "UTIL"
|
||||
_option = "-r {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "sar"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--fields=time/kbmemfree/kbavail/kbmemused/memused/kbbuffers/kbcached/kbcommit/commit/kbactive/kbinact/kbdirty")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"kbmemfree": 1,
|
||||
"kbavail": 2,
|
||||
"kbmemused": 3,
|
||||
"memused": 4,
|
||||
"kbbuffers": 5,
|
||||
"kbcached": 6,
|
||||
"kbcommit": 7,
|
||||
"commit": 8,
|
||||
"kbactive": 9,
|
||||
"kbinact": 10,
|
||||
"kbdirty": 11}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*)\ {1,}(\d*)\ {2,}(\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = MemUtilStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--fields=kbmemfree --fields=memused --fields=kbdirty")
|
||||
@ -1,140 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the vm stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MemVmstat(Monitor):
|
||||
"""To collect the vm stat info"""
|
||||
_module = "MEM"
|
||||
_purpose = "VMSTAT"
|
||||
_option = "{int} 2"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "vmstat"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--fields=procs.r/procs.b/memory.swpd/memory.free/memory.buff/memory.cache/swap.si/swap.so/io.bi/io.bo/system.in/system.cs/cpu.us/cpu.sy/cpu.id/cpu.wa/cpu.st")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"procs.r": 0,
|
||||
"procs.b": 1,
|
||||
"memory.swpd": 2,
|
||||
"memory.free": 3,
|
||||
"memory.buff": 4,
|
||||
"memory.cache": 5,
|
||||
"swap.si": 6,
|
||||
"swap.so": 7,
|
||||
"io.bi": 8,
|
||||
"io.bo": 9,
|
||||
"system.in": 10,
|
||||
"system.cs": 11,
|
||||
"cpu.us": 12,
|
||||
"cpu.sy": 13,
|
||||
"cpu.id": 14,
|
||||
"cpu.wa": 15,
|
||||
"cpu.st": 16,
|
||||
"util.swap": "util.swap",
|
||||
"util.cpu": "util.cpu"}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^\ ?(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
if type(i).__name__ == 'int':
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
elif i == "util.swap":
|
||||
util = int(searchObj[-1][keyword["swap.si"]]) + \
|
||||
int(searchObj[-1][keyword["swap.so"]])
|
||||
ret = ret + " " + str(util)
|
||||
elif i == "util.cpu":
|
||||
util = int(searchObj[-1][keyword["cpu.us"]]) + \
|
||||
int(searchObj[-1][keyword["cpu.sy"]]) + \
|
||||
int(searchObj[-1][keyword["cpu.st"]])
|
||||
ret = ret + " " + str(util)
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = MemVmstat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--fields=memory.free --fields=cpu.us --fields=cpu.id --fields=util.cpu")
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["info", "netstat", "netestat", "topo"]
|
||||
|
||||
from . import *
|
||||
@ -1,56 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the nic config info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NetInfo(Monitor):
|
||||
"""To collect the nic config info"""
|
||||
_module = "NET"
|
||||
_purpose = "INFO"
|
||||
_option = ";-l;-k"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "ethtool"
|
||||
|
||||
def _get(self, para):
|
||||
opts = self._getopt()
|
||||
outputs = ""
|
||||
while True:
|
||||
try:
|
||||
output = subprocess.check_output("{cmd} {opt} {para}".format(
|
||||
cmd=self.__cmd, opt=next(opts), para=para).split())
|
||||
outputs += output.decode()
|
||||
except StopIteration:
|
||||
return outputs
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 4:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path dev')
|
||||
sys.exit(-1)
|
||||
ct = NetInfo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2], sys.argv[3])
|
||||
@ -1,140 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the nic estat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NetEStat(Monitor):
|
||||
"""To collect the nic estat info"""
|
||||
_module = "NET"
|
||||
_purpose = "ESTAT"
|
||||
_option = "-n EDEV {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "sar"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--nic=x, --fields=time/nic/rxerrs/txerrs/colls/rxdrops/txdrops/txcarrs/rxframs/rxfifos/txfifos/errs/util")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"nic": 1,
|
||||
"rxerrs": 2,
|
||||
"txerrs": 3,
|
||||
"colls": 4,
|
||||
"rxdrops": 5,
|
||||
"txdrops": 6,
|
||||
"txcarrs": 7,
|
||||
"rxframs": 8,
|
||||
"rxfifos": 9,
|
||||
"txfifos": 10,
|
||||
"errs": "errs",
|
||||
"util": "util"}
|
||||
|
||||
keys = []
|
||||
nic = "e.*?"
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['nic=', 'fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--nic'):
|
||||
nic = val
|
||||
continue
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {1,}(" +
|
||||
nic +
|
||||
")\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {2,}(\d*\.?\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data for {}".format(nic))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
if type(i).__name__ == 'int':
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
elif i == "errs":
|
||||
errs = float(searchObj[-1][keyword["rxerrs"]]) + \
|
||||
float(searchObj[-1][keyword["txerrs"]])
|
||||
ret = ret + " " + str(errs)
|
||||
elif i == "util":
|
||||
util = float(searchObj[-1][keyword["rxdrops"]]) + \
|
||||
float(searchObj[-1][keyword["txdrops"]]) + \
|
||||
float(searchObj[-1][keyword["rxfifos"]]) + \
|
||||
float(searchObj[-1][keyword["txfifos"]])
|
||||
ret = ret + " " + str(util)
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = NetEStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--nic=lo --fields=rxdrops --fields=txfifos --fields=util")
|
||||
@ -1,126 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the nic stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NetStat(Monitor):
|
||||
"""To collect the nic stat info"""
|
||||
_module = "NET"
|
||||
_purpose = "STAT"
|
||||
_option = "-n DEV {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "sar"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--nic=x, --fields=time/nic/rxpcks/txpcks/rxkBs/txkBs/rxcmps/txcmps/rxmcsts/ifutil")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"nic": 1,
|
||||
"rxpcks": 2,
|
||||
"txpcks": 3,
|
||||
"rxkBs": 4,
|
||||
"txkBs": 5,
|
||||
"rxcmps": 6,
|
||||
"txcmps": 7,
|
||||
"rxmcsts": 8,
|
||||
"ifutil": 9}
|
||||
|
||||
keys = []
|
||||
nic = "e.*?"
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['nic=', 'fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--nic'):
|
||||
nic = val
|
||||
continue
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {1,}(" +
|
||||
nic +
|
||||
")\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {2,}(\d*\.?\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data for {}".format(nic))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = NetStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--nic=lo --fields=txkBs --fields=rxkBs")
|
||||
@ -1,71 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the nic topo.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import re
|
||||
import json
|
||||
import dict2xml
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NetTopo(Monitor):
|
||||
"""To collect the nic topo"""
|
||||
_module = "NET"
|
||||
_purpose = "TOPO"
|
||||
_option = "-c network"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "lshw"
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("xml, json")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json") or (fmt == "xml"):
|
||||
o_json = subprocess.check_output("{cmd} -json".format(
|
||||
cmd=self.__cmd).split(), stderr=subprocess.DEVNULL)
|
||||
info = o_json.decode()
|
||||
all = json.loads(info)
|
||||
|
||||
dict_datas = get_class_type(all, "network")
|
||||
if (fmt == "json"):
|
||||
return json.dumps(dict_datas, indent=2)
|
||||
elif (fmt == "xml"):
|
||||
return dict2xml.dict2xml(dict_datas, "topology")
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = NetTopo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["stat", "top"]
|
||||
|
||||
from . import *
|
||||
@ -1,174 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the perf stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PerfStat(Monitor):
|
||||
"""To collect the perf stat info"""
|
||||
_module = "PERF"
|
||||
_purpose = "STAT"
|
||||
_option = "-a -e cycles,instructions,branches,branch-misses,cache-misses,cache-references,dTLB-load-misses,dTLB-loads,iTLB-load-misses,iTLB-loads,stalled-cycles-backend,r7004,r7005 --interval-print {int} --interval-count 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "perf stat"
|
||||
self.__interval = 1000
|
||||
|
||||
self.__stat = {
|
||||
"cycles": 0,
|
||||
"instructions": 0,
|
||||
"branches": 0,
|
||||
"branch-misses": 0,
|
||||
"cache-misses": 0,
|
||||
"cache-references": 0,
|
||||
"dTLB-load-misses": 0,
|
||||
"dTLB-loads": 0,
|
||||
"iTLB-load-misses": 0,
|
||||
"iTLB-loads": 0,
|
||||
"stalled-cycles-backend": 0,
|
||||
"memstall-load": 0,
|
||||
"memstall-store": 0,
|
||||
"IPC": 0,
|
||||
"BRANCH-MISS-RATIO": 0,
|
||||
"CACHE-MISS-RATIO": 0,
|
||||
"DTLB-LOAD-MISS-RATIO": 0,
|
||||
"ITLB-LOAD-MISS-RATIO": 0,
|
||||
"MPKI": 0,
|
||||
"SBPI": 0,
|
||||
"SBPC": 0,
|
||||
"MEMORY-BOUND": 0,
|
||||
"STORE-BOUND": 0}
|
||||
|
||||
help_info = "--fields="
|
||||
for s in self.__stat:
|
||||
help_info = help_info + s + "/"
|
||||
help_info = help_info.strip("/")
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (help_info)
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val) * 1000
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split(),
|
||||
stderr=subprocess.STDOUT)
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"counts": 1,
|
||||
"unit": 2,
|
||||
"events": 3}
|
||||
|
||||
eventmap = {"memstall-load": "r7004",
|
||||
"memstall-store": "r7005"}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(val)
|
||||
continue
|
||||
|
||||
for s in self.__stat:
|
||||
event = eventmap.get(s)
|
||||
if event is None:
|
||||
event = s
|
||||
pattern = "^\ {2,}(\d.*?)\ {2,}(\d.*?)\ {2,}(\w*)\ {2,}(" + \
|
||||
event + ")\ {1,}.*"
|
||||
searchObj = re.search(pattern, info, re.ASCII | re.MULTILINE)
|
||||
if searchObj is not None:
|
||||
self.__stat[s] = int(
|
||||
searchObj.group(
|
||||
keyword["counts"] +
|
||||
1).replace(
|
||||
",",
|
||||
""))
|
||||
else:
|
||||
self.__stat[s] = -1
|
||||
|
||||
self.__stat["IPC"] = self.__stat["instructions"] / \
|
||||
self.__stat["cycles"]
|
||||
self.__stat["BRANCH-MISS-RATIO"] = self.__stat["branch-misses"] / \
|
||||
self.__stat["branches"] * 100
|
||||
self.__stat["CACHE-MISS-RATIO"] = self.__stat["cache-misses"] / \
|
||||
self.__stat["cache-references"] * 100
|
||||
self.__stat["DTLB-LOAD-MISS-RATIO"] = self.__stat["dTLB-load-misses"] / \
|
||||
self.__stat["dTLB-loads"] * 100
|
||||
self.__stat["ITLB-LOAD-MISS-RATIO"] = self.__stat["iTLB-load-misses"] / \
|
||||
self.__stat["iTLB-loads"] * 100
|
||||
self.__stat["MPKI"] = self.__stat["cache-misses"] / \
|
||||
self.__stat["instructions"] * 1000
|
||||
self.__stat["SBPI"] = self.__stat["instructions"] / \
|
||||
self.__stat["instructions"]
|
||||
self.__stat["SBPC"] = self.__stat["instructions"] / \
|
||||
self.__stat["cycles"]
|
||||
self.__stat["MEMORY-BOUND"] = (self.__stat["memstall-load"] +
|
||||
self.__stat["memstall-store"]) / \
|
||||
self.__stat["cycles"] * 100
|
||||
self.__stat["STORE-BOUND"] = self.__stat["memstall-store"] / \
|
||||
self.__stat["cycles"] * 100
|
||||
|
||||
for event in keys:
|
||||
ret = ret + " " + str(self.__stat[event])
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = PerfStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=5;--fields=cycles --fields=instructions --fields=cache-misses --fields=MPKI --fields=MEMORY-BOUND")
|
||||
@ -1,191 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the perf top snapshot.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
import random
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PerfTop(Monitor):
|
||||
"""To collect the perf top snapshot"""
|
||||
_module = "PERF"
|
||||
_purpose = "TOP"
|
||||
_option = "-a -e {event} sleep {int}"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__interval = 1
|
||||
self.__event = "cycles"
|
||||
self.__keyword = {"overhead": 0,
|
||||
"command": 1,
|
||||
"object": 2,
|
||||
"symbol": 3}
|
||||
|
||||
help_info = "--interval, --event"
|
||||
self._get.__func__.__doc__ = Monitor._get.__doc__ % (help_info)
|
||||
|
||||
help_info = "--fields="
|
||||
for f in self.__keyword:
|
||||
help_info = help_info + f + "/"
|
||||
help_info = help_info.strip("/")
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (help_info)
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(
|
||||
para.split(), None, [
|
||||
'interval=', 'event='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
elif opt in ('--event'):
|
||||
self.__event = val.split(",")[-1]
|
||||
continue
|
||||
|
||||
data_file = "/tmp/perf{}".format(random.random())
|
||||
subprocess.check_output(
|
||||
"perf record -o {data} {opt}".format(
|
||||
opt=self._option.format(
|
||||
event=self.__event,
|
||||
int=self.__interval),
|
||||
data=data_file).split(),
|
||||
stderr=subprocess.DEVNULL)
|
||||
output = subprocess.check_output("perf report --stdio -i {data}".format(
|
||||
data=data_file).split(),
|
||||
stderr=subprocess.STDOUT)
|
||||
return output.decode()
|
||||
|
||||
def __add_merge_entry(self, merge, entry, mask):
|
||||
entry = list(entry)
|
||||
entry[self.__keyword["overhead"]] = float(
|
||||
entry[self.__keyword["overhead"]])
|
||||
entry[self.__keyword["symbol"]] = int(
|
||||
entry[self.__keyword["symbol"]], 16)
|
||||
|
||||
for i in range(len(merge)):
|
||||
if (merge[i][self.__keyword["command"]] == entry[self.__keyword["command"]]) and (
|
||||
merge[i][self.__keyword["object"]] == entry[self.__keyword["object"]]):
|
||||
entry_addr = entry[self.__keyword["symbol"]] & ~mask
|
||||
merged_addr = merge[i][self.__keyword["symbol"]] & ~mask
|
||||
if entry_addr == merged_addr:
|
||||
merge[i][self.__keyword["overhead"]
|
||||
] += entry[self.__keyword["overhead"]]
|
||||
merge[i][self.__keyword["symbol"]] = merged_addr
|
||||
return
|
||||
|
||||
merge.append(entry)
|
||||
return
|
||||
|
||||
def __addr_merge(self, info, mask):
|
||||
merge = []
|
||||
i = 0
|
||||
while True:
|
||||
try:
|
||||
int(info[i][self.__keyword["symbol"]], 16)
|
||||
except ValueError:
|
||||
i += 1
|
||||
continue
|
||||
except IndexError:
|
||||
break
|
||||
self.__add_merge_entry(merge, info.pop(i), mask)
|
||||
|
||||
for i in range(len(merge)):
|
||||
merge[i][self.__keyword["overhead"]
|
||||
] = "%2.2f" % merge[i][self.__keyword["overhead"]]
|
||||
merge[i][self.__keyword["symbol"]
|
||||
] = "0x%016x" % merge[i][self.__keyword["symbol"]]
|
||||
info.append(merge[i])
|
||||
return
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
pattern = re.compile(
|
||||
"^\ {2,}(\d.*?)%\ {2,}(\w.*?)\ {1,}(.*?)\ {2,}\[[.|k]\]\ (\w.*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
keys = []
|
||||
opts, args = getopt.getopt(
|
||||
para.split(), None, [
|
||||
'fields=', 'addr-merge='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(val)
|
||||
continue
|
||||
elif opt in ('--addr-merge'):
|
||||
addr_mask = int(val, 16)
|
||||
continue
|
||||
|
||||
self.__addr_merge(searchObj, addr_mask)
|
||||
|
||||
ret = []
|
||||
for obj in searchObj:
|
||||
entry = []
|
||||
for event in keys:
|
||||
entry.append(obj[self.__keyword[event]])
|
||||
ret.append(entry)
|
||||
return ret
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "raw"):
|
||||
return str(info)
|
||||
elif (fmt == "data"):
|
||||
return info
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = PerfTop("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2 --event=cycles;--fields=overhead --fields=symbol --addr-merge=0xffffffffffffffff")
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["info", "stat", "topo"]
|
||||
|
||||
from . import *
|
||||
@ -1,69 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the CPU info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CpuInfo(Monitor):
|
||||
"""To collect the CPU info"""
|
||||
_module = "CPU"
|
||||
_purpose = "INFO"
|
||||
_option = "-c processor"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "lshw"
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("xml, json")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "xml"):
|
||||
o_xml = subprocess.check_output(
|
||||
"{cmd} -xml {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option).split(),
|
||||
stderr=subprocess.DEVNULL)
|
||||
return o_xml.decode()
|
||||
elif (fmt == "json"):
|
||||
o_json = subprocess.check_output("{cmd} -json {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=subprocess.DEVNULL)
|
||||
return o_json.decode()
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = CpuInfo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,56 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the CPU isolation info.
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
import subprocess
|
||||
from monitor.common import *
|
||||
|
||||
|
||||
class CpuIsolate(Monitor):
|
||||
"""To collect the CPU isolation info"""
|
||||
_module = "CPU"
|
||||
_purpose = "ISOLATION"
|
||||
_path = "/proc/cmdline"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "cat"
|
||||
|
||||
def _get(self, para=None):
|
||||
output = subprocess.check_output("{cmd} {path}".format(
|
||||
cmd=self.__cmd, path=self._path).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para=None):
|
||||
"""
|
||||
decode collected CPU isolation info.
|
||||
|
||||
:param info: The collected info string
|
||||
:param para: The option for decode, para is None
|
||||
:returns info: decoded info
|
||||
"""
|
||||
pattern = re.compile(r"isolcpus=([\d,]*)", re.MULTILINE)
|
||||
cpus = pattern.findall(info)
|
||||
return " ".join(cpus).strip()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = CpuIsolate("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,183 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the CPU stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CpuStat(Monitor):
|
||||
"""To collect the CPU stat info"""
|
||||
_module = "CPU"
|
||||
_purpose = "STAT"
|
||||
_option = "-u -P ALL {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "mpstat"
|
||||
self.__interval = 1
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("json")
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--cpu=n, --fields=time/cpu/usr/nice/sys/iowait/irq/soft/steal/guest/gnice/idle")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json"):
|
||||
o_json = subprocess.check_output(
|
||||
"{cmd} -o JSON {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return o_json.decode()
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"cpu": 1,
|
||||
"usr": 2,
|
||||
"nice": 3,
|
||||
"sys": 4,
|
||||
"iowait": 5,
|
||||
"irq": 6,
|
||||
"soft": 7,
|
||||
"steal": 8,
|
||||
"guest": 9,
|
||||
"gnice": 10,
|
||||
"idle": 11,
|
||||
"cutil": 12}
|
||||
|
||||
keys = []
|
||||
cpu = -1 # -1 means all
|
||||
threshold = 0
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['cpu=', 'threshold=', 'fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--cpu'):
|
||||
if val.isdigit():
|
||||
cpu = val
|
||||
else:
|
||||
err = ValueError("Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
if opt in ('--threshold'):
|
||||
try:
|
||||
threshold = float(val)
|
||||
except ValueError:
|
||||
err = ValueError("Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {2,}(\d*|all)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)\ {2,}(\d*\.\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data for {}".format(cpu))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
stats = []
|
||||
for stat in searchObj:
|
||||
curr = list(stat)
|
||||
curr.append("{:.2f}".format(
|
||||
float(stat[keyword["usr"]]) + float(stat[keyword["nice"]]) +
|
||||
float(stat[keyword["sys"]]) + float(stat[keyword["irq"]]) +
|
||||
float(stat[keyword["soft"]]) + float(stat[keyword["steal"]])))
|
||||
stats.append(curr)
|
||||
|
||||
if cpu == -1 and threshold > 0:
|
||||
cutil_sum = 0
|
||||
cutil_num = 0
|
||||
for i in range(1, len(stats)):
|
||||
if float(stats[i][keyword["cutil"]]) > threshold:
|
||||
cutil_sum += float(stats[i][keyword["cutil"]])
|
||||
cutil_num += 1
|
||||
if cutil_num == 0:
|
||||
stats[0][keyword["cutil"]] = "{:.2f}".format(cutil_sum)
|
||||
else:
|
||||
stats[0][keyword["cutil"]] = "{:.2f}".format(cutil_sum / cutil_num)
|
||||
|
||||
for i in keys:
|
||||
ret = ret + " " + stats[cpu+1][i]
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = CpuStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--threshold=10.0 --fields=usr --fields=sys --fields=iowait --fields=irq --fields=guest")
|
||||
@ -1,69 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the CPU topo.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CpuTopo(Monitor):
|
||||
"""To collect the CPU topo"""
|
||||
_module = "CPU"
|
||||
_purpose = "TOPO"
|
||||
_option = "--no-io --ignore misc"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("xml")
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
if 0 == subprocess.call("which lstopo-no-graphics".split(),
|
||||
stdout=no_print, stderr=no_print):
|
||||
self.__cmd = "lstopo-no-graphics"
|
||||
elif 0 == subprocess.call("which lstopo".split(), stdout=no_print,
|
||||
stderr=no_print):
|
||||
self.__cmd = "lstopo"
|
||||
|
||||
def _get(self, para=None):
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split())
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "xml"):
|
||||
o_xml = subprocess.check_output("{cmd} --of xml {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split())
|
||||
return o_xml.decode()
|
||||
elif (fmt == "table"):
|
||||
o_xml = subprocess.check_output("{cmd} --of ascii {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split())
|
||||
return o_xml.decode()
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = CpuTopo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Init file.
|
||||
"""
|
||||
|
||||
__all__ = ["iostat", "topo"]
|
||||
|
||||
from . import *
|
||||
@ -1,143 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the storage stat info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class IoStat(Monitor):
|
||||
"""To collect the storage stat info"""
|
||||
_module = "STORAGE"
|
||||
_purpose = "STAT"
|
||||
_option = "-xmt {int} 2"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "iostat"
|
||||
self.__interval = 1
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("json")
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--device=x, --fields=dev/rs/ws/rMBs/wMBs/rrqms/wrqms/rrqm/wrqm/r_await/w_await/aqu-sz/rareq-sz/wareq-sz/svctm/util")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json"):
|
||||
o_json = subprocess.check_output(
|
||||
"{cmd} -o JSON {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return o_json.decode()
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"dev": 0,
|
||||
"rs": 1,
|
||||
"ws": 2,
|
||||
"rMBs": 3,
|
||||
"wMBs": 4,
|
||||
"rrqms": 5,
|
||||
"wrqms": 6,
|
||||
"rrqm": 7,
|
||||
"wrqm": 8,
|
||||
"r_await": 9,
|
||||
"w_await": 10,
|
||||
"aqu-sz": 11,
|
||||
"rareq-sz": 12,
|
||||
"wareq-sz": 13,
|
||||
"svctm": 14,
|
||||
"util": 15}
|
||||
|
||||
keys = []
|
||||
dev = "sd.*?"
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['device=', 'fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--device'):
|
||||
dev = val
|
||||
continue
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(" +
|
||||
dev +
|
||||
")\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)\ {1,}(\d*\.\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data for {}".format(dev))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = IoStat("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--fields=wMBs --fields=rMBs")
|
||||
@ -1,70 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the storage topo.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import json
|
||||
import dict2xml
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class StorageTopo(Monitor):
|
||||
"""To collect the storage topo"""
|
||||
_module = "STORAGE"
|
||||
_purpose = "TOPO"
|
||||
_option = "-c storage"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "lshw"
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("xml, json")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json") or (fmt == "xml"):
|
||||
o_json = subprocess.check_output("{cmd} -json".format(
|
||||
cmd=self.__cmd).split(), stderr=subprocess.DEVNULL)
|
||||
info = o_json.decode()
|
||||
all = json.loads(info)
|
||||
|
||||
dict_datas = get_class_type(all, "storage")
|
||||
if (fmt == "json"):
|
||||
return json.dumps(dict_datas, indent=2)
|
||||
elif (fmt == "xml"):
|
||||
return dict2xml.dict2xml(dict_datas, "topology")
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = StorageTopo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,20 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The import content of the package.
|
||||
"""
|
||||
|
||||
__all__ = ["bios", "ldavg", "tasks", "filed", "interrupts"]
|
||||
|
||||
from . import *
|
||||
@ -1,75 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the system BIOS info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import json
|
||||
import dict2xml
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SysBiosInfo(Monitor):
|
||||
"""To collect the system BIOS info"""
|
||||
_module = "SYS"
|
||||
_purpose = "BIOS"
|
||||
_option = "-c system"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "lshw"
|
||||
self.format.__func__.__doc__ = Monitor.format.__doc__ % ("xml, json")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open('/dev/null', 'w') as no_print:
|
||||
output = subprocess.check_output("{cmd} {opt}".format(
|
||||
cmd=self.__cmd, opt=self._option).split(),
|
||||
stderr=no_print)
|
||||
return output.decode()
|
||||
|
||||
def format(self, info, fmt):
|
||||
if (fmt == "json") or (fmt == "xml"):
|
||||
o_json = subprocess.check_output("{cmd} -json".format(
|
||||
cmd=self.__cmd).split(), stderr=subprocess.DEVNULL)
|
||||
info = o_json.decode()
|
||||
all = json.loads(info)
|
||||
|
||||
datas = []
|
||||
walk_class_type(all, "memory", "BIOS", datas)
|
||||
all.pop("children")
|
||||
datas.append(all)
|
||||
dict_datas = {}
|
||||
dict_datas["BIOS"] = datas
|
||||
if (fmt == "json"):
|
||||
return json.dumps(dict_datas, indent=2)
|
||||
elif (fmt == "xml"):
|
||||
return dict2xml.dict2xml(dict_datas, "info")
|
||||
else:
|
||||
return Monitor.format(self, info, fmt)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = SysBiosInfo("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,96 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the file handles util info.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SysFdUtil(Monitor):
|
||||
"""To collect the file handles util info"""
|
||||
_module = "SYS"
|
||||
_purpose = "FDUTIL"
|
||||
_option = "/proc/sys/fs/file-nr"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--fields=allocated/pending/maximum/fd-util")
|
||||
|
||||
def _get(self, para=None):
|
||||
with open(self._option, 'r') as f:
|
||||
fdinfo = f.read()
|
||||
return fdinfo
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"allocated": 0,
|
||||
"pending": 1,
|
||||
"maximum": 2,
|
||||
"fd-util": "fd-util"}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d*)\s{1,}(\d*)\s{1,}(\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
if type(i).__name__ == 'int':
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
elif i == "fd-util":
|
||||
util = int(searchObj[-1][keyword["allocated"]]) / \
|
||||
int(searchObj[-1][keyword["maximum"]]) * 100
|
||||
ret = ret + " " + str(util)
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = SysFdUtil("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
";--fields=fd-util")
|
||||
@ -1,73 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the system device interrupts.
|
||||
"""
|
||||
import getopt
|
||||
import re
|
||||
import sys
|
||||
import subprocess
|
||||
from monitor.common import *
|
||||
|
||||
|
||||
class SysInterrupts(Monitor):
|
||||
"""To collect the system device interrupts"""
|
||||
_module = "SYS"
|
||||
_purpose = "INTERRUPTS"
|
||||
_option = '{print$1$NF}'
|
||||
_path = "/proc/interrupts"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % "--nic=x"
|
||||
self.__cmd = "awk"
|
||||
|
||||
def _get(self, para=None):
|
||||
output = subprocess.check_output("{cmd} {opt} {path}".format(
|
||||
cmd=self.__cmd, opt=self._option, path=self._path).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
"""
|
||||
decode collected CPU interrupts info.
|
||||
|
||||
:param info: The collected info string
|
||||
:param para: The option for decode,
|
||||
[%s]:
|
||||
"--nic=" to select which nic
|
||||
:returns info: decoded info
|
||||
"""
|
||||
if para is None:
|
||||
pattern = re.compile(r"^(\d*):.*", re.MULTILINE)
|
||||
interrupts = pattern.findall(info)
|
||||
return " ".join(interrupts)
|
||||
|
||||
nic = ""
|
||||
opts, _ = getopt.getopt(para.split(), None, ['nic='])
|
||||
for opt, val in opts:
|
||||
if opt in '--nic':
|
||||
nic = val
|
||||
break
|
||||
|
||||
pattern = re.compile(r"^(\d*):{}$".format(nic.strip()), re.MULTILINE)
|
||||
interrupts = pattern.findall(info)
|
||||
return " ".join(interrupts).strip()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = SysInterrupts("UT")
|
||||
ct.report(sys.argv[1], sys.argv[2])
|
||||
@ -1,125 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the system load average statistics.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SysLdavg(Monitor):
|
||||
"""To collect the system load average statistics"""
|
||||
_module = "SYS"
|
||||
_purpose = "LDAVG"
|
||||
_option = "-q {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "sar"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--fields=time/runq-sz/plist-sz/ldavg-1/ldavg-5/ldavg-15/blocked/task-util")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"runq-sz": 1,
|
||||
"plist-sz": 2,
|
||||
"ldavg-1": 3,
|
||||
"ldavg-5": 4,
|
||||
"ldavg-15": 5,
|
||||
"blocked": 6,
|
||||
"task-util": "task-util"}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {1,}(\d*)\ {1,}(\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)\ {1,}(\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
if type(i).__name__ == 'int':
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
elif i == "task-util":
|
||||
with open("/proc/sys/kernel/threads-max", 'r') as f:
|
||||
threads_max = f.read()
|
||||
util = int(searchObj[-1][keyword["plist-sz"]]) / \
|
||||
int(threads_max) * 100
|
||||
ret = ret + " " + str(util)
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = SysLdavg("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--fields=plist-sz --fields=ldavg-5 --fields=task-util")
|
||||
@ -1,113 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The sub class of the monitor, used to collect the task creation and switching statistics.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import subprocess
|
||||
import getopt
|
||||
import re
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, "./../../")
|
||||
from monitor.common import *
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SysTasks(Monitor):
|
||||
"""To collect the task creation and switching statistics"""
|
||||
_module = "SYS"
|
||||
_purpose = "TASKS"
|
||||
_option = "-w {int} 1"
|
||||
|
||||
def __init__(self, user=None):
|
||||
Monitor.__init__(self, user)
|
||||
self.__cmd = "sar"
|
||||
self.__interval = 1
|
||||
self.decode.__func__.__doc__ = Monitor.decode.__doc__ % (
|
||||
"--fields=time/procs/cswchs")
|
||||
|
||||
def _get(self, para=None):
|
||||
if para is not None:
|
||||
opts, args = getopt.getopt(para.split(), None, ['interval='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--interval'):
|
||||
if val.isdigit():
|
||||
self.__interval = int(val)
|
||||
else:
|
||||
err = ValueError(
|
||||
"Invalid parameter: {opt}={val}".format(
|
||||
opt=opt, val=val))
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
continue
|
||||
|
||||
output = subprocess.check_output(
|
||||
"{cmd} {opt}".format(
|
||||
cmd=self.__cmd,
|
||||
opt=self._option.format(
|
||||
int=self.__interval)).split())
|
||||
return output.decode()
|
||||
|
||||
def decode(self, info, para):
|
||||
if para is None:
|
||||
return info
|
||||
|
||||
keyword = {"time": 0,
|
||||
"procs": 1,
|
||||
"cswchs": 2}
|
||||
|
||||
keys = []
|
||||
ret = ""
|
||||
|
||||
opts, args = getopt.getopt(para.split(), None, ['fields='])
|
||||
for opt, val in opts:
|
||||
if opt in ('--fields'):
|
||||
keys.append(keyword[val])
|
||||
continue
|
||||
|
||||
pattern = re.compile(
|
||||
"^(\d.*?)\ {1,}(\d*\.?\d*)\ {1,}(\d*\.?\d*)",
|
||||
re.ASCII | re.MULTILINE)
|
||||
searchObj = pattern.findall(info)
|
||||
if len(searchObj) == 0:
|
||||
err = LookupError("Fail to find data")
|
||||
logger.error(
|
||||
"{}.{}: {}".format(
|
||||
self.__class__.__name__,
|
||||
sys._getframe().f_code.co_name,
|
||||
str(err)))
|
||||
raise err
|
||||
|
||||
for i in keys:
|
||||
ret = ret + " " + searchObj[-1][i]
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print('usage: ' + sys.argv[0] + ' fmt path')
|
||||
sys.exit(-1)
|
||||
ct = SysTasks("UT")
|
||||
ct.report(
|
||||
sys.argv[1],
|
||||
sys.argv[2],
|
||||
"--interval=2;--fields=procs --fields=cswchs")
|
||||
@ -1,266 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
The plugin for monitor and configurator.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import monitor
|
||||
import configurator
|
||||
import time
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ThreadedCall(threading.Thread):
|
||||
"""class for function threaded calling"""
|
||||
|
||||
def __init__(self, func, args=()):
|
||||
super(ThreadedCall, self).__init__()
|
||||
self.func = func
|
||||
self.args = args
|
||||
|
||||
def run(self):
|
||||
self.result = self.func(*self.args)
|
||||
|
||||
def get_result(self):
|
||||
threading.Thread.join(self)
|
||||
try:
|
||||
return self.result
|
||||
except Exception as err:
|
||||
return err
|
||||
|
||||
|
||||
class MPI:
|
||||
"""The monitor plugin"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
Initialize.
|
||||
|
||||
:param: None
|
||||
:returns: None
|
||||
:raises: None
|
||||
"""
|
||||
all_mpis = []
|
||||
all_modules = []
|
||||
all_purposes = []
|
||||
for m in monitor.common.Monitor.__subclasses__():
|
||||
all_mpis.append((m._module, m._purpose))
|
||||
all_modules.append(m._module)
|
||||
all_purposes.append(m._purpose)
|
||||
self.get_monitors.__func__.__doc__ = self.get_monitors.__func__.__doc__ % (
|
||||
set(all_modules), set(all_purposes))
|
||||
self.get_monitor.__func__.__doc__ = self.get_monitor.__func__.__doc__ % (
|
||||
all_mpis)
|
||||
|
||||
@classmethod
|
||||
def get_monitors(self, module=None, purpose=None):
|
||||
"""
|
||||
Get monitors of 'module' for 'purpose'.
|
||||
|
||||
:param module(optional): %s
|
||||
:param purpose(optional): %s
|
||||
:returns list: Success, all found monitors or null
|
||||
:raises: None
|
||||
"""
|
||||
mpis = []
|
||||
for m in monitor.common.Monitor.__subclasses__():
|
||||
if (module is not None) and (m._module != module):
|
||||
continue
|
||||
if (purpose is not None) and (m._purpose != purpose):
|
||||
continue
|
||||
me = m()
|
||||
mpis.append(me)
|
||||
return mpis
|
||||
|
||||
@classmethod
|
||||
def get_monitor(self, module, purpose):
|
||||
"""
|
||||
Get monitor of 'module' for 'purpose'.
|
||||
|
||||
:param module & purpose: %s
|
||||
:returns mpi: Success, the found monitor
|
||||
:raises LookupError: Fail, find monitor error
|
||||
"""
|
||||
mpis = MPI.get_monitors(module, purpose)
|
||||
if len(mpis) != 1:
|
||||
err = LookupError("Find {} {}-{} monitors".format(
|
||||
len(mpis), module, purpose))
|
||||
logger.error("MPI.{}: {}".format(
|
||||
sys._getframe().f_code.co_name, str(err)))
|
||||
raise err
|
||||
else:
|
||||
return mpis[0]
|
||||
|
||||
@classmethod
|
||||
def get_monitor_pooled(self, module, purpose, pool):
|
||||
"""
|
||||
Get monitor of 'module' for 'purpose' in pool.
|
||||
|
||||
:param module & purpose: see get_monitor()
|
||||
:param pool: monitors pool for looking up
|
||||
:returns mpi: Success, the found monitor
|
||||
:raises LookupError: Fail, find monitor error
|
||||
"""
|
||||
mpis = []
|
||||
for m in pool:
|
||||
if (module is not None) and (m._module != module):
|
||||
continue
|
||||
if (purpose is not None) and (m._purpose != purpose):
|
||||
continue
|
||||
mpis.append(m)
|
||||
|
||||
if len(mpis) != 1:
|
||||
err = LookupError("Find {} {}-{} monitors in pool".format(
|
||||
len(mpis), module, purpose))
|
||||
logger.error("MPI.{}: {}".format(
|
||||
sys._getframe().f_code.co_name, str(err)))
|
||||
raise err
|
||||
else:
|
||||
return mpis[0]
|
||||
|
||||
@classmethod
|
||||
def get_monitors_data(self, monitors, pool=None):
|
||||
"""
|
||||
Get given monitors report data in one.
|
||||
|
||||
:param monitors: ((module, purpose, options), ...)
|
||||
options is for report(para)
|
||||
:param pool: monitors pool for looking up
|
||||
:returns list: Success, decoded data strings of all given monitors
|
||||
:returns Exceptions: Success, formatted info
|
||||
:raises LookupError: Fail, find monitor error
|
||||
"""
|
||||
mts = []
|
||||
for m in monitors:
|
||||
if pool is None:
|
||||
mon = MPI.get_monitor(m[0], m[1])
|
||||
else:
|
||||
mon = MPI.get_monitor_pooled(m[0], m[1], pool)
|
||||
mt = ThreadedCall(mon.report, ("data", None, m[2]))
|
||||
mts.append(mt)
|
||||
mt.start()
|
||||
|
||||
rets = []
|
||||
for mt in mts:
|
||||
start = time.time()
|
||||
ret = mt.get_result()
|
||||
end = time.time()
|
||||
logger.debug("MPI.{}: Cost {} s to call {}, ret={}".format(
|
||||
sys._getframe().f_code.co_name, end-start, mt.func, str(ret)))
|
||||
if isinstance(ret, Exception):
|
||||
return ret
|
||||
rets += ret
|
||||
return rets
|
||||
|
||||
|
||||
class CPI:
|
||||
"""The configurator plugin"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
Initialize.
|
||||
|
||||
:param: None
|
||||
:returns: None
|
||||
:raises: None
|
||||
"""
|
||||
all_cpis = []
|
||||
all_modules = []
|
||||
all_submods = []
|
||||
for m in configurator.common.Configurator.__subclasses__():
|
||||
all_cpis.append((m._module, m._submod))
|
||||
all_modules.append(m._module)
|
||||
all_submods.append(m._submod)
|
||||
self.get_configurators.__func__.__doc__ = self.get_configurators.__func__.__doc__ % (
|
||||
set(all_modules), set(all_submods))
|
||||
self.get_configurator.__func__.__doc__ = self.get_configurator.__func__.__doc__ % (
|
||||
all_cpis)
|
||||
|
||||
@classmethod
|
||||
def get_configurators(self, module=None, submod=None):
|
||||
"""
|
||||
Get configurators of 'module'.'submod'.
|
||||
|
||||
:param module(optional): %s
|
||||
:param submod(optional): %s
|
||||
:returns list: Success, all found configurators or null
|
||||
:raises: None
|
||||
"""
|
||||
cpis = []
|
||||
for c in configurator.common.Configurator.__subclasses__():
|
||||
if (module is not None) and (c._module != module):
|
||||
continue
|
||||
if (submod is not None) and (c._submod != submod):
|
||||
continue
|
||||
ce = c()
|
||||
cpis.append(ce)
|
||||
return cpis
|
||||
|
||||
@classmethod
|
||||
def get_configurator(self, module, submod):
|
||||
"""
|
||||
Get configurator of 'module'.'submod'.
|
||||
|
||||
:param module & submod: %s
|
||||
:returns cpi: Success, the found configurator
|
||||
:raises LookupError: Fail, find configurator error
|
||||
"""
|
||||
cpis = CPI.get_configurators(module, submod)
|
||||
if len(cpis) != 1:
|
||||
err = LookupError("Find {} {}-{} configurators".format(
|
||||
len(cpis), module, submod))
|
||||
logger.error("CPI.{}: {}".format(
|
||||
sys._getframe().f_code.co_name, str(err)))
|
||||
raise err
|
||||
else:
|
||||
return cpis[0]
|
||||
|
||||
|
||||
# if __name__ == "__main__":
|
||||
# if len(sys.argv) == 4:
|
||||
# ret = MPI.get_monitors(sys.argv[1], sys.argv[2])
|
||||
# print(ret[0].get(sys.argv[3]))
|
||||
# elif len(sys.argv) == 3:
|
||||
# ret = MPI.get_monitors(sys.argv[1], sys.argv[2])
|
||||
# print(ret[0].get())
|
||||
# else:
|
||||
# print('usage: ' + sys.argv[0] + ' module purpose')
|
||||
|
||||
# if __name__ == "__main__":
|
||||
# if len(sys.argv) == 3:
|
||||
# ret = CPI.get_configurators(sys.argv[1])
|
||||
# print(ret[0].set(sys.argv[2]))
|
||||
# else:
|
||||
# print('usage: ' + sys.argv[0] + ' module key=val')
|
||||
|
||||
if __name__ == "__main__":
|
||||
monitors = (
|
||||
("CPU",
|
||||
"STAT",
|
||||
"--interval=1;--cpu=1 --fields=usr --fields=sys --fields=iowait --fields=irq --fields=guest"),
|
||||
("STORAGE",
|
||||
"STAT",
|
||||
"--interval=3;--device=sda --fields=rMBs --fields=wMBs"),
|
||||
("NET",
|
||||
"STAT",
|
||||
"--interval=2;--nic=lo --fields=wKBs --fields=rKBs"),
|
||||
("PERF",
|
||||
"STAT",
|
||||
"--interval=5;--fields=cycles --fields=instructions --fields=cache-misses"))
|
||||
print(MPI.get_monitors_data(monitors))
|
||||
@ -1,36 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Custom exception and warning.
|
||||
"""
|
||||
|
||||
|
||||
class SetConfigError(Exception):
|
||||
"""For setting config errors"""
|
||||
pass
|
||||
|
||||
|
||||
class GetConfigError(Exception):
|
||||
"""For getting config errors"""
|
||||
pass
|
||||
|
||||
|
||||
class NeedRebootWarning(Warning):
|
||||
"""For needing reboot after config"""
|
||||
pass
|
||||
|
||||
|
||||
class NeedConfigWarning(Warning):
|
||||
"""For needing user handle after config"""
|
||||
pass
|
||||
@ -1,12 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
@ -1,57 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Restful api for classification, in order to provide the method of post.
|
||||
"""
|
||||
|
||||
from flask import abort
|
||||
from flask import current_app
|
||||
from flask_restful import Resource
|
||||
from flask_restful import marshal_with_field
|
||||
|
||||
from optimizer.workload_characterization import WorkloadCharacterization
|
||||
from resources.field import classification_post_field
|
||||
from resources.parser import classification_post_parser
|
||||
from utils import utils
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Classification(Resource):
|
||||
@marshal_with_field(classification_post_field)
|
||||
def post(self):
|
||||
args = classification_post_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
|
||||
model_path = args.get("modelpath")
|
||||
data_path = args.get("data")
|
||||
model = args.get("model", None)
|
||||
data = utils.readFromCsv(data_path)
|
||||
if not data:
|
||||
abort("data path may be not exist")
|
||||
|
||||
classification = WorkloadCharacterization(model_path)
|
||||
resource_limit = ""
|
||||
if model is None:
|
||||
resource_limit, workload_type, percentage = classification.identify(data)
|
||||
else:
|
||||
workload_type, percentage = classification.reidentify(data, model)
|
||||
|
||||
profile_name = {}
|
||||
profile_name["workload_type"] = workload_type
|
||||
profile_name["percentage"] = percentage
|
||||
profile_name["resource_limit"] = resource_limit
|
||||
return profile_name, 200
|
||||
|
||||
@ -1,72 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Restful api with collector, in order to provide the method of post.
|
||||
"""
|
||||
|
||||
from flask import current_app
|
||||
from flask_restful import reqparse, Resource
|
||||
from flask_restful import request, marshal_with, marshal_with_field
|
||||
import os
|
||||
import pandas as pd
|
||||
from resources.field import profile_get_field
|
||||
from resources.parser import collector_post_parser
|
||||
from plugin.plugin import CPI
|
||||
from plugin.plugin import MPI
|
||||
from utils.npipe import getNpipe
|
||||
|
||||
parser = reqparse.RequestParser()
|
||||
|
||||
|
||||
class Collector(Resource):
|
||||
@marshal_with_field(profile_get_field)
|
||||
def post(self):
|
||||
args = collector_post_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
monitors = []
|
||||
mpis = []
|
||||
for monitor in args.get("monitors"):
|
||||
monitors.append([monitor["module"], monitor["purpose"], monitor["field"]])
|
||||
mpis.append(MPI.get_monitor(monitor["module"], monitor["purpose"]))
|
||||
collect_num = args.get("sample_num")
|
||||
|
||||
nPipe = getNpipe(args.get("pipe"))
|
||||
current_app.logger.info(monitors)
|
||||
|
||||
data = []
|
||||
|
||||
for i in range(collect_num):
|
||||
raw_data = MPI.get_monitors_data(monitors, mpis)
|
||||
current_app.logger.info(raw_data)
|
||||
|
||||
float_data = list()
|
||||
for x in raw_data:
|
||||
float_data.append(float(x))
|
||||
|
||||
data.append(float_data)
|
||||
|
||||
str_data = [str(round(data, 3)) for data in float_data]
|
||||
nPipe.write(" ".join(str_data) + "\n")
|
||||
|
||||
path = "/tmp/test.csv"
|
||||
save_file(path, data)
|
||||
result = {}
|
||||
result["path"] = path
|
||||
return result, 200
|
||||
|
||||
|
||||
def save_file(file_name, datas):
|
||||
print(datas)
|
||||
writer = pd.DataFrame(columns=None, data=datas)
|
||||
writer.to_csv(file_name, encoding='utf-8', header=0, index=False)
|
||||
@ -1,105 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Restful api for configurator, in order to provide the method of put and get.
|
||||
"""
|
||||
|
||||
from flask import abort
|
||||
from flask import current_app
|
||||
from flask_restful import Resource
|
||||
from flask_restful import marshal_with_field
|
||||
from resources.parser import property_put_parser
|
||||
from resources.parser import configurator_get_parser
|
||||
from resources.field import configurator_put_field
|
||||
from plugin.plugin import CPI
|
||||
|
||||
|
||||
class Configurator(Resource):
|
||||
@marshal_with_field(configurator_put_field)
|
||||
def put(self):
|
||||
"""
|
||||
calling cpi set method to set the value of the given key
|
||||
:param section: The section of the cpi
|
||||
:param key: the key to be set
|
||||
:param value: the value of the given key
|
||||
:returns status: the status return by the cpi set method
|
||||
:returns value: the message return by the cpi set method
|
||||
"""
|
||||
result = {}
|
||||
args = property_put_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
section = args.get("section").upper()
|
||||
modules = section.split(".")
|
||||
submodule = None
|
||||
if len(modules) > 1:
|
||||
submodule = modules[1]
|
||||
configurators = CPI.get_configurators(modules[0], submodule)
|
||||
if len(configurators) < 1:
|
||||
result["status"] = "module %s is not exist" % (section)
|
||||
return result, 200
|
||||
configurator = configurators[0]
|
||||
|
||||
key = args.get("key")
|
||||
value = args.get("value")
|
||||
params = key + "=" + value
|
||||
ret = configurator.set(params)
|
||||
if not ret:
|
||||
result["status"] = "OK"
|
||||
elif isinstance(ret, Warning):
|
||||
result["status"] = "WARNING"
|
||||
result["value"] = str(ret)
|
||||
elif isinstance(ret, Exception):
|
||||
result["status"] = "ERROR"
|
||||
result["value"] = str(ret)
|
||||
|
||||
return result, 200
|
||||
|
||||
def get(self):
|
||||
"""
|
||||
calling the cpi check method to check the value of the given key
|
||||
:param section: The section of the cpi
|
||||
:param key: the key to be get
|
||||
:param value: the expect value
|
||||
:returns status: the status compare the expect value with the real value
|
||||
:returns value: the system real value
|
||||
"""
|
||||
result = {}
|
||||
args = configurator_get_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
section = args.get("section").upper()
|
||||
key = args.get("key")
|
||||
value = args.get("value", "")
|
||||
|
||||
modules = section.split(".")
|
||||
submodule = None
|
||||
if len(modules) > 1:
|
||||
submodule = modules[1]
|
||||
configurators = CPI.get_configurators(modules[0], submodule)
|
||||
|
||||
if len(configurators) < 1:
|
||||
abort(404)
|
||||
|
||||
configurator = configurators[0]
|
||||
realValue = configurator.get(key)
|
||||
result["value"] = realValue
|
||||
if realValue:
|
||||
if configurator._check(value, realValue):
|
||||
result["status"] = "OK"
|
||||
else:
|
||||
result["status"] = realValue
|
||||
else:
|
||||
result["status"] = "UNKNOWN"
|
||||
|
||||
return result, 200
|
||||
|
||||
@ -1,48 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Fields used by restful api.
|
||||
"""
|
||||
|
||||
from flask_restful import fields
|
||||
|
||||
profile_field = {
|
||||
'profile': fields.String,
|
||||
}
|
||||
|
||||
profile_get_field = fields.Nested(profile_field)
|
||||
|
||||
|
||||
param_field = {
|
||||
'status': fields.String,
|
||||
'value': fields.String,
|
||||
}
|
||||
|
||||
configurator_put_field = fields.Nested(param_field)
|
||||
|
||||
optimizer_field = {
|
||||
'status': fields.String
|
||||
}
|
||||
|
||||
optimizer_post_field = {
|
||||
'task_id': fields.String
|
||||
}
|
||||
|
||||
classification_field = {
|
||||
'resource_limit': fields.String,
|
||||
'workload_type': fields.String,
|
||||
'percentage': fields.Float,
|
||||
}
|
||||
|
||||
classification_post_field = fields.Nested(classification_field)
|
||||
@ -1,90 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Restful api for monitor, in order to provide the method of post and get.
|
||||
"""
|
||||
|
||||
from flask import abort
|
||||
from flask import current_app
|
||||
from flask_restful import reqparse, Resource
|
||||
from resources.parser import monitor_get_parser
|
||||
from resources.parser import monitor_post_parser
|
||||
from plugin.plugin import MPI
|
||||
import getopt
|
||||
|
||||
parser = reqparse.RequestParser()
|
||||
|
||||
|
||||
class Monitor(Resource):
|
||||
def get(self):
|
||||
result = {}
|
||||
args = monitor_get_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
module = args.get("module")
|
||||
purpose = args.get("purpose", None)
|
||||
fmt = args.get("fmt")
|
||||
path = args.get("path")
|
||||
para = args.get("para")
|
||||
|
||||
monitors = MPI.get_monitors(module, purpose)
|
||||
if len(monitors) < 1:
|
||||
return result, 200
|
||||
|
||||
monitor = monitors[0]
|
||||
|
||||
ret = monitor.report(fmt, path, para)
|
||||
if ret is not None:
|
||||
result["status"] = str(ret)
|
||||
else:
|
||||
result["status"] = "OK"
|
||||
|
||||
return result, 200
|
||||
|
||||
def post(self):
|
||||
result = {}
|
||||
args = monitor_post_parser.parse_args()
|
||||
current_app.logger.info(args)
|
||||
|
||||
monitors = []
|
||||
module = args.get("module")
|
||||
purpose = args.get("purpose", None)
|
||||
field = args.get("field")
|
||||
|
||||
monitors = MPI.get_monitors(module, purpose)
|
||||
|
||||
if len(monitors) < 1:
|
||||
abort(404)
|
||||
|
||||
monitor = monitors[0]
|
||||
ret = monitor.report("data", None, field)
|
||||
opts, args = getopt.getopt(field.split(";")[1].split(), None, ['fields=', 'addr-merge='])
|
||||
opt = []
|
||||
for o, v in opts:
|
||||
if o in ("--fields"):
|
||||
opt.append(v)
|
||||
|
||||
events = []
|
||||
for index in range(len(ret)):
|
||||
if isinstance(ret[index], list):
|
||||
event = {}
|
||||
for i in range(len(ret[index])):
|
||||
event[opt[i]] = ret[index][i]
|
||||
events.append(event)
|
||||
else:
|
||||
result[opt[index]] = ret[index]
|
||||
|
||||
if len(events) > 0:
|
||||
result["data"] = events
|
||||
return result, 200
|
||||
|
||||
@ -1,106 +0,0 @@
|
||||
#!/usr/bin/python3
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright (c) 2019 Huawei Technologies Co., Ltd.
|
||||
# A-Tune is licensed under the Mulan PSL v1.
|
||||
# You can use this software according to the terms and conditions of the Mulan PSL v1.
|
||||
# You may obtain a copy of Mulan PSL v1 at:
|
||||
# http://license.coscl.org.cn/MulanPSL
|
||||
# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR
|
||||
# PURPOSE.
|
||||
# See the Mulan PSL v1 for more details.
|
||||
# Create: 2019-10-29
|
||||
|
||||
"""
|
||||
Restful api for optimizer, in order to provide the method of post, get, put and delete.
|
||||
"""
|
||||
from flask import Flask, abort
|
||||
from flask import current_app
|
||||
from flask_restful import reqparse, Resource
|
||||
from flask_restful import request, marshal_with, marshal_with_field
|
||||
import os
|
||||
import uuid
|
||||
from multiprocessing import Queue, Pipe
|
||||
|
||||
import resources.optimizer
|
||||
from resources.field import optimizer_post_field
|
||||
from resources.parser import optimizer_post_parser
|
||||
from resources.parser import optimizer_put_parser
|
||||
from optimizer import optimizer
|
||||
from utils import task_cache
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
parser = reqparse.RequestParser()
|
||||
|
||||
|
||||
class Optimizer(Resource):
|
||||
def get(self, task_id=None):
|
||||
result = []
|
||||
if not task_id:
|
||||
tasks_all = task_cache.TasksCache.getInstance().get_all()
|
||||
for task in tasks_all:
|
||||
result.append(task)
|
||||
else:
|
||||
task = task_cache.TasksCache.getInstance().get(task_id)
|
||||
if not task:
|
||||
abort(404, "taskid {0} not found".format(task_id))
|
||||
result.append(task_id)
|
||||
return result, 200
|
||||
|
||||
# @marshal_with_field(optimizdder_post_field)
|
||||
def post(self):
|
||||
args = optimizer_post_parser.parse_args()
|
||||
logger.info(args)
|
||||
|
||||
if args["max_eval"] < 10:
|
||||
abort(400, "max_eval must be >=10")
|
||||
task_id = str(uuid.uuid1())
|
||||
|
||||
parent_conn, child_conn = Pipe()
|
||||
engine = optimizer.Optimizer(task_id, args["knobs"], child_conn, max_eval=args.get("max_eval"))
|
||||
engine.start()
|
||||
|
||||
value = {}
|
||||
value["process"] = engine
|
||||
value["pipe"] = parent_conn
|
||||
task_cache.TasksCache.getInstance().set(task_id, value)
|
||||
|
||||
result = {}
|
||||
|
||||
result["task_id"] = task_id
|
||||
return result, 201
|
||||
|
||||
def put(self, task_id):
|
||||
if not task_id:
|
||||
abort(404)
|
||||
task = task_cache.TasksCache.getInstance().get(task_id)
|
||||
if not task:
|
||||
abort(404, "taskid {0} not found".format(task_id))
|
||||
|
||||
args = optimizer_put_parser.parse_args()
|
||||
logger.info(args)
|
||||
out_queue = task["pipe"]
|
||||
if args["iterations"] != 0:
|
||||
out_queue.send(args.get("value"))
|
||||
|
||||
result = {}
|
||||
values = out_queue.recv()
|
||||
params = ["%s=%s" % (k, v) for k, v in values.items()]
|
||||
result["param"] = ",".join(params)
|
||||
|
||||
return result, 200
|
||||
|
||||
def delete(self, task_id):
|
||||
if not task_id:
|
||||
abort(404)
|
||||
process = task_cache.TasksCache.getInstance().get(task_id)
|
||||
if not process:
|
||||
abort(404, "taskid {0} not found".format(task_id))
|
||||
process["process"].stopProcess()
|
||||
task_cache.TasksCache.getInstance().get(task_id)["pipe"].close()
|
||||
|
||||
process = task_cache.TasksCache.getInstance().delete(task_id)
|
||||
return {}, 200
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user