first commit

This commit is contained in:
huangtuq 2021-12-28 11:26:52 +08:00
commit 43e64a2a7b
249 changed files with 81661 additions and 0 deletions

8
LICENSE Normal file
View File

@ -0,0 +1,8 @@
The MIT License (MIT)
Copyright © 2021 <copyright holders>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

35
README.md Normal file
View File

@ -0,0 +1,35 @@
# 简介
致力于打造一个集主机管理、配置部署、监控报警、异常诊断、安全审计等一系列功能的自动化运维平台。
探索创新的sysAK、ossre诊断工具及高效的LCCLibbpf Compiler Collection开发编译平台和netinfo网络抖动问题监控系统等
实现系统问题的快速上报、分析与解决,提升集群的全自动运维效率,构建大规模集群运维生态链。
# 目标
通过社区合作,打造出一个自动化运维平台,涵盖云场景中各种典型服务场景,包括线上问题分析诊断、资源和异常事件监控、系统修复业务止血,
安全审计和CVE补丁推送等各种功能提供强大的底层系统运维能力融合到统一的智能运维平台实现自动化运维。
# 功能
* 主机管理
* 配置中心
* 安全审计
* 监控报警
* 智能问题诊断
* 发布部署
# 安装部署
* 打包:执行打包脚本 package.sh生成发布包用于部署。
```
# 前端打包需要本地已经具备yarn环境如不具备需要提前部署yarn环境然后进到 sysom_web 目录执行 yarn 命令安装依赖包。
# mac 环境下 yarn 安装可以采用脚本curl -o- -L https://yarnpkg.com/install.sh | bash
# 安装yarn完成后执行下列命令打包项目
bash package.sh
```
* 部署:将发布包拷贝到目标机器上,解压,进入到目录中执行部署脚本。
```
tar xf sysomRelease-20211207022031.tar.gz
cd sysomRelease-20211207022031
# 默认部署在 /home/sysom 目录,可以接参数自定义安装目录,
bash deploy.sh /usr/local
```

20
package.sh Normal file
View File

@ -0,0 +1,20 @@
#!/bin/bash
green="\033[32m"
RELEASE=sysomRelease-$(date +"%Y%m%d%H%M%S")
APIDIR=sysom_api
WEBDIR=sysom_web
TOOLSDIR=tools
# build web
pushd sysom_web
yarn build
popd
mkdir -p ${RELEASE}
cp -r ${APIDIR}/ ${TOOLSDIR}/ ${RELEASE}/
cp -r ${WEBDIR}/dist/ ${RELEASE}/${WEBDIR}/
mkdir -p ${RELEASE}/${WEBDIR}/download/
cp ${TOOLSDIR}/deploy/deploy.sh ${RELEASE}/
tar czf ${RELEASE}.tar.gz ${RELEASE}/
rm -rf ${RELEASE}
printf "$green The release pacakge is ${RELEASE}.tar.gz"

15
script/monitor/Readme.txt Normal file
View File

@ -0,0 +1,15 @@
本目录脚本主要用于一键部署prometheus + grafana监控
主要有以下步骤:
1、从grafana官网下载grafana安装包并安装启动
2、从prometheus官网下载prometheus的安装包并安装启动设置systemd 服务
3、从prometheus官网下载node_exporter安装包
4、配置grafana的prometheus数据库配置dashboard
直接在sysom server机器执行bash -x monitor_server_deploy.sh脚本即可。
另外,
1、如果受外网访问限制你也可以预先下载好grafana,prometheus,node_exporter等安装包
并把安装包,放置到/usr/local/sysom/monitor目录下。
2、sysom项目当前采用grafana-8.2.5-1.x86_64.rpmprometheus-2.29.1.linux-amd64.tar.gznode_exporter-1.2.2.linux-amd64.tar.gz
等软件包版本如果你需要使用其它软件包版本进行安装你需要修改一下monitor_server_deploy.sh中的版本号信息。

View File

@ -0,0 +1,32 @@
#!/bin/bash -x
GAFANA_CONFIG=/etc/grafana/grafana.ini
##modify grafana.ini
sed 's/;allow_embedding = false/allow_embedding = true/g' -i $GAFANA_CONFIG
sed 's/;disable_login_form = false/disable_login_form = true/g' -i $GAFANA_CONFIG
sed '/enable anonymous access/{n;s/;enabled = false/enabled = true/;}' -i $GAFANA_CONFIG
##login grafana, and get cookie
curl --location --request POST 'http://127.0.0.1:3000/login' --header 'Content-Type: application/json' \
--data-raw '{"user": "admin", "password": "admin"}' -c cookie
##api key
#curl -c cookie -b cookie --location --request POST 'http://127.0.0.1:3000/api/auth/keys' --header 'Content-Type: application/json' \
#--data-raw '{"name": "test_key", "role": "Admin", "secondsToLive": 120}' > api_key.json
#api_key=`cat api_key.json | awk -F"\"" '{print $(NF-1)}'`
##initial database
curl -c cookie -b cookie --location --request POST 'http://127.0.0.1:3000/api/datasources' \
--header 'Content-Type: application/json' \
--data-raw '{"name": "sysom-prometheus", "type": "prometheus", "url": "http://127.0.0.1:9090","access": "proxy", "isDefault": true}'
##initial sysom-dashborad
curl -c cookie -b cookie --location --request POST 'http://127.0.0.1:3000/api/dashboards/db' \
--header 'Content-Type: application/json' \
-d @"sysom-dashboard.json"
rm -f cookie
systemctl restart grafana-server

View File

@ -0,0 +1,6 @@
#!/bin/bash
mkdir -p /usr/local/sysom/monitor/
pushd ../
cp grafana-8.2.5-1.x86_64.rpm prometheus-2.29.1.linux-amd64.tar.gz node_exporter-1.2.2.linux-amd64.tar.gz /usr/local/sysom/monitor/
popd

View File

@ -0,0 +1,35 @@
#!/bin/bash -x
RESOURCE_DIR=/usr/local/sysom/monitor
disable_prometheus()
{
systemctl stop prometheus
systemctl disable prometheus
}
disable_grafana()
{
systemctl stop grafana-server
systemctl disable grafana-server
}
uninstall_pkg()
{
rm -f /usr/lib/systemd/system/prometheus.service
rm -rf $RESOURCE_DIR/prometheus
yum erase -y grafana.x86_64
rm -rf /usr/share/grafana
rm -rf /var/lib/grafana
rm -rf /var/logre/grafana
rm -rf /etc/grafana
}
main()
{
disable_prometheus
disable_grafana
uninstall_pkg
}
main

View File

@ -0,0 +1,174 @@
#!/bin/bash -x
UPLOAD_DIR=/usr/local/sysom/target/sysom_web/download/
GRAFANA_PKG=grafana-8.2.5-1.x86_64.rpm
RESOURCE_DIR=/usr/local/sysom/monitor
PROMETHEUS_VER=2.29.1
PROMETHEUS_ARCH=linux-amd64
PROMETHEUS_PKG=prometheus-${PROMETHEUS_VER}.${PROMETHEUS_ARCH}
PROMETHEUS_TAR=$PROMETHEUS_PKG.tar.gz
NODE_EXPORTER_VER=1.2.2
NODE_EXPORTER_PKG=node_exporter-${NODE_EXPORTER_VER}.${PROMETHEUS_ARCH}
NODE_EXPORTER_TAR=$NODE_EXPORTER_PKG.tar.gz
service_head="
[Unit]
Description=SysOM Monitor Promethues
Documentation=SysOM Monitor Promethues
Wants=network-online.target
After=network-online.target
[Service]
"
service_tail="
[Install]
WantedBy=multi-user.target
"
service_task="ExecStart="
start_grafana_service()
{
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
}
install_grafana()
{
echo "install grafana......"
pushd $RESOURCE_DIR
ls | grep $GRAFANA_PKG 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
wget https://dl.grafana.com/oss/release/$GRAFANA_PKG
ls
fi
yum install -y $GRAFANA_PKG
popd
}
##configure prometheus.yml to auto discovery new nodes
add_auto_discovery()
{
_dir=${PWD}
pushd $RESOURCE_DIR/prometheus
mkdir -p node
cat << EOF >> prometheus.yml
- job_name: 'auto_discovery'
file_sd_configs:
- files:
- "/usr/local/sysom/monitor/prometheus/node/node.json"
refresh_interval: 10s
EOF
popd
cp prometheus_get_node.py $RESOURCE_DIR/prometheus/
}
start_prometheus_service()
{
##create prometheus service
prometheus_exec=${RESOURCE_DIR}/prometheus/prometheus
prometheus_config="--config.file=\"${RESOURCE_DIR}/prometheus/prometheus.yml\""
prometheus_data="--storage.tsdb.path=\"${RESOURCE_DIR}/prometheus/data/\""
prometheus_service_task="$service_task$prometheus_exec $prometheus_config $prometheus_data"
cat << EOF > prometheus.service
$service_head
$prometheus_service_task
$service_tail
EOF
cat prometheus.service
add_auto_discovery
mv prometheus.service /usr/lib/systemd/system
systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus
}
##download and install prometheus
install_prometheus()
{
echo "install prometheus......"
pushd $RESOURCE_DIR
rm -rf prometheus
ls | grep $PROMETHEUS_TAR 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
wget https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VER}/$PROMETHEUS_TAR
ls
fi
tar -zxvf $PROMETHEUS_TAR
##rename the prometheus directory
mv $PROMETHEUS_PKG prometheus
popd
}
##download node_exporter pkg and upload to sysom dir
upload_node_exporter()
{
echo "install node_exporter......"
pushd $RESOURCE_DIR
rm -rf node_exporter
ls | grep $NODE_EXPORTER_TAR 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
echo "wget node_exporter"
wget https://github.com/prometheus/node_exporter/releases/download/v$NODE_EXPORTER_VER/$NODE_EXPORTER_TAR
fi
# tar -zxvf $NODE_EXPORTER_TAR
# mv $NODE_EXPORTER_PKG node_exporter
cp $NODE_EXPORTER_TAR $UPLOAD_DIR
popd
cp node_exporter_deploy.sh $UPLOAD_DIR
}
configure_grafana()
{
bash -x grafana_api_set.sh
}
configure_cron()
{
echo "* * * * * python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
echo "* * * * * sleep 10;python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
echo "* * * * * sleep 20;python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
echo "* * * * * sleep 30;python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
echo "* * * * * sleep 40;python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
echo "* * * * * sleep 50;python3 $RESOURCE_DIR/prometheus/prometheus_get_node.py" >> /var/spool/cron/root
}
main()
{
yum install -y wget
echo "perpare download resource packages: grafana, prometheus, node_exporter"
mkdir -p $RESOURCE_DIR
# bash -x local_copy_pkg.sh
install_grafana
install_prometheus
upload_node_exporter
start_grafana_service
start_prometheus_service
# start_node_exporter_service
configure_grafana
configure_cron
}
main

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,41 @@
#!/bin/bash -x
RESOURCE_DIR=/usr/local/sysom/monitor
NODE_EXPORTER_VER=1.2.2
NODE_EXPORTER_ARCH=linux-amd64
NODE_EXPORTER_PKG=node_exporter-${NODE_EXPORTER_VER}.${NODE_EXPORTER_ARCH}
NODE_EXPORTER_TAR=$NODE_EXPORTER_PKG.tar.gz
##设置node_exporter开机自动启动
cat << EOF > node_exporter.service
[Unit]
Description=SysOM Monitor Promethues
Documentation=SysOM Monitor Promethues
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=${RESOURCE_DIR}/node_exporter/node_exporter
[Install]
WantedBy=multi-user.target
EOF
main()
{
tar -zxvf $NODE_EXPORTER_TAR
rm -rf $RESOURCE_DIR/node_exporter
mv $NODE_EXPORTER_PKG $RESOURCE_DIR/node_exporter
mv node_exporter.service /usr/lib/systemd/system
systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter
ps -elf | grep "/usr/local/sysom/monitor/node_exporter/node_exporter" | grep -v grep 1>/dev/null
if [ $? -ne 0 ]
then
exit 1
fi
exit 0
}
main

View File

@ -0,0 +1,23 @@
#!/usr/bin/python3
import requests
import json
fname="/usr/local/sysom/monitor/prometheus/node/node.json"
host_api="http://localhost/api/v1/host"
hostlist = requests.get(host_api)
res = hostlist.content
hosts = json.loads(res)
host_len = len(hosts["data"]["results"])
iplist=[]
fo = open(fname,"w")
for i in range(0,host_len):
iplist.append(hosts["data"]["results"][i]["ip"]+":9100")
target={"targets":iplist}
res="["+json.dumps(target)+"]"
print(res)
fo.write(res)
fo.close()

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,8 @@
# -*- encoding: utf-8 -*-
"""
@File : __init__.py.py
@Time : 2021/11/22 10:36
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,14 @@
import logging
from django.apps import AppConfig
logger = logging.getLogger(__name__)
class AccountsConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.accounts'
def ready(self) -> None:
logger.info(">>> Accounts module loading success")

View File

@ -0,0 +1,70 @@
# -*- encoding: utf-8 -*-
"""
@File : authentication.py
@Time : 2021/10/29 11:04
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
import logging
import jwt
from django.utils.translation import ugettext as _
from rest_framework_jwt.authentication import BaseAuthentication
from rest_framework_jwt.settings import api_settings
from rest_framework.exceptions import AuthenticationFailed
from rest_framework import serializers
from rest_framework.request import Request
from apps.accounts.models import User
logger = logging.getLogger(__name__)
jwt_decode_handler = api_settings.JWT_DECODE_HANDLER
jwt_get_username_from_payload = api_settings.JWT_PAYLOAD_GET_USERNAME_HANDLER
class Authentication(BaseAuthentication):
def authenticate(self, request: Request):
if '/doc/' in request.META.get('PATH_INFO'):
return None
token = request.META.get('HTTP_AUTHORIZATION')
if not token and '/user/' in request.META.get('PATH_INFO'):
if self._get_all_user_count():
raise AuthenticationFailed("请联系管理员添加账号")
else:
return None
if not token:
raise AuthenticationFailed("没有令牌")
payload = self._check_payload(token)
user = self._check_user(payload=payload)
logger.info(f"{user.username} 身份通过")
return user, token
@staticmethod
def _check_payload(token):
try:
payload = jwt_decode_handler(token)
except jwt.ExpiredSignature:
msg = _('令牌过期!.')
raise AuthenticationFailed(msg)
except jwt.DecodeError:
msg = _('令牌验证失败!.')
raise AuthenticationFailed(msg)
return payload
@staticmethod
def _check_user(payload) -> User:
username = jwt_get_username_from_payload(payload)
if not username:
msg = _('令牌验证失败!')
raise AuthenticationFailed(msg)
try:
user = User.objects.get(username=username)
except User.DoesNotExist:
msg = _("用户不存在!")
raise AuthenticationFailed(msg)
return user
@staticmethod
def _get_all_user_count():
return User.objects.all().count()

View File

@ -0,0 +1,55 @@
from django.db import models
from django.contrib.auth.hashers import make_password, check_password
from lib import BaseModel
class User(BaseModel):
username = models.CharField(max_length=128)
password = models.CharField(max_length=255)
is_admin = models.BooleanField(default=False)
is_agree = models.BooleanField(default=False)
description = models.TextField()
role = models.ManyToManyField(to='Role', verbose_name='关联角色', db_constraint=False)
@staticmethod
def make_password(plain_password: str) -> str:
return make_password(plain_password, hasher='pbkdf2_sha256')
def verify_password(self, plain_password: str) -> bool:
return check_password(plain_password, self.password)
class Meta:
db_table = 'sys_users'
def __str__(self):
return f"用户:{self.username}"
class Role(BaseModel):
role_name = models.CharField(max_length=128, unique=True, verbose_name="角色名称")
permissions = models.ManyToManyField(to='Permission', verbose_name="关联权限", db_constraint=False)
def __str__(self):
return self.role_name
class Meta:
db_table = 'sys_role'
class Permission(BaseModel):
REQUEST_METHOD_CHOICES = (
(0, "GET"),
(1, "POST"),
(2, "DELETE"),
(3, "PUT"),
(4, "PATCH"),
)
path = models.CharField(max_length=64, verbose_name="Api路径")
method = models.IntegerField(choices=REQUEST_METHOD_CHOICES, default=0, verbose_name="请求方式")
class Meta:
db_table = "sys_permission"
def __str__(self):
return f"API{self.path} - Method: {self.get_method_display()}"

View File

@ -0,0 +1,47 @@
# -*- encoding: utf-8 -*-
"""
@File : permissions.py
@Time : 2021/11/9 13:53
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
import logging
from rest_framework.permissions import BasePermission
from rest_framework.request import Request
from rest_framework.views import APIView
from apps.accounts.models import User
logger = logging.getLogger(__name__)
class CustomPermission(BasePermission):
def __init__(self, message=None) -> None:
super().__init__()
self.message = getattr(self.__class__, 'message', '无权限')
self.user: User = None
def init_permission(self, request: Request, view: APIView):
self.user = request.user
def has_permission(self, request: Request, view: APIView):
return True
def has_object_permission(self, request: Request, view: APIView, obj):
return True
class IsAdminPermission(BasePermission):
def has_permission(self, request, view):
return request.user.is_admin
class IsOperationPermission(CustomPermission):
def has_permission(self, request, view):
user = request.user
if user.user_role.role_name == "运维人员":
return True
else:
return False

View File

@ -0,0 +1,23 @@
[
{
"model": "accounts.Role",
"pk": 1,
"fields": {
"role_name": "管理员"
}
},
{
"model": "accounts.Role",
"pk": 2,
"fields": {
"role_name": "运维人员"
}
},
{
"model": "accounts.Role",
"pk": 3,
"fields": {
"role_name": "普通人员"
}
}
]

View File

@ -0,0 +1,139 @@
import logging
from rest_framework import serializers
from rest_framework_jwt.settings import api_settings
from . import models
logger = logging.getLogger(__name__)
jwt_payload_handler = api_settings.JWT_PAYLOAD_HANDLER
jwt_encode_handler = api_settings.JWT_ENCODE_HANDLER
class UserListSerializer(serializers.ModelSerializer):
role = serializers.SerializerMethodField()
class Meta:
model = models.User
exclude = ['password']
def get_role(self, instance: models.User):
roles = instance.role.all()
return RoleListSerializer(instance=roles, many=True).data
class AddUserSerializer(serializers.ModelSerializer):
username = serializers.CharField(error_messages={'required': "用户名必填"})
password = serializers.CharField(error_messages={'required': "密码名必填"}, required=True)
role = serializers.ListField(required=False, write_only=True)
class Meta:
model = models.User
# exclude = ('description', )
fields = "__all__"
def validate_username(self, attr):
try:
models.User.objects.get(username=attr)
except models.User.DoesNotExist:
return attr
raise serializers.ValidationError("用户已存在!")
def validate_password(self, attr):
return models.User.make_password(attr)
def validate_role(self, attr):
return models.Role.objects.filter(id__in=attr)
def save(self, **kwargs):
count = models.User.objects.all().count()
validated_data = {**self.validated_data, **kwargs}
if self.instance is not None:
is_admin = validated_data.get('is_admin', None)
if is_admin:
roles = models.Role.objects.filter(role_name='管理员')
validated_data.update({'role': roles})
self.instance: models.User = self.update(self.instance, validated_data)
assert self.instance is not None, (
'`update()` did not return an object instance.'
)
else:
instance: models.User = self.create(validated_data)
if count == 0:
role = models.Role.objects.filter(role_name='管理员')
instance.role.add(*role)
instance.is_admin = True
else:
is_admin = validated_data.get("is_admin", None)
if is_admin:
role = models.Role.objects.filter(role_name='管理员')
else:
role = models.Role.objects.filter(role_name='普通人员')
instance.role.add(*role)
instance.save()
return self.instance
class UserAuthSerializer(serializers.ModelSerializer):
username = serializers.CharField(error_messages={'required': "用户名必填"})
password = serializers.CharField(error_messages={'required': "密码名必填"})
class Meta:
model = models.User
fields = ("username", "password")
def validate(self, attrs):
user = models.User.objects.get(username=attrs['username'])
if not user.verify_password(attrs['password']):
raise serializers.ValidationError("用户密码不正确!")
return attrs
def validate_username(self, attr):
try:
models.User.objects.get(username=attr)
except models.User.DoesNotExist:
raise serializers.ValidationError(f"用户名: {attr} 不存在!")
return attr
def create_token(self):
user = models.User.objects.get(username=self.data.get('username'))
payload = jwt_payload_handler(user)
token = jwt_encode_handler(payload)
return user, token
class RoleListSerializer(serializers.ModelSerializer):
permissions = serializers.SerializerMethodField()
class Meta:
model = models.Role
exclude = ['deleted_at']
def get_permissions(self, instance: models.Role):
permissions = instance.permissions.all()
return PermissionListSerializer(instance=permissions, many=True).data
class AddRoleSerializer(serializers.ModelSerializer):
permissions = serializers.ListField(required=False, write_only=True)
class Meta:
model = models.Role
fields = "__all__"
class PermissionListSerializer(serializers.ModelSerializer):
method = serializers.SerializerMethodField()
class Meta:
model = models.Permission
exclude = ("deleted_at", )
def get_method(self, instance: models.Permission):
return instance.get_method_display()
class AddPermissionSerializer(serializers.ModelSerializer):
class Meta:
model = models.Permission
fields = "__all__"

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,16 @@
from django.urls import include, path
from rest_framework.routers import DefaultRouter
from . import views
router = DefaultRouter()
router.register('user', views.UserModelViewSet)
router.register('role', views.RoleModelViewSet)
router.register('permission', views.PermissionViewSet)
urlpatterns = [
path('api/v1/', include(router.urls)),
path('api/v1/auth/', views.AuthAPIView.as_view()),
path('api/v1/change_password/', views.ChangePasswordViewSet.as_view())
]

View File

@ -0,0 +1,16 @@
[
{
"model": "accounts.user",
"pk": 1,
"fields": {
"created_at": "2021-12-07 17:38:28",
"deleted_at": null,
"username": "admin",
"password": "pbkdf2_sha256$260000$l7J8CLGVx5TOroHqIrm8Ml$7xshVn1wHfCAeYYNZKHBo7B8bGfklfcKqmhpun6ARSk=",
"is_admin": true,
"is_agree": true,
"description": "",
"role": [1]
}
}
]

View File

@ -0,0 +1,196 @@
import logging
from drf_yasg.utils import swagger_auto_schema
from rest_framework.request import Request
from rest_framework.viewsets import GenericViewSet
from rest_framework import mixins, status
from rest_framework.views import APIView
from apps.accounts.permissions import IsAdminPermission, IsOperationPermission
from apps.accounts.serializer import UserAuthSerializer
from apps.accounts.authentication import Authentication
from . import models
from . import serializer
from lib import success, other_response
logger = logging.getLogger(__name__)
class UserModelViewSet(
GenericViewSet,
mixins.CreateModelMixin,
mixins.ListModelMixin,
mixins.UpdateModelMixin,
mixins.DestroyModelMixin,
mixins.RetrieveModelMixin
):
queryset = models.User.objects.all()
serializer_class = serializer.UserListSerializer
authentication_classes = [Authentication]
# permission_classes = [IsAdminPermission]
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.UserListSerializer
else:
return serializer.AddUserSerializer
def get_permissions(self):
if self.request.method == "GET":
return []
else:
return [permission() for permission in self.permission_classes]
def get_authenticators(self):
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def create(self, request, *args, **kwargs):
response = super().create(request, *args, **kwargs)
return success(result=response.data, message="创建成功")
def list(self, request, *args, **kwargs):
data = super().list(request, *args, **kwargs)
return success(result=data.data)
def update(self, request, *args, **kwargs):
partial = kwargs.pop('partial', False)
instance = self.get_object()
u_serializer = self.get_serializer(instance, data=request.data, partial=partial)
u_serializer.is_valid(raise_exception=True)
self.perform_update(u_serializer)
result = serializer.UserListSerializer(instance=u_serializer.instance, many=False)
return success(result=result.data, message="修改成功")
def destroy(self, request, *args, **kwargs):
super().destroy(request, *args, **kwargs)
return success(result={}, message="删除成功")
def retrieve(self, request, *args, **kwargs):
result = super().retrieve(request, *args, **kwargs)
return success(result=result.data, message="获取成功")
class AuthAPIView(APIView):
authentication_classes = []
@swagger_auto_schema(method='POST')
def post(self, request):
ser = UserAuthSerializer(data=request.data)
ser.is_valid(raise_exception=True)
u, t = ser.create_token()
u_ser = serializer.UserListSerializer(instance=u, many=False)
result = u_ser.data
result.update({"token": t})
return other_response(message="登录成功", code=200, result=result)
class RoleModelViewSet(GenericViewSet,
mixins.CreateModelMixin,
mixins.ListModelMixin,
mixins.UpdateModelMixin,
mixins.RetrieveModelMixin):
queryset = models.Role.objects.all()
serializer_class = serializer.RoleListSerializer
authentication_classes = [Authentication]
permission_classes = [IsAdminPermission]
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.RoleListSerializer
else:
return serializer.AddRoleSerializer
def get_permissions(self):
if self.request.method == "GET":
return []
else:
return [permission() for permission in self.permission_classes]
def get_authenticators(self):
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def retrieve(self, request, *args, **kwargs):
result = super().retrieve(request, *args, **kwargs)
return success(result=result.data)
def create(self, request, *args, **kwargs):
create_serializer = self.get_serializer(data=request.data)
create_serializer.is_valid(raise_exception=True)
self.perform_create(create_serializer)
ser = serializer.RoleListSerializer(instance=create_serializer.instance, many=False)
return other_response(result=ser.data, code=status.HTTP_201_CREATED)
def update(self, request, *args, **kwargs):
partial = kwargs.pop('partial', False)
instance = self.get_object()
u_serializer = self.get_serializer(instance, data=request.data, partial=partial)
u_serializer.is_valid(raise_exception=True)
self.perform_update(u_serializer)
result = serializer.RoleListSerializer(instance=u_serializer.instance, many=False)
return success(message="修改成功", result=result.data)
class PermissionViewSet(GenericViewSet,
mixins.CreateModelMixin,
mixins.ListModelMixin,
mixins.UpdateModelMixin,
):
queryset = models.Permission.objects.all()
serializer_class = serializer.PermissionListSerializer
authentication_classes = [Authentication]
permission_classes = [IsAdminPermission]
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.PermissionListSerializer
else:
return serializer.AddPermissionSerializer
def get_authenticators(self):
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def get_permissions(self):
if self.request.method == "GET":
return []
else:
return [permission() for permission in self.permission_classes]
def create(self, request, *args, **kwargs):
create_serializer = self.get_serializer(data=request.data)
create_serializer.is_valid(raise_exception=True)
self.perform_create(create_serializer)
ser = serializer.PermissionListSerializer(instance=create_serializer.instance, many=False)
return success(result=ser.data)
class ChangePasswordViewSet(APIView):
"""Change User Password"""
required_fields = ['row_password', 'new_password', 'new_password_again']
def post(self, request: Request):
user = request.user
data = request.data
for f in filter(lambda x: not x[1], [(field, data.get(field, None)) for field in self.required_fields]):
return other_response(message=f'{f[0]} 不能为空', code=400, result={})
if data.get('new_password') != data.get('new_password_again'):
return other_response(message="两次密码不一致", code=400, result={})
if not user.verify_password(data.get('row_password')):
return other_response(message="原始密码错误", code=400, result={})
try:
user.password = user.make_password(data.get('new_password'))
user.save()
except Exception as e:
logger.error(e)
return other_response(message=f"数据库错误!{e}", code=400)
return success(result={}, message="密码修成成功")

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,13 @@
import logging
from django.apps import AppConfig
logger = logging.getLogger(__name__)
class HostConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.host'
def ready(self):
logger.info(">>> Host module loading success")

View File

@ -0,0 +1,45 @@
from django.db import models
from lib import BaseModel
from apps.accounts.models import User
from lib.ssh import SSH
class HostModel(BaseModel):
HOST_STATUS_CHOICES = (
(0, 'running'),
(1, 'error'),
(2, 'offline')
)
hostname = models.CharField(max_length=100, unique=True)
ip = models.CharField(max_length=100, unique=True)
port = models.IntegerField()
username = models.CharField(max_length=100)
private_key = models.TextField(null=True)
description = models.CharField(max_length=255, null=True)
status = models.IntegerField(choices=HOST_STATUS_CHOICES, default=2, verbose_name="主机状态")
client_deploy_cmd = models.TextField(verbose_name="client部署命令", default="")
h_type = models.ForeignKey('HostType', on_delete=models.CASCADE, related_name='hosts')
created_by = models.ForeignKey(User, on_delete=models.CASCADE, related_name="c_hosts")
deleted_by = models.ForeignKey(User, null=True, on_delete=models.CASCADE, related_name="d_hosts")
class Meta:
db_table = "sys_host"
def __str__(self):
return f'主机:{self.hostname}'
def get_host_client(self, pkey=None, default_env=None):
pkey = pkey or self.private_key
return SSH(hostname=self.ip, port=self.port, username=self.username, pkey=pkey, default_env=default_env)
class HostType(BaseModel):
type_name = models.CharField(max_length=128, unique=True)
class Meta:
db_table = "sys_host_type"
def __str__(self) -> str:
return f'主机类型: {self.type_name}'

View File

@ -0,0 +1,59 @@
import logging
from rest_framework import serializers
from apps.host.models import HostModel, HostType
logger = logging.getLogger(__name__)
class HostListSerializer(serializers.ModelSerializer):
host_type_id = serializers.SerializerMethodField()
description = serializers.SerializerMethodField()
class Meta:
model = HostModel
fields = [
'id', 'h_type', 'created_at', 'hostname', 'ip', 'port', 'username', 'description',
'host_type_id', 'client_deploy_cmd', 'status'
]
def get_host_type(self, attr: HostModel) -> dict:
ser = HostTypeListSerializer(instance=attr.h_type, many=False)
return ser.data
def get_host_type_id(self, attr: HostModel) -> int:
return attr.h_type.id
def get_description(self, attr: HostModel) -> str:
return attr.description or '暂未填写'
class AddHostSerializer(serializers.ModelSerializer):
hostname = serializers.CharField(error_messages={'required': '该字段必填'})
ip = serializers.CharField(error_messages={'required': '该字段必填'})
port = serializers.IntegerField(error_messages={'required': '该字段必填'})
username = serializers.CharField(error_messages={'required': '该字段必填'})
class Meta:
model = HostModel
fields = ('hostname', 'h_type', 'ip', 'port', 'username', 'private_key', 'description', 'client_deploy_cmd')
class HostTypeListSerializer(serializers.ModelSerializer):
class Meta:
model = HostType
exclude = ('deleted_at',)
class AddHostTypeSerializer(serializers.ModelSerializer):
type_name = serializers.CharField(error_messages={'required': "该字段必填"})
class Meta:
model = HostType
exclude = ('deleted_at',)
def validate_type_name(self, attr):
try:
HostType.objects.get(type_name=attr)
except HostType.DoesNotExist:
return attr
raise serializers.ValidationError("该服务已存在!")

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,15 @@
from django.urls import path
from django.urls.conf import include
from rest_framework.routers import DefaultRouter
from apps.host import views
router = DefaultRouter()
router.register('host', views.HostModelViewSet)
router.register('cluster', views.HostTypeViewSet)
urlpatterns = [
path('api/v1/host/upload_file/', views.SaveUploadFile.as_view()),
path('api/v1/', include(router.urls)),
]

View File

@ -0,0 +1,200 @@
import logging
import os
import uuid
from django.core.files.uploadedfile import InMemoryUploadedFile
from drf_yasg import openapi
from drf_yasg.utils import swagger_auto_schema
from rest_framework.views import APIView
from rest_framework.viewsets import GenericViewSet
from rest_framework import mixins
from django.db.models import Q
from django_filters.rest_framework import DjangoFilterBackend
from django.conf import settings
from apps.host import serializer
from apps.host.models import HostModel, HostType
from apps.accounts.authentication import Authentication
from consumer.executors import SshJob
from apps.task.models import JobModel
from lib import *
from lib.exception import APIException
logger = logging.getLogger(__name__)
class HostModelViewSet(GenericViewSet,
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.CreateModelMixin,
mixins.DestroyModelMixin
):
queryset = HostModel.objects.filter(deleted_at=None)
serializer_class = serializer.HostListSerializer
authentication_classes = [Authentication]
filter_backends = [DjangoFilterBackend]
filterset_fields = ['ip', 'hostname', 'h_type', 'status']
def get_authenticators(self):
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def create(self, request, *args, **kwargs):
password = request.data.pop('host_password', None)
if not password:
return not_found(message=f'host_password required!')
b, k = generate_private_key(
hostname=request.data['ip'], port=request.data['port'], username=request.data['username'], password=password
)
if not b:
return other_response(code=400, message=f"主机验证失败:{k}")
request.data.update({'private_key': k})
create_serializer = self.get_serializer(data=request.data)
create_serializer.is_valid(raise_exception=True)
self.perform_create(create_serializer)
instance = create_serializer.instance
# 检查输入client部署命令 更新host状态
self._client_deploy_cmd_check(instance)
host_list_serializer = serializer.HostListSerializer(instance=instance)
return success(result=host_list_serializer.data)
def perform_create(self, ser):
client_deploy_cmd = settings.CLIENT_DEPLOY_CMD
ser.save(created_by=self.request.user,
client_deploy_cmd=client_deploy_cmd)
def retrieve(self, request, *args, **kwargs):
instance = self.check_instance_exist(request, *args, **kwargs)
if not instance:
return not_found()
ser = self.get_serializer(instance)
return success(result=ser.data)
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.HostListSerializer
else:
return serializer.AddHostSerializer
def destroy(self, request, *args, **kwargs):
instance = self.check_instance_exist(request, *args, **kwargs)
if not instance:
return not_found()
super().destroy(request, *args, **kwargs)
return success(message="删除成功", code=200, result={})
def update(self, request, *args, **kwargs):
response = super().update(request, *args, **kwargs)
return success(result=response.data, message="修改成功")
def check_instance_exist(self, request, *args, **kwargs):
instance = self.get_queryset().filter(**kwargs).first()
return instance if instance else None
@staticmethod
def _client_deploy_cmd_check(instance: HostModel):
task_id = uuid_8()
job_instance = JobModel.objects.create(
command=instance.client_deploy_cmd,
task_id=task_id,
created_by=instance.created_by
)
commands = [
{"instance": instance.ip,
"cmd": instance.client_deploy_cmd
}]
job = SshJob(
commands,
job_instance,
update_host_status=True
)
sche = BackgroundScheduler()
sche.add_job(job.run, )
sche.start()
class HostTypeViewSet(GenericViewSet,
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.DestroyModelMixin,
mixins.CreateModelMixin,
mixins.UpdateModelMixin):
queryset = HostType.objects.filter(Q(deleted_at=None) | Q(deleted_at=""))
serializer_class = serializer.HostTypeListSerializer
def get_authenticators(self):
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.HostTypeListSerializer
else:
return serializer.AddHostTypeSerializer
def retrieve(self, request, *args, **kwargs):
response = super().retrieve(request, *args, **kwargs)
return success(result=response.data)
def create(self, request, *args, **kwargs):
super().create(request, *args, **kwargs)
return success(result={}, message="新增成功")
def update(self, request, *args, **kwargs):
super().update(request, *args, **kwargs)
return success(result={}, message="修改成功")
class SaveUploadFile(APIView):
authentication_classes = []
@swagger_auto_schema(operation_description="上传文件",
request_body=openapi.Schema(
type=openapi.TYPE_OBJECT,
required=["file"],
properties={
"file": openapi.Schema(type=openapi.TYPE_FILE),
"catalogue": openapi.Schema(type=openapi.TYPE_STRING)
}
),
responses={
'200': openapi.Response('save upload success', examples={"application/json": {
"code": 200,
"message": "Upload success",
"data": {}
}}),
'400': openapi.Response('Fail', examples={"application/json": {
"code": 400,
"message": "Required Field: file",
"data": {}
}})
}
)
def post(self, request):
file = request.data.get('file', None)
catalogue = request.data.get('catalogue', None)
if not file:
return APIException(message="Upload Failed: file required!")
file_path = os.path.join(
settings.MEDIA_ROOT, catalogue) if catalogue else settings.MEDIA_ROOT
if not os.path.exists(file_path):
os.mkdir(file_path)
path = os.path.join(file_path, file.name)
try:
with open(path, 'wb') as f:
for chunk in file.chunks():
f.write(chunk)
except Exception as e:
logger.error(e)
raise APIException(message=f"Upload Failed: {e}")
return success(result={}, message="Upload success")

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,13 @@
import logging
from django.apps import AppConfig
logger = logging.getLogger(__name__)
class JobConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.job'
def ready(self):
logger.info(">>> Job module loading success")

View File

@ -0,0 +1,29 @@
from django.db import models
from lib import BaseModel, uuid_8
from apps.accounts.models import User
from apps.host.models import HostModel
from django.contrib.auth.models import AbstractUser
class JobModel(BaseModel):
JOB_STATUS_CHOICES = (
('Ready', 'Ready'),
('Running', 'Running'),
('Success', 'Success'),
('Fail', 'Fail'),
)
instance_id = models.CharField(max_length=64, default=uuid_8, verbose_name="任务实例ID")
status = models.CharField(max_length=32, choices=JOB_STATUS_CHOICES, default='Ready', verbose_name="任务状态")
command = models.TextField(verbose_name="shell文本")
job_result = models.TextField(default="", verbose_name="shell结果")
src_instance = models.CharField(max_length=128, default="")
dest_instance = models.CharField(max_length=128, default="")
host_by = models.ForeignKey(to=HostModel, on_delete=models.CASCADE, related_name="host_jobs")
created_by = models.ForeignKey(to=User, on_delete=models.CASCADE, related_name="user_jobs")
def __str__(self):
return f"Job: {self.instance_id}"
class Meta:
db_table = 'sys_job'

View File

@ -0,0 +1,37 @@
# -*- encoding: utf-8 -*-
"""
@File : seriaizer.py
@Time : 2021/11/22 17:41
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from rest_framework import serializers
from apps.job.models import JobModel
class JobListSerializer(serializers.ModelSerializer):
job_result = serializers.SerializerMethodField()
src_instance = serializers.SerializerMethodField()
dest_instance = serializers.SerializerMethodField()
class Meta:
model = JobModel
exclude = ('deleted_at', )
def get_job_result(self, attr: JobModel):
return attr.job_result or '暂无'
def get_src_instance(self, attr: JobModel):
return attr.src_instance or '暂无'
def get_dest_instance(self, attr: JobModel):
return attr.dest_instance or '暂无'
class JobSerializer(serializers.ModelSerializer):
class Meta:
model = JobModel
fields = ('instance_id', 'command', 'status', 'job_result')

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,19 @@
# -*- encoding: utf-8 -*-
"""
@File : urls.py
@Time : 2021/11/22 10:38
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from django.urls import path, include
from rest_framework.routers import DefaultRouter
from apps.job import views
router = DefaultRouter()
router.register('jobs', views.JobAPIView)
urlpatterns = [
path('api/v1/', include(router.urls)),
]

View File

@ -0,0 +1,25 @@
from rest_framework.viewsets import GenericViewSet
from rest_framework import mixins
from apps.accounts.authentication import Authentication
from apps.job import models, seriaizer
from lib import success
class JobAPIView(GenericViewSet, mixins.ListModelMixin, mixins.RetrieveModelMixin):
queryset = models.JobModel.objects.all()
serializer_class = seriaizer.JobListSerializer
authentication_classes = [Authentication]
def get_queryset(self):
queryset = self.queryset
queryset = queryset.filter(created_by=self.request.user)
queryset = queryset.filter(deleted_at=None)
return queryset
def get_object(self):
pass
def retrieve(self, request, *args, **kwargs):
response = super().retrieve(request, *args, **kwargs)
return success(result=response.data)

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,6 @@
from django.apps import AppConfig
class MonitorConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.monitor'

View File

@ -0,0 +1,3 @@
from django.db import models
# Create your models here.

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,16 @@
# -*- encoding: utf-8 -*-
"""
@File : urls.py
@Time : 2021/11/2 11:07
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from django.urls import path, re_path
from apps.monitor import views
urlpatterns = [
path('api/v1/svg_info/', views.SvgInfoAPIView.as_view()),
re_path(r'^api/v1/download/svg/(?P<svg_name>.*?)$', views.DownloadSvgView.as_view())
]

View File

@ -0,0 +1,67 @@
import logging
import os
from django.conf import settings
from django.http import HttpResponse
from django.views.decorators.clickjacking import xframe_options_exempt
from rest_framework.views import APIView
from apps.accounts.authentication import Authentication
from lib import *
from lib.exception import FileNotFoundException
logger = logging.getLogger(__name__)
DOWNLOAD_SVG_URL = "/api/v1/download/svg/"
def _get_server_svg_list():
"""获取指定路径下的svg文件"""
return os.listdir(settings.SERVICE_SVG_PATH)
class SvgInfoAPIView(APIView):
authentication_classes = [Authentication]
def get_authenticators(self):
# return [auth() for auth in self.authentication_classes]
return []
def get(self, *args, **kwargs):
response = dict()
svg_list = _get_server_svg_list()
response["count"] = len(svg_list)
name_list = [svg_name.split('.')[:-1] for svg_name in svg_list]
master_key_list = list(set([name[0] for name in name_list]))
svg_info_list = list()
for key in master_key_list:
item = dict()
item['name'] = key
item['child'] = [
{'child_name': svg[1], 'download_url': '%s%s' % (DOWNLOAD_SVG_URL, '.'.join(svg)+'.svg')}
for svg in name_list if key in svg
]
svg_info_list.append(item)
response['data'] = svg_info_list
return success(result=response)
class DownloadSvgView(APIView):
authentication_classes = []
@xframe_options_exempt
def get(self, *args, **kwargs):
name = self.get_svg_name(*args, **kwargs)
file_path = os.path.join(settings.SERVICE_SVG_PATH, name)
with open(file_path, 'r') as f:
content = f.read()
response = HttpResponse(content=content, content_type="text/xml")
return response
def get_svg_name(self, *args, **kwargs):
name = kwargs.get('svg_name')
if name not in _get_server_svg_list():
raise FileNotFoundException()
return name

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,13 @@
import logging
from django.apps import AppConfig
logger = logging.getLogger(__name__)
class JobConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.task'
def ready(self):
logger.info(">>> Job module loading success")

View File

@ -0,0 +1,34 @@
from django.db import models
from lib import BaseModel
from apps.accounts.models import User
from apps.host.models import HostModel
class JobModel(BaseModel):
JOB_STATUS_CHOICES = (
('Ready', 'Ready'),
('Running', 'Running'),
('Success', 'Success'),
('Fail', 'Fail'),
)
task_id = models.CharField(max_length=64, default="", verbose_name="任务实例ID")
status = models.CharField(max_length=32, choices=JOB_STATUS_CHOICES, default='Ready', verbose_name="任务状态")
command = models.TextField(verbose_name="shell文本")
job_result = models.TextField(default="", verbose_name="shell结果")
host_by = models.TextField(max_length=64, default="", verbose_name="host_jobs")
created_by = models.ForeignKey(to=User, on_delete=models.CASCADE, related_name="user_jobs")
def __str__(self):
return f"Job: {self.task_id}"
class Meta:
db_table = 'sys_job'
class ServiceModel(BaseModel):
service_name = models.CharField(max_length=64, verbose_name="service名称")
service_script = models.CharField(max_length=64, verbose_name="脚本名称")
class Meta:
db_table = 'service'

View File

@ -0,0 +1,28 @@
# -*- encoding: utf-8 -*-
"""
@File : seriaizer.py
@Time : 2021/11/22 17:41
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from rest_framework import serializers
from apps.task.models import JobModel, ServiceModel
class JobListSerializer(serializers.ModelSerializer):
job_result = serializers.SerializerMethodField()
class Meta:
model = JobModel
exclude = ('deleted_at', )
def get_job_result(self, attr: JobModel):
return attr.job_result or '暂无'
class ServiceSerializer(serializers.ModelSerializer):
class Meta:
model = ServiceModel
exclude = ('deleted_at', )

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,21 @@
# -*- encoding: utf-8 -*-
"""
@File : urls.py
@Time : 2021/11/22 10:38
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from django.urls import path, include
from rest_framework.routers import DefaultRouter
from apps.task import views
router = DefaultRouter()
router.register('jobs', views.JobAPIView)
router.register('tasks', views.TaskAPIView)
router.register('service', views.ServiceAPIView)
urlpatterns = [
path('api/v1/', include(router.urls)),
]

View File

@ -0,0 +1,165 @@
import os
import uuid
import ast
from django_filters.rest_framework import DjangoFilterBackend
from rest_framework.viewsets import GenericViewSet
from rest_framework import mixins
from django.conf import settings
from apps.accounts.authentication import Authentication
from apps.task import seriaizer
from apps.task.models import JobModel, ServiceModel
from apps.host.models import HostModel
from apps.accounts.models import User
from consumer.executors import SshJob
from lib import *
class JobAPIView(GenericViewSet, mixins.ListModelMixin, mixins.RetrieveModelMixin):
queryset = JobModel.objects.all()
serializer_class = seriaizer.JobListSerializer
authentication_classes = []
def get_queryset(self):
queryset = self.queryset
queryset = queryset.filter(created_by=self.request.user)
queryset = queryset.filter(deleted_at=None)
return queryset
def get_object(self):
pass
def retrieve(self, request, *args, **kwargs):
response = super().retrieve(request, *args, **kwargs)
return success(result=response.data)
class TaskAPIView(GenericViewSet,
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.DestroyModelMixin,
mixins.CreateModelMixin
):
queryset = JobModel.objects.filter(deleted_at=None)
serializer_class = seriaizer.JobListSerializer
authentication_classes = []
def create(self, request, *args, **kwargs):
try:
data = request.data
service_name = data.get("service_name", None)
task_id = uuid_8()
if service_name:
sercive = ServiceModel.objects.filter(service_name=service_name).first()
SCRIPTS_DIR = settings.SCRIPTS_DIR
service_path = os.path.join(SCRIPTS_DIR, sercive.service_script)
data["taskid"] = task_id
command = "/bin/bash %s %s" % (service_path, json.dumps(data))
output = os.popen(command)
resp = ast.literal_eval(output.read())
resp_scripts = resp.get("commands")
username = "admin"
user = User.objects.filter(username=username).first()
job_id = str(uuid.uuid4())
self.ssh_job(resp_scripts, task_id, user, job_id)
return success(result={"instance_id": task_id})
else:
return self.default_ssh_job(data, task_id)
except Exception as e:
logger.error(e)
return other_response(message=str(e), code=400)
def retrieve(self, request, *args, **kwargs):
response = super().retrieve(request, *args, **kwargs)
return success(result=response.data)
def default_ssh_job(self, data, task_id):
try:
host_ids = data.get("host_ids")
commands = data.get("command")
if not host_ids:
return other_response(message="请选择执行主机", code=400)
user = User.objects.filter(username='admin').first()
job_id = str(uuid.uuid4())
cmds = []
for i in range(len(host_ids)):
instance = {}
host = HostModel.objects.filter(pk=host_ids[i]).first()
instance["instance"] = host.ip
instance["cmd"] = commands[i]
cmds.append(instance)
self.ssh_job(cmds, task_id, user, job_id)
return success(result={"instance_id": task_id})
except Exception as e:
logger.error(e)
return other_response(message=str(e), code=400)
def ssh_job(self, resp_scripts, task_id, user, job_id):
job_model = JobModel.objects.create(command=resp_scripts, task_id=task_id,
created_by=user)
sch_job = SshJob(resp_scripts, job_model)
scheduler.add_job(sch_job.run, id=job_id, )
def destroy(self, request, *args, **kwargs):
instance = self.get_queryset().filter(**kwargs).first()
if not instance:
return not_found()
self.perform_destroy(instance)
return success(message="删除成功", code=200, result={})
def perform_destroy(self, instance: JobModel):
instance.deleted_at = human_datetime()
instance.deleted_by = self.request.user
instance.save()
def update(self, request, *args, **kwargs):
response = super().update(request, *args, **kwargs)
return success(result=response.data, message="修改成功")
def partial_update(self, request, *args, **kwargs):
response = super().partial_update(request, *args, **kwargs)
return success(result=response.data, message="修改成功")
class ServiceAPIView(GenericViewSet,
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.DestroyModelMixin,
mixins.CreateModelMixin,
):
queryset = ServiceModel.objects.all()
serializer_class = seriaizer.ServiceSerializer
authentication_classes = []
filter_backends = [DjangoFilterBackend]
filterset_fields = ["id", "service_name"]
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
return success(result={})
def destroy(self, request, *args, **kwargs):
instance = self.get_queryset().filter(**kwargs).first()
if not instance:
return not_found()
self.perform_destroy(instance)
return success(message="删除成功", code=200, result={})
def retrieve(self, request, *args, **kwargs):
instance = self.get_queryset().filter(**kwargs).first()
if not instance:
return not_found()
ser = self.get_serializer(instance)
return success(result=ser.data)
def update(self, request, *args, **kwargs):
response = super().update(request, *args, **kwargs)
return success(result=response.data, message="修改成功")
def partial_update(self, request, *args, **kwargs):
response = super().partial_update(request, *args, **kwargs)
return success(result=response.data.get("data"), message="修改成功")

View File

View File

@ -0,0 +1,3 @@
from django.contrib import admin
# Register your models here.

View File

@ -0,0 +1,6 @@
from django.apps import AppConfig
class VmcoreConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'apps.vmcore'

View File

@ -0,0 +1,45 @@
from django.db import models
from lib import BaseModel
# Create your models here.
class Panic(BaseModel):
name = models.CharField(max_length=128,unique=True)
ip = models.CharField(max_length=64,null=True)
hostname = models.CharField(max_length=128)
vertype = models.IntegerField()
status = models.IntegerField()
core_time = models.DateTimeField()
vmcore_file = models.CharField(max_length=256)
dmesg_file = models.CharField(max_length=256)
dmesg = models.TextField()
title = models.CharField(max_length=128)
ver = models.CharField(max_length=128)
rip = models.CharField(max_length=64)
func_name = models.CharField(max_length=64)
comm = models.CharField(max_length=64)
calltrace = models.CharField(max_length=256)
crashkey = models.CharField(max_length=256)
modules = models.CharField(max_length=512)
upload_time = models.IntegerField()
issue_id = models.IntegerField()
panic_type = models.CharField(max_length=64)
panic_class = models.CharField(max_length=64)
class Meta:
db_table = "panic"
class Issue(BaseModel):
calltrace = models.CharField(max_length=256)
crashkey = models.CharField(max_length=256)
solution = models.TextField()
class Meta:
db_table = "issue"
class Calltrace(BaseModel):
name = models.CharField(max_length=128)
line = models.CharField(max_length=128)
idx = models.IntegerField()
vmcore = models.ForeignKey(to='Panic',on_delete=models.CASCADE, to_field='name', related_name='panic_call_trace',default='')
class Meta:
db_table = "call_trace"

View File

@ -0,0 +1,66 @@
import logging
from rest_framework import serializers
from rest_framework_jwt.settings import api_settings
from . import models
logger = logging.getLogger(__name__)
jwt_payload_handler = api_settings.JWT_PAYLOAD_HANDLER
jwt_encode_handler = api_settings.JWT_ENCODE_HANDLER
class PanicListSerializer(serializers.ModelSerializer):
vmcore_link = serializers.SerializerMethodField()
vmcore_check = serializers.SerializerMethodField()
issue_check = serializers.SerializerMethodField()
class Meta:
model = models.Panic
fields = ('id','name','hostname','ip','core_time','ver','vmcore_check','issue_id','vmcore_link','issue_check')
#fields = "__all__"
def get_vmcore_link(self, attr: models.Panic) -> str:
return 'http://127.0.0.1:8000/api/v1/vmcore_detail/?vmcore_name='+attr.name
def get_vmcore_check(self, attr: models.Panic) -> bool:
if attr.vmcore_file != '' and attr.vmcore_file != 'null':
return True
else:
return False
def get_issue_check(self, attr: models.Panic) -> bool:
if attr.issue_id != 0 and attr.issue_id != None:
return True
else:
return False
class PanicDetailSerializer(serializers.ModelSerializer):
class Meta:
model = models.Panic
fields = "__all__"
class AddPanicSerializer(serializers.ModelSerializer):
class Meta:
model = models.Panic
fields = "__all__"
class IssueSerializer(serializers.ModelSerializer):
#permissions = serializers.SerializerMethodField()
class Meta:
model = models.Issue
fields = "__all__"
class AddIssueSerializer(serializers.ModelSerializer):
#permissions = serializers.ListField(required=False, write_only=True)
class Meta:
model = models.Issue
fields = "__all__"

View File

@ -0,0 +1,3 @@
from django.test import TestCase
# Create your tests here.

View File

@ -0,0 +1,15 @@
from django.urls import path,re_path
from django.urls.conf import include
from rest_framework.routers import DefaultRouter
from apps.vmcore import views
router = DefaultRouter()
router.register('vmcore', views.VmcoreViewSet)
router.register('issue', views.IssueModelViewSet)
urlpatterns = [
path('api/v1/', include(router.urls)),
path('api/v1/vmcore_detail/', views.VmcoreDetail.as_view()),
]

View File

@ -0,0 +1,357 @@
import logging
from django.shortcuts import render
from drf_yasg.utils import swagger_auto_schema
from rest_framework.request import Request
from rest_framework.viewsets import GenericViewSet
from rest_framework import mixins
from rest_framework.views import APIView
from django_filters.rest_framework import DjangoFilterBackend
from apps.accounts.permissions import IsAdminPermission, IsOperationPermission
from apps.accounts.serializer import UserAuthSerializer
from apps.accounts.authentication import Authentication
from . import models
from . import serializer
from lib import success, other_response
import datetime
import re
logger = logging.getLogger(__name__)
# Create your views here.
class VmcoreViewSet(GenericViewSet,
mixins.ListModelMixin,
mixins.RetrieveModelMixin,
mixins.UpdateModelMixin,
mixins.CreateModelMixin,
mixins.DestroyModelMixin
):
queryset = models.Panic.objects.filter(deleted_at=None)
serializer_class = serializer.PanicListSerializer
authentication_classes = [Authentication]
filter_backends = [DjangoFilterBackend]
filterset_fields = ['hostname', 'calltrace', 'id', 'name', 'ver']
def get_queryset(self):
data = self.request.GET.dict()
start_time = data.get("startTime", None) or "2021-01-01"
end_time = data.get('endTime', None) or datetime.datetime.now()
return models.Panic.objects.filter(core_time__range=(start_time, end_time))
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.PanicListSerializer
else:
return serializer.AddPanicSerializer
def get_authenticators(self):
return []
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def create(self, request, *args, **kwargs):
data = request.data
if 'similar_dmesg' in data:
#TODO match similar vmcore
dmesg = data.get('similar_dmesg')
data = self.parse_calltrace(dmesg)
return other_response(result=data)
response = super().create(request, *args, **kwargs)
data = response.data
vmcore_id = data['id']
vmcore_calltrace = data['calltrace']
vmcores = models.Panic.objects.filter(calltrace=vmcore_calltrace).exclude(id=vmcore_id)
for vmcore in vmcores:
if vmcore.issue_id != 0:
insert_vmcore = models.Panic.objects.filter(id=vmcore_id)
insert_vmcore = insert_vmcore[0]
insert_vmcore.issue_id = vmcore.issue_id
insert_vmcore.save()
break
return success(result=response.data, message="插入成功")
def list(self, request, *args, **kwargs):
data_get = self.request.GET.dict()
if 'vmcore_id' in data_get and 'similar' in data_get and data_get.get('similar') == '1':
vmcore_id = data_get.get('vmcore_id')
vmcore = models.Panic.objects.filter(id=vmcore_id)
if len(vmcore) == 0:
return other_response(message="没有制定的vmcore", code=400, success=False)
vmcore = vmcore[0]
data = {}
data['calltrace_1'] = []
data['calltrace_2'] = []
data['calltrace_3'] = []
if vmcore.calltrace == '' or vmcore.calltrace == None:
return other_response(result=data)
calltrace_3 = vmcore.calltrace
calltracelist = calltrace_3.split('$')
if len(calltracelist) < 3:
calltrace_3 = ''
if len(calltracelist) >= 2:
calltrace_2 = '$'.join(calltracelist[0:2])
else:
calltrace_2 = ''
calltrace_1 = calltracelist[0]
if calltrace_3 != '':
vmcores = models.Panic.objects.filter(calltrace=calltrace_3).exclude(id=vmcore.id)
for i in vmcores:
data['calltrace_3'].append(serializer.PanicListSerializer(i).data)
if calltrace_2 != '':
vmcores = models.Panic.objects.filter(calltrace__startswith=calltrace_2).exclude(calltrace=calltrace_3)
for i in vmcores:
data['calltrace_2'].append(serializer.PanicListSerializer(i).data)
if calltrace_1 != '':
vmcores = models.Panic.objects.filter(calltrace__startswith=calltrace_1).exclude(calltrace__startswith=calltrace_2).exclude(calltrace=calltrace_3)
for i in vmcores:
data['calltrace_1'].append(serializer.PanicListSerializer(i).data)
return other_response(result=data)
else:
data = super().list(request, *args, **kwargs).data
#if len(data_get) == 0:
if 'hostname' not in data_get and 'start_time' not in data_get and 'end_time' not in data_get and 'vmcore_id' not in data_get and 'similar' not in data_get and 'similar_dmesg' not in data_get :
end_time=datetime.date.today() + datetime.timedelta(days=1)
start_time=end_time + datetime.timedelta(days=-30)
host_sum = models.Panic.objects.values('hostname').distinct().count()
vmcores_sum_30 = models.Panic.objects.filter(core_time__range=(start_time,end_time)).count()
hosts_sum_30 = models.Panic.objects.filter(core_time__range=(start_time,end_time)).count()
start_time=end_time + datetime.timedelta(days=-7)
vmcores_sum_7 = models.Panic.objects.filter(core_time__range=(start_time,end_time)).values('hostname').distinct().count()
hosts_sum_7 = models.Panic.objects.filter(core_time__range=(start_time,end_time)).values('hostname').distinct().count()
rate_30 = hosts_sum_30 / host_sum
rate7 = hosts_sum_7 /host_sum
data['vmcore_30days'] = vmcores_sum_30
data['vmcore_7days'] = vmcores_sum_7
data['rate_30days'] = hosts_sum_30/host_sum
data['rate_7days'] = hosts_sum_7/host_sum
return success(result=data['data'], total=data['total'], success=True, vmcore_30days=vmcores_sum_30, vmcore_7days=vmcores_sum_7, rate_30days=hosts_sum_30/host_sum, rate_7days=hosts_sum_7/host_sum)
return success(result=data['data'], success=True)
def update(self, request, *args, **kwargs):
partial = kwargs.pop('partial', False)
instance = self.get_object()
u_serializer = self.get_serializer(instance, data=request.data, partial=partial)
u_serializer.is_valid(raise_exception=True)
self.perform_update(u_serializer)
result = serializer.PanicListSerializer(instance=u_serializer.instance, many=False)
return success(result=result.data, message="修改成功")
def destroy(self, request, *args, **kwargs):
super().destroy(request, *args, **kwargs)
return success(result={}, message="删除成功")
def match_idx(self, list2, fstr):
i = 0
for x in list2:
if x == fstr:
return i
i = i + 1
return -1
def parse_calltrace(self,dmesg):
if len(dmesg) <= 0:
return []
result=[]
list2=[]
line_pattern1 = re.compile(r'.+[0-9]+\].+\[.*\][? ]* (\S+)\+0x')
line_pattern2 = re.compile(r'.+[0-9]+\][? ]*(\S+)\+0x')
line_pattern3 = re.compile(r'.*#[0-9]+ \[[0-9a-f]+\] (\S+) at')
lines = dmesg.split('\n')
if len(lines) == 1:
lines = dmesg.splitlines()
for r in lines:
if r.find("Call Trace:") > 0 or r.find("exception RIP:") > 0:
del list2[:]
m = line_pattern1.match(r)
if m:
list2.append(m.group(1))
continue
m = line_pattern2.match(r)
if m:
list2.append(m.group(1))
continue
m = line_pattern3.match(r)
if m:
list2.append(m.group(1))
continue
s2=len(list2)
while s2 > 0:
vmcore_names = models.Calltrace.objects.filter(line__in=list2).order_by('name','idx')
last_name = None
last_idx=0
fidx=-1
fcnt=0
fpow=1
for calltrace_set in vmcore_names:
if last_name == None or last_name.name!=calltrace_set.name:
if fcnt > 1:
data = serializer.PanicListSerializer(last_name.vmcore).data
data['rate'] = (fcnt*100/s2)+fpow
result.append(data)
#result[last_name]=(fcnt*100/s2)+fpow
last_idx=0
fidx=-1
fidx=self.match_idx(list2, calltrace_set.line)
fcnt=0
fpow=0
if fidx >= 0:
fcnt=1
if fidx == 0 and calltrace_set.idx <= 2:
fpow=100
elif (last_idx>=calltrace_set.idx-2 and last_idx<=calltrace_set.idx-1) and fidx>=0 and fidx<=s2-1:
if list2[fidx+1]==calltrace_set.line:
fidx=fidx+1
fcnt=fcnt+1
last_name=calltrace_set
last_idx=calltrace_set.idx
if fcnt > 1:
data = serializer.PanicListSerializer(calltrace_set.vmcore).data
data['rate'] = (fcnt*100/s2)+fpow
result.append(data)
if len(result) > 0:
break
list2=list2[1:]
s2=len(list2)
l = sorted(result, key=lambda x:x['rate'], reverse=True)
s = len(l)
if s >= 20:
s = 20
l = l[0:s]
return l
class IssueModelViewSet(GenericViewSet,
mixins.CreateModelMixin,
mixins.ListModelMixin,
mixins.UpdateModelMixin,
mixins.RetrieveModelMixin):
queryset = models.Issue.objects.filter(deleted_at=None)
serializer_class = serializer.IssueSerializer
authentication_classes = [Authentication]
permission_classes = [IsAdminPermission]
filter_backends = [DjangoFilterBackend]
filterset_fields = ['id']
def get_queryset(self):
data = self.request.GET.dict()
if 'issue_id' in data:
return models.Issue.objects.filter(id=data['issue_id'])
elif 'vmcore_id' in data:
vmcore = models.Panic.objects.filter(id=data['vmcore_id'])
if len(vmcore) == 0:
return models.Issue.objects.filter(deleted_at=None)
issue_id = vmcore[0].issue_id
return models.Issue.objects.filter(id=issue_id)
else:
return models.Issue.objects.filter(deleted_at=None)
def get_serializer_class(self):
if self.request.method == "GET":
return serializer.IssueSerializer
else:
return serializer.AddIssueSerializer
def get_permissions(self):
return []
if self.request.method == "GET":
return []
else:
return [permission() for permission in self.permission_classes]
def get_authenticators(self):
return []
if self.request.method == "GET":
return []
else:
return [auth() for auth in self.authentication_classes]
def retrieve(self, request, *args, **kwargs):
result = super().retrieve(request, *args, **kwargs)
return success(result=result.data)
def create(self, request, *args, **kwargs):
vmcore_id = request.data.get('vmcore_id')
issue = {}
issue['solution'] = request.data.get('solution')
vmcore = models.Panic.objects.filter(id=vmcore_id)
if len(vmcore) == 0:
return other_response(message="没有指明vmcore无法提交", code=400)
vmcore = vmcore[0]
if vmcore.issue_id == 0:
issue['calltrace'] = vmcore.calltrace
issue['crashkey'] = vmcore.crashkey
create_serializer = self.get_serializer(data=issue)
create_serializer.is_valid(raise_exception=True)
self.perform_create(create_serializer)
else:
issue_old = models.Issue.objects.filter(id=vmcore.issue_id)
issue_old = issue_old[0]
issue_old.solution = issue['solution']
issue_old.save()
create_serializer = self.get_serializer(issue_old)
instance = create_serializer.instance
vmcore.issue_id = instance.id
vmcore.status = 4
vmcore.save()
vmcores = models.Panic.objects.filter(calltrace=vmcore.calltrace)
if vmcores:
for vmcore in vmcores:
vmcore.issue_id = instance.id
vmcore.status = 4
vmcore.save()
ser = serializer.IssueSerializer(instance=create_serializer.instance, many=False)
return other_response(result=ser.data)
def update(self, request, *args, **kwargs):
partial = kwargs.pop('partial', False)
instance = self.get_object()
u_serializer = self.get_serializer(instance, data=request.data, partial=partial)
u_serializer.is_valid(raise_exception=True)
self.perform_update(u_serializer)
result = serializer.IssueSerializer(instance=u_serializer.instance, many=False)
return success(message="修改成功", result=result.data)
class VmcoreDetail(APIView):
authentication_classes = []
def get(self, request):
try:
data = request.GET.dict()
if 'vmcore_name' not in data and 'vmcore_id' not in data:
return other_response(message='vmcore_name or vmcore_id 不能为空', code=400, result={})
if 'vmcore_name' in data:
vmcore_name = data.get('vmcore_name')
vmcore = models.Panic.objects.filter(name=vmcore_name)
else:
vmcore_id = data.get('vmcore_id')
vmcore = models.Panic.objects.filter(id=vmcore_id)
if len(vmcore) == 0:
return other_response(code=200)
vmcores_data = serializer.PanicDetailSerializer(vmcore[0]).data
return success(result=vmcores_data)
except Exception as e:
logger.error(e)
return other_response(message=str(e), code=400)

File diff suppressed because one or more lines are too long

141
sysom_api/conf/__init__.py Normal file
View File

@ -0,0 +1,141 @@
# -*- encoding: utf-8 -*-
"""
@File : __init__.py.py
@Time : 2021/11/3 11:21
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
import os
import socket
from pathlib import Path
def get_ip_address():
"""ip address"""
hostname = socket.gethostname()
return socket.gethostbyname(hostname)
class BaseConstant:
SERVER_IP = get_ip_address()
BASE_DIR = Path(__file__).resolve().parent.parent
DEBUG = True
DEFAULT_CACHE_REDIS = "redis://127.0.0.1:6379/1"
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'sysom', # Or path to database file if using sqlite3.
'USER': 'sysom', # Not used with sqlite3.
'PASSWORD': 'sysom_admin',
'HOST': '127.0.0.1',
'PORT': '3306',
},
}
LANGUAGE_CODE = 'zh-hans'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_L10N = True
USE_TZ = True
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'index/status')
CLIENT_DEPLOY_CMD = 'ls /root'
# 日志
SERVER_LOGS_FILE = os.path.join(BASE_DIR, 'logs', 'sys_om_info.log')
ERROR_LOGS_FILE = os.path.join(BASE_DIR, 'logs', 'sys_om_error.log')
if not os.path.exists(os.path.join(BASE_DIR, 'logs')):
os.makedirs(os.path.join(BASE_DIR, 'logs'))
# 格式:[2020-04-22 23:33:01][micoservice.apps.ready():16] [INFO] 这是一条日志:
# 格式:[日期][模块.函数名称():行号] [级别] 信息
STANDARD_LOG_FORMAT = '[%(levelname).4s] -- %(asctime)s -- P_%(process) -- d_T_%(thread)d ' \
'- <%(module)s:%(lineno)d>: %(message)s'
CONSOLE_LOG_FORMAT = '[%(levelname).4s] -- %(asctime)s -- P_%(process) -- d_T_%(thread)d ' \
'- <%(module)s:%(lineno)d>: %(message)s'
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': STANDARD_LOG_FORMAT
},
'console': {
'format': CONSOLE_LOG_FORMAT,
'datefmt': '%Y-%m-%d %H:%M:%S',
},
'file': {
'format': CONSOLE_LOG_FORMAT,
'datefmt': '%Y-%m-%d %H:%M:%S',
},
},
'handlers': {
'file': {
'level': 'INFO',
'class': 'logging.handlers.RotatingFileHandler',
'filename': SERVER_LOGS_FILE,
'maxBytes': 1024 * 1024 * 100, # 100 MB
'backupCount': 5, # 最多备份5个
'formatter': 'standard',
'encoding': 'utf-8',
},
'error': {
'level': 'ERROR',
'class': 'logging.handlers.RotatingFileHandler',
'filename': ERROR_LOGS_FILE,
'maxBytes': 1024 * 1024 * 100, # 100 MB
'backupCount': 3, # 最多备份3个
'formatter': 'standard',
'encoding': 'utf-8',
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'console',
}
},
'loggers': {
# default日志
'': {
'handlers': ['console', 'error', 'file'],
'level': 'INFO',
},
'django': {
'handlers': ['console', 'error', 'file'],
'level': 'INFO',
},
'scripts': {
'handlers': ['console', 'error', 'file'],
'level': 'INFO',
},
# 数据库相关日志
'django.db.backends': {
'handlers': [],
'propagate': True,
'level': 'INFO',
},
}
}
env = os.environ.get("env", "testing")
Constant = None
if env == "develop":
from .dev_constant import DevConstant
Constant = DevConstant
print(f"加载了环境 >> {env}")
elif env == "testing":
from .test_constant import TestConstant
Constant = TestConstant
print(f"加载了环境 >> {env}")
elif env == "produce":
from .pro_constant import ProConstant
Constant = ProConstant
print(f"加载了环境 >> {env}")
__all__ = ['Constant']

View File

@ -0,0 +1,23 @@
# -*- encoding: utf-8 -*-
"""
@File : dev_constant.py
@Time : 2021/11/3 14:18
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from . import BaseConstant
class DevConstant(BaseConstant):
DEBUG = True
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'sysom', # Or path to database file if using sqlite3.
'USER': 'sysom', # Not used with sqlite3.
'PASSWORD': 'sysom_admin',
'HOST': '127.0.0.1',
'PORT': '3306',
},
}

View File

@ -0,0 +1,13 @@
# -*- encoding: utf-8 -*-
"""
@File : pro_constant.py
@Time : 2021/11/3 14:17
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from . import BaseConstant
class ProConstant(BaseConstant):
DEBUG = False

View File

@ -0,0 +1,27 @@
# -*- encoding: utf-8 -*-
"""
@File : test_constant.py
@Time : 2021/11/3 14:22
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from . import BaseConstant
class TestConstant(BaseConstant):
DEBUG = True
SERVICE_SVG_PATH = '/Users/fengfuqiu/Documents/project/sysom/netinfo'
SERVICE_URL = 'http://11.238.158.138:8000/'
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'TIMEOUT': 2,
'NAME': 'sysom_test_db', # Or path to database file if using sqlite3.
'USER': 'sysom', # Not used with sqlite3.
'PASSWORD': 'sysomtest',
'HOST': 'rm-tattest-osdh.mysql.rdstest.tbsite.net',
# Set to empty string for localhost. Not used with sqlite3.
'PORT': '3306',
},
}

View File

@ -0,0 +1,2 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-

View File

@ -0,0 +1,103 @@
import json
import logging
from threading import Thread
from channels.generic.websocket import WebsocketConsumer
from channels.exceptions import StopConsumer
from django.contrib.auth.models import AnonymousUser
from django_redis import get_redis_connection
from apps.host.models import HostModel
logger = logging.getLogger(__name__)
def get_host_instance(model, **kwargs):
"""async orm"""
return model.objects.filter(**kwargs).first()
class ExecConsumer(WebsocketConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.rds = get_redis_connection()
def connect(self):
self.accept()
def disconnect(self, code):
self.send(text_data="disconnect")
self.rds.close()
self.close()
def get_response(self):
token = self.scope['url_route']['kwargs']['token']
response = self.rds.brpop(token, timeout=5) # 移除并获取最后一个元素
return response[1] if response else None
def receive(self, **kwargs):
response = self.get_response()
while response:
data = response.decode()
self.send(text_data=data)
response = self.get_response()
self.send(text_data='pong')
class SshConsumer(WebsocketConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.user = None
self.host_id = None
self.ssh = None
self.xterm = None
def connect(self):
self.user = self.scope['user']
self.host_id = self.scope['url_route']['kwargs']['id']
if isinstance(self.user, AnonymousUser):
self.close()
else:
self.accept()
self._connect_host_init()
def _connect_host_init(self):
"""初始化host连接"""
instance = get_host_instance(model=HostModel, pk=self.host_id, created_by=self.user.id)
if not instance:
self.send(bytes_data=b'Not Found host / No Permission\r\n')
self.close()
self.host: HostModel = instance
self.send(bytes_data=b'Connecting ...\r\n')
try:
self.ssh = self.host.get_host_client().get_client()
except Exception as e:
self.send(bytes_data=f'Exception: {e}\r\n'.encode())
self.close()
return
self.xterm = self.ssh.invoke_shell(term='xterm')
self.xterm.transport.set_keepalive(30)
Thread(target=self.loop_read).start()
def loop_read(self):
while True:
data = self.xterm.recv(32 * 1024)
if not data:
self.close()
break
self.send(bytes_data=data)
def receive(self, text_data=None, bytes_data=None):
data = text_data or bytes_data
if data:
data = json.loads(data)
resize = data.get('resize')
if resize and len(resize) == 2:
self.xterm.resize_pty(*resize)
else:
self.xterm.send(data['data'])
def websocket_disconnect(self, message):
raise StopConsumer()

View File

@ -0,0 +1,48 @@
import socket
from apps.host.models import HostModel
from lib.ssh import SSH
from apps.task.models import JobModel
class SshJob:
def __init__(self, resp_scripts, job, **kwargs):
self.resp_scripts = resp_scripts
self.job = job
self.kwargs = kwargs
def run(self):
try:
update_job(instance=self.job, status="Running")
host_ips = []
for script in self.resp_scripts:
ip = script.get("instance", None)
cmd = script.get("cmd", None)
if not ip or not cmd:
update_job(instance=self.job, status="Fail", job_result="script result find not instance or cmd")
break
host_ips.append(ip)
host = HostModel.objects.filter(ip=ip).first()
ssh_cli = SSH(host.ip, host.port, host.username, host.private_key)
with ssh_cli as ssh:
status, result = ssh.exec_command(cmd)
if str(status) != '0':
update_job(instance=self.job, status="Fail", job_result=result, host_by=host_ips)
break
if self.resp_scripts.index(script) == len(self.resp_scripts) - 1:
update_job(instance=self.job, status="Success", job_result=result, host_by=host_ips)
if self.kwargs.get('update_host_status', None):
host.status = status if status == 0 else 1
host.save()
except socket.timeout:
update_job(instance=self.job, status="Fail", job_result="socket time out")
except Exception as e:
update_job(instance=self.job, status="Fail", job_result=str(e))
def update_job(instance: JobModel, **kwargs):
try:
instance.__dict__.update(**kwargs)
instance.save()
return instance
except Exception as e:
raise e

View File

@ -0,0 +1,44 @@
import logging
from channels.db import database_sync_to_async
from django.contrib.auth.models import AnonymousUser
from apps.accounts.models import User
from rest_framework_jwt.settings import api_settings
logger = logging.getLogger(__name__)
jwt_decode_handler = api_settings.JWT_DECODE_HANDLER
jwt_get_username_from_payload = api_settings.JWT_PAYLOAD_GET_USERNAME_HANDLER
@database_sync_to_async
def get_user(user_id: int):
try:
user = User.objects.get(id=user_id)
except User.DoesNotExist:
return AnonymousUser()
return user
class AuthMiddleware:
def __init__(self, application):
self.application = application
async def __call__(self, scope, receive, send, *args, **kwargs):
if "headers" not in scope:
raise ValueError(
"CookieMiddleware was passed a scope that did not have a headers key "
+ "(make sure it is only passed HTTP or WebSocket connections)"
)
user_id = None
token = scope.get('query_string', None)
if not token:
scope['user'] = AnonymousUser()
if token.decode().startswith('user_id='):
user_id = token.decode().replace('user_id=', '')
if not user_id:
scope['user'] = AnonymousUser()
else:
scope['user'] = await get_user(user_id=user_id)
return await self.application(scope, receive, send, *args, **kwargs)

View File

@ -0,0 +1,11 @@
from django.urls import path
from channels.routing import URLRouter
from .consumers import *
from .middleware import AuthMiddleware
ws_router = AuthMiddleware(
URLRouter([
path('ws/exec/<str:token>/', ExecConsumer.as_asgi()),
path('ws/ssh/<int:id>/', SshConsumer.as_asgi()),
])
)

5
sysom_api/dockerfile Normal file
View File

@ -0,0 +1,5 @@
FROM python:3.7.5
WORKDIR /home
RUN pip install django==3.2.3
RUN pip install uwsgi
EXPOSE 8080

12
sysom_api/lib/__init__.py Normal file
View File

@ -0,0 +1,12 @@
# -*- encoding: utf-8 -*-
"""
@File : __init__.py.py
@Time : 2021/10/28 11:04
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from .utils import *
from .base_model import *
from .response import *

View File

@ -0,0 +1,20 @@
# -*- encoding: utf-8 -*-
"""
@File : base_model.py
@Time : 2021/10/28 11:04
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
from django.db import models
from lib import human_datetime
class BaseModel(models.Model):
"""abstract model"""
created_at = models.CharField(max_length=20, default=human_datetime, verbose_name="创建时间")
deleted_at = models.CharField(max_length=20, null=True)
class Meta:
abstract = True

View File

@ -0,0 +1,63 @@
import logging
import traceback
from django.db.models import ProtectedError
from rest_framework.views import set_rollback
from rest_framework import exceptions, status
from rest_framework.exceptions import APIException as DRFAPIException, AuthenticationFailed, NotAuthenticated
from .response import ErrorResponse
logger = logging.getLogger(__name__)
class APIException(Exception):
def __init__(self, code=400, message='API异常', args=('API异常',)):
self.code = code
self.message = message
self.args = args
def __str__(self):
return self.message
class FileNotFoundException(Exception):
def __init__(self, code=404, message='文件不存在'):
self.code = code
self.message = message
def __str__(self):
return self.message
def exception_handler(exc, context):
"""自定义异常处理"""
msg = ''
code = 400
if isinstance(exc, FileNotFoundException):
code = exc.code
msg = exc.message
if isinstance(exc, AuthenticationFailed):
code = 403
msg = exc.detail
elif isinstance(exc, NotAuthenticated):
code = 402
msg = exc.detail
elif isinstance(exc, DRFAPIException):
set_rollback()
# print(exc.detail)
# msg = {str(e) for e in exc.detail}
msg = exc.detail
elif isinstance(exc, exceptions.APIException):
set_rollback()
msg = exc.detail
elif isinstance(exc, ProtectedError):
set_rollback()
msg = "删除失败:该条数据与其他数据有相关绑定"
elif isinstance(exc, Exception):
logger.error(traceback.format_exc())
msg = str(exc) # 原样输出错误
return ErrorResponse(msg=msg, code=code, status=code)

View File

@ -0,0 +1,27 @@
'''
@File: paginations.py
@Time: 2021-12-14 13:46:02
@Author: DM
@Email: wb-msm261421@alibaba-inc.com
@Desc: Local Paginations Class
'''
from rest_framework.pagination import PageNumberPagination
from lib.response import success
class Pagination(PageNumberPagination):
page_query_param = "current"
page_size_query_param = "pageSize"
def paginate_queryset(self, queryset, request, view=None):
self.max_page_size = queryset.count()
return super().paginate_queryset(queryset, request, view=view)
def get_paginated_response(self, data):
return success(message="获取成功", result=data, total=self.page.paginator.count)
def get_page_size(self, request):
if not request.query_params.get(self.page_size_query_param, None):
return self.max_page_size
return super().get_page_size(request)

68
sysom_api/lib/response.py Normal file
View File

@ -0,0 +1,68 @@
from rest_framework.response import Response
from rest_framework import status
from django.http import FileResponse
def _response(data=None, status=None):
return Response(data=data, status=status)
def success(result, message="success", success=True, code=status.HTTP_200_OK, **kwargs):
data = {
"code": code,
"message": message,
"data": result,
"success": success
}
data.update(kwargs)
return _response(data=data, status=code)
def not_found(code=status.HTTP_404_NOT_FOUND, success=False, message="Not Found"):
data = {
"code": code,
"message": message,
"success": success,
}
return _response(data=data, status=code)
def not_permission(code=status.HTTP_403_FORBIDDEN, success=False, message="Not Permission"):
data = {
"code": code,
"success": success,
"message": message
}
return _response(data=data, status=code)
def other_response(result=dict(), message="", success=True, code=status.HTTP_200_OK, **kwargs):
data = {
"code": code,
"message": message,
"data": result,
"success": success
}
data.update(kwargs)
return _response(data=data, status=code)
class ErrorResponse(Response):
"""
标准响应错误的返回,ErrorResponse(msg='xxx')
默认错误码返回400, 也可以指定其他返回码:ErrorResponse(code=xxx)
"""
def __init__(self, data=None, msg='error', code=400, status=None, template_name=None, headers=None,
exception=False, content_type=None):
std_data = {
"code": code,
"data": data or {},
"message": msg
}
super().__init__(std_data, status, template_name, headers, exception, content_type)
class FileResponseAlter(FileResponse):
pass

106
sysom_api/lib/ssh.py Normal file
View File

@ -0,0 +1,106 @@
import logging
from paramiko.client import SSHClient, AutoAddPolicy
from paramiko.rsakey import RSAKey
from io import StringIO
logger = logging.getLogger(__name__)
class SSH:
def __init__(
self,
hostname,
port=22,
username='root',
pkey=None,
password=None,
default_env=None,
connect_timeout=10
):
self.client = None
self.default_env = self.make_env_command(default_env)
self.arguments = {
'hostname': hostname,
'port': port,
'username': username,
'password': password,
'pkey': RSAKey.from_private_key(StringIO(pkey)) if isinstance(pkey, str) else pkey,
'timeout': connect_timeout,
'banner_timeout': 30
}
@staticmethod
def generate_key():
key_obj = StringIO()
key = RSAKey.generate(2048)
key.write_private_key(key_obj)
return key_obj.getvalue(), 'ssh-rsa ' + key.get_base64()
def get_client(self):
if self.client is not None:
return self.client
self.client = SSHClient()
self.client.set_missing_host_key_policy(AutoAddPolicy)
self.client.connect(**self.arguments)
return self.client
def ping(self):
return True
def add_public_key(self, public_key):
command = f'mkdir -p -m 700 ~/.ssh && \
echo {public_key!r} >> ~/.ssh/authorized_keys && \
chmod 600 ~/.ssh/authorized_keys'
exit_code, out = self.exec_command(command)
if exit_code != 0:
raise Exception(f'add public key error: {out}')
def exec_command(self, command, environment=None):
ssh_session = self.client.get_transport().open_session()
if environment:
ssh_session.update_environment(environment)
ssh_session.set_combine_stderr(True)
ssh_session.exec_command(command)
stdout = ssh_session.makefile("rb", -1)
return ssh_session.recv_exit_status(), self.decode(stdout.read())
def put_file(self, local_path, remote_path):
with self as cli:
sftp = cli.client.open_sftp()
sftp.put(local_path, remote_path)
def get_file(self, local_path, remote_path):
with self as cli:
sftp = cli.client.open_sftp()
sftp.get(remote_path, local_path)
def remove_file(self, path):
sftp = self.client.open_sftp()
sftp.remove(path)
def make_env_command(self, environment):
if not environment:
return None
str_envs = []
for k, v in environment.items():
k = k.replace('-', '_')
if isinstance(v, str):
v = v.replace("'", "'\"'\"'")
str_envs.append(f"{k}='{v}'")
str_envs = ' '.join(str_envs)
return f'export {str_envs}'
def decode(self, content):
try:
content = content.decode()
except UnicodeDecodeError:
content = content.decode(encoding='GBK', errors='ignore')
return content
def __enter__(self):
self.get_client()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.client.close()

119
sysom_api/lib/utils.py Normal file
View File

@ -0,0 +1,119 @@
# -*- encoding: utf-8 -*-
"""
@File : utils.py
@Time : 2021/10/28 11:09
@Author : DM
@Email : smmic@isoftstone.com
@Software: PyCharm
"""
import uuid as UUID
import json
import logging
from datetime import datetime, date as datetime_date
from decimal import Decimal
from apscheduler.schedulers.background import BackgroundScheduler
from paramiko import BadAuthenticationType, AuthenticationException
from rest_framework.pagination import PageNumberPagination
from lib.ssh import SSH
logger = logging.getLogger(__name__)
job_defaults = {'max_instances': 10}
scheduler = BackgroundScheduler(job_defaults=job_defaults)
scheduler.start()
CHAR_SET = ("a", "b", "c", "d", "e", "f",
"g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s",
"t", "u", "v", "w", "x", "y", "z", "0", "1", "2", "3", "4", "5",
"6", "7", "8", "9", "A", "B", "C", "D", "E", "F", "G", "H", "I",
"J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V",
"W", "X", "Y", "Z")
def human_datetime(date=None):
if date:
assert isinstance(date, datetime)
else:
date = datetime.now()
return date.strftime('%Y-%m-%d %H:%M:%S')
# 转换时间格式到字符串
def datetime_str(date=None):
return datetime.strptime(date, "%Y-%m-%d %H:%M:%S")
# 日期json序列化
class DateTimeEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return o.strftime('%Y-%m-%d %H:%M:%S')
elif isinstance(o, datetime_date):
return o.strftime('%Y-%m-%d')
elif isinstance(o, Decimal):
return float(o)
return json.JSONEncoder.default(self, o)
def get_request_real_ip(headers: dict):
x_real_ip = headers.get('x-forwarded-for')
if not x_real_ip:
x_real_ip = headers.get('x-real-ip', '')
return x_real_ip.split(',')[0]
def uuid_36():
"""
返回36字符的UUID字符串(十六进制,含有-) bc5debab-95c3-4430-933f-2e3b6407ac30
:return:
"""
return str(UUID.uuid4())
def uuid_32():
"""
返回32字符的UUID字符串(十六进制) bc5debab95c34430933f2e3b6407ac30
:return:
"""
return uuid_36().replace('-', '')
def uuid_8():
"""
返回8字符的UUID字符串(非进制) 3FNWjtlD
:return:
"""
s = uuid_32()
result = ''
for i in range(0, 8):
sub = s[i * 4: i * 4 + 4]
x = int(sub, 16)
result += CHAR_SET[x % 0x3E]
return result
def generate_private_key(hostname, port, username, password=None, pkey=None):
try:
if password:
private_key, public_key = SSH.generate_key()
with SSH(hostname, port, username, password=password) as ssh:
ssh.add_public_key(public_key)
with SSH(hostname, port, username, private_key) as ssh:
ssh.ping()
return True, private_key
if pkey:
SSH(hostname, port, username, pkey)
return True, pkey
except BadAuthenticationType:
return False, "认证类型暂不支持"
except AuthenticationException:
return False, "认证失败,请检查用户名密码或密钥"
except Exception as e:
return False, e

22
sysom_api/manage.py Executable file
View File

@ -0,0 +1,22 @@
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sysom.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()

View File

@ -0,0 +1,70 @@
aioredis==1.3.1
APScheduler==3.8.1
asgiref==3.4.1
async-timeout==3.0.1
attrs==21.2.0
autobahn==21.3.1
Automat==20.2.0
autopep8==1.6.0
bcrypt==3.2.0
cairocffi==1.3.0
CairoSVG==2.5.2
certifi==2021.10.8
cffi==1.15.0
channels==3.0.4
channels-redis==3.3.1
charset-normalizer==2.0.7
constantly==15.1.0
coreapi==2.3.3
coreschema==0.0.4
cryptography==35.0.0
cssselect2==0.4.1
daphne==3.0.2
defusedxml==0.7.1
Django==3.2.8
django-cors-headers==3.10.0
django-filter==21.1
django-redis==5.0.0
djangorestframework==3.12.2
djangorestframework-jwt==1.11.0
drf-yasg==1.20.0
hiredis==2.0.0
hyperlink==21.0.0
idna==3.3
incremental==21.3.0
inflection==0.5.1
install==1.3.4
itypes==1.2.0
Jinja2==3.0.2
MarkupSafe==2.0.1
msgpack==1.0.2
packaging==21.0
paramiko==2.8.0
Pillow==8.4.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycparser==2.20
PyJWT==1.7.1
PyMySQL==1.0.2
PyNaCl==1.4.0
pyOpenSSL==21.0.0
pyparsing==3.0.3
pytz==2021.3
redis==3.5.3
requests==2.26.0
ruamel.yaml==0.17.16
ruamel.yaml.clib==0.2.6
service-identity==21.1.0
six==1.16.0
sqlparse==0.4.2
tinycss2==1.1.0
toml==0.10.2
Twisted==21.7.0
txaio==21.2.1
typing-extensions==3.10.0.2
uritemplate==4.1.1
urllib3==1.26.7
uWSGI==2.0.19.1
webencodings==0.5.1
zope.interface==5.4.0

View File

@ -0,0 +1,15 @@
#!/bin/bash
echo "{
'commands':[
{
'instance': '192.168.0.2',
'cmd': 'ls /root'
},
{
'instance': '192.168.0.3',
'cmd': 'ls /root'
}
],
'taskid': 'xxxx'
}"

View File

@ -0,0 +1,3 @@
import pymysql
pymysql.install_as_MySQLdb()

18
sysom_api/sysom/asgi.py Normal file
View File

@ -0,0 +1,18 @@
"""
ASGI config for sysom project.
It exposes the ASGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/3.2/howto/deployment/asgi/
"""
import os
from channels.routing import ProtocolTypeRouter
from consumer import routing
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sysom.settings')
application = ProtocolTypeRouter({
"websocket": routing.ws_router,
})

171
sysom_api/sysom/settings.py Normal file
View File

@ -0,0 +1,171 @@
"""
Django settings for sysom project.
Generated by 'django-admin startproject' using Django 3.2.8.
For more information on this file, see
https://docs.djangoproject.com/en/3.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.2/ref/settings/
"""
import datetime
import os
from pathlib import Path
BASE_DIR = Path(__file__).resolve().parent.parent
SECRET_KEY = 'django-insecure-^d8b9di9w&-mmsbpt@)o#e+2^z+^m4nhf+z8304%9@8y#ko46l'
ALLOWED_HOSTS = ['*']
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.staticfiles',
'django.contrib.sessions',
'django.contrib.messages',
'apps.accounts',
'apps.host',
'apps.monitor',
'apps.task',
'apps.vmcore',
'consumer',
'rest_framework',
'corsheaders',
'drf_yasg', # 在线API文档
'channels',
'django_filters',
]
MIDDLEWARE = [
'corsheaders.middleware.CorsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
# 'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
from conf import Constant as constant
DEBUG = constant.DEBUG
# 跨域允许
if DEBUG:
CORS_ORIGIN_ALLOW_ALL = True
# Redis缓存
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": constant.DEFAULT_CACHE_REDIS,
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
}
}
# Mysql数据库
DATABASES = constant.DATABASES
ROOT_URLCONF = 'sysom.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'sysom.wsgi.application'
ASGI_APPLICATION = 'sysom.asgi.application'
# channels 配置
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("127.0.0.1", 6379)]
}
}
}
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
LANGUAGE_CODE = constant.LANGUAGE_CODE
TIME_ZONE = constant.TIME_ZONE
USE_I18N = constant.USE_I18N
USE_L10N = constant.USE_L10N
USE_TZ = constant.USE_TZ
STATIC_URL = constant.STATIC_URL
STATIC_ROOT = constant.STATIC_ROOT
# rest_framework settings
REST_FRAMEWORK = {
'DEFAULT_PERMISSION_CLASSES': (
# 'rest_framework.permissions.IsAuthenticated'
),
'DEFAULT_AUTHENTICATION_CLASSES': [
'apps.accounts.authentication.Authentication'
],
'DEFAULT_VERSIONING_CLASS': "rest_framework.versioning.URLPathVersioning",
'DEFAULT_VERSION': 'v1', # 默认版本
'ALLOWED_VERSIONS': ['v1', 'v2'], # 允许的版本
'VERSION_PARAM': 'version',
'DEFAULT_RENDERER_CLASSES': (
'rest_framework.renderers.JSONRenderer',
),
'DEFAULT_PAGINATION_CLASS': 'lib.paginations.Pagination',
'UNICODE_JSON': False,
'EXCEPTION_HANDLER': 'lib.exception.exception_handler'
}
JWT_AUTH = {
'JWT_EXPIRATION_DELTA': datetime.timedelta(days=1),
}
LOGGING = constant.LOGGING
SERVICE_SVG_PATH = os.path.join(BASE_DIR, 'netinfo')
# upload file
MEDIA_URL = '/uploads/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'uploads')
SCRIPTS_DIR = os.path.join(BASE_DIR, 'service_scripts')
SERVER_IP = constant.SERVER_IP
CLIENT_DEPLOY_CMD = constant.CLIENT_DEPLOY_CMD

51
sysom_api/sysom/urls.py Normal file
View File

@ -0,0 +1,51 @@
"""sysom URL Configuration
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/3.2/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path, re_path
from django.urls.conf import include
from django.conf import settings
urlpatterns = [
path('admin/', admin.site.urls),
path('', include("apps.host.urls")),
path('', include("apps.accounts.urls")),
path('', include("apps.monitor.urls")),
path('', include("apps.task.urls")),
path('', include("apps.vmcore.urls")),
]
if settings.DEBUG:
from drf_yasg.views import get_schema_view
from drf_yasg import openapi
from apps.accounts.authentication import Authentication
schema_view = get_schema_view(
openapi.Info(
title="SysOM后端API文档",
default_version='v1',
description="暂无..."
),
public=True,
# permission_classes=(permissions.AllowAny,),
authentication_classes=(Authentication, )
)
urlpatterns += [
path('doc/', schema_view.with_ui('swagger', cache_timeout=0), name='schema-swagger-ui'),
path('redoc/', schema_view.with_ui('redoc', cache_timeout=0), name='schema-redoc'),
]

16
sysom_api/sysom/wsgi.py Normal file
View File

@ -0,0 +1,16 @@
"""
WSGI config for sysom project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sysom.settings')
application = get_wsgi_application()

30
sysom_api/uwsgi.ini Normal file
View File

@ -0,0 +1,30 @@
[uwsgi]
# 配置和nginx链接的socket链接
# socket=0.0.0.0:8080
# 也可以使用http
http=0.0.0.0:8080
# 配置项目路径,项目的所在目录
chdir=/home
# 配置wsgi接口模块文件路劲
wsgi-file=/home/sysom/wsgi.py
# 配置启动的进程数
processes=4
# 配置每个进程的线程数
threads=20
# 配置启动管理主进程
master=True
# 配置存放主进程的进程号文件
pidfile=uwsgi.pid
# 配置dump日志记录
daemonize=/home/logs/uwsgi.log
# 配置静态资源
static-map=/static=/home/index/static

57
sysom_web/README.md Normal file
View File

@ -0,0 +1,57 @@
# Ant Design Pro
This project is initialized with [Ant Design Pro](https://pro.ant.design). Follow is the quick guide for how to use.
## Environment Prepare
Install `node_modules`:
```bash
npm install
```
or
```bash
yarn
```
## Provided Scripts
Ant Design Pro provides some useful script to help you quick start and build with web project, code style check and test.
Scripts provided in `package.json`. It's safe to modify or add additional script:
### Start project
```bash
npm start
```
### Build project
```bash
npm run build
```
### Check code style
```bash
npm run lint
```
You can also use script to auto fix some lint error:
```bash
npm run lint:fix
```
### Test code
```bash
npm test
```
## More
You can view full document on our [official website](https://pro.ant.design). And welcome any feedback in our [github](https://github.com/ant-design/ant-design-pro).

View File

@ -0,0 +1,14 @@
// https://umijs.org/config/
import { defineConfig } from 'umi';
export default defineConfig({
plugins: [
// https://github.com/zthxxx/react-dev-inspector
'react-dev-inspector/plugins/umi/react-inspector',
],
// https://github.com/zthxxx/react-dev-inspector#inspector-loader-props
inspectorConfig: {
exclude: [],
babelPlugins: [],
babelOptions: {},
},
});

View File

@ -0,0 +1,74 @@
// https://umijs.org/config/
import { defineConfig } from 'umi';
import { join } from 'path';
import defaultSettings from './defaultSettings';
import proxy from './proxy';
import routes from './routes';
const { REACT_APP_ENV } = process.env;
export default defineConfig({
hash: true,
antd: {
dark: true,
},
dva: {
hmr: true,
},
layout: {
// https://umijs.org/zh-CN/plugins/plugin-layout
locale: true,
siderWidth: 208,
...defaultSettings,
},
// https://umijs.org/zh-CN/plugins/plugin-locale
locale: {
// default zh-CN
default: 'zh-CN',
antd: true,
// default true, when it is true, will use `navigator.language` overwrite default
baseNavigator: true,
},
dynamicImport: {
loading: '@ant-design/pro-layout/es/PageLoading',
},
targets: {
ie: 11,
},
// umi routes: https://umijs.org/docs/routing
routes,
// Theme for antd: https://ant.design/docs/react/customize-theme-cn
theme: {
'primary-color': defaultSettings.primaryColor,
'root-entry-name': 'default',
},
// esbuild is father build tools
// https://umijs.org/plugins/plugin-esbuild
esbuild: {},
title: false,
ignoreMomentLocale: true,
proxy: proxy[REACT_APP_ENV || 'dev'],
manifest: {
basePath: '/',
},
// Fast Refresh 热更新
fastRefresh: {},
openAPI: [
{
requestLibPath: "import { request } from 'umi'",
// 或者使用在线的版本
// schemaPath: "https://gw.alipayobjects.com/os/antfincdn/M%24jrzTTYJN/oneapi.json"
schemaPath: join(__dirname, 'oneapi.json'),
mock: false,
},
{
requestLibPath: "import { request } from 'umi'",
schemaPath: 'https://gw.alipayobjects.com/os/antfincdn/CA1dOm%2631B/openapi.json',
projectName: 'swagger',
},
],
nodeModulesTransform: {
type: 'none',
},
mfsu: {},
webpack5: {},
exportStatic: {},
});

View File

@ -0,0 +1,11 @@
const Settings = {
navTheme: 'realDark',
primaryColor: '#1890ff',
layout: 'top',
contentWidth: 'Fluid',
fixedHeader: true,
fixSiderbar: true,
title: '系统运维平台',
logo: null,
};
export default Settings;

View File

@ -0,0 +1,593 @@
{
"openapi": "3.0.1",
"info": {
"title": "Ant Design Pro",
"version": "1.0.0"
},
"servers": [
{
"url": "http://localhost:8000/"
},
{
"url": "https://localhost:8000/"
}
],
"paths": {
"/api/currentUser": {
"get": {
"tags": ["api"],
"description": "获取当前的用户",
"operationId": "currentUser",
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CurrentUser"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"x-swagger-router-controller": "api"
},
"/api/login/captcha": {
"post": {
"description": "发送验证码",
"operationId": "getFakeCaptcha",
"tags": ["login"],
"parameters": [
{
"name": "phone",
"in": "query",
"description": "手机号",
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/FakeCaptcha"
}
}
}
}
}
}
},
"/api/login/outLogin": {
"post": {
"description": "登录接口",
"operationId": "outLogin",
"tags": ["login"],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"x-swagger-router-controller": "api"
},
"/api/login/account": {
"post": {
"tags": ["login"],
"description": "登录接口",
"operationId": "login",
"requestBody": {
"description": "登录系统",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoginParams"
}
}
},
"required": true
},
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoginResult"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"x-codegen-request-body-name": "body"
},
"x-swagger-router-controller": "api"
},
"/api/notices": {
"summary": "getNotices",
"description": "NoticeIconItem",
"get": {
"tags": ["api"],
"operationId": "getNotices",
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/NoticeIconList"
}
}
}
}
}
}
},
"/api/rule": {
"get": {
"tags": ["rule"],
"description": "获取规则列表",
"operationId": "rule",
"parameters": [
{
"name": "current",
"in": "query",
"description": "当前的页码",
"schema": {
"type": "number"
}
},
{
"name": "pageSize",
"in": "query",
"description": "页面的容量",
"schema": {
"type": "number"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RuleList"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"post": {
"tags": ["rule"],
"description": "新建规则",
"operationId": "addRule",
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RuleListItem"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"put": {
"tags": ["rule"],
"description": "新建规则",
"operationId": "updateRule",
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RuleListItem"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"delete": {
"tags": ["rule"],
"description": "删除规则",
"operationId": "removeRule",
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"description": "Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
"x-swagger-router-controller": "api"
},
"/swagger": {
"x-swagger-pipe": "swagger_raw"
}
},
"components": {
"schemas": {
"CurrentUser": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"avatar": {
"type": "string"
},
"userid": {
"type": "string"
},
"email": {
"type": "string"
},
"signature": {
"type": "string"
},
"title": {
"type": "string"
},
"group": {
"type": "string"
},
"tags": {
"type": "array",
"items": {
"type": "object",
"properties": {
"key": {
"type": "string"
},
"label": {
"type": "string"
}
}
}
},
"notifyCount": {
"type": "integer",
"format": "int32"
},
"unreadCount": {
"type": "integer",
"format": "int32"
},
"country": {
"type": "string"
},
"access": {
"type": "string"
},
"geographic": {
"type": "object",
"properties": {
"province": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"key": {
"type": "string"
}
}
},
"city": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"key": {
"type": "string"
}
}
}
}
},
"address": {
"type": "string"
},
"phone": {
"type": "string"
}
}
},
"LoginResult": {
"type": "object",
"properties": {
"status": {
"type": "string"
},
"type": {
"type": "string"
},
"currentAuthority": {
"type": "string"
}
}
},
"PageParams": {
"type": "object",
"properties": {
"current": {
"type": "number"
},
"pageSize": {
"type": "number"
}
}
},
"RuleListItem": {
"type": "object",
"properties": {
"key": {
"type": "integer",
"format": "int32"
},
"disabled": {
"type": "boolean"
},
"href": {
"type": "string"
},
"avatar": {
"type": "string"
},
"name": {
"type": "string"
},
"owner": {
"type": "string"
},
"desc": {
"type": "string"
},
"callNo": {
"type": "integer",
"format": "int32"
},
"status": {
"type": "integer",
"format": "int32"
},
"updatedAt": {
"type": "string",
"format": "datetime"
},
"createdAt": {
"type": "string",
"format": "datetime"
},
"progress": {
"type": "integer",
"format": "int32"
}
}
},
"RuleList": {
"type": "object",
"properties": {
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/RuleListItem"
}
},
"total": {
"type": "integer",
"description": "列表的内容总数",
"format": "int32"
},
"success": {
"type": "boolean"
}
}
},
"FakeCaptcha": {
"type": "object",
"properties": {
"code": {
"type": "integer",
"format": "int32"
},
"status": {
"type": "string"
}
}
},
"LoginParams": {
"type": "object",
"properties": {
"username": {
"type": "string"
},
"password": {
"type": "string"
},
"autoLogin": {
"type": "boolean"
},
"type": {
"type": "string"
}
}
},
"ErrorResponse": {
"required": ["errorCode"],
"type": "object",
"properties": {
"errorCode": {
"type": "string",
"description": "业务约定的错误码"
},
"errorMessage": {
"type": "string",
"description": "业务上的错误信息"
},
"success": {
"type": "boolean",
"description": "业务上的请求是否成功"
}
}
},
"NoticeIconList": {
"type": "object",
"properties": {
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/NoticeIconItem"
}
},
"total": {
"type": "integer",
"description": "列表的内容总数",
"format": "int32"
},
"success": {
"type": "boolean"
}
}
},
"NoticeIconItemType": {
"title": "NoticeIconItemType",
"description": "已读未读列表的枚举",
"type": "string",
"properties": {},
"enum": ["notification", "message", "event"]
},
"NoticeIconItem": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"extra": {
"type": "string",
"format": "any"
},
"key": { "type": "string" },
"read": {
"type": "boolean"
},
"avatar": {
"type": "string"
},
"title": {
"type": "string"
},
"status": {
"type": "string"
},
"datetime": {
"type": "string",
"format": "date"
},
"description": {
"type": "string"
},
"type": {
"extensions": {
"x-is-enum": true
},
"$ref": "#/components/schemas/NoticeIconItemType"
}
}
}
}
}
}

39
sysom_web/config/proxy.js Normal file
View File

@ -0,0 +1,39 @@
/**
* 在生产环境 代理是无法生效的所以这里没有生产环境的配置
* -------------------------------
* The agent cannot take effect in the production environment
* so there is no configuration of the production environment
* For details, please see
* https://pro.ant.design/docs/deploy
*/
export default {
dev: {
// localhost:8000/api/** -> https://preview.pro.ant.design/api/**
'/api/v1/': {
// 要代理的地址
//target: 'https://preview.pro.ant.design',
target: 'http://127.0.0.1:8001',
// 配置了这个可以从 http 代理到 https
// 依赖 origin 的功能可能需要这个,比如 cookie
changeOrigin: true,
},
},
test: {
'/api/': {
target: 'https://proapi.azurewebsites.net',
changeOrigin: true,
pathRewrite: {
'^': '',
},
},
},
pre: {
'/api/': {
target: 'your pre url',
changeOrigin: true,
pathRewrite: {
'^': '',
},
},
},
};

106
sysom_web/config/routes.js Normal file
View File

@ -0,0 +1,106 @@
export default [
{
path: '/user',
layout: false,
routes: [
{
path: '/user',
routes: [
{
name: 'login',
path: '/user/login',
component: './user/Login',
},
],
},
{
component: './404',
},
],
},
{
path: '/welcome',
name: 'welcome',
layout: false,
component: './Welcome',
},
{
path: '/host',
name: 'host',
component: './Host',
},
{
path: '/monitor',
name: 'monitor',
routes: [
{
path: '/monitor',
redirect: '/monitor/dashboard',
},
{
path: 'dashboard',
name: 'dashboard',
hideInBreadcrumb:true,
component: './monitor/SystemDashboard',
},
{
path: 'dashboard/:host?',
component: './monitor/SystemDashboard',
},
{
component: './404',
},
],
},
{
path: '/vmcore',
name: 'vmcore',
routes: [
{
path: '/vmcore',
redirect: '/vmcore/list',
},
{
path: '/vmcore/list',
name: 'list',
component: './vmcore/list',
},
{
path: '/vmcore/detail/:id?',
component: './vmcore/Detail',
},
{
path: '/vmcore/match',
name: 'match',
component: './vmcore/match',
},
],
},
{
path: '/diagnose',
name: 'diagnose',
routes: [
{
path: '/diagnose',
redirect: '/diagnose/io',
},
{
path: '/diagnose/io',
name: 'io',
component: './diagnose/io',
},
{
path: '/diagnose/net',
name: 'net',
component: './diagnose/net',
},
],
},
{
path: '/',
redirect: '/welcome',
},
{
component: './404',
},
];

1
sysom_web/dist/CNAME vendored Normal file
View File

@ -0,0 +1 @@
preview.pro.ant.design

Some files were not shown because too many files have changed in this diff Show More