linuxea:gitlab-ci/cd Hygieia dashboard简单配置(十一)


Hygieia
他的发音hi-gee-ya,在希腊语和罗马神话中她是健康的女神/化身
在Hygieia中分为五层,UI,API,DEVOPS TOOLS,数据收集,数据存储,其中:

  • UI层(用户界面) : 是Hygieia的前端,包含用户可以查看的所有图形用户界面(GUI)元素。用户也可以在此处配置仪表板。
  • API层 : Hygieia API层包含Hygieia API和Audit API。Hygieia API包含与源系统数据(由服务任务收集)和Internet一起使用的所有典型REST API服务。Hygieia审计API是API端点的集合,用于审计Hygieia收集器收集的CI / CD数据。该层是本地数据层和源系统数据层的抽象。
  • DEVOPS TOOLS层 :该层需要CI / CD管道中的大量DevOps工具。在图中,Jira,Git,Sonar和XLDeploy作为示例列出。
  • 数据收集层:收集器层从您的DevOps工具中获取数据。反过来,这些数据会显示在您的Hygieia仪表板上。您可以选择从Hygieia收集器清单中安装适用于您的DevOps工具集的收集器。
  • 数据存储层 :Hygieia使用MongoDB作为数据存储和检索的数据库
    架构概述如下图:

Hygieia可以描述CI/CD管道,从本质上讲Hygieia就是一个聚合器,它从团队的CI/CD管道中使用各种devops工具提取数据,使其在仪表板中更易于理解,坦白的说,便是将交付流水全过程反馈到可视化的界面中来
Hygieia的仪表板简化了近乎实时查看CICD管道的能力。仪表板使DevOps工程师和管理人员能够监控代码提交到最终生产中的部署情况。在这两点之间开始(提交)到完成(prod) - 仪表板还提供有关软件操作的整体活力和性能指标的重要信息。
在其中包括:组合视图,流水线视图,产品试图,云环境视图等,阅读参考:https://capitalone.github.io/Hygieia/getting_started.html
本章节是对gitlab和Hygieia,相比较jenkins,还是后者更全面
简单做了安装和配置实践,需要大量的开发套件才能完成,因此与gitlab-ci关系并不大,gitlab本身已经有视图,并不是如此好用而已,本篇仅提供思路,不具备任何参考价值

I. Hygieia与gitlab

安装mvn和node

install mvn

cd /usr/local && curl -Lk http://mirror.rise.ph/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz|tar xz -C ./ && ln -s  apache-maven-3.5.4 apache-maven && echo "export PATH=/usr/local/apache-maven/bin:\$PATH" >> /etc/profile && source /etc/profile && echo -e "\033[32m`mvn -version` \033[0m"

install node

cd /usr/local && curl -Lk https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz|tar xJ -C ./ && ln -s node-v8.11.3-linux-x64 node && echo "export PATH=/usr/local/node/bin:\$PATH" >> /etc/profile && source /etc/profile && echo -e "\033[32m`node -v` \033[0m"

Hygieia install

创建一个普通用户并且切换到普通用户,克隆代码并执行mvn clean install package

[root@LinuxEA-VM-Node202 ~]# useradd Hygieia
[root@LinuxEA-VM-Node202 ~]# su - Hygieia
[Hygieia@LinuxEA-VM-Node202 ~]$ git clone https://github.com/capitalone/Hygieia.git
正克隆到 'Hygieia'...
remote: Counting objects: 34942, done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 34942 (delta 8), reused 44 (delta 3), pack-reused 34849
接收对象中: 100% (34942/34942), 71.17 MiB | 231.00 KiB/s, done.
处理 delta 中: 100% (15509/15509), done.

执行mvn clean install package,这个过程可能会很长

[Hygieia@LinuxEA-VM-Node202 ~]$ cd Hygieia
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia]$ mvn clean install package
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/
省略一万字
[INFO] Reactor Summary:
[INFO] 
[INFO] com.capitalone.dashboard:Hygieia 2.0.5-SNAPSHOT .... SUCCESS [ 49.924 s]
[INFO] com.capitalone.dashboard:core ...................... SUCCESS [03:03 min]
[INFO] com.capitalone.dashboard:api ....................... SUCCESS [ 45.683 s]
[INFO] com.capitalone.dashboard:api-audit ................. SUCCESS [ 13.169 s]
[INFO] com.capitalone.dashboard:rally-collector ........... SUCCESS [  6.604 s]
[INFO] com.capitalone.dashboard:artifactory-artifact-collector SUCCESS [  4.012 s]
[INFO] com.capitalone.dashboard:bamboo-build-collector .... SUCCESS [  3.544 s]
[INFO] com.capitalone.dashboard:jenkins-build-collector ... SUCCESS [  3.855 s]
[INFO] com.capitalone.dashboard:jenkins-cucumber-test-collector SUCCESS [  2.881 s]
[INFO] com.capitalone.dashboard:jenkins-codequality ....... SUCCESS [  7.999 s]
[INFO] com.capitalone.dashboard:sonar-codequality-collector SUCCESS [  4.993 s]
[INFO] com.capitalone.dashboard:aws-cloud-collector ....... SUCCESS [ 36.819 s]
[INFO] com.capitalone.dashboard:udeploy-deployment-collector SUCCESS [  2.942 s]
[INFO] com.capitalone.dashboard:xldeploy-deployment-collector SUCCESS [  2.776 s]
[INFO] com.capitalone.dashboard:jira-feature-collector .... SUCCESS [ 58.628 s]
[INFO] com.capitalone.dashboard:versionone-feature-collector SUCCESS [ 11.287 s]
[INFO] com.capitalone.dashboard:gitlab-feature-collector .. SUCCESS [  3.761 s]
[INFO] com.capitalone.dashboard:chat-ops-collector ........ SUCCESS [  0.929 s]
[INFO] com.capitalone.dashboard:appdynamics-performance-collector SUCCESS [  2.725 s]
[INFO] com.capitalone.dashboard:bitbucket-scm-collector ... SUCCESS [  3.123 s]
[INFO] com.capitalone.dashboard:github-scm-collector ...... SUCCESS [  2.988 s]
[INFO] com.capitalone.dashboard:github-graphql-scm-collector SUCCESS [  4.384 s]
[INFO] com.capitalone.dashboard:subversion-collector ...... SUCCESS [ 21.260 s]
[INFO] com.capitalone.dashboard:gitlab-scm-collector ...... SUCCESS [  3.083 s]
[INFO] com.capitalone.dashboard:hpsm-cmdb-collector ....... SUCCESS [  3.074 s]
[INFO] com.capitalone.dashboard:nexus-iq-collector ........ SUCCESS [  3.144 s]
[INFO] com.capitalone.dashboard:score-collector ........... SUCCESS [  4.487 s]
[INFO] Hygieia Publisher Plugin ........................... SUCCESS [08:25 min]
[INFO] com.capitalone.dashboard:UI ........................ SUCCESS [02:40 min]
[INFO] com.capitalone.dashboard:ui-tests 2.0.5-SNAPSHOT ... SUCCESS [01:44 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25:09 min
[INFO] Finished at: 2018-07-11T09:59:48+08:00
[INFO] ------------------------------------------------------------------------

生成认证密钥

  • 生成一个core的key
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/core/target]$ java -jar core-2.0.5-SNAPSHOT.jar com.capitalone.dashboard.util.Encryption
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Your secret key is:
xIleAiYTyCy1McKXZz2dL6s4+FtiaML+
Sample encrypted string with the above key for 'thisIsMyPassword' is:
0kY/r1UAMiedT2XQCWFQwhJzAWMGiA/k
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/core/target]$ 
  • gitlab tokens
    在gitlab上生产一个 Access Tokens

    点击create pesonal access token

    将生成的Access Token保存:NsDKbGL2P9NYXFQbtMpP
  • mongodb授权
    mongodb安装配置参考:https://www.linuxea.com/1848.html
    创建admin账户
> db.createUser({"user":"admin","pwd":"admin","roles":["root"]}) 
Successfully added user: { "user" : "admin", "roles" : [ "root" ] }

到admin库创建用户和库的授权信息

> use admin
switched to db admin
> db.auth("admin","admin")
1

开始创建linuxeacom

> db.createUser({user: "linuxeacom",pwd: "123456",roles: [{role: "readWrite", db: "linuxeacom"}]})
Successfully added user: {
    "user" : "linuxeacom",
    "roles" : [
        {
            "role" : "readWrite",
            "db" : "linuxeacom"
        }
    ]
}
> 

进入linuxeacom授权linuxeacom用户

> use linuxeacom
switched to db linuxeacom
> db.createUser({user: "linuxeacom",pwd: "123456",roles: [{role: "readWrite", db: "linuxeacom"}]})
Successfully added user: {
    "user" : "linuxeacom",
    "roles" : [
        {
            "role" : "readWrite",
            "db" : "linuxeacom"
        }
    ]
}
> 

验证

> use linuxeacom
switched to db linuxeacom
> db.auth("linuxeacom","123456")
1
> 

配置文件

  • api部分
    将key加入到api配置中
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/api/target]$ cat dashboard.properties 
dbname=linuxeacom
dbhost=10.10.240.203
dbusername=linuxeacom
dbpassword=123456
dbhostport=10.10.240.203:27017
dbport=27017
dbreplicaset=false
server.contextPath=/api
auth.secret = hygsecret
server.port=8080
key=0kY/r1UAMiedT2XQCWFQwhJzAWMGiA/k
  • gitlab配置文件
    gitlab生成的Access Token填写进去,key也填写进去
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/collectors/feature/gitlab/target]$ cat gitlab-application.properties 
# Database Name
dbname=linuxeacom
# Database HostName - default is localhost
dbhost=10.10.240.203
# Database Port - default is 27017
dbport=27017
# MongoDB replicaset
#dbreplicaset=[false if you are not using MongoDB replicaset]
dbreplicaset=false
dbhostport=10.10.240.203:27017,127.0.0.1:27017
# Database Username - default is blank
dbusername=linuxeacom
# Database Password - default is blank
dbpassword=123456
# Logging File location
logging.file=./logs/gitlab.log
#Collector schedule (required)
gitlab.cron=0 0/5 * * * *
#Gitlab host (optional, defaults to 'gitlab.com')
gitlab.host=10.10.240.146
#Gitlab protocol (optional, defaults to 'http')
gitlab.protocol=http
#Gitlab port (optional, defaults to protocol default port)
gitlab.port=80
#Gitlab path (optional, if your instance of gitlab requires a path)
#gitlab.path=
gitlab.apiToken=NsDKbGL2P9NYXFQbtMpP
gitlab.commitThresholdDays=20
gitlab.key=0kY/r1UAMiedT2XQCWFQwhJzAWMGiA/k
gitlab.apiVersion=4
  • UI
    切换到root用户下,安装gulp
[root@LinuxEA-VM-Node202 /home/Hygieia/Hygieia/UI]# npm install -g gulp
npm WARN deprecated gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated graceful-fs@3.0.11: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated graceful-fs@1.2.3: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
npm WARN notice [SECURITY] minimatch has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=minimatch&version=0.2.14 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] minimatch has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=minimatch&version=2.0.10 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=1.0.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
/usr/local/node-v8.11.3-linux-x64/bin/gulp -> /usr/local/node-v8.11.3-linux-x64/lib/node_modules/gulp/bin/gulp.js

+ gulp@3.9.1
  added 253 packages in 7.074s
[root@LinuxEA-VM-Node202 /home/Hygieia/Hygieia/UI]# 

启动

在启动api之前,加密属性的配置

[Hygieia@LinuxEA-VM-Node202 ~]$ java -cp ~/.m2/repository/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar  org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="dbpassword" password=hygieiasecret algorithm=PBEWithMD5AndDES

----ENVIRONMENT-----------------

Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.171-b11 



----ARGUMENTS-------------------

algorithm: PBEWithMD5AndDES
input: dbpassword
password: hygieiasecret



----OUTPUT----------------------

6f/7fe3Mky72zulhSUu6PyujTNtHdQmy

[Hygieia@LinuxEA-VM-Node202 ~]$ 

在root用户下启动

  • 启动api
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/api/target]$ exit
登出
[root@LinuxEA-VM-Node202 ~]# cd /home/Hygieia/Hygieia/api/target/
[root@LinuxEA-VM-Node202 /home/Hygieia/Hygieia/api/target]# 
[root@LinuxEA-VM-Node202 /home/Hygieia/Hygieia/api/target]# java -jar api.jar --spring.config.location=./dashboard.properties -Djasypt.encryptor.password=hygieiasecret
  • 启动scm插件gitlab
[Hygieia@LinuxEA-VM-Node202 ~/Hygieia/collectors/scm/gitlab/target]$ java -jar gitlab-scm-collector-2.0.5-SNAPSHOT.jar --spring.config.name=gitlab --spring.config.location=./gitlab-application.properties
  • 启动UI
[root@LinuxEA-VM-Node202 /home/Hygieia/Hygieia/UI]# gulp serve
[11:07:05] Using gulpfile /home/Hygieia/Hygieia/UI/gulpfile.js
[11:07:05] Starting 'build'...
[11:07:05] Starting 'clean'...
[11:07:05] Finished 'clean' after 57 ms
[11:07:05] Starting 'assets'...
[11:07:05] Starting 'themes'...
[11:07:05] Starting 'fonts'...
[11:07:05] Starting 'js'...
[11:07:05] Starting 'views'...
[11:07:05] Starting 'test-data'...
[11:07:13] Finished 'themes' after 8.05 s
[11:07:13] Finished 'assets' after 8.05 s
[11:07:13] Finished 'test-data' after 8.1 s
[11:07:13] Finished 'views' after 8.23 s
[11:07:13] Finished 'js' after 8.36 s
[11:07:14] Finished 'fonts' after 9.29 s
[11:07:14] Starting 'html'...
[11:07:14] gulp-inject 1 files into index.html.
[11:07:14] gulp-inject 155 files into index.html.
[11:07:14] Finished 'html' after 219 ms
[11:07:14] Finished 'build' after 9.59 s
[11:07:14] Starting 'serve'...
[11:07:14] Finished 'serve' after 162 ms
[BS] Local URL: http://localhost:3000
[BS] External URL: http://10.10.240.202:3000
[BS] Serving files from: dist/

添加防火墙放行

[root@LinuxEA-VM-Node202 ~]# iptables -I INPUT 6 -p tcp --dport 3000 -j ACCEPT

界面配置

点击右上角的Login,在弹出来的界面上点击Sign UP,注册一个即可

创建一个项目 create a new dashboard,在widget management中选择添加
我这里测试repo

填写完成后保存

选中一个查看

II. sonar与Hygieia

  • 启动一个sonarqube
    快速的run一个sonarqube
[root@DS-VM-Node_10_10_240_145 ~]$ docker run -d --name sonarqube --net=host -e SONARQUBE_JDBC_USERNAME=linuxeacom -e SONARQUBE_JDBC_PASSWORD=123 -e SONARQUBE_JDBC_URL=jdbc:postgresql://10.10.240.202/linuxeacom sonarqube:6.7.4

通过IP和端口能够打开

并且使用sonar-scanner检测一次

[gitlab-runner@DS-VM-Node_10_10_240_145 ~/builds/d7f8c868/0/Hygieia_user/linuxea_app]$ docker run -v $(pwd):/root/src  -v  /var/run/docker.sock:/var/run/docker.sock "newtmitch/sonar-scanner" sonar-scanner -Dsonar.host.url=http://10.10.240.145:9000 -Dsonar.projectKey=linuxea_app -Dsonar.projectName=linuxea_app -Dsonar.projectBaseDir=/root/src -Dsonar.sources=./ -Dsonar.java.binaries=. 
  • 配置sonar api
    切换到$PATH/Hygieia/collectors/build/sonar/target/目录下创建配置文件
    配置文件详情请关注备注
[root@DS-VM-Node202 ~]# cd /home/Hygieia/Hygieia/collectors/build/sonar/target/
[root@DS-VM-Node202 /home/Hygieia/Hygieia/collectors/build/sonar/target]# cat sonar-application.properties 
# Database Name
dbname=linuxeacom
# Database HostName - default is localhost
dbhost=10.10.240.203
# Database Port - default is 27017
dbport=27017
# MongoDB replicaset
dbreplicaset=false
dbhostport=[10.10.240.203:27017]
# Database Username - default is blank
dbusername=linuxeacom
# Database Password - default is blank
dbpassword=123456
# Collector schedule (required)
sonar.cron=0 0/5 * * * *
# Sonar server(s) (required) - Can provide multiple
sonar.servers[0]=http://10.10.240.145:9000
# Sonar version, match array index to the server. If not set, will default to version prior to 6.3.
sonar.versions[0]=6.7
# Sonar Metrics - Required. 
#Sonar versions lesser than 6.3
sonar.metrics[0]=ncloc,line_coverage,violations,critical_violations,major_violations,blocker_violations,violations_density,sqale_index,test_success_density,test_failures,test_errors,tests
# For Sonar version 6.3 and above
sonar.metrics[0]=ncloc,violations,new_vulnerabilities,critical_violations,major_violations,blocker_violations,tests,test_success_density,test_errors,test_failures,coverage,line_coverage,sqale_index,alert_status,quality_gate_details
# Sonar login credentials
sonar.username=admin
sonar.password=admin
[root@DS-VM-Node202 /home/Hygieia/Hygieia/collectors/build/sonar/target]# 

而后就在当前目录启动

[root@DS-VM-Node202 /home/Hygieia/Hygieia/collectors/build/sonar/target]# java -jar sonar-codequality-collector-2.0.5-SNAPSHOT.jar  --spring.config.name=sonar --spring.config.location=./sonar-application.properties 

你可能会看到这样的日志

    • 回到UI添加
      添加一个code analysis ,在这个下拉菜单中必须存在sonar中的项目,否则则是失败的,如果失败请检查日志

      保存
2 分享

您可以选择一种方式赞助本站

支付宝扫码赞助

支付宝扫码赞助

日期: 2018-07-13分类: 持续集成

标签: devops

发表评论