linuxea:Prometheus远程存储Promscale和TimescaleDB测试 promscale 是一个开源的可观察性后端，用于由 SQL 提供支持的指标和跟踪。它建立在 PostgreSQL 和 TimescaleDB 的强大和高性能基础之上。它通过 OpenTelemetry Collector 原生支持 Prometheus 指标和 OpenTelemetry 跟踪以及许多其他格式，如 StatsD、Jaeger 和 Zipkin，并且100% 兼容 PromQL。其完整的 SQL 功能使开发人员能够关联指标、跟踪和业务数据，从而获得新的有价值的见解，当数据在不同系统中孤立时是不可能的。它很容易与 Grafana 和 Jaeger 集成，以可视化指标和跟踪。它建立在 PostgreSQL 和 TimescaleDB 之上，继承了坚如磐石的可靠性、高达 90% 的本机压缩、连续聚合以及在全球数百万个实例上运行的系统的操作成熟度。Promscale 可以用作 Grafana和PromLens等可视化工具的 Prometheus 数据源。Promscale 包括两个组件：Promscale 连接器：一种无状态服务，为可观察性数据提供摄取接口，处理该数据并将其存储在 TimescaleDB 中。它还提供了一个使用 PromQL 查询数据的接口。Promscale 连接器自动设置 TimescaleDB 中的数据结构以存储数据并在需要升级到新版本的 Promscale 时处理这些数据结构中的更改。TimescaleDB：存储所有可观察性数据的基于 Postgres 的数据库。它提供了用于查询数据的完整 SQL 接口以及分析函数、列压缩和连续聚合等高级功能。TimescaleDB 提供了很大的灵活性来存储业务和其他类型的数据，然后你可以使用这些数据与可观察性数据相关联。Promscale 连接器使用 Prometheusremote_write接口摄取 Prometheus 指标、元数据和 OpenMetrics 示例。它还使用 OpenTelemetry 协议 (OTLP) 摄取 OpenTelemetry 跟踪。它还可以使用 OpenTelemetry 收集器以其他格式摄取指标和跟踪，以通过 Prometheusremote_write接口和 OpenTelemetry 协议处理和发送它们。例如，你可以使用 OpenTelemetry Collector 将 Jaeger 跟踪和 StatsD 指标摄取到 Promscale。对于 Prometheus 指标，Promscale 连接器公开 Prometheus API 端点，用于运行 PromQL 查询和读取元数据。这允许你将支持 Prometheus API 的工具（例如 Grafana）直接连接到 Promscale 进行查询。也可以向 Prometheus 发送查询，并让 Prometheus 使用接口上的 Promscale 连接器从 Promscale 读取数据 remote_read。你还可以使用 SQL 在 Promscale 中查询指标和跟踪，这允许你使用与 PostgreSQL 集成的许多不同的可视化工具。例如，Grafana 支持通过 PostgreSQL 数据源使用开箱即用的 SQL 查询 Promscale 中的数据我准备通过容器的方式进行尝试，我们先安装docker和docker-composeyum install -y yum-utils \ device-mapper-persistent-data \ lvm2 yum-config-manager \ --add-repo \ https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo yum install docker-ce docker-ce-cli containerd.io docker-compose -y编排我按照官网的docker配置，进行编排了compose进行测试version: '2.2' services: timescaledb: image: timescaledev/promscale-extension:latest-ts2-pg13 container_name: timescaledb restart: always hostname: "timescaledb" network_mode: "host" environment: - csynchronous_commit=off - POSTGRES_PASSWORD=123 volumes: - /etc/localtime:/etc/localtime:ro - /data/prom/timescaledb/data:/var/lib/postgresql/data:rw mem_limit: 512m user: root stop_grace_period: 1m promscale: image: timescale/promscale:0.10 container_name: promscale restart: always hostname: "promscale" network_mode: "host" environment: - PROMSCALE_DB_PASSWORD=123 - PROMSCALE_DB_PORT=5432 - PROMSCALE_DB_NAME=postgres - PROMSCALE_DB_HOST=127.0.0.1 - PROMSCALE_DB_SSL_MODE=allow volumes: - /etc/localtime:/etc/localtime:ro # - /data/prom/postgresql/data:/var/lib/postgresql/data:rw mem_limit: 512m user: root stop_grace_period: 1m grafana: image: grafana/grafana:8.3.7 container_name: grafana restart: always hostname: "grafana" network_mode: "host" #environment: # - GF_INSTALL_PLUGINS="grafana-clock-panel,grafana-simple-json-datasource" volumes: - /etc/localtime:/etc/localtime:ro - /data/grafana/plugins:/var/lib/grafana/plugins mem_limit: 512m user: root prometheus: image: prom/prometheus:v2.33.4 container_name: prometheus restart: always hostname: "prometheus" network_mode: "host" #environment: volumes: - /etc/localtime:/etc/localtime:ro - /data/prom/prometheus/data:/prometheus:rw # NOTE: chown 65534:65534 /data/prometheus/ - /data/prom/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - /data/prom/prometheus/alert:/etc/prometheus/alert #- /data/prom/prometheus/ssl:/etc/prometheus/ssl command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention=45d' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--web.console.templates=/etc/prometheus/consoles' - '--web.enable-lifecycle' - '--web.enable-admin-api' mem_limit: 512m user: root stop_grace_period: 1m node_exporter: image: prom/node-exporter:v1.3.1 container_name: node_exporter user: root privileged: true network_mode: "host" volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)' restart: unless-stoppedgrafana我们这里使用的root用户就是因为需要手动安装下插件bash-5.1# grafana-cli plugins install grafana-clock-panel ✔ Downloaded grafana-clock-panel v1.3.0 zip successfully Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary. bash-5.1# grafana-cli plugins install grafana-simple-json-datasource ✔ Downloaded grafana-simple-json-datasource v1.4.2 zip successfully Please restart Grafana after installing plugins. Refer to Grafana documentation for instructions if necessary.配置grafana在这里下载一些模板https://grafana.com/grafana/dashboards/?pg=hp&plcmt=lt-box-dashboards&search=prometheusVisualize data in Promscaleprometheus我们可以尝试配置远程， 配置参数可以查看官网remote_write: - url: "http://127.0.0.1:9201/write" remote_read: - url: "http://127.0.0.1:9201/read" read_recent: true远程配置如下remote_write: - url: "http://127.0.0.1:9201/write" write_relabel_configs: - source_labels: [__name__] regex: '.*:.*' action: drop remote_timeout: 100s queue_config: capacity: 500000 max_samples_per_send: 50000 batch_send_deadline: 30s min_backoff: 100ms max_backoff: 10s min_shards: 16 max_shards: 16 remote_read: - url: "http://127.0.0.1:9201/read" read_recent: true prometheus.yaml如下global: scrape_interval: 15s evaluation_interval: 15s alerting: alertmanagers: - scheme: http static_configs: - targets: - '127.0.0.1:9093' rule_files: - "alert/host.alert.rules" - "alert/container.alert.rules" - "alert/targets.alert.rules" scrape_configs: - job_name: prometheus scrape_interval: 30s static_configs: - targets: ['127.0.0.1:9090'] - targets: ['127.0.0.1:9093'] - targets: ['127.0.0.1:9100'] remote_write: - url: "http://127.0.0.1:9201/write" write_relabel_configs: - source_labels: [__name__] regex: '.*:.*' action: drop remote_timeout: 100s queue_config: capacity: 500000 max_samples_per_send: 50000 batch_send_deadline: 30s min_backoff: 100ms max_backoff: 10s min_shards: 16 max_shards: 16 remote_read: - url: "http://127.0.0.1:9201/read" read_recent: true重新启动后查看日志ts=2022-03-03T01:35:28.123Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml ts=2022-03-03T01:35:28.137Z caller=dedupe.go:112 component=remote level=info remote_name=797d34 url=http://127.0.0.1:9201/write msg="Starting WAL watcher" queue=797d34 ts=2022-03-03T01:35:28.138Z caller=dedupe.go:112 component=remote level=info remote_name=797d34 url=http://127.0.0.1:9201/write msg="Starting scraped metadata watcher" ts=2022-03-03T01:35:28.277Z caller=dedupe.go:112 component=remote level=info remote_name=797d34 url=http://127.0.0.1:9201/write msg="Replaying WAL" queue=797d34 ts=2022-03-03T01:35:38.177Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=10.053377011s db_storage=1.82µs remote_storage=13.752341ms web_handler=549ns query_engine=839ns scrape=10.038744417s scrape_sd=44.249µs notify=41.342µs notify_sd=6.871µs rules=30.465µs ts=2022-03-03T01:35:38.177Z caller=main.go:896 level=info msg="Server is ready to receive web requests." ts=2022-03-03T01:35:53.584Z caller=dedupe.go:112 component=remote level=info remote_name=797d34 url=http://127.0.0.1:9201/write msg="Done replaying WAL" duration=25.446317635s查看数据[root@localhost data]# docker exec -it timescaledb sh / # su - postgres timescaledb:~$ psql psql (13.4) Type "help" for help. postgres=# 我们查询一个过去五分钟io的指标查询指标SELECT * from node_disk_io_now WHERE time > now() - INTERVAL '5 minutes'; time | value | series_id | labels | device_id | instance_id | job_id ----------------------------+-------+-----------+---------------+-----------+-------------+-------- 2022-03-02 21:03:58.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:04:28.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:04:58.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:05:28.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:05:58.376-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:06:28.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:06:58.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:07:28.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:07:58.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:08:28.373-05 | 0 | 348 | {51,140,91,3} | 140 | 91 | 3 2022-03-02 21:03:58.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:04:28.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:04:58.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:05:28.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:05:58.376-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:06:28.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:06:58.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:07:28.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:07:58.373-05 | 0 | 349 | {51,252,91,3} | 252 | 91 | 3 2022-03-02 21:03:58.373-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3 2022-03-02 21:04:28.373-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3 2022-03-02 21:04:58.373-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3 2022-03-02 21:05:28.373-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3 2022-03-02 21:05:58.376-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3 2022-03-02 21:06:28.373-05 | 0 | 350 | {51,253,91,3} | 253 | 91 | 3在进行一次聚合查询标签键的查询值每个标签键都扩展为自己的列，该列将外键标识符存储为其值。这允许JOIN按标签键和值进行聚合和过滤。要检索由标签 ID 表示的文本，可以使用该val(field_id) 函数。这使你可以使用特定的标签键对所有系列进行聚合等操作。例如，要查找指标的中值node_disk_io_now，按与其关联的工作分组：SELECT val(job_id) as job, percentile_cont(0.5) within group (order by value) AS median FROM node_disk_io_now WHERE time > now() - INTERVAL '5 minutes' GROUP BY job_id;如下postgres=# SELECT postgres-# val(job_id) as job, postgres-# percentile_cont(0.5) within group (order by value) AS median postgres-# FROM postgres-# node_disk_io_now postgres-# WHERE postgres-# time > now() - INTERVAL '5 minutes' postgres-# GROUP BY job_id; job | median ------------+-------- prometheus | 0 (1 row)查询指标的标签集任何度量行中的labels字段表示与测量相关的完整标签集。它表示为标识符数组。要以 JSON 格式返回整个标签集，你可以使用该jsonb()函数，如下所示：SELECT time, value, jsonb(labels) as labels FROM node_disk_io_now WHERE time > now() - INTERVAL '5 minutes';如下 time | value | labels ----------------------------+-------+------------------------------------------------------------------------------------------------------- 2022-03-02 21:09:58.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:28.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:58.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:28.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:58.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:12:28.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:12:58.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:13:28.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:13:58.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:14:28.373-05 | 0 | {"job": "prometheus", "device": "dm-0", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:09:58.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:28.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:58.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:28.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:58.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:12:28.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:12:58.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:13:28.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:13:58.373-05 | 0 | {"job": "prometheus", "device": "dm-1", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:09:58.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:28.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:10:58.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:28.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:11:58.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"} 2022-03-02 21:12:28.373-05 | 0 | {"job": "prometheus", "device": "sda", "__name__": "node_disk_io_now", "instance": "127.0.0.1:9100"}查询node_disk_infopostgres=# SELECT * FROM prom_series.node_disk_info; series_id | labels | device | instance | job | major | minor -----------+------------------------+--------+----------------+------------+-------+------- 250 | {150,140,91,3,324,325} | dm-0 | 127.0.0.1:9100 | prometheus | 253 | 0 439 | {150,253,91,3,508,325} | sda | 127.0.0.1:9100 | prometheus | 8 | 0 440 | {150,258,91,3,507,325} | sr0 | 127.0.0.1:9100 | prometheus | 11 | 0 516 | {150,252,91,3,324,564} | dm-1 | 127.0.0.1:9100 | prometheus | 253 | 1 (4 rows)带标签查询SELECT jsonb(labels) as labels, value FROM node_disk_info WHERE time < now(); Results:labels | value -----------------------------------------------------------------------------------------------------------------------------------+------- {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | NaN {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1 {"job": "prometheus", "major": "253", "minor": "0", "device": "dm-0", "__name__": "node_disk_info", "instance": "127.0.0.1:9100"} | 1通过命令进行查看她的指标视图postgres=# \d+ node_disk_info View "prom_metric.node_disk_info" Column | Type | Collation | Nullable | Default | Storage | Description -------------+--------------------------+-----------+----------+---------+----------+------------- time | timestamp with time zone | | | | plain | value | double precision | | | | plain | series_id | bigint | | | | plain | labels | label_array | | | | extended | device_id | integer | | | | plain | instance_id | integer | | | | plain | job_id | integer | | | | plain | major_id | integer | | | | plain | minor_id | integer | | | | plain | View definition: SELECT data."time", data.value, data.series_id, series.labels, series.labels[2] AS device_id, series.labels[3] AS instance_id, series.labels[4] AS job_id, series.labels[5] AS major_id, series.labels[6] AS minor_id FROM prom_data.node_disk_info data LEFT JOIN prom_data_series.node_disk_info series ON series.id = data.series_id;更多的查询，可以查看官网的教程计划删除promscale里是pg里配删除计划的是90天删除通过SELECT * FROM prom_info.metric;查看我们可以通过调整来修改TimescaleDB 包括一个后台作业调度框架，用于自动化数据管理任务，例如启用简单的数据保留策略。为了添加这样的数据保留策略，数据库管理员可以创建、删除或更改导致drop_chunks根据某个定义的计划自动执行的策略。要在超表上添加这样的策略，不断导致超过 24 小时的块被删除，只需执行以下命令：SELECT add_retention_policy('conditions', INTERVAL '24 hours');随后删除该策略：SELECT remove_retention_policy('conditions');调度程序框架还允许查看已调度的作业：SELECT * FROM timescaledb_information.job_stats;创建数据保留策略以丢弃超过 6 个月的数据块：SELECT add_retention_policy('conditions', INTERVAL '6 months');复制使用基于整数的时间列创建数据保留策略：SELECT add_retention_policy('conditions', BIGINT '600000');我们可以调整prometheus的数据，参考Data RetentionSELECT set_default_retention_period(180 * INTERVAL '1 day')如下postgres=# SELECT postgres-# set_default_retention_period(180 * INTERVAL '1 day'); set_default_retention_period ------------------------------ t (1 row)已经修改为180天我们打开prometheus和grafana都可以正常的查看grafana关于当前版本的压测在slack上，有一个朋友做了压测Promscale版本 2.30.0promscale 0.10.0Hello everyone, I am doing a promscale+timescaledb performance test with 1 promscale(8cpu 32GB memory), 1 timescaledb(postgre12.9+timescale2.5.2 with 16cpu 32G mem), 1 prometheus(8cpu 32G mem), simulate 2500 node_exporters( 1000 metrics/min * 2500 = 2.5 million metrics/min ) . But it seams not stable,做一个promscale+timescaledb性能测试，1个promscale（8cpu 32GB内存），1个timescaledb（postgre12.9+timescale2.5.2 with 16cpu 32G mem），1个prometheus（8cpu 32G mem），模拟2500个node_exporters（1000 指标/分钟 * 2500 = 250 万指标/分钟）。 但它的接缝不稳定异常如下there are warninigs in prometheus: level=info ts=2022-03-01T12:36:33.365Z caller=throughput.go:76 msg="ingestor throughput" samples/sec=35000 metrics-max-sent-ts=2022-03-01T11:21:48.129Z level=info ts=2022-03-01T12:36:34.365Z caller=throughput.go:76 msg="ingestor throughput" samples/sec=35000 metrics-max-sent-ts=2022-03-01T11:21:48.129Z level=warn ts=2022-03-01T12:36:34.482Z caller=watcher.go:101 msg="[WARNING] Ingestion is a very long time" duration=5m9.705407837s threshold=1m0s level=info ts=2022-03-01T12:36:35.365Z caller=throughput.go:76 msg="ingestor throughput" samples/sec=35000 metrics-max-sent-ts=2022-03-01T11:21:48.129Z level=info ts=2022-03-01T12:36:40.365Z caller=throughput.go:76 msg="ingestor throughput" samples/sec=70000 metrics-max-sent-ts=2022-03-01T11:21:48.129Z and errors in prometheus: Mar 01 20:38:55 localhost start-prometheus.sh[887]: ts=2022-03-01T12:38:55.288Z caller=dedupe.go:112 component=remote level=warn remote_name=ceec38 url=http://192.168.105.76:9201/write msg="Failed to send batch, retrying" err="Post \"http://192.168.105.76:9201/write\": context deadline exceeded" any way to increase the thoughput at current configuration?他们推荐使用 remote_write: - url: "http://promscale:9201/write" write_relabel_configs: - source_labels: [__name__] regex: '.*:.*' action: drop remote_timeout: 100s queue_config: capacity: 500000 max_samples_per_send: 50000 batch_send_deadline: 30s min_backoff: 100ms max_backoff: 10s min_shards: 16 max_shards: 16 随后将配置pg 更改为 14.2 和 remote_write 设置，将 promscale mem 增加到 32G，仍然不稳定。请注意，这是2500台的节点压测，那么，这个朋友的测试可以看到至少在目前看来，promscale的开发版本仍然是处于一个初期。官方并没有进行可靠性压测 .我们期待未来的稳定版本

linuxea:清理kube-prometheus历史数据 通常在k8s中，pod是随时可以被替换的，在整个环境里往往我们不太关注某一条鱼，只关注整个鱼群的状态，因此监控数据不会存储太长，因为借鉴意义并不大。但是有时的确想要从 Prometheus 中删除一些指标，如果这些指标不需要，或者只需要释放一些磁盘空间。Prometheus 中的时间序列只能通过管理 HTTP API 删除（默认禁用）。--web.enable-admin-apiAs of Prometheus 2.0, the --web.enable-admin-api flag controls access to the administrative HTTP API which includes functionality such as deleting time series. This is disabled by default. If enabled, administrative and mutating functionality will be accessible under the /api/*/admin/ paths. The --web.enable-lifecycle flag controls HTTP reloads and shutdowns of Prometheus. This is also disabled by default. If enabled they will be accessible under the /-/reload and /-/quit paths.要启用它--web.enable-admin-api，根据安装方法通过启动脚本或 docker-compose 文件将标志传递给 Prometheus。删除时间序列指标使用以下语法删除与某个标签匹配的所有时间序列指标：curl -X POST \ -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={name="linuxea.com"}'要删除与 somejob或匹配的时间序列指标instance，请运行：curl −X POST −g 'http://localhost:9090/api/v1/admin/tsdb/deleteseries?match[]=job="nodeexporter"' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_exporter"}' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="192.168.0.1:9100"}'要从 Prometheus 中删除所有数据，请运行：curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={name=~".+"}'请注意，上述 API 调用不会立即删除数据实际数据仍然存在于磁盘上，将在未来的压缩中被清除。要确定何时删除旧数据，使用--storage.tsdb.retention选项 eg --storage.tsdb.retention='365d'（默认情况下，Prometheus 将数据保留 15 天）要完全删除通过delete_series发送clean_tombstonesAPI 调用删除的数据：curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'删除所有历史数据我们使用的kube-prometheus，添加--web.enable-admin-api方式有所不同打开prometheus-prometheus.yaml 添加enableAdminAPI: true字段 probeNamespaceSelector: {} probeSelector: {} replicas: 2 resources: requests: memory: 400Mi enableAdminAPI: true ruleSelector: matchLabels: prometheus: k8s而后kubectl apply -f prometheus-prometheus.yaml，pod起来后查看 containers: - args: - '--web.console.templates=/etc/prometheus/consoles' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--config.file=/etc/prometheus/config_out/prometheus.env.yaml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention.time=15d' - '--web.enable-lifecycle' - '--storage.tsdb.no-lockfile' - '--web.route-prefix=/' - '--web.enable-admin-api' image: 'quay.io/prometheus/prometheus:v2.26.0' 准备开始删除curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'查看删除前的大小[root@linuxea.com prometheus-db]# ls 01FPZGK1S0BZ9FZMSVPYWA1F52 01FQPP5PJTEMT1BARN8ARJTBKB 01FR222TD3ZWYAZTNRJP5SCJ6M 01FR4VB8N4J0D2DY979PTHZQY1 wal 01FQ59ZR5Z9VNNGY3JSXP6F9AQ 01FQWFJAC72YCBKEN19F39PYCM 01FR46R88Q9NN7GR99MMGX13VG 01FR526TSKK8JTN493PTBQH0XC 01FQB3CCFVNSP890V5KE07YD3M 01FQYDBKTHH8KB8K5GT581BKGE 01FR4MFC9JX6NK81G4Q1W0CJNY chunks_head 01FQGWS2M82Q23H0H2TD0JXK1Y 01FR0B54YSBJ0R54B5171P4Z7R 01FR4VB3HHRN0EE1GQCM6G0A77 queries.active [root@linuxea.com nfs-k8s]# du -sh monitoring-prometheus-k8s-db-prometheus-k8s-0-pvc-ac911baf-f1f1-4bf3-a1f8-af2cb13c5d90/ 7.1G monitoring-prometheus-k8s-db-prometheus-k8s-0-pvc-ac911baf-f1f1-4bf3-a1f8-af2cb13c5d90/找到svc[root@linuxea.com nfs-k8s]# kubectl -n monitoring get svc |grep prometheus-k8s prometheus-k8s NodePort 10.68.110.123 <none> 9090:30090/TCP 128d尝试curl一下[root@linuxea.com nfs-k8s]# curl 10.68.110.123:9090 <a href="/graph">Found</a>.执行[root@linuxea.com manifests]# curl -X POST -g 'http://10.68.110.123:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'而后在查看大小[root@linuxea.com manifests]# kubectl -n monitoring exec -it prometheus-k8s-0 -- /bin/sh Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod. /prometheus $ du -sh ./ 1004.4M ./ /prometheus $ ls 01FR46R88Q9NN7GR99MMGX13VG 01FR4VB8N4J0D2DY979PTHZQY1 chunks_head 01FR4MFC9JX6NK81G4Q1W0CJNY 01FR526TSKK8JTN493PTBQH0XC queries.active 01FR4VB3HHRN0EE1GQCM6G0A77 01FR592J0H8ANAWMFP2VVQ3NX9 wal /prometheus $ du -sh ./ 1006.6M ./完全删除通过delete_series发送clean_tombstonesAPI 调用删除的数据[root@linuxea.com manifests]# curl -X POST -g 'http://10.68.110.123:9090/api/v1/admin/tsdb/clean_tombstones'在查看大小[root@linuxea.com manifests]# kubectl -n monitoring exec -it prometheus-k8s-0 -- /bin/sh Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod. /prometheus $ du -sh ./ 986.6M ./删除完成后，将--web.enable-admin-api关闭。修改enableAdminAPI: false字段即可延申参考tsdb-admin-apisAdd flag to enable prometheus web admin API and fixes #1215kube-prometheusspecprometheus-prometheus.yaml

linuxea:关于kube-prometheus中CPUThrottlingHigh 我们遇到的场景是CPUThrottlingHigh 警报被正常触发，而触发的对象的CPU本身并不高，或者空闲。鉴于此，我们开始怀疑这个警报的必然性。通常在许多情况下，会将此警报修改或者沉默，因为应用程序对延迟不敏感，即使受到限制也可以正常工作，警报基于原因而非症状。因此警报的级别是Info。但是并不能说明此警报是误报。并且沉默只会隐藏背后的真正问题。目前这个问题仍然在讨论中，特别是在这个讨论的特别激烈108,而后在67577也有进一步的讨论表达式如下：sum(increase(container_cpu_cfs_throttled_periods_total{container!="", }[5m])) by (container, pod, namespace) / sum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace) > ( 25 / 100 )目前，总结了几种易处理的方式1，修改警报阈值比例，或者禁止他 2，取消或者修改对这些 pod 的限制3, 内核4.18或者更高3，完全禁止Kubernetes CFS配额(kubelet配置--cpu-cfs-quota=false)我们尝试修改阈值kubectl -n monitoring edit PrometheusRule prometheus-k8s-rules修改 - alert: CPUThrottlingHigh annotations: description: '{{ $value | humanizePercentage }} throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod }}.' runbook_url: https://github.com/prometheus-operator/kube-prometheus/wiki/cputhrottlinghigh summary: Processes experience elevated CPU throttling. expr: | sum(increase(container_cpu_cfs_throttled_periods_total{container!="", }[5m])) by (container, pod, namespace) / sum(increase(container_cpu_cfs_periods_total{}[5m])) by (container, pod, namespace) > ( 75 / 100 ) for: 15m labels: severity: info其他相关参考：https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/108 https://github.com/prometheus-operator/prometheus-operator/issues/2063 https://github.com/kubernetes/kubernetes/issues/67577 https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/ https://bugzilla.kernel.org/show_bug.cgi?id=198197 https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1 https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt https://github.com/prometheus-operator/kube-prometheus/issues/214 https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/453 https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/b71dd35c6a1d509a1ee902eebe7afe943d8ee4b0/alerts/resource_alerts.libsonnet#L13 https://www.youtube.com/watch?v=UE7QX98-kO0 https://github.com/prometheus-operator/kube-prometheus/issues/861 https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/alertmanager.libsonnet#L26-L42 https://devops.stackexchange.com/questions/6494/prometheus-alert-cputhrottlinghigh-raised-but-monitoring-does-not-show-it

linuxea:prometheus基于主机的自动发现(promcr) prometheus自动发现(scrape them)在官网中方式有很多，我这里介绍的是使用consul_sd_config配合registrator来做，registrator作为node节点端，发现容器，且将发现信息注册给consul。拓扑如下：而prometheus中使用consul_sd_config进行重新标记，可以使用的元标签如下：__meta_consul_address：目标的地址__meta_consul_dc：目标的数据中心名称__meta_consul_metadata_<key>：目标的每个节点元数据键值__meta_consul_node：为目标定义的节点名称__meta_consul_service_address：目标的服务地址__meta_consul_service_id：目标的服务ID__meta_consul_service_metadata_<key>：目标的每个服务元数据键值__meta_consul_service_port：目标的服务端口__meta_consul_service：目标所属服务的名称__meta_consul_tags：由标记分隔符连接的目标的标记列表其中目标的IP编号和端口组装为 <__meta_consul_address>:<__meta_consul_service_port>。但是，在一些Consul设置中，相关地址在__meta_consul_service_address。在这些情况下，可以使用relabel 功能替换特殊__address__标签。pcr为了方便使用，我在git创建了一个pcr项目，地址如下https://github.com/marksugar/pcr在此简单说明如何进行基于主机做自动发现，具体可见github说明registratorregistrator运行在每一个node节点，要注册给consul必须是有独立的IP地址，这个ip可能必须是宿主机的ip地址。我们要做的事就是获取每个宿主机的ip地址，并且自动获取。为此，重新封装gliderlabs/registrator:v7容器是很有必要的。添加脚本获取#!/bin/sh # maintainer="linuxea.com" NDIP=`ip a s ${NETWORK_DEVIDE:-eth0}|awk '/inet/{print $2}'|sed -r 's/\/[0-9]{1,}//')` exec /bin/registrator -ip="${NDIP}" ${ND_CMD:--internal=false} consul://${NDIPSERVER_IP:-consul}:8500需要传递几个变量才能使用这个镜象NETWORK_DEVIDE: 网卡名称 NDIPSERVER_IP：consul server ip ND_CMD: 参数Example： environment: - REGISTRATOR_BIND_INTERFACE=eth0 - NETWORK_DEVIDE=eth0 - NDIPSERVER_IP=172.25.250.249 - ND_CMD=-internal=false而在模板中，这里的参数是这样的：- ND_CMD=-internal=false -retry-interval=30 -resync=180-retry-interval=30会在三分钟后自动重新联系CONSUL_SERVERcompose registrator: container_name: registrator image: marksugar/registrator:v7.1 network_mode: "host" depends_on: - consul - cadvisor - node_exporter - alertmanager - grafana - prometheus volumes: - /var/run/docker.sock:/tmp/docker.sock environment: - REGISTRATOR_BIND_INTERFACE=eth0 - NETWORK_DEVIDE=eth0 - NDIPSERVER_IP=172.25.250.249 - ND_CMD=-internal=false -retry-interval=30 -resync=360 cpu_shares: 14 mem_limit: 50m logging: driver: "json-file" options: max-size: "200M" labels: SERVICE_TAGS: prometheusconsulconsul可以是一个集群，也可以说单点，这里已单点的方式运行其中，需要注意绑定的网卡端口-bind '{{ GetInterfaceIP \"eth0\" }}'如果你不是eth0，请修改，如下： consul: container_name: consul image: consul:1.4.0 network_mode: "host" ports: - 8500:8500 command: "agent -server -ui -client=0.0.0.0 -dev -node=node0 -bind '{{ GetInterfaceIP \"eth0\" }}' -bootstrap-expect=1 -data-dir=/consul/data" labels: SERVICE_IGNORE: 'true' environment: - CONSUL_CLIENT_INTERFACE=eth0 cpu_shares: 30 mem_limit: 1024m logging: driver: "json-file" options: max-size: "200M" volumes: - ./consul/config:/consul/config - ./consul/data:/consul/data labels: SERVICE_TAGS: prometheus注册的机器都会在web接口中显示，大致像这样系统与容器在prom/node-exporter:v0.16.0中，选择prom/node-exporter:v0.16.0是有原因的，prom/node-exporter:v0.16.0是可以发现出磁盘，而在prom/node-exporter:v0.17.0经测试，是发现不了的cadvisor启动的端口是18880，这个端口不太会被占用 node_exporter: image: prom/node-exporter:v0.16.0 container_name: node_exporter user: root privileged: true network_mode: "host" volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)' restart: unless-stopped ports: - 9100:9100 cpu_shares: 14 mem_limit: 50m logging: driver: "json-file" options: max-size: "200M" labels: - "SERVICE_TAGS=prometheus" cadvisor: image: google/cadvisor:v0.32.0 container_name: cadvisor network_mode: "host" volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro command: --listen_ip="0.0.0.0" --port=18880 restart: unless-stopped ports: - 18880:18880 cpu_shares: 14 mem_limit: 50m logging: driver: "json-file" options: max-size: "200M" labels: SERVICE_TAGS: prometheusprometheus在prometheus中，我修改了保存的时间，并且配置文件也做了调整 - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention=45d' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--web.console.templates=/etc/prometheus/consoles' - '--web.enable-lifecycle'其中，对于每个容器类型，做了区分，如：cadvisor，如下 - job_name: 'cadvisor' metrics_path: /metrics scheme: http consul_sd_configs: - server: 127.0.0.1:8500 services: ['cadvisor'] relabel_configs: - source_labels: ['__meta_consul_service'] regex: '(.*)' target_label: 'job' replacement: '$1' - source_labels: [__meta_consul_tags] target_label: tags - source_labels: ['__meta_consul_service_address'] regex: '(.*)' target_label: 'instance' replacement: '$1' - source_labels: ['__meta_consul_service_address', '__meta_consul_service_port'] regex: '(.*);(.*)' target_label: '__address__' replacement: '$1:$2' - source_labels: ['__meta_consul_tags'] regex: ',(prometheus|app),' target_label: 'group' replacement: '$1'自动发现字段 relabel_configs: - source_labels: ['__meta_consul_service'] regex: '(.*)' target_label: 'job' replacement: '$1' - source_labels: [__meta_consul_tags] target_label: tags - source_labels: ['__meta_consul_service_address'] regex: '(.*)' target_label: 'instance' replacement: '$1' - source_labels: ['__meta_consul_service_address', '__meta_consul_service_port'] regex: '(.*);(.*)' target_label: '__address__' replacement: '$1:$2'其中：__meta_consul_service发现的地址会重写成job__meta_consul_tags的值重写为tags__meta_consul_service_address获取的ip重写为instance__meta_consul_service_address', '__meta_consul_service_port'分别是ip和端口$1和$2分别对应第一个参数和第二个参数。分组但是这还不够，当发现后，这些信息必须有一个标签来做区分，否则我们是不能够用来灵活使用的，我们需要一个标签，如下所示 - source_labels: ['__meta_consul_tags'] regex: ',(prometheus|app),' target_label: 'group' replacement: '$1'如果__meta_consul_tags中包含prometheus字段或者app字段，就重新成一个组，就是上面所示的意思这也说明了为什么在每个容器后都会打上一个标签的原因 labels: SERVICE_TAGS: prometheus我们可以利用这个标签对项目做分组，灵活的划分grafana有了这个标签之后，我们可以对不同的标签进行划分。如下在项目组中，是有多个组，在每个组下，有多台机器，这里使用ip划分。但是，这样还是不够，有一些容器，没有在k8s中，他在宿主机上运行，那么这个模板就不能完美使用，重新编辑后的，就成了这样子。根据项目，和项目中的容器类型，根据容器来看对应容器下的主机情况。在github中还有其他的模板以供参考。对于如何来分组这个问题，我们首先要有一个标签，或者不同的表情，根据标签来分大组，如类型，项目，web等，而后根据每个标签，或者ip，或者容器名称组合一个序列来看项目地址：https://github.com/marksugar/pcr，这个工具我在使用，逐步更新维护，如果对你有帮助，请移步后顺手点个星星。延伸阅读：https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config延伸阅读：https://prometheus.io/blog/2015/06/01/advanced-service-discovery/

linuxea:prometheus和node_exporter安装使用 node_exporter用Go编写的一个收集器，主要指标来在*NIX内核公开的硬件和操作系统指标。但是每个操作系统上都有不同的支持。如：linux系统的地址解析arp,bcache,开机时间，链接跟踪，cpu信息，磁盘i/o，文件系统，netstat，stat，sockstat，系统时间，vmstat，以及内核4.4的xfs等。默认情况下会启用所有的指标,在高级用途里面是可以进行过滤的。它的安装方式官方提供了好几种，包括docker，由于node_exporter主要监控宿主机，我并不建议使用docker来监控。so，我下载了rpm包，也提供了二进制的安装方式，他们都很好node_exporter这里有yum安装和官网安装包安装yum安装我不建议你使用yum安装，如果你访问不了谷歌，那么你的范围速度相当的慢，鉴于此，我下载好放到github上，你可以试试[root@linuxea-VM-Node_10_10_240_145 ~]$ curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo [root@linuxea-VM-Node_10_10_240_145 ~]$ yum install node_exporter参考：https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/ https://copr-be.cloud.fedoraproject.org/results/ibotty/prometheus-exporters/epel-7-x86_64/00756835-golang-github-prometheus-node_exporter/你可以直接到我github上直接下载我打包的[root@linuxea-VM-Node_10_10_240_145 ~]$ wget https://raw.githubusercontent.com/LinuxEA-Mark/blog-file/master/prometheus/golang-github-prometheus-node_exporter-0.16.0-4.el7.x86_64.zip下载后直接解压安装即可：[root@linuxea-VM-Node_10_10_240_145 ~]$ unzip golang-github-prometheus-node_exporter-0.16.0-4.el7.x86_64.zip Archive: golang-github-prometheus-node_exporter-0.16.0-4.el7.x86_64.zip inflating: golang-github-prometheus-node_exporter-0.16.0-4.el7.x86_64.rpm [root@linuxea-VM-Node_10_10_240_145 ~]$ yum localinstall golang-github-prometheus-node_exporter-0.16.0-4.el7.x86_64.rpm -y [root@linuxea-VM-Node_10_10_240_145 ~]$ systemctl start node_exporter.service 静态二进制文件安装我们打开下载页面下载安装装完后启动，大概是这样复制下面的命令，快速的安装启动[root@linuxea-VM-Node_10_10_240_145 ~]$ curl -Lk https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz|tar xz -C /usr/local && cd /usr/local/node_exporter-0.16.0.linux-amd64 && nohup ./node_exporter 2>&1 &安装完成后，可以通过本地curl 127.0.0.1:/metrics访问查看防火墙放行后可以通过web查看，并且以供服务端抓取数据[root@linuxea-VM-Node_10_10_240_145 ~]$ iptables -I INPUT 5 -p tcp --dport 9100 -j ACCEPT它还提供docker安装，我并不建议，当然你也可以尝试，github地址prometheus配置文件中添加如下：9100端口是node_exporter - job_name: 'gfc' static_configs: - targets: - '10.10.197.3:8888' - '10.10.197.1:8880' - '10.10.197.3:9100' - '10.10.197.1:9100' labels: group: 'gfc_node'reload即可curl -X POST http://10.10.240.143:9090/-/reload,而后在Targets中已经up指标信息在http://IP:PORT/metrics查看假如你已经用了grafana，我这里提供一个找到的模板，属于单个机器的模板，但是非常详细，令人发指，大概长这样,模板下载

linuxea:prometheus通过cAdvisor监控容器状态 cAdvisor（Container Advisor）为容器用户提供了对其运行容器的资源使用和性能特征的理解。它是一个运行守护程序，用于收集，聚合，处理和导出有关正在运行的容器的信息。具体来说，对于每个容器，它保留资源隔离参数，历史资源使用，完整历史资源使用的直方图和网络统计。此数据按容器和机器范围导出。cAdvisor原生支持Docker容器，并且应该支持开箱即用的任何其他容器类型，并且力求更多的支持，如果有使用问题可以在官网提出。cAdvisor的容器抽象基于lmctfy，因此容器本质上是分层嵌套的。在使用cadvisor开始之前，你可能需要安装prometheus和grafana并且配置好，以便于使用，在后面将会使用很多，这很重要，安装和配置这些以便于更好的分析正在运行的容器的资源使用情况和性能特征。cadvisor配置[root@linuxea-vm-Node3 ~]# docker pull google/cadvisor:v0.30.0start 指定IP和端口--listen_ip="0.0.0.0" --port=8888docker run \ --net=host \ --privileged=true \ --volume=/cgroup:/cgroup:ro \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --volume=/dev/disk/:/dev/disk:ro \ --publish=8081:8081 \ --detach=true \ --name=cadvisor \ google/cadvisor:v0.30.0 --listen_ip="0.0.0.0" --port=8888通过浏览器在本地打开，大概是这样的当然，你可能需要一条防火墙规则 iptables -I INPUT 5 -p tcp --dport 8888 -j ACCEPT而后在prometheus端添加到配置文件中(在最下方添加这样)其中job_name定义为gfc,labels的组gfc_node - job_name: 'gfc' static_configs: - targets: - '10.10.197.3:8888' labels: group: 'gfc_node'打开prometheus端的界面IP:PORT，在status的下拉菜单中点击Targets查看状态，大概如下，可以看到10.10.197.3已经是up状态监控指标当然，系统的核心是prometheus服务器，prometheus基于“拉”机制，可以从配置的目标中删除指标。但是，对于短期工作，它提供了一个用于抓取指标的中间推送网关。它还提供PromDash，一个用于收集数据的可视化仪表板，一个带有查询语言的Expression浏览器，用于简化数据过滤，以及一个AlertManager，用于根据基于警报规则引擎的触发警报发送通知。在前面的prometheus中我们提到，prometheus可以和已有的任何指标进行组合，非常灵活，我简单的组合几个参数进行监控查看关于container_memory_usage_bytes信息，{}中的数据和配置信息中的group组对应，name为容器名称度量的条件就是组和名称，得到下图container_memory_usage_bytes{group="gfc_node",name="nginx_server"}但是我还想知道他5分钟内的信息，这样就需要时间窗口的rate()功能 rate(container_memory_usage_bytes{group="gfc_node",name="nginx_server"}[5m])但是这样还是不好，他的信息太长 ，我只要保留名称和ip。那就需要聚合的功能，如：sum() 函数，加上rate()时间窗口，sum()函数将rate()包含在里面，而后使用by保留值sum(rate(container_memory_usage_bytes{group="gfc_node",name="nginx_server"}[5m])) by (name,instance)grafana这里有grafana的展示，在之前grafana安装过了，grafana与prometheus的 配置非常简单，我们直接导入模板即可，模板下载更多可以参考参考github的cadvisonr和cAdvisor运行时选项,指标信息。prometheus参考

linuxea:prometheus2.3.2安装和配置 prometheus诞生2012，许多公司采用，并且该项目目前非常活跃。具有由度量名称和键/值对标识的时间序列数据的多维数据模型。查询语言非常灵活，可以定义已有的各种信息组合不依赖于分布式存储时间序列通过http拉取模型可以通过中间网关(pushgateway)完成托送模型支持服务发现或者静态配置发现，支持精确的报警，基于查询语言支持多客户端库并且支持多集成，如grafana，安装参考，并允许第三方接入，更多内容移步官网，我们来看这张架构图，你或许会了解的更多其实在prometheus的使用中你会发现，他和传统的zabbix等不相同，我认为你在使用prometheus之前就应该对其进行规划，一个个项目的进行规划监控区域，监控指标等，祝你使用愉快prometheus下载[root@linuxea-VM-Node_10_10_240_143 /data]$ curl -Lk https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz|tar xvz -C /usr/local/ [root@linuxea-VM-Node_10_10_240_143 /usr/local]$ ln -s prometheus-2.3.2.linux-amd64 prometheusprometheus start--web.enable-lifecycle启动--web.enable-lifecycle参数，而后便可以使用reload，大概这样curl -X POST http://10.10.240.143:9090/-/reload[root@linuxea-VM-Node_10_10_240_143 /usr/local/prometheus]$ nohup ./prometheus --web.enable-lifecycle --storage.tsdb.path=/data/prometheus >./nohup.out 2>&1 &