78 lines
2.5 KiB
Markdown
Executable File
78 lines
2.5 KiB
Markdown
Executable File
# Prometheus 部署排错（2026-05-30）
|
||
|
||
## 症状
|
||
|
||
`systemctl start prometheus` 失败，进程 crash 后立即 restart loop。
|
||
|
||
## 排错链（按顺序）
|
||
|
||
```bash
|
||
# 1. 查日志
|
||
journalctl -u prometheus -n 30 | tail -15
|
||
|
||
# 2. 确认 systemd 看的 binary 路径
|
||
systemctl show prometheus -p ExecStart
|
||
|
||
# 3. 找实际运行进程
|
||
ps aux | grep prometheus | grep -v grep
|
||
```
|
||
|
||
## 已知失败模式
|
||
|
||
| 错误 | 根因 | 修复 |
|
||
|------|------|------|
|
||
| `permission denied` queries.active | `/var/lib/prometheus/` 权限不对 | `sudo chown -R muc:muc /var/lib/prometheus/` |
|
||
| `address already in use` (端口 9090) | **残留进程占用端口**（常见于手动 kill 后 systemd 启动） | `sudo pkill -9 prometheus; sleep 2; sudo systemctl start prometheus` |
|
||
| `path doesn't exist` | systemd service 文件里写的 binary 路径和实际不符 | 查 `ExecStart`，看 `/usr/local/bin/prometheus` vs `/usr/bin/prometheus` |
|
||
| 0 rule groups loaded | 进程 crash 后重启，规则文件路径变了 | 先杀掉所有进程，确保用 `systemctl start` 启动 |
|
||
|
||
## 残留进程定位
|
||
|
||
```bash
|
||
# 看谁在占用端口（最准确的第一步）
|
||
ss -tlnp | grep 9090
|
||
|
||
# 看残留进程 cmdline
|
||
cat /proc/<PID>/cmdline | tr '\0' ' '
|
||
|
||
# 常见残留来源：
|
||
# - 用户手动启动的（~/.config/prometheus/ 配置）
|
||
# - systemd 之前拉起但已 orphaned 的实例
|
||
```
|
||
|
||
## 验证步骤（启动成功后）
|
||
|
||
```bash
|
||
# 健康检查
|
||
curl -s http://localhost:9090/-/healthy
|
||
# 期望: Prometheus Server is Healthy.
|
||
|
||
# 确认 targets
|
||
curl -s "http://localhost:9090/api/v1/targets" \
|
||
| python3 -c "import sys,json; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
||
# 期望: prometheus up / zhiyid up
|
||
|
||
# 确认告警规则
|
||
curl -s "http://localhost:9090/api/v1/rules" \
|
||
| python3 -c "import sys,json; d=json.load(sys.stdin); [print(g['name'], len(g['rules']), 'rules') for g in d['data']['groups']]"
|
||
# 期望: zhiyi_alerts 8 rules
|
||
```
|
||
|
||
## 关键配置位置
|
||
|
||
- **二进制**: `/usr/bin/prometheus`（systemd service 写的是 `/usr/local/bin/prometheus`，需建 symlink）
|
||
- **配置**: `/etc/prometheus/prometheus.yml`
|
||
- **规则**: `/etc/prometheus/rules/zhiyi.yml`
|
||
- **数据**: `/var/lib/prometheus/data/`
|
||
- **服务**: `/etc/systemd/system/prometheus.service`
|
||
|
||
## 快速修复脚本
|
||
|
||
```bash
|
||
# 一次性修复所有已知问题后重启
|
||
sudo pkill -9 prometheus 2>/dev/null
|
||
sudo ln -sf /usr/bin/prometheus /usr/local/bin/prometheus
|
||
sudo chown -R muc:muc /var/lib/prometheus/
|
||
sleep 3
|
||
sudo systemctl start prometheus && sleep 5
|
||
``` |