Prometheus docke-compose push gateway, client library for PHP, and alert rules based on hour and day

Promethesus push gateway is required for any promethesus client library to push metrics to:

Include the following in your docker-compose.yml services:


Prometheus client library for PHP:
https://github.com/Jimdo/prometheus_client_php

pushgateway:
	image: prom/pushgateway
	ports:
	- 9091:9091

use push gateway example, can use in memory adapter if cron job or task is not a long running one
https://github.com/Jimdo/prometheus_client_php/blob/master/examples/pushgateway.php

configure the push_gateway IP address to be whatever the push gateway IP address and port you used in pushgateway service in docker-compose.yml


Set prometheus alert rules based on time:
https://medium.com/@tom.fawcett/time-of-day-based-notifications-with-prometheus-and-alertmanager-1bf7a23b7695
https://prometheus.io/docs/prometheus/latest/querying/functions/#hour()
https://www.robustperception.io/combining-alert-conditions

Prometheus has hour(), and day_of_month() built in function, just use them in prometheus alert.rules file . For example:
  - alert: high_load
    expr: node_load1 > 0.5 and  hour() >= 16 and day_of_week() == 6  
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} under high load"
      description: "{{ $labels.instance }} of job {{ $labels.job }} is under high load."

Prometheus permission denied && docker useful tips

According to the Dockerfile of prometheus the USER is set to nobody.

So it makes sense that the above dir is not accessible by the service.

So the problem was solved by performing a

chgrp -R nogroup /prometheus_data

https://stackoverflow.com/questions/54232891/docker-compose-directory-permission-errors-on-bind-mount

nogroup is the group analog to the nobody user. It is used for unprivileged processes so that even if something goes wrong the process does not have the permissions to cause any serious damage to an important user or group.

https://unix.stackexchange.com/questions/22520/what-is-nogroup-groups-purpose

useful tips:
// Multiple email in alertmanager
https://stackoverflow.com/questions/47921028/multiple-email-receivers-in-prometheus-alertmanager-to-field

// For docker-compose.yml is always good to add restart always in the each service so when OS rebooted, this service will automatically restart

services:

  xxx:

    image: xxx/xxx

    networks:

      - back    

    ports:

      - 9091:9091

    restart: always  

You can put comma separated email addresses in the
to field.
// Docker rebuild a running container without building rebuilding dependecies

$docker container stop <container_id> / <service_name>

$docker-compose up -d --no-deps --build <service_name>

#<service_name> is the service name specified in docker-compose.yml file

--no-deps - Don't start linked services.

--build - Build images before starting containers.

https://stackoverflow.com/questions/36884991/how-to-rebuild-docker-container-in-docker-compose-yml

// Docker compose build

docker-compose up -d

//  Docker logs

Don't use "docker logs". Use "docker-compose logs flask" to see the logs of that restarting containers. With your options:

docker-compose logs -f --tail=50 

// Go inside docker container

docker exec -it <mycontainer> bash

Prometeus docker compose all in one

Prometheus:
docker compose refer to my own docker repo
https://github.com/iamgoodman/Docker-Compose/tree/master/prometheus


Get started tuotrials:
https://github.com/alexellis/quickstart-prometheus (Docker Compose)
https://github.com/vegasbrianc/prometheus (Docker SWARM)
https://finestructure.co/blog/2016/5/16/monitoring-with-prometheus-grafana-docker-part-1(Tutorial)