Add prometheus

Metrics-Monitoring/prometheus/grafana_on_docker.md

@@ -0,0 +1,43 @@
+# Running the Container
+Reference: [Using Docker](http://docs.grafana.org/installation/docker/)
+
+## Create Storage
+You should probably create some storage:
+```bash
+docker volume create grafana-storage
+```
+
+## Docker Run
+```bash
+docker run \
+--restart=unless-stopped \
+-d \
+-p 3000:3000 \
+--name grafana \
+-v grafana-storage:/var/lib/grafana \
+-e "GF_SECURITY_ADMIN_USER=admin" \
+-e "GF_SECURITY_ADMIN_PASSWORD=MyGrafanaPassword" \
+-e "GF_USERS_ALLOW_SIGN_UP=false" \
+grafana/grafana
+```
+
+# Docker Compose
+```bash
+ grafana:
+    image: grafana/grafana:5.2.1
+    container_name: grafana
+    restart: unless-stopped
+    ports:
+      - "3000:3000"
+    networks:
+      - backend
+    links:
+      - prometheus
+    volumes:
+      - grafana-storage:/var/lib/grafana
+      - /var/batch-shipyard/grafana/provisioning:/etc/grafana/provisioning
+    environment:
+      - GF_SECURITY_ADMIN_USER={GRAFANA_ADMIN_USER}
+      - GF_SECURITY_ADMIN_PASSWORD={GRAFANA_ADMIN_PASSWORD}
+      - GF_USERS_ALLOW_SIGN_UP=false
+```

Metrics-Monitoring/prometheus/installing_node_exporter.md

@@ -0,0 +1,35 @@
+# Introduction
+Node Exporter is a handy tool that with gather Linux metrics fom an OS for Prometheus to grok.
+
+# Installing
+Download the latest release from:
+/~https://github.com/prometheus/node_exporter/releases
+
+```bash
+tar -zxvf node_exporter-*
+mv node_exporter*/node_exporter /usr/sbin/node_exporter
+```
+
+Create a systemd file for it:
+`/etc/systemd/system/node-exporter.service`
+```
+[Unit]
+Description=Node Exporter
+[Service]
+Restart=always
+EnvironmentFile=/etc/node_exporter.conf
+ExecStart=/usr/sbin/node_exporter $OPTIONS
+[Install]
+WantedBy=multi-user.target
+```
+Create a conf file for it, be sure to customize the collectors listed on the github page:
+`/etc/node_exporter.conf
+--no-collector.textfile --no-collector.wifi --no-collector.zfs --no-collector.mdadm --no-collector.xfs`
+
+Enable and start it:
+```bash
+systemctl daemon-reload
+systemctl start node-exporter
+systemctl enable node-exporter
+systemctl --no-pager status node-exporter
+```

Metrics-Monitoring/prometheus/prometheus_on_docker.md

@@ -0,0 +1,60 @@
+# Running it as a container
+```bash
+mkdir /conf
+mkdir /promethus
+chmod 755 /prometheus
+```
+Create your `/conf/prometheus.yml`. A simple one without alerting
+```bash
+global:
+  scrape_interval: 10s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: 'dummy'
+    file_sd_configs:
+      - files:
+        - '/conf/*.json'
+```
+## Create Storage
+```bash
+docker volume create prometheus-storage
+```
+
+## Run docker
+
+Finish setting up prometheus. You can run the below as a script
+```bash
+docker run \
+--restart=unless-stopped \
+--name prometheus \
+-d \
+-v/conf:/conf \
+-vprometheus-storage:/prometheus \
+-p9090:9090 \
+--entrypoint='/bin/prometheus' \
+quay.io/prometheus/prometheus \
+'--config.file=/conf/prometheus.yml' \
+'--storage.tsdb.path=/prometheus' \
+'--web.console.libraries=/usr/share/prometheus/console_libraries' \
+'--web.console.templates=/usr/share/prometheus/consoles'
+```
+From here you should be able to to the host at port 9090 and query.
+
+# Some basic queries
+[Query Basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)  
+[Query Examples](https://prometheus.io/docs/prometheus/latest/querying/examples/)  
+
+Show all  metrics:
+```
+{__name__=~".+"}
+```
+Show free MB on filesytem:
+```
+node_filesystem_free_bytes/(1024^2)
+```
+CPU Metric:
+[Understanding CPU Usage](https://www.robustperception.io/understanding-machine-cpu-usage)
+```
+irate(node_cpu_seconds_total{job="kznode01", mode="user"}[5m])
+```