Prometheus Metrics Dashboard

Prometheus metrics can be viewed directly in the Prometheus dashboard.

Access the Prometheus dashboard at port 9090 at the cluster IP address, for example:

http://cluster_ip_address:9090/targets

The following command reports the name of the Prometheus Service that is listening on port 9090:

PROM_SVC=$(kubectl get svc -n monitoring | grep 9090 | grep -v None | awk '{print $1}')
echo $PROM_SVC

The Prometheus dashboard can also be accessed from the node after starting a port-forward. For example:

http://127.0.0.1:9090/targets

The following command uses the service name discovered above:

kubectl port-forward --namespace monitoring --address 0.0.0.0 svc/${PROM_SVC} 9090:9090

The --address option causes the Prometheus dashboard to be accessible from outside the cluster, after the firewall has been opened for that specific port with commands like the following:

sudo firewall-cmd --zone=public --add-port=9090/tcp --permanent
sudo firewall-cmd --reload
You can then access Prometheus dashboard using the host IP and port: 
http://node_ip:9090/targets
The Prometheus dashboard does not require any login credentials. Using the above targets URL takes you to the Targets list, reporting that all configured targets are running as expected. Selecting the Unhealthy filter allows you to see which of the targets is not responding, if any. Selecting one of the show more links displays details of the scrape endpoint.

To confirm access to MATRIXX SNMP statistics, in the Prometheus user interface, click Graph. Enter the name of any MATRIXX SNMP statistic in the Expression field, then click Execute to display the time-series data for the selected statistic.

Note: It may take 30 to 60 seconds from the time Prometheus is started before it collects any statistics. See Prometheus documentation for more information about statistics collection.

The Prometheus dashboard allows direct queries of the Prometheus time series database using the Graph link, such as the http_server_requests_seconds_count metric reported by Spring MVC for all web apps.