Grafana provides integration with webhooks, email, and more than a dozen external services. Whenever we create an alert, we assign it to a notification channel, along with a message. In our tutorial, we’ll set up two common notification channels: Slack and PagerDuty. Lifecycle of an alert. Hello, I am trying to integrate slack with grafana. When i select notifications slack is not listed. I thought maybe i should use the webhoook option but it doesn’t seem to be working. Everything else i look at references a slack option and i can see a slack.go in the github. How can i install the slack integration?

After moving my blog to the Hugo platform hosted on Netlify I lost the uptime monitoring I had through Jetpack on Wordpress. I had Jetpack connected to my wordpress blog and got a service monitoring if my site was up or down. I would get an email if the site wasn't responding, but not more than that.

New install with Grafana 6.7.0. I am posting using the Slack bot config. Grafana is running in a Docker container. In Grafana logs I see a log entry with no errors: t=2020-03-20T10: lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=0 notification=Slack. Let's deploy Grafana Loki to monitor logs in Grafana and create custom metric based on logs and configure alert accordingly. It propagates and gets notified to the respective Slack channel.

As I've worked quite a bit with Grafana I knew about their Worldping service, but I haven't had any real need to test it before now. Now I had the perfect reason for checking it out as well as look at some alternatives.

Grafana Worldping

I went over to my Grafana installation to check if the worldPing plugin was installed, but it does not ship by default. Luckily installing plugins in Grafana is easy. I headed over to the Grafana website and navigated to the plugins. There you'll find worldPing and the documentation for installing and getting started with it.

Install and configure

First off you install it, the easiest way is to install via the grafana-cli directly:

Or you can download it and unpack the zip file in your plugins directory.

After installing it you enable the plugin. You will be guided through the whole process. You will have to create an account on grafana.com if you haven't got one already and then register the app on the grafana site.

The probing of the endpoints will be run through Grafana and not by your local installation. You'll have your local dashboards connecting to Grafana.com via an API key to pull the probing results.

Grafana Slack Alert

Dashboards

Grafana Slack Community

When you have finished the configuration of the plugin you'll have some dashboards available in your Grafana installation. They'll live under the worldPing menu item in the sidebar menu, but you'll find them in the normal dashboard menu as well.

Summary

Your summary dashboard will give you a quick glance over the health of the endpoint with boxes for DNS, Ping and Http / Https. Further you'll have response graphs by service/protocol where you can see the performance of each probe over time.

Comparison

The comparison dashboard will check your dns performance across the probes configured with single graphs for each probe.

DNS, IP and Web

Grafana Slack Send Image

You'll have individual dashboards for DNS, Ping and Web (ping and web dashboards not shown) which gives a bit more details on the specific service, both overall but also by each probe. These dashboards also have a Map graph which I expect should show the probes plotted on the map, but these doesn't work in my case.

Alerting

Grafana has alerting built in, and I'm using this in my monitoring of my Freenas with an integration that sends alerts to my private Slack.

You can set up alerting on the worldPing dashboards as well, but there is a couple of things to be aware of. First the alerting feature in Grafana doesn't support (in the current release) Template variables in their alert queries.

All of the worldPing dashboards uses variables so if you want to make use of Grafana's alerting feature you'll need to configure the query you want to use for alerting to use specific filters. In my case this isn't a big problem since I only have one site that I want to monitor, but if you have several you would probably need to set up multiple queries with specific values.

After fixing that you can set up your alert query and specify the notification. I've configured mine to send the notification to my private Slack.

This customization leads on to the second thing to be aware of, the worldPing dashboards are delivered by Grafana Labs and are shipped through the plugin installation. When you update the plugin the dashboards will most likely be overwritten by the default ones hence you'll lose your customization. Note that you will be warned about this and given the option to save a copy the edited dashboard.

By this you can also build an 'alerting dashboard' which would allow you to create your specific queries you want to have alerting on. Note that these live in your local Grafana environment which needs to run for the alerting to work.

I've created a small dashboard with some queries that will send a notification to my private Slack if something seems wrong or slow.

Alternatives

If you want a solution that checks if your site is up or down there are lots of services out there i.e. Pingdom, Uptime robot and Uptrends. I haven't tested either of these and can't really vouch for them, but it shows that there are tools available that has a (limited) free tier which might be worth checking out if you don't want to go with Grafana.

Azure Application Insights

You could also use Azure Application Insights which works with any URL. I've set up this in my private Azure subscription for testing.

The setup is easy, you create an Application Insight Resource, create an availability test where you specify the endpoint URL, what kind of test (URL ping test), the locations that should run the test and the criterias and then Azure will run these tests for you. You can also configure alerting to email or a webhook. I've configured both in my example.

After a little while you'll have a dashboard with your results available to you where you also can drill down in to each probe for more details.

With the telemetry data collected I found something interesting, namely some images that loaded slower than others. I did a optimization of my images during the migration process, but seems I forgot some of them.

After optimizing them and publishing, it loaded faster. Even though we are talking milliseconds here, everything counts.

Summary

Grafana Slack Alerts

I really enjoy my current setup with Hugo as the content platform and the hosting on Netlify. Hopefully I will never experience any issues with the availability, but now I at least have some alternatives that can help me monitor it.

I hope this post has shown a few of the alternatives available and given you some ideas on how to configure your own solution.

If you have any questions, comments or suggestions feel free to reach out to me on Twitter!

Related Posts

• Moving to Hugo — April 1, 2019
• vSphere Performance - Telegraf, InfluxDB and Grafana 7 - Configure InfluxDB Authentication and https — August 26, 2020
• vSphere Performance - Telegraf, Influxdb and Grafana 7 - Finished Dashboards for Importing — August 26, 2020
• vSphere Performance - Telegraf, Influxdb and Grafana 7 - Rows and Repeating — August 16, 2020
• vSphere Performance - Telegraf, Influxdb and Grafana 7 - Manual queries — August 16, 2020

Grafana 2.0 shipped with a great feature that enables it to render any graph or panel to a PNG image.

No matter what data source you are using, the PNG image of the Graph will look the same as it does in your browser.

This guide will show you how to install and configure the Hubot-Grafana plugin. This plugin allows you to tell hubot to render any dashboard or graph right from a channel in Slack, Hipchat or Basecamp. The bot will respond with an image of the graph and a link that will take you to the graph.

Amazon S3 Required: The hubot-grafana script will upload the rendered graphs to Amazon S3. Thisis so Hipchat and Slack can show them reliably (they require the image to be publicly available).

What is Hubot?

Hubot is an universal and extensible chat bot that can be used with many chat services and has a huge library of third party plugins that allow you to automate anything from your chat rooms.

Install Hubot

Hubot is very easy to install and host. If you do not already have a bot up and running please read the official Getting Started With Hubot guide.

Install Hubot-Grafana script

In your Hubot project repo install the Grafana plugin using npm:

Edit the file external-scripts.json, and add hubot-grafana to the list of plugins.

Configure

The hubot-grafana plugin requires a number of environment variables to be set in order to work properly.

Grafana server side rendering

The hubot plugin will take advantage of the Grafana server side rendering feature that can render any panel on the server using phantomjs. Grafana ships with a phantomjs binary (Linux only).

To verify that this feature works try the Direct link to rendered image link in the panel share dialog. If you do not get an image when opening this link verify that the required font packages are installed for phantomjs to work.

Grafana API Key

You need to set the environment variable HUBOT_GRAFANA_API_KEY to a Grafana API Key. You can add these from the API Keys page which you find in the Organization dropdown.

Amazon S3

The S3 options are optional but for the images to work properly in services like Slack and Hipchat they need to publicly available. By specifying the S3 options the hubot-grafana script will publish the rendered panel to S3 and it will use that URL when it posts to Slack or Hipchat.

Hubot commands

• hubot graf list
  • Lists the available dashboards
• hubot graf db graphite-carbon-metrics
  • Graph all panels in the dashboard
• hubot graf db graphite-carbon-metrics:3
  • Graph only panel with id 3 of a particular dashboard
• hubot graf db graphite-carbon-metrics:cpu
  • Graph only the panels containing “cpu” (case insensitive) in the title
• hubot graf db graphite-carbon-metrics now-12hr
  • Get a dashboard with a window of 12 hours ago to now
• hubot graf db graphite-carbon-metrics now-24hr now-12hr
  • Get a dashboard with a window of 24 hours ago to 12 hours ago
• hubot graf db graphite-carbon-metrics:3 now-8d now-1d
  • Get only the third panel of a particular dashboard with a window of 8 days ago to yesterday
• hubot graf db graphite-carbon-metrics host=carbon-a
  • Get a templated dashboard with the $host parameter set to carbon-a

Aliases

Some of the hubot commands above can lengthy and you might have to remember the dashboard slug (url id). If you have a few favorite graphs you want to be able check up on often (let’s say from your mobile) you can create hubot command aliases with the hubot script hubot-alias.

Install it:

Now add hubot-alias to the list of plugins in external-scripts.json and restart hubot.

Now you can add an alias like this:

• hubot alias graf-lb=graf db loadbalancers:2 now-20m

Summary

Grafana is going to ship with integrated Slack and Hipchat features some day but you do not have to wait for that. Grafana 2 shipped with a very clever server side rendering feature that can render any panel to a png using phantomjs. The hubot plugin for Grafana is something you can install and use today!