All posts by Blogs on Grafana Labs Blog

timeShift(GrafanaBuzz, 1w) Issue 57

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/17/timeshiftgrafanabuzz-1w-issue-57/

Welcome to TimeShift

August is flying by, but hopefully there’ll still be time to enjoy a few more summer evenings. This week we’re sharing the video demoing the new Explore UI in Grafana from last week’s PromCon, monitoring VMWare’s VKE with Prometheus, hosting a blog on a budget and more.

Come across an article you think might be a good fit for an upcoming issue? Suggestions for new content? Contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned Prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


From the Blogosphere

Video: David Kaltschmidt: Exploring your Prometheus Data in Grafana: Last week we shared David’s slides from PromCon 2018, but it’s so much better to actually see it action.

Installing Prometheus and Grafana on VMware Kubernetes Engine: This post details the process of deploying Prometheus as a monitoring framework for Kubernetes, along with Grafana as the visualization layer. Bahubali covers why monitoring K8s is different, building and preparing your cluster, and installing both Prometheus and Grafana.

Collecting DHCP Scope Data with Grafana: Eric wrote a Python script to help him collect aggregated data about groups of DHCP scopes and how his network users were changing. This lets him total up the number of free and used IPs in each range and visualize them on a graph in Grafana.

How I host this blog, CI and tooling: Vik is a budget conscious blogger and developer. In this article he provides a rundown of the infrastructure he uses for his blog and how he keeps it running for $8.53/month.

Graphite Grafana: Metrics Monitoring Made Easy: The first in a series on metrics monitoring made easy, this article gets you started with spinning up a Graphite/Grafana stack. Learn about the components of Graphite, Grafana, and how to get everything installed – the next article will dive into the actual monitoring.

System monitoring with Grafana, InfluxDB et Collectd: Learn about the components of a responsive dashboard system and how to easily deploy it with Docker.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

2018 Sensu Summit | Portland, OR – August 22-23, 2018:
Brian Gann: The Sensu Plugin for Grafana – Brian will be showing a demo of the new Sensu plugin for Grafana on August 22, and conducting a 30 minute Grafana tutorial on the 23rd!

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

Meetup Workshop: Monitoring with Prometheus and Grafana | Belfast, Northern Ireland – September 18, 2018:

If you’re in Belfast, or are going to be in September, this could be a great Meetup to attend. Topics include: Architecture, Prometheus, Alertmanager, Pushgateway, Telegraf, JMX exporter, Grafana, and more!

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

I love a good heatmap. Let us know if you figure out where that super high latency is coming from.


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 56

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/10/timeshiftgrafanabuzz-1w-issue-56/

Welcome to TimeShift

PromCon was held this week in Munich, Germany, and Grafana Labs was excited to both sponsor and speak at this great conference. If you didn’t have a chance to attend, you can watch a recorded livestream of the event and download slides in the blog roundup below.
Tom Wilkie and David Kaltschmidt
Left to right: Tom Wilkie and David Kaltschmidt prepping for their talks

We’d also like to congratulate the Prometheus project for moving from the incubation stage to graduation as an official CNCF project!

Hope you enjoy this week’s roundup of articles and plugin updates. If you see an article you think might be a good fit for an upcoming issue of TimeShift, please contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned Prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

David Kaltschmidt: Exploring your Prometheus Data in Grafana: David has been working tirelessly on the new explore UI for Grafana. We’ll link to the video once it’s available, but for now he’s made his slides available so you can learn about it, and find out how to enable the feature in the latest version to give it a try.

Tom Wilkie: Prometheus Monitoring Mixins – Using Jsonnet to Package Together Dashboards, Alerts, and Exporters: Tom discusses a technique for using Jsonnet for packaging and deploying “Monitoring Mixins” – extensible and customizable combinations of dashboards, alert definitions and exporters. This technique allows you to publish best-practice monitoring configurations alongside your code, and for users to consume it, customize it and stay up to date. Also, check out the video of his presentation.

Using TimescaleDB + Prometheus to Monitor and Troubleshoot CockroachDB: Diana, who recently joined the Timescale team outlines how to combine four open source technologies to create a powerful monitoring stack, and provides a high-level tutorial on getting everything set up.

Inside Fortnite’s Massive Data Analytics Pipeline: Fortnite is a truly massive MMPG with over 125 million users spanning the globe. But pulling together all the servers, databases, and data pipelines to manage 92 million events per minute is no small feat. This article gives you a rundown of their systems, how they keep them running, and what they’d like to optimize in the future.

Managing your Costs on Kubernetes: Cost is often a factor when discussing migrating to a public cloud solution. When the sky’s the limit when it comes to resources, your monthly bill could be at risk of poking through the stratosphere. The article shows you how to use Prometheus, Grafana, and Kubernetes to create a dashboard that can help you get a better visualization of the cost of your applications.

M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus: Rob Skillington, staff software engineer on the Observability team in the Uber New York City provides a deep dive into M3, the metrics platform they’ve been working on for the past few years. In fact, at our very first GrafanaCon in 2015, Matt Mihic gave a talk about M3.

NetEye 3.14 and NetEye 4.2 Release Notes: The latest version of Würth Phoenix’s NetEye Unified Monitoring Solution has been released which includes Grafana 5 and allows you to take advantage of the new dashboard layout engine, folders, provisioning and more.


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


Grafana Plugin Update

Two new panels to show off this week along with an update to the Clickhouse data source plugin. We’ve made it simple to update your plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

NEW PLUGIN

WindRose Panel – This new panel was developed in a joint effort as part of a research project in Taiwan. A wind rose is a graphic tool used by meteorologists to give a view of how wind speed and direction are distributed at a location.

Install

NEW PLUGIN

LinkSmart SensorThings Data Source – This is a new panel that works with any server that implements the OGC SensorThings API. The SensorThings API is a standard for connecting IoT devices, data and applications over the web. An exciting feature of this is the ability to easily integrate into existing Spatial Data Infrastructures (SDI) or Geographic Information Systems (GIS) and means this data source works well with the Worldmap panel.

Check out their live demo dashboard to see what this data source can do.

Install

UPDATED PLUGIN

Clickhouse Data Source – The Clickhouse plugin is being continually improved and the latest release now has support for annotation queries.

Another new feature is the addition of two new macros: $from and $to that can be used in templating queries. If you are using Grafana’s provisioning feature there is now a config example included in the readme for the Clickhouse plugin.

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

London Hashicorp User Group | London, United Kingdom – August 14, 2018:

Tom Wilkie: Monitoring the Hashistack with Prometheus – Prometheus has become the defacto monitoring system for cloud native applications, but for a while was eschewed by the Hashistack in favour of more traditional technologies. Thats all changing: Hashicorp’s project are beginning to export metrics in the native Prometheus format, and many exporters exist to bridge the gap.

In this talk Tom will give a brief introduction to Prometheus, show you how to piece it all together, and give some recommendation on what to monitor and alert on.

Register Now

How to Use Open Source Projects for Performance Monitoring | Webinar – August 15, 2018 – 1pm EDT:

Check out how you can use 5 of the most popular open source projects, InfluxDB, Telegraf, Chronograf, Kapacitor and Grafana, for performance monitoring of your Infrastructure, Application, & Cloud faster, easier, and to scale. In this webinar, Leonard Gram from Grafana, and Margo Schaedel from InfluxData, will provide you with step by step instruction from download & configure, to collecting metrics and building dashboards and alerts.

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:
Brian Gann: The Sensu Plugin for Grafana – Brian will be showing a demo of the new Sensu plugin for Grafana on August 22, and conducting a 30 minute Grafana tutorial on the 23rd!

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

So glad to hear you’re excited about the new explore UI. It’s still pre-alpha, so there’s lots more to come!


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 55

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/03/timeshiftgrafanabuzz-1w-issue-55/

Welcome to TimeShift

This week we announced the dates for GrafanaCon LA and officially opened up the CFP! While we can’t predict the weather, we can be almost certain it will be blizzard-free this time around. Also, if you’re going to be in Munich next week for PromCon, please be sure and say hello!

Hope you enjoy this week’s roundup of articles and plugin updates. If you see an article you think might be a good fit for an upcoming issue of TimeShift, please contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

The RED Method: How to Instrument Your Services: Grafana Labs’ very own Tom Wilkie gave a fantastic talk at GrafanaCon EU on his microservices-oriented monitoring philosophy called the Red Method, and how to use it in combination with other methods to gain insight on the happiness of both your users and machines.

Plaid.com’s Monitoring System for 9600+ Integrations: Plaid.com is a fintech company that integrates with over 9,600 financial institutions. This article discusses how Plaid.com used AWS Kinesis, Prometheus, Alertmanager and Grafana to solve the challenges of scalability and latency to monitor their vast number of integrations.

How to Monitor an Ubuntu Server with Grafana & Prometheus: This installation guide will have you up and running and monitoring your Linux servers in no time.

Efficient IoT with the ESP8266, Protocol Buffers, Grafana, Go, and Kubernetes: Efficient storage and serialization of data can have a large impact on the battery life and performance of your IoT devices. This write up discusses Protocol Buffers for efficient binary data encoding with IoT devices and getting that data into a TSDB and onto a Grafana dashboard.

Build a Homelab Dashboard: Part 6, Grafana Introduction: Part 6 of the Homelab dashboard series walks us through installing and configuring Grafana and building the first dashboard. You may want to start at part 1 to get up to speed.


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


Grafana Plugin Update

This week, Instana updated their plugin to add additional functionality. There are 2 easy ways to update the Instana (or any) plugin – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

UPDATED PLUGIN

Instana Data Source – Version 1.1.0 has been released which adds a configuration that lets users enable the new Instana application model, yielding new/additional entity types and metrics.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

Uh….


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

The RED Method: How to Instrument Your Services

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/

Paul Dix
Tom Wilkie – Grafana Labs

At GrafanaCon EU in March, we had the pleasure of introducing one of Grafana Labs’ newest team members, Tom Wilkie, who joined as VP, Product with the acquisition of Kausal.

And he came bearing gifts: his popular talk about the RED Method of monitoring microservices, which he created in 2015.

The USE Method

In the talk, Tom first went over the USE Method of instrumenting, which has been popularized by Brendan Gregg:

For every resource, monitor:

  • Utilization (% time that the resource was busy)
  • Saturation (amount of work resource has to do, often queue length)
  • Errors (count of error events)

“It’s a way of building a checklist that goes through everything,” he said. “It helps you know what you don’t know.” But he pointed out that this method, while useful, is somewhat abstract. Applying it to memory, for instance, is hard. “Memory utilization is tricky. What is it? Do you count caches toward utilization?” he said. “Saturation of memory is kind of a weird one… And what is a memory error? And how do you count them in Linux?”

The RED Method

Tom then turned to his RED Method, which he created after a new employee asked what his monitoring philosophy was. “The USE Method doesn’t really apply to services; it applies to hardware, network disks, things like this,” Tom said. “We really wanted a microservices-oriented monitoring philosophy, so we came up with the RED Method.”

For every resource, monitor:

  • Rate (the number of requests per second)
  • Errors (the number of those requests that are failing)
  • Duration (the amount of time those requests take)

(See his presentation slides on the Prometheus implementation of the RED Method)

“Everyone should understand the error rate, the request rate, and then some distribution of latency for those requests,” Tom explained. “You model this for every single service in your architecture, and this gives you a nice, consistent view of how your architecture is behaving. Giving this kind of consistency across services allows you to scale your operational team, and allows you to put people on call for code they didn’t write.”

Plus, he pointed out, “The RED Method is a good proxy to how happy your customers will be. If you’ve got a high error rate, that’s basically going through to your users and they’re getting page load errors. If you’ve got a high duration, your website is slow. So these are really good metrics for building meaningful alerts and measuring your SLA.”

The Four Golden Signals

Finally, Tom looked at a third method: The Four Golden Signals, which is from Google’s SRE book.

For each service, monitor:

  • Latency (time taken to serve a request)
  • Traffic (how much demand is placed on your system)
  • Errors (rate of requests that are failing)
  • Saturation (how “full” your service is)

This is basically the same as the RED Method, but includes saturation. He explained one approach to measuring saturation: “With kube-state-metrics, a little job you run on your Kubernetes cluster that scrapes the Kubernetes API and exports really interesting metadata about your jobs and services and pods and so on, you can compare the amount of CPU a service is using against its quota. Like how much it should be using, or how much is it allowed to use, as a proportion between 1 and 0. This gives you a measure of how ‘full’ your service is, or some proxy for how full your service is at least. And this is super useful because you could for instance build an alert on this.”

Two Sides of the Same Coin

Tom recommended using the USE and RED Methods together. “It’s like the RED Method is about caring about your users and how happy they are,” Tom said, “and the USE Method is about caring about your machines and how happy they are. It’s really just two different views on the same system. They’re complimentary.”

Watch Tom’s full talk in the video below.

Video: The RED Method: How to Instrument Your Services

timeShift(GrafanaBuzz, 1w) Issue 54

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/27/timeshiftgrafanabuzz-1w-issue-54/

Welcome to TimeShift

This week’s big news is that Grafana v5.2.2 was released and includes fixes for Prometheus graphs, dashboard links, loading external plugins, SQL connection leaks, and more. Also, check out our new Grafana Flux plugin and monitor temperature and forecasts using Grafana and the openweathermap API.

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

How the New Influx Query Engine Was Designed—And How to Use It With Grafana: Learn about the design of InfluxData’s new functional query engine Flux in a recap from GrafanaCon EU, and check out the new Flux plugin for Grafana to start using it today!

How to Collect Docker Daemon Metrics: Docker ships with native Prometheus integration – the Docker daemon can generate metrics about it and let you scrape that. Check out how to gather Docker daemon metrics in Linux and MacOS.

Grafana + Prometheus = Awesome: A walkthrough of setting up and configuring a Prometheus/Grafana stack with a few example queries to get you started with your first dashboard.

Playing with Grafana and weather APIs: After receiving a Beewi temperature sensor, Gonzalo wanted to visualize the data from the openweathermap API using Grafana. This article shows you how to collect the data and visualize temperature, UV index, current weather conditions, and forecast in a Grafana dashboard.

Monitoring temperature and humidity with a Raspberry Pi 3, DHT22 sensor, InfluxDB and Grafana: In a second weather related article, Sam needed to track the temperature of his garage for a very important reason – brewing beer. He shares his equipment, setup, scripts and how he intends to improve the setup in the future.

Integration for ServiceNow table API and Grafana: In a follow up on the article about integrating ServiceNow with Grafana, we learn about the newly re-written snow-grafana-proxy plugin, along with the new configuration options, parameters and command line options.


Grafana Plugins

OpenNMS updated their Helm app this week. Depending on your Grafana installation, there are 2 ways to update your plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

UPDATED PLUGIN

OpenNMS Helm App – Version 2.0.0 has been released. It includes a new datasource for querying flow data from OpenNMS and requires Grafana 5.x or greater.

Other new features are:

  • Support for “fallback” attributes to the performance datasource.
  • The ability to configure query timeouts for all of the datasources in the OpenNMS helm plugin.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

This is awesome! Thanks for sharing.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

How the New Influx Query Engine Was Designed—And How to Use It With Grafana

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/26/how-the-new-influx-query-engine-was-designedand-how-to-use-it-with-grafana/

Paul Dix
Left to right: Paul Dix – InfluxData, David Kaltschmidt – Grafana Labs

Flux, the long-awaited new functional query processing engine for InfluxDB, has finally landed. If you’re curious to learn more about the hows and whys of its design, check out this GrafanaCon EU session with InfluxData Cofounder and CTO Paul Dix. Also we’d like to share a recent presentation from David Kaltschmidt, Director, UX for Grafana Labs on the new Flux support in Grafana!

Video: Full Flux Presentation from GrafanaCon:

Download the presentation slides.

Flux + Grafana = ❤

Given the excitement about Flux, we just released a plugin that adds Flux support to Grafana. “Flux is powerful stuff, and you can use it with Grafana today,” says David Kaltschmidt, Director of UX for Grafana Labs, who recently did a live demo of the plugin. Simply download the new plugin from grafana.com to install it. The one requirement is that “you need a super recent version of Grafana (v5.1+), because we expanded the data source plugin model,” he says. “The plugin has syntax highlighting, tag completion, raw table preview, and inline function documentation, which I think is really helpful especially if you’re starting a new language.”

Video: Flux Support in Grafana:

Download the Flux Grafana Plugin Now


WTF(lux)

Dix explained the evolution of his thinking about how best to work with time series data. The first version of Flux (then known as IFQL – Influx Functional Query Language) was based on the same tech that he’d used for errplane, a SaaS server monitoring product, which included Rest API. “The thing that I learned with the Rest API was people understood SQL as a query language for working with their data,” said Dix. “When I kickstarted InfluxDB [in November 2013], the innovation that we introduced was basically a language that looks kind of like SQL.”

To SQL or Not to SQL

A year later, Dix began giving talks with different user groups, and he polled the audience: “Does it make more sense to organize your metrics as a hierarchy, like Graphite does, or does it make more sense to organize it as tags?” Tags won out, and that was the new API introduced in the next version. But on the second question, whether SQL or a functional language was superior for working with this kind of data, the rooms were evenly split. “While I thought functional was the way to do it, I didn’t want to make the change because Influx was gaining in popularity and a lot of people told me the reason we love the project is because of the SQL query language. So basically I was too afraid to switch.”

Fast forward to today, and “there are a bunch of feature requests that people have,” said Dix. “We want to deliver those features, but we were kind of hemmed in by this weird SQL query language that we had created.” And with the new Flux, all of these features are included. “We were able to implement these features in a few months, that we hadn’t been able to get to in two years,” Dix said. “The problem was the query language that we had was really really hard to improve and to change.”

The same could be said with TICKscript, the language of InfluxDB’s monitoring piece, Kapacitor. It was designed to be more functional, but “It is really hard to debug, and it has a super steep learning curve because this doesn’t look like anything anybody’s familiar with,” Dix said. “We essentially created a second language for working with time series data in our platform.”

At the beginning of 2017, Dix decided a rethink was in order: “If I had to do everything brand new, if I had to start Influx today, what would I do?”

One Language to Unite them All

The answer turned out to be pretty simple: “Kapacitor is just background processing, but the truth is, it is the query engine,” he said. “If you’re doing a batch job, that looks exactly like a query a user would submit to a database. InfluxDB is batch interactive. It’s users querying the database. So basically when I thought about 2.0 and what I wanted to accomplish with it, I wanted to unify the API and the language so there’s just one thing you have to learn: one language to unite them all.”

Dix’s solution should also help increase feature velocity, enabled by the fact that the storage will be decoupled from the compute. “We can deploy these features frequently, and the risk of shipping a code update in the query language is not nearly as high because you know your data is safe,” he said. “One feature requests we have frequently now is that people want it to be multi-tenant. The nice thing about having these query processors be stateless systems is you can containerize them and you can put them in a lockbox and say this person can’t mess with this person.”

IFQL Flux

The decision to rebrand IFQL as Flux came in large part “because it’s selling it short to call it a query language,” said Dix. “I think of it more as a language for working with data.”

As for the design philosophy, Dix said it’s this: “You want a user interface for the masses. I know the hardcore people want to write their queries, but my theory is most people don’t want to write queries. They want a point-and-click UI, and they want to see their data.”

Other Guiding Principles for Flux:
  • It’s optimized for readability.
  • It’s flexible.
  • It’s easy for people to contribute to it.
  • Code sharing and reuse will be enabled.
Video Demo Features:
  • More complex windowing behavior enabled than before.
  • Anonymous functions, and pipe forward operators to make clear that data is being piped in from one function to another.
  • Named parameters only, no provisional parameters; for added flexibility and readability.
  • A record looks like a flat object or a row in a table. Anything that starts with an underscore is reserved for system attributes; all data within an InfluxDB would have _measurement and _field.
  • From a user perspective, functions operate in isolation. They have an input and an output.
  • Order doesn’t matter in terms of optimization, but it matters for semantics: “The planner should optimize the query so that it runs in the most efficient way.”
  • There are closures in the language.
  • It’s easy to add functions in the language: “You can define an entire function in three pages’ worth of code.”
  • It will have imports and namespaces.
  • You can query based on metadata in a specific time range.

As Dix put it, “Ideally, I want people to build the language around their use case, so if you have common boilerplate, you could do something like define a function where you just say, ‘Okay, I know the boilerplate is: I’m always asking for a database, I’m always going to ask for a measurement and a field. So I can just do that. And I can pipe the result of that function to the other things.’”

Now that the language has been formalized, it is shipping with the enterprise version of InfluxDB, and there will be commercial support coming soon. “This is a lot of work for the API 2.0 effort,” said Dix. “Flux is the first bit of that.”

Check out Paul’s update to #Fluxlang from InfluxDays London.

timeShift(GrafanaBuzz, 1w) Issue 53

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/20/timeshiftgrafanabuzz-1w-issue-53/

Welcome to TimeShift

This week we’re showing off 2 more brand-new plugins, plus a case study showing how PingCAP troubleshoots more than a thousand metrics from their TiDB clusters. Also, Tweet of the Week returns with a graph celebrating France’s World Cup victory. Enjoy!

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.1

New Features

See everything new in Grafana v5.2.1.

Download Grafana 5.2.1 Now


From the Blogosphere

How Grafana Helps PingCAP Troubleshoot TiDB Deployments: A case study on how PingCAP diagnoses and troubleshoots issues from a thousand metrics on their open source TiDB clusters using Grafana.

Azure Log into Grafana using Stream Analytics jobs: Learn how to export data using Azure Stream Analytics jobs in Azure SQL and visualize the data in Grafana.

How to Monitor ArangoDB using collectd, Prometheus and Grafana: The folks from ArangoDB have put together a tutorial on how to install, configure and visualize metrics for ArangoDB.

Room temperature monitoring with Philips Hue, Node-RED and Grafana: This how-to guide explains how to use a Philips Hue Smart Home lighting system to measure the temperature in a room and visualize the data in a Grafana Dashboard.

Visualizing Data using Grafana: We’ve seen an explosion of interest in using Grafana to visualize data for home automation. This tutorial shows you how to install and run Grafana on a Raspberry Pi with InfluxDB, and Home Assistant.


Grafana Plugins

This week we have an update to the Clickhouse data source and 2 new plugins to share. Depending on your installation, there are 2 ways to update plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

NEW PLUGIN

Akumuli Data source – The Akumuli timeseries database now has a published Grafana datasource plugin. It contains a full-featured query editor and supports templating.

Install

NEW PLUGIN

TrafficLight Panel – A new status panel with traffic lights has been published. The TrafficLight Panel allows you to add a traffic light per query and place it on an image background.

Install

UPDATED PLUGIN

Clickhouse Data source – Added a performance improvement that optimizes memory use for range time series. See this PR for more details.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

Laissez le bon temps rouler!


We’re Hiring!

We’re looking for passionate and talented engineers for positions in New York/Stockholm and remote! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 52

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/13/timeshiftgrafanabuzz-1w-issue-52/

Welcome to TimeShift

It’s good to be back from our short holiday break. This week we have articles on the world’s fastest internet (with a shoutout to Prometheus and Grafana), visualizing real-time and historic weather data, making teams more autonomous, and Grafana + Prometheus + Postgres + TimescaleDB. Hope you have a happy and safe Friday the 13th – see you next week!

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.1

New Features

See everything new in Grafana v5.2.1.

Download Grafana 5.2.1 Now


From the Blogosphere

Ballerina Makeover with Grafana: In this guest blog post, the folks from cloud-native programming language Ballerina show you how you can easily visualize metrics from a Ballerina service with Grafana, walking you step by step through the installation and configuration of the components.

How Grafana gives our teams more autonomy with their data: As the saying goes, “Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime,” Vitor shows how using Grafana has empowered teams across Intuit to get hands on and explore their own data, and provides a step by step guide to starting this in your organization.

World’s Fastest Internet – 1.6 TERABITS per Second: Get an inside look at the world’s fastest internet connection which powered a recent LAN party in Sweden (monitored by Prometheus and Grafana)!

Visualize your data with Grafana: Learn how to visualize some real-time and historical weather data in a Grafana dashboard using the nio platform, Elasticsearch and the OpenWeatherMap API. Very cool stuff.

Uniting SQL and NoSQL for Monitoring: Why PostgreSQL is the ultimate data store for Prometheus: This article shows you how to use Prometheus, PostgreSQL + TimescaleDB, and Grafana for storing, analyzing, and visualizing metrics. If you’re not familiar with TimescaleDB, they spoke at our last GrafanaCon – checkout Erik’s talk.

Introducing Pagination for Grafana Scripted Dashboards: The new release of Würth Phoenix’s NetEye 4 takes advantage of Grafana’s scripted dashboards, but adds some new functionality. Sometimes when you have a large number of panels on a single dashboard it can be slow to load. Würth Phoenix has added the concept of pagination to their default analytics dashboard and they show you how they did it.

Highly Available Grafana running on AWS Fargate and RDS Aurora: Set up a fully operational Grafana that’s load balanced and backed by Amazon Aurora.


We’re Hiring!

We’re looking for passionate and talented engineers for positions in New York/Stockholm and remote! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

TimeSeries Boston | Boston, MA – July 17, 2018:

Jacob Lisi: What does Kubernetes Look Like? Performance Monitoring & Visualization with Grafana – Monitoring Kubernetes is vital to understanding the health and performance of a cluster, but which metrics are most important to add to your dashboards and alert on? Jacob will discuss how to most effectively monitor and visualize your Kubernetes cluster using the Grafana Kubernetes plugin and PromQL. Some of the topics of discussion include(1) how and what to collect when monitoring Kubernetes, (2) how to bring your Grafana dashboards to the next level by using Kubernetes as a data-source, and (3) what to do when managing multiple clusters. All of these topics and more will be discussed to help people get the most out of their Kubernetes monitoring.

RSVP

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

Guest Blog Post: Ballerina Makeover with Grafana

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/12/guest-blog-post-ballerina-makeover-with-grafana/

In this guest blog post from the folks at Ballerina, Anjana shows you how you can easily visualize metrics from a Ballerina service with Grafana, walking you step by step through the installation and configuration of the components. They’ve also extended an offer for a free ticket to their upcoming Ballerinacon to the Grafana community.

Ballerina is a cloud-native programming language gaining adoption in the microservice development world. Ballerina is a general purpose, concurrent, transactional, statically and strongly typed programming language with both textual and graphical syntaxes. It brings fundamental concepts, ideas and tools of distributed system integration into the language. These include distributed transactions, reliable messaging, stream processing, workflows and container management platforms. Ballerina is an open source project under the Apache 2.0 license and is developed by the Ballerina community.

Ballerina is observable by design; where this aspect is baked into the language runtime itself. The idea is to allow users to understand the state of your deployment at any given time, so you can predict the behavior and prepare any corrective actions for the system if necessary. Monitoring the metrics of your system is a prominent aspect of this, and having this data, without having a suitable way to visualize it, is almost useless. This is why Ballerina has chosen Grafana for this task, being the open source industry leader in analytics visualization.

Grafana is a feature rich time series data visualization and dashboarding tool that can visualize data from over 40 different sources, and through its plugin architecture can be extended to visualize data from even more data sources. Ballerina uses Prometheus for collecting the metrics, and we can run queries for additional time series operations and data transformations. And of course, Grafana has Prometheus support as a data source by default. I will be going through the steps on how to monitor and visualize a Ballerina service using Grafana.

Implementing the Ballerina Service

Let’s look at how to write a simple Ballerina service, so we can monitor its invocation metrics.
First, you’ll need to download and install Ballerina.

Create a Ballerina service as shown below, and save it as “hello-service.bal”.

import ballerina/http;
import ballerina/log;
service<http:Service> hello bind { port: 9090 } {
    sayHello(endpoint caller, http:Request req) {
        http:Response res = new;
        res.setPayload("Hello, World!");
        caller->respond(res) but { error e => log:printError(
                           "Error sending response", err = e) };
    }
}

The service is run with the following command:- $ ballerina run hello-world.bal --observe

Note: the “–observe” switch, which is used to enable the observability features of Ballerina, where, at the moment, it is disabled by default.

At this point, the service will start up, and will be ready to accept requests at http://localhost:9090/hello/sayHello. You should see the following output in the console:-

Figure 1 - Starting the Ballerina service

Starting the Ballerina service

You will notice that, it also logs the Prometheus HTTP endpoint, which is an internal service that is brought up to serve the Ballerina metrics to a Prometheus server. The next section will cover how the data is pulled by Prometheus.

Setting up Metrics Collection

The metrics collection is done by Prometheus. For a quick installation, we will use Docker to run Prometheus. As a prerequisite, make sure Docker is installed on your machine.

The first step is to create a “prometheus.yml” file at /tmp, and add the following content:-

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['IP:9797']

Here, “IP” should be your IP address of the network interface; do not put the loopback address here. Basically, with this configuration, Prometheus will talk to the metrics endpoint that was initiated earlier by the Ballerina service. The above prometheus.yml is used to do a volume mount with the Docker instance, so Prometheus will pick that as the configuration file in the container.

Now, execute the following command to run Prometheus in Docker:- $ docker run -p 19090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Figure 2

Running Prometheus in Docker

Now, Prometheus will poll for metrics data from the Ballerina service and store in its time series data store.

Visualizing Metrics with Grafana

Now that we have a queryable data source with Prometheus, we can use Grafana to visualize our metrics in a dashboard.

We will be installing and running Grafana using the following command:- $ docker run -d --name=grafana -p 3000:3000 grafana/grafana

Figure 3

Install and run Grafana

Now, we can go to http://localhost:3000/ to access the Grafana dashboard. From here, you can log in to the system with the default username/password: admin:admin.

Next, we will add a new data source to Grafana, with the following settings:-

Figure 4

Settings for your Prometheus data source

After the data source is saved, the already available Ballerina dashboard can be found at https://grafana.com/dashboards/5841. In order to add it to Grafana, Click [+] -> Import -> 5841 -> Load.
After the dashboard is added, you will be able to immediately see the metrics being visualized by the Grafana dashboard.

Visualizations in the default dashboard

HTTP Service / Client Metrics

  • Service and client level statistics such as request/min, errors/min, error%, HTTP error codes distribution etc…

SQL Client Metrics

  • RDBMS connector related metrics, such as SQL query execution throughput, response time percentiles, SQL error distribution etc…

Ballerina System Metrics

  • Mainly contains the Ballerina scheduler based metrics, such as running worker count, waiting for response worker count etc..
  • In order to generate Ballerina scheduler metrics, the Ballerina environment variable “b7a.runtime.scheduler.statistics” should be set to “true”; $ ballerina run hello.bal --observe -e b7a.runtime.scheduler.statistics=true

Figure 5

The metrics visualization of our hello-world service being invoked

Figure 6

Ballerina scheduler statistics while the hello-world service is invoked repeatedly

Figure 7
Figure 8

Preview of the Grafana dashboard panels, if we were to have any HTTP client operations and SQL operations to be executed, respectively

So we can see that, using Grafana, we were able to create a beautiful and elegant dashboard for Ballerina metrics without any hassle. It also has the functionality of exposing each data source’s unique querying features, to optimally use the features of the backing TSDB, and rich query templating support in the dashboards. All of these features enabled us to create a functional and robust metrics visualization solution for the Ballerina runtime.

Ballerinacon

Date: 2018-07-18

Full day event in San Francisco and also streamed globally. Get intense training on the best practices of microservice development, resiliency, integration, Docker and Kubernetes deployment, Service Meshes, serverless, test-driven microservice development, lifecycle management, observability (including Grafana!), and security.

Grafana community members can attend free: use coupon code BalCon-Grafana when buying the ticket and get your 100% discount.

GrafanaCon Recap: The State of TSDB

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/03/grafanacon-recap-the-state-of-tsdb/

From left to right: Erik Nordstrom – Timescale, Dan Cech – Graphite, Paul Dix – InfluxDB, Tom Wiklie – Prometheus
At GrafanaCon EU, we gathered representatives of the Graphite, Prometheus, InfluxDB, and Timescale projects in the hopes of starting a spirited conversation about the current state of Time Series Databases. They didn’t disappoint! Here are a few highlights from the TSDB panel featuring Erik Nordstrom from Timescale, Dan Cech from Graphite, Paul Dix from InfluxDB, and Tom Wilkie from Prometheus, and moderated by Grafana Labs co-founder and CEO Raj Dutt.

timeShift(GrafanaBuzz, 1w) Issue 51

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/29/timeshiftgrafanabuzz-1w-issue-51/

Welcome to TimeShift With summer holidays approaching and upcoming vacations on the calendar, TimeShift is going to take next week off. We look forward to returning the week after next.
This week we’re proud to announce Grafana v5.2.1 stable is now available! Learn more and download the new stable release below, install the latest plugin updates, and check out this week’s collection of articles from around the web. Enjoy!

Evolution of Telemetry at Bloomberg

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/28/evolution-of-telemetry-at-bloomberg/

From left to right: Sean Hanson and Stig Sorensen
Monitoring in the wild west With 5,000 engineers, 325,000 customers running its software, 2 data centers fully owned and operated, 200 node sites around the world, and a diverse architecture developed over almost four decades, Bloomberg has that many reasons to be committed to monitoring. And as Stig Sorensen, Bloomberg’s Head of Telemetry and Software Developer Sean Hanson described during their GrafanaCon EU talk, the company has taken a long, winding and ongoing journey to achieve the robust telemetry they have now.

Grafana v5.2 Released

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/27/grafana-v5.2-released/

v5.2 Stable Release The recent 5.1 release focused on smoothing out the rough edges from the 5.0 release. The major highlights of the Grafana 5.2 release are these two much-requested, long-awaited features:
Elasticsearch alerting Native Grafana builds for ARM Download Grafana 5.2 Now
Release Highlights There are two new major features included, Elasticsearch alerting and native Grafana builds for ARM.
Another highlight is that Grafana’s docker container now supports Docker secrets which should make it more secure when providing sensitive configuration data to Grafana.

timeShift(GrafanaBuzz, 1w) Issue 50

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/22/timeshiftgrafanabuzz-1w-issue-50/

Welcome to TimeShift TimeShift is turning 1 year old! I really hope you’ve enjoyed reading these weekly roundups as much as I’ve enjoyed writing them. This week we have news on the new Grafana v5.2.0-beta3 release, a bunch of plugin updates to share, and your regular dose of recent blog posts.
Have an article you’d like included in an upcoming TimeShift? Contact Us.

Latest Beta Release: Grafana 5.2.0-beta3 New Features Dashboard: Import dashboard to folder #10796 Minor Updates Build: All rpm packages should be signed #12359 Permissions: Important security fix for API keys with viewer role #12343 Dashboard: Fix so panel titles don’t wrap #11074 Dashboard: Prevent double-click when saving dashboard #11963 Dashboard: Autofocus the add-panel search filter #12189 thx @ryantxu Units: W/m2 (energy), l/h (flow) and kPa (pressure) #11233, thx @flopp999 Units: Litre/min (flow) and milliLitre/min (flow) #12282, thx @flopp999 Alerting: Fix mobile notifications for Microsoft Teams alert notifier #11484, thx @manacker Influxdb: Add support for mode function #12286 Cloudwatch: Fixes panic caused by bad timerange settings #12199 Auth Proxy: Whitelist proxy IP address instead of client IP address #10707 User Management: Make sure that a user always has a current org assigned #11076 Snapshots: Fix: annotations not properly extracted leading to incorrect rendering of annotations #12278 LDAP: Allow use of DN in group_search_filter_user_attribute and member_of #3132, thx @mmolnar Graph: Fix legend decimals precision calculation #11792 Dashboard: Make sure to process panels in collapsed rows when exporting dashboard #12256 Please try the new beta release out and let us know what you think.

Running a Power Plant with Grafana

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/20/running-a-power-plant-with-grafana/

GrafanaCon Recap: Running a Power Plant with Grafana A water and energy innovation company founded in 2005, Natel Energy builds hydropower turbines and designs resilient and distributed hydropower systems. In his talk at GrafanaCon EU, Natel Developer Ryan McKinley gave us a fascinating look at how the company is using Grafana to help run these next-generation power plants.
“It’s a different model for turbine than you’re used to seeing,” McKinley said.

timeShift(GrafanaBuzz, 1w) Issue 49

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/15/timeshiftgrafanabuzz-1w-issue-49/

Welcome to TimeShift This week’s roundup highlights videos from Monitorama, making your own Grafana Plugins, NetApp’s fully open source monitoring stack and more. Have an article you’d like to have included in an upcoming TimeShift? Contact Us.

Latest Beta Release: Grafana 5.2.0-beta1 In addition to Elasticsearch alerting, we have a number of other enhancements including: Downloads now available for ARM, Mac platforms Improved Docker image Prometheus improvements Discord alert notifications Dashboard and panel improvements Please try it out and let us know what you think.

timeShift(GrafanaBuzz, 1w) Issue 48

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/08/timeshiftgrafanabuzz-1w-issue-48/

Welcome to TimeShift We were in Portland this week attending Monitorama – one of our favorite annual conferences. We got the chance to catch up with old friends, make some new ones, and be part of an amazing community of passionate data and monitoring aficionados. Looking forward to Monitorama AMS in September!
Also this week we released Grafana v5.2.0-beta1 and… Elasticsearch alerting has arrived! Download it today and let us know what you think.

timeShift(GrafanaBuzz, 1w) Issue 47

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/06/01/timeshiftgrafanabuzz-1w-issue-47/

Welcome to TimeShift We cover a lot of ground this week with posts on general monitoring principles, home automation, how CERN uses open source projects in their particle acceleration work, and more. Have an article you’d like highlighted here? Get in touch.
We’re excited to be a sponsor of Monitorama PDX June 4-6. If you’re going, please be sure and say hello! Latest Release: Grafana 5.1.3 This latest point release fixes a scrolling issue that was reported in Firefox.

Monitoring with Azure and Grafana

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/05/31/monitoring-with-azure-and-grafana/

Monitoring with Azure and Grafana What is whitebox monitoring?
Why do we monitor our systems?
What is the Azure Monitor plugin and how can I use it to monitor my Azure resources?
Recently, I spoke at Swetugg 2018, a .NET conference held in Stockholm, Sweden to answer these questions. In this video you’ll learn some basic monitoring principles, some of the tools we use to monitor our systems, and get an inside look at the new Azure Monitor plugin for Grafana.

timeShift(GrafanaBuzz, 1w) Issue 46

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/05/24/timeshiftgrafanabuzz-1w-issue-46/

Welcome to TimeShift The day has finally arrived; GDPR is officially in effect! These new policies are meant to provide more transparency about the data companies collect on users, and how that data is used. I for one am just excited that the onslaught of "We’ve updated our privacy policy" emails arriving in my pummeled inbox is nearing its end.
Grafana Labs is no exception. We encourage you to check out our privacy policy, and if you have any questions, feel free to contact us at [email protected]