All posts by Blogs on Grafana Labs Blog

timeShift(GrafanaBuzz, 1w) Issue 64

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/10/12/timeshiftgrafanabuzz-1w-issue-64/

Welcome to TimeShift

This week the Grafana Labs team was busy prepping for the Grafana v5.3 stable release! Grafana v5.3 adds Stackdriver as a core datasource, a new graphical query editor for Postgres, enhancements to TV and kiosk mode, and a lot more. See the release section for a list of all the new features.

GrafanaCon

The GrafanaCon CFP window is scheduled to close Oct 15. Be sure and submit your talk before it’s too late.
Also, don’t forget to grab an early bird ticket to GrafanaCon before they’re sold out!

Know of an article that might be a good fit for an upcoming issue? Contact us!


GrafanaCon LA
Early Bird Tickets Now on Sale

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem. Learn about Grafana and new/upcoming features and projects in the broader ecosystem like Prometheus, Graphite, InfluxDB, Kubernetes, and more.

Get Your Ticket Now


Latest Stable Release: Grafana v5.3

Updates in this stable release
  • Stackdriver: Filter wildcards and regex matching are not yet supported #13495
  • Stackdriver: Support the distribution metric type for heatmaps #13559
  • Cloudwatch: Automatically set graph yaxis unit #13575, thx @mtanda

See everything new in Grafana v5.3.

Download Grafana v5.3 Now


From the Blogosphere

Grafana v5.3 Released: Carl Bergquist wrote an article that dives into the new features of Grafana v5.3 and how to get started using them.

Building a more reliable infrastructure with new Stackdriver tools and partners: This post from the Google Cloud team discusses Stackdriver and announces our partnership developing the Stackdriver datasource for Grafana.

Setting up Prometheus/Grafana Monitoring Dashboard for ForgeRock IDM: This installation walkthrough shows you how to visualize data from ForgeRock Identity Management using Prometheus and Grafana.

Monitor using Grafana and Prometheus.: In the first in a new series on monitoring with Grafana and Prometheus, Nitesh discusses basic components you might need to get a simple monitoring system for your servers up and running.


WEBINAR – Oct 16, 1pm EDT

Making the most out of your upgrade to Graphite 1.1

In this webinar, Graphite project maintainer Dan Cech will provide some history of the project, outline new features of Graphite 1.x, and show you how to best make use of them. Good for Graphite users of all levels, whether you’ve already upgraded to 1.1 or are still on an older version. Hope you can join us!

Register Now


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now

InfluxDays 2018 | San Francisco – November 7-8, 2018:

Matt Toback: Optimizing the Grafana Platform for Flux – Flux, the new InfluxData Data Scripting Language (formerly IFQL), super-charges queries both for analytics and data science. Matt will give a quick overview of the language features as well as the moving parts for a working deployment. Grafana is an open source dashboard solution that shares Flux’s passion for analytics and data science. For that reason, they are very excited to showcase the new Flux support within Grafana, and a couple of common analytics use cases to get the most out of your data.

In this talk, Matt Toback from Grafana Labs will share the latest updates they have made with their Flux builder in Grafana.

Register Now

KubeCon + CloudNativeCon North America 2018 | Seattle, WA – December 10-13, 2018:
David Kaltschmidt: On the OSS Path to Full Observability with Grafana – Grafana is coming "off the wall". To make it more useful for interactive debugging, David and his team have already integrated two pillars of observability – metrics and logs. They are currently adding tracing to complete the incident response experience. All to minimize the cost of context switching during those crucial minutes after getting paged.

This talk will demonstrate the various methods we've used to link the data together. Prometheus is providing the metrics. Via its histograms, request latencies can be extracted to inform each tracing span from Jaeger. Grafana also ensures that lines from your log aggregation system are annotated with span and trace IDs, as well as the other way around: associating logged values with spans.

David will show how these OSS parts should be deployed to achieve full observability in an engaging user experience that saves valuable minutes.

We are also a proud sponsor of the Cloud Native Computing Foundation’s flagship conference. Join Kubernetes, Prometheus, OpenTracing, Fluentd, gRPC, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF, Vitess, CoreDNS, NATS, Linkerd and Helm as the community gathers for four days to further the education and advancement of cloud native computing.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

We are thrilled to be mentioned at Google Cloud Next London ’18!


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

Grafana v5.3 Released

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/10/10/grafana-v5.3-released/

v5.3 Stable released!

Grafana v5.3 brings new features, many enhancements and bug fixes. This article will detail the major new features and enhancements.

Google Stackdriver

Grafana v5.3 ships with built-in support for Google Stackdriver and enables you to visualize your Stackdriver metrics in Grafana.

Getting started with the plugin is easy. Simply create a GCE Service account that has access to the Stackdriver API scope, download the Service Account key file from Google and upload it on the Stackdriver datasource config page in Grafana and you should have a secure server-to-server authentication setup. Like other core plugins, Stackdriver has built-in support for alerting. It also comes with support for heatmaps and basic variables.

If you’re already accustomed to the Stackdriver Metrics Explorer UI, you’ll notice that there are a lot of similarities to the query editor in Grafana. It is possible to add filters using wildcards and regular expressions. You can do Group By, Primary Aggregation and Alignment.

Alias By allows you to format the legend the way you want, and it’s a feature that is not yet present in the Metrics Explorer. Two other features that are only supported in the Grafana plugin are the abilities to manually set the Alignment Period in the query editor and to add Annotations queries.

The Grafana Stackdriver plugin comes with support for automatic unit detection. Grafana will try to map the Stackdriver unit type to a corresponding unit type in Grafana, and if successful the panel Y-axes will be updated accordingly to display the correct unit of measure. This is the first core plugin to provide support for unit detection, and it is our intention to provide support for this in other core plugins in the near future.

The datasource is still in the beta phase, meaning it’s currently in active development and is still missing one important feature – templating queries.
Please try it out, but be aware of that it might be subject to changes and possible bugs. We would love to hear your feedback.

Please read Using Google Stackdriver in Grafana for more detailed information on how to get started and use it.

TV and Kiosk Mode

We’ve improved the TV & kiosk mode to make it easier to use. There’s now an icon in the top bar that will let you cycle through the different view modes.

  1. In the first view mode, the sidebar and most of the buttons in the top bar will be hidden.
  2. In the second view mode, the top bar is completely hidden so that only the dashboard itself is shown.
  3. Hit the escape key to go back to the default view mode.

When switching view modes, the url will be updated to reflect the view mode selected. This allows a dashboard to be opened with a
certain view mode enabled. Additionally, this also enables playlists to be started with a certain view mode enabled.

Notification Reminders

Do you use Grafana alerting and have some notifications that are more important than others? Then it’s possible to set reminders so that you continue to be alerted until the problem is fixed. This is done on the notification channel itself and will affect all alerts that use that channel.
For additional examples of why reminders might be useful for you, see multiple series.

Learn how to enable and configure reminders here.

Postgres Query Builder

Grafana 5.3 comes with a new graphical query builder for Postgres. This brings Postgres integration more in line with some of the other datasources and makes it easier for both advanced users and beginners to work with timeseries in Postgres. Learn more about it in the documentation.

Improved OAuth Support for Gitlab

Grafana 5.3 comes with a new OAuth integration for Gitlab that enables configuration to only allow users that are a member of certain Gitlab groups to authenticate. This makes it possible to use Gitlab OAuth with Grafana in a shared environment without giving everyone access to Grafana.
Learn how to enable and configure it in the documentation.

Annotations

Grafana 5.3 brings improved support for native annotations and makes it possible to use template variables when filtering by tags.
Learn more about it in the documentation.

Variables

Grafana 5.3 ships with a brand new variable type named Text box which makes it easier and more convenient to provide free text input to a variable.
This new variable type will display as a free text input field with an optional prefilled default value.

Changelog

Checkout the CHANGELOG.md file for a complete list
of new features, changes, and bug fixes.

timeShift(GrafanaBuzz, 1w) Issue 63

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/10/05/timeshiftgrafanabuzz-1w-issue-63/

Welcome to TimeShift

This week we released Grafana 5.3.0beta-3 in prep for a stable release that should be available next week. In addition to details on the new beta, we have a lot of new and updated plugins to share, and our weekly roundup of Grafana-related articles from around the Internet.

Don’t forget to grab an early bird ticket to GrafanaCon before they’re sold out! Also, the CFP will close in a little less than two weeks so don’t wait until the last minute to submit a talk. If your talk is selected, we’ll comp your ticket or refund one you’ve already purchased. We’ve received some great proposals, but keep them coming! See grafanacon.org for more information.

Know of an article that might be a good fit for an upcoming issue? Contact us!

Enjoy!


GrafanaCon LA
Early Bird Tickets Now on Sale

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Get Your Ticket Now


Latest Stable Release: Grafana 5.2.4

Bug Fixes
  • GrafanaCli: Fixed issue with grafana-cli install plugin resulting in corrupt http response from source error. Fixes #13079

See everything new in Grafana v5.2.4.

Download Grafana 5.2.4 Now


Latest Beta Release: Grafana 5.3.0-beta3

Major New Features
  • Stackdriver: Fix for missing ngInject #13511
  • Permissions: Fix for broken permissions selector #13507
  • Alerting: Alert reminders deduping not working as expected when running multiple Grafana instances #13492

Download Grafana 5.3.0-beta3 Now


From the Blogosphere

How to gather and display metrics in Red Hat OpenShift: Learn how to use Apache Spark Metrics and Prometheus to gather and display metrics with Grafana on Red Hat OpenShift.

Pillars of Observability: The first in an upcoming series to serve as an introduction to observability and what it means. This post discusses metrics, logs, and tracing and common solutions for each.

Lab #1 – Migrating to Grafana/InfluxDB: While switching from a monolithic approach to a services-oriented architecture, ABC Arbitrage also needed a monitoring tool to gather and report application metrics. This post walks through the decisions they made to build their new monitoring stack.

Real Time Analytics for IoT Data using Mosquitto, AWS Kinesis and InfluxDB: IoT devices generate a large amount of data. Learn how build a data pipeline through common open source components and gain actionable insight from this data.


Grafana Plugin Update

Lots of new plugin and plugin updates were published this week, including updates to two of our premium plugins. To update any of your plugins in your on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

NEW PLUGIN

Dark Sky Datasource – This new datasource plugin allows you to graph historical weather conditions and forecast data from the Dark Sky weather service in Grafana.

Install

NEW PLUGIN

Statusmap Panel – The new Statusmap panel plugin allows you to visualize multiple time series in one panel. The panel has several display modes. Discrete mode allows you to map discrete values to colors. The opacity and spectrum modes are similar to the heatmap panel.

Install

NEW PLUGIN

Annotation List Panel – This new panel shows a table of Grafana user annotations. The annotations in the list are clickable and will navigate to the panel and time range of the chosen annotation.
This can be very useful if you want an overview of events that have occurred and to be able to quickly navigate to that event.

Install

UPDATED PREMIUM PLUGIN


Datadog Datasource – The latest release of the Datadog Datasource now supports some additional functions. Single metric arithmetic allows you to multiply a metric value by a number (value * 100 for example). The following three functions are now also supported:

  • Arithmetic -> Log10 function
  • Smoothing -> Auto Smoother
  • Regression -> Trend Line

Install

UPDATED PREMIUM PLUGIN


Splunk Datasource – A bug for relative date ranges in Grafana has been fixed in the Splunk Datasource. Date ranges like Yesterday and Previous week should now always work.

Install

UPDATED PLUGIN

Trend Box and Alarm Box – The Trend Box and Alarm Box panels both received a fix for a visual bug in Grafana 5.x.

Install Trend Box
Install Alarm Box

UPDATED PLUGIN

Akumuli Datasource – The latest release of the Akumuli Datasource includes a new feature and a bug fix. The new feature is support for template variables of type interval that can be used with Downsample control.

Queries using the Top N feature now work properly when using aliasing (formatting the legend key).

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now

InfluxDays 2018 | San Francisco – November 7-8, 2018:

Matt Toback: Optimizing the Grafana Platform for Flux – Flux, the new InfluxData Data Scripting Language (formerly IFQL), super-charges queries both for analytics and data science. Matt will give a quick overview of the language features as well as the moving parts for a working deployment. Grafana is an open source dashboard solution that shares Flux’s passion for analytics and data science. For that reason, they are very excited to showcase the new Flux support within Grafana, and a couple of common analytics use cases to get the most out of your data.

In this talk, Matt Toback from Grafana Labs will share the latest updates they have made with their Flux builder in Grafana.

Register Now

KubeCon + CloudNativeCon North America 2018 | Seattle, WA – December 10-13, 2018:
David Kaltschmidt: On the OSS Path to Full Observability with Grafana – Grafana is coming "off the wall". To make it more useful for interactive debugging, David and his team have already integrated two pillars of observability – metrics and logs. They are currently adding tracing to complete the incident response experience. All to minimize the cost of context switching during those crucial minutes after getting paged.

This talk will demonstrate the various methods we've used to link the data together. Prometheus is providing the metrics. Via its histograms, request latencies can be extracted to inform each tracing span from Jaeger. Grafana also ensures that lines from your log aggregation system are annotated with span and trace IDs, as well as the other way around: associating logged values with spans.

David will show how these OSS parts should be deployed to achieve full observability in an engaging user experience that saves valuable minutes.

We are also a proud sponsor of the Cloud Native Computing Foundation’s flagship conference. Join Kubernetes, Prometheus, OpenTracing, Fluentd, gRPC, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF, Vitess, CoreDNS, NATS, Linkerd and Helm as the community gathers for four days to further the education and advancement of cloud native computing.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

We are extremely proud of the Grafana open source community. Thank you!!


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 62

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/28/timeshiftgrafanabuzz-1w-issue-62/

Welcome to TimeShift

Big news this week – GrafanaCon early bird tickets are now on sale! We’ve released a limited number of early bird tickets, so grab yours before they’re sold out. Also, call for proposals is open until October 15, so don’t wait until the last minute to submit your talk. We’ve gotten some great proposals already, but the more the merrier. Keep an eye out at grafanacon.org for more updates.

See something we missed? Know of an article that might be a good fit for an upcoming issue? Contact us!

And now, on to the show!


Latest Stable Release: Grafana 5.2.4

Bug Fixes
  • GrafanaCli: Fixed issue with grafana-cli install plugin resulting in corrupt http response from source error. Fixes #13079

See everything new in Grafana v5.2.4.

Download Grafana 5.2.4 Now


Latest Beta Release: Grafana 5.3.0beta-1

Major New Features
  • Alerting: Notification reminders #7330, thx jbaublitz
  • Dashboard: TV & Kiosk mode changes, new cycle view mode button in dashboard toolbar #13025
  • OAuth: Gitlab OAuth with support for filter by groups #5623, thx BenoitKnecht
  • Postgres: Graphical query builder #10095, thx svenklemm

There are a lot of other new features and fixes including the grafana-cli fix included in 5.2.4, so please check out the release notes to see everything that’s new in Grafana 5.3.0beta-1.

Download Grafana 5.3.0beta-1 Now


GrafanaCon LA
Early Bird Tickets Now on Sale

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Get Your Ticker Now


From the Blogosphere

8 Reasons Why You Should Attend GrafanaCon (and how to get your boss to send you): We understand – conferences can be expensive. But the things you learn, people you meet, and memories you make are priceless. This article covers some of the major benefits of attending GrafanaCon, which we hope will convince your company they’d be silly not to send you.

Instrumenting Porcupine With Prometheus & Grafana): Adam Bien’s Porcupine library makes it easy to configure dedicated executor services that act as application bulkheads. Sebastian has created an extension that exposes the Porcupine statistics via MicroProfile Metrics, which can be visualized in Grafana via Prometheus.

Getting Started on Monitoring with Prometheus and Grafana): This presentation from a recent meetup on monitoring with Prometheus touches on the “hows” and “whys” of monitoring as well as diving into that particulars of Prometheus and Grafana. Presentation slides from the meetup are included.

[Video] Boskey Savla – Monitoring VKE Kubernetes clusters with Prometheus/Grafana): In this video from VMware {code} Power Sessions you’ll see how a K8s cluster deployed with VMware Kubernetes Engine can be monitored using Prometheus and Grafana.

Automate Grafana Dashboard Import Process: This article talks about the components and process to automatically import dashboards into Grafana to monitor an application running on a Kubernetes cluster.


Grafana Plugin Update

This week we have an update to one of our premium plugins. To update any of your plugins in your on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

UPDATED PLUGIN

New Relic Datasource – The New Relic Datasource premium plugin now has full support for the Insights API and the NQRL query language. The latest release adds support for annotation queries for the Insights API.

Annotations in Grafana provide a way to mark points on the graph with rich events (like an outage or a deploy). When you hover over an annotation you can get event description and event tags.

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now

KubeCon + CloudNativeCon North America 2018 | Seattle, WA – December 10-13, 2018:
David Kaltschmidt: On the OSS Path to Full Observability with Grafana – Grafana is coming "off the wall". To make it more useful for interactive debugging, David and his team have already integrated two pillars of observability – metrics and logs. They are currently adding tracing to complete the incident response experience. All to minimise the cost of context switching during those crucial minutes after getting paged.

This talk will demonstrate the various methods we've used to link the data together. Prometheus is providing the metrics. Via its histograms, request latencies can be extracted to inform each tracing span from Jaeger. Grafana also ensures that lines from your log aggregation system are annotated with span and trace IDs, as well as the other way around: associating logged values with spans.

David will show how these OSS parts should be deployed to achieve full observability in an engaging user experience that saves valuable minutes.

We are also a proud sponsor of the Cloud Native Computing Foundation’s flagship conference. Join Kubernetes, Prometheus, OpenTracing, Fluentd, gRPC, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF, Vitess, CoreDNS, NATS, Linkerd and Helm as the community gathers for four days to further the education and advancement of cloud native computing.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

8 Reasons Why You Should Attend GrafanaCon (and how to get your boss to send you)

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/27/8-reasons-why-you-should-attend-grafanacon-and-how-to-get-your-boss-to-send-you/

GrafabaCon EU

Mark your calendars! GrafanaCon LA 2019 is scheduled for Feb. 25-26 in Los Angeles, California. The CFP is offically open and we encourage you to submit multiple proposals. All speakers who are selected will have their tickets comped. Speaking of tickets, tickets are officially on sale!

If you’ve attended GrafanaCon before, you already know how awesome it is. But if you’ve never been and you’re on the fence, read on to find out just what we can pack into a two-day conference. And whether you’re a newbie or an alum, stick around for some tips on how to convince your boss to send you.

1. Learning is fundamental.

It goes without saying that there’s a lot of information to be had at GrafanaCon. Because so many different people have embraced Grafana—from the monitoring tinkerer to Fortune 500 companies—GrafanaCon provides the opportunity to get first-hand accounts of monitoring at every scale. Here you can learn monitoring best practices from companies like eBay and Uber, and the ins and outs of the latest open source monitoring tools. GrafanaCon isn’t just about Grafana; it also encompasses the open source monitoring ecosystem at large. You can find out more about Prometheus, Graphite, InfluxDB, Kubernetes, and others. At the last GrafanaCon, attendees heard from more than 30 speakers over two days, covering topics ranging from how Bloomberg built its monitoring infrastructure, to an overview of the RED Method of instrumenting services (from its creator!), to a preview of the new Influx Functional Query Engine.

2. Networking opportunities galore.

Social media can bridge distance, and websites contain a wealth of information, but neither can be a true substitute for the energy of face-to-face interaction and instruction. Here’s your chance to meet your LinkedIn connections and open source co-contributors in real life. Find like-minded people, curious fellow engineers, hobbyists, and people just learning about time series data, data visualization, and open source software. While GrafanaCon has grown year over year, we believe it’s still an intimate environment where people can really get to know their tribe. In addition to the sessions, there are lunches, breaks, and an after-hours party scheduled for attendees to get their networking on.

GrafabaCon EU

3. Hands-on demos.

Where else can you get all the industry leaders and experts in one room? GrafanaCon is the perfect place to learn about new features in Grafana, get in-depth demos and tutorials, tips and tricks, and learn about new products in the extended Grafana landscape—all from the founders and creators themselves.

4. Be part of the open source software community.

More than ever, Fortune 500 companies and other enterprises realize the significant business value of open source software: more rapid feature releases, better code quality, reduced vendor lock-in, reduced costs. And as more and more companies adopt these technologies, being a part of the open source software community is valuable to its employees, too. It’s like a stamp of approval on your CV. Attending GrafanaCon can help you learn how to navigate an open source project, make contributions, and build relationships with maintainers and fellow contributors—all skills that can help you do your current job (or get your dream job).

5. You’ve got wisdom to impart, too.

You’ve been in the trenches, so you have your own monitoring story to tell. At GrafanaCon, the experiences you’ve had and the lessons you’ve learned can be immensely valuable to your fellow attendees. You could even be the hero!

6. Invest in yourself.

You’re awesome, and you’re worth it. Not only will you come back from GrafanaCon recharged with new ideas on monitoring and observability to benefit your company, but you’ll also get the chance to grow both professionally and personally.

7. Share with your coworkers.

Bring what you’ve gleaned back to your colleagues. You can even host a meetup or a training session, and invite your new GrafanaCon connections. Learning new things is great, but passing on what you’ve learned to others is priceless.

8. Have fun.

While the problems we solve through monitoring are important for businesses, we want to provide a safe, supportive, and above all, fun experience. Work hard, play hard: Break out of your shell, and come home with some new life-long connections. Our after-conference receptions are always fun, whether we’re bowling in NYC or hanging out at an arcade in Amsterdam. Also: swag.

Guess what? All of the above can be cited to convince your boss to send you to GrafanaCon. (Okay, maybe not #8. And lose the references to your CV.)

We understand that conferences can be expensive—the price of the ticket, travel, lodging, meals, and time off from our actual jobs. But think about what you get in return: the opportunity to get hands-on trials of new tools, learn how they’re implemented at scale by some of the biggest companies in the world, and crowdsource solutions for monitoring issues we’ve all struggled through. You’ll come back to work invigorated and inspired, armed with new ways of looking at and transforming the data you collect, important industry connections, and fresh ideas that will ultimately help your business succeed. Isn’t that worth your while?

Get Your GrafanaCon Ticket Now!

GrafabaCon EU

timeShift(GrafanaBuzz, 1w) Issue 61

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/21/timeshiftgrafanabuzz-1w-issue-61/

Welcome to TimeShift

This week we share articles covering Grafana’s alpha Explore UI, the many ways Logicify uses Grafana, building your own centralized monitoring stack, and more.

See something we missed? Know of an article that might be a good fit for an upcoming issue? Contact us!


Latest Stable Release: Grafana 5.2.4

Bug Fixes
  • GrafanaCli: Fixed issue with grafana-cli install plugin resulting in corrupt http response from source error. Fixes #13079

See everything new in Grafana v5.2.4.

Download Grafana 5.2.4 Now


Latest Beta Release: Grafana 5.3.0beta-1

Major New Features
  • Alerting: Notification reminders #7330, thx jbaublitz
  • Dashboard: TV & Kiosk mode changes, new cycle view mode button in dashboard toolbar #13025
  • OAuth: Gitlab OAuth with support for filter by groups #5623, thx BenoitKnecht
  • Postgres: Graphical query builder #10095, thx svenklemm

There are a lot of other new features and fixes including the grafana-cli fix included in 5.2.4, so please check out the release notes to see everything that’s new in Grafana 5.3.0beta-1.

Download Grafana 5.3.0beta-1 Now


GrafanaCon LA
Call for Proposals

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit Your Proposal Today


From the Blogosphere

Grafana’s Explore UI: Taking a Deeper Dive into Data with Prometheus Queries: In this recap from the most recent PromCon in Munich, David Kaltschmidt demos the Explore UI alpha. This new interface allows you to iterate quickly through Prometheus queries, while leaving your dashboards intact. You can also try it yourself with a simple config edit.

Grafana, Graphite & Max Data Points: Steph was on a quest to find out why a Grafana singlestat panel was returning wacky values. Her investigation leads her to learning about Graphite’s default runtime consolidation. Our own Dieter Plaetinck wrote an article that touches on this and 24 other gotchas he’s encountered with Graphite, Grafana and Statsd.

Grafana as a Yet Another Tool for Technical Monitoring of Software Products We Build: Logicify uses Grafana in both their internal and external projects from monitoring the temperature of their offices to tracking user behavior in their eCommerce apps.

Services and resources monitoring with Prometheus and Grafana running on Docker: This article discusses building a centralized monitoring stack with Grafana and Prometheus to reduce context switching between multiple monitoring tools and more quickly improve overall system monitoring.

Monitoring Spark 2 performance via Grafana in Ambari-Metrics: This article covers adding a Spark 2 Dashboard to Grafana in Ambari-Metrics to monitor Spark applications for detailed resource usage statistics.


Grafana Plugin Update

One plugin update this week containing a few bug fixes. To update in your on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

UPDATED PLUGIN

Singlestat Math Panel – In this release an off-by-1px error in Firefox is fixed and the panel autosorts the thresholds on the option tab.

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Cloud Native Computing Stockholm | Stockholm, Sweden – September 25, 2018:

Torkel Ödegaard: What’s Going On at Grafana – In this session Torkel will provide an update on the project and discuss what’s new/what’s coming in the future. He’ll also cover some “best practices/interesting things seen in the wild,” and save some time for Q&A.

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now

KubeCon + CloudNativeCon North America 2018 | Seattle, WA – December 10-13, 2018:

We are a proud sponsor of the Cloud Native Computing Foundation’s flagship conference. Join Kubernetes, Prometheus, OpenTracing, Fluentd, gRPC, containerd, rkt, CNI, Envoy, Jaeger, Notary, TUF, Vitess, CoreDNS, NATS, Linkerd and Helm as the community gathers for four days to further the education and advancement of cloud native computing.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

I’m sure they’d give you a volume discount. 🙂


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

Grafana’s Explore UI: Taking a Deeper Dive into Data with Prometheus Queries

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/21/grafanas-explore-ui-taking-a-deeper-dive-into-data-with-prometheus-queries/

GrafabaCon EU
David Kaltschmidt, Director, UX – Grafana Labs

When there’s an incident, Grafana is often the starting point for figuring out a response. Users look at a time series panel and form a hypothesis. And in many situations, they’d like to dive deeper.

To help make that easier, Grafana Labs has created the Explore UI, which allows you to iterate quickly through Prometheus queries, while leaving your dashboards intact. “Grafana is currently on a journey to transition from dashboarding solutions to an observability platform,” Grafana Labs Director, UX David Kaltschmidt explained during his presentation at PromCon in August.

Explore UI was designed to fit into workflows for both troubleshooting and data exploration. And with Prometheus as a data source in 54,000 of over 186,000 Grafana installations—a 3x increase over last year—it made sense to focus on Prometheus queries first.

“We wanted to bring a lot more interactivity to queries and to Grafana,” said Kaltschmidt, “and we iterated a bit and came up with this more query-centric view, as opposed to the classic Grafana dashboards, which are more about visualizations.”

There’s now a new Explore icon in the panel menu of Grafana. This new section, which Kaltschmidt demoed at PromCon, focuses on the queries themselves. “For Prometheus specifically, we want to make the things that should be easy a bit easier,” Kaltschmidt said. “So for example, there’s a little metrics explorer here which groups the available metrics by prefix, so you can have your Alertmanager-related metrics here, and then you just click one, and it fills it out. That’s pretty cool. And then it’s like, hmm, this is a time series that’s monotonically increasing, maybe we should apply a rate function, so then you just click this button, and boom that works.” Especially these last query hints will play a big part in making Prometheus more approachable for novices.


New Explore UI interface

Kaltschmidt then showed how the same could be done for anything with buckets. Explore can fill out “what we hope is a good default function call for your buckets,” he said, by wrapping the expression with the correct histogram function and its parameters. These resulting expressions are meant to be quick starting points, which you can then iterate through queries.

A history was added, so that you can see information such as the last things you queried, how often they’ve been queried in the last 24 hours, and how long ago that was.

“It always executes two queries, one for the instant query for the table, and one for the graphs,” said Kaltschmidt. “You can also toggle the graph, and then you just see the table if you want to iterate more quickly.” It only takes one click on a label to filter by it and drill-down.

We’re already supporting a brand new feature of the Prometheus API: recording rules, which means we have their definitions available in Grafana. “If you want to start debugging a query that also contains recording rules, you can expand the rules and do this for all the recording rules that are in your query, in case you’ve forgotten what you’ve written there.”

A split view allows you to compare different Prometheus instances, for example, Dev vs. Prod. And multiple queries can be displayed in the same graph.

The last part of Kaltschmidt’s demo showed a faceted search within a query. Starting with a long list of jobs you can select a job, e.g., Alertmanager, and “this basically helps you find out what actual queries are available by the Alertmanager job,” he said. “Then you can look for a metric here, and it totally zones in on the labels that are available based on the previous selectors. So you only end up with suggestions that make sense to everything that came before.”


Split view and multuple queries on the same graph

Explore UI is still “super alpha,” Kaltschmidt concluded, but it’s behind the feature flag, and can be accessed if you’re using the latest Docker image. It will likely be released with Grafana v6.0. There are still things that need to be refined: “Support for completion needs a bit more work. We want Prometheus metric metadata from HELP line in the exposition. The Prometheus exposition format has this help text around what a query does, and when the ‘Write-ahead lock’ stuff is implemented, we can possibly show these query metadata, time series metadata also in the UI. That will be pretty cool.”

To enable, simply edit the Grafana config ini file:

[explore]
enabled = true

Then set up a datasource that supports Explore, e.g. Prometheus, and you’re ready to go… Explore. Feedback is welcome (@davkals)!

Watch the live demo:

Download the presentation slides

timeShift(GrafanaBuzz, 1w) Issue 60

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/14/timeshiftgrafanabuzz-1w-issue-60/

Welcome to TimeShift

This week we’re sharing articles on monitoring mixins, cloud native monitoring, monitoring your microservices, and a unique way to know when your software license is going to expire.
See something we missed? Know of an article that might be a good fit for an upcoming issue? Contact us!


Latest Stable Release: Grafana 5.2.4

Bug Fixes
  • GrafanaCli: Fixed issue with grafana-cli install plugin resulting in corrupt http response from source error. Fixes #13079

See everything new in Grafana v5.2.4.

Download Grafana 5.2.4 Now


Latest Beta Release: Grafana 5.3.0beta-1

Major New Features
  • Alerting: Notification reminders #7330, thx jbaublitz
  • Dashboard: TV & Kiosk mode changes, new cycle view mode button in dashboard toolbar #13025
  • OAuth: Gitlab OAuth with support for filter by groups #5623, thx BenoitKnecht
  • Postgres: Graphical query builder #10095, thx svenklemm

There are a lot of other new features and fixes including the grafana-cli fix included in 5.2.4, so please check out the release notes to see everything that’s new in Grafana 5.3.0beta-1.

Download Grafana 5.3.0beta-1 Now


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


From the Blogosphere

Everything You Need to Know About Monitoring Mixins: In this recap from the most recent PromCon in Munich, Tom Wilkie dives into monitoring mixins – a solution for packaging together templates for Grafana dashboards and Prometheus alerts related to a specific piece of software. In addition to the recap, we’ve included the video demo and his presentation slides.

Reporting End of License on Temp Products with Chocolatey, Python, Telegraf and Grafana: Fabien walks us through how he uses Grafana alerts to let him know when his software licenses are about to expire.

IoT / SmartHome – Send EnOcean Sensor Data to Kafka: This article walks you through the steps to get your IoT sensor data visualized in Grafana. An EnOcean sensor will send data to Jeedom, which will feed a MQTT broker and then a synchronizer will send each received message to a Kafka topic. Then the latest value will be exposed to Prometheus and displayed on a Grafana dashboard.

Cloud Native Monitoring and Visualization with Prometheus and Grafana: A brief article on setting up a cloud native monitoring solution. Learn the basics of monitoring and observability, and get familiar with Prometheus and Grafana to make use of them in your Kubernetes environment.

Go Microservices Blog Series, Part 15 – Monitoring with Prometheus.: Part 15 in the Go Microservices blog series tackles monitoring with Prometheus and Grafana. You’ll learn about service discovery, adding endpoints, setting up the stack, and see some example queries to get you going.

Helm Chart Exporter: Ever wanted to know what versions of software are running on your Kubernetes cluster? Now, assuming you’re using Helm to install everything, you can.


Grafana Plugin Update

We have four plugin updates to share this week. Updating Grafana plugins is easy – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

UPDATED PLUGIN

USGS Water Services Data Source – The most important change is that USGS moved their service URL and this now avoids the 301 error. Also, this now nicely picks the mean daily value when you zoom out the instantaneous service to a large time window.

Install

UPDATED PLUGIN

Clickhouse Data Source – Updates to Clickhouse Data Source 1.7 include:

  • Provide $adhoc macros for using ad-hoc filters in inner queries (thx to @vavrusa)
  • Allow to set custom query for ad-hoc filter via adhoc_query_filter variable
  • Provide new Round value $step for auto-rounding according to graph resolution changes

Install

UPDATED PLUGIN

Akumuli Data Source – Minor bug fixed in the latest issue. Limit on number of returned data-points was set using the maxDataPoints value. That values was too small in some cases.

Install

UPDATED PLUGIN

Breadcrumb Panel – The Breadcrumb Panel received some minor style tweaks and fixed an issue with the display of the panel title.

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Meetup Workshop: Monitoring with Prometheus and Grafana | Belfast, Northern Ireland – September 18, 2018:

If you’re in Belfast, or are going to be in September, this could be a great Meetup to attend. Topics include: Architecture, Prometheus, Alertmanager, Pushgateway, Telegraf, JMX exporter, Grafana, and more!

RSVP Now

Cloud Native Computing Stockholm | Stockholm, Sweden – September 25, 2018:

Torkel Ödegaard: What’s Going On at Grafana – In this session Torkel will provide an update on the project and discuss what’s new/what’s coming in the future. He’ll also cover some “best practices/interesting things seen in the wild,” and save some time for Q&A.

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

Indeed! Be sure to checkout talks from the fine folks at FermiLab and CERN from previous GrafanaCons too.


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

Everything You Need to Know About Monitoring Mixins

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/13/everything-you-need-to-know-about-monitoring-mixins/

GrafabaCon EU

There are lots of great community-built dashboards available in the Grafana website, and we’ve made it easier than ever to share and download them. But it’s not uncommon for users to import a dashboard and find that it simply doesn’t work because the labels don’t match up. And the same can happen with Prometheus alerts that are shipped with software like etcd.

So how can we fix this? Tom Wilkie, Prometheus developer and VP Product at Grafana, recently gave a talk at PromCon in Munich about monitoring mixins, a solution for packaging together templates for Grafana dashboards and Prometheus alerts related to a specific piece of software. The basic premise: “Dashboards and alerts should not be opinionated about the labels; labels should be configuration that you can feed into the dashboards and alerts,” said Wilkie. “We need a process where not only can alerts and dashboards not assume the label sets that they’re using in their queries, but also we can inject them dynamically, and we can stay up to date with changes upstream.”

Yaml JSON jinja Go Python Jsonnet

The first step to building mixins was finding a configurable and extensible language for expressing alerts and dashboards, which would allow them to be reusable. Wilkie began a process of elimination: Prometheus alerts are written in yaml, and JSON is a subset of yaml. But these configuration languages “are not very dynamic, or dynamic at all really,” he said. “You can’t really inject labels as I want to, or even inject any kind of configuration.”

The next possibility: string substitution, treating yaml config files as text files and using jinja or Go templates to substitute things in. “The problem there is your substitutions don’t really understand the structure of the configuration you’re manipulating,” he said. “How would you go and add a tag to all of your alerts? Unless that substitution spot has been enabled in the config, you can’t do it.”

Wilkie then presented another increasingly popular strategy of using a Go or Python program to generate config. “The chaps over at Weaveworks have written a Python library called Grafana Lib, and they use Python to generate their dashboards,” he said. “Which is great because you can now write Python code which goes in and modifies your dashboards.” But Wilkie eliminated this method as well, saying “this feels too much like writing programs, and it’s generally a bit of a pain.”

Moreover, he said, “this is not how they did it at Google.”

Just as the cloud native community has taken its cues from Google with Kubernetes (from Google’s Borg) and Prometheus (from Borgmon), Wilkie said he modeled his solution after how that company solved the problem internally. The closest thing available, he said, was the data templating Jsonnet, a powerful extension of JSON that’s easier to write: “It feels a lot like JSON, but it adds variables, conditionals, arithmetic, functions, imports, and error propagation.”

Demystifying Jsonnet

In his live demo at PromCon, Wilkie showed how powerful Jsonnet is, even though it’s quite terse.

First of all, you can refer to things, which is the basis on which configuration can be built. You can have a central config that gets referred to in various different places.

Secondly, you can do merges. “This is really the crux of Jsonnet,” he said. “It has really well defined merge semantics. So now you can imagine you use this system to merge in your config into the existing set of dashboards. And this is really the basis on which all the mixin stuff is built. You would take a mixin and merge in some specialization, some configuration.”

Wilkie then demonstrated recursive merges, hidden fields, and string substitution with Jsonnet—what he called “the tool kit we’re using to build up the mixins. There are only three or four operators you need to remember, and because they’re all single character operators, I think at first glance people tend to think it’s crazy.”

Watch the live demo:

Download the presentation slides

Building a Mixin

There are four top-level keys in a mixin: config, dashboards, alerts, rules.

The alerts and rules use the same structure that’s in Prometheus yaml. The dashboard is a dictionary of JSON files that contains your Grafana JSON.

Using the Kubernetes mixin as an example, Wilkie walked through the code. “In the config, typically every mixin will expose as a configuration the selector it wants to use to select the job, so here we’re looking at a Kube state metric,” said Wilkie. “I like to include the namespace in the job name. This stops me aggregating across jobs in different namespaces. This is how I express my opinion, but you can do it differently because you can override it because it’s Jsonnet.”

The ability to override also solves the problem that occurs when a label changes upstream. Wilkie said that for example, with the Kubernetes mixin, there are default selectors used, but “you can come along and use a different set of labels. So this is my config that overrides those. In my config, I’ve said I want everything to be prefixed with a namespace it lives in. I’ve even made the namespace configurable in certain cases. So this lives in our configuration mono repo, and it gets rendered out to the Kubernetes cluster every time you change anything. And for that I use ksonnet.” (Wilkie explained this further in his KubeCon talk.)

The definition of the alert “isn’t quite as pretty as yaml, but it’s better than JSON,” he said. “You build up your alert groups, you build up your list of rules, you have pod crash looping, and then in the query you do this big string substitution. I like to do it using name substitution, so I pass in the whole config dictionary to every string. And then I name the field in the config dictionary that I’m pulling out. You could just do %s and then do config.kubestateselector. But this helps if, for instance, and this is quite common, you might refer to the same selector multiple times within a single query. You might want to refer to multiple selectors within a single query. You’ll notice [the %s] is part of Jsonnet, and the curly brace is part of the PromQL query. So that’s how we’ve embedded the two languages. And then when it’s rendered, it gets substituted into the thing you’d expect, and it’s rendered out as yaml.”

Making Them Reusable

In an effort to make it easy for people to download, upgrade, and manage these mixins, CoreOS Software Developer and Prometheus contributor Frederic Branczyk built a package manager for Jsonnet called Jsonnet-bundler. “It’s really straightforward,” said Wilkie. “You go and get it, you init, which just writes out an empty log file, and then you can install various mixins. It’s kind of modeled on goget: It builds a vendor tree full of all your mixins, and it will go and recursively fetch other Jsonnet libraries that mixins depend on. So Grafonnet is a Jsonnet library for building Grafana dashboards, so you don’t have to manage the JSON yourself, but you can just do like newpanel, newdashboard, and so on.”

With that vendor tree of all your mixins, you can build Jsonnet snippets. The Kubernetes mixin has a readme that explains how to do it.

Mixins in the Wild

With that information, plus design docs for the package manager and for the mixin spec, available online, Wilkie hopes more people will contribute to the first four mixins (etcd, Consul, Vault, and Kubernetes), which he mostly wrote himself.

The Kubernetes mixin, which he wrote with Branczyk, has gained the most traction. There are now nine more contributors to it, from different companies including Google, MongoDB, and the San Diego Supercomputer Center. And going forward, the Kubernetes mixin’s set of alerts and dashboards will be used in all new versions of OpenShift.

There’s now also a Prometheus ksonnet mixin. “This defines a set of alerts that we think you should be running if you’re running Prometheus,” Wilkie said. “We have a node_exporter mixin. This one’s still quite a work in progress, trying to describe standard alerts and dashboards for your nodes.”

And most excitingly for Wilkie, others in the community seem to be playing with mixins too. “This chap whom I’ve never met before, he’s written a mixin for something extraterrestrial,” Wilkie said. “I haven’t checked it out yet but it’s NASA space weather something, and it’s got a Jsonnet mixin in it.”

timeShift(GrafanaBuzz, 1w) Issue 59

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/09/07/timeshiftgrafanabuzz-1w-issue-59/

Welcome to TimeShift

Last week TimeShift took a vacation, but the Grafana Labs team stayed busy. We announced an important security fix for 5.2.3 and 4.6.4 specifically for LDAP and OAuth authentication. Please read the announcement and upgrade immediately if you haven’t already.

This week we released Grafana 5.2.4 stable and 5.3.0-beta1. More on those releases in the section below.

Come across an article you think might be a good fit for an upcoming issue? Suggestions for new content? Contact us!


Latest Stable Release: Grafana 5.2.4

Bug Fixes
  • GrafanaCli: Fixed issue with grafana-cli install plugin resulting in corrupt http response from source error. Fixes #13079

See everything new in Grafana v5.2.4.

Download Grafana 5.2.4 Now


Latest Beta Release: Grafana 5.3.0beta-1

Major New Features
  • Alerting: Notification reminders #7330, thx jbaublitz
  • Dashboard: TV & Kiosk mode changes, new cycle view mode button in dashboard toolbar #13025
  • OAuth: Gitlab OAuth with support for filter by groups #5623, thx BenoitKnecht
  • Postgres: Graphical query builder #10095, thx svenklemm

There are a lot of other new features and fixes including the grafana-cli fix included in 5.2.4, so please check out the release notes to see everything that’s new in Grafana 5.3.0beta-1.

Download Grafana 5.3.0beta-1 Now


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


From the Blogosphere

Building Telemetry for Tea aka Tealemetry: In the search for true observability Ben is tackling the important issues; the proper temperature to consume tea. Using a temperature sensor, Grafana and its built-in alerting feature, he’s found the sweet spot between scalding your taste buds and something fit to dump down the drain. Very entertaining article.

Deploying Helm / Tiller, Prometheus, AlertManager, Grafana, Elasticsearch On Your Kubernetes Cluster: Part 8 in the series on configuring a 3 node Kubernetes master cluster. This article tackles installing and configuring Helm, Prometheus, Alertmanager, Grafana, and Elasticsearch.

Monitoring Vault on Kubernetes Using Cloud Native Technologies: Bank Vaults is a core building block of the Pipeline PaaS. This post is about monitoring Vault with Prometheus (on Kubernetes) and displaying metrics on Grafana.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Meetup Workshop: Monitoring with Prometheus and Grafana | Belfast, Northern Ireland – September 18, 2018:

If you’re in Belfast, or are going to be in September, this could be a great Meetup to attend. Topics include: Architecture, Prometheus, Alertmanager, Pushgateway, Telegraf, JMX exporter, Grafana, and more!

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

Grafana 5.2.3 and 4.6.4 released with important security fix

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/29/grafana-5.2.3-and-4.6.4-released-with-important-security-fix/

Today we are releasing Grafana 5.2.3 and Grafana 4.6.4. These patch releases includes a very important security fix for all Grafana installations which are configured to use LDAP or OAuth authentication.

Latest stable release in 5.x:

Latest stable release in 4.x:

LDAP & OAuth login vulnerability (CVE-2018-558213)

On the 20th of August at 1800 CEST we were contacted about a potential security issue with the “remember me” cookie Grafana sets upon login. The issue targeted users without a local Grafana password (LDAP & OAuth users) and enabled a potential attacker to generate a valid cookie knowing only a username. We immediately recognized the high severity of this vulnerability and quickly developed a patch for 5.x and 4.x.

Affected versions

Grafana releases 2.0 through 5.2.2 are affected by this vulnerability. All instances
on Grafana Cloud have already been updated to 5.2.3.

Solutions and mitigations

All installations which use the Grafana LDAP or OAuth authentication features must be upgraded as soon as possible. If you cannot upgrade, you should switch authentication mechanisms or put additional protections in front of Grafana such as a reverse proxy.

CVE ID: CVE-2018-558213

We would like to thank Sebastian Solnica for reporting this issue.

Timeline and postmortem

Here is a detailed timeline starting from when we originally learned of the issue. This is the first significant security issue that Grafana Labs has dealt with; we have used this incident as an opportunity to learn from, and will be improving our incident response going forward.

  • 20 Aug 2018 1800 CEST
    • Torkel Ödegaard received request via twitter DM for contact details
    • Received details of vulnerability from Sebastian Solnica 1 hour later
  • 21 Aug 2018 08:00 CEST
    • Confirmed issue and began developing a fix.
    • Fix required a minor new feature in db migration system so commit message did not mention that it had any security implications.
    • Backported fix to 5.2.x branch
  • 22 Aug 2018 11:00 CEST
    • Researched vulnerability disclosure handling
    • Started preparing 5.2.3 release
  • 22 Aug 2018 20:33 CEST
    • Got outside counsel on how to handle vulnerability fixes. Learned that it’s best to request CVE, prepare fix in private, and to give non-public heads up to customers, distributions, etc while maintaining strict public embargo until release time.
    • Requested & received CVE ID CVE-2018-558213
    • Started preparations to handle future vulnerabilities better
  • 23 Aug 2018
    • Started rolling out 5.2.3 to Grafana Cloud customers
    • Worked on backport of fix for old 4.6.x release branch.
    • Decided on making release public on Wed Aug 29 13:00 CEST. The date was chosen to give people time to prepare and not run into the weekend. The time was chosen to fall into main work time of the EU and US while still giving Asia a fair chance to react.
  • 23 Aug 2018 22:05 CEST
    • Proactively contacted Grafana Enterprise customers with details and download links.
    • Completed rollout of 5.2.3 to Grafana Cloud.
  • 29 Aug 2018 13:00 CEST
    • Publish of release & this blog post.

Future considerations

We are not sure if omitting the security nature of the change in the commit message was correct or not. Apache VULNERABILITY HANDLING guidelines state: “No reference should be made to the commit being related to a security vulnerability.”

We did do a private release that we circulated beforehand but the fix (and tag of v5.2.3) was then already in the public repo. The ordering here was obviously wrong. If doing a private release before going public, the fix should also stay private.

We are also putting in new processes for submitting and handling security incidents as you can see in the next section.

Reporting security Issues

If you think you have found a security vulnerability please send a report to [email protected]. This address can be used for all of
Grafana Labs’s open source and commercial products (including but not limited to Grafana, Grafana Cloud, Grafana Enterprise, and grafana.com). We can accept only vulnerability reports at this address. We would prefer if you encrypted your message to us, please use our PGP key. The key fingerprint is

F988 7BEA 027A 049F AE8E 5CAA D125 8932 BE24 C5CA

The key is available from pgp.mit.edu search for grafana.

Security Announcements

We have created a new category on the community site named Security Announcements
where we will post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to email updates to this category if you have a grafana.com account and sign on the community site or via track updates via an RSS feed.

Conclusion

If you run a Grafana installation which uses LDAP or OAuth please upgrade to Grafana 5.2.3 or 4.6.4 as soon as possible.

timeShift(GrafanaBuzz, 1w) Issue 58

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/24/timeshiftgrafanabuzz-1w-issue-58/

Welcome to TimeShift

This week we highlight articles featuring the ultimate guide to monitoring Kubernetes using Prometheus and Grafana, how to build effective dashboards, and a guide to help demystify PromQL.

With the upcoming Labor Day holiday, TimeShift is going to take a short break, but we’ll see you again soon!

Come across an article you think might be a good fit for an upcoming issue? Suggestions for new content? Contact us!


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned Prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


From the Blogosphere

Kubernetes Monitoring with Prometheus, the ultimate guide (part 1): Part 1 in a 4-part, in-depth series on how to implement Kubernetes monitoring with Prometheus. The series opens with an intro to Prometheus and its core concepts, how Prometheus stacks up to veteran monitoring solutions, and an overview of all the components in the architecture.

Digital Dashboard: Making a Dashboard work for you, part one: This series starts off by establishing a baseline for what makes an effective dashboard. John also highlights the strengths he sees in Grafana as well as areas for improvement. This type of thoughtful feedback is always helpful, as we are constantly working to improve the overall user experience. Interested to see where this series goes.

First Steps with Prometheus and Grafana on Kubernetes on Windows: A step-by-step installation guide for monitoring Kubernetes with Prometheus and Grafana on Windows.

PromQL for Humans: If you’re new to Prometheus, PromQL can be daunting. This article attempts to demystify the query language. You’ll also get a handy query cheat-sheet with plenty of real-world examples.

Creating Visualizations in Grafana: This short walkthrough will help you get started creating dashboards in Grafana using your Prometheus metrics.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Grafana Plugin Update

Plugin authors often add new features and fix bugs, so it’s important to keep your plugins up to date. We’ve made the update process simple – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

UPDATED PLUGIN

Single Stat Math Panel – A new feature in this panel adds a circle shape as a background option. This panel is similar to the native Singlestat Panel but also supports math functions across series using the math.js library.

Install


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

Monitorama AMS 2018 | Amsterdam, The Netherlands – September 4-5, 2018:

Monitorama events bring together the brightest minds among the open source development and operations communities. Their goal is to continue to push the boundaries of monitoring and trending software, all while having a great time in a casual setting.

This event takes place in the beautiful city of Amsterdam. If you haven’t already experienced Monitorama, this is a great chance to make their acquaintance.

Get Your Ticket Now

Meetup Workshop: Monitoring with Prometheus and Grafana | Belfast, Northern Ireland – September 18, 2018:

If you’re in Belfast, or are going to be in September, this could be a great Meetup to attend. Topics include: Architecture, Prometheus, Alertmanager, Pushgateway, Telegraf, JMX exporter, Grafana, and more!

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

I see a T-Rex… or a really tall aligator running left.


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 57

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/17/timeshiftgrafanabuzz-1w-issue-57/

Welcome to TimeShift

August is flying by, but hopefully there’ll still be time to enjoy a few more summer evenings. This week we’re sharing the video demoing the new Explore UI in Grafana from last week’s PromCon, monitoring VMWare’s VKE with Prometheus, hosting a blog on a budget and more.

Come across an article you think might be a good fit for an upcoming issue? Suggestions for new content? Contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned Prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


From the Blogosphere

Video: David Kaltschmidt: Exploring your Prometheus Data in Grafana: Last week we shared David’s slides from PromCon 2018, but it’s so much better to actually see it action.

Installing Prometheus and Grafana on VMware Kubernetes Engine: This post details the process of deploying Prometheus as a monitoring framework for Kubernetes, along with Grafana as the visualization layer. Bahubali covers why monitoring K8s is different, building and preparing your cluster, and installing both Prometheus and Grafana.

Collecting DHCP Scope Data with Grafana: Eric wrote a Python script to help him collect aggregated data about groups of DHCP scopes and how his network users were changing. This lets him total up the number of free and used IPs in each range and visualize them on a graph in Grafana.

How I host this blog, CI and tooling: Vik is a budget conscious blogger and developer. In this article he provides a rundown of the infrastructure he uses for his blog and how he keeps it running for $8.53/month.

Graphite Grafana: Metrics Monitoring Made Easy: The first in a series on metrics monitoring made easy, this article gets you started with spinning up a Graphite/Grafana stack. Learn about the components of Graphite, Grafana, and how to get everything installed – the next article will dive into the actual monitoring.

System monitoring with Grafana, InfluxDB et Collectd: Learn about the components of a responsive dashboard system and how to easily deploy it with Docker.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

2018 Sensu Summit | Portland, OR – August 22-23, 2018:
Brian Gann: The Sensu Plugin for Grafana – Brian will be showing a demo of the new Sensu plugin for Grafana on August 22, and conducting a 30 minute Grafana tutorial on the 23rd!

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

Meetup Workshop: Monitoring with Prometheus and Grafana | Belfast, Northern Ireland – September 18, 2018:

If you’re in Belfast, or are going to be in September, this could be a great Meetup to attend. Topics include: Architecture, Prometheus, Alertmanager, Pushgateway, Telegraf, JMX exporter, Grafana, and more!

RSVP Now

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now

All Things Open 2018 | Raleigh, NC – October 21-23, 2018:

Tom Wilkie: The RED Method – How to Instrument your Services – The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.

In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.

Register Now

OSMC 2018 | Nuremberg, Germany – November 5-8, 2018:

David Kaltschmidt: Logging is Coming to Grafana – Grafana is an OSS dashboarding platform with a focus on visualizing time series data as beautiful graphs. Now we’re adding support to show your logs inside Grafana as well. Adding support for log aggregation makes Grafana an even better tool for incident response: First, the metric graphs help in a visual zoning in on the issue. Then you can seamlessly switch over to view and search related log files, allowing you to better understand what your software was doing while the issue was occurring. The main part of this talk shows how to deploy the necessary parts for this integrated experience. In addition I’ll show the latest features of Grafana both for creating dashboards and maintaining their configuration. The last 10-15 will be reserved for a Q&A.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

I love a good heatmap. Let us know if you figure out where that super high latency is coming from.


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 56

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/10/timeshiftgrafanabuzz-1w-issue-56/

Welcome to TimeShift

PromCon was held this week in Munich, Germany, and Grafana Labs was excited to both sponsor and speak at this great conference. If you didn’t have a chance to attend, you can watch a recorded livestream of the event and download slides in the blog roundup below.
Tom Wilkie and David Kaltschmidt
Left to right: Tom Wilkie and David Kaltschmidt prepping for their talks

We’d also like to congratulate the Prometheus project for moving from the incubation stage to graduation as an official CNCF project!

Hope you enjoy this week’s roundup of articles and plugin updates. If you see an article you think might be a good fit for an upcoming issue of TimeShift, please contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned Prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

David Kaltschmidt: Exploring your Prometheus Data in Grafana: David has been working tirelessly on the new explore UI for Grafana. We’ll link to the video once it’s available, but for now he’s made his slides available so you can learn about it, and find out how to enable the feature in the latest version to give it a try.

Tom Wilkie: Prometheus Monitoring Mixins – Using Jsonnet to Package Together Dashboards, Alerts, and Exporters: Tom discusses a technique for using Jsonnet for packaging and deploying “Monitoring Mixins” – extensible and customizable combinations of dashboards, alert definitions and exporters. This technique allows you to publish best-practice monitoring configurations alongside your code, and for users to consume it, customize it and stay up to date. Also, check out the video of his presentation.

Using TimescaleDB + Prometheus to Monitor and Troubleshoot CockroachDB: Diana, who recently joined the Timescale team outlines how to combine four open source technologies to create a powerful monitoring stack, and provides a high-level tutorial on getting everything set up.

Inside Fortnite’s Massive Data Analytics Pipeline: Fortnite is a truly massive MMPG with over 125 million users spanning the globe. But pulling together all the servers, databases, and data pipelines to manage 92 million events per minute is no small feat. This article gives you a rundown of their systems, how they keep them running, and what they’d like to optimize in the future.

Managing your Costs on Kubernetes: Cost is often a factor when discussing migrating to a public cloud solution. When the sky’s the limit when it comes to resources, your monthly bill could be at risk of poking through the stratosphere. The article shows you how to use Prometheus, Grafana, and Kubernetes to create a dashboard that can help you get a better visualization of the cost of your applications.

M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus: Rob Skillington, staff software engineer on the Observability team in the Uber New York City provides a deep dive into M3, the metrics platform they’ve been working on for the past few years. In fact, at our very first GrafanaCon in 2015, Matt Mihic gave a talk about M3.

NetEye 3.14 and NetEye 4.2 Release Notes: The latest version of Würth Phoenix’s NetEye Unified Monitoring Solution has been released which includes Grafana 5 and allows you to take advantage of the new dashboard layout engine, folders, provisioning and more.


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


Grafana Plugin Update

Two new panels to show off this week along with an update to the Clickhouse data source plugin. We’ve made it simple to update your plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

NEW PLUGIN

WindRose Panel – This new panel was developed in a joint effort as part of a research project in Taiwan. A wind rose is a graphic tool used by meteorologists to give a view of how wind speed and direction are distributed at a location.

Install

NEW PLUGIN

LinkSmart SensorThings Data Source – This is a new panel that works with any server that implements the OGC SensorThings API. The SensorThings API is a standard for connecting IoT devices, data and applications over the web. An exciting feature of this is the ability to easily integrate into existing Spatial Data Infrastructures (SDI) or Geographic Information Systems (GIS) and means this data source works well with the Worldmap panel.

Check out their live demo dashboard to see what this data source can do.

Install

UPDATED PLUGIN

Clickhouse Data Source – The Clickhouse plugin is being continually improved and the latest release now has support for annotation queries.

Another new feature is the addition of two new macros: $from and $to that can be used in templating queries. If you are using Grafana’s provisioning feature there is now a config example included in the readme for the Clickhouse plugin.

Install


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

London Hashicorp User Group | London, United Kingdom – August 14, 2018:

Tom Wilkie: Monitoring the Hashistack with Prometheus – Prometheus has become the defacto monitoring system for cloud native applications, but for a while was eschewed by the Hashistack in favour of more traditional technologies. Thats all changing: Hashicorp’s project are beginning to export metrics in the native Prometheus format, and many exporters exist to bridge the gap.

In this talk Tom will give a brief introduction to Prometheus, show you how to piece it all together, and give some recommendation on what to monitor and alert on.

Register Now

How to Use Open Source Projects for Performance Monitoring | Webinar – August 15, 2018 – 1pm EDT:

Check out how you can use 5 of the most popular open source projects, InfluxDB, Telegraf, Chronograf, Kapacitor and Grafana, for performance monitoring of your Infrastructure, Application, & Cloud faster, easier, and to scale. In this webinar, Leonard Gram from Grafana, and Margo Schaedel from InfluxData, will provide you with step by step instruction from download & configure, to collecting metrics and building dashboards and alerts.

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:
Brian Gann: The Sensu Plugin for Grafana – Brian will be showing a demo of the new Sensu plugin for Grafana on August 22, and conducting a 30 minute Grafana tutorial on the 23rd!

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

So glad to hear you’re excited about the new explore UI. It’s still pre-alpha, so there’s lots more to come!


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 55

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/03/timeshiftgrafanabuzz-1w-issue-55/

Welcome to TimeShift

This week we announced the dates for GrafanaCon LA and officially opened up the CFP! While we can’t predict the weather, we can be almost certain it will be blizzard-free this time around. Also, if you’re going to be in Munich next week for PromCon, please be sure and say hello!

Hope you enjoy this week’s roundup of articles and plugin updates. If you see an article you think might be a good fit for an upcoming issue of TimeShift, please contact us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

The RED Method: How to Instrument Your Services: Grafana Labs’ very own Tom Wilkie gave a fantastic talk at GrafanaCon EU on his microservices-oriented monitoring philosophy called the Red Method, and how to use it in combination with other methods to gain insight on the happiness of both your users and machines.

Plaid.com’s Monitoring System for 9600+ Integrations: Plaid.com is a fintech company that integrates with over 9,600 financial institutions. This article discusses how Plaid.com used AWS Kinesis, Prometheus, Alertmanager and Grafana to solve the challenges of scalability and latency to monitor their vast number of integrations.

How to Monitor an Ubuntu Server with Grafana & Prometheus: This installation guide will have you up and running and monitoring your Linux servers in no time.

Efficient IoT with the ESP8266, Protocol Buffers, Grafana, Go, and Kubernetes: Efficient storage and serialization of data can have a large impact on the battery life and performance of your IoT devices. This write up discusses Protocol Buffers for efficient binary data encoding with IoT devices and getting that data into a TSDB and onto a Grafana dashboard.

Build a Homelab Dashboard: Part 6, Grafana Introduction: Part 6 of the Homelab dashboard series walks us through installing and configuring Grafana and building the first dashboard. You may want to start at part 1 to get up to speed.


GrafanaCon LA
CFP Now Open!

Join us in Los Angeles, California February 25-26, 2019 for 2 days of talks focused on Grafana and the open source monitoring ecosystem.

Submit You CFP Today


Grafana Plugin Update

This week, Instana updated their plugin to add additional functionality. There are 2 easy ways to update the Instana (or any) plugin – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click at grafana.com.

UPDATED PLUGIN

Instana Data Source – Version 1.1.0 has been released which adds a configuration that lets users enable the new Instana application model, yielding new/additional entity types and metrics.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

Uh….


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

The RED Method: How to Instrument Your Services

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/

Paul Dix
Tom Wilkie – Grafana Labs

At GrafanaCon EU in March, we had the pleasure of introducing one of Grafana Labs’ newest team members, Tom Wilkie, who joined as VP, Product with the acquisition of Kausal.

And he came bearing gifts: his popular talk about the RED Method of monitoring microservices, which he created in 2015.

The USE Method

In the talk, Tom first went over the USE Method of instrumenting, which has been popularized by Brendan Gregg:

For every resource, monitor:

  • Utilization (% time that the resource was busy)
  • Saturation (amount of work resource has to do, often queue length)
  • Errors (count of error events)

“It’s a way of building a checklist that goes through everything,” he said. “It helps you know what you don’t know.” But he pointed out that this method, while useful, is somewhat abstract. Applying it to memory, for instance, is hard. “Memory utilization is tricky. What is it? Do you count caches toward utilization?” he said. “Saturation of memory is kind of a weird one… And what is a memory error? And how do you count them in Linux?”

The RED Method

Tom then turned to his RED Method, which he created after a new employee asked what his monitoring philosophy was. “The USE Method doesn’t really apply to services; it applies to hardware, network disks, things like this,” Tom said. “We really wanted a microservices-oriented monitoring philosophy, so we came up with the RED Method.”

For every resource, monitor:

  • Rate (the number of requests per second)
  • Errors (the number of those requests that are failing)
  • Duration (the amount of time those requests take)

(See his presentation slides on the Prometheus implementation of the RED Method)

“Everyone should understand the error rate, the request rate, and then some distribution of latency for those requests,” Tom explained. “You model this for every single service in your architecture, and this gives you a nice, consistent view of how your architecture is behaving. Giving this kind of consistency across services allows you to scale your operational team, and allows you to put people on call for code they didn’t write.”

Plus, he pointed out, “The RED Method is a good proxy to how happy your customers will be. If you’ve got a high error rate, that’s basically going through to your users and they’re getting page load errors. If you’ve got a high duration, your website is slow. So these are really good metrics for building meaningful alerts and measuring your SLA.”

The Four Golden Signals

Finally, Tom looked at a third method: The Four Golden Signals, which is from Google’s SRE book.

For each service, monitor:

  • Latency (time taken to serve a request)
  • Traffic (how much demand is placed on your system)
  • Errors (rate of requests that are failing)
  • Saturation (how “full” your service is)

This is basically the same as the RED Method, but includes saturation. He explained one approach to measuring saturation: “With kube-state-metrics, a little job you run on your Kubernetes cluster that scrapes the Kubernetes API and exports really interesting metadata about your jobs and services and pods and so on, you can compare the amount of CPU a service is using against its quota. Like how much it should be using, or how much is it allowed to use, as a proportion between 1 and 0. This gives you a measure of how ‘full’ your service is, or some proxy for how full your service is at least. And this is super useful because you could for instance build an alert on this.”

Two Sides of the Same Coin

Tom recommended using the USE and RED Methods together. “It’s like the RED Method is about caring about your users and how happy they are,” Tom said, “and the USE Method is about caring about your machines and how happy they are. It’s really just two different views on the same system. They’re complimentary.”

Watch Tom’s full talk in the video below.

Video: The RED Method: How to Instrument Your Services

timeShift(GrafanaBuzz, 1w) Issue 54

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/27/timeshiftgrafanabuzz-1w-issue-54/

Welcome to TimeShift

This week’s big news is that Grafana v5.2.2 was released and includes fixes for Prometheus graphs, dashboard links, loading external plugins, SQL connection leaks, and more. Also, check out our new Grafana Flux plugin and monitor temperature and forecasts using Grafana and the openweathermap API.

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.2

Bug Fixes

  • Prometheus: Fix graph panel bar width issue in aligned prometheus queries #12379
  • Dashboard: Dashboard links not updated when changing variables #12506
  • Postgres/MySQL/MSSQL: Fix connection leak #12636 #9827
  • Plugins: Fix loading of external plugins #12551
  • Dashboard: Remove unwanted scrollbars in embedded panels #12589
  • Prometheus: Prevent error using $__interval_ms in query #12533, thx @mtanda

See everything new in Grafana v5.2.2.

Download Grafana 5.2.2 Now


From the Blogosphere

How the New Influx Query Engine Was Designed—And How to Use It With Grafana: Learn about the design of InfluxData’s new functional query engine Flux in a recap from GrafanaCon EU, and check out the new Flux plugin for Grafana to start using it today!

How to Collect Docker Daemon Metrics: Docker ships with native Prometheus integration – the Docker daemon can generate metrics about it and let you scrape that. Check out how to gather Docker daemon metrics in Linux and MacOS.

Grafana + Prometheus = Awesome: A walkthrough of setting up and configuring a Prometheus/Grafana stack with a few example queries to get you started with your first dashboard.

Playing with Grafana and weather APIs: After receiving a Beewi temperature sensor, Gonzalo wanted to visualize the data from the openweathermap API using Grafana. This article shows you how to collect the data and visualize temperature, UV index, current weather conditions, and forecast in a Grafana dashboard.

Monitoring temperature and humidity with a Raspberry Pi 3, DHT22 sensor, InfluxDB and Grafana: In a second weather related article, Sam needed to track the temperature of his garage for a very important reason – brewing beer. He shares his equipment, setup, scripts and how he intends to improve the setup in the future.

Integration for ServiceNow table API and Grafana: In a follow up on the article about integrating ServiceNow with Grafana, we learn about the newly re-written snow-grafana-proxy plugin, along with the new configuration options, parameters and command line options.


Grafana Plugins

OpenNMS updated their Helm app this week. Depending on your Grafana installation, there are 2 ways to update your plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

UPDATED PLUGIN

OpenNMS Helm App – Version 2.0.0 has been released. It includes a new datasource for querying flow data from OpenNMS and requires Grafana 5.x or greater.

Other new features are:

  • Support for “fallback” attributes to the performance datasource.
  • The ability to configure query timeouts for all of the datasources in the OpenNMS helm plugin.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

This is awesome! Thanks for sharing.


We’re Hiring!

We’ve added new open positions at Grafana Labs! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

How the New Influx Query Engine Was Designed—And How to Use It With Grafana

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/26/how-the-new-influx-query-engine-was-designedand-how-to-use-it-with-grafana/

Paul Dix
Left to right: Paul Dix – InfluxData, David Kaltschmidt – Grafana Labs

Flux, the long-awaited new functional query processing engine for InfluxDB, has finally landed. If you’re curious to learn more about the hows and whys of its design, check out this GrafanaCon EU session with InfluxData Cofounder and CTO Paul Dix. Also we’d like to share a recent presentation from David Kaltschmidt, Director, UX for Grafana Labs on the new Flux support in Grafana!

Video: Full Flux Presentation from GrafanaCon:

Download the presentation slides.

Flux + Grafana = ❤

Given the excitement about Flux, we just released a plugin that adds Flux support to Grafana. “Flux is powerful stuff, and you can use it with Grafana today,” says David Kaltschmidt, Director of UX for Grafana Labs, who recently did a live demo of the plugin. Simply download the new plugin from grafana.com to install it. The one requirement is that “you need a super recent version of Grafana (v5.1+), because we expanded the data source plugin model,” he says. “The plugin has syntax highlighting, tag completion, raw table preview, and inline function documentation, which I think is really helpful especially if you’re starting a new language.”

Video: Flux Support in Grafana:

Download the Flux Grafana Plugin Now


WTF(lux)

Dix explained the evolution of his thinking about how best to work with time series data. The first version of Flux (then known as IFQL – Influx Functional Query Language) was based on the same tech that he’d used for errplane, a SaaS server monitoring product, which included Rest API. “The thing that I learned with the Rest API was people understood SQL as a query language for working with their data,” said Dix. “When I kickstarted InfluxDB [in November 2013], the innovation that we introduced was basically a language that looks kind of like SQL.”

To SQL or Not to SQL

A year later, Dix began giving talks with different user groups, and he polled the audience: “Does it make more sense to organize your metrics as a hierarchy, like Graphite does, or does it make more sense to organize it as tags?” Tags won out, and that was the new API introduced in the next version. But on the second question, whether SQL or a functional language was superior for working with this kind of data, the rooms were evenly split. “While I thought functional was the way to do it, I didn’t want to make the change because Influx was gaining in popularity and a lot of people told me the reason we love the project is because of the SQL query language. So basically I was too afraid to switch.”

Fast forward to today, and “there are a bunch of feature requests that people have,” said Dix. “We want to deliver those features, but we were kind of hemmed in by this weird SQL query language that we had created.” And with the new Flux, all of these features are included. “We were able to implement these features in a few months, that we hadn’t been able to get to in two years,” Dix said. “The problem was the query language that we had was really really hard to improve and to change.”

The same could be said with TICKscript, the language of InfluxDB’s monitoring piece, Kapacitor. It was designed to be more functional, but “It is really hard to debug, and it has a super steep learning curve because this doesn’t look like anything anybody’s familiar with,” Dix said. “We essentially created a second language for working with time series data in our platform.”

At the beginning of 2017, Dix decided a rethink was in order: “If I had to do everything brand new, if I had to start Influx today, what would I do?”

One Language to Unite them All

The answer turned out to be pretty simple: “Kapacitor is just background processing, but the truth is, it is the query engine,” he said. “If you’re doing a batch job, that looks exactly like a query a user would submit to a database. InfluxDB is batch interactive. It’s users querying the database. So basically when I thought about 2.0 and what I wanted to accomplish with it, I wanted to unify the API and the language so there’s just one thing you have to learn: one language to unite them all.”

Dix’s solution should also help increase feature velocity, enabled by the fact that the storage will be decoupled from the compute. “We can deploy these features frequently, and the risk of shipping a code update in the query language is not nearly as high because you know your data is safe,” he said. “One feature requests we have frequently now is that people want it to be multi-tenant. The nice thing about having these query processors be stateless systems is you can containerize them and you can put them in a lockbox and say this person can’t mess with this person.”

IFQL Flux

The decision to rebrand IFQL as Flux came in large part “because it’s selling it short to call it a query language,” said Dix. “I think of it more as a language for working with data.”

As for the design philosophy, Dix said it’s this: “You want a user interface for the masses. I know the hardcore people want to write their queries, but my theory is most people don’t want to write queries. They want a point-and-click UI, and they want to see their data.”

Other Guiding Principles for Flux:
  • It’s optimized for readability.
  • It’s flexible.
  • It’s easy for people to contribute to it.
  • Code sharing and reuse will be enabled.
Video Demo Features:
  • More complex windowing behavior enabled than before.
  • Anonymous functions, and pipe forward operators to make clear that data is being piped in from one function to another.
  • Named parameters only, no provisional parameters; for added flexibility and readability.
  • A record looks like a flat object or a row in a table. Anything that starts with an underscore is reserved for system attributes; all data within an InfluxDB would have _measurement and _field.
  • From a user perspective, functions operate in isolation. They have an input and an output.
  • Order doesn’t matter in terms of optimization, but it matters for semantics: “The planner should optimize the query so that it runs in the most efficient way.”
  • There are closures in the language.
  • It’s easy to add functions in the language: “You can define an entire function in three pages’ worth of code.”
  • It will have imports and namespaces.
  • You can query based on metadata in a specific time range.

As Dix put it, “Ideally, I want people to build the language around their use case, so if you have common boilerplate, you could do something like define a function where you just say, ‘Okay, I know the boilerplate is: I’m always asking for a database, I’m always going to ask for a measurement and a field. So I can just do that. And I can pipe the result of that function to the other things.’”

Now that the language has been formalized, it is shipping with the enterprise version of InfluxDB, and there will be commercial support coming soon. “This is a lot of work for the API 2.0 effort,” said Dix. “Flux is the first bit of that.”

Check out Paul’s update to #Fluxlang from InfluxDays London.

timeShift(GrafanaBuzz, 1w) Issue 53

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/20/timeshiftgrafanabuzz-1w-issue-53/

Welcome to TimeShift

This week we’re showing off 2 more brand-new plugins, plus a case study showing how PingCAP troubleshoots more than a thousand metrics from their TiDB clusters. Also, Tweet of the Week returns with a graph celebrating France’s World Cup victory. Enjoy!

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.1

New Features

See everything new in Grafana v5.2.1.

Download Grafana 5.2.1 Now


From the Blogosphere

How Grafana Helps PingCAP Troubleshoot TiDB Deployments: A case study on how PingCAP diagnoses and troubleshoots issues from a thousand metrics on their open source TiDB clusters using Grafana.

Azure Log into Grafana using Stream Analytics jobs: Learn how to export data using Azure Stream Analytics jobs in Azure SQL and visualize the data in Grafana.

How to Monitor ArangoDB using collectd, Prometheus and Grafana: The folks from ArangoDB have put together a tutorial on how to install, configure and visualize metrics for ArangoDB.

Room temperature monitoring with Philips Hue, Node-RED and Grafana: This how-to guide explains how to use a Philips Hue Smart Home lighting system to measure the temperature in a room and visualize the data in a Grafana Dashboard.

Visualizing Data using Grafana: We’ve seen an explosion of interest in using Grafana to visualize data for home automation. This tutorial shows you how to install and run Grafana on a Raspberry Pi with InfluxDB, and Home Assistant.


Grafana Plugins

This week we have an update to the Clickhouse data source and 2 new plugins to share. Depending on your installation, there are 2 ways to update plugins – for on-prem Grafana, use the grafana-cli tool, or for Hosted Grafana update with one-click.

NEW PLUGIN

Akumuli Data source – The Akumuli timeseries database now has a published Grafana datasource plugin. It contains a full-featured query editor and supports templating.

Install

NEW PLUGIN

TrafficLight Panel – A new status panel with traffic lights has been published. The TrafficLight Panel allows you to add a traffic light per query and place it on an image background.

Install

UPDATED PLUGIN

Clickhouse Data source – Added a performance improvement that optimizes memory use for range time series. See this PR for more details.

Install


Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard or monitoring related tweet and show it off! #monitoringLove

Laissez le bon temps rouler!


We’re Hiring!

We’re looking for passionate and talented engineers for positions in New York/Stockholm and remote! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions


Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.

timeShift(GrafanaBuzz, 1w) Issue 52

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/07/13/timeshiftgrafanabuzz-1w-issue-52/

Welcome to TimeShift

It’s good to be back from our short holiday break. This week we have articles on the world’s fastest internet (with a shoutout to Prometheus and Grafana), visualizing real-time and historic weather data, making teams more autonomous, and Grafana + Prometheus + Postgres + TimescaleDB. Hope you have a happy and safe Friday the 13th – see you next week!

Have an article you’d like included in an upcoming TimeShift? Contact Us.


Latest Stable Release: Grafana 5.2.1

New Features

See everything new in Grafana v5.2.1.

Download Grafana 5.2.1 Now


From the Blogosphere

Ballerina Makeover with Grafana: In this guest blog post, the folks from cloud-native programming language Ballerina show you how you can easily visualize metrics from a Ballerina service with Grafana, walking you step by step through the installation and configuration of the components.

How Grafana gives our teams more autonomy with their data: As the saying goes, “Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime,” Vitor shows how using Grafana has empowered teams across Intuit to get hands on and explore their own data, and provides a step by step guide to starting this in your organization.

World’s Fastest Internet – 1.6 TERABITS per Second: Get an inside look at the world’s fastest internet connection which powered a recent LAN party in Sweden (monitored by Prometheus and Grafana)!

Visualize your data with Grafana: Learn how to visualize some real-time and historical weather data in a Grafana dashboard using the nio platform, Elasticsearch and the OpenWeatherMap API. Very cool stuff.

Uniting SQL and NoSQL for Monitoring: Why PostgreSQL is the ultimate data store for Prometheus: This article shows you how to use Prometheus, PostgreSQL + TimescaleDB, and Grafana for storing, analyzing, and visualizing metrics. If you’re not familiar with TimescaleDB, they spoke at our last GrafanaCon – checkout Erik’s talk.

Introducing Pagination for Grafana Scripted Dashboards: The new release of Würth Phoenix’s NetEye 4 takes advantage of Grafana’s scripted dashboards, but adds some new functionality. Sometimes when you have a large number of panels on a single dashboard it can be slow to load. Würth Phoenix has added the concept of pagination to their default analytics dashboard and they show you how they did it.

Highly Available Grafana running on AWS Fargate and RDS Aurora: Set up a fully operational Grafana that’s load balanced and backed by Amazon Aurora.


We’re Hiring!

We’re looking for passionate and talented engineers for positions in New York/Stockholm and remote! Do you love open source software? Do you thrive on tackling complex challenges to build the future? Want to work with awesome people? Be the next to join our team!

View our Open Positions

Upcoming Events

In between code pushes we like to speak at, sponsor and attend all kinds of conferences and meetups. We also like to make sure we mention other Grafana-related events happening all over the world. If you’re putting on just such an event, let us know and we’ll list it here.

TimeSeries Boston | Boston, MA – July 17, 2018:

Jacob Lisi: What does Kubernetes Look Like? Performance Monitoring & Visualization with Grafana – Monitoring Kubernetes is vital to understanding the health and performance of a cluster, but which metrics are most important to add to your dashboards and alert on? Jacob will discuss how to most effectively monitor and visualize your Kubernetes cluster using the Grafana Kubernetes plugin and PromQL. Some of the topics of discussion include(1) how and what to collect when monitoring Kubernetes, (2) how to bring your Grafana dashboards to the next level by using Kubernetes as a data-source, and (3) what to do when managing multiple clusters. All of these topics and more will be discussed to help people get the most out of their Kubernetes monitoring.

RSVP

PromCon 2018 | Munich, Germany – August 9-10, 2018:

Tom Wilkie: Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts, and Exporters – Prometheus offers powerful open source monitoring and alerting – but that comes with higher degrees of freedom, making pre-configured monitoring “packages” hard to build. Simultaneously, it’s becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration.

In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” – extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul, Vault, and Cassandra.

David Kaltschmidt: Explore Your Prometheus Data in Grafana – Grafana is the de-facto dashboarding solution for Prometheus. Now imagine you received a page. Grafana is often the starting point for your incident response. You look at a time series panel, form a hypothesis, and would like to dive deeper. We’ve built a whole new section that allows you to do just that by iterating quickly through Prometheus queries while leaving your dashboards intact. I’ll show-case our new Explore UI and how it can fit into your workflows for both troubleshooting and data exploration.

We’re also a proud PromCon 2018 sponsor of the evening social event and a diversity scholarship. We hope to see you there!

Register Now

2018 Sensu Summit | Portland, OR – August 22-23, 2018:

We are a proud sponsor of this year’s Sensu Summit! Come enjoy Portland in the summer and learn a ton from the sharpest operations engineers in monitoring!

More Info

CloudNative London 2018 | London, United Kingdom – September 26-28, 2018:

Tom Wilkie: Monitoring Kubernetes With Prometheus – In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin.

Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Register Now


How are we doing?

Hope you enjoyed this issue of TimeShift. What do you think? Are there other types of content you’d like to see here? Submit a comment on this issue below, or post something at our community forum.

Follow us on Twitter, like us on Facebook, and join the Grafana Labs community.