Getting Up and Running With Grafana Loki

Published in

The Startup

6 min readNov 6, 2020

Grafana Labs recently launched the v2 of their log aggregation solution — Loki. Loki makes some interesting choices around log aggregation to support horizontal scaling and to keep the operational costs low. It is inspired by Prometheus. Loki supports querying the logs for errors, patterns and much more.

In this post, we’ll go over the setup for Loki on Kubernetes. Before we setup Loki, we’ll install some microservices that we can use to generate logs and events and then consume those via Loki. We’ll use FluentBit to forward logs to Loki and then use the Loki datasource on Grafana to analyze the logs.

Setup the sample application

We’ll use an Istio enabled namespace on a local Kubernetes cluster(docker-desktop to be precise) to install the microservices-demo application. If you don’t have Istio installed, you can follow the step from my previous post to add it to your Kubernetes cluster. Once the Istio setup is done, let’s create the new namespace

$ kubectl create ns demo
$ kubectl label namespace demo istio-injection=enabled #Enable istio sidecar injection

Next, let’s install the microservices-demo app.

# Clone the repo with all microservices to be deployed
$ git clone git@github.com:GoogleCloudPlatform/microservices-demo.git$ cd microservices-demo# Install the services
$ kubectl apply -n demo  -f ./release/kubernetes-manifests.yaml# Setup istio gateway and virtual services for routing
$ kubectl apply -n demo  -f ./release/istio-manifests.yaml

Once the installation finishes, quickly verify the running pods using kubectl -n demo get pods . The demo application can now be accessed at the Istio gateway endpoint for your environment.

Microservice demo application — Microservices demo application

The microservices demo application is an online boutique backed my multiple services written in different programming languages. The application can be used to add items to a cart, change transaction currencies and perform a checkout. All these actions are powered by different services.

Install Loki and FluentBit

After the sample application is up and running, let’s begin installing Loki.

# Add the loki helm chart
$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo update

The above step would pull the Loki installation chart and it is now ready to be installed.

# Create the namespace to install loki
$ kubectl create ns loki# Apply the loki helm chart
$ helm upgrade --install loki --namespace=loki loki/loki  --set fluent-bit.enabled=true,promtail.enabled=false,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=true,prometheus.server.persistentVolume.enabled=true

Ensure that fluent-bit.enabled=true is set to use FluentBit as the log forwarder.

Next, install FluentBit

$ helm upgrade --install fluent-bit --namespace=loki loki/fluent-bit --set loki.serviceName=loki.loki.svc.cluster.local

Check the running pods in the loki namespace with kubectl -n loki get pods. You might see the loki pod crashing with an error level=error caller=out_loki.go:49 [flb-go="failed to launch" error="failed to parse BatchSize: 1.048576e+06" . This is a known issue with value escaping on the FluentBit config map for Loki. The workaround is pretty straightforward

$ kubectl -n loki edit cm/fluent-bit-fluent-bit-loki
# Replace the value of BatchSize to 1048576 under the [Output] section of the config. Save the configmap

At this point you should see all services and pods in the loki namespace working fine. To make sure that things are working as expected, we’ll install Grafana to do some queries using Loki.

Install Grafana

Let’s pull the Grafana chart for helm

$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo update

We’ll use the existing namespace to install Grafana although this is not a requirement.

$ helm upgrade --install grafana --namespace=loki grafana/grafana

You can use kubectl -n loki get pods to verify that all pods are running as expected.

Checking if pods are running fine after Loki, FluentBit and Grafana have been installed.

Adding Loki to Grafana

To access the Grafana dashboard, port forward to the Grafana service and open the url http://localhost:8080 in the browser.

$ kubectl -n loki port-forward svc/grafana 8080:80
# Navigate to http://localhost:8080 in the browser

You’ll be prompted for a username and a password. Use the username admin and follow the steps below to get the password

$ kubectl get secret --namespace loki grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
# The password for the admin user

Once you are logged in, we can add Loki as a data source. From the Left side-panel, select Configuration -> Data Sources

In the Data sources view, click Add data source and then select Loki

Give the datasource a name and then set the Loki URL. The Grafana instance should be able to access the Loki URL. In our case, since both Loki and Grafana are running in the same namespace and don’t need to use any authentication scheme, no other configuration is required. Click “Save & Test”. You should see a confirmation message like “Data source connected and labels found.”

Queries with Loki

Now that Grafana has been connected to Loki, we can query the logs from our sample application deployed in the demo namespace to observe its behaviour. We can hunt for errors, lookup transactions by trace id and do many more powerful things including visualizing the patterns and trends. The query language for Loki is called LogQL.

In the Grafana application, select “Explore” from the side panel. Select the Loki data source added in the previous step. In the query box, add {namespace="demo"} and hit “Run query”. You can see the logs being returned

A basic LogQL query using Loki in Grafana

You can add more filters and labels to narrow the logs you want to see. eg: To view the logs coming from the Istio-sidecars in the demo namespace, use the query {container="istio-proxy", namespace="demo"} . We can also look for http requests that don’t end with a desired status code {namespace="demo"} |= "http.resp.status"!= "200"`

We can also perform some aggregations over time on the data from the logs

Aggregating http status codes in the demo namespace in the last 5 minutes. The green line above represents 302 status code and the yellow represents 200 status code. No other status codes in the last 5 mins.

You can do some advance operations and learn more about LogQL here. Your query history is saved so that you can revisit a query in the future if you’d like. The query inspector reports metrics about the query execution.

To generate some logs with a spread, you can use the sample application to perform some actions like add to card and checkout.

Lemme know here or on twitter about your Loki setup journey and about the use cases you are able to complement with Loki.