Monitor a Golang application with Prometheus and Grafana

Learn how to containerize a Golang application and monitor it with Prometheus and Grafana.

The guide teaches you how to containerize a Golang application and monitor it with Prometheus and Grafana.

Acknowledgment
Docker would like to thank Pradumna Saraf for his contribution to this guide.

Overview

To make sure your application is working as intended, monitoring is important. One of the most popular monitoring tools is Prometheus. Prometheus is an open-source monitoring and alerting toolkit that is designed for reliability and scalability. It collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. To visualize the metrics, you can use Grafana. Grafana is an open-source platform for monitoring and observability that allows you to query, visualize, alert on, and understand your metrics no matter where they are stored.

In this guide, you will be creating a Golang server with some endpoints to simulate a real-world application. Then you will expose metrics from the server using Prometheus. Finally, you will visualize the metrics using Grafana. You will containerize the Golang application, and using the Docker Compose file, you will connect all the services: Golang, Prometheus, and Grafana.

What will you learn?

Create a Golang application with custom Prometheus metrics.
Containerize a Golang application.
Use Docker Compose to run multiple services and connect them together to monitor a Golang application with Prometheus and Grafana.
Visualize the metrics using Grafana dashboards.

Prerequisites

A good understanding of Golang is assumed.
You must me familiar with Prometheus and creating dashboards in Grafana.
You must have familiarity with Docker concepts like containers, images, and Dockerfiles. If you are new to Docker, you can start with the Docker basics guide.

Next steps

You will create a Golang server and expose metrics using Prometheus.

Building the application

Prerequisites

You have a Git client. The examples in this section use a command-line based Git client, but you can use any client.

You will be creating a Golang server with some endpoints to simulate a real-world application. Then you will expose metrics from the server using Prometheus.

Getting the sample application

Clone the sample application to use with this guide. Open a terminal, change directory to a directory that you want to work in, and run the following command to clone the repository:

$ git clone https://github.com/dockersamples/go-prometheus-monitoring.git

Once you cloned you will see the following content structure inside go-prometheus-monitoring directory,

go-prometheus-monitoring
├── CONTRIBUTING.md
├── Docker
│   ├── grafana.yml
│   └── prometheus.yml
├── dashboard.json
├── Dockerfile
├── LICENSE
├── README.md
├── compose.yaml
├── go.mod
├── go.sum
└── main.go

main.go - The entry point of the application.
go.mod and go.sum - Go module files.
Dockerfile - Dockerfile used to build the app.
Docker/ - Contains the Docker Compose configuration files for Grafana and Prometheus.
compose.yaml - Compose file to launch everything (Golang app, Prometheus, and Grafana).
dashboard.json - Grafana dashboard configuration file.
Dockerfile - Dockerfile used to build the Golang app.
compose.yaml - Docker Compose file to launch everything (Golang app, Prometheus, and Grafana).
Other files are for licensing and documentation purposes.

Understanding the application

The following is the complete logic of the application you will find in main.go.

package main

import (
	"strconv"

	"github.com/gin-gonic/gin"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// Define metrics
var (
	HttpRequestTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_total",
		Help: "Total number of requests processed by the API",
	}, []string{"path", "status"})

	HttpRequestErrorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_error_total",
		Help: "Total number of errors returned by the API",
	}, []string{"path", "status"})
)

// Custom registry (without default Go metrics)
var customRegistry = prometheus.NewRegistry()

// Register metrics with custom registry
func init() {
	customRegistry.MustRegister(HttpRequestTotal, HttpRequestErrorTotal)
}

func main() {
	router := gin.Default()

	// Register /metrics before middleware
	router.GET("/metrics", PrometheusHandler())
	
	router.Use(RequestMetricsMiddleware())
	router.GET("/health", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Up and running!",
		})
	})
	router.GET("/v1/users", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello from /v1/users",
		})
	})

	router.Run(":8000")
}

// Custom metrics handler with custom registry
func PrometheusHandler() gin.HandlerFunc {
	h := promhttp.HandlerFor(customRegistry, promhttp.HandlerOpts{})
	return func(c *gin.Context) {
		h.ServeHTTP(c.Writer, c.Request)
	}
}

// Middleware to record incoming requests metrics
func RequestMetricsMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		path := c.Request.URL.Path
		c.Next()
		status := c.Writer.Status()
		if status < 400 {
			HttpRequestTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		} else {
			HttpRequestErrorTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		}
	}
}

In this part of the code, you have imported the required packages gin, prometheus, and promhttp. Then you have defined a couple of variables, HttpRequestTotal and HttpRequestErrorTotal are Prometheus counter metrics, and customRegistry is a custom registry that will be used to register these metrics. The name of the metric is a string that you can use to identify the metric. The help string is a string that will be shown when you query the /metrics endpoint to understand the metric. The reason you are using the custom registry is so avoid the default Go metrics that are registered by default by the Prometheus client. Then using the init function you are registering the metrics with the custom registry.

import (
	"strconv"

	"github.com/gin-gonic/gin"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// Define metrics
var (
	HttpRequestTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_total",
		Help: "Total number of requests processed by the API",
	}, []string{"path", "status"})

	HttpRequestErrorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_error_total",
		Help: "Total number of errors returned by the API",
	}, []string{"path", "status"})
)

// Custom registry (without default Go metrics)
var customRegistry = prometheus.NewRegistry()

// Register metrics with custom registry
func init() {
	customRegistry.MustRegister(HttpRequestTotal, HttpRequestErrorTotal)
}

In the main function, you have created a new instance of the gin framework and created three routes. You can see the health endpoint that is on path /health that will return a JSON with {"message": "Up and running!"} and the /v1/users endpoint that will return a JSON with {"message": "Hello from /v1/users"}. The third route is for the /metrics endpoint that will return the metrics in the Prometheus format. Then you have RequestMetricsMiddleware middleware, it will be called for every request made to the API. It will record the incoming requests metrics like status codes and paths. Finally, you are running the gin application on port 8000.

func main() {
	router := gin.Default()

	// Register /metrics before middleware
	router.GET("/metrics", PrometheusHandler())
	
	router.Use(RequestMetricsMiddleware())
	router.GET("/health", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Up and running!",
		})
	})
	router.GET("/v1/users", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello from /v1/users",
		})
	})

	router.Run(":8000")
}

Now comes the middleware function RequestMetricsMiddleware. This function is called for every request made to the API. It increments the HttpRequestTotal counter (different counter for different paths and status codes) if the status code is less than or equal to 400. If the status code is greater than 400, it increments the HttpRequestErrorTotal counter (different counter for different paths and status codes). The PrometheusHandler function is the custom handler that will be called for the /metrics endpoint. It will return the metrics in the Prometheus format.

// Custom metrics handler with custom registry
func PrometheusHandler() gin.HandlerFunc {
	h := promhttp.HandlerFor(customRegistry, promhttp.HandlerOpts{})
	return func(c *gin.Context) {
		h.ServeHTTP(c.Writer, c.Request)
	}
}

// Middleware to record incoming requests metrics
func RequestMetricsMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		path := c.Request.URL.Path
		c.Next()
		status := c.Writer.Status()
		if status < 400 {
			HttpRequestTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		} else {
			HttpRequestErrorTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		}
	}
}

That's it, this was the complete gist of the application. Now it's time to run and test if the app is registering metrics correctly.

Running the application

Make sure you are still inside go-prometheus-monitoring directory in the terminal, and run the following command. Install the dependencies by running go mod tidy and then build and run the application by running go run main.go. Then visit http://localhost:8000/health or http://localhost:8000/v1/users. You should see the output {"message": "Up and running!"} or {"message": "Hello from /v1/users"}. If you are able to see this then your app is successfully up and running.

Now, check your application's metrics by accessing the /metrics endpoint. Open http://localhost:8000/metrics in your browser. You should see similar output to the following.

# HELP api_http_request_error_total Total number of errors returned by the API
# TYPE api_http_request_error_total counter
api_http_request_error_total{path="/",status="404"} 1
api_http_request_error_total{path="//v1/users",status="404"} 1
api_http_request_error_total{path="/favicon.ico",status="404"} 1
# HELP api_http_request_total Total number of requests processed by the API
# TYPE api_http_request_total counter
api_http_request_total{path="/health",status="200"} 2
api_http_request_total{path="/v1/users",status="200"} 1

In the terminal, press ctrl + c to stop the application.

Note
If you don't want to run the application locally, and want to run it in a Docker container, skip to next page where you create a Dockerfile and containerize the application.

Summary

In this section, you learned how to create a Golang app to register metrics with Prometheus. By implementing middleware functions, you were able to increment the counters based on the request path and status codes.

Next steps

In the next section, you'll learn how to containerize your application.

Containerize a Golang application

Containerization helps you bundle the application and its dependencies into a single package called a container. This package can run on any platform without worrying about the environment. In this section, you will learn how to containerize a Golang application using Docker.

To containerize a Golang application, you first need to create a Dockerfile. The Dockerfile contains instructions to build and run the application in a container. Also, when creating a Dockerfile, you can follow different sets of best practices to optimize the image size and make it more secure.

Creating a Dockerfile

Create a new file named Dockerfile in the root directory of your Golang application. The Dockerfile contains instructions to build and run the application in a container.

The following is a Dockerfile for a Golang application. You will also find this file in the go-prometheus-monitoring directory.

# Use the official Golang image as the base
FROM golang:1.24-alpine AS builder

# Set environment variables
ENV CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64

# Set working directory inside the container
WORKDIR /build

# Copy go.mod and go.sum files for dependency installation
COPY go.mod go.sum ./

# Download dependencies
RUN go mod download

# Copy the entire application source
COPY . .

# Build the Go binary
RUN go build -o /app .

# Final lightweight stage
FROM alpine:3.21 AS final

# Copy the compiled binary from the builder stage
COPY --from=builder /app /bin/app

# Expose the application's port
EXPOSE 8000

# Run the application
CMD ["bin/app"]

Understanding the Dockerfile

The Dockerfile consists of two stages:

Build stage: This stage uses the official Golang image as the base and sets the necessary environment variables. It also sets the working directory inside the container, copies the go.mod and go.sum files for dependency installation, downloads the dependencies, copies the entire application source, and builds the Go binary.
You use the golang:1.24-alpine image as the base image for the build stage. The CGO_ENABLED=0 environment variable disables CGO, which is useful for building static binaries. You also set the GOOS and GOARCH environment variables to linux and amd64, respectively, to build the binary for the Linux platform.
Final stage: This stage uses the official Alpine image as the base and copies the compiled binary from the build stage. It also exposes the application's port and runs the application.
You use the alpine:3.21 image as the base image for the final stage. You copy the compiled binary from the build stage to the final image. You expose the application's port using the EXPOSE instruction and run the application using the CMD instruction.
Apart from the multi-stage build, the Dockerfile also follows best practices such as using the official images, setting the working directory, and copying only the necessary files to the final image. You can further optimize the Dockerfile by other best practices.

Build the Docker image and run the application

One you have the Dockerfile, you can build the Docker image and run the application in a container.

To build the Docker image, run the following command in the terminal:

$ docker build -t go-api:latest .

After building the image, you can run the application in a container using the following command:

$ docker run -p 8000:8000 go-api:latest

The application will start running inside the container, and you can access it at http://localhost:8000. You can also check the running containers using the docker ps command.

$ docker ps

Summary

In this section, you learned how to containerize a Golang application using a Dockerfile. You created a multi-stage Dockerfile to build and run the application in a container. You also learned about best practices to optimize the Docker image size and make it more secure.

Related information:

Next steps

In the next section, you will learn how to use Docker Compose to connect and run multiple services together to monitor a Golang application with Prometheus and Grafana.

Connecting services with Docker Compose

Now that you have containerized the Golang application, you will use Docker Compose to connect your services together. You will connect the Golang application, Prometheus, and Grafana services together to monitor the Golang application with Prometheus and Grafana.

Creating a Docker Compose file

Create a new file named compose.yml in the root directory of your Golang application. The Docker Compose file contains instructions to run multiple services and connect them together.

Here is a Docker Compose file for a project that uses Golang, Prometheus, and Grafana. You will also find this file in the go-prometheus-monitoring directory.

services:
  api:
    container_name: go-api
    build:
      context: .
      dockerfile: Dockerfile
    image: go-api:latest
    ports:
      - 8000:8000
    networks:
      - go-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
    develop:
      watch:
        - path: .
          action: rebuild
      
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.55.0
    volumes:
      - ./Docker/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    networks:
      - go-network
  
  grafana:
    container_name: grafana
    image: grafana/grafana:11.3.0
    volumes:
      - ./Docker/grafana.yml:/etc/grafana/provisioning/datasources/datasource.yaml
      - grafana-data:/var/lib/grafana
    ports:
      - 3000:3000
    networks:
      - go-network
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=password

volumes:
  grafana-data:

networks:
  go-network:
    driver: bridge

Understanding the Docker Compose file

The Docker Compose file consists of three services:

Golang application service: This service builds the Golang application using the Dockerfile and runs it in a container. It exposes the application's port 8000 and connects to the go-network network. It also defines a health check to monitor the application's health. You have also used healthcheck to monitor the health of the application. The health check runs every 30 seconds and retries 5 times if the health check fails. The health check uses the curl command to check the /health endpoint of the application. Apart from the health check, you have also added a develop section to watch the changes in the application's source code and rebuild the application using the Docker Compose Watch feature.
Prometheus service: This service runs the Prometheus server in a container. It uses the official Prometheus image prom/prometheus:v2.55.0. It exposes the Prometheus server on port 9090 and connects to the go-network network. You have also mounted the prometheus.yml file from the Docker directory which is present in the root directory of your project. The prometheus.yml file contains the Prometheus configuration to scrape the metrics from the Golang application. This is how you connect the Prometheus server to the Golang application.
global: scrape_interval: 10s evaluation_interval: 10s scrape_configs: - job_name: myapp static_configs: - targets: ["api:8000"]
In the prometheus.yml file, you have defined a job named myapp to scrape the metrics from the Golang application. The targets field specifies the target to scrape the metrics from. In this case, the target is the Golang application running on port 8000. The api is the service name of the Golang application in the Docker Compose file. The Prometheus server will scrape the metrics from the Golang application every 10 seconds.
Grafana service: This service runs the Grafana server in a container. It uses the official Grafana image grafana/grafana:11.3.0. It exposes the Grafana server on port 3000 and connects to the go-network network. You have also mounted the grafana.yml file from the Docker directory which is present in the root directory of your project. The grafana.yml file contains the Grafana configuration to add the Prometheus data source. This is how you connect the Grafana server to the Prometheus server. In the environment variables, you have set the Grafana admin user and password, which will be used to log in to the Grafana dashboard.
apiVersion: 1 datasources: - name: Prometheus (Main) type: prometheus url: http://prometheus:9090 isDefault: true
In the grafana.yml file, you have defined a Prometheus data source named Prometheus (Main). The type field specifies the type of the data source, which is prometheus. The url field specifies the URL of the Prometheus server to fetch the metrics from. In this case, the URL is http://prometheus:9090. prometheus is the service name of the Prometheus server in the Docker Compose file. The isDefault field specifies whether the data source is the default data source in Grafana.

Apart from the services, the Docker Compose file also defines a volume named grafana-data to persist the Grafana data and a network named go-network to connect the services together. You have created a custom network go-network to connect the services together. The driver: bridge field specifies the network driver to use for the network.

Building and running the services

Now that you have the Docker Compose file, you can build the services and run them together using Docker Compose.

To build and run the services, run the following command in the terminal:

$ docker compose up

The docker compose up command builds the services defined in the Docker Compose file and runs them together. You will see the similar output in the terminal:

 ✔ Network go-prometheus-monitoring_go-network  Created                                                           0.0s 
 ✔ Container grafana                            Created                                                           0.3s 
 ✔ Container go-api                             Created                                                           0.2s 
 ✔ Container prometheus                         Created                                                           0.3s 
Attaching to go-api, grafana, prometheus
go-api      | [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
go-api      | 
go-api      | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
go-api      |  - using env:     export GIN_MODE=release
go-api      |  - using code:    gin.SetMode(gin.ReleaseMode)
go-api      | 
go-api      | [GIN-debug] GET    /metrics                  --> main.PrometheusHandler.func1 (3 handlers)
go-api      | [GIN-debug] GET    /health                   --> main.main.func1 (4 handlers)
go-api      | [GIN-debug] GET    /v1/users                 --> main.main.func2 (4 handlers)
go-api      | [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
go-api      | Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
go-api      | [GIN-debug] Listening and serving HTTP on :8000
prometheus  | ts=2025-03-15T05:57:06.676Z caller=main.go:627 level=info msg="No time or size retention was set so using the default time retention" duration=15d
prometheus  | ts=2025-03-15T05:57:06.678Z caller=main.go:671 level=info msg="Starting Prometheus Server" mode=server version="(version=2.55.0, branch=HEAD, revision=91d80252c3e528728b0f88d254dd720f6be07cb8)"
grafana     | logger=settings t=2025-03-15T05:57:06.865335506Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
grafana     | logger=settings t=2025-03-15T05:57:06.865337131Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
grafana     | logger=ngalert.state.manager t=2025-03-15T05:57:07.088956839Z level=info msg="State
.
.
grafana     | logger=plugin.angulardetectorsprovider.dynamic t=2025-03-15T05:57:07.530317298Z level=info msg="Patterns update finished" duration=440.489125ms

The services will start running, and you can access the Golang application at http://localhost:8000, Prometheus at http://localhost:9090/health, and Grafana at http://localhost:3000. You can also check the running containers using the docker ps command.

$ docker ps

Summary

In this section, you learned how to connect services together using Docker Compose. You created a Docker Compose file to run multiple services together and connect them using networks. You also learned how to build and run the services using Docker Compose.

Related information:

Next, you will learn how to develop the Golang application with Docker Compose and monitor it with Prometheus and Grafana.

Next steps

In the next section, you will learn how to develop the Golang application with Docker. You will also learn how to use Docker Compose Watch to rebuild the image whenever you make changes to the code. Lastly, you will test the application and visualize the metrics in Grafana using Prometheus as the data source.

Developing your application

In the last section, you saw how using Docker Compose, you can connect your services together. In this section, you will learn how to develop the Golang application with Docker. You will also see how to use Docker Compose Watch to rebuild the image whenever you make changes to the code. Lastly, you will test the application and visualize the metrics in Grafana using Prometheus as the data source.

Developing the application

Now, if you make any changes to your Golang application locally, it needs to reflect in the container, right? To do that, one approach is use the --build flag in Docker Compose after making changes in the code. This will rebuild all the services which have the build instruction in the compose.yml file, in your case, the api service (Golang application).

docker compose up --build

But, this is not the best approach. This is not efficient. Every time you make a change in the code, you need to rebuild manually. This is not is not a good flow for development.

The better approach is to use Docker Compose Watch. In the compose.yml file, under the service api, you have added the develop section. So, it's more like a hot reloading. Whenever you make changes to code (defined in path), it will rebuild the image (or restart depending on the action). This is how you can use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
services:
  api:
    container_name: go-api
    build:
      context: .
      dockerfile: Dockerfile
    image: go-api:latest
    ports:
      - 8000:8000
    networks:
      - go-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
    develop:
      watch:
        - path: .
          action: rebuild

Once you have added the develop section in the compose.yml file, you can use the following command to start the development server:

$ docker compose watch

Now, if you modify your main.go or any other file in the project, the api service will be rebuilt automatically. You will see the following output in the terminal:

Rebuilding service(s) ["api"] after changes were detected...
[+] Building 8.1s (15/15) FINISHED                                                                                                        docker:desktop-linux
 => [api internal] load build definition from Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 704B                                                                                                                      0.0s
 => [api internal] load metadata for docker.io/library/alpine:3.17                                                                                        1.1s
  .                             
 => => exporting manifest list sha256:89ebc86fd51e27c1da440dc20858ff55fe42211a1930c2d51bbdce09f430c7f1                                                    0.0s
 => => naming to docker.io/library/go-api:latest                                                                                                          0.0s
 => => unpacking to docker.io/library/go-api:latest                                                                                                       0.0s
 => [api] resolving provenance for metadata file                                                                                                          0.0s
service(s) ["api"] successfully built

Testing the application

Now that you have your application running, head over to the Grafana dashboard to visualize the metrics you are registering. Open your browser and navigate to http://localhost:3000. You will be greeted with the Grafana login page. The login credentials are the ones provided in Compose file.

Once you are logged in, you can create a new dashboard. While creating dashboard you will notice that is default data source is Prometheus. This is because you have already configured the data source in the grafana.yml file.

The optional settings screen with the options specified.

You can use different panels to visualize the metrics. This guide doesn't go into details of Grafana. You can refer to the Grafana documentation for more information. There is a Bar Gauge panel to visualize the total number of requests from different endpoints. You used the api_http_request_total and api_http_request_error_total metrics to get the data.

You created this panel to visualize the total number of requests from different endpoints to compare the successful and failed requests. For all the good requests, the bar will be green, and for all the failed requests, the bar will be red. Plus it will also show the from which endpoint the request is coming, either it's a successful request or a failed request. If you want to use this panel, you can import the dashboard.json file from the repository you cloned.

Summary

You've come to the end of this guide. You learned how to develop the Golang application with Docker. You also saw how to use Docker Compose Watch to rebuild the image whenever you make changes to the code. Lastly, you tested the application and visualized the metrics in Grafana using Prometheus as the data source.

What can I help you with?

Monitor a Golang application with Prometheus and Grafana