Build a RAG application using Ollama and Docker

This guide demonstrates how to use Docker to deploy Retrieval-Augmented Generation (RAG) models with Ollama.

The Retrieval Augmented Generation (RAG) guide teaches you how to containerize an existing RAG application using Docker. The example application is a RAG that acts like a sommelier, giving you the best pairings between wines and food. In this guide, you’ll learn how to:

• Containerize and run a RAG application
• Set up a local environment to run the complete RAG stack locally for development

Start by containerizing an existing RAG application.

Containerize a RAG application

Overview

This section walks you through containerizing a RAG application using Docker.

Note

You can see more samples of containerized GenAI applications in the GenAI Stack demo applications.

Get the sample application

The sample application used in this guide is an example of RAG application, made by three main components, which are the building blocks for every RAG application. A Large Language Model hosted somewhere, in this case it is hosted in a container and served via Ollama. A vector database, Qdrant, to store the embeddings of local data, and a web application, using Streamlit to offer the best user experience to the user.

Clone the sample application. Open a terminal, change directory to a directory that you want to work in, and run the following command to clone the repository:

$ git clone https://github.com/mfranzon/winy.git

You should now have the following files in your winy directory.

├── winy/
│ ├── .gitignore
│ ├── app/
│ │ ├── main.py
│ │ ├── Dockerfile
| | └── requirements.txt
│ ├── tools/
│ │ ├── create_db.py
│ │ ├── create_embeddings.py
│ │ ├── requirements.txt
│ │ ├── test.py
| | └── download_model.sh
│ ├── docker-compose.yaml
│ ├── wine_database.db
│ ├── LICENSE
│ └── README.md

Containerizing your application: Essentials

Containerizing an application involves packaging it along with its dependencies into a container, which ensures consistency across different environments. Here’s what you need to containerize an app like Winy :

1. Dockerfile: A Dockerfile that contains instructions on how to build a Docker image for your application. It specifies the base image, dependencies, configuration files, and the command to run your application.

2. Docker Compose File: Docker Compose is a tool for defining and running multi-container Docker applications. A Compose file allows you to configure your application's services, networks, and volumes in a single file.

Run the application

Inside the winy directory, run the following command in a terminal.

$ docker compose up --build

Docker builds and runs your application. Depending on your network connection, it may take several minutes to download all the dependencies. You'll see a message like the following in the terminal when the application is running.

server-1  |   You can now view your Streamlit app in your browser.
server-1  |
server-1  |   URL: http://0.0.0.0:8501
server-1  |

Open a browser and view the application at http://localhost:8501. You should see a simple Streamlit application.

The application requires a Qdrant database service and an LLM service to work properly. If you have access to services that you ran outside of Docker, specify the connection information in the docker-compose.yaml.

winy:
  build:
    context: ./app
    dockerfile: Dockerfile
  environment:
    - QDRANT_CLIENT=http://qdrant:6333 # Specifies the url for the qdrant database
    - OLLAMA=http://ollama:11434 # Specifies the url for the ollama service
  container_name: winy
  ports:
    - "8501:8501"
  depends_on:
    - qdrant
    - ollama

If you don't have the services running, continue with this guide to learn how you can run some or all of these services with Docker. Remember that the ollama service is empty; it doesn't have any model. For this reason you need to pull a model before starting to use the RAG application. All the instructions are in the following page.

In the terminal, press ctrl+c to stop the application.

Summary

In this section, you learned how you can containerize and run your RAG application using Docker.

Next steps

In the next section, you'll learn how to properly configure the application with your preferred LLM model, completely locally, using Docker.

Use containers for RAG development

Prerequisites

Complete Containerize a RAG application.

Overview

In this section, you'll learn how to set up a development environment to access all the services that your generative RAG application needs. This includes:

• Adding a local database
• Adding a local or remote LLM service

Note

You can see more samples of containerized GenAI applications in the GenAI Stack demo applications.

Add a local database

You can use containers to set up local services, like a database. In this section, you'll explore the database service in the docker-compose.yaml file.

To run the database service:

1. In the cloned repository's directory, open the docker-compose.yaml file in an IDE or text editor.

2. In the docker-compose.yaml file, you'll see the following:

   services:
     qdrant:
       image: qdrant/qdrant
       container_name: qdrant
       ports:
         - "6333:6333"
       volumes:
         - qdrant_data:/qdrant/storage

   Note

   To learn more about Qdrant, see the Qdrant Official Docker Image.

3. Start the application. Inside the winy directory, run the following command in a terminal.

   $ docker compose up --build
   

4. Access the application. Open a browser and view the application at http://localhost:8501. You should see a simple Streamlit application.

5. Stop the application. In the terminal, press ctrl+c to stop the application.

Add a local or remote LLM service

The sample application supports both Ollama. This guide provides instructions for the following scenarios:

• Run Ollama in a container
• Run Ollama outside of a container

While all platforms can use any of the previous scenarios, the performance and GPU support may vary. You can use the following guidelines to help you choose the appropriate option:

• Run Ollama in a container if you're on Linux, and using a native installation of the Docker Engine, or Windows 10/11, and using Docker Desktop, you have a CUDA-supported GPU, and your system has at least 8 GB of RAM.
• Run Ollama outside of a container if running Docker Desktop on a Linux Machine.

Choose one of the following options for your LLM service.

Run your RAG application

At this point, you have the following services in your Compose file:

• Server service for your main RAG application
• Database service to store vectors in a Qdrant database
• (optional) Ollama service to run the LLM service

Once the application is running, open a browser and access the application at http://localhost:8501.

Depending on your system and the LLM service that you chose, it may take several minutes to answer.

Summary

In this section, you learned how to set up a development environment to provide access all the services that your GenAI application needs.

Related information:

• Dockerfile reference
• Compose file reference
• Ollama Docker image
• GenAI Stack demo applications

Next steps

See samples of more GenAI applications in the GenAI Stack demo applications.