Docker Syslog

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging. But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space.

The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd).

Docker ELK stack with syslog input. Run the latest version of the ELK (Elasticsearch, Logstash, Kibana) stack to process the syslog logs via Docker and Docker-compose.You can redirect your logs to this stack with syslog driver in your Docker instances and get the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and the visualization power of Kibana. Edit the docker configuration file (/etc/sysconfig/docker on RHEL/CentOS based systems). Add the log driver parameter (-log-driver=syslog -log-opt syslog-facility=local6 -log-level=warn) to the OPTIONS line. # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS='-selinux-enabled. The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd). As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation.

As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation during development but a highly-available, scalable logging solution in production.

If you would rather explore an alternate Docker logging architecture, using a dedicated container that senses all other containers and routes their log events, see my article on Docker logspout.

Spring Boot logging to the console

Using our Spring Boot example fully described here, we will have it output DEBUG level logs to the console. It uses SLF4J/Logback so we configure ‘src/main/resources/logback.xml’ to send events to its ConsoleAppender.

Later in this article we put this application into a Docker container, but for now let’s run it directly on our Ubuntu host machine.

[Shell]
$ sudo apt-get install curl git openjdk-7-jdk maven -y
$ git clone https://github.com/fabianlee/gs-rest-service.git
$ cd gs-rest/service/complete
$ ./mvnw package
$ ufw allow 8080/tcp
$ java -jar target/gs-rest-service-0.1.0.jar
[/Shell]

Then you can make a client call into this service by using curl:

[Shell]
$ curl http://localhost:8080/greeting
[/Shell]

Each call to the service will output debug level logs to the console. This console level output will be redirected using the appropriate Docker logging driver in the subsequent sections.

Spring Boot project deployed in Docker container

Now that we have seen our example project deployed directly on our host machine, let’s move on to deploying it into a Docker container. If you have not installed Docker, here are instructions for Ubuntu.

If you want to build the Docker image and run gs-rest-service in a docker container where port 8080 is mapped directly to the docker host:

[Shell]
$ sudo docker build -t gs-rest-service -f src/main/docker/Dockerfile .
$ sudo docker run -p 8080:8080 gs-rest-service
[/Shell]

We have the container’s port 8080 mapped directly to the host, so you can call curl on localhost:8080 exactly like before.

Docker container sending logs to json-file

Break out of the container started earlier, and start the container again explicitly specifying a json-file logging driver.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=json-file gs-rest-service
[/Shell]

Now you can view the latest logs and check the path of the persisted json file by using:

[Shell]
$ sudo docker logs $(docker ps -q)
$ sudo docker inspect –format='{{.LogPath}}’ $(docker ps -q)
[/Shell]

Note: This .json file is located on the Docker host, not the container.

Docker container sending logs to syslog

Before we send syslog output to a host, let’s make sure the host either has rsyslog enabled (see my article here for Ubuntu), or we use netcat on the host to echo what comes into port 514 via UDP (use sudo or root because port 514 < 1024).

[Shell]

# ufw allow 514/udp

# while true; do { nc -vlu 514; } done

[/Shell]

Now break out of the Docker container started earlier, and start the container again explicitly specifying a syslog logging driver to the host where you have enabled the syslog listener on 514/udp.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=syslog –log-opt syslog-address=udp://192.168.1.2:514 –log-opt syslog-facility=daemon –log-opt syslog-format=rfc5424 –log-opt tag=”{{.ImageName}}/{{.ID}}” gs-rest-service
[/Shell]

The ‘Docker logs’ command does not have the ability to show you these network logs, but you should have received logs from netcat or ‘/var/log/syslog’ that looks like:

[RAW]

<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.806 [http-nio-8080-exec-3] DEBUG hello.GreetingController – debug message slf4j
<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.807 [http-nio-8080-exec-3] INFO hello.GreetingController – info message slf4j

[/RAW]

Summary

The overarching idea is that these container log messages are now being directed by the person deploying the Docker container, and not the developer who packages the application.

This loose coupling makes the logging implementation an orthogonal concern that can be centrally operated for production environments (e.g. ELK), yet simple to debug during the development lifecycle.

REFERENCES

https://www.loggly.com/blog/top-5-docker-logging-methods-to-fit-your-container-deployment-strategy/

docker run -it ubuntu:trusty /bin/bash

docker ps -q (last container id)

docker ps -a (all container id, even stopped)

docker inspect -f ‘{{.Name}} – {{.NetworkSettings.IPAddress }}’ $(docker ps -q)

docker exec -it $(docker ps -q) bash (get to shell of latest container)

logger -p local0.warn -d -n myhost “test message to catchall” -u /ignore/socket

docker inspect –format='{{.LogPath}}’ $(docker ps -q)

When building containerized applications, logging is definitely one of the most important things to get right from a DevOps standpoint. Log management helps DevOps teams debug and troubleshoot issues faster, making it easier to identify patterns, spot bugs, and make sure they don’t come back to bite you!

In this article, we’ll refer to Docker logging in terms of container logging, meaning logs that are generated by containers. These logs are specific to Docker and are stored on the Docker host. Later on, we’ll check out Docker daemon logs as well. These are the logs that are generated by Docker itself. You will need those to debug errors in the Docker engine.

Docker Logging: Why Are Logs Important When Using Docker

Docker Syslog Server Gui

The importance of logging applies to a much larger extent to Dockerized applications. When an application in a Docker container emits logs, they are sent to the application’s stdout and stderr output streams.

The container’s logging driver can access these streams and send the logs to a file, a log collector running on the host, or a log management service endpoint.

By default, Docker uses a json-file driver, which writes JSON-formatted logs to a container-specific file on the host where the container is running. More about this in the section below called “What’s a Logging Driver?”

The example below shows JSON logs created using the json-file driver:

If that wasn’t complicated enough, you have to deal with Docker daemon logs and host logs apart from container logs. All of them are vital in troubleshooting errors and issues when using Docker.

We know how challenging handling Docker logs can be. Check out Top 10 Docker Logging Gotchas to see some of the best practices we discovered along the years.

Before moving on, let’s go over the basics.

What Is a Docker Container

A container is a unit of software that packages an application, making it easy to deploy and manage no matter the host. Say goodbye to the infamous “it works on my machine” statement!

How? Containers are isolated and stateless, which enables them to behave the same regardless of the differences in infrastructure. A Docker container is a runtime instance of an image that’s like a template for creating the environment you want.

What Is a Docker Image?

A Docker image is an executable package that includes everything that the application needs to run. This includes code, libraries, configuration files, and environment variables.

Why Do You Need Containers?

Containers allow breaking down applications into microservices – multiple small parts of the app that can interact with each other via functional APIs. Each microservice is responsible for a single feature so development teams can work on different parts of the application at the same time. That makes building an application easier and faster.

How Is Docker Logging Different

Most conventional log analysis methods don’t work on containerized logging – troubleshooting becomes more complex compared to traditional hardware-centric apps that run on a single node and need less troubleshooting. You need more data to work with so you must extend your search to get to the root of the problem.

Here’s why:

Containers are Ephemeral

Docker containers emit logs to the stdout and stderr output streams. Because containers are stateless, the logs are stored on the Docker host in JSON files by default. Why?

The default logging driver is json-file. The logs are then annotated with the log origin, either stdout or stderr, and a timestamp. Each log file contains information about only one container.

You can find these JSON log files in the /var/lib/docker/containers/ directory on a Linux Docker host. Here’s how you can access them:

That’s where logging comes into play. You can collect the logs with a log aggregator and store them in a place where they’ll be available forever. It’s dangerous to keep logs on the Docker host because they can build up over time and eat into your disk space. That’s why you should use a central location for your logs and enable log rotation for your Docker containers.

Containers are Multi-Tiered

This is one of the biggest challenges to Docker logging. However basic your Docker installation is, you will have to work with two levels of aggregation. One refers to the logs from the Dockerized application inside the container. The other involves the logs from the host servers, which consist of the system logs, as well as the Docker Daemon logs which are usually located in /var/log or a subdirectory within this directory.

A simple log aggregator that has access to the host can’t just pull application log files as if they were host log files. Instead, it must be able to access the file system inside the container to collect the logs. Furthermore, your infrastructure will, inevitably, extend to more containers and you’ll need to find a way to correlate log events to processes rather than their respective containers.

Docker Logging Strategies and Best Practices

Needless to say, logging in Docker could be challenging. But there are a few best practices to have in mind when working with containerized apps.

Logging via Application

This technique means that the application inside the containers handles its own logging using a logging framework. For example, a Java app could use a Log4j2 to format and send the logs from the app to a remote centralized location skipping both Docker and the OS.

On the plus side, this approach gives developers the most control over the logging event. However, it creates extra load on the application process. If the logging framework is limited to the container itself, considering the transient nature of containers, any logs stored in the container’s filesystem will be wiped out if the container is terminated or shut down.

To keep your data, you’ll have to either configure persistent storage or forward logs to a remote destination like a log management solution such as Elastic Stack or Sematext Cloud. Furthermore, application-based logging becomes difficult when deploying multiple identical containers, since you would need to find a way to tell which log belongs to which container.

Logging Using Data Volumes

As we’ve mentioned above, one way to work around containers being stateless when logging is to use data volumes.

With this approach you create a directory inside your container that links to a directory on the host machine where long-term or commonly-shared data will be stored regardless of what happens to your container. Now, you can make copies, perform backups, and access logs from other containers.

You can also share volume across multiple containers. But on the downside, using data volumes make it difficult to move the containers to different hosts without any loss of data.

Logging Using the Docker Logging Driver

Another option to logging when working with Docker, is to use logging drivers. Unlike data volumes, the Docker logging driver reads data directly from the container’s stdout and stderr output. The default configuration writes logs to a file on the host machine, but changing the logging driver will allow you to forward events to syslog, gelf, journald, and other endpoints.

Since containers will no longer need to write to and read from log files, you’ll likely notice improvements in terms of performance. However, there are a few disadvantages of using this approach as well: Docker log commands work only with the json-file log driver; the log driver has limited functionality, allowing only log shipping without parsing; and containers shut down when the TCP server becomes unreachable.

Logging Using a Dedicated Logging Container

Another solution is to have a container dedicated solely to logging and collecting logs, which makes it a better fit for the microservices architecture. The main advantage of this approach is that it doesn’t depend on a host machine. Instead, the dedicated logging container allows you to manage log files within the Docker environment. It will automatically aggregate logs from other containers, monitor, analyze, and store or forward them to a central location.

This logging approach makes it easier to move containers between hosts and scale your logging infrastructure by simply adding new logging containers. At the same time, it enables you to collect logs through various streams of log events, Docker API data, and stats.

This is the approach we suggest you should use. You can set up Logagent as a dedicated logging container and have all Docker logs ship to Sematext Logs in under a few minutes as explained a bit further down.

Logging Using the Sidecar Approach

For larger and more complex deployments, using a sidecar is among the most popular approaches to logging microservices architectures.

Similarly to the dedicated container solution, it uses logging containers. The difference is that this time, each application container has its own dedicated container, allowing you to customize each app’s logging solution. The first container saves log files to a volume which are then tagged and shipped by the logging container to a third-party log management solution.

One of the main advantages of using sidecars is that it allows you to set up additional custom tags to each log, making it easier for you to identify their origin.

There are some drawbacks, however – it can be complex and difficult to set up and scale, and it can require more resources than the dedicated logging method. You must ensure that both application container and sidecar container are working as a single unit, otherwise, you might end up losing data.

Get Started with Docker Container Logs

When you’re using Docker, you work with two different types of logs: daemon logs and container logs.

What Are Docker Container Logs?

Docker container logs are generated by the Docker containers. They need to be collected directly from the containers. Any messages that a container sends to stdout or stderr is logged then passed on to a logging driver that forwards them to a remote destination of your choosing.

Docker Syslog Log Driver

Here are a few basic Docker commands to help you get started with Docker logs and metrics:

Show container logs: docker logs containerName
Show only new logs: docker logs -f containerName
Show CPU and memory usage: docker stats
Show CPU and memory usage for specific containers: docker stats containerName1 containerName2
Show running processes in a container: docker top containerName
Show Docker events: docker events
Show storage usage: docker system df

Watching logs in the console is nice for development and debugging, however in production you want to store the logs in a central location for search, analysis, troubleshooting and alerting.

What Is a Logging Driver?

Logging drivers are Docker’s mechanisms for gathering data from running containers and services to make it available for analysis. Whenever a new container is created, Docker automatically provides the json-file log driver if no other log driver option has been specified. At the same time, it allows you to implement and use logging driver plugins if you would like to integrate other logging tools.

Here’s an example of how to run a container with a custom logging driver, in this case syslog:

How to Configure the Docker Logging Driver?

When it comes to configuring the logging driver, you have two options:

setup a default logging driver for all containers
specify a logging driver for each container

In the first case, the default logging driver is a JSON file, but, as mentioned above, you have many other options such as logagent, syslog, fluentd, journald, splunk, etc. You can switch to another logging driver by editing the Docker configuration file and changing the log-driver parameter, or using your preferred log shipper.

Alternatively, you can choose to configure a logging driver on a per-container basis. As Docker provides a default logging driver when you start a new container, you need to specify the new driver from the very beginning by using the -log-driver and -log-opt parameters.

Where Are Docker Logs Stored By Default?

The logging driver enables you to choose how and where to ship your data. The default logging driver as I mentioned above is a JSON file located on the local disk of your Docker host:

Have in mind, though, that when you use another logging driver than json-file or journald you will not find any log files on your disk. Docker will send the logs over the network without storing any local copies. This is risky if you ever have to deal with network issues.

In some cases Docker might even stop your container, when the logging driver fails to ship the logs. This issue might happen depending on what delivery mode you are using.

Learn more about where Docker logs are stored from our post about Docker logs location.

Where Are Delivery Modes?

Docker containers can write logs by using either the blocking or non-blocking delivery mode. The mode you choose will determine how the container prioritizes logging operations relative to its other tasks.

Direct/Blocking

Blocking is Docker’s default mode. It will interrupt the application each time it needs to deliver a message to the driver.

It makes sure all messages are sent to the driver, but can introduce latency in the performance of your application. if the logging driver is busy, the container delays the application’s other tasks until it has delivered the message.

Depending on the logging driver you use, the latency differs. The default json-file driver writes logs very quickly since it writes to the local filesystem, so it’s unlikely to block and cause latency. However, log drivers that need to open a connection to a remote server can block for longer periods and cause noticeable latency.

That’s why we suggest you use the json-file driver and blocking mode with a dedicated logging container to get the most of your log management setup. Luckily it’s the default log driver setup, so you don’t need to configure anything in the /etc/docker/daemon.json file.

Non-blocking

In non-blocking mode, a container first writes its logs to an in-memory ring buffer, where they’re stored until the logging driver is available to process them. Even if the driver is busy, the container can immediately hand off application output to the ring buffer and resume executing the application. This ensures that a high volume of logging activity won’t affect the performance of the application running in the container.

When running in non-blocking mode, the container writes logs to an in-memory ring buffer. The logs are stored in the ring-buffer until it’s full. Only then is the log shipped. Even if the driver is unavailable, the container sends logs to the ring buffer and continues executing the application. This ensures high volume of logging without impacting performance. But there are downsides.

Non-blocking mode does not guarantee that the logging driver will log all the events. If the buffer runs out of space, buffered logs will be deleted before they are sent. You can use the max-buffer-size option to set the amount of RAM used by the ring buffer. The default value for max-buffer-size is 1 MB, but if you have more RAM available, increasing the buffer size can increase the reliability of your container’s logging.

Although blocking mode is Docker’s default for new containers, you can set this to non-blocking mode by adding a log-opts item to Docker’s daemon.json file.

Alternatively, you can set non-blocking mode on an individual container by using the --log-opt option in the command that creates the container:

Logging Driver Options

The log file format for the json-file logging driver is machine readable JSON format with a timestamp, stream name and the log message. Therefore users prefer the docker logs command to see the logs on their console.

On the other hand the machine readable log format is a good base for log shippers to ship the logs to log management platforms, where you can search, visualise and alert on log data.

However, you have other log driver options as follows:

logagent: A general purpose log shipper. The Logagent Docker image is pre-configured for log collection on container platforms. Logagent collects not only logs, it also adds meta-data such as image name, container id, container name, Swarm service or Kubernetes meta-data to all logs. Plus it handles multiline logs and can parse container logs.
syslog: Ships log data to a syslog server. This is a popular option for logging applications.
journald: Sends container logs to the systemd journal.
fluentd: Sends log messages to the Fluentd collector as structured data.
elf: Writes container logs to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash.
awslogs: Sends log messages to AWS CloudWatch Logs.
splunk: Writes log messages to Splunk using HTTP Event Collector (HEC).
cplogs: Ships log data to Google Cloud Platform (GCP) Logging.
logentries: Writes container logs to Rapid7 Logentries.
etwlogs: Writes log messages as Event Tracing for Windows (ETW) events, thus only available on Windows platforms.

Use the json-file Log Driver With a Log Shipper Container

The most reliable and convenient way of log collection is to use the json-file driver and set up a log shipper to ship the logs. You always have a local copy of logs on your server and you get the advantage of centralized log management.

If you were to use Sematext Logagent there are a few simple steps to follow in order to start sending logs to Sematext. After creating a Logs App, run these commands in a terminal.

This will start sending all container logs to Sematext.

How to Work With Docker Container Logs Using the docker logs Command?

Docker has a dedicated command which lists container logs. The docker logs command. The flow will usually involve you checking your running containers with docker ps, then check the logs by using a container’s ID.

This command will list all logs for the specified container. You can add a timestamp flag and list logs for particular dates.

What you’ll end up doing will be tailing these logs, either to check the last N number of lines or tailing the logs in real time.

The --tail flag will show the last N lines of logs:

Using the --follow flag will tail -f (follow) the Docker container logs:

But what if you only want to see specific logs? Luckily, grep works with docker logs as well.

This command will only show errors:

Once an application starts growing, you tend to start using Docker Compose. Don’t worry, it has a logs command as well.

This will display the logs from all services in the application defined in the Docker Compose configuration file.

Get started with Docker with our Docker Commands Cheat Sheet!

How to Work with Docker Container Logs Using a Log Shipper?

While everyone’s infrastructure is growing – nowadays, mostly in the container space – so are the monitoring needs. However, monitoring containers is different – and more challenging – from traditional server monitoring.

Unlike non-containerized applications that write logs into files, containers write their logs to the standard output and standard error stream. Container logs can be a mix of plain text messages from start scripts and structured logs from applications, which makes it difficult for you to tell which log event belongs to what container and app, then parse it correctly and so on.

Although Docker log drivers can ship logs to log management tools, most of them don’t allow you to parse container logs. You need a separate tool called a log shipper, such as Logagent, Logstash or rsyslog to structure logs before shipping them to storage. The problem is that when your logging solution uses multiple tools with dependencies for log processing, the chances your logging pipeline will crash increases with every new tool.

But there are a few Docker logging driver alternatives that can help make your job easier, one of them being Sematext Logagent.

Docker Syslog-address

Logagent is an all-in-one general-purpose solution for container log processing that allows you to monitor container logs, as well as your whole infrastructure and applications if paired with the Sematext Agent container.

You can read more about how Logagent works and how to use it for monitoring logs in our post on Docker Container Monitoring with Sematext.

What About Docker Daemon Logs

Docker daemon logs are generated by the Docker platform and located on the host. Depending on the host operating system, daemon logs are written to the system’s logging service or to a log file.

If you were to collect only container logs you’d get insight into the state of your services. However, by traditional logging methods, you also need to be aware of the state of your Docker platform, which is what Docker daemon logs are for. They paint a clear picture of your overall microservices architecture.

On that note, the Docker daemon logs two types of events:

Events generated by the Docker service itself
Commands sent to the daemon through Docker’s Remote API

Depending on your Operating System, the Docker daemon log file is stored in different locations.

Check out Guide to Docker Logs Location to find out more.

Docker Log Management: Logging Tools & Software

Logging is a key part of gathering insight into the state of your infrastructure, but only if it’s analyzed. However, log data comes in huge volumes so doing it manually would be like looking for a needle in a haystack. Which is why you need a log data analysis platform. You can opt for open-source solutions or commercial software to get the most out of your Docker logs.

Open-Source Log Analysis Solutions

With open-source solutions, you need an expert team ready to handle everything from setup to configuration, providing infrastructure, maintenance, and management.

The most popular open source log analysis software is Elastic Stack (formerly known as ELK Stack). It’s a robust platform comprising three different tools – Elasticsearch to store log data, Logstash to process it, and Kibana to visualize log data.

For more information on Elasticsearch, check out our Elasticsearch Complete Guide.

If you still want to use Elasticsearch and Kibana but don’t want to manage it yourself, Sematext Cloud has an Elasticsearch API and integrated Kibana in the UI, if you feel like using it instead of the default Sematext Dashboards. This makes migrating to a managed Elasticsearch cluster a walk in the park. In your log shipper configuration, you’d only change the Elasticsearch endpoints from your local Elasticsearch cluster to the Sematext Cloud Elasticsearch API endpoint.

Commercial Log Analysis Tools: Logging as a Service

If you don’t have the resources to deal with Docker log data on your own, you can reach out to vendors who provide “logging as a service” as part of a full log management solution. You only need to point out the Docker logs and they’ll take over managing your log data from collection to storage, analysis, monitoring, and presentation.

Sematext as a Log Management Solution for Docker Logs

Installing Syslog On The Raspberry Pi Using Syslog-NG

The Docker logging driver and log delivery mode you choose can have a noticeable effect on the performance of your containerized applications. We recommend using the json-file driver for reliable logging, consistent performance, and observability by using a centralized logging tool like Sematext Logs, If you want to see how

We want to give you an all-in-one solution that provides hassle-free log management and analytics for your infrastructure and applications. It allows you to filter, analyze, and alert on logs from all your applications.

Docker Syslog Container

By storing your Docker logs, you can detect and troubleshoot issues easier, but also gather actionable insights from both your infrastructure and Dockerized applications.

If you want to earn more about Sematext Logs, see how it stacks against similar solutions in our dedicated articles about the best log management tools, log aggregation tools, log analysis software, and cloud logging services.

For optimum performance, we recommend you collect logs along with metrics and traces. We talked more about this is our Docker monitoring series. Chek it out if you’re into that:

Now that you know how logging works in Docker, you can take a look at Kubernetes logs as well. Learn more from our Kubernetes logging guide.