Using Graylog with Log4j2

Do you need to get your project integrated with a Graylog system? If yes, this article is for you. I will give the exact steps to add a gelf appender in your project and try to share as much detail as I can along with my experiences about exceptional cases you will encounter when you are packaging an uber-jar (aka fat-jar)

Before getting into the real stuff we will explain a little bit about Grayloy. Let’s see the formal definition put by the Graylog team itself.

What is Graylog?

Graylog is a leading centralized log management solution built to open standards for capturing, storing, and enabling real-time analysis of terabytes of machine data.

graylog.org

Graylog uses GELF instead of the plain old syslog style. GELF stands for Graylog Extended Log Format. Logging to graylog is more than just writing simply a line of text into a log file. It is more like firing structured log events in json format as shown in the code snippet below. You can log using UDP or TCP depending on your needs. It also supports compression and chunking if you want (Chunking is only available through UDP) For Graylog options and settings please check Graylog DOCs. This article will be about how to get a Java application talk to Graylog by using log4j2.

{
  "version": "1.1",
  "host": "example.org",
  "short_message": "A short message that helps you identify what is going on",
  "full_message": "Backtrace here\n\nmore stuff",
  "timestamp": 1385053862.3072,
  "level": 1,
  "_user_id": 9001,
  "_some_info": "foo",
  "_some_env_var": "bar"
}

GELF Implementations for Java

Currently there isn’t an official GELF implementation published by Graylog for any of the Java logging frameworks . However there is an opensource library that the Graylog community widely relies on. It is logstash-gelf.

Logstash-Gelf

logstash-gelf requires as of version 1.14.0 Java 7 or higher. Version 1.13.x and older require Java 6

logstash-gelf

Logstash-gelf is quite an all-rounder. It is not just supporting log4j2, it supports all the major java logging frameworks. For the log4j2 part it implements a log4j2 plugin, catches log events and creates GELF messages out of them. That’s why in order to get integrated into Graylog by using logstash-gelf, all you need to do is just having the right dependencies in your classpath and updating log4j2.xml accordingly.

Steps to Get Started

1. Add logstash-gelf in your dependencies. Your logging related dependencies should end up looking like below:

2. Update your log4j2.xml adding the gelf appender accordingly. Host is the only mandatory thing.

That’s all. Now your logs will be appended to Graylog as well.

For the details of the settings, check the list that I quoted from the logstash-gelf docs.

Things you need to know when you are using Graylog

MDC

The acronym stands for Mapped Diagnostic Context. MDC is not a Graylog specific concept. It is a logging instrument employed to enrich log messages with more useful information. If you have some data that you want to be available when the actual log message is written, you put it into MDC. Speaking of log4j2, everything you put will be accessible in the appender.

This is how you set your custom gelf message fields programmaticly. If you want a field named _userId to be available in Graylog then you put it into ThreadContext before firing the log events. All the log events will be, sort of, paired with the information you put inside the ThreadContext untill you call ThreadContext.clearMap()

Streams

The Graylog streams are a mechanism to route messages into categories in realtime while they are processed. You define rules that instruct Graylog which message to route into which streams.

docs.graylog.org

If your Graylog system uses streams than it expects you to tell which stream you want your log messages to be in. So you should be providing the correct stream key-value pair either as a log4j2 appender field (like I did in the log4j2.xml above) or using MDC. For example in my case the stream identifying gelf field was logStream so I’ve set it’s value to MYAWESOMEAPPS.

Fat-Jar Issues

During runtime, plugins are located by the PluginManager of log4j2. PluginManager locates them inside the classpath by looking at a metadata file that is supposed to list all the plugins available. If you are packaging a fat-jar, this metadata file is generated during compilation by the corresponding annotation processor of log4j2-core. It does not append the entries for custom plugins to the metadata and always overwrites with the default value that comes from log4j2-core artifact. In order to get your plugin of interest listed inside this metadata file, you need to set a transformer in your pom.xml. Check the corresponding pom.xml changes:

Since fat-jar packaging always tends to mess things up with the classpath metadata files you need to be careful for similar issues for your other dependencies. For example if you are using Apache CXF you will need one more transformer:

<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
   <resource>META-INF/cxf/bus-extensions.txt</resource>
</transformer>

Example Project

I have created a sample project and tested in my environment. Please feel free to fork and clone on GitHub. https://github.com/HalitTalha/Graylog-FatJar-Example