Introduction to a Handy Python Package - structlog

When starting with Python’s logging module, many of us have encountered confusion, as with the following code example:

import logging

log = logging.getLogger(__name__)
log.info('Hello')

We might expect this code to print the string Hello, but it doesn’t.

This is because the default log level for Python’s logging module is WARNING, so only messages at WARNING, ERROR, or CRITICAL levels are displayed.

To properly use the logging module, setting it up is necessary, which often means consulting the Python documentation. This isn’t a problem with Python per se, but rather a design philosophy difference.

So, is there a package more intuitive and easier to use than the built-in logging module?

The answer is “yes,” and that is the structlog package, which this article introduces.

Environment #

• Python 3
• structlog
• rich
• better_exceptions
• flask

$ pip install structlog rich better_exceptions flask

structlog #

structlog is a logging package designed to be simple, user-friendly, and fast. Unlike the default logging that only outputs certain levels, structlog outputs all logs to the standard output regardless of the log level, with formatting automatically handled and logs color-coded based on severity:

>>> import structlog
>>> log = structlog.get_logger()
>>> log.info('Hello World')
2023-08-30 13:20:28 [info     ] Hello World
>>> log.info('Error')
2023-08-30 13:20:32 [error    ] Error

Furthermore, structlog is designed to seamlessly replace the existing logging module, supporting %-format string just like the logging module. Here’s an example:

>>> log.info('Hello %s', 'Amo')
2023-08-30 13:21:31 [info     ] Hello Amo

However, with structlog, you can directly use f-string, like so:

>>> name = 'Amo'
>>> log.info(f'Hello {name}')

structlog offers much more, including:

1. Bound Loggers that allow developers to attach key-value pairs to logs without manual formatting.
2. Context Variables to store context-related values, like user IDs or IPs, allowing logs to include this info without repeated retrieval.
3. Richer output formats, like JSON, for better integration with tools like ELK, Logstash, or Graylog.
4. Support for asyncio.

Bound Logger #

Bound Loggers let developers attach arbitrary key-value pairs to logs without setting the format manually. For instance, you can specify any key-value pair, and structlog will automatically append it in a key=value format, sorted alphabetically for convenience:

>>> log.info('Hello %s', 'Amo', current_value='Amo', foo='bar')
>>> 2023-08-30 13:23:56 [info     ] Hello Amo                      current_value=Amo foo=bar

Context Variables #

Context Variables allow developers to store context-related values. For example, in Flask, you can use context variables to log conveniently, ensuring that all subsequent logs automatically include certain details:

import uuid

import structlog
from structlog.contextvars import (
    bind_contextvars,
    clear_contextvars,
    merge_contextvars,
)

import flask


logger = structlog.get_logger()
app = flask.Flask(__name__)


@app.before_request
def setup_log():
    clear_contextvars()
    bind_contextvars(
       request_id=str(uuid.uuid4()),
       ip=flask.request.access_route[0],
    )


def do_job():
    log = logger.bind()
    log.warning('Doing the job', func=do_job.__name__)



@app.route('/', methods=['GET'])
def index():
    log = logger.bind()
    log.info('GET /')
    do_job()
    return {'ok': True}


if __name__ == "__main__":
   app.run()

Upon running this Flask example, accessing http://127.0.0.1:5000 will show a detailed log:

In the app.before_request callback, we register a function to set up a context with request_id and ip. Functions index() and do_job() can then call logger.bind() to obtain and log context values seamlessly.

Context Variables are also excellent for tracking nested code changes.

For more about Context Variables, check structlog.contextvars documentation.

Logging in JSON Format #

To log in JSON format with structlog, define the structlog processors. The processors serve as pipelines for log messages, processed sequentially as specified in a list, enabling customization:

import structlog

structlog.configure(
    processors=[
        structlog.processors.dict_tracebacks,
        structlog.processors.JSONRenderer(),
    ],
)

log = structlog.get_logger()

log.info('Hi', value_1=1, valu2='2')

The above example produces the following output:

{"value_1": 1, "valu2": "2", "event": "Hi"}

Log messages are stored under the event key. Avoid passing another event parameter to prevent TypeError. Other values are stored in a Python dictionary with key-value pairs.

Note: structlog.processors.dict_tracebacks transforms exceptions into dictionary format for successful JSON output via JSONRenderer.

asyncio #

The following example from the official documentation shows that structlog supports asyncio. Use methods like ainfo, adebug, awarning, aerror prefixed with ‘a’ for async logging:

import asyncio
>>> logger = structlog.get_logger()
>>> async def f():
...     await logger.ainfo("async hi!")

Integrating better-exceptions, rich Packages #

With better-exceptions and rich installed, structlog outputs exceptions in a more human-readable format, aligning with the philosophy that logs should be clear and easy to understand. Here’s an example with an intentional division by zero:

import structlog

log = structlog.get_logger()

value = 1

try:
    value / 0
except Exception as e:
    log.exception(e)

The output is enhanced, providing a neat error message with locals for variable states, simplifying understanding and debugging:

Conclusion #

structlog is a highly practical Python logging package. It’s not only easy to use but also offers many developer-friendly features, with excellent backward compatibility for seamless transition from Python’s logging module.

It also provides integrations with OpenTelemetry, Django, Flask, Pyramid, Celery, Twisted, making it a mature package worth exploring!

Enjoy!

References #

structlog 23.1.0 documentation