Easily manage settings with Pydantic

During development, configuration files are often needed to easily change the behavior of the system. For example, the switch of Debug mode, database related settings, etc. are usually placed in configuration files.

In addition to the built-in module configparser provided by Python to easily implement configuration files, it can also easily implement configuration files using classes, such as the following two files are examples of configuration

# settings.py
class Settings(object):
    DB_HOST = 'localhost'
    DB_PORT = 8888


settings = Settings()

# test.py
from settings import settings


print(settings.DB_HOST, settings.DB_PORT)

However, nowadays development projects also often use dotenv (e.g. python-dotenv) to make configuration easier.

In addition to developing the functionality of dotenv by yourself, you can actually choose to integrate class-style configuration files and dotenv easily with pydantic.

Requirements #

• Python 3.7
• pydantic 1.5.1

$ pip install pydantic==1.5.1

pydantic #

pydantic is a package that provides data type validation and configuration management through type annotations.

For example, the following example defines the Car class with pydantic’s BaseModel, and also annotates the types of each attribute in the class:

from datetime import datetime
from pydantic import BaseModel


class Car(BaseModel):
    brand_id: int
    brand_name: str
    wheels: int = 4
    created_at: datetime = datetime.now()

When we try to create an instance of the class, pydantic will perform type validation.

Car(brand_id='bad_id', brand_name='Bad name')

The above execution results will show errors similar to the following, indicating that we should set brand_id to an integer type:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    Car(brand_id='bad_id', brand_name='Bad name')
  File "pydantic/main.py", line 338, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Car
brand_id
  value is not a valid integer (type=type_error.integer)

After getting an idea of what pydantic looks like, you can then try out the configuration management capabilities provided by pydantic.

Managing Configuration with Pydantic #

The configuration management of pydantic can be implemented by inheriting the BaseSettings class, which is the biggest difference from the BaseModel class in that it provides integration with environment variables and dotenv.

When we attempt to create an instance of the BaseSettings class, BaseSettings will attempt to load environment variables.

from pydantic import BaseSettings


class Settings(BaseSettings):
    HOME: str


print(Settings())

The result of the above example is as follows, we can see that we haven’t set a value to HOME, and pydantic will automatically load the HOME in the environment variable to the HOME in Settings:

HOME='/Users/<username>'

Once we know the purpose of BaseSettings, we can design different settings for different environments, such as in the following example, we put the common settings in the Settings class and override the settings that are different due to the environment by inheriting the Settings class.

import os

from pydantic import BaseSettings


class Settings(BaseSettings):
    API_BASE: str
    DB_HOST: str = 'localhost'
    DB_PORT: int = 8888
    DB_NAME: str
    TESTING: bool = False


class Production(Settings):
    API_BASE = 'https://example.com/api'
    DB_NAME = 'production'


class Testing(Settings):
    API_BASE = 'https://testing.example.com/api'
    DB_NAME = 'testing'
    TESTING = True


def get_settings():
    env = os.getenv('ENV', 'TESTING')
    if env == 'PRODUCTION':
        return Production()
    return Testing()


settings = get_settings()

print('DB Host =', settings.DB_HOST)
print('DB Port =', settings.DB_PORT)

print('Override =', Production(TESTING=True).dict())

The above example execution result as follow:

DB Host = localhost
DB Port = 8888
Override = {'API_BASE': 'https://example.com/api', 'DB_HOST': 'localhost', 'DB_PORT': 8888, 'DB_NAME': 'production', 'TESTING': True}

Let’s take a look at the data validation function provided by pydantic, when we deliberately set DB_PORT as a string:

Production(DB_PORT='abc')

The following error will appear, which indicates that DB_PORT must be an integer. This type of validation can also prevent problems with future configuration files due to mismatched types:

Traceback (most recent call last):
  File "pydantic_test.py", line 40, in <module>
    Production(DB_PORT='abc')
  File "pydantic/env_settings.py", line 28, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 338, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Production
DB_PORT
  value is not a valid integer (type=type_error.integer)

Finally, take a look at the dotenv integration feature provided by pydantic.

In the BaseSettings class, dotenv can be easily integrated by defining the Config class and specifying the env_file property, for example:

import os

from pydantic import BaseSettings


class Settings(BaseSettings):
    API_BASE: str
    DB_HOST: str
    DB_PORT: int
    DB_NAME: str
    TESTING: bool = False

    class Config:
        env_file = '.env'


class Production(Settings):
    API_BASE = 'https://example.com/api'
    DB_NAME = 'production'


class Testing(Settings):
    API_BASE = 'https://testing.example.com/api'
    DB_NAME = 'testing'

    class Config:
        env_file = '.testing.env'


def get_settings():
    env = os.getenv('ENV', 'TESTING')
    if env == 'PRODUCTION':
        return Production()
    return Testing()


settings = get_settings()

print('DB Host =', settings.DB_HOST)
print('DB Port =', settings.DB_PORT)

print('Override =', Production(TESTING=True).dict())

The 13th, 14th, 26th and 27th lines in the above example are the parts for loading different dotenv configurations for different environments.

The contents and location of the two files, .env and .testing.env, in the file system are as follows:

.
├── .env
├── .testing.env
└── settings.py

The contents of .env:

DB_HOST=localhost
DB_PORT=8888
BLAH_BLAH=Banana

The content of testing.env:

DB_HOST=127.0.0.1
DB_PORT=6666

The result after integrating dotenv is:

DB Host = localhost
DB Port = 8888
Override = {'API_BASE': 'https://example.com/api', 'DB_HOST': 'localhost', 'DB_PORT': 8888, 'DB_NAME': 'production', 'TESTING': True}

It should be noted that BLAH_BLAH=Banana in .env is not one of the attributes defined in the abovementioned two categories of Settings and Production, so it will not be automatically loaded into Production. If you want to load BLAH_BLAH, you must define the BLAH_BLAH property in the Settings or Production category.

Above is a simple introduction to configuration management using pydantic.

In fact, pydantic has other different uses, such as generating JSON schema, integrating Python 3.7 dataclasses, etc. If you are interested, you can read the official documents in detail, which is sure to bring a lot of benefits.

References #

pydantic