Python Programming - Advanced Level

Practical Python Performance Profiling - From cProfile to py-spy

It’s a well-known fact that Python applications run slower compared to those written in languages like Go or C. However, Python’s ease of use significantly speeds up development, and its philosophy of batteries included along with its rich ecosystem can greatly reduce the need to reinvent the wheel, shortening the path from concept to product.

The issue of slower execution can be mitigated through various techniques such as using distributed systems, multithreading, multiprocessing, optimizing algorithms, or extending Python with more efficient languages (e.g., using the ctypes module). Although it might not achieve the same high efficiency as Go or C, it can certainly reach an acceptable level of performance.

Before tackling real efficiency issues, it’s often the case that inefficiencies are due to poorly written Python applications rather than limitations of Python itself.

Analyzing Python code to find hidden performance issues is crucial. This article starts by introducing the cProfile module and demonstrates with examples how to identify bottlenecks in Python programs. We then use the py-spy tool to precisely locate issues, improving the efficiency of finding problems.

Posted on  Dec 9, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 6 min read

Practical Use of Fil to Improve Python Memory Usage

In the post Identifying Peak Memory Usage with Python’s Resource Module, we showed how to use the Python resource module to identify peak memory usage. However, this module doesn’t provide detailed memory stats nor does it show which parts of your Python code consume the most memory. Therefore, we need a tool to profile memory usage in detail to help pinpoint problems.

In this post, we’ll introduce how to use Fil to profile Python programs’ memory usage. We’ll go through a simple example to identify and optimize the parts of the program that consume the most memory.

Posted on  Nov 29, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 4 min read

Easily manage settings with Pydantic

During development, configuration files are often needed to easily change the behavior of the system. For example, the switch of Debug mode, database related settings, etc. are usually placed in configuration files.

In addition to the built-in module configparser provided by Python to easily implement configuration files, it can also easily implement configuration files using classes, such as the following two files are examples of configuration

# settings.py
class Settings(object):
    DB_HOST = 'localhost'
    DB_PORT = 8888


settings = Settings()

# test.py
from settings import settings


print(settings.DB_HOST, settings.DB_PORT)

However, nowadays development projects also often use dotenv (e.g. python-dotenv) to make configuration easier.

In addition to developing the functionality of dotenv by yourself, you can actually choose to integrate class-style configuration files and dotenv easily with pydantic.

Posted on  Jul 1, 2020  in  Useful Python Modules , Python Programming - Advanced Level  by  Amo Chen  ‐ 5 min read

Learning Google Protocol Buffers with Python - Part 3

This article is part of a tutorial series:

• Learning Google Protocol Buffers with Python - Part 1
• Learning Google Protocol Buffers with Python - Part 2
• Learning Google Protocol Buffers with Python - Part 3

In the last article, we introduced some key syntax and features of proto3 with Google Protocol Buffers. This time, we’ll delve into how to write more complex messages, and introduce some convenient data types along with how to use them.

Posted on  Nov 9, 2018  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 5 min read

Learning Google Protocol Buffers with Python - Part 2

This post is part of a tutorial series:

• Learning Google Protocol Buffers with Python - Part 1
• Learning Google Protocol Buffers with Python - Part 2
• Learning Google Protocol Buffers with Python - Part 3

In the previous post, we introduced the basics of using Google Protocol Buffers (proto3) with Python. In this post, we’ll delve further into some key syntax and features of proto3.

Posted on  Nov 3, 2018  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 4 min read

Learning Google Protocol Buffers with Python - Part 1

This article is part of a series:

• Learning Google Protocol Buffers with Python - Part 1
• Learning Google Protocol Buffers with Python - Part 2
• Learning Google Protocol Buffers with Python - Part 3

The blog post What I Learned from Quip on How to Build a Product on 8 Different Platforms with Only 13 Engineers explains how Quip managed to build products for 8 different platforms with only a 13-member team. It’s definitely something worth learning from.

A key concept from the post is Build once, use multiple times. It encourages minimizing the repetition of creating the same components, thereby increasing the reusability of components. The article also reveals that Quip heavily uses Google Protocol Buffers. By defining data structures using Google Protocol Buffers, automatic code generation can occur for reading and writing the same data structure across various languages or platforms. It can even act as a data exchange format to transfer between different platforms, reducing repetitive development costs and thus improving development efficiency.

With such a handy tool, let’s learn Google Protocol Buffers using Python!

Posted on  Oct 27, 2018  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 5 min read