profiling

Practical Python Performance Profiling - From cProfile to py-spy

It’s a well-known fact that Python applications run slower compared to those written in languages like Go or C. However, Python’s ease of use significantly speeds up development, and its philosophy of batteries included along with its rich ecosystem can greatly reduce the need to reinvent the wheel, shortening the path from concept to product.

The issue of slower execution can be mitigated through various techniques such as using distributed systems, multithreading, multiprocessing, optimizing algorithms, or extending Python with more efficient languages (e.g., using the ctypes module). Although it might not achieve the same high efficiency as Go or C, it can certainly reach an acceptable level of performance.

Before tackling real efficiency issues, it’s often the case that inefficiencies are due to poorly written Python applications rather than limitations of Python itself.

Analyzing Python code to find hidden performance issues is crucial. This article starts by introducing the cProfile module and demonstrates with examples how to identify bottlenecks in Python programs. We then use the py-spy tool to precisely locate issues, improving the efficiency of finding problems.

Posted on  Dec 9, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 6 min read

Practical Use of Fil to Improve Python Memory Usage

In the post Identifying Peak Memory Usage with Python’s Resource Module, we showed how to use the Python resource module to identify peak memory usage. However, this module doesn’t provide detailed memory stats nor does it show which parts of your Python code consume the most memory. Therefore, we need a tool to profile memory usage in detail to help pinpoint problems.

In this post, we’ll introduce how to use Fil to profile Python programs’ memory usage. We’ll go through a simple example to identify and optimize the parts of the program that consume the most memory.

Posted on  Nov 29, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 4 min read