python

How to Neatly Display Parallel Output in Terminal?

When using modules like multiprocessing or threading for parallel processing, have you ever wondered how to print string messages from each process or thread effectively?

Most people would simply use print() in the terminal. However, when the output is too long or too much, it might not be particularly useful. In such scenarios, it’s often best to write the outputs to a file so you can trace back any issues later.

Still, most of the time, we want to know the latest status of each executing unit, like which step it’s currently at. If these outputs continue to pile up in the terminal, it inevitably makes the terminal quite messy.

In this article, I will introduce a method to refresh and overlay outputs of parallel processing, so you can see the latest status of each unit while keeping the terminal clean and elegant!

Posted on  Mar 13, 2024  in  Python Programming - Intermediate Level  by  Amo Chen  ‐ 3 min read

Introduction to the Python Package - JMESPath (A JSON Query Language Similar to jq)

JMESPath is a Python package with functions similar to jq, enabling Python developers to query and restructure JSON data with a syntax akin to jq. This is done by converting JSON into Python’s native data types using the json module. Proper use of JMESPath can simplify code and enhance readability.

This article will introduce the methods for using JMESPath.

Posted on  Jul 12, 2023  in  Python Module Recommendations  by  Amo Chen  ‐ 5 min read

Great Article Share - Automating Kubernetes Deployments using Python

The article, “Automating Kubernetes Deployments using Python,” introduces how to automate the control of K8s-related resources using the Python Kubernetes package. It covers five major resources: Deployment, Service, ConfigMap, Secret, and Ingress. These are typically encountered when deploying applications to K8s. If you’re looking to automate the process of deploying applications to K8s with Python, this article is a great starting point and very valuable as a reference. Automating Deployment of Applications using Kubernetes Python SDK

Posted on  Jul 8, 2023  by  Amo Chen  ‐ 1 min read

What? You Haven't Used K8s Yet?

With the rise of containerization, more and more companies are adopting Kubernetes (or K8s) to run various containerized services within K8s clusters. Not only does this provide a standard approach for deployments (since everyone writes YAML configuration files to describe how services are deployed), it also offers more flexibility in system resource scheduling and scaling (using K8s commands makes it easy to adjust the number or resources of service containers). So usually, after learning Docker and docker-compose, the next step is to learn how to use K8s.

Posted on  Mar 30, 2023  by  Amo Chen  ‐ 1 min read

Python Module Tutorial - dataclasses

Python’s dataclasses is a new module added in Python 3.7, mainly used to define structured data in the form of classes.

The dataclasses module provides some convenient features to help automatically generate commonly used class methods such as __init__, __repr__, __eq__, etc., saving developers time in writing repetitive code.

Using dataclasses can make Python programs more concise and improve code readability.

Are you ready to show off your skills using dataclasses in your Python code?

Posted on  Feb 1, 2023  in  Python Programming - Intermediate Level  by  Amo Chen  ‐ 8 min read

Designed for Counting - Python Counter Class

The Python collections module provides several convenient classes for developers to use, among which the Counter class (a subclass of dict) can be applied in counting-related scenarios:

A Counter is a dict subclass for counting hashable objects.

This article will introduce the usage of the Counter class and compare its performance with dict and defaultdict.

Posted on  Apr 19, 2022  in  Python Programming - Beginner Level  by  Amo Chen  ‐ 5 min read

Practical Python Performance Profiling - From cProfile to py-spy

It’s a well-known fact that Python applications run slower compared to those written in languages like Go or C. However, Python’s ease of use significantly speeds up development, and its philosophy of batteries included along with its rich ecosystem can greatly reduce the need to reinvent the wheel, shortening the path from concept to product.

The issue of slower execution can be mitigated through various techniques such as using distributed systems, multithreading, multiprocessing, optimizing algorithms, or extending Python with more efficient languages (e.g., using the ctypes module). Although it might not achieve the same high efficiency as Go or C, it can certainly reach an acceptable level of performance.

Before tackling real efficiency issues, it’s often the case that inefficiencies are due to poorly written Python applications rather than limitations of Python itself.

Analyzing Python code to find hidden performance issues is crucial. This article starts by introducing the cProfile module and demonstrates with examples how to identify bottlenecks in Python programs. We then use the py-spy tool to precisely locate issues, improving the efficiency of finding problems.

Posted on  Dec 9, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 6 min read

Practical Use of Fil to Improve Python Memory Usage

In the post Identifying Peak Memory Usage with Python’s Resource Module, we showed how to use the Python resource module to identify peak memory usage. However, this module doesn’t provide detailed memory stats nor does it show which parts of your Python code consume the most memory. Therefore, we need a tool to profile memory usage in detail to help pinpoint problems.

In this post, we’ll introduce how to use Fil to profile Python programs’ memory usage. We’ll go through a simple example to identify and optimize the parts of the program that consume the most memory.

Posted on  Nov 29, 2021  in  Python Programming - Advanced Level  by  Amo Chen  ‐ 4 min read