Three tricks to prevent your data project from memory overflow
A few years ago, I wrote a blog post on how to write memory-efficient loops in Python which became quite popular. The positive response encouraged me to write a second part, where I delve into additional memory optimization methods.
When writing python code, loops are not the only place where we need to be mindful of memory usage. In data-related projects and object-oriented code development, it is important to make sure that our classes are also memory efficient. Often, we invest a significant amount of time designing and writing complex and intricate classes, only to discover that they perform poorly in testing or production due to the large amount of data they need to carry.
By following the techniques and approaches discussed in the article, you can create classes that optimize memory usage and improve overall performance. This blog post explores three techniques and recommended approaches for creating memory-efficient Python classes.
Using Python’s __slots__ dunder, you can explicitly define the attributes that a class can ever possess. This generally helps optimize the memory usage of our classes by avoiding the creation of a dynamic dictionary for attribute storage.
By default, Python classes store their instance attributes in a private dictionary (__dict__). This dictionary allows for a lot of flexibility, as you can add, modify, or delete the class attributes at runtime. However, this flexibility usually comes at the cost of memory overhead. Each instance of the class has a dictionary that stores attribute names and values as key-values pairs. When using __slots__, Python reserves only a fixed amount of space for the specified attributes directly in each instance, instead of using the default dictionary.
Here’s an example of a Python class that uses __slots__ to increase memory efficiency:
