Python: __init__ is NOT a constructor: a deep dive in Python object creation
Tinkering with Python’s constructor to create fast, memory-efficient classes
Did you know that the __init__ method is not a constructor? But if __init__ doesn’t create the object, then what does? How do objects get created in Python? Does Python even have a constructor?
The goal of this article is to better understand how Python creates objects and manipulate this process to make better appplications.
First we’ll take a deep dive in how Python creates objects. Next we’ll apply this knowledge and discuss some interesting use cases with some practical examples. Let’s code!
1. Theory: Creating objects in Python
In this part we’ll figure out what Python does under the hood when you create an object. In the next part we’ll take this new knowledge and apply in in part 2.
How to create an object in Python?
This should be pretty simple; you just create an instance of a class. Alternatively you could create a new built-in type like a str or an int. In the code below an instance is created of a basic class. It just contains an __init__ function and a say_hello method:
class SimpleObject:
greet_name:str
def __init__(self, name:str):
self.greet_name = name
def say_hello(self) -> None:
print(f"Hello {self.greet_name}!")
my_instance = SimpleObject(name="bob")
my_instance.say_hello()Notice the __init__ method. It receives a name parameter and stores its value on the greet_name attribute of the SimpleObject instance. This allows our instance to keep state.
Now the question rises: in order to save the state, we need to have something to save the state on. Where does __init__ get the object from?
So, is __init__ a constructor?
The answer: technically no. Constructors actually create the new object; the __init__ method is only responsible for setting the state of the object. It just receives values through it’s parameters and assigns them to the class attributes like greet_name.
In Python, the actual creation of an object right before initialization. For object creation, Python uses a method called __new__ that’s present on each object.
Mike Huls
What does __new__ do?
__new__ is a class method, meaning it is called on the class itself, not on an instance of the class. It is present on each object and is responsible for actually creating and returning the object. The most important aspect of __new__ is that it must return an instance of the class. We’ll tinker with this method later in this article.
Where does the __new__ method come from?
The short answer: everything in Python is an object, and the object class has a __new__ method. You can think of this as “each class inherits from the object class”.
Notice that even though our SimpleObject class inherit from anything, we can still proof that it’s an instance of object:
# SimpleObject is of type 'object'
my_instance = SimpleObject(name="bob")
print(isinstance(my_instance, object)) # <-- True
# but all other types as well:
print(isinstance(42, object)) # <-- True
print(isinstance('hello world', object)) # <-- True
print(isinstance({"my": "dict"}, object)) # <-- TrueIn summary, everything is an object, object defines a __new__ method thus everything in Python has a __new__ method.
How does __new__ differ from __init__?
The __new__ method is used for actually creating the object: allocating memory and returning the new object. Once the object is created we can initialize it with __init__; setting up the initial state.
Mike Huls
What does Python’s process of object creation look like?
Internally, the functions below get executed when you create a new object:
__new__: allocates memory and returns the new object__init__: initialize newly created object; set state
In the code below we demonstrate this by overriding __new__ with our own function. In the next part we’ll use this principle to do some interesting things:
class SimpleObject:
greet_name:str
def __new__(cls, *args, **kwargs): # <-- newly added function
print("__new__ method")
return super().__new__(cls)
def __init__(self, name:str):
print("__init__ method")
self.greet_name = name
def say_hello(self) -> None:
print(f"Hello {self.greet_name}!")
my_instance = SimpleObject(name="bob")
my_instance.say_hello()(we’ll explain why and how this code works in the next parts).
This will print the following:
__new__ method
__init__ method
Hello bob!This means that we have access to the function that initialized an instance of our class! We also see that __new__ executes first. In the next part we’ll understand the behaviour of __new__: what does super().__new__(cls) mean?
How does __new__ work?
The default behaviour of __new__ looks like the code below. In this part we’ll try to understand what’s going on so that we tinker with it in the practical examples in the next part.
class SimpleObject:
def __new__(cls, *args, **kwargs):
return super().__new__(cls)Notice that __new__ is being called on the super() method, which returns a “reference” (it’s actually a proxy-object) to the parent-class of SimpleObject. Remember that SimpleObject inherits from object, where the __new__method is defined.
Breaking it down:
- we get a “reference” to the base class of the class we’re in. In the case of
SimpleObjectwe get a “reference” toobject - We call
__new__on the “reference” soobject.__new__ - We pass in
clsas an argument.
This is how class methods like__new__work; it’s a reference to the class itself
Putting it all together: we ask SimpleObject's parent-class to create a new instance of SimpleObject.
This is the same as my = object.__new__(SimpleObject)
Can I then also create a new instance using __new__?
Yes, remember that the default __new__ implementation actually calls it directly: return super().__new__(cls). So the approaches in the code below do the same:
# 1. __new__ and __init__ are called internally
my_instance = SimpleObject(name='bob')
# 2. __new__ and __init__ are called directly:
my_instance = SimpleObject.__new__(SimpleObject)
my_instance.__init__(name='bob')
my_instance.say_hello()What happens in the direct method:
- we call the
__new__function onSimpleObject, passing it theSimpleObjecttype. SimpleObject.__new__calls__new__on it’s parent class (object)object.__new__creates and returns a instance ofSimpleObjectSimpleObject.__new__returns the new instance- we call
__init__to initialize it.
These things also happen in the non-direct method but they are handled under the hood so we don’t notice.
Mike Huls
Practical application 1: subclassing immutable types
Now that we know how __new__ works we can use if to do some interesting things. We’ll put the theory to practice and subclass an immutable type. This way we can have our own, special type with it’s own methods defined on a very fast, built-in type.
The goal
We have an application that processes many coordinates. For this reason we want our coordinates stored in tuples since they’re small and memory-efficient.
We will create our own Point` class that inherits from tuple`. This way Point` is a tuple` so it’s very fast and small and we can add functionalities like:
- control over object creation (only create a new object if all coordinates are positive e.g.)
- additional methods like calculating the distance between two coordinates.
Mike Huls
The Point class with a __new__ override
In our first try we just create a Point class that inherits from tuple and tries to initialize the tuple with a x,y coordinate. This won’t work:
class Point(tuple):
x: float
y: float
def __init__(self, x:float, y:float):
self.x = x
self.y = y
p = Point(1,2) # <-- tuple expects 1 argument, got 2The reason that this fails is because our class is a subclass of the tuple, which are immutable. Remember that the tuple is created by __new__, after which __init__ runs. At the time of initialization, the tuple is already created and cannot be altered anymore since they are immutable.
We can fix this by overriding __new__:
class Point(tuple):
x: float
y: float
def __new__(cls, x:float, y:float): # <-- newly added method
return super().__new__(cls, (x, y))
def __init__(self, x:float, y:float):
self.x = x
self.y = yThis works because in __new__ we use super() to get a reference to the parent of Point, which is tuple. Next we use tuple.__new__ and pass it an iterable ((x, y)) create a new tuple. This is the same as tuple((1, 2)).
Controlling instance creation and additional methods
The result is a Point class that is a tuple under the hood but we can add all kinds of extras:
class Point(tuple):
x: int
y: int
def __new__(cls, x:float, y:float):
if x < 0 or y < 0: # <-- filter inputs
raise ValueError("x and y must be positive")
return super().__new__(cls, (x, y))
def __init__(self, x:float, y:float):
self.x = x
self.y = y
def distance_from(self, other_point: Point): # <-- new method
return math.sqrt(
(other_point.x - self.x) ** 2 + (other_point.y - self.y) ** 2
)
p = Point(1, 2)
p2 = Point(3, 1)
print(p.distance_from(other_point=p2)) # <-- 2.23606797749979Notice that we’ve added a method for calculating distances between Points, as well as some input validation. We now check in __new__ if the provided Xand y values are positive and prevent object creation altogether when this is not the case.
Mike Huls
Practical application 2: adding metadata
In this example we’re creating a subclass from an immutable float and add some metadata. The class below will produce a true float but we’ve added some extra information about the symbol to use.
class Currency(float):
def __new__(cls, value: float, symbol: str):
obj = super(Currency, cls).__new__(cls, value)
obj.symbol = symbol
return obj
def __str__(self) -> str:
return f"{self.symbol} {self:.2f}" # <-- returns symbol & float formatted to 2 decimals
price = Currency(12.768544, symbol='€')
print(price) # <-- prints: "€ 12.74"As you see we inherit from float , which makes an instance of Currency an actual float. As you see, we also have access to metadata like a symbol for pretty printing.
Also notice that this is an actual float; we can perform float operations without a problem:
print(isinstance(price, float)) # True
print(f"{price.symbol} {price * 2}") # prints: "€ 25.48"
Practical application 3: Singleton pattern
There are cases when you don’t want to return a new object every time you instantiate a class. A database connection for example. A singleton restricts the instantiation of a class to one single instance. This pattern is used to ensure that a class has only one instance and provides a global point of access to that instance:
class Singleton:
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = super(Singleton, cls).__new__(cls)
return cls._instance
singleton1 = Singleton()
singleton2 = Singleton()
print(id(singleton1))
print(id(singleton2))
print(singleton1 is singleton2) # TrueThis code creates an instance of the Singleton class if it doesn’t exist yet and saves it as an attribute on the cls. When Singleton is called once more it returns the instance that is has stored before.
Mike Huls
Other Practical applications
Some other applications include:
- Controlling instance creation
We’ve seen this in thePointexample: add additional logic before creating an instance. This can include input validation, modification, or logging. - Factory Methods
Determine in__new__which class will be returned, based on inputs. - Caching
For resource-intensive object creation. Like with the Singleton pattern, we can store previously created objects on the class itself. We can check in__new__if an equivalent object already exists and return it instead of creating a new one.
Mike Huls
Conclusion
In this article we took a deep dive into Python object creation, learnt a lot about how and why it works. Then we looked at some practical examples that demonstrate that we can do a lot of interesting things with our newly acquired knowledge. Controlling object creation can enable you to create efficient classes and professionalize your code significantly.
To improve your code even further, I think the most important part is to truly understand your code, how Python works and apply the right data structures. For this, check out my other articles here or this this presentation.
I hope this article was as clear as I hope it to be but if this is not the case please let me know what I can do to clarify further. In the meantime, check out my other articles on all kinds of programming-related topics.
Happy coding!
— Mike
P.S: like what I’m doing? Follow me:
