For some time now I have been interested in the question of operator overloading in Python. Until now, there is no
such trick in Python as method overloading, although this is a classic trick from OOP. There is
a functools.singledispatch crutch, but it is not a native overload. And I don't like the syntax either. Luciano Ramalho, author of "Fluent Python" book says: "The @singledispatch decorator is not intended to method overloading in Python in the spirit of Java", and i agree with him.
I want to see a beautiful overload, like in the C# language. Look at this beauty implemented in C#:
public class Overloading
{
public int add(int a, int b)
{
return a + b;
}
public int add(int a, int b, int c)
{
return a + b + c;
}
public float add(float a, float b, float c, float d)
{
return a + b + c + d;
}
}
And look at Java example:
class Overloading {
private String formatNumber(int value) {
return String.format("%d", value);
}
private String formatNumber(double value) {
return String.format("%.3f", value);
}
private String formatNumber(String value) {
return String.format("%.2f", Double.parseDouble(value));
}
}
Imagine if a reference like this was possible in Python:
class Overloading:
def fetch(self, value: str): ...
def fetch(self, value: int): ...
def fetch(self, value: tuple): ...
I'm not sure if the Python client interface can achieve this behavior, but I decided to make my own implementation of
method overloading in Python without diving into the rabbit hole of semantics, metaclasses and C reference.
The first decision that comes to mind is decoration. Decorating methods is a good idea for many reasons.
Firstly, it will allow existing code to be used without modification, since the purpose of a decorator is to modify the behavior of a function or method, without changing the semantics of the function/method body.
Secondly, since the decoration allows to execute any custom parameterized code, our decorator will be able to receive the necessary parameters as input, which using we will be able to understand what to do next.
The logic that our code will decorate will be to create a hash table that will contain all the declared variations of the semantics of a particular function.
This is what you need! Let's look at an example below:
from functools import lru_cache
from typing import Callable, Optional
# argument to string (string as hash table key)
def determine_types(args, kwargs):
args_str = "&".join(str(a) for a in args)
kwargs_str = "&".join(f"{k}={v}" for k, v in kwargs.items())
result = f"{args_str}&{kwargs_str}"
return result
class Overload:
__DEFAULT_HASH_TABLE = dict()
def __init__(self, *args, separate_hash_table: dict = None, **kwargs):
self.args = args
self.kwargs = kwargs
self.func: Optional[Callable] = None
self.hash_table = separate_hash_table \
if separate_hash_table else self.__DEFAULT_HASH_TABLE
def __call__(self, *args, **kwargs):
"""
:param args: args[0] - wrapped func or method
:param kwargs: empty!
:return:
"""
self.func = args[0]
# hash function algo
hash_function_key = abs(
hash(
determine_types(self.args, self.kwargs) +
str(id(self.func))
)
)
# Definition case
# Saving function in hash table for hash_function_key
self.hash_table.update({
hash_function_key: self.func
})
# Evaluation case
# Return a wrapper that will return the
# target function from the hash table
def call_function_by_hash_table_key(*a, **kw):
return self.hash_table \
.get(self.func.__name__) \
.get(hash_function_key)(*a, **kw)
return call_function_by_hash_table_key
# We don't care that the semantics of our code redefines
# the fetch function name. Due to the fact that we store each function in
# our hash table, we are not afraid of redefining the function name.
# Thus, we get the purity of function names and access
# to each function redefinen by our semantics
class Test:
@Overload(str, int, callback=None)
def fetch(self, a, b, callback=None):
return a * b
@Overload(int, int, callback=lru_cache)
def fetch(self, a, b, callback=lru_cache):
return a * b, callback
# when we calling the `fetch` function, the interpreter will refer
# to the last defined name `fetch`, and thanks to the logic of our
# decorator, it will receive for execution exactly the decorated
# function that is stored in our hash table by the corresponding key,
# which is the semantics of the input types
test = Test()
print(test.fetch("1", 2, callback=None))
print(test.fetch(1, 2, callback=lru_cache))
>>> ('11', None)
>>> (2, <function lru_cache at 0x104d43f40>)
This implementation is not perfect, because it requires additional memory for the hash table, but the result does not look bad, agree?
The next step I would like to get rid of the excessive type declaration inside the Overload() call. I want to modify
the decorator so that it doesn't need input parameters at all. It takes as input a function that already has everything
it needs for a successful snapping. To do this, I want to use inspect.getfullargspec, to parse the function inside the wrapper and search for the types of all arguments. This will make it mandatory to type annotation at the input of the function. Here's how I want to see it:
# Current implementation view
@Overload(int, int, callback=lru_cache)
def fetch(self, a, b, callback=lru_cache): ...
# Desired implementation view
@Overload()
def fetch(self, a: int, b: int, callback: Callable = lru_cache): ...
And at this point, I come up with a brilliant idea - to see the implementations of Python's method overloading on
Github! 🤦♂️
I found something that has already been done for me. Greetings, to the overloading package 🤦♂️🤦♂️🤦♂️
Using this package will display the following:
@overload
def biggest(items: Iterable[int]):
return max(items)
@overload
def biggest(items: Iterable[str]):
return max(items, key=len)
Looks exactly the way I want it to, but let's see how it's implemented. The first line of code immediately catches your
eye import ast. It seems that someone has already implemented my idea lol.
To be honest, I think the source code for this package is quite overloaded. I don't understand why there are 800+ lines of code with such a very trivial functionality. There is so much code that I will be honest, I have not read 10 percent of the contents of this package. But perhaps my implementation will eventually cease to yield in the complexity of this implementation.
Back to my realization! I want to leave the decorator with an empty call when the type annotation is mandatory!
I remove all arguments when calling the Overload decorator and annotate the method argument types:
class Test:
@Overload()
def fetch(self, a: int, b: int, callback: Callable = None):
return a * b
Now, for the decorator to work again, we need to explore the types of arguments using inspect.getfullargspec, and slightly redo the locke of the hash key formation for our hash table!
Now our client class looks like this:
class Test:
@Overload()
def fetch(self, a: int, b: int = 0, callback: Callable = None):
return a * b
@Overload()
def fetch(self, a: str, b: int, callback: Callable = None):
return a * b, callback
And the decorating class benefits like this:
class Overload:
__DEFAULT_HASH_TABLE = dict()
def __init__(self, separate_hash_table: dict = None):
self.func: Optional[Callable] = None
self.hash_table = separate_hash_table \
if separate_hash_table else self.__DEFAULT_HASH_TABLE
def __call__(self, *args, **kwargs):
"""
:param args: args[0] - wrapped func or method
:param kwargs: empty!
:return: Callable wrapper
"""
self.func = args[0]
# Definition case
# Inspecting wrapped function for getting types annotations
argspec: inspect.FullArgSpec = inspect.getfullargspec(self.func)
# Common case
# Declaring key for hash table (it's not unique yet)
key = f"{argspec.args}:{argspec.defaults}"
# Common case
# Hash function algo
hash_function_key = abs(
# making hash with declaring key and wrapped function id
# (its fully unique now, and it's ready for hash table)
hash(
key + str(id(self.func))
)
)
# Definition case
# Saving function in hash table for hash_function_key
self.hash_table.update({
hash_function_key: self.func
})
# Evalueation case
# Return a wrapper that will return the
# target function from the hash table
def call_function_by_hash_table_key(*a, **kw):
return self.hash_table.get(hash_function_key)(*a, **kw)
return call_function_by_hash_table_key
Please note that *args, **kwargs as input parameters in the Overload initializer have been removed, as our
decorator no longer needs an additional type declaration!
In the __call__ method, a check of the input types of the decorated function was added:
# Definition case
# Inspecting wrapped function for getting types annotations
argspec: inspect.FullArgSpec = inspect.getfullargspec(self.func)
And based on the types of input arguments and their default values, a hash key was constructed:
# Common case
# Declaring key for hash table (it's not unique yet)
key = f"{argspec.args}:{argspec.defaults}"
Accordingly, the hashing algorithm has changed slightly, since in the previous implementation, the algorithm implied
hashing the return value of the determine_types function, which we no longer need. The hashing algorithm now looks like this:
# Common case
# Hash function algo
hash_function_key = abs(
# making hash with declaring key and wrapped function id
# (its fully unique now, and it's ready for hash table)
hash(
key + str(id(self.func))
)
)
A very simple, but at the same time reliable algorithm for generating hash keys for our table hash. We will never get
duplicate functions in our hash table unless two or more absolutely identical names of functions are declared in
one Python namespace. In practice, I have never seen such a thing. Overloading methods assumes different semantics of input parameters with methods of the same name. But if the semantics of the input parameters are identical, then this is no longer an overload of methods, but their overriding! Just imagine two alsolute identical functions:
def foo(a: int, b: int): ...
def foo(a: int, b: int): ...
There can be no question of any overload of methods in such a case! This is a pure redefinition of the method for all canonons. Therefore, if the purity of the differentiality of semantics is observed with a 99% probability, we can say that our hash table will not have collisions!
>>> test = Test()
>>> print(test.fetch("1", 2, callback=None))
>>> print(test.fetch(1, 2, callback=lru_cache))
>>> ('11', None)
>>> (2, <function lru_cache at 0x104bb3f40>)
Good!
I like the result more than the very first decision. But I guess I'll go further, because I have another idea. It's completely insane because it doesn't make any sense, but for my moral satisfaction, I want to make it happen!
I want to add another parameter to the key generation algorithm for the table hash, namely the type of the return value. This will generate code that resembles the following:
class Test:
@Overload()
def fetch(self, a: int, b: int,
callback: Callable = None) -> int:
return a * b
@Overload()
def fetch(self, a: int, b: int,
callback: Callable) -> tuple[int, Callable]:
return a * b, callback
It's an absolutely crazy and dumb idea. There's no sense in it, because in practice we will never, under any circumstances, probably need this functionality, but I don't care! I want - I do!
The first two parameters a: int, b: int in both methods are identical. The difference is in the default value of the callback parameter and in the types of values returned. In the first method, there is a default value for the callback parameter and a return type of int. In the second method, there are no default values, respectively, the callback parameter in this case is mandatory. And to demonstrate how method overload works, the type of the return value is different from the value in the first method.
Our hash key generation algorithm will separate both cases, and return from the scenery the one that corresponds to the semantics of the input parameters when called. That is, I expect a behavior in which in the absence of callback as an input parameter - the decorator will return to us the first method with the default value of callback and the return type int. And in the case when callback is present as an input parameter, the decorator will return us to the second method with the return type tuple[int, Callable].
Absolutely frostbitten, delusional, useless idea. But I don't care, I like it!
In Python, no one will ever need to overload methods of this level, but I want to remind you what a real method
overload looks like on the example of the C# language:
public class Overloading
{
// Return type int
public int add(int a, int b)
{
return a + b;
}
// Return type int
public int add(int a, int b, int c)
{
return a + b + c;
}
// Return type float
public float add(float a, float b, float c, float d)
{
return a + b + c + d;
}
}
Note that the types of return values also participate in method overload semantics. And that means my job isn't over!
By the way, it's already 11 am, and I want to finish this article like crazy, because I myself became interested!
In fact, already at this stage, both methods are stored in our hash table under different keys. That is, using the current codebase, you can create a class with overloaded methods of this level:
class Test:
@Overload()
def fetch(self, a: int, b: int,
callback: Callable = None):
return a * b
@Overload()
def fetch(self, a: int, b: int,
callback: Callable):
return a * b, callback
print(test.fetch("1", 2, callback=None))
print(test.fetch(1, 2, callback=lru_cache))
We get the following output:
>>> ('11', None)
>>> (2, <function lru_cache at 0x10507ff40>)
Let's see how our decorated functions are saved in the hash table. Add a print to the end of the __call__ method:
print(self.hash_table)
return call_function_by_hash_table_key
>>> {
>>> 2435609516054484431: <function Test.fetch at 0x1050dd090>
>>> }
>>> {
>>> 2435609516054484431: <function Test.fetch at 0x1050dd090>,
>>> 7828357193962827624: <function Test.fetch at 0x1050dd1b0>
>>> }
Let me remind you that the default values are involved in the formation of hash keys for our decorated functions, which
means that everything is logical. Obviously, the semantics of the method def fetch(self, a: int, b: int, callback: Callable = None) and the semantics of the method def fetch(self, a: int, b: int, callback: Callable) are different. And if the semantics are different, then the hash keys are different! That is, we could stop there, but as I said, I want to add the types of return values to the algorithm for generating hash keys! Moreover, for this we need quite a bit, namely, to change the line of formation of the hash key:
# Common case
# Declaring key for hash table (it's not unique yet)
key = f"{argspec.args}:{argspec.defaults}:{argspec.annotations.get('return')}"
Thats All!
My implementation is functionally ready! I did some refactoring. You can see the result below:
import inspect
from typing import Callable, Optional
class Overload:
__DEFAULT_HASH_TABLE = dict()
def __init__(self, separate_hash_table: dict = None):
self.func: Optional[Callable] = None
self.hash_table = separate_hash_table \
if separate_hash_table else self.__DEFAULT_HASH_TABLE
self.argspec_key_pattern = "{args}:{defaults}:{returns}"
def get_function_hash(self, key):
# Default hash function algo
# You can customize hashing algorithm you want
return abs(
# Making hash with declaring key and wrapped function id
# (its fully unique now, and it's ready for hash table)
hash(
key + str(id(self.func))
)
)
def __call__(self, *args, **kwargs):
"""
:param args: args[0] - wrapped func or method
:param kwargs: empty!
:return: Callable wrapper
"""
self.func = args[0]
# Definition case
# Inspecting wrapped function for getting types annotations
argspec: inspect.FullArgSpec = inspect.getfullargspec(self.func)
# Ccomon case
# Declaring key for hash table (it's not unique yet)
key = self.argspec_key_pattern.format(
args=argspec.args,
defaults=argspec.defaults,
returns=argspec.annotations.get('return')
)
# Common case
# Hash function algo
function_hash_key = self.get_function_hash(key)
# Definition case
# Saving function in hash table for hash_function_key
self.hash_table.update({
function_hash_key: self.func
})
# Evaluation case
# Return a wrapper that will return the
# target function from the hash table
wrapper = self.get_mapped_function(function_hash_key)
return wrapper
def get_mapped_function(self, function_hash_key):
# Default wrapper
# You can customize it you want
def wrapper(*a, **kw):
try:
return self.hash_table[function_hash_key](*a, **kw)
except KeyError as e:
# Exception case when function_hash_key
# was not found in hash table
# In this case will be returns input wrapped func
return self.func(*a, **kw)
return wrapper
P.S A perfect representation of method overload that will most likely never appear in Python
Personally, I imagine the ideal implementation of method overloading in Python something like this:
from typing import Protocol
class Overloaded(Protocol):
# some protocol declaring
__overloaded__ = ()
...
class A(Overloaded):
__overloaded__ = ('fetch',)
def fetch(self) -> None: ...
def fetch(self, value: int) -> int: ...
def fetch(self, value: str) -> str: ...
def fetch(self, value: Sequence) -> Sequence: ...
Perhaps there will be a continuation of this article if I have the desire to understand the creation of my own Python
protocols to implement a protocol that implements my method overload! If you like this idea, write a comment below!