Query About Error Retry Mechanism in Function Calls

[lzcmian]

Hello Lithops Team,

Firstly, I would like to express my gratitude for your work on the Lithops project. It has been immensely helpful in my endeavors.

I'm currently using lithops for parallel execution of a potentially unstable function that can throw errors randomly, as demonstrated in the code snippet below:

import lithops

def unreliable_function(x):
    """This is an unstable function that may throw an error."""
    import random
    if random.random() > 0.7:
        raise ValueError("An error occurred")
    return x * x

with lithops.FunctionExecutor() as fexec:
    futures = fexec.map(unreliable_function, range(10))
    results = fexec.get_result(throw_except=False)
    print(results)  # Output: [0, None, None, None, 16, 25, 36, None, 64, 81]

I am seeking a feature or method to automatically retry the function call when an error occurs in any of the workers. I have reviewed the documentation but could not find any mention of such a feature. Perhaps I might have missed something, or there might be a recommended pattern to handle such cases using existing functionalities?

Could you please advise if there is a built-in mechanism in Lithops to handle retries of function calls when exceptions occur, or if there is any suggested method to implement such functionality?

Thank you very much for your assistance and for the great work on Lithops.

[JosepSampe]

Hi @lzcmian, in Lithops there are 2 different ways to retry failed function calls:

1. One is to retry all the failed calls manually at the end of the execution. You can find a simple example here.
2. The second is to use the RetryingFunctionExecutor. This option is a new feature contributed by @tomwhite in Add a mechanism to automatically retry failed tasks #1291, that allows to automatically retry failed calls as soon as they fail instead of doing it manually, but it is not yet released an documented. If you want to try this option, you have to use master branch, and from lithops import RetryingFunctionExecutor. We will add examples and docs in the near future. For now you can check out the tests in here

[lzcmian]

I understand, thank you for your help.

[tomwhite]

Another option to consider is to retry failed calls internally in your code - using something like Tenacity you would annotate unreliable_function with @retry. This is a lightweight solution as the retry is done within the same Lithops task container, whereas the other solutions use a new container (which is more robust if the container itself is slow for some reason).

[lzcmian]

I've learned a lot, thank you very much.