The mapping functions are handled in the same function? #761
-
Lithops with AWS Lambda backend handles user map function requests by using multiprocess lib to enable parallel processing. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
Independently of the memory size, AWS Lambda provides the parallelism by starting a separate container for each activation of a For memory sizes > 1769MB, where you get more than one cpu per container, you can use the |
Beta Was this translation helpful? Give feedback.
Independently of the memory size, AWS Lambda provides the parallelism by starting a separate container for each activation of a
map()
. By default, each started container processes 1 element of theiterable
. For example, if theiterable
of amap(func, itrerable)
contains 1000 elements, Lithops + AWS lambda will start 1000 separate containers, each one processing one element of theiterable
.For memory sizes > 1769MB, where you get more than one cpu per container, you can use the
worker_processes
config key to send multiple activation to the same worker, this way you will leverage all the CPUs in a container.