Multi-thread operations not compatible with UC

[Karuiizawa]

Hi guys,

Well after many tries I just can't get UC to work with multithreads (Celery), the issue comes from the patcher.py that doesn't support concurrent race runs :

Please upgrade your database plan.
2023-04-22T17:56:28.641855+00:00 app[worker.1]: [2023-04-22 17:56:28,641: ERROR/ForkPoolWorker-1] Task myapp.tasks.search_product[a5aac04e-6ade-479a-9e40-7b08dda70e11] raised unexpected: FileNotFoundError(2, 'No such file or directory')
2023-04-22T17:56:28.641875+00:00 app[worker.1]: Traceback (most recent call last):
2023-04-22T17:56:28.641876+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
2023-04-22T17:56:28.641876+00:00 app[worker.1]:     R = retval = fun(*args, **kwargs)
2023-04-22T17:56:28.641876+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
2023-04-22T17:56:28.641877+00:00 app[worker.1]:     return self.run(*args, **kwargs)
2023-04-22T17:56:28.641877+00:00 app[worker.1]:   File "/app/myapp/tasks.py", line 546, in search_product
2023-04-22T17:56:28.641877+00:00 app[worker.1]:     results.extend(future.result())
2023-04-22T17:56:28.641878+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-04-22T17:56:28.641878+00:00 app[worker.1]:     return self.__get_result()
2023-04-22T17:56:28.641879+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-04-22T17:56:28.641879+00:00 app[worker.1]:     raise self._exception
2023-04-22T17:56:28.641879+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2023-04-22T17:56:28.641879+00:00 app[worker.1]:     result = self.fn(*self.args, **self.kwargs)
2023-04-22T17:56:28.641879+00:00 app[worker.1]:   File "/app/myapp/tasks.py", line 433, in scrape_website
2023-04-22T17:56:28.641880+00:00 app[worker.1]:     driver = create_selenium_instance(current_thread().ident)
2023-04-22T17:56:28.641880+00:00 app[worker.1]:   File "/app/myapp/tasks.py", line 384, in create_selenium_instance
2023-04-22T17:56:28.641880+00:00 app[worker.1]:     temp_driver = uc.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"))
2023-04-22T17:56:28.641880+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/undetected_chromedriver/__init__.py", line 246, in __init__
2023-04-22T17:56:28.641881+00:00 app[worker.1]:     self.patcher.auto()
2023-04-22T17:56:28.641881+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/undetected_chromedriver/patcher.py", line 128, in auto
2023-04-22T17:56:28.641881+00:00 app[worker.1]:     self.unzip_package(self.fetch_package())
2023-04-22T17:56:28.641881+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.10/site-packages/undetected_chromedriver/patcher.py", line 181, in unzip_package
2023-04-22T17:56:28.641882+00:00 app[worker.1]:     os.rename(os.path.join(self.zip_path, self.exe_name), self.executable_path)
2023-04-22T17:56:28.641884+00:00 app[worker.1]: FileNotFoundError: [Errno 2] No such file or directory: '/app/.local/share/undetected_chromedriver/undetected/chromedriver' -> '/app/.local/share/undetected_chromedriver/undetected_chromedriver'

I tried to patch the issue in the patcher using shutil and also tried to assign to each thread it's own instance of chromedriver :

import shutil

def create_selenium_instance(thread_id):
    thread_chromedriver_path = os.path.join(os.path.dirname(os.environ.get("CHROMEDRIVER_PATH")), f"chromedriver_{thread_id}")

    if not os.path.exists(thread_chromedriver_path):
        # Create a separate instance of undetected_chromedriver.Chrome
        temp_driver = uc.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"))
        temp_driver.quit()

        # Copy the patched chromedriver to the thread-specific path
        shutil.copy2(temp_driver.binary_path, thread_chromedriver_path)

    options = webdriver.ChromeOptions()
    options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
    options.add_argument("--headless")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--no-sandbox")

    driver = uc.Chrome(executable_path=thread_chromedriver_path, options=options)
    return driver

But still I get the same issue.

That's unfortunate since we really need multithread support in order to scale the process
Anyone got it to work ?

[f999145]

I don't know how much this solves the problem. But uc works great in multiprocessing.
I see you're using handle. I never managed to apply handless. The author said that the work of handless is unstable.

[ericmclachlan]

@f999145 : When you're saying "handle" here, do you mean headless?

[f999145]

Yes I meant handless