I am building a webscraper using Python: Selenium is the backend, and PyQt5 is the frontend. The purpose of the frontend is to gather information from the user that affects decision-making in the backend.

A big part of webscraping using Selenium is using WebDriverWait to wait until a certain element is present. Whenever Selenium is waiting, or completing some other task of a noticeable length, the frontend freezes. You can't interact with it at all.

In order to make the frontend responsive while the backend is working, I tried putting backend activity into a QThread. It seems like this would normally be a good solution, except Selenium hates being anywhere other than the main thread and crashes.

What is the best way to make the frontend responsive while the backend is running? Could the frontend be moved to a thread instead of the backend? Is multiprocessing an option? (It seems unnecessarily complicated). Any ideas would be great. Thanks!