Run an async function from a sync function within an already-running event loop
06:50 02 Aug 2024

In my Python application, I have a sync function boo() that is called inside a running event loop. boo() has to get some data from foo(arg1, arg2), which is an async function.

Unfortunately, I can't turn boo() into an async function. It must stay synchronized. (This constraint is out of my hands).

How can I call foo(arg1, arg2) from within boo(), wait until it completes, and continue the execution?

Minimal Reproducible Example

I tried to create a minimal reproducible example. This is the closest I could get. The real application is big and complex, and may behave differently.

import time
import asyncio

async def work_for_data():
    time.sleep(3)
    return 42

# sync function, calling async function
def get_number():
    return asyncio.get_event_loop().run_until_complete(work_for_data())

async def get_data():
    return get_number()

async def run():
    loop = asyncio.get_event_loop()
    task = asyncio.create_task(get_data())
    loop.run_until_complete(task)


if __name__ == "__main__":
    asyncio.run(run())

This code raises:

  File "./minimal_example.py", line 9, in get_number
    return asyncio.get_event_loop().run_until_complete(work_for_data())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 629, in run_until_complete
    self._check_running()
  File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 588, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

Attempts To Solve The Problem

I made a lot of attempts to solve it, all of them didn't work.

Attempt 1

data = asyncio.run(foo(arg1, arg2))

Raised the following exception:

    data = asyncio.run(foo(arg1, arg2))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pycharm_helpers/pydevd_asyncio/pydevd_nest_asyncio.py", line 143, in run
    loop.run_until_complete(task)
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 518, in uvloop.loop.Loop._run
RuntimeError: this event loop is already running.

Attempt 2

loop = asyncio.get_event_loop()
data = loop.run_until_complete(foo(arg1, arg2))

Raised the following exception:

    data = loop.run_until_complete(foo(arg1, arg2))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 518, in uvloop.loop.Loop._run
RuntimeError: this event loop is already running.

Attempt 3

loop = asyncio.get_running_loop()
    with ThreadPoolExecutor() as executor:
        future = executor.submit(lambda: asyncio.run_coroutine_threadsafe(foo(arg1, arg2), loop).result())
        data = future.result()

The interpreter got stuck when executing future.result()

Attempt 4

    loop = asyncio.get_event_loop()
    future = asyncio.Future()

    def callback(task):
        if task.exception():
            future.set_exception(task.exception())
        else:
            future.set_result(task.result())

    task = asyncio.run_coroutine_threadsafe(foo(arg1, arg2), loop)
    task.add_done_callback(callback)

    result = task.result()  ## Stuck here
    return result

The interpreter got stuck when executing task.result()

python async-await python-asyncio future coroutine