await in a python for loop never finishing - google-colaboratory

Question

I'm trying a new thing for me, using playwright in google colab.
this combination requires/forces async programming.
I've got a context manager which is able to handle the login and logout called "Login". That works great!
The internal page I'm trying to get to has datasets, with no links, just div's to click on.
the locator (I believe) is working fine and should return multiple elements when combined with .element_handles() I'm assuming.
from playwright.async_api import async_playwright
import asyncio
from IPython.display import Image
import nest_asyncio
nest_asyncio.apply()
# browser is set to webkit in the Login() context manager
...
async def loop_over_datasets(browser=None, page=None):
print("starting")
datasets = page.locator("div.horizontal.clickable")
print("continuing")
datasets = await asyncio.gather(datasets.element_handles())
for ds in datasets:
print(f'inside the loop, ds is {ds}')
print("doesn't get here in tact")
# for each dataset I want to launch a new page where the dataset is clicked but I'll settle for sync programming at this point.
# new_page = await ds.click()
# ds_page = await browser.new_page(new_page)
# ds_page.click()
async def get_all_new_info():
async with Login() as (b,l):
await loop_over_datasets(browser=b,page = l)
asyncio.run(get_all_new_info()) #has to be killed manually or it will run forever.
In the line datasets = await asyncio.gather(datasets.element_handles()) gather() doesn't actually work without await and await never returns
which means I don't get "inside the loop...".
without await I get the "ds" variable but it's not anything I can do something with.
How is this supposed to be used?

Answer 1

Without full code it's a little bit hard to test but wanted to share few things that may help:
datasets = await asyncio.gather(datasets.element_handles())
As far as I can see in Playwright documentation element_handles() returns <List[ElementHandle]> and your are trying to pass this list to asyncio.gather which needs awaitable objects which are coroutines, Tasks, and Futures and probably thats why it's not working, so I would just done
datasets = datasets.element_handles()
Now, I assume you'd like to go through those datasets in an asynchronous manner. You should be able to put the content of the for loop into a coroutine and based on that create tasks that will be executed by gather.
async def process_dataset(ds):
new_page = await ds.click()
ds_page = await browser.new_page(new_page)
ds_page.click()
tasks = []
for ds in datasets:
tasks.append(asyncio.create_task(process_dataset(ds)))
await asyncio.gather(*tasks)