Fix "Future attached to a different loop" error by ensuring tasks are created in the correct event loop

- Explicitly retrieve and use the correct event loop when creating tasks to avoid cross-loop issues.
- Ensures proper task scheduling in environments with multiple event loops.
This commit is contained in:
Aravind Karnam
2024-11-26 14:05:02 +05:30
parent a98d51a62c
commit a888c91790
3 changed files with 8 additions and 4 deletions

View File

@@ -264,7 +264,8 @@ class BFSScraperStrategy(ScraperStrategy):
self.stats.current_depth = depth
if parallel_processing:
task = asyncio.create_task(
loop = asyncio.get_event_loop()
task = loop.create_task(
self.process_url(url, depth, crawler, queue, visited, depths)
)
pending_tasks.add(task)