docs(api): add streaming mode documentation and examples
Add comprehensive documentation for the new streaming mode feature in arun_many(): - Update arun_many() API docs to reflect streaming return type - Add streaming examples in quickstart and multi-url guides - Document stream parameter in configuration classes - Add clone() helper method documentation for configs This change improves documentation for processing large numbers of URLs efficiently.
This commit is contained in:
@@ -265,9 +265,21 @@ async def quick_parallel_example():
|
||||
"https://example.com/page3"
|
||||
]
|
||||
|
||||
run_conf = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
|
||||
run_conf = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
stream=True # Enable streaming mode
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Stream results as they complete
|
||||
async for result in await crawler.arun_many(urls, config=run_conf):
|
||||
if result.success:
|
||||
print(f"[OK] {result.url}, length: {len(result.markdown_v2.raw_markdown)}")
|
||||
else:
|
||||
print(f"[ERROR] {result.url} => {result.error_message}")
|
||||
|
||||
# Or get all results at once (default behavior)
|
||||
run_conf = run_conf.clone(stream=False)
|
||||
results = await crawler.arun_many(urls, config=run_conf)
|
||||
for res in results:
|
||||
if res.success:
|
||||
@@ -279,8 +291,13 @@ if __name__ == "__main__":
|
||||
asyncio.run(quick_parallel_example())
|
||||
```
|
||||
|
||||
The example above shows two ways to handle multiple URLs:
|
||||
1. **Streaming mode** (`stream=True`): Process results as they become available using `async for`
|
||||
2. **Batch mode** (`stream=False`): Wait for all results to complete
|
||||
|
||||
For more advanced concurrency (e.g., a **semaphore-based** approach, **adaptive memory usage throttling**, or customized rate limiting), see [Advanced Multi-URL Crawling](../advanced/multi-url-crawling.md).
|
||||
|
||||
---
|
||||
|
||||
## 8. Dynamic Content Example
|
||||
|
||||
|
||||
Reference in New Issue
Block a user