feat(crawler): add network request and console message capturing
Implement comprehensive network request and console message capturing functionality: - Add capture_network_requests and capture_console_messages config parameters - Add network_requests and console_messages fields to models - Implement Playwright event listeners to capture requests, responses, and console output - Create detailed documentation and examples - Add comprehensive tests This feature enables deep visibility into web page activity for debugging, security analysis, performance profiling, and API discovery in web applications.
This commit is contained in:
17
tests/general/test_crawlers.py
Normal file
17
tests/general/test_crawlers.py
Normal file
@@ -0,0 +1,17 @@
|
||||
|
||||
# example_usageexample_usageexample_usage# example_usage.py
|
||||
import asyncio
|
||||
from crawl4ai.crawlers import get_crawler
|
||||
|
||||
async def main():
|
||||
# Get the registered crawler
|
||||
example_crawler = get_crawler("example_site.content")
|
||||
|
||||
# Crawl example.com
|
||||
result = await example_crawler(url="https://example.com")
|
||||
|
||||
print(result)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user