perf(crawler): major performance improvements & raw HTML support

Browse Source

- Switch to lxml parser (~4x speedup)
- Add raw HTML & local file crawling support
- Fix cache headers & async cleanup
- Add browser process monitoring
- Optimize BeautifulSoup operations
- Pre-compile regex patterns

Breaking: Raw HTML handling requires new URL prefixes
Fixes: #256, #253

...

This commit is contained in:

UncleCode

2024-11-13 19:40:40 +08:00

parent 61b93ebf36

commit c38ac29edb

11 changed files with 2953 additions and 130 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

2179

tests/async/sample_wikipedia.html Normal file

File diff suppressed because one or more lines are too long