ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add use_stemming option to BM25ContentFilter (#1192)

Browse Source
This commit is contained in:
UncleCode
2025-06-08 06:57:37 +02:00
parent 4eb90b41b6
commit ef6f4329fa
3 changed files with 20 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/md_v2/core/markdown-generation.md
View File

@@ -200,7 +200,7 @@ config = CrawlerRunConfig(markdown_generator=md_generator)
- **`user_query`**: The term you want to focus on. BM25 tries to keep only content blocks relevant to that query.
- **`bm25_threshold`**: Raise it to keep fewer blocks; lower it to keep more.
- **`use_stemming`**: If `True`, variations of words match (e.g., “learn,” “learning,” “learnt”).
- **`use_stemming`** *(default `True`)*: If enabled, variations of words match (e.g., “learn,” “learning,” “learnt”).
**No query provided?** BM25 tries to glean a context from page metadata, or you can simply treat it as a scorched-earth approach that discards text with low generic score. Realistically, you want to supply a query for best results.