`.
+## 3. BM25ContentFilter
+### 3.1 Usage Example
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+from crawl4ai.content_filter_strategy import BM25ContentFilter
+from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
+
+async def main():
+ # 1) A BM25 filter with a user query
+ bm25_filter = BM25ContentFilter(
+ user_query="startup fundraising tips",
+ # Adjust for stricter or looser results
+ bm25_threshold=1.2
+ )
+
+ # 2) Insert into a Markdown Generator
+ md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)
+
+ # 3) Pass to crawler config
+ config = CrawlerRunConfig(
+ markdown_generator=md_generator
+ )
+
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url="https://news.ycombinator.com",
+ config=config
+ )
+ if result.success:
+ print("Fit Markdown (BM25 query-based):")
+ print(result.markdown.fit_markdown)
+ else:
+ print("Error:", result.error_message)
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+### 3.2 Parameters
+- **`user_query`** (str, optional): E.g. `"machine learning"`. If blank, the filter tries to glean a query from page metadata.
+- **`bm25_threshold`** (float, default 1.0):
+ - Higher → fewer chunks but more relevant.
+ - Lower → more inclusive.
+> In more advanced scenarios, you might see parameters like `language`, `case_sensitive`, or `priority_tags` to refine how text is tokenized or weighted.
+## 4. Accessing the “Fit” Output
+After the crawl, your “fit” content is found in **`result.markdown.fit_markdown`**.
+```python
+fit_md = result.markdown.fit_markdown
+fit_html = result.markdown.fit_html
+```
+If the content filter is **BM25**, you might see additional logic or references in `fit_markdown` that highlight relevant segments. If it’s **Pruning**, the text is typically well-cleaned but not necessarily matched to a query.
+## 5. Code Patterns Recap
+### 5.1 Pruning
+```python
+prune_filter = PruningContentFilter(
+ threshold=0.5,
+ threshold_type="fixed",
+ min_word_threshold=10
+)
+md_generator = DefaultMarkdownGenerator(content_filter=prune_filter)
+config = CrawlerRunConfig(markdown_generator=md_generator)
+```
+### 5.2 BM25
+```python
+bm25_filter = BM25ContentFilter(
+ user_query="health benefits fruit",
+ bm25_threshold=1.2
+)
+md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)
+config = CrawlerRunConfig(markdown_generator=md_generator)
+```
+## 6. Combining with “word_count_threshold” & Exclusions
+```python
+config = CrawlerRunConfig(
+ word_count_threshold=10,
+ excluded_tags=["nav", "footer", "header"],
+ exclude_external_links=True,
+ markdown_generator=DefaultMarkdownGenerator(
+ content_filter=PruningContentFilter(threshold=0.5)
+ )
+)
+```
+1. The crawler’s `excluded_tags` are removed from the HTML first.
+3. The final “fit” content is generated in `result.markdown.fit_markdown`.
+## 7. Custom Filters
+If you need a different approach (like a specialized ML model or site-specific heuristics), you can create a new class inheriting from `RelevantContentFilter` and implement `filter_content(html)`. Then inject it into your **markdown generator**:
+```python
+from crawl4ai.content_filter_strategy import RelevantContentFilter
+
+class MyCustomFilter(RelevantContentFilter):
+ def filter_content(self, html, min_word_threshold=None):
+ # parse HTML, implement custom logic
+ return [block for block in ... if ... some condition...]
+
+```
+1. Subclass `RelevantContentFilter`.
+2. Implement `filter_content(...)`.
+3. Use it in your `DefaultMarkdownGenerator(content_filter=MyCustomFilter(...))`.
+## 8. Final Thoughts
+- **Summaries**: Quickly get the important text from a cluttered page.
+- **Search**: Combine with **BM25** to produce content relevant to a query.
+- **BM25ContentFilter**: Perfect for query-based extraction or searching.
+- Combine with **`excluded_tags`, `exclude_external_links`, `word_count_threshold`** to refine your final “fit” text.
+- Fit markdown ends up in **`result.markdown.fit_markdown`**; eventually **`result.markdown.fit_markdown`** in future versions.
+- Last Updated: 2025-01-01
+
+
+# Content Selection
+Crawl4AI provides multiple ways to **select**, **filter**, and **refine** the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove certain domains and images, **`CrawlerRunConfig`** offers a wide range of parameters.
+## 1. CSS-Based Selection
+There are two ways to select content from a page: using `css_selector` or the more flexible `target_elements`.
+### 1.1 Using `css_selector`
+A straightforward way to **limit** your crawl results to a certain region of the page is **`css_selector`** in **`CrawlerRunConfig`**:
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+async def main():
+ config = CrawlerRunConfig(
+ # e.g., first 30 items from Hacker News
+ css_selector=".athing:nth-child(-n+30)"
+ )
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url="https://news.ycombinator.com/newest",
+ config=config
+ )
+ print("Partial HTML length:", len(result.cleaned_html))
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+**Result**: Only elements matching that selector remain in `result.cleaned_html`.
+### 1.2 Using `target_elements`
+The `target_elements` parameter provides more flexibility by allowing you to target **multiple elements** for content extraction while preserving the entire page context for other features:
+```python
+import asyncio
+from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
+
+async def main():
+ config = CrawlerRunConfig(
+ # Target article body and sidebar, but not other content
+ target_elements=["article.main-content", "aside.sidebar"]
+ )
+ async with AsyncWebCrawler() as crawler:
+ result = await crawler.arun(
+ url="https://example.com/blog-post",
+ config=config
+ )
+ print("Markdown focused on target elements")
+ print("Links from entire page still available:", len(result.links.get("internal", [])))
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+**Key difference**: With `target_elements`, the markdown generation and structural data extraction focus on those elements, but other page elements (like links, images, and tables) are still extracted from the entire page. This gives you fine-grained control over what appears in your markdown content while preserving full page context for link analysis and media collection.
+## 2. Content Filtering & Exclusions
+### 2.1 Basic Overview
+```python
+config = CrawlerRunConfig(
+ # Content thresholds
+ word_count_threshold=10, # Minimum words per block
+
+ # Tag exclusions
+ excluded_tags=['form', 'header', 'footer', 'nav'],
+
+ # Link filtering
+ exclude_external_links=True,
+ exclude_social_media_links=True,
+ # Block entire domains
+ exclude_domains=["adtrackers.com", "spammynews.org"],
+ exclude_social_media_domains=["facebook.com", "twitter.com"],
+
+ # Media filtering
+ exclude_external_images=True
+)
+```
+- **`word_count_threshold`**: Ignores text blocks under X words. Helps skip trivial blocks like short nav or disclaimers.
+- **`excluded_tags`**: Removes entire tags (`