Enhance crawler capabilities and documentation

- Add llm.txt generator
  - Added SSL certificate extraction in AsyncWebCrawler.
  - Introduced new content filters and chunking strategies for more robust data extraction.
  - Updated documentation.

docs/llm.txt/6_chunking_strategies.q.md

@@ -1,53 +1,10 @@
-### Hypothetical Questions
-
-1. **General Purpose of Chunking**
-   - *"Why is chunking text important before applying cosine similarity or building RAG pipelines?"*
-   - *"How does dividing large texts into smaller chunks improve retrieval accuracy and scalability?"*
-   
-2. **Regex-Based Chunking**
-   - *"How can I split text into chunks using a custom regular expression?"*
-   - *"What are typical use cases for Regex-based chunking, and when should I prefer it over other methods?"*
-   
-3. **Sentence-Based Chunking**
-   - *"How do I break text into individual sentences using an NLP approach like `sent_tokenize`?"*
-   - *"When should I prefer sentence-based chunking over regex-based or fixed-length chunking?"*
-   
-4. **Topic-Based Segmentation**
-   - *"What is topic-based segmentation, and how does it produce thematically coherent chunks?"*
-   - *"How can I integrate TextTiling or other topic segmentation algorithms into my chunking pipeline?"*
-   
-5. **Fixed-Length Word Chunking**
-   - *"How do I evenly distribute text into fixed-size word chunks?"*
-   - *"What are the benefits and drawbacks of using a fixed-length chunking strategy?"*
-   
-6. **Sliding Window Chunking**
-   - *"What is a sliding window approach, and how does overlapping chunks improve context retention?"*
-   - *"How do I choose appropriate window sizes and step values for my sliding window chunking?"*
-   
-7. **Cosine Similarity Integration**
-   - *"How do I apply cosine similarity to identify the most relevant chunks for a given query?"*
-   - *"What preprocessing steps are necessary before computing cosine similarity between a query and the generated chunks?"*
-   
-8. **RAG (Retrieval-Augmented Generation) Applications**
-   - *"How can chunking strategies facilitate integration with Retrieval-Augmented Generation systems?"*
-   - *"Which chunking method is best suited for maintaining context in RAG-based pipelines?"*
-   
-9. **Practical Considerations & Best Practices**
-   - *"How do I choose the right chunking strategy for my specific use case (e.g., documents, transcripts, webpages)?"*
-   - *"What are some best practices for combining chunking, vectorization, and similarity scoring methods?"*
-   
-10. **Advanced Use Cases**
-    - *"Can I combine multiple chunking strategies, such as applying sentence tokenization followed by a sliding window?"*
-    - *"How do I handle very large documents or corpora with chunking and similarity extraction at scale?"*
-
-### Topics Discussed in the File
-
-- **Purpose of Chunking Strategies**: Facilitating cosine similarity retrieval and RAG system integration.
-- **Regex-Based Chunking**: Splitting text based on patterns (e.g., paragraphs, blank lines).
-- **Sentence-Based Chunking**: Using NLP techniques to create sentence-level segments for fine-grained analysis.
-- **Topic-Based Segmentation**: Grouping text into topical units for thematic coherence.
-- **Fixed-Length Word Chunking**: Dividing text into uniform word count segments for consistent structure.
-- **Sliding Window Chunking**: Overlapping segments to preserve contextual continuity.
-- **Integrating Cosine Similarity**: Pairing chunked text with a query to retrieve the most relevant content.
-- **Applications in RAG Systems**: Enhancing retrieval workflows by organizing content into meaningful chunks.
-- **Comparison of Chunking Methods**: Trade-offs between simplicity, coherence, and context preservation.
+chunking_overview: Chunking strategies divide large texts into manageable parts for content processing and extraction | text segmentation, content division, document splitting | None
+cosine_similarity_integration: Chunking prepares text segments for semantic similarity analysis using cosine similarity | semantic search, relevance matching | from sklearn.metrics.pairwise import cosine_similarity
+rag_integration: Chunks can be integrated into RAG (Retrieval-Augmented Generation) systems for structured workflows | retrieval augmented generation, RAG pipeline | None
+regex_chunking: Split text using regular expression patterns for basic segmentation | regex splitting, pattern-based chunking | RegexChunking(patterns=[r'\n\n'])
+sentence_chunking: Divide text into individual sentences using NLP tools | sentence tokenization, NLP chunking | from nltk.tokenize import sent_tokenize
+topic_chunking: Create topic-coherent chunks using TextTiling algorithm | topic segmentation, TextTiling | from nltk.tokenize import TextTilingTokenizer
+fixed_length_chunking: Segment text into chunks with fixed word count | word-based chunking, fixed size segments | FixedLengthWordChunking(chunk_size=100)
+sliding_window_chunking: Generate overlapping chunks for better context preservation | overlapping segments, windowed chunking | SlidingWindowChunking(window_size=100, step=50)
+cosine_similarity_extraction: Extract relevant chunks using TF-IDF and cosine similarity comparison | similarity search, relevance extraction | from sklearn.feature_extraction.text import TfidfVectorizer
+chunking_workflow: Combine chunking with cosine similarity for enhanced content retrieval | content extraction, similarity workflow | CosineSimilarityExtractor(query).find_relevant_chunks(chunks)