ayrisdev/crawl4ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Adding LLMContext generator to website.

Browse Source
This commit is contained in:
UncleCode
2025-05-24 20:37:09 +08:00
parent ad078c3f18
commit a06710ff03
38 changed files with 101901 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
docs/md_v2/llmtxt/build.md Normal file
View File

@@ -0,0 +1,75 @@
O**Prompt for AI Coding Assistant: Create an Interactive LLM Context Builder Page**
**Objective:**
Your task is to create an interactive HTML webpage with JavaScript functionality that allows users to select and combine different `crawl4ai` LLM context files into a single downloadable Markdown (`.md`) file. This tool will empower users to craft tailored context for their AI assistants based on their specific needs.
**Core Functionality:**
1. **Display `crawl4ai` Components:** The page will list all available `crawl4ai` documentation components.
2. **Select Context Types:** For each component, users can select which types of context they want to include:
* Memory (API facts)
* Reasoning (How-to/why)
* Examples (Code snippets)
(All should be selected by default for each initially selected component).
3. **Special "Aggregate" Contexts:** Include options for special, pre-combined contexts:
* "Vibe Coding" (a curated mix for general AI prompting)
* "All Library Context" (a comprehensive aggregation of all memory, reasoning, and examples for the entire library).
4. **Fetch and Concatenate:** When the user clicks a "Download Combined Context" button:
* The JavaScript will fetch the content of all selected Markdown files from the server (from a predefined folder, e.g., `/llmtxt/`).
* It will concatenate the content of these files into a single string.
5. **Client-Side Download:** The concatenated content will be offered to the user as a download (e.g., `custom_crawl4ai_context.md`).
**Input/Assumptions:**
* **Context Files Location:** All individual context Markdown files are located on the server in a publicly accessible folder named `llmtxt/`.
* **File Naming Convention:** Files follow the pattern: `crawl4ai_{{component_name}}_[memory|reasoning|examples]_content.llm.md`.
* `{{component_name}}` can contain underscores (e.g., `deep_crawling`, `config_objects`).
* The special contexts will have names like `crawl4ai_vibe_content.llm.md` and `crawl4ai_all_content.llm.md`.
* **Component List:** You will be provided with a list of `crawl4ai` components. For this implementation, use the following list:
* `core`
* `config_objects`
* `deep_crawling`
* `deployment` (covers Installation & Docker Deployment)
* `extraction` (covers Structured Data Extraction)
* `markdown` (covers Markdown Generation Algorithm)
* `pdf_processing`
* *(No separate "Vibe Coding" or "All Library Context" in this list, as they are special top-level selections)*
**Detailed UI/UX Requirements:**
1. **Main Page Structure:**
* **Header:** "Crawl4AI Interactive LLM Context Builder"
* **Introduction:** Briefly explain the purpose of the tool (from the `USING_LLM_CONTEXTS.md` content you helped draft: "Supercharging Your AI Assistant...").
* **Selection Area:**
* **Special Aggregate Contexts (Radio Buttons or Prominent Checkboxes):**
* [ ] "Vibe Coding Context" (`crawl4ai_vibe_content.llm.md`)
* [ ] "All Library Context (Comprehensive)" (`crawl4ai_all_content.llm.md`)
* *Behavior:* Selecting one of these might disable individual component selections (or vice-versa) to avoid redundancy, or simply add them to the list. Consider user experience here. A simple approach is that if an aggregate is selected, it's the *only* thing downloaded.
* **Individual Component Selection (Table or List of Checkboxes):**
* A section titled "Select Individual Components & Context Types:"
* For each component in the provided list:
* A master checkbox for the component itself (e.g., `[ ] Core Functionality`). Selected by default.
* Nested checkboxes (indented or grouped) for context types, enabled only if the parent component is checked:
* `[x] Memory (API Facts)`
* `[x] Reasoning (How-to/Why)`
* `[x] Examples (Code Snippets)`
(These three sub-checkboxes should be selected by default if the parent component is selected).
* **Action Button:**
* A button: "Generate & Download Combined Context"
* **Status/Feedback Area:** (Optional, but good UX)
* Display messages like "Fetching files...", "Combining context...", "Download starting..." or error messages.
**Final Output:**
* A single HTML file (e.g., `interactive_context_builder.html`).
* Associated JavaScript code (can be inline within `<script>` tags or in a separate `.js` file).
* Associated CSS code (can be inline within `<style>` tags or in a separate `.css` file).
This interactive tool will greatly enhance the user experience for `crawl4ai` developers looking to leverage your specialized LLM contexts. Please ensure the JavaScript is robust and provides good user feedback.
---
This prompt should give your AI coding assistant a very clear set of requirements and guidelines for building the interactive context builder. Remember to provide it with the list of components as mentioned in the "Input/Assumptions" section.

142
docs/md_v2/llmtxt/index.html Normal file
View File

@@ -0,0 +1,142 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Crawl4AI LLM Context Builder</title>
<link rel="stylesheet" href="llmtxt.css">
</head>
<body>
<div class="container">
<header class="header">
<h1><span class="logo">🚀🤖</span> Crawl4AI LLM Context Builder</h1>
</header>
<section class="intro">
<div class="intro-header">
<h2>🧠 A New Approach to LLM Context</h2>
<p>
Traditional <code>llm.txt</code> files often fail with complex libraries like Crawl4AI. They dump massive amounts of API documentation, causing <strong>information overload</strong> and <strong>lost focus</strong>. They provide the "what" but miss the crucial "how" and "why" that makes AI assistants truly helpful.
</p>
</div>
<div class="intro-solution">
<h3>💡 The Solution: Multi-Dimensional, Modular Contexts</h3>
<p>
Inspired by modular libraries like Lodash, I've redesigned how we provide context to AI assistants. Instead of one monolithic file, Crawl4AI's documentation is organized by <strong>components</strong> and <strong>perspectives</strong>.
</p>
<div class="dimensions">
<div class="dimension">
<span class="badge memory">Memory</span>
<h4>The "What"</h4>
<p>Precise API facts, parameters, signatures, and configuration objects. Your unambiguous reference.</p>
</div>
<div class="dimension">
<span class="badge reasoning">Reasoning</span>
<h4>The "How" & "Why"</h4>
<p>Design principles, best practices, trade-offs, and workflows. Teaches AI to think like an expert.</p>
</div>
<div class="dimension">
<span class="badge examples">Examples</span>
<h4>The "Show Me"</h4>
<p>Runnable code snippets demonstrating patterns in action. Pure practical implementation.</p>
</div>
</div>
</div>
<div class="intro-benefits">
<p>
<strong>Why this matters:</strong> You can now give your AI assistant exactly what it needs - whether that's quick API lookups, help designing solutions, or seeing practical implementations. No more information overload, just focused, relevant context.
</p>
<p class="learn-more">
<a href="/blog/articles/llm-context-revolution" class="learn-more-link">📖 Read the full story behind this approach →</a>
</p>
</div>
</section>
<section class="builder">
<div class="special-contexts">
<h2>Quick Presets</h2>
<div class="preset-options">
<label class="preset-option">
<input type="radio" name="preset" value="vibe" id="preset-vibe">
<div class="preset-card">
<h3>🎯 Vibe Coding</h3>
<p>Curated context for general AI prompting - perfect for exploring capabilities</p>
</div>
</label>
<label class="preset-option">
<input type="radio" name="preset" value="all" id="preset-all">
<div class="preset-card">
<h3>📚 Complete Library</h3>
<p>Comprehensive context including all components and perspectives</p>
</div>
</label>
<label class="preset-option">
<input type="radio" name="preset" value="custom" id="preset-custom" checked>
<div class="preset-card">
<h3>🔧 Custom Selection</h3>
<p>Choose specific components and context types</p>
</div>
</label>
</div>
</div>
<div class="component-selector" id="component-selector">
<h2>Select Components & Context Types</h2>
<div class="select-all-controls">
<button class="btn-small" id="select-all">Select All</button>
<button class="btn-small" id="deselect-all">Deselect All</button>
</div>
<div class="component-table-wrapper">
<table class="component-selection-table">
<thead>
<tr>
<th width="50"></th>
<th>Component</th>
<th class="clickable-header" data-type="memory">Memory</th>
<th class="clickable-header" data-type="reasoning">Reasoning</th>
<th class="clickable-header" data-type="examples">Examples</th>
</tr>
</thead>
<tbody id="components-tbody">
<!-- Components will be dynamically inserted here -->
</tbody>
</table>
</div>
</div>
<div class="action-area">
<button class="download-btn" id="download-btn">
<span class="icon">⬇</span> Generate & Download Context
</button>
<div class="status" id="status"></div>
</div>
</section>
<section class="reference-table">
<h2>Available Context Files</h2>
<div class="table-wrapper">
<table class="context-table">
<thead>
<tr>
<th>Component</th>
<th>Memory</th>
<th>Reasoning</th>
<th>Examples</th>
<th>Full</th>
</tr>
</thead>
<tbody id="reference-table-body">
<!-- Table rows will be dynamically inserted here -->
</tbody>
</table>
</div>
</section>
</div>
<script src="llmtxt.js"></script>
</body>
</html>

456
docs/md_v2/llmtxt/llmtxt.css Normal file
View File

@@ -0,0 +1,456 @@
/* Terminal Theme CSS for LLM Context Builder */
:root {
--background-color: #070708;
--font-color: #e8e9ed;
--primary-color: #50ffff;
--primary-dimmed: #09b5a5;
--secondary-color: #d5cec0;
--tertiary-color: #a3abba;
--accent-color: rgb(243, 128, 245);
--error-color: #ff3c74;
--code-bg-color: #3f3f44;
--border-color: #3f3f44;
--hover-bg: #1a1a1c;
--success-color: #50ff50;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 0;
font-family: dm, Monaco, Courier New, monospace;
font-size: 14px;
line-height: 1.5;
background-color: var(--background-color);
color: var(--font-color);
}
.container {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
/* Header */
.header {
text-align: center;
margin-bottom: 40px;
border-bottom: 1px dashed var(--tertiary-color);
padding-bottom: 20px;
}
.header h1 {
font-size: 24px;
color: var(--primary-color);
margin: 0;
text-transform: uppercase;
letter-spacing: 2px;
}
.logo {
font-size: 28px;
vertical-align: middle;
}
/* Intro Section */
.intro {
background-color: var(--code-bg-color);
border: 1px solid var(--border-color);
padding: 30px;
margin-bottom: 30px;
}
.intro-header h2 {
color: var(--primary-color);
margin: 0 0 15px 0;
font-size: 20px;
}
.intro-header p {
line-height: 1.6;
margin-bottom: 0;
}
.intro-header code {
background-color: var(--hover-bg);
padding: 2px 6px;
color: var(--primary-dimmed);
}
.intro-solution {
margin-top: 5px;
padding-top: 25px;
border-top: 1px dashed var(--border-color);
}
.intro-solution h3 {
color: var(--secondary-color);
margin: 0 0 15px 0;
font-size: 18px;
}
.dimensions {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 20px;
margin: 20px 0;
}
.dimension {
background-color: var(--hover-bg);
padding: 20px;
border: 1px solid var(--border-color);
transition: all 0.2s ease;
}
.dimension:hover {
border-color: var(--primary-dimmed);
}
.dimension h4 {
color: var(--font-color);
margin: 10px 0 8px 0;
font-size: 16px;
}
.dimension p {
font-size: 13px;
line-height: 1.5;
color: var(--tertiary-color);
margin: 0;
}
.intro-benefits {
margin-top: 0px;
padding-top: 0x;
border-top: 1px dashed var(--border-color);
}
.intro-benefits strong {
color: var(--primary-color);
}
.learn-more {
margin-top: 15px;
}
.learn-more-link {
color: var(--primary-dimmed);
text-decoration: none;
font-weight: bold;
transition: color 0.2s ease;
}
.learn-more-link:hover {
color: var(--primary-color);
text-decoration: underline;
}
.badge {
display: inline-block;
padding: 2px 8px;
font-size: 12px;
text-transform: uppercase;
margin-right: 8px;
}
.badge.memory {
background-color: var(--primary-dimmed);
color: var(--background-color);
}
.badge.reasoning {
background-color: var(--accent-color);
color: var(--background-color);
}
.badge.examples {
background-color: var(--secondary-color);
color: var(--background-color);
}
/* Builder Section */
.builder {
margin-bottom: 40px;
}
.builder h2 {
color: var(--primary-color);
font-size: 18px;
margin-bottom: 20px;
text-transform: uppercase;
}
/* Preset Options */
.preset-options {
display: flex;
gap: 20px;
margin-bottom: 30px;
}
.preset-option {
flex: 1;
cursor: pointer;
}
.preset-option input[type="radio"] {
display: none;
}
.preset-card {
border: 2px solid var(--border-color);
padding: 20px;
transition: all 0.2s ease;
background-color: var(--code-bg-color);
}
.preset-card h3 {
margin: 0 0 10px 0;
color: var(--secondary-color);
font-size: 16px;
}
.preset-card p {
margin: 0;
font-size: 12px;
color: var(--tertiary-color);
}
.preset-option input:checked + .preset-card {
border-color: var(--primary-color);
background-color: var(--hover-bg);
}
.preset-card:hover {
border-color: var(--primary-dimmed);
}
/* Component Selector */
.component-selector {
margin-bottom: 30px;
}
.select-all-controls {
margin-bottom: 20px;
}
.btn-small {
background-color: var(--code-bg-color);
color: var(--font-color);
border: 1px solid var(--border-color);
padding: 5px 15px;
margin-right: 10px;
cursor: pointer;
font-family: inherit;
font-size: 12px;
text-transform: uppercase;
transition: all 0.2s ease;
}
.btn-small:hover {
background-color: var(--primary-dimmed);
color: var(--background-color);
}
/* Component Selection Table */
.component-table-wrapper {
overflow-x: auto;
border: 1px solid var(--border-color);
margin-top: 20px;
}
.component-selection-table {
width: 100%;
border-collapse: collapse;
background-color: var(--code-bg-color);
}
.component-selection-table th,
.component-selection-table td {
padding: 12px;
text-align: left;
border-bottom: 1px solid var(--border-color);
}
.component-selection-table th {
background-color: var(--hover-bg);
color: var(--primary-color);
text-transform: uppercase;
font-size: 12px;
letter-spacing: 1px;
font-weight: bold;
}
.component-selection-table th.clickable-header {
cursor: pointer;
user-select: none;
transition: background-color 0.2s ease;
}
.component-selection-table th.clickable-header:hover {
background-color: var(--primary-dimmed);
color: var(--background-color);
}
.component-selection-table th:not(:first-child) {
text-align: center;
width: 120px;
}
.component-selection-table td {
font-size: 13px;
}
.component-selection-table td:not(:first-child) {
text-align: center;
}
.component-selection-table tr:hover td {
background-color: var(--hover-bg);
}
.component-name {
color: var(--primary-color);
font-weight: bold;
}
.component-selection-table input[type="checkbox"] {
cursor: pointer;
width: 16px;
height: 16px;
}
/* Disabled row state */
.component-selection-table tr.disabled td:not(:first-child) {
opacity: 0.5;
pointer-events: none;
}
/* Action Area */
.action-area {
text-align: center;
margin: 40px 0;
}
.download-btn {
background-color: var(--primary-dimmed);
color: var(--background-color);
border: none;
padding: 15px 40px;
font-size: 16px;
font-family: inherit;
cursor: pointer;
text-transform: uppercase;
letter-spacing: 1px;
transition: all 0.2s ease;
display: inline-flex;
align-items: center;
gap: 10px;
}
.download-btn:hover {
background-color: var(--primary-color);
transform: translateY(-2px);
}
.download-btn .icon {
font-size: 20px;
}
.status {
margin-top: 20px;
font-size: 14px;
min-height: 30px;
}
.status.loading {
color: var(--primary-color);
}
.status.success {
color: var(--success-color);
}
.status.error {
color: var(--error-color);
}
/* Reference Table */
.reference-table {
margin-top: 60px;
}
.reference-table h2 {
color: var(--primary-color);
font-size: 18px;
margin-bottom: 20px;
text-transform: uppercase;
}
.table-wrapper {
overflow-x: auto;
border: 1px solid var(--border-color);
}
.context-table {
width: 100%;
border-collapse: collapse;
background-color: var(--code-bg-color);
}
.context-table th,
.context-table td {
padding: 12px;
text-align: left;
border-bottom: 1px solid var(--border-color);
}
.context-table th {
background-color: var(--hover-bg);
color: var(--primary-color);
text-transform: uppercase;
font-size: 12px;
letter-spacing: 1px;
}
.context-table td {
font-size: 13px;
}
.context-table tr:hover td {
background-color: var(--hover-bg);
}
.file-link {
color: var(--primary-dimmed);
text-decoration: none;
}
.file-link:hover {
color: var(--primary-color);
text-decoration: underline;
}
.file-size {
color: var(--tertiary-color);
font-size: 11px;
}
/* Responsive Design */
@media (max-width: 768px) {
.preset-options {
flex-direction: column;
}
.components-grid {
grid-template-columns: 1fr;
}
.container {
padding: 10px;
}
}

456
docs/md_v2/llmtxt/llmtxt.js Normal file
View File

@@ -0,0 +1,456 @@
// Crawl4AI LLM Context Builder JavaScript
// Component definitions
const components = [
{
id: 'core',
name: 'Core Functionality',
description: 'Basic crawling and scraping features'
},
{
id: 'config_objects',
name: 'Configuration Objects',
description: 'Browser and crawler configuration'
},
{
id: 'deep_crawling',
name: 'Deep Crawling',
description: 'Multi-page crawling strategies'
},
{
id: 'deployment',
name: 'Deployment',
description: 'Installation and Docker setup'
},
{
id: 'extraction',
name: 'Data Extraction',
description: 'Structured data extraction strategies'
},
{
id: 'markdown',
name: 'Markdown Generation',
description: 'Content-to-markdown conversion'
},
{
id: 'vibe',
name: 'Vibe Coding',
description: 'General-purpose AI context',
special: false
}
];
// Context types
const contextTypes = ['memory', 'reasoning', 'examples'];
// State management
const state = {
preset: 'custom',
selectedComponents: new Set(),
selectedContextTypes: new Map()
};
// Initialize the application
document.addEventListener('DOMContentLoaded', () => {
setupPresetHandlers();
renderComponents();
renderReferenceTable();
setupActionHandlers();
setupColumnHeaderHandlers();
// Initialize all components as selected with all context types
components.forEach(comp => {
if (!comp.special) {
state.selectedComponents.add(comp.id);
state.selectedContextTypes.set(comp.id, new Set(contextTypes));
}
});
updateComponentUI();
});
// Setup preset radio button handlers
function setupPresetHandlers() {
const presetRadios = document.querySelectorAll('input[name="preset"]');
presetRadios.forEach(radio => {
radio.addEventListener('change', (e) => {
state.preset = e.target.value;
updatePresetSelection();
});
});
}
// Update UI based on preset selection
function updatePresetSelection() {
const componentSelector = document.getElementById('component-selector');
if (state.preset === 'custom') {
componentSelector.style.display = 'block';
} else {
componentSelector.style.display = 'none';
}
}
// Render component selection table
function renderComponents() {
const tbody = document.getElementById('components-tbody');
tbody.innerHTML = '';
components.filter(c => !c.special).forEach(component => {
const row = createComponentRow(component);
tbody.appendChild(row);
});
}
// Create a component table row
function createComponentRow(component) {
const tr = document.createElement('tr');
tr.id = `component-${component.id}`;
// Component checkbox cell
const checkboxCell = document.createElement('td');
checkboxCell.innerHTML = `
<input type="checkbox" id="check-${component.id}"
data-component="${component.id}">
`;
tr.appendChild(checkboxCell);
// Component name cell
const nameCell = document.createElement('td');
nameCell.innerHTML = `<span class="component-name">${component.name}</span>`;
tr.appendChild(nameCell);
// Context type cells
contextTypes.forEach(type => {
const td = document.createElement('td');
td.innerHTML = `
<input type="checkbox" id="check-${component.id}-${type}"
data-component="${component.id}" data-type="${type}">
`;
tr.appendChild(td);
});
// Add event listeners
const mainCheckbox = tr.querySelector(`#check-${component.id}`);
mainCheckbox.addEventListener('change', (e) => {
handleComponentToggle(component.id, e.target.checked);
});
// Add event listeners for context type checkboxes
contextTypes.forEach(type => {
const typeCheckbox = tr.querySelector(`#check-${component.id}-${type}`);
typeCheckbox.addEventListener('change', (e) => {
handleContextTypeToggle(component.id, type, e.target.checked);
});
});
return tr;
}
// Handle component checkbox toggle
function handleComponentToggle(componentId, checked) {
if (checked) {
state.selectedComponents.add(componentId);
// Select all context types when component is selected
if (!state.selectedContextTypes.has(componentId)) {
state.selectedContextTypes.set(componentId, new Set(contextTypes));
} else {
// If component was already partially selected, select all
state.selectedContextTypes.set(componentId, new Set(contextTypes));
}
} else {
state.selectedComponents.delete(componentId);
state.selectedContextTypes.delete(componentId);
}
updateComponentUI();
}
// Handle component selection based on context types
function updateComponentSelection(componentId) {
const types = state.selectedContextTypes.get(componentId) || new Set();
if (types.size > 0) {
state.selectedComponents.add(componentId);
} else {
state.selectedComponents.delete(componentId);
}
}
// Handle context type checkbox toggle
function handleContextTypeToggle(componentId, type, checked) {
if (!state.selectedContextTypes.has(componentId)) {
state.selectedContextTypes.set(componentId, new Set());
}
const types = state.selectedContextTypes.get(componentId);
if (checked) {
types.add(type);
} else {
types.delete(type);
}
updateComponentSelection(componentId);
updateComponentUI();
}
// Update UI to reflect current state
function updateComponentUI() {
components.filter(c => !c.special).forEach(component => {
const row = document.getElementById(`component-${component.id}`);
const mainCheckbox = row.querySelector(`#check-${component.id}`);
const hasSelection = state.selectedComponents.has(component.id);
const selectedTypes = state.selectedContextTypes.get(component.id) || new Set();
// Update main checkbox
mainCheckbox.checked = hasSelection;
// Update row disabled state
row.classList.toggle('disabled', !hasSelection);
// Update context type checkboxes
contextTypes.forEach(type => {
const typeCheckbox = row.querySelector(`#check-${component.id}-${type}`);
typeCheckbox.checked = selectedTypes.has(type);
});
});
}
// Setup action button handlers
function setupActionHandlers() {
// Select/Deselect all buttons
document.getElementById('select-all').addEventListener('click', () => {
components.filter(c => !c.special).forEach(comp => {
state.selectedComponents.add(comp.id);
state.selectedContextTypes.set(comp.id, new Set(contextTypes));
});
updateComponentUI();
});
document.getElementById('deselect-all').addEventListener('click', () => {
state.selectedComponents.clear();
state.selectedContextTypes.clear();
updateComponentUI();
});
// Download button
document.getElementById('download-btn').addEventListener('click', handleDownload);
}
// Setup column header click handlers
function setupColumnHeaderHandlers() {
const headers = document.querySelectorAll('.clickable-header');
headers.forEach(header => {
header.addEventListener('click', () => {
const type = header.getAttribute('data-type');
toggleColumnSelection(type);
});
});
}
// Toggle all checkboxes in a column
function toggleColumnSelection(type) {
// Check if all are currently selected
let allSelected = true;
components.filter(c => !c.special).forEach(comp => {
const types = state.selectedContextTypes.get(comp.id);
if (!types || !types.has(type)) {
allSelected = false;
}
});
// Toggle all
components.filter(c => !c.special).forEach(comp => {
if (!state.selectedContextTypes.has(comp.id)) {
state.selectedContextTypes.set(comp.id, new Set());
}
const types = state.selectedContextTypes.get(comp.id);
if (allSelected) {
types.delete(type);
} else {
types.add(type);
}
updateComponentSelection(comp.id);
});
updateComponentUI();
}
// Handle download action
async function handleDownload() {
const statusEl = document.getElementById('status');
statusEl.textContent = 'Preparing context files...';
statusEl.className = 'status loading';
try {
const files = getSelectedFiles();
if (files.length === 0) {
throw new Error('No files selected. Please select at least one component or preset.');
}
statusEl.textContent = `Fetching ${files.length} files...`;
const contents = await fetchFiles(files);
const combined = combineContents(contents);
downloadFile(combined, 'crawl4ai_custom_context.md');
statusEl.textContent = 'Download complete!';
statusEl.className = 'status success';
setTimeout(() => {
statusEl.textContent = '';
statusEl.className = 'status';
}, 3000);
} catch (error) {
statusEl.textContent = `Error: ${error.message}`;
statusEl.className = 'status error';
}
}
// Get list of selected files based on current state
function getSelectedFiles() {
const files = [];
if (state.preset === 'vibe') {
files.push('crawl4ai_vibe.llm.full.md');
} else if (state.preset === 'all') {
// Use the dedicated aggregated files for all components
files.push('crawl4ai_all_memory_content.llm.md');
files.push('crawl4ai_all_reasoning_content.llm.md');
files.push('crawl4ai_all_examples_content.llm.md');
} else {
// Custom selection
state.selectedComponents.forEach(compId => {
const types = state.selectedContextTypes.get(compId);
if (types) {
types.forEach(type => {
files.push(`crawl4ai_${compId}_${type}_content.llm.md`);
});
}
});
}
return files;
}
// Fetch multiple files
async function fetchFiles(fileNames) {
// Use /assets/llmtxt/ path with .txt extension
const baseUrl = '/assets/llmtxt/';
const promises = fileNames.map(async (fileName) => {
// Convert .md to .txt for fetching
const txtFileName = fileName.replace('.md', '.txt');
try {
const response = await fetch(baseUrl + txtFileName);
if (!response.ok) {
console.warn(`Failed to fetch ${txtFileName} from ${baseUrl + txtFileName}`);
return { fileName, content: `<!-- Failed to load ${fileName} -->` };
}
const content = await response.text();
return { fileName, content };
} catch (error) {
console.warn(`Error fetching ${txtFileName} from ${baseUrl + txtFileName}:`, error);
return { fileName, content: `<!-- Error loading ${fileName} -->` };
}
});
return Promise.all(promises);
}
// Combine file contents with headers
function combineContents(fileContents) {
const header = `# Crawl4AI Custom LLM Context
Generated on: ${new Date().toISOString()}
Total files: ${fileContents.length}
---
`;
const sections = fileContents.map(({ fileName, content }) => {
const componentName = extractComponentName(fileName);
const contextType = extractContextType(fileName);
return `## ${componentName} - ${contextType}
Source: ${fileName}
${content}
---
`;
});
return header + sections.join('\n');
}
// Extract component name from filename
function extractComponentName(fileName) {
// Pattern: crawl4ai_{component}_{type}_content.llm.md
const match = fileName.match(/crawl4ai_(.+?)_(memory|reasoning|examples|llm\.full)/);
if (match) {
const compId = match[1];
const component = components.find(c => c.id === compId);
return component ? component.name : compId.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase());
}
return 'Unknown Component';
}
// Extract context type from filename
function extractContextType(fileName) {
if (fileName.includes('_memory_')) return 'Memory';
if (fileName.includes('_reasoning_')) return 'Reasoning';
if (fileName.includes('_examples_')) return 'Examples';
if (fileName.includes('.llm.full')) return 'Complete Context';
return 'Context';
}
// Download file to user's computer
function downloadFile(content, fileName) {
const blob = new Blob([content], { type: 'text/markdown' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = fileName;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
}
// Render reference table
function renderReferenceTable() {
const tbody = document.getElementById('reference-table-body');
tbody.innerHTML = '';
// Since vibe is no longer special, just show all components the same way
components.forEach(component => {
const row = document.createElement('tr');
row.innerHTML = `
<td><strong>${component.name}</strong></td>
<td><a href="/assets/llmtxt/crawl4ai_${component.id}_memory_content.llm.txt" class="file-link" target="_blank">Memory</a></td>
<td><a href="/assets/llmtxt/crawl4ai_${component.id}_reasoning_content.llm.txt" class="file-link" target="_blank">Reasoning</a></td>
<td><a href="/assets/llmtxt/crawl4ai_${component.id}_examples_content.llm.txt" class="file-link" target="_blank">Examples</a></td>
<td><a href="/assets/llmtxt/crawl4ai_${component.id}.llm.full.txt" class="file-link" target="_blank">Full</a></td>
`;
tbody.appendChild(row);
});
}
// Check if examples file exists (all components have examples)
function hasExamplesFile(componentId) {
// All components have examples files
return true;
}
// Check if full file exists (all components have full files)
function hasFullFile(componentId) {
// All components have full files
return true;
}
// Utility function to capitalize first letter
function capitalizeFirst(str) {
return str.charAt(0).toUpperCase() + str.slice(1);
}

37
docs/md_v2/llmtxt/why.md Normal file
View File

@@ -0,0 +1,37 @@
# Supercharging Your AI Assistant: My Journey to Better LLM Contexts for `crawl4ai`
When I started diving deep into using AI coding assistants with my own libraries, particularly `crawl4ai`, I quickly realized that the common approach to providing context via a simple `llm.txt` or even a beefed-up `README.md` just wasn't cutting it. This document explains the problems I encountered and how I've tried to create a more effective system for `crawl4ai`, allowing you (and your AI assistant) to get precisely the right information.
## My Frustration with Standard `llm.txt` Files
My experience with generic `llm.txt` files for complex libraries like `crawl4ai` revealed several pain points:
1. **Information Overload & Lost Focus:** I found that when I threw a massive, monolithic context file at an LLM, it often struggled. The sheer volume of information seemed to dilute its focus. If I asked a specific question about a niche feature, the LLM might get sidetracked by more prominent but currently irrelevant parts of the library. It felt like trying to find a single sentence in a thousand-page novel – the information was *there*, but not always accessible or prioritized correctly by the AI.
2. **The "What" Without the "How" or "Why":** Most `llm.txt` files I encountered were essentially API dumps – a list of functions, classes, and parameters. This is the "what" of a library. But to truly use a library effectively, especially one as flexible as `crawl4ai`, you need the "how" (idiomatic usage patterns, best practices for common tasks) and the "why" (the design rationale behind certain features). Without this, I noticed my AI assistant would often generate syntactically correct but practically inefficient or non-idiomatic code. It was guessing the *intent* and the *best way* to use the library, and those guesses weren't always right.
3. **No Guidance on "Thinking" Like an Expert:** A static list of facts doesn't teach an LLM the *art* of using the library. It doesn't convey the trade-offs an experienced developer considers, the common pitfalls they've learned to avoid, or the clever ways to combine features to solve complex problems. I wanted my AI assistant to not just recall an API, but to help me *reason* about the best way to build a solution with `crawl4ai`.
## Inspiration: Selective Inclusion & Multi-Dimensional Understanding
I've always admired how libraries like Lodash or jQuery (in its modular days) allowed developers to pick and choose only the parts they needed, resulting in smaller, more focused bundles. This idea of modularity and selective inclusion resonated deeply with me as I thought about LLM context. Why force-feed an LLM the entire library's details when I'm only working on a specific component or task?
This led me to develop a new approach for `crawl4ai`: **multi-dimensional, modular contexts**.
Instead of one giant `llm.txt`, I've broken down the `crawl4ai` documentation into:
1. **Logical Components:** Context is organized around the major functional areas of the library (e.g., Core, Data Extraction, Deep Crawling, Markdown Generation, etc.). This allows you to select context relevant only to the task at hand.
2. **Three Dimensions of Context for Each Component:**
* **`_memory.md` (Foundational Memory):** This is the "what." It contains the precise, factual information about the component's public API, data structures, configuration objects, parameters, and method signatures. It's the detailed, unambiguous reference.
* **`_reasoning.md` (Reasoning & Problem-Solving Framework):** This is the "how" and "why." It includes design principles, common task workflows with decision guides, best practices, anti-patterns, illustrative code examples solving real problems, and explanations of trade-offs. It aims to guide the LLM in "thinking" like an expert `crawl4ai` user.
* **`_examples.md` (Practical Code Examples):** This is pure "show-me-the-code." It's a collection of runnable snippets demonstrating various ways to use the component's features and configurations, with minimal explanatory text. It’s for quickly seeing different patterns in action.
**The Goal:**
My aim is to provide you with a flexible system. You can give your AI assistant:
* Just the **memory** files for quick API lookups.
* The **reasoning** files (perhaps with memory) for help designing solutions.
* The **examples** files for seeing practical implementations.
* A **combination** of these across one or more components tailored to your specific task.
* Or, for broader understanding, special aggregate contexts like the "Vibe Coding" context or the "All Library Context."
By providing these structured, multi-faceted contexts, I hope to significantly improve the quality and relevance of the assistance you get when using AI to code with `crawl4ai`. The following sections will guide you on how to select and use these different context files.