Commit Graph

Select branches

Hide Pull Requests

0.3.5

0.3.6

0.3.7

0.3.72

0.3.73

0.3.74

0.3.742

0.3.743

0.3.744

0.3.745

0.3.75

0.4.0

0.4.1

0.4.2

2025-JUN-1

add-claude-github-actions-1759553116682

bug/proxy_config

bugfix/arun-many-cdp-managed-browser

claude/fix-update-pyopenssl-security-011CUPexU25DkNvoxfu5ZrnB

claude/implement-webhook-crawl-feature-011CULZY1Jy8N5MUkZqXkRVp

coderabbitai/docstrings/14vTVzYa3bH06l5wYNY9jTghrrj9FxxWL

codex/add-httpx-and-https-http2]-packages

codex/add-memory_wait_timeout-parameter-to-memoryadaptivedispatche

codex/add-use_stemming-parameter-to-bm25contentfiler

codex/add-vnc-streaming-endpoint-to-docker-server

codex/find-and-fix-a-bug

codex/fix-indexerror-in-browser-manager-py-with-use-managed-browse

copilot/modify-page-creation-and-logging

deploy

develop

devin/1748137705-fix-bm25contentfilter-docs

docker-test

docker/add_features

docker/base_config_overrides

docker/fix_sig

docs

docs-llm-strategies-update

docs-proxy-security

extract-media

feat/ahmed_dev

feat/undetected-browser

feature/agent-oai

feature/async-llm-extaction

feature/c4a-script

feature/configHealthMonitor

feature/content-filter

feature/content-filter-nasrin-1

feature/docker-cluster

feature/docker-hooks

feature/docker-llm-parameters

feature/marketplace-sponsor-logo

feature/nasrin-cli-deep-crawl

feature/scraper

feature/scraping-strategy

feature/telemetry

fix-async-url-seeder-redirect-verification

fix-cors-disable-web-security

fix/adaptive-crawler-llm-config

fix/async-llm-extraction-arunMany

fix/case_senstive_params

fix/cdp

fix/configurable-backoff

fix/deep-crawl-scoring

fix/deep-crawl-scoring-priority

fix/deprecated_pydantic

fix/dfs_deep_crawling

fix/docker

fix/docker-filter

fix/docker-jwt

fix/docker-llmEnvFile

fix/exit_with_q

fix/https-reditrect

fix/json-infinity-serialization

fix/linkPreviewScoring

fix/marketplace

fix/n-playwright-stealth

fix/playwright-stealth

fix/proxy_deprecation

fix/relative_url

fix/release-notes-demo-code

fix/request-crawl-stream

fix/serialize-proxy-config

fix/sitemap_seeder

fix/viewport_in_managed_browser

format-inline-tags

hooks

image-description

image-filterizer

implement-webhook-crawl-feature-011CULZY1Jy8N5MUkZqXkRVp

main

main-0.3.7

main-1

main-75

main-img-captionify

main-v0.2.72

merge-pr971

new-release-0.0.2

new-release-0.0.2-no-spacy

next

next-2-batch-crawl

next-JUN

next-MAY

next-alpine-docker

next-browser-farm

patch/generate_schema

pdf_processing

proxy-support

pull-84

release/v0.7.0

release/v0.7.1

release/v0.7.2

release/v0.7.3

release/v0.7.4

release/v0.7.5

release/v0.7.6

release/v0.7.7

release/v0.7.8

release/v0.8.0

run-many-deep-crawling

scraper-uc

scrapper

sponsors/thor_data

ssh-server

staging

unclecode-patch-1

unclecode-patch-2

unclecode-patch-3

unclecode-patch-4

unclecode-patch-5

unclecode-patch-6

unclecode-patch-7

unclecode-patch-8

unclecode/issue157

unclecode/issue167

v0.2.74

v0.2.76

v0.4.24

v0.4.241

v0.4.242

v0.4.243

v0.5.5

vr0.4.244

vr0.4.245

vr0.4.246

vr0.4.267

vr0.4.3b1

vr0.4.3b2

vr0.4.3b3

vr0.5.0.post1

vr0.5.0.post5

#1004

#1030

#1054

#1058

#1058

#1059

#1059

#1060

#1062

#1065

#1065

#1068

#1073

#1074

#1077

#1078

#108

#1081

#1081

#1083

#1085

#1085

#109

#1090

#1093

#1093

#1094

#1098

#1098

#1100

#1102

#1104

#1106

#1106

#1107

#1108

#1110

#1113

#1122

#1123

#1124

#1124

#1133

#1137

#1140

#1145

#1152

#1155

#1155

#1156

#1157

#1159

#1159

#1161

#1170

#1175

#1179

#1179

#1180

#1180

#1184

#1186

#119

#1192

#1193

#1195

#1200

#1200

#1207

#1207

#1208

#1209

#1210

#1211

#1211

#1212

#1214

#1220

#1220

#1223

#1223

#1225

#1225

#1232

#1234

#1234

#1238

#1238

#1239

#1245

#1249

#125

#1255

#1255

#1263

#1263

#1265

#1266

#1267

#1272

#1272

#1274

#128

#1281

#1282

#1285

#1289

#1289

#129

#1290

#1290

#1296

#13

#1303

#1304

#1305

#1307

#1308

#1308

#1313

#1319

#1334

#1334

#1336

#1337

#1339

#134

#135

#1351

#1356

#1358

#1361

#1364

#1366

#1368

#1369

#1371

#1372

#1373

#1376

#1378

#1381

#1383

#1384

#1386

#1387

#1388

#1389

#139

#1390

#1393

#1395

#1398

#1399

#14

#1402

#1408

#1413

#1416

#1416

#1417

#1417

#1420

#1422

#1425

#1425

#1426

#1432

#1433

#1435

#1436

#1440

#1441

#1444

#1447

#1448

#1450

#1451

#1454

#1463

#1464

#1465

#1467

#1469

#1470

#1471

#1478

#1482

#1483

#1483

#1486

#1488

#1488

#149

#1494

#1494

#1495

#1496

#1497

#1501

#1508

#1513

#1514

#1518

#1519

#1525

#1527

#1528

#1529

#1530

#1531

#1532

#1533

#1533

#1535

#1536

#1537

#1539

#1546

#1547

#1548

#1550

#1554

#1555

#1556

#1557

#1558

#1560

#1565

#1568

#1569

#1570

#1572

#1572

#1576

#158

#1580

#1580

#1588

#1589

#1590

#1592

#1592

#1595

#1596

#1597

#1598

#1599

#1600

#1605

#1607

#1609

#1612

#1613

#1617

#1617

#1619

#1620

#1622

#1622

#1623

#1624

#1628

#1630

#1633

#1637

#1640

#1641

#1643

#1645

#1648

#1650

#1650

#1653

#1655

#1661

#1662

#1667

#1668

#1668

#1674

#1674

#1676

#1677

#1681

#1683

#1683

#1685

#1689

#1689

#169

#1694

#1696

#1697

#1698

#1700

#1702

#1702

#1703

#1706

#1706

#1707

#1707

#1710

#1712

#1713

#1714

#1715

#1715

#1716

#1716

#1717

#1718

#1719

#172

#1720

#1721

#1722

#1722

#1723

#1724

#1729

#1729

#1730

#1730

#1733

#1734

#1734

#1744

#1746

#1752

#1755

#1756

#1756

#1759

#1759

#176

#1760

#1760

#1761

#1761

#1763

#1763

#1764

#1764

#1765

#1765

#1766

#1766

#194

#200

#215

#218

#229

#232

#234

#24

#249

#255

#269

#271

#279

#286

#288

#293

#294

#298

#299

#3

#300

#304

#312

#312

#313

#314

#324

#33

#332

#332

#335

#335

#337

#34

#357

#358

#369

#37

#379

#387

#389

#390

#394

#403

#410

#411

#416

#416

#419

#419

#427

#440

#444

#445

#458

#462

#462

#465

#472

#475

#475

#496

#510

#562

#581

#60

#605

#605

#606

#609

#612

#617

#618

#622

#64

#640

#65

#657

#658

#66

#662

#671

#671

#679

#680

#681

#681

#685

#687

#706

#708

#723

#723

#724

#729

#734

#741

#741

#749

#75

#752

#754

#775

#776

#777

#788

#792

#799

#799

#80

#800

#800

#806

#808

#821

#84

#84

#846

#85

#864

#865

#868

#891

#899

#901

#901

#903

#914

#915

#916

#918

#929

#93

#931

#945

#948

#948

#95

#961

#967

#967

#969

#970

#971

#973

#977

#983

#983

#988

#988

#990

#994

#999

#999

0.3.4

docker-rebuild-v0.7.5

docker-rebuild-v0.7.6

docker-rebuild-v0.7.7

docker-rebuild-v0.7.8

docker-rebuild-v0.8.0

v.3.72

v0.0.75

v0.1.0

v0.2.0

v0.2.1

v0.2.2

v0.2.4

v0.2.6

v0.2.7

v0.2.71

v0.2.72

v0.2.73

v0.2.74

v0.2.77

v0.3.0

v0.3.3

v0.3.6

v0.3.745

v0.3.746

v0.4.24

v0.4.243

v0.5.0.post1

v0.6.3

v0.7.0

v0.7.1

v0.7.2

v0.7.3

v0.7.4

v0.7.5

v0.7.6

v0.7.7

v0.7.8

v0.8.0

vr0.6.0

vr0.6.0rc1

vr0.6.3

Mono Color

• b2f3cb0dfa WIP: logger migriate to rich wakaka6 2025-04-10 23:02:19 +08:00
• 18e8227dfb feat(crawler): add console message capture functionality UncleCode 2025-04-10 23:26:09 +08:00
• 7c358a1aee fix(browser): add null check for crawlerRunConfig.url UncleCode 2025-04-10 23:25:07 +08:00
• 108b2a8bfb Fixed capturing console messages for case the url is the local file. Update docker configuration (work in progress) UncleCode 2025-04-10 23:22:38 +08:00
• 66ac07b4f3 feat(crawler): add network request and console message capturing unclecode 2025-04-10 16:03:48 +08:00
• a2061bf31e feat(crawler): add MHTML capture functionality UncleCode 2025-04-09 15:39:04 +08:00
• 6f7ab9c927 fix: Revert changes to session management in AsyncHttpWebcrawler and solve the underlying issue by removing the session closure in finally block of session context. Aravind Karnam 2025-04-08 18:31:00 +05:30
• 9038e9acbd Merge branch 'main' into next UncleCode 2025-04-08 17:43:42 +08:00
• 02e627e0bd fix(crawler): simplify page retrieval logic in AsyncPlaywrightCrawlerStrategy UncleCode 2025-04-08 17:43:36 +08:00
• 72d8e679ad feat(pipeline): add high-level Crawler utility class for simplified web crawling next-2-batch-crawl UncleCode 2025-04-07 22:50:44 +08:00
• 67a790b4a6 Add test file for Pipeline batch crawl. UncleCode 2025-04-06 19:38:31 +08:00
• 5b66208a7e Refactor next branch UncleCode 2025-04-06 18:33:09 +08:00
• d95b2dc9f2 Some refactoring, movie pipelin submodule folder into the main. UncleCode 2025-04-06 18:28:28 +08:00
• 591f55edc7 refactor(browser): rename methods and update type hints in BrowserHub for clarity UncleCode 2025-04-06 18:22:05 +08:00
• e1d9e2489c refactor(docs): update import statement in quickstart.py for improved clarity UncleCode 2025-04-05 23:12:06 +08:00
• b1693b1c21 Remove old quickstart files UncleCode 2025-04-05 23:10:25 +08:00
• 49d904ca0a refactor(docs): enhance quickstart_examples.py with improved configuration and file handling UncleCode 2025-04-05 22:57:45 +08:00
• ca9351252a refactor(docs): update import paths and clean up example code in quickstart_examples.py UncleCode 2025-04-05 22:55:56 +08:00
• 935d9d39f8 Add quickstart example set UncleCode 2025-04-05 21:37:25 +08:00
• f8213c32b9 Merge branch 'vr0.5.0.post8' UncleCode 2025-04-05 21:36:17 +08:00
• 14894b4d70 feat(config): set DefaultMarkdownGenerator as the default markdown generator in CrawlerRunConfig feat(logger): add color mapping for log message formatting options UncleCode 2025-04-03 20:34:19 +08:00
• 7155778eac chore: move from faust-cchardet to chardet Aravind Karnam 2025-04-03 17:42:51 +05:30
• 4133e5460d typo-fix: https://github.com/unclecode/crawl4ai/pull/918 Aravind Karnam 2025-04-03 17:42:24 +05:30
• 73fda8a6ec fix: address the PR review: https://github.com/unclecode/crawl4ai/pull/899#discussion_r2024639193 Aravind Karnam 2025-04-03 13:47:13 +05:30
• 86df20234b fix(crawler): handle exceptions in get_page call to ensure page retrieval UncleCode 2025-04-02 21:25:24 +08:00
• 179921a131 fix(crawler): update get_page call to include additional return value UncleCode 2025-04-02 19:01:30 +08:00
• 9e16a4bb26 Merge next and resolve conflicts Aravind Karnam 2025-04-02 12:18:23 +05:30
• c5cac2b459 feat(browser): add BrowserHub for centralized browser management and resource sharing UncleCode 2025-04-01 20:35:02 +08:00
• 555455d710 feat(browser): implement browser pooling and page pre-warming UncleCode 2025-03-31 21:55:07 +08:00
• 765f856ed4 Merge pull request #808 from dvschuyl/bug/parse-srcset-fix-float-width Aravind 2025-03-31 18:21:09 +05:30
• 757e3177ed fix: https://github.com/unclecode/crawl4ai/issues/839 Aravind Karnam 2025-03-31 17:10:04 +05:30
• d8357e80d2 Merge pull request #915 from maggie-edkey/css-selector Aravind 2025-03-31 13:03:35 +05:30
• ef1f0c4102 fix:https://github.com/unclecode/crawl4ai/issues/701 Aravind Karnam 2025-03-31 12:43:32 +05:30
• 1119f2f5b5 fix: https://github.com/unclecode/crawl4ai/issues/911 maggie.wang 2025-03-31 14:05:54 +08:00
• bb02398086 refactor(browser): improve browser strategy architecture and lifecycle management UncleCode 2025-03-30 20:58:39 +08:00
• 3ff7eec8f3 refactor(browser): consolidate browser strategy implementations UncleCode 2025-03-28 22:47:28 +08:00
• d8cbeff386 fix: https://github.com/unclecode/crawl4ai/issues/842 Aravind Karnam 2025-03-28 19:31:05 +05:30
• 64f20ab44a refactor(docker): update Dockerfile and browser strategy to use Chromium UncleCode 2025-03-28 15:59:02 +08:00
• 57e0423b3a fix:target_element should not affect link extraction. -> https://github.com/unclecode/crawl4ai/issues/902 Aravind Karnam 2025-03-28 12:56:37 +05:30
• c635f6b9a2 refactor(browser): reorganize browser strategies and improve Docker implementation UncleCode 2025-03-27 21:35:13 +08:00
• 7be5427283 Merge branch 'next' into 2025-MAR-ALPHA-1 Aravind Karnam 2025-03-27 12:29:32 +05:30
• 7f93e88379 refactor(tests): remove unused imports in test_docker_browser.py UncleCode 2025-03-26 15:19:29 +08:00
• 40d4dd36c9 chore(version): bump version to 0.5.0.post8 and update post-installation setup UncleCode 2025-03-25 21:56:49 +08:00
• d8f38f2298 chore(version): bump version to 0.5.0.post7 UncleCode 2025-03-25 21:47:19 +08:00
• 5c88d1310d feat(cli): add output file option and integrate LXML web scraping strategy UncleCode 2025-03-25 21:38:24 +08:00
• 4a20d7f7c2 feat(cli): add quick JSON extraction and global config management UncleCode 2025-03-25 20:30:25 +08:00
• 585e5e5973 fix: https://github.com/unclecode/crawl4ai/issues/733 Aravind Karnam 2025-03-25 15:17:59 +05:30
• e3111d0a32 fix: prevent session closing after each request to maintain connection pool. Fixes: https://github.com/unclecode/crawl4ai/issues/867 Aravind Karnam 2025-03-25 13:46:55 +05:30
• 2f0e217751 Chore: Add brotli as dependancy to fix: https://github.com/unclecode/crawl4ai/issues/867 Aravind Karnam 2025-03-25 13:44:41 +05:30
• 6405cf0a6f Merge branch 'vr0.5.0.post5' into next UncleCode 2025-03-25 14:51:29 +08:00
• 6eed4adc65 Merge branch 'vr0.5.0.post5' UncleCode 2025-03-25 12:24:07 +08:00
• bdd9db579a chore(version): bump version to 0.5.0.post6 vr0.5.0.post5 UncleCode 2025-03-25 12:01:36 +08:00
• 1107fa1d62 feat(cli): enhance markdown generation with default content filters UncleCode 2025-03-25 11:56:00 +08:00
• efa73257c5 Merge branch 'next' into 2025-MAR-ALPHA-1 Aravind Karnam 2025-03-24 21:57:29 +05:30
• 4dfd270161 fix: #855 run-many-deep-crawling UncleCode 2025-03-24 22:54:53 +08:00
• 8c08521301 feat(browser): add Docker-based browser automation strategy UncleCode 2025-03-24 21:36:58 +08:00
• 462d5765e2 fix(browser): improve storage state persistence in CDP strategy UncleCode 2025-03-23 21:06:41 +08:00
• 6eeb2e4076 feat(browser): enhance browser context creation with user data directory support and improved storage state handling UncleCode 2025-03-23 19:07:13 +08:00
• 0094cac675 refactor(browser): improve parallel crawling and browser management UncleCode 2025-03-23 18:53:24 +08:00
• 4ab0893ffb feat(browser): implement modular browser management system UncleCode 2025-03-21 22:50:00 +08:00
• e01d1e73e1 fix: link normalisation in BestFirstStrategy Aravind Karnam 2025-03-21 17:34:13 +05:30
• 471d110c5e fix: url normalisation ref: https://github.com/unclecode/crawl4ai/issues/841 Aravind Karnam 2025-03-21 16:48:07 +05:30
• f89113377a fix: Move adding of visited urls to the 'visited' set, when queueing the URLs instead of after dequeuing, this is to prevent duplicate crawls. https://github.com/unclecode/crawl4ai/issues/843 Aravind Karnam 2025-03-21 13:44:57 +05:30
• 6740e87b4d fix: remove trailing slash when the path is empty. This is causing dupicate crawls Aravind Karnam 2025-03-21 13:41:31 +05:30
• 8b761f232b fix: improve logged url readability by decoding encoded urls Aravind Karnam 2025-03-21 13:40:23 +05:30
• e0c2a7c284 chore: remove mistakenly commited deps.txt file Aravind Karnam 2025-03-21 11:06:46 +05:30
• ac2f9ae533 fix: streamline url status logging via single entrypoint i.e. logger.url_status Aravind Karnam 2025-03-20 18:59:15 +05:30
• eedda1ae5c fix: Truncate long urls in middle than end since users are confused that same url is being scraped several times. Also remove labels on status and timer to be replaced with symbols to save space and display more URL Aravind Karnam 2025-03-20 18:56:19 +05:30
• 8cecbec7a7 Merge branch 'next' into 2025-MAR-ALPHA-1 Aravind Karnam 2025-03-20 17:07:53 +05:30
• 6432ff1257 feat(browser): add builtin browser management system UncleCode 2025-03-20 12:13:59 +08:00
• 4359b12003 docs + fix: Update example for full page screenshot & PDF export. Fix the bug Error: crawl4ai.async_webcrawler.AsyncWebCrawler.aprocess_html() got multiple values for keyword argument - for screenshot param. https://github.com/unclecode/crawl4ai/issues/822#issuecomment-2732602118 Aravind Karnam 2025-03-18 17:20:24 +05:30
• 5358ac0fc2 refactor: clean up imports and improve JSON schema generation instructions UncleCode 2025-03-18 18:53:34 +08:00
• 529a79725e docs: remove hallucinations from docs for CrawlerRunConfig + Add chunking strategy docs in the table Aravind Karnam 2025-03-18 16:14:00 +05:30
• 9109ecd8fc chore: Raise an exception with clear messaging when body tag is missing in the fetched html. The message should warn users to add appropriate wait_for condition to wait until body tag is loaded into DOM. fixes: https://github.com/unclecode/crawl4ai/issues/804 Aravind Karnam 2025-03-18 15:26:20 +05:30
• 84883be513 Merge branch 'next' into 2025-MAR-ALPHA-1 Aravind Karnam 2025-03-18 15:12:21 +05:30
• 79328e4292 Create main.yml (#846) Aravind 2025-03-17 18:17:57 +05:30
• a24799918c feat(llm): add additional LLM configuration parameters UncleCode 2025-03-14 21:36:23 +08:00
• a31d7b86be feat(changelog): update CHANGELOG for version 0.5.0.post5 with new features, changes, fixes, and breaking changes UncleCode 2025-03-14 15:26:37 +08:00
• 7884a98be7 feat(crawler): add experimental parameters support and optimize browser handling UncleCode 2025-03-14 14:39:24 +08:00
• c190ba816d refactor: Instead of custom validation of question, rely on the built in FastAPI validator, so generated API docs also reflects this expectation correctly Aravind Karnam 2025-03-14 09:40:50 +05:30
• a3954dd4c6 refactor: Move the checking of protocol and prepending protocol inside api handlers Aravind Karnam 2025-03-14 09:39:10 +05:30
• 6e3c048328 feat(api): refactor crawl request handling to streamline single and multiple URL processing UncleCode 2025-03-13 22:30:38 +08:00
• b750542e6d feat(crawler): optimize single URL handling and add performance comparison UncleCode 2025-03-13 22:15:15 +08:00
• cbb8755972 Merge branch 'next' into 2025-MAR-ALPHA-1 Aravind Karnam 2025-03-13 10:42:22 +05:30
• dc36997a08 feat(schema): improve HTML preprocessing for schema generation UncleCode 2025-03-12 22:40:46 +08:00
• 1630fbdafe feat(monitor): add real-time crawler monitoring system with memory management UncleCode 2025-03-12 19:05:24 +08:00
• 341b7a5f2a 🐛 Truncate width to integer string in parse_srcset dvschuyl 2025-03-11 11:05:14 +01:00
• 3ea3c0520d Add all 5 deployments solution for testing deploy UncleCode 2025-03-10 18:57:14 +08:00
• 9547bada3a feat(content): add target_elements parameter for selective content extraction UncleCode 2025-03-10 18:54:51 +08:00
• 9d69fce834 feat(scraping): add smart table extraction and analysis capabilities UncleCode 2025-03-09 21:31:33 +08:00
• c6a605ccce feat(filters): add reverse option to URLPatternFilter UncleCode 2025-03-08 18:54:41 +08:00
• 4aeb7ef9ad refactor(proxy): consolidate proxy configuration handling UncleCode 2025-03-07 23:14:11 +08:00
• a68cbb232b feat(browser): add standalone CDP browser launch and lxml extraction strategy UncleCode 2025-03-07 20:55:56 +08:00
• e1b3bfe6fb Merge branch 'vr0.5.0.post4' UncleCode 2025-03-06 22:46:44 +08:00
• f78c46446b feat(deep-crawling): improve URL normalization and domain filtering UncleCode 2025-03-06 22:45:57 +08:00
• 1b72880007 chore(version): bump version to 0.5.0.post3 UncleCode 2025-03-06 20:32:32 +08:00
• 29f7915b79 fix(models): support float timestamps in CrawlStats UncleCode 2025-03-06 20:30:57 +08:00
• 2327db6fdc refactor(crawler): introduce CrawlResultContainer and simplify interfaces UncleCode 2025-03-05 22:23:08 +08:00
• fd02dc782d Merge branch 'main' of https://github.com/unclecode/crawl4ai UncleCode 2025-03-05 17:15:48 +08:00
• 3a234ec950 fix(auth): make JWT authentication optional with fallback UncleCode 2025-03-05 17:14:42 +08:00