A powerful, intelligent CLI tool to crawl dynamic and static websites with full JavaScript rendering support and convert them to clean, well-formatted Markdown files. Perfect for archiving documentation, creating offline knowledge bases, and preserving web content.
- π Dynamic Site Support: Full JavaScript rendering via Playwright (Vue/React/Angular/Next.js)
- π― Smart Content Extraction: Automatically identifies and extracts core content, removing navigation, ads, and sidebars
- π Recursive Crawling: Intelligently crawls subpages with configurable depth and count limits
- οΏ½οΈ Media Downloads: Optional image and video downloading with lazy-loading support
- π Base URL Intelligence: Uses browser's
document.baseURIfor accurate relative path resolution - π Local Link Conversion: Automatically converts HTML links to local Markdown relative paths
- π§Ή Clean Output: Preserves tables, code blocks, images, links, and heading hierarchies
- π SSL Flexibility: Handles sites with certificate issues gracefully
- π Cross-Platform: Works on Windows, macOS, and Linux (Python 3.8+)
- π Universal Compatibility: Generated Markdown works with Typora, Obsidian, VS Code, and more
pip3 install web2mdgit clone https://github.com/floatinghotpot/web2md.git
cd web2md
python3 -m pip install -e .# Install Chromium driver (required for JavaScript rendering)
python3 -m playwright install chromium
# Linux only: Install system dependencies
python3 -m playwright install-deps chromium# Crawl a single page (auto-generated save directory)
web2md https://docs.python.org/3/tutorial/
# Specify custom save directory
web2md https://docs.python.org/3/tutorial/ ./python-docs
# Crawl with images
web2md https://example.com/docs --picture
# Limit crawl depth and count
web2md https://example.com/docs --depth 2 --count 10
# Crawl with images and videos
web2md https://example.com/docs --picture --video --depth 3web2md -hweb2md [URL] [SAVE_DIR] [OPTIONS]
| Argument | Required | Description |
|---|---|---|
web_url |
β Yes | Target webpage URL (must start with http/https) |
save_folder |
β No | Local save directory (auto-generated from URL if omitted) |
| Option | Default | Description |
|---|---|---|
--depth N |
5 |
Maximum relative crawl depth from base URL |
--count N |
999 |
Maximum number of pages to crawl (0 = unlimited) |
--picture |
False |
Download and save images to local images/ directory |
--video |
False |
Download and save videos to local videos/ directory |
-h, --help |
- | Show help message and exit |
web2md https://company.com/docs/home company-docs --depth 2- Crawls all pages within 2 levels of
/docs/ - Saves to
./company-docs/
web2md https://company.com/docs/home company-docs --depth 2 --count 5- Stops after crawling 5 pages
- Useful for testing or sampling large sites
web2md https://company.com/docs/home --picture --count 3- Downloads images to
images/subdirectory - Converts image URLs to local relative paths in Markdown
web2md https://company.com/docs/home --depth 1 --count 10- Auto-creates directory:
company_com_docs/
The tool automatically determines a base URL from your target URL:
- Target:
https://company.com/docs/homeβ Base:https://company.com/docs/ - All crawling is scoped to pages under this base URL
Uses the browser's document.baseURI to correctly resolve relative URLs:
- Handles
<base>tags in HTML - Respects redirects and trailing slashes
- Resolves lazy-loaded images with
data-src,srcset, etc.
Automatically identifies core content using priority selectors:
<main>tag.article-contentor.article_content#main-content.content<article>tag- Fallback to
<body>(with cleanup)
When --picture or --video is enabled:
- Downloads media files to
images/orvideos/subdirectories - Generates unique filenames with MD5 hash to prevent duplicates
- Converts URLs to local relative paths in Markdown
- Supports lazy-loading attributes:
data-src,data-original,srcset
MD filenames are generated from URLs:
- Remove base URL prefix
- Replace
/with_ - Filter illegal characters
- Example:
https://company.com/docs/api/authβapi_auth.md
PLAYWRIGHT_CONFIG = {
"headless": False, # Set to True for background crawling
"timeout": 60000, # Page load timeout (ms)
"wait_for_load": "networkidle", # Wait strategy
"sleep_after_load": 2, # Additional wait time (seconds)
"user_agent": "Mozilla/5.0..." # Custom user agent
}MEDIA_CONFIG = {
"timeout": 30000, # Media download timeout (ms)
"image_dir": "images", # Image save subdirectory
"video_dir": "videos", # Video save subdirectory
"allowed_img_ext": [".jpg", ".jpeg", ".png", ".gif", ".bmp", ".svg", ".webp"],
"allowed_vid_ext": [".mp4", ".avi", ".mov", ".webm", ".flv", ".mkv"]
}REMOVE_TAGS = ["nav", "header", "footer", "aside", "script", "style", "iframe", "sidebar"]
CORE_CONTENT_SELECTORS = [
("main", {}),
("div", {"class_": "article-content"}),
("article", {})
]DEFAULT_CRAWL_CONFIG = {
"max_depth": 5, # Default max depth
"max_count": 999, # Default max pages
"allowed_schemes": ["http", "https"],
"exclude_patterns": [r"\.pdf$", r"\.zip$", r"\.exe$"]
}Edit web2md/cli.py and set:
PLAYWRIGHT_CONFIG = {
"headless": False, # Shows browser window
...
}Add site-specific selectors to CORE_CONTENT_SELECTORS:
CORE_CONTENT_SELECTORS = [
("main", {}),
("div", {"class_": "documentation-content"}), # Custom selector
("article", {})
]Install and use playwright-stealth:
pip3 install playwright-stealthAdd to get_dynamic_html() in web2md/cli.py:
from playwright_stealth import stealth_sync
page = context.new_page()
stealth_sync(page) # Add this line
page.goto(url, ...)Add login logic in get_dynamic_html() before page.goto():
page.goto("https://example.com/login")
page.fill("#username", "your-username")
page.fill("#password", "your-password")
page.click("#login-button")
time.sleep(2)The tool automatically disables SSL verification for downloads. If you encounter issues, check your network/firewall settings.
Increase timeout in PLAYWRIGHT_CONFIG:
"timeout": 120000, # 2 minutes- Check if content is in
<main>or common content tags - Add custom selectors to
CORE_CONTENT_SELECTORS - Run with
headless: Falseto debug visually
- Verify image URLs are accessible
- Check if images require authentication
- Some CDNs may block automated downloads
Automatically installed via pip:
- playwright - Browser automation and JS rendering
- beautifulsoup4 - HTML parsing and manipulation
- lxml - Fast XML/HTML parser
- markdownify - HTML to Markdown conversion
- urllib3 - HTTP client utilities
Contributions are welcome! Please follow these steps:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run tests (if available)
- Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
git clone https://github.com/floatinghotpot/web2md.git
cd web2md
python3 -m pip install -e .
python3 -m playwright install chromiumThis project is licensed under the MIT License - see the LICENSE file for details.
- Playwright for powerful browser automation
- BeautifulSoup for HTML parsing
- markdownify for clean Markdown conversion
Made with β€οΈ for developers, researchers, and documentation enthusiasts.
If you find this tool useful, please consider giving it a β on GitHub!