How do I ignore certain URLs when using deadlink?

Add hosts to the 'ignore_hosts' list in the deadlink.toml config file, e.g., ['www.example.com'], to skip checking URLs from those domains. This is useful for excluding internal or temporary sites.

Deadlink vs lychee: which should I use for link checking?

Deadlink is a Python tool focused on configurability and caching for text files, ideal for Python-centric workflows. Lychee is a Rust-based alternative with potentially faster performance and broader file support; choose Deadlink for its TOML configurability, lychee for speed and cross-language compatibility.

Can deadlink check links in binary files like PDFs?

No, deadlink only parses text files with allowed extensions (default includes txt, yaml, c). It does not support binary formats, so you'll need pre-processing to extract text first.

How to set up deadlink in a CI/CD pipeline?

Install deadlink via pip in your pipeline script, then run 'deadlink check' on target directories. Configure timeouts and ignore hosts in deadlink.toml to handle network issues and exclude irrelevant domains.

Does deadlink handle HTTP redirects properly?

Yes, deadlink follows HTTP redirects and reports the final status code. You can adjust the timeout_s in config to manage slow or infinite redirects during checks.

What file extensions does deadlink support by default?

Deadlink checks common text-based extensions like .txt, .md, and .yaml, but you can customize this via the 'allow_file_extensions' setting in the config file to include or exclude specific types.

How to cache results with deadlink to speed up repeated runs?

Deadlink automatically caches URL results based on the cache_ttl_s setting in the config file. Set a TTL (e.g., 259200 seconds for 3 days) to balance freshness and performance, avoiding redundant network requests.

Open-Awesome

deadlink

A Python tool that parses text files for HTTP URLs and checks if they are still valid.

GitHub

176 stars4 forks0 contributors

What is deadlink?

Deadlink is a Python-based command-line tool that scans text files to find HTTP URLs and verifies their validity. It helps developers and technical writers maintain working links in documentation and code by automatically detecting broken URLs before they cause problems.

Target Audience

Developers, technical writers, and maintainers who need to ensure URLs in their codebases, README files, and documentation remain functional over time.

Value Proposition

Deadlink offers a simple, configurable solution for URL validation with concurrent checking and caching, making it faster than manual verification and more focused than general web crawlers.

Overview

:skull: Checks and fixes URLs in code and documentation.

Use Cases

Best For

Checking URLs in Markdown documentation before releases
Validating links in code comments and configuration files
Automating link checks in CI/CD pipelines
Maintaining large documentation sets with many external references
Finding broken links in legacy codebases
Preventing 404 errors in published technical content

Not Ideal For

Projects requiring validation of non-HTTP URLs like mailto: or file:// links
Websites with heavy JavaScript rendering where links load dynamically
Large-scale web crawling tasks needing complex navigation or dynamic content handling
Environments where Python is not installed or preferred for tooling

Pros & Cons

Pros

Concurrent URL Checking

Supports configurable concurrent requests (e.g., num_concurrent_requests = 5) for faster scanning, as shown in the config example, reducing overall check time for multiple URLs.

Flexible TOML Configuration

Allows customization of timeout, ignored hosts, and file extensions via a TOML config file, enabling adaptation to different project needs without code changes.

Built-in Caching System

Includes a cache with TTL (e.g., cache_ttl_s = 259200) to avoid redundant checks on the same URLs, improving efficiency and reducing network load.

Simple CLI Interface

Offers an easy command-line interface with commands like 'deadlink check', making it straightforward to integrate into scripts or manual workflows.

Cons

Limited URL Protocol Support

Only validates HTTP/HTTPS links, ignoring other common protocols like mailto: or ftp:, which can be a gap for comprehensive link checking in mixed-content projects.

No JavaScript Rendering

Cannot parse or check links that are generated or modified by JavaScript, making it ineffective for modern web applications or dynamic documentation sites.

Python Dependency Required

Relies on Python and pip for installation and use, which may not be ideal in environments without Python or where minimal dependencies are preferred.

Frequently Asked Questions

Related Projects

aria2

aria2 is a lightweight multi-protocol & multi-source, cross platform download utility operated in command-line. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink.

Stars41,512

Forks3,876

Last commit28 days ago

alex

Catch insensitive, inconsiderate writing

Stars5,095

Forks214

Last commit1 year ago

kill-tabs

Kill all Chrome tabs to improve performance, decrease battery usage, and save memory

Stars393

Forks21

Last commit5 months ago

crawley

The unix-way web crawler

Stars340

Forks18

Last commit6 days ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub