Here's a description you can use for the GitHub repository:
This project is a Python-based recursive file downloader that allows you to download all files from a given URL, including subdirectories. The downloader supports various features such as:
- Resumable Downloads: Automatically resumes interrupted downloads using HTTP Range requests.
- Speed Limiting: Allows optional download speed limiting.
- Pause and Resume: Supports manual pause and resume during the download process.
- Retry Mechanism: Implements exponential backoff for retrying failed downloads.
- Progress Monitoring: Uses
tqdmto display download progress in a user-friendly manner.
- Provide the base URL as a command-line argument.
- Downloads files into
downloaded_files/followed by the rest of the URL path structure. - Handles edge cases such as incomplete downloads, broken connections, and retry logic for failed attempts.
To run the downloader, use the command:
python main.py <base_url>
Example:
python main.py https://example.com/files/
This script is helpful for downloading a large number of files recursively from a web server while providing robust error handling and user control.