wm webmirror threaded website mirroring

Threaded website mirroring.

webmirror crawls same-host content, streams discovery into download workers, rewrites HTML and CSS for local browsing, and writes a JSON run report that can be used to resume failed and partial work.

Jump to commands View report example Download 1.12 tarball
Workers default concurrency with configurable worker count
Reports JSON output for auditing, failure review, and resume input
Output modes tui2 for terminal monitoring or plain for logs
webmirror session
$ ./webmirror --threads 16 --delay 0.25s --resume ./mirror/mirror-report.json https://example.com
same-host crawl, local rewrite, resumable failures
stream discovery into downloads by default Ctrl+C keeps shutdown graceful
Discovery 0 HTML, CSS, srcset, meta refresh, sitemaps, and robots sitemap entries.
Recovery 0 Resume partial transfers, retry transient failures, and continue from prior reports.
Scope 0 Configurable worker range with crawl depth, page count, size, and queue limits.
Output 0 Interactive TUI for live monitoring or plain logs for scripts and non-TTY runs.

Terminal session

This embedded demo is the existing terminal capture from the project. It shows the actual interface style used during a run.

Animated terminal demo of webmirror crawling and downloading resources

What the demo shows

webmirror is operated from the command line. The TUI shows active workers, scan progress, queue pressure, throughput, and failure state while the run is live.

  • Terminal-first interface with either tui2 or plain output.
  • Streaming discovery keeps the worker pool busy during long scans.
  • Write failures, retry behavior, and run completion stay visible in-session.
  • The same run state is also persisted into mirror-report.json.
Download latest tar.gz artifact: webmirror-1.12.tar.gz

Core capabilities

01

Same-host crawling with bounded scope

Starts at one URL, stays on the host, and keeps crawl scope inside the starting path subtree unless you begin at /.

02

Download while you discover

Streaming mode feeds discovered resources directly to workers, so large sites start transferring immediately instead of waiting for a full scan.

03

Local browsing that actually works

Downloaded HTML and CSS are rewritten to local paths so the mirrored site can be opened and navigated offline.

04

Resume partial and failed work

Supports .part files, HTTP Range requests, and report-driven resume for previous failed or unfinished items.

05

Queue overflow without losing the crawl

Oversized scan queues can spill to disk with --overflow, keeping large runs from stalling in memory.

06

Operational controls, not just flags

Rate limiting, delays, proxy support, robots handling, file-size caps, verbose logging, and path naming controls are all first-class options.

Run model

1

Start from one URL

Parse config, pick output mode, set worker counts, and initialize HTTP behavior such as proxying, retries, and delay strategy.

2

Extract and filter

Discover resources from HTML, CSS, srcset, meta refresh, and sitemap sources while respecting include or exclude rules.

3

Download concurrently

Workers transfer files, resume partial content when possible, and keep progress visible in either the TUI or plain logs.

4

Write a resumable report

At the end, get counts, failed items, partial items, command metadata, and enough detail to resume the next run cleanly.

Important flags

  • --proxy accepts http://, https://, socks5://, and socks5h:// proxy URLs.
  • --proxy also accepts multiple proxies as a comma-separated list.
  • --resume retries failed and partial URLs from a previous report.
  • --overflow spills oversized scan queues to disk.
  • --delay and --rate-limit control request pacing.
  • --view selects auto, tui2, or plain.
  • --wget-compatible switches path naming and enables listing-date matching heuristics.

Operational notes

  • --delay accepts fixed values like 2s, bare second values like 1.5, and randomized ranges like 0.5-1.5s.
  • --retries is a legacy shortcut that sets both --http-retries and --transfer-retries.
  • Query strings are part of local path mapping unless report redaction removes them from display output.
  • Root URLs mirror to index.html; extensionless paths mirror to .../index.html.
  • --warn and --exit are mutually exclusive.
  • Local paths are generated under the chosen output directory.
  • The report is updated during the run and written to <output>/mirror-report.json.

Command examples

Select a common usage pattern to inspect the command shape. The examples below reflect the flags currently documented in the repository README.

Basic mirror


        

        

      

    


    

      

        
Report structure

        

          Each run writes <output>/mirror-report.json with counts, failures, partial items, active options, and resume metadata.
        

        

          
{
  "command_args": [
    "--delay",
    "0.5-1.5s",
    "--resume",
    "./mirror/mirror-report.json",
    "https://example.com"
  ],
  "downloaded": 42,
  "failed": 1,
  "failed_items": [
    {
      "url": "https://example.com/app.js",
      "resume_url": "https://example.com/app.js?cache=bypass",
      "error": "503 Service Unavailable",
      "referenced_from": [
        "https://example.com/"
      ]
    }
  ],
  "partial_items": [
    "https://example.com/assets/site.css"
  ]
}

        

      


      

        
Report contents

        
    
          
  • Aggregate counts for discovered, downloaded, skipped, excluded, and failed items.
    • 
          
  • Failure entries with URL, error text, and referrers when available.
    • 
          
  • resume_url when a redacted display URL still needs original query data for resume.
    • 
          
  • Partial items and command metadata from the original invocation.
    • 
          
  • Queue statistics when deferred scan entries or spill files are involved.
    • 
        

      

    


    

      

        Option reference
        

          

            Full command line option list based on the current flag definitions in the tool.
          

          

            
              

                -o, --output
                Output directory. Default: ./mirror.
              

              

                -t, --threads
                Number of download threads. Default: 8.
              

              

                --max-file-size
                Maximum bytes per file download. 0 means unlimited.
              

              

                --max-resources
                Maximum resources to discover. 0 means unlimited.
              

              

                --rate-limit
                Maximum HTTP requests per second. 0 means unlimited.
              

              

                --proxy
                Accepts http://, https://, socks5://, and socks5h:// URLs. Multiple proxies can be provided as a comma-separated list.
              

              

                --delay
                Supports 0, fixed values such as 2s, bare numeric seconds such as 1.5, and ranges such as 0.5-1.5s.
              

              

                --overflow
                Optional overflow queue file path. It can be used with an explicit file path or without one to auto-generate a secure runtime file.
              

              

                --exclude
                Exclude wildcard patterns. Repeatable or comma-separated.
              

              

                --exclude-file
                YAML file with scoped URL regex to path wildcard exclusions.
              

              

                --exclusive
                Mirror only wildcard patterns. Repeatable or comma-separated.
              

              

                --exclusive-file
                YAML file with scoped URL regex to path wildcard inclusions.
              

              

                --config
                Load options from a file, or use $HOME/.mirrorrc when provided without a value.
              

              

                --resume
                Path to mirror-report.json to seed failed and partial URL retries.
              

              

                --discover-backup
                Attempt backup-file variants from discovered URLs.
              

              

                --partial
                Resume partial downloads using HTTP Range requests and .part files.
              

              

                --no-stream
                Disable the streaming discovery-to-download pipeline.
              

              

                --wget-compatible
                Use wget-like path naming behavior and metadata heuristics.
              

              

                --match-listing-dates
                Set local mtime from directory listing timestamps when available.
              

              

                --redact-report-query
                Strip query strings from URLs stored in mirror-report.json.
              

              

                --view
                Output view mode: auto, tui2, or plain.
              

              

                --obey-robots
                Obey robots.txt rules.
              

              

                --ua
                Alias for --user-agent.
              

              

                --user-agent
                HTTP user agent string.
              

              

                --timeout
                Request timeout. Default: 20s.
              

              

                --http-retries
                Retry count for transient HTTP failures.
              

              

                --transfer-retries
                Retry count for mid-transfer or download failures.
              

              

                --retries
                Legacy shortcut that sets both --http-retries and --transfer-retries.
              

              

                --max-pages
                Maximum pages to crawl. 0 means unlimited.
              

              

                --max-depth
                Maximum crawl depth. 0 means unlimited.
              

              

                --max-referrers
                Maximum stored referrers per failed resource in reports. 0 means unlimited.
              

              

                --scan-queue-limit
                Maximum pending crawl queue entries. 0 uses the automatic default.
              

              

                --insecure
                Alias for --insecure-skip-tls-verify.
              

              

                --insecure-skip-tls-verify
                Skip TLS certificate verification.
              

              

                --verbose
                Enable verbose logging.
              

              

                --warn
                On write failures, print a warning and continue.
              

              

                --exit
                On write failures, fail fast with exit code 1.
              

            
          

        

      

    


    

      

        .mirrorrc example
        

          

            Example CLI-style config file based on examples/mirrorrc. The file is tokenized like shell arguments, and quoted values are supported.
          

          

            
# webmirror config file (CLI-style flags)
# Lines are tokenized like shell args; quotes are supported.

--output ./mirror
--overflow
--threads 12
--timeout 30s
--proxy "socks5h://127.0.0.1:9050,http://127.0.0.1:8080"
--http-retries 8
--transfer-retries 12
--rate-limit 20
--scan-queue-limit 50000
--view plain
# write-failure policy (choose one): default prompt, --warn, or --exit
# --warn
--user-agent "mirror/ops profile"
--obey-robots
--discover-backup
--exclude "**/*.tmp,**/*.bak"
--exclude-file ./examples/exclude.yaml
--exclusive-file ./examples/exclusive.yaml

          

        

      

    


    

      

        exclude.yaml example
        

          

            Example scoped exclusion rules based on examples/exclude.yaml.
          

          

            
rules:
  - url: https?://(www\.)?example\.com/admin/.*
    patterns:
      - "**"
  - url: https?://(www\.)?example\.com/files/.*
    patterns:
      - "*.log"
      - "*.tmp"
      - "**/incoming/*"
  - url: https?://(www\.)?example\.com/.*
    patterns:
      - "*.jpg"
      - "*.png"

          

        

      

    


    

      

        exclusive.yaml example
        

          

            Example scoped inclusion rules based on examples/exclusive.yaml.
          

          

            
rules:
  - url: https?://(www\.)?example\.com/downloads/.*
    patterns:
      - "*.pdf"
      - "*.txt"
  - url: https?://(www\.)?example\.com/filedump/.*
    patterns:
      - "**/incoming/*"
      - "*.tar.gz"
  - url: https?://(www\.)?example\.com/.*
    patterns:
      - "docs/**"

          

        

      

    


    

      

        
Downloads

        

          Download the source tarball or one of the currently built binaries present in this workspace.
        

        

          

            
Source tarball

            
Packaged source snapshot for the current version.

            webmirror-1.12.tar.gz
            Download
          

          

            
Linux aarch64

            
Native ARM64 Linux binary

            webmirror-linux-aarch64
            Download
          

          

            
Linux amd64

            
Cross-compiled Linux binary.

            webmirror-linux-amd64
            Download
          

          

            
Windows amd64

            
Cross-compiled Windows binary.

            webmirror-windows-amd64.exe
            Download
          

          

            
macOS Darwin amd64

            
Natively compiled macOS Darwin binary.

            webmirror-darwin-amd64
            Download
          

          

            
macOS Darwin ARM

            
Natively compiled macOS Darwin ARM binary.

            webmirror-darwin-arm
            Download