Introduction §
This appendix provides a curated catalog of tools, software, services, and resources essential for Archaeobytological practice. Tools are organized by function and annotated with:
- Purpose: What the tool does
- Skill Level: Beginner, Intermediate, Advanced
- Cost: Free, Freemium, Paid
- Platform: Windows, macOS, Linux, Web-based
- Open Source: Yes/No
Tools are current as of 2025 but the digital preservation landscape evolves rapidly. Check the Archaeobytology community wiki (archaeobytology.org/wiki) for updates.
I. Web Archiving & Scraping Tools §
1. Wget §
Purpose: Command-line tool for downloading websites recursively
Skill Level: Beginner-Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.gnu.org/software/wget/
What It Does: Downloads web pages and their linked resources (images, CSS, JavaScript). Creates mirror copies of websites on your local machine.
Basic Usage:
wget --recursive --level=2 --no-parent --wait=1 https://example.com
Best For: Static HTML sites, simple scraping projects
Limitations: Doesn’t handle JavaScript-heavy sites well, can’t navigate login walls
2. HTTrack §
Purpose: Website copier with GUI interface
Skill Level: Beginner
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.httrack.com/
What It Does: Similar to wget but with graphical interface. Easier for beginners who don’t want command-line tools.
Best For: One-time website archiving, beginners
Limitations: Slower than command-line tools, less flexible configuration
3. ArchiveBox §
Purpose: Self-hosted web archiving platform
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux, Docker
Open Source: Yes
Website: https://archivebox.io/
What It Does: Creates permanent archives of web pages including HTML, screenshots, PDFs, videos, and git repositories. Provides web interface for browsing archives.
Features:
- Multiple capture methods (wget, Chrome headless, youtube-dl, etc.)
- Scheduled archiving (cron jobs)
- Full-text search
- Deduplication
Best For: Personal archiving projects, research collections, small organizations
Setup Complexity: Requires server or Docker knowledge
4. Webrecorder (now Conifer) §
Purpose: Browser-based interactive web archiving
Skill Level: Beginner
Cost: Free
Platform: Web (browser extension also available)
Open Source: Yes
Website: https://conifer.rhizome.org/
What It Does: Records your browsing session including JavaScript interactions, videos, and dynamic content. Creates WARC (Web ARChive) files you can replay.
Features:
- Captures JavaScript-heavy sites
- Records social media feeds (Twitter, Instagram)
- Exports to standard WARC format
- Replay archives offline
Best For: Social media archiving, dynamic websites, personal projects
Unique Advantage: Works in browser, no installation required
5. Heritrix §
Purpose: Industrial-strength web crawler
Skill Level: Advanced
Cost: Free
Platform: Java (cross-platform)
Open Source: Yes
Website: https://github.com/internetarchive/heritrix3
What It Does: Internet Archive’s production crawler. Designed for massive-scale archiving (billions of URLs).
Features:
- Highly configurable crawl policies
- Distributed crawling
- Respects robots.txt
- Creates WARC files
Best For: Large institutions, comprehensive web archiving
Limitations: Steep learning curve, requires significant infrastructure
6. Browsertrix Crawler §
Purpose: High-fidelity browser-based crawling
Skill Level: Intermediate-Advanced
Cost: Free
Platform: Docker
Open Source: Yes
Website: https://github.com/webrecorder/browsertrix-crawler
What It Does: Uses real browsers (Chrome) to capture JavaScript-heavy sites with perfect fidelity. Creates WARC files.
Best For: Modern web apps, single-page applications, sites requiring JavaScript
7. Archive-It §
Purpose: Subscription web archiving service
Skill Level: Beginner
Cost: Paid (subscription based on storage)
Platform: Web-based
Open Source: No
Website: https://archive-it.org/
What It Does: Managed web archiving service by Internet Archive. Point-and-click interface for creating and managing web archives.
Features:
- Scheduled recurring crawls
- Metadata management
- Public or private collections
- Integration with Wayback Machine
Best For: Institutions without technical staff, organizations needing reliable managed service
Cost: Starts ~$1,500/year for small collections
8. Wayback Machine Downloader §
Purpose: Retrieve websites from the Internet Archive
Skill Level: Intermediate
Cost: Free
Platform: Ruby (cross-platform)
Open Source: Yes
Website: https://github.com/hartator/wayback-machine-downloader
What It Does: Downloads entire websites from the Wayback Machine to your local computer.
Basic Usage:
wayback_machine_downloader http://example.com
Best For: Resurrecting dead sites that were not archived locally but exist in IA.
II. Media Preservation Tools §
9. yt-dlp §
Purpose: Video downloader for YouTube and 1000+ sites
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://github.com/yt-dlp/yt-dlp
What It Does: Downloads videos from streaming platforms including metadata, subtitles, thumbnails. The active fork of youtube-dl.
Basic Usage:
yt-dlp --write-description --write-info-json --write-thumbnail https://youtube.com/watch?v=VIDEO_ID
Best For: Video archiving, preserving YouTube/Vimeo/TikTok content
10. gallery-dl §
Purpose: Image gallery downloader
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://github.com/mikf/gallery-dl
What It Does: Downloads images from image hosting sites (Imgur, Flickr, DeviantArt, Twitter, etc.).
Best For: Image archiving, art preservation, meme collections
11. FFmpeg §
Purpose: Multimedia conversion and processing
Skill Level: Intermediate-Advanced
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://ffmpeg.org/
What It Does: The Swiss Army knife of audio/video. Converts formats, extracts frames, creates thumbnails, transcodes for preservation.
Best For: Format migration, creating preservation masters, generating access copies
Example:
ffmpeg -i input.flv -c:v libx264 -c:a aac output.mp4
12. ShareX / CleanShot X §
Purpose: Advanced screen capture
Skill Level: Beginner
Cost: ShareX (Free), CleanShot (Paid)
Platform: Windows (ShareX), macOS (CleanShot)
Open Source: ShareX (Yes)
Website: https://getsharex.com/
What It Does: Captures screenshots, GIFs, and scrolling windows. Essential for documenting “ephemeral” interfaces that cannot be scraped (e.g., Snapchats, dying apps).
Best For: Documentation of UI/UX, capturing unscrapable content.
III. Emulation & Obsolescence Tools §
13. Flashpoint Archive §
Purpose: Flash game and animation preservation
Skill Level: Beginner
Cost: Free
Platform: Windows, Linux
Open Source: Partially
Website: https://bluemaxima.org/flashpoint/
What It Does: Preserves and plays 150,000+ Flash games and animations using embedded emulators.
Best For: Playing preserved Flash content, research, nostalgia
14. Ruffle §
Purpose: Flash Player emulator in Rust
Skill Level: Beginner
Cost: Free
Platform: Web (browser extension), Desktop
Open Source: Yes
Website: https://ruffle.rs/
What It Does: Open-source Flash Player replacement that runs in browsers. Critical for “resurrecting” Petribytes without the original proprietary plugin.
Best For: Viewing archived Flash content, embedding Flash in modern websites
15. MAME (Multiple Arcade Machine Emulator) §
Purpose: Arcade game preservation
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.mamedev.org/
What It Does: Emulates arcade hardware to preserve vintage arcade games.
Best For: Arcade game preservation, historical research
16. DOSBox §
Purpose: DOS emulator
Skill Level: Beginner
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.dosbox.com/
What It Does: Emulates MS-DOS environment for running old DOS games and software.
Best For: 1980s-1990s software preservation
17. RetroArch §
Purpose: Frontend for emulators
Skill Level: Intermediate
Cost: Free
Platform: Cross-platform
Open Source: Yes
Website: https://www.retroarch.com/
What It Does: A unified interface for running emulators (cores) for dozens of consoles (NES, SNES, PlayStation, etc.).
Best For: Console history preservation, gaming.
IV. Forensics & Data Recovery §
18. The Sleuth Kit (TSK) / Autopsy §
Purpose: Digital forensics and file recovery
Skill Level: Advanced
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.sleuthkit.org/
What It Does: Analyzes disk images, recovers deleted files, examines file systems. Autopsy is the GUI; TSK is the command line.
Best For: Forensic analysis of hard drives, recovering deleted content, “Digging in the Dirt” (Chapter 8).
19. FTK Imager §
Purpose: Disk imaging tool
Skill Level: Intermediate
Cost: Free
Platform: Windows
Open Source: No
Website: https://www.exterro.com/ftk-imager
What It Does: Creates forensic disk images (bit-by-bit copies) without altering the original evidence.
Best For: Creating preservation masters of physical media.
20. PhotoRec §
Purpose: File recovery tool
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.cgsecurity.org/wiki/PhotoRec
What It Does: Recovers deleted files from hard drives, memory cards, etc., by recognizing file signatures (magic numbers).
Best For: Recovering accidentally deleted content, salvaging corrupted media.
21. Hex Editors (HxD, 0xED) §
Purpose: Raw binary editing
Skill Level: Advanced
Cost: Free
Platform: Windows (HxD), macOS (0xED)
Open Source: Varies
Website: https://mh-nexus.de/en/hxd/
What It Does: View and edit the raw binary data of a file. Essential for identifying “Magic Numbers” when file extensions are missing.
Best For: Forensic analysis, fixing corrupted headers.
V. Metadata & Organization §
22. ExifTool §
Purpose: Metadata reading/writing
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://exiftool.org/
What It Does: The industry standard for reading, writing, and editing meta-information in a wide variety of files.
Best For: Extracting metadata, adding preservation info (provenance) to files.
23. DROID (Digital Record Object Identification) §
Purpose: File format identification
Skill Level: Intermediate
Cost: Free
Platform: Windows, macOS, Linux (Java)
Open Source: Yes
Website: https://digital-preservation.github.io/droid/
What It Does: Identifies file formats and versions using the PRONOM registry.
Best For: Surveying collections, format migration planning, identifying “Xenobytes.”
24. Zotero §
Purpose: Reference management
Skill Level: Beginner
Cost: Free
Platform: Windows, macOS, Linux
Open Source: Yes
Website: https://www.zotero.org/
What It Does: Organize research sources, generate citations, save snapshots of web pages.
Best For: Academic research, maintaining a personal bibliography.
25. Obsidian §
Purpose: Knowledge base
Skill Level: Beginner-Intermediate
Cost: Free for personal use
Platform: Cross-platform
Open Source: No (Modules are open)
Website: https://obsidian.md/
What It Does: A knowledge base that works on top of a local folder of plain text Markdown files.
Best For: Maintaining “sovereign” personal knowledge graphs, documentation.
VI. Sovereignty & Infrastructure (The Anvil) §
26. Ghost §
Purpose: Sovereign publishing platform
Skill Level: Beginner (hosted) to Intermediate (self-hosted)
Cost: Freemium (Ghost Pro) or Free (self-hosted)
Platform: Web-based (Node.js)
Open Source: Yes
Website: https://ghost.org/
What It Does: Open-source platform for blogging, newsletters, and memberships.
Best For: “Ground” ownership, “POSSE” publishing (Chapter 17).
27. Mastodon §
Purpose: Federated social networking server
Skill Level: Advanced (self-hosting), Beginner (using)
Cost: Free (software)
Platform: Web-based (Ruby)
Open Source: Yes
Website: https://joinmastodon.org/
What It Does: Essential for the “Connection” pillar and federated infrastructure.
Best For: Social networking without corporate control.
28. IPFS (InterPlanetary File System) §
Purpose: Distributed file storage protocol
Skill Level: Advanced
Cost: Free
Platform: Cross-platform
Open Source: Yes
Website: https://ipfs.tech/
What It Does: Content-addressed, peer-to-peer file system. Files stored across network, retrieved by hash.
Best For: “Seed Bank” model, censorship-resistant storage.
29. Tor Browser / Onion Services §
Purpose: Anonymity and censorship resistance
Skill Level: Beginner
Cost: Free
Platform: Cross-platform
Open Source: Yes
Website: https://www.torproject.org/
What It Does: Routes traffic through a distributed network to conceal location and usage.
Best For: Privacy, bypassing censorship, accessing “Dark Archives.”
30. Signal §
Purpose: Encrypted communication
Skill Level: Beginner
Cost: Free
Platform: Mobile, Desktop
Open Source: Yes
Website: https://signal.org/
What It Does: End-to-end encrypted messaging.
Best For: “War Room” coordination, secure team communication.
VII. Storage & Backup §
31. Nextcloud §
Purpose: Self-hosted cloud storage
Skill Level: Intermediate
Cost: Free (self-hosted)
Platform: Web-based (PHP)
Open Source: Yes
Website: https://nextcloud.com/
What It Does: Personal cloud storage like Dropbox but self-hosted. Sync files across devices.
Best For: “Ground” ownership, data sovereignty.
32. Syncthing §
Purpose: Peer-to-peer file synchronization
Skill Level: Beginner-Intermediate
Cost: Free
Platform: Cross-platform
Open Source: Yes
Website: https://syncthing.net/
What It Does: Syncs files between devices without central server.
Best For: Personal backups, avoiding “The Cloud.”
33. Restic §
Purpose: Encrypted backup program
Skill Level: Intermediate
Cost: Free
Platform: Cross-platform
Open Source: Yes
Website: https://restic.net/
What It Does: Fast, encrypted, deduplicated backups to local or cloud storage.
Best For: Secure “3-2-1” backups.
VII. Reference Resources §
34. Archive Team Wiki §
Website: https://wiki.archiveteam.org/ Purpose: Documentation of rescue projects (“War Rooms”), specific platform scripts, and warrior appliances.
35. PRONOM §
Website: https://www.nationalarchives.gov.uk/PRONOM Purpose: The technical registry of file formats. Used by DROID to identify files.
36. COPTR (Community Owned digital Preservation Tool Registry) §
Website: https://coptr.digipres.org/ Purpose: A wiki listing hundreds of digital preservation tools.
End of Appendix B