My personal toolbox

Shell, Python, and Perl scripts I wrote/use on a daily basis; i.e. eating my own dog food.

Quick overview by category

The summary/index below was generated by local AI (DSv4 Flash); keep in mind that many utilities below are probably WAY out of date... Use your own judgement about them - that is, check whether they fit your own needs before using them.

Mostly, this repo establishes a common baseline in any machine I work in; a baseline that I've muscle-memoried over the years.

Even if none of the utilities below are useful to you, the approach itself - i.e. building a publicly accessible arsenal you can 'git clone' at a moment's notice... that is definitely recommended :-)

Sandboxing & container isolation

Script	What it does
isolate.sh	Firejail-based sandbox: network allowlisting, read-only $HOME, per-path rw/hide. The core isolation engine
vimisolated.sh	Launch vim inside isolate.sh, auto-detecting writable paths
pi.isolated.sh	Run pi (AI coding agent) inside isolate.sh, tunneling through a Unix socket to reach a local LLM. Symlinked as pi.sh
desert_island_container_execution.sh	Run untrusted TCP server code in a container with all outbound traffic dropped; the host reaches it on 127.0.0.1:<port>
dockerme.sh	Docker container with $PWD mapped to /workdir and X11/PulseAudio/DRI mounted; -n enables network, -r runs as root, -p exposes a port
parse-isolation-options-common.sh	Shared argument parser for isolate.sh / pi.isolated.sh (sourced, not standalone)

Example usage of isolate.sh to isolate vim (note that this is just a generic example of isolation, for isolating vim in particular, vimisolated.sh is much better):

# Read-only $HOME, no network
isolate.sh -- vim ~/bin/pi.sh

# Allow specific servers, make some paths writable, hide SSH keys from any vim-spawned subprocesses (Language servers, etc)
isolate.sh --rw ~/bin --rw ~/.viminfo --hide ~/.ssh \
           --servers servers.txt --dns 1.1.1.1 -- vim ~/bin/pi.sh

# ...so I built vimisolated.sh - auto-mapping of r/w folders, etc
vimisolated.sh ~/bin/pi.sh

Example usage of isolated pi.dev harness:

# Assumes a local LLM at localhost:8081
pi.isolated.sh [--port PORT] [isolate.sh options] [-- pi options]

File indexing & syncing

Script	What it does
indexer.py	Scan folders, store MD5s in SQLite, detect changes. Parallel MD5 with ProcessPoolExecutor
syncer.py	Cross-reference a new folder against an existing indexer.db: finds already-existing files (by MD5) and similar filenames (Levenshtein distance)
vidir	Make the moreutils/vidir work properly with my isolated vim
cmpDir1andDir2andRemoveIdenticalFromDir1	Remove from dir1 files that are identical (size+MD5) in dir2
cmpDir1andDir2andRemoveIdenticalFromDir1BasedOnSizeAndTimestamp	Same, but uses size + timestamp (faster, no MD5)

Example usage:

# Scan folders, store MD5s
indexer.py /mnt/photos

Duplicate detection

Script	What it does
find_dup_videos.py	Detect duplicate videos by duration + perceptual hash (imagehash)
find_dup_images.py	Find duplicate images via perceptual hashing

# Requires ffmpeg/ffprobe + python3 -m pip install ImageHash pillow
find_dup_videos.py /path/to/videos/

Video related

Script	What it does
minimalVideo.py	Transcode any video to x264 at ~0.06 b/p, mux with original audio via mkvmerge
minimalVideo265.py	Same concept, but using HEVC (x265)
minimalVideoCRF_265.sh	HEVC encode with CRF mode (quality-based), opus audio
minimalVideoCRF_265_copy_audio.sh	HEVC CRF encode, copy audio track directly
minimalVideoCRF_265_forQuest.sh	HEVC CRF encode variant for VR headsets (Quest)
vaapi_encode.sh	VAAPI HW-accelerating both decode + HEVC encode
vaapi_encode.just_encode.sh	VAAPI HW-accelerated HEVC encode only
vaapi_decode_in_SW_encode_in_HW.sh	Decode on CPU, upload frames to the GPU, HEVC-encode via VAAPI with opus audio
cuda_encode_hevc.sh	NVENC HEVC encode (NVIDIA GPU)
cuda_encode_av1.sh	NVENC AV1 encode (NVIDIA GPU)
fps_2x.sh	Double frame-rate via VAAPI motion interpolation (60fps)
fps_2x_vapoursynth/	Vapoursynth-based 2x FPS scripts
any2mp3.sh	Extract audio track to LAME mp3 from any media file
show_video_bpp.py	Show bits-per-pixel stats for a video; also invocable as bpp
show_audio_bpp.sh	Print each file's audio bitrate next to its name; also invocable as app
show_video_details.py	Dump codec/res/fps/bitrate of a video file
m4a2aac.sh	Extract AAC from M4A container
mpvYoutubeNoVideo.sh	Play YouTube audio-only (no video, saves CPU)
jpg2lep.sh	Lossless JPEG recompression via Dropbox Lepton
png2webp.sh	Convert PNG to WebP via ffmpeg
recordX11.sh	Lossless X11 screen capture
flashVideo.sh	Extract flash video from browser plugin's pipe
misc.py	Shared utility module for video scripts

Pi (AI coding agent) helpers

Script	What it does
pi.isolated.sh	Primary launcher: sandboxed pi under isolate.sh -- local LLM over a Unix-socket tunnel, or a remote endpoint via --url. Symlinked as pi.sh
pi.google.sh	Interactive pi TUI against Google AI Studio through pi.isolated.sh -- read-only $HOME, egress only to localAI/google-servers.txt; key from ai.google.key
pi.google_run.sh	One-shot pi prompt against Google AI Studio (--mode json -p) through pi.isolated.sh; key from ai.google.key
pi_parse_stream.py	See localAI/pi.subagent/AGENTS.md (streaming JSON/pi)
pilog.sh	Render a ~/.pi/agent/sessions JSONL session as Markdown
cclog.sh	Render a Claude Code session JSONL as Markdown with a containerized cclog Go tool
localAI/	Local LLM serving: llama.cpp launch scripts, recorded benchmark runs, pi subagent prompt and extensions, hosted-session allowlist
Dockerfiles/	Dockerfiles (e.g. the yt-dlp image, etc)

System monitoring & statistics

Script	What it does
dstat	Vendored, locally patched dstat (system resource monitor); see vendored code
dstat.sh	Wrapper: dstat -clnv --fs --vm
stats.py	Pipe in numbers -> colored statistics (mean, stddev, median, min/max)
statsLive.py	Real-time mean+stddev over stdin stream (Welford's online algorithm)
histogram.py	matplotlib histogram from a file/stdin (-); -r drops outliers, -p out.png for non-GUI output
histogram.sh	pipe to it to get text histogram and datamash percentiles (-n no histogram, -r ignore outliers)
asciigraph.py	matplotlib line graph of a stdin data series, one number per line (title required as $1)
asciigraph	text mode ASCII line graph -- one number per line on stdin
crystaldiskmark.sh	CrystalDiskMark clone for Linux using fio
benchmark.nvme.via.io.uring.sh	NVMe sequential read benchmark via io_uring
percentile.sh	Quick percentile calculator
performance	Pin all CPU cores to the performance governor and disable ASLR before benchmarking; needs root
ondemand	Return cores to the load-based powersave governor and re-enable ASLR after benchmarking; needs root

# Colored stats
for i in {1..100}; do echo $i; done | stats.py

# Streaming data stats
for i in {1..100} ; do echo $i ; sleep 1 ; done | statsLive.py
Mean: 3.0000  StdDev: 1.4142 # updated live as data arrive

# Histogram
cat numbers.txt | histogram.py -      # matplotlib
cat numbers.txt | histogram.sh        # text histogram + percentiles

# ASCII graph (vendored binary)
for i in {1..100}; do echo $i; done | asciigraph -h 20

Networking

Script	What it does
block_user_via_iptables.sh	Block a Unix user's outgoing traffic via iptables
unblock_user_via_iptables.sh	Remove the iptables block for a user
sync_NIC_HW_timestamping_clock_from_host.py	Sync NIC hardware timestamp clock (phc2sys) for accurate tcpdump timestamps
tcpdump2binary.py	Extract raw Ethernet frames from a pcap into binary files (uses scapy)
myip.sh	Show my public IP
nointernet.for.genymotion.sh	Run Genymotion with no internet but a working private loopback (localhost adb only)
youtubeFromClipboard.sh	daemon; auto-plays video URLs found in the clipboard via mpv, hands zathura: http... URLs to Firefox
yt-dlp.sh	Run yt-dlp safely (JS madness!) from inside a container that maps $PWD inside it
SendAtWarp9 / ReceiveAtWarp9	Push files across a LAN: tar | lzop -1 | nc on one side, nc -l | lzop -d | tar x on the other, port 9656
SendAtWarp9r / ReceiveAtWarp9r	Same transfer with the roles reversed -- the sender listens, the receiver connects

Subtitle tools

Script	What it does
get_subs_tmux.sh	Download YouTube subtitles, convert to text, launch an interactive pi session to summarize
vtt2text.py	Convert VTT subtitle files to clean text
subrip.sh	Subtitle RIP/conversion utility
sub_auto_fixer.py	Match subtitle files to video files by Dice coefficient + LCS

File/text utilities

Script	What it does
latest.py	List recently modified files in a directory, sorted by timestamp
latest.sh	Bash-only poor-mans-version of latest.py
exclude.sh	Interactively filter lines from a file by regex (used with latest.py)
excludeFilter.sh	Same, but via stdin pipe
greedy.py	Pack files to maximally fit a target size
lost_my_space.py	Compares current fs against previous snapshot of file sizes (-u takes a new snapshot)
lost_my_space.sh	wrapper around the .py: keeps the DB in /root/bin.local, then filters the output via exclude.sh
mydu.sh	Better du: sums file sizes (filesystem-agnostic) per directory
logDurations.py	Parse timestamped logs and show duration of each run
sortXML.py	Sort XML elements while preserving structure
csvToHTML_bootstrap.py	CSV -> styled HTML table (Bootstrap)
csvToHTML_barebones.py	CSV -> minimal HTML table
excelToCsv.py	Convert Excel (.xls/.xlsx) to CSV
countMoviesLength.py	Sum total duration of video files in a folder
rmdirRecursive.sh	Remove all empty directories recursively (post-cleanup)
inlineDataInHTML.sh	Generate data: URI for embedding files in HTML
nocolor.sh	Strip ANSI color codes from pipe
quote.sh	Wrap each line in double-quotes, escaping inner quotes
nth	Print field N of each line, negative indexes allowed -- the awk '{print $3}' shortcut
supergrep.pl	Minimal grep over stdin/files with -n, -v, -i
gdv	Open every file modified in the current git repo in vim, ready for GitGutter staging
encfs	encfs wrapper that makes both arguments absolute and adds -o allow_root
opus.it.sh	Convert a media file's audio track to 64kbps opus
showPackages	List Debian packages by installed size, largest last

Clipboard & automation

Script	What it does
clipboardDaemon.py	Clipboard monitoring daemon
execOnChange.sh	Run a command whenever files matching a pattern change (like make for file changes)
batch.SMP.processing.with.bash	Simple parallel processing with bash
lock.sh	Lock screen via xscreensaver
setTitle.sh	Set XTerm title
tmux-attach.sh	Attach to a tmux session
tmux-restore.sh	Restore tmux sessions from saved state
tmux_colors.sh	Show tmux color palette for config
fzf-preview.sh	File preview while browsing with fzf + bat
timestamped_logger.sh	Run a command, log stdout+stderr with timestamps

Developer tools

Script	What it does
cpptypefmt	C++ type formatting helper
clean-template-mess.py	Pretty-print C++ template instantiations with indentation
findClassInJars.sh	Find a class name inside a collection of JAR files
scalaRunner	Run a Scala script with -savecompiled -deprecation
make-phase1	Ask make what it knows: dump its variables and implicit rules for a target, with env/autovar noise filtered
strace_resumed.awk	awk -f filter that re-joins strace's <unfinished ...> / <... resumed> pairs per PID
findIn	Flexible file content search
git-lg	Git log with graph (one-line)
diffIgnoreCaseAndWS.sh	diff -u -i -b wrapper
diffu.meld	Diff with meld
xmlprettyprint.sh	XML pretty-printer (symlink to tidyXML.sh)
tidyXML.sh	XML tidy/format
html5check.py	HTML5 validation
make_ada_tags.py	Generate ctags for Ada sources (gnatxref)
pylintAll.sh	Run pylint on all Python scripts in current dir
showBashCmdline.sh	Show the currently executing command line inside a bash shell
vimcat.sh	Cat a file with vim-based ANSI syntax highlighting
svndiff	SVN diff wrapper
svndiffMELD	SVN diff via Meld
svndiffWS	SVN diff (whitespace-aware)
svndiffXXDIFF	SVN diff via xxdiff

Format conversion & encoding

Script	What it does
ansi2html.sh	Convert ANSI-colored terminal output to HTML
htmlEntities.pl	HTML-encode text (Perl)
unicodeUnescape.py	Decode \uXXXX escape sequences
hex2utf.py	Hex -> UTF-8 converter
mime_decoder_inplace_pipe.py	Decode MIME-encoded files in-place
epochToHuman.pl	Convert Unix epoch to human date
tohex	Decimal -> hex
todec	Hex -> decimal
tozero.sh	feed safely from a pipe to xargs -0
href.pl	Extract href links from HTML
img.pl	Extract img src links from HTML
unescape.pl	Decode URL-encoded (%XX) and HTML entities
speed.pl	Monitor download speed/progress of a file

System administration

Script	What it does
swap.usage.sh	Show swap usage per process
zswapStatus.sh	Check zswap status
hyperthreading	Toggle hyperthreading
brightScreen.sh	Set backlight brightness to max
256color.pl	Display 256-color terminal palette
waitForCoolCPU.sh	Wait until CPU temperature drops
waitForProcToDie.pl	Wait for a process to exit, then run a command
dumpMemoryOfPID.sh	Dump all memory pages of a PID (as root)
dump_all_thread_callstacks_for_pid.sh	Dump all thread callstacks of a PID via GDB
callstacks_of_threads.sh	Symlink to above
thread_callstacks.sh	Symlink to above
snap_remove_old_versions.sh	Clean old snap package versions
archLinux.orphans.ordered.by.size.sh	List orphaned pacman packages sorted by size
x11.off.sh	Blank/lock the console with i3lock
x11.off.saver	Turn display off via xset dpms/xset s
fixClock.sh	Sync system clock via NTP (pool.ntp.org)
7z-ttsiod-compress	7z compression wrapper
makeSelfSignedCertificate.sh	Generate self-signed SSL certificate (legacy, use Let's Encrypt instead)
bcmp.sh	Binary compare two files (shows offset + hex bytes)
setSubtract.pl	Subtract one list of strings from another
setUnversionedToSVNignore.sh	Mark all ? files as svn:ignore
USB_drives_need_no_sleep.py	Keep USB drives awake (prevent spin-down on ZFS server)
loopaesmount.py	Mount loopback AES-encrypted containers
pulseaudio_record.sh	Record PulseAudio output to OGG
sync_music_to_android_with_m3u_for_folder_symlinks.sh	Sync music to Android with .m3u playlist files
mergefs_show_realpath.sh	Show mergerfs real path for files
pdfCountPages.sh	Sum page counts of PDF files
yq_outliers.sh	Flatten YAML to dot-notation for grepping
x16.py	GDB helper: dump 16-column memory (vs default 8)
unchroot.pl	Break out of a chroot jail (security testing)

Vendored third-party code

Everything here is MIT-licensed (see LICENSE), except the files below, which are other people's work and keep their own upstream licence.

File	Upstream	Licence
dstat	dstat-real/dstat (formerly dagwieers/dstat, which redirects there) -- archived, last commit 2020-06-18; the original homepage dag.wieers.com now answers HTTP 500. This copy is upstream master plus Python-3 compatibility fixes only (shebang, collections.abc, raw-string regex literals). The still-maintained Python-3 fork is scottchiefbaker/dool.	GPL-2.0+ (upstream ships the text as COPYING)
ansi2html.sh	Pádraig Brady / pixelbeat, also mirrored in pixelb/scripts	LGPL-2.0 (per the file's own header)
vimcat.sh	Matthew Wozniski's AnsiHighlight-based vimcat, mirrored at vim-scripts/vimcat; actively maintained successor: ofavre/vimcat. The file is a bash wrapper that sources itself as Vim script (vim:ft=vim).	BSD (per the file header, which asks to be credited)
html5check.py	Client for the Nu HTML checker, defaulting to http://html5.validator.nu/. It sends a User-Agent, which that Jetty-based service requires (it answers 400 Bad Request without one). Only http:// endpoints are supported -- --service=https://... exits with 7.	MIT (Mozilla Foundation header in the file)
256color.pl	Todd Larason's xterm colour test	not MIT (upstream xterm/XFree86)
swap.usage.sh	answer at stackoverflow.com/q/479953	CC-BY-SA (StackOverflow default), not MIT
unchroot.pl	pentestmonkey's chroot-breakout script; the blog post is named in the file header	none stated in the file

Contributions on record

• csvToHTML_bootstrap.py -- contributed by Karanveer (commit daee254, Oct 2019).
• rmdirRecursive.sh -- carries the -empty fix contributed by Tim Stackhouse (commit 2d24eac, Oct 2014).

Dependencies

• Most scripts: bash, standard POSIX tools
• Python scripts: Python 3, various modules (see docstrings)
• nth: pure stdlib; waitForCoolCPU.sh and pulseaudio_record.sh depend on it
• Warp9 transfers: lzop, nc (classic netcat -l -p syntax); no encryption -- trusted LANs only
• Video scripts: ffmpeg / ffprobe, mplayer, mkvmerge / mkvextract, x264/x265, VAAPI or NVENC drivers
• isolate.sh: firejail, ip, getent
• dockerme.sh: Docker
• lost_my_space.py: zstd / zstdcat, plus pv when using the wrapper
• pi.isolated.sh: A locally running LLM endpoint (llama.cpp, vLLM, etc.)
• histogram.sh: datamash, column; histogram.py / asciigraph.py: matplotlib and numpy
• crystaldiskmark.sh, benchmark.nvme.via.io.uring.sh: fio
• clipboardDaemon.py, youtubeFromClipboard.sh: xclip; tmux helpers: tmux; pi.isolated.sh tunnel: socat
• dump_all_thread_callstacks_for_pid.sh, dumpMemoryOfPID.sh, x16.py: gdb
• find_dup_images.py: ImageHash, pillow (pip) and feh to review the groups
• excelToCsv.py: xlrd
• Single-tool scripts: wget (myip.sh), bc (tohex, todec), mimetype (inlineDataInHTML.sh), pdfinfo (pdfCountPages.sh), yq + jq (yq_outliers.sh), exiftool (png2webp.sh), lepton (jpg2lep.sh), lm-sensors (waitForCoolCPU.sh), pactl/parec + oggenc (pulseaudio_record.sh), fzf + bat (fzf-preview.sh), xscreensaver (lock.sh), i3lock (x11.off.sh)