3 February 2007Own project · Linux & UnixOwn post

Six Linux one-liners from 2007 and what you type instead today

Summary

In early 2007 I kept a small list of one-liners on bk99.de, “everything normal people need day to day”: finding duplicate files, editing files over ssh, sending disk images compressed across the network and rescuing failing disks. Three of them still work today, two contained errors, and one no longer works that way with today’s standard tool. This post from my old blog shows the list, what each line does and how you would write it today.

The list from 2007

The page was called “Linux Quickies” and was a table of command and description. This is how it looked online in February 2007 (descriptions in German as in the original):

find . -exec md5sum {} 2>/dev/null \; | sort | uniq -w 32 -D     Doppelte Dateinamen finden
ssh -t root@tyr 'vi /etc/passwd'                                ssh mit -t Pseudo Terminal
buffer ssh | bzip2 | buffer | ssh "cat > image.bz2"
rsync | ssh                                                     rsync oder ssh with -C
dd_rescue -r /dev/hdb /dev/stdout | ssh root@bla "cat > image"  Von hinten nach stdout und dann per ssh
dd_rescue /dev/sda1 - | ssh username@machineip ‘cat /datarecovery/backup.img’

Finding duplicate files

The first line calculates the MD5 checksum of every file, sorts the list and lets uniq print only the lines whose first 32 characters, i.e. the checksum, occur more than once. It therefore finds files with identical content, not identical file names as the description claims. Tested today, it still works.

Two improvements make it faster and cleaner: -type f skips directories, to which md5sum only responds with error messages, and {} + passes many files at once instead of starting a separate md5sum for each.

find . -type f -exec md5sum {} + | sort | uniq -w 32 -D

For large collections there are dedicated tools, all included in Debian: fdupes -r compares sizes first and then contents, rdfind and jdupes can replace duplicates with hard links. That is faster because files with a unique size are not read at all.

A terminal for the command on the other side

ssh with a command at the end runs it without a terminal. That is fine for ls or df, but an editor like vi needs a terminal, otherwise the screen stays blank or keys arrive wrongly. -t requests a pseudo-terminal on the remote side.

ssh -t root@tyr 'vi /etc/passwd'

For exactly this file I would use vipw today. It locks the file while editing and checks for errors afterwards instead of saving a broken passwd: ssh -t admin@tyr sudo vipw.

Compressed images across the network

The third line is more of a reminder than a command: produce data, buffer it, pack it with bzip2, buffer again and write it via ssh into a file on the other machine. The buffer program smoothed out the jumps between a fast disk, a slow compressor and the network. The fourth line meant that rsync runs over ssh and that ssh -C or rsync -z compress the data on the way.

Today zstd replaces bzip2: it is many times faster, uses all cores with -T0, and pv shows progress and throughput. If needed, mbuffer takes over the buffering role.

dd if=/dev/sdX bs=4M | pv | zstd -T0 | ssh backup@ziel 'cat > /backup/sdX.img.zst'
rsync -az --info=progress2 quelle/ backup@ziel:/backup/quelle/

Rescuing failing disks

The last two lines use dd_rescue by Kurt Garloff. Unlike dd it does not stop at read errors but skips them, and with -r it reads the disk from back to front. That helps when a damaged area at the start holds up the read head. Output went to standard output and via ssh to another machine so that nothing is written to the sick disk.

The last line could never have worked, for two reasons: instead of straight quotes it contains typographic quotation marks, which the shell does not recognise as quoting. And cat reads the file instead of writing it; what was meant is cat > /datarecovery/backup.img.

Debian no longer includes dd_rescue; the package last existed in Debian 6. The usual tool today is GNU ddrescue from the gddrescue package. It keeps a map file recording which areas have already been read, can also read backwards with -R and can resume a run at any time. It does not support the route via standard output, though, because it needs an output file it can jump around in. The image is therefore created on a healthy disk or a mounted network drive and compressed afterwards.

ddrescue -d -n /dev/sdX /mnt/rettung/sdX.img /mnt/rettung/sdX.map
ddrescue -d -r3 /dev/sdX /mnt/rettung/sdX.img /mnt/rettung/sdX.map

The first run with -n quickly fetches everything that can be read without effort, the second tries the difficult areas up to three times. -d reads directly from the disk, bypassing the cache.

Numbers at a glance

The note “Linux Quickies” was in the old Plone blog under /dokumente/linux-quickys; the archived version is dated 3 February 2007.
uniq -w 32 compares only the first 32 characters of a line, exactly the length of an MD5 checksum in hexadecimal.
According to its manual, GNU ddrescue needs input and output files with random access and therefore does not write to a pipe.
In Debian 13 the package for GNU ddrescue is called gddrescue; dd_rescue last existed in Debian 6.

References

Remarks

In 2026 I took the table over verbatim from the Web Archive, as a text block instead of a table; later additions up to 2009 have not survived there.
tyr from the second line was one of my machines at the time, like odin and Hödur named after gods of Norse mythology.

Original note in the Web Archive