lndup

Hardlink duplicate files. Ultra fast!

asynchronous I/O
one file one stat() call
full-speed hash least files
customize filters and keys with javascript

Installation

$ npm i -g lndup

Usage

Usage: lndup [OPTION]... [PATH]...
Hardlink duplicate files.
  -n, --dry-run  don't link
  -v, --verbose  explain what is being done
  -q, --quiet    don't output extra information
  -i, --stdin    read more paths from stdin
  -f, --file     add a file filter
                 (stats: fs.Stats, path: string): boolean
  -d, --dir      add a directory filter
                 (stats: fs.Stats, path: string, files: string[]): boolean
  -k, --key      add a key to differentiate files
                 (stats: fs.Stats, path: string): any
  -H, --hash     select a digest algorithm, default: sha1
                 run 'openssl list -digest-algorithms' for available algorithms.
  -h, --help     display this help and exit
  -V, --version  output version information and exit
See <https://github.com/chinory/lndup>

Not follow symbolic links.

Example

All outputs are executable unix-shell code, using comment to carry extra information.

$ lndup -v .
#Stat: probe: readdir       204B   3
#Stat: probe: stat     144.02MiB  23
#Stat: probe: select   144.02MiB  19
#Time: probe: 7.351ms
#Stat: verify: internal         0B  0
#Stat: verify: external  144.00MiB  9
#Stat: verify: total     144.00MiB  9
#Time: verify: 183.209ms
#Stat: solve: current  112.00MiB  7
#Time: solve: 0.110ms
ln -f -- '16M/null_2' '16M/null_3'
ln -f -- '16M/null_2' '16M/null_1'
ln -f -- '16M/ran1_1' '16M/ran1_2'
ln -f -- 'root/ran4_1' 'root/ran4_2'
ln -f -- 'root/ran4_1' 'root/ran4_2' #Error: EACCES: permission denied, rename 'root/ran4_2' -> 'root/ran4_2.e8c70ebe0635ab41'
#Stat: execute: todo  64.00MiB  3  4
#Stat: execute: done  48.00MiB  2  3
#Stat: execute: fail  16.00MiB  1  1
#Time: execute: 8.331ms

Customize filter & key

File Filters: If you don’t want to hardlink files smaller than 1024 bytes:

$ lndup /path -f 'stats=>stats.size>=1024'

Directory Filters: While you don’t want to consider a directory with more than 100 files:

$ lndup /path -f 'stats=>stats.size>=1024' -d '(s,p,f)=>f.length<=100'

Extra keys: Obviously, you don’t want to hardlink the same files with different user, group and mode:

$ lndup /path -k 's=>s.uid' -k 's=>s.gid' -k 's=>s.mode'

Require more: Finally, you have a super idea:

$ lndup /path -k 'require("/path/to/keyfunc.js")' -f 'require("/path/to/filter.js")'

Notice

Failed operation will be output to stderr like following:

ln -f -- 'root/ran4_1' 'root/ran4_2' #Error: EACCES: permission denied, rename 'root/ran4_2' -> 'root/ran4_2.e8c70ebe0635ab41'

Beware of mv and rm fails, they are remedies of failed link operation, you need to complete them manually. Fortunately, the remedies rarely fail, as they are all counter-operations of just-successful operations.

Requirement

Node.js >=9

Introduction

data structure

// nested maps
devMap // devMap instanceof Map
sizeMap = devMap[stat.dev] // sizeMap instanceof Map
exkeyMap = sizeMap[stat.size] // exkeyMap instanceof Map
contentMap = exkeyMap[value of extra keys] // contentMap instanceof Map
inoMap = contentMap[hash.digest] // inoMap instanceof Map
paths = inoMap[stat.ino] // paths instanceof Array

processing

probe(paths).then(verify).then(solve).then(execute)

probe: Traverse the input paths asynchronously while use the stat()’s result to group files.

verify: Find out least files to hash, and group files by the digest.

solve: Make solution that instruct to hardlink the file whose inode is majority to other files.

execute: Execute that solution or just dry-run.