download in pdf format
Status: in progress

Motivation

1. Have an idea of how long is it going to take
2. Interruption does not mean corruption
3. Faster downloading

aria2c

aria2c -x 16 -s 16 -c https://server/file.fastq.gz

parallel -j 4 aria2c -c {} :::: urls.txt

for sra

prefetch --max-size 100G SRR123456
fasterq-dump --threads 8 SRR123456

Simple examples and syntax

Understand how big a folder is

du -sh /path/to/folder

Dry run to understand what gets copied

rsync -avhn --stats source/ destination/

• -a for sensible defaults (permissions, recursivity etc.)
• -v for verbose
• -h for human readible file sizes
• -n for not run

Copy from source to destination displaying progress bar

rsync -ah --info=progress2 source/ destination/

Check all get copied by checking checksum

rsync -a --checksum --dry-run source/ destination/

Delete the source if needed

rm -r source/