Bash Utility Usages

Permissions

1, 3, 5, 7 xyz x = file owner y = group owner z = everyone else 7 = r + w + x 5 = r + x

755 = read/write/execute to owner, read/execute to group owner 775 = owner read/write/execute

Time utility

real - wall clock time user - amount of cpu time used while executing sys - amount of cpu time in hte kernel in the process

Compression and Decompression

tar examples

Compress

tar --create --file /tmp/archive.tar --verbose /tmp/archive/ # Create archive

List files

tar -tzf archive.tar.gz

Decompress

tar xf <archive name>.tar # Unpack archive
tar -xzf archive.tar.gz

pigz examples

Compress

tar --use-compress-program=pigz -cf bldC.tar.gz bin/*

Decompress

# 3m real time decompression
pigz -dc bldC.tar.gz | tar --directory=/tmp/cpp -x -f -
pigz -dc bldC.tar.gz | tar -xf - -C cpp/

zstd examples

Compress

# 2m 5s
tar -cf - cpp/ | zstd --long -T0 -3 > bldC.tar.zst
# 4m 30s
tar -cf - cpp/ | zstd > bldC.tar.zst

Decompress

# 1m 20s - so 3x as fast
zstd -d -c bldC.tar.zst | tar -xf - -C cpp/
# no speed up using long -T0 options from above

Symlinking

Relative symlinks can have unintended consequences - I try to avoid

readlink /tmp/mysymlink
ln -s /tmp/target.a /tmp/symlink_to_target.a

Remove files - rm too long error

Can encounter errors with fuzzy matching if too many files return

#rm -rf ./*.dat # old style
find . -name "*.rst" -print0 -exec echo {} +
find . -name "*.rst" -delete
find . -mtime +N # modified more than N days ago

Sed commands

find /path/dir -type f -name "*.txt" -exec sed -i 's/oldpattern/newpattern/g' {} +
# on mac, after brew install gsed
find /path/dir -type f -name "*.txt" -exec gsed -i 's/oldpattern/newpattern/g' {} +

Kill Procs

ps aux | grep python| awk '{ print $2 }' # List out processes to kill

Kill processes

kill $(ps aux | grep python | awk '{ print $2 }')
# get hanging proces
kill -9 $(ps aux | grep python | awk '{ print $2 }')
# Or, with pkill
pkill -f python
# Or, with killall
killall -r python

Postgres Commands

View and reload HBA File

SHOW hba_file;
SHOW data_directory;
SELECT pg_reload_conf();

pg_dump and pg_restore examples

pg_dump -Z 6 -F c db_name > /tmp/db_dump.dump
pg_restore -F c ./db_dump.dump -d db_name -v

Quick and dirty backups

Improved version

hosts=("example" "example2")
for h in "${hosts[@]}"; do
    echo "Runnning $h"
    ssh "$h" 'rsync -avP ~/ archivehost:/tmp/archive/$HOST/home/'
done

Simple version using excel to generate lists

HOST=example; ssh "$HOST" 'rsync -avP ~/ "archivehost:/tmp/archive/$HOST/home/' &
HOST=example2; ssh "$HOST" 'rsync -avP ~/ "archivehost:/tmp/archive/$HOST/home/' &

Iperf3 usage

iperf3 -s -p 5201 # On server
iperf3 -c <server-ip> -p 5201 # On Client
iperf3 -c <server-ip> -p 5201 -u # udp
for port in {5201..5210}; do iperf3 -c <server-ip> -p $port; done # multiple ports

RPM Commands

rpm -qa package # Find out where in file system package is installed
rpm -qa | grep package # Lookup where packages are installed

Lsof commands

lsof /path/to/file #  See What’s Using a File or Directory
lsof -u username
kill -9 $(lsof -t -i :5000)
lsof -p <PID>                # List open files for process ID

Cleanup remote git branches

git fetch
for branch in $(git branch -r --merged | grep -v HEAD | grep -v develop | grep -v master | grep -v master | sed /\*/d); do
        if [ -z "$(git log -1 --since='Jun 15, 2020' -s ${branch})" ]; then
                echo -e `git show --format="%ci %cr %an" ${branch} | head -n 1` \\t$branch
                remote_branch=$(echo ${branch} | sed 's#origin/##' )
        fi
done

Format xml file in place

xmllint -format -recover a.xml > b.xml