Launch Bash Commands From Text Files

My version of multithreading may not be the most effective or efficient, but it's so easy to remember when in the middle of a task plus the overhead is very light.

while read list; do $list & done < list.txt

Simple Way to Rename File Using Excel

I often have the need to modify a large amount of filenames, and I have often found it easiest to turn to Excel and a simple bash script.

Combine this formula with a sequence number and you have a very easy way to rename a massive amount of files and keep the filenames unique.

 =concatenate("cp ",char(34),A1,char(34)," ",char(34),"renamed/", B1,char(34))

cp "oldfilename" "renamed/newfilename" 

  1. Drag that formula down the entire column.
  2. Copy the column to a text editor and save as rename.sh. 
  3. Open a terminal/console and change to the directory containing the files you wish to rename.
  4. If you haven't made a directory called renamed, type mkdir renamed. 
  5. Run bash rename.sh 

Generate Blank Media

Sometimes I have the need to generate truly blank/empty media on demand. Here are some solutions I found using ffmpeg and imagemagick:

-30 Seconds of blank/empty audio:
    ffmpeg -ar 48000 -t 30 -f s16le -acodec pcm_s16le -ac 2 -i /dev/zero -acodec libmp3lame -aq 4 blank.mp3

-30 seconds of blank/empty video:
    touch 1.txt && convert 1.txt -page Letter 1.png && ffmpeg -loop 1 -i 1.png -c:v libx264 -t 30 -pix_fmt yuv420p -aspect 16:9 blank.mp4 && rm 1.*

-5 page blank/empty pdf:
    for f in {1..5}; do touch temp.txt && convert temp.txt -page Letter temp.$f.png; done && convert temp.*.png blank.pdf && rm temp.*

-Blank/empty png file:
    touch 1.txt && convert 1.txt -page Letter blank.png && rm 1.txt

Corrupt PDFs

As a digital/media asset manager, corrupt files are the bane of my existence.

To programmatically find corrupt pdf files, I landed with the following utilizing pdftotext from xpdf.

for f in *.pdf; \
do pdftotext -q -f 1 -l 2 $f $f.txt; \
err=$?; \
if [ $err -ne 0 ]; then mv $f _failed_$f; \
elif [ $err -eq 3 ]; then mv $f _locked_$f; \
else rm $f.txt; fi;
done

Kick off multiple instances of script

I had the need to launch about 100 instances of a script all at once to scrape some web data. This was the loop I used:

for f in *.txt; do python inStock.py $f & done

Note:

  • Semicolon launches consecutively, waiting for previous script to finish:
    • for f in *.txt; do foobar.py $f; done
  • Ampersand launches in the background rather than waiting for the consecutive script before it to finish:
    • for f in *.txt; do foobar.py $f & done

 

Remove duplicate files by MD5

I recently had the need to delete an unknown number of duplicate files out of a batch of ~10,000 images. The filenames were all different, but the md5 hash was known and all of the duplicates had the same.

md5 -r *.jpg | grep "37d1b8f9d6f02f31cmb192a28b96cade" | awk '{ print $2 }' | xargs rm

Split a text file in half

I often have the need to split long text files into smaller chunks, and my need requires they are split by line.

Here's my one liner solution:

f=$"filename"; s=$(wc -l $f | awk '{print $1}'); \
h=$(echo "scale=0;" $(($s/2+1)) | bc -q); \
split -l $h $f "output_"; for file in output_*; \
do mv "$file" "$file.txt"; done

Notes:

  • BC can handle floating point numbers, so scale=0 limits any decimal output.

  • Bash math requires double parentheses.