Parsing "ls -l" output can be very difficult. Recently I've had a need to do just that in order to migrate files from one computer to another in an intelligent way, meaning only updating a file if the migrated file was "newer" then the existing file. To facilitate this, I needed a "dropbox" where I could deposit the migrating files, where they could be compared with the existing files by their date-time stamps. That was easy ... I just created a "dropbox" folder on each of my Mac computers at the $HOME level. Now, and machine can transmit files to a "dropbox" on another computer. Then I could login to that computer and finish the migration using a script.
I wanted another script on each computer that would select the files to migrate from a single directory that had modification date for the current "month" or for "today". I created "movtoday" to handle BOTH options by having a "movmonth" symlink pointing at "movtoday". Within that script, I look at the $0 parameter to determine if it is "movmonth" or "movtoday" that is being called. All the actions within the script are the same, with a single exception which determines the pattern within "ls -l" output in the date portion. For "month", I just want the month's name, like Jan or Feb. For "today", I want both the month's name AND today's date, from 1 through 31. Unfortunately, $(date '+%b %d') returns a two-digit value for %d, but the output for "ls -l" returns ' x' instead of '0x' for the first nine days of a month. There a trick for handling that, as you'll see in the actual "movtoday" code.
I then added a "movyear" symlink pointing to "movtoday", and with minor modifications to "movtoday" was now able to move all files in a directory that had "hh:mm" time-stamps.
So how can I parse the "ls -l" output to obtain only those files I want to migrate? I chose to migrate only real files, not directories, or sockets, of symlinks. However, I added an option to the call to either "movtoday", "movmonth", or "movyear" that allows me to do an "ls -lL" which follows symlinks to their target file. That allows me to move the actual file, based upon its date-time, to a symlink on the destination side that mirrors the symlinks on the source side.
If you copy/paste thollowing is "movtoday" code, be sure to eliminate blank lines:
#!/bin/bash
# normally, symlinks are not moved, but you can add "L" parm
# to follow the symlinks, and move the target files themselves.
# - - - - Insure we have an empty dropbox - - - -
mybox="$HOME/dropbox"
if [ ! -d "$mybox" ]; then
echo "You need to create $mybox to use this code." ; exit 0
fi
mytest=$(ls -1 $mybox | wc -l | tr -d ' \t')
if [ "$mytest" != "0" ]; then
echo "$mybox is not empty." ; exit 0
fi
# - - - - Eliminate any old migration list so we don't migrate it - - - -
rm -f put.today 2>/dev/null # eliminate old version
# - - - - Determine the "destination" directory - - - -
if [ -f mysym ]; then
mysym=$(cat mysym)
else
if [ "$PWD" = "$HOME" ]; then
mysym='myhome'
else
mysym=${PWD#$HOME/}
if [ -z "$mysym" ]; then
mysym="$PWD"
fi
fi
echo "$mysym" >mysym
fi
# - - - - Create a migration command list in a /tmp file - - - -
mytemp="/tmp/today$$"
rm -f $mytemp 2>/dev/null
touch "$mytemp"
echo "cd $mybox" >$mytemp
echo "rm *" >>$mytemp
# - - - - Determine migration by "month" or "today" - - - -
myself=$(echo "$0" | grep 'today')
if [ -n "$myself" ]; then
mydate=$(date '+%b %d')
mydate=${mydate/ 0/ } # - - turn days like " 0x" into " x" - -
else
myself=$(echo "$0" | grep 'year')
if [ -n "$myself" ]; then mydate=''
else mydate=$(date '+%b')
fi
fi
# - - - - Parse the "ls -l" command to create the migration "put" commands - - - -
ls -l$1 | grep " $mydate " | grep '^-' | awk -F' [0-9]{2}:[0-9]{2} ' '/:/{print $2}' | sed 's/ /?/g' | xargs -L1 echo 'put -p' >>$mytemp
# Alternate: replace "awk -F' [0-9]{2}:[0-9]{2} ' '/:/{print $2}'" above by "grep ' [0-9]*:[0-9]* ' | tr -s ' ' | cut -f9- -d' '"
# - - - - Add "mysym" file to list if it isn't already included - - - -
mypath=$(grep ' mysym$' $mytemp)
if [ -z "$mypath" ]; then echo 'put -p mysym' >>$mytemp ; fi
# - - - - Terminate the "sftp" command list - - - -
echo 'quit' >>$mytemp
# - - - - Move the command list into "put.today", eliminating log-files - - - -
cat $mytemp | grep -v '\.log$' >put.today
rm -f $mytemp 2>/dev/null # - - - cleanup - - -
cat put.today # create new version
echo "" ; echo "Use 'sfx dmv put.today #' from here to move--After the move, use movtest at destination"
exit 0
The key line is this:
ls -l$1 | grep " $mydate " | grep '^-' | awk -F' [0-9]{2}:[0-9]{2} ' '/:/{print $2}' | sed 's/ /?/g' | xargs -L1 echo 'put -p' >>$mytemp
It starts with the "ls -l" (or "ls -lL" if you specified the L parameter). The first "grep" parses for the " mon " or " mon day " fields. This retains all files with matching month or month-day requirements. The next grep keeps all lines that begin with a dash (-), which are simple files, NOT directories, sockets. If "-lL" is used, symlink are followed, otherwise, they are eliminated as well. The "awk" command retains all lines with "hh:mm" pattern, which is the time-stamp field. In older files, it is the "year" in ccyy format. So we now have found the break between the front part of the line, and the last part that contains the file's name. "awk" retains the last part (print $2). Some of these file names may have blanks in them, and "sftp" can handle them if we change blanks into question-marks. That's what "sed" does. Finally, "xargs' handles every line separately and executes the "echo" command which creates "put -p filename" commands into our command file ($mytemp).
What's currently missing here is the "movtest" script used on the destination machine, and the "sfx" and 'dmv" scripts used to drive the "sftp" process using "put.today".