I was looking for a way to tidy up out photos on the NAS at home and have tried a number of things that just did not fit the bill.

they were either just to difficult or completely wrong.

i then stumbled upon this blog post http://falesafe.wordpress.com/2009/07/07/photo-management/

What a gem. if you can install ruby on a machine you have to sort your photos this is it.

The post is from 2009 and so I had to update some of the gems it uses as well as change some of the code. but it was not much work.

Thanks to Falesafe for making it available it had another bonus in that I found out that we have 76000 photos.

I feel some culling is needed.

EDIT:

I had to work with the script a bit as the EXIF attribute it was using was causing my photos to be sorted incorrectly namely (date_time).
So I have updated the script to use the (date_time_original) atribute and this has now sorted my photos properly for me. The original post that was written has comments that are closed so I will upload the adjusted script here if you want to use it.

 

#!/usr/bin/ruby
# == Synopsis
#
# This script examines a source directory for photos and movie files and moves them to
# a destination directory.  The destination directory will contain a date-hierarchy of folders.
#
# == Usage
#
# ruby photo_organizer.rb [ -h | --help ] source_dir destination_dir
#
# == Author
# Doug Fales, Falesafe Consulting, Inc.
# 
# == Change Log
# LANCE HAIG = changed the EXIF attribute used to determine photo date taken to .date_time_original
#
# == Copyright
# Copyright (c) 2009 Doug Fales.
# Licensed under the same terms as Ruby.
require 'rubygems'
require 'exifr'
require 'find'
require 'logger'
require 'optparse'
require 'pathname3'
require 'digest/sha3'

STDOUT.sync = true


#$log = Logger.new("photo_organizer.log", 3, 20*1024*1024)  # Log files up to 20MB, keep at least three around
#$log.info("Photo organizer started...")

def log
  @log ||= Logger.new("photo_organizer.log", 3, 20*1024*1024)  # Log files up to 20MB, keep at least three around
  @log
end

log.info("Photo organizer started...")


def usage()
puts <<-USAGE

   photo_organizer.rb takes a source directory and a destination directory.  All photos
   in the source directory will be scanned for EXIF timestamps and then moved to a
   folder path in the destination directory that corresponds to the year, month, and day
   found in the photo's EXIF data.  Movies will be moved to a "movies" directory
   under the destination dir.

      Usage: ruby photo_organizer.rb [ -h | --help ] source_directory destination_directory

USAGE
exit
end




if ARGV.length < 2
  usage
  exit
end

$file_counter = 0

@source_dir = Pathname.new(ARGV[0]).realpath
@dest_dir = Pathname.new(ARGV[1]).realpath
@movies_dir = File.expand_path(File.join(@dest_dir, "movies"))

log.info("Source: #{@source_dir}")
log.info("Destination: #{@dest_dir}")

opts = OptionParser.new
opts.on("-h", "--help") { usage }
opts.parse!(ARGV) rescue usage

def increment_counter
  $file_counter += 1
  file_counter_string = sprintf("%10d", $file_counter)
  print(13.chr + "Processed: #{file_counter_string} Files")
end

def get_time(f)
  if(is_movie(f))
    time = File.ctime(f)
  else
    time = EXIFR::JPEG.new(f).date_time_original
  end
  log.info("Time for: #{f} is #{time}")
  return time
end

def get_dest_dirs(t)
  year = t.year.to_s
  month = t.strftime("%m_%b_%Y").upcase
  log.info("Destination will be: #{year}/#{month}")
  return [year, month]
end

def get_filename(t, f)
  "#{t.strftime("%Y%m%d")}_#{File.basename(f)}"
end

def rand_hash
  return Digest::SHA3.hexdigest(Time.now.to_i.to_s + rand(10000).to_s)[0..10]
end

def uniquefy(fname, destination)
  unique_name = File.join(destination, "#{File.basename(fname)}_#{rand_hash}#{File.extname(fname)}")
  while(File.exist?(unique_name))
    unique_name = File.join(destination, "#{File.basename(fname)}_#{rand_hash}#{File.extname(fname)}")
  end
  log.info("Made unique_name: #{unique_name}")
  unique_name
end

def exists_any_extension?(filename)
  ext = File.extname(filename)
  up_ext = ext.upcase
  dn_ext = ext.downcase
  return (File.exist?(filename.sub(ext, up_ext)) or File.exist?(filename.sub(ext, dn_ext)) or File.exist?(filename))
end

def move_movie(f)
  log.info("Destination directory is: #{@movies_dir}")
  filename = File.basename(f)
  destination_filename =  File.expand_path(File.join(@movies_dir, filename))
  log.info("Destination file is: #{destination_filename}")
  
  # Move to subdir, with new name
  if(exists_any_extension?(destination_filename))
    destination_filename = uniquefy(filename, @movies_dir)
  end
  
  if(File.exist?(destination_filename)) 
    log.info("ERROR: For some reason, #{destination_filename} still exists after uniquefy.  Will not attempt to move #{f}!")
    return false
  end
  log.info("Doing 'mv #{f} #{destination_filename}'")
  
  FileUtils.mv(f, destination_filename)
  return true
end

def move_image(f, time)
  (year, month) = get_dest_dirs(time)
  destination = File.expand_path(File.join(@dest_dir, year, month))
  log.info("Destination directory is: #{destination}")
  
  #    Make subdir
  FileUtils.mkpath(destination) unless File.exist?(destination)
  
  filename = get_filename(time, f)
  destination_filename =  File.expand_path(File.join(destination, filename))
  
  log.info("Destination file is: #{destination_filename}")
  
  # Move to subdir, with new name
  if(exists_any_extension?(destination_filename))
    destination_filename = uniquefy(filename, destination)
  end
  
  if(File.exist?(destination_filename)) 
    log.info("ERROR: For some reason, #{destination_filename} still exists after uniquefy.  Will not attempt to move #{f}!")
    return false
  end
  
  log.info("Doing 'mv #{f} #{destination_filename}'")
  FileUtils.mv(f, destination_filename)
  return true
end

def is_movie(f)
  return (File.extname(f) =~ /avi|mov|mpg|mp4|arw|raw/i)
end

def is_raw(f)
  return (File.extname(f) =~ /arw|raw/i)
end

unless File.exist?(@movies_dir)
  FileUtils.mkpath(@movies_dir)
end

Find.find(@source_dir) do |f|
  case
  when File.file?(f)

    if(is_movie(f))
      was_moved = move_movie(f)
      increment_counter if was_moved
      next
    end

    begin
      log.info("Processing file: #{f}")
      time = get_time(f)
    rescue => e
      if(f =~ /.DS_Store/)
        log.info("Skipping .DS_Store")
        next
      elsif (e.message =~ /malformed JPEG/)
        log.info("Malformed JPEG: #{f}")
        next
      end
    end

    if(time.nil?) 
      log.info("WARNING: No EXIF time for: #{f}.  Will skip it.")
      next
    end

    was_moved = move_image(f, time)
    increment_counter if was_moved

  when File.directory?(f)
    log.info("Processing directory: #{f}")
  else "?"
    log.info("Non-dir, non-file: #{f}")
  end
end

puts "nFinished."
									

I am currently helping a customer migrate from Novell to Microsoft and they are using the Quest migrator product to move their data to new DFS servers.
They currently have a large amount of data stored on a number of volumes. The sheer number of volumes and data have required that they deploy a large number of the copy engine servers.
The copy engine does not utilize a central logging facility, it stores the logfiles in a folder alongside the copy engine.
This unfortunately has a side affect, that there are now quite a few log files and some are reaching over 1.5GB in size.
Trying to load these files into a text editor as proven impossible and unworkable and another way was needed.

I decided that the best way to achieve this was to use a script that would parse the log files and extract the errors from the files into another file that would be smaller and easier to work with.

I decided to use Powershell as the scripting language as it would run on the new infrastructure and could be run on a copy engine server with enough disk space.

I undertook quite a bit of research and trial and error but eventually I have a working script.

This script is not signed so you will either need to sign the script to run it or elevate the privileges with set-Executionpolicy on the system you are going to be using.

The script uses two files the main script file and a csv file with the volume names and copy engine server names.

Below you will find a copy of both. I have also created a git repository that you can find on GitHub if you would like to help make it better

Original PowerShell Script

# Quest Migrator Logfile Parser
$d = Get-Date
$logFileRPath = "D:Logslogs"
$logFilePath = "D:Logs"
$logFileEPath = "D:Logserrors"
$logFileDate = $d.Day - 1
$logFiles = Get-ChildItem $logFileRPath | where { $_.LastWriteTime.Day -eq $logFileDate}
# Add more copy machines to the file below.
$copyServers = Import-Csv $logFilePathqmcopyservers.csv
#Write-Host "LogDate Is" $logFileDate
Write-Host
Write-Host "Copy Servers To Use"
foreach ($server in $copyServers) {
        $copyMachine = $server.CopyMachine
        $copyVolume = $server.Volume
        $sourcePath = "\$copyMachinendsmigLogsFile Migration"
        Write-Host $copyMachine
        #Write-Host $sourcePath
        $sourceLog = Get-ChildItem $sourcePath*Scan.log | where { $_.LastWriteTime.Day -eq $logFileDate}
        #Write-Host $sourceLog.LastWriteTime
        Copy-Item -path $sourceLog -destination $logFileRPath"$copyVolume"-Scan.log
        }
Write-Host
Write-Host "Log Files Selected"
foreach ($logFile in $logFiles) {
        #Write-Host $logFile
        }
Write-Host 
foreach ($objFile in $logFiles) 
    {
        Write-Host "Processing LogFile" $objFile
        gc $logFileRPath$objFile -read 10000 | %{$_} | ? {$_ -like '*Error   *'} | Out-File $logFileEPath"$objFile"-errors.log
        }
									

Original CSV File

Volume, CopyMachine
NWVOLUME1,COPYMACHINE1
NWVOLUME2,COPYMACHINE2
NWVOLUME3,COPYMACHINE3
NWVOLUME4,COPYMACHINE4
NWVOLUME5,COPYMACHINE5