renaming jpegs based on f-spot tags

f-spot does not have an option for renaming files, so if you want to rename your images you need to export them from f-spot, rename them, then re-import. It sounds a hassle but actually it’s quite easy to streamline, especially since it’s possible just to drag and drop the images you’re interested in. The Drag’n’Drop ability also gets around the problem where really long filenames can’t be exported because of their filename length.

I rename my jpegs based on the the tags I’ve created in f-spot. I don’t know how f-spot chooses the order to write the tags to the EXIF fields, and I don’t much care. It’s sufficiently robust for me, and it means I can rapidly locate images by using the ‘locate’ command.

Here’s the script I use:

#       Script: rename_fspot_jpegs
#       Notes:  Renames jpegs based on their embedded tags
#               Modify the BASEDIR variable as required

scriptname="$(basename $0)"

# modify this section as required for your setup. 
# you should be able to just change the BASEDIR to your own home directory,
# and export your jpegs from f-spot into jpegs_in.

# Change this to whatever you want to separate the tags in your filenames

mkdir -p ${OUTDIR} || exit 2
echo "STARTING script: ${scriptname}"

# process jpegs and extract tags
echo "Process file list for renaming ..."
cd ${INDIR} || exit 2

ls | while read fname 
# END each filename with datestamp
        echo "Processing ${fname}"
        exif_created=`exiftool -overwrite_original -d %c -CreateDate "${fname}" | sed 's///-/g' | sed 's/^.*: //' | sed 's/:/-/g'`
        if [[ -z ${exif_created} ]] ; then
# Create date not found. Using original
                exif_created=`exiftool -overwrite_original -d %c -DateTimeOriginal "${fname}" | sed 's/^.*: //' | sed 's/:/-/g'`

#       Copy the contents of Description field into UserComment field.
#       UserComment field often ends up garbled and appears to be used
#       by some gallery sites - e.g. zenfolio.
        exif_description=`exiftool -overwrite_original -Description "${fname}" | sed 's/^.*: //'`
        exiftool -overwrite_original -UserComment="$exif_description" "${fname}"
# f-spot writes its tags to the Subject field.

        exif_tags=`exiftool -overwrite_original -Subject "${fname}" | sed 's/^.*: //' | sed "s/,/ ${TAGSEP}/g"`
        echo "Tags: ${exif_tags}"
        if [[ ! -z "${exif_tags}" ]] ; then
                newfname="${exif_tags} ${TAGSEP} ${exif_created}"

# we need the --backup=numbered option in case we have duplicate filenames
        cp "${fname}" "${OUTDIR}/${newfname}" --backup=numbered

# now we have a directory full of filenames with no .jpg suffix
cd ${OUTDIR} || exit 2
ls | while read fname
        mv "${fname}" "${fname}".jpg

# EOF: rename_fspot_jpegs

f-spot, exiftool, exiv2, and exif header weirdness

I decided to revisit an old bug that I logged about a problem I have with f-spot handling of exif headers. Now I’m more confused than ever. I can see what’s happening, but not how or why.

The problem I’m having is that when I upload some photos to gallery websites the handling of exif data doesn’t appear consistent. For example, have a look at this photo. If you look at the right hand side of the screen where the photo title is shown it also shows the caption. It looks like this:

From Zenfolio gallery

Now have a look at the same photo uploaded to picasaweb (the sizes are slightly different – I think picasaweb changed them on the fly when I uploaded via google plus). In this case there is no garbled text and picasaweb has used the exif description field for the photo caption. i.e:

from picasaweb

So what’s happening here? Well, here’s what I think is happening. Picasaweb is extracting the caption from the exif field Description, and Zenfolio is extracting the caption from the field UserComment. Here’s what I get if I run exiftool on the image file (you might need to click on the image to see it properly):

exiftool -UserComment -Description test1.jpg

As you can see the UserComment field contains garbled characters.

A lot of my photos have this and the inconsistencies can cause me headaches. I decided on a brute-force scan of my photo collection to address the issue. I thought, why not just duplicate the Description field into the UserComment field? That way, whatever package or gallery reads the image file will probably get the caption. So here’s the script I used:


#       Process all jpegs and overwrite UserComment field with Description
#       field where they differ.
find /jpegs -type f -iname '*.jpg' | while read fname
        field_usercomment=$(exiftool -UserComment "${fname}" | sed 's/^.*: //')
        field_description=$(exiftool -Description "${fname}" | sed 's/^.*: //')

        if [[ "${field_usercomment}" != "${field_description}" ]] ; then

                echo "$fname"
                echo "User Comment: ${field_usercomment}"
                echo "Description: ${field_description}"

                echo "${fname}"
                exiftool -overwrite_original -UserComment="${field_description}" "${fname}"

After a satisfying hour or two my photos all looked a bit tidier. So now I have a cludge – something that will get round the problem. But I’d really like to treat the cause, and not the symptom. So what’s the cause?

My first guess has always been the camera. I have a knackered old Nikon Coolpix that gives me problems. But I see this problem with other cameras too, my Canon DSLR and compact camera have acquired the garbled field in some pictures too.

So I turn to f-spot, the photo-management software I use in Linux Mint. I checked the exiftool output from a test file, then imported it into f-spot. I then added a tag so that f-spot would write out the exif data (I have f-spot configured to Store Tags and Description inside image files where possible) then had a look at the image file. Here’s what I got:

exiftool output

The output from the first invocation of exiftool is on the jpeg before importing into f-spot. The second example shows the output of exiftool run on the file after it has been imported into f-spot. It looks like f-spot does something to the image that mangles the UserComment field.

At this point I assumed (erroneously I think) that f-spot must be storing the UserComment and Description fields in its sqlite database. Using sqlite3 on the command line I had a look at the structure of the photos table:

dougie@phoenix ~ $ echo '.schema photos' | sqlite3 .config/f-spot/photos.db
    time            INTEGER NOT NULL,
    base_uri        STRING NOT NULL,
    filename        STRING NOT NULL,
    description        TEXT NOT NULL,
    roll_id            INTEGER NOT NULL,
    default_version_id    INTEGER NOT NULL,
    rating            INTEGER NULL
dougie@phoenix ~ $

There’s the description field, but no UserComment field. But it looks to me that f-spot is certainly writing to the UserComment field when it updates an image file. Perhaps it’s dependent on the type of contents of the Description field. It doesn’t, however, make any difference as far as I can tell what the contents of the Description field are. In this example the text ends with an exclamation mark, but I can’t detect any pattern in the text I put in Description fields that might provide a clue.

So if f-spot isn’t storing the UserComment field directly what, when, and why is it writing to the field in the image file? It seems that here we enter the murky world of XMP and I rapidly lose the tenuous grasp I had on the underlying technology up to this point.

Back in 2008 there was a discussion on the f-spot mailing list about how f-spot stores its data in the image file. Re-reading that thread it sounds as if f-spot writes its tags to the XMP metadata and not all utilities are able to read that. This was 2008 mind, and f-spot has jumped a few versions since then.

Out of the same thread came a hint on using exiv2. This is a tool I use a lot, especially for adjusting ‘date taken’ timestamps, but hadn’t found it any use for extracting the UserComment field. It turns out I hadn’t RTFM properly and it is possible if you tell exiv2 to look at XMP data. This leads to some interesting output. Here is the example file again, using both exiv2 and exiftool to examine the field contents:

dougie@phoenix ~ $
dougie@phoenix ~ $ exiftool -UserComment -Description /jpegs/2011/07/17/test1.jpg
User Comment                    : 桷⁹畲污湯⁧桴⁥敢捡⁨桷湥礠畯挠湡挠瑵琠敨挠牯敮ⅲ††††††††††††
Description                     : why run along the beach when you can cut the corner!
dougie@phoenix ~ $
dougie@phoenix ~ $ exiv2 -p a /jpegs/2011/07/17/test1.jpg
Exif.Image.ImageDescription                  Ascii      77  why run along the beach when you can cut the corner!                        
Exif.Image.Software                          Ascii      21  f-spot version 0.8.0
Exif.Image.ExifTag                           Long        1  162
Exif.Photo.DateTimeOriginal                  Ascii      20  2011:07:17 09:37:13
Exif.Photo.UserComment                       Undefined  84  why run along the beach when you can cut the corner!                        
Exif.Image.GPSTag                            Long        1  296
Exif.GPSInfo.GPSVersionID                    Byte        4
Exif.GPSInfo.GPSLatitudeRef                  Ascii       2  North
Exif.GPSInfo.GPSLatitude                     Rational    3  55deg 32' 25.410"
Exif.GPSInfo.GPSLongitudeRef                 Ascii       2  West
Exif.GPSInfo.GPSLongitude                    Rational    3  1deg 37' 58.960"
Exif.GPSInfo.GPSAltitudeRef                  Byte        1  Above sea level
Exif.GPSInfo.GPSAltitude                     Rational    1  9.5 m
Exif.GPSInfo.GPSTimeStamp                    SRational   3  09:37:13
Exif.GPSInfo.GPSMapDatum                     Ascii       7  WGS-84
Exif.GPSInfo.GPSDateStamp                    Ascii      11  2011:07:17
Xmp.xmp.CreateDate                           XmpText    19  17/07/2011 09:37:13
Xmp.xmp.Rating                               XmpText     1  0
Xmp.xmp.CreatorTool                          XmpText    20  f-spot version 0.8.0
Xmp.dc.description                           LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
Xmp.dc.subject                               XmpBag      2  Dougie Nisbet, Coastal Run - 2011
Xmp.exif.UserComment                         LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
dougie@phoenix ~ $

Interestingly, exiv2 is displaying the UserComment data in clear text, in two places, which I’m guessing must be something to do with the different types of metadata – XMP and EXIF.

So does this mean this has nothing to do with f-spot at all? If exiv2 is able to read the text, perhaps it’s something to do with the utility used to examine the image file? If I have a look at the file using geeqie and open up the EXIF window geeqie seems to understand the text too:

geeqie EXIF window

Perhaps this is an unrelated setting I need to have a look at. Language, Locale, who knows. The issue seems to crop up in different places so it may very well be unrelated to f-spot. For the time being I have a fairly straightforward work-around, I just need to remember to run a script to copy the description field into the UserComment field for any images I update in f-spot. It’s a bit of a nuisance but fairly easy to automate, and a lot less time-consuming than trying to get to the bottom of the mystery!

How to deal with exif stuff in Coolpix images that f-spot doesn’t like

I have a battered and knackered old Nikon Coolpix S600 camera. The zoom no longer works and it can be quite cranky. It’s not surprising as it has a tough time. It often gets carried on fell races and road races in, er, ‘hostile’ conditions. Its compact size and ‘sports’ setting makes it handy for firing and forgetting. I have a Canon compact that gets similar treatment but on the whole the Canon can’t take the punishment that the Nikon can.

The problem I have with the Nikon is that there seems to be issues with the exif data. Whether it’s the camera, the camera’s firmware, or the software I use on the PC to process images, I don’t know. The upshot is the same, though, I have problems writing exif data back to the jpegs from the photo management software f-spot.

I always start f-spot on the command line in debug mode and save the output to a file. This way I can get a better idea of what’s going on. i.e.

f-spot --debug 2>&1 | tee /home/dougie/f-spot.out

(there’s a wrapper script involved too that takes a backup of the old database first).

Once I have the images in f-spot I start applying tags. I have f-spot configured so that it writes tags to the image file itself as this gives me greater flexibility if I start copying photos around the place or into other packages. This setting is in f-spot under Edit / Preferences. i.e.

write image data to file

So now I have my photos and I’ve applied some tags to them. But if I have a look at the terminal session where I started f-spot, I can see there are problems. The messages will typically look something like this:

[4 Debug 21:39:15.516] open uri = file:///jpegs/2011/06/11/DSCN5936.JPG
[4 Debug 21:39:15.940] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5934.JPG
[4 Debug 21:39:15.942] open uri = file:///jpegs/2011/06/11/DSCN5934.JPG
[14 Debug 21:39:15.968] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5928.JPG)...
[14 Warn 21:39:15.970] Metadata of file file:///jpegs/2011/06/11/DSCN5928.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.359] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5931.JPG
[4 Debug 21:39:16.360] open uri = file:///jpegs/2011/06/11/DSCN5931.JPG
[14 Debug 21:39:16.569] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5924.JPG)...
[14 Warn 21:39:16.571] Metadata of file file:///jpegs/2011/06/11/DSCN5924.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.786] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5928.JPG
[4 Debug 21:39:16.787] open uri = file:///jpegs/2011/06/11/DSCN5928.JPG
[14 Debug 21:39:17.153] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5914.JPG)...
[14 Warn 21:39:17.154] Metadata of file file:///jpegs/2011/06/11/DSCN5914.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.

The problem for me is two-fold

  1. f-spot has a problem with the metadata in the jpeg.
  2. It’s creating extra .xmp files that I don’t want.

I’ve tried several avenues to resolve this and I thought the following would do the trick:

jhead -purejpg

This uses the Linux utility jhead to rewrite the image header with a standard one. Unfortunately the problem persists. So I took a more brutal approach. I used the linux utility exiftool to obliterate all meta data from the image.

At first I’d copy all the images from the memory card into a directory and then run exiftool on all the files there. i.e.

exiftool -all= *

(Note that syntax; there’s a space between the equal sign and asterix)

This is fine but unfortunately it pretty much destroys any useful or interesting extra information. Most I can live without, except the date and time that the photo was taken. It automatically inherits the file modification time instead.

The solution is to first read the jpegs into f-spot, then, while f-spot is still running, go to the directory where the images have been imported to, then run the exiftool command there. In my setup, I have f-spot configured to copy all imported jpegs to /jpegs where they are automatically arranged in directories according to date and time. So the procedure for me is as follows:

  1. Import jpegs into f-spot
  2. With f-spot running, change to destination directory where jpegs were imported to
  3. Run exiftool -all= *
  4. carry out tagging operations in f-spot

The advantage of this is that despite obliterating the meta data in the files using exiftool, f-spot still knows the data and times of these photos in its own database. So the next time you write to them with f-spot, by adding a tag for instance, they will get the date and time from f-spot. You lose pretty much everything else, which is a shame, but the date and time are what’s most important for me.

Typically what I tend to do is read all the photos from the memory card into a temporary folder, e.g. ~dougie/in, where I can carry out a quick pass using something like geeqie to delete any truly terrible photos. Then I import the photos into f-spot. Once that’s done I change to the destination directory where f-spot has copied the photos and run exifool. e.g.

dougie@phoenix /jpegs/2011/06/11 $ exiftool -all= *
60 image files updated
dougie@phoenix /jpegs/2011/06/11 $

Note the syntax for the exiftool command. That’s a space after the equal sign and before the asterix. It’s a powerful command so it’s best to check the man page first to ensure that you know what it’s going to do.