Overlaying exif tags onto viewed image

Being blessed with a pretty rubbish memory I’ve become a big fan of Spaced Repetition Systems over the years. I’ve tried several but have settled down with the reassuringly simple mnemosyne program. mnemosyne has been out for a long time and the currently stable release is 1.2 although there is now a beta version 2.x.

I use images a lot in my mnemosyne deck especially for learning plants and trees. One of the problems I have is that mnemosyne doesn’t allow full screen viewing of the images or zooming in and out, although I think that is addressed in the new beta. Furthermore, the 1.x release doesn’t really support the idea of cramming, or ad-hoc testing, so if I, for example, want to have a browse through all my tree winter ident photos it’s not particularly straightforward.

Most of my photos are pretty well tagged and are currently enjoying a somewhat haphazard journey from f-spot to shotwell, but on the whole they are tagged, with the meta data physically written to the files themselves. For some time I’ve been musing over a quick and easy way of browsing my images based on simple search criteria but with limited success. Often I rename images which has the benefit of making them rapidly findable using the linux locate command but this is not without its problems.

Ideally what I was after was a way of browsing images full-screen with the ability to zoom in and out, skipping forward randomly or sequentially, and, importantly, when I wanted to, easily viewing the exif tags written to the image.

I’m pretty much there, thanks to a shy little program called qiv. I don’t know where qiv has been all my life but it’s pretty much everything I like about a utility. Lean, mean and does what it does well. Using a bit of scripting and linking I get the images I’m interested together and chuck them somewhere, such as /jlinks/imagebatch1. Then I invoke qiv with something like:

qiv --autorotate --maxpect --fullscreen --delay=10 --random --no_statusbar --file /tmp/flist

where /tmp/flist contains the list of files I want to browse through.

This works pretty well. But there are problems.

If I want to see some information about the image I’m viewing, such as its filename, I can press ‘i’. This displays the status bar and if you happen to be using meaningful filenames as I usually do this can be sufficient to find out the name of the plant, bird, whatever, you’re looking at. The problem is that the text in the status line is too small for me. I can read it but it’s a bit of an effort. I searched for a solution to this, assuming that it would be possible to change the text size. If it is I guess it needs a bit of programming as it’s not readily obvious if it’s possible to reconfigure it in any other way.

What I wanted really was a way to superimpose the filename onto the image itself, in nice big letters. As is often the way I spent a lot of hours looking for a solution that was staring me in the face.

qiv allows you to call an external command based on certain keypresses. It took me a while to realise that, far from being quite complicated, it was simply a matter of taking the sample qiv-command script that is shipped with qiv, chucking it into my search path, and then hacking it to bits.

I’m still experimenting and for the moments I have a few options in my qiv-command file that display the information I’m interested in. Most of the time I am interested in the exif keyword tags embedded in the image as they reliably tell me what it is I’m looking at. They are retrieved in no particular order but that doesn’t matter – it does the job. I also found another utility called gnome-osd-client was pretty handy for overlaying the text I wanted.

Here, for example, is the section of my qiv-command file for what happens if there is a keypress of 0.

 title=$(exiftool -t -title "${filename}")
 where=$(exiftool -t -Country -State -City -Location "${filename}")
 rating=$(exiftool -T -Rating "${filename}")
 tags=$(exiftool -t -Subject "${filename}")
 gnome-osd-client "${title} ""${tags}"" (Rating=${rating}) ${where}"

This uses exiftool to get the information I’m interested in, then gnome-osd-client to display it. gnome-osd-client isn’t actually necessary, and other options I have are

  exiftool -t -title -Subject -Country -State -City -Location -model -Rating "$filename"

and even the brief (producing verbose)

  exiftool "${filename}"

One curiosity I discovered during all this is that the -T switch on exiftool seems to cause consternation with gnome-osd-client. Unfortunately I was using this a lot earlier on and assumed that it was something I was doing wrong, whereas it just seems to be a quirky clash between the two utilities. If you use -T anywhere with exiftool in the gnome-osd-client command you will get an error. e.g.

  gnome-osd-client "$(exiftool -T -title -Subject -Country -State -City -Location -model -Rating "${filename}")"

will fail.

Of course, this doesn’t give you any of the spaced repetition logic that you’d get from using a system such as mnemosyne, and perhaps with mnemosyne 2.x this sort of facility might be buiilt in anyway. However it’s quite nice to just specify a search keyword and get a slab of browsable images on the screen.

f-spot, exiftool, exiv2, and exif header weirdness

I decided to revisit an old bug that I logged about a problem I have with f-spot handling of exif headers. Now I’m more confused than ever. I can see what’s happening, but not how or why.

The problem I’m having is that when I upload some photos to gallery websites the handling of exif data doesn’t appear consistent. For example, have a look at this photo. If you look at the right hand side of the screen where the photo title is shown it also shows the caption. It looks like this:

From Zenfolio gallery

Now have a look at the same photo uploaded to picasaweb (the sizes are slightly different – I think picasaweb changed them on the fly when I uploaded via google plus). In this case there is no garbled text and picasaweb has used the exif description field for the photo caption. i.e:

from picasaweb

So what’s happening here? Well, here’s what I think is happening. Picasaweb is extracting the caption from the exif field Description, and Zenfolio is extracting the caption from the field UserComment. Here’s what I get if I run exiftool on the image file (you might need to click on the image to see it properly):

exiftool -UserComment -Description test1.jpg

As you can see the UserComment field contains garbled characters.

A lot of my photos have this and the inconsistencies can cause me headaches. I decided on a brute-force scan of my photo collection to address the issue. I thought, why not just duplicate the Description field into the UserComment field? That way, whatever package or gallery reads the image file will probably get the caption. So here’s the script I used:


#       Process all jpegs and overwrite UserComment field with Description
#       field where they differ.
find /jpegs -type f -iname '*.jpg' | while read fname
        field_usercomment=$(exiftool -UserComment "${fname}" | sed 's/^.*: //')
        field_description=$(exiftool -Description "${fname}" | sed 's/^.*: //')

        if [[ "${field_usercomment}" != "${field_description}" ]] ; then

                echo "$fname"
                echo "User Comment: ${field_usercomment}"
                echo "Description: ${field_description}"

                echo "${fname}"
                exiftool -overwrite_original -UserComment="${field_description}" "${fname}"

After a satisfying hour or two my photos all looked a bit tidier. So now I have a cludge – something that will get round the problem. But I’d really like to treat the cause, and not the symptom. So what’s the cause?

My first guess has always been the camera. I have a knackered old Nikon Coolpix that gives me problems. But I see this problem with other cameras too, my Canon DSLR and compact camera have acquired the garbled field in some pictures too.

So I turn to f-spot, the photo-management software I use in Linux Mint. I checked the exiftool output from a test file, then imported it into f-spot. I then added a tag so that f-spot would write out the exif data (I have f-spot configured to Store Tags and Description inside image files where possible) then had a look at the image file. Here’s what I got:

exiftool output

The output from the first invocation of exiftool is on the jpeg before importing into f-spot. The second example shows the output of exiftool run on the file after it has been imported into f-spot. It looks like f-spot does something to the image that mangles the UserComment field.

At this point I assumed (erroneously I think) that f-spot must be storing the UserComment and Description fields in its sqlite database. Using sqlite3 on the command line I had a look at the structure of the photos table:

dougie@phoenix ~ $ echo '.schema photos' | sqlite3 .config/f-spot/photos.db
    time            INTEGER NOT NULL,
    base_uri        STRING NOT NULL,
    filename        STRING NOT NULL,
    description        TEXT NOT NULL,
    roll_id            INTEGER NOT NULL,
    default_version_id    INTEGER NOT NULL,
    rating            INTEGER NULL
dougie@phoenix ~ $

There’s the description field, but no UserComment field. But it looks to me that f-spot is certainly writing to the UserComment field when it updates an image file. Perhaps it’s dependent on the type of contents of the Description field. It doesn’t, however, make any difference as far as I can tell what the contents of the Description field are. In this example the text ends with an exclamation mark, but I can’t detect any pattern in the text I put in Description fields that might provide a clue.

So if f-spot isn’t storing the UserComment field directly what, when, and why is it writing to the field in the image file? It seems that here we enter the murky world of XMP and I rapidly lose the tenuous grasp I had on the underlying technology up to this point.

Back in 2008 there was a discussion on the f-spot mailing list about how f-spot stores its data in the image file. Re-reading that thread it sounds as if f-spot writes its tags to the XMP metadata and not all utilities are able to read that. This was 2008 mind, and f-spot has jumped a few versions since then.

Out of the same thread came a hint on using exiv2. This is a tool I use a lot, especially for adjusting ‘date taken’ timestamps, but hadn’t found it any use for extracting the UserComment field. It turns out I hadn’t RTFM properly and it is possible if you tell exiv2 to look at XMP data. This leads to some interesting output. Here is the example file again, using both exiv2 and exiftool to examine the field contents:

dougie@phoenix ~ $
dougie@phoenix ~ $ exiftool -UserComment -Description /jpegs/2011/07/17/test1.jpg
User Comment                    : 桷⁹畲污湯⁧桴⁥敢捡⁨桷湥礠畯挠湡挠瑵琠敨挠牯敮ⅲ††††††††††††
Description                     : why run along the beach when you can cut the corner!
dougie@phoenix ~ $
dougie@phoenix ~ $ exiv2 -p a /jpegs/2011/07/17/test1.jpg
Exif.Image.ImageDescription                  Ascii      77  why run along the beach when you can cut the corner!                        
Exif.Image.Software                          Ascii      21  f-spot version 0.8.0
Exif.Image.ExifTag                           Long        1  162
Exif.Photo.DateTimeOriginal                  Ascii      20  2011:07:17 09:37:13
Exif.Photo.UserComment                       Undefined  84  why run along the beach when you can cut the corner!                        
Exif.Image.GPSTag                            Long        1  296
Exif.GPSInfo.GPSVersionID                    Byte        4
Exif.GPSInfo.GPSLatitudeRef                  Ascii       2  North
Exif.GPSInfo.GPSLatitude                     Rational    3  55deg 32' 25.410"
Exif.GPSInfo.GPSLongitudeRef                 Ascii       2  West
Exif.GPSInfo.GPSLongitude                    Rational    3  1deg 37' 58.960"
Exif.GPSInfo.GPSAltitudeRef                  Byte        1  Above sea level
Exif.GPSInfo.GPSAltitude                     Rational    1  9.5 m
Exif.GPSInfo.GPSTimeStamp                    SRational   3  09:37:13
Exif.GPSInfo.GPSMapDatum                     Ascii       7  WGS-84
Exif.GPSInfo.GPSDateStamp                    Ascii      11  2011:07:17
Xmp.xmp.CreateDate                           XmpText    19  17/07/2011 09:37:13
Xmp.xmp.Rating                               XmpText     1  0
Xmp.xmp.CreatorTool                          XmpText    20  f-spot version 0.8.0
Xmp.dc.description                           LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
Xmp.dc.subject                               XmpBag      2  Dougie Nisbet, Coastal Run - 2011
Xmp.exif.UserComment                         LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
dougie@phoenix ~ $

Interestingly, exiv2 is displaying the UserComment data in clear text, in two places, which I’m guessing must be something to do with the different types of metadata – XMP and EXIF.

So does this mean this has nothing to do with f-spot at all? If exiv2 is able to read the text, perhaps it’s something to do with the utility used to examine the image file? If I have a look at the file using geeqie and open up the EXIF window geeqie seems to understand the text too:

geeqie EXIF window

Perhaps this is an unrelated setting I need to have a look at. Language, Locale, who knows. The issue seems to crop up in different places so it may very well be unrelated to f-spot. For the time being I have a fairly straightforward work-around, I just need to remember to run a script to copy the description field into the UserComment field for any images I update in f-spot. It’s a bit of a nuisance but fairly easy to automate, and a lot less time-consuming than trying to get to the bottom of the mystery!

How to deal with exif stuff in Coolpix images that f-spot doesn’t like

I have a battered and knackered old Nikon Coolpix S600 camera. The zoom no longer works and it can be quite cranky. It’s not surprising as it has a tough time. It often gets carried on fell races and road races in, er, ‘hostile’ conditions. Its compact size and ‘sports’ setting makes it handy for firing and forgetting. I have a Canon compact that gets similar treatment but on the whole the Canon can’t take the punishment that the Nikon can.

The problem I have with the Nikon is that there seems to be issues with the exif data. Whether it’s the camera, the camera’s firmware, or the software I use on the PC to process images, I don’t know. The upshot is the same, though, I have problems writing exif data back to the jpegs from the photo management software f-spot.

I always start f-spot on the command line in debug mode and save the output to a file. This way I can get a better idea of what’s going on. i.e.

f-spot --debug 2>&1 | tee /home/dougie/f-spot.out

(there’s a wrapper script involved too that takes a backup of the old database first).

Once I have the images in f-spot I start applying tags. I have f-spot configured so that it writes tags to the image file itself as this gives me greater flexibility if I start copying photos around the place or into other packages. This setting is in f-spot under Edit / Preferences. i.e.

write image data to file

So now I have my photos and I’ve applied some tags to them. But if I have a look at the terminal session where I started f-spot, I can see there are problems. The messages will typically look something like this:

[4 Debug 21:39:15.516] open uri = file:///jpegs/2011/06/11/DSCN5936.JPG
[4 Debug 21:39:15.940] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5934.JPG
[4 Debug 21:39:15.942] open uri = file:///jpegs/2011/06/11/DSCN5934.JPG
[14 Debug 21:39:15.968] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5928.JPG)...
[14 Warn 21:39:15.970] Metadata of file file:///jpegs/2011/06/11/DSCN5928.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.359] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5931.JPG
[4 Debug 21:39:16.360] open uri = file:///jpegs/2011/06/11/DSCN5931.JPG
[14 Debug 21:39:16.569] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5924.JPG)...
[14 Warn 21:39:16.571] Metadata of file file:///jpegs/2011/06/11/DSCN5924.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.786] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5928.JPG
[4 Debug 21:39:16.787] open uri = file:///jpegs/2011/06/11/DSCN5928.JPG
[14 Debug 21:39:17.153] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5914.JPG)...
[14 Warn 21:39:17.154] Metadata of file file:///jpegs/2011/06/11/DSCN5914.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.

The problem for me is two-fold

  1. f-spot has a problem with the metadata in the jpeg.
  2. It’s creating extra .xmp files that I don’t want.

I’ve tried several avenues to resolve this and I thought the following would do the trick:

jhead -purejpg

This uses the Linux utility jhead to rewrite the image header with a standard one. Unfortunately the problem persists. So I took a more brutal approach. I used the linux utility exiftool to obliterate all meta data from the image.

At first I’d copy all the images from the memory card into a directory and then run exiftool on all the files there. i.e.

exiftool -all= *

(Note that syntax; there’s a space between the equal sign and asterix)

This is fine but unfortunately it pretty much destroys any useful or interesting extra information. Most I can live without, except the date and time that the photo was taken. It automatically inherits the file modification time instead.

The solution is to first read the jpegs into f-spot, then, while f-spot is still running, go to the directory where the images have been imported to, then run the exiftool command there. In my setup, I have f-spot configured to copy all imported jpegs to /jpegs where they are automatically arranged in directories according to date and time. So the procedure for me is as follows:

  1. Import jpegs into f-spot
  2. With f-spot running, change to destination directory where jpegs were imported to
  3. Run exiftool -all= *
  4. carry out tagging operations in f-spot

The advantage of this is that despite obliterating the meta data in the files using exiftool, f-spot still knows the data and times of these photos in its own database. So the next time you write to them with f-spot, by adding a tag for instance, they will get the date and time from f-spot. You lose pretty much everything else, which is a shame, but the date and time are what’s most important for me.

Typically what I tend to do is read all the photos from the memory card into a temporary folder, e.g. ~dougie/in, where I can carry out a quick pass using something like geeqie to delete any truly terrible photos. Then I import the photos into f-spot. Once that’s done I change to the destination directory where f-spot has copied the photos and run exifool. e.g.

dougie@phoenix /jpegs/2011/06/11 $ exiftool -all= *
60 image files updated
dougie@phoenix /jpegs/2011/06/11 $

Note the syntax for the exiftool command. That’s a space after the equal sign and before the asterix. It’s a powerful command so it’s best to check the man page first to ensure that you know what it’s going to do.