f-spot, exiftool, exiv2, and exif header weirdness

I decided to revisit an old bug that I logged about a problem I have with f-spot handling of exif headers. Now I’m more confused than ever. I can see what’s happening, but not how or why. The problem I’m having is that when I upload some photos to gallery websites the handling of exif … Continue reading “f-spot, exiftool, exiv2, and exif header weirdness”

I decided to revisit an old bug that I logged about a problem I have with f-spot handling of exif headers. Now I’m more confused than ever. I can see what’s happening, but not how or why.

The problem I’m having is that when I upload some photos to gallery websites the handling of exif data doesn’t appear consistent. For example, have a look at this photo. If you look at the right hand side of the screen where the photo title is shown it also shows the caption. It looks like this:

From Zenfolio gallery

Now have a look at the same photo uploaded to picasaweb (the sizes are slightly different – I think picasaweb changed them on the fly when I uploaded via google plus). In this case there is no garbled text and picasaweb has used the exif description field for the photo caption. i.e:

from picasaweb

So what’s happening here? Well, here’s what I think is happening. Picasaweb is extracting the caption from the exif field Description, and Zenfolio is extracting the caption from the field UserComment. Here’s what I get if I run exiftool on the image file (you might need to click on the image to see it properly):

exiftool -UserComment -Description test1.jpg

As you can see the UserComment field contains garbled characters.

A lot of my photos have this and the inconsistencies can cause me headaches. I decided on a brute-force scan of my photo collection to address the issue. I thought, why not just duplicate the Description field into the UserComment field? That way, whatever package or gallery reads the image file will probably get the caption. So here’s the script I used:

#!/bin/bash

#
#       Process all jpegs and overwrite UserComment field with Description
#       field where they differ.
#
find /jpegs -type f -iname '*.jpg' | while read fname
do
        field_usercomment=$(exiftool -UserComment "${fname}" | sed 's/^.*: //')
        field_description=$(exiftool -Description "${fname}" | sed 's/^.*: //')

        if [[ "${field_usercomment}" != "${field_description}" ]] ; then

                echo "$fname"
                echo "User Comment: ${field_usercomment}"
                echo "Description: ${field_description}"
                echo

                echo "${fname}"
                exiftool -overwrite_original -UserComment="${field_description}" "${fname}"
                echo
        fi
done

After a satisfying hour or two my photos all looked a bit tidier. So now I have a cludge – something that will get round the problem. But I’d really like to treat the cause, and not the symptom. So what’s the cause?

My first guess has always been the camera. I have a knackered old Nikon Coolpix that gives me problems. But I see this problem with other cameras too, my Canon DSLR and compact camera have acquired the garbled field in some pictures too.

So I turn to f-spot, the photo-management software I use in Linux Mint. I checked the exiftool output from a test file, then imported it into f-spot. I then added a tag so that f-spot would write out the exif data (I have f-spot configured to Store Tags and Description inside image files where possible) then had a look at the image file. Here’s what I got:

exiftool output

The output from the first invocation of exiftool is on the jpeg before importing into f-spot. The second example shows the output of exiftool run on the file after it has been imported into f-spot. It looks like f-spot does something to the image that mangles the UserComment field.

At this point I assumed (erroneously I think) that f-spot must be storing the UserComment and Description fields in its sqlite database. Using sqlite3 on the command line I had a look at the structure of the photos table:

dougie@phoenix ~ $ echo '.schema photos' | sqlite3 .config/f-spot/photos.db
CREATE TABLE photos (
    id            INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    time            INTEGER NOT NULL,
    base_uri        STRING NOT NULL,
    filename        STRING NOT NULL,
    description        TEXT NOT NULL,
    roll_id            INTEGER NOT NULL,
    default_version_id    INTEGER NOT NULL,
    rating            INTEGER NULL
);
dougie@phoenix ~ $

There’s the description field, but no UserComment field. But it looks to me that f-spot is certainly writing to the UserComment field when it updates an image file. Perhaps it’s dependent on the type of contents of the Description field. It doesn’t, however, make any difference as far as I can tell what the contents of the Description field are. In this example the text ends with an exclamation mark, but I can’t detect any pattern in the text I put in Description fields that might provide a clue.

So if f-spot isn’t storing the UserComment field directly what, when, and why is it writing to the field in the image file? It seems that here we enter the murky world of XMP and I rapidly lose the tenuous grasp I had on the underlying technology up to this point.

Back in 2008 there was a discussion on the f-spot mailing list about how f-spot stores its data in the image file. Re-reading that thread it sounds as if f-spot writes its tags to the XMP metadata and not all utilities are able to read that. This was 2008 mind, and f-spot has jumped a few versions since then.

Out of the same thread came a hint on using exiv2. This is a tool I use a lot, especially for adjusting ‘date taken’ timestamps, but hadn’t found it any use for extracting the UserComment field. It turns out I hadn’t RTFM properly and it is possible if you tell exiv2 to look at XMP data. This leads to some interesting output. Here is the example file again, using both exiv2 and exiftool to examine the field contents:

dougie@phoenix ~ $
dougie@phoenix ~ $ exiftool -UserComment -Description /jpegs/2011/07/17/test1.jpg
User Comment                    : 桷⁹畲污湯⁧桴⁥敢捡⁨桷湥礠畯挠湡挠瑵琠敨挠牯敮ⅲ††††††††††††
Description                     : why run along the beach when you can cut the corner!
dougie@phoenix ~ $
dougie@phoenix ~ $ exiv2 -p a /jpegs/2011/07/17/test1.jpg
Exif.Image.ImageDescription                  Ascii      77  why run along the beach when you can cut the corner!                        
Exif.Image.Software                          Ascii      21  f-spot version 0.8.0
Exif.Image.ExifTag                           Long        1  162
Exif.Photo.DateTimeOriginal                  Ascii      20  2011:07:17 09:37:13
Exif.Photo.UserComment                       Undefined  84  why run along the beach when you can cut the corner!                        
Exif.Image.GPSTag                            Long        1  296
Exif.GPSInfo.GPSVersionID                    Byte        4  2.0.0.0
Exif.GPSInfo.GPSLatitudeRef                  Ascii       2  North
Exif.GPSInfo.GPSLatitude                     Rational    3  55deg 32' 25.410"
Exif.GPSInfo.GPSLongitudeRef                 Ascii       2  West
Exif.GPSInfo.GPSLongitude                    Rational    3  1deg 37' 58.960"
Exif.GPSInfo.GPSAltitudeRef                  Byte        1  Above sea level
Exif.GPSInfo.GPSAltitude                     Rational    1  9.5 m
Exif.GPSInfo.GPSTimeStamp                    SRational   3  09:37:13
Exif.GPSInfo.GPSMapDatum                     Ascii       7  WGS-84
Exif.GPSInfo.GPSDateStamp                    Ascii      11  2011:07:17
Xmp.xmp.CreateDate                           XmpText    19  17/07/2011 09:37:13
Xmp.xmp.Rating                               XmpText     1  0
Xmp.xmp.CreatorTool                          XmpText    20  f-spot version 0.8.0
Xmp.dc.description                           LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
Xmp.dc.subject                               XmpBag      2  Dougie Nisbet, Coastal Run - 2011
Xmp.exif.UserComment                         LangAlt     1  lang="x-default" why run along the beach when you can cut the corner!                        
dougie@phoenix ~ $

Interestingly, exiv2 is displaying the UserComment data in clear text, in two places, which I’m guessing must be something to do with the different types of metadata – XMP and EXIF.

So does this mean this has nothing to do with f-spot at all? If exiv2 is able to read the text, perhaps it’s something to do with the utility used to examine the image file? If I have a look at the file using geeqie and open up the EXIF window geeqie seems to understand the text too:

geeqie EXIF window

Perhaps this is an unrelated setting I need to have a look at. Language, Locale, who knows. The issue seems to crop up in different places so it may very well be unrelated to f-spot. For the time being I have a fairly straightforward work-around, I just need to remember to run a script to copy the description field into the UserComment field for any images I update in f-spot. It’s a bit of a nuisance but fairly easy to automate, and a lot less time-consuming than trying to get to the bottom of the mystery!

The Katsura Tree

As the title of this blog might suggest, I have a particular fondness for the Katsura tree (Cercidiphyllum japonicum). I have three in the garden, all youngsters. One is getting on for 2 or 3 metres tall, has been in for a few years (planted Nov 2007), and is looking good. Around the corner in … Continue reading “The Katsura Tree”

As the title of this blog might suggest, I have a particular fondness for the Katsura tree (Cercidiphyllum japonicum). I have three in the garden, all youngsters. One is getting on for 2 or 3 metres tall, has been in for a few years (planted Nov 2007), and is looking good.

Cercidiphyllum japonicum
Healthy tree

Around the corner in the front garden there are a couple of tiny trees that I snapped up for a fiver each from Dawyck a couple of years ago. They are about 3 or 4 metres apart and have been absolutely fine since the day they were put in. I occasionally mulch with grass clippings or clear encroaching roses or beech from them, but all, in all, they’ve seemed happy.

But something has happened recently. The one on the right (click for a bigger picture) looks pretty healthy. Not a big tree but happy enough.

 

And then I walk about 3 metres along the flower bed to the next one. And it doesn’t look well at all.

Cercidiphyllum japonicum
Not a healthy tree

The mush you can see around the base is old grass clippings (no fertiliser, or weed and feed), and it is kept away from the base of the sapling. Same for the healthy one.

A closer look at the leaves shows them brown and mostly dead.

So, what is wrong with this picture? Why is one alive and apparently healthy, and one is dead, or nearly so? They are only a few metres apart and have no visible differences in light, soil or moisture.

spell checker ignores bits in Word 2010

Well that was weird. I am used to Microsoft Word automatically spell checking as I go. And then I realised it had stopped doing it. Then I realised it was doing it in bits of my document, and not others. So I copied and pasted and experimented and cursed. Then I switched on the option that shows all the hidden codes and it all looked ok. I couldn’t see why it was ignoring just sections of the document. e.g.

Word spell checker ignoring some words
The same words ignored later on line

What I didn’t realise is that Word appears to allow you to switch off spell-checking on certain parts of your document. But it isn’t obvious which bits. I found that when I dropped the mouse about in the unchecked bit then selected: Review / Language / Set Proofing Language it was configured to ignore that section. Or Word. Letter. Character. Dunno how or why it decided to ignore that bit. I don’t know where the not-checking section begins and ends, or how to find out. i.e.

Configuring Word to ignore spell check
Configured NOT to check

So I unchecked Do not check spelling and grammar and everything is hunky dory. Mystery solved.

Well, not really. Why did that option get checked in the first place? I didn’t do it.

I have a theory that it’s something to do with something that happened a couple of days ago. Out of the blue Word popped up with a glittering selection of all my spelling mistakes and dictionary additions for the past few months, and asked me if I would mind awfully if it sent them anonymously to Microsoft. I said I did mind, politely declined the request, and asked not to be asked again. Perhaps it was something to do with that. Perhaps Bill is cross with me.

new kids on the (fat) blocks

As well as the occasional woodpecker, bullfinch and jackdaw, the bird feeders are buzzing with birds, old and new. A closer look shows the gape of the young blue tits amongst the grown-ups.

As well as the occasional woodpecker, bullfinch and jackdaw, the bird feeders are buzzing with birds, old and new. A closer look shows the gape of the young blue tits amongst the grown-ups.

Suppertime
a mixture of birds, old and new

new kids on the (fat) blocks

As well as the occasional woodpecker, bullfinch and jackdaw, the bird feeders are buzzing with birds, old and new. A closer look shows the gape of the young blue tits amongst the grown-ups.

As well as the occasional woodpecker, bullfinch and jackdaw, the bird feeders are buzzing with birds, old and new. A closer look shows the gape of the young blue tits amongst the grown-ups.

Suppertime
a mixture of birds, old and new

I can feed myself

After last night’s observation of an adult male Greater Spotted feeding a juvenile I decided to set up the Big Camera on the tripod with the remote cable release and wait, hopeful to see the same behaviour again. It’s easy to forget the importance of depth-of-field with a fast shutter speed, and I got literally … Continue reading “I can feed myself”

After last night’s observation of an adult male Greater Spotted feeding a juvenile I decided to set up the Big Camera on the tripod with the remote cable release and wait, hopeful to see the same behaviour again. It’s easy to forget the importance of depth-of-field with a fast shutter speed, and I got literally hundreds of photos of a crystal-sharp-focussed bird feeder, and an ever-so-slightly fuzzy woodpecker.

But amongst the fuzzyness there were a couple of nice sharp shots. Of junior. It looks like he (or she) is quite capable of looking after himself.

father and son (or daughter)

There’s nothing like getting a clear, sharp, vibrant photograph of a dazzling bird like the woodpecker to make your day. Sure enough, this is nothing like it. Lots of my favourite photos, however, lack technical merit. So when I glanced out the kitchen window yesterday and saw an adult male greater spotted woodpecker at the … Continue reading “father and son (or daughter)”

There’s nothing like getting a clear, sharp, vibrant photograph of a dazzling bird like the woodpecker to make your day. Sure enough, this is nothing like it. Lots of my favourite photos, however, lack technical merit.

father and son
father and son (or daughter) at the feeder

So when I glanced out the kitchen window yesterday and saw an adult male greater spotted woodpecker at the fat feeder along with its youngster my heart skipped a beat. It’s a great sight and I crept to the window with the compact camera and shot a couple of photos through the grime.

Here you can see the juvenile woodpecker sitting on the top of the feeder, identifiable by the red crown. This redtop will fade away in the months ahead but for the moment it makes the juvenile birds stand out clearly. The adult, with the red spot on the nape of his neck is an adult male. The female doesn’t have this mark. The presence or absence of these red splodges make the sex of greater spotted woodpecker pretty easy to identify.

Just before they flew of I managed to capture a grainy moment where the adult was feeding the youngster a blob of fat from the feeder. It won’t win any photo competitions but I love it!

Adult male woodpecker feeding youngster
An adult male Greater Spotted Woodpecker feeds a youngster

 

windows / linux filename compatibility

It would be quite nice to be able to backup my photo collection to a USB drive in VFAT format. Unfortunately an indeterminate number of my photos have long filenames, although none have the dreaded microsoft hated colon(:). 250 characters seems to be the Windows7 limit. I could write a script to parse my /jpegs … Continue reading “windows / linux filename compatibility”

It would be quite nice to be able to backup my photo collection to a USB drive in VFAT format. Unfortunately an indeterminate number of my photos have long filenames, although none have the dreaded microsoft hated colon(:). 250 characters seems to be the Windows7 limit. I could write a script to parse my /jpegs directory and report on all filenames >250 characters in length as well as any containing colons. This would give me a report. If it’s not too long it might be easiest just to go through them individually with f-spot and re-tag and rename them.

How to deal with exif stuff in Coolpix images that f-spot doesn’t like

I have a battered and knackered old Nikon Coolpix S600 camera. The zoom no longer works and it can be quite cranky. It’s not surprising as it has a tough time. It often gets carried on fell races and road races in, er, ‘hostile’ conditions. Its compact size and ‘sports’ setting makes it handy for … Continue reading “How to deal with exif stuff in Coolpix images that f-spot doesn’t like”

I have a battered and knackered old Nikon Coolpix S600 camera. The zoom no longer works and it can be quite cranky. It’s not surprising as it has a tough time. It often gets carried on fell races and road races in, er, ‘hostile’ conditions. Its compact size and ‘sports’ setting makes it handy for firing and forgetting. I have a Canon compact that gets similar treatment but on the whole the Canon can’t take the punishment that the Nikon can.

The problem I have with the Nikon is that there seems to be issues with the exif data. Whether it’s the camera, the camera’s firmware, or the software I use on the PC to process images, I don’t know. The upshot is the same, though, I have problems writing exif data back to the jpegs from the photo management software f-spot.

I always start f-spot on the command line in debug mode and save the output to a file. This way I can get a better idea of what’s going on. i.e.

f-spot --debug 2>&1 | tee /home/dougie/f-spot.out

(there’s a wrapper script involved too that takes a backup of the old database first).

Once I have the images in f-spot I start applying tags. I have f-spot configured so that it writes tags to the image file itself as this gives me greater flexibility if I start copying photos around the place or into other packages. This setting is in f-spot under Edit / Preferences. i.e.

write image data to file

So now I have my photos and I’ve applied some tags to them. But if I have a look at the terminal session where I started f-spot, I can see there are problems. The messages will typically look something like this:

[4 Debug 21:39:15.516] open uri = file:///jpegs/2011/06/11/DSCN5936.JPG
[4 Debug 21:39:15.940] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5934.JPG
[4 Debug 21:39:15.942] open uri = file:///jpegs/2011/06/11/DSCN5934.JPG
[14 Debug 21:39:15.968] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5928.JPG)...
[14 Warn 21:39:15.970] Metadata of file file:///jpegs/2011/06/11/DSCN5928.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.359] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5931.JPG
[4 Debug 21:39:16.360] open uri = file:///jpegs/2011/06/11/DSCN5931.JPG
[14 Debug 21:39:16.569] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5924.JPG)...
[14 Warn 21:39:16.571] Metadata of file file:///jpegs/2011/06/11/DSCN5924.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.
[4 Debug 21:39:16.786] Invalid thumbnail, reloading: file:///jpegs/2011/06/11/DSCN5928.JPG
[4 Debug 21:39:16.787] open uri = file:///jpegs/2011/06/11/DSCN5928.JPG
[14 Debug 21:39:17.153] Syncing metadata to file (file:///jpegs/2011/06/11/DSCN5914.JPG)...
[14 Warn 21:39:17.154] Metadata of file file:///jpegs/2011/06/11/DSCN5914.JPG may be corrupt, refusing to write to it, falling back to XMP sidecar.

The problem for me is two-fold

  1. f-spot has a problem with the metadata in the jpeg.
  2. It’s creating extra .xmp files that I don’t want.

I’ve tried several avenues to resolve this and I thought the following would do the trick:

jhead -purejpg

This uses the Linux utility jhead to rewrite the image header with a standard one. Unfortunately the problem persists. So I took a more brutal approach. I used the linux utility exiftool to obliterate all meta data from the image.

At first I’d copy all the images from the memory card into a directory and then run exiftool on all the files there. i.e.

exiftool -all= *

(Note that syntax; there’s a space between the equal sign and asterix)

This is fine but unfortunately it pretty much destroys any useful or interesting extra information. Most I can live without, except the date and time that the photo was taken. It automatically inherits the file modification time instead.

The solution is to first read the jpegs into f-spot, then, while f-spot is still running, go to the directory where the images have been imported to, then run the exiftool command there. In my setup, I have f-spot configured to copy all imported jpegs to /jpegs where they are automatically arranged in directories according to date and time. So the procedure for me is as follows:

  1. Import jpegs into f-spot
  2. With f-spot running, change to destination directory where jpegs were imported to
  3. Run exiftool -all= *
  4. carry out tagging operations in f-spot

The advantage of this is that despite obliterating the meta data in the files using exiftool, f-spot still knows the data and times of these photos in its own database. So the next time you write to them with f-spot, by adding a tag for instance, they will get the date and time from f-spot. You lose pretty much everything else, which is a shame, but the date and time are what’s most important for me.

Typically what I tend to do is read all the photos from the memory card into a temporary folder, e.g. ~dougie/in, where I can carry out a quick pass using something like geeqie to delete any truly terrible photos. Then I import the photos into f-spot. Once that’s done I change to the destination directory where f-spot has copied the photos and run exifool. e.g.

dougie@phoenix /jpegs/2011/06/11 $ exiftool -all= *
60 image files updated
dougie@phoenix /jpegs/2011/06/11 $

Note the syntax for the exiftool command. That’s a space after the equal sign and before the asterix. It’s a powerful command so it’s best to check the man page first to ensure that you know what it’s going to do.

Serendipity

A few weeks ago, while chasing butterflies at Low Barns, I chanced upon this striking beastie. I was struck by the colours and thought it wouldn’t be difficult to identify what it is, but I had butterflies to chase and other things to do. I added the photo to my library, and tagged it as … Continue reading “Serendipity”

A few weeks ago, while chasing butterflies at Low Barns, I chanced upon this striking beastie. I was struck by the colours and thought it wouldn’t be difficult to identify what it is, but I had butterflies to chase and other things to do. I added the photo to my library, and tagged it as unidentified. Adding it to many other similarly tagged photos that I have queued up to sort out.

 

 

I can’t recall the host tree but the leaf looks cherry-like, although looking at it again that looks like it could be goat willow / pussy willow (Salix caprea).

Here’s a closer view of the creatures.

I didn’t think any more about it and consigned the image to the back-burner of idents-to-investigate, until I read a blog that showed the same creature in the same county around the same time. This is the froghopper (Cercopis vulnerata).

It’s one of the reasons I try to follow local natural history blogs and the news on local natural history sites. Chances are, what is being seen and written about across the country might very well be on my own doorstep.