Exceeding limit of searchable field

[Eckhard Henkel]

Recently I changed metadata field entries by setting "searchable" (and "browsable") to "true" as described at Lightroom SDK guide chapter "Defining metadata fields", "Metadata field entries", page 74. As a result I got this error message:

Could not upgrade your catalog for plug-in metadata.

An internal error has occurred: Searchable strings must be less than 512 bytes; this value is 513 bytes

So, the plugin can't update the catalog and as a result the plugin is disabled.

The error message fits to the SDK description, stating "Strings stored in this field must not exceed 511 bytes."

The error message states not, which image of my catalog (and which metadata field) causes the error. So I don't have an idea how to identify the image and its field which has exceeded the limit. Without identification of the image(s) I'm not able to shorten the field entry/entries.
Any idea how I could deal with this situation?

[johnrellis]

The easiest way I can think of is to write a script that iterates over every photo and tests the length of each custom field. 

You could also use SQLite to examine the catalog database for too-long fields (but that would take me significantly longer to reverse engineer that part of the database schema).

[johnrellis]

Another option is the List View plugin.   It can export a CSV file containing the custom metadata fields of all plugins for all photos.

[Eckhard Henkel]

Dear John,

thanks a lot for your hints and proposals!

I will give the "List View" plugin a try.

This might help me to identify the image(s) which exceed the 511-bytes-limit. But, maybe, it's a special case of my catalog and the exceedance I did at some fields.

Beside this, I'm looking for a more general solution for cases of other plugin's users.

To make my concerns a bit more clear: I'm working at this open source project: LrMediaWiki

Up to now, the plugin doesn't support searchable custom metadata fields. Due to a user request, I'll change several custom metadata fields to be searchable and browsable at a future version of the plugin.

Now a decision has to be made: Which fields should be searchable/browsable and which fields not?

My assumption is, the usage of the first three description fields might exceed the limit, others not. Of course, it's an assumption and highly depends on the use cases and the specific semantic of the fields.

Therefore my actual approach is to define the most fields to be searchable/browsable and to avoid the three description fields which might exceed the limit to let them not to be searchable/browsable.

The intention is to provide as much as possible fields to be searchable/browsable. On the other hand, fields which might exceed the limit shouldn't be searchable/browsable to avoid users to run into the same problem as me.

What do you think about this approach?

[johnrellis · Accepted solution]

My assumption is, the usage of the first three description fields might exceed the limit, others not. Of course, it's an assumption and highly depends on the use cases and the specific semantic of the fields. Therefore my actual approach is to define the most fields to be searchable/browsable and to avoid fields which might exceed the limit to let them not to be searchable/browsable. The intention is to provide as much as possible fields to be searchable/browsable. On the other hand, fields which might exceed the limit shouldn't be searchable/browsable to avoid users to run into the same problem as me.

Seems plausible.  But the fields that are the longest might be precisely the ones that users most want to search?

In my Any Comment plugin, I worked around the limit by having the plugin maintain two versions of a field, one with the user-supplied contents (not searchable) and one that's searchable and guaranteed to be less than 511 bytes.  For example, it has "comments" (user-supplied contents; not searchable) and "searchableComments" (contents constructed from "comments"; searchable).   The "searchableComments" contents consists of the least frequently occurring words in "comments" (with no duplicates), as many as will fit under 511 bytes.   (Users are more likely to search for unique names and words than commonly used words like "a", "the", etc.)

[Eckhard Henkel]

John, thanks again for your support!

> Seems plausible.

Therefore I decided to go the mentioned way.

> But the fields that are the longest might be precisely the ones that users most want to search?

Maybe, maybe not. I don't know. As often happens, it's not easy to predict user's use cases and expectations.

My decision fulfills a request of a single user who is interested in a special field to be searchable. In addition I added as much other fields as possible. If any other user will report a similar request related to the fields I omitted, I'll have so see, what can be done.

Maintaining two versions of a field might be an option – but up to now, I try to avoid this effort.

However, I consider this thread to be resolved.