Skip navigation
Currently Being Moderated

[export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post

Jul 19, 2009 4:18 AM

I opened a Bug with Adobe, and they suggested I post here. I hope that someone within the LUA developer group hears this...

 

Here's how I described it to Adobe (case #0180990524):

And I quote:
"
When using LrHttp.post or LrHttp.postMultipart, the post transaction fails with the target website, because of invalid TCP header in
formation. This only occurs on large transfers, like multipart posts containing image files.

 

Using tcpdump command on Mac OS 10.5, I captured the header of the packets for a successful post, and a failed post.

 

Successful:
19:48:24.212750 IP Macintosh.home.56274 > perfora.net.http: P 1:382(381) ack 1 win 65535
0x0000: 4500 01a5 6116 4000 4006 72a4 c0a8 0102 E...a.@.@.r.....

FAILED:
19:38:12.580630 IP Macintosh.home.56240 > perfora.net.http: P 3983702875:3983703256(381) ack 3995041646 win 65535
0x0000: 4500 01a5 c23b 4000 4006 117f c0a8 0102 E....;@.@.......

 

In the failed case, note the VERY large counter in the first line after "P".
"

 

The problem is that at the TCP layer, the computers have to agree on what part of the message is being sent. In the first case, it is from byte 1 to byte 382. In the second case, it STARTS with part 3983702875, which is completely invalid. The web server ignores the packet as corrupted, and the post command is unsuccessful. My guess is that Adobe has an un-initialized variable in their LrHttp library.

 

Here's Adobe's response:
"
I understand that you would like assistance with LrHttp.post and LrHttp.postMultipart. Adobe does not provide support for these methods at this level. For free assistance with this, you may refer to the forums online: http://www.adobe.com/support/forums/

Status: Withdrawn
"

 

They don't want to accept that they have a bug in their library. (I didn't withdraw it, they closed it that way)

I tried re-writing the export plugin to use only the "LrHttp.Post()" API, but with a large post (like an 15k image), it had the same problem.
If anyone else has experience to share on this topic, please reply. (This is not an offer to help with a bug inside Adobe, just curious if anyone else has this problem)

 
Replies
  • Currently Being Moderated
    Jul 20, 2009 6:53 AM   in reply to bpub

    Interesting.  Can you post an example of your export code?  There are plenty of of export plug-ins that post images to websites and work.  I have tested my latest export plug-in with files up to 80Mb so I wonder if it is something specific about your code or the file you are testing with.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 2, 2009 9:13 PM   in reply to bpub

    I know there are a few LR to Gallery plugins out there that do work. Mine are only going up to 5MB as that's the server limit for those services.. no issues though.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 1, 2009 10:34 AM   in reply to bpub

    Hi,

     

    I am the developper of the plugin, so I guess I can bring a little more information here. As far as I know, most users don't have any problem to upload large pictures, but I know some can not, for reasons I can not explain. I don't have the capacity nor time to test all configurations of my users, however, I try to stay as OS agnostic as I can, and for the moment, there is nothing I know of that is OS specific in my code.

     

    Here is some code to explain how it all works :

     

    The loop to process generated pictures :

        for i, rendition in exportContext:renditions{ stopIfCanceled = true } do
        
            -- Get next photo.
            local photo = rendition.photo
            local success, pathOrMessage = rendition:waitForRender()
            
            -- Check for cancellation again after photo has been rendered.
            if progressScope:isCanceled() then
                break
             end
            
            if success then
    
                local filename = LrPathUtils.leafName( pathOrMessage )
    
                local caption = ''
        
                -- Set the caption
                if captionSetting == 'none' then
                    caption = ''
                elseif captionSetting == 'filename' then
                    caption = filename
                elseif captionSetting == 'title' then
                    photo.catalog:withCatalogDo( function() 
                        caption = photo:getFormattedMetadata ( 'title' )
                    end ) 
                elseif captionSetting == 'caption' then
                    photo.catalog:withCatalogDo( function() 
                        caption = photo:getFormattedMetadata ( 'caption' )
                    end ) 
                end    
                
                -- upload file to gallery
                success = GalleryRemoteProtocol.uploadImage( serverId, albumName, pathOrMessage, caption )
                
                if success ~= '0' then
                    -- if we can't upload that file, log it.  For example, maybe user has exceeded disk
                    -- quota, or the file already exists and we don't have permission to overwrite, or
                    -- we don't have permission to write to that directory, etc....
                    table.insert( failures, filename )
                end
                        
                -- When done with photo, delete temp file. There is a cleanup step that happens later,
                -- but this will help manage space in the event of a large upload.
                LrFileUtils.delete( pathOrMessage )
            end
    

     

    And here is the function to upload the file :

    function GalleryRemoteProtocol.uploadImage(serverId, album, imagePath, caption)
        log:trace("Calling GalleryRemoteProtocol.uploadImage( "..prefs.serverTable[serverId].label..", "..album..", "..imagePath..", "..caption.." )")
        
        -- get server
        local server = prefs.serverTable[serverId].server
        
        local filename = LrPathUtils.leafName( imagePath )
        
        local response, debug, content
        if GalleryRemoteProtocol.galleryVersion == '1' then
            -- construct operation
            content =
                { { name='cmd', value='add-item' },
                  { name='protocol_version', value='2.0' },
                  { name='set_albumName', value=string.encodeEntities( album ) },
                  { name='caption', value=string.encodeEntities( caption ) },
                  { name='userfile_name', value=filename },
                  { name = 'userfile',
                    fileName = filename,
                    filePath = imagePath,
                    contentType = 'multipart/form-data'
                  }
                }
        else
            -- construct operation
            content =
                { { name='g2_form[cmd]', value='add-item' },
                  { name='g2_form[protocol_version]', value='2.0' },
                  { name='g2_form[set_albumName]', value=album },
                  { name='g2_form[caption]', value=caption },
                  { name='g2_form[force_filename]', value=filename },
                  { name='g2_authToken', value=GalleryRemoteProtocol.authToken },
                  { name = 'g2_userfile',
                    fileName = filename,
                    filePath = imagePath,
                    contentType = 'multipart/form-data'
                  }
                }
        end
        -- send POST request to the gallery server
        response, debug = LrHttp.postMultipart( GalleryRemoteProtocol.getGalleryRemoteURL(server), content )
        
        -- parse reponse into easy to digest table format
        responseTable = parseGalleryResponse( response )
            
        -- get server response status
        local serverStatus = getResponse( responseTable, 'status' )
        
        return serverStatus
    end
    

     

    As you can see, this is pretty dumb code.

     

    By the way, I had the chance to have Eric Scouten review my code. I got valuable information and one fix. However, he did not mention any potential problem in the upload section.

     

    If this can help...

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 1, 2009 10:54 AM   in reply to Arnaud73

    I did forward your message to one of our testers. He took the time to set up a Gallery server internally and create a reproducible test case that demonstrates a failure to upload large files with this plug-in. I haven't had the time to investigate this bug yet, but it is on my plate for LR3. (Sorry, can't provide info on when that will be available.)

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 1, 2009 6:32 PM   in reply to Arnaud73

    A possible cause of this issue is the wrong content type in your MIME chunk for the image file:

    { name = 'g2_userfile',
      fileName = filename,
      filePath = imagePath,
      contentType = 'multipart/form-data'
    }

     

    Try changing it as follows: contentType='image/jpeg'

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 2, 2009 4:45 AM   in reply to Vladimir Vinogradsky

    Vladimir Vinogradsky wrote:

     

    A possible cause of this issue is the wrong content type in your MIME chunk for the image file:

    { name = 'g2_userfile',
      fileName = filename,
      filePath = imagePath,
      contentType = 'multipart/form-data'
    }

     

    Try changing it as follows: contentType='image/jpeg'

    Right !

    I changed this and made some tests. It seams OK.

     

    However, I don't think this is the reason of the initial problem

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 2, 2009 4:58 AM   in reply to escouten

    Eric, thanks for the information!

     

    I think, this is going to be very tricky to nail down and maybe to reproduce. The described problem seams to be at the TCP level, so the Gallery server is important there.

    Bpub, can you tell :

    • Which OS/machine is used to run Gallery
    • Which webserver is used (and version)
    • Which PHP engine is used (and version)

     

    Maybe we won't be able to reproduce the same conditions. I have a similar case with someone else running Gallery 1, and can not upload large pictures, but I can upload the same pictures on his Gallery installation from my Windows XP laptop. Weird...

     

    Eric, if you need a test account on a Gallery 2, I can provide you with one. Maybe Bpub could do too, it might help.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 2, 2009 10:47 AM   in reply to Arnaud73

    Arnaud, I'll keep this in mind, but at this point, our tester has indicated that he was able to create a pretty solid reproducible case in-house. Hopefully that will be sufficient to figure out what's going wrong.

     

    Message was edited by: Eric Scouten (correcting a typo)

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 3, 2009 5:29 AM   in reply to escouten

    Additionnal information : I reinstalled Windows last week-end on my machine. This time I decided to give Windows 7 a try (after XP).

     

    I tested the plugin and LR on Windows 7 and almost everything seems to be fine, except I cannot create any album. the Plugin seams to stop in LrHttp.PostMultipart (the last try I have is from the GetGalleryRemoteUrl within the parameters. So I think, something gets stuck in PostMultipart. However, I managed to upload very large pictures to the same Gallery using also PostMultipart.

     

    By the way here is the code of the function to create an album. As you can see, this is not rocket science

    -- Add a new album to the server 
    function GalleryRemoteProtocol.addAlbum(serverId, parentAlbum, albumName, albumTitle, albumDescription)
        log:trace("Calling GalleryRemoteProtocol.addAlbum( "..prefs.serverTable[serverId].label..", "..parentAlbum..", "..albumName..", "..albumTitle..", "..albumDescription.." )")
        
        -- get server
        local server = prefs.serverTable[serverId].server
        
        local response, debug
        if GalleryRemoteProtocol.galleryVersion == '1' then
            response, debug = LrHttp.get( GalleryRemoteProtocol.getGalleryRemoteURL(server) ..
                "?cmd=new-album&protocol_version=2.1&set_albumName=" .. parentAlbum ..
                "&newAlbumName=" .. albumName ..
                "&newAlbumTitle=" .. albumTitle ..
                "&newAlbumDesc=" .. albumDescription )
        else
            -- construct operation
            local operation =
                { { name='g2_form[cmd]', value='new-album' },
                  { name='g2_form[protocol_version]', value='2.1' },
                  { name='g2_form[set_albumName]', value=parentAlbum },
                  { name='g2_form[newAlbumName]', value=albumName },
                  { name='g2_form[newAlbumTitle]', value=albumTitle },
                  { name='g2_form[newAlbumDesc]', value=albumDescription },
                  { name='g2_authToken', value=GalleryRemoteProtocol.authToken },
                }    
            
            -- define headers
            local headers =
                { { name='Content-Encoding', value='utf-8'},
                }
            
            -- send POST request to the gallery server
            response, debug = LrHttp.postMultipart( GalleryRemoteProtocol.getGalleryRemoteURL(server), operation, headers )
        end
        -- parse reponse into easy to digest table format
        responseTable = parseGalleryResponse( response )
        
        -- get server response status
        local serverStatus = getResponse( responseTable, 'status' )
        
        return serverStatus, getResponse( responseTable, 'album_name' )
    end
    
     
    |
    Mark as:
  • Currently Being Moderated
    Sep 17, 2009 2:25 AM   in reply to Arnaud73

    My mistake on this one...

     

    In the headers, I specified "name" instead of "field", LR 2.0 to 2.3 seamed OK with it, but LR 2.4 and LR 2.5 could not execute the LrHttp.postMultipart witouth dying with no error message.

     

    This one is corrected and not related to the initial problem.

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 16, 2009 1:17 PM   in reply to bpub

    Is this issue already confirmed or is there a solution/workaround for it.

    I just discovered that my plugin has problems with 100+ MB photos.

     

    One user gets an "not enough memory" error message. While I get (with the same photo) a 'An internal error occured: LuaRunException'

    This occurs in the PostMultipart call and after that any PostMultipart will fail unit lightroom is closed (ie force a kill of the lightroom process) and restarted.

     

    Any suggestions?

     

    Paul

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points