13 Replies Latest reply: Nov 16, 2009 1:17 PM by pact software RSS

    [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post

    bpub

      I opened a Bug with Adobe, and they suggested I post here. I hope that someone within the LUA developer group hears this...

       

      Here's how I described it to Adobe (case #0180990524):

      And I quote:
      "
      When using LrHttp.post or LrHttp.postMultipart, the post transaction fails with the target website, because of invalid TCP header in
      formation. This only occurs on large transfers, like multipart posts containing image files.

       

      Using tcpdump command on Mac OS 10.5, I captured the header of the packets for a successful post, and a failed post.

       

      Successful:
      19:48:24.212750 IP Macintosh.home.56274 > perfora.net.http: P 1:382(381) ack 1 win 65535
      0x0000: 4500 01a5 6116 4000 4006 72a4 c0a8 0102 E...a.@.@.r.....

      FAILED:
      19:38:12.580630 IP Macintosh.home.56240 > perfora.net.http: P 3983702875:3983703256(381) ack 3995041646 win 65535
      0x0000: 4500 01a5 c23b 4000 4006 117f c0a8 0102 E....;@.@.......

       

      In the failed case, note the VERY large counter in the first line after "P".
      "

       

      The problem is that at the TCP layer, the computers have to agree on what part of the message is being sent. In the first case, it is from byte 1 to byte 382. In the second case, it STARTS with part 3983702875, which is completely invalid. The web server ignores the packet as corrupted, and the post command is unsuccessful. My guess is that Adobe has an un-initialized variable in their LrHttp library.

       

      Here's Adobe's response:
      "
      I understand that you would like assistance with LrHttp.post and LrHttp.postMultipart. Adobe does not provide support for these methods at this level. For free assistance with this, you may refer to the forums online: http://www.adobe.com/support/forums/

      Status: Withdrawn
      "

       

      They don't want to accept that they have a bug in their library. (I didn't withdraw it, they closed it that way)

      I tried re-writing the export plugin to use only the "LrHttp.Post()" API, but with a large post (like an 15k image), it had the same problem.
      If anyone else has experience to share on this topic, please reply. (This is not an offer to help with a bug inside Adobe, just curious if anyone else has this problem)

        • 1. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
          Mark J M Wilson Community Member

          Interesting.  Can you post an example of your export code?  There are plenty of of export plug-ins that post images to websites and work.  I have tested my latest export plug-in with files up to 80Mb so I wonder if it is something specific about your code or the file you are testing with.

          • 2. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
            bpub Community Member

            It's for LR to Gallery. Available at:

             

            http://www.starway.org/blogs/Photographie/lightroom-gallery/lightroom-to-gallery-english/

             

            Look for GalleryRemoteProtocol.lua -> uploadimage()

             

            The bug was first reported in this forum: http://gallery.menalto.com/node/68277

             

            "Tracing through the log the LrHttp::Postmultipart call just doesn't return it seems"

             

            This has occurred on Mac OS 10.4 and 10.5. I'm not sure of other platforms.

            • 3. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
              seanmcfoto ACP

              I know there are a few LR to Gallery plugins out there that do work. Mine are only going up to 5MB as that's the server limit for those services.. no issues though.

              • 4. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                Arnaud73

                Hi,

                 

                I am the developper of the plugin, so I guess I can bring a little more information here. As far as I know, most users don't have any problem to upload large pictures, but I know some can not, for reasons I can not explain. I don't have the capacity nor time to test all configurations of my users, however, I try to stay as OS agnostic as I can, and for the moment, there is nothing I know of that is OS specific in my code.

                 

                Here is some code to explain how it all works :

                 

                The loop to process generated pictures :

                    for i, rendition in exportContext:renditions{ stopIfCanceled = true } do
                    
                        -- Get next photo.
                        local photo = rendition.photo
                        local success, pathOrMessage = rendition:waitForRender()
                        
                        -- Check for cancellation again after photo has been rendered.
                        if progressScope:isCanceled() then
                            break
                         end
                        
                        if success then
                
                            local filename = LrPathUtils.leafName( pathOrMessage )
                
                            local caption = ''
                    
                            -- Set the caption
                            if captionSetting == 'none' then
                                caption = ''
                            elseif captionSetting == 'filename' then
                                caption = filename
                            elseif captionSetting == 'title' then
                                photo.catalog:withCatalogDo( function() 
                                    caption = photo:getFormattedMetadata ( 'title' )
                                end ) 
                            elseif captionSetting == 'caption' then
                                photo.catalog:withCatalogDo( function() 
                                    caption = photo:getFormattedMetadata ( 'caption' )
                                end ) 
                            end    
                            
                            -- upload file to gallery
                            success = GalleryRemoteProtocol.uploadImage( serverId, albumName, pathOrMessage, caption )
                            
                            if success ~= '0' then
                                -- if we can't upload that file, log it.  For example, maybe user has exceeded disk
                                -- quota, or the file already exists and we don't have permission to overwrite, or
                                -- we don't have permission to write to that directory, etc....
                                table.insert( failures, filename )
                            end
                                    
                            -- When done with photo, delete temp file. There is a cleanup step that happens later,
                            -- but this will help manage space in the event of a large upload.
                            LrFileUtils.delete( pathOrMessage )
                        end
                

                 

                And here is the function to upload the file :

                function GalleryRemoteProtocol.uploadImage(serverId, album, imagePath, caption)
                    log:trace("Calling GalleryRemoteProtocol.uploadImage( "..prefs.serverTable[serverId].label..", "..album..", "..imagePath..", "..caption.." )")
                    
                    -- get server
                    local server = prefs.serverTable[serverId].server
                    
                    local filename = LrPathUtils.leafName( imagePath )
                    
                    local response, debug, content
                    if GalleryRemoteProtocol.galleryVersion == '1' then
                        -- construct operation
                        content =
                            { { name='cmd', value='add-item' },
                              { name='protocol_version', value='2.0' },
                              { name='set_albumName', value=string.encodeEntities( album ) },
                              { name='caption', value=string.encodeEntities( caption ) },
                              { name='userfile_name', value=filename },
                              { name = 'userfile',
                                fileName = filename,
                                filePath = imagePath,
                                contentType = 'multipart/form-data'
                              }
                            }
                    else
                        -- construct operation
                        content =
                            { { name='g2_form[cmd]', value='add-item' },
                              { name='g2_form[protocol_version]', value='2.0' },
                              { name='g2_form[set_albumName]', value=album },
                              { name='g2_form[caption]', value=caption },
                              { name='g2_form[force_filename]', value=filename },
                              { name='g2_authToken', value=GalleryRemoteProtocol.authToken },
                              { name = 'g2_userfile',
                                fileName = filename,
                                filePath = imagePath,
                                contentType = 'multipart/form-data'
                              }
                            }
                    end
                    -- send POST request to the gallery server
                    response, debug = LrHttp.postMultipart( GalleryRemoteProtocol.getGalleryRemoteURL(server), content )
                    
                    -- parse reponse into easy to digest table format
                    responseTable = parseGalleryResponse( response )
                        
                    -- get server response status
                    local serverStatus = getResponse( responseTable, 'status' )
                    
                    return serverStatus
                end
                

                 

                As you can see, this is pretty dumb code.

                 

                By the way, I had the chance to have Eric Scouten review my code. I got valuable information and one fix. However, he did not mention any potential problem in the upload section.

                 

                If this can help...

                • 5. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                  escouten Adobe Employee

                  I did forward your message to one of our testers. He took the time to set up a Gallery server internally and create a reproducible test case that demonstrates a failure to upload large files with this plug-in. I haven't had the time to investigate this bug yet, but it is on my plate for LR3. (Sorry, can't provide info on when that will be available.)

                  • 6. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                    Vladimir Vinogradsky

                    A possible cause of this issue is the wrong content type in your MIME chunk for the image file:

                    { name = 'g2_userfile',
                      fileName = filename,
                      filePath = imagePath,
                      contentType = 'multipart/form-data'
                    }

                     

                    Try changing it as follows: contentType='image/jpeg'

                    • 7. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                      Arnaud73 Community Member

                      Vladimir Vinogradsky wrote:

                       

                      A possible cause of this issue is the wrong content type in your MIME chunk for the image file:

                      { name = 'g2_userfile',
                        fileName = filename,
                        filePath = imagePath,
                        contentType = 'multipart/form-data'
                      }

                       

                      Try changing it as follows: contentType='image/jpeg'

                      Right !

                      I changed this and made some tests. It seams OK.

                       

                      However, I don't think this is the reason of the initial problem

                      • 8. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                        Arnaud73 Community Member

                        Eric, thanks for the information!

                         

                        I think, this is going to be very tricky to nail down and maybe to reproduce. The described problem seams to be at the TCP level, so the Gallery server is important there.

                        Bpub, can you tell :

                        • Which OS/machine is used to run Gallery
                        • Which webserver is used (and version)
                        • Which PHP engine is used (and version)

                         

                        Maybe we won't be able to reproduce the same conditions. I have a similar case with someone else running Gallery 1, and can not upload large pictures, but I can upload the same pictures on his Gallery installation from my Windows XP laptop. Weird...

                         

                        Eric, if you need a test account on a Gallery 2, I can provide you with one. Maybe Bpub could do too, it might help.

                        • 9. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                          escouten Adobe Employee

                          Arnaud, I'll keep this in mind, but at this point, our tester has indicated that he was able to create a pretty solid reproducible case in-house. Hopefully that will be sufficient to figure out what's going wrong.

                           

                          Message was edited by: Eric Scouten (correcting a typo)

                          • 10. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                            bpub Community Member

                            Here's the version details:

                             

                            Os: Mac OS 10.5

                            Server: Linux infong 2.4 #1 SMP Thu May 28 16:09:02 UTC 2009 i686 GNU/Linux

                            PHP: Version 5.2.10

                            • 11. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                              Arnaud73 Community Member

                              Additionnal information : I reinstalled Windows last week-end on my machine. This time I decided to give Windows 7 a try (after XP).

                               

                              I tested the plugin and LR on Windows 7 and almost everything seems to be fine, except I cannot create any album. the Plugin seams to stop in LrHttp.PostMultipart (the last try I have is from the GetGalleryRemoteUrl within the parameters. So I think, something gets stuck in PostMultipart. However, I managed to upload very large pictures to the same Gallery using also PostMultipart.

                               

                              By the way here is the code of the function to create an album. As you can see, this is not rocket science

                              -- Add a new album to the server 
                              function GalleryRemoteProtocol.addAlbum(serverId, parentAlbum, albumName, albumTitle, albumDescription)
                                  log:trace("Calling GalleryRemoteProtocol.addAlbum( "..prefs.serverTable[serverId].label..", "..parentAlbum..", "..albumName..", "..albumTitle..", "..albumDescription.." )")
                                  
                                  -- get server
                                  local server = prefs.serverTable[serverId].server
                                  
                                  local response, debug
                                  if GalleryRemoteProtocol.galleryVersion == '1' then
                                      response, debug = LrHttp.get( GalleryRemoteProtocol.getGalleryRemoteURL(server) ..
                                          "?cmd=new-album&protocol_version=2.1&set_albumName=" .. parentAlbum ..
                                          "&newAlbumName=" .. albumName ..
                                          "&newAlbumTitle=" .. albumTitle ..
                                          "&newAlbumDesc=" .. albumDescription )
                                  else
                                      -- construct operation
                                      local operation =
                                          { { name='g2_form[cmd]', value='new-album' },
                                            { name='g2_form[protocol_version]', value='2.1' },
                                            { name='g2_form[set_albumName]', value=parentAlbum },
                                            { name='g2_form[newAlbumName]', value=albumName },
                                            { name='g2_form[newAlbumTitle]', value=albumTitle },
                                            { name='g2_form[newAlbumDesc]', value=albumDescription },
                                            { name='g2_authToken', value=GalleryRemoteProtocol.authToken },
                                          }    
                                      
                                      -- define headers
                                      local headers =
                                          { { name='Content-Encoding', value='utf-8'},
                                          }
                                      
                                      -- send POST request to the gallery server
                                      response, debug = LrHttp.postMultipart( GalleryRemoteProtocol.getGalleryRemoteURL(server), operation, headers )
                                  end
                                  -- parse reponse into easy to digest table format
                                  responseTable = parseGalleryResponse( response )
                                  
                                  -- get server response status
                                  local serverStatus = getResponse( responseTable, 'status' )
                                  
                                  return serverStatus, getResponse( responseTable, 'album_name' )
                              end
                              
                              • 12. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                                Arnaud73 Community Member

                                My mistake on this one...

                                 

                                In the headers, I specified "name" instead of "field", LR 2.0 to 2.3 seamed OK with it, but LR 2.4 and LR 2.5 could not execute the LrHttp.postMultipart witouth dying with no error message.

                                 

                                This one is corrected and not related to the initial problem.

                                • 13. Re: [export plugin] Bug in LrHttp.PostMultipart and LrHttp.Post
                                  pact software

                                  Is this issue already confirmed or is there a solution/workaround for it.

                                  I just discovered that my plugin has problems with 100+ MB photos.

                                   

                                  One user gets an "not enough memory" error message. While I get (with the same photo) a 'An internal error occured: LuaRunException'

                                  This occurs in the PostMultipart call and after that any PostMultipart will fail unit lightroom is closed (ie force a kill of the lightroom process) and restarted.

                                   

                                  Any suggestions?

                                   

                                  Paul