13 Replies Latest reply on Nov 16, 2009 2:54 PM by Vladimir Vinogradsky

    LrHttp.post and HTTP redirect

    jartp

      Hi,

       

      I am writing a plugin that uses HTTP to upload images to a site. The site uses cookies to maintain a user's session. It works like this:

       

      1. Browser:POST to a sigin script sending username and password

      2. Server: Respond with HTTP 303 (redirect) including Set-Cookie header for session cookie and URL to redirect to

      3. Browser: GET redirected URL, including sending cookie header as part of the GET request

      4. Server: Respond HTTP 200 with redirected URL contents (no Set-Cookie header)

       

      To allow my plugin to grab the session cookie, I need to be able to see the 303 response from step 2.

       

      However, the way LrHttp.post works is to hide steps 2 and 3 from me, and return the html from the successful re-direct.

       

      I understand that this is nice and correct behaviour from Lr.Http to automatically interpret the redirect, but this prevents me from seeing the Set-Cookie header and hence I cannot establish a session with the server from Lightroom.

       

      Any ideas?

       

      James

        • 1. Re: LrHttp.post and HTTP redirect
          jartp Level 1

          Further to this, I looked at another site I'd like to develop an export plug-in for and it exhibits the same behaviour but using HTTP 302 instead of 303.

           

          So I did some research and it appears it is perfectly valid, and common, behaviour for a site to receive a POST request with login details, and return a set cooke header with a session cookie along with a HTTP 302 or 303 redirect to the "successful login" page.

           

          I don't see how it is possible to write a plug-in for this type of site unless LrHttp.post can be (optionally) told not to follow redirects.

           

          Anyone see it differently or know of a workaround?

           

          James

          • 2. Re: LrHttp.post and HTTP redirect
            jimkeir

            Hi,

             

            Same problem here. It would be nice to have a parameter to disable the automatic following of redirects, or alternatively to pass through any Set-Cookie directives that are present on the original response.

             

            Cheers,

            Jim

            • 3. Re: LrHttp.post and HTTP redirect
              Vladimir Vinogradsky Level 1

              This is undocumented, but based on my own experience, LrHttp.post and LrHttp.get process cookies automatically - both these methods extract cookies from HTTP responses and include them in subsequent requests, and you don't have to lift a finger.

              • 4. Re: LrHttp.post and HTTP redirect
                jimkeir Level 1

                Hi,

                 

                Vladimir Vinogradsky wrote:

                 

                This is undocumented, but based on my own experience, LrHttp.post and LrHttp.get process cookies automatically - both these methods extract cookies from HTTP responses and include them in subsequent requests, and you don't have to lift a finger.

                You're right - but that's exactly the problem. I *need* to lift a finger. One of those cookies is a server response message which isn't available elsewhere. It arrives with the first response, which is a 302 (redirect). LR automatically redirects and returns the contents of *that* request - which doesn't have the same cookies.

                 

                I found a way round it by scraping another webpage, but it's not ideal.

                 

                Cheers,

                Jim

                • 5. Re: LrHttp.post and HTTP redirect
                  Vladimir Vinogradsky Level 1

                  Pardon me if I still don't understand the issue.

                  The way cookies work, they are generated on the server and sent to the client with the only purpose of getting them back along with subsequent requests from the client so the server identify the client and the session with that client. Generally speaking, clients should not care what those cookies represent - their sole concern must be to send cookies back to the server along with any  requests. This is exactly what LrHttp does - it takes the load of extracting and injecting cookies off of the programmer by doing its magic in the background.

                   

                  When the server responses with a redirect (or any other type of response) containing cookies, those cookes are retrieved by LrHttp and sent back along with your next GET or POST. Let me reiterate the steps of your scenario with some comments:

                   

                  1. Browser:POST to a sigin script sending username and password

                   

                  2. Server: Respond with HTTP 303 (redirect) including Set-Cookie header for session cookie and URL to redirect to

                  LrHttp extracts cookies from Set-Cookie headers and saves them for later communication with the server

                   

                  3. Browser: GET redirected URL, including sending cookie header as part of the GET request

                  LrHttp includes saved cookies in the request and passes them back to server along with your query string

                   

                  4. Server: Respond HTTP 200 with redirected URL contents (no Set-Cookie header)

                   

                  I don't mean to sound arrogant, I just don't see a reason to concern yourself with having access to Set-Cookie headers if your goal is simply to have the cookies included with your next GET (or POST) requests.

                  • 6. Re: LrHttp.post and HTTP redirect
                    jartp Level 1

                    Hi Vladimir,

                     

                    While you are correct with what you are saying, I have two scenarios where I cannot develop the functionality I require because of a set-cookie being hidden by a re-direct.

                     

                    Scenario A - redirect from SSL url to non-SSL url.

                    I am trying to authenticate a plugin to a site.

                    1. The login URL is an https URL, I can send valid authentication parameters to this url

                    2. On successful login, the resulting HTTP sent from the server is a redirect containing a set-cookie, domain is set in the set-cookie header. The redirect is to an http URL in the same domain.

                    3. LrHttp does NOT appear to send the cookie back with the next get/post. (Note: If the redirect is to an https URL, then the cookie is sent - as you describe).

                     

                    Scenario B - need to access the cookie for application logic purposes

                    In this case, the cookie contains data on the user's configuration, it is not just a session cookie. I want to read the cookie and do something with it...

                     

                    But regardless of the reasons why, LrHttp exposes any set-cookies returned with a response, so it is not access to set-cookie headers themselves that we are talking about here - it is a given that developers would want to, and can, access set-cookie headers (otherwise why would LrHttp provide access to them?).

                     

                    The issue is that because LrHttp intelligently follows re-directs, because of this we don't have the ability to see set-cookie headers that are part of the redirect. Hence, what is needed is the option to have LrHttp follow re-directs or not... that's the real issue.

                     

                    You don't sound arrogant - healthy debate is good!

                     

                    James

                    • 7. Re: LrHttp.post and HTTP redirect
                      jimkeir Level 1

                      Hi,

                       

                      James is right, questions are healthy!

                       

                      Under some circumstances, what you have is quite correct. As long as the cookies get sent back with the redirected request, there's no problem. However, it's not always the case. It's quite common to have an authentication request redirected. You log in, the server returns some kind of token and redirects you to your home page. You are then expected to return that token with every subsequent request, not just the next one, to prove that you're authenticated:

                       

                      My request : HTTP GET /login.asp?username=me&password=metoo

                      Initial Response : 302 Redirect to /homepage.asp; Set-Cookie: MyAuthKey=0x1234-5678-abcdef01-2345

                       

                      Lightroom's request : HTTP GET /homepage.asp; Cookie MyAuthKey=...

                      Server's response : 200 (contents of /homepage.asp)

                       

                      *** At this point, I get the contents of /homepage.asp with no MyAuthKey cookie.

                       

                      The situation is that I need that MyAuthKey value to include in every subsequent request I send, not just to get the contents of /homepage.asp which are in fact totally irrelevant. The MyAuthKey is sent *only* when you log in, and expected on every request after that.

                       

                      Cheers,

                      Jim

                      • 8. Re: LrHttp.post and HTTP redirect
                        Vladimir Vinogradsky Level 1

                        I see your point (both jartp and jimkeir).

                        In situations where the cookie contains information which you need to process or interpret, not having the access to it may ruin your entire design. I truly hope the SDK will grow to offer more functionality and flexibility for advanced plugin developers.

                         

                        There is just one thing I am still puzzled about... I am pretty sure that LrHttp saves cookies and pushes them  out not only with the very next request, but with every subsequent request. In fact, even when you close your plugin and invoke it again, the very first GET already contains cookies saved from the last session. I've seen this by snooping after Lightroom with a HTTP sniffer. First I was surprised, but then I though maybe Lightroom has some sort of an internal cookie cache (per host, similar to a web browser). This way it can grab those cookies from its cache and insert them in every subsequent request to that host. Only after you restart Lightroom that cache is reset and the very first HTTP request contains no cookies.

                         

                        I am going to do some more testing on the weekend to see if I can confirm that. I am also going to test whether cookies returned in a redirect to a non secure URL after an HTTPS POST are retained. I am expecting to confirm (or invalidate) this with captures from my HTTP sniffer and Apache server logs.

                         

                        Thanks both for providing your input!

                        • 9. Re: LrHttp.post and HTTP redirect
                          Vladimir Vinogradsky Level 1

                          I was able to confirm that my guess was correct, Lightroom does has a cookie cache, and it does insert received cookies in every subsequent request to the original host (under some conditions). Bellow are some excerpts from the communication log between a plugin and my test server. I'm not a fan of long posts but please bear with me as the following contains pretty interesting findings.

                          Scenario

                          1.Plugin requests a login screen over HTTP

                          Lightroom performs GET http://<domain>/code/lr/login.php"


                          Web Server receives GET /code/lr/login.php
                             Cookies: <none>
                             Request Vars: <none>

                           

                          Web Server response
                             Set-Cookie:PHPSESSID=2kl5vj2ksvtk0oovhq79msrma1; path=/
                             RESPONSE HTML: <1563 bytes>


                           

                          Lightroom receives
                            Set-Cookie: PHPSESSID=2kl5vj2ksvtk0oovhq79msrma1; path=/
                            Content-Length: 1563


                          2. Plugin performs a post over HTTPS to the URL specified in the form's action tag. The server responds with a redirect to a non-secure URL, the redirect response contains a user_id cookie. Lightroom performs internal redirect to the non-secure URL and requests its content via GET.

                          Lightroom performs POST https://<domain>/code/lr/login.php?mode=login
                          fldPassword=LightroomPassword&fldUser=LightroomUser

                           

                          Web server receives POST /code/lr/login.php?mode=login
                          Cookies
                             PHPSESSID: 2kl5vj2ksvtk0oovhq79msrma1
                          Request Vars
                             mode: login
                             fldPassword: LightroomPassword
                             fldUser: LightroomUser

                          Web server response
                            Set-Cookie: user_id=user_0123456789; path=/
                            Location: /code/lr/login.php?mode=user_login

                          Lightroom performs an internal redirect

                          Web server receives GET /code/lr/login.php?mode=user_login
                          Cookies
                            PHPSESSID: 2kl5vj2ksvtk0oovhq79msrma1
                            user_id: user_0123456789
                          Request Vars
                            mode: user_login

                          Web server responses
                            Set-Cookie: <none>
                            RESPONSE TEXT: You have requested page [user_login]

                           

                          Plugin receives
                             Set-Cookie: <none>
                             Content-Length: 36
                             CONTENT: You have requested page [user_login]


                          3. Just to be completely sure, plugin performs another GET from a different URL, and as we can see from the log, the user_id cookie is also supplied. Now, in the lua code I do not extract or insert any cookies in server requests, all manipulations with cookies happen behind the scenes, thanks to the SDK.

                          Plugin performs GET http://<domain>/code/lr/url2.php?mode=page2

                           

                          Web server receives GET /code/lr/url2.php?mode=page2
                          Cookies
                             PHPSESSID: 2kl5vj2ksvtk0oovhq79msrma1
                             user_id: user_0123456789
                          Request Vars
                             mode: page1

                           

                          Web server responses
                             Set-cookie  <none>
                             RESPONSE TEXT: URL2= You have requested page [page2]

                           

                          Plugin receives
                            
                          Set-Cookie: <none>
                             Content-Length: 37
                             CONTENT: URL2= You have requested page [page2]

                          Important Discovery
                          While testing and playing with the code I made one little discovery, which might be crucial to solving the original problem. All this magic works only when the server specifies a cookie path. The path can be "path=/" (the cookie is valid within the entire domain), or "path=/foo/" (the cookie is valid within the foo directory and its subdirectories). If the server doesn't specify a path at all, the cookie will not be included by the SDK in any subsequent request. This is where SDK behaves badly! As per HTTP specification, any cookie without a path should be available within the current directory that the cookie is being set in. In my example, the current directory is /code/lr/, therefore any script located in that directory must receive the "pathless" cookie back, but the SDK doesn't supply it.

                          Conclusion
                          Plugin developers don't really need to worry about extracting and inserting cookies (except for the rare occasions when the application logic depends on the cookie value), as long  as the web application responds with a domain-scope or directory-scope cookie. When the web server sets path-less cookies, the SDK behaves badly by not including cookies in any subsequent request, even those that match the directory path where the cookie was set in. I believe the LrHttp needs to be improved in both aspects - it should provide access to stored cookies, and it should correctly process pathless cookies. Other that that, it works beautifully!

                          • 10. Re: LrHttp.post and HTTP redirect
                            escouten Adobe Employee

                            Vladimir, thanks for the detailed info. I've copied this into a bug report and we'll investigate.

                            • 11. Re: LrHttp.post and HTTP redirect
                              Vladimir Vinogradsky Level 1

                              I'd like to revisit this topic once again.

                              It appears when the server performs multiple redirects, cookies are not being saved for intermediate redirects.

                              Let's say there's a chain of two redirects - URL1 responds with "Cookie1" and a redirect to URL2. URL2, in turn, responds with Cookie 2 and a redirect to URL3. As a result, Cookie 1 never reaches the plugin, it also doesn't get saved in the SDK's internal cookie cache. I am not sure about the last redirect, but I am a 100% certain any cookie set before the final redirect will be lost.

                              This would have been only half bad if not for the other serious omission in the SDK - there's no way to disable redirects and handle them from within the plugin.

                              I hope this get addressed in LR3's SDK.

                              • 12. Re: LrHttp.post and HTTP redirect
                                escouten Adobe Employee

                                Hi Vladimir, this bug is still open/ToFix in my queue for LR3. I haven't had time to look into it in more detail yet. It will probably be 2-3 weeks yet.

                                • 13. Re: LrHttp.post and HTTP redirect
                                  Vladimir Vinogradsky Level 1

                                  Eric,

                                   

                                  I don't mean to give you a hard time with this issue, it's just there is something I have recently stumbled on and that is somehow related to this topic.

                                   

                                  Recently I attempted to implement an export plugin by integrating with a web site that employs a rather convoluted authorization process. Instead of using HTTP POST to submit the user name and password for authentication, it expects a GET request from the client with the user information supplied as a cookie created on the client and added to the list of cookies initially supplied by the server at the beginning of the session. It appears the existing implementation of LrHttp does not provide a way to inject a cookie if the cookie cache is non-empty. I tried supplying a Cookie header, however the added cookie is being ignored. The request goes out with all of the cookies received from the server originally, but the new cookie is not included.

                                  If I rename the "field" field of the headers array to "Cookie2", the new header does show up in the outgoing request.

                                  I believe LrHttp simply overwrites the supplied Cookie header with the information from its cookie cache, if there are any cookies stored.

                                   

                                  I hope in addition to fixing certain obvious issues with handling cookies, the next version of the SDK gives developers full control over redirects and cookies. I also believe this entire topic is rather critical as the current behavior bars Lightroom from many exciting integration opportunities.

                                  Thanks again!