11 Replies Latest reply on Mar 29, 2018 4:48 PM by stib_at_work

    AE using socket to import picture from URL - Need Your Help

    videocp Level 1

      I have a project that works with local network using the path name. I want to update the project to replace all placeholders with a picture located at a URL address instead of the local network. I have no experience with socket coding and can't seem to find the right snippets of code on the internet to get me through this point of using URL as a file location.

       

      I tried this code (using a picture at this URL as an example);

       

      webConnect= new Socket;

      if(webConnect.open("http://distilleryimage3.s3.amazonaws.com")) {

               webConnect.write('GET /2940b7e0236511e2af9022000a1f9a23_7.jpg HTTP/1.0nn');

                response=  webConnect.read();

                webConnect.close();

                alert('complete');

      } else {

               alert('fail');

                }

       

      Any help will be appreciated. I need the results of the socket connection to be a object I can use in the statement

       

      curItem.replace(new File( <picture path> );

       

      Thanks again.

        • 1. Re: AE using socket to import picture from URL - Need Your Help
          chauffeurdevan Level 2

          Just looked in the adobe javascript documentation and you should not put the http:// and you should put the http port number 80 at the end. So the connection should be "distilleryimage3.s3.amazonaws.com:80".

           

          At the GET, make sure you have slashes before both "n".

          • 2. Re: AE using socket to import picture from URL - Need Your Help
            videocp Level 1

            Thanks, that was most helpful. At least I'm getting a response from the server. Working on the idea of image buckets and such. Some trial and error and more reading I was able to get this socket response; This was response from

            webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');

             

            How do I save the results into an object that curItem.replace(new File( <Socket response> ); will work with? I did change the GET to 'binary' insteaed of HTTP:/1.0\n\n but just don't know what to do with it next.

            screenshot-socket.jpg

             

            You help and patience with my questions has been greatly appreciated. Learning curves and patience are not easily to over come or use at time. Thanks again.

            • 3. Re: AE using socket to import picture from URL - Need Your Help
              chauffeurdevan Level 2

              Now you need to parse that response.

               

              Here to get you started :

               

              var response =  webConnect.readln(); var statusCode = new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/g).exec(q)[1]; if (parseInt(statusCode) != 200) webConnect.close(); else{     response =  webConnect.readln();     /*read all the necessary lines;*/     webConnect.close(); }

               

               

              I suppose there is an undefined number of line in that header. So you'll read lines by lines and look for content that you want. Eventually you'll need to know the content-length that you'll feed to the webConnect.read(contentLengthValue). You might do it using regexp like i did for the first line (http://gskinner.com/RegExr/). You should look at the definition of a http response for how to finish processing this header, but it looks like there is an empty line before feeding the data.

              • 4. Re: AE using socket to import picture from URL - Need Your Help
                videocp Level 1

                Thanks again for your help. I'm inching towards a working bit of code. I'm still researching socket service protocol because I'm just typing in code and isntructions sometimes and just don't know why I'm doing it.


                What is the \n\n after the HTTP1.0 in the GET command?

                 

                It does appear that the data (picture) is at the end of the initial response, I'm just not able to separate it from the header and create a valid image file.

                 

                Perhaps it is understanding what the \n\n means. Also do I need the\\n\n if I use BINARY instead of HTTP1.0 ?

                 

                Thanks again for all of you help. I feel as though I'm one step away from this working.

                • 5. Re: AE using socket to import picture from URL - Need Your Help
                  chauffeurdevan Level 2

                  Ok, to get the data at the end, it is like I said, you have to get the value of content-length and use that value for webConnect.read(content-lengthValue). From what I understand, the empty line means the end of the header, after that, it is the data.

                   

                  To process the header, you'll do an while loop with a readln command that you'll parse. (right now, you should replace it by a for loop to prevent and infinite loop)

                   

                  So something like  (i didn't test it, so there is possiblity there is some errors):

                   

                  var contentLength = ""

                  var statusCode = ""

                  for (var i = 0; i < 20; i++){

                       var response =  webConnect.readln();

                       if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

                            statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1] );

                       }

                       else if (new RegExp(/Content-Length: ([0-9]+)/).test(response)){

                            contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1] );

                       }

                       else if (new RegExp(/^\s*$/).test(response)){

                            break;

                       }

                  }

                  if (statusCode != 200) webConnect.close()

                  else{

                      var data = webConnect.read(contentLength);

                       var file= new File(Folder.temp.absoluteURI   + "/httpSocket.jpeg");

                       file.encoding='BINARY'

                       file.open ("w");

                       file.write(data);

                       file.close ();

                       file.execute();

                  }

                   

                   

                  If not, I don't know the thing wbout BINARY instead of HTTP.... for the \n\n, it is just empty line to tell the server to stop reading. And in fact, it should be \r\n from what i saw on the web - cause of the line carriage and line return diffrences.

                  • 6. Re: AE using socket to import picture from URL - Need Your Help
                    videocp Level 1

                    OK, this is my working code, that appears to fetch the right size of data after the text (the pic is 121KB) but it is not saving a valid jpg file. What appears to be a blank line between the text and the data is something so I changed the last else to recognize the last text line. The new RegExp(/^\s*$/).test(response) was not breaking and it was reading too many lines

                     




                    webConnect= new Socket;

                     

                    if(webConnect.open( "distilleryimage5.s3.amazonaws.com:80")) {

                            webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');

                     

                         var file= new File("httpSocket.jpeg");

                         file.encoding='BINARY'

                         file.open ("w");



                    var contentLength = ""

                    var statusCode = ""

                    for (var i = 0; i < 20; i++){

                         var response =  webConnect.readln();

                     

                         if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

                              statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1]);

                         }

                         else if (new RegExp(/Content-Length: ([0-9]+)/).test(response) ){

                              contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1]);

                         }

                         else if (new RegExp(/Server: AmazonS3/).test(response) ) {



                      break;

                         }

                    }

                     

                     

                    if (statusCode != 200) webConnect.close()

                    else{


                    alert('statusCode=' + statusCode + ' contentLength=' + contentLength);

                        var data = webConnect.read(contentLength);


                    alert(data.length);

                         file.write(data);

                    }

                         file.close();

                         file.execute();

                    }

                    • 7. Re: AE using socket to import picture from URL - Need Your Help
                      videocp Level 1

                      OK, this comes close to working. I can read the jpg file with QuickTime, though it is corrupted and the image is different color or hue. Nothing else will read the jpg file, it so my efforts to use After Effects have still not been met.

                       

                      I have not idea why I have to read 1 byte at the beginning of the data set and discard it. I can't find enough information about the jpg header format but it seems there is corruption at the top of the downloaded file. Since I'm downloading the 'contentLength', it's odd that midstream I get corrupted data. It appears to be consistent too.

                       

                      Anyway, I am about at a lose.

                       




                      webConnect= new Socket;

                       

                       


                      if(webConnect.open( "distilleryimage5.s3.amazonaws.com:80","binary")) {


                      webConnect.write('GET /distilleryimage5/17bdfa64237411e2a23c22000a1f9d66_7.jpg HTTP/1.0\n\n');






                           var file= new File("httpSocket.jpg");

                           file.encoding='BINARY'

                           file.open ("w","binary");



                      var contentLength = ""

                      var statusCode = ""

                      for (var i = 0; i < 20; i++){

                           var response =  webConnect.readln();

                       

                       

                           if (new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).test(response)){

                                statusCode = parseInt( new RegExp(/HTTP\/[0-9.]+\s([0-9]+)/).exec(response)[1]);

                           }

                           else if (new RegExp(/Content-Length: ([0-9]+)/).test(response) ){

                                contentLength = parseInt( new RegExp(/Content-Length: ([0-9]+)/).exec(response)[1]);

                           }

                           else if (new RegExp(/Server: AmazonS3/).test(response) ) {



                        break;

                           }

                      }

                       

                       

                      if (statusCode != 200) webConnect.close()

                      else{


                      alert('statusCode=' + statusCode + ' contentLength=' + contentLength);

                          var data = webConnect.read(1);

                       



                      var data = webConnect.read(contentLength);


                      file.write(data);

                       

                       

                      }

                           file.close();

                           file.execute();

                      }

                      • 8. Re: AE using socket to import picture from URL - Need Your Help
                        chauffeurdevan Level 2

                        The extra byte is cause you changed the regexp from /^\s*$/ to /Server: AmazonS3/. Put it back to /^\s*$/

                         

                        I have the reading part a little bit better but it is always shorter doesnt' understand why. At least, the script stops when/if the server close the connection.

                         

                        var data = "";

                        while (data.length <= contentLength){

                            data = data + webConnect.read(contentLength);

                            if (!webConnect.connected) break;

                        }

                        webConnect.close()

                        delete webConnect;

                         

                        I also found this page that is really good explained. From what I understand, it could be that the GET should be implemented for HTTP/1.1 and maybe adding a keep-alive connection arguments.

                        http://www3.ntu.edu.sg/home/ehchua/programming/webprogramming/http_basics.html

                        • 9. Re: AE using socket to import picture from URL - Need Your Help
                          stevelewisza Level 1

                          I have found a solution to this for anyone who may be interested:

                           

                          It's fairly simple, instead of using the socket object, what I've done is use wget, which is a command prompt tool that can download files via command prompt, since you can execute commands through extendscript with system.callSystem, you can execute the wget tool to download the file from the url, all you need to do is download the wget tool from here: GNU Wget 1.19.4 for Windows then place wget.exe into the folder you want to download images to

                           

                          I've written a snippet that works with After Effects that will download the image with wget, then import that file into After Effects, just change the paths to suit:

                           

                           

                           

                          var folderLocation = "d:/wget";

                          var imageURL = "https://i.imgur.com/pKY6r1K.jpg";

                           

                          var myCommand = "cd /d " + folderLocation + " & wget " + imageURL;

                          system.callSystem("cmd /c \"" + myCommand + "\"");

                           

                          var path = new File(folderLocation + "/" + (imageURL.split("/")[imageURL.split("/").length - 1]));

                           

                          if (path.exists) {

                            var io = new ImportOptions(path);

                            var videoFile = app.project.importFile(io);

                          } else {

                            alert("There was a problem importing the image");

                          }

                          1 person found this helpful
                          • 10. Re: AE using socket to import picture from URL - Need Your Help
                            jsimons@vectorworks.net

                            stevelewisza – thanks for posting! This brings me a little closer to a solution to a variation on this technique I've been trying to wrap my head around, which is pulling items from an Adobe Shared Library into AE. I think because the libraries are cloud based, each asset should have a web address, which could be called using your snippet. (Since I can't figure out a way to access Adobe Libraries directly inside of AE).

                             

                            Many of the end users of my projects will use Macs, however. Do you know of a similar software that would be compatible with Mac? I suppose it wouldn't be too difficult to version the  script out for each OS. If there is something similar, would the process of calling the .dng via system.callSystem work the same way it does to call an .exe in Windows?

                             

                            Any insight would be greatly appreciated!

                            • 11. Re: AE using socket to import picture from URL - Need Your Help
                              stib_at_work Level 2

                              curl is probably your best bet on macs, (wget is available, but curl is built-in).

                              1 person found this helpful