5 Replies Latest reply on Dec 20, 2013 8:37 AM by areohbee

    improving the reliability of external command calls

    victorvodka Level 1

      This is probably a Rob Cole question since I'm doing things under the Elare Framework, but it might apply more generally.  I'm

      using a service call that looks something like

      app:call( Service:new{ name=serviceName, async=true,   main=function( call )
                     local catPath =  LrPathUtils.child( _PLUGIN.path, "appDb.sqlite")
                     if not str:is( catPath ) then
                          --app:show{ warning="Catalog path can not be blank." }
                     if catPath == catalog:getPath() then
                     local LrLogger = import 'LrLogger'
                     local expResp = app:getPref( 'expectResponse' )
                     local outHdl
                     if expResp then
                          outHdl = 'del'
                          outHdl = nil
                     if _win_env() then
                          sqlite = LrPathUtils.child( _PLUGIN.path, 'sqlite3.exe' )
                          sqlite = LrPathUtils.child( _PLUGIN.path, 'sqlite3' )
                     if string.sub(strSQL, string.len(strSQL), string.len(strSQL))~=";" then
                          strSQL=strSQL ..";"
                     local s, m = fso:writeFile( scriptPath, strSQL )
                     local param = str:fmt( '"^1" ".read "^2""', catPath:gsub( "\\", "\\\\" ), scriptPath:gsub( "\\", "\\\\" ) ) 
                     local s, m, c = app:executeCommand( sqlite, param, nil, tempFileName, nil ) -- nil => no specific targets, nil => temp-file for capturing output is fine, 'del' => get output, then delete output file.
                     local out=fso:readFile(tempFileName )
                     if not bwlCachedQuery then
                     updateDialog(out, typeIn, catId, generatorId, generator, imageId, strSQL, timeThrough)
                end } )

      It basically writes some SQL into a temp file, calls executeCommand to run sqlite3.exe or the Mac equivalent, reads the result file, getting pipe-delimited data, and then it deletes the temp files. (This is an oversimplification because sometimes it actually caches the result files and uses them later without having to do the Sqlite query.)   I've noticed, though, that some fraction of the time, a good query that should result in valid data produces an empty file.  I have no idea why, but it's a source of some vexing unreliability.  I've actually put in some code to re-run the query when this happens (all my Updates and Inserts have a small select query on the end so they always produce some data).  I was wondering why this might be happening? I should mention that the names of my temp files include long random numbers, so it's not a problem of one file overwriting another.

        • 1. Re: improving the reliability of external command calls
          areohbee Level 5

          Hmm... - I don't see anything wrong with the code. Of course you should be checking statuses and error messages returned in case they provide a clue. I'll keep thinking about it and see if I come up with any other ideas. I've never had such unreliability, but I'm struggling with my own brand of unreliability at the moment - all commands hang sometimes (infrequently, but enough...), not just sqlite, and not just plugins (not just Lightroom either), although that's where I notice it the most.


          Consider trying the latest version of sqlite, maybe they've fixed a bug:



          • 2. Re: improving the reliability of external command calls
            DawMatt Level 3



            I haven't looked at Rob's framework in a while, but I have occassionally found similar issues when working directly with the Lightroom SDK. I suspect part of the problem is files not having been completely written to disk at the time you try to read it in.  This might be due to a timing issue or interference with file system writes by another program (e.g. a virus scanner).


            In my code I've tried a couple of different techniques:

            - add a loop and some brief sleeps to see if the file content becomes available prior to a pre-defined timeout. I think my current maximum sleep time configured is 0.2 seconds and it has been working OK on my test equipment (using Win 7 x64 and OS/X 10.7.5). This can be problematic if you are running a large volume of commands outputing small amounts of text, but otherwise an up to 0.2 second delay should not have a noticeable impact on the user

            - avoided using LrFileUtils.readFile() . I've found it sometimes fails to read file content even though I've just confirmed a file isReadable(). Instead I've been using the io calls directly. Please note this issue was most visible during Lightroom startup, or periods of high disk activity (e.g. backups were running).


            It migth be worth checking some of the framework internals to see if either of these could be impacting you here.



            1 person found this helpful
            • 3. Re: improving the reliability of external command calls
              victorvodka Level 1

              Well, I wasn't doing a test on isReadable() so I've added that.  And I haven't seen a failure of data since.  Here's hoping!

              • 4. Re: improving the reliability of external command calls
                areohbee Level 5

                Would you mind posting the code so we can see where isReadable() is going. Also, are you checking statuses returned from calls like writeFile & executeCommand?

                • 5. Re: improving the reliability of external command calls
                  areohbee Level 5

                  Matt Dawson wrote:


                  It migth be worth checking some of the framework internals to see if either of these could be impacting you here.

                  Thank you Matt - your feedback here has helped me squash a particularly insidious and intermittent bug in the framework.


                  I am now checking isReadable and no longer using LrFileUtils.readFile and so far haven't experienced the bug I was experiencing - knock on wood.


                  Have you reported these bugs to Adobe?


                  Also, the problem with NOT using LrFileUtils.readFile is that lua's io library doesn't work if there are non-ascii characters in filesystem path, which is (presumably) one of the main reasons the function exists in the first place.


                  I run modified versions of dofile/loadfile for this reason, i.e.

                      local LrPathUtils = import 'LrPathUtils'
                      local LrFileUtils = import 'LrFileUtils'
                      _G.loadfile = function( file )
                          -- Note: file is optional, and if not provided means stdin.
                          -- Shouldn't happen in Lr env(?) but just in case...
                          if file == nil then
                              return nil, "load-file from stdin not supported in Lr env."
                          local filename = LrPathUtils.leafName( file )
                          local status, contents = pcall( LrFileUtils.readFile, file ) -- lr-doc says nothing about throwing errors, or if contents returned could be nil: best to play safe...
                          if status then
                              if contents then
                                  local func, err = loadstring( contents, filename ) -- returns nil, errm if any troubles: no need for pcall (short chunkname required for debug).
                                  if func then
                                      return func
                                      --return nil, "loadstring was unable to load contents returned from: " .. tostring( file or 'nil' ) .. ", error message: " .. err -- lua guarantees a non-nil error message string.
                                      local x = err:find( filename )
                                      if x then
                                          err = err:sub( x ) -- strip the funny business at the front: just get the good stuff...
                                      elseif err:len() > 77 then -- dunno if same on Mac ###2
                                          err = err:sub( -77 )
                                      return nil, err -- return *short* error message
                                  -- return nil, "Unable to obtain contents from file: " .. tostring( file ) -- probably also too long.
                                  return nil, "No contents in: " .. filename
                              -- return nil, "Unable to read file: " .. tostring( file ) .. ", error message: " .. tostring( contents or 'nil' ) -- probably also too long.
                              return nil, "Unable to read file: " .. filename
                      _G.dofile = function( file )
                          local func, err = loadfile( file ) -- returns nil, errm if any problems.
                          if func then
                              local result = {} -- hokey handling here has since been improved in later versions..
                              result[1], result[2], result[3], result[4], result[5], result[6], result[7], result[8], result[9], result[10], result[11], result[12], result[13], result[14], result[15], result[16], result[17], result[18], result[19], result[20], result[21] = func() -- throw error, if any.
                              if result[21] ~= nil then
                                  error( "Modified dofile only supports 20 return values" )
                                  return unpack( result )
                              error( err ) -- error message guaranteed.