22 Replies Latest reply on Feb 21, 2017 3:34 PM by johnrellis

    Getting yield errors when attempting require of debugscript

    LoweMo.photo Level 1

      I've been using johnrellis's excellent Debugging Toolkit (debugscript) to help improve my Lightroom plugin development flow. It's been useful in a couple of plugins I've worked on, but in one particular project I've been collaborating on, if even all I do is include the "require lines", the plugin fails to operate properly and I find the following in the Plugin Manager error log:

      **** Error 1
      An error occurred while attempting to run one of the plug-in’s scripts.
      attempt to yield across metamethod/C-call boundary
      
      **** Error 2
      Could not load the post-processing filter from this plug-in.
      attempt to yield across metamethod/C-call boundary
      

       

      I initially had tried wrapping all the (documentation-hinted) functions in the Debug.showErrors() calls and thought I must have made a mistake somewhere when I found the plugin wasn't working. So I reverted my changes and found things working again; at least things which I wanted to debug were working in their buggy state. (Now I think it's getting near perfection.) It wasn't till later that I found that trying to include the debugscript, even for simple things like a call to Debug.lognpp(), would fail (even before making any calls to its functionality.

       

      I wonder if anyone else has seen this error. To replicate it, simply check out the project and uncomment the require lines (currently only in a couple of the files (e.g. in DialogTagging.lua):

       

      local Require = require 'Require'.path ("../debugscript.lrdevplugin")
      local Debug = require 'Debug'.init ()
      require 'strict'
      

       

      And, well, you would also need to create a free developer account on Clarifai.com and follow the installation/config to be able to attempt to run the script, but I promise that it will be worth the effort if you want a really cool keywording tool, integrated into Lightroom. I'd just love to be able to use the Debugging Toolkit with it and have no idea what would cause this kind of error. I searched and there are not any explicit calls to yield, so it must be some Lightroom quirk, but there is likely a workaround for our use (I hope)...

       

      Happy to hear any kind of response.

        • 1. Re: Getting yield errors when attempting require of debugscript
          johnrellis Most Valuable Participant
          attempt to yield across metamethod/C-call boundary

          That error occurs when you call an API call that needs to be executed in a task started by LrTasks, e.g. photo:getDevelopSettings().

          • 2. Re: Getting yield errors when attempting require of debugscript
            LoweMo.photo Level 1

            Thank you, johnrellis. I'm sure there must be something like that. Strange that the error only occurs if the debugscript is included, though... but I'll do further investigation to see if there might be such a task that I've missed.

             

            I don't think anything like that is in the parts I've contributed, since I've paid careful attention to that, but I did find something along those lines (a call to LrHttp.post() that needed to be within an LrFunctionContext.postAsyncTaskWithContext function block). So far, no changes I've tried have resolved this issue, though.

            • 3. Re: Getting yield errors when attempting require of debugscript
              DFBurns Level 1

              For what it's worth, I came here to post the same question and found yours. I'm having the same issue. I decided to ditch my homegrown logging and move wholesale to John's debugging library. Now, my export plugin's dialog works fine but I get an error from Lightroom that "Could not load the post-processing filter from this plugin. Attempt to yield across ... blah blah."

               

              (One odd thing is that Lightroom proceeds with some built-in process because it goes ahead and exports a JPEG of the selected image.)

               

              Any strategies for figuring this out? This used to work fine before changing over to John's library so I thought I had all the tasks and contexts worked out but maybe I was lucky all along?

               

              db

              • 4. Re: Getting yield errors when attempting require of debugscript
                johnrellis Most Valuable Participant

                I've personally never used the Debugging Toolkit with a full-blown export plugin.  (Some of my plugins do exports internally, though.)  But I've used the toolkit extensively with all sorts of async tasks and SDK API calls that require such tasks.

                 

                One easy thing to try: Try renaming the plugin folder from myplugin.lrdevplugin to myplugin.lrplugin.  That will automatically disable Debug without needing to edit the code.  Do you still get the "attempt to yield across metamethod/C-call boundary"

                 

                I'm wondering if the problem has something to do with Debug.showErrors ().  When Debug is disabled, showErrors(func) uses LR's recommended method for ensuring errors are displayed:

                 

                    return function (...)
                        return LrFunctionContext.callWithContext("wrapped", 
                            function (context)
                                LrDialogs.attachErrorDialogToFunctionContext (context)
                                return func (unpack (arg))
                                end)
                

                 

                But when debugging is enabled, showErrors (func) uses pcall () or LrTasks.pcall (), depending on whether the code is running on a task created by LrTasks.startAsyncTask() or is on the main task.  If you try to call an API function that yields internally within pcall (), you'll get the "attempt to yield across metamethod/C-call boundary" error. 

                 

                I don't have any hypothesis yet why showErrors() might be causing problems. 

                • 5. Re: Getting yield errors when attempting require of debugscript
                  johnrellis Most Valuable Participant

                  Another alternative is to try debugging with the ZeroBrane IDE: Debugging Adobe Lightroom plugins with ZeroBrane Studio - ZeroBrane . I haven't tried it, but others have reported here that it worked for them.  It appears to be a full-fledged remote debugger.

                  • 6. Re: Getting yield errors when attempting require of debugscript
                    DFBurns Level 1

                    @johnrellis, I got things working last night. On a lark, I went into the Plugin Manager and turned *off* the option to reload the plugin on each export. I haven't dived into your code yet but I think your code does some sort of caching of required files and maybe this conflicts with LR in some unexpected way? It's a drag because I have to reload after every code change but at least I can focus on my own code now.

                     

                    I'd be curious to hear your thoughts on this and why it might be behaving this way. And yes, I've been meaning to try ZeroBrane for a long time. Maybe soon.

                     

                    Cheers,

                    db

                    • 7. Re: Getting yield errors when attempting require of debugscript
                      johnrellis Most Valuable Participant

                      I think your code does some sort of caching of required files and maybe this conflicts with LR in some unexpected way? I

                      Debugscript's require() won't reload a module if it's already loaded, unless Debug is enabled, in which case it always reloads.   In particular, if the plugin folder is named myplugin.lrdevplugin, it will always reload; if it is named myplugin.lrplugin, it won't reload.

                       

                      I don't have any hypotheses why reloading might affect things.  Try renaming the module to myplugin.lrplugin and see if the problem occurs.

                      • 8. Re: Getting yield errors when attempting require of debugscript
                        johnrellis Most Valuable Participant

                        Thinking about the yield errors overnight, I've got a (weak) hypothesis. Perhaps LrTasks.canYield () doesn't work properly in the export-plugin environment.  If so, that would cause Debug.showErrors() to use pcall() rather than LrTasks.pcall(), which would cause the yield errors.

                         

                        The easiest way to test that is to rename the extension of your plugin folder from .lrdevplugin to .lrplugin and see if the errors still occur.

                        • 9. Re: Getting yield errors when attempting require of debugscript
                          DFBurns Level 1

                          johnrellis I tried a 2x2 matrix: with "Reload plugin on each export" on and off, and with my plugin's directory named .lrplugin and .lrdevplugin. From all 4 tests, the "Dev" suffix made no difference. The only thing that made LR find my functions properly was turning the Reload option off in the plugin manager.

                           

                          I still haven't looked deeply into your code yet. Anything I can help look for? If it matters, I'm on Lightroom 2015.8 and macOS 10.12.1.

                           

                          Thanks,

                          db

                          • 10. Re: Getting yield errors when attempting require of debugscript
                            johnrellis Most Valuable Participant

                            The only thing that made LR find my functions properly was turning the Reload option off in the plugin manager.

                            What precisely are the symptoms you mean by "LR find my functions properly"? I thought you were getting the error messages described in the original post?

                             

                            Unless you have explicitly invoked Require.reload (true), then when the plugin folder ends with .lrplugin, Require doesn't reload any files (just like LR's built-in require()). So when the folder ends with .lrplugin, and then you disable the plugin manager's Reload, the only effect is stop LR's reloading of the file(s) referenced directly by Info.lua.  I have no hypotheses as to why that might affect things.

                            • 11. Re: Getting yield errors when attempting require of debugscript
                              DFBurns Level 1

                              Ah, sorry if unclear. The error messages I get are the ones I mentioned. And btw, they are not caught and displayed by the Debug library, they are only shown in the plugin manager dialog.

                               

                              What I meant by not finding my functions is that, even though Lightroom seems to find some functions from my service provider script without issue (the ones that define top and bottom sections for the export dialog), the two others specified there (updateExportSettings and processRenderedPhotos) are never even invoked by Lightroom. I have logging first thing in each function and it never appears in the debug.log file.

                               

                              What *does* happen when I tell Lightroom to export is that, rather than completely failing when it says "Could not load the post-processing filter from this plugin," it seems to fallback to a built-in default and try to export my image as a jpeg. Strange.

                               

                              I doubt it will help, but for avoidance of doubt, here is that section of my provider script. I've tried a version of this without wrapping functions with Debug.showErrors() and it made no difference.

                               

                              sectionsForBottomOfDialog = Debug.showErrors( LIVExportDialog.sectionsForBottomOfDialog ),
                              sectionsForTopOfDialog  = Debug.showErrors( LIVExportDialog.sectionsForTopOfDialog ),
                              updateExportSettings  = Debug.showErrors( LIVExportDialog.updateExportSettings ),
                              processRenderedPhotos  = Debug.showErrors( LIVTask.processRenderedPhotos ),

                               

                              I can make progress now with auto-reload turned off but it'd be great to find a way to turn it back on and have things work. Again, I'm happy to dive into code, do extra logging if it helps, etc. etc.

                               

                              db

                               

                              • 12. Re: Getting yield errors when attempting require of debugscript
                                johnrellis Most Valuable Participant

                                Anything I can help look for? If it matters, I'm on Lightroom 2015.8 and macOS 10.12.1.

                                Did the problem occur with 2015.7 or earlier?

                                 

                                The only thing I can suggest is to do painful binary search with logging to see precisely where the yield error message first arises. If the problem might be with Debug itself (as suggested by the original poster), then you could use a copy of Debug's logging code within Debug.lua or any other .lua file.  Make sure this block is executed first to create a global "log":

                                 

                                local LrDate = import 'LrDate'
                                local LrLogger = import 'LrLogger'
                                local LrPathUtils = import 'LrPathUtils'
                                local logFilename = LrPathUtils.child (_PLUGIN.path, "digdeep.log")
                                
                                log = LrLogger (_PLUGIN.id .. "digdeep")
                                
                                log:enable (function (msg)
                                    local f = io.open (logFilename, "a")
                                    if f == nil then return end
                                    f:write (
                                        LrDate.timeToUserFormat (LrDate.currentTime (), "%y/%m/%d %H:%M:%S"),
                                        msg, MAC_ENV and "\n" or "\r\n")
                                    f:close ()
                                    end)
                                

                                 

                                Then insert log:trace() statements, e.g.

                                 

                                log:trace ("Goodbye cruel world")
                                


                                They'll be logged in the file "digdeep.log" in the plugin folder.

                                • 13. Re: Getting yield errors when attempting require of debugscript
                                  johnrellis Most Valuable Participant

                                  Also, it's unfortunate that the LR SDK doesn't allow us to trap errors before it unwinds the stack, so you can see the stack as it exists when the error occurs, rather than after it is unwound to LR's error handler or Debug's error handler, which are usually near the bottom of the stack.

                                   

                                  Lua provides xpcall() for such trapping, but it isn't available via the SDK; I've hacked LR to allow Debug to access xpcall(), but it doesn't work with LR's tasks.   LR appears to have something internally called coxpcall(), but I haven't figured out how to use it with plugins.  I don't think the ZeroBrane LR debugger has solved the problem either, but I might be wrong.

                                  • 14. Re: Getting yield errors when attempting require of debugscript
                                    DFBurns Level 1

                                    Amen to that. I've been writing software for 30 years and Lightroom is one of the least developer-friendly environments I've ever worked in.

                                    • 15. Re: Getting yield errors when attempting require of debugscript
                                      DFBurns Level 1

                                      I remember that the problem was also in 2015.7 but I can't rollback to try it again without a little pain.

                                       

                                      The problem with trying to find where the yield error arises is that it doesn't seem to be in my code. All of my dialog functions exit cleanly. Those two export functions never get called. So where is the error occurring? As you say, sure would be great to have a callstack.

                                      • 16. Re: Getting yield errors when attempting require of debugscript
                                        johnrellis Most Valuable Participant

                                        So where is the error occurring?

                                        Perhaps in Require.lua or Debug.lua, either when they are initializing or in Require.require()?  That's what the original poster's symptoms seemed to indicate.

                                        • 17. Re: Getting yield errors when attempting require of debugscript
                                          DFBurns Level 1

                                          Ok. I'll do some experimenting tonight if I can.

                                          • 18. Re: Getting yield errors when attempting require of debugscript
                                            DFBurns Level 1

                                            johnrellis I need to call it a night but here's what I know so far. Things are dying in Require.findFile when it calls LrFileUtils.exists(). Not on every call - just one and under specific circumstances. When the plugin is reloaded in the plugin manager, all files are require'd and all goes well. When the Export dialog runs, all the same files are required in the same order and all goes well. When I click Export in the dialog, it tries to require all the files in the same order but when it gets to one of my files (DFB.lua but I can't believe the name matters?), that's when it chokes.

                                             

                                            What I don't understand is that I insert this test code right before the call and it always works:

                                             

                                            if pcall( LrFileUtils.exists, filePath) then
                                               slog:trace( '.exists SAFE' )

                                            else
                                               slog:trace( '.exists NOT SAFE' )

                                            end

                                             

                                            But then the next line is the actual call to LrFileUtils.exists( filePath ) and that seems to fail (logging immediately afterward doesn't appear and my export throws the error about attempting to yield across C-call boundary.

                                             

                                            Is this a possibly useful clue? I added code right before the .exists() call to log the result of LrTasks.canYield(). It is always false for requires when reloading in the plugin manager and when starting the Export dialog but after clicking Export, it's false until right before the attempt to require the file where things fail. In that case, canYield() returns true.

                                             

                                            At this point, it may be useful to continue this offline until there's known solution. If you're willing, can you PM me a good email address to discuss?

                                             

                                            Thanks,

                                            db

                                            • 19. Re: Getting yield errors when attempting require of debugscript
                                              johnrellis Most Valuable Participant

                                              Using a copy of DFBurns' code, I narrowed down the problem to a bug in LR.  The following expression fails when called from an LrExportServiceProvider script:

                                              LrTasks.pcall (LrFileUtils.exists, ".")

                                              It generates the error "attempt to yield across metamethod/C-call boundary". Of course, LrTasks.pcall() should never generate an error (provided its first argument is a function). 

                                               

                                              Please add your me-too vote and opinion to this bug report in the official Adobe feedback forum: Lightroom SDK: LrTasks.pcall fails when called from an LrExportServiceProvider script | Photoshop Family Customer Commun…  . Though I very much doubt Adobe will ever fix this, based on the amount of love the SDK has received in the last many years.

                                               

                                              I suspect that LR isn't properly setting up the plugin execution environment when it calls the LrExportServiceProvider script.  I last tested the toolkit with the sample export service way back in LR 3, so the bug must have been introduced in the intervening six years.

                                               

                                              I haven't tried it, but here's one way that the bug might be avoided while still using the toolkit:

                                               

                                              - Don't use Require at all in the LrExportServiceProvider script or any scripts it requires.  Place copies of Debug.lua and strict.lua in your folder plugin so the built-in require() can find them.

                                               

                                              - Don't require Debug.lua in the LrExportServiceProvider script itself.

                                               

                                              - Don't use Debug.showErrors () to wrap the various functions in the table returned by the LrExportServiceProvider script.   E.g. instead of writing:

                                              startDialog  = Debug.showErrors( LIVExportDialog.startDialog )

                                              do this instead:

                                              startDialog  = LIVExportDialog.startDialog

                                              - In the file defining the functions returned by the LrExportServiceProvider, wrap the function bodies with Debug.showErrors:

                                              LiveExportDialog.startDialog = Debug.showErrors (function ()

                                                  ...

                                                  end)

                                               

                                              If this gets too painful, it might be a good time to investigate the ZeroBrane IDE and debugger.

                                              • 20. Re: Getting yield errors when attempting require of debugscript
                                                johnrellis Most Valuable Participant

                                                Dave Burns asked some questions in a private email.  I thought I’d answer in this forum, since the answers involve some learnings about LR’s task model and pcall() that others might find useful.

                                                 

                                                The answer to both questions is hidden in the implementation of Debug.showErrors() – in particular, this code:

                                                To understand why the code does this, first some background: LR’s “main task” cannot call a number of potentially long-running API calls that require the ability to yield to other tasks (e.g. calls that access the catalog).   Such calls must be made from “asynchronous tasks” created by LrTasks.startAsynchronousTask() (or the internal equivalent).   The core engine of the user interface appears to execute on the main task, including plugin callbacks provided to LrView controls.  The top-level scripts of plugins are also executed on the main task.

                                                 

                                                Lua’s built-in pcall () somehow interferes with the ability of asynchronous tasks to yield.  If you try to call a long-running API call from within pcall(), you’ll get the error “attempt to yield across metamethod/C-call boundary”. (I don’t understand much about the Lua and LR implementations to know definitely why.  But the standard implementation of Lua’s pcall() uses C’s “longjmp” mechanism, and Lua’s coroutines may rely on longjmp as well.  LR’s tasks are layered on Lua’s coroutines, and perhaps LR’s task scheduler relies on longmp too, and perhaps these uses of longjmp are incompatible.)

                                                 

                                                So it’s ok to use the built-in pcall() on the main task (where you can’t call the long-running API calls in any case). And it’s also ok to use it on asynchronous tasks, as long as you don’t invoke long-running API calls within the pcall(). 

                                                 

                                                But if you need to trap errors in asynchronous tasks for code that contains long-running API calls, you need to use LrTasks.pcall().  The documentation says:

                                                Simulates Lua's standard pcall(), but in a way that allows a call to LrTasks.yield() to occur inside it.

                                                It appears to “simulate” the pcall() by creating a new task to run the called function.  The caller then waits for that task to complete.  (You can verify that by using Lua’s built-in coroutine.running() to get the coroutine identifiers of the caller and callee.) 

                                                 

                                                The downside of the implementation of LrTasks.pcall() is that Lua’s built-in stack tracing (debug.getinfo ()) doesn’t know how to traverse from the callee (in one task) to the caller (in another task).  So a debugger like Debug (and perhaps Zerobrane) will show a stack trace stopping at LrTasks.pcall().

                                                 

                                                So (deep breath) that is why Debug.showErrors() avoids the use of LrTasks.pcall() when it can, i.e. when the caller is the main task.  The idea was to make it a little more likely a debugger could show the full call stack.  But six years later, that potential benefit is not large, since very little code instrumented with Debug.showErrors () is being called from the main task. (At least that’s true with my plugins, which execute almost entirely within asynchronous tasks, to simplify the problem of calling long-running API calls.)

                                                 

                                                To further pop the stack to your original questions:

                                                Your steps to repro included this code “LrTasks.pcall (LrFileUtils.exists, ".”)” but down the stack in findfile, I was using pcall with that just fine and then the call that seems to fail is a call to .exists() without using pcall or LrTasks.pcall at all.

                                                The implementation of Require.require() wraps the code chunk compiled from the loaded file with Debug.showErrors().  So any call to LrFileUtils.exists() executed from a file loaded by Require will execute within the context of Debug.showErrors(). If that Require was called from a main task, then Debug.showErrors() will use pcall(); if it’s called from an asynchronous task, it will use LrTasks.pcall().  It’s that use of LrTasks.pcall() combined with LrFileUtils.exists() that triggers the bug.

                                                 

                                                LrFileUtils.exists() is not normally a long-running call, and it can be safely called from within the built-in pcall(). So wrapping it within pcall() doesn’t have any effect.  But wrapping it within LrTasks.pcall() (as Debug.showErrors() does) can trigger the LR bug.

                                                 

                                                Why doesn’t the first require in the chain of files I require hit the bug? That is, why does it not fail until (in my case) DFB.lua is required? If I understand how things are working, seems to me it’s going to fail on the first require?

                                                 

                                                I think that when LR first loads the plugin, it is loaded from LR’s main task (as are all plugin top-level scripts).  All the Debug.showErrors() (including the hidden ones in Require.require()) thus use the built-in pcall().    But when you check the Reload option in the Plug-in Manager, the subsequent reloads are initiated from an asynchronous task.  Thus, Debug.showErrors() uses LrTask.pcall(), triggering the LR bug when LrFileUtils.exists() is called.

                                                • 21. Re: Getting yield errors when attempting require of debugscript
                                                  johnrellis Most Valuable Participant

                                                  LrTasks.pcall () can also fail when invoked from within a property table's observer function: Lightroom SDK: LrTasks.pcall fails when called from an LrExportServiceProvider script | Photoshop Family Customer Commun…

                                                   

                                                  Something is obviously wrong with LrTasks.pcall().

                                                  • 22. Re: Getting yield errors when attempting require of debugscript
                                                    johnrellis Most Valuable Participant

                                                    I released a new version of the Debugging Toolkit: New version of Debugging Toolkit . The new version includes documentation on how to use it with export and publish-service plugins, while getting back the ability to reload your source files each time you execute the plugin in development mode (avoiding the LR bug discussed extensively in this thread).