9 Replies Latest reply on Feb 12, 2017 3:44 PM by polarapfel

    Adobe RAW Converter failing with "There was not enough memory" error

    polarapfel

      Hi,

       

      I'm running the Adobe DNG Converter against a large number of files (14587) which are stored on a network share that is mapped to a drive letter. The output is written to a locally attached USB drive. The machine is an older i7 (Lenovo T420S), 8GB RAM running the latest Windows 10. The source files are mostly Olympus RAW files (.ORF), a small number of files are Nokia Lumia DNG files from a Lumia 1020 and 930. The Lumia 1020 files are about 40 mega pixels in resolution.

       

      For a small number of files, the tool fails to convert due to "There was not enough memory" error. This seems to only affect the large Lumia 1020 files. See screenshot.

       

      Any idea what went wrong? After this is finished (in several hours), how do I conveniently rerun the tool just on the failed files?

       

      thanks,

       

      Tobias W.

       

      notenoughmemoryadoberaw.jpg

        • 1. Re: Adobe RAW Converter failing with "There was not enough memory" error
          Mohit Goyal Adobe Employee (Admin)

          Hi polarapfel,

           

          The error you are getting because of large number of files getting converted together and the system memory is not able to handle that task.

           

          You can do this process in bits & pieces and please try and make sure that you are not running any other heavy software at time on your computer.

           

          Let us know if that helps.

           

          Regards,

          ~Mohit

          • 2. Re: Adobe RAW Converter failing with "There was not enough memory" error
            polarapfel Level 1

            I am a little confused here. My assumption is that the converter takes one source file at a time, loads it into memory and then writes the converted target file to disk. This is also visible in the status column - the converter is only processing a single file at any time. So the amount of memory needed is based on the size of that single file plus whatever space the tool needs to produce the output for that single file. With a large number of files nested in subdirectories, the tool also needs to keep that meta data in memory, so granted that adds some more memory usage. Looking into Task Manager, I can see that the DNG Converter uses in between 275MB to 285MB at any time while chunking through the list of files. CPU is at 40% to 80%. The system still has 3GB of memory available according to Task Manager. I really don't see how it shouldn't be able to perform as expected - given that it's not working in parallel on more than one file, so the number of files to work through is effectively irrelevant. With enough memory available and serial execution of files, any "There is not enough memory" situation seems like a bug to me.

             

            I'll take a look at running the converter on the command line and wrapping it with a script that calls the converter on a single file at time and creates the source list through a tree structure itself, checking on return code from each converter call to log which files failed.

            • 3. Re: Adobe RAW Converter failing with "There was not enough memory" error
              ssprengel Adobe Community Professional & MVP

              The DNG Converter, at least 8.7 which is the most recent version I have installed, is a 32-bit program so probably has less than 1GB of memory available and maybe that memory is fragmented.  Obviously if Adobe made a 64-bit version available it'd run out of memory less often, but that must not be a high priority.

               

              Will the DNG Converter skip conversion for photos that already exist in the destination?  If so you can just run it more than once.  And if you can narrow down where the Lumina DNGs are just convert those separately on the subfolders they exist in, if things are organized that way and maybe it won't run out of memory.

               

              First test to make sure a single Lumina DNG will convert.

              • 4. Re: Adobe RAW Converter failing with "There was not enough memory" error
                polarapfel Level 1

                Individual Lumia dng files convert without issues.

                 

                I am currently seeing a lot of them fail sequentially. Looking into Task Manager, I can see that the converter now uses in between 480MB and 550MB of memory when failing to convert. There are still 3GB of memory free and using 550MB memory is still only half of what you suggested might be the upper limit for a 32 bit binary on Windows.

                 

                Also, given the files are high resolution and large in size, this suggests that the converter has limits as to what files it can work with (in numbers suitable for batch processing). The 16MB Olympus raw files are no issue. So, does this mean that users with files from one of the new high resolution Sony sensors with an equal amount of pixels and even more depth (measured in bits) than the Lumia raw files will not be able to process more than a handful of files at a time with this tool?

                 

                Another suggestion that's easy enough to implement would be an option for the converter to write a log file (that can be easily parsed with other command line tools), so I don't have to run it against all files again, just to watch it burn CPU cycles skipping through the ones it detected as duplicates.

                 

                One more suggestion: make the status text field a table. It should be possible to copy and paste the content from that table.

                 

                Also, it would be a great option to tell the converter to ignore specific source raw formats. In my case for example, I have a folder tree with Lumia dng files and Olympus orf files. It would be great to run the tool recursively against a root folder, only converting files ending in orf. This could be extended to actually honor a regular expression as a filter. If this overcomplicates the UI, then just put these options into the command line version of the converter.

                 

                Also, the converter needs to be able to handle more available memory. So maybe, a 64bit version makes sense now.

                 

                thanks,

                 

                Tobias

                • 5. Re: Adobe RAW Converter failing with "There was not enough memory" error
                  ssprengel Adobe Community Professional & MVP

                  I remember when Help / System Info… in 32-bit PS or 32-bit LR, I forget which, only had 760MB or so memory available on an 8GB system, so it’s possible that 550MB + whatever new allocation is being asked for, is too much.  The theoretical maximum is 2GB, but it’s often much less than that in reality.

                   

                  You could try the conversion of a newly-booted system without running any other apps, and see if any more is available. 

                   

                  Does it help if you set the embedded JPG Preview size to something smaller?  I wonder if the DNG Converter uses the video-driver for some operations and it is somehow running out of memory. 

                   

                  Can you recursively hide all the DNG files on the source, or recursively rename their extension temporarily, to keep the DNG converter from seeing them?

                   

                  As far as Adobe making failure logs, this is just speculation based on never having seen anything like it, I’m sure the coding engineers would be all for it, but it seems against the philosophy from others in the design and implementation team.  Adobe does produce installation logs for their own troubleshooting but doesn’t produce user-consumable logs for purposes like you’re wanting. 

                   

                  The DNG Converter, for years and years, doesn’t know how to select a folder that you’ve opened it into when selecting the source, unlike almost all other software include Adobe software that can.  You’d think they’d change how it works, but it’s remained poorly implemented for a long time causing endless questions on the forums.  I highly doubt Adobe would even consider making a savable, parsable log for user developers to work with.  The DNG Converter loses them money because people can use it to bypass paid upgrades.  Unless it’s some highly-placed developer’s holiday charity project I doubt things will change with how it works even though it makes sense to us developers.

                  • 6. Re: Adobe RAW Converter failing with "There was not enough memory" error
                    polarapfel Level 1

                    Well, sometimes nagging works!

                     

                    I will probably end up wrapping the converter in a Ruby script that manages all the features I described above and calling the converter on a single input file at a time to avoid memory buildup and fragmentation. I know starting it on each file again and again creates some overhead, but at least it gives me the flexibility to work around its shortcomings.

                     

                    My main reason to use the converter in the first place is to convert and embed my non-dng files to dng, so they don't count against my Amazon CloudDrive quota on Amazon Prime. So having a Ruby script to wrap the converter is something I need anyway as I can include the upload to Amazon CloudDrive within the same script as an option. I was hoping the converter was a bit more sophisticated.

                     

                    thanks,

                     

                    Tobias W.

                    • 7. Re: Adobe RAW Converter failing with "There was not enough memory" error
                      norimbo

                      Dear Tobias did you create the script that will send only one photo to DNG converter ?

                      • 8. Re: Adobe RAW Converter failing with "There was not enough memory" error
                        norimbo Level 1

                        Dear Mohit

                        If I have 1000 photos and have to start process of conversion for each 100 photos I will have to spend whole day for this work. That's crazy and good reason to change software to other. Please prepare 64 bit version of adobe DNG converter or prepare the script which will allow to convert more photos.

                        • 9. Re: Adobe RAW Converter failing with "There was not enough memory" error
                          polarapfel Level 1

                          Hi norimbo,

                           

                          I didn't get to that yet, but I've actually started working on Lr related stuff, so I'll try to get it done in the next couple of weeks.

                           

                          Tobias