32 Replies Latest reply on May 16, 2016 12:14 PM by EddieLotter

    Coldfusion errors with undefined element, dump during error processing shows element defined

    Steven6282 Level 1

      Ok, so I've got a strange problem that I've been beating my head against for over a week now.  I'm holding out a little hope that maybe someone here has seen something like this and has a solution or at least can point me in a direction to make some sense of it.


      We are running an enterprise application on CF9 currently.  Recently we made a large update with some advancements and in part of the advancements I'm using some request scoped variables.

       

      Said request variable is a structure that is set in Application.cfc OnRequestStart.  Randomly since the changes have been put on our production servers we get an error email (we have an error handler defined to capture and email errors when they occur), stating that key.subkey is undefined in request scoped variable.  What makes this so bizarre is in the email from the error handler (that is processed in the same request) shows the key.subkey existing in the request scoped variable in a <cfdump var="#request#"> in the email.  How could it possibly be erroring on a part of the page that runs after OnRequestStart, but before OnError, saying that this structure element doesn't exist yet it does exist during the error handler caused by it.  The request scoped variable btw is not defined or set anywhere else outside of the Application.cfc.

       

      I should also mention that other keys in the structure are referenced in the request prior to the point of failure without erroring, some in the same function as the point of error even.  It's almost as if the structure is not fully defined yet while it's processing the page request, but it is fully defined by the time it dumps it in the error email....

       

      This problem is really baffling me, and it makes it worse that I can't find any way to reproduce it on our testing servers.

        • 1. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
          Carl Von Stetten Adobe Community Professional & MVP

          Is the request.key.subkey variable perhaps empty (undefined, null)?  That will result in the struct key showing in a dump but not be able to be referenced.  You might have to wrap that code in a <cfif StructKeyExists( request.key, "subkey" )> to check.

          • 2. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
            EddieLotter Level 3

            Steven6282 wrote:

            This problem is really baffling me, and it makes it worse that I can't find any way to reproduce it on our testing servers.

            Can you reproduce it in a separate test application on the production server?

             

            Cheers

            Eddie

            • 3. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
              Carl Von Stetten Adobe Community Professional & MVP

              Also, a screenshot of the <cfdump> (with any sensitive information redacted/blurred) would be helpful.

              • 4. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                Steven6282 Level 1

                The keys are not empty. They usually have numbers in them in the dumped data.  It depends on exactly which spot the error randomly happens in as to what data is in the struct.  Occasionally there is one with an empty string, but I have verified that it is an empty string, not an undefined value.

                 

                No, sorry I should have been more specific and said I can't even reproduce it on the production server personally.  I've tried doing a lot of different things thinking maybe our users were doing something really weird to cause it to happen, but no matter how many times I refresh or try to load the page a different way I can't find a way to reproduce it.  We have over 4000 users on the production server and over 20,000 requests to the pages that can generate the error per day.  But the error only happens maybe 50 to 100 times a day, so it's a rare hard to reproduce error and obviously doesn't happen every single request.

                 

                Unfortunately I can't practically screenshot any of the data and share it on the internet.  This is a health record application so I would have to redact nearly everything in the dump to share a screenshot without violating HIPAA laws.

                 

                I can give you a little more details on the structure with dummy data though.  Essentially the point of this request scoped variable is to load in client data that is being used across multiple files in the request.  So for example one spot that it commonly errors on the structure it says is undefined is

                 

                Request.Client.Admission.CSMG (csmg being case manager in this case, and is always a number, a client cannot have an admission without having an assigned case manager in the system).

                 

                In the dump it will show Request -> Client -> Admission -> CSMG: 9999, but the point of error says it's not defined.  And two lines of code before the point of error there is a reference to Request.Client.Admission.ADMNO (admission number), that works fine without erroring.

                 

                I tried adding a <cflock scope="session" around the <cfset request.Client = ... > tag today just to be sure, and it did not make a difference.

                • 5. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                  EddieLotter Level 3

                  Have you reviewed all the ColdFusion log files? It's possible that they might contain a clue as to what might be happening.

                   

                  I know it is a production server, but since you're experiencing the error anyway, I recommend you try to access the key immediately after assigning a value to it. That way, when the problem occurs, it will be informative to note where it happens. If it doesn't happen immediately after the assignment then it will be a matter of figuring out what is happening to the key between the assignment and when you access it later in your application.

                   

                  Cheers

                  Eddie

                  • 6. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                    Steven6282 Level 1

                    I have looked in the logs and have not seen anything relevant near the same time stamp as the error.

                     

                    It's random which key it complains about being undefined, so there isn't any single specific key I can output immediately after the assignment.  The main object .Request.Client is a entire structure be assigned in one <cfset>.  I have already verified all the keys in that structure are accurate at structure creation time.

                    • 7. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                      EddieLotter Level 3

                      Steven6282 wrote:

                       

                      The main object .Request.Client is a entire structure be assigned in one <cfset>.

                      I don't understand what you mean by this. Can you give a practical example?

                      Steven6282 wrote:

                       

                      I have already verified all the keys in that structure are accurate at structure creation time.

                      By this, do you mean you verified that each key has a valid value?

                       

                      Cheers

                      Eddie

                      • 8. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                        EddieLotter Level 3

                        It occurs to me that you are talking about assigning one structure to another. This is probably the root of your problem because you're getting a reference instead of a copy of the structure or some of its keys.

                         

                        Use the Duplicate() function to ensure that you get a deep copy when assigning one structure to another.

                         

                        Cheers

                        Eddie

                        • 9. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                          BKBK Adobe Community Professional & MVP

                          Steven6282 wrote:

                           

                          What makes this so bizarre is in the email from the error handler (that is processed in the same request) shows the key.subkey existing in the request scoped variable in a <cfdump var="#request#"> in the email. How could it possibly be erroring on a part of the page that runs after OnRequestStart, but before OnError, saying that this structure element doesn't exist yet it does exist during the error handler caused by it.

                          Your application might inadvertently be triggering an additional request. Although the variable is read in onRequestStart the error occurs where Coldfusion writes the variable.

                           

                          To get information about where that happens, replace the dump of the request scope (in onError) with:

                           

                          <cfdump var="#arguments.exception#">

                          • 10. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                            Steven6282 Level 1

                            EddieLotter wrote:

                             

                            Steven6282 wrote:

                             

                            The main object .Request.Client is a entire structure be assigned in one <cfset>.

                            I don't understand what you mean by this. Can you give a practical example?

                            Steven6282 wrote:

                             

                            I have already verified all the keys in that structure are accurate at structure creation time.

                            By this, do you mean you verified that each key has a valid value?

                             

                            Cheers

                            Eddie

                            In another part of the code these structures are built and stored in a session variable, there can be multiples and a url parameter in the request determines which one is used for that request.

                             

                            So in another part of the code it does a query against a database and does something like:

                             

                            <cfset var storedVariable = {}>

                            <cfloop list="#structKeyList( query.colList )#" index="column">

                                 <cfset structInsert( storedVariable, column, query[ column ] )>

                            </cfloop>

                             

                            This isn't exactly the same it's just an example.  There is some additional processing done on the query data before it's stored in the structure, like formatting dates, trimming strings, etc.  These are also 1 record length queries for the record, that is ensured before the code runs.  In this code I have output all the variables post storing them in the structure and verified them as valid accurate values.

                             

                            We are essentially doing this to cache the processed data for the session.

                             

                            I cannot use duplicate because this should be a reference assignment not a copy.  There are times in the code when the structure will be updated in the request scope and it should update the original session structure.  I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being.  Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

                             

                            This is not happening where the variable is being written.  I already have the exception as part of my dump data.  The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

                             

                            Also just to note, I don't see how this could sometimes be multiple requests.  Some of the spots where it errors happens through an XMLHttpRequest.  That can't be more than one request serverside and still return accurate data afaik.

                            • 11. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                              EddieLotter Level 3

                              Steven6282 wrote:

                               

                              I cannot use duplicate because this should be a reference assignment not a copy. There are times in the code when the structure will be updated in the request scope and it should update the original session structure. I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being. Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

                               

                              I think I'm getting a better idea of what is happening. If you are not protecting the reads and updates to the session variables with cfLock then chances are good that some of your users are opening multiple browser tabs/windows which is causing overlapping updates to the session variables. Taking a reference to a session variable and putting it in the request scope does not alleviate the need to do locking.

                               

                              Cheers

                              Eddie

                              • 12. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                Steven6282 Level 1

                                EddieLotter wrote:

                                 

                                Steven6282 wrote:

                                 

                                I cannot use duplicate because this should be a reference assignment not a copy. There are times in the code when the structure will be updated in the request scope and it should update the original session structure. I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being. Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

                                 

                                I think I'm getting a better idea of what is happening. If you are not protecting the reads and updates to the session variables with cfLock then chances are good that some of your users are opening multiple browser tabs/windows which is causing overlapping updates to the session variables. Taking a reference to a session variable and putting it in the request scope does not alleviate the need to do locking.

                                 

                                Cheers

                                Eddie

                                As I said,

                                The updates are currently disabled and the errors still occur.  Where the main structure is original built at is in in a session scope lock.

                                • 13. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                  EddieLotter Level 3

                                  As I said, you need to lock your reads as well.

                                   

                                  Cheers

                                  Eddie

                                  • 14. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                    Steven6282 Level 1

                                    Lock a read?? I do not understand the purpose for that?  Can you please explain what that would accomplish?  If the data were possibly being modified somewhere else simultaneously I could understand, but since in this case it's not possible, I'm not really following what that would achieve.  I mean I can put a read-only lock around one of the more common spots I see the error to satisfy testing it, but I'd still like to understand the idea behind the necessity of it.

                                    • 15. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                      EddieLotter Level 3

                                      Steven6282 wrote:

                                       

                                      Lock a read?? I do not understand the purpose for that? Can you please explain what that would accomplish? If the data were possibly being modified somewhere else simultaneously I could understand, but since in this case it's not possible, I'm not really following what that would achieve. I mean I can put a read-only lock around one of the more common spots I see the error to satisfy testing it, but I'd still like to understand the idea behind the necessity of it.

                                      Why do you think it's not possible? When a user opens more than one tab in their browser and make requests to your Web server, the server has no idea which tab the request is coming from. The same session scope is being used for all those tabs and they can interfere with each other. It's for this reason that cfLock exists.

                                       

                                      For more information, and to better understand what ColdFusion is doing, read Using Persistent Data and Locking in the CFML Reference Guide.

                                       

                                      Cheers

                                      Eddie

                                      • 16. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                        Steven6282 Level 1

                                        Because I have code in place that preserves the integrity session data across tabs exactly for that reason.  Using HTML5 sessionStroage which is tab specific, and creating a unique tab id, to pass along with requests the session data is scoped specific to that tab.

                                        • 17. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                          EddieLotter Level 3

                                          What happens when your users don't use an HTML5 capable browser?

                                           

                                          Cheers

                                          Eddie

                                          • 18. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                            Steven6282 Level 1

                                            Those users don't exist for us.  This is an internal business application where all user PCs are on the domain and their software is managed by the domain.  All of our users currently use IE11.  The effect pages on the website would not work correctly at all in a browser that doesn't support HTML5, there are canvas used in some places and other HTML5 elements.

                                            • 19. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                              EddieLotter Level 3

                                              For the record, I am not saying that this is the root cause of the problem. What I am saying is that not locking access to session variables can cause the intermittent and spurious problems that you are experiencing, which is why I brought it up. It is in your best interests to rule it out as a possible cause by using ColdFusion the way that it is intended to be used.

                                               

                                              Intermittent problems are among the worst to troubleshoot. You have been given ideas by several people in this thread and I don't think there is anything more we can do to help.

                                               

                                              Good luck with the bug hunt. If you get to the bottom of it, it will be helpful to the community if you report back here about it.

                                               

                                              Cheers

                                              Eddie

                                              • 20. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                Steven6282 Level 1

                                                Ok well thanks for trying.  I figured it was to obscure of a problem.  That is usually what happens these days, by the time I need to ask for help, it's to weird or complex of a problem for there to be any ready solution

                                                 

                                                That is also why we don't bother contacting adobe support anymore.  By the time we get to a point that we can't solve an adobe problem on our own, it completely baffles their tier 1 support and they will have us chasing our tales or giving us inaccurate information for months lol.  Last time we contacted them about a known page break bug in cfdocument, after 8 months of back and forth and them not being able to come up with a solution they finally just said that it was a 3rd party issue and they would no longer support it haha.  Talk about bad support... if it's included in your product you can't just write it off as a 3rd part issue.

                                                 

                                                Anyway, I digress.  Thanks again for trying, if I do find a solution I will come back and post it, but it's looking more and more likely that I'm going to have to chalk it up to some obscure CF bug, won't be the first one and probably not the last either.  Already found a couple of others while working on this project, like the JSON bug and CF sending strings as numbers causing a json parse error ( "0000" for example will be output in the json string as parameter: 0000 instead of parameter: "0000" ).

                                                • 21. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                  BKBK Adobe Community Professional & MVP

                                                  Did you use my suggestion and dump arguments.exception instead of request?

                                                  • 22. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                    Steven6282 Level 1

                                                    BKBK wrote:

                                                     

                                                    Did you use my suggestion and dump arguments.exception instead of request?

                                                     

                                                     

                                                    Yes

                                                    Steven Hickerson wrote:

                                                     

                                                    This is not happening where the variable is being written.  I already have the exception as part of my dump data.  The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

                                                    • 23. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                      BKBK Adobe Community Professional & MVP

                                                      Ah, in your response to EddieLotter, which I only skimmed through. Thanks for the repeat.

                                                       

                                                      I think, rather than find a workaround, we should determine the cause. For if we cannot, then this will point to a bug.

                                                      • 24. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                        BKBK Adobe Community Professional & MVP

                                                        Steven Hickerson wrote:

                                                         

                                                        This is not happening where the variable is being written. I already have the exception as part of my dump data. The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

                                                        Could you, for the sake of clarity, show us the line at which the error occurs.

                                                        • 25. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                          Steven6282 Level 1

                                                          I have a theory on the "cause" but it's just a theory.  I've been able to duplicate it one time out of hundreds of attempts.  I think what is happening is the one and only time that coldfusion should be writing the object, it's progressing to the next line of code before the write is fully complete in memory.  So if the system is for some reason slow about completing the write to memory or it processes other code fast enough to try and read from memory before the write is complete it gets the error.  This might be a behavior of Java behind the scenes, I don't know.


                                                          Like I said I've been able to duplicate it once with a structure that was made intentionally very large (I stuffed a few 100 MB binary blobs in it and 500 other variables).  I then set a request variable = to it, and in the next part of the request attempted to access the last element in the structure.  One time out of at least over a hundred refreshes on that test page I was able to get an undefined error.

                                                          • 26. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                            BKBK Adobe Community Professional & MVP

                                                            It is more likely that a second request is involved. It attempts to access the last element in the structure while the first request is still writing to the structure. If so, then a solution would be to put a lock, of exclusive type, around the first code.

                                                            1 person found this helpful
                                                            • 27. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                              EddieLotter Level 3

                                                              BKBK wrote:

                                                              a solution would be to put a lock, of exclusive type, around the first code.

                                                               

                                                              That's what I recommended earlier in this thread, but Steven seems averse to locking code even though it ensures data integrity. Locking really does have a valid purpose.

                                                               

                                                              Cheers

                                                              Eddie

                                                              • 28. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                                BKBK Adobe Community Professional & MVP

                                                                So you did. As they say, EddieLotter, when you're right you're right.

                                                                • 29. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                                  EddieLotter Level 3

                                                                  BKBK wrote:

                                                                   

                                                                  As they say...

                                                                  Who's been talking behind my back??

                                                                   

                                                                  Cheers

                                                                  Eddie

                                                                  • 30. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                                    Steven6282 Level 1

                                                                    EddieLotter wrote:

                                                                     

                                                                    BKBK wrote:

                                                                    a solution would be to put a lock, of exclusive type, around the first code.

                                                                     

                                                                    That's what I recommended earlier in this thread, but Steven seems averse to locking code even though it ensures data integrity. Locking really does have a valid purpose.

                                                                     

                                                                    Cheers

                                                                    Eddie

                                                                    Actually, you might want to check the definition of adverse because I said I would add the cflock to see if it made a difference.

                                                                     

                                                                    Further, in the test case I described above that I was able to reproduce the error in, there was an exclusive lock around the write and a readonly lock around where the error occurred.  So the locks apparently didn't help.

                                                                     

                                                                    I'm also 100% positive in the test case there was a single request.

                                                                    • 31. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                                      BKBK Adobe Community Professional & MVP

                                                                      I do believe we're missing something. What does Coldfusion say in the dump of arguments.exception ? What is the error message literally? Even better, could you share the stack trace with us, together with the offending code block?

                                                                      • 32. Re: Coldfusion errors with undefined element, dump during error processing shows element defined
                                                                        EddieLotter Level 3

                                                                        Steven6282 wrote:

                                                                        Actually, you might want to check the definition of adverse because I said I would add the cflock to see if it made a difference.

                                                                         

                                                                        I said you were averse to doing it, not that you wouldn't do it, based on your response to my mentioning it. Take a deep breath, we're trying to help you here.

                                                                         

                                                                        Steven6282 wrote:

                                                                        Further, in the test case I described above that I was able to reproduce the error in, there was an exclusive lock around the write and a readonly lock around where the error occurred. So the locks apparently didn't help.

                                                                        If you have managed to create a test case that can reproduce the problem, then post that code so that others can test it in their environments and report back with their findings.

                                                                         

                                                                        Cheers

                                                                        Eddie