30 Replies Latest reply: Jun 8, 2012 5:59 AM by BKBK RSS

    Duplication of emails using cfmail

    Gary__F Community Member

      Hi. I'm having a problem with cfmail sending out the same email twice. The mailsent.log shows the same emails sent again at exactly the same time to the very second. It's not caused by double-clicking the submit button, it happens with a single click. There are no loops in my code, not even a query or a cfc. It only happens on our prd servers, not our dev server. The main differences is prd uses a multi-instance installation and dev does not, plus dev doesn't have all the hotfixes that prd has.

       

      What could cause CF to send the same email twice within the same second? Someone reported this 4 years ago in this thread but there was no resolution!

       

      Setup: CF8.01 Ent with hotfixes, Win2008 R2, IIS7.5, Microsoft SMTP server.

       

      Thanks in advance.

        • 1. Re: Duplication of emails using cfmail
          Gary__F Community Member

          BUMP!

           

          To help debug this I'm now generating a big random number at the top of my code and another random number within the cfmail tag itself. Both random numbers are inserted into the email message. If the page is called twice or cfmail is looped you'd expect to see the duplicate emails showing the same random numbers. This has not been the case. All numbers are totally unique. Therefore the page and cfmail is run once only.

           

          Some emails have been sent in triplicate by the way! So something weird is happening outside of my code. CF is logging each duplicate in mailsent.log so it's not the SMTP server at fault.

           

          The code is no more complicated than this after a form post.

           

          <cfif IsDefined("form.email")>

          <cfset emailTo="admin@mydomain.com">

          <cfset randomID=RandRange(100,1000000,"SHA1PRNG")>

          <cfmail to="#emailTo#" from="server@mydomain.com" replyto="#form.email#" subject="Test message" type="html" failto="failedmail@mydomain.com" mailerid="My Special Mailer">

          Dear Administrator,<br>

          Blah blah...<br><br>

          #form.message#<br>

          <!--- insert randomID generated outside of cfmail, then insert a random number generated inside cfmail --->

          Page ref: #randomID#. Mail ref: #RandRange(100,1000000,"SHA1PRNG")#

          </cfmail>

          </cfif>

           

          Ignore the weirdly displayed email addresses, this forum keeps turning them into funny links.

          • 2. Re: Duplication of emails using cfmail
            BKBK MVP

            Strip the code down to its most basic, such as below. Does the issue persist?

             

            <cfif IsDefined("form.email")>

            <cfmail to="admin@mydomain.com" from="server@mydomain.com" replyto="#form.email#" subject="Test message" type="html" failto="failedmail@mydomain.com" mailerid="My Special Mailer">

            test mail

            </cfmail>

            </cfif>

            • 3. Re: Duplication of emails using cfmail
              Gary__F Community Member

              I've simplified the code to just the cfmail tag and I still get duplicate emails. I just worked out that it only happens from one of our CF servers. Let's call it server 2. When the pages are served by server 1 then I don't get any duplicate emails. Clearly it's a server specific issue.

               

              Both servers are behind a hardware load balancer using sticky sessions so a visitor will always stay with the same CF server. The servers are identical in terms of hardware, OS, CF version and configuration. The cfm files are on a DFS network share so both servers use the same files.

               

              Why would one CF server send & log the same email several times, but it's twin sister does it once?

              • 4. Re: Duplication of emails using cfmail
                BKBK MVP

                Strange. Are you using any form of session replication? Is it possible 2 or more session threads open the action page?

                 

                Run the following test. Do you get duplicate log entries? What test values appear in the logs? (I have assumed the form page and action page are not the same. If they are, use <cfparam> instead.)

                 

                On the form page:

                <cfset session.mailTest = 0>

                 

                On action page:

                <cfif IsDefined("form.email")>

                <cfset session.mailtest=session.mailTest+1>

                <cflog text="mail test: #session.mailTest#" file="mailTest">

                </cfif>

                • 5. Re: Duplication of emails using cfmail
                  Gary__F Community Member

                  I'm not using session replication, the servers are blissfully unaware of each other's existence and sessions. I tried your test code and got the email in triplicate and a single log entry: "mail test: 1".

                   

                  Thanks for your time thinking of ideas, BKBK. Any other thoughts please?

                  • 6. Re: Duplication of emails using cfmail
                    BKBK MVP

                    Could we have a look at the form page?

                    • 7. Re: Duplication of emails using cfmail
                      Gary__F Community Member

                      Sure, here's the actual stripped down page that's causing 2 to 5 duplicaton emails to be sent most of the time, but it only happens on 1 of our servers.

                       

                      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
                      <HTML>
                      <BODY>

                      <cfif IsDefined("form.email")>

                       

                      <cfmail to="test@mydomain.com" from="noreply@mydomain.com" replyto="#form.email#" subject="dupe email test" type="html" failto="failedmail@mydomain.com" mailerid="My Special Mailer">
                      Test email.
                      <cfoutput>Mail ref:#RandRange(100,1000000,"SHA1PRNG")#</cfoutput>
                      </cfmail>
                      Thank you for submitting the form.<br>
                      <cfelse>
                      <FORM name="form2" id="form2" method="post" action="samepage.cfm">
                      <INPUT name="email" type="text" id="email" size="30">
                      <INPUT name="submitBttn" id="submitBttn" type="submit" value="Submit">     
                      </FORM>
                      </cfif>

                       

                      </BODY>
                      </HTML>

                      • 8. Re: Duplication of emails using cfmail
                        BKBK MVP

                        Just bear with me. What is the result when you use the following, even more basic, code?

                         

                        <cfif IsDefined("form.email")>

                            <cfmail to="test@mydomain.com" from="noreply@mydomain.com" subject="dupe email test">

                            Test email.

                            </cfmail>

                        <cfelse>

                            <cfoutput><FORM name="form2" id="form2" action="#cgi.script_name#"></cfoutput>

                            <INPUT name="email" type="text" id="email" size="30">

                            <INPUT name="submitBttn" id="submitBttn" type="submit" value="Submit">    

                            </FORM>

                        </cfif>

                        • 9. Re: Duplication of emails using cfmail
                          Gary__F Community Member

                          BKBK, your code also sends duplicate emails from "server 2". It sounds like you had another idea of what it could be?

                          • 10. Re: Duplication of emails using cfmail
                            BKBK MVP

                            On reading up on the issue, I found that the possible causes could be:

                             

                            1) Page being requested twice. We have ruled that out(cflog ran once only, session counter not incremented). To be sure, we had to rule out double-clicking, too. Let's do that now.

                             

                            Does the following code (just a CFM page, no form submission) send duplicate e-mails?

                             

                            <cfmail to="test@mydomain.com" from="noreply@mydomain.com" subject="dupe email test">

                                Look, no submit button!

                                </cfmail>

                             

                            2) Type="HTML", possibly together with form post from Internet Explorer. We have ruled out e-mail type. Not yet the browser type. Does issue occur with Firefox?

                             

                            3) Mail server. Open the source file of each e-mail in a set of duplicates. Read off the message IDs. If they are the same, then the mail server is likely causing the issue. If they are distinct, the ColdFusion server is likely causing the problem, as it must have instructed the mail server multiple times to send an e-mail.

                             

                            4) When a similar issue was discovered in Railo last year, the following workaround seemed to help: spoolenable="no". Hence, I would test with

                             

                            <cfmail to="test@mydomain.com" from="noreply@mydomain.com" subject="dupe email test" spoolenable="no">

                            • 11. Re: Duplication of emails using cfmail
                              Gary__F Community Member

                              1) A plain old cfmail tag without a form post generated duplicate emails.

                               

                              2) Using Firefox and Chrome also caused duplicate emails. We know the problem only ocurrs when a specific server is being used, other servers don't have this problem.

                               

                              3) CF is logging each duplicate email in its mailsent.log file so we know the duplication is taking place before mail is sent to the mail servers. The Message-ID in the received duplicate emails is different.

                               

                              4) After adding spoolenable="no" there was no more duplicate email coming from server 2. (Partial yay!) Does this suggest a bug with CF's spooling system? The golden question is what could be causing this problem to surface on one server but not on any others in the same environment with identical configurations?

                               

                              Are there any other drawbacks of using spoolenable="no" apart from losing unsent emails if the server crashes before they're sent to the SMTP server?

                               

                              The server sends bulk mail in another part of the application using cfmail query="myquery", but this has never generated duplicate emails. About 3 million emails a year are sent using this particular method (non-spam I must add!) compared to just 30,000 1-off emails that don't use a query or a loop. I find that quite interesting and wonder if it provides a clue to the cause of the duplication?

                               

                              Thanks for your help, BKBK. I will disable spooling for the 1-off emails as a temporary fix for now.

                              • 12. Re: Duplication of emails using cfmail
                                Adam Cameron. Community Member

                                It sounds to me like you've got an additional (orphaned) jrun process running on that server.  Sometimes if a CF instance seems to crash, JRun will start it up again, however the original one sometimes also recovers, leaving two CF instances servicing each request.

                                 

                                How many JRun processes have you got running?

                                 

                                --

                                Adam

                                • 13. Re: Duplication of emails using cfmail
                                  Gary__F Community Member

                                  Hi Adam. I checked the processes and there's 1x jrun.exe and 1x jrunsvc.exe on each of the CF servers. That's useful info to know but it doesn't look like that's happened here from what I can see.

                                   

                                  You've just given me a thought. We used to have problems with CF's MailSpoolService failing so I use a schedule script to check if we have aging unsent mail and if so restart the service. The code to restart is:

                                   

                                  <cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>

                                  <cfset MailSpoolService = sFactory.mailSpoolService>

                                  <cfset MailSpoolService.stop()>

                                  <cfscript>sleep(2000);</cfscript>

                                  <cfset MailSpoolService.start()>

                                   

                                  Is it possible that there are several mailspool services running if one recovered by itself after dying? As these are Java services and aren't shown in Windows Task Manager is there another way to check?

                                  • 14. Re: Duplication of emails using cfmail
                                    Adam Cameron. Community Member

                                    I have never had any luck with that tactic... all I ever ended up with is multiple mail spool services running...

                                     

                                     

                                    That said, this does not really explain the situation you mentioned wherein each mail message had its own unique random number.  CF deals with the random number side of things, the mail spooler just processes the mail.  So if you had one CF process and multiple mail processes, you'd get multiple mail messages with the same CF-generated random number (if that makes sense).

                                     

                                    It definitely looks like CF is processing those templates more than once.

                                     

                                    Dunno what further to suggest at the moment, but I'm giving it some thought.

                                     

                                    --

                                    Adam

                                    • 15. Re: Duplication of emails using cfmail
                                      Gary__F Community Member

                                      Adam, I probably didn't explain the random number test result properly where a random number is inserted into each cfmail message. Each duplicate of the same email message had an identical random number. This proves CF runs the script once despite multiple emails being created by CF. Otherwise a different random number would be inserted into each email if the script ran more than once.

                                       

                                      (Not to be confused with the Message-ID inserted into the email header which is different for each duplication of the same email.)

                                      • 16. Re: Duplication of emails using cfmail
                                        Adam Cameron. Community Member

                                        Ah right.  OK, that points to perhaps multiple mail spooler processes runing, perhaps caused by (or at least not helped by ~) your code above.

                                         

                                        Can you get rid of that code, and just restart the CF server that's causing the problem?

                                         

                                        I'm guessing (and it's just a guess) that the mail spooler runs as some sort of daemon within the JVM, so restarting CF should get rid.

                                         

                                        It sounds like you've got more than one prod box, so this should be OK to test at some stage, shouldn't it?

                                         

                                        --

                                        Adam

                                        • 17. Re: Duplication of emails using cfmail
                                          Gary__F Community Member

                                          If I get rid of the code to restart the mailspoolservice then it would mean restarting the whole server if there's a problem. That would mean users losing their sessions and being kicked out and requests being killed. That's not really acceptable and would invoke complaints or support tickets if that happens. As we moved away from CF's session replication we can't seemlessly transfer sessions from one server to another. (Not that CF ever did that seemlessly when we did use session replication!)

                                           

                                          The service factory has some other functionality that might reveal info about how many mailspoolservices are running. It's a black art and isn't documented but I found some more args that might help:

                                          http://www.cflib.org/udf/cfMailFactory

                                           

                                          As an experiment I'll reboot the "faulty" server tonight and kill the schedule task so the mailspoolservice isn't restarted, then test to see if it duplicates emails.

                                           

                                          Thanks.

                                          • 18. Re: Duplication of emails using cfmail
                                            Gary__F Community Member

                                            I rebooted server 2, disabled the script that stops/starts the mailspoolservice and got it to send a few emails without the spoolenable="no" setting. So far I've not seen any duplicate emails.

                                             

                                            If we've identified the cause of the problem by disabling my schedule script then I need to find a better way of detecting a dead mailspoolservice and restarting it. I played around with the servicefactory and there are no functions that disclose how many instances are running or if there are no instances running.

                                            • 19. Re: Duplication of emails using cfmail
                                              BKBK MVP

                                              Gary__F wrote:

                                               

                                              I rebooted server 2, disabled the script that stops/starts the mailspoolservice and got it to send a few emails without the spoolenable="no" setting. So far I've not seen any duplicate emails.

                                               

                                              If we've identified the cause of the problem by disabling my schedule script then I need to find a better way of detecting a dead mailspoolservice and restarting it. I played around with the servicefactory and there are no functions that disclose how many instances are running or if there are no instances running.

                                              In view of the recent developments, my take on it now is that the cause was a fault in the spooling mechanism. It resulted in each e-mail being spooled twice or more.

                                               

                                              I think the server restart (essentially a reset) is a sufficient solution. I would further do nothing with the Mailspoolservice.

                                              • 20. Re: Duplication of emails using cfmail
                                                Adam Cameron. Community Member

                                                I need to find a better way of detecting a dead mailspoolservice and restarting it. I played around with the servicefactory and there are no functions that disclose how many instances are running or if there are no instances running.

                                                 

                                                I wonder if it's possible to set up a second CF instance that simply handles the mail spooling?

                                                 

                                                Looking @ neo-mail.xml, one could perhaps set the number of mail threads on your prod box to be zero (one cannot set it to zero via the CFAdmin UI), and then on the second instance point its spool dir to be your prod instance's mail dir.  This might cause the prod instance to still write the files to the spool dir, but not try to send them (this is predicated on the spool threads also not having a hand in getting the files to the dir in the first place, I guess), and the second instance will send 'em.  And if the second instance stops working, then it's low-impact to restart it.

                                                 

                                                Is that an option?

                                                 

                                                And, hey: I dunno if that'll work, and I don't have the resources here to test it, sorry.

                                                 

                                                --

                                                Adam

                                                • 21. Re: Duplication of emails using cfmail
                                                  Gary__F Community Member

                                                  @Adam, that's a good idea. Perhaps slightly off the wall but I follow your logic! I could probably script the new instance to reboot itself if emails in the spool dir starts to age. I'll give that a try. Thank you very much.

                                                   

                                                  @BKBK, yes it's the flaky spooling service that's probably running twice. It's a shame it can't be more finely controlled, or at least probed to see how many instances are running before my script restarts it. Thanks for your ideas and tests. I'll give Adam's suggestion a try and see if it works.

                                                   

                                                  My long term goal is to write my own pre-spooling system that puts requests to create a new email into the db so I can control how many at a time get generated by CF. If CF or its spooling service crashes nothing will be lost because the db would track which emails have been generated and sent. It could run as a new CF instance.

                                                   

                                                  I wonder if the mailspoolservice has been updated (made more resilient) in CF9 or CF10? I don't recall any documentation to say that it has been.

                                                  • 22. Re: Duplication of emails using cfmail
                                                    BKBK MVP

                                                    Gary__F wrote:

                                                     

                                                    Setup: CF8.01 Ent with hotfixes ...

                                                    On an entirely different note now, which hotfixes did you install (on server 2 in particular)? There were some hotfixes for CF8.0.1 to fix cfmail issues.

                                                    • 23. Re: Duplication of emails using cfmail
                                                      Gary__F Community Member

                                                      I installed all hotfixes for 8.01 that addressed cfmail issues. CF Admin tells me there 2 are installed: hf801-00001 and chf8010004. I'm sure I installed more but maybe the little security fixes don't show up on the info page.

                                                       

                                                      I've had several mail related issues over the years with CF; some were fixed using a workaround, others with a hotfix.

                                                      • 24. Re: Duplication of emails using cfmail
                                                        BKBK MVP

                                                        Gary__F wrote:

                                                         

                                                        I installed all hotfixes for 8.01 that addressed cfmail issues. CF Admin tells me there 2 are installed: hf801-00001 and chf8010004.

                                                        I spotted a problem. ColdFusion hotfixes of these types are cumulative. I do believe that you had to delete the JAR file for hotfix 1 before installing hotfix 4. See Step 6 of the CF8.0.1 hotfix 4 installation notes. In any case, I suppose it is all right to stop ColdFusion, reverse both hotfix 1and hotfix 4 installations, and then reinstall hotfix 4.

                                                        • 25. Re: Duplication of emails using cfmail
                                                          Gary__F Community Member

                                                          Ah, that's interesting! I'm a bit confused with the file naming conventions.

                                                           

                                                          I downloaded the hotfix file from the URL in your last post and it's called hf801-00004.jar and is claimed to be a security hotfix. However, there's also a hotfix file called chf8010004.jar and the download page for it lists all of the bugs it fixes from hotfix 1 through to 4. I think you linked to a security hotfix and the file I've got installed is the cumalative hotfix that addresses bugs (like a cfmail bug) rather than security issues.

                                                           

                                                          The file dates and sizes are different:

                                                          • hf801-00004.jar 492,516 bytes, 15-Nov-2011
                                                          • chf8010004.jar  648,997 bytes, 15-Nov-2009

                                                           

                                                          I believe these files should be installed on the same server because one is for security fixes and the other is for bug fixes (and a couple of minor enhancements). Would you agree or have I got this wrong?

                                                          • 26. Re: Duplication of emails using cfmail
                                                            BKBK MVP

                                                            Look at the last digit, the hotfix number. You are talking about 4 and 4. I was actually talking about the presence of 1 and 4 on the same server.

                                                            • 27. Re: Duplication of emails using cfmail
                                                              Gary__F Community Member

                                                              From what I've read the files "chf*" are patches for bug fixes and enhancements and the "hf*" files are security patches. So I need one of each file installed. i.e. there is no conflict so long as there's only one chf* file and one hf* file installed. I think the latest chf* number is 4 (from 2009) and the latest hf* number is 5 from March 2012.

                                                              • 28. Re: Duplication of emails using cfmail
                                                                BKBK MVP

                                                                This is important. Why not ask Adobe whether the JARs hf801-00001 and chf8010004 should be installed simultaneously.

                                                                • 29. Re: Duplication of emails using cfmail
                                                                  Gary__F Community Member

                                                                  I would, but don't Adobe still charge $500 to open a support ticket so you can speak to a junior who'll mess you around with useless answers until they eventually go and speak to a more senior support person? (Not cynicism but based on past experiences)

                                                                  • 30. Re: Duplication of emails using cfmail
                                                                    BKBK MVP

                                                                    OK. Let's research it ourselves and save the $ for beer.