39 Replies Latest reply on Oct 7, 2009 8:59 AM by goalie2

    Adobe Updater - ASA 5510 problems

    Level 1
      Hello, the company I work for has an Cisco ASA 5510 firewall and when anybody in our company tries to update their Adobe using Adobe Updater it causes our bandwidth to peak. If it goes unchecked it will peak for as long as they're trying to run the updater, meaning that the internet is essentially down for 100+ people.

      I figure I have 2 options to try and resolve this.

      For one I can do a startup script that will disable the updater in the registry. I did this maybe 4 months ago and it didn't work like I had anticipated. Does anybody know the registry key that is used to disable it?

      I could also just block the address from our firewall. Does anybody know the actual address that adobe looks at to run its updater?

      Any help would be greatly appreciated.
        • 1. Re: Adobe Updater - ASA 5510 problems
          MichaelKazlow MVP & Adobe Community Professional
          It appears you either don't have enough bandwidth or something is
          severely misconfigured if one user updating Reader can cause the
          problems you are seeing. My bet you have a configuration problem or that
          your ISP is not providing the bandwidth you think it is.

          Mike
          • 2. Re: Adobe Updater - ASA 5510 problems
            (Aandi_Inston) Level 1
            I agree. If a single application user can flood the link, this is a
            configuration problem or fault with the router/firewall. It is
            perfectly normal for any product to make connections and to try to
            download as fast as possible - that's for routers to manage.

            Aandi Inston
            • 3. Re: Adobe Updater - ASA 5510 problems
              Level 1
              Thanks guys I appreciate your feedback but we've already spent a lot of time trying to figure out what the problem is when trying to download from our ASA without any luck. I'm sure it has something to do with the Adobe downloader, but we haven't been able to pin down exactly why it's acting that. Regardless I still want to stop Adobe from updating altogether so we still go back to my first two questions.

              "For one I can do a startup script that will disable the updater in the registry. I did this maybe 4 months ago and it didn't work like I had anticipated. Does anybody know the registry key that is used to disable it?

              I could also just block the address from our firewall. Does anybody know the actual address that adobe looks at to run its updater? "
              • 4. Re: Adobe Updater - ASA 5510 problems
                Level 1
                Hey,
                I have an ASA 5520 with the CSC module and I am experiencing exactly the same issue.
                I have tried both explicit permits and denies on the ip address range that I 'think' updater is using but it still floods out my outside interface... using every available bit of bandwidth.
                I have a 20meg link and am fully ccsp qualified so I'm afraid the blame here is pointing towards the updater architecture and not my cisco configuration. The problem with updater is that once it starts it doesn't stop.

                If anyone can help resolve this I would be extremely grateful.
                • 5. Re: Adobe Updater - ASA 5510 problems
                  Level 1
                  Oh. Forgot to add.
                  The address range I suspect updater is using is:-

                  84.53.134.0/24
                  • 6. Re: Adobe Updater - ASA 5510 problems
                    (Aandi_Inston) Level 1
                    If a single application on a single computer can flood your network,
                    no matter how badly designed the application might be, then there is a
                    flaw in the network - surely? So, I would recommend asking Cisco for
                    what protection you have against inadvertant (or deliberate) DOS.

                    Aandi Inston
                    • 7. Re: Adobe Updater - ASA 5510 problems
                      Level 1
                      'fraid not.
                      updater port hops every port from 0 - 65500.
                      This is the problem and is the reason it "holds on" to a translation connection; opening multiple ports at the same time.
                      I'm usually quick to blame cisco but not this time.
                      It is a crap app. Simple as.
                      • 8. Re: Adobe Updater - ASA 5510 problems
                        (Aandi_Inston) Level 1
                        Doesn't this mean that your network is totally vulnerable to an
                        internal DOS attack?

                        Aandi Inston
                        • 9. Re: Adobe Updater - ASA 5510 problems
                          MichaelKazlow MVP & Adobe Community Professional
                          Two users of the same hardware having the same problem, people without
                          the hardware are not having the problem. Appears to me to be a hardware
                          problem.

                          Many years ago I had a Netgear router with firewall capabilities state
                          package inspection. After having the router for years, I started having
                          problems accessing Mac/PC Connection. I called them up about having
                          problems with their site, the checked things out and we eventually tried
                          my computer without the router and the problem disappeared. I contacted
                          Netgear, they had me update the router and poof the problem disappeared.
                          I would have sworn to you the problem was the Mac/PC Connection website.
                          It was they only website I had problems with. I would have been very
                          very wrong.

                          Mike
                          • 10. Re: Adobe Updater - ASA 5510 problems
                            Level 1
                            Regardless of whether its a problem with Adobe's Updater or my firewall...how am I going to resolve this?
                            • 11. Re: Adobe Updater - ASA 5510 problems
                              (Aandi_Inston) Level 1
                              Well, the symptoms suggest that Adobe Reader is repeately retrying.
                              Presumably the router blocks the effort.

                              Since network traffic normally has to time out before anyone retries,
                              a process taking a few seconds, this suggests that the firewall is
                              dropping the connection and sending a response to that effect.

                              Maybe the firewall could just drop the connection without a response.
                              Then Reader would, perhaps, only retry at intervals.

                              Aandi Inston
                              • 12. Re: Adobe Updater - ASA 5510 problems
                                Level 1
                                We are having the exact same behavior, with a Fortigate 100 firewall. It has taken 5 weeks to track it down because it's such a moving target. Some users kill the updater when it hangs, others leave it minimized for weeks.

                                My firewall consultant said the thing Fortigate has in common with the ASA 5510 is virus checking in stream (something like that) and when he finds a way to block it, we'll post the solution. In the meantime, I'm killing Adobe Updater wherever I can on individual systems.
                                • 13. Re: Adobe Updater - ASA 5510 problems
                                  Level 1
                                  To follow up, my firewall consultant things whitelisting adobe.com on the firewall's antivirus app will clear this up. Hope this help, would love to know how other people solve it.
                                  • 14. Re: Adobe Updater - ASA 5510 problems
                                    Level 1
                                    Alright, here is what I've done... I've whitelisted adobe.com, and I touched every single computer and deleted the adobeupdater.dll file.... Problem is neither of them worked and I just had the exact same problem. There has got to be an easier way...any ideas?
                                    • 15. Re: Adobe Updater - ASA 5510 problems
                                      Level 1
                                      I made McAfee ePolicy deny access to the api file to stop it running.
                                      Only way I could do it.
                                      • 16. Re: Adobe Updater - ASA 5510 problems
                                        Level 1
                                        I'm back...Has anyone found a solution for this yet?
                                        • 17. Re: Adobe Updater - ASA 5510 problems
                                          Level 1
                                          We had no problems when our Fortinet firewall was handling duties with IDS/IPS services turned on, but as soon as we migrated the company over to an ASA 5520 two weeks ago, we all of a sudden were seeing single machines making over 50+ connections to Adobe update servers and slamming shut our Internet pipe (two T1's multi-linked). I enabled NetFlows on the I-Net router to see the top destinations and then had to setup an access list & capture on the ASA to find out who the requesting nodes were on our network. That is when I literally saw one computer take down I-Net for a 300+ user company. We have the Trend CSC-SSM piece installed and I backed off on the security settings to allow all file downloads, but it didn't make a difference. Trying to block servers other than on a temporary basis would not be smart since Adobe contracts with Level3 and others to host their content and media distributions.

                                          Any info that any one has would be greatly appreciated by myself as well.

                                          Thanks,
                                          Terry
                                          • 18. Re: Adobe Updater - ASA 5510 problems
                                            Level 1
                                            Uncheck the checkbox on the updater when you find it is running that says something to the effect 'only use bandwidth during inactivity'. This checkbox is the issue for sure, as soon as I uncheck it everything downloads in minutes and during the download process the network DOS is resolved.

                                            VERY POORLY WRITTEN FOR SURE...ANY SOLUTIONS??
                                            • 19. Re: Adobe Updater - ASA 5510 problems
                                              Level 1
                                              We too have an issue with Cisco ASA and Trend CSC modules, but I don't think its the Trend module. A workaround is to block 84.53.178.40 and 84.53.178.48 on the inside interface. That stops traffic leaving and therefore starting the connection. Now need a grouppol adm to turn off the updater.
                                              • 20. Re: Adobe Updater - ASA 5510 problems
                                                Level 1
                                                Having same problem here. It appears that Adobe Updater is not updating but it keeps trying. If you block the IP's they keep changing. Different computers are hitting different server IP for the updates.
                                                • 21. Re: Adobe Updater - ASA 5510 problems
                                                  Level 1
                                                  The IP's I popsted are the server IP's on the Internet. Blocking them stops the PC from establishing a connection, which for me stopped the ASA overloading the outside interface. Seems that those 2 IP's bombard the outside interface as a result of a client PC requesting many updates.
                                                  • 22. Re: Adobe Updater - ASA 5510 problems
                                                    Level 1
                                                    The problem is the IP addresses change... must be some form of load balancing going on.
                                                    My IPs of shame yesterday were:-

                                                    84.53.134.26
                                                    84.53.134.24
                                                    • 23. Re: Adobe Updater - ASA 5510 problems
                                                      Level 1
                                                      Not seen our server IP's change since the problem started. We have had issue on and off for nearly a year. You can't block the whole subnet as it serves content for many sites.

                                                      Upgrading the ASA to IOS V8 supports treat detection and shunning which also helps the firewall block DOS attacks.

                                                      My problem is that all devices connect via a proxy, so I may just ban ardownload.adobe.com.
                                                      • 24. Re: Adobe Updater - ASA 5510 problems
                                                        Level 1
                                                        I've finally had time to look at this today and it does seem to be the CSC module in my ASA 5520 that is causing some sort of issue.
                                                        If I remove the port-object for 80 (www or http) that is assigned to the CSC module then everything flows through nicely.

                                                        Now to work out how to fix this as the whole point of my CSC is to scan http traffic.
                                                        • 25. Re: Adobe Updater - ASA 5510 problems
                                                          Level 1
                                                          Ok heres the fix...Under CSC module in the GUI there is a command for large files. Under HTTP scanning I enabled deferred scanning for files larger than 2MB and my bandwidth went from maxed out to hardly nothing. I am running ASA 5510 with CSC module.

                                                          Talked to Cisco Tac I think they are getting a lot of calls also about this issue.
                                                          • 26. Re: Adobe Updater - ASA 5510 problems
                                                            Level 1
                                                            Have the same issue with FG400A... already blocked ardownload.adobe.com but 4 hours later seeing some more traffic. will analyze more on Monday AM. trying a method of deploying a registry hack to disable auto updater (updater5).
                                                            • 27. Re: Adobe Updater - ASA 5510 problems
                                                              Level 1
                                                              disabled antivirus for http and so far so good... seeing if this fixes it. if it does will try deferring as per above. maybe fortinet can contact adobe for poorly written code in their updater app.
                                                              • 28. Re: Adobe Updater - ASA 5510 problems
                                                                Level 1
                                                                I noticed with my troubles that it wasnt just Adobe, but also Java Updates and Symantec Endpoint Protection. It seems the client times out waiting for the antivirus module to scan the update to see if its infected.

                                                                Everything now is still running great on my ASA 5510 since I made the change on how it scans large files.
                                                                • 29. Re: Adobe Updater - ASA 5510 problems
                                                                  Level 1
                                                                  Same here.
                                                                  Raised it with Trend and Cisco. My plus lic is up in a month and I won't be renewing unless this is resolved.
                                                                  iTunes works for the privileged on the network now, as well ;-)
                                                                  • 30. Re: Adobe Updater - ASA 5510 problems
                                                                    Level 1
                                                                    Chris.Jones...Thank You! I made the same change for deferred scanning and after a few days of troubleshooting this I've finally got my bandwidth back.

                                                                    Looks like this might also be the reason I've been having problems with LiveUpdate for SEP.

                                                                    For me, the strange part is that this problem seems to have just started someime early last week. We've been running the same config on our ASA and CSC-SSM for 8+ months now. Anyone have any idea what caused the change in behavior?

                                                                    Edit: just as an additional note for anyone who might be interested, the 2 IPs that were slamming our Outside Interface were 204.213.179.200 and .201. These resolve back to deploy.akamaitechnologies.com. Since Adobe is using Akamai for distributing updates, it's not surprising that the IPs are not the same for everyone.
                                                                    • 31. Re: Adobe Updater - ASA 5510 problems
                                                                      Level 1
                                                                      I've had an ASA5510 w/ CSC for 2 months. I noticed it first on October 22nd. My bandwidth shot up after noon and was maxed out all night but the next morning it went back down to normal. After that no problems. But on November 6th my Internet bandwidth stayed pegged out all day long, the next day was a little better but Monday November 10th it was slammed again. So then I started trying solutions to figure out what was going on.

                                                                      Maybe Adobe sent the download signal out...???
                                                                      • 32. Re: Adobe Updater - ASA 5510 problems
                                                                        Level 1
                                                                        Similar issue here. The adobe update client connects through a chain of proxies (ISA performing caching, then a McAfee device performs virus checking).
                                                                        Similar issue here. The adobe update client connects through a chain of proxies, first a Microsft ISA server, then a McAfee device performs virus checking on downloaded content.

                                                                        Viewed from the workstation, the client establishes a TCP connection to ISA, then sends its GET request for the update file URL. Most recently,
                                                                        ardownload.adobe.com/pub/adobe/reader/win/8.x/8.1.3/enu/AdbeRdr813_en_US.msi

                                                                        An ACK is sent from ISA to the client. After 15 seconds, no data is received at the client, and the client sends a RST to the proxy. This is followed one second later by a new TCP connection establishment, a GET request, 15 seconds, and a RST. The pattern repeats indefinitely.

                                                                        Somewhere in our proxy chain the RST packet is not being transmitted. I suspect the McAfee device, which continues to download every requested file. Over a couple hours, a single adobe update client can request hundreds update files. With update files often in excess of 30 MB, this traffic, on top of our normal 50-70% utilization, saturates the 20MB internet pipe we have available for our 2300 users.

                                                                        The McAfee device has a data trickling feature (which I've seen in other similar web anti-virus proxies). This feature is disabled here. Data trickling allows small amounts of data to be periodically sent to waiting clients as a sort of "keep-alive". Once the whole file is downloaded and virus scanned, the remainder is transferred to the client. We're experimenting with data trickling, configured to send data to the client every ten seconds in order to prevent the 15 second timeout and retry pattern we've been seeing.

                                                                        Dan Lynch, CISSP
                                                                        Information Technology Analyst
                                                                        County of Placer
                                                                        Auburn, CA
                                                                        • 33. Re: Adobe Updater - ASA 5510 problems
                                                                          Level 1
                                                                          It appears the workaround/solution on a Fortigate is to either whitelist *adobe.com and/or change antivirus http scan threshold to 2MB... so far no issues today.
                                                                          • 34. Re: Adobe Updater - ASA 5510 problems
                                                                            Level 1
                                                                            Dears,
                                                                            I have the same problem on my ASA 5510 witch CSC SSM 10 plus licence. During work hours my internet connection goes very slow and it seems that the CSC SSM in consuming the whole bandwitdh.

                                                                            Does anybody have an ideo on how to solve this issue ?
                                                                            Thank you,
                                                                            • 35. Re: Adobe Updater - ASA 5510 problems
                                                                              Level 1
                                                                              elie1234,

                                                                              The solution was posted above by chris.jones and I have confirmed it to work. On the CSC-SSM admin page, under HTTP, you need to enable deferred scanning for files larger than 2MB. This solution was recommended to chris.jones by Cisco TAC.
                                                                              • 36. Re: Adobe Updater - ASA 5510 problems
                                                                                Level 1
                                                                                Does anyone know what settings resolve this issue on a Checkpoint UTM firewall?

                                                                                Thanks.
                                                                                • 37. Re: Adobe Updater - ASA 5510 problems
                                                                                  FCC Technician Level 1

                                                                                  Hi

                                                                                   

                                                                                  We have been experiencing this same issue with multiple client sites. The bandwidth spikes and holds at the limit of the pipe.

                                                                                   

                                                                                  At first we had thought it was a web marshal issue as when we stopped the web marshal proxy service the traffic would stop instantly but start straight back up when the proxy service was started back up.

                                                                                   

                                                                                  The sites we use have Fortigate firewalls, when you disable the protection profile the traffic drops nearly instantly.

                                                                                   

                                                                                  It appears that what is happening is the client calls for adobe updates to be downloaded but the firewall holds the file till it can download the file for a antivirus scan. While it catches the file for the scan it does not send any data to the client calling for the updates which forces the adobe updater to call for another download restarting the cycle and causing the bandwidth to get eaten alive.

                                                                                   

                                                                                  Inside of the fortigate I have been changing the scan profiles to use a setting known as comfort client.

                                                                                   

                                                                                  Comfort client is a setting where the firewall will send a certain amount of data to the clients at intervals rather than not sending any data through at all to keep the connection alive and stop it downloading via another port.

                                                                                   

                                                                                  I have set the comfort client interval time at 5 second delays and set it to send 5 bytes of data.

                                                                                   

                                                                                  So far this has resulted in a rather large reduction in the amount of incoming traffic, I will continue to monitor it over the next few weeks.

                                                                                   

                                                                                  Hopefully this helps others in the same boat.

                                                                                  • 38. Re: Adobe Updater - ASA 5510 problems
                                                                                    matt.baker Level 1

                                                                                    Just happened across this post while doing some Google-fu.  We're having the exact same problem with a Cisco PIX/Trend Micro IWSS Proxy setup.  In order to recover bandwidth we initially just blocked all traffic to ardownload.adobe.com.  After reading this, we may look at some whitelisting to see what happens.

                                                                                    • 39. Re: Adobe Updater - ASA 5510 problems
                                                                                      goalie2 Level 1

                                                                                      Matt,

                                                                                       

                                                                                      We are running Cisco PIX 515E with Trend IWSS as well and are having the same problems with bandwdth being absolutley choked a couple of times a day.  I have blocked ardownload.com on our Surfcontrol Web filter just now to see if that will help.  Did you do any whitelisting on IWSS?

                                                                                       

                                                                                      Cheers