22 Replies Latest reply on Feb 14, 2010 6:49 AM by Andre Linux

    stratus.adobe.com is down

    Andre Linux Level 1

      Hello

      Today I have noticed that adobe stratus service (rtmfp://stratus.adobe.com/) is down.

      After significant delay of "connecting" state (80 seconds) or so I get the following error:

      ScriptDebug: p2pConnectionHandler event: NetConnection.Connect.Closed

      Earlier this day the service worked just fine.

      My NAT is configured properly and UDP traffic is not blocked.
      Just for checking, http://cc.rtmfp.net/ connects just fine.

      I also can't ping stratus.adobe.com

      We also can't connect from another computer which is situated in another country.

       

      Screenshot attached:

      service.png

      Thanks a lot, your service is great and we really appreciate that you provide this service for free, I am really looking forward to resolve this issue!

       

        • 1. Re: stratus.adobe.com is down
          Michael Thornburgh Adobe Employee

          oh dear.  it looks like there was a power failure at the data center where Stratus is located.  that's my guess at least, since all of the servers went down at the same time.

           

          i've restarted Stratus.  it looks like the service went down at around 22:45 PST saturday and was restarted at 00:42 PST sunday (about two hours later).  that's really gonna cut into our nines!

           

          the preliminary report from the facility is that one or more UPS units failed.

           

          -mike

          • 2. Re: stratus.adobe.com is down
            Andre Linux Level 1

            Thanks for fixing it up

            • 3. Re: stratus.adobe.com is down
              Andre Linux Level 1

              Adobe Stratus service is down again!

              • 5. Re: stratus.adobe.com is down
                evanFLASH Level 1

                Makes me a little nervous building rtmfp into my app and relying on stratus..even if it's just for executive demos at this point, i wish it would be more reliable :-\

                • 6. Re: stratus.adobe.com is down
                  Andre Linux Level 1

                  Indeed. It used to work very well, but it is offline for an hour already

                  • 7. Re: stratus.adobe.com is down
                    Michael Thornburgh Adobe Employee

                    we are investigating right now.  all of the servers in the Stratus cluster seem to have run out of memory/swap at the same time.  (8:11am pacific time).

                     

                    i'm restarting one of them to get the service going again, but i want to understand this failure before i reboot any of the others. 

                     

                    -mike

                    • 8. Re: stratus.adobe.com is down
                      blitter Level 1

                      Thanks for bringing Stratus back up. I want to voice my support, encouragement appreciation to the development  team.

                      Cheers,

                      Arnie

                      • 9. Re: stratus.adobe.com is down
                        evanFLASH Level 1

                        Agreed! RTMFP is one of the most exciting developments ever to come to the flash player ... and that's saying a lot :-)

                        • 10. Re: stratus.adobe.com is down
                          Andre Linux Level 1

                          Of course!

                          We all love Stratus

                          Keep up the great work

                           

                           

                          I know how it feels to wake up and realize that your servers have failed while you were sleeping for unknown reason, good luck with identifying the reason which caused server crash !

                          • 11. Re: stratus.adobe.com is down
                            Michael Thornburgh Adobe Employee

                            we have identified the likely root cause of this morning's failure to be a lame bug related to time comparisons which, in a highly contrived but apparently possible situation, could cause memory to be consumed a little at a time until it all runs out.

                             

                            we have corrected this bug and loaded new software onto the cluster.  all new connections are to the new, fixed code.  the old code is still running for existing connections while we allow them to disconnect naturally.  after an hour or two we might bump any remaining connections to the old code for the safety of the cluster.  for anyone so dropped, they should be able to reconnect immediately and will be connected to the fixed code.

                             

                            -mike

                            1 person found this helpful
                            • 12. Re: stratus.adobe.com is down
                              rajeshell Level 1

                              is Still stratus.adobe.com down?

                               

                              I am not able to ping the server...

                               

                              Thanks for the reply!stratusPing.JPG

                              • 13. Re: stratus.adobe.com is down
                                Andre Linux Level 1

                                Nope, stratus is up and working, stratus.adobe.com is just not answering to ICMP requests (ping)

                                You can use this application to determine whether Stratus is working or not:

                                http://labs.adobe.com/technologies/stratus/samples/#resources

                                • 14. Re: stratus.adobe.com is down
                                  evanFLASH Level 1

                                  This sample application has not been working for me at all today.

                                  It's now failing with this message:

                                   

                                   

                                  Connecting to rtmfp://stratus.adobe.com
                                  NetConnection event: NetConnection.Connect.Success
                                  Connected, my ID: 26b6862ab020e41002cbaf828e269fa220bf4826576e6d121b05a63f1021e9fb
                                  ID event: idManagerError
                                  Error description: HTTP error: (mx.messaging.messages::ErrorMessage)#0
                                    body = (null)
                                    clientId = "DirectHTTPChannel0"
                                    correlationId = "AA16393A-FEBA-09FE-B1C6-722B287ACE58"
                                    destination = ""
                                    extendedData = (null)
                                    faultCode = "Channel.Security.Error"
                                    faultDetail = "Destination: DefaultHTTP"
                                    faultString = "Security error accessing url"
                                    headers = (Object)#1
                                      DSStatusCode = 0
                                    messageId = "C988B3CA-6C7F-6791-36CD-722B29267551"
                                    rootCause = (flash.events::SecurityErrorEvent)#2
                                      bubbles = false
                                      cancelable = false
                                      currentTarget = (flash.net::URLLoader)#3
                                        bytesLoaded = 0
                                        bytesTotal = 0
                                        data = (null)
                                        dataFormat = "text"
                                      errorID = 0
                                      eventPhase = 2
                                      target = (flash.net::URLLoader)#3
                                      text = "Error #2048: Security sandbox violation: http://labs.adobe.com/technologies/stratus/samples/VideoPhoneLabs.swf cannot load data from http://76.74.170.61/cgi-bin/reg?username=eflath&identity=26b6862ab020e41002cbaf828e269fa220bf4826576e6d121b05a63f1021e9fb."
                                      type = "securityError"
                                    timestamp = 0
                                    timeToLive = 0
                                  Disconnecting.
                                  Hanging up call
                                  NetConnection event: NetConnection.Connect.Closed
                                  

                                   

                                  stratus_fail.jpg

                                  • 15. Re: stratus.adobe.com is down
                                    Michael Thornburgh Adobe Employee

                                    the sample application uses a simple web service that runs on one of the servers in the Stratus cluster.  that particular computer became so wedged yesterday during the out-of-memory situation that we couldn't log in remotely to reboot it.  we put in a request for a forced reboot, but it took a little while to get that processed.

                                     

                                    that system has since been rebooted and the web service is once again accessible.  i tested the sample app and it appears to now work.

                                     

                                    -mike

                                    • 16. Re: stratus.adobe.com is down
                                      dmutantz Level 1

                                      Hi Guys

                                       

                                      Sorry to add to your woes but today at 11.10am GMT Stratus was down.  Thanks for any help.  Are there any plans to evaluate ways to get a dependable service going , I know you are doing this as a free service and I really am very grateful for your efforts and Adobes generosity; but I would really like to explore the potential of Stratus further but the crahes are getting more not less frequent.

                                      • 17. Re: stratus.adobe.com is down
                                        Andre Linux Level 1

                                        I don't think that we should expect fully reliable service at this time, because Adobe Stratus service is a beta prerelease, and we help them to develop good service by testing current beta version and reporting the bugs.

                                         

                                        Quote from http://labs.adobe.com/technologies/stratus/ "Product Details"

                                         

                                        "This release of the Stratus is prerelease and is designed for evaluation purposes only. The service is not final; but, many portions of the technology are fully implemented and ready for you to try and discuss."

                                         

                                        Even so, service worked really great for many months now, except for a few periods of downtime, so I don't think that we have to worry much about the last servers crash, it just happens in all services from time to time.

                                         

                                        Thanks to Stratus team for everything

                                        • 18. Re: stratus.adobe.com is down
                                          dmutantz Level 1

                                          Yes of course you are correct and its working fine now as well.

                                           

                                          Regards

                                          Dave

                                          • 19. Re: stratus.adobe.com is down
                                            Michael Thornburgh Adobe Employee

                                            i examined our logs for the time in question (11:10am GMT == 3:10am Pacific time, where we and Stratus are located).  i saw no unusual drops in number of users or rate of connections.  i examined the log for several hours before and after and everything looked completely normal in that timeframe (actually, all night long).

                                             

                                            Stratus is located on the west coast of the United States and has no geographic diversity (because it's a free, beta, unsupported, non-production, non-commercial developer tool).  perhaps at the time you were experiencing problems there was an Internet connectivity problem somewhere on the path between your location and the location of the data center hosting Stratus that affected a small enough number of people that it didn't show up in our logs.

                                             

                                            again, all night (before, during, and after Jan 28 11:10am GMT) Stratus appeared to be going strong with lots of connected users and handling new connections the whole time.

                                             

                                            -mike

                                            • 20. Re: stratus.adobe.com is down
                                              Michael Thornburgh Adobe Employee

                                              in the past 2 years that Stratus has been up and running, we've had the following outages:

                                               

                                                1) about a year ago (i don't remember the exact date), a programming error left the service unstable (but functioning) after an upgrade for about an hour until we reverted to an older version.  we fixed the problem and rolled in the new corrected software with no interruption.

                                               

                                                2) on nov 22 2009, a power failure at our data center (including the failure of the data center's battery and generator power) took us down for about two hours.

                                               

                                                3) on tuesday of this week, a combination of load on the service and other conditions managed to trigger a programming error that had been present in the code for over a year that brought down the service for several hours.  part of this failure wedged one of the physical servers (the one hosting a needed piece of the sample application) so badly that it required an in-person "remote hands" reboot, which took much longer than desired to arrange and execute.

                                               

                                              while not quite five 9s, i don't think that's too bad for a free, beta, unsupported evaluation service. 

                                               

                                              -mike

                                              1 person found this helpful
                                              • 21. Re: stratus.adobe.com is down
                                                rajeshell Level 1

                                                Thanks for making it Clear Michael!

                                                • 22. Re: stratus.adobe.com is down
                                                  Andre Linux Level 1

                                                  Thanks for making it clear.

                                                  It seems to be very high uptime. I would say 99,9%, at least. Even commercial services sometimes don't have uptime like that. Keep up the good work

                                                  -Andrey