21 Replies Latest reply on Jun 17, 2010 8:44 AM by Michael Thornburgh

    FYI: Stratus slow

    Michael Thornburgh Adobe Employee

      a very popular application has been using Stratus.  the very large load of random-looking UDP traffic overwhelmed the front-end networking equipment at our data center today, which negatively impacted real, paid Adobe hosted services.  our data center managers have imposed a network throttle on the Stratus cluster while they work to come up with a permanent, scalable solution.

       

      this may appear as significant delays to connect to Stratus or as complete failure to connect (most of the time, just a long delay).  we're sorry for the inconvenience.  we hope things will be up to full strength again soon, but we currently have no ETR for this issue.

       

      please remember that Stratus is a free, beta, experimental, "Adobe Labs" developer tool, and should not be relied upon for production services.  we expect this won't be the last growing pain we experience.

       

      on the plus side: the load of this very popular application has given us very valuable real-world information about the challenges of scaling RTMFP servers, which will directly benefit future Adobe server products and deployment guidance.

       

      -mike

       

      ps. this very popular application also stressed the Stratus servers themselves over the last week or so.  we have already made some improvements on the server end.  these server improvements have given us some breathing room while we acquire new hardware.

       

      i can't divulge the "very popular application", but its developer is welcome to come forward and continue this thread.

        • 1. Re: FYI: Stratus slow
          MichaelHealy

          chatroulette, by any chance? and does the fact that they are serving adverts make it a commercial application?

          • 2. Re: FYI: Stratus slow
            nbilyk

            This is unfortunate..

             

            I'd love to see someone make an open source linux install for a Stratus server.  I've been waiting for a year on news of where Stratus is going.

            • 3. Re: FYI: Stratus slow
              ktowes (BD) Adobe Employee

              We're excited to see the usage of RTMFP technology in Flash.  We are looking at addressing the demand and popularity of Stratus that will allow more developers to start exploring what the benefits might be in your business, and to create new experiences on the web.

               

              If you are interested in testing other future technologies, you can always follow the link below to sign up for new beta programs.   As we get ready to start testing new Flash Media Server technology - we're very interested in hearing from the developer community.

               

              Sign up here (make sure you select "Flash Media Server")

              https://www.adobe.com/cfusion/mmform/index.cfm?name=prerelease_interest

               

              Also - if you haven't seen it - we have a new Stratus 2 article now available on Adobe.com:

              http://www.adobe.com/devnet/flashmediaserver/articles/p2p_rtmfp_groups.html

              • 4. Re: FYI: Stratus slow
                stratamark

                ktowes:

                 

                Do you have any more information regarding the likely deployment of Stratus-like P2P rendezvous in the form of an update to FMS, or a service like LiveCycle?

                 

                Thank you

                • 5. Re: FYI: Stratus slow
                  Mickey79 Level 1

                  chatroulette was down yesterday as well, what a coincidence

                  • 6. Re: FYI: Stratus slow
                    Jack van Gils

                    It is no longer acceptable for us to rely on a service that is in development/test for a long time now. We have build an application that depends on this service. I would like to have an alternative in my own control or more commitment from Adobe for this service. You have launched the peer-2-peer capability in FlashPlayer 10 a long time ago already. We have bought licenses for Flash Media Server under the agreement that al this peer-2-peer capability would be available in a soon to be released next-release of that server software soon. That is some 2 years ago now.

                     

                    How do I get a reliable service soon? How can I bring this in my own premises asap so I do not depend on the tweeks en turns of a development Adobe service?

                    • 7. Re: FYI: Stratus slow
                      Jack van Gils Level 1

                      "In addition to Stratus and LiveCycle Collaboration Service, the RTMFP protocol will also be available in a future release of Flash Media Server."

                      Still this is the promise today on the forums. So now the question is when can I operate RTMFP from my own Flash Media Server release?

                      • 8. Re: FYI: Stratus slow
                        Michael Thornburgh Adobe Employee

                        as i just posted in this thread:

                         

                           http://forums.adobe.com/message/2628282#2628282

                         

                        Stratus performance is currently back to normal.  we hope that the short-term changes that were made will suffice until our longer-term solutions can be executed.  we have some capacity breathing room right now, but until our longer-term capacity plan is executed, we can't support a significantly higher load than we currently have.

                        • 9. Re: FYI: Stratus slow
                          Jack van Gils Level 1

                          Michael,

                           

                          thank you for the information and restoring the service to a usable level.

                          However this still does not answer my main question: When is a new release coming to be self supportive on RTMFP support? \

                          As Adobe states since a few years now: "In addition to Stratus and LiveCycle Collaboration Service, the RTMFP protocol will also be available in a future release of Flash Media Server."

                           

                          Can you guide me somewhere to get this resolved?

                           

                          Thank you

                          Jack

                          • 10. Re: FYI: Stratus slow
                            Michael Thornburgh Adobe Employee

                            as i just posted in this thread:

                             

                               http://forums.adobe.com/message/2898215#2898215

                             

                            we have completed our upgrade to the Stratus cluster and network infrastructure.  while the Stratus service has been "normal" for several months, we were on borrowed time and were not in a position to handle significant growth.  with the new, faster computers and a network configuration more appropriate for RTMFP traffic, we believe we are equipped to handle the Loads of the Future.

                            • 11. Re: FYI: Stratus slow
                              jazz_matazz

                              Ouch,  again this morning, our remote applications  came to join a group (NetGroup) quickly, and for several hours peers do not find them.

                              Our group specifier is encoded as follows:

                                          var g:GroupSpecifier = new GroupSpecifier(groupName);
                                          g.objectReplicationEnabled = true;
                                          g.multicastEnabled = true;
                                          g.postingEnabled = false;
                                          g.serverChannelEnabled = true;
                                          g.ipMulticastMemberUpdatesEnabled = true;

                              Yesterday,  one peer detected a neighbor connection (on the group) in few secondes. Today, it seems sometimes take several minutes.

                              • 12. Re: FYI: Stratus slow
                                jazz_matazz Level 1

                                I confirm a really problem since the 15 juin. Even if I open the firewall (UDP and TCP) for software, the result is the same.

                                Here is the Adobe sample application available online:

                                http://jazzmatazz.free.fr/TestStratus/

                                 

                                If this sample is execute on two network computer different, they  discover themself after several minutes. This  problem didn't exist before.

                                • 13. Re: FYI: Stratus slow
                                  Michael Thornburgh Adobe Employee

                                  what happens if you reverse the order of the computers as they join the group?  in other words, you have computers A and B.  when A joins first and then B joins, it takes "a while" for them to find each other.  what happens if B joins first and then A joins?

                                   

                                  if they join together immediately, then the problem is NAT/firewall related, most likely (but not necessarily) on A's end.

                                   

                                  the code running on the new Stratus cluster is exactly the same as what had been running on the old cluster.  the only differences are "new computers" and "much less obstructed UDP pathway".

                                   

                                  on an unrelated note: i saw in your GroupSpecifier that you enable ipMulticastMemberUpdatesEnabled.  you must also add an IP multicast to the groupspec for that option to do anything.  that won't affect computers on different networks of course (since member updates are sent with TTL/hoplimit=1).  but that option can dramatically improve the performance of large groups where more than one member is on a LAN.

                                  • 14. Re: FYI: Stratus slow
                                    jazz_matazz Level 1

                                    Thanx for this intersting "ipMulticastMemberUpdatesEnabled" note.

                                     

                                    I work on rtmfp://stratus.rtmfp.net/ address.

                                    We have developped a small chat application test which works wonderfully well since 6 months ago.

                                    Since exactly the 15 juin, all connection scenarios with 2 peers doesn't work (they find themselves after several minutes), and some connection scenarios with 3 peers doesn't work.

                                     

                                    A :  instance 1 on a computer

                                    A' : instance 2 on the same computer

                                    B :  instance 1 on a another computer on different lan

                                    B' :  instance 2 on the another computer on different lan

                                     

                                    By order of  connection at the group we have this:

                                    B, A      => NOK
                                    A, B      => NOK
                                    A, B, A' => OK
                                    A, A', B => NOK
                                    B, A, A' => NOK
                                    B, A, B' => OK
                                    B, B', A => NOK
                                    A, B, B' => NOK

                                     

                                    Note that firewall for the test is opened for the internet browser used.

                                    We conducted tests on  a dozen computers on which the application were correctly running daily for 6  months.

                                    I don't understand.

                                    • 15. Re: FYI: Stratus slow
                                      jazz_matazz Level 1

                                      I have just tested a  direct P2P connection (without group), and it works fine, on all computers.

                                      • 16. Re: FYI: Stratus slow
                                        Michael Thornburgh Adobe Employee

                                        try it now.

                                        • 17. Re: FYI: Stratus slow
                                          jazz_matazz Level 1

                                          Woua

                                          It is working again as well as before!

                                          Thanks a lot for your forcefulness!

                                          And, what was the problem? A french network particularity?

                                          • 18. Re: FYI: Stratus slow
                                            Michael Thornburgh Adobe Employee

                                            it was a subtle configuration oversight on my part related to the different networking configuration of the new server cluster.  sorry for the inconvenience.

                                            • 19. Re: FYI: Stratus slow
                                              jazz_matazz Level 1

                                              No pb, thank you.

                                              Happy  to have contributed to solve an issue.

                                              • 20. Re: FYI: Stratus slow
                                                jazz_matazz Level 1

                                                I wonder : When it wasn't work, if I had used "addMemberHint()" to add  manually peer to group (in admitting that I have a database containing peers ids) is what it would have  unlocked the stratus latency?

                                                • 21. Re: FYI: Stratus slow
                                                  Michael Thornburgh Adobe Employee

                                                  no, that wouldn't've made a difference.  a fundamental part of how Stratus does NAT and firewall traversal wasn't working fully because i left an important option out of the command-line arguments for the server processes.