10 Replies Latest reply on Jun 18, 2010 4:33 AM by Matoo-

    When will Stratus performance be back to normal?

    24z.me

      Dear Adobe,

       

      We are the developers of CamCarousel, a Chatroulette clone with the full code available for the Flex developer community. Since we've been covered by ReadWriteWeb, we are experiencing extreme long latency for every interaction with the Stratus service. We understand that Chatroulette is putting quite some heat on the service.

       

      Since the service "interruption" already lasts for more than a day, do you have any estimate as to when you'll make Stratus scale?

        • 1. Re: When will Stratus performance be back to normal?
          Michael Thornburgh Adobe Employee

          we are currently working on a three-phase plan.

           

          phase 1: our data center managers have updated the software on the firewall, which is the current bottleneck.  the expectation is the new software should be able to handle the *current* load, and they are slowly relaxing the throttle on the Stratus cluster to ensure we don't overload (and negatively impact) production Adobe services.  the hope is that by end-of-business monday (USA pacific time) the throttle will be lifted completely and, assuming the firewall and our servers can keep up with whatever the load looks like by then, we should be at full speed.  however, if the load is too great, some amount of throttle will need to be kept in place, but hopefully less severe than right now.

           

          phase 2: our data center managers are working to create a new, non-stateful, UDP-friendly network zone into which to move the Stratus cluster.  this should completely eliminate any network bottlenecks for the foreseeable future (until the ISP runs out of packets-per-second for us).  we don't have a date for this change yet.

           

          phase 3: we have new computers on order to replace the dated computers that Stratus runs on today.  we estimate the new computers are about 4 times faster than the current ones.

           

          after phase 3, we should be in a position where we can add capacity to Stratus by adding more computers to the cluster.

          • 2. Re: When will Stratus performance be back to normal?
            Powerofjurai

            Is the Stratus service currently available for use in commercial applications? because it sure seems CamCarousel is selling their code commercially with the implication that it relies upon the Stratus service to function....

            • 3. Re: When will Stratus performance be back to normal?
              24z.me Level 1

              We are selling the code for educational purposes and to help others learn how to build a Chatroulette-like application. We are not offering any commercial service relying on Stratus. - Thanks for asking and giving us a chance to clarify your confusion!

              • 5. Re: When will Stratus performance be back to normal?
                Michael Thornburgh Adobe Employee

                phase 1 is now complete.  while the throttle on Stratus isn't actually *gone*, it is set to higher than the current traffic level on Stratus, so it is effectively running at full speed.  we're also seeing significantly less traffic than we were a week ago.  we hope that traffic levels will stay reasonable until phases 2 and 3 can be executed, at which point we'll be in a better position to grow our capacity.

                 

                as it is right now, we have "breathing room".

                 

                if traffic increases dramatically, the new throttle setting may start to have an effect and performance may degrade again.

                • 6. Re: When will Stratus performance be back to normal?
                  Michael Thornburgh Adobe Employee

                  after much longer than anticipated, Stratus is now running on brand new hardware with a brand new, more UDP friendly network path.  in other words, "phase 3" is complete.

                   

                  please post if you notice any new problems (things that worked yesterday no longer work).  it *should* appear and behave identically (except for different IP addresses of the servers).

                   

                  the performance of Stratus has been "normal" for quite some time, but we should now be in a much better position to handle growth of the service.

                  • 7. Re: When will Stratus performance be back to normal?
                    Matoo-

                    RTMFP used to work perfectly until two days ago but now client-to-client stream playback fails all the time, even when using the sample Adobe Video Phone demo. We can connect to Stratus just fine but when trying to play the farID of another Flash client then the stream never starts. In Video Phone words, the "connect" works just fine but the "call" is never received by the targetted nickname.

                     

                    When trying to connect two clients running on the same computer it works fine though. Looks like the UDP port used for the client-to-client connection has been changed and we'll have to update the client's firewall/router configuration accordingly.

                    • 8. Re: When will Stratus performance be back to normal?
                      Michael Thornburgh Adobe Employee

                      if there were special firewall rules to allow connections to Stratus by IP address and/or port, then the change we made yesterday will definitely cause a problem, since the IP addresses of the Stratus servers changed.

                       

                      there was a configuration problem on the new cluster that would have affected NAT and firewall traversal.  it was corrected several hours ago.  please try again and see if things are working now or if you are still having problems.

                      • 9. Re: When will Stratus performance be back to normal?
                        Matoo- Level 1

                        Connection to Stratus works flawlessly. It's the P2P connection that doesn't work. Network activity shows that UDP packets are sent from one of the FP clients to the the other one on port 4293 but connection is never established. When the two FP clients are on the same computer it works just fine.

                         

                        I thought about opening UDP port 4293 but that's actually a bad idea, RTMFP P2P connections use random UDP port numbers don't they? And it worked just fine before two days ago. It might just be related to the NAT/Firewall traversal issues you mentioned. We'll try again tonight.

                        • 10. Re: When will Stratus performance be back to normal?
                          Matoo- Level 1

                          Good news, it's working fine again.