Skip navigation

Cirrus

Michael Thornburgh

Currently Being Moderated

FYI: Stratus slow

Feb 23, 2010 2:18 PM

a very popular application has been using Stratus. the very large load of random-looking UDP traffic overwhelmed the front-end networking equipment at our data center today, which negatively impacted real, paid Adobe hosted services. our data center managers have imposed a network throttle on the Stratus cluster while they work to come up with a permanent, scalable solution.

this may appear as significant delays to connect to Stratus or as complete failure to connect (most of the time, just a long delay). we're sorry for the inconvenience. we hope things will be up to full strength again soon, but we currently have no ETR for this issue.

please remember that Stratus is a free, beta, experimental, "Adobe Labs" developer tool, and should not be relied upon for production services. we expect this won't be the last growing pain we experience.

on the plus side: the load of this very popular application has given us very valuable real-world information about the challenges of scaling RTMFP servers, which will directly benefit future Adobe server products and deployment guidance.

-mike

ps. this very popular application also stressed the Stratus servers themselves over the last week or so. we have already made some improvements on the server end. these server improvements have given us some breathing room while we acquire new hardware.

i can't divulge the "very popular application", but its developer is welcome to come forward and continue this thread.

4679 Views 21 Replies Latest reply: Michael Thornburgh, Jun 17, 2010 8:44 AM

Replies

Currently Being Moderated

1. MichaelHealy,

Feb 23, 2010 5:01 PM in reply to Michael Thornburgh

Report

chatroulette, by any chance? and does the fact that they are serving adverts make it a commercial application?

|
Mark as:
Currently Being Moderated

2. nbilyk,

Feb 24, 2010 4:02 PM in reply to MichaelHealy

Report

This is unfortunate..

I'd love to see someone make an open source linux install for a Stratus server. I've been waiting for a year on news of where Stratus is going.

|
Mark as:
Currently Being Moderated

3. ktowes (Product Mgr),

Feb 24, 2010 4:50 PM in reply to Michael Thornburgh

Report

We're excited to see the usage of RTMFP technology in Flash. We are looking at addressing the demand and popularity of Stratus that will allow more developers to start exploring what the benefits might be in your business, and to create new experiences on the web.

If you are interested in testing other future technologies, you can always follow the link below to sign up for new beta programs. As we get ready to start testing new Flash Media Server technology - we're very interested in hearing from the developer community.

Sign up here (make sure you select "Flash Media Server")
https://www.adobe.com/cfusion/mmform/index.cfm?name=prerelease_interes t

Also - if you haven't seen it - we have a new Stratus 2 article now available on Adobe.com:
http://www.adobe.com/devnet/flashmediaserver/articles/p2p_rtmfp_groups .html

|
Mark as:
Currently Being Moderated

4. stratamark,

Feb 24, 2010 6:37 PM in reply to ktowes (Product Mgr)

Report

ktowes:

Do you have any more information regarding the likely deployment of Stratus-like P2P rendezvous in the form of an update to FMS, or a service like LiveCycle?

Thank you

|
Mark as:
Currently Being Moderated

5. Mickey79,

Feb 24, 2010 8:43 PM in reply to Michael Thornburgh

Report

chatroulette was down yesterday as well, what a coincidence

|
Mark as:
Currently Being Moderated

6. Jack van Gils,

Feb 25, 2010 1:12 AM in reply to Michael Thornburgh

Report

It is no longer acceptable for us to rely on a service that is in development/test for a long time now. We have build an application that depends on this service. I would like to have an alternative in my own control or more commitment from Adobe for this service. You have launched the peer-2-peer capability in FlashPlayer 10 a long time ago already. We have bought licenses for Flash Media Server under the agreement that al this peer-2-peer capability would be available in a soon to be released next-release of that server software soon. That is some 2 years ago now.

How do I get a reliable service soon? How can I bring this in my own premises asap so I do not depend on the tweeks en turns of a development Adobe service?

|
Mark as:
Currently Being Moderated

7. Jack van Gils,

Feb 25, 2010 1:30 AM in reply to Jack van Gils

Report

"In addition to Stratus and LiveCycle Collaboration Service, the RTMFP protocol will also be available in a future release of Flash Media Server."
Still this is the promise today on the forums. So now the question is when can I operate RTMFP from my own Flash Media Server release?

|
Mark as:
Currently Being Moderated

8. Michael Thornburgh,

Mar 1, 2010 5:12 PM in reply to Michael Thornburgh

Report

as i just posted in this thread:

http://forums.adobe.com/message/2628282#2628282

Stratus performance is currently back to normal. we hope that the short-term changes that were made will suffice until our longer-term solutions can be executed. we have some capacity breathing room right now, but until our longer-term capacity plan is executed, we can't support a significantly higher load than we currently have.

|
Mark as:
Currently Being Moderated

9. Jack van Gils,

Mar 1, 2010 11:16 PM in reply to Michael Thornburgh

Report

Michael,

thank you for the information and restoring the service to a usable level.
However this still does not answer my main question: When is a new release coming to be self supportive on RTMFP support? \
As Adobe states since a few years now: "In addition to Stratus and LiveCycle Collaboration Service, the RTMFP protocol will also be available in a future release of Flash Media Server."

Can you guide me somewhere to get this resolved?

Thank you
Jack

|
Mark as:
Currently Being Moderated

10. Michael Thornburgh,

Jun 15, 2010 11:06 AM in reply to Michael Thornburgh

Report

as i just posted in this thread:

http://forums.adobe.com/message/2898215#2898215

we have completed our upgrade to the Stratus cluster and network infrastructure. while the Stratus service has been "normal" for several months, we were on borrowed time and were not in a position to handle significant growth. with the new, faster computers and a network configuration more appropriate for RTMFP traffic, we believe we are equipped to handle the Loads of the Future.

|
Mark as:
Currently Being Moderated

11. jazz_matazz,

Jun 15, 2010 1:35 PM in reply to Michael Thornburgh

Report

Ouch, again this morning, our remote applications came to join a group (NetGroup) quickly, and for several hours peers do not find them.
Our group specifier is encoded as follows:
            var g:GroupSpecifier = new GroupSpecifier(groupName);
            g.objectReplicationEnabled = true;
            g.multicastEnabled = true;
            g.postingEnabled = false;
            g.serverChannelEnabled = true;
            g.ipMulticastMemberUpdatesEnabled = true;
Yesterday, one peer detected a neighbor connection (on the group) in few secondes. Today, it seems sometimes take several minutes.

|
Mark as:
Currently Being Moderated

12. jazz_matazz,

Jun 16, 2010 2:14 AM in reply to jazz_matazz

Report

I confirm a really problem since the 15 juin. Even if I open the firewall (UDP and TCP) for software, the result is the same.
Here is the Adobe sample application available online:
http://jazzmatazz.free.fr/TestStratus/

If this sample is execute on two network computer different, they discover themself after several minutes. This problem didn't exist before.

|
Mark as:
Currently Being Moderated

13. Michael Thornburgh,

Jun 16, 2010 10:49 AM in reply to jazz_matazz

Report

what happens if you reverse the order of the computers as they join the group? in other words, you have computers A and B. when A joins first and then B joins, it takes "a while" for them to find each other. what happens if B joins first and then A joins?

if they join together immediately, then the problem is NAT/firewall related, most likely (but not necessarily) on A's end.

the code running on the new Stratus cluster is exactly the same as what had been running on the old cluster. the only differences are "new computers" and "much less obstructed UDP pathway".

on an unrelated note: i saw in your GroupSpecifier that you enable ipMulticastMemberUpdatesEnabled. you must also add an IP multicast to the groupspec for that option to do anything. that won't affect computers on different networks of course (since member updates are sent with TTL/hoplimit=1). but that option can dramatically improve the performance of large groups where more than one member is on a LAN.

|
Mark as:
Currently Being Moderated

14. jazz_matazz,

Jun 16, 2010 11:50 AM in reply to Michael Thornburgh

Report

Thanx for this intersting "ipMulticastMemberUpdatesEnabled" note.

I work on rtmfp://stratus.rtmfp.net/ address.
We have developped a small chat application test which works wonderfully well since 6 months ago.
Since exactly the 15 juin, all connection scenarios with 2 peers doesn't work (they find themselves after several minutes), and some connection scenarios with 3 peers doesn't work.

A : instance 1 on a computer
A' : instance 2 on the same computer
B : instance 1 on a another computer on different lan
B' : instance 2 on the another computer on different lan

By order of connection at the group we have this:
B, A => NOK
A, B => NOK
A, B, A' => OK
A, A', B => NOK
B, A, A' => NOK
B, A, B' => OK
B, B', A => NOK
A, B, B' => NOK

Note that firewall for the test is opened for the internet browser used.
We conducted tests on a dozen computers on which the application were correctly running daily for 6 months.
I don't understand.

|
Mark as:
Currently Being Moderated

15. jazz_matazz,

Jun 16, 2010 12:02 PM in reply to jazz_matazz

Report

I have just tested a direct P2P connection (without group), and it works fine, on all computers.

|
Mark as:
Currently Being Moderated

16. Michael Thornburgh,

Jun 16, 2010 2:56 PM in reply to jazz_matazz

Report

try it now.

|
Mark as:
Currently Being Moderated

17. jazz_matazz,

Jun 16, 2010 4:05 PM in reply to Michael Thornburgh

Report

Woua
It is working again as well as before!
Thanks a lot for your forcefulness!
And, what was the problem? A french network particularity?

|
Mark as:
Currently Being Moderated

18. Michael Thornburgh,

Jun 16, 2010 4:17 PM in reply to jazz_matazz

Report

it was a subtle configuration oversight on my part related to the different networking configuration of the new server cluster. sorry for the inconvenience.

|
Mark as:
Currently Being Moderated

19. jazz_matazz,

Jun 17, 2010 12:58 AM in reply to Michael Thornburgh

Report

No pb, thank you.
Happy to have contributed to solve an issue.

|
Mark as:
Currently Being Moderated

20. jazz_matazz,

Jun 17, 2010 5:58 AM in reply to jazz_matazz

Report

I wonder : When it wasn't work, if I had used "addMemberHint()" to add manually peer to group (in admitting that I have a database containing peers ids) is what it would have unlocked the stratus latency?

|
Mark as:
Currently Being Moderated

21. Michael Thornburgh,

Jun 17, 2010 8:44 AM in reply to jazz_matazz

Report

no, that wouldn't've made a difference. a fundamental part of how Stratus does NAT and firewall traversal wasn't working fully because i left an important option out of the command-line arguments for the server processes.

|
Mark as:

Forum guidelines

Answers + Points = Status

10 points awarded for Correct Answers
5 points awarded for Helpful Answers
10,000+ points
1,001-10,000 points
501-1,000 points
5-500 points