Skip navigation
Currently Being Moderated

ColdFusion 10 intermittent "service not available"

Jun 14, 2012 10:29 AM

Have been programming over 30 years and CF in its various forms since 1998, but have to say this upgrade is NOT straightforward and after 3 very frustrating days I've finally reverted to CF9.

 

CF10 Standard upgrade from CF9. Running on dedicated IIS 7.5 on WIN 2008 R2 Server. All 64bit.

 

Upgrade installation worked correctly. RDS was selected as was the Upgrade all IIS sites.

 

Initially CF10 failed to start and checking IIS showed the connectors had NOT been installed. CF10 Administrator just showed raw machine code.

 

Used the CF10 Web Server Configuration tool on "ALL Sites" - no difference.

 

Removed "All sites" and then installed the connectors against individual sites. This worked - but then I noticed each site was running the CF9 connectors!

 

Removed each. Then ran the CF9 Web Server Configuration tool to remove all CF9 connectors. Confirmed they had been removed in IIS, then ran CF10 Web Server Configuration tool to add the connectors and all worked well.

 

This morning I rebooted the server to confirm CF had "installed" correctly, 3 of my 5 sites too 10 minutes before they could be accessed. Then after 30 minutes I'd get a simple white screen with "service unavailable".

 

Can't afford to have our sites and clients sites intermittent so have reverted to the CF9 connectors and took CF10 offline.

 

Hopefully someone else has had similar experience and can shed some light on this issue.

 

Thanks

Peter

 
Replies 1 2 3 Previous Next
  • Currently Being Moderated
    Jun 18, 2012 6:11 AM   in reply to Ripples

    Had similar issues and attempted install/reinstall.  Found that the Handler Mappings were completely wrong.  Changed them to point to the correct location and all the problems disappeared. 

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 20, 2012 6:15 AM   in reply to Ripples

    Knock on wood, but I haven't experienced this issue yet.  My guess is that it is more the Apache integration.  Are all the services restarting?  If this is happening repeatedly, I would file a bug over at https://bugbase.adobe.com/

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 5, 2012 8:44 PM   in reply to Ripples

    Re Durring CF start:
    org.apache.catalina.core.AprLifecycleListener init
    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\CF10\\etc

     

    Was discussed in this post:

    http://forums.adobe.com/message/4446569#4446569

     

    Maybe the details in that post to resolve that message will partly provide solution to your issue.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 6, 2012 5:01 PM   in reply to Ripples

    Peter I recall seeing Jrun (CF6 7 8?) and getRealPathFromConn errors before but have not directly been involved with matter so it does not come to mind as to solution. Perhaps if you search without tomcat details you might find a useful lead since CF has run Jrun for longer than tomcat.

     

    HTH again, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 12, 2012 4:27 PM   in reply to Ripples

    Hi Peter

    Have you figured the way to resolve your issue with cf10 yet? Or you have reverted back to cf 9. It seems we may face the similar issue. Can you please check your system error log to see whether you got some similar erroes as I did. I have posted the detail in

    http://forums.adobe.com/message/4609125

    My production server is really unstable at the moment and this forcing me to monitor the server 24/7 in case it hangs.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 17, 2012 5:23 PM   in reply to Ripples

    Peter,

    Thank you for your response.

    There are two bugs which might associate to our issue have been submitted to Adobe and I believe they are all confirmed by Coldfusion Team. As far as I know, they are working on it at the moment, but unfortunately they are unable to put the fix in hotfix 1 they have released early this month or late last month.

    https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

    https://bugbase.adobe.com/index.cfm?event=bug&id=3222748

    We are waiting for the fix from Adobe, since we are not going to rollback to where we came from.

    My temperate solution is increasing the number of error IIS can handle before shut itself off.

    It is in application pool > Advance setting > Rapid-Faill Protection - Maximum Failures

    Since I have found the IIS will shut itself off after 5 errors been sent from connector, by increasing the number here, it temperately solved my problem (but my event log still full of errors from connector)

     

    Ryan

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 22, 2012 10:19 PM   in reply to Ripples

    Peter, I hear your frustration, and I have some thoughts.

     

    I don't see anyone suggesting this yet: is it possible that this is all down to a permissions issue? I realize that at this point, you have likely uninstalled CF10 and may not care to futz with it anymore, but in case you may, or in case others may find this, here are a couple of suggestions.

     

    First, did you notice how things would be ok until you rebooted? That leads me to think that something that was "changed" was really only changed in memory and could not be written to disk, so that on reboot, the "change" was "lost". I can't right now think what that may have been, but it sounds like a clue worth considering.

     

    One thing I do know is that it may help when running the web server configuration tool if you first right-click on it and "run as administrator". This is one of those vagaries that arises from the increased security in Windows 2008. I do think that sometimes CF is getting an unfair black eye due to things like this.There are other apps which work funky if you don't "run as admin". Some even give no error, but still don't do what you think they should do.

     

    Also, there are important differences in IIS 7 from how things worked before, which could also be related (assuming you are not using IIS 6 Compatibily mode). Some folks may have noticed that a web.config file may be written to the root of your IIS sites which have been configured for CF. That could be created by changes you make in the IIS interface, or by the CF web server config tool (or installer).

     

    Well, as for the latter here's another potential gotcha: perhaps CF doesn't have permissions to that directory. For those who change the user under which CF runs, they may not consider any need to give write permissions for that user to their website docroot. If CF can't write to it, then I could see the web server config tool (or installer) not implementing things correctly.

     

    Not all configurations of CF and IIS require creation of the web.config file, so I do realize some will say they see none there and yet things are working. Put briefly, the differences have to do with if you are or are not running IIS in IIS 6 compatibility mode. If you are not, then it works quite differently than it did before.

     

    First, it seems that if you configure "all sites", CF configures things (like what you see in handler mappings) in an applicationhost.config file in \Windows\System32\inetsrv\config, and any changes from that made for a given site are written to its web.config. And second, if you configure a specific site, then it writes the needed settings to that site's web.config. In either case, again, if there are any challenges writing to that file, then settings won't be persisted. I grant that I'm still guessing a bit here, so I'll welcome thoughts from others.

     

    BTW, this use of the web.config could also explain some of the confusion you were hitting when you had things configured for both CF 9 and CF10. Perhaps you may have had (as I did at one time) a site set to connect using CF9, and another set to connect using CF10, both pointing to the same web site docroot. That was always ok in CF traditionally, to test things on one server with different sites against different versions of CF.

     

    But now (if not using IIS 6 Compat mode) then CF and IIS are using these config files, and they are NOT writing different ones for different versions. If 2 sites point to the same docroot, changes to each basically overwrite the other. It can definitely be confusing. 

     

    And for many people, CF10 is the first release where they are deploying CF on IIS 7. You may know that CF 9.0 and below did not formally support it, but people found they could get it to work by enabling IIS 6 compatibility. And while 9.0.1 added full support for IIS 7, many just continued to use IIS 6 Compat when they upgraded from 9.0 to 9.0.1. I suspect now we have more people going to IIS 7 for the first time with 10, or maybe putting it on new boxes where they did not think to even turn on IIS 6 compat, and so they are experiencing the results of this new approach to things for the first time.

     

    Again, some of what I offer here is speculation. A lot of this is new for all of us. I just wanted to offer these thoughts as I'd not see anyone else do so yet here. I appreciate the severe annoyance some have felt in things seeming "not to work". But I'll just say that there are people who are running CF10 fine, so it is possible to configure things correctly.

     

    This is somewhat like the pain we went through in the move from CF 5 to CFMX (6), where things with Java were quite different and some even gave up on CF (or just never moved up past 5, or not for many years.) Then there's the added differene that CF10 is running on Tomcat rather than JRun, and Adobe did have to create a new connector (and couldn't use the stock one that comes with Tomcat). With so many moving parts, it can be hard to know for sure when something is a "bug", when something is just a misconfiguration, or a misunderstanding, or a failure to set proper permissions, or to run as administrator, and on and on.

     

    As in that time fo the switch to CFMX, there may be growing pains but there will also be growing understanding as more and more people find and resolve problems such as these, and share their knowledge. It's little consolation for those who are on the front lines deploying the new release in production.

     

    But I hope, Peter, that perhaps some of what I've shared here might either help solve problems or at least help get you further down the road to resolving them, again, if and when you may be willing to give deployment of CF10 another shot.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2012 11:58 AM   in reply to Ryan Luo

    Ryan,

     

    Just wanted to confirm that we experienced the issues with IIS listed in those bugs. I was able to get Adobe's attention via social media and we are testing out new jars that fix the issue with the isapi connector.

     

    We ran for almost 24 hours before having to roll back to our CF9 servers on Friday. (We had to rollback because of this DateConvert() bug.) The new jars appear to fix the connector issue. Hopefully there will be an update soon to address this. But, since the instructions from adobe included dissconnecting sites and running wsconfig after replacing the wsconfig.jar, I'm not sure that a regular update will fix it automatically.

     

    If you want to test the new jars, Rupesh has an offer to send the jars to people who would test them on this bug page in the notes. (rukumar (at) adobe (dot) com)

     

    We have again added the CF10 servers into the pool and they are ramping up. Hope to be fully on CF10 by Tuesday.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 2, 2012 5:25 PM   in reply to cf_chris

    Hi Chris

    Thanks for your reply and the information you have provided.

    I am afraid I couldn't try the new jar due to the server is in production. I will wait the official patch for cf10 and hopefully it can resolve the issue troubles the community.

     

    Thanks

    Ryan

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 9, 2012 2:30 PM   in reply to Ripples

    Ripples, sorry to hear your pain.

     

    We upgrade to CF 10 running Windows 2008 servers with IIS 7, connecting to MySQL 5 DBs. We put our highest profile customers on these new boxes and promised them a great, upgraded service. We tested, and tested each site was then put live on the new VPSs. Site failures became rampant. IIS got hung up several times a day. Restarting IIS solves the problem, but it is an absurd problem for a middle ware that is on version 10.

     

    Seems that if there is only 1 site running there is no problem, but if mulitple sites run on the same server, IIS gets hung up.

     

    I am glad to read that going back to CF 9 solved your problems. This is what we may do.

     

    Charlie Arehart, above has some good suggestions it seems, and from what I can tell, it is really a matter of CF10 working propertly with IIS 7. Problem can be solved by running IIS 7 with IIS 6 compatibility enabled?

     

    If anyone has come up with a stable config of the above specs - would love to hear it

     

    Thanks,

    james

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 21, 2012 4:07 PM   in reply to colophonnewmedia

    James,

     

    Update 4 had the connector fix in it. If you apply Update 4 and use wsconfig to disconnect and reconnect your instances to IIS 7, you should be good to go. The crashing should stop.

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 26, 2012 7:58 AM   in reply to cf_chris

    Chris,

     

    Thank you for the note. We are have applied this fix, and so far so good.

     

    We also had to update a setting in CF admin setting for: “Maximum number of simultaneous Template requests”. The default for CF 10 is set to 25. The default install of our local copy of CF 10 has this set to 25. Our box was set to 10, as it was upgraded from 8 and 9 and we think this was left over from CF 8 and 9.

     

    However, according to Adobe, when the limit for Template requests is reached, the system is supposed to queue the rest. In our case it does not queue, it stops. So there is still some concern. 

     

    We also don’t see how a request is equal to site activity for 1 site. We thought that “worker processes” would be related to a number of different processes that happened when web templates run.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 25, 2013 8:52 AM   in reply to Ripples

    Just updated Coldfusion to latest version (Hotfix 7). We are still having this issue. System runs for a week or so and then our main application pool dies with this error over and over:

     

    SEVERE: Error in getRealPathFromConn
    java
    .net.SocketException: Connection reset
        at java
    .net.SocketInputStream.read(SocketInputStream.java:168)
    at org
    .apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:319)
    at org
    .apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:445)
    at org
    .apache.coyote.ajp.AjpProcessor.readString(AjpProcessor.java:425)
    at org
    .apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:407)
    at org
    .apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1154)
    at org
    .apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)
    at org
    .apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)
    at org
    .apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)
    at org
    .apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)
    at org
    .apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:335)
    at coldfusion
    .runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:179)
    at coldfusion
    .runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)
    at coldfusion
    .runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)
    at coldfusion
    .runtime.TemplateProxyFactory.getFullName(TemplateProxyFactory.java:1092)
    at coldfusion
    .runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:202)
    at coldfusion
    .runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159)
    at coldfusion
    .runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120)
    at coldfusion
    .cfc.CFCProxy.<init>(CFCProxy.java:138)
    at coldfusion
    .cfc.CFCProxy.<init>(CFCProxy.java:84)
    at coldfusion
    .runtime.AppEventInvoker.<init>(AppEventInvoker.java:64)
    at coldfusion
    .filter.ApplicationFilter.invoke(ApplicationFilter.java:226)
    at coldfusion
    .filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
    at coldfusion
    .filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
    at coldfusion
    .filter.PathFilter.invoke(PathFilter.java:112)
    at coldfusion
    .filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
    at coldfusion
    .filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
    at coldfusion
    .filter.BrowserFilter.invoke(BrowserFilter.java:38)
    at coldfusion
    .filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
    at coldfusion
    .filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
    at coldfusion
    .filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
    at coldfusion
    .filter.CachingFilter.invoke(CachingFilter.java:62)
    at coldfusion
    .CfmServlet.service(CfmServlet.java:219)
    at coldfusion
    .bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
    at org
    .apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org
    .apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at coldfusion
    .monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
    at coldfusion
    .bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
    at org
    .apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org
    .apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org
    .apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
    at org
    .apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
    at org
    .apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org
    .apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org
    .apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org
    .apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
    at org
    .apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org
    .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)
    at org
    .apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)
    at org
    .apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
    at org
    .apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
    at java
    .util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java
    .util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java
    .lang.Thread.run(Thread.java:662)

     

    While the server error is the same each time, the trace is different with each one. The first one always looks like the above, and the rest of them after it look similar to the below where the connection is reset by peer. However the full traces and paths to different files changes with each error usually. Sometimes three or four in a row of the same, sometimes just one. Either way, there are like 40 or 50 in a row.

     

    Jan 25, 2013 8:10:04 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer
    SEVERE: Error in getRealPathFromConn
    java.net.SocketException: Connection reset by peer: socket write error
              at java.net.SocketOutputStream.socketWrite0(Native Method)
              at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
              at java.net.SocketOutputStream.write(SocketOutputStream.java:124)
              at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:401)
              at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1154)
              at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)
              at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)
              at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)
              at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)
              at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:335)
              at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)
              at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)
              at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)
              at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1046)
              at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1060)
              at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:858)
              at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:830)
              at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:794)
              at coldfusion.runtime.TemplateProxyFactory.getResolvedFile(TemplateProxyFactory.java:1263)
              at coldfusion.runtime.TemplateProxyFactory.getTemplateFileHelper(TemplateProxyFactory.java:1571)
              at coldfusion.cfc.ComponentProxyFactory.getProxy(ComponentProxyFactory.java:51)
              at coldfusion.runtime.CFPage.CreateObject(CFPage.java:5560)
              at coldfusion.runtime.CFPage.CreateObject(CFPage.java:5575)
              at cfApplication2ecfc851739670.runPage(D:\www\facilitator\Application.cfc:30)
              at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:244)
              at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)
              at coldfusion.runtime.TemplateProxyFactory.resolveComponentHelper(TemplateProxyFactory.java:538)
              at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:234)
              at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159)
              at coldfusion.runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120)
              at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:138)
              at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:84)
              at coldfusion.runtime.AppEventInvoker.<init>(AppEventInvoker.java:64)
              at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:226)
              at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
              at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
              at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)
              at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
              at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
              at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
              at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
              at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
              at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
              at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
              at coldfusion.CfmServlet.service(CfmServlet.java:219)
              at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
              at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
              at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
              at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)
              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)
              at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
              at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
    

    This is a critical error and is really bugging the crap out of me. Strange thing is, as far as I know, this has not happened until the last month or so. We've been running CF10 on IIS7 for about 6 months. The first appearance of the error is on Aug 7, however it only fired once. The on Nov 5 once. Then Jan 10 twice. Then on Jan12 it started firing the error repeatedly in a row causing IIS to stop the pool. I updated CF shortly after that to attempt to fix the issue, but it remains an issue. Also, this error has occured 0 times in our development machine. The machines are for all intents and purposes identical. (HDD/RAM is different but nothing else)

     

    The only other thing I can think of, you can slap me for my ignorance, is there are other things in the log file:

     

    A bunch of these:

     

    Jan 18, 2013 8:13:59 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
    SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@cde21de]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1613372400000,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="America/Denver",offset=-25200000,dstSavings=3600000,useDaylight=true,transitions=157,lastRule=java.util.SimpleTimeZone[id=America/Denver,offset=-25200000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2021,MONTH=1,WEEK_OF_YEAR=8,WEEK_OF_MONTH=3,DAY_OF_MONTH=15,DAY_OF_YEAR=46,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=0,HOUR=0,HOUR_OF_DAY=0,MINUTE=0,SECOND=0,MILLISECOND=0,ZONE_OFFSET=-25200000,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
    

    And These:

     

    Jan 17, 2013 10:42:07 PM org.apache.catalina.core.AprLifecycleListener init
    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\ColdFusion10\\cfusion\lib;C:\\ColdFusion10\\cfusion\jintegra\bin;C:\\ColdFusion10\\cfusion\jintegra\bin\international;C:\\ColdFusion10\\cfusion\lib\oosdk\classes\win
    

    You won't suprise me if you say these items may be related but I really don't know what those two other errors mean. I've tried googling them with no luck on what to do about them.

     

    Also looking at the exception log, usually around the same time I get an error stating that Coldfusion can't find something. Usually at the top of the stack the method its running is:

     

    at coldfusion.runtime.TemplateProxyFactory.getResolvedFile(TemplateProxyFactory.java:1344)

     

    At the time I am usually running something to get the path to create a component or something like that if I follow the trace down. This seems like it is the result of the main error that Coldfusion can't the path. Usually these exceptions are found just in the application.cfc or when the application is initializing. Just so everyone is aware, all our web files are on our D: drive not the system drive where Coldfusion is installed. It's not an external drive. This is a virtural server hosted by our IT department.

     

    All of the IIS6 Compatability components are there in the Role Services. Do I need to "turn them on?"

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 11, 2013 10:36 AM   in reply to Lee Bartelme

    @Lee

    FWIW I am having the exact same issue.  Installed CF10 on a fresh win 2k r2 box IIS7.5 running a Mura instance.  Developed the site over the last 6 months or so.  Ran into an issue with the site returning blank pages with a few errors.  I did see all of the ones you describe above.  Fiddled with the JVM settings to no avail.

     

    I hope adding my voice to the crowd gathering here helps get things fixed.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 12, 2013 7:13 PM   in reply to byrons1

    Lee and Byron, you may have what Byron says are the same issues, but your solutions may be different.

     

    First, Lee, you say you just applied update 7. Had you applied any CF10 updates prior to that? If so, had you re-run the web server configuration tool (to remove and re-add your connections)? That’s mandatory for updates 3-5.

     

    What many often don’t notice, though, is that if they may have “skipped” those updates and just applied a later one (like the latest, 7), then they STILL have to remember to rerun the web server config tool after that later update. The technote and update text (in the updates feature) do “recommend” it, but the challenge is that they don’t require it because technically it’s not required if you’re going from, say, update 6 to update 7. 

     

    But again, if you are either applying this as your first update, or had skipped one of those earlier ones, or had applied them but had NOT rerun the web server config tool, then you are currently running with the old connector. And THAT could be the cause of your problem.

     

    Look at the isapi_redirect.dll  file in each subdirectory of C:\ColdFusion10\config\wsconfig\ (meaning, C:\ColdFusion10\config\wsconfig\1, C:\ColdFusion10\config\wsconfig\2, and so on). Each of these represents a connection of IIS to an instance. You may have only one, you may have many such numbered subfolders. Look at the date of that file: if it’s not dated November 8 2012 (or later) then it’s not been updated. The connector in each subfolder must be updated.

     

    (And you run the tool for each instance from within that instance, such as C:\ColdFusion10\cfusion\instance1\bin\wsconfig.exe. Again, don’t forget to launch the tool with “run as administrator”, if on Windows Vista, 7 or 2008.)

     

    Finally, Byron, the reason I have proposed you may have the “same issue but different solution” is that you say you “Installed CF10 on a fresh win 2k r2”. If by that you mean you had not applied any updates at all, then you have 2 things to do: apply the latest update (7) and then run the web server config tool to reconfigure any connections you may have made during the installer.

     

    Let us know if this may have helped either of you (or other readers, seeing this in the future.)

     

     

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 12, 2013 7:25 PM   in reply to Charlie Arehart

    I think on our dev box, it made me do this when updating to hot fix 6. I don't remember specifically doing this on our prod box.  I tried searching for wsconfig.exe but didn't find it. Then I looked on adobes site on how to do this and the steps were more than confusing. I have no clue if I completed this step. I restarted the machine hoping it would complete this for me.  I'll check the date on the file and if it's older than what you said, look into doing that step again.  I swear my dev box prompted me to do this, but I don't remember the prod box doing so. If I remember right, the wsconfig tool just opens a single box witha single field right. I have no idea what I need to do at with the tool as there is little/no documentation on the subject. Can you point me in the right direction. I think on the dev box I just left whatever was in there and moved on.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 13, 2013 10:27 PM   in reply to Lee Bartelme

    No, Lee, it will not “complete it for you”. And I agree that if one looks at the CF docs (in the “Admin and Config” as well as “Install” guides) there is no real step-by-step guide to running the web server config tool. That is surprising.  It focuses only on how to “start” it, not how to “use it”.  I suppose some could argue, “it’s a gui, so it’s obvious”, but I would agree that someone not familiar with it would not know for sure how to work with it, and what all the buttons are about.

     

    (And those who think Lee is being whiny, consider also that someone who merely installed CF and chose to connect it to IIS at the time would NOT have previously seen this web server config tool. They only see it now, when the CF10 update calls for them to re-build the web server configuration. So this problem is perhaps more significant than it’s ever been for CFers without lots of CF admin/config experience.)

     

    But yes, Lee, you would need to run it (with that “run as administrator” option, if you see one when you right-click on it), and you would remove and then re-add whatever connections you see (taking note of what they were, if more than one, in terms of what “site” was connected to what “instance”, again if you run more than one instance of CF, as is supported in the Enterprise/Trial/Developer edition). A restart of CF won’t do it for you.

     

    Let us know if that helps.

     

     

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 2:00 PM   in reply to Charlie Arehart

    I updated the connectors last week, just had a crash 5 minuets ago.

     

    As a side note, I added a scheduled task on my development machine that would "ping" each of our applications every 20 minuets. If the server returned anything other than a 200 status, I have it send me an email. I had this script running for a for a few weeks before I updated my connector and the server never crashed. I stopped the scheduled task on the same day I updated the connector, wanting to see if it was my task that was keeping the server alive, or if this config would truly fix the issue.

     

    Oddly enough, when the application pools just stopped, I do not see the same getRealPathFromConn() error. This is in my error log at the time of the event:

     

    Feb 19, 2013 2:40:56 PM org.apache.catalina.core.StandardServer await

    INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.

    Feb 19, 2013 2:40:56 PM org.apache.coyote.AbstractProtocol pause

    INFO: Pausing ProtocolHandler ["ajp-bio-8012"]

    Feb 19, 2013 2:40:56 PM org.apache.catalina.core.StandardService stopInternal

    INFO: Stopping service Catalina

    Feb 19, 2013 2:41:58 PM org.apache.catalina.core.AprLifecycleListener init

    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\ColdFusion10\\cfusion\lib;C:\\ColdFusion10\\cfusion\jintegra\bin; C:\\ColdFusion10\\cfusion\jintegra\bin\international;C:\\ColdFusion10\ \cfusion\lib\oosdk\classes\win

    Feb 19, 2013 2:42:01 PM org.apache.coyote.AbstractProtocol init

    INFO: Initializing ProtocolHandler ["ajp-bio-8012"]

    Feb 19, 2013 2:42:01 PM org.apache.catalina.core.StandardService startInternal

    INFO: Starting service Catalina

    Feb 19, 2013 2:42:01 PM org.apache.catalina.core.StandardEngine startInternal

    INFO: Starting Servlet Engine: Apache Tomcat/7.0.23

    Feb 19, 2013 2:42:13 PM org.apache.catalina.core.ApplicationContext log

    INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

    Feb 19, 2013 2:42:13 PM org.apache.catalina.core.ApplicationContext log

    INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 20.4-b02

    Feb 19, 2013 2:42:21 PM org.apache.catalina.core.ApplicationContext log

    INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

    Feb 19, 2013 2:42:26 PM org.apache.coyote.AbstractProtocol start

    INFO: Starting ProtocolHandler ["ajp-bio-8012"]

    Feb 19, 2013 2:42:27 PM com.adobe.coldfusion.launcher.Launcher run

    INFO: Server startup in 41013 ms

     

    I don't see an error. The first event there I believe (valid shutdown command) is when I hit restart on the windows machine. No errors. So what the heck? I had previously increased my IIS rapid fail to 50 and in event viewer of course there are 50 of them. Basically these two over and over again:

     

    • A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4924'. The data field contains the error number.
    • A process serving application pool 'domain.com' terminated unexpectedly. The process id was '10336'. The process exit code was '0xff'.

     

    And at 2:36 Application pool 'domain.com' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

     

    There are no coldfusion errors thrown today at all before 2:40, essentially when I restarted the server. Nothing in the exception log. I feel like all my CF update did was hide it from the log as the problem remains. I've turned my development scheduled task again in an attempt to keep the applcations running hoping there will be no issues. Any thoughts on this. I've been doing a lot of reading and it seems that the IIS 7.5, Win Server 2008, CF10 combo is entirely unstable.

     

    The isapi_redirect.dll located in Coldfusion10\config\wsconfig\1 is indeed dated 11/8/2012 1:18 PM.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 2:50 PM   in reply to colophonnewmedia

    Chris,

     

    Thank you for the note. We are have applied this fix, and so far so good.

     

    We also had to update a setting in CF admin setting for: “Maximum number of simultaneous Template requests”. The default for CF 10 is set to 25. The default install of our local copy of CF 10 has this set to 25. Our box was set to 10, as it was upgraded from 8 and 9 and we think this was left over from CF 8 and 9.

     

    However, according to Adobe, when the limit for Template requests is reached, the system is supposed to queue the rest. In our case it does not queue, it stops. So there is still some concern.

     

    We also don’t see how a request is equal to site activity for 1 site. We thought that “worker processes” would be related to a number of different processes that happened when web templates run.

     

    Ours is set to 0. Is that bad/incorrect. Does 0 in this case mean no limit?

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 2:30 PM   in reply to Lee Bartelme

    Lee, Maybe some of the CF10\config\wsconfig\1\isapi_redirect.log around 2:40pm to 1 hour before would be interesting?

    Probably less so but still could be good to check anything of concern in that time frame for CF10\cfusion\logs\coldfusion-out.log?

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 2:49 PM   in reply to carl type3

    So in the isapi_redirect.log there are alot of these at 14:34. And I mean A LOT, but they exist throughout:

     

    • [Tue Feb 19 14:34:02.404 2013] [9372:10556] [info] init_jk::jk_isapi_plugin.c (2634): Starting Jakarta/ISAPI/isapi_redirector/1.2.32 ()
    • [Tue Feb 19 14:34:02.404 2013] [9372:10556] [error] init_jk::jk_isapi_plugin.c (2779): Initializing shm:(null) errno=-1. Load balancing workers will not function properly
    • [Tue Feb 19 14:34:02.420 2013] [9372:10556] [info] init_jk::jk_isapi_plugin.c (2813): Jakarta/ISAPI/isapi_redirector/1.2.32 () initialized
    • [Tue Feb 19 14:36:04.818 2013] [12000:11812] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 1 of max reuse connection 250 and total endpoint count 250 (LESS OF THESE)

     

    After the restart sets of these:

     

    • [Tue Feb 19 14:41:00.206 2013] [9720:11712] [info] TerminateFilter::jk_isapi_plugin.c (2343): Jakarta/ISAPI/isapi_redirector/1.2.32 () stopping
    • [Tue Feb 19 14:42:03.081 2013] [2944:2988] [info] init_jk::jk_isapi_plugin.c (2634): Starting Jakarta/ISAPI/isapi_redirector/1.2.32 ()
    • [Tue Feb 19 14:42:28.805 2013] [2624:3044] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 1 of max reuse connection 250 and total endpoint count 250

     

    There is nothing in the outlog that looks out of the ordinary or that looks like an error.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 3:26 PM   in reply to Lee Bartelme

    Lee, case it helps here is an extraction of what ISAPI log looks like while working normally take from CF stop to load commencing.

     

    [Tue Feb 19 12:10:34.919 2013] [2596:1912] [info] TerminateFilter::jk_isapi_plugin.c (2343): Jakarta/ISAPI/isapi_redirector/1.2.32 () stopping
    [Tue Feb 19 13:25:26.442 2013] [5888:5868] [info] init_jk::jk_isapi_plugin.c (2634): Starting Jakarta/ISAPI/isapi_redirector/1.2.32 ()
    [Tue Feb 19 13:25:26.473 2013] [5888:5868] [info] init_jk::jk_isapi_plugin.c (2813): Jakarta/ISAPI/isapi_redirector/1.2.32 () initialized
    [Tue Feb 19 13:25:26.677 2013] [5888:5868] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 1 of max reuse connection 250 and total endpoint count 250
    [Tue Feb 19 13:25:30.677 2013] [5888:3148] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 2 of max reuse connection 250 and total endpoint count 250
    [Tue Feb 19 13:25:30.692 2013] [5888:5868] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 3 of max reuse connection 250 and total endpoint count 250
    [Tue Feb 19 13:53:50.503 2013] [5888:3124] [info] ajp_send_request::jk_ajp_common.c (1658): (cfusion) all endpoints are disconnected, detected by connect check (3), cping (0), send (0)
    [Tue Feb 19 13:59:04.607 2013] [5888:2804] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 4 of max reuse connection 250 and total endpoint count 250
    [Tue Feb 19 13:59:05.060 2013] [5888:3124] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 5 of max reuse connection 250 and total endpoint count 250
    [Tue Feb 19 13:59:05.107 2013] [5888:4892] [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 6 of max reuse connection 250 and total endpoint count 250

     

    > [info] init_jk::jk_isapi_plugin.c (2634): Starting Jakarta/ISAPI/isapi_redirector/1.2.32 ()
    I would think that odd if a stop did not proceed it. Did IIS WWW CF stop?
    > [error] init_jk::jk_isapi_plugin.c (2779): Initializing shm:(null) errno=-1. Load balancing workers will not function properly
    "Error" usually means there is a problem that needs to be resolved. I have not come across this one yet. Suggest follow that error detail perhaps will lead to solution.
    > [info] init_jk::jk_isapi_plugin.c (2813): Jakarta/ISAPI/isapi_redirector/1.2.32 () initialized
    There we have part of a start-up process.
    > [info] ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 1 of max reuse connection 250 and total endpoint count 250
    Normal usage. Depending on load value for "reuse count is" will increment.

     

    While you can get more ISAPI log details via changing isapi_redirect.properties altering log_level=info to debug I would not recommend it in a production environment. If you know what the process is that leads up to crash and can simulate that process occurring then log_level=debug can be useful.

     

    Again, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 3:38 PM   in reply to Lee Bartelme

    @Lee
    >INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path

     

    I don't think that will necessarily be causing a problem and you may not be using APR. Find information about APR here:


    http://tomcat.apache.org/native-doc/index.html

     

    You could download APR extract tcnative-1.dll (take care there is Win 32 and 64 bit) and copy that to CF10\cfusion\lib then restart CF10 Application Service. You will notice the log details change to EG:


    org.apache.catalina.core.AprLifecycleListener init
    INFO: Loaded APR based Apache Tomcat Native library
    org.apache.catalina.core.AprLifecycleListener init
    INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false],
    random [true].
    org.apache.coyote.AbstractProtocol init
    INFO: Initializing ProtocolHandler ["ajp-apr-8012"]
    etc

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 3:49 PM   in reply to carl type3

    After Googling " [error] init_jk::jk_isapi_plugin.c (2779): Initializing shm:(null) errno=-1. Load balancing workers will not function properly" I have come across others with the same issue dealing with the redirect.dll file. They speak of patching the file, but I don't think that is an option for me, as this dll should have to come from Adobe/CF correct?

     

    Should I be running my applicatin pools with Enable 32-bit set to true. They are set to false currently. This really seems like an Adobe CF/IIS issue doesn't it. What the hell is going on. I read in one thread they think this is happening durring applicatin recycle at some point or possibly due to load. My scheduled task that hits my applications every 20 minuets I think keeps all application/worker processes from shutting down, thus keeping the application pools alive, but my uneducation may prove it's all in my head.

     

    I have no idea if we are using APR or how to even tell. Should I just download the file and see what happenes. Is there some sort of drawback to doing so? Any harm that can be done? My thinking is this is a load issue as it NEVER happens on our development machine.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 4:19 PM   in reply to Lee Bartelme

    As you say we can’t just go get the current Tomcat Connector (1.2.37) extract ISAPI DLL and use that. CF have given us a modified 1.2.32 Connector and I guess CF Dev team have good reasons for doing so.

     

    Having an error is a help, finding what the error really means can be a bit like following bread crumb trail. What to recommend I am not sure yet. Perhaps your workers.properties syntax would be interesting. I have been using JMX with jconsole to monitor load at the CF end of the Connector and found that helpful. Let me know if you want more details on that.

     

    Apply APR DDL if you want. I have on some CF10 environments with APR DLL applied but for now I think time spent on APR will distract from what the real problem is. Perhaps another day. I noticed it in the thread and so thought I would pass on what I know.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 4:22 PM   in reply to carl type3

    Thanks for your input. I'm a programmer assigned to also keep the server running if you know what I mean, so my skills as far as a Server Admin are pretty basic. I know what I am doing to get things done, but as far as real debugging/troubleshooting on the server side of things, I am lacking. I feel like my only solution is to wait for the next CF patch. I think my scheduled task keeps things running for now, and while not ideal, the server hasn't crashed while running the script yet.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 5:18 PM   in reply to Lee Bartelme

    Lee - know where your coming from.

     

    With the ISAPI log mention of "error" and "load balance" curious what is syntax of CF10\config\wsconfig\1\workers.properties EG:

     

    worker.list=cfusion


    worker.cfusion.type=ajp13
    worker.cfusion.host=localhost
    worker.cfusion.port=8012
    worker.cfusion.max_reuse_connections=250

     

    Read this informative blog, I know there is a lot there:

     

    http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connecto r-configuration

     

    With your mention of running schedule task to keep things running that can point to timeout issues. You might do well to add time out syntax. IIS WWW and CF need restarting to apply changes. EG

     

    workers.properties add -
    worker.cfusion.connection_pool_timeout = 60

     

    server.xml CF10\cfusion\runtime\conf alter AJP section -
    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" connectionTimeout="60000" />

     

    HTH again, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 19, 2013 5:27 PM   in reply to carl type3

    My workers.properties looks exactly as you have displayed. Should I try to corespond my connection_pool_timeout with my application timeout in my CF applications? Is 60 = 60 minuets? I'll read through your link in the meantime and hopefully it will answer my question of if I should have to edit the XML and the workers.properties or if it's an either or type of thing. Thanks again for your expertise.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 20, 2013 5:54 PM   in reply to Lee Bartelme

    workers.properties connection_pool_timeout = 60 is in seconds where as server.xml AJP connectionTimeout="60000" is in milliseconds. General recommendation is the time outs should match and references offer 60 seconds or 600 (10 minutes) for the ISAPI pool and AJP threads. I would not set to 60 minutes however as always do your own testing to know what value suits best. You are welcome - but I am no expert.

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 4, 2013 10:32 PM   in reply to Lee Bartelme

    @Lee it has been a while. How is that going for you? Regards, Carl.

     
    |
    Mark as:
1 2 3 Previous Next
Actions

More Like This

  • Retrieving data ...

Bookmarked By (1)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points