• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

Help! Unable to create new native thread

New Here ,
Jun 23, 2010 Jun 23, 2010

Copy link to clipboard

Copied

Ok I see this problem all over the web but most of the posts are many years old and still leave me a little confused.  Often at random times on the same tags like CFLDAP I get the error "Unable to create new native thread".  Then a few minutes later the same tag on the same page works just fine.

I have 4GB of memory in this box with 1400 MB set as the maximum Java heap size by the way.

Here are my arguments from the jvm.config file.

# Arguments to VM

java.args=-server  -Xmx1400m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib -Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/flex/jars,{application.home}/../wwwroot/WEB-INF/cfform/jars,"C:\\Program Files\\Apache Software Foundation\\Apache2.2\\htdocs\\InterWeb-Prod\\includes\\classes"

#

Here is a typical error that comes through.

struct     

Browser     Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.70 Safari/533.4    

DateTime    {ts '2010-06-23 10:07:44'}  

Diagnostics unable to create new native thread null <br>The error occurred on line 122.

GeneratedContent      

HTTPReferer http://--------------------/index.cfm?FuseAction=HR.main    

Mailto      interweb@--------------------- 

Message     unable to create new native thread      

QueryString fuseaction=ManagedLists.MailSubscriptions     

RemoteAddress     192.168.250.124

RootCause   struct    

Message     unable to create new native thread      

StackTrace          java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:597) at com.sun.jndi.ldap.Connection.<init>(Connection.java:208) at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:112) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2504) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:263) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:76) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288) at javax.naming.InitialContext.init(InitialContext.java:223) at javax.naming.InitialContext.<init>(InitialContext.java:197) at javax.naming.directory.InitialDirContext.<init>(InitialDirContext.java:82) at coldfusion.tagext.net.LdapTag.do_ActionQuery(LdapTag.java:839) at coldfusion.tagext.net.LdapTag.doStartTag(LdapTag.java:616) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661) at cfadfunctions2ecfm1473603830$funcADLOOKUPUSERNAME.runFunction(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\includes\adfunctions.cfm:122) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at cfadfunctions2ecfm1473603830$funcADLOOKUPMULTIPLEUSERNAMES.runFunction(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\includes\adfunctions.cfm:54) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:418) at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:324) at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:59) at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:277) at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:192) at coldfusion.runtime.CfJspPage._invokeUDF(CfJspPage.java:2471) at cfdsp_managedlists2ecfm55598920.runPage(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\hr\managedlists\dsp_managedlists.cfm:232) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661) at cffbx_switch2ecfm2121232114.runPage(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\hr\managedlists\fbx_switch.cfm:23) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661) at cffbx_fusebox30_cf502ecfm1180471387._factor4(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\fbx_fusebox30_cf50.cfm:241) at cffbx_fusebox30_cf502ecfm1180471387._factor5(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\fbx_fusebox30_cf50.cfm:1) at cffbx_fusebox30_cf502ecfm1180471387.runPage(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\fbx_fusebox30_cf50.cfm:1) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661) at cfindex2ecfm749274359.runPage(C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\InterWeb-Prod\index.cfm:23) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:273) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:86) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.CfmServlet.service(CfmServlet.java:175) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

I could really use some help figuring this out.  We are getting random errors a few times a day and usually in places where we are using CFLDAP.

Thanks,

David

Views

7.6K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jun 23, 2010 Jun 23, 2010

Copy link to clipboard

Copied

This is one of those times when an "outofmemory" error is NOT what it seems.  The answer is NOT because you're running out of heap space, and the solution is NOT to increase the heap space (which as you may know, if on 32-bit version of CF, you're already about maxed out at anyway at 1.4gb).

This may shock some, but I would bet the answer for you is to lower your max heap size instead. Heresy, I realize.

I don't have time to elaborate, but again if this is a 32-bit OS (or CF is running in 32-bit mode), you may know you can only get 2gb of user space per process. If you do use up 1.4gb of it with heap, and then also the 192m you show for permgen (which is the default and usually an ok value), and then add in some OS overhead, what's left gets used for the "stack space" (a place to allocate a stack for each running thread). If there's not enough room to allocate a stack for a running thread, you get that error.

TO be clear, you can't control the size of the stack, so instead I'm proposing that you lower the heap to give it more breathing room.

You will read of some proposing to use the -xss switch--that's to control the size of each entry in the stack (not the total stack space size). Sadly, it's risky business trying to fix things by changing the xss. For one thing, I've not found a good java reference to confirm what the current size is on any given jvm, so how do you know if the value you use (from some blog entry suggestion, for instance) is making the stack entry size bigger or smaller than your current default?

Some may wonder, "so if a stack entry gets allocated for each thread, is this related to the size of the simultaneous request threads entry in the CF Admin?" Well, it is, but that's really only just one of several thread pools in a CF system. There are threads for every database connection, threads for request waiting to run (the queue), threads to manager client variable repositories, threads used for internal java processes, and so on.

You may sometimes find that the error above is driven more by an over-abundance of threads being created, but in my experience, that's not the problem. It's the stack space being allocated in leftover memory space being squeezed by the heap and permgen. Those are easier to lower, though I'd focus on the larger heap. Some servers really don't need the heap as large as they make it. As long as you don't get corresponding outofmemory errors reporting "out of heap" problem if you lower the heap, you're generally ok.

As so many things, it's a balancing act. HTH.

/charlie


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 24, 2010 Jun 24, 2010

Copy link to clipboard

Copied

I had actually wondered about that and did consider bringing it down a little.  I learned a couple more little facts yesterday that may help in diagnosing the issue.  The error was happening at random but ususally on either queries or CFLDAP tags.  there was one template in particular that was erroring more often than others.  On that page there was a CFLDAP tag in a lengthy loop.  If I were to reload that template over and over, the memory that JRun was using would go up and up until about 1GB and that is when the crash would happen.  I reproduce it over and over.  I got the crashing to stop by changing the -server switch to -Xint but the site runs a little slower now.

The reason I set the heap size to 1400 is because at 1024 the site was really slow.  We upped the memory in the box and the heap size.

David

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jun 29, 2010 Jun 29, 2010

Copy link to clipboard

Copied

I have an additional suggestion, you have a maximum setting for the Permanent Generation XX:MaxPermSize=192m I would add a start setting and up the max as follows, after reducing the total heap size to a maximum of 1200

XX:MaxSize=128m XX:MaxPermSize=256m

CFWhisperer http://www.cfwhisperer.com/

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 26, 2010 Sep 26, 2010

Copy link to clipboard

Copied

Absolutely unbelievable that this has been going on for 4 YEARS and there are still NO definitive answers from Adobe!!

I have a 24x7 production server that services 180,000 clients in 300 countries around the clock that cannot afford to be down for more than a few minutes at a time.

I'm running Windows 2008 32-bit with 4GB of RAM because I have some graphics software that cannot tolerate a 64-bit OS.  Everything has been running fine for the past few months, but now it's starting to crash more and more frequently.  I have CF 8.0.1.195765 and Java 1.6.0_04

One of my other symptoms I have noticed is that any work done using RDP starts getting really really bogged down.  Opening Windows Explorer and searching for files can take minutes.  Restarting the CF service will sometimes fail to complete, and you have to wait a few minutes before the manual Start option appears in the dropdown.  Maybe coincidentally, the Performance Monitor will fail to add any of the ColdFusion metrics, for example Running Requests and Queued Requests, and the Event Monitor shows an error relating to an 8-byte boundary not being maintained.

Can it be that companies are really suffering with multiple crashes per day and NOBODY has found a cure for this???????????

Dave Morris

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

Absolutely unbelievable that this has been going on for 4 YEARS and there are still NO definitive answers from Adobe!!

There's no definite answer from Adobe, because there's no definitive single question.  These errors come up because - for any number of possible reasons - the JVM becomes overloaded.  And it's almost always down to an issue with the application code being run, and nothing to do with CF, per-se.  So what's Adobe supposed to do?  Come and fix your application for you?  There are plenty of diagnostic tools out there - some built into CF itself - which you could use to sort the problem out.  Or you could hire someone like Mike Brunt to come and have a look at your set up.  All it takes to fix these things is to actually do something about fixing it, rather than slamming one's fist on the desk saying "why doesn't Adobe do something!"

I thought - and my knowledge in the area isn't expansive - the inability to allocate threads was down to the app being choked up: threads are a finite commodity, and if requests are queuing up faster than the JVM can clear them, you're gonna get errors eventually.  What causes this sort of thing?  More traffic than the server can process.  This could be because of server kit, or it could be due to code that doesn't scale to the requirement.  Usually the latter.  Basically one cannot polish a turd.

I have a 24x7 production server that services 180,000 clients in 300 countries around the clock that cannot afford to be down for more than a few minutes at a time.

I'm running Windows 2008 32-bit with 4GB of RAM because I have some graphics software that cannot tolerate a 64-bit OS.  Everything has been running fine for the past few months, but now it's starting to crash more and more frequently.  I have CF 8.0.1.195765 and Java 1.6.0_04

How many concurrent requests for 180000 clients generate?  The number of clients matters less than how heavily the site is used.  But, to be honest, I get the impression the implication is "it's a busy site", in which case I think your hardware is a bit on the lean side.  Also, if it's a mission-critical app, running it on a single server is probably ill-advised, just from a fault-tolerance perspective even before one starts thinking about performance.

If I was in your situation, I'd consider the following options:

* upgrade your JVM to the most recent version (1.6.0_21?)

* put more RAM in the box, and run a second CF instance on it.

* get another box.

* refactor the environment so the graphics software runs on a different box, upgrading the CF server to 64-bit (again, with more RAM)

* run some diagnostics on the server, seeing what's taking up threads and where any bottlenecks might be

* get an expert in to do the analysis side of things.

* stop expecting Adode to fix "something".

One of my other symptoms I have noticed is that any work done using RDP starts getting really really bogged down.  Opening Windows Explorer and searching for files can take minutes.  Restarting the CF service will sometimes fail to complete, and you have to wait a few minutes before the manual Start option appears in the dropdown.  Maybe coincidentally, the Performance Monitor will fail to add any of the ColdFusion metrics, for example Running Requests and Queued Requests, and the Event Monitor shows an error relating to an 8-byte boundary not being maintained.

You don't, by any chance, store your client variables in the registry do you?  The only time I've seen this happen is when the CF box is munging the OS environment, and the only way I can think this could happen if it was clogging the registry.  Never store client variables in the registry.

--

Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

Well, if it were my application at fault, I'd be happy to know that. But other than converting instances of the custom tag CFX_IMAGE over to use the native CFIMAGE tag in more of my templates, nothing has changed between the time that the server would stay up for weeks on end, to now when it is crashing several times per day.

I despise the Registry and don't store anything there.

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

Well, if it were my application at fault, I'd be happy to know that. But other than converting instances of the custom tag CFX_IMAGE over to use the native CFIMAGE tag in more of my templates, nothing has changed between the time that the server would stay up for weeks on end, to now when it is crashing several times per day.

Well, equally, it sounds like nothing has changed with the CF install either.  So it makes no more sense to point the finger @ CF than it does to point the finger at your app.

And regarding "nothing's changed"... you've not updated the codebase at all?  You've not added more features to the site?  You've not had an increase in clients or traffic?  The site's usage patterns haven't changed?

--

Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

In July I moved everything to a new server, and it has been running nicely since then.  For almost a month it ran without any glitches.  I have been doing some development work, but only occasionally testing it, so nothing much has changed in the production environment other than the CFImage use that I described.  I still have over 120GB of hard disk on the C drive, so I'm not running into disk space problems.  The number of simultaneous users online at any given time is around 60 to 150, but the crashes seem to also take place at night when there may only be 40 online. So I don't think that loading is the issue.

This appears to be something that builds up over the course of the day, getting worse and worse, until it finally melts down.  And that has nothing to do with my application.

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

the crashes seem to also take place at night when there may only be 40 online. So I don't think that loading is the issue.

Like when you might be running some scheduled tasks to reindex stuff, or [whatever]?  So when there is "non-standard" activity taking place?  Could the DB be doing any scheduled maintenance?  Or even the network doing a back-up or something like that?  If the traffic is low, could something in the app be timing out or something, and cause some reinitialisation code to run that is a bit hairy?

Have you actually done any profiling - like with FusionReactor, or JConsole or whatever - to see what's going on?

This appears to be something that builds up over the course of the day, getting worse and worse, until it finally melts down.  And that has nothing to do with my application.


Other than the fact that it's your application that's the thing that is running, you mean?

I don't mean to be obtuse, but if your application wasn't running... the server wouldn't crash right?  So it is almost certainly going to be something about your application that is causing this.  That is more likely than part of CF is bung on your server and not bung on anyone else's server.  Even if it's some edge case like there being a conflict caused when a <cfldap> call is made straight after a <cfimage> call then there's a null pointer and the thread hangs up (obviously that's silly, but you take my point), then it's your code causing that to happen.  Even if your code is revealing an underlying bug in CF.

I think it might be a good approach to change from "it's not me, it's not me", to "it probably is me, but I just don't understand why it's me".  If for no other reason than you're not going to get anywhere if you just sit there going "it's not my fault".

--

Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

This isn't my first rodeo.  I've been developing ColdFusion since V4.5 and was writing software in 1975 on PDP-11 machines.  I know how to determine whether something is my fault or the system's fault.

When Performance Monitor won't let me add the ColdFusion metrics because they are not aligning on an 8-byte boundary, I am pretty certain that something is out of whack, and it's not my applications.

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

LATEST

Well I guess mileage varies then: I wouldn't use an OS-level tool to investigate an issue with my JVM, I'd use a Java profiling tool (like JConsole).  Or a specific CF profiling tool (like FusionReaction).

But you seem to know best, so I'll leave it with you.

Best of luck.

--

Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Sep 27, 2010 Sep 27, 2010

Copy link to clipboard

Copied

One of the things I have learned is that many of the JVM messages can be misleading, for instance there is a default maximum of 1,000 threads available to JRun so it is very unlikely that you ran out of all threads.  Speaking of threads, this is a good thread which hopefully many will learn from and I really appreciate the mention.  In the interests of helping the community, can you send me your jvm.config and jrun.xml files to mbrunt AT go2ria.net, also do you have a Standard or Multi-instance install?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation