SEVERE: The web application [/] created a ThreadLocal

I'm getting the same output in my ColdFusion-error.log files.  It seems to be preceded by the following lines

INFO: Stopping service Catalina

Nov 2, 2012 10:24:21 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

Maybe it's just me, but anytime I see SEVERE in an error log I have some concern.

I currently have ColdFusion 10 Update 4 installed and I was getting the same errors with Update 2

Anybody have some input on this?

Since you mention having applied updater 4, have you re-run the web server configuration? That’s a requirement for the CF10 updaters (mentioned on the updater page, but easily overlooked).

it may not make any difference, but I wanted throw it out in case it might.

Similarly, be aware that sometimes when you run the web server config tool, it may seem to “work” when it has not. I’ve found that it’s necessary to “run it as admin”, which means doing start>adobe>coldfusion 10 and then right-clicking on “web server onfig tool” to choose “run as administrator”.

Hope that may help.

/charlie

I did run the wsconfig tool as instructed, but I did not "run as administrator" as you suggested.  I'll give that a shot and let you know if anything changes.

I used the "run as administrator" on the wsconfig tool and I'm still getting the SEVERE errors.  They seems to show up in the ColdFusion-error.log file after the "ColdFusion 10 Application Server" service is restarted.

Any ideas?

Thanks,

Christian

OK. I’d next recommend you check the logs for the CF10 updater 4 you say you applied, to make sure it was really applied and without any errors. Since it was supposed to fix these sort of problems, we’d really want to make sure it was applied before concluding that it did not. The logs for the udpaters are in a hf-updates dir, within the CF10 instance where they were applied (such as C:\ColdFusion10\cfusion\).

Speaking of that, are you by any chance running multiple instances? And is your web server set to connect to one of the instances other than the default cfusion instance? If so, can you confirm as well that you applied the updater to THAT instance? Besides looking in the CF Admin for that instance, again look in the logs for the update, such as C:\ColdFusion10\instancex\hf-updates\.

/charlie

I've installed Update 4 on both my Dev and Test servers.  They are both single instances and they are also both behaving the same.  The SEVERE errors exist before and after Update 4.

I looked at the log files for Update 4 and everything says "Status: SUCCESSFUL" so it looks like it was installed succesfully on both servers.  I can email the log file directly to you if you'd like.

My servers are 64bit Windows Server 2008 R2 on VM ESX 4.0 for your reference.

Thanks,

Christian

Well, emailing the logs to me won’t help much. And especially if they say that all operations were “successful”.

But along those lines, are you confirming that it’s saying that for all actions within the log? Worth double-checking.

If that’s not the issue, and you are still having issues, I’d recommend you raise your concern on one of the blog entries (at blogs.coldfusion.com), to get the attention of the engineers. They are not required to respond here (or there, technically) but you just may get their attention their as far more people read the blog than the forums, I’d suspect.

Hope you resolve the problem eventually.

/charlie

That is correct, all of the actions have "Status: SUCCESSFUL" in the Update 4 log file.

I'll check out the blogs.  I appreciate your input, Charlie.

Thanks,

Christian

This definitely points to a severe problem. But it is not with your code. This occurs at the level of the Java Virtual Machine, and a ColdFusion library is the cause. A datetime class in the Java package coldfusion.util behaves in a way that can cause a memory leak.

It is significant that this occurs during shutdown. As the error message says, one or more ThreadLocal date objects that had been created couldn't be garbage-collected during shut-down. Such runaway rogue threads may retain references that survive a restart of a web application. This can eventually lead to a memory leak.

The following line from the error message tells us why we shouldn't panic: "Threads are going to be renewed over time to try and avoid a probable memory leak". However, the ColdFusion team should look into this while the going is still good.

Were you able to find a resolution for this?  I too am experiencing this error on Update 2 and now Update 4.

SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@202d2c30]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1352746379829,areFieldsSet=true,areAllFieldsSet=true,le nient=false,zone=sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstS avings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=Ameri ca/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode= 3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,end Month=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalD aysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=10,WEEK_OF_YEAR=46,WEEK_OF_MONTH=3,DAY_OF_MONTH=12, DAY_OF_YEAR=317,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=2,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE =52,SECOND=59,MILLISECOND=829,ZONE_OFFSET=-28800000,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

I checked out http://blogs.coldfusion.com/ but don't see any info.

Thanks,

Mark

and I have the exact same issue with Updater 6 installed. Has there been any resolution to this?

I am havng issues witrh getting the console to work in CFB after updating to CF10 on all 3 of my dev machines (this thread: http://forums.adobe.com/thread/1113052) and was looking through the CF logs for possible reasons and stumbled across this error.

So not saying they have any relation, but now that makes 2 items that are troubling after moving from CF9 to 10...

I am also seeing this error.  Our CF10 box is a fresh install not an upgrade (not that that makes any difference) with Update 6 installed.  I do have multiple instances installed on this box and have veriffied that update 6 was successful on all.  I have re-run the configuration tool for each instance as "Administrator" and we still see the issue.  Has anyone made any progress with this??

SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@1b99ba1]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1355777649150,areFieldsSet=true,areAllFieldsSet=true,le nient=false,zone=sun.util.calendar.ZoneInfo[id="America/New_York",offset=-18000000,dstSavi ngs=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/ New_York,offset=-18000000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,star tMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth= 10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInF irstWeek=1,ERA=1,YEAR=2012,MONTH=11,WEEK_OF_YEAR=51,WEEK_OF_MONTH=4,DAY_OF_MONTH=17,DAY_OF _YEAR=352,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=1,HOUR=3,HOUR_OF_DAY=15,MINUTE=54,SEC OND=9,MILLISECOND=150,ZONE_OFFSET=-18000000,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Any updates on this I am getting the same still in CHF 8 on ubuntu 12.04. Server is crashing about once every 24 hr, unknown yet if this is related to the cause.

Thanks,

~ Ben

I am having the same problem. 

When I run wsconfig.exe I have an empty list under "Configured Web Servers."  When I click Add... I get the following:

Note that the OK button is greyed out. If I choose another web server and then select IIS again, I do get the OK button. When I click Ok I get the message "An unknown error ocurred."

I'm confused by all this, because my server is functioning, so ... why does this tool think I'm not connected?

Hi,

Did you try running the tool in Admin mode? If that doesn't work, please refer to http://help.adobe.com/en_US/ColdFusion/10.0/Admin/WSc3ff6d0ea77859461172e0811cbf364104-7fd 9.html for running the wsconfig tool via command line.

Regards,

Anit Kumar

Hi there,

We've also been experiencing the same errors in our logs after the clock change to BST here in the UK.

I have been unable to find any other information relating to this issue online apart from this thread so have posted a bug to https://bugbase.adobe.com/index.cfm?event=bug&id=3540118

I will update this with any further information I find.

Kind regards,

Tom

Hi Tom,

I have added a note and a vote to your bug post.

The note i have added is below. If anyone has any suggestions for a workaround please do post them.

I have tried running the webserver config as administrator but this has not resolved the situation.
Changing your clock to Pacific time seems to stop the error, but then the server is on the wrong time.

Kind regards
Luke

Note added to Toms bug report:
I am also seeing this behavior, this is a new install updated to CF10 update 9. I have re-ran the webserver config tool as admin as per suggestions here: http://forums.adobe.com/thread/1085240 but to no effect.

The server clock is set to time zone (UTC) Dublin, Edinburgh, Lisbon, London.

If i change the clock to (UTC -0800) Pacific Time (US & Canada) i do not get the error.

The error in the logs is: Apr 11, 2013 4:02:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@6bbdcdd5]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1365690925920,areFieldsSet=true,areAllFieldsSet=true,le nient=false,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000 ,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offse t=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,s tartDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfW eek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR =2013,MONTH=3,WEEK_OF_YEAR=15,WEEK_OF_MONTH=2,DAY_OF_MONTH=11,DAY_OF_YEAR=101,DAY_OF_WEEK= 5,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=3,HOUR_OF_DAY=15,MINUTE=35,SECOND=25,MILLISECOND=920 ,ZONE_OFFSET=0,DST_OFFSET=3600000]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak

For anyone who has this issue with time and CF10 - are you using the version of Java that comes bundled with CF?

I was and i found that installing Java as a stand-alone (ie to c:\program files) then modifying the JVM.config file to point java home to the new location has solved the error.

Download Java from here: http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

install it

navigate to C:\ColdFusion10\cfusion\bin\jvm.config

backup
open in notepad

change "java.home=C:\\ColdFusion10\\jre" to the new location - this was "java.home=C:\\Program Files\\Java\\jdk1.7.0_17\\jre" for me

save

restart CF

We had four SEVERE errors on the pre-shipped JRE wirh CF10 64 bit. We installed the latest Sun JRE and upon restarting the server this has reduced the errors to just one. However, on a second restart we are back to 4 SEVERE errors again.

We also noticed that our memory heap is large when starting the server (we use FusionReactor) and unless a garbage dump is done, this memory usage stays high. The garbage dump seems to release a massive amount of memory too (254MB!).

Anyone got any ideas to solve this one?

Interesting observation, wombat. So to be clear, are you referring to JVM 6 or 7? CF 10 originally “pre-shipped” (as of May 2012) with Java 6, but a new installer (for Windows only) in March of 2013 causes it to come with Java 1.7.  (And a hotfixe released at the same time also allowed anyone running CF10 or CF9 to change to Java 1.7.

So it may be helpful for readers to know which you are referring to (for both what you found “pre-shipped” and what you changed to).

Also, while I don’t mean to cause embarrassment or pick a fight, I do want to offer another clarification of a common misconception: doing a GC (or a “garbage dump”, as you referred to it) is never necessary to solve a memory problem. Even if you find the heap to be high (and yes, CF can indeed hold the heap high for a few minutes after startup), there’s no reason for you to do a GC to try to “make that go away”. The fact is that if and when the JVM gets memory pressure (because of requests running) and more heap is needed, it is THEN when the JVM would do a GC--on its own.

Now, if it does (or you try to do) a GC, but it can’t collect much, then that’s a different problem. Something is holding memory, and if that is not relieved, then you may well get an outofmemory error.

But bottom line: doing a GC is only something that helps you see how much memory can be garbage collected. It’s not something you need to do. The JVM will do it itself, if it senses a need to.

Hope that’s helpful.

/charlie

We have this error too - could it be a problem with scheduled tasks being initialised on start up ?

The error seems to happen once for every scheduled task we have in CFADMIN.

We're on CF10 Standard which had an issue with schedule tasks after release ???

Nov 13, 2013 5:21:29 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@7ce474af]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1384362572889,areFieldsSet=true,areAllFieldsSet=true,le nient=false,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000 ,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offse t=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,s tartDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfW eek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR =2013,MONTH=10,WEEK_OF_YEAR=46,WEEK_OF_MONTH=3,DAY_OF_MONTH=13,DAY_OF_YEAR=317,DAY_OF_WEEK =4,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=5,HOUR_OF_DAY=17,MINUTE=9,SECOND=32,MILLISECOND=889 ,ZONE_OFFSET=0,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

. . . . . . We get 14of these

followed by 4 of

SEVERE: The web application [/] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@194ad86c]) and a value of type [org.apache.derby.iapi.services.context.ContextManager] (value [org.apache.derby.iapi.services.context.ContextManager@6f4582b8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

We're on CF10 Standard Update 11 ( it happened on update 4 too )  Win 2008,  IIS7.5  We're on GMT

It may be related but we have a very serious issue with the server not responding to flash remote calls under very modest load.

as follows - REMOTEOBJECT ERROR Client.Packet.Encoding Error deserializing client message. null

This problem started after a restart on November 5th ( GMT - first restart after the clocks changed ?? )

Could this be due to running out of threads / Mem leak due to the date error?

We are experiencing this problem as well. It sounds like lukester85 has solved it by having CF use an install of Java outside of the internal Coldfusion Java install.  Has anyone tried this who has been eperiencing this problem?

This solved it for us too.

Hi,

I tried to install a JRE outside of Coldfusion Java install, but I still have the SEVERE log entries...

Hi Stephane/All,

Have you solved this issue?

I have intalled a newer Java version, but it did not help.

Thanks!

Hi Feb,

Not yet solved for me...

We are still seeing the same problem as well.  We are running ColdFusion 10 multi-instance, and only seeing the server freeze up randomly in one particular instance.

Still having this problem too and it's driving me nuts.  I don't know what changed, because CF10 was working fine for a while.  Now Coldfusion stops responding one or more times a day with a number of threads in the wait chain.  The only way to fix it is to restart the service.  When I do, I get the same error messages as others in this thread in the Coldfusion-error.log on startup:

"SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

Some of these are followed by this error message somehow related to Solr collections:

"SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."

May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

It seems like the number of errors is about the same as the number of scheduled tasks.  It doesn't look like any useful error, exception, or access messages get logged before Coldfusion stops responding.

My setup is a virtualized 2008 Win Server-64bit, Apache 2.2 (stock 32bit), CF10-64bit, update 13

I've tried: 

1. Various security improvements
2. Searching for cfml errors or recursive code
3. Adding RAM (8GB now)
4. Pausing scheduled tasks
5. Removing & reconnecting the server connector as Administrator as suggested
6. Increasing the JVM heap size
7. Using the latest 1.7.0_55  JRE externally

Next I guess I'll have to try reinstalling Coldfusion or rebuilding the server from scratch.  I have to say that moving to Railo is looking more appealing than upgrading to CF11.  This should just work without so much grief...