• Global community
    • Language:
      • Deutsch
      • English
      • EspaƱol
      • FranƧais
      • PortuguĆŖs
  • ę—„ęœ¬čŖžć‚³ćƒŸćƒ„ćƒ‹ćƒ†ć‚£
    Dedicated community for Japanese speakers
  • ķ•œźµ­ ģ»¤ė®¤ė‹ˆķ‹°
    Dedicated community for Korean speakers
Exit
0

Help: CF10 (all hotfixes) on Win2K3 starts to consume up to 100% cpu (coldfusion.exe)

Explorer ,
Jan 03, 2013 Jan 03, 2013

Copy link to clipboard

Copied

hi

We have a CF9 latest updates running on a Win2K3 server  under Apache 2.2.22 and JRun4 for years. Now, I wanted to upgrade to CF10.

I first updated to Apache 2.2.23 with ssl 1.0.1c

For reassurance, I just disabled all JRUN CF9 services as seen in the Task Manager.

Then I installed CF10. The installer mentioend different  NET Ports. I accepted them.

Install continued without a problem, CF10 was setup and I configured it. I then ran our main ap which also worked fine.

Then, after some few hours while I was checking the health of it, I figured that  COLDFUSION.EXE takes up all CPU time.

Server response  is still there but sluggish.I have not seen any reason why the cpu usage increases.

I have to kill coldfusion.exe to cut cpu load. however, a few minutes after that (about 10-20 minutes) coldfusion.exe again begins to climb up. it takes about 5-10 minutes to hit 100% ...

I have no idea how to deal with that (except going back to CF9)

On my developer CF10 (same config) I have not seen that behavior. Howewer, this server is not exposed to the net, the productive one is ... there might be some deep links pointing somewhere ... however, our server is not a busy server, so it should not face such a cpu load .. as said it ran under CF9 for years without such problems.

any hints how to detect the cause of this? Can I use any CF10 tools to figure out this problem?

any other clues to that?

it's urging ... as usual ...

Thank

Martin

Views

3.4K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 06, 2013 Jan 06, 2013

Copy link to clipboard

Copied

Hi Martin. Any errors or warnings in log files CF10\cfusion\logs\ coldfusion-error.log coldfusion-out.log ?

How about enable CF10 Metrics logging, CF10admin > Debugging & Logging > Debug Output Settings > then check metrics.log after CF restart to enable?

I know post CF10 update 5 WSCONFIG had some webserver connector updates in case of Windows and IIS. I guess with Apache you would be using a connector / extension / mod_jk but I have not been following CF10 update matters that relate to Apache webserver. Could be worth knowing if you need to apply a CF10 WSCONFIG webserver update with Apache in use?

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 14, 2013 Jan 14, 2013

Copy link to clipboard

Copied

hi carl

Thanks for your hint - it did not help. Cause CF10 update page says to reapply the wsconfig, it tried that as well, to no avail. I switched back to CF9 ..

Strange is another thing: I have a code for a cfdiv with works without any problem in CF9 but stalls CF10 ... i.e. the ajax call then never returns until I abort in the browser and then I see bad gateway error ...

This call also stalls my development server (as I said the other day, that server does not see the 100% cpu load).

If I disable the call CF10 runs the page.

This simple call stalls:

<cfdiv id="scal"

          style="#calpos#"

          bind="url:/ajutils.cfm?op=getShowCalendar&trid=#url.trid#&year=2000&month=#month(jetzt)#"

          bindonload=true

          />

When I switched back to CF9, this code again worked flawlessly.

BTW: cf10\coldfusion-error.log (of a restarted service) lists this first ... something to be worried?

---

Jan 15, 2013 4:45:10 AM org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal performance

in production environments was not found on the java.library.path:

x:\\ColdFusion10\\cfusion\lib;x:\\ColdFusion10\\cfusion\jintegra\bin;x:\\ColdFusion10\\cfus

ion\jintegra\bin\international;x:\\ColdFusion10\\cfusion\lib\oosdk\classes\win

---

My problems could relate to some mail problems ... we have a scheduled mail poll process ...

---

"Error","scheduler-1","01/04/13","01:10:57",,"javax.mail.MessagingException: Exception reading response;   nested exception is:      java.net.SocketTimeoutException: Read timed out"

"Error","scheduler-1","01/04/13","01:24:59",,"javax.mail.MessagingException: Exception reading response;   nested exception is:      java.net.SocketTimeoutException: Read timed out"

---

This could cause the high CPU load ... I just observe the machine right now at 4:50am .. so no load but the CPU load goes up .. cause the mail poller of course does run.

Martin

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 21, 2013 Jan 21, 2013

Copy link to clipboard

Copied

LATEST

Re -  cf10\coldfusion-error.log (of a restarted service) lists this first ... something to be worried?

Apache Tomcat Native Library - What is it? Some paste from documents say-
The Apache Tomcat Native Library provides portable API for features not found in contemporary JDK's. It uses Apache Portable Runtime as operating system abstraction layer and OpenSSL for SSL networking and allows optimal performance in production environments.

Download tomcat-native-1.1.23-win32-bin

note \bin contains tcnative-1.dll ie 32 bit and bin\x64 has the 64 bit tcnative-1.dll

Copy correct bit dll to CF10\cfusion\lib

Restart CF10 to apply

CF10 coldfusion-error.log will now report:

org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-apr-8012"]
org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
etc

You may well not be using "Apache Tomcat Native Library" however if you like to be rid of the INFO optimal performance message ā€“ thatā€™s how.

Once more, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 16, 2013 Jan 16, 2013

Copy link to clipboard

Copied

Hi Martin,

we just migrated last week and ran into a similar issue, having to restart CF every couple of hours.  We thought we had a memory leak

but.....  it turns out that we were running server monitor and it was sucking all the resources.  Go into cf admin >> server monitor>>monitor settings  and if they are enabled, uncheck them.

We did this and it solved our problems.

hope this helps.

JB

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 16, 2013 Jan 16, 2013

Copy link to clipboard

Copied

Hi jn

did not help ... cause the standard server does not have server monitoring enabled.

HOWEVER ... the latest CF 10 update 7 probably solved my issues. I noted that all scheduled tasks are gone. maybe there was something buried somewhere that caused my pain ... after installing the scheduled task again ...  the tasks has been running now for more than one start ... (it's an indefinite job, running all 5 minutes) and I have not seen any CPU 100% flatline again so far .... well, let's hope.

Thanks

Tinu

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

Unfortunately, CF10 Updater 7 is no remedy.

When I got into office, I watch and see the CPU again hitting the ceiling ... :-((((

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

BTW:

in the logfile SERVER.LOG I find

Jan 17, 2013      5:59 AM      Error      ajp-bio-8012-exec-90

It could be that the time is the point where the CPU again ran havoc.

What it this ajp-bio-8012 thingy?

Martin

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

AJP is the protocol "CF10 / java / tomcat" uses to the mod_jk connector which inturn talks to your apache webserver.
If it was IIS I would suggest check CF10\config\wsconfig\N\isapi_redirect.log however I have not done CF10 with apache so hard for me to say. Is there a log that goes with mod_jk (it is late for me so tomorrow perhaps I will search detail)? I still think CF10\cfusion\logs\coldfusion-error.log warning or error messages would be interesting.


Your server.xml AJP section would be interesting to know eg:
\CF10\cfusion\runtime\conf\server.xml
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8012" protocol="AJP/1.3"
redirectPort="8445"
tomcatAuthentication="false"
maxThreads= etc
/>

Also Martin you can monitor AJP with JMX settings, JDK tools Jconsole and Jvisualvm. If your are familiar with JMX and Jconsole
use that to monitor MBeans, Catalina (aka tomcat) threadpool AJP particularly currentthreadcount currenthreadbusy. Get back to me if you are interested in more details on any of that.

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

hi carl

thanks for the hints ...

I checked the mod_jk.log and found such lines

---

[Thu Jan 17 15:02:10 2013] [4908:1128] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): Failed opening socket to (127.0.0.1:8012) (errno=61)

[Thu Jan 17 15:02:10 2013] [4908:1128] [error] ajp_send_request::jk_ajp_common.c (1649): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)

[Thu Jan 17 15:02:10 2013] [4908:1128] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Thu Jan 17 15:02:11 2013] [4908:1128] [info] jk_open_socket::jk_connect.c (626): connect to 127.0.0.1:8012 failed (errno=61)

[Thu Jan 17 15:02:11 2013] [4908:1128] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1027): Failed opening socket to (127.0.0.1:8012) (errno=61)

[Thu Jan 17 15:02:11 2013] [4908:1128] [error] ajp_send_request::jk_ajp_common.c (1649): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)

[Thu Jan 17 15:02:11 2013] [4908:1128] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)

[Thu Jan 17 15:02:11 2013] [4908:1128] [error] ajp_service::jk_ajp_common.c (2648): (cfusion) connecting to tomcat failed.

[Thu Jan 17 15:02:11 2013] [4908:1128] [info] jk_handler::mod_jk.c (2725): Service error=-3 for worker=cfusion

---

and this is my server.xml (all comments removed):

---

<?xml version='1.0' encoding='utf-8'?>

<Server port="8007" shutdown="SHUTDOWN">

   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

   <Listener className="org.apache.catalina.core.JasperListener" />

  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />

  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <GlobalNamingResources>

    <Resource name="UserDatabase" auth="Container"

              type="org.apache.catalina.UserDatabase"

              description="User database that can be updated and saved"

              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"

              pathname="conf/tomcat-users.xml" />

  </GlobalNamingResources>

  <Service name="Catalina">

    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"

        maxThreads="150" minSpareThreads="4"/>

    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" />

    <Engine name="Catalina" defaultHost="localhost" jvmRoute="cfusion">

      <Realm className="org.apache.catalina.realm.LockOutRealm">

        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"

               resourceName="UserDatabase"/>

      </Realm>

      <Host name="localhost"  appBase="webapps"

            unpackWARs="true" autoDeploy="false">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

               prefix="localhost_access_log." suffix=".txt"

               pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/>

      </Host>

    </Engine>

  </Service>

</Server>

----

What drives me crazy is that i never seem to catch the start of process when the CPU begins to climb. I can stare hours with the remote desktop to see the tast manager almost idling (as it should), then I go away and close teh rd session ... later, for curiosity, i get back and see the CPU agin on the ceiling ...

if Windows Perfmon would allow me to add the cf10 counters, i hope i could corelate the time of the cpu ƶoad climb with a request to my vhosts ... i NEED to find that call that might start all this after a cf10 restart ...

Martin

BTW & FWIW:

I have CF9 Enterprise (JRUN stuff) on the same server, of course all services disabled. by this i mean i did not uninstall it before installing cf10 the first time.  could this be a problem?

As i wrote that, the cpu again begins to climb. I then did a cfstat -s because PerfMon does not work. cfstat says this

---

Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

0   0   0   0   -1  -1  0     0     0     0      0      2889   0      0

---

and while i atch the cpu climb, i use cfstat over and over and see the AvgDB count increasing constantly ...

---

Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

0   0   0   0   -1  -1  0     0     0     0      0      2889   0      0

0   0   0   0   -1  -1  0     0     0     0      0      2897   0      0

0   0   4   4   -1  -1  0     0     0     0      0      3731   0      0

0   0   0   0   -1  -1  0     0     0     0      0      4255   0      0

0   0   0   0   -1  -1  0     0     0     0      0      4286   0      0

0   0   0   0   -1  -1  0     0     0     0      0      4286   0      0

---

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

I think the AJP + mod_jk needs tuning for production environment. Some of the default tomcat AJP + mod_jk settings may not meet your load.
While I have not done any (until today) CF10 + Windows + Apache I have become familiar with CF10 + Windows + IIS which uses an isapi connector. Tuning the AJP + isapi connector is important for performance and stability. Likely AJP + mod_jk similarly needs tuning.


What values to tune for AJP and connector?

Refer to some paste from tomcat doc:
maxThreads
The maximum number of request processing threads to be created by
this Connector, which therefore determines the maximum number of
simultaneous requests that can be handled. If not specified, this
attribute is set to 200.
minSpareThreads
The number of request processing threads that will be created when
this Connector is first started. The connector will also make sure it
has the specified number of idle processing threads available. This
attribute should be set to a value smaller than that set for
maxThreads. The default value is 10.
ConnectionTimeout
The number of milliseconds this Connector will wait, after accepting a
connection, for the request URI line to be presented. The default
value for AJP protocol connectors is -1 (i.e. infinite).

How to apply those settings?

Edit (take a copy of first so you got a backup) CF10\cfusion\runtime\conf\server.xml AJP part -
Add values for maxthreads, minspare, timeout 

Edit (backup copy first) CF10\config\wsconfig\N\workers.properties -
Add values for connection_pool_size, connection_pool_timeout, connection_pool_minsize .

Restart Apache and CF10 service to apply changes.


EG

Server.xml part:

<Connector port="8012"
protocol="AJP/1.3"
redirectPort="8445"
tomcatAuthentication="false"
maxThreads="n00"
minSpareThreads="n00/2"
connectionTimeout="n0000"
/>

workers.properties:

worker.list=cfusion


worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.max_reuse_connections=250
worker.cfusion.connection_pool_size = n00
worker.cfusion.connection_pool_timeout = n0
worker.cfusion.connection_pool_minsize=n00/3

But what values?

It is hard for me to say since I have not seen problem server or have any monitoring details (eg JXM + Jconsole).
Note server.xml AJP timeout part is in milliseconds where as workers.properties timeout value is in seconds.
I do not want to say one set of values is a cure all  however as an initial change I would double the default values for maximum, more than double minimum (like Java JVM matters sometimes the minimum settings are the most important) and add timeout. EG -

Server.xml AJP part:

<Connector port="8012"
protocol="AJP/1.3"
redirectPort="8445"
tomcatAuthentication="false"
maxThreads="400"
minSpareThreads="80"
connectionTimeout="60000"
/>

workers.properties:

worker.list=cfusion


worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.max_reuse_connections=500
worker.cfusion.connection_pool_size = 400
worker.cfusion.connection_pool_timeout = 60
worker.cfusion.connection_pool_minsize=80

Well I typed all that before your update with CFSTAT and lack of PerfMon. I will try comment on that soon.

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

hi carl

thanks again for caring

fortunately, our services are currently not that required ... but still some do. so, we have no load that could persistently climb over 30% cpu .... recall that we've been using CF 9 on the same physical assets for years. ... it was jsut an upgrade to cf10 ...

I made some scripts to start/stop cf9/cf10 .... when in cf9, the server behaves as before ... no strnage load, no nothing. fireing um cf10 ... and sooner or later cpu load rises.

so, settings for the jvm should not be ANY problem right now because of the low request hits on the entire machine ....

Martin

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 17, 2013 Jan 17, 2013

Copy link to clipboard

Copied

Hi Martin,

I do not think the stopped CF9 would effect things other than a possible issue with the Webserver (apache) having a Jrun connector on it as well as a tomcat connector. Seems to me since you are managing to change CF10 to CF9 versa visa you have your apache change to tomcat or jrun connector versa visa sorted.

Could the CPU load climb after a period due to tomcat connector default timeout settings are infinite? Guessing over time all threads are used none handed back to the queue. Harking on here about AJP (server.xml) and mod_jk (workers.properties) timeout settings.

I find CF10 CFSTAT odd say in comparison with CFSTAT on CF9 and earlier. There is probably more I can say on that but will not just now.
CF10admin > Debugging & Logging > Debug Output Settings > What is your value for "Connector Port"?

Cheers, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 18, 2013 Jan 18, 2013

Copy link to clipboard

Copied

hi carl

i'm closing in on the problem ... hoever, strange

having nothing except cfstat, I did notice the AvgDB Time which was unclear to me why this simply goes up after any request I do. After having run CF10 for a long time (2.5 hours), I see such values

---

Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

0   0   16  16  -1  -1  0     0     0     0      0      145    0      0

0   0   0   0   -1  -1  0     0     1     0      0      218255 0      0

0   0   0   0   -1  -1  0     0     1     0      0      240414 0      0

---

the first value is the one after a cf service restart. the 2nd is after 2.5 hours and the 3rd is immediately after the 2nd.

So, the problem seems to be found in the database connection. As said, everything remained the same except CF. So are jConn to connect to a Sybase db. I also used PerfMon to observe the Sybase engine to no avail, it does not have locks not anyhing else the could cause the problems.

Somehow CF changed the way it interacts with jConn and/or there are incompatibilities that arise now with CF10 which were not there with CF9.

I will now update all database related stuff to see if anything fixes my problems on the way ...

First, I will see if there is an option to stop long running database queries in CF itself. I hardly recall that there is something ... only covering the db queries ...

Martin

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 18, 2013 Jan 18, 2013

Copy link to clipboard

Copied

Some closing thoughts from me. Following DB updates if you want to touch on this again just touch the thread.

Tomcat connector logs [info] messages are normal tho [error] and [warning] are matters of concern. The mod_jk log is showing some [error] so stop these from happening and system should work without failing.

For CFSTAT what is your CF10admin > Debugging & Logging > Debug Output Settings > "Connector Port" value?

Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jan 20, 2013 Jan 20, 2013

Copy link to clipboard

Copied

hi carl

the port is 8500.

fwiw, I had to schedule CF 10 restart every 15 minutes to have it running at least a bit more stable. our main customer falready faced 503 ...

it seems as if worker thread are not terminating sometimes. because shutting down CF takes quite a long time when the cpu is up ...

anyway ...

thanks for your ideas

Martin

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 20, 2013 Jan 20, 2013

Copy link to clipboard

Copied

Regarding CFSTAT. Your server.xml and workers.properties define connector port or AJP protocol on port 8012.

Debugging & Logging > Debug Output Settings > Connector Port should also be that value 8012 (not 8500) to provide counters for the other statistics. CF may need a restart to apply.

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 20, 2013 Jan 20, 2013

Copy link to clipboard

Copied

Restart CF10 every 15 minutes thatā€™s got to be awful for all concerned.

I think I have made a fairly good suggestion to backup and apply a change for tomcat parameters. I figure you got everything to gain and nothing lost by trying a change and given a backup copy you can always copy back to original configuration.

Certainly applying a change without having some benchmark logs or monitoring to refer, before and after, can leave you unaware as to outcome. Some may offer changing many parameters at once cumbersome and I agree but I am not remotely connected to the server to be hands on and have diagnosis to refer upon. If however a change is applied mod_jk.log stops reporting [error] and CF10 does not need restart  you have a good idea that changes have been beneficial.

Again, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation