Additional research. I've found our development CF11 install was an early version and actually installed with Java 7 (1.7) and has seen none of these "Metaspace" errors. Our production machines were newly downloaded and installed with Java 8 (1.8)...where this new Metaspace concept was introduced. This would explain why the development machine never had these issues!
My increase in the Metaspace size seems to have only reduced the number of occurrences of this error. It is still happening.
I found this article explaining Metaspace:
I have 7 or 8 custom JAR tags in addition to all everything CF installs. I find it hard to believe no one else has run into this problem.
I'm going to try increase the MaxMetaspaceSize to 1024. If that doesn't work I can try removing it altogether. We're also contemplating moving back down to Java 7.
Rule of thumb I have noticed - if you know some values for PermGen with Java 7 that work well for your environment use those as a starting point for MetaSpace. EG:
-XX:PermSize=312m -XX:MaxPermSize=512m Java 7
-XX:MetaspaceSize=312m -XX:MaxMetaspaceSize=512m Java 8
One can do well to assign a value for initial setting to MetaSpace. See above example.
Perhaps related, how much memory is being consumed by Compressed Class Space? JMX monitoring with JDK tools like JMC or Jconsole might be helpful to check plus will also show MetaSpace use along with garbage collection information.
We have also been working on CF11 in Dev with Java 1.8_45(currently) for 1 year and have moved production to CF11 three times and reverted back to CF10 Java 1.7 each time because of memory issues.
Have you made any progress with your JVM tuning on 1.8 or has success switching back to Java 1.7.