A blog by a system administrator and programmer.

java OutOfMemoryError Cont. 
Thursday, November 17, 2011, 09:00 PM - System
Posted by Freddy Chu
If you still face the problem of

java.lang.OutOfMemoryError: PermGen space

Event you have increase the -XX:MaxPermSize.

You can try the following jvm args.

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

Usually only java web container like tomcat / jboss / jetty with many contexts will need that flag.

But remember enabling this will decrease the performance. Use with care.

P.S. java 1.6 seems do not support CMSPermGenSweepingEnabled.
3 comments ( 350 views )   |  permalink   |   ( 2.9 / 731 )
Java Out of Memory problems 
Sunday, September 18, 2011, 02:31 PM - Programming
Posted by Freddy Chu

Java heap

java.lang.OutOfMemoryError: Java heap

Just simply apply -Xmx will fix the issue. But notice that one thing there have max. value for difference os.
32bit Windows around 2G
32bit Linux around 2.5G
64bit I only tried to use 4G, the limit seems much higher than 32bit systems.

PermGen space

java.lang.OutOfMemoryError: PermGen space

Presenting the Permanent Generation

By my understanding, PermGen space is for loading classes specifications. Usually you will not able to see this exception. Expecte that you have many lib need to be load and use.
Using Jboss with more than 1 big web application may hit this. You can change this limit by using -XX:PermSize and -XX:MaxPermSize

GC overhead limit exceeded

I think this is most uncommon exception that will be hitted
java.lang.OutOfMemoryError: GC overhead limit exceeded

Excessive GC Time and OutOfMemoryError
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.


The best way is to solve the coding problem. It is believed that the code have genereated too many trivial object. Especially in loops, if that is the case rather reuse the object than new an object.

If you really don't want to change the code or you cannot found the problem you can try the following way. Hopefully it will solve the issue, but not the best way and may require a long time to finish.

JVM have 3 difference garbage collectors you can try switching between them.

serial collector
single processor
jvm flag: -XX:+UseSerialGC

parallel collector
high throughput sometimes with pause
jvm flag: -XX:+UseParallelGC

concurrent collector
low lag time & moderate throughput
jvm flag: -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

3019 comments ( 21993 views )   |  permalink   |   ( 2.9 / 500 )
my Apache production checklist 
Sunday, February 20, 2011, 04:23 PM - System
Posted by Freddy Chu

Apache (httpd)

Lower KeepAliveTimeout
Default usually around 15 but I will choose from 5-10 but remember do not set the value too low as it will cause tcp overhead

Reduce extra dns lookup for log
HostnameLookups off

Disable directory listing
Remove "Indexes" from Options

Disable .htaccess files
Reduce file IO to search and access permission files, put all access control into your apache configuration files
just simply comment out all AccessFileName lines

Make sure Apache is not run by root
remember to check the "User" and "Group" in configure file

Hide system information
ServerTokens Prod
ServerSignature off

If you really want to hide the name of Apache, you will need to modify the source code and compile for yourself. I believe it is not a necessary step as there still have many ways to discover your web server easily.

Disable weak cipher
SSLProtocol -ALL +SSLv3 +TLSv1

Limit the use of mod_status
If you really need that better change the Location to non default links and make sure it is protected by source host or any authentication

Turn on FollowSymLinks but disable SymLinksIfOwnerMatch
which reduce disk IO to check the file type but make sure that nobody put links in your server that point to your private files
Options FollowSymLinks

remove "SymLinksIfOwnerMatch" from Options

Enable compress module (if application do not implement compression.)
<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

Disable all useless Apache modules
in some OS the default config files include many useless moduels

install some protection modules (optional)


Disable access time update
Set noatime to your web document root if your OS support
/dev/md0 /var/www ext3 defaults,noatime 0 0

Tune network options of the OS
net.core.netdev_max_backlog = 3000
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Here is just some examples by my experience.

Check disk usage
Make sure there is enough space for log file and don't forget to check the log rotation config.


Change the session name
session.name = SESSION
It is my habit that do not use default session name

hiding php version information X-Powered-By
expose_php = Off

Deploy php accelerator
List of accelerators
Alternative PHP Cache
ionCube PHP Accelerator
Zend Accelerator
Windows Cache Extension for PHP
3499 comments ( 28250 views )   |  permalink   |   ( 3 / 7717 )
file_get_contents getting mad? 
Tuesday, April 20, 2010, 08:20 PM - Programming
Posted by Freddy Chu
My blog is died for awhile after php upgrade to 5.2.13. It is full of error about putting a non-array into rsort.

The root cause of the problem is due to the this blog store entries in file base, some of the file listing is serialized array. If you copy those content and try to unserialize it, it is perfect no error.

It really make me shocked as i expected the problem is come from serialization algorithm but it is not. It really cost me some time to figure out that the problem is from file_get_contents. It is really tricky, it won't show on command line mode but only happens in php modules of apache.

The REAL reason of unserializable is because file_get_contents add slashes into the output string. I have really no idea why i happens...

So I use the most DIRTY way to fix that ... add a stripslashes after call file_get_contents.

If anybody know the reasons please let me know :(

4422 comments ( 97358 views )   |  permalink   |   ( 3 / 9200 )
undefined symbol: dav_register_provider 
Monday, April 12, 2010, 05:24 PM - System
Posted by Freddy Chu
It has been a long time that i haven't update my apache...

It is very easy to do with Gentoo but this time it give me an error. :(

/usr/lib/apache2/modules/mod_dav_svn.so: undefined symbol: dav_register_provider

After googled awhile ... i found that is related to dav of apache, at that time i really don't know why there exist such problem. As my svn server have been up for at least 5 years. It never get such problem.

Finally it works after I load the dav before svn module.

LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule dav_lock_module modules/mod_dav_lock.so
DavLockDB "/var/lib/dav/lockdb"

if you are using the same OS as me and you have compiled apache with dav options. Just add "-DDAV" to APACHE2_OPTS in /etc/conf.d/apache2

2804 comments ( 24383 views )   |  permalink   |   ( 3 / 1731 )
Too many CLOSE_WAIT 
Thursday, May 21, 2009, 11:18 AM - System
Posted by Freddy Chu
Currently I found that jetty / tomcat on Linux will have many CLOSE_WAIT on busy system especially your network is not in good condition.

These CLOSE_WAIT will disappear untill you stop the server. These CLOSE_WAIT will use up all of you tcp connection and hang up your web server. Many people claimed that is the bug of jvm. Although I have tried most java ver., the problem still exist.

Here is another dirty way to fix that issue... although it is not the best solution........
add the following lines to /etc/sysctl.conf
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 1800

And then execute
sysctl -p

or do a reboot

3002 comments ( 18572 views )   |  permalink   |   ( 3 / 9137 )
Jetty disable weak cipher 
Thursday, April 16, 2009, 02:07 PM - System
Posted by Freddy Chu
inorder to disable weak SSL cipher in jetty you can add the xml below into SslSocketConnector

<Set name="ExcludeCipherSuites">
<Array type="java.lang.String">

9 comments ( 320 views )   |  permalink   |   ( 3 / 516 )
java.io.IOException: Too many open files 
Saturday, October 25, 2008, 12:06 AM - System, Programming
Posted by Administrator
Yesterday I have face a funny java exception on my Linux server.

java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:537)

It have cost me few minutes to figure out what is that problem.

At first i think it is caused by sysctl
but i found
fs.file-max = 65535

and my lsof -nn | wc -l is only around 10xx so i know that is not the problem.

After that i think about ulimit, if you are careless you may fake by default result the command ulimit's output. unlimited

When you execute ulimit -a you will see the whole story.

#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16370
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16370
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Now you know the point is there ... most Linux default openfile per user is limited to 1024. So you must edit the file /etc/security/limits.conf

add those 2 lines below to override the default limit.

* soft nofile 65536
* hard nofile 65536

4425 comments ( 2153 views )   |  permalink   |   ( 3 / 8787 )
Oracle moving index to another table space 
Tuesday, April 8, 2008, 05:04 PM - Programming
Posted by Freddy Chu
These days I am super busy. Nearly no time to write my blog. :(

There is so many people asking me how to move the index after created. As you know if you use plsql developer's GUI it will drop the constraint and add again with index. I don't know why it work like that but there is some simple solution.


Here also show you a simple procedure to move all index of a table from one tablespace to another tablespace.

create or replace procedure MOVE_INDEX_BETWEEN_TABLESPACE(from_ts in string,
to_ts in string,
tablename in string) is
cursor index_names is
select user_indexes.index_name
from user_indexes
where user_indexes.table_name like upper(tablename)
and user_indexes.tablespace_name = upper(from_ts);
index_name user_indexes.index_name%type;
open index_names;
fetch index_names
into index_name;
exit when index_names%notfound;
if index_name is not null and to_ts is not null then
end if;
end loop;

3 comments ( 211 views )   |  permalink   |   ( 3 / 399 )
Happy lunar new year~~ 
Sunday, February 10, 2008, 11:01 AM - General
Posted by Freddy Chu
This year I am lucky that I can have a good location to take photos. :D

For more photos please go to my gallery.

Fireworks 2008

4 comments ( 220 views )   |  permalink   |   ( 3 / 1451 )

<Back | 1 | 2 | Next> Last>>