A blog by a system administrator and programmer.

java OutOfMemoryError Cont.

If you still face the problem of

java.lang.OutOfMemoryError: PermGen space

Event you have increase the -XX:MaxPermSize.

You can try the following jvm args.

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

Usually only java web container like tomcat / jboss / jetty with many contexts will need that flag.

But remember enabling this will decrease the performance. Use with care.

P.S. java 1.6 seems do not support CMSPermGenSweepingEnabled.

Java Out of Memory problems

Java heap

java.lang.OutOfMemoryError: Java heap

Just simply apply -Xmx will fix the issue. But notice that one thing there have max. value for difference os.
e.g.
32bit Windows around 2G
32bit Linux around 2.5G
64bit I only tried to use 4G, the limit seems much higher than 32bit systems.

PermGen space

java.lang.OutOfMemoryError: PermGen space

Presenting the Permanent Generation

By my understanding, PermGen space is for loading classes specifications. Usually you will not able to see this exception. Expecte that you have many lib need to be load and use.
Using Jboss with more than 1 big web application may hit this. You can change this limit by using -XX:PermSize and -XX:MaxPermSize
e.g.

-XX:PermSize=128m
-XX:MaxPermSize=128m

GC overhead limit exceeded

I think this is most uncommon exception that will be hitted
java.lang.OutOfMemoryError: GC overhead limit exceeded

Excessive GC Time and OutOfMemoryError
The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.

source

The best way is to solve the coding problem. It is believed that the code have genereated too many trivial object. Especially in loops, if that is the case rather reuse the object than new an object.

If you really don't want to change the code or you cannot found the problem you can try the following way. Hopefully it will solve the issue, but not the best way and may require a long time to finish.

JVM have 3 difference garbage collectors you can try switching between them.

serial collector
single processor
jvm flag: -XX:+UseSerialGC

parallel collector
high throughput sometimes with pause
jvm flag: -XX:+UseParallelGC

concurrent collector
low lag time & moderate throughput
jvm flag: -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

my Apache production checklist

Apache (httpd)

Lower KeepAliveTimeout
Default usually around 15 but I will choose from 5-10 but remember do not set the value too low as it will cause tcp overhead

Reduce extra dns lookup for log
HostnameLookups off

Disable directory listing
Remove "Indexes" from Options

Disable .htaccess files
Reduce file IO to search and access permission files, put all access control into your apache configuration files
just simply comment out all AccessFileName lines

Make sure Apache is not run by root
remember to check the "User" and "Group" in configure file

Hide system information

ServerTokens Prod
ServerSignature off

If you really want to hide the name of Apache, you will need to modify the source code and compile for yourself. I believe it is not a necessary step as there still have many ways to discover your web server easily.

Disable weak cipher

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:!EXPORT

Limit the use of mod_status
If you really need that better change the Location to non default links and make sure it is protected by source host or any authentication

Turn on FollowSymLinks but disable SymLinksIfOwnerMatch
which reduce disk IO to check the file type but make sure that nobody put links in your server that point to your private files

Options FollowSymLinks

remove "SymLinksIfOwnerMatch" from Options

Enable compress module (if application do not implement compression.)

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

Disable all useless Apache modules
in some OS the default config files include many useless moduels
e.g.

ldap_module
proxy_ajp_module
proxy_balancer_module
proxy_connect_module
proxy_ftp_module
proxy_http_module
proxy_module
speling_module

install some protection modules (optional)
mod_securitye
mod_evasive

OS

Disable access time update
Set noatime to your web document root if your OS support
e.g.

/dev/md0 /var/www ext3 defaults,noatime 0 0

Tune network options of the OS

net.core.netdev_max_backlog = 3000
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

Here is just some examples by my experience.

Check disk usage
Make sure there is enough space for log file and don't forget to check the log rotation config.

PHP

Change the session name
session.name = SESSION
It is my habit that do not use default session name

hiding php version information X-Powered-By
expose_php = Off

Deploy php accelerator
List of accelerators
Alternative PHP Cache
eaccelerator
ionCube PHP Accelerator
XCache
Zend Accelerator
Windows Cache Extension for PHP

file_get_contents getting mad?

My blog is died for awhile after php upgrade to 5.2.13. It is full of error about putting a non-array into rsort.

The root cause of the problem is due to the this blog store entries in file base, some of the file listing is serialized array. If you copy those content and try to unserialize it, it is perfect no error.

It really make me shocked as i expected the problem is come from serialization algorithm but it is not. It really cost me some time to figure out that the problem is from file_get_contents. It is really tricky, it won't show on command line mode but only happens in php modules of apache.

The REAL reason of unserializable is because file_get_contents add slashes into the output string. I have really no idea why i happens...

So I use the most DIRTY way to fix that ... add a stripslashes after call file_get_contents.

If anybody know the reasons please let me know :(

undefined symbol: dav_register_provider

It has been a long time that i haven't update my apache...

It is very easy to do with Gentoo but this time it give me an error. :(

/usr/lib/apache2/modules/mod_dav_svn.so: undefined symbol: dav_register_provider

After googled awhile ... i found that is related to dav of apache, at that time i really don't know why there exist such problem. As my svn server have been up for at least 5 years. It never get such problem.

Finally it works after I load the dav before svn module.

LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule dav_lock_module modules/mod_dav_lock.so
DavLockDB "/var/lib/dav/lockdb"

if you are using the same OS as me and you have compiled apache with dav options. Just add "-DDAV" to APACHE2_OPTS in /etc/conf.d/apache2

Too many CLOSE_WAIT

Currently I found that jetty / tomcat on Linux will have many CLOSE_WAIT on busy system especially your network is not in good condition.

These CLOSE_WAIT will disappear untill you stop the server. These CLOSE_WAIT will use up all of you tcp connection and hang up your web server. Many people claimed that is the bug of jvm. Although I have tried most java ver., the problem still exist.

Here is another dirty way to fix that issue... although it is not the best solution........
add the following lines to /etc/sysctl.conf

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 1800

And then execute

sysctl -p

or do a reboot

Jetty disable weak cipher

inorder to disable weak SSL cipher in jetty you can add the xml below into SslSocketConnector


  
    SSL_RSA_WITH_DES_CBC_SHA
    SSL_DHE_RSA_WITH_DES_CBC_SHA
    SSL_DHE_DSS_WITH_DES_CBC_SHA
    SSL_RSA_EXPORT_WITH_RC4_40_MD5
    SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
    SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
    SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA

java.io.IOException: Too many open files

Yesterday I have face a funny java exception on my Linux server.

java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:537)

It have cost me few minutes to figure out what is that problem.

At first i think it is caused by sysctl
but i found

fs.file-max = 65535

and my lsof -nn | wc -l is only around 10xx so i know that is not the problem.

After that i think about ulimit, if you are careless you may fake by default result the command ulimit's output. unlimited

When you execute ulimit -a you will see the whole story.

#ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16370
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16370
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Now you know the point is there ... most Linux default openfile per user is limited to 1024. So you must edit the file /etc/security/limits.conf

add those 2 lines below to override the default limit.

* soft nofile 65536
* hard nofile 65536

Oracle moving index to another table space

These days I am super busy. Nearly no time to write my blog. :(

There is so many people asking me how to move the index after created. As you know if you use plsql developer's GUI it will drop the constraint and add again with index. I don't know why it work like that but there is some simple solution.

ALTER INDEX REBUILD TABLESPACE

Here also show you a simple procedure to move all index of a table from one tablespace to another tablespace.


create or replace procedure MOVE_INDEX_BETWEEN_TABLESPACE(from_ts   in string,
                                                          to_ts     in string,
                                                          tablename in string) is
  cursor index_names is
    select user_indexes.index_name
      from user_indexes
     where user_indexes.table_name like upper(tablename)
       and user_indexes.tablespace_name = upper(from_ts);
  index_name user_indexes.index_name%type;
begin
  open index_names;
  loop
    fetch index_names
      into index_name;
    exit when index_names%notfound;
    if index_name is not null and to_ts is not null then
      EXECUTE IMMEDIATE 'ALTER INDEX ' || index_name ||
                        ' REBUILD TABLE_SPACE ' || to_ts;
    end if;
  end loop;
end MOVE_INDEX_BETWEEN_TABLESPACE;

Happy lunar new year~~

This year I am lucky that I can have a good location to take photos. :D

For more photos please go to my gallery.

Fireworks 2008

Java disk usage

It have been a long time for me to found a way to check the disk usage in Java.

Finially i get the solutions. It is in Java 6.0. java.io.File

There is 2 functions.

getTotalSpace() --- Returns the size of the partition named by this abstract pathname.
getUsableSpace() --- Returns the number of bytes available to this virtual machine on the partition named by this abstract pathname.

If your application is limited to use versions before 6.0. Then you can only keep using external commands.
Runtime.getRuntime().exec(commands);
e.g.
Linux/Unix: df
Windows: fsutil volume diskfree c:

Alternative solutions for system independent solutions is to setup a snmp on the server you want to check and use java snmp client to query the result. That will be quite complex.

Useful links:
snmp4j
netsnmp

Happy New Year 2008

Happy New Year to All :D

compile error of subversion

This problem usually exist in solaris 5.7

libtool[4766]: : is not an identifier
make: *** [subversion/svn/svn] Segmentation Fault. (core dumped)

Many people suggested to compile it as static link, then the problem will be solved. I have tried but without success. So that i start debug the compile code, lastly i found the problem is caused by ksh.

Solution:

1. do configure as usual
2. replace all ksh to bash or sh in Makefiles and libtool
3. make

P.S. becareful of compile neon, db4, openssl, expat, apr, apr-util for solaris. It seems that gcc < 3.0 have some bugs in solaris system, you better compile them with shared flags

Compiler error in Solaris

These days I have facing an error during compile DB4 in solaris 5.7.

when I am doing configure It shows

configure: error: C compiler cannot create executables

Here is the error in config.log.

configure:3855: found /usr/ucb/cc
configure:3866: result: cc
configure:3902: checking for C compiler version
configure:3909: cc --version >&5
/usr/ucb/cc: language optional software package not installed
configure:3912: $? = 1
configure:3919: cc -v >&5
/usr/ucb/cc: language optional software package not installed
configure:3922: $? = 1
configure:3929: cc -V >&5
/usr/ucb/cc: language optional software package not installed
configure:3932: $? = 1
configure:3955: checking for C compiler default output file name
configure:3982: cc -O -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS conftest.c >&5
/usr/ucb/cc: language optional software package not installed

I have googled for sometime but seems no any solutions. >"<

But suddently I think about tricky way. Changed the compiler

so that i export CC=gcc

It works lucky.

Do anybody know another way?

syslog-ng concurrent connections exceeded

These days I get a very funny warning in the syslog.

syslog-ng[4484]: Number of allowed concurrent connections exceeded; num='10', max='10'

I have googled it for a while, it seems is the default values of syslog-ng limited the concurrent connections of any stream device to 10.

It can simply sloved by increase the limit.

in syslog-ng.conf

unix-stream("/dev/log" max-connections(20));

Simple MySQL Replication HOWTO

Requirements:

* Slave server should be newer version than the master server. It is adviced that both servers are in same version
* Network port 3306 of both server should be avaliable
* log-bin is enabled

Procedure

1. edit my.cnf to add log-bin and server-id, server-id should be unique and from 1 to 2^32 - 1.

[mysqld]
server-id = 1
log-bin

2. restart the mysql server both server if you have do many modification in step 1

3. create replicate user, if you are using MySQL before 4.0.2 replace "REPLICATION SLAVE" by "FILE"

GRANT REPLICATION SLAVE ON *.* TO 'Username'@'Host' IDENTIFIED BY 'password';

4. lock all tables on master server

mysql> FLUSH TABLES WITH READ LOCK;

5. Copy all mysql files to the slave server

6. At master server run show master status, remember the file and position info

mysql> SHOW MASTER STATUS;
+--------------+----------+--------------+------------------+
| File | Position | Binlog_do_db | Binlog_ignore_db |
+--------------+----------+--------------+------------------+
| log-bin.002 | 189 | | |
+--------------+----------+--------------+------------------+

7. At slave server run change master to use the account create in step 3 and info in step 6

CHANGE MASTER TO
MASTER_HOST='master_server_ip',
MASTER_USER='slave_user_name',
MASTER_PASSWORD='slave_password',
MASTER_LOG_FILE='file_in_step_7',MASTER_LOG_POS=position_in_step_7;

8. run Slave start

9. unlock tables on master server

mysql>UNLOCK TABLES;

Now you will have a replicate MySQL server, just test to modify any records. :D

Mysql problems

Get Error 1033 "Incorrect information in file xxxx.frm" when run REPAIR

* Found a backup frm file if you can and overwrite the existing one
* If you do not have frm backup, you may need to find the original "create table" statement

1. Backup those table's frm, MYI, MYD first
2. use the original create table statement to create a new table with another name
3. overwrite the table's frm with the new table's frm
4. run REPAIR TABLE again

mysqldump: Got error: 1105: not found (Errcode: 24) when using LOCK TABLES

* this happens usually caused by open_files_limit too low
* edit my.cnf at [mysqld]" part

[mysqld]
open_files_limit = 8192

* restart mysql server

About my blog

It has been a long time that i want to have my own blog, but

I am really lazy....

I spent most of my in World of Warcraft in the past. I think i have spent almost around 2 years in the game. It is a good timing to leave the virtual world and put more time on work and study. :D

This blog is mainly about Unix servers and programming problems.

I will keep update it when i solve any interesting problems.