Apache Semaphore issue

Semaphore issues

This is one of the rarest conditions which I came across. The webserver failed to load with following error messages in the server log file. The server had fairly good uptime and was a busy server.

============================================================

[emerg] (28)No space left on device: Couldn't create accept lock [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [notice] Digest: generating secret for digest authentication ... [notice] Digest: done [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [emerg] (28)No space left on device: Couldn't create accept lock

 ===========================================================

However the disk space as well as quota limit were  fine.  On a detailed analysis, it is found to be  a semaphore issue.

 A semaphore is an inter-process communication tool that is used by Apache to communicate with its child processes. Semaphore marks memory locations as locked and may not release it upon completion of process. In most case, if parent process dies before the child.

Due to the marking of  too many memory locations as being used while they are not,  the system will run out of memory locations after some time. This can happen in very busy servers with high uptime. In the previous case, the error message indicates that the apache server fails to allocate semaphores for new child processes.Check to see how many semaphores are currently in use. If Apache is running correctly, you should see something like this

# ipcs -s

If Apache is stopped, and you still see these semaphores, then you can safely kill them by running this command for each semaphore id (in the second column)

#ipcrm -s <semid>

To destroy all semaphores, you can run this from the command line (with “apache” being the apache-user:

#for semid in `ipcs -s | grep apache | cut -f2 -d" "`; do ipcrm -s $semid; done

or

#ipcs -s | grep apache | perl -e ‘while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}’

#ipcs -s | grep nobody | perl -e ‘while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}’   or #ipcs -m | grep nobody | perl -e ‘while () { @a=split(/\s+/); print `ipcrm -m $a[1]`}’

OR

for i in `ipcs -s | awk ‘/httpd/ {print $2}’`; do (ipcrm -s $i); done

Increase semaphore limit

To view the current parameters:

#ipcs -l

To change these parameters, modify the file /etc/sysctl.conf and add the following lines:

kernel.msgmni = 1024   kernel.sem = 250 256000 32 1024

Then load these settings with the command:

#sysctl -p