NGINX Semaphores Arrays

General Discussion of atomic repo and development projects.

Ask for help here with anything else not covered by other forums.
FrancoisD
Forum User
Forum User
Posts: 5
Joined: Wed Mar 18, 2015 3:20 pm
Location: QC, CANADA

NGINX Semaphores Arrays

Unread post by FrancoisD »

i found a weird bug, i have no clues if its documented or not at Atomic.

we are using NGINX from ATOMIC for laodbalance purposes. and we came up with some problem that we needed to reboot from times to times. until i found what was going on and apply a TEMP FIX until Atomic fixes this but it never came out. so my collegues are pushing me to post here.

we are having this problem on all our NGINX enabled server, using the ATOMIC repo for NGINX. which is arround 45 servers total in our infrastructure. all Linux, centos 6.6 64 bit.

we are using;
#nginx -V nginx version: nginx/1.6.2
nginx.x86_64 1.6.2-23.el6.art @atomic
onto a centos 6.6
Linux ESL-DR2-jb-01 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

i will not put every details of what i did and how i reached that conclusion but, long story short, nginx produces entry in the Semaphore Arrays, that upon restart or load, does not clear.

to be even more exact, when you service start, it creates 2 semaphore. when you service stop, it removes the sempahore, which is perfect BUT the problem is;
if you start nginx, creates 2 semaphores.
if you service configtest, it creates 2 new semaphore that are not getting deleted,
then do a service reload, it creates 4 new sempahore
then if you service stop, it only removes the last 2 semaphores.

even, if you do no service command, just the log rotation will generates 2 new ipcs.

to illustrate this;

Code: Select all

225/1877MB      0.00 0.00 0.00 1/275 7486 UNKOWN ZONE
[7214:7213] [0:1395] 04:42:20 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1395)# banner start

 #####   #######     #     ######   #######
#     #     #       # #    #     #     #
#           #      #   #   #     #     #
 #####      #     #     #  ######      #
      #     #     #######  #   #       #
#     #     #     #     #  #    #      #
 #####      #     #     #  #     #     #


225/1877MB      0.00 0.00 0.00 1/275 7486 UNKOWN ZONE
[7214:7213] [0:1396] 04:42:27 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1396)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


225/1877MB      0.00 0.00 0.00 1/275 7486 UNKOWN ZONE
[7214:7213] [0:1397] 04:42:31 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1397)# service nginx start
Starting nginx:                                            [  OK  ]

327/1877MB      0.04 0.01 0.00 1/277 7838 UNKOWN ZONE
[7214:7213] [0:1398] 04:42:36 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1398)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 6651906    4294967295 600        1
0x00000000 6684675    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


327/1877MB      0.04 0.01 0.00 1/277 7838 UNKOWN ZONE
[7214:7213] [0:1399] 04:42:38 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1399)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

327/1877MB      0.04 0.01 0.00 1/277 7838 UNKOWN ZONE
[7214:7213] [0:1400] 04:42:49 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1400)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 6651906    4294967295 600        1
0x00000000 6684675    4294967295 600        1
0x00000000 6717444    4294967295 600        1
0x00000000 6750213    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


327/1877MB      0.04 0.01 0.00 1/277 7838 UNKOWN ZONE
[7214:7213] [0:1401] 04:42:54 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1401)# service nginx reload
Reloading nginx:                                           [  OK  ]

327/1877MB      0.04 0.01 0.00 1/277 7838 UNKOWN ZONE
[7214:7213] [0:1402] 04:43:02 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1402)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 6651906    4294967295 600        1
0x00000000 6684675    4294967295 600        1
0x00000000 6717444    4294967295 600        1
0x00000000 6750213    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1
0x00000000 6782984    4294967295 600        1
0x00000000 6815753    4294967295 600        1
0x00000000 6848522    4294967295 600        1
0x00000000 6881291    4294967295 600        1


324/1877MB      0.02 0.01 0.00 1/277 7900 UNKOWN ZONE
[7214:7213] [0:1403] 04:43:06 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1403)# service nginx stop
Stopping nginx:                                            [  OK  ]

324/1877MB      0.02 0.01 0.00 1/277 7900 UNKOWN ZONE
[7214:7213] [0:1404] 04:43:22 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1404)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 6651906    4294967295 600        1
0x00000000 6684675    4294967295 600        1
0x00000000 6717444    4294967295 600        1
0x00000000 6750213    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1
0x00000000 6782984    4294967295 600        1
0x00000000 6815753    4294967295 600        1


324/1877MB      0.02 0.01 0.00 1/277 7900 UNKOWN ZONE
[7214:7213] [0:1405] 04:43:24 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1405)# ps -ef |grep nginx
root      7920  7214  0 16:43 pts/1    00:00:00 /bin/grep --color=always nginx

324/1877MB      0.02 0.01 0.00 1/277 7900 UNKOWN ZONE
[7214:7213] [0:1406] 04:43:35 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1406)# banner stop

 #####   #######  #######  ######
#     #     #     #     #  #     #
#           #     #     #  #     #
 #####      #     #     #  ######
      #     #     #     #  #
#     #     #     #     #  #
 #####      #     #######  #


329/1877MB      0.01 0.01 0.00 1/275 7927 UNKOWN ZONE
[7214:7213] [0:1407] 04:43:46 Wed Mar 18 root@l-2001a  +1 /home/p7685
SUDO MODE ON
(1:1407)#
as we start to see those semaphore groing, i was wondering what could do this, then logrotate got me thinking that it generates 2 entry each day. due to the checkconfig embeded in the reload command, but reload itself is doing a kill hup on the process id.

Code: Select all

reload() {
    configtest_q || return 6
    echo -n $"Reloading $prog: "
    killproc -p $pidfile $prog -HUP
    echo
}

so, to test it, start nginx as you would normaly, then just type a couple of kill -HUP {with the master process id}. for the sake of the long message, i will just paste 1 kill command but in fact, i executed it 10 times.

Code: Select all

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1456] 04:54:27 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1456)# banner start

 #####   #######     #     ######   #######
#     #     #       # #    #     #     #
#           #      #   #   #     #     #
 #####      #     #     #  ######      #
      #     #     #######  #   #       #
#     #     #     #     #  #    #      #
 #####      #     #     #  #     #     #


326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1457] 04:54:30 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1457)# service nginx start
Starting nginx:                                            [  OK  ]

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1458] 04:54:38 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1458)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 7766018    4294967295 600        1
0x00000000 7798787    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1459] 04:54:43 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1459)# ps -ef |grep nginx
root      8457     1  0 16:54 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx     8458  8457  0 16:54 ?        00:00:00 nginx: worker process
root      8464  7214  0 16:54 pts/1    00:00:00 /bin/grep --color=always nginx

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1460] 04:54:50 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1460)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1461] 04:54:57 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1461)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1462] 04:54:57 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1462)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1463] 04:54:58 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1463)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1464] 04:54:58 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1464)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1465] 04:54:59 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1465)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1466] 04:54:59 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1466)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1467] 04:55:00 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1467)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1468] 04:55:00 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1468)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1469] 04:55:00 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1469)# kill -HUP 8457

326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1470] 04:55:01 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1470)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 7766018    4294967295 600        1
0x00000000 7798787    4294967295 600        1
0x00000000 7831556    4294967295 600        1
0x00000000 7864325    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1
0x00000000 7897096    4294967295 600        1
0x00000000 7929865    4294967295 600        1
0x00000000 7962634    4294967295 600        1
0x00000000 7995403    4294967295 600        1
0x00000000 8028172    4294967295 600        1
0x00000000 8060941    4294967295 600        1
0x00000000 8093710    4294967295 600        1
0x00000000 8126479    4294967295 600        1
0x00000000 8159248    4294967295 600        1
0x00000000 8192017    4294967295 600        1
0x00000000 8224786    4294967295 600        1
0x00000000 8257555    4294967295 600        1
0x00000000 8290324    4294967295 600        1
0x00000000 8323093    4294967295 600        1
0x00000000 8355862    4294967295 600        1
0x00000000 8388631    4294967295 600        1
0x00000000 8421400    4294967295 600        1
0x00000000 8454169    4294967295 600        1


326/1877MB      0.00 0.00 0.00 1/275 8247 UNKOWN ZONE
[7214:7213] [0:1471] 04:55:04 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1471)#
now lest demonstrate, that even stoped, nginx can generate semaphore

Code: Select all

330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1476] 04:57:46 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1476)# banner start

 #####   #######     #     ######   #######
#     #     #       # #    #     #     #
#           #      #   #   #     #     #
 #####      #     #     #  ######      #
      #     #     #######  #   #       #
#     #     #     #     #  #    #      #
 #####      #     #     #  #     #     #


330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1477] 04:57:50 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1477)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1478] 04:57:52 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1478)# ps -ef |grep nginx
root      8654  7214  0 16:58 pts/1    00:00:00 /bin/grep --color=always nginx

330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1479] 04:58:25 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1479)# service nginx status
nginx is stopped

330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1480] 04:58:29 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1480)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1481] 04:58:31 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1481)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1482] 04:58:40 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1482)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 8486914    4294967295 600        1
0x00000000 8519683    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


330/1877MB      0.00 0.00 0.00 1/273 8623 UNKOWN ZONE
[7214:7213] [0:1483] 04:58:43 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1483)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 8486914    4294967295 600        1
0x00000000 8519683    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1


330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1484] 04:58:46 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1484)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1485] 04:58:54 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1485)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1486] 04:58:54 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1486)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1487] 04:58:55 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1487)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1488] 04:58:55 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1488)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8690 UNKOWN ZONE
[7214:7213] [0:1489] 04:58:55 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1489)# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

330/1877MB      0.00 0.00 0.00 1/273 8763 UNKOWN ZONE
[7214:7213] [0:1490] 04:58:56 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1490)# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 8486914    4294967295 600        1
0x00000000 8519683    4294967295 600        1
0x00000000 8552452    4294967295 600        1
0x00000000 8585221    4294967295 600        1
0x00000000 2326534    p7685      600        1
0x00000000 2359303    p7685      600        1
0x00000000 8617992    4294967295 600        1
0x00000000 8650761    4294967295 600        1
0x00000000 8683530    4294967295 600        1
0x00000000 8716299    4294967295 600        1
0x00000000 8749068    4294967295 600        1
0x00000000 8781837    4294967295 600        1
0x00000000 8814606    4294967295 600        1
0x00000000 8847375    4294967295 600        1
0x00000000 8880144    4294967295 600        1
0x00000000 8912913    4294967295 600        1


330/1877MB      0.00 0.00 0.00 1/273 8763 UNKOWN ZONE
[7214:7213] [0:1491] 04:58:58 Wed Mar 18 root@l-2001a  +1 /etc/logrotate.d
SUDO MODE ON
(1:1491)#
as you can see, those semaphore are generated. the actual list is maximum if 128 IPCS queues. if you reach that, then any other process that required a semaphore will not start. and you will get this message, in my case, i ran config test 64 time, to get 128 semaphores, then tried to start apache httpd

Code: Select all

[Wed Mar 18 17:03:47 2015] [error] (28)No space left on device: Cannot create SSLMutex
those semaphore that nginx generates, are coded to be unkowned. a good thing would be that Atomic code the semaphores owned by the user nginx.

but i have another question, why the release of nginx from ATOMIC is generating all those semaphores, while EPEL and the original release from NGINX repo are not generating any of these, this leads me to think that some code changes are made in NGINX that uses semaphore while the other REPO are not.

im sory for the long email, but i worked like my butt on this in the office to find out these! and now i am sharing with the WORLD!

my own personl fix, is that i changed the /etc/init.d/nginx script, i added this command

Code: Select all

for i in `ipcs -s | awk '/4294967295/ {print $2}'`; do (ipcrm -s $i); done
after each function that the script is calling, so i am clearing those semaphore as the apears. and this leads me to another fact, that NGINX does not even uses them, as the application do not fail, crash or whatsoever... well i think that nxing is not using those semaphores!

on this, have a good one, if by mistake, ive post this in the bad forom, please push it to the proper place.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: NGINX Semaphores Arrays

Unread post by scott »

Semaphore whackamole here :P The deviations we have off of epel and others are we have 3 modules added, and any one of them could be the culprit. they are:

* ngx_cache_purge
* ngx_pagespeed
* mod_security

The only one we really couldnt live without is mod_security, however I know for a fact that its still really really beta on nginx. The test case would be to remove those individually from the package and have you run a build without any of them to isolate if it is indeed nginx, and if it is not re-test with each module disabled to see if we can bracket which one is the culprit.
FrancoisD
Forum User
Forum User
Posts: 5
Joined: Wed Mar 18, 2015 3:20 pm
Location: QC, CANADA

Re: NGINX Semaphores Arrays

Unread post by FrancoisD »

that looks a great idea! i am in.

does that means, you will build packages and i test them ? or i have to dowload the source and remove those packages and compile them ? i could do it, but ill require some technical knowledge that i probably dont have. i know that "make" is building things, thats about it! i'm more of a sysadmin, less of a developper.

i'm comming from a HP-UX background.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: NGINX Semaphores Arrays

Unread post by scott »

Ah HP-UX... very familiar with that as well :P Back in the day you'd see the HP-UX "Exploit of the week", as I recall the guy that ran that site eventually got paid by HP to stop running it :P We used to call it PHUX.

I could do the test builds for you, thats where I was going with this. What dist/arch are you on?
FrancoisD
Forum User
Forum User
Posts: 5
Joined: Wed Mar 18, 2015 3:20 pm
Location: QC, CANADA

Re: NGINX Semaphores Arrays

Unread post by FrancoisD »

we are on

we are using;
#nginx -V nginx version: nginx/1.6.2
nginx.x86_64 1.6.2-23.el6.art @atomic
onto a centos 6.6
Linux ESL-DR2-jb-01 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
paulie
Forum User
Forum User
Posts: 76
Joined: Tue Apr 20, 2010 2:49 am

Re: NGINX Semaphores Arrays

Unread post by paulie »

Hi,

I'll save you 1 hour (for which I'll need some kind of payment in kind later) its kidneystones, sorry mod_security.

I've replicated this with mod_security + pagespeed and just mod_security on latest nginx (I don't use that other module).

Oh and there's also this (so its not an atomic problem if that makes you feel any better) :

http://serverfault.com/questions/668151 ... 4294967295

I've also just stumbled across this which may have some more insight (I'm still trying to work out what it says) :

https://github.com/SpiderLabs/ModSecurity/issues/658

There are more issues on github that relate directly to this semaphore problem but for some reason I can't find them anymore.

When I get time I'm going to play with the refactoring branch ( https://github.com/SpiderLabs/ModSecuri ... efactoring ), but I believe this branch also has the semaphore problem (my bigger problems are my worker processes only lasting a couple of minutes and mod_sec enabled with no rules breaking Roundcube login and imap connections (probably because of the processes dying I suspect)).

Hope this helps,

Paul.
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: NGINX Semaphores Arrays

Unread post by scott »

That was what I was worried it might be. And I'll bet you arent even using mod_security on nginx in this case, right?
FrancoisD
Forum User
Forum User
Posts: 5
Joined: Wed Mar 18, 2015 3:20 pm
Location: QC, CANADA

Re: NGINX Semaphores Arrays

Unread post by FrancoisD »

i do not know scott if our dev team are using mod_secutiry, the only requirement i had was to take nginx from Atomic!

for the work around, that's what i did prior sending you the messages, i did added the ipcrm command in the init.d script under the stop.

what is strange tho, its that with the epel version and the original nginx version, there are no semaphore, this is why i figured it was atomic. in my test, which i tested all other distro source, only atomic generated semaphore.

i also have added in nagios, some script with the event tag, that if by some magical mistakes, it start piling up, it will clean those semaphore. this means, if someone is doing like 15 service nginx configtest, those are not doing stop service, therefore, semaphore are piling, until someone do a stop.

best regards!

(i am thinking of what type of payments you'd need!)

a tap on the back ?! thanks a lot for those searches Paulie
scott
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
Posts: 8355
Joined: Wed Dec 31, 1969 8:00 pm
Location: earth
Contact:

Re: NGINX Semaphores Arrays

Unread post by scott »

what is strange tho, its that with the epel version and the original nginx version, there are no semaphore, this is why i figured it was atomic. in my test, which i tested all other distro source, only atomic generated semaphore.
Its because its built with mod_security support. Nginx is not like apache where you can turn that off at run time. We could look into building a custom repo for you that did not have that, you can reach out to sales@atomicorp.com for details there.

Other workarounds would be that nagios option or some self-healing rules in ASL.
FrancoisD
Forum User
Forum User
Posts: 5
Joined: Wed Mar 18, 2015 3:20 pm
Location: QC, CANADA

Re: NGINX Semaphores Arrays

Unread post by FrancoisD »

thank you scott! so for now, i have put in place 2 work arrounds,

nagios event to clear sempahore when it get to a certain treshold

and added ipcrm in the init.d script.

that should cover everything, i did posted here my findings so other people could benefits of what i did.

i did added the init.d script changes in our cf-engine, so if atomic is pushing a new init.d script, it will be auto-re-edited to add those ipcrm.

best regards! and have a great summer time!
Post Reply