Recently we noticed that Sahi would start giving "Too many open files" errors within 5 minutes of a restart. We Googled about this error and found that in Unix, the system as well as each user has a maximum number of files it can open simultaneously.
In our case, Sahi is run by the user "vnc". So we first increased the maximum number of open files by the user vnc. Here are the steps:
Open "/etc/security/limits.conf". In this file, enter the following at the end:
vnc soft nofile 4096 vnc hard nofile 16384
Open "/etc/pam.d/common-session". In this file, enter the following at the end:
session required pam_limits.so
Now restart the machine. Once it's back, execute "su - vnc" to make yourself the vnc user. Then execute the following commands:
ulimit -Hn ulimit -Sn
The first command should return a value of 4096 and the second 16384.
After making the above changes, we expected that Sahi will still give "Too many open files" errors within 15 minutes. So we decided to debug further.
We looked at the files that are getting opened by Sahi. You can do this by executing "lsof -p <process-id>" where <process-id> is Sahi's process id. We noticed that Sahi is opening multiple socket connections (which are treated as files by Unix) every second. Most of these connections were requested by external websites (probably spammy ones). The number of requests was so large that after a while, Sahi will reach its maximum number of open files.
We fixed this by closing Sahi's port to external connections. Sahi, by default, runs on port 9999. We used iptables to close port 9999 for external requests that are not originating from the same machine (localhost).
First install "iptables-persistent" which simplifies reloading of iptables.
apt-get install iptables-persistent
Create an iptables rule to block all external TCP requests to port 9999.
iptables -I INPUT ! -i lo -p tcp --dport 9999 -j DROP
Save this rule and reload iptables.
/etc/init.d/iptables-persistent save /etc/init.d/iptables-persistent reload
With this new rule, external requests couldn't reach Sahi and the "Too many open files" error disappeared.
