Since upgrading to QuickBuild 9.0.23 from 8.0.39, we've experienced 'Too many open files' error on two different CentOS 7 64-bit servers, approximately a week or so after each got upgraded. Increasing user limits and restarting the java process is helping us get by, but numbers are still climbing and we're wondering if there might be a file descriptor leak in QB9 somewhere?
As an example, our open files on a major server this morning that experienced the issue were over 1.08 million. After shutting down QuickBuild, that dropped to 3,000 without any QuickBuild processes running. After starting it back up, I've clocked numbers (lsof | wc -l
) at 231,228, followed by 223,402 around 30 minutes later, followed by 278,831 30 minutes after that. A third QuickBuild server, which has not (yet?) encountered the error and has well over 1 million open files, but we also preemptively increased the user limit for open files before installing QB9 in anticipation of the issue.
Note:
running the command cat /proc/sys/fs/file-nr
gives 3,360 files open currently, not counting duplicates
running the command lsof
yields a LOT of connections through TCPv6