Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

'Too many open files' since upgrade to QB9 #4188

JShelton ·

Since upgrading to QuickBuild 9.0.23 from 8.0.39, we've experienced 'Too many open files' error on two different CentOS 7 64-bit servers, approximately a week or so after each got upgraded. Increasing user limits and restarting the java process is helping us get by, but numbers are still climbing and we're wondering if there might be a file descriptor leak in QB9 somewhere?

As an example, our open files on a major server this morning that experienced the issue were over 1.08 million. After shutting down QuickBuild, that dropped to 3,000 without any QuickBuild processes running. After starting it back up, I've clocked numbers (lsof | wc -l) at 231,228, followed by 223,402 around 30 minutes later, followed by 278,831 30 minutes after that. A third QuickBuild server, which has not (yet?) encountered the error and has well over 1 million open files, but we also preemptively increased the user limit for open files before installing QB9 in anticipation of the issue.

Note:
running the command cat /proc/sys/fs/file-nr gives 3,360 files open currently, not counting duplicates
running the command lsof yields a LOT of connections through TCPv6

  • replies 2
  • views 1155
  • stars 0
robinshen ADMIN ·

Please switch to page "Administration / Plugin Management" to find "Grid Measurement Plugin", then disable it to see if open files still increase. If still increase, please get a memory dump for analysis by running:
/path/to/jdk/bin/jmap -dump:format=b, file=/tmp/dump.hprof

You may put the memory dump on DropBox and send me the link

robinshen ADMIN ·