I am running a genome assembly program *Trinity, http://trinityrnaseq.sourceforge.net/, if interested) on one of the XSEDE resources. The hardware limits the number of threads to 2500, which the program always wants to exceed... It there an easy way to limit the number of threads executed? I have tried -XX:ParallelGCThreads=16, but this seems to introduce new errors.
So, is there a runtime command to limit the total number of threads??
You can use a custom queue that runs as a separate process that handles the number of threads limitation. The advantage of this is that you can opt to either limit the threads or you can still keep adding the number of threads. You will probably have a addToQueue(Thread t) class and subsequently a consumer consuming all these threads. The Queue will know how many threads are actively running. The daemon process will fire the consume() method of this queue at will if the threads are well within the range. And after every thread finishes or compeltes it job, it reports back to the queue. The Queue that you maintain can be a priority queue if you feel there should be a priority on the running tasks. This not only removes the dependency on the JVM but also makes your program look cleaner.