I am trying to take advantage of a quad-core machine by parallelizing a costly operation that is performed on a list of about 1000 items.
I am using R's parallel::mclapply function currently:
res = rbind.fill(parallel::mclapply(lst, fun, mc.cores=3, mc.preschedule=T))
Which works. Problem is, any additional subprocess that is spawned has to allocate a large chunk of memory:
Ideally, I would like each core to access shared memory from the parent R process, so that as I increase the number of cores used in mclapply, I don't hit RAM limitations before core limitations.
I'm currently at a loss on how to debug this issue. All of the large data structures that each process accesses are globals (currently). Is that somehow the issue?
I did increase my shared memory max setting for the OS to 20 GB (available RAM):
$ cat /etc/sysctl.conf
kern.sysv.shmmax=21474836480
kern.sysv.shmall=5242880
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
kern.maxproc=2048
I thought that would fix things, but the issue still occurs.
Any other ideas?