I have a Rails app that processes a large (millions) number of records in a mysql database. Once it starts working, its memory use quickly grows at a speed of 50MB per second. With tools like oink I was able to narrow down the root cause to one loop that goes through all the records in a big table in the database.
I understand if I use something like Person.all.each, all the records will be loaded into memory. However if I switch to find_each, I still see the same memory issue. To further isolate the problem I created the following test controller, which does nothing but looping through the records. I suppose find_each only keeps a small number of objects in memory each time, but memory use grows linearly as it executes.
class TestController < ApplicationController
def memory_test
Person.find_each do |person|
end
end
I suspect it has to do with ActiveRecord caching the query results. But I checked my environment settings and I do have all the caching related options set to false in development (I am using the default settings created by rails). I did some search online but couldn't find a solution.
I am using rails 3.1.0 rc1 and ruby 1.9.2
Thanks!