Alex Rivera | Logout

How to get Rails to automatically reestablish database connections after a database downtime

Asked 2012-09-18T01:07:28.700
20

After a database downtime, Rails will first throw this error once:

ActiveRecord::StatementInvalid: NativeException: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

From then on, every database call with have the following error, even after the database is back up:

ActiveRecord::StatementInvalid: ActiveRecord::JDBCError: This connection has been closed.

To get the server running again, I have to restart the rails server. This is not ideal for us, as our prod engineers would like to do maintenance on our databases without having to also bring back up all the services that depend on the database. So, I'm wondering - is there a way to automatically get Rails to try to reestablish the database connection or a recommended way to get this behavior?

Things I have tried:

I have already tried setting reconnect to true in my database options, and with that, I can kill individual database connections, and rails will reestablish the connections. However, it will not after a database outage. I found that from a command console I could get the connection back up by calling

ActiveRecord::Base::establish_connection

So maybe finding a clean place for rails to call the above command this would work? Any suggestions?

sql ruby-on-rails database-connection
Edit
Report

1 Answer

0

I'm not sure how to do exactly what you're asking, but I have another 'process' suggestion: set up simple scripts so your prod engineers can easily stop and start all applications.

Develop a set of capistrano recipes (or other scripts) that your prod engineers can use to stop and start all applications. For a normal Rails app, all you should really need to do is put a maintenance page, so that nginx or apache serves that page instead of forwarding requests to the rails instances. Ideally then, rails workers stop getting requests, db goes down, db comes up, then the maintenance page gets taken down and the workers get requests again, never realizing the database went away for a while.

In the case of background workers, they may need to be actually stopped and started by the script unless their queue is empty and stays empty. Any scheduled rake tasks or other scheduled jobs will probably fail if they depend on the database and run while it's down, so you'll want to try to schedule them to run outside the window when you normally do db maintenance.

If your prod engineers don't like running scripts (!), you could probably set up a nice web interface to make it easy for them. This will probably prove useful for more than just dealing with database connection errors, as it will empower more people in your organization to take care of basic things like stopping and starting your apps.

answered 2012-11-01T13:17:24.133

Your Answer