Nickserver hangs unresponsive after EMFILE: Too many open files
This happened today on alpaca.
I tried to query nickserver with:
--- git/pixelated-user-agent ‹master› » curl -k -X POST -d address=varac@riseup.net https://nicknym.mail.bitmask.net:6425
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<hr>
<address>Apache/2.4.10 (Debian) Server at nicknym.mail.bitmask.net Port 6425</address>
</body></html>
/var/log/nickserver.log
didn't show any activity after the last crash dump, and only worked again after restarting nickserver:
E, [2017-05-17T15:14:48.296568 #16167] ERROR -- : /srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/reactor.rb:16:in `new'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/reactor.rb:16:in `initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/mailbox/evented.rb:10:in `new'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/mailbox/evented.rb:10:in `initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/mailbox.rb:6:in `initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:106:in `new'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:106:in `initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:21:in `new'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:21:in `initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid.rb:195:in `new'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid.rb:195:in `new'
/srv/leap/nickserver/lib/nickserver/source.rb:8:in `initialize'
/srv/leap/nickserver/lib/nickserver/request_handlers/leap_email_handler.rb:16:in `new'
/srv/leap/nickserver/lib/nickserver/request_handlers/leap_email_handler.rb:16:in `source'
/srv/leap/nickserver/lib/nickserver/request_handlers/leap_email_handler.rb:24:in `nicknym_email?'
/srv/leap/nickserver/lib/nickserver/request_handlers/leap_email_handler.rb:10:in `handle'
/srv/leap/nickserver/lib/nickserver/request_handlers/base.rb:6:in `call'
/srv/leap/nickserver/lib/nickserver/handler_chain.rb:42:in `try_handler'
/srv/leap/nickserver/lib/nickserver/handler_chain.rb:30:in `block in handle'
/srv/leap/nickserver/lib/nickserver/handler_chain.rb:30:in `each'
/srv/leap/nickserver/lib/nickserver/handler_chain.rb:30:in `find'
/srv/leap/nickserver/lib/nickserver/handler_chain.rb:30:in `handle'
/srv/leap/nickserver/lib/nickserver/dispatcher.rb:41:in `handle'
/srv/leap/nickserver/lib/nickserver/dispatcher.rb:34:in `respond_to'
/srv/leap/nickserver/lib/nickserver/reel_server.rb:41:in `handle_request'
/srv/leap/nickserver/lib/nickserver/reel_server.rb:30:in `block in on_connection'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/connection.rb:73:in `each_request'
/srv/leap/nickserver/lib/nickserver/reel_server.rb:29:in `on_connection'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/server.rb:50:in `call'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/server.rb:50:in `handle_connection'
/srv/leap/nickserver/lib/nickserver/reel_server.rb:24:in `block in handle_connection'
/srv/leap/nickserver/lib/kernel_ext.rb:11:in `block in silence_warnings'
/srv/leap/nickserver/lib/kernel_ext.rb:24:in `with_warnings'
/srv/leap/nickserver/lib/kernel_ext.rb:11:in `silence_warnings'
/srv/leap/nickserver/lib/nickserver/reel_server.rb:23:in `handle_connection'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
E, [2017-05-17T15:55:27.539107 #16167] ERROR -- : Actor crashed!
Errno::EMFILE: Too many open files - accept(2)
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_server.rb:42:in `accept_nonblock'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_server.rb:42:in `accept_nonblock'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_server.rb:37:in `accept'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/server.rb:38:in `block in run'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/server.rb:38:in `loop'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/reel-0.6.1/lib/reel/server.rb:38:in `run'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
/srv/leap/nickserver/vendor/bundle/ruby/2.1.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
Starting nickserver 127.0.0.1:64250
I, [2017-05-24T10:11:06.540295 #13912] INFO -- : GET /?address=nobody@riseup.net
D, [2017-05-24T10:11:06.540517 #13912] DEBUG -- : {"address"=>["nobody@riseup.net"]}
I, [2017-05-24T10:11:12.685720 #13912] INFO -- : -> 200