bigcouch nodes doesn't get registered as cluster members
In preparation for the demo in DC, we decided to consolidate the couchdb servers in seattle. So I removed the one that wasn't there in seattle, and built a new one. I purged couchdb on all the existing nodes and re-deployed to them (including to the webapp), but now when you try to go to bitmask.net, you get the rails error, "We're sorry, but something went wrong".
I find this is put in the production.log:
RestClient::Conflict (409 Conflict): vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!' vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result' vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit' /usr/lib/ruby/1.9.1/net/http.rb:745:in `start' vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit' vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute' vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute' vendor/bundle/ruby/1.9.1/gems/couchrest-1.1.3/lib/couchrest/rest_api.rb:94:in `execute' vendor/bundle/ruby/1.9.1/gems/couchrest-1.1.3/lib/couchrest/rest_api.rb:50:in `put' vendor/bundle/ruby/1.9.1/gems/couchrest-1.1.3/lib/couchrest/database.rb:130:in `save_doc' vendor/bundle/ruby/1.9.1/gems/couchrest_session_store-0.1.3/lib/couchrest_session_store.rb:74:in `set_session' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:327:in `commit_session' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:211:in `context' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/cookies.rb:341:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `_run__3576451226799408762__call__870488930827515836__callbacks' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_call_callbacks' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:27:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/remote_ip.rb:31:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:32:in `call_app' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `block in call' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/tagged_logging.rb:22:in `tagged' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `call' vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/request_id.rb:22:in `call' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call' vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/cache/strategy/local_cache.rb:72:in `call' vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call' vendor/bundle/ruby/1.9.1/gems/rack-ssl-1.3.3/lib/rack/ssl.rb:27:in `call' vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward' vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch' vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup' vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!' vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/engine.rb:479:in `call' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:223:in `call' vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in `method_missing' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/request_handler.rb:96:in `process_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:479:in `safe_fork' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:180:in `start' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/application_spawner.rb:129:in `start' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize' :10:in `synchronize' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server_collection.rb:79:in `synchronize' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:137:in `spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:357:in `server_main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/abstract_server.rb:206:in `start_synchronously' /usr/share/phusion-passenger/helper-scripts/passenger-spawn-server:99:in `'
(from redmine: created on 2013-09-03, closed on 2013-09-04, relates #3712 (closed))