[Archivesspace_Users_Group] Indexer failing on restart of service - anyone else seen this?

[Joshua D. Shaw]

Hey All-

This may be related to some of the indexer problems that Maureen and others have noticed from time to time, but I'm throwing this out there in case others want to chime in with a solution (best!) or some insight.

Anyway, here's the scenario. Our setup for production is configured as shown below the fold. Basically 4 frontend servers behind an F5. We've noticed that when we do a stop/start of the frontends (for plugin updates, etc) the indexer for one of the frontends sometimes fails. The logs for that server show a lot of errors of the following type:

D, [2015-10-07T17:53:28.369000 #16298] DEBUG -- : Thread-9382: POST /users/staff_system/login [session: nil]
D, [2015-10-07T17:53:28.377000 #16298] DEBUG -- : Thread-9382: Post-processed params: {:username=>"staff_system", :password=>"[FILTERED]", :expiring=>false}
D, [2015-10-07T17:53:28.529000 #16298] DEBUG -- : Thread-9382: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 161.0ms
D, [2015-10-07T17:53:28.542000 #16298] DEBUG -- : Thread-9384: POST /update_monitor [session: nil]
D, [2015-10-07T17:53:28.548000 #16298] DEBUG -- : Thread-9384: Post-processed params: {:active_edits=>#<JSONModel(:active_edits) {"jsonmodel_type"=>"active_edits", "active_edits"=>[]}>}
D, [2015-10-07T17:53:28.550000 #16298] DEBUG -- : Thread-9384: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"26"}, ["{\"error\":\"Access denied\"}\n"]]... in 8.0ms
D, [2015-10-07T17:53:28.558000 #16298] DEBUG -- : Thread-5526: POST /users/staff_system/login [session: nil]
D, [2015-10-07T17:53:28.564000 #16298] DEBUG -- : Thread-5526: Post-processed params: {:username=>"staff_system", :password=>"[FILTERED]", :expiring=>false}
D, [2015-10-07T17:53:28.723000 #16298] DEBUG -- : Thread-5526: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 165.0ms
D, [2015-10-07T17:53:32.879000 #16298] DEBUG -- : Thread-5518: POST /users/search_indexer/login [session: nil]
D, [2015-10-07T17:53:32.885000 #16298] DEBUG -- : Thread-5518: Post-processed params: {:username=>"search_indexer", :password=>"[FILTERED]", :expiring=>false}
D, [2015-10-07T17:53:33.050000 #16298] DEBUG -- : Thread-5518: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 171.0ms
#<RuntimeError: Authentication to backend failed: {"error":"Login failed"}

Sometimes a simple start/stop of that particular server solves the issue, but we have had to replicate a known good server to the bad to clear things up.

Has anyone else seen something similar? Any thoughts on what a potential cause (and cure) might be?

Thanks!
Joshua

Setup details with some obfuscation for security.

-          ArchivesSpace version: 1.3
-          Java version: 1.7.0_85-b15
-          CPU cores: 4 cores (each server)
-          Memory: 36gb (each server)

At the top level, clients connection for our hardware load balancer, Big-IP’s F5. The F5 terminates SSL and sends HTTP traffic to one of four servers running Oracle HTTP Server (OHS), which is essentially Apache. Two of these servers also run the ArchivesSpace application. OHS is configured to send the HTTP traffic to the two application servers.

[cid:EE5F9913-10DF-4257-876B-AB3E04343E46]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20151013/34451e8f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DartmouthAS-production.png
Type: image/png
Size: 49971 bytes
Desc: DartmouthAS-production.png
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20151013/34451e8f/attachment.png>