<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Hey All-</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
This may be related to some of the indexer problems that Maureen and others have noticed from time to time, but I'm throwing this out there in case others want to chime in with a solution (best!) or some insight.</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Anyway, here's the scenario. Our setup for production is configured as shown below the fold. Basically 4 frontend servers behind an F5. We've noticed that when we do a stop/start of the frontends (for plugin updates, etc) the indexer for one of the frontends
sometimes fails. The logs for that server show a lot of errors of the following type:</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<div>D, [2015-10-07T17:53:28.369000 #16298] DEBUG -- : Thread-9382: POST /users/staff_system/login [session: nil]</div>
<div>D, [2015-10-07T17:53:28.377000 #16298] DEBUG -- : Thread-9382: Post-processed params: {:username=>"staff_system", :password=>"[FILTERED]", :expiring=>false}</div>
<div>D, [2015-10-07T17:53:28.529000 #16298] DEBUG -- : Thread-9382: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 161.0ms</div>
<div>D, [2015-10-07T17:53:28.542000 #16298] DEBUG -- : Thread-9384: POST /update_monitor [session: nil]</div>
<div>D, [2015-10-07T17:53:28.548000 #16298] DEBUG -- : Thread-9384: Post-processed params: {:active_edits=>#<JSONModel(:active_edits) {"jsonmodel_type"=>"active_edits", "active_edits"=>[]}>}</div>
<div>D, [2015-10-07T17:53:28.550000 #16298] DEBUG -- : Thread-9384: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"26"}, ["{\"error\":\"Access denied\"}\n"]]... in 8.0ms</div>
<div>D, [2015-10-07T17:53:28.558000 #16298] DEBUG -- : Thread-5526: POST /users/staff_system/login [session: nil]</div>
<div>D, [2015-10-07T17:53:28.564000 #16298] DEBUG -- : Thread-5526: Post-processed params: {:username=>"staff_system", :password=>"[FILTERED]", :expiring=>false}</div>
<div>D, [2015-10-07T17:53:28.723000 #16298] DEBUG -- : Thread-5526: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 165.0ms</div>
<div>D, [2015-10-07T17:53:32.879000 #16298] DEBUG -- : Thread-5518: POST /users/search_indexer/login [session: nil]</div>
<div>D, [2015-10-07T17:53:32.885000 #16298] DEBUG -- : Thread-5518: Post-processed params: {:username=>"search_indexer", :password=>"[FILTERED]", :expiring=>false}</div>
<div>D, [2015-10-07T17:53:33.050000 #16298] DEBUG -- : Thread-5518: Responded with [403, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"25"}, ["{\"error\":\"Login failed\"}\n"]]... in 171.0ms</div>
<div>#<RuntimeError: Authentication to backend failed: {"error":"Login failed"}</div>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Sometimes a simple start/stop of that particular server solves the issue, but we have had to replicate a known good server to the bad to clear things up.</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Has anyone else seen something similar? Any thoughts on what a potential cause (and cure) might be?</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Thanks!</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Joshua</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
Setup details with some obfuscation for security.</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<div>- ArchivesSpace version: 1.3</div>
<div>- Java version: 1.7.0_85-b15</div>
<div>- CPU cores: 4 cores (each server)</div>
<div>- Memory: 36gb (each server)</div>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div><font face="Tahoma,sans-serif">At the top level, clients connection for our hardware load balancer, Big-IP’s F5. The F5 terminates SSL and sends HTTP traffic to one of four servers running Oracle HTTP Server (OHS), which is essentially Apache. Two of these
servers also run the ArchivesSpace application. OHS is configured to send the HTTP traffic to the two application servers.</font></div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<img src="cid:EE5F9913-10DF-4257-876B-AB3E04343E46" type="image/png" width="508.500000" height="337.500000"></div>
<div style="color: rgb(0, 0, 0); font-family: Tahoma, sans-serif; font-size: 14px;">
<div id="MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
</body>
</html>