<div dir="ltr"><div><div><div>Chelsea,<br><br></div>Thanks for your response! We aren't seeing any resource usage spikes during/before outages, and the disk isn't filling up, so that isn't the issue on our end. We've already limited ASpace to our internal network, so it's not a question of extreme traffic.<br><br></div>Thanks,<br></div>--Alex<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 28, 2017 at 12:47 PM, Chelsea Lobdell <span dir="ltr"><<a href="mailto:clobdel1@swarthmore.edu" target="_blank">clobdel1@swarthmore.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Alex,<br><br></div>Not sure if this is going to be helpful or not but we were seeing a lot of CPU load warnings on our dev server that would bring the frontend of the application to a grinding halt until it was restarted. I was unable to see anything in the logs or any sort of malicious traffic so I put a few stop gaps in place. First, I adjusted IP Table and Firewall rules to lock down the application to only campus IP ranges. The application is only used by staff currently and they know if they are coming from off campus they need to use EZProxy or VPN for access. Secondly, I set a nightly restart cron from the root user's crontab: <br># restart archivesspace every night at 2am<br>00 02 * * * /etc/init.d/archivesspace stop<br>05 02 * * * /etc/init.d/archivesspace start<br><br></div>We haven't experience CPU load warnings since I put these stop gaps in place. So I'm still uncertain of what was causing the issue but maybe these stop gaps can help you too. <br><br></div>- Chelsea<br></div><div class="gmail_extra"><br clear="all"><div><div class="m_3286751538356617560gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="color:rgb(128,128,128)"><em><span style="font-size:small">---------------</span></em></span><br><span style="color:rgb(128,128,128)"><em><span style="font-size:small">Chelsea Lobdell</span></em></span><br><span style="color:rgb(128,128,128)"><em><span style="font-size:small">Library Web Developer/ Swarthmore College</span></em></span><br><span style="color:rgb(128,128,128)"><em><span style="font-size:small"><a href="mailto:clobdel1@swarthmore.edu" target="_blank">clobdel1@swarthmore.edu</a> / <a href="tel:(610)%20690-6818" value="+16106906818" target="_blank">(610)690-6818</a></span></em></span></div></div></div></div>
<br><div class="gmail_quote"><div><div class="h5">On Tue, Feb 28, 2017 at 12:27 PM, Alexander Duryee <span dir="ltr"><<a href="mailto:alexanderduryee@nypl.org" target="_blank">alexanderduryee@nypl.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div><div>Our production instance of ASpace has been suffering from intermittent outages for the past few weeks. Roughly once or twice a day, the application will go down for 1-5 minutes, then bring itself back up. Perplexingly, this doesn't seem to impact the backend - for example, the application went down during a reindex, but the reindex continued (and was logged) during the outage. However, API requests are dropped and the frontend is inaccessible during the outage.<br><br></div>We haven't identified a cause for the outages (e.g. heavy traffic, lots of database calls), as there's no consistent pattern. There's nothing in archivesspace.out to indicate any troubles.<br><br></div>Has anyone else experienced similar issues? Is there another log that we could refer to (e.g. ASpace's Apache instance) to diagnose the outages?<br clear="all"><div><div><div><br></div><div>Thanks!<br></div><div>--Alex<span class="m_3286751538356617560HOEnZb"><font color="#888888"><br></font></span></div><span class="m_3286751538356617560HOEnZb"><font color="#888888"><div><br>-- <br><div class="m_3286751538356617560m_5487311650241406326gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Alexander Duryee<div>Metadata Archivist</div><div>New York Public Library</div><div><a href="tel:(917)%20229-9590" value="+19172299590" target="_blank">(917)-229-9590</a></div><div><a href="mailto:alexanderduryee@nypl.org" target="_blank">alexanderduryee@nypl.org</a></div></div></div></div></div>
</div></font></span></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyra<wbr>lists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/m<wbr>ailman/listinfo/archivesspace_<wbr>users_group</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Alexander Duryee<div>Metadata Archivist</div><div>New York Public Library</div><div>(917)-229-9590</div><div><a href="mailto:alexanderduryee@nypl.org" target="_blank">alexanderduryee@nypl.org</a></div></div></div></div></div>
</div>