[Archivesspace_Users_Group] Intermittent Outages

[Alexander Duryee]

Chelsea,

Thanks for your response!  We aren't seeing any resource usage spikes
during/before outages, and the disk isn't filling up, so that isn't the
issue on our end.  We've already limited ASpace to our internal network, so
it's not a question of extreme traffic.

Thanks,
--Alex

On Tue, Feb 28, 2017 at 12:47 PM, Chelsea Lobdell <clobdel1 at swarthmore.edu>
wrote:

> Hi Alex,
>
> Not sure if this is going to be helpful or not but we were seeing a lot of
> CPU load warnings on our dev server that would bring the frontend of the
> application to a grinding halt until it was restarted. I was unable to see
> anything in the logs or any sort of malicious traffic so I put a few stop
> gaps in place. First, I adjusted IP Table and Firewall rules to lock down
> the application to only campus IP ranges. The application is only used by
> staff currently and they know if they are coming from off campus they need
> to use EZProxy or VPN for access. Secondly, I set a nightly restart cron
> from the root user's crontab:
> # restart archivesspace every night at 2am
> 00 02 * * * /etc/init.d/archivesspace stop
> 05 02 * * * /etc/init.d/archivesspace start
>
> We haven't experience CPU load warnings since I put these stop gaps in
> place. So I'm still uncertain of what was causing the issue but maybe these
> stop gaps can help you too.
>
> - Chelsea
>
> *---------------*
> *Chelsea Lobdell*
> *Library Web Developer/ Swarthmore College*
> *clobdel1 at swarthmore.edu <clobdel1 at swarthmore.edu> / (610)690-6818
> <(610)%20690-6818>*
>
> On Tue, Feb 28, 2017 at 12:27 PM, Alexander Duryee <
> alexanderduryee at nypl.org> wrote:
>
>> Our production instance of ASpace has been suffering from intermittent
>> outages for the past few weeks.  Roughly once or twice a day, the
>> application will go down for 1-5 minutes, then bring itself back up.
>> Perplexingly, this doesn't seem to impact the backend - for example, the
>> application went down during a reindex, but the reindex continued (and was
>> logged) during the outage.  However, API requests are dropped and the
>> frontend is inaccessible during the outage.
>>
>> We haven't identified a cause for the outages (e.g. heavy traffic, lots
>> of database calls), as there's no consistent pattern.  There's nothing in
>> archivesspace.out to indicate any troubles.
>>
>> Has anyone else experienced similar issues?  Is there another log that we
>> could refer to (e.g. ASpace's Apache instance) to diagnose the outages?
>>
>> Thanks!
>> --Alex
>>
>> --
>> Alexander Duryee
>> Metadata Archivist
>> New York Public Library
>> (917)-229-9590 <(917)%20229-9590>
>> alexanderduryee at nypl.org
>>
>> _______________________________________________
>> Archivesspace_Users_Group mailing list
>> Archivesspace_Users_Group at lyralists.lyrasis.org
>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>
>>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>


-- 
Alexander Duryee
Metadata Archivist
New York Public Library
(917)-229-9590
alexanderduryee at nypl.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20170228/55a9e689/attachment.html>