[Archivesspace_Users_Group] Performance tuning

[Blake Carver]

> ASPACE_JAVA_XMX="-Xmx16144m"

If the site needs 16 gigs of RAM to keep going, something is probably wrong.

> AppConfig[:indexer_records_per_thread] = 25
> AppConfig[:indexer_thread_count] = 7
> AppConfig[:indexer_solr_timeout_seconds] = 300

Messing around with those indexer settings is generally a good idea for sites with ALLOTTA records and/or a decent number of people adding new records. "It Depends" is about the only answer for those settings. If you have really deep/complex resources you'll probably want to change those up. Try slowing it down a bit. Try 2/50 or 2/25. You could even slow it WAY down to 1/1. It'll take forever to finish, but you can see exactly what it's doing each time.

BUT

If your site was OK and now it's not, maybe something else has gone wrong.

Check your logs. Make sure you're running the log_level on debug to catch everything possible.

- Look for ERROR or FATAL, maybe Java out of memory errors?
Sometimes there will be an Error or error but it's usually ERROR

-Is your indexer just running forever? Maybe it's caught in a crashing loop?
 "grep Indexed" do those numbers keep going up and down?
You sould see:
 Indexed 102303 of 102303 archival_object records in repository
Only once, you don't want to see that count going up and up and then suddenly
 Indexed 1 of 102303 archival_object records in repository

- Also check your proxy logs, maybe there's a bad bot hitting the PUI way too hard.

- Also just check that the server doesn't have anything new running and using up all the resources.

The answer is usually in the logs.

________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Cowing [he], Jared <jec15 at williams.edu>
Sent: Tuesday, September 13, 2022 12:33 PM
To: archivesspace_users_group at lyralists.lyrasis.org <archivesspace_users_group at lyralists.lyrasis.org>
Subject: [Archivesspace_Users_Group] Performance tuning

Hi all,

I'm currently working with our campus IT to identify the cause(s) of a degradation in performance that we've been seeing. I'm relatively new to our library and to ArchivesSpace generally, but I've been told it's been slowing bit by bit for a few years, and the problem has escalated just in the past few weeks. Neither full re-indexes nor our nightly re-starts seem to help.

I'm aware of this page on tuning<https://archivesspace.github.io/tech-docs/provisioning/tuning.html>, which has been quite helpful in addition to suggestions already posted to this list. We're hopeful that moving to external Solr with our next upgrade will also help (currently on 2.7.1), but are still trying other measures just in case it doesn't. While we look more into general best practices for tuning our hosting environment, I'd also like to check with all of you to see if there are common issues that are more specific to ArchivesSpace that we have overlooked and should focus our attention on.

Here are a few of our key settings. Our Java memory variables are below. I get the sense that they are higher than average, is that so?
ASPACE_JAVA_XMX="-Xmx16144m"
ASPACE_JAVA_XSS="-Xss8m"

Indexer settings from config.rb:
AppConfig[:indexer_records_per_thread] = 25
AppConfig[:indexer_thread_count] = 7
AppConfig[:indexer_solr_timeout_seconds] = 300


And copied from our innodb settings:
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=8G
symbolic-links=0
max_allowed_packet=128M
open_files_limit=9182

I appreciate any tips on what we ought to be looking for. I know it's hard to give advice from afar when each institution's situation is different, but thought it worth asking in case anything jumped out before we turn to technical support.

Thanks,
--

Jared Cowing | Systems Librarian | he/him
WILLIAMS COLLEGE LIBRARIES<https://library.williams.edu/>  | Williamstown, MA | (413)597-3061
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20220915/9f779975/attachment.html>