<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div class="elementToProof">> ASPACE_JAVA_XMX="-Xmx16144m"</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br class="ContentPasted0">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
If the site needs 16 gigs of RAM to keep going, something is probably wrong.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div class="ContentPasted0 elementToProof">> AppConfig[:indexer_records_per_thread] = 25</div>
<div class="ContentPasted0 elementToProof">> AppConfig[:indexer_thread_count] = 7</div>
<div class="elementToProof">> AppConfig[:indexer_solr_timeout_seconds] = 300</div>
<div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">Messing around with those indexer settings is generally a good idea for sites with
ALLOTTA records and/or a decent number of people adding new records. "It Depends" is about the only answer for those settings. If you have really deep/complex resources you'll probably want to change those up. Try slowing it down a bit. Try 2/50 or 2/25. You
could even slow it WAY down to 1/1. It'll take forever to finish, but you can see exactly what it's doing each time.</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">BUT</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">If your site was OK and now it's not, maybe something else has gone wrong.</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">Check your logs. Make sure you're running the log_level on debug to catch everything
possible.</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">- Look for ERROR or FATAL, maybe Java out of memory errors?</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">Sometimes there will be an Error or error but it's usually ERROR</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">-Is your indexer just running forever? Maybe it's caught in a crashing loop? <br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<span style="font-size: 11pt;"> "grep Indexed" do those numbers keep going up and down?</span><font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family: Calibri, Arial, Helvetica, sans-serif; margin: 0px;">
<span style="font-size: 14.6667px;">You sould see:</span></div>
<div name="divtagdefaultwrapper" style="font-family: Calibri, Arial, Helvetica, sans-serif; margin: 0px;">
<span style="font-size: 14.6667px;" class="ContentPasted1"> Indexed 102303 of 102303 archival_object records in repository<br>
</span></div>
<div name="divtagdefaultwrapper" style="font-family: Calibri, Arial, Helvetica, sans-serif; margin: 0px;">
<span style="font-size: 14.6667px;" class="ContentPasted1">Only once, you don't want to see that count going up and up and then suddenly</span></div>
<div name="divtagdefaultwrapper" style="font-family: Calibri, Arial, Helvetica, sans-serif; margin: 0px;">
<span style="font-size: 14.6667px;" class="ContentPasted1 ContentPasted2"> Indexed 1 of 102303 archival_object records in repository<br>
</span></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">- Also check your proxy logs, maybe there's a bad bot hitting the PUI way too hard.</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt">- Also just check that the server doesn't have anything new running and using up
all the resources.</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="3" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
<div id="signature_bookmark"></div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof">
The answer is usually in the logs.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Cowing [he],
Jared <jec15@williams.edu><br>
<b>Sent:</b> Tuesday, September 13, 2022 12:33 PM<br>
<b>To:</b> archivesspace_users_group@lyralists.lyrasis.org <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> [Archivesspace_Users_Group] Performance tuning</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I'm currently working with our campus IT to identify the cause(s) of a degradation in performance that we've been seeing. I'm relatively new to our library and to ArchivesSpace generally, but I've been told it's been slowing bit by bit for a few years,
and the problem has escalated just in the past few weeks. Neither full re-indexes nor our nightly re-starts seem to help.</div>
<div><br>
</div>
<div>I'm aware of <a href="https://archivesspace.github.io/tech-docs/provisioning/tuning.html" data-auth="NotApplicable">
this page on tuning</a>, which has been quite helpful in addition to suggestions already posted to this list. We're hopeful that moving to external Solr with our next upgrade will also help (currently on 2.7.1), but are still trying other measures just in case
it doesn't. While we look more into general best practices for tuning our hosting environment, I'd also like to check with all of you to see if there are common issues that are more specific to ArchivesSpace that we have overlooked and should focus our attention
on.</div>
<div><br>
</div>
<div>Here are a few of our key settings. Our Java memory variables are below. I get the sense that they are higher than average, is that so?</div>
<div>ASPACE_JAVA_XMX="-Xmx16144m"<br>
ASPACE_JAVA_XSS="-Xss8m"<br>
<br>
Indexer settings from config.rb:</div>
<div>AppConfig[:indexer_records_per_thread] = 25<br>
AppConfig[:indexer_thread_count] = 7<br>
AppConfig[:indexer_solr_timeout_seconds] = 300
<pre class="x_gmail-aLF-aPX-K0-aPE" style="font-family:"Courier New",Courier,monospace,arial,sans-serif; margin-top:0px; margin-bottom:0px; white-space:pre-wrap; color:rgb(0,0,0); font-size:14px"><br></pre>
And copied from our innodb settings:<br>
innodb_file_per_table<br>
innodb_flush_method=O_DIRECT<br>
innodb_log_file_size=1G<br>
innodb_buffer_pool_size=8G<br>
symbolic-links=0<br>
max_allowed_packet=128M<br>
open_files_limit=9182
<div><br>
</div>
<div>I appreciate any tips on what we ought to be looking for. I know it's hard to give advice from afar when each institution's situation is different, but thought it worth asking in case anything jumped out before we turn to technical support.</div>
<div><br>
</div>
<div>Thanks,</div>
-- <br>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<p><span style="font-size:11.0pt; font-family:"Calibri",sans-serif"><b>Jared Cowing</b><span style="color:#323e4f">
</span><b><span style="color:black">|</span></b><span style="color:#323e4f"> </span>
<font color="#999999">Systems Librarian</font><span style="color:#323e4f"> </span>
<b><span style="color:black">| </span></b><font color="#999999">he/him</font><br>
</span><u><a href="https://library.williams.edu/" data-auth="NotApplicable"><u><font color="#9900ff">W<font size="1">ILLIAMS</font> C<font size="1">OLLEGE</font> L<font size="1">IBRARIES</font></font></u></a></u><span style="color:rgb(23,54,93)"> </span><span style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(23,54,93)"> </span><b><span style="font-size:11.0pt; font-family:"Calibri",sans-serif; color:black">|</span></b><span style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(23,54,93)">
</span><span style="font-size:11pt; font-family:Calibri,sans-serif"><font color="#999999">Williamstown, MA </font></span><b style="font-family:Calibri,sans-serif; font-size:14.6667px"><span style="color:black">| </span></b><span style="color:rgb(153,153,153); font-family:Calibri,sans-serif; font-size:14.6667px">(413)597-3061</span></p>
<p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>