<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Michael</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
These aren't answers, but I think it might help the group if we knew a bit more about how your instance is structured - both from a tech perspective (memory allocation to the app and Solr) and things like how many repos and how many objects (resources, AOs,
etc) are in the DB. The structure of your resources may also be useful. IE are they wide or deep or both? Wide meaning a lot of siblings at each level, but not a lot of levels in the hierarchy and deep meaning a lot of levels in the hierarchy, but not as many
siblings at each level.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
The plugins that you are using probably aren't the culprit, but they <b>can</b> add/override index functionality, so listing those out may help as well.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
It might also be good to know how many edits are made concurrently on average.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Couple of things that sprang to mind to check (if you haven't already). Have you noticed this same behavior in an instance that is not in use? IE have you set up a clone of your production instance, let it do its initial full index, and then just let it sit?
Do you see errors in the app log that have any bearing on the problem or pop up around or just before the app goes unresponsive or OOM?<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
In case it helps for comparison, Dartmouth is running 3.3.1 (skipped 3.2.0) and allocating 4GB each to the app and Solr - everything running in containers. We have 5 repos, though only one is utilized much. That repo has about 15k resources and 670k AOs with
30k top containers and 15k agents. We have relatively few events or subjects. The resources tend to be wide with max 4 levels of hierarchy. Our largest resource has 10s of thousands of AOs in the hierarchy. We also run a huge number of plugins. We have relatively
few editors - less than 5 at any one time. <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Full index typically takes about 24 hours. We have not seen memory issues in any of our instances, though I have occasionally seen indexer timeouts during a full index. We have stock settings for the indexer (4, 1, 25) - though I had to raise the solr timeout
a huge amount to 7200 for 3.3.1 to avoid solr timeouts. We do run the PUI, so much of the full index time is the PUI index churning away. Staff side indexing takes about 6-8 hours.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Best,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Joshua<br>
</div>
<div id="appendonsend"></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size: 11pt; color: rgb(0, 0, 0);" face="Calibri, sans-serif"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Michael
Smith <mismith@nla.gov.au><br>
<b>Sent:</b> Tuesday, May 23, 2023 7:52 PM<br>
<b>To:</b> archivesspace_users_group@lyralists.lyrasis.org <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> [Archivesspace_Users_Group] Diagnosing issues with ArchivesSpace</font>
<div> </div>
</div>
<div style="word-wrap:break-word" lang="EN-AU">
<table style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none" width="100%" cellspacing="0" cellpadding="0" border="0" align="left">
<tbody style="display:block">
<tr>
<td cellpadding="7px 2px 7px 2px" style="padding: 7px 2px; background-color: rgb(166, 166, 166);" width="1px" valign="middle">
</td>
<td cellpadding="7px 5px 7px 15px" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, Segoe UI, Segoe WP, Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);" width="100%" valign="middle">
<div>You don't often get email from mismith@nla.gov.au. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable" data-loopstyle="link">
Learn why this is important</a></div>
</td>
<td cellpadding="7px 5px 7px 5px" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, Segoe UI, Segoe WP, Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);" width="75px" valign="middle" align="left">
</td>
</tr>
</tbody>
</table>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">Hello, </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">Our team has been facing recurring issues with our ArchivesSpace setup since October last year, which we've been unable to fully resolve despite concerted efforts.</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">We’re currently running v3.2 on Red Hat Enterprise Linux Server 7.9 (Maipo) and we do have a few custom plugins developed by Hudmol. These don’t appear to be causing the issues that we’re seeing but we haven’t ruled that out yet.</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">The primary problem involves intermittent system slowdowns and shutdowns, requiring frequent reboots to regain functionality. This occurs on average 3-4 times weekly but can sometimes be more frequent. This issue is affecting
multiple teams across our organization.</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">The most common symptom of our problem that we are seeing now looks to be a connection pool leak where what looks like indexer threads are holding connections in a closed wait state and preventing them from being used for other
requests. This leads to the main page timing out and staff seeing 504 errors, when unresponsive in this manner we usually restart the application. If the application hits an OOM, it will restart itself.</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">Some of the things we’ve attempted so far,</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<ul style="margin-bottom: 0cm;margin-top:0cm" type="disc">
<li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">changed default config settings for indexer records per thread, thread count and solr timeout to 10, 2 & 300</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">modified archivesspace.sh to increase memory available (ASPACE_JAVA_XMX="-Xmx35g")</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">disabled both PUI and PUI indexer</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">application logging to a circular log</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">changed the garbage collection policies (ASPACE_GC_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:NewRatio=1 -XX:+ExitOnOutOfMemoryError -XX:+UseGCOverheadLimit")</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">checked top_containers with empty relationships (0 results)</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">checked for duplicate event relationships (0 results)</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">checked for empty indexer state files per record type (0 empty state files)</span></li><li class="x_MsoListParagraph" style="margin: 0cm 0cm 0cm 36pt; font-size: 10pt; font-family: "Calibri", sans-serif;margin-left:0cm">
<span style="font-size:11.0pt">nightly restarts of the system</span></li></ul>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">Any advice with further diagnosis / troubleshooting would be appreciated. If you need additional information about our setup or the issues we're encountering, please let us know.</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt">Regards,</span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;">
<span style="font-size:11.0pt"> </span></p>
<div>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;margin-bottom:12.0pt; line-height:14.0pt">
<b><span style="font-size:11.0pt; font-family:"Arial",sans-serif">Michael Smith</span></b><span style="font-size: 11pt; font-family: "Arial", sans-serif; color: black;"> |
</span><span style="font-size:11.0pt; font-family:"Arial",sans-serif">Software Developer<span style="color: black;"><br>
02 6262 </span>1029<span style="color: black;"> | </span></span><a href="mailto:mismith@nla.gov.au" data-auth="NotApplicable" title="mailto:Servicename@nla.gov.au" data-loopstyle="link"><span style="font-size:11.0pt; font-family:"Arial",sans-serif">mismith@nla.gov.au</span></a><span style="font-size: 11pt; font-family: "Arial", sans-serif; color: black;">
| National Library of Australia</span><span style="font-size:11.0pt"></span></p>
<p class="x_MsoNormal" style="margin: 0cm; font-size: 10pt; font-family: "Calibri", sans-serif;margin-right:4.25pt; margin-bottom:25.0pt; margin-left:0cm; line-height:14.0pt">
<span style="font-size: 11pt; font-family: "Arial", sans-serif; color: black;">The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect
to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.</span><span style="font-size:11.0pt"></span></p>
</div>
</div>
</div>
</div>
</body>
</html>