<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Tom,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Not sure if it will help (and not sure if you shared your config.rb file in a previous message), but have you tried:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<ul>
<li>turning off the Solr backups during the re-indexing (<a href="https://github.com/archivesspace/archivesspace/blob/d207e8a7bb01c2b7b6f42ee5c0025d95f35ee7ae/common/config/config-defaults.rb#L76" id="LPlnk">https://github.com/archivesspace/archivesspace/blob/d207e8a7bb01c2b7b6f42ee5c0025d95f35ee7ae/common/config/config-defaults.rb#L76</a>).
Just going back to Dave's suggestion about keeping an eye out on disk space.</li><li>updating the record inheritance settings and removing the bit about inheriting scope and contents notes, which really bloats the index since most finding aids won't have lower-level descriptive notes (<a href="https://github.com/archivesspace/archivesspace/blob/d207e8a7bb01c2b7b6f42ee5c0025d95f35ee7ae/common/config/config-defaults.rb#L411-L413" id="LPlnk">https://github.com/archivesspace/archivesspace/blob/d207e8a7bb01c2b7b6f42ee5c0025d95f35ee7ae/common/config/config-defaults.rb#L411-L413</a>).
For our record inheritance settings , <a href="https://github.com/YaleArchivesSpace/aspace-deployment/blob/master/prod/config.rb#L204-L244" id="LPlnk">https://github.com/YaleArchivesSpace/aspace-deployment/blob/master/prod/config.rb#L204-L244</a>), we only
inherit two notes currently: access notes and preferred citation notes.<br>
</li><li>Turning off the PUI indexer until the staff indexing is done, and then turning the PUI indexer back on?</li></ul>
<div>When testing re-indexing locally, I usually bump up the two values that Blake listed below, at least until I start getting Java heapspace errors and don't' have any more RAM to allot to ASpace
<span id="🙂">🙂. <span style="margin:0px;caret-color:rgb(0, 0, 0)"><span style="margin:0px">But even just waiting for the archival objects can take a while, depending on the settings in your config.rb file. While waiting for a full re-index once when things
were re-indexing on a server that I didn't have access to, I'd periodically look at the last archival object ID that was indexed, and then run a database query to see how many more archival objects were left in that repo, since a full re-index seems to go
in order of the primary keys. e.g.</span></span>
<div style="margin:0px;caret-color:rgb(0, 0, 0)"><span style="margin:0px"><br>
</span></div>
<blockquote style="caret-color:rgb(0, 0, 0);margin-top:0px;margin-bottom:0px">
<div style="margin:0px">select count(*) from archival_object
<div style="margin:0px">where id > {archival_object id}</div>
and repo_id = {repo id};</div>
</blockquote>
</span></div>
<div><span>That would at least give me a sense of how much longer it might be for all the archival objects were left in one of our repos.</span></div>
<div><span><br>
</span></div>
<div><span>As for the slowdown, after all the archival objects are indexed in a repository, the next thing that happens (although it can take quite a while) will be for all the tree indexes to be created and finally committed to Solr. See <a href="https://github.com/archivesspace/archivesspace/blob/82c4603fe22bf0fd06043974478d4caf26e1c646/indexer/app/lib/pui_indexer.rb#L136" id="LPlnk936966">https://github.com/archivesspace/archivesspace/blob/82c4603fe22bf0fd06043974478d4caf26e1c646/indexer/app/lib/pui_indexer.rb#L136</a>.
If i recall correctly, there won't be any specific mentions in the logs about that, but after you get a message about all of the archival objects being indexed in a specfic repository, you'll get another message about the archival objects being indexed again
sometime later, at which point the full trees have been reindexed again, and then the indexer will be off to the next repo (or record type, like classification records, etc.)</span></div>
<div><br>
</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Tom Hanstra
<hanstra@nd.edu><br>
<b>Sent:</b> Monday, March 22, 2021 11:21 AM<br>
<b>To:</b> Archivesspace Users Group <archivesspace_users_group@lyralists.lyrasis.org><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] PUI indexing issues</font>
<div> </div>
</div>
<div>
<div dir="ltr">Thanks, Blake.
<div><br>
</div>
<div>In your testing, how big was the repository that you were testing against. Mine has "763368 archival_object records" and I consistently get into the 670K range for staff and 575 range for PUI before things really slow down. I'm now trying to really increase
the Java settings to see if that will help. So far, the problem is similar: real slow downs after zipping through the first records. I'll also try some of the settings you have there to see if fewer but larger threads work better than multiple smaller threads. </div>
<div><br>
</div>
<div>Tom</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Mar 22, 2021 at 10:52 AM Blake Carver <<a href="mailto:blake.carver@lyrasis.org">blake.carver@lyrasis.org</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="margin:0px; font-size:12pt">I did some experimenting this weekend, messing around with indexer speeds, and found I could get it to succeed with the right indexer settings. I think the answer is going to be "it depends" and you'll need to experiment
with what works on your set up with your data. I started with the defaults, then dropped it to reallllly slow (1 thread 1 per), then just tried to dial it up and down. The last one I tried worked fine, it was fast enough to finish in a reasonable amount of
time and didn't slow down or crash. Your settings may not look like this, but here's something to try.</span>
<div style="margin:0px; font-size:12pt"><br>
</div>
<div style="margin:0px; font-size:12pt">AppConfig[:pui_indexer_records_per_thread] = 50</div>
<span style="margin:0px; font-size:12pt">AppConfig[:pui_indexer_thread_count] = 1</span><br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_gmail-m_-8641307548135412358Signature">
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div></div>
<div></div>
<div id="x_gmail-m_-8641307548135412358divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<br>
</div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
So some extra detail for the mailing list archives... if your site keeps crashing before the indexers finish and you're not seeing any particular errors in the logs that make you think you have a problem with your data, try turning the knobs on your indexer
speed and see if that helps.</div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<br>
</div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
It looks like maybe the indexer just eats up too much memory on BIG records and having too many (too many being 15ish) threads running causes it to crash. I know BIG is pretty subjective, if you have a bunch of resources (maybe a few thousand) AND those resources
all have ALLOTA (maybe a few thousand) children with ALLOTA subjects/agents/notes/stuff, then you might hit this problem. Seems like it's not the total number of resources, it's probably because those resources are big/complex/deep. </div>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px">
<br>
</div>
</div>
</div>
</div>
<div id="x_gmail-m_-8641307548135412358appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-8641307548135412358divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> <<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Sent:</b> Thursday, March 18, 2021 11:24 AM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] PUI indexing issues</font>
<div> </div>
</div>
<div>
<div dir="ltr">Dave,
<div><br>
</div>
<div>Thanks for the suggestion, but unless there is some direct limitation within Solr, that should not be an issue. My disk is at only about 50% of capacity and Solr should be able to expand as needed. In my case, I don't think there has been much addition
to Solr because I'm reindexing records which have been indexed already. So the deleted records are growing, but not the overall number of records. My index is currently at about 6GB.</div>
<div><br>
</div>
<div>Any other thoughts out there? </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tom</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Mar 18, 2021 at 10:51 AM Mayo, Dave <<a href="mailto:dave_mayo@harvard.edu" target="_blank">dave_mayo@harvard.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">This is a little bit of a shot in the dark, but have you looked at disk space on whatever host Solr is resident on? (the ASpace server if you’re not running an external one)?<br>
<br>
A thing we’ve hit a couple times is that Solr, at least in some configurations, needs substantial headroom on disk to perform well – I think it’s related to how it builds and maintains the index? So it might be worth looking to see if Solr is filling up the
disk enough that it can’t efficiently handle itself.<u></u><u></u></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">--<u></u><u></u></p>
<p style="margin-top: 0px; margin-bottom: 0px;">Dave Mayo (he/him)<u></u><u></u></p>
</div>
<p style="margin-top: 0px; margin-bottom: 0px;">Senior Digital Library Software Engineer<br>
Harvard University > HUIT > LTS<u></u><u></u></p>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
<div style="border-right:none; border-bottom:none; border-left:none; border-top:1pt solid rgb(181,196,223); padding:3pt 0in 0in">
<p style="margin-top: 0px; margin-bottom: 0px;"><b><span style="font-size:12pt; color:black">From:
</span></b><span style="font-size:12pt; color:black"><<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Tom Hanstra <<a href="mailto:hanstra@nd.edu" target="_blank">hanstra@nd.edu</a>><br>
<b>Reply-To: </b>Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Date: </b>Wednesday, March 17, 2021 at 11:43 AM<br>
<b>To: </b>Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject: </b>Re: [Archivesspace_Users_Group] PUI indexing issues<u></u><u></u></span></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
<div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0in 0in 0in 6pt; margin-left:4.8pt; margin-right:0in">
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;margin-bottom:12pt"><u></u> <u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">- What really bothers me is the slowdown. That indicates to me that some resource is being lost along the way. Anyone have thoughts on what that might be?<u></u><u></u></p>
</div>
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">Just to follow up on my earlier post, I did get even lower numbers from Blake to try based upon what he used for our hosted account. But I'm seeing the same pattern in terms of slowdowns regarding the number of
records that get processed/hour. Is this typical? Is it just hitting records that have more work to be done? Or do I still have a resource issue.<u></u><u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">I note that the number of docs in Solr has not changed at all throughout the last couple of attempts, which again leads me to believe it has already handled these records (at least once) before and thus there
is no more indexing to really be done with the records which it is running through the PUI indexer again. Which leads back to the "why does PUI indexing restart each time from 0" question. How does one add an enhancement request to have this reviewed and (perhaps)
changed?<u></u><u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">Thanks,<u></u><u></u></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;">Tom<u></u><u></u></p>
</div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0in 0in 0in 6pt; margin-left:4.8pt; margin-right:0in">
<div>
<blockquote style="border-top:none; border-right:none; border-bottom:none; border-left:1pt solid rgb(204,204,204); padding:0in 0in 0in 6pt; margin-left:4.8pt; margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<p style="margin-top: 0px; margin-bottom: 0px;">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><b><span style="font-size:9.5pt; font-family:Arial,sans-serif; color:rgb(136,136,136)">Tom Hanstra</span></b><u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><i><span style="font-size:9.5pt; font-family:Arial,sans-serif; color:rgb(136,136,136)">Sr. Systems Administrator</span></i><span style="font-size:9.5pt; color:rgb(136,136,136)"><u></u><u></u></span></p>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:9.5pt; color:rgb(136,136,136)"><a href="mailto:hanstra@nd.edu" target="_blank"><span style="font-family:Arial,sans-serif; color:rgb(17,85,204)">hanstra@nd.edu</span></a><u></u><u></u></span></p>
</div>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:9.5pt; color:rgb(136,136,136)"><u></u> <u></u></span></p>
</div>
</div>
<div>
<p style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size:9.5pt; color:rgb(136,136,136)"><img border="0" id="x_gmail-m_-8641307548135412358x_gmail-m_710270332817758421_x0000_i1025" src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flyralists.lyrasis.org%2Fmailman%2Flistinfo%2Farchivesspace_users_group&data=04%7C01%7Cmark.custer%40yale.edu%7C7f1f33055593492d4ce008d8ed462e73%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637520233005628409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ADhPxo7qubWUyufL9kPnYAkrEY2QyEz9%2Fc%2F1q5VSnA8%3D&reserved=0" originalsrc="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" shash="j53SA6gu53SO2g3RscNrXrUVj+/MyW/1rcWx3Os6Vn4HNV4InJ2Qog+1ggD4RB4HFWYTLun1i2cDoj30QHUBGUSeqe8xM7SSPxjEdqjbD0m1RiKloxRPKfU+al0noxwWQ8arxvclsbOkiSZq3Pn281G5B/YXU3+aIuISrUzmp3M=" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif; font-size:12.7273px; color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136); font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px; font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" target="_blank" style="color:rgb(17,85,204); font-size:12.7273px; font-family:arial,helvetica,sans-serif">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flyralists.lyrasis.org%2Fmailman%2Flistinfo%2Farchivesspace_users_group&data=04%7C01%7Cmark.custer%40yale.edu%7C7f1f33055593492d4ce008d8ed462e73%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637520233005628409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ADhPxo7qubWUyufL9kPnYAkrEY2QyEz9%2Fc%2F1q5VSnA8%3D&reserved=0" originalsrc="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" shash="j53SA6gu53SO2g3RscNrXrUVj+/MyW/1rcWx3Os6Vn4HNV4InJ2Qog+1ggD4RB4HFWYTLun1i2cDoj30QHUBGUSeqe8xM7SSPxjEdqjbD0m1RiKloxRPKfU+al0noxwWQ8arxvclsbOkiSZq3Pn281G5B/YXU3+aIuISrUzmp3M=" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b style="font-family:arial,helvetica,sans-serif; font-size:12.7273px; color:rgb(136,136,136)">Tom Hanstra</b><br>
</div>
<div style="color:rgb(136,136,136); font-size:12.8px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.7273px">
<div>
<div><i style="font-size:12.7273px; font-family:arial,helvetica,sans-serif">Sr. Systems Administrator</i></div>
<div><a href="mailto:hanstra@nd.edu" target="_blank" style="color:rgb(17,85,204); font-size:12.7273px; font-family:arial,helvetica,sans-serif">hanstra@nd.edu</a><br>
</div>
</div>
<div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div>
</div>
<div style="font-size:12.7273px"><img src="https://docs.google.com/uc?export=download&id=1GFX1KaaMTtQ2Kg2u8bMXt1YwBp96bvf0&revid=0B7APN9POn6xAQ244WWFYMFU3aVJwZ0lxbmVHK3FxNXlCd0RRPQ"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>