<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Truncating the "deleted_records" table will prevent the OAI-PMH
service from being able to send out deletion notifications. Maybe
it is worth trying on a testing system, but probably not a good
idea on a production system.</p>
<p><br>
</p>
<p>Also note that is this...</p>
<p><br>
</p>
<p>> Deleted 186992 documents</p>
<p><br>
</p>
<p>... is not logging the deletion of deleted records. It is logging
the deletion of Solr documents for unpublished records. But few,
if any, exist to be deleted. ArchivesSpace sent 186992 IDs to Solr
to delete, just in case any of them were unpublished immediately
before this index run. Solr returned a 200 OK response, even if
none were found, so ArchivesSpace reports them as all deleted.<br>
</p>
<p><br>
</p>
<p>The other logged deletions, before that, are likewise not "real"
99.9% of the time. These batches...</p>
<p><br>
</p>
<p>> Deleted 100 documents<br>
</p>
<p><br>
</p>
<p>...are when ArchivesSpace tells Solr to delete the tree nodes for
all archival objects without children, just in case any of them
had children before this index run. Only the PUIIndexer does this,
which is part of why it is slower (it is also allocated fewer
threads in default config.) Again, 99.99% of the time there's
nothing for Solr to delete, but it has to search its indexes for
them anyway.<br>
</p>
<p><br>
</p>
<p>And these...</p>
<p><br>
</p>
<p>> Deleted 25 documents<br>
</p>
<p><br>
</p>
<p>...are the deletion of URIs in the deleted_records table. But
most of those were deleted long ago by previous index runs. Again,
it is just in case any new ones were recently deleted (and even
those were probably deleted by the RealtimeIndexer.)<br>
</p>
<p><br>
</p>
<p>This "belt and braces" approach prevents a few stray records
remaining in the PUI when they've been deleted or unpublished, but
it seems to be the cause of the longest wait times for commits
when re-indexing large repositories. Maybe something has changed
in newer versions of Solr to make this process slower, possibly
specifically for deletions?<br>
</p>
<br>
<p>Andrew.</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p></p>
<div class="moz-cite-prefix">On 27/01/2023 14:01, Blake Carver
wrote:<br>
</div>
<blockquote type="cite" cite="mid:DM6PR22MB23093DD35E020B3FCEE436FE9FCC9@DM6PR22MB2309.namprd22.prod.outlook.com">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div class="elementToProof"><span style="font-family: Calibri,
Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0,
0, 0); background-color: rgb(255, 255, 255);"></span></div>
> I'm running default config values for the AS log levels so
they are all set to 'debug'.
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">I'm seeing "INFO" and
not "DEBUG" there.
</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">> Deleted 186992
documents</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">How much is in the
deleted_records table? Try truncating that. ArchivesSpace is
going and deleting anything in that table.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">> So I'm falling
back to this just being super slow for some reason.
</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">Could be some complex
records, could be there's way too much in the deleted table.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">> but I'm not sure
why the PUI indexer would become so much slower (21 hours)
</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0 elementToProof">Yep,
sounds about right. The PUI is slow.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">> We do have some
collections that are quite large (10s of thousands of AOs), so
maybe that's part of the issue.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
No doubt that's slowing it down too.
<div class="elementToProof">
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt;
background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt;
background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org"><archivesspace_users_group-bounces@lyralists.lyrasis.org></a>
on behalf of Joshua D. Shaw
<a class="moz-txt-link-rfc2396E" href="mailto:Joshua.D.Shaw@dartmouth.edu"><Joshua.D.Shaw@dartmouth.edu></a><br>
<b>Sent:</b> Thursday, January 26, 2023 6:02 PM<br>
<b>To:</b> Archivesspace Users Group
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group@lyralists.lyrasis.org"><archivesspace_users_group@lyralists.lyrasis.org></a><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] External Solr
- Memory Allocation?</font>
<div> </div>
</div>
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div dir="ltr">
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Thanks, Blake! <br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
I'm running default config values for the AS log levels so
they are all set to 'debug'. I took a closer look, and the
timeout message happens exactly after the timeout amount I set
(as you'd expect). Interestingly, Solr is in the middle of
deleting documents when it goes silent<br>
<br>
<span style="font-family:"Courier New",monospace">I,
[2023-01-26T09:18:40.357101 #78764] INFO -- : Thread-3384:
Deleted 100 documents: #<a class="moz-txt-link-rfc2396E" href="Net::HTTPOK:0x72b3d9e"><Net::HTTPOK:0x72b3d9e></a></span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
.... 40 minutes pass with all the other AS log chatter ...</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<span style="font-family:"Courier New",monospace">E,
[2023-01-26T09:58:40.400971 #78764] ERROR -- : Thread-3384:
SolrIndexerError when deleting records: Timeout error with
POST {....}</span><br class="x_ContentPasted1">
<span style="font-family:"Courier New",monospace">I,
[2023-01-26T09:58:40.410522 #78764] INFO -- : Thread-3384:
Deleted 100 documents: #<a class="moz-txt-link-rfc2396E" href="Net::HTTPOK:0x4ab44e31"><Net::HTTPOK:0x4ab44e31></a></span></div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
This continuing delete phase goes on for a bit until it stops
logging batch deletes.
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted5
x_ContentPasted6" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<span style="font-family:"Courier New",monospace">I,
[2023-01-26T09:59:11.734200 #78764] INFO -- : Thread-3384:
Deleted 9 documents: #<a class="moz-txt-link-rfc2396E" href="Net::HTTPOK:0x1be6c3e9"><Net::HTTPOK:0x1be6c3e9></a></span></div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted5
x_ContentPasted6" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted5
x_ContentPasted6 x_ContentPasted11" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
.... 40 minutes pass with all the other AS log chatter ... And
then the commit error pops up<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted5
x_ContentPasted6" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<span style="font-family:"Courier New",monospace">E,
[2023-01-26T10:39:11.746166 #78764] ERROR -- : Thread-3384:
SolrIndexerError when committing:
</span>
<div class="x_ContentPasted4"><span style="font-family:"Courier New",monospace">Timeout
error with POST {"commit":{"softCommit":false}}.</span></div>
<div class="x_ContentPasted4"><br>
</div>
<div class="x_ContentPasted4">Then after some more time</div>
<div class="x_ContentPasted4"><br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7"><span style="font-family:"Courier New",monospace">I,
[2023-01-26T11:06:35.678926 #78764] INFO -- :
Thread-3384: Deleted 186992 documents:
#<a class="moz-txt-link-rfc2396E" href="Net::HTTPOK:0x7e298af9"><Net::HTTPOK:0x7e298af9></a></span><br class="x_ContentPasted7">
</div>
<div class="x_ContentPasted4 x_ContentPasted7"><br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7">.... This all
seems to indicate to me that the commit phase is taking an
inordinate amount of time (almost 2 hours - maybe that's
what I need to set the timeout to?). After that, the indexer
starts the 2nd repo<br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7"><br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7
x_ContentPasted8"><span style="font-family:"Courier
New",monospace">I, [2023-01-26T11:06:35.765797
#78764] INFO -- : Thread-3384: PUI Indexer [2023-01-26
11:06:35 -0500] Indexed 2 additional PUI records in
repository Sherman</span><br class="x_ContentPasted8">
</div>
<div class="x_ContentPasted4 x_ContentPasted7
x_ContentPasted8"><br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7
x_ContentPasted8">.... The indexer waits for a looong time
with no timeout and no messaging - even though this is a
tiny repo - and then starts the 3rd repo<br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7
x_ContentPasted8"><br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7 x_ContentPasted8
x_ContentPasted9">
<span style="font-family:"Courier New",monospace">I,
[2023-01-26T11:31:32.795602 #78764] INFO -- :
Thread-3384: PUI Indexer [2023-01-26 11:31:32 -0500]
Indexed 188 additional PUI records in repository Rauner-XO</span><br class="x_ContentPasted9">
</div>
<div class="x_ContentPasted4 x_ContentPasted7 x_ContentPasted8
x_ContentPasted9">
<br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7 x_ContentPasted8
x_ContentPasted9">
And then the indexer starts the 4th repo soon after and
seems to go on to complete normally<br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7 x_ContentPasted8
x_ContentPasted9">
<br>
</div>
<div class="x_ContentPasted4 x_ContentPasted7 x_ContentPasted8
x_ContentPasted9 x_ContentPasted10">
<span style="font-family:"Courier New",monospace">I,
[2023-01-26T11:31:33.369369 #78764] INFO -- :
Thread-3384: PUI Indexer [2023-01-26 11:31:33 -0500] ~~~
Indexed 25 of 74785 archival_object records in repository
thedartmouth</span><br class="x_ContentPasted10">
<br>
</div>
The Solr logs indicate that Solr is working this entire time
doing adds and deletes. For example in one of the quiet
phases:</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<span style="font-family:"Courier New",monospace">2023-01-26
10:23:35.928 INFO (qtp2101153819-523) [ x:archivesspace]
o.a.s.u.p.LogUpdateProcessorFactory [archivesspace]
webapp=/solr path=/update params={}{add=...</span></div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<span style="font-family:"Courier New",monospace">2023-01-26
10:23:38.195 INFO (qtp2101153819-468) [ x:archivesspace]
o.a.s.u.p.LogUpdateProcessorFactory [archivesspace]
webapp=/solr path=/update params={}{deleteByQuery=...</span></div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
So I'm falling back to this just being <b>super</b> slow for
some reason. I do have some custom indexer addons, but I'm not
sure why the PUI indexer would become
<b>so</b> much slower (21 hours) when the Staff indexer
completes in a normal amount of time (a little under 6 hours).
For previous versions this hasn't been quite that different
(6hrs vs about 13hrs). We do have some collections that are
quite large (10s of thousands of AOs), so maybe that's part of
the issue.<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
I haven't checked to see if the PUI indexer is gathering that
much more data (and traversing the tree more times - maybe?)
than it was in 3.1.1, but that's on my 'to check' list.<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0 x_ContentPasted1
x_ContentPasted2 x_ContentPasted3 x_ContentPasted4
x_ContentPasted12 x_ContentPasted13" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Joshua<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org"><archivesspace_users_group-bounces@lyralists.lyrasis.org></a>
on behalf of Blake Carver <a class="moz-txt-link-rfc2396E" href="mailto:blake.carver@lyrasis.org"><blake.carver@lyrasis.org></a><br>
<b>Sent:</b> Thursday, January 26, 2023 4:12 PM<br>
<b>To:</b> Archivesspace Users Group
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group@lyralists.lyrasis.org"><archivesspace_users_group@lyralists.lyrasis.org></a><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] External
Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_x_elementToProof">That's... interesting. </div>
<div class="x_x_elementToProof"><br>
</div>
<div class="x_x_elementToProof">That RAM allocation seems
fine. That Solr timeout is way higher than I would think is
needed.</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Maybe set the loglevel to debug and see if it spits out
something more useful? Maybe you'll be able to see what it's
up to during that looooong time. I like your theory on
that. </div>
<div class="x_x_elementToProof">
<div id="x_x_Signature">
<div>
<div id="x_x_divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
<div name="x_x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; margin:0px">
<font size="3" face="Calibri,Arial,Helvetica,sans-serif" color="black"><span dir="ltr" style="font-size:12pt; background-color:white"><font size="2"><span style="font-size:11pt"><br>
</span></font></span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org"><archivesspace_users_group-bounces@lyralists.lyrasis.org></a>
on behalf of Joshua D. Shaw
<a class="moz-txt-link-rfc2396E" href="mailto:Joshua.D.Shaw@dartmouth.edu"><Joshua.D.Shaw@dartmouth.edu></a><br>
<b>Sent:</b> Thursday, January 26, 2023 3:38 PM<br>
<b>To:</b> Archivesspace Users Group
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group@lyralists.lyrasis.org"><archivesspace_users_group@lyralists.lyrasis.org></a><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] External
Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Following up on this. And looking for some advice!<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Even with the Solr timeout set to 40 minutes, I'm seeing
some random Solr Timeout errors, though these do
<b>NOT</b> cause the indexer to restart. In the latest
test run I see one Solr Timeout for delete and one for
commit - both following the PUI indexer run for AOs for
the first and largest repo (~630k AOs).<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
The indexer throws the delete timeout error, waits for a
loooong time with seemingly no activity, throws the commit
timeout error, waits again, and then picks back up as if
nothing had gone wrong and continues with the initial
index run. All of the index data looks correct (ie correct
number of objects in both the staff and PUI).</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
My theory is that the Solr update phase really is taking a
super loooong time, but that the data has all been sent to
Solr so the timeouts are really just ArchivesSpace waiting
for Solr in between indexing one object type and the next
and no index data is lost.<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
There are no corresponding log entries in the Solr logs
that I can find.</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
I'm running solr 8.11.6 with 4GB and AS 3.3.1 with 4GB.
Both bare metal on my laptop, so no container issues that
might be at play. Solr memory use peaks at around 3.5GB.<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
I've kept the stock thread and records per thread settings
and just upped the timeout (to 2400). I guess the next
step is to set the timeout even higher - maybe an hour
(3600)? I don't see a reason to run a lower thread or
record count, but can certainly try that as well, though
I'm not looking forward to the time it will take (the
current run takes 21 hours as it is - up from about 15 for
3.1.1)</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Any advice appreciated!<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Thanks!</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Joshua<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org"><archivesspace_users_group-bounces@lyralists.lyrasis.org></a> on
behalf of Joshua D. Shaw
<a class="moz-txt-link-rfc2396E" href="mailto:Joshua.D.Shaw@dartmouth.edu"><Joshua.D.Shaw@dartmouth.edu></a><br>
<b>Sent:</b> Tuesday, January 24, 2023 6:56 AM<br>
<b>To:</b> Archivesspace Users Group
<a class="moz-txt-link-rfc2396E" href="mailto:archivesspace_users_group@lyralists.lyrasis.org"><archivesspace_users_group@lyralists.lyrasis.org></a><br>
<b>Subject:</b> [Archivesspace_Users_Group] External
Solr - Memory Allocation?</font>
<div> </div>
</div>
<div dir="ltr">
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Hey all</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
We're about to jump to v3.3.1 and I'm wondering if
anyone has any suggestions for memory allocation for
Solr?
<br>
</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Currently we're running 6GB for the entire suite in
v3.1.1 and are looking to keep the same overall memory
footprint. Wondering if something like a 75/25 split (ie
4GB for AS and 2GB for Solr) would be a reasonable
allocation? Or are people finding that Solr is more
demanding?</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<br>
</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Thanks!</div>
<div class="x_x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
Joshua<br>
</div>
<div class="x_x_x_x_elementToProof">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_x_x_Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
___________________</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Joshua Shaw (he, him)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Library Web & Application Developer<br>
</div>
<span style="font-family:Calibri,Helvetica,sans-serif">Digital
Library Technologies Group</span><br>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Dartmouth College<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
603.646.0405<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Archivesspace_Users_Group mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a>
<a class="moz-txt-link-freetext" href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a>
</pre>
</blockquote>
</body>
</html>