[Archivesspace_Users_Group] External Solr - Memory Allocation?

[Brian Hoffman]

<< Why does one even have to run a periodic indexer? Aren't there guarantees in
the system that updates are seen through to the index in realtime, do bulk
updates not trigger a refresh of updated records selectively? Reading the code
seems to suggest that updates are queued until processed, does AS need a more
durable queue? >>

In theory, if your index is “up to date” (according to the indexer_state directory) the periodic indexer should have no work to do. I think this is part of a class of problems that arise when for some reason the periodic indexer cannot get through its workload and therefore tries and tries again. That is what happens, for example, when a MySQL database contains a record with a bad timestamp due to DST. If someone could file a JIRA issue with as much info as possible for recreating the problem (and maybe someone who could be contacted to supply a database copy) then it could probably be prioritized and addressed.

From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Peter Heiner <ph448 at cam.ac.uk>
Date: Saturday, January 28, 2023 at 3:51 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] External Solr - Memory Allocation?
Joshua D. Shaw wrote on 2023-01-26 23:02:10:
> Thanks, Blake!
>
> I'm running default config values for the AS log levels so they are all set to 'debug'. I took a closer look, and the timeout message happens exactly after the timeout amount I set (as you'd expect). Interestingly, Solr is in the middle of deleting documents when it goes silent
>
> I, [2023-01-26T09:18:40.357101 #78764]  INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x72b3d9e>
>
> .... 40 minutes pass with all the other AS log chatter ...
>
> E, [2023-01-26T09:58:40.400971 #78764] ERROR -- : Thread-3384: SolrIndexerError when deleting records: Timeout error with  POST {....}
> I, [2023-01-26T09:58:40.410522 #78764]  INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x4ab44e31>
>
> This continuing delete phase goes on for a bit until it stops logging batch deletes.
>
> I, [2023-01-26T09:59:11.734200 #78764]  INFO -- : Thread-3384: Deleted 9 documents: #<Net::HTTPOK:0x1be6c3e9>
>
> .... 40 minutes pass with all the other AS log chatter ... And then the commit error pops up
>
> E, [2023-01-26T10:39:11.746166 #78764] ERROR -- : Thread-3384: SolrIndexerError when committing:
> Timeout error with  POST {"commit":{"softCommit":false}}.
>
> Then after some more time
>
> I, [2023-01-26T11:06:35.678926 #78764]  INFO -- : Thread-3384: Deleted 186992 documents: #<Net::HTTPOK:0x7e298af9>
>
> .... This all seems to indicate to me that the commit phase is taking an inordinate amount of time (almost 2 hours - maybe that's what I need to set the timeout to?). After that, the indexer starts the 2nd repo

We're experiencing the exact same issue at least in the largest of our 30-odd
repositories:

I, [2023-01-28T07:24:48.015356 #2036632]  INFO -- : Thread-2006: Staff Indexer [2023-01-28 07:24:48 +0000] ~~~ Indexed 536300 of 587664 archival_object records in repository CUL
E, [2023-01-28T07:25:47.217953 #2036632] ERROR -- : Thread-2016: SolrIndexerError when committing:
Timeout error with  POST {"commit":{"softCommit":false}}.

We've had this problem from the start but were unable to dig deeper because
our in-house monitoring wasn't granular or even capable enough. Our crude
solution was using an external Solr since 2.5 and an external indexer since
around 2.7, and periodic restarts out of hours, but we've started getting
problems despite that. Our Solr is allocated 6GB of memory and timeout is set
to 1200 seconds, but the problem is that searches in AS fail during the wait
and that makes AS unusable, so we're reluctant to increase that.

Why does one even have to run a periodic indexer? Aren't there guarantees in
the system that updates are seen through to the index in realtime, do bulk
updates not trigger a refresh of updated records selectively? Reading the code
seems to suggest that updates are queued until processed, does AS need a more
durable queue?

Thanks,
p
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230128/f055edf9/attachment.html>