[Archivesspace_Users_Group] External Solr - Memory Allocation?

[Peter Heiner]

Joshua D. Shaw wrote on 2023-01-26 23:02:10:
> Thanks, Blake!
> 
> I'm running default config values for the AS log levels so they are all set to 'debug'. I took a closer look, and the timeout message happens exactly after the timeout amount I set (as you'd expect). Interestingly, Solr is in the middle of deleting documents when it goes silent
> 
> I, [2023-01-26T09:18:40.357101 #78764]  INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x72b3d9e>
> 
> .... 40 minutes pass with all the other AS log chatter ...
> 
> E, [2023-01-26T09:58:40.400971 #78764] ERROR -- : Thread-3384: SolrIndexerError when deleting records: Timeout error with  POST {....}
> I, [2023-01-26T09:58:40.410522 #78764]  INFO -- : Thread-3384: Deleted 100 documents: #<Net::HTTPOK:0x4ab44e31>
> 
> This continuing delete phase goes on for a bit until it stops logging batch deletes.
> 
> I, [2023-01-26T09:59:11.734200 #78764]  INFO -- : Thread-3384: Deleted 9 documents: #<Net::HTTPOK:0x1be6c3e9>
> 
> .... 40 minutes pass with all the other AS log chatter ... And then the commit error pops up
> 
> E, [2023-01-26T10:39:11.746166 #78764] ERROR -- : Thread-3384: SolrIndexerError when committing:
> Timeout error with  POST {"commit":{"softCommit":false}}.
> 
> Then after some more time
> 
> I, [2023-01-26T11:06:35.678926 #78764]  INFO -- : Thread-3384: Deleted 186992 documents: #<Net::HTTPOK:0x7e298af9>
> 
> .... This all seems to indicate to me that the commit phase is taking an inordinate amount of time (almost 2 hours - maybe that's what I need to set the timeout to?). After that, the indexer starts the 2nd repo

We're experiencing the exact same issue at least in the largest of our 30-odd
repositories:

I, [2023-01-28T07:24:48.015356 #2036632]  INFO -- : Thread-2006: Staff Indexer [2023-01-28 07:24:48 +0000] ~~~ Indexed 536300 of 587664 archival_object records in repository CUL
E, [2023-01-28T07:25:47.217953 #2036632] ERROR -- : Thread-2016: SolrIndexerError when committing: 
Timeout error with  POST {"commit":{"softCommit":false}}.

We've had this problem from the start but were unable to dig deeper because
our in-house monitoring wasn't granular or even capable enough. Our crude
solution was using an external Solr since 2.5 and an external indexer since
around 2.7, and periodic restarts out of hours, but we've started getting
problems despite that. Our Solr is allocated 6GB of memory and timeout is set
to 1200 seconds, but the problem is that searches in AS fail during the wait
and that makes AS unusable, so we're reluctant to increase that.

Why does one even have to run a periodic indexer? Aren't there guarantees in
the system that updates are seen through to the index in realtime, do bulk
updates not trigger a refresh of updated records selectively? Reading the code
seems to suggest that updates are queued until processed, does AS need a more
durable queue?

Thanks,
p