[Archivesspace_Users_Group] External Solr - Memory Allocation?

[Blake Carver]

That's... interesting.

That RAM allocation seems fine. That Solr timeout is way higher than I would think is needed.

Maybe set the loglevel to debug and see if it spits out something more useful? Maybe you'll be able to see what it's up to during that looooong time. I like your theory on that.


________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Joshua D. Shaw <Joshua.D.Shaw at dartmouth.edu>
Sent: Thursday, January 26, 2023 3:38 PM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] External Solr - Memory Allocation?

Following up on this. And looking for some advice!

Even with the Solr timeout set to 40 minutes, I'm seeing some random Solr Timeout errors, though these do NOT cause the indexer to restart. In the latest test run I see one Solr Timeout for delete and one for commit - both following the PUI indexer run for AOs for the first and largest repo (~630k AOs).

The indexer throws the delete timeout error, waits for a loooong time with seemingly no activity, throws the commit timeout error, waits again, and then picks back up as if nothing had gone wrong and continues with the initial index run. All of the index data looks correct (ie correct number of objects in both the staff and PUI).

My theory is that the Solr update phase really is taking a super loooong time, but that the data has all been sent to Solr so the timeouts are really just ArchivesSpace waiting for Solr in between indexing one object type and the next and no index data is lost.

There are no corresponding log entries in the Solr logs that I can find.

I'm running solr 8.11.6 with 4GB and AS 3.3.1 with 4GB. Both bare metal on my laptop, so no container issues that might be at play. Solr memory use peaks at around 3.5GB.

I've kept the stock thread and records per thread settings and just upped the timeout (to 2400). I guess the next step is to set the timeout even higher - maybe an hour (3600)? I don't see a reason to run a lower thread or record count, but can certainly try that as well, though I'm not looking forward to the time it will take (the current run takes 21 hours as it is - up from about 15 for 3.1.1)

Any advice appreciated!

Thanks!
Joshua

________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Joshua D. Shaw <Joshua.D.Shaw at dartmouth.edu>
Sent: Tuesday, January 24, 2023 6:56 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: [Archivesspace_Users_Group] External Solr - Memory Allocation?

Hey all

We're about to jump to v3.3.1 and I'm wondering if anyone has any suggestions for memory allocation for Solr?

Currently we're running 6GB for the entire suite in v3.1.1 and are looking to keep the same overall memory footprint. Wondering if something like a 75/25 split (ie 4GB for AS and 2GB for Solr) would be a reasonable allocation? Or are people finding that Solr is more demanding?

Thanks!
Joshua

___________________
Joshua Shaw (he, him)
Library Web & Application Developer
Digital Library Technologies Group
Dartmouth College
603.646.0405
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230126/cc8c77dc/attachment.html>