[Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Fri Jul 22 02:37:06 EDT 2016

[apologies for cross-posting]

I haven't personally seen many out of memory errors during an upgrade to v1.5.x*, but my testing has always been with "optimal" resources -- plenty of available cpu, ram, disk, swap etc. with a well tuned configuration.

So I pulled a largish database (600k archival objects) to my workstation for testing this evening. Applying some minimal config I was able to complete the upgrade. I let things run for a while and didn't observe any issues.

Given that top container conversion and solr reindexing are costly operations for larger databases I suggest trying to break this into two steps to see if it helps:

Before the upgrade set enable_indexer and enable_solr to false (you can also do this for frontend and public). Delete any existing Solr index.

Start the upgrade. It will complete and then ... nothing =) The other apps are disabled, but you have an upgraded database. Stop ArchivesSpace and restart MySQL.

Re-enable the other apps in your config file. Start ArchivesSpace again, it will skip container conversion and now indexing will kick-in (without the cumulative impact of conversion + indexing on the database and system resources). Results are different / better?

* I have seen the OOM (out of memory) error twice and neither time was it related to insufficient hardware or configuration, but was actually due to something in the data.

One time we isolated the issue to the barcoder plugin run over a repository (though we didn't have the time to dig deeper) -- without barcoder and generating the barcodes a different way the upgrade could complete.

The other time there was an instance in the database that was associated with a non-existent archival object (possibly from a migration). This created a failure state during indexing which led to looping (and eventual out of memory errors). Deleting the problem instance restored things to normality. I suspect something like this is more likely in play than hardware or configuration, particularly if those things haven't been an issue before.

Hope there's something in there that's helpful to someone,

Best,
Mark

Mark Cooper
Technical Lead, Hosting and Support
LYRASIS
email: mark.cooper at lyrasis.org
skype: mark_c_cooper
________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Jason Loeffler <j at minorscience.com>
Sent: Thursday, July 21, 2016 5:53:38 PM
To: Archivesspace Users Group
Subject: Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Just wanted to confirm Mark Cooper's note from earlier in the thread. Deleting and rebuilding the index is recommended here<https://groups.google.com/forum/#!topic/archivesspace/uqDXPUv-yCo>.

As a result, I ran a couple of test migrations to 1.5 yesterday (~275,000 records) with no issues.

On Thu, Jul 21, 2016 at 8:19 PM, Kevin Clair <Kevin.Clair at du.edu<mailto:Kevin.Clair at du.edu>> wrote:
To contribute another data point, this happens with ours as well, but we only have 130,000 archival objects in our database so typically it will finish indexing those but then run out of memory when it moves on to the digital objects. Upon restarting AS once the index will finish normally. Our Solr index has a little over 200,000 documents, which sounds perhaps like it’s the lower limit at which 1.5 starts to have memory issues with the initial index. I haven’t been copying over the /data directory at all when doing test upgrades.  -k

From: <archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> on behalf of Patrick Galligan <PGalligan at rockarch.org<mailto:PGalligan at rockarch.org>>
Reply-To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Date: Thursday, July 21, 2016 at 1:48 PM
To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Hi all,

I’ve just tried to index again, with no luck. Same error that we’re running out of memory, despite high memory allocations.

This is a pretty big blocker for us. We were hoping to migrate to 1.5 in early August, but we obviously can’t if the index never finishes successfully.

Patrick Galligan
Rockefeller Archive Center
Assistant Digital Archivist
914-366-6386<tel:914-366-6386>

From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>] On Behalf Of Dallas Pillen
Sent: Wednesday, July 20, 2016 3:50 PM
To: Archivesspace Users Group
Subject: Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Hi all,

Just want to jump in to say that we have been experiencing the same issue during a test migration to 1.5 RC3.

For context, we're migrating 2,910 EADs and in the process creating about 850,000 archival objects and 43,000 top containers. We have also not used the barcoder plugin (though we have added temporary fake barcodes to top containers in our EADs).

Dallas

On Wed, Jul 20, 2016 at 3:15 PM, Vandermillen, Michael <michael_vandermillen at harvard.edu<mailto:michael_vandermillen at harvard.edu>> wrote:
Thanks Noah! Yes, it always happens in the middle of archival objects, then starts over. We never ran the barcode tool, so that’s not the issue, at least for us. Hopefully we can get more info from Lyrasis.

Michael

From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>] On Behalf Of Noah Huffman
Sent: Wednesday, July 20, 2016 3:08 PM
To: Archivesspace Users Group
Subject: Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Michael,

I’ve experienced this exact problem when running test migrations to 1.5RC3.  Increasing the heap size never helped either.  From examining my log file (archivesspace.out), it looks like the re-indexing process runs out of memory (typically when indexing archival object records) and then just starts indexing at archival object #1 again.  It seems to continue in this loop indefinitely.  To successfully migrate to 1.5RC3, I had to stop and start the app several times until it finally indexed all the archival objects.

No matter how high I set the heap size, the app seems to jump up to that limit shortly after starting it.

I communicated this problem to Lyrasis and there was some suspicion it was caused by running the barcoder plugin over my entire repository.

FWIW, we’ve also experience out of memory issues in our production instance during periodic indexing (not a complete re-index).  It seems like we have to restart the app every couple of weeks due to memory issues.

I’m curious if others have experienced these same memory problems too.  Thanks for raising the issue.

-Noah

================
Noah Huffman
Archivist for Metadata, Systems, and Digital Records
David M. Rubenstein Rare Book & Manuscript Library
Duke University | 919-660-5982<tel:919-660-5982>
http://library.duke.edu/rubenstein/<https://urldefense.proofpoint.com/v2/url?u=http-3A__library.duke.edu_rubenstein_&d=CwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VrQdYlh6WQ9nrFC-sjqrXhpO3c_dIW1HO3Q189A7J80&m=X71gxo3lvFklEbvZRx8LX86CpdM-tILccnang2Hetx4&s=MrGtvBJTB3c-Xfhv3l51heHE0bRimo5uYXOjQCIoU2k&e=>

From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Vandermillen, Michael
Sent: Wednesday, July 20, 2016 2:42 PM
To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>) <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

Hi,

As we import more and more finding aids into our archivesspace test instance, I get out of memory errors when trying to reindex (by deleting indexer_state and solr_index):

SEVERE: auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit

For 1.4x (and 1.5 RC 1), we had aspace heap set at the default (1024m) and could successfully reindex, after loading 4000+ finding aids. We have now loaded 5000+ finding aids, but even when I increase available heap to 8 gig (more doesn’t seem reasonable), I still get out of memory errors during a reindex (using 1.5 RC3).

Has anyone else run into this? Any suggestions?

Michael

Michael Vandermillen
Digital Library Software Engineer
Harvard University IT

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

--
Dallas Pillen
Assistant Archivist for Metadata and Digital Projects

[https://webapps.lsa.umich.edu/dean/lsa_emails/bentley-sig-em.png]
  Bentley Historical Library<http://bentley.umich.edu/>
  1150 Beal Avenue
  Ann Arbor, Michigan 48109-2113
  734.647.3559<tel:734.647.3559>
  Twitter<https://twitter.com/umichBentley> Facebook <https://www.facebook.com/bentleyhistoricallibrary>

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20160722/3388329b/attachment.html>