[Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3

[Dallas Pillen]

Hi all,

Just want to jump in to say that we have been experiencing the same issue
during a test migration to 1.5 RC3.

For context, we're migrating 2,910 EADs and in the process creating about
850,000 archival objects and 43,000 top containers. We have also not used
the barcoder plugin (though we have added temporary fake barcodes to top
containers in our EADs).

Dallas

On Wed, Jul 20, 2016 at 3:15 PM, Vandermillen, Michael <
michael_vandermillen at harvard.edu> wrote:

> Thanks Noah! Yes, it always happens in the middle of archival objects,
> then starts over. We never ran the barcode tool, so that’s not the issue,
> at least for us. Hopefully we can get more info from Lyrasis.
>
>
>
> Michael
>
>
>
> *From:* archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:
> archivesspace_users_group-bounces at lyralists.lyrasis.org] *On Behalf Of *Noah
> Huffman
> *Sent:* Wednesday, July 20, 2016 3:08 PM
> *To:* Archivesspace Users Group
> *Subject:* Re: [Archivesspace_Users_Group] out of memory errors during
> reindexing, 1.5 RC3
>
>
>
> Michael,
>
>
>
> I’ve experienced this exact problem when running test migrations to
> 1.5RC3.  Increasing the heap size never helped either.  From examining my
> log file (archivesspace.out), it looks like the re-indexing process runs
> out of memory (typically when indexing archival object records) and then
> just starts indexing at archival object #1 again.  It seems to continue in
> this loop indefinitely.  To successfully migrate to 1.5RC3, I had to stop
> and start the app several times until it finally indexed all the archival
> objects.
>
>
>
> No matter how high I set the heap size, the app seems to jump up to that
> limit shortly after starting it.
>
>
>
> I communicated this problem to Lyrasis and there was some suspicion it was
> caused by running the barcoder plugin over my entire repository.
>
>
>
> FWIW, we’ve also experience out of memory issues in our production
> instance during periodic indexing (not a complete re-index).  It seems like
> we have to restart the app every couple of weeks due to memory issues.
>
>
>
> I’m curious if others have experienced these same memory problems too.
> Thanks for raising the issue.
>
>
>
> -Noah
>
>
>
> ================
>
> Noah Huffman
>
> Archivist for Metadata, Systems, and Digital Records
>
> David M. Rubenstein Rare Book & Manuscript Library
>
> Duke University | 919-660-5982
>
> http://library.duke.edu/rubenstein/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__library.duke.edu_rubenstein_&d=CwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VrQdYlh6WQ9nrFC-sjqrXhpO3c_dIW1HO3Q189A7J80&m=X71gxo3lvFklEbvZRx8LX86CpdM-tILccnang2Hetx4&s=MrGtvBJTB3c-Xfhv3l51heHE0bRimo5uYXOjQCIoU2k&e=>
>
>
>
>
>
> *From:* archivesspace_users_group-bounces at lyralists.lyrasis.org [
> mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org
> <archivesspace_users_group-bounces at lyralists.lyrasis.org>] *On Behalf Of *Vandermillen,
> Michael
> *Sent:* Wednesday, July 20, 2016 2:42 PM
> *To:* Archivesspace Users Group (
> archivesspace_users_group at lyralists.lyrasis.org) <
> archivesspace_users_group at lyralists.lyrasis.org>
> *Subject:* [Archivesspace_Users_Group] out of memory errors during
> reindexing, 1.5 RC3
>
>
>
> Hi,
>
>
>
> As we import more and more finding aids into our archivesspace test
> instance, I get out of memory errors when trying to reindex (by deleting
> indexer_state and solr_index):
>
>
>
> SEVERE: auto commit error...:java.lang.IllegalStateException: this writer
> hit an OutOfMemoryError; cannot commit
>
>
>
> For 1.4x (and 1.5 RC 1), we had aspace heap set at the default (1024m) and
> could successfully reindex, after loading 4000+ finding aids. We have now
> loaded 5000+ finding aids, but even when I increase available heap to 8 gig
> (more doesn’t seem reasonable), I still get out of memory errors during a
> reindex (using 1.5 RC3).
>
>
>
> Has anyone else run into this? Any suggestions?
>
>
>
> Michael
>
>
>
> Michael Vandermillen
>
> Digital Library Software Engineer
>
> Harvard University IT
>
>
>
>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>


-- 

*Dallas Pillen*Assistant Archivist for Metadata and Digital Projects


  Bentley Historical Library <http://bentley.umich.edu/>
  1150 Beal Avenue
  Ann Arbor, Michigan 48109-2113
  734.647.3559
  Twitter <https://twitter.com/umichBentley> Facebook
<https://www.facebook.com/bentleyhistoricallibrary>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20160720/98c6d66b/attachment.html>