<div dir="ltr">Just wanted to confirm Mark Cooper's note from earlier in the thread. Deleting and rebuilding the index is recommended <a href="https://groups.google.com/forum/#!topic/archivesspace/uqDXPUv-yCo">here</a>.<div><br></div><div>As a result, I ran a couple of test migrations to 1.5 yesterday (~275,000 records) with no issues.</div><div><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div><br></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Jul 21, 2016 at 8:19 PM, Kevin Clair <span dir="ltr"><<a href="mailto:Kevin.Clair@du.edu" target="_blank">Kevin.Clair@du.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri">To contribute another data point, this happens with ours as well, but we only have 130,000 archival objects in our database so typically it will finish indexing those but then run out of
memory when it moves on to the digital objects. Upon restarting AS once the index will finish normally. Our Solr index has a little over 200,000 documents, which sounds perhaps like it’s the lower limit at which 1.5 starts to have memory issues with the initial
index. I haven’t been copying over the /data directory at all when doing test upgrades. -k<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>> on behalf of Patrick Galligan <<a href="mailto:PGalligan@rockarch.org" target="_blank">PGalligan@rockarch.org</a>><br>
<b>Reply-To: </b>Archivesspace Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Date: </b>Thursday, July 21, 2016 at 1:48 PM<br>
<b>To: </b>Archivesspace Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject: </b>Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">Hi all,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">I’ve just tried to index again, with no luck. Same error that we’re running out of memory, despite high memory allocations.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">This is a pretty big blocker for us. We were hoping to migrate to 1.5 in early August, but we obviously can’t if the index never finishes successfully.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">Patrick Galligan</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">Rockefeller Archive Center</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)">Assistant Digital Archivist</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)"><a href="tel:914-366-6386" value="+19143666386" target="_blank">914-366-6386</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma">From:</span></b><span style="font-size:10pt;font-family:Tahoma"> <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a> [mailto:<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>]
<b>On Behalf Of </b>Dallas Pillen<br>
<b>Sent:</b> Wednesday, July 20, 2016 3:50 PM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3</span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Just want to jump in to say that we have been experiencing the same issue during a test migration to 1.5 RC3. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">For context, we're migrating 2,910 EADs and in the process creating about 850,000 archival objects and 43,000 top containers. We have also not used the barcoder plugin (though we have added temporary fake barcodes to top containers in our
EADs). <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Dallas<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Jul 20, 2016 at 3:15 PM, Vandermillen, Michael <<a href="mailto:michael_vandermillen@harvard.edu" target="_blank">michael_vandermillen@harvard.edu</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Thanks Noah! Yes, it always happens in the middle of archival objects, then starts over. We never ran the barcode tool, so that’s not the issue, at least
for us. Hopefully we can get more info from Lyrasis.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Michael</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> [mailto:<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>]
<b>On Behalf Of </b>Noah Huffman<br>
<b>Sent:</b> Wednesday, July 20, 2016 3:08 PM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Michael,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">I’ve experienced this exact problem when running test migrations to 1.5RC3. Increasing the heap size never helped either. From examining my log file
(archivesspace.out), it looks like the re-indexing process runs out of memory (typically when indexing archival object records) and then just starts indexing at archival object #1 again. It seems to continue in this loop indefinitely. To successfully migrate
to 1.5RC3, I had to stop and start the app several times until it finally indexed all the archival objects.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">No matter how high I set the heap size, the app seems to jump up to that limit shortly after starting it.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">I communicated this problem to Lyrasis and there was some suspicion it was caused by running the barcoder plugin over my entire repository.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">FWIW, we’ve also experience out of memory issues in our production instance during periodic indexing (not a complete re-index). It seems like we have
to restart the app every couple of weeks due to memory issues.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">I’m curious if others have experienced these same memory problems too. Thanks for raising the issue.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">-Noah</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)">================</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)">Noah Huffman</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)">Archivist for Metadata, Systems, and Digital Records</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)">David M. Rubenstein Rare Book & Manuscript Library</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)">Duke University |
<a href="tel:919-660-5982" target="_blank">919-660-5982</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial Narrow";color:rgb(64,64,64)"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__library.duke.edu_rubenstein_&d=CwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VrQdYlh6WQ9nrFC-sjqrXhpO3c_dIW1HO3Q189A7J80&m=X71gxo3lvFklEbvZRx8LX86CpdM-tILccnang2Hetx4&s=MrGtvBJTB3c-Xfhv3l51heHE0bRimo5uYXOjQCIoU2k&e=" target="_blank"><span style="color:rgb(64,64,64)">http://library.duke.edu/rubenstein/</span></a>
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b>
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">
archivesspace_users_group-bounces@lyralists.lyrasis.org</a> [<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org</a>]
<b>On Behalf Of </b>Vandermillen, Michael<br>
<b>Sent:</b> Wednesday, July 20, 2016 2:42 PM<br>
<b>To:</b> Archivesspace Users Group (<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>) <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@lyralists.lyrasis.org</a>><br>
<b>Subject:</b> [Archivesspace_Users_Group] out of memory errors during reindexing, 1.5 RC3<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">As we import more and more finding aids into our archivesspace test instance, I get out of memory errors when trying to reindex (by deleting indexer_state and solr_index):<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">SEVERE: auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">For 1.4x (and 1.5 RC 1), we had aspace heap set at the default (1024m) and could successfully reindex, after loading 4000+ finding aids. We have now loaded 5000+ finding aids, but
even when I increase available heap to 8 gig (more doesn’t seem reasonable), I still get out of memory errors during a reindex (using 1.5 RC3).<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Has anyone else run into this? Any suggestions?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Michael<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Michael Vandermillen<u></u><u></u></p>
<p class="MsoNormal">Digital Library Software Engineer<u></u><u></u></p>
<p class="MsoNormal">Harvard University IT<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.5pt">Dallas Pillen<br>
</span></b><span style="font-size:9.5pt">Assistant Archivist for Metadata and Digital Projects</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;color:black"><img border="0" width="351" height="40" src="https://webapps.lsa.umich.edu/dean/lsa_emails/bentley-sig-em.png"></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:black"> <a href="http://bentley.umich.edu/" target="_blank"><span style="color:rgb(17,85,204)">Bentley Historical Library</span></a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:black"> 1150 Beal Avenue</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:black"> Ann Arbor, Michigan 48109-2113</span><span style="font-size:10pt;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;color:black"> </span><span style="font-size:9.5pt"><a href="tel:734.647.3559" value="+17346473559" target="_blank">734.647.3559</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:black"> <a href="https://twitter.com/umichBentley" target="_blank"><span style="color:rgb(17,85,204)">Twitter</span></a> <a href="https://www.facebook.com/bentleyhistoricallibrary" target="_blank"><span style="color:rgb(17,85,204)">Facebook </span></a></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
<br></blockquote></div><br></div></div></div>