<div dir="ltr"><div class="gmail_extra">Thanks so much, James. Can you tell me if there's a script available for generating an arbitrary number of test records? Something similar to Drupal's devel <a href="https://api.drupal.org/api/devel/functions/8.x-1.x">generate</a> hooks?</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">Jason Loeffler</font></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">Technology Consultant | </font><span style="color:rgb(102,102,102);font-family:garamond,serif">The American Academy in Rome</span></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">Minor Science | Application Development & Metadata Strategy</font></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">Brooklyn, New York</font></div></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666"><a href="mailto:jason@minorscience.com" target="_blank">jason@minorscience.com</a></font></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">(347) 405-0826</font></div><div><font size="2" face="garamond, serif" style="background-color:rgb(255,255,255)" color="#666666">minorscience (Skype)</font></div><div><br><br></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Nov 15, 2016 at 6:17 PM, James Bullen <span dir="ltr"><<a href="mailto:james@hudmol.com" target="_blank">james@hudmol.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Hi all,</div><div><br></div><div>As part of our new role as ArchivesSpace development partner, we will be addressing issues like this as a priority.</div><div><br></div><div>It is still early days and we’re working with the folks at the ArchivesSpace program on a release schedule, so we can’t make any definitive statements yet, but please know that we are aware of this issue and will address it as soon as we are able.</div><div><br></div><div><br></div><div>Cheers,</div><div>James</div><div><br></div><div><br></div><div>—</div><div>James Bullen</div><div>Hudson Molonglo</div><div><br></div><div><br></div><br><div><blockquote type="cite"><div>On Nov 16, 2016, at 7:46 AM, Joshua D. Shaw <<a href="mailto:Joshua.D.Shaw@dartmouth.edu" target="_blank">Joshua.D.Shaw@dartmouth.edu</a>> wrote:</div><br class="m_-1845926206139756334Apple-interchange-newline"><div><div class="m_-1845926206139756334WordSection1" style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">Hi all –<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">We at Dartmouth have experienced similar issues. We have some large resources as well (one has 60K+ objects in the tree) and anything that involves a save or rearrangement (moving a file around, etc) can take a *<b>lot</b>* of time (many minutes) and may cause an error – typically of the “another user is modifying this record” type.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">If we have to do any modifications to a resource of that size, we a) budget a lot of time and b) do things in small increments – ie don’t move more than a couple of files around at a time. It’s not a great solution, but it does minimize some of the headache.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">I *<b>think</b>* (but haven’t had the time to really dig into this) that one reason the error comes about is because the indexer steps on/collides with the process that the save/arrangement kicked off. We are still running 1.3 and hope that some of our issues will be mitigated when we move to 1.5.1, though we know that not all of them have been resolved yet.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">One other data point is that we’ve got a plugin that runs as a background job doing a bunch of importing. This background job touches some of the larger resources, but does *<b>not</b>* cause the errors and long save times, which leads me to believe that a lot of the problem is in the frontend – perhaps with the way the tree is populated - as Jason pointed out.<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">Best,<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri">Joshua<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:11pt;font-family:Calibri"><u></u> <u></u></span></div><div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><b><span style="font-family:Calibri">From:<span class="m_-1845926206139756334Apple-converted-space"> </span></span></b><span style="font-family:Calibri"><<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" style="color:purple;text-decoration:underline" target="_blank">archivesspace_users_<wbr>group-bounces@lyralists.<wbr>lyrasis.org</a>> on behalf of Jason Loeffler <<a href="mailto:j@minorscience.com" style="color:purple;text-decoration:underline" target="_blank">j@minorscience.com</a>><br><b>Reply-To:<span class="m_-1845926206139756334Apple-converted-space"> </span></b>Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" style="color:purple;text-decoration:underline" target="_blank">archivesspace_users_group@<wbr>lyralists.lyrasis.org</a>><br><b>Date:<span class="m_-1845926206139756334Apple-converted-space"> </span></b>Tuesday, November 15, 2016 at 3:25 PM<br><b>To:<span class="m_-1845926206139756334Apple-converted-space"> </span></b>Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" style="color:purple;text-decoration:underline" target="_blank">archivesspace_users_group@<wbr>lyralists.lyrasis.org</a>><br><b>Cc:<span class="m_-1845926206139756334Apple-converted-space"> </span></b>"<a href="mailto:archivesspace@googlegroups.com" style="color:purple;text-decoration:underline" target="_blank">archivesspace@<wbr>googlegroups.com</a>" <<a href="mailto:archivesspace@googlegroups.com" style="color:purple;text-decoration:underline" target="_blank">archivesspace@googlegroups.<wbr>com</a>><br><b>Subject:<span class="m_-1845926206139756334Apple-converted-space"> </span></b>Re: [Archivesspace_Users_Group] Problems working with archival object with large number of direct children<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div><div><div><div><div><div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">Hi Sally,<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">Definitely, yes. We have many resources with 5,000 or more archival object records. We've deployed on some pretty decent Amazon EC2 boxes (16GB memory, burstable CPU, etc.) with negligible improvement. I have a feeling that this is not a resource allocation issue. Looking at the web inspector, most of the time is spent negotiating <a href="http://jstree.com/" style="color:purple;text-decoration:underline" target="_blank">jstree</a> and/or loading<i><span class="m_-1845926206139756334Apple-converted-space"> </span>all JSON objects</i><span class="m_-1845926206139756334Apple-converted-space"> </span>associated with a resource into the browser. Maybe an ASpace dev can weigh in.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><br><br><u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">From the sysadmin side, Maureen Callahan at Yale commissioned Percona to evaluate ArchivesSpace and MySQL performance. I've attached the report. Let me know if you need any help interpreting the report.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">At some point, and quite apart from this thread, I hope we can collectively revisit the staff interface architecture and recommend improvements. <u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">JL<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div></div></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'">On Tue, Nov 15, 2016 at 2:37 PM, Sally Vermaaten <<a href="mailto:sally.vermaaten@nyu.edu" style="color:purple;text-decoration:underline" target="_blank">sally.vermaaten@nyu.edu</a>> wrote:<u></u><u></u></div><blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in" type="cite"><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">Hi everyone,</span><u></u><u></u></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt"><u></u> <u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">We're running into an issue with a large resource record in ArchivesSpace and wonder if anyone has experienced a similar issue. In one resource record, we have a series/archival object with around 19,000 direct children/archival objects. We've found that: <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt 47.25pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:10pt;font-family:Symbol"><span>·<span style="font-style:normal;font-variant-caps:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'"> <span class="m_-1845926206139756334Apple-converted-space"> </span></span></span></span><span style="font-size:9.5pt">it takes several minutes to open the series in the 'tree' navigation view and then, once opened scrolling through series is very slow / laggy<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt 47.25pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:10pt;font-family:Symbol"><span>·<span style="font-style:normal;font-variant-caps:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'"> <span class="m_-1845926206139756334Apple-converted-space"> </span></span></span></span><span style="font-size:9.5pt">it takes a couple of minutes to open any archival object in the series in edit mode and <u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt 47.25pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:10pt;font-family:Symbol"><span>·<span style="font-style:normal;font-variant-caps:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'Times New Roman'"> <span class="m_-1845926206139756334Apple-converted-space"> </span></span></span></span><span style="font-size:9.5pt">it takes a couple of minutes to save changes to any archival object within the series<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">Does anyone else have a similarly large archival object in a resource record? If so, have you observed the same long load/save time when editing the component records? <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt"><u></u> <u></u></span></div></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">The slow load time does not seem to be affected by memory allocation; we've tried increasing the speed / size of the server and it seemed to have no effect. We'd definitely appreciate any other suggestions for how we might fix or work around the problem.<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt"><u></u> <u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">We also wonder if this performance issue is essentially caused by the queries being run to generate the UI view - i.e. perhaps in generating the resource 'tree' view, all data for the whole series (all 19k archival objects) is being retrieved and stored in memory? If so, we wondered if it would be possible and would make sense to change the queries running during tree generation, etc. to only retrieve some batches at a time, lazy loading style? <u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt"><u></u> <u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="font-size:9.5pt">Thanks,<u></u><u></u></span></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span class="m_-1845926206139756334gmail-m5887280598291089947gmail-il"><span style="font-size:9.5pt">Weatherly</span></span><span style="font-size:9.5pt"> and Sally<u></u><u></u></span></div></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="color:rgb(136,136,136)"><u></u> <u></u></span></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span class="m_-1845926206139756334gmail-hoenzb"><span style="color:rgb(136,136,136)">--<span class="m_-1845926206139756334Apple-converted-space"> </span></span><u></u><u></u></span></div><div><div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="color:rgb(136,136,136)">Sally Vermaaten<br>Project Manager, Archival Systems<br>New York University Libraries</span><u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><span style="color:rgb(136,136,136)"><a href="tel:1-212-992-6259" style="color:purple;text-decoration:underline" target="_blank">1-212-992-6259</a><u></u><u></u></span></div></div></div></div></div></div><p class="MsoNormal" style="margin:0in 0in 12pt;font-size:12pt;font-family:'Times New Roman'"><br>______________________________<wbr>_________________<br>Archivesspace_Users_Group mailing list<br><a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" style="color:purple;text-decoration:underline" target="_blank">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br><a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" style="color:purple;text-decoration:underline" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><u></u><u></u></p></blockquote></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:'Times New Roman'"><u></u> <u></u></div></div></div></div><span style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">!DSPAM:<wbr>582b7444314351074817778!<span class="m_-1845926206139756334Apple-converted-space"> </span></span><span style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">_____<wbr>______________________________<wbr>____________</span><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><span style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">Archivesspace_Users_Group mailing list</span><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"><span style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);float:none;display:inline!important">!DSPAM:<wbr>582b7444314351074817778!</span><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)"></div></blockquote></div><br></div><br>______________________________<wbr>_________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><br>
<br></blockquote></div><br></div></div>