[Archivesspace_Users_Group] Maximum finding aid size

Wed Oct 10 12:24:26 EDT 2018

Thanks all!

And thanks Mark; it's good to know that large clusters of linked resources
can become an issue, as we definitely have records that include hundreds of
correspondents linked as agents.

On Fri, Oct 5, 2018 at 5:08 PM Custer, Mark <mark.custer at yale.edu> wrote:

> Kevin,
>
> In addition to what Nancy and Steve have already mentioned, I'll add a few
> other things to consider about the size of the records:
>
>
>    - Right now, a note cannot exceed 65k characters in ArchivesSpace.
>    The overall file might not even be that large, but if you have any really,
>    really, really long notes, you won't be able to import the file without
>    splitting up the note.
>    - If you have a single archival object that's linked to, say, 1,000 or
>    more digital objects, top containers, subjects, agents, or the like, then
>    things might not behave so well in the staff interface, etc.  I've found
>    the number of linked records clustered on a single record to be more
>    problematic than a deeply hierarchical finding aid.  As long as you don't
>    have many (or any) linked to over 500 things, I'd say that's good for
>    ASpace.  We have a few above that, but not many.
>    - The tree view for finding aids has also recently been optimized to
>    work better for really flat listing, as well.  See these good-natured notes
>    for a great explanation of the work done behind the scenes to accommodate
>    all sorts of different finding aids:
>    *https://github.com/archivesspace/archivesspace/blob/615fe43e946cbf672e2c35a6c9b79ac6b11a136a/backend/app/model/large_tree.rb#L1-L49*
>    <https://github.com/archivesspace/archivesspace/blob/615fe43e946cbf672e2c35a6c9b79ac6b11a136a/backend/app/model/large_tree.rb#L1-L49>
>
>
> I can also say that we don't have any finding aids as large as 100MB 😊  We
> have a few that run the gamut from 10 - 40 MB, though, and those are
> working well.
>
> Mark
>
>
>
> -----Original Message-----
> From: archivesspace_users_group-bounces at lyralists.lyrasis.org [
> mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org
> <archivesspace_users_group-bounces at lyralists.lyrasis.org>] On Behalf Of
> Kennedy, Nancy
> Sent: Friday, 05 October, 2018 4:08 PM
> To: Archivesspace Users Group <
> archivesspace_users_group at lyralists.lyrasis.org>
> Subject: Re: [Archivesspace_Users_Group] Maximum finding aid size
>
> We also have finding aids that large (about 10 or so that are larger than
> 10MB).  I'd definitely recommend 2.5 if you will need to work with large
> records.  Prior to updating to 2.5, the load times could mean long wait
> times to view or edit.  In the last few weeks, we've added a record that is
> now over 100MB, and takes about 15 minutes to export to EAD.
>
> In the past, in previous systems, we were unable to keep records of this
> size united and had split a few very large records into separate resources
> due to size. It's always been hard to re-unite / present that scenario in
> discovery systems.  So far, archivesspace is doing a better job of scaling
> to handle our size extremes.
>
> Nancy
>
>
>
>
>
>
> -----Original Message-----
> From: archivesspace_users_group-bounces at lyralists.lyrasis.org <
> archivesspace_users_group-bounces at lyralists.lyrasis.org> On Behalf Of
> Majewski, Steven Dennis (sdm7g)
> Sent: Friday, October 05, 2018 3:28 PM
> To: Archivesspace Users Group <
> archivesspace_users_group at lyralists.lyrasis.org>
> Subject: Re: [Archivesspace_Users_Group] Maximum finding aid size
>
>
> I’ve recently imported a 25MB EAD file as well as some around 10-15MB.
> It took a significant amount of time to ingest — about an hour.
> This was on a test server with minimal load other than that job, so you
> may want to schedule your ingest at off hours.
> Definitely do it on a test server first, because it’s difficult to back
> out of if you need to fix something and try again.
>
> ( This was using the current version (v2.5.0). Some very early versions
> were unacceptably slow. )
>
>
> I think the primary overhead is in creating objects, more than the EAD
> parsing, so it probably scales by some other order of complexity rather
> than just text size.
> i.e. creating a very large bioghist or other note is still only creating a
> single object and is probably a single MySQL operation, while turning a
> long list of agent names into agent records and linking them to the
> resource is many operations.
>
>
> — Steve Majewski
>
>
>
> > On Oct 5, 2018, at 3:00 PM, Kevin W. Schlottmann <kws2126 at columbia.edu>
> wrote:
> >
> > Hi all,
> >
> > Does anyone have a sense of where the size limitations are in
> > ArchivesSpace?  We have some very large finding aids (10+ MB EAD) that
> > we hope to import, manage, and publish using AS. I would image that
> > other institutions have large finding aids, and I'm curious to know
> > what issues we might run into.
> >
> > Best,
> >
> > Kevin
> > --
> > Kevin Schlottmann
> > Head of Archives Processing
> > Rare Book & Manuscript Library
> > Butler Library, Room 801
> > Columbia University
> > 535 W. 114th St., New York, NY  10027
> > (212) 854-8483
> >
> > _______________________________________________
> > Archivesspace_Users_Group mailing list
> > Archivesspace_Users_Group at lyralists.lyrasis.org
> > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flyrali
> > sts.lyrasis.org%2Fmailman%2Flistinfo%2Farchivesspace_users_grou&da
> > ta=02%7C01%7Cmark.custer%40yale.edu%7C4ceb88a237be4d5f3f5e08d62afe373d
> > %7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636743668695267673&s
> > data=mbglfUrCPnQFTF1VkDRbbm1hd7nJnr%2F%2B4%2F%2BMFgRKi6A%3D&reserv
> > ed=0
> > p
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
>
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flyralists.lyrasis.org%2Fmailman%2Flistinfo%2Farchivesspace_users_group&data=02%7C01%7Cmark.custer%40yale.edu%7C4ceb88a237be4d5f3f5e08d62afe373d%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636743668695267673&sdata=fD%2FaVX3oK%2FW%2FrvpBVZxy23ofjIixdmTtHyHCSlIYqzA%3D&reserved=0
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>

-- 
Kevin Schlottmann
Head of Archives Processing
Rare Book & Manuscript Library
Butler Library, Room 801
Columbia University
535 W. 114th St., New York, NY  10027
(212) 854-8483
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20181010/3921844d/attachment.html>