From ns96 at nyu.edu Mon Dec 1 08:44:34 2014 From: ns96 at nyu.edu (Nathan Stevens) Date: Mon, 1 Dec 2014 08:44:34 -0500 Subject: [Archivesspace_Users_Group] Validation errors preventing migration of data to ArchivesSpace In-Reply-To: References: Message-ID: This seems to be an issue with the ref id plugin, and not the migration plugin sinces it worked before with no problem. On Nov 27, 2014 7:09 AM, "Megan Williams" wrote: > We are in the midst of migrating our data (tight timeframe, of course) > from Archivists? Toolkit 2.0.0 Update 15 to ArchivesSpace 1.1.0. We are > using both the migration plugin and the refid plugin, and have encountered > validation errors that prevent most of our resource records from migrating. > > > > As per advice from Nathan Steven, we are using the refid plugin with the > option of ?refid_none? to ensure that no refids are copied and that > ArchivesSpace assigns new ones instead. > > > > The following config changes were made in the ArchivesSpace v1.1.0 config > file for loading the "refid_rules" plugin: > > > > AppConfig[:refid_rule] = "<%= repository['repo_code'] %>_<%= > resource['formatted_id'] %>_<%= SecureRandom.hex %>" > > ## Plug-ins to load. They will load in the order specified > > AppConfig[:plugins] = ['refid_rules'] > > > > Post-migration we found that most of our resource records had not > migrated. The migration error log is attached; the errors appear to be > primarily validation errors relating to "refids", i.e.: > > > > {"errors": ["Server error: #<:ValidationException: > :errors=>{\"ref_id\"=>[\"Did not match regular expression: > \\\\A[a-zA-Z0-9\\\\-_:\\\\.]*\\\\z\"]}}>"],"saved": []} > > > > We were previously able to migrate all of our resources in an earlier > trial migration to ArchivesSpace version 1.0.7.1 using the default > migration settings on the migration plugin (without the refid plugin). > > > > Any ideas/advice on where we went wrong would be greatly appreciated! > > > > Many thanks > > > > Megan > > > > *Megan Williams* *|* A/g Assistant Curator *|* Pictures & Manuscripts > Branch *|* National Library of Australia* |* Canberra ACT 2600 *| *w: > www.nla.gov.au > > > > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ns96 at nyu.edu Mon Dec 1 10:10:05 2014 From: ns96 at nyu.edu (Nathan Stevens) Date: Mon, 1 Dec 2014 10:10:05 -0500 Subject: [Archivesspace_Users_Group] Validation errors preventing migration of data to ArchivesSpace In-Reply-To: References: Message-ID: Also, the option to specify no refids is "-refid_none", not "refid_none" (notice the "-"), so this will likely cause the errors you seeing since the migration plugin will just use the default setting of generating random ref_ids if it doesn't understand the option. On Thu, Nov 27, 2014 at 1:05 AM, Megan Williams wrote: > We are in the midst of migrating our data (tight timeframe, of course) > from Archivists? Toolkit 2.0.0 Update 15 to ArchivesSpace 1.1.0. We are > using both the migration plugin and the refid plugin, and have encountered > validation errors that prevent most of our resource records from migrating. > > > > As per advice from Nathan Steven, we are using the refid plugin with the > option of ?refid_none? to ensure that no refids are copied and that > ArchivesSpace assigns new ones instead. > > > > The following config changes were made in the ArchivesSpace v1.1.0 config > file for loading the "refid_rules" plugin: > > > > AppConfig[:refid_rule] = "<%= repository['repo_code'] %>_<%= > resource['formatted_id'] %>_<%= SecureRandom.hex %>" > > ## Plug-ins to load. They will load in the order specified > > AppConfig[:plugins] = ['refid_rules'] > > > > Post-migration we found that most of our resource records had not > migrated. The migration error log is attached; the errors appear to be > primarily validation errors relating to "refids", i.e.: > > > > {"errors": ["Server error: #<:ValidationException: > :errors=>{\"ref_id\"=>[\"Did not match regular expression: > \\\\A[a-zA-Z0-9\\\\-_:\\\\.]*\\\\z\"]}}>"],"saved": []} > > > > We were previously able to migrate all of our resources in an earlier > trial migration to ArchivesSpace version 1.0.7.1 using the default > migration settings on the migration plugin (without the refid plugin). > > > > Any ideas/advice on where we went wrong would be greatly appreciated! > > > > Many thanks > > > > Megan > > > > *Megan Williams* *|* A/g Assistant Curator *|* Pictures & Manuscripts > Branch *|* National Library of Australia* |* Canberra ACT 2600 *| *w: > www.nla.gov.au > > > > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mang.sun at rice.edu Mon Dec 1 16:01:01 2014 From: mang.sun at rice.edu (Mang Sun) Date: Mon, 01 Dec 2014 15:01:01 -0600 Subject: [Archivesspace_Users_Group] partially update a record through API In-Reply-To: <1416996592484.31780@lyrasis.org> References: <546E2632.6020909@rice.edu> <1416923932879.32734@lyrasis.org> <6d2575da7fba4a3f8313bec0989fe94e@CY1PR0501MB1612.namprd05.prod.outlook.com> <1F1E72103798F04D96C355B296D7C61B3FD150@mb2-uts.du.edu>, <1416996592484.31780@lyrasis.org> Message-ID: <547CD70D.3020308@rice.edu> Thank you very much Christ for the script. For the time being, It looks like rubygems.org is down. I will give the script a try later. Mang On 11/26/2014 4:09 AM, Chris Fitzpatrick wrote: > > Hi, > > Ok, so have a look at this rather rough script : > > https://gist.github.com/9eb97ad3f6dd772f875f.git > > It's in Ruby, and you'll have to have rest_client and highline installed ( gem install rest_client highline ) > > Run this with a file called template.json with the values you want to have added. You provide your URI, user, password, repository, and the type of record you're wanting to change. This will go through, get the record's json, merge in the json from your template.json, and update the record. > > Let me know if you have any questions...and fyi global replace features are expected to be done during a sprint in early 2015. > > best, chris. > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > > ________________________________________ > From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Steven Majewski > Sent: Tuesday, November 25, 2014 6:04 PM > To: Archivesspace Users Group > Subject: Re: [Archivesspace_Users_Group] partially update a record through API > > jq would be handy here: > http://stedolan.github.io/jq/ > > something like: > GET some object with curl | jq edit | PUT same object with curl > > where you pass in object URI and jq edit command. > > It would only work for objects where there was a symmetrical GET & PUT. > ( i.e. not for GET /repositories/$REPO/resources/$ID ) > > > ? Steve. > > > On Nov 25, 2014, at 10:06 AM, Kevin Clair wrote: > >> Hello, >> >> I'd be interested as well. I've written one already, which I can share if there's interest, but it's written for very specific migration tasks that we have here at DU so it might not be the most efficient way of going about this. -k >> >> -----Original Message----- >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Noah Huffman >> Sent: Tuesday, November 25, 2014 7:27 AM >> To: Archivesspace Users Group >> Subject: Re: [Archivesspace_Users_Group] partially update a record through API >> >> Hi Chris, >> >> If you write this script, could you share with the list? I'd be interested as well. >> >> I expect that these sorts of batch editing tasks will be fairly common. >> >> Thanks, >> -Noah >> >> ================ >> Noah Huffman >> Archivist for Metadata and Encoding >> David M. Rubenstein Rare Book & Manuscript Library Duke University http://library.duke.edu/rubenstein/ >> >> >> >> -----Original Message----- >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chris Fitzpatrick >> Sent: Tuesday, November 25, 2014 8:59 AM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Subject: Re: [Archivesspace_Users_Group] partially update a record through API >> >> >> Hi Mang Sun, >> >> >> If you're doing it this way, you'll need to get the JSON first and add the publish: false value, then POST that back. >> >> I can write a small script for you that does this, if you're interested... >> b,chris >> >> Chris Fitzpatrick | Developer, ArchivesSpace >> Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ >> >> ________________________________________ >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Mang Sun >> Sent: Thursday, November 20, 2014 6:34 PM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Subject: [Archivesspace_Users_Group] partially update a record through API >> >> I am trying to partially update accession records by setting PUBLISH to false using REST API, but I can't get it done successfully. >> For example, run the following command to update one accession record, curl_as username password -d "@accession.json" -v "the AS API url/repositaries/2/accession/1" >> >> accession.json contains >> { >> "publish" : false >> } >> >> but I get the following complaints >> >> {"error":{"id_0":["Property is required but was missing"],"accession_date":["Property is required but was >> missing"]},"warning":null,"invalid_object":"#> {\"publish\"=>true, \"jsonmodel_type\"=>\"accession\", >> \"external_ids\"=>[], \"related_accessions\"=>[], \"subjects\"=>[], \"linked_events\"=>[], \"extents\"=>[], \"dates\"=>[], \"external_documents\"=>[], \"rights_statements\"=>[], \"deaccessions\"=>[], \"related_resources\"=>[], \"restrictions_apply\"=>false, \"access_restrictions\"=>false, \"use_restrictions\"=>false, \"linked_agents\"=>[], \"instances\"=>[]}>"} >> >> >> Any idea? >> >> Mang Sun >> Rice U. >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > From mariella at caltech.edu Mon Dec 1 18:02:59 2014 From: mariella at caltech.edu (Soprano, Maria (Mariella)) Date: Mon, 1 Dec 2014 23:02:59 +0000 Subject: [Archivesspace_Users_Group] questions about Classification module Message-ID: Hi, We have a couple of questions about the Classification module. 1. We would like to understand how the Classification module is used by other institutions and would like to have some feedback. 2. Has anybody migrated legacy data to the Classification module? If so, how and from what system? Thanks, Mariella Mariella Soprano | Senior Archivist for Collection Management | Caltech Archives & Special Collections | Mail Code 015A-74 | Pasadena CA 91125 | Phone 626.395.2501 | Fax 626.395.4073 | mariella at caltech.edu | Caltech | caltech.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mang.sun at rice.edu Tue Dec 2 10:40:44 2014 From: mang.sun at rice.edu (Mang Sun) Date: Tue, 02 Dec 2014 09:40:44 -0600 Subject: [Archivesspace_Users_Group] partially update a record through API In-Reply-To: <1416996657959.39415@lyrasis.org> References: <546E2632.6020909@rice.edu> <1416923932879.32734@lyrasis.org> <6d2575da7fba4a3f8313bec0989fe94e@CY1PR0501MB1612.namprd05.prod.outlook.com> <1F1E72103798F04D96C355B296D7C61B3FD150@mb2-uts.du.edu>, , <1416996592484.31780@lyrasis.org> <1416996657959.39415@lyrasis.org> Message-ID: <547DDD7C.5090800@rice.edu> Chris, The script runs as expected. Thanks a lot. Mang On 11/26/2014 4:10 AM, Chris Fitzpatrick wrote: > > Sorry, just realized I sent the git url. Here's the gist: > > > https://gist.github.com/cfitz/9eb97ad3f6dd772f875f > > > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > > ________________________________________ > From: Chris Fitzpatrick > Sent: Wednesday, November 26, 2014 11:09 AM > To: Archivesspace Users Group > Subject: Re: [Archivesspace_Users_Group] partially update a record through API > > Hi, > > Ok, so have a look at this rather rough script : > > https://gist.github.com/9eb97ad3f6dd772f875f.git > > It's in Ruby, and you'll have to have rest_client and highline installed ( gem install rest_client highline ) > > Run this with a file called template.json with the values you want to have added. You provide your URI, user, password, repository, and the type of record you're wanting to change. This will go through, get the record's json, merge in the json from your template.json, and update the record. > > Let me know if you have any questions...and fyi global replace features are expected to be done during a sprint in early 2015. > > best, chris. > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > > ________________________________________ > From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Steven Majewski > Sent: Tuesday, November 25, 2014 6:04 PM > To: Archivesspace Users Group > Subject: Re: [Archivesspace_Users_Group] partially update a record through API > > jq would be handy here: > http://stedolan.github.io/jq/ > > something like: > GET some object with curl | jq edit | PUT same object with curl > > where you pass in object URI and jq edit command. > > It would only work for objects where there was a symmetrical GET & PUT. > ( i.e. not for GET /repositories/$REPO/resources/$ID ) > > > ? Steve. > > > On Nov 25, 2014, at 10:06 AM, Kevin Clair wrote: > >> Hello, >> >> I'd be interested as well. I've written one already, which I can share if there's interest, but it's written for very specific migration tasks that we have here at DU so it might not be the most efficient way of going about this. -k >> >> -----Original Message----- >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Noah Huffman >> Sent: Tuesday, November 25, 2014 7:27 AM >> To: Archivesspace Users Group >> Subject: Re: [Archivesspace_Users_Group] partially update a record through API >> >> Hi Chris, >> >> If you write this script, could you share with the list? I'd be interested as well. >> >> I expect that these sorts of batch editing tasks will be fairly common. >> >> Thanks, >> -Noah >> >> ================ >> Noah Huffman >> Archivist for Metadata and Encoding >> David M. Rubenstein Rare Book & Manuscript Library Duke University http://library.duke.edu/rubenstein/ >> >> >> >> -----Original Message----- >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chris Fitzpatrick >> Sent: Tuesday, November 25, 2014 8:59 AM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Subject: Re: [Archivesspace_Users_Group] partially update a record through API >> >> >> Hi Mang Sun, >> >> >> If you're doing it this way, you'll need to get the JSON first and add the publish: false value, then POST that back. >> >> I can write a small script for you that does this, if you're interested... >> b,chris >> >> Chris Fitzpatrick | Developer, ArchivesSpace >> Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ >> >> ________________________________________ >> From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Mang Sun >> Sent: Thursday, November 20, 2014 6:34 PM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Subject: [Archivesspace_Users_Group] partially update a record through API >> >> I am trying to partially update accession records by setting PUBLISH to false using REST API, but I can't get it done successfully. >> For example, run the following command to update one accession record, curl_as username password -d "@accession.json" -v "the AS API url/repositaries/2/accession/1" >> >> accession.json contains >> { >> "publish" : false >> } >> >> but I get the following complaints >> >> {"error":{"id_0":["Property is required but was missing"],"accession_date":["Property is required but was >> missing"]},"warning":null,"invalid_object":"#> {\"publish\"=>true, \"jsonmodel_type\"=>\"accession\", >> \"external_ids\"=>[], \"related_accessions\"=>[], \"subjects\"=>[], \"linked_events\"=>[], \"extents\"=>[], \"dates\"=>[], \"external_documents\"=>[], \"rights_statements\"=>[], \"deaccessions\"=>[], \"related_resources\"=>[], \"restrictions_apply\"=>false, \"access_restrictions\"=>false, \"use_restrictions\"=>false, \"linked_agents\"=>[], \"instances\"=>[]}>"} >> >> >> Any idea? >> >> Mang Sun >> Rice U. >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > From sdm7g at virginia.edu Tue Dec 2 11:27:27 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Tue, 2 Dec 2014 11:27:27 -0500 Subject: [Archivesspace_Users_Group] solr_home Message-ID: Indexing in my ArchivesSpace instances appear to be working properly, however when I go to the solr web console and click on either Schema or Config for Collection1, I see a stack trace: 5001java.lang.NullPointerException at org.apache.solr.handler.admin.ShowFileRequestHandler.showFromFileSystem(ShowFileRequestHandler.java:209) at org.apache.solr.handler.admin.ShowFileRequestHandler.handleRequestBody(ShowFileRequestHandler.java:122) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1302) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:448) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1067) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:377) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1001) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:360) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:744) 500 And the Dashboard shows a non-existing directory for solr_home and ?Instance? : /usr/local/projects/archivesspace/data/solr_home/collection1 Just wondering what all that means! ( I was having a problem with a devserver instance, which was actually due to my forgetting to start the indexer:devserver, however, in hunting for the problem, I was misled by these error messages, until I discovered my functioning test server showed the same errors. ) ? Steve Majewski / UVA Alderman Library -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From PGalligan at rockarch.org Tue Dec 2 12:33:47 2014 From: PGalligan at rockarch.org (Galligan, Patrick) Date: Tue, 2 Dec 2014 12:33:47 -0500 Subject: [Archivesspace_Users_Group] Large PDF export failing Message-ID: We've been doing some testing with PDF exports recently, and have noticed that if a resource record gets to a certain size, it will lock up AS for a couple of minutes and send the user to an error page. We have had no problem with smaller PDF files, and haven't had issues with EAD export. Has anyone else run into this issue? Our IT department has been unable to track down the issue. We thought it might be a mysql lock wait problem, but that ended up being fruitless. Any thoughts? Here's a log of what's happening: 2014-12-02 12:21:16.741:INFO:/:Started GET "/resources/11211/download_ead?include_daos=false&include_unpublished=true&numbered_cs=false&print_pdf=true" for 192.168.50.129 at 2014-12-02 12:21:16 -0500| 2014-12-02 12:21:16.761:INFO:/: Parameters: {"include_daos"=>"false", "include_unpublished"=>"true", "numbered_cs"=>"false", "print_pdf"=>"true", "id"=>"11211"}| D, [2014-12-02T12:21:16.946000 #25469] DEBUG -- : Thread-3486: GET /repositories/2/resource_descriptions/11211.pdf/metadata [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:16.959000 #25469] DEBUG -- : Thread-3486: Post-processed params: {:id=>11211, :repo_id=>2, :fmt=>"pdf"} D, [2014-12-02T12:21:17.049000 #25469] DEBUG -- : Thread-3470: GET /repositories/2/resource_descriptions/11211.pdf?include_unpublished=true&print_pdf=true&include_daos=false&numbered_cs=false [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:17.057000 #25469] DEBUG -- : Thread-3470: Post-processed params: {:id=>11211, :include_unpublished=>true, :include_daos=>false, :numbered_cs=>false, :print_pdf=>true, :repo_id=>2} D, [2014-12-02T12:21:20.720000 #25469] DEBUG -- : Thread-3470: GET /repositories/2/resources/11211/tree?include_unpublished=true&print_pdf=true&include_daos=false&numbered_cs=false [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:20.728000 #25469] DEBUG -- : Thread-3470: Post-processed params: {:id=>11211, :repo_id=>2} D, [2014-12-02T12:21:31.415000 #25469] DEBUG -- : Thread-3470: Responded with [200, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"3850900"}, ["{\"title\":\"Nelson A. Rockefeller papers\",\"id\":11211,\"node_type\":\"resource\",\"publish\":true,\"suppressed\":false,\"children\":[{\"title\":\"Activities, 1930-1979, bulk 1946-1971\",\"id\":658077,\"record_uri\":\"/repositories/2/archival_objects/658077\",\"publish\":true,\"suppressed\":false,\"node_type\":\"archival_object\",\"level\":\"series\",\"has_children\":true,\"chi... in 10529.0ms Patrick Galligan Rockefeller Archive Center Assistant Digital Archivist 914-366-6386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From reese.2179 at osu.edu Tue Dec 2 12:45:12 2014 From: reese.2179 at osu.edu (Reese, Terry P.) Date: Tue, 2 Dec 2014 17:45:12 +0000 Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record In-Reply-To: <782EFBE964BFC34E893C7FF255B5F20F363B7B92@CIO-KRC-D2MBX11.osuad.osu.edu> References: <782EFBE964BFC34E893C7FF255B5F20F363B7B92@CIO-KRC-D2MBX11.osuad.osu.edu> Message-ID: Hi, I wanted to add to this message a little bit since I'd had a chance to talk with Cate and have some screenshots to help illustrate the problem. The issue seems to be a UI issue, as the data is definitely there and stored - but when in edit mode, it seems the UI only will display the first three attached instances or objects for edit. It's almost like there should be a load more button and it's missing (though I'm sure that's likely not the case). I had Cate provide me a screen shot so folks could see what is happening. In the view interface, I can see the 6 instances attached to the record: [cid:image001.jpg at 01D00E2D.D01998C0] However, in the edit mode, only 3 instances are visible and there is no button or option (that we can see) to have the interface load the remaining instance data for edit. (see below). [cid:image005.jpg at 01D00E2D.D01998C0] There is a very good possibility that I'm just blind and missing something, so any feedback would be appreciated. Thanks, --tr From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Putirskis, Crystal E. (Cate) Sent: Wednesday, November 26, 2014 1:16 PM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record I'm adding instances at the collection level of a resource record. After saving the record, I find (in Edit mode) that I can only access the first three instances entered-there does not seem to be a way to list, access, or edit any of the others. [In View mode, I can see all of my instances.] So, in Edit mode, how do I access instances 4 and beyond? (Also, in View mode, are there any development plans to include the container 1 indicator (at minimum) alongside the container 1 type? A list of 40 lines that just say 'Box' is not ideal.) Thanks, Cate [The Ohio State University] Cate Putirskis Special Collections Processing Coordinator University Libraries - Special Collections Description & Access 016 Thompson Library | 1858 Neil Ave Mall, Columbus, OH 43210 614-292-8114 putirskis.1 at osu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 43526 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 57504 bytes Desc: image005.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.png Type: image/png Size: 8636 bytes Desc: image006.png URL: From Kevin.Clair at du.edu Tue Dec 2 16:36:39 2014 From: Kevin.Clair at du.edu (Kevin Clair) Date: Tue, 2 Dec 2014 21:36:39 +0000 Subject: [Archivesspace_Users_Group] automatically adding Dates of Existence to Agent sort names Message-ID: <1F1E72103798F04D96C355B296D7C61B400418@mb2-uts.du.edu> Hello, Our archivists have reported some confusion about the way ArchivesSpace generates sort names for Agents. When we began our migration, "Dates" was still a field within the Name Form section of the Agent data model, and so when we used it to manage life dates for a person, the sort title pulled them automatically. Now that the life dates are stored in the Dates of Existence section, the sort title doesn't pull them automatically anymore. I've told them to override the option for automatically generating sort titles if they're creating Agent records where the life dates are present, but it would be nice to be able to assign a primary date of existence in an Agent record (similar to authorized/display forms of names) so that we can automatically have an expression of a single date or date range included in the sort title again. Not sure if others have encountered this as an issue, but since it's been reported to me a few times now, I thought I'd bring it up on the list. thanks! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Wed Dec 3 05:32:40 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Wed, 3 Dec 2014 10:32:40 +0000 Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record In-Reply-To: References: <782EFBE964BFC34E893C7FF255B5F20F363B7B92@CIO-KRC-D2MBX11.osuad.osu.edu>, Message-ID: <1417602761908.41920@lyrasis.org> Hi Terry, Yes, this bug has been tracked here => https://www.pivotaltracker.com/story/show/82065358 It's a problem with the "Show All" button being bumped out for Instances. There's a fix in the aspace-maintance plugin ( https://github.com/archivesspace/aspace-110-plugin ) that resolves this. Let me know if this works for you. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Reese, Terry P. Sent: Tuesday, December 2, 2014 6:45 PM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record Hi, I wanted to add to this message a little bit since I'd had a chance to talk with Cate and have some screenshots to help illustrate the problem. The issue seems to be a UI issue, as the data is definitely there and stored - but when in edit mode, it seems the UI only will display the first three attached instances or objects for edit. It's almost like there should be a load more button and it's missing (though I'm sure that's likely not the case). I had Cate provide me a screen shot so folks could see what is happening. In the view interface, I can see the 6 instances attached to the record: [cid:image001.jpg at 01D00E2D.D01998C0] However, in the edit mode, only 3 instances are visible and there is no button or option (that we can see) to have the interface load the remaining instance data for edit. (see below). [cid:image005.jpg at 01D00E2D.D01998C0] There is a very good possibility that I'm just blind and missing something, so any feedback would be appreciated. Thanks, --tr From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Putirskis, Crystal E. (Cate) Sent: Wednesday, November 26, 2014 1:16 PM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record I'm adding instances at the collection level of a resource record. After saving the record, I find (in Edit mode) that I can only access the first three instances entered-there does not seem to be a way to list, access, or edit any of the others. [In View mode, I can see all of my instances.] So, in Edit mode, how do I access instances 4 and beyond? (Also, in View mode, are there any development plans to include the container 1 indicator (at minimum) alongside the container 1 type? A list of 40 lines that just say 'Box' is not ideal.) Thanks, Cate [The Ohio State University] Cate Putirskis Special Collections Processing Coordinator University Libraries - Special Collections Description & Access 016 Thompson Library | 1858 Neil Ave Mall, Columbus, OH 43210 614-292-8114 putirskis.1 at osu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 43526 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 57504 bytes Desc: image005.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.png Type: image/png Size: 8636 bytes Desc: image006.png URL: From reese.2179 at osu.edu Wed Dec 3 08:17:34 2014 From: reese.2179 at osu.edu (Reese, Terry P.) Date: Wed, 3 Dec 2014 13:17:34 +0000 Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record In-Reply-To: <1417602761908.41920@lyrasis.org> References: <782EFBE964BFC34E893C7FF255B5F20F363B7B92@CIO-KRC-D2MBX11.osuad.osu.edu>, <1417602761908.41920@lyrasis.org> Message-ID: Thanks Chris, good to know. --tr From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chris Fitzpatrick Sent: Wednesday, December 3, 2014 5:33 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record Hi Terry, Yes, this bug has been tracked here => https://www.pivotaltracker.com/story/show/82065358 It's a problem with the "Show All" button being bumped out for Instances. There's a fix in the aspace-maintance plugin ( https://github.com/archivesspace/aspace-110-plugin ) that resolves this. Let me know if this works for you. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org > on behalf of Reese, Terry P. > Sent: Tuesday, December 2, 2014 6:45 PM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record Hi, I wanted to add to this message a little bit since I'd had a chance to talk with Cate and have some screenshots to help illustrate the problem. The issue seems to be a UI issue, as the data is definitely there and stored - but when in edit mode, it seems the UI only will display the first three attached instances or objects for edit. It's almost like there should be a load more button and it's missing (though I'm sure that's likely not the case). I had Cate provide me a screen shot so folks could see what is happening. In the view interface, I can see the 6 instances attached to the record: [cid:image001.jpg at 01D00ED1.96471920] However, in the edit mode, only 3 instances are visible and there is no button or option (that we can see) to have the interface load the remaining instance data for edit. (see below). [cid:image002.jpg at 01D00ED1.96471920] There is a very good possibility that I'm just blind and missing something, so any feedback would be appreciated. Thanks, --tr From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Putirskis, Crystal E. (Cate) Sent: Wednesday, November 26, 2014 1:16 PM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Cannot view/edit more than 3 instances in a Resource Record I'm adding instances at the collection level of a resource record. After saving the record, I find (in Edit mode) that I can only access the first three instances entered-there does not seem to be a way to list, access, or edit any of the others. [In View mode, I can see all of my instances.] So, in Edit mode, how do I access instances 4 and beyond? (Also, in View mode, are there any development plans to include the container 1 indicator (at minimum) alongside the container 1 type? A list of 40 lines that just say 'Box' is not ideal.) Thanks, Cate [The Ohio State University] Cate Putirskis Special Collections Processing Coordinator University Libraries - Special Collections Description & Access 016 Thompson Library | 1858 Neil Ave Mall, Columbus, OH 43210 614-292-8114 putirskis.1 at osu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 43526 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 57504 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 8636 bytes Desc: image003.png URL: From Chris.Fitzpatrick at lyrasis.org Wed Dec 3 10:39:46 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Wed, 3 Dec 2014 15:39:46 +0000 Subject: [Archivesspace_Users_Group] Large PDF export failing In-Reply-To: References: Message-ID: <1417621187601.71340@lyrasis.org> Hi Patrick, What's most likely happening is that you're just hitting a webserver timeout error. The problem is that large resources can take a long time to generate EAD XML, and it'll be even longer to take that XML, transform it into HTML, and transform that HTML into a PDF. You could bump up the timeout time to a crazy high number, which could fix the problem. The ultimate fix is going to have to be to background EAD exports to jobs, like they are for imports. We have this slated for a release shortly after the beginning of the new year. I do have a utility here that might help out => https://github.com/archivesspace/ead2pdf/releases You can take an EAD XML file and run it through the PDF creation process locally. To run it, simply run the : java -jar ead2pdf.jar ead_file.xml output_pdf.pdf And it'll use the same XSLT that ASpace uses. Hope this helps... b,chris. [https://avatars2.githubusercontent.com/u/1311559?v=3&s=400] Releases ? archivesspace/ead2pdf ? GitHub ead2pdf - ASpace EAD2PDF Read more... Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Galligan, Patrick Sent: Tuesday, December 2, 2014 6:33 PM To: Archivesspace Users Group Subject: [Archivesspace_Users_Group] Large PDF export failing We?ve been doing some testing with PDF exports recently, and have noticed that if a resource record gets to a certain size, it will lock up AS for a couple of minutes and send the user to an error page. We have had no problem with smaller PDF files, and haven?t had issues with EAD export. Has anyone else run into this issue? Our IT department has been unable to track down the issue. We thought it might be a mysql lock wait problem, but that ended up being fruitless. Any thoughts? Here?s a log of what?s happening: 2014-12-02 12:21:16.741:INFO:/:Started GET "/resources/11211/download_ead?include_daos=false&include_unpublished=true&numbered_cs=false&print_pdf=true" for 192.168.50.129 at 2014-12-02 12:21:16 -0500| 2014-12-02 12:21:16.761:INFO:/: Parameters: {"include_daos"=>"false", "include_unpublished"=>"true", "numbered_cs"=>"false", "print_pdf"=>"true", "id"=>"11211"}| D, [2014-12-02T12:21:16.946000 #25469] DEBUG -- : Thread-3486: GET /repositories/2/resource_descriptions/11211.pdf/metadata [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:16.959000 #25469] DEBUG -- : Thread-3486: Post-processed params: {:id=>11211, :repo_id=>2, :fmt=>"pdf"} D, [2014-12-02T12:21:17.049000 #25469] DEBUG -- : Thread-3470: GET /repositories/2/resource_descriptions/11211.pdf?include_unpublished=true&print_pdf=true&include_daos=false&numbered_cs=false [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:17.057000 #25469] DEBUG -- : Thread-3470: Post-processed params: {:id=>11211, :include_unpublished=>true, :include_daos=>false, :numbered_cs=>false, :print_pdf=>true, :repo_id=>2} D, [2014-12-02T12:21:20.720000 #25469] DEBUG -- : Thread-3470: GET /repositories/2/resources/11211/tree?include_unpublished=true&print_pdf=true&include_daos=false&numbered_cs=false [session: #"jmorillo", :login_time=>2014-12-02 09:32:54 -0500, :expirable=>true}, @id="e8486be22af564ff33cd7254844716d921777beae4daeafadf29ac8c676a21e6">] D, [2014-12-02T12:21:20.728000 #25469] DEBUG -- : Thread-3470: Post-processed params: {:id=>11211, :repo_id=>2} D, [2014-12-02T12:21:31.415000 #25469] DEBUG -- : Thread-3470: Responded with [200, {"Content-Type"=>"application/json", "Cache-Control"=>"private, must-revalidate, max-age=0", "Content-Length"=>"3850900"}, ["{\"title\":\"Nelson A. Rockefeller papers\",\"id\":11211,\"node_type\":\"resource\",\"publish\":true,\"suppressed\":false,\"children\":[{\"title\":\"Activities, 1930-1979, bulk 1946-1971\",\"id\":658077,\"record_uri\":\"/repositories/2/archival_objects/658077\",\"publish\":true,\"suppressed\":false,\"node_type\":\"archival_object\",\"level\":\"series\",\"has_children\":true,\"chi... in 10529.0ms Patrick Galligan Rockefeller Archive Center Assistant Digital Archivist 914-366-6386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From francine.snyder at guggenheim.org Wed Dec 3 15:12:26 2014 From: francine.snyder at guggenheim.org (Francine Snyder) Date: Wed, 3 Dec 2014 20:12:26 +0000 Subject: [Archivesspace_Users_Group] Location workflow Message-ID: <927A17016A6A094EA6F33793D11E0604865D9DFC@NY345EX10.na.GUGGENHEIM.ORG> Hello - While I know ArchivesSpace hasn't been fully implemented at many places, I'm throwing these questions out to the group anyways to see if anyone has an insight. Not sure how to search pivot tracker so perhaps (hopefully) some of these are currently being addressed. We've migrated our data into ArchivesSpace and are experimenting with some of our workflows. We migrated from MS Access (accessions) and EAD (resources). A few location-specific questions for the group. 1. Location in accession/resource records: as we have multiple storage locations and boxes frequently move between storage, processing, and research, we are constantly using and updating this field. We're finding the position of the location field in the accession and resource records cumbersome (buried under instances and must be clicked open to view). It is my understanding this is similar in AT; are others planning on using the location module actively or are repositories using other methods being used to track locations? (We're open to other methods. Questions below are if location field is used.) 2. Location in search and browse: we are interested in the ability to add an accession's "inventory" and "location" fields to the search and browse result columns. Are there plans to make the search and browse result columns customizable? Likewise we would like be able to filter by location in advanced search. 3. Location in rapid entry (resource): Are there plans to add the option for instance location to the columns in rapid data entry? Currently, we have to manually add this to each file record after completing. 4. Location modification: Is there a way to rapidly change the location of all instances with the same box number? Thanks, Francine Francine Snyder Director of Library and Archives Guggenheim Museum -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmargalo at udel.edu Wed Dec 3 16:37:27 2014 From: jmargalo at udel.edu (Jaime Margalotti) Date: Wed, 3 Dec 2014 16:37:27 -0500 Subject: [Archivesspace_Users_Group] Nested issue Message-ID: Hi Chris & Co., I am continuing to have issues with imported EAD in regards to the . Our existing EAD files have the nested within our . Upon import, ArchivesSpace is recognizing the and properly populating that field, but it is also leaving the content of that tag in the . I've attached an image. I thought the nesting issue was supposed to be addressed in this version of the program (version 1.1.0), so I looked at the notes for the release on github. The problem sounds like BUG #72326468 , which is listed with the resolved bugs: https://github.com/archivesspace/archivesspace/releases/tag/v1.1.0 Thanks, Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267302-831-0554jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: aspacedates.png Type: image/png Size: 292252 bytes Desc: not available URL: From jmargalo at udel.edu Wed Dec 3 17:07:38 2014 From: jmargalo at udel.edu (Jaime Margalotti) Date: Wed, 3 Dec 2014 17:07:38 -0500 Subject: [Archivesspace_Users_Group] Classifications - a bug and a feature request Message-ID: We're excited about using the Classifications feature in ArchivesSpace and I have added the categories that we use to classify all of our collections. In doing so, I've come across one small bug and would like to request a feature. I apologize if this bug is addressed elsewhere in the larger issues with special characters. Several of our Classification titles have ampersands (&). They appear with no problem in every part of the Classification record except for the blue linked text at the top of the record (see attached image). This is obviously quite minor, but figured it was worth mentioning. The feature I'd like to see involves the Identifier. The Identifiers that we use are numeric and it would be handy to be able to sort by the Identifier in the Browse screen for the Classifications. You can change the Browse options for Accessions, Resources, and Digital Objects, but not Classifications. Thanks! Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: aspace_ampersand.png Type: image/png Size: 303388 bytes Desc: not available URL: From bmg17 at psu.edu Wed Dec 3 17:08:55 2014 From: bmg17 at psu.edu (Ben Goldman) Date: Wed, 3 Dec 2014 17:08:55 -0500 (EST) Subject: [Archivesspace_Users_Group] note contents/ead export In-Reply-To: <321207997.344997.1417643490154.JavaMail.zimbra@psu.edu> Message-ID: <555005133.351487.1417644535600.JavaMail.zimbra@psu.edu> Hello, I apologize if this has been addressed elsewhere, but I couldn't find an answer after searching in the Google Group or Pivotal Tracker (side note: is there an easy way to search this listserv's archives?). I've noticed that the block text found in the Content fields of Notes, when exported as EAD, is wrapping that text in this: Any idea why this is or what we can do to adjust it? On a related topic, a lot of our Bio/Hist and Scope/Content are multi-paragraph notes that had

tags embedded appropriately in AT, but we apparently lost these tags/embedded content for our notes in the migration to ASpace. ASpace applies bookend

tags around the entirety of the note on EAD export, but this results in code like this:

Paragraph. New paragraph. Last paragraph.

We're currently preparing to republish all our finding aids using the ASpace API, but would love to fix this first. Does anyone have ideas on how we might remediate this without manually checking/editing every resource? We're still running 1.09, so if someone tells me the answer is an upgrade to 1.1, I will be thrilled! Thanks, Ben Ben Goldman Digital Records Archivist Penn State University Libraries University Park, PA 814-863-8333 http://www.libraries.psu.edu/psul/speccolls.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Wed Dec 3 17:23:10 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Wed, 3 Dec 2014 17:23:10 -0500 Subject: [Archivesspace_Users_Group] note contents/ead export In-Reply-To: <555005133.351487.1417644535600.JavaMail.zimbra@psu.edu> References: <555005133.351487.1417644535600.JavaMail.zimbra@psu.edu> Message-ID: Ben: I believe this is fixed in 1.1.0 + aspace-110-plugin. ( or if you build from github master ) It is fixed on export in the serializer, which means that if you upgrade ArchivesSpace and keep existing data they will export without CDATA escaping. ? Steve Majewski On Dec 3, 2014, at 5:08 PM, Ben Goldman wrote: > Hello, > > I apologize if this has been addressed elsewhere, but I couldn't find an answer after searching in the Google Group or Pivotal Tracker (side note: is there an easy way to search this listserv's archives?). > > I've noticed that the block text found in the Content fields of Notes, when exported as EAD, is wrapping that text in this: > > > > Any idea why this is or what we can do to adjust it? > > On a related topic, a lot of our Bio/Hist and Scope/Content are multi-paragraph notes that had

tags embedded appropriately in AT, but we apparently lost these tags/embedded content for our notes in the migration to ASpace. ASpace applies bookend

tags around the entirety of the note on EAD export, but this results in code like this: > >

Paragraph. > > New paragraph. > > Last paragraph.

> > We're currently preparing to republish all our finding aids using the ASpace API, but would love to fix this first. Does anyone have ideas on how we might remediate this without manually checking/editing every resource? We're still running 1.09, so if someone tells me the answer is an upgrade to 1.1, I will be thrilled! > > Thanks, > Ben > > > > Ben Goldman > Digital Records Archivist > Penn State University Libraries > University Park, PA > 814-863-8333 > http://www.libraries.psu.edu/psul/speccolls.html > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From Chris.Fitzpatrick at lyrasis.org Thu Dec 4 09:37:38 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Thu, 4 Dec 2014 14:37:38 +0000 Subject: [Archivesspace_Users_Group] Nested issue In-Reply-To: References: Message-ID: <1417703861239.9701@lyrasis.org> Hi Jaime, Yes, this looks like a case of dueling requests, since this ticket: https://www.pivotaltracker.com/story/show/72190678 then wanted mixed content added back into the unittitle. I think the nuance I missed there is that regular tags should be allowed and kepy, except for unitdate, which should be stripped out. Will update the original ticket and add a fix.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: Jaime Margalotti Sent: Wednesday, December 3, 2014 10:37 PM To: archivesspace_users_group at lyralists.lyrasis.org; Chris Fitzpatrick Subject: Nested issue Hi Chris & Co., I am continuing to have issues with imported EAD in regards to the . Our existing EAD files have the nested within our . Upon import, ArchivesSpace is recognizing the and properly populating that field, but it is also leaving the content of that tag in the . I've attached an image. I thought the nesting issue was supposed to be addressed in this version of the program (version 1.1.0), so I looked at the notes for the release on github. The problem sounds like BUG #72326468, which is listed with the resolved bugs: https://github.com/archivesspace/archivesspace/releases/tag/v1.1.0 Thanks, Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554 jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmg17 at psu.edu Thu Dec 4 09:42:01 2014 From: bmg17 at psu.edu (Ben Goldman) Date: Thu, 4 Dec 2014 09:42:01 -0500 (EST) Subject: [Archivesspace_Users_Group] note contents/ead export In-Reply-To: References: <555005133.351487.1417644535600.JavaMail.zimbra@psu.edu> Message-ID: <614555488.525723.1417704121565.JavaMail.zimbra@psu.edu> Steve, great news, thanks! Do we know whether 1.1 still applies the

tags in notes in the same manner on EAD export? -Ben ----- Original Message ----- From: "Steven Majewski" To: "Archivesspace Users Group" Sent: Wednesday, December 3, 2014 5:23:10 PM Subject: Re: [Archivesspace_Users_Group] note contents/ead export Ben: I believe this is fixed in 1.1.0 + aspace-110-plugin. ( or if you build from github master ) It is fixed on export in the serializer, which means that if you upgrade ArchivesSpace and keep existing data they will export without CDATA escaping. ? Steve Majewski On Dec 3, 2014, at 5:08 PM, Ben Goldman < bmg17 at psu.edu > wrote: Hello, I apologize if this has been addressed elsewhere, but I couldn't find an answer after searching in the Google Group or Pivotal Tracker (side note: is there an easy way to search this listserv's archives?). I've noticed that the block text found in the Content fields of Notes, when exported as EAD, is wrapping that text in this: Any idea why this is or what we can do to adjust it? On a related topic, a lot of our Bio/Hist and Scope/Content are multi-paragraph notes that had

tags embedded appropriately in AT, but we apparently lost these tags/embedded content for our notes in the migration to ASpace. ASpace applies bookend

tags around the entirety of the note on EAD export, but this results in code like this:

Paragraph. New paragraph. Last paragraph.

We're currently preparing to republish all our finding aids using the ASpace API, but would love to fix this first. Does anyone have ideas on how we might remediate this without manually checking/editing every resource? We're still running 1.09, so if someone tells me the answer is an upgrade to 1.1, I will be thrilled! Thanks, Ben Ben Goldman Digital Records Archivist Penn State University Libraries University Park, PA 814-863-8333 http://www.libraries.psu.edu/psul/speccolls.html _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmargalo at udel.edu Thu Dec 4 09:56:26 2014 From: jmargalo at udel.edu (Jaime Margalotti) Date: Thu, 4 Dec 2014 09:56:26 -0500 Subject: [Archivesspace_Users_Group] Nested issue In-Reply-To: <1417703861239.9701@lyrasis.org> References: <1417703861239.9701@lyrasis.org> Message-ID: Thanks very much! As far as I can tell, is the only legal nested tag in that it makes sense to purge. Best, Jaime On Thu, Dec 4, 2014 at 9:37 AM, Chris Fitzpatrick < Chris.Fitzpatrick at lyrasis.org> wrote: > > Hi Jaime, > > > Yes, this looks like a case of dueling requests, since this ticket: > > https://www.pivotaltracker.com/story/show/72190678 > > > then wanted mixed content added back into the unittitle. > > > I think the nuance I missed there is that regular tags should be allowed > and kepy, except for unitdate, which should be stripped out. > > > Will update the original ticket and add a fix.. > > > b,chris. > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > ------------------------------ > *From:* Jaime Margalotti > *Sent:* Wednesday, December 3, 2014 10:37 PM > *To:* archivesspace_users_group at lyralists.lyrasis.org; Chris Fitzpatrick > *Subject:* Nested issue > > Hi Chris & Co., > > I am continuing to have issues with imported EAD in regards to the > . Our existing EAD files have the nested within our > . Upon import, ArchivesSpace is recognizing the and > properly populating that field, but it is also leaving the content of that > tag in the . I've attached an image. > > I thought the nesting issue was supposed to be addressed in this version > of the program (version 1.1.0), so I looked at the notes for the release on > github. The problem sounds like BUG #72326468 > , which is listed > with the resolved bugs: > https://github.com/archivesspace/archivesspace/releases/tag/v1.1.0 > > Thanks, > Jaime > > -- > > Jaime L. Margalotti > Senior Assistant Librarian > Manuscripts and Archives Department University of Delaware Library > Newark, DE 19717-5267302-831-0554jmargalo at udel.edu > > -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.custer at yale.edu Thu Dec 4 10:03:35 2014 From: mark.custer at yale.edu (Custer, Mark) Date: Thu, 4 Dec 2014 15:03:35 +0000 Subject: [Archivesspace_Users_Group] note contents/ead export In-Reply-To: <614555488.525723.1417704121565.JavaMail.zimbra@psu.edu> References: <555005133.351487.1417644535600.JavaMail.zimbra@psu.edu> <614555488.525723.1417704121565.JavaMail.zimbra@psu.edu> Message-ID: Ben, I haven?t tested this thoroughly, but my understanding is that the exporter in the aspace-110-plugin (and maybe the core code, too) will now apply

tags wherever it encounters singe line breaks in a notes field, which is a bit different than how the AT used a double line break for the same purpose. That said, neither system will change a single line break to an EAD element, so those will need to be hand-encoded. Mark From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Ben Goldman Sent: Thursday, December 04, 2014 9:42 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] note contents/ead export Steve, great news, thanks! Do we know whether 1.1 still applies the

tags in notes in the same manner on EAD export? -Ben ________________________________ From: "Steven Majewski" > To: "Archivesspace Users Group" > Sent: Wednesday, December 3, 2014 5:23:10 PM Subject: Re: [Archivesspace_Users_Group] note contents/ead export Ben: I believe this is fixed in 1.1.0 + aspace-110-plugin. ( or if you build from github master ) It is fixed on export in the serializer, which means that if you upgrade ArchivesSpace and keep existing data they will export without CDATA escaping. ? Steve Majewski On Dec 3, 2014, at 5:08 PM, Ben Goldman > wrote: Hello, I apologize if this has been addressed elsewhere, but I couldn't find an answer after searching in the Google Group or Pivotal Tracker (side note: is there an easy way to search this listserv's archives?). I've noticed that the block text found in the Content fields of Notes, when exported as EAD, is wrapping that text in this: Any idea why this is or what we can do to adjust it? On a related topic, a lot of our Bio/Hist and Scope/Content are multi-paragraph notes that had

tags embedded appropriately in AT, but we apparently lost these tags/embedded content for our notes in the migration to ASpace. ASpace applies bookend

tags around the entirety of the note on EAD export, but this results in code like this:

Paragraph. New paragraph. Last paragraph.

We're currently preparing to republish all our finding aids using the ASpace API, but would love to fix this first. Does anyone have ideas on how we might remediate this without manually checking/editing every resource? We're still running 1.09, so if someone tells me the answer is an upgrade to 1.1, I will be thrilled! Thanks, Ben Ben Goldman Digital Records Archivist Penn State University Libraries University Park, PA 814-863-8333 http://www.libraries.psu.edu/psul/speccolls.html _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmargalo at udel.edu Thu Dec 4 10:37:13 2014 From: jmargalo at udel.edu (Jaime Margalotti) Date: Thu, 4 Dec 2014 10:37:13 -0500 Subject: [Archivesspace_Users_Group] Changing Labels of User Defined Fields Message-ID: I was able to successfully import data into the User Defined Fields for Accessions, but I haven't been able to figure out how to change their labels to something more useful. The help documentation doesn't provide any guidance. How do I make these changes? Thanks, Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From PGalligan at rockarch.org Thu Dec 4 10:42:54 2014 From: PGalligan at rockarch.org (Galligan, Patrick) Date: Thu, 4 Dec 2014 10:42:54 -0500 Subject: [Archivesspace_Users_Group] Changing Labels of User Defined Fields In-Reply-To: References: Message-ID: Jaime, You need to change the user_defined fields in en.yml. You can find the file in your locales directory. For instance: Change ?date_1: Date 1? to ?date_1: Delivery Date?. Hope that helps! Patrick Galligan Rockefeller Archive Center Assistant Digital Archivist 914-366-6386 From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Jaime Margalotti Sent: Thursday, December 04, 2014 10:37 AM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Changing Labels of User Defined Fields I was able to successfully import data into the User Defined Fields for Accessions, but I haven't been able to figure out how to change their labels to something more useful. The help documentation doesn't provide any guidance. How do I make these changes? Thanks, Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554 jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Thu Dec 4 10:44:49 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Thu, 4 Dec 2014 15:44:49 +0000 Subject: [Archivesspace_Users_Group] Changing Labels of User Defined Fields In-Reply-To: References: Message-ID: <1417707892802.55970@lyrasis.org> Hi Jaime, Have a look at this: https://github.com/archivesspace/archivesspace/wiki/Customizing-&-Theming-ArchivesSpace You'll need to edit label values in the locales/en.yml file for those user_defined fields ( starting a line #1672 ). b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Jaime Margalotti Sent: Thursday, December 4, 2014 4:37 PM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Changing Labels of User Defined Fields I was able to successfully import data into the User Defined Fields for Accessions, but I haven't been able to figure out how to change their labels to something more useful. The help documentation doesn't provide any guidance. How do I make these changes? Thanks, Jaime -- Jaime L. Margalotti Senior Assistant Librarian Manuscripts and Archives Department University of Delaware Library Newark, DE 19717-5267 302-831-0554 jmargalo at udel.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Clair at du.edu Thu Dec 4 10:48:08 2014 From: Kevin.Clair at du.edu (Kevin Clair) Date: Thu, 4 Dec 2014 15:48:08 +0000 Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' Message-ID: <1F1E72103798F04D96C355B296D7C61B401E07@mb2-uts.du.edu> Hello, We've been playing around with MARCXML exports from ArchivesSpace, and have noticed an issue with the resource records we get out. We have Agents set as subjects in collections with "Local sources" set as the source for the heading, but when we get the MARCXML out, it doesn't export a subfield 2 even though the second indicator is correctly set to '7'. Attached is a MARCXML record affected by this, as well as the EAC-CPF export for one of the 600 field entries in that record (to show that the source is set properly). Not sure if this is a bug or an issue with our data. thanks! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: codu_14096_Goldberg_eac.xml Type: text/xml Size: 1311 bytes Desc: codu_14096_Goldberg_eac.xml URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: B366_marc21.xml Type: text/xml Size: 4407 bytes Desc: B366_marc21.xml URL: From Kevin.Clair at du.edu Thu Dec 4 10:49:12 2014 From: Kevin.Clair at du.edu (Kevin Clair) Date: Thu, 4 Dec 2014 15:49:12 +0000 Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' In-Reply-To: <5486_1417708094_5480823E_5486_9695_5_1F1E72103798F04D96C355B296D7C61B401E07@mb2-uts.du.edu> References: <5486_1417708094_5480823E_5486_9695_5_1F1E72103798F04D96C355B296D7C61B401E07@mb2-uts.du.edu> Message-ID: <1F1E72103798F04D96C355B296D7C61B401E46@mb2-uts.du.edu> (I should add that this seems only to be affecting agents; when it's e.g. a 650 field with local sources, the subfield 2 outputs correctly.) From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Thursday, December 04, 2014 8:48 AM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' Hello, We've been playing around with MARCXML exports from ArchivesSpace, and have noticed an issue with the resource records we get out. We have Agents set as subjects in collections with "Local sources" set as the source for the heading, but when we get the MARCXML out, it doesn't export a subfield 2 even though the second indicator is correctly set to '7'. Attached is a MARCXML record affected by this, as well as the EAC-CPF export for one of the 600 field entries in that record (to show that the source is set properly). Not sure if this is a bug or an issue with our data. thanks! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad.westbrook at lyrasis.org Thu Dec 4 11:37:46 2014 From: brad.westbrook at lyrasis.org (Brad Westbrook) Date: Thu, 4 Dec 2014 16:37:46 +0000 Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' In-Reply-To: <1F1E72103798F04D96C355B296D7C61B401E46@mb2-uts.du.edu> References: <5486_1417708094_5480823E_5486_9695_5_1F1E72103798F04D96C355B296D7C61B401E07@mb2-uts.du.edu> <1F1E72103798F04D96C355B296D7C61B401E46@mb2-uts.du.edu> Message-ID: <5a76fa1aeb9743d2992a3b7091945e8c@DM2PR0801MB0862.namprd08.prod.outlook.com> Hi, Kevin, Could you please send a screen shot of how the linked "name record" appears in the staff interface? Thanks, Brad W. Bradley D. Westbrook Program Manager brad.westbrook at lyrasis.org 800.999.8558 x2910 678.235.2910 bradley_d_westbrook (Skype) [cid:image003.png at 01CE734E.FD759D30] From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Thursday, December 04, 2014 10:49 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' (I should add that this seems only to be affecting agents; when it's e.g. a 650 field with local sources, the subfield 2 outputs correctly.) From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Thursday, December 04, 2014 8:48 AM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' Hello, We've been playing around with MARCXML exports from ArchivesSpace, and have noticed an issue with the resource records we get out. We have Agents set as subjects in collections with "Local sources" set as the source for the heading, but when we get the MARCXML out, it doesn't export a subfield 2 even though the second indicator is correctly set to '7'. Attached is a MARCXML record affected by this, as well as the EAC-CPF export for one of the 600 field entries in that record (to show that the source is set properly). Not sure if this is a bug or an issue with our data. thanks! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 7645 bytes Desc: image001.png URL: From Kevin.Clair at du.edu Thu Dec 4 11:40:11 2014 From: Kevin.Clair at du.edu (Kevin Clair) Date: Thu, 4 Dec 2014 16:40:11 +0000 Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' In-Reply-To: <5a76fa1aeb9743d2992a3b7091945e8c@DM2PR0801MB0862.namprd08.prod.outlook.com> References: <5486_1417708094_5480823E_5486_9695_5_1F1E72103798F04D96C355B296D7C61B401E07@mb2-uts.du.edu> <1F1E72103798F04D96C355B296D7C61B401E46@mb2-uts.du.edu>, <5a76fa1aeb9743d2992a3b7091945e8c@DM2PR0801MB0862.namprd08.prod.outlook.com> Message-ID: <1F1E72103798F04D96C355B296D7C61B401FDA@mb2-uts.du.edu> Hi Brad, Attached. -k ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org [archivesspace_users_group-bounces at lyralists.lyrasis.org] on behalf of Brad Westbrook [brad.westbrook at lyrasis.org] Sent: Thursday, December 04, 2014 9:37 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' Hi, Kevin, Could you please send a screen shot of how the linked ?name record? appears in the staff interface? Thanks, Brad W. Bradley D. Westbrook Program Manager brad.westbrook at lyrasis.org 800.999.8558 x2910 678.235.2910 bradley_d_westbrook (Skype) [cid:image003.png at 01CE734E.FD759D30] From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Thursday, December 04, 2014 10:49 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' (I should add that this seems only to be affecting agents; when it?s e.g. a 650 field with local sources, the subfield 2 outputs correctly.) From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Thursday, December 04, 2014 8:48 AM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] 600 $2 not exporting 'local' when ind2='7' Hello, We?ve been playing around with MARCXML exports from ArchivesSpace, and have noticed an issue with the resource records we get out. We have Agents set as subjects in collections with ?Local sources? set as the source for the heading, but when we get the MARCXML out, it doesn?t export a subfield 2 even though the second indicator is correctly set to ?7?. Attached is a MARCXML record affected by this, as well as the EAC-CPF export for one of the 600 field entries in that record (to show that the source is set properly). Not sure if this is a bug or an issue with our data. thanks! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 7645 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2014-12-04 at 9.39.38 AM.png Type: image/png Size: 184908 bytes Desc: Screen Shot 2014-12-04 at 9.39.38 AM.png URL: From sdm7g at virginia.edu Thu Dec 4 15:01:50 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Thu, 4 Dec 2014 15:01:50 -0500 Subject: [Archivesspace_Users_Group] Puma caught this error: exceeded available parameter key space (RangeError) Message-ID: Ever seen this ? Any idea what this error message means ? ? Steve Majewski curl -H "X-ArchivesSpace-Session: 8ebd995c8155efcc516a53f99ed5315010fec1cd7fe606fb507b054c70482d0b" "-d @imports/UVA-SC/viu00005.json http://localhost:4567/repositories/2/batch_imports? Puma caught this error: exceeded available parameter key space (RangeError) /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:464:in `[]=' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:111:in `normalize_params' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:96:in `parse_nested_query' org/jruby/RubyArray.java:1613:in `each' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:93:in `parse_nested_query' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:332:in `parse_query' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:209:in `POST' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:221:in `params' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:789:in `call!' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:780:in `call' app/main.rb:249:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/nulllogger.rb:9:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/head.rb:9:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:124:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1499:in `synchronize' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:490:in `handle_request' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:488:in `handle_request' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:361:in `process_client' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:357:in `process_client' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:254:in `run' org/jruby/RubyProc.java:271:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/thread_pool.rb:92:in `spawn_thread'URL http://localhost:4567 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From Chris.Fitzpatrick at lyrasis.org Thu Dec 4 15:54:12 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Thu, 4 Dec 2014 20:54:12 +0000 Subject: [Archivesspace_Users_Group] Puma caught this error: exceeded available parameter key space (RangeError) In-Reply-To: References: Message-ID: <1417726452074.48088@lyrasis.org> Yeah, I actually have seen that before...it's a Rack thing, which is limiting the number of parameter keys submitted ( the default is limited 65536 ). uh...here's the description of it => https://github.com/rack/rack/issues/318 So, you could break up the JSON, or you could increase the Rack::Utils.key_space_limit to something much higher, although as the github issue suggests could leave you more open to a DOS attack ( since someone could just throw big wads of JSON at you and your web server will die just trying to serialize it ). Might be good to have this added to the config.rb so people can tweak the number when they want to do an import? Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Steven Majewski Sent: Thursday, December 4, 2014 9:01 PM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] Puma caught this error: exceeded available parameter key space (RangeError) Ever seen this ? Any idea what this error message means ? ? Steve Majewski curl -H "X-ArchivesSpace-Session: 8ebd995c8155efcc516a53f99ed5315010fec1cd7fe606fb507b054c70482d0b" "-d @imports/UVA-SC/viu00005.json http://localhost:4567/repositories/2/batch_imports? Puma caught this error: exceeded available parameter key space (RangeError) /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:464:in `[]=' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:111:in `normalize_params' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:96:in `parse_nested_query' org/jruby/RubyArray.java:1613:in `each' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/utils.rb:93:in `parse_nested_query' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:332:in `parse_query' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:209:in `POST' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/request.rb:221:in `params' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:789:in `call!' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:780:in `call' app/main.rb:249:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/nulllogger.rb:9:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/rack-1.4.5/lib/rack/head.rb:9:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:124:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1499:in `synchronize' /projects/Archivespace/dcs-archivesspace/build/gems/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:490:in `handle_request' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:488:in `handle_request' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:361:in `process_client' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:357:in `process_client' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/server.rb:254:in `run' org/jruby/RubyProc.java:271:in `call' /projects/Archivespace/dcs-archivesspace/build/gems/gems/puma-2.8.2-java/lib/puma/thread_pool.rb:92:in `spawn_thread'URL http://localhost:4567 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Thu Dec 4 16:26:22 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Thu, 4 Dec 2014 21:26:22 +0000 Subject: [Archivesspace_Users_Group] solr_home In-Reply-To: References: Message-ID: <1417728380692.58947@lyrasis.org> Hi Steven, Yeah, so Solr is having a hard time setting the "instanceDir" because the solrconfig and schema are actually bundled up in the WAR file's classpath. The rest of the admin ui's schema browser stuff should still work, but I think you'll have to monkey around a bit to get the admin ui to load the config and schema xml file....or you can look at them here : https://github.com/archivesspace/archivesspace/tree/master/solr b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Steven Majewski Sent: Tuesday, December 2, 2014 5:27 PM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] solr_home Indexing in my ArchivesSpace instances appear to be working properly, however when I go to the solr web console and click on either Schema or Config for Collection1, I see a stack trace: 5001java.lang.NullPointerException at org.apache.solr.handler.admin.ShowFileRequestHandler.showFromFileSystem(ShowFileRequestHandler.java:209) at org.apache.solr.handler.admin.ShowFileRequestHandler.handleRequestBody(ShowFileRequestHandler.java:122) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1302) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:448) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1067) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:377) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1001) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:360) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:744) 500 And the Dashboard shows a non-existing directory for solr_home and "Instance" : /usr/local/projects/archivesspace/data/solr_home/collection1 Just wondering what all that means! ( I was having a problem with a devserver instance, which was actually due to my forgetting to start the indexer:devserver, however, in hunting for the problem, I was misled by these error messages, until I discovered my functioning test server showed the same errors. ) - Steve Majewski / UVA Alderman Library -------------- next part -------------- An HTML attachment was scrubbed... URL: From ktasker at library.berkeley.edu Thu Dec 4 19:14:01 2014 From: ktasker at library.berkeley.edu (Kate Tasker) Date: Thu, 4 Dec 2014 16:14:01 -0800 Subject: [Archivesspace_Users_Group] XML export error message Message-ID: Hi All, Could anyone help decipher an error message I'm encountering when exporting an XML file? The file cuts off after the and displays "error on line 95 at column 8: Extra content at the end of the document". I'm guessing there must be something amiss in the container list but haven't been able to pinpoint the problem. We're usually able to export finding aids created in ASpace like this one without too much trouble. Has anyone seen this before? (By the way, the finding aid was created in ASpace v1.0.9 and I'm exporting it with numbered tags.) Any insight would be much appreciated, thanks! -Kate Kate Tasker Digital Collections Archivist The Bancroft Library University of California Berkeley, CA 94720-6000 -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah.huffman at duke.edu Fri Dec 5 09:20:24 2014 From: noah.huffman at duke.edu (Noah Huffman) Date: Fri, 5 Dec 2014 14:20:24 +0000 Subject: [Archivesspace_Users_Group] XML export error message In-Reply-To: References: Message-ID: Kate, I?m not sure this answers your question, but I?ve noticed that occasionally EAD records fail to export completely when exporting from the staff client. Basically, when you open the EAD file, you?ll notice that it terminates abruptly somewhere in the middle of the components. For example, it may end with something like . I just replicated this in version 1.1.0, using the option to export with numbered components and DAOs. It?s almost as if the application just times out during the export. FWIW, I?ve never had any problem getting complete EAD exports through the API. -Noah ================ Noah Huffman Archivist for Metadata and Encoding David M. Rubenstein Rare Book & Manuscript Library Duke University noah.huffman at duke.edu 919-660-5982 http://library.duke.edu/rubenstein/ From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kate Tasker Sent: Thursday, December 04, 2014 7:14 PM To: ArchivesSpace Users Group Subject: [Archivesspace_Users_Group] XML export error message Hi All, Could anyone help decipher an error message I'm encountering when exporting an XML file? The file cuts off after the and displays "error on line 95 at column 8: Extra content at the end of the document". I'm guessing there must be something amiss in the container list but haven't been able to pinpoint the problem. We're usually able to export finding aids created in ASpace like this one without too much trouble. Has anyone seen this before? (By the way, the finding aid was created in ASpace v1.0.9 and I'm exporting it with numbered tags.) Any insight would be much appreciated, thanks! -Kate Kate Tasker Digital Collections Archivist The Bancroft Library University of California Berkeley, CA 94720-6000 -------------- next part -------------- An HTML attachment was scrubbed... URL: From smithkr at mit.edu Fri Dec 5 14:06:26 2014 From: smithkr at mit.edu (Kari R Smith) Date: Fri, 5 Dec 2014 19:06:26 +0000 Subject: [Archivesspace_Users_Group] Still Time to Submit: Call for Proposals DUE SOON for IS&T Archiving 2015 Message-ID: <29F559819ACA9A4FBF208407D4B63ABBB92A1438@OC11expo28.exchange.mit.edu> Apologies for Cross-posting. Please forward this call as appropriate. - Kari Smith, Technical Program Chair, Archiving 2015 The IS&T Archiving Conference brings together a unique community of imaging novices and experts from libraries, archives, records management, and information technology institutions to discuss and explore the expanding field of digital archiving and preservation. Join us for four days of platform presentations, interactive poster sessions, tours, and short-courses. Prospective authors are invited to submit abstracts describing original work for presentation at the Archiving 2015 conference to be held at the Getty Center in LA May 19-22, 2015. [Conference Call at: http://www.imaging.org/ist/conferences/archiving/index.cfm The Program will cover the general fields of: * Digital Preservation: Infrastructure, repositories, and web harvesting and archiving * Creating and Preserving Dynamic Media: Sound, film, digital art * Imaging Technology: Including digital documentation and forensic analysis of art * Using Tools, Systems, and Services: Quality assurance, managing file formats including image compression, and digital forensics * Managing Content and Digital Curation: * Policies, processes, metrics for services, illustrating value and ROI, and systems * Access rights management * Data privacy and PII (personally identifiable information) * Share Economies and Partnerships * Innovative Software, Projects, and Services More information about the event and the template for submitting abstracts is available at http://imaging.org/ist/conferences/archiving/index.cfm A PDF version of the Call for Papers is available at http://imaging.org/ist/conferences/archiving/Archiving2015_Call_for_Papers.pdf **Please feel free to contact me if you have questions about a proposal you'd like to submit. Kari R. Smith, Technical Program Chair, Archiving 2015 http://www.imaging.org/ist/conferences/archiving/index.cfm Digital Archivist, MIT Libraries, Institute Archives and Special Collections 617-258-5568 | smithkr (at) mit.edu http://libraries.mit.edu/archives/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Mon Dec 8 08:22:38 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Mon, 8 Dec 2014 13:22:38 +0000 Subject: [Archivesspace_Users_Group] XML export error message In-Reply-To: References: , Message-ID: <1418044966568.58442@lyrasis.org> Hi, Yes, can you upgrade to v1.1.0? This resolved an issue with UTF8 characters causing problems with the export, which is what you might be running into now. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Noah Huffman Sent: Friday, December 5, 2014 3:20 PM To: ktasker at library.berkeley.edu; Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] XML export error message Kate, I'm not sure this answers your question, but I've noticed that occasionally EAD records fail to export completely when exporting from the staff client. Basically, when you open the EAD file, you'll notice that it terminates abruptly somewhere in the middle of the components. For example, it may end with something like . I just replicated this in version 1.1.0, using the option to export with numbered components and DAOs. It's almost as if the application just times out during the export. FWIW, I've never had any problem getting complete EAD exports through the API. -Noah ================ Noah Huffman Archivist for Metadata and Encoding David M. Rubenstein Rare Book & Manuscript Library Duke University noah.huffman at duke.edu 919-660-5982 http://library.duke.edu/rubenstein/ From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kate Tasker Sent: Thursday, December 04, 2014 7:14 PM To: ArchivesSpace Users Group Subject: [Archivesspace_Users_Group] XML export error message Hi All, Could anyone help decipher an error message I'm encountering when exporting an XML file? The file cuts off after the and displays "error on line 95 at column 8: Extra content at the end of the document". I'm guessing there must be something amiss in the container list but haven't been able to pinpoint the problem. We're usually able to export finding aids created in ASpace like this one without too much trouble. Has anyone seen this before? (By the way, the finding aid was created in ASpace v1.0.9 and I'm exporting it with numbered tags.) Any insight would be much appreciated, thanks! -Kate Kate Tasker Digital Collections Archivist The Bancroft Library University of California Berkeley, CA 94720-6000 -------------- next part -------------- An HTML attachment was scrubbed... URL: From christine.dibella at lyrasis.org Mon Dec 8 11:42:17 2014 From: christine.dibella at lyrasis.org (Christine DiBella) Date: Mon, 8 Dec 2014 16:42:17 +0000 Subject: [Archivesspace_Users_Group] FW: Public Components not loading on public resource under prefix In-Reply-To: References: Message-ID: <3a66f75b882c46eb9ccd0dc7657e2465@DM2PR0801MB0848.namprd08.prod.outlook.com> Forwarded on behalf of Sarah Reid. From: Reid, Sarah N. [mailto:reid.419 at osu.edu] Sent: Monday, December 8, 2014 11:37 AM To: archivesspace_users_group-bounces at lyralists.lyrasis.org Subject: Public Components not loading on public resource under prefix Hi, We had an issue with the Components section not loading when viewing a record from the Public side of ArchivesSpace. The Components section just had a continual loading icon. [cid:F988270F-B027-4BDA-B365-09F10414AF2E] After some troubleshooting, I found that it was because our version of ArchivesSpace is running under a prefix, and the URL gathering the Components data was missing a forward-slash after our prefix. Here's what the failing URL looked like, which was missing the '/' after 'collectionguides' https://library.osu.edu/collectionguidestree?uri=%2Frepositories%2F5%2Fresources%2F10. I was able to find the javascript responsible for building the URL and add a forward-slash to fix our local instance for now. Before I totally commit to that fix, though, I was wondering if there were there any steps I may have missed for building ArchivesSpace under a prefix that would fix this? Or is this a bug that could be fixed for future releases? Thanks, Sarah [cid:A9F74B0F-55C0-40B8-B1D8-C3419B88DDCC] Sarah Reid Systems Developer / Engineer The Ohio State University Libraries Information Technology 320E 18th Avenue Library, 175 W 18th Ave, Columbus, OH 43210 reid.419 at osu.edu library.osu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: osu-emailsig.png Type: image/png Size: 5675 bytes Desc: osu-emailsig.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2014-12-08 at 11.20.32 AM[1].png Type: image/png Size: 145133 bytes Desc: Screen Shot 2014-12-08 at 11.20.32 AM[1].png URL: From sdm7g at virginia.edu Mon Dec 8 17:20:14 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Mon, 8 Dec 2014 17:20:14 -0500 Subject: [Archivesspace_Users_Group] Problems & Strategies for importing very large EAD files Message-ID: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I?ve never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c? The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? ? Steve Majewski / UVA Alderman Library -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From ktasker at library.berkeley.edu Mon Dec 8 17:24:37 2014 From: ktasker at library.berkeley.edu (Kate Tasker) Date: Mon, 8 Dec 2014 14:24:37 -0800 Subject: [Archivesspace_Users_Group] XML export error message [ArchivesSpace Users Group Digest Vol 17, Issue 15] Message-ID: Hi Noah and Chris, Thanks for your replies. We'll be upgrading to v1.1.0 in the near future, so hopefully that will make the difference. Best, -Kate Kate Tasker Digital Collections Archivist The Bancroft Library University of California Berkeley, CA 94720-6000 On Mon, Dec 8, 2014 at 5:22 AM, Chris Fitzpatrick < Chris.Fitzpatrick at lyrasis.org> wrote: > Hi, > > > > Yes, can you upgrade to v1.1.0? This resolved an issue with UTF8 > characters causing problems with the export, which is what you might be > running into now. > > > b,chris. > > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > ------------------------------ > *From:* archivesspace_users_group-bounces at lyralists.lyrasis.org < > archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of > Noah Huffman > *Sent:* Friday, December 5, 2014 3:20 PM > *To:* ktasker at library.berkeley.edu; Archivesspace Users Group > *Subject:* Re: [Archivesspace_Users_Group] XML export error message > > > Kate, > > > > I?m not sure this answers your question, but I?ve noticed that > occasionally EAD records fail to export completely when exporting from the > staff client. Basically, when you open the EAD file, you?ll notice that it > terminates abruptly somewhere in the middle of the components. For > example, it may end with something like . > > > > I just replicated this in version 1.1.0, using the option to export with > numbered components and DAOs. > > > > It?s almost as if the application just times out during the export. > > > > FWIW, I?ve never had any problem getting complete EAD exports through the > API. > > > > -Noah > > > > ================ > > Noah Huffman > > Archivist for Metadata and Encoding > > David M. Rubenstein Rare Book & Manuscript Library > > Duke University > > noah.huffman at duke.edu > > 919-660-5982 > > http://library.duke.edu/rubenstein/ > > > > > > > > > > *From:* archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto: > archivesspace_users_group-bounces at lyralists.lyrasis.org] *On Behalf Of *Kate > Tasker > *Sent:* Thursday, December 04, 2014 7:14 PM > *To:* ArchivesSpace Users Group > *Subject:* [Archivesspace_Users_Group] XML export error message > > > > Hi All, > > > > Could anyone help decipher an error message I'm encountering when > exporting an XML file? The file cuts off after the and displays > "error on line 95 at column 8: Extra content at the end of the document". > I'm guessing there must be something amiss in the container list but > haven't been able to pinpoint the problem. We're usually able to export > finding aids created in ASpace like this one without too much trouble. Has > anyone seen this before? > > > > (By the way, the finding aid was created in ASpace v1.0.9 and I'm > exporting it with numbered tags.) > > > > Any insight would be much appreciated, thanks! > > > > -Kate > > > Kate Tasker > > Digital Collections Archivist > > The Bancroft Library > > University of California > > Berkeley, CA 94720-6000 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Tue Dec 9 03:28:56 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Tue, 9 Dec 2014 08:28:56 +0000 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> Message-ID: <1418113747338.66332@lyrasis.org> Hi Steven, Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). Try that and see if helps the issue.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace at googlegroups.com on behalf of Steven Majewski Sent: Monday, December 8, 2014 11:20 PM To: archivesspace_users_group at lyralists.lyrasis.org Cc: archivesspace at googlegroups.com Subject: [archivesspace] Problems & Strategies for importing very large EAD files The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I?ve never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c? The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? ? Steve Majewski / UVA Alderman Library -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. From reese.2179 at osu.edu Tue Dec 9 09:08:24 2014 From: reese.2179 at osu.edu (Reese, Terry P.) Date: Tue, 9 Dec 2014 14:08:24 +0000 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: <1418113747338.66332@lyrasis.org> References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> Message-ID: Chris, I wanted to follow up something Steven said -- is there documentation noting what ArchivesSpace uses in validation. We have a number of legacy finding aids that have loading issues based on validation (which is fine) -- but it would be nice to be able to create a validation file that we could use to validate data prior to uploading it. --tr -----Original Message----- From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chris Fitzpatrick Sent: Tuesday, December 9, 2014 3:29 AM To: archivesspace_users_group at lyralists.lyrasis.org; archivesspace at googlegroups.com Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files Hi Steven, Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). Try that and see if helps the issue.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace at googlegroups.com on behalf of Steven Majewski Sent: Monday, December 8, 2014 11:20 PM To: archivesspace_users_group at lyralists.lyrasis.org Cc: archivesspace at googlegroups.com Subject: [archivesspace] Problems & Strategies for importing very large EAD files The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased '-Xmx300m' to '-Xmx500m' in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... "}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I've never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides - sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c... The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn't help, and if we don't find any other way to improve import performance, I'm considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? - Steve Majewski / UVA Alderman Library -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group From christine.dibella at lyrasis.org Tue Dec 9 16:33:56 2014 From: christine.dibella at lyrasis.org (Christine Di Bella) Date: Tue, 9 Dec 2014 21:33:56 +0000 Subject: [Archivesspace_Users_Group] Fwd: Georgia Tech Archives and ArchivesSpace References: <1937120455.5281535.1418160436862.JavaMail.root@mail.gatech.edu> Message-ID: <96BEBE99-421C-4EBD-8F67-F255C7B5E4BB@lyrasis.org> Forwarded on behalf of Jody Thompson. Begin forwarded message: From: "Thompson, Jody A" > Date: December 9, 2014, 4:27:16 PM EST To: >, > Subject: Georgia Tech Archives and ArchivesSpace Hello. In ArchivesSpace's current state, the Georgia Tech Archives will not be adopting it in 2014. We have tested and evaluated ArchivesSpace and feel it lacks the following: -barcode and batch location control functionality is not robust enough yet in ASpace to support our collection move -functionality to manage digital objects in bulk without sys admin privileges is not robust enough yet to support our digital archives and repository work -admin and public UI are still confusing for staff (extremely nested structure of information, use of jargon in labels, etc.) and will be too confusing for researchers We will move to a high density storage facility in 2015, we must go with a system that can properly handle barcodes/batch function, and so for the meantime, we will continue to use Archivists' Toolkit. We look forward to learning about ASpace improvements and hope to adopt it in late 2015. Thank you, Jody Thompson -- Jody Lloyd Thompson Head, Archives and Records Management Dept. Library & Learning Excellence Georgia Institute of Technology Atlanta, Georgia 30332-0900 t:404-894-9626 f:404-894-9421 www.library.gatech.edu/archives/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bmg17 at psu.edu Wed Dec 10 16:16:58 2014 From: bmg17 at psu.edu (Ben Goldman) Date: Wed, 10 Dec 2014 16:16:58 -0500 (EST) Subject: [Archivesspace_Users_Group] EAD export In-Reply-To: <1352773390.366963.1418245842813.JavaMail.zimbra@psu.edu> Message-ID: <1399896150.369792.1418246218076.JavaMail.zimbra@psu.edu> Hi All, How configurable is the EAD export functionality in ASpace? In particular, I am wondering if it's possible to configure the naming convention of the exported XML files, and whether we could export with the stylesheet declared. Thanks, Ben Ben Goldman Digital Records Archivist Penn State University Libraries University Park, PA 814-863-8333 http://www.libraries.psu.edu/psul/speccolls.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Thu Dec 11 06:38:59 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Thu, 11 Dec 2014 11:38:59 +0000 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org>, Message-ID: <1418297940316.23031@lyrasis.org> Hi Terry, The mappings can be found here : http://archivesspace.org/importexport in a spreadsheet. But for the actual validation rules, the easiest way to look at these is by looking at the schema files, which you can see here : https://github.com/archivesspace/archivesspace/tree/master/common/schemas It's a nice declaritive syntax that I think is pretty easy to follow...for example, for resources, check out: https://github.com/archivesspace/archivesspace/blob/master/common/schemas/resource.rb which declares things like: "id_0" => {"type" => "string", "ifmissing" => "error", "maxLength" => 255}, ( first part of the 4 part id field is a string, will error if missing, and has a max length of 255 ) The other thing to check that resource is an extention of abstract_archival_object ( declared in the "parent" => "abstract_archival_object" ), so you can see additional validation rules in the abstract_archival_object schema ( https://github.com/archivesspace/archivesspace/blob/master/common/schemas/abstract_archival_object.rb ). It does seem like it might be a good idea to have a tool that checks the EAD for imports as well...might try and work on that next week. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Reese, Terry P. Sent: Tuesday, December 9, 2014 3:08 PM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files Chris, I wanted to follow up something Steven said -- is there documentation noting what ArchivesSpace uses in validation. We have a number of legacy finding aids that have loading issues based on validation (which is fine) -- but it would be nice to be able to create a validation file that we could use to validate data prior to uploading it. --tr -----Original Message----- From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Chris Fitzpatrick Sent: Tuesday, December 9, 2014 3:29 AM To: archivesspace_users_group at lyralists.lyrasis.org; archivesspace at googlegroups.com Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files Hi Steven, Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). Try that and see if helps the issue.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace at googlegroups.com on behalf of Steven Majewski Sent: Monday, December 8, 2014 11:20 PM To: archivesspace_users_group at lyralists.lyrasis.org Cc: archivesspace at googlegroups.com Subject: [archivesspace] Problems & Strategies for importing very large EAD files The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased '-Xmx300m' to '-Xmx500m' in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... "}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I've never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides - sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c... The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn't help, and if we don't find any other way to improve import performance, I'm considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? - Steve Majewski / UVA Alderman Library -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group From sdm7g at virginia.edu Thu Dec 11 12:43:22 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Thu, 11 Dec 2014 12:43:22 -0500 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: <1418113747338.66332@lyrasis.org> References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> Message-ID: I increased memory sizes in JAVA_OPTS: JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" This for a command line program running: EADConverter.new( eadxml ) Backend server also running on the same machine because I called JSONModel:init in client_mode, and it wants a server to talk to. I fixed the date problem in my xml that caused it to be incomplete on the previous run. I added some debug trace statements in the JSONModel validate calls to track progress. ( Comparing the amount of debug output vs. the size of JSON output, I don?t think this had a significant effect on performance, but I could be wrong. ) Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 memory and SSD. I ran Activity Monitor during the run to check on CPU and Memory usage, which seemed to peak at around 700GB. The process seemed to run at 100% +/- 3% CPU. I ran another parse operation in parallel on another smaller file for a time. This didn?t seem to have a bit impact: both cores were running close to 100% rather than just one. ( I assume there isn?t much parallelism to be wrung out of the parse operation. ) That 2nd file was just over 6MB and took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t nearby when it finished. Real time was much greater, but I?m not sure that the laptop didn?t sleep for part of that time. Time for the 14MB file was over 24 hours CPU time (greater than that in actual real clock time). The previous attempt ( with less memory allocated ) was 24 hours or more in real time ? I did not measure CPU time on that attempt ? but it got a validation error and the resulting JSON file was incomplete : there were internal URI references that were never defined in the JSON file. After completion, I attempted to POST it to backend server /repositories/$REPO/batch_imports. First to the backend development instance running on my laptop, then to our actual test server. That attempt to post to the test server took too long to finish while at the office, and my VPN at home usually times out after an hour or two, so I copied the JSON file up to the server and did the ?nohup curl_as? to the backend port on the same server, and left it running overnight. So success, but there was an incredible amount of overhead in the parsing and validation. Any ideas where the bottleneck is in this operation ? IF the overhead is due to the validation (and here, I?m just guessing), that would be another argument for validating the EAD against a schema before JSON parsing, and perhaps providing a ?trusted path? parser that skips that validation. I?m learning my way around the code base, but not quite sure I?m ready to tackle that myself yet! I did run the code with 'raise_errors=false? , but that still runs the validations. We may attempt this again in the future with more memory and better instrumentation, but for now, I just needed to determine whether we could actually successfully ingest our largest guides. ? Steve Majewski / UVA Alderman Library On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick wrote: > > > Hi Steven, > > Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. > > If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). > > Try that and see if helps the issue.. > b,chris. > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > > ________________________________________ > From: archivesspace at googlegroups.com on behalf of Steven Majewski > Sent: Monday, December 8, 2014 11:20 PM > To: archivesspace_users_group at lyralists.lyrasis.org > Cc: archivesspace at googlegroups.com > Subject: [archivesspace] Problems & Strategies for importing very large EAD files > > The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. > > I have tried importing these guides from the frontend webapp import: my time or patience has always been > hit after a couple of hours, before they have managed to import successfully. > > I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, > and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple > of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or > more of sleep time while my laptop was in-transit ). This failed with the error: > > failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> > > which I eventually traced to this: > May 1916-June 1916 > > > But before fixing this and trying again I wanted to ask if others had experience with importing large files. > > Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? > > 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. > Does this type of performance seem expected ? Is there anything else that should be reconfigured ? > > [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that > describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and > I?ve never seen an XSLT stylesheet take that long to validate. ] > > > Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: > xxxx-a, xxxx-b, xxxx-c? > The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace > to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? > > If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering > disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to > the guide a few at a time via the backend API. > > Anyone have any other ideas or experience to report ? > > > ? Steve Majewski / UVA Alderman Library > > > > > > -- > You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. > To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From sdm7g at virginia.edu Thu Dec 11 13:07:42 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Thu, 11 Dec 2014 13:07:42 -0500 Subject: [Archivesspace_Users_Group] ArchivesSpace admin/frontend URLs are context dependent Message-ID: <8329DD40-17EC-4EB4-B330-EBBE974B0489@virginia.edu> Not sure if it?s a bug or a mis-feature: We just discovered that ArchivesSpace admin/frontend URLs are context dependent on which repo is selected. When I send a URL to a colleague and they attempt to open it, if they have a different repo selected from the one I have selected, they get the error: Record Not Found The record you've tried to access may no longer exist or you may not have permission to view it. This is going to make collaboration somewhat awkward! The public links don?t have that context dependency, and to link back to the frontend EDIT uri, the EDIT button uses an indirect link: http://archives-test.lib.virginia.edu:8080/resolve/edit?uri=/repositories/8/resources/3978&autoselect_repo=true ? Steve Majewski / UVA Alderman Library -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From ns96 at nyu.edu Thu Dec 11 14:08:09 2014 From: ns96 at nyu.edu (Nathan Stevens) Date: Thu, 11 Dec 2014 14:08:09 -0500 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> Message-ID: You should try importing those EADs into the AT (www.archiviststoolkit.org) to establish some kind of baseline since those import times seems way too long for relatively small datasets. For example, migration of a 600MB AT database into ASpace takes under 5 hours. You can then use the AT migration plugin to transfer those records over to ASpace and see how long that takes. You send a couple of those EAD to me directly (ns96 at nyu.edu) and I can try importing into AT, and transferring to ASpace on our end. On Thu, Dec 11, 2014 at 12:43 PM, Steven Majewski wrote: > > > I increased memory sizes in JAVA_OPTS: > > JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" > > This for a command line program running: EADConverter.new( eadxml ) > Backend server also running on the same machine because I called > JSONModel:init in client_mode, > and it wants a server to talk to. > > I fixed the date problem in my xml that caused it to be incomplete on the > previous run. > I added some debug trace statements in the JSONModel validate calls to > track progress. > ( Comparing the amount of debug output vs. the size of JSON output, I > don?t think this > had a significant effect on performance, but I could be wrong. ) > > Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 > memory and SSD. > > I ran Activity Monitor during the run to check on CPU and Memory usage, > which seemed to peak at around 700GB. > The process seemed to run at 100% +/- 3% CPU. I ran another parse > operation in parallel on another smaller file > for a time. This didn?t seem to have a bit impact: both cores were running > close to 100% rather than just one. > ( I assume there isn?t much parallelism to be wrung out of the parse > operation. ) That 2nd file was just over 6MB and > took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t > nearby when it finished. Real time was much > greater, but I?m not sure that the laptop didn?t sleep for part of that > time. Time for the 14MB file was over 24 hours > CPU time (greater than that in actual real clock time). The previous > attempt ( with less memory allocated ) was > 24 hours or more in real time ? I did not measure CPU time on that attempt > ? but it got a validation error and the > resulting JSON file was incomplete : there were internal URI references > that were never defined in the JSON file. > > > After completion, I attempted to POST it to backend server > /repositories/$REPO/batch_imports. First to the backend > development instance running on my laptop, then to our actual test server. > That attempt to post to the test server took > too long to finish while at the office, and my VPN at home usually times > out after an hour or two, so I copied the JSON > file up to the server and did the ?nohup curl_as? to the backend port on > the same server, and left it running overnight. > > So success, but there was an incredible amount of overhead in the parsing > and validation. > > Any ideas where the bottleneck is in this operation ? > > IF the overhead is due to the validation (and here, I?m just guessing), > that would be another argument for validating > the EAD against a schema before JSON parsing, and perhaps providing a > ?trusted path? parser that skips that validation. > > > I?m learning my way around the code base, but not quite sure I?m ready to > tackle that myself yet! > I did run the code with 'raise_errors=false? , but that still runs the > validations. > > We may attempt this again in the future with more memory and better > instrumentation, but for now, I just needed to > determine whether we could actually successfully ingest our largest > guides. > > > ? Steve Majewski / UVA Alderman Library > > > On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick < > Chris.Fitzpatrick at lyrasis.org> wrote: > > > > Hi Steven, > > Yeah, you're probably going to want to have the -Xmx at around the 1G > default. I've dropped down to 512M ( which is what the > sandbox.archivesspace.org runs on ), but it'll be rather sluggish. > > If can go that high, you can start the application with the public and > index apps disabled, which will reduce some of the overhead ( especially > the indexer ). > > Try that and see if helps the issue.. > b,chris. > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > > ________________________________________ > From: archivesspace at googlegroups.com on > behalf of Steven Majewski > Sent: Monday, December 8, 2014 11:20 PM > To: archivesspace_users_group at lyralists.lyrasis.org > Cc: archivesspace at googlegroups.com > Subject: [archivesspace] Problems & Strategies for importing very large > EAD files > > The majority of our EAD guides are less than 1MB in size. We have a few > larger ones, the two largest being around 13-14 MB. > > I have tried importing these guides from the frontend webapp import: my > time or patience has always been > hit after a couple of hours, before they have managed to import > successfully. > > I recently attempted importing the largest file again by calling > EADConverter.new( eadfile ) from a ruby command line program, > and writing out the resulting json file to later import using the backend > API. The first attempt ran out of memory after a couple > of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd > attempt took just over 24 hours ( with an hour or > more of sleep time while my laptop was in-transit ). This failed with the > error: > > failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be > before begin"]}, :import_context=>" ... ?}> > > which I eventually traced to this: > May 1916-June 1916 > > > But before fixing this and trying again I wanted to ask if others had > experience with importing large files. > > Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize > as well ? > > 24 hours for 14MB does not seem to be a linear projection from the 3 or > 4MB files I have managed to import. > Does this type of performance seem expected ? Is there anything else > that should be reconfigured ? > > [ Waiting 24 hours for an error message, then repeat again, seems like > another good reason to want a schema that > describes the flavor of EAD that ArchivesSpace accepts: it would be nice > to be able to do a pre-flight check, and > I?ve never seen an XSLT stylesheet take that long to validate. ] > > > Also: a lot of those smaller EAD files are collections that are split up > into separated guides ? sometimes due to separate accessions: > xxxx-a, xxxx-b, xxxx-c? > The more recent tendency has been to unify a collection into a single > guide, and we were expecting to perhaps use ArchivesSpace > to unify some of these separate guides after they had been separately > imported. Should we be reconsidering that plan ? > > If increasing memory further doesn?t help, and if we don?t find any other > way to improve import performance, I?m considering > disassembling the file: strip out much of the and import a skeleton > guide, and then process the , adding resources to > the guide a few at a time via the backend API. > > Anyone have any other ideas or experience to report ? > > > ? Steve Majewski / UVA Alderman Library > > > > > > -- > You received this message because you are subscribed to the Google Groups > "ArchivesSpace" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to archivesspace+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From russellr at rice.edu Thu Dec 11 14:25:37 2014 From: russellr at rice.edu (Rebecca Russell) Date: Thu, 11 Dec 2014 13:25:37 -0600 Subject: [Archivesspace_Users_Group] Importing CSV files for digital objects Message-ID: <5489EFB1.1060103@rice.edu> Apologies if this issue/question has been covered before... I am attempting to import a csv file for digital objects that have a file version accessible online (from our Institutional Repository). I have been unsuccessful in determining the correct import field header that will map to the File version field. We have thousands of digital objects that we would like to import from our IR into ArchivesSpace and would like to avoid having to manually input the uri data. Many thanks, Rebecca -- Rebecca Russell, CA Archivist / Special Collections Librarian Woodson Research Center RA Hanszen College Rice University russellr at rice.edu 713-348-5133 -------------- next part -------------- An HTML attachment was scrubbed... URL: From PGalligan at rockarch.org Thu Dec 11 14:38:08 2014 From: PGalligan at rockarch.org (Galligan, Patrick) Date: Thu, 11 Dec 2014 14:38:08 -0500 Subject: [Archivesspace_Users_Group] Importing CSV files for digital objects In-Reply-To: <5489EFB1.1060103@rice.edu> References: <5489EFB1.1060103@rice.edu> Message-ID: I'd be interested in an updated digital object template for CSV as well. Specifically one that we can use to import full file version info, date and date expression, object type, METS identifier, Actuate/Show value, and different notes. Is this possible at all? Patrick Galligan Rockefeller Archive Center Assistant Digital Archivist 914-366-6386 From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Rebecca Russell Sent: Thursday, December 11, 2014 2:26 PM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Importing CSV files for digital objects Apologies if this issue/question has been covered before... I am attempting to import a csv file for digital objects that have a file version accessible online (from our Institutional Repository). I have been unsuccessful in determining the correct import field header that will map to the File version field. We have thousands of digital objects that we would like to import from our IR into ArchivesSpace and would like to avoid having to manually input the uri data. Many thanks, Rebecca -- Rebecca Russell, CA Archivist / Special Collections Librarian Woodson Research Center RA Hanszen College Rice University russellr at rice.edu 713-348-5133 -------------- next part -------------- An HTML attachment was scrubbed... URL: From PGalligan at rockarch.org Thu Dec 11 16:19:25 2014 From: PGalligan at rockarch.org (Galligan, Patrick) Date: Thu, 11 Dec 2014 16:19:25 -0500 Subject: [Archivesspace_Users_Group] Collection Management enumeration error Message-ID: Hi, We're getting an error where the Processing Priority is throwing back "translation missing: en.enumerations.collection_management_processing_priority.none" in our migrated accession records. Is there a change I can make in the enumerations to get this to display differently? Thanks, Patrick Galligan Rockefeller Archive Center Assistant Digital Archivist 914-366-6386 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tommy at library.caltech.edu Thu Dec 11 20:02:24 2014 From: tommy at library.caltech.edu (Tommy Ingulfsen) Date: Fri, 12 Dec 2014 01:02:24 +0000 Subject: [Archivesspace_Users_Group] adding a new field? Message-ID: <0C3D8CDFA307EB49B9FAA73625FAC3480106C7C2EE@CLSX9.cls.caltech.edu> Is there a somewhat easy way to add a brand new field to a record type in ArchivesSpace? Or add a new value to an enumerated type field? My immediate need is to add a couple of values to the note type field for agents (specifically the "Note Type" drop down list for persons). Thanks! tommy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tommy at library.caltech.edu Thu Dec 11 20:09:12 2014 From: tommy at library.caltech.edu (Tommy Ingulfsen) Date: Fri, 12 Dec 2014 01:09:12 +0000 Subject: [Archivesspace_Users_Group] help documentation? Message-ID: <0C3D8CDFA307EB49B9FAA73625FAC3480106C7C2FD@CLSX9.cls.caltech.edu> When I click on the little question mark icons in my ArchivesSpace installation it takes me to docs.archivesspace.org and asks for a login. Thinking that this could be a useful thing, I registered yesterday and am still waiting for the confirmation email to use my account (there is no spam filter on the email address I provided). Would anyone be able to hook me up with a working account? Thank you - in advance tommy -------------- next part -------------- An HTML attachment was scrubbed... URL: From christine.dibella at lyrasis.org Thu Dec 11 21:10:15 2014 From: christine.dibella at lyrasis.org (Christine Di Bella) Date: Fri, 12 Dec 2014 02:10:15 +0000 Subject: [Archivesspace_Users_Group] user documentation for ArchivesSpace Message-ID: Hi everyone, Since it's come up a few times recently and it never hurts to give it a boost: the user documentation for ArchivesSpace is available through the authentication system at http://docs.archivesspace.org, which is also linked from the ArchivesSpace website under Services --> Member Services. Each member institution has one or more people designated in the authentication system as administrators. (This designation happens initially at the time of enrollment and can be updated anytime.) These administrators can set up accounts for anyone else in their institution. Building on the efforts of the User Documentation subgroup of the Users Advisory Council, you will see enhancements and improvements to the documentation in the months to come. If you have any difficulties using the authentication system, or if you would like assistance in setting up an account, please contact me and I'll be glad to help. Best, Christine Christine Di Bella Program Assistant christine.dibella at lyrasis.org 800.999.8558 x2905 678-235-2905 cdibella13 (Skype) [cid:image003.png at 01CE734E.FD759D30] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 7645 bytes Desc: image001.png URL: From Chris.Fitzpatrick at lyrasis.org Fri Dec 12 03:29:24 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Fri, 12 Dec 2014 08:29:24 +0000 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> , Message-ID: <1418372964251.47235@lyrasis.org> Hi Steve, Ok, so you're running the process in the jirb console? That will be not very effectient, as the irb mode is not really built with performance in mind... Validating it against a schema is not a bad idea, but we don't have an XML schema to validate it against. The EAD schema is very loose in terms of what it allows, so having something that's EAD valid won't really be helpful. Someone would need to make a ArchivesSpace XML schema that we can validate against...b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace at googlegroups.com on behalf of Nathan Stevens Sent: Thursday, December 11, 2014 8:08 PM To: Archivesspace Users Group Cc: archivesspace at googlegroups.com Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files You should try importing those EADs into the AT (www.archiviststoolkit.org) to establish some kind of baseline since those import times seems way too long for relatively small datasets. For example, migration of a 600MB AT database into ASpace takes under 5 hours. You can then use the AT migration plugin to transfer those records over to ASpace and see how long that takes. You send a couple of those EAD to me directly (ns96 at nyu.edu) and I can try importing into AT, and transferring to ASpace on our end. On Thu, Dec 11, 2014 at 12:43 PM, Steven Majewski > wrote: I increased memory sizes in JAVA_OPTS: JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" This for a command line program running: EADConverter.new( eadxml ) Backend server also running on the same machine because I called JSONModel:init in client_mode, and it wants a server to talk to. I fixed the date problem in my xml that caused it to be incomplete on the previous run. I added some debug trace statements in the JSONModel validate calls to track progress. ( Comparing the amount of debug output vs. the size of JSON output, I don?t think this had a significant effect on performance, but I could be wrong. ) Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 memory and SSD. I ran Activity Monitor during the run to check on CPU and Memory usage, which seemed to peak at around 700GB. The process seemed to run at 100% +/- 3% CPU. I ran another parse operation in parallel on another smaller file for a time. This didn?t seem to have a bit impact: both cores were running close to 100% rather than just one. ( I assume there isn?t much parallelism to be wrung out of the parse operation. ) That 2nd file was just over 6MB and took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t nearby when it finished. Real time was much greater, but I?m not sure that the laptop didn?t sleep for part of that time. Time for the 14MB file was over 24 hours CPU time (greater than that in actual real clock time). The previous attempt ( with less memory allocated ) was 24 hours or more in real time ? I did not measure CPU time on that attempt ? but it got a validation error and the resulting JSON file was incomplete : there were internal URI references that were never defined in the JSON file. After completion, I attempted to POST it to backend server /repositories/$REPO/batch_imports. First to the backend development instance running on my laptop, then to our actual test server. That attempt to post to the test server took too long to finish while at the office, and my VPN at home usually times out after an hour or two, so I copied the JSON file up to the server and did the ?nohup curl_as? to the backend port on the same server, and left it running overnight. So success, but there was an incredible amount of overhead in the parsing and validation. Any ideas where the bottleneck is in this operation ? IF the overhead is due to the validation (and here, I?m just guessing), that would be another argument for validating the EAD against a schema before JSON parsing, and perhaps providing a ?trusted path? parser that skips that validation. I?m learning my way around the code base, but not quite sure I?m ready to tackle that myself yet! I did run the code with 'raise_errors=false? , but that still runs the validations. We may attempt this again in the future with more memory and better instrumentation, but for now, I just needed to determine whether we could actually successfully ingest our largest guides. ? Steve Majewski / UVA Alderman Library On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick > wrote: Hi Steven, Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). Try that and see if helps the issue.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace at googlegroups.com > on behalf of Steven Majewski > Sent: Monday, December 8, 2014 11:20 PM To: archivesspace_users_group at lyralists.lyrasis.org Cc: archivesspace at googlegroups.com Subject: [archivesspace] Problems & Strategies for importing very large EAD files The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I?ve never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c? The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? ? Steve Majewski / UVA Alderman Library -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Fri Dec 12 03:37:35 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Fri, 12 Dec 2014 08:37:35 +0000 Subject: [Archivesspace_Users_Group] ArchivesSpace admin/frontend URLs are context dependent In-Reply-To: <8329DD40-17EC-4EB4-B330-EBBE974B0489@virginia.edu> References: <8329DD40-17EC-4EB4-B330-EBBE974B0489@virginia.edu> Message-ID: <1418373456299.90293@lyrasis.org> Hi Steve, Yes, that's a feature. The idea, I think, is in the staff UI the select repository serves as the context. So searching and browsing is restricted to the selected repository. However, I do see a use for having a specific staff ui link that can be shared and redirect to directly to an object...currently you'd have to provide the link and say what repository it's in. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Steven Majewski Sent: Thursday, December 11, 2014 7:07 PM To: Archivesspace Users Group Subject: [Archivesspace_Users_Group] ArchivesSpace admin/frontend URLs are context dependent Not sure if it's a bug or a mis-feature: We just discovered that ArchivesSpace admin/frontend URLs are context dependent on which repo is selected. When I send a URL to a colleague and they attempt to open it, if they have a different repo selected from the one I have selected, they get the error: Record Not Found The record you've tried to access may no longer exist or you may not have permission to view it. This is going to make collaboration somewhat awkward! The public links don't have that context dependency, and to link back to the frontend EDIT uri, the EDIT button uses an indirect link: http://archives-test.lib.virginia.edu:8080/resolve/edit?uri=/repositories/8/resources/3978&autoselect_repo=true - Steve Majewski / UVA Alderman Library -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Fri Dec 12 15:56:20 2014 From: sdm7g at virginia.edu (Steven Majewski) Date: Fri, 12 Dec 2014 15:56:20 -0500 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: <1418372964251.47235@lyrasis.org> References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> , <1418372964251.47235@lyrasis.org> Message-ID: <48467E89-BA9E-445E-B84E-35D3C228C8C2@virginia.edu> On Dec 12, 2014, at 3:29 AM, Chris Fitzpatrick wrote: > > Hi Steve, > > > Ok, so you're running the process in the jirb console? > That will be not very effectient, as the irb mode is not really built with performance in mind... > No, not running in the jirb console ? just a script running from jruby. Here is the code: ( A slight variation on a previously posted version: trying to figure out the correct environment in which to run jobs like this. Using frontend config and client_mode because I was intending to add code to do the batch_imports POST directly. ) require 'frontend/config/boot' require 'frontend/config/application' require 'jsonmodel' require 'backend/app/converters/ead_converter' require 'fileutils' require 'backend/app/lib/logging' BACKEND_URL='http://localhost:4567/' def json_path(xmlpath) xmlpath.slice(0..xmlpath.rindex('.xml')) + 'json' if xmlpath.end_with? ".xml" end JSONModel::init( :strict => false, :client_mode => true, :url => BACKEND_URL ) ARGV.each do |eadxml| begin puts eadxml converter = EADConverter.new( eadxml ) converter.run puts eadxml + " : ok." $stderr.puts '#@@++ ' + eadxml + " : ok." rescue Exception => e puts eadxml + " : failed: " + e.message $stderr.puts e.backtrace $stderr.puts '#@@-- ' + eadxml + " : failed: " + e.message ensure FileUtils::move( converter.get_output_path, json_path(eadxml), :verbose => true ) end end > Validating it against a schema is not a bad idea, but we don't have an XML schema to validate it against. The EAD schema is very loose in terms of what it allows, so having something that's EAD valid won't really be helpful. Someone would need to make a ArchivesSpace XML schema that we can validate against...b,chris. > > Extending or restricting the EAD schema is not very difficult if you know what rules you need to add. The problem is that those rules are undocumented except in the code. So far, we?ve been discovering them by a trial and error experimental process. I?m only just now getting familiar enough with the code base to attempt to reverse engineer them from the code. XML Schemas are just one possible way of documenting those rules. You could argue that the JSONModel schemas are another form of documentation. However, it seems to take quite a lot of cross referencing across several schemas ( due to inheritance and inclusions ) and the mappings elsewhere in the code to get a full picture. Sometimes XML Schemas or DTDs present similar cross-referencing problems, but they are older and more standardized formats and there are tools for dealing with them. Are there any tools, for example, to expand those JSONModel schemas inline into something more readable ? ? Steve Majewski > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > From: archivesspace at googlegroups.com on behalf of Nathan Stevens > Sent: Thursday, December 11, 2014 8:08 PM > To: Archivesspace Users Group > Cc: archivesspace at googlegroups.com > Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files > > You should try importing those EADs into the AT (www.archiviststoolkit.org) to establish some kind of baseline since those import times seems way too long for relatively small datasets. For example, migration of a 600MB AT database into ASpace takes under 5 hours. You can then use the AT migration plugin to transfer those records over to ASpace and see how long that takes. > > You send a couple of those EAD to me directly (ns96 at nyu.edu) and I can try importing into AT, and transferring to ASpace on our end. > > On Thu, Dec 11, 2014 at 12:43 PM, Steven Majewski wrote: > > I increased memory sizes in JAVA_OPTS: > > JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" > > This for a command line program running: EADConverter.new( eadxml ) > Backend server also running on the same machine because I called JSONModel:init in client_mode, > and it wants a server to talk to. > > I fixed the date problem in my xml that caused it to be incomplete on the previous run. > I added some debug trace statements in the JSONModel validate calls to track progress. > ( Comparing the amount of debug output vs. the size of JSON output, I don?t think this > had a significant effect on performance, but I could be wrong. ) > > Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 memory and SSD. > > I ran Activity Monitor during the run to check on CPU and Memory usage, which seemed to peak at around 700GB. > The process seemed to run at 100% +/- 3% CPU. I ran another parse operation in parallel on another smaller file > for a time. This didn?t seem to have a bit impact: both cores were running close to 100% rather than just one. > ( I assume there isn?t much parallelism to be wrung out of the parse operation. ) That 2nd file was just over 6MB and > took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t nearby when it finished. Real time was much > greater, but I?m not sure that the laptop didn?t sleep for part of that time. Time for the 14MB file was over 24 hours > CPU time (greater than that in actual real clock time). The previous attempt ( with less memory allocated ) was > 24 hours or more in real time ? I did not measure CPU time on that attempt ? but it got a validation error and the > resulting JSON file was incomplete : there were internal URI references that were never defined in the JSON file. > > > After completion, I attempted to POST it to backend server /repositories/$REPO/batch_imports. First to the backend > development instance running on my laptop, then to our actual test server. That attempt to post to the test server took > too long to finish while at the office, and my VPN at home usually times out after an hour or two, so I copied the JSON > file up to the server and did the ?nohup curl_as? to the backend port on the same server, and left it running overnight. > > So success, but there was an incredible amount of overhead in the parsing and validation. > > Any ideas where the bottleneck is in this operation ? > > IF the overhead is due to the validation (and here, I?m just guessing), that would be another argument for validating > the EAD against a schema before JSON parsing, and perhaps providing a ?trusted path? parser that skips that validation. > > > I?m learning my way around the code base, but not quite sure I?m ready to tackle that myself yet! > I did run the code with 'raise_errors=false? , but that still runs the validations. > > We may attempt this again in the future with more memory and better instrumentation, but for now, I just needed to > determine whether we could actually successfully ingest our largest guides. > > > ? Steve Majewski / UVA Alderman Library > > > On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick wrote: > >> >> >> Hi Steven, >> >> Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. >> >> If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). >> >> Try that and see if helps the issue.. >> b,chris. >> >> Chris Fitzpatrick | Developer, ArchivesSpace >> Skype: chrisfitzpat | Phone: 918.236.6048 >> http://archivesspace.org/ >> >> ________________________________________ >> From: archivesspace at googlegroups.com on behalf of Steven Majewski >> Sent: Monday, December 8, 2014 11:20 PM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Cc: archivesspace at googlegroups.com >> Subject: [archivesspace] Problems & Strategies for importing very large EAD files >> >> The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. >> >> I have tried importing these guides from the frontend webapp import: my time or patience has always been >> hit after a couple of hours, before they have managed to import successfully. >> >> I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, >> and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple >> of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or >> more of sleep time while my laptop was in-transit ). This failed with the error: >> >> failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> >> >> which I eventually traced to this: >> May 1916-June 1916 >> >> >> But before fixing this and trying again I wanted to ask if others had experience with importing large files. >> >> Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? >> >> 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. >> Does this type of performance seem expected ? Is there anything else that should be reconfigured ? >> >> [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that >> describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and >> I?ve never seen an XSLT stylesheet take that long to validate. ] >> >> >> Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: >> xxxx-a, xxxx-b, xxxx-c? >> The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace >> to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? >> >> If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering >> disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to >> the guide a few at a time via the backend API. >> >> Anyone have any other ideas or experience to report ? >> >> >> ? Steve Majewski / UVA Alderman Library >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. >> To unsubscribe from this group and stop receiving emails from it, send an email toarchivesspace+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > > -- > Nathan Stevens > Programmer/Analyst > Digital Library Technology Services > New York University > > 1212-998-2653 > ns96 at nyu.edu > -- > You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. > To unsubscribe from this group and stop receiving emails from it, send an email toarchivesspace+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4943 bytes Desc: not available URL: From ns96 at nyu.edu Sat Dec 13 15:42:52 2014 From: ns96 at nyu.edu (Nathan Stevens) Date: Sat, 13 Dec 2014 15:42:52 -0500 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: <48467E89-BA9E-445E-B84E-35D3C228C8C2@virginia.edu> References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> <1418372964251.47235@lyrasis.org> <48467E89-BA9E-445E-B84E-35D3C228C8C2@virginia.edu> Message-ID: All Testing was conducted on a windows 7 32 bit/Core i5 3.2Gz/4GB machine using Oracles JRE1.7.0_67 and ASpace v1.1.0 First. the provided EAD file needed some slight modifications to validate against the EAD schema. Once validated, import fails about 2.5 hours with the error message below. For comparison, the same EAD imported into AT in about 50 seconds (not a typo) and can be migrated to the same ASpace instance in 12 minutes using the migration plugin. Not sure where the bottle neck is, but clearly the ASpace EAD importer needs some work to match the AT importer. viuh00010.xml ================================================== !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORT ERROR !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! The following errors were found: extents : At least 1 item(s) is required title : Property is required but was missing id_0 : Property is required but was missing level : Property is required but was missing For JSONModel(:resource): #"resource", "external_ids"=>[], "subjects"=>[], "linked_events"=>[], "extents"=>[], "dates"=>[], "external_documents"=>[], "rights_statements"=>[], "linked_agents"=>[], "restrictions"=>false, "instances"=>[], "deaccessions"=>[], "related_accessions"=>[], "notes"=>[], "uri"=>"/repositories/import/resources/import_082e59ec-ae98-4c49-984a-91bdfb90a9e5", "finding_aid_title"=>"A Guide to the Philip S. Hench Walter Reed Yellow Fever Collectioncirca 1800- circa 1998", "finding_aid_author"=>"William B. Bean, Donna L. Purvis, Mark Mones, Henry K. Sharp, Janet Pearson, and Dan Cavanaugh.", "finding_aid_edition_statement"=>"3rd edition\n This edition reflects substantial changes made to the 2nd edition finding aid. The repository staff made these changes to prepare the digital collection for inclusion in the University of Virginia Library's digital repository. ", "finding_aid_language"=>"Description is inEnglish", "language"=>"eng"}> In : ... On Fri, Dec 12, 2014 at 3:56 PM, Steven Majewski wrote: > > > On Dec 12, 2014, at 3:29 AM, Chris Fitzpatrick < > Chris.Fitzpatrick at lyrasis.org> wrote: > > > Hi Steve, > > > Ok, so you're running the process in the jirb console? > That will be not very effectient, as the irb mode is not really built with > performance in mind... > > > No, not running in the jirb console ? just a script running from jruby. > Here is the code: > ( A slight variation on a previously posted version: trying to figure out > the correct environment in which to run jobs like this. > Using frontend config and client_mode because I was intending to add > code to do the batch_imports POST directly. ) > > require 'frontend/config/boot' > require 'frontend/config/application' > require 'jsonmodel' > require 'backend/app/converters/ead_converter' > require 'fileutils' > require 'backend/app/lib/logging' > > BACKEND_URL='http://localhost:4567/' > > def json_path(xmlpath) > xmlpath.slice(0..xmlpath.rindex('.xml')) + 'json' if > xmlpath.end_with? ".xml" > end > > > JSONModel::init( :strict => false, :client_mode => true, :url => > BACKEND_URL ) > > ARGV.each do |eadxml| > begin > puts eadxml > converter = EADConverter.new( eadxml ) > converter.run > puts eadxml + " : ok." > $stderr.puts '#@@++ ' + eadxml + " : ok." > rescue Exception => e > puts eadxml + " : failed: " + e.message > $stderr.puts e.backtrace > $stderr.puts '#@@-- ' + eadxml + " : failed: " + e.message > ensure > FileUtils::move( converter.get_output_path, json_path(eadxml), > :verbose => true ) > end > end > > > > > Validating it against a schema is not a bad idea, but we don't have an XML > schema to validate it against. The EAD schema is very loose in terms of > what it allows, so having something that's EAD valid won't really be > helpful. Someone would need to make a ArchivesSpace XML schema that we can > validate against...b,chris. > > > > > Extending or restricting the EAD schema is not very difficult if you know > what rules you need to add. > The problem is that those rules are undocumented except in the code. > So far, we?ve been discovering them by a trial and error experimental > process. > I?m only just now getting familiar enough with the code base to attempt to > reverse engineer them from the code. > > XML Schemas are just one possible way of documenting those rules. > You could argue that the JSONModel schemas are another form of > documentation. > However, it seems to take quite a lot of cross referencing across several > schemas ( due to inheritance and inclusions ) > and the mappings elsewhere in the code to get a full picture. > Sometimes XML Schemas or DTDs present similar cross-referencing problems, > but they are older and more > standardized formats and there are tools for dealing with them. Are there > any tools, for example, to expand > those JSONModel schemas inline into something more readable ? > > > ? Steve Majewski > > > > Chris Fitzpatrick | Developer, ArchivesSpace > Skype: chrisfitzpat | Phone: 918.236.6048 > http://archivesspace.org/ > ------------------------------ > *From:* archivesspace at googlegroups.com > on behalf of Nathan Stevens > *Sent:* Thursday, December 11, 2014 8:08 PM > *To:* Archivesspace Users Group > *Cc:* archivesspace at googlegroups.com > *Subject:* Re: [Archivesspace_Users_Group] [archivesspace] Problems & > Strategies for importing very large EAD files > > You should try importing those EADs into the AT (www.archiviststoolkit.org) > to establish some kind of baseline since those import times seems way too > long for relatively small datasets. For example, migration of a 600MB AT > database into ASpace takes under 5 hours. You can then use the AT > migration plugin to transfer those records over to ASpace and see how long > that takes. > > You send a couple of those EAD to me directly (ns96 at nyu.edu) and I can > try importing into AT, and transferring to ASpace on our end. > > On Thu, Dec 11, 2014 at 12:43 PM, Steven Majewski > wrote: >> >> >> I increased memory sizes in JAVA_OPTS: >> >> JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" >> >> This for a command line program running: EADConverter.new( eadxml ) >> Backend server also running on the same machine because I called >> JSONModel:init in client_mode, >> and it wants a server to talk to. >> >> I fixed the date problem in my xml that caused it to be incomplete on the >> previous run. >> I added some debug trace statements in the JSONModel validate calls to >> track progress. >> ( Comparing the amount of debug output vs. the size of JSON output, I >> don?t think this >> had a significant effect on performance, but I could be wrong. ) >> >> Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 >> memory and SSD. >> >> I ran Activity Monitor during the run to check on CPU and Memory usage, >> which seemed to peak at around 700GB. >> The process seemed to run at 100% +/- 3% CPU. I ran another parse >> operation in parallel on another smaller file >> for a time. This didn?t seem to have a bit impact: both cores were >> running close to 100% rather than just one. >> ( I assume there isn?t much parallelism to be wrung out of the parse >> operation. ) That 2nd file was just over 6MB and >> took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t >> nearby when it finished. Real time was much >> greater, but I?m not sure that the laptop didn?t sleep for part of that >> time. Time for the 14MB file was over 24 hours >> CPU time (greater than that in actual real clock time). The previous >> attempt ( with less memory allocated ) was >> 24 hours or more in real time ? I did not measure CPU time on that >> attempt ? but it got a validation error and the >> resulting JSON file was incomplete : there were internal URI references >> that were never defined in the JSON file. >> >> >> After completion, I attempted to POST it to backend server >> /repositories/$REPO/batch_imports. First to the backend >> development instance running on my laptop, then to our actual test >> server. That attempt to post to the test server took >> too long to finish while at the office, and my VPN at home usually times >> out after an hour or two, so I copied the JSON >> file up to the server and did the ?nohup curl_as? to the backend port on >> the same server, and left it running overnight. >> >> So success, but there was an incredible amount of overhead in the parsing >> and validation. >> >> Any ideas where the bottleneck is in this operation ? >> >> IF the overhead is due to the validation (and here, I?m just guessing), >> that would be another argument for validating >> the EAD against a schema before JSON parsing, and perhaps providing a >> ?trusted path? parser that skips that validation. >> >> >> I?m learning my way around the code base, but not quite sure I?m ready to >> tackle that myself yet! >> I did run the code with 'raise_errors=false? , but that still runs the >> validations. >> >> We may attempt this again in the future with more memory and better >> instrumentation, but for now, I just needed to >> determine whether we could actually successfully ingest our largest >> guides. >> >> >> ? Steve Majewski / UVA Alderman Library >> >> >> On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick < >> Chris.Fitzpatrick at lyrasis.org> wrote: >> >> >> >> Hi Steven, >> >> Yeah, you're probably going to want to have the -Xmx at around the 1G >> default. I've dropped down to 512M ( which is what the >> sandbox.archivesspace.org runs on ), but it'll be rather sluggish. >> >> If can go that high, you can start the application with the public and >> index apps disabled, which will reduce some of the overhead ( especially >> the indexer ). >> >> Try that and see if helps the issue.. >> b,chris. >> >> Chris Fitzpatrick | Developer, ArchivesSpace >> Skype: chrisfitzpat | Phone: 918.236.6048 >> http://archivesspace.org/ >> >> ________________________________________ >> From: archivesspace at googlegroups.com on >> behalf of Steven Majewski >> Sent: Monday, December 8, 2014 11:20 PM >> To: archivesspace_users_group at lyralists.lyrasis.org >> Cc: archivesspace at googlegroups.com >> Subject: [archivesspace] Problems & Strategies for importing very large >> EAD files >> >> The majority of our EAD guides are less than 1MB in size. We have a few >> larger ones, the two largest being around 13-14 MB. >> >> I have tried importing these guides from the frontend webapp import: my >> time or patience has always been >> hit after a couple of hours, before they have managed to import >> successfully. >> >> I recently attempted importing the largest file again by calling >> EADConverter.new( eadfile ) from a ruby command line program, >> and writing out the resulting json file to later import using the backend >> API. The first attempt ran out of memory after a couple >> of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd >> attempt took just over 24 hours ( with an hour or >> more of sleep time while my laptop was in-transit ). This failed with >> the error: >> >> failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be >> before begin"]}, :import_context=>" ... ?}> >> >> which I eventually traced to this: >> May 1916-June 1916 >> >> >> But before fixing this and trying again I wanted to ask if others had >> experience with importing large files. >> >> Should I try increasing -Xmx memory radically more, or perhaps >> MaxPermSize as well ? >> >> 24 hours for 14MB does not seem to be a linear projection from the 3 or >> 4MB files I have managed to import. >> Does this type of performance seem expected ? Is there anything else >> that should be reconfigured ? >> >> [ Waiting 24 hours for an error message, then repeat again, seems like >> another good reason to want a schema that >> describes the flavor of EAD that ArchivesSpace accepts: it would be nice >> to be able to do a pre-flight check, and >> I?ve never seen an XSLT stylesheet take that long to validate. ] >> >> >> Also: a lot of those smaller EAD files are collections that are split up >> into separated guides ? sometimes due to separate accessions: >> xxxx-a, xxxx-b, xxxx-c? >> The more recent tendency has been to unify a collection into a single >> guide, and we were expecting to perhaps use ArchivesSpace >> to unify some of these separate guides after they had been separately >> imported. Should we be reconsidering that plan ? >> >> If increasing memory further doesn?t help, and if we don?t find any other >> way to improve import performance, I?m considering >> disassembling the file: strip out much of the and import a >> skeleton guide, and then process the , adding resources to >> the guide a few at a time via the backend API. >> >> Anyone have any other ideas or experience to report ? >> >> >> ? Steve Majewski / UVA Alderman Library >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ArchivesSpace" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email toarchivesspace+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> > > -- > Nathan Stevens > Programmer/Analyst > Digital Library Technology Services > New York University > > 1212-998-2653 > ns96 at nyu.edu > > -- > You received this message because you are subscribed to the Google Groups > "ArchivesSpace" group. > To unsubscribe from this group and stop receiving emails from it, send an > email toarchivesspace+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From danelle.moon at sjsu.edu Mon Dec 15 14:58:32 2014 From: danelle.moon at sjsu.edu (Danelle Moon) Date: Mon, 15 Dec 2014 11:58:32 -0800 Subject: [Archivesspace_Users_Group] Contiued EAD export issues Message-ID: Chris, I sent you a message last week off list, and I need to get back to me addressing the plugin failure and are continued problem getting full EAD to export. I sent you an example in my message, and we need you to contact us to set-up a webx meeting to review. Please respond ASAP. Thanks Danelle Danelle Moon Full Librarian Director SJSU Special Collections One Washington Square San Jose, CA 95192-0028 408/808-2061 office 408/808-2063 fax Danelle.Moon at sjsu.ed u -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Fitzpatrick at lyrasis.org Mon Dec 15 18:31:39 2014 From: Chris.Fitzpatrick at lyrasis.org (Chris Fitzpatrick) Date: Mon, 15 Dec 2014 23:31:39 +0000 Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files In-Reply-To: References: <71F50386-EBFE-411A-8AD1-8D9888B6D367@virginia.edu> <1418113747338.66332@lyrasis.org> <1418372964251.47235@lyrasis.org> <48467E89-BA9E-445E-B84E-35D3C228C8C2@virginia.edu>, Message-ID: Hi, Yes, I actually work expect this. We did some work to improve performance on the migration endpoints, but I don't think we've done any on the EAD importer. For example, I'd bet a donut that one bottleneck is the component reordering issue we ran into for migration ( when adding a component the tree is reorder in the db, since the normal use case done via the UI would assume this is necessary. ) Also, the AT and archon ead importers are certainly much more mature at this point ,so I think for large ead it's not a bad idea to do as Nathan suggest,if your a at or archon institution. B,Chris. Sent from my HTC ----- Reply message ----- From: "Nathan Stevens" To: "Archivesspace Users Group" Cc: "archivesspace at googlegroups.com" Subject: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files Date: Sat, Dec 13, 2014 21:42 All Testing was conducted on a windows 7 32 bit/Core i5 3.2Gz/4GB machine using Oracles JRE1.7.0_67 and ASpace v1.1.0 First. the provided EAD file needed some slight modifications to validate against the EAD schema. Once validated, import fails about 2.5 hours with the error message below. For comparison, the same EAD imported into AT in about 50 seconds (not a typo) and can be migrated to the same ASpace instance in 12 minutes using the migration plugin. Not sure where the bottle neck is, but clearly the ASpace EAD importer needs some work to match the AT importer. viuh00010.xml ================================================== !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! IMPORT ERROR !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! The following errors were found: extents : At least 1 item(s) is required title : Property is required but was missing id_0 : Property is required but was missing level : Property is required but was missing For JSONModel(:resource): #"resource", "external_ids"=>[], "subjects"=>[], "linked_events"=>[], "extents"=>[], "dates"=>[], "external_documents"=>[], "rights_statements"=>[], "linked_agents"=>[], "restrictions"=>false, "instances"=>[], "deaccessions"=>[], "related_accessions"=>[], "notes"=>[], "uri"=>"/repositories/import/resources/import_082e59ec-ae98-4c49-984a-91bdfb90a9e5", "finding_aid_title"=>"A Guide to the Philip S. Hench Walter Reed Yellow Fever Collectioncirca 1800- circa 1998", "finding_aid_author"=>"William B. Bean, Donna L. Purvis, Mark Mones, Henry K. Sharp, Janet Pearson, and Dan Cavanaugh.", "finding_aid_edition_statement"=>"3rd edition\n This edition reflects substantial changes made to the 2nd edition finding aid. The repository staff made these changes to prepare the digital collection for inclusion in the University of Virginia Library's digital repository. ", "finding_aid_language"=>"Description is inEnglish", "language"=>"eng"}> In : ... On Fri, Dec 12, 2014 at 3:56 PM, Steven Majewski > wrote: On Dec 12, 2014, at 3:29 AM, Chris Fitzpatrick > wrote: Hi Steve, Ok, so you're running the process in the jirb console? That will be not very effectient, as the irb mode is not really built with performance in mind... No, not running in the jirb console ? just a script running from jruby. Here is the code: ( A slight variation on a previously posted version: trying to figure out the correct environment in which to run jobs like this. Using frontend config and client_mode because I was intending to add code to do the batch_imports POST directly. ) require 'frontend/config/boot' require 'frontend/config/application' require 'jsonmodel' require 'backend/app/converters/ead_converter' require 'fileutils' require 'backend/app/lib/logging' BACKEND_URL='http://localhost:4567/' def json_path(xmlpath) xmlpath.slice(0..xmlpath.rindex('.xml')) + 'json' if xmlpath.end_with? ".xml" end JSONModel::init( :strict => false, :client_mode => true, :url => BACKEND_URL ) ARGV.each do |eadxml| begin puts eadxml converter = EADConverter.new( eadxml ) converter.run puts eadxml + " : ok." $stderr.puts '#@@++ ' + eadxml + " : ok." rescue Exception => e puts eadxml + " : failed: " + e.message $stderr.puts e.backtrace $stderr.puts '#@@-- ' + eadxml + " : failed: " + e.message ensure FileUtils::move( converter.get_output_path, json_path(eadxml), :verbose => true ) end end Validating it against a schema is not a bad idea, but we don't have an XML schema to validate it against. The EAD schema is very loose in terms of what it allows, so having something that's EAD valid won't really be helpful. Someone would need to make a ArchivesSpace XML schema that we can validate against...b,chris. Extending or restricting the EAD schema is not very difficult if you know what rules you need to add. The problem is that those rules are undocumented except in the code. So far, we?ve been discovering them by a trial and error experimental process. I?m only just now getting familiar enough with the code base to attempt to reverse engineer them from the code. XML Schemas are just one possible way of documenting those rules. You could argue that the JSONModel schemas are another form of documentation. However, it seems to take quite a lot of cross referencing across several schemas ( due to inheritance and inclusions ) and the mappings elsewhere in the code to get a full picture. Sometimes XML Schemas or DTDs present similar cross-referencing problems, but they are older and more standardized formats and there are tools for dealing with them. Are there any tools, for example, to expand those JSONModel schemas inline into something more readable ? ? Steve Majewski Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________ From: archivesspace at googlegroups.com > on behalf of Nathan Stevens > Sent: Thursday, December 11, 2014 8:08 PM To: Archivesspace Users Group Cc: archivesspace at googlegroups.com Subject: Re: [Archivesspace_Users_Group] [archivesspace] Problems & Strategies for importing very large EAD files You should try importing those EADs into the AT (www.archiviststoolkit.org) to establish some kind of baseline since those import times seems way too long for relatively small datasets. For example, migration of a 600MB AT database into ASpace takes under 5 hours. You can then use the AT migration plugin to transfer those records over to ASpace and see how long that takes. You send a couple of those EAD to me directly (ns96 at nyu.edu) and I can try importing into AT, and transferring to ASpace on our end. On Thu, Dec 11, 2014 at 12:43 PM, Steven Majewski > wrote: I increased memory sizes in JAVA_OPTS: JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=512m -Xmx1080m -Xss8m" This for a command line program running: EADConverter.new( eadxml ) Backend server also running on the same machine because I called JSONModel:init in client_mode, and it wants a server to talk to. I fixed the date problem in my xml that caused it to be incomplete on the previous run. I added some debug trace statements in the JSONModel validate calls to track progress. ( Comparing the amount of debug output vs. the size of JSON output, I don?t think this had a significant effect on performance, but I could be wrong. ) Running on a MacBook Pro w/ 2 GHz Intel Core i7 and 16 GB 1600 MHz DDR3 memory and SSD. I ran Activity Monitor during the run to check on CPU and Memory usage, which seemed to peak at around 700GB. The process seemed to run at 100% +/- 3% CPU. I ran another parse operation in parallel on another smaller file for a time. This didn?t seem to have a bit impact: both cores were running close to 100% rather than just one. ( I assume there isn?t much parallelism to be wrung out of the parse operation. ) That 2nd file was just over 6MB and took over 6 hours of CPU time. ( Not sure of the exact time, as I wasn?t nearby when it finished. Real time was much greater, but I?m not sure that the laptop didn?t sleep for part of that time. Time for the 14MB file was over 24 hours CPU time (greater than that in actual real clock time). The previous attempt ( with less memory allocated ) was 24 hours or more in real time ? I did not measure CPU time on that attempt ? but it got a validation error and the resulting JSON file was incomplete : there were internal URI references that were never defined in the JSON file. After completion, I attempted to POST it to backend server /repositories/$REPO/batch_imports. First to the backend development instance running on my laptop, then to our actual test server. That attempt to post to the test server took too long to finish while at the office, and my VPN at home usually times out after an hour or two, so I copied the JSON file up to the server and did the ?nohup curl_as? to the backend port on the same server, and left it running overnight. So success, but there was an incredible amount of overhead in the parsing and validation. Any ideas where the bottleneck is in this operation ? IF the overhead is due to the validation (and here, I?m just guessing), that would be another argument for validating the EAD against a schema before JSON parsing, and perhaps providing a ?trusted path? parser that skips that validation. I?m learning my way around the code base, but not quite sure I?m ready to tackle that myself yet! I did run the code with 'raise_errors=false? , but that still runs the validations. We may attempt this again in the future with more memory and better instrumentation, but for now, I just needed to determine whether we could actually successfully ingest our largest guides. ? Steve Majewski / UVA Alderman Library On Dec 9, 2014, at 3:28 AM, Chris Fitzpatrick > wrote: Hi Steven, Yeah, you're probably going to want to have the -Xmx at around the 1G default. I've dropped down to 512M ( which is what the sandbox.archivesspace.org runs on ), but it'll be rather sluggish. If can go that high, you can start the application with the public and index apps disabled, which will reduce some of the overhead ( especially the indexer ). Try that and see if helps the issue.. b,chris. Chris Fitzpatrick | Developer, ArchivesSpace Skype: chrisfitzpat | Phone: 918.236.6048 http://archivesspace.org/ ________________________________________ From: archivesspace at googlegroups.com > on behalf of Steven Majewski > Sent: Monday, December 8, 2014 11:20 PM To: archivesspace_users_group at lyralists.lyrasis.org Cc: archivesspace at googlegroups.com Subject: [archivesspace] Problems & Strategies for importing very large EAD files The majority of our EAD guides are less than 1MB in size. We have a few larger ones, the two largest being around 13-14 MB. I have tried importing these guides from the frontend webapp import: my time or patience has always been hit after a couple of hours, before they have managed to import successfully. I recently attempted importing the largest file again by calling EADConverter.new( eadfile ) from a ruby command line program, and writing out the resulting json file to later import using the backend API. The first attempt ran out of memory after a couple of hours. I increased ?-Xmx300m? to ?-Xmx500m? in the JAVA_OPTS. The 2nd attempt took just over 24 hours ( with an hour or more of sleep time while my laptop was in-transit ). This failed with the error: failed: #<:ValidationException: {:errors=>{"dates/0/end"=>["must not be before begin"]}, :import_context=>" ... ?}> which I eventually traced to this: May 1916-June 1916 But before fixing this and trying again I wanted to ask if others had experience with importing large files. Should I try increasing -Xmx memory radically more, or perhaps MaxPermSize as well ? 24 hours for 14MB does not seem to be a linear projection from the 3 or 4MB files I have managed to import. Does this type of performance seem expected ? Is there anything else that should be reconfigured ? [ Waiting 24 hours for an error message, then repeat again, seems like another good reason to want a schema that describes the flavor of EAD that ArchivesSpace accepts: it would be nice to be able to do a pre-flight check, and I?ve never seen an XSLT stylesheet take that long to validate. ] Also: a lot of those smaller EAD files are collections that are split up into separated guides ? sometimes due to separate accessions: xxxx-a, xxxx-b, xxxx-c? The more recent tendency has been to unify a collection into a single guide, and we were expecting to perhaps use ArchivesSpace to unify some of these separate guides after they had been separately imported. Should we be reconsidering that plan ? If increasing memory further doesn?t help, and if we don?t find any other way to improve import performance, I?m considering disassembling the file: strip out much of the and import a skeleton guide, and then process the , adding resources to the guide a few at a time via the backend API. Anyone have any other ideas or experience to report ? ? Steve Majewski / UVA Alderman Library -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email toarchivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email toarchivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -- You received this message because you are subscribed to the Google Groups "ArchivesSpace" group. To unsubscribe from this group and stop receiving emails from it, send an email to archivesspace+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Clair at du.edu Thu Dec 18 16:59:36 2014 From: Kevin.Clair at du.edu (Kevin Clair) Date: Thu, 18 Dec 2014 21:59:36 +0000 Subject: [Archivesspace_Users_Group] success with jasper reports? Message-ID: <1F1E72103798F04D96C355B296D7C61B40B4D2@mb2-uts.du.edu> Hello, Has anyone on the list successfully managed to get a Jasper report working in ArchivesSpace? I have a couple of simple ones I've written just to see how it works, but after following the instructions on Github they're still not displaying in the staff UI for me. Not sure if there's something wrong with the way I have ArchivesSpace configured (we're on v1.1.0), or if it's another issue. thanks, -k -------------- next part -------------- An HTML attachment was scrubbed... URL: From EJOLLEY at nla.gov.au Thu Dec 18 17:18:26 2014 From: EJOLLEY at nla.gov.au (Emma Jolley) Date: Thu, 18 Dec 2014 22:18:26 +0000 Subject: [Archivesspace_Users_Group] success with jasper reports? In-Reply-To: <1F1E72103798F04D96C355B296D7C61B40B4D2@mb2-uts.du.edu> References: <1F1E72103798F04D96C355B296D7C61B40B4D2@mb2-uts.du.edu> Message-ID: <81FF938BA2407B4DA1E134E7FA5C09EC016CD80B8D@EXMBX1.shire.nla.gov.au> Hi All I have been waiting for the procedures/guidelines on how to generate reports in ArchivesSpace. Did I miss an email advertising these? Many thanks Emma Emma Jolley| Curator of Digital Archives, Pictures and Manuscripts Branch|National Library of Australia Canberra ACT 2600 e: emma.jolley at nla.gov.au|t: 02 6262 1456| www.nla.gov.au/ms http://www.nla.gov.au/support-us/make-a-collection-offer-pictures-and-manuscripts From: archivesspace_users_group-bounces at lyralists.lyrasis.org [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] On Behalf Of Kevin Clair Sent: Friday, 19 December 2014 9:00 AM To: Archivesspace Users Group (archivesspace_users_group at lyralists.lyrasis.org) Subject: [Archivesspace_Users_Group] success with jasper reports? Hello, Has anyone on the list successfully managed to get a Jasper report working in ArchivesSpace? I have a couple of simple ones I've written just to see how it works, but after following the instructions on Github they're still not displaying in the staff UI for me. Not sure if there's something wrong with the way I have ArchivesSpace configured (we're on v1.1.0), or if it's another issue. thanks, -k -------------- next part -------------- An HTML attachment was scrubbed... URL: From cornelison at mitre.org Fri Dec 19 07:25:53 2014 From: cornelison at mitre.org (Cornelison, Lee) Date: Fri, 19 Dec 2014 12:25:53 +0000 Subject: [Archivesspace_Users_Group] Java CLASSPATH startup issues Message-ID: Chris, we are installing ArchivesSpace 1.1 on a Windows server, where we are using the procrun approach to run as a Windows service against a remote MySQL database. When we launch ArchivesSpace as a foreground service (running the archivesspace.bat script) the app starts up fine. When we try to start up the ArchivesSpace as the service instance your scripts created, I get the following errors: "db_url": "jdbc:mysql://mysqlsrv2.mitre.org:3306/archivesspace?user=archivesspace&[SECRET]useUnicode=true&characterEncoding=UTF-8", D:/archivesspace/gems/gems/jdbc-mysql-5.1.13/lib/jdbc/mysql.rb:3 warning: already initialized constant VERSION E, [2014-12-18T13:01:02.544000 #5164] ERROR -- : Thread-3238: DB connection failed: missing class or uppercase package name (`com.mysql.jdbc.Driver') E, [2014-12-18T13:01:03.153000 #5164] ERROR -- : Thread-3238: ***** DATABASE CONNECTION FAILED ***** I know the MySQL connector is OK, and the Windows %JAVA_HOME% variable is being read. Any ideas how to solve this Java class path issue? Lee Cornelison Electronic Records Management Service Architect R504 - Electronic Content and Records Services The MITRE Corporation | Bedford, MA 202 Burlington Road, Bedford, MA 01730 1-781-271-5051 | cornelison at mitre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 92 bytes Desc: image001.gif URL: From ns96 at nyu.edu Fri Dec 19 09:39:43 2014 From: ns96 at nyu.edu (Nathan Stevens) Date: Fri, 19 Dec 2014 09:39:43 -0500 Subject: [Archivesspace_Users_Group] success with jasper reports? In-Reply-To: <1F1E72103798F04D96C355B296D7C61B40B4D2@mb2-uts.du.edu> References: <1F1E72103798F04D96C355B296D7C61B40B4D2@mb2-uts.du.edu> Message-ID: Hi, We are working on streamlining the reports workflow for the next version of ASpace. Take a look at https://github.com/archivesspace/archivesspace/tree/master/reports for more information. P.S. The current instructions for setting up reports is incomplete so that why it's not working. On Thu, Dec 18, 2014 at 4:59 PM, Kevin Clair wrote: > > Hello, > > > > Has anyone on the list successfully managed to get a Jasper report working > in ArchivesSpace? I have a couple of simple ones I?ve written just to see > how it works, but after following the instructions on Github they?re still > not displaying in the staff UI for me. Not sure if there?s something wrong > with the way I have ArchivesSpace configured (we?re on v1.1.0), or if it?s > another issue. > > > > thanks, > > -k > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > -- Nathan Stevens Programmer/Analyst Digital Library Technology Services New York University 1212-998-2653 ns96 at nyu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkottman at ku.edu Mon Dec 22 13:48:03 2014 From: mkottman at ku.edu (Kottman, Miloche) Date: Mon, 22 Dec 2014 18:48:03 +0000 Subject: [Archivesspace_Users_Group] Enhancement suggestion: Expanding data fields Message-ID: <220B4CB68E19834D965E60AC94D69BD823F29DCA@EXCH10-MBX-05.home.ku.edu> I have a suggestion that will help when editing existing resource records. When creating a new resource record, the Notes fields can be enlarged by dragging the bottom right corner of the data field. However, when editing a resource record that has already been saved, the functionality to expand the field is lost. I've only noticed the loss of this functionality on Notes fields. [cid:image001.jpg at 01D01DE5.86546AE0] --Miloche Kottman University of Kansas Libraries -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 7581 bytes Desc: image001.jpg URL: From angela.spinazze at lyrasis.org Mon Dec 22 16:37:32 2014 From: angela.spinazze at lyrasis.org (Angela Spinazze) Date: Mon, 22 Dec 2014 21:37:32 +0000 Subject: [Archivesspace_Users_Group] Happy Holidays and Thanks for a terrific 2014! Message-ID: ArchivesSpace Community, As 2014 comes to a close, I would like to thank you for your commitment to and support of ArchivesSpace. This year has brought with it many changes, including my participation in ArchivesSpace in my new role as Senior Director of Collaborative Programs at LYRASIS. Since late summer, I have been working primarily behind the scenes to learn from and support the program staff, provide guidance and leadership, and introduce some new project management practices. As a member of LYRASIS? management team, I serve on the ArchivesSpace Board. My experience includes more than 20 years of project management: facilitating community design conversations, managing community supported software development, and implementing software to manage collections for a variety of museums and special collections holding organizations around the world. I would like to share with you a bit about what we?ve been working on and what is in store for the year ahead. Our first priority is to be of service to you. As most of you already know, the program team has expanded. As of January 5th, we will be a team of six. * Christine DiBella * Chris Fitzpatrick * Brian Hoffman * Angela Spinazz? * Nathan Stevens * Brad Westbrook In addition, there are over a dozen LYRASIS colleagues who provide support and service to us and to you each day. We heard from many of you this year about your needs. Thank you! You said that you need: * More regular communications * Greater transparency * More structure around software development and releases Towards that end, we have been working to streamline our processes, eliminate redundant systems, and update our project management tools. The visible results in 2015 will include: * Greater use of the wiki and email lists for communications (community calendar, development roadmap, council activities, etc.) * A move to Jira for all software development activities (planning, voting, tasks, bugs, etc.) * Greater member engagement (we will be seeking your input on the roadmap and engaging council members to help us refine priorities, in addition to current activities) * And more! We ask for your patience as we rollout some new approaches and support tools in the new year. I am proud to lead your program team and to contribute to the continued positive growth of ArchivesSpace with such a dynamic group of colleagues. And, whilst I have had the opportunity of meeting some of you in 2014, I look forward to meeting many more of you in 2015. Best wishes for a peaceful holiday, Angela Please note: LYRASIS offices are closed from December 24th (2:00pm EST) through January 4, 2015. We?ll be back in the office on Monday, January 5th! **Apologies for cross-postings** ========================= Angela Spinazz? Senior Director of Collaborative Programs and CollectionSpace Program Director LYRASIS email: angela.spinazze at lyrasis.org voice: +1 800-999-8558 x2922 www.archivesspace.org www.collectionspace.org www.lyrasis.org -------------- next part -------------- An HTML attachment was scrubbed... URL: