From Kevin.Clair at du.edu Thu Jul 11 10:22:27 2019 From: Kevin.Clair at du.edu (Kevin Clair) Date: Thu, 11 Jul 2019 14:22:27 +0000 Subject: [Archivesspace_Users_Group] question on fine-tuning reports Message-ID: <8EF74095-4357-45D9-AA42-76CCD1DF2881@du.edu> Good morning, We have a handful of custom reports in ArchivesSpace. When I run them in the test instance on my local computer, they finish running in a matter of seconds or minutes; when I run them on our (hosted) production instance, they can take several hours to complete. Figuring out why this is the case is beyond my skill set; what sorts of factors should I be considering if I were to work with our hosting provider to solve this issue? Thanks for any help you all can provide! -k -------------- next part -------------- An HTML attachment was scrubbed... URL: From npwb2 at cam.ac.uk Tue Jul 16 05:45:26 2019 From: npwb2 at cam.ac.uk (Nick Butler) Date: Tue, 16 Jul 2019 09:45:26 +0000 Subject: [Archivesspace_Users_Group] User defined booleans appearing in record view Message-ID: Hi all, Just wondering if anyone else has experienced this issue and might know how to solve it? We aren't currently using any of the three Boolean user defined fields in Archivesspace, and are hiding them from view using a combination of the user_defined_in_basic plugin and commenting out large swathes of the user_defined template. When we click to edit the resources, these Boolean fields are not displayed, as desired. However when we click to view the resources, the section appears, with Boolean 1: False, Boolean 2: False and Boolean 3: False all appearing. I've attached a screenshot of what we're seeing. I may be wrong, but I don't think this is caused by a plugin we have (at least, commenting out the plugin line in our config file and restarting doesn't seem to fix it). Is there anything we can do to resolve this? We'd like none of these fields to appear, and for the entire user defined section not to appear if there's nothing in it. Many thanks, Nick -- Nick Butler Software Developer Digital Services Cambridge University Library West Road Cambridge CB3 9DR, UK npwb2 at cam.ac.uk Internal tel: 33067 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot from 2019-07-16 10-43-13.png Type: image/png Size: 17729 bytes Desc: Screenshot from 2019-07-16 10-43-13.png URL: From npwb2 at cam.ac.uk Tue Jul 16 07:11:02 2019 From: npwb2 at cam.ac.uk (Nick Butler) Date: Tue, 16 Jul 2019 11:11:02 +0000 Subject: [Archivesspace_Users_Group] User defined booleans appearing in record view In-Reply-To: References: Message-ID: <473499654df2230e5758a4064c61a8c28a57514e.camel@cam.ac.uk> Just realised you can override the user_defined schemas file by copying it into the plugin (in the schemas directory), then fix the issue just by commenting out the boolean lines. Apologies for not trying this before... Best wishes, Nick -----Original Message----- From: Nick Butler > Reply-To: Archivesspace Users Group > To: archivesspace_users_group at lyralists.lyrasis.org > Subject: [Archivesspace_Users_Group] User defined booleans appearing in record view Date: Tue, 16 Jul 2019 09:45:26 +0000 Hi all, Just wondering if anyone else has experienced this issue and might know how to solve it? We aren't currently using any of the three Boolean user defined fields in Archivesspace, and are hiding them from view using a combination of the user_defined_in_basic plugin and commenting out large swathes of the user_defined template. When we click to edit the resources, these Boolean fields are not displayed, as desired. However when we click to view the resources, the section appears, with Boolean 1: False, Boolean 2: False and Boolean 3: False all appearing. I've attached a screenshot of what we're seeing. I may be wrong, but I don't think this is caused by a plugin we have (at least, commenting out the plugin line in our config file and restarting doesn't seem to fix it). Is there anything we can do to resolve this? We'd like none of these fields to appear, and for the entire user defined section not to appear if there's nothing in it. Many thanks, Nick -- Nick Butler Software Developer Digital Services Cambridge University Library West Road Cambridge CB3 9DR, UK npwb2 at cam.ac.uk Internal tel: 33067 _______________________________________________ Archivesspace_Users_Group mailing list Archivesspace_Users_Group at lyralists.lyrasis.org http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -- Nick Butler Software Developer Digital Services Cambridge University Library West Road Cambridge CB3 9DR, UK npwb2 at cam.ac.uk Internal tel: 33067 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jessica.Crouch at lyrasis.org Tue Jul 16 12:18:47 2019 From: Jessica.Crouch at lyrasis.org (Jessica Crouch) Date: Tue, 16 Jul 2019 16:18:47 +0000 Subject: [Archivesspace_Users_Group] LYRASIS 2019 Accessibility Report Now Available Message-ID: <0DB5F2DB-710D-486E-9FE2-F9D6F99BF719@lyrasis.org> LYRASIS announces the release of a new study, Understanding the Landscape of Library Accessibility for Online Materials. The study was undertaken in an effort to map the landscape of accessibility policies and practices for online resources as managed by libraries and archives across the United States. The study can be accessed on the LYRASIS website at https://www.lyrasis.org/technology/Pages/Accessibility-Survey-Report.aspx. Authored by Hannah Rosen, LYRASIS Scholarly Communication Specialist and Digitization Program Coordinator, and Jill Grogg, Strategist for the Content and Scholarly Communication Initiatives team, the study is based on a survey conducted in early 2019 as a mechanism to better understand how (primarily academic) libraries within the United States are handling accessibility for their online content, and more specifically, where they stand in terms of policy and implementation. Key findings of the study include: ? Libraries are the most progressive in terms of accessibility when they maintain the most control over their content. ? National policies and community technical guidelines on accessibility hold more prominence than local or institutional mandates. ? Most accessibility training is self-initiated; more infrastructure is needed to train librarians in accessibility mandates and tools. Celeste Feather, Senior Director of Content and Scholarly Communication Initiatives at LYRASIS says of the report, ?This study is an important piece of the work LYRASIS is doing to understand the needs of collections holding institutions across the country, and helps unite libraries in their efforts to make materials more accessible to their users and communities. Our goal is to leverage the reach we have at LYRASIS, with more than 1000 members, to help build conversations around these vital topics and use our research abilities to inform the wider community about these important trends and best practices. The report and all survey materials, including original survey questions and anonymized responses are accessible on the LYRASIS webpage. Visit https://www.lyrasis.org/technology/Pages/Accessibility-Survey-Report.aspxx for more information. LYRASIS Content and Scholarly Communication Initiatives Team [cid:D9C43C04-DF4E-467C-B0BC-358EF417F998] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 4483 bytes Desc: image001.png URL: From bzhang at csus.edu Mon Jul 22 18:45:35 2019 From: bzhang at csus.edu (Zhang, Bin) Date: Mon, 22 Jul 2019 22:45:35 +0000 Subject: [Archivesspace_Users_Group] CAS authentication? Message-ID: <122FEF07-E36C-4C41-8114-78508434DFDF@csus.edu> This is a newbie question: How does one setup AS to use CAS for authentication? What plugin would you use and how to configure? Thanks --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.carver at lyrasis.org Mon Jul 22 22:23:21 2019 From: blake.carver at lyrasis.org (Blake Carver) Date: Tue, 23 Jul 2019 02:23:21 +0000 Subject: [Archivesspace_Users_Group] CAS authentication? In-Reply-To: <122FEF07-E36C-4C41-8114-78508434DFDF@csus.edu> References: <122FEF07-E36C-4C41-8114-78508434DFDF@csus.edu> Message-ID: There's a plugin: https://github.com/lyrasis/aspace-oauth ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Monday, July 22, 2019 6:45 PM To: Archivesspace Group Subject: [Archivesspace_Users_Group] CAS authentication? This is a newbie question: How does one setup AS to use CAS for authentication? What plugin would you use and how to configure? Thanks --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From trthorn2 at ncsu.edu Tue Jul 23 09:05:18 2019 From: trthorn2 at ncsu.edu (Trevor Thornton) Date: Tue, 23 Jul 2019 09:05:18 -0400 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) Message-ID: Hi everybody- I'm building a service using these API endpoints (or I think I am): [:GET] /repositories/:repo_id/resources/:id/tree/root [:GET] /repositories/:repo_id/resources/:id/tree/node These incorporate the concept of "waypoints", which I admit that I'm not familiar with in this context, and it isn't explained very well in the documentation. This is what I have to work with (these are elements included in the API response): - child_count ? the number of immediate children - waypoints ? the number of ?waypoints? those children are grouped into - waypoint_size ? the number of children in each waypoint - precomputed_waypoints ? a collection of arrays (keyed on child URI) in the same format as returned by the ?/waypoint? endpoint. Since a fetch for a given node is almost always followed by a fetch of the first waypoint, using the information in this structure can save a backend call. Can anyone explain what exactly waypoints are and how they are different from children? In the examples I've seen, the "precomputed_waypoints" element in the response looks like a convoluted way (an array value of the lone element in an object, which is itself the value of the lone element in another object) to provide the children nodes of the given node (or root). What's the difference? Thanks, Trevor -- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries -------------- next part -------------- An HTML attachment was scrubbed... URL: From kk1210 at georgetown.edu Tue Jul 23 09:18:38 2019 From: kk1210 at georgetown.edu (Kimberli Kelmor) Date: Tue, 23 Jul 2019 09:18:38 -0400 Subject: [Archivesspace_Users_Group] proxy rewriting gone awry Message-ID: All of a sudden today the public link to the repositories is broken. The public can list the repositories, but when they click on a link it gets a 404 not found. The url for the list of repositories is website.edu/public/repositories -- this works The url for one of the repositories is website.edu/repositories/2 -- this does not work If I manually make it website.edu/public/repositories/2 -- this works for the actual content, but the images are broken. I'm new to all this, but is something messing up the proxy settings or is that barking up the wrong tree? Everything (config.rb and apache conf.d) is set up per the 'running archivesspace under a prefix' documentation. Running version 2.6.0 -- but that has been running for about a week with no problem. Any help greatly appreciated! Kim -- Kimberli M. Kelmor Head of Law Library Technology Georgetown University Law Library kk1210 at georgetown.edu 202-662-9158 -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.carver at lyrasis.org Tue Jul 23 09:29:39 2019 From: blake.carver at lyrasis.org (Blake Carver) Date: Tue, 23 Jul 2019 13:29:39 +0000 Subject: [Archivesspace_Users_Group] proxy rewriting gone awry In-Reply-To: References: Message-ID: I'd start by looking in the logs (Both Apache and ArchivesSpace), any errors in there? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Kimberli Kelmor Sent: Tuesday, July 23, 2019 9:18 AM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] proxy rewriting gone awry All of a sudden today the public link to the repositories is broken. The public can list the repositories, but when they click on a link it gets a 404 not found. The url for the list of repositories is website.edu/public/repositories -- this works The url for one of the repositories is website.edu/repositories/2 -- this does not work If I manually make it website.edu/public/repositories/2 -- this works for the actual content, but the images are broken. I'm new to all this, but is something messing up the proxy settings or is that barking up the wrong tree? Everything (config.rb and apache conf.d) is set up per the 'running archivesspace under a prefix' documentation. Running version 2.6.0 -- but that has been running for about a week with no problem. Any help greatly appreciated! Kim -- Kimberli M. Kelmor Head of Law Library Technology Georgetown University Law Library kk1210 at georgetown.edu 202-662-9158 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob.swanson at uconn.edu Tue Jul 23 09:39:05 2019 From: bob.swanson at uconn.edu (Swanson, Bob) Date: Tue, 23 Jul 2019 13:39:05 +0000 Subject: [Archivesspace_Users_Group] proxy rewriting gone awry In-Reply-To: References: Message-ID: I?m not very good at this, there are smarter folks than me out there, But when I run into a funky problem like that I re-index and it fixes it. https://archivesspace.github.io/archivesspace/user/re-creating-indexes/ From: archivesspace_users_group-bounces at lyralists.lyrasis.org On Behalf Of Kimberli Kelmor Sent: Tuesday, July 23, 2019 9:19 AM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] proxy rewriting gone awry All of a sudden today the public link to the repositories is broken. The public can list the repositories, but when they click on a link it gets a 404 not found. The url for the list of repositories is website.edu/public/repositories -- this works The url for one of the repositories is website.edu/repositories/2 -- this does not work If I manually make it website.edu/public/repositories/2 -- this works for the actual content, but the images are broken. I'm new to all this, but is something messing up the proxy settings or is that barking up the wrong tree? Everything (config.rb and apache conf.d) is set up per the 'running archivesspace under a prefix' documentation. Running version 2.6.0 -- but that has been running for about a week with no problem. Any help greatly appreciated! Kim -- Kimberli M. Kelmor Head of Law Library Technology Georgetown University Law Library kk1210 at georgetown.edu 202-662-9158 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob.swanson at uconn.edu Tue Jul 23 09:40:09 2019 From: bob.swanson at uconn.edu (Swanson, Bob) Date: Tue, 23 Jul 2019 13:40:09 +0000 Subject: [Archivesspace_Users_Group] proxy rewriting gone awry In-Reply-To: References: Message-ID: BTW, I re-index overnight. It takes a LONG time. From: archivesspace_users_group-bounces at lyralists.lyrasis.org On Behalf Of Swanson, Bob Sent: Tuesday, July 23, 2019 9:39 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] proxy rewriting gone awry I?m not very good at this, there are smarter folks than me out there, But when I run into a funky problem like that I re-index and it fixes it. https://archivesspace.github.io/archivesspace/user/re-creating-indexes/ From: archivesspace_users_group-bounces at lyralists.lyrasis.org

> On Behalf Of Kimberli Kelmor Sent: Tuesday, July 23, 2019 9:19 AM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] proxy rewriting gone awry All of a sudden today the public link to the repositories is broken. The public can list the repositories, but when they click on a link it gets a 404 not found. The url for the list of repositories is website.edu/public/repositories -- this works The url for one of the repositories is website.edu/repositories/2 -- this does not work If I manually make it website.edu/public/repositories/2 -- this works for the actual content, but the images are broken. I'm new to all this, but is something messing up the proxy settings or is that barking up the wrong tree? Everything (config.rb and apache conf.d) is set up per the 'running archivesspace under a prefix' documentation. Running version 2.6.0 -- but that has been running for about a week with no problem. Any help greatly appreciated! Kim -- Kimberli M. Kelmor Head of Law Library Technology Georgetown University Law Library kk1210 at georgetown.edu 202-662-9158 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Tue Jul 23 10:52:41 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Tue, 23 Jul 2019 14:52:41 +0000 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: References: Message-ID: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> I believe the rationale of the waypoints was that initially, it was expected that resource children/ archival objects would fall into a more balanced tree structure, but it turned out that there were many flat hierarchies with hundreds of top level children, and getting all of the children at once was not working very efficiently. So with they waypoint calls, you may only be getting some of the children, but the display can start populating the tree display while making additional calls for the rest. I may have some postman examples and internal notes around somewhere: I?ll see what I can dig out. ? Steve. > On Jul 23, 2019, at 9:05 AM, Trevor Thornton wrote: > > Hi everybody- > > I'm building a service using these API endpoints (or I think I am): > [:GET] /repositories/:repo_id/resources/:id/tree/root > [:GET] /repositories/:repo_id/resources/:id/tree/node > > These incorporate the concept of "waypoints", which I admit that I'm not familiar with in this context, and it isn't explained very well in the documentation. This is what I have to work with (these are elements included in the API response): > child_count ? the number of immediate children > waypoints ? the number of ?waypoints? those children are grouped into > waypoint_size ? the number of children in each waypoint > precomputed_waypoints ? a collection of arrays (keyed on child URI) in the same format as returned by the ?/waypoint? endpoint. Since a fetch for a given node is almost always followed by a fetch of the first waypoint, using the information in this structure can save a backend call. > Can anyone explain what exactly waypoints are and how they are different from children? In the examples I've seen, the "precomputed_waypoints" element in the response looks like a convoluted way (an array value of the lone element in an object, which is itself the value of the lone element in another object) to provide the children nodes of the given node (or root). What's the difference? > > Thanks, > Trevor > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3598 bytes Desc: not available URL: From trthorn2 at ncsu.edu Tue Jul 23 11:08:09 2019 From: trthorn2 at ncsu.edu (Trevor Thornton) Date: Tue, 23 Jul 2019 11:08:09 -0400 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> Message-ID: Thanks, Steve. That makes sense, and I tested with a resource with >1000 top level children and I see that only 200 of them are included, which corresponds to the value for "waypoint_size" in the response: { > "child_count":4780, > "waypoints":24, > "waypoint_size":200 > ... So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) < sdm7g at virginia.edu> wrote: > I believe the rationale of the waypoints was that initially, it was > expected that resource children/ archival objects would fall into a more > balanced tree structure, but it turned out that there were many flat > hierarchies with hundreds of top level children, and getting all of the > children at once was not working very efficiently. So with they waypoint > calls, you may only be getting some of the children, but the display can > start populating the tree display while making additional calls for the > rest. > > I may have some postman examples and internal notes around somewhere: I?ll > see what I can dig out. > > ? Steve. > > > On Jul 23, 2019, at 9:05 AM, Trevor Thornton wrote: > > Hi everybody- > > I'm building a service using these API endpoints (or I think I am): > [:GET] /repositories/:repo_id/resources/:id/tree/root > > [:GET] /repositories/:repo_id/resources/:id/tree/node > > > These incorporate the concept of "waypoints", which I admit that I'm not > familiar with in this context, and it isn't explained very well in the > documentation. This is what I have to work with (these are elements > included in the API response): > > - child_count ? the number of immediate children > - waypoints ? the number of ?waypoints? those children are grouped into > - waypoint_size ? the number of children in each waypoint > - precomputed_waypoints ? a collection of arrays (keyed on child URI) > in the same format as returned by the ?/waypoint? endpoint. Since a fetch > for a given node is almost always followed by a fetch of the first > waypoint, using the information in this structure can save a backend call. > > Can anyone explain what exactly waypoints are and how they are different > from children? In the examples I've seen, the "precomputed_waypoints" > element in the response looks like a convoluted way (an array value of the > lone element in an object, which is itself the value of the lone element in > another object) to provide the children nodes of the given node (or root). > What's the difference? > > Thanks, > Trevor > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > -- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Tue Jul 23 11:57:43 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Tue, 23 Jul 2019 15:57:43 +0000 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> Message-ID: <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> > So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? You call /repositories/$repo/resources/$id/tree/waypoint?offset=$N 23 times. ( You already got the first batch in .precomputed_waypoints in the call to /ress/root ) I found the documentation note in the source I was looking for: https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/large_tree.rb # What's the big idea? # # ArchivesSpace has some big trees in it, and sometimes they look a lot like big # sticks. Back in the dark ages, we used JSTree for our trees, which in general # is perfectly cromulent. We recognized the risk of having some very large # collections, so dutifully configured JSTree to lazily load subtrees as the # user expanded them (avoiding having to load the full tree into memory right # away). # # However, time makes fools of us all. The JSTree approach works fine if your # tree is fairly well balanced, but that's not what things look like in the real # world. Some trees have a single root node and tens of thousands of records # directly underneath it. Lazy loading at the subtree level doesn't save you # here: as soon as you expand that (single) node, you're toast. # # This "large tree" business is a way around all of this. It's effectively a # hybrid of trees and pagination, except we call the pages "waypoints" for # reasons known only to me. So here's the big idea: # # * You want to show a tree. You ask the API to give you the root node. # # * The root node tells you whether or not it has children, how many children, # and how many waypoints that works out to. # # * Each waypoint is a fixed-size page of nodes. If the waypoint size is set # to 200, a node with 1,000 children would have 5 waypoints underneath it. # # * So, to display the records underneath the root node, you fetch the root # node, then fetch the first waypoint to get the first N nodes. If you need # to show more nodes (i.e. if the user has scrolled down), you fetch the # second waypoint, and so on. # # * The records underneath the root might have their own children, and they'll # have their own waypoints that you can fetch in the same way. It's nodes, # waypoints and turtles the whole way down. # # All of this interacts with the largetree.js code in the staff and public # interfaces. You open a resource record, and largetree.js fetches the root # node and inserts placeholders for each waypoint underneath it. As the user # scrolls towards a placeholder, the code starts building tracks ahead of the # train, fetching that waypoint and rendering the records it contains. When a # user expands a node to view its children, that process repeats again (the node # is fetched, waypoint placeholders inserted, etc.). # # The public interface runs the same code as the staff interface, but with a # small twist: it fetches its nodes and waypoints from Solr, rather than from # the live API. We hit the API endpoints at indexing time and store them as # Solr documents, effectively precomputing all of the bits of data we need when # displaying trees. > On Jul 23, 2019, at 11:08 AM, Trevor Thornton wrote: > > Thanks, Steve. That makes sense, and I tested with a resource with >1000 top level children and I see that only 200 of them are included, which corresponds to the value for "waypoint_size" in the response: > > { > "child_count":4780, > "waypoints":24, > "waypoint_size":200 > ... > > So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? > > On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) > wrote: > I believe the rationale of the waypoints was that initially, it was expected that resource children/ archival objects would fall into a more balanced tree structure, but it turned out that there were many flat hierarchies with hundreds of top level children, and getting all of the children at once was not working very efficiently. So with they waypoint calls, you may only be getting some of the children, but the display can start populating the tree display while making additional calls for the rest. > > I may have some postman examples and internal notes around somewhere: I?ll see what I can dig out. > > ? Steve. > > >> On Jul 23, 2019, at 9:05 AM, Trevor Thornton > wrote: >> >> Hi everybody- >> >> I'm building a service using these API endpoints (or I think I am): >> [:GET] /repositories/:repo_id/resources/:id/tree/root >> [:GET] /repositories/:repo_id/resources/:id/tree/node >> >> These incorporate the concept of "waypoints", which I admit that I'm not familiar with in this context, and it isn't explained very well in the documentation. This is what I have to work with (these are elements included in the API response): >> child_count ? the number of immediate children >> waypoints ? the number of ?waypoints? those children are grouped into >> waypoint_size ? the number of children in each waypoint >> precomputed_waypoints ? a collection of arrays (keyed on child URI) in the same format as returned by the ?/waypoint? endpoint. Since a fetch for a given node is almost always followed by a fetch of the first waypoint, using the information in this structure can save a backend call. >> Can anyone explain what exactly waypoints are and how they are different from children? In the examples I've seen, the "precomputed_waypoints" element in the response looks like a convoluted way (an array value of the lone element in an object, which is itself the value of the lone element in another object) to provide the children nodes of the given node (or root). What's the difference? >> >> Thanks, >> Trevor >> >> -- >> Trevor Thornton >> Applications Developer, Digital Library Initiatives >> North Carolina State University Libraries >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3598 bytes Desc: not available URL: From trthorn2 at ncsu.edu Tue Jul 23 12:11:59 2019 From: trthorn2 at ncsu.edu (Trevor Thornton) Date: Tue, 23 Jul 2019 12:11:59 -0400 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> Message-ID: Just found that file in the repo before I saw your message and I think I understand now - thanks! So, if you're looking at a node below the root (an ArchivalObject) that has >200 children, you would hit the ".../tree/waypoint" endpoint however many times and include "parent_node" in the GET params with the ArchivalObject URI, right? On Tue, Jul 23, 2019 at 11:57 AM Majewski, Steven Dennis (sdm7g) < sdm7g at virginia.edu> wrote: > > So the next question is how do you make the subsequent calls to retrieve > the next 200, etc.? > > > > You call /repositories/$repo/resources/$id/tree/waypoint?offset=$N 23 > times. > ( You already got the first batch in .precomputed_waypoints in the call to > /ress/root ) > > > I found the documentation note in the source I was looking for: > > https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/large_tree.rb > > > # What's the big idea? > # > # ArchivesSpace has some big trees in it, and sometimes they look a lot > like big > # sticks. Back in the dark ages, we used JSTree for our trees, which in > general > # is perfectly cromulent. We recognized the risk of having some very large > # collections, so dutifully configured JSTree to lazily load subtrees as > the > # user expanded them (avoiding having to load the full tree into memory > right > # away). > # > # However, time makes fools of us all. The JSTree approach works fine if > your > # tree is fairly well balanced, but that's not what things look like in > the real > # world. Some trees have a single root node and tens of thousands of > records > # directly underneath it. Lazy loading at the subtree level doesn't save > you > # here: as soon as you expand that (single) node, you're toast. > # > # This "large tree" business is a way around all of this. It's > effectively a > # hybrid of trees and pagination, except we call the pages "waypoints" for > # reasons known only to me. So here's the big idea: > # > # * You want to show a tree. You ask the API to give you the root node. > # > # * The root node tells you whether or not it has children, how many > children, > # and how many waypoints that works out to. > # > # * Each waypoint is a fixed-size page of nodes. If the waypoint size is > set > # to 200, a node with 1,000 children would have 5 waypoints underneath > it. > # > # * So, to display the records underneath the root node, you fetch the > root > # node, then fetch the first waypoint to get the first N nodes. If you > need > # to show more nodes (i.e. if the user has scrolled down), you fetch the > # second waypoint, and so on. > # > # * The records underneath the root might have their own children, and > they'll > # have their own waypoints that you can fetch in the same way. It's > nodes, > # waypoints and turtles the whole way down. > # > # All of this interacts with the largetree.js code in the staff and public > # interfaces. You open a resource record, and largetree.js fetches the > root > # node and inserts placeholders for each waypoint underneath it. As the > user > # scrolls towards a placeholder, the code starts building tracks ahead of > the > # train, fetching that waypoint and rendering the records it contains. > When a > # user expands a node to view its children, that process repeats again > (the node > # is fetched, waypoint placeholders inserted, etc.). > # > # The public interface runs the same code as the staff interface, but with > a > # small twist: it fetches its nodes and waypoints from Solr, rather than > from > # the live API. We hit the API endpoints at indexing time and store them > as > # Solr documents, effectively precomputing all of the bits of data we need > when > # displaying trees. > > > > > > On Jul 23, 2019, at 11:08 AM, Trevor Thornton wrote: > > Thanks, Steve. That makes sense, and I tested with a resource with >1000 > top level children and I see that only 200 of them are included, which > corresponds to the value for "waypoint_size" in the response: > > { >> "child_count":4780, >> "waypoints":24, >> "waypoint_size":200 >> ... > > > So the next question is how do you make the subsequent calls to retrieve > the next 200, etc.? > > On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) < > sdm7g at virginia.edu> wrote: > >> I believe the rationale of the waypoints was that initially, it was >> expected that resource children/ archival objects would fall into a more >> balanced tree structure, but it turned out that there were many flat >> hierarchies with hundreds of top level children, and getting all of the >> children at once was not working very efficiently. So with they waypoint >> calls, you may only be getting some of the children, but the display can >> start populating the tree display while making additional calls for the >> rest. >> >> I may have some postman examples and internal notes around somewhere: >> I?ll see what I can dig out. >> >> ? Steve. >> >> >> On Jul 23, 2019, at 9:05 AM, Trevor Thornton wrote: >> >> Hi everybody- >> >> I'm building a service using these API endpoints (or I think I am): >> [:GET] /repositories/:repo_id/resources/:id/tree/root >> >> [:GET] /repositories/:repo_id/resources/:id/tree/node >> >> >> These incorporate the concept of "waypoints", which I admit that I'm not >> familiar with in this context, and it isn't explained very well in the >> documentation. This is what I have to work with (these are elements >> included in the API response): >> >> - child_count ? the number of immediate children >> - waypoints ? the number of ?waypoints? those children are grouped >> into >> - waypoint_size ? the number of children in each waypoint >> - precomputed_waypoints ? a collection of arrays (keyed on child URI) >> in the same format as returned by the ?/waypoint? endpoint. Since a fetch >> for a given node is almost always followed by a fetch of the first >> waypoint, using the information in this structure can save a backend call. >> >> Can anyone explain what exactly waypoints are and how they are different >> from children? In the examples I've seen, the "precomputed_waypoints" >> element in the response looks like a convoluted way (an array value of the >> lone element in an object, which is itself the value of the lone element in >> another object) to provide the children nodes of the given node (or root). >> What's the difference? >> >> Thanks, >> Trevor >> >> -- >> Trevor Thornton >> Applications Developer, Digital Library Initiatives >> North Carolina State University Libraries >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> > > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > -- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Tue Jul 23 12:51:26 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Tue, 23 Jul 2019 16:51:26 +0000 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> Message-ID: <7A53BF80-1E97-4AA4-83DF-C512371BBE1B@virginia.edu> I believe for the next level of archival_objects, you have to get /repositories/$REPO/archival_objects/$ID/children , but check the API docs. Note that there is also a GET /repositories/$REPO/resources/$ID/ordered_records method that gives you the whole hierarchy, but minimal info about each resource: { ref: display_string:, depth:, level: } I don?t think I knew about that one the first time I was wrestling with this sort of task. If you?re doing backend API and not worried about real time display update, it might make more sense to walk the output ordered_records If you want more complete info on resource children. ? Steve. > On Jul 23, 2019, at 12:11 PM, Trevor Thornton wrote: > > Just found that file in the repo before I saw your message and I think I understand now - thanks! > > So, if you're looking at a node below the root (an ArchivalObject) that has >200 children, you would hit the ".../tree/waypoint" endpoint however many times and include "parent_node" in the GET params with the ArchivalObject URI, right? > > On Tue, Jul 23, 2019 at 11:57 AM Majewski, Steven Dennis (sdm7g) > wrote: > >> So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? > > > > You call /repositories/$repo/resources/$id/tree/waypoint?offset=$N 23 times. > ( You already got the first batch in .precomputed_waypoints in the call to /ress/root ) > > > I found the documentation note in the source I was looking for: > https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/large_tree.rb > > > # What's the big idea? > # > # ArchivesSpace has some big trees in it, and sometimes they look a lot like big > # sticks. Back in the dark ages, we used JSTree for our trees, which in general > # is perfectly cromulent. We recognized the risk of having some very large > # collections, so dutifully configured JSTree to lazily load subtrees as the > # user expanded them (avoiding having to load the full tree into memory right > # away). > # > # However, time makes fools of us all. The JSTree approach works fine if your > # tree is fairly well balanced, but that's not what things look like in the real > # world. Some trees have a single root node and tens of thousands of records > # directly underneath it. Lazy loading at the subtree level doesn't save you > # here: as soon as you expand that (single) node, you're toast. > # > # This "large tree" business is a way around all of this. It's effectively a > # hybrid of trees and pagination, except we call the pages "waypoints" for > # reasons known only to me. So here's the big idea: > # > # * You want to show a tree. You ask the API to give you the root node. > # > # * The root node tells you whether or not it has children, how many children, > # and how many waypoints that works out to. > # > # * Each waypoint is a fixed-size page of nodes. If the waypoint size is set > # to 200, a node with 1,000 children would have 5 waypoints underneath it. > # > # * So, to display the records underneath the root node, you fetch the root > # node, then fetch the first waypoint to get the first N nodes. If you need > # to show more nodes (i.e. if the user has scrolled down), you fetch the > # second waypoint, and so on. > # > # * The records underneath the root might have their own children, and they'll > # have their own waypoints that you can fetch in the same way. It's nodes, > # waypoints and turtles the whole way down. > # > # All of this interacts with the largetree.js code in the staff and public > # interfaces. You open a resource record, and largetree.js fetches the root > # node and inserts placeholders for each waypoint underneath it. As the user > # scrolls towards a placeholder, the code starts building tracks ahead of the > # train, fetching that waypoint and rendering the records it contains. When a > # user expands a node to view its children, that process repeats again (the node > # is fetched, waypoint placeholders inserted, etc.). > # > # The public interface runs the same code as the staff interface, but with a > # small twist: it fetches its nodes and waypoints from Solr, rather than from > # the live API. We hit the API endpoints at indexing time and store them as > # Solr documents, effectively precomputing all of the bits of data we need when > # displaying trees. > > > > > >> On Jul 23, 2019, at 11:08 AM, Trevor Thornton > wrote: >> >> Thanks, Steve. That makes sense, and I tested with a resource with >1000 top level children and I see that only 200 of them are included, which corresponds to the value for "waypoint_size" in the response: >> >> { >> "child_count":4780, >> "waypoints":24, >> "waypoint_size":200 >> ... >> >> So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? >> >> On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) > wrote: >> I believe the rationale of the waypoints was that initially, it was expected that resource children/ archival objects would fall into a more balanced tree structure, but it turned out that there were many flat hierarchies with hundreds of top level children, and getting all of the children at once was not working very efficiently. So with they waypoint calls, you may only be getting some of the children, but the display can start populating the tree display while making additional calls for the rest. >> >> I may have some postman examples and internal notes around somewhere: I?ll see what I can dig out. >> >> ? Steve. >> >> >>> On Jul 23, 2019, at 9:05 AM, Trevor Thornton > wrote: >>> >>> Hi everybody- >>> >>> I'm building a service using these API endpoints (or I think I am): >>> [:GET] /repositories/:repo_id/resources/:id/tree/root >>> [:GET] /repositories/:repo_id/resources/:id/tree/node >>> >>> These incorporate the concept of "waypoints", which I admit that I'm not familiar with in this context, and it isn't explained very well in the documentation. This is what I have to work with (these are elements included in the API response): >>> child_count ? the number of immediate children >>> waypoints ? the number of ?waypoints? those children are grouped into >>> waypoint_size ? the number of children in each waypoint >>> precomputed_waypoints ? a collection of arrays (keyed on child URI) in the same format as returned by the ?/waypoint? endpoint. Since a fetch for a given node is almost always followed by a fetch of the first waypoint, using the information in this structure can save a backend call. >>> Can anyone explain what exactly waypoints are and how they are different from children? In the examples I've seen, the "precomputed_waypoints" element in the response looks like a convoluted way (an array value of the lone element in an object, which is itself the value of the lone element in another object) to provide the children nodes of the given node (or root). What's the difference? >>> >>> Thanks, >>> Trevor >>> >>> -- >>> Trevor Thornton >>> Applications Developer, Digital Library Initiatives >>> North Carolina State University Libraries >>> _______________________________________________ >>> Archivesspace_Users_Group mailing list >>> Archivesspace_Users_Group at lyralists.lyrasis.org >>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> >> -- >> Trevor Thornton >> Applications Developer, Digital Library Initiatives >> North Carolina State University Libraries >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3598 bytes Desc: not available URL: From trthorn2 at ncsu.edu Tue Jul 23 14:22:04 2019 From: trthorn2 at ncsu.edu (Trevor Thornton) Date: Tue, 23 Jul 2019 14:22:04 -0400 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: <7A53BF80-1E97-4AA4-83DF-C512371BBE1B@virginia.edu> References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> <7A53BF80-1E97-4AA4-83DF-C512371BBE1B@virginia.edu> Message-ID: First of all, great documentation (in the code, API documentation less so but we're working on that) ? To close the loop on this thread (for anyone still interested): For what I'm doing I need the container info, which is included in the .../tree/... responses. Basically I'm re-creating a version of the AS resource tree to provide a browsable view of a resource hierarchy in another application. So the process will be something like this (for a resource with URI */repositories/1/resources/123):* 1. Call * /repositories/1/resources/123/tree/root* to get the resource-level data and its children (up to 200) 2. If the value for "waypoints" in the response is greater than 1, call */repositories/1/resources/123/tree/waypoints&offset=n* for each additional waypoint (n = 1 through # of waypoints - 1) to get the rest of the children 3. Then for each child record with other children, I'll provide a link to see the next level, which will call * /repositories/1/resources/123/tree/node&node_uri=[URI OF THE RECORD THAT WAS CLICKED] **(NOTE: node_uri is a required parameter for this endpoint but that's not mentioned in API the documentation)* This provides a response similar to the *.../tree/root* endpoint but with data for the archival object record instead of the resource 4. Repeat step 2 if there is more than one waypoint at this level, including the current node URI as *parent_id* in the GET params 5. Repeat steps 3 & 4 until you get to the end I *think* this is close to right. Thanks again for your help! On Tue, Jul 23, 2019 at 12:51 PM Majewski, Steven Dennis (sdm7g) < sdm7g at virginia.edu> wrote: > > I believe for the next level of archival_objects, you have to get > /repositories/$REPO/archival_objects/$ID/children , but check the API docs. > > > Note that there is also a GET > /repositories/$REPO/resources/$ID/ordered_records method that gives you the > whole hierarchy, but minimal info about each resource: { ref: > display_string:, depth:, level: } > > I don?t think I knew about that one the first time I was wrestling with > this sort of task. > If you?re doing backend API and not worried about real time display > update, it might make more sense to walk the output ordered_records > If you want more complete info on resource children. > > > ? Steve. > > > On Jul 23, 2019, at 12:11 PM, Trevor Thornton wrote: > > Just found that file in the repo before I saw your message and I think I > understand now - thanks! > > So, if you're looking at a node below the root (an ArchivalObject) that > has >200 children, you would hit the ".../tree/waypoint" endpoint however > many times and include "parent_node" in the GET params with the > ArchivalObject URI, right? > > On Tue, Jul 23, 2019 at 11:57 AM Majewski, Steven Dennis (sdm7g) < > sdm7g at virginia.edu> wrote: > >> >> So the next question is how do you make the subsequent calls to retrieve >> the next 200, etc.? >> >> >> >> You call /repositories/$repo/resources/$id/tree/waypoint?offset=$N 23 >> times. >> ( You already got the first batch in .precomputed_waypoints in the call >> to /ress/root ) >> >> >> I found the documentation note in the source I was looking for: >> >> https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/large_tree.rb >> >> >> # What's the big idea? >> # >> # ArchivesSpace has some big trees in it, and sometimes they look a lot >> like big >> # sticks. Back in the dark ages, we used JSTree for our trees, which in >> general >> # is perfectly cromulent. We recognized the risk of having some very >> large >> # collections, so dutifully configured JSTree to lazily load subtrees as >> the >> # user expanded them (avoiding having to load the full tree into memory >> right >> # away). >> # >> # However, time makes fools of us all. The JSTree approach works fine if >> your >> # tree is fairly well balanced, but that's not what things look like in >> the real >> # world. Some trees have a single root node and tens of thousands of >> records >> # directly underneath it. Lazy loading at the subtree level doesn't save >> you >> # here: as soon as you expand that (single) node, you're toast. >> # >> # This "large tree" business is a way around all of this. It's >> effectively a >> # hybrid of trees and pagination, except we call the pages "waypoints" for >> # reasons known only to me. So here's the big idea: >> # >> # * You want to show a tree. You ask the API to give you the root node. >> # >> # * The root node tells you whether or not it has children, how many >> children, >> # and how many waypoints that works out to. >> # >> # * Each waypoint is a fixed-size page of nodes. If the waypoint size >> is set >> # to 200, a node with 1,000 children would have 5 waypoints underneath >> it. >> # >> # * So, to display the records underneath the root node, you fetch the >> root >> # node, then fetch the first waypoint to get the first N nodes. If >> you need >> # to show more nodes (i.e. if the user has scrolled down), you fetch >> the >> # second waypoint, and so on. >> # >> # * The records underneath the root might have their own children, and >> they'll >> # have their own waypoints that you can fetch in the same way. It's >> nodes, >> # waypoints and turtles the whole way down. >> # >> # All of this interacts with the largetree.js code in the staff and public >> # interfaces. You open a resource record, and largetree.js fetches the >> root >> # node and inserts placeholders for each waypoint underneath it. As the >> user >> # scrolls towards a placeholder, the code starts building tracks ahead of >> the >> # train, fetching that waypoint and rendering the records it contains. >> When a >> # user expands a node to view its children, that process repeats again >> (the node >> # is fetched, waypoint placeholders inserted, etc.). >> # >> # The public interface runs the same code as the staff interface, but >> with a >> # small twist: it fetches its nodes and waypoints from Solr, rather than >> from >> # the live API. We hit the API endpoints at indexing time and store them >> as >> # Solr documents, effectively precomputing all of the bits of data we >> need when >> # displaying trees. >> >> >> >> >> >> On Jul 23, 2019, at 11:08 AM, Trevor Thornton wrote: >> >> Thanks, Steve. That makes sense, and I tested with a resource with >1000 >> top level children and I see that only 200 of them are included, which >> corresponds to the value for "waypoint_size" in the response: >> >> { >>> "child_count":4780, >>> "waypoints":24, >>> "waypoint_size":200 >>> ... >> >> >> So the next question is how do you make the subsequent calls to retrieve >> the next 200, etc.? >> >> On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) < >> sdm7g at virginia.edu> wrote: >> >>> I believe the rationale of the waypoints was that initially, it was >>> expected that resource children/ archival objects would fall into a more >>> balanced tree structure, but it turned out that there were many flat >>> hierarchies with hundreds of top level children, and getting all of the >>> children at once was not working very efficiently. So with they waypoint >>> calls, you may only be getting some of the children, but the display can >>> start populating the tree display while making additional calls for the >>> rest. >>> >>> I may have some postman examples and internal notes around somewhere: >>> I?ll see what I can dig out. >>> >>> ? Steve. >>> >>> >>> On Jul 23, 2019, at 9:05 AM, Trevor Thornton wrote: >>> >>> Hi everybody- >>> >>> I'm building a service using these API endpoints (or I think I am): >>> [:GET] /repositories/:repo_id/resources/:id/tree/root >>> >>> [:GET] /repositories/:repo_id/resources/:id/tree/node >>> >>> >>> These incorporate the concept of "waypoints", which I admit that I'm not >>> familiar with in this context, and it isn't explained very well in the >>> documentation. This is what I have to work with (these are elements >>> included in the API response): >>> >>> - child_count ? the number of immediate children >>> - waypoints ? the number of ?waypoints? those children are grouped >>> into >>> - waypoint_size ? the number of children in each waypoint >>> - precomputed_waypoints ? a collection of arrays (keyed on child >>> URI) in the same format as returned by the ?/waypoint? endpoint. Since a >>> fetch for a given node is almost always followed by a fetch of the first >>> waypoint, using the information in this structure can save a backend call. >>> >>> Can anyone explain what exactly waypoints are and how they are different >>> from children? In the examples I've seen, the "precomputed_waypoints" >>> element in the response looks like a convoluted way (an array value of the >>> lone element in an object, which is itself the value of the lone element in >>> another object) to provide the children nodes of the given node (or root). >>> What's the difference? >>> >>> Thanks, >>> Trevor >>> >>> -- >>> Trevor Thornton >>> Applications Developer, Digital Library Initiatives >>> North Carolina State University Libraries >>> _______________________________________________ >>> Archivesspace_Users_Group mailing list >>> Archivesspace_Users_Group at lyralists.lyrasis.org >>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >>> >>> >>> _______________________________________________ >>> Archivesspace_Users_Group mailing list >>> Archivesspace_Users_Group at lyralists.lyrasis.org >>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >>> >> >> >> -- >> Trevor Thornton >> Applications Developer, Digital Library Initiatives >> North Carolina State University Libraries >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> > > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > -- Trevor Thornton Applications Developer, Digital Library Initiatives North Carolina State University Libraries -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Tue Jul 23 14:42:06 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Tue, 23 Jul 2019 18:42:06 +0000 Subject: [Archivesspace_Users_Group] Retrieving tree info via API (what are "waypoints"?) In-Reply-To: References: <50F3A05C-E9E4-4D6C-BE3C-AFEDD246929A@virginia.edu> <63934353-7A2B-498E-92C9-8EC43174B162@virginia.edu> <7A53BF80-1E97-4AA4-83DF-C512371BBE1B@virginia.edu> Message-ID: <48121C73-C4AD-429A-9CE3-15DE6A75991C@virginia.edu> AH! Thanks I hadn?t needed to drill down to deeper levels on what I was initially using this endpoint for, so I didn?t catch onto the use of /tree/node?node_uri= Sounds right! ? Steve. > On Jul 23, 2019, at 2:22 PM, Trevor Thornton wrote: > > First of all, great documentation (in the code, API documentation less so but we're working on that) ? > > To close the loop on this thread (for anyone still interested): > For what I'm doing I need the container info, which is included in the .../tree/... responses. Basically I'm re-creating a version of the AS resource tree to provide a browsable view of a resource hierarchy in another application. So the process will be something like this (for a resource with URI /repositories/1/resources/123): > Call /repositories/1/resources/123/tree/root to get the resource-level data and its children (up to 200) > If the value for "waypoints" in the response is greater than 1, call /repositories/1/resources/123/tree/waypoints&offset=n for each additional waypoint (n = 1 through # of waypoints - 1) to get the rest of the children > Then for each child record with other children, I'll provide a link to see the next level, which will call /repositories/1/resources/123/tree/node&node_uri=[URI OF THE RECORD THAT WAS CLICKED] > (NOTE: node_uri is a required parameter for this endpoint but that's not mentioned in API the documentation) > This provides a response similar to the .../tree/root endpoint but with data for the archival object record instead of the resource > Repeat step 2 if there is more than one waypoint at this level, including the current node URI as parent_id in the GET params > Repeat steps 3 & 4 until you get to the end > I *think* this is close to right. > > Thanks again for your help! > > On Tue, Jul 23, 2019 at 12:51 PM Majewski, Steven Dennis (sdm7g) > wrote: > > I believe for the next level of archival_objects, you have to get /repositories/$REPO/archival_objects/$ID/children , but check the API docs. > > > Note that there is also a GET /repositories/$REPO/resources/$ID/ordered_records method that gives you the whole hierarchy, but minimal info about each resource: { ref: display_string:, depth:, level: } > > I don?t think I knew about that one the first time I was wrestling with this sort of task. > If you?re doing backend API and not worried about real time display update, it might make more sense to walk the output ordered_records > If you want more complete info on resource children. > > > ? Steve. > > >> On Jul 23, 2019, at 12:11 PM, Trevor Thornton > wrote: >> >> Just found that file in the repo before I saw your message and I think I understand now - thanks! >> >> So, if you're looking at a node below the root (an ArchivalObject) that has >200 children, you would hit the ".../tree/waypoint" endpoint however many times and include "parent_node" in the GET params with the ArchivalObject URI, right? >> >> On Tue, Jul 23, 2019 at 11:57 AM Majewski, Steven Dennis (sdm7g) > wrote: >> >>> So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? >> >> >> >> You call /repositories/$repo/resources/$id/tree/waypoint?offset=$N 23 times. >> ( You already got the first batch in .precomputed_waypoints in the call to /ress/root ) >> >> >> I found the documentation note in the source I was looking for: >> https://github.com/archivesspace/archivesspace/blob/master/backend/app/model/large_tree.rb >> >> >> # What's the big idea? >> # >> # ArchivesSpace has some big trees in it, and sometimes they look a lot like big >> # sticks. Back in the dark ages, we used JSTree for our trees, which in general >> # is perfectly cromulent. We recognized the risk of having some very large >> # collections, so dutifully configured JSTree to lazily load subtrees as the >> # user expanded them (avoiding having to load the full tree into memory right >> # away). >> # >> # However, time makes fools of us all. The JSTree approach works fine if your >> # tree is fairly well balanced, but that's not what things look like in the real >> # world. Some trees have a single root node and tens of thousands of records >> # directly underneath it. Lazy loading at the subtree level doesn't save you >> # here: as soon as you expand that (single) node, you're toast. >> # >> # This "large tree" business is a way around all of this. It's effectively a >> # hybrid of trees and pagination, except we call the pages "waypoints" for >> # reasons known only to me. So here's the big idea: >> # >> # * You want to show a tree. You ask the API to give you the root node. >> # >> # * The root node tells you whether or not it has children, how many children, >> # and how many waypoints that works out to. >> # >> # * Each waypoint is a fixed-size page of nodes. If the waypoint size is set >> # to 200, a node with 1,000 children would have 5 waypoints underneath it. >> # >> # * So, to display the records underneath the root node, you fetch the root >> # node, then fetch the first waypoint to get the first N nodes. If you need >> # to show more nodes (i.e. if the user has scrolled down), you fetch the >> # second waypoint, and so on. >> # >> # * The records underneath the root might have their own children, and they'll >> # have their own waypoints that you can fetch in the same way. It's nodes, >> # waypoints and turtles the whole way down. >> # >> # All of this interacts with the largetree.js code in the staff and public >> # interfaces. You open a resource record, and largetree.js fetches the root >> # node and inserts placeholders for each waypoint underneath it. As the user >> # scrolls towards a placeholder, the code starts building tracks ahead of the >> # train, fetching that waypoint and rendering the records it contains. When a >> # user expands a node to view its children, that process repeats again (the node >> # is fetched, waypoint placeholders inserted, etc.). >> # >> # The public interface runs the same code as the staff interface, but with a >> # small twist: it fetches its nodes and waypoints from Solr, rather than from >> # the live API. We hit the API endpoints at indexing time and store them as >> # Solr documents, effectively precomputing all of the bits of data we need when >> # displaying trees. >> >> >> >> >> >>> On Jul 23, 2019, at 11:08 AM, Trevor Thornton > wrote: >>> >>> Thanks, Steve. That makes sense, and I tested with a resource with >1000 top level children and I see that only 200 of them are included, which corresponds to the value for "waypoint_size" in the response: >>> >>> { >>> "child_count":4780, >>> "waypoints":24, >>> "waypoint_size":200 >>> ... >>> >>> So the next question is how do you make the subsequent calls to retrieve the next 200, etc.? >>> >>> On Tue, Jul 23, 2019 at 10:52 AM Majewski, Steven Dennis (sdm7g) > wrote: >>> I believe the rationale of the waypoints was that initially, it was expected that resource children/ archival objects would fall into a more balanced tree structure, but it turned out that there were many flat hierarchies with hundreds of top level children, and getting all of the children at once was not working very efficiently. So with they waypoint calls, you may only be getting some of the children, but the display can start populating the tree display while making additional calls for the rest. >>> >>> I may have some postman examples and internal notes around somewhere: I?ll see what I can dig out. >>> >>> ? Steve. >>> >>> >>>> On Jul 23, 2019, at 9:05 AM, Trevor Thornton > wrote: >>>> >>>> Hi everybody- >>>> >>>> I'm building a service using these API endpoints (or I think I am): >>>> [:GET] /repositories/:repo_id/resources/:id/tree/root >>>> [:GET] /repositories/:repo_id/resources/:id/tree/node >>>> >>>> These incorporate the concept of "waypoints", which I admit that I'm not familiar with in this context, and it isn't explained very well in the documentation. This is what I have to work with (these are elements included in the API response): >>>> child_count ? the number of immediate children >>>> waypoints ? the number of ?waypoints? those children are grouped into >>>> waypoint_size ? the number of children in each waypoint >>>> precomputed_waypoints ? a collection of arrays (keyed on child URI) in the same format as returned by the ?/waypoint? endpoint. Since a fetch for a given node is almost always followed by a fetch of the first waypoint, using the information in this structure can save a backend call. >>>> Can anyone explain what exactly waypoints are and how they are different from children? In the examples I've seen, the "precomputed_waypoints" element in the response looks like a convoluted way (an array value of the lone element in an object, which is itself the value of the lone element in another object) to provide the children nodes of the given node (or root). What's the difference? >>>> >>>> Thanks, >>>> Trevor >>>> >>>> -- >>>> Trevor Thornton >>>> Applications Developer, Digital Library Initiatives >>>> North Carolina State University Libraries >>>> _______________________________________________ >>>> Archivesspace_Users_Group mailing list >>>> Archivesspace_Users_Group at lyralists.lyrasis.org >>>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >>> >>> _______________________________________________ >>> Archivesspace_Users_Group mailing list >>> Archivesspace_Users_Group at lyralists.lyrasis.org >>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >>> >>> >>> -- >>> Trevor Thornton >>> Applications Developer, Digital Library Initiatives >>> North Carolina State University Libraries >>> _______________________________________________ >>> Archivesspace_Users_Group mailing list >>> Archivesspace_Users_Group at lyralists.lyrasis.org >>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group >> >> >> -- >> Trevor Thornton >> Applications Developer, Digital Library Initiatives >> North Carolina State University Libraries >> _______________________________________________ >> Archivesspace_Users_Group mailing list >> Archivesspace_Users_Group at lyralists.lyrasis.org >> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group > > > -- > Trevor Thornton > Applications Developer, Digital Library Initiatives > North Carolina State University Libraries > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4974 bytes Desc: not available URL: From elouthen at scu.edu Tue Jul 23 14:48:42 2019 From: elouthen at scu.edu (Erin Louthen) Date: Tue, 23 Jul 2019 11:48:42 -0700 Subject: [Archivesspace_Users_Group] Export ASpace accession records to MARC? Message-ID: Hello ASpace Community, I would appreciate advice regarding the following: is there a way to export ASpace accession records into MARC records? I know you can do this with resource records, but can't locate any documentation about exporting accession records. Thank you! Erin -- Erin Louthen University Archivist Archives & Special Collections University Library Santa Clara University 500 El Camino Real Santa Clara, CA 95053-0500 elouthen at scu.edu Tel.:408-554-4117 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kk1210 at georgetown.edu Tue Jul 23 15:26:13 2019 From: kk1210 at georgetown.edu (Kimberli Kelmor) Date: Tue, 23 Jul 2019 15:26:13 -0400 Subject: [Archivesspace_Users_Group] proxy rewriting gone awry Message-ID: I will try the reindexing tonight. The apache error log had a ton of proxy errors, all of which were blocking localhost or refusing connection. A connection will be refused, a connection will fail, 'worker' for local host will be disabled for 60 seconds, and finally local host is disabled. Then the cycle repeats. All the localhost calls are supposed to be re-written aren't they? Thanks again, Kim I?m not very good at this, there are smarter folks than me out there, But when I run into a funky problem like that I re-index and it fixes it. https://archivesspace.github.io/archivesspace/user/re-creating-indexes/ From: archivesspace_users_group-bounces at lyralists.lyrasis.org

> On Behalf Of Natalie Adams Sent: Thursday, July 25, 2019 11:26 AM To: archivesspace_users_group at lyralists.lyrasis.org Subject: [Archivesspace_Users_Group] Adding user-defined fields to archival object records Good afternoon all, This is a question about user-defined fields and archival objects. Here in Cambridge we have been working on migration ?setting up crosswalks to map data from our existing systems (including AtoM) to ArchivesSpace. There are a small number of fields we have not been able to find a home for in AS and we had envisaged mapping them to user-defined fields. However, it looks as though it?s not possible to add a set of user-defined fields to archival objects, though they can be added to other types of record (eg Resources). We are looking at migrating some substantial datasets into ArchivesSpace: in some cases several thousands of records will be nested beneath one parent/resource description (for example the archives of the University of Cambridge) and we want to map all our existing fields to ArchivesSpace (whether the AS records are archival objects or resource descriptions). I would be really interested in hearing whether anyone else has faced a similar challenge and has any advice about how we might tackle this. I?d also be interested to understand more about the background to the difference between archival objects and resources. Many thanks in advance for any help with this and best wishes, Natalie Natalie Adams Systems Archivist Cambridge University Library West Road Cambridge Cambridge, CB3 9DR Tel 01223 766377 www.lib.cam.ac.uk/ Normal working days are Monday-Wednesday -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jessica.Crouch at lyrasis.org Thu Jul 25 14:26:23 2019 From: Jessica.Crouch at lyrasis.org (Jessica Crouch) Date: Thu, 25 Jul 2019 18:26:23 +0000 Subject: [Archivesspace_Users_Group] ArchivesSpace Update - July 2019 Message-ID: <23C9BD55-4153-48D4-9F17-ABB3421E3CD7@lyrasis.org> July 2019 Update Development We continue to work on some needed infrastructure improvements for the application. We're also doing work on agents and recording language information that will make ArchivesSpace more standards-compliant in these areas. We're aiming to have these ready to go into the application in a release later this summer. Are you interested in helping make the ArchivesSpace application better for the community by contributing code for small bug fixes and features? Our Development Prioritization sub-team recently worked with the program team to launch a Ready for Community Developer status to make it easier to identify JIRA issues that are especially appropriate for community contributions. Development Prioritization sub-team leaders Lydia Tang and Maggie Hughes will be talking about it at our Annual Member Forum in Austin next week. You can also reach out to Christine Di Bella (christine.dibella at lyrasis.org) or Laney McGlohon (laney.mcglohon at lyrasis.org) for more information about how to get involved with working on these issues. Registration ends soon for the 5th Annual Member Forum! Registration closes on July 30th for the ArchivesSpace Fifth Annual Member Forum on August 2, 2019, at the Thompson Conference Center at ArchivesSpace member institution the University of Texas at Austin. Just as in previous years, this will be a free opportunity for staff from ArchivesSpace member institutions to meet and share information with each other and the program team about all things ArchivesSpace. Check out our wiki for details and registration information! Attending SAA? Here are all the places you can find ArchivesSpace on the SAA schedule: Visit us at the ArchivesSpace booth! Can?t attend the Member Forum but still want to connect with the ArchivesSpace program team? Visit us at the ArchivesSpace booth in the exhibit hall. The exhibit hall is open Sunday, August 3, from 5:30pm-7:00pm and Monday, August 4, from 8:00am-5:00pm. Check out these sessions featuring ArchivesSpace at SAA! Saturday, August 3 8:30am-10:30am - Museum Section Symposium (9:30am-10:30am) ArchivesSpace at the Whitney Museum of American Art Tara Hart, Whitney Museum of American Art Sunday, August 4 2:30pm-3:00pm - Professional Posters Wrangling a Hidden Fine Art Collection, or a Librarian, an Archivist and a Curator (Cross)walk into ArchivesSpace Sally Benny, New England Historic Genealogical Society 3:00pm-4:00pm - Session 308 Linked Data for Everyone: Practical (and Implementable!) Solutions for Enhancing Discovery on the Web Interested in linked data but don't know where to start? The panelists attempt to demystify linked data by exploring simple, implementable ways to begin creating and consuming linked data to enhance discovery of archival resources. Discussion topics include ArchivesSpace, Wikidata, and Schema.org. This conversation will give participants the clarity, courage, and practical tips needed to jump in and get engaged with the semantic web. Monday, August 5 10:30am-11:45am - Session 609 Standards for Sustainability: Ensuring Sustainable Programs with It Takes a Village What does it take to make sure programs are sustainable and enduring? The It Takes a Village (ITAV) Guidebook can act as a roadmap for all types of programs, not just open source software. Using the four facets outlined in the Guidebook?Technology, Governance, Resources, and Community Engagement?the panelists discuss how ITAV can be a tangible tool for ensuring sustainability, regardless of where a program currently sits within its life cycle. Laurie Arp, Director of Collections Services and Community Supported Software at LYRASIS, and ArchivesSpace Program Manager Christine Di Bella will be on the panel. 12:30pm-1:00pm - Professional Posters Archives to the People: Piloting a New Public Program at the Brooklyn Museum J.E. Molly Seegers, Brooklyn Museum Membership Update We are excited to welcome our newest members to our community! Our new members since June 30 include: * Museum of Ventura County (Ventura, CA) * Newark Public Library (Newark, NJ) * Des Moines Area Community College (Ankeny, IA) Renewals have been sent to all current members for 2019/2020. If your institution has not received its renewal, or if you have any questions about it, please email us. As of July 23, we have 394 General members, 19 Educational Program members, and 3 Registered Service Providers. If you are interested in your institution becoming a member of ArchivesSpace, please email us at ArchivesSpaceHome at lyrasis.org for more information. ________________________________ ArchivesSpace monthly updates provide news about ArchivesSpace community and program activities and are sent to our member listservs, the ArchivesSpace Google Group, and SAA?s Collection Management Tools Section listserv, as well as being posted on the ArchivesSpace website. Please feel free to share this update with people you know who have an interest in ArchivesSpace but may not be on one of these lists. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alinn at dalton.org Thu Jul 25 14:27:04 2019 From: alinn at dalton.org (Amanda Linn) Date: Thu, 25 Jul 2019 14:27:04 -0400 Subject: [Archivesspace_Users_Group] Archivesspace_Users_Group Digest, Vol 77, Issue 1 Message-ID: Thank you for your message. I am away from the office for the summer with limited email access. I will respond as soon as possible. All the best, Amanda Linn Archivist/Librarian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzhang at csus.edu Thu Jul 25 16:26:19 2019 From: bzhang at csus.edu (Zhang, Bin) Date: Thu, 25 Jul 2019 20:26:19 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: Message-ID: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu> We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.carver at lyrasis.org Thu Jul 25 17:15:18 2019 From: blake.carver at lyrasis.org (Blake Carver) Date: Thu, 25 Jul 2019 21:15:18 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu> References: , <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu> Message-ID: "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Thursday, July 25, 2019 4:26 PM To: Archivesspace Group Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzhang at csus.edu Fri Jul 26 12:31:21 2019 From: bzhang at csus.edu (Zhang, Bin) Date: Fri, 26 Jul 2019 16:31:21 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu> Message-ID: Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). -- Bin From: on behalf of Blake Carver Reply-To: Archivesspace Group Date: Thursday, July 25, 2019 at 2:15 PM To: Archivesspace Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Thursday, July 25, 2019 4:26 PM To: Archivesspace Group Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From benn.joseph at northwestern.edu Fri Jul 26 12:58:47 2019 From: benn.joseph at northwestern.edu (Benn Joseph) Date: Fri, 26 Jul 2019 16:58:47 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu>

Message-ID: <08cb704e7ee2468aba45d0492add219d@evcspmbx04.ads.northwestern.edu> When we had the same ampersand problem, we did have to fix them all by hand. We?d export the EAD for records that were known ampersand offenders, find the problem characters, and then fix them directly in the Resource record/archival object/etc. But, it was only for ampersands that were not followed by a space! So ?AT&T? would cause an error, but ?AT & T? is totally fine. You may find it?s the same for you. Benn Joseph Head of Archival Processing Northwestern University Libraries Northwestern University www.library.northwestern.edu benn.joseph at northwestern.edu 847.467.6581 From: archivesspace_users_group-bounces at lyralists.lyrasis.org On Behalf Of Zhang, Bin Sent: Friday, July 26, 2019 11:31 AM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). -- Bin From: > on behalf of Blake Carver > Reply-To: Archivesspace Group > Date: Thursday, July 25, 2019 at 2:15 PM To: Archivesspace Group > Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org

> on behalf of Zhang, Bin > Sent: Thursday, July 25, 2019 4:26 PM To: Archivesspace Group > Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From blake.carver at lyrasis.org Fri Jul 26 13:03:10 2019 From: blake.carver at lyrasis.org (Blake Carver) Date: Fri, 26 Jul 2019 17:03:10 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu> , Message-ID: What version are you running? There's a bunch of fixes in the new version for the dreaded ampersands! ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Friday, July 26, 2019 12:31 PM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). -- Bin From: on behalf of Blake Carver Reply-To: Archivesspace Group Date: Thursday, July 25, 2019 at 2:15 PM To: Archivesspace Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Thursday, July 25, 2019 4:26 PM To: Archivesspace Group Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzhang at csus.edu Fri Jul 26 13:11:00 2019 From: bzhang at csus.edu (Zhang, Bin) Date: Fri, 26 Jul 2019 17:11:00 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu>

Message-ID: We are running 2.6.0 (latest I thought) -- Bin From: on behalf of Blake Carver Reply-To: Archivesspace Group Date: Friday, July 26, 2019 at 10:03 AM To: Archivesspace Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF What version are you running? There's a bunch of fixes in the new version for the dreaded ampersands! ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Friday, July 26, 2019 12:31 PM To: Archivesspace Users Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). -- Bin From: on behalf of Blake Carver Reply-To: Archivesspace Group Date: Thursday, July 25, 2019 at 2:15 PM To: Archivesspace Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin Sent: Thursday, July 25, 2019 4:26 PM To: Archivesspace Group Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. Any help is greatly appreciated. Bin Log Follow Log Generating report Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) java.lang.Thread.run(java/lang/Thread.java:748) Done. --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang -------------- next part -------------- An HTML attachment was scrubbed... URL: From sdm7g at virginia.edu Fri Jul 26 13:14:21 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Fri, 26 Jul 2019 17:14:21 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu>

Message-ID: <4416A461-A41B-4112-A39C-F30309CEDB22@virginia.edu> v2.6.0 added a fix for the dreaded unescaped ampersands in EAD export ( and PDF export from staff interface, which generates EAD and transforms to PDF, where malformed XML in the export stage is the usually problem when PDF export fails. ), https://github.com/archivesspace/archivesspace/commit/18f2f5100e8aa4510c61158a9accf279fb60e895 but I don?t think it will fix this issue in the report generator. I think adding some code to this clean_invalid_xml_characters() function to detect and escape unescaped ampersands might be the place to fix it: https://github.com/archivesspace/archivesspace/blob/master/backend/app/lib/reports/report_generator.rb#L126-L136 ? Steve. > On Jul 26, 2019, at 1:03 PM, Blake Carver wrote: > > What version are you running? There's a bunch of fixes in the new version for the dreaded ampersands! > From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin > Sent: Friday, July 26, 2019 12:31 PM > To: Archivesspace Users Group > Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF > > Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). > > -- > Bin > > From: on behalf of Blake Carver > Reply-To: Archivesspace Group > Date: Thursday, July 25, 2019 at 2:15 PM > To: Archivesspace Group > Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF > > > "The entity name must immediately follow the '&' in the entity reference." > > That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. > > Is there anything in there with ampersands? Like "J&J Automotive" or something? > From: archivesspace_users_group-bounces at lyralists.lyrasis.org on behalf of Zhang, Bin > Sent: Thursday, July 25, 2019 4:26 PM > To: Archivesspace Group > Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF > > We are having problem creating Accession Report in PDF format. All other formats (JSON, CSV, RTF) are working, just not the PDF format. I have attached log below. > > Any help is greatly appreciated. > > Bin > > Log > Follow Log > Generating report > > Can't load the XML resource (using TRaX transformer). org.xml.sax.SAXParseException; lineNumber: 316990; columnNumber: 95; The entity name must immediately follow the '&' in the entity reference. > > org.xhtmlrenderer.resource.XMLResource$XMLResourceBuilder.createXMLResource(org/xhtmlrenderer/resource/XMLResource.java:191) > > org.xhtmlrenderer.resource.XMLResource.load(org/xhtmlrenderer/resource/XMLResource.java:71) > > org.xhtmlrenderer.swing.NaiveUserAgent.getXMLResource(org/xhtmlrenderer/swing/NaiveUserAgent.java:211) > > org.xhtmlrenderer.pdf.ITextRenderer.loadDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:134) > > org.xhtmlrenderer.pdf.ITextRenderer.setDocument(org/xhtmlrenderer/pdf/ITextRenderer.java:149) > > java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) > > org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) > > org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) > > RUBY.generate_pdf(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:54) > > RUBY.generate(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/reports/report_generator.rb:26) > > RUBY.run(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/job_runners/reports_runner.rb:39) > > data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.invokeOther45:run(data/archivesspace/data/tmp/jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_/webapp/WEB_minus_INF/app/lib//data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) > > data.archivesspace.data.tmp.jetty_minus_0_dot_0_dot_0_dot_0_minus_8089_minus_backend_dot_war_minus___minus_any_minus_.webapp.WEB_minus_INF.app.lib.background_job_queue.run_pending_job(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:126) > > RUBY.block in start_background_thread(/data/archivesspace/data/tmp/jetty-0.0.0.0-8089-backend.war-_-any-/webapp/WEB-INF/app/lib/background_job_queue.rb:169) > > org.jruby.RubyProc.call(org/jruby/RubyProc.java:289) > > org.jruby.RubyProc.call(org/jruby/RubyProc.java:246) > > java.lang.Thread.run(java/lang/Thread.java:748) > > Done. > > > > --- > Bin Zhang > Digital Information Services Librarian > Library Systems & IT Services, University Library > California State University, Sacramento > +1 (916) 278-5664 | bzhang at csus.edu > Zoom: https://csus.zoom.us/my/bzhang > > > _______________________________________________ > Archivesspace_Users_Group mailing list > Archivesspace_Users_Group at lyralists.lyrasis.org > http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4974 bytes Desc: not available URL: From sdm7g at virginia.edu Fri Jul 26 17:00:13 2019 From: sdm7g at virginia.edu (Majewski, Steven Dennis (sdm7g)) Date: Fri, 26 Jul 2019 21:00:13 +0000 Subject: [Archivesspace_Users_Group] Mariadb errors from reports: "portion" Message-ID: <469F7FC2-1D1B-478B-802A-48591CF8A434@virginia.edu> Anyone else using MariaDB ? We?re running mysql on our production servers, but I?m running a current version of Maria-DB on my development laptop. While investigating the unescaped ampersand in reports issue, I discovered I?m getting errors running some reports. Resource List Report says: Generating report Java::ComMysqlJdbcExceptionsJdbc4::MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'portion, number as extent_number, extent_type_id as extent_type, contai' at line 2 And by trying that select statement in the command line, it appears that ?portion? must now be a keyword in MariaDB. This generates an error: MariaDB [archivesspace]> select portion_id as portion from extent; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'portion from extent' at line 1 But this works: MariaDB [archivesspace]> select portion_id as `portion` from extent; +---------+ | portion | +---------+ | 924 | | 924 | | 924 | | 924 | | 924 | | 924 | https://github.com/archivesspace/archivesspace/blob/master/reports/custom/extent_subreport.rb#L17 ?Portion? no listed here: https://mariadb.com/kb/en/library/reserved-words/ But this list in source code is canonical source: https://github.com/MariaDB/server/blob/10.5/sql/lex.h#L474 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4974 bytes Desc: not available URL: From bzhang at csus.edu Fri Jul 26 17:08:20 2019 From: bzhang at csus.edu (Zhang, Bin) Date: Fri, 26 Jul 2019 21:08:20 +0000 Subject: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF In-Reply-To: <4416A461-A41B-4112-A39C-F30309CEDB22@virginia.edu> References: <10CF2207-2F96-4A03-B0C0-AB74757AEC51@csus.edu>

<4416A461-A41B-4112-A39C-F30309CEDB22@virginia.edu> Message-ID: <586076C6-EA70-4652-8FD1-6FF6062345E5@csus.edu> Thanks Steven and Blake! Now can someone with power fix this soon? --- Bin Zhang Digital Information Services Librarian Library Systems & IT Services, University Library California State University, Sacramento +1 (916) 278-5664 | bzhang at csus.edu Zoom: https://csus.zoom.us/my/bzhang From: on behalf of "Majewski, Steven Dennis (sdm7g)" Reply-To: Archivesspace Group Date: Friday, July 26, 2019 at 10:14 AM To: Archivesspace Group Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF v2.6.0 added a fix for the dreaded unescaped ampersands in EAD export ( and PDF export from staff interface, which generates EAD and transforms to PDF, where malformed XML in the export stage is the usually problem when PDF export fails. ), https://github.com/archivesspace/archivesspace/commit/18f2f5100e8aa4510c61158a9accf279fb60e895 but I don?t think it will fix this issue in the report generator. I think adding some code to this clean_invalid_xml_characters() function to detect and escape unescaped ampersands might be the place to fix it: https://github.com/archivesspace/archivesspace/blob/master/backend/app/lib/reports/report_generator.rb#L126-L136 ? Steve. On Jul 26, 2019, at 1:03 PM, Blake Carver > wrote: What version are you running? There's a bunch of fixes in the new version for the dreaded ampersands! ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org

> on behalf of Zhang, Bin > Sent: Friday, July 26, 2019 12:31 PM To: Archivesspace Users Group > Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF Thanks Blake! We do have ?&? in accession records (in title and various other fields). It?s going to be a lot of work to replace all of them. Is this something the software can look for and take care of? We were able to get the same report in other formats with no problem (JSON, RTF, HTML). -- Bin From: > on behalf of Blake Carver > Reply-To: Archivesspace Group > Date: Thursday, July 25, 2019 at 2:15 PM To: Archivesspace Group > Subject: Re: [Archivesspace_Users_Group] FW: Unable to create Accession Report in PDF "The entity name must immediately follow the '&' in the entity reference." That makes me think there's an ampersand in that record and it's choking on it when it tries to make the PDF. Is there anything in there with ampersands? Like "J&J Automotive" or something? ________________________________ From: archivesspace_users_group-bounces at lyralists.lyrasis.org