[Archivesspace_Users_Group] Container Checker

Wed Apr 20 16:08:10 EDT 2016

Hi,

Yeah...so that cc.sh script is pretty hacky. It's something I wrote just to run data checking easier on my laptop...but I hope it shows how the process works. 

( And yeah...funny how we live in a world now where javascript is the fastest and most reliable option ) 

I've found it ok to ignore those errors since the overflow is always in relation to a longish title. The main thing that's important to check are the container and location related cells, which should be small. 

Also...I just remembered that if you're running daff against very very large CSV file, you might need to tweak Node's heap size...to do that, do a "which daff" to find you executiable ( probably in /usr/local/bin ) and change the shebang to be :

#!/usr/bin/env node --max-old-space-size=8192 

( thatll increase it to 8g.. make it small if you don't have that http://www.codexpedia.com/javascript/increasing-the-memory-limit-in-node-js/ ) 

Chris Fitzpatrick | Developer, ArchivesSpace
Skype: chrisfitzpat  | Phone: 918.236.6048
http://archivesspace.org/

________________________________________
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Majewski, Steven Dennis (sdm7g) <sdm7g at eservices.virginia.edu>
Sent: Wednesday, April 20, 2016 8:56 PM
To: Archivesspace Users Group
Subject: Re: [Archivesspace_Users_Group] Container Checker

I initially missed the requirement for the javascript version on the first read.
I can report that neither the python or ruby versions will work properly.

I had some issues with the permissions on the mysql output in /tmp on my test server installation,
which was causing the ‘mv’ to fail, but correcting for that, it worked.

But lots of "Ignored overflowing row 38203 with cell '\N' after: \N,(172)” ...  messages.
Are these significant ?

— Steve Majewski

> On Apr 20, 2016, at 2:46 PM, Kevin Clair <Kevin.Clair at du.edu> wrote:
>
> I did but not the javascript version; that fixed it. Thanks!  -k
>
> From: <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of "chris.fitzpatrick at lyrasis.org" <Chris.Fitzpatrick at lyrasis.org>
> Reply-To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org>
> Date: Wednesday, April 20, 2016 at 12:41 PM
> To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org>
> Subject: Re: [Archivesspace_Users_Group] Container Checker
>
>
> Hi Kevin,
>
> Yeah, did you install daff?
> https://github.com/paulfitz/daff
>
> The script is using the javascript version of daff ( npm install daff -g ).
> ( you have to have nodejs installed ).
>
> b,chris
>
>
> Chris Fitzpatrick | Developer, ArchivesSpace
> Skype: chrisfitzpat  | Phone: 918.236.6048
> http://archivesspace.org/
>
>
> From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Kevin Clair <Kevin.Clair at du.edu>
> Sent: Wednesday, April 20, 2016 8:34 PM
> To: Archivesspace Users Group
> Subject: Re: [Archivesspace_Users_Group] Container Checker
>
> Hello,
>
> Tried to run this; couldn’t figure out how to run ‘daff’ from command line. Has anyone been successful with that? The project readme isn’t very clear on it (to me, anyway).  -k
>
> From: <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of "chris.fitzpatrick at lyrasis.org" <Chris.Fitzpatrick at lyrasis.org>
> Reply-To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org>
> Date: Monday, April 18, 2016 at 5:38 AM
> To: Archivesspace Group <archivesspace_users_group at lyralists.lyrasis.org>
> Subject: [Archivesspace_Users_Group] Container Checker
>
> Hi,
>
> I put together some SQL queries that might help some people verify their data post-container conversion.
>
> Have a look at : https://github.com/archivesspace/container_checker
>
> There's a few things here:
>
> A SQL script to dump out a CSV file of all your containers and their associated resources/accessions/archival objects and locations.
>
> Another SQL script to pull out a CSV file of all you top_containers and their associated resources/accessions/archival objects and locations
>
> A shell script to run both the container and top_container SQL queries and diff them using the excellent Daff library and make a HTML table that shows the changes. ( script is linux only...sorry <OutlookEmoji-&#X02639.png> )
>
> In theory, the CSV files should be identical after conversion. In reality, this won't be the case since the data can change for a variety of reasons ( most of them beneficial ). However, you can get an overview of what's going on with this and look at it in the comfort of Excel or your preferred Excel alternative.
>
> Please give a go and let us know what you think. If you don't feel capable, or are too busy, or your IT guy is too busy, you can email me an MySQL dump of your database ( zipped up, please ) and I can send you back csv files and html output.
>
> I'm also working on something to do a pre-flight conversion check that will try and call out some potential issues with conversion, which I should have done in the next 24 hours or so.
>
> Thanks and let us know what you think.
>
> best, Chris.
>
>
>
>
>
>
>
>
>
> Chris Fitzpatrick | Developer, ArchivesSpace
> Skype: chrisfitzpat  | Phone: 918.236.6048
> http://archivesspace.org/
> <OutlookEmoji-&#X02639.png>_______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group

_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group