<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} .ms-cui-menu {background-color:#ffffff;border:1px rgb(171, 171, 171) solid;font-family:'Segoe UI WPC', 'Segoe UI', Tahoma, 'Microsoft Sans Serif', Verdana, sans-serif;font-size:11pt;color:rgb(51, 51, 51);} .ms-cui-menusection-title {display:none;} .ms-cui-ctl {vertical-align:text-top;text-decoration:none;color:rgb(51, 51, 51);} .ms-cui-ctl-on {background-color:rgb(223, 237, 250);opacity: 0.8;} .ms-cui-img-cont-float {display:inline-block;margin-top:2px} .ms-cui-smenu-inner {padding-top:0px;} .ms-owa-paste-option-icon {margin: 2px 4px 0px 4px;vertical-align:sub;padding-bottom: 2px;display:inline-block;} .ms-rtePasteFlyout-option:hover {background-color:rgb(223, 237, 250) !important;opacity:1 !important;} .ms-rtePasteFlyout-option {padding:8px 4px 8px 4px;outline:none;} .ms-cui-menusection {float:left; width:85px;height:24px;overflow:hidden}--></style>
</head>
<body>
<div style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Steven,<br>
</p>
<p><br>
</p>
<p>Wow, thanks for this. I'm am going over this and it really helps for the improved error messaging we are trying to setup. <br>
</p>
<p><br>
</p>
<p>I definitely think it should be doable to strip out any empty XML tags and not have them create JSON nodes.<br>
</p>
<p><br>
</p>
<p>Also looking at the diff you sent..it seems to cause some problems with the test suite, but I need to figure out what's going on there and way this is stripping out some of these error messages. Will update soon. <br>
</p>
<p><br>
</p>
<p>But yes, until then a good work around would be to strip out empty EAD tags prior to import....<br>
</p>
<p><br>
</p>
<p>best,chris. <br>
</p>
<p><br>
</p>
<div>
<p><br>
</p>
<div class="BodyFragment"><font size="2">
<div class="PlainText">Chris Fitzpatrick | chris.fitzpatrick@lyrasis.org<br>
Developer, ArchivesSpace<br>
http://archivesspace.org/<br>
</div>
</font></div>
</div>
<div style="color: #282828;">
<hr tabindex="-1" style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Steven Majewski
<sdm7g@virginia.edu><br>
<b>Sent:</b> Saturday, March 01, 2014 5:14 PM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] EAD Import - cryptic error messages</font>
<div> </div>
</div>
<div>
<div><br>
</div>
<div>I found a way to get info on the EAD -> JSON_schema mappings, and I’ve managed to fix those notes/0/content errors as well as several others.</div>
<div><br>
</div>
<div>For debugging purposes, I make this temporary change to jsonmodel_wrap.rb to ignore all validation errors,</div>
<div>and I run my command line EAD import parser. ( I haven’t tried running this code on the backend server — no idea what that might break. )</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #692828; background-color: #d4c0ac; position: static; z-index: auto;">
<b>--- a/backend/app/converters/lib/jsonmodel_wrap.rb</b></div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #692828; background-color: #d4c0ac; position: static; z-index: auto;">
<b>+++ b/backend/app/converters/lib/jsonmodel_wrap.rb</b></div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; background-color: #d4c0ac; position: static; z-index: auto;">
<span style="color: #0193a0;">@@ -13,10 +13,10 @@</span> module ASpaceImport</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; background-color: #d4c0ac; position: static; z-index: auto;">
# TODO - speed things up by avoiding this another way</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; background-color: #d4c0ac; position: static; z-index: auto;">
rescue JSONModel::ValidationException => e</div>
<p style="margin: 0px; font-size: 16px; font-family: menlo; background-color: #d4c0ac; min-height: 19px; position: static; z-index: auto;">
<br class="webkit-block-placeholder">
</p>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #a61709; background-color: #d4c0ac; position: static; z-index: auto;">
- e.errors.reject! {|path, mssg|</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #a61709; background-color: #d4c0ac; position: static; z-index: auto;">
- e.attribute_types &&</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #a61709; background-color: #d4c0ac; position: static; z-index: auto;">
- e.attribute_types.has_key?(path) &&</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #a61709; background-color: #d4c0ac; position: static; z-index: auto;">
- e.attribute_types[path] == 'ArchivesSpaceDynamicEnum'}</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #019500; background-color: #d4c0ac; position: static; z-index: auto;">
+ e.errors.reject! {|path, mssg| true }</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #019500; background-color: #d4c0ac; position: static; z-index: auto;">
+# e.attribute_types &&</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #019500; background-color: #d4c0ac; position: static; z-index: auto;">
+# e.attribute_types.has_key?(path) &&</div>
<div style="margin: 0px; font-size: 16px; font-family: menlo; color: #019500; background-color: #d4c0ac; position: static; z-index: auto;">
+# e.attribute_types[path] == 'ArchivesSpaceDynamicEnum'}</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>This generates json files for almost all of the EAD files. ( except for about 30, which I assume are the ones with errors other</div>
<div>than #<:ValidationException…> ). The ones that would not have normally validated correctly will still generate validation</div>
<div>errors if POSTED to /repositories/$ID/batch_imports. However, I can pipe them thru json_pp and search for the schema </div>
<div>property in the error message. So far, this has yielded enough context information to identify the source of the problem </div>
<div>in the EAD file. </div>
<div><br>
</div>
<div>Most of these problems seem to trace back to empty elements in the EAD file. </div>
<div><br>
</div>
<div>In a few cases, there is a missing required element ( unitid, for example ), but in most cases, removing the empty element</div>
<div>fixes the problem. Is this something that could be fixed in the parser ? : if the element is empty, don’t create a JSON property</div>
<div>for it ? </div>
<div>( For now, I’m adding templates for all of the glitches I’ve found to a AS fixup stylesheet run as a pre-process to AS import. ) </div>
<div><br>
</div>
<div><br>
</div>
<div>— Steve Majewski</div>
<div><br>
</div>
<br>
<div>
<div>On Feb 28, 2014, at 8:43 AM, Brad Westbrook <<a href="mailto:brad.westbrook@lyrasis.org">brad.westbrook@lyrasis.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div lang="EN-US" style="font-family: helvetica; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="WordSection1">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;">Hi, Steve,</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;">I won’t be able to address your mapping request until next week.</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;">We are working on a public release now which will address the LDAP security hole reported a couple of weeks ago and include a number of enhancements made since the 1.0.4 release
on Jan. 20. We are aiming to announce the release later today, but it might not be until Monday, depending on resolution of one item.</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;">Brad</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;">Bradley D. Westbrook</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;">Program Manager</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;"><a href="mailto:brad@archivesspace.org" style="color: purple; text-decoration: underline;"><span style="color: #1f497d;">brad.westbrook@lyrasis.org</span></a></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;">800.999.8558 x2910</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;">678.235.2910</span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif; color: #1f497d;">bradley_d_westbrook (Skype) </span><span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: arial, sans-serif;"><image001.png></span><span style="font-size: 11pt; font-family: calibri, sans-serif;"></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<span style="font-size: 11pt; font-family: calibri, sans-serif; color: #1f497d;"> </span></div>
<div>
<div style="border-style: solid none none; border-top-color: #b5c4df; border-top-width: 1pt; padding: 3pt 0in 0in;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
<b><span style="font-size: 10pt; font-family: tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: tahoma, sans-serif;"><span class="Apple-converted-space"> </span><a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">archivesspace_users_group-bounces@lyralists.lyrasis.org</a>
[<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org">mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org</a>]<b>On Behalf Of<span class="Apple-converted-space"> </span></b>Steven Majewski<br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Friday, February 28, 2014 8:33 AM<br>
<b>To:</b><span class="Apple-converted-space"> </span>Archivesspace Users Group<br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Archivesspace_Users_Group] EAD Import - cryptic error messages</span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
Brad:</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
Can you express this requirement in terms of EAD elements instead of JSONModel schema types ? </div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
It’s that mapping that is giving me trouble: trying to turn the schema references in those error messages</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
into elements in the imported EAD that need to be addressed. </div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
Any ETA for that next release ? </div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
I’ve managed to fixup some of the import problems with a stylesheet: I’m up to 2749 files out of 4074 parsing successfully ( up from 0 and 300+ on my </div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
initial efforts ). That notes/0/content message is my greatest outstanding issue: </div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;">1210 #<:ValidationException: {:errors=>{"notes/0/content"=>["At least 1 item(s) is required"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 31 Unexpected Object Type in Queue: Expected archival_object got container</span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 30 #<:ValidationException: {:errors=>{"dates"=>["one or more required (or enter a Title)"], "title"=>["must not be an empty string (or enter a Date)"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 11 #<:ValidationException: {:errors=>{"instances/0/container/indicator_1"=>["Property is required but was missing"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 11 #<:ValidationException: {:errors=>{"id_0"=>["Property is required but was missing"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 8 #<:ValidationException: {:errors=>{"extents"=>["At least 1 item(s) is required"], "notes/0/content"=>["At least 1 item(s) is required"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 6 #<:ValidationException: {:errors=>{"extents"=>["At least 1 item(s) is required"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 5 #<:ValidationException: {:errors=>{"notes/0/content"=>["At least 1 item(s) is required"], "id_0"=>["Property is required but was missing"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 5 #<:ValidationException: {:errors=>{"ead_id"=>["Must be 255 characters or fewer"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 2 #<:ValidationException: {:errors=>{"instances/0/container/type_1"=>["Property is required but was missing"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 Invalid schema given: string</span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 #<:ValidationException: {:errors=>{"record"=>["Can't unambiguously match {:reference_text=>\"(In non correspondence -legal)\"} against schema types: [\"JSONModel(:note_index_item) object\"]. Resolve this by adding
a 'jsonmodel_type' property to {:reference_text=>\"(In non correspondence -legal)\"}"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 #<:ValidationException: {:errors=>{"notes/7/subnotes/0/content"=>["Must be 65000 characters or fewer"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 #<:ValidationException: {:errors=>{"notes/0/content"=>["At least 1 item(s) is required"], "notes/8/subnotes/0/content"=>["Must be 65000 characters or fewer"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 #<:ValidationException: {:errors=>{"instances/0/container/type_1"=>["Property is required but was missing"], "instances/0/container/indicator_1"=>["Property is required but was missing"]}}></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif; background-color: #d4c0ac;">
<span style="font-family: menlo, serif;"> 1 #<:ValidationException: {:errors=>{"extents"=>["At least 1 item(s) is required"], "ead_id"=>["Must be 255 characters or fewer"]}}></span></div>
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
— Steve M.</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'times new roman', serif;">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>