<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Dallas,<br>
</p>
<p><br>
</p>
<p>Have you tried adding "charset=utf-8" to the content-type headers? Also feel free to send me an EAD file that produces the error and I'll try to look into it over the weekend.<br>
</p>
<p><br>
</p>
<p>Best,<br>
</p>
<p>Mark<br>
</p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<span dir="ltr"><font color="black" face="Calibri,Arial,Helvetica,sans-serif" size="3"><span style="font-size:12pt; background-color:white"><font face="Calibri,Arial,Helvetica,sans-serif"><font color="#282828" face="Calibri,Arial,Helvetica,sans-serif"><font color="#2F2F2F" face="Arial,sans-serif"></font></font></font></span></font></span></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div>Mark Cooper<br>
</div>
<div>Technical Lead, Hosting and Support</div>
<div>LYRASIS</div>
<div>email: mark.cooper@lyrasis.org</div>
<div>skype: mark_c_cooper<br>
</div>
</div>
</div>
</div>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Dallas Pillen
<djpillen@umich.edu><br>
<b>Sent:</b> Tuesday, April 7, 2015 9:09 AM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Import EAD via the API?</font>
<div> </div>
</div>
<div>
<div dir="ltr">Thanks for the replies!
<div><br>
</div>
<div>Noah: we have talked about first importing our EADs into AT and then migrating to ArchivesSpace, but we're trying to do a lot of preliminary data cleanup and have made some modifications to the AS EAD importer in an attempt to make our legacy data as clean
and AS-friendly as possible before migrating. Ideally, we'll account for most of the errors we've identified during our EAD cleanup and won't need to rely on the more forgiving nature of the AT importer. But it's something we're open to taking a look at at
some point, at least as a supplemental error discovery/identification tool.</div>
<div><br>
</div>
<div>Mark and Steve: Something along those lines just might work. As part of our EAD cleanup we are going through our extents, subjects, and other potential sources of namespace pollution so those sorts of issues will be controlled for in our EADs before our
actual migration to AS. This should also make a pre-migration conversion to the AS JSONModel more feasible since, when we are ready to do so, our EADs will theoretically convert to the AS JSONModel with minimal errors.</div>
<div><br>
</div>
<div>Mark, I tried your plugin on a batch of EADs and there were a few cases in which the conversion failed due to something like the following:<br>
</div>
<div><br>
</div>
<div>(UndefinedConversionError) ""\xC3\xA9"" from UTF-8 to US-ASCII<br>
</div>
<div><br>
</div>
<div>The EAD that threw this particular error has some '<span style="color:rgb(0,0,0); font-family:'Times New Roman'; font-size:medium; text-align:center; background-color:rgb(248,248,248)">é</span>'<span style="color:rgb(0,0,0); font-family:'Times New Roman'; font-size:medium; text-align:center; background-color:rgb(248,248,248)"> </span>characters
that the AS EAD converter imports without issue. Any thoughts on that? Other than that, converting our EADs to JSON and then bulk importing each of the JSON files would essentially accomplish what I was thinking of doing by importing EAD through the API.</div>
<div><br>
</div>
<div>Still, like Noah, I'd still be interested in knowing if anyone else has figured out how to start EAD import jobs through the API, partially out of curiosity and also to be able to compare some of these potential conversion/import strategies.</div>
<div><br>
</div>
<div>Thanks again for all the helpful suggestion!</div>
<div><br>
</div>
<div>Dallas </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Apr 6, 2015 at 4:35 PM, Steven Majewski <span dir="ltr">
<<a href="mailto:sdm7g@virginia.edu" target="_blank">sdm7g@virginia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div style="word-wrap:break-word">
<div><br>
</div>
<div>What I had been doing was running EADConverter locally on a batch of files, saving the JSON</div>
<div>output if successful, and posting those JSON files to </div>
<div><a href="http://archivesspace.github.io/archivesspace/doc/file.API.html#post-repositoriesrepoidbatchimports" target="_blank">http://archivesspace.github.io/archivesspace/doc/file.API.html#post-repositoriesrepoidbatchimports</a></div>
<div><br>
</div>
<div>( This two stage process was also very useful in earlier versions when the error reporting</div>
<div> was missing the context: ArchivesSpace would tell you what was missing, but it didn’t point</div>
<div> to where it was missing from. It was possible to inspect the JSON and look for the null or</div>
<div> missing value. ) </div>
<div><br>
</div>
<div>You want to:</div>
<div><br>
</div>
<div>
<div style="margin:0px; font-size:15px; background-color:rgb(189,238,237)"><span style="white-space:pre-wrap"></span>converter = EADConverter.new( eadxml )</div>
<div style="margin:0px; font-size:15px; background-color:rgb(189,238,237)"><span style="white-space:pre-wrap"></span>converter.run</div>
</div>
<div>and then do something (move, copy or sent directly to batch_imports API) with:</div>
<div>
<div style="margin:0px; font-size:15px; background-color:rgb(189,238,237)"><span style="white-space:pre-wrap"></span>converter.get_output_path</div>
</div>
<div><br>
</div>
<div>and if you wrap it in a begin/rescue block, you can catch and report the errors. </div>
<div><br>
</div>
<div><br>
</div>
<div>I’ve experimented with a couple of variations on the error catching and processing.</div>
<div>For example, if you move the JSON output in the ensure clause ( begin/rescue/ensure ),</div>
<div>you can save the JSON to inspect even if it’s not complete enough to successfully </div>
<div>import with /batch_imports, but you might not want to mix “good” and “bad” JSON in</div>
<div>your output files. </div>
<div><br>
</div>
<div>More recently, I’ve been experimenting with using an alternate EAD importer with </div>
<div>looser schema validation rules. </div>
<div><br>
</div>
<div>One problem with importing thousands of EAD files by this batch method is that </div>
<div>we have had problems with “namespace pollution” of the controlled vocab lists </div>
<div>for extents and containers. These values are controlled from the webapp and editor,</div>
<div>but importing from EAD adds to the values in the database. If you import a few </div>
<div>EAD files at a time, it’s not difficult to merge and clean up these values, but</div>
<div>importing several thousand EAD files that aren’t very controlled for those values</div>
<div>led to an explosion that makes the drop down lists of those values unusable. </div>
<div><br>
</div>
<div>See a previous message about this: </div>
<div><a href="http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2015-March/001216.html" target="_blank">http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2015-March/001216.html</a></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>— Steve Majewski / UVA Alderman Library </div>
<div><br>
</div>
<br>
<div>
<div>
<div class="h5">
<div>On Apr 6, 2015, at 3:08 PM, Dallas Pillen <<a href="mailto:djpillen@umich.edu" target="_blank">djpillen@umich.edu</a>> wrote:</div>
<br>
</div>
</div>
<blockquote type="cite">
<div>
<div class="h5">
<div dir="ltr">Hello all,
<div><br>
</div>
<div>I was curious if anyone has had any success starting EAD import jobs via the API?</div>
<div><br>
</div>
<div>I was thinking this could be done using POST /repositories/:repo_id/jobs_with_files described here: <a href="http://archivesspace.github.io/archivesspace/doc/file.API.html#post-repositoriesrepoidjobswithfiles" target="_blank">http://archivesspace.github.io/archivesspace/doc/file.API.html#post-repositoriesrepoidjobswithfiles</a></div>
<div><br>
</div>
<div>However, I am not entirely sure how the job and file parameters should be sent in the POST request, and I haven't seen anyone ask this question before or give an example of how it might work. I've tried sending the POST request several different ways and
each time I am met with: {"error":{"job":["Parameter required but no value provided"],"files":["Parameter required but no value provided"]}}. </div>
<div><br>
</div>
<div>I suppose it's worth mentioning that the reason I want to do this is that, at some point, we will be importing several thousand EADs into ArchivesSpace. We're doing a lot of preliminary work to make our EADs import successfully, but know there will likely
be some that will fail. Right now, the only way to do a batch import of EADs is to do a batch as a single import job. If one EAD in that job has an error, the entire job fails. For that reason, I would like to be able to import each EAD as a separate job so
that the EADs that will import successfully will do so without being impacted by the EADs with errors. However, starting several thousand individual import jobs would be very tedious, and I'm looking for a way to automate that process. If anyone else has come
up with any creative solutions or knows of a better way to do that than the API, I would be very interested to know.</div>
<div><br>
</div>
<div>The end goal would be to have a script that would batch start the import jobs, get the ID for each job, check up on the jobs every so often and, once there are no longer any active jobs, output some information about each of the jobs that failed. I've
figured out how to do most of that using the API, but I'm stumped on how to get the whole process started.</div>
<div><br>
</div>
<div>Thanks!</div>
<div><br>
</div>
<div>Dallas<br>
<div><br clear="all">
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.8000001907349px"><b>Dallas Pillen<br>
</b>Project Archivist<b><br>
</b></div>
<div style="font-size:12.8000001907349px"><br>
</div>
<div style="font-size:12.8000001907349px"><img height="40" width="351" style="-webkit-user-select: none;" src="https://webapps.lsa.umich.edu/dean/lsa_emails/bentley-sig-em.png"><br>
</div>
<div style="font-size:12.8000001907349px"><font size="1"> <a href="http://bentley.umich.edu/" target="_blank" style="color:rgb(17,85,204)">Bentley Historical Library</a></font></div>
<div style="font-size:12.8000001907349px"><font size="1"> 1150 Beal Avenue</font></div>
<div style="font-size:12.8000001907349px"><font size="1"> Ann Arbor, Michigan 48109-2113</font><span style="font-size:x-small"> </span></div>
<div style="font-size:12.8000001907349px"><a value="+17347643482" style="font-size:x-small"> </a><a value="+17347643482" style="color:rgb(34,34,34); font-size:x-small">734.647.3559</a></div>
<div style="font-size:12.8000001907349px"><font size="1"> <a href="https://twitter.com/umichBentley" target="_blank" style="color:rgb(17,85,204)">Twitter</a> <a href="https://www.facebook.com/bentleyhistoricallibrary" target="_blank" style="color:rgb(17,85,204)">Facebook </a></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div style="font-size:12.8000001907349px"><b>Dallas Pillen<br>
</b>Project Archivist<b><br>
</b></div>
<div style="font-size:12.8000001907349px"><br>
</div>
<div style="font-size:12.8000001907349px; color:rgb(0,0,0)"><img height="40" width="351" style="-webkit-user-select: none;" src="https://webapps.lsa.umich.edu/dean/lsa_emails/bentley-sig-em.png"><br>
</div>
<div style="font-size:12.8000001907349px; color:rgb(0,0,0)"><font size="1"> <a href="http://bentley.umich.edu/" target="_blank" style="color:rgb(17,85,204)">Bentley Historical Library</a></font></div>
<div style="font-size:12.8000001907349px; color:rgb(0,0,0)"><font size="1"> 1150 Beal Avenue</font></div>
<div style="font-size:12.8000001907349px; color:rgb(0,0,0)"><font size="1"> Ann Arbor, Michigan 48109-2113</font><span style="font-size:x-small"> </span></div>
<div style="font-size:12.8000001907349px"><a value="+17347643482" style="font-size:x-small"><font color="#000000"> </font></a><a value="+17347643482" style="color:rgb(34,34,34); font-size:x-small">734.647.3559</a></div>
<div style="font-size:12.8000001907349px; color:rgb(0,0,0)"><font size="1"> <a href="https://twitter.com/umichBentley" target="_blank" style="color:rgb(17,85,204)">Twitter</a> <a href="https://www.facebook.com/bentleyhistoricallibrary" target="_blank" style="color:rgb(17,85,204)">Facebook </a></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>