<div dir="ltr">I've also received the same UTF8 error when importing legacy accession records that have <b>valid</b> diacritical marks in the title and/or agent name.<div><br></div><div>Lisa</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 15, 2017 at 2:17 PM, Reese, Terry P. <span dir="ltr"><<a href="mailto:reese.2179@osu.edu" target="_blank">reese.2179@osu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_8494465543217940090WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I guess my question would be – is your legacy data UTF8? For whatever reason, I’ve found that historically, Archives have often used other charactersets when encoding their
EAD files (though to be fair, I see this in MARC records as well; confusion between MARC8, ISO8859-1, and codepage 1252). The simply solution (and this would maintain your characters) would be to convert the character set to UTF8. Otherwise, even if you
held on to these values – they wouldn’t display in any form that you could read; and in fact, that is what the error message is trying to tell you. That as a UTF8 value, your data is going to be gibberish, regardless of if you keep it or not.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">--tr<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-<wbr>bounces@lyralists.lyrasis.org</a> [mailto:<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_<wbr>group-bounces@lyralists.<wbr>lyrasis.org</a>]
<b>On Behalf Of </b>Stasiulatis, Suzanne<br>
<b>Sent:</b> Wednesday, February 15, 2017 3:12 PM</span></p><div><div class="h5"><br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@<wbr>lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Enumerations Findings<u></u><u></u></div></div><p></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I totally agree that we shouldn’t have special characters if at all possible, but a large amount of our legacy data uses them. Especially in titles, staff want
to use those characters as they are reflected on original materials.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Suzanne<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-<wbr>bounces@lyralists.lyrasis.org</a> [<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">mailto:archivesspace_users_<wbr>group-bounces@lyralists.<wbr>lyrasis.org</a>]
<b>On Behalf Of </b>Reese, Terry P.<br>
<b>Sent:</b> Wednesday, February 15, 2017 2:58 PM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Enumerations Findings<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Why would you want to retain invalid special characters? My guess is that one of the reasons for this error is that invalid characters would cause problems with indexing for
search, as well as impact display and export. I would think you’d want to use the error as a flag to identify data that needs to be corrected. Or am I missing something?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">--tr<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-<wbr>bounces@lyralists.lyrasis.org</a> [<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">mailto:archivesspace_users_<wbr>group-bounces@lyralists.<wbr>lyrasis.org</a>]
<b>On Behalf Of </b>Stasiulatis, Suzanne<br>
<b>Sent:</b> Wednesday, February 15, 2017 2:52 PM<br>
<b>To:</b> Archivesspace Users Group <<a href="mailto:archivesspace_users_group@lyralists.lyrasis.org" target="_blank">archivesspace_users_group@<wbr>lyralists.lyrasis.org</a>><br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Enumerations Findings<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">This also came up for me recently. If invalid special characters are present in the content titles, I get this error. I’m not sure quite how to adjust to accept
those special characters.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><img border="0" width="892" height="407" style="width:9.2916in;height:4.2416in" id="m_8494465543217940090Picture_x0020_4" src="cid:image002.png@01D2879E.AE4E3830"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:navy">Suzanne Stasiulatis
</span></b><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:navy">| Archivist II<br>
Pennsylvania Historical and Museum Commission<b> </b>| Pennsylvania State Archives<br>
350 North Street | Harrisburg, PA 17120-0090<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:navy">Phone: <a href="tel:(717)%20787-5953" value="+17177875953" target="_blank">717-787-5953</a><u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.0pt;font-family:"Verdana",sans-serif;color:#003e7e"><a href="http://www.phmc.pa.gov/" target="_blank">http://www.phmc.pa.gov</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:#17365d"><a href="mailto:sustasiula@pa.gov" target="_blank">sustasiula@pa.gov</a></span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#17365d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">
<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">archivesspace_users_group-<wbr>bounces@lyralists.lyrasis.org</a> [<a href="mailto:archivesspace_users_group-bounces@lyralists.lyrasis.org" target="_blank">mailto:archivesspace_users_<wbr>group-bounces@lyralists.<wbr>lyrasis.org</a>]
<b>On Behalf Of </b>Majewski, Steven Dennis (sdm7g)<br>
<b>Sent:</b> Wednesday, February 15, 2017 2:36 PM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Enumerations Findings<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have run into the case that some EAD attribute values are required to be NMTOKENs, thus no embedded spaces or other disallowed characters. We replaced enumerations with embedded spaces with underscores. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This has only come to my attention in the last week or so, so I haven’t made a thorough investigation of which attributes or which enumerations this applies to — just fixed them as I’ve encountered that error. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So it may be intentional that it is using the non translated value. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">( And I wouldn’t be surprised, if for simplicity, it may be over applying that rule in places where it’s not actually required. ) <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">— Steve. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 15, 2017, at 2:09 PM, Carlos Lemus <<a href="mailto:carlos.lemus@unlv.edu" target="_blank">carlos.lemus@unlv.edu</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hello, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">At UNLV Special Collections, we've been working on cleaning up our enumeration values because in many cases there were duplicates caused by imports (i.e value: linear_feet vs value: Linear feet vs Linear Feet). We wanted to stick as close
as possible to ArchivesSpace standards and decided to make our enumeration values all lowercase seperated by an underscore and then merge any records with incorrect enumerations into that correct value (i.e value: linear Feet into linear_feet). We also have
some custom enumerations such as: value: oversized_box, translation: Oversized Box; digital_file; Digital File<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">After we had that set up correctly, we had some findings and was wondering if anyone has experienced the same things or had a standard we could use. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. When generating PDFs and EADs the enumeration values that were custom (such as the oversized_box) would come out as machine readable oversized_box instead of using our local en.yml value (located in the local plugin).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> This was something I found in the EAD serializer (<a href="https://github.com/archivesspace/archivesspace/blob/master/backend/app/exporters/serializers/ead.rb#L490" target="_blank">https://github.com/<wbr>archivesspace/archivesspace/<wbr>blob/master/backend/app/<wbr>exporters/serializers/ead.rb#<wbr>L490</a>) and
was able to create a temporary solution of generating it , but required altering the enumeration instead of referencing our file. I thought i'd point it out because anyone creating custom enumerations even with a translation in an en.yml file would not see
their change reflected in the EAD export. (I've attached an image reflecting this) Anyone experience this?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">2. Another example of this case was in the container "type" attribute. Before something like Oversized Box would be export to EAD as is because that was it's value in the enumeration. After we changed the value correctly to oversized_box,
it would export to the EAD container "type" as is and translate to the PDF as well. With some XSLT manipulation I was able to get it to show up as oversized box (shown in attachments). I've looked through <a href="https://www.loc.gov/ead/tglib/elements/container.html" target="_blank">https://www.loc.gov/<wbr>ead/tglib/elements/container.<wbr>html</a>
and cannot find an example of a two+ attribute value. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Should attributes be machine readable (i.e oversized_box), human readable (Oversized Box), or does it even matter? Of course, exporting it as Oversized Box would be easiest to translate a user friendly version to the user. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Excuse me for the lengthy post, I'm trying to be thorough with my explenation, but please let me know if you've come accross something similar or have a finite solution.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Carlos Lemus<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt">Application Programmer, Special Collections Technical Services</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt">University Libraries, University of Nevada, Las Vegas</span><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><i><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#252525;background:white">How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth? - Sherlock
Holmes</span></i><span style="font-size:9.5pt"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><enumeration_ead.PNG><<wbr>containers_enum.PNG>__________<wbr>______________________________<wbr>_______<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org" target="_blank">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
Archivesspace_Users_Group mailing list<br>
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@<wbr>lyralists.lyrasis.org</a><br>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group" rel="noreferrer" target="_blank">http://lyralists.lyrasis.org/<wbr>mailman/listinfo/<wbr>archivesspace_users_group</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br>Head of Archival Processing<br><br>University of Minnesota Libraries<br>Archives and Special Collections<br>Elmer L. Andersen Library, Suite 315<br>222-21st Ave. S.<br>Minneapolis MN 55455<br><br>Phone: <a value="+16126248812" style="color:rgb(17,85,204)">612.626.2531</a><br></div></div></div></div>
</div>