<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:47386626;
mso-list-type:hybrid;
mso-list-template-ids:-1274392126 67698711 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-number-format:alpha-lower;
mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US">
<div>Hello,</div>
<div><br>
</div>
<div>If you put a robots.txt file in the config folder of your ArchivesSpace system, it will be served by a request for /robots.txt, after the next restart. I cannot remember where I read that, and cannot find it now, but can confirm it works, since I believe
2.6.0.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Andrew Morrison</div>
<div>Software Engineer</div>
<div>Bodleian Digital Library Systems and Services</div>
<div><a href="https://www.bodleian.ox.ac.uk/bdlss">https://www.bodleian.ox.ac.uk/bdlss</a></div>
<div><br>
</div>
<div><br>
</div>
<div>On Tue, 2019-05-21 at 13:59 +0000, Swanson, Bob wrote:</div>
<blockquote type="cite">
<div class="WordSection1">
<p class="MsoNormal">Please forgive me if this is posted twice, I sent the following yesterday before I submitted the “acceptance Email” to the ArchivesSpace Users Group. I don’t see where it was posted on the board (am I doing this correctly?).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So far as I can tell, this is how I’m supposed to ask questions regarding ArchivesSpace.<o:p></o:p></p>
<p class="MsoNormal">Please forgive and correct me if I’m going about this incorrectly.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am new to ArchivesSpace, Ruby, JBOD and web development, so I’m pretty dumb.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The PUI Pre-Launch checklist advises creating and updating <b>
robots.txt</b>,<o:p></o:p></p>
<p class="MsoNormal">So we would like to set up a <b>robots.txt</b> file to control what crawlers can access when they crawl our ArvhivesSpace site
<a href="https://archivessearch.lib.uconn.edu/">https://archivessearch.lib.uconn.edu/</a>.<o:p></o:p></p>
<p class="MsoNormal">I understand that <b>robots.txt</b> is supposed to go in the web root directory of the website.<o:p></o:p></p>
<p class="MsoNormal">In a normal apache configuration that’s simple enough.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But,<o:p></o:p></p>
<p class="MsoNormal">We are serving ArchivesSpace via HTTPS.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><!--[if !supportLists]--><span style="mso-list:Ignore">a)<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->All Port 80 traffic is redirected to Port 443.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><!--[if !supportLists]--><span style="mso-list:Ignore">b)<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->443 traffic is proxied to 8081 (for the public interface) per the ArchivesSpace documentation.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <b>RequestHeader set X-Forwarded-Proto "https"<o:p></o:p></b></p>
<p class="MsoNormal" style="margin-left:1.0in"><b> ProxyPreserveHost On<o:p></o:p></b></p>
<p class="MsoNormal" style="margin-left:1.0in"><b> ProxyPass / <a href="http://localhost:8081/">
http://localhost:8081/</a> retry=1 acquire=3000 timeout=600 Keepalive=on<o:p></o:p></b></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:1.0in">
<b> ProxyPassReverse / <a href="http://localhost:8081/">http://localhost:8081/</a></b><o:p></o:p></p>
<p class="MsoNormal">So, my web root directory (var/www/html) is empty (save some garbage left over from when I was testing).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve read the documentation on <a href="http://www.robotstxt.org">
www.robotstxt.org</a> but I can’t find anything that pertains to my situation.<o:p></o:p></p>
<p class="MsoNormal">I have to imagine that most ArchivesSpace sites are now https and use robots.txt, so this should be a somewhat a somewhat standard implementation.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don not find much information on the Users Group site pertaining to this,<o:p></o:p></p>
<p class="MsoNormal">I find reference to plans for this being implemented at the web server level back in 2016,<o:p></o:p></p>
<p class="MsoNormal">But nothing beyond that.<o:p></o:p></p>
<p class="MsoNormal"><a href="http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html">http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A search of the ArchivesSpace Technical Documentation for “robots” comes up empty as well.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you please direct me to any documentation that may exist on setting up a robots.txt file in a proxied HTTPS instance of ArchviceSpace?<o:p></o:p></p>
<p class="MsoNormal">Thank you, and please tolerate my naivety.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Bob Swanson<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">UConn Libraries<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">860-486-5260 – Office<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">860-617-1188 - Mobile<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<pre>_______________________________________________
Archivesspace_Users_Group mailing list
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a>
</pre>
</blockquote>
</body>
</html>