<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Look at that!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="https://github.com/archivesspace/archivesspace/commit/0bfb91e7f27a18b4cb6e0a27527be1041c877237#diff-f266d24dcc6fcbe9020ee4f31cf538f7">https://github.com/archivesspace/archivesspace/commit/0bfb91e7f27a18b4cb6e0a27527be1041c877237#diff-f266d24dcc6fcbe9020ee4f31cf538f7</a><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Yep, sure looks like that'll work as well. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So it seems like the easiest way to serve up a robots file is just throw it in your config directory.</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Andrew Morrison
<andrew.morrison@bodleian.ox.ac.uk><br>
<b>Sent:</b> Tuesday, May 21, 2019 11:37 AM<br>
<b>To:</b> Archivesspace Users Group<br>
<b>Subject:</b> Re: [Archivesspace_Users_Group] Help with robots.txt</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style>
<div lang="EN-US">
<div>Hello,</div>
<div><br>
</div>
<div>If you put a robots.txt file in the config folder of your ArchivesSpace system, it will be served by a request for /robots.txt, after the next restart. I cannot remember where I read that, and cannot find it now, but can confirm it works, since I believe
2.6.0.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Andrew Morrison</div>
<div>Software Engineer</div>
<div>Bodleian Digital Library Systems and Services</div>
<div><a href="https://www.bodleian.ox.ac.uk/bdlss">https://www.bodleian.ox.ac.uk/bdlss</a></div>
<div><br>
</div>
<div><br>
</div>
<div>On Tue, 2019-05-21 at 13:59 +0000, Swanson, Bob wrote:</div>
<blockquote type="cite">
<div class="x_WordSection1">
<p class="x_MsoNormal">Please forgive me if this is posted twice, I sent the following yesterday before I submitted the “acceptance Email” to the ArchivesSpace Users Group. I don’t see where it was posted on the board (am I doing this correctly?).</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">So far as I can tell, this is how I’m supposed to ask questions regarding ArchivesSpace.</p>
<p class="x_MsoNormal">Please forgive and correct me if I’m going about this incorrectly.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I am new to ArchivesSpace, Ruby, JBOD and web development, so I’m pretty dumb.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">The PUI Pre-Launch checklist advises creating and updating
<b>robots.txt</b>,</p>
<p class="x_MsoNormal">So we would like to set up a <b>robots.txt</b> file to control what crawlers can access when they crawl our ArvhivesSpace site
<a href="https://archivessearch.lib.uconn.edu/">https://archivessearch.lib.uconn.edu/</a>.</p>
<p class="x_MsoNormal">I understand that <b>robots.txt</b> is supposed to go in the web root directory of the website.</p>
<p class="x_MsoNormal">In a normal apache configuration that’s simple enough.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">But,</p>
<p class="x_MsoNormal">We are serving ArchivesSpace via HTTPS.</p>
<p class="x_MsoListParagraph" style="text-indent:-.25in"><span style="">a)<span style="font:7.0pt "Times New Roman"">
</span></span>All Port 80 traffic is redirected to Port 443.</p>
<p class="x_MsoListParagraph" style="text-indent:-.25in"><span style="">b)<span style="font:7.0pt "Times New Roman"">
</span></span>443 traffic is proxied to 8081 (for the public interface) per the ArchivesSpace documentation.</p>
<p class="x_MsoNormal" style="margin-left:1.0in"> <b>RequestHeader set X-Forwarded-Proto "https"</b></p>
<p class="x_MsoNormal" style="margin-left:1.0in"><b> ProxyPreserveHost On</b></p>
<p class="x_MsoNormal" style="margin-left:1.0in"><b> ProxyPass / <a href="http://localhost:8081/">
http://localhost:8081/</a> retry=1 acquire=3000 timeout=600 Keepalive=on</b></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:12.0pt; margin-left:1.0in">
<b> ProxyPassReverse / <a href="http://localhost:8081/">http://localhost:8081/</a></b></p>
<p class="x_MsoNormal">So, my web root directory (var/www/html) is empty (save some garbage left over from when I was testing).</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I’ve read the documentation on <a href="http://www.robotstxt.org">
www.robotstxt.org</a> but I can’t find anything that pertains to my situation.</p>
<p class="x_MsoNormal">I have to imagine that most ArchivesSpace sites are now https and use robots.txt, so this should be a somewhat a somewhat standard implementation.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I don not find much information on the Users Group site pertaining to this,</p>
<p class="x_MsoNormal">I find reference to plans for this being implemented at the web server level back in 2016,</p>
<p class="x_MsoNormal">But nothing beyond that.</p>
<p class="x_MsoNormal"><a href="http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html">http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html</a></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">A search of the ArchivesSpace Technical Documentation for “robots” comes up empty as well.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Can you please direct me to any documentation that may exist on setting up a robots.txt file in a proxied HTTPS instance of ArchviceSpace?</p>
<p class="x_MsoNormal">Thank you, and please tolerate my naivety.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="color:#1F497D">Bob Swanson</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">UConn Libraries</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">860-486-5260 – Office</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">860-617-1188 - Mobile</span></p>
<p class="x_MsoNormal"> </p>
</div>
<pre>_______________________________________________
Archivesspace_Users_Group mailing list
<a href="mailto:Archivesspace_Users_Group@lyralists.lyrasis.org">Archivesspace_Users_Group@lyralists.lyrasis.org</a>
<a href="http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group">http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group</a>
</pre>
</blockquote>
</div>
</body>
</html>