<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'll be sure to add this one to the docs, so let me know if this works!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think you'll need to do an alias, something like this for Apache:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><Location "/robots.txt"><br>
</span>
<div> SetHandler None<br>
</div>
<div> Require all granted<br>
</div>
<div></Location><br>
</div>
<div>Alias /robots.txt /var/www/robots.txt<br>
</div>
<div><br>
</div>
<div>nginx, more like this:</div>
<div><span> location /robots.txt {<br>
</span>
<div> alias /var/www/robots.txt;<br>
</div>
<div> }<br>
</div>
<span></span><br>
</div>
<span></span><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> archivesspace_users_group-bounces@lyralists.lyrasis.org <archivesspace_users_group-bounces@lyralists.lyrasis.org> on behalf of Swanson, Bob
<bob.swanson@uconn.edu><br>
<b>Sent:</b> Tuesday, May 21, 2019 9:59 AM<br>
<b>To:</b> archivesspace_users_group@lyralists.lyrasis.org<br>
<b>Subject:</b> [Archivesspace_Users_Group] Help with robots.txt</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="x_WordSection1">
<p class="x_MsoNormal">Please forgive me if this is posted twice, I sent the following yesterday before I submitted the “acceptance Email” to the ArchivesSpace Users Group. I don’t see where it was posted on the board (am I doing this correctly?).</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">So far as I can tell, this is how I’m supposed to ask questions regarding ArchivesSpace.</p>
<p class="x_MsoNormal">Please forgive and correct me if I’m going about this incorrectly.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I am new to ArchivesSpace, Ruby, JBOD and web development, so I’m pretty dumb.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">The PUI Pre-Launch checklist advises creating and updating
<b>robots.txt</b>,</p>
<p class="x_MsoNormal">So we would like to set up a <b>robots.txt</b> file to control what crawlers can access when they crawl our ArvhivesSpace site
<a href="https://archivessearch.lib.uconn.edu/">https://archivessearch.lib.uconn.edu/</a>.</p>
<p class="x_MsoNormal">I understand that <b>robots.txt</b> is supposed to go in the web root directory of the website.</p>
<p class="x_MsoNormal">In a normal apache configuration that’s simple enough.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">But,</p>
<p class="x_MsoNormal">We are serving ArchivesSpace via HTTPS.</p>
<p class="x_MsoListParagraph" style="text-indent:-.25in"><span style="">a)<span style="font:7.0pt "Times New Roman"">
</span></span>All Port 80 traffic is redirected to Port 443.</p>
<p class="x_MsoListParagraph" style="text-indent:-.25in"><span style="">b)<span style="font:7.0pt "Times New Roman"">
</span></span>443 traffic is proxied to 8081 (for the public interface) per the ArchivesSpace documentation.</p>
<p class="x_MsoNormal" style="margin-left:1.0in"> <b>RequestHeader set X-Forwarded-Proto "https"</b></p>
<p class="x_MsoNormal" style="margin-left:1.0in"><b> ProxyPreserveHost On</b></p>
<p class="x_MsoNormal" style="margin-left:1.0in"><b> ProxyPass / <a href="http://localhost:8081/">
http://localhost:8081/</a> retry=1 acquire=3000 timeout=600 Keepalive=on</b></p>
<p class="x_MsoNormal" style="margin-right:0in; margin-bottom:12.0pt; margin-left:1.0in">
<b> ProxyPassReverse / <a href="http://localhost:8081/">http://localhost:8081/</a></b></p>
<p class="x_MsoNormal">So, my web root directory (var/www/html) is empty (save some garbage left over from when I was testing).</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I’ve read the documentation on <a href="http://www.robotstxt.org">
www.robotstxt.org</a> but I can’t find anything that pertains to my situation.</p>
<p class="x_MsoNormal">I have to imagine that most ArchivesSpace sites are now https and use robots.txt, so this should be a somewhat a somewhat standard implementation.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I don not find much information on the Users Group site pertaining to this,</p>
<p class="x_MsoNormal">I find reference to plans for this being implemented at the web server level back in 2016,</p>
<p class="x_MsoNormal">But nothing beyond that.</p>
<p class="x_MsoNormal"><a href="http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html">http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/2016-August/003916.html</a></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">A search of the ArchivesSpace Technical Documentation for “robots” comes up empty as well.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Can you please direct me to any documentation that may exist on setting up a robots.txt file in a proxied HTTPS instance of ArchviceSpace?</p>
<p class="x_MsoNormal">Thank you, and please tolerate my naivety.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="color:#1F497D">Bob Swanson</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">UConn Libraries</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">860-486-5260 – Office</span></p>
<p class="x_MsoNormal"><span style="color:#1F497D">860-617-1188 - Mobile</span></p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</body>
</html>