[Archivesspace_Users_Group] Staging upgrade using Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet no data displayed in ASpace application
Jesse Martinez
jesse.martinez at bc.edu
Thu Jul 13 11:24:25 EDT 2023
Hi Mark,
It's normal for the archivesspace core not to have a Data import handler
set up. And it's also normal to see that warning message about security in
the solr logs.
Thank you for checking that the solr schema.xml file matches. The last
thing I can recommend for troubleshooting solr is to bootstrap it with solr
data from production, and see if that loads records into the ASpace
interface.
Jesse
On Thu, Jul 13, 2023 at 10:49 AM Mark Cyzyk <mcyzyk at gmail.com> wrote:
>
> In the Solr Web Gui --> Archivesspace Core --> Dataimport, there is this
> error message:
>
> The solrconfig.xml file for this index does not have an operational
> DataImportHandler defined!
>
>
> My solrconfig.xml, straight from 3.2.0 codebase:
>
> <?xml version="1.0" encoding="utf-8"?><config>
> <lib dir="${solr.install.dir}/dist/" regex="solr-analysis-extras.*\.jar" />
> <lib dir="${solr.install.dir}/contrib/analysis-extras/lucene-libs" regex="lucene-analyzers-icu-.*\.jar" />
> <lib dir="${solr.install.dir}/contrib/analysis-extras/lib/" regex="icu4j-*.*\.jar" />
> <schemaFactory class="ClassicIndexSchemaFactory"/>
> <luceneMatchVersion>8.10</luceneMatchVersion>
> <dataDir>${solr.data.dir:}</dataDir>
> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}" />
> <updateHandler class="solr.DirectUpdateHandler2">
> <maxPendingDeletes>1000</maxPendingDeletes>
> <autoCommit>
> <maxDocs>1000</maxDocs>
> <maxTime>60000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> </updateHandler>
> <query>
> <maxBooleanClauses>1024</maxBooleanClauses>
> <filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
> <queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
> <documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
> <queryResultWindowSize>20</queryResultWindowSize>
> <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
> <useColdSearcher>false</useColdSearcher>
> <maxWarmingSearchers>2</maxWarmingSearchers>
> </query>
> <requestDispatcher handleSelect="false">
> <requestParsers enableRemoteStreaming="true" multipartUploadLimitInKB="2048000" />
> <httpCaching never304="true" />
> </requestDispatcher>
> <requestHandler name="/select" class="solr.SearchHandler">
> <lst name="defaults">
> <str name="defType">edismax</str>
> <str name="echoParams">explicit</str>
> <int name="rows">10</int>
> <str name="df">fullrecord</str>
> <str name="pf">four_part_id^50</str>
> <str name="qf">title^25 four_part_id^50 fullrecord</str>
> <str name="bq">primary_type:resource^100</str>
> <str name="bq">primary_type:accession^100</str>
> <str name="bq">primary_type:subject^50</str>
> <str name="bq">primary_type:agent_person^50</str>
> <str name="bq">primary_type:agent_corporate_entity^30</str>
> <str name="bq">primary_type:agent_family^30</str>
> </lst>
> </requestHandler>
> <requestHandler name="/update" class="solr.UpdateRequestHandler"></requestHandler>
> <requestHandler name="/analysis/document" class="solr.DocumentAnalysisRequestHandler" startup="lazy" />
> <requestHandler name="/analysis/field" startup="lazy" class="solr.FieldAnalysisRequestHandler" />
> <requestHandler name="/replication" class="solr.ReplicationHandler" startup="lazy" />
> <queryResponseWriter name="json" class="solr.JSONResponseWriter">
> <str name="content-type">text/plain; charset=UTF-8</str>
> </queryResponseWriter>
> <admin>
> <defaultQuery>*:*</defaultQuery>
> </admin>
> <requestHandler name="/admin/luke" class="org.apache.solr.handler.admin.LukeRequestHandler" /></config>
>
> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
> Mark Cyzyk, M.A., M.L.S.
> Library Applications Group
> The Sheridan Libraries
> The Johns Hopkins Universitymcyzyk at jhu.edu
>
> Verba volant, scripta manent.
>
> On 7/13/23 10:40, Mark Cyzyk wrote:
>
> Hi Mark,
>
> If you go to this URL
>
> http://localhost:8983/solr/#/archivesspace/files?file=schema.xml
>
> does the schema file shown there match the same one you copied over?
>
> https://github.com/archivesspace/archivesspace/blob/v3.2.0/solr/schema.xml
>
> Jesse
>
>
>
> Yes, they match:
>
> <schema name="archivesspace" version="1.5">
>
>
> Thinking...
>
> Mark
>
> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
> Mark Cyzyk, M.A., M.L.S.
> Library Applications Group
> The Sheridan Libraries
> The Johns Hopkins Universitymcyzyk at jhu.edu
>
> Verba volant, scripta manent.
>
> On 7/12/23 16:20, Jesse Martinez wrote:
>
> Hi Mark,
>
> If you go to this URL
>
> http://localhost:8983/solr/#/archivesspace/files?file=schema.xml
>
> does the schema file shown there match the same one you copied over?
>
> https://github.com/archivesspace/archivesspace/blob/v3.2.0/solr/schema.xml
>
> Jesse
>
>
> On Wed, Jul 12, 2023 at 3:44 PM Mark Cyzyk <mcyzyk at gmail.com> wrote:
>
>> Another clue, from the Log in the Solr Web GUI:
>>
>> 7/12/2023, 3:19:27 PM WARN false
>> CoreContainer Not all security plugins configured!
>> authentication=disabled authorization=disabled. Solr is only as secure as
>> you make it. Consider configuring authentication/authorization before
>> exposing Solr to users internal or external. See
>> https://s.apache.org/solrsecurity for more info
>>
>>
>> Maybe ASpace simply does not have permission to add to Solr? (And yet,
>> this is just a warning. And yet, for the sake of replicability, this very
>> same Solr setup worked in Vagrant during our last ASpace upgrade.)
>>
>> Here, setting up Solr in my Vagrant file (which I recently ran
>> line-by-line, manually, without error):
>>
>> solr_provision = <<SCRIPT
>> cd /opt
>> sudo wget
>> https://archive.apache.org/dist/lucene/solr/8.10.0/solr-8.10.0.tgz
>> sudo tar zxf solr-8.10.0.tgz
>> sudo bash /opt/solr-8.10.0/bin/install_solr_service.sh
>> /opt/solr-8.10.0.tgz
>> sudo mkdir -p /opt/solr/server/solr/configsets/archivesspace/conf
>> sudo cp -r /vagrant/archivesspace_CURRENT/solr/*
>> /opt/solr/server/solr/configsets/archivesspace/conf/
>> sudo chown -R solr:solr /opt/solr
>> sudo su - solr -c "/opt/solr/bin/solr create -c archivesspace -p 8983 -d
>> archivesspace"
>> sudo systemctl restart solr
>> SCRIPT
>>
>>
>> Reading about Solr and Permissions....
>>
>> And sorry for all the emails in this thread. And yet, once this is
>> resolved it may have value for others in future?
>>
>> Mark
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>> Mark Cyzyk, M.A., M.L.S.
>> Library Applications Group
>> The Sheridan Libraries
>> The Johns Hopkins Universitymcyzyk at jhu.edu
>>
>> On 7/12/23 11:46 AM, Joshua D. Shaw wrote:
>>
>> Hey Mark
>>
>> After the updates you've made to the config, try shutting down AS and
>> clearing the indexer state directories and restarting. That should tell AS
>> to try a new index.
>>
>> Also, I think that +0000 is just a timestamp offset. Its not counting the
>> objects indexed. That log line looks something like: PUI Indexer ...
>> indexed x of y resources
>>
>> Joshua
>>
>> ------------------------------
>> *From:* archivesspace_users_group-bounces at lyralists.lyrasis.org
>> <archivesspace_users_group-bounces at lyralists.lyrasis.org>
>> <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of
>> Mark Cyzyk <mcyzyk at gmail.com> <mcyzyk at gmail.com>
>> *Sent:* Wednesday, July 12, 2023 11:32 AM
>> *To:* Brian Hoffman <brian.hoffman at lyrasis.org>
>> <brian.hoffman at lyrasis.org>; mcyzyk at jhu.edu <mcyzyk at jhu.edu>
>> <mcyzyk at jhu.edu>; Archivesspace Users Group
>> <archivesspace_users_group at lyralists.lyrasis.org>
>> <archivesspace_users_group at lyralists.lyrasis.org>; jesse.martinez at bc.edu
>> <jesse.martinez at bc.edu> <jesse.martinez at bc.edu>
>> *Subject:* Re: [Archivesspace_Users_Group] Staging upgrade using
>> Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet
>> no data displayed in ASpace application
>>
>>
>> Brian,
>>
>> I am trying to duplicate our Production ASpace 3.2.0 in a local Vagrant
>> box so I can stage an upgrade to the latest codebase.
>>
>> Try uncommenting that line and changing it to:
>>
>>
>> AppConfig[:solr_url] = "http://localhost:8983/solr/archivesspace"
>> <http://localhost:8983/solr/archivesspace>
>>
>>
>> Done. And yet:
>>
>> ************************************************************
>> Welcome to ArchivesSpace!
>> You can now point your browser to http://localhost:8080
>> ************************************************************
>> I, [2023-07-12T15:28:45.335247 #51936] INFO -- : Thread-3306: PUI
>> Indexer [2023-07-12 15:28:45 +0000] Running index round
>> I, [2023-07-12T15:28:47.533454 #51936] INFO -- : Thread-3306: PUI
>> Indexer [2023-07-12 15:28:47 +0000] Index round complete
>> I, [2023-07-12T15:28:50.634218 #51936] INFO -- : Thread-2956: Staff
>> Indexer [2023-07-12 15:28:50 +0000] Running index round
>> I, [2023-07-12T15:28:52.715507 #51936] INFO -- : Thread-2956: Staff
>> Indexer [2023-07-12 15:28:52 +0000] Index round complete
>>
>>
>> Still +0000!
>>
>> Mark
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>> Mark Cyzyk, M.A., M.L.S.
>> Library Applications Group
>> The Sheridan Libraries
>> The Johns Hopkins Universitymcyzyk at jhu.edu
>>
>> Verba volant, scripta manent.
>>
>> On 7/12/23 11:10, Brian Hoffman wrote:
>>
>> Hi Mark,
>>
>>
>>
>> What version of ASpace are you using?
>>
>>
>>
>> Try uncommenting that line and changing it to:
>>
>>
>>
>> AppConfig[:solr_url] = "http://localhost:8983/solr/archivesspace"
>> <http://localhost:8983/solr/archivesspace>
>>
>>
>>
>> *From: *Mark Cyzyk <mcyzyk at gmail.com> <mcyzyk at gmail.com>
>> *Date: *Wednesday, July 12, 2023 at 10:59 AM
>> *To: *Brian Hoffman <brian.hoffman at lyrasis.org>
>> <brian.hoffman at lyrasis.org>, mcyzyk at jhu.edu <mcyzyk at jhu.edu>
>> <mcyzyk at jhu.edu>, Archivesspace Users Group
>> <archivesspace_users_group at lyralists.lyrasis.org>
>> <archivesspace_users_group at lyralists.lyrasis.org>, jesse.martinez at bc.edu
>> <jesse.martinez at bc.edu> <jesse.martinez at bc.edu>
>> *Subject: *Re: [Archivesspace_Users_Group] Staging upgrade using
>> Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet
>> no data displayed in ASpace application
>>
>>
>> Brian,
>>
>> That value, commented out because it's the default, in my config is:
>>
>> ## The ArchivesSpace Solr index listens on port 8090 by default. You can
>> ## set it to something else below.
>> #AppConfig[:solr_url] = "http://localhost:8090" <http://localhost:8090>
>>
>>
>> Mark
>>
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>>
>> Mark Cyzyk, M.A., M.L.S.
>>
>> Library Applications Group
>>
>> The Sheridan Libraries
>>
>> The Johns Hopkins University
>>
>> mcyzyk at jhu.edu
>>
>>
>>
>> Verba volant, scripta manent.
>>
>> On 7/12/23 10:34, Brian Hoffman wrote:
>>
>> Hi Mark,
>>
>>
>>
>> I am referring to the value for :solr_url in the file ‘config/config.rb’
>>
>>
>>
>> Brian
>>
>>
>>
>>
>>
>> *From: *Mark Cyzyk <mcyzyk at gmail.com> <mcyzyk at gmail.com>
>> *Date: *Wednesday, July 12, 2023 at 10:32 AM
>> *To: *Brian Hoffman <brian.hoffman at lyrasis.org>
>> <brian.hoffman at lyrasis.org>, mcyzyk at jhu.edu <mcyzyk at jhu.edu>
>> <mcyzyk at jhu.edu>, Archivesspace Users Group
>> <archivesspace_users_group at lyralists.lyrasis.org>
>> <archivesspace_users_group at lyralists.lyrasis.org>, jesse.martinez at bc.edu
>> <jesse.martinez at bc.edu> <jesse.martinez at bc.edu>
>> *Subject: *Re: [Archivesspace_Users_Group] Staging upgrade using
>> Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet
>> no data displayed in ASpace application
>>
>> Thanks, Brian,
>>
>> Here is what I see on screen:
>>
>>
>>
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>>
>> Mark Cyzyk, M.A., M.L.S.
>>
>> Library Applications Group
>>
>> The Sheridan Libraries
>>
>> The Johns Hopkins University
>>
>> mcyzyk at jhu.edu
>>
>>
>>
>> Verba volant, scripta manent.
>>
>> On 7/12/23 10:24, Brian Hoffman wrote:
>>
>> Hi Mark,
>>
>>
>>
>> I believe you have incorrectly set the solr url to a screen on the
>> control panel.
>>
>>
>>
>> Try replacing:
>>
>> http://localhost:8983/solr/#/~cores/archivesspace
>>
>> with
>>
>>
>>
>> http://localhost:8983/solr/archivesspace
>> <http://localhost:8983/solr/#/~cores/archivesspace>
>>
>>
>>
>> *From: *archivesspace_users_group-bounces at lyralists.lyrasis.org
>> <archivesspace_users_group-bounces at lyralists.lyrasis.org>
>> <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of
>> Mark Cyzyk <mcyzyk at gmail.com> <mcyzyk at gmail.com>
>> *Date: *Wednesday, July 12, 2023 at 10:19 AM
>> *To: *jesse.martinez at bc.edu <jesse.martinez at bc.edu>
>> <jesse.martinez at bc.edu>, mcyzyk at jhu.edu <mcyzyk at jhu.edu> <mcyzyk at jhu.edu>
>> *Cc: *Archivesspace Users Group
>> <archivesspace_users_group at lyralists.lyrasis.org>
>> <archivesspace_users_group at lyralists.lyrasis.org>
>> *Subject: *Re: [Archivesspace_Users_Group] Staging upgrade using
>> Vagrant/Virtualbox, data in MySQL tables, Solr core present, reindexed, yet
>> no data displayed in ASpace application
>>
>> Darn:
>>
>>
>>
>>
>> {
>>
>> "responseHeader":{
>>
>> "status":0,
>>
>> "QTime":187,
>>
>> "params":{
>>
>> "q":"*:*",
>>
>> "indent":"true",
>>
>> "q.op":"OR",
>>
>> "_":"1689171084647"}},
>>
>> "response":{"numFound":0,"start":0,"numFoundExact":true,"docs":[]
>>
>> }}
>>
>>
>> I will look more carefully at my Solr Aspace config (and yet the steps in
>> my Vagrant file for setting this up are the same as last summer when I did
>> this!).
>>
>> Mark
>>
>>
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>>
>> Mark Cyzyk, M.A., M.L.S.
>>
>> Library Applications Group
>>
>> The Sheridan Libraries
>>
>> The Johns Hopkins University
>>
>> mcyzyk at jhu.edu
>>
>>
>>
>> Verba volant, scripta manent.
>>
>> On 7/12/23 10:09, Jesse Martinez wrote:
>>
>> Thanks for sharing that, Mark!
>>
>>
>>
>> I have another quick follow up question.
>>
>>
>>
>> What do you see when you run a solr search like this one:
>>
>>
>>
>> http://localhost:8983/solr/#/archivesspace/query?q=*:*&q.op=OR&indent=true
>>
>>
>>
>> You should see all your indexed records. But if you don't see any results
>> then it may indicate that your archivesspace configuration directory is not
>> correct.
>>
>>
>>
>> Jesse
>>
>>
>>
>> On Wed, Jul 12, 2023 at 10:05 AM Mark Cyzyk <mcyzyk at gmail.com> wrote:
>>
>>
>> Thanks, Jesse,
>>
>> Here is the output of those commands:
>>
>> ps aux | grep solr | grep -v grep
>> solr 898 0.0 0.1 19096 9812 ? Ss Jul11 0:00
>> /lib/systemd/systemd --user
>> solr 910 0.0 0.0 169492 3368 ? S Jul11 0:00
>> (sd-pam)
>> solr 1223 0.5 12.5 4018692 742508 ? Sl Jul11 7:41 java
>> -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem
>> -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages
>> -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -verbose:gc
>> -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
>> -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/solr/logs/solr_gc.log
>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
>> -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes=
>> -Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983
>> -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow
>> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs
>> -Djetty.home=/opt/solr/server -Dsolr.solr.home=/var/solr/data
>> -Dsolr.data.home= -Dsolr.install.dir=/opt/solr
>> -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
>> -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k
>> -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip
>>
>> Puzzling!
>>
>> Mark
>>
>>
>>
>> <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
>>
>> Mark Cyzyk, M.A., M.L.S.
>>
>> Library Applications Group
>>
>> The Sheridan Libraries
>>
>> The Johns Hopkins University
>>
>> mcyzyk at jhu.edu
>>
>>
>>
>> Verba volant, scripta manent.
>>
>> On 7/12/23 09:52, Jesse Martinez wrote:
>>
>> ps aux | grep solr | grep -v grep
>>
>>
>>
>>
>>
>>
>> --
>>
>> Jesse Martinez
>>
>> Senior Library Applications Developer
>>
>> O'Neill Library, Boston College
>>
>> jesse.martinez at bc.edu
>>
>> 617-552-2509
>>
>> he/him/his
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Archivesspace_Users_Group mailing list
>> Archivesspace_Users_Group at lyralists.lyrasis.org
>> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>>
>
>
> --
> Jesse Martinez
> Senior Library Applications Developer
> O'Neill Library, Boston College
> jesse.martinez at bc.edu
> 617-552-2509
> he/him/his
>
>
>
>
--
Jesse Martinez
Senior Library Applications Developer
O'Neill Library, Boston College
jesse.martinez at bc.edu
617-552-2509
he/him/his
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230713/8dd2f7e4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snapshot1.png
Type: image/png
Size: 98031 bytes
Desc: not available
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230713/8dd2f7e4/attachment.png>
More information about the Archivesspace_Users_Group
mailing list