[Archivesspace_Users_Group] hELP - No record found after test migration from v1.4.2 to 1.5.0
Mang Sun
mang.sun at rice.edu
Thu Sep 15 11:53:55 EDT 2016
Patrick,
I did test with 1.5.0 and 1.5.1 and ran into the same problem.
Mang
On 9/15/2016 10:39 AM, Galligan, Patrick wrote:
>
> Rachel,
>
> It was a known issue that 1.5.0 had indexing problems. These have
> generally been fixed in 1.5.1. I’d suggest using that version for your
> test migration.
>
> Patrick Galligan
>
> Rockefeller Archive Center
>
> Assistant Digital Archivist
>
> 914-366-6386
>
> *From:*archivesspace_users_group-bounces at lyralists.lyrasis.org
> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] *On
> Behalf Of *Maderik, Rachel A
> *Sent:* Thursday, September 15, 2016 11:30 AM
> *To:* 'Archivesspace Users Group'
> *Subject:* Re: [Archivesspace_Users_Group] hELP - No record found
> after test migration from v1.4.2 to 1.5.0
>
> I was able to replicate this problem using Rice’s data. I set up a
> fresh instance running 1.4.2 using their database dump from that
> version, and it built the index fine (took 10-15 minutes). When I
> migrated it to 1.5, the indexer seems to get hung up after indexing
> the top containers (which finished about 2 hrs ago). Below is an error
> message that appears right after it finishes the top containers (and
> continues to appear repeatedly in the log file). It seems to be
> related to indexing archival objects, though I can’t say I understand
> anything more than that:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> ~~~ Indexed 12884 top_container records in 600 seconds ~~~
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Sep 15, 2016 9:33:57 AM
> org.eclipse.jetty.server.handler.ContextHandler$Context log
>
> INFO: DEBUG: resetting rack response due exception
>
> Failure in periodic indexer worker thread: <!DOCTYPE HTML PUBLIC
> "-//W3C//DTD HTML 4.01 Transitional//EN"
> "http://www.w3.org/TR/html4/loose.dtd">
>
> …
>
> <body>
>
> <div id="summary">
>
> <h1>Internal Server Error <span>(500)</span></h1>
>
> <table class="meta">
>
> <tr>
>
> <th>Request Method:</th>
>
> <td>GET</td>
>
> </tr>
>
> <tr>
>
> <th>Request URL:</th <URL:%3c/th>>
>
> <td>http://localhost:8089/repositories/2/archival_objects?id_set=1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16%2C17%2C18%2C19%2C20%2C21%2C22%2C23%2C24%2C25&resolve%5B%5D=location_profile&resolve%5B%5D=container_profile&resolve%5B%5D=container_locations&resolve%5B%5D=subjects&resolve%5B%5D=linked_agents&resolve%5B%5D=linked_records&resolve%5B%5D=classifications&resolve%5B%5D=digital_object</td>
>
> </tr>
>
> </table>
>
> </div>
>
> <div id="info">
>
> <p>(NoMethodError) undefined method `related_records' for
> nil:NilClass</p>
>
> </div>
>
> <div id="explanation">
>
> <p>
>
> You're seeing this error because you use
> <code>JRuby::Rack::ErrorApp::ShowStatus</code>.
>
> </p>
>
> </div>
>
> </body>
>
> -------------------------------
>
> Rachel Maderik
>
> Systems and Technology Librarian
>
> 501D Preston Library
>
> Virginia Military Institute
>
> Lexington, VA 24450
>
> 540-464-7572
>
> *From:*archivesspace_users_group-bounces at lyralists.lyrasis.org
> <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>
> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org] *On
> Behalf Of *Mang Sun
> *Sent:* Wednesday, September 14, 2016 3:00 PM
> *To:* Archivesspace Users Group
> <archivesspace_users_group at lyralists.lyrasis.org
> <mailto:archivesspace_users_group at lyralists.lyrasis.org>>
> *Subject:* Re: [Archivesspace_Users_Group] hELP - No record found
> after test migration from v1.4.2 to 1.5.0
>
> Christine and other AS community members who have successfully
> migrated AS 1.4.2 to AS 1.5.x,
>
> We are trying to draw attention and seek help from AS development
> team and other community members who have successfully migrated AS
> 1.4.2 to AS 1.5.x. If necessary, we are willing to provide our MySQL
> dump file for you to test.
>
> Test migration from AS 1.4.2 to AS1.5.x really drives us mad. Though
> we don't quite see any (out of mem) error in the log, we still blindly
> pump up heap size (Xmx )to 6GB, stack size set o 3MB and rerun rebuild
> index or the complete migration steps, the problem is still there -
> Solr seemingly very silently stops writing or committing index files
> when the number of docs in Solr reaches to about 15,xxx as opposed to
> 500,xxx indexed docs found in our 1.4.2 instance, though thereafter
> we still can see the " end_commit_flush" message pop up in the log
> from time to time .Our test box environments are
>
> 1. Redhat 6.8 ,JAVA build 1.7.0_67 . MySQL 5.1.7 and 8GB memory
> 2.. Redhat 7.2 Virtual Machine,NYSQL 5.5.45, java 1.80_65,and 4GB.
> We also can confirm database side migration is largely done with
> new tables and fields added, existing records converted and top
> container records added. The problem is sticking to rebuilding the
> whole index - Solr stops writing and committing index prematurely
> after the number of docs grows to a small number(12xxx-15xxx) though
> commit flush can still be seen thereafter .
>
> Now we have exhausted ideas and means in figuring out why this "no
> record found " problem that seemingly only happens to us.
>
> Other experiment we did is export EAD file of one resource record from
> our production and load it into a 1.4.2 test instance and then upgrade
> this test instance to 1.5.x. this time the resource record can be
> indexed and appear(some issue with the Name ) in the test 1.5.x instance.
>
> I think there is some fundamental but very sneaky issue with 1.5.x
> indexing (rebuild index). The challenge is that no explicit error
> message that could be easily found in the log that can give us some
> hint and therefore this is a nightmare given the long waiting time
> before you can confirm a successful indexing or not. May be a
> standalone tool allowing user to manually rebuild index could be very
> helpful and straightforward, and less arcane.
>
> As mentioned earlier, records have been converted and migrated but not
> indexed, what else we tried is use direct route path (can be inferred
> from our production) to pull records out and update them by toggling
> through Suppress/Unsuppress or Publisesh/Unpublished , then we can
> get those updated records indexed. in the following diagram, we can
> see 3 resources, 2 agent records and some other record types
> highlighted in yellow appear in the types field, as opposed to the
> types (not highlighted) originally indexed and generated by the
> migration process. Also in the same diagram, the number of indexed
> docs is too small (we can see about half million docs in our 1.4.2
> production instance).Also we found a side issue with agent record, no
> matter what you set for Publish, Solr only sees it unpublished.But
> At least, this experiment can confirm there is not obvious problem
> with dynamic indexing.
>
> If AS developers or other AS users want to reproduce the issue or give
> a try for a solution, we can provide our v1.4.2 database dump or the
> v1.5.x database dump.
>
> Thank you very much.
>
> Mang Sun
>
> Rice U.
>
> On 9/9/2016 11:55 AM, Christine Di Bella wrote:
>
> Mang,
>
> How long ago did you start the reindex and how big is your
> database? The v1.5.1 upgrade improved indexing speed, especially
> for large databases, but the indexing does still take some time.
> People on this list can probably give you a sense of how long it
> took them for different sizes of databases. It’s probably been
> most common to run it overnight or over a weekend, when use of the
> system is low.
>
> (Some people were large databases were experiencing problems with
> indexing taking an excessive amount of time or not completing at
> all, which was one of the reasons for the quick release of v1.5.1.)
>
> Christine
>
> Christine Di Bella
>
> Community Outreach Manager
>
> christine.dibella at lyrasis.org <mailto:christine.dibella at lyrasis.org>
>
> 800.999.8558 x2905
>
> 678-235-2905
>
> cdibella13 (Skype)
>
> cid:image003.png at 01CE734E.FD759D30
>
> *From:*archivesspace_users_group-bounces at lyralists.lyrasis.org
> <mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>
> [mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org]
> *On Behalf Of *Mang Sun
> *Sent:* Friday, September 9, 2016 12:31 PM
> *To:* Archivesspace Users Group
> <archivesspace_users_group at lyralists.lyrasis.org>
> <mailto:archivesspace_users_group at lyralists.lyrasis.org>
> *Subject:* Re: [Archivesspace_Users_Group] No record found after
> test migration from v1.4.2 to 1.5.0
>
>
> I even didn't copy the data folder of the v1.4.2 into v.1.5.0.
> but even after I follow the recommended way by copying the data
> folder and removing those two subfolders mentioned by Joshua the
> problem is still there.Peeping into Solr schema, I found field
> "types" only have the following four types but not resource and
> digital_object,subject and agent . The missing types may explain
> why we find no records under every record type tab. Puzzled~ Mang
>
> ·12608
>
> otop_container
> <http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22top_container%22>
>
> ·1,169
>
> otree_view
> <http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22tree_view%22>
>
> ·75
>
> oarchival_object
> <http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22archival_object%22>
>
> ·1
>
> orepository
> <http://archives-test.library.rice.edu:8090/#/collection1/query?q=types:%22repository%22>
>
> On 9/9/2016 8:52 AM, Sally Vermaaten wrote:
>
> Hi Mang Sun,
>
> I agree with Joshua - this sounds to me like an indexing
> problem to me. In the course of upgrading to 1.5.0 did you
> delete the data/solr_index/index directory and all files in
> the data/indexer_state directory? This is one of the
> recommended steps in the instructions for upgrading to v.1.5.0
> <https://github.com/archivesspace/archivesspace/blob/master/UPGRADING_1.5.0.md>.
> We accidentally missed this step once in a running a test
> upgrade and definitely experienced some indexing problems
> though I don't remember the details. As it was a test upgrade
> (and it sounds like that's what you're doing as well) we just
> started afresh and ran the upgrade process over again. When we
> included that step, things were fine.
>
> On a related note - there's a 1.5.1 release of ArchivesSpace
> <https://github.com/archivesspace/archivesspace/releases/tag/v1.5.1> that
> includes several fixes for bugs that were found in the weeks
> after 1.5.0 was released. If are looking to update to a 1.5.x
> version, you may want to consider upgrading to 1.5.1 one
> rather than 1.5.0.
>
> Good luck,
>
> Sally
>
> On Fri, Sep 9, 2016 at 9:50 AM, Joshua D. Shaw
> <Joshua.D.Shaw at dartmouth.edu
> <mailto:Joshua.D.Shaw at dartmouth.edu>> wrote:
>
> I believe the suggestion for migration to 1.5+ was to
> delete the existing index and indexer states which will
> trigger a complete reindex.
>
> Best,
> Joshua
>
>
>
>
>
> On Sep 9, 2016, at 8:20 AM, "Mang Sun" <mang.sun at rice.edu
> <mailto:mang.sun at rice.edu>> wrote:
>
> > i
> >
> > After we migrate our v.1.4.2 test instance to v1.5.0, we
> can't find any records but just no record in Staff
> interface and public interface.
> >
> > But if we open a migrated record (saycollection record)
> using the direct route path to that record and resave it,
> this record but only this one starts to show up under
> Collections when browsing collections. This problem does
> not come with any new record created within v.1.5.0. Any
> suggestion?
> >
> > Thank you.
> >
> > Mang Sun
> >
> > Rice U.
> >
> >
> >
> > _______________________________________________
> > Archivesspace_Users_Group mailing list
> > Archivesspace_Users_Group at lyralists.lyrasis.org
> <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
> >
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>
>
> --
>
> Sally Vermaaten
> Project Manager, Archival Systems
> New York University Libraries
>
> 1-212-992-6259
>
>
>
>
> _______________________________________________
>
> Archivesspace_Users_Group mailing list
>
> Archivesspace_Users_Group at lyralists.lyrasis.org
> <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>
>
> _______________________________________________
>
> Archivesspace_Users_Group mailing list
>
> Archivesspace_Users_Group at lyralists.lyrasis.org
> <mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
>
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
>
>
>
> _______________________________________________
> Archivesspace_Users_Group mailing list
> Archivesspace_Users_Group at lyralists.lyrasis.org
> http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20160915/0cddb1cc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 41892 bytes
Desc: not available
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20160915/0cddb1cc/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 7645 bytes
Desc: not available
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20160915/0cddb1cc/attachment-0001.png>
More information about the Archivesspace_Users_Group
mailing list