[Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089 error
Gadsby, Eric T.
egadsby at towson.edu
Fri May 19 11:32:53 EDT 2023
Hi again so it looks like it’s Spunk….
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
chronyd 1018 chrony 5u IPv4 23682 0t0 UDP 127.0.0.1:323
chronyd 1018 chrony 6u IPv6 23683 0t0 UDP [::1]:323
sshd 1041 root 3u IPv4 24217 0t0 TCP *:22 (LISTEN)
sshd 1041 root 4u IPv6 24219 0t0 TCP *:22 (LISTEN)
omiengine 1184 omi 6u IPv6 27027 0t0 TCP *:1270 (LISTEN)
mysqld 1222 mysql 25u IPv6 29018 0t0 TCP *:3306 (LISTEN)
winbindd 1269 root 26u IPv4 26524 0t0 TCP 10.8.3.3:48320->136.160.181.196:49686 (ESTABLISHED)
winbindd 1269 root 27u IPv4 26557 0t0 TCP 10.8.3.3:50518->136.160.181.196:49669 (ESTABLISHED)
winbindd 1269 root 28u IPv4 88332 0t0 TCP 10.8.3.3:42876->136.160.181.196:389 (ESTABLISHED)
master 1819 root 14u IPv4 29098 0t0 TCP 127.0.0.1:25 (LISTEN)
master 1819 root 15u IPv6 29099 0t0 TCP [::1]:25 (LISTEN)
java 1825 solr 46u IPv6 31552 0t0 TCP 127.0.0.1:7983 (LISTEN)
java 1825 solr 156u IPv6 32642 0t0 TCP *:8983 (LISTEN)
splunkd 2323 splunk 4u IPv4 34989 0t0 TCP 127.0.0.1:8089 (LISTEN)
splunkd 2323 splunk 62u IPv4 89331 0t0 TCP 10.8.3.3:52400->136.160.181.121:9997 (ESTABLISHED)
splunkd 2323 splunk 65u IPv4 89328 0t0 TCP 10.8.3.3:57076->136.160.180.122:8089 (ESTABLISHED)
sshd 2512 root 4u IPv4 36189 0t0 TCP 10.8.3.3:22->10.19.130.72:59108 (ESTABLISHED)
sshd 2545 egadsby-admin 4u IPv4 36189 0t0 TCP 10.8.3.3:22->10.19.130.72:59108 (ESTABLISHED)
Am I reading this right? Thanks!
[Towson University logo]<http://www.towson.edu/>
Eric T. Gadsby
Pronouns: he/him/his
IT Operations Specialist | Albert S. Cook Library
—
P: 410-704-3340
egadsby at towson.edu<mailto:egadsby at towson.edu> | libraries.towson.edu<http://www.towson.edu/https:/libraries.towson.edu>
—
Confidentiality Notice: This message may contain information that is confidential, privileged, proprietary, or otherwise legally exempt from disclosure. If you are not the intended recipient, you are notified that you are not authorized to read, print, copy or disseminate this message, any part of it, or any attachments. If this message has been sent to you in error, please notify the sender by replying to this transmission, or by calling Albert S. Cook Library at 410-704-3340 .
From: archivesspace_users_group-bounces at lyralists.lyrasis.org <archivesspace_users_group-bounces at lyralists.lyrasis.org> on behalf of Tom Hanstra <hanstra at nd.edu>
Date: Friday, May 19, 2023 at 11:30 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org>
Subject: Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089 error
[ CAUTION: This email is from outside of TU. Use caution before clicking links or opening attachments. If suspicious, report to phishing at towson.edu. ]
Yes, that is just Solr. But if this is a new server could you just turn off Splunk for a while just to see that it is not the conflict? Nothing in what you've shown us so far would suggest that there is a conflict on port 8089. But if Splunk might be the issue, try turning it off and then starting ArchivesSpace
On Fri, May 19, 2023 at 11:22 AM Gadsby, Eric T. <egadsby at towson.edu<mailto:egadsby at towson.edu>> wrote:
ps -ef | grep java =
“java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX :MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filec ount=9,filesize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -Dsolr.log.dir=/var/solr/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/logs -Djetty.home =/opt/solr/server -Dsolr.solr.home=/var/solr/data -Dsolr.data.home= -Dsolr.install.dir=/opt/solr -Dsolr.default.confdir=/opt/solr/server/solr/configsets/ _default/conf -Dlog4j.configurationFile=/var/solr/log4j2.xml -Xss256k -Dsolr.log.muteconsole -jar start.jar --module=http --module=gzip
egadsby+ 6561 2546 0 11:18 pts/0 00:00:00 grep --color=auto java”
Right now it just seems to be Solr running…. Based on my read of this output. I might be wrong.
It is a new 3.3.1 server with a fresh install with the data base from an old server being imported from a SQL dump and then migrated using the script. Thanks!
[Towson University logo]<http://www.towson.edu/>
Eric T. Gadsby
Pronouns: he/him/his
IT Operations Specialist | Albert S. Cook Library
—
P: 410-704-3340
egadsby at towson.edu<mailto:egadsby at towson.edu> | libraries.towson.edu<http://www.towson.edu/https:/libraries.towson.edu>
—
Confidentiality Notice: This message may contain information that is confidential, privileged, proprietary, or otherwise legally exempt from disclosure. If you are not the intended recipient, you are notified that you are not authorized to read, print, copy or disseminate this message, any part of it, or any attachments. If this message has been sent to you in error, please notify the sender by replying to this transmission, or by calling Albert S. Cook Library at 410-704-3340 .
From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> <archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> on behalf of Tom Hanstra <hanstra at nd.edu<mailto:hanstra at nd.edu>>
Date: Friday, May 19, 2023 at 11:14 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089<http://0.0.0.0/0.0.0.0:8089> error
[ CAUTION: This email is from outside of TU. Use caution before clicking links or opening attachments. If suspicious, report to phishing at towson.edu<mailto:phishing at towson.edu>. ]
When connected to the server, use the command:
ps -ef | grep java
and look to see if there are java processes running. If there is an old one, you will have to stop/kill it before starting the new one.
Is this a completely new server only running 3.3.1 ? If so, the solr process will also show up using that command.
Tom
On Fri, May 19, 2023 at 10:55 AM Gadsby, Eric T. <egadsby at towson.edu<mailto:egadsby at towson.edu>> wrote:
I wondered that, how would I check? Going to a browser yields nothing… that said my firewall rules may not be in place? Thanks!
Error! Filename not specified.<http://www.towson.edu/>
Eric T. Gadsby
Pronouns: he/him/his
IT Operations Specialist | Albert S. Cook Library
—
P: 410-704-3340
egadsby at towson.edu<mailto:egadsby at towson.edu> | libraries.towson.edu<http://www.towson.edu/https:/libraries.towson.edu>
—
Confidentiality Notice: This message may contain information that is confidential, privileged, proprietary, or otherwise legally exempt from disclosure. If you are not the intended recipient, you are notified that you are not authorized to read, print, copy or disseminate this message, any part of it, or any attachments. If this message has been sent to you in error, please notify the sender by replying to this transmission, or by calling Albert S. Cook Library at 410-704-3340 .
From: archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org> <archivesspace_users_group-bounces at lyralists.lyrasis.org<mailto:archivesspace_users_group-bounces at lyralists.lyrasis.org>> on behalf of Tom Hanstra <hanstra at nd.edu<mailto:hanstra at nd.edu>>
Date: Friday, May 19, 2023 at 10:53 AM
To: Archivesspace Users Group <archivesspace_users_group at lyralists.lyrasis.org<mailto:archivesspace_users_group at lyralists.lyrasis.org>>
Subject: Re: [Archivesspace_Users_Group] Trying to spin up a new server and it won't load... Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089<http://0.0.0.0/0.0.0.0:8089> error
[ CAUTION: This email is from outside of TU. Use caution before clicking links or opening attachments. If suspicious, report to phishing at towson.edu<mailto:phishing at towson.edu>. ]
First thought would be that the new service is trying to start while the old one is still running. Could that be?
Tom
On Fri, May 19, 2023 at 10:44 AM Gadsby, Eric T. <egadsby at towson.edu<mailto:egadsby at towson.edu>> wrote:
Dear Friends,
I am in the process of moving our old v2.7.1 to V3.3.1. I have been able to set up Solr, the application, move the data and start ArchivesSpace twice in our testing environment (CentOS8 Stream). Today I ran the same procedure in our production environment (RHEL8.5) and ArchivesSpace won't start.
The log reports this error: "Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089<http://0.0.0.0/0.0.0.0:8089>" -- of at least I think that's the problem (I'm going to provide the whole log:
"$ cat archivesspace.out
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Loading ArchivesSpace configuration file from path: /opt/archivesspace/config/config.rb
Loading ArchivesSpace configuration file from path: /opt/archivesspace/config/config.rb
Loading ArchivesSpace configuration file from path: /opt/archivesspace/config/config.rb
May 19, 2023 10:01:02 AM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @7281ms to org.eclipse.jetty.util.log.Slf4jLog
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9332; jvm 11.0.19+7-LTS
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: node0 Scavenging every 600000ms
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.handler.ContextHandler$Contextlog
INFO: INFO: jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.19+7-LTS on 11.0.19+7-LTS +jit [linux-x86_64]
May 19, 2023 10:01:03 AM org.eclipse.jetty.server.handler.ContextHandler$Contextlog
INFO: INFO: using a shared (threadsafe!) runtime
uri:classloader:/jruby/rack/response.rb:294: warning: constant ::Fixnum is deprecated
uri:classloader:/jruby/rack/core_ext.rb:26: warning: constant ::NativeExceptionis deprecated
Loading ArchivesSpace configuration file from path: /opt/archivesspace/config/config.rb
ArchivesSpaceThreadDump: Touch the file '/opt/archivesspace/thread_dump_backend.txt' to trigger a thread dump
I, [2023-05-19T10:01:14.847038 #1570] INFO -- : Thread-2002: Connecting to database: jdbc:mysql://localhost:3306/archivesspace?user=[REDACTED]&password=[REDACT ED]&useUnicode=true&characterEncoding=UTF-8. Max connections: 28
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver classis `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
I, [2023-05-19T10:01:15.866939 #1570] INFO -- : Thread-2002: All tables checkedand confirmed set to UTF-8. Nice job!
I, [2023-05-19T10:01:16.615214 #1570] INFO -- : Thread-2002: Solr config checksum verification ok.
I, [2023-05-19T10:01:19.772389 #1570] INFO -- : Thread-2002: Starting job scheduler
/opt/archivesspace/gems/gems/rufus-scheduler-2.0.24/lib/rufus/sc/cronline.rb:80:warning: constant ::Fixnum is deprecated
I, [2023-05-19T10:01:20.446880 #1570] INFO -- : Thread-2002: Updating system preferences
I, [2023-05-19T10:01:21.067956 #1570] INFO -- : Thread-2914: Starting background job thread 1 (2914)
I, [2023-05-19T10:01:21.074939 #1570] INFO -- : Thread-2916: Starting background job thread 2 (2916)
May 19, 2023 10:01:21 AM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started o.e.j.w.WebAppContext at 26f5e45d{/,file:///opt/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5382710903715848903/webapp/,AVAILABLE}{/opt/archivesspace/wars/backend.war}<mailto:o.e.j.w.WebAppContext at 26f5e45d%7b/,file:///opt/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5382710903715848903/webapp/,AVAILABLE%7d%7b/opt/archivesspace/wars/backend.war%7d>
========================================================================
A trace file has been written to the following location: /tmp/aspace_diagnostic_1684504881.txt
This file contains information that will assist developers in diagnosing
problems with your ArchivesSpace installation. Please review the file's
contents for sensitive information (such as passwords) that you might not
want to share.
========================================================================
Unhandled Java exception: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089<http://0.0.0.0/0.0.0.0:8089>
java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8089<http://0.0.0.0/0.0.0.0:8089>
openAcceptChannel at org/eclipse/jetty/server/ServerConnector.java:349
open at org/eclipse/jetty/server/ServerConnector.java:310
doStart at org/eclipse/jetty/server/AbstractNetworkConnector.java:80
doStart at org/eclipse/jetty/server/ServerConnector.java:234
start at org/eclipse/jetty/util/component/AbstractLifeCycle.java:73
doStart at org/eclipse/jetty/server/Server.java:401
start at org/eclipse/jetty/util/component/AbstractLifeCycle.java:73
invoke0 at jdk/internal/reflect/NativeMethodAccessorImpl.java:-2
invoke at jdk/internal/reflect/NativeMethodAccessorImpl.java:62
invoke at jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43
invoke at java/lang/reflect/Method.java:566
invokeDirectWithExceptionHandling at org/jruby/javasupport/JavaMethod.java:441
invokeDirect at org/jruby/javasupport/JavaMethod.java:305
invokeOther132:start at launcher/launcher.rb:90
start_server at launcher/launcher.rb:90
invokeOther193:start_server at launcher/launcher.rb:144
main at launcher/launcher.rb:144
invokeOther303:main at launcher/launcher.rb:242
<main> at launcher/launcher.rb:242
invokeWithArguments at java/lang/invoke/MethodHandle.java:710
runScript at org/jruby/Ruby.java:1205
runNormally at org/jruby/Ruby.java:1128
runNormally at org/jruby/Ruby.java:1146
runFromMain at org/jruby/Ruby.java:958
doRunFromMain at org/jruby/Main.java:400
internalRun at org/jruby/Main.java:292
run at org/jruby/Main.java:234
main at org/jruby/Main.java:206
Caused by:
java.net.BindException: Address already in use
bind0 at sun/nio/ch/Net.java:-2
bind at sun/nio/ch/Net.java:459
bind at sun/nio/ch/Net.java:448
bind at sun/nio/ch/ServerSocketChannelImpl.java:227
bind at sun/nio/ch/ServerSocketAdaptor.java:80
openAcceptChannel at org/eclipse/jetty/server/ServerConnector.java:344
open at org/eclipse/jetty/server/ServerConnector.java:310
doStart at org/eclipse/jetty/server/AbstractNetworkConnector.java:80
doStart at org/eclipse/jetty/server/ServerConnector.java:234
start at org/eclipse/jetty/util/component/AbstractLifeCycle.java:73
doStart at org/eclipse/jetty/server/Server.java:401
start at org/eclipse/jetty/util/component/AbstractLifeCycle.java:73
invoke0 at jdk/internal/reflect/NativeMethodAccessorImpl.java:-2
invoke at jdk/internal/reflect/NativeMethodAccessorImpl.java:62
invoke at jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43
invoke at java/lang/reflect/Method.java:566
invokeDirectWithExceptionHandling at org/jruby/javasupport/JavaMethod.java:441
invokeDirect at org/jruby/javasupport/JavaMethod.java:305
call at org/jruby/java/invokers/InstanceMethodInvoker.java:32
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:355
call at org/jruby/runtime/callsite/CachingCallSite.java:144
invokeOther132:start at launcher/launcher.rb:90
RUBY$method$start_server$1 at launcher/launcher.rb:90
call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:80
call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:179
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:397
call at org/jruby/runtime/callsite/CachingCallSite.java:206
invokeOther193:start_server at launcher/launcher.rb:144
RUBY$method$main$6 at launcher/launcher.rb:144
call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:156
cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:355
call at org/jruby/runtime/callsite/CachingCallSite.java:144
invokeOther303:main at launcher/launcher.rb:242
RUBY$script at launcher/launcher.rb:242
invokeWithArguments at java/lang/invoke/MethodHandle.java:710
load at org/jruby/ir/Compiler.java:89
runScript at org/jruby/Ruby.java:1205
runNormally at org/jruby/Ruby.java:1128
runNormally at org/jruby/Ruby.java:1146
runFromMain at org/jruby/Ruby.java:958
doRunFromMain at org/jruby/Main.java:400
internalRun at org/jruby/Main.java:292
run at org/jruby/Main.java:234
main at org/jruby/Main.java:206"
I have Solr and ArchivesSpace set to start as init scripts. I would appreciate any insight that folks could offer would be great. Thanks!
[Towson University logo]<http://www.towson.edu/>
Eric T. Gadsby
Pronouns: he/him/his
IT Operations Specialist | Albert S. Cook Library
—
P: 410-704-3340
egadsby at towson.edu<mailto:egadsby at towson.edu> | libraries.towson.edu<http://www.towson.edu/https:/libraries.towson.edu>
—
Confidentiality Notice: This message may contain information that is confidential, privileged, proprietary, or otherwise legally exempt from disclosure. If you are not the intended recipient, you are notified that you are not authorized to read, print, copy or disseminate this message, any part of it, or any attachments. If this message has been sent to you in error, please notify the sender by replying to this transmission, or by calling Albert S. Cook Library at 410-704-3340 .
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>
Error! Filename not specified.
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>
Error! Filename not specified.
_______________________________________________
Archivesspace_Users_Group mailing list
Archivesspace_Users_Group at lyralists.lyrasis.org<mailto:Archivesspace_Users_Group at lyralists.lyrasis.org>
http://lyralists.lyrasis.org/mailman/listinfo/archivesspace_users_group
--
Tom Hanstra
Sr. Systems Administrator
hanstra at nd.edu<mailto:hanstra at nd.edu>
[Image removed by sender.]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230519/217a388f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1826 bytes
Desc: image001.jpg
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230519/217a388f/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1826 bytes
Desc: image001.jpg
URL: <http://lyralists.lyrasis.org/pipermail/archivesspace_users_group/attachments/20230519/217a388f/attachment-0001.jpg>
More information about the Archivesspace_Users_Group
mailing list