I want to use gsutil to download cloud storage's bucket file, but it is continuing retrying request. I use the gsutil command in Mac Terminal.
The "gcloud int" has succeeded. Besides, gsutil can work in Cloud Shell.
gsutil ls gs://tm_ufs
INFO 0604 14:17:49.749002 retry_util.py] Retrying request, attempt #1...
INFO 0604 14:18:51.684501 retry_util.py] Retrying request, attempt #2...
INFO 0604 14:19:54.875942 retry_util.py] Retrying request, attempt #3...
INFO 0604 14:21:03.459085 retry_util.py] Retrying request, attempt #4...
INFO 0604 14:22:22.220589 retry_util.py] Retrying request, attempt #5...
INFO 0604 14:23:48.716924 retry_util.py] Retrying request, attempt #6...
INFO 0604 14:25:20.733894 retry_util.py] Retrying request, attempt #7...
INFO 0604 14:26:52.750410 retry_util.py] Retrying request, attempt #8...
INFO 0604 14:28:24.763793 retry_util.py] Retrying request, attempt #9...
There has been a previous discussion on the topic:
gcloud can connect but gsutil cannot
Turned out to be proxy issue. I suspect it might be the problem in this case as well.
Related
I was using wget to download GEDI data from LP DAAC data pool. It always returns an error of "unable to establish SSL connection". I attempted wget in promote or Pycharm and added the "--no-check-certificate" configuration.
The wget is the newest release (1.21.3,64bit).
OS: windows11.
from the following massages, I guess the connection to EarthData is successful because it returns the data downloading link that I can open manually in the browser and then can start downloading. This error could happen in the last step that wget starts accessing the returned link and then downloading.
returned messages:
--2022-08-14 09:51:09-- https://e4ftl01.cr.usgs.gov//GEDI_L1_L2/GEDI/GEDI01_B.002/2019.04.20/GEDI01_B_2019110092939_O01996_01_T03334_02_005_01_V002.h5
Resolving e4ftl01.cr.usgs.gov (e4ftl01.cr.usgs.gov)... 2001:49c8:4000:127d::133:130, 152.61.133.130
Connecting to e4ftl01.cr.usgs.gov (e4ftl01.cr.usgs.gov)|2001:49c8:4000:127d::133:130|:443... failed: Bad file descriptor.
Connecting to e4ftl01.cr.usgs.gov (e4ftl01.cr.usgs.gov)|152.61.133.130|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?scope=uid&app_type=401&client_id=ijpRZvb9qeKCK5ctsn75Tg&response_type=code&redirect_uri=https%3A%2F%2Fe4ftl01.cr.usgs.gov%2Foauth&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292Ly9HRURJX0wxX0wyL0dFREkvR0VESTAxX0IuMDAyLzIwMTkuMDQuMjAvR0VESTAxX0JfMjAxOTExMDA5MjkzOV9PMDE5OTZfMDFfVDAzMzM0XzAyXzAwNV8wMV9WMDAyLmg1 [following]
--2022-08-14 09:51:55-- https://urs.earthdata.nasa.gov/oauth/authorize?scope=uid&app_type=401&client_id=ijpRZvb9qeKCK5ctsn75Tg&response_type=code&redirect_uri=https%3A%2F%2Fe4ftl01.cr.usgs.gov%2Foauth&state=aHR0cHM6Ly9lNGZ0bDAxLmNyLnVzZ3MuZ292Ly9HRURJX0wxX0wyL0dFREkvR0VESTAxX0IuMDAyLzIwMTkuMDQuMjAvR0VESTAxX0JfMjAxOTExMDA5MjkzOV9PMDE5OTZfMDFfVDAzMzM0XzAyXzAwNV8wMV9WMDAyLmg1
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 2001:4d0:241a:4081::89, 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|2001:4d0:241a:4081::89|:443... failed: Bad file descriptor.
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
Unable to establish SSL connection.
I am trying to install a simple job in Gitlab-CI. It should just copy most of the files (.php) in the repository to a different folder (from repository to Apache's /var/www/html/...).
The output goes like this:
Using Shell executor...
Preparing environment
Running on my.server...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialisierte existierendes Git-Projektarchiv in /home/gitlab-runner/builds/TyGXtawE/0/sdp-dev/ibo-php/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.sustainable-data-platform.org/sdp-dev/ibo-php.git/':
The requested URL returned error: 500
So I can find the corresponding problem in /var/log/gitlab/gitlab-rails/production.log
Started POST "/api/v4/jobs/request" for ip.ip.ip.ip at 2022-04-26 14:27:41 +0200
Started GET "/sdp-dev/ibo-php.git/info/refs?service=git-upload-pack" for ip.ip.ip.ip at 2022-04-26 14:27:44 +0200
Processing by Repositories::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "repository_path"=>"sdp-dev/ibo-php.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 14ms (Views: 1.3ms | ActiveRecord: 1.5ms | Elasticsearch: 0.0ms | Allocations: 4656)
Started GET "/sdp-dev/ibo-php.git/info/refs?service=git-upload-pack" for ip.ip.ip.ip at 2022-04-26 14:27:44 +0200
Processing by Repositories::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "repository_path"=>"sdp-dev/ibo-php.git"}
Completed 500 Internal Server Error in 34ms (ActiveRecord: 7.6ms | Elasticsearch: 0.0ms | Allocations: 7564)
JWT::DecodeError (Nil JSON web token):
lib/gitlab/jwt_authenticatable.rb:17:in `decode_jwt_for_issuer'
lib/gitlab/workhorse.rb:210:in `decode_jwt'
# ... stacktrace
I have installed the runner from "https://gitlab-runner-downloads.s3.amazonaws.com/latest/rpm/gitlab-runner_amd64.rpm" and gave it the token from Gitlab's GUI.
Gitlab (v 14.5.2) and the runner (v 14.10.0) are running on the same machine.
All this might be overhead for some cp -r, but I would like to understand the error for upcoming more complex jobs. Any idea how to debug the registration-token, how to find the problem?
Update
403 for Gitlab Runner from Gitlab?
It seems like gitlab-runner just gets a 403 (forbidden) from Gitlab. So I might need to change something in the corresponding (Apache). But how can I debug this (ERROR: Checking for jobs... forbidden)?
How to register a runner?
There is this command: sudo gitlab-runner register --url $URL --registration-token $REGISTRATION_TOKEN. But when I put the right data (URL, REGISTRATION_TOKEN) in this command (not using bash variables!), I am asked for the same data again (URL, TOKEN) and I just do c/p. E.g. Type the URL [MY_URL]:, then I c/p MY_URL and hit enter. Is this OK?
token / unhealthy runner problems
No success after creating a new token in gitlab and registering a new runner, also after updating Gitlab to v14.10.
But I can see some problems in /var/log/messages.
systemd: Started GitLab Runner.
gitlab-runner: Runtime platform #033[0;m arch#033[0;m=amd64 os#033[0;m=linux pid#033[0;m=20848 revision#033[0;m=c6bb62f6 version#033[0;m=14.10.0
gitlab-runner: Starting multi-runner from /etc/gitlab-runner/config.toml...#033[0;m builds#033[0;m=0
gitlab-runner: Running in system-mode. #033[0;m
gitlab-runner: #033[0;m
gitlab-runner: Configuration loaded #033[0;m builds#033[0;m=0
gitlab-runner: listen_address not defined, metrics & debug endpoints disabled#033[0;m builds#033[0;m=0
gitlab-runner: [session_server].listen_address not defined, session endpoints disabled#033[0;m builds#033[0;m=0
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Runner {URL}/{token} is not healthy and will be disabled!#033[0;m
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Runner {URL}/{token} is not healthy and will be disabled!#033[0;m
I am interested in listen_address not defined, metrics & debug endpoints disabled and Checking for jobs... forbidden.
You can see this problem elsewhere in SO, but I could not find a solution (googling etc.).
Here? https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28965
Some things you might try:
Create a new token
Update Gitlab
We have a Kerberised cluster and I'm trying to run a Java action in Oozie where I make a JDBC connection to Hive. This JDBC connections works fine on the Sandbox without Kerberos.
The connection string is as simple as the following, where I'm providing username and password in it:
Connection con = DriverManager.getConnection("jdbc:hive2://W12345:10000/control;principal=hive/W12345.companynet.net#COMPANYNET.NET","user123","passw123");
The Oozie action (strangely) completes succesfully, and the Java action log does not present any error:
1742 [main] INFO org.apache.hive.jdbc.Utils - Supplied authorities: W12345:10000
1742 [main] INFO org.apache.hive.jdbc.Utils - Resolved authority: W12345:10000
1766 [main] INFO org.apache.hive.jdbc.HiveConnection - Will try to open client transport with JDBC Uri: jdbc:hive2://W12345:10000/control;principal=hive/W12345.companynet.net#COMPANYNET.NET
<<< Invocation of Main class completed <<<
Oozie Launcher ends
1785 [main] INFO org.apache.hadoop.mapred.Task - Task:attempt_1464245290012_0129_m_000000_0 is done. And is in the process of committing
1847 [main] INFO org.apache.hadoop.mapred.Task - Task attempt_1464245290012_0129_m_000000_0 is allowed to commit now
1854 [main] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_1464245290012_0129_m_000000_0' to hdfs://danskehadoop/user/user123/oozie-oozi/0000013-160527101253015-oozie-oozi-W/JavaAction--java/output/_temporary/1/task_1464245290012_0129_m_000000
1909 [main] INFO org.apache.hadoop.mapred.Task - Task 'attempt_1464245290012_0129_m_000000_0' done.
But in reality the Java main does not complete correctly the execution (and does not execute the needed queries) because the JDBC connection fails with an exception that I can see only in the Hive log:
ERROR [HiveServer2-Handler-Pool: Thread-78363]: server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
I'm actually connected to the cluster, and already done further kinit on my username.
Does anybody know what could the cause of this exception be?
Thanks in advance for the help!
Antonio
This happened to me on MapR hadoop distribution platform.
In my case it was Keepalived checking Hive port every 5 seconds and producing such error. I simply used "nc" command to check if Hive port is in use and did not use any authentication method. Later I switched to "maprcli" command which uses SASL authentication and the error was gone.
I'm trying to run the samples from https://github.com/aspnet/home.
But I'm having problems updating the references. The restore is very slow and fails for some packages:
GET http://www.myget.org/F/aspnetvnext/api/v2/FindPackagesById()?id='Microsoft.Framework.Configuration.EnvironmentVariables'
Warning: FindPackagesById: Microsoft.Framework.Configuration.EnvironmentVariables
HTTP request timed out. Retrying.
GET http://www.nuget.org/api/v2/FindPackagesById()?id='Microsoft.Framework.Configuration.EnvironmentVariables'
Warning: FindPackagesById: Microsoft.Framework.Configuration.Ini
HTTP request timed out. Retrying.
GET http://www.myget.org/F/aspnetvnext/api/v2/FindPackagesById()?id='Microsoft.Framework.Configuration.Ini'
Warning: FindPackagesById: Microsoft.Framework.Configuration.Ini
HTTP request timed out. Retrying.
GET http://www.nuget.org/api/v2/FindPackagesById()?id='Microsoft.Framework.Configuration.Ini'
Warning: FindPackagesById: Microsoft.Framework.Logging
HTTP request timed out. Retrying.
GET http://www.myget.org/F/aspnetvnext/api/v2/FindPackagesById()?id='Microsoft.Framework.Logging'
Warning: FindPackagesById: Microsoft.Framework.Logging
HTTP request timed out. Retrying.
GET http://www.nuget.org/api/v2/FindPackagesById()?id='Microsoft.Framework.Logging'
Warning: FindPackagesById: Newtonsoft.Json
HTTP request timed out. Retrying.
GET http://www.myget.org/F/aspnetvnext/api/v2/FindPackagesById()?id='Newtonsoft.Json'
Warning: FindPackagesById: Newtonsoft.Json
HTTP request timed out. Retrying.
GET http://www.nuget.org/api/v2/FindPackagesById()?id='Newtonsoft.Json'
Error: FindPackagesById: Microsoft.AspNet.Mvc.Razor
HTTP request timed out. Exiting.
I tried with different urls using nuget, myget, http, https.
I checked my internet connection.
There is a related issue in the dnx repo, but it did not help.
Any workaround to try?
I'm using mono 4.0.3.20 on CentOS.
Here is the workaround:
export MONO_THREADS_PER_CPU=2000
from http://www.mono-project.com/archived/articlethreadpool_deadlocks/
I'm using Vagrant ssh to connect to a remote Grinder agent while running the console locally:
vagrant ssh agent01 -c "./startAgent.sh" -- -R 6372:localhost:6372
The console can talk to the agent, and start the agent's threads:
2015-07-24 10:12:54,391 INFO agent: The Grinder 3.11
2015-07-24 10:12:54,507 INFO agent: connected to console at /127.0.0.1:6372
2015-07-24 10:12:54,507 INFO agent: waiting for console signal
2015-07-24 10:12:57,869 INFO agent: received a start message
2015-07-24 10:12:57,887 INFO agent: Worker process command line: java '-javaagent:grinder-3.11/lib/grinder-dcr-agent-3.11.jar' -classpath 'grinder-3.11/lib/grinder.jar' net.grinder.engine.process.WorkerProcessEntryPoint
2015-07-24 10:12:57,967 INFO agent: worker agent01-0 started
2015-07-24 10:12:57,976 INFO agent: worker agent01-1 started
2015-07-24 10:12:58,007 INFO agent: worker agent01-2 started
2015-07-24 10:12:58,032 INFO agent: worker agent01-3 started
2015-07-24 10:12:58,059 INFO agent: worker agent01-4 started
2015-07-24 10:12:58,095 INFO agent: worker agent01-5 started
2015-07-24 10:12:58,156 INFO agent: worker agent01-6 started
2015-07-24 10:12:58,182 INFO agent: worker agent01-7 started
2015-07-24 10:12:58,214 INFO agent: worker agent01-8 started
2015-07-24 10:12:58,250 INFO agent: worker agent01-9 started
Shortly thereafter, though, the agents say:
2015-07-24 10:13:38,016 INFO agent01-2: Report to console failed
net.grinder.communication.CommunicationException: Exception whilst sending message
at net.grinder.communication.AbstractSender.send(AbstractSender.java:57) ~[grinder-core-3.11.jar:na]
at net.grinder.communication.QueuedSenderDecorator.flush(QueuedSenderDecorator.java:60) ~[grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess.sendStatusMessage(GrinderProcess.java:638) [grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess.access$1100(GrinderProcess.java:110) [grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess$ReportToConsoleTimerTask.run(GrinderProcess.java:615) ~[grinder-core-3.11.jar:na]
at net.grinder.engine.process.GrinderProcess.run(GrinderProcess.java:465) [grinder-core-3.11.jar:na]
at net.grinder.engine.process.WorkerProcessEntryPoint.run(WorkerProcessEntryPoint.java:86) [grinder-core-3.11.jar:na]
at net.grinder.engine.process.WorkerProcessEntryPoint.main(WorkerProcessEntryPoint.java:59) [grinder-core-3.11.jar:na]
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_75]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_75]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_75]
...
and never provide data to the console.
What causes this? Is there ssh configuration I need to tweak?
To fix this, I added this line to /etc/ssh/sshd_config on the grinder agent host:
MaxStartups 1000
and changed the grinder agent properties from:
grinder.processes = 10
grinder.threads = 100
to
grinder.processes = 4
grinder.threads = 250