Failed s3fs mount due to Timezone skew - amazon-s3

Apr 22 05:54:59 ubuntuserver s3fs[10143]: s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF]
Apr 22 05:54:59 ubuntuserver s3fs[10143]: PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
Apr 22 05:54:59 ubuntuserver s3fs[10145]: init v1.85(commit:381835e) with OpenSSL
Apr 22 05:54:59 ubuntuserver s3fs[10145]: check services.
Apr 22 05:54:59 ubuntuserver s3fs[10145]: check a bucket.
Apr 22 05:54:59 ubuntuserver s3fs[10145]: curl.cpp:ResetHandle(1879): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later.
Apr 22 05:54:59 ubuntuserver s3fs[10145]: URL is https://s3-us-west-2.amazonaws.com/bucketubuntuserver/
Apr 22 05:54:59 ubuntuserver s3fs[10145]: URL changed is https://bucketubuntuserver.s3-us-west-2.amazonaws.com/
Apr 22 05:55:01 ubuntuserver s3fs[10145]: curl.cpp:RequestPerform(2273): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message>
<RequestTime>Mon, 22 Apr 2019 05:54:59 GMT</RequestTime>
<ServerTime>2019-04-22T06:23:01Z</ServerTime>
<MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds>
<RequestId>2CDB15BFC9072D0D</RequestId><HostId>grA/XIvT7zLUh9jLUxYGAs8jOtMs762CPMX+TM6GdAVvAB36/b8hH0dVOugVBWRpHX3O63V2Bv8=</HostId></Error>
Apr 22 05:55:01 ubuntuserver s3fs[10145]: curl.cpp:CheckBucket(3305): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message>
<RequestTime>Mon, 22 Apr 2019 05:54:59 GMT</RequestTime>
<ServerTime>2019-04-22T06:23:01Z</ServerTime>
<MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>2CDB15BFC9072D0D</RequestId><HostId>grA/XIvT7zLUh9jLUxYGAs8jOtMs762CPMX+TM6GdAVvAB36/b8hH0dVOugVBWRpHX3O63V2Bv8=</HostId></Error>
Apr 22 05:55:01 ubuntuserver s3fs[10145]: s3fs.cpp:s3fs_check_service(3868): invalid credentials(host=https://s3-us-west-2.amazonaws.com) - result of checking service.
Apr 22 05:55:01 ubuntuserver s3fs[10145]: Pool full: destroy the oldest handler
Apr 22 05:55:01 ubuntuserver s3fs[10145]: s3fs.cpp:s3fs_exit_fuseloop(3444): Exiting FUSE event loop due to errors
Apr 22 05:55:01 ubuntuserver s3fs[10145]: destroy
i had my credentials correct,but i wasnt able to mount s3 due to the clock difference. My server is using UTC which was late by 26 minutes. My problem is solved by fixing ntp sync but:-
1) I want to confirm if the s3fs or any aws tool i use also send the Clock information to S3 ? as is present but its GMT instead of UTC. The s3 seems to be using UTC when comparing it to servers properly synced to ntp.
2) Can we use any timezone provided that is properly synced with good NTP server ?

S3 signs requests including the client's current time to prevent attackers from replaying requests at a later time. Thus if your client has the incorrect time, the server will treat it as an invalid request. Both the client and server use UTC/GMT; the time zone does not matter. Configuring ntp as you did should resolve these issues.

Related

Redis service crashes with "Failed opening the RDB file systemdd (in server root dir /etc/cron.d) for saving: Permission denied"

I am running Redis server version 6.0.6 on Ubuntu 20.04. The process is run by "redis" user.
Sometimes, the Redis process crashes and gets restarted on its own and when this happens, lot of data cached in Redis becomes unavailable. This happens every few days/weeks. I can see the following messages in the logs - saving was working fine till 2:32:43 and suddenly failed at 2:34:15:
133121:C 23 Jun 2021 02:27:54.383 * RDB: 22 MB of memory used by copy-on-write
105798:M 23 Jun 2021 02:27:54.511 * Background saving terminated with success
105798:M 23 Jun 2021 02:29:46.279 * 10000 changes in 60 seconds. Saving...
105798:M 23 Jun 2021 02:29:46.354 * Background saving started by pid 133125
133125:C 23 Jun 2021 02:30:16.363 * DB saved on disk
133125:C 23 Jun 2021 02:30:16.464 * RDB: 18 MB of memory used by copy-on-write
105798:M 23 Jun 2021 02:30:16.583 * Background saving terminated with success
105798:M 23 Jun 2021 02:32:14.138 * 10000 changes in 60 seconds. Saving...
105798:M 23 Jun 2021 02:32:14.222 * Background saving started by pid 133131
133131:C 23 Jun 2021 02:32:42.924 * DB saved on disk
133131:C 23 Jun 2021 02:32:42.988 * RDB: 22 MB of memory used by copy-on-write
105798:M 23 Jun 2021 02:32:43.123 * Background saving terminated with success
105798:M 23 Jun 2021 02:34:14.958 * DB saved on disk
105798:M 23 Jun 2021 02:34:15.705 # Failed opening the RDB file systemdd (in server root dir /etc/cron.d) for saving: Permission denied
=== REDIS BUG REPORT START: Cut & paste starting from here ===
105798:M 23 Jun 2021 02:34:15.705 # Redis 6.0.6 crashed by signal: 11
105798:M 23 Jun 2021 02:34:15.705 # Crashed running the instruction at: 0x55f2e7e35099
105798:M 23 Jun 2021 02:34:15.705 # Accessing address: 0x149968
105798:M 23 Jun 2021 02:34:15.705 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/usr/bin/redis-server 172.16.106.88:6379(je_malloc_usable_size+0x89)[0x55f2e7e35099]
Backtrace:
/usr/bin/redis-server 172.16.106.88:6379(logStackTrace+0x4f)[0x55f2e7db2bcf]
/usr/bin/redis-server 172.16.106.88:6379(sigsegvHandler+0xb5)[0x55f2e7db33d5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fb934c173c0]
/usr/bin/redis-server 172.16.106.88:6379(je_malloc_usable_size+0x89)[0x55f2e7e35099]
/usr/bin/redis-server 172.16.106.88:6379(+0x50b79)[0x55f2e7d72b79]
/usr/bin/redis-server 172.16.106.88:6379(rdbSave+0x2ba)[0x55f2e7d9345a]
/usr/bin/redis-server 172.16.106.88:6379(saveCommand+0x67)[0x55f2e7d94ab7]
/usr/bin/redis-server 172.16.106.88:6379(call+0xb1)[0x55f2e7d6a8b1]
/usr/bin/redis-server 172.16.106.88:6379(processCommand+0x4a6)[0x55f2e7d6b446]
/usr/bin/redis-server 172.16.106.88:6379(processCommandAndResetClient+0x14)[0x55f2e7d799e4]
/usr/bin/redis-server 172.16.106.88:6379(processInputBuffer+0x18f)[0x55f2e7d7e39f]
/usr/bin/redis-server 172.16.106.88:6379(+0xe10ac)[0x55f2e7e030ac]
/usr/bin/redis-server 172.16.106.88:6379(aeProcessEvents+0x303)[0x55f2e7d63b83]
/usr/bin/redis-server 172.16.106.88:6379(aeMain+0x1d)[0x55f2e7d63ebd]
/usr/bin/redis-server 172.16.106.88:6379(main+0x4e5)[0x55f2e7d603d5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fb934a370b3]
/usr/bin/redis-server 172.16.106.88:6379(_start+0x2e)[0x55f2e7d606ae]
The service restarts on its own and the Redis server starts working fine for a few days/weeks and crashes again with the same error!
I have checked several posts in SO, but none of them resolve my issue since:
a) The instance where the Redis server is running is in a private network (public access is disabled).
b) The DB file name and dir have not been corrupted as observed from "config get dbfilename" and "config get dir" commands. They show the default values.
c) The permissions of the directories are correct (/var/lib/redis is owned by redis with 755 permissions and /var/lib/redis/dump.rdb is owned by redis with 660 permissions)
Can anyone help me identify the root cause of this issue please?

“500 Internal Server Error” with job artifacts on minio

I'm running gitlab-ce on-prem with min.io as a local S3 service. CI/CD caching is working, and basic connectivity with the S3-compatible minio is good. (Versions: gitlab-ce:13.9.2-ce.0, gitlab-runner:v13.9.0, and minio/minio:latest currently c253244b6fb0.)
Is there additional configuration to differentiate between job-artifacts and pipeline-artifacts and storing them in on-prem S3-compatible object storage?
In my test repo, the "build" stage builds a sparse R package. When I was using local in-gitlab job artifacts, it succeeds and moves on to the "test" and "deploy" stages, no problems. (And that works with S3-stored cache, though that configuration is solely within gitlab-runner.) Now that I've configured minio as a local S3-compatible object storage for artifacts, though, it fails.
...
Created cache
Uploading artifacts for successful job
Uploading artifacts...
/builds/git/mygroup/citest/ci/build/*.tar.gz: found 1 matching files and directories
/builds/git/mygroup/citest/ci/lib: found 67 matching files and directories
WARNING: Uploading artifacts as "archive" to coordinator... failed id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
FATAL: invalid argument
Cleaning up file based variables
ERROR: Job failed: exit code 1
The only fail is in the artifact-storage attempt. (With in-gitlab artifacts, this all succeeded and worked.)
I can find no reference of "invalid argument" within the minio logs (which say nothing during this time). I'm not certain that it is a minio-problem.
The relevant portion of the gitlab config:
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'host' => "minio.mydomain.com",
# 'region' => '',
'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>',
'path_style' => true
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts-storage'
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['lfs']['enabled'] = false
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads-storage'
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false
That configuration is adapted from https://docs.gitlab.com/ee/administration/object_storage.html, deselecting storage of components I don't think I need. I added 'path_style' => true because without it, the default of bucket.minio.mydomain.com was not resolved correctly (so this is mostly-S3-compatible, not perfectly AWS). I have also tried this with 'proxy_download' of true, no change.
I'm logged into the minio console, and the gitlab-artifacts-storage bucket exists. In fact, after these failed "build" tests, I'm seeing newly created job artifacts stored in this bucket (as job.log), so I know that basic connectivity (i.e., access key and secret) works:
In my research, I've seen similar errors linked to nginx rev-proxy issues, which suggests that something in the traefik configuration (or just its presence) might be an issue. It is merely passing traffic, it is doing no path-translation/stripping. Regardless, all of the rev-proxy discussion was on nginx, often attempting path-munging of some sort, and was resolved with some seemingly-unrelated change to the nginx configuration. I haven't found any that map into the traefik domain.
traefik access.log
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44742","ClientHost":"172.19.0.1","ClientPort":"44742","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":3472517,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":3422850,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":49667,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131470,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:53.890788923Z","StartUTC":"2021-03-18T20:37:53.890788923Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:53 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45D8DDC4B3","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:53 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45D8DDC4B3","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203753Z","time":"2021-03-18T20:37:53Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44748","ClientHost":"172.19.0.1","ClientPort":"44748","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1571234,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1525159,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":46075,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131472,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.090024071Z","StartUTC":"2021-03-18T20:37:54.090024071Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E4BD6115","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E4BD6115","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44754","ClientHost":"172.19.0.1","ClientPort":"44754","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1205368,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1145885,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":59483,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131473,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.162829022Z","StartUTC":"2021-03-18T20:37:54.162829022Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E9114D33","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E9114D33","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44758","ClientHost":"172.19.0.1","ClientPort":"44758","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1087332,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1031618,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":55714,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131474,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.206750001Z","StartUTC":"2021-03-18T20:37:54.206750001Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EBAE7A4E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EBAE7A4E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44762","ClientHost":"172.19.0.1","ClientPort":"44762","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1126408,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1068170,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":58238,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131475,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.243629562Z","StartUTC":"2021-03-18T20:37:54.243629562Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EDE0062E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EDE0062E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44766","ClientHost":"172.19.0.1","ClientPort":"44766","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1279861,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1227773,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":52088,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131476,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.277572935Z","StartUTC":"2021-03-18T20:37:54.277572935Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EFE57F90","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EFE57F90","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44774","ClientHost":"172.19.0.1","ClientPort":"44774","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":4317260,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":4261951,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":55309,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131480,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.503920248Z","StartUTC":"2021-03-18T20:37:55.503920248Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4639004631","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4639004631","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203755Z","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44778","ClientHost":"172.19.0.1","ClientPort":"44778","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1879630,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1830988,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":48642,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131482,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.703174362Z","StartUTC":"2021-03-18T20:37:55.703174362Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4644E482E7","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4644E482E7","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44782","ClientHost":"172.19.0.1","ClientPort":"44782","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":2076600,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":2009920,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":66680,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131484,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:56.652353505Z","StartUTC":"2021-03-18T20:37:56.652353505Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:56 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A467D7A9332","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:56 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A467D7A9332","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203756Z","time":"2021-03-18T20:37:56Z"}
I understand there is a difference between job artifacts and pipeline artifacts, so I think what I'm missing is a configuration distinction between "job" and "pipeline".
.gitlab-ci.yml snippet
variables:
GIT_DEPTH: 10
R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
BUILD_DIR: "$CI_PROJECT_DIR/ci/build"
CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/$CI_PROJECT_NAME.Rcheck"
default:
image: rocker/shiny-verse:4.0.3
interruptible: true
build-package:
stage: build
script:
- mkdir -p "$R_LIBS_USER" "$BUILD_DIR"
- R -e '
devtools::install_deps(dependencies = TRUE, lib = Sys.getenv("R_LIBS_USER")) ;
devtools::build(path = Sys.getenv("BUILD_DIR")) ;'
artifacts:
paths:
- $BUILD_DIR/*.tar.gz
- $R_LIBS_USER
cache:
key: "${CI_COMMIT_REF_SLUG}__cilib"
paths:
- $R_LIBS_USER
(Yes, it is both cached and an artifact. This is an instance of me testing the CI subsystem, not something I intend to maintain.)
This is in a docker-swarm behind a traefik reverse-proxy (and SSL terminator).
The answer is to bypass the empty-string test; the underlying protocol does not support region-less configuration, nor is there a configuration option to support it.
The trick is able to work because the use of 'endpoint' causes the 'region' to be ignored. With that, setting the region to something and forcing the endpoint allows it to work:
gitlab_rails['object_store']['connection'] = {
'provider' => 'AWS',
'host' => "minio.mydomain.com",
'region' => 'us-east-1', # this must be non-empty, but is ignored ...
'endpoint' => 'https://minio.mydomain.com', # ... because of 'endpoint'
'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>',
'path_style' => true
}
(I owe discovery of this to Florian, in gitlab-org/gitlab#297227.)

Redis service automatically stops after few minutes of running

On my Ubuntu machine, redis server was running fine and suddenly it stops. After I started it, again it automatically stops after few minutes. So I start again, and so on. Why is this happening?
Here are the logs when I start redis:
21479:C 29 Apr 21:59:10.986 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21479:C 29 Apr 21:59:10.987 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=21479, just started
21479:C 29 Apr 21:59:10.987 # Configuration loaded
21480:M 29 Apr 21:59:10.990 * Increased maximum number of open files to 10032 (it was originally set to 1024).
21480:M 29 Apr 21:59:10.991 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
21480:M 29 Apr 21:59:10.992 # Server initialized
21480:M 29 Apr 21:59:14.588 * DB loaded from disk: 3.596 seconds
21480:M 29 Apr 21:59:14.591 * Ready to accept connections

Failed opening .rdb for saving: Permission denied - started after a while of running successfully

I have had a node web service running successfully on an aws ubuntu server for over a month, with the requests cached using redis.
Yesterday I started getting the following error from some of my routes:
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
I was able to stop the error occurring by using:
config set stop-writes-on-bgsave-error no
as suggested in the answers to this question, but it doesn't actually solve the underlying problem.
To find the underlying problem I checked the logs and found the following had started happening:
[1105] 09 Aug 13:17:14.800 - 0 clients connected (0 slaves), 797680 bytes in use
[1105] 09 Aug 13:17:15.101 * 1 changes in 900 seconds. Saving...
[1105] 09 Aug 13:17:15.101 * Background saving started by pid 28090
[28090] 09 Aug 13:17:15.101 # Failed opening .rdb for saving: Permission denied
[1105] 09 Aug 13:17:15.201 # Background saving error
Over the weekend no one had been using the server, but before the weekend the logs were fine, and we were getting no errors:
[12521] 06 Aug 04:49:27.308 - 0 clients connected (0 slaves), 803352 bytes in use
[12521] 06 Aug 04:49:29.012 * 1 changes in 900 seconds. Saving...
[12521] 06 Aug 04:49:29.012 * Background saving started by pid 26663
[26663] 06 Aug 04:49:29.014 * DB saved on disk
[26663] 06 Aug 04:49:29.014 * RDB: 2 MB of memory used by copy-on-write
[12521] 06 Aug 04:49:29.112 * Background saving terminated with success
As I said, no one has touched this server in the intervening time.
Looking around for people having the same problem I found this question. I checked the ownership and permissions on the directory and db file as suggested in the answers there:
drwxr-xr-x 2 redis redis 26 Aug 6 06:55 redis
-rw-r--r-- 1 redis redis 18 Aug 6 06:55 dump-6379.rdb
The permissions and ownership both look ok to me, but I have noticed that the date on the file and folder is between the last time I saw the service working and the first time it failed. Unfortunately that hasn't really helped me with what to do next and I am at a bit of a loss.
I am looking for suggestions for next steps to find the cause of the problem, or at least a way of making redis able to write again.

CPanel/WHM Unknown License File Error

So my issue is like the title suggests. However I have tried the following suggestions from this page (https://documentation.cpanel.net/display/ALD/Installation+Guide+-+Troubleshoot+Your+Installation#InstallationGuide-TroubleshootYourInstallation-Licenseerrors) with no results.
1.) curl -L http://cpanel.net/showip.cgi (shows my ip address on the server for use on the verify.cpanel.net script), this can be verified also here... (http://verify.cpanel.net/index.cgi?ip=xxx.xxx.xxx.xx) (I don't like showing my IP, but trust me it was verified.)
2.) /usr/local/cpanel/cpkeyclt
Updating cPanel license...Done. Update Failed!
Error message:
A License check appears to already be running.
Building global cache for cpanel...Done
So the above didn't work.
I then tried these commands.
3.) /usr/local/cpanel/etc/init/stopcpsrvd and then /usr/local/cpanel/scripts/upcp --sync to attempt to resynchronize.
This appears to successfully run but I still get the same error. Attached below is the error message I get when I attempt to login to WHM.
4.) I then tried running rdate -s rdate.cpanel.net as suggested in some other posts to have the times match up and then when I run (/usr/local/cpanel/cpkeyclt) it seems to time out and nothing ever happens.
Looking at the logs for the cpanel license (/usr/local/cpanel/logs/license_log) I see this.
Tue Jul 26 16:23:30 2016: Trying server 208.74.125.22
Tue Jul 26 16:23:45 2016: Timed out while connecting to port 2089
Tue Jul 26 16:24:00 2016: Timed out while connecting to port 80
Tue Jul 26 16:24:15 2016: Timed out while connecting to port 110
Tue Jul 26 16:24:30 2016: Timed out while connecting to port 143
Tue Jul 26 16:24:45 2016: Timed out while connecting to port 25
Tue Jul 26 16:25:00 2016: Timed out while connecting to port 23
Tue Jul 26 16:25:15 2016: Timed out while connecting to port 993
Tue Jul 26 16:25:30 2016: Timed out while connecting to port 995
Tue Jul 26 16:30:14 2016: License Update Request
Tue Jul 26 16:30:14 2016: Using full manual DNS resolution
Tue Jul 26 16:30:14 2016: Trying server 208.74.121.85
Tue Jul 26 16:30:29 2016: Timed out while connecting to port 2089
Any help is appreciated!
Notes
Results of running /usr/local/cpanel/etc/init/stopcpsrvd
/usr/local/cpanel/etc/init/stopcpsrvd
Waiting for “cpsrvd” to stop ……Gracefully Terminating processes: cpsrvd: with pids 20842 and owner root.......waited 1 second(s) for 1 process(es) to terminate....Done
…finished.
Startup Log
Starting PID 20839: /usr/local/cpanel/libexec/cpsrvd-dormant
Results of running /usr/local/cpanel/scripts/upcp –sync (Couldn't show everything because of text character limitations)
[2016-07-26 15:39:39 -0400] Detected cron=0 (Terminal detected)
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (21620) at Tue Jul 26 15:41:53 2016
[2016-07-26 15:41:53 -0400] Maintenance completed successfully
[2016-07-26 15:41:54 -0400] 95% complete
[2016-07-26 15:41:54 -0400] Running Standardized hooks
[2016-07-26 15:41:54 -0400] 100% complete
[2016-07-26 15:41:54 -0400]
[2016-07-26 15:41:54 -0400] cPanel update completed
[2016-07-26 15:41:54 -0400] A log of this update is available at /var/cpanel/updatelogs/update.1469561979.log
[2016-07-26 15:41:54 -0400] Removing upcp pidfile
[2016-07-26 15:41:54 -0400]
[2016-07-26 15:41:54 -0400] Completed all updates
=> Log closed Tue Jul 26 15:41:54 2016
It turns out the answer was IPTables. Before that it was the rDate command that was necessary to fix it, but my IPTables was blocking the connections.
To temporarily disable your firewall do this.
iptables-save > /root/current.ipt
iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT
iptables -F INPUT; iptables -F OUTPUT
ping -c 3 google.com
iptables-restore < /root/current.ipt
rm -f /root/current.ipt
The first command saves a copy of your firewall settings.
The next 2 commands make it so all input/output are allowed (for outgoing and incoming connections)
Finally test by pinging the ip address that was giving the issue for cPanel in your log file.
If it works that means the update license command will work.
Simply run:
/usr/local/cpanel/cpkeyclt
and you are good to go.
You can restore back your rules by using the last 2 commands if you want:
iptables-restore < /root/current.ipt
rm -f /root/current.ipt
Be warned that you will be blocked again, unless you fix the firewall.