Add new control plane node got failed k8s 1.21.0 [closed] - ssl

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I want to add a new control plane node into the cluster.
So, I run in an existing control plane server:
kubeadm token create --print-join-command
I run this command in new control plane node:
kubeadm join 10.0.0.151:8443 --token m3g8pf.gdop9wz08yhd7a8a --discovery-token-ca-cert-hash sha256:634db22bc69b47b8f2b9f733d2f5e95cf8e56b349e68ac611a56d9da0cf481b8 --control-plane --apiserver-advertise-address 10.0.0.10 --apiserver-bind-port 6443 --certificate-key 33cf0a1d30da4c714755b4de4f659d6d5a02e7a0bd522af2ebc2741487e53166
I got this message:
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
error execution phase control-plane-prepare/download-certs: error downloading certs: the Secret does not include the required certificate or key - name: external-e
tcd.crt, path: /etc/kubernetes/pki/apiserver-etcd-client.crt
I run in an existing production control plane node:
kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
0a3f5486c3b9303a4ace70ad0a9870c2605d67eebcd500d68a5e776bbd628a3b
Re-run this command in the new control plane node:
kubeadm join 10.0.0.151:8443 --token m3g8pf.gdop9wz08yhd7a8a --discovery-token-ca-cert-hash sha256:634db22bc69b47b8f2b9f733d2f5e95cf8e56b349e68ac611a56d9da0cf481b8 --control-plane --apiserver-advertise-address 10.0.0.10 --apiserver-bind-port 6443 --certificate-key 0a3f5486c3b9303a4ace70ad0a9870c2605d67eebcd500d68a5e776bbd628a3b
I got the same message:
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
error execution phase control-plane-prepare/download-certs: error downloading certs: the Secret does not include the required certificate or key - name: external-etcd.crt, path: /etc/kubernetes/pki/apiserver-etcd-client.crt
To see the stack trace of this error execute with --v=5 or higher
What's I am wrong?
I have all certs in the new node installed before doing this op:
# ls /etc/kubernetes/pki/
apiserver.crt apiserver.key ca.crt front-proxy-ca.crt front-proxy-client.key
apiserver-etcd-client.crt apiserver-kubelet-client.crt ca.key front-proxy-ca.key sa.key
apiserver-etcd-client.key apiserver-kubelet-client.key etcd front-proxy-client.crt sa.pub
I didn't see how to specify etcd certs files:
Usage:
kubeadm init phase upload-certs [flags]
Flags:
--certificate-key string Key used to encrypt the control-plane certificates in the kubeadm-certs Secret.
--config string Path to a kubeadm configuration file.
-h, --help help for upload-certs
--kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf")
--skip-certificate-key-print Don't print the key used to encrypt the control-plane certificates.
--upload-certs Upload control-plane certificates to the kubeadm-certs Secret.
Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level)
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity

You also need to pass the --config flag to your kubeadm init phase command (use sudo if needed). So instead of:
kubeadm init phase upload-certs --upload-certs
you should for example run:
kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml
This topic is also explained by Uploading control-plane certificates to the cluster docs.

Related

EJBCA: Authorization Denied Admin GUI

I am attempting to upgrade EJBCA.
I attempted to run this on ubuntu 20.04, locally, using wildfly 18. Wildfly 18 results in this error: "CAUSE: Client certificate or OAuth bearer token required."
I have tried this two ways, by importing the keystore, truststore and superadmin from another instance and by creating the CA fresh and using the resulting superadmin.p12.
The home page loads, but the administration gives me the following error:
"AUTHORIZATIONDENIED
CAUSE: Client certificate or OAuth bearer token required. "
I can really use some help with this.
Things I have tried:
(1) I have downloaded superadmin.p12 and imported it into my browsers
(2) I have attempted to upload the superdmin cert:
bin/ejbca.sh ca importcacert ${NAME} ${NAME}.cacert.pem -initauthorization -superadmincn SuperAdmin
This results in The CA certificate is already imported.
(3) Both my keystore.jks and truststore.jks are moved into /ejbca/p12 and /opt/wildfly/standalone/configuration/keystore
(4) I did set "web.reqcertindb=false"
(6) I did try to enable ssl on wildfly 14 (https://docs.bitnami.com/bch/infrastructure/wildfly/administration/enable-ssl-wildfly/)
(7) I have tried a fresh Management_CA as well
The log of /ejbca/adminweb:
"08:20:01,270 ERROR [org.ejbca.ui.web.admin.configuration.EjbcaJSFHelperImpl] (default task-4) org.cesecore.authentication.AuthenticationFailedException: Client certificate or OAuth bearer token required.
08:20:01,279 WARN [org.ejbca.ui.web.admin.configuration.EjbcaWebBeanImpl] (default task-4) Language was not initialized for this session
08:20:01,279 WARN [org.ejbca.ui.web.admin.configuration.EjbcaWebBeanImpl]
I can provide more information if needs be.
Thank you
So, I have it running today. Here is what I learned:
It seems that if you set wildfly up as a service (per instructions) it is going to set up wildfly to run with launch.sh. Launch.sh is going to result in a cipher mistmatch. I needed to run the standalone.sh file instead
Adminweb must be contacted on 8443
if you need to run this thing on domain setup your going to need to post another question
Best,

KAFKA and SSL : java.lang.OutOfMemoryError: Java heap space when using kafka-topics command on KAFKA SSL cluster

this is my first post on Stackoverflow, i hope i didnt choose the wrong section.
Context :
Kafka HEAP size is configured on following file :
/etc/systemd/system/kafka.service
With following parameter :
Environment="KAFKA_HEAP_OPTS=-Xms6g -Xmx6g"
OS is "CentOS Linux release 7.7.1908".
Kafka is "confluent-kafka-2.12-5.3.1-1.noarch", installed from the following repository :
# Confluent REPO
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/5.3/7
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1
[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/5.3
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/5.3/archive.key
enabled=1
I activated SSL on a 3-machine KAFKA cluster few days ago, and suddently, the following command stopped working :
kafka-topics --bootstrap-server <the.fqdn.of.server>:9093 --describe --topic <TOPIC-NAME>
Which return me the following error :
[2019-10-03 11:38:52,790] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':(org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1152)
at java.lang.Thread.run(Thread.java:748)
On the server's log, the following line appears when i try to request it via "kafka-topics" :
/var/log/kafka/server.log :
[2019-10-03 11:41:11,913] INFO [SocketServer brokerId=<ID>] Failed authentication with /<ip.of.the.server> (SSL handshake failed) (org.apache.kafka.common.network.Selector)
I was able to use this command properly BEFORE implementing SSL on the cluster. Here is the configuration i'm using.
All functionnality work properly (consumers, producers...) except "kafka-topics" :
# SSL Configuration
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>
ssl.keystore.type=<keystore-type>
ssl.keystore.location=<keystore-path>
ssl.keystore.password=<keystore-password>
# Enable SSL between brokers
security.inter.broker.protocol=SSL
# Listeners
listeners=SSL://<fqdn.of.the.server>:9093
advertised.listeners=SSL://<fqdn.of.the.server>:9093
There is no problem with the certificate (which is signed by internal CA, internal CA which i added to the truststore specified on the configuration). OpenSSL show no errors :
openssl s_client -connect <fqdn.of.the.server>:9093 -tls1
>> Verify return code: 0 (ok)
The following command is working pretty well with SSL, thanks to parameter "-consumer.config client-ssl.properties"
kafka-console-consumer --bootstrap-server <fqdn.of.the.server>:9093 --topic <TOPIC-NAME> -consumer.config client-ssl.properties
"client-ssl.properties" content is :
security.protocol=SSL
ssl.truststore.location=<truststore-path>
ssl.truststore.password=<truststore-password>
Right now, i'm forced to use "--zookeeper", which according to the documentation, is deprecated :
--zookeeper <String: hosts> DEPRECATED, The connection string for
the zookeeper connection in the form
host:port. Multiple hosts can be
given to allow fail-over.
And of course, it's working pretty well :
kafka-topics --zookeeper <fqdn.of.the.server>:2181 --describe --topic <TOPIC-NAME>
Topic:<TOPIC-NAME> PartitionCount:3 ReplicationFactor:2
Configs:
Topic: <TOPIC-NAME> Partition: 0 Leader: <ID-3> Replicas: <ID-3>,<ID-1> Tsr: <ID-1>,<ID-3>
Topic: <TOPIC-NAME> Partition: 1 Leader: <ID-1> Replicas: <ID-1>,<ID-2> Isr: <ID-2>,<ID-1>
Topic: <TOPIC-NAME> Partition: 2 Leader: <ID-2> Replicas: <ID-2>,<ID-3> Isr: <ID-2>,<ID-3>
So, my question is : why am i unable to use "--bootstrap-server" atm ? Because of the "zookeeper" deprecation, i'm worried about not to be able to consult my topics, and their details...
I believe that kafka-topics needs the same option than kafka-console-consumer, aka "-consumer.config"...
Ask if any additionnal precision needed.
Thanks a lot, hope my question is clear and readable.
Blyyyn
I finally found a way to deal with this SSL error. The key is to use the following setting :
--command-config client-ssl.properties
This is working with the most part of KAFKA commands, like kafka-consumer-groups, and of course kafka-topics. See examples below :
kafka-consumer-groups --bootstrap-server <kafka-hostname>:<kafka-port> --group <consumer-group> --topic <topic> --reset-offsets --to-offset <offset> --execute --command-config <ssl-config>
kafka-topics --list --bootstrap-server <kafka-hostname>:<kafka-port> --command-config client-ssl.properties
ssl-config was "client-ssl.properties",see initial post for content.
Beware, by using IP address on , you'll have an error if the machine certificate doesnt have alternative name with that IP address. Try to have correct DNS resolution and use FQDN if possible.
Hope this solution will help, cheers!
Blyyyn
Stop your Brokers and run below ( assuming you have more that 1.5GB RAM on your server)
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
then start your Brokers on all 3 nodes and then try it.
Note that for consumer and producer clients you need to prefix security.protocol accordingly inside your client-ssl.properties.
For Kafka Consumers:
consumer.security.protocol=SASL_SSL
For Kafka Producers:
producer.security.protocol=SASL_SSL

Problem getting complete .pem from ansible letsencrypt / acme_certificate module

I was using Ansible 2.4 and included the letsencrypt module in one of my roles hoping to get a complete `.pem' format file at the end (key, chain, cert). There was no problem generating the key or using the csr to request the new cert, and no problem with the challenge, but when everything was done, I was only getting the certificate back, no chain.
When I tried to use them, Apache would fail to start saying that the key and the cert did not match. I assumed that this was because I didn't include the chain which was missing.
According to the docs here: https://docs.ansible.com/ansible/latest/modules/acme_certificate_module.html the chain|chain_dest and fullchain|fullchain_dest parameters weren't added until Ansible 2.5. So I upgraded to Ansible 2.7 (via git), and I'm still running into the exact same error...
FAILED! => {
"changed": false,
"msg": "
Unsupported parameters for (letsencrypt) module: chain_dest, fullchain_dest
Supported parameters include: account_email, account_key, acme_directory, agreement,
challenge, csr, data, dest, remaining_days"
}
I've tried the aliases and current names for both but nothing is working. Here is my current challenge-response call:
- name: Let the challenge be validated and retrieve the cert and intermediate certificate
letsencrypt:
account_key: /etc/ssl/lets_encrypt.key
account_email: ###########.###
csr: /etc/ssl/{{ myhost.public_hostname }}.csr
dest: /etc/ssl/{{ myhost.public_hostname }}.crt
chain_dest: /etc/ssl/{{ myhost.public_hostname }}.int
fullchain_dest: /etc/ssl/{{ myhost.public_hostname }}.pem
challenge: dns-01
acme_directory: https://acme-v01.api.letsencrypt.org/directory
remaining_days: 60
data: "{{ le_com_challenge }}"
tags: sslcert
The documentation says that this is valid, but the error response does not include chain|chain_dest or fullchain|fullchain_dest as valid parameters.
I would, from the docs, expect that this response should result in the new certificate being created (.crt), the chain being created (.int), and the fullchain to be created (.pem).
Any help would be appreciated.
Should have waited 5 minutes... seems that the newer parameters are only available under the newer module name acme_certificate, even though it says letsencrypt was a valid alias. As soon as I updated this it worked.

SSL certificate pinning with libcurl

I'd like to know if this example is enough to provide certificate pinning with libcurl:
http://curl.haxx.se/libcurl/c/cacertinmem.html
because I have found that curl also allows http://curl.haxx.se/libcurl/c/CURLOPT_PINNEDPUBLICKEY.html
Since I'll be using a self-signed certificate and only trust on it I don't know if it's truly necessary to pinn it too.
resume: Can the connection be compromised if I only add my certificate (self-signed) to the x509 certificate store like the example? do I need to add extra checks? do I need to use the CURLOPT_PINNEDPUBLICKEY option?
Thanks.
You can find another example in the implementation of the new curl option in git 2.8 (March 2016):
See commit aeff8a6 (15 Feb 2016) by Christoph Egger (siccegge).
(Merged by Junio C Hamano -- gitster -- in commit e79112d, 24 Feb 2016)
http: implement public key pinning
Add the http.pinnedpubkey configuration option for public key pinning. It allows any string supported by libcurl -- base64(sha256(pubkey)) or filename of the full public key.
If cURL does not support pinning (is too old) output a warning to the user.
The git config man page mentions:
http.pinnedpubkey:
Public key of the https service.
It may either be the filename of a PEM or DER encoded public key file or a string starting with 'sha256//' followed by the base64 encoded sha256 hash of the public key.
See also libcurl 'CURLOPT_PINNEDPUBLICKEY'.
git will exit with an error if this option is set but not supported by cURL.
With Git 2.34 (Q4 2021), HTTPS error handling is updated when it comes to SSL certificate pinning:
See commit 3e8084f (24 Sep 2021) by Ævar Arnfjörð Bjarmason (avar).
(Merged by Junio C Hamano -- gitster -- in commit 97492aa, 11 Oct 2021)
http: check CURLE_SSL_PINNEDPUBKEYNOTMATCH when emitting errors
Signed-off-by: Ævar Arnfjörð Bjarmason
Change the error shown when a http.pinnedPubKey doesn't match to point the http.pinnedPubKey variable added in aeff8a6 ("http: implement public key pinning", 2016-02-15, Git v2.8.0-rc0 -- merge listed in batch #8), e.g.:
git -c http.pinnedPubKey=sha256/someNonMatchingKey ls-remote https://github.com/git/git.git
fatal: unable to access 'https://github.com/git/git.git/' with http.pinnedPubkey configuration: SSL: public key does not match pinned public key!
Before this we'd emit the exact same thing without the " with http.pinnedPubkey configuration".
The advantage of doing this is that we're going to get a translated message (everything after the ":" is hardcoded in English in libcurl), and we've got a reference to the git-specific configuration variable that is causing the error.
Unfortunately we can't test this easily, as there are no tests that require https:// in the test suite, and t/lib-httpd.sh doesn't know how to set up such tests.
See this thread for the start of a discussion about what it would take to have divergent "t/lib-httpd/apache.conf" test setups.

Puppet agent fail to connect to master [closed]

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 7 years ago.
Improve this question
I'm struggling since 2 days with signing of Puppet-Agents now. The Problem is as follows:
On the master I delete all existing signatures with
puppet cert clean esx-poc-1.xxx.de
On the agent I delete the whole ssl directory with
rm -rf /var/lib/puppet/ssl/
After running one of the following commands on the agent...
puppet certificate generate esx-poc-1.xxx.de --ca-location remote
puppet agent --server puppetmaster.int.xxx.com --waitforcert 60 --test
...I can list the certificates on the master with:
puppet cert list --all
The output is:
"esx-poc-1.xxx.de" (SHA256)
71:72:D8:3E:09:9E:B1:5C:DA:78:A8:B8:A1:2B:E4:09:B8:00:8A:AF:49:02:CC:B2:B5:C3:25:79:59:0D:A8:F5
+ "puppetmaster.int.xxx.com" (SHA256) 7B:00:8C:4F:CE:B2:0D:2F:A1:BB:A7:C4:25:B0:11:01:2B:EC:90:46:D1:CB:BE:AA:AD:3F:B4:70:0C:83:3F:78
(alt names: "DNS:puppet", "DNS:puppet.xxx.de",
"DNS:puppetmaster.int.xxx.com")
After signing the agent with:
puppet cert sign esx-poc-1.xxx.de
The fingerprint differs from the one above:
"esx-poc-1.xxx.de" (SHA256) 49:F6:59:FD:3C:28:C6:54:7F:6E:A7:56:56:DB:64:9A:E2:08:10:90:11:83:7A:A6:0E:E1:CD:39:F0:E0:1C:25
Is that correct?
Performing an agent-run aferwards results in the following error:
Warning: Unable to fetch my node definition, but the agent run will continue: Warning: Error 400 on SERVER: Could not retrieve facts for esx-poc-1.xxx.de: Failed to submit 'replace facts' command for esx-poc-1.xxx.de to PuppetDB at puppetmaster.int.xxx.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=puppetmaster.int.xxx.com] Info: Retrieving plugin Info: Loading facts in /var/lib/puppet/lib/facter/last_run.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rb Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Info: Loading facts in /var/lib/puppet/lib/facter/iptables.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for esx-poc-1.xxx.de to PuppetDB at puppetmaster.int.xxx.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=puppetmaster.int.xxx.com] Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
Thx for any help!
Ran into this exact same issue myself. The problem ended up being that the puppetdb-terminus package was at version 1.1.0 while puppetdb itself was still at 1.0.5.
After downgrading puppetdb-terminus down to 1.0.5 everything operated normally.
In puppet 3.4 I noticed that if the hostnames are not set this error could be thrown.
For example; I had two debian boxes; one was named debian1 and the other debian2 in the hosts file. But, both of their /etc/hostname settings where debian; after I changed their name with hostname and set their names in /etc/hostname they worked just fine.
This might be a dumb question, but, do you have a node definition for this machine? I.e.,
node 'esx-poc-1.xxx.de' {
.....
}
I had this error after changing permission of files in /etc/puppet.
Changing them back to 'pe-puppet' (for an enterprise version) solved it for me
After completely reinstalling puppetdb it's finally working...