RabbitMQ in Kubernetes - Create User as part of Statefulset deployment kind - rabbitmq

I am new to the Kubernetes and learning by experimenting. I have created RabbitMQ statefulset and it's working. However, the issue I am facing is the way I use it's admin portal.
By default RabbitMQ provides the guest/guest credential but that works only with localhsot. It gives me a thought that I supposed to have another user for admin as well as for my connection string at API side to access RabbitMQ. (currently in API side also I use guest:guest#.... as bad practice)
I like to change but I don't know how. I can manually login to the RabbitMQ admin portal (after deployment and using guest:guest credential) can create new user. But I thought of automating that as part of Kubernetes Statefulset deployment.
I have tried to add post lifecycle hook of kubernetes but that did not work well. I have following items:
rabbitmq-configmap:
rabbitmq.conf: |
## Clustering
#cluster_formation.peer_discovery_backend = k8s
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_partition_handling = autoheal
#cluster_formation.k8s.hostname_suffix = rabbitmq.${NAMESPACE}.svc.cluster.local
#cluster_formation.node_cleanup.interval = 10
#cluster_formation.node_cleanup.only_log_warning = true
rabbitmq-serviceaccount:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rabbitmq
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs:
- get
- list
- watch
rabbitmq-statefulset:
initContainers:
- name: "rabbitmq-config"
image: busybox
volumeMounts:
- name: rabbitmq-config
mountPath: /tmp/rabbitmq
- name: rabbitmq-config-rw
mountPath: /etc/rabbitmq
command:
- sh
- -c
# the newline is needed since the Docker image entrypoint scripts appends to the config file
- cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf;
cp /tmp/rabbitmq/enabled_plugins /etc/rabbitmq/enabled_plugins;
containers:
- name: rabbitmq
image: rabbitmq
ports:
- containerPort: 15672
Any help?

There are multiple way to do it
You can use the RabbitMQ CLI to add the user into it.
Add the environment variables and change the username/password instead of guest .
image: rabbitmq:management-alpine
environment:
RABBITMQ_DEFAULT_USER: user
RABBITMQ_DEFAULT_PASS: password
Passing argument to image
https://www.rabbitmq.com/cli.html#passing-arguments
Mounting the configuration file to RabbitMQ volume.
Rabbitmq.conf file
auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = AMQPLAIN
loopback_users.guest = false
listeners.tcp.default = 5672
#default_pass = admin
#default_user = admin
hipe_compile = false
#management.listener.port = 15672
#management.listener.ssl = false
management.tcp.port = 15672
management.load_definitions = /etc/rabbitmq/definitions.json
#default_pass = admin
#default_user = admin
definitions.json
{
"users": [
{
"name": "user",
"password_hash": "password",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}
],
"vhosts":[
{"name":"/"}
],
"queues":[
{"name":"qwer","vhost":"/","durable":true,"auto_delete":false,"arguments":{}}
]
}
Another option
Dockerfile
FROM rabbitmq
# Define environment variables.
ENV RABBITMQ_USER user
ENV RABBITMQ_PASSWORD password
ADD init.sh /init.sh
EXPOSE 15672
# Define default command
CMD ["/init.sh"]
init.sh
#!/bin/sh
# Create Rabbitmq user
( sleep 5 ; \
rabbitmqctl add_user $RABBITMQ_USER $RABBITMQ_PASSWORD 2>/dev/null ; \
rabbitmqctl set_user_tags $RABBITMQ_USER administrator ; \
rabbitmqctl set_permissions -p / $RABBITMQ_USER ".*" ".*" ".*" ; \
echo "*** User '$RABBITMQ_USER' with password '$RABBITMQ_PASSWORD' completed. ***" ; \
echo "*** Log in the WebUI at port 15672 (example: http:/localhost:15672) ***") &
# $# is used to pass arguments to the rabbitmq-server command.
# For example if you use it like this: docker run -d rabbitmq arg1 arg2,
# it will be as you run in the container rabbitmq-server arg1 arg2
rabbitmq-server $#
You can read more here

Related

How to copy a file and add dynamic content on startup of an EC2 instance using CloudFormation?

I need to copy the content of a cert file which comes from the secretsmanager into an EC2 instance on startup using CloudFormation.
Edit:
I added an IAM Role, a Policy, and an InstanceProfile in my code to ensure that I can access the SecretsManager value using UserData
The code looks like this now:
SecretsManagerAccessRole:
Type: AWS::IAM::Role
Properties:
RoleName: CloudFormationSecretsManagerAccessRole
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: sts:AssumeRole
Path: "/"
SecretsManagerInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles: [ !Ref SecretsManagerAccessRole ]
SecretsManagerInstancePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: SecretsManagerAccessPolicy,
PolicyDocument:
Statement:
- Effect: Allow
Action: secretsmanager:GetSecretValue
Resource: <arn-of-the-secret>
Roles: [ !Ref SecretsManagerAccessRole ]
LinuxEC2Instance:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile: !Ref SecretsManagerInstanceProfile
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum update -y
groupadd -g 110 ansible
adduser ansible -g ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
aws secretsmanager get-secret-value \
--secret-id <arn-of-the-secret> \
--region ${AWS::Region} \
--query 'SecretString' \
--output text > /home/ansible/.ssh/authorized_keys
chmod 000644 /home/ansible/.ssh/authorized_keys
chown -R ansible.ansible /home/ansible/.ssh/
cat /home/ansible/.ssh/authorized_keys
During startup of the instance, I get this issue here:
Unable to locate credentials. You can configure credentials by running "aws configure".
It seems like the user did not get the necessary role to perform this action in UserData? Why is that?
I tried few things, but all failed. The only thing that worked was to use UserData.
For example, you could have the following:
LinuxEC2Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-08f3d892de259504d # AL2 in us-east-1
InstanceType: t2.micro
IamInstanceProfile: <name-of-instance-profile>
KeyName: MyKeyPair
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum update -y
groupadd -g 110 ansible
adduser ansible -g ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
secret_value=$(aws secretsmanager get-secret-value \
--secret-id <arn-of-the-secret> \
--region ${AWS::Region} \
--query 'SecretString' \
--output text)
# have to check the exact command here of jq
echo ${!secret_value} | jq -r '.key' > /home/ansible/.ssh/authorized_keys
chmod 000644 /home/ansible/.ssh/authorized_keys
chown -R ansible.ansible /home/ansible/.ssh/
You also would need to add an instance role/profile to the instance
so that it can read the secret. The role could contain the following
policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadSecretValue",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "<arn-of-secret>"
}
]
}
edit:
In you KMS is used for encryption of the secret, the instance role would need to have permissions for KMS as well.
Ok, I got it working, this is the full answer, the code below worked for me, in addition, I needed to add 'kms:GenerateDataKey', 'kms:Decrypt' to the permissions for it to properly retrieve the secret, finally I needed to use jq to retrieve the value out of the JSON format I got from secrets manager:
CFNInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref CFNAccessRole
CFNAccessRole:
Type: AWS::IAM::Role
Properties:
RoleName: CFNAccessRole
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
Path: /
CFNInstancePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: SecretsManagerAccessPolicy,
PolicyDocument:
Statement:
- Effect: Allow
Action: ['secretsmanager:GetSecretValue', 'kms:GenerateDataKey', 'kms:Decrypt']
Resource: '*'
Roles:
- !Ref CFNAccessRole
# EC2 Instance creation
LinuxEC2Instance:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile: !Ref CFNInstanceProfile
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum update -y
groupadd -g 110 ansible
adduser ansible -g ansible
mkdir -p /home/ansible/.ssh
chmod 700 /home/ansible/.ssh
aws secretsmanager get-secret-value \
--secret-id <arn-of-the-secret> \
--region ${AWS::Region} \
--query 'SecretString' \
--output text | jq -r ".key" > /home/ansible/.ssh/authorized_keys
chmod 000644 /home/ansible/.ssh/authorized_keys
chown -R ansible.ansible /home/ansible/.ssh/
cat /home/ansible/.ssh/authorized_keys

Assign variable within kubernetes yaml job

I would like to run a command within the yaml file for kubernetes:
Here is the part of the yaml file that i use
The idea is to calculate a precent value based on mapped and unmapped values. mapped and unmapped are set properly but the percent line fails
I think the problem comes from the single quotes in the BEGIN statement of the awk command which i guess need to escape ???
If mapped=8 and unmapped=7992
Then percent is (8/(8+7992)*100) = 0.1%
command: ["/bin/sh","-c"]
args: ['
...
echo "Executing command" &&
map=${grep -c "^#" outfile.mapped.fq} &&
unmap=${grep -c "^#" outfile.unmapped.fq} &&
percent=$(awk -v CONVFMT="%.10g" -v map="$map" -v unmap="$unmap" "BEGIN { print ((map/(unmap+map))*100)}") &&
echo "finished"
']
Thanks to the community comments: Ed Morton & david
For those files with data, please create configmap:
outfile.mapped.fq
outfile.unmapped.fq
kubectl create configmap config-volume --from-file=/path_to_directory_with_files/
Create pod:
apiVersion: v1
kind: Pod
metadata:
name: awk-ubu
spec:
containers:
- name: awk-ubuntu
image: ubuntu
workingDir: /test
command: [ "/bin/sh", "-c" ]
args:
- echo Executing_command;
map=$(grep -c "^#" outfile.mapped.fq);
unmap=$(grep -c "^#" outfile.unmapped.fq);
percent=$(awk -v CONVFMT="%.10g" -v map="$map" -v unmap="$unmap" "BEGIN { print ((map/(unmap+map))*100)}");
echo $percent;
echo Finished;
volumeMounts:
- name: special-config
mountPath: /test
volumes:
- name: special-config
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: config-volume
restartPolicy: Never
Once completed verify the result:
kubectl logs awk-ubu
Executing_command
53.3333
Finished

Terraform remote-exec on windows with ssh

I have setup a Windows server and installed ssh using Chocolatey. If I run this manually I have no problems connecting and running my commands. When I try to use Terraform to run my commands it connects successfully but doesn't run any commands.
I started by using winrm and then I could run commands but due to some problem with creating a service fabric cluster over winrm I decided to try using ssh instead and when running things manually it worked and the cluster went up. So that seems to be the way forward.
I have setup a Linux VM and got ssh working by using the private key. So I have tried to use the same config as I did with the Linux VM on the Windows but it still asked me to use my password.
What could the reason be for being able to run commands over ssh manually and using Terraform only connect but no commands are run? I am running this on OpenStack with Windows 2016
null_resource.sf_cluster_install (remote-exec): Connecting to remote host via SSH...
null_resource.sf_cluster_install (remote-exec): Host: 1.1.1.1
null_resource.sf_cluster_install (remote-exec): User: Administrator
null_resource.sf_cluster_install (remote-exec): Password: true
null_resource.sf_cluster_install (remote-exec): Private key: false
null_resource.sf_cluster_install (remote-exec): SSH Agent: false
null_resource.sf_cluster_install (remote-exec): Checking Host Key: false
null_resource.sf_cluster_install (remote-exec): Connected!
null_resource.sf_cluster_install: Creation complete after 4s (ID: 5017581117349235118)
Here is the script im using to run the commands:
resource "null_resource" "sf_cluster_install" {
# count = "${local.sf_count}"
depends_on = ["null_resource.copy_sf_package"]
# Changes to any instance of the cluster requires re-provisioning
triggers = {
cluster_instance_ids = "${openstack_compute_instance_v2.sf_servers.0.id}"
}
connection = {
type = "ssh"
host = "${openstack_networking_floatingip_v2.sf_floatIP.0.address}"
user = "Administrator"
# private_key = "${file("~/.ssh/id_rsa")}"
password = "${var.admin_pass}"
}
provisioner "remote-exec" {
inline = [
"echo hello",
"powershell.exe Write-Host hello",
"powershell.exe New-Item C:/tmp/hello.txt -type file"
]
}
}
Put the connection block inside the provisioner block:
provisioner "remote-exec" {
connection = {
type = "ssh"
...
}
inline = [
"echo hello",
"powershell.exe Write-Host hello",
"powershell.exe New-Item C:/tmp/hello.txt -type file"
]
}

kafka connect transforms RegExRouter exiting with unrecoverable exception

I have made a kafka pipeline to copy a sqlserver table to s3
During sink, i'm trying to transform topic names dropping prefix with the regexrouter function :
"transforms":"dropPrefix",
"transforms.dropPrefix.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex":"SQLSERVER-TEST-(.*)",
"transforms.dropPrefix.replacement":"$1"
The sink fails with the message :
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:586)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:322)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:188)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:564)
... 10 more
If i remove the transform, the pipeline works fine
Problem can be reproduced with this docker-compose :
version: '2'
services:
smtproblem-zookeeper:
image: zookeeper
container_name: smtproblem-zookeeper
ports:
- "2181:2181"
smtproblem-kafka:
image: confluentinc/cp-kafka:5.0.0
container_name: smtproblem-kafka
ports:
- "9092:9092"
links:
- smtproblem-zookeeper
- smtproblem-minio
environment:
KAFKA_ADVERTISED_HOST_NAME : localhost
KAFKA_ZOOKEEPER_CONNECT: smtproblem-zookeeper:2181/kafka
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://smtproblem-kafka:9092
KAFKA_CREATE_TOPICS: "_schemas:3:1:compact"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
smtproblem-schema_registry:
image: confluentinc/cp-schema-registry:5.0.0
container_name: smtproblem-schema-registry
ports:
- "8081:8081"
links:
- smtproblem-kafka
- smtproblem-zookeeper
environment:
SCHEMA_REGISTRY_HOST_NAME: http://smtproblem-schema_registry:8081
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://smtproblem-kafka:9092
SCHEMA_REGISTRY_GROUP_ID: schema_group
smtproblem-kafka-connect:
image: confluentinc/cp-kafka-connect:5.0.0
container_name: smtproblem-kafka-connect
command: bash -c "wget -P /usr/share/java/kafka-connect-jdbc http://central.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/6.4.0.jre8/mssql-jdbc-6.4.0.jre8.jar && /etc/confluent/docker/run"
ports:
- "8083:8083"
links:
- smtproblem-zookeeper
- smtproblem-kafka
- smtproblem-schema_registry
- smtproblem-minio
environment:
CONNECT_BOOTSTRAP_SERVERS: smtproblem-kafka:9092
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: "connect_group"
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 1000
CONNECT_CONFIG_STORAGE_TOPIC: "connect_config"
CONNECT_OFFSET_STORAGE_TOPIC: "connect_offsets"
CONNECT_STATUS_STORAGE_TOPIC: "connect_status"
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: "io.confluent.connect.avro.AvroConverter"
CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: "http://smtproblem-schema_registry:8081"
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://smtproblem-schema_registry:8081"
CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_REST_ADVERTISED_HOST_NAME: "smtproblem-kafka_connect"
CONNECT_LOG4J_ROOT_LOGLEVEL: INFO
CONNECT_LOG4J_LOGGERS: org.reflections=ERROR
CONNECT_PLUGIN_PATH: "/usr/share/java"
AWS_ACCESS_KEY_ID: localKey
AWS_SECRET_ACCESS_KEY: localSecret
smtproblem-minio:
image: minio/minio:edge
container_name: smtproblem-minio
ports:
- "9000:9000"
entrypoint: sh
command: -c 'mkdir -p /data/datalake && minio server /data'
environment:
MINIO_ACCESS_KEY: localKey
MINIO_SECRET_KEY: localSecret
volumes:
- "./minioData:/data"
smtproblem-sqlserver:
image: microsoft/mssql-server-linux:2017-GA
container_name: smtproblem-sqlserver
environment:
ACCEPT_EULA: "Y"
SA_PASSWORD: "Azertyu&"
ports:
- "1433:1433"
Create a database in sqlserver container :
$ sudo docker exec -it smtproblem-sqlserver bash
# /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P 'Azertyu&'
Create a test database :
create database TEST
GO
use TEST
GO
CREATE TABLE TABLE_TEST (id INT, name NVARCHAR(50), quantity INT, cbMarq INT NOT NULL IDENTITY(1,1), cbModification smalldatetime DEFAULT (getdate()))
GO
INSERT INTO TABLE_TEST VALUES (1, 'banana', 150, 1); INSERT INTO TABLE_TEST VALUES (2, 'orange', 154, 2);
GO
exit
exit
Create a source connector :
curl -X PUT http://localhost:8083/connectors/sqlserver-TEST-source-bulk/config -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.password": "Azertyu&",
"validate.non.null": "false",
"tasks.max": "3",
"table.whitelist": "TABLE_TEST",
"mode": "bulk",
"topic.prefix": "SQLSERVER-TEST-",
"connection.user": "SA",
"connection.url": "jdbc:sqlserver://smtproblem-sqlserver:1433;database=TEST"
}'
Create the sink connector :
curl -X PUT http://localhost:8083/connectors/sqlserver-TEST-sink/config -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{
"topics": "SQLSERVER-TEST-TABLE_TEST",
"topics.dir": "TABLE_TEST",
"s3.part.size": 5242880,
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"tasks.max": 3,
"schema.compatibility": "NONE",
"s3.region": "us-east-1",
"schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
"format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
"s3.bucket.name": "datalake",
"store.url": "http://smtproblem-minio:9000",
"flush.size": 1,
"transforms":"dropPrefix",
"transforms.dropPrefix.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex":"SQLSERVER-TEST-(.*)",
"transforms.dropPrefix.replacement":"$1"
}'
Error can be shown in Kafka connect UI, or with curl status command :
curl -X GET http://localhost:8083/connectors/sqlserver-TEST-sink/status
Thanks for your help
So, if we debug, we can see what it is trying to do...
There is a HashMap with the original topic name (SQLSERVER_TEST_TABLE_TEST-0), and the transform has already been applied (TABLE-TEST-0), so if we lookup the "new" topicname, it cannot find the S3 writer for the TopicPartition.
Therefore, the map returns null, and the subsequent .buffer(record) throws an NPE.
I had a similar use case for this before -- writing more than one topic into a single S3 path, and I ended up having to write a custom partitioner, e.g. class MyPartitioner extends DefaultPartitioner.
If you build a JAR using some custom code like that, put it under usr/share/java/kafka-connect-storage-common, then edit the connector config for partitioner.class, it should work as expected.
I'm not really sure if this is a "bug", per say, because back up the call stack, there is no way to get a reference to the regex transform at the time the topicPartitionWriters are declared with the source topic name(s).
If anything, the storage connector configurations should allow a separate regex transform that can edit the encodedPartition (the path where it writes the files)

How do I add my own public key to Vagrant VM?

I got a problem with adding an ssh key to a Vagrant VM. Basically the setup that I have here works fine. Once the VMs are created, I can access them via vagrant ssh, the user "vagrant" exists and there's an ssh key for this user in the authorized_keys file.
What I'd like to do now is: to be able to connect to those VMs via ssh or use scp. So I would only need to add my public key from id_rsa.pub to the authorized_keys - just like I'd do with ssh-copy-id.
Is there a way to tell Vagrant during the setup that my public key should be included? If not (which is likely, according to my google results), is there a way to easily append my public key during the vagrant setup?
You can use Ruby's core File module, like so:
config.vm.provision "shell" do |s|
ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
s.inline = <<-SHELL
echo #{ssh_pub_key} >> /home/vagrant/.ssh/authorized_keys
echo #{ssh_pub_key} >> /root/.ssh/authorized_keys
SHELL
end
This working example appends ~/.ssh/id_rsa.pub to the ~/.ssh/authorized_keys of both the vagrant and root user, which will allow you to use your existing SSH key.
Copying the desired public key would fall squarely into the provisioning phase. The exact answer depends on what provisioning you fancy to use (shell, Chef, Puppet etc). The most trivial would be a file provisioner for the key, something along this:
config.vm.provision "file", source: "~/.ssh/id_rsa.pub", destination: "~/.ssh/me.pub"
Well, actually you need to append to authorized_keys. Use the the shell provisioner, like so:
Vagrant.configure(2) do |config|
# ... other config
config.vm.provision "shell", inline: <<-SHELL
cat /home/vagrant/.ssh/me.pub >> /home/vagrant/.ssh/authorized_keys
SHELL
# ... other config
end
You can also use a true provisioner, like Puppet. For example see Managing SSH Authorized Keys with Puppet.
There's a more "elegant" way of accomplishing what you want to do. You can find the existing private key and use it instead of going through the trouble of adding your public key.
Proceed like this to see the path to existing private key (look below for IdentityFile):
run vagrant ssh-config
result:
$ vagrant ssh-config
Host magento2.vagrant150
HostName 127.0.0.1
User vagrant
Port 3150
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile "/Users/madismanni/m2/vagrant-magento/.vagrant/machines/magento2.vagrant150/virtualbox/private_key"
IdentitiesOnly yes
LogLevel FATAL
Then you can use the private key like this, note also the switch for switching off password authentication
ssh -i /Users/madismanni/m2/vagrant-magento/.vagrant/machines/magento2.vagrant150/virtualbox/private_key -o PasswordAuthentication=no vagrant#127.0.0.1 -p 3150
This excellent answer was added by user76329 in a rejected Suggested Edit
Expanding on Meow's example, we can copy the local pub/private ssh keys, set permissions, and make the inline script idempotent (runs once and will only repeat if the test condition fails, thus needing provisioning):
config.vm.provision "shell" do |s|
ssh_prv_key = ""
ssh_pub_key = ""
if File.file?("#{Dir.home}/.ssh/id_rsa")
ssh_prv_key = File.read("#{Dir.home}/.ssh/id_rsa")
ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
else
puts "No SSH key found. You will need to remedy this before pushing to the repository."
end
s.inline = <<-SHELL
if grep -sq "#{ssh_pub_key}" /home/vagrant/.ssh/authorized_keys; then
echo "SSH keys already provisioned."
exit 0;
fi
echo "SSH key provisioning."
mkdir -p /home/vagrant/.ssh/
touch /home/vagrant/.ssh/authorized_keys
echo #{ssh_pub_key} >> /home/vagrant/.ssh/authorized_keys
echo #{ssh_pub_key} > /home/vagrant/.ssh/id_rsa.pub
chmod 644 /home/vagrant/.ssh/id_rsa.pub
echo "#{ssh_prv_key}" > /home/vagrant/.ssh/id_rsa
chmod 600 /home/vagrant/.ssh/id_rsa
chown -R vagrant:vagrant /home/vagrant
exit 0
SHELL
end
A shorter and more correct code should be:
ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
config.vm.provision 'shell', inline: 'mkdir -p /root/.ssh'
config.vm.provision 'shell', inline: "echo #{ssh_pub_key} >> /root/.ssh/authorized_keys"
config.vm.provision 'shell', inline: "echo #{ssh_pub_key} >> /home/vagrant/.ssh/authorized_keys", privileged: false
Otherwise user's .ssh/authorized_keys will belong to root user.
Still it will add a line at every provision run, but Vagrant is used for testing and a VM usually have short life, so not a big problem.
I end up using code like:
config.ssh.forward_agent = true
config.ssh.insert_key = false
config.ssh.private_key_path = ["~/.vagrant.d/insecure_private_key","~/.ssh/id_rsa"]
config.vm.provision :shell, privileged: false do |s|
ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
s.inline = <<-SHELL
echo #{ssh_pub_key} >> /home/$USER/.ssh/authorized_keys
sudo bash -c "echo #{ssh_pub_key} >> /root/.ssh/authorized_keys"
SHELL
end
Note that we should not hard code path to /home/vagrant/.ssh/authorized_keys since some vagrant boxes not using the vagrant username.
None of the older posts worked for me although some came close. I had to make rsa keys with keygen in the terminal and go with custom keys. In other words defeated from using Vagrant's keys.
I'm on Mac OS Mojave as of the date of this post. I've setup two Vagrant boxes in one Vagrantfile. I'm showing all of the first box so newbies can see the context. I put the .ssh folder in the same folder as the Vagrant file, otherwise use user9091383 setup.
Credit for this solution goes to this coder.
Vagrant.configure("2") do |config|
config.vm.define "pfbox", primary: true do |pfbox|
pfbox.vm.box = "ubuntu/xenial64"
pfbox.vm.network "forwarded_port", host: 8084, guest: 80
pfbox.vm.network "forwarded_port", host: 8080, guest: 8080
pfbox.vm.network "forwarded_port", host: 8079, guest: 8079
pfbox.vm.network "forwarded_port", host: 3000, guest: 3000
pfbox.vm.provision :shell, path: ".provision/bootstrap.sh"
pfbox.vm.synced_folder "ubuntu", "/home/vagrant"
pfbox.vm.provision "file", source: "~/.gitconfig", destination: "~/.gitconfig"
pfbox.vm.network "private_network", type: "dhcp"
pfbox.vm.network "public_network"
pfbox.ssh.insert_key = false
ssh_key_path = ".ssh/" # This may not be necessary. I may remove.
pfbox.vm.provision "shell", inline: "mkdir -p /home/vagrant/.ssh"
pfbox.ssh.private_key_path = ["~/.vagrant.d/insecure_private_key", ".ssh/id_rsa"]
pfbox.vm.provision "file", source: ".ssh/id_rsa.pub", destination: ".ssh/authorized_keys"
pfbox.vm.box_check_update = "true"
pfbox.vm.hostname = "pfbox"
# VirtualBox
config.vm.provider "virtualbox" do |vb|
# vb.gui = true
vb.name = "pfbox" # friendly name for Oracle VM VirtualBox Manager
vb.memory = 2048 # memory in megabytes 2.0 GB
vb.cpus = 1 # cpu cores, can't be more than the host actually has.
end
end
config.vm.define "dbbox" do |dbbox|
...
This is an excellent thread that helped me solve a similar situation as the original poster describes.
While I ultimately used the settings/logic presented in smartwjw’s answer, I ran into a hitch since I use the VAGRANT_HOME environment variable to save the core vagrant.d directory stuff on an external hard drive on one of my development systems.
So here is the adjusted code I am using in my Vagrantfile to accommodate for a VAGRANT_HOME environment variable being set; the “magic” happens in this line vagrant_home_path = ENV["VAGRANT_HOME"] ||= "~/.vagrant.d":
config.ssh.insert_key = false
config.ssh.forward_agent = true
vagrant_home_path = ENV["VAGRANT_HOME"] ||= "~/.vagrant.d"
config.ssh.private_key_path = ["#{vagrant_home_path}/insecure_private_key", "~/.ssh/id_rsa"]
config.vm.provision :shell, privileged: false do |shell_action|
ssh_public_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
shell_action.inline = <<-SHELL
echo #{ssh_public_key} >> /home/$USER/.ssh/authorized_keys
SHELL
end
For the inline shell provisioners - it is common for a public key to contains spaces, comments, etc. So make sure to put (escaped) quotes around the var that expands to the public key:
config.vm.provision 'shell', inline: "echo \"#{ssh_pub_key}\" >> /home/vagrant/.ssh/authorized_keys", privileged: false
A pretty complete example, hope this helps someone who visits next. Moved all the concrete values to external config files. IP assignment is just for trying out.
# -*- mode: ruby -*-
# vi: set ft=ruby :
require 'yaml'
vmconfig = YAML.load_file('vmconfig.yml')
=begin
Script to created VMs with public IPs, VM creation governed by the provided
config file.
All Vagrant configuration is done below. The "2" in Vagrant.configure
configures the configuration version (we support older styles for
backwards compatibility). Please don't change it unless you know what
you're doing
Default user `vagrant` is created and ssh key is overridden. make sure to have
the files `vagrant_rsa` (private key) and `vagrant_rsa.pub` (public key) in the
path `./.ssh/`
Same files need to be available for all the users you want to create in each of
these VMs
=end
uid_start = vmconfig['uid_start']
ip_start = vmconfig['ip_start']
vagrant_private_key = Dir.pwd + '/.ssh/vagrant_rsa'
guest_sshkeys = '/' + Dir.pwd.split('/')[-1] + '/.ssh/'
Vagrant.configure('2') do |config|
vmconfig['machines'].each do |machine|
config.vm.define "#{machine}" do |node|
ip_start += 1
node.vm.box = vmconfig['vm_box_name']
node.vm.box_version = vmconfig['vm_box_version']
node.vm.box_check_update = false
node.vm.boot_timeout = vmconfig['vm_boot_timeout']
node.vm.hostname = "#{machine}"
node.vm.network "public_network", bridge: "#{vmconfig['bridge_name']}", auto_config: false
node.vm.provision "shell", run: "always", inline: "ifconfig #{vmconfig['ethernet_device']} #{vmconfig['public_ip_part']}#{ip_start} netmask #{vmconfig['subnet_mask']} up"
node.ssh.insert_key = false
node.ssh.private_key_path = ['~/.vagrant.d/insecure_private_key', "#{vagrant_private_key}"]
node.vm.provision "file", source: "#{vagrant_private_key}.pub", destination: "~/.ssh/authorized_keys"
node.vm.provision "shell", inline: <<-EOC
sudo sed -i -e "\\#PasswordAuthentication yes# s#PasswordAuthentication yes#PasswordAuthentication no#g" /etc/ssh/sshd_config
sudo systemctl restart sshd.service
EOC
vmconfig['users'].each do |user|
uid_start += 1
node.vm.provision "shell", run: "once", privileged: true, inline: <<-CREATEUSER
sudo useradd -m -s /bin/bash -U #{user} -u #{uid_start}
sudo mkdir /home/#{user}/.ssh
sudo cp #{guest_sshkeys}#{user}_rsa.pub /home/#{user}/.ssh/authorized_keys
sudo chown -R #{user}:#{user} /home/#{user}
sudo su
echo "%#{user} ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/#{user}
exit
CREATEUSER
end
end
end
It's rather an old Question but maybe this would help someone nowadays, hopefully.
What works like a charm for me is:
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.define "debian-1"
config.vm.hostname = "debian-1"
# config.vm.network "private_network", ip: "192.168.56.2" # this enables Internal network mode for VirtualBox
config.vm.network "private_network", type: "dhcp" # this enables Host-only network mode for VirtualBox
config.vm.network "forwarded_port", guest: 8081, host: 8081 # with this you can hit http://mypc:8081 to load the web service configured in the vm..
config.ssh.host = "mypc" # use the base host's hostname.
config.ssh.insert_key = true # do not use the global public image key.
config.ssh.forward_agent = true # have already the agent keys preconfigured for ease.
config.vm.provision "ansible" do |ansible|
ansible.playbook = "../../../ansible/playbooks/configurations.yaml"
ansible.inventory_path = "../../../ansible/inventory/hosts.ini"
ansible.extra_vars = {
nodes: "#{config.vm.hostname}",
username: "vagrant"
}
ansible.ask_vault_pass = true
end
end
Then my Ansible provisioner playbook/role configurations.yaml contains this:
- name: Create .ssh folder if not exists
file:
state: directory
path: "{{ ansible_env.HOME }}/.ssh"
- name: Add authorised key (for remote connection)
authorized_key:
state: present
user: "{{ username }}"
key: "{{ lookup('file', 'eos_id_rsa.pub') }}"
- name: Add public SSH key in ~/.ssh
copy:
src: eos_id_rsa.pub
dest: "{{ ansible_env.HOME }}/.ssh"
owner: "{{ username }}"
group: "{{ username }}"
- name: Add private SSH key in ~/.ssh
copy:
src: eos_id_rsa
dest: "{{ ansible_env.HOME }}/.ssh"
owner: "{{ username }}"
group: "{{ username }}"
mode: 0600
Madis Maenni answer is closest to best solution:
just do:
vagrant ssh-config >> ~/.ssh/config
chmod 600 ~/.ssh/config
then you can just ssh via hostname.
To get list of hostnames configured in ~/.ssh/config
grep -E '^Host ' ~/.ssh/config
My example:
$ grep -E '^Host' ~/.ssh/config
Host web
Host db
$ ssh web
[vagrant#web ~]$
Generate a rsa key pair for vagrant authentication ssh-keygen -f ~/.ssh/vagrant
You might also want to add the vagrant identity files to your ~/.ssh/config
IdentityFile ~/.ssh/vagrant
IdentityFile ~/.vagrant.d/insecure_private_key
For some reason we can't just specify the key we want to insert so we take a
few extra steps to generate a key ourselves. This way we get security and
knowledge of exactly which key we need (+ all vagrant boxes will get the same key)
Can't ssh to vagrant VMs using the insecure private key (vagrant 1.7.2)
How do I add my own public key to Vagrant VM?
config.ssh.insert_key = false
config.ssh.private_key_path = ['~/.ssh/vagrant', '~/.vagrant.d/insecure_private_key']
config.vm.provision "file", source: "~/.ssh/vagrant.pub", destination: "/home/vagrant/.ssh/vagrant.pub"
config.vm.provision "shell", inline: <<-SHELL
cat /home/vagrant/.ssh/vagrant.pub >> /home/vagrant/.ssh/authorized_keys
mkdir -p /root/.ssh
cat /home/vagrant/.ssh/authorized_keys >> /root/.ssh/authorized_keys
SHELL