reading dynamic configurations split in multiple properties file from config server - spring-cloud-config

I want to add several hundred configuration files in this pattern:
/application.yaml
/1000/application.yaml
/2000/application.yaml
/3000/application.yaml
/4000/application.yaml
Where 1000, 2000, 3000 are sender codes. When a REST call is made to my API, it will have a parameter 'senderCode' which have values like 1000, 2000, etc
Based on that I want to read the configs form the corresponding application.yaml from config server.
My config server's application yaml has:
cloud:
config:
server:
git:
uri: http://example.com/my-configurations
search-paths: 1000, 1001, 1002, 1003
With above settings, I can configure properties int the client application like this:
#ConfigurationProperties(prefix = "1000")
public class CodeBasedConfig {
String senderName;
String senderSource;
}
But that mean creating thousands of files like the one above. I want to be able to load configs from multiple files into a map like this:
key: 1000
value: configs for 1000
key: 2000
value: configs for 2000
On a side note, it will be bonus, if I could read configurations on demand for a given sender, instead of loading them all upfront.

I changed the file structure a bit but grouping them under a folder named clients
/application.yaml
clients/1000/application.yaml
clients/2000/application.yaml
clients/3000/application.yaml
clients/4000/application.yaml
After this I updated the spring cloud server's searchPaths like this:
spring:
cloud:
config:
server:
git:
uri: <my git url>
username: ${git_username}
password: ${git_password}
label: develop
searchPaths: clients/*
With this the config sever is supposed to read all application.yaml's under 'clients' folder and merge them in a unified configuration.
As an FYI, my application.yaml's under clients folder have content like this:
1000/application.yaml
clients:
'1000':
attribute1: value1
attribute2: value2
2000/application.yaml
clients:
'2000':
attribute1: value1
attribute2: value2
When config server serves these configs to clients the above configs appear as:
clients:
'1000':
attribute1: value1
attribute2: value2
'2000':
attribute1: value1
attribute2: value2
This way I could break/refactor large configuration into more maintainable configuration files

Related

How to define name for s3bucket for different environment in Kafka Sink

I am currently setting up my aws s3 bucket for different environments so I can have data in dev, tqa, stg, and prd. The name of my bucket in dev is s3.dev.kafka.sink while in tqa it is named as s3.tqa.kafka.sink each associated with its correct env. The documentation in the Kafka Connect website doesn't specify how to be set the environments, so I did the following way, however I keep getting errors that the bucket name is not named properly.
I put it in the secret yaml file
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: kafka-sink-s3-secret
namespace: namespace
spec:
backendType: secretManager
data:
-key: s3.tqa.kafka.sink
name: bucket_name
property: bucket_name
While in deployment file
env:
-name: bucket_name
valueFrom:
secretKeyRef:
name:kaka-sink-s3-secret
key: bucket_name
And I will specify the bucket name in the config:
"s3.bucket.name":"'"$bucket_name"'"
But it fails to deploy. Any idea how can i specify as s3.{{ENV}}.kafka.sink so it runs the correct bucket name in their own env in aws
Out of the box, Kafka Connect doesn't have any way to access environment variables other than those defined by the AWS SDK (the keys and profile, at least)
Sounds like you will need to use a ConfigProvider of the Kafka Connect API
Here's one example on Github, which you'd need to compile and load into your Docker images - https://github.com/giogt/kafka-env-config-provider
Inside the connector properties, use like this
"bucket.name": "${env:ENVIRONMENT_VARIABLE_NAME}"
You should be able to use Helm to better separate/template out the full bucket name within the secret/deployment resource definition

Connect App Engine to Google cloud SQL fails

I'm following this guide
I'm filling the config like this:
val datasourceConfig = HikariConfig().apply {
jdbcUrl = "jdbc:mysql:///$DB_NAME"
username = DB_PASS
password = DB_USER
mapOf(
"cloudSqlInstance" to CLOUD_SQL_CONNECTION_NAME,
"socketFactory" to "com.google.cloud.sql.mysql.SocketFactory",
"ipTypes" to "PUBLIC,PRIVATE",
).forEach {
addDataSourceProperty(
it.key,
it.value
)
}
}
output of the gcloud sql instances describe project-name:
backendType: SECOND_GEN
connectionName: project-name:europe-west1:project-name-db
databaseVersion: MYSQL_5_7
failoverReplica:
available: true
gceZone: europe-west1-d
instanceType: CLOUD_SQL_INSTANCE
ipAddresses:
- ipAddress: *.*.*.*
type: PRIMARY
kind: sql#instance
name: project-name-db
project: project-name
region: europe-west1
from which I'm filling my env variables:
DB_NAME=project-name-db
CLOUD_SQL_CONNECTION_NAME=project-name:europe-west1:project-name-db
On the deployed app line val dataSource = HikariDataSource(datasourceConfig) crashes with the following exception:
com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Cannot connect to MySQL server on localhost:3,306.
Make sure that there is a MySQL server running on the machine/port you are trying to connect to and that the machine this software is running on is able to connect to this host/port (i.e. not firewalled). Also make sure that the server has not been started with the --skip-networking flag.
update: I've tried adding google between second and third slashes("jdbc:mysql://google/$DB_NAME"), according to this answer, now I get:
Cannot connect to MySQL server on google:3,306.
I was missing the following dependency:
implementation("com.google.cloud.sql:mysql-socket-factory-connector-j-8:1.2.2")
more info here
Also DB_NAME is not name of gcloud sql instances output, but a database name that should be created in Console -> Project -> Sql -> Databases

About how to configure karate with Http and ssl in one project

I have one Karate project which including test Client Http apis and our internal webservices (using SSL). So I don't know how to set configuration in Karate.js file.
e.g.
1> karate.configure('proxy', { uri: 'http://clientapiaddress',
username: 'xxx', password: 'xxx' }); 2> karate.configure('ssl',
{keyStore: ' xxx.p12', keyStorePassword: 'xxx'});
When I run feature which is testing internal web services, always get error
"DNS_Fail" as: " The host name resolution (DNS lookup) for this host
name ( ) has failed. The
Internet address may be misspelled or obsolete, the host
( ) may be temporarily
unavailable, or the DNS server may be unresponsive. "
Thanks
you can do * configure ssl = true as your background step

Spring Cloud Config - Multiple Composite Repositories?

Is it possible configure Spring Cloud Config with multiple composite repositories? Our setup uses multiple team-based repositories:
spring:
cloud:
config:
server:
git:
repos:
teamA:
cloneOnStart: true
pattern: teama-*
searchPaths: '{profile}'
uri: file:///etc/config/teama
teamB:
cloneOnStart: true
pattern: teamb-*
searchPaths: '{profile}'
uri: file:///etc/config/teamb
We want each team to now pull from 2 different git repositories. I think multiple Composite repositories (https://cloud.spring.io/spring-cloud-config/single/spring-cloud-config.html#composite-environment-repositories) is what we want, but I can't figure out how to combine them, or if it is even possible.
Clarification:
I want each team to pull configuration data from two repos instead of just the 1. In pseudo code:
spring:
cloud:
config:
server:
git:
repos:
teamA:
repo1:
key1: repo1
key2: repo1
repo2:
key1: repo2
key2: repo2
In this case your can use composite configuration. Next configuration is working for me, client service is using properties from 2 repositories
spring:
profiles:
active: composite
cloud:
config:
server:
composite:
-
type: git
cloneOnStart: true
uri: https://github.com/..../test-config
-
type: git
cloneOnStart: true
uri: https://github.com/..../test-config-2
You may need to configure 'searchPaths' for each for your needs. Profile should be composite (at least one of profiles)

Pattern matching for profile in Spring Cloud Config Server

Context
I am attempting to separate configuration information for our applications using the pattern-matching feature in Spring Cloud Config Server. I have created a repo for "production" environment with a property file floof.yaml. I have created a repo for "development" environment with a property file floof-dev.yaml.
My server config:
spring:
application:
name: "bluemoon"
cloud:
config:
server:
git:
uri: file://${user.home}/tmp/prod
repos:
development:
pattern:
- \*/dev
uri: file://${user.home}/tmp/dev
After starting the server instance, I can successfully retrieve the config content using curl, and can verify which content was served by referring to the "source" element as well as the values for the properties themselves.
Expected Behavior
When I fetch http://localhost:8080/floof/prod I expect to see the source "$HOME/tmp/prod/floof.yaml" and the values from that source, and the actual results match that expectation.
When I fetch http://localhost:8080/floof/dev I expect to see the source "$HOME/tmp/dev/floof-dev.yaml" and the values from that source, but the actual result is the "production" file and contents (the same as if I had fetched .../floof/prod instead.
My Questions
Is my expectation of behavior incorrect? I assume not since there is an example in the documentation in the "Git backend" section that suggests separation by profile is a thing.
Is my server config incorrectly specifying the "development" repo? I turned up the logging verbosity in the server instance and saw nothing in there that called attention to itself in terms of misconfiguration.
Are the property files subject to a naming convention that I'm not following?
I had the same issue. Here is how I resolved::
spring cloud config pattern match for profile
Also, check if you are using Brixton.M5 version.
After some debugging on the PatternMatching source code here is how I resolved the issue: You can choose one of the two ways.
application.yml
server:
port: 8888
spring:
cloud:
config:
server:
git:
uri: ssh://xxxx#github/sample/cloud-config-properties.git
repos:
development:
pattern: '*/development' ## give in quotes
uri: ssh://git#xxxgithub.com/development.git
OR
development:
pattern: xx*/development,*/development ##since it is not allowed to have a value starting with a wildcard( '*' )after pattern I first gave a generic matching but the second value is */development. Since pattern takes multiple values, the second pattern will match with the profile.
uri: ssh://git#xxxgithub.com/development.git
pattern: */development.Error on yml file- expected alphabetic or numeric character, but found but found /.
The reason the profile pattern git repo was not identified because : although spring allows multiple array values for pattern beginning with a '-' in the yml file, the pattern matcher was taking the '-' as string to be matched. i.e it is looking for a pattern '-*/development' instead of '*/development'.
repos:
development:
pattern:
-*/development
-*/staging
Another issue i observed was, I was getting a compilation error on yml file if i had to mention the pattern array as '- */development' - note space after hyphen(which is meant to show that it can hold multiple values as array) and start with a '*/development' with an error: expected alphabetic or numeric character, but found but found /
repos:
development:
pattern:
- */development
- */staging