I have a docker-compose.yml file with well-known environment variables to reach our corporate proxy:
---
version: '2.2'
# adapted from:
# https://github.com/SeleniumHQ/docker-selenium/wiki/Getting-Started-with-Docker-Compose
# docker-compose --force-recreate
services:
chrome:
privileged: True
image: "selenium/standalone-chrome:3.11.0"
ports:
- "4444:4444"
volumes:
- /dev/shm:/dev/shm
environment:
- TZ="UT"
- http_proxy=http://proxy.lan:8080
- https_proxy=http://proxy.lan:8080
- no_proxy=
#- SE_OPTS=-Dhttp.proxyHost=proxy.lan -Dhttp.proxyPort=8080
network_mode: host
When I run wget in the container, then the proxy is used as expected.
--2018-05-02 12:30:45-- http://google.com/
Resolving proxy.lan (proxy.lan)... 192.168.33.141
Connecting to proxy.lan (proxy.lan)|192.168.33.141|:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://www.google.com/ [following]
--2018-05-02 12:30:45-- http://www.google.com/
Reusing existing connection to proxy.lan:8080.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’
/dev/null [ <=> ] 10.81K --.-KB/s in 0s
2018-05-02 12:30:46 (310 MB/s) - ‘/dev/null’ saved [11073]
However, when I try to run the container with SE_OPTS="-Dhttp.proxyHost=proxy.lan -Dhttp.proxyPort=8080", then I see a stacktrace:
Exception in thread "main" com.beust.jcommander.ParameterException: Was passed main parameter '"-Dhttp.proxyHost=proxy.lan' but no main parameter was defined in your arg class.
There is an unmerged PR, but I fear the urgency to use a proxy in testing in corporate environment might not be felt by the Se dev. Maybe there is an alternative solution.
Related
I have my docker-compose.override.yml set up below in Visual Studio 2022
elasticsearch:
container_name: elasticsearch
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.type=single-node
restart: always
ports:
- "9200:9200"
- "9300:9300"
networks:
elastic:
kibana:
container_name: kibana
restart: always
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
depends_on:
- elasticsearch
ports:
- "5601:5601"
networks:
elastic:
networks:
elastic:
driver: bridge
Every time I try to configure kibana, after supplying a token, I get the following error from the kibana container w/out kibana getting fully configured.
2023-02-06 21:40:38 [2023-02-07T04:40:38.210+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-02-06 21:40:38 [2023-02-07T04:40:38.366+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
2023-02-06 21:40:38 [2023-02-07T04:40:38.508+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
2023-02-06 21:40:39 [2023-02-07T04:40:39.605+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
2023-02-06 21:40:39 [2023-02-07T04:40:39.741+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 192.168.128.3:60856, Remote: 192.168.128.2:9200
2023-02-06 21:40:40 [2023-02-07T04:40:40.773+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
2023-02-06 21:40:42 [2023-02-07T04:40:42.194+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 192.168.128.3:44378, Remote: 192.168.128.2:9200
I am a bit at a loss as to why I can't get it configured. Any help appreciated.
I'm trying out traefik and developing a dummy plugin right now. After countless of errors, I finally get rid all of the errors and don't get any error anymore.
But, the plugin doesn't work as intended. It doesn't throw errors, but it doesn't seem to work either. Is there any way to confirm that the Plugin actually runs? By "running", I mean that I configure it properly, but the function just doesn't return what I want.
This is the output after running the docker compose.
echo-server | Echo server listening on port 8080.
traefik-proxy | time="2022-03-21T07:58:28Z" level=info msg="Configuration loaded from flags."
That's just it. No errors, no exit. And it throws a GET log after refreshing the web page, so I assume there are no errors blocking the code. But I'm still not what's wrong. Is it the plugin code or the configuration?
If this is necessary, this is some of my codes:
# docker-compose.yml
version: "3.3"
networks:
traefik-proxy:
volumes:
traefik-proxy:
services:
traefik-proxy:
image: "traefik:latest"
container_name: "traefik-proxy"
networks:
- traefik-proxy
command:
# - "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--experimental.localPlugins.traefik-denyuseragent.modulename=github.com/xxx/denyuseragent"
ports:
- "80:80"
- "8080:8080"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./plugins-local/:/plugins-local/
echo-server:
image: "xxx/echo-server"
container_name: "echo-server"
networks:
- traefik-proxy
labels:
- "traefik.enable=true"
- "traefik.http.routers.echoserver.rule=Host(`echoserver.localhost`)"
- "traefik.http.routers.echoserver.entrypoints=web"
- "traefik.http.routers.echoserver.middlewares=traefik-denyuseragent"
# .traefik.yml
displayName: Plugin
type: middleware
import: github.com/xxx/denyuseragent
summary: 'Example'
testData:
userAgent:
- Firefox
- Mozilla/5.0
# traefik.yml
experimental:
localPlugins:
traefik-denyuseragent:
modulename: "github.com/xxx/denyuseragent"
# config.yml
http:
routers:
my-router:
rule: host(`echoserver.localhost`)
service: service-echoserver
entryPoints:
- web
middlewares:
- traefik-denyuseragent
services:
service-echoserver:
loadBalancer:
servers:
- url: http://127.0.0.1:5000
middlewares:
traefik-denyuseragent:
plugin:
traefik-denyuseragent:
userAgent:
- Mozilla/5.0
Anyone can confirm if it could work properly or not? Is there some kinds of code I can run to make sure if the Plugin is configured properly or not? Because it'd be great if it is so I can move on to another task, I've spent days just try to configure it.
I just wanna know if the problem is on the plugin or the configuration.
Error: I won't open a connection to 127.0.0.1 (only to 172.20.0.1) (code=500)
My docker file is as below, I have tried passive ports range too but not sure about --expose here, I am able to connect at first through my code but at a second attempt or reusing it is throwing the above error.
I understand it has something with the passive mode but I have the configuration but not sure what is missing, please help.
version: '3'
services:
ftpd_server:
image: stilliard/pure-ftpd
container_name: pure-ftpd
privileged: true
ports:
- "21:21"
- "30000-30099:30000-30099"
volumes: # remember to replace /folder_on_disk/ with the path to where you want to store the files on the host machine
#- "/tmp/test/data:/home/username/"
#- "/tmp/test/passwd:/etc/pure-ftpd/passwd"
- "/upload:/home/upload"
#- "./certs:/etc/ssl/private"
environment:
PUBLICHOST: "localhost"
FTP_USER_NAME: lktransfer
FTP_USER_PASS: lktransfer
FTP_USER_HOME: /home/upload/
#FTP_PASSIVE_PORTS: "30000-30009"
FTP_MAX_CLIENTS: 50
FTP_PASSIVE_PORTS: 30000:30099
FTP_MAX_CONNECTIONS: 50
ADDED_FLAGS: "--tls=1"
#ADDED_FLAGS: "--tls=2"
TLS_USE_DSAPRAM: "true"
TLS_CN: "localhost"
TLS_ORG: "Ellkay"
TLS_C: "IN"
restart: always
I am trying to get Traefik setup in a Docker and am having a heck of a time. Following this guide and using Cloudflare (DNS only to trafeik.mydomain.com), to connect, I am getting "This site can't be reached oauth.mydomain.com's server IP address could not be found".
wget https://traefik.mydomain.com/dashboard
--2020-09-26 19:19:38-- https://traefik.mydomain.com/dashboard
Resolving traefik.mydomain.com (traefik.mydomain.com)... <ip address>
Connecting to traefik.mydomain.com (traefik.mydomain.com)|<ip address>|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://accounts.google.com/o/oauth2/auth?client_id=6597174190-33npvgec044jtcrj4scmfgt561.apps.googleusercontent.com&prompt=select_account&redirect_uri=https%3A%2F%2Foauth.mydomain.com%2F_oauth&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=fc4c85a1e11f4914247d1e7c95b031%3Agoogle%3Ahttps%3A%2F%2Ftraefik.mydomain.com%2Fdashboard [following]
--2020-09-26 19:19:38-- https://accounts.google.com/o/oauth2/auth?client_id=6597114190-33npkhvge44jtcrj4scmuafgt561.apps.googleusercontent.com&prompt=select_account&redirect_uri=https%3A%2F%2Foauth.mydomain.com%2F_oauth&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=fc4c85a1e11f4914247d1e7c94a5b031%3Agoogle%3Ahttps%3A%2F%2Ftraefik.mydomain.com%2Fdashboard
Resolving accounts.google.com (accounts.google.com)... 172.217.1.205, 2607:f8b0:400f:805::200d
Connecting to accounts.google.com (accounts.google.com)|172.217.1.205|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://accounts.google.com/AccountChooser?oauth=1&continue=https%3A%2F%2Faccounts.google.com%2Fsignin%2Foauth%2Flegacy%2Fconsent%3Fauthuser%3Dunknown%26part%3DAJi8hAPSHE1cMVuaeTQ61pcXpMEfDhbxN02IAjg5jH0GYWG7yCVc5EC9ug_kuK3j7hUBGCiY_q_SGL-0xSPDtiliIthwpmOTvP_MV5upFvDdYTpTTlraXSxx_7f8vhJteA8UjJoKSqgeUvFWns_BdFn8z73XALchawMrWA1vVl0xJYpUYHUXxD3K0zl4TbcgpVOljSfZM0vkQAHwTm54OjNiw51GTMCJAwiGwh_ANodLXY1n07UrO6-AgJ1pEeRksrlKs-O2W2Az1Fj4QWMej3OJ8HiHfRVlBt8c8zStbROoFMIce9ldHm5FF-l54b3xQcBp4xLi6ABqcrciv_Y0TAFuuwwotfgqrl1_uMHfyX9KJogk_gntcEiG2489OMNwFinOVAPUCg1Z-gn-ps7g_oBl4MB-FzsIiVpfyy_qRD7SMyhvnVe4Bj-%26as%3DS-2012888342%253A160116957872%23 [following]
--2020-09-26 19:19:38-- https://accounts.google.com/AccountChooser?oauth=1&continue=https%3A%2F%2Faccounts.google.com%2Fsignin%2Foauth%2Flegacy%2Fconsent%3Fauthuser%3Dunknown%26part%3DAJi8hAPSk6oHE1cMVRfegIuaeTQ61pcXpMEfD2FXah02IAjg5GYWG7yCVc5EC9ug_kuK3j7hUBGCiY_q_SGL-0xSPDtiliIthwpmOTvP_MV5upFvDdYTpTTSxx_7f8vhJteA8UjJoKSqgeUvFWns_BdFn8z73XALchawMrWA1vVXbAl0xJYpUYHUXzl4TbcgpVOljSfZM0vkQAHwTmhD54OjNiw51GTMCJAwiGwh_ANodLXY1n07UrO6-AgJ1pEeRksrlKs-O2W2Az1Fj4QWMej3OJ8HiHfRVlBt8c8zStbROoFMIce9ldHm5FF-l54b3xQcBpABqcrciv_Y0TAFuuwwotfgqrl1_uMHfyX9KJogk_gntcEiG2489OMNwFAPUCg1Z-gn-ps7g_oBl4MB-FzsIiVpfyy_qRDBteJ7SMyhvnVe4Bj-%26as%3DS-2012888342%253A1601169578900872%23
Reusing existing connection to accounts.google.com:443.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://accounts.google.com/ServiceLogin?continue=https%3A%2F%2Faccounts.google.com%2Fsignin%2Foauth%2Flegacy%2Fconsent%3Fauthuser%3Dunknown%26part%3DAJi8hAPSkE1cMVRIuaeTQ61ppMEfD2FXahbxN02IAjg5jH0GYWG7yCVc5EC9ug_kuK3j7hUBGCiY_q_SGL-0xSPDtiliIthwpmOTvP_MV5upFvDdYTpTTlraXSxx_7f8vhJteA8UjJoKSqgs_BdFn8z73XALchawMrWA1vVXbAl0xJYpUYHUXxD3K0zl4TbcgpVOljSfZM0vkQAHwTmhD54OjNiw51GTMCJAwiGwh_ANodLXY1n07UrO6-AgJ1pEeRksrlKs-O2W2Az1Fj4QWMej3OJ8HiHBt8c8zStbROoFMIce9ldHm5FF-l54b3xQcBp4xLi6ABqcrciv_Y0TAFuuwwotfgqrl1_uMHfyX9KJntcEiG2489OMNwFinOVAPUCg1Z-gn-ps7g_oBl4MB-FzsIiVpfyy_qRDBteJ7SMyhvnVe4Bj-%26as%3DS-2012888342%253A1601178900872%23&sacu=1&oauth=1&rip=1 [following]
--2020-09-26 19:19:39-- https://accounts.google.com/ServiceLogin?continue=https%3A%2F%2Faccounts.google.com%2Fsignin%2Foauth%2Flegacy%2Fconsent%3Fauthuser%3Dunknown%26part%3DAJi8hAPSk6ogIuaeTQ61pcXpMEfD2FXahbxN02IAjg5jH0GYWG7yCVc5EC9ug_kuK3j7hUBGCiY_q_SGL-0xSPDtiliIthwpmOTvP_MV5upFvDdYTpTTlraXSxx_7f8vhJteA8UjJoKSqgeUvFWns_BdFn8z73XALchawMrWA1vVXbAl0xJYpUYHUXxD3K0zl4jSfZM0vkQAHwTmhD54OjNiw51GTMCJAwiGwh_ANodLXY1n07UrO6-AgJ1pEeRksrlKs-O2W2Az1Fj4QWMej3OVlBt8c8zStbROoFMIce9ldHm5FF-l54b3xQcBp4xLi6ABqcrciv_Y0TAFuuwwotfgqrl1_uKJogk_gntcEiG2489OMNwFinOVAPUCg1Z-gn-ps7g_oBl4MB-FzsIiVpfyy_qRDBteJ7SMyhvnVe4Bj-%26as%3DS-2012888342%253A1601169578900872%23&sacu=1&oauth=1&rip=1
Reusing existing connection to accounts.google.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘dashboard.1’
dashboard.1 [ <=> ] 58.82K --.-KB/s in 0.03s
2020-09-26 19:19:39 (1.64 MB/s) - ‘dashboard.1’ saved [60236]
The Docker log says:
level=debug msg="Remote error http://oauth:4181. StatusCode: 307"
middlewareType=ForwardedAuthType middlewareName=middlewares-oauth#file
This is my docker-compose.yml file:
version: "3.3"
########################### NETWORKS
networks:
t2_proxy:
external:
name: t2_proxy
default:
driver: bridge
########################### SERVICES
services:
# All services / apps go below this line
# Traefik 2 - Reverse Proxy
traefik:
container_name: traefik
image: traefik:chevrotin # the chevrotin tag refers to v2.2.x but introduced a breaking change in 2.2.2
restart: unless-stopped
command: # CLI arguments
- --global.checkNewVersion=true
- --global.sendAnonymousUsage=false
- --entryPoints.http.address=:80
- --entryPoints.https.address=:443
# Allow these IPs to set the X-Forwarded-* headers - Cloudflare IPs: https://www.cloudflare.com/ips/
- --entrypoints.https.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22
- --entryPoints.traefik.address=:8080
- --api=true
# - --api.insecure=true
# - --serversTransport.insecureSkipVerify=true
- --log=true
- --log.level=DEBUG # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC
- --accessLog=true
- --accessLog.filePath=/traefik.log
- --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines
- --accessLog.filters.statusCodes=400-499
- --providers.docker=true
- --providers.docker.endpoint=unix:///var/run/docker.sock
- --providers.docker.defaultrule=Host(`{{ index .Labels "com.docker.compose.service" }}.$DOMAINNAME`)
- --providers.docker.exposedByDefault=false
- --providers.docker.network=t2_proxy
- --providers.docker.swarmMode=false
- --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory.
# - --providers.file.filename=/path/to/file # Load dynamic configuration from a file.
- --providers.file.watch=true # Only works on top level files in the rules folder
# - --certificatesResolvers.dns-cloudflare.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # LetsEncrypt Staging Server - uncomment when testing
- --certificatesResolvers.dns-cloudflare.acme.email=$CLOUDFLARE_EMAIL
- --certificatesResolvers.dns-cloudflare.acme.storage=/acme.json
- --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.provider=cloudflare
- --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.resolvers=1.1.1.1:53,1.0.0.1:53
# networks:
# t2_proxy:
# ipv4_address: 192.168.90.254 # You can specify a static IP
networks:
- t2_proxy
security_opt:
- no-new-privileges:true
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8080
published: 8080
protocol: tcp
mode: host
volumes:
- $DOCKERDIR/traefik2/rules:/rules
- /var/run/docker.sock:/var/run/docker.sock:ro
- $DOCKERDIR/traefik2/acme/acme.json:/acme.json # cert location - you must touch this file and change permissions to 600
- $DOCKERDIR/traefik2/traefik.log:/traefik.log # for fail2ban - make sure to touch file before starting container
- $DOCKERDIR/shared:/shared
environment:
- CF_API_EMAIL=$CLOUDFLARE_EMAIL
- CF_API_KEY=$CLOUDFLARE_API_KEY
labels:
- "traefik.enable=true"
# HTTP-to-HTTPS Redirect
- "traefik.http.routers.http-catchall.entrypoints=http"
- "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
# HTTP Routers
- "traefik.http.routers.traefik-rtr.entrypoints=https"
- "traefik.http.routers.traefik-rtr.rule=Host(`traefik.$DOMAINNAME`)"
- "traefik.http.routers.traefik-rtr.tls=true"
# - "traefik.http.routers.traefik-rtr.tls.certResolver=dns-cloudflare" # Comment out this line after first run of traefik to force the use of wildcard certs
- "traefik.http.routers.traefik-rtr.tls.domains[0].main=$DOMAINNAME"
- "traefik.http.routers.traefik-rtr.tls.domains[0].sans=*.$DOMAINNAME"
# - "traefik.http.routers.traefik-rtr.tls.domains[1].main=$SECONDDOMAINNAME" # Pulls main cert for second domain
# - "traefik.http.routers.traefik-rtr.tls.domains[1].sans=*.$SECONDDOMAINNAME" # Pulls wildcard cert for second domain
## Services - API
- "traefik.http.routers.traefik-rtr.service=api#internal"
## Middlewares
- "traefik.http.routers.traefik-rtr.middlewares=chain-oauth#file"
# Google OAuth - Single Sign On using OAuth 2.0
oauth:
container_name: oauth
image: thomseddon/traefik-forward-auth:latest
restart: unless-stopped
networks:
- t2_proxy
security_opt:
- no-new-privileges:true
environment:
- CLIENT_ID=$GOOGLE_CLIENT_ID
- CLIENT_SECRET=$GOOGLE_CLIENT_SECRET
- SECRET=$OAUTH_SECRET
- COOKIE_DOMAIN=$DOMAINNAME
- INSECURE_COOKIE=false
- AUTH_HOST=oauth.$DOMAINNAME
- URL_PATH=/_oauth
- WHITELIST=$MY_EMAIL
- LOG_LEVEL=debug
- LOG_FORMAT=text
- LIFETIME=2592000 # 30 days
- DEFAULT_ACTION=auth
- DEFAULT_PROVIDER=google
labels:
- "traefik.enable=true"
## HTTP Routers
- "traefik.http.routers.oauth-rtr.entrypoints=https"
- "traefik.http.routers.oauth-rtr.rule=Host(`oauth.$DOMAINNAME`)"
- "traefik.http.routers.oauth-rtr.tls=true"
## HTTP Services
- "traefik.http.routers.oauth-rtr.service=oauth-svc"
- "traefik.http.services.oauth-svc.loadbalancer.server.port=4181"
## Middlewares
- "traefik.http.routers.oauth-rtr.middlewares=chain-oauth#file"
Finally, the end of my middlewares.toml file looks like this:
[http.middlewares.middlewares-oauth]
[http.middlewares.middlewares-oauth.forwardAuth]
address = "http://oauth:4181" # Make sure you have the OAuth service in docker-compose.yml
trustForwardHeader = true
authResponseHeaders = ["X-Forwarded-User"]
I searched around and checked everything I found already suggested but no luck. Seems like it's gotta be something small though.
In Cloudflare, I changed oauth.mydomain.com from "Proxied" to "DNS Only" and now I am no longer getting redirected.
I have been able to successfully connect the GCS connector without SASL or SSL enabled. When I enable SASL and SSL; connect-standalone does not seem to be able to communicate with the brokers.
The problem appears to be with the gcs-sink-license-manager. This is what I have found from the logs but they aren't super helpful for me to actually figuring out what the issue is....
LOGS
[2018-12-19 16:29:05,645] INFO [AdminClient clientId=gcs-sink-license-manager] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:238)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
[2018-12-19 16:29:05,647] ERROR WorkerConnector{id=gcs-sink} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:119)
org.apache.kafka.connect.errors.ConnectException: Timed out while checking for or creating topic(s) '_confluent-command'. This could indicate a connectivity issue, unavailable topic partitions, or if this is your first use of the topic it may have taken too long to create.
at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:251)
at io.confluent.license.LicenseStore$1.run(LicenseStore.java:159)
at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
at io.confluent.license.LicenseStore.start(LicenseStore.java:187)
at io.confluent.license.LicenseManager.<init>(LicenseManager.java:42)
at io.confluent.connect.gcs.GcsSinkConnector.checkLicense(GcsSinkConnector.java:80)
at io.confluent.connect.gcs.GcsSinkConnector.start(GcsSinkConnector.java:67)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:111)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:136)
at org.apache.kafka.connect.runtime.WorkerConnector.transitionTo(WorkerConnector.java:195)
at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:241)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startConnector(StandaloneHerder.java:297)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:206)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2018-12-19 16:29:05,649] INFO Finished creating connector gcs-sink (org.apache.kafka.connect.runtime.Worker:257)
[2018-12-19 16:29:05,650] INFO Skipping reconfiguration of connector gcs-sink since it is not running (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:329)
[2018-12-19 16:29:05,652] INFO Created connector gcs-sink (org.apache.kafka.connect.cli.ConnectStandalone:104)
Connector Properties
connector.class="io.confluent.connect.gcs.GcsSinkConnector"
storage.class="io.confluent.connect.gcs.storage.GcsStorage"
bootstrap.servers=kafka1:19092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/share/java,/usr/share/confluent-hub-components
gcs.sasl.properties
#Connector
format.class=io.confluent.connect.gcs.format.json.JsonFormat
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
flush.size=3
# confluent.license=
#GCS
name=gcs-sink
connector.class=io.confluent.connect.gcs.GcsSinkConnector
gcs.bucket.name=kafka-bucket-4c
gcs.part.size=5242880
gcs.credentials.path=/usr/share/assets/gcs-key.json
confluent.topic.bootstrap.servers=kafka1:19092
topics=sandbox
confluent.topic.replication.factor=1
#Storage
storage.class=io.confluent.connect.gcs.storage.GcsStorage
client.id=gcs-standalone-sink
# Sink authentication settings
consumer.log4j.root.loglevel=DEBUG
consumer.bootstrap.servers=kafka1:19092
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_PLAINTEXT
consumer.ssl.endpoint.identification.algorithm=
Dockerfile
FROM confluentinc/cp-kafka-connect
ADD assets /usr/share/assets
# ENV CONNECT_OPTS "-Djava.security.auth.login.config=/usr/share/assets/kafka_admin_account.conf -Djavax.net.ssl.trustStore=/usr/share/assets/secrets/kafka.client.truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
ENV KAFKA_OPTS "-Djava.security.auth.login.config=/usr/share/assets/secrets/kafka_admin_account.conf -Djavax.net.debug=all"
ENV CONNECT_OPTS "-Djava.security.auth.login.config=/usr/share/assets/secrets/kafka_admin_account.conf -Djavax.net.debug=all"
COPY assets/secrets/cacerts /usr/lib/jvm/zulu-8-amd64/jre/lib/security/cacerts
CMD ["/bin/bash", "-c", "connect-standalone ${CONNECT_PROPS} ${GCS_PROPS}"]
docker-compose file
kafka1:
image: company-kafka-secure
# build: ./
depends_on:
- zookeeper
ports:
- 19091:19091
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka1:19092,EXT://localhost:19091
KAFKA_LISTENERS: SASL_PLAINTEXT://:19092,EXT://:19091
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL_PLAINTEXT:SASL_PLAINTEXT,EXT:SASL_PLAINTEXT
KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
ZOOKEEPER_SASL_ENABLED: "false"
KAFKA_AUTHORIZER_CLASS_NAME: com.us.digital.kafka.security.authorization.KafkaAuthorizer
CONFLUENT_METRICS_ENABLE: "false"
volumes:
- ./secrets:/etc/kafka/secrets
networks:
- message_hub
kafka_gcs_connect:
build: ./kafka-connect
ports:
- 28082:28082
depends_on:
- kafka1
- kafka3
- kafka2
- zookeeper
environment:
CONNECT_PROPS: /usr/share/assets/connect-standalone.sasl.properties
CONNECT_REST_PORT: 28082
GCS_PROPS: /usr/share/assets/gcs.sasl.properties
networks:
- message_hub
CONNECT_BOOTSTRAP_SERVERS=kafka1:19092,kafka2:29092,kafka3:39092
CONNECT_CONFLUENT_TOPIC_BOOTSTRAP_SERVERS=kafka1:19092,kafka2:29092,kafka3:39092
CONNECT_CONFLUENT_LICENSE=
CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false
CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=false
CONNECT_CONFIG_STORAGE_TOPIC=connect-config
CONNECT_OFFSET_STORAGE_TOPIC=connect-offsets
CONNECT_STATUS_STORAGE_TOPIC=connect-status
CONNECT_REPLICATION_FACTOR=1
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
CONNECT_SECURITY_PROTOCOL=SASL_PLAINTEXT
CONNECT_SASL_MECHANISM=PLAIN
CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
CONNECT_CONSUMER_BOOTSTRAP_SERVERS=kafka1:19092,kafka2:29092,kafka3:39092
CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_PLAINTEXT
CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
CONNECT_CONSUMER_SASL_MECHANISM=PLAIN
CONNECT_GROUP_ID=gcs-kafka-connector
CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
CONNECT_REST_PORT=28082
CONNECT_PLUGIN_PATH=/usr/share/java,/usr/share/confluent-hub-components
KAFKA_OPTS=-Djava.security.auth.login.config=/usr/share/assets/kafka_admin_account.conf
Here is all of the properties I found I needed to get SASL working with a gcs connector.