failed to open the plugin manifest /.traefik.yml: open /.traefik.yml: no such file or directory - traefik

I'm new to Traefik. I'm trying to develop a plugin and "connect" it to my webserver using Docker Compose. I use docker compose for development. When I try to run the Docker Compose I encountered this error:
/dev/echo-docker-compose $ docker-compose up --build
Starting echo-server ... done
Recreating traefik ... done
Attaching to echo-server, traefik
echo-server | Echo server listening on port 8080.
traefik | time="2022-03-18T07:40:11Z" level=info msg="Configuration loaded from flags."
traefik | time="2022-03-18T07:40:11Z" level=info msg="Traefik version 2.6.1 built on 2022-02-14T16:50:25Z"
traefik | time="2022-03-18T07:40:11Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":8080\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"insecure\":true,\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"pilot\":{\"dashboard\":true},\"experimental\":{\"localPlugins\":{\"traefik-plugin-example\":{\"moduleName\":\"github.com/usergithub/traefik-plugin-example\"}}}}"
traefik | time="2022-03-18T07:40:11Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik | 2022/03/18 07:40:11 traefik.go:79: command traefik error: 1 error occurred:
traefik | * failed to open the plugin manifest plugins-local/src/github.com/usergithub/traefik-plugin-example/.traefik.yml: open plugins-local/src/github.com/usergithub/traefik-plugin-example/.traefik.yml: no such file or directory
traefik |
traefik |
traefik exited with code 1
I think I can conclude that the cause of this error is that it can't find the .traefik.yml file which is quite obvious. But, I already have the file in that directory.
This is my full directory structure
dev/plugins-local/
└── src
└── github.com
└── traefik
└── traefik-plugin-example
├── plugin.go
├── plugin_test.go
├── .traefik.yml
├── go.mod
├── Dockerfile
├── Makefile
├── traefik
├── traefik-plugin.go
├── rules-plugin.go
dev/echo-docker-compose/
└── docker-compose.yml
└── cmd
└── ....
For some reasons, I think Docker Compose can't read .traefik.yml. But, it's also the same case when I try to run ./traefik --configfile traefik-plugin.yml from dev/plugins-local. It also throws the same error where it can't read the .traefik.yml.
Anyone knows how to fix this problem?
This is the relevant codes:
On server repo
# dev/echo-docker-compose/.env
PLUGIN_MODULE=github.com/usergithub/traefik-plugin-example
PLUGIN_NAME=traefik-plugin-example
# dev/echo-docker-compose/docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.6"
container_name: "traefik"
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--experimental.localPlugins.${PLUGIN_NAME}.moduleName=${PLUGIN_MODULE}"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
echo-server:
image: "usergithub/echo-server"
container_name: "echo-server"
labels:
- "traefik.enable=true"
# The domain the service will respond to
- "traefik.http.routers.echoserver.rule=Host(`echoserver.localhost`)"
# Allow request only from the predefined entry point named "web"
- "traefik.http.routers.echoserver.entrypoints=web"
- "traefik.http.routers.echoserver.entrypoints=web"
- "traefik.http.routers.echoserver.middlewares=echoserver-demo"
- "traefik.http.middlewares.echoserver-demo.plugin.${PLUGIN_NAME}.headers.DoesPluginWork=YES"
- "traefik.http.routers.echoserver.tls.certresolver=default"
On plugin repo
# dev/plugins-local/traefik-plugin.go
pilot:
token: "token"
api:
dashboard: true
insecure: true
experimental:
localPlugins:
traefik-plugin-example:
moduleName: github.com/usergithub/traefik-plugin-example
entryPoints:
http:
address: ":8000"
forwardedHeaders:
insecure: true
providers:
file:
filename: rules-plugin.yaml
# dev/plugins-local/src/github.com/usergithub/traefik-plugin-example/.traefik.yml
displayName: Traefik Example Plugin
type: middleware
import: github.com/usergithub/traefik-plugin-example
summary: 'Write plugin'
testData:
userAgent:
- Firefox
- Mozilla/5.0

according to the doc, "There must be a .traefik.yml file at the root of your project describing your plugin" and from what I see in your directory structure, you placed .traefik.yml in the "src" folder with several sub folder in between.
Putting the file in the right place should solve the problem, please note that go.mod should also be at the root, (and the plugin must use git tag).
As an additional ressource, you can check the demo plugin,
Also for anyone reading, I must add a disclaimer because Traefik plugins are still an "Experimental Features" and should be used with caution

Related

CodeDeploy not properly copying code from GitHub

I have CodeDeploy pull code from my GitHub repo. In the deployment Commit ID (for GitHub) I have specified the Commit ID that I want to deploy. My repo has the following structure:
my-service/
README.md
.gitignore
scripts/
deploy.sh
src/
<lots of code here>
pm2.dev.json
appspec.yml
My appspec.yml file looks like:
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu
hooks:
BeforeInstall:
- location: scripts/deploy.sh
timeout: 300
runas: root
My scripts/deploy.sh looks like:
sudo npm install pm2 -g
pwd
pm2 start /home/ubuntu/my-service/pm2.dev.json
When I run the CodeDeployment deployment for this, it fails with the following error:
Script at specified location: scripts/deploy.sh run as user root failed with exit code 1
When I look at the logs I see:
LifecycleEvent - BeforeInstall
Script - scripts/deploy.sh
[stderr]npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
[stdout]changed 182 packages, and audited 183 packages in 8s
[stdout]
[stdout]12 packages are looking for funding
[stdout] run `npm fund` for details
[stdout]
[stdout]found 0 vulnerabilities
[stdout]/opt/codedeploy-agent
[stderr][PM2][ERROR] File /home/ubuntu/my-service/pm2.dev.json not found
Sure enough when I look in /home/ubuntu/my-service, I do not see a pm2.dev.json file, because this server had been manually configured several weeks ago before a pm2.dev.json file was added to the project. I would have expected CodeDeploy to have written whats in the repo to the server under /home/ubuntu.
Can anyone spot anything wrong with my appspec.yml or other configuration? Could it be a bad GitHub setup?
Had to change BeforeInstall to Install.
BeforeInstall runs before it copies over the source code (specified under files/source). But Install run just after that copy occurs, hence files will be available on the file system.

Cypress E2E Testing VueJS with GitLab Runner returns 404 not found

I have the following simple GitLab CI file
stages:
- e2e_test
e2e_test:
stage: e2e_test
tags:
- test
script:
- npm config set strict-ssl false
- npm install
- npm run test:e2e:ci
I am running the above script in a shell runner for very specific corporate reasons.
This script results in a 404 error with the following log
npm run test:e2e:ci
> todo-app#0.1.0 test:e2e:ci /home/gitlab-runner/builds/UHz6XzwM/0/microservices-and-cognitive-solutions/demos/microservice-concept/todo-app
> vue-cli-service test:e2e --headless
INFO Starting e2e tests...
INFO Starting development server...
DONE Compiled successfully in 8865ms6:51:36 PM
App running at:
- Local: http://localhost:8083/
- Network: http://10.108.201.150:8083/
App is served in production mode.
Note this is for preview or E2E testing only.
====================================================================================================
(Run Starting)
tput: No value for $TERM and no -T specified
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 3.8.3 │
│ Browser: Electron 78 (headless) │
│ Specs: 1 found (test.js) │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: test.js (1 of 1)
Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`
My First Test
1) Visits the app root url
0 passing (341ms)
1 failing
1) My First Test Visits the app root url:
CypressError: cy.visit() failed trying to load:
http://localhost:8083/
The response we received from your web server was:
> 404: Not Found
This was considered a failure because the status code was not '2xx'.
If you do not want status codes to cause failures pass the option: 'failOnStatusCode: false'
at Object.cypressErr (http://localhost:8083/__cypress/runner/cypress_runner.js:86207:11)
at Object.throwErr (http://localhost:8083/__cypress/runner/cypress_runner.js:86162:18)
at Object.throwErrByPath (http://localhost:8083/__cypress/runner/cypress_runner.js:86194:17)
at http://localhost:8083/__cypress/runner/cypress_runner.js:71493:31
at visitFailedByErr (http://localhost:8083/__cypress/runner/cypress_runner.js:71006:12)
at http://localhost:8083/__cypress/runner/cypress_runner.js:71474:22
at tryCatcher (http://localhost:8083/__cypress/runner/cypress_runner.js:120203:23)
at Promise._settlePromiseFromHandler (http://localhost:8083/__cypress/runner/cypress_runner.js:118139:31)
at Promise._settlePromise (http://localhost:8083/__cypress/runner/cypress_runner.js:118196:18)
at Promise._settlePromise0 (http://localhost:8083/__cypress/runner/cypress_runner.js:118241:10)
at Promise._settlePromises (http://localhost:8083/__cypress/runner/cypress_runner.js:118316:18)
at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (http://localhost:8083/__cypress/runner/cypress_runner.js:114928:16)
at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8083/__cypress/runner/cypress_runner.js:114938:10)
at Async.drainQueues (http://localhost:8083/__cypress/runner/cypress_runner.js:114812:14)
If I clone the same repository on the same server and manually run the following commands in the terminal it works fine.
npm install
npm run test:e2e:ci
Any idea why the same commands would fail in a GitLab runner shell but working in a user shell
I found the problem to fix my issue. I needed to set a no_proxy environment entry in my gitlab-runner config.toml file. This won't affect everyone, but since i am behind a corporate proxy, I needed it.
export no_proxy="localhost"

Filebeat not starting TCP server (input)

So I have configured filebeat to accept input via TCP. This is filebeat.yml file.
filebeat.inputs:
- type: tcp
host: ["localhost:9000"]
max_message_size: 20MiB
For some reason filebeat does not start the TCP server at port 9000. I have verified this using wireshark. Wireshark shows nothing at port 9000.
This is output of command "filebeat -e -d "*"" run on terminal
2019-08-14T09:12:40.745-0600 INFO instance/beat.go:468 Home path: [/usr/local/Cellar/filebeat/6.2.4] Config path: [/usr/local/etc/filebeat] Data path: [/usr/local/var/lib/filebeat] Logs path: [/usr/local/var/log/filebeat]
2019-08-14T09:12:40.745-0600 DEBUG [beat] instance/beat.go:495 Beat metadata path: /usr/local/var/lib/filebeat/meta.json
2019-08-14T09:12:40.745-0600 INFO instance/beat.go:475 Beat UUID: 764da0fd-ea93-4777-b1ea-63149be0d6b6
2019-08-14T09:12:40.745-0600 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2019-08-14T09:12:40.745-0600 DEBUG [beat] instance/beat.go:230 Initializing output plugins
2019-08-14T09:12:40.745-0600 DEBUG [processors] processors/processor.go:49 Processors:
2019-08-14T09:12:40.745-0600 INFO pipeline/module.go:76 Beat name: Ad-MBP.domain
2019-08-14T09:12:40.745-0600 ERROR fileset/modules.go:95 Not loading modules. Module directory not found: /usr/local/Cellar/filebeat/6.2.4/module
2019-08-14T09:12:40.745-0600 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2019-08-14T09:12:40.745-0600 INFO instance/beat.go:301 filebeat start running.
2019-08-14T09:12:40.745-0600 DEBUG [registrar] registrar/registrar.go:90 Registry file set to: /usr/local/var/lib/filebeat/registry
2019-08-14T09:12:40.746-0600 INFO registrar/registrar.go:110 Loading registrar data from /usr/local/var/lib/filebeat/registry
2019-08-14T09:12:40.746-0600 INFO registrar/registrar.go:121 States Loaded from registrar: 0
2019-08-14T09:12:40.746-0600 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-14T09:12:40.746-0600 INFO crawler/crawler.go:48 Loading Prospectors: 1
2019-08-14T09:12:40.746-0600 DEBUG [registrar] registrar/registrar.go:152 Starting Registrar
2019-08-14T09:12:40.746-0600 DEBUG [cfgfile] cfgfile/reload.go:95 Checking module configs from: /usr/local/etc/filebeat/modules.d/*.yml
2019-08-14T09:12:40.746-0600 DEBUG [cfgfile] cfgfile/reload.go:109 Number of module configs found: 0
2019-08-14T09:12:40.746-0600 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 0
2019-08-14T09:12:40.746-0600 INFO cfgfile/reload.go:127 Config reloader started
2019-08-14T09:12:40.748-0600 DEBUG [cfgfile] cfgfile/reload.go:151 Scan for new config files
2019-08-14T09:12:40.748-0600 DEBUG [cfgfile] cfgfile/reload.go:170 Number of module configs found: 0
2019-08-14T09:12:40.748-0600 INFO cfgfile/reload.go:219 Loading of config files completed.
I am not sure what I am doing wrong..
I believe filebeat inputs are only available from filebeat 6.3+, anything older used filebeat prospectors.
6.3 TCP input documentation, nothing available for 6.2 or older as it uses prospectors:
https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-tcp.html
Your logs show that you are on filebeat version 6.24, could you try out your configuration with 6.3+?

Problem with filebeat yml file on Windows

I am quite new to the Elastic stack and trying to experiment with visualization of apache log files in Kibana. I am using filebeat to ingest the apache logs. However when I run .\filebeat.exe setup -e, I get the following error:
2019-02-05T20:53:10.515+0530 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-02-05T20:53:10.520+0530 INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.6.0
2019-02-05T20:53:10.520+0530 INFO kibana/client.go:118 Kibana url: http://localhost:5601
2019-02-05T20:53:10.567+0530 WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled
2019-02-05T20:53:10.572+0530 ERROR instance/beat.go:911 Exiting: 1 error: error loading config file: invalid con
fig: yaml: line 4: did not find expected hexdecimal number
My filebeat.yml file looks like this:
filebeat.inputs:
- type: log
enabled: true
paths: C:\Users\bigdataadmin\Downloads\ApacheLogs\*
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: C:\Program Files\Filebeat\modules.d\*.yml
reload.enabled: true
reload.period: 60s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
I also checked the yml on http://www.yamllint.com/ but didn't find any problems. I can't seem to figure out what's wrong with line 4 of this file.
I am using filebeat 6.6
The path key(on line 4) is an array. So you need to represent an array there.
Example :
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\Users\bigdataadmin\Downloads\ApacheLogs\*
Please be very cautious about the data type you are representing in such config files, I had made the same mistake while I was working on Filebeat and I had to spend a lot of time for a small mistake...

ERROR/MainProcess consumer: Cannot connect to amqp:// (Docker Compose and Celery-RabbitMQ)

I'm trying to make Flask, Celery and RabbitMQ work using docker-compose service. I am able to get the things up and running, but the Celery client isn't able to connect to the RabbitMQ server. On the celery server start, I get this message:
[2017-02-27 17:52:17,466: ERROR/MainProcess] consumer: Cannot connect to amqp://radmin:**#174.138.76.93:5672//: [Errno 104] Connection reset by peer.
Trying again in 2.00 seconds...
[2017-02-27 17:52:19,481: ERROR/MainProcess] consumer: Cannot connect to amqp://radmin:**#174.138.76.93:5672//: Socket closed.
Trying again in 4.00 seconds...
My folder structure for the project is:
├── docker-compose.yml
├── order
│   ├── Dockerfile
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── __pycache__
│   │   └── __init__.cpython-35.pyc
│   ├── requirements.txt
│   ├── main.py
├── celery
│   ├── Dockerfile
│   ├── __init__.py
│   ├── requirements.txt
│   └── tasks.py
├── proxy
│   ├── Dockerfile
│   └── order.conf
The order folder is where the main flask server is and the celery has the folder where my celery tasks are present. I'm creating a separate celery worker, wherein the tasks submitted by main.py through
task = celery.send_task('mytasks.outputservice', args=[arg1, arg2])
would be received by tasks.py #app.task(name='mytasks.outputservice')
I've implemented this locally and it works fine (the remote celery tasks thing), I guess the issue is with the RabbitMQ and Celery connections.
My docker-compose file is:
version: '2.1'
services:
# Application
ordermanagement:
restart: always
build: ./ordermanagement
expose:
- "8080"
ports:
- "8080:8080"
volumes:
- /usr/src/app/static
depends_on:
- rabbit
- worker
#env_file: .env
#command: /usr/local/bin/gunicorn -w 2 --timeout 90 --keep- alive 90 --bind :8080 scheduledpurchases:app
# RabbitMQ
rabbit:
hostname: rabbit
image: rabbitmq:management
environment:
- RABBITMQ_DEFAULT_USER=radmin
- RABBITMQ_DEFAULT_PASS=radmin
ports:
- "5672:5672" # we forward this port because it's useful for debugging
- "15672:15672" # here, we can access rabbitmq management plugin
# Nginx server
proxy:
restart: always
build: ./proxy
expose:
- "80"
ports:
- "85:80"
depends_on:
- ordermanagement
links:
- ordermanagement:ordermanagement
# Celery worker
worker:
build: ./ordermanagement_celery
user: nobody
depends_on:
- rabbit
I've seen other questions related to Celery and RabbitMQ, but majority of the problems were either resolved (ip address resolution) or some other small issue.
If someone could give me lead on how to resolve this.
PS: I'm directly deploying to DigitalOcean servers from terminal, checked the ip for RabbitMQ (Celery Broker URL, it is the same as the docker-machine ip).