How to use Traefik and Envoy in the same project to handle grpc-web? - traefik

Question

I've got a service up and running on traefik with LetsEncrypt at grpc.mydomain.com. However, traefik doesn't support routing grpc-web request due to some issue with CORS (https://github.com/containous/traefik/issues/4210). Envoy appears to be an alternative to traefik which works with grpc-web, but I don't want to go about reconfiguring everything.
If I put envoy at envoy.mydomain.com then it actually hits traefik first and traefik can't route the grpc-web requests to envoy. So this doesn't work.
If I put envoy outside of traefik (mydomain.com:9091) then envoy doesn't have the TLS support that traefik has.
Do I need to switch everything to envoy? Is there an alternative I haven't considered? Any guidance welcome :)
Current Traefik Setup:
traefik:
image: traefik:v2.0.0
container_name: traefik
command:
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.grpc.address=:8090
- --providers.docker
- --api
- --serversTransport.rootCAs=/certs/grpc.cert
# Lets Encrypt Resolvers
- --certificatesresolvers.leresolver.acme.email=${EMAIL}
- --certificatesresolvers.leresolver.acme.storage=/etc/acme/cert.json
- --certificatesresolvers.leresolver.acme.tlschallenge=${TLS_CHALLENGE}
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/acme/:/etc/acme/
- ./secrets/grpc.cert:/certs/grpc.cert
# Dynamic Configuration
labels:
# Dashboard
- "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)"
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.tls.certresolver=leresolver"
- "traefik.http.routers.traefik.entrypoints=websecure"
- "traefik.http.routers.traefik.middlewares=authtraefik"
# https://docs.traefik.io/middlewares/basicauth/
# password generated from `echo $(htpasswd -nb admin $PASSWORD) | sed -e s/\\$/\\$\\$/g`
- "traefik.http.middlewares.authtraefik.basicauth.users=admin:$$apr1$$6VzI3S0N$$29FC82dYEbjFN9tPSfWLX1"
# global redirect to https
- "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.entrypoints=web"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
# middleware redirect
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
ports:
- 80:80
- 443:443
- 8090:8090
networks:
- internal
- proxied
grpc_server:
image: ${GRPC_IMAGE}
container_name: grpc_server
volumes:
- /tmp/keyset.json:/tmp/keyset.json
- ./secrets/:/secrets/
working_dir: /app/__main__/
labels:
- "traefik.http.routers.combined_server.rule=Host(`grpc.${DOMAIN}`)"
- "traefik.http.routers.combined_server.entrypoints=grpc"
- "traefik.http.routers.combined_server.tls=true"
- "traefik.http.routers.combined_server.tls.certresolver=leresolver"
# http
- "traefik.http.services.grpc-svc.loadbalancer.server.scheme=h2c"
- "traefik.http.services.grpc-svc.loadbalancer.server.port=8090"
expose:
- 8090
networks:
- internal
- proxied
I also tried setting these to fix the CORS error but got nowhere.
- "traefik.http.middlewares.testheader.headers.accesscontrolallowmethods=GET,PUT,DELETE,POST,OPTIONS"
- "traefik.http.middlewares.testheader.headers.accesscontrolallowheaders=keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout"
- "traefik.http.middlewares.testheader.headers.accesscontrolmaxage=100"
- "traefik.http.middlewares.testheader.headers.addvaryheader=true"
- "traefik.http.middlewares.testheader.headers.alloworigin=*"

Answer 1

A way to approach this issue is by using 2 different URLs that are both being handled by traefik at first. One URL is being used for "direct grpc" (grpc.mydomain.com), the other one for grpc-web (let's call it grpc-web.mydomain.com). Traefik does TLS termination for both.
The grpc.mydomain.com traffic is directly passed to the container running the grpc_server. The grpc-web.mydomain.com traffic is passed to envoy which acts as a grpc-web-proxy and then passes the traffic to the grpc_server.
So as you are using docker-compose, you would need to add an envoy service to your docker-compose.yml:
---
version: '3'
services:
traefik:
# traefik configuration from your question
# ...
grpc-server:
# grpc_server configuration from your question
# ...
envoy:
image: envoyproxy/envoy:v1.14.1
restart: unless-stopped
volumes:
- ./envoy.yaml:/etc/envoy/envoy.yaml
labels:
- traefik.enable=true
- traefik.http.routers.envoy.rule=Host(`grpc-web.mydomain.com`)
- traefik.http.services.envoy.loadbalancer.server.port=8080
- traefik.http.routers.envoy.tls=true
- traefik.http.routers.envoy.tls.certresolver=leresolver
The envoy.yaml configuration (mounted in the volumes section above) looks like this:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: grpc_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.grpc_web
- name: envoy.cors
- name: envoy.router
clusters:
- name: grpc_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{ socket_address: { address: grpc-server, port_value: 8090 }}]
This is a pretty basic grpc-web config for envoy. The important part to notice is that we set address: grpc-server, port_value: 8090 in the configuration of the "grpc_service" cluster configuration to the service name from the docker-compose.yml and to the port your grpc-server is listening on. Please note I renamed your service from grpc_server to grpc-server as the underscore is not a valid charater in hostnames.
On the client side, use:
"grpc-web.mydomain.com" in your javascript (grpc-web) code.
"grpc.mydomain.com" when writing a client in another language (like golang for example).
I created a working example, which can be found under: https://github.com/rbicker/greeter

Answer 2

If you want to get rid of deprecated warnings in envoy, you can update envoy.yaml from this answer with those three changes:
replace:
- name: envoy.http_connection_manager
config:
with:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
replace
- name: envoy.grpc_web
- name: envoy.cors
- name: envoy.router
with
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
replace
hosts: [{ socket_address: { address: grpc-server, port_value: 8090 }}]
with
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: grpc-server
port_value: 8090