I recently went through the tutorial for load balancing apps in DCOS using marathon-lb (in the example they balance some nginx containers: https://dcos.io/docs/1.9/networking/marathon-lb/marathon-lb-advanced-tutorial/). I am trying to use this approach to internally load balance my own custom application. The custom app I am using is a play scala app. I have the internal marathon-lb set up and can successfully use it for the nginx container but when I try to use my own docker image I cannot get this to work. I start up my service with my custom image and I can access the service fine by using the IP and port that gets assigned to it (i.e. if the service gets deployed on 10.0.0.0 and is available on port 1234 then curl http://10.0.0.0:1234/ works as expected and I can also make my api calls as defined in my application routes). However, when I try to access the app through the load balancer (curl -i http://marathon-lb-internal.marathon.mesos:10002, where 10002 is the service port) then I get this message:
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
For reference, here is my json file I'm using to start my custom service:
{
"id": "my-app",
"container": {
"type": "DOCKER",
"docker": {
"image": "my_repo/my_image:1.0.0",
"network": "BRIDGE",
"portMappings": [
{ "hostPort": 0, "containerPort": 9000, "servicePort": 10002, "protocol": "tcp" }
],
"parameters": [
{ "key": "env", "value": "USER_NAME=user" },
{ "key": "env", "value": "USER_PASSWORD=password" }
],
"forcePullImage": true
}
},
"instances": 1,
"cpus": 1,
"mem": 1000,
"healthChecks": [{
"protocol": "HTTP",
"path": "/v1/health",
"portIndex": 0,
"timeoutSeconds": 10,
"gracePeriodSeconds": 10,
"intervalSeconds": 2,
"maxConsecutiveFailures": 10
}],
"labels":{
"HAPROXY_GROUP":"internal"
},
"uris": [ "https://s3.amazonaws.com/my_bucket/my_docker_credentials" ]
}
I had the same problem and found the solution here
marathon-lb health check failing on all spray.io containers
Need to add
"HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": " http-send-name-header Host\n timeout check {healthCheckTimeoutSeconds}s\n"
To your config so that the REST layer doesn't bark on the health check from marathon
Related
I am using AspNetCoreRateLimit version 4.0.1 and I have done all the setup in .net core 6 web api. I can see rate limit is working when I send a call via postman.
However, when I add IpRateLimitPolicies with specific IP address, the settings won't be applied.
I use postman and this time in the proxy I added the ip address to 127.0.0.1. I can see the ip hitting the api is set correctly when I use Request.HttpContext.Connection.RemoteIpAddress;
I registered them as follow in program.cs:
_serviceCollection.AddOptions();
_serviceCollection.AddMemoryCache();
_serviceCollection.Configure<IpRateLimitOptions>(builder.Configuration.GetSection("IpRateLimiting"));
_serviceCollection.Configure<IpRateLimitPolicies>(builder.Configuration.GetSection("IpRateLimitPolicies"));
_serviceCollection.AddInMemoryRateLimiting();
_serviceCollection.AddSingleton<IIpPolicyStore, MemoryCacheIpPolicyStore>();
_serviceCollection.AddSingleton<IRateLimitCounterStore, MemoryCacheRateLimitCounterStore>();
_serviceCollection.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
_serviceCollection.AddSingleton<IRateLimitConfiguration, RateLimitConfiguration>();
Also added:
app.UseIpRateLimiting();
My appsettings also looks like:
{
"IpRateLimiting": {
"EnableEndpointRateLimiting": false,
"StackBlockedRequests": false,
"RealIPHeader": "X-Real-IP",
"ClientIdHeader": "X-ClientId",
"IpWhitelist": [ ],
"EndpointWhitelist": [],
"ClientWhitelist": [],
"HttpStatusCode": 429,
"GeneralRules": [
{
"Endpoint": "*",
"Period": "10s",
"Limit": 1
}
]
},
"IpRateLimitPolicies": {
"IpRules": [
{
"Ip": "127.0.0.1",
"Rules": [
{
"Endpoint": "*",
"Period": "20s",
"Limit": 2
}
]
}
]
}
But apparently the settings under IpRateLimitPolicies won't be applied.
I wonder if I have missed anything here?
Thank you
After testing, I think the AspNetCoreRateLimit package is not compatible in .net5 and .net6. Maybe the .net core3.1 version will be more stable.
You can submit issues on github.
Apparently I have missed some configuration in program.cs or startup.cs:
https://github.com/stefanprodan/AspNetCoreRateLimit/issues/305
Since we are using startup I have added the following in Configure method
var ipPolicyStore = app.ApplicationServices.GetRequiredService();
ipPolicyStore.SeedAsync().GetAwaiter().GetResult();
I'm trying to use GoDaddy's API to create a subdomain using the following http request:
PATCH /v1/domains/domainName.com/records
Host: api.ote-godaddy.com
Authorization: sso-key API_KEY:API_SECRET
Content-Type: application/json
Content-Length: 100
[
{
"data": "111.111.111.111",
"name": "subdomainName",
"ttl": 6000,
"type": "A"
}
]
but I get the following response:
{
"code": "UNKNOWN_DOMAIN",
"message": "The given domain is not registered, or does not have a zone file"
}
Please changes the host name as https://api.godaddy.com. your request will be work only production URL.
Please generate Production level API Key & SECRET.
Body: (Raw - Json Type)
[
{
"data": "YourServerIp",
"name": "subdomainName",
"port": 80,
"priority": 10,
"protocol": "string",
"service": "string",
"ttl": 600,
"type": "A"
}
]
Note:
It's only happened when primary domain already exists on your go daddy account
I figured out these requests only work using the production authorization against the production url but won't work using ote-authorization against the ote url. Maybe the url has to be set as an ote domain and not a production domain. Not sure.
Hi I was trying to implement ocelot for our experimental tests on dev.
Here is end-point of api that I want to reach by via ocelot. using 443 port for both of project.
but getting 502 bad gateway all the time.
end point => https://localhost/document/api/v1/Documents/XYZ
"ReRoutes": [
{
"DownstreamPathTemplate": "/document/api/v1/Documents/{name}",
"DownstreamScheme": "https",
"DownstreamHostAndPorts": [
{
"Host": "localhost",
"Port": 443
}
],
"UpstreamPathTemplate": "/apigateway/{name}/document",
"UpstreamHttpMethod": [ "Post" ],
"Priority": 0
}
],
"GlobalConfiguration": {
"BaseUrl": "https://localhost:443"
}
}
Microgateway alias name =>"apigateway"
Api alias name => "document"
In addition this I was able to debug on visiual studio but whenever I host both app on my local IIS getting 502 bad gateway
It appears that the configuration you have used is redirecting the request to the gateway itself, resulting in a circle.
i.e. the upstream call to the base URL of "localhost:443" is redirecting to the downstream "localhost:443" - the same.
Furthermore, the later versions of Ocelot appear to look for Routes in the configuration instead of ReRoutes documentation
I have been playing around with load-testing my application on a single agent cluster in AKS. During the testing, the connection to the dashboard stalled and never resumed. My application seems down as well, so I am assuming the cluster is in a bad state.
The API server is restate-f4cbd3d9.hcp.centralus.azmk8s.io
kubectl cluster-info dump shows the following error:
{
"name": "kube-dns-v20-6c8f7f988b-9wpx9.14fbbbd6bf60f0cf",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/events/kube-dns-v20-6c8f7f988b-9wpx9.14fbbbd6bf60f0cf",
"uid": "47f57d3c-d577-11e7-88d4-0a58ac1f0249",
"resourceVersion": "185572",
"creationTimestamp": "2017-11-30T02:36:34Z",
"InvolvedObject": {
"Kind": "Pod",
"Namespace": "kube-system",
"Name": "kube-dns-v20-6c8f7f988b-9wpx9",
"UID": "9d2b20f2-d3f5-11e7-88d4-0a58ac1f0249",
"APIVersion": "v1",
"ResourceVersion": "299",
"FieldPath": "spec.containers{kubedns}"
},
"Reason": "Unhealthy",
"Message": "Liveness probe failed: Get http://10.244.0.4:8080/healthz-kubedns: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",
"Source": {
"Component": "kubelet",
"Host": "aks-agentpool-34912234-0"
},
"FirstTimestamp": "2017-11-30T02:23:50Z",
"LastTimestamp": "2017-11-30T02:59:00Z",
"Count": 6,
"Type": "Warning"
}
As well as some Pod Sync errors in Kube-System.
Example of issue:
az aks browse -g REstate.Server -n REstate
Merged "REstate" as current context in C:\Users\User\AppData\Local\Temp\tmp29d0conq
Proxy running on http://127.0.0.1:8001/
Press CTRL+C to close the tunnel...
error: error upgrading connection: error dialing backend: dial tcp 10.240.0.4:10250: getsockopt: connection timed out
You'll probably need to ssh to the node to see if the Kubelet service is running. For future you can set Resource quotas from exhausting all resources in the cluster nodes.
Resource Quotas -https://kubernetes.io/docs/concepts/policy/resource-quotas/
I have a web app on OpenShift v3 (all-in-One), using the Wildfly Builder Image. In addition, I created a service named xyz, to point to an external host+IP. Something like this:
"kind": "Service",
"apiVersion": "v1",
"metadata": { "name": "xyz" },
"spec": {
"ports": [
{ "port": 61616,
"protocol": "TCP",
"targetPort": 61616
}
],
"selector": {}
}
I also have an endpoint, pointing externally, but that is not relevant for this question.
When deployed, my program can access an environment variable named XYZ_PORT=tcp://172.30.192.186:61616
However, I cannot figure out how to see all the values of all such variables either via the web-console, or using the CLI. Using the web-console, I cannot see it being injected into the YAML.
I tried some of the oc env options, but none seem to list what I want.
Let's say you are deploying kitchensink, then the below CLI should list all the environment variables:
oc env bc/kitchensink --list