How does Consul resolve the new redis master ip address after the old redis master get takedown ?
For example :
I did while true; do dig redis.service.google.consul +short; sleep 2; done the response is
192.168.248.43
192.168.248.41
192.168.248.42
192.168.248.41
192.168.248.42
192.168.248.43
...
My expectation is it's only resolve to 192.168.248.41 because it's master. But when the master is down, consul should resolve to 192.168.248.42 or 192.168.248.43, according which one is master
Here is my consul services config in 192.168.248.41
...
"services": [
{
"id": "redis01-xxx",
"name": "redis",
"tags": ["staging"],
"address": "192.168.248.41",
"port": 6379
}
]
...
Here is my consul services config in 192.168.248.42
...
"services": [
{
"id": "redis01-xxx",
"name": "redis",
"tags": ["staging"],
"address": "192.168.248.42",
"port": 6379
}
]
...
And same thing with 192.168.248.43.
My expectation is like this video https://www.youtube.com/watch?v=VHpZ0o8P-Ts
when i do dig, the consul will resolve to only one IP address (master). When the master is down and redis sentinel selects the new master. The consul will resolve to new redis master IP address.
I am very new in Consul. So, i am really appreciate if someone can give short example and suggestion feature of consul, so i can catch it faster.
When you register multiple services with the same "name", they will appear under <name>.service.consul (if they are healthy of course). In your case, you didn't define any health-checks for your redis services, that's why you see all of the service IPs in your DNS query response.
For use-case, where you only need IP of the Redis master, I would recommend registering service with a health check . I'm not familiar how one would query Redis, to find out if the node is the master. But you can use for an example script check, where your script would return 0, if current Redis node was elected master by the Redis sentinel. Consul will automatically recognize that service instance as healthy, and you will see only it's IP in the dig redis.service.consul query to Consul DNS interface.
Related
I've decided to give Nomad a try, and I'm setting up a small environment for side projects in my company.
Although the documentation on Nomad/Consul is nice and detailed, they don't reach the simple task of exposing a small web service to the world.
Following this official tutorial to use Traefik as a load balancer, how can I make those exposed services reachable?
The tutorial has a footnote stating that the services could be accessed from outside the cluster by port 8080.
But in a cluster where I have 3 servers and 3 clients, where should I point my DNS to?
Should a DNS with failover pointing to the 3 clients be enough?
Do I still need a load balancer for the clients?
There are multiple ways you could handle distributing the requests across your servers. Some may be more preferable than the other depending on your deployment environment.
The Fabio load balancer docs have a section on deployment configurations which I'll use as a reference.
Direct with DNS failover
In this model, you could configure DNS to point to the IPs of all three servers. Clients would receive all three IPs back in response to a DNS query, and randomly connect to one of the available instances.
If an IP is unhealthy, the client should retry the request to one of the other IPs, but clients may experience slower response times if a server is unavailable for an extended period of time and the client is occasionally routing requests to that unavailable IP.
You can mitigate this issue by configuring your DNS server to perform health checking of backend instances (assuming it supports it). AWS Route 53 provides this functionality (see Configuring DNS failover). If your DNS server does not support health checking, but provides an API to update records, you can use Consul Terraform Sync to automate adding/removing server IPs as the health of the Fabio instances changes in Consul.
Fabio behind a load balancer
As you mentioned the other option would be to place Fabio behind a load balancer. If you're deploying in the cloud, this could be the cloud provider's LB. The LB would give you better control over traffic routing to Fabio, provide TLS/SSL termination, and other functionality.
If you're on-premises, you could front it with any available load balancer like F5, A10, nginx, Apache Traffic Server, etc. You would need to ensure the LB is deployed in a highly available manner. Some suggestions for doing this are covered in the next section.
Direct with IP failover
Whether you're running Fabio directly on the Internet, or behind a load balancer, you need to make sure the IP which clients are connecting to is highly available.
If you're deploying on-premises, one method for achieving this would be to assign a common loopback IP each of the Fabio servers (e.g., 192.0.2.10), and then use an L2 redundancy protocol like Virtual Router Redundancy Protocol (VRRP) or an L3 routing protocol like BGP to ensure the network routes requests to available instances.
L2 failover
Keepalived is a VRRP daemon for Linux. There can find many tutorials online for installing and configure in.
L3 failover w/ BGP
GoCast is a BGP daemon built on GoBGP which conditionally advertises IPs to the upstream network based on the state of health checks. The author of this tool published a blog post titled BGP based Anycast as a Service which walks through deploying GoCast on Nomad, and configuring it to use Consul for health information.
L3 failover with static IPs
If you're deploying on-premises, a more simple configuration than the two aforementioned solutions might be to configure your router to install/remove static routes based on health checks to your backend instances. Cisco routers support this through their IP SLA feature. This tutorial walks through a basic setup configuration http://www.firewall.cx/cisco-technical-knowledgebase/cisco-routers/813-cisco-router-ipsla-basic.html.
As you can see, there are many ways to configure HA for Fabio or an upstream LB. Its hard to provide a good recommendation without knowing more about your environment. Hopefully one of these suggestions will be useful to you.
In the following case a network of nomad nodes are in 192.168.8.140-250 range and a floating IP 192.168.8.100 are the one that is DNAT from the firewall for 80/443 ports.
The traefik is coupled to a keepalived in the same group. The keepalived will assign its floating ip to the node where traefik is running. There will be only one keepalived in master state.
its not the keepalived "use case" but it's good at broadcast arp when it comes alive.
job "traefik" {
datacenters = ["dc1"]
type = "service"
group "traefik" {
constraint {
operator = "distinct_hosts"
value = "true"
}
volume "traefik_data_le" {
type = "csi"
source = "traefik_data"
read_only = false
attachment_mode = "file-system"
access_mode = "multi-node-multi-writer"
}
network {
port "http" {
static = 80
}
port "https" {
static = 443
}
port "admin" {
static = 8080
}
}
service {
name = "traefik-http"
provider = "nomad"
port = "http"
}
service {
name = "traefik-https"
provider = "nomad"
port = "https"
}
task "keepalived" {
driver = "docker"
env {
KEEPALIVED_VIRTUAL_IPS = "192.168.8.100/24"
KEEPALIVED_UNICAST_PEERS = ""
KEEPALIVED_STATE = "MASTER"
KEEPALIVED_VIRTUAL_ROUTES = ""
}
config {
image = "visibilityspots/keepalived"
network_mode = "host"
privileged = true
cap_add = ["NET_ADMIN", "NET_BROADCAST", "NET_RAW"]
}
}
task "server" {
driver = "docker"
config {
image = "traefik:2.9.6"
network_mode = "host"
ports = ["admin", "http", "https"]
args = [
"--api.dashboard=true",
"--entrypoints.web.address=:${NOMAD_PORT_http}",
"--entrypoints.websecure.address=:${NOMAD_PORT_https}",
"--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
"--certificatesresolvers.letsencryptresolver.acme.email=email#email",
"--certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json",
"--certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web",
"--entrypoints.web.http.redirections.entryPoint.to=websecure",
"--entrypoints.web.http.redirections.entryPoint.scheme=https",
"--providers.nomad=true",
"--providers.nomad.endpoint.address=http://192.168.8.140:4646" ### IP to your nomad server
]
}
volume_mount {
volume = "traefik_data_le"
destination = "/letsencrypt/"
}
}
}
}
For keepalived to run, you should allow some CAP in docker plugin config
plugin "docker" {
config {
allow_privileged = true
allow_caps = [...,"NET_ADMIN","NET_BROADCAST","NET_RAW"]
}
}
I have following configuration to connect with RabbitMQ server. While configuring Sensu , rabbitmq password given in plain text. For security issue, I need to hide or encrypted this password. How can we do it?
File content : /etc/sensu/rabbitmq.json
{
"rabbitmq": {
"host": "127.0.0.1",
"port": 5672,
"vhost": "/sensu",
"user": "sensu",
"password": "**secret**"
}
}
Thanks in advance!!
If your goal is to have encryption in-flight with TLS, you'll want to look into enabling TLS listeners for RabbitMQ.
Sensu's official documentation also shows how to configure Sensu to communicate with a TLS-enabled RabbitMQ.
I installed rabbit mq via docker image on a machine including the management and rabbitmq_auth_backend_ip_range plugins. I want to restrict access to the ports 5671/2 and 15672 to only allow certain IPs accessing them.
As 15672 is the web interface, I have not current solution for that. Any ideas on that?
For 5671/2 (which one is the secure one?) I want to use the plugin rabbitmq_auth_backend_ip_range because as far as I understood, that's its purpose.
My current rabbitmq.config looks like this:
[
{rabbit, [
{auth_backends, [{rabbit_auth_backend_ip_range}]}
]},
{rabbitmq_auth_backend_ip_range, [
{tag_masks,
[{'administrator', [<<"::FFFF:192.168.0.0/112">>]}]
}
]}
].
According to the documentation that allows access only for accounts tagged with administrator. But if I do a telnet nothing changed:
telnet ip-address 5672
I can access it. How do you pass over credentials via telnet? How is ip restriction done with rabbit mq?
rabbitmq-auth-backend-ip-range is only providing authentication mechanism to login/talk to rabbitmq server. That doesn't mean your 5672 port is not open.
You will still be able to telnet on 5672 but if some administrator user tries to connect particularly to RabbitMQ server than it should match with the given IP address otherwise authentication failed will return
For RabbitMQ Management you can define IP address something like this:
{rabbitmq_management, [
{listener, [{port, 15672}, {ip, "127.0.0.1"}]}
]}
Rabbitmq-auth-backend-ip-range link is community plugin for client authorization based on source IP address. With this community plugin, we can restrict access to client on the basis of IP address
Steps To configure plugin in rabbitmq version 3.6.X
wget https://dl.bintray.com/rabbitmq/community-plugins/3.6.x/rabbitmq_auth_backend_ip_range/rabbitmq_auth_backend_ip_range-20180116-3.6.x.zip
unzip content to /usr/lib/rabbitmq/lib/rabbitmq_server-3.x/plugins
Enable plugin:rabbitmq-plugins enable rabbitmq_auth_backend_ip_range
Set a custom tag to which this plugin will block for certain IP address
rabbitmqctl set_user_tags custom_user custom_tag
Configure rabbitmqctl configuration file
vi /etc/rabbitmq/rabbitmq.config
[
{rabbit, [
{tcp_listeners, [5672]},
{auth_backends, [
{rabbit_auth_backend_internal,
[rabbit_auth_backend_internal, rabbit_auth_backend_ip_range]
}
]
}
]},
{rabbitmq_auth_backend_ip_range, [
{tag_masks,
[{'customtag', [<<"::FFFF:172.xx.xx.xxx">>]}]},
{default_masks, [<<"::0/0">>]}
]}
].
this configuration will effect in such a way that the user with tag customtag will able to connect to rabbitmq server with IP address 172.xx.xx.xxx and all other tags can access from any IP address
sudo service rabbitmq-server restart
PS: As there is no valid link online to configure the rabbitmq_auth_backend_ip_range plugin, so I answered this question with the configuration steps
I have a redis server, logstash indexer server, and an elasticsearch server.
How can I have the indexer server or even the shipper servers include the IPs in the log so that it's easier to sort in Kibana?
Or is this something that is done in the elasticsearch config?
When your input log to logstash, logstash will create an event and add hostname to the event. Logstash use hostname instead of IP because one server will have several IP. For example, 127.0.0.1, public IP etc. therefore it doesn't know which IP to use. So logstash use hostname.
Did it. I added this:
filter {
dns {
add_field => [ "IPs", "Logs, from %{host}" ]
}
}
filter {
dns {
type => [ "MESSAGES" ]
resolve => [ "host" ]
action => [ "replace" ]
}
}
Reason why I used a double filter was so that I still was able to keep the hostname after "replace" overwrote the host value to the IP address.
We are implementing scale-out for our SignalR app and trying to avoid a single point of failure in our cluster. Thus, more than one Redis message bus server is required.
The problem with implementing Redis Sentinel is that upon fail-over, a the client needs to connect to a new endpoint [address], which would require the SignalR application to be restarted (Redis endpoint defined in Application_Start()).
Not an option.
I'm trying to understand if/how Booksleeve will help, and would like some explain this.
The issue is that we can only have one single endpoint defined for message bus. A hardware solution is not currently an option.
Would the SignalR application connect to a Booksleeve wrapper, which maintains the list of master/slaves?
Another option using Azure Service Bus. However, the instructions for Wiring Up the Windows Azure Service Bus Provider indicate there are still problems with this:
Note, this web site is an ASP.NET site that runs in an Azure web role.
As of 1.0alpha2 there are some bugs in AzureWebSites due to which
ServiceBus Scale out scenarios do not work well. We are working on
resolving this for the future
I don't know the specifics of how SignalR does the connect, but: BookSleeve already offers some concessions towards failover nodes. In particular, the ConnectionUtils.Connect method takes a string (ideal for web.config configuration values etc), which can include multiple redis nodes, and BookSleeve will then try to locate the most appropriate node to connect to. If the nodes mentioned in the string are regular redis nodes, it will attempt to connect to a master, otherwise falling back to a slave (optionally promoting the slave in the process). If the nodes mentioned are sentinel nodes, it will ask sentinel to nominate a serer to connect to.
What BookSleeve doesn't offer at the moment is a redundant connection wrapper that will automatically reconnect. That is on the road-map, but isn't difficult to do in the calling code. I plan to add more support for this at the same time as implementing redis-cluster support.
But: all that is from a BookSleeve perspective - I can't comment on SignalR specifically.
BookSleeve 1.3.41.0 supports Redis sentinel. Deployment configuration we use: 1 master redis, 1 slave redis. Each box has sentinel (one for master, one for slave). Clients connect to sentinel first, sentinel then redirects them to active master.
This is how it is implemented in client code:
public class OwinStartup
{
public void Configuration(IAppBuilder app)
{
var config = new WebClientRedisScaleoutConfiguration();
GlobalHost.DependencyResolver.UseRedis(config);
app.MapSignalR();
}
}
public class WebClientRedisScaleoutConfiguration : RedisScaleoutConfiguration
{
public WebClientRedisScaleoutConfiguration()
: base(() => getRedisConnection(), WebUIConfiguration.Default.Redis.EventKey)
{ }
private static BookSleeve.RedisConnection _recentConnection;
private static BookSleeve.RedisConnection getRedisConnection()
{
var log = new TraceTextWriter();
var connection = BookSleeve.ConnectionUtils.Connect("sentinel1:23679,sentinel2:23679,serviceName=WebClient", log);
if (connection != null)
{
_recentConnection = connection;
return connection;
}
if (_recentConnection != null)
{
return _recentConnection;
}
// Cannot return null nor throw exception -- this will break reconnection cycle.
return new BookSleeve.RedisConnection(string.Empty);
}
}
Hot to configure redis.
Common steps
Download Redis for windows http://redis.io/download
Unzip to c:\redis
Master (only very first redis box, only one such config)
Create Redis service: execute command within redis directory redis-server --service-install redis.conf --service-name redis
Start Redis service
Ensure Redis is listing port 6379
Slave (other boxes)
Update redis.conf: add line slaveof masterAddr 6379 where masterAddr
is address where redis in master mode is running, 6379 is default
redis port.
Create Redis service: execute command within redis directory redis-server --service-install redis.conf --service-name redis
Start Redis service
Ensure Redis is listing port 6379
Sentinel (common for master and slave)
Create file redis-sentinel.conf with content:
port 26379
logfile "redis-sentinel1.log"
sentinel monitor WebClient masterAddr 6379 1
where masterAddr is address where redis in master mode is running,
6379 is default redis port, 1 is quorum (number of host that makes
decision is server down or not). WebClient is group name. You specify it in client code ConnectionUtils.Connect("...,serviceName=WebClient...")
Create redis sentinel service:execute command within redis directory redis-server --service-install redis-sentinel.conf --service-name redis-sentinel --sentinel
Start redis-sentinel service