Apache (ec2) --- Client (ELB)
| |
|-------[1.]FIN------->|
| |
|<-----[2.]FIN+ACK-----|
| |
|---------ACK--------->|
| |
With Wireshark I'd like to extract only the packet "[1.]FIN" described above figure which is emitted by server's 80 port and which "initiates" FIN sequence.
I've tried a filter:
tcp.flags.fin && tcp.srcport==80
but the filter also extracts the extra "[2.]FIN+ACK" packets.
How can I filter out only [1.] packet considering "FIN" sequence initiator?
Background:
I'm struggling to get rid of 504 errors with AWS ELB and ec2 (apache), where "FIN - FIN/ACK - ACK" sequence is initiated by the backend apache-side. In such environment FIN sequence initiated by ELB is ideal as AWS official sais: http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html
According to https://aws.amazon.com/jp/premiumsupport/knowledge-center/504-error-classic/, I've tried changing replace MPM (event -> worker) and disabling TCP_DEFER_ACCEPT, which slightly reduced 504 errors. However the situation is not much improved.
The point I think will be to find the cause which makes apache initiate active-close sequence, thus I'm firstly trying to extract initiating FIN packet from apache among at most 512 parallel connections between ELB and EC2 (apache).
tcp.flags.fin == 1 && tcp.flags.ack == 0
A filter such as tcp.flags.fin only checks for the presence of the parameter. To find certain values of a parameter, a comparison is needed. That is why filters like "tcp" work to find TCP packets.
The filter match for FIN does not exclude other flags being set or not set, so a comparison is needed for each flag that should be part of the filter.
Related
I'm using prometheus to detect pauses in inbound traffic for an API endpoint. The query for this seems to be working nicely: alert if there's no change in HTTP requests for a given endpoint over a one-hour time period:
sum by(path, custom_identifier, kubernetes_namespace) (
delta(http_request_duration_seconds_count{
status_code="201",
method="POST",
path="/api/v1/path/to/endpoint"}[1h])
) == 0
What I want, though, is for this alert to resolve immediately upon resumption of traffic. As currently written, it won't resolve until traffic has been restored for the same time period - 1h.
Is this a thing that Prometheus can do? Is there a different method of accomplishing this goal?
- alert: AlertName
expr: sum by(path, custom_identifier, kubernetes_namespace) (
delta(http_request_duration_seconds_count{
status_code="201",
method="POST",
path="/api/v1/path/to/endpoint"}[1m])) == 0 #recovers after 1min if resolved
for: 1h
For testing, I wanted to shrink my 3 node cluster to 2 nodes, to later go and do the same thing for my 5 node cluster.
However, after following the best practice of shrinking a cluster:
Back up all tables
For all tables: alter table xyz set (number_of_replicas=2) if it was less than 2 before
SET GLOBAL PERSISTENT discovery.zen.minimum_master_nodes = <half of the cluster + 1>;
3 a. If the data check should always be green, set the min_availability to 'full':
https://crate.io/docs/reference/configuration.html#graceful-stop
Initiate graceful stop on one node
Wait for the data check to turn green
Repeat from 3.
When done, persist the node configurations in crate.yml:
gateway.recover_after_nodes: n
discovery.zen.minimum_master_nodes:[![enter image description here][1]][1] (n/2) +1
gateway.expected_nodes: n
My cluster never went back to "green" again, and I also have a critical node check failing.
What went wrong here?
crate.yml:
...
################################## Discovery ##################################
# Discovery infrastructure ensures nodes can be found within a cluster
# and master node is elected. Multicast discovery is the default.
# Set to ensure a node sees M other master eligible nodes to be considered
# operational within the cluster. Its recommended to set it to a higher value
# than 1 when running more than 2 nodes in the cluster.
#
# We highly recommend to set the minimum master nodes as follows:
# minimum_master_nodes: (N / 2) + 1 where N is the cluster size
# That will ensure a full recovery of the cluster state.
#
discovery.zen.minimum_master_nodes: 2
# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
# discovery.zen.ping.timeout: 3s
#
# Time a node is waiting for responses from other nodes to a published
# cluster state.
#
# discovery.zen.publish_timeout: 30s
# Unicast discovery allows to explicitly control which nodes will be used
# to discover the cluster. It can be used when multicast is not present,
# or to restrict the cluster communication-wise.
# For example, Amazon Web Services doesn't support multicast discovery.
# Therefore, you need to specify the instances you want to connect to a
# cluster as described in the following steps:
#
# 1. Disable multicast discovery (enabled by default):
#
discovery.zen.ping.multicast.enabled: false
#
# 2. Configure an initial list of master nodes in the cluster
# to perform discovery when new nodes (master or data) are started:
#
# If you want to debug the discovery process, you can set a logger in
# 'config/logging.yml' to help you doing so.
#
################################### Gateway ###################################
# The gateway persists cluster meta data on disk every time the meta data
# changes. This data is stored persistently across full cluster restarts
# and recovered after nodes are started again.
# Defines the number of nodes that need to be started before any cluster
# state recovery will start.
#
gateway.recover_after_nodes: 3
# Defines the time to wait before starting the recovery once the number
# of nodes defined in gateway.recover_after_nodes are started.
#
#gateway.recover_after_time: 5m
# Defines how many nodes should be waited for until the cluster state is
# recovered immediately. The value should be equal to the number of nodes
# in the cluster.
#
gateway.expected_nodes: 3
So there are two things that are important:
The number of replicas is essentially the number of nodes you can loose in a typical setup (2 is recommended so that you can scale down AND loose a node in the process and still be ok)
The procedure is recommended for clusters > 2 nodes ;)
CrateDB will automatically distribute the shards across the cluster in a way that no replica and primary share a node. If that is not possible (which is the case if you have 2 nodes and 1 primary with 2 replicas, the data check will never return to 'green'. So in your case, set the number of replicas to 1 in order to get the cluster back to green (alter table mytable set (number_of_replicas = 1)).
The critical node check is due to the cluster not having received an updated crate.yml yet: Your file also still has the configuration of a 3-node cluster in it, hence the message. Since CrateDB only loads the expected_nodes at startup (it's not a runtime setting), a restart of the whole cluster is required to conclude scaling down. It can be done with a rolling restart, but be sure to set SET GLOBAL PERSISTENT discovery.zen.minimum_master_nodes = <half of the cluster + 1>; properly, otherwise the consensus will not work...
Also, it's recommended to scale down one-by-one in order to avoid overloading the cluster with rebalancing and accidentally loosing data.
I am working on Mosquitto and plan to use Redis as the back end to handle both username/password pair authentication and ACL. I am using JPmens' authentication plugin to do this. The authentication works well, but I can't make the ACL work. Redis uses unique keys and the usernames (keys in my cases) are used in username/password pairs for authentication purposes. I have tried to mix user name,password and topics together in sets/list, but none of them work.
the mosquitto conf:
auth_plugin /etc/mosquitto/auth-plug.so
auth_opt_backends redis
auth_opt_redis_host 127.0.0.1
auth_opt_redis_port 6379
auth_opt_redis_userquery GET %s
auth_opt_redis_aclquery GET %s-%s
Following name/password pairs are working fine for the authentication
SET user1 PBKDF2$sha256$901$Qh18ysY4wstXoHhk$g8d2aDzbz3rYztvJiO3dsV698jzECxSg
SET user2 PBKDF2$sha256$901$R74X2ae3MufMS20M$CAbXZFDmXJN7Cc28Dm/Z97OfM8Tz1JHn
...
Following settings won't work for the ACL: (a/b... as topics)
sadd user22 PBKDF2$sha256$901$Qh18ysY4wstXoHhk$g8d2aDzbz3rYztvJiO3dsV698jzECxSg a/b c/d
rpush user33 PBKDF2$sha256$901$q5/N74O6Iaf/e8Cg$dEA3tZSi/sJeXKAkX39Gd3agy2WY96gE e/f
What's the correct way to do so?
In the Redis API, aclrequery shows that:
Single stepping until exit from function be_redis_aclcheck, which has no line number information.
redisCommand (c=0x6537d0, format=0x6561c0 "GET user1-t/c") at hiredis.c:1345
1345 void *redisCommand(redisContext *c, const char *format, ...) {
(gdb) bt
0 redisCommand (c=0x6537d0, format=0x6561c0 "GET my-t/c") at hiredis.c:1345
1 0x00007ffff5e61376 in be_redis_aclcheck () from /etc/mosquitto/auth-plug.so
2 0x00007ffff5e5c351 in mosquitto_auth_acl_check ()
from /etc/mosquitto/auth-plug.so
Here, user1 is the user name and t/c is the topic. GET user1-t/c seems to tell me a string type is expected in the Redis database. Can anyone give me an example of how to get this to work?
Thanks
I have figured out how it works. If MQTT broker only allow client user1 to pub and sub "a/b" and "c/d" topics, the correct ACL data in Redis for the JPmens plugin will be:
user1-a/b 2
user1-c/d 2
"user1-a/c" is the key and 2 is the value.
It's not preferred, if Redis goes down for any reason your entire system will be down also.
It will be a SPF (single point of failure) in your architecture.
we're going to use rabbitmq in our project, but facing a problem that, we want to debug on our dev machines, so the response message have to be send to machine which originally send the request message out. How we're going to achive that, is there an existing solution in spring-rabbitmq framework?
We have considered several solutions. such as declare a set of queues for each machine, the queue name prefix by machine name. Is that feasible?
Define set of queues (debug queue A-Z) and bind them to headers exchange with attributes x-match=any, from=[A-Z], to=[A-Z] respectively to . Then bind headers exchange to you main working exchange (one or more) to receive all messages you interested in, so when your consumer publish response it will be duplicated to your debug exchange and then routed to appropriate queue.
[sender X] [ worker ] [consumer on queue X]
| ^ |
[request] | [response from=X, to=X] [duped request from=X|
\ | | [duplicated response from=X, to=X]
\ [request from=X] | ^
v | V |
[working topic exchange] -------> [debug headers exchange]
/ | \ / | \
{bindings by routing key mask} {bindings by any headers from=[A-Z], to=[A-Z]}
/ | \ / | \
[working queue 1] ... [working queue N] [debug queue A] ... [debug queue Z]
To bind request and response messages you can use applicationId and correlationId message attributes.
Note, that both request and response messages will be duplicated to debug queues. You may also specify separate queue for request and response messages by binding queues to match only specific headers, something like x-match=all, from=[A-Z] or x-match=all, to=[A-Z] and publish response and request messages with only that headers (only from or only to), but it is up to you.
The pros:
easy to implement
requires minimal code changes
easy to turn on/off
may be safely run in production environment
Cons:
use more resource power from RabbitMQ side
Alternatively, you can utilize RPC pattern somehow if you debugging process requires realtime response receiving. But this will block publisher until response processed, which may differ from real-world app usage and break business logic.
Pros:
step-by-step debugging process
Cons:
hard to implement
may require a lot of code changes
break business logic
hard to enable/disable
not production environment safe
p.s.: sorry for ascii graph
I have created a RAW socket and set the socket option as "IP_HDRINCL". I am setting the identification field in the outgoing UDP packet and
sending the same out. I see that this field is changed to a random number when the packet is sent out. I can see the same in the sniffer capture.
The same code when built and tested in a different Windows XP machine, works fine ie. I can see that the identification field in the IP header of the outgoing packet carries the same value that I have set.
Not sure whats going wrong. Please let me if you have any insights.
Thanks,
Renu
Raw sockets with IP_HDRINCL automatically set the values of certain field of ip packets that are going to get sent out.
In your case, the packet id will get overwritten in case is zero. (is it your case?)
This are the cases: (taken from Raw Socket man page)
+---------------------------------------------------+
|IP Header fields modified on sending by IP_HDRINCL |
+----------------------+----------------------------+
|IP Checksum |Always filled in. |
+----------------------+----------------------------+
|Source Address |Filled in when zero. |
+----------------------+----------------------------+
|Packet Id |Filled in when zero. |
+----------------------+----------------------------+
|Total Length |Always filled in. |
+----------------------+----------------------------+