Geode Redis Adaptor - gemfire

Hi all, hoping someone can assist me with some queries/configuration for the use of the Geode Redis Adapter. I'm having some difficulty ascertaining how/whether I can configure a number of Redis servers within my Geode cluster to function in a high availability setup.
I'm very new to Geode, but understand that in a traditional Geode application, the client interacts with a locator process to access data from the cluster and balance load. Given that the aim of this adapter is to function as a drop-in replacement for Redis (i.e. no change required on the client) I imagine it functions somewhat differently.
Here is what I have tried so far:
I have built from source according to this link and successfully got the gfsh cli up on 2 CentOS 7 VMs:
192.168.0.10: host1
192.168.0.15: host2
On host1, I run the following commands:
gfsh>start locator --name=locator1 --bind-address=192.168.0.10 --port=10334
gfsh>start server --name=redis --redis-bind-address=192.168.0.10 --redis-port=11211 --J=-Dgemfireredis.regiontype=PARTITION_REDUNDANT
On host2, I run the following command:
gfsh>start server --name=redis2 --redis-bind-address=192.168.0.15 --redis-port=11211 --J=-Dgemfireredis.regiontype=PARTITION_REDUNDANT --locators=192.168.0.10[10334]
On host1, I examine the current configuration:
gfsh>list members
Name | Id
-------- | -------------------------------------------------
locator1 | 192.168.0.10(locator1:16629:locator)<ec><v0>:1024
redis2 | 192.168.0.15(redis2:6022)<ec><v2>:1024
redis | 192.168.0.10(redis:16720)<ec><v1>:1025
gfsh>list regions
List of regions
-----------------
__HlL
__ReDiS_MeTa_DaTa
__StRiNgS
For each of the regions, I can see both server redis & redis2 as Hosting Members - e.g.
gfsh>describe region --name=__StRiNgS
..........................................................
Name : __StRiNgS
Data Policy : normal
Hosting Members : redis2
redis
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ----- | -----
Region | size | 0
| scope | local
At this point, I turned to the redis-cli for some testing. Given the previous output, my expectation was that if I set a key on one server, I should be able to read it back from the other server:
192.168.0.10:11211> set foo 'bar'
192.168.0.10:11211> get foo
"bar"
192.168.0.15:11211> get foo
(nil)
gfsh>describe region --name=__StRiNgS
..........................................................
Name : __StRiNgS
Data Policy : normal
Hosting Members : redis2
redis
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ----- | -----
Region | scope | local
Non-Default Attributes Specific To The Hosting Members
Member | Type | Name | Value
------ | ------ | ---- | -----
redis2 | Region | size | 0
redis | Region | size | 1
As you can see, a query against host2 for the key added on host1 returned (nil). I'd greatly appreciate any help here. Is it possible achieve what I'm aiming for here or does the Redis adapter only allow you to scale out a single server?

This may not be an answer, but it is probably too long for a comment.
I am not familiar with the specific Geode Redis Adapter you are talking about here. But from my experience with Gemfire/Geode, there are things you may want to check:
You started the first host without locators param, I am not sure whether this will cause any problem with cluster formation. There are two ways in Gemfire to form a cluster: by specifying mcast port or by specifying locators.
Scope of the region you are inspecting looks wrong. "local" will not replicate any updates. When you set it up correctly, it should show up as DISTRIBUTED_NO_ACK / DISTRIBUTED_ACK / GLOBAL I suppose.
Hope this helps

Xiawei is correct, a scope "local" regions will not replicate the entry on other members. The workaround for this could have been to just create a region named __StRiNgS from gfsh, but since region names starting with two underscores are for internal use only, that's not possible.
I have filed this issue https://issues.apache.org/jira/browse/GEODE-1921 to fix the problem. I also attached a patch for this issue. With the patch applied I see that the __StRiNgS region now is PARTITION.
gfsh>start locator --name=locator1
gfsh>start server --name=redis --redis-port=11211
gfsh>list regions
List of regions
-----------------
HlL
StRiNgS
__ReDiS_MeTa_DaTa
gfsh>describe region --name=/StRiNgS
..........................................................
Name : StRiNgS
Data Policy : partition
Hosting Members : redis
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ----------- | ---------
Region | size | 0
| data-policy | PARTITION

On host1:
start locator --name=locator1 --bind-address=192.168.0.10 --port=10334
start server --name=redis --redis-bind-address=192.168.0.10 --redis-port=11211 --J=-Dgemfireredis.regiontype=REPLICATE
NOTE: Yoy have to use regiontype as "REPLICATE" if you wish the data to be replicated from one region to another.
On host2:
start server --name=redis2 --redis-bind-address=192.168.0.15 --redis-port=11211 --J=-Dgemfireredis.regiontype=REPLICATE --locators=192.168.0.10[10334]
https://geode.apache.org/docs/guide/11/developing/region_options/region_types.html

Related

How to change the default number of pods per node?

I'm playing around with AWS EKS and I've set up a cluster with a t3.small node. Somehow the number of allocatable pods is set to 8:
Allocatable:
cpu: 2
ephemeral-storage: 19316009748
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 1902056Ki
pods: 8
It seems to me that the resource request on the node is on the low side
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
310m (15%) 0 (0%) 140Mi (7%) 340Mi (18%)
and so I'd like to ask:
Is 8 a default value?
How can I change that?
Thanks.
The calculation of the max number of pods in a given node in AWS is defined by this formula:
Max Pods = (Maximum Network Interfaces for instance type) * (IPv4 Addresses per Interface) - 1
Since you used t3.small, the value comes down to 3 * 4 - 1 = 11. The number 11 also depends upon the compute resources you're associating with each pod.
The list of network interfaces are listed here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

Wireshark filter for packets which initiates FIN (connection close) sequence from the server-side

Apache (ec2) --- Client (ELB)
| |
|-------[1.]FIN------->|
| |
|<-----[2.]FIN+ACK-----|
| |
|---------ACK--------->|
| |
With Wireshark I'd like to extract only the packet "[1.]FIN" described above figure which is emitted by server's 80 port and which "initiates" FIN sequence.
I've tried a filter:
tcp.flags.fin && tcp.srcport==80
but the filter also extracts the extra "[2.]FIN+ACK" packets.
How can I filter out only [1.] packet considering "FIN" sequence initiator?
Background:
I'm struggling to get rid of 504 errors with AWS ELB and ec2 (apache), where "FIN - FIN/ACK - ACK" sequence is initiated by the backend apache-side. In such environment FIN sequence initiated by ELB is ideal as AWS official sais: http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html
According to https://aws.amazon.com/jp/premiumsupport/knowledge-center/504-error-classic/, I've tried changing replace MPM (event -> worker) and disabling TCP_DEFER_ACCEPT, which slightly reduced 504 errors. However the situation is not much improved.
The point I think will be to find the cause which makes apache initiate active-close sequence, thus I'm firstly trying to extract initiating FIN packet from apache among at most 512 parallel connections between ELB and EC2 (apache).
tcp.flags.fin == 1 && tcp.flags.ack == 0
A filter such as tcp.flags.fin only checks for the presence of the parameter. To find certain values of a parameter, a comparison is needed. That is why filters like "tcp" work to find TCP packets.
The filter match for FIN does not exclude other flags being set or not set, so a comparison is needed for each flag that should be part of the filter.

Mixing fanout and direct exchanges with AMQP

I have two kinds of workers for a same event.
I would like a message be dispatched to only one among some of my workers (like "direct" exchanges). But the other workers should all process the message (like fanout).
It's a bit hard to explain but the idea is here. And maybe the following schema will help you to understand what I would like.
Do you have a solution?
Kind regards,
Ben
If I understand you correctly, you would like to have "type-1" as worker where only a single worker work on an item while "type-2" can be treated as multiple handlers (like log-handlers) where all should accept the event.
If i'm right , then you might be able to chain two queues (exchanges) .
exchange1 - fanout - all the "type-2" (loggers) will wait here so they will all get the event
exchange2 - direct - all your "type-1" will wait here so only one will get the event
the trick - you need to make sure that you have a consumer listening on "exchange1" that will also publish to "exchange2".
your best option is to use routing keys with multiple bindings between your exchange and your queues.
i would recommend either direct or topic exchange for this, but not fanout.
to model your example image above, your configuration would look like this:
| exchange | routing key | queue |
|----------|-------------|---------|
| some.ex | type.1 | queue.1 |
| some.ex | type.1 | queue.2 |
| some.ex | type.1 | queue.3 |
| some.ex | type.2 | queue.4 |
| some.ex | type.2 | queue.5 |
basically, you need to have a routing key per queue and a queue per worker.
you may want to read up a bit more on exchanges, queues and bindings, to get a better understanding of when they would be used. i have a few ebooks that cover this (along with other RMQ usage scenarios) at https://leanpub.com/u/derickbailey

Could not write data in gemfire

I am trying to write to gemfire server in region 'trade'.
My class is like :
public class TradeDetails{
String exchange;
String Product;
String Account;
String Quantity;
//getter and setter }
I have deployed the jar in the gfsh console.
The query I am running on gfsh console is
put --key=1 --value=('exchange':'xyz','Product':'abc','Account':'xyz','Quantity':'123L') --region=/trade --value-class=model.TradeDetails
But I am getting an error
Couldn't convert JSON to Object of type class model.TradeDetails.
What could be the cause?
Well, according to the GemFire documenation, your Gfsh put command appears to be correct...
put --key=1 --value=('exchange':'xyz','Product':'abc','Account':'xyz','Quantity':'123L') --region=/trade --value-class=model.TradeDetails
However, your key value 1 is a bit suspect. If you used a key constraint of java.lang.Long on your "/trade" Region then you also need to specify the --key-class option on the put.
I was able to successfully perform the following...
$ gfsh
_________________________ __
/ _____/ ______/ ______/ /____/ /
/ / __/ /___ /_____ / _____ /
/ /__/ / ____/ _____/ / / / /
/______/_/ /______/_/ /_/ v8.2.0
Monitor and Manage GemFire
gfsh>connect
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=10.99.199.3, port=1099] ..
Successfully connected to: [host=10.99.199.3, port=1099]
gfsh>list members
Member Count : 1
Coordinator : SpringGemFireDataServer (10.99.199.3(SpringGemFireDataServer:77179)<v0>:47312)
Name | Id
----------------------- | ----------------------------------------------------
SpringGemFireDataServer | 10.99.199.3(SpringGemFireDataServer:77179)<v0>:47312
gfsh>describe member --name=SpringGemFireDataServer
Name : SpringGemFireDataServer
Id : 10.99.199.3(SpringGemFireDataServer:77179)<v0>:47312
Host : 10.99.199.3
Regions : People
PID : 77179
Groups :
Used Heap : 229M
Max Heap : 3641M
Working Dir : /Users/jblum/pivdev/spring-data-gemfire-tests-workspace/spring-data-gemfire-tests/target
Log file : /Users/jblum/pivdev/spring-data-gemfire-tests-workspace/spring-data-gemfire-tests/target
Locators : localhost[10334]
Cache Server Information
Server Bind : localhost
Server Port : 40404
Running : true
Client Connections : 0
gfsh>list regions
List of regions
---------------
People
gfsh>describe region --name=/People
..........................................................
Name : People
Data Policy : partition
Hosting Members : SpringGemFireDataServer
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ---- | -----
Region | size | 0
gfsh>
gfsh>put --region=/People --key=1 --key-class=java.lang.Long --value=('firstName':'Jon','lastName':'Doe') --value-class=org.spring.data.gemfire.app.beans.Person
Result : true
Key Class : java.lang.Long
Key : 1
Value Class : org.spring.data.gemfire.app.beans.Person
Value
------
<NULL>
gfsh>
gfsh>describe region --name=/People
..........................................................
Name : People
Data Policy : partition
Hosting Members : SpringGemFireDataServer
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ---- | -----
Region | size | 1
Note, my "/People" Region has a key type of java.lang.Long and value type of org.spring.data.gemfire.app.beans.Person.
Although, when I attempted to read "Jon Doe" back out, Gfsh puked...
gfsh>get --region=/People --key=1 --key-class=java.lang.Long --value-class=org.spring.data.gemfire.app.beans.Person
Exception occurred. null
However, I did go onto create a (Spring Boot-based) GemFire client cache application (with a base configuration of SpringGemFireClient) that pulled the Person back out successfully.
Person is [Jon Doe]
You might try annotating your model.TradeDetails application domain type with the Jackson mapping annotations, though I am not certain Gfsh actually uses them to perform the mapping since I think (when I lasted checked) Gfsh was not using Jackson. But, it wouldn't hurt either. Either way.
Note, my Server was started with this SpringGemFireDataServer, which is based on SpringGemFireServer.
Hope this helps (a little :-).
Cheers!
John

How do you set the channel on a XBee PRO series 2?

I've got three XBees. 2x PROs and a standard, all series 2s.
I've configured one PRO and one standard to be router/endpoints on channel 0 and PAN 234 (this is the default channel and PAN ID when selecting the "XBP24-B"/"XB24-B" "ZNET2.5 ROUTER/END DEVICE AT" profile (version 1247 for both).
However the one PRO I've set with the "ZNET 2.5 COORDINATOR AT" profile has a channel of E (though if I keep flashing the device with the same profile, this changes from 12-F).
Obviously if the coordinator doesn't have the same channel, nothing will work, but I can't see any way of setting the channel manually..?
The CH setting in X-CTU is read only, and I can see any other UI element to change the channel:
I've even been into the terminal and typed (words in brackets are what the terminal returns):
+++ (OK)
ATCH (E)
ATCH0 (ERROR)
ATCH 0 (ERROR)
ATCH00 (ERROR)
ATCH 00 (ERROR)
ATCH E (ERROR)
ATCHE (ERROR)
I've Googled and Googled to no avail. incredibly frustrating, can anyone help?!
I've had them working previous as a matter of fluke as I kept flashing the hardware until the channel numbers match up, but this is obviously ridiculous!
Channel selection with the XBee ZB (S2, S2B, S2C) series of modules works differently than with the XBee 802.15.4 (S1) modules. Channel selection is automatic with ZB (as opposed to it being manual with the 802.15.4 modules).
You normally never need to manipulate the channel selection parameters with ZB. Modules find each other and associate with each other if they can.
If your modules just can't seem to find each other it usually comes down to a mismatch in the PAN settings (ID), security settings (LK), or network joining permission settings on the coordinator (NJ).
Not a lot of information exists on the web outside of the Digi's XBee ZB OEM manual. For reference sake, channel selection with ZB works like this:
XBee ZB Coordinator is powered up
The XBee ZB Coordinator reads its SC parameter and builds a list of candidate channels to scan
The XBee ZB Coordinator then performs an energy scan on each candidate channel
The XBee ZB Coordinator then chooses the channel with the least amount of energy on it
This procedure aims to pick a channel with the least amount of noise on it be it from microwave ovens, WiFi networks, or anything else that might be transmitting on the 2.4GHz frequency band.
Any router or end devices joining a network with consult their SC parameters first, then they will try and search for networks they can join which match their PAN and security parameters. They will join and stay joined to the first network they can--with some minor exceptions (see the JV and NW parameters, for example).
If you want to force a channel selection you must set the SC parameter to enable only a single channel. The SC parameter is a bitmask1. Each bit set in the mask will enable one additional channel. What's tricky about this parameter is that the first bit (bit 0) is not channel 0, it's channel 11 (0x0B). For ease of use, if you wanted to lock an XBee ZB to a single channel here would be the values:
+---------------+---------------+------------------+-------------------------+
| Channel (Dec) | Channel (Hex) | XBee ZB SC Value | XBee Availability |
+---------------+---------------+------------------+-------------------------+
| 11 | 0xB | 0x1 | All |
| 12 | 0xC | 0x2 | All |
| 13 | 0xD | 0x4 | All |
| 14 | 0xE | 0x8 | All |
| 15 | 0xF | 0x10 | All |
| 16 | 0x10 | 0x20 | All |
| 17 | 0x11 | 0x40 | All |
| 18 | 0x12 | 0x80 | All |
| 19 | 0x13 | 0x100 | All |
| 20 | 0x14 | 0x200 | All |
| 21 | 0x15 | 0x400 | All |
| 22 | 0x16 | 0x800 | All |
| 23 | 0x17 | 0x1000 | All |
| 24 | 0x18 | 0x2000 | All |
| 25 | 0x19 | 0x4000 | S1, S2B, S2C (not S2) |
| 26 | 0x1A | 0x8000 | S1 only |
+---------------+---------------+------------------+-------------------------+
Obviously if the coordinator doesn't
have the same channel, nothing will
work, but I can't see any way of
setting the channel manually..?
I'm not sure if what you say above is right. From http://ftp1.digi.com/support/documentation/90000976_C.pdf (you should look at the correct version for your hardware, though), it looks like the purpose of a coordinator is to automatically determine the channel:
Coordinator Operation
Forming a Network
The coordinator is responsible for
selecting the channel, PAN ID (16-bit
and 64-bit), security policy, and
stack profile for a network. Since a
coordinator is the only device type
that can start a network, each ZigBee
network must have one coordinator.
After the coordinator has started a
network, it can allow new devices to
join the network. It can also route
data packets and communicate with
other devices on the network. To
ensure the coordinator starts on a
good channel and unused PAN ID, the
coordinator performs a series of scans
to discover any RF activity on
different channels (energy scan) and
to discover any nearby operating PANs
(PAN scan). The process for selecting
the channel and PAN ID are described
in the following sections.
Channel Selection
When starting a
network, the coordinator must select a
"good" channel for the network to
operate on. To do this, it performs an
energy scan on multiple channels
(frequencies) to detect energy levels
on each channel. Channels with
excessive energy levels are removed
from its list of potential channels to
start on.
I've actually never used the Digi XBee radios (just some of their other radios), so I don't know much about coordinators. I think that their user interface typically does expose all of the valid commands, so it probably won't work to try sending them manually (as you discovered).
You may find more experts on the Digi forums.
Hmm. strange, keep going over the settings again and finally got it working with 2-way comms?! :S
Setup one a coordinator
Setup other two as router/end devices
Set the Device High (DH) to 0 and the Device Low (DL) to FFFF (this means everything sent from this module should be received by everyone)
Set Negotiate Channel to 1 (Enabled), which means the only time a router/end device can set it's own channel is when it finds a coordinator on the same channel (this clearly wasn't happening in my case)
Everything else was left as default.
Leave coordinator powered on when configuring router/end devices so you can check they pair correctly.
As I said, I've used this configuration before and it just didn't work, so I don't know what kicked it into life this time, but it worked?!
One thing to take into account is that ZigBee channels extend from 11 to 26, zero is not a valid option.
Since that's the case, are you able to try the command ATCH11 ?
If that command succeeds, then perhaps the ATCH command wants a decimal input between 11 and 26?