I have two tables on two different machines both running zookeeper as a daemon, one of them has myid 1 the other has myid 2.
The table is configured as such on the machine "1":
CREATE TABLE test(values)
ReplicatedMergeTree('/clickhouse/tables/01/test', '1', date, timestamp, 8192)
The table is configured as such on the machine "2":
CREATE TABLE test(values)
ReplicatedMergeTree('/clickhouse/tables/01/test', '2', date, timestamp, 8192)
The 01 in the '/clickhouse/tables/01/test' is probably the values messing with me... since I '{layer}-{shard}' with it.... but since I have no remote_servers configured I assumed I can just use any value and it wouldn't matter, since I don't want query distribution, just data duplication.
Also the tables seem to be "aware" of eachother since creating two tables with the replica number '2' or '1' results in an error saying such a table already exists.
Do I need to configure a remote_server by default if I want to use replication ? Do I need to configure zookeeper specifically for the tables I want to replicated ? Do you guys have any in-details examples of setting up data replication with clickhouse ?
It seems that you have done everything correctly. {layer}-{shard} part is optional - it just simplifies creation of similar tables on many servers. Bare minimum for configuring a replicated table is a path in zookeeper that is common for all replicas (by convention this path starts with /clickhouse/tables/ but in fact any unique path will do) and an identifier that is unique to each replica ('1' and '2' are totally OK, but more descriptive names allowing you to identify the corresponding host are recommended).
Configuring remote_servers is not necessary for replication, but zookeeper is required. You can use a single zookeeper cluster for all replicated tables.
Related
I am testing out NiFi to replace our current ingestion setup which imports data from multiple MySQL shards of a table and store it in HDFS.
I am using GenerateTableFetch and ExecuteSQL to achieve this.
Each incoming flow file will have a database.name attribute which is being used by DBCPConnectionPoolLookup to select the relevant shard.
Issue is that, let's say I have 2 shards to pull data from, shard_1 and shard_2 for table accounts and also I have updated_at as Maximum Value Columns, it is not storing state for the for the table#updated_at per shard. There is only 1 entry per table in state.
When I check in Data Provenance, I see the shard_2 flowfile file getting dropped without being passed to ExecuteSQL. And my guess is it's because shard_1 query gets executed first and then when shard_2 query comes, it's records are checked against shard_1's updated_at and since it returns empty, it drops the file.
Has anyone faced this issue? Or am I missing something?
The ability to choose different databases via DBCPConnectionPoolLookup was added after the scheme to store state in the database fetch processors (QueryDatabaseTable, GenerateTableFetch, e.g.). Also, getting the database name differs between RDBMS drivers, it might be in the DatabaseMetaData or ResultSetMetaData, possibly in getCatalog() or getSchema() or neither.
I have written NIFI-5590 to cover this improvement.
Been searching for a specific info but couldn't find; forgive me for being new at this.
I will try to replicate a Firebird DB using SymmetricsDS. This is an ERP database; which in my mind will have 1 master and 2 slaves. I will have 2 slave servers which will work locally and local machines will connect them as clients.
Say for example I am a client of local slave 1. I am creating a new customer which will automatically get a customer ID 100. At the same time a client of the local slave (server) 2 creates a new customer and it takes the same customer ID. Now when these two slaves sync to the master; there will be a conflict.
I know this sounds quite noob; you know you can't hide it.
What would be the best approach to prevent this; rather solving?
I don't think there is one "the best" approach. It depends on system specific details what works best... anyway, some options are:
UUID
Use UUID as customer ID. Since version 2.5 Firebird has some built in support for generating and converting UUIDs.
Segmented generators
On each local slave init the customer ID sequence so that IDs generated by it doesn't overlap with other slaves. Ie if you use 32 bit integers as PK and need max two slaves you dedicate top bit as "slave ID". That means that on first slave you start the sequence from zero while at the second you starti it from 2147483648 (bin 1000 0000 0000 0000 0000 0000 0000 0000). See the ALTER SEQUENCE statement for how to set the sequence's starting value.
ID server
You could have a service which generates IDs. Whenever slave needs ID for a customer it recuests it from a special service. To help with the perfomance it probably makes sense to request new IDs in patches and cache them for later use.
I suppose the system is legacy and you don't have the ability to change how it works. In a similar occasion I have solved this problem letting each slave generating sequences. I've added a write filter in symmetricDs on the master node that will intercept each push from a slave and add a unique prefix per slave. If data has to be synced back to the slaves after data is routed to each slave add a write filter to symmetric slave that will strip the added prefix.
For example maximum number of slaves is 99. Let's say slave 1 creates a sequence 198976, assuming the sequence length is 10, use slave's ID, pad left the sequence with zeros and add the slave id as prefix: (0)100198976. If slave 17 generated the same sequence, master node's filter would change it to 1700198976.
If the same data has is changed on the master and has to be sent back to the slave that generated it, write filter on the slave will strip the first two digits (after left padding with 0 in case of one digit slave IDs). Slave 1's sequence from master (0)100198976 will become again 198976; and slave 17's sequence from master 1700198976 will become 198976.
If the whole length of the ID column has been used on the slaves, alter the column on the master by widening the it to accommodate for the width of slave IDs
I am new to Apache Ignite and come from a Data Warehousing background.
So pardon me if I try to relate to Ignite through DBMS jargon.
I have gone through forums but I am still unclear about some of the basics.
I also would like specific answers to the scenario I have posted later.
1.) CacheMode=PARTITIONED
a.) When a cache is declared as partitioned, does the data get equally
partitioned across all nodes by default?
b.) Is there an option to provide a "partition key" based on which the data
would be distributed across the nodes? Is this what we call the Affinity
Key?
c.) How is partitioning different from affinity and can a cache have both
partition and affinity key?
2.) Affinity Concept
With an Affinity Key defined, when I load data (using loadCache()) into a partitioned cache, will the source rows be sent to the node they belong to or all the nodes on the cluster?
3.) If I create one index on the cache, does it by default become the partition/
affinity key as well? In such a scenario, how is a partition different from index?
SCNEARIO DESCRIPTION
I want to load data from a persistent layer into a Staging Cache (assume ~2B) using loadCache(). The cache resides on a 4 node cluster.
a.) How to load data such that each node has to process only 0.5B records?
Is is by using Partitioned Cache mode and defining an Affinity Key?
Then I want to read transactions from the Staging Cache in TRANSACTIONAL atomicity mode, lookup a Target Cache and do some operations.
b.) When I do the lookup on Target Cache, how can I ensure that the lookup is happening only on the node where the data resides and not do lookup on all the nodes on which Target Cache resides?
Would that be using the AffinityKeyMapper API? If yes, how?
c.) Lets say I wanted to do a lookup on a key other than Affinity Key column, can creating an index on the lookup column help? Would I end up scanning all nodes in that case?
Staging Cache
CustomerID
CustomerEmail
CustomerPhone
Target Cache
Seq_Num
CustomerID
CustomerEmail
CustomerPhone
StartDate
EndDate
This is answered on Apache Ignite users forum: http://apache-ignite-users.70518.x6.nabble.com/Understanding-Cache-Key-Indexes-Partition-and-Affinity-td11212.html
Ignite uses AffinityFunction [1] for data distribution. AF implements two mappings: key->partition and partition->node.
Key->Partition mapping is definitely map entry to partition. It doesn't bother of backups, but data collocation\distribution over partitions.
Usually, entry key (actually it's hashcode) is used to calculate partition entry belongs to.
But you can use AffinityKey [2] that would be use instead to manage data collocation. See also 'org.apache.ignite.cache.affinity.AffinityKey' javadoc.
Partition->Node mapping determines primary and backup nodes for partition. It doesn't bother of data collocation, but backups and partition distribution among nodes
Cache.loadCache just makes all nodes to call localLoadCache method. Which calls CacheStore.loadCache. So, each of grid nodes will load all the data from cache store and then discard data that is not local for the node.
Same data may resides on several nodes if you use a backups. AffinityKey should be a part of entry key and if AffinityKey mapping is configured then AffinityKey will be used instead of entry key for entry->partition mapping
and AffinityKey will be passed to AffinityFunction.
Indexes always resides on same node with the data.
a. To achieve this you should implement CacheStore.loadCache method to load data for certain partitions. E.g. you can store partitionID for each row in database.
However, if you change AF or partitions numbers you should update partitionID for entries in database as well.
The other way. If it is posible, you can load all the data in single node and then add other nodes to the grid. Data will rebalanced over nodes automatically.
b. AffinityKey is always used if it is as it shoud be part of entry key. So, lookup will always be happening on the node where the data resides.
c. I can't understand the question. Would you please clarify if it still is actual?
Summary:
I need to collect data reliably from the satelite SQL servers using the SQL Server Service Broker. I need to design the protokol that allows a kind of plug-in another satelite SQL server smoothly. (I am going to refine the question based on your suggestions -- including pictures. But I need to start somehow.)
Background:
I do have one central SQL server (Microsoft, SQL 2008 R2 Standard ed.) and several small ones near the production machines (the same version, Express edition). The small server collects temperatures from sensors into the table defined this way:
CREATE TABLE dbo.line_sensor_values (
UTC DATETIME NOT NULL,
line_no TINYINT NOT NULL, -- line number: 1, 2, 3, etc.
sensor_no TINYINT NOT NULL, -- sensor number: 1, 2, 3, etc.
sensor_value float NULL, -- the measured value
PRIMARY KEY CLUSTERED (
UTC ASC,
line_no ASC,
sensor_no ASC
)
)
The line_no is constant for the small SQL at the production line. The same table is created at the central SQL server, that can be temporarily physically disconnected from the small servers. (You know, the real, physical environment.)
The goal is to transwer all the collected data from the small SQL servers to the central ones. All the servers have the tables created; however, they know nothing about the data at the other side of communication. This way, some protocol must be designed to make the data collection working. A kind of hanshake must be designed to know where to continue with the data transfer after reconnection or after the failure of the sensor data collection.
The central server uses the collected sensor data to be processed as finalisation of some tasks. Say, the data points of certain sensor from certain line (known to the task) must be processed to form the chart. The task knows the time interval for which the sensor values are to be collected. However, the task database environment is not synchronized by events with collection of data. This way, UTC interval is the only way to determine whether the sensor data belong to the task or not.
Again, the data sensor sampling interval is independent on the task, and the SQL servers may be disconnected temporarily. Sometimes, the sensor may be broken, or there can be another reason for missing physical data from the sensor. However, if there is the sensor data with the UTC time, it means that all previous values or exist in the table or they never existed. Consequently, the way to know whether the data for the task are complete is equal to the knowledge that there are newer data for the sensor (produced after the UTC range interval for the task).
The goal is that no collected sensor value is lost. The ideal goal is that there is no need for any other special invocation of the functionality (i.e. via any kind of scheduler).
What was done already:
Basically, the sensor inserts the data into the dedicated table (other than the mentioned dbo.line_sensor_values above). The trigger gets the data and transforms them, and inserts them to the dbo.line_sensor_values. In other words, the table at the satelite machine is already collecting the data. It already works. This trigger or another mean could be used to send the sensor value via the Service Broker.
The stored procedure that takes the task, checks the table at the central SQL server for the sensor data, and makes the chart if the data is present was already designed and it works. However, it was use only manually as a proof of the concept.
The Service Broker setting was already suggested earlier. But the Service Broker communication for the purpose was not designed nor partly tested, yet.
I understand this is a broad question. This way I am going to split it to separate questions...
Separate questions to be solved:
SQL Service Broker: Collecting data — plug-in scenario analysis
Thanks for your time and experience, Petr
Sounds like something you don't need Service Broker for. You could add a new column IsReplicated bit not null default(0) to the slave machines. Then you need to regularly copy all data where IsReplicated = 0 to the central server and mark the data as IsReplicated = 1 at the slaves.
This is a very simple synchronization scheme. Would that work for you?
The only purpose of this comment is to close the question that should be considered a summary of the questions that discuss related details. I do not consider the existing answer really answering the question (no offence).
Now I have two choices.
I have the same schema for all the data. the record stand for the connection between to hosts.So one record belongs to two hosts. now I do the action that once I need to get the connection of 1 host. I will insert the record in to h2. So if there is a connection between host1 and host2. Every time I query the connections of host1, I have already store the connection between host1 and host2. So when I query the info about host2. the record will be stored twice in the table. So I am thinking about to create a table for each host.
Here is the dilemma. Since there are a lots of hosts. So if I create a table for each host,the quantity of the tables will be huge. Is to query the huge but only one table faster or to query lots of smaller tables faster?(no join)
Thanks
Indexing the one table with lots of records is the way to go. It can become a maintenance nightmare if you don't, and indexing will take care of your search speed in the table. Plus let's assume that you have a huge amount of records, in Sql Server 2008 (and in 2005) you can partition the table into separate files which will help with speed as well. Even not in sql server, keeping the data in the same table is the way to go, especially if your schema for the table is the same for each host.