I recently "inherited" a Carbon/Graphite setup from a colleague which I have to redesign. The current setup is:
Datacenter 1 (DC1): 2 servers (server-DC1-1 and server-DC1-2) with 1 carbon-relay and 4 carbon caches
Datacenter 2 (DC2): 2 servers (server-DC2-1 and server-DC2-2) with 1 carbon-relay and 4 carbon caches
All 4 carbon-relays are configured with a REPLICATION_FACTOR of 2, consistent hashing and all carbon-caches ( 2(DCs) * 2(Servers) * 4(Caches) ). This had the effect that some metrics exist only on 1 server (they probably were hashed to a different cache on the same server). With over 1 million metrics this problem affects about 8% of all metrics.
What I would like to do is a multi-tiered setup with redundancy, so that I mirror all metrics across the datacenters and inside the datacenter I use consistent hashing to distribute the metrics evenly across 2 servers.
For this I need help with the configuration (mainly) of the relays. Here is a picture of what I have in mind:
The clients would send their data to the tier1relays in their respective Datacenters ("loadbalancing" would occur on client side, so that for example all clients with an even number in the hostname would send to tier1relay-DC1-1 and clients with an odd number would send to tier1relay-DC1-2).
The tier2relays use consistent hashing to distribute the data in the datacenter evenly across the 2 servers. For example the "pseudo" configuration for tier2relay-DC1-1 would look like this:
RELAY_METHOD = consistent-hashing
DESTINATIONS = server-DC1-1:cache-DC1-1-a, server-DC1-1:cache-DC1-1-b, (...), server-DC1-2:cache-DC1-2-d
What I would like to know: how do I tell tier1relay-DC1-1 and tier1relay-DC1-2 that they should send all metrics to the tier2relays in DC1 and DC2 (replicate the metrics across the DCs) and do some kind of "loadbalancing" between tier2relay-DC1-1 and tier2relay-DC1-2.
On another note: I also would like to know what happens inside the carbon-relay if I use consistent hashing, but one or more of the destinations are unreachable (server down) - do the metrics get hashed again (against the reachable caches) or will they simply be dropped for the time? (Or to ask the same question from a different angle: when a relay receives a metric does it do the hashing of the metric based on the list of all configured destinations or based on the currently available destinations?)
https://github.com/grobian/carbon-c-relay
Which exactly does what you need. Also it give you a great boost in performance.
Related
I am looking for a cloud service provider to host a SQL DB in and access through API calls. After looking through multiple providers I have seen that Azure has a 12-month free trial but only 250 GB S0 instance with 10 database transaction units.
Could anyone explain to be what they mean by 10 DB transaction units? Any help is greatly appreciated.
For reference our database would not be large in scale just holding candidate and judges applications which we only get maximum 600 candidates per year.
I tried looking transactional units online and saw it make be a single REST API call which seems absurd to me.
Please examine the output of the following query:
SELECT * FROM sys.dm_user_db_resource_governance
That will tell you the following information about the current service tier:
min_cores (cores available on the service)
max_dop (the MAX_DOP value for the user workload)
max_sessions (the maximum number of sessions allowed)
max_db_max_size_in_mb (the maximum max_size value for a data file, in MB)
log_size_in_mb
instance_max_worker_threads (worker thread limit for the SQL Server instance)
The above information will give the details of what 10 DTU means in terms of resources available. You can run this query every time you change service tier of the database.
I am trying to track server uptimes using redis.
So the approach I have chosen is as follows:
server xyz will keep on sending my service ping indicating that it was alive and working in the last 30 seconds.
My service will store a list of all time intervals during which the server was active. This will be done by storing a list of {startTime, endTime} in redis, with key as name of the server (xyz)
Depending on a user query, I will use this list to generate server uptime metrics. Like % downtime in between times (T1, T2)
Example:
assume that the time is T currently.
at T+30, server sends a ping.
xyz:["{start:T end:T+30}"]
at T+60, server sends another ping
xyz:["{start:T end:T+30}", "{start:T+30 end:T+60}"]
and so on for all pings.
This works fine , but an issue is that over a large time period this list will get a lot of elements. To avoid this currently, on a ping, I pop the last element of the list, check if it can be merged with the latest time interval. If it can be merged, I coalesce and push a single time interval into the list. if not then 2 time intervals are pushed.
So with this my list becomes like this after step 2 : xyz:["{start:T end:T+60}"]
Some problems I see with this approach is:
the merging is being done in my service, and not redis.
incase my service is distributed, The list ordering might get corrupted due to multiple readers and writers.
Is there a more efficient/elegant way to handle this , like maybe handling merging of time intervals in redis itself ?
We are looking for NewSQL database.
We were using ScyllaDB (Cassandra) which in ~ 10 days test just had fatal crash (all nodes are unresponsive to API, CSQL is working, one node after reset has lost data for > 5 days (100GB), after reset failed to rebuild).
This is the most tragic crash that any DB that we've tested had, so we are looking for something else.
In terms of data we're talking about 200-500TB per month, > 10 nodes.
~ 50k writes a second, ~ 200k reads a second.
We need something that:
supports SQL (Cassandra was a huge headache during design stage, impossible to replace certain aspects of our MySQL database (which is fine, but we are testing new product that has much higher requirements)
can scale horizontally well from 5-10-50 nodes + regional support (we have farms on in 8 main locations, they have to replicate fairly well, but clients from i.e. USA West won't access data in USA East)
can support read/write scaling
we do not use transactions as per se, but want to be able to where X = 'E' and Date > '2020'
I want to understand how the namespace/topic replication works in Apache Pulsar and what affect does the change in cluster size have on the replication factor of the existing and new namespaces/topics.
Consider the following scenario:
I am starting with a single node with the following broker configuration:
# Number of bookies to use when creating a ledger
managedLedgerDefaultEnsembleSize=1
# Number of copies to store for each message
managedLedgerDefaultWriteQuorum=1
# Number of guaranteed copies (acks to wait before write is complete)
managedLedgerDefaultAckQuorum=1
After a few months I decide to increase the cluster size to two with the following configuration for the new broker:
# Number of bookies to use when creating a ledger
managedLedgerDefaultEnsembleSize=2
# Number of copies to store for each message
managedLedgerDefaultWriteQuorum=2
# Number of guaranteed copies (acks to wait before write is complete)
managedLedgerDefaultAckQuorum=2
In the above scenario what will be the behaviour of the cluster:
Does this change the replication factor(RF) of the existing topics?
Do newly created topics have the old RF or the new specified RF?
How does the namespace/topic(Managed Ledger) -> Broker ownership work?
Please note that the two broker nodes have different configurations at this point.
TIA
What you are changing is the default replication settings (ensemble, write, ack). You shouldn't be using different defaults on different brokers, because then you'll get inconsistent behavior depending on which broker the client connects to.
The replication settings are controlled at namespace level. If you don't explicitly set them, you get the default settings. However, you can change the settings on individual namespaces using the CLI or the REST interface. If you start with settings of (1 ensemble, 1 write, 1 ack) on the namespace and then change to (2 ensemble, 2 write, 2 ack), then the following happens:
All new topics in the namespace use the new settings, storing 2 copies of each message
All new messages published to existing topics in the namespace use the new settings, storing 2 copies. Messages that are already stored in existing topics are not changed. They still have only 1 copy.
An important point to note is that the number of brokers doesn't affect the message replication. In Pulsar, the broker just handles the serving (producing/consuming) of the message. Brokers are stateless and can be scaled horizontally. The messages are stored on Bookkeeper nodes (bookies). The replication settings (ensemble, write, ack) refer to Bookkeeper nodes, not brokers. Here is an diagram from the Pulsar website that illustrates this:
So, to move from a setting of (1 ensemble, 1 write, 1 ack) to (2 ensemble, 2 write, 2 ack), you need to add a Bookkeeper node to your cluster (assuming you start with just 1), not another broker.
I'm trying to use HAProxy for rate-limiting.
I need to keep track of several endpoints and limit them individually.
So far I was using general purpose counters. However, there is only 3 of them, sc0 to sc2.
In the documentation, it mentions that all the operations on these counters take an optional table parameter, but it's not clear, then, if I can track different things on different tables, but using the same counters.
In other words: is the limit of 3 general purpose counters global, or per sticky table?
If, after proper table definition and track instructions, I do
sc1_inc_gpc0(table1)
(and, under different conditions)
sc1_inc_gpc0(table2)
And then have 2 acl rules like
acl X sc1_get_gpc0(table1) gt 1
acl Y sc1_get_gpc0(table2) gt 1
Will the two acls work indepentently, or they would to all effects track the same counter?
Thanks for all help!
(In case you are wondering: for a number of reasons, at the moment I could not use a different solution than HAProxy for rate-limiting)
Self answered after looking at the source code and testing.
Yes it is possible to use the same counter on different tables
Moreover, you can also increment the number of available counters at build time. The default is 3, but it can be set up to 10 for sure. Then generic versions of the functions like sc_gpc0_rate(<ctr>[,<table>]) can be used, passing the index of the new counter as first argument.