Let's say we have 3 nodes A,B,C. Each having 15 GBs of filesize on disk. I want to bring up another node D but give it very little filesize size, because i will use this box only for data back up purpose. So I want to move to state
A 15GB,B 15GB, C 15GB, D 1GB
But I am not able to make it work, when i add the 4th node, it starts getting data which needs more filesize than what i have allocated. And process dies eventually.
Is it possible to configure different file sizes for each node in aerospike? While assigning bucket ranges to all the nodes, will the data distribution algorithm take file size into account?
All nodes in the cluster are expected to be homogeneous with respect to capacity. See https://discuss.aerospike.com/t/setting-up-a-cluster-with-different-memory-sizes-on-nodes/4387
I would like to understand how having node D would help facilitate data backup.
I don't quite understand what you are trying to do but if on node D, you do not define the namespace you have on A, B and C, say ns_dat, and, you do need one namespace on D - call it ns_temp, then D will not have any data from ns_dat. So, A, B, C are configured with namespace ns_dat, D is configured with ns_temp, and then D can be 1GB while A, B, C are 15GB. So you will have a 4 node cluster - what you want - and D will not take any data from A, B, C's namespace. This is not a recommended way to configure Aerospike. Again, not sure what you are trying to achieve.
Related
My question is about the different CP types triggers on a NetApp filer.
I have looked a lot and found good description for most of them but some explanations are a bit general.
Here is the list of the CP types (as shown at sysstat command) along with an explanation to the ones I already know. Please help me understand the rest (and correct me if I got anything wrong):
T - Time. CP occurs every 10 seconds since the last CP if no other trigger has caused it.
F - Full NVLog. The NVRAM is divided into two sections (4 when working in an HA pair configuration - half is a mirror of the HA partner) - if one is filled up the CP occurs and the data is flushed to disks, in the meantime the other half is used for incoming writes.
B - Back to back. While a CP is commited, the second half of the NVLog is full and needs to flush before the first one finished. This situation causes for latency problems and means that the filer is having hard times keeping up with the write loads.
b - I need help from you guys about this one, all the places I read only declare that this is also back to back that is worse than B but no one explains exactly what is the difference and when this is shown instead of the other.
S - Snapshot. Right before the filer is taking snapshot it is committing CP so it will be in a consistent state.
Z - I need your help for this one as well, everything I found just says that this is CP that happens in order to sync the machine and happens before snapshots. So, what is the need for this one if we have the S? what is the difference between them?
H - High water mark. I AM NOT SURE I GOT THIS ONE CORRECT BUT - When there is a lot of changed data in the memory buffers (RAM not NVRAM!) the filer is committing CP in order to flush and get the buffers clean.
L - Low water mark. I AM NOT SURE I GOT THIS ONE CORRECT BUT - When there is low space left on the memory buffers (RAM not NVRAM!) he filer is committing CP in order to flush and get the buffers clean. So the difference between this and H is that H is about changed data threshold and this is about data in buffers as a whole (if I got it right).
U - flUsh. When application using asynchronous writes asks that it's data will be flushed down to a persistent storage.
V - low Virtual buffers. I have no idea what that one means, help?
M - low Mbufs. I have no idea what that one means, help?
D - low Datavects. I have no idea what that one means, help?
N - max entries on NVLog. What is the difference between this one an F?
So, in summary I need help at:
Difference between B and b (and a real one - not that b is worse)
Difference between S and Z
Difference between F and N
Any information about V, M & D types
A validation that I got things right, specifically L, H and U will be appreciated
Thanks in advance.
Lowercase b stands for deferred back-to-back CPs. They are deferred because the existing b2b CPs can't be written down to disk fast enough, which means that your b2b CPs are falling further and further behind... which is why it is even worse than regular uppercase B b2b CPs. You are essentially throwing more I/O load at the controller than it can handle, and this can really impact your write latency.
When I unload a table from amazon redshift to S3, it always splits the table into two parts no matter how small the table. I have read the redshift documentation regarding unloading, but no answers other than it says sometimes it splits the table (I've never seen it not do that). I have two questions:
Has anybody every seen a case where only one file is created?
Is there a way to force redshift to unload into a single file?
Amazon recently added support for unloading to a single file by using PARALLEL OFF in the UNLOAD statement. Note that you still can end up with more than one file if it is bigger than 6.2GB.
By default, each slice creates one file (explanation below). There is a known workaround - adding a LIMIT to the outermost query will force the leader node to process whole response - thus it will create only one file.
SELECT * FROM (YOUR_QUERY) LIMIT 2147483647;
This only works as long as your inner query returns fewer than 2^31 - 1 records, as a LIMIT clause takes an unsigned integer argument.
How files are created? http://docs.aws.amazon.com/redshift/latest/dg/t_Unloading_tables.html
Amazon Redshift splits the results of a select statement across a set of files, one or more files per node slice, to simplify parallel reloading of the data.
So now we know that at least one file per slice is created. But what is a slice? http://docs.aws.amazon.com/redshift/latest/dg/t_Distributing_data.html
The number of slices is equal to the number of processor cores on the node. For example, each XL compute node has two slices, and each 8XL compute node has 16 slices.
It seems that the minimal number of slices is 2, and it will grow larger when more nodes or more powerful nodes is added.
As of May 6, 2014 UNLOAD queries support a new PARALLEL options. Passing PARALLEL OFF will output a single file if your data is less than 6.2 gigs (data is split into 6.2 GB chunks).
I have a graph like this:
Source | Sink | Timestamp
A B 2012-08-01 03:02:00
B C 2012-08-01 03:02:00
C D 2012-08-01 03:02:00
...
I am constructing this table from another table. I would like to design my table so that:
It uses the minimum storage without comprising on being able to get the most recent graph (I don't care about previous graphs for a real-time scenario)
It should be possible to study the graph evolution (how fast is something changing etc.)
Currently, other than storing Source, Sink and a Timestamp, there are no other optimizations. Considering that every snapshot contains 800K links, storing the graph in its entirety is not possible so I am looking for possible delta based approaches. Any suggestions on how to approach this problem?
The graph itself is highly dynamic i.e. nodes and links can be added or removed at each snapshot.
I think you are looking at something like this:
SNAPSHOT_VER ("version") is monotonically incremented with each new snapshot. Each new snapshot is a delta that adds or removes nodes and edges relative to the previous snapshot:
When adding new nodes/edges, you simply create a new SNAPSHOT and use its SNAPSHOT_VER for NODE.CREATED_VER and EDGE.CREATED_VER. Leave DELETED_VER at NULL for now.
When deleting the nodes/edges, simply set the DELETED_VER according to the snapshot at which they were deleted.
When querying for the graph as it existed at the given snapshot, you should be able to do it similar to this:
SELECT *
FROM NODE
WHERE
CREATED_VER <= #snapshot_ver
AND (DELETED_VER IS NULL OR #snapshot_ver < DELETED_VER)
(Equivalent query can be constructed for EDGE.)
This will get all the nodes that were created at the given snapshot or earlier, and were either not deleted at all or were deleted later than the given snapshot.
When querying for the latest graph, you can simply:
SELECT *
FROM NODE
WHERE DELETED_VER IS NULL
You'll probably need a composite index on {DELETED_VER, CREATED_VER DESC} (in both NODE and EDGE) for good performance in both of these cases.
I have a data structure consisting of a set of objects which are arranged into a multiply-linked list which is also (isomorphically) a valid DAG. It could be viewed as one single multiply-linked list, or as a series of n doubly-linked lists which may share members. (This is the same data structure from Algorithm for quickly obtaining a partial ordering over multiple linked lists, for those of you following my questions.)
I am looking for a general technique, in no specific SQL dialect, for expressing this multiply-linked list/DAG in SQL, such that it's easy to take a given node and obtain:
The previous and next links in the DAG, given a topological ordering of the DAG
The previous and next links in each doubly-linked list to which this node belongs
Using the example data from that other question:
first = [a, b, d, f, h, i];
second = [a, b, c, f, g, i];
third = [a, e, f, g, h, i];
I'd want to be able to, given node f, obtain [(c|d|e), g] from the overall DAG's topology and also {first: [d, h], second: [c, g], third: [e, g]} from each of the lists orderings.
Here's the fun part: n, the number of doubly-linked lists, is not fixed and may grow at any time. I'd rather not redo the schema each time that happens.
All of the algorithms I've come up with so far either (a) stuff a big pickle into the DB and pull it out in order to calculate orderings, or (b) require that the lists be explicitly enumerated as recursive relations in the DB.
I'll go with an option in (b) if I can't find something better but I'm hoping that there's something magical out there to make this easier.
Pre:
This is a question and answer forum, not 'lets sit down, group think for a bit, and solve the whole problem' forum.
I think what you want to investigate in a technique called 'modified preordered tree traversal' a mouthful i know, but it allows the storing of hierarchical data in a flat database and individual enties. Sadly, you do have to do some rewriting on inserts, but the selects can be done in a single query, so it's best for 'many view/ few changes' situations like a website. Luckily, you rarely have to rewrite the whole dataset (only the parts you changed and those hierarchically after them.
I remember a good article on the basics on it ( a couple years ago) but can't find the bookmark atm, so start with just a google search.
EDIT/UPDATE:
link: http://www.sitepoint.com/hierarchical-data-database/
No matter what, from dealing with this issue extensively, you will have to choose were to put the brunt of the work, on view, or on change. Depending on the size of the 'master' tree, you may (like me) decide to break the tree up into parts and use a tree of trees, limiting the update cost.
Is there a way to compute the difference between two sorted sets (zset) or do I have to use simple sets for this?
Problem:
Set F contains a list of sorted id's (sorted set, full list)
Set K contains a list of id's (simple set, subset of F)
I want to retrieve every entry in F, in order, that's not in K.
Is this possible using Redis alone or do I have to do the computation on the application? If yes, what is the best way?
EDIT: SDIFF does not suit this purpose as it doesn't allow sorted sets.
Make a copy of F as a simple set. Let's call it G. Now perform the SDIFF.
Or...
Make a copy of F as a sorted set. Let's call it G. Iterate through K and remove each element from G.
SDIFF really should work on sorted sets, regular sets, or combinations. But, at this time, it does not.
Also, if F is very large, you may see some performance hits when you make a copy of it. In this case, create a set G in your Redis DB that it updated when K is updated. That is, F and G are initially equal. As you add elements to K, remove the element from G.