I want to know, if the two settings node_auto_indexing and relationship_auto_indexing in the neo4j.properties concerning the ids of nodes and rels?
or creates neo4j automatically an index for the ids of the inserted nodes and rels?
the auto index creates index for all properties defined at the *_keys_indexable line in the neo4j.properties file.
the index then bounds the node ID with the specific property value. thus, searching the index for the the property value will return the node.
since your question is a bit unclear to me, you might want to take a look at official docu:
http://docs.neo4j.org/chunked/milestone/auto-indexing.html
No you shouldn't add your ID to the auto index. There is no use for it, since you can already retrieve nodes by ID, without using auto index.
There are however occassions where the usual ID is not sufficient. For instance, when working with users, you may have a user id of some kind. You'd then store this in a property, and add that property to the auto index. This way, you can search by user id. Underlying, Neo4J matches your custom user ID, with the actual node id.
Important to keep in mind here is that per definition, auto index is not unique. You need to design your application in such a fashion that the property is in fact unique, if you're expecting a single node result.
Related
I recently got to know Redis, integrated it into my project and now I am facing the following use case.
My question in short:
Which data type can I use to get all entries sorted AND to be able to overwrite single entries?
My question in long:
I have a huge amount of point cloud models that I want to store and work with via Redis.
My point cloud model consists of three things:
Unique id (stays the same)
Point Cloud as a string (changes over time)
Priority as an integer (changes over time)
Basically I would like to be able to do only two things with Redis. However, if I understand the documentation correctly, these are seen as benefits of two different data types, so I can't find a data type that exactly fits my use case. I hope, however, that I am wrong about this and that someone here can help me.
Use case:
Get quick all models, all already sorted
Overwrite/update a specific model
Sorted Sets
Advantage
Get all entries in sorted order
my model property Priority can be used here as a score, which determines the order.
Disadvantage
No possibility to access a special value via a key and overwrite it.
Hashes:
Advantage
Overwrite specific entry via Key > Field
Get all entries via Key
Disadvantage
No sorting
I would suggest to just use two distinct data types:
a hash with all the properties of your model, with the exception of the priority;
a sorted set which allows to easily sort your collection and deal with the scores / priorities.
You could then link the two by storing each hash key (or a distinctive value which allows to reconstruct the final hash key) as the related sorted set member.
For example:
> HSET point-cloud:123 foo bar baz suppiej
> ZADD point-clouds-by-priority 42 point-cloud:123
You will keep all the advantages you mentioned, with no disadvantages at all.
I need to understand how one can search attributes of a DynamoDB that is part of an array.
So, in denormalising a table, say a person that has many email addresses. I would create an array into the person table to store email addresses.
Now, as the email address is not part of the sort key, and if I need to perform a search on an email address to find the person record. I need to index the email attribute.
Can I create an index on the email address, which is 1-many relationship with a person record and it's stored as an array as I understand it in DynamoDB.
Would this secondary index be global or local? Assuming I have billions of person records?
If I could create it as either LSI or GSI, please explain the pros/cons of each.
thank you very much!
Its worth getting the terminology right to start with. DynamoDB supported data types are
Scalar - String, number, binary, boolean
Document - List, Map
Sets - String Set, Number Set, Binary Set
I think you are suggesting you have an attribute that contains a list of emails. The attribute might look like this
Emails: ["one#email.com", "two#email.com", "three#email.com"]
There are a couple of relevant points about Key attributes described here. Firstly keys must be top-level attributes (they cant be nested in JSON documents). Secondly they must be of scalar types (i.e. String, Number or Binary).
As your list of emails is not a scalar type, you cannot use it in a key or index.
Given this schema you would have to perform a scan, in which you would set the FilterExpression on your Emails attribute using the CONTAINS operator.
Stu's answer has some great information in it and he is right, you can't use an Array it's self as a key.
What you CAN sometimes do is concatenate several variables (or an Array) into a single string with a known seperator (maybe '_' for example), and then use that string as a Sort Key.
I used this concept to create a composite Sort Key that consisted of multiple ISO 8061 date objects (DyanmoDB stores dates as ISO 8061 in String type attributes). I also used several attributes that were not dates but were integers with a fixed character length.
By using the BETWEEN comparison I am able to individually query each of the variables that are concatenated into the Sort Key, or construct a complex query that matches against all of them as a group.
In other words a data object could use a Sort Key like this:
email#gmail.com_email#msn.com_email#someotherplace.com
Then you could query that (assuming you knew what the partition key is) with something like this:
SELECT * FROM Users
WHERE User='Bob' AND Emails LIKE '%email#msn.com%'
YOU MUST know the partition key in order to perform a Query no matter what you choose as your Sort Key and no matter how that Sort Key is constructed.
I think the real question you are asking is what should my sort keys and partition keys be? That will depend on exactly which queries you want to make and how frequently each type of query is used.
I have found that I have way more success with DynamoDB if I think about the queries I want to make first, and then go from there.
A word on Secondary Indexes (GSI / LSI)
The issue here is that you still need to 'know' the Partition Key for your secondary data structure. GSI / LSI help you avoid needing to create additional DynamoDB tables for the sole purpose of improving data access.
From Amazon:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html
To me it sounds more like the issue is selecting the Keys.
LSI (Local Secondary Index)
If (for your Query case) you don't know the Partition Key to begin with (as it seems you don't) then a Local Secondary Index won't help — since it has the SAME Partition Key as the base table.
GSI (Global Secondary Index)
A Global Secondary Index could help in that you can have a DIFFERENT Partition Key and Sort Key (presumably a partition key that you could 'know' for this query).
So you could use the Email attribute (perhaps composite) as the Sort Key on your GSI and then something like a service name, or sign-up stage, as your Partition Key. This would let you 'know' what partition that user would be in based on their progress or the service they signed up from (for example).
GSI / LSI still need to generate unique values using their keys so keep that in mind!
How does activerecord assign an ID to a newly created record? The ID values seem to be all over the place. Sometimes they are sequential, but sometimes they seem to be some kind of a hash.
Is there a way to control the behavior?
Within a relational database you'll see that IDs are usually sequential. This happens to be an automatically incrementing field called id by default in these databases with Rails. This is the 99% case, meaning that 99% of the time you can expect to see it done this way. It's the sane way.
However, There are some cases in which the "id" field within the database may not be automatically incrementing and may instead be a string. In a database I am working with at the moment, the id field is called client_id, is a 6-character string such as "RAB001" and needs to be manually assigned by the code itself. This is due to a legacy system we are supporting and there's nothing we can do to fix that. It's just how it is.
In other databases such as Mongoid the ids are, once again, generated automatically. There's a difference here though: instead of them being automatically incrementing numbers they are a hash. In a Mongo database I happen to have handy, one of the object's _id fields (note the underscore) is this lovely, easy-to-understand1 hash: 4e22b5812f8b7d6f6d000001. This is automatically generated by Mongo and I don't really care what it is except for when I need to find an object and there's no other way of finding it by another unique value.
I would recommend sticking with an automatically generating ID system, be it something provided by the traditional database systems such as PostgreSQL or MySQL or something by Mongo.
Any system where you need to generate the primary key for a record manually needs to have a huge "HERE BE DRAGONS" label on it and should be handled like a case of nitroglycerin or similarly to this apt analogy. Avoid this system if you can.
1 I am being sarcastic here.
I am creating the model for a web application. The tables have ID fields as primary keys. My question is whether one should define ID as a property of the class?
I am divided on the issue because it is not clear to me whether I should treat the object as a representation of the table structure or whether I should regard the table as a means to persist the object.
If I take the former route then ID becomes a property because it is part of the structure of the database table, however if I take the latter approach then ID could be viewed as a peice of metadata belonging to the database which is not strictly a part of the objects model.
And then we arrive at the middle ground. While the ID is not really a part of the object I'm trying to model, I do realise that the the objects are retrieved from and persisted to the database, and that the ID of an object in the database is critical to many operations of the system so it might be advantageous to include it to ease interactions where an ID is used.
I'm a solo developer, so I'd really like some other, probably more experienced perspectives on the issue
Basically: yes.
All the persistence frameworks ive used (including Hibernate, Ibatis) do require the ID to be on the Object.
I understand your point about metadata, but an Object from a database should really derive its identity in the same way the database does - usually an int primary key. Then Object-level equality should be derived from that.
Sometimes you have primary keys that are composite, e.g first name and last name (don't ever do this!), in which cases the primary key doesn't become 'metadata' because it is part of the Object's identity.
I generally reserve the ID column of an object for the database. My opinion is that to use it for any 'customer-facing' purpose, (for example, use the primary key ID as a customer number) you will always shoot yourself in the foot later.
If you ever make changes to the existing data (instead of exclusively adding new data), you need the PK. Otherwise you don't know which record to change in the DB.
You should have the ID in the object. It is essential.
The easiest use case to give as an example is testing equality:
public bool Equals(Object a, Object b) { return {a.ID = b.ID}; }
Anything else is subject to errors, and you'll find that out when you start getting primary key violations or start overwriting existing data.
By counterargument:
Say you don't have the ID in the object. Once you change an object, and don't have it's ID from the database, how will you know which record to update?
At the same time, you should note that the operations I mention are really private to the object instance, so ID does not necessarily have to be a public property.
I include the ID as a property. Having a simple unique identifier for an object is often very handy regardless of whether the object is persisted in a database or not. It also makes your database queries much more simple.
I would say that the table is just a means to persist an object, but that doesn't mean the object can't have an ID.
I'm very much of the mindset that the table is a means to persist the object, but, even so, I always expose the IDs on my objects for two primary reasons:
The database ID is the most convenient way to uniquely identify an object, either within a class (if you're using a per-table serial/autonumber ID) or universally (if you're maintaining a separate "ID-to-class" mapping). In the context of web applications, it makes everything much simpler and more efficient if your forms are able to just specify <input type=hidden name=id value=12345> instead of having to provide multiple fields which collectively contain sufficient information to identify the target object (or, worse, use some scheme to concatenate enough identifying information into a single string, then break it back down when the form is submitted).
It needs to have an ID anyhow in order to maintain a sane database structure and there's no reason not to expose it.
Should the ID in the object read-only or not? In my mind it should be read-only as by definition the ID will never change (as it uniquely identifies a record in the database).
This creates a problem when you create a new object (ID not set yet), save it in the database through a stored procedure which returns the newly created ID then how do you store it back in the object if the ID property is read-only?
Example:
Employee employee = new Employee();
employee.FirstName="John";
employee.LastName="Smith";
EmployeeDAL.Save(employee);
How does the Save method (which actually connects to the database to save the new employee) update the EmployeeId property in the Employee object if this property is read-only (which should be as the EmployeeId will never ever change once it's created).
Is it possible in hibernate to have an entity where some IDs are assigned and some are generated?
For instance:
Some objects have an ID between 1-10000 that are generated outside of the database; while some entities come in with no ID and need an ID generated by the database.
You could use 'assigned' as the Id generation strategy, but you would have to give the entity its id before you saved it to the database. Alternately you could build your own implementation of org.hibernate.id.IdentifierGenerator to provide the Id in the manner you've suggested.
I have to agree w/ Cade Roux though, and doing so seems like it be much more difficult than using built in increment, uuid, or other form of id generation.
I would avoid this and simply have an auxiliary column for the information about the source of the object and a column for the external identifier (assuming the external identifier was an important value you wanted to keep track of).
It's generally a bad idea to use columns for mixed purposes - in this case to infer from the nature of a surrogate key the source of an object.
Use any generator you like, make sure it can start at an offset (when you use a sequence, you can initialize it accordingly).
For all other entities, call setId() before you insert them. Hibernate will only generate an id if the id property is 0. Note that you should first insert objects with ids into the db and then work with them. There is a lot of code in Hibernate which expects the object to be in the DB when id != 0.
Another solution is to use negative ids for entities which come with an id. This will also make sure that there are no collisions when you insert an new object.