I've come to the point where I need to store some additional data about where a particular field comes from in my Lucene.Net index. Specifically, I want to attach a guid to certain fields of a document when the field is added to the document, and retrieve it again when I get the document from a search result.
Is this possible?
Edit:
Okay, let me clarify a bit by giving an example.
Let's say I have an object that I want to allow the user to tag with custom tags like "personal", "favorite", "some-project". I do this by adding multiple "tag" fields to the document, like so:
doc.Add( new Field( "tag", "personal" ) );
doc.Add( new Field( "tag", "favorite" ) );
The problem is I now need to record some meta data about each individual tag itself, specifically a guid representing where that tag came from (imagine it as a user id). Each tag could potentially have a different guid, so I can't simply create a "tag-guid" field (unless the order of the values is preserved---see edit 2 below). I don't need this metadata to be indexed (and in fact I'd prefer it not to be, to avoid getting hits on metadata), I just need to be able to retrieve it again from the document/field.
doc.GetFields( "tag" )[0].Metadata...
(I'm making up syntax here, but I hope my point is clear now.)
Edit 2:
Since this is a completely different question, I've posted a new question for this approach: Is the order of multi-valued fields in Lucene stable?
Okay let's try another approach... The key problem area is the indeterminacy of the multiple field values under the same field name (e.g. "tag"). If I could introduce or obtain some kind of determinacy here, I might be able to store the metadata in another field.
For example, if I could rely on the order of the values of the field never changing, I could use an index in the set of values to identify exactly which tag I am referring to.
Is there any guarantee that the order I add the values to a field will remain the same when I retrieve the document at a later time?
Depending on your search requirements for this index, this may be possible. That way you can control the order of fields. It would require updating both fields as the tag list changes of course, but the overhead may be worth it.
doc.Add(new Field("tags", "{personal}|{favorite}"));
doc.Add(new Field("tagsref", "{1234}|{12345}"));
Note: using the {} allows you to qualify your search for uniqueness where similar values exist.
Example: If values were stored as "person|personal|personage" searching for "person" would return a document that has any one of person, personal or personage. By qualifying in curly brackets like so: "{person}|{personal}|{personage}", I can search for "{person}" and be sure it won't return false positives. Of course, this assumes you don't use curly brackets in your values.
I think you're asking about payloads.
Edit: From your use case, it sounds like you have no desire to use this metadata in your search, you just want it there. (Basically, you want to use Lucene as a database system.)
So, why can't you use a binary field?
ExtraData ed = new ExtraData { Tag = "tag", Type = "personal" };
byte[] byteData = BinaryFormatter.Serialize(ed); // this isn't the correct code, but you get the point
doc.Add(new Field("myData", byteData, Field.Store.YES));
Then you can deserialize it on retrieval.
Related
I am trying to create an index that returns entire data object of the documents in a collection;
here is the code:
CreateIndex({
name: "users_by_data",
source: Collection("users"),
values: { field: ['data'] }
})
but after creation it says: Values Not set (using ref by default)
If I specifically define fields (separately by their name), it will behave as expected, but data isn't working. the question is:
Is it impossible (e.g. for performance reasons) or am I doing it wrong?
side note: I am aware that I can do Lambda function on Paginate and achieve similar result, but this question specifically is about the Index level;
You can currently index regular values (strings, numbers, dates, etc) and you can index an array which will more or less 'unroll' the array into separate index entries. However, what you are trying, indexing an object is not possible at this point. An object (like data) will be ignored if you try to index it.
Currently, you have two options:
as you mentioned, using Map/Get at query time.
listing all values of the data object in the index since you can select specific values of an object in the index (which is however less flexible if new attributes arrive later on in the object)
We intend to support indexing of objects in the future, I can't provide an ETA yet though. There is a feature request on our forums as well that you can vote up: https://forums.fauna.com/t/object-as-terms-instead-of-scalar-s/628
You're going to want to use the Select function on the Ref you get back from the Index if you only want the data field back.
For an individual document, you can do something like this
Select( "data",
Get(
Match(
Index("yourIndexName"),
**yourIndexTerm // Could point to String/Number/FQL Ref
)
)
)
For a list of documents, you can use Paginate as you said but you can still pull the data property out of each document
Map(
Paginate(
Match(
Index("yourIndexName"),
**yourIndexTerm // Could point to String/Number/FQL Ref
)
),
Lambda("doc", Select("data", Get(Var("doc"))))
)
I'm new to Aerospike and am probably missing something fundamental, but I'm trying to see an enumeration of the Keys in a Set (I'm purposefully avoiding the word "list" because it's a datatype).
For example,
To see all the Namespaces, the docs say to use SHOW NAMESPACES
To see all the Sets, we can use SHOW SETS
If I want to see all the unique Keys in a Set ... what command can I use?
It seems like one can use client.scan() ... but that seems like a super heavy way to get just the key (since it fetches all the bin data as well).
Any recommendations are appreciated! As of right now, I'm thinking of inserting (deleting) into (from) a meta-record.
Thank you #pgupta for pointing me in the right direction.
This actually has two parts:
In order to retrieve original keys from the server, one must -- during put() calls -- set policy to save the key value server-side (otherwise, it seems only a digest/hash is stored?).
Here's an example in Python:
aerospike_client.put(key, {'bin': 'value'}, policy={'key': aerospike.POLICY_KEY_SEND})
Then (modified Aerospike's own documentation), you perform a scan and set the policy to not return the bin data. From this, you can extract the keys:
Example:
keys = []
scan = client.scan('namespace', 'set')
scan_opts = { 'concurrent': True, 'nobins': True, 'priority': aerospike.SCAN_PRIORITY_MEDIUM }
for x in (scan.results(policy=scan_opts)): keys.append(x[0][2])
The need to iterate over the result still seems a little clunky to me; I still think that using a 'master-key' Record to store a list of all the other keys will be more performant, in my case -- in this way, I can simply make one get() call to the Aerospike server to retrieve the list.
You can choose not bring the data back by setting includeBinData in ScanPolicy to false.
Following is a Node we having in DB
P:Person { name:"xxx", skill:"Java" }
and after awhile, we would like to change the Skill to skill array, is it possible?
P:Person { name:"xxx", skill:["Java", "Javascript"] }
Which Cypher query should I use?
If you have a single skill value in skill, then just do
MATCH (p:Person)
WHERE HAS (p.skill)
SET p.skill=[p.skill]
If there are multiple values you need to convert to an array such as P:Person { name:"xxx", skill:"Java","JavaScript" } then this should work:
MATCH (p:P)
SET p.skill= split(p.skill,",")
In fact, I think your real problem here is not how to get an array property in a node, but how to store it. Your data model is wrong in my opinion, storign data as array in neo4j is not common, since you have relations to store multiple skills (in your example).
How to create your data model
With your question, I can already see that you have one User, and one User can have 1..n skills.
I guess that one day (maybe tomorrow) you will need to know which users are able to use Java, C++, PHP, and every othre skills.
So, Here you can already see that every skill should have its own node.
What is the correct model in this case?
I think that, still with only what you said in question, you should have something like this:
(:Person{name:"Foo"})-[:KNOWS]->(:Skill{name:"Bar"})
using such a data model, you can get every Skill known by a Person using this query:
MATCH (:Person{name:"Foo"})-[:KNOWS]->(skill:Skill)
RETURN skill //or skill.name if you just want the name
and you can also get every Person who knows a Skill using this:
MATCH (:Skill{name:"Bar"})<-[:KNOWS]-(person)
RETURN person //Or person.name if you just want the name
Keep in mind
Storing array values in properties should be the last option when you are using neo4j.
If a property can be found in multiple nodes, having the same value, you can create a node to store it, then you will be able to link it the other nodes using relations, and finding every node having the property X = Y will be easier.
I have a field that might have apostrophes in it.
I want to be able to:
1. store the value as is in the index
2. search based on the value ignoring any apostrophes.
I am thinking of using:
doc.add(new Field("name", value, Store.YES, Index.NO));
doc.add(new Field("name", value.replaceAll("['‘’`]",""), Store.NO, Index.ANALYZED));
if I then do the same replace when searching I guess it should work and use the cleared value to index/search and the value as is for display.
am I missing any other considerations here ?
Performing replaceAll directly on the value its a bad practice in Lucene, since it would a much better practice to encapsulate your tokenization recipe in an Analyzer. Also I don't see the benefit of appending fields in your use case (See Document.add).
If you want to Store the original value and yet be able to search without the apostrophes simply declare your field like this:
doc.add(new Field("name", value, Store.YES, Index.ANALYZED);
Then simply hook up a custom Tokenizer that will replace apostrophes (I think the Lucene's StandardAnalyzer already includes this transformation).
If you are storing the field with the aim of using highlighting you should also consider using Field.TermVector.WITH_POSITIONS_OFFSETS.
I want to find all documents in the index that have a certain field, regardless of the field's value. If at all possible using the query language, not the API.
Is there a way?
If you know the type of data stored in your field, you can try a range query. Per example, if your field contain string data, a query like field:[a* TO z*] would return all documents where there is a string value in that field.
I've done some experimenting, and it seems the simplest way to achieve this is to create a QueryParser and call SetAllowLeadingWildcard( true ) and search for field:* like so:
var qp = new QueryParser( Lucene.Net.Util.Version.LUCENE_29, field, analyzer );
qp.SetAllowLeadingWildcard( true );
var query = qp.Parse( "*" ) );
(Note I am setting the default field of the QueryParser to field in its constructor, hence the search for just "*" in Parse()).
I cannot vouch for how efficient this method is over other methods, but being the simplest method I can find, I would expect it to be at least as efficient as field:[* TO *], and it avoids having to do hackish things like field:[0* TO z*], which may not account for all possible values, such as values starting with non-alphanumeric characters.
Another solution is using a ConstantScoreQuery with a FieldValueFilter
new ConstantScoreQuery(new FieldValueFilter("field"))