Index analyzer not working with Elasticsearch.net And Nest - nest

my code is it.. using nest, elasticsearch 2.3.0 Version
i want mapping( + custom analyzer) & create index ...
but mapping is not unsuccessful low level call error!
please , check my code and review for me
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
var client = new ElasticClient(settings);
var request = new IndexExistsRequest("aa");
var result = client.IndexExists(request);
if (result.Exists == true)
{
client.DeleteIndex("aa", null);
}
var ilhee_Custom = new CustomAnalyzer
{
Filter = new List<string> { "lowercase", "stop", "standard", "snowball" },
Tokenizer = "standard"
};
List<Person> categList = new List<Person>();
var Person = new Person
{
id = 1,
Firstname = "an apples bananas boxes, the sun.",
Lastname = "a beautiful womens with a good guys in there"
};
categList.Add(Person);
var response = client.CreateIndex("aa");
var mappingResponse = client.Map<Person>(d => d
.Properties(props => props
.String(s => s
.Name(p => p.Firstname)
.Index(FieldIndexOption.Analyzed)
.Analyzer("ilhee_Custom")
)
.String(s1 => s1
.Name(p1 => p1.Lastname)
.NotAnalyzed()
)
)
.Index("aa")
.Type("person")
);
var b = client.IndexMany<Person>(categList, "aa", "person");

You create a custom analyzer but you don't send it to Elasticsearch, so when it comes to using it in a mapping, Elasticsearch knows nothing about the custom analyzer.
You can create a new index with analysis and mappings in one request. Here's an example of creating an index, adding your custom analyzer and mapping as part of the index creation
void Main()
{
var node = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(node)
// set "aa" as the default index; if no index
// is specified for a type or in the request,
// the default index will be used
.DefaultIndex("aa");
var client = new ElasticClient(settings);
var indexExistsResponse = client.IndexExists("aa");
if (indexExistsResponse.Exists)
{
client.DeleteIndex("aa", null);
}
var people = new List<Person>{
new Person
{
id = 1,
Firstname = "an apples bananas boxes, the sun.",
Lastname = "a beautiful womens with a good guys in there"
}
};
var createIndexResponse = client.CreateIndex("aa", c => c
.Settings(s => s
.Analysis(a => a
.Analyzers(ad => ad
// give the custom analyzer a name
.Custom("ilhee_Custom", ca => ca
.Tokenizer("standard")
.Filters("lowercase", "stop", "standard", "snowball")
)
)
)
)
.Mappings(m => m
.Map<Person>(d => d
.Properties(props => props
.String(s => s
.Name(p => p.Firstname)
.Analyzer("ilhee_Custom")
)
.String(s1 => s1
.Name(p1 => p1.Lastname)
.NotAnalyzed()
)
)
)
)
);
var indexManyResponse = client.IndexMany<Person>(people, "aa");
// refresh the index after indexing, so that newly indexed documents
// are available in search results.
client.Refresh("aa");
var searchResponse = client.Search<Person>(s => s
.Query(q => q
.Match(m => m
.Field(p => p.Firstname)
.Query("boxes")
)
)
);
}
public class Person
{
public int id { get; set; }
public string Firstname { get; set; }
public string Lastname { get; set;}
}
The search returns back our indexed document as expected
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.15342641,
"hits" : [ {
"_index" : "aa",
"_type" : "person",
"_id" : "1",
"_score" : 0.15342641,
"_source" : {
"id" : 1,
"firstname" : "an apples bananas boxes, the sun.",
"lastname" : "a beautiful womens with a good guys in there"
}
} ]
}
}

Related

Importing a contact column into Podio

Which app_id should be used for importing into a contact column? Also, what should the mappings parameter look like?
podio.ImporterService.ImportAppItems(fileId, appId, new List<ImportMappingField> {
new ImportMappingField { FieldId = primaryFieldId, Unique = false, Value = new { column_id = "0" }},
new ImportMappingField { FieldId = contactfieldId, Unique = false, Value = new { column_id = "1", app_id = ???, mappings = new []{ ??? }}}
})
Edit:
I figured it out. Below is an example that works for me.
podio.ImporterService.ImportAppItems(373063497, 18803129, new List<ImportMappingField> {
new ImportMappingField {
FieldId = 148580608,
Unique = false,
Value = new { column_id = "0" }
},
new ImportMappingField {
FieldId = 148580614,
Unique = false,
Value = new {
mappings = new []{
new {
field_key = "mail",
unique = "true",
column_id = "4"
}
}
}
}
});
See the API documentation [1]
[1] https://developers.podio.com/doc/contacts

Why does RavenDB return a wrong total result count?

As shown in the following unit test I expect stats.TotalResults to be 2 but is 3. Why is that?
[Test]
public void RavenQueryStatisticsTotalResultsTest1()
{
using (var db = _documentStore.OpenSession())
{
db.Store(new Club { Name = "Foo1", Type = "Amateur" }); // --> Matches all conditions
db.Store(new Club { Name = "Foo2", Type = "Professional" });
db.Store(new Club { Name = "Foo3", Type = "Amateur" }); // --> Matches all conditions
db.Store(new Club { Name = "Boo1", Type = "Amateur" });
db.Store(new Club { Name = "Boo2", Type = "Professional" });
db.SaveChanges();
}
WaitForIndexing(_documentStore);
using (var db = _documentStore.OpenSession())
{
RavenQueryStatistics stats;
var query = db.Query<Club>()
.Statistics(out stats)
.Where(club => club.Type == "Amateur")
.Intersect()
.Search(club => club.Name, "Foo*", escapeQueryOptions: EscapeQueryOptions.AllowPostfixWildcard);
var clubs = query.ToList();
Assert.AreEqual(2, clubs.Count);
Assert.AreEqual(2, stats.TotalResults); // I expect 2 but was 3! Why is that?
}
}
You need Intersect when using multiple where clauses. To combine Where and Search, you have to pass SearchOptions.And to Search:
using (var db = _documentStore.OpenSession()) {
RavenQueryStatistics stats;
var query = db.Query<Club>()
.Customize(_ => _.WaitForNonStaleResultsAsOfLastWrite())
.Statistics(out stats)
.Where(club => club.Type == "Amateur")
.Search(
club => club.Name,
"Foo*",
escapeQueryOptions: EscapeQueryOptions.AllowPostfixWildcard,
options:SearchOptions.And);
var clubs = query.ToList();
Assert.AreEqual(2, clubs.Count);
Assert.AreEqual(2, stats.TotalResults);
}

Using system.linq.lookup values for dropdownfor mvc 4

In MVC 4 I am using razor to get items from collection and assign it to a var. The var is of type
{System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping}
This is my code
var ChairList = Model.Interviewers.Where(d => d.LocKey == Convert.ToString(location.LocationKey) && d.IsChair && d.Date == date.Date).GroupBy(x => new { x.FullDetails, x.InterviewerId, x.Preference }).ToList();
And how it looks in watch:
- ChairList Count = 1 System.Collections.Generic.List<System.Linq.IGrouping<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>>
- [0] {System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping} System.Linq.IGrouping<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer> {System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping}
+ [System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping] {System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping} System.Linq.Lookup<<>f__AnonymousType0<string,System.Guid,string>,IMEModels.InterviewManagement.Interviewer>.Grouping
- Key { FullDetails = "TEST - Richard Jackson - (80020937)", InterviewerId = {ff1efad7-7176-4fab-a1bb-30f6656c8880}, Preference = "Available" } <Anonymous Type>
FullDetails "TEST - Richard Jackson - (80020937)" string
+ InterviewerId {ff1efad7-7176-4fab-a1bb-30f6656c8880} System.Guid
Preference "Available" string
+ Raw View
I want to use this for a dropdownfor but it doesn't recognise the key and value that I am giving for it:
#Html.DropDownListFor(m => m.InterviewSchedules[location.InterviewDates.IndexOf(date)].ChairId, new SelectList(ChairList, "InterviewerId", "FullDetails"))
Can someone help me with this piece of code? It's possible that there is an easier way of doing this that I am unaware of.
For view
#Html.DropDownListFor(m => m.InterviewSchedules[location.InterviewDates.IndexOf(date)].ChairId,, ViewData["ReturnList"] as SelectList, new { #class = "form-control" })
For Code (Return as viewData)
public SelectList ReturnList(Guid UID) {
var ChairList = Model.Interviewers.Where(d => d.LocKey == Convert.ToString(location.LocationKey) && d.IsChair && d.Date == date.Date).GroupBy(x => new { x.FullDetails, x.InterviewerId, x.Preference }).ToList();
List<SelectListItem> selectItems = ChairList.Select(s => new SelectListItem() {
Text =FullDetails,
Value = InterviewerId.ToString(),
Selected = false
}
).ToList();
selectItems.Insert(0, new SelectListItem() {
Text = " --Select -- ",
Value = null,
Selected = false
});
SelectList selectList = new SelectList(selectItems, "Value", "Text");
return selectList;
}

Work around for NEST not supporting pattern replace char filter

NEST doesn't appear to support the pattern replace char filter described here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-pattern-replace-charfilter.html
I've created an issue at https://github.com/elasticsearch/elasticsearch-net/issues/543.
Most of my indexing is working so I would like to continue to use NEST. Is there a way I can work around this using some manual json injection at some point during the index configuration? I'm new to NEST so not sure if this is doable.
Specifically I'm hoping to use the pattern replace char filter to remove unit numbers from a street address before they are run through a custom analyzer (i.e. #205 - 1260 Broadway becomes 1260 Broadway). Because of the custom analyzer, I believe I need to use this char filter to accomplish this.
My current configuration looks like this:
elasticClient.CreateIndex("geocoding", c => c
.Analysis(ad => ad
.Analyzers(ab => ab
.Add("address-index", new CustomAnalyzer()
{
Tokenizer = "whitespace",
Filter = new List<string>() { "lowercase", "synonym" }
})
.Add("address-search", new CustomAnalyzer()
{
Tokenizer = "whitespace",
Filter = new List<string>() { "lowercase" },
CharFilter = new List<string>() { "drop-unit" }
})
)
.CharFilters(cfb => cfb
.Add("drop-unit", new CharFilter()) //missing char filter here
)
.TokenFilters(tfb => tfb
.Add("synonym", new SynonymTokenFilter()
{
Expand = true,
SynonymsPath = "analysis/synonym.txt"
})
)
)
UPDATE:
As of May 2014, NEST now supports the pattern replace char filter: https://github.com/elasticsearch/elasticsearch-net/pull/637
Instead of using the fluent settings during your index creation you can use the Settings.Add approach to add to the FluentDictionary in a more manual way, but with complete control over what settings are passed in. An example of this is shown in Create Index of the NEST Documentation. I am using this approach for a very similar reason.
You configuration would look something similar to the following:
elasticClient.CreateIndex("geocoding", c => c.
.Settings(s => s.
.Add("analysis.analyzer.address-index.type", "custom")
.Add("analysis.analyzer.address-index.tokenizer", "whitespace")
.Add("analysis.analyzer.address-index.filter.0", "lowercase")
.Add("analysis.analyzer.address-index.filter.1", "synonym")
.Add("anaylsis.analyzer.address-search.type", "custom")
.Add("analysis.analyzer.address-search.tokenizer", "whitespace")
.Add("analysis.analyzer.address-search.filter.0", "lowercase")
.Add("analysis.analyzer.address-search.char_filter.0", "drop-unit")
.Add("analysis.char_filter.drop-unit.type", "mapping")
.Add("analysis.char_filter.drop-unit.mappings.0", "<mapping1>")
.Add("analysis.char_filter.drop-unit.mappings.1", "<mapping2>")
...
)
);
You will need to replace <mapping1> and <mapping2> above with your actual char_filter mappings that you want to use. Please note that I have not used a char_filter before, so the settings values may be a little off, but should get you going in the right direction.
Just to provide a follow-up to Paige's very helpful answer, it looks like you can combine the fluent and manual Settings.Add approaches. The following worked for me:
elasticClient.CreateIndex("geocoding", c => c
.Settings(s => s
.Add("analysis.char_filter.drop_unit.type", "pattern_replace")
.Add("analysis.char_filter.drop_unit.pattern", #"#\d+\s-\s")
.Add("analysis.char_filter.drop_unit.replacement", "")
)
.Analysis(ad => ad
.Analyzers(ab => ab
.Add("address_index", new CustomAnalyzer()
{
Tokenizer = "whitespace",
Filter = new List<string>() { "lowercase", "synonym" }
})
.Add("address_search", new CustomAnalyzer()
{
CharFilter = new List<string> { "drop_unit" },
Tokenizer = "whitespace",
Filter = new List<string>() { "lowercase" }
})
)
.TokenFilters(tfb => tfb
.Add("synonym", new SynonymTokenFilter()
{
Expand = true,
SynonymsPath = "analysis/synonym.txt"
})
)
)
EsClient.CreateIndex("universal_de", c => c
.NumberOfReplicas(1)
.NumberOfShards(5)
.Settings(s => s //just as an example
.Add("merge.policy.merge_factor", "10")
.Add("search.slowlog.threshold.fetch.warn", "1s")
.Add("analysis.char_filter.drop_chars.type", "pattern_replace")
.Add("analysis.char_filter.drop_chars.pattern", #"[^0-9]")
.Add("analysis.char_filter.drop_chars.replacement", "")
.Add("analysis.char_filter.drop_specChars.type", "pattern_replace")
.Add("analysis.char_filter.drop_specChars.pattern", #"[^0-9a-zA-Z]")
.Add("analysis.char_filter.drop_specChars.replacement", "")
)
.Analysis(descriptor => descriptor
.Analyzers(bases => bases
.Add("folded_word", new CustomAnalyzer()
{
Filter = new List<string> { "lowercase", "asciifolding", "trim" },
Tokenizer = "standard"
}
)
.Add("trimmed_number", new CustomAnalyzer()
{
CharFilter = new List<string> { "drop_chars" },
Tokenizer = "standard",
Filter = new List<string>() { "lowercase" }
})
.Add("trimmed_specChars", new CustomAnalyzer()
{
CharFilter = new List<string> { "drop_specChars" },
Tokenizer = "standard",
Filter = new List<string>() { "lowercase" }
})
)
)
.AddMapping<Business>(m => m
//.MapFromAttributes()
.Properties(props => props
.MultiField(mf => mf
.Name(t => t.DirectoryName)
.Fields(fs => fs
.String(s => s.Name(t => t.DirectoryName).Analyzer("standard"))
.String(s => s.Name(t => t.DirectoryName.Suffix("folded")).Analyzer("folded_word"))
)
)
.MultiField(mf => mf
.Name(t => t.Phone)
.Fields(fs => fs
.String(s => s.Name(t => t.Phone).Analyzer("trimmed_number"))
)
)
This is how you create the index and add the mapping.
Now to search i have something like this :
var result = _Instance.Search<Business>(q => q
.TrackScores(true)
.Query(qq =>
{
QueryContainer termQuery = null;
if (!string.IsNullOrWhiteSpace(input.searchTerm))
{
var toLowSearchTerm = input.searchTerm.ToLower();
termQuery |= qq.QueryString(qs => qs
.OnFieldsWithBoost(f => f
.Add("directoryName.folded", 5.0)
)
.Query(toLowSearchTerm));
termQuery |= qq.Fuzzy(fz => fz.OnField("directoryName.folded").Value(toLowSearchTerm).MaxExpansions(2));
termQuery |= qq.Term("phone", Regex.Replace(toLowSearchTerm, #"[^0-9]", ""));
}
return termQuery;
})
.Skip(input.skip)
.Take(input.take)
);
NEW: I managed to use the pattern replace in a better way like this :
.Analysis(descriptor => descriptor
.Analyzers(bases => bases
.Add("folded_word", new CustomAnalyzer()
{
Filter = new List<string> { "lowercase", "asciifolding", "trim" },
Tokenizer = "standard"
}
)
.Add("trimmed_number", new CustomAnalyzer()
{
CharFilter = new List<string> { "drop_chars" },
Tokenizer = "standard",
Filter = new List<string>() { "lowercase" }
})
.Add("trimmed_specChars", new CustomAnalyzer()
{
CharFilter = new List<string> { "drop_specChars" },
Tokenizer = "standard",
Filter = new List<string>() { "lowercase" }
})
.Add("autocomplete", new CustomAnalyzer()
{
Tokenizer = new WhitespaceTokenizer().Type,
Filter = new List<string>() { "lowercase", "asciifolding", "trim", "engram" }
}
)
)
.TokenFilters(i => i
.Add("engram", new EdgeNGramTokenFilter
{
MinGram = 3,
MaxGram = 15
}
)
)
.CharFilters(cf => cf
.Add("drop_chars", new PatternReplaceCharFilter
{
Pattern = #"[^0-9]",
Replacement = ""
}
)
.Add("drop_specChars", new PatternReplaceCharFilter
{
Pattern = #"[^0-9a-zA-Z]",
Replacement = ""
}
)
)
)

sample of kendo combo for relation table with id and value

i'm using kendo ui in my asp.net mvc 4 with razor views and encounter problem with kendo combo when the list load from an action via ajax with sending parameters to the server like the sample here:HERE
becuase the table has more then 2,000 rows.
when i load the edit page, the combo load and filter the data as expected, but the value of this field is - [object object].
i did declare the .DataTextField("ProffName") + .DataValueField("ID")
My ClientsController:
public ActionResult Edit(int id = 0)
{
Clients clients = db.Clients.Find(id);
if (clients == null)
{
return HttpNotFound();
}
ViewData["MyAgency"] = new SelectList(db.Agency, "ID", "AgentName", clients.AgencyId);
ViewData["MyCategory1"] = new SelectList(db.CategoryTbl, "ID", "category", clients.CategoryId);
List<SelectListItem> obj = new List<SelectListItem>();
obj.Add(new SelectListItem { Text = "male", Value = "1" });
obj.Add(new SelectListItem { Text = "female", Value = "2" });
obj.Add(new SelectListItem { Text = "choose", Value = "0" });
ViewData["MyMin"] = obj;
ViewBag.ProffID = new SelectList(db.ProfTBL, "ID", "ProffName", clients.ProffID);
ViewBag.Metapel = new SelectList(db.Workers, "ID", "WorkerName", clients.Metapel);
return View(clients);
}
My ProffController:
public ActionResult ProffVM_Read(string text)
{
var Proff_Tbl = db.ProfTBL.Select(proff => new ProffVm { ID = proff.ID, ProffName = proff.ProffName });
if (!string.IsNullOrEmpty(text))
{
Proff_Tbl = Proff_Tbl.Where(p => p.ProffName.Contains(text));
}
return Json(Proff_Tbl, JsonRequestBehavior.AllowGet);
}
and the Kendo combo:
#Html.Label("Proff")
#(Html.Kendo().ComboBoxFor(model => model.ProffID)
.Name("proffCbo")
.DataTextField("ProffName")
.DataValueField("ID")
.Events(e => e
.Select("proffCbo_select")
.Change("proffCbo_change")
)
.DataSource(source =>
{
source.Read(read =>
{
read.Action("ProffVM_Read", "Proff")
.Data("onAdditionalData");
});
})
)
where am i wrong ???
i can change this combo to textbox but... i have to realize the magic.
Thanks
Change these two lines
var Proff_Tbl = db.ProfTBL.Select(proff => new ProffVm { ID = proff.ID, ProffName = proff.ProffName });
Proff_Tbl = Proff_Tbl.Where(p => p.ProffName.Contains(text));
to
var Proff_Tbl = db.ProfTBL.Select(proff => new ProffVm { ID = proff.ID, ProffName = proff.ProffName }).ToList();
Proff_Tbl = Proff_Tbl.Where(p => p.ProffName.Contains(text)).ToList();