Changes b/w ElasticSearch 1.x and 2.x - nest

Does documentation exist on how to change code written in NEST 1.x to 2.x?
I've looked at these sites and they're incomplete:
https://github.com/elastic/elasticsearch-net/blob/master/docs/2.0-breaking-changes/nest-breaking-changes.md
https://github.com/elastic/elasticsearch-net
https://www.elastic.co/blog/ga-release-of-nest-2-0-our-dot-net-client-for-elasticsearch
For example I'd like to know how to replace the following:
1)
given ISearchResponse<T> searchResults = ...
How to do:
searchResults.ConnectionStatus
searchResults.RequestInformation.Request
2)
client.Get<T>(s => s.Id(id));
3)
Given QueryContainer query
new SearchDescriptor<T>()
.From(from)
.Size(pageSize)
.Query(query); //this dosen't work anymore
4)
MatchQuery doesn't accept fuziness as double and type parameters as string as it used to
5) QueryDescriptor seems gone gasp
6) client.Update is busted
var result = client.Update<CustomerProfile>(request => request
.Id(customer.CustomerId)
.Doc(customer)
.Refresh()
);
7) client.Get is busted in a similar way to client.Update
8) In Mappings the following setup doesn't work anymore
CreateIndexDescriptor cid = ...
cid.NumberOfReplicas(numReplicas)
.NumberOfShards(numShards)
.Settings(s => s
.Add("merge.policy.merge_factor", "10")
.Add("search.slowlog.threshold.fetch.warn", "1s")
)
.Analysis(a => a.TokenFilters etc etc
EDIT
9) Date Ranges:
startDate and endDate are DateTime type
var qd = new QueryContainerDescriptor<EsActivity>();
QueryContainer qc = qd.Range(r =>
r.Field("esactivity.timestamp")
.GreaterThanOrEquals(DateMath.Anchored(startDate))
.LessThanOrEquals(DateMath.Anchored(endDate))
);
.GreaterThanOrEquals expects a double parameter but on the documentation page it takes DateMath.Anchored(startDate)
10) Highlighting:
highlightFields: List<string>
Action<HighlightFieldDescriptor<T>> [] tmp = highlightFields.Select(field =>
new Action<HighlightFieldDescriptor<T>>(
highlighter => highlighter.Field(field)
)
).ToArray();
sd:SearchDescriptor<..>..
sd.Highlight(h => h
.PreTags(preTag)
.PostTags(postTag)
.OnFields(tmp)
);
I see I can replace OnFields(tmp) with .Fields(f=>f.OnAll()) but I'd still like to specify the fields myself in some way.
And how come there is a HighlightQuery option available since we already apply highlighting on a query object.. now there are 2 query calls.
I've converted the highlighting above to
var tmp = highlightFields.Select(field =>
Tuple.Create<Field, IHighlightField>(
Field.Create(field),
new HighlightField()
)
).ToDictionary(x => x.Item1, x => x.Item2);
sd.Highlight(h => new Highlight
{
PreTags = new[] { preTag },
PostTags = new[] { postTag },
Fields = tmp
}
);

1) searchResults.ApiCall replaces searchResults .ConnectionStatus.
You can get the request bytes with searchResults.ApiCall.RequestBodyInBytes and you will also need to set .DisableDirectStreaming() on ConnectionSettings in order to capture the bytes as the request is written to the request stream directly by default.
2) Use client.Get<T>(id) - The first parameter is a DocumentPath<T> type.
3) To pass a QueryContainer to a Fluent API descriptor, just return it from the Func<QueryContainerDescriptor<T>, QueryContainer>
new SearchDescriptor<T>()
.From(from)
.Size(pageSize)
.Query(_ => query);
4) match query fuzziness as a double mapped to a formula to calculate edit distance in Elasticsearch 1.x. Since this was removed in Elasticsearch 2.x, it is also gone from NEST. You can set fuzziness edit distance with
client.Search<Document>(s => s
.Query(q => q
.Match(m => m
.Query("this is my query")
.Fuzziness(Fuzziness.EditDistance(3))
)
)
);
Not sure what you're referring to with type, but I think you're referring to document type? If that's the case, document type takes a Types type which string implicitly converts to
client.Search<Document>(s => s
.Type("other-type")
.MatchAll()
);
5) QueryDescriptor<T> was renamed to QueryContainerDescriptor<T> to better reflect the fact that it's a descriptor for building a QueryContainer
6) Update API works
// specifying id
client.Update<Document>("document-id", u => u
.Doc(document)
.Refresh()
);
Since the first parameter is a DocumentPath<T>, the document instance (if you have it) can be passed as the first parameter
client.Update<Document>(document, u => u
.Doc(document)
.Refresh()
);
where index, type and id will be inferred from the document instance
7) See above
8) Create index settings have been revised to reflect the level at which the settings appear in the REST API json call
client.CreateIndex("index-name", c => c
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(2)
.SlowLog(sl => sl
.Search(sls => sls
.Fetch(slsf => slsf
.ThresholdWarn("1s")
)
)
)
.Analysis(a => a) // etc...
)
);
You can also use strings for settings if you prefer, although the fluent API will ensure the correct setting values are sent e.g. "search.slowlog.threshold.fetch.warn" is now "index.search.slowlog.threshold.fetch.warn"
client.CreateIndex("index-name", c => c
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(2)
.Setting("index.search.slowlog.threshold.fetch.warn", "1s")
.Analysis(a => a) // etc...
)
);
merge.policy.merge_factor is removed in Elasticsearch 2.0

Related

Google Sheet API - How to get only updated rows

I could not able to find in the documentation on how to get only updated records/rows from Google Sheets API.
Is there a way, that I can get a timestamp of each record when it was last modified?
any guidance or any links that would solve this issue.
Thanks!
You cannot do this directly with Sheets API. You can keep track of the changes in a file using Drive API, though, but I don't think this is what you want to do.
I'd propose using an onEdit trigger using Apps Script. Every time the spreadsheet is modified, you could retrieve the data of the edited range and store it somewhere, as well as the current date.
It could be something on the following lines:
function onEdit(e) {
var timestamp = new Date();
var range = e.range;
var editedRow = range.getRow();
// Store timestamp and editedRow index somewhere you can retrieve it later (it could be in the spreadsheet itself)
}
Update:
You can create the trigger remotely using Apps Script API. First you should create a project bound to your spreadsheet and then add the corresponding code by calling projects.updateContent (you should add two files, the script itself, which contains the onEdit trigger, and the manifest file). Just beware that you can only use simple triggers with this API, not installable ones. But in your situation, that's more than enough.
I hope this is of any help.
WRITE OPERATION:
For $response = $service->spreadsheets_values->update() process (Writting file after creating it), response will be as follow:
Google_Service_Sheets_UpdateValuesResponse Object
(
[spreadsheetId] => XXX
[updatedCells] => 7
[updatedColumns] => 7
[updatedDataType:protected] => Google_Service_Sheets_ValueRange
[updatedDataDataType:protected] =>
[updatedRange] => Sheet1!A1:G1
[updatedRows] => 1
[internal_gapi_mappings:protected] => Array
(
)
[modelData:protected] => Array
(
)
[processed:protected] => Array
(
)
)
To Get Rows = $response->getUpdatedRows();
To Get Cells = $response->getUpdatedCells();
To Get Columns= $response->getUpdatedColumns();
and so on...
APPEND OPERATION:
For $response = $service->spreadsheets_values->append() process, response will be as follow:
Google_Service_Sheets_AppendValuesResponse Object
(
[spreadsheetId] => XXXX
[tableRange] => Sheet1!A1:G1
[updatesType:protected] => Google_Service_Sheets_UpdateValuesResponse
[updatesDataType:protected] =>
[internal_gapi_mappings:protected] => Array
(
)
[modelData:protected] => Array
(
[updates] => Array
(
[spreadsheetId] => XXXX
[updatedRange] => Sheet1!A2:G7
[updatedRows] => 6
[updatedColumns] => 7
[updatedCells] => 42
)
)
[processed:protected] => Array
(
)
)
To Get Rows = $response->getUpdates()->getUpdatedRows();
To Get Cells = $response->getUpdates()->getUpdatedCells();
To Get Columns= $response->getUpdates()->getUpdatedColumns();
and so on...

Does RavenDb's `Include` support constructing a document id, not just selecting one?

This code here is not working as expected. Specifically, it seems the calls to Include are not causing any extra documents to be added to the session. Thus, the Loads are each costing a full db hit. This is RavenDb 3.5.
I couldn't find examples that actually use Include this way. They all seem to emit a "selector" that is used to find a member on the document, and that member holds the literal document id to include. Whereas I am constructing the document id and returning it as a string.
session.Query<Coil>()
.Include(c => nameof(CoilState) + "/" + id)
.Include(c => nameof(CoilExt) + "/" + id)
.Include(c => nameof(Material) + "/" + c.MaterialCode)
.Where(c => c.CoilId == id)
.ToList()
.Select(c =>
Coil_Dto.ToCoilDto(
c
, session.Load<CoilState>(nameof(CoilState) + "/" + c.CoilId)
, session.Load<CoilExt>(nameof(CoilExt) + "/" + c.CoilId)
, session.Load<Material>(nameof(Material) + "/" + c.MaterialCode)
)
)
.SingleOrDefault()
My question is: can Include be used in this way? If not, is there some way to achieve that "include" functionality using my constructed ids?
It sounds like you want to use Lazy, rather than Include.
Include
The .Include method should be passed the name of a property on your object(s) which contains an ID.
// Good:
// Note that StateId, ExtId, and MaterialId are all properties on our Coil class.
session.Query<Coil>()
.Include(c => c.StateId)
.Include(c => c.ExtId)
.Inclide(c => c.MaterialId);
// Bad:
session.Query<Coil>
.Include(c => "CoilStates/123")
.Include(c => "CoilExts/456")
.Include(c => "Materials/789")
In short, make sure your .Include calls are passed the name of properties on the Coil class. Those properties should contain string values which are IDs of other documents. Then they'll all be loaded in a single trip to the database.
Lazy
If Include doesn't make sense for your scenario, but you still want to load disparate objects in a single DB call, use the lazy API
// Lazily load the coil. No DB trip yet.
var lazyCoil = session.Query<Coil>()
.Where(...)
.Lazily();
// Lazily load a CoilState. No DB trip yet.
var lazyCoilState = session.Advanced.Lazily.Load<CoilState>("CoilStates/123");
// Lazily load a Material. Still no DB trip.
var lazyMaterial = session.Advanced.Lazily.Load<Material>("Materials/456");
// Grab one of the values. This will fetch all lazy loaded items in 1 trip.
var coil = lazyCoil.Value;
// Grab the other values. No DB trip needed; they're already loaded!
var coilState = lazyCoilState.Value;
var material = lazyMaterial.Value;

Elasticsearch Autocomplete Issues

I am trying to create an Autocompletion using Elasticsearch.net, but i keep getting an Invalid response.
But cant figure out why?
My Request looks like this
var descriptor = new SearchDescriptor<EmployeeDocument>()
.Index("employees").Type("employee").From(page - 1).Size(pageSize)
.Suggest(
s => s.Completion(
"my-completion-suggest",
c => c
.Field(f1 => f1.Description)
.Field(f1 => f1.empfirstname)
.Contexts(
queriesDescriptor => queriesDescriptor.Context(
"query-descriptor",
queryDescriptor => queryDescriptor.Prefix(true).Context(query)))));
var response3 = await this.client.SearchAsync<EmployeeDocument>(descriptor);
the error i am getting is
Invalid NEST response built from a unsuccessful low level call on POST: /employees/employee/_search
Audit trail of this API call:
- [1] BadResponse: Node: http://192.168.2.29:9200/ Took: 00:00:00.3543244
ServerError: ServerError: 400Type: search_phase_execution_exception Reason: "all shards failed"
This is how i am calling the method
var results1 = await service.SearchAsync("brenda", page, pageSize);
var results8 = await service.SearchAsync("something else", page, pageSize);
My model is also very straightforward. I left out some properties
[ElasticsearchType(Name = "employee")]
public class EmployeeDocument
{
//[Text(Name = "pkempid")]
public long pkempid { get; set; }
//[Text(Name = "empfirstname")]
public string empfirstname { get; set; }
}
Description and empfirstname need to be mapped as completion field data types. The CompletionField type in NEST can be used for the property type, which will be mapped as a completion data type through automapping.
Additionally, a completion suggester can only specify one field, so chaining multiple calls to .Field() will not work as expected (the last call will be the field used). You can however specify multiple suggesters in one request targeting different fields. It's more usual though, rather than having multiple completion fields in a mapping, to specify multiple input values to a single completion field.
The use case for the completion suggester is to provide fast "search as you type" autocompletion functionality, trading off the power of more complex analysis chains that can be performed with text field data types.
Ok, so I figured out how to do autocomplete, I ended up using Edge NGram Tokenizer First thing I needed to do was setup my indexes with the correct filters.
var response = this.client.CreateIndex(
ElasticConfig.IndexName,
index => index.Mappings(
ms => ms.Map<EmployeeDocument>(
m => m.Properties(
p => p
.Text(t => t.Name(n => n.EmpFirstName).Analyzer("auto-complete").Fields(ff => ff.Keyword(k => k.Name("keyword"))))
.Text(t => t.Name(n => n.pkEmpID).Analyzer("auto-complete-id").Fields(ff => ff.Keyword(k => k.Name("keyword"))))
.Text(t => t.Name(n => n.Description).Analyzer("auto-complete").Fields(ff => ff.Keyword(k => k.Name("keyword")))))))
.Settings(f => f.Analysis(
analysis => analysis
.Analyzers(
analyzers => analyzers
.Custom("auto-complete", a => a.Tokenizer("standard").Filters("lowercase", "auto-complete-filter"))
.Custom("auto-complete-id", a => a.Tokenizer("standard").Filters("lowercase", "auto-complete-id-filter")))
.TokenFilters(tokenFilter => tokenFilter
.EdgeNGram("auto-complete-filter", t => t.MinGram(3).MaxGram(5))
.EdgeNGram("auto-complete-id-filter", t => t.MinGram(1).MaxGram(5))))));
Then for the actual search
var response = await this.client.SearchAsync<EmployeeDocument>(
x => x.Index("default-index").Type("employee").From(page - 1).Size(pageSize)
.Query(q => q
.MultiMatch(m => m
.Query(query)
.Fields(f => f
.Field(_ => _.EmpFirstName)
.Field(_ => _.pkEmpID)
.Field(_ => _.Description))))
.Highlight(
h => h.PreTags("<mark>").PostTags("</mark>").Fields(
f => f.Field(p => p.EmpFirstName),
f => f.Field(p => p.pkEmpID),
f => f.Field(p => p.Description))));

PDO: Passing extra parameters to a prepared statment than needed

Can you send more parameters than needed to a prepared statement using PDO with no undesired side effects?
That mights seem like a strange question but I ask because I have 4 queries in a row which all use similar and different parameters. The relevant parts of the queries:
1st (select, different table to others):
WHERE threadID = :tid
2nd (select):
WHERE user_ID = :u_ID AND thread_ID = :tid
3rd (update if 2nd was successful):
SET time = :current_time WHERE user_ID = :u_ID AND thread_ID = :tid
4th (insert if 2nd was unsuccessful):
VALUES (:u_ID, :tid, :current_time)
Can I declare one array with the three parameters at the beginning and use it for all 4 queries?
To sort out any confusion, the queries would be executed seperately. It is the parameters variable being reused and so that would mean some queries would receive parameters they don't need. So something like:
$parameters = array(':tid' => $tid, ':u_ID' => $u_ID, ':current_time' => $time);
$1st = $db->prepare($query1);
$1st->execute($parameters);
$2nd = $db->prepare($query2);
$2nd->execute($parameters);
$3rd = $db->prepare($query3);
$3rd->execute($parameters);
$4th = $db->prepare($query4);
$4th->execute($parameters);
If I can, should I? Will this slow down or cause security flaws to my database or scripts?
If I can make this question a bit clearer, please ask.
Thank you!
Perhaps the documentation has been updated since this question was first asked, but now it is quite clearly stated "No"
You cannot bind more values than specified; if more keys exist in input_parameters than in the SQL specified in the PDO::prepare(), then the statement will fail and an error is emitted.
These answers should be useful in filtering out the extra parameters.
I know this is already answered and it's only asking about whether you can send extra params, but I thought people might arrive at this question, and want to know how to get around this limitation. Here's the solution I use:
$parameters = array('tid' => $tid, 'u_ID' => $u_ID, 'current_time' => $time);
$1st = $db->prepare($query1);
$1st->execute(array_intersect_key($parameters, array_flip(array('tid'))));
$2nd = $db->prepare($query2);
$2nd->execute(array_intersect_key($parameters, array_flip(array('u_ID', 'tid'))));
$3rd = $db->prepare($query3);
$3rd->execute(array_intersect_key($parameters, array_flip(array('u_ID', 'tid', 'current_time'))));
$4th = $db->prepare($query4);
$4th->execute(array_intersect_key($parameters, array_flip(array('u_ID', 'tid', 'current_time'))));
That array_interset_key and array_flip maneuver could be extracted to its own function, like:
function filter_fields($params,$field_names) {
return array_intersect_key($params, array_flip($field_names))
}
I just haven't got around to it yet.
The function flips your array of key names, so you have an array with no values, but the right keys. Then intersect filters the first array so you only have the keys that are in both arrays (in this case, only the ones in your array_flipped array). But you get the values for the original array (not the empties). So you make one array of parameters, but specify which params are actually sent to PDO.
So, with the function, you'd do:
$parameters = array('tid' => $tid, 'u_ID' => $u_ID, 'current_time' => $time);
$1st = $db->prepare($query1);
$1st->execute(filter_fields($parameters, array('tid')));
$2nd = $db->prepare($query2);
$2nd->execute(filter_fields($parameters, array('u_ID', 'tid')));
$3rd = $db->prepare($query3);
$3rd->execute(filter_fields($parameters, array('u_ID', 'tid', 'current_time')));
$4th = $db->prepare($query4);
$4th->execute(filter_fields($parameters, array('u_ID', 'tid', 'current_time')));
If you have PHP 5.4, you can use the square bracket array syntax, to make it even cooler:
$parameters = array('tid' => $tid, 'u_ID' => $u_ID, 'current_time' => $time);
$1st = $db->prepare($query1);
$1st->execute(filter_fields($parameters, ['tid']));
$2nd = $db->prepare($query2);
$2nd->execute(filter_fields($parameters, ['u_ID', 'tid']));
$3rd = $db->prepare($query3);
$3rd->execute(filter_fields($parameters, ['u_ID', 'tid', 'current_time']));
$4th = $db->prepare($query4);
$4th->execute(filter_fields($parameters, ['u_ID', 'tid', 'current_time']));
I got a chance to test my question, and the answer is you cannot send more parameters than the query uses. You get the following error:
PDOException Object
(
[message:protected] => SQLSTATE[HY093]: Invalid parameter number: parameter was not defined
[string:Exception:private] =>
[code:protected] => HY093
[file:protected] => C:\Destination\to\file.php
[line:protected] => line number
[trace:Exception:private] => Array
(
[0] => Array
(
[file] => C:\Destination\to\file.php
[line] => line number
[function] => execute
[class] => PDOStatement
[type] => ->
[args] => Array
(
[0] => Array
(
[:u_ID] => 1
[:tid] => 1
[:current_time] => 1353524522
)
)
)
[1] => Array
(
[file] => C:\Destination\to\file.php
[line] => line number
[function] => function name
[class] => class name
[type] => ->
[args] => Array
(
[0] => SELECT
column
FROM
table
WHERE
user_ID = :u_ID AND
thread_ID = :tid
[1] => Array
(
[:u_ID] => 1
[:tid] => 1
[:current_time] => 1353524522
)
)
)
)
[previous:Exception:private] =>
[errorInfo] => Array
(
[0] => HY093
[1] => 0
)
)
I don't know a huge amount about PDO, hence my question, but I think that because :current_time is sent but not used and the error message is "Invalid parameter number: parameter was not defined" you cannot send extra parameters which are not used.
Additionally the error code HY093 is generated. Now I can't seem to find any documentation explaining PDO codes anywhere, however I came across the following two links specifically about HY093:
What is PDO Error HY093
SQLSTATE[HY093]
It seems HY093 is generated when you incorrectly bind parameters. This must be happening here because I am binding too many parameters.
executing different type of multiple queries with one execute leads to problems. you can run multiple selects or multiple updates with one execute. For this case to create different prepared statements objects and pass the the parameters accordingly.
// for WHERE threadID = :tid
$st1 = $db->prepare($sql);
$st1->bindParam(':tid', $tid);
$st1->execute();
or
$st1->execute(array(':tid'=>$tid);
// for WHERE user_ID = :u_ID AND thread_ID = :tid
$st2 = $db->prepare($sql);
$st2->bindParam(':u_ID', $u_ID);
$st2->bindParam(':tid', $tid);
$st2->execute();
or
$st2->execute(array(':tid'=>$tid, ':u_ID' => $u_ID);
// for SET time = :current_time WHERE user_ID = :u_ID AND thread_ID = :tid
$st3 = $db->prepare($sql);
$st3->bindParam(':u_ID', $u_ID);
$st3->bindParam(':tid', $tid);
$st3->bindParam(':current_time', $current_time);
$st3->execute();
or
$st3->execute(array(':tid'=>$tid, ':u_ID' => $u_ID, ':current_time' => $current_time);
// for VALUES (:u_ID, :tid, :current_time)
$st4 = $db->prepare($sql);
$st4->bindParam(':u_ID', $u_ID);
$st4->bindParam(':tid', $tid);
$st4->bindParam(':current_time', $current_time);
$st4->execute();
or
$st4->execute(array(':tid'=>$tid, ':u_ID' => $u_ID, ':current_time' => $current_time);

NHibernate 3 - type safe way to select a distinct list of values

I am trying to select a distinct list of values from a table whilst ordering on another column.
The only thing working for me so far uses magic strings and an object array. Any better (type-safe) way?
var projectionList = Projections.ProjectionList();
projectionList.Add(Projections.Property("FolderName"));
projectionList.Add(Projections.Property("FolderOrder"));
var list = Session.QueryOver<T>()
.Where(d => d.Company.Id == SharePointContextHelper.Current.CurrentCompanyId)
.OrderBy(t => t.FolderOrder).Asc
.Select(Projections.Distinct(projectionList))
.List<object[]>()
.ToList();
return list.Select(l => new Folder((string)l[0])).ToList();
btw, doing it with linq won't work, you must select FolderOrder otherwise you'll get a sql error (ORDER BY items must appear in the select list if SELECT DISTINCT is specified.
)
and then doing that gives a known error : Expression type 'NhDistinctExpression' is not supported by this SelectClauseVisitor. regarding using anonymous types with distinct
var q = Session.Query<T>()
.Where(d => d.Company.Id == SharePointContextHelper.Current.CurrentCompanyId)
.OrderBy(d => d.FolderOrder)
.Select(d => new {d.FolderName, d.FolderOrder})
.Distinct();
return q.ToList().Select(f => new Folder(f));
All seems a lot of hoops and complexity to do some sql basics....
To resolve the type-safety issue, the syntax is:
var projectionList = Projections.ProjectionList();
projectionList.Add(Projections.Property<T>(d => d.FolderName));
projectionList.Add(Projections.Property<T>(d => d.FolderOrder));
the object [] thing is unavoidable, unless you define a special class / struct to hold just FolderName and FolderOrder.
see this great introduction to QueryOver for type-saftey, which is most certainly supported.
best of luck.