I am new in elastic search.
How I can convert the following SQL statement into elastic search query?
select sum(totaldevicecount),datasource from
(select distinct oskey,custkey,productkey,
timekey,totaldevicecount,datasource from es_reporting_data_new)
group by datasource;
Thanks
After simplifying your query to below
select sum(totaldevicecount),datasource from es_reporting_data_new group by datasource;
ES query would be
{
"aggs": {
"data_source": {
"terms": {
"field": "datasource"
},
"aggs": {
"total_device_count": {
"sum": {
"field": "totaldevicecount"
}
}
}
}
}
}
For more details see also Elastic Search Sum aggregation with group by and where condition.
terms query for distinct, filters for groupBy and aggregations for nested selects and sum if I remember correctly.
As answer is already provided. I'll add extra bit.
I would highly recommend reading elastic documenatation assuming you need to convert many queries like this.
As elastic query parameter is different from sql. So in depth understanding of elastic query param will help you convert SQL query if possible.
Logstash input support JDBC inputs. One way to execute the queries and load the resultset is supported by ELK. You can directly use the queries or preferred views for larger queries. Your output will be directly ingested into ELK.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html
The other way, if you want to query the indexes, you can writes queries directly in Dev tools like below:
POST _xpack/sql?format=txt
{
"query":"DESCRIBE \"indice-name-*\""
}
POST _xpack/sql
{
"query":"Select data1, data2 from \"indice-name-*\" where data1 = 'ABC' and ID = '11223333'"
}
Related
I have a JSON document within CosmosDB that looks like this
{
"id":"1234-11111-22222-2222"
"Feedbacks": [
{ "id": "abc"
}
]
}
I would like to first find all the documents where the ids are >15 and then loop over and find all the feedbacks that are associated with this result. How would I do this using the LogicApp? I have an idea and this is what I have done so far:
I am not sure if I have to perform another query? if so, what would this be?
Select c.Feedbacks from ...?
Any help or suggestion is appreciated.
Just ad I know about logic, we can not do that. Your really purpose is that get the all the data which id>15 and update the Feedback ids.
I think Data Factory is more suitable for you.
You could using Data flow to achieve that. Create Cosmos DB as Source, using Mapping data flow transformation to update the Feedback ids.
Ref tutorial: Copy and transform data in Azure Cosmos DB (SQL API) by using Azure Data Factory
Update:
We can do a nested query like: for each (document d array) // iterate over documents array do for each (feedback f in d.feedback) // iterate over feedback array in each document do something with f ... done done
Thanks for the sharing.
I am developing a system using Google Data-Store, where there's a Kind - Posts and which has 2 properties
1. message (string)
2. hashtags (list)
I wanted to query the distinct hashtags with the number. For example
say The posts are
{
{
"message":"msg1",
"tags":["abc","cde","efr"]
},
{
"message":"msg2",
"tags":["abc,"efgh","efk"]
},
{
"message":"msg3",
"tags":["abc,"efgh","efr"]
}
}
The output should be
{
"abc":3
"cde":1
"efk":1
"efgh":2
"efr":2
}
But in NoSQL implementation Data-store I can't directly query this. In order to query I have to load all the messages and find distinct queries. It will be a time-consuming event.
But I have seen a distinct function db.collection.distinct() which I think might have optimize this problem. If It has to be done on any NoSQL what may be the optimum solution for this?
Unfortunately, projection queries with 'distinct on' will only return a single result per distinct value (https://cloud.google.com/datastore/docs/concepts/queries#projection_queries). It will not provide a count of each distinct value. You'll need to do the count yourself, but you can use a projection query to save cost by only returning the tag values instead of the full entities.
I want use gather all data within a database in RavenDB. Similar to SELECT * in SQL.
from #all_docs
select {
Name: Value
}
Instead of typing in all possible names and their corresponding values, I am wondering if there is an "all" character.
from #all_docs
should return all the documents
I have this function that takes data from the database and also has search. The problem is that when I search with Entity framework it's slow, but if I use the same query I got from the log and use it in SSMS it's fast. I must also say that there are allot of movies, 388262. I also tried adding an index on title at movie, but didn't help.
Query I use in SSMS:
SELECT *
FROM Movie
WHERE title LIKE '%pirate%'
ORDER BY ##ROWCOUNT
OFFSET 0 ROWS FETCH NEXT 30 ROWS ONLY
Entity code (_movieRepository.GetAll() returns Queryable not all movies):
public IActionResult Index(MovieIndexViewModel vm) {
IQueryable<Movie> query = _movieRepository.GetAll().AsNoTracking();
if (!string.IsNullOrWhiteSpace(vm.Search)) {
query = query.Where(m => m.title.ToLower().Contains(vm.Search.ToLower()));
}
vm.TotalItemCount = query.Count();
vm.Movies = query.Skip(_pageSize * (vm.Page - 1)).Take(_pageSize);
vm.PageSize = _pageSize;
return View(vm);
}
Caveat: I don't have much experience with the Entity framework.
However, you might find useful debugging tips available in the Entity Framework Performance Article from Simple talk. Looking at what you've posted you might be able to improve your query performance by:
Choosing only the specific column you're interested in (it sounds like you're only interested in querying for the 'Title' column).
Pay special attention to your data-types. You might want to convert your NVARCHAR variables to VARCHAR(40) (or some appropriate character limit)
try removing all of the ToLower() stuff,
if (!string.IsNullOrWhiteSpace(vm.Search)) {
query = query.Where(m => m.title.Contains(vm.Search)));
}
sql server (unlike c#) is not case sensitive by default (though you can configure it to be that way). Your query is forcing sql server to lower case every record in the table and then do the comparison.
If using a SQL 'Like' statement to query data it will return data even if its only partially matched. For instance, if I'm searching for food and there's an item in my db called "raisins" when using SQL the query would return "raisins" even if my search only contained "rai". In elasticsearch, the query won't return a record unless the entire name (in this case "raisins") is specified. How can I get elasticsearch to behave similar to the SQL statement. I'm using Rails 3.1.1 and PostgreSQL. Thanks!
While creating index of model for elasticsearch use tokenizer on index which will fulfil your requirement. For. e.g.
tokenizer: {
:name_tokenizer => {type: "edgeNGram", max_gram: 100, min_gram: 3, side: "front"}
}
this will create tokens of size from 3 to 100 of your fields and as side is given as front it will check from the starting. You can get more details from here http://www.slideshare.net/clintongormley/terms-of-endearment-the-elasticsearch-query-dsl-explained