What is the ordering of the keys in an Ignite cache (without using indexing) and is it possible to do the equivalent of the following RocksDB snippet
try (final RocksIterator rocksIterator =
rocksDB.newIterator(columnFamilyHandleList.get(1))) {
for (rocksIterator.seek(prefixKey);
i.e. jump to the next entry starting with a given byte[] or String?
The way you'd do that in Ignite is by using SQL.
var query = new SqlFieldsQuery("select x,y,z from table where z like ? order by x").setArgs("prefix%");
try (var cursor = cache.query(query)) {
for (var r : cursor) {
Long id = (Long) r.get(0);
BigDecimal value = (BigDecimal) r.get(1);
String name = (String) r.get(2);
}
}
Related
I'm trying to run scan on my redis cluster using Jedis. I tried using the .scan(...) method as follows for a match pattern but I get the following error:
"JedisCluster only supports SCAN commands with MATCH patterns containing hash-tags"
my code is as follows (excerpted):
private final JedisCluster redis;
...
String keyPrefix = "helloWorld:*";
ScanParams params = new ScanParams()
.match(keyPrefix)
.count(100);
String cur = SCAN_POINTER_START;
boolean done = false;
while (!done) {
ScanResult<String> resp = redis.scan(cur, params);
...
cur = resp.getStringCursor();
if (resp.getStringCursor().equals(SCAN_POINTER_START)) {
done = true;
}
}
When I run my code, it gives a weird error talking about hashtags:
"JedisCluster only supports SCAN commands with MATCH patterns containing hash-tags"
In the redis-cli I could just use match patterns like that what I wrote for the keyPrefix variable. Why am I getting an error?
How do I get Jedis to show me all the the keys that match a given substring?
The problem was that the redis variable is a RedisCluster object and not a Redis object.
A redis cluster object has a collection of nodes and scanning that is different than scanning an individual node.
To solve the issue, you can scan through each individual node as such:
String keyPrefix = "helloWorld:*";
ScanParams params = new ScanParams()
.match(keyPrefix)
.count(100);
redis.getClusterNodes().values().stream().forEach(pool -> {
boolean done = false;
String cur = SCAN_POINTER_START;
try (Jedis jedisNode = pool.getResource()) {
while (!done) {
ScanResult<String> resp = jedisNode.scan(cur, params);
...
cur = resp.getStringCursor();
if (cur.equals(SCAN_POINTER_START)) {
done = true;
}
}
}
});
I use Hibernate Search 5.11 on my Spring Boot 2 application, allowing to make full text research.
This librairy require to index documents.
When my app is launched, I try to re-index manually data of an indexed entity (MyEntity.class) each five minutes (for specific reason, due to my server context).
I try to index data of the MyEntity.class.
MyEntity.class has a property attachedFiles, which is an hashset, filled with a join #OneToMany(), with lazy loading mode enabled :
#OneToMany(mappedBy = "myEntity", cascade = CascadeType.ALL, orphanRemoval = true)
private Set<AttachedFile> attachedFiles = new HashSet<>();
I code the required indexing process, but an exception is thrown on "fullTextSession.index(result);" when attachedFiles property of a given entity is filled with one or more items :
org.hibernate.TransientObjectException: The instance was not associated with this session
The debug mode indicates a message like "Unable to load [...]" on entity hashset value in this case.
And if the HashSet is empty (not null, only empty), no exception is thrown.
My indexing method :
private void indexDocumentsByEntityIds(List<Long> ids) {
final int BATCH_SIZE = 128;
Session session = entityManager.unwrap(Session.class);
FullTextSession fullTextSession = Search.getFullTextSession(session);
fullTextSession.setFlushMode(FlushMode.MANUAL);
fullTextSession.setCacheMode(CacheMode.IGNORE);
CriteriaBuilder builder = session.getCriteriaBuilder();
CriteriaQuery<MyEntity> criteria = builder.createQuery(MyEntity.class);
Root<MyEntity> root = criteria.from(MyEntity.class);
criteria.select(root).where(root.get("id").in(ids));
TypedQuery<MyEntity> query = fullTextSession.createQuery(criteria);
List<MyEntity> results = query.getResultList();
int index = 0;
for (MyEntity result : results) {
index++;
try {
fullTextSession.index(result); //index each element
if (index % BATCH_SIZE == 0 || index == ids.size()) {
fullTextSession.flushToIndexes(); //apply changes to indexes
fullTextSession.clear(); //free memory since the queue is processed
}
} catch (TransientObjectException toEx) {
LOGGER.info(toEx.getMessage());
throw toEx;
}
}
}
Does someone have an idea ?
Thanks !
This is probably caused by the "clear" call you have in your loop.
In essence, what you're doing is:
load all entities to reindex into the session
index one batch of entities
remove all entities from the session (fullTextSession.clear())
try to index the next batch of entities, even though they are not in the session anymore... ?
What you need to do is to only load each batch of entities after the session clearing, so that you're sure they are still in the session when you index them.
There's an example of how to do this in the documentation, using a scroll and an appropriate batch size: https://docs.jboss.org/hibernate/search/5.11/reference/en-US/html_single/#search-batchindex-flushtoindexes
Alternatively, you can just split your ID list in smaller lists of 128 elements, and for each of these lists, run a query to get the corresponding entities, reindex all these 128 entities, then flush and clear.
Thanks for the explanations #yrodiere, they helped me a lot !
I chose your alternative solution :
Alternatively, you can just split your ID list in smaller lists of 128 elements, and for each of these lists, run a query to get the corresponding entities, reindex all these 128 entities, then flush and clear.
...and everything works perfectly !
Well seen !
See the code solution below :
private List<List<Object>> splitList(List<Object> list, int subListSize) {
List<List<Object>> splittedList = new ArrayList<>();
if (!CollectionUtils.isEmpty(list)) {
int i = 0;
int nbItems = list.size();
while (i < nbItems) {
int maxLastSubListIndex = i + subListSize;
int lastSubListIndex = (maxLastSubListIndex > nbItems) ? nbItems : maxLastSubListIndex;
List<Object> subList = list.subList(i, lastSubListIndex);
splittedList.add(subList);
i = lastSubListIndex;
}
}
return splittedList;
}
private void indexDocumentsByEntityIds(Class<Object> clazz, String entityIdPropertyName, List<Object> ids) {
Session session = entityManager.unwrap(Session.class);
List<List<Object>> splittedIdsLists = splitList(ids, 128);
for (List<Object> splittedIds : splittedIdsLists) {
FullTextSession fullTextSession = Search.getFullTextSession(session);
fullTextSession.setFlushMode(FlushMode.MANUAL);
fullTextSession.setCacheMode(CacheMode.IGNORE);
Transaction transaction = fullTextSession.beginTransaction();
CriteriaBuilder builder = session.getCriteriaBuilder();
CriteriaQuery<Object> criteria = builder.createQuery(clazz);
Root<Object> root = criteria.from(clazz);
criteria.select(root).where(root.get(entityIdPropertyName).in(splittedIds));
TypedQuery<Object> query = fullTextSession.createQuery(criteria);
List<Object> results = query.getResultList();
int index = 0;
for (Object result : results) {
index++;
try {
fullTextSession.index(result); //index each element
if (index == splittedIds.size()) {
fullTextSession.flushToIndexes(); //apply changes to indexes
fullTextSession.clear(); //free memory since the queue is processed
}
} catch (TransientObjectException toEx) {
LOGGER.info(toEx.getMessage());
throw toEx;
}
}
transaction.commit();
}
}
I have a native query
searchSql = "Select firstname,lastname from students order by id desc"
and then i do
Query query = entityManager.createNativeQuery(searchSql);
List<Map<String, Object>> results = query.getResultList();
Now if i print the results with KEYS
List<String>headers = new ArrayList<>();
for(String header : results.get(0).keySet()){
headers.add(header);
}
i get random order of the column names.
How can i get the exact order as in the select statement ?
LinkedHashMap should be the answer but i get class cast exceptions... Any generic ideas?
I checked if LinkedHashMap causes the randomness of columns:
#Test
public void linkedHashMap() {
Map<String, Integer> map = new LinkedHashMap<>();
IntStream.range(0, 10).forEach(integer -> {
map.put(
String.valueOf(integer),
integer
);
});
for (String val : map.keySet()){
System.out.println(val);
}
}
but instead it prints:
0
1
2
3
4
5
6
7
8
9
The randomness seems to be a limitation of Hibernate, as stated here
If simply changing the signature to List<Object[]> results = query.getResultList(); does NOT work, then you can give a shot at SqlResultSetMapping. That way, the implicit map insertion would be avoided and order would be maintained I guess (I am not 100% sure, needs to be tested)
See https://docs.oracle.com/javaee/7/api/javax/persistence/SqlResultSetMapping.html.
I have a problem with RavenDB indexing.
Simple query looks like this:
var values =
myCollection.Query.Where(w =>
w.MyId == MyId &&
w.IsReady == false &&
w.IsDeleted &&
w.Rate > 0)
During execution Raven creates dynamic index:
from doc in docs.MyCollection
select new { Rate = doc.Rate, IsReady = doc.IsReady, IsDeleted = doc.IsDeleted, MyId = doc.MyId }
with extra options:
Field -> Rate;
Storage -> No;
Indexing -> Default;
Sort -> Double;
Field Rate has decimal type.
Problem:
I wanted to add static index, but when I specified index like this:
public class MyIndex : AbstractIndexCreationTask<MyCollection> {
public MyIndex () {
Map = d => d.Select(s => new { Rate = s.Rate, IsReady = s.IsReady, IsDeleted = s.IsDeleted, MyId = s.MyId });
Sort(x => x.Rate, SortOptions.Double);
}
}
Raven is creating index slightly different:
from doc in docs.MyCollection
select new { Rate = (decimal)doc.Rate, IsReady = doc.IsReady, IsDeleted = doc.IsDeleted, MyId = doc.MyId }
with extra options:
Field -> Rate;
Storage -> No;
Indexing -> Default;
Sort -> Double;
The only difference is that I have casting in static index, because my field type is decimal and I'm using Double sort option.
Because of that Raven is not using my static index but instead creates dynamic one every time query is being executed.
I tried to do some casting inside Sort() but then index has not been created at all. One way to overcome this issue is to manually modify static index from management console after it was created, but it's not good solution.
Any ideas how to deal with that?
Thanks.
Edit:
Another example:
Type of field DateTime and querying using DateTime values as predicates (greater than / less than). Raven in dynamic index creation picks String as a SortOption, and when I try to prepare static index I get casting issue.
You can use the IDocumentSession.Query(string indexName, [bool isMapReduce]) or the IDocumentSession.Query<TResult, TIndexCreator>() overloads to explicitly specify a static index. So in your specific case, either IDocumentSession.Query<MyCollection, MyIndex>() or IDocumentSession.Query("MyIndex").
Using Dapper-dot-net...
The following yields no results in the data object:
var data = conn.Query(#"
select top 25
Term as Label,
Type,
ID
from SearchTerms
WHERE Term like '%#T%'",
new { T = (string)term });
However, when I just use a regular String Format like:
string QueryString = String.Format("select top 25 Term as Label, Type, ID from SearchTerms WHERE Term like '%{0}%'", term);
var data = conn.Query(QueryString);
I get 25 rows back in the collection. Is Dapper not correctly parsing the end of the parameter #T?
Try:
term = "whateverterm";
var encodeForLike = term => term.Replace("[", "[[]").Replace("%", "[%]");
string term = "%" + encodeForLike(term) + "%";
var data = conn.Query(#"
select top 25
Term as Label,
Type,
ID
from SearchTerms
WHERE Term like #term",
new { term });
There is nothing special about like operators, you never want your params inside string literals, they will not work, instead they will be interpreted as a string.
note
The hard-coded example in your second snippet is strongly discouraged, besides being a huge problem with sql injection, it can cause dapper to leak.
caveat
Any like match that is leading with a wildcard is not SARGable, which means it is slow and will require an index scan.
Yes it does. This simple solution has worked for me everytime:
db.Query<Remitente>("SELECT *
FROM Remitentes
WHERE Nombre LIKE #n", new { n = "%" + nombre + "%" })
.ToList();
Best way to use this to add concat function in query as it save in sql injecting as well, but concat function is only support above than sql 2012
string query = "SELECT * from country WHERE Name LIKE CONCAT('%',#name,'%');"
var results = connection.query<country>(query, new {name});
The answer from Sam wasn't working for me so after some testing I came up with using the SQLite CONCAT equivalent which seems to work:
string sql = "SELECT * FROM myTable WHERE Name LIKE '%' || #NAME || '%'";
var data = IEnumerable data = conn.Query(sql, new { NAME = Name });
Just to digress on Sam's answer, here is how I created two helper methods to make searches a bit easier using the LIKE operator.
First, creating a method for generating a parameterized query, this method uses dynamic: , but creating a strongly typed generic method should be more desired in many cases where you want static typing instead of dynamic.
public static dynamic ParameterizedQuery(this IDbConnection connection, string sql, Dictionary<string, object> parametersDictionary)
{
if (string.IsNullOrEmpty(sql))
{
return null;
}
string missingParameters = string.Empty;
foreach (var item in parametersDictionary)
{
if (!sql.Contains(item.Key))
{
missingParameters += $"Missing parameter: {item.Key}";
}
}
if (!string.IsNullOrEmpty(missingParameters))
{
throw new ArgumentException($"Parameterized query failed. {missingParameters}");
}
var parameters = new DynamicParameters(parametersDictionary);
return connection.Query(sql, parameters);
}
Then adding a method to create a Like search term that will work with Dapper.
public static string Like(string searchTerm)
{
if (string.IsNullOrEmpty(searchTerm))
{
return null;
}
Func<string, string> encodeForLike = searchTerm => searchTerm.Replace("[", "[[]").Replace("%", "[%]");
return $"%{encodeForLike(searchTerm)}%";
}
Example usage:
var sql = $"select * from products where ProductName like #ProdName";
var herringsInNorthwindDb = connection.ParameterizedQuery(sql, new Dictionary<string, object> { { "#ProdName", Like("sild") } });
foreach (var herring in herringsInNorthwindDb)
{
Console.WriteLine($"{herring.ProductName}");
}
And we get our sample data from Northwind DB:
I like this approach, since we get helper extension methods to do repetitive work.
My solution simple to this problem :
parameter.Add("#nomeCliente", dfNomeCliPesquisa.Text.ToUpper());
query = "SELECT * FROM cadastrocliente WHERE upper(nome) LIKE " + "'%" + dfNomeCliPesquisa.Text.ToUpper() + "%'";