Aerospike aql: How to fetch records from aerospike using predicate based on the Map field - aerospike

I've below class definition for data transfer object. I use spring-data-aerospike to persistence.
import org.springframework.data.annotation.Id;
public class User implements Serializable {
#Id
String uid;
Map<String, Object> ext = new HashMap<String, Object>();
// getters & setters
}
Sample data in the database is like -
select ext from department.User;
+-------+-------------------------+----------------------------------+
| PK | ext || #_class |
+-------+-------------------------+----------------------------------+
| "123" | MAP('{"idfa": "xyz"}') | "com.tut.dto.User" |
| "234" | MAP('{}') | "com.tut.dto.User" |
+-------+-------------------------+----------------------------------+
I need to query the database now that it should only return the records which have "idfa" key string in the ext field column.
I tried the following. But it didn't work
1.
select * from test.UserRecord where ext.idfa is not null;
Unsupported command format with token - '.'
Make sure string values are enclosed in quotes.
Type " aql --help " from console or simply "help" from within the aql-prompt.
```
2.
select * from test.UserRecord where ext contains 'idfa';
Unsupported command format with token - ''idfa''
Make sure string values are enclosed in quotes.
Type " aql --help " from console or simply "help" from within the aql-prompt.
How can I make it work?

Try executing:
select * from department.user in mapkeys where ext = "idfa"
Based on the following structure:
SELECT <bins> FROM <ns>[.<set>] IN <indextype> WHERE <bin> = <value>
Assuming you've created an index with collection type of "MAPKEYS", you can create it using #Indexed annotation on the ext field in your User class.
If your using Spring Data Aerospike latest version it should look something like that:
#Indexed(name = "indexName", type = IndexType.STRING, collectionType = IndexCollectionType.MAPKEYS)
Map<String, Object> ext = new HashMap<String, Object>();
If you're interested in interacting with Aerospike List/Map bins through code (instead of AQL) you should read about CDT (Collection Data Type) operations.
CDT documentation:
https://docs.aerospike.com/docs/guide/cdt.html
Map Operation class (including methods documentation):
https://github.com/aerospike/aerospike-client-java/blob/master/client/src/com/aerospike/client/cdt/MapOperation.java
And some examples in the Aerospike Java Client:
https://github.com/aerospike/aerospike-client-java/blob/master/test/src/com/aerospike/test/sync/basic/TestOperateMap.java

Related

What's the proper syntax for SETTING a table value

In the Dao Interface I need to use
#Query("UPDATE table SET user_name = value")
fun addValue(value: String)
Here value is not recognized as the input from the function addValue(value:String)
IDE reports value is an Unresolved symbol
How do make value from the function be recognised as input in the SQL statement
I'm assuming it might have something to do with Entities what Entities to I need to include in my Database class
Here
#Database(Entities = MyEntity::class], version = 1)
abstract class MyEntityDatabase: Room database()
The room provides colon operator to resolve the argument in the query. So you query #Query("UPDATE table SET user_name = value") will be changed to Query("UPDATE table SET user_name = :value") then room can resolve the argument.
So it should be like
#Query("UPDATE table SET user_name = :argName")
fun addValue(argName: String)

Does Apache Spark SQL support MERGE clause?

Does Apache Spark SQL support MERGE clause that's similar to Oracle's MERGE SQL clause?
MERGE into <table> using (
select * from <table1>
when matched then update...
DELETE WHERE...
when not matched then insert...
)
Spark does support MERGE operation using Delta Lake as storage format. The first thing to do is to save the table using the delta format to provide support for transactional capabilities and support for DELETE/UPDATE/MERGE operations with spark
Python/scala:
df.write.format("delta").save("/data/events")
SQL: CREATE TABLE events (eventId long, ...) USING delta
Once the table exists, you can run your usual SQL Merge command:
MERGE INTO events
USING updates
ON events.eventId = updates.eventId
WHEN MATCHED THEN
UPDATE SET events.data = updates.data
WHEN NOT MATCHED
THEN INSERT (date, eventId, data) VALUES (date, eventId, data)
The command is also available in Python/Scala:
DeltaTable.forPath(spark, "/data/events/")
.as("events")
.merge(
updatesDF.as("updates"),
"events.eventId = updates.eventId")
.whenMatched
.updateExpr(
Map("data" -> "updates.data"))
.whenNotMatched
.insertExpr(
Map(
"date" -> "updates.date",
"eventId" -> "updates.eventId",
"data" -> "updates.data"))
.execute()
To support Delta Lake format, you also need the delta package as dependency in your spark job:
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_x.xx</artifactId>
<version>xxxx</version>
</dependency>
See https://docs.delta.io/latest/delta-update.html#upsert-into-a-table-using-merge for more details
As of Spark 3.0, Spark offers a very clean way of doing the merge operation using the spark delta table.
https://docs.delta.io/latest/delta-update.html#upsert-into-a-table-using-merge
It does not. As of now (it might change in the future) Spark doesn't support UPDATES, DELETES or any other variant of record modification.
It can only overwrite existing storage (with different implementation depending on the source) or append with plain INSERT.
you can write your custom code: Below code you can edit to go with merge instead of Insert. Make sure this is computation heavy operations. but get y
df.rdd.coalesce(2).foreachPartition(partition => {
val connectionProperties = brConnect.value
val jdbcUrl = connectionProperties.getProperty("jdbcurl")
val user = connectionProperties.getProperty("user")
val password = connectionProperties.getProperty("password")
val driver = connectionProperties.getProperty("Driver")
Class.forName(driver)
val dbc: Connection = DriverManager.getConnection(jdbcUrl, user, password)
val db_batchsize = 1000
var pstmt: PreparedStatement = null
partition.grouped(db_batchsize).foreach(batch => {
batch.foreach{ row =>
{
val id = row.id
val fname = row.fname
val lname = row.lname
val userid = row.userid
println(id, fname)
val sqlString = "INSERT employee USING " +
" values (?, ?, ?, ?) "
var pstmt: PreparedStatement = dbc.prepareStatement(sqlString)
pstmt.setLong(1, row.id)
pstmt.setString(2, row.fname)
pstmt.setString(3, row.lname)
pstmt.setString(4, row.userid)
pstmt.addBatch()
pstmt.executeBatch()
}
}
//pstmt.executeBatch()
dbc.commit()
pstmt.close()
})
dbc.close()
} )
If you are working over Spark, maybe this answers could help you to lead with the merge issue using DataFrames.
Anyway, reading some documentation of Hortonworks, it says that Merge sentence is supported in Apache Hive 0.14 and later.
There is an Apache project - Apache Iceberg - which creates a table format type with editing capabilities, including MERGE:
https://iceberg.apache.org/docs/latest/spark-writes/

QueryDSL like operation SimplePath

Similarly to this question I would like to perform an SQL "like" operation using my own user defined type called "AccountNumber".
The QueryDSL Entity class the field which defines the column looks like this:
public final SimplePath<com.myorg.types.AccountNumber> accountNumber;
I have tried the following code to achieve a "like" operation in SQL but get an error when the types are compared before the query is run:
final Path path=QBusinessEvent.businessEvent.accountNumber;
final Expression<AccountNumber> constant = Expressions.constant(AccountNumber.valueOfWithWildcard(pRegion.toString()));
final BooleanExpression booleanOperation = Expressions.booleanOperation(Ops.STARTS_WITH, path, constant);
expressionBuilder.and(booleanOperation);
The error is:
org.springframework.dao.InvalidDataAccessApiUsageException: Parameter value [7!%%] did not match expected type [com.myorg.types.AccountNumber (n/a)]
Has anyone ever been able to achieve this using QueryDSL/JPA combination?
Did you try using a String constant instead?
Path<?> path = QBusinessEvent.businessEvent.accountNumber;
Expression<String> constant = Expressions.constant(pRegion.toString());
Predicate predicate = Expressions.predicate(Ops.STARTS_WITH, path, constant);
In the end, I was given a tip by my colleague to do the following:
if (pRegion != null) {
expressionBuilder.and(Expressions.booleanTemplate("{0} like concat({1}, '%')", qBusinessEvent.accountNumber, pRegion));
}
This seems to do the trick!
It seems like there is bug/ambiguity. In my case, I need to search by couple fields with different types (String, Number), e.g. SQL looks like:
SELECT * FROM table AS t WHERE t.name = "%some%" OR t.id = "%some%";
My code looks like:
BooleanBuilder where = _getDefaultPredicateBuilder();
BooleanBuilder whereLike = new BooleanBuilder();
for(String likeField: _likeFields){
whereLike = whereLike.or(_pathBuilder.getString(likeField).contains(likeValue));
}
where.and(whereLike);
If first _likeFields is type of String - request works fine, otherwise it throws Exception.

How to query Oracle v$ tables using Groovy

I'm trying to query the Oracle v$session table using Groovy (imported groovy.sql.SQL) like this:
sql.eachRow("""
Select
'Y' as runInd
from v$session
where upper(module) = ?
having count(*) > 1
""", [programName]) {row -> etc...}
But Groovy keeps telling me: "Groovy:Apparent variable 'session' was found in a static scope but doesn't refer to a local variable, static field or class."
It apparently doesn't like the table called v$session. I've tried many things, I'm not sure why I can't find out how to do this. Any help is appreciated. Thanks!
Tom
Instead of """ which marks it as a multi-line templated groovy string, try ''' which shouldn't try to template things following a $:
sql.eachRow( '''Select
| 'Y' as runInd
| from v$session
| where upper(module) = ?
| having count(*) > 1'''.stripMargin(), [programName]) { row ->
etc...
}

Modeshape querying mixinTypes

I'm using Modeshape and modeshape-connector-jdbc-metadata. I want to get all nodes representing tables in the storage. That nodes have [mj:catalog] mixin type.
I'm querying storage using next code:
public List getDatabases() throws RepositoryException {
// Obtain the query manager for the session ...
QueryManager queryManager = dbSession.getWorkspace().getQueryManager();
// Create a query object ...
Query query = queryManager.createQuery("SELECT * FROM [mj:table]"
, Query.JCR_SQL2);
// Execute the query and get the results ...
QueryResult result = query.execute();
// Iterate over the nodes in the results ...
NodeIterator nodeIter = result.getNodes();
List stringResult = new ArrayList();
while (nodeIter.hasNext()) {
stringResult.add(nodeIter.nextNode().getName());
}
return stringResult;
}
But it always returns empty list.
I also tried to query using next queries:
SELECT unst.*, tbl.* FROM [nt:unstructured] AS unst
JOIN [mj:table] AS tbl ON ISSAMENODE(unst,tbl)
SELECT * FROM [nt:unstructured] WHERE [jcr:mixinTypes] = [mj:table]
But result remains the same.
What I'm doing wrong?
Thank you for any help.
There is a known issue that the database metadata nodes are not indexed automatically. A simple workaround is to cast the JCR Session's getWorkspace() instance to org.modeshape.jcr.api.Workspace (the public API for ModeShape's workspace) and call the reindex(String path) method and passing in the path to the database catalog node (or an ancestor if desired).