In sql we can do something like this:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
Is there any way to do multiple/bulk/batch inserts or updates in Slick?
Can we do something similar, at least using SQL plain queries ?
For inserts, as Andrew answered, you use insertALL.
def insertAll(items:Seq[MyCaseClass])(implicit session:Session) = {
(items.size) match {
case s if s > 0 =>
try {
// basequery is the tablequery object
baseQuery.insertAll(tempItems :_*)
} catch {
case e:Exception => e.printStackTrace()
}
Some(tempItems(0))
case _ => None
}
}
For updates, you're SOL. Check out Scala slick 2.0 updateAll equivalent to insertALL? for what I ended up doing. To paraphrase, here's the code:
private def batchUpdateQuery = "update table set value = ? where id = ?"
/**
* Dropping to jdbc b/c slick doesnt support this batched update
*/
def batchUpate(batch:List[MyCaseClass])(implicit session:Session) = {
val pstmt = session.conn.prepareStatement(batchUpdateQuery)
batch map { myCaseClass =>
pstmt.setString(1, myCaseClass.value)
pstmt.setString(2, myCaseClass.id)
pstmt.addBatch()
}
session.withTransaction {
pstmt.executeBatch()
}
}
In Slick, you are able to use the insertAll method for a Table. An example of insertAll is given in the Getting Started page on Slick's website.
http://slick.typesafe.com/doc/0.11.1/gettingstarted.html
Related
I am new to elasticsearch so I will need some help. Unfortunately, I didnt found the answer in other topics here on SO.
I have some .net core application which I inherited and now there is a need to implement some changes.
I already have a method of getting data from elasticsearch, but after getting them, I am not sure how to change it and use it in application.
To be precise, I need to parse first and last name and to remove special characters, specific serbian latin letters like "šđžčć" etc... I already have a method for this parsing written but not sure how to call it...
So, my question is can I and how can I do this?
What I have now is the following:
var result = await _elasticClient.SearchAsync<CachedUserEntity>(
s =>
s.Index(_aliasName)
.Query(q => andQuery));
CachedUserEntity, among others, contains property about FirstName and LastName.
Inside results.Documents, I am getting the data about FirstName and LastName from elasticsearch, but I am not sure how to access it in order to update it via aformentioned NameParser ...
Sorry if the question is too easy, not to say stupid :)
I wont use updateByQuery here, for some reasons. I would scroll on documents (i use matchAll on my exemple, you obviously need to replace it with your query), or, if you dont know how to identify documents to update, only update usefull documents in UpdateManyWithIndex/UpdateManyPartial function.
For performance, we have to update severals documents at once, so we use bulk/updateMany function.
You can use both solution, the classic update, or the second (partial update) with an object containing the targeteds fields.
On server sides, both solutions will have the same cost / performance.
var searchResponse = Client.Search<CachedUserEntity>(s => s
.Query(q => q
MatchAll()
)
.Scroll("10s")
);
while (searchResponse.Documents.Any())
{
List<CachedUserEntity> NewSearchResponse = RemoveChar(searchResponse);
UpdateManyWithIndex<CachedUserEntity>(NewSearchResponse, _aliasName);
searchResponse = Client.Scroll<Project>("2h", searchResponse.ScrollId);
}
public void UpdateManyWithIndex<C>(List<C> obj, string index) where C : class {
var bulkResponse = Client.Bulk(b => b
.Index(index).Refresh(Elasticsearch.Net.Refresh.WaitFor) // explicitly provide index name
.UpdateMany<C>(obj, (bu, d) => bu.Doc(d)));
}
Or, using partial update object
Note: in this case Indix is already set on my client (add .index if needed)
var searchResponse = Client.Search<CachedUserEntity>(s => s
.Query(q => q
MatchAll()
)
.Scroll("2h")
);
while (searchResponse.Documents.Any())
{
List<object> listPocoPartialObj = GetPocoPartialObjList(searchResponse);
UpdateManyPartial(listPocoPartialObj);
searchResponse = Client.Scroll<Project>("2h", searchResponse.ScrollId);
}
private List<object> GetPocoPartialObjList(List<CachedUserEntity> cachedList) {
List<object> listPoco = new List<object>();
//note if you dont have cachedList.Id, take a look at result.source, comments if needed
foreach (var eltCached in cachedList) {
listPoco.Add( new object() { Id = cachedList.Id, FirstName = YOURFIELDWITHOUTSPECIALCHAR, LastName = YOURSECONDFIELDWITHOUTSPECIALCHAR});
}
return listPoco;
}
public bool UpdateManyPartial(List<object> partialObj)
{
var bulkResponse = Client.Bulk(b => b
.Refresh(Elasticsearch.Net.Refresh.WaitFor)
.UpdateMany(partialObj, (bu, d) => bu.Doc(d))
);
if (!bulkResponse.IsValid)
{
GetErrorMsgs(bulkResponse);
}
return (bulkResponse?.IsValid == true);
}
I'm trying to do a select count query in Sql Server using Dapper. The expected response should be 0 when a profile does not exist. When I do the query in SSMS it returns correctly, but in the API using Dapper it returns 1. Any idea why this is happening?
public IActionResult GetProfileCount(string profileId)
{
int profileCount = 0;
using (IDbConnection db = new SqlConnection(connectionString))
{
try
{
profileCount = db.Query($"select count(*) from Profile where Id='{profileId}'").Count();
}
catch(Exception ex)
{
Console.WriteLine($"Error retrieving count for ProfileId: {profileId}", ex.Message);
}
}
return Ok(profileCount);
}
I see you added your own answer but could I recommend not doing it that way. When you do
profileCount = db.Query($"select * from Profile where Id='{profileId}'").Count();
What you are actually doing is selecting every field from the database, pulling it into your C# application, and then counting how many results you got back. Then you are binning all that data you got back, very inefficient!
Change it to this :
profileCount = db.QueryFirst<int>($"select count(*) from Profile where Id = #profileId", new { profileId })");
Instead you are selecting an "int" from the result set, which just so happens to be your count(*). Perfect!
More on querying in Dapper here : https://dotnetcoretutorials.com/2019/08/05/dapper-in-net-core-part-2-dapper-query-basics/
Also notice that (similar to the other answer), I am using parameterized queries. I also heavily recommend this as it protects you from SQL Injection. Your initial example is very vulnerable!
You can read a little more about SQL Injection in C#/MSSQL here https://dotnetcoretutorials.com/2017/10/11/owasp-top-10-asp-net-core-sql-injection/ But just know that Dapper protects you from it as long as you use the inbuilt helpers to add parameters to your queries.
Another option is use the method ExecuteScalar for "select count" queries:
profileCount = db.ExecuteScalar<int>("select count(*) from Profile where Id=#profileId", new { profileId });
Ref.: https://www.learndapper.com/selecting-scalar-values
Try and change your query to the following:
db.Query($"select count(*) from Profile where Id = #ProfileId", new { ProfileId = profileId }).Count()
I figured it out. The .Count() is counting the rows of the result, which is going to be 1 because the result is one row displaying the number 0. I switched my code to this and it works now.
public IActionResult GetProfileCount(string profileId)
{
int profileCount = 0;
using (IDbConnection db = new SqlConnection(connectionString))
{
try
{
profileCount = db.Query($"select * from Profile where Id='{profileId}'").Count();
}
catch(Exception ex)
{
Console.WriteLine($"Error retrieving count for ProfileId: {profileId}", ex.Message);
}
}
return Ok(profileCount);
}
I have an "UPDATE" query and I want to check if this query has affected any rows and if it has I would like to send an email to myself. I'm not really experienced with perl or sql. (I'm using postgresql).
EDIT: I know that UPDATE returns "count" which tells how many rows if any have been updated, but I still don't know how to get to it.
The code looks like this:
my $updateQuery - //UPDATE query
if(//updateQuery has effected rows){
//send mail
}else
//do nothing
The problem is I don't know what to put into the if, what flag should I set? Is there any easy check that answers "has effected rows"?
Shortcut version using do()
my $update_query = '...';
my $ret = $dbh->do($update_query);
if ($ret) {
if ($ret eq '0E0') {
# no rows updated
} else {
# rows updated
}
} else {
# error
}
Full prepare()/execute() version.
my $update_query = '...';
my $sth = $dbh->prepare($update_query);
my $ret = $sth->execute;
if ($ret) {
if ($ret eq '0E0') {
# no rows updated
} else {
# rows updated
}
} else {
# error
}
How do I create a timeout for this operation: ?
def db = Sql.newInstance("jdbc:mysql://${mysql_host}:3306/${dbName}",
user, pass, 'com.mysql.jdbc.Driver')
db.eachRow(query) { row ->
// do something with the row
}
I believe the correct way would be something like this:
sql = Sql.newInstance("jdbc:oracle:thin:#localhost:1521:XE", "user",
"pwd", "oracle.jdbc.driver.OracleDriver")
sql.withStatement {
stmt -> stmt.queryTimeout = 10
}
sql.eachRow("select * from someTable", {
println it
} )
of course, this is where I'd used Oracle, but I hope this can give you an idea.
I believe there might not be a general answer, but rather a database/driver-specific answer via parameters to the connection URL.
E.g. for mysql, I think that adding connectTimeout=something&socketTimeout=something might do the trick.
I am working on a backend edit page for my custom entity. I have almost everything working, including saving a bunch of different text fields. I have a problem, though, when trying to set the value of a boolean field.
I have tried:
$landingPage->setEnabled(1);
$landingPage->setEnabled(TRUE);
$landingPage->setEnabled(0);
$landingPage->setEnabled(FALSE);
None seem to persist a change to my database.
How are you supposed to set a boolean field using magento ORM?
edit
Looking at my database, mysql is storing the field as a tinyint(1), so magento may be seeing this as an int not a bool. Still can't get it to set though.
This topic has bring curiosity to me. Although it has been answered, I'd like to share what I've found though I didn't do intense tracing.
It doesn't matter whether the cache is enabled / disabled, the table schema will be cached.
It will be cached during save process.
Mage_Core_Model_Abstract -> save()
Mage_Core_Model_Resource_Db_Abstract -> save(Mage_Core_Model_Abstract $object)
Mage_Core_Model_Resource_Db_Abstract
public function save(Mage_Core_Model_Abstract $object)
{
...
//any conditional will eventually call for:
$this->_prepareDataForSave($object);
...
}
protected function _prepareDataForSave(Mage_Core_Model_Abstract $object)
{
return $this->_prepareDataForTable($object, $this->getMainTable());
}
Mage_Core_Model_Resource_Abstract
protected function _prepareDataForTable(Varien_Object $object, $table)
{
$data = array();
$fields = $this->_getWriteAdapter()->describeTable($table);
foreach (array_keys($fields) as $field) {
if ($object->hasData($field)) {
$fieldValue = $object->getData($field);
if ($fieldValue instanceof Zend_Db_Expr) {
$data[$field] = $fieldValue;
} else {
if (null !== $fieldValue) {
$fieldValue = $this->_prepareTableValueForSave($fieldValue, $fields[$field]['DATA_TYPE']);
$data[$field] = $this->_getWriteAdapter()->prepareColumnValue($fields[$field], $fieldValue);
} else if (!empty($fields[$field]['NULLABLE'])) {
$data[$field] = null;
}
}
}
}
return $data;
}
See the line: $fields = $this->_getWriteAdapter()->describeTable($table);
Varien_Db_Adapter_Pdo_Mysql
public function describeTable($tableName, $schemaName = null)
{
$cacheKey = $this->_getTableName($tableName, $schemaName);
$ddl = $this->loadDdlCache($cacheKey, self::DDL_DESCRIBE);
if ($ddl === false) {
$ddl = parent::describeTable($tableName, $schemaName);
/**
* Remove bug in some MySQL versions, when int-column without default value is described as:
* having default empty string value
*/
$affected = array('tinyint', 'smallint', 'mediumint', 'int', 'bigint');
foreach ($ddl as $key => $columnData) {
if (($columnData['DEFAULT'] === '') && (array_search($columnData['DATA_TYPE'], $affected) !== FALSE)) {
$ddl[$key]['DEFAULT'] = null;
}
}
$this->saveDdlCache($cacheKey, self::DDL_DESCRIBE, $ddl);
}
return $ddl;
}
As we can see:
$ddl = $this->loadDdlCache($cacheKey, self::DDL_DESCRIBE);
will try to load the schema from cache.
If the value is not exists: if ($ddl === false)
it will create one: $this->saveDdlCache($cacheKey, self::DDL_DESCRIBE, $ddl);
So the problem that occurred in this question will be happened if we ever save the model that is going to be altered (add column, etc).
Because it has ever been $model->save(), the schema will be cached.
Later after he add new column and "do saving", it will load the schema from cache (which is not containing the new column) and resulting as: the data for new column is failed to be saved in database
Delete var/cache/* - your DB schema is cached by Magento even though the new column is already added to the MySQL table.