Related
I am using get_or_create to insert objects to database but the problem is that doing 1000 at once takes too long time.
I tried bulk_create but it doesn't provide functionality I need (creates duplicates, ignores unique value, doesn't trigger post_save signals I need).
Is it even possible to do get_or_create in bulk via customized sql query?
Here is my example code:
related_data = json.loads(urllib2.urlopen(final_url).read())
for item in related_data:
kw = item['keyword']
e, c = KW.objects.get_or_create(KWuser=kw, author=author)
e.project.add(id)
#Add m2m to parent project
related_data cotains 1000 rows looking like this:
[{"cmp":0,"ams":3350000,"cpc":0.71,"keyword":"apple."},
{"cmp":0.01,"ams":3350000,"cpc":1.54,"keyword":"apple -10810"}......]
KW model also sends signal I use to create another parent model:
#receiver(post_save, sender=KW)
def grepw(sender, **kwargs):
if kwargs.get('created', False):
id = kwargs['instance'].id
kww = kwargs['instance'].KWuser
# KeyO
a, b = KeyO.objects.get_or_create(defaults={'keyword': kww}, keyword__iexact=kww)
KW.objects.filter(id=id).update(KWF=a.id)
This works but as you can imagine doing thousands of rows at once takes long time and even crashes my tiny server, what bulk options do I have?
As of Django 2.2, bulk_create has an ignore_conflicts flag. Per the docs:
On databases that support it (all but Oracle), setting the ignore_conflicts parameter to True tells the database to ignore failure to insert any rows that fail constraints such as duplicate unique values
This post may be of use to you:
stackoverflow.com/questions/3395236/aggregating-saves-in-django
Note that the answer recommends using the commit_on_success decorator which is deprecated. It is replaced by the transaction.atomic decorator. Documentation is here:
transactions
from django.db import transaction
#transaction.atomic
def lot_of_saves(queryset):
for item in queryset:
modify_item(item)
item.save()
If I understand correctly, "get_or_create" means SELECT or INSERT on the Postgres side.
You have a table with a UNIQUE constraint or index and a large number of rows to either INSERT (if not yet there) and get the newly create ID or otherwise SELECT the ID of the existing row. Not as simple as it may seem on the outside. With concurrent write load, the matter is even more complicated.
And there are various parameters that need to be defined (how to handle conflicts exactly):
How to use RETURNING with ON CONFLICT in PostgreSQL?
I have a column in my database that has multiple values, here is an example of what these values are:
The SQL column is called: extra_fields and is type text
[{"id":"2","value":"Ser mayor de edad"},
{"id":"3","value":"Prueba de extension"},
{"id":"4","value":"99.999.99"},
{"id":"5","value":"10"}}
I have this in a PHP function where I get the values I want to modify:
$db = $this->getDbo();
$query = 'SELECT * FROM #__k2_items where id='.$item_id;
$db->setQuery($query);
$resultado = $db->loadObject();
$campos_extra = json_decode($resultado->extra_fields);
$num_max_cup_descargar = $campos_extra[2]->value;
$num_max_cup_canjear = $campos_extra[3]->value;
$num_max_cup_x_dia = $campos_extra[4]->value;
$num_max_cup_uti_canje_x_user_dia = $campos_extra[5]->value;
$num_max_cup_desc_user = $campos_extra[6]->value;
I am trying to update one of these values, how can I do this?
EDIT: Im using MySQL database.
You will not be able to Update the substring of some value in a text column with only SQL. MySQL doesn't care if you have a stringified JSON object in there or part of a Shakespeare play. You'll have to update this value by doing it in PHP:
$campos_extra = json_decode($resultado->extra_fields);
$num_max_cup_descargar = $campos_extra[2]->value;
$num_max_cup_canjear = $campos_extra[3]->value;
$num_max_cup_x_dia = $campos_extra[4]->value;
$num_max_cup_uti_canje_x_user_dia = $campos_extra[5]->value;
$num_max_cup_desc_user = $campos_extra[6]->value;
$num_max_cup_desc_user = "changedUser";
$campos_extra[6]->value = $num_max_cup_desc_user;
$campos_extra_updated = json_encode($campos_extra);
And put $campos_extra_updated into that field of the database.
In order to dynamically support such a queries you might use two different approaches:
Programmatically manipulate the data
Well, this is rather simple - to summarize it nicely it means that you have to write the logic to manipulate the data end to end. The great benefit of that is a simple fact that your solution will not be dependent on the underlying DB engine.
Use the database specific extensions
Most of the current RDBMS engines have some support for dynamic columns or array data stored in the tables. In case of MySQL there is an open source clone called MariaDB that have support for such a mechanism - see the documentation of Dynamic Columns in MariaDB.
With PostgreSQL you have an option to store JSON and manipulate that directly (limited), array column, composite types if your properties are well defined or hstore to create a schema-less design.
All of the above implementations will have their pros and cons, so you must pick wisely. But all of them comes with added benefits of indexing (and, therefore, queries) against to columns, some general SQL support, etc. Choose wisely :-)
Does Spring have any features which allow for a batch select? I basically have n number of selects to execute depending on the number of records in a list. At the moment because the size of the list is always changing i have to dynamically build the SQL which will be executed. The end product looks something like this
select * from record_details t WHERE id IN ((?),(?),(?))
However the code to generate this SQL on the fly is messy and I'm wondering if there is a nicer approach for this type of problem?
The NamedParameterJdbcTemplate (and according support class) does have that support.
public void someRepoMethod(List ids) {
String query = "select * from record_details where id in (:ids)";
getNamedParameterJdbcTemplate().query(query, Collections.singletonMap("ids", ids), new YourRowMapper());
}
If you don't want to generate SQL yourself you have to use some existing framework. From what i know myBatis, is more lightweight than hibernate so it can suit you more, but there may be other more suitable.
I saw a query run in a log file on an application. and it contained a query like:
SELECT ID FROM CUST_ATTR49 WHERE 1=0
what is the use of such a query that is bound to return nothing?
A query like this can be used to ping the database. The clause:
WHERE 1=0
Ensures that non data is sent back, so no CPU charge, no Network traffic or other resource consumption.
A query like that can test for:
server availability
CUST_ATTR49 table existence
ID column existence
Keeping a connection alive
Cause a trigger to fire without changing any rows (with the where clause, but not in a select query)
manage many OR conditions in dynamic queries (e.g WHERE 1=0 OR <condition>)
This may be also used to extract the table schema from a table without extracting any data inside that table. As Andrea Colleoni said those will be the other benefits of using this.
A usecase I can think of: you have a filter form where you don't want to have any search results. If you specify some filter, they get added to the where clause.
Or it's usually used if you have to create a sql query by hand. E.g. you don't want to check whether the where clause is empty or not..and you can just add stuff like this:
where := "WHERE 0=1"
if X then where := where + " OR ... "
if Y then where := where + " OR ... "
(if you connect the clauses with OR you need 0=1, if you have AND you have 1=1)
As an answer - but also as further clarification to what #AndreaColleoni already mentioned:
manage many OR conditions in dynamic queries (e.g WHERE 1=0 OR <condition>)
Purpose as an on/off switch
I am using this as a switch (on/off) statement for portions of my Query.
If I were to use
WHERE 1=1
AND (0=? OR first_name = ?)
AND (0=? OR last_name = ?)
Then I can use the first bind variable (?) to turn on or off the first_name search criterium. , and the third bind variable (?) to turn on or off the last_name criterium.
I have also added a literal 1=1 just for esthetics so the text of the query aligns nicely.
For just those two criteria, it does not appear that helpful, as one might thing it is just easier to do the same by dynamically building your WHERE condition by either putting only first_name or last_name, or both, or none. So your code will have to dynamically build 4 versions of the same query. Imagine what would happen if you have 10 different criteria to consider, then how many combinations of the same query will you have to manage then?
Compile Time Optimization
I also might add that adding in the 0=? as a bind variable switch will not work very well if all your criteria are indexed. The run time optimizer that will select appropriate indexes and execution plans, might just not see the cost benefit of using the index in those slightly more complex predicates. Hence I usally advice, to inject the 0 / 1 explicitly into your query (string concatenating it in in your sql, or doing some search/replace). Doing so will give the compiler the chance to optimize out redundant statements, and give the Runtime Executer a much simpler query to look at.
(0=1 OR cond = ?) --> (cond = ?)
(0=0 OR cond = ?) --> Always True (ignore predicate)
In the second statement above the compiler knows that it never has to even consider the second part of the condition (cond = ?), and it will simply remove the entire predicate. If it were a bind variable, the compiler could never have accomplished this.
Because you are simply, and forcedly, injecting a 0/1, there is zero chance of SQL injections.
In my SQL's, as one approach, I typically place my sql injection points as ${literal_name}, and I then simply search/replace using a regex any ${...} occurrence with the appropriate literal, before I even let the compiler have a stab at it. This basically leads to a query stored as follows:
WHERE 1=1
AND (0=${cond1_enabled} OR cond1 = ?)
AND (0=${cond2_enabled} OR cond2 = ?)
Looks good, easily understood, the compiler handles it well, and the Runtime Cost Based Optimizer understands it better and will have a higher likelihood of selecting the right index.
I take special care in what I inject. Prime way for passing variables is and remains bind variables for all the obvious reasons.
This is very good in metadata fetching and makes thing generic.
Many DBs have optimizer so they will not actually execute it but its still a valid SQL statement and should execute on all DBs.
This will not fetch any result, but you know column names are valid, data types etc. If it does not execute you know something is wrong with DB(not up etc.)
So many generic programs execute this dummy statement for testing and fetching metadata.
Some systems use scripts and can dynamically set selected records to be hidden from a full list; so a false condition needs to be passed to the SQL. For example, three records out of 500 may be marked as Privacy for medical reasons and should not be visible to everyone. A dynamic query will control the 500 records are visible to those in HR, while 497 are visible to managers. A value would be passed to the SQL clause that is conditionally set, i.e. ' WHERE 1=1 ' or ' WHERE 1=0 ', depending who is logged into the system.
quoted from Greg
If the list of conditions is not known at compile time and is instead
built at run time, you don't have to worry about whether you have one
or more than one condition. You can generate them all like:
and
and concatenate them all together. With the 1=1 at the start, the
initial and has something to associate with.
I've never seen this used for any kind of injection protection, as you
say it doesn't seem like it would help much. I have seen it used as an
implementation convenience. The SQL query engine will end up ignoring
the 1=1 so it should have no performance impact.
Why would someone use WHERE 1=1 AND <conditions> in a SQL clause?
If the user intends to only append records, then the fastest method is open the recordset without returning any existing records.
It can be useful when only table metadata is desired in an application. For example, if you are writing a JDBC application and want to get the column display size of columns in the table.
Pasting a code snippet here
String query = "SELECT * from <Table_name> where 1=0";
PreparedStatement stmt = connection.prepareStatement(query);
ResultSet rs = stmt.executeQuery();
ResultSetMetaData rsMD = rs.getMetaData();
int columnCount = rsMD.getColumnCount();
for(int i=0;i<columnCount;i++) {
System.out.println("Column display size is: " + rsMD.getColumnDisplaySize(i+1));
}
Here having a query like "select * from table" can cause performance issues if you are dealing with huge data because it will try to fetch all the records from the table. Instead if you provide a query like "select * from table where 1=0" then it will fetch only table metadata and not the records so it will be efficient.
Per user milso in another thread, another purpose for "WHERE 1=0":
CREATE TABLE New_table_name as select * FROM Old_table_name WHERE 1 =
2;
this will create a new table with same schema as old table. (Very
handy if you want to load some data for compares)
An example of using a where condition of 1=0 is found in the Northwind 2007 database. On the main page the New Customer Order and New Purchase Order command buttons use embedded macros with the Where Condition set to 1=0. This opens the form with a filter that forces the sub-form to display only records related to the parent form. This can be verified by opening either of those forms from the tree without using the macro. When opened this way all records are displayed by the sub-form.
In ActiveRecord ORM, part of RubyOnRails:
Post.where(category_id: []).to_sql
# => SELECT * FROM posts WHERE 1=0
This is presumably because the following is invalid (at least in Postgres):
select id FROM bookings WHERE office_id IN ()
It seems like, that someone is trying to hack your database. It looks like someone tried mysql injection. You can read more about it here: Mysql Injection
I have this idea that using SQL VIEWS to abstract simple database computations (such as a count on a relation) is sufficient, and you don't need procedures (== procedural code)
A simple sql view + a where clause >> a stored procedure with parameters sometimes
While making this point I imagined a way of retrieving table/view data without writing SQL and without writing the where clause..
But, to my surprise, there does not seem a way to accomplish this in ADO.NET 2.0 or later.
Let me tell you what I tried:
SqlDataAdapter + SqlCommandBuilder still requires you to write "SELECT ... FROM" and the WHERE CLAUSE in strings (plus, if you put the 'where', you dont have much use of Update/Insert/DeleteCommand)
typed DataSets only allow you to retrieve _entire DataTable_s and then applying filters to them. Filters are strings, without escaping aid... (must double the single quote!)
SQL to Entities looked promising but they seem to: be limited to MSSQL, generate bloated SQL queries, generate a whole new stack of DAOs (besides the existing Domain Model classes), reqiuire .net 3.5+ for all this etc. (that is, all these are disadvantages for me)
Other ORMs have similar problems as SQL to Entities.
What I'm looking for is a strong-typed method of accessing database tables/views that:
doesn't come with another set of DAOs (K.I.S.S)
allows me to query a table without writing "SELECTs" in strings (strong-typed)
allows me to filter(WHERE) a table with properly-escaped parameters (and without retrieving the whole data beforehand)
can later issue updates/inserts/deletes
I am fairly new to .Net but not stupid: does this exist?
Thanks.
FluentADO
Subsonic has a fairly lightweight query tool that you can use to directly query against the database with a Query object that abstracts the SQL. If you want to, you can also use its code generation facility to map your database tables to POCOs, or to only create a strongly typed schema (for column/table names and so on).
I don't really believe that what you want to do is achievable without using some sort of ORM, or a specialized DSL with a compiler that somehow knows about your database schema, type/column information, etc.
Take into account that C# is a general purpose language, and its compiler is totally unaware of your database types, that's why you cannot bind them without using some abstraction layer, which usually involves ad hoc SQL queries(strings), NHibernate or similar mapping files (more strings) and/or DAOs.
If you don't want to write the WHERE clause, one way is to use a Filter object and add the conditions you want. For example:
var sc = new Filter();
sc.Add("Contacttitle", "Sales Agent");
sc.Add("city", "london", Logical.Or);
var customers = D2Bk.Fetch(sc, new Customers());
But you don't want to use DAOs (Customers above is such), so you would have to write the SQL statement and specify the where clause:
DataSet ds = D2Bk.Fetch("SELECT * FROM Customers WHERE Contacttitle=#PAR1 OR City=#PAR2", "Sales Agent", "london");
I would assume that you've looked at LINQ and ADO.Net Data Services and these do not meet some of your requirements?
Native ADO.Net is a database provider, and therefore it provides a direct SQL interface into the underlying data sources. There is various CRUB based solutions out there that simulate what you suggest to various degrees.
We have a strong internal trend to leave the database to the Database team and use Webservices as a main interface to our databases, mainly because of a Delphi codebase that still needs to be supported.
Actually can't believe I forgot to add ADO.Net Entity Framework which is used by ADO.Net Data Services among others. There is also a LINQ to Entities provider.
I did something like this with a stored procedure once. Basically, I wanted to specify any permutation of fields to match on in my WHERE clause, but I didn't want to write 100 sprocs with slightly different param lists and where clauses.
So, I did something like this:
CREATE PROCEDURE [GetSimpleCustomers]
(
#ID varchar(50) = null,
#Name varchar(50) = null,
#IsActive bit = null,
#Address1 varchar(50) = null,
#Address2 varchar(50) = null,
#City varchar(50) = null,
#State varchar(50) = null,
#Zip varchar(50) = null
)
AS
SELECT ID, Name, IsActive, Address1, Address2, City, State, Zip
FROM SimpleCustomerExample
WHERE (ID = #ID OR #ID is NULL)
AND (Name = #Name OR #Name is NULL)
AND (IsActive = #IsActive or #IsActive is NULL)
AND (Address1= #Address1 or #Address1 is NULL)
AND (Address2= #Address2 or #Address2 is NULL)
AND (City= #City or #City is NULL)
AND (State= #State or #State is NULL)
AND (Zip= #Zip or #Zip is NULL)
This will let you call the sproc in your code and only pass the params you are interested in filtering on, and the rest will not be factored in if you leave them null.
So, you can do something like
public List<SimpleCustomer> GetAllCustomersFromOhio()
{
List<SimpleCustomer> list = new List<SimpleCustomer>();
using (SqlCommand cmd = new SqlCommand(blah blah))
{
cmd.Parameters.AddWithValue("State", "Ohio");//or "OH" depending on your convention
using(IDataReader oDR = cmd.ExecuteReader())
{
//hydrate your list of SimpleCustomers from the record set.
}
}
return list;
}
EDIT:
In reply to comment:
You could easily enough alter the GetSimpleCustomers to be DeleteSimpleCustomers by changing the
SELECT <columns> FROM SimpleCustomers
to
DELETE FROM SimpleCustomers
and keep the same logic. The same is true for an Update.
Also, I'll answer a question with a question: How many tables do you have that actually need this level of custom filtering? The syntax would be so similar you could pound it all out in a day (or less if you hammered together a simple script to write it for you).
If you are using strongly-typed DataSets, you can create parameterized queries in the Visual Studio Editor by adding identifiers prefixed with # in the query. Create a DataSet XSD file in Visual Studio and create a new table called Products, then add a new query to it.
For example:
select * from Products where Category = #category;
This will autogenerate methods for filled datasets or getting datatables that take the extra parameter. It will also handle propery escaping the strings (uses parameters on the command objects). I used this to create some super simple prototype web apps fairly quickly.
I recently wrote a query 'framework' for generating SQL where clauses.
The primary component is a BaseQueryArgs class with a ToWhereClause() function that uses reflection to convert properties into string sections. This needs to handle the work of escaping and properly formatting values.
Any class inheriting BaseQueryArgs simply needs to declare public properties, and you end up with a strongly typed query object. For optional properties, you make the value nullable (ref type or Nullable<>) and the SQL generator filters out null values.
You can use custom attributes to define extra features for each property:
custom column name that differs from the property name
custom value handling (such as a date value being used as a BETWEEN test expression)
This can be used to create queries with a strongly-typed query object like so:
MyCustomQueryArgs args = new MyCustomQueryArgs
{
ProductFamilyID = 17,
Region = Regions.Northwest,
Active = true
};
List<Product> product = QueryProcessor.GetProductsWhere(args);
GetProductsWhere() would obviously call some data method that accesses the view with the generated SQL.
I don't have a solution for updates/deletes, but it doesn't seem that hard to write a method that converts an object instance into a SQL statement using a switch or attribute to determine table name.
This is very "roll your own", but that gives you the freedom to customize it for your needs, and doesn't include lots of heavy ORM/DAO wrapping.
Have a look at Mindscapes Lightspeed products
It builds stongly typed LINQ queriable models that results in efficient SQL code accross a variety of database engines and includes Memcached and Lucene support
I have used XPO on several projects and their newer version has better support for queries.
http://www.devexpress.com/Products/NET/ORM/
The implementation, like all of them is not without its drawbacks however.
I use Data Abstract for my projects.