I am facing a situation here.
So below are the steps I am using to transform my DataFrame
val filteringRecordsToExpire = collectAllActiveRecords.join(collectingSrcSysIdsToExpire, Seq("trans_id"), "leftsemi")
filteringRecordsToExpire contains few of the IDS which I need to make Invalid
val expiredList = filteringRecordsToExpire.select("trans_id").distinct().collect()
expiredList.foreach(v => expireRecords(v(0).toString)) --> Here I am Updating each record
Now I want to use those Same IDs that I expired and further want to re-Enter them in the same Table with some new Values.
But I am getting an Empty DataFrame After I perform the Expire ( Which is basically updating the existing table for those same IDs )
collectingSrcSysIdsToExpire - So this DataFrame holds all those IDs which further I want to modify and INSERT into the Table.
But in this process The Whole Dataframe is going empty.
I have tried persisting this dataframe and Also registered to a Temp Table and tried using it. But nothing is working.
Any Help or suggestion would be a big help. Thanks in advance.
-----------------------------solution----------------------------------
So here is how I solved this issue.
As Suggested I used MERGE INTO which was a lot faster and as I am using unique transaction ids so I dint have any duplicate issues. Previously I was Updating the table for those transaction IDS then tried to use those same unique IDs with modified values and INSERT INTO the same table.
As a solution I first picked distinct transaction ids from my source and INSERT INTO the table with my updated values , then stored those same list of transaction ids and updated existing older record entries from the table.
val filteringRecordsToExpire = delta.join(collectingSrcSysIdsToExpire, Seq("trans_id"), "leftsemi")
.distinct()
collectingSrcSysIdsToExpire.select(TargetTable.schema.map(f => col(f.name)): _*).write.insertInto(Table)
val sqlUpdateQry =
s""" MERGE INTO TargetTable as tgtTable
USING expireSrsIds as source
ON tgtTable.trans_id = source.trans_id
AND few more conditions
WHEN MATCHED
THEN UPDATE SET
expiring older entries
So somehow INSERT then UPDATE works sequentially.
But UPDATE then INSERT does not work.
The foreach is by definition doesn't return any data - you can see from API docs that the return type is Unit. I also don't recommend to update individual records - it will be too slow as it will rewrite the data for each record separately. Instead, use the MERGE operation, with something like this (it's not Scala, just algorithm):
sourceTable
.as("source")
.merge(
dfUpdates.as("updates"),
"source.id = updates.id")
.whenMatched
.updateExpr(
Map(
"status" -> "'expired"
))
See MERGE documentation for full details. Also instead of updating records, you can delete them.
For example, if I want to frequently update a user's score during a session,
is there a more performant way than - UPDATE score FROM [databaseName] WHERE name = [username]
I feel like I should not be continuously needing to search through the entire database when the value's location has been previously found.
Thanks in advance
You wouldn't search through the entire database if you had an index on name:
create index idx_<table>_name on <table>(name);
where <table> is the name of the table you are referring to, not the database.
Incidentally, if you wanted to update the table, you would use update, not select. But update can still use the index.
You can use properties here you can search and update only the database when you need it.
private int score;
public int Score
{
get
{
score = searchDatabase();
return score;
}
set
{
UpdateDatabase(value);
score = value;
}
}
Debug.Log(Score);
The only thing that comes to mind that resembles what you mention as the value's location, is the value of the primary key column of the row with the value that you search for.
So, if the primary key in the table is id and you only have the name of the person, you can query once to get the row's id:
SELECT id FROM tablename WHERE name = ?
and then use that id in all subsequent updates:
UPDATE tablename SET score = ? WHERE id = ?
This is the fastest way to do the updates.
Someone deployed a SQL table with the schema
ConfigOptions
name VARCHAR(50)
value VARCHAR(50)
and the following logic for saving options:
int i = ExecuteNonQuery("UPDATE ConfigOptions SET value=#value WHERE name=#name");
if(i==0) i = ExecuteNonQuery("INSERT INTO ConfigOptions (name,value) (#name,#value)");
We now saw that this table is littered with duplicates, and we want to change this.
As far as I can tell, the logic is: whenever the UPDATE affected zero rows, another row is inserted. If I am not mistaken, this can be caused by:
a row by the name of #name does not exist or
the row exists, but already contains value #value
So, all rows with same name should be full duplicates. If now, something is completely wrong (and behaviour may be undefined).
Now I have to fix this problem of duplicates, so I want to add a PK on name. Before I can do this, I have to remove all rows with duplicate names, only keeping one of each.
In the installer (only the installer is allowed to change schema), I only have SQL queries at hand, so I can't do it with C# logic:
Dictionary<string, int> dic = new Dictionary<string, int>();
SqlDataReader sdr = ExecuteReader("SELECT name,COUNT(value) FROM ConfigOptions GROUP BY name HAVING COUNT(value)>1");
while (sdr.Read()) dic.Add(sdr.GetString(0), sdr.GetInt32(1));
sdr.Close();
foreach (var kv in dic) {
AddParameter("#name", System.Data.SqlDbType.VarChar, 50, kv.Key);
ExecuteNonQuery("DELETE TOP " + (kv.Value - 1) + " FROM ConfigOptions WHERE name=#name");
}
ExecuteNonQuery("ALTER TABLE program_options ADD PRIMARY KEY (name)");
Is there a way to put this into SQL logic?
Using %%physloc%%, the phys(ical) loc(ation) of the row, should do the trick:
DELETE FROM ConfigOptions
WHERE %%physloc%% NOT IN (
SELECT MIN(%%physloc%%)
FROM ConfigOptions
GROUP BY name);
After this cleanup, you can add the primary key to the table.
NOTE: this will leave you with only one row for every name. If the value column is different in two records with the same name, you will lose the newest record. If you want to change this, use GROUP BY name, value.
I have a MySQL table with a primary key field that has AUTO_INCREMENT on.
After reading other posts on here I've noticed people with the same problem and with varied answers. Some recommend not using this feature, others state it can't be 'fixed'.
I have:
table: course
fields: courseID, courseName
Example: number of records in the table: 18. If I delete records 16, 17 and 18 - I would expect the next record entered to have the courseID of 16, however it will be 19 because the last entered courseID was 18.
My SQL knowledge isn't amazing but is there anyway to refresh or update this count with a query (or a setting in the phpMyAdmin interface)?
This table will relate to others in a database.
Given all the advice, I have decided to ignore this 'problem'. I will simply delete and add records whilst letting the auto increment do it's job. I guess it doesn't really matter what the number is since it's only being used as a unique identifier and doesn't have a (as mentioned above) business meaning.
For those who I may have confused with my original post: I do not wish to use this field to know how many records I have. I just wanted the database to look neat and have a bit more consistency.
What you're trying to do sounds dangerous, as that's not the intended use of AUTO_INCREMENT.
If you really want to find the lowest unused key value, don't use AUTO_INCREMENT at all, and manage your keys manually. However, this is NOT a recommended practice.
Take a step back and ask "why you need to recycle key values?" Do unsigned INT (or BIGINT) not provide a large enough key space?
Are you really going to have more than 18,446,744,073,709,551,615 unique records over the course of your application's lifetime?
ALTER TABLE foo AUTO_INCREMENT=1
If you've deleted the most recent entries, that should set it to use the next lowest available one. As in, as long as there's no 19 already, deleting 16-18 will reset the autoincrement to use 16.
EDIT: I missed the bit about phpmyadmin. You can set it there, too. Go to the table screen, and click the operations tab. There's an AUTOINCREMENT field there that you can set to whatever you need manually.
Primary autoincrement keys in database are used to uniquely identify a given row and shouldn't be given any business meaning. So leave the primary key as is and add another column called for example courseOrder. Then when you delete a record from the database you may want to send an additional UPDATE statement in order to decrement the courseOrder column of all rows that have courseOrder greater than the one you are currently deleting.
As a side note you should never modify the value of a primary key in a relational database because there could be other tables that reference it as a foreign key and modifying it might violate referential constraints.
Try :
SET #num := 0;
UPDATE your_table SET id = #num := (#num+1);
ALTER TABLE `your_table` AUTO_INCREMENT = 1;
That'll reset the autoincremented value, and then count every row while a new value is created for it.
example : before
1 : first value here
2 : second value here
X : deleted value
4 : The rest of the table
5 : The rest of the rest..
so the table will display the array : 1,2,4,5
Example : AFTER (if you use this command you will obtain)
1 : first value here
2 : second value here
3 : The rest of the table
4 : the rest of the rest
No trace of the deleted value, and the rest of the incremented continues with this new count.
BUT
If somewhere on your code something use the autoincremented value... maybe this attribution will cause problem.
If you don't use this value in your code everything should be ok.
You shouldn't be relying on the AUTO_INCREMENT id to tell you how many records you have in the table. You should be using SELECT COUNT(*) FROM course. ID's are there to uniquely identifiy the course and can be used as references in other tables, so you shouldn't repeat ids and shouldn't be seeking to reset the auto increment field.
I came here looking for an answer to the Title question "MySQL - Auto Increment after delete" but I could only find an answer for that in the questions
How to delete certain row from mysql table?
How to reset AUTO_INCREMENT in MySQL?
By using something like:
DELETE FROM table;
ALTER TABLE table AUTO_INCREMENT = 1;
Note that Darin Dimitrov's answer explain really well AUTO_INCREMENT and it's usage. Take a look there before doing something you might regret.
PS: The question itself is more "Why you need to recycle key values?" and Dolph's answer cover that.
What you are trying to do is very dangerous. Think about this carefully. There is a very good reason for the default behaviour of auto increment.
Consider this:
A record is deleted in one table that has a relationship with another table. The corresponding record in the second table cannot be deleted for auditing reasons. This record becomes orphaned from the first table. If a new record is inserted into the first table, and a sequential primary key is used, this record is now linked to the orphan. Obviously, this is bad. By using an auto incremented PK, an id that has never been used before is always guaranteed. This means that orphans remain orphans, which is correct.
There is actually a way to fix that. First you delete the auto_incremented primary key column, and then you add it again, like this:
ALTER TABLE table_name DROP column_name;
ALTER TABLE table_name ADD column_name int not null auto_increment primary key first;
you can select the ids like so:
set #rank = 0;
select id, #rank:=#rank+1 from tbl order by id
the result is a list of ids, and their positions in the sequence.
you can also reset the ids like so:
set #rank = 0;
update tbl a join (select id, #rank:=#rank+1 as rank from tbl order by id) b
on a.id = b.id set a.id = b.rank;
you could also just print out the first unused id like so:
select min(id) as next_id from ((select a.id from (select 1 as id) a
left join tbl b on a.id = b.id where b.id is null) union
(select min(a.id) + 1 as id from tbl a left join tbl b on a.id+1 = b.id
where b.id is null)) c;
after each insert, you can reset the auto_increment:
alter table tbl auto_increment = 16
or explicitly set the id value when doing the insert:
insert into tbl values (16, 'something');
typically this isn't necessary, you have count(*) and the ability to create a ranking number in your result sets. a typical ranking might be:
set #rank = 0;
select a.name, a.amount, b.rank from cust a,
(select amount, #rank:=#rank+1 as rank from cust order by amount desc) b
where a.amount = b.amount
customers ranked by amount spent.
I can think of plenty of scenarios where you might need to do this, particularly during a migration or development process. For instance, I just now had to create a new table by cross-joining two existing tables (as part of a complex set-up process), and then I needed to add a primary key after the event. You can drop the existing primary key column, and then do this.
ALTER TABLE my_table ADD `ID` INT NOT NULL AUTO_INCREMENT FIRST, ADD PRIMARY KEY (`ID`);
For a live system, it is not a good idea, and especially if there are other tables with foreign keys pointing to it.
I got a very simple but tricky method.
While deleting a row, you can preserve the IDs into another temporary table. After that, when you will insert new data into the main table then you can search and pick IDs from the temporary table. So use a checking here. If the temporary table has no IDs then calculate maximum ID into the main table and set the new ID as: new_ID = old_max_ID+1.
NB: You can not use auto-increment feature here.
You may think about making a trigger after delete so you can update the value of autoincrement and the ID value of all rows that does not look like what you wanted to see.
So you can work with the same table and the auto increment will be fixed automaticaly whenever you delete a row the trigger will fix it.
You can use your mysql client software/script to specify where the primary key should start from after deleting the required records.
Its definitely not recommendable. If you have a large database with multiple tables, you may probably have saved a userid as id in table 2. if you rearrange table 1 then probably the intended userid will not end up being the intended table 2 id.
MYSQL Query
Auto Increment Solution. It works perfect when you have inserted many records during testing phase of software. Now you want to launch your application live to your client and You want to start auto increment from 1.
To avoid any unwanted problems, for safer side
First export .sql file.
Then follow the below steps:
Step 1)
First Create the copy of an existing table
MySQL Command to create Copy:
CREATE TABLE new_Table_Name SELECT * FROM existing_Table_Name;
The exact copy of a table is created with all rows except Constraints.
It doesn’t copy constraints like Auto Increment and Primary Key into new_Table_name
Step 2)
Delete All rows If Data is not inserted in testing phase and it is not useful.
If Data is important then directly go to Step 3.
DELETE from new_Table_Name;
Step 3) To Add Constraints, Goto Structure of a table
3A) Add primary key constraint from More option (If You Require).
3B) Add Auto Increment constraint from Change option. For this set Defined value as None.
3C) Delete existing_Table_Name and
3D) rename new_Table_Name to existing_Table_Name.
Now It will work perfectly. The new first record will take first value in Auto Increment column.
Here is a step to solve your problem.
On your .php file, just add this query given below:
<?php
$servername = "localhost";
$username = "root";
$password = "";
$dbname = "";
$conn = new mysqli($servername, $username, $password, $dbname);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
//write the number or id you want to start with the next user in AUTO_INCREMENT
$sql = "ALTER TABLE `table_name` AUTO_INCREMENT = number";
$conn->query($sql);
?>
I hope your problem will be solved.
if($id == 1){ // deleting first row
mysqli_query($db,"UPDATE employees SET id=id-1 WHERE id>1");
}
else if($id>1 && $id<$num){ // deleting middle row
mysqli_query($db,"UPDATE employees SET id=id-1 WHERE id>$id");
}
else if($id == $num){ // deleting last row
mysqli_query($db,"ALTER TABLE employees AUTO_INCREMENT = $num");
}
else{
echo "ERROR";
}
mysqli_query($db,"ALTER TABLE employees AUTO_INCREMENT = $num");
here is a function that fix your problem
public static void fixID(Connection conn, String table) {
try {
Statement myStmt = conn.createStatement();
ResultSet myRs;
int i = 1, id = 1, n = 0;
boolean b;
String sql;
myRs = myStmt.executeQuery("select max(id) from " + table);
if (myRs.next()) {
n = myRs.getInt(1);
}
while (i <= n) {
b = false;
myRs = null;
while (!b) {
myRs = myStmt.executeQuery("select id from " + table + " where id=" + id);
if (!myRs.next()) {
id++;
} else {
b = true;
}
}
sql = "UPDATE " + table + " set id =" + i + " WHERE id=" + id;
myStmt.execute(sql);
i++;
id++;
}
} catch (SQLException e) {
e.printStackTrace();
}
}
Let's say I have a table called Product, with three columns: Id, CustomerId, Name. Id is the primary key. The schema is outside of the control of my group, and we now have a requirement to always provide CustomerId as a parameter for all queries (selects, updates, deletes). It's a long story I'd rather not get into ... it involves triggers :-P
So my question is, when I have an attached entity in LinqToEntities, and I want to save some updates (say I'm updating the name in this case). How can I get it to generate the SQL:
update Product set Name = #Name where Id=#Id and CustomerId=#CustomerId
Where the customerId parameter is included in the where clause in addition to the primary key.
Thanks :-)
Does the CustomerId help uniquely identify the row past #Id? I didn't really follow the "triggers" bit, since the predicate used for the update is not known by the trigger. Or you do want to re-update the CustomerId each time (detectable from UPDATE(...) in the trigger)
The easiest option is to do it as object updates:
var qry = from product in model.Products
where Id == #Id && CustomerId == #CustomerId
select product;
foreach(Product p in qry) {
p.Name = #Name;
}
model.SaveChanges(); // or whatever the method is in EF
If you know you are expecting one record, you could use:
Product prod = (from product in model.Products
where Id == #Id && CustomerId == #CustomerId
select product).Single();
prod.Name = #Name;
mode.SaveChanges(); // ditto
You might also be able to write it as Entity-SQL, but I'm not sure I'd bother, personally... (update: I've just checked, and I don't think Entity-SQL includes DML, so no, you can't - you'd have to use either the above, or a regular SQL command/SPROC)
One way would be to use a stored proc to do the update. This gives you complete control over the SQL.
Another way is to add the CustomerId to the entity key.