Anonymise SQLite database? - sql

Question

Problem: I have a table with first name, surname, and gender columns. I need to partially anonymise the database, by replacing all the names in this table with arbitrary made-up names. I also have a spreadsheet with lots of gender-specific arbitrary names.
Given this, how do I iterate through the rows of this table, and replace each name in turn with a name from the spreadsheet?
I can do this in C fairly trivially, but it's a days work - export the spreadsheet as CSV, and then iterate through the rows of the table, updating each name with the next one from the CSV file. However, I can't help feeling that there's a much simpler way to do this by turning the CSV name data into a script, but I've got no idea how to iterate through the table from a script. Any pointers/ideas appreciated.

Answer 1

I believe you are on the right track with the application route either with C or Python or whatever you feel convenient. Here is a different method that can be scripted.
Export data from Excel as CSV
$ cat test.csv
Jacob Jacobs,M
Rogers Bogers,M
Marsha Darsha,F
Tina Fina,F
Mono Bono,M
Import this into sqlite
sqlite> .mode csv
sqlite> .import test.csv proxy
sqlite> select * from proxy2;
"Jacob Jacobs",M
"Rogers Bogers",M
"Marsha Darsha",F
"Tina Fina",F
"Mono Bono",M
Remember count of males and females
Let's say your table was called main in which you have real names, and you want to change them to names from proxy table randomly.
sqlite> .schema
CREATE TABLE proxy (fullname text, gender text);
CREATE TABLE main(fullname TEXT,gender TEXT,age INT);
sqlite> select * from main;
fullname,gender,age
"John Smith",M,20
"Marshall Dubin",M,20
"Kate Ortiz",F,20
"Ron Bunsh",M,20
"Kelly Torro",F,20
sqlite> select count(*) from main where gender='M';
count(*)
3
sqlite> select count(*) from main where gender='F';
count(*)
2
Have your application remember this information that there are 3 Males and 2 Females.
Execute update statement repeatedly with different offset
sqlite> update main
...> set fullname = (
...> select fullname from proxy where gender='M' order by random() limit 1)
...> where rowid = (
...> select rowid from main where gender='M' order by rowid limit 0,1);
Change the limit 0,1 to limit 1,1 and re-execute. Go on till you reach limit 2,1. Since you have 3 records for Males, go from limit 0,1 to limit 2,1.
Repeat the same thing to anonymize Female records. Change gender='M' to gender='F'. Since there are only 2 females, you will execute update two times. Once with limit 0,1 and then with limit 1,1.
If you run this in a transaction, my hope that your script should be able to churn through the updates quite fast.
End Result
WAS
fullname gender age
---------- ---------- ----------
John Smith M 20
Marshall D M 20
Kate Ortiz F 20
Ron Bunsh M 20
Kelly Torr F 20
IS
fullname gender age
------------- ---------- ----------
Rogers Bogers M 20
Jacob Jacobs M 20
Tina Fina F 20
Jacob Jacobs M 20
Jasmine F 20
Example of scripting SQLite with Bash - http://andreaolivato.tumblr.com/post/133473114/using-sqlite3-in-bash
Other option
In your application, hold the fake names in two arrays - one for male and one for female. The idea is to be able to pull a random fake name by gender on demand
Do a select rowid, gender from main order by rowid
Iterate through the records
If gender is male, pull a random fake record from male array; likewise for female record
Run update main set fullname=<fake-record> where rowid=<selected-row-id>