Efficient way to find if a given string is _not_ listed in (sqlite3) table - sql

Question

I have a Db table listing media files which have been archived to LTO (4.3 million of them). The ongoing archiving process is manual, carried out by different people as and when downtime arises. We need an efficient way of determining which files in a folder are not archived so we can complete the job if needed, or confidently delete the folder if it's all archived.
(For the sake of argument let's assume all filenames are unique, we do need to handle duplicates but that's not this question.)
I should probably just fire up Perl/Python/Ruby and talk to the Db thru them. But it would take me quite a while to get back up to speed in those and I have a nagging feeling that it would be overkill.
I can think of a two simpler approaches, but each has drawbacks and I wonder if there's a yet better way?
Method 1: is to simply bash-recurse down each directory structure, invoking sqlite3 per-file and outputting the filename if the query returns and empty result
This is probably less efficient than
Method 2: recurse through the directory structure and produce an sql file which will:
create a table with all our on-disk files in it (let's call it the "working table")
compare that with the archive table - select all files in the working table but not in the archive table
destroy the working table, or quit without saving
While 2 seems likely more efficient than 1, it seems that building the comparison table in the first place might incur some overhead and I did kind of imagine the backup table as a monolithic read-only thing that people refer to and don't write into.
Is there any way in pure SQL to just output a list of not-founds (without them existing in another table)?

Answer 1

Finding values not in some other table is easy:
SELECT *
FROM SomeTable
WHERE File NOT IN (SELECT File
FROM OtherTable);
To create the other table, you can write a series of INSERT statements, or just use the .import command of the shell from a plain text file.
A temporary table will not be saved.

Answer 2

Sooo, I think I have to answer my own question.
tl;dr - use a scripting language (the thing I was hoping to avoid)
Trying that and the other two approaches (details below) on my system yields the following numbers when checking a 33-file directory structure against the 4.3 million record Db:
A Ruby script: 0.27s
Bash running sqilte3 once per file ("Method 1"): 0.73s
SQL making a temp table and using "NOT IN" (Method 2): 8s
The surprising thing for me is that the all-sql is an order of magnitude slower than bash. This was true using the macOS (10.12) commandline sqlite3 and the GUI "DB Browser for SQLite"
The details
Script method
This is the crux of my Ruby script. Ruby of course is not the fastest language out there and you could probably do better than this (but if you really need speed, it might be time for C)
require "sqlite3"
db = SQLite3::Database.open 'path/to/mydb.db'
# This will skip Posix hidden files, which is fine by me
Dir.glob("search_path/**/*") do |f|
file = File.stat(f)
next unless file.file?
short_name = File.basename(f)
qouted_short_name = short_name.gsub("'", "''")
size = File.size(f)
sql_cmd = "select * from 'Backup_Table' where filename='#{qouted_short_name}' and sizeinbytesincrsrc=#{size}"
count = db.execute(sql_cmd).length
if count == 0
puts "UNARCHIVED: #{f}"
end
end
(Note the next two are Not The Answer, but I'll include them if anyone wants to check my methodology)
Bash
This is a crude Bash recurse-through-files which will print a list of files that are backed up (not what I want, but gives me an idea of speed):
#! /bin/bash
recurse() {
for file in *; do
if [ -d "${file}" ]; then
thiswd=`pwd`
(cd "${file}" && recurse)
cd "${thiswd}"
elif [ -f "${file}" ]; then
fullpath=`pwd`${file}
filesize=`stat -f%z "${file}"`
sqlite3 /path/to/mydb.db "select filename from 'Backup_Table' where filename='$file'"
fi
done
}
cd "$1" && recurse
SQL
CL has detailed method 2 nicely in his/her answer