PostgreSQL import from CSV NULL values are text - Need null - sql

I had exported a bunch of tables (>30) as CSV files from MySQL database using phpMyAdmin. These CSV file contains NULL values like:
"id","sourceType","name","website","location"
"1","non-commercial","John Doe",NULL,"California"
I imported many such csv to a PostgreSQL database with TablePlus. However, the NULL values in the columns are actually appearing as text rather than null.
When my application fetches the data from these columns it actually retrieves the text 'NULL' rather than a null value.
Also SQL command with IS NULL does not retrieve these rows probably because they are identified as text rather than null values.
Is there a SQL command I can do to convert all text NULL values in all the tables to actual NULL values? This would be the easiest way to avoid re-importing all the tables.

PostgreSQL's COPY command has the NULL 'some_string' option that allows to specify any string as NULL value: https://www.postgresql.org/docs/current/sql-copy.html
This would of course require re-importing all your tables.
Example with your data:
The CSV:
"id","sourceType","name","website","location"
"1","non-commercial","John Doe",NULL,"California"
"2","non-commercial","John Doe",NULL,"California"
The table:
CREATE TABLE import_with_null (id integer, source_type varchar(50), name varchar(50), website varchar(50), location varchar(50));
The COPY statement:
COPY import_with_null (id, source_type, name, website, location) from '/tmp/import_with_NULL.csv' WITH (FORMAT CSV, NULL 'NULL', HEADER);
Test of the correct import of NULL strings as SQL NULL:
SELECT * FROM import_with_null WHERE website IS NULL;
id | source_type | name | website | location
----+----------------+----------+---------+------------
1 | non-commercial | John Doe | | California
2 | non-commercial | John Doe | | California
(2 rows)
The important part that transforms NULL strings into SQL NULL values is NULL 'NULL' and could be any other value NULL 'whatever string'.

UPDATE For whoever comes here looking for a solution
See answers for two potential solutions
One of the solutions provides a SQL COPY method which must be performed before the import itself. The solution is provided by Michal T and marked as accepted answer is the better way to prevent this from happening in the first place.
My solution below uses a script in my application (Built in Laravel/PHP) which can be done after the import is already done.
Note- See the comments in the code and you could potentially figure out a similar solution in other languages/frameworks.
Thanks to #BjarniRagnarsson suggestion in the comments above, I came up with a short PHP Laravel script to perform update queries on all columns (which are of type 'string' or 'text') to replace the 'NULL' text with NULL values.
public function convertNULLStringToNULL()
{
$tables = DB::connection()->getDoctrineSchemaManager()->listTableNames(); //Get list of all tables
$results = []; // an array to store the output results
foreach ($tables as $table) { // Loop through each table
$columnNames = DB::getSchemaBuilder()->getColumnListing($table); //Get list of all columns
$columnResults = []; // array to store the results per column
foreach ($columnNames as $column) { Loop through each column
$columnType = DB::getSchemaBuilder()->getColumnType($table, $column); // Get the column type
if (
$columnType == 'string' || //check if column type is string or text
$columnType == 'text'
) {
$query = "update " . $table . " set \"" . $column . "\"=NULL where \"" . $column . "\"='NULL'"; //Build the update query as mentioned in comments above
$r = DB::update($query); //perform the update query
array_push($columnResults, [
$column => $r
]); //Push the column Results
}
}
array_push($results, [
$table => $columnResults
]); // push the table results
}
dd($results); //Output the results
}
Note I was using Laravel 8 for this.

Related

Sql query 'IN ' operator is not work error?

I am using CI 'in'operator is not work sql error please check its and share valuable idea...
table
enter image description here
id | coach_name
------------------
9 | GS
------------------
10 | SLR
view and function
$coachID = explode(',',$list['coach']);
$coachname = $this->rail_ceil_model->display_coach_name($coachID);
show result
SLR
need result
GS,SLR
last query result here
SELECT coach_name FROM mcc_coach WHERE id IN('9', '10')
CI code
public function display_coach_name($coachID='')
{
$db2 = $this->load->database('rail',TRUE);
$db2->select('coach_name');
$db2->from('mcc_coach');
$db2->where_in('id',$coachID);
$query = $db2->get();
echo $db2->last_query(); die;
if ($query->num_rows() > 0):
//return $query->row()->coach_name;
else:
return 0;
endif;
}
You must provide an array to in operator so #coachId must be an array not a string
If you are writing this query
SELECT coach_name FROM mcc_coach WHERE id IN('9,10')
it means you are applying in operator on a single id which contains a comma separated value.
So, right query will be
SELECT coach_name FROM mcc_coach WHERE id IN('9','10')

How to use where in list items

I have a database as below:
TABLE_B:
ID Name LISTID
1 NameB1 1
2 NameB2 1,10
3 NameB3 1025,1026
To select list data of table with ID. I used:
public static List<ListData> GetDataById(string id)
{
var db = Connect.GetDataContext<DataContext>("NameConnection");
var sql = (from tblB in db.TABLE_B
where tblB.LISTID.Contains(id)
select new ListData
{
Name= tblB.Name,
});
return sql.ToList();
}
When I call the function:
GetDataById("10") ==> Data return "NameB2, NameB3" are not correct.
The data correct is "NameB2". Please help me about that?
Thanks!
The value 10 will cause unintended matches because LISTID is a string/varchar type, as you already saw, and the Contains function does not know that there delimiters that should be taken into account.
The fix could be very simple: surround both the id that you are looking for and LISTID with extra commas.
So you will now be looking for ,10,.
The value ,10, will be found in ,1,10, and not in ,1025,1026,
The LINQ where clause then becomes this:
where ("," + tblB.LISTID + ",").Contains("," + id + ",")

get data from table and from relative foreign tables in SQL

Hi have a text search input that looks for matching records in the DB and gets all the data from a table:
let's say like this:
$q = Input::get('items');
$q = "\"" . "($q)" . "\"";
$items = DB::table('my_items')->whereRaw(
'MATCH(item_name) AGAINST(? IN BOOLEAN MODE)',
array($q)
)->get();
So I get all the items in the DB from my textsearch, then I send the result as json to some script that updates my page with the items:
return Response()->json($items);
The relations are:
My_item:
public function brand(){
return $this->hasOne('App\Brand', 'id', 'brand_id');
}
Brand:
public function My_item(){
return $this->belongsToMany('App\My_item');
}
Now the problem here is that in 'my_items' table I have some data as IDs that reference foreign tables.
For example I will have a 'brand_id' that for example references a 'brands' table where I can have information regarding the brand.
So for example I could have brand_id = 3 that means 'Microsoft' in my brands table (id = 3, name = microsoft).
Now what I need to do is not only passing the brand_id to my view but also the actual information (name), in this case Microsoft so that I can put that info in the item description.
But, how can I get that information before sending with that query? Is there some sort of flag I can use in the query like $items = DB::table bla bla with foreign?
this way works, DB:: method is dropped for:
$items = My_item::with('brand')->where('item_name', 'LIKE', "%$q%")->get();
this one doesn't:
DB::table('my_items')::with('brand')->where('item_name', 'LIKE', "%$q%")->get();
First of all, you can simplify your search query to something like this:
My_item::where('item_name', 'LIKE', "%$q%")->get();
Now, assign relations the relation to your other tables in your Models. Then you can get all information using the following syntax:
My_item::with('brand')->where('item_name', 'LIKE', "%$q%")->get();
Read more about relations here: https://laravel.com/docs/5.1/eloquent-relationships

How to change column name if it contains certain characters

How to change the name of table columns using SQL Server.
The Database table looks like this:
| Column 1 | Column 2 | Column 3 | Q115($) | Q215($) | Q315($) | .... | QXYY($)|
Where new columns are added over time in the format "Quarter"+"Year"+"($)"
I want to write a query that does the following in Microsoft SQL Server:
For all columns that contains "($)":
Change Name of Column from QXYY($) to 20YY QX
Changing column names is probably something you want to do manually instead of as a mass change. You can get a list of all the columns in all tables in your DB using the sys.all_columns table so something like
select all_objects.name as Table_nm, all_columns.name as Column_nm
from sys.all_columns
inner join sys.all_objects
on all_objects.object_id = all_columns.object_id
where all_columns.name like '%$%'
It looks like renaming columns is done through a Stored Procedure, so you could take the output of that query to create a string of other queries to update all the offending records.
https://msdn.microsoft.com/en-us/library/ms188617.aspx
Good luck!
Powershell:
push-location;
import-module sqlps -disablenamechecking;
pop-location;
$s = new-object microsoft.sqlserver.management.smo.server '.';
$tbl = $s.databases['MyDB'].tables['MyTbl'];
foreach ($col in $tbl.Columns) {
if ($col -match '^Q([1-4])([0-9][0-9])\(\$\)$') {
$newName = "20$($matches[2]) Q$($matches[1])";
$col.Rename( $newName );
}
}

How can I store the results of a SQL query as a hash with unique keys?

I have a query that returns multiple rows:
select id,status from store where last_entry = <given_date>;
The returned rows look like:
id status
-----------------
1131A correct
1132B incorrect
1134G empty
I want to store the results like this:
$rows = [
{
ID1 => '1131A',
status1 => 'correct'
},
{
ID2 => '1132B',
status2 => 'incorrect'
},
{
ID3 => '1134G',
status3 => 'empty'
}
];
How can I do this?
What you are looking for is a hash of hash in Perl. What you do is
Iterate over the results of your query.
Split each entry by tab
Create a hash with the id as key and status as value
Now to store the hash created by each such query you create another hash. Here the key could be something like 'given_date' in your case so you could write
$parent_hash{given_date}=\%child_hash
This will results in the parent hash having a reference of each query result.
For more you can refer to these resources:
http://perldoc.perl.org/perlref.html
http://www.thegeekstuff.com/2010/06/perl-array-reference-examples/
Have a look at DBI documentation.
Here is part of script that does what you want:
my $rows;
while(my $hash_ref = $sth->fetchrow_hashref) {
push #$rows, $hash_ref;
}
You can do this by passing a Slice option to DBI's selectall_arrayref:
my $results = $dbh->selectall_arrayref(
'select id,status from store where last_entry = ?',
{ Slice => {} },
$last_entry
);
This will return an array reference with each row stored in a hash. Note that since hash keys must be unique, you will run into problems if you have duplicate column names in your query.
This is the kind of question that raises an immediate red flag. It's somewhat of an odd request to want a collection (array/array reference) of data structures that are heterogeneous---that's the whole point of a collection. If you tell us what you intend to do with the data rather than what you want the data to look like, we can probably suggest a better solution.
You want something like this:
# select the data as an array of hashes - retured as an arrayref
my $rows = $dbh->selectall_arrayref($the_query, {Slice => {}}, #any_search_params);
# now make the id keys unique
my $i = 1;
foreach my $row ( #$rows) {
# remove each column and assign the value to a uniquely named column
# by adding a numeric suffix
$row->{"ID" . $i} = delete $row->{ID};
$row->{"status" . $i} = delete $row->{status};
$i += 1;
}
Add your own error checking.
So you said "save as a hash," but your example is an array of hashes. So there would be a slightly different method for a hash of hashes.