Importing CSV columns into SQL table without Sqlcmd - sql

I’m currently importing a csv file and inserting some of the results into a table.
Unfortunately I’m using Sqlcmd, but the module isn’t installed on the relevant server and can’t be installed (out of my control).
Is there a way to manage the exact same as the below, outside of SqlCMD?
Example Code:
$server = 's'
$database = 'd'
$table = 't'
Import-CSV C:\Test\Test.csv | ForEach-Object {Invoke-Sqlcmd `
-Database $database -ServerInstance $server `
-Query "INSERT INTO $table VALUES ('$($_."Column One")',
'$($_."Column Two")',
NULL,
'$($_.""Column Three"")')"
}

You can try the ADO.Net objects:
# Hardcode the table name, unless you *REALLY* trust your users.
$SQL = "INSERT INTO TableName VALUES (#Column1, #Column2, NULL, #Column3)"
$conn = new-Object System.Data.SqlClient.SqlConnection
$conn.ConnectionString = "Server=$server;Database=$database;Integrated Security=True;"
$cmd = new-Object System.Data.SqlClient.SqlCommand($SQL, $conn)
# Use actual types and lengths from the database here
$c1 = $cmd.Parameters.Add("#Column1", [System.Data.SqlDbType]::NVarChar, 20)
$c2 = $cmd.Parameters.Add("#Column2", [System.Data.SqlDbType]::NVarChar, 20)
$c3 = $cmd.Parameters.Add("#Column3", [System.Data.SqlDbType]::NVarChar, 20)
$conn.Open()
Import-CSV C:\Test\Test.csv | ForEach-Object {
$c1.Value = $_."Column One"
$c2.Value = $_."Column Two"
$c3.Value = $_."Column Three"
$cmd.ExecuteNonQuery()
}
$conn.Close()
This will also fix any issues with apostrophes in the data, which would have caused HUGE problems in the original.

You could try and use the bulk insert command, search for a simple SQL connection and query execution - the exact one that Joel Coehoorn posted, if you have bulk permissions and the server has access to the file itself. you could also add parameters to it and then call it however you see fit. Meaning you could foreach files in folder, call the bulk insert with your connection.
BULK INSERT TableName
FROM 'LinkToSourceFile.csv' WITH (
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\n',
FIRSTROW = 2
);

Related

Oracle SQL query returns some data when it should be empty-PowerShell

I am writing a script in PowerShell Core 7.2. I get a list of files from a folder that I check against Oracle db. I need the data of Description, and NC_Name column if the file is in db.
The issue is that even when the file is not in db it still returns the data but of some other file.
For example: I have a list of files, File#1, File#2,File#3. If File#2 is not in the db it still returns the data of File#1.
I have tried counting the number of rows and putting it as a condition. As in
$rowNum = $connection.count
The issue with this is that $rowNum is never zero because it returns data for some other file; because the variable $fileName is never empty.
I also tried checking for the file name in the query itself but it gave a lot of errors. The query was
$query="DECLARE record_exists INTEGER; BEGIN SELECT COUNT(*) INTO record_exists FROM NC_PROGRAMS WHERE NC_PROGRAMS.NC_NAME = '$fileName' AND ROWNUM = 1; IF record_exists = 1 THEN Select DESCRIPTION, NC_NAME"
The code is:
#Get all files
$result = $start.EnumerateDirectories() | ForEach-Object -Parallel {
$_.GetFiles('*.EIA', $using:enum)
}
$result | Format-Table -AutoSize
foreach($item in $result){
$fileName = $item.BaseName
#Oracle connection
Add-Type -Path C:\lib\netstandard2.1\Oracle.ManagedDataAccess.dll
$query = "Select DESCRIPTION, NC_NAME From NC_PROGRAMS WHERE
NC_PROGRAMS.NC_NAME = '$fileName' "
$connectionString = "connectionString"
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.Open()
$command = $connection.CreateCommand()
$command.CommandText = $query
$reader = $command.ExecuteReader()
$rowNum = $connection.count
Write-host "Number of rows-"$rowNum
while($reader.Read()) {
$description=$reader.GetString(0)
$fastemsFileName = $reader.GetString(1)
}
$connection.Close()
}

PowerShell Query SQL IF Column = 0

I'm a bit out of my wheelhouse on this one, but what I'm trying to do is a have a web form gather all our new user information, dump it in a table then use those columns to populate the data for my PowerShell user provisioning scripts.
The part I'm struggling with is querying the data and getting it into variables in PowerShell. Assume a table with basic GivenName,SurName,Title etc. columns. I want to pull all rows of data (new users) that the "Created" column equals 0, and put each column into a variable to use. At the end of the user creation script, I will update the "Created" column to 1 to avoid the user being re-created next batch run but the row will remain.
Any tips on the best way to go about this?
So we do this alot, so much that we created a function we always leverage for taking data in or out of SQL.
function Get-SQL ($query,$server,$database, $username, $password) {
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
if ($PSBoundParameters.ContainsKey('username') -eq $true -and $PSBoundParameters.ContainsKey('password') -eq $true) {
$SqlConnection.ConnectionString = "Server = $server; Database = $database; User ID=$username; Password=$password;"
} else {
$SqlConnection.ConnectionString = "Server = $server; Database = $database; Integrated Security = True"
}
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $query
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet) | Out-Null
$SqlConnection.Close()
return $DataSet.Tables[0]
}
Now dont get scared of all that craziness, just put it at the top of your script (we actually have it in a global functional library we call on all of our automation scripts but putting it inline will be fine)
Once that function is there you can simply do this:
$MyData = get-sql -server "MyServer" -database "MyDatabaseName" -Query "select * from myTable where created = 0"
Then you will be able to access the all columns via the $MyData object. But in reality you will want to do a foreach loop to really work with each record.
foreach ($whatever in $myData) {
write-output "Now working on record: $($whatever.GivenName) $($whatever.SurName)"
# Do more stuff you want
#now Update SQL
get-sql -server "MyServer" -database "MyDatabaseName" -Query "update myTable set created = 1 where UniqueUserID = '$($whatever.UniqueUserID)'"
}
Last things to note, you need a unique User ID. This can be anything, here we always use objectGUID from Active Directory but if you dont have that then setup UserID field in the db, and have it auto increment an int. You always want a immutableID (meaning something that will never change like a name or an email will). Also notice when using vars in a string I always use $($var) syntax, works when your strings are in double quotes and can help avoid issues.
This should be enough to get you going... good luck!

Powershell script executing SQL procedure with parameters has truncated output

I have a Powershell script that is successfully connecting to a SQL Server database and executing a procedure. The SQL procedure contains a select as follows:
SELECT #sql AS 'ColDemo1'
The variable #sql is nvarchar(max). I want the full contents of #sql returned to a new sql file however I only get the first line and an unwanted column heading as below:
ColDemo1
------------
First line of data...
The Powershell script is as follows:
$server = "SERVER\DEMO"
$database = "dbDemo"
$pZero = "PVal0"
$pOne = "PVal1"
$pTwo = "PVal2"
$pThree = "PVal3"
function Run-SQLUSP {
param (
$pZero,
$pOne,
$pTwo,
$pThree
)
$conn = New-Object System.Data.SqlClient.SqlConnection("Server=${server};Database='${database}';Integrated Security=TRUE")
$conn.Open()
$cmd = $conn.CreateCommand()
$cmd.CommandText = "dbo.demoSp '$pZero', '$pOne', '$pTwo', '$pThree'"
$adapter = New-Object System.Data.SqlClient.SqlDataAdapter($cmd)
$dataset = New-Object System.Data.DataSet
[void]$adapter.Fill($dataset)
$dataset.tables[0]
}
Run-SQLUSP $pZero $pOne $pTwo $pThree | Out-File "c:\DemoFolder\DemoScriptName.sql" -width 8000
How can I amend my Powershell script to return all lines of the output to the new sql file? When executing the procedure in SSMS the resultset is returned in a single table cell (not multiple rows).
Two changes were required:
To avoid truncating the output: Change Out-File to Set-Content -Path and remove -width 8000.
To only return the first cell: Append .rows[0][0] to $dataset.tables[0] within the function.
Below is the revised code in full:
$server = "SERVER\DEMO"
$database = "dbDemo"
$pZero = "PVal0"
$pOne = "PVal1"
$pTwo = "PVal2"
$pThree = "PVal3"
function Run-SQLUSP {
param (
$pZero,
$pOne,
$pTwo,
$pThree
)
$conn = New-Object System.Data.SqlClient.SqlConnection("Server=${server};Database='${database}';Integrated Security=TRUE")
$conn.Open()
$cmd = $conn.CreateCommand()
$cmd.CommandText = "dbo.demoSp '$pZero', '$pOne', '$pTwo', '$pThree'"
$adapter = New-Object System.Data.SqlClient.SqlDataAdapter($cmd)
$dataset = New-Object System.Data.DataSet
[void]$adapter.Fill($dataset)
$dataset.tables[0].rows[0][0]
}
Run-SQLUSP $pZero $pOne $pTwo $pThree | Set-Content -Path "c:\DemoFolder\DemoScriptName.sql"
I was having the same issue, then discovered I could fix it if I used the Column Name to access the cell value from the table row.
#Trunacted
$data.Tables[0].Rows[0][0]
#Complete data
$data.Tables[0].Rows[0]["XMLData"]

How to return the value of a row, without column name in a query?

I am writing a Powershell script that extracts data via the SQLPS module, executing a query directly to the SQL Server. If I do a plain
Select <column A> from <table B>
I get the column listed as well, as stated like this:
Column A
--------
Value C
Here I wish to only retrieve the Value C, for storing it as a variable.
If you are not bound to use this SQLPS module then this might be a easier way to do it:
$connection = new-object System.Data.SqlClient.SqlConnection("Data Source=.;Initial Catalog=TestDB;Integrated Security=True");
$connection.Open()
$query = "SELECT [A] FROM [dbo].[Tablename]"
$cmd = new-object "System.Data.SqlClient.SqlCommand" ($query, $connection)
$cmd.CommandTimeout = 0
$executeReader = $cmd.ExecuteReader()
while ($executeReader.Read()) {
$Name = $executeReader.GetValue(0)
//Do what you desire with the resultset.
$Name + "`r`n" >> D:\PathToResultFolder\result.txt
}
$executeReader.Close()
$connection.Close()
Also I read and think that this should be handled outside of the Query as it is not normal for a Query to not show column-names.

SQL Query with Powershell - Return Values not maching

So I have some code I wrote to take the input form a text file and run some sql checks against another database I populated:
$volOutput = gc C:\Users\<user>\Desktop\mutant.txt
foreach ($m in $volOutput) {
$check = $m.split()[-1] | select -Unique
foreach ($c in $check) {
#$c - this lists all of them so the foreach is working...
# Build the connection and search the db for $c names.
$conn = New-Object System.Data.SqlClient.SqlConnection
$conn.ConnectionString = "Server=(localdb)\mutex; Database=MutexObjects"
$conn.Open()
$db = $conn.CreateCommand()
$db.CommandText = "select Names from Objects WHERE Names='$c'"
$db.ExecuteScalar()
$conn.Close()
} # Foreach Check
} # First foreach
The return values I get are:
PS C:\> B:\Programming\powershell\parse_vol.ps1
ZonesCounterMutex
ZoneAttributeCacheCounterMutex
ZonesCacheCounterMutex
ZoneAttributeCacheCounterMutex
ZonesLockedCacheCounterMutex
ZonesCounterMutex
ZoneAttributeCacheCounterMutex
ZonesCacheCounterMutex
ZoneAttributeCacheCounterMutex
ZonesLockedCacheCounterMutex
Which is correct, but it's also missing a lot more. If I take individual samples and run queries from within SQL management studio for example, I get:
I populated the word "test" in each list as a....test.
Select Names From Objects WHERE Names='test'
Names
test
But I don't see test on the output from the above code. There are about 5 or 6 more than it's missing that I have validated manually by querying the db w/in SQL management studio.
Any help is much appreciated.
Compare the file content against the full list of Names from the database:
$filecontent = Get-Content "C:\Users\<user>\Desktop\mutant.txt" `
| % { $_.split()[-1] } `
| select -Unique
$conn = New-Object System.Data.SqlClient.SqlConnection
$conn.ConnectionString = "Server=(localdb)\mutex; Database=MutexObjects"
$conn.Open()
$dbwrite = $conn.CreateCommand()
$dbwrite.CommandText = "SELECT Names FROM Objects"
$reader = $dbwrite.ExecuteReader([System.Data.CommandBehavior]::CloseConnection)
$dbcontent = while ( $reader.Read() ) { $reader[0] }
$conn.Close()
Compare-Object $filecontent $dbcontent
Does Compare-Object show differences?