Insert records in table using Powershell - sql

I need to copy all data in MyDatabase on SERVER1 to SERVER2.
SERVER1 and SERVER2 are in different domains.
SERVER1 can be reached from SERVER2 using SQL Authentication.
SERVER1 and SERVER2 are not linked.
I only have SELECT rights on Mydatabase on SERVER1.
Only tooling to be used is PowerShell
My idea was to write a powershell script that get the names of all tables in MyDatabase on SERVER1, create a SELECT * FROM table statement and execute those one by one.. Insert the returned records in a new table in MyDatabase on SERVER2.
I used that trick to write the data from a database to CSV files. However, now I want to write the data not to a file but to a table.
See the code below
$server_src = "SERVER1" #$args[0]
$database_src = "MyDatabase" #$args[1]
$auth_src = "UID=MyUser;PWD=Mypassword" #$args[2]
$server_dst = "SERVER2" #$args[3]
$database_dst = "MyDatabase" #$args[4]
$auth_dst = "UID=MyUser;PWD=Mypassword" #$args[5]
# set source
$connection_string_src = "Server = $server_src; Database = $database_src; $auth_src;"
$connection_src = New-Object System.Data.SqlClient.SqlConnection $connection_string_src
# set destination
$connection_string_dst = "Server = $server_dst; Database = $database_dst; $auth_dst;"
$connection_dst = New-Object System.Data.SqlClient.SqlConnection $connection_string_dst
# sql to execute on source - get all tables in database $database_src
$sql_src = "select distinct
'$database_src' as dbname
,s.name as schemaname
,t.name as tablename
, 'SELECT * FROM ' + quotename('$database_src')+'.'+
quotename(s.name)+'.'+
quotename(t.name)
AS selectquery
,UPPER(s.name+'.'+t.name) as [dst_name]
from $database_src.sys.schemas s
inner join $database_src.sys.tables t on s.schema_id = t.schema_id"
# actually open the connection and execute $sql_src
$command_src = New-Object System.Data.SqlClient.SqlCommand $sql_src,$connection_src
$connection_src.Open()
$adapter_src = New-Object System.Data.SqlClient.SqlDataAdapter $command_src
$dataset_src = New-Object System.Data.DataSet
[void] $adapter_src.Fill($dataset_src)
$connection_src.Close()
#loop through source query's and insert data in $records
[array] $rows=($dataset_src.Tables | Select-Object -Expand Rows)
$i=0
foreach ($row in $rows)
{
$query = $rows[$i].selectquery
$dst_name = $rows[$i].dst_name
$command_src = New-Object System.Data.SqlClient.SqlCommand $query,$connection_src
$connection_src.Open()
$adapter_src = New-Object System.Data.SqlClient.SqlDataAdapter $command_src
$wanteddata_src = New-Object System.Data.DataSet
[void] $adapter_src.Fill($wanteddata_src)
$connection_src.Close()
# I think I only need to alter the 2 lines below to write the data to a table called $dst_name in the database $database_dst on the server $server_dstinstead of a file. Any existing table on the destination should be destroyed.
$outputfile = "C:\ExportedData\"+$def_filename+ ".csv"
$records= ($wanteddata_src.Tables | Select-Object -Expand Rows) | export-csv -Path $outputfile -Delimiter "|" -NoTypeInformation
$i++
}

Related

How to export the output of a SQL query to Excel using PowerShell?

I have a script which pulls data from a SQL database and gives the o/p in excel but the o/p comes under single column. I want to have it in different colums. This is my script:
$SQLServer = ""
$SQLDBName = ""
$SqlQuery = ""
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $SQLDBName; Integrated Security = True; User ID = $uid; Password = $pwd;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSet.Tables[0] | out-file "C:\Users\sql.csv"
I also tried this script:
$SQLServer = ""
$SQLDBName = ""
$SqlQuery = "select request_number,actioneer,audited_date from form_submission where actioneer ='XXXX' AND audited_date between '2020-05-19 00:00:00:000' and '2020-05-19 19:16:00:000';"| | Select #{n='request_number';e={request_number}}, #{n='actioneer';e={actioneer}} ,#{n ='audited_date';e={audited_date}}
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database =
$SQLDBName; Integrated Security = True; User ID = $uid; Password = $pwd;"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSet.Tables[0] | out-file "C:\Users\sql.csv"
But this leads to the following error:
Exception calling "Fill" with "1" argument(s): "Must declare the scalar variable "#"."
At C:\Users\sraghur\Documents\sql powershell.ps1:12 char:1
+ $SqlAdapter.Fill($DataSet)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SqlException
Is it really necessary to use .net methods instead of "Invoke-Sqlcmd"?
I usually use something like
Invoke-Sqlcmd -ServerInstance $strSQLServer -Database $strDatabase -QueryTimeout 180 -ConnectionTimeout 180 -query "select sqlcolumn1, sqlcolumn2, sqlcolumn3 from xxx" `
| Select-Object -Property sqlcolumn1, sqlcolumn2, sqlcolumn3 `
| Export-Csv "C:\Temp\SQLOutput.csv" -Encoding UTF8 -Delimiter ";" -NoTypeInformation
(You might have to install powershell module "SqlServer" beforehand)
On the other hand: Depending on the delimiter char used, the method of opening the file in Excel might be the problem: Restart Excel and choose "File > Open > ..." instead of double-clicking.
In case both doesn't work, you could also select - as LordPupazz recommended - the "single column" you were talking about and use "data > text to columns" (+ choose the correct delimiter).
Try this
Import-Module -Name SqlServer -Force
#Download ImportExcel Module if you don't have it already: https://www.powershellgallery.com/packages/ImportExcel/7.1.0
Import-Module -Name "C:\Program Files\WindowsPowerShell\Modules\ImportExcel\ImportExcel.psm1" -force
##########################################################################################################################################################################################
#Create Excel File in Temp
##########################################################################################################################################################################################
$Folder = "C:\Temp";
$ReportName = "ReportName";
#$FileNameCSV = $Folder + "\" + $ReportName + ".csv"
$FileNameExcel = $Folder + "\" + $ReportName +"_"+ $(get-date -f MM-dd-yyyy) + ".xlsx"
#$TestPath1 = Test-Path -Path $FileNameCSV;
$TestPath2 = Test-Path -Path $FileNameExcel;
#If($TestPath1 -eq $true){Remove-Item -Path $FileNameCSV -Force -ErrorAction Ignore};
If($TestPath2 -eq $true){Remove-Item -Path $FileNameExcel -Force -ErrorAction Ignore};
##########################################################################################################################################################################################
#Connect to SQL and load report table to PS variable
##########################################################################################################################################################################################
#Variables for your source SQL Server information
$SqlServer = "sqlserver.domain,port"
$SQLDB = "main"
$SQLQuery = "SELECT * FROM dbo.FOO"
#Wrap everything into a PS variable
$SQLProc = #{
ServerInstance = ""
Database = ""
Query = ""
}
#Set PS variable values
$SQLProc.ServerInstance = $SqlServer
$SQLProc.Database = $SQLDB
$SQLProc.Query = $SQLQuery
#Run SQL Query
$Results = Invoke-Sqlcmd #SQLProc
##########################################################################################################################################################################################
##Load data to Excel file on the specified Excel tab
##########################################################################################################################################################################################
#From the results dataset, select what columns you want in the Excel File
$Export = $Results | Select Column1,Column2,Column3
$Export | Export-Excel -Path $FileNameExcel -WorkSheetname "SheetName" -TableName "SQLData" -TableStyle Medium15 -AutoSize -BoldTopRow
Since you are using PowerShell already, why not to try dbatools.io and ImportExcel.
dbatools will help you to get data from the SQL with an easy way. Here is an example (for more check my series about dbatools:
$data = Invoke-DbaQuery -SqlInstance "localhost,1433" -Query "SELECT name,database_id from sys.databases"
The other will help you to get CSV exported to the XLSX.
$data | Export-Excel -Path HowToExcel.xlsx -Show
See how to use ImportExcel on my blog too

Change the default date format for query output

I'm running an enormous PowerShell script, which loops through every table and view in my database, and creates *.CSV text files for each table/view, containing the entire dataset.
(SELECT * FROM <table/view name>)
Any columns that are DATE or DATETIME are produced in the format MM/DD/YYYY
I need them to come out in the format YYYY/MM/DD
There are over 1,200 tables and views, so I can't manually specify the format for each affected column.
Can I change the default output format at the server level?
... or at the database level?
... or at the session level?
Here's my PowerShell code:
$server = "myserver"
$database = "mydb"
$tablequery = "SELECT s.name AS schema_name, t.name AS table_name FROM sys.tables t LEFT JOIN sys.schemas s ON s.schema_id = t.schema_id UNION ALL SELECT s.name, v.name FROM sys.views v LEFT JOIN sys.schemas s ON s.schema_id = v.schema_id"
#Delcare Connection Variables
$connectionTemplate = "Data Source={0};Integrated Security=SSPI;Initial Catalog={1};"
$connectionString = [string]::Format($connectionTemplate, $server, $database)
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$command = New-Object System.Data.SqlClient.SqlCommand
$command.CommandText = $tablequery
$command.Connection = $connection
#Load up the Tables in a dataset
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$SqlAdapter.SelectCommand.CommandTimeout=600
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
# Loop through all tables and export a CSV of the Table Data
foreach ($Row in $DataSet.Tables[0].Rows)
{
$queryData = "SELECT * FROM [$($Row[0])].[$($Row[1])]"
#Specify the output location of your dump file
$extractFile = "C:\mydb\$($Row[0])_$($Row[1]).csv"
$command.CommandText = $queryData
$command.Connection = $connection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
$DataSet.Tables[0] | Export-Csv $extractFile -NoTypeInformation
}
What if you add this line:
[System.Threading.Thread]::CurrentThread.CurrentCulture=[System.Globalization.CultureInfo]"fr-CA"
in this way:
...
...
# Loop through all tables and export a CSV of the Table Data
[System.Threading.Thread]::CurrentThread.CurrentCulture=[System.Globalization.CultureInfo]"fr-CA"
foreach ($Row in $DataSet.Tables[0].Rows)
{
$queryData = "SELECT * FROM [$($Row[0])].[$($Row[1])]"
#Specify the output location of your dump file
$extractFile = "C:\mydb\$($Row[0])_$($Row[1]).csv"
$command.CommandText = $queryData
$command.Connection = $connection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $command
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$connection.Close()
$DataSet.Tables[0] | Export-Csv $extractFile -NoTypeInformation
}

Copy multiple tables result to Excel

I have a massive SQL query that returns 4 tables. I'm able to return the first table but not the other tables here is the code I'm using below. I'm using a powershell script that I found online. I was about to get the second table by changing $DataSetTable = $DataSet.Tables[0] to $DataSetTable = $DataSet.Tables[1] the second table was return in the excel sheet but not the first one.
How can I all the tables?
$SQLServer = 'SQLCL01'
$Database = 'home'
$SqlQuery = #' Query goes here'#
## - Connect to SQL Server using non-SMO class 'System.Data':
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server = $SQLServer; Database = $Database; Integrated Security = True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
## - Extract and build the SQL data object '$DataSetTable':
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$SqlAdapter.Fill($DataSet)
$DataSetTable = $DataSet.Tables[0]
## --------- Working with Excel ------- ##
## - Create an Excel Application instance:
$xlsObj = New-Object -ComObject Excel.Application
## - Create new Workbook and Sheet (Visible = 1 / 0 not visible)
$xlsObj.Visible = 1
$xlsWb = $xlsobj.Workbooks.Add()
$xlsSh = $xlsWb.Worksheets.item(1)
$xlsSh = $xlsWb.Worksheets.Item("Sheet1")
$xlsSh.Name = "Back date dump"
$xlsSh = $xlsWb.ActiveSheet
## - Build the Excel column heading:
[Array] $getColumnNames = $DataSetTable.Columns | Select ColumnName
## - Build column header:
[int] $RowHeader = 1
foreach ($ColH in $getColumnNames) {
$xlsSh.Cells.item(1, $RowHeader).font.bold = $true
$xlsSh.Cells.item(1, $RowHeader) = $ColH.ColumnName
$RowHeader++
}
## - Adding the data start in row 2 column 1:
[int] $rowData = 2
[int] $colData = 1
foreach ($rec in $DataSetTable.Rows) {
foreach ($Coln in $getColumnNames) {
## - Next line convert cell to be text only:
$xlsSh.Cells.NumberFormat = "#"
## - Populating columns:
$xlsSh.Cells.Item($rowData, $colData) = $rec.$($Coln.ColumnName).ToString()
$ColData++
}
$rowData++; $ColData = 1
}
## - Adjusting columns in the Excel sheet:
$xlsRng = $xlsSH.usedRange
$xlsRng.EntireColumn.AutoFit()
Loop over all tables in the collection:
$xlsObj = New-Object -ComObject Excel.Application
foreach ($DataSetTable in $DataSet.Tables) {
...
}

Powershell update SQL Server column

Hi I am trying to update a column though doesnt seem to work. It extracts the data and does the date conversion in the string but the update section of my code below does not want to insert the changes Powershell makes.
## connect to db and do stuff
$Connection = new-object system.data.SqlClient.SQLConnection("Data Source= (local);Integrated Security=SSPI;Initial Catalog=TESTDB");
$sqlqry = "select row_id, DestFileName from FL_Input Where DestFileName like '%[0-9] ' + 'Jan%' + ' [0-9]%'"
$Command = new-object system.data.sqlclient.sqlcommand
$Command.CommandText = $sqlqry
$Command.Connection = $Connection
$Connection.Open()
$UpdateCmd = new-object System.Data.SqlClient.SqlCommand
$UpdateCmd.Connection = $Connection
## for each filename, update timestamp - this replaces your $file in $files loop.
$Result = $Command.ExecuteReader()
while ($Result.Read()) {
$destfile = $Result['DestFileName']
$srcfile = $destfile
$row_id = $Result['row_id']
## do all your regex stuff here
$destfile -match '\d{2}\s\w+\s\d{2,4}' | Out-Null <# Test #>
$destfile -match '\d{2}\-\w+\-\d{4}' | Out-Null <# Test #>
$destFile -replace "$(($matches).values)" , "$(get-date "$(($matches).Values)" -Format yyyyMMdd)"
write-host "{0}->{1}" -f $destfile, $srcfile
# when you're finished doing your regex, push the result result back to the db:
$updateqry = "update FL_Input set DestFileName='{0}' WHERE Row_ID = {1} " -f $destfile, $row_id
$UpdateCmd.CommandText = $updateqry
$UpdateCmd.ExecuteNonQuery()
}
## all done
$Connection.Close
Aren't you missing single quotes around the row_id e.g.:
$updateqry = "update FL_Input set DestFileName='$destfile' WHERE Row_ID = '$row_id'"

Extract Millisecond data from SQL stored procedure

I have looked for answers on this subject and I have posted in another forum but this seems to be the font of all knowledge. I am trying to pull data from a SQL Server 2000 database using PowerShell. The Powershell script calls a stored procedure, this then extracts the data and export-csv outs it to, well a CSV. The problem is that the datetime fields in the outputted data have lost their milliseconds. This happens when PowerShell extracts the data from the database into a temporary table. Here are the two bits of code.
PowerShell script
#VARIABLES
$SqlQuery = "SAP_proc"
#Connection Strings
$Server = "server"
$Database = "db_dab"
#Output Files
$OutputPath = "c:\Sapout.csv"
#END OF VARIABLES
#SQL Connection
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$Server;Database=$Database;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = "SAP_proc"
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$DataOut = $SqlAdapter.Fill($DataSet)
#$DataOut = $SqlAdapter.FillSchema($DataSet,[System.Data.SchemaType]::Source)
#Data Manipulation
$DataOut | Out-Null
#Export Data to Hash Table
$DataTable = $DataSet.Tables[0]
#Export Has table to CSV
$DataTable | Export-CSV -Delimiter "," -Encoding Unicode -Path $OutputPath -NoTypeInformation
$SqlConnection.Close()
Stored procedure
ALTER proc [dbo].[SAP_proc]
as
DECLARE #return_value int
DECLARE #starttime datetime
DECLARE #endtime datetime
SET #starttime = DATEADD (dd, -30, CURRENT_TIMESTAMP)
SET #endtime = CURRENT_TIMESTAMP
EXEC #return_value = [dbo].[sp_agent]
#starttime
,#endtime
SELECT 'Return Value' = #return_value
Because the other stored procedure SP_agent is created by the software I can't edit it. Also I don't want to replicate the software defined stored procedure (with SELECT convert to varchar for datetime) in my command text string as it is a behemoth stored procedure.
Any help would be massively useful.
It's not a Powershell issue or a temp table issue. This is because your datetime column is converted to a string when you call export-csv using the default tostring method which doesn't include milliseconds. If you want milliseconds then specify it in the tostring method call:
$a = get-date
$a.ToString("d/M/yyyy hh:mm:ss.fff tt")
#To to something similar with your export-csv of a datatable you can create an expression:
$DataTable | select column1, column2, #{n=datecolumn;e={$_.datecolumn.ToString("d/M/yyyy hh:mm:ss.fff tt")}} | export-csv <rest of code>
Edited 10/17/2012
It seems like you are having trouble with this still. So here's a complete script which I've tested outputs milliseconds. You'll need to change the variable section to your environment. I hope this helps:
#VARIABLES
$SqlQuery = "select 'test' as column1, 'milliseconds' as column2, getdate() as datecolumn"
#Connection Strings
$Server = "$env:computername\sql1"
$Database = "tempdb"
#Output Files
$OutputPath = "./millisec.csv"
#END OF VARIABLES
#SQL Connection
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$Server;Database=$Database;Integrated Security=True"
$SqlCmd = New-Object System.Data.SqlClient.SqlCommand
$SqlCmd.CommandText = $SqlQuery
$SqlCmd.Connection = $SqlConnection
$SqlAdapter = New-Object System.Data.SqlClient.SqlDataAdapter
$SqlAdapter.SelectCommand = $SqlCmd
$DataSet = New-Object System.Data.DataSet
$DataOut = $SqlAdapter.Fill($DataSet)
#$DataOut = $SqlAdapter.FillSchema($DataSet,[System.Data.SchemaType]::Source)
#Data Manipulation
$DataOut | Out-Null
#Export Data to Hash Table
$DataTable = $DataSet.Tables[0]
#Export Has table to CSV
#$DataTable | Export-CSV -Delimiter "," -Encoding Unicode -Path $OutputPath -NoTypeInformation
$DataTable | select column1, column2, #{n='datecolumn';e={$_.datecolumn.ToString("M/d/yyyy hh:mm:ss.fff tt")}} | export-csv $OutputPath -NoTypeInformation -force
$SqlConnection.Close()
#Another code example with explanation. Added on 10/23/2012
#VARIABLES
$SqlQuery = "select getdate() as datecolumn"
#Connection Strings
$Server = "$env:computername\sql1"
$Database = "tempdb"
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$Server;Database=$Database;Integrated Security=True"
$SqlConnection.Open()
$SqlCmd = new-Object System.Data.SqlClient.SqlCommand($SqlQuery, $SqlConnection)
$data = $SqlCmd.ExecuteScalar()
$SqlConnection.Close()
#Notice NO millisecond
Write-Output $data
#See $data is an object of type System.DateTime
$data | gm
#There's a property called millisecond
#Millisecond Property int Millisecond {get;}
#Although you don't "see" millisecond's on screen it's still there
$data.Millisecond
#Powershell uses a types and format rules to define how data is display on screen. You can see this by looking at
#C:\Windows\System32\WindowsPowerShell\v1.0\types.ps1xml and searching for "DataTime". This file along with format file define how data is displayed
#When you display datatime screen it's implicitly calling
#Searching for datetime in types.ps1xml you'll find this line:
#"{0} {1}" -f $this.ToLongDateString(), $this.ToLongTimeString()
#for our example
"{0} {1}" -f $data.ToLongDateString(), $data.ToLongTimeString()
#So millisecond is still there, but if you want millisecond in your output to CSV, screen or file you'll need to call a ToString method with a date format. Here's an example:
$data.ToString("M/d/yyyy hh:mm:ss.fff tt")