Doing this query in sql server - it returns 3 rows of data. Running the script with the write-host $1_resultsDataTable and comment out the other variable $2_resultsDataTable- it returns only one row of the data array. Now if I reverse the comments so the $2_resultsDataTable is active for the write-host, it returns 6 rows of data.
How do I set this up so I would see the same 3 rows assigned to both $1_resultsDataTable and $2_resultsDataTable when I dump these variables to view the data results?
[string] $Server= "SERVER"
[string] $Database = "mvTest"
[string] $UserSqlQuery= $("select m.created_date, m.additional_data as ReasonDown from aeroscout.mv_audit m where m.created_date >= '2020-01-18' and m.additional_data like '%query-text%'")
#
$1_resultsDataTable, $2_resultsDataTable = foreach ($x in 1..2) {
$resultsDataTable = New-Object System.Data.DataTable
$resultsDataTable = ExecuteSqlQuery $Server $Database $UserSqlQuery
$resultsDataTable # first loop sends output to $1_resultsDataTable, second loop send to $2_resultsDataTable
Start-Sleep 3
}
# executes a query and populates the $datatable with the data
function ExecuteSqlQuery ($Server, $Database, $SQLQuery) {
$Datatable = New-Object System.Data.DataTable
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';Integrated Security=True;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
$Command.CommandText = $SQLQuery
$Reader = $Command.ExecuteReader()
If ($Reader.HasRows) {
while($Reader.Read()) {
$props = #{}
for($i = 0; $i -lt $Reader.FieldCount; $i+=1) {
$name = $Reader.GetName($i)
$value = $Reader.item($i)
$props.Add($name, $value)
}
$obj = new-object PSObject -Property $props
Write-Output $obj
}
}
return $obj
$SqlConnection.Close()
}
#validate we got data
write-host $1_resultsDataTable
Start-Sleep 3
write-host $2_resultsDataTable
[string] $Server= 'NERD\PAULSDB'
[string] $Database = "myPhotos"
$ShootDate= Get-Content -Path W:\W-SQL\PS\ShootDate.txt
ForEach ($S_Date in $ShootDate){
$SqlQuery=#"
SELECT * FROM Target.U_PhotoYears where StrmydateTaken= '$S_Date'
"#
$resultsDataTable = ExecuteSqlQuery $Server $Database $SqlQuery
function ExecuteSqlQuery ($Server, $Database, $SQLQuery) {
$Datatable = New-Object System.Data.DataTable
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';Integrated Security=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
$Command.CommandText = $SQLQuery
$Reader = $Command.ExecuteReader()
$Datatable.Load($Reader)
$Connection.Close()
return $Datatable
}
$data=$(foreach($a in $resultsDataTable)
{
$a.P_Filename+$a.myYearTaken
}
)
$data
}
Text file has a distinct list of dates from all photo shoots. The first ForEach appears to be retrieving data ok. Then the Powershell script queries the myPhotos db based on values from text file. The query return the correct records for a given date.
However I cant get the records out of the second foreach to Export-Csv.
Any suggestions would be really appreciated
I think you shoud use bulkcopy for send your datas into temporary table. its really better for performance (you do only 3 actions on database and not X actions => X dates into you files)
try this (not tested)
$Connection=$null
$dttemporary=$null
$sqlBulkCopy=$null
$FinalData=$null
$Command=$null
$Reader=$null
$ErrorActionPreference = "Stop" #for stop script on first error
try
{
#create datatable for send data
$dttemporary = New-Object System.Data.Datatable
[void]$dttemporary.Columns.Add("mydate")
#add data to datatable
Get-Content -Path W:\W-SQL\PS\ShootDate.txt | %{
$NewRow=$dttemporary.NewRow()
$NewRow["mydate"]=$_
$dttemporary.Rows.Add($NewRow)
}
#Create connexion
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';Integrated Security=true;"
#create bulkcopy structure for send all data
$sqlBulkCopy = New-Object (“Data.SqlClient.SqlBulkCopy”) -ArgumentList $Connection
$sqlBulkCopy.DestinationTableName = “#TmpTable”
#Create datatable for final data
$FinalData = New-Object System.Data.DataTable
#create command
$Command = $Connection.CreateCommand()
$Connection.Open()
#create temporary table (exsit while connexion stay open)
$Command.CommandText = "CREATE TABLE #TmpTable(mydate varchar(30) NOT NULL)"
$Command.ExecuteNonQuery()
#send data to temporary command with bulkcopy
$sqlBulkCopy.WriteToServer($dttemporary)
#query for get your final datas
$Command.CommandText = "SELECT * FROM Target.U_PhotoYears f1 inner join #TmpTable f2 on f1.StrmydateTaken=f2.mydate"
$Reader = $Command.ExecuteReader()
$FinalData.Load($Reader)
$Connection.Close()
#loop on data for traitment
$FinalData | %{
$_["P_Filename"].ToString() + $_["StrmydateTaken"].ToString()
}
}
catch
{
$_.Exception.Message
}
finally
{
if ($Connection)
{
$Connection.Close()
$Connection.Dispose()
}
if ($dttemporary) {$dttemporary.Dispose()}
if ($sqlBulkCopy) {$sqlBulkCopy.Dispose()}
if ($FinalData) {$FinalData.Dispose()}
if ($Command) {$Command.Dispose()}
if ($Reader) {$Reader.Dispose()}
}
I have multiple text files with need to be imported in SQL Server management studio. I have written a code in powershell to import certain files. But I have a file which ends with a row contains only '---------------------------------------'
For example
test|test2|test3
A|B|C
Q|W|E
'-----------'
is there a way to ignore or skip this row. I already tried select-object -skiplast 1
Function AutoImportCommaFlatFiles($location, $file, $extension, $server, $database)
{
$full = $location + $file + $extension
## $columns = Get-Content $full | Select-Object -skip 1 | set-Content $full
$all = Get-Content $full | select-Object -skip 3
$columns = $all[0]
$columns = $columns.Replace(" ","")
$columns = $columns.Replace("||","Column Emtpy|Column Empty 2|")
$columns = $columns.Replace("","Column Emtpy3")
$columns = $columns.TrimEnd('|')
$columns = $columns.Replace("|","] VARCHAR(100), [")
$table = "CREATE TABLE " + $file + "([" + $columns + "] VARCHAR(100))"
$connection = New-Object System.Data.SqlClient.SqlConnection
$buildTable = New-Object System.Data.SqlClient.SqlCommand
$insertData = New-Object System.Data.SqlClient.SqlCommand
$connection.ConnectionString = "Data Source=" + $server + ";Database=" + $database + ";integrated security=true"
$buildTable.CommandText = $table
$buildTable.Connection = $connection
## Added to function;
$x = 0
$insertData.CommandText = "EXECUTE stp_CommaBulkInsert #1,#2"
$insertData.Parameters.Add("#1", $full)
$insertData.Parameters.Add("#2", $file)
$insertData.Connection = $connection
$connection.Open()
$buildTable.ExecuteNonQuery()
$connection.Close()
## Added to function
$x = 1
if ($x = 1)
{
$connection.Open()
$insertData.ExecuteNonQuery()
$connection.Close()
}
}
and like this :
(Get-Content $full | Select-Object -skiplast 1) | set-Content $full
$all = Get-Content $full
Why you dont use import-csv command, Something like this :
$all=import-csv c:\temp\result.csv -Delimiter '|' -Header col1, col2, col3 | where col2 -ne $null
I have a powershell script that writes every file and its attributes recursively starting from a specific directory. This works but the directories could have as many as 1,000,000 files. What I want to do is batch them at 1000 inserts per transaction. Here is the original PS:
$server = ""
$Database = ""
$Path = "C:\Test"
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
foreach($file in Get-ChildItem -Verbose -Recurse -Path $Path | Select-Object Name,Length,Mode, Directory,CreationTime, LastAccessTime, LastWriteTime) {
$fileName = $file.Name
$fileSize = ([int]$file.Length)
$fileMode = $file.Mode
$fileDirectory = $file.Directory
$fileCreationTime = [datetime]$file.CreationTime
$fileLastAccessTime = [datetime]$file.LastAccessTime
$fileLastWriteTime = [datetime]$file.LastWriteTime
$sql = "
begin
insert TestPowerShell
select '$fileName', '$fileSize', '$fileMode', '$fileDirectory', '$fileCreationTime', '$fileLastAccessTime', '$fileLastWriteTime'
end
"
$Command.CommandText = $sql
echo $sql
$Command.ExecuteNonQuery()
}
$Connection.Close()
My thoughts are to implement some sort of counter that will keep appending the insert until it reaches 1000 and then jump out of the loop and execute. I cannot figure out with this current setup how to batch at 1000, execute and then pick back up with the get-childitem loop.
Something like this should do:
function Execute-SqlQuery($query){
Write-Host "Executing query:"
Write-Host $query;
}
$data = #(1,2,3,4,5,6,7,8,9,10,11);
$batchSize = 2;
$counter = 0;
$sql = "";
foreach($item in $data){
if($counter -eq $batchSize){
Execute-SqlQuery $sql;
$counter = 0;
$sql = "";
}
$sql += "insert into myTable(id) values($item) `n";
$counter += 1;
}
Execute-SqlQuery $sql;
$server = ""
$Database = ""
$Path = "C:\Test"
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
# new variables to handle batching
$batchcounter=0
$batchsize=1000
$sqlValues = New-Object Collections.ArrayList
foreach($file in Get-ChildItem -Verbose -Recurse -Path $Path | Select-Object Name,Length,Mode, Directory,CreationTime, LastAccessTime, LastWriteTime) {
$fileName = $file.Name
$fileSize = ([int]$file.Length)
$fileMode = $file.Mode
$fileDirectory = $file.Directory
$fileCreationTime = [datetime]$file.CreationTime
$fileLastAccessTime = [datetime]$file.LastAccessTime
$fileLastWriteTime = [datetime]$file.LastWriteTime
$sqlValues.Add("('$fileName', '$fileSize', '$fileMode', '$fileDirectory', '$fileCreationTime', '$fileLastAccessTime', '$fileLastWriteTime')")
$batchcounter++
# if the counter hits batchsize, run the insert, using lots of:
# insert into table
# values (1,2,3)
# , (4,5,6)
# , (7,8,9)
if ($batchcounter % $batchsize -eq 0) {
$sql = "insert TestPowerShell values {0}" -f ($sqlValues.ToArray() -join "`r`n,")
$Command.CommandText = $sql
Write-Host $sql
$Command.ExecuteNonQuery()
$sqlValues.Clear()
}
}
# catch any remaining files
if ($batchcounter -gt 0) {
$sql = "insert TestPowerShell values {0}" -f ($sqlValues.ToArray() -join "`r`n,")
$Command.CommandText = $sql
Write-Host $sql
$Command.ExecuteNonQuery()
$sqlValues.Clear()
}
$Connection.Close()
For anyone interested - this is one way to do it:
function WriteBatch {
echo $sql
$Command.CommandText = $sql
$Command.ExecuteNonQuery()
}
$server = ""
$Database = ""
$Path = ""
$Counter = 0
$Connection = New-Object System.Data.SQLClient.SQLConnection
$Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
$Connection.Open()
$Command = New-Object System.Data.SQLClient.SQLCommand
$Command.Connection = $Connection
[string]$sql = "
begin
insert into TestPowerShell(NameString, FileSize, Mode, Directory, CreationTime, LastAccessTime, LastWriteTime)
values "
foreach($file in Get-ChildItem -Verbose -Recurse -Path $Path | Select-Object Name, Length, Mode, Directory, CreationTime, LastAccessTime, LastWriteTime) {
$fileName = $file.Name
$fileSize = ([int]$file.Length)
$fileMode = $file.Mode
$fileDirectory = $file.Directory
$fileCreationTime = [datetime]$file.CreationTime
$fileLastAccessTime = [datetime]$file.LastAccessTime
$fileLastWriteTime = [datetime]$file.LastWriteTime
$sql = $sql + "('$fileName', '$fileSize', '$fileMode', '$fileDirectory', '$fileCreationTime', '$fileLastAccessTime', '$fileLastWriteTime'),"
$sql += "`n"
$Counter++
If($Counter -eq 900) {
$sql = $sql.Trim().Trim(',')
$sql = $sql + " End"
WriteBatch
$Counter = 0
$sql = "
begin
insert into TestPowerShell(NameString, FileSize, Mode, Directory, CreationTime, LastAccessTime, LastWriteTime)
values "
}
}
if ($Counter -gt 0){
$sql = $sql.Trim().Trim(',')
$sql = $sql + " End"
WriteBatch
}
$Connection.Close()
I have a small powershell script that is meant to get column ServerName from a remote SQL database called, Hal0Test > from table, ServerList. However, I can not figure out this Powershell Error.
Newest Code:
Write-Output " `n Start of Hal0 `n";
$connectionString = "Server=QAUTILITYDB01;Database=Hal0Test;Integrated Security=True;"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$command = $connection.CreateCommand()
$ServerArray = [System.Collections.ArrayList]#()
$query = "SELECT ServerName FROM ServerList"
$command.CommandText = $query
$ServerNames = $command.ExecuteReader()
$table = new-object “System.Data.DataTable”
$table.Load($ServerNames)
$ServerArray = $table | select -Expand ServerName
$ServerArray | ForEach-Object {
# $Server returns each server name
$os = Get-WmiObject -Class Win32_OperatingSystem -Computer $_
$disks = Get-WmiObject -Class Win32_LogicalDisk -Computer $_ |
Where-Object {$_.DriveType -eq 3} |
ForEach-Object {
'{0} {1:D} MB Free/{2:D} MB Used' -f $_.DeviceID,
[int]($_.FreeSpace/1MB), [int]($_.Size/1MB)
}
New-Object -Type PSCustomObject -Property #{
'FQDN' = $_
'ServerName' = $os.PSComputerName
'OperatingSystem' = $os.Caption
'Disks' = $disks -join ' | '
}
$command.CommandText = "UPDATE ServerList SET FQDN = '$_', OS = '$os.Caption' WHERE ServerName = '$os.PSComputerName';"
$result = $command.ExecuteNonQuery()
} | Export-Csv 'C:\Users\king\Desktop\HalO\output.csv' -Delimiter '|' -NoType
Write-Output "`n End of Hal0";
SQL Table:
You changed my ForEach-Object loop to a foreach loop. If you want to use the latter you need to change the current object variable $_ to your loop variable $Server:
foreach ($Server in $ServerArray) {
$os = Get-WmiObject -Class Win32_OperatingSystem -Computer $Server
$disks = Get-WmiObject -Class Win32_LogicalDisk -Computer $Server | ...
...
}
otherwise you need to change the loop back to a ForEach-Object loop:
$ServerArray | ForEach-Object {
$os = Get-WmiObject -Class Win32_OperatingSystem -Computer $_
$disks = Get-WmiObject -Class Win32_LogicalDisk -Computer $_ | ...
...
}
Also, there's no pipe between } and Export-Csv:
$result = $command.ExecuteNonQuery()
} Export-Csv 'C:\Users\mdaraghmeh\Desktop\HalO\output.csv' -Delimiter '|' -NoType
^
here
And even if there were would the foreach loop still be unable to feed its output into the pipeline. If you want to use foreach with a pipeline you must assign the output to a variable:
$output = foreach ($Server in $ServerArray) { ... }
$output | Export-Csv ...
or run it in an expression:
(foreach ($Server in $ServerArray) { ... }) | Export-Csv ...
For direct pipeline processing you need a ForEach-Object loop:
$ServerArray | ForEach-Object { ... } | Export-Csv ...
You need to use a [datatable] to store the result of your select:
$connectionString = "Server=QAUTILITYDB01;Database=Hal0Test;Integrated Security=True;"
$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = $connectionString
$connection.Open()
$command = $connection.CreateCommand()
$ServerArray = [System.Collections.ArrayList]#()
$query = "SELECT ServerName FROM ServerList"
$command.CommandText = $query
$ServerNames = $command.ExecuteReader()
$table = new-object "System.Data.DataTable"
$table.Load($ServerNames)
Now $table has your servers name list.