Conflict with Get-Item - sql

I have a line of code that has some conflict in it and I just cannot see what it is.
Here is the program
$Date = Get-Date -Format MM-dd-yyyy-HH-mm-ss-tt
<#*****SQL Session Variables*****#>
$Username = "sa"
$Password = "xyz" | ConvertTo-SecureString -asPlainText -Force
$Credentials = New-Object System.Management.Automation.PSCredential($Username,$Password)
$DatabaseNames = Invoke-SQLcmd -Query "Select * From sys.databases WHERE Name NOT LIKE '%Master%' AND Name NOT LIKE '%Tempdb%' AND Name NOT LIKE '%Model%' AND Name NOT LIKE '%Msdb%';" -ServerInstance "localhost" | Select Name -ExpandProperty Name
Write-Host "****Database Names****" -ForeGroundColor Red -BackGroundColor Black
$DatabaseNames
"`n`n"
<#****Get Database Size****#>
Push-Location
Import-Module sqlps -disablenamechecking
Invoke-Sqlcmd -Query "select name, physical_name, size * 8.0 / 1024 size from sys.master_files WHERE Type_Desc NOT LIKE '%LOG%' AND Name NOT LIKE '%Master%' AND Name NOT LIKE '%Tempdb%' AND Name NOT LIKE '%Model%' AND Name NOT LIKE '%Msdb%';" -ServerInstance "localhost" | Select #{Label="Database Name";Expression={$_.name}}, #{Label="Location";Expression={$_.physical_name}}, #{Label="SIze (MB)";Expression={$_.size}}
Pop-Location
"`n"
Start-Sleep 4
<#****Backup Databases****#>
ForEach ($DatabaseName in $DatabaseNames)
{
$SQLBackupPath1 = "C:\T2\SQlBackup\"
$SQLBackupPath2 = "$DatabaseName"
$SQLBackupPath3 = "_"
$SQLBackupPath4 = $Date
$SQLBackupPath5 = "_.bak"
$SQLBackupPath = $SQLBackupPath1 + $SQLBackupPath2 + $SQLBackupPath3 + $SQLBackupPath4 + $SQLBackupPath5
$SQLBackupPath
Backup-SQLDatabase -ServerInstance localhost -Database $DatabaseName -BackupFile $SQLBackupPath -Credential $Credentials
Write-Host "Database Backup of " -NoNewLine
Write-Host "$DatabaseName" -ForeGroundColor Red -BackGroundColor Black -NoNewLine
Write-Host " has been completed"
"`n"
}
<#****Delete Databases****#>
ForEach ($DatabaseName in $DatabaseNames)
{
Push-Location
Import-Module sqlps -disablenamechecking
Invoke-Sqlcmd -Query "EXEC msdb.dbo.sp_delete_database_backuphistory #database_name = N'$DatabaseName';" -ServerInstance "localhost"
Invoke-Sqlcmd -Query "USE [master];" -ServerInstance "localhost"
Invoke-Sqlcmd -Query "ALTER DATABASE [$DatabaseName] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;" -ServerInstance "localhost"
Invoke-Sqlcmd -Query "DROP DATABASE [$DatabaseName];" -ServerInstance "localhost"
Pop-Location
Write-Host "Database: " -NoNewLine
Write-Host "$DatabaseName" -ForeGroundColor Red -BackGroundColor Black -NoNewLine
Write-Host " has been Dropped"
"`n"
}
Get-ChildItem -Path C:\T2\SQLBackup\ | Where-Object {$_.Name -match "ApOps*" } | Sort-Object LastWriteTime -Descending
If I move the final GetChildItem line under the first line $Date I get output. However if I use this in a loop or where it currently sits no results are returned. This makes me believe that there is some conflicting statement in the code. If so I cannot seem to find it and I am losing my mind looking for it. Please help.

You're being bitten by the output formatting, and being a bit careless with your output. Consider this, I'm in a directory with a couple of files:
List the files, simple and works fine:
PS D:\test> Get-ChildItem # list the files
Directory: D:\test
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 02/11/2016 07:01 1640 log.txt
-a---- 02/11/2016 07:01 0 testfile
Output an empty string, then list the files, simple and works fine. One blank line extra at the start:
PS D:\test> ""; Get-ChildItem
Directory: D:\test
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 02/11/2016 07:01 1640 log.txt
-a---- 02/11/2016 07:01 0 testfile
Get the string length, then list the files. Wait what? The output formatter is taking property you asked for from the first output it receives (string Length) and is then applying that to everything it receives (files) until the pipeline is empty:
PS D:\test> "" | select Length; Get-ChildItem # select string length, then...
Length # missing columns ??!!
------
0 # string length
1640 # file length
0 # file length
Get some property which doesn't exist, then list the files. This is the same idea, only this time it's not showing anything for the string or anything for the files, because Foobar doesn't exist on either type:
PS D:\test> "" | select Foobar; Get-ChildItem # select property which doesn't exist
Foobar
------
# ???? this is your "no output"
You are doing this kind of thing - sending mixed types to the output pipeline. They still go to the output, if you captured them in variables or wrote them to files the data would be there, but the output formatter which tries to work out how to display them on the console takes the first thing it gets and formats the output against that, assuming you'll have a consistent output. And the first complex objects it gets come from your code here:
Invoke-Sqlcmd -Query "select name, physical_name, size * 8.0 / 1024 size from sys.master_files WHERE Type_Desc NOT LIKE '%LOG%' AND Name NOT LIKE '%Master%' AND Name NOT LIKE '%Tempdb%' AND Name NOT LIKE '%Model%' AND Name NOT LIKE '%Msdb%';" -ServerInstance "localhost" | Select #{Label="Database Name";Expression={$_.name}}, #{Label="Location";Expression={$_.physical_name}}, #{Label="SIze (MB)";Expression={$_.size}}
# where you select these properties for display:
# Database Name, Location, Size (MB)
# None of these properties exist on files or directories,
# so they don't get shown on screen.
Fix:
$x = Invoke-SqlCmd ___
$y = Get-ChildItem
$x | Format-Table
$y | Format-Table
Or use more Write-Host, or output to files.

Related

powershell not exporting

hi i am running the following query in powershell:
Import-Module Hall.psm1
$Database = 'Report'
$Server = '192.168.1.2'
$Query = 'SELECT all * FROM [Report].[dbo].[TestView]'
$LogLocation = "\\Report\LogFile.csv"
$DynamicYear = (Get-Date).Year
$DynamicMonth = (Get-Culture).DateTimeFormat.GetMonthName((Get-Date).Month)
$FileDestination = "\\Report\MONTHLY REPORTS\"+$DynamicYear+"\"+$DynamicMonth+"\"
$Outputfilename='TestView-'+(Get-Date).ToString('MM-dd-yyyy')+'.csv'
$LocalCreate = 'C:\Scripts\LocalCreate\'
$FolderPathExtension = "Microsoft.PowerShell.Core\FileSystem::"
$CodeDestination = $FolderPathExtension+$FileDestination
$filedest=$LocalCreate+$outputfilename
$Logfile = $FolderPathExtension+$LogLocation
Invoke-sqlcmd -querytimeout 120 -query "
$Query
" -database $database -serverinstance $server |
ConvertTo-Csv -NoTypeInformation | # Convert to CSV string data without the type metadata
Select-Object -Skip 0 | # Trim header row, leaving only data columns
% {$_ -replace '"',''} | # Remove all quote marks
Set-Content -Path $filedest
(gc $filedest) | ? {$_.trim() -ne "" } | set-content $filedest
if(Test-Path ($filedest)) {
Move-Item -Path $filedest -Destination $CodeDestination -Force
$LogType = 'INFO'
$LogEntry = "$filedest MovedTo $To"
Write-Log -Message $LogEntry -Level $LogType -Logfile $Logfile
}
which works fine without any issue if the query has data.
however, if the query does not have any data it does not create a .csv. how can i get it to create a blank .csv? or .csv with headers only?
Use New-Item -ItemType File -Path $filedest before your Invoke-SqlCmd Or ConvertTo-Csv

Read the sql files by Exclude the list of files from sql table and Execute all the files in a single Transaction using PowerShell

I have not an Expert in power shell. I Need a script/Approch which handles the below requirement.
I have list of files in a folder and the file Names like below.
001_File.sql
002_File.sql
003_File.sql
004_File.sql
Also, I have a table in sql server which holds the file Name Information.
TableName: Executedfile with a column FileName.
002_File.sql
004_File.sql
My requirement is to read the files which is available in the folder but not in the table.
I have to the Read files only:
001_File.sql
003_File.sql
Now, I Need to Execute these two file in the sequential order under the same Transaction on SQL Server. As I Need to rollback all the transaction if any Error occurred.
As of now I wrote a power shell something below.
$QueryResult = Invoke-Sqlcmd -ServerInstance 'MyServer' -Database 'MyDb' -Query "SELECT DISTINCT FNames from TableName"
Get-ChildItem "E:\Testing\" -Filter *.sql | Sort-Object $_.Name|
Foreach-Object {
$FileFullpath= $_.FullName
Write-Host $FileFullpath
$FileName = $_.Name
Write-Host $FileName
if(!$QueryResult.FName.Contains($FileName))
{
invoke-sqlcmd -inputfile $FileFullpath -serverinstance "servername\serverinstance" -database "mydatabase"
}
}
Please suggest me some script.
Challenges:
How to read the files in Sequential order as it has leading Zeros. Does the Above `Sort-Object $_.Name ' will Sort ?
How to Execute all the list of files under one transaction.
Thanks
Finally I did something like this.
$QueryResult = Invoke-Sqlcmd -ServerInstance 'MyServer' -Database 'MyDb' -Query "SELECT DISTINCT FNames from TableName"
$FullScript = #()
$FullScript += "BEGIN TRANSACTION;"
Get-ChildItem "E:\Testing\" -Filter *.sql | Sort-Object $_.Name|
Foreach-Object {
if(!$QueryResult.FName.Contains($_.Name))
{
$FullScript += Get-Content $_.FullName
}
}
$FullScript += "COMMIT TRANSACTION;"
sqlcmd -S localhost -d test -Q "$FullScript"
Try this...
#get list of filenames from database...
$QueryResult = Invoke-Sqlcmd -ServerInstance 'MyServer' -Database 'MyDB' -Query "SELECT DISTINCT FNames from TableName" | Select-Object -ExpandProperty FileName
#get files from folder whose names are not in $queryresult...
$files = Get-ChildItem -Path E:\Testing -Filter *.sql | ? {(!($QueryResult.Contains($_.BaseName)))} | Sort-Object Name
#get the content of each $file and replace "GO" with empty string, etc...
$queries = #()
foreach ($file in $files) {
$queries += (Get-Content $file.FullName).replace("GO","")
}
#join each query into a single T-SQL statement...
$singleTransaction = $queries -join ";"
#execute statement...
Invoke-Sqlcmd -ServerInstance 'SERVER' -Database 'DB' -Query $singleTransaction
To really achieve a 'single transaction'...you may have to have a consistent input to modify and put into one statement. I am not sure how you will need to do that.
Finally I wrote the Script using SMO Objects to handle the GO Statement and Transactions.
$SqlFilePath = "D:\Roshan\Testing\SQL\"
$serverName = "MyServer"
$databaseName = "MyDB"
$QueryResult = Invoke-Sqlcmd -ServerInstance $serverName -Database $databaseName -Query "SELECT DISTINCT FName from dbo.TableName" -AS DataRows
$connection = new-object system.data.SqlClient.SQLConnection("Data Source=$serverName;Integrated Security=SSPI;Initial Catalog=$databaseName;Connection Timeout=600;Max Pool Size=10");
$Server = new-Object Microsoft.SqlServer.Management.Smo.Server(New-Object Microsoft.SqlServer.Management.Common.ServerConnection($connection))
$script_contents ="SET XACT_ABORT ON
GO
BEGIN TRANSACTION
GO"
Get-ChildItem $SqlFilePath -Filter *.sql| Sort-Object $_.Name|
ForEach-Object {
if(!$QueryResult.FName.Contains($_.Name))
{
Write-Host $_.Name -ForegroundColor Magenta
#[string]$script_contents = Get-Content $_.FullName
$script_contents += [IO.File]::ReadAllText($_.FullName)
#Write-Host $script_contents
#$Server.ConnectionContext.ExecuteNonQuery($script_contents)
}
}
$script_contents+= " COMMIT TRANSACTION;"
$Server.ConnectionContext.ExecuteNonQuery($script_contents)
You can write some thing in your shell script
select filename from tablename; >> file.out
--->002_File.sql
grep -v 'file.out' * >> excludedfile.out

Invoke-Sqlcmd : Conversion failed when converting date and/or time from character string

My PowerShell script keeps failing when trying to import a CSV file.
The error message reads: "Invoke-Sqlcmd : Conversion failed when converting date and/or time from character string."
In my CSV file, I have a column called "LastWriteTime"
Here is my PowerShell script
$database = 'test'
$server = 'leasesql'
$table = 'dbo.ssis'
Import-CSV C:\temp\text.csv | ForEach-Object {
Invoke-Sqlcmd -Database $database -ServerInstance $server -Query "INSERT INTO $table (PSComputerName, FullName, Extension, LastWriteTime)
VALUES ('$_.PSComputerName','$_.FullName','$_.Extension','$_.LastWriteTime,')"
}
I was able to find out that I need to put the columns in $($_.colmunname)
Import-CSV C:\temp\ssis.csv | ForEach-Object {
Invoke-Sqlcmd -Database $database -ServerInstance $server -Query "INSERT INTO $table VALUES ('$($.PSComputerName)','$($.FullName)','$($.Extension)','$($.LastWriteTime)')"

Print sql query results to file automatically

Hello everyone I am currently trying to write the results of a query to a tab delimited csv file automatically on a schedule. Currently I am using powershell to do this
$results = Invoke-SQLCmd -ServerInstance $Server -Database $Database -query "select distinct * from WHIProducts"
$results | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "$inventorypath\inventory_$date\$filename" -fo -en ascii
The problem with this is that the results are so big I am getting a system.outofmemoryexception error. I have tried increasing the maxmemorypershell but I still get the same error. I need to this automatically so going into SSMS and doing it manually is not enough. Any ideas?
I am trying to print ~170k rows. There will be more eventually, probably up to about 300k. Here is the powershell error.
ConvertTo-csv : Exception of type 'System.OutOfMemoryException' was
thrown. At
C:\Users\pmaho\Dropbox\MASSFILEStest\scripts\daily_inventory‌​.ps1:59
char:12 + $results | ConvertTo-csv -NoTypeInformation -Delimiter "`t"
| Out-Fil ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
CategoryInfo : NotSpecified: (:) [ConvertTo-Csv], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.Co‌​nvertToCsvCommand
–
I am using SQL Server express edition
Try piping the output of Invoke-SqlCmd to the Export-CSV cmdlet. Here is the script.
Push-Location; Import-Module SQLPS -DisableNameChecking; Pop-Location
$SQLServer = "localhost\inst1"
$DBName = "ExportCSVTesting"
$ExportFile = "C:\Users\BIGRED-7\Documents\Git\csvfiles\addresses.csv"
$Counter = 0
while ( $true )
{
# Remove the export file
if (Test-Path -Path $ExportFile -PathType Leaf) {
Remove-Item $ExportFile -Force
}
# Clear the buffer cache to make sure each test is done the same
$ClearCacheSQL = "DBCC DROPCLEANBUFFERS"
Invoke-Sqlcmd -ServerInstance $SQLServer -Query $ClearCacheSQL
# Export the table through the pipeline and capture the run time. Only the export is included in the run time.
$ExportSQL = "SELECT * FROM [addresses] ;"
$sw = [Diagnostics.Stopwatch]::StartNew()
Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $ExportSQL | Export-CSV -Path $ExportFile -NoTypeInformation
$sw.Stop()
$sw.Elapsed
$Milliseconds = $sw.ElapsedMilliseconds
# Get a row count for display
$RowCountSQL = "SELECT COUNT(0) AS [Count] FROM [addresses] ;"
$RowCount = Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $RowCountSQL
$RowCount = $RowCount.Count
$Counter++
Write-Output ("Run $Counter of RowCount: $RowCount")
# Log the run statistics
$StatsSQL = "INSERT INTO [RunStats] (Counter,Milliseconds,Notes) VALUES ($RowCount,$Milliseconds,'Pipeline')"
Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $StatsSQL
}

PowerCLI and SQL import issue with var

I am using this form post as a baseline for a PowerCLI script I am trying to write.
Powershell Get-QADUser results to SQL table
My modified scrip looks like this:
Add-PSSnapin VMware.VimAutomation.Core
Add-PSSnapin SqlServerCmdletSnapin100
Add-PSSnapin SqlServerProviderSnapin100
$db_server = "127.0.0.1\VIM_SQLEXP"
$db = "Billing"
$table = "vmdata"
$username = "import"
$pwd = "myPassWord"
# First, clear existing table
$sql_query_del = "DELETE FROM $table"
Invoke-Sqlcmd -ServerInstance $db_server -Database $db -Username $username -Password $pwd -Query $sql_query_del
# Get vms and resources add to DB
Get-resourcepool -name finance | get-vm | foreach {
$name = $_.Name
$PowerState = $_.PowerState
$NumCPUs = $_.'Num CPUs'
$MemoryGB = $_.MemoryGB
Write-Host " Name : $Name PowerState : $PowerState MemoryGB : $MemoryGB Num CPUs : $NumCPUs"
$sql_query = "INSERT INTO $table (Name, PowerState, MemoryGB, NumCPUs) VALUES ('$Name', '$PowerState', '$MemoryGB', '$NumCPUs')"
Invoke-Sqlcmd -ServerInstance $db_server -Database $db -Username $username -Password $pwd -Query $sql_query
}
My problem is the Num CPUs that has a space in it when just executing this command
Get-resourcepool -name finance | get-vm
Name PowerState Num CPUs MemoryGB
---- ---------- -------- --------
GreatPlains01 PoweredOn 4 8.000
what is the appropriate method to reference this to have the result display?
When executing this via power shell I get the following result:
Name : GreatPlains01 PowerState : PoweredOn MemoryGB : 8 Num CPUs :
So the variable I am using isn't correct I have tried [] "" and ' ' around Num CPUs with no success.
Although the column header in the output of the Get-VM cmdlet is 'Num CPUs', the property name is NumCPU. So you should use:
$NumCPUs = $_.NumCPU