I'm having trouble to come up with solution that would compare LineNumbers of matching pairs from two lists. I will show you what I mean on example.
I have one SQL script, where I am inserting some data into existing tables. For ensuring repeatability of the script, before every insert into I am deleting the previous content of the table with "delete" statement. I am able to parse the file and check If every "insert into database1.table1" also have "delete from database1.table1" in the file. But i don't know how to check if the delete statement of the particular table is before the insert into statement (you need to delete the content of the table before you load new data into it). I figured I would need to use the LineNumber property, but I really don't know how to combine it with the database.table check.
This is what i got into first variable with this command:
$insertinto = Get-ChildItem "$packagepath\Init\" -Include 03_Init_*.txt -Recurse | Select-String -Pattern "insert into "
#content of variable
C:\Users\hanus\Documents\sql_init.txt:42:insert into database1.table1
C:\Users\hanus\Documents\sql_init.txt:130:insert into database1.table2
C:\Users\hanus\Documents\sql_init.txt:282:insert into database2.table3
Here is what I got into second variable with this command:
$deletefrom = Get-ChildItem "$packagepath\Init\" -Include 03_Init_*.txt -Recurse | Select-String -Pattern "delete from "
#content of the variable
C:\Users\hanus\Documents\sql_init.txt:40:delete from database1.table1;
C:\Users\hanus\Documents\sql_init.txt:128:delete from database1.table2;
C:\Users\hanus\Documents\sql_init.txt:280:delete from database2.table3;
The expected output would be something like: This"delete from" statement is not before "insert into" statement, even though it's in the file.
I hope I described the problem well. I am new to Powershell and scripting so be please patient with me. Thank you for any help in advance!
You're already using Select-String, so this should be pretty simple. The content of those variables is far more than you're seeing there. Run this:
$deletefrom | Format-List * -Force
You'll see that each match contains an object with properties for what file the match is from, what line number the match was found on, and more. I think if you capture the table that is being modified in your Select-String with a look behind of what you're searching on now you could group on that, and then alert on times where the delete happens after the insert.
Get-ChildItem "$packagepath\Init\*" -Include 03_Init_*.txt -Recurse |
Select-String "(?<=delete from |insert into )([^;]+)" |
Group-Object {$_.Matches[0].value} |
ForEach-Object {
if($_.group[0] -notmatch 'delete from'){Write-Warning "Inserting into $($_.Name) before deleting"}
}
Related
For my example, I'm looking to compare a source file with some changes made to the attributes in a table - lets say in the form of another source file.
What i want to achieve is
Sourcefile.csv
Newfile.csv
Deltafile.csv
(this will export only the changes deltas (rows) between the two files)
What i would like to achieve is that the row with the changes is exported as the delta, not just the column attribute.
All other rows that match do not need to be updated.
100,Renie,Stav,Renie.Stav#yopmail.com,Renie.Stav#gmail.com,CHANGE
101,Neila,Germann,CHANGE,Neila.Germann#gmail.com,developer
I've looked at Powershell, FC and SSIS incremental loading to see if this will work for my needs but a need some guidance in the right direction. Any help is greatly appreciated! :)
**Current Method **
Looking indepth at the powershell i tried in https://www.reddit.com/r/PowerShell/comments/cea8ax/compare_2_csv_files_and_export_the_rows_that_do/
Which is
# Compare work
$csv1 = Import-Csv -Path C:\Users\G23\Documents\Hackingfolder\source.csv
$csv2 = Import-Csv -Path C:\Users\G23\Documents\Hackingfolder\new.csv
$head = (Get-Content -Path C:\Users\G23\Documents\Hackingfolder\source.csv | Select-Object -First 1) -split ","
Compare-Object $csv1 $csv2 -Property $head -PassThru| Export-Csv C:\Users\G23\Documents\Hackingfolder\TheDiff.csv -NoTypeInformation
# Remove side indicator if you dont care to know where the diff came from
Compare-Object $csv1 $csv2 -Property $head | Select-Object -Property $head
Ignoring the side indicators i would get the rows that would not match, both of them. its not smart enough to know which one is the updated one. e.g I want the exported delta changes rows only.
PowerShell output
instead of the desired below changes only in csv
100,Renie,Stav,Renie.Stav#yopmail.com,Renie.Stav#gmail.com,CHANGE
101,Neila,Germann,CHANGE,Neila.Germann#gmail.com,developer
Thanks!
I have 4 files with the same csv header as following
Column1,Column2,Column3,Column4
But I only required data from Column2,Column3,Column4 for import the data into SQL database using BCP . I am using the PowerShell to select the columns that I want and import the required data using BCP but my powershell executed with no error and there are not data updated in my database table. May I know how to set the BCP to import the output from Powershell to database table. Here are my powershell script
$filePath = Get-ChildItem -Path 'D:\test\*' -Include $filename
$desiredColumn = 'Column2','Column3','Column4'
foreach($file in $filePath)
{
write-host $file
$test = import-csv $file | select $desiredColumn
write-host $test
$action = bcp <myDatabaseTableName> in $test -T -c -t";" -r"\n" -F2 -S <MyDatabase>
}
These are the output from the powershell script
D:\test\sample1.csv
#{column2=111;column3=222;column4=333} #{column2=444;column3=555;column4=666}
D:\test\sample2.csv
#{column2=777;column3=888;column4=999} #{column2=aaa;column3=bbb;column4=ccc}
First off, you can't update a table with bcp. It is used to bulk load data. That is, it will either insert new rows or export existing data into a flat file. Changing existing rows, usually called as updating, is out of scope for bcp. If that's what you need, you need to use another a tool. Sqlcmd works fine, and Powershell's got Invoke-Sqlcmd for running arbitary TSQL statements.
Anyway, the BCP utility has notoriously tricky syntax. As far as I know, one cannot bulk load data by passing the data as parameter to bcp, a source file must be used. Thus you need to save the filtered file and pass its name to bcp.
Exporting a filtered CSV is easy enough, just remember to use -NoTypeInformation switch, lest you'll get #TYPE Selected.System.Management.Automation.PSCustomObject as your first row of data. Assuming the bcp arguments are well and good (why -F2 though? And Unix newlines?).
Stripping double quotes requires another an edit to the file. Scrpting Guy has a solution.
foreach($file in $filePath){
write-host $file
$test = import-csv $file | select $desiredColumn
# Overwrite filtereddata.csv, should one exist, with filtered data
$test | export-csv -path .\filtereddata.csv -NoTypeInformation
# Remove doulbe quotes
(gc filtereddata.csv) | % {$_ -replace '"', ''} | out-file filtereddata.csv -Fo -En ascii
$action = bcp <myDatabaseTableName> in filtereddata.csv -T -c -t";" -r"\n" -F2 -S <MyDatabase>
}
Depending on your locale, column separator might be semicolon, colon or something else. Use -Delimiter '<character>' switch to pass whatever you need or change bcp's argument.
Erland's got a helpful page about bulk operations. Also, see Redgate's advice.
Without need to modify the file first, there is an answer here about how bcp can handle quoted data.
BCP in with quoted fields in source file
Essentially, you need to use the -f option and create/use a format file to tell SQL your custom field delimiter (in short, it is no longer a lone comma (,) but it is now (",")... comma with two double quotes. Need to escape the dblquotes and a small trick to handle the first doulbe quote on a line. But it works like a charm.
Also, need the format file to ignore column(s)... just set the destination column number to zero. All with no need to modify the file before load. Good luck!
I have a powershell script which I have written, it also works as well. The problem that I now have is that originally the requirement was to save the results onto a database, now I want to email the results as well. I thought about a couple of options, but finding it difficult, now an easy way out which i thought of was to export the result to CSV then attach that CSV to the email.
The code below is inside my loop.
$sql = " INSERT INTO dbo.tb_checks ([ServerName],[Directory],[DirectoryFile] ,[FileCreationDate]) SELECT '$ServerName', '$Filepath', '$fileName', '$FileDate'"
Invoke-Sqlcmd2 -serverinstance $DBServer -database $Database -query $sql
SELECT '$ServerName', '$Filepath', '$fileName', '$FileDate' | Export-csv $Outfilename -append
The CSV file gets generated, but with no data.
Another idea which i thought of was to have the data stored in an array, then loop through/spit out the entire content of the array in an email.
Can someone help please ?
The reason that your CSV is empty is because you aren't feeding it an array that it can work with. What headers would it use in your script? It has no idea, it's just having random stuff thrown at it. Change that last line to this:
New-Object PSObject -Property #{Server=$ServerName;FilePath=$Filepath;FileName=$fileName;FileDate=$FileDate} | Export-csv $Outfilename -Append -NoTypeInformation
Assuming that your variables are set right it should output the file you want.
If you want to make it a table and put it in an email make an empty array before your loop, then do something like:
$LoopArray = #()
<start of loop>
$LoopArray += New-Object PSObject -Property #{Server=$ServerName;FilePath=$Filepath;FileName=$fileName;FileDate=$FileDate}
$LoopArray | Export-csv $Outfilename -Append -NoTypeInformation
<end of loop>
Then afterwards you have the array to work with that has all your data in the CSV stored and can be injected into an email.
I'm trying to set up a script designed to change a bit over 100 placeholders in probably some 50 files. In general I got a list of possible placeholders, and their values. I got some applications that have exe.config files as well as ini files. These applications are stored in c:\programfiles(x86)\ and in d:\In general I managed to make it work with one path, but not with two. I could easily write the code to replace twice, but that leaves me with a lot of messy code and would be harder for others to read.
ls c:\programfiles(x86) -Recurse | where-object {$_.Extension -eq ".config" -or $_.Extension -eq ".ini"} | %{(gc $PSPath) | %{
$_ -replace "abc", "qwe" `
-replace "lkj", "hgs" `
-replace "hfd", "fgd"
} | sc $_PSPath; Write-Host "Processed: " + $_.Fullname}
I've tried to include 2 paths by putting $a = path1, $b = path2, c$ = $a + $b and that seems to work as far as getting the ls command to run in two different places. however, it does not seem to store the path the files are in, and so it will try to replace the filenames it has found in the folder you are currently running the script from. And thus, even if I might be in one of the places where the files is supposed to be, it's not in the other ...
So .. Any idea how I can get Powershell to list files in 2 different places and replace the same variables in both places without haveing to have the code twice ? I thought about putting the code I would have to use twice into a variable, calling it when I needed to instead of writing it again, but it seemed to resolve the code before using it, and that didn't exactly give me results since the data comes from the first part.
If you got a cool pipeline, then every problem looks like ... uhm ... fluids? objects? I have no clue. But anyway, just add another layer (and fix a few problems along the way):
$places = 'C:\Program Files (x86)', 'D:\some other location'
$places |
Get-ChildItem -Recurse -Include *.ini,*.config |
ForEach-Object {
(Get-Content $_) -replace 'abc', 'qwe' `
-replace 'lkj', 'hgs' `
-replace 'hfd', 'fgd' |
Set-Content $_
'Processed: {0}' -f $_.FullName
}
Notable changes:
Just iterate over the list of folders to crawl as the first step.
Doing the filtering directly in Get-ChildItem makes it faster and saves the Where-Object.
-replace can be applied directly to an array, no need for another ForEach-Object there.
If the number of replacements is large you may consider using a hashtable to store them so that you don't have twenty lines of -replace 'foo', 'bar'.
I have a power-shell script with which I am trying to back up a constantly changing number of SQL databases. Fortunately all of these databases are listed in a registry key. I am leveraging this in a for-each loop. The issue that I am having is that after grabbing the registry value that I want, when I try to pass it into my function to back up the databases there seems to be information in the variable that I can get rid of. If I output the contents of the variable to the screen by just calling the variable ($variable) is shows just fine. But if I write-host the variable to the screen the extra "content" that shows up when calling the function also shows up.
Here is the part of the script that generates the contents of the variable.
foreach ($childitem in get-childitem "HKLM:\SOFTWARE\Wow6432Node\Lanovation\Prism Deploy\Server Channels")
{$DBName = get-itemproperty Registry::$childitem | select "Channel Database Name"
write-host $DBname}
Here is what write-host displays :
#{Channel Database Name=Prism_Deploy_Sample_268CBD61_AC9E_4853_83DE_E161C72458DE}
but what I need is only this part :
Prism_Deploy_Sample_268CBD61_AC9E_4853_83DE_E161C72458DE
I have tried looking online at how to do this, and what I've found mentions things similar to $variable.split and then specifying my delimiters. But when I try this I get an error saying "Method invocation failed because [System.Management.Automation.PSCustomObject] doesn't contain a method named 'split'."
I'm at a loss as to where to go from where I'm at currently.
select-object will return an object that has the named properties that you "select". To get just value of that property, just access it by name:
write-host $DBname."Channel Database Name"
Sounds like it's returning a hash table row object.
Try
write-host $DBName.value
or, failing that, do a
$DBName | Get-member
When in doubt, get-member gives you a nice idea of what you are dealing with.
You should be able to write
foreach ($childitem in get-childitem "HKLM:\SOFTWARE\Wow6432Node\Lanovation\Prism Deploy\Server Channels")
{$DBName = get-itemproperty Registry::$childitem | select "Channel Database Name"
write-host $DBname.Name}
to get what you are looking for