can anyone tell me how I can update the following procedure to handle big files please (size <= 10 G):
proc read_text_file { file } {
set fp [open ${file} r]
set return_data ""
while { [gets $fp each_line] != -1 } {
lappend return_data ${each_line}
}
close $fp
return ${return_data}
}
my objective is to read a huge file line by line in a better runtime
Thanks
When you have a very large file, you categorically want to avoid bringing it all into memory at once. (Also, Tcl 8.* has a memory chunk allocation limit that makes bringing in 50GB of data intensely exciting. That's a long-standing API bug that's fixed in 9.0 — in alpha — but you'll have to put up with it for now.)
If you can, do a pass over the file to identify where the interesting sub-chunks of it are. For the sake of argument, let's assume that those are the lines that match a pattern; here's an example that finds where procedures are in a Tcl script (under some simple assumptions).
proc buildIndices {filename} {
set f [open $filename]
set indices {}
try {
while {![eof $f]} {
set idx [tell $f]
set line [gets $f]
if {[regexp {^proc (\w+)} $line -> name]} {
dict set indices $name $idx
}
}
return $indices
} finally {
close $f
}
}
Now you have the indices, you can then pull in a procedure from the file like this:
proc obtainProcedure {filename procName indices} {
set f [open $filename]
try {
seek $f [dict get $indices $procName]
set procedureDefinition ""
while {[gets $f line] >= 0} {
append procedureDefinition $line "\n"
if {[info complete $procedureDefinition]} {
# We're done; evaluate the script in the caller's context
tailcall eval $procedureDefinition
}
}
} finally {
close $f
}
}
You'd use that like this:
# Once (possibly even save this to its own file)
set indices [buildIndices somefile.tcl]
# Then, to use
obtainProcedure somefile.tcl foobar $indices
If you're doing this a lot, convert your code to use a database; they're a lot more efficient in the long run. The index building is equivalent to building the database and the other procedure is equivalent to doing a DB query.
I'm setting up macros, Set, and Say. Defined in procedures.
proc Set {key value args} {
set ::$key $value
set "::key2" "$key"
}
proc Say {key} {
puts $key
}
proc Say2 {key} {
set key3 [regsub "\%" $key "\$"]
puts $key3
eval puts $key3
}
Which allows me to execute the following:
Set "test" "this should display this test text"
Say $key2 ;#should display the key "test" which is what we just set
Say $test ;#presents the value of the key test
Output
% Set "test" "this should display this test text"
test
% Say $key2 ;#should display the key "test" which is what we just set
test
% Say $test ;#presents the value of the key test
this should display this test text
So now lets say I want to reassign the variable $ to %
Set "mouse" "squeak" ;#set key mouse with value string of "squeak"
Say $mouse ;#displays the value as set above correctly
Say2 %mouse ;#start using our own characters to represent variables - switch the % for a $ and then output
However I then get when using eval,
can't read "mouse": no such variable
Output
% Set "mouse" "squeak" ;#set key mouse with value string of "squeak"
mouse
% Say $mouse ;#displays the value as set above correctly
squeak
% Say2 %mouse ;#start using our own characters to represent variables
$mouse
can't read "mouse": no such variable
I'm finding this weird because we set it above, we can recall the value using the standard $ And I can prove that the regsub in Say2 is working as it should replacing % with $.
%mouse becomes $mouse which is a valid variable.
Eval $mouse outputs with no such variable
Am I missing something?
Thanks
The issue is with the proc:
proc Say2 {key} {
set key3 [regsub {%} $key {$}]
puts $key3
eval puts $key3 ;# here
}
$mouse does not exist in this proc. It was not passed as a parameter, nor was it created with set. It exists however in the global namespace. One way to reach for it is to use uplevel in this case:
proc Say2 {key} {
set key3 [regsub {%} $key {$}]
puts $key3
uplevel puts $key3
}
Another option I often use is upvar to bring the variable inside (though in this case, we don't want the $ anymore):
proc Say2 {key} {
set key3 [regsub {%} $key {}]
puts $key3
upvar $key3 var
puts $var
}
PS: I also went ahead and removed some backslashes since they aren't really needed in that scenario.
I'm trying to change a variable inside a ScriptBlock.
What am I doing wrong?
$reader=(New-Object System.Xml.XmlNodeReader $xaml)
$Window=[Windows.Markup.XamlReader]::Load($reader)
$Window.Add_SourceInitialized( {
$timer = new-object System.Windows.Threading.DispatcherTimer
$timer.Interval = [TimeSpan]"0:0:0.25"
$timer.Add_Tick( $updateBlock )
$timer.Start()
} )
$count = 0
$updateBlock = { Write-Host $count; $count++; Write-Host $count}
The Output is a repeating sequence of 0 and 1. So how do I access the variable and not only a copy of it?
When you modify $count inside the scope of the ScriptBlock, a local copy is created, and the original $Count variable in the parent scope is left untouched.
There are a few ways to modify $count in the parent scope, either with an explicit scope qualifier:
$updateBlock = { Write-Host $count; $script:count++; Write-Host $count}
Or by retrieving the variable with Get-Variable and the relative -Scope parameter (-Scope 1 refers to the immediate parent scope):
$updateBlock = { Write-Host $count; (Get-Variable -Scope 1 -Name count).Value++; Write-Host $count}
Or (as pointed out by #PetSerAl), use the [ref] keyword:
$updateBlock = { Write-Host $count; ([ref]$count).Value++; Write-Host $count}
I've got a strange issue with Powershell. I've created a script that collects info from the SQL server and I need to crop it all into 1 table. I've tried using the SqlServerCmdletSnapin but on some of my servers it returns nothing. So i decided to try and go old school and use connection strings. However when I do that the Join-Object operation returns nothing. Here are shortened versions of the scripts. The first is using the Cmdlet, the second is the old way.
1st
function AddItemProperties($item, $properties, $output)
{
if($item -ne $null)
{
foreach($property in $properties)
{
$propertyHash =$property -as [hashtable]
if($propertyHash -ne $null)
{
$hashName=$propertyHash["name"] -as [string]
if($hashName -eq $null)
{
throw "there should be a string Name"
}
$expression=$propertyHash["expression"] -as [scriptblock]
if($expression -eq $null)
{
throw "there should be a ScriptBlock Expression"
}
$_=$item
$expressionValue=& $expression
$output | add-member -MemberType "NoteProperty" -Name $hashName -Value $expressionValue
}
else
{
# .psobject.Properties allows you to list the properties of any object, also known as "reflection"
foreach($itemProperty in $item.psobject.Properties)
{
if ($itemProperty.Name -like $property)
{
$output | add-member -force -MemberType "NoteProperty" -Name $itemProperty.Name -Value $itemProperty.Value
}
}
}
}
}
}
function WriteJoinObjectOutput($leftItem, $rightItem, $leftProperties, $rightProperties, $Type)
{
$output = new-object psobject
if($Type -eq "AllInRight")
{
# This mix of rightItem with LeftProperties and vice versa is due to
# the switch of Left and Right arguments for AllInRight
AddItemProperties $rightItem $leftProperties $output
AddItemProperties $leftItem $rightProperties $output
}
else
{
AddItemProperties $leftItem $leftProperties $output
AddItemProperties $rightItem $rightProperties $output
}
$output
}
<#
.Synopsis
Joins two lists of objects
.DESCRIPTION
Joins two lists of objects
.EXAMPLE
Join-Object $a $b "Id" ("Name","Salary")
#>
function Join-Object
{
[CmdletBinding()]
[OutputType([int])]
Param
(
# List to join with $Right
[Parameter(Mandatory=$true,
Position=0)]
[object[]]
$Left,
# List to join with $Left
[Parameter(Mandatory=$true,
Position=1)]
[object[]]
$Right,
# Condition in which an item in the left matches an item in the right
# typically something like: {$args[0].Id -eq $args[1].Id}
[Parameter(Mandatory=$true,
Position=2)]
[scriptblock]
$Where,
# Properties from $Left we want in the output.
# Each property can:
# - Be a plain property name like "Name"
# - Contain wildcards like "*"
# - Be a hashtable like #{Name="Product Name";Expression={$_.Name}}. Name is the output property name
# and Expression is the property value. The same syntax is available in select-object and it is
# important for join-object because joined lists could have a property with the same name
[Parameter(Mandatory=$true,
Position=3)]
[object[]]
$LeftProperties,
# Properties from $Right we want in the output.
# Like LeftProperties, each can be a plain name, wildcard or hashtable. See the LeftProperties comments.
[Parameter(Mandatory=$true,
Position=4)]
[object[]]
$RightProperties,
# Type of join.
# AllInLeft will have all elements from Left at least once in the output, and might appear more than once
# if the where clause is true for more than one element in right, Left elements with matches in Right are
# preceded by elements with no matches. This is equivalent to an outer left join (or simply left join)
# SQL statement.
# AllInRight is similar to AllInLeft.
# OnlyIfInBoth will cause all elements from Left to be placed in the output, only if there is at least one
# match in Right. This is equivalent to a SQL inner join (or simply join) statement.
# AllInBoth will have all entries in right and left in the output. Specifically, it will have all entries
# in right with at least one match in left, followed by all entries in Right with no matches in left,
# followed by all entries in Left with no matches in Right.This is equivallent to a SQL full join.
[Parameter(Mandatory=$false,
Position=5)]
[ValidateSet("AllInLeft","OnlyIfInBoth","AllInBoth", "AllInRight")]
[string]
$Type="OnlyIfInBoth"
)
Begin
{
# a list of the matches in right for each object in left
$leftMatchesInRight = new-object System.Collections.ArrayList
# the count for all matches
$rightMatchesCount = New-Object "object[]" $Right.Count
for($i=0;$i -lt $Right.Count;$i++)
{
$rightMatchesCount[$i]=0
}
}
Process
{
if($Type -eq "AllInRight")
{
# for AllInRight we just switch Left and Right
$aux = $Left
$Left = $Right
$Right = $aux
}
# go over items in $Left and produce the list of matches
foreach($leftItem in $Left)
{
$leftItemMatchesInRight = new-object System.Collections.ArrayList
$null = $leftMatchesInRight.Add($leftItemMatchesInRight)
for($i=0; $i -lt $right.Count;$i++)
{
$rightItem=$right[$i]
if($Type -eq "AllInRight")
{
# For AllInRight, we want $args[0] to refer to the left and $args[1] to refer to right,
# but since we switched left and right, we have to switch the where arguments
$whereLeft = $rightItem
$whereRight = $leftItem
}
else
{
$whereLeft = $leftItem
$whereRight = $rightItem
}
if(Invoke-Command -ScriptBlock $where -ArgumentList $whereLeft,$whereRight)
{
$null = $leftItemMatchesInRight.Add($rightItem)
$rightMatchesCount[$i]++
}
}
}
# go over the list of matches and produce output
for($i=0; $i -lt $left.Count;$i++)
{
$leftItemMatchesInRight=$leftMatchesInRight[$i]
$leftItem=$left[$i]
if($leftItemMatchesInRight.Count -eq 0)
{
if($Type -ne "OnlyIfInBoth")
{
WriteJoinObjectOutput $leftItem $null $LeftProperties $RightProperties $Type
}
continue
}
foreach($leftItemMatchInRight in $leftItemMatchesInRight)
{
WriteJoinObjectOutput $leftItem $leftItemMatchInRight $LeftProperties $RightProperties $Type
}
}
}
End
{
#produce final output for members of right with no matches for the AllInBoth option
if($Type -eq "AllInBoth")
{
for($i=0; $i -lt $right.Count;$i++)
{
$rightMatchCount=$rightMatchesCount[$i]
if($rightMatchCount -eq 0)
{
$rightItem=$Right[$i]
WriteJoinObjectOutput $null $rightItem $LeftProperties $RightProperties $Type
}
}
}
}
}
Add-PSSnapin SqlServerCmdletSnapin100
Add-PSSnapin SqlServerProviderSnapin100
#------------------------------------------------------------------------------------------------------------------------------------------------------
$Sizehash = $null
$Sizehash = #{}
$SizeQuery = "SELECT
DatabaseName = DB_NAME(database_id),
Total_MB = CAST(SUM(size) * 8. / 1024 AS DECIMAL(8,2))
FROM sys.master_files WITH(NOWAIT)
WHERE database_id > 4
GROUP BY database_id
order by DatabaseName"
$Sizehash = Invoke-Sqlcmd -Query $SizeQuery -ServerInstance PMMCSQL1
#------------------------------------------------------------------------------------------------------------------------------------------------------
$CThash = $null
$CThash = #{}
$CTQuery = "exec sp_msforeachdb 'IF ''?'' NOT IN (''master'',''model'',''tempdb'',''msdb'',''pubs'')
BEGIN
USE ?;
select DB_NAME() AS CTName, value AS ClientType
FROM fn_listextendedproperty(default, default, default, default, default, default, default)
where name = ''Client type'';
END'"
$CThash = Invoke-Sqlcmd -Query $CTQuery -ServerInstance PMMCSQL1
#------------------------------------------------------------------------------------------------------------------------------------------------------
$JoinCT = $null
$JoinCT = Join-Object `
-Left $Sizehash `
-Right $CThash `
-LeftProperties DatabaseName,Total_MB `
-RightProperties ClientType `
-Type AllinBoth `
-Where {$args[0].DatabaseName -eq $args[1].CTName}
$JoinCT
2nd
> function AddItemProperties($item, $properties, $output)
{
if($item -ne $null)
{
foreach($property in $properties)
{
$propertyHash =$property -as [hashtable]
if($propertyHash -ne $null)
{
$hashName=$propertyHash["name"] -as [string]
if($hashName -eq $null)
{
throw "there should be a string Name"
}
$expression=$propertyHash["expression"] -as [scriptblock]
if($expression -eq $null)
{
throw "there should be a ScriptBlock Expression"
}
$_=$item
$expressionValue=& $expression
$output | add-member -MemberType "NoteProperty" -Name $hashName -Value $expressionValue
}
else
{
# .psobject.Properties allows you to list the properties of any object, also known as "reflection"
foreach($itemProperty in $item.psobject.Properties)
{
if ($itemProperty.Name -like $property)
{
$output | add-member -force -MemberType "NoteProperty" -Name $itemProperty.Name -Value $itemProperty.Value
}
}
}
}
}
}
function WriteJoinObjectOutput($leftItem, $rightItem, $leftProperties, $rightProperties, $Type)
{
$output = new-object psobject
if($Type -eq "AllInRight")
{
# This mix of rightItem with LeftProperties and vice versa is due to
# the switch of Left and Right arguments for AllInRight
AddItemProperties $rightItem $leftProperties $output
AddItemProperties $leftItem $rightProperties $output
}
else
{
AddItemProperties $leftItem $leftProperties $output
AddItemProperties $rightItem $rightProperties $output
}
$output
}
<#
.Synopsis
Joins two lists of objects
.DESCRIPTION
Joins two lists of objects
.EXAMPLE
Join-Object $a $b "Id" ("Name","Salary")
#>
function Join-Object
{
[CmdletBinding()]
[OutputType([int])]
Param
(
# List to join with $Right
[Parameter(Mandatory=$true,
Position=0)]
[object[]]
$Left,
# List to join with $Left
[Parameter(Mandatory=$true,
Position=1)]
[object[]]
$Right,
# Condition in which an item in the left matches an item in the right
# typically something like: {$args[0].Id -eq $args[1].Id}
[Parameter(Mandatory=$true,
Position=2)]
[scriptblock]
$Where,
# Properties from $Left we want in the output.
# Each property can:
# - Be a plain property name like "Name"
# - Contain wildcards like "*"
# - Be a hashtable like #{Name="Product Name";Expression={$_.Name}}. Name is the output property name
# and Expression is the property value. The same syntax is available in select-object and it is
# important for join-object because joined lists could have a property with the same name
[Parameter(Mandatory=$true,
Position=3)]
[object[]]
$LeftProperties,
# Properties from $Right we want in the output.
# Like LeftProperties, each can be a plain name, wildcard or hashtable. See the LeftProperties comments.
[Parameter(Mandatory=$true,
Position=4)]
[object[]]
$RightProperties,
# Type of join.
# AllInLeft will have all elements from Left at least once in the output, and might appear more than once
# if the where clause is true for more than one element in right, Left elements with matches in Right are
# preceded by elements with no matches. This is equivalent to an outer left join (or simply left join)
# SQL statement.
# AllInRight is similar to AllInLeft.
# OnlyIfInBoth will cause all elements from Left to be placed in the output, only if there is at least one
# match in Right. This is equivalent to a SQL inner join (or simply join) statement.
# AllInBoth will have all entries in right and left in the output. Specifically, it will have all entries
# in right with at least one match in left, followed by all entries in Right with no matches in left,
# followed by all entries in Left with no matches in Right.This is equivallent to a SQL full join.
[Parameter(Mandatory=$false,
Position=5)]
[ValidateSet("AllInLeft","OnlyIfInBoth","AllInBoth", "AllInRight")]
[string]
$Type="OnlyIfInBoth"
)
Begin
{
# a list of the matches in right for each object in left
$leftMatchesInRight = new-object System.Collections.ArrayList
# the count for all matches
$rightMatchesCount = New-Object "object[]" $Right.Count
for($i=0;$i -lt $Right.Count;$i++)
{
$rightMatchesCount[$i]=0
}
}
Process
{
if($Type -eq "AllInRight")
{
# for AllInRight we just switch Left and Right
$aux = $Left
$Left = $Right
$Right = $aux
}
# go over items in $Left and produce the list of matches
foreach($leftItem in $Left)
{
$leftItemMatchesInRight = new-object System.Collections.ArrayList
$null = $leftMatchesInRight.Add($leftItemMatchesInRight)
for($i=0; $i -lt $right.Count;$i++)
{
$rightItem=$right[$i]
if($Type -eq "AllInRight")
{
# For AllInRight, we want $args[0] to refer to the left and $args[1] to refer to right,
# but since we switched left and right, we have to switch the where arguments
$whereLeft = $rightItem
$whereRight = $leftItem
}
else
{
$whereLeft = $leftItem
$whereRight = $rightItem
}
if(Invoke-Command -ScriptBlock $where -ArgumentList $whereLeft,$whereRight)
{
$null = $leftItemMatchesInRight.Add($rightItem)
$rightMatchesCount[$i]++
}
}
}
# go over the list of matches and produce output
for($i=0; $i -lt $left.Count;$i++)
{
$leftItemMatchesInRight=$leftMatchesInRight[$i]
$leftItem=$left[$i]
if($leftItemMatchesInRight.Count -eq 0)
{
if($Type -ne "OnlyIfInBoth")
{
WriteJoinObjectOutput $leftItem $null $LeftProperties $RightProperties $Type
}
continue
}
foreach($leftItemMatchInRight in $leftItemMatchesInRight)
{
WriteJoinObjectOutput $leftItem $leftItemMatchInRight $LeftProperties $RightProperties $Type
}
}
}
End
{
#produce final output for members of right with no matches for the AllInBoth option
if($Type -eq "AllInBoth")
{
for($i=0; $i -lt $right.Count;$i++)
{
$rightMatchCount=$rightMatchesCount[$i]
if($rightMatchCount -eq 0)
{
$rightItem=$Right[$i]
WriteJoinObjectOutput $null $rightItem $LeftProperties $RightProperties $Type
}
}
}
}
}
#------------------------------------------------------------------------------------------------------------------------------------------------------
$ConnTimeout = 30
$QueryTimeout = 120
$DB = "master"
$conn=New-Object System.Data.SqlClient.SQLConnection
$conn.ConnectionString = "Server=PMMCSQL1;Database=$DB;Integrated Security=True;Connect Timeout=$ConnTimeout"
$SQLcmd=New-Object system.Data.SqlClient.SqlCommand
$SQLcmd.CommandTimeout=$QueryTimeout
$conn.Open()
$SizeQuery = "SELECT
DatabaseName = DB_NAME(database_id),
Total_MB = CAST(SUM(size) * 8. / 1024 AS DECIMAL(8,2))
FROM sys.master_files WITH(NOWAIT)
where database_id > 4
group BY database_id
order by DatabaseName"
$SQLcmd.CommandTimeout = $QueryTimeout
$SqlCmd.CommandText = $SizeQuery
$Sqlcmd.Connection = $conn
$SQLAdap=New-Object system.Data.SqlClient.SqlDataAdapter($SQLcmd)
$DataSet=New-Object system.Data.DataSet
$SQLAdap.fill($DataSet) | Out-Null
$conn.Close()
$Sizehash = $DataSet.Tables | Format-Table
$SQLAdap = $null
$DataSet = $null
#------------------------------------------------------------------------------------------------------------------------------------------------------
$conn.Open()
$CTQuery = "exec sp_msforeachdb 'IF ''?'' NOT IN (''master'',''model'',''tempdb'',''msdb'',''pubs'')
BEGIN
USE ?;
select DB_NAME() AS CTName, value AS ClientType
FROM fn_listextendedproperty(default, default, default, default, default, default, default)
where name = ''Client type'';
END'"
$SQLcmd.CommandTimeout = $QueryTimeout
$SqlCmd.CommandText = $CTQuery
$Sqlcmd.Connection = $conn
$SQLAdap=New-Object system.Data.SqlClient.SqlDataAdapter($SQLcmd)
$DataSet=New-Object system.Data.DataSet
$SQLAdap.fill($DataSet) | Out-Null
$conn.Close()
$CThash = $DataSet.Tables | Format-Table
$SQLAdap = $null
$DataSet = $null
#------------------------------------------------------------------------------------------------------------------------------------------------------
$JoinCT = $null
$JoinCT = Join-Object `
-Left $Sizehash `
-Right $CThash `
-Where {$args[0].DatabaseName -eq $args[1].CTName} `
-LeftProperties DatabaseName,Total_MB `
-RightProperties ClientType `
-Type AllinBoth
$JoinCT
Why reinvent the wheel? If you are trying to join tables from different databases on different servers create a linked server and let SQL do the joins.
sql query for join two tables of different databases that are in two Servers
($Sizehash | Select DatabaseName,Total_MB) | Join ($CThash | Select CTName,ClientType) -on DatabaseName -eq CTName
See: In Powershell, what's the best way to join two tables into one?
How can I declare variables and assign values to them at run time.
Reason: I am fetching these variables values from sql server and these variable values are configurable in nature
Code which I have tried till now
[array]$varArray = #($($ServerName),$($HostName))
foreach($varname in $varArray)
{
$varname = "some test value"
}
Write-Host $ServerName
Write-Host $HostName
The simplest way of using dynamically named variables would be a dictionary:
$vars = #{} # create empty dictionary
# add key/value pairs to dictionary:
$vars["foo"] = 23
$vars["bar"] = "foobar"
$vars["baz"] = Get-Content C:\sample.txt
Another way would be to declare variables on the fly:
$name = "foo"
$value = "bar"
New-Variable $name $value
echo $foo
Or you could create a custom object and add properties as Kyle C suggested. That approach is similar to a dictionary, although technically different.
You could try adding a NoteProperty to the object.
$varname | Add-Member -type NoteProperty -name TestProperty -value "some test value" -PassThru
Also see this for what types of objects you can add a member to: What objects are suitable for Add-Member?