I am pretty new to Tcl and have been writing snippets to improve the automation of the process flow in our Work. I want to compare the value of a variable to its previous value so that the code knows its a new flow. The problem is: How to store the old value of a variable? or more precisely, how can we store the value of a variable that is assigned during previous flow?(Is it even possible?)
The following is how our workflow looks like
Start compilation
A) Start phase1 and run flow.tcl script twice
B) Start phase2 and run flow.tcl script twice
...
End compilation
Here in this example, the variable is assigned a new value every time it is run in a different phase. But since I am unable to store the value of the variable to compare, am stuck at trying different options but in vain. This might be totally impossible but as far as I know Tcl can handle almost everything.
Any help is greatly appreciated
Thanks in advance
Hemanth
Edit: simple solution found. Have the data written to text files and read in back again. Thanks
You can save variables in an array and load the variables back into Tcl. The command "array get" serializes the data and "array set" puts it back into an array.
flow.tcl
#!/usr/bin/tclsh
proc load_data {data_file array_name} {
upvar $array_name data
if {[file exists $data_file]} {
set fp [open $data_file r]
array set data [read $fp]
close $fp
}
}
proc save_data {data_file array_name} {
upvar $array_name data
set fp [open $data_file w]
puts $fp [array get data]
close $fp
}
set now [clock seconds]
# Set defaults. If you need new keys in your data file you can add them here.
set data(count) 0
set data(last_timestamp) $now
# Load existing data over default values. If the key doesn't exist the default will be used.
load_data "flow.dat" data
# Use the saved data to find elapsed time.
set elapsed [expr $now - $data(last_timestamp)]
set count $data(count)
# Save new data.
set data(last_timestamp) $now
set data(count) [incr count]
save_data "flow.dat" data
puts "It's been $elapsed seconds since last run. You have run this $count times."
output
% ./flow.tcl
It's been 0 seconds since last run. You have run this 1 times.
flow.dat
% cat flow.dat
count 1 last_timestamp 1427142892
Related
I am trying to figure out how to check if a channel is empty or not.
For instance, I have two processes. The first process runs only if a combination of parameters/flags are set and if so, checks also if its input file from another process (input via a channel) is not empty, then it creates a new input file for a second process (to eventually replace the default one). As a simplified example:
.....
.....
// create the channel here to force nextflow to wait for the first process
_chNewInputForProcessTwo = Channel.create()
process processOne {
when:
params.conditionOne && parameters.conditionTwo
input:
file inputFile from _channelUpstreamProcess
output:
file("my.output.file") into _chNewInputForProcessTwo
script:
"""
# check if we need to produce new input for second process (i.e., input file not empty)
if [ -s ${inputFIle} ]
then
<super_command_to_generate_new_fancy_input_for_second_process> > "my.output.file"
else
echo "No need to create new input"
fi
"""
}
// and here I would like to check if new input was generated or leave the "default" one
_chInputProcessTwo = Channel.from(_chNewInputForProcessTwo).ifEmpty(Channel.value(params.defaultInputProcessTwo))
process secondProcess {
input:
file inputFile from _chInputProcessTwo
......
......
etc.
When I try running with this approach it fails because the channel _chNewInputForProcessTwo contains DataflowQueue(queue=[]) therefore, not being actually empty.
I've tried several things looking at the documentation and the threads on google groups and on gitter. trying to set it to empty, but then it complains i am trying to use the channel twice. putting create().close(), etc.
Is there a clean/reasonable way to do this? I could do it using a value channel and have the first process output some string on the stdout to be picked up and checked by the second process, but that seems pretty dirty to me.
Any suggestions/feedback is appreciated. Thank you in advance!
Marius
Best to avoid trying to check if the channel is empty. If your channel could be empty and you need a default value in your channel, you can use the ifEmpty operator to supply one. Note that a single value is implicitly a value channel. I think all you need is:
myDefaultInputFile = file(params.defaultInputProcessTwo)
chInputProcessTwo = chNewInputForProcessTwo.ifEmpty(myDefaultInputFile)
Also, calling Channel.create() is usually unnecessary.
Can anyone please help me in writing a script in AHK based on below requirement.
Requirement:
I have a CSV/TXT file in my windows environment which contains 20,000+ records in below format.
So, when I run the script it should prompt a InputBox to enter an instance name.
Example : If i enter Instance4 , it should display result in MsgBox as ServerName4
Sample Format:
ServerName1,ServerIP,Instance1,Type
ServerName2,ServerIP,Instance2,Type
ServerName3,ServerIP,Instance3,Type
ServerName4,ServerIP,Instance4,Type
ServerName5,ServerIP,Instance5,Type
.
.
.
Also as the CSV/TXT file contains large no of records , pls also consider the best way to avoid delay in fetching the results.
Please post your code, or at least show what you've already done.
You can use a Parsing Loop with CSV as the delimiter, and make a variable for each 'Instance' who's value is that of the current row's 'ServerName'.
The steps are to first FileRead the data from the file, then Loop, Parse like so:
Loop, Parse, data, CSV
{
; Parses row by row, then column by column in each row.
; A_LoopField // Current value
; A_Index // Current loop's index
; Write a script that makes a variable named with the current value of column 3, and give it the value of column 1
}
After that, you can make a Goto loop that spams InputBox and following a command that prints out the needed variable using the MsgBox command, like so:
MsgBox % %input%
I have a big text file about 4GB, and more than 8 million lines, I'm writing a perl script to read this file line by line, do some processing and update the info to sybase, i did this in a batch way, 1000 lines per batch for update commit, but here comes the problem, at first, a batch only costs 10 to 20 seconds, but with the processing goes, updating a batch becomes slower and slower, a batch costs 3 to 4 min, I definitely have no idea why this is happening! Any body can help me analys this what may be the cause? Thanks in advance, on my knee...
==>I'm writing a perl script to read this file line by line, do some processing and update the info to sybase
Please do entire processing at one go mean process your source file at one go; Prepare data structure using hash, array as per requirement and then start inserting data into database.
Please keep below points in mind while inserting large data into database.
1- If each column data is not too large then you can insert entire data at one go also.( you may require good RAM not sure about size because it depend on dataset u need to process).
2- You should use execute_array of perl DBI so that you can insert data at one go.
3- If you have not sufficient RAM to insert data at one go then please devide your data ( may be in 8 part, 1 million lines each time).
4- Also Make it sure that you are preparing the statement once. In every run you are just executing with new data set.
5- Set your auto_commit off.
A sample code to use execute_array of perl DBI. I have used this to insert around 10 million of data into mysql.
Please keep ur data in arrays like below in the form of array.
#column1_data, #column2_data, #column3_data
print $logfile_handle, "Total records to insert--".scalar(#column1_data);
print $logfile_handle, "Inserting data into database";
my $sth = $$dbh_ref->prepare("INSERT INTO $tablename (column1,column2,column3) VALUES (?,?,?)")
or print ($logfile_handle, "ERROR- Couldn't prepare statement: " . $$dbh_ref->errsr) && exit;
my $tuples = $sth->execute_array(
{ ArrayTupleStatus => \my #tuple_status },
\#column1_data,
\#column2_data,
\#column3_data
);
$$dbh_ref->do("commit");
print ($logfile_handle,"Data Insertion Completed.");
if ($tuples) {
print ($logfile_handle,"Successfully inserted $tuples records\n");
} else {
##print Error log or those linese which are not inserted
my $status = $tuple_status[$tuple];
$status = [0, "Skipped"] unless defined $status;
next unless ref $status;
print ($logfile_handle, "ERROR- Failed to insert (%s,%s,%s): %s\n",
$column1_data[$tuple], $column2_data[$tuple],$column3_data[$tuple], $status->[1]);
}
}
Here is the simple perl script fetching data from SQL.
Read data and write on a file OUTFILE, and print the data on screen for every 10000th line.
One thing I am curious is that the printing the data on screen terminates very quickly(in 30 seconds), however, data fetching and writing on a file ends very slowly(30 minutes later).
The amount of data is not large. The output files size is less than 100Mbyte.
while ( my ($a,$b) = $curSqlEid->fetchrow_array() )
{
printf OUTFILE ("%s,%d\n", $a,$b);
$counter ++;
if($counter % 10000 == 0){
printf ("%s,%d\n", $a,$b);
}
}
$curSqlEid->finish();
$dbh->disconnect();
close(OUTFILE);
You are suffering from buffering.
Handles other than STDERR are buffered by default, and most handles use a block buffering. That means Perl will wait until there is 8KB* of data to write before sending anything to the system.
STDOUT is special. When is attached to a terminal (and only then), it uses a different kind of buffering: line buffering. When using line buffering, the data is flushed every time a newline is encountered in the data to write.
You can see this by running
$ perl -e'print "abc"; print "def"; sleep 5; print "\n"; sleep 5;'
[ 5 seconds pass ]
abcdef
[ 5 seconds pass ]
$ perl -e'print "abc"; print "def"; sleep 5; print "\n"; sleep 5;' | cat
[ 10 seconds pass ]
abcdef
The solution is to turn off buffering.
use IO::Handle qw( ); # Not needed on Perl 5.14 or later
OUTFILE->autoflush(1);
* — 8KB is the default. It can be configured when Perl is compiled. It used to be a non-configurable 4KB until 5.14.
I think you are seeing the output file size as 0 while the script is running and displaying on the console. Do not go by that. The file size will show up only once the script has finished. This is due to output buffering.
Anyways, the delay cannot be as large as 30 min. Once the script is done, you should see the output file data.
I tried various things, but the final conclusion is that python and perl has basically different handling data flow from DB. It looks like in perl, it is possible to handle data line by line while the data is transferred from DB. However, in Python it needs to wait until the entire data download from the server to process it.
Let me start with I am very new to powershell and programming for that matter. I have a powershell script that takes some arguments and that outputs a value.
The result of the script is going to be something like 9/10 where 9 would be the number active out of the total amount of nodes. I want to assign the output to a variable so I can then call another script based on the value.
This is what I have tried, but it does not work:
$active = (./MyScript.ps1 lb uid **** site)
I have also tried the following which seems to assign the variable an empty string
$active = (./MyScript.ps1 lb uid **** site | out-string)
In both cases they run and give me the value immediately instead of assigning it to the variable. When I call the variable, I get no data.
I would embrace PowerShell's object-oriented nature and rather than output a string like "9/10", create an object with properties like NumActiveNodes and TotalNodes e.g. in your script output like so:
new-object psobject -Property #{NumActiveNodes = 9; TotalNodes = 10}
Of course, substitute in the dynamic values for num active and total nodes. Note that uncaptured objects will automatically appear on your script's output. Then, if this is your scripts only output, you can do this:
$obj = .\MyScript.ps1
$obj.NumActiveNodes
9
$obj.TotalNodes
10
It will make it nicer for those consuming the output of your script. In fact the output is somewhat self-documenting e.g.:
C:\PS> .\MyScript.ps1
NumActiveNodes TotalNodes
-------------- ----------
9 10
P.S. When did StackOverflow start sucking so badly at formatting PowerShell script?
If you don't want to change the script ( and assuming only that $avail_count/$total_count line is written by the script), you can do:
$var= powershell .\MyScript.ps1
Or just drop the write-host and have just $avail_count/$total_count
and then do:
$var = .\MyScript.ps1
you could just do a $global:foobar in your script and it will persist after the script is closed
I know, the question is a bit older, but it might help someone to find the right answer.
I had the similar problem with executing PS script with another PS script and saving the output into variable, here are 2 VERY good answers:
Mathias
mklement0
Hope it helps!
Please up-vote them if so, because they are really good!