expect inside while loop - loops forever - while-loop

I'm new to expect scripts, so please forgive my stumbling...
Below is the meat of my expect script. The intent is to scroll thru several screens of output, after each of which the user is prompted with "Continue? [y/n]".
Finally, when there are no more screens, a "% " prompt is displayed, which SHOULD cause execution to fall out of the while loop.
set more_screens 1
while {$more_screens > 0} {
sleep 10
expect {
"\[y/n]" { send "y\r"}
"% " { set more_screens 0 }
}
}
What happens instead is... it stays in the while loop forever, sending "y" over and over and over again. I have set "exp_internal 1", and from that output it "seems" like the expect keeps re-reading text that it has already matched on, and so keeps seeing "[y/n]", and keeps sending "y" when, in fact, there are only 2 screens of output, and thus only 2 "Continue? [y/n]" prompts.
(The sleep statement is probably not necessary - i just added it to maybe solve the problem - it did NOT - and to allow me to digest the debug output better.)
Bottom line... Are there any obvious blunders in my code? I'll take any suggestions at improving this and eliminating the endless looping.
EDIT BELOW added to this question, after James made a helpful suggestion.
Thanks James for your quick response, and your helpful suggestion! But...
The same problem persists with your approach (although, yours IS much more elegant, and I'll add this to my expect tool kit.)
The problem sure seems to be, as noted initially, each execution of the expect statement RE-READS TEXT THAT WAS ALREADY READ AND COMPARED. Below is output when I execute the "exp_continue" code from James, and I set "exp_internal 1" to get debug output on my screen...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>expect: does "get dump tables\r\n\r% get dump tables\n\r\r\nIfn TableName Configured >MaxUse InUse LastDropTime\r\n3 cdm_app 100002 190 33 >\r\n3 cdm_conv 2000002 675180 4813 \r\n3 cdm_pdisc 250002 >250002 1304 01-24-2014-19:14:59\r\n3 cdm_kpi 100001 141 25 >\r\n3 cdm_qoe 500003 204918 1578 \r\n3 cdm_qoe_hd 2500003 >582993 1578 \r\n3 cdm_kpi_error_app 100001 5 2 \r\n3 >cdm_kpi_error 100001 7 2 \r\n3 asr_cache 1000000 >1000000 999995 \r\n3 asr_sess 2000000 62670 29748 \r\n3 >asr_conn 3000000 64428 31147 \r\n3 asr_sess_keys 1500000 >1015269 1009049 \r\n3 asr_conn_opts 6000000 0 0 \r\n3 >asr_events 4000000 5239 144 \r\n3 skt_table 2000000 >2000000 2000000 \r\n3 skt_trans 1000000 408020 254674 \r\n3 >ses_sip_db 5000 0 0 \r\n3 ses_gtp_mob_txn 5000 >0 0 \r\nContinue? [y/n]: " (spawn_id exp6) match glob pattern "[y/n]"? yes
>expect: set expect_out(0,string) "n"
>expect: set expect_out(spawn_id) "exp6"
>expect: set expect_out(buffer) "get dump tables\r\n\r% get dump tables\n\r\r\nIfn"
>send: sending "y\r" to { exp6 }
>expect: continuing expect
>
>
>expect: does " TableName Configured MaxUse InUse LastDropTime\r\n3 >cdm_app 100002 190 33 \r\n3 cdm_conv 2000002 >675180 4813 \r\n3 cdm_pdisc 250002 250002 1304 01-24-2014->\r\n3 cdm_kpi 100001 141 25 \r\n3 cdm_qoe 500003 >204918 1578 \r\n3 cdm_qoe_hd 2500003 582993 1578 \r\n3 >cdm_kpi_error_app 100001 5 2 \r\n3 cdm_kpi_error 100001 >7 2 \r\n3 asr_cache 1000000 1000000 999995 \r\n3 >asr_sess 2000000 62670 29748 \r\n3 asr_conn 3000000 >64428 31147 \r\n3 asr_sess_keys 1500000 1015269 1009049 \r\n3 >asr_conn_opts 6000000 0 0 \r\n3 asr_events 4000000 >5239 144 \r\n3 skt_table 2000000 2000000 2000000 \r\n3 >skt_trans 1000000 408020 254674 \r\n3 ses_sip_db 5000 >0 0 \r\n3 ses_gtp_mob_txn 5000 0 0 \r\nContinue?
>[y/n]: " (spawn_id exp6) match glob pattern "[y/n]"? yes
>expect: set expect_out(0,string) "n"
>expect: set expect_out(spawn_id) "exp6"
>expect: set expect_out(buffer) " TableName Con"
>send: sending "y\r" to { exp6 }
>^Csighandler: handling signal(2)
>async event handler: Tcl_Eval(exit 130)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
After the expect matches on "[y/n]" the first time, it then does a "expect: continuing expect" (middle of the above text output), then the block of text it reads next is, except for the first few words, THE SAME TEXT BLOCK IT ALREADY READ AND COMPARED.
Am I missing anything??? This has to be a problem if the expect statements re-reads already processed output, yeah? (I have looked at the actual output sent by the target system, and it does NOT send out the same text block a second time.)
Again, I'm new at expect scripts, but I can't see any other explanation for what the debug output above shows. (And, I apologize for having trouble formatting the output correctly - I really am trying!)
Thanxks to anyone who has the patience to read all of the above, and perhaps has an explanation or suggestion.

You need to be rescued by the exp_continue command. :-)
What that command does when encountered is stay within the expect block and try to match again, with whatever new input may come.
So you can really shorten your above code to be:
expect {
"\[y/n]" {
send "y\r"
exp_continue
}
"% " {
# Do whatever is needed here, after which program flow will continue *outside* of the expect block
}
}
Let me know if that works out for you!
EDIT - based on #feenyman99 additional info:
Ok, I see what it is. You have the wrong pattern. By using "[y/n]", a match is being produced with a single 'n' character. There's your matched string:
expect: set expect_out(0,string) "n"
expect_out(0,string) holds the matched pattern. expect_out(buffer) holds the portion of input removed from the buffer, which holds all input up to and including the matched pattern (from then on, the next expect operation will look for a match on input after the last matched pattern). As you can see, it holds the input up to and including the first literal 'n' character that is found (newlines don't count):
expect: set expect_out(buffer) "get dump tables\r\n\r% get dump tables\n\r\r\nIfn"
So what is happening is that your script is sending the "y\r" way before the yes/no prompt is presented. And, although I'm not seeing the rest of the logs, I'm guessing the next match happens shortly after, upon hitting the next 'n' character.
So you need to change your pattern matching statement to be able to match the yes/no prompt. Better make that a regexp match (-re). I tested the following, and it works (tested on Tcl 8.4.13):
expect {
-re "\\\[y/n]" { send "y\r"}
The multiple backslashes are because the backslash is also an escape character within the pattern matcher. Kind of tricky, but sometimes they are needed.
Let me know how that goes. You should be all set now.
PS: This may come in handy: http://www.tcl.tk/doc/howto/regexp81.tml

Related

Output progress over time in hashcat

I am analysing the amount of hashes cracked over a set period of time.
I am looking to save the current status of the crack every 10 seconds.
'''
Recovered........: 132659/296112 (44.80%) Digests, 0/1 (0.00%) Salts
Recovered/Time...: CUR:3636,N/A,N/A AVG:141703,8502198,204052756 (Min,Hour,Day)
Progress.........: 15287255040/768199139595 (1.99%)
'''
I want these 3 lines of the status saved every 10 seconds or so.
Is it possible to do this within hashcat or will I need to make a separate script in python?
Getting the status every 10 seconds
You can enable printing the status with --status and you can set the status to prints every X seconds with --status-timer X. You can see these command line arguments on the hashcat options wiki page, or hashcat --help.
Example: hashcat -a 0 -m 0 example.hash example.dict --status --status-timer 10
Saving all the statuses
I'm assuming that you just want to save everything that gets printed by hashcat while it's running. An easy way to do this is just copy everything from stdout into a file. This is a popular s/o question, so we'll just use this answer.
To be safe, let's use -a which appends to the file, so we don't accidentally overwrite previous runs. All we need to do is put | tee -a file.txt after our hashcat call.
Solution
Give this a shot, it should save all the statuses (and everything else from stdout) to output.txt:
hashcat -a A -m M hashes.txt dictionary.txt --status --status-timer 10 | tee -a output.txt
Just swap out A, M, hashes.txt, and dictionary.txt with the arguments you're using.
If you need help getting just the "Recovered" lines from this output file, or if this doesn't work on your computer (I'm on OSX), let me know in a comment.
In addition to Andrew Zick's answer, note that for machine-readable status, hashcat has native support for machine-readable output - see the --machine-readable option. This produces tab-separated output like so:
STATUS 5 SPEED 111792 1000 EXEC_RUNTIME 0.007486 CURKU 1 PROGRESS 62 62 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 14247323 1000 EXEC_RUNTIME 0.038953 CURKU 36 PROGRESS 2232 2232 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 36929864 1000 EXEC_RUNTIME 1.661804 CURKU 1296 PROGRESS 80352 80352 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 66538858 1000 EXEC_RUNTIME 3.237319 CURKU 46656 PROGRESS 28926722892672 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
STATUS 5 SPEED 63562975 1000 EXEC_RUNTIME 3.480536 CURKU 1679616 PROGRESS 104136192 104136192 RECHASH 0 1 RECSALT 0 1 REJECTED 0 UTIL -1
... which is exactly what tools like Hashtopolis use to provide a front-end to hashcat output.
For machine-readable output, the options --outfile, and --outfile-format are available. See the Format section of the output of hashcat --help for the options to --outfile-format:
- [ Outfile Formats ] -
# | Format
===+========
1 | hash[:salt]
2 | plain
3 | hex_plain
4 | crack_pos
5 | timestamp absolute
6 | timestamp relative

Calculate GSM message variable length code using awk

I need a awk code which will calculate the message length using a given length.
For instance:
a message having character length is 600 and calculation formula will be like this;
first minus 600 from 160 and answer ll be 1 and then in next iteration 153 will minus from 540 and so on.
if message character lenght is less than 160 than consider as 1 message and exit
600-160=540 1 message
540-153=387 1+1 message
387-153=233 1+1+1 message
233-153=81 1+1+1+1 message
81 less than 153 1+1+1+1+1 message
this iteration will run until value is not less than 153.
600 character message is equal to 5 messages i mean.

Equal loading for parallel task distribution

I have a large number of independent tasks I would like to run, and I would like to distribute them on a parallel system such that each processor does the same amount of work, and maximizes my efficiency.
I would like to know if there is a general approach to finding a solution to this problem, or possibly just a good solution to my exact problem.
I have T=150 tasks I would like to run, and the time each task will take is t=T. That is, task1 takes 1 one unit of time, task2 takes 2 units of time... task150 takes 150 units of time. Assuming I have n=12 processors, what is the best way to divide the work load between workers, assuming the time it takes to begin and clean up tasks is negligible?
Despite my initial enthusiasm for #HighPerformanceMark's ingenious approach, I decided to actually benchmark this using GNU Parallel with -j 12 to use 12 cores and simulated 1 unit of work with 1 second of sleep.
First I generated a list of the jobs as suggested with:
paste <(seq 1 72) <(seq 150 -1 79)
That looks like this:
1 150
2 149
3 148
...
...
71 80
72 79
Then I pass the list into GNU Parallel and pick up the remaining 6 jobs at the end in parallel:
paste <(seq 1 72) <(seq 150 -1 79) | parallel -k -j 12 --colsep '\t' 'sleep {1} ; sleep {2}'
sleep 73 &
sleep 74 &
sleep 75 &
sleep 76 &
sleep 77 &
sleep 78 &
wait
That runs in 16 mins 24 seconds.
Then I used my somewhat simpler approach, which is just to run big jobs first so you are unlikely to be left with any big ones at the end and thereby get imbalance in CPU load because just one big job needs to run and the rest of your CPUs have nothing to do:
time parallel -j 12 sleep {} ::: $(seq 150 -1 1)
And that runs in 15 minutes 48 seconds, so it is actually faster.
I think the problem with the other approach is that after the first 6 rounds of 12 pairs of jobs, there are 6 jobs left the longest of which takes 78 seconds, so effectively 6 CPUs sit there doing nothing for 78 seconds. If the number of tasks was divisible by the number of CPUs, that would not occur but 150 doesn't divide by 12.
The solution I came to was similar to those mentioned above. Here is the pseudo-code if anyone is interested:
N_proc = 12.0
Jobs = range(1,151)
SerialTime = sum(Jobs)
AverageTime = SerialTime / N_proc
while Jobs remaining:
for proc in range(0,N_proc):
if sum(proc) < AverageTime:
diff = AverageTime - sum(proc)
proc.append( max( Jobs <= diff ) )
Jobs.pop( max( Jobs <= diff ) )
else:
proc.append( min(Jobs) )
Jobs.pop( min(Jobs) )
This seemed to be the optimal method for me. I tried it on many different distributions of job run-times, and it seems to do a decent job of evenly distributing the work, so long as N_proc << N_jobs.
This is a slight modification from largest first, in that each processor first tries to avoid doing more than it's "fair share". If it must go over it's fair share, then it will attempt to stay near the fair answer by grabbing the smallest remaining task from the queue.

predefined preemption points among processes

I have forked many child processes and assigned priority and core to each of them. Porcess A executes at period of 3 sec and process B at a period of 6 sec. I want them to execute in such a way that the higher priority processes should preempt lower priority ones only at predefined points and tried to acheive it with semaphores. I have used this same code snippets within the 2 processes with different array values in both.
'bubblesort_desc()' sorts the array in descending order and prints it. 'bubblesort_asc()' sorts in ascending order and prints.
while(a<3)
{
printf("time in sort1.c: %d %ld\n", (int)request.tv_sec, (long int)request.tv_nsec);
int array[SIZE] = {5, 1, 6 ,7 ,9};
semaphore_wait(global_sem);
bubblesort_desc(array, SIZE);
semaphore_post(global_sem);
semaphore_wait(global_sem);
bubblesort_asc(array, SIZE);
semaphore_post(global_sem);
semaphore_wait(global_sem);
a++;
request.tv_sec = request.tv_sec + 6;
request.tv_nsec = request.tv_nsec; //if i add 1ms here like an offset to the lower priority one, it works.
semaphore_post(global_sem);
semaphore_close(global_sem); //close the semaphore file
//sleep for the rest of the time after the process finishes execution until the period of 6
clk = clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &request, NULL);
if (clk != 0 && clk != EINTR)
printf("ERROR: clock_nanosleep\n");
}
I get the output like this whenever two processes get activated at the same time. For example at time units of 6, 12,..
time in sort1.c: 10207 316296689
time now in sort.c: 10207 316296689
9
99
7
100
131
200
256
6
256
200
5
131
100
99
1
1
5
6
7
9
One process is not supposed to preempt the other while one set of sorted list is printing. But it's working as if there are no semaphores. I defined semaphores as per this link: http://linux.die.net/man/3/pthread_mutexattr_init
Can anyone tell me what can be the reason for that? Is there a better alternative than semaphores?
Its printf that's causing the ambiguous output. If the results are printed without '\n' then we get a more accurate result. But its always better to avoid printf statements for real time applications. I used trace-cmd and kernelshark to visualize the behaviour of the processes.

Pulling from a very very specific location imbedded in a text file

I finished every piece of code in my program save for one tid bit, how to pull two numbers from a text file. I know how to pull lines, I know how to pull search strings, but I cant figure out this one to save my life.
Anyways here is a sample of the automatically generated text that I need to pull from...
.......................................................................
Applications Memory Usage (kB):
Uptime: 6089044 Realtime: 6089040
** MEMINFO in pid 764 [com.lookout] **
native dalvik other total
size: 27908 8775 N/A 36683
allocated: 3240 4216 N/A 7456
free: 24115 4559 N/A 28674
(Pss): 1454 1142 6524 *9120*
(priv dirty): 1436 628 5588 *7652*
Objects
Views: 0 ViewRoots: 0
AppContexts: 0 Activities: 0
Assets: 3 AssetManagers: 3
Local Binders: 15 Proxy Binders: 41
Death Recipients: 3
OpenSSL Sockets: 0
SQL
heap: 98 MEMORY_USED: 98
PAGECACHE_OVERFLOW: 16 MALLOC_SIZE: 50
DATABASES
pgsz dbsz Lookaside(b) Dbname
1 14 120 google_analytics.db
Asset Allocations
zip:/system/app/com.lookout_6.0.1_r8234_Release.apk:/resources.arsc: 161K
.............................................................................
The two numbers that I need out of this are the two ones that I put in the **'s (the asterisks are not normally there). These numbers will be different every time this sheet is generated, and the number placement might be different as well as some of the numbers could have 4 digits, 5 digits, or 6 digits.
If anyone could shed any light on the subject it would be greatly appreciated
Thanks,
Zach
You just need to read in the last word of the line and convert it to a number. Use String.LastIndexOf to find the last space " " in the file and read the data from that point forwards.
Dim line as String = " (Pss): 1454 1142 6524 9120"
Dim value as Integer
If line.IndexOf("(Pss)") > 0 Then
value = CInt(line.Substring(line.LastIndexOf(" ") + 1))
End If