What would be the best method to tail a file in Clojure? I haven't come across any utilities that would help do so, but ideas on how to build one would be appreciated!
Thanks.
As kotarak stated, you could use RandomAccessFile to seek to the end of the File. Unfortunately you have to busy-wait/sleep for changes.
Using a lazy sequence you can process the lines "on-the-fly":
(import java.io.RandomAccessFile)
(defn raf-seq
[#^RandomAccessFile raf]
(if-let [line (.readLine raf)]
(lazy-seq (cons line (raf-seq raf)))
(do (Thread/sleep 1000)
(recur raf))))
(defn tail-seq [input]
(let [raf (RandomAccessFile. input "r")]
(.seek raf (.length raf))
(raf-seq raf)))
; Read the next 10 lines
(take 10 (tail-seq "/var/log/mail.log"))
Update:
Something like tail -f /var/log/mail.log -n 0, using doseq, so the changes are actually consumed.
(doseq [line (tail-seq "/var/log/mail.log")] (println line))
You can use RandomAccessFile to seek directly to the end of the file and search for linebreaks from there. Not as elegant and short as the take-last approach, but also not O(n) which might matter for big file sizes.
No solution for tail -f, though. Some inspiration might be found in JLogTailer.
Something like:
(take-last 10 (line-seq (clojure.contrib.io/reader "file")))
Related
I have been creating some PCB footprints in KiCad recently, which are stored in s-expression files with data that looks like this:
(fp_text user %R (at 0 5.08) (layer F.Fab)
(effects (font (size 1 1) (thickness 0.15)))
)
(fp_line (start -27.04996 -3.986) (end -27.24996 -3.786) (layer F.Fab) (width 0.1))
(pad "" np_thru_hole circle (at 35.56 0) (size 3.175 3.175) (drill 3.175) (layers *.Cu *.Mask)
(clearance 1.5875))
(pad 96 smd rect (at 1.25 3.08473) (size 0.29972 1.45034) (layers F.Cu F.Paste F.Mask)
(clearance 0.09906))
I would like to be able to write shell one-liners to efficiently edit multiple parameters. I would normally use Awk for something like this, but the recursive nature of s-expressions makes it ill-suited for the task. I would like to know if there is a programming language with an interpreter designed to handle piped data and can process s-expressions natively. Perhaps a data-driven dialect of Lisp would do this, but I'm not sure where to look.
In summary, I would like to be able to make quick edits to an s-expression file in a similar manner to the way Awk lets me process columns of data line-by-line; only in the case of s-expressions the processing would be performed level-by-level.
Example: find all of the pad expressions of type smd with (size 0.29972 1.45034), and renumber each one based its position.
Simple script
Here is an example in Common Lisp, assuming your input is in file "/tmp/ex.cad" (it could also be obtained by reading the output stream of a process).
The main processing loop consists in opening the file in order to obtain an input stream in (which is automatically closed at the end of with-open-file), loop over all forms in the file, process them and possibly output them to standard output. You could complexify the process as much as you want, but the following is good enough:
(with-open-file (in #"/tmp/ex.cad")
(let ((*read-eval* nil))
(ignore-errors
(loop (process-form (read in))))))
Suppose you want to increase the width of fp_line entries, ignore fp_text and otherwise print the form unmodified, you could define process-form as follows:
(defun process-form (form)
(destructuring-bind (header . args) form
(print
(case header
(fp_line (let ((width (assoc 'width args)))
(when width (incf (second width) 3)))
form)
(fp_text (return-from process-form))
(t form)))))
Running the previous loop would then output:
(FP_LINE (START -27.04996 -3.986) (END -27.24996 -3.786) (LAYER F.FAB) (WIDTH 3.1))
(PAD "" NP_THRU_HOLE CIRCLE (AT 35.56 0) (SIZE 3.175 3.175) (DRILL 3.175) (LAYERS *.CU *.MASK) (CLEARANCE 1.5875))
(PAD 96 SMD RECT (AT 1.25 3.08473) (SIZE 0.29972 1.45034) (LAYERS F.CU F.PASTE F.MASK) (CLEARANCE 0.09906))
More safety
From there, you can build more elaborate pipelines, with the help of pattern matching or macros if you want. You have to take into account some safety measures, like binding *read-eval* to nil, using with-standard-io-syntax
and binding *print-circte* to T as suggested by tfb, disallowing fully qualified symbols (by having #\: signal an error), etc. Ultimately, like Shell scripts one-liners, the amount of precautions you add is based on how much you trust your inputs:
;; Load libraries
(ql:quickload '(:alexandria :optima))
;; Import symbols in current package
(use-package :optima)
(use-package :alexandria)
;; Transform source into a stream
(defgeneric ensure-stream (source)
(:method ((source pathname)) (open source))
(:method ((source string)) (make-string-input-stream source))
(:method ((source stream)) source))
;; make reader stop on illegal characters
(defun abort-reader (&rest values)
(error "Aborting reader: ~s" values))
Dedicated package for KiCad symbols (exporting is optional):
(defpackage :kicad
(:use)
(:export #:fp_text
#:fp_line
#:pad
#:size))
Loop over forms:
(defmacro do-forms ((form source &optional result) &body body)
"Loop over forms from source, eventually return result"
(with-gensyms (in form%)
`(with-open-stream (,in (ensure-stream ,source))
(with-standard-io-syntax
(let ((*read-eval* nil)
(*print-circle* t)
(*package* (find-package :kicad))
(*readtable* (copy-readtable)))
(set-macro-character #\: #'abort-reader nil)
(loop
:for ,form% := (read ,in nil ,in)
:until (eq ,form% ,in)
:do (let ((,form ,form%)) ,#body)
:finally (return ,result)))))))
Example:
;; Print lines at which there is a size parameter, and its value
(let ((line 0))
(labels ((size (alist) (second (assoc 'kicad:size alist)))
(emit (size) (when size (print `(:line ,line :size ,size))))
(process (options) (emit (size options))))
(do-forms (form #P"/tmp/ex.cad")
(match form
((list* 'kicad:fp_text _ _ options) (process options))
((list* 'kicad:fp_line options) (process options))
((list* 'kicad:pad _ _ _ options) (process options)))
(incf line))))
Output
(:LINE 2 :SIZE 3.175)
(:LINE 3 :SIZE 0.29972)
Just write a simple Lisp or Scheme script which loops on reading and processes recursively your s-expr as required. On Linux I would recommend using Guile (a good Scheme interpreter) or perhaps Clisp (a simple Common Lisp implementation) or even SBCL (a very powerful Common Lisp).
(You might consider DSSSL, but in your case it is overkill)
Notice that your sample input is not an S-expression, because (layer F.Fab) is not one (since after the dot you should have another s-expression, not an atom like Fab). I guess it is a typo and should be (layer "F.Fab"); or maybe your KiCad software don't process S-expressions, but some other input language (which should be specified, probably in EBNF notation) inspired by S-expressions.
Notice also that KiCad is a free software and has a community with forums and a mailing list. Perhaps you should ask your actual problem there?
PS. We don't know what transformation you have in mind, but Scheme and Common Lisp are really fit for such tasks. In most cases they are extremely simple to code (probably a few lines only).
I have a problem with ksh in that a while loop is failing to obey the "while" condition. I should add now that this is ksh88 on my client's Solaris box. (That's a separate problem that can't be addressed in this forum. ;) I have seen Lance's question and some similar but none that I have found seem to address this. (Disclaimer: NO I haven't looked at every ksh question in this forum)
Here's a very cut down piece of code that replicates the problem:
1 #!/usr/bin/ksh
2 #
3 go=1
4 set -x
5 tail -0f loop-test.txt | while [[ $go -eq 1 ]]
6 do
7 read lbuff
8 set $lbuff
9 nwords=$#
10 printf "Line has %d words <%s>\n" $nwords "${lbuff}"
11 if [[ "${lbuff}" = "0" ]]
12 then
13 printf "Line consists of %s; time to absquatulate\n" $lbuff
14 go=0 # Violate the WHILE condition to get out of loop
15 fi
16 done
17 printf "\nLooks like I've fallen out of the loop\n"
18 exit 0
The way I test this is:
Run loop-test.sh in background mode
In a different window I run commands like "echo some nonsense >>loop_test.txt" (w/o the quotes, of course)
When I wish to exit, I type "echo 0 >>loop-test.txt"
What happens? It indeed sets go=0 and displays the line:
Line consists of 0; time to absquatulate
but does not exit the loop. To break out I append one more line to the txt file. The loop does NOT process that line and just falls out of the loop, issuing that "fallen out" message before exiting.
What's going on with this? I don't want to use "break" because in the actual script, the loop is monitoring the log of a database engine and the flag is set when it sees messages that the engine is shutting down. The actual script must still process those final lines before exiting.
Open to ideas, anyone?
Thanks much!
-- J.
OK, that flopped pretty quick. After reading a few other posts, I found an answer given by dogbane that sidesteps my entire pipe-to-while scheme. His is the second answer to a question (from 2013) where I see neeraj is using the same scheme I'm using.
What was wrong? The pipe-to-while has always worked for input that will end, like a file or a command with a distinct end to its output. However, from a tail command, there is no distinct EOF. Hence, the while-in-a-subshell doesn't know when to terminate.
Dogbane's solution: Don't use a pipe. Applying his logic to my situation, the basic loop is:
while read line
do
# put loop body here
done < <(tail -0f ${logfile})
No subshell, no problem.
Caveat about that syntax: There must be a space between the two < operators; otherwise it looks like a HEREIS document with bad syntax.
Er, one more catch: The syntax did not work in ksh, not even in the mksh (under cygwin) which emulates ksh93. But it did work in bash. So my boss is gonna have a good laugh at me, 'cause he knows I dislike bash.
So thanks MUCH, dogbane.
-- J
After articulating the problem and sleeping on it, the reason for the described behavior came to me: After setting go=0, the control flow of the loop still depends on another line of data coming in from STDIN via that pipe.
And now that I have realized the cause of the weirdness, I can speculate on an alternative way of reading from the stream. For the moment I am thinking of the following solution:
Open the input file as STDIN (Need to research the exec syntax for that)
When the condition occurs, close STDIN (Again, need to research the syntax for that)
It should then be safe to use the more intuitive:while read lbuffat the top of the loop.
I'll test this out today and post the result. I'd hope someone else benefit from the method (if it works).
Is it possible to print something in the screen and, at the same time, that what is being printed is also written in a file?
Right now, I have something like this:
print *, root1, root2
open(unit=10,file='result.txt'
write(10,*), root1, root2
close(10)
I feel like I'm wasting lines and making the code longer that it should be. I actually want to print/write much more lines that these, so that's why I'm looking for a cleaner way to do it.
Writing to standard output and writing to file are two different things, so you will always need separate instructions. But you don't have to open and close the file for every line you write.
Honestly, I don't think it's that much more of an effort:
open(unit=10, file='result.txt', status='replace', form='formatted')
....
write( *, *) "Here comes the data"
write(10, *) "Here comes the data"
....
write( *, *) root1, root2
write(10, *) root1, root2
....
close(10)
this is only one line more than what you would have to do anyway per write statement.
If you really think that you have too many write statements in your code, here are a few ideas that you might try:
If you are running on a Linux or Unix system (including MacOS), you can write a program that only writes to standard out, and pipe the output into a file, like this:
$ ./my_program | tee result.txt
This will both output the data to the screen, and write it into the file result.txt
Or you could write the output to a file in the program, and 'follow' the file externally:
$ ./my_program &
$ tail -f result.txt
I just had another idea: If you really often have the issue that you need to write data to both the screen and the file, you can place that into a subroutine:
program my_program
implicit none
real :: root1, root2, root3
....
open(10, 'result.txt', status='replace', form='formatted')
....
call write_output((/ root1, root2 /))
....
call write_output((/ root1, root2, root3 /))
....
call write_output((/ root1, root2 /))
....
close(10)
....
contains
subroutine write_output(a)
real, dimension(:), intent(in) :: a
write( *, *) a
write(10, *) a
end subroutine write_output
end program my_program
I am passing the values to be written here as an array because that gives you more flexibility in the number of variables that you might want to print. On the other hand, you can only use this subroutine to write real values, for others (integer, character, etc) or combinations thereof you'd need to still have two write statements, or write other specific 'write to both' routines.
I am new to DCL.
I want to get the out put of a command in a variable and iterate result one by one.
filePath=dir /since="time_now" [.SUBDIR]*.PNG/noheader/notrail
That's just not how we roll with DCL.
We don't do pipes, we do, but not really.
DIR/SINCE=NOW ... will not give anything by definition, since nothing exists since now.
Use /OUT to stick the directory output into a file, and then read ans parse (F$PARSE and/or F$ELEMENT and/or F$LOC)
Check out HELP OPEN; HELP READ [/END]; HELP LEXICAL
Google for examples.
More advanced DCL scripts use F$PARSE, F$SEARCH and F$FILE(file,CDT) to avoid activating images and creating temp files: $ HELP LEXICAL
Google for examples.
Check out yesterday stack-exhange entry ?! : OpenVMS - DELETE Line if TEXT like x
But if you are just starting... IMHO just skip DCL and stick to PERL
$ perl -e "for (<[.SUBDIR]*.PNG>) { next unless -M > 0.123; print; ... }"
Good luck!
Hein
top:
file = f$search("[.subdir]*.PNG")
if (file .eqs. "")then goto cont
mtime=f$file_attribute(file,"RDT")
if mtime.ges.build_start_time then -
name=f>parse(file,,,"NAME")
call CHECK "''name'"
goto top
cont:
#Hein please review this code and suggest changes
I'm using
while( (getline line < "filename") > 0 )
within my BEGIN statement, but this while loop only seems to read the last line of the file instead of each line. I think it may be a newline character problem, but really I don't know. Any ideas?
I'm trying to read the data in from a file other than the main input file.
The same syntax actually works for one file, but not another, and the only difference I see is that the one for which it DOES work has "^M" at the end of each line when I look at it in Vim, and the one for which it DOESN'T work doesn't have ^M. But this seems like an odd problem to be having on my (UNIX based) Mac.
I wish I understood what was going with getline a lot better than I do.
You would have to specify RS to something more vague.
Here is a ugly hack to get things working
RS="[\x0d\x0a\x0d]"
Now, this may require some explanation.
Diffrent systems use difrent ways to handle change of line.
Read http://en.wikipedia.org/wiki/Carriage_return and http://en.wikipedia.org/wiki/Newline if you are
interested in it.
Normally awk hadles this gracefully, but it appears that in your enviroment, some files are being naughty.
0x0d or 0x0a or 0x0d 0x0a (CR+LF) should be there, but not mixed.
So lets try a example of a mixed data stream
$ echo -e "foo\x0d\x0abar\x0d\x0adoe\x0arar\x0azoe\x0dqwe\x0dtry" |awk 'BEGIN{while((getline r )>0){print "r=["r"]";}}'
Result:
r=[foo]
r=[bar]
r=[doe]
r=[rar]
try]oe
We can see that the last lines are lost.
Now using the ugly hack to RS
$ echo -e "foo\x0d\x0abar\x0d\x0adoe\x0arar\x0azoe\x0dqwe\x0dtry" |awk 'BEGIN{RS="[\x0d\x0a\x0d]";while((getline r )>0){print "r=["r"]";}}'
Result:
r=[foo]
r=[bar]
r=[doe]
r=[rar]
r=[zoe]
r=[qwe]
r=[try]
We can see every line is obtained, reguardless of the 0x0d 0x0a junk :-)
Maybe you should preprocess your input file with for example dos2unix (http://sourceforge.net/projects/dos2unix/) utility?