Lua script via SSH with file io operations - ssh

I am trying to run a script remotely (remote is server B). When I run the script locally on server A I get the expected results. When I run the script via SSH from server A to server B, I get the following error:
/usr/bin/ssh admin#server.domain.com "lua/rpi.init"
lua: lua/rpi.init:8: attempt to index upvalue 'logFile' (a nil value) stack traceback:
lua/rpi.init:8: in function 'logMsg'
lua/rpi.init:47: in main chunk
#!/usr/bin/env lua
local f = assert(io.popen("sudo netstat -a | grep ^tcp[^6] | grep LISTEN | grep [^0-9]22[0-9][0-9]", 'r'))
local ports22 = {}
local logFile
function logMsg(msg)
logFile = io.open("logs/pi.init.log", "a+")
logFile:write(os.date("%b %d %Y %X ") .. tostring(msg) .. "\n")
logFile:close()
end
function getPorts22()
logMsg("Getting available ports...")
while true do
line = f:read()
if line == nil then break end
port = string.sub(line, 40, 44)
table.insert(ports22, port)
end
f:close()
table.sort(ports22)
end
function getNextOpenPort22()
local openPort = 2222
if #ports22 == 0 then
logMsg("Returning port :" .. openPort)
return openPort
end
for i=1, #ports22 + 1 do
if tonumber(ports22[i]) == openPort then
openPort = openPort + 1
else
logMsg("Returning port: " .. openPort)
return openPort
end
end
end
function printPorts()
msg = table.concat(ports22, ", ")
logMsg("Found ports in use: " .. tostring(msg))
end
logMsg("Script called to run.")
getPorts22()
printPorts()
print(getNextOpenPort22())
Is it possible to run scripts via SSH (bash, lua or otherwise) and have them perform io operations on the remote machine?
Also, what is the best way to return values from the remote script to the local host? I return the value from my script by calling print() in order to have something for my local script to actual use.

You get this error because io.open calls returns a nil value and an error message. You may need to change this line logFile = io.open("logs/pi.init.log", "a+") to something like the following:
local logFile, err = io.open("logs/pi.init.log", "a+")
if not logFile then error(err, 2) end
This will print the error message when io.open call is not successful. In your case it may be caused by non-existent logs folder, insufficient permissions, or something else; the error message should point you in the right direction.

Related

channel checks as empty even if it has content

I am trying to have a process that is launched only if a combination of conditions is met, but when checking if a channel has a path to a file, it always returns it as empty. Probably I am doing something wrong, in that case please correct my code. I tried to follow some of the suggestions in this issue but no success.
Consider the following minimal example:
process one {
output:
file("test.txt") into _chProcessTwo
script:
"""
echo "Hello world" > "test.txt"
"""
}
// making a copy so I check first if something in the channel or not
// avoids raising exception of MultipleInputChannel
_chProcessTwo.into{
_chProcessTwoView;
_chProcessTwoCheck;
_chProcessTwoUse
}
//print contents of channel
println "Channel contents: " + _chProcessTwoView.toList().view()
process two {
input:
file(myInput) from _chProcessTwoUse
when:
(!_chProcessTwoCheck.toList().isEmpty())
script:
def test = _chProcessTwoUse.toList().isEmpty() ? "I'm empty" : "I'm NOT empty"
println "The outcome is: " + test
}
I want to have process two run if and only if there is a file in the _chProcessTwo channel.
If I run the above code I obtain:
marius#dev:~/pipeline$ ./bin/nextflow run test.nf
N E X T F L O W ~ version 19.09.0-edge
Launching `test.nf` [infallible_gutenberg] - revision: 9f57464dc1
[c8/bf38f5] process > one [100%] 1 of 1 ✔
[- ] process > two -
[/home/marius/pipeline/work/c8/bf38f595d759686a497bb4a49e9778/test.txt]
where the last line are actually the contents of _chProcessTwoView
If I remove the when directive from the second process I get:
marius#mg-dev:~/pipeline$ ./bin/nextflow run test.nf
N E X T F L O W ~ version 19.09.0-edge
Launching `test.nf` [modest_descartes] - revision: 5b2bbfea6a
[57/1b7b97] process > one [100%] 1 of 1 ✔
[a9/e4b82d] process > two [100%] 1 of 1 ✔
[/home/marius/pipeline/work/57/1b7b979933ca9e936a3c0bb640c37e/test.txt]
with the contents of the second worker .command.log file being: The outcome is: I'm empty
I tried also without toList()
What am I doing wrong? Thank you in advance
Update: a workaround would be to check _chProcessTwoUse.view() != "" but that is pretty dirty
Update 2 as required by #Steve, I've updated the code to reflect a bit more the actual conditions i have in my own pipeline:
def runProcessOne = true
process one {
when:
runProcessOne
output:
file("inputProcessTwo.txt") into _chProcessTwo optional true
file("inputProcessThree.txt") into _chProcessThree optional true
script:
// this would replace the probability that output is not created
def outputSomething = false
"""
if ${outputSomething}; then
echo "Hello world" > "inputProcessTwo.txt"
echo "Goodbye world" > "inputProcessThree.txt"
else
echo "Sorry. Process one did not write to file."
fi
"""
}
// making a copy so I check first if something in the channel or not
// avoids raising exception of MultipleInputChannel
_chProcessTwo.into{
_chProcessTwoView;
_chProcessTwoCheck;
_chProcessTwoUse
}
//print contents of channel
println "Channel contents: " + _chProcessTwoView.view()
println _chProcessTwoView.view() ? "Me empty" : "NOT empty"
process two {
input:
file(myInput) from _chProcessTwoUse
when:
(runProcessOne)
script:
"""
echo "The outcome is: ${myInput}"
"""
}
process three {
input:
file(defaultInput) from _chUpstreamProcesses
file(inputFromProcessTwo) from _chProcessThree
script:
def extra_parameters = _chProcessThree.isEmpty() ? "" : "--extra-input " + inputFromProcessTwo
"""
echo "Hooray! We got: ${extra_parameters}"
"""
}
As #Steve mentioned, I should not even check if a channel is empty, NextFlow should know better to not initiate the process. But I think in this construct I will have to.
Marius
I think part of the problem here is that process 'one' creates only optional outputs. This makes dealing with the optional inputs in process 'three' a bit tricky. I would try to reconcile this if possible. If this can't be reconciled, then you'll need to deal with the optional inputs in process 'three'. To do this, you'll basically need to create a dummy file, pass it into the channel using the ifEmpty operator, then use the name of the dummy file to check whether or not to prepend the argument's prefix. It's a bit of a hack, but it works pretty well.
The first step is to actually create the dummy file. I like shareable pipelines, so I would just create this in your baseDir, perhaps under a folder called 'assets':
mkdir assets
touch assets/NO_FILE
Then pass in your dummy file if your '_chProcessThree' channel is empty:
params.dummy_file = "${baseDir}/assets/NO_FILE"
dummy_file = file(params.dummy_file)
process three {
input:
file(defaultInput) from _chUpstreamProcesses
file(optfile) from _chProcessThree.ifEmpty(dummy_file)
script:
def extra_parameters = optfile.name != 'NO_FILE' ? "--extra-input ${optfile}" : ''
"""
echo "Hooray! We got: ${extra_parameters}"
"""
}
Also, these lines are problematic:
//print contents of channel
println "Channel contents: " + _chProcessTwoView.view()
println _chProcessTwoView.view() ? "Me empty" : "NOT empty"
Calling view() will emit all values from the channel to stdout. You can ignore whatever value it returns. Unless you enable DSL2, the channel will then be empty. I think what you're looking for here is a closure:
_chProcessTwoView.view { "Found: $it" }
Be sure to append -ansi-log false to your nextflow run command so the output doesn't get clobbered. HTH.

Cannot use threads to insert data to PostgreSQL with DBIish. What's going wrong?

Edit: This was solved by moritz. I've added a note to the code on the line that's wrong.
My application is a web server talking to a game client. The server is multithreaded, which Postgres allows. When loading client data into the database, I noticed parallel requests fail with several different errors, none of which make sense to me.
This short-ish test case dumps a nested hash into the database. When run without start, it works perfectly. When run with threads, it almost always gives one or more of the following errors:
DBDish::Pg: Error: (7) in method prepare at
D:\rakudo\share\perl6\site\sources\BAD7C1548F63C7AA7BC86BEDDA0F7BD185E141AD
(DBDish::Pg::Connection) line 48 in block at testcase.p6 line 62
in sub add-enum-mappings at testcase.p6 line 59 in block at
testcase.p6 line 91
DBDish::Pg: Error: ERROR: prepared statement
"pg_3448_16" already exists (7) in method prepare at
D:\rakudo\share\perl6\site\sources\BAD7C1548F63C7AA7BC86BEDDA0F7BD185E141AD
(DBDish::Pg::Connection) line 46 in block at testcase.p6 line 62
in sub add-enum-mappings at testcase.p6 line 59 in block at
testcase.p6 line 91
DBDish::Pg: Error: Wrong number of arguments to
method execute: got 1, expected 0 (-1) in method enter-execute at
D:\rakudo\share\perl6\site\sources\65FFB78EFA3030486D1C4D339882A410E3C94AD2
(DBDish::StatementHandle) line 40 in method execute at
D:\rakudo\share\perl6\site\sources\B3190B6E6B1AA764F7521B490408245094C6AA87
(DBDish::Pg::StatementHandle) line 52 in sub add-enum-mappings at
testcase.p6 line 54 in block at testcase.p6 line 90
message type 0x31 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x74 arrived from server while idle
message type 0x6e arrived from server while idle
message type 0x5a arrived from server while idle
Here's the code. (If you choose to run it, remember to set the right password. It creates/manipulates a table called "enummappings", but does nothing else.) The meat is in add-enum-mappings(). Everything else is just setup. Oh, and dbh() creates a separate DB connection for each thread. This is necessary, according to the PostgreSQL docs.
#!/usr/bin/env perl6
use DBIish;
use Log::Async;
my Lock $db-lock;
my Lock $deletion-lock;
my Lock $insertion-lock;
INIT {
logger.send-to($*ERR);
$db-lock .= new;
$deletion-lock .= new;
$insertion-lock .= new;
}
# Get a per-thread database connection.
sub dbh() {
state %connections;
my $dbh := %connections<$*THREAD.id>; # THIS IS WRONG. Should be %connections{$*THREAD.id}.
$db-lock.protect: {
if !$dbh.defined {
$dbh = DBIish.connect('Pg', :host<127.0.0.1>, :port(5432), :database<postgres>,
:user<postgres>, :password<PASSWORD>);
}
};
return $dbh;
}
sub create-table() {
my $name = 'enummappings';
my $column-spec =
'enumname TEXT NOT NULL, name TEXT NOT NULL, value INTEGER NOT NULL, UNIQUE(enumname, name)';
my $version = 1;
my $sth = dbh.prepare("CREATE TABLE IF NOT EXISTS $name ($column-spec);");
$sth.execute;
# And add the version number to a version table:
dbh.execute:
"CREATE TABLE IF NOT EXISTS tableversions (name TEXT NOT NULL UNIQUE, version INTEGER NOT NULL);";
$sth = dbh.prepare:
'INSERT INTO tableversions (name, version) VALUES (?, ?)
ON CONFLICT (name)
DO
UPDATE SET version = ?;';
$sth.execute($name, $version, $version);
}
sub add-enum-mappings($enumname, #names, #values --> Hash) {
$deletion-lock.protect: {
my $sth = dbh.prepare('DELETE FROM enummappings WHERE enumname = ?;');
$sth.execute($enumname);
};
my #rows = (^#names).map: -> $i {$enumname, #names[$i], #values[$i]};
info "Inserting #rows.elems() rows...";
$insertion-lock.protect: {
my $sth = dbh.prepare('INSERT INTO enummappings (enumname,name,value) VALUES '~
('(?,?,?)' xx #rows.elems).join(',') ~ ';');
$sth.execute(#rows>>.list.flat);
};
return %(status => 'okay');
}
# Create a bunch of long enums with random names, keys, and values.
sub create-enums(--> Hash[Hash]) {
my #letters = ('a'..'z', 'A'..'Z').flat;
my Hash %enums = ();
for ^36 {
my $key = #letters.pick(10).join;
for ^45 {
my $sub-key = #letters.pick(24).join;
%enums{$key}{$sub-key} = (0..10).pick;
}
}
return %enums;
}
sub MAIN() {
create-table;
await do for create-enums.kv -> $enum-name, %enum {
start {
add-enum-mappings($enum-name, %enum.keys, %enum.values);
CATCH { default { note "Got error adding enum: " ~ .gist; } }
};
}
}
I'm on Windows 10, with a 8-core computer. I know I could insert the data single-threadedly, but what if the game gets a hundred connections at once? I need to fix this for good.
I suspect your problem is here:
my $dbh := %connections<$*THREAD.id>;
The %hash<...> syntax is only for literals. You really need to write %connections{$*THREAD.id}.
With your error in place, you have just one DB connection that's shared between all threads, and I guess that's what DBIish (or the underlying postgresql C client library) is unhappy about.

lua variable type nil even though i just assigned it

I have a part of a lua script here:
local cmp = require("component")
local r = cmp.br_reactor
local e = require("event")
local unicode = require ("unicode")
local exit = false
local buffersize = 10000000
local last_tick_percent = 1
print(type(last_tick_percent))
function stored_energy()
local rf_stored = r.getEnergyStored()
local rf_percent = (rf_stored/buffersize)*100
print(type(rf_precent))
print(type(last_tick_percent))
local delta_percent = rf_percent - last_tick_percent
last_tick_percent = re_percent
return rf_percent.."% ["..rf_stored.."] | "..unicode.char(916)..": "..delta_percent.."%"
end
The first print is not even executed for some reason. Inside the function, the first print returns Number while the second print returns nil.
Now I am getting the error attempt to perform arithmetic on upvalue "last_tick_percent" ( a nil value), obviously because last_tick_percent is nil which the print(type(..)) showed.
But i just assigned it literally 5 lines above.
So the questions are:
Why is last_tick_percent nil and how can i fix that?
why is the first print not executed?
you are assigning re_percent which is not declared in your script to last_tick_percent inside stored_energy. i am assuming you meant to assign rf_percent.

How and when and where jvm change the max open files value of Linux?

In linux there is a limit for max open files for every process of each login user, as below:
$ ulimit -n
1024
When I study java nio, I'd like to check this value. Because channel also is a file in Linux,I wrote a client code to create socketChannel continuely until throwing below exception:
java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:423)
at sun.nio.ch.Net.socket(Net.java:416)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:104)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
But I found it till created about 4085 socketChannel, it will throw this exception. This number is more than 1024. Somebody told me jvm changed the value implicitly. And I wrote a java program to execute ulimit command, and found jvm do change the value. As below:
String [] cmdArray = {"sh","-c","ulimit -n"};
Process p = Runtime.getRuntime().exec(cmdArray);
BufferedInputStream in = new BufferedInputStream(p.getInputStream());
byte[] buf = new byte[1024];
int len = in.read(buf);
System.out.println(new String(buf, 0, len)); //4096
Does anybody know when and where and how jvm changes this value? Does exist some sys log to record this change or some sys tool could monitor this change?
$ strace -f -o HelloWorld.strace java HelloWorld
Hello World!
$ vi HelloWorld.strace
...
16341 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
16341 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0
...
Download openjdk, then cd into hotspot dir,
$ grep -r setrlimit
...
src/os/linux/vm/os_linux.cpp: status = setrlimit(RLIMIT_NOFILE, &nbr_files);
...
$ vi src/os/linux/vm/os_linux.cpp
...
if (MaxFDLimit) {
// set the number of file descriptors to max. print out error
// if getrlimit/setrlimit fails but continue regardless.
struct rlimit nbr_files;
int status = getrlimit(RLIMIT_NOFILE, &nbr_files);
if (status != 0) {
if (PrintMiscellaneous && (Verbose || WizardMode))
perror("os::init_2 getrlimit failed");
} else {
nbr_files.rlim_cur = nbr_files.rlim_max;
status = setrlimit(RLIMIT_NOFILE, &nbr_files);
if (status != 0) {
if (PrintMiscellaneous && (Verbose || WizardMode))
perror("os::init_2 setrlimit failed");
}
}
...
If you modify above code, e.g.
//nbr_files.rlim_cur = nbr_files.rlim_max;
nbr_files.rlim_cur = 2048;
then rebuild this openjdk, then use this new jdk to execute above code, you'll find the output is 2048.

Capture Variable Changes In Lua

I'm unsure where to look as far as what I could use to capture a variable change in a program loaded within another program.
Here's my code, as messy as it is:
function launch()
shell.run("clear")
print("Preparing for first time run.")
sleep(1)
print("")
local program = netTest()
local file = loadstring(program)
file()
sleep(3)
shell.run("clear")
end
function netTest()
local output = http.get("http://pastebin.com/raw.php?i=hzZv3YH2")
if output then
local contents = output.readAll()
output.close()
return contents
else
print("Empty response")
return false
end
end
local program = netTest()
local file = loadstring(program)
launch()
Here's the code it's calling on:
function ping()
fails = 0
pingResult = 0
print("Testing Internet Connection.")
print("")
oldx, oldy = term.getCursorPos()
print("Connection Test 1")
http.request("http://www.google.com/")
if os.pullEvent() == "http_success" then
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Passed")
else
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Failed")
fails = fails+1
end
sleep(1)
print("Connection Test 2")
http.request("http://www.microsoft.com/")
if os.pullEvent() == "http_success" then
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Passed")
else
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Failed")
fails = fails+1
end
sleep(1)
print("Connection Test 3")
http.request("http://www.example-failure.com/")
if os.pullEvent() == "http_success" then
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Passed")
else
local oldx, oldy = term.getCursorPos()
term.setCursorPos(46,oldy-1)
io.write("Failed")
fails = fails+1
end
sleep(1)
if fails == 0 then
print("")
print("")
print("Test Complete, no failures detected.")
sleep(1.5)
elseif fails == 1 then
print("")
print("")
print("1 connection failures detected. A Website Host might be down however connectivity is still there.")
print("")
print("Test Complete.")
sleep(1.5)
elseif fails == 2 then
print("")
print("")
print("2 connection failures detected. Possible limited web connectivity.")
print("")
print("Test Complete.")
sleep(1.5)
elseif fails == 3 then
print("")
print("")
print("Catastrophic connection failure detected. A firewall or improper internet settings may be the problem.")
print("")
print("Test Complete.")
pingResult = __pingResult + 3
sleep(1.5)
end
end
ping()
What it's doing as you can see is running a program externally which will test the connection by making http requests to a couple pages to make sure there's connection to the internet. (I know, it's a bit lame, but I'm still learning).
Basically when and if the connection reads failure on all 3 stages, it'll make my pingResult variable = 3. What I'm wanting to do from the first program that called on my internet utility is record what that variable is set to. If it's set to 3 upon closing, to record that variable being set, and then for simplicity's sake, print that variable so I can see it's either 0 or 3. I'll be doing something with it later but you get the gist of it.
I've tried a couple other things but can't seem to figure out on how I go about doing this. Seeing as I'm still new I tried random stuff but none seemed to work, or I just did them wrong so I don't know what to do with it. Been trying at this one for a couple days to no success.
First of all, most of the code you posted is noise -- print statements, cursor manipulation, fetching Lua code from pastebin (?), all kinds of logic which has absolutely nothing to do with your question, including code which literally does nothing. That's why you haven't got a response yet.
See: How to Ask Questions the Smart Way: Volume is Not Precision.
If we strip all the extraneous stuff from your post, we're left with:
-- main.lua
local file = loadfile('nettest.lua')
file()
-- nettest.lua
pingResult = 123
Now to address your questions:
What it's doing as you can see is running a program externally:
It's not running anything externally. You've grabbed external code, but it's executed locally. Any changes that code makes to the global state is visible to you. In this case, main.lua has access to pingResult. For instance, you can write:
-- main.lua
local file = loadfile('nettest.lua')
file()
print(pingResult)
Of course, if you want a cleaner separation between your script and ping module, you should probably have that code return a value rather than write to a global:
-- main.lua
local file = loadfile('nettest.lua')
local pingResult = file()
print(pingResult)
-- nettest.lua
local pingResult
-- ... code the computes the result an stores it in pingResult ...
return pingResult