I can't get the output of a script run through singularity.
I have a python script, at the end of which the output is saved with:
...
with open('saveOut.pkl','wb') as myFile:
pickle.dump(myTable,myFile)
I want to run this script with singularity on a distant machine. Since I am learning singularity, I made a 'sand box' debian image (not compiled into a single 'img' file yet) in the directory /tmp/debian; in this image I copied the python script test.py in /usr/src and I run it with the command:
sudo singularity exec /tmp/debian python3.5 /usr/src/test.py
The problem:
It works well as long as I have only displayed results. with the pickle example described above, I don't get any saveOut.pkl file anywhere: this file is just not written anywhere but I don't see any message. I tried to write an explicit path in the python script. For instance /usr/src/saveOut.pkl, but this is the same.
How could I write a result ?
What was your expected result i.e. in which directory did you expect
to find the output file?
I expect a file saveOutput.pkl anywhere, in the container or not, I don't care the location. Currently I don't get it at all: neither in the container's current directory, nor in the container's /usr/src/, nor on the host, nor anywhere.
Did you look for it on the host or in the container?
both, I don't see it anywhere
What's happening here is that your python script is writing the pickle file to its current location (/usr/src/ in the container). Then, since the output from your script is not persistent (due to the sandbox not being writable on execution), it gets deleted at the end of the run.
I believe you could change your script:
with open('/opt/saveOut.pkl','wb') as myFile:
pickle.dump(myTable,myFile)
and then bind the local directory and get the output you're looking for:
sudo singularity exec -B ./:/opt /tmp/debian python3.5 /usr/src/test.py
This worked for me, anyway.
Related
This might be a silly question, but I didn't find anything on the internet. I am trying to run the object_detection API of tensorflow in Windows 10. I've installed all the packages correctly and I've also added the "models" directory to the PYTHONPATH, using windows cmd:
set PYTHONPATH=C:\Program Files\Tensorflow\models
And I also verify that the PYTHONPATH is set correctly, by typing set PYTHONPATH, which gives me this output:
PYTHONPATH=C:\Program Files\Tensorflow\models
Running the following test
python research/object_detection/builders/model_builder_tf2_test.py
I get the output:
Ran 20 tests in 37.331s
OK (Skipped 1)
Which is great. However, If I open another terminal and type
python research/object_detection/builders/model_builder_tf2_test.py
I get the error: ModuleNotFoundError: No Module named official
However, If i type set PYTHONPATH command to verify that the PYTHONPATH is correct I get:
PYTHONPATH=C:\Users\Kohli\AppData\Local\Programs\Python\Python37\Scripts\Tensorflow\models;C:\Users\Kohli\AppData\Local\Programs\Python\Python37\Scripts\Tensorflow\models\official;C:\Users\Kohli\AppData\Local\Programs\Python\Python37\Scripts\Tensorflow\models\research
which is different from the previous terminal output. Moreover, when I open the "Enviroment Variables" tab, I can see that PYTHONPATH exists.
If I add the models directory to PYTHONPATH once again, It will work, but Only for that particular terminal. However, I don't understand why this is happening. Does the "set PYTHONPATH" command creates a virtual environment for that particular terminal or something? How can i permanently add models dir to PYTHONPATH, without typing set PYTHONPATH or os.path.append('path/to/models') ?
Thanks in advance
I have the problem that I want to enable logging of a screen session at the start of it which then saves the log to a specific file.
What I have until now was:
screen -AmdSL cod2war /home/cod2server/scripts/service_28969.sh
while service_28969.sh is a shell script that will call other scripts which produce output.
I started multiple of those screen-sessions with different names, for example
screen -AmdSL cod2sd /home/cod2server/scripts/service_28962.sh
-L enables logging as the screen's man say, and will safe the ouput in a file called 'screenlog.0', now since I have multiple of those screens only one of it produces output saved in that log file (I can't find other 'screenlog.*' files in that folder).
I thought to use the -Logfile "file" option from the same man page, but it doesn't work for me and I can't find out what I'm doing wrong..
screen -Logfile cod2sd.log -AmdS cod2sd /home/u268450/cod2server/scripts/service_28962.sh
will produce the following error:
Use: screen [-opts] [cmd [args]]
or: screen -r [host.tty]
Options:
[...]
Error: Unknown option Logfile
and
screen -AmdS cod2sd /home/u268450/cod2server/scripts/service_28962.sh -Logfile cod2sd.log
will run without any error and start the screen but without the logging at all..
You can specify a logfile from within the default startup ~/.screenrc file using a line like
logfile mylog.log
To do this from the command line you can create a file mystartup to hold the above line, then use option -c mystartup to tell screen to read this file for setup instead of the default. If you also need to have ~/.screenrc read, you can add the source command to your startup file. The final result would look something like:
echo 'logfile mylog.log
source ~/.screenrc' >mystartup
screen -AmdSL cod2war -c mystartup /home/cod2server/scripts/service_28969.sh
This works for me:
screen -L -Logfile /Logs/Screen/`date +%Y%m%d`_screen.log
The configs I checked:
screen version 4.08.00 (GNU) 05-Feb-20 on FreeBSD 12.2
and
version 4.06.02 (GNU) 23-Oct-17 on Debian GNU/Linux 10 (buster)
and
version 4.00.03 (FAU) 23-Oct-06 on Mac OS X 10.9.5.
I just ran into this error myself and found this solution that worked with my python file, wanted to share for anyone else who might run into this issue:
screen -L -Logfile LOGFILENAME.LOG -dmS SCREENNAME python3 ./FILENAME.PY
I have no idea if this is the 'correct' way but it works.
-L enables logging
-Logfile LOGFILENAME.LOG declares what to call the log file and file format
-dmS SCREENNAME, dm runs in detached mode and S allows you to name the session
python3 ./FILENAME.PY in this case is my script but I assume that any other script here functions
I have tried a different ordering of these commands and this was the only way I managed to have them all run without issues. Hopes this helps.
I am trying to run an application which uses pagemap in gem5 FS mode.
But I am not able to use pagemap in gem5. It throws below error -
"assert(pagemap>=0) failed"
The line of code is:
int pagemap = open("/proc/self/pagemap", O_RDONLY);
assert(pagemap >= 0);
Also, If I try to run my application on gem5 terminal with sudo ,it throws error-
sudo command not found
How can I use sudo in gem5 ??
These problems are not gem5 specific, but rather image / Linux specific, and would likely happen on any simulator or real hardware. So I recommend that you remove gem5 from the equation completely, and ask a Linux or image specific question next time, saying exactly what image your are using, kernel configs, and provide a minimal C example that reproduces the problem: this will greatly improve the probability that you will get help.
I have just done open("/proc/self/pagemap", O_RDONLY) successfully with: this program and on this fs.py setup on aarch64, see also these comments.
If /proc/<pid>/pagemap is not present for any file, do the following:
ensure that procfs is mounted on /proc. This is normally done with an fstab entry of type:
proc /proc proc defaults 0 0
but your init script needs to use fstab as well.
Alternatively, you can mount proc manually with:
mount -t proc proc proc/
you will likely want to ensure that /sys and /dev are mounted as well.
grep the kernel to see if there is some config controlling the file creation.
These kinds of things are often easy to find without knowing anything about the kernel.
If I do:
git grep '"pagemap'
to find the pagemap string, which is likely the creation point, on v4.18 this leads me to fs/proc/base.c, which contains:
#ifdef CONFIG_PROC_PAGE_MONITOR
REG("pagemap", S_IRUSR, proc_pagemap_operations),
#endif
so make sure CONFIG_PROC_PAGE_MONITOR is set.
sudo: most embedded / simulator images don't have it, you just login as root directly and can do anything by default without it. This can be seen by the conventional # in the prompt instead of $.
I was working with the tensorflow(GPU version) module in Pycharm. If I run a script from terminal, it works as expected. However when I run the script from pycharm, it says:
ImportError: libcudart.so.7.5: cannot open shared object file: No
such file or directory
How do I resolve this?
Pycharm interpreter shows tensorflow as a package.
In the terminal, when I check for the version of tensorflow, it was the same as in pycharm (0.10.0rc0)
Looks like your CUDA_HOME or LD_LIBRARY_PATH configured correctly in the console, but not in PyCharm. You can check and compare their values, in console do
echo $CUDA_HOME
echo $LD_LIBRARY_PATH
In PyCharm (say, in your main script):
import os
print(os.environ.get('CUDA_HOME'))
print(os.environ.get('LD_LIBRARY_PATH'))
You can configure them for the given Run Configuration in Environment Variables section.
Better approach would be configuring those environment variables globally, so every process in the system would have an access to them. To do that you have to edit /etc/environment file and add original values, which you got from console.
Here are very similar problems: one, two, three.
I'm playing in Elm, and whenever I use elm make I get an error from Windows Script Host:
The error states that there is an error in the build elm.js file. And when I look at the given line, it's the following:
return {
keys: keys, // A hash of key name to index
free: free, // An array of unkeyed item indices
} // line 10547
So I'm guessing it's bitching about the unneeded , on line 10546: free: free,.
But now my question is, why is Windows Script Host doing stuff with my build elm.js file anyway, and how can I fix it?
I already tried disabling Windows Script Host, but then I just get an error stating that it doesn't allow script to be run.
Since you have a file in that directory called elm.js, the Windows Command Line tries executing that when you type elm make. It thinks you are intending this: elm.js make.
You can get around this in a few ways:
Change the name of the output file from elm.js to something else
Move the generated elm.js file to a subdirectory
Run elm-make from the command line instead of elm make
Use another shell like Powershell, Git Bash, or Cygwin