I am receiving this error when I run my Snakefile:
Error in rule get_ha_frequency:
jobid: 0
output: freq.annotated.tsv
RuleException:
ModuleNotFoundError in line 29 of file/path/Snakefile:
No module named 'nbformat.v1.convert'
File "file/path/Snakefile", line 29, in __rule_get_ha_frequency
File "/home/users/USER/mambaforge/envs/snakemake/lib/python3.9/site-packages/nbformat/__init__.py", line 13, in <module>
File "/home/users/USER/mambaforge/envs/snakemake/lib/python3.9/site-packages/nbformat/v1/__init__.py", line 23, in <module>
File "/home/users/USER/mambaforge/envs/snakemake/lib/python3.9/concurrent/futures/thread.py", line 52, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
The Snakefile works without this rule. I haven't needed this module for any of my other rules and the script associated with this rule doesn't seem to be different than the other rules that are working (i.e. same input file type, same output file type, only module used is pandas). Why is nbformat necessary for this rule, but not the others? Is there a way to circumvent installing nbformat?
I'm not sure how, but something became messed up in the snakemake env and reinstalling snakemake helped fix this issue. I didn't even have to delete snakemake to reinstall. I just followed the Full Installation instructions here and it gave me the option to replace my snakemake env with the fresh install.
Related
I have a workflow that runs correctly on my own machine, but fails when being submitted to a cluster. The error seems to be a shell problem on the jobscript written where the environment variables are set:
/nfs/mypath/test/.snakemake/tmp.m80omc9q/snakejob.retrieve_data.3.sh:
line 3: NEW_ACCESSION='E-CORN-1' ACCESSIONS='E-MTAB-4395,E-MTAB-4342,E-MTAB-4128,E-MTAB-3826,E-MTAB-3173,E-MTAB-964,E-GEOD-62778,E-GEOD-54272'
COVARIATE_TYPE='characteristic'
BATCH='study' COVARIATE='organism part'
CACHE_PATH='/other-path/baseline-merge-cache': No such file or directory
however, because the jobscript gets deleted, I cannot check the shell script to understand the issue. Is there any way of keeping this file after snakemake finishes/fails for inspection?
I've installed the locusts-plugins along with the other required modules as shown in the setup.py:
"locust>=1.4.3",
"psycogreen",
"psycopg2-binary",
"websocket-client",
"python-dateutil",
"pymongo",
"confluent-kafka",
"selenium>=4.0.0.b2",
"lxml",
"opencensus-ext-azure"
But when I attempt to run any of the examples using the webdrivers I get the error:
File "c:\users\mjohns33\pycharmprojects\pythonproject\venv\lib\site-packages\gevent\subprocess.py", line 1270, in _execute_child
hp, ht, pid, tid = CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
2021-04-12T20:11:44Z <Greenlet at 0x55012d0: > failed with FileNotFoundError
I have the standalone server running and chromedriver in the path, is there something I'm missing?
Thanks for the full callstack. The command that it is complaining about is killall, which is not available on Windows.
I've only ever run WebdriverUser on Mac and Linux. It might work if you just comment out that line (c:\users\mjohns33\pycharmprojects\pythonproject\venv\lib\site-packages\locust_plugins\users\webdriver.py, line 127)
(and manually kill chromedriver as needed)
I figured out that the locust-plugins is all-inclusive, meaning you don't want to install selenium or any other modules. Python, locust and locust-plugins, that's it.
I still see new errors but just having what's needed clears up a lot of these issues.
I've already got Gem5 installed.
I'm trying to do a full system simulation.
I added M5_PATH
echo "export M5_PATH==/home/sam/security/gem5/full/" >> ~/.bashrc
I put all the system image and config files under following directory:
:~/security/gem5/full$ ls
binaries configs config-x86.tar.bz2 disks x86-system.tar.bz2 x86-system.tar.bz2.1
I changed the path in the SysPaths.py file to following:
paths = [ '/dist/m5/system', 'full' ]
And updated the following line in Benchmark.py
return env.get('LINUX_IMAGE', disk('linux-x86.img'))
Gem5 is compiled successfully and I'm running the following:
./build/X86/gem5.opt configs/example/fs.py --disk-image=/home/sam/security/gem5/full/disks/linux-x86.img
But I get an error that it can't find a path to system files.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "build/X86/python/m5/main.py", line 438, in main
exec(filecode, scope)
File "configs/example/fs.py", line 335, in <module>
test_sys = build_test_system(np)
File "configs/example/fs.py", line 93, in build_test_system
cmdline=cmdline)
File "/home/sam/security/gem5/configs/common/FSConfig.py", line 614, in makeLinuxX86System
makeX86System(mem_mode, numCPUs, mdesc, self, Ruby)
File "/home/sam/security/gem5/configs/common/FSConfig.py", line 539, in makeX86System
disk0.childImage(mdesc.disk())
File "/home/sam/security/gem5/configs/common/Benchmarks.py", line 59, in disk
return disk(self.diskname)
File "/home/sam/security/gem5/configs/common/SysPaths.py", line 63, in __call__
raise IOError("Can't find a path to system files.")
IOError: Can't find a path to system files.
Update 2020-01
As of 82f6d6e90f36e400db1f38eef5fe17430313458e reviewed at https://gem5-review.googlesource.com/c/public/gem5/+/23672/7 the CLI insanity has reduced substantially:
M5_PATH is not required anymore on X86 if you point to all required files explicitly:
fs.py --kernel path/to/vmlinux --disk-image path/to/rootfs.ext2
The second disk named linux-bigswap2.img and x86_64-vmlinux-2.6.22.9 are not needed anymore, and you can pass multiple disks at will with multiple --disk-image options, see: How to attach multiple disk images in a simulation with gem5 fs.py?
on ARM, M5_PATH can also be dispensed, but you also need to point the bootloader with:
fs.py --bootloader ./system/arm/bootloader/arm64/boot.arm64
If you miss any of those files, M5_PATH gets used.
Note that just like the PATH search algorithm, paths without / are only searched under M5_PATH, so if you want to point to a file in the current directory you need to add ./ as in:
fs.py --kernel ./vmlinux
see also: Why do you need ./ (dot-slash) before executable or script name to run it in bash?
Old answer
gem5 is picky about some path names, but you don't need to patch it to achieve a decent image setup.
For example, this working setup with gem5 e2656006df442a995bf80ee03fa9700d6ec14537 essentially runs:
M5_PATH=/full/path/to/system \
build/X86/gem5.opt \
configs/example/fs.py \
--disk-image /any/path/to/rootfs.ext2 \
--kernel /any/path/to/vmlinux
and /full/path/to/system contains:
./disks/linux-bigswap2.img
./binaries/x86_64-vmlinux-2.6.22.9
Both of those files are dummies which I generated from here with:
dd if=/dev/zero of=./binaries/linux-bigswap2.img count=1 bs=16k
touch disks/x86_64-vmlinux-2.6.22.9
Yes, this is a horrible workaround to gem5's opinionated image searching... someone should really patch gem5 to not look for those images if you pass --disk-image and --kernel yourself...
As always, have a try at debugging it with prints and PDB: it should then be simple to figure out why something didn't work for you.
I have a Windows 10 x64 based PC. I am trying to get this PDF-DIFF Python tool here: https://github.com/JoshData/pdf-diff
to run and it hits this error when it is running a line from Python Lib file called 'subprocess.py' file.
I posted this issue with full screenshots of the error and still waiting for response:
https://github.com/JoshData/pdf-diff/issues/30
Just to note that before installation of this github tool I :
1) installed Python 3.6 (tried both 32 bit and 64) and currently have 32 bit
2) installed Microsoft Visual Studio 14.0 Build tools (C++ build tools)
3) in command prompt ran command: pip install lxml so that I could have
the xml libraries downloaded and satisfy the tool's requirements
4) additionally have checked other stackoverflow threads suggesting that 'file not found' error is due to 'COMSPEC' variable being set differently in regedit vs. Python's subprocess.py.
In windows key registry, 'ComSpec' key set to: %SystemRoot%\system32\cmd.exe;
In python subprocess call is being made as such (last line is 997):
if shell:
startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = _winapi.SW_HIDE
comspec = os.environ.get("COMSPEC", "cmd.exe")
args = '{} /c "{}"'.format (comspec, args)
#Start the process
try:
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
# no special security
None, None,
int(not close_fds),
creationflags,
env,
os.fspath(cwd) if cwd is not None else
None,
startupinfo)
I'm out of ideas, and feel the program requirements are satisfied.
So, when I run the command in the prompt to get my two PDF files compared and output comparison spit out:
C:\Python36\Scripts\pdf_diff>pdf-diff 683000.pdf 17368000.pdf > comparison_output.png
*I immediately get error shown below:
Traceback (most recent call last):
File "C:\Python36\Scripts\pdf_diff\pdf-diff-script.py", line 11, in
load_entry_point('pdf-diff==0.9.0', 'console_scripts', 'pdf-diff')()
File "C:\Python36\lib\site-packages\pdf_diff-0.9.0-py3.6.egg\pdf_diff\command_line.py", line 496, in main
changes = compute_changes(args.files[0], args.files[1], top_margin=float(args.top_margin), bottom_margin=float(args.bottom_margin))
File "C:\Python36\lib\site-packages\pdf_diff-0.9.0-py3.6.egg\pdf_diff\command_line.py", line 14, in compute_changes
docs = [serialize_pdf(0, pdf_fn_1, top_margin, bottom_margin), serialize_pdf(1, pdf_fn_2, top_margin, bottom_margin)]
File "C:\Python36\lib\site-packages\pdf_diff-0.9.0-py3.6.egg\pdf_diff\command_line.py", line 29, in serialize_pdf
for run in box_generator:
File "C:\Python36\lib\site-packages\pdf_diff-0.9.0-py3.6.egg\pdf_diff\command_line.py", line 102, in mark_eol_hyphens
for next_box in boxes:
File "C:\Python36\lib\site-packages\pdf_diff-0.9.0-py3.6.egg\pdf_diff\command_line.py", line 64, in pdf_to_bboxes
xml = subprocess.check_output(["pdftotext", "-bbox", fn, "/dev/stdout"])
File "C:\Python36\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Python36\lib\subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Python36\lib\subprocess.py", line 709, in init
restore_signals, start_new_session)
File "C:\Python36\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I believe I found out the problem here. In the process of installing different versions of Python, namely 2.7 and 3.6 I broke something. I believe it could be the importing of different modules for the PDF-diff program. The PDF-DIFF program, according to manual is coded for Python 3. So, I believe the issue above was a versioning issue. I have for now removed Python 3.6 from my machine as I need 2.7 for other projects.
Whenever I try build WebRTC for Windows, I get this error when running gclient runhooks:
________ running 'C:\path\to\depot_tools\python276_bin\python.exe src/build
/landmines.py' in 'C:\path\to\webrtc\src\chromium' Please follow the instructions
at http://www.chromium.org/developers/how-tos/bui ld-instructions-windows
Traceback (most recent call last):
File "src/build/landmines.py", line 215, in <module>
sys.exit(main())
File "src/build/landmines.py", line 202, in main
gyp_environment.SetEnvironment()
File "C:\path\to\webrtc\src\chromium\src\build\gyp_environment.py", line 33, in
SetEnvironment
vs_toolchain.SetEnvironmentAndGetRuntimeDllDirs()
File "C:\path\to\webrtc\src\chromium\src\build\vs_toolchain.py", line 34, in
SetEnvironmentAndGetRuntimeDllDirs
Update()
File "C:\path\to\webrtc\src\chromium\src\build\vs_toolchain.py", line 179, in
Update
subprocess.check_call(get_toolchain_args)
File "C:\path\to\depot_tools\python276_bin\lib\subprocess.py", line 540, in
check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\path\\to\\depot_tools\\python276_bin\\python.exe',
'C:\\path\\to\\depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json',
'C:\\path\\to\\webrtc\\src\\chromium\\src\\build\\win_toolchain.json',
'ee7d718ec60c2dc5d255bbe325909c2021a7efef']' returned non-zero exit status 1
Error: Command C:\path\to\depot_tools\python276_bin\python.exe src/build/landmines.py returned non-zero exit status 1 in C:\path\to\webrtc\src\chromium
Hook ''C:\path\to\depot_tools\python276_bin\python.exe' src/build/landmines.py' took 68.55 secs
Error: Command C:\path\to\depot_tools\python276_bin\python.exe - u src/sync_chromium.py --target-revision 271c6cca48a6cef32c0f3add3b17b700707deec 5 returned non-zero exit status 2 in C:\path\to\webrtc
Hook ''C:\path\to\depot_tools\python276_bin\python.exe' -u src/sync_chromium.py --target-revision 271c6cca48a6cef32c0f3add3b17b700707deec5' took 3842.34 secs
I'm using 64-bit Windows 7 with VS 2013 installed. I've already tried setting the following variables:
GYP_DEFINES='windows_sdk_path="C:\path\to\windows\sdk" platform=win
component=shared_library'
DEPOT_TOOLS_WIN_TOOLCHAIN=0
GYP_GENERATORS=msvs
GYP_MSVS_VERSION=2013
Attempting to build chromium also gives the same error. Fetching with nohooks works fine, it's just when I try gclient sync or runhooks that the error occurs. I've also tried building with ninja, and that fails as well. I've got all SDKs and such all up to date. Any help with this would be greatly appreciated.
I have found this possible solution at Google Code
as per braveyao#webrtc.org's sugestion
Supporting msvs is not the focus of Chromium. You can try ninja which
should always work. Also you could try to set the
"GYP_GENERATORS=msvs-ninja,ninja". Then you could build with ninja and
debug with VS2013.
Also, chromium is now 64bit only so you need to change GYP_DEFINES aswell
so your envirnment variables should be:
GYP_DEFINES=target_arch=x64
DEPOT_TOOLS_WIN_TOOLCHAIN=0
GYP_GENERATORS=msvs-ninja,ninja
GYP_MSVS_VERSION=2013
Of course replace 2013 if that's not the version of Visual Studio you have installed
you may need to delete what you already have downloaded and do a fresh fetch webrtc after that change if you thinkered with it
I should also add that the Windows must be english version (it's ok if you after install some other language pack but it must be originally english) - i have a PT-PT version at home and that one always gives an error with some python file when running gclient sync - at work though i originally had english version and installed the PT-PT language pack but it works fine
This is what worked for me. Does that work for you?