I'm trying to use a file in a DAG.
The codes I want to use is basically this:
conf_device_info = OmegaConf.load(f"./config/{dag_name}/config_dtype.json")
and my bucket is currently like this:
my-bucket
--/ dags
-- /config
--/{dag_name}
--/config_dtype.json
-- dag_with_the_code.py
-- /utils
--s3_manager.py
When I import s3_manager with "import utils.s3_manager" , it goes fine.
When I try to run the code the OmegaConf code, it says
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/airflow/config/{dag_name}/config_dtype.json'
What should I do to do what I'm trying to acheive?
Why is import working and referencing file with absolute path not working..?
Thanks in advance.
Related
I was trying to read csv file in jupyter notebook but it showed error of filenotfound. Then I tried to check whether my file is present then it shoewd false as output. But I have checked the file location in my files explorer and the csv file is present .How should I read the file?
import os
os.path.isfile(r'C:\Users\Ritesh\Downloads\Data\Amazon_Products.csv')
Screenshot of code and error
maybe try:
import pandas as pd
df = pd.read_csv('your-filepath')
you could also try to move the file into your project directory so that it is in the same folder as the .ipynb
I have faced a problem with Sphinx in Python. Even if I have followed the instructions from https://groups.google.com/forum/#!topic/sphinx-users/lO4CeNJQWjg I was not able to solve it.
I have Docs/source folder which contains:
conf.py
index.rst
RstFiles (the folder which contains .rst files for each module).
In conf.py I specify the abs path in the following way:
sys.path.insert(0, os.path.abspath('..'))
In index.rst I call all the modules from RstFiles folder in the following way:
.. toctree::
:maxdepth: 2
:caption: Contents:
BatchDataContainer.rst
BatchDefaultValues.rst
BatchTypes.rst
And finally, the content of each .rst file is in the following way:
BatchDataContainer
==================
.. automodule:: RstFiles.BatchDataContainer
:members:
When I run sphinx-build I get 2 main errors:
D:\hfTools\Projects\Validation-Source\Docs\source\RstFiles\BatchDataContainer.rst:
WARNING: document isn't included in any toctree
and
WARNING: autodoc: failed to import module 'BatchDataContainer' from
module 'RstFiles'; the following exception was raised: No module named
'RstFiles'
Any ideas what might be wrong cause I have tried different things already and nothing has helped?
If conf.py is located in this directory,
D:\hfTools\Projects\Validation-Source\Docs\source,
and the project's Python modules (including BatchDataContainer.py) are in
D:\hfTools\Projects\Validation-Source\Products,
then you need sys.path.insert(0, os.path.abspath('../..')) in conf.py.
The automodule directive needs to be updated as well:
.. automodule:: Products.BatchDataContainer
:members:
Tensorflow 1.4.0 comes with the S3 filesystem driver by default. I'm having trouble using it, and have this minimal example, that does not work for me:
import tensorflow as tf
f = tf.gfile.Open("s3://bucket/plipp", mode='w')
f.write("foo")
f.close()
which gives the following error:
Traceback (most recent call last):
File "test2.py", line 5, in <module>
f.close()
File "/Users/me/venv3/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 234, in close
pywrap_tensorflow.Set_TF_Status_from_Status(status, ret_status)
File "/Users/me/venv3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: : Unable to connect to endpoint
From what I can see, it seems like "Unable to connect to endpoint" is an error from the C++ AWS SDK. I've given myself * permissions to the bucket.
My bucket is in eu-west-1 and I've tried doing export S3_ENDPOINT=https://s3-eu-west-1.amazonaws.com and export S3_REGION=eu-west-1 since it seems that those variables are consumed by the S3 driver, but this changes nothing.
I've also tried using s3://bucket.s3-eu-west-1.amazonaws.com/plipp as the path, instead of just using the bucket name.
I can copy files to the bucket fine:
~> aws s3 cp foo s3://bucket/plipp
upload: ./foo to s3://bucket/plipp
Any ideas what I might be doing wrong? How can I debug further?
I'm not quite sure what went wrong last time I tried this, but now I got it working by just doing export S3_REGION=eu-west-1 and writing to the bucket with
with tf.gfile.Open("s3://bucket/plipp", mode='w') as f:
f.write("foo")
So, don't export the S3_ENDPOINT variable.
So I have a custom pipeline that extends Scrapy's current FilesPipeline. However, I'm having trouble with setting the FILES_STORE variable. My current file structure is:
my_scraper.py
files/
#this is where I want the files to download to
so, I set FILES_STORE=/files/ and run the spider. But when I do that I get the following error:
PermissionError: [Errno 13] Permission denied: '/files/'
Why does this happen? Is there anything that I am doing wrong?
If it's useful to anyone else, it was simple error - FILES_STORE requires the full path, not just the relative path from the folder.
I downloaded this Lua module and installed it with make. However, when I try to use it in my code, I get the following error:
VirtualBox:~/Downloads$ lua socket_test.lua
lua: socket_test.lua:1: module 'nixio.util' not found:
no field package.preload['nixio.util']
no file './nixio/util.lua'
no file '/usr/local/share/lua/5.1/nixio/util.lua'
no file '/usr/local/share/lua/5.1/nixio/util/init.lua'
no file '/usr/local/lib/lua/5.1/nixio/util.lua'
no file '/usr/local/lib/lua/5.1/nixio/util/init.lua'
no file '/usr/share/lua/5.1/nixio/util.lua'
no file '/usr/share/lua/5.1/nixio/util/init.lua'
no file './nixio/util.so'
no file '/usr/local/lib/lua/5.1/nixio/util.so'
no file '/usr/lib/i386-linux-gnu/lua/5.1/nixio/util.so'
no file '/usr/lib/lua/5.1/nixio/util.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file './nixio.so'
no file '/usr/local/lib/lua/5.1/nixio.so'
no file '/usr/lib/i386-linux-gnu/lua/5.1/nixio.so'
no file '/usr/lib/lua/5.1/nixio.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
socket_test.lua:1: in main chunk
[C]: ?
And the first line of my code:
local nixio = require "nixio", require "nixio.util"
Any help would be appreciated!
I see these options:
Try make install and hope that it works. Perhaps set some variable such as PREFIX nothing.
Manually move dist/usr/local/share/lua/5.1/nixio to /usr/local/share/lua/5.1/nixio and dist/usr/local/lib/lua/5.1/nixio.so to /usr/local/lib/lua/5.1/nixio.so.
Add the dist paths to the environment: LUA_CPATH='dist/usr/local/lib/lua/5.1/?.so;;' and LUA_PATH='dist/usr/local/share/lua/5.1/?.lua;;'.
Add the dist paths in Lua: package.cpath='dist/usr/local/lib/lua/5.1/?.so;'..package.cpath and package.path='dist/usr/local/share/lua/5.1/?.lua;'..package.path.