i want to make a DAG file (apache airflow) for uploading a rar file to s3 bucket any one tried.? plz suggest ,
and i tried these things on my DAG file but there is showing some error
from airflow.operators import SimpleHttpOperator, HttpSensor, , EmailOperator, S3KeySensor
The error is
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'SimpleHttpOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'HttpSensor' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
how to solve this issue.?
This is simply a warning, not an error. A DeprecationWarning usually hints that something you're doing will work now, but may break in future versions. If your task is failing, ignore these messages and look for a proper error.
Code for operators have always been located under airflow.operators.[operator_module], but it was also made available under airflow.operators directly for convenience. For example, SimpleHttpOperator is defined in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/http_operator.py, so importing it from airflow.operators.http_operator will definitely work. However, importing it from airflow.operators will also work due to the code that currently exists in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/init.py#L97-L99, at least for now in your current version of Airflow. Basically, you can address these warnings by updating your imports to the following:
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.email_operator import EmailOperator
from airflow.sensors.http_sensor import HttpSensor
from airflow.sensors.s3_key_sensor import S3KeySensor
Just heads up, currently only on the master branch, not yet in any released version, some of the third party operators and sensors have also been moved again. For example, S3KeySensor will be found under providers.amazon.aws.sensors.s3_key.py. As expected, importing from the "old" path will get you a similar deprecation message, https://github.com/apache/airflow/blob/97a429f9d0cf740c5698060ad55f11e93cb57b55/airflow/sensors/s3_key_sensor.py#L25-L28.
Related
I am currently trying to push data from Salesforce to BigQuery using Airflow. I am new to airflow and currently I'm following this link:
https://github.com/TheF1rstPancake/airflow-salesforce
When I try to import the plugin
from airflow.operators import SalesforceToFileOperator,
I get the error message stating ImportError: cannot import name 'SalesforceToFileOperator'
How do I import this plugin on Airflow?
Current Directory Structure
DAGS
salesforce_bg.py
plugins
airflow-salesforce
__init__.py
hooks
operators
What am I doing wrong?
From the official Airflow documentation: here
The python modules in the plugins folder get imported, and hooks, operators, sensors, macros, executors and web views get integrated to Airflow’s main collections and become available for use.
So, you only need to place salesforce_bg.py into the /plugins folder of your Airflow server.
Defining the dependency of the version 5.5.10:
dependencies {
...
compile 'com.itextpdf:itextg:5.5.10'
}
works fine.
But 5.5.11 or 5.5.12 do not compile.
Has this been released yet? If not, any idea when it may be?
I am QA & release engineer at iText Software, and I am possibly best placed to answer this question.
iTextG 5.5.11 and 5.5.12 have not yet been released. That is in the backlog, and not yet planned. It certainly won't happen this month.
Are there any changes in 5.5.11 or 5.5.12 that you urgently need? Because they are bugfix releases, without any new features, and if you are not affected by the fixed bugs, then there is no need to upgrade. See iText changelogs for more details (on the iText website, or the git log on GitHub).
Using the version 5.5.11 indicated in the docs:
dependencies {
compile 'com.itextpdf:itextg:5.5.11'
}
or the latest release 5.5.13 doesn´t work, probably it doesn´t exists for download via gradle:
I had to change to :
compile 'com.itextpdf:itextg:5.5.10'
Also import following class to generate pdf
import com.itextpdf.text.*;
import com.itextpdf.text.Document;
import com.itextpdf.text.DocumentException;
import com.itextpdf.text.Image;
import com.itextpdf.text.Paragraph;
import com.itextpdf.text.pdf.PdfContentByte;
import com.itextpdf.text.pdf.PdfPCell;
import com.itextpdf.text.pdf.PdfPTable;
import com.itextpdf.text.pdf.PdfWriter;
You can also refer more methods and techniques to generate pdf with inbuilt facility on below official link:
https://developers.itextpdf.com/examples/itext-action-second-edition/chapter-1
If anyone like my help to make it easy than give Up Arrow and Right Click for the same.
Thankyou.
I'm simply trying to import 'aurelia-event-aggregator' into a TypeScript class and I get this error in my console. Even though it says it's loaded and configured aurelia-event-aggregator on lines 5 & 6, it says SystemJS still can't find it. This is especially strange since I'm also (successfully) importing autoinject from 'aurelia-dependency-injection' and customElement from 'aurelia-framework' without a problem, and they seem to be configured/installed identically to the event aggregator on my system via jspm.
My config.js and package.json are here.
I haven't the slightest idea how to start troubleshooting this. Please let me know if there's any other information I can provide.
This is going to sound dumb, but you need to run jspm install aurelia-event-aggregator. I know it's already in your project, but jspm 0.16 doesn't support peer dependencies (at least that's what I remember), so you have to explicitly install it as a top level dep for your app and then it will start working.
In the latest Apache poi download(poi-3.15-beta2), while taking screenshot, I need to use FileUtils.copyFile. In its previous version, the imported package was import org.apache.commons.io.FileUtils;. In the latest download, this package is not coming, and it is giving error in my existing executable code. Now I tried to remove the previous import and it gave import org.apache.tools.ant.util.FileUtils;
Code:
FileUtils.copyFile(
scrFile,
new File(location+"LR_"+strDate+"_scr1.png")
);
Gives the error:
Cannot make a static reference to the non-static method
`copyFile(File, File)` from the type `FileUtils`
Apache POI never bundled or required Apache Commons IO, which contains the FileUtils class and so it seems some other project dragged in this code previously, but does not any longer. See http://poi.apache.org/overview.html#components for the list of third-party projects that Apache POI uses.
You should simply add a recent commons-io dependency to your project depending on which type of buildsystem you use, e.g. a normal dependency in Gradle/Maven or the actual jar-file if you have a buildsystem without full dependency-support.
Use the code below:
FileUtils.getFileUtils().copyFile(sourceFile, new File(directory + filename));
And import file should be:
import org.apache.tools.ant.util.FileUtils;
I am using openpyxl from different Python version using following way.
sys.path.insert(0,
'/remote/Python-2.7.2-shared/linux32/lib/python2.7/site-packages/openpyxl-1.6.1-py2.7.egg')
sys.path.insert(1,
'/remote/Python-2.7.2-shared/linux32/lib/python2.7/site-packages')
I will not receive any warning when I directly use particular version.
Python-2.7.2-shared/linux32/lib/python2.7/site-packages/openpyxl-1.6.1-py2.7.egg/openpyxl/shared/compat/elementtree.py:30:
UserWarning: Unable to import 'xml.etree.cElementree'. Falling back on
'xml.etree.Elementree'
I am reading more than 100 xlsx File and did manual testing previously and need to provide quick fix.
As per my understanding, I am reading xlsx File and does not contain any xml element.
So it should not impact any reading data in xlsx File. can be confirm it or can I ignore this warning.
One small thing not related to openpyxl.
is it possible to hide this warning. I do not have root permission
You can ignore warnings. What you are doing is not recommended. The warning is just that without cElementTree your code may run slow. Python does support install packages for in user home directories but using virtual environments (virtualenv) is preferable.