I am currently trying to push data from Salesforce to BigQuery using Airflow. I am new to airflow and currently I'm following this link:
https://github.com/TheF1rstPancake/airflow-salesforce
When I try to import the plugin
from airflow.operators import SalesforceToFileOperator,
I get the error message stating ImportError: cannot import name 'SalesforceToFileOperator'
How do I import this plugin on Airflow?
Current Directory Structure
DAGS
salesforce_bg.py
plugins
airflow-salesforce
__init__.py
hooks
operators
What am I doing wrong?
From the official Airflow documentation: here
The python modules in the plugins folder get imported, and hooks, operators, sensors, macros, executors and web views get integrated to Airflow’s main collections and become available for use.
So, you only need to place salesforce_bg.py into the /plugins folder of your Airflow server.
Related
I tried to install EZGmail module for Python. When I check my interpreter settings, it shows it as installed in the PyCharm list. I then use this same interpreter for my project, but I get a module not found error when trying to import EZGmail. What should I check?
Looks like you're trying to import the module in a different casing than it's mentioned on the document.
You're trying to do:
import EZGmail
while the Quickstart document says:
import ezgmail
i want to make a DAG file (apache airflow) for uploading a rar file to s3 bucket any one tried.? plz suggest ,
and i tried these things on my DAG file but there is showing some error
from airflow.operators import SimpleHttpOperator, HttpSensor, , EmailOperator, S3KeySensor
The error is
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'SimpleHttpOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'HttpSensor' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python3.6/dist-packages/airflow/utils/helpers.py:439: DeprecationWarning: Importing 'EmailOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
how to solve this issue.?
This is simply a warning, not an error. A DeprecationWarning usually hints that something you're doing will work now, but may break in future versions. If your task is failing, ignore these messages and look for a proper error.
Code for operators have always been located under airflow.operators.[operator_module], but it was also made available under airflow.operators directly for convenience. For example, SimpleHttpOperator is defined in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/http_operator.py, so importing it from airflow.operators.http_operator will definitely work. However, importing it from airflow.operators will also work due to the code that currently exists in https://github.com/apache/airflow/blob/1.10.9/airflow/operators/init.py#L97-L99, at least for now in your current version of Airflow. Basically, you can address these warnings by updating your imports to the following:
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.email_operator import EmailOperator
from airflow.sensors.http_sensor import HttpSensor
from airflow.sensors.s3_key_sensor import S3KeySensor
Just heads up, currently only on the master branch, not yet in any released version, some of the third party operators and sensors have also been moved again. For example, S3KeySensor will be found under providers.amazon.aws.sensors.s3_key.py. As expected, importing from the "old" path will get you a similar deprecation message, https://github.com/apache/airflow/blob/97a429f9d0cf740c5698060ad55f11e93cb57b55/airflow/sensors/s3_key_sensor.py#L25-L28.
I am currently working on creating a Test Suite using Cucumber framework. I have created feature files for each of the functionality I have to test.
The feature file never shows the conditions to be covered via glue code. On click of Recalculate steps it throws an error : An internal error occurred during: "Scanning for step definitions".
java.lang.NullPointerException
I tried to change the path of glue in the Test Runner class but it did not solve the problem as well. The feature can be executed without any issue and the code runs fine, only issue is feature files keep on stating that there is no matching glue code.
#OrderInteraction
#Orders
Feature: Validating Order functionality
Background: Pre-requisites of Order Functionality
Given WebDriver is initialized
And Website is up and running
When User Enter "Username" and "Password" as Credentials
Then Validate User Login
Test Runner :
package Cucumber;
import org.junit.runner.RunWith;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
#RunWith(Cucumber.class)
#CucumberOptions(features = "Features", glue=
{"src/main/java/CucumberStepDefinitions"})
public class CucumberTestRunner {
}
I would like to find a way in which Cucumber feature files can recognize the glue code present. Feel free to guide me to a existing question which answers this or a documentation which will be of help.
There is a possible solution in this discussion https://github.com/cucumber/cucumber-eclipse/issues/303
Which basically says to make sure you don't have
import cucumber.api.java.en.*;
in your steps files, and to use
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
instead.
Which fixed it for me
I finally was able to find the solution to this issue. Issue lies with the cucumber plugin version, I referred to Cucumber-Eclipse-Update-SiteSnapshot for resolution.
The steps were as below :
1.From Eclipse, go to menu Help > Install New software
2.Work with: https://cucumber.github.io/cucumber-eclipse-update-site-snapshot
3.Select the check-box for Cucumber Eclipse Plugin
4.Select Next as per the instruction shown during installation.
5.Restart your Eclipse after completion of instruction.
Once the plugin was updated Feature file automatically scanned for steps and found the matching glue code.
In the latest Apache poi download(poi-3.15-beta2), while taking screenshot, I need to use FileUtils.copyFile. In its previous version, the imported package was import org.apache.commons.io.FileUtils;. In the latest download, this package is not coming, and it is giving error in my existing executable code. Now I tried to remove the previous import and it gave import org.apache.tools.ant.util.FileUtils;
Code:
FileUtils.copyFile(
scrFile,
new File(location+"LR_"+strDate+"_scr1.png")
);
Gives the error:
Cannot make a static reference to the non-static method
`copyFile(File, File)` from the type `FileUtils`
Apache POI never bundled or required Apache Commons IO, which contains the FileUtils class and so it seems some other project dragged in this code previously, but does not any longer. See http://poi.apache.org/overview.html#components for the list of third-party projects that Apache POI uses.
You should simply add a recent commons-io dependency to your project depending on which type of buildsystem you use, e.g. a normal dependency in Gradle/Maven or the actual jar-file if you have a buildsystem without full dependency-support.
Use the code below:
FileUtils.getFileUtils().copyFile(sourceFile, new File(directory + filename));
And import file should be:
import org.apache.tools.ant.util.FileUtils;
I have a MyFile entity with sql db file:
class MyFile(ndb.Model):
data = ndb.BlobProperty()
data property store sql db file. So in requestHandler I need to parse this file.
I am trying to using standard sqlite3 library, but getting error on gae side:
ImportError: No module named _sqlite3
With PyDbLite library I am getting error too:
from PyDbLite.SQLite import Database, Table
File "libs/PyDbLite/SQLite.py", line 81, in <module>
from pysqlite2 import dbapi2 as sqlite
Is there any way to parse sql database in goggle app engine?
I know that gae uses NoSQL datastore, I just need to handle sql db file.
The module _sqlite3 may be in your python path locally, but when deployed, the module cannot be found. You can debug your python path by printing it when in production and making sure the module is visible on that path (keep in mind that any old file or folder isn't considered a module ready for import by python, it needs to have __init__.py).
The only reason you would receive that error is if you either
A) failed to upload the module you're using as part of your project, mistakenly believing that since it was visible to your local python code it should be visible when deployed
or
B) failed to reference the uploaded module correctly. My comments in the first paragraph should help you debug this case.