Airflow version: 2.0.0
When I use the sftp command to manually connect to the host from any airflow worker everything works fine. Here is the error log from when I try to use the operator which under the hood uses the paramiko library to transfer files:
{ssh.py:202} WARNING - No Host Key Verification. This wont protect against Man-In-The-Middle attacks
{transport.py:1819} INFO - Connected (version 2.0, client 1.91)
{transport.py:1819} INFO - Auth banner: b'MOMENTUM SYSTEMS - SSH Server\nAuthentication Methods Supported:\nPUBLICKEY, PASSWORD'
{transport.py:1819} INFO - Authentication continues...
{transport.py:1819} INFO - Disconnect (code 2): unexpected service request
{taskinstance.py:1396} ERROR - Authentication failed.
Traceback (most recent call last):
File "/home/centos/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/centos/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/centos/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
result = task_copy.execute(context=context)
File "/home/centos/airflow-dags/utils/operators/s3_to_sftp.py", line 76, in execute
sftp_client = ssh_hook.get_conn().open_sftp()
File "/home/centos/.local/lib/python3.7/site-packages/airflow/providers/ssh/hooks/ssh.py", line 225, in get_conn
client.connect(**connect_kwargs)
File "/home/centos/.local/lib/python3.7/site-packages/paramiko/client.py", line 446, in connect
passphrase,
File "/home/centos/.local/lib/python3.7/site-packages/paramiko/client.py", line 764, in _auth
raise saved_exception
File "/home/centos/.local/lib/python3.7/site-packages/paramiko/client.py", line 751, in _auth
self._transport.auth_password(username, password)
File "/home/centos/.local/lib/python3.7/site-packages/paramiko/transport.py", line 1509, in auth_password
return self.auth_handler.wait_for_response(my_event)
File "/home/centos/.local/lib/python3.7/site-packages/paramiko/auth_handler.py", line 236, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.
The Airflow connection that I use has the password and no additional options in extra.
The answer provided to the linked question worked for my use case:
Multi-factor authentication (password and key) with Paramiko
Related
I'm trying to user google sheets API service, which requires an HTTPS connection. I'm getting the following error
Exception Type: InsecureTransportError at my_site/google/success/
Exception Value: (insecure_transport) OAuth 2 MUST utilize https.
I am using Heroku, and on my settings it says AMC Status: ok. I verified that I'm using HTTPS by running curl -vI https://my_site/google/success
which returned:
SSL certificate verify ok
From my perspective it seems that I am using HTTPS, but I am getting this error. What could I be doing wrong? Surely I have something misconfigured Anything else I need to provide from troubleshooting? Here is the Full traceback:
traceback (most recent call last):
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/app/google_api/views.py", line 56, in authorize_success
flow.fetch_token(authorization_response=code)
File "/app/.heroku/python/lib/python3.8/site-packages/google_auth_oauthlib/flow.py", line 286, in fetch_token
return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
File "/app/.heroku/python/lib/python3.8/site-packages/requests_oauthlib/oauth2_session.py", line 239, in fetch_token
self._client.parse_request_uri_response(
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 203, in parse_request_uri_response
response = parse_authorization_code_response(uri, state=state)
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 256, in parse_authorization_code_response
raise InsecureTransportError()
import os
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
I am trying to get a connection with the Nessus server with the bellow command in python but it failed with an error message can you tell me what can be the cause. I have checked my network connection it is fine.
requests.post( 'https://164.99.175.30:8834/'+ '/session',data={'username':'admin','password':'micro#123'},verify=False)```
error message
Traceback (most recent call last):
File "nessus.py", line 425, in <module>
login()
File "nessus.py", line 111, in login
res = requests.post(url + '/session',data={'username':username,'password':password},verify=verify)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=8834): Max retries exceeded with url: /session (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f46f2d6d410>: Failed to establish a new connection: [Errno 111] Connection refused',))
The nessus api is depreciated as of version 7.x, this is the best source I could find.
EDIT: I have found a better source directly from tenable
What has been removed from Nessus 7:
There is a restriction in scan API capabilities.
The ability to manage scans via API and CLI has been removed in v7. All Nessus Pro scanning operations must be done through the user interface.
So currently the ability of the Nessus API is as follows:
Removed the ability to run scans or reports and create new objects
The Read features, where the ability to pull scan data so GET /scan/scan ID now works again and this aids with some of the integration processes.
https://community.tenable.com/s/article/The-differences-between-Nessus-6-and-Nessus-7
This is only for Nessus pro versions
TL;DR: When I try to use flask db migrate with a remote database, I get: MySQLdb._exceptions.OperationalError: (1045, "Access denied for user ''#'localhost' (using password: YES)"). But I'm certain that the username and password are correct, the remote user has the SELECT privilege, and I don't actually have any models declared yet.
I have an SSH tunnel set up for access to a remote database.
$ ssh -L <local_port>:localhost:<remote_internal_port> <mysql_user>#<server> -p <remote_external_port>
I have credentials for a read-only user on the remote MySQL DB (really MariaDB). This is not the database the users are stored in, so I'm using SQLALCHEMY_BINDS to add the second database.
My environment looks like this:
DATABASE_URL=sqlite:////tmp/dev.db
DATABASE_BINDS="remote=mysql+mysqldb://<mysql_user>:<pass>#localhost:<local_port>/<db_name>"
Then in the config file:
from environs import Env
env = Env()
env.read_env()
SQLALCHEMY_DATABASE_URI = env.str("DATABASE_URL") # local
SQLALCHEMY_BINDS = env.dict("DATABASE_BINDS") # remote, etc
I don't presently have any models declared for the remote database at all. I'm planning to use reflection to load them.
The initial run of flask db init --multidb works fine. It creates the migration folder. But when I run flask db migrate -m "Initial DB", I get this:
Traceback (most recent call last):
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
return fn()
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
return _ConnectionFairy._checkout(self)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get
return self._create_connection()
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/path/to/project/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/path/to/project/venv/lib/python3.7/site-packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/path/to/project/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 166, in __init__
super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (1045, "Access denied for user '<mysql_user>'#'localhost' (using password: YES)")
I've checked and double-checked that the username and password I've declared in the environment are correct. I can successfully use a GUI client with these settings to browse the remote MySQL database via the SSH tunnel. I can use the MySQL CLI client on the remote server to USE and SELECT on the appropriate database.
I'm at a loss for where to look here. I think the SSH tunnel is a red herring, since it's working well with the GUI client, but I don't know how to get more logging to see what the problem really is. This answer implies that the issue might in fact be one of permissions, not with the l/p themselves. Does Flask-Migrate require something other than the SELECT privilege?
The problem was that I was using "localhost" instead of "127.0.0.1".
I want use the python library ncclient 0.6.6 with Python 2.7.15 to connect to a NETCONF server (netopeer2) and read out the running config.
I tried to follow the example from the manual, running this code in the console:
with manager.connect(host="*the IP adress*", port=*the port*, timeout=None, username="*user*", password="*pwd*") as m:
c = m.get_config(source='running').data_xml
with open("%s.xml" % host, 'w') as f:
f.write(c)
As written in the manual, I try to disable public-key authentification with allow_agent and look_for_keys as False. Unfortunately, this does not work properly, because I get the error message:
File "<stdin>", line 1, in <module>
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 177, in connect
return connect_ssh(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 143, in connect_ssh
session.connect(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/transport/ssh.py", line 481, in connect
raise SSHUnknownHostError(known_hosts_lookup, fingerprint)
ncclient.transport.errors.SSHUnknownHostError: Unknown host key [e3:8d:35:a9:43:f9:3c:8a:f4:d3:88:5b:a9:36:93:59] for [[192.168.56.2]:1831]
I do not get why it still complains about the unknown host key, even though I explicitly disabled public-key authentification.
The netopeer NETCONF server is definitely running, for I get a "Hello" Message as soon as I try to SSH into it from out of the terminal.
Did I miss something?
m = manager.connect(host="172.17.0.2", port=830, username="netconf", password="netconf", hostkey_verify=False)
Did the trick. Hostkey_verify has to be false.
I've been trying to set up my EndpointsV2 backend (billing has been enabled) with Firebase authentication but am always ending up with either an error when I attempt to generate the OpenAPI spec or a 401 Unauthorized when accessing the API.
My current configuration is as follows (errors out on OpenAPI generation):
main.py:
firebase_issuer = endpoints.Issuer(
issuer='https://securetoken.google.com/[GAE/FIREBASE_APP_ID]',
jwks_uri=('https://www.googleapis.com/service_accounts/v1/metadata/x509/'
'securetoken#system.gserviceaccount.com'))
issuers = {
"firebase": firebase_issuer
}
#endpoints.api("user", "v1", api_key_required=True,
issuers=issuers)
class UserApi(remote.Service):
#models.UserModel.query_method(
user_required=True,
path="info",
name="user.info",
http_method="GET")
def info(self, query):
return query.filter(
models.UserModel.owner == endpoints.get_current_user())
Error on OpenAPI Gen:
$ python lib/endpoints/endpointscfg.py get_openapi_spec main.UserApi --hostname [SERVICE-NAME].endpoints.[GAE_APP_ID].appspot.com
No handlers could be found for logger "endpoints.apiserving"
Traceback (most recent call last):
File "lib/endpoints/endpointscfg.py", line 633, in <module>
main(sys.argv)
File "lib/endpoints/endpointscfg.py", line 629, in main
args.callback(args)
File "lib/endpoints/endpointscfg.py", line 487, in _GenOpenApiSpecCallback
application_path=args.application)
File "lib/endpoints/endpointscfg.py", line 332, in _GenOpenApiSpec
application_path=application_path)
File "lib/endpoints/endpointscfg.py", line 214, in GenApiConfig
services, hostname=hostname))
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 973, in pretty_print_config_to_json
descriptor = self.get_openapi_dict(services, hostname)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 959, in get_openapi_dict
return self.__api_openapi_descriptor(services, hostname=hostname)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 872, in __api_openapi_descriptor
security_definitions)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 691, in __method_descriptor
service.api_info.audiences, security_definitions)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 722, in __x_security_descriptor
_INVALID_AUTH_ISSUER % default_auth_issuer)
api_exceptions.ApiConfigurationError: No auth issuer named google_id_token defined in this Endpoints API.
The error leads me to think that if I simply swap "firebase" for "google_id_token"in issuers that it may work but I have tried that and just gotten a 401.
I've also started to see this error in GAE Logging:
Cannot decode and verify the auth token. The backend will not be able to retrieve user info (/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/control/wsgi.py:588)
Traceback (most recent call last):
File "/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/control/wsgi.py", line 585, in __call__
service_name)
File "/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/auth/tokens.py", line 83, in authenticate
"allow provider id: " + provider_id)
UnauthenticatedException: The requested method does not allow provider id: google_id_token
The idToken I am sending with my testing (done through Postman using the header Authorization: Bearer [idToken]) was generated using pyrebase and I checked jwt.io to confirm that it is in fact a valid token.
My initial thought is that I haven't set up the audiences correctly, but I'm not sure what it should be (though I have tried appending a ClientID from API Manager > Credentials).
Is it possible to authenticate with firebase using endpoints-proto-datastore on the endpointsv2 framework? If so, what do I need to do differently to get that working? If not, would the code that I have work right away without using endpoints-proto-datastore?
Edit:
Looks like this is an issue with the endpoints library itself
https://github.com/cloudendpoints/endpoints-python/issues/32