Getting the follow running the example from the docs. Anyone have ideas why?
Looks like gprc. Maybe this is related? https://github.com/grpc/grpc/issues/15933
docker run --network host -e FLYTE_PLATFORM_URL='127.0.0.1:30081' lyft/flytesnacks:v0.1.0 pyflyte -p flytesnacks -d development -c sandbox.config register workflows
Using configuration file at /app/sandbox.config
Flyte Admin URL 127.0.0.1:30081
Running task, workflow, and launch plan registration for flytesnacks, development, ['workflows'] with version 46045e6383611da1cb763d64d846508806fce1a4
Registering Task: workflows.edges.edge_detection_canny
Traceback (most recent call last):
File "/app/venv/bin/pyflyte", line 11, in <module>
sys.exit(main())
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/clis/sdk_in_container/register.py", line 86, in workflows
register_all(project, domain, pkgs, test, version)
File "/app/venv/lib/python3.6/site-packages/flytekit/clis/sdk_in_container/register.py", line 24, in register_all
o.register(project, domain, name, version)
File "/app/venv/lib/python3.6/site-packages/flytekit/common/exceptions/scopes.py", line 158, in system_entry_point
return wrapped(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/common/tasks/task.py", line 141, in register
_engine_loader.get_engine().get_task(self).register(id_to_register)
File "/app/venv/lib/python3.6/site-packages/flytekit/engines/flyte/engine.py", line 234, in register
self.sdk_task
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/friendly.py", line 50, in create_task
spec=task_spec.to_flyte_idl()
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/raw.py", line 12, in handler
return fn(*args, **kwargs)
File "/app/venv/lib/python3.6/site-packages/flytekit/clients/raw.py", line 77, in create_task
return self._stub.CreateTask(task_create_request)
File "/app/venv/lib/python3.6/site-packages/grpc/_channel.py", line 604, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/app/venv/lib/python3.6/site-packages/grpc/_channel.py", line 506, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"#1578482148.715995870","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3876,"referenced_errors":[{"created":"#1578482148.715993301","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":395,"grpc_status":14}]}"
I think the problem is in FLYTE_PLATFORM_URL='127.0.0.1:30081'.Minikube runs in VM on OSX.
When you run docker run --network host, the docker daemon shares the networking namespace with this host - https://docs.docker.com/network/host/
Thus if you have the docker daemon and Kubernetes running on the same host (not a VM), you should be able to refer to services (nodeport) or ingress on Kubernetes using 127.0.0.1.
But, since minikube runs in a VM on osx, you will need to use minikube ip to find the ip address of the minikube machine and then set
FLYTE_PLATFORM_URL=':30081'
Related
I am running a CherryPy Local server. But After sometime server is timing out. I am new to CherryPy server and I found in error.log this message and unable to solve this issue.
CherryPy==18.8.0
Cherrot==8.6.0
Python==3.8.1
Traceback (most recent call last):
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\server.py", line 1300, in communicate
req.respond()
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\server.py", line 1090, in respond
self.server.gateway(self).respond()
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\wsgi.py", line 145, in respond
self.write(chunk)
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\wsgi.py", line 231, in write
self.req.write(chunk)
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\server.py", line 1146, in write
self.conn.wfile.write(chunk)
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\makefile.py", line 438, in write
res = super().write(val, *args, **kwargs)
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\makefile.py", line 36, in write
self._flush_unlocked()
File "C:\Users\test\Envs\eps\lib\site-packages\cheroot\makefile.py", line 45, in _flush_unlocked
n = self.raw.write(bytes(self._write_buf))
File "c:\python381\lib\socket.py", line 687, in write
return self._sock.send(b)
File "c:\python381\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
socket.timeout: The write operation timed out
Im trying to use rabbitmq as a broker for celery and using Hashicorp Vault for root credential rotation. How do I use vault's rabbitmq secret engine with celery?
def get_broker_url():
url = "http://localhost:8200/v1/rabbitmq/creds/dev-role"
payload = {}
headers = {
'X-Vault-Token': 'VAULT_TOKEN'
}
response = requests.get(url, headers=headers, data=payload).json()
return "amqp://{username}:{password}#127.0.0.1:5672/".format(username=response["data"]["username"], password=response["data"]["password"])
And while initialising celery
app = Celery(app_name, broker=get_broker_url(), backend="rpc://")
Im getting this error when lease expires
[2022-11-24 13:05:54,489: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 628, in start
c.loop(*c.loop_args())
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
cb(*cbargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/base.py", line 235, in on_readable
reader(loop)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/base.py", line 217, in _read
drain_events(timeout=0)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 525, in drain_events
while not self.blocking_read(timeout):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 531, in blocking_read
return self.on_inbound_frame(frame)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/method_framing.py", line 53, in on_frame
callback(channel, method_sig, buf, None)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 538, in on_inbound_method
method_sig, payload, content,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
listener(*args)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 668, in _on_close
(class_id, method_id), ConnectionError)
amqp.exceptions.ConnectionForced: (0, 0): (320) CONNECTION_FORCED - user 'root-3a978f79-e2fb-2889-08d4-0c69d2bcbffd' is deleted
[2022-11-24 13:05:54,498: WARNING/MainProcess] /Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
[2022-11-24 13:05:54,505: CRITICAL/MainProcess] Unrecoverable error: AccessRefused(403, 'ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.', (0, 0), '')
Traceback (most recent call last):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/connection.py", line 21, in start
c.connection = c.connect()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 428, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 435, in connection_for_read
self.app.connection_for_read(heartbeat=heartbeat))
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 462, in ensure_connected
callback=maybe_shutdown,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 381, in ensure_connection
self._ensure_connection(*args, **kwargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 437, in _ensure_connection
callback, timeout=timeout
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 877, in _connection_factory
self._connection = self._establish_connection()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/connection.py", line 812, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
conn.connect()
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 329, in connect
self.drain_events(timeout=self.connect_timeout)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 525, in drain_events
while not self.blocking_read(timeout):
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 531, in blocking_read
return self.on_inbound_frame(frame)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/method_framing.py", line 53, in on_frame
callback(channel, method_sig, buf, None)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 538, in on_inbound_method
method_sig, payload, content,
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
listener(*args)
File "/Users/vg/celery-mq-demo/venv/lib/python3.7/site-packages/amqp/connection.py", line 668, in _on_close
(class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
Since broker root user's lease expires based on the ttl, How to add new broker credential for celery to use?
celery version - 5.2.7
I have an anaconda environment with selenium installed. When I try to run I get this error:
Traceback (most recent call last):
File "c:\Users\Nick\Desktop\Code\product-scraper\sephora-scraper\scraper.py", line 31, in <module>
ChromeDriverManager().install(), options=options)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\webdriver_manager\chrome.py", line 34, in install
driver_path = self._get_driver_path(self.driver)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\webdriver_manager\manager.py", line 21, in _get_driver_path
driver_version = driver.get_version()
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\webdriver_manager\driver.py", line 40, in get_version
return self.get_latest_release_version()
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\webdriver_manager\driver.py", line 63, in get_latest_release_version
resp = requests.get(f"{self._latest_release_url}_{self.browser_version}")
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Nick\anaconda3\envs\web-scraper\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='chromedriver.storage.googleapis.com', port=443): Max retries exceeded with url: /LATEST_RELEASE_88.0.4324 (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))
I'm new to anaconda so I don't know what else to provide. Please leave a comment if I need to anything and I will add it right away. Thanks.
Try to add this path to your environment variable:
..\Anaconda3
..\Anaconda3\scripts
..\Anaconda3\Library\bin
You might need to restart windows after set up environment path
I want to use my Raspberry Pi 3, running Rapian Stretch for a web scraping project. For python i use the berryconada distribution.
When I run my Spider, I get
ValueError: Unknown ECC curve
On my Laptop (Xubuntu 16.04) everything runs fine. Maybe I need to install an additional library or something?
Down below the full traceback.
Traceback (most recent call last):
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/utils/defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 63, in download_request
return agent.download_request(request)
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 300, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/web/client.py", line 1633, in request
endpoint = self._getEndpoint(parsedURI)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/web/client.py", line 1617, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/web/client.py", line 1494, in endpointForURI
uri.port)
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/contextfactory.py", line 59, in creatorForNetloc
return ScrapyClientTLSOptions(hostname.decode("ascii"), self.getContext())
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/contextfactory.py", line 56, in getContext
return self.getCertificateOptions().getContext()
File "/home/pi/berryconda3/lib/python3.6/site-packages/scrapy/core/downloader/contextfactory.py", line 51, in getCertificateOptions
acceptableCiphers=DEFAULT_CIPHERS)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/python/deprecate.py", line 792, in wrapped
return wrappee(*args, **kwargs)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 1595, in __init__
self._ecCurve = _OpenSSLECCurve(_defaultCurveName)
File "/home/pi/berryconda3/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 1744, in __init__
raise ValueError("Unknown ECC curve.")
I dropped berryconda and pip installed scrapy. If you're getting this error on Jessie, moving to Stretch gives you access to the newer openssl libs which contain the missing things.
After I upgraded to Stretch I cut berryconda from my path, pip uninstalled cryptography, twisted, pyopenssl, and scrapy.
Then with the no cache option I pip installed scrapy, which brought all those packages back, and now my spider is running.
I have an issue on my Centos7 system behind a corporate proxy
cfy bootstrap --install-plugins -p /opt/cfy/cloudify-manager-blueprints-commercial/simple-manager-blueprint.yaml -i /opt/cfy/cloudify-manager-blueprints-commercial/simple-manager-blueprint-inputs.yaml
returns
SSL: CERTIFICATE_VERIFY_FAILED
env http_proxy and https_proxy is set and I've exported
CLOUDIFY_SSL_TRUST_ALL=true;
Collecting https://github.com/cloudify-cosmo/cloudify-fabric-plugin/archive/1.3.1.zip (from -r /tmp/requirements_oUJf1I.txt (line 1))
Command 'pip install -r /tmp/requirements_oUJf1I.txt' executed with an error.
code: 2
error: Exception:
Traceback (most recent call last):
File "/opt/cfy/env/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
wb.build(autobuilding=True)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/opt/cfy/env/lib/python2.7/site-packages/pip/req/req_set.py", line 577, in _prepare_file
session=self.session, hashes=hashes)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/download.py", line 810, in unpack_url
hashes=hashes
File "/opt/cfy/env/lib/python2.7/site-packages/pip/download.py", line 649, in unpack_http_url
hashes)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/download.py", line 842, in _download_http_url
stream=True,
File "/opt/cfy/env/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/opt/cfy/env/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)
output: None
It's hard to know what went wrong without looking at the blueprint and the inputs.yaml (after removing all credentials).
Without more data I would simply try to export
CLOUDIFY_SSL_TRUST_ALL=True;
Let me know if it works