# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
from google.cloud import bigquery_datatransfer_v1
def sample_start_manual_transfer_runs():
# Create a client
client = bigquery_datatransfer_v1.DataTransferServiceClient()
# Initialize request argument(s)
request = bigquery_datatransfer_v1.StartManualTransferRunsRequest(
)
# Make the request
response = client.start_manual_transfer_runs(request=request)
# Handle the response
print(response)
I am trying to write a script that manually triggers s3 to bigquery. However, I am unsure of how to integrate other types and functions provided from DataTransferServiceClient.
For example, how do I integrate transfer_config into the script above. Also, I am not quite sure how to get config_id from transfer_config once I have it.
Related
Is there a way to listen to redis publisher message, through Websocket of dash-extensions package?
from dash_extensions import WebSocket
...
html.Div(id="message", className='p-4 border'),
WebSocket(url='ws://127.0.0.1:6379/1', id='ws')
This gives error in redis-server
# Possible SECURITY ATTACK detected. It looks like somebody is sending POST or Host: commands to Redis. This is likely due to an attacker attempting to use Cross Protocol Scripting to compromise your Redis instance. Connection aborted.
Is such provision limited to specified redis clients only?
I believe that you are missing a component that reads the appropriate data from the Redis server and passes them through the websocket. This could be done in many different ways. Here is a small self-contained example written purely in Python,
import redis
import dash_html_components as html
from gevent import sleep
from dash import Dash
from dash.dependencies import Input, Output
from dash_extensions import WebSocket
from dash_extensions.websockets import SocketPool, run_server
# Read data from Redis and pass it through the web socket.
def ws_handler(ws):
while True:
sleep(2) # delay between updates, here 1s
value = redis.Redis().get('foo').decode("utf-8") # get redis value
ws.send(f"Redis value is [{value}]") # send data
# Inject some dummy data into Redis.
redis.Redis().set('foo', 'bar')
# Create example app.
app = Dash(prevent_initial_callbacks=True)
socket_pool = SocketPool(app, handler=ws_handler)
app.layout = html.Div([html.Div("Listening for message", id="log"), WebSocket(id="ws")])
#app.callback(Output("log", "children"), [Input("ws", "message")])
def update_graph(msg):
return msg['data']
if __name__ == '__main__':
run_server(app, port=5000) # 5000 if the default port
I have been struggling to combine flask_ldap3_login and flask_httpauth into a simple Flask app and test the result against the only free LDAP server that's available: ldap.forumsys.com
There a lot of information out there but none showed how to set all the config parameters correctly to get it to work just right. After a lot of trial and error, I got it to work.
If you have a better/simpler solution, please share!
Here's the code (I'll also be sharing in https://gist.github.com/jeromegit/f987b081afa3cbc09ed47a3089b32079):
(Also check out the information re: the free LDAP server: https://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/ )
from flask import Flask
import logging
from flask_ldap3_login import LDAP3LoginManager, AuthenticationResponseStatus
from flask_httpauth import HTTPBasicAuth
# Extra logging to help troubleshoot
logging.getLogger('flask_ldap3_login').setLevel(logging.DEBUG)
logger = logging.getLogger('flask_ldap3_login')
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
app = Flask(__name__)
auth = HTTPBasicAuth(realm="Private!")
# Setup LDAP Configuration Variables. Change these to your own settings.
# All configuration directives can be found in the documentation.
# See https://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/
config = dict()
# Hostname of your LDAP Server
config['LDAP_HOST'] = 'ldap.forumsys.com'
# The Username to bind to LDAP with
config['LDAP_BIND_USER_DN'] = 'cn=read-only-admin,dc=example,dc=com'
# The Password to bind to LDAP with
config['LDAP_BIND_USER_PASSWORD'] = 'password'
# Base DN of your directory
config['LDAP_BASE_DN'] = 'dc=example,dc=com'
# Users DN to be prepended to the Base DN
#config['LDAP_USER_DN'] = 'ou=scientists'
# Groups DN to be prepended to the Base DN
config['LDAP_GROUP_DN'] = ''
# The RDN attribute for your user schema on LDAP
config['LDAP_USER_RDN_ATTR'] = 'uid'
# The Attribute you want users to authenticate to LDAP with.
config['LDAP_USER_LOGIN_ATTR'] = 'uid'
config['LDAP_GROUP_OBJECT_FILTER'] = '(objectclass=*)'
# Setup a LDAP3 Login Manager.
ldap_manager = LDAP3LoginManager()
# Init the manager with the config
ldap_manager.init_config(config)
#auth.verify_password
def verify_password(username, password):
# Check if the credentials are correct
response = ldap_manager.authenticate(username, password)
if response.status == AuthenticationResponseStatus.success:
return username
else:
print(response.status)
#app.route('/')
#auth.login_required
def index():
return "Hello, {}!".format(auth.current_user())
if __name__ == '__main__':
app.run(port=5003)
I was able to setup the CherryPy HTTPServer to require an SSL client certificate using the following code:
ssl_certificate = os.environ.get("SSL_CERTIFICATE")
ssl_adapter = BuiltinSSLAdapter(
certificate=ssl_certificate,
private_key=os.environ["SSL_PRIVATE_KEY"],
certificate_chain=os.environ.get("SSL_CERTIFICATE_CHAIN")
)
verify_mode = ssl.CERT_REQUIRED
ssl_adapter.context.verify_mode = verify_mode
HTTPServer.ssl_adapter = ssl_adapter
Now I am trying to get the SSL client certification information from my request handler, but I can't figure how. After reading https://github.com/cherrypy/cheroot/blob/master/cheroot/ssl/builtin.py#L419 it seems that the wsgi environment variables should be populated with SSL_CLIENT* variables. I could not find any method/property from the request objectwould allow me to fetch such information
How can I obtain this variables from a request handler ?
I have learned the answer from a conversation with CherryPy maintainers on Gitter.
An CherryPy request object may contain more attributes than those documented in the API source, such attributes are set dynamically once the request object is created as part of the WSGI handling:
https://github.com/cherrypy/cherrypy/blob/master/cherrypy/_cpwsgi.py#L319
...
request.multithread = self.environ['wsgi.multithread']
request.multiprocess = self.environ['wsgi.multiprocess']
request.wsgi_environ = self.environ
...
Knowing this, to obtain the WSGI environment which includes the SSL* variables, we just need to access it by importing the request object:
import cherrypy.request
...
print(cherrypy.request.wsgi_environ)
...
I would like to mask the version or remove the header altogether.
To change the 'Server:' http header, in your conf.py file:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'Microsoft-IIS/6.0'
And use an invocation along the lines of gunicorn -c conf.py wsgi:app
To remove the header altogether, you can monkey-patch gunicorn by replacing its http response class with a subclass that filters out the header. This might be harmless, but is probably not recommended. Put the following in conf.py:
from gunicorn.http import wsgi
class Response(wsgi.Response):
def default_headers(self, *args, **kwargs):
headers = super(Response, self).default_headers(*args, **kwargs)
return [h for h in headers if not h.startswith('Server:')]
wsgi.Response = Response
Tested with gunicorn 18
This hasn't been clearly written here so I'm gonna confirm that the easiest way for the latest version of Gunicorn (20.1.x) is to add following lines into configuration file:
import gunicorn
gunicorn.SERVER = 'undisclosed'
For newer releases (20.0.4): Create a gunicorn.conf.py file with the content below in the directory from where you will run the gunicorn command:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'My WebServer'
It's better to change it to something unique than remove it. You don't want to risk, e.g., spiders thinking you're noncompliant. Changing it to the name of software you aren't using can cause similar problems. Making it unique will prevent the same kind of assumptions ever being made. I recommend something like this:
import gunicorn
gunicorn.SERVER_SOFTWARE = 'intentionally-undisclosed-gensym384763'
You can edit __init__.py to set SERVER_SOFTWARE to whatever you want. But I'd really like the ability to disable this with a flag so I didn't need to reapply the patch when I upgrade.
My mocky-patch free solution, involves wrapping the default_headers method:
import gunicorn.http.wsgi
from six import wraps
def wrap_default_headers(func):
#wraps(func)
def default_headers(*args, **kwargs):
return [header for header in func(*args, **kwargs) if not header.startswith('Server: ')]
return default_headers
gunicorn.http.wsgi.Response.default_headers = wrap_default_headers(gunicorn.http.wsgi.Response.default_headers)
This doesn't directly answer to the question but could address the issue as well and without monkey patching gunicorn.
If you are using gunicorn behind a reverse proxy, as it usually happens, you can set, add, remove or perform a replacement in a response header coming downstream from the backend. In our case the Server header.
I guess every Webserver should have an equivalent feature.
For example, in Caddy 2 (currently in beta) it would be something as simple as:
https://localhost {
reverse_proxy unix//tmp/foo.sock {
header_down Server intentionally-undisclosed-12345678
}
}
For completeness I still add a minimal (but fully working) Caddyfile to handle Server header modification even in manual http->https redirect process (Caddy 2 does it automatically, if you don't override it), which could a bit tricky to figure it out correctly.
http://localhost {
# Fact: the `header` directive has less priority than `redir` (which means
# it's evaluated later), so the header wouldn't be changed (and Caddy would
# shown instead of the faked value).
#
# To override the directive ordering only for this server, instead of
# change the "order" option globally, put the configuration inside a
# route directive.
# ref.
# https://caddyserver.com/docs/caddyfile/options
# https://caddyserver.com/docs/caddyfile/directives/route
# https://caddyserver.com/docs/caddyfile/directives#directive-order
route {
header Server intentionally-undisclosed-12345678
redir https://{host}{uri}
}
}
https://localhost {
reverse_proxy unix//tmp/foo.sock {
header_down Server intentionally-undisclosed-12345678
}
}
To check if it works just use curl as curl --insecure -I http://localhost and curl --insecure -I http://localhost (--insecure because localhost certs are automatically generated as self signed).
It's so simple to setup that you could also think to use it in development (with gunicorn --reload), especially if it resembles your staging/production environment.
I am attempting to run my buildbot master server behind a cherokee reverse proxy with the buildbot instance as cherokee's information source in a round robin reverse proxy layout.
This is the buildbot master.cfg configuration file:-
# -*- python -*-
# ex: set syntax=python:
# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.
# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}
####### BUILDSLAVES
# The 'slaves' list defines the set of recognized buildslaves. Each element is
# a BuildSlave object, specifying a unique slave name and password. The same
# slave name and password must be configured on the slave.
from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave("example-slave", "pass")]
# 'slavePortnum' defines the TCP port to listen on for connections from slaves.
# This must match the value configured into the buildslaves (with their
# --master option)
c['slavePortnum'] = 9989
####### CHANGESOURCES
# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes. Here we point to the buildbot clone of pyflakes.
from buildbot.changes.gitpoller import GitPoller
c['change_source'] = []
c['change_source'].append(GitPoller(
'git://github.com/buildbot/pyflakes.git',
workdir='gitpoller-workdir', branch='master',
pollinterval=300))
####### SCHEDULERS
# Configure the Schedulers, which decide how to react to incoming changes. In this
# case, just kick off a 'runtests' build
from buildbot.schedulers.basic import SingleBranchScheduler
from buildbot.schedulers.forcesched import ForceScheduler
from buildbot.changes import filter
c['schedulers'] = []
c['schedulers'].append(SingleBranchScheduler(
name="all",
change_filter=filter.ChangeFilter(branch='master'),
treeStableTimer=None,
builderNames=["runtests"]))
c['schedulers'].append(ForceScheduler(
name="force",
builderNames=["runtests"]))
####### BUILDERS
# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which slaves can execute them. Note that any particular build will
# only take place on one slave.
from buildbot.process.factory import BuildFactory
from buildbot.steps.source import Git
from buildbot.steps.shell import ShellCommand
factory = BuildFactory()
# check out the source
factory.addStep(Git(repourl='git://github.com/buildbot/pyflakes.git', mode='copy'))
# run the tests (note that this will require that 'trial' is installed)
factory.addStep(ShellCommand(command=["trial", "pyflakes"]))
from buildbot.config import BuilderConfig
c['builders'] = []
c['builders'].append(
BuilderConfig(name="runtests",
slavenames=["example-slave"],
factory=factory))
####### STATUS TARGETS
# 'status' is a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot/status/*.py has a variety to choose from,
# including web pages, email senders, and IRC bots.
c['status'] = []
from buildbot.status import html
from buildbot.status.web import authz, auth
authz_cfg=authz.Authz(
# change any of these to True to enable; see the manual for more
# options
auth=auth.BasicAuth([("pyflakes","pyflakes")]),
gracefulShutdown = False,
forceBuild = 'auth', # use this to test your slave once it is set up
forceAllBuilds = False,
pingBuilder = False,
stopBuild = False,
stopAllBuilds = False,
cancelPendingBuild = False,
)
c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
####### PROJECT IDENTITY
# the 'title' string will appear at the top of this buildbot
# installation's html.WebStatus home page (linked to the
# 'titleURL') and is embedded in the title of the waterfall HTML page.
c['title'] = "Pyflakes"
c['titleURL'] = "http://divmod.org/trac/wiki/DivmodPyflakes"
# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server (usually the html.WebStatus page) is visible. This
# typically uses the port number set in the Waterfall 'status' entry, but
# with an externally-visible host name which the buildbot cannot figure out
# without some help.
c['buildbotURL'] = "http://localhost:8010/"
####### DB URL
c['db'] = {
# This specifies what database buildbot uses to store its state. You can leave
# this at its default for all but the largest installations.
'db_url' : "sqlite:///state.sqlite",
}
# change any of these to True to enable; see the manual for more
# options
auth=auth.BasicAuth([("pyflakes","pyflakes")]),
And this is the cherokee configuration:-
Unfortunately, I get 502 Bad gateway when I go to my web url but on the other hand, I know that my buildbot master server instance is working correctly because going to the same web url and appending :8010 behind the web url gives me the "Welcome to the Buildbot ..." page.
Is your proxy on the same machine as the buildbot? If not, you will need to adjust the URL in cherokee, to point to the machine running buildbot (localhost points to the machine cherokee is running on).
In any case, c['buildbotURL'] should be changed to point to the public URL that the buildbot is available under (i.e. what cherokee exposes, rather than the URL being proxied).