DefaultHeadersMiddleware can do all UserAgentMiddleware can do? So what's the meaning of UserAgentMiddleware?
I believe it's just for the reason of simplicity.
User agent is a very common thing to change, so being able to set it through the USER_AGENT setting or the spiders user_agent attribute can be very helpful at times.
For example, changing the user agent from the command line is as simple as:
scrapy <command> -s USER_AGENT=something
Doing the same thing where you change the default headers would be much more complicated.
Related
I have a command in discord.py where I would like to have, for example, #has_permissions(administrator=True), but I want to be able to run the command with, say, a custom check that checks if it is a certain user, so that the user/role/whatever can run the command, regardless of whether or not they have the permission. So I guess my question is this: is there a way to allow a command if a user passes at least 1 check? Thank you in advance!
Yes you can write your own check as described in the docs here: https://discordpy.readthedocs.io/en/master/ext/commands/api.html?highlight=has%20permissions#discord.ext.commands.check
def check_if_it_is_me(ctx):
return ctx.message.author.id == 85309593344815104
#bot.command()
#commands.check(check_if_it_is_me)
async def only_for_me(ctx):
await ctx.send('I know you!')
I found an answer! After reviewing the docs, I found the #commands.check_any() decorator. You can input checks for it to check, and if any of the inputted checks will be passed, it adds that check.
Read more about this decorator in the documentation.
Is there a way to make the logged user (on superset) to make the queries on impala?
I tried to enable the "Impersonate the logged on user" option on Databases but with no success because all the queries run on impala with superset user.
I'm trying to achieve the same! This will not completely answer this question since it does not still work but I want to share my research in order to maybe help another soul that is trying to use this instrument outside very basic use cases.
I went deep in the code and I found out that impersonation is not implemented for Impala. So you cannot achieve this from the UI. I found out this PR https://github.com/apache/superset/pull/4699 that for whatever reason was never merged into the codebase and tried to copy&paste code in my Superset version (1.1.0) but it didn't work. Adding some logs I can see that the configuration with the impersonation is updated, but then the actual Impala query is with the user I used to start the process.
As you can imagine, I am a complete noob at this. However I found out that the impersonation thing happens when you create a cursor and there is a constructor parameter in which you can pass the impersonation configuration.
I managed to correctly (at least to my understanding) implement impersonation for the SQL lab part.
In the sql_lab.py class you have to add in the execute_sql_statements method the following lines
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
where cursor_kwargs is defined in db_engine_specs/impala.py as the following
#classmethod
def get_configuration_for_impersonation(cls, uri, impersonate_user, username):
logger.info(
'Passing Impala execution_options.cursor_configuration for impersonation')
return {'execution_options': {
'cursor_configuration': {'impala.doas.user': username}}}
#classmethod
def get_cursor_configuration_for_impersonation(cls, uri, impersonate_user,
username):
logger.debug('Passing Impala cursor configuration for impersonation')
return {'configuration': {'impala.doas.user': username}}
Finally, in models/core.py you have to add the following bit in the get_sqla_engine def
params = extra.get("engine_params", {}) # that was already there just for you to find out the line
self.cursor_kwargs = self.db_engine_spec.get_cursor_configuration_for_impersonation(
str(url), self.impersonate_user, effective_username) # this is the line I added
...
params.update(self.get_encrypted_extra()) # already there
#new stuff
configuration = {}
configuration.update(
self.db_engine_spec.get_configuration_for_impersonation(
str(url),
self.impersonate_user,
effective_username))
if configuration:
params.update(configuration)
As you can see I just shamelessy pasted the code from the PR. However this kind of works only for the SQL lab as I already said. For the dashboards there is an entirely different way of querying Impala that I did not still find out.
This means that queries for the dashboards are handled in a different way and there isn't something like this
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
My gut (and debugging) feeling is that you need to first understand the sqlalchemy part and extend a new ImpalaEngine class that uses a custom cursor with the impersonation conf. Or something like that, however it is not simple (if we want to call this simple) as the sql_lab part. So, the trick is to find out where the query is executed and create a cursor with the impersonation configuration. Easy, isnt'it ?
I hope that this could shed some light to you and the others that have this issue. Let me know if you did find out another way to solve this issue, or if this comment was useful.
Update: something really useful
A colleague of mine succesfully implemented impersonation with impala without touching any superset related, but instead working directly with the impyla lib. A PR was open with the code to change. You can apply the patch directly in the impyla src used by superset. You have to edit both dbapi.py and hiveserver2.py.
As a reminder: we are still testing this and we do not know if it works with different accounts using the same superset instance.
I´m currently writing tests for my application and therefore, I have to test some click.group commands I defined:
Let´s say I defined them like:
#click.group(cls=MyGroup)
#click.pass_context
def myapp(ctx):
init_stuff()
#myapp.command()
#click.option('--myOption')
def foo(myOption: str) -> None:
do_stuff() # change some files, print, create other files
I know that I could use the CliRunner from click.testing. However, I just want to make sure, that the command is called, but I DONT WANT it to execute any code (for example by applying the CliRunner.invoke()).
How could this be done?
I couldn´t come up with a solution using mocking with foo for example. Or do I have to execute code lets say using the isolated_filesystem() which CliRunner provides?
So the question is: What would be the most efficient way to test my commands when defined like shown above?
Many thanks in advance
You could add a --dry-run flag to your group or some commands, and save it it inside the context, and if the flag is enabled, do not execute any code. Then you can use CliRunner.invoke() with the --dry-run flag enabled and just check your invocations have happened, without actually executing the code.
The docs of Scrapy says about HttpProxyMiddleware is like this:
This middleware sets the HTTP proxy to use for requests, by setting the proxy meta value for Request objects.
Like the Python standard library modules urllib and urllib2, it obeys the following environment variables:
http_proxy
https_proxy
no_proxy
You can also set the meta key proxy per-request, to a value like http://some_proxy_server:port or http://username:password#some_proxy_server:port. Keep in mind this value will take precedence over http_proxy/https_proxy environment variables, and it will also ignore no_proxy environment variable.
docs:https://doc.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=Proxy#module-scrapy.downloadermiddlewares.httpproxy
But there are no examples in the docs.
I have no ideas how to write a HttpProxyMiddleware.
Are there any suggestions?
In settings.py just do this.
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 100
}
And then while yielding each request do this
yield Request(meta={'proxy': "http://%s"%(random.choice(["IP:PORT", "IP:PORT"]))})
That's it!
You don't need to write one. HttpProxyMiddleware already exists in Scrapy.
As docs state, there are two ways of letting Scrapy know you need your requests to go through a proxy:
Setting environment variables
(e.g. from the command line)
export http_proxy="http://username:password#host:port"
You can also set the meta key "proxy" per-request, to a value like http://some_proxy_server:port or http://username:password#some_proxy_server:port.
Keep in mind this value will take precedence over http_proxy/https_proxy environment variables, and it will also ignore no_proxy environment variable
e.g.
yield Request("http://google.com", meta={'proxy':'http://username:password#some_proxy_server:port'}, callback=self.some_method)
I'm looking for a variable that can tell me which number 'won' the call on a multi-target Dial command.
Example:
Dial(SIP/1000&SIP/1001&SIP/1002,30)
Set(the_unlucky_winner=${...})
I'm not getting anything from the ${DIALEDPEERx} variables. Sounds like these vars are broken but I don't know if this is what I should be using.
Ancient version 1.2.14 deployed at this site. All clients are SIP
Thanks anyone
Only realistic way do that - cal via Local channle like freepbx do(check freepbx.org source) or use Macro on answer(i am afraid not work in 1.2)
Parse the contents of the CDR record for the file. One of the fields is dstchannel which will hold a value like SIP/1002-9786b0b0.
Also keep in mind that the call variable stack is wiped on hangup, unless you have an "h" (hangup) extension defined for the context. So, you can most easily handle your post-call processing there.
Further Reading:
http://www.asteriskdocs.org/en/3rd_Edition/asterisk-book-html-chunk/asterisk-SysAdmin-SECT-1.html
Please Note:
if this answer turns out to solve your problem, please "accept" it for the benefit of others trying to solve the same problem later
Hi all I have a solution to this problem. It is working fine for both normal dial and multi target dial.
In dialstring add a macro, here I am adding "followme" macro.
M(followme)
$agi->exec("dial", "SIP/6001#sip.example.com&SIP/6002#sip.example.com,rtTgM(followme)");
Then after call is answered it ll go to context
[macro-followme]
In this context you write one script to get the connected calls information by
$dstchannel=$agi->get_variable("DIALEDPEERNUMBER");
The way I managed to do it is as follows
Dial(SIP/1000&SIP/1001&SIP/1002,30,M(whoanswered))
[macro-whoanswered]
exten => s,1,NoOp(${CHANNEL})
You will see that the actual extension that answered is containes in ${CHANNEL}
If 1001 answered the channel will be something like SIP/1001-00017cf1
Just use the CUT command to cut it by / and -