Python mechanize br.open() 503 Error: service unavailable? - apache

I am running an apache 2.4 server with PHP 5.6. I have a form called form1.html that sends it's data to another PHP script that prints out the form's data.
I am trying to use Python with mechanize to automatically fill out that form. I need to do all this for automation for work.
So with this code:
import mechanize
br=mechanize.Browser()
br.set_handle_robots(False)
response = br.open("http://127.0.0.1/form1.html")
... it then gives this:
Traceback (most recent call last):
File "mechTest.py", line 21, in <module>
File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 203, in open
return self._mech_open(url, data, timeout=timeout)
File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 255, in _mech_open
raise response
mechanize._response.httperror_seek_wrapper: HTTP Error 503: Service Unavailable
I am stuck on this issue, and I need mechanize to work. Is there anyway to fix this, such as fixing the browser header? Thanks.

Related

recognize_google gives "recognition request failed: Forbidden" error

I am using an Ubuntu 20.04 Virtual Machine and running the following code :
import speech_recognition as sr
# Initialize recognizer class (for recognizing the speech)
r = sr.Recognizer()
# Reading Audio file as source
# listening the audio file and store in audio_text variable
with sr.AudioFile('I-dont-know.wav') as source:
audio_text = r.listen(source)
# recoginize_() method will throw a request error if the API is unreachable, hence using exception handling
try:
# using google speech recognition
text = r.recognize_google(audio_text)
print('Converting audio transcripts into text ...')
print(text)
except Exception as ex:
print(ex)
It gives the error message "recognition request failed: Forbidden"
I have also tried generating a key from google cloud and setting its path on the device but still the same error occurs.
If anyone can suggest something, it would be great help. Any alternate and efficient methods for speech to text conversion can also be suggested.
Thanks

DefaultCredentialsError for running query in Jupyter notebook

I am following this Google cloud tutorial. Running codes in Jupyter notebook. Step 5 is fine and when I run step 7 I get this error:
DefaultCredentialsError Traceback (most recent call last)
<ipython-input-8-a1696640ccb4> in <module>
----> 1 get_ipython().run_cell_magic('bigquery', '', 'SELECT\n source_year AS year,\n COUNT(is_male) AS birth_count\nFROM `bigquery-public-data.samples.natality`\nGROUP BY year\nORDER BY year DESC\nLIMIT 15\n')
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2379 with self.builtin_trap:
2380 args = (magic_arg_s, cell)
-> 2381 result = fn(*args, **kwargs)
2382 return result
2383
The DefaultCredentialsError means that the Application Default Credentials (ADC) that the client library is trying to use for authentication are not properly set.
You may have missed the step in the section Setting up a local Jupyter environment with the link for Getting Started with Authentication.
In short, you need to set your default credentials by running a cell with the following code.
export GOOGLE_APPLICATION_CREDENTIALS="path/to/key.json"
Where your path points to a valid service account key file.
You might also be able to set default credentials instead with the following in a cell if you have the gcloud sdk installed locally:
gcloud auth application-default login

Agent connection server error in robot framework while debugging only/Script is running fine [duplicate]

For some reason I get the following error only when I open up a nested webdriver instance. No idea what is happening here.
I am using Windows 10, geckodriver 0.21.0, and Python 3.7.
ConnectionAbortedError: [WinError 10053]
An established connection was aborted by the software in your host machine
Part of Script That Is Working Fine
tab_backers = ff.find_element_by_xpath('//a[#gogo-test="backers_tab"]')
try:
funding_backers_count = int(''.join(filter(str.isdigit, str(tab_backers.text))))
except ValueError:
funding_backers_count = 0
if funding_backers_count > 0:
tab_backers.click()
see_more_backers = WebDriverWait(ff, 10).until(
EC.element_to_be_clickable((By.XPATH, '//ui-view//a[text()="See More Backers"]'))
)
clicks = 0
while clicks < 0:
clicks += 1
ff.WebDriverWait(ff, 5).until(
see_more_backers.click()
)
for container in ff.find_elements_by_xpath('//ui-view//div[#class="campaignBackers-pledge ng-scope"]'):
backers_profile = container.find_elements_by_xpath('./*/div[#class="campaignBackers-pledge-backer-details"]/a')
if len(backers_profile) > 0:
backers_profile = backers_profile[0].get_attribute('href')
else:
backers_profile = 'Unknown'
backers_name = safe_encode(container.find_element_by_xpath('(./*/div[#class="campaignBackers-pledge-backer-details"]/*)[1]').text)
backers_timestamp = container.find_element_by_xpath('./*/div[#class="campaignBackers-pledge-backer-details"]/div[contains(#class, "campaignBackers-pledge-backer-details-note")]').text
backers_contribution = container.find_element_by_xpath('./*//*[contains(#class, "campaignBackers-pledge-amount-bold")]').text
if backers_contribution != 'Private':
backers_contribution = int(''.join(filter(str.isdigit, str(backers_contribution))))
if backers_profile != 'Unknown':
Part of Script Causing System to Abort Connection
_ff = create_webdriver_instance()
_ff.get(backers_profile)
_ff.quit()
Traceback
Traceback (most recent call last):
File "C:\Users\Anthony\Desktop\test.py", line 271, in <module>
backers_profile = container.find_elements_by_xpath('./*/div[#class="campaignBackers-pledge-backer-details"]/a')
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 381, in find_elements_by_xpath
return self.find_elements(by=By.XPATH, value=xpath)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 680, in find_elements
{"using": by, "value": value})['value']
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 318, in execute
response = self.command_executor.execute(driver_command, params)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 472, in execute
return self._request(command_info[0], url, body=data)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 495, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1055, in _send_output
self.send(chunk)
File "C:\Users\Anthony\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 977, in send
self.sock.sendall(data)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
geckodriver.log
Here it is in a codepen, since it is wayyy too long!
create_webdriver_instance Function
def create_webdriver_instance():
options = Options()
options.add_argument('-headless')
try:
ua_string = random.choice(ua_strings)
profile = webdriver.FirefoxProfile()
profile.set_preference('general.useragent.override', ua_string)
return webdriver.Firefox(profile) # profile, firefox_options=options
except IndexError as error:
print('\nSection: Function to Create Instances of WebDriver\nCulprit: random.choice(ua_strings)\nIndexError: {}\n'.format(error))
return webdriver.Firefox() # firefox_options=options
Does anybody have any idea whatsoever what might be causing the connection to abort?
This error message...
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
...implies that the initialization of a new WebBrowsing Session i.e. Firefox Browser session was aborted.
An established connection was aborted by the software in your host machine
As per your code attempt the error is clearly coming out create_webdriver_instance() function which contains:
try:
ua_string = random.choice(ua_strings)
profile = webdriver.FirefoxProfile()
profile.set_preference('general.useragent.override', ua_string)
return webdriver.Firefox(profile)
And:
except IndexError as error:
print('\nSection: Function to Create Instances of WebDriver\nCulprit: random.choice(ua_strings)\nIndexError: {}\n'.format(error))
return webdriver.Firefox()
So it is not exactly clear from which function you are facing this issue among return webdriver.Firefox(profile) or webdriver.Firefox().
Perhaps a closer look at the logs within codepen suggests the error is coming out from webdriver.Firefox(profile).
Reasons
There can be multiple reason behind this error:
Presence of anti-virus softwares.
Firewall blocking the ports.
Network Configuration.
Problem can be caused by CORS.
Due to enabling of HTTP keep-alive connections
Solution
The initial step would be find out if any software is blocking the connection to/from the server in your computer. Apart from that the probable solutions are:
Disable anti-virus softwares.
Disable firewall.
Ensure that /etc/hosts on your system contains the following entry:
127.0.0.1 localhost.localdomain localhost
As per Connection was aborted by the software in your host machine you need to allow localhost routes like http://localhost:8080/reactive-commands
As per Keep-Alive connection to geckodriver 0.21.0 dropped after 5s of inactivity without re-connection using Selenium Python client
AutomatedTester: This issue is not because we are not connected at the time of doing a request. If that was the issue we would be getting a httplib.HTTPConnection exception being thrown. Instead a BadStatusLine is thrown when we do a connection and close it and try parse the response. Now this could be the python stdlib bug, httplib bug or selenium bug. A Python client to replace urllib with something else that does not exhibit the same defect with Keep-Alive connections is WIP.
andreastt: The geckodriver team is working on extending the server-side timeout value to something more reasonable. As I said, this would help mitigate this issue but not fundamentally fix it. In any case it is true that five seconds is probably too low to get real benefit from persistent HTTP connections, and that increasing it to something like 60 seconds would have greater performance.
Conclusion
Selenium 3.14.0 has just been released. If you are affected by this issue please upgrade accordingly.
References:
Flask broken pipe with requests
Cross-Origin Resource Sharing (CORS)
Connection was aborted by the software in your host machine
[WinError 10053] with the 0.21.0
Keep-Alive connection to geckodriver 0.21.0 dropped after 5s of inactivity without re-connection using Selenium Python client
Support keep alive connections
Struct hyper::server::Server
Urllib3
As documentation says:
Software caused connection abort. An established connection was
aborted by the software in your host computer, possibly due to a data
transmission time-out or protocol error.
Possible reasons:
Timeout or other network-level error.
Network connection died
Firewall closed the connection because it was open too long
Connection was closed before process has been finished
AntiVirus blocks the connection
and etc.
Also try to downgrade geckodriver 0.21.0 to geckodriver 0.20.1. You can download it here. It seems to be a problem with geckodriver 0.21.0 https://stackoverflow.com/a/51236719/8625512
PS:
options.add_argument('-headless')
should be:
options.add_argument('--headless')
This problem happened to me and as the error was intermittent, I initially believed that it was some firewall or antivirus problem, but it was much simpler.
I had a form that was being submitted twice when the SEND button was clicked. The button was set to type="submit" and a javascript code submitted this form when this button was clicked. I changed the button to type="button" and the problem was solved.
Another reason:
When a page served by python is loading, user may cancel the loading before transmission is complete, by pressing Esc or by clicking 'X'(Stop loading this page button). This is usually done when the server is slow.
E.g. if user navigates to your page and Press Esc before the image is fully loaded, this error will occur. In that case, you have to Handle it.
Windows has implemented Controlled folder access, which is to block un-authorized app to access your important files(means almost anything installed by windows). This is to prevent data encryption and ransom by malware.
Windows will pop-out notification if it blocks any app.
You can allow apps/programs access through Controlled folder access.
I solved this problem with a browser configuration in Firefox: Go to Settings -> general and scroll down to network configuration. in the window that opens I go to the end I selected DNS PROXY USE SOCKS and activated DNS over HTTPS. I chose all three options available.
I had the same issue. The issue is happening if the same request is sending multiple times to the server.
This problem happened to me and as the error was intermittent, I initially believed that it was some firewall or antivirus problem, but it was much simpler.
I had a form that was being submitted twice when the SEND button was clicked. The button was set to type="submit" and a javascript code submitted this form when this button was clicked. I changed the button to type="button" and the problem was solved.
Another reason in browser (not webdriver): An extension that doesn't update frequently. In my case this issue was solved when disabling an extension that had some deprecated code. Try on an incognito window for a quick check.

multiple file upload to bigquery

I am trying to do multiple file upload simultaneously to google big-query using command line tool. I got following error :
BigQuery error in load operation: Could not connect with BigQuery server.
Http response status: 503
Http response content:
Service Unavailable
Any way to workaround this problem ?
How do I upload multiple files simultaneously to google big-query using command line tool.
Multiple file upload should work (and we use it every day). If you're getting a 503, that indicates something is wrong with the service. One thing you might want to make sure of is that if you're using a * in your command line that you have it quoted so that the shell doesn't expand it automatically before it gets passed to bq.
If you're getting a 503 error, can you retry the command the flag --apilog=- (this needs to be one of the first params) which will dump the interaction with the server to stdout. The problem may be obvious from that log, but if it isn't can you update your question with the relevant portions of the log? If you're not comfortable posting that information on a public forum, can you e-mail it to me at tigani at google dot com?

Phirehose Setup Problems

Trying to use the Phirehose class from https://github.com/fennb/phirehose
After uploading everything to the server, when i try to run sample.php i get the following errors. I've tried to manually run through the code but can't get to the bottom of this one. Any help would be greatly appreciated...
Phirehose: Connecting to twitter stream: http://stream.twitter.com/1/statuses/sample.json with params: array ( 'delimited' => 'length',)
Phirehose: Resolved host stream.twitter.com to 199.59.148.138
Phirehose: Connecting to 199.59.148.138
Phirehose: Connection established to 199.59.148.138
Phirehose: HTTP failure 1 of 20 connecting to stream: HTTP ERROR : (). Sleeping for 10 seconds.
So, there are multiple different versions of the script. The .zip availble for phirehose in githubs download section is a different version than just copying down the most recent version of the script.
Most recent version of the script appears to work perfectly.