I created a basic coding in visualbasic to download from NSE website.
While the coding still downloads the previous years data, it gives an download error for the current new year.
The RAW URL is https://www.nseindia.com/products/content/equities/equities/archieve_eq.htm If you choose a date (say today) and then select BHAVCOPY report, the site will provide you with a link to download the csv.zip file.
However, if you click on the link directly (https://www.nseindia.com/content/historical/EQUITIES/2017/JAN/cm02JAN2017bhav.csv.zip), the URL returns an error: Access Denied
You don't have permission to access "THE LINK" on this server.
Reference #18.11367a5c.1483362327.35d38c1b
What might be the problem with change in year?
i also facing same issue. fixed by adding 2 http header property.
"User-Agent" : "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
"Referer" : "https://www1.nseindia.com/products/content/equities/equities/archieve_eq.htm"
After a bit tweaking I noticed it was something to do with browser. Blocked the cookies and everything is working fine.
Related
This is a part of the code I am using:
req = urllib.request.Request(url, headers = user_agent)
Then I have the following commands:
resp = urllib.request.urlopen(req)
resp_data = resp.read()
print(resp_data)
When I read the command line output from print(resp_data) I see the following message:
Loading\n \n\n \n Unsupported Browser\n Please use IE 10+, Microsoft Edge, Chrome, Firefox, or Safari.\n We apologize for any inconvenience.
Clearly, this reads that the website I am requesting sees the browser Python is connecting with as invalid. I am not sure how to remedy this...
Currently, my user_agent variable is coded as follows:
user_agent = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
I have researched around and have played with what Google says to be valid user agents; however, I have not found one that works.
I am very new to the urllib module and, honestly, Python in general. Any help would be greatly appreciated!
I was trying an IOT project where in I want to update the database remotely using ESP8266-01 module. I have the php file to update the database, but to execute that I am trying with GET method. Unfortunately, it wan't working. The server received the request in the access.log, but wan't updating the database. Wanted to debug, so I had this question.
The entry in the access.log is as follows:
192.168.43.150 - - [18/Mar/2017:20:23:40 +0000] "GET collectdata.php?status=1 HTTP/1.1\r\nHost: 192.168.43.92\r\n\r\n" 400 0 "-" "-"
This looks wrong: GET collectdata.php...
That needs to be a full path, eg. GET /collectdata.php or GET /scripts/collectdata.php, or similar.
The 400 response code you're seeing in the log means "Bad request", and the lack of a leading slash (and the rest of the path, if needed) is what Apache is complaining about.
We have a Tomcat 7 running behind an Apache2 Server, connected via AJP.
Some AJAX-Requests we receive at the Tomcat do not contain any request Parameters.
This applies to GET and to POST requests as well.
Inspecting the Apache access_log the requests were all answered with a HTTP Status Code of 103:
IP - - [SNIP] "POST /redacted/ticket/1234567 HTTP/1.1" 103 655 "https://redactedhost.com/redacted/ticket/1234567" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"
I can't figure where this Status Code comes from. Has anybody ever heard of it? As far as I have examined it is not a Custom Response from our Application and it is not defined in our Tomcat or Apache configuration.
HTTP 103 has been approved as status code for indicating hints to client.
More details here IETF
Official IANA Registry [1] says that code 103 is unassigned.
Some googling finds "103 Checkpoint" from "Resumable HTTP Requests Proposal" [2] [3] (I don't know whether that document in [3] is up-to-date, it is just from first page of search results. Usually such proposals should go further as an IETF draft). Note that it is not official, and response code 308 from [3] has already been assigned for different purpose by RFC 7538 [1].
As far as I have examined it is not a Custom Response from our Application
If it is not in plain view, it may be one of filters that you are using. Look through your libraries. You may try running with a debugger.
Do you see that response code at Apache HTTPD side only, or in Tomcat access log as well?
The Status Code 103 was an "unofficial" implementation which ended up in some (very few) products/software source codes, reason why people stumbled upon it in (rare) occasions.
History tells that the Status Code 103 was planned to respond when the request was installation/implementation dependent, which means its should appear when something (required) was missing or faulty in the Server installation or Network.
The message "Access denied while creating Web Service" was originally intended for the Status Code 103, but only few implementations used it. There were also cases it was simply translated as "Checkpoint".
For a while those messages made little to no sense, and the Status Code 103 was kept listed as "unofficial", until when it was deprecated even before becoming official, which happened some point in time around 2012.
However, in December 20, 2017, suggested by K. Oku, the IETF (Internet Engineering Task Force) retrieved and reset the Status Code 103 from "Deprecated" to "Experimental", retrofitting it as "Early Hints", as it has been listed in the RFC 8297.
You can read more its new purpose and utilization here: An HTTP Status Code for Indicating Hints
In 2018 The Status Code 103 was officially listed in the IANA HTTP Registry, even before being set as official by the IETF. Based on that, the odds are that the new meaning of the Status Code 103 should be safe to use and soon enough it shall became "official" by the IETF too. That is how it has been happening for a long time for other Status Codes.
I must express my appreciation and thank Ujjwal Gulecha for taking the time to bring this information forward on an old thread. It helps a lot keeping the information up-to-date in Stack Overflow when something in the industry change and affect the answers provided. Good job, bother!
This method that I wrote was working fine a week ago but now it downloads an incomplete file. The original file is nearly 10mb but the file that is being downloaded is 2k. My code is basically this
Dim URL as string = "http://www.cqc.org.uk/sites/default/files/cqc_locations_export.csv"
Dim path as string = "C:\temp"
Dim webClient As New WebClient
webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729)")
webClient.DownloadFile(URL, path)
Any idea what is going wrong here ?
Cheers
This is not my area, but you may be missing the size of the file, if it's binary, char encoding and other header data. Also, this 2k of file maybe a part of the file or metadata.
The Second argument to DownloadFile requires a full filename not just a path (and the root of the C:\ drive is protected on windows 7 so the method may throw an exception if you try to write here)
Note: Don't forget to dispose of the webclient when done.
Note 2: I would suggest you should avoid naming collisions by using names other than the .NET class names. Don't forget VB is case insensitive (unlike C#)
The following works fine for me:
Dim URL As String = "http://www.cqc.org.uk/sites/default/files/cqc_locations_export.csv"
Dim filename As String = "C:\temp\temp.csv"
Using wc As New WebClient
wc.Headers.Add("user-agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729)")
wc.DownloadFile(URL, filename)
End Using
It could be a problem with the site that is hosting the file, or how that site handles GET requests. I had the same problem downloading a URI I had shared from DropBox. The file was about 3 MB, but only about the first 112 KB was downloaded. When I downloaded the same file from Screencast.com, the whole file was downloaded.
On ground you may want to try three things :
Run your application with Admin rights as C drive is protected in
Windows 7 and above.
Temporary stop your antivirus program and try downloading again.
Only dispose webClient when download is complete (On DownloadFileCompleted Event)
The docs seems to suggest either (two different variations)
Dim client As New WebClient()
or
Dim client As WebClient
I want to know the analytical algorithm that how to get the uv number from apache log.
the log like this:
218.19.140.242 – - [10/Dec/2010:09:31:17 +0800] “GET
/query/trendxml/district/todayreturn/month/2009-12-14/2010-12-09/haizhu_tianhe.xml HTTP/1.1″
200 1933 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Gecko/20100722
Firefox/3.6.8 (.NET CLR 3.5.30729)”
thanks.
or
who knows how awstats or webtrends works to get the unique visitor.
I had known , In awstats( http://awstats.sourceforge.net/docs/awstats_glossary.html ), Unique Visitors are tracked by IP address, so if multiple users are accessing your site from the same IP (such as a home or office network), they will be counted as a single unique visitor.
To know more ,here http://digdeeply.info/archives/03191870.html