I'm trying to scrape data from a website but I have no experience with scraping or APIs except for making a Discord Bot once. So I followed the steps described here to find the API:
http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api
The Request URL in the Headers tab with the important information is this one:
https://api.amiami.com/api/v1.0/item?gcode=FIGURE-119023&lang=eng
When I try to open this page, like he does, it only returns:
{"RSuccess":false,"RValue":{"HttpStatusCode":400},"RMessage":"Invalid access."}
If you want to try getting the Request URL yourself, the original page I used was:
https://www.amiami.com/eng/detail/?gcode=FIGURE-119023
Removing the language argument doesn't seem to change anything either. So I guess there's something that detects that I'm not accessing it in a normal way. Any Ideas on how to fix this?
Related
I know this is probably strictly case-specific, but I do feel like I encounter this problem a lot so I will make an effort to try and understand it better.
I am new to using APIs, but I have never succeeded in using one without copying someone's code. In this case, I can't even find any examples on forums, nor in the API documentation.
I'm trying to pull my balance value from my investment bank "NordNet" to scroll, amongst other things, on an Arduino display I've made. Right now I use python Selenium to automatically but "physically" login to NordNet and grab my balance from the DOM. As I'm afraid I might get "punished" for such botted behavior, and because the script is fairly high maintenance (as the HTML changes over time), I would obviously much rather get this information through NordNet's new API.
Link to NordNets API doc
Every time I try to utilize an API doc it's always the same, it looks easy, but I can never get it to work.
This time I tried to just play a little with the API before exploring further.
I use PostMan to send the simplest request:
https://www.nordnet.se/api/2
And I get a successful code 200 JSON response.
I then try to take it a step further to access my account data using this endpoint:
https://www.nordnet.se/api/2/accounts
For this one, I obviously need some authentication of some sort
The doc looks like this:
So I set my PostMan client up like this and get the response showcased:
I've put my NordNet login into the "Auth" tab as "basic auth" and I then see PostMan encrypts this info some way, in the "Headers" tab.
I'm getting an unauthorized response code and I have no idea why. Am I using PostMan wrong (probably)? Is the API faulty (probably not)? There is a mention of a session_id that should contain both password and username? Maybe something completely else...
I hope you can help
The documentation says to use session_id as username and password for that api ,
so try logging in and then get the session id (try with both sid and ssid) . from network tab and pass it as username and password for authorization .
sid- is for http and ssid for https i guess , try with both
I have a vb.NET App that uses System.Net.WebClient to query an API. I'm able to get the information I'm requesting just fine.
The people that supply the API are requesting that I
"set a custom User header when requesting data to determine the source application."
Am I supposed to pre-send something first, or append something to the url for the WebClient to processes? The API only accepts get requests and it doesn't have a parameter for an identification.
I'm stuck in terminology here. A search for that phrase, here, came up with server-side topics so I don't know what to look for. Can someone translate?
I am trying to to add a custom Tumblr follow button to my site and I've been trying to figure out how the follow api works.
I've been using https://apigee.com/console/tumblr to test various requests and so for I can make the get requests work, but not the post requests. Specifically, I would like to be able to follow a blog.
For instance, if i want to follow mcupdate The post request should look like this
http://api.tumblr.com/v2/user/follow?url=http%3A%2F%2Fmcupdate.tumblr.com
This returns 404 page not found. As do any and all variations that I can think of. Since the get requests work fine, I am assuming that the credentials that apigee.com generates are fine. Does anyone know what is the problem?
Thanks.
I am playing around with Github's API and I noticed that they allow anyone to request all users that have signed up in chronological order.
https://api.github.com/users
http://developer.github.com/v3/users/
I was trying to get the second page but for some reason their pagination isn't working for me. I wasn't sure what I was doing wrong.
https://api.github.com/users?page=2
https://api.github.com/users?start_page=2
http://developer.github.com/v3/ Under "Pagination".
Anyone know the right way to do this?
Check out the returned HTTP headers for the https://api.github.com/users resource. Specifically, look for the Link header, which will look like this:
Link:<https://api.github.com/users?since=135>; rel="next", <https://api.github.com/users{?since}>; rel="first"
So, what you need to do is do an HTTP GET on https://api.github.com/users?since=135 to get the next page. After that, check the Link header again and you will get to the next page, etc. Also notice the provided URI template https://api.github.com/users{?since} which enables you to start at any id.
I am trying to scrape a this page with scrapy:
http://www.barnesandnoble.com/s?dref=4815&sort=SA&startat=7391
and the response which I get is different than what I see in the browser. Browser response has the correct page, while scrapy response is:
http://www.barnesandnoble.com/s?dref=4815&sort=SA&startat=1
page. I have tried with urllib2 but still have the same issue. Any help is much appreciated.
I don't really understand the issue, but usually a different response for a browser and scrapy is caused by one these:
the server analyzes your User-Agent header, and returns a specially crafted page for mobile clients or bots;
the server analyzes the cookies, and does something special when it looks like you are visiting for the first time;
you are trying to make a POST request via scrapy like the browser does, but you forgot some form fields, or put wrong values
etc.
There is no universal way to determine what's wrong, because it depends on the server logic, which you don't know. If you are lucky, you will analyze and fix all the mentioned issues and will make it work.