Is there any way to download the own direct messages to archive them?
The Twitter API limits the call to the latest 200 DMs, which cannot download a full archive for longer conversations.
The official Twitter Archive seems not to contain the messages at all. And most thirdparty services (which you might not want to let them access your messages anyway) will be using the api and the best they can do is to poll often enough not to miss the 200 DM limit.
Is there any other way to get the messages from twitter? Scrolling back on the site seems to work, but they always load older messages in small steps and copy&paste from there gives an rather ugly result, too.
It does not need to full twitter-api information, just handle, time and message (maybe media links, if possible) should be available.
I have created a tool (https://github.com/Mincka/DMArchiver) to download my direct messages, with the ability to also download the uploaded images, videos and GIFs (as MP4).
Because it does not rely on the API, it is possible to download more than 200 messages. The script just simulate the "scrolling method" described by dimethylarginine and parse the result.
The main idea is to make requests in loop by calling the following URL with a valid auth_token cookie value for the authentication and parse the json response:
https://twitter.com/messages/with/conversation?id=1337&max_entry_id=1337
The max_entry_id value is not required for the first request. You need to use the value of the min_entry_id variable in the response as the new max_entry_id in each subsequent iteration to get the next 20 older tweets. When max_entry_id is not in the json response, you are at the begin of the thread.
Some headers are also required to get a proper response from Twitter:
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0'
'Accept': 'application/json, text/javascript, */*; q=0.01'
'X-Requested-With': 'XMLHttpRequest'
Currently, the output of the tool is only available as an IRC-like conversation but I may add other output styles in the future (HTML, JSON, XML...).
Sorry to bring this thread back to life, but I've been having a look at this too. There are no third-party apps that can do this (as far as I'm aware) but having had a look at the source code of Twitter, it seems like with some fiddling I MIGHT be able to get them downloaded. My 'research' has shown that when you scroll up, it uses JS to load the tweets above it, using a call similar to the API. If you look through Chrome's network log you should be able to see where it gets called and the URL of the call. One can't access this through a browser, but I'll let you know if I get any further.
Sorry this isn't much of an answer yet - I'm just posting this so I can update it and not forget to do it.
Update - 10/05/2016 - I've managed to download all of my Twitter DMs. It is not elegant AT ALL, but it works. It involves leaving your computer on overnight, with the middle mouse button clicked to scroll up your DMs. Once theat has been done (it's reached the first DM), you can download the webpage and with it your DMs. This uses a lot of memory, so be careful! I am working on parsing the HTML now so it is more readable the the soup it is currently.
Related
I've found several posts looking for guidance on how to use the Spotify Add Custom Playlist Cover Image API, but no responses with examples of how to use it successfully (for example). Zero responses to a recent question about this on the Spotify developers community forum.
Has anyone actually succeeded in making this work? Can you share an example?
I'm making a PUT call to https://api.spotify.com/v1/playlists/{my playlist ID}/images with headers Accept: application/json, Content-Type: image/jpeg (I've also tried application/json), and Authorization: Bearer {token}
I've encoded my image filed with base64 and confirmed there's no preface text with image/jpeg. My question is how to actually post the encoded image data? I don't think it's part of the URL call (https://api.spotify.com/ ... /images/image=), but rather in the request body. All of my various attempts are generating a 400 error: "Bad request"
Appreciate any help out there.
I know this is probably strictly case-specific, but I do feel like I encounter this problem a lot so I will make an effort to try and understand it better.
I am new to using APIs, but I have never succeeded in using one without copying someone's code. In this case, I can't even find any examples on forums, nor in the API documentation.
I'm trying to pull my balance value from my investment bank "NordNet" to scroll, amongst other things, on an Arduino display I've made. Right now I use python Selenium to automatically but "physically" login to NordNet and grab my balance from the DOM. As I'm afraid I might get "punished" for such botted behavior, and because the script is fairly high maintenance (as the HTML changes over time), I would obviously much rather get this information through NordNet's new API.
Link to NordNets API doc
Every time I try to utilize an API doc it's always the same, it looks easy, but I can never get it to work.
This time I tried to just play a little with the API before exploring further.
I use PostMan to send the simplest request:
https://www.nordnet.se/api/2
And I get a successful code 200 JSON response.
I then try to take it a step further to access my account data using this endpoint:
https://www.nordnet.se/api/2/accounts
For this one, I obviously need some authentication of some sort
The doc looks like this:
So I set my PostMan client up like this and get the response showcased:
I've put my NordNet login into the "Auth" tab as "basic auth" and I then see PostMan encrypts this info some way, in the "Headers" tab.
I'm getting an unauthorized response code and I have no idea why. Am I using PostMan wrong (probably)? Is the API faulty (probably not)? There is a mention of a session_id that should contain both password and username? Maybe something completely else...
I hope you can help
The documentation says to use session_id as username and password for that api ,
so try logging in and then get the session id (try with both sid and ssid) . from network tab and pass it as username and password for authorization .
sid- is for http and ssid for https i guess , try with both
Does anybody know where documentation for the Metacritic api is/if it still works. There used to be a Metacritic API at https://market.mashape.com/byroredux/metacritic-v2#get-user-details which disappeared today.
Otherwise I'm trying to scrape the site myself but keeping getting a blocked by a 429 Slow down. I got data like 3 times this hour and haven't been able to get anymore in the last 20 minutes which is making testing difficult and application possibly useless. Please let me know if there's anything else I can be doing to scape I don't know about.
I was using that API as well for an app I wrote a while ago. Looks like the creator removed it from Mashape. I just sent him an email to ask whether it'll be back up. I did find this scraper online. It only has a few endpoints but following the examples given you could easily add more. Let me know if you make any progress!
Edit: Looks like CBS requested it to be taken down. The ToS prohibits scraping:
[…] you agree not to do the following, or assist others to do the following:
Engage in unauthorized spidering, “scraping,” data mining or harvesting of Content, or use any other unauthorized automated means to gather data from or about the Services;
Though I was hoping for a Javascript way of doing this, the creator of the API also told me some info.
He says I was getting blocked for not having a User agent in the header and should use a 429 handling procedure i.e. re-request with longer pauses in between.
A PHP plugin available as well: http://datalinx.io/shop/metacritic-api/
I had to add a user agent like JCDJulian said and now it allows me to scrape. So for Ruby:
agent = Mechanize.new
agent.user_agent_alias = "Mac Firefox"
Then it stopped giving me the 403 Forbidden error.
Is it a bad practice when writing a RESTful API to use custom HTTP response codes like:
417 - Password not provided
418 - Database error
I see there is a list of standard HTTP response codes. However, from looking at Twitter's API, it appears Twitter tries to return standard HTTP response codes when available but their own error codes when they cannot align the error with a standard HTTP response (correct me if I am wrong).
What is the best practice for response codes (especially for errors) while creating a RESTful API? Any comments on the practice which Twitter chose to use?
Yes, yes it is bad practice... mostly.
One of the tenets of REST is that you work with the underlying protocols, as such HTTP has already defined a good set of response codes.
However, not every situation is catered for perfectly. Take Twitters 'arrest your calm', that response code is used when the request was valid, it simply is not being handled due to too many request being made.
I don't see another response code that quite matches that. The other two options are to either lie, and tell the user the request failed for some other response or give a generic 400 'you did something bad' (then in the body give a more detailed explanation).
I would favour using the generic X00 codes, and use headers or the body to add more detail about what actually went wrong. This at least conforms better to standards and less brittle.
Note though, it is terrible to take an existing error code, and repurpose it. 404 should always be used only for 'not found' errors. Don't start using it because the user can't make that request at this time of day.
The problems in using your own codes are:
The code you choose may get officially assigned to something completely different, and that could break your API in the future. (e.g. compare a 306 with a 301)
Intermediaries don't know what your code means, so cannot optimise anything. The internet works so well because it is a distributed system, not an end-to-end system.
There are generic responses for each category, x00, which should be used if nothing better exists.
You can send your own more specific error code in either the response body or (not as good) a response header. There should be no need to make up your own codes. If you have truly found something that would benefit the rest of the internet and no-one else has thought of until now, you can always submit an Internet Draft to the IETF (this is fairly easy to do).
I would not hold up Twitter as a shining example of good internet practice, though. :)
I'm writing one of my first apps for consuming a web service in Objective C, it's a Lighthouse API client. I'm able to execute all the GETs and XML parsing correctly and quickly, but I'm having extreme trouble trying to create a new ticket via POST (http://lighthouseapp.com/api).
I'm using ASIHTTPRequest.
I tried including the parameters on the URL (i.e. POST /projects/#{project_id}/tickets.xml?title=boo).
I've tried putting the ticket XML in the request body.
<ticket><title>boo</title></ticket>
Nothing is working. (server always sends a response back saying it needs a title) I'm very new to web services - am I missing something obvious?
I had a quick look at the Lighthouse API and here's how you go about creating a new ticket.
Request URL is http://{yourCustomURL}.lighthouseapp.com/projects/{ProjectID}/tickets.xml where {ProjectID} is a 5 digit number - in my case 72945.
Method is POST
Content type should be set to application/xml
Body should be in the format below. All fields are optional so I only included the title
<ticket> <assigned-user-id type="integer"></assigned-user-id> <body></body> <milestone-id type="integer"></milestone-id> <state></state> <title>Testing new ticket creation</title></ticket>
(sorry about the formatting of the code above, SO doesn't seem to like XML formatted code somehow?
This worked for me with a new ticket created under projectID 72945 - response received was 201 Created
If you want to make sure your POST request is working before diving into ASIHTTPRequest, download a Firefox add-on called POSTER from here. This will allow you to send an authenticated post request with all the fields above. Once you get that working, it should be a piece of cake to get ASIHTTPRequest to do the same.