I was scraping all carousel medias using cheerio and regex to find all urls inside the page body.
To know if a media was a carousel, I checked if the request response was type 'video' and didn't have a 'videos' property. Like in this thread null.jpg being returned for carousel posts via Instagram API.
But now, Instagram changed the way carousels come in the requests, now, the Json response comes identical an image response if the first slide of the carousel is an image, and identical a video response if the firt slide is a video.
I want to know if anyone can differentiate carousels from other types of media in requests from the Instagram API.
Since last April 24th we are receiving posts of type "carousel" from the API. This posts contain the property "carousel_media" with all the carousel images/videos.
Related
I am using a request:
https://www.googleapis.com/youtube/v3/search?part=snippet&type=video&q={search value}&maxResults=3&key={key}`
This way I am getting json with info about video thumbnails and descriptions what i need but I don't have an url to channel picture and don't have infro how many views has a video.
I found the way to get the info about channel URL logo in separate request
https://www.googleapis.com/youtube/v3/search?part=snippet&channelId={id}
but it doesn't look right to make additional separate requests for every single logo of the list and Views statistics.
Also, is it possible to get in the same request for embedded video URL's as well?
You are right, it's impossible from a search query q=${searchValue} to get the associated channel thumbnail URL in a single request.
So, I'm trying to scrape a website with infinite scrolling.
I'm following this tutorial on scraping infinite scrolling web pages: https://blog.scrapinghub.com/2016/06/22/scrapy-tips-from-the-pros-june-2016
But the example given looks pretty easy, it's an orderly JSON object with the data you want.
I want to scrape this https://www.bahiablancapropiedades.com/buscar#/terrenos/venta/bahia-blanca/todos-los-barrios/rango-min=50.000,rango-max=350.000
The XHR response for each page is weird, looks like corrupted html code
This is how the Network tab looks
I'm not sure how to navigate the items inside "view". I want the spider to enter each item and crawl some information for every one.
In the past I've succesfully done this with normal pagination and rules guided by xpaths.
https://www.bahiablancapropiedades.com/buscar/resultados/0
This is XHR url.
While scrolling the page it will appear the 8 records per request.
So do one thing get all records XPath. these records divide by 8. it will appear the count of XHR requests.
do below process. your issue will solve. I get the same issue as me. I applied below logic. it will resolve.
pagination_count = xpath of presented number
value = int(pagination_count) / 8
for pagination_value in value:
url = https://www.bahiablancapropiedades.com/buscar/resultados/+[pagination_value]
pass this url to your scrapy funciton.
It is not corrupted HTML, it is escaped to prevent it from breaking the JSON. Some websites will return simple JSON data and others, like this one, will return the actual HTML to be added.
To get the elements you need to get the HTML out of the JSON response and create your own parsel Selector (this is the same as when you use response.css(...)).
You can try the following in scrapy shell to get all the links in one of the "next" pages:
scrapy shell https://www.bahiablancapropiedades.com/buscar/resultados/3
import json
import parsel
json_data = json.loads(response.text)
sel = parsel.Selector(json_data['view']) # view contains the HTML
sel.css('a::attr(href)').getall()
The YouTube API has a page for thumbnails and a page for videos. If I want the thumbnails for a certain YouTube video, which API should I be using?
I'm inclined to say the thumbnail API, but I can't figure out how to get information out of it? Can I only set with it? The first bullet point on the thumbnail API docs seems to indicate that I can get images from it:
A resource's snippet.thumbnails property is an object that identifies the thumbnail images available for that resource.
The only method available on thumbnails ressource is set means :
Uploads a custom video thumbnail to YouTube and sets it for a video.
https://developers.google.com/youtube/v3/docs/thumbnails
So you can't retrieve a thumbnail of a video with the resource thumbnails.
You need to use the ressource videos.list https://developers.google.com/youtube/v3/docs/videos/list with paramter snippet and specific videoId
I'm building a navigation app for Windows 8 using Html & JS. For some of my div's, I am handling the onclick to do the following code, WinJS.Navigation.navigate("/pages/video/video.html"), which is a video page that simply plays a video. Before that code is called, I'm setting a hidden input element in default.html , <input id="currentVideoId" type="hidden" /> with the value of the video id that is clicked.
That way, on my video page, I can grab the current value of the hidden input to figure out which video to ajax load for the user.
Question: how can I instead navigate the user to /pages/video/video.html?id=555 ? I tried that, but my video.html did not seem to load at all. If I can do that, then in my video.js file I can look for the query string var to get the id of the video it should play.
Issue: if I don't do this, then if I go from 1 video to another video (I have an "Up next" control on each video page pointing to the next video in sequence), the back button doesn't work correctly as it just goes to the last video played.
You can use the same navigate function with the second parameter:
WinJS.Navigation.navigate("/pages/video/video.html", videoId);
Can anyone provide an example of how to retrieve captions for YouTube video?
Based on API docs, API request https://gdata.youtube.com/feeds/api/videos?q=surfing&caption&v=2
returns list of videos about surfing with captions.
However a request for the first video from the list
https://gdata.youtube.com/feeds/api/videos/bQX4fustG9Q?v=2&key=DEVELOPERKEY
returns feed that shows no traces of captions
What am I missing?
Can someone please post an example of request that will retrieve captions for that video?
Thank you.
According to the docs, you don't need to send any special parameters in the initial request, but instead retrieve videos normally and then look for the link with rel=http://gdata.youtube.com/schemas/2007#video.captionTracks.
Note that the caption link is only visible to the owner of a video.