Exporting links in google scholar results - google-scholar

Question

I want to export data from google scholar. In particular, I want to export a list of articles that cite a particular paper. If I click the Cited By link I can get this page. One way I can export these data is to add all of them to my library. Then you can export in 4 different formats (BibTex, Refman, Endnote, CSV). However, none of these export formats include the HTML link (URL) to each paper.
The other strategy would be to scrape the data, but I don't want to do that as I know that this can be very tricky with google scholar's captchas.
Is there a way to export the results of a google scholar search that includes the URLs of each paper ?

Answer 1

For the page you're on you mean? From the console (F12) do:
copy($$('li > a').map(a => a.href))
Now they're in your clipboard.

Answer 2

To extract Cited by data, you'll need an ID of a Google Scholar organic search result that the Cited by link belongs to. You can find the ID inside data-cid html attribute.
You can then query the next link to retrieve the data: https://scholar.google.com/scholar?q=info: this_is_where_you_put_the_cite_id:scholar.google.com/&output=cite
There is also a third party solution like SerpApi to do this for you. It's a paid API with a free trial.
Example python code (available in other libraries also):
from serpapi import GoogleSearch
params = {
"engine": "google_scholar_cite",
"q": "FDc6HiktlqEJ",
"api_key": "secret_api_key",
}
search = GoogleSearch(params)
results = search.get_dict()
Example JSON output:
"citations": [
{
"title": "MLA",
"snippet": "Schwertmann, U. T. R. M., and Reginald M. Taylor. \"Iron oxides.\" Minerals in soil environments 1 (1989): 379-438."
},
{
"title": "APA",
"snippet": "Schwertmann, U. T. R. M., & Taylor, R. M. (1989). Iron oxides. Minerals in soil environments, 1, 379-438."
},
{
"title": "Chicago",
"snippet": "Schwertmann, U. T. R. M., and Reginald M. Taylor. \"Iron oxides.\" Minerals in soil environments 1 (1989): 379-438."
},
{
"title": "Harvard",
"snippet": "Schwertmann, U.T.R.M. and Taylor, R.M., 1989. Iron oxides. Minerals in soil environments, 1, pp.379-438."
},
{
"title": "Vancouver",
"snippet": "Schwertmann UT, Taylor RM. Iron oxides. Minerals in soil environments. 1989 Jan 1;1:379-438."
}
],
"links": [
{
"name": "BibTeX",
"link": "https://scholar.googleusercontent.com/scholar.bib?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=4&ct=citation&cd=-1&hl=en"
},
{
"name": "EndNote",
"link": "https://scholar.googleusercontent.com/scholar.enw?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=3&ct=citation&cd=-1&hl=en"
},
{
"name": "RefMan",
"link": "https://scholar.googleusercontent.com/scholar.ris?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=2&ct=citation&cd=-1&hl=en"
},
{
"name": "RefWorks",
"link": "https://scholar.googleusercontent.com/scholar.rfw?q=info:FDc6HiktlqEJ:scholar.google.com/&output=citation&scisdr=CgXpniNQGAA:AAGBfm0AAAAAYMu3WkYJI4po_pgcUVKgwwFp1dl5uNYk&scisig=AAGBfm0AAAAAYMu3WlZR_joxo-i8FTZ1CphjzmW_d447&scisf=1&ct=citation&cd=-1&hl=en"
}
]
Check out the documentation for more details.
Disclaimer: I work at SerpApi.