I'd like to get all the links in this website https://www.sciencedirect.com/browse/journals-and-books?accessType=openAccess&accessType=containsOpenAccess
Then I'd like go on every link to extract the text in `" after clicking on the button "View Full Aims & Scope".
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
wait = WebDriverWait(driver, 20)
url = "https://www.sciencedirect.com/browse/journals-and-books?accessType=openAccess&accessType=containsOpenAccess"
driver.get(url)
page_description = wait.until(EC.presence_of_element_located((By.XPATH, "//span[#class='pagination-pages-label u-margin-s-left-from-sm u-margin-s-right-from-sm']")))
index_of = wait.until(EC.presence_of_element_located((By.XPATH, "//span[#class='pagination-pages-label u-margin-s-left-from-sm u-margin-s-right-from-sm']"))).text.index('of')
index_number = index_of + 3
time.sleep(2) #otherwise sometimes it doesn't work
length = len(page_description.text)
pages = int(page_description.text[index_number:length])
allLi = []
for i in range(pages):
index = i + 1
url = "https://www.sciencedirect.com/browse/journals-and-books?page="+str(index)+"&accessType=containsOpenAccess&accessType=openAccess"
driver.get(url)
currentAli = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//a[#class='anchor js-publication-title anchor-default']"))
for li in currentAli:
link = li.get_attribute('href');
allLi.append(link)
for li in allLi:
driver.get(li)
button = wait.until(EC.element_to_be_clickable(By.XPATH, "//button[#class='button-link button-link-secondary']"))
button.click()
descrip = wait.until(EC.presence_of_all_elements_located(By.XPATH, "//span[#class='spaced']"))
print(descrip)
First it doesn't work. Then I also have a problem with I try to compute length or pages. Sometimes it works, sometimes it doesn't. Is there anything with the asynchronism? I have to add time.sleep(2). I know it is not a good practice.
Thanks!
Here is a way of getting that information you're after, without the overheads of Selenium (though you might have to add a 2 or 3 seconds between network calls, just to be nice to ScienceDirect server's resources):
from bs4 import BeautifulSoup as bs
from tqdm import tqdm ## if using Jupyter, do from tqdm.notebook import tqdm
import pandas as pd
import cloudscraper
scraper = cloudscraper.create_scraper(disableCloudflareV1=True)
initial_list = []
journal_stuffs = []
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
for x in tqdm(range(1, 39)):
r = scraper.get(f'https://www.sciencedirect.com/browse/journals-and-books?page={x}&accessType=containsOpenAccess&accessType=openAccess')
soup = bs(r.text, 'html.parser')
links = ['https://www.sciencedirect.com' + y.get('href') for y in soup.select('a[class="anchor js-publication-title anchor-default"]')]
initial_list.extend(links)
print('there are ', len(set(initial_list)), 'journal links')
for url in tqdm(initial_list[:20]):
r = scraper.get(url)
soup = bs(r.text, 'html.parser')
title = soup.select_one('a[class="anchor js-title-link anchor-default anchor-has-background-color anchor-has-inherit-color"]').get_text(strip=True)
try:
more_info = soup.select_one('div[class="slide-out"]').get_text(strip=True)
except Exception as e:
more_info = 'Without full aim and scope. A bit pointless, really.'
journal_stuffs.append((title, more_info))
df = pd.DataFrame(journal_stuffs, columns = ['Title', 'Info'])
print(df)
Result in terminal:
100%
38/38 [00:11<00:00, 3.49it/s]
there are 3718 journal links
100%
20/20 [00:12<00:00, 1.46it/s]
Title Info
0 AACE Clinical Case Reports Aims & ScopeAACE Clinical Case Reports is an online journal that publishes case reports with accompanying commentaries six times a year. The primary mission of the journal is to present the most up-to-date information for practicing endocrinologists, fellows in endocrinology and health care professionals dealing with endocrine disorders including diabetes, obesity, osteoporosis, thyroid and other general endocrine disorders.
1 AASRI Procedia Without full aim and scope. A bit pointless, really.
2 Academic Pathology Aims & ScopeAcademic Pathologyis the official open-access journal of theAssociation of Pathology Chairs, established to give voice to innovations in education, practice, and management from academic departments of pathology and laboratory medicine, with the potential for broad impact on medicine, medical research, and the delivery of care.Academic Pathologyaddresses methods for improving patient care (clinical informatics, genomic testing and data management, lab automation, electronic health record integration, and annotate biorepositories); best practices in inter-professional clinical partnerships; innovative pedagogical approaches to medical education and educational program evaluation in pathology; models for training academic pathologists and advancing academic career development; administrative and organizational models supporting the discipline; and leadership development in academic medical centers, health systems, and other relevant venues. Intended authorship and audiences forAcademic Pathologyare international and reach beyond academic pathology itself, including but not limited to healthcare providers, educators, researchers, and policy-makers.Academic Pathologypublishes original research, reviews, brief reports, and educational cases. All articles are rigorously peer-reviewed for relevance and quality.
3 Academic Pediatrics Aims & ScopeAcademic Pediatrics, the official journal of theAcademic Pediatric Association, is a peer-reviewed publication whose purpose is to strengthen the research and educational base of academic generalpediatrics. The journal provides leadership in pediatric education, research, patient care and advocacy. Content areas includepediatric education,emergency medicine,injury,abuse,behavioral pediatrics,holistic medicine,child health servicesandhealth policy,and theenvironment. The journal provides an active forum for the presentation of pediatric educational research in diverse settings, involving medical students, residents, fellows, and practicing professionals. The journal also emphasizes important research relating to the quality of child health care, health care policy, and the organization of child health services. It also includes systematic reviews of primary care interventions and important methodologic papers to aid research in child health and education.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
4 Academic Radiology Aims & ScopeAcademic Radiologypublishes original reports of clinical and laboratory investigations indiagnostic imaging, the diagnostic use ofradioactive isotopes,computed tomography,positron emission tomography,magnetic resonance imaging,ultrasound,digital subtraction angiography,image-guided interventionsand related techniques.\nIt also includes brief technical reports describing original observations, techniques, and instrumental developments; state-of-the-art reports on clinical issues, new technology and other topics of current medical importance; meta-analyses; scientific studies and opinions on radiologic education; and letters to the Editor.
5 ACC Current Journal Review Without full aim and scope. A bit pointless, really.
6 Accident Analysis & Prevention Aims & ScopeAccident Analysis & Preventionprovides wide coverage of the general areas relating toaccidental injuryand damage, including the pre-injury and immediate post-injury phases. Published papers deal with medical, legal, economic, educational, behavioral, theoretical or empirical aspects of transportation accidents, as well as withaccidentsat other sites. Selected topics within the scope of the Journal may include: studies of human, environmental and vehicular factors influencing the occurrence, type and severity of accidents and injury; the design, implementation and evaluation of countermeasures; biomechanics of impact and human tolerance limits to injury; modelling and statistical analysis of accident data; policy, planning and decision-making in safety.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
7 Accounting Forum Without full aim and scope. A bit pointless, really.
8 Accounting, Organizations and Society Aims & ScopeAccounting, Organizations & Society is a leading international interdisciplinary journal concerned with the relationships among accounting and human behaviour, organizational and institutional structures and processes, and the wider socio-political environment of the enterprise. It aims to challenge and extend our understanding of the roles of accounting and related emergent and calculative practices in the construction of economic and societal actors, and their modes of economic organizing, including ways in which such practices influence and are influenced by the development of market and other infrastructures.We aim to publish high quality work which draws upon diverse methodologies and theoretical developments from across the social sciences, and which illuminates the development, processes and effects of accounting within its organizational, political, historical and social contexts. AOS particularly wishes to attract innovative work which analyses accounting phenomena to advance theory development in, for example, the psychological, social psychological, organizational, sociological and human sciences.The journal's unique focus covers, but is not limited to, such topics as:•\tThe roles of accounting in organizations and society;•\tThe contribution of accounting practices to the emergence, maintenance and transformation of organizational and societal institutions;•\tThe roles of accounting in the development of new organizational and institutional forms, both public and private;•\tThe relationships between accounting, auditing, accountability, ethics and social justice;•\tBehavioural studies of accounting practices and the providers, verifiers, and users of accounting information, including cognitive aspects of accounting, judgment and decision-making processes, and the behavioural aspects of planning, control and valuation processes;•\tOrganizational process studies of the design, implementation and use of accounting, information and management control systems;•\tAccounting for human actors, and the impact of accounting technologies upon human subjectivities and evaluations;•\tThe roles of accounting in shaping the design, operation and delivery of public service providers, not-for-profit entities, government bodies, as well as local, national and transnational governmental organizations;•\tSocial, organizational, political, and psychological studies of the standard-setting process, and the effects of accounting regulations and rules;•\tThe roles and practices of audit, auditors and accounting firms in the construction and understanding of organizational and societal valuations;•\tAccounting for sustainability and the environment, including studies of environmental and social reporting;•\tHistorical studies of the emergence, transformation and impact of accounting calculations, practices, and representations, including the development and the changing roles of accounting theories, techniques, individual and teams of practitioners and their firms, professional associations, and regulators.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
9 Achievements in the Life Sciences Without full aim and scope. A bit pointless, really.
10 ACOG Clinical Review Without full aim and scope. A bit pointless, really.
11 Acta Anaesthesiologica Taiwanica Without full aim and scope. A bit pointless, really.
12 Acta Astronautica Aims & ScopeActa Astronauticais sponsored by theInternational Academy of Astronautics. Content is based on original contributions in all fields of basic,engineering,lifeandsocial space sciencesand ofspace technologyrelated to:The peaceful scientific exploration of space,Its exploitation for human welfare and progress,Conception, design, development and operation of space-borne and Earth-based systems,In addition to regular issues, the journal publishes selected proceedings of the annual International Astronautical Congress (IAC), transactions of the IAA and special issues on topics of current interest, such asmicrogravity,space station technology,geostationary orbits, andspace economics. Other subject areas includesatellite technology,space transportationandcommunications,space energy,power and propulsion,astrodynamics,extraterrestrial intelligenceandEarth observations.For more information on the International Academy of Astronautics (IAA), visit their home page:http://www.iaaweb.org. Members of the IAA are eligible for a discount on a personal subscription toActa Astronautica. Please clickhereto download an order form.
13 Acta Biomaterialia Aims & ScopeActa Biomaterialiais an international journal that publishes peer-reviewed original research reports, review papers and communications in the broadly defined field ofbiomaterials science. The emphasis of the journal is on the relationship betweenbiomaterial structureandfunctionat all length scales.The scope ofActa Biomaterialiaincludes:Hypothesis-driven design of biomaterialsBiomaterial surface science linking structure to biocompatibility, including protein adsorption and cellular interactionsBiomaterial mechanical characterization and modeling at all scalesMolecular, statistical and other types of modeling applied to capture biomaterial behaviorInteractions of biological species with defined surfacesCombinatorial approaches to biomaterial developmentStructural biology as it relates structure to function for biologically derived materials that have application as a medical material, or as it aids in understanding the biological response to biomaterialsMethods for biomaterial characterizationProcessing of biomaterials to achieve specific functionalityMaterials development for arrayed genomic and proteomic screeningBenefits to authorsFree and automatic manuscript deposit service to meet NIH public access requirements at one year;Multiple options for data-sharing (seehttp://www.materialstoday.com/materials-genome-initiative/);Free author pdf and Sharelink share your article with your peers (seehttps://www.elsevier.com/journal-authors/share-link);And more information on our author services can be foundherePlease see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
14 Acta Colombiana de Cuidado Intensivo Aims & ScopeActa Colombiana de Cuidado Intensivois the official publication of the Asociación Colombiana de Medicina Crítica y Cuidado Intensivo (Colombian Association of Critical Medicine and Intensive Care). It is published every three months in March, June, September and December and is intended to be a means of dissemination in all areas associated with the management of the critically ill patient.All the manuscripts received by theActa Colombiana de Cuidado Intensivoare reviewed using a double blind system by experts in the specialty (peer review).The Journal publishes articles on research (Originals), Reviews, Case Reports , and Case Series, as well as Articles on Reflections, and Clinical Comments. Also, it offers the possibility of publishing supplements on specific topics that allows the reader to get into a particular area of knowledge in depth.The development of Intensive Care has encouraged certain areas of specialisation within the specialists dedicated to the care of the critically ill patient. Responding to this need,Acta Colombiana de Cuidado Intensivopays particular attention to certain areas of interests which are made up by experts. The subject matter organisation of the Journal enables it to approach not just technical subjects, but also those related to the logistic organisation of the practice of intensive care.The areas of interest of the Journal are the following:• epidemiology• infection and sepsis• coagulation and inflammation• cardiovascular critical care• mechanical ventilation• bioethics• nutrition and metabolism• quality and costs• neurological intensive care• toxicology• trauma• obstetrics intensive care• sedation and analgesia• paediatrics intensive careLa revistaActa Colombiana de Cuidado Intensivoes el órgano oficial de la Asociación Colombiana de Medicina Crítica y Cuidado Intensivo. Se publica trimestralmente en los meses de marzo, junio, septiembre y diciembre y pretende ser un órgano de divulgación en todas las áreas relacionadas con el manejo del paciente críticamente enfermo.Todos los manuscritos recibidos porActa Colombiana de Cuidado Intensivoson revisados mediante el sistema de doble ciego por expertos de la especialidad.La revista publica artículos de investigación (Originales), de Revisión, Reportes de Casos y Series de Casos, así como Artículos de Reflexión y Comentarios Clínicos. Además, ofrece la posibilidad de publicar suplementos sobre temas específicos que permitan al lector profundizar a fondo en un área particular del conocimiento.El desarrollo del cuidado intensivo ha promovido ciertas áreas de especialización dentro de los especialistas dedicados al cuidado del paciente críticamente enfermo. Respondiendo a esta necesidad,Acta Colombiana de Cuidado Intensivopresta especial atención a determinadas áreas de interés en las que se agrupa a los expertos. La organización temática de la revista permite abordar no solo temas técnicos sino también aquellos relacionados con la organización logística de la práctica del cuidado intensivo.Las áreas de interés de la revista son las siguientes:• epidemiologia• infecciones y sepsis• coagulación e inflamación• cuidado crítico cardiovascular• ventilación mecánica• bioética• nutrición y metabolismo• calidad y costos• cuidado intensivo neurológico• toxicología• trauma• cuidado intensivo obstétrico• sedación y analgesia• cuidado intensivo pediátrico
15 Acta Ecologica Sinica Aims & ScopeActa Ecologica Sinica (International Journal)is a bimonthly academic journal sponsored by the Ecological Society of China and the Research Center for Eco-Environmental Sciences, Chinese Academy of Sciences.Acta Ecologica Sinicapublishes novel research inecology, promotes the exchange and cooperation of ecologists and ecological research between developing and developed countries. The Journal aims to show the scientific mechanism of the interaction between life and environment and facilitates the academic dissemination and scientific development of ecological research in the world, especially in developing countries.Position of the journalActa Ecologica Sinicais a comprehensive journal devoted to the development of Ecology and its sub-disciplines. It unites ecological scientists in the world with the aim of publishing high-quality papers on innovative research. Published papers should unveil mechanisms of the interactions between life and environment, and contribute to the innovation and sustainable development of ecological science in the world.International perspectiveMembers of the editorial board of Acta Ecologica Sinica are all internationally renowned ecologists, and its presentEditor-in-Chiefis the academician of theChinese Academy of Sciences (CAS)and theChinese Academy of Engineering (CAE). In recent years, Acta Ecologica Sinica is receiving an increasing international attention, and its editorial members come from 8 different countries and regions in various areas of ecological research, which strengthens the journal`s impact worldwide. It is anticipated that Acta Ecologica Sinica will further gain more international recognition and have a great prospect of development.Journal coverageThis journal publishes papers on animal ecology, plant ecology, microbial ecology, agro-ecology, forestry ecology, grassland ecology, soil ecology, ocean and aquatic ecosystems, landscape ecology, chemical ecology, contaminant ecology, urban and human ecology. We particularly welcome reviews on recent developments in ecology, novel experimental studies, and short communications, new theories, methodologies, new techniques, book reviews, and research news and laboratory introductions.
16 Acta Histochemica Aims & ScopeActa Histochemicais a classic scientific journal established in 1954 currently focused on basic research and methodological innovations in cell and tissue biology. The aim of the journal is to promote the peer-reviewed publication of original articles and short communications reporting novel results and experimental approaches in the field, as well as comprehensive reviews, letters to the editor and meeting reports, serving as an open forum for the cell and histochemical research community. Manuscripts analysing the mechanisms of functional regulation of living systems at a cell/tissue level, in physiological or pathological conditions, or reporting new techniques and methodological approaches to quantify/visualize cellular activities are particularly welcomed.
17 Acta de Investigación Psicológica Without full aim and scope. A bit pointless, really.
18 Acta Materialia Aims & ScopeActa Materialiaprovides a forum for publishing full-length, original papers and commissioned overviews that advance the in-depth understanding of the relationship between the processing, the structure and the properties of inorganic materials. Papers that have a high impact potential and/or substantially advance the field are sought. The structure encompasses atomic and molecular arrangements, chemical and electronic structures, and microstructure. Emphasis is on either the mechanical or functional behavior of inorganic solids at all length scales down to nanostructures.The following aspects of the science and engineering of inorganic materials are of particular interest:(i) Cutting-edge experiments and theory as they relate to the understanding of the properties,(ii) Elucidation of the mechanisms involved in the synthesis and processing of materials specifically as they relate to the understanding of the properties,and(iii) Characterization of the structure and chemistry of materials specifically as it relates to the understanding of the properties.Acta Materialiawelcomes papers that employ theory and/or simulation (or numerical methods) that substantially advance our understanding of the structure and properties of inorganic materials. Such papers should demonstrate relevance to the materials community by, for example, making a comparison with experimental results (in the literature or in the present study), making testable microstructural or property predictions or elucidating an important phenomenon. Papers that focus primarily on model parameter studies, development of methodology or those employing existing software packages to obtain standard or incremental results are discouraged.Short communications and comments to papers published inActa Materialiamay besubmitted toScripta Materialia.
19 Acta Metallurgica Without full aim and scope. A bit pointless, really.
I have a screen that displays article information thats been pulled from a Wordpress API call and returns json (inclusive of all its lovely HTML tags).
<Text style={styles.summary}>{htmlRegex(item.content.rendered)}{"\n"}{Moment(item.date, "YYYYMMDD").fromNow()}</Text>
I have a function that strips out all of the HTML tags, tidies up any unicode, etc...
function htmlRegex(string) {
string = string.replace(/<\/?[^>]+(>|$)/g, "")
string = string.replace(/…/g,"...")
let changeencode = entities.decode(string);
return changeencode;
}
The challenge is that the tags returned in the content appear to be causing odd line spacing issues, as shown in the screen grab;
The content.rendered contains;
rendered: "
<figure class="wp-block-image size-large"><img data-attachment-id="655" data-permalink="https://derbyfutsal.com/derby-futsal-club-women-name-change-june20/" data-orig-file="https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png" data-orig-size="1024,512" data-comments-opened="1" data-image-meta="{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}" data-image-title="derby-futsal-club-women-name-change-june20" data-image-description="" data-medium-file="https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=300" data-large-file="https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=730" src="https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=1024" alt="" class="wp-image-655" srcset="https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png 1024w, https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=150 150w, https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=300 300w, https://derbyfutsal.files.wordpress.com/2020/06/derby-futsal-club-women-name-change-june20.png?w=768 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p>Derby Futsal Club Ladies’ team are renamed Derby Futsal Club Women.</p>
<p>The change in name reflects Derby Futsal’s work in developing all aspects of futsal on and off the court.</p>
<p>It reflects the way the league (FA National Futsal Women’s Super Series), the players, the fans and the management refer to the game.</p>
<p>Hannah Roberts, Derby Futsal Club Women captain, believes “the change from Ladies to Women’s is a subtle but important one. Many professional sports teams have moved towards ‘Women’s’ in the last five years in order to stay modern and in touch, and as a forward-thinking club it’s important for Derby Futsal to do the same. We’re making so many strides in our community work and marketing, and this name change is another step forward to the future for the club”.</p>
<p>Derby Futsal Club Women first team coach, Matt Hardy feels this name change signifies evolution for the team; “the future of the women’s game both at Derby and nationally is looking bright. So it’s only right that we have a name that is modern, and inline with the national game”. </p>
<p>This news follows similar moves in professional football. Chelsea, Manchester City and Arsenal have all renamed their women’s team recently. It is something Professor Kath Woodward from the Open University, an expert on sociology and sport agrees with, “the use of ladies suggests a physical frailty and need for protection”.</p>
<p>Alex Scott, former Arsenal Women captain, adds: “the term ‘Women’s’ delineates between men and women without as many stereotypes or preconceived notions and it is in keeping with modern-day thinking on equality”.</p>
<p></p>
",
My question is, how do you handle the tags so that the return line white space is managable?
Put this in your css:
p {
margin: 0;
padding: 0;
}
And just replace 0 with whatever suits (0.5rem, 20px, whatever floats your boat really).
I have been trying to create distractors (false answers) for multiple choice questions. Using word vectors, I was able to get decent results for single-word nouns.
When dealing with compound nouns (such as "car park" or "Donald Trump"), my best attempt was to compute similar words for each part of the compound and combine them. The results are very entertaining:
Car park -> vehicle campground | automobile zoo
Fire engine -> flame horsepower | fired motor
Donald Trump -> Richard Jeopardy | Jeffrey Gamble
Barrack Obama -> Obamas McCain | Auschwitz Clinton
Unfortunately, these are not very convincing. Especially in case of named entities, I want to produce other named entities, which appear in similar contexts; e.g:
Fire engine -> Fire truck | Fireman
Donald Trump -> Barrack Obama | Hillary Clinton
Niagara Falls -> American Falls | Horseshoe Falls
Does anyone have any suggestions of how this could be achieved? Is there are a way to generate similar named entities/noun chunks?
I managed to get some good distractors by searching for the named entities on Wikipedia, then extracting entities which are similar from the summary. Though I'd prefer to find a solution using just spacy.
If you haven't seen it yet, you might want to check out sense2vec, which allows learning context-sensitive vectors by including the part-of-speech tags or entity labels. Quick usage example of the spaCy extension:
s2v = Sense2VecComponent('/path/to/reddit_vectors-1.1.0')
nlp.add_pipe(s2v)
doc = nlp(u"A sentence about natural language processing.")
most_similar = doc[3]._.s2v_most_similar(3)
# [(('natural language processing', 'NOUN'), 1.0),
# (('machine learning', 'NOUN'), 0.8986966609954834),
# (('computer vision', 'NOUN'), 0.8636297583580017)]
See here for the interactive demo using a sense2vec model trained on Reddit comments. Using this model, "car park" returns things like "parking lot" and "parking garage", and "Donald Trump" gives you "Sarah Palin", "Mitt Romney" and "Barack Obama". For ambiguous entities, you can also include the entity label – for example, "Niagara Falls|GPE" will show similar terms to the geopolitical entitiy (GPE), e.g. the city as opposed to the actual waterfalls. The results obviously depend on what was present in the data, so for even more specific similarities, you could also experiment with training your own sense2vec vectors.
I need to display a brief description about some topic. Like, when we search "India" on Google, it will show a brief description about India on the right side of the page, but that content is not an actual paragraph from wikipedia. does we get this content from wiki api?
https://www.mediawiki.org/wiki/Extension:TextExtracts#API, for example:
https://en.wikipedia.org/w/api.php?action=query&prop=extracts&exintro=1&explaintext=1&titles=India&continue=&format=json&formatversion=2 outputs
{
"batchcomplete": true,
"query": {
"pages": [
{
"pageid": 14533,
"ns": 0,
"title": "India",
"extract": "India (/ˈɪndiə/), officially the Republic of India (Bhārat Gaṇarājya), is a country in South Asia. It is the seventh-largest country by area, the second-most populous country with over 1.2 billion people, and the most populous democracy in the world. Bounded by the Indian Ocean on the south, the Arabian Sea on the south-west, and the Bay of Bengal on the south-east, it shares land borders with Pakistan to the west; China, Nepal, and Bhutan to the north-east; and Burma (Myanmar) and Bangladesh to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; in addition, India's Andaman and Nicobar Islands share a maritime border with Thailand and Indonesia.\nHome to the ancient Indus Valley Civilisation and a region of historic trade routes and vast empires, the Indian subcontinent was identified with its commercial and cultural wealth for much of its long history. Four religions—Hinduism, Buddhism, Jainism, and Sikhism—originated here, whereas Zoroastrianism and the Abrahamic religions of Judaism, Christianity, and Islam arrived in the 1st millennium CE and also helped shape the region's diverse culture. Gradually annexed by and brought under the administration of the British East India Company from the early 18th century and administered directly by the United Kingdom after the Indian Rebellion of 1857, India became an independent nation in 1947 after a struggle for independence that was marked by non-violent resistance led by Mahatma Gandhi.\nThe Indian economy is the world's seventh-largest by nominal GDP and third-largest by purchasing power parity (PPP). Following market-based economic reforms in 1991, India became one of the fastest-growing major economies; it is considered a newly industrialised country. However, it continues to face the challenges of poverty, corruption, malnutrition, inadequate public healthcare, and terrorism. A nuclear weapons state and a regional power, it has the third-largest standing army in the world and ranks ninth in military expenditure among nations. India is a federal constitutional republic governed under a parliamentary system consisting of 29 states and 7 union territories. India is a pluralistic, multilingual, and a multi-ethnic society. It is also home to a diversity of wildlife in a variety of protected habitats."
}
]
}
}