I'm trying to analyse sequences with TraMineR
UKTUS records people activities for every 10 minutes defined by the dataset as act1_1, act1_2,...,act1_144 variables (144 x 10 minutes).
Each time step (act1_1, act1_2, act1_3) is defined by one of the following code:
-1 Not applicable
0 Unspecified personal care
110 Sleep
111 Sleep: In bed not asleep
120 Sleep: Sick in bed
210 Eating
300 Other personal care: Unspecified other personal care
310 Other personal care: Wash and dress
390 Other personal care: Other specified personal care
1000 Unspecified employment
1100 Main job: unspecified main job
1110 Main job: Working time in main job
1120 Main job: Coffee and other breaks in main job
1210 Second job: Working time in second job
1220 Second job: Coffee and other breaks in second job
1300 Activities related to employment: Unspecified activities related to employment
1310 Activities related to employment: Lunch break
1390 Activities related to employment: Other specified activities related to employment
1391 Activities related to employment: Activities related to job seeking
1399 Activities related to employment: Other unspecified activities related to employment
2000 Study: Unspecified study school or university
2100 Study: Unspecified activities related to school or university
2110 Study: Classes and lectures
2120 Study: Homework
2190 Study: other specified activities related to school or university
2210 Free time study
3000 Unspecified household and family care
3100 Unspecified food management
3110 Food preparation and baking
3130 Dish washing
3140 Preserving
3190 Other specified food management
3200 Unspecified household upkeep
3210 Cleaning dwelling
3220 Cleaning yard
3230 Heating and water
3240 Arranging household goods and materials
3250 Disposal of waste
3290 Other or unspecified household upkeep
3300 Unspecified making and care for textiles
3310 Laundry
3320 Ironing
3330 Handicraft and producing textiles
3390 Other specified making and care for textiles
3410 Gardening
3420 Tending domestic animals
3430 Caring for pets
3440 Walking the dog
3490 Other specified gardening and pet care
3500 Unspecified construction and repairs
3510 House construction and renovation
3520 Repairs of dwelling
3530 Making repairing and maintaining equipment
3531 Woodcraft metalcraft sculpture and pottery
3539 Other specified making repairing and maintaining equipment
3540 Vehicle maintenance
3590 Other specified construction and repairs
3600 Unspecified shopping and services
3610 Unspecified shopping
3611 Shopping mainly for food
3612 Shopping mainly for clothing
3613 Shopping mainly related to accommodation
3614 Shopping or browsing at car boot sales or antique fairs
3615 Window shopping or other shopping as leisure
3619 Other specified shopping
3620 Commercial and administrative services
3630 Personal services
3690 Other specified shopping and services
3710 Household management not using the internet
3713 Shopping for and ordering clothing via the internet
3720 Unspecified household management using the internet
3721 Shopping for and ordering unspecified goods and services via the internet
3722 Shopping for and ordering food via the internet
3724 Shopping for and ordering goods and services related to accommodation via the internet
3725 Shopping for and ordering mass media via the internet
3726 Shopping for and ordering entertainment via the internet
3727 Banking and bill paying via the internet
3729 Other specified household management using the internet
3800 Unspecified childcare
3810 Unspecified physical care & supervision of a child
3811 Feeding the child
3819 Other and unspecified physical care & supervision of a child
3820 Teaching the child
3830 Reading playing and talking with child
3840 Accompanying child
3890 Other or unspecified childcare
3910 Unspecified help to a non-dependent eg injured adult household member
3911 Physical care of a non-dependent e.g. injured adult household member
3914 Accompanying a non-dependent adult household member e.g. to hospital
3919 Other specified help to a non-dependent adult household member
3920 Unspecified help to a dependent adult household member
3921 Physical care of a dependent adult household member e.g. Alzheimic parent
3924 Accompanying a dependent adult household member e.g. Alzheimic
3929 Other specified help to a dependent adult household member
4000 Unspecified volunteer work and meetings
4100 Unspecified organisational work
4110 Work for an organisation
4120 Volunteer work through an organisation
4190 Other specified organisational work
4200 Unspecified informal help to other households
4210 Food management as help to other households
4220 Household upkeep as help to other households
4230 Gardening and pet care as help to other households
4240 Construction and repairs as help to other households
4250 Shopping and services as help to other households
4260 Help to other households in employment and farming
4270 Unspecified childcare as help to other households
4271 Physical care and supervision of child as help to other household
4272 Teaching non-coresident child
4273 Reading playing & talking to non-coresident child
4274 Accompanying non-coresident child
4275 Physical care and supervision of own child as help to other household
4277 Reading playing & talking to own non-coresident child
4278 Accompanying own non-coresident child
4279 Other specified childcare as help to other household
4280 Unspecified help to an adult of another household
4281 Physical care and supervision of an adult as help to another household
4282 Accompanying an adult as help to another household
4283 Other specified help to an adult member of another household
4289 Other specified informal help to another household
4290 Other specified informal help
4300 Unspecified participatory activities
4310 Meetings
4320 Religious activities
4390 Other specified participatory activities
5000 Unspecified social life and entertainment
5100 Unspecified social life
5110 Socialising with family
5120 Visiting and receiving visitors
5130 Celebrations
5140 Telephone conversation
5190 Other specified social life
5200 Unspecified entertainment and culture
5210 Cinema
5220 Unspecified theatre or concerts
5221 Plays musicals or pantomimes
5222 Opera operetta or light opera
5223 Concerts or other performances of classical music
5224 Live music other than classical concerts opera and musicals
5225 Dance performances
5229 Other specified theatre or concerts
5230 Art exhibitions and museums
5240 Unspecified library
5241 Borrowing books records audiotapes videotapes CDs VDs etc. from a library
5242 Reference to books and other library materials within a library
5243 Using internet in the library
5244 Using computers in the library other than internet use
5245 Reading newspapers in a library
5249 Other specified library activities
5250 Sports events
5290 Other unspecified entertainment and culture
5291 Visiting a historical site
5292 Visiting a wildlife site
5293 Visiting a botanical site
5294 Visiting a leisure park
5295 Visiting an urban park playground designated play area
5299 Other or unspecified entertainment or culture
5310 Resting - Time out
6000 Unspecified sports and outdoor activities
6100 Unspecified physical exercise
6110 Walking and hiking
6111 Taking a walk or hike that lasts at least miles or 1 hour
6119 Other walk or hike
6120 Jogging and running
6130 Biking skiing and skating
6131 Biking
6132 Skiing or skating
6140 Unspecified ball games
6141 Indoor pairs or doubles games
6142 Indoor team games
6143 Outdoor pairs or doubles games
6144 Outdoor team games
6149 Other specified ball games
6150 Gymnastics
6160 Fitness
6170 Unspecified water sports
6171 Swimming
6179 Other specified water sports
6190 Other specified physical exercise
6200 Unspecified productive exercise
6210 Hunting and fishing
6220 Picking berries mushroom and herbs
6290 Other specified productive exercise
6310 Unspecified sports related activities
6311 Activities related to sports
6312 Activities related to productive exercise
7000 Unspecified hobbies games and computing
7100 Unspecified arts
7110 Unspecified visual arts
7111 Painting drawing or other graphic arts
7112 Making videos taking photographs or related photographic activities
7119 Other specified visual arts
7120 Unspecified performing arts
7121 Singing or other musical activities
7129 Other specified performing arts
7130 Literary arts
7140 Other specified arts
7150 Unspecified hobbies
7160 Collecting
7170 Correspondence
7190 Other specified or unspecified arts and hobbies
7220 Computing - programming
7230 Unspecified information by computing
7231 Information searching on the internet
7239 Other specified information by computing
7240 Unspecified communication by computer
7241 Communication on the internet
7249 Other specified communication by computing
7250 Unspecified other computing
7251 Skype or other video call
7259 Other specified computing
7300 Unspecified games
7310 Solo games and play
7320 Unspecified games and play with others
7321 Billiards pool snooker or petanque
7322 Chess and bridge
7329 Other specified parlour games and play
7330 Computer games
7340 Gambling
7390 Other specified games
8000 Unspecified mass media
8100 Unspecified reading
8110 Reading periodicals
8120 Reading books
8190 Other specified reading
8210 Unspecified TV video or DVD watching
8211 Watching a film on TV
8212 Watching sport on TV
8219 Other specified TV watching
8220 Unspecified video watching
8221 Watching a film on video
8222 Watching sport on video
8229 Other specified video watching
8300 Unspecified listening to radio and music
8310 Unspecified radio listening
8311 Listening to music on the radio
8312 Listening to sport on the radio
8319 Other specified radio listening
8320 Listening to recordings
9000 Travel related to unspecified time use
9010 Travel related to personal business
9100 Travel to/from work
9110 Travel in the course of work
9120 Travel to work from home and back only
9130 Travel to work from a place other than home
9210 Travel related to education
9230 Travel escorting to/ from education
9310 Travel related to household care
9360 Travel related to shopping
9370 Travel related to services
9380 Travel escorting a child other than education
9390 Travel escorting an adult other than education
9400 Travel related to organisational work
9410 Travel related to voluntary work and meetings
9420 Travel related to informal help to other households
9430 Travel related to religious activities
9440 Travel related to participatory activities other than religious activities
9500 Travel to visit friends/relatives in their homes not respondents household
9510 Travel related to other social activities
9520 Travel related to entertainment and culture
9600 Travel related to other leisure
9610 Travel related to physical exercise
9620 Travel related to hunting & fishing
9630 Travel related to productive exercise other than hunting & fishing
9710 Travel related to gambling
9720 Travel related to hobbies other than gambling
9800 Travel related to changing locality
9810 Travel to holiday base
9820 Travel for day trip/just walk
9890 Other specified travel
9940 Punctuating activity
9941 Unknown: at home
9950 Filling in the time use diary
9960 No main activity no idea what it might be
9970 No main activity some idea what it might be
9980 Illegible activity
9990 Unspecified time use
9999 Queryable
I created a matrix in R with 129 columns and 16533 rows.
Activities <-uktus15_diary_wide[,c ("serial", "pnum","ddayw","DVAge", "dmonth", "dyear","WhenDiary","AfterDiaryDay","WhereStart","WhereEnd","RushedD","Ordinary","KindOfDay","Trip","enjm1","act1_1, "act1_2", "act1_3", "act1_4", "act1_5", "act1_6", "act1_7", "act1_8", "act1_9", "act1_10", "act1_11", "act1_12", "act1_13", "act1_14","act1_15", "act1_16", "act1_17", "act1_18", "act1_19", " "act1_21", "act1_22", "act1_23", "act1_24", "act1_25", "act1_26", "act1_27", "act1_28", "act1_29", "act1_30",
"act1_31", "act1_32", "act1_33", "act1_34", "act1_35", "act1_36", "act1_37", "act1_38", "act1_39", "act1_40",
"act1_41", "act1_42", "act1_43", "act1_44", "act1_45", "act1_46", "act1_47", "act1_48", "act1_49", "act1_50",
"act1_51", "act1_52", "act1_53", "act1_54", "act1_55", "act1_56", "act1_57", "act1_58", "act1_59", "act1_60",
"act1_61", "act1_62", "act1_63", "act1_64", "act1_65", "act1_66", "act1_67", "act1_68", "act1_69", "act1_70",
"act1_71", "act1_72", "act1_73", "act1_74", "act1_75", "act1_76", "act1_77", "act1_78", "act1_79", "act1_80",
"act1_81", "act1_82", "act1_83", "act1_84", "act1_85", "act1_86", "act1_87", "act1_88", "act1_89", "act1_90",
"act1_91", "act1_92", "act1_93", "act1_94", "act1_95", "act1_96", "act1_97", "act1_98", "act1_99", "act1_100",
"act1_101", "act1_102", "act1_103", "act1_104", "act1_105", "act1_106", "act1_107", "act1_108", "act1_109",
"act1_110", "act1_111", "act1_112", "act1_113", "act1_114")]
Example how the data looks like (I included just serial pnum act1_28 (activity between 8:30-8:40), act1_29 (activity between 8:40-8:50 and act1_30 (activity between 8:50-0:90 variables)
serial pnum act1_28 act1_29 act1_30
11011202 1 3110 3110 3110
11011202 2 3310 3310 7241
11011202 4 9210 9210 9210
My question is can we use TraMineR in this case to mine sequences? Can act1_1, act1_2...act1_144 be defined as a sequence? Can activity codes be used to define states?
Here is how you create the state sequence object from your example data
library(TraMineR)
dat <- matrix(c(11011202, 1, 3110, 3110, 3110,
11011202, 2, 3310, 3310, 7241,
11011202, 4, 9210, 9210, 9210),
nrow=3, ncol=5, byrow=TRUE)
names(dat) <- c("serial", "pnum", "act1_28", "act1_29", "act1_30")
## creating the state sequence object
s <- seqdef(dat[,3:5])
seqiplot(s)
Here, only four different states are involved.
However, the alphabet for your whole dataset is much larger. Several features of TraMineR will either not work or not produce useful outcomes when the alphabet contains too many states. E.g., chronograms or index plots with so many different states would be unreadable. Moreover, TraMineR automatically assigns colors to states only as long as the size of the alphabet does not exceed 12. Of course, many functions such as computing the complexity of the individual sequences (seqici), the sequence of cross-sectional distributions at the successive time intervals and their entropies (seqstatd), or even computing pairwise dissimilarities (seqdist) should work with large alphabets.
Nevertheless, to use TraMineR, I would strongly recommend that you consider to recoding your states so as to drastically reduce your alphabet to say less than 20 different states.
Related
I was able to scrape the title, date, links, and content of news on these links: https://www.news24.com/news24/southafrica/crime-and-courts and https://www.news24.com/news24/southafrica/education. The output is saved in an excel file. However, I noticed that not all the contents inside the articles were scrapped. I have tried different methods on my "Getting content section of my code" Any help with this will be appreciate. Below is my code:
import sys, time
from bs4 import BeautifulSoup
import requests
import pandas as pd
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from datetime import timedelta
art_title = [] # to store the titles of all news article
art_date = [] # to store the dates of all news article
art_link = [] # to store the links of all news article
pagesToGet = ['southafrica/crime-and-courts', 'southafrica/education']
for i in range(0, len(pagesToGet)):
print('processing page : \n')
url = 'https://www.news24.com/news24/'+str(pagesToGet[i])
print(url)
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
try:
driver.get("https://www.news24.com/news24/" +str(pagesToGet[i]))
except Exception as e:
error_type, error_obj, error_info = sys.exc_info()
print('ERROR FOR LINK:', url)
print(error_type, 'Line:', error_info.tb_lineno)
continue
time.sleep(3)
scroll_pause_time = 1
screen_height = driver.execute_script("return window.screen.height;")
i = 1
while True:
driver.execute_script("window.scrollTo(0, {screen_height}{i});".format(screen_height=screen_height, i=i))
i += 1
time.sleep(scroll_pause_time)
scroll_height = driver.execute_script("return document.body.scrollHeight;")
if (screen_height) * i > scroll_height:
break
soup = BeautifulSoup(driver.page_source, 'html.parser')
news = soup.find_all('article', attrs={'class': 'article-item'})
print(len(news))
# Getting titles, dates, and links
for j in news:
titles = j.select_one('.article-item__title span')
title = titles.text.strip()
dates = j.find('p', attrs={'class': 'article-item__date'})
date = dates.text.strip()
address = j.find('a').get('href')
news_link = 'https://www.news24.com' + address
art_title.append(title)
art_date.append(date)
art_link.append(news_link)
df = pd.DataFrame({'Article_Title': art_title, 'Date': art_date, 'Source': art_link})
# Getting Content Section
news_articles = [] # to store the content of each news artcle
news_count = 0
for link in df['Source']:
print('\n')
start_time = time.monotonic()
print('Article No. ', news_count)
print('Link: ', link)
# Countermeasure for broken links
try:
if requests.get(link):
news_response = requests.get(link)
else:
print("")
except requests.exceptions.ConnectionError:
news_response = 'N/A'
# Auto sleep trigger after saving every 300 articles
sleep_time = ['100', '200', '300', '400', '500']
if news_count in sleep_time:
time.sleep(12)
else:
""
try:
if news_response.text:
news_data = news_response.text
else:
print('')
except AttributeError:
news_data = 'N/A'
news_soup = BeautifulSoup(news_data, 'html.parser')
try:
if news_soup.find('div', {'class': 'article__body'}):
art_cont = news_soup.find('div','article__body')
art = []
article_text = [i.text.strip().replace("\xa0", " ") for i in art_cont.findAll('p')]
art.append(article_text)
else:
print('')
except AttributeError:
article = 'N/A'
print('\n')
news_count += 1
news_articles.append(art)
end_time = time.monotonic()
print(timedelta(seconds=end_time - start_time))
print('\n')
# Create a column to add all the scraped text
df['News'] = news_articles
df.drop_duplicates(subset ="Source", keep = False, inplace = True)
# Dont store links
df.drop(columns=['Source'], axis=1, inplace=True)
df.to_excel('SA_news24_3.xlsx')
driver.quit()
I tried the following code in the Getting Content Section as well. However, it produced the same output.
article_text = [i.get_text(strip=True).replace("\xa0", " ") for i in art_cont.findAll('p')]
The site has various types of URLs so your code was omitting them since they found it malformed or some had to be subscribed to read.For the ones that has to be subscribed to read i have added "Login to read" followers by the link in articles . I ran this code till article number 670 and it didn't give any error. I had to change it from .xlsx to .csv since it was giving an error of openpyxl in python 3.11.0.
Full Code
import time
import sys
from datetime import timedelta
import pandas as pd
from bs4 import BeautifulSoup
import requests
import json
art_title = [] # to store the titles of all news article
art_date = [] # to store the dates of all news article
art_link = [] # to store the links of all news article
pagesToGet = ['southafrica/crime-and-courts',
'southafrica/education', 'places/gauteng']
for i in range(0, len(pagesToGet)):
print('processing page : \n')
if "places" in pagesToGet[I]:
url = f"https://news24.com/api/article/loadmore/tag?tagType=places&tag={pagesToGet[i].split('/')[1]}&pagenumber=1&pagesize=100&ishomepage=false&ismobile=false"
else:
url = f"https://news24.com/api/article/loadmore/news24/{pagesToGet[i]}?pagenumber=1&pagesize=1200&ishomepage=false&ismobile=false"
print(url)
r = requests.get(url)
soup = BeautifulSoup(r.json()["htmlContent"], 'html.parser')
news = soup.find_all('article', attrs={'class': 'article-item'})
print(len(news))
# Getting titles, dates, and links
for j in news:
titles = j.select_one('.article-item__title span')
title = titles.text.strip()
dates = j.find('p', attrs={'class': 'article-item__date'})
date = dates.text.strip()
address = j.find('a').get('href')
# Countermeasure for links with full url
if "https://" in address:
news_link = address
else:
news_link = 'https://www.news24.com' + address
art_title.append(title)
art_date.append(date)
art_link.append(news_link)
df = pd.DataFrame({'Article_Title': art_title,
'Date': art_date, 'Source': art_link})
# Getting Content Section
news_articles = [] # to store the content of each news artcle
news_count = 0
for link in df['Source']:
start_time = time.monotonic()
print('Article No. ', news_count)
print('Link: ', link)
news_response = requests.get(link)
news_data = news_response.content
news_soup = BeautifulSoup(news_data, 'html.parser')
art_cont = news_soup.find('div', 'article__body')
# Countermeasure for links with subscribe form
try:
try:
article = art_cont.text.split("Newsletter")[
0]+art_cont.text.split("Sign up")[1]
except:
article = art_cont.text
article = " ".join((article).strip().split())
except:
article = f"Login to read {link}"
news_count += 1
news_articles.append(article)
end_time = time.monotonic()
print(timedelta(seconds=end_time - start_time))
print('\n')
# Create a column to add all the scraped text
df['News'] = news_articles
df.drop_duplicates(subset="Source", keep=False, inplace=True)
# Dont store links
df.drop(columns=['Source'], axis=1, inplace=True)
df.to_csv('SA_news24_3.csv')
Output
,Article_Title,Date,News
0,Pastor gets double life sentence plus two 15-year terms for rape and murder of women,2h ago,"A pastor has been sentenced to two life sentences and two 15-year jail terms for rape and murder His modus operandi was to take the women to secluded areas before raping them and tying them to trees.One woman managed to escape after she was raped but the bodies of two others were found tied to the trees.The North West High Court has sentenced a 50-year-old pastor to two life terms behind bars and two 15-year jail terms for rape and murder.Lucas Chauke, 50, was sentenced on Monday for the crimes, which were committed in 2017 and 2018 in Temba in the North West.According to North West National Prosecuting Authority spokesperson Henry Mamothame, Chauke's first victim was a 53-year-old woman.He said Chauke pretended that he would assist the woman with her spirituality and took her to a secluded place near to a dam.""Upon arrival, he repeatedly raped her and subsequently tied her to a tree before fleeing the scene,"" Mamothame said. The woman managed to untie herself and ran to seek help. She reported the incident to the police, who then started searching for Chauke.READ | Kidnappings doubled nationally: over 4 000 cases reported to police from July to SeptemberOn 10 May the following year, Chauke pounced on his second victim - a 55-year-old woman.He took her to the same secluded area next to the dam, raped her and tied her to a tree before fleeing. This time his victim was unable to free herself.Her decomposed body was later found, still tied to the tree, Mamothame said. His third victim was targeted months later, on 3 August, in the same area. According to Mamothame, Chauke attempted to rape her but failed.""He then tied her to a tree and left her to die,"" he said. Chauke was charged in connection with her murder.He was linked to the crimes via DNA.READ | 'These are not pets': Man gives away his two pit bulls after news of child mauled to deathIn aggravation of his sentence, State advocate Benny Kalakgosi urged the court not to deviate from the prescribed minimum sentences, saying that the offences Chauke had committed were serious.""He further argued that Chauke took advantage of unsuspecting women, who trusted him as a pastor but instead, [he] took advantage of their vulnerability,"" Mamothame said. Judge Frances Snyman agreed with the State and described Chauke's actions as horrific.The judge also alluded to the position of trust that he abused, Mamothame said.Chauke was sentenced to life for the rape of the first victim, 15 years for the rape of the second victim and life for her murder, as well as 15 years for her murder. He was also declared unfit to possess a firearm."
1,'I am innocent': Alleged July unrest instigator Ngizwe Mchunu pleads not guilty,4h ago,"Former Ukhozi FM DJ Ngizwe Mchunu has denied inciting the July 2022 unrest.Mchunu pleaded not guilty to charges that stem from the incitement allegations.He also claimed he had permission to travel to Gauteng for work during the Covid-19 lockdown.""They are lying. I know nothing about those charges,"" alleged July unrest instigator, former Ukhozi FM DJ Ngizwe Brian Mchunu, told the Randburg Magistrate's Court when his trial started on Tuesday.Mchunu pleaded not guilty to the charges against him, which stems from allegations that he incited public violence, leading to the destruction of property, and convened a gathering in contravening of Covid-19 lockdown regulations after the detention of former president Jacob Zuma in July last year.In his plea statement, Mchunu said all charges were explained to him.""I am a radio and television personality. I'm also a poet and cultural activist. In 2020, I established my online radio.READ | July unrest instigators could face terrorism-related charges""On 11 July 2021, I sent invitations to journalists to discuss the then-current affairs. At the time, it was during the arrest of Zuma.""I held a media briefing at a hotel in Bryanston to show concerns over Zuma's arrest. Zuma is my neighbour [in Nkandla]. In my African culture, I regard him as my father.""Mchunu continued that he was not unhappy about Zuma's arrest but added: ""I didn't condone any violence. I pleaded with fellow Africans to stop destroying infrastructure. I didn't incite any violence.I said to them, 'My brothers and sisters, I'm begging you as we are destroying our country.'""He added:They are lying. I know nothing about those charges. I am innocent. He also claimed that he had permission to travel to Gauteng for work during the lockdown.The hearing continues."
2,Jukskei River baptism drownings: Pastor of informal 'church' goes to ground,5h ago,"A pastor of the church where congregants drowned during a baptism ceremony has gone to ground.Johannesburg Emergency Medical Services said his identity was not known.So far, 14 bodies have been retrieved from the river.According to Johannesburg Emergency Medical Services (EMS), 13 of the 14 bodies retrieved from the Jukskei River have been positively identified.The bodies are of congregants who were swept away during a baptism ceremony on Saturday evening in Alexandra.The search for the other missing bodies continues.Reports are that the pastor of the church survived the flash flood after congregants rescued him.READ | Jukskei River baptism: Families gather at mortuary to identify loved onesEMS spokesperson Robert Mulaudzi said they had been in contact with the pastor since the day of the tragedy, but that they had since lost contact with him.It is alleged that the pastor was not running a formal church, but rather used the Jukskei River as a place to perform rituals for people who came to him for consultations. At this stage, his identity is not known, and because his was not a formal church, Mulaudzi could not confirm the number of people who could have been attending the ceremony.Speaking to the media outside the Sandton fire station on Tuesday morning, a member of the rescue team, Xolile Khumalo, said: Thirteen out of the 14 bodies retrieved have been identified, and the one has not been identified yet.She said their team would continue with the search. ""Three families have since come forward to confirm missing persons, and while we cannot be certain that the exact number of bodies missing is three, we will continue with our search."""
3,Six-month-old infant ‘abducted’ in Somerset West CBD,9h ago,"Authorities are on high alert after a baby was allegedly abducted in Somerset West on Monday.The alleged incident occurred around lunchtime, but was only reported to Somerset West police around 22:00. According to Sergeant Suzan Jantjies, spokesperson for Somerset West police, the six-month-old baby boy was taken around 13:00. It is believed the infant’s mother, a 22-year-old from Nomzamo, entrusted a fellow community member and mother with the care of her child before leaving for work on Monday morning. However, when she returned home from work, she was informed that the child was taken. Police were apparently informed that the carer, the infant and her nine-year-old child had travelled to Somerset West CBD to attend to Sassa matters. She allegedly stopped by a liquor store in Victoria Street and asked an unknown woman to keep the baby and watch over her child. After purchasing what was needed and exiting the store, she realised the woman and the children were missing. A case of abduction was opened and is being investigated by the police’s Family Violence, Child Protection and Sexual Offences (FCS) unit. Police obtained security footage which shows the alleged abductor getting into a taxi and making off with the children. The older child was apparently dropped off close to her home and safely returned. However, the baby has still not been found. According to a spokesperson, FCS police members prioritised investigations immediately after the case was reported late last night and descended on the local township, where they made contact with the visibly “traumatised” parent and obtained statements until the early hours of Tuesday morning – all in hopes of locating the child and the alleged suspect.Authorities are searching for what is believed to be a foreign national woman with braids, speaking isiZulu.Anyone with information which could aid the investigation and search, is urged to call Captain Trevor Nash of FCS on 082 301 8910."
4,Have you herd: Dubai businessman didn't know Ramaphosa owned Phala Phala buffalo he bought - report,8h ago,"A Dubai businessman who bought buffaloes at Phala Phala farm reportedly claims he did not know the deal was with President Cyril Ramaphosa.Hazim Mustafa also claimed he was expecting to be refunded for the livestock after the animals were not delivered.He reportedly brought the cash into the country via OR Tambo International Airport, and claims he declared it.A Dubai businessman who reportedly bought 20 buffaloes from President Cyril Ramaphosa's Phala Phala farm claims that he didn't know the deal with was with the president, according to a report.Sky News reported that Hazim Mustafa, who reportedly paid $580 000 (R10 million) in cash for the 20 buffaloes from Ramaphosa's farm in December 2019, said initially he didn't know who the animals belonged to.A panel headed by former chief justice Sandile Ngcobo released a report last week after conducting a probe into allegations of a cover-up of a theft at the farm in February 2020.READ | Ramaphosa wins crucial NEC debate as parliamentary vote on Phala Phala report delayed by a weekThe panel found that there was a case for Ramaphosa to answer and that he may have violated the law and involved himself in a conflict between his official duties and his private business.In a statement to the panel, Mustafa was identified as the source of the more than $500 000 (R9 million) that was stolen from the farm. Among the evidence was a receipt for $580 000 that a Phala Phala employee had written to ""Mr Hazim"".According to Sky News, Mustafa said he celebrated Christmas and his wife's birthday in Limpopo in 2019, and that he dealt with a broker when he bought the animals.He reportedly said the animals were to be prepared for export, but they were never delivered due to the Covid-19 lockdown. He understood he would be refunded after the delays.He also reportedly brought the cash into the country through OR Tambo International Airport and said he declared it. Mustafa also told Sky News that the amount was ""nothing for a businessman like [him]"".READ | Here's the Sudanese millionaire - and his Gucci wife - who bought Ramaphosa's buffaloThe businessman is the owner Sudanese football club Al Merrikh SC. He is married to Bianca O'Donoghue, who hails from KwaZulu-Natal. O'Donoghue regularly takes to social media to post snaps of a life of wealth – including several pictures in designer labels and next to a purple Rolls Royce Cullinan, a luxury SUV worth approximately R5.5 million.Sudanese businessman Hazim Mustafa with his South African-born wife, Bianca O'Donoghue.Facebook PHOTO: Bianca O'Donoghue/Facebook News24 previously reported that he also had ties to former Sudanese president, Omar al-Bashir.There have been calls for Ramaphosa to step down following the saga. A motion of no confidence is expected to be submitted in Parliament.He denied any wrongdoing and said the ANC's national executive committee (NEC) would decide his fate.Do you have a tipoff or any information that could help shape this story? Email tips#24.com"
5,Hefty prison sentence for man who killed stranded KZN cop while pretending to offer help,9h ago,"Two men have been sentenced – one for the murder of a KwaZulu-Natal police officer, and the other for an attempt to rob the officer.Sergeant Mzamiseni Mbele was murdered in Weenen in April last year.He was attacked and robbed when his car broke down on the highway while he was on his way home.A man who murdered a KwaZulu-Natal police officer, after pretending that he wanted to help him with his broken-down car, has been jailed.A second man, who was only convicted of an attempt to rob the officer, has also been sentenced to imprisonment.On Friday, the KwaZulu-Natal High Court in Madadeni sentenced Sboniso Linda, 36, to an effective 25 years' imprisonment, and Nkanyiso Mungwe, 25, to five years' imprisonment.READ | Alleged house robber shot after attack at off-duty cop's homeAccording to Hawks spokesperson, Captain Simphiwe Mhlongo, 39-year-old Sergeant Mzamiseni Mbele, who was stationed at the Msinga police station, was on his way home in April last year when his car broke down on the R74 highway in Weenen.Mbele let his wife know that the car had broken down. While stationary on the road, Linda and Mungwe approached him and offered to help.Mhlongo said: All of a sudden, [they] severely assaulted Mbele. They robbed him of his belongings and fled the scene. A farm worker found Mbele's body the next dayA case of murder was reported at the Weenen police station and the Hawks took over the investigation.The men were arrested.""Their bail [application] was successfully opposed and they appeared in court several times until they were found guilty,"" Mhlongo added.How safe is your neighbourhood? Find out by using News24's CrimeCheckLinda was sentenced to 20 years' imprisonment for murder and 10 years' imprisonment for robbery with aggravating circumstances. Half of the robbery sentence has to be served concurrently, leaving Linda with an effective sentence of 25 years.Mungwe was sentenced to five years' imprisonment for attempted robbery with aggravating circumstances."
I'd like to get all the links in this website https://www.sciencedirect.com/browse/journals-and-books?accessType=openAccess&accessType=containsOpenAccess
Then I'd like go on every link to extract the text in `" after clicking on the button "View Full Aims & Scope".
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
wait = WebDriverWait(driver, 20)
url = "https://www.sciencedirect.com/browse/journals-and-books?accessType=openAccess&accessType=containsOpenAccess"
driver.get(url)
page_description = wait.until(EC.presence_of_element_located((By.XPATH, "//span[#class='pagination-pages-label u-margin-s-left-from-sm u-margin-s-right-from-sm']")))
index_of = wait.until(EC.presence_of_element_located((By.XPATH, "//span[#class='pagination-pages-label u-margin-s-left-from-sm u-margin-s-right-from-sm']"))).text.index('of')
index_number = index_of + 3
time.sleep(2) #otherwise sometimes it doesn't work
length = len(page_description.text)
pages = int(page_description.text[index_number:length])
allLi = []
for i in range(pages):
index = i + 1
url = "https://www.sciencedirect.com/browse/journals-and-books?page="+str(index)+"&accessType=containsOpenAccess&accessType=openAccess"
driver.get(url)
currentAli = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//a[#class='anchor js-publication-title anchor-default']"))
for li in currentAli:
link = li.get_attribute('href');
allLi.append(link)
for li in allLi:
driver.get(li)
button = wait.until(EC.element_to_be_clickable(By.XPATH, "//button[#class='button-link button-link-secondary']"))
button.click()
descrip = wait.until(EC.presence_of_all_elements_located(By.XPATH, "//span[#class='spaced']"))
print(descrip)
First it doesn't work. Then I also have a problem with I try to compute length or pages. Sometimes it works, sometimes it doesn't. Is there anything with the asynchronism? I have to add time.sleep(2). I know it is not a good practice.
Thanks!
Here is a way of getting that information you're after, without the overheads of Selenium (though you might have to add a 2 or 3 seconds between network calls, just to be nice to ScienceDirect server's resources):
from bs4 import BeautifulSoup as bs
from tqdm import tqdm ## if using Jupyter, do from tqdm.notebook import tqdm
import pandas as pd
import cloudscraper
scraper = cloudscraper.create_scraper(disableCloudflareV1=True)
initial_list = []
journal_stuffs = []
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
for x in tqdm(range(1, 39)):
r = scraper.get(f'https://www.sciencedirect.com/browse/journals-and-books?page={x}&accessType=containsOpenAccess&accessType=openAccess')
soup = bs(r.text, 'html.parser')
links = ['https://www.sciencedirect.com' + y.get('href') for y in soup.select('a[class="anchor js-publication-title anchor-default"]')]
initial_list.extend(links)
print('there are ', len(set(initial_list)), 'journal links')
for url in tqdm(initial_list[:20]):
r = scraper.get(url)
soup = bs(r.text, 'html.parser')
title = soup.select_one('a[class="anchor js-title-link anchor-default anchor-has-background-color anchor-has-inherit-color"]').get_text(strip=True)
try:
more_info = soup.select_one('div[class="slide-out"]').get_text(strip=True)
except Exception as e:
more_info = 'Without full aim and scope. A bit pointless, really.'
journal_stuffs.append((title, more_info))
df = pd.DataFrame(journal_stuffs, columns = ['Title', 'Info'])
print(df)
Result in terminal:
100%
38/38 [00:11<00:00, 3.49it/s]
there are 3718 journal links
100%
20/20 [00:12<00:00, 1.46it/s]
Title Info
0 AACE Clinical Case Reports Aims & ScopeAACE Clinical Case Reports is an online journal that publishes case reports with accompanying commentaries six times a year. The primary mission of the journal is to present the most up-to-date information for practicing endocrinologists, fellows in endocrinology and health care professionals dealing with endocrine disorders including diabetes, obesity, osteoporosis, thyroid and other general endocrine disorders.
1 AASRI Procedia Without full aim and scope. A bit pointless, really.
2 Academic Pathology Aims & ScopeAcademic Pathologyis the official open-access journal of theAssociation of Pathology Chairs, established to give voice to innovations in education, practice, and management from academic departments of pathology and laboratory medicine, with the potential for broad impact on medicine, medical research, and the delivery of care.Academic Pathologyaddresses methods for improving patient care (clinical informatics, genomic testing and data management, lab automation, electronic health record integration, and annotate biorepositories); best practices in inter-professional clinical partnerships; innovative pedagogical approaches to medical education and educational program evaluation in pathology; models for training academic pathologists and advancing academic career development; administrative and organizational models supporting the discipline; and leadership development in academic medical centers, health systems, and other relevant venues. Intended authorship and audiences forAcademic Pathologyare international and reach beyond academic pathology itself, including but not limited to healthcare providers, educators, researchers, and policy-makers.Academic Pathologypublishes original research, reviews, brief reports, and educational cases. All articles are rigorously peer-reviewed for relevance and quality.
3 Academic Pediatrics Aims & ScopeAcademic Pediatrics, the official journal of theAcademic Pediatric Association, is a peer-reviewed publication whose purpose is to strengthen the research and educational base of academic generalpediatrics. The journal provides leadership in pediatric education, research, patient care and advocacy. Content areas includepediatric education,emergency medicine,injury,abuse,behavioral pediatrics,holistic medicine,child health servicesandhealth policy,and theenvironment. The journal provides an active forum for the presentation of pediatric educational research in diverse settings, involving medical students, residents, fellows, and practicing professionals. The journal also emphasizes important research relating to the quality of child health care, health care policy, and the organization of child health services. It also includes systematic reviews of primary care interventions and important methodologic papers to aid research in child health and education.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
4 Academic Radiology Aims & ScopeAcademic Radiologypublishes original reports of clinical and laboratory investigations indiagnostic imaging, the diagnostic use ofradioactive isotopes,computed tomography,positron emission tomography,magnetic resonance imaging,ultrasound,digital subtraction angiography,image-guided interventionsand related techniques.\nIt also includes brief technical reports describing original observations, techniques, and instrumental developments; state-of-the-art reports on clinical issues, new technology and other topics of current medical importance; meta-analyses; scientific studies and opinions on radiologic education; and letters to the Editor.
5 ACC Current Journal Review Without full aim and scope. A bit pointless, really.
6 Accident Analysis & Prevention Aims & ScopeAccident Analysis & Preventionprovides wide coverage of the general areas relating toaccidental injuryand damage, including the pre-injury and immediate post-injury phases. Published papers deal with medical, legal, economic, educational, behavioral, theoretical or empirical aspects of transportation accidents, as well as withaccidentsat other sites. Selected topics within the scope of the Journal may include: studies of human, environmental and vehicular factors influencing the occurrence, type and severity of accidents and injury; the design, implementation and evaluation of countermeasures; biomechanics of impact and human tolerance limits to injury; modelling and statistical analysis of accident data; policy, planning and decision-making in safety.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
7 Accounting Forum Without full aim and scope. A bit pointless, really.
8 Accounting, Organizations and Society Aims & ScopeAccounting, Organizations & Society is a leading international interdisciplinary journal concerned with the relationships among accounting and human behaviour, organizational and institutional structures and processes, and the wider socio-political environment of the enterprise. It aims to challenge and extend our understanding of the roles of accounting and related emergent and calculative practices in the construction of economic and societal actors, and their modes of economic organizing, including ways in which such practices influence and are influenced by the development of market and other infrastructures.We aim to publish high quality work which draws upon diverse methodologies and theoretical developments from across the social sciences, and which illuminates the development, processes and effects of accounting within its organizational, political, historical and social contexts. AOS particularly wishes to attract innovative work which analyses accounting phenomena to advance theory development in, for example, the psychological, social psychological, organizational, sociological and human sciences.The journal's unique focus covers, but is not limited to, such topics as:•\tThe roles of accounting in organizations and society;•\tThe contribution of accounting practices to the emergence, maintenance and transformation of organizational and societal institutions;•\tThe roles of accounting in the development of new organizational and institutional forms, both public and private;•\tThe relationships between accounting, auditing, accountability, ethics and social justice;•\tBehavioural studies of accounting practices and the providers, verifiers, and users of accounting information, including cognitive aspects of accounting, judgment and decision-making processes, and the behavioural aspects of planning, control and valuation processes;•\tOrganizational process studies of the design, implementation and use of accounting, information and management control systems;•\tAccounting for human actors, and the impact of accounting technologies upon human subjectivities and evaluations;•\tThe roles of accounting in shaping the design, operation and delivery of public service providers, not-for-profit entities, government bodies, as well as local, national and transnational governmental organizations;•\tSocial, organizational, political, and psychological studies of the standard-setting process, and the effects of accounting regulations and rules;•\tThe roles and practices of audit, auditors and accounting firms in the construction and understanding of organizational and societal valuations;•\tAccounting for sustainability and the environment, including studies of environmental and social reporting;•\tHistorical studies of the emergence, transformation and impact of accounting calculations, practices, and representations, including the development and the changing roles of accounting theories, techniques, individual and teams of practitioners and their firms, professional associations, and regulators.Benefits to authorsWe also provide many author benefits, such as free PDFs, a liberal copyright policy, special discounts on Elsevier publications and much more. Please click here for more information on ourauthor services.Please see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
9 Achievements in the Life Sciences Without full aim and scope. A bit pointless, really.
10 ACOG Clinical Review Without full aim and scope. A bit pointless, really.
11 Acta Anaesthesiologica Taiwanica Without full aim and scope. A bit pointless, really.
12 Acta Astronautica Aims & ScopeActa Astronauticais sponsored by theInternational Academy of Astronautics. Content is based on original contributions in all fields of basic,engineering,lifeandsocial space sciencesand ofspace technologyrelated to:The peaceful scientific exploration of space,Its exploitation for human welfare and progress,Conception, design, development and operation of space-borne and Earth-based systems,In addition to regular issues, the journal publishes selected proceedings of the annual International Astronautical Congress (IAC), transactions of the IAA and special issues on topics of current interest, such asmicrogravity,space station technology,geostationary orbits, andspace economics. Other subject areas includesatellite technology,space transportationandcommunications,space energy,power and propulsion,astrodynamics,extraterrestrial intelligenceandEarth observations.For more information on the International Academy of Astronautics (IAA), visit their home page:http://www.iaaweb.org. Members of the IAA are eligible for a discount on a personal subscription toActa Astronautica. Please clickhereto download an order form.
13 Acta Biomaterialia Aims & ScopeActa Biomaterialiais an international journal that publishes peer-reviewed original research reports, review papers and communications in the broadly defined field ofbiomaterials science. The emphasis of the journal is on the relationship betweenbiomaterial structureandfunctionat all length scales.The scope ofActa Biomaterialiaincludes:Hypothesis-driven design of biomaterialsBiomaterial surface science linking structure to biocompatibility, including protein adsorption and cellular interactionsBiomaterial mechanical characterization and modeling at all scalesMolecular, statistical and other types of modeling applied to capture biomaterial behaviorInteractions of biological species with defined surfacesCombinatorial approaches to biomaterial developmentStructural biology as it relates structure to function for biologically derived materials that have application as a medical material, or as it aids in understanding the biological response to biomaterialsMethods for biomaterial characterizationProcessing of biomaterials to achieve specific functionalityMaterials development for arrayed genomic and proteomic screeningBenefits to authorsFree and automatic manuscript deposit service to meet NIH public access requirements at one year;Multiple options for data-sharing (seehttp://www.materialstoday.com/materials-genome-initiative/);Free author pdf and Sharelink share your article with your peers (seehttps://www.elsevier.com/journal-authors/share-link);And more information on our author services can be foundherePlease see ourGuide for Authorsfor information on article submission. If you require any further information or help, please visit ourSupport Center
14 Acta Colombiana de Cuidado Intensivo Aims & ScopeActa Colombiana de Cuidado Intensivois the official publication of the Asociación Colombiana de Medicina Crítica y Cuidado Intensivo (Colombian Association of Critical Medicine and Intensive Care). It is published every three months in March, June, September and December and is intended to be a means of dissemination in all areas associated with the management of the critically ill patient.All the manuscripts received by theActa Colombiana de Cuidado Intensivoare reviewed using a double blind system by experts in the specialty (peer review).The Journal publishes articles on research (Originals), Reviews, Case Reports , and Case Series, as well as Articles on Reflections, and Clinical Comments. Also, it offers the possibility of publishing supplements on specific topics that allows the reader to get into a particular area of knowledge in depth.The development of Intensive Care has encouraged certain areas of specialisation within the specialists dedicated to the care of the critically ill patient. Responding to this need,Acta Colombiana de Cuidado Intensivopays particular attention to certain areas of interests which are made up by experts. The subject matter organisation of the Journal enables it to approach not just technical subjects, but also those related to the logistic organisation of the practice of intensive care.The areas of interest of the Journal are the following:• epidemiology• infection and sepsis• coagulation and inflammation• cardiovascular critical care• mechanical ventilation• bioethics• nutrition and metabolism• quality and costs• neurological intensive care• toxicology• trauma• obstetrics intensive care• sedation and analgesia• paediatrics intensive careLa revistaActa Colombiana de Cuidado Intensivoes el órgano oficial de la Asociación Colombiana de Medicina Crítica y Cuidado Intensivo. Se publica trimestralmente en los meses de marzo, junio, septiembre y diciembre y pretende ser un órgano de divulgación en todas las áreas relacionadas con el manejo del paciente críticamente enfermo.Todos los manuscritos recibidos porActa Colombiana de Cuidado Intensivoson revisados mediante el sistema de doble ciego por expertos de la especialidad.La revista publica artículos de investigación (Originales), de Revisión, Reportes de Casos y Series de Casos, así como Artículos de Reflexión y Comentarios Clínicos. Además, ofrece la posibilidad de publicar suplementos sobre temas específicos que permitan al lector profundizar a fondo en un área particular del conocimiento.El desarrollo del cuidado intensivo ha promovido ciertas áreas de especialización dentro de los especialistas dedicados al cuidado del paciente críticamente enfermo. Respondiendo a esta necesidad,Acta Colombiana de Cuidado Intensivopresta especial atención a determinadas áreas de interés en las que se agrupa a los expertos. La organización temática de la revista permite abordar no solo temas técnicos sino también aquellos relacionados con la organización logística de la práctica del cuidado intensivo.Las áreas de interés de la revista son las siguientes:• epidemiologia• infecciones y sepsis• coagulación e inflamación• cuidado crítico cardiovascular• ventilación mecánica• bioética• nutrición y metabolismo• calidad y costos• cuidado intensivo neurológico• toxicología• trauma• cuidado intensivo obstétrico• sedación y analgesia• cuidado intensivo pediátrico
15 Acta Ecologica Sinica Aims & ScopeActa Ecologica Sinica (International Journal)is a bimonthly academic journal sponsored by the Ecological Society of China and the Research Center for Eco-Environmental Sciences, Chinese Academy of Sciences.Acta Ecologica Sinicapublishes novel research inecology, promotes the exchange and cooperation of ecologists and ecological research between developing and developed countries. The Journal aims to show the scientific mechanism of the interaction between life and environment and facilitates the academic dissemination and scientific development of ecological research in the world, especially in developing countries.Position of the journalActa Ecologica Sinicais a comprehensive journal devoted to the development of Ecology and its sub-disciplines. It unites ecological scientists in the world with the aim of publishing high-quality papers on innovative research. Published papers should unveil mechanisms of the interactions between life and environment, and contribute to the innovation and sustainable development of ecological science in the world.International perspectiveMembers of the editorial board of Acta Ecologica Sinica are all internationally renowned ecologists, and its presentEditor-in-Chiefis the academician of theChinese Academy of Sciences (CAS)and theChinese Academy of Engineering (CAE). In recent years, Acta Ecologica Sinica is receiving an increasing international attention, and its editorial members come from 8 different countries and regions in various areas of ecological research, which strengthens the journal`s impact worldwide. It is anticipated that Acta Ecologica Sinica will further gain more international recognition and have a great prospect of development.Journal coverageThis journal publishes papers on animal ecology, plant ecology, microbial ecology, agro-ecology, forestry ecology, grassland ecology, soil ecology, ocean and aquatic ecosystems, landscape ecology, chemical ecology, contaminant ecology, urban and human ecology. We particularly welcome reviews on recent developments in ecology, novel experimental studies, and short communications, new theories, methodologies, new techniques, book reviews, and research news and laboratory introductions.
16 Acta Histochemica Aims & ScopeActa Histochemicais a classic scientific journal established in 1954 currently focused on basic research and methodological innovations in cell and tissue biology. The aim of the journal is to promote the peer-reviewed publication of original articles and short communications reporting novel results and experimental approaches in the field, as well as comprehensive reviews, letters to the editor and meeting reports, serving as an open forum for the cell and histochemical research community. Manuscripts analysing the mechanisms of functional regulation of living systems at a cell/tissue level, in physiological or pathological conditions, or reporting new techniques and methodological approaches to quantify/visualize cellular activities are particularly welcomed.
17 Acta de Investigación Psicológica Without full aim and scope. A bit pointless, really.
18 Acta Materialia Aims & ScopeActa Materialiaprovides a forum for publishing full-length, original papers and commissioned overviews that advance the in-depth understanding of the relationship between the processing, the structure and the properties of inorganic materials. Papers that have a high impact potential and/or substantially advance the field are sought. The structure encompasses atomic and molecular arrangements, chemical and electronic structures, and microstructure. Emphasis is on either the mechanical or functional behavior of inorganic solids at all length scales down to nanostructures.The following aspects of the science and engineering of inorganic materials are of particular interest:(i) Cutting-edge experiments and theory as they relate to the understanding of the properties,(ii) Elucidation of the mechanisms involved in the synthesis and processing of materials specifically as they relate to the understanding of the properties,and(iii) Characterization of the structure and chemistry of materials specifically as it relates to the understanding of the properties.Acta Materialiawelcomes papers that employ theory and/or simulation (or numerical methods) that substantially advance our understanding of the structure and properties of inorganic materials. Such papers should demonstrate relevance to the materials community by, for example, making a comparison with experimental results (in the literature or in the present study), making testable microstructural or property predictions or elucidating an important phenomenon. Papers that focus primarily on model parameter studies, development of methodology or those employing existing software packages to obtain standard or incremental results are discouraged.Short communications and comments to papers published inActa Materialiamay besubmitted toScripta Materialia.
19 Acta Metallurgica Without full aim and scope. A bit pointless, really.
I am trying to extract all locations from a web page that are grouped into a large id=results div.
https://www.dynacast.com/en-gb/discover-dynacast/locations
My current code produces an empty list that I cannot interact with:
browser = mechanicalsoup.StatefulBrowser(soup_config={'features': 'lxml'}
locations_page = browser.page
locations_list = locations_page.find_all("div", id="location-list")
for locations_grid in locations_list:
grid = locations_grid.find_all_next("div", class_="grid-4")
print(grid)
Output = []
My aim is to extract all the text from this locations table (North America, Europe etc.) and the locations within the continents and append it to a list. Thank you in advance.
The page is dynamic so you won't get that html from a simple request. My suggestion is just get the data in json format from the api and can use pandas to normalize it:
import requests
import pandas as pd
url = 'https://www.dynacast.com/api/sitecore/Location/LocationsByServiceAjax'
payload = {
'serviceId': '',
'contextItemId': 'a9640857-ede5-47b3-a692-1606ee151d7c'}
jsonData = requests.get(url, params=payload).json()
df = pd.json_normalize(jsonData)
Output: 1st 5 of 15 rows below
print(df.head(5).to_string())
Id LocationName LocationAddress Phone Continent Country DirectionsLink CountryUrl Content Latitude Longitude SummaryContent SalesPhone ItemUrl IsFeatured
0 7ec388a8-adfa-4652-90b7-af1a089dfba1 Dynacast Tunisia Parc d'Activités Economiques de Bizerte\r\nSite de Menzel Bourguiba II BP 130\r\n7050 Menzel Bourguiba\r\nTunisie +216 72418363 Africa Tunisia https://www.google.com/maps?daddr=Parc+d%27Activit%C3%A9s+Economiques%2BMenzel+Bourguiba%2BBizerte%2BTunisia <p>Dynacast acquired Taurus Tunisia in 2016; it is located in Menzel Bourguiba, Bizerte, Tunisia just 30 minutes from the Tunis International Airport. This 1,900m<sup>2</sup> facility specializes in precision zinc die casting. Additional details about this location include:</p>\n<ul>\n<li>12 conventional zinc die casting machines ranging from 20T to 135T</li>\n<li>Production range from 0.5g to 0.9 kg</li>\n<li>Assembly operations </li>\n<li>In-house tool maintenance and repair</li>\n<li>Wide range of secondary operations and surface finishes including milling, thread cutting, sand blasting, and tumbling</li>\n</ul> 37.1450007 9.8045612 <p>Dynacast acquired Taurus Tunisia in 2016; it is located in Menzel Bourguiba, Bizerte, Tunisia. This 1,900m<sup>2</sup> facility specializes in precision zinc die casting.</p> /en/discover-dynacast/locations/dynacast-tunisia False
1 a7138364-2693-45b7-a886-4d526a325428 Sundaram Dynacast India No. 25 A/2, Sidco Industrial Park\nNorth Phase, Ambattur\nChennai, Tamil Nadu 600098 +91.44.2625.9169 Asia India https://www.google.com/maps?daddr=No.+25+A%2F2%2C+Sidco+Industrial+Park%2BChennai%2BTamil+Nadu%2BIndia <p>Sundaram Dynacast is a 68,000 square foot facility located in Ambattur, Chennai, India. The facility is the largest zinc high pressure hot chamber die caster in this area. Below are additional details about what is offered at this location:</p>\n<ul>\n<li>19 multi-slide proprietary zinc die casting machines ranging from 2.5 to 4.5 tons</li>\n<li>7 standard zinc die casting machines ranging from 25 to 63 tons</li>\n<li>8 aluminum die casting machine ranging from 125T to 350 tons</li>\n<li>Precision tool building with in-house capabilities</li>\n<li>Wide range of secondary operations including CNC machining, tapping, reaming, drilling, and assembly available through certified/dedicated partners</li>\n<li>Wide range of surface finishes are available in-house including tumbling, polishing, shot blasting, de-burring etc.. Plating, powder coating etc. are supported through our certified partners</li>\n</ul>\n<p /> 13.1094448 80.1666248 <p>Sundaram Dynacast is a 68,000 square foot facility located in Ambattur, Chennai, India. The facility is the largest zinc high pressure hot chamber die caster in this area. </p> /en/discover-dynacast/locations/sundaram-dynacast-india False
2 a2129999-b0d7-458b-bdd4-ae8a15dd4276 Dynacast Singapore 6 Second Chin Bee Road\r\nJurong, 618773 65.6268.7644 ext. 135 Asia Singapore <p>Dynacast Singapore focuses on multi-slide zinc, conventional zinc, aluminum, and magnesium die casting. Below are additional details about what is offered at the Singapore location:</p>\n<ul>\n<li>40 multi-slide proprietary zinc die casting machines ranging from 4 to 27 tons</li>\n<li>1 standard zinc die casting machines up to 125 tons</li>\n<li>4 aluminum die casting machines up to 135 tons</li>\n<li>1 multi-slide magnesium die casting machine up to 20 tons</li>\n<li>Precision tool building with in-house capabilities</li>\n<li>Wide range of secondary operations including CNC machining, tapping, reaming, and drilling available in house and through certified partners</li>\n<li>Design, prototyping, and modeling services</li>\n<li>Wide range of finishing surfaces available including tumbling, polishing, shot blasting and thermal de-burring</li>\n</ul>\n<p>www.dynacast.com.sg</p>\n<p><br /><strong>Dynacast Singapore works with OptiMIM, utilizing state-of-the-art metal injection molding technology to create high performing small precision metal components to almost any level of complexity and at virtually any volume.</strong></p> 1.332509 103.714238 <p>Dynacast Singapore focuses on multi-slide zinc, conventional zinc, aluminum, and magnesium die casting. </p> /en/discover-dynacast/locations/dynacast-singapore False
3 f2dd9429-d9d4-4028-b8ca-227972771980 Dynacast Tokyo (Regional Support Office) Azabu Green Terrace 5F\r\nMinamiazabu 3-20-1\r\nMinato-ku, Tokyo 106-0047 +03 6859-8512 Asia Japan <p>Dynacast Tokyo is a Regional Support Office that is strategically located to help with the demand for die casting components in the Asian market. Our sales team is available to help with any of your small precision component needs. </p>\n<p><a rel="noopener noreferrer" href="http://www.dynacast.jp" target="_blank">www.dynacast.jp</a></p> 35.649991 139.7307153 <p>Dynacast Tokyo is a sales office that is strategically located to help with the demand for die casting components in the Asian market. Our sales team is available to help with any of your small precision component needs. </p> /en/discover-dynacast/locations/dynacast-tokyo False
4 a4535e88-d1c3-4489-b351-cb75c4658f61 Dynacast Dongguan No. 88 Qiufu Road District\r\nFumin Industrial Park 2\r\nDalang, Dongguan, Guangdong 523778 86.769.8222.0638 (8801) Asia China https://www.google.com/maps?daddr=No.+88+Qiufu+Road+District%2BDalang%2C+Dongguan%2BGuangdong%2BChina <p>Dynacast Dongguan is a 14,000 square meter facility that is located in the heart of the Guangdong Province. The facility delivers high quality parts to the leading manufacturing hub in Southern China and focuses on zinc, aluminum, and magnesium die casting. Below are additional details about what is offered at the Dongguan location:</p>\n<ul>\n<li>28 zinc die casting machines ranging from 2 to 250 tons</li>\n<li>20 aluminum die casting machines ranging from 125 to 350 tons</li>\n<li>4 standard magnesium die casting machines ranging from 60 to 350 tons</li>\n<li>4 aluminum & magnesium die cast machines ranging from 60 to 350 tons </li>\n<li>2 standard plastic injection machines up to 100 tons</li>\n<li>Precision tool building with in-house capabilities</li>\n<li>Wide range of secondary operations including CNC machining, tapping, reaming, and drilling available in house and through certified partners</li>\n<li>Design, prototyping, and modeling services</li>\n<li>Wide range of finishing surfaces available including E-coating, electroless nickel plating, bright chrome, black chrome, spray painting, black anodising, micro-arc oxidation, copper nickle chrome, powder coating and passivation</li>\n</ul>\n<p>www.dynacast.cn</p>\n<p /> 22.9455755 113.9313944 <p>Dynacast Dongguan is a 14,000 square meter facility that is located in the heart of the Guangdong Province. The facility delivers high quality parts to the leading manufacturing hub in Southern China and focuses on zinc, aluminum, and magnesium die casting.</p> /en/discover-dynacast/locations/dynacast-dongguan False
hey this was kind of explained to me before but having trouble appying the same thing now to almost the same page...
page = 'http://www.imdb.com/genre/action/?ref_=gnr_mn_ac_mp'
table = soup.find_all("table", {"class": "results"})
for item in list(table):
for info in item.contents[1::2]:
info.a.extract()
link = info.a['href']
print(link)
name = info.text.strip()
print(name)
code above tries to capture the link to each page of each film contained in the a tag in the variable info... and the text in it has the name of each film but instead i get all the text. is there any way of just getting the name?
thanks guys in advance!!!
Just just need to pull the text from the anchor tag inside the td with the class title:
In [15]: from bs4 import BeautifulSoup
In [16]: import requests
In [17]: url = "http://www.imdb.com/genre/action/?ref_=gnr_mn_ac_mp"
In [18]: soup = BeautifulSoup(requests.get(url,"lxml").content)
In [19]: for td in soup.select("table.results td.title"):
....: print(td.a.text)
....:
X-Men: Apocalypse
Warcraft
Captain America: Civil War
The Do-Over
Teenage Mutant Ninja Turtles: Out of the Shadows
The Angry Birds Movie
The Nice Guys
Batman v Superman: Dawn of Justice
Suicide Squad
Deadpool
Gods of Egypt
Zootopia
13 Hours: The Secret Soldiers of Benghazi
Now You See Me 2
The Brothers Grimsby
Hardcore Henry
Monster Trucks
Independence Day: Resurgence
Star Trek Beyond
The Legend of Tarzan
Deepwater Horizon
X-Men: Days of Future Past
Star Wars: The Force Awakens
X-Men: First Class
The 5th Wave
Pretty much all the data you would want is inside the td with the title class:
So if you wanted the outline also all you need is the text from the span.outline:
In [24]: for td in soup.select("table.results td.title"):
....: print(td.a.text)
....: print(td.select_one("span.outline").text)
....:
X-Men: Apocalypse
With the emergence of the world's first mutant, Apocalypse, the X-Men must unite to defeat his extinction level plan.
Warcraft
The peaceful realm of Azeroth stands on the brink of war as its civilization faces a fearsome race of...
Captain America: Civil War
Political interference in the Avengers' activities causes a rift between former allies Captain America and Iron Man.
The Do-Over
Two down-on-their-luck guys decide to fake their own deaths and start over with new identities, only to find the people they're pretending to be are in even deeper trouble.
Teenage Mutant Ninja Turtles: Out of the Shadows
As Shredder joins forces with mad scientist Baxter Stockman and henchmen Bebop and Rocksteady to take over the world, the Turtles must confront an even greater nemesis: the notorious Krang.
The Angry Birds Movie
Find out why the birds are so angry. When an island populated by happy, flightless birds is visited by mysterious green piggies, it's up to three unlikely outcasts - Red, Chuck and Bomb - to figure out what the pigs are up to.
The Nice Guys
A mismatched pair of private eyes investigate the apparent suicide of a fading porn star in 1970s Los Angeles.
Batman v Superman: Dawn of Justice
Fearing that the actions of Superman are left unchecked, Batman takes on the Man of Steel, while the world wrestles with what kind of a hero it really needs.
Suicide Squad
A secret government agency recruits imprisoned supervillains to execute dangerous black ops missions in exchange for clemency.
Deadpool
A former Special Forces operative turned mercenary is subjected to a rogue experiment that leaves him with accelerated healing powers, adopting the alter ego Deadpool.
Gods of Egypt
Mortal hero Bek teams with the god Horus in an alliance against Set, the merciless god of darkness, who has usurped Egypt's throne, plunging the once peaceful and prosperous empire into chaos and conflict.
Zootopia
In a city of anthropomorphic animals, a rookie bunny cop and a cynical con artist fox must work together to uncover a conspiracy.
13 Hours: The Secret Soldiers of Benghazi
During an attack on a U.S. compound in Libya, a security team struggles to make sense out of the chaos.
Now You See Me 2
The Four Horsemen resurface and are forcibly recruited by a tech genius to pull off their most impossible heist yet.
The Brothers Grimsby
A new assignment forces a top spy to team up with his football hooligan brother.
Hardcore Henry
Henry is resurrected from death with no memory, and he must save his wife from a telekinetic warlord with a plan to bio-engineer soldiers.
Monster Trucks
Looking for any way to get away from the life and town he was born into, Tripp (Lucas Till), a high school senior...
Independence Day: Resurgence
Two decades after the first Independence Day invasion, Earth is faced with a new extra-Solar threat. But will mankind's new space defenses be enough?
Star Trek Beyond
The USS Enterprise crew explores the furthest reaches of uncharted space, where they encounter a mysterious new enemy who puts them and everything the Federation stands for to the test.
The Legend of Tarzan
Tarzan, having acclimated to life in London, is called back to his former home in the jungle to investigate the activities at a mining encampment.
Deepwater Horizon
A story set on the offshore drilling rig Deepwater Horizon, which exploded during April 2010 and created the worst oil spill in U.S. history.
X-Men: Days of Future Past
The X-Men send Wolverine to the past in a desperate effort to change history and prevent an event that results in doom for both humans and mutants.
Star Wars: The Force Awakens
Three decades after the defeat of the Galactic Empire, a new threat arises. The First Order attempts to rule the galaxy and only a ragtag group of heroes can stop them, along with the help of the Resistance.
X-Men: First Class
In 1962, the United States government enlists the help of Mutants with superhuman abilities to stop a malicious dictator who is determined to start World War III.
The 5th Wave
Four waves of increasingly deadly alien attacks have left most of Earth decimated. Cassie is on the run, desperately trying to save her younger brother.
For runtime td.select_one("span.runtime").text etc..
Just like how you got the link by doing
info.a['href']
You can also get the title of the movie by doing
info.a['title']
Hopefully this is what you're looking for!
I'd like to get back only one row per ID, even if there are multiple rows associated with an ID.
The scenario is that the data reflects a site selling land / properties, and that for each listing, there may be multiple images associated with each ID. I need only one.
Thank You.
As requested, sample data:
RowNumber ID UserID CountryID tbl_detailsID Name Description Price AddressLine1 AddressLine2 City State Zip AddressGPS Space Amenities Policies URL DateEntered DateExpires NumViews IsCompanyProperty EnforceMinStay IsOtherSite rate_Avg isFeatured DayPrice ImageURL Name check_in check_out currency bathroom_count pet unit_size elder handicap occupancy bedroom_count LinkURL smoking unit_size_Units property_Type children Minimum_stay Minimum_stay_Unit CountryName
2 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-007-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
3 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-006-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
4 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-005-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
5 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-004-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
6 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-003-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
7 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-002-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
8 247586 1 1 161398 Beautiful Oceanfront Home in Cambria California Located on the Central Coast of California 10 minutes from Hearst Castle.
Phone 1: Toll Free (800) 240-2277
Phone 2: (805) 927-0306 (California, USA)
direct email contact: debbie (#) maisonsdecambria.com
Leopold Cove" is a beautiful Oceanfront Home located on the Central Coast of California. It has fabulous ocean views and access to the water from a private stairway. The living area has large plate glass windows that feature 180 degree views of the Pacific Ocean. A comfortable platform style bed & a warm, cozy fireplace are also located in the living area so you can relax for hours, or even days, enjoying the beauty that surrounds you. There is a full kitchen & a separate bedroom with 2 twin beds. The property also has a picnic area & places to sit, perhaps with a good book & a glass of wine while you watch the sunset. If you are a nature lover, you will enjoy the otters, seals & birdlife that we are privileged to have so close to us. During the gray whale migration times, this is a p 2183 Sherwood Drive Cambria California 93428 35.544712|-121.095444 http://www.flipkey.com/cambria-vacation-rentals/p202618/ 2012-10-01 10:52:00 2012-10-15 10:52:00 0 0 0 703 5.0 0 NULL http://images1.flipkey.com/img/photos/302459/243298/large_302459-243298-001-1346943508.jpg Beautiful Oceanfront Home in Cambria California USD 1 Ask 0 Ask Ask 4 1 http://www.flipkey.com/cambria-vacation-rentals/p202618/ Ask House Ask 0 NULL United States
And the Omni-query I that I am constructing to replace a few hundred lines of temp tables:
SELECT DISTINCT ROW_NUMBER() OVER(ORDER BY tbl_Properties.ID DESC) AS [RowNumber], tbl_Properties.ID, tbl_Properties.UserID, tbl_Properties.CountryID, tbl_Properties.tbl_detailsID, tbl_Properties.Name,
tbl_Properties.[Description], tbl_Properties.Price, tbl_Properties.AddressLine1, tbl_Properties.AddressLine2, tbl_Properties.City, tbl_Properties.[State], tbl_Properties.Zip, tbl_Properties.AddressGPS,
tbl_Properties.[Space],
tbl_Properties.Amenities,
tbl_Properties.Policies,
tbl_Properties.URL,
tbl_Properties.DateEntered,
tbl_Properties.DateExpires,
tbl_Properties.NumViews,
tbl_Properties.IsCompanyProperty,
tbl_Properties.EnforceMinStay,
tbl_Properties.IsOtherSite,
tbl_Properties.rate_Avg,
tbl_Properties.isFeatured,
tbl_Properties.DayPrice,
tbl_Images.ImageURL,
tbl_Details.Name,
tbl_Details.check_in,
tbl_Details.check_out,
tbl_Details.currency,
tbl_Details.bathroom_count,
tbl_Details.pet,
tbl_Details.unit_size,
tbl_Details.elder,
tbl_Details.handicap,
tbl_Details.occupancy,
tbl_Details.bedroom_count,
tbl_Details.url As [LinkURL],
tbl_Details.smoking,
tbl_Details.unit_size_Units,
tbl_Details.property_Type,
tbl_Details.children,
tbl_Details.Minimum_stay,
tbl_Details.Minimum_stay_Unit,
tbl_Countries.Name As [CountryName]
FROM [tbl_Details]
INNER JOIN [tbl_Properties] ON [tbl_Details].[detailsID] = [tbl_Properties].[tbl_detailsID]
INNER JOIN [tbl_Images] ON [tbl_Images].PropertyID = [tbl_Properties].ID
INNER JOIN [tbl_Countries] ON [tbl_Countries].ID = [tbl_Properties].CountryID
The query itself is, of course, not finished yet.
If your SQLServer version supports row_number() OVER (2008,2012 support)
select * from
(select t.*,
row_number() over (partition by id order by id) as rownumber from t
) t1
where t1.rownumber=1
Here is the SQLFiddle example
Since there's no sample data you'll need to bend this response to suit your needs
select
Listings.*, SingleImage.Column1, SingleImage.Column2
from Listings
cross apply (
select top 1 *
from Images
where Listings.ListingID = Images.ListingID
) SingleImage
That needs SQL 2005 or newer (for the "cross apply" to work)
There are ways to make it work in SQL 2000 too but they're not as efficient
try this:
SELECT TOP 1 * FROM
imagestable
WHERE imagestable.landID = <theID>
This will return the first image in the table that belongs to the property.
You can INNER JOIN this query with your property table if you want to return all property info as a single record.
In SQL Server 2005+ you can use this
use tempdb
go
create table tbl (id int, s varchar(50))
insert into tbl values (1,'qwe'),(1,'asd'), (1,'qwff'),(2,'ert'),(2,'ykf'),(3,'')
;with cte
AS
(
SELECT id,s, ROW_NUMBER() OVER (PARTITION BY id ORDER BY s ) n
FROM tbl
)
SELECT id,s
FROM CTE
WHERE n=1
drop table tbl