if the class is not found append a int/Nan in bs4 - selenium
I've been trying to figure out how to append an integer or a Nan when the class isn't found, but it seems not working, and instead is skipping the if/else statement.
any idea?
rooms_bed = []
for a in soup.find_all("span", attrs={"class":"no-svg-bed-icon bed-icon seperator"}):
for s in a.select('title'):
if len(s) == 0:
rooms_bed.append(0)
else:
rooms_bed.append(s.get_text())
number_bedrooms = []
for x in rooms_bed:
if "bedrooms" in x:
number_bedrooms.append(int(x.replace('bedrooms', '')))
I am somehow fairly convinced this is yet another XY Problem.
Therefore, here is one tested method of obtaining all data from that page:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
import re
import json
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
}
url = 'https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E1268&index=120&propertyTypes=&includeSSTC=false&mustHave=&dontShow=&furnishTypes=&keywords='
soup = bs(requests.get(url, headers=headers).text, 'html.parser')
data = soup.find(string=re.compile("^window.jsonModel")).split(' = ')[1]
df = pd.json_normalize(json.loads(data)['properties'])
print(df)
Result in terminal:
id bedrooms bathrooms numberOfImages numberOfFloorplans numberOfVirtualTours summary displayAddress countryCode propertySubType premiumListing featuredProperty distance transactionType commercial development residential students auction feesApply feesApplyText displaySize showOnMap propertyUrl contactUrl staticMapUrl channel firstVisibleDate keywords keywordMatchType saved hidden onlineViewingsAvailable hasBrandPlus displayStatus enquiredTimestamp heading formattedBranchName addedOrReduced propertyTypeFullDescription enhancedListing isRecent formattedDistance location.latitude location.longitude propertyImages.images propertyImages.mainImageSrc propertyImages.mainMapImageSrc listingUpdate.listingUpdateReason listingUpdate.listingUpdateDate price.amount price.frequency price.currencyCode price.displayPrices customer.branchId customer.brandPlusLogoURI customer.contactTelephone customer.branchDisplayName customer.branchName customer.brandTradingName customer.branchLandingPageUrl customer.development customer.showReducedProperties customer.commercial customer.showOnMap customer.enhancedListing customer.developmentContent customer.buildToRent customer.buildToRentBenefits customer.brandPlusLogoUrl productLabel.productLabelText productLabel.spotlightLabel lozengeModel.matchingLozenges
0 128841404 3 1 19 1 0 Bridgfords Hazel Grove are over the moon to we... Deneside Crescent, Hazel Grove, Stockport, Che... GB Semi-Detached True True None buy False False True False False False None True /properties/128841404#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-07T17:22:14Z [] no_keyword False False False True None Featured Property by Bridgfords, Hazel Grove Reduced on 23/11/2022 3 bedroom semi-detached house for sale False False 53.382230 -2.110010 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-23T22:14:22Z 280000 not specified GBP [{'displayPrice': '£280,000', 'displayPriceQua... 93 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 4990 Bridgfords, Hazel Grove Hazel Grove Bridgfords /estate-agents/agent/Bridgfords/Hazel-Grove-93... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Corner Plot False []
1 114569264 3 2 14 1 1 A well presented DETACHED BUNGALOW enjoying a ... Parsonage Gardens, Marple GB Bungalow True False None buy False False True False False False None True /properties/114569264#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2021-10-08T16:59:54Z [] no_keyword False False True True None by Edward Mellor Ltd, Marple Added on 08/07/2022 3 bedroom bungalow for sale False False 53.388240 -2.060480 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-07-08T15:07:43Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 28598 /company/clogo_rmchoice_12942_0000.png 0161 524 6040 Edward Mellor Ltd, Marple Marple Edward Mellor Ltd /estate-agents/agent/Edward-Mellor-Ltd/Marple-... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Cul-De-Sac False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
2 128949689 3 1 15 0 1 A rare opportunity to acquire this three bedro... Delamere Close, Stockport, SK7 GB Detached Bungalow True False None buy False False True False False False None True /properties/128949689#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-10T12:42:26Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, Hazel... Added on 10/11/2022 3 bedroom detached bungalow for sale False False 53.381600 -2.102170 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-10T12:48:02Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 12271 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 9316 Ian Tonge Property Services Limited, Hazel Grove Hazel Grove Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
3 127513385 4 2 17 1 1 Julian Wadden are delighted to be selling this... Boddens Hill Road, Heaton Mersey, Stockport, SK4 GB Detached False False None buy False False True False False False None True /properties/127513385#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-28T15:41:42Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Added on 28/09/2022 4 bedroom detached house for sale False False 53.411211 -2.194055 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-09-28T15:47:04Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... False []
4 128983526 6 2 15 1 1 A well presented large family home in a popula... Shawbrook Road, Manchester, Greater Manchester... GB Semi-Detached True False None buy False False True False False False None True /properties/128983526#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-11T11:10:01Z [] no_keyword False False False True None by Bridgfords, Withington Added on 11/11/2022 6 bedroom semi-detached house for sale False False 53.428500 -2.194710 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-11T11:15:05Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 828 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 2534 Bridgfords, Withington Withington Bridgfords /estate-agents/agent/Bridgfords/Withington-828... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
5 125170034 4 2 31 3 1 A beautiful family home with this TASTEFULLY M... Grasmere Crescent, High Lane, Stockport, SK6 GB Detached True False None buy False False True False False False None True /properties/125170034#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-07-20T12:02:21Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, High ... Reduced on 24/10/2022 4 bedroom detached house for sale False False 53.370442 -2.073502 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-24T10:12:21Z 499000 not specified GBP [{'displayPrice': '£499,000', 'displayPriceQua... 12272 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 7425 Ian Tonge Property Services Limited, High Lane High Lane Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
6 127657994 4 3 13 1 0 Looking for a family home with bags of space a... Harrisons Drive, Woodley, Stockport GB Detached False False None buy False False True False False False None True /properties/127657994#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-03T13:21:25Z [] no_keyword False False False True None by On the Move Property Boutique, Hyde Added on 03/10/2022 4 bedroom detached house for sale False False 53.428014 -2.086527 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-10-03T13:27:02Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 123220 /124k/123220/branch_rmchoice_logo_123220_0005.png 0161 524 8664 On the Move Property Boutique, Hyde Hyde On the Move Property Boutique /estate-agents/agent/On-the-Move-Property-Bout... False True False True False None False [] https://media.rightmove.co.uk:443/dir/124k/123... False []
7 128921357 4 1 16 1 1 An impressive, fully refurbished 4 bedroom det... Hermitage Gardens, Romiley, Stockport, Cheshir... GB Detached True False None buy False False True False False False None True /properties/128921357#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-09T16:10:16Z [] no_keyword False False False True None by Lawler & Co, Marple Added on 09/11/2022 4 bedroom detached house for sale False False 53.415631 -2.065066 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-09T16:16:02Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 108866 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 3922 Lawler & Co, Marple Marple Lawler & Co /estate-agents/agent/Lawler-and-Co/Marple-1088... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... High Specification False []
8 127524668 3 1 17 1 0 An immaculately presented and well maintained ... Alvaston Avenue, Heaton Moor GB Detached False False None buy False False True False False False None True /properties/127524668#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-28T18:00:38Z [] no_keyword False False False True None by John Mellor Independent Estate Agents, Hea... Reduced on 19/10/2022 3 bedroom detached house for sale False False 53.415292 -2.186009 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-19T15:45:04Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 4639 /5k/4639/branch_rmchoice_logo_4639_0000.jpeg 0161 524 4074 John Mellor Independent Estate Agents, Heaton ... Heaton Moor John Mellor Independent Estate Agents /estate-agents/agent/John-Mellor-Independent-E... False True False True False None False [] https://media.rightmove.co.uk:443/dir/5k/4639/... False []
9 127965329 3 1 19 1 0 A wonderful larger than average three double b... Carlton Road, Heaton Mersey, Stockport, SK4 GB Semi-Detached True False None buy False False True False False False None True /properties/127965329#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-12T16:16:41Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Reduced on 14/12/2022 3 bedroom semi-detached house for sale False False 53.415290 -2.194650 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-14T14:30:07Z 490000 not specified GBP [{'displayPrice': '£490,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False []
10 129519590 4 2 18 1 1 Occupying a delightful position at the head of... Gloucester Avenue, Marple GB Detached True False None buy False False True False False False None True /properties/129519590#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-28T10:55:20Z [] no_keyword False False True True None by Edward Mellor Ltd, Marple Added on 28/11/2022 4 bedroom detached house for sale False False 53.396840 -2.067260 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-28T11:01:02Z 490000 not specified GBP [{'displayPrice': '£490,000', 'displayPriceQua... 28598 /company/clogo_rmchoice_12942_0000.png 0161 524 6040 Edward Mellor Ltd, Marple Marple Edward Mellor Ltd /estate-agents/agent/Edward-Mellor-Ltd/Marple-... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Garage False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
11 129229553 5 2 2 1 0 COMING TO THE MARKET SOON - We are inviting yo... Kennerley Road, Davenport, Stockport, SK2 GB Semi-Detached False False None buy False False True False False False None True /properties/129229553#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-18T15:51:02Z [] no_keyword False False False True None by Julian Wadden, Stockport Added on 18/11/2022 5 bedroom semi-detached house for sale False False 53.392527 -2.150971 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-18T15:57:01Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 67998 /68k/67998/branch_rmchoice_logo_67998_0000.jpeg 0161 524 7110 Julian Wadden, Stockport Stockport Julian Wadden /estate-agents/agent/Julian-Wadden/Stockport-6... False True False True False None False [] https://media.rightmove.co.uk:443/dir/68k/6799... False []
12 128381072 4 1 17 1 1 A stunning family home located a short walk fr... Brooklands Road, Stockport, SK7 GB Semi-Detached True False None buy False False True False False False None True /properties/128381072#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-25T12:14:51Z [] no_keyword False False False True None by Purplebricks, covering Stockport Reduced on 02/12/2022 4 bedroom semi-detached house for sale False False 53.372003 -2.121311 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-02T11:08:12Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 75825 /brand/brand_rmchoice_logo_87698_0003.png 020 3834 7974 Purplebricks, covering Stockport covering Stockport Purplebricks /estate-agents/agent/Purplebricks/covering-Sto... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
13 129576503 5 2 23 3 1 NOT BEEN ON THE MARKET FOR 35 YEARS! THIS VERS... Bowfell Drive, Stockport, SK6 GB Detached True False None buy False False True False False False None True /properties/129576503#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-29T16:32:21Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, High ... Added on 29/11/2022 5 bedroom detached house for sale False False 53.369764 -2.071565 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-29T16:38:04Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 12272 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 7425 Ian Tonge Property Services Limited, High Lane High Lane Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Off-Street Parking False []
14 127281005 4 2 20 1 1 An impressive, THREE/FOUR BEDROOM, DETACHED FA... Tennyson Close, Heaton Mersey, Stockport, SK4 GB Detached True False None buy False False True False False False None True /properties/127281005#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-21T16:06:17Z [] no_keyword False False True True None by Philip James Kennedy, Heaton Moor Reduced on 15/11/2022 4 bedroom detached house for sale False False 53.411088 -2.188637 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-15T12:58:24Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 7571 /8k/7571/branch_rmchoice_logo_7571_0008.jpeg 0161 524 3546 Philip James Kennedy, Heaton Moor Heaton Moor Philip James Kennedy /estate-agents/agent/Philip-James-Kennedy/Heat... False True False True False None False [] https://media.rightmove.co.uk:443/dir/8k/7571/... Premium Listing False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
15 127483358 3 2 37 1 0 Fabulous Victorian c1901 semi-detached of spac... Davenport Road, Hazel Grove, Stockport SK7 4HA GB Semi-Detached False False None buy False False True False False False None True /properties/127483358#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-27T19:59:44Z [] no_keyword False False False True None by Woodhall Properties, Hazel Grove Reduced on 31/10/2022 3 bedroom semi-detached house for sale False False 53.379861 -2.125069 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-31T13:53:57Z 479950 not specified GBP [{'displayPrice': '£479,950', 'displayPriceQua... 922 /company/clogo_189_0000.jpg 0161 483 5100 Woodhall Properties, Hazel Grove Hazel Grove Woodhall Properties /estate-agents/agent/Woodhall-Properties/Hazel... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False []
16 124938890 5 2 26 1 1 ** OFFERS INVITED ** Rare opportunity to purch... Bower Avenue, Heaton Norris, Stockport, SK4 GB Semi-Detached True False None buy False False True False False False None True /properties/124938890#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-07-13T17:59:48Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Reduced on 01/11/2022 5 bedroom semi-detached house for sale False False 53.415697 -2.175796 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-01T14:41:21Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False []
17 122332742 4 5 16 1 1 Extended and very much improved detached with ... Craig Road, Heaton Mersey, Stockport, SK4 GB Detached True False None buy False False True False False False None True /properties/122332742#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-04-12T22:56:29Z [] no_keyword False False False True None by Purplebricks, covering Stockport Added on 12/04/2022 4 bedroom detached house for sale False False 53.408552 -2.195049 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-04-12T23:02:01Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 75825 /brand/brand_rmchoice_logo_87698_0003.png 020 3834 7974 Purplebricks, covering Stockport covering Stockport Purplebricks /estate-agents/agent/Purplebricks/covering-Sto... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
18 127330814 4 2 12 1 2 An impressive FOUR DOUBLE BEDROOM detached wit... Kinross Avenue, Woodsmoor, Stockport, Cheshire... GB Detached False False None buy False False True False False False None True /properties/127330814#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-22T18:11:23Z [] no_keyword False False True True None by Lawler & Co, Hazel Grove Reduced on 14/10/2022 4 bedroom detached house for sale False False 53.380745 -2.144872 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-14T11:01:12Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 120697 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 8681 Lawler & Co, Hazel Grove Hazel Grove Lawler & Co /estate-agents/agent/Lawler-and-Co/Hazel-Grove... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
19 127937918 2 1 28 1 1 Set in a prime central village location this h... Longhurst Lane, Marple Bridge, SK6 GB Detached True False None buy False False True False False False None True /properties/127937918#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-12T09:29:39Z [] no_keyword False False True True None by Julian Wadden, Marple Added on 12/10/2022 2 bedroom detached house for sale False False 53.399128 -2.052152 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-10-12T09:35:03Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 123928 /124k/123928/branch_rmchoice_logo_123928_0002.... 0161 524 3443 Julian Wadden, Marple Marple Julian Wadden /estate-agents/agent/Julian-Wadden/Marple-1239... False True False True False None False [] https://media.rightmove.co.uk:443/dir/124k/123... Village location False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
20 129435305 4 1 14 1 0 Bridgfords are over the moon to welcome to the... Shepley Close, Hazel Grove, Stockport, SK7 GB Detached True False None buy False False True False False False None True /properties/129435305#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-24T14:33:00Z [] no_keyword False False False True None by Bridgfords, Hazel Grove Added on 24/11/2022 4 bedroom detached house for sale False False 53.370440 -2.122900 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-24T14:38:03Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 93 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 4990 Bridgfords, Hazel Grove Hazel Grove Bridgfords /estate-agents/agent/Bridgfords/Hazel-Grove-93... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... No Chain False []
21 129803789 3 1 16 1 0 *****NO CHAIN*****A superb opportunity to purc... Marsham Road, SK7 GB Detached True False None buy False False True False False False None True /properties/129803789#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-12-07T15:13:51Z [] no_keyword False False False True None by Leighton Snow, Bramhall Added on 07/12/2022 3 bedroom detached house for sale False False 53.373750 -2.136140 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-12-07T15:19:05Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 195578 /company/clogo_rmchoice_59957_0000.jpeg 0161 524 6749 Leighton Snow, Bramhall Bramhall Leighton Snow /estate-agents/agent/Leighton-Snow/Bramhall-19... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Potential to extend (STPP) False []
22 128790602 4 1 25 4 1 A superb opportunity to acquire this unqiue fa... Dialstone Lane, Stockport, SK2 GB Detached True False None buy False False True False False False None True /properties/128790602#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-06T11:15:03Z [] no_keyword False False False True None by Strike, Nationwide Reduced on 06/12/2022 4 bedroom detached house for sale False False 53.393875 -2.131655 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-06T17:00:07Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 29376 /company/clogo_rmchoice_7671_0010.png 020 3835 4350 Strike, Nationwide Nationwide Strike /estate-agents/agent/Strike/Nationwide-29376.html False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Substantial Plot False []
23 127181315 3 2 11 1 0 A brand-new development of 4 modern semi-detac... Broadstone Road, Heaton Chapel, Stockport, SK4 GB Semi-Detached True False None buy False True True False False False None True /properties/127181315#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-17T13:18:30Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Added on 17/09/2022 3 bedroom semi-detached house for sale False False 53.429003 -2.174603 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-09-17T13:24:01Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False [{'type': 'NEW_HOME', 'priority': 3}]
24 126025148 4 1 12 1 1 A rare opportunity to acquire an Edwardian Sty... Mile End Lane, Stockport, Cheshire, SK2 GB Semi-Detached False False None buy False False True False False False None True /properties/126025148#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-08-12T17:05:36Z [] no_keyword False False True True None by Lawler & Co, Hazel Grove Reduced on 08/11/2022 4 bedroom semi-detached house for sale False False 53.395507 -2.134795 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-08T16:31:25Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 120697 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 8681 Lawler & Co, Hazel Grove Hazel Grove Lawler & Co /estate-agents/agent/Lawler-and-Co/Hazel-Grove... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
You can slice & dice that json object further, to get only the info you're after. Relevant pandas documentation can be found here.
Related
Working with a multiindex dataframe, to get summation results over a boolean column, based on a condition from another column
We have a multiindex dataframe that looks like: date condition_1 condition_2 item1 0 2021-06-10 06:30:00+00:00 True False 1 2021-06-10 07:00:00+00:00 False True 2 2021-06-10 07:30:00+00:00 True True item2 3 2021-06-10 06:30:00+00:00 True False 4 2021-06-10 07:00:00+00:00 True True 5 2021-06-10 07:30:00+00:00 True True item3 6 2021-06-10 06:30:00+00:00 True True 7 2021-06-10 07:00:00+00:00 False True 8 2021-06-10 07:30:00+00:00 True True The value of date repeats between items (because the df is a result of a default concat on a dictionary of dataframes). The logic we basically want to vectorize is "for every date where condition_1 is true for all items: sum the occurrences where condition_2 is true in a new results column for all of them". The result would basically look like this based on the above example (comments on how it's derived: next to the results column): date condition_1 condition_2 result item1 0 2021-06-10 06:30:00+00:00 True False 1 [because condition_1 is True for all items and condition_2 is True once] 1 2021-06-10 07:00:00+00:00 False True 0 [condition_1 is not True for all items so condition_2 is irrelevant] 2 2021-06-10 07:30:00+00:00 True True 3 [both conditions are True for all 3 items] item2 3 2021-06-10 06:30:00+00:00 True False 1 [a repeat for the same reasons] 4 2021-06-10 07:00:00+00:00 True True 0 [a repeat for the same reasons] 5 2021-06-10 07:30:00+00:00 True True 3 [a repeat for the same reasons] item3 6 2021-06-10 06:30:00+00:00 True True 1 [a repeat for the same reasons] 7 2021-06-10 07:00:00+00:00 False True 0 [a repeat for the same reasons] 8 2021-06-10 07:30:00+00:00 True True 3 [a repeat for the same reasons]
Here is what I came up with. def cond_sum(s): return s.cond1.all() * s.cond2.sum() df.reset_index(level=0, inplace=True) df['result'] = df.groupby('date').apply(cond_sum) df.set_index('item', append=True) Then if you want the original index, you can add it back. df.set_index('item', append=True).swaplevel() Note, you mentioned vectorized, so you could swap that out for: dfg = df.groupby(level=0).agg({'cond1': 'all', 'cond2': 'sum'}) df['result'] = dfg.cond1 * dfg.cond2
Pandas True False Matching
For this table: I would like to generate the 'desired_output' column. One way to achieve this maybe: All the True values from col_1 are transferred straight across to desired_output (red arrow) In desired_output, place a True value above any existing True value (green arrow) Code I have tried: df['desired_output']=df.col_1.apply(lambda x: True if x.shift()==True else False) Thankyou
You can chain by | for bitwise OR original with shifted values by Series.shift: d = {"col1":[False,True,True,True,False,True,False,False,True,False,False,False]} df = pd.DataFrame(d) df['new'] = df.col1 | df.col1.shift(-1) print (df) col1 new 0 False True 1 True True 2 True True 3 True True 4 False True 5 True True 6 False False 7 False True 8 True True 9 False False 10 False False 11 False False
try this df['desired_output'] = df['col_1'] df.loc[1:, 'desired_output'] = df.col_1[1:].values | df.col_1[:-1].values print(df)
In case those are saved as string. all_caps (TRUE / FALSE) Input: col_1 0 True 1 True 2 False 3 True 4 True 5 False 6 Flase 7 True 8 False Code: df['desired']=df['col_1'] for i, e in enumerate(df['col_1']): if e=='True': df.at[i-1,'desired']=df.at[i,'col_1'] df = df[:(len(df)-1)] df Output: col_1 desired 0 True True 1 True True 2 False True 3 True True 4 True True 5 False False 6 Flase True 7 True True 8 False False
How to vectorize in Pandas when values depend on prior values
I'd like to use Pandas to implement a function that keeps a running balance, but I'm not sure it can be vectorized for speed. In short, the problem I'm trying to solve is to keep track consumption, generation, and the "bank" of over-generation. "consumption" means how much is used in a given time period. "generation" is how much is generated. When generation is greater than consumption then the homeowner can "bank" the extra generation, to be applied in subsequent time periods. they can apply it if their consumption exceeds their generation for a later month. This will be for many entities, hence the "id" field. The time sequence is defined by "order" Very basic example: Month 1 generates 13 consumes 8 -> therefore banks 5 month 2 generates 8 consumes 10 -> therefore uses 2 from the the bank, and still has 3 left over Month 3 generates 7 consumes 20 -> exhausts remaining 3 from bank, and has no bank left over. Code import numpy as np import pandas as pd id = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2] order = [1,2,3,4,5,6,7,8,9,18,11,12,13,14,15,1,2,3,4,5,6,7,8,9,10,11] consume = [10, 17, 20, 11, 17, 19, 20, 10, 10, 19, 14, 12, 10, 14, 13, 19, 12, 17, 12, 18, 15, 14, 15, 20, 16, 15] generate = [20, 16, 17, 21, 9, 13, 10, 16, 12, 10, 9, 9, 15, 13, 100, 15, 18, 16, 10, 16, 12, 12, 13, 20, 10, 15] df = pd.DataFrame(list(zip(id, order, consume, generate)), columns =['id','Order','Consume', 'Generate']) begin_bal = [0,10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0] end_bal = [10,9,6,16,8,2,0,6,8,0,0,0,5,4,91,0,6,5,3,1,0,0,0,0,0,0] withdraw = [0,1,3,0,8,6,2,0,0,8,0,0,0,1,4,0,0,1,2,2,1,0,0,0,0,0] df_solution = pd.DataFrame(list(zip(id, order, consume, generate, begin_bal, end_bal, withdraw)), columns =['id','Order','Consume', 'Generate', 'begin_bal', 'end_bal', 'Withdraw']) def bank(df): # deposit all excess when generation exceeds consumption deposit = (df['Generate'] > df['Consume']) * (df['Generate'] - df['Consume']) df['end_bal'] = 0 # beginning balance = prior period ending balance df = df.sort_values(by=['id', 'Order']) df['begin_bal'] = df['end_bal'].shift(periods=1) df.loc[df['Order']==1, 'begin_bal'] = 0 # set first month beginning balance of each customer to 0 # calculate withdrawal df['Withdraw'] = 0 ok_to_withdraw = df['Consume'] > df['Generate'] df.loc[ok_to_withdraw,'Withdraw'] = np.minimum(df.loc[ok_to_withdraw, 'begin_bal'], df.loc[ok_to_withdraw, 'Consume'] - df.loc[ok_to_withdraw, 'Generate'] - deposit[ok_to_withdraw]) # ending balance = beginning balance + deposit - withdraw df['end_bal'] = df['begin_bal'] + deposit - df['Withdraw'] return df df = bank(df) df.head() id Order Consume Generate end_bal begin_bal Withdraw 0 1 1 10 20 10.0 0.0 0.0 1 1 2 17 16 0.0 0.0 0.0 2 1 3 20 17 0.0 0.0 0.0 3 1 4 11 21 10.0 0.0 0.0 4 1 5 17 9 0.0 0.0 0.0 df_solution.head() id Order Consume Generate begin_bal end_bal Withdraw 0 1 1 10 20 0 10 0 1 1 2 17 16 10 9 1 2 1 3 20 17 9 6 3 3 1 4 11 21 6 16 0 4 1 5 17 9 16 8 9 I tried to implement with various iterations of cumsum and shift . . . but the fact remains that value of each row seems like it needs to be recalculated based on the prior row, and I'm not sure this is possible to vectorize. Code to generate some test datasets: def generate_testdata(): random.seed(42*42) np.random.seed(42*42) numids = 10 numorders = 12 id = [] order = [] for i in range(numids): id = id + [i]*numorders order = order + list(range(1,numorders+1)) consume = np.random.uniform(low = 10, high = 40, size = numids*numorders) generate = np.random.uniform(low = 10, high = 40, size = numids*numorders) df = pd.DataFrame(list(zip(id, order, consume, generate)), columns =['id','Order','Consume', 'Generate']) return df
Here is a numpy-ish approach, mostly because I'm not that familiar with pandas: The idea is to first compute the free cumsum and then to subtract the cumulative minimum if it is negative. import numpy as np import pandas as pd id = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2] order = [1,2,3,4,5,6,7,8,9,18,11,12,13,14,15,1,2,3,4,5,6,7,8,9,10,11] consume = [10, 17, 20, 11, 17, 19, 20, 10, 10, 19, 14, 12, 10, 14, 13, 19, 12, 17, 12, 18, 15, 14, 15, 20, 16, 15] generate = [20, 16, 17, 21, 9, 13, 10, 16, 12, 10, 9, 9, 15, 13, 8, 15, 18, 16, 10, 16, 12, 12, 13, 20, 10, 15] df = pd.DataFrame(list(zip(id, order, consume, generate)), columns =['id','Order','Consume', 'Generate']) begin_bal = [0,10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0] end_bal = [10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0,0] withdraw = [0,1,3,0,9,6,2,0,0,8,0,0,0,1,4,0,0,1,2,2,1,0,0,0,0,0] df_solution = pd.DataFrame(list(zip(id, order, consume, generate, begin_bal, end_bal, withdraw)), columns =['id','Order','Consume', 'Generate', 'begin_bal', 'end_bal', 'Withdraw']) def f(df): # find block bondaries ids = df["id"].values bnds, = np.where(np.diff(ids, prepend=ids[0]-1, append=ids[-1]+1)) # find raw balance change delta = (df["Generate"] - df["Consume"]).values # find offset, so cumulative min does not interfere across ids safe_total = (np.minimum(delta.min(), 0)-1) * np.diff(bnds[:-1]) # must apply offset just before group switch, so it aligns the first # begin_bal, not end_bal, of the next group # also keep a copy of original values at switches delta_orig = delta[bnds[1:-1]-1] delta[bnds[1:-1]-1] += safe_total - np.add.reduceat(delta, bnds[:-2]) # form free cumsum acc = delta.cumsum() # correct acc -= np.minimum(0, np.minimum.accumulate(acc)) # write solution back to df shft = np.empty_like(acc) shft[1:] = acc[:-1] shft[0] = 0 # reinstate last end_bal of each group acc[bnds[1:-1]-1] = np.maximum(0, shft[bnds[1:-1]-1] + delta_orig) df["begin_bal"] = shft df["end_bal"] = acc df["Withdraw"] = np.maximum(0, df["begin_bal"] - df["end_bal"]) Test: f(df) df == df_solution Prints: id Order Consume Generate begin_bal end_bal Withdraw 0 True True True True True True True 1 True True True True True True True 2 True True True True True True True 3 True True True True True True True 4 True True True True True True False 5 True True True True True True True 6 True True True True True True True 7 True True True True True True True 8 True True True True True True True 9 True True True True True True True 10 True True True True True True True 11 True True True True True True True 12 True True True True True True True 13 True True True True True True True 14 True True True True True True True 15 True True True True True True True 16 True True True True True True True 17 True True True True True True True 18 True True True True True True True 19 True True True True True True True 20 True True True True True True True 21 True True True True True True True 22 True True True True True True True 23 True True True True True True True 24 True True True True True True True 25 True True True True True True True There is one False but that appears to be a typo in the expected output provided.
Using #PaulPanzer's logic here is a pandas version. def CalcEB(x): delta = x['Generate'] - x['Consume'] return delta.cumsum() - delta.cumsum().cummin().clip(-np.inf,0) df['end_bal'] = df.groupby('id', as_index=False).apply(CalcEB).values df['begin_bal'] = df.groupby('id')['end_bal'].shift().fillna(0) df['Withdraw'] = (df['begin_bal'] - df['end_bal']).clip(0,np.inf) df_pandas = df.copy() #Note the typo mentioned by Paul Panzer df_pandas.reindex(df_solution.columns, axis=1) == df_solution Output (check dataframes) id Order Consume Generate begin_bal end_bal Withdraw 0 True True True True True True True 1 True True True True True True True 2 True True True True True True True 3 True True True True True True True 4 True True True True True True False 5 True True True True True True True 6 True True True True True True True 7 True True True True True True True 8 True True True True True True True 9 True True True True True True True 10 True True True True True True True 11 True True True True True True True 12 True True True True True True True 13 True True True True True True True 14 True True True True True True True 15 True True True True True True True 16 True True True True True True True 17 True True True True True True True 18 True True True True True True True 19 True True True True True True True 20 True True True True True True True 21 True True True True True True True 22 True True True True True True True 23 True True True True True True True 24 True True True True True True True 25 True True True True True True True
I am not sure I understood your question fully, but I am going to give a go at answering. I will re-phrase what I understood... 1. Source data There is source data, which is a DataFrame with four columns: id - ID number of an entity order - indicates the sequence of periods consume - how much was consumed during the period generate - how much was generated during the period 2. Calculations For each id, we want to calculate: diff which is the difference between generate and consume for each period opening balance which is the closing balance from the previous order closing balance which is the cumulative sum of the diff 3. Code I will try to solve this with groupby, cumsum and shift. # Make sure the df is sorted df = df.sort_values(['id','order']) df['diff'] = df['generate'] - df['consume'] df['closing_balance'] = df.groupby('id')['diff'].cumsum() # Opening balance equals the closing balance from the previous period df['opening_balance'] = df.groupby('id')['closing_balance'].shift(1) I definitely misunderstood something, feel free to correct me and I will try to come up with a better answer. In particular, I wasn't sure how to handle the closing_balance going into negative numbers. Should it show negative balance? Should it nullify the "debts"?
How to count trues/falses from a list?
df = [bigdataframe[['Action', 'Adventure','Animation', 'Childrens', 'Comedy', 'Crime','Documentary', 'Drama', 'Fantasy', 'FilmNoir', 'Horror', 'Musical', 'Mystery', 'Romance','SciFi', 'Thriller', 'War', 'Western']].sum(axis=1) > 1] df Out[8]: [0 True 1 True 2 True 3 True 4 True 5 False 6 True 7 True 8 False 9 True 10 False 11 True 12 True 13 True 14 True 15 False 16 True 17 False 18 True 19 False 20 False 21 True 22 True 23 True 24 False 25 True 26 True 27 True 28 True 29 True 99970 True 99971 True 99972 False 99973 True 99974 True 99975 True 99976 True 99977 True 99978 False 99979 False 99980 True 99981 False 99982 True 99983 False 99984 True 99985 True 99986 True 99987 True 99988 False 99989 True 99990 True 99991 True 99992 False 99993 True 99994 True 99995 True 99996 True 99997 True 99998 True 99999 False Length: 100000, dtype: bool] I have tried: len(df[df==True]) Masking They are in a list so shouldn't I just be able to count them? Or do I need to assign them numerical values, 1 for true and 0 for false and then use the count or sum function to find how many are true?
Demo: In [386]: df = pd.DataFrame(np.random.rand(5,3), columns=list('ABC')) In [387]: df Out[387]: A B C 0 0.228687 0.647431 0.526471 1 0.795122 0.915011 0.950481 2 0.386244 0.705412 0.420596 3 0.343213 0.928993 0.192527 4 0.201023 0.209281 0.304799 In [388]: df[['A','B','C']].sum(axis=1).gt(1.5) Out[388]: 0 False 1 True 2 True 3 False 4 False dtype: bool In [389]: df[['A','B','C']].sum(axis=1).gt(1.5).sum() Out[389]: 2
to count number of true in a list sum(unlist(your.list.object))
Groupby pandas calculate percentage
I have a groupby object as follows after i ran: grouped_mask=L2014_2.groupby(['state']) grouped_mask.mask.value_counts() state mask AL False 105931 True 77 AR False 67788 True 1774 AZ False 90068 True 151 CA False 586184 True 4 CO False 75188 True 14360 CT False 78270 True 1 Now i need to calculate what is percentage of true in each state. Is there a way to do this?
Also you can set the normalize parameter to obtain the relative frequencies: grouped_mask.mask.value_counts(normalize=True) just multiply by 100 to get the percentages :-) regards
You can groupby on the first level and then apply a lambda that divides the True/False counts against the sum: In [20]: df.groupby(level=0).apply(lambda x: x/x.sum() * 100) Out[20]: Count state mask AL False 99.927364 True 0.072636 AR False 97.449757 True 2.550243 AZ False 99.832629 True 0.167371 CA False 99.999318 True 0.000682 CO False 83.963908 True 16.036092 CT False 99.998722 True 0.001278 To filter the above to get just the False labels you can use advanced indexing using slices: In [33]: gp = df.groupby(level=0).apply(lambda x: x/x.sum() * 100) gp.loc(axis=0)[slice(None),False] Out[33]: Count state mask AL False 99.927364 AR False 97.449757 AZ False 99.832629 CA False 99.999318 CO False 83.963908 CT False 99.998722
gp=grouped_mask.mask.value_counts().groupby(level=0).apply(lambda x: 100*x/float(x.sum())) state mask AL False 94.37 True 5.63 AR False 73.85 True 26.15 AZ False 91.88 True 8.12 CA False 99.57 True 0.43 CO False 64.66 True 35.34 gp.sort_index(level=0) gp.loc(axis=0)[slice(None),False] KeyError: 'MultiIndex Slicing requires the index to be fully lexsorted tuple len (2), lexsort depth (1)'