if the class is not found append a int/Nan in bs4 - selenium

I've been trying to figure out how to append an integer or a Nan when the class isn't found, but it seems not working, and instead is skipping the if/else statement.
any idea?
rooms_bed = []
for a in soup.find_all("span", attrs={"class":"no-svg-bed-icon bed-icon seperator"}):
for s in a.select('title'):
if len(s) == 0:
rooms_bed.append(0)
else:
rooms_bed.append(s.get_text())
number_bedrooms = []
for x in rooms_bed:
if "bedrooms" in x:
number_bedrooms.append(int(x.replace('bedrooms', '')))

I am somehow fairly convinced this is yet another XY Problem.
Therefore, here is one tested method of obtaining all data from that page:
import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
import re
import json
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
}
url = 'https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E1268&index=120&propertyTypes=&includeSSTC=false&mustHave=&dontShow=&furnishTypes=&keywords='
soup = bs(requests.get(url, headers=headers).text, 'html.parser')
data = soup.find(string=re.compile("^window.jsonModel")).split(' = ')[1]
df = pd.json_normalize(json.loads(data)['properties'])
print(df)
Result in terminal:
id bedrooms bathrooms numberOfImages numberOfFloorplans numberOfVirtualTours summary displayAddress countryCode propertySubType premiumListing featuredProperty distance transactionType commercial development residential students auction feesApply feesApplyText displaySize showOnMap propertyUrl contactUrl staticMapUrl channel firstVisibleDate keywords keywordMatchType saved hidden onlineViewingsAvailable hasBrandPlus displayStatus enquiredTimestamp heading formattedBranchName addedOrReduced propertyTypeFullDescription enhancedListing isRecent formattedDistance location.latitude location.longitude propertyImages.images propertyImages.mainImageSrc propertyImages.mainMapImageSrc listingUpdate.listingUpdateReason listingUpdate.listingUpdateDate price.amount price.frequency price.currencyCode price.displayPrices customer.branchId customer.brandPlusLogoURI customer.contactTelephone customer.branchDisplayName customer.branchName customer.brandTradingName customer.branchLandingPageUrl customer.development customer.showReducedProperties customer.commercial customer.showOnMap customer.enhancedListing customer.developmentContent customer.buildToRent customer.buildToRentBenefits customer.brandPlusLogoUrl productLabel.productLabelText productLabel.spotlightLabel lozengeModel.matchingLozenges
0 128841404 3 1 19 1 0 Bridgfords Hazel Grove are over the moon to we... Deneside Crescent, Hazel Grove, Stockport, Che... GB Semi-Detached True True None buy False False True False False False None True /properties/128841404#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-07T17:22:14Z [] no_keyword False False False True None Featured Property by Bridgfords, Hazel Grove Reduced on 23/11/2022 3 bedroom semi-detached house for sale False False 53.382230 -2.110010 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-23T22:14:22Z 280000 not specified GBP [{'displayPrice': '£280,000', 'displayPriceQua... 93 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 4990 Bridgfords, Hazel Grove Hazel Grove Bridgfords /estate-agents/agent/Bridgfords/Hazel-Grove-93... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Corner Plot False []
1 114569264 3 2 14 1 1 A well presented DETACHED BUNGALOW enjoying a ... Parsonage Gardens, Marple GB Bungalow True False None buy False False True False False False None True /properties/114569264#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2021-10-08T16:59:54Z [] no_keyword False False True True None by Edward Mellor Ltd, Marple Added on 08/07/2022 3 bedroom bungalow for sale False False 53.388240 -2.060480 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-07-08T15:07:43Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 28598 /company/clogo_rmchoice_12942_0000.png 0161 524 6040 Edward Mellor Ltd, Marple Marple Edward Mellor Ltd /estate-agents/agent/Edward-Mellor-Ltd/Marple-... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Cul-De-Sac False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
2 128949689 3 1 15 0 1 A rare opportunity to acquire this three bedro... Delamere Close, Stockport, SK7 GB Detached Bungalow True False None buy False False True False False False None True /properties/128949689#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-10T12:42:26Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, Hazel... Added on 10/11/2022 3 bedroom detached bungalow for sale False False 53.381600 -2.102170 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-10T12:48:02Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 12271 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 9316 Ian Tonge Property Services Limited, Hazel Grove Hazel Grove Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
3 127513385 4 2 17 1 1 Julian Wadden are delighted to be selling this... Boddens Hill Road, Heaton Mersey, Stockport, SK4 GB Detached False False None buy False False True False False False None True /properties/127513385#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-28T15:41:42Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Added on 28/09/2022 4 bedroom detached house for sale False False 53.411211 -2.194055 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-09-28T15:47:04Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... False []
4 128983526 6 2 15 1 1 A well presented large family home in a popula... Shawbrook Road, Manchester, Greater Manchester... GB Semi-Detached True False None buy False False True False False False None True /properties/128983526#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-11T11:10:01Z [] no_keyword False False False True None by Bridgfords, Withington Added on 11/11/2022 6 bedroom semi-detached house for sale False False 53.428500 -2.194710 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-11T11:15:05Z 500000 not specified GBP [{'displayPrice': '£500,000', 'displayPriceQua... 828 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 2534 Bridgfords, Withington Withington Bridgfords /estate-agents/agent/Bridgfords/Withington-828... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
5 125170034 4 2 31 3 1 A beautiful family home with this TASTEFULLY M... Grasmere Crescent, High Lane, Stockport, SK6 GB Detached True False None buy False False True False False False None True /properties/125170034#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-07-20T12:02:21Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, High ... Reduced on 24/10/2022 4 bedroom detached house for sale False False 53.370442 -2.073502 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-24T10:12:21Z 499000 not specified GBP [{'displayPrice': '£499,000', 'displayPriceQua... 12272 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 7425 Ian Tonge Property Services Limited, High Lane High Lane Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
6 127657994 4 3 13 1 0 Looking for a family home with bags of space a... Harrisons Drive, Woodley, Stockport GB Detached False False None buy False False True False False False None True /properties/127657994#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-03T13:21:25Z [] no_keyword False False False True None by On the Move Property Boutique, Hyde Added on 03/10/2022 4 bedroom detached house for sale False False 53.428014 -2.086527 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-10-03T13:27:02Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 123220 /124k/123220/branch_rmchoice_logo_123220_0005.png 0161 524 8664 On the Move Property Boutique, Hyde Hyde On the Move Property Boutique /estate-agents/agent/On-the-Move-Property-Bout... False True False True False None False [] https://media.rightmove.co.uk:443/dir/124k/123... False []
7 128921357 4 1 16 1 1 An impressive, fully refurbished 4 bedroom det... Hermitage Gardens, Romiley, Stockport, Cheshir... GB Detached True False None buy False False True False False False None True /properties/128921357#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-09T16:10:16Z [] no_keyword False False False True None by Lawler & Co, Marple Added on 09/11/2022 4 bedroom detached house for sale False False 53.415631 -2.065066 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-09T16:16:02Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 108866 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 3922 Lawler & Co, Marple Marple Lawler & Co /estate-agents/agent/Lawler-and-Co/Marple-1088... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... High Specification False []
8 127524668 3 1 17 1 0 An immaculately presented and well maintained ... Alvaston Avenue, Heaton Moor GB Detached False False None buy False False True False False False None True /properties/127524668#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-28T18:00:38Z [] no_keyword False False False True None by John Mellor Independent Estate Agents, Hea... Reduced on 19/10/2022 3 bedroom detached house for sale False False 53.415292 -2.186009 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-19T15:45:04Z 495000 not specified GBP [{'displayPrice': '£495,000', 'displayPriceQua... 4639 /5k/4639/branch_rmchoice_logo_4639_0000.jpeg 0161 524 4074 John Mellor Independent Estate Agents, Heaton ... Heaton Moor John Mellor Independent Estate Agents /estate-agents/agent/John-Mellor-Independent-E... False True False True False None False [] https://media.rightmove.co.uk:443/dir/5k/4639/... False []
9 127965329 3 1 19 1 0 A wonderful larger than average three double b... Carlton Road, Heaton Mersey, Stockport, SK4 GB Semi-Detached True False None buy False False True False False False None True /properties/127965329#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-12T16:16:41Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Reduced on 14/12/2022 3 bedroom semi-detached house for sale False False 53.415290 -2.194650 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-14T14:30:07Z 490000 not specified GBP [{'displayPrice': '£490,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False []
10 129519590 4 2 18 1 1 Occupying a delightful position at the head of... Gloucester Avenue, Marple GB Detached True False None buy False False True False False False None True /properties/129519590#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-28T10:55:20Z [] no_keyword False False True True None by Edward Mellor Ltd, Marple Added on 28/11/2022 4 bedroom detached house for sale False False 53.396840 -2.067260 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-28T11:01:02Z 490000 not specified GBP [{'displayPrice': '£490,000', 'displayPriceQua... 28598 /company/clogo_rmchoice_12942_0000.png 0161 524 6040 Edward Mellor Ltd, Marple Marple Edward Mellor Ltd /estate-agents/agent/Edward-Mellor-Ltd/Marple-... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Garage False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
11 129229553 5 2 2 1 0 COMING TO THE MARKET SOON - We are inviting yo... Kennerley Road, Davenport, Stockport, SK2 GB Semi-Detached False False None buy False False True False False False None True /properties/129229553#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-18T15:51:02Z [] no_keyword False False False True None by Julian Wadden, Stockport Added on 18/11/2022 5 bedroom semi-detached house for sale False False 53.392527 -2.150971 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-18T15:57:01Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 67998 /68k/67998/branch_rmchoice_logo_67998_0000.jpeg 0161 524 7110 Julian Wadden, Stockport Stockport Julian Wadden /estate-agents/agent/Julian-Wadden/Stockport-6... False True False True False None False [] https://media.rightmove.co.uk:443/dir/68k/6799... False []
12 128381072 4 1 17 1 1 A stunning family home located a short walk fr... Brooklands Road, Stockport, SK7 GB Semi-Detached True False None buy False False True False False False None True /properties/128381072#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-25T12:14:51Z [] no_keyword False False False True None by Purplebricks, covering Stockport Reduced on 02/12/2022 4 bedroom semi-detached house for sale False False 53.372003 -2.121311 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-02T11:08:12Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 75825 /brand/brand_rmchoice_logo_87698_0003.png 020 3834 7974 Purplebricks, covering Stockport covering Stockport Purplebricks /estate-agents/agent/Purplebricks/covering-Sto... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
13 129576503 5 2 23 3 1 NOT BEEN ON THE MARKET FOR 35 YEARS! THIS VERS... Bowfell Drive, Stockport, SK6 GB Detached True False None buy False False True False False False None True /properties/129576503#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-29T16:32:21Z [] no_keyword False False False True None by Ian Tonge Property Services Limited, High ... Added on 29/11/2022 5 bedroom detached house for sale False False 53.369764 -2.071565 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-29T16:38:04Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 12272 /brand/brand_rmchoice_logo_5724_0002.jpeg 0161 524 7425 Ian Tonge Property Services Limited, High Lane High Lane Ian Tonge Property Services Limited /estate-agents/agent/Ian-Tonge-Property-Servic... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Off-Street Parking False []
14 127281005 4 2 20 1 1 An impressive, THREE/FOUR BEDROOM, DETACHED FA... Tennyson Close, Heaton Mersey, Stockport, SK4 GB Detached True False None buy False False True False False False None True /properties/127281005#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-21T16:06:17Z [] no_keyword False False True True None by Philip James Kennedy, Heaton Moor Reduced on 15/11/2022 4 bedroom detached house for sale False False 53.411088 -2.188637 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-15T12:58:24Z 485000 not specified GBP [{'displayPrice': '£485,000', 'displayPriceQua... 7571 /8k/7571/branch_rmchoice_logo_7571_0008.jpeg 0161 524 3546 Philip James Kennedy, Heaton Moor Heaton Moor Philip James Kennedy /estate-agents/agent/Philip-James-Kennedy/Heat... False True False True False None False [] https://media.rightmove.co.uk:443/dir/8k/7571/... Premium Listing False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
15 127483358 3 2 37 1 0 Fabulous Victorian c1901 semi-detached of spac... Davenport Road, Hazel Grove, Stockport SK7 4HA GB Semi-Detached False False None buy False False True False False False None True /properties/127483358#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-27T19:59:44Z [] no_keyword False False False True None by Woodhall Properties, Hazel Grove Reduced on 31/10/2022 3 bedroom semi-detached house for sale False False 53.379861 -2.125069 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-31T13:53:57Z 479950 not specified GBP [{'displayPrice': '£479,950', 'displayPriceQua... 922 /company/clogo_189_0000.jpg 0161 483 5100 Woodhall Properties, Hazel Grove Hazel Grove Woodhall Properties /estate-agents/agent/Woodhall-Properties/Hazel... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False []
16 124938890 5 2 26 1 1 ** OFFERS INVITED ** Rare opportunity to purch... Bower Avenue, Heaton Norris, Stockport, SK4 GB Semi-Detached True False None buy False False True False False False None True /properties/124938890#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-07-13T17:59:48Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Reduced on 01/11/2022 5 bedroom semi-detached house for sale False False 53.415697 -2.175796 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-01T14:41:21Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False []
17 122332742 4 5 16 1 1 Extended and very much improved detached with ... Craig Road, Heaton Mersey, Stockport, SK4 GB Detached True False None buy False False True False False False None True /properties/122332742#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-04-12T22:56:29Z [] no_keyword False False False True None by Purplebricks, covering Stockport Added on 12/04/2022 4 bedroom detached house for sale False False 53.408552 -2.195049 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-04-12T23:02:01Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 75825 /brand/brand_rmchoice_logo_87698_0003.png 020 3834 7974 Purplebricks, covering Stockport covering Stockport Purplebricks /estate-agents/agent/Purplebricks/covering-Sto... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... Premium Listing False []
18 127330814 4 2 12 1 2 An impressive FOUR DOUBLE BEDROOM detached wit... Kinross Avenue, Woodsmoor, Stockport, Cheshire... GB Detached False False None buy False False True False False False None True /properties/127330814#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-22T18:11:23Z [] no_keyword False False True True None by Lawler & Co, Hazel Grove Reduced on 14/10/2022 4 bedroom detached house for sale False False 53.380745 -2.144872 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-10-14T11:01:12Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 120697 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 8681 Lawler & Co, Hazel Grove Hazel Grove Lawler & Co /estate-agents/agent/Lawler-and-Co/Hazel-Grove... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
19 127937918 2 1 28 1 1 Set in a prime central village location this h... Longhurst Lane, Marple Bridge, SK6 GB Detached True False None buy False False True False False False None True /properties/127937918#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-10-12T09:29:39Z [] no_keyword False False True True None by Julian Wadden, Marple Added on 12/10/2022 2 bedroom detached house for sale False False 53.399128 -2.052152 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-10-12T09:35:03Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 123928 /124k/123928/branch_rmchoice_logo_123928_0002.... 0161 524 3443 Julian Wadden, Marple Marple Julian Wadden /estate-agents/agent/Julian-Wadden/Marple-1239... False True False True False None False [] https://media.rightmove.co.uk:443/dir/124k/123... Village location False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
20 129435305 4 1 14 1 0 Bridgfords are over the moon to welcome to the... Shepley Close, Hazel Grove, Stockport, SK7 GB Detached True False None buy False False True False False False None True /properties/129435305#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-24T14:33:00Z [] no_keyword False False False True None by Bridgfords, Hazel Grove Added on 24/11/2022 4 bedroom detached house for sale False False 53.370440 -2.122900 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-11-24T14:38:03Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 93 /brand/brand_rmchoice_logo_10_0001.jpeg 0161 524 4990 Bridgfords, Hazel Grove Hazel Grove Bridgfords /estate-agents/agent/Bridgfords/Hazel-Grove-93... False True False True False None False [] https://media.rightmove.co.uk:443/dir/brand/br... No Chain False []
21 129803789 3 1 16 1 0 *****NO CHAIN*****A superb opportunity to purc... Marsham Road, SK7 GB Detached True False None buy False False True False False False None True /properties/129803789#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-12-07T15:13:51Z [] no_keyword False False False True None by Leighton Snow, Bramhall Added on 07/12/2022 3 bedroom detached house for sale False False 53.373750 -2.136140 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-12-07T15:19:05Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 195578 /company/clogo_rmchoice_59957_0000.jpeg 0161 524 6749 Leighton Snow, Bramhall Bramhall Leighton Snow /estate-agents/agent/Leighton-Snow/Bramhall-19... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Potential to extend (STPP) False []
22 128790602 4 1 25 4 1 A superb opportunity to acquire this unqiue fa... Dialstone Lane, Stockport, SK2 GB Detached True False None buy False False True False False False None True /properties/128790602#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-11-06T11:15:03Z [] no_keyword False False False True None by Strike, Nationwide Reduced on 06/12/2022 4 bedroom detached house for sale False False 53.393875 -2.131655 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-12-06T17:00:07Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 29376 /company/clogo_rmchoice_7671_0010.png 020 3835 4350 Strike, Nationwide Nationwide Strike /estate-agents/agent/Strike/Nationwide-29376.html False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... Substantial Plot False []
23 127181315 3 2 11 1 0 A brand-new development of 4 modern semi-detac... Broadstone Road, Heaton Chapel, Stockport, SK4 GB Semi-Detached True False None buy False True True False False False None True /properties/127181315#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-09-17T13:18:30Z [] no_keyword False False False True None by Julian Wadden, Heaton Moor Added on 17/09/2022 3 bedroom semi-detached house for sale False False 53.429003 -2.174603 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... new 2022-09-17T13:24:01Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 81832 /82k/81832/branch_rmchoice_logo_81832_0004.jpeg 0161 524 6124 Julian Wadden, Heaton Moor Heaton Moor Julian Wadden /estate-agents/agent/Julian-Wadden/Heaton-Moor... False True False True False None False [] https://media.rightmove.co.uk:443/dir/82k/8183... Premium Listing False [{'type': 'NEW_HOME', 'priority': 3}]
24 126025148 4 1 12 1 1 A rare opportunity to acquire an Edwardian Sty... Mile End Lane, Stockport, Cheshire, SK2 GB Semi-Detached False False None buy False False True False False False None True /properties/126025148#/?channel=RES_BUY /property-for-sale/contactBranch.html?property... None BUY 2022-08-12T17:05:36Z [] no_keyword False False True True None by Lawler & Co, Hazel Grove Reduced on 08/11/2022 4 bedroom semi-detached house for sale False False 53.395507 -2.134795 [{'srcUrl': 'https://media.rightmove.co.uk:443... https://media.rightmove.co.uk:443/dir/crop/10:... https://media.rightmove.co.uk:443/dir/crop/10:... price_reduced 2022-11-08T16:31:25Z 475000 not specified GBP [{'displayPrice': '£475,000', 'displayPriceQua... 120697 /company/clogo_rmchoice_41135_0001.jpeg 0161 524 8681 Lawler & Co, Hazel Grove Hazel Grove Lawler & Co /estate-agents/agent/Lawler-and-Co/Hazel-Grove... False True False True False None False [] https://media.rightmove.co.uk:443/dir/company/... False [{'type': 'ONLINE_VIEWINGS', 'priority': 6}]
You can slice & dice that json object further, to get only the info you're after. Relevant pandas documentation can be found here.

Related

Working with a multiindex dataframe, to get summation results over a boolean column, based on a condition from another column

We have a multiindex dataframe that looks like:
date condition_1 condition_2
item1 0 2021-06-10 06:30:00+00:00 True False
1 2021-06-10 07:00:00+00:00 False True
2 2021-06-10 07:30:00+00:00 True True
item2 3 2021-06-10 06:30:00+00:00 True False
4 2021-06-10 07:00:00+00:00 True True
5 2021-06-10 07:30:00+00:00 True True
item3 6 2021-06-10 06:30:00+00:00 True True
7 2021-06-10 07:00:00+00:00 False True
8 2021-06-10 07:30:00+00:00 True True
The value of date repeats between items (because the df is a result of a default concat on a dictionary of dataframes).
The logic we basically want to vectorize is "for every date where condition_1 is true for all items: sum the occurrences where condition_2 is true in a new results column for all of them".
The result would basically look like this based on the above example (comments on how it's derived: next to the results column):
date condition_1 condition_2 result
item1 0 2021-06-10 06:30:00+00:00 True False 1 [because condition_1 is True for all items and condition_2 is True once]
1 2021-06-10 07:00:00+00:00 False True 0 [condition_1 is not True for all items so condition_2 is irrelevant]
2 2021-06-10 07:30:00+00:00 True True 3 [both conditions are True for all 3 items]
item2 3 2021-06-10 06:30:00+00:00 True False 1 [a repeat for the same reasons]
4 2021-06-10 07:00:00+00:00 True True 0 [a repeat for the same reasons]
5 2021-06-10 07:30:00+00:00 True True 3 [a repeat for the same reasons]
item3 6 2021-06-10 06:30:00+00:00 True True 1 [a repeat for the same reasons]
7 2021-06-10 07:00:00+00:00 False True 0 [a repeat for the same reasons]
8 2021-06-10 07:30:00+00:00 True True 3 [a repeat for the same reasons]
Here is what I came up with.
def cond_sum(s):
return s.cond1.all() * s.cond2.sum()
df.reset_index(level=0, inplace=True)
df['result'] = df.groupby('date').apply(cond_sum)
df.set_index('item', append=True)
Then if you want the original index, you can add it back.
df.set_index('item', append=True).swaplevel()
Note, you mentioned vectorized, so you could swap that out for:
dfg = df.groupby(level=0).agg({'cond1': 'all', 'cond2': 'sum'})
df['result'] = dfg.cond1 * dfg.cond2

Pandas True False Matching

For this table:
I would like to generate the 'desired_output' column. One way to achieve this maybe:
All the True values from col_1 are transferred straight across to desired_output (red arrow)
In desired_output, place a True value above any existing True value (green arrow)
Code I have tried:
df['desired_output']=df.col_1.apply(lambda x: True if x.shift()==True else False)
Thankyou
You can chain by | for bitwise OR original with shifted values by Series.shift:
d = {"col1":[False,True,True,True,False,True,False,False,True,False,False,False]}
df = pd.DataFrame(d)
df['new'] = df.col1 | df.col1.shift(-1)
print (df)
col1 new
0 False True
1 True True
2 True True
3 True True
4 False True
5 True True
6 False False
7 False True
8 True True
9 False False
10 False False
11 False False
try this
df['desired_output'] = df['col_1']
df.loc[1:, 'desired_output'] = df.col_1[1:].values | df.col_1[:-1].values
print(df)
In case those are saved as string. all_caps (TRUE / FALSE)
Input:
col_1
0 True
1 True
2 False
3 True
4 True
5 False
6 Flase
7 True
8 False
Code:
df['desired']=df['col_1']
for i, e in enumerate(df['col_1']):
if e=='True':
df.at[i-1,'desired']=df.at[i,'col_1']
df = df[:(len(df)-1)]
df
Output:
col_1 desired
0 True True
1 True True
2 False True
3 True True
4 True True
5 False False
6 Flase True
7 True True
8 False False

How to vectorize in Pandas when values depend on prior values

I'd like to use Pandas to implement a function that keeps a running balance, but I'm not sure it can be vectorized for speed.
In short, the problem I'm trying to solve is to keep track consumption, generation, and the "bank" of over-generation.
"consumption" means how much is used in a given time period.
"generation" is how much is generated.
When generation is greater than consumption then the homeowner can "bank" the extra generation, to be applied in subsequent time periods. they can apply it if their consumption exceeds their generation for a later month.
This will be for many entities, hence the "id" field. The time sequence is defined by "order"
Very basic example:
Month 1 generates 13 consumes 8 -> therefore banks 5
month 2 generates 8 consumes 10 -> therefore uses 2 from the the bank, and still has 3 left over
Month 3 generates 7 consumes 20 -> exhausts remaining 3 from bank, and has no bank left over.
Code
import numpy as np
import pandas as pd
id = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2]
order = [1,2,3,4,5,6,7,8,9,18,11,12,13,14,15,1,2,3,4,5,6,7,8,9,10,11]
consume = [10, 17, 20, 11, 17, 19, 20, 10, 10, 19, 14, 12, 10, 14, 13, 19, 12, 17, 12, 18, 15, 14, 15, 20, 16, 15]
generate = [20, 16, 17, 21, 9, 13, 10, 16, 12, 10, 9, 9, 15, 13, 100, 15, 18, 16, 10, 16, 12, 12, 13, 20, 10, 15]
df = pd.DataFrame(list(zip(id, order, consume, generate)),
columns =['id','Order','Consume', 'Generate'])
begin_bal = [0,10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0]
end_bal = [10,9,6,16,8,2,0,6,8,0,0,0,5,4,91,0,6,5,3,1,0,0,0,0,0,0]
withdraw = [0,1,3,0,8,6,2,0,0,8,0,0,0,1,4,0,0,1,2,2,1,0,0,0,0,0]
df_solution = pd.DataFrame(list(zip(id, order, consume, generate, begin_bal, end_bal, withdraw)),
columns =['id','Order','Consume', 'Generate', 'begin_bal', 'end_bal', 'Withdraw'])
def bank(df):
# deposit all excess when generation exceeds consumption
deposit = (df['Generate'] > df['Consume']) * (df['Generate'] - df['Consume'])
df['end_bal'] = 0
# beginning balance = prior period ending balance
df = df.sort_values(by=['id', 'Order'])
df['begin_bal'] = df['end_bal'].shift(periods=1)
df.loc[df['Order']==1, 'begin_bal'] = 0 # set first month beginning balance of each customer to 0
# calculate withdrawal
df['Withdraw'] = 0
ok_to_withdraw = df['Consume'] > df['Generate']
df.loc[ok_to_withdraw,'Withdraw'] = np.minimum(df.loc[ok_to_withdraw, 'begin_bal'],
df.loc[ok_to_withdraw, 'Consume'] -
df.loc[ok_to_withdraw, 'Generate'] -
deposit[ok_to_withdraw])
# ending balance = beginning balance + deposit - withdraw
df['end_bal'] = df['begin_bal'] + deposit - df['Withdraw']
return df
df = bank(df)
df.head()
id Order Consume Generate end_bal begin_bal Withdraw
0 1 1 10 20 10.0 0.0 0.0
1 1 2 17 16 0.0 0.0 0.0
2 1 3 20 17 0.0 0.0 0.0
3 1 4 11 21 10.0 0.0 0.0
4 1 5 17 9 0.0 0.0 0.0
df_solution.head()
id Order Consume Generate begin_bal end_bal Withdraw
0 1 1 10 20 0 10 0
1 1 2 17 16 10 9 1
2 1 3 20 17 9 6 3
3 1 4 11 21 6 16 0
4 1 5 17 9 16 8 9
I tried to implement with various iterations of cumsum and shift . . . but the fact remains that value of each row seems like it needs to be recalculated based on the prior row, and I'm not sure this is possible to vectorize.
Code to generate some test datasets:
def generate_testdata():
random.seed(42*42)
np.random.seed(42*42)
numids = 10
numorders = 12
id = []
order = []
for i in range(numids):
id = id + [i]*numorders
order = order + list(range(1,numorders+1))
consume = np.random.uniform(low = 10, high = 40, size = numids*numorders)
generate = np.random.uniform(low = 10, high = 40, size = numids*numorders)
df = pd.DataFrame(list(zip(id, order, consume, generate)),
columns =['id','Order','Consume', 'Generate'])
return df
Here is a numpy-ish approach, mostly because I'm not that familiar with pandas:
The idea is to first compute the free cumsum and then to subtract the cumulative minimum if it is negative.
import numpy as np
import pandas as pd
id = [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2]
order = [1,2,3,4,5,6,7,8,9,18,11,12,13,14,15,1,2,3,4,5,6,7,8,9,10,11]
consume = [10, 17, 20, 11, 17, 19, 20, 10, 10, 19, 14, 12, 10, 14, 13, 19, 12, 17, 12, 18, 15, 14, 15, 20, 16, 15]
generate = [20, 16, 17, 21, 9, 13, 10, 16, 12, 10, 9, 9, 15, 13, 8, 15, 18, 16, 10, 16, 12, 12, 13, 20, 10, 15]
df = pd.DataFrame(list(zip(id, order, consume, generate)),
columns =['id','Order','Consume', 'Generate'])
begin_bal = [0,10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0]
end_bal = [10,9,6,16,8,2,0,6,8,0,0,0,5,4,0,0,6,5,3,1,0,0,0,0,0,0]
withdraw = [0,1,3,0,9,6,2,0,0,8,0,0,0,1,4,0,0,1,2,2,1,0,0,0,0,0]
df_solution = pd.DataFrame(list(zip(id, order, consume, generate, begin_bal, end_bal, withdraw)),
columns =['id','Order','Consume', 'Generate', 'begin_bal', 'end_bal', 'Withdraw'])
def f(df):
# find block bondaries
ids = df["id"].values
bnds, = np.where(np.diff(ids, prepend=ids[0]-1, append=ids[-1]+1))
# find raw balance change
delta = (df["Generate"] - df["Consume"]).values
# find offset, so cumulative min does not interfere across ids
safe_total = (np.minimum(delta.min(), 0)-1) * np.diff(bnds[:-1])
# must apply offset just before group switch, so it aligns the first
# begin_bal, not end_bal, of the next group
# also keep a copy of original values at switches
delta_orig = delta[bnds[1:-1]-1]
delta[bnds[1:-1]-1] += safe_total - np.add.reduceat(delta, bnds[:-2])
# form free cumsum
acc = delta.cumsum()
# correct
acc -= np.minimum(0, np.minimum.accumulate(acc))
# write solution back to df
shft = np.empty_like(acc)
shft[1:] = acc[:-1]
shft[0] = 0
# reinstate last end_bal of each group
acc[bnds[1:-1]-1] = np.maximum(0, shft[bnds[1:-1]-1] + delta_orig)
df["begin_bal"] = shft
df["end_bal"] = acc
df["Withdraw"] = np.maximum(0, df["begin_bal"] - df["end_bal"])
Test:
f(df)
df == df_solution
Prints:
id Order Consume Generate begin_bal end_bal Withdraw
0 True True True True True True True
1 True True True True True True True
2 True True True True True True True
3 True True True True True True True
4 True True True True True True False
5 True True True True True True True
6 True True True True True True True
7 True True True True True True True
8 True True True True True True True
9 True True True True True True True
10 True True True True True True True
11 True True True True True True True
12 True True True True True True True
13 True True True True True True True
14 True True True True True True True
15 True True True True True True True
16 True True True True True True True
17 True True True True True True True
18 True True True True True True True
19 True True True True True True True
20 True True True True True True True
21 True True True True True True True
22 True True True True True True True
23 True True True True True True True
24 True True True True True True True
25 True True True True True True True
There is one False but that appears to be a typo in the expected output provided.
Using #PaulPanzer's logic here is a pandas version.
def CalcEB(x):
delta = x['Generate'] - x['Consume']
return delta.cumsum() - delta.cumsum().cummin().clip(-np.inf,0)
df['end_bal'] = df.groupby('id', as_index=False).apply(CalcEB).values
df['begin_bal'] = df.groupby('id')['end_bal'].shift().fillna(0)
df['Withdraw'] = (df['begin_bal'] - df['end_bal']).clip(0,np.inf)
df_pandas = df.copy()
#Note the typo mentioned by Paul Panzer
df_pandas.reindex(df_solution.columns, axis=1) == df_solution
Output (check dataframes)
id Order Consume Generate begin_bal end_bal Withdraw
0 True True True True True True True
1 True True True True True True True
2 True True True True True True True
3 True True True True True True True
4 True True True True True True False
5 True True True True True True True
6 True True True True True True True
7 True True True True True True True
8 True True True True True True True
9 True True True True True True True
10 True True True True True True True
11 True True True True True True True
12 True True True True True True True
13 True True True True True True True
14 True True True True True True True
15 True True True True True True True
16 True True True True True True True
17 True True True True True True True
18 True True True True True True True
19 True True True True True True True
20 True True True True True True True
21 True True True True True True True
22 True True True True True True True
23 True True True True True True True
24 True True True True True True True
25 True True True True True True True
I am not sure I understood your question fully, but I am going to give a go at answering.
I will re-phrase what I understood...
1. Source data
There is source data, which is a DataFrame with four columns:
id - ID number of an entity
order - indicates the sequence of periods
consume - how much was consumed during the period
generate - how much was generated during the period
2. Calculations
For each id, we want to calculate:
diff which is the difference between generate and consume for each period
opening balance which is the closing balance from the previous order
closing balance which is the cumulative sum of the diff
3. Code
I will try to solve this with groupby, cumsum and shift.
# Make sure the df is sorted
df = df.sort_values(['id','order'])
df['diff'] = df['generate'] - df['consume']
df['closing_balance'] = df.groupby('id')['diff'].cumsum()
# Opening balance equals the closing balance from the previous period
df['opening_balance'] = df.groupby('id')['closing_balance'].shift(1)
I definitely misunderstood something, feel free to correct me and I will try to come up with a better answer.
In particular, I wasn't sure how to handle the closing_balance going into negative numbers. Should it show negative balance? Should it nullify the "debts"?

How to count trues/falses from a list?

df = [bigdataframe[['Action', 'Adventure','Animation',
'Childrens', 'Comedy', 'Crime','Documentary',
'Drama', 'Fantasy', 'FilmNoir', 'Horror',
'Musical',
'Mystery', 'Romance','SciFi', 'Thriller', 'War',
'Western']].sum(axis=1) > 1]
df
Out[8]:
[0 True
1 True
2 True
3 True
4 True
5 False
6 True
7 True
8 False
9 True
10 False
11 True
12 True
13 True
14 True
15 False
16 True
17 False
18 True
19 False
20 False
21 True
22 True
23 True
24 False
25 True
26 True
27 True
28 True
29 True
99970 True
99971 True
99972 False
99973 True
99974 True
99975 True
99976 True
99977 True
99978 False
99979 False
99980 True
99981 False
99982 True
99983 False
99984 True
99985 True
99986 True
99987 True
99988 False
99989 True
99990 True
99991 True
99992 False
99993 True
99994 True
99995 True
99996 True
99997 True
99998 True
99999 False
Length: 100000, dtype: bool]
I have tried:
len(df[df==True])
Masking
They are in a list so shouldn't I just be able to count them? Or do I need to assign them numerical values, 1 for true and 0 for false and then use the count or sum function to find how many are true?
Demo:
In [386]: df = pd.DataFrame(np.random.rand(5,3), columns=list('ABC'))
In [387]: df
Out[387]:
A B C
0 0.228687 0.647431 0.526471
1 0.795122 0.915011 0.950481
2 0.386244 0.705412 0.420596
3 0.343213 0.928993 0.192527
4 0.201023 0.209281 0.304799
In [388]: df[['A','B','C']].sum(axis=1).gt(1.5)
Out[388]:
0 False
1 True
2 True
3 False
4 False
dtype: bool
In [389]: df[['A','B','C']].sum(axis=1).gt(1.5).sum()
Out[389]: 2
to count number of true in a list
sum(unlist(your.list.object))

Groupby pandas calculate percentage

I have a groupby object as follows after i ran:
grouped_mask=L2014_2.groupby(['state'])
grouped_mask.mask.value_counts()
state mask
AL False 105931
True 77
AR False 67788
True 1774
AZ False 90068
True 151
CA False 586184
True 4
CO False 75188
True 14360
CT False 78270
True 1
Now i need to calculate what is percentage of true in each state. Is there a way to do this?
Also you can set the normalize parameter to obtain the relative frequencies:
grouped_mask.mask.value_counts(normalize=True)
just multiply by 100 to get the percentages :-)
regards
You can groupby on the first level and then apply a lambda that divides the True/False counts against the sum:
In [20]:
df.groupby(level=0).apply(lambda x: x/x.sum() * 100)
Out[20]:
Count
state mask
AL False 99.927364
True 0.072636
AR False 97.449757
True 2.550243
AZ False 99.832629
True 0.167371
CA False 99.999318
True 0.000682
CO False 83.963908
True 16.036092
CT False 99.998722
True 0.001278
To filter the above to get just the False labels you can use advanced indexing using slices:
In [33]:
gp = df.groupby(level=0).apply(lambda x: x/x.sum() * 100)
gp.loc(axis=0)[slice(None),False]
Out[33]:
Count
state mask
AL False 99.927364
AR False 97.449757
AZ False 99.832629
CA False 99.999318
CO False 83.963908
CT False 99.998722
gp=grouped_mask.mask.value_counts().groupby(level=0).apply(lambda x: 100*x/float(x.sum()))
state mask
AL False 94.37
True 5.63
AR False 73.85
True 26.15
AZ False 91.88
True 8.12
CA False 99.57
True 0.43
CO False 64.66
True 35.34
gp.sort_index(level=0)
gp.loc(axis=0)[slice(None),False]
KeyError: 'MultiIndex Slicing requires the index to be fully lexsorted tuple len (2), lexsort depth (1)'