Sup, ive been stuck on this problem since last night , i am trying to increase the scatter plots on
Scattergeo dependent on the magnitude of a earthquake, i keep getting Nonetype error here's the code, when i add size to :
[Scattergeo(lon=lons,lat=lats,size=[5*mag for mag in mags])]
import json
from plotly import offline
from plotly.graph_objs import *
with open('all_month.geojson.json',encoding='utf8') as geo :
file = json.load(geo)
with open('earthquakes','w') as eq :
json.dump(file,eq,indent=4)
file = file['features']
mags,lons,lats = [],[],[]
for columns in file :
mag = columns['properties']['mag']
lon = columns['geometry']['coordinates'][0]
lat = columns['geometry']['coordinates'][1]
mags.append(mag)
lons.append(lon)
lats.append(lat)
data = [Scattergeo(lon=lons,lat=lats)]
my_layout = Layout(title='Earthquakes around the world for the past 30 days')
offline.plot({'data':data,'layout':my_layout},filename='Eq.html')
OK, i figured it out, when i downloaded the data, there were mutiple options as to the format and i recommend downloading the +1 magnitude and above
.
here is the output
Related
My program is optimizing the charging and decharging of a home battery to minimize the cost of electricity at the end of the year. In this case there also is a PV, which means that sometimes you're injecting electricity into the grid and receive money. The net offtake is the result of the usage of the home and the PV installation. So these are the possible situations:
Net offtake > 0 => usage home > PV => discharge from battery or take from grid
Net offtake < 0 => usage home < PV => charge battery or injection into grid
The electricity usage of homes is measured each 15 minutes, so I have 96 measurement point in 1 day. I want to optimilize the charging and decharging of the battery for 2 days, so that day 1 takes the usage of day 2 into account.
I wrote a controller that reads the data and gives each time the input values for 2 days to the optimization. With a rolling principle, it goes to the next 2 days and so on. Below you can see the code from my controller.
from gekko import GEKKO
from simulationModel2_2d_1 import getSimulation2
from exportModel2 import exportToExcelModel2
import numpy as np
#import matplotlib.pyplot as plt
import pandas as pd
import time
import math
# ------------------------ Import and read input data ------------------------
file = r'Data Sim 2.xlsx'
data = pd.read_excel(file, sheet_name='Input', na_values='NaN')
dataRead = pd.DataFrame(data, columns= ['Timestep','Verbruik woning (kWh)','Netto afname (kWh)','Prijs afname (€/kWh)',
'Prijs injectie (€/kWh)','Capaciteit batterij (kW)',
'Capaciteit batterij (kWh)','Rendement (%)',
'Verbruikersprofiel','Capaciteit PV (kWp)','Aantal dagen'])
timestep = dataRead['Timestep'].to_numpy()
usage_home = dataRead['Verbruik woning (kWh)'].to_numpy()
net_offtake = dataRead['Netto afname (kWh)'].to_numpy()
price_offtake = dataRead['Prijs afname (€/kWh)'].to_numpy()
price_injection = dataRead['Prijs injectie (€/kWh)'].to_numpy()
cap_batt_kW = dataRead['Capaciteit batterij (kW)'].iloc[0]
cap_batt_kWh = dataRead['Capaciteit batterij (kWh)'].iloc[0]
efficiency = dataRead['Rendement (%)'].iloc[0]
usersprofile = dataRead['Verbruikersprofiel'].iloc[0]
days = dataRead['Aantal dagen'].iloc[0]
pv = dataRead['Capaciteit PV (kWp)'].iloc[0]
# ------------- Optimization model & Rolling principle (2 days) --------------
# Initialise model
m = GEKKO()
# Output data
ts = []
charging = [] # Amount to charge/decharge batterij
e_batt = [] # Amount of energy in the battery
usage_net = [] # Usage after home, battery and pv
p_paid = [] # Price paid for energy of 15min
# Energy in battery to pass
energy = 0
# Iterate each day for one year
for d in range(int(days)-1):
d1_timestep = []
d1_net_offtake = []
d1_price_offtake = []
d1_price_injection = []
d2_timestep = []
d2_net_offtake = []
d2_price_offtake = []
d2_price_injection = []
# Iterate timesteps
for i in range(96):
d1_timestep.append(timestep[d*96+i])
d2_timestep.append(timestep[d*96+i+96])
d1_net_offtake.append(net_offtake[d*96+i])
d2_net_offtake.append(net_offtake[d*96+i+96])
d1_price_offtake.append(price_offtake[d*96+i])
d2_price_offtake.append(price_offtake[d*96+i+96])
d1_price_injection.append(price_injection[d*96+i])
d2_price_injection.append(price_injection[d*96+i+96])
# Input data simulation of 2 days
ts_temp = np.concatenate((d1_timestep, d2_timestep))
net_offtake_temp = np.concatenate((d1_net_offtake, d2_net_offtake))
price_offtake_temp = np.concatenate((d1_price_offtake, d2_price_offtake))
price_injection_temp = np.concatenate((d1_price_injection, d2_price_injection))
if(d == 7):
print(ts_temp)
print(energy)
# Simulatie uitvoeren
charging_temp, e_batt_temp, usage_net_temp, p_paid_temp, energy_temp = getSimulation2(ts_temp, net_offtake_temp, price_offtake_temp, price_injection_temp, cap_batt_kW, cap_batt_kWh, efficiency, energy, pv)
# Take over output first day, unless last 2 days
energy = energy_temp
if(d == (days-2)):
for t in range(1,len(ts_temp)):
ts.append(ts_temp[t])
charging.append(charging_temp[t])
e_batt.append(e_batt_temp[t])
usage_net.append(usage_net_temp[t])
p_paid.append(p_paid_temp[t])
elif(d == 0):
for t in range(int(len(ts_temp)/2)+1):
ts.append(ts_temp[t])
charging.append(charging_temp[t])
e_batt.append(e_batt_temp[t])
usage_net.append(usage_net_temp[t])
p_paid.append(p_paid_temp[t])
else:
for t in range(1,int(len(ts_temp)/2)+1):
ts.append(ts_temp[t])
charging.append(charging_temp[t])
e_batt.append(e_batt_temp[t])
usage_net.append(usage_net_temp[t])
p_paid.append(p_paid_temp[t])
print('Simulation day '+str(d+1)+' complete.')
# ------------------------ Export output data to Excel -----------------------
a = exportToExcelModel2(ts, usage_home, net_offtake, price_offtake, price_injection, charging, e_batt, usage_net, p_paid, cap_batt_kW, cap_batt_kWh, efficiency, usersprofile, pv)
print(a)
The optimization with Gekko happens in the following code:
from gekko import GEKKO
def getSimulation2(timestep, net_offtake, price_offtake, price_injection,
cap_batt_kW, cap_batt_kWh, efficiency, start_energy, pv):
# ---------------------------- Optimization model ----------------------------
# Initialise model
m = GEKKO(remote = False)
# Global options
m.options.SOLVER = 1
m.options.IMODE = 6
# Constants
speed_charging = cap_batt_kW/4
m.time = timestep
max_cap_batt = m.Const(value = cap_batt_kWh)
min_cap_batt = m.Const(value = 0)
max_charge = m.Const(value = speed_charging) # max battery can charge in 15min
max_decharge = m.Const(value = -speed_charging) # max battery can decharge in 15min
# Parameters
usage_home = m.Param(net_offtake)
price_offtake = m.Param(price_offtake)
price_injection = m.Param(price_injection)
# Variables
e_batt = m.Var(value=start_energy, lb = min_cap_batt, ub = max_cap_batt) # energy in battery
price_paid = m.Var() # price paid each 15min
charging = m.Var(lb = max_decharge, ub = max_charge) # amount charge/decharge each 15min
usage_net = m.Var(lb=min_cap_batt)
# Equations
m.Equation(e_batt==(m.integral(charging)+start_energy)*efficiency)
m.Equation(-charging <= e_batt)
m.Equation(usage_net==usage_home + charging)
price = m.Intermediate(m.if2(usage_net*1e6, price_injection, price_offtake))
price_paid = m.Intermediate(usage_net * price / 100)
# Objective
m.Minimize(price_paid)
# Solve problem
m.options.COLDSTART=2
m.solve()
m.options.TIME_SHIFT=0
m.options.COLDSTART=0
m.solve()
# Energy to pass
energy_left = e_batt[95]
#m.cleanup()
return charging, e_batt, usage_net, price_paid, energy_left
The data you need for input can be found in this Excel document:
https://docs.google.com/spreadsheets/d/1S40Ut9-eN_PrftPCNPoWl8WDDQtu54f0/edit?usp=sharing&ouid=104786612700360067470&rtpof=true&sd=true
With this code, it always ends at day 17 with the "Solution Not Found" Error.
I already tried extending the default iteration limit to 500 but it didn't work.
I also tried with other solvers but also no improvement.
By presolving with COLDSTART it already reached day 17, without this it ends at day 8.
I solved the days were my optimization ends individually and then the solution was always found immediately with the same code.
Someone who can explain this to me and maybe find a solution? Thanks in advance!
This is kind of big to troubleshoot, but here are some general ideas that might help. This assumes, as you said, that the model solves fine for day 1-2, and day 3-4, and day 5-6, etc. And that those results pass inspection (aka the basic model is working as you say).
Then something is (obviously) amiss around day 17. Some things to look at and try:
Start the model at day 16-17, see if it works in isolation
gather your results as you go and do a time series plot of the key variables, maybe one of them is on an obvious bad trend towards a limit causing an infeasibility... Perhaps the e_batt variable is slowly declining because not enough PV energy is available and hits minimum on Day 17
Radically change the upper/lower bounds on your variables to test them to see if they might be involved in the infeasibility (assuming there is one)
Make a different (fake) dataset where the solution is assured and kind of obvious... all constants or a pattern that you know will produce some known result and test the model outputs against it.
It might also be useful to pursue the tips in this excellent post on troubleshooting gekko, and edit your question with the results of that effort.
edit: couple ideas from your comment...
You didn't say what the result was from troubleshooting and whether the error is infeasible or max iterations, or ???. But...
If the model seems to crunch after about 15 days, I'm betting that it is becoming infeasible. Did you plot the battery level over course of days?
Also, I'm suspicious of your equation for the e_batt level... Why are you multiplying the prior battery state by the efficiency factor? That seems incorrect. That is charge that is already in the battery. Odds are you are (incorrectly) hitting the battery charge level every day with the efficiency tax and that the max charge level isn't sufficient to keep up with demand.
In addition to tips above try:
fix your efficiency formula to not multiply the efficiency times the previous state
change the efficiency to 100%
make the upper limit on charge huge
As an aside: I don't really see the connection to PV energy available here. What you are basically modeling is some "mystery battery" that you can charge and discharge anytime you want. I would get this debugged first and then look at the energy available by time of day...you aren't going to be generating charge at midnight. :).
Im trying to add coördinates to a set of addresses that are saved in an excel file using the google geocoder API. See code below:
for i, row in df.iterrows():
#below combines the address columns together in one variable, to push to the geocoder API.
apiAddress = str(df.at[i, 'adresse1']) + ',' + str(df.at[i, 'postnr']) + ',' + str(df.at[i, 'By'])
#below creates a dictionary with the API key and the address info, to push to the Geocoder API on each iteration
parameters = {
'key' : API_KEY,
'address' : apiAddress
}
#response from the API, based on the input url + the dictionary above.
response = requests.get(base_url, params = parameters).json()
#when you look at the response, it is given as a dictionary. with this command I access the geometry part of the dictionary.
geometry = response['results'][0]['geometry']
#within the geometry party of the dictionary given by the API, I access the lat and lng respectively.
lat = geometry['location']['lat']
lng = geometry['location']['lng']
#here I append the lat / lng to a new column in the dataframe for each iteration.
df.at[i, 'Geo_Lat_New'] = lat
df.at[i, 'Geo_Lng_New'] = lng
#printing the first 10 rows.
print(df.head(10))
the above code works perfectly fine for 20 addresses. But when I try to run it on the entire dataset of 90000 addresses; using iterrows() I get a IndexError:
File "C:\Users\...", line 29, in <module>
geometry = response['results'][0]['geometry']
IndexError: list index out of range
Using itertuples() instead, with:
for i, row in df.itertuples():
I get a ValueError:
File "C:\Users\...", line 22, in <module>
for i, row in df.itertuples():
ValueError: too many values to unpack (expected 2)
when I use:
for i in df.itertuples():
I get a complicated KeyError. That is to long to put here.
Any suggestions on how to properly add coördinates for each address in the entire dataframe?
Update, in the end I found out what the issue was. The google geocoding API only handles 50 request per second. Therefore I used to following code to take a 1 second break after every 49 requests:
if count == 49:
print('Taking a 1 second break, total count is:', total_count)
time.sleep(1)
count = 0
Where count keeps count of the number of loops, as soon as it hits 49, the IF statement above is executed, taking a 1 second break and resetting the count back to zero.
Although you have already found the error - Google API limits the amount of requests that can be done - it isn't usually good practice to use for with pandas. Therefore, I would re write your code to take advantage of pd.DataFrame.apply.
def get_geometry(row: pd.Series, API_KEY: str, base_url: str, tries: int = 0):
apiAddress = ",".join(row["adresse1"], row["postnr"], row["By"])
parameters = {"key": API_KEY, "address": apiAddress}
try:
response = requests.get(base_url, params = parameters).json()
geometry = response["results"][0]["geometry"]
except IndexError: # reach limit
# sleep to make the next 50 requests, but
# beware that consistently reaching limits could
# further limit sending requests.
# this is why you might want to keep track of how
# many tries you have already done, as to stop the process
# if a threshold has been met
if tries > 3: # tries > arbitrary threshold
raise
time.sleep(1)
return get_geometry(row, API_KEY, base_url, tries + 1)
else:
geometry = response["results"][0]["geometry"]
return geometry["location"]["lat"], geometry["location"]["lng"]
# pass kwargs to apply function and iterate over every row
lat_lon = df.apply(get_geometry, API_KEY = API_KEY, base_url = base_url, axis = 1)
df["Geo_Lat_New"] = lat_lon.apply(lambda latlon: latlon[0])
df["Geo_Lng_New"] = lat_lon.apply(lambda latlon: latlon[1])
I am currently trying to save a df frame where the index is a Date (no minute and seconds).
Pasted Index
While the saving works with no issues with the following code
path = "test.h5"
store = pd.HDFStore(path, mode="a", encoding = "utf-8")
store.append("df",dftest)
store.close()
and loading as well with the following code
path = "test.h5"
store = pd.HDFStore(path, mode="a", encoding = "utf-8")
c = store.select_column("df", "index")
dftest = store.select("df", where=c[c.isin(date)].index)
store.close()
The index output changes to a number
Loaded Index
Does anyone have any clue why. I also tried to save it as a full date format with hours minutes seconds but get an error.
Thanks for your help
Edit:
date is a filter variables containing a list of dates
e.g.
date = [dt.date(2022, 8, 30]
I'm currently trying to implement my own source block on GNURADIO. What this source block does is takes the values of 4 arguments (titled Value1,Value2,Value3 and Value4) and converts them to binary values with varying lengths. For example, Value1 is converted into a binary value with a length of 11 bits, whilst Value2 is converted into a binary value with a length of 16 bits etc. The conversion of these values work.
The issue I'm having is that I am unable to output these converted values onto a vector sink. when the work function returns (len(output[0]),when I execute the code using QA code. I end up with an endless loop, which when interrupted, outputs an extremely large vector with values of "0L".
Now from reading similar questions, I understand that as my block is set as a source, it will send out data as long as the vector sink has enough room in its input buffer. Therefore I set the return value to be -1 to force the flow graph to stop. This gives me an empty value "()" in the Vector sink.
The code is shown below:
import numpy
import time
import sys
import random
import os
from gnuget import *
from gnuradio import gr
from gnuradio import blocks
from gnuradio.blocks.blocks_swig1 import vector_source_b
class ConvertVariables(gr.sync_block):
"""
docstring for block ConvertVariables
"""
def __init__(self,Value1,Value2,Value3,Value4):
self.value1 = Value1
self.value2 = Value2
self.value3 = Value3
self.value4 = Value4
gr.sync_block.__init__(self,
name="ConvertVariables",
in_sig=None,
out_sig=[(numpy.byte)])
def work(self, input_items, output_items):
out = output_items[0]
self.value1_R = getvalue1(self.value1)
self.value2_R = getvalue2(self.value2)
self.value3_R = getvalue3(self.value3)
self.value4_R = getvalue4(self.value4)
self.value1_B = bytearray(self.value1_R,'utf-8')
self.value2_B = bytearray(self.value2_R,'utf-8')
self.value3_B = bytearray(self.value3_R,'utf-8')
self.value4_B = bytearray(self.value4_R,'utf-8')
print(type(self.value1_B))
print(type(self.value2_B))
print(type(self.value3_B))
print(type(self.value4_B))
print(len(output_items),"output_items")
self.out[:] = [self.value1_B,self.value2_B,self.value3_B,self.value4_B]
print(len(output_items[0]))
return (len(output_items[0]))
So ideally when the arguments(2034,5000,50,5) are sent. The expected result in the vector sink is
('11111110010','0001001110001000','110010','0101')
However what appears instead is either
(0L,0L,0L,0L)
(The length of the vector is not consistent, if I set the return value of work to len(output[0]), the terminal windows fills with 0L)
or
()
This only occurs if the return value of work is set to -1
I am trying to find maximum values of a sinusoidal and show it in a subplot. And trying to update it like an animation. But the subplot of the maximum values gives all zero values. When I print the array it is not zeroes. I think it is not updating y values. I couldn't figure out the reason. Any help would be kindly appreciated.
I will put my code it is executable:
from pylab import *
import time
ion()
fs = 1e6
Ts = 1/fs
SNR=10
sinfreq=2*pi*1e5
pack= 512
t = Ts*arange(0,pack)
f = fs*(arange(0,pack)-pack/2)/pack
max_y = zeros (len(t))
y=sin(sinfreq*t)
y=y+randn(size(y))/sqrt(10^(SNR/10)*2)
subplot(211)
line1, = plot(y)
subplot(212)
line2, = plot(max_y)
for i1 in arange(1,1000):
y=sin(sinfreq*t)
y=y+randn(size(y))/sqrt(10^(SNR/10)*2)
line1.set_ydata(y)
mk=0
for mk in range(0,len(y)):
if y[mk] > max_y[mk]:
max_y[mk] = y[mk]
print max_y
line2.set_ydata(max_y)
draw()
waitforbuttonpress(timeout=0.5)
You've forgotten an indent in the second for loop.
That should actually give you an IndentationError, so I don't understand how you can say that the program is executable (in fact, I edited your entry to remove the left-over "enter code here" statement; if you had checked and copy-pasted your entry, you'd probably found both mistakes).
But, are you sure you don't want simply
max_y[:] = max(y)
instead of that for loop?
Ok I have found the solution incidentally, I don't know the reason but I update my max_y plot any other value firstly then do my real update, then the plot shows my changes. Other than this it is not showing. I try with different for loops they are ploting with one update but for my loop it wanted to update twice.
I have also add set_ylim to see better the limits. I put stars to places that I have changed. I am putting the new code also. I wish it will help people that have the same problem.
from pylab import *
import time
ion()
fs = 1e6
Ts = 1/fs
SNR=10
sinfreq=2*pi*1e5
pack= 512
t = Ts*arange(0,pack)
f = fs*(arange(0,pack)-pack/2)/pack
max_y = zeros (len(t))
y=sin(sinfreq*t)
y=y+randn(size(y))/sqrt(10^(SNR/10)*2)
subplot(211)
line1, = plot(y)
sub2=subplot(212) #****
line2, = plot(max_y)
for i1 in arange(1,1000):
y=sin(sinfreq*t)
y=y+randn(size(y))/sqrt(10^(SNR/10)*2)
line1.set_ydata(y)
mk=0
for mk in range(0,len(y)):
if y[mk] > max_y[mk]:
max_y[mk] = y[mk]
#print max_y
line2.set_ydata(zeros(len(max_y)))#****
line2.set_ydata(max_y)
sub2.set_ylim(min(max_y),max(max_y)) #****
draw()
waitforbuttonpress(timeout=0.5)