I know that some other languages, such as PHP, support a concept of "variable variable names" - that is, the contents of a string can be used as part of a variable name.
I heard that this is a bad idea in general, but I think it would solve some problems I have in my Python code.
Is it possible to do something like this in Python? What can go wrong?
If you are just trying to look up an existing variable by its name, see How can I select a variable by (string) name?. However, first consider whether you can reorganize the code to avoid that need, following the advice in this question.
You can use dictionaries to accomplish this. Dictionaries are stores of keys and values.
>>> dct = {'x': 1, 'y': 2, 'z': 3}
>>> dct
{'y': 2, 'x': 1, 'z': 3}
>>> dct["y"]
2
You can use variable key names to achieve the effect of variable variables without the security risk.
>>> x = "spam"
>>> z = {x: "eggs"}
>>> z["spam"]
'eggs'
For cases where you're thinking of doing something like
var1 = 'foo'
var2 = 'bar'
var3 = 'baz'
...
a list may be more appropriate than a dict. A list represents an ordered sequence of objects, with integer indices:
lst = ['foo', 'bar', 'baz']
print(lst[1]) # prints bar, because indices start at 0
lst.append('potatoes') # lst is now ['foo', 'bar', 'baz', 'potatoes']
For ordered sequences, lists are more convenient than dicts with integer keys, because lists support iteration in index order, slicing, append, and other operations that would require awkward key management with a dict.
Use the built-in getattr function to get an attribute on an object by name. Modify the name as needed.
obj.spam = 'eggs'
name = 'spam'
getattr(obj, name) # returns 'eggs'
It's not a good idea. If you are accessing a global variable you can use globals().
>>> a = 10
>>> globals()['a']
10
If you want to access a variable in the local scope you can use locals(), but you cannot assign values to the returned dict.
A better solution is to use getattr or store your variables in a dictionary and then access them by name.
New coders sometimes write code like this:
my_calculator.button_0 = tkinter.Button(root, text=0)
my_calculator.button_1 = tkinter.Button(root, text=1)
my_calculator.button_2 = tkinter.Button(root, text=2)
...
The coder is then left with a pile of named variables, with a coding effort of O(m * n), where m is the number of named variables and n is the number of times that group of variables needs to be accessed (including creation). The more astute beginner observes that the only difference in each of those lines is a number that changes based on a rule, and decides to use a loop. However, they get stuck on how to dynamically create those variable names, and may try something like this:
for i in range(10):
my_calculator.('button_%d' % i) = tkinter.Button(root, text=i)
They soon find that this does not work.
If the program requires arbitrary variable "names," a dictionary is the best choice, as explained in other answers. However, if you're simply trying to create many variables and you don't mind referring to them with a sequence of integers, you're probably looking for a list. This is particularly true if your data are homogeneous, such as daily temperature readings, weekly quiz scores, or a grid of graphical widgets.
This can be assembled as follows:
my_calculator.buttons = []
for i in range(10):
my_calculator.buttons.append(tkinter.Button(root, text=i))
This list can also be created in one line with a comprehension:
my_calculator.buttons = [tkinter.Button(root, text=i) for i in range(10)]
The result in either case is a populated list, with the first element accessed with my_calculator.buttons[0], the next with my_calculator.buttons[1], and so on. The "base" variable name becomes the name of the list and the varying identifier is used to access it.
Finally, don't forget other data structures, such as the set - this is similar to a dictionary, except that each "name" doesn't have a value attached to it. If you simply need a "bag" of objects, this can be a great choice. Instead of something like this:
keyword_1 = 'apple'
keyword_2 = 'banana'
if query == keyword_1 or query == keyword_2:
print('Match.')
You will have this:
keywords = {'apple', 'banana'}
if query in keywords:
print('Match.')
Use a list for a sequence of similar objects, a set for an arbitrarily-ordered bag of objects, or a dict for a bag of names with associated values.
Whenever you want to use variable variables, it's probably better to use a dictionary. So instead of writing
$foo = "bar"
$$foo = "baz"
you write
mydict = {}
foo = "bar"
mydict[foo] = "baz"
This way you won't accidentally overwrite previously existing variables (which is the security aspect) and you can have different "namespaces".
Use globals() (disclaimer: this is a bad practice, but is the most straightforward answer to your question, please use other data structure as in the accepted answer).
You can actually assign variables to global scope dynamically, for instance, if you want 10 variables that can be accessed on a global scope i_1, i_2 ... i_10:
for i in range(10):
globals()['i_{}'.format(i)] = 'a'
This will assign 'a' to all of these 10 variables, of course you can change the value dynamically as well. All of these variables can be accessed now like other globally declared variable:
>>> i_5
'a'
Instead of a dictionary you can also use namedtuple from the collections module, which makes access easier.
For example:
# using dictionary
variables = {}
variables["first"] = 34
variables["second"] = 45
print(variables["first"], variables["second"])
# using namedtuple
Variables = namedtuple('Variables', ['first', 'second'])
v = Variables(34, 45)
print(v.first, v.second)
The SimpleNamespace class could be used to create new attributes with setattr, or subclass SimpleNamespace and create your own function to add new attribute names (variables).
from types import SimpleNamespace
variables = {"b":"B","c":"C"}
a = SimpleNamespace(**variables)
setattr(a,"g","G")
a.g = "G+"
something = a.a
If you don't want to use any object, you can still use setattr() inside your current module:
import sys
current_module = module = sys.modules[__name__] # i.e the "file" where your code is written
setattr(current_module, 'variable_name', 15) # 15 is the value you assign to the var
print(variable_name) # >>> 15, created from a string
You have to use globals() built in method to achieve that behaviour:
def var_of_var(k, v):
globals()[k] = v
print variable_name # NameError: name 'variable_name' is not defined
some_name = 'variable_name'
globals()[some_name] = 123
print(variable_name) # 123
some_name = 'variable_name2'
var_of_var(some_name, 456)
print(variable_name2) # 456
Variable variables in Python
"""
<?php
$a = 'hello';
$e = 'wow'
?>
<?php
$$a = 'world';
?>
<?php
echo "$a ${$a}\n";
echo "$a ${$a[1]}\n";
?>
<?php
echo "$a $hello";
?>
"""
a = 'hello' #<?php $a = 'hello'; ?>
e = 'wow' #<?php $e = 'wow'; ?>
vars()[a] = 'world' #<?php $$a = 'world'; ?>
print(a, vars()[a]) #<?php echo "$a ${$a}\n"; ?>
print(a, vars()[vars()['a'][1]]) #<?php echo "$a ${$a[1]}\n"; ?>
print(a, hello) #<?php echo "$a $hello"; ?>
Output:
hello world
hello wow
hello world
Using globals(), locals(), or vars() will produce the same results
#<?php $a = 'hello'; ?>
#<?php $e = 'wow'; ?>
#<?php $$a = 'world'; ?>
#<?php echo "$a ${$a}\n"; ?>
#<?php echo "$a ${$a[1]}\n"; ?>
#<?php echo "$a $hello"; ?>
print('locals():\n')
a = 'hello'
e = 'wow'
locals()[a] = 'world'
print(a, locals()[a])
print(a, locals()[locals()['a'][1]])
print(a, hello)
print('\n\nglobals():\n')
a = 'hello'
e = 'wow'
globals()[a] = 'world'
print(a, globals()[a])
print(a, globals()[globals()['a'][1]])
print(a, hello)
Output:
locals():
hello world
hello wow
hello world
globals():
hello world
hello wow
hello world
Bonus (creating variables from strings)
# Python 2.7.16 (default, Jul 13 2019, 16:01:51)
# [GCC 8.3.0] on linux2
Creating variables and unpacking tuple:
g = globals()
listB = []
for i in range(10):
g["num%s" % i] = i ** 10
listB.append("num{0}".format(i))
def printNum():
print "Printing num0 to num9:"
for i in range(10):
print "num%s = " % i,
print g["num%s" % i]
printNum()
listA = []
for i in range(10):
listA.append(i)
listA = tuple(listA)
print listA, '"Tuple to unpack"'
listB = str(str(listB).strip("[]").replace("'", "") + " = listA")
print listB
exec listB
printNum()
Output:
Printing num0 to num9:
num0 = 0
num1 = 1
num2 = 1024
num3 = 59049
num4 = 1048576
num5 = 9765625
num6 = 60466176
num7 = 282475249
num8 = 1073741824
num9 = 3486784401
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9) "Tuple to unpack"
num0, num1, num2, num3, num4, num5, num6, num7, num8, num9 = listA
Printing num0 to num9:
num0 = 0
num1 = 1
num2 = 2
num3 = 3
num4 = 4
num5 = 5
num6 = 6
num7 = 7
num8 = 8
num9 = 9
I'm answering the question How to get the value of a variable given its name in a string?
which is closed as a duplicate with a link to this question. (Editor's note: It is now closed as a duplicate of How can I select a variable by (string) name?)
If the variables in question are part of an object (part of a class for example) then some useful functions to achieve exactly that are hasattr, getattr, and setattr.
So for example you can have:
class Variables(object):
def __init__(self):
self.foo = "initial_variable"
def create_new_var(self, name, value):
setattr(self, name, value)
def get_var(self, name):
if hasattr(self, name):
return getattr(self, name)
else:
raise "Class does not have a variable named: " + name
Then you can do:
>>> v = Variables()
>>> v.get_var("foo")
'initial_variable'
>>> v.create_new_var(v.foo, "is actually not initial")
>>> v.initial_variable
'is actually not initial'
I have tried both in python 3.7.3, you can use either globals() or vars()
>>> food #Error
>>> milkshake #Error
>>> food="bread"
>>> drink="milkshake"
>>> globals()[food] = "strawberry flavor"
>>> vars()[drink] = "chocolate flavor"
>>> bread
'strawberry flavor'
>>> milkshake
'chocolate flavor'
>>> globals()[drink]
'chocolate flavor'
>>> vars()[food]
'strawberry flavor'
Reference:
https://www.daniweb.com/programming/software-development/threads/111526/setting-a-string-as-a-variable-name#post548936
The consensus is to use a dictionary for this - see the other answers. This is a good idea for most cases, however, there are many aspects arising from this:
you'll yourself be responsible for this dictionary, including garbage collection (of in-dict variables) etc.
there's either no locality or globality for variable variables, it depends on the globality of the dictionary
if you want to rename a variable name, you'll have to do it manually
however, you are much more flexible, e.g.
you can decide to overwrite existing variables or ...
... choose to implement const variables
to raise an exception on overwriting for different types
etc.
That said, I've implemented a variable variables manager-class which provides some of the above ideas. It works for python 2 and 3.
You'd use the class like this:
from variableVariablesManager import VariableVariablesManager
myVars = VariableVariablesManager()
myVars['test'] = 25
print(myVars['test'])
# define a const variable
myVars.defineConstVariable('myconst', 13)
try:
myVars['myconst'] = 14 # <- this raises an error, since 'myconst' must not be changed
print("not allowed")
except AttributeError as e:
pass
# rename a variable
myVars.renameVariable('myconst', 'myconstOther')
# preserve locality
def testLocalVar():
myVars = VariableVariablesManager()
myVars['test'] = 13
print("inside function myVars['test']:", myVars['test'])
testLocalVar()
print("outside function myVars['test']:", myVars['test'])
# define a global variable
myVars.defineGlobalVariable('globalVar', 12)
def testGlobalVar():
myVars = VariableVariablesManager()
print("inside function myVars['globalVar']:", myVars['globalVar'])
myVars['globalVar'] = 13
print("inside function myVars['globalVar'] (having been changed):", myVars['globalVar'])
testGlobalVar()
print("outside function myVars['globalVar']:", myVars['globalVar'])
If you wish to allow overwriting of variables with the same type only:
myVars = VariableVariablesManager(enforceSameTypeOnOverride = True)
myVars['test'] = 25
myVars['test'] = "Cat" # <- raises Exception (different type on overwriting)
Any set of variables can also be wrapped up in a class.
"Variable" variables may be added to the class instance during runtime by directly accessing the built-in dictionary through __dict__ attribute.
The following code defines Variables class, which adds variables (in this case attributes) to its instance during the construction. Variable names are taken from a specified list (which, for example, could have been generated by program code):
# some list of variable names
L = ['a', 'b', 'c']
class Variables:
def __init__(self, L):
for item in L:
self.__dict__[item] = 100
v = Variables(L)
print(v.a, v.b, v.c)
#will produce 100 100 100
It should be extremely risky...
but you can use exec():
a = 'b=5'
exec(a)
c = b*2
print (c)
Result:
10
The setattr() method sets the value of the specified attribute of the specified object.
Syntax goes like this –
setattr(object, name, value)
Example –
setattr(self,id,123)
which is equivalent to self.id = 123
As you might have observed, setattr() expects an object to be passed along with the value to generate/modify a new attribute.
We can use setattr() with a workaround to be able to use within modules. Here’ how –
import sys
x = "pikachu"
value = 46
thismodule = sys.modules[__name__]
setattr(thismodule, x, value)
print(pikachu)
I'm trying to use an SQLite database via SQLAlchemy 1.4 ORM to filter data based on relationships.
Data
My example data consists of groups and members in a simple many-to-many schema:
[...]
member_f = Member(name="F")
group_1 = Group(name="1", members=[member_a, member_b, member_c]) <-- find this group via member names
group_2 = Group(name="2", members=[member_a, member_b, member_f])
group_3 = Group(name="3", members=[member_a, member_c, member_d])
group_4 = Group(name="4", members=[member_d, member_e, member_f])
[...]
Full running example code (schema, objects, queries):
from sqlalchemy import Table, Column, Integer, String, ForeignKey
from sqlalchemy import create_engine, select, func, text, and_
from sqlalchemy.orm import relationship, declarative_base, sessionmaker
engine = create_engine("sqlite+pysqlite:///:memory:", future=True, echo=False)
Session = sessionmaker(bind=engine, future=True)
# Schema
Base = declarative_base()
groups_members = Table("groups_members", Base.metadata,
Column("group_id", ForeignKey("groups.id")),
Column("member_name", ForeignKey("members.name")),
)
class Group(Base):
__tablename__ = "groups"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String, nullable=False)
members = relationship("Member", secondary=groups_members, backref="groups", lazy="subquery")
def __repr__(self):
return f"<Group: {self.name}>"
class Member(Base):
__tablename__ = "members"
name = Column(String, primary_key=True)
def __repr__(self):
return f"<Member: {self.name}>"
Base.metadata.create_all(engine)
# Objects
member_a = Member(name="A")
member_b = Member(name="B")
member_c = Member(name="C")
member_d = Member(name="D")
member_e = Member(name="E")
member_f = Member(name="F")
group_1 = Group(name="1", members=[member_a, member_b, member_c])
group_2 = Group(name="2", members=[member_a, member_b, member_f])
group_3 = Group(name="3", members=[member_a, member_c, member_d])
group_4 = Group(name="4", members=[member_d, member_e, member_f])
print(f"{member_a}: {member_a.groups}") # OK
with Session() as session:
session.add(group_1)
session.add(group_2)
session.add(group_3)
session.add(group_4)
session.commit()
print(session.query(Group).all()) # OK
# Query users example
def get_members_in_more_than_2_groups():
with Session() as session:
return session.execute(
select(Member, func.count(groups_members.columns.group_id).label('group_members_count'))
.join(groups_members)
.group_by(Member.name)
.having(text('group_members_count > 2'))
).all()
for m in get_members_in_more_than_2_groups():
print(m) # OK
# Query groups problem: associated with A and B but not with E or F
def get_groups_by_member_names(member_names_included, member_names_excluded):
with Session() as session:
included = session.execute(select(Member).where(Member.name.in_(member_names_included))).all()
excluded = session.execute(select(Member).where(Member.name.in_(member_names_excluded))).all()
return session.execute(
select(Group)
.join(Group.members)
.where(
and_(
Group.members.contains(included),
~Group.members.contains(excluded),
)
)
.group_by(Group.id)
).scalars().all()
for g in get_groups_by_member_names(member_names_included=["A", "B"], member_names_excluded=["E", "F"]):
print(g) # Expected output: <Group: 1>
Goal
Now I'm trying to find all groups that
have both members with the names A and B (that's groups 1 and 2)
and don't have any member named E or F (removing group 2)
resulting in just group 1.
Problem
The relevant (and failing) function in the example code is get_groups_by_member_names and with my lack of database knowledge, I'm quite stuck.
Most existing questions that I could find on SO only need to filter by one relationship value. But I need them to consider the lists of included and excluded member names.
I have tried to get the members as SQLAlchemy objects first and inserting those into the query but without any luck. I may have done that completely wrong, though.
I also tried joining the tables, filtering with the names list and counting the grouped results... It's hard for me to tell whether I'm on the right track or not at all.
Running over all groups in Python and applying the filtering there would be my fallback workaround. But with many items, the database can probably handle it more efficiently.
Any help greatly appreciated, I am happy with anything that works. I could probably also work my way up from a functioning SQL statement.
Thanks for your time!
Edit 1:
I found this answer https://stackoverflow.com/a/21104689/5123171 and while it works on small data sets, it's terribly slow on larger ones (about 60 seconds for 500 members and 10k groups):
def get_group_by_members(member_names_included, member_names_excluded):
with Session() as session:
return session.query(Group).join(groups_members).filter(
groups_members.columns.member_name.in_(member_names_included)).group_by(Group.id).having(func.count(groups_members.columns.member_name) == len(member_names_included),
).filter(
~Group.members.any(Member.name.in_(member_names_excluded)),
).all()
Ok, here's what I ended up with via trial & error in an SQL editor. This is faster than the previous attempt (50 milliseconds on the same data).
The comments in the code correspond to the following steps:
Find groups containing excluded members
Filter those out of the assignment table
Filter remaining assignment table by included members
Group by group IDs
Return all remaining groups matching the number of included members
SQL
SELECT * FROM groups
WHERE id IN (
SELECT group_id FROM groups_members
WHERE member_name IN ("A", "B") // 3.
AND group_id NOT IN ( // 2
SELECT group_id FROM groups_members
WHERE member_name IN ("E", "F") // 1.
)
GROUP BY group_id // 4.
HAVING count(member_name) == 2 // 5.
)
SQLAlchemy
session.query(Group)
.where(Group.id.in_(
session.query(groups_members.c.group_id)
.where(
groups_members.c.member_name.in_(member_names_included), # 3.
groups_members.c.group_id.not_in( # 2.
session.query(groups_members.c.group_id).where(
groups_members.c.member_name.in_(member_names_excluded) # 1.
)
),
)
.group_by(groups_members.c.group_id) # 4.
.having(
func.count(groups_members.c.member_name)
== len(member_names_included) # 5.
))
)
.all()
And the full running example in one piece:
from sqlalchemy import Table, Column, Integer, String, ForeignKey
from sqlalchemy import create_engine, select, func, text, and_, not_
from sqlalchemy.orm import relationship, declarative_base, sessionmaker
engine = create_engine("sqlite+pysqlite:///:memory:", future=True, echo=False)
Session = sessionmaker(bind=engine, future=True)
# Schema
Base = declarative_base()
groups_members = Table(
"groups_members",
Base.metadata,
Column("group_id", ForeignKey("groups.id")),
Column("member_name", ForeignKey("members.name")),
)
class Group(Base):
__tablename__ = "groups"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String, nullable=False)
members = relationship(
"Member", secondary=groups_members, backref="groups", lazy="subquery"
)
def __repr__(self):
return f"<Group {self.id}: {self.name}>"
class Member(Base):
__tablename__ = "members"
name = Column(String, primary_key=True)
def __repr__(self):
return f"<Member: {self.name}>"
Base.metadata.create_all(engine)
# Objects
member_a = Member(name="A")
member_b = Member(name="B")
member_c = Member(name="C")
member_d = Member(name="D")
member_e = Member(name="E")
member_f = Member(name="F")
group_1 = Group(name="1", members=[member_a, member_b, member_c])
group_2 = Group(name="2", members=[member_a, member_b, member_f])
group_3 = Group(name="3", members=[member_a, member_c, member_d])
group_4 = Group(name="4", members=[member_d, member_e, member_f])
with Session() as session:
session.add(group_1)
session.add(group_2)
session.add(group_3)
session.add(group_4)
session.commit()
# Query
member_names_included = ["A", "B"]
member_names_excluded = ["E", "F"]
# Raw SQL variant
with Session() as session:
result = session.execute(
f"""
SELECT * FROM groups
WHERE id IN (
SELECT group_id FROM groups_members
WHERE member_name IN ("A", "B")
AND group_id NOT IN (
SELECT group_id FROM groups_members
WHERE member_name IN ("E", "F")
)
GROUP BY group_id
HAVING count(member_name) == 2
)
"""
).all()
groups = [Group(**r) for r in result]
for r in groups:
print(f'SQL {r}')
# ORM Variant
with Session() as session:
result = (
session.query(Group)
.where(
Group.id.in_(
session.query(groups_members.c.group_id)
.where(
# Matching any included members
groups_members.c.member_name.in_(member_names_included),
# Removing any groups containing excluded members
groups_members.c.group_id.not_in(
session.query(groups_members.c.group_id).where(
groups_members.c.member_name.in_(member_names_excluded)
)
),
)
# This is to make sure that all included members exist in a group, not just a few
.group_by(groups_members.c.group_id)
.having(
func.count(groups_members.c.member_name)
== len(member_names_included)
)
)
)
.all()
)
for r in result:
print(f'ORM {r}')
I hope this is helpful to anyone and if you have suggestions for improvement, please let me know.
I have this query:
UPDATE test SET cucc = 'akarmi' WHERE id IN (1, 2)
Laravel equivalent:
DB::table('test')
->whereIn('id', [1, 2])
->update(['cucc' => 'akarmi']);
But i'd like to append new values to column cucc:
UPDATE test SET cucc = CONCAT(cucc, 'akarmi') WHERE id IN (1, 2)
Laravel equivalent:
DB::table('test')
->whereIn('id', [1, 2])
->update(['cucc' => DB::raw("CONCAT(cucc, 'plus')")]);
But if the 'plus' string comes from a user input variable, it need to escape. I tried this:
DB::table('test')
->whereIn('id', [1, 2])
->update(['cucc' => DB::raw('CONCAT(cucc, ?)', 'plus')]);
But DB::raw not supports replace ? to escaped value.
How can I solve this?
DB::raw() doesnt have a parameter for bindings but the query builder does
DB::table('test')
->whereRaw('id IN (?,?)')
->setBindings(['plus', 1, 2])
->update(['cucc' => DB::raw('CONCAT(cucc, ?)')]);
You can also use the bindings parameter of the update() method
(should work but didnt test it)
DB::table('test')
->whereIn('id', [1, 2])
->update(['cucc' => DB::raw('CONCAT(cucc, ?)')], ['plus']);
I wrote a function for found all pois around a track
controller :
def index
#track = Track.friendly.find(params[:track_id])
#tracks = Track.where(way_id: #track.id)
#way = Way.find(1)
#poi_start = Poi.find(#way.point_start)
#pois = #track.pois.sleepsAndtowns
#pois = #way.poi_around_track_from(#poi_start, 50000, #pois)
end
way.rb
def poi_around_track_from(poi, dist, pois)
around_sql = <<-SQL
SELECT
ST_DWithin(
ST_LineSubstring(
way.path,
ST_LineLocatePoint(way.path, pta.lonlat::geometry) + #{dist} / ST_Length(way.path::geography),
ST_LineLocatePoint(way.path, pta.lonlat::geometry) + 100000 / ST_Length(way.path::geography)
),
ptb.lonlat,
2000) is true as pois
FROM ways way, pois pta, pois ptb
WHERE way.id = #{self.id}
and pta.id = #{poi.id}
and ptb.id = #{pois.ids}
SQL
Poi.find_by_sql(around_sql).pois
end
This function return :
syntax error at or near "["
LINE 13: and ptb.id = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
What's wrong, how can I fix it ?
Since you are using standard sql to build the query, (not the ActiveRecord), you will have to use the standard IN clues with where
It looks like pois.ids is returning an array, so, you will have to turn it to a string in the format as below
[1,2] #=> (1,2)
Change,
WHERE way.id = #{self.id}
and pta.id = #{poi.id}
and ptb.id = #{pois.ids}
to
WHERE way.id = #{self.id}
and pta.id = #{poi.id}
and ptb.id IN (#{pois.ids.join(',')})
You can change pois.ids as #semeera207 wrote to string or go another way and compare ptb.id to pois.ids as an array.
WHERE way.id = #{self.id}
and pta.id = #{poi.id}
and array[ptb.id] && #{pois.ids}
To make it faster create gin index
Create index on pois using gin((array[id]));