Understanding PsychoPy's data logging - psychopy

Question

I have a test PsychoPy Builder script that I am using to investigate some counter-intuitive behaviour. The structure is four routines:
"Init", not in a loop, the following code in "Begin Experiment":
x = 0
y = 0
z = 0
foo = [0, 0, 0]
"One", in a loop, the following code in "End Routine":
x = x + 1
foo[0] = foo[0] + 1
thisExp.addData("x", x)
thisExp.addData("y", y)
thisExp.addData("z", z)
thisExp.addData("foo", foo)
"Two", in a loop, the following code in "End Routine":
y = y + 2
foo[1] = foo[1] + 2
thisExp.addData("x", x)
thisExp.addData("y", y)
thisExp.addData("z", z)
thisExp.addData("fooY", foo[1])
thisExp.addData("foo", foo)
"Three", in a loop, the following code in "End Routine":
z = z + 3
foo[2] = foo[2] + 3
thisExp.addData("x", x)
thisExp.addData("y", y)
thisExp.addData("z", z)
thisExp.addData("foo", foo)
There is no other code, no other components. The routines "One", "Two", and "Three" form a loop in that order executed five times. The relevant columns of the CSV output file are as follows:
trials.thisRepN trials.thisTrialN trials.thisN trials.thisIndex x y z foo fooY
0 0 0 0 1 2 3 [5, 10, 15] 2
1 0 1 0 2 4 6 [5, 10, 15] 4
2 0 2 0 3 6 9 [5, 10, 15] 6
3 0 3 0 4 8 12 [5, 10, 15] 8
4 0 4 0 5 10 15 [5, 10, 15] 10
Is this the expected output? If so, why? Note that the individual variables, x, y, and z, are displaying updated values each time through the loop (at the end of the loop), while the list foo shows only the final value after the loop iterates all five times, but it shows this in every line. But calling out individual elements of the list displays as individual variables do.
What is the logic and rationale behind this?
Is there a way to make the list output perform as the others do?
Is there a way to force the output to capture/display any of these variables as they are when the addData() is invoked rather than waiting until the end of the loop?

Answer 1

I think I know what is going wrong here. It's probably because python assigns by reference rather than copy. This is explained in detail elsewhere but briefly,
original = [1, 2]
new = original # new is simply a reference to original! It is not a copy.
new[0] = 'Oops' # original is now ['Oops', 2] as is new (which is just a reference or pointer
In your case, the TrialHandler receives the reference, which simply points to the "foo" variable which is updated throughout the experiment. Since the log is only saved in the end of the experiment, all the rows in "foo" now points to the "foo variable" which now holds the value [5, 10, 15].
This assignment-by-reference can be extremely beautiful and handy, but sometimes cause headache like in your example. It applies to all python mutables: lists, dicts, functions, and classes. But not for immutables, like numbers, tuples and strings! That's why your script works for digits but not for the list.
There are different solutions. The simplest is probably to replace the addData calls with thisExp.addData("foo", tuple(foo)) which converts the mutable list to an immutable tuple. One can also do thisExp.addData("foo", [x for x in foo]). A more all-round solution for all kinds of objects is to run import copy in the beginning of the experiment and then add data like thisExp.addData("foo", copy.copy(foo)) in the other codeblocks (if you have a complicated object, use copy.deepcopy instead).