I'm trying to prototype using the SmartRedis Python client to interact with the SmartSim Orchestrator. Is it possible to launch the orchestrator without any other models in the experiment? If so, what would be the best way to do so?
It is entirely possible to do that. A SmartSim Experiment can contain different types of 'entities' including Models, Ensembles (i.e. groups of Models), and Orchestrator (i.e. the Redis-backed database). None of these entities, however, are 'required' to be in the Experiment.
Here's a short script that creates an experiment which includes only a database.
from SmartSim import Experiment
NUM_DB_NODES = 3
exp = Experiment("Database Only")
db = exp.create_database(db_nodes=NUM_DB_NODES)
exp.generate(db)
exp.start(db)
After this, the Orchestrator (with the number of shards specified by NUM_DB_NODES) will have been spunup. You can then connect the Python client using the following line:
client = smartredis.Client(db.get_address()[0],NUM_DB_NODES>1)
Related
I have a celery app which has to be pinged by another app. This other app uses json to serialize celery task parameters, but my app has a custom serialization protocol. When the other app tries to ping my app (app.control.ping), it throws the following error:
"Celery ping failed: Refusing to deserialize untrusted content of type application/x-stjson (application/x-stjson)"
My whole codebase relies on this custom encoding, so I was wondering if there is a way to configure a json serialization but only for this ping, and to continue using the custom encoding for the other tasks.
These are the relevant celery settings:
accept_content = [CUSTOM_CELERY_SERIALIZATION, "json"]
result_accept_content = [CUSTOM_CELERY_SERIALIZATION, "json"]
result_serializer = CUSTOM_CELERY_SERIALIZATION
task_serializer = CUSTOM_CELERY_SERIALIZATION
event_serializer = CUSTOM_CELERY_SERIALIZATION
Changing any of the last 3 to [CUSTOM_CELERY_SERIALIZATION, "json"] causes the app to crash, so that's not an option.
Specs: celery=5.1.2
python: 3.8
OS: Linux docker container
Any help would be much appreciated.
Changing any of the last 3 to [CUSTOM_CELERY_SERIALIZATION, "json"] causes the app to crash, so that's not an option.
Because result_serializer, task_serializer, and event_serializer doesn't accept list but just a single str value, unlike e.g. accept_content
The list for e.g. accept_content is possible because if there are 2 items, we can check if the type of an incoming request is one of the 2 items. It isn't possible for e.g. result_serializer because if there were 2 items, then what should be chosen for the result of task-A? (thus the need for a single value)
This means that if you set result_serializer = 'json', this will have a global effect where all the results of all tasks (the returned value of the tasks which can be retrieved by calling e.g. response.get()) would be serialized/deserialized using the json-serializer. Thus, it might work for the ping but it might not for the tasks that can't be directly serialized/deserialized to/from JSON which really needs the custom stjson-serializer.
Currently with Celery==5.1.2, it seems that task-specific setting of result_serializer isn't possible, thus we can't set a single task to be encoded in 'json' and not 'stjson' without setting it globally for all, I assume the same case applies to ping.
Open request to add result_serializer option for tasks
A short discussion in another question
Not the best solution but a workaround is that instead of fixing it in your app's side, you may opt to just add support to serialize/deserialize the contents of type 'application/x-stjson' in the other app.
other_app/celery.py
import ast
from celery import Celery
from kombu.serialization import register
# This is just a possible implementation. Replace with the actual serializer/deserializer for stjson in your app.
def stjson_encoder(obj):
return str(obj)
def stjson_decoder(obj):
obj = ast.literal_eval(obj)
return obj
register(
'stjson',
stjson_encoder,
stjson_decoder,
content_type='application/x-stjson',
content_encoding='utf-8',
)
app = Celery('other_app')
app.conf.update(
accept_content=['json', 'stjson'],
)
You app remains to accept and respond stjson format, but now the other app is configured to be able to parse such format.
I have a multi-tenancy structure set up where each client has a schema set up for them. The structure mirrors the "parent" schema, so any migration that happens needs to happen for each schema identically.
I am using Flask-Script with Flask-Migrate to handle migrations.
What I tried so far is iterating over my schema names, building a URI for them, scoping a new db.session with the engine generated from the URI, and finally running the upgrade function from flask_migrate.
#manager.command
def upgrade_all_clients():
clients = clients_model.query.all()
for c in clients:
application.extensions["migrate"].migrate.db.session.close_all()
application.extensions["migrate"].migrate.db.session = db.create_scoped_session(
options={
"bind": create_engine(generateURIForSchema(c.subdomain)),
"binds": {},
}
)
upgrade()
return
I am not entirely sure why this doesn't work, but the result is that it only runs the migration for the db that was set up when the application starts.
My theory is that I am not changing the session that was originally set up when the manager script runs.
Is there a better way to migrate each of these schemas without setting multiple binds and using the --multidb parameter? I don't think I can use SQLALCHEMY_BINDS in the config since these schemas need to be able to be dynamically created/destroyed.
For those who are encountering the same issue, the answer to my specific situation was incredibly simple.
#manager.command
def upgrade_all_clients():
clients = clients_model.query.all()
for c in clients:
print("Upgrading client '{}'...".format(c.subdomain))
db.engine.url.database = c.subdomain
_upgrade()
return
The database attribute of the db.engine.url is what targets the schema. I don't know if this is the best way to solve this, but it does work and I can migrate each schema individually.
I'm working on integrating an experiment in psychopy with the eyelink eyetracking system. The way to do this seems to be through pylink. Unfortunately I'm really unfamiliar with pylink and I was hoping there was a sample of an experiment that combines the two. I haven't been able to find one. If anyone would be able to share an example or point me towards a more accessible manual than the pylink api that sr-research provides I'd be really grateful.
Thanks!
I am glad you found your solution. I have not used iohub, but we do use psychopy and an eyelink and therefore some of the following code may be of use to others who wish to invoke more direct communication. Note that our computers use Archlinux. If none of the following makes any sense to you, don't worry about it, but maybe it will help others who are stumbling along the same path we are.
Communication between experimental machine and eye tracker machine
First, you have to establish communication with the eyelink. If your experimental machine is turned on and plugged into a live Eyelink computer then on linux you have to first set your ethernet card up, and then set the default address that Eyelink uses (this also works for the Eyelink 1000 - they kept the same address). Note your ethernet will probably have a different name than enp4s0. Try simply with ip link and look for something similar. NB: these commands are being typed into a terminal.
#To set up connection with Eyelink II computer:
#ip link set enp4s0 up
#ip addr add 100.1.1.2/24 dev enp4s0
Eyetracker functions
We have found it convenient to write some functions for talking to the Eyelink computer. For example:
Initialize Eyetracker
sp refers to the tuple of screenx, screeny sizes.
def eyeTrkInit (sp):
el = pl.EyeLink()
el.sendCommand("screen_pixel_coords = 0 0 %d %d" %sp)
el.sendMessage("DISPLAY_COORDS 0 0 %d %d" %sp)
el.sendCommand("select_parser_configuration 0")
el.sendCommand("scene_camera_gazemap = NO")
el.sendCommand("pupil_size_diameter = %s"%("YES"))
return(el)
NB: the pl function comes from import pylink as pl. Also, note that there is another python library called pylink that you can find on line. It is probably not the one you want. Go through the Eyelink forum and get pylink from there. It is old, but it still works.
Calibrate Eyetracker
el is the name of the eyetracker object initialized above. sp screen size, and cd is color depth, e.g. 32.
def eyeTrkCalib (el,sp,cd):
pl.openGraphics(sp,cd)
pl.setCalibrationColors((255,255,255),(0,0,0))
pl.setTargetSize(int(sp[0]/70), int(sp[1]/300))
pl.setCalibrationSounds("","","")
pl.setDriftCorrectSounds("","off","off")
el.doTrackerSetup()
pl.closeGraphics()
#el.setOfflineMode()
Open datafile
You can talk to the eye tracker and do things like opening a file
def eyeTrkOpenEDF (dfn,el):
el.openDataFile(dfn + '.EDF')
Drift correction
Or drift correct
def driftCor(el,sp,cd):
blockLabel=psychopy.visual.TextStim(expWin,text="Press the space bar to begin drift correction",pos=[0,0], color="white", bold=True,alignHoriz="center",height=0.5)
notdone=True
while notdone:
blockLabel.draw()
expWin.flip()
if keyState[key.SPACE] == True:
eyeTrkCalib(el,sp,cd)
expWin.winHandle.activate()
keyState[key.SPACE] = False
notdone=False
Sending and getting messages.
There are a number of built-in variables you can set, or you can add your own. Here is an example of sending a message from your python program to the eyelink
eyelink.sendMessage("TRIALID "+str(trialnum))
eyelink.startRecording(1,1,1,1)
eyelink.sendMessage("FIX1")
tFix1On=expClock.getTime()
Gaze contingent programming
Here is a portion of some code that uses the eyelink's most recent sample in the logic of the experimental program.
while notdone:
if recalib==True:
dict['recalib']=True
eyelink.sendMessage("RECALIB END")
eyelink.startRecording(1,1,1,1)
recalib=False
eventType=eyelink.getNextData()
if eventType==pl.STARTFIX or eventType==pl.FIXUPDATE or eventType==pl.ENDFIX:
sample=eyelink.getNewestSample()
if sample != None:
if sample.isRightSample():
gazePos = sample.getRightEye().getGaze()
if sample.isLeftSample():
gazePos = sample.getLeftEye().getGaze()
gazePosCorFix = [gazePos[0]-scrx/2,-(gazePos[1]-scry/2)]
posPix = posToPix(fixation)
eucDistFix = sqrt((gazePosCorFix[0]-posPix[0])**2+(gazePosCorFix[1]-posPix[1])**2)
if eucDistFix < tolFix:
core.wait(timeFix1)
notdone=False
eyelink.resetData()
break
Happy Hacking.
rather than PyLink, you might want to look into using the ioHub system within PsychoPy. This is a more general-purpose eye tracking system that also allows for saving data in a common format (integrated with PsychoPy events), and provides tools for data analysis and visualisation.
ioHUb is built to be agnostic to the particular eye tracker you are using. You just need to create a configuration file specific to your EyeLink system, and thereafter use the generic functions ioHiv provides for calibration, accessing gaze data in real-time, and so on.
There are some teaching resources accessible here: http://www.psychopy.org/resources/ECEM_Python_materials.zip
For future readers, I wanted to share my library for combining pylink and psychopy. I've recently updated it to work with python 3. It provides simple to use, high level functions.
https://github.com/colinquirk/templateexperiments/tree/master/eyelinker
You could also work at a lower level with the PsychoPyCustomDisplay class (see the pylink docs for more info about EyeLinkCustomDisplay).
For an example of it in use, see:
https://github.com/colinquirk/ChangeDetectionEyeTracking
(At the time of writing, this experiment code is not yet python 3 ready, but it should still be a useful example.)
The repo also includes other modules for creating experiments and recording EEG data, but they are not necessary if you are just interested in the eyelinker code.
I'm using mongodb to store some data. Then I have a function that gets the object with the latest timestamp and one with the oldest. I haven't experienced any issues during development or production with this method but when I try to implement a test for it the test fails approx 20% of the times. I'm using rspec to test this method and I'm not using mongoid or mongomapper. I create three objects with different timestamps but get a nil response since my dataset contains 0 objects. I have read a lot of articles about write_concern and that it might be the problem with "unsafe writes" but I have tried almost all the different combinations with these parameters (w, fsync, j, wtimeout) without any success. Does anyone have any idea how to solve this issue? Perhaps I have focused too much with the write_concern track and that the problems lies somewhere else.
This is the method that fetches the latest and oldest timestamp.
def first_and_last_timestamp(customer_id, system_id)
last = collection(customer_id).
find({sid:system_id}).
sort(["t",Mongo::DESCENDING]).
limit(1).next()
first = collection(customer_id).
find({sid:system_id}).
sort(["t",Mongo::ASCENDING]).
limit(1).next()
{ min: first["t"], max: last["t"] }
end
Im inserting data using this method where data is a json object.
def insert(customer_id, data)
collection(customer_id).insert(data)
end
I have reverted back to use the default for setting up my connection
Mongo::MongoClient.new(mongo_host, mongo_port)
I'm using the gem mongo (1.10.2). I'm not using any fancy setup for my mongo database. I've just installed mongo using brew on my mac and started it. The version of my mongo database is v2.6.1.
I've used the following command to generate a topic model from some documents:
bin/mallet train-topics --input topic-input.mallet --num-topics 100 --output-state topic-state.gz
I have not, however, used the --output-model option to generate a serialized topic trainer object. Is there any way I can use the state file to infer topics for new documents? Training is slow, and it'll take a couple of days for me to retrain, if I have to create the serialized model from scratch.
We did not use the command line tools shipped with mallet, we just use the mallet api to create the serialized model for inferences of the new document. Two point need special notice:
You need serialize out the pipes you used just after you finish the training (For my case, it is SerialPipes)
And of cause the model need also to be serialized after you finish the training(For my case, it is ParallelTopicModel)
Please check with the java doc:
http://mallet.cs.umass.edu/api/cc/mallet/pipe/SerialPipes.html
http://mallet.cs.umass.edu/api/cc/mallet/topics/ParallelTopicModel.html
Restoring a model from the state file appears to be a new feature in mallet 2.0.7 according to the release notes.
Ability to restore models from gzipped "state" files. From the new
TopicTrainer, use the --input-state [filename] argument. Note that you
can manually edit this file. Any token with topic set to -1 will be
immediately resampled upon loading.
If you mean you want to see how new documents fit into a previously trained topic model, then I'm afraid there is no simple command you can use to do it right.
The class cc.mallet.topics.LDA in mallet 2.0.7's source code provides such a utility, try to understand it and use it in your program.
P.S., If my memory serves, there is some problem with the implementation of the function in that class:
public void addDocuments(InstanceList additionalDocuments,
int numIterations, int showTopicsInterval,
int outputModelInterval, String outputModelFilename,
Randoms r)
You have to rewrite it.