In simulating a trajectory for a OpenAI gym environment, such as the Mujoco Walker2d, one feeds the current observation and action into the gym step function to produce the next observation. However, there appears to be no way render a given trajectory of observations only (this is all it needs for rendering)! Does anyone know how to do this, or if it is even possible?
Related
I wanted to use the contextual bandit agents (LinearThompson Sampling agent) in the tf_Agents.
I am using a custom environment and my rewards are delayed by 3 days. Hence for training, the observations are generated from the saved historical tables (predictions generated 3 days ago) and their corresponding rewards (Also in the table).
Given this, only during training, how do I make the policy to output an action, for a given observation, from the historical tables? And during evaluation I want the policy to behave the usual way, generating the actions using the policy it learned from.
Looks like I need to write a custom policy, that behaves in a way during training and behaves it's usual self (linearthompsonsampling.policy) during evaluation. Unfortunately I couldn't find any examples or documentation for this usecase. Can someone please explain how to code this - an example would be very useful
I am carrying out a task which I need to know which is the next element based on a list.
Real example:
I have a list of triangles with BLUE and PINK colors of different sizes and these sizes can be alternated.
What I need is to predict what the color of the next triangle will be based on the current list passed. (the size would be great).
I tried to search for Watson AI, TensorFlow and etc. but I couldn't find anything.
I am new to this area and I am looking for that learning.
Could someone indicate something or a set of things that I can use to get closer to that result?
If possible send me an example code or something like that..
If the shapes and colours are random, then it's just guesswork, and a random number generator is probably the best you are going to get.
If there is a limited set, like a pack of cards, then you could use probabilities to guess what is next.
If there is an unlimited set and always a defined pattern where the next colour can be predicted, then you might be able to use a machine learning model. To get started just explore the web and find a getting started with machine learning tutorial. They will be python biased, and scikit-learn is probably all you need.
I want to train a reinforcement learning agent on a model which i build in OpenModelica. By using pyFMI, it is no problem to import the FMU, simulate it, and get some results.
My problem is that i don´t have a possibility to "pause" a simulation after each step, getting the states, feeding my RL-agent with it and returning his proposed action as an input.
ModelicaGym seems to be a way to solve this problem by starting a simulation, stopping, getting the results, defining the next action and starting the simulation again with the last end-time as starting time.
Reading a paper from Lund University (https://portal.research.lu.se/portal/files/7201641/pyfmi_tech.pdf) made me think about an other idea:
Creating a FMU with the Learner, and connecting the two FMUs via PyFMI.Master.
Something along these lines:
from pyfmi import load_fmu
from pyfmi.master import Master
controller = load_fmu("controller.fmu")
Circuit = load_fmu("circuit.fmu")
connections = [( Circuit ,"currentSensor1.i",controller ,"feedback1.u2"),
(controller ,"PID.y",Circuit ,"signalVoltage1.v")]
models = [Circuit , controller]
master_simulator = Master(models , connections)
res = master_simulator.simulate(final_time =1)
Controlling the circuit with an other FMU with a PID controller inside works, but is it possible to create a FMU with a Reinforcement Learning Agent, including all other requiered Libraries, packages (Keras, Tensorflow?)
According to my point of view, such an implementation could have a pretty good performance, especially for models and learners with a higher complexity, this could be an interesting approach.
Or am I just chasing some dreams, because implementing a Reinforcement Learning algorithm in a FMU is not possible or causing other troubles?
Actually, i was a little surprised of not finding other people trying to implement this.
Best regards
Henrik
This answer might be plentifully late, but nevertheless I found your question during my research for the exact same problem. Your question is - to my understanding - taken up in the paper
Integration and Evaluation of Deep Reinforcement Learning Controller in a Building CoSimulation Environment
[PDF found here]
However in our context, the co-simulation environment
that is used for the study of Cyber-Physical Systems is a
master of co-simulation, and need to be able to call AI based Python libraries or tools from within the co-simulation. In the co-simulation, the controller requires an
interaction with the controlled component at each
iteration. Therefore, it is necessary to generate the AI based control components in the form of an FMU as is the
case for the other components.
They used a tool called DACCOSIM NG but later introduced their own methodology to make this approach more streamline.
Hope you long since found your own solution.
maybe you can update your question so it is more clear how the learning agent is implemented, but I understand that it can be used from Python?
The example fmu_with_input_function.py from the PyFMI documentation illustrates how to use a function as input an FMU. I suppose you can retrieve information from the FMU in this function like so (untested pseudo code):
from pyfmi import load_fmu
define input_object(model):
response = model.get('response_variable_name')
return ('input_var_name', learner(response))
model = load_fmu('model.fmu')
res = model.simulate(final_time=30, input=input_object(model))
You have to set up your model FMU so that the variables your learner should change (input_var_name), are input variables or tunable parameters. If you use parameters without variability="tunable", you cannot change them in the course of the simulation.
I would first try with input variables, because tunable parameters are a bit more complicated to treat and might not be implemented correctly in some tools.
Machine learning and deep learning model I know how to code but not know how to do it in real time like stock market and get real time predicted value.
This is a simple toy example. Let us assume that for stock market prediction, you use a fixed time window, e.g the past 10 candlesticks. You query the data, pass it to the model and make your prediction (for example predicting the next 5 candle sticks). When the 11th candle stick appears in real time, query again the previous 10 candle sticks [2nd-11th] and make predictions for the next 5. You can adjust overlapping predictions by averaging them for instance.
This scheme would work for machine learning and deep learning. Of course that is just a toy example, hence there are more sophisticated of doing so. So get creative and read a bunch of research papers to see how it is done in the industry.
I'm participating in small data analysis competition in our school.
We use Fitbit wearable devices, which is loaned to each participants by host of contest.
For 2 months during the contest, they walk and sleep with this small device 24/7,
allow it to gather data about participant's walk count with heart rate(bpm), etc.
and we need to solve some problems based on these participants' data
like, example,
show the relations between rainy days and participants' working out rate using the chart,
i think purpose of problem is,
because of rain, lot of participants are expected to be at home.
can you show some cause and effect numerically?
i'm now studying python library numpy, pandas with ipython notebook.
but still i have no idea about solving these problems..
could you recommend some projects or sites use for references? i really eager to win this competition.:(
and lastly, sorry for my poor English.
Thank you.
that's a fun project. I'm working on something kind of similar.
Here's what you need to do:
Learn the fitbit API and stream the data from the fitbit accelerometer and gyroscope. If you can combine this with heart rate data, great. The more types of data you have, the more effective your algorithm will be. You can store this data in a simple csv file (streaming the accel/gyro data at 50Hz is recommended). Or setup a web server and store it in a database for easy access
Learn how to use pandas and scikit learn
[optional but recommended]: Learn matplotlib so you can graph you data and get a feel for how it looks
Load the data into pandas and create features on the data - notably using 1-2 second sliding window analysis with 50% overlap. Good features include (for all three Accel X, Y, Z): max, min, standard deviation, root mean square, root sum square and tilt. Polynomials will help.
Since this is a supervised classification problem, you will need to create some labelled data - so do this manually (state 1 = rainy day, state 2 = non-rainy day) and then train a classification algorithm. I would recommend a random forest
Test using unlabeled data - don't forget to use cross validation
Voila, you now have a highly accurate model and will win the competition. Plus you've learned about a bunch of really cool Python and machine learning stuff.
For more tutorials on how all this stuff works, I'd highly recommend the Kaggle tutorial projects
BONUS: If you want to take it to a new level, you can start adding smoothers on top of your classifier, for example by using a Hidden Markov Model as explained in this talk
BONUS 2: Go get a PhD in Human Activity Recognition.