I mean to get a series of snapshots, at a sequence of time steps, of a layout with two views (one RenderView + one LineChartView).
For this I put together a script, see below.
I do
ssh -X myserver
and there I run
~/ParaView-5.4.1-Qt5-OpenGL2-MPI-Linux-64bit/bin/pvbatch myscript.py
The script is extremely slow to run. I conceive the following reasons/bottlenecks:
Communication of the graphic part (ssh -X) from the remote server to my computer.
Display of graphics in my computer.
Processing in the server.
Is there a way to assess which is the bottleneck, with my current resources?
(For instance, I know I could get a faster communication to assess item 1, but I cannot do that now.)
Is there a way to accelerate pvbatch?
The answer likely depends on my system, but perhaps there are generic actions I can take.
Creation of the layout with two views
...
ans = GetAnimationScene()
time_steps = ans.TimeKeeper.TimestepValues
for istep in range(len(time_steps)) :
tstep = time_steps[istep]
ans.AnimationTime = tstep
fname = "combo" + '-' + '{:08d}'.format(istep) + '.png'
print( "Exporting image " + fname + " for time step " + str(tstep) )
SaveScreenshot(fname, viewLayout1, quality=100)
Why do you need the -X ?
Just set DISPLAY to :0 and do not forward graphics.
The bottleneck is most likely the rendering on your local display. If your server has a X server, you can perform the rendering on your server by setting accordingly the DISPLAY environnement variable as Mathieu explained.
If your server does not have a X server running, then the best option is to build Paraview on your server using either the OSMesa backend or the EGL backend (if you have a compatible graphic card on it).
Related
I've three different machine learning models in python. To improve performance, I run them on different terminals in parallel. They are communicating and sharing data with one another through files. These models are creating batches of files to make available for other. All the processes are running in parallel but dependent on data prepared by other process. Once a process A prepares a batch of data, it creates a file to give signal to other process that data is ready, then process B starts processing it, while looking for other batch too simultaneously. How can this huge data be shared with next process without creating files? Is there any better way to communicate among these processes without creating/deleting temporary files in python?
Thanks
You could consider running up a small Redis instance... a very fast, in-memory data structure server.
It allows you to share strings, lists, queues, hashes, atomic integers, sets, ordered sets between processes very simply.
As it is networked, you can share all these data structures not only within a single machine, but across multiple machines.
As it has bindings for C/C++, Python, bash, Ruby, Perl and so on, it also means you can use the shell, for example, to quickly inject commands/data into your app to change its behaviour, or get debugging insight by looking at how variables are set.
Here's an example of how to do multiprocessing in Python3. Instead of storing results in a file the results are stored in a dictionary (see output)
from multiprocessing import Pool, cpu_count
def multi_processor(function_name):
file_list = []
# Test, put 6 strings in the list so your_function should run six times
# with 6 processors in parallel, (assuming your CPU has enough cores)
file_list.append("test1")
file_list.append("test2")
file_list.append("test3")
file_list.append("test4")
file_list.append("test5")
file_list.append("test6")
# Use max number of system processors - 1
pool = Pool(processes=cpu_count()-1)
pool.daemon = True
results = {}
# for every item in the file_list, start a new process
for aud_file in file_list:
results[aud_file] = pool.apply_async(your_function, args=("arg1", "arg2"))
# Wait for all processes to finish before proceeding
pool.close()
pool.join()
# Results and any errors are returned
return {your_function: result.get() for your_function, result in results.items()}
def your_function(arg1, arg2):
try:
print("put your stuff in this function")
your_results = ""
return your_results
except Exception as e:
return str(e)
if __name__ == "__main__":
some_results = multi_processor("your_function")
print(some_results)
The output is
put your stuff in this function
put your stuff in this function
put your stuff in this function
put your stuff in this function
put your stuff in this function
put your stuff in this function
{'test1': '', 'test2': '', 'test3': '', 'test4': '', 'test5': '', 'test6': ''}
Try using a sqlite database to share files.
I made this for this exact purpose:
https://pypi.org/project/keyvalue-sqlite/
You can use it like this:
from keyvalue_sqlite import KeyValueSqlite
DB_PATH = '/path/to/db.sqlite'
db = KeyValueSqlite(DB_PATH, 'table-name')
# Now use standard dictionary operators
db.set_default('0', '1')
actual_value = db.get('0')
assert '1' == actual_value
db.set_default('0', '2')
assert '1' == db.get('0')
I am trying to establish a bluetooth serial communication link between a Raspberry Pi Zero W, running Raspbian Jessie [03-07-2017], and an Arduino (UNO).
I am currently able to write data to the Arduino using bluetoothctl.
The application requires that we are able to write data to a particular BLE Slave. There are multiple [HM-10] Slaves to switch between, the Slave needs to be chosen during the program execution.
There is no BAUD rate preference. Currently, we are using 9600 universally.
Functions have been created that automatically connect and then write data to an "attribute", this shows up as data on the Serial Monitor of the Arduino.
Python Code - using BlueZ 5.44 (manually installed):
import subprocess
from subprocess import Popen, PIPE
# Replaces the ':' with '_' to allow the MacAddress to be in the form
# of a "Path" when "selecting an attribute"
def changeMacAddr(word):
return ''.join(c if c != ':' else '_' for c in word)
# Connects to a given MacAddress and then selects the attribute to write to
def connBT(BTsubProcess, stringMacAddr):
BTsubProcess.stdin.write(bytes("".join("connect "+stringMacAddr +"\n"), "utf-8"))
BTsubProcess.stdin.flush()
time.sleep(2)
stringFormat = changeMacAddr(stringMacAddr)
BTsubProcess.stdin.write(bytes("".join("select-attribute /org/bluez/hci0/dev_"
+ stringFormat +
"/service0010/char0011" + "\n"), "utf-8"))
BTsubProcess.stdin.flush()
# Can only be run once connBT has run - writes the data in a list [must have numbers 0 - 255 ]
def writeBT(BTsubProcess, listOfData):
stringList = [str('{0} ').format(elem) for elem in listOfData]
BTsubProcess.stdin.write(bytes("".join("write " + "".join(stringList) + "\n"), "utf-8"))
BTsubProcess.stdin.flush()
# Disconnects
def clostBT(BTsubProcess):
BTsubProcess.communicate(bytes("disconnect\n", "utf-8"))
# To use the functions a subprocess "instance" of bluetoothctl must be made
blt = subprocess.Popen(["bluetoothctl"], stdin=subprocess.PIPE, shell=True)
# blt with then be passed into the function for BTsubProcess
# Note: the MacAddresses of the Bluetooth modules were pre-connected and trusted manually via bluetoothctl
This method works fine for small sets of data, but my requirements require me to stream data to the Arduino very quickly.
The current set up is:
Sensor data (accelerometer, EEG) via USB serial is received by the Pi
The Pi processes the data
Commands are then sent to the Arduino via the in built bluetooth of the Pi Zero W
However, while using this method the bluetooth data transmission would delay (temporarily freeze) when the sensor data changed.
The data transmission was flawless when using two pre-paired HM-10 modules, the Pi's GPIO serial port was configured using PySerial.
The following methods have also been tried:
Using WiringPi to set-up a bluetooth serial port on the /dev/ttyAMA0
using Python sockets and rfcomm
When attempting to use both of these methods. The Python code compiles, however, once the Serial Port is opened the data is seemingly not written and does not show up on the Arduino's Serial Monitor.
This then cripples the previous functions. Even when using bluetoothctl manually, the module cannot be unpaired/disconnected. Writing to the appropriate attribute does not work either.
A restart is required to regain normal function.
Is this approach correct?
Is there a better way to send data over BLE?
UPDATE: 05/07/2017
I am no longer working on this project. But troubleshooting has led me to believe that a "race condition" in the code may have led to the program not functioning as intended.
This was verified during the testing phase where a more barebones code was created that functioned very well.
I am trying to understand, how pipe lining in redis works? According to one blog I read, For this code
Pipeline pipeline = jedis.pipelined();
long start = System.currentTimeMillis();
for (int i = 0; i < 100000; i++) {
pipeline.set("" + i, "" + i);
}
List<Object> results = pipeline.execute();
Every call to pipeline.set() effectively sends the SET command to Redis (you can easily see this by setting a breakpoint inside the loop and querying Redis with redis-cli). The call to pipeline.execute() is when the reading of all the pending responses happens.
So basically, when we use pipe-lining, when we execute any command like set above, the command gets executed on the server but we don't collect the response until we executed, pipeline.execute().
However, according to the documentation of pyredis,
Pipelines are a subclass of the base Redis class that provide support for buffering multiple commands to the server in a single request.
I think, this implies that, we use pipelining, all the commands are buffered and are sent to the server, when we execute pipe.execute(), so this behaviour is different from the behaviour described above.
Could someone please tell me what is the right behaviour when using pyreids?
This is not just a redis-py thing. In Redis, pipelining always means buffering a set of commands and then sending them to the server all at once. The main point of pipelining is to avoid extraneous network back-and-forths-- frequently the bottleneck when running commands against Redis. If each command were sent to Redis before the pipeline was run, this would not be the case.
You can test this in practice. Open up python and:
import redis
r = redis.Redis()
p = r.pipeline()
p.set('blah', 'foo') # this buffers the command. it is not yet run.
r.get('blah') # pipeline hasn't been run, so this returns nothing.
p.execute()
r.get('blah') # now that we've run the pipeline, this returns "foo".
I did run the test that you described from the blog, and I could not reproduce the behaviour.
Setting breakpoints in the for loop, and running
redis-cli info | grep keys
does not show the size increasing after every set command.
Speaking of which, the code you pasted seems to be Java using Jedis (which I also used).
And in the test I ran, and according to the documentation, there is no method execute() in jedis but an exec() and sync() one.
I did see the values being set in redis after the sync() command.
Besides, this question seems to go with the pyredis documentation.
Finally, the redis documentation itself focuses on networking optimization (Quoting the example)
This time we are not paying the cost of RTT for every call, but just one time for the three commands.
P.S. Could you get the link to the blog you read?
Is it possible to have one Redis Lua script hit more than one database? I currently have information of one type in DB 0 and information of another type in DB 1. My normal workflow is doing updates on DB 1 based on an API call along with meta information from DB 0. I'd love to do everything in one Lua script, but can't figure out how to hit multiple dbs. I'm doing this in Python using redis-py:
lua_script(keys=some_keys,
args=some_args,
client=some_client)
Since the client implies a specific db, I'm stuck. Ideas?
It is usually a wrong idea to put related data in different Redis databases. There is almost no benefit compared to defining namespaces by key naming conventions (no extra granularity regarding security, persistence, expiration management, etc ...). And a major drawback is the clients have to manually handle the selection of the correct database, which is error prone for clients targeting multiple databases at the same time.
Now, if you still want to use multiple databases, there is a way to make it work with redis-py and Lua scripting.
redis-py does not define a wrapper for the SELECT command (normally used to switch the current database), because of the underlying thread-safe connection pool implementation. But nothing prevents you to call SELECT from a Lua script.
Consider the following example:
$ redis-cli
SELECT 0
SET mykey db0
SELECT 1
SET mykey db1
The following script displays the value of mykey in the 2 databases from the same client connection.
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
r = redis.Redis(connection_pool=pool)
lua1 = """
redis.call("select", ARGV[1])
return redis.call("get",KEYS[1])
"""
script1 = r.register_script(lua1)
lua2 = """
redis.call("select", ARGV[1])
local ret = redis.call("get",KEYS[1])
redis.call("select", ARGV[2])
return ret
"""
script2 = r.register_script(lua2)
print r.get("mykey")
print script2( keys=["mykey"], args = [1,0] )
print r.get("mykey"), "ok"
print
print r.get("mykey")
print script1( keys=["mykey"], args = [1] )
print r.get("mykey"), "misleading !!!"
Script lua1 is naive: it just selects a given database before returning the value. Its usage is misleading, because after its execution, the current database associated to the connection has changed. Don't do this.
Script lua2 is much better. It takes the target database and the current database as parameters. It makes sure that the current database is reactivated before the end of the script, so that next command applied on the connection still run in the correct database.
Unfortunately, there is no command to guess the current database in the Lua script, so the client has to provide it systematically. Please note the Lua script must reset the current database at the end whatever happens (even in case of previous error), so it makes complex scripts cumbersome and awkward.
Following code works on my pc but gives error on other pc's. how is it possible to run this successfully on all machines.
QSqlQuery query;
QString queryString = "SELECT * FROM " + parameter3->toAscii() + " WHERE " + parameter1->toAscii() + " = \"" + parameter2->toAscii() + "\"";
bool retX = query.exec(queryString);
What pre requisite should be fulfilled for this to run on any pc
In troubleshooting, if you isolate your query and it returns the result you anticipated ( such as you have done utilizing qt creator to verify the query returns a result of true), the next step would be to take a close look at your code and verify that you are passing the proper parameters into the query for execution.
I have a virgin machine I utilize for this purpose. I am a software engineer by trade and I am fully aware that i have a ton of software installed on my PC which the common user may/will not have installed. So the virgin allows me to test the code in stand-alone form.
I suggest implementing a message box prior to the execution of your query which shows the query to be executed. This will verify the query is correct on the "other machines".
Certain dll's were needed. in my case qtguid4.dll, qtcored4.dll and qtsqld4.dll. There was a size difference. Once matched it worked on a pc. However, on other pc's i still get an error "The application failed to initialize 0xc000007b ....."
How is it possible to make an application run.
Brgds,
kNish