I'm writting in a h5py file in parallel using mpi4py. I notice that the programs seems to freeze when i try to close the h5file. Do you have an idea what can cause this issue?
Thanks a lot
I process as follow :
`f = h5py.File('tttt.hdf5', 'w',driver= 'mpio', comm=Comm,libver='latest')
f.create_dataset('Spect',shape = (100,),maxshape=(100000,),dtype='f', chunks = True)
###### spect calculation ##########
###### spect recording ############
f['Spect'].resize(np.array(Spect.shape))
f['Spect'][:] = Spect
print fclosing
f.close() # Here the program seems to be stuck
print "finish" # doesn't appear in the control `
Related
Python v3.5, Windows 10
I'm using multiple processes and trying to captures user input. Searching everything I see there are odd things that happen when using input() with multiple processes. After 8 hours+ of trying, nothing I implement worked, I'm positive I am doing it wrong but I can't for the life of me figure it out.
The following is a very stripped down program that demonstrates the issue. Now it works fine when I run this program within PyCharm, but when I use pyinstaller to create a single executable it fails. The program constantly is stuck in a loop asking the user to enter something as shown below:.
I am pretty sure it has to do with how Windows takes in standard input from things I've read. I've also tried passing the user input variables as Queue() items to the functions but the same issue. I read you should put input() in the main python process so I did that under if __name__ = '__main__':
from multiprocessing import Process
import time
def func_1(duration_1):
while duration_1 >= 0:
time.sleep(1)
print('Duration_1: %d %s' % (duration_1, 's'))
duration_1 -= 1
def func_2(duration_2):
while duration_2 >= 0:
time.sleep(1)
print('Duration_2: %d %s' % (duration_2, 's'))
duration_2 -= 1
if __name__ == '__main__':
# func_1 user input
while True:
duration_1 = input('Enter a positive integer.')
if duration_1.isdigit():
duration_1 = int(duration_1)
break
else:
print('**Only positive integers accepted**')
continue
# func_2 user input
while True:
duration_2 = input('Enter a positive integer.')
if duration_2.isdigit():
duration_2 = int(duration_2)
break
else:
print('**Only positive integers accepted**')
continue
p1 = Process(target=func_1, args=(duration_1,))
p2 = Process(target=func_2, args=(duration_2,))
p1.start()
p2.start()
p1.join()
p2.join()
You need to use multiprocessing.freeze_support() when you produce a Windows executable with PyInstaller.
Straight out from the docs:
multiprocessing.freeze_support()
Add support for when a program which uses multiprocessing has been frozen to produce a Windows executable. (Has been tested with py2exe, PyInstaller and cx_Freeze.)
One needs to call this function straight after the if name == 'main' line of the main module. For example:
from multiprocessing import Process, freeze_support
def f():
print('hello world!')
if __name__ == '__main__':
freeze_support()
Process(target=f).start()
If the freeze_support() line is omitted then trying to run the frozen executable will raise RuntimeError.
Calling freeze_support() has no effect when invoked on any operating system other than Windows. In addition, if the module is being run normally by the Python interpreter on Windows (the program has not been frozen), then freeze_support() has no effect.
In your example you also have unnecessary code duplication you should tackle.
The Tello sdk has the streamon and streamoff commands. These start and end the 720p video stream. Is there a way to get a 5 megapixel image through the sdk, like when you put the control app into photo mode? I'm guessing if such a way exists, it is not documented.
while there isn't a UDP command that you can send to the drone to switch, there is some python code that you could try in order to make it work. and to make something like that work easily without coding it over again putting it in a function and calling it like "takePhoto()" is the best solution I have. Here's some code:
# # simple ex_4: Tello_photo
# <\>
from djitellopy import tello
import cv2
me = tello.Tello()
me.connect()
print(me.get_battery())
def TakePhoto(me):
me.streamoff()
me.streamon()
img = me.get_frame_read().frame
# print(img)
# img = cv2.resize(img, (360, 240)) # optional
cv2.imshow("photo", img) # optional display photo
cv2.waitKey(0)
# # example condition, it can also just be called normally like "TakePhoto"
"""
if (1 + 1) == 2:
TakePhoto(me)
"""
# # another example, with an actual used if statement
"""
bat = me.get_battery()
if bat < 50:
TakePhoto(me)
else:
print("battery too low (for ex)")
"""
# # for now, we'll call it just like this for testing:
TakePhoto(me)
# </>
the code was tested on windows 10 - pycharm 2021.2 > python3.x >> opencv 4.5.1.48
I hope this helped and I hope it works for you.
I have developed a machine learning python script (let's call it classify_obj written with python 3.6) that imports TensorFlow. It was developed initially for bulk analysis but now I find the need to run this script repeatedly on smaller datasets to cater for more real time usage. I am doing this on Linux RH7.
Process Flow:
Master tool (written in Java) call classify_obj with object input to categorize.
classify_obj generates the classification result as a csv (takes about 7-10s)
Master tool reads the result from #2
Master tool proceeds to do other logic
Repeat #1 with next object input
To breakdown the time taken, I switched off the main logic and just do the modules import without performing any other action. I found that the import takes about 4-5s out of the 7-10s run time on the small dataset. The classification takes about 2s. I am also looking at other ways to reduce the run time for other areas but the bulk seems to be from the import.
Import time: 4-6s
Classify time: 1s
Read, write and other logic time: 0.2s
I am thinking what options are there to reduce the import time?
One idea I had was to modify the classify_obj into a "stay alive" process. The master tool after completing all its activity will stop this process/service. The intent (not sure if this would be the case) is that all the required libraries are already loaded during the process start and when the master tool calls that process/service, it will only incur the classification time instead of needing to import the libraries repeated.
What do you think about this? Also how can I set this up on Linux RHEL 7.4? Some reference links would be greatly appreciated.
Other suggestion would be greatly appreciated.
Thanks and have a great day!
This is the solution I designed to achieve the above.
Reference: https://realpython.com/python-sockets/
I have to create 2 scripts.
1. client python script: Used to pass the raw data to be classified to the server python script using socket programming.
server python script: Loads the keras (tensorflow) lib and model at launch. Continues to stay alive until a 'stop' request from client (to exit the while loop). When the client script sends the data to the server script, server script will process the incoming data and return a ok/not ok output back to the client script.
In the end, the classification time is reduced to 0.1 - 0.3s.
Client Script
import socket
import argparse
from argparse import ArgumentParser
def main():
parser = ArgumentParser(description='XXXXX')
parser.add_argument('-i','--input', default='NA', help='Input txt file path')
parser.add_argument('-o','--output', default='NA', help='Output csv path with class')
parser.add_argument('-stop','--stop', default='no', help='Stop the server script')
args = parser.parse_args()
str = args.input + ',' + args.output + ',' + args.stop
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 65432 # The port used by the server
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
bytedata = str.encode()
sock.send(bytedata)
data = sock.recv(1024)
print('Received', data)
if __name__== "__main__":
main()
Server Script
def main():
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Port to listen on (non-privileged ports are > 1023)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST,PORT))
sock.listen(5)
stop_process = 'no'
while (stop_process == 'no'):
# print('Waiting for connection')
conn, addr = sock.accept()
data = ''
try:
# print('Connected by', addr)
while True:
data = conn.recv(1024)
if data:
stop_process = process_input(data) # process_input function processes incoming data. If client sends 'yes' for the stop argument, the stop_process variable will be set to 'yes' by the function.
byte_reply = stop_process.encode()
conn.sendall(byte_reply) # send reply back to client
else:
break
conn.close()
# print('Closing connection',addr)
finally:
conn.close()
if __name__== "__main__":
main()
I am looking for a way to print 3D pdf from the results ABAQUS/Viewer. This will make it easy to communicate the results with others who are interested in the results of simulation but do not have access to ABAQUS.
The best way is to export a vrml file and convert it using Tetra4D or pdf3D and Adobe Acrobat professional. The 3D pdfs can look very good. However, the commercial software would cost over £800 per year. I did create a Python script to create a 3D pdf directly from Abaqus/CAE & Viewer which uses 2 open source tools: 1) Meshlab (http://www.meshlab.net/) to create a U3D file, 2) MiKTeX (https://miktex.org/) to convert the U3D file into a pdf. The output is not as polished as Tetra4D but it works. I have not tried this with the latest version of Meshlab. Just run this script from Abaqus/CAE or Abaqus/Viewer.
# Abaqus CAE/Viewer Python Script to create a 3D pdf directly from Abaqus/CAE or Abaqus/Viewer.
# You must first install meshlab (meshlabserver.exe)and MiKTeX (pdflatex.exe)
# Edit this script to reflect the installed locations of meshlabserver.exe and pdflatex.exe
# It will export a stl or obj file the mesh of current viewport and convert into 3D pdf
# Or run in Abaqus/viewer and it will create a VRML file and convert to 3D pdf.
# If contours are displayed in Abaqus Viewer, then it will create a contour 3D pdf
from abaqus import *
from abaqusConstants import *
from viewerModules import *
import os
import subprocess
import sys
# -----------------------------------------------------------------------------
pdfName='try'
meshlab_path="C:/Program Files/VCG/MeshLab/meshlabserver.exe"
pdfLatex_path="C:/Program Files (x86)/MiKTeX 2.9/miktex/bin/pdflatex.exe"
# -----------------------------------------------------------------------------
currView=session.viewports[session.currentViewportName]
try: # for Abaqus Viewer
cOdbD=currView.odbDisplay
odb = session.odbs[cOdbD.name]
name=odb.name.split(r'/')[-1].replace('.odb','')
module='Vis'
except: # Abaqus CAE
#name=currView.displayedObject.modelName
import stlExport_kernel
name = repr(currView.displayedObject).split('[')[-1].split(']')[0][1:-1] # allows for either main or visulation modules
module='CAE'
print module
if module=='CAE':
#All instances must be meshed
cOdbD=None
try:
ext='.stl'
stlExport_kernel.STLExport(moduleName='Assembly', stlFileName=pdfName + ext, stlFileType='BINARY')
except:
try:
ext='.obj'
session.writeOBJFile(fileName=os.path.join(directory,pdfName + ext), canvasObjects= (currView, ))
except:
print 'Either your assembly is not fully meshed or something else'
directory=(os.getcwd())
else: # Abaqus/Viewer
if cOdbD.viewCut:
session.graphicsOptions.setValues(antiAlias=OFF) # Better with anti aliasing off
odb = session.odbs[cOdbD.name]
directory=odb.path.replace(odb.path.split('/')[-1],'').replace('/','\\')
# Turn off most of the stuff in the viewport
currView.viewportAnnotationOptions.setValues(triad=OFF,
legend=OFF, title=OFF, state=OFF, annotations=OFF, compass=OFF)
ext='.wrl'
session.writeVrmlFile(fileName=os.path.join(directory,pdfName + ext),
compression=0, canvasObjects= (currView, ))
pdfFilePath=os.path.join(directory,pdfName+'-out.pdf')
if os.path.isfile(pdfFilePath):
os.remove(pdfFilePath)
#Check file was deleted
if os.path.isfile(pdfFilePath):
print "Aborted because pdf file of same name cant be deleted. Please close programs which it might be open in"
1/0 #a dodgy way to exit program
# Invoke meshlab to convert to a .u3d file
if cOdbD: #If in Abaqus/viewer
if 'CONTOURS' in repr(cOdbD.display.plotState[0]): # If contours are displayed. Output contoured pdf
p=subprocess.Popen([meshlab_path,'-i',pdfName + ext, '-o',pdfName + '.u3d','-m','vc']) #'vn fn fc vt'
else:
p=subprocess.Popen([meshlab_path,'-i',pdfName + ext, '-o',pdfName + '.u3d'])
else:
p=subprocess.Popen([meshlab_path,'-i',pdfName + ext, '-o',pdfName + '.u3d'])
p.communicate() # Wait for meshlab to finish
file_fullPathName=os.path.join(directory, pdfName + '.tex')
#Read the .tex file which meshlab has just created
with open(file_fullPathName, 'r') as texFile:
lines = texFile.read()
#Edit the .tex file
lines=lines.replace("\usepackage[3D]{movie15}","\\usepackage[3D]{movie15}\n\\usepackage[margin=-2.2in]{geometry}")
if cOdbD:
if 'CONTOURS' in repr(cOdbD.display.plotState[0]):
lines=lines.replace("3Dlights=CAD,","3Dlights=CAD,\n\t3Drender=SolidWireframe,")
lines=lines.replace("\n\end{document}","{---------------------------------------------------------------------------------Click above! MB1 - rotate, MB2 wheel or MB3 - zoom, Ctrl-MB1 - pan--------------}\n\\end{document}")
file_fullPathName=os.path.join(directory, pdfName + '-out.tex')
with open(file_fullPathName, "w") as outp:
outp.write(lines)
p=subprocess.Popen([
pdfLatex_path,
pdfName + '-out.tex',
])
p.communicate()
print 'Conversion to pdf complete'
print file_fullPathName
The simplest way of printing the Abaqus *.odb results are using Tecplot 360 which is read the Abaqus *.odb files and you can get the *.tif and *.png results with any resolutions and you can also rotate the model in 3D and change the fonts and all the things you need.
I have a little issue with matplotlib. While trying to save some figures in a loop hundred times with savefig(), it finally ends with the following error:
IOError: [Errno 24] Too many open files: 'test_421.png'
I have checked that I close every figure after saving it, but it doesn't seem to be efficient.
Here is a code sample to illustrate how I get this error:
def displayFig(input, display, savePath):
fig = plt.figure(figsize=(22, 5))
# add some subplots, axis and colorbars...
if display:
plt.show()
else:
plt.savefig(savePath, dpi = 100)
plt.close(fig)
for i, inp in enumerate(inpArray):
savePath = 'test_%d.png' % i
displayFig(inp, False, savePath)
Why is this error raised? Could it be a multi-threading issue or a memory leak?
Try upgrading your version of matplotlib. I believe from matplotlib v1.2.0 this was addressed (by this PR https://github.com/matplotlib/matplotlib/issues/1466/).