I am currently reading the documentation for numpy, however to get a more thorough understanding of the library, it would be helpful if there was a way to debug the workflow of the library as I call a particular function.
I have tried debugging when numpy was imported as a third party module. However, when I try to step into it, it is actually stepping over.
Therefore, I am building it from source and thereby trying to build it locally in an attempt to run it.
I find the documentation provided in the numpy website for developers to be a bit vague for beginners like me.
I would highly appreciate any comments that would set me on the right path, as I have tried everything that I know of.
Thanks!
I am currently reading the documentation for numpy, however to get a more thorough understanding of the library, it would be helpful if there was a way to debug the workflow of the library as I call a particular function.
Unless you plan to fix a bug in Numpy, help Numpy developpers or you are a contributor, you should not debug Numpy directly.
I have tried debugging when numpy was imported as a third party module. However, when I try to step into it, it is actually stepping over.
By default, Numpy enable compiler optimizations like -O2 or -O3 or even using annotations in the code so to tell the compiler to use a given optimization level (so to better vectorize it for example). Such optimizations tends to make debugging harder and unreliable. The maximal optimization level for debugging should be -Og and the minimal one is -O0. Using -O1/-O2/-O3 tends to causes issues. You also need to enable debugging informations with -g.
The standard way to run and debug Numpy is to use gdb --args python runtests.py -g --python mytest.py. The -g flag should compile Numpy with compiling options -O0 -ggdb. Adding --debug-info may help you to understand if everything is built correctly. For more information see this and that. You can also see the above informations in the runtests.py script.
If you still have issues with the above method, the last desperate option is to add printf directly in the code (and take care to flush stdout frequently). It is not very clean and force Numpy to be frequently recompiled which is a bit slow but it is a pretty good solution when gdb is unstable (ie. crashes or just bogus) for example.
Thank you for contributing to Numpy.
Related
I am new to LLVM backend. What I am trying to do is inserting several register loading and storing between different instructions. So I create a machine function pass to do this job. When I disable the optimisation via -O0, everything works fine.
However, when I enable the optimisation, I found that the code has been optimised in a wrong way. For example, some labels of -O0 code has been kept.
For example it still keeps jl label1; while label1 is not exist in -O3 code.
I am trying to figure out a way to bypass it. My thought is that maybe we can decide the execution order of Machine Function Pass and I could then run my Pass at the very end. Or maybe there is other ways bypass it?
I have been searching it for a while, but I didn't find anything useful.
Thanks for your kind helps !
I'm working on a very demanding project (actually an interpreter), exclusively written in D, and I'm wondering what type of optimizations would generally be recommended. The project makes heavy use of GC, classes, asssociative arrays, and pretty much anything.
Regarding compilation, I've already experimented both with DMD and LDC flags and LDC with -flto=full -O3 -Os -boundscheck=off seems to be making a difference.
However, as rudimentary as this may sound, I would like you to suggest anything that comes to your mind that could help speed up the performance, related or not to the D language. (I'm sure I'm missing several things).
Compiler flags: I would add -mcpu=native if the program will be running on your machine. Not sure what effect -Os has in addition to -O3.
Profiling has been mentioned in comments. Personally under Linux I have a script which dumps a process's stack trace and I do that a few times to get an idea of where it's getting hung up on.
Not sure what you mean by GS.
Since you mentioned classes: in D, methods are virtual by default; virtual methods add indirections and are not inlineable. Make sure only those methods that must be virtual are. See if you can rewrite your program using a form of polymorphism that doesn't involve indirections, such as using template metaprogramming.
Since you mentioned associative arrays: these make heavy use of the GC; to speed them up, switch to a third-party library that works on top of std.allocator, such as https://github.com/dlang-community/containers
If some parts of your code are parallelizable, std.parallelism is a good tool for this.
Since you mentioned that the project is an interpreter: there are many avenues for optimizing them, up to JIT/AOT compilation. Perhaps you could link to an existing library such as LLVM or libjit.
I'm using scipy.ndimage.zoom and I get this annoying warning:
UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
I'm not sure what I should get from it, I started using it with SciPy 1.0.0 so I don't believe it really affects me.
I guess calling it UserWarning is a bit questionable given it's not intended for user consumption, but maybe the intended user is the developer importing the library.
I'm using multiprocessing and I get one warning per process, even more annoying.
Is there a sane way to silent it?
It was easier than I thought, leaving the question for future reference in case anyone needs this.
import warnings
warnings.filterwarnings('ignore', '.*output shape of zoom.*')
Your proposed solution did not work for me. But what does work is:
import warnings
# other nice code
with warnings.catch_warnings():
warnings.simplefilter("ignore")
x = scipy.ndimage.interpolation.zoom(...)
Seems like a silly question, but I'm scratching my head about this.
Jython 2.7.0 "final release", platform W7
I simply want to run my app so that __debug__ is False and so that assert statements are skipped.
I tried going
jython -O my_app.py
and it said
Unknown option: O
... did some googling but to no avail...
anyone seeking the answer
https://www.safaribooksonline.com/library/view/jython-essentials/9781449397364/apds02.html
Major Design Differences
Debug and optimization
Jython does not currently recognize the CPython command-line switch -O, which removes assert statements and other debug
information. In Jython, the __debug__ global value, which controls
this behavior, is always set to 1 and is not writable.
... so (for the moment) you have to implement your own (switch-offable) assert-style mechanisms. Potentially somewhat problematic, because even having a boolean "is_production_run" is going to take time to evaluate, though for all I know this might just be one machine instruction.
I wonder if the Jython high command have got any plans to implement this feature one day? I can't imagine it would be that complicated...
I sometimes test Python modules as I develop them by running a Python interactive prompt in a terminal, importing my new module and testing out the functionality. Of course, since my code is in development there are bugs, and frequent restarts of the interpreter are required. This isn't too painful when I've only executed a couple of interpreter lines before restarting: my key sequence when the interpreter restart looks like Up Up Enter Up Up Enter... but extrapolate it to 5 or more statements to be repeated and it gets seriously painful!
Of course I could put my test code into a script which I execute with python -i, but this is such a scratch activity that it doesn't seem quite "above threshold" for opening a text editor :) What I'm really pining for is the Ctrl-r behaviour from the bash shell: executing a sequence of 10 commands in sequence in bash involves finding the command in history (repeated Up or Ctrl-r for a search -- both work in the Python interpreter shell) and then just pressing Ctrl-o ten times. One of my favourite bash shell features.
The problem is that while lots of other readline binding functionality like Ctrl-a, Ctrl-e, Ctrl-r, and Ctrl-s work in the Python interpreter, Ctrl-o does not. I've not been able to find any references to this online, although perhaps the readline module can be used to add this functionality to the python prompt. Any suggestions?
Edit: Yes, I know that using the interactive interpreter is not a development methodology that scales beyond a few lines! But it is convenient for small tests, and IMO the interactiveness can help to work out whether a developing API is natural and convenient, or too heavy. So please confine the answers to the technical question of whether readline history-stepping can be made to work in python, rather than the side-opinion of whether one should or shouldn't choose to (sometimes) work this way!
Edit: Since posting I realised that I am already using the readline module to make some Python interpreter history functions work. But the Ctrl-o binding to the operate-and-get-next readline command doesn't seem to be supported, even if I put readline.parse_and_bind("Control-o: operate-and-get-next") in my PYTHONSTARTUP file.
I often test Python modules as I develop them by running a Python interactive prompt in a terminal, importing my new module and testing out the functionality.
Stop using this pattern and start writing your test code in a file and your life will be much easier.
No matter what, running that file will be less trouble.
If you make the checks automatic rather than reading the results, it will be quicker and less error-prone to check your code.
You can save that file when you're done and run it whenever you change your code or environment.
You can perform metrics on the tests, like making sure you don't have parts of your code you didn't test.
Are you familiar with the unittest module?
Answering my own question, after some discussion on the python-ideas list: despite contradictory information in some readline documentation it seems that the operate-and-get-next function is in fact defined as a bash extension to readline, not by core readline.
So that's why Ctrl-o neither behaves as hoped by default when importing the readline module in a Python interpreter session, nor when attempting to manually force this binding: the function doesn't exist in the readline library to be bound.
A Google search reveals https://bugs.launchpad.net/ipython/+bug/382638, on which the GNU readline maintainer gives reasons for not adding this functionality to core readline and says that it should be implemented by the calling application. He also says "its implementation is not complicated", although it's not obvious to me how (or whether it's even possible) to do this as a pure Python extension to the readline module behaviour.
So no, this is not possible at the moment, unless the operate-and-get-next function from bash is explicitly implemented in the Python readline module or in the interpreter itself.
This isn't exactly an answer to your question, but if that is your development style you might want to look at DreamPie. It is a GUI wrapper for the Python terminal that provides various handy shortcuts. One of these is the ability to drag-select across the interpreter display and copy only the code (not the output). You can then paste this code in and run it again. I find this handy for the type of workflow you describe.
Your best bet will be to check that project : http://ipython.org
This is an example with a history search with Ctrl+R :
EDIT
If you are running debian or derivated :
sudo apt-get install ipython