I use IntelliJ premium, I updated the whole app, and I saw a very annoying new output style of it's Jupyter notebook. Then I reinstalled an older version, I see the jupyter output is still in new format. I guess it is because of Jupyter's package update.
How can I have the old-style table format?
the new style shows only 10 rows and for every run, you should change 10 to a higher number which becomes annoying after a couple of minutes. It has gotten really slow too.
Related
The code is taking about 20 minutes to load a month for each variable with 168 time steps for the cycle of 00 and 12 UTC of each day. When it comes to saving to csv, the code takes even longer, it's been running for almost a day and it still hasn't saved to any station. How can I improve the code below?
Reading .grib files using xr.open_mfdataset() and cfgrib:
I can speak to the slowness of reading grib files using xr.open_mfdataset(). I had a similar task where I was reading in many grib using xarray and it was taking forever. Other people have experienced similar issues with this as well (see here).
According to the issue raised here, "cfgrib is not optimized to handle files with a huge number of fields even if they are small."
One thing that worked for me was converting as many of the individual grib files as I could to one (or several) netcdf files and then read in the newly created netcdf file(s) to xarray instead. Here is a link to show you how you could do this with several different methods. I went with the grib_to_netcdf command via ecCodes tool.
In summary, I would start with converting your grib files to netcdf, as it should be able to read in the data to xarray in a more performant manner. Then you can focus on other optimizations further down in your code.
I hope this helps!
I am working on pre-processing a few geo-tif files. I want to obtain the minimum value for my raster image. My OS is Linux and Rasterio version 1.3.3. When I run my script in a conda environment, I realized that the minimum value is given as zero which is incorrect.
I am using the following code:
with rasterio.open(file) as tif_input_obj:
tif_data = tif_input_obj.read()
print(tif_data.min())
I tried running the same lines of code by typing python in my conda environment. This time the minimum value was obtained correctly. I have also tried gdal and faced the same problem.
has anyone faced this issue before?
Thank you for your help.
I tried to run a project (ipynb extension) from GitHub using Google colab.
I have managed to run the program, but when compared with the author’s output, mine is a little different.
For example, train_df.describe() does not print some of the columns (‘target' column in particular because that is used to plot a graph.
Why is it that I run the same program but get different result?
I am doing a project in the jupyter notebook/lab environment, and am wondering if anyone knows the speed of csv's deleting rows vs just writing a whole new csv file. I am going to do around 20 filters and calculations, and would prefer to keep the filters separate for error checking. (some are required to be done separately)
I am very new to speed of functions, and can't find documentation as which would be faster.
Basically, would I be better off just having a single file that I keep removing things from as I filter things out, or writing to a new file.
I have previously been using writing to a new file, but the size of this dataset is concerning for me. So far I haven't had errors with anything, but speed is already an issue with small sizes.
I've also debated just truncating a file instead of making a new one.
I have recently begun learning to use Julia, converting over from Matlab/Octave. I decided that the best way to get some experience was to convert some code I was already working on i Octave - a Newton solver for a complicated multidimensional problem. I have been able to convert the code over successfully (and with noticeable speedup relative to Octave, without devectorisation or other performance-based changes), with only one issue arising.
I have chosen to use PyPlot for plotting, due to its similarity to Matlab/Octave's plotting functionality. However, there is some behaviour from PyPlot that is undesired. I use the plotting function to display the current state of the vector I am trying to get to zero (using the Newton solver part of the code), so that I can see what it is doing, and adjust the code to try to improve this behaviour. I input the number of Newton steps to take before the code stops, and then I can make adjustments or re-issue the command to continue attempting to converge.
I have the code set up to plot the current state every few steps, so that I can, for instance, have the code take 200 steps, but show me the status after every 10 steps. In Octave, this works perfectly, providing me with up-to-date information - should the behaviour of the code not be desirable, I can quickly cancel the code with Ctrl-C (this part works in Julia, too).
However, Julia does not produce or update the plots when the plot() command is used; instead, it produces the plot, or updates it if the plot window is already open, only when the code finishes. This entirely defeats the purpose of the intermittent plotting within the code. Once the code has completed, the plot is correctly generated, so I know that the plot() command itself is being used correctly.
I have tried adding either draw() or show() immediately after the plot command. I have also tried display(gcf()). None of these have modified the result. I have confirmed that isinteractive() outputs "true". I have also tried turning interactivity off (ioff()) and switching whether to use the python or julia backend (pygui(true) and pygui(false)), with no effect on this behaviour.
Have I missed something? Is there another package or option that needs to be set in order to force PyPlot to generate the current plot immediately, rather than waiting until Julia finishes its current code run to generate the plot?
Or is it perhaps possible that scope is causing a problem, here, as the intermittent plotting happens inside a while loop?
I am using xubuntu 12.10 with Julia 0.2.1.
PyPlot defaults to this behavior in the REPL. To make it show the plots as they are plotted type ion(). To turn it off again type ioff().
ion() is only effective for the current season so if you want it to stay on across sessions just add it to your .juliarc file.
If you're using iPython ion() will plot to a new window but ioff() will plot inline.