I know how to read the CSV into numpy and do it from a Python script, and that is good enough for my use case.
But since it has a GUI with data loading functionality, I was expecting it would just work for such an universal data format.
So I tried to go on the menu:
File
Load data
Open file
but when I select a simple CSV file:
i=0; while [ "$i" -lt 10 ]; do echo "$i,$((2*i)),$((4*i))"; i=$((i+1)); done > main.csv
which contains:
0,0,0
1,2,4
2,4,8
3,6,12
4,8,16
5,10,20
6,12,24
7,14,28
8,16,32
9,18,36
an error popup shows on the GUI:
No suitable reader found for file /home/ciro/main.csv
Google led me to this interesting file in the source tree: https://github.com/enthought/mayavi/blob/e2569be1096be3deecb15f8fa8581a3ae3fb77d3/mayavi/tools/data_wizards/csv_loader.py but that just looks like an example of how to do it from a script.
Tested in Mayavi 4.6.2.
From the documentation
One needs to have some data or the other loaded before a Module or Filter may be used. Mayavi supports several data file formats most notably VTK data file formats. Alternatively, mlab can be used to load data from numpy arrays. For advanced information on data structures, refer to the Data representation in Mayavi section.
I've tested importing using the GUI on a Asus Laptop Intel CoreTM i7-4510U CPU # 2.00 GHz with 8 GBs de RAM, using Windows 10, both in and out of a Python virtualenv and always got the same problem:
It all points to CSV files not being directly supported, so had to find another workaround.
My favorite was to use a virtual environment and install on it mayavi, jupyterlab, PyQt5 and Pandas.
Then, using PowerShell, start a Jupyter notebook (jupyter notebook) > Upload > Select the .csv. This imported a 1,25 GBs (153543233 rows x 3 columns) .csv in around 20s, which then became available for usage.
Related
I'm reading the csv file at https://github.com/VinitaSilaparasetty/julia-beginners/blob/master/data/nba/nba19-20.csv
I get a DataFrame and I save it as XLSX. When I try to read it in jupyterlab I get the error the file is not UTF-8 encoded and therefore the file is not read.
This is my code:
using HTTP, XLSX, CSV, DataFrames
df = CSV.read(HTTP.get("https://raw.githubusercontent.com/VinitaSilaparasetty/julia-beginners/master/data/nba/nba19-20.csv").body)
# first(df,5) # first shows the top five rows ok
XLSX.writetable("data/nba/nba19-20.XLSX", collect(eachcol(df)), names(df), overwrite = true)
The file is saved in my data folder. When I try to open it with jupyterlab, I get a pop up with the file is not UTF-8 encoded and the file is not opened.
When I try to open the file in Ubuntu (with LibreOffice) I do not see anything suspicious.
As I'm new to Julia I'm struggling to understand where the problem lies or how to fix it.
I tried to see if I could encode the dataframe in UTF-8 (after saving the file to disk) with
data = DataFrame(CSV.File(open(read,"data/nba/nba19-20.csv", enc"utf-8")))
But I did not see any change. Any suggestion is welcome.
Do you have the jupyterlab-spreadsheet plugin installed? JupyterLab by default doesn't support opening xlsx files (it isn't mentioned in the file formats list here for example).
See also this similar question involving Python pandas (which says pretty much the same thing).
I am new to geospatial analytics and using NetCDF and GeoTIFF files. I am trying to convert the NetCDF file into GeoTIFF file. I came across this reference: netcdf-to-geotiff-file-conversion. I have successfully installed and can run gdal_translate in my MacOS terminal. However, I get this message I am trying to understand what this means and what I am missing? This appears to be warning, however it didn't generate any output.
My code executed:
gdal_translate some_nc_file_name.nc output.tif
Error/Message:
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Input file contains subdatasets. Please, select one of them for reading.
Here is my data output I previewed in Python:
You appear to have multiple variables in the file, so need to select one. Example (following https://nsidc.org/support/how/how-convert-golive-netcdf-variables-geotiff).
gdal_translate NETCDF:"Input_FileName.nc":variable_name Output_FileName.tif
Based on the warning message, the netCDF file lacks certain important attributes, so some issues may exist with the coordinates etc. in the tif.
After simulation is finished Dymola runs dsres2sdf.exe to convert the results to sdf-format (if that option is enabled in the simulation setup output tab).
Usually this runs smoothly but sometimes it generates a sdf file that is very small (800 Byte) and empty.
Starting the dsres2sdf.exe manually from command line generates the same empty file.
I suspect that happens if the *.mat-File is very large (>1 GB)
Anybody has any clue how to get a proper sdf-File?
The SDF Editor and the SDF libraries for Python and MATLAB can read Dymola result files (*.mat) transparently (as if they were SDFs) and allow you to save them as *.sdf.
For example with Python:
import sdf
# load the Dymola result file
data = sdf.load('DoublePendulum.mat')
# re-save as SDF
sdf.save('DoublePendulum.sdf', data)
I'm trying to export via assimp command line tool two models: one from .fbx (2013) and one from .lwo both to .obj file format.
However I'm not able to extract the textures from the original files.
The command that I perform is:
assimp export INPUT_FILE.xxx OUTPUT_FILE.obj -cts, -gsn, -jiv, -icl, -lbw, -rrm, -slm, -tri, -guv, -sbpt, -fd, -fiv, -fi, -vds -om
The model are:
https://nasa3d.arc.nasa.gov/detail/cubesat-1RU
and
https://nasa3d.arc.nasa.gov/detail/acrimsat
previously converted to the .fbx 2013 file format via FbxConverterUI
At the moment we do not export the embedded textures as far as I know. We opened an issue to get this fixed.
I have a binary file (capture.bin) from the rtl_sdr tool. I convert it to a .cfile with this manual http://sdr.osmocom.org/trac/wiki/rtl-sdr#Usingthedata
Where can I get the data in this file? The goal is to get a numerical format output from the the source. Is this possible?
That actually is covered by a GNU Radio FAQ entry.
What is the file format of a file_sink? How can I read files produced by a file sink?
All files are in pure binary format. Just bits. That’s it. A floating point data stream is saved as 32 bits in the file, one after the other. A complex signal has 32 bits for the real part and 32 bits for the imaginary part. Reading back a complex number means reading in 32 bits, saving that to the real part of a complex data structure, and then reading in the next 32 bits as the imaginary part of the data structure. And just keep reading the data.
Take a look at the Octave and Python files in gr-utils for reading in data using Octave and Python’s Scipy module.
The exception to the format is when using the metadata file format. These files are produced by the File Meta Sink: http://gnuradio.org/doc/doxygen/classgr_1_1blocks_1_1file__meta__sink.html block and read by the File Meta Source block. >See the manual page on the metadata file format for more information about how to deal with these files.
A one-line Python command to read the entire file into a numpy array is:
f = scipy.fromfile(open("filename"), dtype=scipy.uint8)
Replace the dtype with scipy.int16, scipy.int32, scipy.float32, scipy.complex64 or >whatever type you were using.
Update
scipy.fromfile will be deprecated in v2.0 so instead use numpy library
f = numpy.fromfile(open("filename"), dtype=numpy.uint8)