I am trying to read the view angles from a Sentinel-2 image (L1C SAFE compact format) for executing an atmospheric correction algorithm. I can get those values by parsing the file MTD_TL.xml, but I am not able to get them through rasterio.
I have tried to access to those data using the xml:SENTINEL2 and the xml:VRT metadata domains, but I can only access to the values from the file MTD_MSIL1C.xml (the main metadata file).
The whole point of using rasterio is being able of using GDAL's virtual file system, as the images will be read from S3 buckets. Any alternatives for easily reading MTD_TL.xml through the virtual file system would be also valid (and really appreciated).
Thank you!!
I answer to myself.
I could not find how to get the values I require, but according to https://gdal.org/user/virtual_file_systems.html the function VSIFOpenL may be used for opening the file. After that, manual parsing will do the trick :)
Ps. I must read the documentation slowly.
Related
I can already see three ways, but none are quick (not compared to say, accessing a raw file on github)
Fork/download (requires registration)
Follow instructions here (i.e. download, open up in jupyter/ipython notebook)
Copy the code blocks manually, one by one (bad for long notebooks)
Is there an easier way? (I hoping, ideally, to add raw to the url somewhere, just like on github)
In case it's useful to others, put the notebook url here to extract raw code
I have to create a sub routine using VB.Net that compress some files into a "file.zip" file, but the problem is that this "file.zip" MUST have the maximum size of 2 MB.
I don't know how to do it, even if it's possible.
It would be nice if someone has some example to show me.
It is not possible to do this in the general case. For example if you have a 2GB movie file, no lossless compression algorithm will ever get it to 2MB.
One solution is to "chunk" your ZIP file. That is, divide it into parts that are individually no more than 2MB. 7-Zip has support for this. You can use their .NET API from VB.Net. I'm not sure whether the API provides direct support for chunking. If not, you can start 7-Zip from your program using Process.Start().
There are really two questions that revolve around the use of --add-section. The simple one is in the title. Based on my reading, I haven't been able to figure out how one could execute --add-section.
To use add-section, I have to pass a section name. If I use an existing section name the program responds with "can't add section '.data': File in wrong format." Perhaps I just need to pass another parameter. If I use a new section name, which I would prefer to do, I'm warned that "allocated section '.blob' not in segment."
Now, I have gotten my feature to work as I need it to aside from the "not in segment" warning. I'd like to figure out if there is a legitimate way to put this blob into the executable. I would link it in, but that isn't so easy because the data I'm adding is generated from the contents of the executable itself.
The second question is really what I care about. Is there a way to do the following given that the blob cannot be computed until after the link is complete.
Link ELF file
Generate blob from ELF file and other data
Add blob to ELF file so that it is loaded at run-time to the correct location in memory
objcopy --add-section .blob=blob.o \
--set-section-flags .blob=alloc,contents,load,readonly \
--change-section-address .blob=ADDRESS \
program.elf program.blobbed.elf
I'd be happy to add a section and/or segment to the ELF file as part of the link and insert this blob there. I'm not sure how to do that.
It has occurred to me that I could accomplish this feat with a second link, but objcopy would be cleaner.
Link ELF file
Generate blob from ELF file and other data
Re-link ELF file including new blob.o
UPDATE: This last strategy may be workable as long as the relink doesn't change something in the portion of the program that was produced by the first link. It doesn't on first attempts, but it may be possible to work around it. Hence, the desire to use --add-section to add in this blob instead of going through a second link.
You may add that section, fill it with, say, NULs, and then compute your blob. Then patch that blob into this section. Later, when you check the integrity of the ELF, do as if that section was full of NULs and compute the blob again. Finally, compare both computed blob and blob stored in section.
Not specifically answering your question but one approach I used to use for this sort of thing was to link in a placeholder block and then just patch the correct value in afterwards.
I know this isn't what you want to do, but it is a pretty simple and reliable way to do it. And has the major advantage of being tool chain/platform agnostic.
I have a custom algorithm for processing a graph which accepts a txt file as input. Because it is a large scale graph I want to implement it in the apache giraph framework. I' ve done a lot of research but I am still not sure if I am in the right path.
I am reading a .csv file which contain the graph data and using a parser I am converting it to the txt file and uploading to the HDFS file system of hadoop.
I have read the SimpleShortestPathsVertex example from the apache quick start guide and I can see that processes the data from a file in HDFS using the jar-with-dependencies jar file.
My problem is that I haven't yet understood how can I add my algorithm in the apache giraph framework and start the process of the graph. Can I add my algorithm to apache framework using eclipse and modify it from there or there is any other way?
Thank you!
Have a look here:
https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example
Where you able to run this example?
If yes.
Familiarize yourself with the different Writable formats of hadoop! Else it is hard to use these to your algorithm.
All computation concerning the graph is done in the compute() function.
(If you're more advanced have a look into the workerContext preSuperstep and Aggregators!)
You can change the example, but as soon as you use other data types you have to change your VertexReader and VertexWriter.
If you have a specific Algorithm in mind, make up your mind what you need for the computation and specify the layout of your input file. Then adapt your VertexReader and -Writer. And then finaly start the implement your compute() function!
Of course you can use eclipse! Simply Reference the Giraph jar (For me it is "giraph-0.1-jar-with-dependencies.jar") And start coding.
All you need is a instance of these files specific to your algorithm:
YourGiraphJob (the file starting the Hadoop/Giraph job)
YourVertex (Specifies your compute() function executed on each Vertex)
YourInputFormat (Specifying the Writable formats of YourReader)
YourOutputFormat (Specifying the Writable formats of YourWriter)
YourReader (Specifies how your inputFile is transformed e.g. that for each line a Vertex can be initialized using given information)
YourWriter (Specifies how your outputFile is generated from the vertices)
(optionaly a WorkerContext if you want to use Aggregators.)
Simply checkout: http://giraph.apache.org/source-repository.html
using eclipse and you should have the code including an example application you can toy around with!
This is a bit of a two part question, for working with 40mb xml files.
• What’s a reasonable size to store in memory for a program running continually in the background?
• How to find what has changed in an XML file.
So on the first read the XML is loaded into NSData, then uploaded to the server.
Now instead of uploading a 40mb XML every time it changes, I would prefer to upload a “delta” file containing only what has changed. The program would monitor the file for change, and activate when it’s been modified. From what I can see, I would need to parse an old version of the xml file and parse the modified xml file, then compare them? Is it unreasonable to store 80mb in memory like this every time the file is modified?. Now I’m assuming that this has to be done with a DOM parser because I can’t see how you could compare two files like that with a SAX parser since it only has part of the file stored?
I'm a newbie at this so any help would be appreciated!
To compare two files:
There are many ways to do, (As file is to be considered, I may not be correct):
sdiff file1.xml file2.xml A unix command
You can use this command with apple script.
-[NSFileManager contentsEqualAtPath:andPath:]
This method checks to see if two files at given path are the same file, then compares their size, and finally compares their contents.
For other part:
What size is considered for background process, I dont think so, for an application it matters. You can save these into temporary files. Even safari uses 130+ MB as you can easily check through Activity monitor.
NSXMLParser ended up being the most useful for this