I need to process MODIS ocean level 2 data and I obtained an external plugin for ENVI https://github.com/dawhite/EPOC/releases. Now, I want to batch process hundreds of images for which I modified the code like the following code. The code is running fine, but I have to select the input file every time. Can anyone please help me to make the program fully automatic? I really appreciate and thanks a lot for your help!
Pro OCL2convert
dir = 'C:\MODIS\'
CD, dir
; batch processing of level 2 ocean chlorophyll data
files=file_search('*.L2_LAC_OC.x.hdf', count=numfiles)
; this command will search for all files in the directory which end with
; the specified one
counter=0
; this is a counter that tells IDL which file is being read-starts at 0
While (counter LT numfiles) Do begin
; this command tells IDL to start a loop and to only finish when the counter
; is equal to the number of files with the name specified
name=files(counter)
openr, 1, name
proj = envi_proj_create(/utm, zone=40, datum='WGS-84')
ps = [1000.0d,1000.0d]
no_bowtie = 0 ;same as not setting the keyword
no_msg = 1 ;same as setting the keyword
;OUTPUT CHOICES
;0 -> standard product only
;1 -> georeferenced product only
;2 -> standard and georeferenced products
output_choice = 2
;RETURNED VALUES
;r_fid -> ENVI FID for the standard product, if requested
;georef_fid -> ENVI FID for the georeferenced product, if requested
convert_oc_l2_data, fname=fname, output_path=output_path, $
proj=proj, ps=ps, output_choice=output_choice, r_fid=r_fid, $
georef_fid=georef_fid, no_bowtie=no_bowtie, no_msg=no_msg
print,'done!'
close, 1
counter=counter+1
Endwhile
End
Not knowing what convert_oc_l2_data does (it appears to be a program you created, there is no public documentation for it), I would say that the problem might be that the out_path variable is not defined in the rest of your program.
Related
I am working with a nextflow workflow that, at a certain stage, groups a series of files by their sample id using groupTuple(), and resulting in a channel that looks like this:
[sample_id, [file_A, file_B, ... , file_N]]
[sample_id, [file_A, file_B, ... , file_N]]
...
[sample_id, [file_A, file_B, ... , file_N]]
Note that this is the same channel structure that you get from .fromFilePairs().
I want to use these channel items in a process in such a way that, for each item, the process reads the sample_id from the first field and all the files from the inner tuple at once.
The nextflow documentation is somewhat cryptic about this, and it is hard to find how to declare this type of input in a channel, so I thought I'd create a question on stack overflow and then answer it myself for anyone who will ever be looking for this answer.
How does one declare the inner tuple in the input section of a nextflow process?
In the example given above, my inner tuple contains items of only one type (files). I can therefore pass the whole second term of the tuple (i.e. the inner tuple) as a single input item under the file() qualifier. Like this:
input:
tuple \
val(sample_id), \
file(inner_tuple) \
from Input_channel
This will ensure that the tuple content is read as file (one by one), the same way as performing .collect() on a channel of files, in the sense that all files will then be available in the nextflow temp directory where the process is executed.
The question is how you come up with sample_id, but in case they just have different file extensions you might use something like this:
all_files = Channel.fromPath("/path/to/your/files/*")
all_files.map { it -> [it.simpleName, it] }
.groupTuple()
.set { grouped_files }
The path qualifier (previously the file qualifier) can be used to stage a single (file) value or a collection of (file) values into the process execution directory. The note at the bottom of the multiple input files section in the docs also mentions:
The normal file input constructs introduced in the input of files
section are valid for collections of multiple files as well.
This means, you can use a script variable, e.g.:
input:
tuple val(sample_id), path(my_files)
In which case, the variable will hold the list of files (preserving the original filenames). You could use it directly to refer to all of the files in the list, or, you could access specific (file) elements (if you need them) using square bracket (slice) notation.
This is the syntax you will want most of the time. However, if you need predicable filenames or if you need to deal with files with the identical filenames, you may need a different approach:
Alternatively, you could specify a target filename, e.g.:
input:
tuple val(sample_id), path('my_file')
In the case where a single file is received by the process, the file would be staged with the target filename. However, when a collection of files is received by the process, the filename will be appended with a numerical suffix representing its ordinal position in the list. For example:
process test {
tag { sample_id }
debug true
stageInMode 'rellink'
input:
tuple val(sample_id), path('fastq')
"""
echo "${sample_id}:"
ls -g --time-style=+"" fastq*
"""
}
workflow {
readgroups = Channel.fromFilePairs( '*_{1,2}.fastq' )
test( readgroups )
}
Results:
$ touch {foo,bar,baz}_{1,2}.fastq
$ nextflow run .
N E X T F L O W ~ version 22.04.4
Launching `./main.nf` [scruffy_caravaggio] DSL2 - revision: 87a80d6d50
executor > local (3)
[65/66f860] process > test (bar) [100%] 3 of 3 ✔
baz:
lrwxrwxrwx 1 users 20 fastq1 -> ../../../baz_1.fastq
lrwxrwxrwx 1 users 20 fastq2 -> ../../../baz_2.fastq
foo:
lrwxrwxrwx 1 users 20 fastq1 -> ../../../foo_1.fastq
lrwxrwxrwx 1 users 20 fastq2 -> ../../../foo_2.fastq
bar:
lrwxrwxrwx 1 users 20 fastq1 -> ../../../bar_1.fastq
lrwxrwxrwx 1 users 20 fastq2 -> ../../../bar_2.fastq
Note that the names of staged files can be controlled using the * and ? wildcards. See the links above for a table that shows how the wildcards are replaced depending on the cardinality of the input collection.
I'm currently running a code and I'm always getting to the same end. I am trying to read an input file and it returns the error:
Fortran runtime error: End of file
In an other post they said to put in the iostat specifier so now my code looks like this:
INTEGER :: m
INTEGER :: st
Open(Unit = 13,action='read',file='Data_Inp.dat',status='old')
read (13,*, iostat = st) m
write (*,*) st
write (*,*) m
ALLOCATE(winkel(m),energie(m))
Do i = 1,m
read(13,*),winkel(i),energie(i)
End Do
And the input file looks like this:
12
-17.83 -0.019386527878
-15.83 -0.020125057233
-12.83 -0.020653853148
-11.83 -0.020840036028
-9.83 -0.020974157405
-8.83 -0.021056401707
-6.83 -0.021065517811
-5.83 -0.020992571816
-4.83 -0.020867828448
-1.83 -0.02069158012
Now the terminal prints a -1 for iostat and a constantly changing number for m.
If the first read command is causing an error, check for extraneous characters before or after "12" in your input file, especially if you created it on one platform (Windows?) and using it on another platform (Linux? Mac?)
I don't understand the envi_setup_head. Could anyone help me write it in IDL code format?
I have maps that were produced in IDL and I need to process them in ENVI. I don't know how to save the images in a folder and be able open them in ENVI. Does anyone know how to do it?
To create an ENVI header for an image file, you can try something like the IDL procedure below. It creates a small image file and uses envi_setup_head to create an ENVI header file. Essentially all you have to do is provide it with the number of samples, lines, data-type, etc., and you are good to go.
pro enviHeaderTest
compile_opt idl2
; Create the data and write to a file.
ns = 100
nl = 100
data = dist(ns, nl)
fname = 'mydatafile.dat'
openw, lun, fname, /GET_LUN
writeu, lun, data
close, lun
; Open a headless ENVI.
nv = envi(/HEADLESS)
; Create some map info for the raster.
mc = [0,0,0,0] ;Tie point: [x pixel, ypixel, x map, y map]
ps = [1D/3600, 1D/3600] ; Pixel size
mapInfo = envi_map_info_create(/GEOGRAPHIC, MC=mc, PS=ps)
; Create the header.
envi_setup_head, FNAME=fname, $ ; file name
NS=ns, $ ; number of samples
NL=nl, $ ; number of lines
NB=1, $ ; number of bands
DATA_TYPE=4, $ ; IDL data type (float in this case)
INTERLEAVE=0, $ ; BSQ
MAP_INFO=mapInfo, $
/WRITE
; Close ENVI.
nv.close
end
Then, you can read the image into ENVI, either from the File->Open menu, or via the IDL command line like so:
IDL> nv = envi()
ENVI> view = nv.getview()
ENVI> raster = nv.openraster('mydatafile.dat')
ENVI> layer = view.createlayer(raster)
I'm doing some optimization using a model whose number of constraints and variables exceeds the cap for the student version of, say, AMPL, so I've found a webpage [http://www.neos-server.org/neos/solvers/milp:Gurobi/AMPL.html] which can solve my type of model.
I've found however that when using a solver where you can provide a commandfile (which I assume is the same as a .run file) the documentation of NEOS server tells that you should see the documentation of the input file. I'm using AMPL input which according to [http://www.neos-guide.org/content/FAQ#ampl_variables] should be able to print the decision variables using a command file with the appearance:
solve;
display _varname, _var;
The problem is that NEOS claim that you cannot add the:
data datafile;
model modelfile;
commands into the .run file, resulting in that the compiler cannot find the variables.
Does anyone know of a way to work around this?
Thanks in advance!
EDIT: If anyone else has this problem (which I believe many people have based on my Internet search). Try to remove any eventual reset; command from the .run file!
You don't need to specify model or data commands in the script file submitted to NEOS. It loads the model and data files automatically, solves the problem, and then executes the script (command file) you provide. For example submitting diet1.mod model diet1.dat data and this trivial command file
display _varname, _var;
produces the output which includes
: _varname _var :=
1 "Buy['Quarter Pounder w/ Cheese']" 0
2 "Buy['McLean Deluxe w/ Cheese']" 0
3 "Buy['Big Mac']" 0
4 "Buy['Filet-O-Fish']" 0
5 "Buy['McGrilled Chicken']" 0
6 "Buy['Fries, small']" 0
7 "Buy['Sausage McMuffin']" 0
8 "Buy['1% Lowfat Milk']" 0
9 "Buy['Orange Juice']" 0
;
As you can see this is the output from the display command.
I want to acquire image data from stereo camera simultaneously, or in parallel, save somewhere and read the data when need.
Currently I am doing
for i=1:100
start([vid1 vid2]);
imageData1=getdata(vid1,1);
imageData2=getdata(vid2,1);
%do several calculations%
....
end
In this cameras are working serially and it is very slow. How can I make 2 cameras work at a time???
Please help..
P.S : I also tried parfor but it does not help .
Regards
No Parallel Computing Toolbox required!
The following solution can generally solve problems like yours:
First the videos, I just use some vectors as "data" and save them to the workspace, these would be your two video files:
% Creating of some "videos"
fakevideo1 = [1 ; 1 ; 1];
save('fakevideo1','fakevideo1');
fakevideo2 = [2 ; 2 ; 2];
save('fakevideo2','fakevideo2');
The basic trick is to create a function which generates another instance of Matlab:
function [ ] = parallelinstance( fakevideo_number )
% create command
% -sd (set directory), pwd (current directory), -r (run function) ...
% finally "&" to indicate background computation
command = strcat('matlab -sd',{' '},pwd,{' '},'-r "processvideo(',num2str(fakevideo_number),')" -nodesktop -nosplash &');
% call command
system( command{1} );
end
Most important is the use of & at the end of the terminal command!
Within this function another function is called where the actual video processing is done:
function [] = processvideo( fakevideo_number )
% create file and variable name
filename = strcat('fakevideo',num2str(fakevideo_number),'.mat');
varname = strcat('fakevideo',num2str(fakevideo_number));
% load video to workspace or whatever
load(filename);
A = eval(varname);
% do what has to be done
results = A*2;
% save results to workspace, file, grandmothers mailbox, etc.
save([varname 'processed'],'results');
% just to show that both processes run parallel
pause(5)
exit
end
Finally call the two processes in your main script:
% function call with number of video: parallelinstance(fakevideo_number)
parallelinstance(1);
parallelinstance(2);
My code is completely executable, so just play around a bit. I tried to keep it simple.
After all you will find two .mat files with the processed video "data" in your workspace.
Be aware to adjust the string fakevideo to name root of all your video files.