I have a scenario where i need to Zip two different folders in different paths to a single Zip file using MSBuild.ExtensionPack.Compression.Zip. Is that possible, if so how i can achieve it.
Ex: C:\Folder 2 and D:\Folder 1 , to a single zip file Folder1.Zip.
Related
I have a folder structure like this as a source
Source/2021/01/01/*.xlsx files
Source/2021/03/02/*.xlsx files
Source/2021/04/03/.*xlsx files
Source/2021/05/04/.*xlsx files
I want to drop all these excel files into a different folder called Output.
Method 1:
When I am trying this, I used copy activity and I am able to get Files with folder structure( not a requirement) in Output folder. I used Binary file format.
Method 2:
Also, I am able to get files as some random id .xlsx in my output folder. I used Flatten Hierrachy.
My requirement is to get files with the same name as source.
This is what i suggest and I have implemented something in the past and i am pretty confident this should work .
Steps
Use getmetada activity and try to loop through all the folder inside Source/2021/
Use a FE loop and pass the ItemType as folder ( so that you get folder only and NO files , I know at this time you dont; have file )
Inside the IF , add a Execute pipeline activity , they should point to a new pipeline which will take a parameter like
Source/2021/01/01/
Source/2021/03/02/
The new pipeline should have a getmetadata activity and FE loop and this time we will look for files only .
Inside the FE loop add a copy activity and now will have to use the full file name as source name .
I need to delete many folders with an exempt list which has a list of folders and files that I should not delete. So I tried delete activity and tried to use if function of Add dynamic content to check whether the file or folder name is the same of the specified ones. But I do not have what should be the parameters #if(). In other words, to use these functions, how do we get the file name or folder name?
It is difficult to get all files and folders by using GetMetadata activity.
As workarounds, you can try these ways:
1.create a Delete activity and select List of files option. Then create a file filled with those path of files and folders need to be deleted.(relative path to the path configured in the dataset).
2.using blob SDK to do this thing.
We use Pig to load files from directories containing thousands of files, transform them, and then output files that are a consolidation of the input.
We've noticed that the output files contain the header record of every file processed, i.e. the header appears multiple times in each file.
Is there any way to have the header only once per output file?
raw_data = LOAD '$INPUT'
USING org.apache.pig.piggybank.storage.CSVExcelStorage(',')
DO SOME TRANSFORMS
STORE data INTO '$OUTPUT'
USING org.apache.pig.piggybank.storage.CSVExcelStorage('|')
Did you try this option?
SKIP_INPUT_HEADER
See https://github.com/apache/pig/blob/31278ce56a18f821e9c98c800bef5e11e5396a69/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/CSVExcelStorage.java#L85
Need SQL code to find the following details in a Drive/Path: - Folder Name and File Name - Folder Size and File Size in MB - Date Created and Date Modified and Date Accessed - File Flag (1/0) Yes or no based on whether the line item is a file or a folder - Path/URL/Location of the file or folder Need the details for all the folders and sub-folders and files in a given Path/URL
I have multiple codes which provides each of these details individually, but unable to merge it in a single query or join post execution.
Expected output:
http://s6.postimg.org/cn845zwdd/expected_output.jpg
Parts of the Codes are available in the Blogs as below:
http://sqljourney.wordpress.com/2010/06/08/get-list-of-files-from-a-windows-directory-to-sql-server/#comment-1674 (This link provides a function that gives date modified/path)
Get each file size inside a Folder using SQL (this link provides file size, file flag)
The above links provides parts of the query but i am unable to join them to get them in a single Store procedure..
I am trying to make a conversion program which converts multiple text files from a cad design file into a file that the machine can read.
each file has multiple values and is laid out like this:
X -0.0001
Y 1.0500
Z 1.5700
LOCATION 0.0050
Each file stands for a location that the machine is supposed to go to and do something. The output needs to look something like this:
X-0.0001Y1.0500Z1.5700L0.0050
Other information regarding position is here also.
so it's a fairly simple conversion. But what I'm wondering is what the best way to go about it is. Do I convert each file individually then combine them? The other information has to go at the bottom of the file. So where there are more files it would go:
Location 1
Location 2
Location 1 parameters
location 2 parameters
I have tried a couple different ways and still cannot come up with the best way to do it.
essentially what I'm asking is what the best/ most efficient way would be to convert these files. Sorry if this is confusing.
Note I am using vb.net for the programming language
if this is a project of enormous scale (e.g. millions of files), you might want to look into something like map reduce.
if not (which I'm guessing), I would suggest as follows:
parse each file sequentially, adding (appending)the results to each of TWO files. Finally, combine the two files and you're done.
LOCATIONS_FILE (FILE 1)
Location 1
Location 2
(etc)
METADATA_FILE (FILE 2)
location 1 params
location 2 params
(etc)
when all files are parsed, append the contents of FILE 2 to those of FILE 1.
FINAL FILE
Location 1
Location 2
(etc)
location 1 params
location 2 params
(etc)
I don't use VB.NET. But, pseudocode would be something like:
fn parse_file(file,locations_filehandle, metadata_filehandle):
file.extract_locations() -> append(locations_filehandle)
file.extract_metadata() -> append(metadata_filehandle)
fn main():
for file in files:
parse_file(file,locations_filehandle,metadata_filehandle)
finalfile=locations_filehandle.read() + metadata_filehandle.read()
finalfile.writeToDisk()
main()