I've been developing an SSIS package from a flat file source. The file comes daily and the file name has datetime indication like this:
Filename_20190509042908.txt
I was wondering how I can pass until the date part; I want the package to read the file dynamically, but it should pass without the last 6 digits I just don't need the last 6 digit numbers as it is not consistent.
I want to pass Filename_20190509.txt
I have figured out how to take the filename until date removing the time part. Hence, I've trouble to let the package read the file name dynamically by ignoring the last 6 digits before file extension.
Can anyone help me with this please?
Remove the time part from the full file path
Assuming that the full file path is stored within a variable named #[User::FilePath]
You have to add a variable of type string (example: #[User::Filename]), Before the data flow task add an Expression Task and use the following expression:
#[User::Filename] = SUBSTRING(#[User::FilePath], 1, LEN(#[User::FilePath]) -
FINDSTRING(REVERSE(#[User::FilePath]), "\\", 1)) + "\\" +
LEFT(TOKEN(#[User::FilePath],"\\",TOKENCOUNT(#[User::FilePath],"\\")),
LEN(TOKEN(#[User::FilePath],"\\",TOKENCOUNT(#[User::FilePath],"\\"))) - 10) + ".txt"
Example:
If the value of #[User::FilePath] is
C:\New Folder\1\Filename_20190503001221.txt
Then #[User::Filename] will be:
C:\New Folder\1\Filename_20190503.txt
If You have only the file name as
filename_20190503001221.txt
and the folder path is stored in another variable, just use the following expression:
#[User::Filename] = #[User::Folderpath] + "\\" +
LEFT(TOKEN(#[User::FilePath],"\\",TOKENCOUNT(#[User::FilePath],"\\")),
LEN(TOKEN(#[User::FilePath],"\\",TOKENCOUNT(#[User::FilePath],"\\"))) - 10) + ".txt"
Read File Source from Variable
Click on the flat file connection manager used to read the source file, press F4 to show the properties tab, Click on the expression property and assign the following expression to the connectionstring property:
#[User::Filename]
Now change the Data Flow Task Delay validation property value to True.
Dynamic Flat File Connections in SQL Server Integration Services
I have to assume you are using a foreach loop already as the filename is changing but here is how to change the fully qualified name to what you want:
TOKEN(character_expression, delimiter_string, occurrence)
Your usage:
this will get you the full filename:
exp = TOKEN(#filename,"\",LEN(#filename)-LEN(replace(#filename,"\",""))
then from there you need use left plus add .txt
left(exp,LEN(exp)-10) + ".txt"
Related
I'm trying to create a python program that allows you to dynamically choose the folder to save the download.
'outtmpl' lets you choose an output as a simple path such that
'e:/python/downloadedsongs/%(title)s.%(ext)s'
is a valid path and will let you save to the downloadedsongs folder
But With the function that defines the choose folder button
def openLocation():
global Folder_Name
Folder_Name = filedialog.askdirectory()
print(Folder_Name)
why does
f'{Folder_name}/%(title)s.%(ext)s'
not return a valid path, the program outright ignores the variable. I thought at first i was having a fstring issue because %s was pythons old format for f'strings but I have also tried these variances with no success
'outtmpl': Folder_Name + '/' + '%(title)s.%(ext)s', (no errors but cannot find output)
'outtmpl': '%(Folder_Name)s%(title)s.%(ext)s',
%()s is the old version of fstrings I was conflating the two.
answer ended up looking like 'outtmpl': Folder + '/%(title)s.%(ext)s'
I just started learning Python and now I'm trying to integrate that with my GIS knowledge. As the title suggests, I'm attempting to convert an Excel sheet to a table but I keep getting errors, one which is wholly undecipherable to me and the other which seems to be suggesting that my file does not exist which, I know is incorrect since I copied it's location directly from it's properties.
Here is a screenshot of my environment. Please help if you can and thanks in advance.
Environment/Error
Simply set, you put the workspace directory inside the filename variable so when arcpy handles it, it tries to acess a file that does not exist, in an unknown workspace.
Try this.
arcpy.env.workspace = "J:\egis_work\dpcd\projects\SHARITA\Python\"
arcpy.ExcelToTable_conversion("Exceltest.xlsx", "Bookstorestable", "Sheet1")
Arcpy uses the following syntax to convert geodatabase tables to excel
It is straight forward.
Example
Excel tables cannot be stored in the geodatabase. Most reasonable thing is to store them in the rootfolder in which the geodatabase with the table is. Say I want to convert table below into excel and save it in the root folder or in the folder in which the geodatabase is.
I will go as follows: I have put the explanations after the #.
import arcpy
import os
from datetime import datetime, date, time
# Set environment settings
in_table= r"C:\working\Sunderwood\Network Analyst\MarchDistances\Centroid.gdb\SunderwoodFirstArcpyTable"
#os.path.basename(in_table)
out_xls= os.path.basename(in_table)+ datetime.now().strftime('%Y%m%d') # Here
#os.path.basename(in_table)- Gives the base name of pathname. In this case, it returns the name table
# + is used in python to concatenate
# datetime.now()- gives todays date
# Converts todays date into a string in the format YYYMMDD
# Please add all the above statements and you notice you have a new file name which is the table you input plus todays date
#os.path.dirname() method in Python is used to get the directory name from the specified path
geodatabase = os.path.dirname(in_table)
# In this case, os.path.dirname(in_table) gives us the geodatabase
# The The join() method takes all items in an iterable and joins them into one string
SaveInFolder= "\\".join(geodatabase.split('\\')[:-1])
# This case, I tell python take \ and join on the primary directory above which I have called geodatabase. However, I tell it to remove some characters. I will explain the split below.
# I use split method. The split() method splits a string into a list
#In the case above it splits into ['W:\\working\\Sunderwood\\Network', 'Analyst\\MarchDistances\\Centroid.gdb']. However, that is not what I want. I want to remove "\\Centroid.gdb" so that I remain with the follwoing path ['W:\\working\\Sunderwood\\Network', 'Analyst\\MarchDistances']
#Before I tell arcpy to save, I have to specify the workspace in which it will save. So I now make my environment the SaveInFolder
arcpy.env.workspace =SaveInFolder
## Now I have to tell arcpy what I will call my newtable. I use os.path.join.This method concatenates various path components with exactly one directory separator (‘/’) following each non-empty part except the last path component
newtable = os.path.join(arcpy.env.workspace, out_xls)
#In the above case it will give me "W:\working\Sunderwood\Network Analyst\MarchDistances\SunderwoodFirstArcpyTable20200402"
# You notice the newtable does not have an excel extension. I resort to + to concatenate .xls onto my path and make it "W:\working\Sunderwood\Network Analyst\MarchDistances\SunderwoodFirstArcpyTable20200402.xls"
table= newtable+".xls"
#Finally, I call the arcpy method and feed it with the required variables
# Execute TableToExcel
arcpy.TableToExcel_conversion(in_table, table)
print (table + " " + " is now available")
New to PDI here. Need to output data from a view in a postgresql database to a file daily. The output file will be like xxxx_20160427.txt, so need to append the dynamic date in the file name. How to do it?
EDIT-----------------
I was not clear here by asking how to add dynamic date, I was trying to add not just date but optional other parts to the file name. E.g adding a serial no (01) at the end: xxxx_2016042701.txt etc. So my real question is how to make a dynamic file name? In other ETL tool e.g. SSIS it will be a simple expression. Not sure how it is done in PDI?
In your Text file output step, simply check "Include date in filename?" under the files tab.
You can create a dynamic filename variable with a Modified Java Script value STEP.
and then in the Text File Output STEP click on "Accept file name from field", and select your variable declared from previous step (filename_var on this example).
I'm attempting to create a process to import data. I created the entire process and it works, but I'm having trouble creating the variable to find the file name of the csv i want to import automatically. Each time a new csv is uploaded to me it has a timestamp on it. I want to be able to grab that file no matter what the name is and do work to it.
So for example this week the file name would be
filename_4-14-2014.csv
And next week
filename_4_21_2014.csv
And so on into eternity. . .
Is there a way to create a variable that picks up the full file name even though its changing?
After doing some poking around, I've discovered the following...
You can use a file system task to perform the copy operation I was referring to. You can set the input file and the output file as variables. This way you can always know that the file you use for import is always named the same, and has the right data.
You just need to add the variables and a File System Task to your package.
Ok so to accomplish what I wanted I created a Foreach Loop Container. Using the foreach loop container I had it look for any files ending with .csv in my specified folder by using a wildcard [denoted by asterisk: *.csv] .
Within the Foreach Loop container is as follows.
Step 1: File System Task - rename file.
Step 2: Data Flow Task - Import data to sql
Step 3: File System Task - Copy the file to another folder, append datetime to filename
Step 4: File System Task - Delete source file.
I used variables to get all the file and folder names plus datetimes.
I ran into this problem when uploading a file with a super long name - my database field was only set to 50 characters. Since then, I have increased my database field length, but I'd like to have a way to check the length of the filename before uploading. Below is my code. The validation returns '85' as the character length. And it returns the same count for every different file I upload (none of which have a file name length of 85).
<cfscript>
missing_info = "<p>There was a slight problem with your submission. The following are required or invalid:</p><ul>";
// Check the length of the file name for our database field
if ( len(Form["ResumeFile1"]) gt 100 )
{
missing_info = missing_info & "<li>'Resume File 1' is invalid. Character length must be less than 100. Current count is " & len(Form["ResumeFile1"]) & ".</li>";
validation_error = true;
ResumeFileInvalidMarker = true;
}
</cfscript>
Anyone see anything wrong with this?
Thanks!
http://www.cfquickdocs.com/cf9/#cffile.upload
After you upload the file, the variable "clientFileName" will give you the name of the uploaded file, without a file extension.
The only way to read the filename before you upload it would be to use JavaScript to read and parse the value (file path) in the file field.
A quick clarification in the wording of your question. By the time your code executes the file upload has already happened. The file resides in a temporary directory on the ColdFusion server and the form field related to the file upload contains the temporary filename for that file. Aside from checking to see if a file has been specified, do not do anything directly with that file or you'll be circumventing some built in security.
You want to use the cffile tag with the upload action (or equivalent udf) to move the temp file into a folder of your choosing. At that point you get access to a structure containing lots of information. Usually I "upload" into a temporary directory for the application, which should be outside of the webroot for security.
At this point you'll then want to do any validation against the file, such as filename length, file type, file size, etc and delete the file if it fails any checks. If it passes all checks then you move it into it's final destination which may be inside the webroot.
In your case you'll want to check the cffile structure element clientFile which is the original filename including extension (which you'll need to check, since an extension doesn't need to be present and can be any length).