I have the following problem:
I have a data table that is fed by data from a SQL query.
The query works just fine, but not all the data is displayed. I deleted one of the columns before and no wanted to readd it, but it does not show.
Is there a way to get this to work?
Basically, I have those columns:
Name, First name, birthday, gender
Now I deleted gender:
Name, First name, birthday
After a while, I wanted to readd gender, but the data table shows the following:
Name, first name, birthday
It does work, if I change the column name from gender to sex in the SQL query, but that is not a solution I can live with.
If I change the name, then rename the column header, on the next refresh, the name is reinstated. If I rename the column header, then change the column name in the SQL query, the column disappears on the next refresh.
Anyone with a solution?
I'm guessing you have Preserve column/sort/filter/layout checked in the External Data Properties dialog (right-click> Table> External Data Properties). Try unchecking it, refreshing, and then checking it again. Save first!
I had the same issue, and finally found an easy solution for adding columns. Click on the table, then Query>Edit>Advanced Editor (under the home tab).
You should see the source code for the query. In the first line of code, you will see Columns= (followed by your number of columns).
You need to change this number to reflect the correct number of columns in the new CSV file. I originally had 17 columns. I added two data columns, so I changed this number to 19.
Close the editor and refresh, and you should be all set.
For anyone who needs it:
I did not find how to follow method described by Sullivan
I found different one. It needs editing XML in the unzipped XLSX file.
1. Add column to your table and rename it to column name you want to restore
2. Open in the notepad queryTable xml file extracted from XLSX (you have to find proper one)
3. Open (in notepad or IE) proper Table[#].xml with your table, find by name and remember ID of your column as your column ID
4. Find tag with name of column you need to restore and remove this tag.
5. Find column with text ' tableColumnId="[your column ID]" '
6. Add atrybute 'name = "[column name]"' and delete atribute 'dataBound="0"'.
7. save querytable[#].xml zip all folders to one file , rename to xlsx (never zip one folder that contains all, you need to select all objects and zip).
[#] of querytable is not always the same as # of table.
Relation is described in xl\tables_rels\table[#].xml.rels
Related
I have a problem with SSRS report builder. Basically what I want to do is to delete a column. I have a report that someone else made and there is 1 column (xxx) that no longer exists in data source tables so I need to delete it.
When I go to query designer and delete this column from the code and run it there, it works. I close query designer window and see that list of columns (fields) is updated now and xxx column is not there. Then, I delete this column manually in designer (default screen) and when I try to run the report, It doesn't work:
"The Value expression for the text box ‘XXXDataField’ refers to the field ‘xxx’. Report item expressions can only refer to fields within the current dataset scope or, if inside an aggregate, the specified dataset scope. Letters in the names of fields must use the correct case."
But that field should be already deleted. So I don't know what else I can do, or what it can be linked to. I just want to delete it. Any Idea?
Thank you
If your dataset does not contain xxx but your DataTable has this error is normal
You either delete that column from your table or at lease delete DataSet Binding from table so that SSRS does not try to retrieve that column from DataSet
It will be a reference to the field in another field. For example, if you deleted a column that showed an OrderShipped status, then you might have another text box highlighted based on this.
The error is telling you which textbox is in error. So, click somewhere on the designer, then in the properties window, right at the top, click the dropdown which allows you to choose specific report items, choose XXXDataField (the one named in the error message) and then check the value expression. In there you will find the reference to the column you deleted.
I'm new to PDI and Kettle, and what I thought was a simple experiment to teach myself some basics has turned into a lot of frustration.
I want to check a database to see if a particular record exists (i.e. vendor). I would like to get the name of the vendor from reading a flat file (.CSV).
My first hurdle selecting only the vendor name from 8 fields in the CSV
The second hurdle is how to use that vendor name as a variable in a database query.
My third issue is what type of step to use for the database lookup.
I tried a dynamic SQL query, but I couldn't determine how to build the query using a variable, then how to pass the desired value to the variable.
The database table (VendorRatings) has 30 fields, one of which is vendor. The CSV also has 8 fields, one of which is also vendor.
My best effort was to use a dynamic query using:
SELECT * FROM VENDORRATINGS WHERE VENDOR = ?
How do I programmatically assign the desired value to "?" in the query? Specifically, how do I link the output of a specific field from Text File Input to the "vendor = ?" SQL query?
The best practice is a Stream lookup. For each record in the main flow (VendorRating) lookup in the reference file (the CSV) for the vendor details (lookup fields), based on its identifier (possibly its number or name or firstname+lastname).
First "hurdle" : Once the path of the csv file defined, press the Get field button.
It will take the first line as header to know the field names and explore the first 100 (customizable) record to determine the field types.
If the name is not on the first line, uncheck the Header row present, press the Get field button, and then change the name on the panel.
If there is more than one header row or other complexities, use the Text file input.
The same is valid for the lookup step: use the Get lookup field button and delete the fields you do not need.
Due to the fact that
There is at most one vendorrating per vendor.
You have to do something if there is no match.
I suggest the following flow:
Read the CSV and for each row look up in the table (i.e.: the lookup table is the SQL table rather that the CSV file). And put default upon not matching. I suggest something really visible like "--- NO MATCH ---".
Then, in case of no match, the filter redirect the flow to the alternative action (here: insert into the SQL table). Then the two flows and merged into the downstream flow.
I need guidance regarding the most approriate approach to perform a index function using pentaho Data integration ( kettle )
my situation is as following :
using the GLOBAL voip system report, I stored all data in a Mysql Database, which gives me several id number + name and lastname but whithout the departement name.
each departement name has it's own excel reports that can be identified by the group file name, which is not available in the Global file.
what i am trying to achieve is a lookup for each identification number to identify the departement where he belongs using the report filename and store it on the approriate column.
Any help will be appreciated.
Assuming you're using the Excel File Input step, there is an option on the Additional Output Fields tab that will allow you to specify the Full Filename Field. You can name this whatever you want, and it will add an additional column to your incoming Excel data that has the name of the file as one of the columns. You may need to do some regex cleanup on that fields since it's the full file path, not just the filename.
As far as doing the lookup, there are many lookup options to merge streams in the Lookup category of the design tab. I think the Stream Lookup is the step you'll want.
As far as I understood your need, you have to first build a "mapping table" of two columns: the department (aka the start of the xls filename) and the employee (aka its ID).
This table does not need to be materialized and may stay in a step of the the PDI. So
Read all the xls files with a Microsoft Excel File. In case you do not know how to do it: Browse to any of these file, press the Add button, then in the Selected files table, remove the filename to keep only its directory path and write .*\.xls in the Regex wildcard. Check you select the appropriates files with the Show filename button.
In the same step, define the Sheet to be "Fiche technique" (assuming they are all the same). Define the field to be "A" with type String (an empty column) and "ID" also with type String (otherwise you'll have a un-trappable error on "Agent ID" and "Total". Also follow #eicherjc suggestion and keep the filename, although I suggest you keep the Short file name and call it filename.
You should get a two column stream: ID and filename, which need some bit of data massage before to be used. The ID contains non-integer fields and the file name contains extra characters.
The simplest way to do this is with a Modified Javascript Value. I may suggest the code:
var ID = Number(ID);
var regex = filename.match(/(.*)__\d+\.xls/);
if(regex) filename = regex[1];
and do not forget specify the the ID has now a type Integer and to put a "Y" in the Replace value in field of the Fields`` table at the bottom.
The first line will convert any number in its value, and non-number in a 0, which is an ID that does not exists.
The next lines will extract the department from the filename with a Regex. If you do not like regex, you may use a filename = filename.substr(0, filename.indexOf('__')), or any formula that will do the job.
Now you have a stream ready to be used, except that some employees may, right or wrong, be in more than one department. If it does not matter which one, then leave it like that. Otherwise you have to provide some logic to filter the correct department.
You can now use a Lookup Stream to read the department of each employee. The Lookup step is the Modified Javascript value (or whatever name you gave to this step). The field to lookup is the field of the ID in your mySql. The Lookup field is the ID (or whatever name you gave to the column B of your xls files). And the field to retrieveenter code here is the filename (or more precisely, the department name extracted from the filename).
I am working on Excel file to load the data into sql table using SSIS VS2013.
How do I extract the column names into a temptable?
In the image attached,there is the "2017 Marketing Sales - Actual" column in the first row, I want to extract the YYYY from the column name and pass that value to a variable and update the table field with YYYY info.
Can you anyone help me how to do this?
In your Excel Connection Manager, uncheck the "First row has column names" check box. This should allow you to access that first row. You'll need to setup a Data Flow Task using this Excel Connection Manager, followed by a derived column to extract the left 4 characters from that 1st row. Unless you somehow limit the rest of the Excel data source, you'll probably also get a lot of extraneous rows importing into your destination. Might need to do some clean up to get it down to just that year.
I have a file which consists of a few part numbers.Using this file i need to exclude data in dashboard in another table which also has part numbers.How to filter data out of the table based on the part numbers present in the file if the part numbers in the file can change over time?
When you import the file with a list of part numbers, add a calculated column under transformations (also make sure that it's not reading the first record of your part list file as a header row--I don't know what your file looks like). In the expression box, just enter something simple like 1. Call this new dataset something like part_list. This column represents a flag that we will add to the table that is already in your dashboard. Let's suppose that table is called data.
Once the file is imported, click Insert > Columns... and ensure that data is selected in the "Add columns to data table:" drop down box, and that part_list is selected in the "Add columns from:" menu. Click Next. Match the part number column in both tables, and click next. Add the flag column to data with a left outer join (assuming this makes sense with your data). Once the column is added, you can filter out the 1's.
If this does not answer your question, consider providing more details about what your data looks like.