I am using PDI 8.2 CE and Penatho BI Server 8.2CE. Transformation works fine locally where the .ktrs and files are saved in local folders, using ${Internal.Entry.Current.Directory} to map a text file that is used in the ktr. However once this is uploaded to the Penatho BI Server (saving the text file in the same folder as the ktr) and try to run it the ${Internal.Entry.Current.Directory} is resolved as "file:///C:/public/ErrorHandlingSample/UnitTestData_MultiType.txt" instead of the relative repository directory !!!
I don't know what change can make it working again? If anyone hit the same issue please help...
Thanks heaps..
Some logs below,
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - An error occurred while creating field mapping
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - Could not read from "file:///C:/public/ErrorHandlingSample/UnitTestData_MultiType.txt" because it is not a file.
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 -
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - Could not read from "file:///C:/public/ErrorHandlingSample/UnitTestData_MultiType.txt" because it is not a file.
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 -
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.steps.csvinput.CsvInput.readFieldNamesFromFile(CsvInput.java:463)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.steps.csvinput.CsvInput.createFieldMapping(CsvInput.java:427)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.steps.csvinput.CsvInput.openNextFile(CsvInput.java:332)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.steps.csvinput.CsvInput.processRow(CsvInput.java:135)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at java.lang.Thread.run(Thread.java:748)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - Caused by: org.apache.commons.vfs2.FileNotFoundException: Could not read from "file:///C:/public/ErrorHandlingSample/UnitTestData_MultiType.txt" because it is not a file.
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1123)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.apache.commons.vfs2.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:349)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.core.vfs.KettleVFS.getInputStream(KettleVFS.java:263)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.pentaho.di.trans.steps.csvinput.CsvInput.readFieldNamesFromFile(CsvInput.java:443)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - ... 5 more
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - Caused by: java.io.FileNotFoundException: C:\public\ErrorHandlingSample\UnitTestData_MultiType.txt (The system cannot find the path specified)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at java.io.FileInputStream.open0(Native Method)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at java.io.FileInputStream.open(FileInputStream.java:195)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at java.io.FileInputStream.<init>(FileInputStream.java:138)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - at org.apache.commons.vfs2.provider.local.LocalFile.doGetInputStream(LocalFile.java:215)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0
at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1119)
2019/08/13 13:02:29 - Scenario 1 Input Data 2.0 - ... 8 more
The Pentaho repository is not intended to be a store of any file type other than Pentaho reports, and Pentaho transformations/jobs. The ${Internal.Entry.Current.Directory} variable will only work to reference things like text files while in the Windows/Mac/Linux file systems. While inside the Pentaho Server, that variable is used to reference other transformations or jobs, it won't work to reference things like text files. You should create a direct path to the text file that the Pentaho Server JVM can access.
Related
While using kafka input, I want to output only when json data contains a specific string.
I tried setting "include_lines" in filebeat.yml, but it was not filtered properly.
When the filebit.yml setting is as follows and data-set1 and 2 are input, not only data-set1 but also data-set2 are output.
I expected only data-set 1 to be output, but it wasn't.
What did I make a mistake?
part of the filebeat.yml
filebeat.inputs:
- type: kafka
hosts:
- qa-parkbae-01.hanpda.com:9092,
- qa-parkbae-02.hanpda.com:9092,
- qa-parkbae-03.hanpda.com:9092
topics: ["parkbae-test-topic1"]
group_id: "test123"
ssl.enabled: false
include_lines: ['\"event\":\"basket\"']
input data-set1 :
{"id":"parkbae","event":"basket","data":"test1"}
input data-set2 :
{"id":"parkbae","event":"ball","data":"test2"}
This question already has an answer here:
How do I invoke my JS Function for a particular column in a csv file in the Karate Framework?
(1 answer)
Closed 1 year ago.
#noinspection CucumberTableInspection
Feature: Serverless
Background:
* def data = read('input.csv')
* csv data = data
* def isValid = function(x){ return (x.name && x.unit) ? x.cost.length != 0 : true }
Scenario Outline:
* assert isValid(__row)
Examples:
| data |
The test runs but doesn't check anything.
Karate version: 1.1.0.RC4
elapsed: 1,77 | threads: 1 | thread time: 0,00
features: 0 | skipped: 1 | efficiency: 0,00
scenarios: 0 | passed: 0 | failed: 0
HTML report: (paste into browser to view) | Karate version: 1.1.0.RC4
===================================================================
Process finished with exit code 0
The * csv data = data is not needed and was used to explain something totally different earlier. When "read" a CSV becomes a JSON array automatically: https://github.com/intuit/karate#csv-files
If you are new to Karate I recommend you don't use CSV files and the Scenario Outline. Please just get used to Examples first and then move on from there. You are combining 3 advanced topics - CSV, dynamic outlines and conditional assertions. Do your tests really need to be so clever and complex ? Read this please: https://stackoverflow.com/a/54126724/143475
And please please read the docs and the examples: https://github.com/intuit/karate#dynamic-scenario-outline
I'm trying to read an Access database file, .mdb file with Spoon,
using Microsoft Access Input in Design tab.
But it is not working.
When I press Get Tables in Content tab I get this error: Looking for usage map at page 32000, but page type is 0
If I put one table and press preview rows I get this error:
019/08/27 11:35:32 - Spoon - Spoon
2019/08/27 11:52:22 - /Transformación 1 - Iniciado despacho de la transformación [/Transformación 1]
2019/08/27 11:52:22 - Microsoft Access Input.0 - ERROR (version 8.0.0.0-28, build 8.0.0.0-28 from 2017-11-05 07.27.50 by buildguy) : Couldn't open file #1 : file:///home/pentaho/Escritorio/general.mdb --> java.io.IOException: Looking for usage map at page 32000, but page type is 0
2019/08/27 11:52:22 - Microsoft Access Input.0 - Procesamiento finalizado (I=0, O=0, R=0, W=0, U=0, E=1)
2019/08/27 11:52:22 - /Transformación 1 - Transformación detectada
2019/08/27 11:52:22 - /Transformación 1 - Transformación esta matando los otros pasos!
Thank you.
I am trying to obtain a list of all the DB Tables that will give me visibility on what tables I may need to JOIN for running SQL scripts.
For example, in TCL when I run "LIST.DICT" it returns "Name of File:" for input. I then enter "PRODUCT" and it returns a list of all available fields.
However, Where can I get a list of all my available Tables or list of my options that I can enter after "Name of File:"?
Here is what I am trying to achieve. In the screen shot below, I would like to run a SQL script that gives me the latest Log File Activity, Date - Time - Description. I would like the script to return '8/13/14 08:40am BR: 3;BuyPkg'
Thank you in advance for your help.
From TCL within the database account containing your database files, type: LISTF
Sample output:
FILES in your vocabulary 03:21:38pm 29 Jun 2015 Page 1
Filename........................... Pathname...................... Type Modulo
File - Contains all logical device names
DICT &DEVICE& /u1/uv/D_&DEVICE& 2 1
DATA &DEVICE& /u1/uv/&DEVICE& 2 3
File - Used by MAKE.MAP.FILE
DICT &MAP& /u1/uv/D_&MAP& 2 1
DATA &MAP& /u1/uv/&MAP& 6 7
File - Contains all parts of Distributed Files
DICT &PARTFILES& /u1/uv/D_&PARTFILES& 2 1
DATA &PARTFILES& /u1/uv/&PARTFILES& 18 7
DICT &PH& D_&PH& 3 1
DATA &PH& &PH& 1
DICT &SAVEDLISTS& D_&SAVEDLISTS& 3 1
DATA &SAVEDLISTS& &SAVEDLISTS& 1
File - Used by uniVerse to access the current directory.
DICT &UFD& /u1/uv/D_UFD 2 1
DATA &UFD& . 19 1
DICT &XML& D_&XML& 18 3
DATA &XML& &XML& 19 1
Firstly, UniVerse has no Log File Activity Date and Time.
However, you can still obtain the table's modified/ accessed date from the file system however.
To do this,
You need to have a subroutine accepting a path of the table to return a date or a time.
e.g. SUBROUTINE GET.FILE.MOD.DATE(DAT.MOD, S.FILE.PATH)
Inside the subroutine, you can use EXECUTE to run shell command like istat for getting these info on a unix e.g.
Please beware that for a dynamic file e.g. there are Data and Overflow parts under a directory. You should compare the dates obtained and return only the latest one.
Globally catalog the subroutine
Create an I-Desc in VOC, e.g. I.FILE.MOD.DATE in the field definition of this I-Desc: SUBR("*GET.FILE.MOD.DATE",F2) and Conv Code as "D/MDY2"
Create another I-Desc e.g. I.FILE.MOD.TIME
Finally, you can
LIST VOC I.FILE.MOD.DATE I.FILE.MOD.TIME DESC WITH TYPE LIKE "F..."
alternatively in SQL,
SELECT I.FILE.MOD.DATE, I.FILE.MOD.TIME, VOC.DESC FROM VOC WHERE TYPE LIKE "F%";
I'm trying to setup Awstats to formate Amazon S3 log files,but it keeps saying the log doesn't match the LogFormat, below is the configuration and log content:
LogFormat="%other %extra1 %time1 %host %logname %other %method %url
%otherquot %code %extra2 %bytesd %other %extra3 %extra4 %refererquot
%uaquot %other"
0dfbd34f831f30a30832ff62edcb8a93158c056f27cebd6b746e35309d19039c
looxcie-data1 [18/Dec/2011:04:30:15 +0000] 75.101.241.228
arn:aws:iam::062105025988:user/s3-user E938CC6E4B848BEA
REST.GET.BUCKET - "GET
/?delimiter=/&prefix=data/prod/looxciemp4/0/20/&max-keys=1000
HTTP/1.1" 200 - 672 - 44 41 "-" "-" -
Then I execute the command and get following result:
root#test:/usr/local/awstats/wwwroot/cgi-bin# perl awstats.pl -update - config=www.awstats.apache.com
Create/Update database for config "/etc/awstats/awstats.www.awstats.apache.com.conf" by AWStats version 7.0 (build 1.971)
From data in log file "/var/log/httpd/access.log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Jumped lines in file: 0
Parsed lines in file: 1
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 1 corrupted records,
Found 0 old records,
Found 0 new qualified records.
Can anyone help to figure it out?
===========================================
I found that the format "%logname" can not match such name
arn:aws:iam::062105025988:user/s3-user
It is wired, but "%lognamequot" is able to match "arn:aws:iam::062105025988:user/s3-user";
This is the cause of this problem;
But our system log file does include logname like arn:aws:iam::062105025988:user/s3-user;
Is there anyone can help to figure it out why it doesn't match it?