I would like to save Transformation as a file. For that purpose I use Kettle REST Api Command POST /kettle/registerTrans/?xml=Y
Please see:
https://help.hitachivantara.com/Documentation/Pentaho/8.1/Developer_Center/REST_API/Carte/020
The Transformation is registered and saved in PDI Database, so I can execute it via Web GUI (please see attached image).
According to documentation, the *.ktr transformation should be saved. In spite the configured repository as a file repository, the transformation is not saved and can't be executed with the REST Command GET /kettle/executeTrans/?rep=my_repository&user=my_user&pass=my_password&trans=my_trans&level=INFO
If I add manually the file to pre-defined repository folder, the transformation can be executed with REST API Command mentioned below.
Here is my repositories.xml
<?xml version="1.0" encoding="UTF-8"?>
<repositories>
<repository id="KettleFileRepository">
<id>KettleFileRepository</id>
<name>bi_source</name>
<description>File repository</description>
<is_default>false</is_default>
<base_directory>/home/pentaho/repositories/bi_source</base_directory>
<read_only>N</read_only>
<hides_hidden_files>N</hides_hidden_files>
</repository>
</repositories>
Please tell me, what I am doing wrong and why cannot I save the persistent file.
Related
Background
I was planning to use S3 to store the Flink's checkpoints using the FsStateBackend. But somehow I was getting the following error.
Error
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
Flink version: I am using Flink 1.10.0 version.
I have found the solution for the above issue, so here I am listing it in steps that are required.
Steps
We need to add some configs in the flink-conf.yaml file which I have listed below.
state.backend: filesystem
state.checkpoints.dir: s3://s3-bucket/checkpoints/ #"s3://<your-bucket>/<endpoint>"
state.backend.fs.checkpointdir: s3://s3-bucket/checkpoints/ #"s3://<your-bucket>/<endpoint>"
s3.access-key: XXXXXXXXXXXXXXXXXXX #your-access-key
s3.secret-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx #your-secret-key
s3.endpoint: http://127.0.0.1:9000 #your-endpoint-hostname (I have used Minio)
After completing the first step we need to copy the respective(flink-s3-fs-hadoop-1.10.0.jar and flink-s3-fs-presto-1.10.0.jar) JAR files from the opt directory to the plugins directory of your Flink.
E.g:--> 1. Copy /flink-1.10.0/opt/flink-s3-fs-hadoop-1.10.0.jar to /flink-1.10.0/plugins/s3-fs-hadoop/flink-s3-fs-hadoop-1.10.0.jar // Recommended for StreamingFileSink
2. Copy /flink-1.10.0/opt/flink-s3-fs-presto-1.10.0.jar to /flink-1.10.0/plugins/s3-fs-presto/flink-s3-fs-presto-1.10.0.jar //Recommended for checkpointing
Add this in checkpointing code
env.setStateBackend(new FsStateBackend("s3://s3-bucket/checkpoints/"))
After completing all the above steps re-start the Flink if it is already running.
Note:
If you are using both(flink-s3-fs-hadoop and flink-s3-fs-presto) in Flink then please use s3p:// specificly for flink-s3-fs-presto and s3a:// for flink-s3-fs-hadoop instead of s3://.
For more details click here.
I am attempting to set my zeppelin.bigquery.project_id (or any bigquery configuration property) via my zeppelin-site.xml, but my changes are not loaded when I start Zeppelin. The project ID always defaults to ' '. I am able to change other configuration properties (ex. zeppelin.notebook.storage). I am using Zeppelin 0.7.3 from https://hub.docker.com/r/apache/zeppelin/.
zeppelin-site.xml (created before starting Zeppelin, before an interpreter.json file exists):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.S3NotebookRepo</value>
<description>notebook persistence layer implementation</description>
</property>
... etc ...
<property>
<name>zeppelin.bigquery.project_id</name>
<value>my-project-id</value>
<description>Google Project Id</description>
</property>
</configuration>
Am I configuring the interpreter incorrectly? Could this parameter be overridden elsewhere?
I am not really familiar with Apache Zeppelin, but I have found some documentation pages that make me think that you should actually store the BigQuery configuration parameters in your Interpreter configuration file:
This entry in the GCP blog explains how to use the BigQuery Interpreter for Apache Zeppelin. It includes some examples on how to use it with Dataproc, Apache Spark and the Interpreter.
The BigQuery Interpreter documentation for Zeppelin 0.7.3 mentions that zeppelin.bigquery.project_id is the right parameter to configure, so that is not the issue here. Here there is some information on how to configure the Zeppelin Interpreters.
The GitHub page of the BigQuery Interpreter states that you have to configure the properties during Interpreter creation, and then you should enable is by using %bigquery.sql.
Finally, make sure that you are specifying the BigQuery interpreter in the appropriate field in the zeppelin-site.xml (like done in the template) or instead enable it by clicking on the "Gear" icon and selecting "bigquery".
Edit /usr/lib/zeppelin/conf/interpreter.json, change zeppelin.bigquery.project_id to be the value of your project and run sudo systemctl restart zeppelin.service.
I am trying to define one global configuration package for all my .dtsx files.
I have a login there:
<Configuration ConfiguredType="Property" Path="\Package.Connections[SourceConnectionOLEDB].Properties[UserName]" ValueType="String">
<ConfiguredValue> exampleLoginHere </ConfiguredValue>
</Configuration>
This login appears in many places.
So, what i'm trying to do is to set this login into variable and change only in one place instead of do that in all occurrences.
I found This solution but when i put
<!DOCTYPE DTSConfiguration [
<!ENTITY sourceLogin "exampleLoginHere">
]>
and then change
<ConfiguredValue> exampleLoginHere </ConfiguredValue>
to
<ConfiguredValue> &sourceLogin; </ConfiguredValue>
my dtsx after start return:
Warning: Cannot load the XML configuration file. The XML configuration file may be malformed or not valid
Am I doing something wrong? I forgot about something?
Package Configuration files are nothing but regular xml files and the rules that apply for them should be working for this as well. Having said that to address your need of having a variable across multiple packages you can set it up as a 'Indirect Configuration' and have the value coming from a 'SQL Server' table. Here is a link that gives more detailed breakdown of how it works -
http://bi-blogger.typepad.com/etlbi_blogger/2008/05/using-indirect-configuration-with-ssis.html
I'm trying to deploy Mule project to cloudhub using command prompt. I'm getting the below error:
" Illegal path attribute "/nexus". Path of origin: "/snapshots/org/mule/modules/mule-module- cloudhub/3.0.0-SNAPSHOT/mule-module-cloudhub-3.0.0-20140909.225040-1.pom"
I have tried to remove repository for nexus, but I got the same error. Also referred the url
Connection to http://dev.ee.mulesource.com refused: Operation timed out
Why do Mule Studio 3.4 Builds with Maven Fail or Hangs Indefinitely?
But, when I try to deploy using anypoint GUI, it is being deployed in cloudhub. Problem arises when it comes to console.
Please find my pom repository for Nexus
<repository>
<id>mule-ee-releases</id>
<name>Mule Release Repository</name>
<url>https://repository-master.mulesoft.org/nexus/content/repositories/ci-releases/</url>
</repository>
</repositories>
Settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<pluginGroups>
<pluginGroup>org.mule.tools</pluginGroup>
</pluginGroups>
<servers>
<server>
<id>cloudhub.io</id>
<username>my_UserName</username>
<password>PWD</password>
</server>
</servers>
</settings>
Looking at the screenshot, it seems you are trying to deploy your project to https://repository-master.mulesoft.org/snapshots/ You apparently haven't provided the credentials to connect to this server so you can't upload your build artifacts there.
You need to either:
provide credentials that authorize you to deploy to repository-master.mulesoft.org, via: https://maven.apache.org/settings.html#Servers
deploy to another repository where you are authorized to do so, via: https://maven.apache.org/pom.html#Distribution_Management
If you are using a reverse proxy and you have different context paths on either side of it you'll need to map cookies as well.
For Apache you can use:
ProxyPassReverseCookiePath /nexus /
For Nexus you can use:
proxy_cookie_path /nexus /;
I have configured a SVN file in TeamCity server. And also set the property "Trigger a build on each check-in". So the file Build is triggered after every check-in is happened. Now I want to access the build information of configured SVN files while build is triggering.
I have a web space in Mediawiki. Mediawiki has some api to edit/create pages. I wish to call Mediawiki api with SVN build information from TeamCity server itself.
Is there any way to do this ?
what is the HSQL2 database in Teamcity server. Can we use this DB to access build information ?
You can use the TeamCity REST API to get the build changes, example:
curl /httpAuth/app/rest/changes?build=id:%teamcity.build.id%
As a command line build step will yield something like:
<changes count="3">
<change href="/httpAuth/app/rest/changes/id:217404" id="217404" version="b6b97a0d7789d97df6df908ab582ebb723235f43" webLink="http://teamcity.my-domain.com/viewModification.html?modId=217404&personal=false"/>
<change href="/httpAuth/app/rest/changes/id:217403" id="217403" version="064ecef5552ec2fb7875d7c26fe54cdf94b67bec" webLink="http://teamcity.my-domain.com/viewModification.html?modId=217403&personal=false"/>
<change href="/httpAuth/app/rest/changes/id:217402" id="217402" version="9bc3a34c952059bbfc7bebeb79ba5a3c894ef555" webLink="http://teamcity.my-domain.com/viewModification.html?modId=217402&personal=false"/>
</changes>
Then you can loop through the change and hit its href which will yield like:
<change date="20130918T133404-0600" username="welsh" href="/httpAuth/app/rest/changes/id:217397" id="217397" version="51e925e354a83deccde881b30a76974b2ff745f4" webLink="http://teamcity.my-domain.com/viewModification.html?modId=217397&personal=false">
<comment>
My comments are here
</comment>
<files>
<file before-revision="90acd4da1972814094c22e3020c5073521a7b640#141323126c0" after-revision="51e925e354a83deccde881b30a76974b2ff745f4#1413290abe0" file="grails-app/views/layouts/global.gsp" relative-file="grails-app/views/layouts/global.gsp"/>
</files>
<user href="/httpAuth/app/rest/users/id:1" id="1" name="Welsh" username="welsh"/>
</change>
Then you can create something to take this information to push it to your media wiki.