I have read the document and reference about the Apache Flink 1.6 and want to find some api like Spark HiveContext to read data from Hive, but only find the HDFS api and JDBC api. Is there any api as same as Spark HiveContext to connect Hive ?
You can use HCatInputFormat like so:
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.createInput(new HCatInputFormat<>(database, table));
You probably need to add the dependency too:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hcatalog</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
From Flink 1.9.0, we are officially supporting using Flink to read from and write to Hive. https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/
Are you still looking into this option? Feel free to reach out to me personally
Related
I am working on a feature to upload/resume/pause files to amazon S3 bucket and using following dependency (among others) in pom.xml for these features,
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
<version>2.19.1</version>
</dependency>
I came across this link (though its for aws android sdk), it mentions adding listener to persist the state transferManager.upload(putRequest, new S3ProgressListener() {
I am using S3TransferManager in my implementation and thought of adding transfer listener to UploadFileRequest.Builder to achieve whats mentioned in first link but could not find any references to do so. Are there any references for this? Please advise.
I am using zerocode-rest-bdd maven lib with following version. I have my applications host and port defined in the JUnit runner with "#TargetEnv("app_host.properties")".
<dependency>
<groupId>org.jsmart</groupId>
<artifactId>zerocode-rest-bdd</artifactId>
<version>1.2.15</version>
</dependency>
I want to access more hosts/ports(boundary application IPs) and common tokens(SAML, OAuth etc) into my JSON test case which I am unable to access using ${{app_host}}.
Is there any other way to extend or configure these extra properties so that I can access them and validate my boundary contracts?
app_host.properties contains:
web.application.endpoint.host=https://api.github.local
web.application.endpoint.port=443
web.application.endpoint.context=
#Can not access these below properties
app_host_1=https://app1.host.local.uk
saml_token=<SAML>sdf-wer</SAML>
Accessing in the test case as below:
"url": "${app_host_1}/users/u123",
Seems like you are using an older version of Zerocode lib. You can update to <version>1.2.17</version> or to the latest as below.
<dependency>
<groupId>org.jsmart</groupId>
<artifactId>zerocode-tdd</artifactId>
<version>1.3.x</version>
</dependency>
Then, once you define your properties e.g.
micro_host_1=https://micro.host1.local.uk
xyz_key=abc_value
the values can be resolved via ${micro_host_1} or ${xyz_key} to, https://micro.host1.local.uk or abc_value respectively in the test cases.
Hope this helps!
I am using Spring 2.0.1.RELEASE and have setup all projects (2 services and the cloud config server) with spring-cloud-bus
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
</dependency>
The config server also has the spring-cloud-config-monitor
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-monitor</artifactId>
</dependency>
I edit a file in my Git reposiroty (using local files with native profile of Spring Cloud Config). The change is detected, and I see the following line in the
Cloud Config Server:
17:59:25.201 [task-scheduler-3] INFO o.s.cloud.bus.event.RefreshListener - Received remote refresh request. Keys refreshed [version.client.min]
However, none of the other services receive the notification about updated keys.
On the other hand, if I manually call the bus-refresh endpoint of any other service, I see that all modules receive the updated key. The config server itself also receives the notification, but it says that there is no key updated, which makes sense since it already detected the change.
The documentation did not mention any special property to set apart from the RabbitMQ properties (which seem to be well configured since the bus-refresh endpoint is working as expected.)
I saw that there are already a few posts about this, one is even pointing to a bug that has been marked as resolved (https://github.com/spring-cloud/spring-cloud-bus/issues/101) but it does not seem to be working on my side.
Any property to enable for the config server to notify the bus?
Any hint regarding how to debug this?
Easy fix (after a lots of research!)
Changed all dependencies of org.springframework.cloud from FINCHLEY.M9 to 2.0.0.RC1 and suddenly, everything started working!
Perhaps your bootstrap.properties file aren't getting loaded on project startup
So a few things firsthand:
If you're using, (in your Cloud Config project),
<spring-cloud.version> 2020.0.0 (to be found under <DependencyManagement> or either specified in <properties>)
Then Spring Boot versions lower then 2.4.1 (in the same project) will not start up the cloud config server under its default dependencies.
So if you áre using the above versions, and maybe versions above,
Then for the projects that need updating by cloud Bus (using bootstrap.properties for instance) should contain the Starter Bootstrap dependency (of course along with the cloud-starter-config and the bus-amqp dependency)
<!-- Spring Cloud Config + Cloud Bus + Bootstrap-->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
Plus check that the Spring cloud version is 2020.0.0-M6 or Hoxton.BUILD-SNAPSHOT depending on which Spring Boot version you're using. Here's a screenshot of which Spring Cloud versions are compatible with which Spring Boot versions
v v v v v v v
I have multiple Resource class and i don't know if i can use #SwaggerDefinition in all of them so i am trying to write a custom bootstrap servlet
I am fallowing the example in the fallowing link
Bootstrap for swagger-jersery2
and i am not able to find SwaggerContextService class in any package .
So i tried
BeanConfig beanConfig = new BeanConfig();
beanConfig.configure(swagger);
even that is not working
My maven dependency looks like this
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-jersey2-jaxrs</artifactId>
<version>1.5.0</version>
</dependency>
can some one help me with this
It looks like you are using a much older version of swagger. SwaggerContextService was added in 1.5.7 and the latest stable version is 1.5.13.
I have made new application Mobilefirst 8.0 using CLI.
I have followed Link to create sql adapter.
Added SQL databse file in root folder of project (root folder/Utils) is this right coz in 7.1 we have to add sql file under server folder.
Also added jdbc lib file in root folder. But when I trying to invoke adapter I am getting "Exception was thrown while invoking procedure: getAccountTransactions2 in adapter: SampleAdapter SQL connection creation failed" in logs.
Can someone let me know what I am doing wrong. Below is my code uploaded in drive.
Code here
Edit: I see what you did... you got it all wrong. You cannot include a "database file" in the application and expect the adapter to "connect" to this database.
Here is a diagram:
[application] ----> [mobilefirst server][adapter] ---> [database].
The application sends a request to the server to call the adapter which will send a request to the database, and then the response propagates back until it reaches the application, which send the original request.
You need to run your database in an actual server, not in the application.
Added SQL databse file in root folder of project (root folder/Utils) is this right coz in 7.1 we have to add sql file under server folder.
I assume you are referring to the connector driver... This is wrong.
In v8.0 you add the connector as a Maven dependency in the adapter's pom.xml file.
Learn about maven dependencies:
https://mobilefirstplatform.ibmcloud.com/tutorials/en/foundation/8.0/adapters/creating-adapters/#dependencies
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
Depending on your database type, search for a connector in the maven repository site: http://search.maven.org/
Once you found it, add its reference to the pom.xml file and re-build your adapter.
Be sure though to add the correct values for your database, in the adapter.xml file (URL to database, username, password, ...).
For example for MySQL:
pom.xml
<dependencies>
<dependency>
<groupId>com.ibm.mfp</groupId>
<artifactId>adapter-maven-api</artifactId>
<scope>provided</scope>
<version>[8.0.0,9.0.0)</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.6</version>
</dependency>
</dependencies>
adapter.xml
...
...
<dataSourceDefinition>
<driverClass>com.mysql.jdbc.Driver</driverClass>
<url>jdbc:mysql://localhost:3306/mobilefirst_training</url>
<user>mobilefirst</user>
<password>mobilefirst</password>
</dataSourceDefinition>