Schemaless Bigquery kafka sink connector SMT not able to save data to bigquery on boolean.
MapsUtil.debugPrint on recordValue before returning from apply(R record).
active = true java.lang.String
Schema definition
{
"mode": "NULLABLE",
"name": "active",
"type": "BOOLEAN"
}
Deserialiser
public class BooleanDeserialiser extends JsonDeserializer<Boolean> {
#Override
public Boolean deserialize(JsonParser parser, DeserializationContext context)
throws IOException {
return !"0".equals(parser.getText());
}
Serialiser
public class BooleanSerialiser extends JsonSerializer<Boolean> {
#Override
public void serialize(Boolean value, JsonGenerator gen, SerializerProvider serializers)
throws IOException {
gen.writeString(value ? "true" : "false");
}
Error
[row index 76]: invalid: Cannot convert string value to boolean: 1
at com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor.maybeThrowEncounteredErrors(KCBQThreadPoolExecutor.java:108)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.put(BigQuerySinkTask.java:233)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:539)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:322)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I'm leaving this as community wiki response for community visibility for this kind of questions. As mention on your question made to the connector devs, your request may need a feature implemented on bigquery that needs to be requested to by opening a BigQuery Feature Request. This will bring this kind of requests into developers spotlight so it can be taken into consideration.
It's worth to mention, you can see the list of issues/bugs for the confluentinc kafka-connect-bigquery on the project issue dashboard.
Related
I have an apache geode setup where there is one locator and one Server. we have a region employee in that. we were trying to implement in-line cache where a cache miss will lookup into database and will fill apache geode, but after deployment of Jars when i am trying to alter the region . It shows exception
Stack Trace:
[error 2021/04/09 15:18:30.513 IST <Function Execution Processor2> tid=0x3a] Error instantiating class: <com.abc.geode.ApacheGeode.EmployeeCacheLoader>
java.lang.RuntimeException: Error instantiating class: <com.abc.geode.ApacheGeode.EmployeeCacheLoader>
at org.apache.geode.management.internal.configuration.domain.DeclarableTypeInstantiator.newInstance(DeclarableTypeInstantiator.java:43)
at org.apache.geode.management.internal.cli.functions.RegionAlterFunction.alterRegion(RegionAlterFunction.java:202)
at org.apache.geode.management.internal.cli.functions.RegionAlterFunction.executeFunction(RegionAlterFunction.java:67)
at org.apache.geode.management.cli.CliFunction.execute(CliFunction.java:37)
at org.apache.geode.internal.cache.MemberFunctionStreamingMessage.process(MemberFunctionStreamingMessage.java:201)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:372)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:436)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:475)
at org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:393)
at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassCastException: com.abc.geode.ApacheGeode.EmployeeCacheLoader cannot be cast to org.apache.geode.cache.Declarable
at org.apache.geode.management.internal.configuration.domain.DeclarableTypeInstantiator.newInstance(DeclarableTypeInstantiator.java:34)
public class EmployeeCacheLoader implements CacheLoader<Long,Employee>, Declarable {
#Override
public Employee load(LoaderHelper<Long, Employee> helper) throws CacheLoaderException {
Employee e=new Employee();
e.setEmail("a#b.com");
e.setIdemployee(2L);
return e;
}
#Override
public void close() {
}
#Override
public void init(Properties props) {
}
Things tried
I tried by not implementing Declarable but still no success.
I have not done any change for serializer.
I've just tried the scenario using Apache Geode 1.13.2 and it works just fine, you can find the example here. Do you have multiple versions of the same jar within the server's class path?, that might be the reason for the exception.
Cheers.
In a sample Java servlet project in my IntelliJ IDE, i have created this class in Kotlin :
#WebServlet(name = "TwitterAPIServlet", description = "This is used to test the servlet api", urlPatterns = ["/twitterAPIServlet"])
class TwitterAPIServlet : HttpServlet() {
#Throws(IOException::class)
override fun doGet(req: HttpServletRequest, resp: HttpServletResponse) {
// Print answer
val out = resp.writer
out.println("Request Done : </br>")
}
}
When i am trying to call this with my .jsp page, i have this error :
2020-05-10 15:30:34.218:WARN:oejs.HttpChannel:qtp60559178-27: /twitterAPIServlet
java.lang.NoClassDefFoundError: kotlin/jvm/internal/Intrinsics
at main.java.co.rezo.api.internal.v1.TwitterAPIServlet.doGet(TwitterAPIServlet.kt)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
When i am trying the same code in Java, it's working :
#WebServlet(
name = "TwitterAPIServlet",
description = "This is used to test the servlet api",
urlPatterns = "/twitterAPIServlet")
public class TwitterAPIServlet extends HttpServlet {
#Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
PrintWriter out = resp.getWriter();
out.println("Request Done : </br>");
}
}
What can i do to make my Kotlin code working?
Most probably the issue is that Kotlin's standard library is not included in the generated WAR (or is not on a classpath if you run the exploded app). Make sure to include it in your build.
Instructions for Gradle can be found in the official examples repository: kotlin-examples.
Maven is similar: https://kotlinlang.org/docs/reference/using-maven.html.
In case of pure IDEA (though it's recommended to use one of the build tools above), check the projects settings:
You should see Kotlin's stdlib included in the artifact:
I've been developing a RESTful web service using Web API 2 and OWIN. Originally my controllers inherited from ApiController, and the GET actions supported OData filtering/queries i.e. marked with [EnableQuery].
We've now decided to look at whether it's feasible to expose a true OData service, and so make our controllers inherit from ODataController instead of ApiController. While this seems to be working well for the most part, $select no longer works.
public class BaseController : ODataController
{
... some properties here, not related to issue...
}
public class EmployeesController : BaseController
{
private readonly AppDbContext _context = new AppDbContext();
[EnableQuery]
public IQueryable<Employee> Get()
{
return _context.Employees;
}
...
}
The error I'm seeing is:
{
"error": {
"code": "",
"message": "An error has occurred.",
"innererror": {
"message": "'DbQuery`1' cannot be serialized using the ODataMediaTypeFormatter.",
"type": "System.Runtime.Serialization.SerializationException",
"stacktrace": " at System.Web.OData.Formatter.ODataMediaTypeFormatter.GetSerializer(Type type, Object value, ODataSerializerProvider serializerProvider)\r\n at System.Web.OData.Formatter.ODataMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, HttpContent content, HttpContentHeaders contentHeaders)\r\n at System.Web.OData.Formatter.ODataMediaTypeFormatter.WriteToStreamAsync(Type type, Object value, Stream writeStream, HttpContent content, TransportContext transportContext, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Owin.HttpMessageHandlerAdapter.<BufferResponseContentAsync>d__13.MoveNext()"
}
}
}
I'm puzzled how this works with ApiController, but not with ODataController! Is there something I'm missing?
Cheers!
Ok, I've figured out what was happening:
I tried to update my OData packages to the latest version for OData v4 support. The Microsoft .Net OData library namespaces have changed between OData v3 (System.Web.Http.OData) and v4 (System.Web.OData). I had somehow managed to mix these libraries in such a way that I was referencing the EnableQuery attribute in the old OData library, which caused the serialization issue.
Not an obvious problem to track down - it's not obvious what's happening when the attributes have the same name but are in different namespaces and in fact belong to entirely different versions!
I have put in the password which is "root" and it keeps popping back up. How can I suppress this or get rid of it. I am using spring boot and spring security.
application.properties
spring.datasource.driverClassName=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/springbootpractice
spring.datasource.username=root
spring.jpa.database = MYSQL
spring.jpa.show-sql = true
# Hibernate
hibernate.dialect: org.hibernate.dialect.MySQL5Dialect
hibernate.show_sql: true
hibernate.hbm2ddl.auto: update
entitymanager.packagesToScan: /
I am using intellij 14 if that matters.
----Update 1-----
#Configuration
#EnableWebMvcSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
#Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/", "/index").permitAll()
.anyRequest().authenticated()
.and()
.formLogin()
.loginPage("/index")
.permitAll()
.and()
.logout()
.permitAll();
}
#Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/", "/index").permitAll()
.anyRequest().authenticated()
.and()
.formLogin()
.loginPage("/index")
.permitAll()
.and()
.logout()
.permitAll();
}
#Autowired
public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception {
auth
.inMemoryAuthentication()
.withUser("user").password("password").roles("USER");
}
}
This class has to be in a parent package of all other packages:
WebSecurityConfig.
Also in application.properties set:
security.basic.enabled=false
ACV's answer is probably the easiest way to turn off the authentication completely by adding security.basic.enabled=false to the application.properties file which is usually located under src/main/resources folder.
or you just type in the password :)
1. use default password
When you run your spring application, there is usually a whole bunch of logging printed, which people usually don't read. The password is actually generated and printed to the screen at the startup. and the username is simply user. If you are testing using a browser and it probably only need you enter it once and caches it, so once for all, you should be securely logged in without authenticating every time.
(however, every time you restart your app, it will generate a new password)
2. customize your password
Add the following properties to your application.properties if you want to customize your username and password:
security.user.name=myuser
security.user.password=mypassword
And here is how it looks like with your own username and password
Reference:
Spring Boot Features - Security
Monitoring and Management over HTTP
You can bypass this spring boot security mechanism. See an example below for this:
#SpringBootApplication
#EnableAutoConfiguration(exclude = {SecurityAutoConfiguration.class})
public class SampleSpringBootApplication {
public static void main(String[] args) {
SpringApplication.run(SampleSpringBootApplication.class, args);
}
}
When Spring Security is in the classpath, Spring Boot by default secures all your pages with Basic authentication. That's why you are being asked for userid and password.
You will need to configure the security. To do so, commonly people would extend a WebSecurityConfigurerAdapter, like this:
#Configuration
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
#Override
protected void configure(HttpSecurity http) throws Exception {
http
.authorizeRequests()
.antMatchers("/", "/home").permitAll()
.anyRequest().authenticated()
.and()
...
Refer this Spring Security guide for more details.
Here was the issues
(1) .loginPage("/index") was saying my login page was at index, however I just wanted to use spring's default login page.
(2) had to to move the security package inside the demo package (the main package). Thanks to #Sanjay for suggesting that. I tried to use #ComponantScan but it could not get it to work.
I am trying to use the JAX-RS Client API to request a resource through HTTP GET, by using the following code: (I used jersey-client v2.12 and also resteasy-client v3.0.8.Final to test the implementation)
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.InvocationCallback;
public class StackOverflowExample {
public static void main(String[] args) {
Client client = ClientBuilder.newClient();
client.target("http://example.com/").request().async().get(new InvocationCallback<String>() {
#Override
public void completed(String s) {
System.out.println("Async got: " + s);
}
#Override
public void failed(Throwable throwable) {
System.out.println("Async failure...");
}
});
}
}
As I expected the String is printed almost immediately. But the process keeps running about one minute, although there isn't any code that should be executed.
The JAX-RS spec just says that we should use the InvocationCallback and nothing else that matters to my issue. But even if I use a Future the same effect happens. I also tested, if this has something to do with a timeout, which was very unlikely and wrong. The debugger shows that there are some threads running namely DestroyJavaVM and jersey-client-async-executor-0 or pool-1-thread-1 in the case of resteasy.
Do you have any idea what is going wrong here?
It is allways helpful to consult the JavaDoc. Concerning my issue it says:
Clients are heavy-weight objects that manage the client-side communication infrastructure. Initialization as well as disposal of a Client instance may be a rather expensive operation. It is therefore advised to construct only a small number of Client instances in the application. Client instances must be properly closed before being disposed to avoid leaking resources.
If I close the client properly everything is working as expected.
public class StackOverflowExample {
public static void main(String[] args) {
Client client = ClientBuilder.newClient();
// request here
client.close();
}
}