Using Apache Ignite v2.6
As Apache Ignite version does not support JDBC connection pool against JDBC Thin Driver, what should be the approach to issue multiple parallel queries to Ignite? Document says that Ignite support 128 ResultSet/Cursor at a time. does this mean that 'A client will be able to shoot multiple queries in parallel, using a single connection'??
Any suggestions please.
I guess, you are referring to maxOpenCursorsPerConnection parameter. Its value doesn't specify the number of concurrent operations, it only limits the number of open cursors per connection.
A single JDBC connection cannot be used from multiple threads concurrently. You can use a connection pool instead. For example, take a look at Hikari Connection Pool.
Related
I have multiple standalone NiFi instances (approx. 10) that I want to use to send data to a NiFi cluster (3 NiFi instances) using RPG (Site-to-Site). But, the flow from the standalone instances to the cluster seems to be slow.
Is this the right approach?
How many Site-to-Site Connections does NiFi allow?
Are there any best practices for Site-to-Site NiFi Data Flow?
You may want to first rule out your network. You could ssh to one of the standalone nodes and then try to SCP a large file from the standalone node to one of the nodes in the NiFi cluster. If that is slow then it is more of a network problem and there won't be much you can do to make it go faster in NiFi.
In NiFi, you can tune each side of the site-to-site config...
On the central cluster you can right-click on the remote Input Port and configure the concurrent tasks which defaults to 1. This is the number of threads that can concurrently process data received on the port.
On the standalone NiFi instances you can also configure the concurrent tasks used to send data to a given port. Right-click on the RPG and select "Manage remote ports", and then change the concurrent tasks for whichever port.
I would like to know answers for below questions:
1) In case if Ignite server is restarted, I need to restart the client (web applications). Is there any way client can reconnect to server on server restart. I know when server restarts it allocates a different ID and because of this the current existing connection becomes stale. Is there way to overcome this problem and if so, which version of Ignite supports this feature. Currently I utilize version 1.7
2) Can I have client cache like how Ehcache provides. I don’t want client cache as a front–end to a distributed cache. When I looked at the Near Cache API, it doesn’t have cache name properties like cache configuration and it acts only as a front-end to a distributed cache. Is it possible to create client only cache in Ignite
3) If I have a large object to cache, I find Serialization and Deserialization takes a longer time in Ignite and retrieving from distributed cache is slow. Is there any way we can speed up large objects retrieval from Ignite DataGrid.
This topic is discussed on Apache Ignite users mailing list: http://apache-ignite-users.70518.x6.nabble.com/Questions-on-Client-Reconnect-and-Client-Cache-td10018.html
Ignite has two modes, one is Server mode, and the other is client mode.I am reading https://apacheignite.readme.io/docs/clients-vs-servers, but didn't get a good understanding of these two modes.
In my opinion, there are two use cases:
If the Ignite is used as an embedded server in a java application, they the Ignite should be in server mode, that is, Ignite should be started with
Ignite ignite = Ignition.start(configFile)
If I have setup an Ignite cluster that are running as standalone processes. Then in my java code, I should start Ignite in client mode, so that the client mode Ignite can connect to the Ignite cluster, and CRUD the cache data that resides in the ignite cluster?
Ignition.setClientMode(true);
Ignite ignite = Ignition.start(configFile)
Yeah, this is correct understanding.
Ignite client mode intended as lightweight mode (which do not store data and do not execute compute tasks). Client node should communicate with a cluster and should not utilize self resources.
Client does not even started without server node presented in topology.
In order to further add to #Makros answer, Ignite Client stores data if near cache is enabled. This is done in order to increase the performance of cache retrievals.
Yeah, you are right in ignite client has IgniteConfiguration.setClientMode(true); and for server IgniteConfiguration.setClientMode(false);, which is default value. if set IgniteConfiguration.setClientMode(false); in you code or forget to set setClientMode(); it will work as server.
what is difference between org.apache.hive.jdbc.HiveDriver and org.apache.hadoop.hive.jdbc.HiveDriver ?
Which one to use to Write a JDBC Client to connect to hive ?
Hive 0.11 includes a new JDBC driver that works with HiveServer2, enabling users to write JDBC applications against Hive. The application needs to use the JDBC driver class and specify the network address and port in the connection URL in order to connect to Hive.
HiveServer2 (HIVE-2935), brings concurrency, authentication, and a foundation for authorization to Hive
HiveServer2 is an improved version of HiveServer that supports Kerberos authentication and multi-client concurrency and use the driver "org.apache.hive.jdbc.HiveDriver"
HiveServer1 or Thrift server cannot handle concurrent requests from more than one client. This is actually a limitation imposed by the Thrift interface that HiveServer exports, and can't be resolved by modifying the HiveServer code. The Driver which Hive Server "org.apache.hadoop.hive.jdbc.HiveDriver"
Please find the links which will help you to understand more.
org.apache.hadoop.hive.jdbc.HiveDriver and org.apache.hive.jdbc.HiveDriver
To work with, it depend on ur requirement which version you are having and how you have done Hive Configuration.
I currently have an application deployed using Tomcat that interacts with a Postgres database via JDBC. The queries are very expensive, so what I'm seeing is a timeout caused by Tomcat or Apache (Apache sits in front of Tomcat in my configuration). I'm trying to limit the connections to the database to 20-30 simultaneous connections, so that the database is not overwhelmed. I've done this using the \.. configuration, setting maxActive to 30 and maxIdle to 20. I also bumped up the maxWait.
In this scenario I'm limiting the USE of the database, but I want the connections/requests to be POOLED within Tomcat. Apache can accept 250 simultaneous requests. So I need to ensure Tomcat can also accept this many, but handle them appropriately.
Tomcat has two settings in the HTTP Connector config file:
maxThreads - "Max number of request processing threads to be created by the Http Connector, which therefore determines the max number of simultaneous requests that can be handled."
acceptCount - "The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused."
So I'm guessing that if I set maxThreads to the max number of JDBC connections (30), then I can set acceptCount to 250-30 = 220.
I don't quite understand the difference between a thread that is WAITING on a JDBC connection to open up from the pool, versus a thread that is queued... My thought is that a queued thread is consuming less cycles whereas a running thread, waiting on the JDBC pool, will be spending cycles checking the pool for a free thread...?
Note that the HTTP connector is for incoming HTTP requests, and unrelated to JDBC. You probably want to configure the JDBC connection pool separately, such as the connectionProperties for the JDBC connector:
http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
Unless your application handles request in a matter where it directly connects to the database on a per http connection basis, then you should configure your JDBC connection pool based on what your database software is set to/ can handle and your maxthreads to what your application / hardware can handle.
Keeping maxActive value (of db connection pooling) lesser than maxThreads (i.e. number of concurrent threads) makes sense in most of the cases. You can set acceptCount to a higher value depending upon what traffic you are expecting in your website and how fast one request can be processed.