Was using CacheConfiguration in Ignite until I stuck with issue on how to authenticate.
Because of that I was starting to change the CacheConfiguration to clientCacheConfiguration. However after converting it to CacheConfiguration I started to notice that it
does not able to save into table because it lack of method setIndexedTypes eg.
Before
CacheConfiguration<String, IgniteParRate> cacheCfg = new CacheConfiguration<>();
cacheCfg.setName(APIConstants.CACHE_PARRATES);
cacheCfg.setIndexedTypes(String.class, IgniteParRate.class);
New
ClientCacheConfiguration cacheCfg = new ClientCacheConfiguration();
cacheCfg.setName(APIConstants.CACHE_PARRATES);
//cacheCfg.setIndexedTypes(String.class, IgniteParRate.class); --> this is not provided
I still need the table to be populated so it easier for us to verify ( using Client IDE like DBeaver)
Any way to solve this issue?
If you need to create tables/cache dynamically using the thin-client, you'll need to use the setQueryEntities() method to define the columns available to SQL "manually". (Passing in the classes with annotations is basically a shortcut for defining the query entities.) I'm not sure why setIndexedTypes() isn't available in the thin-client; maybe a question for the developer mailing list.
Alternatively, you can define your caches/tables in advance using a thick client. They'll still be available when using the thin-client.
To add to existing answer, you can also try to use cache templates for that.
https://apacheignite.readme.io/docs/cache-template
Pre-configure templates, use them when creating caches from thin client.
Related
I have a multi-tenancy structure set up where each client has a schema set up for them. The structure mirrors the "parent" schema, so any migration that happens needs to happen for each schema identically.
I am using Flask-Script with Flask-Migrate to handle migrations.
What I tried so far is iterating over my schema names, building a URI for them, scoping a new db.session with the engine generated from the URI, and finally running the upgrade function from flask_migrate.
#manager.command
def upgrade_all_clients():
clients = clients_model.query.all()
for c in clients:
application.extensions["migrate"].migrate.db.session.close_all()
application.extensions["migrate"].migrate.db.session = db.create_scoped_session(
options={
"bind": create_engine(generateURIForSchema(c.subdomain)),
"binds": {},
}
)
upgrade()
return
I am not entirely sure why this doesn't work, but the result is that it only runs the migration for the db that was set up when the application starts.
My theory is that I am not changing the session that was originally set up when the manager script runs.
Is there a better way to migrate each of these schemas without setting multiple binds and using the --multidb parameter? I don't think I can use SQLALCHEMY_BINDS in the config since these schemas need to be able to be dynamically created/destroyed.
For those who are encountering the same issue, the answer to my specific situation was incredibly simple.
#manager.command
def upgrade_all_clients():
clients = clients_model.query.all()
for c in clients:
print("Upgrading client '{}'...".format(c.subdomain))
db.engine.url.database = c.subdomain
_upgrade()
return
The database attribute of the db.engine.url is what targets the schema. I don't know if this is the best way to solve this, but it does work and I can migrate each schema individually.
Is there a way to make the logged user (on superset) to make the queries on impala?
I tried to enable the "Impersonate the logged on user" option on Databases but with no success because all the queries run on impala with superset user.
I'm trying to achieve the same! This will not completely answer this question since it does not still work but I want to share my research in order to maybe help another soul that is trying to use this instrument outside very basic use cases.
I went deep in the code and I found out that impersonation is not implemented for Impala. So you cannot achieve this from the UI. I found out this PR https://github.com/apache/superset/pull/4699 that for whatever reason was never merged into the codebase and tried to copy&paste code in my Superset version (1.1.0) but it didn't work. Adding some logs I can see that the configuration with the impersonation is updated, but then the actual Impala query is with the user I used to start the process.
As you can imagine, I am a complete noob at this. However I found out that the impersonation thing happens when you create a cursor and there is a constructor parameter in which you can pass the impersonation configuration.
I managed to correctly (at least to my understanding) implement impersonation for the SQL lab part.
In the sql_lab.py class you have to add in the execute_sql_statements method the following lines
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
where cursor_kwargs is defined in db_engine_specs/impala.py as the following
#classmethod
def get_configuration_for_impersonation(cls, uri, impersonate_user, username):
logger.info(
'Passing Impala execution_options.cursor_configuration for impersonation')
return {'execution_options': {
'cursor_configuration': {'impala.doas.user': username}}}
#classmethod
def get_cursor_configuration_for_impersonation(cls, uri, impersonate_user,
username):
logger.debug('Passing Impala cursor configuration for impersonation')
return {'configuration': {'impala.doas.user': username}}
Finally, in models/core.py you have to add the following bit in the get_sqla_engine def
params = extra.get("engine_params", {}) # that was already there just for you to find out the line
self.cursor_kwargs = self.db_engine_spec.get_cursor_configuration_for_impersonation(
str(url), self.impersonate_user, effective_username) # this is the line I added
...
params.update(self.get_encrypted_extra()) # already there
#new stuff
configuration = {}
configuration.update(
self.db_engine_spec.get_configuration_for_impersonation(
str(url),
self.impersonate_user,
effective_username))
if configuration:
params.update(configuration)
As you can see I just shamelessy pasted the code from the PR. However this kind of works only for the SQL lab as I already said. For the dashboards there is an entirely different way of querying Impala that I did not still find out.
This means that queries for the dashboards are handled in a different way and there isn't something like this
with closing(engine.raw_connection()) as conn:
# closing the connection closes the cursor as well
cursor = conn.cursor(**database.cursor_kwargs)
My gut (and debugging) feeling is that you need to first understand the sqlalchemy part and extend a new ImpalaEngine class that uses a custom cursor with the impersonation conf. Or something like that, however it is not simple (if we want to call this simple) as the sql_lab part. So, the trick is to find out where the query is executed and create a cursor with the impersonation configuration. Easy, isnt'it ?
I hope that this could shed some light to you and the others that have this issue. Let me know if you did find out another way to solve this issue, or if this comment was useful.
Update: something really useful
A colleague of mine succesfully implemented impersonation with impala without touching any superset related, but instead working directly with the impyla lib. A PR was open with the code to change. You can apply the patch directly in the impyla src used by superset. You have to edit both dbapi.py and hiveserver2.py.
As a reminder: we are still testing this and we do not know if it works with different accounts using the same superset instance.
I have a standalone java application which will interact with my web application running on node. I am trying to add new rules dynamically through web UI. So far I am unable to figure it out, how to create and add rules. Any suggestions for the right direction would be helpful
This is basically a duplicate of https://stackoverflow.com/questions/25036973 so the following is basically a duplicate of my answer to that question...
It's probably best to just look at the Drools examples source code. For instance the KieFileSystem example shows how to create a rule from a String and launch a session which includes it.
The essentials are that you create a KieServices, which contains a virtual file system. You then add rules to that file system. A little bit like the following:
KieServices ks = KieServices.Factory.get();
KieRepository kr = ks.getRepository();
KieFileSystem kfs = ks.newKieFileSystem();
kfs.write("src/main/resources/my/rules/therule.drl", "The source code of the rule");
KieBuilder kb = ks.newKieBuilder(kfs);
kb.buildAll();
you can add multiple Compiled rule DRL files like
knowledgebuilder.add(new ByteArrayResource(compiledDRL.getBytes()),ResourceType.DRL);
Get all the knowledgePackages and fire the all rules
knowledgeBase kbase = knowledgeBaseFactory.newKnowledgeBase();
kbase.addknowledgePackages(knowledgeBuilder.getKnowledgePackages());
knowledgeSession ksession = kbase.newStatefullKnowledgeSession();
ksession.insert(inputObject);
ksession.fireAllRules();
ksession.dispose();
I have about 85 SSIS packages that are using the same connection manager.
I understand that each package has its own connection manager.
I am trying to decide what would be the best configurations approach to simply set the connectionstring of the connection manager based on the server the packages are residing on.
I have visited all kinds of suggestions online, but cannot find anywhere the practice where I can simply copy the configuration from one package to the rest of the packages.
There are obviously many approaches such as XML file, SQL Server, Environment Variable, etc.
All the articles out there are pointing to use an Indirect method by using XML or SQL approach. Why would using an environment variable for just holding a connection string is such a bad approach?
Any suggestions are highly appreciated.
Thanks!
Why would using an environment variable for just holding a connection string is such a bad approach?
I find the environment variable or registry key configuration approach to be severely limited by the fact that it can only configure one item at a time. For a connection string, you'd need to define an environment variable for each catalog on a given server. Maybe it's only 2 or 3 and that's manageable. We had a good 30+ per database instance and we had multi-instanced machines so you can see how quickly this problem explodes into a maintenance nightmare. Contrast that with a table or xml based approach which can hold multiple configuration items for a given configuration key.
...best configurations approach to simply set the connectionstring of the connection manager based on the server the packages are residing on.
If you go this route, I'd propose creating a variable, ConnectionString and using it to configure the property. It's an extra step but again I find it's easier to debug a complex expression on a variable versus a complex expression on a property. With a variable, you can always pop a breakpoint on the package and look at the locals window to see the current value.
After creating a variable named ConnectionString, I right click on it, select Properties and set EvaluateAsExpression equal to True and the Expression property to something like "Data Source="+ #[System::MachineName] +"\\DEV2012;Initial Catalog=FOO;Provider=SQLNCLI11.1;Integrated Security=SSPI;"
When that is evaluated, it'd fill in the current machine's name (DEVSQLA) and I'd have a valid OLE DB connection string that connects to a named instance DEV2012.
Data Source=DEVSQLA\DEV2012;Initial Catalog=FOO;Provider=SQLNCLI11.1;Integrated Security=SSPI;
If you have more complex configuration needs than just the one variable, then I could see you using this to configure a connection manager to a sql table that holds the full repository of all the configuration keys and values.
...cannot find anywhere the practice where I can simply copy the configuration from one package to the rest of the packages
I'd go about modifying all 80something packages through a programmatic route. We received a passel of packages from a third party and they had not followed our procedures for configuration and logging. The code wasn't terribly hard and if you describe exactly the types of changes you'd make to solve your need, I'd be happy to toss some code onto this answer. It could be as simple as the following. After calling the function, it will modify a package by adding a sql server configuration on the SSISDB ole connection manager to a table called dbo.sysdtsconfig for a filter named Default.2008.Sales.
string currentPackage = #"C:\Src\Package1.dtsx"
public static void CleanUpPackages(string currentPackage)
{
p = new Package();
p.app.LoadPackage(currentPackage, null);
Configuration c = null;
// Apply configuration Default.2008.Sales
// ConfigurationString => "SSISDB";"[dbo].[sysdtsconfig]";"Default.2008.Sales"
// Name => MyConfiguration
c = p.Configurations.Add();
c.Name = "SalesConfiguration";
c.ConfigurationType = DTSConfigurationType.SqlServer;
c.ConfigurationString = #"""SSISDB"";""[dbo].[sysdtsconfig]"";""Default.2008.Sales""";
app.SaveToXml(sourcePackage, p, null);
}
Adding a variable in to the packages would not take much more code. Inside the cleanup proc, add code like this to add a new variable into your package that has an expression like the above.
string variableName = string.Empty;
bool readOnly = false;
string nameSpace = "User";
string variableValue = string.Empty;
string literalExpression = string.Empty;
variableName = "ConnectionString";
literalExpression = #"""Data Source=""+ #[System::MachineName] +""\\DEV2012;Initial Catalog=FOO;Provider=SQLNCLI11.1;Integrated Security=SSPI;""";
p.Variables.Add(variableName, readOnly, nameSpace, variableValue);
p.Variables[variableName].EvaluateAsExpression = true;
p.Variables[variableName].Expression = literalExpression;
Let me know if I missed anything or you'd like clarification on any points.
Firstly, I would like to state our environment details.
We are trying to use EJB-hibernate with sql Azure to create apps on Azure cloud using Eclipse.
We needed to create and transact on databases dynamically. We are able to create databases dynamically. However, on trying to transact on these we are getting an error:
"java.sql.SQLException: No suitable driver found for connection url"
When we tried statically transacting using jpa was not a problem. However, dynamic transactions cannot be done. The entitymanager object is created but not able to connect database.
Could someone help us and explain how we can handle transactions using JPA for dynamically created databases.
Thanks,
Saugata
[edit] We are using the following persistence.xml:
>org.hibernate.ejb.HibernatePersistence
java:jboss/EDS</jta-data-source> -->
net.oauth.database.Co
net.oauth.database.Cr
value="org.hibernate.transaction.JTATransactionFactory" />
value="org.hibernate.transaction.JBossTransactionManagerLookup" />
Our code to connect to the db is as follows:
Map configOverrides = new HashMap();
configOverrides.put("hibernate.connection.password", "");
configOverrides.put("hibernate.connection.username", "");
configOverrides.put("hibernate.connection.driver_class","com.microsoft.sqlserver.jdbc.SQLServerDriver");
configOverrides.put("hibernate.connection.url", "jdbc:sqlsever://;" + "databaseName=;user=;password=");
EntityManagerFactory factory = Persistence.createEntityManagerFactory(ENTERPRISE_UNIT_NAME, configOverrides);
Please note that we are trying to create and connect to db dynamically and hence to do not the db created statically.
For this we are getting the error:
"java.sql.SQLException: No suitable driver found for connection url"
Create a persistence.xml with a persistence unit and put everything there which is static (eg database dialect, logging parameters, etc.)
Then use the following method to create the entity manager:
javax.persistence.Persistence.createEntityManagerFactory(String persistenceUnitName, Map properties);
Supply the variable parameters in the map, like this:
properties.put("hibernate.connection.url", "jdbc:postgresql://127.0.0.1/test");
properties.put("hibernate.connection.username", "joe");
properties.put("hibernate.connection.password", "pass");