Apache Ignite doesn't support creating schema with DDL yet. So is there any way I can create a new schema in runtime? I tried to add a template cache configuration in a thick client, but schema config doesn't work.
CacheConfigration cfg = new CacheConfigration();
cfg.setName("MyTemplte*");
cfg.setSqlSchema("NEW_SCHEMA");
ignite.addCacheConfigration(cfg);
Use ignite.createCache(cfg) or ignite.getOrCreateCache(cfg) to create a new cache with the specified schema.
addCacheConfiguration only creates a template, but does not create any schemas or caches.
See https://ignite.apache.org/docs/latest/SQL/schemas for more details.
Related
I have a Ignite cluster of 2 or more nodes (max of 4) in server mode.
Let's say I have an Ignite cache defined by Java class called Employee (let's say this is version 1) loaded and used. If I update this Employee class with a new member field (version 2), how would I go about updating the loaded class with the new version (ie update the cache definition)? How does Ignite handle objects (cache records) created previously based on Employee version 1 vs new cache records created with Employee version 2? If I have SQL queries using new fields as defined in version 2, is that going to fail because the Employee version 1 based objects/cache records are not compatible with new SQL using the newly defined field(s) in Employee version 2?
I can delete db folder from the working directory, reload the new class as part of restarting the Ignite service. But, I lose all previous data.
Cluster member with updated Employee class definition will not join other nodes in the cluster still loaded with original Employee version 1 class. Again, I need to shutdown all members in the cluster and reload the new Employee version and restart all members in the cluster.
Ignite doesn't store code versions. The latest deployed class is in use.
in order to preserve the fields, Ignite builds binary meta for a customer type and stores it for validation. If you are going to add new fields and leave the old ones untouched, Ignite will update the meta automatically, nothing to configure/change. A old record will be deserialised with new fields set to null.
For SQL it's recommended to go with DDL to adjust the schema accordingly:
ALTER TABLE "schema".MyTable DROP COLUMN oldColumn
ALTER TABLE "schema".MyTable ADD COLUMN newColumn VARCHAR;
You can check available meta using control script --meta command (not sure if it's available in Ignite edition though)
control.sh --meta list
Ignite won't propagate POJO changes automatically using peerClassLoading. You should either update the JARs manually or rely on some deployment SPI, like URL deployment.
Overall, you should not remove your db folder each time you are going to make changes to your POJOs/SQL tables. Adding new fields should be totally OK. Do not remove the old fields, it's better to mark them as deprecated.
I am new to HIVE, I am trying to setup a hive metastore service with standalone MySQL DB, and I realized that I need to config hive.metastore.warehouse.dir in the hive-site.xml, but I am having a hard time to understand what it is for?
1, None of the metadata will be stored in this location, because all of the metadata will be stored in the MySQL db.
2, None of the data files will be stored in this location, because I am not setting up a Hive data service, it is just a metastore service. And when creating hive tables, I will specify the location of the table.
Why do I still need to set this configuration?
spark.sql.warehouse.dir is a static configuration property that sets Hive’s hive.metastore.warehouse.dir property, i.e. the location of default database for the Hive warehouse
That is correct. This directory indicates where the actual data in the tables will reside.
It sounds like in most of your situations, the data will reside outside of what you set for this directory. However, if a user were to forget to set the location or if there are any internal/automated calls that use the "default" database. This is where your "default" data will reside.
I have a question regarding Flyway and managing multiple schemas. I have multiple schemas (schema1, schema2, schema3) with different deployment schedules and different folder locations (sql/schema1, sql/schema2, sql/schema3) with different code.
I want to Flyway to create the schemas before the code deployment but how do I set this up in a single config file? I read the Flyway doc (https://flywaydb.org/documentation/faq#multiple-schemas) but is the example using a single config file? or do i need to create multiple config files (one per schema)?
Can i achieve the same setting comma delimited schema list? will "Schema1" only look in the "sql/Schema1" location? I really dont want Schema1 pulling code from a different folder i.e. sql/Schema2, etc.
Thanks in advance!
When using Flyway with multiple schemas, you need to explicitly say in the sql statements which schema the sql is going to change. You can do this by putting an ALTER SESSION SET CURRENT_SCHEMA=schema1 at the top of each migration file, or prefixing all your statements like CREATE TABLE schema1.bananas.
If this is not practical, it would be best to create a number of config files, each with a single schema specified, and a single location specified. e.g.
flyway.schemas=schema1
flyway.locations=filesystem:sql/schema1
Then you can run Flyway with each config file individually to migrate that particular schema.
I'm new to Nifi so could you help me understand this platform and its capabilities.
Would I be able to use a Nifi process to create a new table in Hive and move data into it weekly from a teradata database in the way I've defined below?
How would I go about it? Not sure if I'm building a sensible flow.
Would the following process suffice: QueryDatabaseTable (and configure a pooling service for teradata and define a new tablename and schedule ingestion) --> PutHiveStreaming (create the table defined earlier)
and then how do i pull the teradata schema into the new table?
If you want to create new hive table along with the ingestion process then
Method1:
Using ConvertAvroToOrc processor adds hive.ddl(external table) attribute to the flowfile as we can use this attribute and execute using PutHiveQL processor then we are able to create table in hive.
If you want to create transactional table then needs to change the hive.ddl attribute.
Refer to this link for more details.
If you wan to pull only the delta records from the source then you can use
ListDatabaseTables(list all tables from source db) + GenerateTableFetch(stores the state) Processors
Flow:
Method2:
QuerydatabaseTable processor will result flowfile in Avro Format then you can use ExtractAvroMetaData processor to extract the avro schema by using some script we can create a new attribute with the required schema(i.e. managed/external/transactional table).
I have a custom table in MODX database set up and working, thanks to this article:
http://bobsguides.com/custom-db-tables.html
and now I need to add new column to this existing table. How can I do this the "MODX way"? Or do I have to create the component from scratch again?
You can manually add the new column to the database, then update your xml schema and map files to include the new column metadata. If you have a build script you could simply run it again after amending the schema to regenerate the map files.
I could be more specific if you paste in your existing schema and description of the column you want to add.
I believe MigxDB plugin (part of migx plugin) sets up a utility under manager page to just do that.
Install migx as instructed (you need to do an extra step to set it up so read the instruction)
load your modified schema in midx-package manager and do 'parse schema' and then 'add field'.
Make sure you have package name and pre-fix specified when loading your schema. modx forum has a dedicated section for migx if you need further clarification.