I am looking for an approach to import data from mssql to gemfire, Is it possible either from java program or Command line?
There's no out of box functionality to import data from a RDBMS into GemFire, you'll basically need to iterate over your rows, transform them into Java Objects and insert them into the target region manually.
There is, however, a feature for Keeping the Cache in Sync with Outside Data Sources that you can use to slowly populate the region whenever there's a cache miss for a particular key, which might be useful here.
Hope this helps. Cheers.
Related
Want to migrate bulk files (e.g VSAM) from Mainframe to Azure in the beginning of the Project, how that can be achieved ?
Any utility or do we need to write own scripts?
I suspect there are some utilities out there but I suspect they are most / all priced products. Since VSAM datasets are not defined using a language construct like DDL you will likely have to do most of the heavy lifting. Either writing your own programs or custom scripts. You didn’t mention operating system but I assume you’re working on z/OS.
Here are some things to consider:
The structure of the VSAM dataset is basically record oriented. There are three basic types you’ll run into that host application data:
Key Sequenced Datasets (KSDS)
Entry Sequenced Datasets (ESDS)
Relative Record datasets (RRDS)
Familiarize yourself with the means of defining the datasets as it will give you some insight into the dataset specifics. DFSMS Access Method Services Commands will show the utilities used to create them and get information like Keylength and offest of the key. DEFINE CLUSTER is the command to create the dataset. You mentioned you are moving the data toi Azure but this will help you understand the characteristics of the data you are moving.
Since there is no DDL for VSAM datasets you will generally find the structure in the programs that manipulate them like COBOL Copybooks, HLASM DSECTs and similar constructs. This is the long pole in the tent for you.
Consider what are the semantics of accessing the data. VSAM as an access method does have some ability to control read/write access on a macro level using a DEFINE CLUSTER option called SHAREOPTIONS. The SHAREOPTIONS instruct the operating system how to handle the VSAM buffers in terms of reading and writing so that multiple processes can access the same data. Its primitive if compared to sahred files systems like NFS. VSAM allows the application to control access (or serialization) using ENQ / DEQ functions. These enable applications to express intent in the cluster about a VSAM file and coordinate their own activities.
You might find that converting a VSAM file to a relational form like Db2 is better for you. Again, you’ll have to create the DDL to describe the tables, data formats and the like.
Another consideration is data conversion. You’ll find there is character data that is most likely in EBCDIC and needs to be converted to a new code page. Numeric data can be in Packed Decimal, Binary, or even text and will need to be converted.
The short answer is there isn’t an “Easy Button” to do what you want. Consider the data is only one of the questions that needs to be answered. Serialization and access to the data, codepage conversion, if you are moving some data but not others will you need to be able to map some of the converted data back to data on the mainframe.
Consider exploring IBM CDC classic replication. You can achieve it with click of buttons.
I have not done for Azure. So not sure about support.
I'm fairly new to SQL Server management and am currently looking in to building a solution with SSIS.
My question will be mostly about "is my logic correct" and some smaller things about what the best practice would be.
But let me paint you a picture to start!
I have an application which provides me with DB Views, to offload the stress on that database I would like to transfer the data of these views periodically to a secondary database/different instance on which I can then also set more specific permissions/transformations/other views build on that data. My views initially provided are pretty much fixed in how I can get them.
After some reading it looked to me that the way to go was to use SSIS. I started building my package and used the "SSIS Import and Export wizard" to do an initial transfer.
Now for my first question, would this be the proper way to transfer the data and is SSIS the right tool for the job?
Secondly, I noticed that the wizard made multiple SQL Preperation tasks and Data flow tasks.
For me it would seem logical to split each view that becomes a table in to a seperate SQL Preperation Task and a seperate Data flow task just to keep a clear picture and as much control as possible. While that would take some time to set up (>100 views/tables) it seems cleaner than how the wizard did it by just grouping some of them together.
Also, since the preperation tasks already create tables these fail when executed a 2nd time as they already exist. Is there a quick workarround for this besides adding a IFEXISTS clause to each query?
Any thoughts on this would be appreciated or hints towards a better solution if I'm approaching this completly from the wrong direction.
The idea is to later on add some SSAS to the system and provide some data analytics on these tables/data as well.
Thanks!
If you are transferring the data to a different instance then SSIS is probably your best bet. Your next question is to work out whether you want to import all the data each time or just the new/updated items.
If you are exporting all the data every time, this is much simpler and assumig you have a suitable maintenance window (such as overnight) that you can complete the process in without affecting your end users, you can get away with simply truncating the data and re-loading. This obviously has consequences related to increased data transfer volumes.
If you want to only export the new/updated data, you will now need to work out whether or not you can actually work out which rows are new or different without simply comparing them all to what you have in your secondary database. Ideally your source tables will have a reliable LastUpdateDate column or better yet a rowversion column, using which you can export all rows with a more recent value than can be seen in the corresponding table in your secondary instance.
There is a lot of reading to be done regarding the updates only route, for which I would strongly suggest you avoid the Slowly Changing Dimension transformation like the plague.
You are also right in thinking that there is a lot of repetitive tasks when you want to do simple operations across a large number of similar objects, such as adding that ifexists to the table creation in your post. The best way to tackle this I have found is to learn how to use Biml to automate the repetitive tasks based on metadata.
Good Luck!
With the load data option that Liquibase provides, one can specify seed data in a CSV format. Is there a way I can provide say, a JSON or XML file with data that Liquibase would understand?
The use case is we are trying to put in some sample data which is hierarchical. E.g. Category - Subcategory relation which would require putting in parent id for all related categories. If there is a way to avoid including the ids in the seed data via say, JSON.
{
"MainCat1": ["SubCat11", "SubCat12"],
"MainCat2": ["SubCat21", "SubCat22"]
}
Very likely to have this as not supported (couldn't make Google help me) but is there a way to write a plugin or something that does this? Pointer to a guide (if any) would help.
NOTE: This is not about specifying the change log in that format.
This not currently supported and supporting it robustly would be pretty difficult. The main difficultly lies in the fact that Liquibase is designed to be database-platform agnostic, combined with the design goal of being able to generate the SQL required to do an operation without actually doing the operation live.
Inserting data like you want without knowing the keys and just generating SQL that could be run later is going to be very difficult, perhaps even impossible. I would suggest approaching Nathan, who is the main developer for Liquibase, more directly. The best way to do that might be through the JIRA bug database for Liquibase.
If you want to have a crack at implementing it, you could start by looking at the code for the LoadDataChange class (source in Github), which is where the CSV support currently lives.
I just started iOS development am currently developing an application that just reads data from a server and displays it onto the screen. What I am not sure of is whether to use XML or sqlite3 to store the data. Which method should be more preferred and why? thanks in advance.
It is important to remember they are two different things, suited to different tasks. Choose the one that fits the problem. (In this case I would likely use XML or "just plain text" because it sounds like just a simple download-cache. Either the raw response could be kept or, perhaps the data already transformed into objects and then automatically serialized into XML or whatnot. In any case, keep it simple.)
XML is (at the very core) a markup format. XML documents are a (hopefully well-defined) structure. There is a large set of tooling that supports manipulation and querying within a hierarchical "document" model. I use XML a good bit for a serialization format and also use it for local caching if appropriate (e.g. there are no non-hierarchical relationships). XML is often loaded entirely into memory (e.g. a DOM) for manipulation.
SQLite is a relational database that is designed around tables and relationships between sets of tables. Being able to run (complex) queries is where a relational database really shines. SQLite is also very fast and can process large data-sets which can't all fit in memory. Columns in SQLite can also contain text (read: XML) so the approaches are not orthogonal.
Happy coding.
Probably all depends on how data is processed after it was stored. If data must be sorted, uses specific selection etc. then, sqlite is better solution.
Second, not so important, concern is how much data will be stored, if it's just one "table" with 10 rows then sqlite is probably too much for it.
If you want to read data from server and want to display on screen and don't need to save it locally then use XML.
If you want to store it locally and don't want to fetch from server then use XML files or sqlite database in your project.
If you want to fetch from server and also to store it locally then first use XML to fetch data and then use sqlite to store it locally.
and look at #pst answer for what is the difference between them.
I've created a Core Data schema in xcode (3.2.5 if it matters) so I have the .xcdatamodel file with the proper entities and relations.
Now - How can I insert data, edit data and/or delete data from it, NOT from within the code ?
Like what phpMyAdmin is for MySql.
Thanks.
Core Data is meant to be used programmatically. Once you run the app once, it should create a file somewhere on disk (exactly where is probably specified in the AppDelegate class). It is likely that this file will be a SQLite database, but it doesn't have to be (the point of Core Data is to abstract your data away from the file format used to store it). It could also be an XML file or a binary file.
If it's a SQLite file, then you can open it in your favorite SQLite editor.
HOWEVER
The schema used in the SQLite format is not documented. If you go mucking around in it, you might get stuff to work, but it's also very likely that you could irreparably screw it up. (If it's an XML file or a binary file, you're probably totally out of luck)
In the end, Core Data is supposed to be used programmatically. To use it in a different way (such as what you're asking for) would be to use it in a way for which it was not intended and therefore not designed.
I don't know if you already solved your problem, but there's this SQLite Manager plug-in for firefox: http://code.google.com/p/sqlite-manager/
I haven't tried importing data or using the INSERT command to insert individual rows, but you could give it a try. It's free and works very well for me as is.
There's quite a few database management tools available for sqlite that allow you to do this. I've tried a few but to be honest none of them have impressed me that much as yet.
Would be great to have something like Toad available.
Anyway, find wherever your database file is, then drop it onto whichever application.
You can then add, delete, and edit rows and columns.
Of course, you will need to maintain any foreign keys and such like.
I find the generated Core Data models to be pretty easy to understand.
Example tools are SQLite Database Browser (free), SQLiteManager (not free), and Base. A quick Google search should reveal those and a few more.
I normally use SQLite Database Browser although it does crash occasionally.
See Christian Kienle's Core data editor. It's not free, but is designed to work directly with core data models and stores via Apple's API, supports binary data, builds relationships and even triggers validation, etc. I've found it's worth the $20.