Is it possible to use some opensource project to help parse the sql query and convert it into the custom key-value db command (CRUD on the key). I saw the architecture of the tidb, and it seems like they have an abstracted layer. I would appreciated anyone pointing to some resources to help start the process.
I did try to research other project out there, and it seems like most of them have ad-hoc and embeded the parsing layers into their db, so I cannot really decouple and use their solutions.
You can refer to this blog:https://www.pingcap.com/blog/tidb-internal-computing/
But I don't think it is easy to decouple the code from TiDB.
You could use an SQL parser like https://github.com/pingcap/tidb/tree/master/parser but then you would have to add something that does the actual storage operations.
If you want transactions you might want to check this:
https://tikv.org/deep-dive/distributed-transaction/percolator/
Related
My apologies in advance for sounding like a newbie. This is really just a curiosity question I have as an outsider observing my team clash with our client. Please ask any questions you have, and I will try my best to answer it.
Currently, we are storing our transformation queries in a DynamoDB table. When needed, we pull into Databricks and execute the query. Simple as that. Our client has called this out as “hard coding” (more on that soon)
Our client has come up with an alternative that involves creating JSON config files containing the transformation rules (all tables/attributes required, target table names, Alias names, join keys, etc. etc.). From here, the SQL query is dynamically created. This approach is still “hard coding” since these config files would need to be manually edited anytime there is a change in the rules.
The way I see this: I think storing the transform rules in JSON is more business user friendly, but that’s about where I see the pros end. It brings in much more complexity to the code and likely will need to be continuously developed to support new queries. Also, I don’t see anyway to prevent “hard coding”. The client business leads seem to think there is some magical tool to convert plain English text to complex SQL queries
I just wanted to get some experts thoughts on this. Which solution is better, or is there another approach that should be taken?
I would like to create my own query language for a web api I wrote. But I have no idea where to start for it.
The language should be like SQL. For that I looked up the NHibernate code, cause I know, that they have the HQL (Hibernate Query Language) but it didn't help.
Are there any instructions or sth.? If this question is wrong here please move and/or tell me where I should ask else.
The first step is a lot of design work, starting be answering the following question:
Is this new Query Language going to be converted to SQL which will be
executed by a standard database engine, or are you going to write your
own database server as well?
If it's going to be converted to SQL (just like HQL) then map out the translations from your language to SQL on paper (and make sure you know the possible SQL constructs you may have to use). Once you've got that, you can start implementing it. Yes, this sounds like BDUF, but the language should be defined in one pass, I think, as it will be more consistent and easier to use if you do it that way. You can always implement it in a more Agile way once you've got that.
If you're going to write own database server, you're on your own....
Has anyone used the EmpireDB from Apache? I am planning to use it and would like to know if there are any known issues with this new concept?
I have used ORM like Hibernate and JPA. But EmpireDB sounds like it easy.
Anyone knows its limitations then please let me know.
Thanks.
EmpireDB places impositions on your model classes; you can't just persist your designed model. You add a dependence to their codebase (which could change later). You have to write a lot more code to use it (using its Tutorial) than you would using JDO or JPA. How that is somehow "easy" i can't think.
If you need full control over the sql, you don't want to use entities for everything and hate sql/jpql strings it is the perfect tool.
It depends largely on what you plan to do with it.
I am creating a quick and dirty prototype (C#) of an object-relational mapping tool. I would like to support at least two kinds of databases - one will be Microsoft SQL Server 2005/2008 and the other most probably MySQL.
Is there any way to use a single data base access mechanism for both database engines and what would it be?
Of course, I know that there will be differences in SQL query syntax, but in my case it is not that important - I'll use a tool to generate SQL queries which suit the certain db engine and user will be able to optimize those SQL queries.
The main idea is to have as flexible data provider solution as possible. Can it be done or not and how can it be done easier?
Note that I am not using this for a production system, just for a prototype, but still I'm curious how it is achieved in production OR/M tools - are they using completely separate access mechanism for each data provider or there are something common? And are they using DataReaders or there is some more appropriate way to retrieve data if I intend to transform data to business objects?
Thanks for any ideas, links etc.
Ok, I found it:
http://www.15seconds.com/issue/040127.htm
the solution is to use IDbxxx or Dbxxx as described in msdn.microsoft.com/en-us/library/ms379620(VS.80).aspx
Now I can specify only once what kind of DataProvider I use and then just use Db/IDb everywhere else.
I recommend nhibernate - which does what you want I think.
nhibernate.info
I currently have a relatively small (4 or 5 tables, 5000 rows) MySQL database that I would like to convert to an sqlite database. As I'd potentially have to do this more than once, I'd be grateful if anyone could recommend any useful tools, or at least any easily-replicated method.
(I have complete admin access to the database/machines involved.)
I've had to do similar things a few times. The easiest approach for me has been to write a script that pulls from one data source and produces an output for the new data source. Just do a SELECT * query for each table in your current database, and then dump all the rows into an INSERT INTO query for your new database. You can either dump this into a file or pipe it straight into the database frontend.
It's not pretty, but honestly, pretty hardly seems to be a major concern for things like this. This technique is quick to write, and it works. Those are my primary criteria for things like this.
You might want to check out this thread, too. It looks like a couple of people have already put together basically what you need. I didn't look that far into it, though, so no guarantees.
As long as a MySQL dump file doesn't exceed the SQLite query language, you should be able to migrate fairly easily:
tgl#moto~$ mysqldump old-database > old-database-dump.sql
tgl#moto~$ sqlite3 -init old-database-dump.sql new-database
I haven't tried this myself.
UPDATE:
Looks like you'll need to do a couple edits of the MySQL dump. I'd use sed, or Google for it.
Just the comment syntax, auto_increment & TYPE= declaration, and escape characters differ.
Here is a list of converters:
http://www.sqlite.org/cvstrac/wiki?p=ConverterTools
An alternative method that would work nicely but is rarely mentioned is: use a ORM class that abstracts the specific database differences away for you. e.g. you get these in PHP (RedBean), Python (Django's ORM layer, Storm, SqlAlchemy), Ruby on Rails ( ActiveRecord), Cocoa (CoreData)
i.e. you could do this:
Load data from source database using the ORM class.
Store data in memory or serialize to disk.
Store data into source database using the ORM class.
If it's just a few tables you could probably script this in your preferred scripting langauge and have it all done by the time it'd take to read all the replies or track down a suitable tool. I would any way. :)