I'm currently looking into automating a software build process that includes a database schema defined in MySQL Workbench.
Using Workbench's scripting capabilities, I'd like to open a Workbench document and export its schema as an SQL CREATE script.
What I'd like to know is if there is a function that exports the entire schema in one step as Workbench's File | Export | Forward Engineer SQL CREATE Script, automatically handling any dependencies between tables.
I've found some candidates in the DbMySQL module that might do that (generateSQL(GrtNamedObject, dict, string) and makeSQLExportScript(GrtNamedObject, dict, dict, dict)), however I'm confused about the parameters they expect – the first one could be the schema object, but what are the other arguments ?
Could anyone tell me if my assumption is correct and/or provide me with usage examples ?
So far, I've come up with a manual solution (note that this currently does not sort the tables according to their FK relations):
local o = assert(io.open("/tmp/create.sql", "wb"));
foreach_table_all(function (t)
o:write(DbMySQL:makeCreateScriptForObject(t) .. ";\n\n")
end)
o:close()
The question is related to How to generate SQL Script from MySQL Workbench using Command Line?, however the answer found there is really abstract and tells nothing about actually using the scripting features of MySQL Workbench.
Seems that other linked question got answered in Dec 2013, courtesy of madhead, albeit with minor trivial code glitches, and in Python rather than Lua, so here the Python version that is working for me:
# -*- coding: utf-8 -*-
# MySQL Workbench Python script
# <description>
# Written in MySQL Workbench 6.0.8
import os
import grt
from grt.modules import DbMySQLFE
c = grt.root.wb.doc.physicalModels[0].catalog
DbMySQLFE.generateSQLCreateStatements(c, c.version, {
'GenerateDrops' : 1,
'GenerateSchemaDrops' : 1,
'OmitSchemata' : 1,
'GenerateUse' : 1
})
DbMySQLFE.generateSQLCreateStatements(c, c.version, {})
DbMySQLFE.createScriptForCatalogObjects(os.path.dirname(grt.root.wb.docPath) + '/ddl.sql', c, {})
Looks rather big compared to the loop variant but might bring some benefits (haven't tested, but I could imagine Workbench being able to figure out the proper order to create tables etc.).
Also I am unsure about whether this has existed when I was asking the question, but anyway, this works on a recent version.
Related
I have multiple ecospold (version 1) files with LCI data that I want to convert to a custom excel format. I need all data given in the ecospold file. For my own convinience I want to use python to complete this task.
My research until now has lead me to the following conclusions:
There exist at least two converters (by GLAD and openLCA) to convert ecospold formats (1 and 2) to e.g. the ILCD. But those formats are not helping me to go anywhere, since I need to have all the data accessible in python and in order to then write it into my custom excel format.
To get the data in python, the package pyecospold (https://github.com/sami-m-g/pyecospold) seems to be a suitable choice.
According to the README that can be found at the pyecospold github repository,
ecoSpold = parse_file("data/v1/v1_1.xml") # Replace with your own XML file
should do the job. So I implemented the following lines:
import os
from pyecospold import parse_file, save_file, Defaults
from lxml import etree
cd = os.getcwd()
path_input = cd + r'\inputs\ecospold_test.xml'
# Parse the required XML file to EcoSpold class.
es = parse_file('inputs/ecospold_test.xml')
Now I run into the error:
TypeError: parse_file() missing 2 required positional arguments: 'schema_path' and 'ecospold_lookup'
I understood that a schema in xsd format is needed, therefore I got the schema files from the github and amended my last line of code:
es = parse_file('inputs/ecospold_test.xml', 'inputs/schemas/v1/EcoSpold01Dataset.xsd')
Now there is still one argument missing:
TypeError: parse_file() missing 1 required positional argument: 'ecospold_lookup'
Since I have no experience in parsing xml files in python, I have no idea what to do with this. Additionally, I am confused why the README does not say anything about those additionally needed arguments.
My second idea was to use brightway to get the data into python. But since brightway itself is quite an extensive package, I could not find a simple (or any) way to do this. (Sadly, the notebooks linked in the answer of this question Import Ecoinvent 2.2 Ecospold files into Brightway do not exist anymore)
Another option would of course be to write my own parser. But because I am lacking experience and pyecospold does exactly this (at least in my understanding), I would like to avoid this option.
Additionally, there in openLCA it is possible to read in ecospold files and then export them to an excel format. From this excel format I could of course make my custom excel format. The problem here is that I have no idea how to automize this, because I do not want to read in and export each file individually and manually in openLCA.
If anyone has an idea on how to solve one of my subproblems or a good alternative on how to solve my general problem, I would be very thankful. :)
So, I created a minigame bot on telegram. The bot just contains a fishing game, and it's already running. I want if a user fishes and gets a fish, the fish will be stored in a database. So the user can see what he got while fishing. Does this is require SQL?
I haven't tried anything, because I don't understand about storing data in python. If there is a tutorial related to this, please share it in the comments. Thank you
You can use anything to store user data, including text files.
The simplest approaches to storing data can be serializing a dictionary to JSON with the builtin json module:
DATABASE = 'database.json' # (name or extension don't actually matter)
import json
# loading
with open(DATABASE, 'r', encoding='utf-8') as fd:
user_data = json.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'w', encoding='utf-8') as fd:
json.dump(user_data, fd)
This would only support simple data-types. If you need to store custom classes, as long as you don't upgrade your Python version, you can use the built-in pickle module:
DATABASE = 'database.pickle' # (name or extension don't actually matter)
import pickle
# loading
with open(DATABASE, 'rb') as fd:
user_data = pickle.load(fd)
user_data[1234] = 5 # pretend user 1234 scored 5 points
# saving
with open(DATABASE, 'wb') as fd:
pickle.dump(user_data, fd)
Whether this is a good idea or not depends on how many users you expect your bot to have. If it's even a hundred, these approaches will work just fine. If it's in the thousands, perhaps you could use separate files per user, and still be okay. If it's more than that, then yes, using any database, including the built-in sqlite3 module, would be a better idea. There are many modules for different database engines, but using SQLite is often enough (and there are also libraries that make using SQLite easier).
Telethon itself uses the sqlite3 module to store the authorization key, and a cache for users it has seen. It's not recommended to reuse that same file for your own needs though. It's better to create your own database file if you choose to use sqlite3.
Using a txt file as database is a terrible idea, go with SQL
Stata's ado packages quite nicely accommodate end users running different versions of Stata. For example:
program define MyGreatProgram
if int(_caller())<8 {
display as res "MyGreatProgram does not support this version of Stata, sorry."
exit
}
else {
if int(_caller())<14 {
MyGreatProgram8 `0'
}
else {
MyGreatProgram14 `0'
}
}
end
Stata's improvements with newer versions have extended to improving the possibilities in help files. For example, in Stata versions 14+, one can incorporate Unicode, and this may be very helpful in documentation (e.g., Greek characters, mathematical operators, etc.). However, in my code above, an end user of MyGreatProgram running Stata version 11, would not find help files with Unicode especially legible, while a user running Stata 15 might think they looked just fine.
Is it possible to have Stata automatically recognize separate help files for different versions of Stata, or to embed version-specific directives into Stata .sthlp files?
The comment by #NickCox suggests the following kinda clunky, but definitely workable non-automated solution:
Write different help files for different versions/version ranges a la MyGreatProgram8.sthlp, MyGreatProgram14.sthlp, etc.
Write a "foyer" help file, MyGreatProgram.sthlp, which serves as a directory to version-specific help files a la:
help MyGreatProgram
-------------------------------------------------------
Title
MyGreatProgram -- Précis of MyGreatProgram
Directory of MyGreatProgram documentation for
[Stata v 8 to Stata v 13 users]
[Stata v 14+ users]
Where [Stata v 8 to Stata v 13 users] links to MyGreatProgram8.sthlp, and [Stata v 14+ users] links to MyGreatProgram14.sthlp.
Is it possible to write to 2nd BigQuery table after writing to 1st has finished in a batch pipeline using Wait.on() method(new feature in Apache Beam 2.4)? The example given in the Apache Beam documentation is:
PCollection<Void> firstWriteResults = data.apply(ParDo.of(...write to first database...));
data.apply(Wait.on(firstWriteResults))
// Windows of this intermediate PCollection will be processed no earlier than when
// the respective window of firstWriteResults closes.
.apply(ParDo.of(...write to second database...));
But why would I write to database from within ParDo? Can we not do the same by using the I/O transforms given in Dataflow?
Thanks.
Yes this is possible, although there are some known limitations and there is currently some work being done to further support this.
In order to make this work you can do something like the following:
WriteResult writeResult = data.apply(BigQueryIO.write()
...
.withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
);
data.apply(Wait.on(writeResults.getFailedInserts()))
.apply(...some transform which writes to second database...);
It should be noted that this only works with streaming inserts and wont work with file loads. At the same time there is some work being done currently to better support this use case that you can follow here
Helpful references:
http://moi.vonos.net/cloud/beam-send-pubsub/
http://osdir.com/apache-beam-users/msg02120.html
I need to write to a non-VSAM dataset in the mainframe. I know that we need to use the ZFile library to do it and I found how to do it here
I am running my Java batch job in the WebSphere Liberty on zOS. How do I specify the dataset? Can I directly give the DataSet a name like this?
dsnFile = new ZFile("X.Y.Z", "wb,type=record,noseek");
I am able to write it to a text file on the server itself using Java's File Writers but I don't know how to access a mvs dataset.
I am relatively new to the world of zOS and mainframe.
It sounds like you might be asking more generally how to use the ZFile API on WebSphere Liberty on z/OS.
Have you tried something like:
String pdsName = ZFile.getSlashSlashQuotedDSN("X.Y.Z");
ZFile zfile = new ZFile(pdsName , ...options...)
As far as batch-specific use cases, you might obviously have to differentiate between writing to a new file that's created for the first time on an original execution, as opposed to appending to an already-existing one on a restart.
You also might find some useful snipopets in this doctorbatch.io repo, along with the original link you posted.
For reference, I'll copy/paste from the ZFile Javadoc:
ZFile dd = new ZFile("//DD:MYDD", "r");
Opens the DD namee MYDD for reading
ZFile dsn = new ZFile("//'SYS1.HELP(ACCOUNT)'", "rt");
Opens the member ACCOUNT from the PDS SYS1.HELP for reading text records
ZFile dsn = new ZFile("//SEQ", "wb,type=record,recfm=fb,lrecl=80,noseek");
Opens the data set {MVS_USER}.SEQ for sequential binary writing. Note that ",noseek" should be specified with "type=record" if access is sequential, since performance is greatly improved.
One final note, another couple useful ZFile helper methods are: bpxwdyn() and getFullyQualifiedDSN().