My goal is to somehow be sending (Maybe via an OCI) process an sql script to determine whether if it is a DML script (All of the code blocks are not DDL nor DCL) or an hybrid with other data languages.
Is it possible? If so, how?
Let's say maybe with OCI you can't do it. How would you automatically do a check to validate this?
Use OCI_ATTR_STMT_TYPE as explained by https://stackoverflow.com/a/13528133/103724 and you will get one of the values in this table http://docs.oracle.com/cd/E14072_01/appdev.112/e10646/oci04sql.htm#CIHEHCEJ.
There's also OCI_ATTR_SQLFNCODE which provides more detailed information, but as the question above showed, it can be tricky to use. --DD
Related
We are repointing a large number (>1000) DataStage jobs from one database to another. As part of this, we will need to make the same changes to a single stage for many jobs.
So far, we have been able to export jobs to XML, edit and reimport. This seems to work, but will require a lot of parsing logic. We also have looked at dsjob, but that tool does not seem to have the ability to edit jobs.
What is the best method (UI or CLI/API) to bulk edit job stages?
Scenarios like this are the reason for using parameters for Databases - I recommend using ParameterSets with DBName, User, Password and Schema parameters.
This allows an easy and quick change in one place of a project: the ParameterSet
Hard coding all these things will give you a hard time - the export method is one option you know already.
There is a connector migration wizzard - I am not sure if this tool could be helpful as well - you might want to search for documentation on that.
Perhaps you can try the RJUT (Rapid Job Update Tool) or the CMT (Connector Migration Tool).
RJUT: https://www.ibm.com/support/pages/rapid-job-update-tool-ibm-infosphere-information-server-datastage-jobs
CMT: https://www.ibm.com/docs/en/iis/11.7?topic=connectors-using-command-line-migrate-jobs
I want to create an automate alert for our data such as null, unequal amount, etc. It'll send to our email. The data stored in bigquery. How do i create this?
it automatically send to our email if the database have null value, empty value, unequal.
e.g
Look!
Table invoice have 0 rows detected. You can check from XX, and XY table.
or for the advance
Look!
Table invoice have 0 rows detected. You can check from XX, and XY table. This causes because XYZ.
The easiest way to resolve this request is to develop a custom solution on top of app engine or cloud functions, so that you can run your own queries and trigger email notifications.
This approach will not alert you in real-time but based on an schedule.
The other option if your use case is related to INSERT statements and real-time analysis is to take advantage of the logging alerts.
This will trigger an alert based on INSERT events, so you need to build your own query and setup the alert accordingly.
You can start with something like this:
resource.type="bigquery_resource"
protoPayload.methodName="jobservice.insert"
If you don't want to create a custom solution, it sounds like the Great Expectations framework could be a good fit for you.
This is a system for defining properties you expect of your data. It can be connected to BigQuery. It can be configured to send you emails if your tests ('expectations') fail.
You would still need to host the system yourself (they have a cloud solution coming in 2023).
If you want to get started fast and don't have easy access to hosting, you could also consider DBT (data build tool). DBT can connect to BigQuery and allows you to run tests (though this is not its main purpose). These tests can include checks for null data. You can write custom tests for more complex checks.
I mention it because they already have a paid for cloud solution, DBT Cloud, and yes, you can send email notifications if your tests fail. Depending on circumstances, this could be the fastest way to get what you want.
I'm in a situation where I need to query modifications out of an DirX Directory Server (LDAP).
In more commonly products like OpenDS, Oracle DSEE, etc. there is usually come kind of changelog that can be queried, which gives you the sequence of modifications performed in that server.
Unfortunately, there is basically no information available online that helps me with this question.
Can anybody with some insight to DirX give some hints if DirX provides anything like this?
DirX doesn't provide the cn=changelog node/subtree that you're looking for.
DirX changelogs are written as LDIF change files. These files can simply be dumped to the filesystem for later use/processing, or as they are written you can invoke any application/script you like do do something with the LDIF data. For example, you can pipe the ldif data to ldapmodify and send every change made in DirX out to another ldap server in real-time. You could pipe the data to a custom application or script that filters it for certain types of operations and writes the wanted info to a sql db, or to whatever output you want. There really aren't any limits here. You just need to read LDIF.
The LDIF data can be written (and piped to your application/script) on change to handle real-time requirements, or on a scheduled basis for batch based processes.
BTW, I've seen implementations where the cn=changelog node (like you'd find on Oracle DSEE) is created in DirX using the LDIF changelog data. i.e. as the LDIF data is written on change, the data is piped to a script that creates the entries you expect under cn=changelog. Obviously this was done to provide more familiar changelog functionality for Oracle DSEE users.
Check whether DirX supports the persistent search control. If it does, this provides change notification, but not history like the UnboundID change log or the retro-changelog of DSEE.
For example I've often wanted to search stackoverflow with
SELECT whatever FROM questions WHERE
views * N + votes * M > answers AND NOT(answered) ORDER BY views;
or something like that.
Is there any reasonable way to allow users to use SQL as a search/filter language?
I see a few problems with it:
Accessing/changing stuff (a carefully setup user account should fix that)
SQL injection (given the previous the worst they should be able to do is get back junk and crash there session).
DOS attacks with pathological queries
What indexes do you give them?
Edit: I'd like to allow joins and what not as well.
Accessing/changing stuff
No problem, just run the query with a crippled user, with permissions only to select
SQL injection
Just sanitize the query
DOS attacks
Time-out the query and throttle the access by IP. I guess you can also throttle the CPU usage in some servers
If you do SQLEncode your users' input (and make sure to remove all ; as well!), I see no huge safety flaw (other than that we're still handing nukes out to psychos...) in having three input boxes - one for table, one for columns and one for conditions. They won't be able to have strings in their conditions, but queries like your example should work. You will do the actual pasting together of the SQL statement, so you'll be in control of what is actually executed. If your setup is good enough you'll be safe.
BUT, I wouldn't for my life let my user enter SQL like that. If you want to really customize search options, give either a bunch of flags for the search field, or a bunch of form elements that can be combined at will.
Another option is to invent some kind of "markup language", sort of like Markdown (the framework SO uses for formatting all these questions and answers...), that you can translate to SQL. Then you can make sure that only "harmless" selects are performed, and you can protect user data etc.
In fact, if you ever implement this, you should see if you could run the commands from a separate account on the SQL server, which only has access to the very basic needs, and obviously only read access.
Facebook does this with FQL. See the blog post or presentation.
I just thought of a strong sanitize method that could be used to restrict what can be used.
Use MySQL and grab it's lex/yacc files
use the lex file as is
gut the yacc file to only the things you want to allow
use action rules that spit out the input on success.
Part of the setup routine for the product I'm working on installs a database update utility. The utility checks the current version of the users database and (if necessary) executes a series of SQL statements that upgrade the database to the current version.
Two key features of this routine:
Once initiated, it runs without user interaction
SQL operations preserve the integrity of the users data
The goal is to keep the setup/database routine as simple as possible for the end user (the target audience is non-technical). However, I find that in some cases, these two features are at odds. For example, I want to add a unique index to one of my tables - yet it's possible that existing data already breaks this rule. I could:
Silently choose what's "right" for the user and discard (or archive) data; or
Ask the user to understand what a unique index is and get them to choose what data goes where
Neither option sounds appealing to me. I could compromise and not create a unique index at all, but that would suck. I wonder what others do in this situation?
Check out SQL Packager from Red-Gate. I have not personally used it, but these guys make good tools overall and this seems to do what you're looking for. It let's you modify the script to customize the install:
http://www.red-gate.com/products/SQL_Packager/index.htm
You never throw a users data out. One possible option is to try and create the unique index. If the index creation fails, let them know it failed, tell them what they need to research, and provide them a script they can run if they find they have a data error that they choose to fix up.