I have a table in liquibase. I want to retrieve "name" field for a particular "type" from that table and then convert name to upper case. How can I use changeset or ServletContextListener to do the same?
For example:
My table structure is table1{ id, name, type} and i retrieve names associated to a given type. Now I want to convert these names (can be more than one) to upper case before my data is actually populated in the database.
This is more of an ETL task than a schema task. You could use Liquibase for it, but it would probably require custom code. If it is just a small set of things you might be able to do it using custom SQL.
Related
I want to create new table with empty columns and specify the datatype later. Is it possible? I try to do so on myscompiler.io and it works. I don't know if it's just possible in such site or is it actually possible to create that once I use other tools to write my SQL.
No. The SQL syntax requires that a table be well-defined, with column names and data types. This is true in every database that I can think of.
You could possibly do what you want in one of three ways:
Your database might support some sort of generic type which you could use to define the column. For instance, SQL Server has a sql_variant type.
You could define the table with a specific type such as a string and change the type later using alter table.
You could define the table with a single primary key column and add columns as you decide what they are.
I don't recommend any of these approaches. Instead, I would suggest that you need to re-think how your application is structured. Tables represent entities and entities have properties. Generally when using databases, these things are known before you start doing any work. There may be some cases where dynamic table creation is useful, but that is definitely not the common approach when using databases.
Our project has got a particular requirement. I understand it but I don't think it can be easy to implement.
Description: There is a table in database with 500 columns.
Now, the end users (power users or admins) may want to define derived columns. A derived column is a column based on other columns. The derived column can technically be a actual physical column or just a logical column which we generate at runtime by dynamic SQL query.
Example: say there is a column called 'WageOnDay' and there is another called 'WageDate'. Now end user may want to define a derived column called 'WageForPastMonth' or 'WageForPastYear' or just simply a derived column with 'WageDate'+2000. The user may want to define any number of such additional derived fields and would want to define them via a UI. The user would also want to specify a custom display name for these derived columns. Example: user may want to call the derived column for 'WageDate'+2000 as 'WageWithBonus' or 'WageAfterAdjustment'. The user may change the display name of the derived column and its definition (current definition 'WageDate'+2000 to 'WageDate'+4000) from time to time..
The derived column definition could also be an aggregation like sum over the wages for a date range etc.
Now - we thought over this from technical aspect. And there does not seem to be a way to implement this.
The user may define an aggregation for derived column or a simple expression etc.. we could go on about actually modifying the table definition and adding these additional fields as per the users definition for derived columns..
Other approach would be to store these derived column definitions in another configuration table and use those to to generate a dynamic SQL which will have to generate the derived columns according to the definition.
Currently, we think implementing this would require a huge effort. We would need to also create a semantic language which will store the derived column definition and would have to implement a parser which generates part of the SQL based on the user definition.
I know there are reporting tools like QlikView, CrystalReports and JasperReport which have the functionality I described.
But, can we implement it in our project ? If not, then is integrating with any of the other tools an option.
Please share your thoughts and suggestions.
Let us know if such a requirement has been addressed and a high level technical approach.
You could add an XML column to the table for custom fields, then just manipulate it at the application layer.
Context
I have three tables in my SQL Server database: 1) School, 2) College, 3) University.
Then I have another table: Tags.
Each of the three tables (School, College, University) can have Tags associated with them. For which purpose I have three association tables: SchoolTags, CollegeTags, UniversityTags.
Problem
I am trying to create a user-defined function that will take the name of association table as parameter (i.e. 'SchoolTags') and the Id of the entity (school/college/university) and will return a list of tags associated with that entityId.
The issue I am having is I have got to join Tags with a table whose name will come in as parameter. For that I am creating a dynamic query. And we can not run dynamic queries in SQL Server user-defined functions.
Question
Any idea how can that be acheived?
Note: I want separate association tables as I have created and do not want to convert them into a generic association table and I do not want to add If-Else based on table names in my function so that if a new association table is created, I do not need to update my function.
I am using Microsoft SQL Server.
Whatever language you are using, you would probably use if:
begin
if table = 'school' then
begin
. . .
end;
else if table = 'college' then
. . .
end;
The exact syntax depends on the scripting language for the database you are using.
What you desire is impossible. You cannot pass a table name as a parameter to a UDF and use dynamic sql in the UDF to then create and execute a statement that is specific to the table passed as the argument. You already know that you have no choice but to use if-else statements in your UDF to achieve your goal - it is your pipe-dream of "never having to update (or verify) your code when the schema changes" (yes - I rephrased it to make your issue more obvious) that is a problem.
There are likely to be other ways of implementing some useful functionality - but I suggest that you are thinking too far ahead and trying to implement generic functions without a clear purpose. And that is a very difficult and trouble-prone path that requires sophisticated tsql skills.
And to re-iterate the prior responses, you have a schema problem. You purposely created three different entities - and now you want a common function to use with any of them. So before you spend much time on this particular aspect, you should take some time to think carefully about how you intend to use (i.e., write queries against) these tables. If you find yourself using unions frequently to combine these entities into a common resultset, then you have might have a mismatch between your actual business and your model (schema) of it.
Consider normalizing your database in related, logical groupings for one EducationInstitution table and one JoinEducTags table. Those tables sound like they maintain the same structure but of different typology and hence should be saved in one table with different Type field for School, College, University, etc.
Then, add necessary constraints, primary/foreign keys for the one-to-many relationship between all three sets:
You never want to keep restructuring your schema (i.e., add tables) for each new type. With this approach, your user-defined function would just need to receive value parameters not identifiers like tables to be run in dynamic querying. Finally, this approach scales better with efficient storage. And as you will see normalization saves on complex querying.
I want to update an empty column from type Boolean to string in big-query.
How can i do it without overwrite the table and load all the data?
thanks!
You can only add new fields at the end of the table. On the old columns, you have the option to change required to nullable. So what you want is not possible, only if you add a new field, or as you say completely overwriting the table.
There are two table operations Update and Patch.
You need to use the Update command, to add new columns to your schema.
Important side notes:
order is important. If you change the ordering, it will look like an incompatible schema.
you can only add new fields at the end of the table. On the old columns, you have the option to change required to nullable.
you cannot add a required field to an existing schema.
you cannot remove old fields, once a table's schema has been specified you cannot change it without first deleting all the of the data associated with it. If you want to change a table's schema, you must specify a writeDisposition of WRITE_TRUNCATE. For more information, see the Jobs resource.
Here is an example of a curl session that adds fields to a schema. It should be relatively easy to adapt to Java. It uses auth.py from here
When using Table.Update(), you must include the full table schema again. If you don't provide an exact matching schema you can get: Provided Schema does not match Table. For example I didn't paid attention to details and in one of my update calls I didn't include an old field like created and it failed.
The user wants to add new fields in UI dynamically. This new field should get stored in database and they should be allowed to perform CRUD on it.
Now I can do this by specifying a XML but I wanted a better way where these new columns are searchable. Also the idea of firing ALTER statement and adding a new column seems wrong.
Can anyone help me with a design pattern on database server side of how to solve this problem?
This can be approached using a key value system. You create a table with the primary key column(s) of the table you want to annotate, a column for the name of the attribute, and a column for its value. When you user wants to add an attribute (say height) to the record of person 123 you add a row to the new table with the values (123, 'HEIGHT', '140.5').
In general you cast the values to TEXT for storage but if you know all the attributes will be numeric you can choose a different type for the value column. You can also (not recommended) use several different value columns depending on the type of the data.
This technique has the advantage that you don't need to modify the database structure to add new attributes and attributes are only stored for those records that have them. The disadvantage is that querying is not as straightforward as if the columns were all in the main data table.
There is no reason why a qualified business user should not be allowed to add a column to a table. It is less likely to cause a problem than just about anything else you can imagine including adding a new row to a table or changing. the value of a data element.
Using either of the methods described above do not avoid any risk; they are simply throwbacks to COBOL filler fields or unnecessary embellishments of the database function. The result can still be unnormalized and inaccurate.
These same business persons add columns to spreadsheets and tables to Word documents without DBAs getting in their way.
Of course, just adding the column is the smallest part of getting an information system to work, but it is often the case that it is perceived to be an almost insurmountable barrier. It is in fact 5 min worth of work assuming you know where to put it. Adding a column to the proper table with the proper datatype is easy to do, easy to use, and has the best chance of encouraging data quality.
Find out what the maximum number of user-added fields will be and add them before hand. For example 'User1', 'User2', 'User3', 'User4'...etc. You can then enable the fields on the UI based on some configurable settings.