I want to pull the meta data found in the details tab of BigQuery tables into a query. This contains labels, row counts, description, clusters, partition and other very useful information to the table. The labels alone contain information in our case that connects MDM and governance details. These are added at build time, but struggling to be able to query.
I'm aware of the information_schema.tables and information_schema.columns views. That has good info too, but not the items above. What I am looking for are all the meta-data items contained in the "details" tab when browsing the BigQuery tables visually through the console.
The details tab looks something like this:
Probably worth checking out https://cloud.google.com/bigquery/docs/information-schema-intro for the list of views. I'm guessing what you're after is in some combination of the TABLE_OPTIONS view and/or the various storage views.
Related
I'm new to CDS/Dataverse, coming from the SQL Server world. I created a new Dataverse table and there are over a dozen columns in my "new" table (e.g. "status", "version number"). Apparently these are added automatically. Why is this?
Also, there doesn't seem to be a way to view a grid of data (like I can with SQL Server) for quick review/modification of the data. Is there a way to view data visually like this?
Any tips for a new user, coming from SQL Server, would be appreciated. Thanks.
Edit: clarified the main question with examples (column names). (thanks David)
I am also new to CDS/Dataverse, so the following is a limited understanding from what I have explored so far.
The idea behind Dataverse is that it gives you a pre-built schema that follows best-practice for you build off of, so that you spend less time worrying about building a comprehensive data schema, creating tables, and how to relate them all together, and more time building applications in Power Apps.
For example, amongst the several dozen tables it generates from the get-go is Account and Contact. The former is for organisational entities and the latter is for single-person entities. You can go straight into adding your user records in one of these tables and take advantage of bits of Power Apps functionality already hooked up to these tables. You do not have to spend time thinking up column names, creating the table, making sure it hooks up to all the other Dataverse tables, testing whether the Power Apps functionality works with it correctly etc.
It is much the same story with the automatically generated columns for new tables: they are all there to maintain a best-practice schema and functionality for Power Apps. For example, the extra columns give you good auditing with the data you add, including when a row was created, modified, who created the row etc. The important thing is to start from what you want to build, and not get too caught up in the extra tables/columns. After a bit of research, you'll probably find you can utilise some more tables/columns in your design.
Viewing and adding data is very tedious -- it seems to take 5 clicks and several seconds to load the bit of data you want, which is eons in comparison to doing it in SQL Server. I believe it is how it is due to Microsoft's attempt to make it "user friendly".
Anyhow, the standard way to view data, starting from the main Power Apps view is:
From the right-hand side pane, click Data
Click Tables
From the list of tables, click your table
Along the top row, click Data
There is an alternative method that allows you to view the Dataverse tables in SSMS – see link below:
https://www.strategy365.co.uk/using-sql-to-query-the-common-data-service/
To import data in bulk:
Click on Data from the top drop-down menu > Get data.
Importing data from Excel is free. To import from other sources, including SQL Server, I believe is a paid service (although I think you may be able to do this on the free Community Plan).
I am trying to populate populate and collect metadata for the business in GBQ. Basically, the business doesn't have access to tables, we create authorised views for them that they use in their reports.
The problem is, if I populate the column description field in the table, the views based on that table won't inherit the metadata, the same with sharded tables.
There is going to be a degree of data entry to populate the metadata, but I would really like to be able to share it across related views.
Is it possible to automate BQ metadata in any way?
There are some different options to both get information about a table or a view (https://cloud.google.com/bigquery/docs/tables#getting_information_about_tables), and to update that information (https://cloud.google.com/bigquery/docs/updating-views#update-description).
Depending on your specific case you can use the bq command line or a programming language SDK to automatize the process for retrieving and updating BigQuery's metadata.
I'm new to Redash and trying to use it to analyze data.
I have created several queries and added to a dashboard.
Is Redash possible to change the datasouce on dashboard level instead of changing the datasouce of query one by one? (all datasource with the same data structure, so query can share to use)
Nope not yet. The queries underneath are tied to datasources, not the dashboard.
If you want to bulk update, you should be able to update the column in the queries table in postgres. The column data_source_id should allow you to do a bulk update.
I am using report builder 3.0 (very similar to SQL server reporting services) to create reports for users on an application using SQL server 2012 database.
To set the scene, we have a database with over 1200 tables. We actually only need about 100 of these for reporting purposes. But it is very common that we need to combine fields from multiple tables together to get a common resource of data that my colleagues and I need for our reports.
Eg if I want a view of a customer, I would want to bring in information about the customer from the customer_table, information about his phone details from the Phone table, information about his account(s) from the accounts table and so on. Then I might need another view of the accounts - account type, various balance amounts, opening date, status etc.
What I would love to do is create a "customer view" where we combine all these fields into a single combined virtual table. Then we have an "Accounts view". It would be easier to use, easier to manage etc. Then we use this for all our reports going forwards. And when we need to, we can combine the customer and accounts view to use on a report plus actual tables into one combo-dataset to use on a report.
I am unsure about the right way to do this.
I see I can create a data source. This doesn't seem right as this appears to be what one might do if working off 2 or more databases. We are using just 1 database.
Then there are report models. It seems these are being deprecated and phased out so this doesn't seem a good option.
Finally I see we can create shared datasets. However, this option (as far as I can tell) won't allow me to combine this with another dataset. So using the example above, I won't be able to combine the customer view and the account view with this approach to use for a report to display details about the customer and his/her accounts.
Would appreciate guidance on the best way to achieve what I am trying to do...
Thanks
I can only speak from personal experience, but using the the data source approach has been good for our purposes. We have a single database with 50+ tables in it. This is linked to as a shared data source in the project so is available to all 50+ reports.
We then use Stored Procedures to make the information in the databases available to the reports, each report has it's own Stored Procedure that joins as many tables as required to provide the data for the report. The advantage of using Stored Procedures also allows you to only return rows you are interested in, rather than entire tables.
I'm not certain if this is the kind of answer that you were after, but describes how we solve a similar (smaller) issue.
I've got a nice SSAS tabular model with users processing away. Certain users need access to certain information, such as confidential info (e.g., SS numbers), that should not be visible to everyone. How should I handle this?
This indicates that there is no way to use roles to remove columns, only rows. Is my only option to make a copy of the model and maintain both? This can't be such an edge case...
I guess I can jury-rig something with a scm fork and code-generation, but I'd rather not go down that road.
Alternatively, is there any way to hide the columns (per user/role), so that at least they don't show up in client tools?
One method that requires very little additional development is to use the method described in the following blog post. http://blog.westmonroepartners.com/a-workaround-for-column-security-in-the-sql-server-analysis-services-bism-tabular-model/
The blog contains a link to an SSIS package which will replicate an existing cube, with the exception of the sensitive data columns. The users who cannot view the sensitive data columns can be given access to the second cube that does not contain sensitive data columns.
One way to achieve this is to create Perspectives. You can create different perspectives for different group of users. And end users can connect to their specific model.