Error: Not found: Dataset my-project-name:domain_public was not found in location US - google-bigquery

I need to make a query for a dataset provided by a public project. I created my own project and added their dataset to my project. There is a table named: domain_public. When I make query to this table I get this error:
Query Failed
Error: Not found: Dataset my-project-name:domain_public was not found in location US
Job ID: my-project-name:US.bquijob_xxxx
I am from non-US country. What is the issue and how to fix it please?
EDIT 1:
I change the processing location to asia-northeast1 (I am based in Singapore) but the same error:
Error: Not found: Dataset censys-my-projectname:domain_public was not found in location asia-northeast1
Here is a view of my project and the public project censys-io:
Please advise.
EDIT 2:
The query I used to type is based on censys tutorial is:
#standardsql
SELECT domain, alexa_rank
FROM domain_public.current
WHERE p443.https.tls.cipher_suite = 'some_cipher_suite_goes_here';
When I changed the FROM clause to:
FROM `censys-io.domain_public.current`
And the last line to:
WHERE p443.https.tls.cipher_suite.name = 'some_cipher_suite_goes_here';
It worked. Shall I understand that I should always include the projectname.dataset.table (if I'm using the correct terms) and point the typo the Censys? Or is this special case to this project for some reason?

BigQuery can't find your data
How to fix it
Make sure your FROM location contains 3 parts
A project (e.g. bigquery-public-data)
A database (e.g. hacker_news)
A table (e.g. stories)
Like so
`bigquery-public-data.hacker_news.stories`
*note the backticks
Examples
Wrong
SELECT *
FROM `stories`
Wrong
SELECT *
FROM `hacker_news.stories`
Correct
SELECT *
FROM `bigquery-public-data.hacker_news.stories`

In Web UI - click Show Options button and than select your location for "Processing Location"!
Specify the location in which the query will execute. Queries that run in a specific location may only reference data in that location. For data in US/EU, you may choose Unspecified to run the query in the location where the data resides. For data in other locations, you must specify the query location explicitly.
Update
As it stated above - Queries that run in a specific location may only reference data in that location
Assuming that censys-io.domain_public dataset has its data in US - you need to specify US for Processing Location

The problem turned out to be due to wrong table name in the FROM clause.
The right FROM clause should be:
FROM `censys-io.domain_public.current`
While I was typing:
FROM domain_public.current
So the project name is required in the FROM and `` are required because of - in the project name.

Make sure your FROM location contains 3 parts as #stevec mentioned
A project (e.g. bigquery-public-data)
A database (e.g. hacker_news)
A table (e.g. stories)
But in my case, I was using the LegacySql within the Google script editor, so in that case you need to state that to false, for example:
var projectId = 'xxxxxxx';
var request = {
query: 'select * from project.database.table',
useLegacySql: false
};
var queryResults = BigQuery.Jobs.query(request, projectId);

check exact case [upper or lower] and spelling of table or view name.
copy it from table definition and your problem will be solved.
i was using FPL009_Year_Categorization instead of FPL009_Year_categorization
using c as C and getting the error "not found in location asia-south1"
I copied with exact case and problem is resolved.

On your Big Query console, go to the Data Explorer on the left pane, click the small three dots, then select query option from the list. This step confirms you choose the correct project and dataset. Then you can edit the query on the query pane on the right.

may be dataset name changed in create dataset option. it should be US or default location
enter image description here

Related

Understanding the "Not found: Dataset ### was not found in location US" error

I know this topic has come up many times but still here I am. Data processing location seems consistent (dataset, US; query: US) and I am using backticks & long format in the FROM clause
Below are two sequences of code. The first one works perfectly:
SELECT station_id
FROM `bigquery-public-data.austin_bikeshare.bikeshare_stations`
Whereas the following returns an error message:
SELECT bikeshare_stations.station_id
FROM `bigquery-public-data.austin_bikeshare`
Not found: Dataset glassy-droplet-347618:bigquery-public-data was not found in location US
My question, thus, is why do the first lines of text work while the second doesn't?
You need to understand the different parts of the backticks:
bigquery-public-data is the name of the project;
austin_bikeshare is the name of the schema (aka dataset in BQ); and
bikeshare_stations is the name of the table/view.
Therefore, the shorter format you are looking for is: austin_bikeshare.bikeshare_stations (instead of bigquery-public-data.austin_bikeshare).
Using bigquery-public-data.austin_bikeshare means that you have a schema called bigquery-public-data that contains a table called austin_bikeshare , when this is not true.

error in google datastudio on using custom query

I am using a simple sql block of statements to execute and return a set of results in big query, This is working fine in big query and getting the results,I need to export this data to data studio, so in data studio i use bigquery as connector and select the project and custom query and in that I paste the contents below:
Declare metricType String;
SET metricType="compute.googleapis.com/instance/cpu/utilization";
BEGIN
IF (metricType="compute.googleapis.com/instance/cpu/usage_time")
THEN
SELECT m.value as InstanceName,metric.type as metricType,point.value.double_value as usagetime,point.interval.start_time as StartTime,point.interval.end_time as EndTime,h.value as instance_id FROM `myproject.metric_export.sd_metrics_export_fin`, unnest(resource.labels) as h,unnest(metric.labels) as m where metric.type='compute.googleapis.com/instance/cpu/usage_time' and h.key="project_id";
ELSE IF (metricType="compute.googleapis.com/instance/cpu/utilization")
THEN
SELECT m.value as InstanceName,metric.type as metricType,point.value.double_value as utilizationrate,point.interval.start_time as
StartTime,point.interval.end_time as EndTime,h.value as instance_id FROM `myproject-.metric_export.sd_metrics_export_fin`,unnest(resource.labels) as h,unnest(metric.labels) as m where metric.type='compute.googleapis.com/instance/cpu/utilization' and h.key="project_id";
END IF;
END IF;
END;
but after click "ADD" button i get the below error:
I am not sure what is this error about? I have not used any stored procedure and I am just pasting it as custom query.
Also If I try to save the results of the BigQuery into a view from the Bigquery console results pane, it gives me the error,
Syntax error: Unexpected keyword DECLARE at [1:1]
I am extremely new to datastudio and also to bigquery. Kindly Help thanks
First, I would like to make some considerations about your query. You are using Scripting in order to declare and create a loop within your query. However, since you declare and set the metricsType in the beginning of the query, it will never enter in the first IF. This happens because the value is set and it is not looping through anything.
I would suggest you to use CASE WHEN instead, as below:
SELECT m.value as InstanceName,metric.type as metricType,
CASE WHEN metric.type = #parameter THEN point.value.double_value ELSE 0 END AS usagetime,
CASE WHEN metric.type = #parameter THEN point.value.double_value ELSE 0 END AS utilizationrate,
point.interval.start_time as StartTime,point.interval.end_time as EndTime,h.value as instance_id
FROM `myproject.metric_export.sd_metrics_export_fin`, unnest(resource.labels) as h,unnest(metric.labels) as m
WHERE metric.type=#parameter and h.key="project_id";
Notice that I am using the concept of Parameterized queries. Also, for this reason this query won't work in the console. In addition, pay attention that whem you set the #parameter to "compute.googleapis.com/instance/cpu/utilization", it will have a non-null column with the usagetime and a null column named utilizationrate.
Secondly, in order to add a new data source in DataStudio, you can follow this tutorial from the documentation. After, selecting New Report, click on the BigQuery Connector > Custom Query> Write your Project id, you need to click in ADD PARAMETER (below the query editor). In the above query, I would add:
Name: parameter
Display name: parameter
Data type: text
Default value: leave it in blank
Check the box Allow "parameter" to be modified in reports. This means you will be able to use this parameter as a filter and modify its value within the reports.
Following all the steps above, the data source will be added smoothly.
Lastly, I must point that if your query ran in the Console you are able to save it as a view by clicking on Save view, such as described here.

Filter parent data source with a field which is in another table in D365

Some times we need to filter a form grid based on the status of the transaction reference Id. Assume that we want to show purchase orders with confirm document state in arrival overview form. The document state field is located in the purch table. For this aim, I try to outer join WMSArrivalOverviewTmp to purch table and add Range. However the results is not as I expect.
This is the code I have tried in the initialized data source event:
[FormDataSourceEventHandler(formDataSourceStr(WMSArrivalOverview, WMSArrivalOverviewTmp), FormDataSourceEventType::Initialized)]
public static void WMSArrivalOverviewTmp_OnInitialized(FormDataSource sender, FormDataSourceEventArgs e)
{
QueryBuildDataSource qbds = sender.queryBuildDataSource();
QueryBuildDataSource qbdsPO;
qbdsPO = qbds.addDataSource(tableNum(PurchTable));
qbdsPO.clearRange(fieldNum(PurchTable, DocumentState));
qbdsPO.joinMode(JoinMode::OuterJoin);
qbdsPO.fetchMode(QueryFetchMode::One2One);
qbdsPO.addLink(fieldNum(WMSArrivalOverviewTmp, InventTransRefId ),fieldNum(PurchTable, PurchId), qbds.name());
qbdsPO.relations(false);
qbdsPO.addRange(fieldNum(PurchTable, DocumentState)).value(SysQuery::value(VersioningDocumentState::Confirmed));
info(sender.query().toString());
}
This is the query which has been shown :
SELECT FIRSTFAST * FROM WMSArrivalOverviewTmp(WMSArrivalOverviewTmp)
OUTER JOIN FROM PurchTable(PurchTable_1) ON
WMSArrivalOverviewTmp.InventTransRefId = PurchTable.PurchId AND
((DocumentState = 40))
Also, I have changed the event type to query executing but I get errors:
[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error converting
data type nvarchar to bigint.
Cannot select a record in Purchase orders (PurchTable). The SQL
database has issued an error.
P.S. I have noticed that the error comes when I click on the update button after I fill one of the field in the filter section(arrival option). For example when I fill account number or warehouse. And also I have noticed that when I click on the update button while the fields are not filled in the filter section the join operation has not applied correctly, i.e., the purchase orders which have draft status are shown.
Your actual query looks correct to me. I would add that you may want to add the link between the WMSArrivalOverviewTmp.InventTransType == InventTransType::Purch (or set qbds.relations(true) so that you use the table's built in relation), but I doubt that is the root of your problem. Of course it wouldn't hurt to make sure your query works in the manner that you expect it to by converting the results of the info() call you have into SQL code and run it in SSMS to double check the result set.
Also, I'm curious why would you put the event on the datasource you are trying to manipulate? I would recommend investigating to see if it can occur on the OnModified event of the "status of the transaction reference Id". Although perhaps I am misunderstanding - if this situation is more of a "we need to happen all the time when the form loads" than a "some times we need" as you start your question off - the place to change the query is to modify it in a handler of the datasource's OnQueryExecuting event.
There are two ways of modifying the query on an event when an event occurs.
FormRun formRun = sender.formRun() as FormRun;
FormDataSource formDataSource = formRun.dataSource(formDataSourceStr(WMSArrivalOverview, WMSArrivalOverviewTmp));
//First way - get the base query and use executeQuery() to reload changes
Query query = formDataSource.query();
//modified query here.
formDataSource.executeQuery();
//Second way - get the current queryRun query and use research() to reload changes
Query query = formDataSource.queryRun.query();
//modified query here.
formDataSource.research();

Peoplesoft CreateRowset with related display record

According to the Peoplebook here, CreateRowset function has the parameters {FIELD.fieldname, RECORD.recname} which is used to specify the related display record.
I had tried to use it like the following (just for example):
&rs1 = CreateRowset(Record.User, Field.UserId, Record.UserName);
&rs1.Fill();
For &k = 1 To &rs1.ActiveRowCount
MessageBox(0, "", 999999, 99999, &rs1(&k).UserName.Name.Value);
End-for;
(Record.User contains only UserId(key), Password.
Record.UserName contains UserId(key), Name.)
I cannot get the Value of UserName.Name, do I misunderstand the usage of this parameter?
Fill is the problem. From the doco:
Note: Fill reads only the primary database record. It does not read
any related records, nor any subordinate rowset records.
Having said that, it is the only way I know to bulk-populate a standalone rowset from the database, so I can't easily see a use for the field in the rowset.
Simplest solution is just to create a view, but that gets old very soon if you have to do it a lot. Alternative is to just loop through the rowset yourself loading the related fields. Something like:
For &k = 1 To &rs1.ActiveRowCount
&rs1(&k).UserName.UserId.value = &rs1(&k).User.UserId.value;
&rs1(&k).UserName.SelectByKey();
End-for;

MOSS 2007: What is the source of "Directories"?

I'm trying to generate a new SharePoint list item directly using SQL server. What's stopping me is damn tp_DirName column. I have no ideas how to create this value.
Just for instance, I have selected all tasks from AllUserData, and there are possible values for the column: 'MySite/Lists/Task', 'Lists/Task' and even 'MySite/Lists/List2'.
MySite is the FullUrl value from Webs table. I can obtain it. But what about 'Lists/Task' and '/Lists/List2'? Where they are stored?
If try to avoid SQL context, I can formulate it the following way: what is the object, that has such attribute as '/Lists/List2'? Where can I set it up in GUI?
Just a FYI. It is VERY not supported to try and write directly to SharePoint's SQL Tables. You should really try and write something that utilizes the SharePoint Object Model. Writing to the SharePoint database directly mean Microsoft will not support the environment.
I've discovered, that [AllDocs] table, in contrast to its title, contains information about "directories", that can be used to generate tp_DirName. At least, I've found "List2" and "Task" entries in [AllDocs].[tp_Leaf] column.
So the solution looks like this -- concatenate the following 2 components to get tp_DirName:
[Webs].[FullUrl] for the web, containing list, containing item.
[AllDocs].[tp_Leaf] for the list, containing item.
Concatenate the following 2 components to get tp_Leaf for an item:
(Item count in the list) + 1
'_.000'
Regards,
Well, my previous answer was not very useful, though it had a key to the magic. Now I have a really useful one.
Whatever they said, M$ is very liberal to the MOSS DB hackers. At least they provide the following documents:
http://msdn.microsoft.com/en-us/library/dd304112(PROT.13).aspx
http://msdn.microsoft.com/en-us/library/dd358577(v=PROT.13).aspx
Read? Then, you know that all folders are listed in the [AllDocs] table with '1' in the 'Type' column.
Now, let's look at 'tp_RootFolder' column in AllLists. It looks like a folder id, doesn't it? So, just SELECT the single row from the [AllDocs], where Id = tp_RootFolder and Type = 1. Then, concatenate DirName + LeafName, and you will know, what the 'tp_DirName' value for a newly generated item in the list should be. That looks like a solid rock solution.
Now about tp_LeafName for the new items. Before, I wrote that the answer is (Item count in the list) + 1 + '_.000', that corresponds to the following query:
DECLARE #itemscount int;
SELECT #itemscount = COUNT(*) FROM [dbo].[AllUserData] WHERE [tp_ListId] = '...my list id...';
INSERT INTO [AllUserData] (tp_LeafName, ...) VALUES(CAST(#itemscount + 1 AS NVARCHAR(255)) + '_.000', ...)
Thus, I have to say I'm not sure that it works always. For items - yes, but for docs... I'll inquire into the question. Leave a comment if you want to read a report.
Hehe, there is a stored procedure named proc_AddListItem. I was almost right. MS people do the same, but instead of (count + 1) they use just... tp_ID :)
Anyway, now I know THE SINGLE RIGHT answer: I have to call proc_AddListItem.
UPDATE: Don't forget to present the data from the [AllUserData] table as a new item in [AllDocs] (just insert id and leafname, see how SP does it itself).