I am running a query on BigQuery and everything runs properly. However, when trying to save this query as a new table (in BQ) I get the following error:
enter image description here
"Not found: Dataset mycompany-data:google_analytics_de was not found in location europe-west1 "
The aforementioned table location on BigQuery is europe-west1.
I have double-checked for spelling errors and access permissions, but this error persists whatsoever.
Can you assist me with this?
Related
We are trying to replicate data from AlloyDB to Bigquery using Datastream.
We Get "An unknown error occurred. Please try again. If the error persists, contact Google support."
In the Datastream console --> objects list, we see all source tables with Object Status "Failed" and Backfill status "Completed".
In Bigquery we see only a subset of the tables (not all the "Completed" objects were synced).
In the Logs Explorer I can see this error on BQ:
I also see this error: error: {
code: 11
message: "Unsupported primary key column either does not exist or is a pseudocolumn at [1:401]"
}
The column referred in the error is of type enum.
The desired situation is having all the AlloyDB tables replicated into Bigquery.
The error message is not very informative...
What does it mean?
What would be the best way to go about troubleshooting this?
We're actively working on making these error messages be more informative, and improvements are continuously being rolled out as we identify more edge cases. Assuming you followed all the steps in the documentation, then you may need to open a ticket with support for further investigation. If a support ticket isn't an option, you can still report the issue using the public issue tracker
I just had this same issue but connecting to a PostgreSQL in AWS RDS:
Beginning with Postgres 10, passwords are encrypted using SCRAM-SHA-256 in PostgreSQL. Google DataStream still expects MD5 password encryption, or it will generate an "unknown error" in the logs and fail the backfills.
You'll need to update your postgresql.conf (or RDS Cluster Parameter Group if you're using AWS like me):
password_encryption = 'MD5'
Restart the database and make sure the parameter has changed with:
SHOW password_encryption;
Reset the password of your users:
ALTER USER "{username}" with password '{password}';
More info from the PostgreSQL docs: https://www.postgresql.org/docs/current/auth-password.html
I'm trying to set up a sample cluster with asterixDB on my M1 mac. I have my environment up and running and I am able to successfully make SQL queries with the following code:
drop dataverse csv if exists;
create dataverse csv;
use csv;
create type csv_type as {
lat: int32,
long: int32
};
create dataset csv_set (csv_type)
primary key lat;
However, when I try to load the dataset with a CSV file it seems to brick my sample cluster and throws the error: Error Code: 1 "HYR0010: Node asterix_nc2 does not exist". The code which causes this is below.
use csv;
load dataset csv_set using localfs
(("path"="127.0.0.1:///Users/nicholassantini/Downloads/test.csv"),
("format"="delimited-text"));
Thus far I have tried both java's newest release of version 18 and 17.0.3 as well as a variety of ports for the queries. I'm not sure what else to try. Some logs that I think are relevant say that it is failing to connect to the node. Not sure if that's an issue with the port or the node itself. Here is a snippet of those logs.
image.png
Also in case it matters, my CSV is a simple 2 column 2 row file with all single-digit integer values.
I appreciate any and all help.
After consulting the developer help email thread, I was able to find that the issue stems from the release of asterixDB that I was using (0.9.7.1). Upgrading to the newest release(0.9.8) fixed this issue.
The link can be found here:
https://ci-builds.apache.org/job/AsterixDB/job/asterixdb-snapshot-integration/lastSuccessfulBuild/artifact/asterixdb/asterix-server/target/asterix-server-0.9.8-SNAPSHOT-binary-assembly.zip
I own a Magento CE 1.7 based website where we keep getting variations of the following error, usually after the daily morning site reindex:
"SQL ERROR: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'main_table.data' in 'field list'
SQL QUERY: SELECT DISTINCT main_table.data, main_table.lifetime, main_table.expire, main_table.priority, additional_table.*, IFNULL(al.value, main_table.frontend_label) AS store_label FROM eav_attribute AS main_table
INNER JOIN catalog_eav_attribute AS additional_table ON additional_table.attribute_id = main_table.attribute_id
LEFT JOIN eav_attribute_label AS al ON al.attribute_id = main_table.attribute_id AND al.store_id = 1 WHERE (main_table.entity_type_id (...)"
I have a website down detector set up so that I can know immediately when the site is unavailable. However it obviously doesn't detect whenever the site is unusable due to errors such as “SQL ERROR: SQLSTATE[42S22]”. That means that my team must be constantly manually (!!!) monitoring if the site is indeed working properly (by clicking on product and category links)!
Usually we surpass the error by just cleaning the cache or performing a new reindex.
My questions are the following:
1 - Is there a way of automatically perform a detection of this type of errors in Magento so that - if it happens - a cache cleaning (or a site reindex) is immediately run and an alert is sent to the webmaster?
2 – If such an error is detected, is there a way of it not be shown to the person accessing the site? That is, if the error is detected, is it possible to immediately display a message (“We’ll be back soon”) while the cache is being cleaned or the site reindexed?
I will be grateful for any help you can provide.
Thank you!
When doing "DBT run" I get the following error
{{ config(materialized='table') }}
SELECT customer_id FROM `hello-data-pipeline.adwords.google_ads_campaign_stats`
I am making sure that my FROM location contains 3 parts
A project (hello-data-pipeline)
A database (adwords)
A table (google_ads_campaign_stats)
But I get the following error
15:41:51 | 2 of 3 START table model staging_benjamin.yo......................... [RUN]
15:41:51 | 2 of 3 ERROR creating table model staging_benjamin.yo................ [ERROR in
0.32s]
Runtime Error in model yo (models/yo.sql)
404 Not found: Dataset hello-data-pipeline:staging_benjamin was not found in location EU
NB. Bigquery does not show any error when doing this query in Bigquery Editor.
NB 2 DBT does not show any error when "running sql" command directly in the script editor
What I am doing wrong ?
You may need to specify a location where your query will run. Queries that run in a specific location may only reference data in that location. You may choose auto-select to run the query in the location where the data resides.
Read more about Dataset locations
OK I found. I needed to specify the location in the profile.yml file.
=> https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile/#dataset-locations
In DBT cloud you will find it when setting up your project
I had a similar error to your 'hello-data-pipeline:staging_benjamin was not found in location EU'
However, my issue was not that the dataset was not in the incorrect location. If was that DBT was not targeting the schema I wanted.
e.g. for your example it would be that hello-data-pipeline:staging_benjamin would actually not be the target schema you initially wanted.
Adding this bit of code on top of my query solved the issue.
{{ config(schema='marketing') }}
select ...
cf DBT's schemas: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas
here is another doc that helped me understand why this was happening:
"dbt Cloud IDE: The values are defined by your connection and credentials. To check any of these values, head to your account (via your profile image in the top right hand corner), and select the project under "Credentials".
https://docs.getdbt.com/reference/dbt-jinja-functions/target
I am using Cloud Storage Text to BigQuery template on Cloud Composer.
The template is kicked from Python google api client.
The same program
works fine in US location (for Dataflow and BigQuery).
fails in asia-northeast1 location.
works fine with the fewer (less than 10000) input files in asia-northeast location.
Does anybody have an idea about this?
I want to execute in the asia-northeast location for business reason.
More details about failure:
The program worked until "ReifyRenameInput", and the failed .
dataflow job failed
with the error message below:
java.io.IOException: Unable to insert job: beam_load_textiotobigquerydataflow0releaser0806214711ca282fc3_8fca2422ccd74649b984a625f246295c_2a18c21953c26c4d4da2f8f0850da0d2_00000-0, aborting after 9 .
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:231)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:202)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startCopyJob(BigQueryServicesImpl.java:196)
at org.apache.beam.sdk.io.gcp.bigquery.WriteRename.copy(WriteRename.java:144)
at org.apache.beam.sdk.io.gcp.bigquery.WriteRename.writeRename(WriteRename.java:107)
at org.apache.beam.sdk.io.gcp.bigquery.WriteRename.processElement(WriteRename.java:80)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found { "code" : 404, "errors" : [ { "domain" : "global", "message" : "Not found: Dataset pj:datasetname", "reason" : "notFound" } ], "message" : "Not found: Dataset pj:datasetname" }
(pj and dataset name are not real name, and they are project name and dataset name for outputTable parameter)
Although the error message said the dataset is not found, the dataset surely existed.
Moreover, some new tables which seems to be tempory tables were created in the dataset after the program.
This is a known issue related to your Beam SDK version according to this public issue tracker. The Beam 2.5.0 SDK version doesn't have this issue.