When I try to seed some demo data back to my organization, i get this 'time out' error, what could be the possible solution?
➜ teees-store-sanity git:(main) ✗ commercelayer seed -b multi_market
Reading OpenAPI schema... done ✔
Reading business model multi_market from url https://data.commercelayer.app/seeder... done ✔
Seeding data for organization teees-store using business model multi_market...
✔ Create addresses: [2]
✔ Create merchants: [1]
⠏ Create stock_locations: stock_location_2
Create inventory_models
Create inventory_stock_locations
Create inventory_return_locations
Create price_lists
Create markets
Create shipping_categories
Create shipping_zones
Create shipping_methods
Create payment_gateways
Create payment_methods
Create skus
Create stock_items
Create prices
› Error: timed out
Related
I want to prevent the scenario where my model runs, even though any number of source tables are (erroneously) empty. The phrase coming to mind is a "pre-hook," although I'm not sure that's the right terminology.
Ideally I'd run dbt run --select MY_MODEL and as a part of that, these tests for non-emptiness in the source tables would run. However, I'm not sure dbt works like that. Currently I'm thinking I'll have to apply these tests to the sources and run those tests (according to this document), prior to executing dbt run.
Is there a more direct way of having dbt run fail if any of these sources are empty?
Personally the way I'd go about this would be to define your my_source.yml
to have not_null tests on every column using something like this docs example
version: 2
sources:
- name: jaffle_shop
database: raw
schema: public
loader: emr # informational only (free text)
loaded_at_field: _loaded_at # configure for all sources
tables:
- name: orders
identifier: Orders_
loaded_at_field: updated_at # override source defaults
columns:
- name: id
tests:
- not_null
- name: price_in_usd
tests:
- not_null
And then in your run / build, use the following order of operations:
dbt test --select source:*
dbt build
In this circumstance, I'd highly recommend making your own variation on the generate_source macro from dbt-codegen which automatically defines your sources with columns & not_null tests included.
I'm using DBT connected to Snowflake. I use DBT Cloud, but we are moving to using VS Code for our DBT project work.
I have an incremental DBT model that compiles and works without error when I issue the DBT RUN command in DBT Cloud. Yet when I attempt to run the exact same model from the same git branch using the DBT RUN command from the terminal in VS Code I get the following error:
Database Error in model dim_cifs (models\core_data_warehouse\dim_cifs.sql)
16:45:31 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
The table in Snowflake defines this column as VARCHAR(50). I have no idea why DBT is attempting to change the data length or why it only happens when the command is run from VS Code Terminal. There is no need to make this DDL change to the table.
When I view the compiled SQL in the Target folder there is nothing that indicates a DDL change.
When I look in the logs I find the following, but don't understand what is triggering the DDL change:
describe table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS"
16:45:31.354314 [debug] [Thread-9 (]: SQL status: SUCCESS 36 in 0.09 seconds
16:45:31.378864 [debug] [Thread-9 (]:
In "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS":
Schema changed: True
Source columns not in target: []
Target columns not in source: []
New column types: [{'column_name': 'LOAN_MGMT_SYS', 'new_type': 'character varying(3)'}]
16:45:31.391828 [debug] [Thread-9 (]: Using snowflake connection "model.xxxxxxxxxx.dim_cifs"
16:45:31.391828 [debug] [Thread-9 (]: On model.xxxxxxxxxx.dim_cifs: /* {"app": "dbt", "dbt_version": "1.1.1", "profile_name": "xxxxxxxxxx", "target_name": "dev", "node_id": "model.xxxxxxxxxx.dim_cifs"} */
alter table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS" alter "LOAN_MGMT_SYS" set data type character varying(3);
16:45:31.546962 [debug] [Thread-9 (]: Snowflake adapter: Snowflake query id: 01a5bc8d-0404-c9c1-0000-91b5178ac72a
16:45:31.548895 [debug] [Thread-9 (]: Snowflake adapter: Snowflake error: 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
Any help is greatly appreciated.
I'm a new user of dbt, trying to write a relationship test:
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
I am getting this error:
Compilation Error
Invalid test config given in models/Animal_Protocols/schema.yml:
test definition dictionary must have exactly one key, got [('relationships', None), ('to', "ref('Animal_Protocols')"), ('field', 'id')] instead (3 keys)
#: UnparsedNodeUpdate(original_file_path='model...ne)
"unique" and "not-null" tests in the same file are working fine, but I have a similar error with "accepted_values".
I am using dbt cli version 0.21.0 with Snowflake on MacOS Big Sur 11.6.
You are very close! I'm 96% sure that this is an indentation issue -- the #1 pain point of working with YAML. The solution is that both to and field need to be indented below the relationships key as opposed to at the same level.
See the Tests dbt docs page for an example
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
I'm also new to DevOps, but let me see if I can explain my situation, also if anyone has any advice on doing post deployment scripts and can share their experiences I will be most grateful.
I use SQL Source control from Redgate, Git and Azure DevOps. I have managed to get the build and deployment working perfectly. Additionally we would like to make some data changes so we have additional (Data Change) scripts we wish to run. Linking these tables as static data is not an option as we would end up having most tables linked and end up with extremely timeous build and deploy times.
The script inserts data to a table, thereby triggering the insert trigger which calls a procedure to write to an audit table. The actual error from the deployment log is:
Task : DbUp Migration
2021-01-25T12:52:17.2629353Z Description : Runs SQL Server change scripts, and only those which have not been run already.
2021-01-25T12:52:17.2629405Z Version : 2.1.4
2021-01-25T12:52:17.2629671Z Author : Johan Classon
2021-01-25T12:52:17.2629721Z Help : [More Information](https://github.com/johanclasson/vso-agent-tasks)
2021-01-25T12:52:17.2629775Z ==============================================================================
2021-01-25T12:52:19.6610608Z Beginning database upgrade
2021-01-25T12:52:19.6689401Z Checking whether journal table exists..
2021-01-25T12:52:19.6728756Z Journal table does not exist
2021-01-25T12:52:19.7775348Z Executing Database Server script '001 - EX27605 - rdl.sql'
2021-01-25T12:52:19.7908913Z Checking whether journal table exists..
2021-01-25T12:52:19.7929056Z Creating the [dbo].[_SchemaVersions] table
2021-01-25T12:52:19.8077238Z The [dbo].[_SchemaVersions] table has been created
2021-01-25T12:52:20.2264323Z ifExists - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2277308Z insert - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2805780Z SQL exception has occured in script: '001 - EX27605 - rdl.sql'
2021-01-25T12:52:20.2942912Z ##[error]Script block number: 0; Block line 74; Message: Trig_After_Ins_Upd_Del_rdl_Rule_Definition_Lookup
2021-01-25T12:52:20.4528752Z ##[error]System.Data.SqlClient.SqlException (0x80131904): p_dte_Audit_Backend: (Line: 108) [dte_admin]
So the 1st error refers to line 74 of the trigger, if I read this correctly which is a commit on a procedure call to write to the audit table, line 108 of my procedure sets the userID:
DECLARE #user VARCHAR(40)
SET #user =
CASE
WHEN suser_sname() = 'NT SERVICE\SQLAgent$'+##servicename THEN 'dte_admin'
WHEN suser_sname() = 'NT SERVICE\SQLSERVERAGENT' THEN 'dte_admin'
ELSE suser_sname()
END
--###############
select #udt_Audit = udt_Audit
from udt_User_Detail (nolock)
where udt_User_Id = #user--suser_sname()
--select #udt_Audit = udt_Audit from udt_User_Detail
--where udt_User_Id = suser_sname()
if ##rowcount = 0
begin
declare #udt_User_Id varchar(30)
set #udt_User_Id = suser_sname()
The last line being line 108.
Important to note that dte_admin is a login on the SQL Server, it is a sysadmin account and mapped to the database, the dte_admin user is also a user at database level and exists as a user in my user table udt_user_detail.
During the build and deployment I am using variables and they are set to dte_admin username and password.
My question is then, why does this fail with this user?
Deployment Pipeline result
I have a Dataflow pipeline, running locally. The objective is to read a JSON file using TEXTIO, make sessions and load it into BigQuery. Given the structure I have to create a temp directory in GCS and then load it into BigQuery using that. Previously I had a data schema error that prevented me to load the data, see here. That issue is resolved.
So now when I run the pipeline locally it ends with dumping a temporary JSON newline delimited file into GCS. The SDK then gives me the following:
Starting BigQuery load job beam_job_xxxx_00001-1: try 1/3
INFO [main] (BigQueryIO.java:2191) - BigQuery load job failed: beam_job_xxxx_00001-1
...
Exception in thread "main" com.google.cloud.dataflow.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: Failed to create the load job beam_job_xxxx_00001, reached max retries: 3
at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:187)
at pedesys.Dataflow.main(Dataflow.java:148)
Caused by: java.lang.RuntimeException: Failed to create the load job beam_job_xxxx_00001, reached max retries: 3
at com.google.cloud.dataflow.sdk.io.BigQueryIO$Write$WriteTables.load(BigQueryIO.java:2198)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$Write$WriteTables.processElement(BigQueryIO.java:2146)
The errors are not very descriptive and the data is still not loaded in BigQuery. What is puzzling is that if I go to the BigQuery UI and load the same temporary file from GCS that was dumped by the SDK's Dataflow pipeline manually, in the same table, it works beautifully.
The relevant code parts are as follows:
PipelineOptions options = PipelineOptionsFactory.create();
options.as(BigQueryOptions.class)
.setTempLocation("gs://test/temp");
Pipeline p = Pipeline.create(options)
...
...
session_windowed_items.apply(ParDo.of(new FormatAsTableRowFn()))
.apply(BigQueryIO.Write
.named("loadJob")
.to("myproject:db.table")
.withSchema(schema)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
);
The SDK is swallowing the error/exception and not reporting it to users. It's most likely a schema problem. To get the actual error that is happening you need to fetch the job details by either:
CLI - bq show -j job beam_job_<xxxx>_00001-1
Browser/Web: use "try it" at the bottom of the page here.
#jkff has raised an issue here to improve the error reporting.