I am using dbt version 0.19.1.
I have a pipeline that automatically runs dbt docs generate whenever i push some changes to my git repository.
From a certain point in time, pipelines started to fail with these error message:
Running with dbt=0.19.1
Found 776 models, 0 tests, 0 snapshots, 0 analyses, 359 macros, 0 operations, 0 seed files, 145 sources, 0 exposures
14:44:45 | Concurrency: 8 threads (target='default')
14:44:45 |
14:45:07 | Done.
14:45:07 | Building catalog
Encountered an error while generating catalog: 'dtype'
dbt encountered 1 failure while writing the catalog
The problem is that I have many models and changes happening in the source code and it is not easy to understand from this message where the error is coming from.
If I check the logs files in the logs folder they are very messy, containing debug information from all the threads, i cannot find the source of the issue.
Is there a way to understand where this issue is coming from?
Is it possible to better debug dbt docs generate command?
Thank you all for your replies in advance!
Francesco
Is it possible to better debug dbt docs generate command?
To better debug the issue you can use
dbt --debug docs generate
That will contain the debug information from all the threads, but it should also include a stack trace from where the error was raised.
Thank you Josh, adding another point to look for the error.
I suggest who have the same issues and has thousands of rows of logs like me to look for the string "Error while running" in the debug logs to find the place where the exception should have raised.
Hopes this could help someone else.
Related
I’m doing some local testing using dbt core and trying to figure out how to only rerun models associated with failed tests from the prior dbt build invocation.
I’ve setup a test scenario with 3 models (models a, b, and c). I ran dbt build. The models were built and tests were run. The scenario has a failed not_null test, which impacts models a and c.
Now, I want to rerun only the models associated with the failed test (a and c), without having to manually select them.
I tried doing dbt build --select 1+result:fail which I found here. That resulted in
Encountered an error:
No comparison run_results
I also tried dbt build –-select result:error+ –-defer -–state ./target which I found here. That resulted in
dbt: error: unrecognized arguments: –-select result:error+ –-defer -–state ./target
I’m guessing running only models impacted by failed tests relies on run_results.json which I see in ./target. So, I'm thinking I’m missing some kind of configuration step, but I’m having a hard time finding info on how to rerun dbt from the point of failure. Any suggestions or resources for this issue?
Ok, I ended up having some unnoticed em dashes in the second command I tried.
This is what I was after
dbt build --select 1+result:fail+ --defer --state ./target
Answered in dbt slack.
I have been having some issues lately when including applications in task sequences. When running any task sequence that contains applications, it automatically fails, and says "The referenced package cannot be found." I've checked my distribution points and boundary groups, and verified the application content is distributed. When checking the logs, it just states it failed to find an application, I track down the application it's referencing, and redistribute, or even remove it from the task sequence, and try running it again, I get the same error, except for the next application content ID's. When adding packages to a task sequence, it seems to run successfully. Has anyone else encountered this?
EDIT: I've also been seeing 'content hash value mismatch' errors in the logs.
Any help is greatly appreciated. Some extra info:
I have already restored the site server VM, rebuilt the distribution point.
Failed to find CCM_ApplicationCIAssignment object for AdvertID="***2017A", ModelName="ScopeId_E6E2F6FB-692F-4938-8AC6-146644EAE93F/Application_ce95b2ac-bf5a-4de2-b930-6f9b74b7dfd0"
"Failed to resolve selected task sequence dependencies. Code(0x80040104)"
I have a fully developed set of functions which work fine in the "dev" stage and it's now time for me deploy to production. Unfortunately every time I try to deploy it goes for a long time but after printing "Checking Stack update progress" it fails with a 404 error:
An error occurred: SentinelLambdaFunction - Function not found: arn:aws:lambda:us-east-1:837955377040:function:xyz-services-prod-sentinel (Service: AWSLambda; Status Code: 404; Error Code: ResourceNotFoundException; Request ID: 38f86b7a-99cd-11e8-af06-fffd92e40dc5).
This error is non-sensical to me as this function does exist and executing precisely the same full deployment to "dev" results in no error. Note that in both environment/stage, we are deploying 10 functions with a fully deployment.
I tried removing the function which was being complained about first, with the hope that I could re-include it on a second deployment but then it simply complained about a different function not existing.
I also thought maybe the "--force" parameter might push this deployment into place but it has had no impact on the error I get.
The cycle time for each attempt is very long so I'd be very grateful if anyone could help to point me in the right direction on this.
Below is a screenshot of the output when run in "verbose" mode:
In attempt to get around the error I thought maybe I'd have a better chance if I went into CloudFormation and explicitly deleted the template for prod. I attempted to do this from the GUI and got the following:
This actually has further convinced me that this removal is important but I'm not sure what to do next.
For me, the solution was:
serverless remove
and then try deploying again.
So the solution to this problem was to ensure that all previous traces of the CloudFront stack was removed. In my case I had manually taken out a few functions from Lambda and the 401 errors I was getting were likely occuring in the removal attempts rather than my assumption that it was related to adding these functions.
Bear in mind you may find yourself -- like I did -- where the first attempt to delete fails. In this case try again and make sure to check off any checkboxes exposed by UI that indicate what had caused the issues the prior attempt.
Once I'd done that I was able to deploy as per normal from the serverless framework.
Is there a way to get Karate to automatically print the name of each scenario as it is executed into the logs? We have a reasonably large suite that generates ~25MB of log data in our Jenkins pipeline console output and sometimes it’s a little tricky trying to match a line where com.intuit.karate logs an ERROR to the failure summary at the end of the run. It is most likely possible to obtain the scenario name and print() it but that would mean adding code to many hundred scenarios which I’d like to avoid.
As part of the fix for this issue Karate will log the Scenario name (if not empty) along with any failure trace.
A beta version with this fix is available 0.6.1.2 it would be great if you can try it and confirm.
If you feel more has to be done, do open a ticket and we'll do our best to get this into the upcoming 0.6.2 release.
I've got a graph that isn't behaving as it should in CloudConnect.
I'm running it locally, and it's completing, but not doing its work.
In an effort to figure out why this is, I've added printLog calls in many places, like the following
printLog(warn, 'transfrom from file ' + $in.0.fileName);
printLog(debug, 'joining etc');
The Phase consists of a FileList into a SimpleCopy, into a LookupJoin, a Reformat (produce SQL) and a DBInsert.
However, while I see logs for phases above, I'm not seeing anything produced in the log for any part of my phase. All parts of the phase do report running successfully in log. I've also done Enable Debugging on all connections in this phase.
Am I missing something to enable logging? Is there a better way to debug processing in CloudConnect?
Discovered the problem - the FileList will succeed if the source file cannot be found, but none of the subsequent steps will then fire. It's somewhat unintuitive, since the log files says 'succeeded'.
For debugging, after run you can access the data by right clicking on the connection, and selecting "View Data"
Sorry for the elementary question, but documentation didn't seem to cover this clearly, at least for a GoodData noob. I'll leave it up for anyone with the same problem!