"transforms.Reroute.topic.regex" and "table.include.list" in property Json of Debezium SQL connector - sql

``As per below property files, expected behavior is to route events from "table.include.list": "dbo.idm_assets,dbo.idm_datapoints" to queue - debezium_assets and from "table.include.list": "dbo.idm_workorder,dbo.idm_activity" to queue - debezium_events.`
`{
"name": "sql-server-kafkacdc-idm-connection",
"config": {
"snapshot.mode": "initial_schema_only",
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "localhost",
"database.port": "1433",
"database.user": "cdc_idm_owner",
"database.password": "cdcIdm#2022",
"database.dbname": "kafkacdc",
"database.server.name": "kafkacdc",
"tasks.max": "1",
"decimal.handling.mode": "string",
"tombstones.on.delete": false,
"table.include.list": "dbo.idm_assets,dbo.idm_datapoints",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.topic.regex": "(.*)",
"transforms.Reroute.topic.replacement": "debezium_assets",
"database.history":"io.debezium.relational.history.MemoryDatabaseHistory"
}
}`
{
"name": "sql-server-kafkacdc-idm-all-001",
"config": {
"snapshot.mode": "initial_schema_only",
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "localhost",
"database.port": "1433",
"database.user": "cdc_idm_owner",
"database.password": "cdcIdm#2022",
"database.dbname": "kafkacdc",
"database.server.name": "kafkacdc",
"tasks.max": "1",
"decimal.handling.mode": "string",
"tombstones.on.delete": false,
"table.include.list": "dbo.idm_workorder,dbo.idm_activity",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.topic.regex": "(.*)",
"transforms.Reroute.topic.replacement": "debezium_events",
"database.history":"io.debezium.relational.history.MemoryDatabaseHistory"
}
}
But currently, events are being sent to both the queues even if we trigger an event from table of another property file which isn't expected.
Observation : We have deleted one of the connectors(Assets) and tried to reproduce the issue. Triggered an event from Assets table and could see it in Events queue which is not expected.
Please find the current and expected behaviors:
Current Behavior
Expected Behavior`

Related

Azure Devops API Release definitions expand recursively (I need workflowTasksfrom deployPhases from environments)

Does the Azure Devops REST API allow me to expand multiple levels? When using the release definitions I specifically need the workflowtasks, which are buried a couple of lists deep.
More context:
I'm optimizing an Azure Devops extension that scans pipelines for compliancy. Right now there's a rule that scans the workflowtasks. To get the information required on all relevant pipelines, we do the following call to the Azdo API for each release definition:
https://vsrm.dev.azure.com/{Organization}/{Project}/_apis/release/definitions/{definitionID}?api-version=6.0
This returns a completely decked-out release definition including environments like this:
"environments": [{
"id": 10,
"name": "Stage 1",
... etc
"deployPhases": [{
"deploymentInput": {
"parallelExecution": {
"parallelExecutionType": "none"
},
...etc
"rank": 1,
...etc
"workflowTasks": [{
"environment": {},
"taskId": "obfuscated",
"version": "2.*",
"name": "obfuscated",
"refName": "",
"enabled": true,
"alwaysRun": false,
"continueOnError": false,
"timeoutInMinutes": 0,
"retryCountOnTaskFailure": 0,
"definitionType": "task",
"overrideInputs": {},
"condition": "succeeded()",
"inputs": {
"template": "obfuscated",
"assets": "obfuscated",
"duration": "60",
"title": "",
"description": "",
"implementationPlan": "obfuscated"
}
}, {
"environment": {},
"taskId": "obfuscated",
"version": "2.*",
"name": "obfuscated",
"refName": "",
"enabled": true,
"alwaysRun": false,
"continueOnError": false,
"timeoutInMinutes": 0,
"retryCountOnTaskFailure": 0,
"definitionType": "task",
"overrideInputs": {},
"condition": "succeeded()",
"inputs": {
"changeClosureCode": "1",
"changeClosureComments": "Successful implementation",
"changeId": ""
}
}
]
}
],
...etc
}
],
But when I try and get the list as a whole, using the following URL:
https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?$expand=environments&api-version=6.0
My Environments arrays (there is one for each definition obviously) looks nothing like the previous one. It doesn't include deployPhases (not even as an empty array).
Since we have 2300 release definitions, you can Imagine how inconvenient it is to call the release/definitions/{definitionID} endpoint instead of the release/definitions one that fetches all of them at the same time.
Is there a way to expand the release/definitions call to fetch all environments including workflowTasks and maybe other stuff? Is there a syntax that allows for this? Something like $expand=environments>deployPhases>workflowTasks?
Is there a way to expand the release/definitions call to fetch all environments including workflowTasks and maybe other stuff? Is there a syntax that allows for this? Something like $expand=environments>deployPhases>workflowTasks?
I am afraid there is no such syntax allow you to fetch all environments including workflowTasks.
You could use the REST API with some powershell scripts to fetch all environments including workflowTasks:
The sample powershell scripts:
$url = "https://vsrm.dev.azure.com/{Organization}/{Project}/_apis/release/definitions/{definitionID}?api-version=6.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "workflowTasks = $($pipeline.environments.deployPhases.workflowTasks | ConvertTo-Json -Depth 100)"
The output is:

Creating an event through the ST Developer Portal's API Console

I'm trying to create an event using the API console and keep getting errors. Any ideas why?
I've been using different versions of the example value:
{
"name": "string",
"description": "string",
"status": "string",
"event_id": "string",
"start_epoch": 0,
"end_epoch": 0,
"industry": "string",
"archived": true,
"deleted": true,
"legacy_id": 0,
"is_public": true
}
I get the following back. Any thoughts?
{
"code": "BadRequestError",
"message": "[\"Has time can't be blank\",\"true is not included in the list\"]"
}
You will need to fetch the user/team information first
Once you have your oauth token from above and set to the Authorization header, make a call to https://developer-portal.socialtables.com/api-console#!/Authentication/get_4_0_oauth_token
This will give you the user and team object back to make subsequent calls to make events
Once you have the team_id you can now make events
You can POST to /4.0/events
Swagger doc: https://developer-portal.socialtables.com/api-console#!/Events/post_4_0_events
Example POST payload:
{
"name": "NAME",
"description": "DESCRIPTION",
"status": "new",
"start_epoch": TIME_IN_MS,
"end_epoch": TIME_IN_MS,
"industry": "INDUSTRY_TYPE",
“has_time”: 1 // 0 = all day event, 1 = from/to a specific time in day
}
- This will return the event ID under data.event.id in the response from the above POST
- You can then link the user to:
https://home.socialtables.com/events/EVENT_ID

Apache Druid segment merge task submition failure

I am using Druid 0.9.1.1 and trying to merge all the segment of a datasource per day to a single segment. Whereas the merge task initiation fails with error :
{"error":"Instantiation of [simple type, class io.druid.timeline.DataSegment] value failed: null (through reference chain: java.util.ArrayList[0])"}
I have got the segment details from segment metadata query. There is no help from driud documents as only specify raw structure of the overall query, but not the required segment detail structure(Below is how druid document suggests).
{
"type": "merge",
"id": <task_id>,
"dataSource": <task_datasource>,
"aggregations": <list of aggregators>,
"segments": <JSON list of DataSegment objects to merge>
}
example queries :
{
"type": "merge",
"id": "envoy_merge_task",
"dataSource": "dcap.envoy.diskmounts.kafka",
"segments": [{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5460959,"numRows":41577,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_1","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5448881,"numRows":41577,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_2","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5454452,"numRows":41571,"aggregators":null,"queryGranularity":null},{"id":"dcap.sermon.threshold.kafka_2017-05-22T00:00:00.000Z_2017-05-23T00:00:00.000Z_2017-05-22T07:00:02.951Z_3","intervals":["2017-05-22T00:00:00.000Z/2017-05-23T00:00:00.000Z"],"columns":{},"size":5456267,"numRows":41569,"aggregators":null,"queryGranularity":null}] }
I have tried different forms of structure for "segments" key, results in same error.
example :
"segments": [{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_1"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_2"},{"id":"dcap.envoy.diskmounts.kafka_2017-05-21T06:00:00.000Z_2017-05-21T07:00:00.000Z_2017-05-21T06:02:43.482Z_3"}]
What is right structure for segment-merge tasks.
the format i used for segments is
"segments":[
{
"dataSource": "wikiticker88",
"interval": "2015-09-12T02:00:00.000Z/2015-09-12T03:00:00.000Z",
"version": "2018-01-16T07:23:16.425Z",
"loadSpec": {
"type": "local",
"path": "/home/linux/druid-0.11.0/var/druid/segments/wikiticker88/2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z/2018-01-16T07:23:16.425Z/0/index.zip"
},
"dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user",
"metrics": "count,added,deleted,delta,user_unique",
"shardSpec": {
"type": "none"
},
"binaryVersion": 9,
"size": 198267,
"identifier": "wikiticker88_2015-09-12T02:00:00.000Z_2015-09-12T03:00:00.000Z_2018-01-16T07:23:16.425Z"
},
]
use this to get your metadata of segments
/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full

ARM - How can I get the access key from a storage account to use in AppSettings later in the template?

I'm creating an Azure Resource Manager template that instantiates multiple resources, including an Azure storage account and an Azure App Service with a Web App.
I'd like to be able to capture the primary access key (or the full connection string, either way is fine) from the newly-created storage account, and use that as a value for one of the AppSettings for the Web App.
Is that possible?
Use the listkeys helper function.
"appSettings": [
{
"name": "STORAGE_KEY",
"value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
}
]
This quickstart does something similar:
https://azure.microsoft.com/en-us/documentation/articles/cache-web-app-arm-with-redis-cache-provision/
The syntax has changed since the other answer was accepted. The error you will now hit is 'Template language expression property 'key1' doesn't exist, available properties are 'keys'
Keys are now represented as an array of keys, and the syntax is now:
"StorageAccount": "[Concat('DefaultEndpointsProtocol=https;AccountName=',variables('StorageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('StorageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]",
See: http://samcogan.com/retrieve-azure-storage-key-in-arm-script/
I faced with this issue two times. First in the 2015 and last today in May of 2017.
I need to add connection strings to the WebApp - I want to add strings automatically from generated resources during deployment from the ARM template. It can help later to not add manually this values.
First time I used old version of the function listKeys (it looks like old version returns result not as object but as value):
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2015-05-01-preview').key1)]"
},
Today last version of the working template is:
"resources": [
{
"apiVersion": "2015-08-01",
"type": "config",
"name": "connectionstrings",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites/', parameters('webSiteName'))]"
],
"properties": {
"DefaultConnection": {
"value": "[concat('Data Source=tcp:', reference(resourceId('Microsoft.Sql/servers/', parameters('sqlserverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=', parameters('databaseName'), ';User Id=', parameters('administratorLogin'), '#', parameters('sqlserverName'), ';Password=', parameters('administratorLoginPassword'), ';')]",
"type": "SQLServer"
},
"AzureWebJobsStorage": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
},
"AzureWebJobsDashboard": {
"type": "Custom",
"value": "[concat(variables('storageConnectionString'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageName')), '2016-01-01').keys[0].value)]"
}
}
},
Thanks.
below is example for adding storage account to ADLA
"storageAccounts": [
{
"name": "[parameters('DataLakeAnalyticsStorageAccountname')]",
"properties": {
"accessKey": "[listKeys(variables('storageAccountid'),'2015-05-01-preview').key1]"
}
}
],
in variable you can keep
"variables": {
"apiVersion": "[providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]]",
"storageAccountid": "[concat(resourceGroup().id,'/providers/','Microsoft.Storage/storageAccounts/', parameters('DataLakeAnalyticsStorageAccountname'))]"
},

Ember - Cannot read property 'replace' of undefined

My application
Im trying to build a fairly simple application using Laravel as a RESTfull API server and Ember as my fontend framework
My backend server lives on http://api.example.com/1.0/
My frontend lives on http://www.example.com/
Ive just started this project so I'm only a few hours into the devlopment, so there might be some configuration issues that Im missing here, but anyway.
Im trying to get a list of products from my server and display them in my ember application using ember-data
Im running ember 2.0.2 and ember-data 2.0.0
Im getting the following error in chrome.
Error
Error while processing route: product Cannot read property 'replace'
of undefined TypeError: Cannot read property 'replace' of undefined
at Object.func (http://localhost:4200/assets/vendor.js:45832:15)
at Object.Cache.get (http://localhost:4200/assets/vendor.js:23421:36)
at decamelize (http://localhost:4200/assets/vendor.js:45874:29)
at Object.func (http://localhost:4200/assets/vendor.js:45789:12)
at Object.Cache.get (http://localhost:4200/assets/vendor.js:23421:36)
at Object.dasherize (http://localhost:4200/assets/vendor.js:45878:35)
at ember$data$lib$system$normalize$model$name$$normalizeModelName (http://localhost:4200/assets/vendor.js:66295:27)
at ember$data$lib$serializers$json$serializer$$default.extend.modelNameFromPayloadKey
(http://localhost:4200/assets/vendor.js:75184:67)
at ember$data$lib$serializers$json$serializer$$default.extend._normalizeResourceHelper
(http://localhost:4200/assets/vendor.js:75064:30)
at Array.map (native)
Files
In ember I have generated a product resource giving my the following files.
// app/routes/product.js
import Ember from 'ember';
export default Ember.Route.extend({
model() {
return this.store.findAll('product');
}
});
// app/model/product.js
import DS from 'ember-data';
export default DS.Model.extend({
name: DS.attr(),
price: DS.attr()
});
JSON response
The Json returned from my api http://api.example.com/1.0/products
{
"data": [
{
"id": "1",
"name": "dolores",
"price": "59015",
"created_at": "2015-09-06 16:18:13",
"updated_at": "2015-09-06 16:18:13"
},
{
"id": "2",
"name": "debitis",
"price": "57449",
"created_at": "2015-04-07 14:45:16",
"updated_at": "2015-04-07 14:45:16"
},
...
]
}
This is adapter/serializer error, it's not descriptive though. Payload is wrong for the JSONAPIAdapter (the default adapter)
You should modify payload as:
{
"data": [
{
"id": "1",
"type": "products",
"attributes": {
"name": "dolores",
"price": "59015",
"created-at": "2015-09-06 16:18:13",
"updated-at": "2015-09-06 16:18:13"
}
}, {
"id": "2",
"type": "products",
"attributes": {
"name": "debitis",
"price": "57449",
"created-at": "2015-04-07 14:45:16",
"updated-at": "2015-04-07 14:45:16"
}
}]
}
or use RESTAdapter/Serializer with a such payload:
{
"products": [{
"id": "1",
"name": "dolores",
"price": "59015",
"created_at": "2015-09-06 16:18:13",
"updated_at": "2015-09-06 16:18:13"
}, {
"id": "2",
"name": "debitis",
"price": "57449",
"created_at": "2015-04-07 14:45:16",
"updated_at": "2015-04-07 14:45:16"
}]
}
If you can't change response payload, you have to customize Adapter/Serializer pair. Check related questions on SO.
Links for details:
Guides
JSONAPISerializer payload example
RESTSerializer payload example
The same issue happened to me.
Version details:
DEBUG: -------------------------------
ember.debug.js:DEBUG: Ember : 1.11.0
ember.debug.js:DEBUG: Ember Data : 1.0.0-beta.14.1
ember.debug.js:DEBUG: jQuery : 1.11.1
DEBUG: -------------------------------
Cause of the issue:
It seems that adding attributes to Handlebars elements using inline if helpers also causes the issue (whether the property that you are using on your condition is true, false, null or undefined).
my-component.hbs
<button class="btn btn-solve-my-problems" {{action 'solveIt}} {{if isNotSolvable 'disabled'}}>
Solve my problems!
</button>
my-component.coffee
isNotSolveble: (->
if #get('somethingMeaningful') then true else null
).property('somethingMeaningful')
The solution:
Instead of using the inline if helper, I am just attributing the value of isNotSolvable to the disabled attribute (nothing was changed on the my-component.coffee file).
my-component.hbs
<button class="btn btn-solve-my-problems" {{action 'solveIt}} disabled="{{isNotSolvable}}">
Solve my problems!
</button>
PS.: the null being returned from the method has to do with the manipulation of the disabled attribute. It has nothing to do with the issue reported on this thread. It only reflects my scenario and how I've solved the issue. More details here.