Looked all over Azure Data Explorer documentation for migration scenarios and I didn't manage to find an article on this.
What I'm trying to do is to apply migration to incoming data and I thought of putting it in the Update Policy. I don't know if this is a good idea or not, let me know. Aside for this, I don't know if what I'm doing is good enough or if it could be made better.
I have table Target and table Source. Source has a dynamic Payload column and I'm mapping that column to the table Target IF it has a certain property. I did it as such:
let new_data = Source
| where Payload.Name == 'NameImLookingFor'
;
let good_data = new_data
| where isnull(Payload.DeprecatedField)
| project
FieldA = todouble(Payload.FieldA),
FieldB = todouble(Payload.FieldB),
FieldC = todouble(Payload.FieldC)
;
let migrated_data = new_data
| where isnotnull(Payload.DeprecatedField)
| project
FieldA = iff(toint(Payload.DeprecatedField)==0,todouble(Payload.DeprecatedFieldValue), Payload.UndefinedMemeber),
FieldB = iff(toint(Payload.DeprecatedField)==1,todouble(Payload.DeprecatedFieldValue), Payload.UndefinedMemeber),
FieldC = iff(toint(Payload.DeprecatedField)==2,todouble(Payload.DeprecatedFieldValue), Payload.UndefinedMemeber)
;
good_data
| union migrated_data
I have some questions and incertitudes:
iff must have an else value specified. I want it to be null, but that type doesn't exist so I'm using Payload. some field that I'm sure it doesn't exists on the object so I have an empty value. Is this good enough? Could it be better?
I'm calling that iff 3 times, could a function be made for it? If yes, how and where? Should I place that in the update policy also or define it somewhere else?
Could it be done in a single query? I looked into case statement but I didn't feel like it would make my life easier.
Thanks.
Using an update policy is valid (though, ideally, you'd fix the data at its source, if possible, before ingestion into Kusto/ADX).
You could replace your logic with the following:
Source
| where Payload.Name == 'NameImLookingFor'
| extend df = toint(Payload.DeprecatedField)
| project FieldA = case(isnull(df), todouble(Payload.FieldA), case(df == 0, todouble(Payload.DeprecatedFieldValue), double(null))),
FieldB = case(isnull(df), todouble(Payload.FieldB), case(df == 1, todouble(Payload.DeprecatedFieldValue), double(null))),
FieldC = case(isnull(df), todouble(Payload.FieldC), case(df == 2, todouble(Payload.DeprecatedFieldValue), double(null)))
Related
I have a table as shown
ID (int) | DATA (bytea)
1 | \x800495356.....
The contents of the data column have been stored via a python script
result_dict = {'datapoint1': 100, 'datapoint2': 2.334'}
table.data = pickle.dumps(result_dict)
I can easily read the data back using
queried_dict = pickle.loads(table.data)
But I don't know how to query it directly as a json or even as plain text in postgres alone. I have tried the following query and many versions of it but it doesn't seem to work
-- I don't know what should come between SELECT and FROM
SELECT encode(data, 'escape') AS res FROM table WHERE id = 1;
-- I need to get this or somewhere close to this as the query result
res |
{"datapoint1": 100, "datapoint2": 2.33}
Thanks a lot in advance to everyone trying to help.
Hi i want to update a specific row of my oracle database via apex. So i need to reach/address the specific row via where clause.
example entries:
GATTUNG
ART
UNTERART
ABART
VARIETÄT
AAA
AAA
AAA
NULL
NULL
AAA
AAA
NULL
NULL
AAA
AAA
AAA
NULL
NULL
NULL
Now i have two approaches.
first:
UPDATE TBL_PFLANZE
SET NAMEN =:P1_NAMEN
WHERE (GATTUNG = :P1_GATTUNG AND ART = :P1_ART_KREUZUNG)
AND ((UNTERART=:P1_UNTERART OR :P1_UNTERART IS NULL) AND (VARIETÄT=:P1_VARIETAET OR :P1_VARIETAET IS NULL) AND (ABART=:P1_ABART OR :P1_ABART IS NULL));
second:
UPDATE TBL_PFLANZE
SET NAMEN ='&P1_NAMEN.'
WHERE (GATTUNG = '&P1_GATTUNG.' AND ART = '&P1_ART_KREUZUNG.')
AND (UNTERART &P1_WHERE_UNTERART. AND VARIETÄT &P1_WHERE_VARIETAET. AND ABART &P1_WHERE_ABART.);
the differences of both approaches are the P1_WHERE_...variables.
the first one use for example :P1_UNTERART and contains the whole value
the second one use for example &P1_WHERE_UNTERART. and contains = '&P1_UNTERART.' OR IS NULL
my problem is:
the first one updates all entries if i only set GATTUNG and ART (if i do not specify one of the other variables)
the second will work, but is not the right approach as I should be using binding variables instead of &VAR.
So my question is, how can i use the first approach which the desired result... :(
Does this do the trick ? It will update the rows if any of the columns have the same value as the corresponding page item or the database column and page item are both null:
UPDATE tbl_pflanze
SET
namen = :P1_NAMEN
WHERE
( gattung = :P1_GATTUNG AND art = :P1_ART_KREUZUNG)
AND
( ( NVL(unterart,'X') = NVL(:P1_UNTERART,'X')) AND
( NVL(varietät,'X') = NVL(:P1_VARIETAET,'X')) AND
( NVL(abart,'X') = NVL(:P1_ABART,'X')) );
The way I see it, the 1st query is correct while the 2nd is wrong.
Why is it wrong? Because you're using strings. This:
SET NAMEN ='&P1_NAMEN.'
WHERE (GATTUNG = '&P1_GATTUNG.'
would want to put litally &P1_NAMEN. string into the NAMEN column for all rows whose GATTUNG column contains literally &P1_GATTUNG. string in it (and there's most probably none, so the 2nd query won't do anything, ever).
So, what is your problem, exactly? Why do you think that the 1st query doesn't work? Did you run the page in debug mode and reviewed debug results? Could it be that P1 page's items aren't stored into session state so UPDATE statement doesn't see them?
Also, where do you execute that UPDATE? As a process which runs after button is pressed? Something else?
My employer has switched data systems and reporting tools. We used to use Report Builder with a nicely built data model that allowed me to do some complex filtering easily. Then we used Business Objects, and though I didn't like it very much, it also let me do some complex filtering. Now we're back to Report Builder, but the data model is different, and the only filtering I seem to be able to do is a string of AND operators.
(Note: I'm self-taught on both Report Builder and Business Objects. I have minimal experience with the SQL coding language itself. Also, actual data labels have been changed in this example.)
I'm pulling from a large amount of data, so I need to filter on the query level. I first need to include data based on five criteria, like this.
| SYSTEM.REGION.REGION_STATUS_CODE = N'1'
| SYSTEM.STATE.STATE_STATUS_CODE = N'1'
AND | SYSTEM.ORDERS.DISCARDED_DATE IS NULL
| SYSTEM.SERVICE.SERVICE_DISCARDED_DATE IS NULL
| SYSTEM.SERVICE.SERVICE_STATUS_CODE = N'01'
Then I need to include data that fits one of two pairings, like this.
| | SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Retail'
| AND | SYSTEM.ORDERS.DISCOUNT_CODE = N'N/A'
OR |
| | SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Wholesale'
| AND | SYSTEM.ORDERS.DISCOUNT_CODE != N'N/A'
After I built my query using the query designer and switched to text mode, it gave me this.
WHERE
SYSTEM.REGION.REGION_STATUS_CODE = N'1'
AND SYSTEM.STATE.STATE_STATUS_CODE = N'1'
AND SYSTEM.ORDERS.DISCARDED_DATE IS NULL
AND SYSTEM.SERVICE.SERVICE_DISCARDED_DATE IS NULL
AND SYSTEM.SERVICE.SERVICE_STATUS_CODE = N'01'
AND SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Retail'
AND SYSTEM.ORDERS.DISCOUNT_CODE = N'N/A'
AND SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Wholesale'
AND SYSTEM.ORDERS.DISCOUNT_CODE != N'N/A'
I've tried putting parentheses in, but I must have done it wrong because the query ran for ages before essentially giving me the entire database.
Anybody care to help a SQL newbie?
Presuming everything else is right, it should just be about applying parenthesis to get the logic right. Using slightly exaggerated whitespace to try and make it clear:
WHERE
SYSTEM.REGION.REGION_STATUS_CODE = N'1'
AND SYSTEM.STATE.STATE_STATUS_CODE = N'1'
AND SYSTEM.ORDERS.DISCARDED_DATE IS NULL
AND SYSTEM.SERVICE.SERVICE_DISCARDED_DATE IS NULL
AND SYSTEM.SERVICE.SERVICE_STATUS_CODE = N'01'
AND (
(SYSTEM.ORDERS.DISCOUNT_CODE = N'N/A'
AND SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Retail')
OR
(SYSTEM.ORDERS.DISCOUNT_CODE != N'N/A'
AND SYSTEM.SERVICE.SERVICE_CONTRACT_CODE = N'Wholesale')
)
(It still may run forever, but that's more a factor of database size and indexing.)
I have the following models:
class Collection(models.Model):
...
class Record(models.Model):
collection = models.ForeignKey(Collection, related_name='records')
filename = models.CharField(max_length=256)
checksum = models.CharField(max_length=64)
class Meta:
unique_together = (('filename', 'collection'),)
I want to perform the following query:
For each filename of Record I want to know the Collections that:
Do not provide a Record with that filename
or that provide such a Record but has a differing checksum
I have in mind an output like that:
| C1 C2 C3 <- collections
-----------+------------
file-1.txt | x
file-2.txt | x
file-3.txt | ! ! !
file-4.txt | x ! !
file-5.txt | ! ! x
x = missing
! = different checksum
What I've com up so far is that I create a query for each Collection, excluding all filenames that are within this collection but exist in others.
for collection in collections:
other_collections = [c for c in collections if c is not collection]
results[collection] = qs.filter(collection__in=other_collections).exclude(
filename__in=qs.filter(
collection=collection
).values_list('filename', flat=True)
).order_by('filename').values_list('filename', flat=True)
This somewhat solves the first part of my question, but is rather quirky and requires post-processing to get to the format I desire. And, more importantly, it does not address the checksum comparison.
Is it possible to perform the two queries in one combined step to get the results in the format I described above?
The solution would not necessarily have to use the QuerySet APIs, a fallback to raw SQL is fine by me too.
It is not possible to write a SQL query that returns a variable number of columns, although you can achieve that effect if you wrap everything in an array or JSON object.
If you know the collections, you could write SQL like this:
SELECT r.filename,
(SELECT r.checksum = r2.checksum FROM records r2 WHERE r.filename = r2.filename AND r2.collection_id = 1) AS c1,
(SELECT r.checksum = r2.checksum FROM records r2 WHERE r.filename = r2.filename AND r2.collection_id = 2) AS c2,
...
FROM records r
WHERE r.collection_id = 1
GROUP BY r.filename, r.checksum
For each filename/collection pair, you will get NULL if the collection doesn't have the record, true if the collection has it with the right checksum, or false if the collection has it with a different checksum.
I include WHERE r.collection_id = 1 because otherwise for the checksum comparison, you have to answer "different from what?"
Whats the best way to do a query like this, which will update more than one row:
UPDATE B
SET col_a = 1
Where col_a = 0
I can load the entities then loop them to do the EntitySave for each one, however this seems like overkill. Should I use HQL to do the update? Can I use HQL to do the update?
You can absolutely run an update on more than one row using ORMExecuteQuery. This is an example of a project I worked on:
ORMExecuteQuery("UPDATE Part SET ReleaseDate = :ReleaseDate, ChangeNote = :ChangeNote WHERE ID IN (#Arguments.PartIDs#)", {ReleaseDate = Arguments.ReleaseDate, ChangeNote = Arguments.ChangeNote});
Easy as that... Notice that you specify the parameters as :VarName and feed them in as the second argument of the ORMExecuteQuery function.