I have a column named Con_TYPE in which there are multiple types of connections such as fiberoptic, satellite, 3g etc.
And I want to sort them only into 2 rows:
fiberoptic
5
others
115
Can anybody help me?
Thanks in advance
You can use Calculated dimension or Mapping load
Lets imagine that the data, in its raw form, looks like this:
dimension: Con_TYPE
measure: Sum(value)
Calculated dimension
You can add expressions inside the dimension. If we have a simple if statement as an expression then the result is:
dimension: =if(Con_TYPE = 'fiberoptic', Con_TYPE, 'other')
measure: Sum(Value)
Mapping load
Mapping load is a script function so we'll have to change the script a bit:
// Define the mapping. In our case we want to map only one value:
// fiberoptic -> fiberoptic
// we just want "fiberoptic" to be shown the same "fiberoptic"
TypeMapping:
Mapping
Load * inline [
Old, New
fiberoptic, fiberoptic
];
RawData:
Load
Con_TYPE,
value,
// --> this is where the mapping is applied
// using the TypeMapping, defined above we are mapping the values
// in Con_TYPE field. The third parameter specifies what value
// should be given if the field value is not found in the
// mapping table. In our case we'll chose "Other"
ApplyMap('TypeMapping', Con_TYPE, 'Other') as Con_TYPE_Mapped
;
Load * inline [
Con_TYPE , value
fiberoptic, 10
satellite , 1
3g , 7
];
// No need to drop "TypeMapping" table since its defined with the
// "Mapping" prefix and Qlik will auto drop it at the end of the script
And we can use the new field Con_TYPE_Mapped in the ui. And the result is:
dimension: Con_TYPE_Mapped
measure: Sum(Value)
Pros/Cons
calculated dimension
+ easy to use
+ only UI change
- leads to performance issues on mid/large datasets
- have to be defined (copy/paste) per table/chart. Which might lead to complications if have to be changed across the whole app (it have to be changed in each object where defined)
mapping load
+ no performance issues (just another field)
+ the mapping table can be defined inline or loaded from an external source (excel, csv, db etc)
+ the new field can be used across the whole app and changing the values in the script will not require table/chart change
- requires reload if the mapping is changed
P.S. In both cases selecting Other in the tables will correctly filter the values and will show data only for 3g and satellite
Related
Using TypeORM, I'm trying to create ViewEntities that depend on each other, for example "View B" select from "View A". No matter what I do I can't get the ViewEntities to get created in the order of dependency. Sometimes "View B" is created first, and the synchronization process fails, because it can't find "View A", since it's not created yet.
The error:
QueryFailedError: relation "public.course_item_view" does not exist
Solutions I have tried:
Renaming the ViewEntity files (to check if the system uses ABC ordering on file names)
Renaming the ViewEntity classes (to check if the system uses ABC ordering on class names)
Renaming the ViewEntity's "name" property (to check if the system uses ABC ordering on the final SQL view names)
Reordering the ViewEntity class references in the "entities: []" array of the connection options
Reordering the ViewEntity class imports in the file where I declare the connection options
Removing/Adding the file again (to check if the system uses Creation Date based ordering)
Modifying the files (to check if the system uses Modification Date based ordering)
All of these failed. I cannot figure out how the system determines the order in which the view's are created.
Any help would be GREATLY appreciated!!
Expected Behavior
The view's should be created in an order that is either specified by a property inside the views, or the order should be resolved automatically from the SELECT statements (dependency array), or it should be based on the order in which I reference the ViewEntities in the "entities: []" array of the connection options, or any other solution would be perfect where one could determine the order in which the ViewEntities are created.
Actual Behavior
The ViewEntites are created in an order that I honestly can't understand. Sometimes a dependent ViewEntity is created before the ViewEntitiy it depends on. This causes the synchronization to fail.
File name: "CourseItemView" which resolves to: "course_item_view"
#ViewEntity({
expression: `
SELECT
"uvcv"."userId",
"uvcv"."courseId",
"uvcv"."videoId",
CAST (null AS integer) AS "examId",
"uvcv"."isComplete" AS "isComplete"
FROM public.video_completed_view AS "uvcv"
UNION ALL
SELECT
"uecv"."userId",
"uecv"."courseId",
CAST (null AS integer) AS "videoId",
"uecv"."examId",
"uecv"."isCompleted" AS "isComplete"
FROM public.user_exam_completed_view AS "uecv"
.
.
File name: "CourseItemStateView" which resolves to: "course_item_state_view"
This DEPENDS on the "course_item_view", as you can see in the SQL
#ViewEntity({
expression: `
SELECT
"course"."id" AS "courseId",
"user"."id" AS "userId",
"civ"."videoId" AS "videoId",
"civ"."isComplete" AS "isVideoCompleted",
"civ"."examId" AS "examId",
"civ"."isComplete" AS "isExamCompleted"
FROM public."course"
LEFT JOIN public."user"
ON 1 = 1
LEFT JOIN public.course_item_view AS "civ" ------------------- HERE
ON "civ"."courseId" = "course"."id"
AND "civ"."userId" = "user"."id"
ORDER BY "civ"."videoId","civ"."examId"
`
})
.
.
My connection options:
const postgresOptions = {
// properties, passwords etc...
entities: [
// entities....
// ...
// ...
// views
VideoCompletedView,
UserExamCompletedView,
UserExamAnswerSessionView,
UserVideoMaxWatchedSecondsView,
CourseItemView, --------------------------------HERE
CourseItemStateView ---------------------------HERE
],
} as ConnectionOptions;
createConnection(postgresOptions )
Steps to Reproduce
Create ViewEntites that depend on each other
You will run into this issue, but is hard to say exactly why and when, this is the main problem.
I have three models:
class Customer(models.Model):
pass
class IssueType(models.Model):
pass
class IssueTypeConfigPerCustomer(models.Model):
customer=models.ForeignKey(Customer)
issue_type=models.ForeignKey(IssueType)
class Meta:
unique_together=[('customer', 'issue_type')]
How can I find all tuples of (custmer, issue_type) where there is no IssueTypeConfigPerCustomer object?
I want to avoid a loop in Python. A solution which solves this in the DB would be preferred.
Background: for every customer and for every issue-type, there should be a config in the DB.
If you can afford to make one database trip for each issue type, try something like this untested snippet:
def lacking_configs():
for issue_type in IssueType.objects.all():
for customer in Customer.objects.filter(
issuetypeconfigpercustomer__issue_type=None
):
yield customer, issue_type
missing = list(lacking_configs())
This is probably OK unless you have a lot of issue types or if you are doing this several times per second, but you may also consider having a sensible default instead of making a config object mandatory for each combination of issue type and customer (IMHO it is a bit of a design-smell).
[update]
I updated the question: I want to avoid a loop in Python. A solution which solves this in the DB would be preferred.
In Django, every Queryset is either a list of Model instances or a dict (values querysets), so it is impossible to return the format you want (a list of tuples of Model) without some Python (and possibly multiple trips to the database).
The closest thing to a cross product would be using the "extra" method without a where parameter, but it involves raw SQL and knowing the underlying table name for the other model:
missing = Customer.objects.extra(
select={"issue_type_id": 'appname_issuetype.id'},
tables=['appname_issuetype']
)
As a result, each Customer object will have an extra attribute, "issue_type_id", containing the id of one IssueType. You can use the where parameter to filter based on NOT EXISTS (SELECT 1 FROM appname_issuetypeconfigpercustomer WHERE issuetype_id=appname_issuetype.id AND customer_id=appname_customer.id). Using the values method you can have something close to what you want - this is probably enough information to verify the rule and create the missing records. If you need other fields from IssueType just include them in the select argument.
In order to assemble a list of (Customer, IssueType) you need something like:
cross_product = [
(customer, IssueType.objects.get(pk=customer.issue_type_id))
for customer in
Customer.objects.extra(
select={"issue_type_id": 'appname_issuetype.id'},
tables=['appname_issuetype'],
where=["""
NOT EXISTS (
SELECT 1
FROM appname_issuetypeconfigpercustomer
WHERE issuetype_id=appname_issuetype.id
AND customer_id=appname_customer.id
)
"""]
)
]
Not only this requires the same number of trips to the database as the "generator" based version but IMHO it is also less portable, less readable and violates DRY. I guess you can lower the number of database queries to a couple using something like this:
missing = Customer.objects.extra(
select={"issue_type_id": 'appname_issuetype.id'},
tables=['appname_issuetype'],
where=["""
NOT EXISTS (
SELECT 1
FROM appname_issuetypeconfigpercustomer
WHERE issuetype_id=appname_issuetype.id
AND customer_id=appname_customer.id
)
"""]
)
issue_list = dict(
(issue.id, issue)
for issue in
IssueType.objects.filter(
pk__in=set(m.issue_type_id for m in missing)
)
)
cross_product = [(c, issue_list[c.issue_type_id]) for c in missing]
Bottom line: in the best case you make two queries at the cost of legibility and portability. Having sensible defaults is probably a better design compared to mandatory config for each combination of Customer and IssueType.
This is all untested, sorry if some homework was left for you.
We have a weekly backup process which exports our production Google Appengine Datastore onto Google Cloud Storage, and then into Google BigQuery. Each week, we create a new dataset named like YYYY_MM_DD that contains a copy of the production tables on that day. Over time, we have collected many datasets, like 2014_05_10, 2014_05_17, etc. I want to create a data set Latest_Production_Data that contains a view for each of the tables in the most recent YYYY_MM_DD dataset. This will make it easier for downstream reports to write their query once and always retrieve the most recent data.
To do this, I have code that gets the most recent dataset and the names of all the tables that dataset contains from the BigQuery API. Then, for each of these tables, I fire a tables.insert call to create a view that is a SELECT * from the table I am looking to create a reference to.
This fails for tables that contain a RECORD field, from what looks to be a pretty benign column-naming rule.
For example, I have this table:
For which I issue this API call:
{
'tableReference': {
'projectId': 'redacted',
'tableId': u'AccountDeletionRequest',
'datasetId': 'Latest_Production_Data'
}
'view': {
'query': u'SELECT * FROM [2014_05_17.AccountDeletionRequest]'
},
}
This results in the following error:
HttpError: https://www.googleapis.com/bigquery/v2/projects//datasets/Latest_Production_Data/tables?alt=json returned "Invalid field name "__key__.namespace". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long.">
When I execute this query in the BigQuery web console, the columns are renamed to translate the . to an _. I kind of expected the same thing to happen when I issued the create view API call.
Is there an easy way I can programmatically create a view for each of the tables in my dataset, regardless of their underlying schema? The problem I'm encountering now is for record columns, but another problem I anticipate is for tables that have repeated fields. Is there some magic alternative to SELECT * that will take care of all these intricacies for me?
Another idea I had was doing a table copy, but I would prefer not to duplicate the data if I can at all avoid it.
Here is the workaround code I wrote to dynamically generate a SELECT statement for each of the tables:
def get_leaf_column_selectors(dataset, table):
schema = table_service.get(
projectId=BQ_PROJECT_ID,
datasetId=dataset,
tableId=table
).execute()['schema']
return ",\n".join([
_get_leaf_selectors("", top_field)
for top_field in schema["fields"]
])
def _get_leaf_selectors(prefix, field):
if prefix:
format = prefix + ".%s"
else:
format = "%s"
if 'fields' not in field:
# Base case
actual_name = format % field["name"]
safe_name = actual_name.replace(".", "_")
return "%s as %s" % (actual_name, safe_name)
else:
# Recursive case
return ",\n".join([
_get_leaf_selectors(format % field["name"], sub_field)
for sub_field in field["fields"]
])
We had a bug where you needed to need to select out the individual fields in the view and use an 'as' to rename the fields to something legal (i.e they don't have '.' in the name).
The bug is now fixed, so you shouldn't see this issue any more. Please ping this thread or start a new question if you see it again.
I am trying to get the ID under "General" from a feature item in rally. This is my query:
body = { "find" => {"_ProjectHierarchy" => projectID, "_TypeHierarchy" => "PortfolioItem/Feature"
},
"fields" => ["FormattedID","Name","State","Release","_ItemHierarchy","_TypeHierarchy","Tags"],
"hydrate" => ["_ItemHierarchy","_TypeHierarchy","Tags"],
"fetch"=>true
}
I am not able to get any value for FormattedID, I tried using "_UnformattedID" but it pulls up an entirely different value than the FormattedID. Any help would be appreciated.
LBAPI does not have FormattedID field. You are correct using _UnformattedID. It is the FormattedID without the prefix. For example, this query:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/1111/artifact/snapshot/query.js?find={"_ProjectHierarchy":2222,"_TypeHierarchy":"PortfolioItem/Feature","State":"Developing",_ValidFrom: {$gte: "2013-06-01TZ",$lt: "2013-09-01TZ"}},sort:{_ValidFrom:-1}}&fields=["_UnformattedID","Name","State"]&hydrate=["State"]&compress=true&pagesize:200
shows _UnformattedID that correspond to FormattedID as this screenshot shows:
I noticed your are using fields and fetch . Per LBAPI's documentation, it uses fields rather than fetch. If you want to get all fields, use fields=true
As far as the missing custom fields, make sure that the custom field value was set within the dates of the query.
Compare these almost identical queries: the first query does not return a custom field, the second query does.
Query #1:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/1111/artifact/snapshot/query.js?find={"_ProjectHierarchy":2222,"_TypeHierarchy":"PortfolioItem/Feature","State":"Developing",_ValidFrom: {$gte: "2013-06-01TZ",$lt: "2013-09-01TZ"}}}&fields=["_UnformattedID","Name","State","c_PiCustomField"]&hydrate=["State","c_PiCustomField"]
Query #2:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/11111/artifact/snapshot/query.js?find={"_ProjectHierarchy":2222,"_TypeHierarchy":"PortfolioItem/Feature","State":"Developing",__At: "current"}&fields=["_UnformattedID","Name","State","c_PiCustomField"]&hydrate=["State","c_PiCustomField"]
The first query uses time period: _ValidFrom: {$gte: "2013-06-01TZ",$lt: "2013-09-01TZ"}
The second query uses __At: "current"
Let's say I just create a new custom field on PortfolioItem. It is not possible to create a custom field on PorfolioItem/Feature, so the field is created on PI, but both queries still use "_TypeHierarchy":"PortfolioItem/Feature".
After I created this custom field, called PiCustomField, I set a value of that field for a specific Feature, F4.
The first query does not have a single snapshot that includes that field because that field did not exist in the time period we lookback. We can't change the past.
The second query returns this field for F4. It does not return it for other Features because all other Features do not have this field set.
Here is the screenshot:
I am missing the SQL out of this to Bulk update attributes by SKU/UPC.
Running EE1.10 FYI
I have all the rest of the code working but I"m not sure the who/what/why of
actually updating our attributes, and haven't been able to find them, my logic
is
Open a CSV and grab all skus and associated attrib into a 2d array
Parse the SKU into an entity_id
Take the entity_id and the attribute and run updates until finished
Take the rest of the day of since its Friday
Here's my (almost finished) code, I would GREATLY appreciate some help.
/**
* FUNCTION: updateAttrib
*
* REQS: $db_magento
* Session resource
*
* REQS: entity_id
* Product entity value
*
* REQS: $attrib
* Attribute to alter
*
*/
See my response for working production code. Hope this helps someone in the Magento community.
While this may technically work, the code you have written is just about the last way you should do this.
In Magento, you really should be using the models provided by the code and not write database queries on your own.
In your case, if you need to update attributes for 1 or many products, there is a way for you to do that very quickly (and pretty safely).
If you look in: /app/code/core/Mage/Adminhtml/controllers/Catalog/Product/Action/AttributeController.php you will find that this controller is dedicated to updating multiple products quickly.
If you look in the saveAction() function you will find the following line of code:
Mage::getSingleton('catalog/product_action')
->updateAttributes($this->_getHelper()->getProductIds(), $attributesData, $storeId);
This code is responsible for updating all the product IDs you want, only the changed attributes for any single store at a time.
The first parameter is basically an array of Product IDs. If you only want to update a single product, just put it in an array.
The second parameter is an array that contains the attributes you want to update for the given products. For example if you wanted to update price to $10 and weight to 5, you would pass the following array:
array('price' => 10.00, 'weight' => 5)
Then finally, the third and final attribute is the store ID you want these updates to happen to. Most likely this number will either be 1 or 0.
I would play around with this function call and use this instead of writing and maintaining your own database queries.
General Update Query will be like:
UPDATE
catalog_product_entity_[backend_type] cpex
SET
cpex.value = ?
WHERE cpex.attribute_id = ?
AND cpex.entity_id = ?
In order to find the [backend_type] associated with the attribute:
SELECT
backend_type
FROM
eav_attribute
WHERE entity_type_id =
(SELECT
entity_type_id
FROM
eav_entity_type
WHERE entity_type_code = 'catalog_product')
AND attribute_id = ?
You can get more info from the following blog article:
http://www.blog.magepsycho.com/magento-eav-structure-role-of-eav_attributes-backend_type-field/
Hope this helps you.