I am new to integration and MuleSoft so I need your help. I have a flat file with different invoice line items per salesID, like this:
SalesOrderID OrderQty UnitPrice
43659 70 2024.994
43659 70 2024.994
43660 1 419.4589
43660 1 874.794
43661 1 809.76
I want to insert total invoice amount and quantity in another CSV file using Mule, something like this:
SalesOrderID OrderQty UnitPrice
43659 140 4049.988
43660 2 1294.4589
43661 1 809.76
I know how to do this in informatica,but im trying to figure out a way to do this in MuleSoft. How can I sum up all the line items and group them by SalesOrderID? Any help/clue will be really appreciated.
Thanks.
There are a number of ways to do this, let me explain my personal favourite assuming you are using Mule community edition.
Use the superb library: SuperCSV, it will allow you not only to parse the data (including those dot numbers) but also validate it and give back an exact report on why the file is broken in case it is.
This could be done in a transformer that transform the inbound stream and return either a map of maps (or even better a iterator that handles the whole thing but this is more difficult) or a custom exception with the error report.
Given that this requirement is one that faces Mule developers even today, it's useful to see a solution based on the Mule 4.x runtime and Dataweave 2.x.
If the data came from a file or is otherwise a monolithic stream of text, use the splitBy() function to get an array of text lines.
payload splitBy '\n'
remove the first line as the headers should not be calculated
payload[1 to -1] // this is my favorite way to do it
Now use the reduce() function to iterate over each of the lines in turn, updating the accumulator each time to account for quantity and price.
Hopefully that helps
Related
I'm trying to write a query that returns the vulnerabilities found by "Built-in Qualys vulnerability assessment" in log analytics.
It was all going smoothly I was getting the values from the properties Json and turning then into separated strings but I found out that some of the terms posses more than one value, and I need to get all of them in a single cell.
My query is like this right now
securityresources | where type =~ "microsoft.security/assessments/subassessments"
| extend assessmentKey=extract(#"(?i)providers/Microsoft.Security/assessments/([^/]*)", 1, id), IdAzure=tostring(properties.id)
| extend IdRecurso = tostring(properties.resourceDetails.id)
| extend NomeVulnerabilidade=tostring(properties.displayName),
Correcao=tostring(properties.remediation),
Categoria=tostring(properties.category),
Impacto=tostring(properties.impact),
Ameaca=tostring(properties.additionalData.threat),
severidade=tostring(properties.status.severity),
status=tostring(properties.status.code),
Referencia=tostring(properties.additionalData.vendorReferences[0].link),
CVE=tostring(properties.additionalData.cve[0].link)
| where assessmentKey == "1195afff-c881-495e-9bc5-1486211ae03f"
| where status == "Unhealthy"
| project IdRecurso, IdAzure, NomeVulnerabilidade, severidade, Categoria, CVE, Referencia, status, Impacto, Ameaca, Correcao
Ignore the awkward names of the columns, for they are in Portuguese.
As you can see in the "Referencia" and "CVE" columns, I'm able to extract the values from a specific index of the array, but I want all links of the whole array
Without sample input and expected output it's hard to understand what you need, so trying to guess here...
I think that summarize make_list(...) by ... will help you (see this to learn how to use make_list)
If this is not what you're looking for, please delete the question, and post a new one with minimal sample input (using datatable operator), and expected output, and we'll gladly help.
I would like to know all stock symbols that are no longer being traded for the last 1 year. How can I do this using AFL code via Amibroker's Auto Analysis/Scan tool?
Explore on 1 recent bar:
Gap=252;
Filter=Now(3)-DateNum()>=Gap;
I am missing the SQL out of this to Bulk update attributes by SKU/UPC.
Running EE1.10 FYI
I have all the rest of the code working but I"m not sure the who/what/why of
actually updating our attributes, and haven't been able to find them, my logic
is
Open a CSV and grab all skus and associated attrib into a 2d array
Parse the SKU into an entity_id
Take the entity_id and the attribute and run updates until finished
Take the rest of the day of since its Friday
Here's my (almost finished) code, I would GREATLY appreciate some help.
/**
* FUNCTION: updateAttrib
*
* REQS: $db_magento
* Session resource
*
* REQS: entity_id
* Product entity value
*
* REQS: $attrib
* Attribute to alter
*
*/
See my response for working production code. Hope this helps someone in the Magento community.
While this may technically work, the code you have written is just about the last way you should do this.
In Magento, you really should be using the models provided by the code and not write database queries on your own.
In your case, if you need to update attributes for 1 or many products, there is a way for you to do that very quickly (and pretty safely).
If you look in: /app/code/core/Mage/Adminhtml/controllers/Catalog/Product/Action/AttributeController.php you will find that this controller is dedicated to updating multiple products quickly.
If you look in the saveAction() function you will find the following line of code:
Mage::getSingleton('catalog/product_action')
->updateAttributes($this->_getHelper()->getProductIds(), $attributesData, $storeId);
This code is responsible for updating all the product IDs you want, only the changed attributes for any single store at a time.
The first parameter is basically an array of Product IDs. If you only want to update a single product, just put it in an array.
The second parameter is an array that contains the attributes you want to update for the given products. For example if you wanted to update price to $10 and weight to 5, you would pass the following array:
array('price' => 10.00, 'weight' => 5)
Then finally, the third and final attribute is the store ID you want these updates to happen to. Most likely this number will either be 1 or 0.
I would play around with this function call and use this instead of writing and maintaining your own database queries.
General Update Query will be like:
UPDATE
catalog_product_entity_[backend_type] cpex
SET
cpex.value = ?
WHERE cpex.attribute_id = ?
AND cpex.entity_id = ?
In order to find the [backend_type] associated with the attribute:
SELECT
backend_type
FROM
eav_attribute
WHERE entity_type_id =
(SELECT
entity_type_id
FROM
eav_entity_type
WHERE entity_type_code = 'catalog_product')
AND attribute_id = ?
You can get more info from the following blog article:
http://www.blog.magepsycho.com/magento-eav-structure-role-of-eav_attributes-backend_type-field/
Hope this helps you.
How can i add additional information to the Magento Packingslip PDF. I am using integrated label paper, so i would like to add repeat the customers delivery address at the footer and also, some details like total quantity of items in the order and the cost of the items in the order. I am currently modifying local files in: Mage/Sales/Model/Order/Pdf/ but I have only managed to change to font.
EDIT:
Okay, i have made good progress and added most of the information i need, however, i have now stumbled across a problem.. I would like to get the total weight of all items in each order and Quantity.
I am using this code in the shipment.php:
Under the foreach (This is useful in case an order has a split delivery - as you can have one order with multiple "shipments" So this is why I have the code after here:
foreach ($shipment->getAllItems() as $item){
if ($item->getOrderItem()->getParentItem()) {
continue;
...
then I have this further down:
$shippingweight=0;
$shippingweight= $item->getWeight()*$item->getQty();
$page->drawText($shippingweight . Mage::helper('sales'), $x, $y, 'UTF-8');
This is great for Row totals. But not for the whole shipment. What I need to do is have this bit of code "added up" for each item in the shipment to create the total Weight of the whole shipment. It is very important that I only have the total of the shipment - not the order as I will be splitting shipments.
Almost at the end of getPdf(...) in Mage_Sales_Model_Order_Pdf_Shipment you will find the code that inserts new pages (lines 93-94 in 1.5.0.1)
if($this->y<15)
$page = $this->newPage(....)
happening in a for-loop.
You can change the logic here to make it change page earlier to make room for your extra information and then add it before the page shift. You should also place code after the for-block if you want it to appear on the last page as well.
Note: You should not change files in app/code/code/Mage directly. Instead, you should place your changed files under app/code/local/Mage using the same folder structure. That way your changes won't accidently get overwritten in an upgrade.
This should get you the total number of items in your order (there might be a faster way but don't know it off the top of my head):
$quote = Mage::getModel('sales/quote')->load($order->getQuoteId());
$itemsCount = $quote->getItemsSummaryQty();
For invoice PDF
at app/code/local/Mage/Sales/Model/Order/Pdf/Items/Invoice/Default.php
add this before $lineBlock
$product = Mage::getModel('catalog/product')->loadByAttribute('sku', $this->getSku($item), array('weight'));
$lines[][] = array(
'text' => 'Weight: '. $product->getData('weight')*1 .'Kg/ea. Total: ' .$product->getData('weight')*$item->getQty() . 'Kg',
'feed' => 400
);
I have a wierd issue. I am loading 1k invoice objects, header first then details in my DAL. I am using VB.NET on this project. I am able to get the invoice headers just fine. When I get to loading the details for each invoice I am getting a timeout on SQL Server. I increased the timeout to 5 minutes but still the same thing. If I reduce the invoice count to 200 it works fine.
Here is what I am doing
//I already loaded the invoice headers. I am now iterating each invoice to get it's detail
For Each invoice As Invoice In invoices
drInvoiceItems = DBSqlHelperFactory.ExecuteReader(CONNECTION_STRING, CommandType.StoredProcedure, "dbo.getinvoiceitem", _
New SqlParameter("#invoicenumber", invoice.InvoiceNumber))
While drInvoiceItems.Read()
invoice.LineItems.Add(New InvoiceLine(drInvoiceItems("id"), drInvoiceItems("inv_id"), drInvoiceItems("prodid"), drInvoiceItems("name"), drInvoiceItems("barcode"), drInvoiceItems("quantity"), drInvoiceItems("costprice")))
End While
Next
Return invoices
I am aware that I am firing 1k connections to the DB due to the iterations. Can't I load all the line items with one select statement and then do something like
For Each invoice As Invoice In invoices
invoice.Items.Add(invoiceItems.Find(Function(i as InvoiceItem),i.InvoiceNumber = invoice.InvoiceNumber))
Next
I get the error whenusing the lambda funcion above
Error 1 Value of type 'System.Collections.Generic.List(Of BizComm.InvoiceLine)' cannot be converted to 'BizComm.InvoiceLine'. C:\Projects\BizComm\InvoiceDAL.vb 75 35 BizComm
One thing I have done when iterating through items in the past is use the same Connection object for all the necessary read activities. It seems to greatly enhance performance.
I'd also look at the database to see whether the dbo.getinvoiceitem procedure can be improved, or if another procedure can be written which will give you all the line items for a group of invoices (perhaps by date or customer/vendor) rather than just one header at a time. Then you can more effectively apply your iteration over the invoice collection and add the lines to the headers.
You can also check to see whether there is an effective index on column that the #invoicenumber parameter references.
From your code, it looks like you are not closing the connections and datareaders. See if you can place your connections and datareaders in a USING statement:
Using con As New SqlConnection(connectionString)
....
End Using
The DBSqlHelperFactory opens a connection, but can't close it since the connection is needed after its return. I'd modify the code, so that you open one connection and pass it to DBSqlHelperFactory as a parameter.
To quickly pick up these issues, I always debug with:
Max Pool Size=1;
added to the end of the connection string. That will quickly throw an error any time you forget to close a connection.
Why load InvoiceItems before hand? Can't you load it on demand?
i.e. when you need to get the Items, call a method on Invoice instance (myInvoice.GetItems)
EDIT: It will be better to understand the full picture of what you are trying to do.
Is it really required to get all Invoices as well?
Why not select all the line items for all the invoices you need in a single query. then split the results up into multiple invoice objects?
Re: how do I map between collections?
One implementation could be: create 1000 anemic Invoice object, chuck them in a Dictionary which goes from Id to Invoice. Then when you select the line items you include the invoice id, look up the anemic invoice and add the line to it.