Python Pandas: How can I group different days of a month that are part of my dataset into one month? - pandas

I am currently working on my master thesis and I have a problem with regards to data organization in pandas. I downloaded multiple economic indicators that are published once a month and consolidated them in one dataframe.
However, the economic indicators are released on different days each month. Therefore my dataframe has for example five different rows for January 2020 (e.g. January 1st, January 5th, January 13th, January 28th, January 31st) and many „NaN“ values in each row.
I want to organize my data so that I have one row for each month, so for example one row for January 2020. However, I cannot figure out how to solve this problem in pandas.
Another challenge represents the fact that sometimes the data is released on March 1st and on March 31st. Therefore consolidating everything in one month could also lead to new problems if the values are summed up.
The table below visualizes my problem. My index column in the dataframe are the dates.
| Dates | Indicator 1 | Indicator 2
| ———————— | ——————————— | ———————————
| 01.01.20 | 1 | 1
| 08.01.20 | 2 | NaN
| 02.02.20 | 5 | 5
| 01.03.20 | 8 | 6
| 31.03.20 | 7 | 7
I already tried pd.to_period or pd.groupby, but I could not solve the problem.

Related

Subtract two aggregated values in Bar Chart

My data is like -
+-----------+------------------+-----------------+-------------+
| Issue Num | Created On | Closed at | Issue Owner |
+-----------+------------------+-----------------+-------------+
| 1 | 12/21/2016 15:26 | 1/13/2017 9:48 | Name 1 |
| 2 | 1/10/2017 7:38 | 1/13/2017 9:08 | Name 2 |
| 3 | 1/13/2017 8:57 | 1/13/2017 8:58 | Name 2 |
| 4 | 12/20/2016 20:30 | 1/13/2017 5:46 | Name 2 |
| 5 | 12/21/2016 19:30 | 1/13/2017 1:14 | Name 1 |
| 6 | 12/20/2016 20:30 | 1/12/2017 9:11 | Name 1 |
| 7 | 1/9/2017 17:44 | 1/12/2017 1:52 | Name 1 |
| 8 | 12/21/2016 19:36 | 1/11/2017 16:59 | Name 1 |
| 9 | 12/20/2016 19:54 | 1/11/2017 15:45 | Name 1 |
+-----------+------------------+-----------------+-------------+
What I am trying to achieve is
Number of issues created per week
Number of issues closed per week
Net number of issues remaining per week
I am able to resolve the top two points but unable to approach the last.
My attempt -
This gives me number of issues created every week.
Similarly I have done for Closed per week.
For Net number of issues (Created-Closed) -
I tried adding Closed At column along with Created On but I can't see second bar in the chart along with Created On either.
Something like this
I tried doing the same in excel -
I want something of this sort but with another column as the difference of
number of issues created that week - number of issues closed that week.
In this case, 8-6=2.
You could use a calculated field(Analysis->Create Calculated Field). Something like this:
{FIXED [Create Date]:Count(if DATEPART('year',[Create Date]) = 2016 then [Number of Records] end)} - {FIXED [Closed Date]:Count(if DATEPART('year',[Closed Date]) = 2016 then [Number of Records] end)}
This function is using LOD expressions to pull back both sets of values. It will filter on all 2016 results for both date sets and then minus them from each other.
For more on LOD's see here:
https://www.tableau.com/about/blog/LOD-expressions
Use this as your measure and pull in one of your date fields as the dimension.
The normal way to solve this problem is to reshape the data so you have one row per status change instead of one row per issue, with a column named [Date] and a column named [Action]. The action can be submit and close (or in a more complex world include approve, reject, whatever - tracking the history.
You can do the reshaping without modifying your source data by using a UNION to get two copies of each row with appropriate calculated fields to make the visible columns make sense (e.g., create calculated a field called Date that returns the submission date or closing date depending on whether the row is from the first or second union, with a similar one called Action whose value depends on that as well. Filter out Close actions that have a null date)
Or you can preprocess the data to reshape it.
Or you can use data blending to make two sources that point to the same data source but customizing the linking fields to line up the submit and close dates (e.g., duplicate the data connection and rename both date fields to have the same name). But in this case, you probably want to create scaffolding source that has every date, but no other data, to use as the primary data source to avoid filtering out data from the secondary for dates that don't appear in the primary. The blending approach can be brittle.
Assuming you used the UNION approach instead of Data Blending, then you can count the number of submissions and closures within a certain date range, or compute a running total of the difference to see the backlog size over time.

Access Query: get difference of dates with a twist

I'm going to do my best to explain this so I apologize in advance if my explanation is a little awkward. If I am foggy somewhere, please tell me what would help you out.
I have a table filled with circuits and dates. Each circuit gets trimmed on a time cycle of about 36 months or 48 months. I have a column that gives me this info. I have one record for every time the a circuit's trim cycle has been completed. I am attempting to link a known circuit outage list, to a table with their outage data, to a table with the circuit's trim history. The twist is the following:
I only want to get back circuits that have exceeded their trim cycles by 6 months. So I would need to take all records for a circuit, look at each individual record, find the most recent previous record relative to the record currently being examined (I will need every record examined invididually), calculate the difference between the two records in months, then return only the records that exceeded 6 months of difference between any two entries for a given feeder.
Here is an example of the data:
+----+--------+----------+-------+
| ID | feeder | comp | cycle |
| 1 | 123456 | 1/1/2001 | 36 |
| 2 | 123456 | 1/1/2004 | 36 |
| 3 | 123456 | 7/1/2007 | 36 |
| 4 | 123456 | 3/1/2011 | 36 |
| 5 | 123456 | 1/1/2014 | 36 |
+----+--------+----------+-------+
Here is an example of the result set I would want (please note: cycle can vary by circuit, so the value in the cycle column needs to be in the calculation to determine if I exceeded the cycle by 6 months between trimmings):
+----+--------+----------+-------+
| ID | feeder | comp | cycle |
| 3 | 123456 | 7/1/2007 | 36 |
| 4 | 123456 | 3/1/2011 | 36 |
+----+--------+----------+-------+
This is the query I started but I'm failing really hard at determining how to make the date calculations correctly:
SELECT temp_feederList.Feeder, Temp_outagesInfo.causeType, Temp_outagesInfo.StormNameThunder, Temp_outagesInfo.deviceGroup, Temp_outagesInfo.beginTime, tbl_Trim_History.COMP, tbl_Trim_History.CYCLE
FROM (temp_feederList
LEFT JOIN Temp_outagesInfo ON temp_feederList.Feeder = Temp_outagesInfo.Feeder)
LEFT JOIN tbl_Trim_History ON Temp_outagesInfo.Feeder = tbl_Trim_History.CIRCUIT_ID;
I wasn't really able to figure out where I need to go from here to get that most recent entry and perform the mathematical comparison. I've never been asked to do SQL this complex before, so I want to thank all of you for your patience and any assistance you're willing to lend.
I'm making some assumptions, but this uses a subquery to give you rows in the feeder list where the previous completed date was greater than the number of months ago indicated by the cycle:
SELECT tbl_Trim_History.ID, tbl_Trim_History.feeder,
tbl_Trim_History.comp, tbl_Trim_History.cycle
FROM tbl_Trim_History
WHERE tbl_Trim_History.comp>
(SELECT Max(DateAdd("m", tbl_Trim_History.cycle, comp))
FROM tbl_Trim_History T2
WHERE T2.feeder = tbl_Trim_History.feeder AND
T2.comp < tbl_Trim_History.comp)
If you needed to check for longer than 36 months you could add an arbitrary value to the months calculated by the DateAdd function.
Also I don't know if the value of cycle specified the number of month from the prior cycle or the number of months to the next one. If the latter I would change tbl_Trim_History.cycle in the DateAdd function to just cycle.
SELECT tbl_trim_history.ID, tbl_trim_history.Feeder,
tbl_trim_history.Comp, tbl_trim_history.Cycle,
(select max(comp) from tbl_trim_history T
where T.feeder=tbl_trim_history.feeder and
t.comp<tbl_trim_history.comp) AS PriorComp,
IIf(DateDiff("m",[priorcomp],[comp])>36,"x") AS [Select]
FROM tbl_trim_history;
This query identifies (with an X in the last column) the records from tbl_trim_history that exceed the cycle time - but as noted in the comments I'm not entirely sure if this is what you need or not, or how to incorporate the other 2 tables. Once you see what it is doing you can modify it to only keep the records you need.

Date dimension and cumulative Months

The Date dimension is in the following state:
In SSAS looks like this:
Based on this how can I build an Hierarchy in SSAS with the following structrue:
2016
Jan-Feb
Jan-Mrz
Jan-Apr
Jan-Mai
Jan-Jun
...
where Jan-Apr will be the cumulative Amount from January until April in 2016.
Downvotes are probably because this is a bad dimension design. You can get a cumulative aggregation by dropping all this Jan-Feb etc stuff, using a simple Date dimension and using the PeriodsToDate function.
Your hierarchy should go as Year -> Month -> day and so on. To support cummilative figures combine your hierarchy with a range dimension that includes month ranges like Jan - Apr for example. Hierarchy is not used as ranges directly. you have to combine both together to get the requirement.
Range dimension will have minimum value which will be your starting month and max value which will be your ending month of one particular range. Example dimension as follows
+------------+---------------------+
| DIM_Range |
+------------+---------------------+
|Sur_Key | Min Value | Max Value |
+------------+---------------------+
| 01 | Jan | Apr |
| 02 | May | August |
+------------+---------------------+

DAX SUMMARIZE() with filter - Powerpivot

Rephrasing a previous question after further research. I have a denormalised hierarchy of cases, each with an ID, a reference to their parent (or themselves) and a closure date.
Cases
ID | Client | ParentMatterName | MatterName | ClaimAmount | OpenDate | CloseDate
1 | Mr. Smith | ABC Ltd | ABC Ltd | $40,000 | 1 Jan 15 | 4 Aug 15
2 | Mr. Smith | ABC Ltd | John | $0 |20 Jan 15 | 7 Oct 15
3 | Mr. Smith | ABC Ltd | Jenny | $0 | 1 Jan 15 | 20 Jan 15
4 | Mrs Bow | JQ Public | JQ Public | $7,000 | 1 Jan 15 | 4 Aug 15
After the help of greggyb I also have another column, Cases[LastClosed], which will be true if the current row is closed, and is the last closed of the parent group.
There is also a second table of payments, related to Cases[ID]. These payments could be received in parent or child matters. I sum payments received as follows:
Recovery All Time:=CALCULATE([Recovery This Period], ALL(Date_Table[dateDate]))
I am looking for a new measure which will calculate the total recovered for a unique ParentMatterName, if the last closed matter in this group was closed in the Financial Year we are looking at - 30 June end date.
I am now looking at the SUMMARIZE() function to do the first part of this, but I don't know how to filter it. The layers of calculate are confusing. I've looked at This MSDN blog but it appears that this will filter to only show the total payments for that matter that was last closed (not adding the related children).
My current formula is:
Recovery on Closed This FY :=
CALCULATE (
SUMX (
SUMMARIZE (
MatterListView,
MatterListView[UniqueParentName],
"RecoveryAllTime", [Recovery All Time]
),
[RecoveryAllTime]
)
)
All help appreciated.
Again, your solution is much more easily solved with a model addition. Remember, storage is cheap, your end users are impatient.
Just store in your Cases table a column with the LastClosedDate of every parent matter, which indicates the date associated with the last closed child matter. Then it's a simple filter to return only those payments/matters that have LastClosedDate in the current fiscal year. Alternately, if you know for certain that you are only concerned with the year, you could store just LastClosedFiscalYear, to make your filter predicate a bit simpler.
If you need help with specific measures or how you might implement the additional field, let us know (I'd recommend adding these fields at the source, or deriving them in the source query rather than using calculated columns).

How to calculate the broadcast year and month out of the given date?

Is there a way to calculate the the broadcast year and month for a given gregorian date?
The advertising broadcast calendar differs from the regular calendar, in the way that every month needs to start on a Monday and end on a Sunday and have exactly 4 or 5 weeks. You can read about it here: http://en.wikipedia.org/wiki/Broadcast_calendar
This is a pretty common thing in TV advertising, so I guess there is a standard mathematical formula for it, that uses a combination of date functions (week(), month(), etc...).
Here is an example mapping between gregorian and broadcast dates:
| gregorian_date | broadcast_month | broadcast_year |
+----------------+-----------------+----------------+
| 2014-12-27 | 12 | 2014 |
| 2014-12-28 | 12 | 2014 |
| 2014-12-29 | 1 | 2015 |
| 2014-12-30 | 1 | 2015 |
| 2014-12-31 | 1 | 2015 |
| 2015-01-01 | 1 | 2015 |
| 2015-01-02 | 1 | 2015 |
Here is example how the broadcast calendar looks for 2015:
http://www.rab.com/public/reports/BroadcastCalendar_2015.pdf
As far as I can see, the pattern is that the first of the Gregorian month always falls within the first week of the Broadcast calendar, and any days from the previous month are pulled forward into that month to create full weeks. In Excel, you can use the following formula in cell B2 (first of your broadcast months above) to calculate the broadcast month:
=MONTH(A2+(7-WEEKDAY(A2,2)))
Similarly, in cell C2:
=IF(AND(MONTH(A2)=12,B2=1),YEAR(A2)+1,YEAR(A2))
This will return the broadcast month and year for any dates you put into your data set.
Hope that helps!
month,first,last
2018_1,2018-01-01,2018-01-28
2018_2,2018-01-29,2018-02-25
2018_3,2018-02-26,2018-03-25
2018_4,2018-03-26,2018-04-29
2018_5,2018-04-30,2018-05-27
2018_6,2018-05-28,2018-06-24
2018_7,2018-06-25,2018-07-29
2018_8,2018-07-30,2018-08-26
2018_9,2018-08-27,2018-09-30
2018_10,2018-10-01,2018-10-28
2018_11,2018-10-29,2018-11-25
2018_12,2018-11-26,2018-12-30
2019_1,2018-12-31,2019-01-27
2019_2,2019-01-28,2019-02-24
2019_3,2019-02-25,2019-03-31
2019_4,2019-04-01,2019-04-28
2019_5,2019-04-29,2019-05-26
2019_6,2019-05-27,2019-06-30
2019_7,2019-07-01,2019-07-28
2019_8,2019-07-29,2019-08-25
2019_9,2019-08-26,2019-09-29
2019_10,2019-09-30,2019-10-27
2019_11,2019-10-28,2019-11-24
2019_12,2019-11-25,2019-12-29
2020_1,2019-12-30,2020-01-26
2020_2,2020-01-27,2020-02-23
2020_3,2020-02-24,2020-03-29
2020_4,2020-03-30,2020-04-26
2020_5,2020-04-27,2020-05-31
2020_6,2020-06-01,2020-06-28
2020_7,2020-06-29,2020-07-26
2020_8,2020-07-27,2020-08-30
2020_9,2020-08-31,2020-09-27
2020_10,2020-09-28,2020-10-25
2020_11,2020-10-26,2020-11-29
2020_12,2020-11-30,2020-12-27
2021_1,2020-12-28,2021-01-31
2021_2,2021-02-01,2021-02-28
2021_3,2021-03-01,2021-03-28
2021_4,2021-03-29,2021-04-25
2021_5,2021-04-26,2021-05-30
2021_6,2021-05-31,2021-06-27
2021_7,2021-06-28,2021-07-25
2021_8,2021-07-26,2021-08-29
2021_9,2021-08-30,2021-09-26
2021_10,2021-09-27,2021-10-31
2021_11,2021-11-01,2021-11-28
2021_12,2021-11-29,2021-12-26
2022_1,2021-12-27,2022-01-30
2022_2,2022-01-31,2022-02-27
2022_3,2022-02-28,2022-03-27
2022_4,2022-03-28,2022-04-24
2022_5,2022-04-25,2022-05-29
2022_6,2022-05-30,2022-06-26
2022_7,2022-06-27,2022-07-31
2022_8,2022-08-01,2022-08-28
2022_9,2022-08-29,2022-09-25
2022_10,2022-09-26,2022-10-30
2022_11,2022-10-31,2022-11-27
2022_12,2022-11-28,2022-12-25
2023_1,2022-12-26,2023-01-29
2023_2,2023-01-30,2023-02-26
2023_3,2023-02-27,2023-03-26
2023_4,2023-03-27,2023-04-30
2023_5,2023-05-01,2023-05-28
2023_6,2023-05-29,2023-06-25
2023_7,2023-06-26,2023-07-30
2023_8,2023-07-31,2023-08-27
2023_9,2023-08-28,2023-09-24
2023_10,2023-09-25,2023-10-29
2023_11,2023-10-30,2023-11-26
2023_12,2023-11-27,2023-12-31
2024_1,2024-01-01,2024-01-28
2024_2,2024-01-29,2024-02-25
2024_3,2024-02-26,2024-03-31
2024_4,2024-04-01,2024-04-28
2024_5,2024-04-29,2024-05-26
2024_6,2024-05-27,2024-06-30
2024_7,2024-07-01,2024-07-28
2024_8,2024-07-29,2024-08-25
2024_9,2024-08-26,2024-09-29
2024_10,2024-09-30,2024-10-27
2024_11,2024-10-28,2024-11-24
2024_12,2024-11-25,2024-12-29
2025_1,2024-12-30,2025-01-26
2025_2,2025-01-27,2025-02-23
2025_3,2025-02-24,2025-03-30
2025_4,2025-03-31,2025-04-27
2025_5,2025-04-28,2025-05-25
2025_6,2025-05-26,2025-06-29
2025_7,2025-06-30,2025-07-27
2025_8,2025-07-28,2025-08-31
2025_9,2025-09-01,2025-09-28
2025_10,2025-09-29,2025-10-26
2025_11,2025-10-27,2025-11-30
2025_12,2025-12-01,2025-12-28