Trying to fit an OLS model but not successful.
2 responses were gotten as error:
missingdataerror: exog contains inf or nan
olsmodel1 not defined
Below is my code:
olsmodel1 =sm.OLS(y_train, x_train).fit()
print(olsmodel1.summary())
I have checked for missing values using :
df1.isna().sum()
Here is the output:
brand_name 0
os 0
screen_size 0
4g 0
5g 0
main_camera_mp 0
selfie_camera_mp 0
int_memory 0
ram 0
battery 0
weight 0
release_year 0
days_used 0
normalized_used_price 0
normalized_new_price 0
I will appreciate of someone can help me out.
Thank you
I was actually expecting the OLS result to be generated
The PDF content below renders with the correct vertical positions, but how?
1 0 0 -1 0 792 cm
q
.75 0 0 .75 72 192.75 cm
BT
/F4 14.666667 Tf
1 0 0 -1 0 .80265617 Tm
0 -13.2773438 Td <0030> Tj
12.2087708 0 Td <0024> Tj
8.6870575 0 Td <003C> Tj
9.7756042 0 Td <0032> Tj
11.4001007 0 Td <0035> Tj
ET
Q
q
.75 0 0 .75 72 222.75 cm
BT
/F4 14.666667 Tf
1 0 0 -1 4.0719757 .80265617 Tm
0 -13.2773438 Td <002C> Tj
4.0719757 0 Td <0003> Tj
4.0719757 0 Td <0057> Tj
4.0719757 0 Td <004B> Tj
8.1511078 0 Td <004C> Tj
3.2561493 0 Td <0051> Tj
8.1511078 0 Td <004E> Tj
ET
Q
Renders correctly:
MAJOR
I think
However I can't understand how the y positions are calculated to do this (x is fine). The Text Rendering Matrix (TRM) is given by Text Matrix (TM) multiplied by Current Transformation Matrix (CTM) PDF1.7 Reference section 9.4.4. CTM is the identity matrix multiplied by each "cm" operation.
So for the first snippet,
CTM = [1 0 0 -1 0 792] x [0.75 0 0 0.75 72 192.75] = [0.75 0 0 -0.75 72 786.75]
TRM is TM x CTM:
TRM = [1 0 0 -1 0 0.8026] x [0.75 0 0 -0.75 72 786.75] = [0.75 0 0 0.75 72 786.1]
So, ignoring small details, the text will be rendered around y = 786 (actually 776 I reckon)
For the second snippet,
CTM = [1 0 0 -1 0 792] x [0.75 0 0 0.75 72 222.75] = [0.75 0 0 -0.75 72 816.75]
TRM = [1 0 0 -1 4.072 0.802] x [0.75 0 0 -0.75 72 816.75] = [0.75 0 0 0.75 75.05 816.4]
Again, ignoring small details, the text will be rendered around y = 816 (actually 806 I reckon)
But the y origin is the bottom of the page, and 816 is greater than 786. So how come the second snippet of text renders correctly below the first? I'm clearly missing something in the calculations, but I can't see what. Any ideas?
The error in your calculations is that you apply the cm matrix by multiplication from the right side. You instead have to apply it from the left side.
I.e. for the first snippet you have
CTM = [0.75 0 0 0.75 72 192.75] Γ [1 0 0 -1 0 792] = [0.75 0 0 -0.75 72 599.25]
and for the second snippet
CTM = [0.75 0 0 0.75 72 222.75] Γ [1 0 0 -1 0 792] = [0.75 0 0 -0.75 72 569.25]
With these current transformation matrices the rendered result is to be expected.
If you wonder how you should have known that you need to multiply from the left side...
This result is true in general for PDF: when a sequence of transformations is carried out, the matrix representing the combined transformation (Mβ²) is calculated by premultiplying the matrix representing the additional transformation (MT) with the one representing all previously existing transformations (M):
πβ² = ππ Γ π
(ISO 32000-2 section 8.3.4 "Transformation matrices")
Without going deep into matrices (not my forte, there is a slight error in my initial maths so images new corrected) you are working downwards from top left based on an inverted start point of 0 792 cm (Top Left corner)
The start of that snippet is above MAJOR 72 192.75 cm
Without outher transformations the text would be "UpsideDown" with M facing towards the bottom then the second 1 0 0 -1 mirrors it back upright and 0.8 "raises" it towards bottom so baseline is 193.5 ish from topleft at which point you "add" 0 -13.2773438 Td so the baseline is now about 205 from top left
Likewise, the origin for the second row is 72 222.75 cm down from above datum.
In both cases you placed their mirrored baseline even lower at 0 -13.2773438 Td thus both lines will be lower than shown above. In part due to the matrix inversions.
so here the second baseline is now at about 72 234 cm down from top left as subject to similar maths is roughly 222.75+.802+13.277 down but scale can also have effect.
Generally its best to use real time viewer of alterations (however this is not the best way just an example that by playing with rounded values I can see the effects).
I am trying to come up with a way to check what are the most influential factors of a person not paying back a loan (defaulting). I have worked with the sklearn library quite intensively, but I feel like I am missing something quite trivial...
The dataframe looks like this:
0 7590-VHVEG Female Widowed Electronic check Outstanding loan 52000 20550 108 0.099 288.205374 31126.180361 0 No Employed No Dutch No 0
1 5575-GNVDE Male Married Bank transfer Other 42000 22370 48 0.083 549.272708 26365.089987 0 Yes Employed No Dutch No 0
2 3668-QPYBK Male Registered partnership Bank transfer Study 44000 24320 25 0.087 1067.134272 26678.356802 0 No Self-Employed No Dutch No 0
The distribution of the "DefaultInd" column (target variable) is this:
0 0.835408
1 0.164592
Name: DefaultInd, dtype: float64
I have label encoded the data to make it look like this, :
CustomerID Gender MaritalStatus PaymentMethod SpendingTarget EstimatedIncome CreditAmount TermLoanMonths YearlyInterestRate MonthlyCharges TotalAmountPayments CurrentLoans SustainabilityIndicator EmploymentStatus ExistingCustomer Nationality BKR_Registration DefaultInd
0 7590-VHVEG 0 4 2 2 52000 20550 108 0.099 288.205374 31126.180361 0 0 0 0 5 0 0
1 5575-GNVDE 1 1 0 1 42000 22370 48 0.083 549.272708 26365.089987 0 1 0 0 5 0 0
2 3668-QPYBK 1 2 0 4 44000 24320 25 0.087 1067.134272 26678.356802 0 0 2 0 5 0
After that I have removed NaNs and cleaned it up some more (removing capitalizion, punctuation etc)
After that, I try to run this cell:
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
y = df['DefaultInd']
X = df.drop(['CustomerID','DefaultInd'],axis=1)
X = X.astype(float)
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.20,random_state=42)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print(classification_report(y_test, y_pred))
Which results in this:
precision recall f1-score support
0 0.83 1.00 0.91 1073
1 0.00 0.00 0.00 213
accuracy 0.83 1286
macro avg 0.42 0.50 0.45 1286
weighted avg 0.70 0.83 0.76 1286
As you can see, the "1" class does not get predicted 1 time, I am wondering whether or not this behaviour is to be expected (I think it is not). I tried to use class_weightd = βbalancedβ, but that resulted in an average f1 score of 0.59 (instead of 0.76)
I feel like I am missing something, or is this kind of behaviour expected and should I rebalance the dataset before fitting? I feel like the division is not that skewed (Β±80/20), there should not be this big of a problem.
Any help would be more than appreciated :)
I have a column in a dataset called 'Crop', that represents the crops grown in a field over a period of time. The column might have a single string, like Cotton, or it may have multiple strings, like Cotton, Soy. And, depending on the dataset, there may be crops that are categories, but not represented in the particular dataset I'm training with at the time.
I've tried this:
possible_categories = list(['Corn', 'Sorghum', 'Hemp', 'Cotton', 'Soy'])
#df = (X.Crop).str.split(', ', expand=True)
#ohe_crop = pd.get_dummies(df, columns=possible_categories, sparse=True)
#print(ohe_crop)
X.Crop = (X.Crop).astype(pd.CategoricalDtype(categories=possible_categories))
ohe_crop = pd.get_dummies(X.Crop, columns=possible_categories, sparse=True)
which yields this:
Corn Sorghum Hemp Cotton Soy
0 0 0 0 0 0
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
... ... ... ... ...
35512 0 0 0 0 1
35513 0 0 0 0 1
35514 0 0 0 0 1
35515 0 0 0 0 1
35516 0 0 0 0 1
[35517 rows x 5 columns]
In reality, the 1st row of Crop was Cotton, Soy, Sorghum so I expected:
Corn Sorghum Hemp Cotton Soy
0 0 1 0 1 1
I think what happened here is that get_dummies() creates dummy columns for all possible permutations of the crop data:
Corn, Cotton, Soy
Corn, Soy Cotton ...
Hemp, Cotton,
Soy Hemp,
Soy Soy
so unless the field had crops that fit into these patterns, the row gets a 0.
I'd like to specify the possible categories, split Crop into multiple columns delimited by the commas that are in the rows, and then be able to populate multiple columns if there were multiple crops grown, but I can't figure out how to make it happen. Any advice?
I gone through different solutions on CTM matrix calculations(someof them are this and this).
What I know about content stream is when "q " encounters we need to push identity matrix in a graphics_stack and keep multiply with next position operator(cm , Tm, Td, TD) CTM. When "Q" encounters we need to pop the last matrix.
For text positioning parsing when "BT" encounters push in identity matrix in position_stack and keep multiply with next position operator(cm , Tm, Td, TD) CTM. When "ET" encounters we need to pop the last matrix.
Here the some times we need to multiply with last CTM matrix and some times just multiply with identity matrix. When these cases are occurs?
Case 1:
From as shown in image 1 and 2 case merely replacement of new matrix from Td to Tm. And from 2 to 3 it's again last CTM multiplication. How I know?(By visually I can tell by looks)
Case 2:
In this case which how the matrix pushing and calculation will be there ?
Case 3:
BT
TT_1 20 Tf
35.56 150.24533 Td _______________ 1
(some sample text) Tj
50.526 250.36 Td ________________ 2
(second line new replace) Tj
0 -16.2 Td _____________________ 3
(Line end.) Tj
ET
This case 1 and 2 are merely replacement, 2 and 3 previous matrix multiplication. How do I know?
Case 4:
Please parse these positions at least till 10. The source file of this pdf
Case 5:
In above one need to calculate the l position. I highlighted with 1 to 4 numbers. I need to calculate the positions of each and every l How can do that? pdf
case6:
What is the change in calculation when the page(pdf) is rotated to 90 or 180 or 270 and 315 degrees?
These are some cases what I saw. What else might cases I can encounter and What is the generic approach to solve this ?
Operators of interest
First of all, I get the impression you mix up two different aspects. You have the current transformation matrix (CTM) and you have the text and text line matrices. The CTM is subject to cm, q, and Q. The text and text line matrices are subject to BT, Tm, Td, ... And to determine the exact position and direction of drawn text you need the product of text matrix and CTM at the time that text is drawn.
How those operators change the matrices, is described in the PDF specification, ISO 32000 part 1 or 2.
From ISO 32000-1 Table 57 β Graphics State Operators β
cm: Modify the current transformation matrix (CTM) by concatenating the specified matrix
q: Save the current graphics state including the CTM on the graphics state stack
Q: Restore the graphics state including the CTM by removing the most recently saved state from the stack and making it the current state
From ISO 32000-1 Table 107 β Text object operators β
BT: Begin a text object, initializing the text matrix, Tm, and the text line matrix, Tlm, to the identity matrix.
From ISO 32000-1 Table 108 β Text-positioning operators β
tx ty Td: Move to the start of the next line, offset from the start of the current line by (tx, ty). More precisely, this operator shall perform these assignments:
a b c d e f Tm: Set the text matrix, Tm, and the text line matrix, Tlm:
Furthermore, TD, T*, ', and " operate on Tm and Tlm in a way specified using the Td operator.
From ISO 32000-1 section 9.4.4 β Text Space Details β
Whenever a glyph is drawn, its entire transformation from text space may be represented by a text rendering matrix, Trm:
where Tfs is the current font size, Th is the current horizontal scaling factor, and Trise is the current text rise value.
After drawing that glyph, Tm is updated according to the glyph displacement
In horizontal mode tx is the displacement and ty is zero, in vertical mode tx is zero and ty is the displacement. The applicable value is calculated as
Example 1
In the following paragraphs I use rounded values to concentrate on the essentials.
The CTM starts as the Identity matrix, and as there is no cm operation here, it remains so all the time. Tm and Tlm on the other hand do change:
BT
Tm and Tlm are both set to the identity matrix
/GS0 gs
/T1_0 10 Tf
No change to Tm or Tlm.
317 65 Td
This multiplies a translation matrix as described above from the left to the former value of the Tlm and sets Tm and Tlm to the result:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 317 65 1 0 0 1 317 65 1
As that former value was the identity, this may look like a replacement by the Td translation matrix but it actually is a multiplication.
(F)Tj
This draws a glyph transformed by the text rendering matrix
10 Γ 1 0 0 1 0 0 1 0 0 10 0 0
T = 0 10 0 * 0 1 0 * 0 1 0 = 0 10 0
rm 0 0 1 317 65 1 0 0 1 317 65 1
Thereafter Tm is updated as described above. Unfortunately we don't have the widths of the font T1_0, so we cannot calculate the updated value.
1 0 0 1 370 87 Tm
This sets the text matrix, Tm, and the text line matrix, Tlm, to the given matrix:
1 0 0
T = T = 0 1 0
m lm 370 87 1
So now we know the current Tm value again.
-47 -22 Td
This multiplies a translation matrix as described above from the left to the former value of the Tlm and sets Tm and Tlm to the result:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm -47 -22 1 370 87 1 323 65 1
[(igure)-251(2.3:)-621(P)16...] TJ
This draws the strings in the argument array with a starting text rendering matrix of
10 Γ 1 0 0 1 0 0 1 0 0 10 0 0
T = 0 10 0 * 0 1 0 * 0 1 0 = 0 10 0
rm 0 0 1 323 65 1 0 0 1 323 65 1
updating Tm again and again as described above.
Example 2
In the following paragraphs I use rounded values to concentrate on the essentials.
The CTM starts as the Identity matrix.
q
This saves the current graphics state, including the current CTM. As there is no Q operation in the example, though, we can ignore that for now.
.24 0 0 .24 91 740 cm
This updates the CTM by the given matrix:
0.24 0 0 1 0 0 0.24 0 0
CTM = 0 0.24 0 * 0 1 0 = 0 0.24 0
91 740 1 0 0 1 91 740 1
BT
Tm and Tlm are both set to the identity matrix
133 0 0 133 0 0 Tm
This sets the text matrix, Tm, and the text line matrix, Tlm, to the given matrix:
133 0 0
T = T = 0 133 0
m lm 0 0 1
/TT1.0 1 Tf
.002 Tc
No change to CTM, Tm, or Tlm.
[(The)1( )1(Long )1(Tai)1(l)]TJ
This draws the strings in the argument array with a starting text rendering matrix of
1 Γ 1 0 0 133 0 0 0.24 0 0 32 0 0
T = 0 1 0 * 0 133 0 * 0 0.24 0 = 0 32 0
rm 0 0 1 0 0 1 91 740 1 91 740 1
updating Tm again and again as described above.
Example 3
BT
TT_1 20 Tf
35.56 150.24533 Td _______________ 1
(some sample text) Tj
50.526 250.36 Td ________________ 2
(second line new replace) Tj
0 -16.2 Td _____________________ 3
(Line end.) Tj
ET
In the following paragraphs I use rounded values to concentrate on the essentials.
The CTM starts as the Identity matrix, and as there is no cm operation here, it remains so all the time. Tm and Tlm on the other hand do change:
BT
Tm and Tlm are both set to the identity matrix
TT_1 20 Tf
No change to Tm or Tlm.
36 150 Td
This multiplies a translation matrix as described above from the left to the former value of the Tlm and sets Tm and Tlm to the result:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 36 150 1 0 0 1 36 150 1
As that former value was the identity, this may look like a replacement by the Td translation matrix but it actually is a multiplication.
(some sample text) Tj
This draws glyphs transformed by the text rendering matrix
20 Γ 1 0 0 1 0 0 1 0 0 20 0 0
T = 0 20 0 * 0 1 0 * 0 1 0 = 0 20 0
rm 0 0 1 36 150 1 0 0 1 36 150 1
Thereafter Tm is updated as described above. Unfortunately we don't have the widths of the font TT_1, so we cannot calculate the updated value.
51 250 Td
This multiplies a translation matrix as described above from the left to the former value of the Tlm and sets Tm and Tlm to the result:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 51 250 1 36 150 1 87 400 1
So now we know the current Tm value again.
(second line new replace) Tj
This draws glyphs transformed by the text rendering matrix
20 Γ 1 0 0 1 0 0 1 0 0 20 0 0
T = 0 20 0 * 0 1 0 * 0 1 0 = 0 20 0
rm 0 0 1 87 400 1 0 0 1 87 400 1
Thereafter Tm is updated as described above. Unfortunately we don't have the widths of the font TT_1, so we cannot calculate the updated value.
0 -16 Td
This multiplies a translation matrix as described above from the left to the former value of the Tlm and sets Tm and Tlm to the result:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 0 -16 1 87 400 1 87 384 1
So now we know the current Tm value again.
(Line end.) Tj
This draws glyphs transformed by the text rendering matrix
20 Γ 1 0 0 1 0 0 1 0 0 20 0 0
T = 0 20 0 * 0 1 0 * 0 1 0 = 0 20 0
rm 0 0 1 87 384 1 0 0 1 87 384 1
Thereafter Tm is updated as described above. Unfortunately we don't have the widths of the font TT_1, so we cannot calculate the updated value.
Example 4
We discussed example 4 in your answer and the comments to it.
Example 5
q 0.1 0 0 0.1 0 0 cm
/R108 gs
0 g
q
...
Q
0 0 1 rg
q
...
Q
4.05 w
0 G
722.023 4082.13 m
722.023 4490.28 l
S
723.961 4488.25 m
2872.98 4488.25 l
S
404.1 w
0 0 0.199951 0 K
723.961 4284.18 m
2872.98 4284.18 l
S
4.05 w
0 G
720 4080.2 m
2876.94 4080.2 l
S
2874.91 4082.13 m
2874.91 4490.28 l
S
0 g
q
Why does your view of those instructions show the numbers inaccurately? The above is copy&pasted from the stream contents, there is no need to change the numbers like that...
q
0.1 0 0 0.1 0 0 cm
Sets the CTM to
0.1 0 0
CTM = 0 0.1 0
0 0 1
/R108 gs
...
0 G
Nothing happens to the CTM
722.023 4082.13 m
722.023 4490.28 l
S
We have to apply the CTM to these coordinates
0.1 0 0
[722.023 4082.13] * 0 0.1 0 = [72.2023 408.213]
0 0 1
0.1 0 0
[722.023 4490.28] * 0 0.1 0 = [72.2023 449.028]
0 0 1
Thus, a line is stroked from 72.2023,408.213 to 72.2023,449.028.
723.961 4488.25 m
2872.98 4488.25 l
S
Just like above, a line is drawn from 72.3961,448.825 to 287.298,448.825.
404.1 w
0 0 0.199951 0 K
723.961 4284.18 m
2872.98 4284.18 l
S
And again, a line is drawn from 72.3961,428.418 to 287.298,428.418. The only notable thing here is that the line is quite wide, ca. 40 units, so this "line" actually looks more like a filled rectangle and represents the background of the text box with the Lorentz force characterization.
4.05 w
0 G
720 4080.2 m
2876.94 4080.2 l
S
Another line is drawn, narrow again and, therefore, looking like a line, this time from 72,408.02 to 287.694 408.02.
2874.91 4082.13 m
2874.91 4490.28 l
S
And finally the last line, this time from 287.491,408.213 to 287.491,449.028.
Example 4 solution:
The CTM starts as the Identity matrix.
q
This saves the current graphics state (Graphics state = Identity ---- 1).
1 0 0 1 62.692 277.671 cm
This updates the CTM by the given matrix:
1 0 0 1 0 0 1 0 0
CTM = 0 0 0 * 0 1 0 = 0 1 0 ----- 2
62.692 277.67 1 0 0 1 62.692 277.67 1
CTM is updated to above result
q
push the CTM save to graphics state(Graphics state = 1, 2)
q
Again push the same CTM matrix to graphics state(Graphics state = 1, 2, 2)
1 0 0 1 286.59 207.54 cm
Update the CTM by current matrix.
1 0 0 1 0 0 1 0 0
CTM = 0 1 0 * 0 0 0 = 0 1 0 ----- 3
286.49 207.54 1 62.692 277.67 1 349.18 485.21 1
CTM updated to above result matrix.
q
push the CTM save to graphics state(Graphics state = 1, 2,2,3)
.75 .75 .75 RG
n
11.33 19.84 171.67 232.146 re
S
This will nothing change in position matrices.
Q
Assign the CTM to last available graphics state CTM = 3. Q will remove the last graphics state from graphics stack.(Graphics state = 1, 2,2)
1 0 0 1 17.007 23.52 cm
Update the CTM by current matrix.
1 0 0 0 1 0 0 1 0
CTM = 0 1 0 * 0 0 0 = 0 1 0 ----- 4
17.007 23.52 1 349.18 485.21 1 366.18 508.73 1
CTM updated to above result matrix.
q
push the last CTM to graphics state(Graphics state = 1, 2,2,4)
Skip rg, RG
BT
assign Tm and Tlm to Identity matrix.
1 0 0 1 0 5.6 Tm
This sets the text matrix, Tm, and the text line matrix, Tlm, to the given matrix:
1 0 0
T = T = 0 1 0
m lm 0 5.6 1
Tm and Tlm updated.
46.22 0 Td
Translate the matrix according to above Td matrix.
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 46.22 0 1 0 5.6 1 46.22 5.6 1
update the Tm and Tlm with above matrix. Next Tf you can leave.
12 TL
This sets the graphics state leading text parameter, no direct influence on Tm or Tlm.
(William Shakespeare) Tj
This draws the string glyph-by-glyph, the first glyph transformed by this text rendering matrix
8 Γ 1 0 0 1 0 0 1 0 0 8 0 0
T = 0 8 0 * 0 1 0 * 0 1 0 = 0 8 0
rm 0 0 1 46.22 5.6 1 366.18 508.73 1 412.4 514.33 1
While the string is rendered glyph-by-glyph, Tm is updated as described in ISO 32000-1 section 9.4.4, and so is the Trm.
T*
As the text leading currently is 12, T* is equivalent to 0 -12 Td, so:
1 0 0 1 0 0 1 0 0
T = T = 0 1 0 * 0 1 0 = 0 1 0
m lm 0 -12 1 46.22 5.6 1 46.22 -6.4 1