Related
I have a large dataframe (thousands of rows by hundreds of columns), a short excerpt is as the following:
data = {'Step':['', '', '', 'First', 'First', 'Second', 'Third', 'Second', 'First', 'Second', 'First', 'First', 'Second', 'Second'],
'Stuff':['tot', 'white', 'random', 7583, 3563, 824, 521, 7658, 2045, 33, 9823, 5, 8090, 51],
'Mark':['marking', '', '', 1, 5, 5, 5, 1, 27, 27, 1, 6, 1, 9],
'A':['item_a', 100, 'st1', 142, 2, 2, 2, 100, 150, 105, 118, 118, 162, 156],
'B':['skill', 66, 'abc', 160, 2, 130, 140, 169, 1, 2, 130, 140, 144, 127],
'C':['item', 50, 'st1', 2000, 2, 65, 2001, 1999, 1, 2, 2000, 4, 2205, 2222],
'D':['item_c', 100, 'st1', 433, 430, 150, 170, 130, 1, 2, 300, 4, 291, 606],
'E':['test', 90, 'st1', 111, 130, 5, 10, 160, 1, 2, 232, 4, 144, 113],
'F':['done', 80, 'abc', 765, 755, 5, 10, 160, 1, 2, 733, 4, 666, 500],
'G':['nd', 90, 'mag', 500, 420, 5, 10, 160, 1, 2, 300, 4, 469, 500],
'H':['prt', 100, 'st1', 999, 200, 5, 10, 160, 1, 2, 477, 4, 620, 7],
'Name':['NS', '', '', "Pat", "Lucy", "Lucy", "Lucy", "Nick", "Kirk", "Kirk", "Joe", "Nico", "Nico", "Bryan"],
'Value':[ -1, 0, 0, 0, 3, 6, 5, 0, 7, 7, 0, 6, 0, 1]}
df = pd.DataFrame(data)
I need to sort this dataframe according to the following conditions that have to be satisfied all together:
In the "Name" column, names that are the same are to remain grouped (e.g. there are 3
records of "Lucy" next to each other, and they cannot be moved apart)
For each group of names, the appearance order has to remain the one
given by the "Step" column (e.g. the first appearance of "Lucy" is
related to the value "First" in the "Step" column, the second to
"Second" and so on)
All the remaining names that in the "Value" column have a value = 0,
have to be moved below the others (e.g. "Pat" can be moved after the
others, but not "Nico" because there are two records of "Nico" and
the other one has a value = 6)
The first three rows cannot be moved
What I have done is to concatenate different sub-dataframes:
df_groupnames=df[df.duplicated(subset=['Name'], keep=False)]
df_nogroup = df[~df.duplicated(subset=['Name'], keep=False)]
df_nogroup_high = df_nogroup[df_nogroup["Value"] > 0 ]
df_nogroup_null = df_nogroup[df_nogroup["Value"] == 0]
# Let's concatenate these dataframes to get the sorted one
df_sorted = pd.concat([df_groupnames, df_nogroup_high, df_nogroup_null])
It works, but I wonder if there's a smarter, simpler way, and maybe faster, to obtain the same.
Thank you for your attention.
I have three arrays that I read from FITS files that contain images (I can open them with astroimagej. They don't seem corrupted)
band = fits.open("band.fits")[0].data
In : r
Out:
array([[ 40, 608, 829, ..., 652, 297, 306],
[ 70, 886, 786, ..., 1088, 519, 314],
[ 14, 112, 518, ..., 885, 1454, 0],
...,
[1648, 471, 0, ..., 40, 558, 68],
[1456, 536, 0, ..., 42, 257, 108],
[ 235, 858, 177, ..., 113, 203, 108]], dtype=uint16)
In : g
Out:
array([[ 916, 0, 130, ..., 339, 84, 546],
[ 0, 0, 836, ..., 0, 0, 0],
[ 0, 1726, 1712, ..., 0, 0, 505],
...,
[1025, 0, 129, ..., 1485, 2915, 0],
[ 0, 0, 1129, ..., 990, 1815, 0],
[ 659, 0, 296, ..., 0, 0, 0]], dtype=uint16)
In : b
Out:
array([[ 916, 0, 130, ..., 339, 84, 546],
[ 0, 0, 836, ..., 0, 0, 0],
[ 0, 1726, 1712, ..., 0, 0, 505],
...,
[1025, 0, 129, ..., 1485, 2915, 0],
[ 0, 0, 1129, ..., 990, 1815, 0],
[ 659, 0, 296, ..., 0, 0, 0]], dtype=uint16)
When I try to run the make_lupton_rbg function from the astropy library, I get the error message
UFuncTypeError: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('uint16') with casting rule 'same_kind'
Yet it seems to me like every array here is of the same type. What might be going wrong?
I also tried to run the function with one of these arrays repeated three times e.g. make_lupton_rgb(r, r, r), to see if there was a problem with one individual FITS file. I get the same error message in all cases.
Converting each array to float64 worked, though I don't know why.
I am currently trying to INSERT INTO my SQL database a row of 144 columns.
The problem is that the last 10 values of the new row are NULL while they are supposed to be float and int.
That's an example of what I have in my DB after the INSERT INTO :
First column
Before last column
Last column
1
NULL
NULL
That's the SQL request I am using
INSERT INTO "historic_data2"
VALUES (28438, 163, 156, 1, 'FIST 2', 91, 81, 82, 84, 90, 6, '2 Pts Int M', 'Offensive', 0, '91_81_82_84_90', 86, 85, 0, 36, 62, 24, 0, 132, 86, 0, 83, 0, 0, 0, 0, 42, 77, 24, 0, 173, 107, 0, 204, 0, 0, 0, 0, 42, 77, 24, 0, 173, 107, 0, 204, 0, 0, 0, 81, 62, 34, 23, 19, 45, 32, 18, 9, 19, 0.5555555555555556, 0.5161290322580645, 0.5294117647058824, 0.391304347826087, 1.0, 82, 54, 34, 18, 28, 49, 27, 17, 8, 28, 0.5975609756097561, 0.5, 0.5, 0.4444444444444444, 1.0, 302, 233, 132, 89, 69, 168, 116, 69, 35, 69, 0.5562913907284768, 0.4978540772532189, 0.5227272727272727, 0.39325842696629215, 1.0, 214, 161, 84, 73, 53, 119, 79, 39, 36, 53, 0.5560747663551402, 0.4906832298136646, 0.4642857142857143, 0.4931506849315068, 1.0, 717, 544, 298, 233, 173, 416, 285, 175, 97, 173, 0.5801952580195258, 0.5238970588235294, 0.587248322147651, 0.41630901287553645, 1.0, 466, 315, 183, 128, 151, 357, 233, 138, 91, 151, 0.7660944206008584, 0.7396825396825397, 0.7540983606557377, 0.7109375,1.0,112)
I can't figure out how to solve this issue. My guess would be that there is a hard limit on how much column you can insert at once but I don't know how to solve that.
Thank you in advance for your help
I have a data set that looks like this:
structure(list(landings = c(116, 31, 0, 0, 0,
0, 0, 0, 0, 120, 0, 241, 9, 0, 64, 326, 142, 605, 139, 410,
212, 470, 416, 309, 1269, 474, 22, 135, 395, 464, 451, 32,
2537, 210, 299, 1522, 184, 550, 666, 429, 1372, 184, 147,
1208, 159, 951, 1000, 1100, 301, 144, 244, 0, 0, 281, 0,
0, 0, 0, 0, 0, 0, 0, 0, 42, 594, 26, 747, 436, 0, 914, 182,
8, 275, 175, 766, 130, 930, 31, 177, 123, 895, 88, 107, 0,
4, 481, 909, 511, 877, 402, 295, 336, 645, 310, 301, 398,
411, 0, 205, 293, 49, 454, 162, 138, 1171, 0, 138, 0, 111,
0, 0, 36, 78, 114, 0, 0, 134, 44, 549, 0, 378, 716, 739,
393, 203, 839, 70, 454, 132, 651, 63, 1850, 217, 403, 55,
0, 408, 43, 17, 12, 26, 2, 811, 581, 1216, 154, 1059, 89,
1862, 1310, 297, 29, 680, 0, 0, 29, 0, 0, 0, 0, 0, 0, 17,
6, 0, 0, 0, 44, 909, 0, 0, 0, 194, 0, 212, 18, 46, 44, 56,
365, 37, 0, 73, 11, 16, 19, 0, 0, 0, 23, 0, 92, 0, 216, 0,
16, 0, 80, 319, 59, 35, 929, 47, 0, 0, 356, 0, 0, 33, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 0, 0, 91, 362, 0,
0, 0, 0, 0, 29, 0, 0, 392, 105, 0, 94, 15, 222, 34, 44, 178,
1867, 0, 224, 241, 23, 1502, 492, 168, 0, 234, 299, 453,
0, 406, 149, 0, 39, 57, 86, 0, 28, 23, 265, 0, 0, 0, 168,
31, 20, 0, 28, 78, 244, 13, 0, 99, 168, 861, 52, 649, 0,
174, 0, 0, 2462, 64, 178, 0, 61, 0, 321, 391, 33, 17, 227,
241, 248, 294, 1119, 37, 90, 0, 85, 37, 89, 0, 0, 0), Date = c(2014,
2014.01916495551, 2014.03832991102, 2014.05749486653, 2014.07665982204,
2014.09582477755, 2014.11498973306, 2014.13415468857, 2014.15331964408,
2014.17248459959, 2014.1916495551, 2014.21081451061, 2014.22997946612,
2014.24914442163, 2014.26830937714, 2014.28747433265, 2014.30663928816,
2014.32580424367, 2014.34496919918, 2014.36413415469, 2014.3832991102,
2014.40246406571, 2014.42162902122, 2014.44079397673, 2014.45995893224,
2014.47912388775, 2014.49828884326, 2014.51745379877, 2014.53661875428,
2014.55578370979, 2014.5749486653, 2014.59411362081, 2014.61327857632,
2014.63244353183, 2014.65160848734, 2014.67077344285, 2014.68993839836,
2014.70910335387, 2014.72826830938, 2014.74743326489, 2014.7665982204,
2014.78576317591, 2014.80492813142, 2014.82409308693, 2014.84325804244,
2014.86242299795, 2014.88158795346, 2014.90075290897, 2014.91991786448,
2014.93908281999, 2014.9582477755, 2014.97741273101, 2014.99657768652,
2015.01574264203, 2015.03490759754, 2015.05407255305, 2015.07323750856,
2015.09240246407, 2015.11156741958, 2015.13073237509, 2015.1498973306,
2015.16906228611, 2015.18822724162, 2015.20739219713, 2015.22655715264,
2015.24572210815, 2015.26488706366, 2015.28405201916, 2015.30321697467,
2015.32238193018, 2015.34154688569, 2015.3607118412, 2015.37987679671,
2015.39904175222, 2015.41820670773, 2015.43737166324, 2015.45653661875,
2015.47570157426, 2015.49486652977, 2015.51403148528, 2015.53319644079,
2015.5523613963, 2015.57152635181, 2015.59069130732, 2015.60985626283,
2015.62902121834, 2015.64818617385, 2015.66735112936, 2015.68651608487,
2015.70568104038, 2015.72484599589, 2015.7440109514, 2015.76317590691,
2015.78234086242, 2015.80150581793, 2015.82067077344, 2015.83983572895,
2015.85900068446, 2015.87816563997, 2015.89733059548, 2015.91649555099,
2015.9356605065, 2015.95482546201, 2015.97399041752, 2015.99315537303,
2016.01232032854, 2016.03148528405, 2016.05065023956, 2016.06981519507,
2016.08898015058, 2016.10814510609, 2016.1273100616, 2016.14647501711,
2016.16563997262, 2016.18480492813, 2016.20396988364, 2016.22313483915,
2016.24229979466, 2016.26146475017, 2016.28062970568, 2016.29979466119,
2016.3189596167, 2016.33812457221, 2016.35728952772, 2016.37645448323,
2016.39561943874, 2016.41478439425, 2016.43394934976, 2016.45311430527,
2016.47227926078, 2016.49144421629, 2016.5106091718, 2016.52977412731,
2016.54893908282, 2016.56810403833, 2016.58726899384, 2016.60643394935,
2016.62559890486, 2016.64476386037, 2016.66392881588, 2016.68309377139,
2016.7022587269, 2016.72142368241, 2016.74058863792, 2016.75975359343,
2016.77891854894, 2016.79808350445, 2016.81724845996, 2016.83641341547,
2016.85557837098, 2016.87474332649, 2016.893908282, 2016.91307323751,
2016.93223819302, 2016.95140314853, 2016.97056810404, 2016.98973305955,
2017.00889801506, 2017.02806297057, 2017.04722792608, 2017.06639288159,
2017.0855578371, 2017.10472279261, 2017.12388774812, 2017.14305270363,
2017.16221765914, 2017.18138261465, 2017.20054757016, 2017.21971252567,
2017.23887748118, 2017.25804243669, 2017.2772073922, 2017.29637234771,
2017.31553730322, 2017.33470225873, 2017.35386721424, 2017.37303216975,
2017.39219712526, 2017.41136208077, 2017.43052703628, 2017.44969199179,
2017.4688569473, 2017.48802190281, 2017.50718685832, 2017.52635181383,
2017.54551676934, 2017.56468172485, 2017.58384668036, 2017.60301163587,
2017.62217659138, 2017.64134154689, 2017.6605065024, 2017.67967145791,
2017.69883641342, 2017.71800136893, 2017.73716632444, 2017.75633127995,
2017.77549623546, 2017.79466119097, 2017.81382614648, 2017.83299110199,
2017.85215605749, 2017.871321013, 2017.89048596851, 2017.90965092402,
2017.92881587953, 2017.94798083504, 2017.96714579055, 2017.98631074606,
2018.00547570157, 2018.02464065708, 2018.04380561259, 2018.0629705681,
2018.08213552361, 2018.12046543463, 2018.13963039014, 2018.15879534565,
2018.17796030116, 2018.19712525667, 2018.21629021218, 2018.23545516769,
2018.2546201232, 2018.27378507871, 2018.29295003422, 2018.31211498973,
2018.33127994524, 2018.35044490075, 2018.36960985626, 2018.38877481177,
2018.40793976728, 2018.42710472279, 2018.4462696783, 2018.46543463381,
2018.48459958932, 2018.50376454483, 2018.52292950034, 2018.54209445585,
2018.56125941136, 2018.58042436687, 2018.59958932238, 2018.61875427789,
2018.6379192334, 2018.65708418891, 2018.67624914442, 2018.69541409993,
2018.71457905544, 2018.73374401095, 2018.75290896646, 2018.77207392197,
2018.79123887748, 2018.81040383299, 2018.8295687885, 2018.84873374401,
2018.86789869952, 2018.88706365503, 2018.90622861054, 2018.92539356605,
2018.94455852156, 2018.96372347707, 2018.98288843258, 2019.00205338809,
2019.0212183436, 2019.04038329911, 2019.05954825462, 2019.07871321013,
2019.09787816564, 2019.11704312115, 2019.13620807666, 2019.15537303217,
2019.17453798768, 2019.19370294319, 2019.2128678987, 2019.23203285421,
2019.25119780972, 2019.27036276523, 2019.28952772074, 2019.30869267625,
2019.32785763176, 2019.34702258727, 2019.36618754278, 2019.38535249829,
2019.4045174538, 2019.42368240931, 2019.44284736482, 2019.46201232033,
2019.48117727584, 2019.50034223135, 2019.51950718686, 2019.53867214237,
2019.55783709788, 2019.57700205339, 2019.5961670089, 2019.61533196441,
2019.63449691992, 2019.65366187543, 2019.67282683094, 2019.69199178645,
2019.71115674196, 2019.73032169747, 2019.74948665298, 2019.76865160849,
2019.787816564, 2019.80698151951, 2019.82614647502, 2019.84531143053,
2019.86447638604, 2019.88364134155, 2019.90280629706, 2019.92197125257,
2019.94113620808, 2019.96030116359, 2019.9794661191))
I am running a GAM that looks like this:
gam1<-gam(landings~s(Date))
I am using draw to plot my data:
draw(gam1)
I have been looking to figure out what the uncertainty is measured by in draw() with no success. Is this a 95% confidence interval or standard error that is used to plot the uncertainty in this plot?
It's an approximate 95% credible interval (drawn at 2 * the standard error of the smooth), the same as you'd get from mgcv:::plot.gam().
I should make this clearer, and allow users to control what coverage they want for the interval, in the package.
I have ECC public and private generated with BouncyCastle:
Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
ECNamedCurveParameterSpec ecSpec = ECNamedCurveTable
.getParameterSpec("secp192r1");
KeyPairGenerator g = KeyPairGenerator.getInstance("ECDSA", "BC");
g.initialize(ecSpec, new SecureRandom());
KeyPair pair = g.generateKeyPair();
System.out.println(Arrays.toString(pair.getPrivate().getEncoded()));
System.out.println(Arrays.toString(pair.getPublic().getEncoded()));
byte[] privateKey = new byte[]{48, 123, 2, 1, 0, 48, 19, 6, 7, 42, -122, 72, -50, 61, 2, 1, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, 4, 97, 48, 95, 2, 1, 1, 4, 24, 14, 117, 7, -120, 15, 109, -59, -35, 72, -91, 99, -2, 51, -120, 112, -47, -1, -115, 25, 48, -104, -93, 78, -7, -96, 10, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, -95, 52, 3, 50, 0, 4, 64, 48, -104, 32, 41, 13, 1, -75, -12, -51, -24, -13, 56, 75, 19, 74, -13, 75, -82, 35, 1, -50, -93, -115, -115, -34, -81, 119, -109, -50, -39, -57, -20, -67, 65, -50, 66, -122, 96, 84, 117, -49, -101, 54, -30, 77, -110, -122}
byte[] publicKey = new byte[]{48, 73, 48, 19, 6, 7, 42, -122, 72, -50, 61, 2, 1, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, 3, 50, 0, 4, 64, 48, -104, 32, 41, 13, 1, -75, -12, -51, -24, -13, 56, 75, 19, 74, -13, 75, -82, 35, 1, -50, -93, -115, -115, -34, -81, 119, -109, -50, -39, -57, -20, -67, 65, -50, 66, -122, 96, 84, 117, -49, -101, 54, -30, 77, -110, -122}
How to convert them into traditional format which can be reused later in https://github.com/kmackay/micro-ecc/blob/master/uECC.h? I need 24 bytes private and 48 public key while now it is 125 and 75.
Gives 24 and 48, sometimes when 0 is added at the beginning 25 or 49:
ECPrivateKey ecPrivateKey = (ECPrivateKey)privateKey;
System.out.println(ecPrivateKey.getS().toByteArray().length);
ECPublicKey ecPublicKey = (ECPublicKey)publicKey;
System.out.println(ecPublicKey.getW().getAffineX().toByteArray().length + ecPublicKey.getW().getAffineY().toByteArray().length);