Hi there I was wondering whether someone might assist with combining plots generated using the example provide on this page Depth Profiling visualization where I have analyzed data for salinity and depth, however I have a categorical variable dividing three estuaries based on whether the mouth is "closed", "open", or "semi-closed". I used the code of Depth Profiling Visualization, however each plot has its own salinity legend scale per plot.
Here is the data.
State Distance Depth pH DO Chla Salinity Max.depth
1 Closed 0.60 0.0 8.66 10.64 0.8880000 18.49 -1.3
2 Closed 0.60 0.5 8.68 10.79 1.4800000 18.51 -1.3
3 Closed 0.60 1.3 8.73 11.26 1.1840000 18.51 -1.3
4 Closed 1.00 0.0 8.48 9.07 5.3280000 18.18 -0.8
5 Closed 1.00 0.8 8.47 8.30 2.9600000 18.35 -0.8
6 Closed 1.60 0.0 8.38 9.70 1.1840000 18.38 -2.0
7 Closed 1.60 0.5 8.40 9.33 NA 18.39 -2.0
8 Closed 1.60 1.0 8.40 9.27 1.1840000 18.39 -2.0
9 Closed 1.60 1.5 8.41 9.27 NA 18.41 -2.0
10 Closed 1.60 2.0 8.47 9.23 1.4800000 18.57 -2.0
11 Closed 2.15 0.0 8.40 9.85 2.6640000 18.26 -3.5
12 Closed 2.15 0.5 8.41 9.95 NA 18.27 -3.5
13 Closed 2.15 1.0 8.42 9.16 1.1840000 18.28 -3.5
14 Closed 2.15 2.0 8.42 9.82 NA 18.28 -3.5
15 Closed 2.15 3.5 8.38 9.17 0.5920000 18.30 -3.5
16 Closed 3.50 0.0 8.30 9.82 2.0720000 17.71 -5.0
17 Closed 3.50 0.5 8.31 9.78 NA 17.71 -5.0
18 Closed 3.50 1.0 8.32 9.75 1.4800000 17.72 -5.0
19 Closed 3.50 2.0 8.32 9.73 NA 17.78 -5.0
20 Closed 3.50 3.0 8.30 9.20 NA 17.95 -5.0
21 Closed 3.50 4.0 8.29 8.80 NA 18.00 -5.0
22 Closed 3.50 5.0 8.24 7.47 1.4800000 18.06 -5.0
23 Closed 4.85 0.0 8.21 10.10 2.9600000 17.33 -1.6
24 Closed 4.85 0.5 8.21 9.90 2.0720000 17.33 -1.6
25 Closed 4.85 1.0 8.21 9.73 NA 17.32 -1.6
26 Closed 4.85 1.6 8.22 9.60 1.1840000 17.32 -1.6
27 Closed 6.00 0.0 8.07 9.07 4.4400000 16.65 -1.5
28 Closed 6.00 0.5 8.06 8.98 5.6240000 16.65 -1.5
29 Closed 6.00 1.0 8.06 8.81 NA 16.67 -1.5
30 Closed 6.00 1.5 8.10 8.80 4.1440000 16.67 -1.5
31 Closed 6.70 0.0 7.83 9.25 0.0000000 13.90 -0.5
32 Open 0.60 0.0 7.56 8.42 1.1840000 1.62 -0.5
33 Open 0.60 0.5 7.62 8.40 1.9733333 1.79 -0.5
34 Open 1.00 0.0 7.67 8.55 1.1840000 1.10 -0.4
35 Open 1.00 0.4 7.62 8.49 1.5786667 1.10 -0.4
36 Open 1.60 0.0 7.48 8.40 1.5786667 0.98 -1.0
37 Open 1.60 0.5 7.47 8.33 NA 0.98 -1.0
38 Open 1.60 1.0 7.45 8.33 2.7626667 0.99 -1.0
39 Open 2.15 0.0 7.19 7.99 1.1840000 0.85 -1.0
40 Open 2.15 0.5 7.19 7.96 NA 0.86 -1.0
41 Open 2.15 1.0 7.18 7.98 1.1840000 0.89 -1.0
42 Open 3.50 0.0 7.14 7.56 0.3946667 0.55 -4.8
43 Open 3.50 0.5 7.20 7.50 NA 0.55 -4.8
44 Open 3.50 1.0 7.28 7.38 1.9733333 0.55 -4.8
45 Open 3.50 2.0 7.38 7.10 NA 0.55 -4.8
46 Open 3.50 3.0 7.56 6.15 NA 0.56 -4.8
47 Open 3.50 4.0 7.20 4.43 NA 2.65 -4.8
48 Open 3.50 4.8 6.93 2.25 1.9733333 6.76 -4.8
49 Open 4.85 0.0 6.90 8.29 1.1840000 0.26 -1.2
50 Open 4.85 0.5 6.77 8.20 0.7893333 0.27 -1.2
51 Open 4.85 1.2 6.55 8.20 0.7893333 0.39 -1.2
52 Open 6.00 0.0 6.49 9.53 1.1840000 0.13 -1.0
53 Open 6.00 0.5 6.59 9.53 NA 0.13 -1.0
54 Open 6.00 1.0 6.79 9.53 1.1840000 0.13 -1.0
55 Open 6.70 0.0 6.48 9.48 0.7893333 0.11 -0.5
56 Semi-closed 0.60 0.0 8.05 6.30 19.7300000 18.86 -1.4
57 Semi-closed 0.60 0.5 8.04 5.19 19.7300000 24.07 -1.4
58 Semi-closed 0.60 1.0 8.00 5.98 NA 30.50 -1.4
59 Semi-closed 0.60 1.4 7.87 6.19 5.1300000 31.18 -1.4
60 Semi-closed 1.00 0.0 7.99 5.75 22.8900000 18.81 -0.9
61 Semi-closed 1.00 0.5 7.95 5.10 NA 19.08 -0.9
62 Semi-closed 1.00 0.9 7.86 3.42 11.8400000 26.60 -0.9
63 Semi-closed 1.60 0.0 7.88 6.05 11.4500000 17.29 -1.7
64 Semi-closed 1.60 0.5 7.87 5.78 NA 17.32 -1.7
65 Semi-closed 1.60 1.0 7.86 4.74 8.6800000 17.44 -1.7
66 Semi-closed 1.60 1.5 7.84 3.90 NA 19.65 -1.7
67 Semi-closed 1.60 1.7 7.91 3.75 9.0800000 21.07 -1.7
68 Semi-closed 2.15 0.0 7.91 6.95 22.8900000 16.50 -1.3
69 Semi-closed 2.15 0.5 7.92 6.76 26.4400000 16.50 -1.3
70 Semi-closed 2.15 1.0 7.91 5.99 NA 17.40 -1.3
71 Semi-closed 2.15 1.3 7.97 4.10 7.1000000 18.79 -1.3
72 Semi-closed 3.50 0.0 7.75 6.13 18.5500000 15.86 -4.5
73 Semi-closed 3.50 0.5 7.72 5.90 NA 15.86 -4.5
74 Semi-closed 3.50 1.0 7.65 4.38 9.0800000 16.38 -4.5
75 Semi-closed 3.50 1.5 7.56 1.59 NA 20.09 -4.5
76 Semi-closed 3.50 2.0 7.55 0.38 NA 22.11 -4.5
77 Semi-closed 3.50 3.0 7.53 0.42 NA 30.36 -4.5
78 Semi-closed 3.50 4.0 7.52 0.52 NA 31.50 -4.5
79 Semi-closed 3.50 4.5 7.54 0.68 1.1800000 31.84 -4.5
80 Semi-closed 4.85 0.0 7.66 6.31 21.7100000 15.41 -1.6
81 Semi-closed 4.85 0.5 7.65 6.18 NA 15.44 -1.6
82 Semi-closed 4.85 1.0 7.65 5.57 21.3100000 15.54 -1.6
83 Semi-closed 4.85 1.6 7.52 0.76 6.7100000 22.60 -1.6
84 Semi-closed 6.00 0.0 7.74 8.50 87.6200000 13.11 -1.0
85 Semi-closed 6.00 0.5 7.66 7.38 NA 13.92 -1.0
86 Semi-closed 6.00 1.0 7.60 3.20 7.5000000 15.42 -1.0
87 Semi-closed 6.70 0.0 8.55 6.94 0.0000000 0.25 -0.5
I was hoping someone might be able to assist to unify the scales of the three legends from the three mouth conditions of estuary so that only one legend describing salinity for all plots is possible.
I have numpy array with the sape of 178 rows X 14 columns like this:
0 1 2 3 4 5 6 7 8 9 10 11 \
0 1.0 14.23 1.71 2.43 15.6 127.0 2.80 3.06 0.28 2.29 5.64 1.04
1 1.0 13.20 1.78 2.14 11.2 100.0 2.65 2.76 0.26 1.28 4.38 1.05
2 1.0 13.16 2.36 2.67 18.6 101.0 2.80 3.24 0.30 2.81 5.68 1.03
3 1.0 14.37 1.95 2.50 16.8 113.0 3.85 3.49 0.24 2.18 7.80 0.86
4 1.0 13.24 2.59 2.87 21.0 118.0 2.80 2.69 0.39 1.82 4.32 1.04
.. ... ... ... ... ... ... ... ... ... ... ... ...
173 3.0 13.71 5.65 2.45 20.5 95.0 1.68 0.61 0.52 1.06 7.70 0.64
174 3.0 13.40 3.91 2.48 23.0 102.0 1.80 0.75 0.43 1.41 7.30 0.70
175 3.0 13.27 4.28 2.26 20.0 120.0 1.59 0.69 0.43 1.35 10.20 0.59
176 3.0 13.17 2.59 2.37 20.0 120.0 1.65 0.68 0.53 1.46 9.30 0.60
177 3.0 14.13 4.10 2.74 24.5 96.0 2.05 0.76 0.56 1.35 9.20 0.61
12 13
0 3.92 1065.0
1 3.40 1050.0
2 3.17 1185.0
3 3.45 1480.0
4 2.93 735.0
.. ... ...
173 1.74 740.0
174 1.56 750.0
175 1.56 835.0
176 1.62 840.0
177 1.60 560.0
[178 rows x 14 columns]
I tried to print it in dataframe for all the rows and only the first (index 0) column and the output worked like this:
0
0 1.0
1 1.0
2 1.0
3 1.0
4 1.0
.. ...
173 3.0
174 3.0
175 3.0
176 3.0
177 3.0
[178 rows x 1 columns]
using the same logic, I want totake all the rows and only the first column with the value is below 2. I tried to do it like this and it doesn't work:
reduced = data[data[:,0:1]<=2]
I got an
IndexError
like this:
IndexError Traceback (most recent call last)
<ipython-input-159-7eab0abd8f99> in <module>()
----> 1 reduced = data[data[:,0:1]<=2]
IndexError: boolean index did not match indexed array along dimension 1; dimension is 14 but corresponding boolean dimension is 1.
anybody can help me?
thank in advance
Solved it.
It is just as simple as just convert the numpy array to dataframe and then select rows based on condition in dataframe:
reduced = data[data['class'] <= 2]
i have below dataframe belows .. and i wanna convert it to numpy array.
when i tried.. time order is broken converted to numpy array.
may it's because it is time-series data (19:00~0:00:00~07:00:00)
how can i keep time-order convert dataframe to numpy array?
aaa \
Date 2015-12-06 2015-12-13 2015-12-20 2015-12-23 2015-12-26 2016-01-03
Time
19:00:00 4.72 8.50 3.87 7.95 1.76 9.82
19:15:00 4.54 8.00 3.72 8.14 1.74 9.77
19:30:00 4.44 8.17 3.72 7.99 1.75 9.77
19:45:00 4.37 7.92 3.28 7.94 1.89 9.61
20:00:00 4.03 7.54 2.48 7.99 1.98 9.46
20:15:00 3.74 7.86 3.30 7.68 1.63 9.30
20:30:00 3.48 8.41 3.52 7.88 1.52 9.22
20:45:00 3.31 8.52 3.81 7.83 1.54 9.08
21:00:00 3.17 8.23 3.97 7.96 1.63 9.14
21:15:00 2.99 8.23 3.37 7.61 1.87 9.14
21:30:00 2.96 8.26 3.23 7.63 2.03 9.13
21:45:00 2.69 7.89 3.10 7.34 2.12 9.04
22:00:00 2.62 7.83 2.94 7.21 2.11 9.04
22:15:00 2.55 7.78 2.83 7.26 2.39 9.01
22:30:00 2.49 7.73 2.89 7.15 2.30 9.08
22:45:00 2.48 7.80 2.79 7.02 2.22 8.92
23:00:00 2.38 7.71 2.92 7.17 2.43 8.80
23:15:00 2.23 7.74 3.01 7.24 2.33 8.56
23:30:00 2.29 7.51 3.10 7.14 2.38 8.32
23:45:00 2.29 7.31 3.00 6.89 2.10 8.02
00:00:00 2.17 6.84 2.84 6.89 1.82 7.86
00:15:00 2.13 6.84 2.65 7.06 1.36 7.95
00:30:00 2.21 6.78 2.63 6.98 0.92 7.97
00:45:00 2.19 6.41 2.18 7.08 1.05 7.80
01:00:00 2.13 6.24 1.56 7.20 0.81 7.73
01:15:00 2.14 5.90 1.39 7.31 1.01 7.89
01:30:00 2.13 5.74 1.81 7.58 0.79 7.91
01:45:00 2.11 5.82 1.60 7.47 1.19 8.02
02:00:00 1.72 6.01 0.90 7.14 1.27 8.09
02:15:00 1.94 6.04 1.12 7.33 0.95 8.13
02:30:00 2.05 6.00 1.44 7.06 1.15 8.15
02:45:00 1.96 6.03 1.45 6.86 1.05 7.95
03:00:00 1.63 6.28 1.62 6.85 1.22 7.43
03:15:00 1.79 6.14 1.41 6.94 1.05 6.97
03:30:00 1.37 6.03 1.29 6.98 1.27 6.97
03:45:00 1.44 5.84 1.01 7.29 1.31 6.90
04:00:00 1.37 5.62 0.92 7.13 1.35 6.77
04:15:00 1.62 5.75 0.95 7.18 1.21 7.09
04:30:00 1.64 5.71 1.06 7.18 1.32 7.27
04:45:00 1.40 5.46 0.79 7.17 1.55 7.35
05:00:00 1.51 5.48 0.64 6.83 1.42 7.27
05:15:00 1.46 5.80 0.52 6.58 1.60 7.21
05:30:00 1.61 5.59 0.35 6.98 1.54 7.13
05:45:00 1.49 5.28 0.46 6.58 1.58 7.04
06:00:00 1.55 5.00 0.17 6.35 1.88 7.10
06:15:00 1.94 4.94 -0.18 6.12 1.94 7.11
06:30:00 1.45 5.01 -0.31 6.02 1.90 7.14
06:45:00 1.36 4.90 -0.17 5.83 2.06 7.17
07:00:00 1.25 4.75 0.20 5.70 2.35 7.18
You need transpose DataFrame by T and convert to array:
arr = df.T.values
Or first convert to array and then transpose:
arr = df.values.T
I want to use np.savetxt(file,array,fmt='%8.1f') to save as txt
1958 6.4 1.8 7.7 70.1 41.4 38.5 65.4 25.7
1959 27.2 42.5 63.3 86.2 101.5 71.4 114.2 137.9
1960 22.9 18.3 28.7 106.5 159.1 50.4 203 121.6
1961 4.4 26.9 47.1 67.9 53.6 64.8 95 42
1962 20.9 31.2 60.6 38.8 66.2 37.9 67.9 62.3
1963 11.9 14.5 59 56 83.1 110.9 77.1 93.5
each element take up 8 spaces one by one(no seperation between each one).
First cols year format is %8d, and others is %8.1f. flush right.
How to do this in numpy? or using pandas?
n = len(df.columns)
fmt = ('{:8.0f}' + '{:8.1f}' * (n - 1)).format
print(df.apply(lambda x: fmt(*x), 1).to_csv(index=None, header=None))
1958 6.4 1.8 7.7 70.1 41.4 38.5 65.4 25.7
1959 27.2 42.5 63.3 86.2 101.5 71.4 114.2 137.9
1960 22.9 18.3 28.7 106.5 159.1 50.4 203.0 121.6
1961 4.4 26.9 47.1 67.9 53.6 64.8 95.0 42.0
1962 20.9 31.2 60.6 38.8 66.2 37.9 67.9 62.3
1963 11.9 14.5 59.0 56.0 83.1 110.9 77.1 93.5
I'm looking to extract and print a specific line from a table I have in a long log file. It looks something like this:
******************************************************************************
XSCALE (VERSION July 4, 2012) 4-Jun-2013
******************************************************************************
Author: Wolfgang Kabsch
Copy licensed until 30-Jun-2013 to
academic users for non-commercial applications
No redistribution.
******************************************************************************
CONTROL CARDS
******************************************************************************
MAXIMUM_NUMBER_OF_PROCESSORS=16
RESOLUTION_SHELLS= 20 10 6 4 3 2.5 2.0 1.9 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8
MINIMUM_I/SIGMA=4.0
OUTPUT_FILE=fae-ip.ahkl
INPUT_FILE= /dls/sci-scratch/Sam/FC59251/fr6_1/XDS_ASCII.HKL
THE DATA COLLECTION STATISTICS REPORTED BELOW ASSUMES:
SPACE_GROUP_NUMBER= 97
UNIT_CELL_CONSTANTS= 128.28 128.28 181.47 90.000 90.000 90.000
***** 16 EQUIVALENT POSITIONS IN SPACE GROUP # 97 *****
If x',y',z' is an equivalent position to x,y,z, then
x'=x*ML(1)+y*ML( 2)+z*ML( 3)+ML( 4)/12.0
y'=x*ML(5)+y*ML( 6)+z*ML( 7)+ML( 8)/12.0
z'=x*ML(9)+y*ML(10)+z*ML(11)+ML(12)/12.0
# 1 2 3 4 5 6 7 8 9 10 11 12
1 1 0 0 0 0 1 0 0 0 0 1 0
2 -1 0 0 0 0 -1 0 0 0 0 1 0
3 -1 0 0 0 0 1 0 0 0 0 -1 0
4 1 0 0 0 0 -1 0 0 0 0 -1 0
5 0 1 0 0 1 0 0 0 0 0 -1 0
6 0 -1 0 0 -1 0 0 0 0 0 -1 0
7 0 -1 0 0 1 0 0 0 0 0 1 0
8 0 1 0 0 -1 0 0 0 0 0 1 0
9 1 0 0 6 0 1 0 6 0 0 1 6
10 -1 0 0 6 0 -1 0 6 0 0 1 6
11 -1 0 0 6 0 1 0 6 0 0 -1 6
12 1 0 0 6 0 -1 0 6 0 0 -1 6
13 0 1 0 6 1 0 0 6 0 0 -1 6
14 0 -1 0 6 -1 0 0 6 0 0 -1 6
15 0 -1 0 6 1 0 0 6 0 0 1 6
16 0 1 0 6 -1 0 0 6 0 0 1 6
ALL DATA SETS WILL BE SCALED TO /dls/sci-scratch/Sam/FC59251/fr6_1/XDS_ASCII.HKL
******************************************************************************
READING INPUT REFLECTION DATA FILES
******************************************************************************
DATA MEAN REFLECTIONS INPUT FILE NAME
SET# INTENSITY ACCEPTED REJECTED
1 0.1358E+03 1579957 0 /dls/sci-scratch/Sam/FC59251/fr6_1/XDS_ASCII.HKL
******************************************************************************
CORRECTION FACTORS AS FUNCTION OF IMAGE NUMBER & RESOLUTION
******************************************************************************
RECIPROCAL CORRECTION FACTORS FOR INPUT DATA SETS MERGED TO
OUTPUT FILE: fae-ip.ahkl
THE CALCULATIONS ASSUME FRIEDEL'S_LAW= TRUE
TOTAL NUMBER OF CORRECTION FACTORS DEFINED 720
DEGREES OF FREEDOM OF CHI^2 FIT 357222.9
CHI^2-VALUE OF FIT OF CORRECTION FACTORS 1.024
NUMBER OF CYCLES CARRIED OUT 4
CORRECTION FACTORS for visual inspection by XDS-Viewer DECAY_001.cbf
XMIN= 0.6 XMAX= 1799.3 NXBIN= 36
YMIN= 0.00049 YMAX= 0.44483 NYBIN= 20
NUMBER OF REFLECTIONS USED FOR DETERMINING CORRECTION FACTORS 396046
******************************************************************************
CORRECTION FACTORS AS FUNCTION OF X (fast) & Y(slow) IN THE DETECTOR PLANE
******************************************************************************
RECIPROCAL CORRECTION FACTORS FOR INPUT DATA SETS MERGED TO
OUTPUT FILE: fae-ip.ahkl
THE CALCULATIONS ASSUME FRIEDEL'S_LAW= TRUE
TOTAL NUMBER OF CORRECTION FACTORS DEFINED 7921
DEGREES OF FREEDOM OF CHI^2 FIT 356720.6
CHI^2-VALUE OF FIT OF CORRECTION FACTORS 1.023
NUMBER OF CYCLES CARRIED OUT 3
CORRECTION FACTORS for visual inspection by XDS-Viewer MODPIX_001.cbf
XMIN= 5.4 XMAX= 2457.6 NXBIN= 89
YMIN= 40.0 YMAX= 2516.7 NYBIN= 89
NUMBER OF REFLECTIONS USED FOR DETERMINING CORRECTION FACTORS 396046
******************************************************************************
CORRECTION FACTORS AS FUNCTION OF IMAGE NUMBER & DETECTOR SURFACE POSITION
******************************************************************************
RECIPROCAL CORRECTION FACTORS FOR INPUT DATA SETS MERGED TO
OUTPUT FILE: fae-ip.ahkl
THE CALCULATIONS ASSUME FRIEDEL'S_LAW= TRUE
TOTAL NUMBER OF CORRECTION FACTORS DEFINED 468
DEGREES OF FREEDOM OF CHI^2 FIT 357286.9
CHI^2-VALUE OF FIT OF CORRECTION FACTORS 1.022
NUMBER OF CYCLES CARRIED OUT 3
CORRECTION FACTORS for visual inspection by XDS-Viewer ABSORP_001.cbf
XMIN= 0.6 XMAX= 1799.3 NXBIN= 36
DETECTOR_SURFACE_POSITION= 1232 1278
DETECTOR_SURFACE_POSITION= 1648 1699
DETECTOR_SURFACE_POSITION= 815 1699
DETECTOR_SURFACE_POSITION= 815 858
DETECTOR_SURFACE_POSITION= 1648 858
DETECTOR_SURFACE_POSITION= 2174 1673
DETECTOR_SURFACE_POSITION= 1622 2230
DETECTOR_SURFACE_POSITION= 841 2230
DETECTOR_SURFACE_POSITION= 289 1673
DETECTOR_SURFACE_POSITION= 289 884
DETECTOR_SURFACE_POSITION= 841 326
DETECTOR_SURFACE_POSITION= 1622 326
DETECTOR_SURFACE_POSITION= 2174 884
NUMBER OF REFLECTIONS USED FOR DETERMINING CORRECTION FACTORS 396046
******************************************************************************
CORRECTION PARAMETERS FOR THE STANDARD ERROR OF REFLECTION INTENSITIES
******************************************************************************
The variance v0(I) of the intensity I obtained from counting statistics is
replaced by v(I)=a*(v0(I)+b*I^2). The model parameters a, b are chosen to
minimize the discrepancies between v(I) and the variance estimated from
sample statistics of symmetry related reflections. This model implicates
an asymptotic limit ISa=1/SQRT(a*b) for the highest I/Sigma(I) that the
experimental setup can produce (Diederichs (2010) Acta Cryst D66, 733-740).
Often the value of ISa is reduced from the initial value ISa0 due to systematic
errors showing up by comparison with other data sets in the scaling procedure.
(ISa=ISa0=-1 if v0 is unknown for a data set.)
a b ISa ISa0 INPUT DATA SET
1.086E+00 1.420E-03 25.46 29.00 /dls/sci-scratch/Sam/FC59251/fr6_1/XDS_ASCII.HKL
FACTOR TO PLACE ALL DATA SETS TO AN APPROXIMATE ABSOLUTE SCALE 0.4178E+04
(ASSUMING A PROTEIN WITH 50% SOLVENT)
******************************************************************************
STATISTICS OF SCALED OUTPUT DATA SET : fae-ip.ahkl
FILE TYPE: XDS_ASCII MERGE=FALSE FRIEDEL'S_LAW=TRUE
186 OUT OF 1579957 REFLECTIONS REJECTED
1579771 REFLECTIONS ON OUTPUT FILE
******************************************************************************
DEFINITIONS:
R-FACTOR
observed = (SUM(ABS(I(h,i)-I(h))))/(SUM(I(h,i)))
expected = expected R-FACTOR derived from Sigma(I)
COMPARED = number of reflections used for calculating R-FACTOR
I/SIGMA = mean of intensity/Sigma(I) of unique reflections
(after merging symmetry-related observations)
Sigma(I) = standard deviation of reflection intensity I
estimated from sample statistics
R-meas = redundancy independent R-factor (intensities)
Diederichs & Karplus (1997), Nature Struct. Biol. 4, 269-275.
CC(1/2) = percentage of correlation between intensities from
random half-datasets. Correlation significant at
the 0.1% level is marked by an asterisk.
Karplus & Diederichs (2012), Science 336, 1030-33
Anomal = percentage of correlation between random half-sets
Corr of anomalous intensity differences. Correlation
significant at the 0.1% level is marked.
SigAno = mean anomalous difference in units of its estimated
standard deviation (|F(+)-F(-)|/Sigma). F(+), F(-)
are structure factor estimates obtained from the
merged intensity observations in each parity class.
Nano = Number of unique reflections used to calculate
Anomal_Corr & SigAno. At least two observations
for each (+ and -) parity are required.
SUBSET OF INTENSITY DATA WITH SIGNAL/NOISE >= -3.0 AS FUNCTION OF RESOLUTION
RESOLUTION NUMBER OF REFLECTIONS COMPLETENESS R-FACTOR R-FACTOR COMPARED I/SIGMA R-meas CC(1/2) Anomal SigAno Nano
LIMIT OBSERVED UNIQUE POSSIBLE OF DATA observed expected Corr
20.00 557 66 74 89.2% 2.7% 3.0% 557 58.75 2.9% 100.0* 45 1.674 25
10.00 5018 417 417 100.0% 2.4% 3.1% 5018 75.34 2.6% 100.0* 2 0.812 276
6.00 18352 1583 1584 99.9% 2.8% 3.3% 18351 65.55 2.9% 100.0* 11* 0.914 1248
4.00 59691 4640 4640 100.0% 3.2% 3.5% 59690 64.96 3.4% 100.0* 4 0.857 3987
3.00 112106 8821 8822 100.0% 4.4% 4.4% 112102 50.31 4.6% 99.9* -3 0.844 7906
2.50 147954 11023 11023 100.0% 8.7% 8.6% 147954 29.91 9.1% 99.8* 0 0.829 10096
2.00 332952 24698 24698 100.0% 21.4% 21.6% 332949 14.32 22.3% 99.2* 1 0.804 22992
1.90 106645 8382 8384 100.0% 56.5% 57.1% 106645 5.63 58.8% 94.7* -2 0.767 7886
1.80 138516 10342 10343 100.0% 86.8% 87.0% 138516 3.64 90.2% 87.9* -2 0.762 9741
1.70 175117 12897 12899 100.0% 140.0% 140.1% 175116 2.15 145.4% 69.6* -2 0.732 12188
1.60 209398 16298 16304 100.0% 206.1% 208.5% 209397 1.35 214.6% 48.9* -2 0.693 15466
1.50 273432 20770 20893 99.4% 333.4% 342.1% 273340 0.80 346.9% 23.2* -1 0.644 19495
1.40 33 27 27248 0.1% 42.6% 112.7% 12 0.40 60.3% 88.2 0 0.000 0
1.30 0 0 36205 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
1.20 0 0 49238 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
1.10 0 0 68746 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
1.00 0 0 98884 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
0.90 0 0 147505 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
0.80 0 0 230396 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
total 1579771 119964 778303 15.4% 12.8% 13.1% 1579647 14.33 13.4% 99.9* -1 0.755 111306
========== STATISTICS OF INPUT DATA SET ==========
R-FACTORS FOR INTENSITIES OF DATA SET /dls/sci-scratch/Sam/FC59251/fr6_1/XDS_ASCII.HKL
RESOLUTION R-FACTOR R-FACTOR COMPARED
LIMIT observed expected
20.00 2.7% 3.0% 557
10.00 2.4% 3.1% 5018
6.00 2.8% 3.3% 18351
4.00 3.2% 3.5% 59690
3.00 4.4% 4.4% 112102
2.50 8.7% 8.6% 147954
2.00 21.4% 21.6% 332949
1.90 56.5% 57.1% 106645
1.80 86.8% 87.0% 138516
1.70 140.0% 140.1% 175116
1.60 206.1% 208.5% 209397
1.50 333.4% 342.1% 273340
1.40 42.6% 112.7% 12
1.30 -99.9% -99.9% 0
1.20 -99.9% -99.9% 0
1.10 -99.9% -99.9% 0
1.00 -99.9% -99.9% 0
0.90 -99.9% -99.9% 0
0.80 -99.9% -99.9% 0
total 12.8% 13.1% 1579647
******************************************************************************
WILSON STATISTICS OF SCALED DATA SET: fae-ip.ahkl
******************************************************************************
Data is divided into resolution shells and a straight line
A - 2*B*SS is fitted to log<I>, where
RES = mean resolution (Angstrom) in shell
SS = mean of (sin(THETA)/LAMBDA)**2 in shell
<I> = mean reflection intensity in shell
BO = (A - log<I>)/(2*SS)
# = number of reflections in resolution shell
WILSON LINE (using all data) : A= 14.997 B= 29.252 CORRELATION= 0.99
# RES SS <I> log(<I>) BO
1667 8.445 0.004 2.3084E+06 14.652 49.2
2798 5.260 0.009 1.5365E+06 14.245 41.6
3547 4.106 0.015 2.0110E+06 14.514 16.3
4147 3.480 0.021 1.2910E+06 14.071 22.4
4688 3.073 0.026 7.3586E+05 13.509 28.1
5154 2.781 0.032 4.6124E+05 13.042 30.3
5568 2.560 0.038 3.1507E+05 12.661 30.6
5966 2.384 0.044 2.4858E+05 12.424 29.2
6324 2.240 0.050 1.8968E+05 12.153 28.5
6707 2.119 0.056 1.3930E+05 11.844 28.3
7030 2.016 0.062 9.1378E+04 11.423 29.0
7331 1.926 0.067 5.4413E+04 10.904 30.4
7664 1.848 0.073 3.5484E+04 10.477 30.9
7934 1.778 0.079 2.4332E+04 10.100 31.0
8193 1.716 0.085 1.8373E+04 9.819 30.5
8466 1.660 0.091 1.4992E+04 9.615 29.7
8743 1.609 0.097 1.1894E+04 9.384 29.1
9037 1.562 0.102 9.4284E+03 9.151 28.5
9001 1.520 0.108 8.3217E+03 9.027 27.6
HIGHER ORDER MOMENTS OF WILSON DISTRIBUTION OF CENTRIC DATA
AS COMPARED WITH THEORETICAL VALUES. (EXPECTED: 1.00)
# RES <I**2>/ <I**3>/ <I**4>/
3<I>**2 15<I>**3 105<I>**4
440 8.445 0.740 0.505 0.294
442 5.260 0.762 0.733 0.735
442 4.106 0.888 0.788 0.717
439 3.480 1.339 1.733 2.278
438 3.073 1.168 1.259 1.400
440 2.781 1.215 1.681 2.269
438 2.560 1.192 1.603 2.405
450 2.384 1.117 1.031 0.891
432 2.240 1.214 1.567 2.173
438 2.119 0.972 0.992 0.933
445 2.016 1.029 1.019 0.986
441 1.926 1.603 1.701 1.554
440 1.848 1.544 1.871 2.076
436 1.778 0.927 0.661 0.435
444 1.716 1.134 1.115 1.197
440 1.660 1.271 1.618 2.890
436 1.609 1.424 1.045 0.941
448 1.562 1.794 1.447 1.423
426 1.520 2.517 1.496 2.099
8355 overall 1.253 1.255 1.455
HIGHER ORDER MOMENTS OF WILSON DISTRIBUTION OF ACENTRIC DATA
AS COMPARED WITH THEORETICAL VALUES. (EXPECTED: 1.00)
# RES <I**2>/ <I**3>/ <I**4>/
2<I>**2 6<I>**3 24<I>**4
1227 8.445 1.322 1.803 2.340
2356 5.260 1.167 1.420 1.789
3105 4.106 1.010 1.046 1.100
3708 3.480 1.055 1.262 1.592
4250 3.073 0.999 1.083 1.375
4714 2.781 1.061 1.232 1.591
5130 2.560 1.049 1.178 1.440
5516 2.384 1.025 1.117 1.290
5892 2.240 1.001 1.058 1.230
6269 2.119 1.060 1.140 1.233
6585 2.016 1.109 1.344 1.709
6890 1.926 1.028 1.100 1.222
7224 1.848 1.060 1.150 1.348
7498 1.778 1.143 1.309 1.655
7749 1.716 1.182 1.299 1.549
8026 1.660 1.286 1.376 1.538
8307 1.609 1.419 1.481 1.707
8589 1.562 1.663 1.750 2.119
8575 1.520 2.271 2.172 5.088
111610 overall 1.253 1.354 1.804
======= CUMULATIVE INTENSITY DISTRIBUTION =======
DEFINITIONS:
<I> = mean reflection intensity
Na(Z)exp = expected number of acentric reflections with I <= Z*<I>
Na(Z)obs = observed number of acentric reflections with I <= Z*<I>
Nc(Z)exp = expected number of centric reflections with I <= Z*<I>
Nc(Z)obs = observed number of centric reflections with I <= Z*<I>
Nc(Z)obs/Nc(Z)exp versus resolution and Z (0.1-1.0)
# RES 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
440 8.445 0.75 0.95 0.98 1.00 0.98 0.99 1.00 1.00 1.02 1.02
442 5.260 1.18 1.11 1.09 1.09 1.07 1.08 1.08 1.08 1.07 1.06
442 4.106 0.97 1.01 0.98 0.97 0.96 0.94 0.92 0.91 0.92 0.94
439 3.480 0.91 0.88 0.91 0.91 0.89 0.90 0.90 0.89 0.89 0.93
438 3.073 0.92 0.92 0.90 0.93 0.94 0.99 1.02 0.99 0.96 0.96
440 2.781 0.98 1.01 1.02 1.05 1.04 1.03 1.04 1.02 1.01 1.01
438 2.560 1.02 1.10 1.05 1.03 1.01 1.03 1.04 1.01 1.04 1.02
450 2.384 0.78 0.93 0.92 0.93 0.89 0.89 0.92 0.95 0.96 0.95
432 2.240 0.69 0.82 0.84 0.86 0.91 0.92 0.93 0.94 0.95 0.95
438 2.119 0.75 0.87 0.95 1.02 1.09 1.09 1.12 1.12 1.10 1.08
445 2.016 0.86 0.86 0.87 0.90 0.91 0.93 0.98 0.99 1.00 1.00
441 1.926 0.88 0.79 0.79 0.81 0.82 0.84 0.85 0.85 0.86 0.86
440 1.848 1.00 0.89 0.85 0.83 0.85 0.85 0.88 0.90 0.90 0.92
436 1.778 1.03 0.87 0.79 0.79 0.80 0.84 0.85 0.87 0.90 0.92
444 1.716 1.09 0.85 0.81 0.78 0.80 0.80 0.81 0.81 0.84 0.85
440 1.660 1.27 1.01 0.93 0.88 0.85 0.84 0.84 0.85 0.88 0.91
436 1.609 1.34 1.00 0.89 0.83 0.80 0.80 0.80 0.81 0.80 0.83
448 1.562 1.39 1.09 0.93 0.86 0.81 0.78 0.77 0.79 0.78 0.78
426 1.520 1.38 1.03 0.88 0.83 0.82 0.80 0.78 0.76 0.75 0.74
8355 overall 1.01 0.95 0.92 0.91 0.91 0.91 0.92 0.92 0.93 0.93
Na(Z)obs/Na(Z)exp versus resolution and Z (0.1-1.0)
# RES 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
1227 8.445 1.10 1.22 1.21 1.21 1.14 1.10 1.12 1.10 1.11 1.09
2356 5.260 1.15 1.10 1.09 1.03 1.03 1.03 1.01 1.01 1.01 1.00
3105 4.106 0.91 0.96 0.99 1.01 1.02 1.00 1.00 0.99 0.99 1.00
3708 3.480 0.93 0.97 1.00 1.06 1.05 1.04 1.04 1.04 1.04 1.05
4250 3.073 0.94 1.02 1.01 1.00 1.01 1.00 1.00 1.01 1.02 1.02
4714 2.781 1.11 1.04 1.02 1.02 1.02 1.01 1.01 1.01 1.00 1.00
5130 2.560 1.00 1.10 1.06 1.03 1.01 1.02 1.01 1.01 1.01 1.02
5516 2.384 1.09 1.08 1.05 1.04 1.04 1.02 1.01 1.01 1.01 1.01
5892 2.240 0.98 0.99 1.00 1.01 1.01 1.01 1.00 1.00 1.00 1.00
6269 2.119 1.14 1.04 1.02 1.00 1.00 1.00 1.01 1.02 1.02 1.01
6585 2.016 1.17 1.02 1.01 1.02 1.02 1.03 1.02 1.02 1.02 1.02
6890 1.926 1.35 1.07 1.00 0.99 1.00 1.01 1.01 1.00 1.00 1.01
7224 1.848 1.52 1.11 1.01 0.97 0.96 0.98 0.98 0.98 0.98 0.99
7498 1.778 1.80 1.22 1.03 0.97 0.95 0.94 0.95 0.95 0.95 0.96
7749 1.716 2.01 1.28 1.07 0.99 0.94 0.92 0.92 0.92 0.93 0.93
8026 1.660 2.31 1.41 1.13 1.01 0.95 0.92 0.90 0.89 0.89 0.89
8307 1.609 2.62 1.54 1.19 1.04 0.95 0.90 0.88 0.87 0.86 0.87
8589 1.562 2.94 1.69 1.29 1.10 1.00 0.93 0.89 0.86 0.85 0.85
8575 1.520 3.14 1.78 1.34 1.13 1.01 0.93 0.88 0.85 0.83 0.83
111610 overall 1.73 1.24 1.09 1.03 0.99 0.97 0.96 0.96 0.96 0.96
List of 33 reflections *NOT* obeying Wilson distribution (Z> 10.0)
h k l RES Z Intensity Sigma
72 11 61 1.52 17.34 0.2886E+06 0.2367E+05 "alien"
67 53 6 1.50 15.85 0.2638E+06 0.1128E+06 "alien"
35 10 25 3.17 14.39 0.2118E+08 0.2364E+06 "alien"
46 17 99 1.50 14.16 0.2357E+06 0.9588E+05 "alien"
34 32 2 2.75 13.44 0.1239E+08 0.1279E+06 "alien"
79 6 15 1.60 13.10 0.3117E+06 0.2477E+05 "alien"
61 20 33 1.88 12.54 0.8900E+06 0.3054E+05 "alien"
44 4 48 2.30 12.38 0.4695E+07 0.6072E+05 "alien"
66 25 19 1.79 11.89 0.5788E+06 0.2739E+05 "alien"
66 25 11 1.81 11.88 0.5781E+06 0.2771E+05 "alien"
60 43 61 1.50 11.77 0.1959E+06 0.9769E+05 "alien"
72 11 17 1.74 11.64 0.4278E+06 0.2619E+05 "alien"
80 24 26 1.50 11.41 0.1899E+06 0.9793E+05 "alien"
41 21 26 2.59 11.09 0.6988E+07 0.7945E+05 "alien"
44 18 20 2.59 11.08 0.6982E+07 0.7839E+05 "alien"
23 3 62 2.59 11.06 0.6971E+07 0.9154E+05 "alien"
69 7 22 1.80 11.06 0.5383E+06 0.2564E+05 "alien"
73 10 15 1.72 10.98 0.4036E+06 0.2356E+05 "alien"
70 17 35 1.68 10.96 0.3286E+06 0.2415E+05 "alien"
57 24 41 1.88 10.91 0.7746E+06 0.2842E+05 "alien"
82 24 6 1.50 10.74 0.1787E+06 0.1019E+06 "alien"
69 25 62 1.50 10.67 0.1775E+06 0.8689E+05 "alien"
24 20 44 2.91 10.45 0.9641E+07 0.1017E+06 "alien"
66 43 5 1.63 10.37 0.2468E+06 0.2294E+05 "alien"
81 4 29 1.53 10.36 0.1725E+06 0.2364E+05 "alien"
60 40 26 1.72 10.32 0.3792E+06 0.2578E+05 "alien"
39 18 57 2.18 10.24 0.3885E+07 0.5573E+05 "alien"
70 41 15 1.57 10.19 0.1922E+06 0.2281E+05 "alien"
55 36 41 1.79 10.16 0.4942E+06 0.2967E+05 "alien"
37 4 81 1.88 10.15 0.7202E+06 0.3357E+05 "alien"
56 27 5 2.06 10.14 0.1854E+07 0.3569E+05 "alien"
44 39 29 2.06 10.09 0.1844E+07 0.3805E+05 "alien"
65 46 29 1.56 10.06 0.1898E+06 0.2270E+05 "alien"
List of 33 reflections *NOT* obeying Wilson distribution (sorted by resolution)
Ice rings could occur at (Angstrom):
3.897,3.669,3.441, 2.671,2.249,2.072, 1.948,1.918,1.883,1.721
h k l RES Z Intensity Sigma
82 24 6 1.50 10.74 0.1787E+06 0.1019E+06
67 53 6 1.50 15.85 0.2638E+06 0.1128E+06
80 24 26 1.50 11.41 0.1899E+06 0.9793E+05
60 43 61 1.50 11.77 0.1959E+06 0.9769E+05
69 25 62 1.50 10.67 0.1775E+06 0.8689E+05
46 17 99 1.50 14.16 0.2357E+06 0.9588E+05
72 11 61 1.52 17.34 0.2886E+06 0.2367E+05
81 4 29 1.53 10.36 0.1725E+06 0.2364E+05
65 46 29 1.56 10.06 0.1898E+06 0.2270E+05
70 41 15 1.57 10.19 0.1922E+06 0.2281E+05
79 6 15 1.60 13.10 0.3117E+06 0.2477E+05
66 43 5 1.63 10.37 0.2468E+06 0.2294E+05
70 17 35 1.68 10.96 0.3286E+06 0.2415E+05
73 10 15 1.72 10.98 0.4036E+06 0.2356E+05
60 40 26 1.72 10.32 0.3792E+06 0.2578E+05
72 11 17 1.74 11.64 0.4278E+06 0.2619E+05
66 25 19 1.79 11.89 0.5788E+06 0.2739E+05
55 36 41 1.79 10.16 0.4942E+06 0.2967E+05
69 7 22 1.80 11.06 0.5383E+06 0.2564E+05
66 25 11 1.81 11.88 0.5781E+06 0.2771E+05
61 20 33 1.88 12.54 0.8900E+06 0.3054E+05
57 24 41 1.88 10.91 0.7746E+06 0.2842E+05
37 4 81 1.88 10.15 0.7202E+06 0.3357E+05
56 27 5 2.06 10.14 0.1854E+07 0.3569E+05
44 39 29 2.06 10.09 0.1844E+07 0.3805E+05
39 18 57 2.18 10.24 0.3885E+07 0.5573E+05
44 4 48 2.30 12.38 0.4695E+07 0.6072E+05
44 18 20 2.59 11.08 0.6982E+07 0.7839E+05
41 21 26 2.59 11.09 0.6988E+07 0.7945E+05
23 3 62 2.59 11.06 0.6971E+07 0.9154E+05
34 32 2 2.75 13.44 0.1239E+08 0.1279E+06
24 20 44 2.91 10.45 0.9641E+07 0.1017E+06
35 10 25 3.17 14.39 0.2118E+08 0.2364E+06
cpu time used by XSCALE 25.9 sec
elapsed wall-clock time 28.1 sec
I would like to extract the second last line where the 11th column has a number followed by an asterisk (xy.z*). E.g. in this table the line I'm looking for would contain "23.2*" from the 11th column (CC(1/2)). I would like the second last because the last would be the line that starts with total, and this was a lot easier to extract with a simple grep command.
So the expected output for the code in this case would be to print the line:
1.50 273432 20770 20893 99.4% 333.4% 342.1% 273340 0.80 346.9% 23.2* -1 0.644 19495
In a different file the second last value in the 11th with an asterisk after may correspond to 1.6 in the first column so the expected output would be:
1.60 216910 5769 5769 100.0% 207.5% 214.7% 216910 1.72 210.4% 26.0* -3 0.654 5204
And so on for all the different possible positions of the asterisk in the table.
I've tried using things like grep "[0-9, 0-9, ., 0-9*]" file.name and various other grep and fgrep things but I'm pretty new to this and can't get it to work.
Any help would be greatly appreciated.
Sam
GNU sed
(for your updated script)
sed -n '/LIMIT/,/=/{/^\s*\(\S*\s*\)\{10\}[0-9.-]*\*/H;x;s/^.*\n\(.*\n.*\)$/\1/;x;/=/{x;P;q}}' file
.. output is:
1.50 273432 20770 20893 99.4% 333.4% 342.1% 273340 0.80 346.9% 23.2* -1 0.644 19495
To print the entire second last line which matches that regex, you can do something like this:
awk '$11~/[0-9.]+\*/{secondlast=last;last=$0}END{print secondlast}' logFile
This one liner can do it:
$ awk '{if ($11 ~ /\*/) {i++; a[i]=$0}} END {print a[i -1]}' file
1.50 274090 20781 20874 99.6% 333.7% 341.9% 274015 0.80 347.1% 24.8* 0 0.645 19516
Explanation
It add to the array a[] all lines that contain * the 11th field. Then prints not the last but the previous one.
Update
Since your log is very big and asterisks appear all around, I update my code to:
$ awk '{if ($11 == /[0-9]*.[0-9]*\*/) {i++; a[i]=$0}} END {print a[i -1]}' a
0.90 0 0 147505 0.0% -99.9% -99.9% 0 -99.00 -99.9% 0.0 0 0.000 0
so it looks for lines with NNN.XXX* format.
awk '$11~/^[0-9.]+\*$/ {prev=val; val=$11+0} END {print prev}' log
I add 0 to the value of $11 to convert the string "23.2*" to the number 23.2.
Alternately, when I hear "nth from the end", I think: reverse it and take the nth from the top:
tac log | awk '$11~/^[0-9.]+\*$/ && ++n == 2 {print $11+0; exit}'