I have received this in my daily logwatch report. I do not know what this means. I am concerned that my server is about to fail. I do not know enough to know what this data means.
--------------------- Kernel Begin ------------------------
WARNING: Kernel Errors Present
res 41/01:00:90:20:6c/00:00:1b:00:00/40 Emask 0x401 (device error) <F> ...: 4 Time(s)
res 41/40:00:90:20:6c/00:00:1b:00:00/40 Emask 0x409 (media error) <F> ...: 1 Time(s)
res 51/01:00:90:20:6c/00:00:1b:00:00/e0 Emask 0x1 (device error) ...: 6 Time(s)
res 51/40:00:90:20:6c/00:00:1b:00:00/e0 Emask 0x9 (media error) ...: 7 Time(s)
ata1.00: NCQ disabled due to excessive errors ...: 1 Time(s)
ata1.00: error: { UNC } ...: 8 Time(s)
sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocat ...: 2 Time(s)
sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descr ...: 3 Time(s)
3 Time(s): 1b 6c 20 90
2 Time(s): 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
1 Time(s): 72 03 13 00 00 00 00 0c 00 0a 80 00 00 00 00 00
3 Time(s): Descriptor sense data with sense descriptors (in hex):
13 Time(s): ata1.00: cmd 25/00:08:90:20:6c/00:00:1b:00:00/e0 tag 0 dma 4096 in
5 Time(s): ata1.00: cmd 60/08:00:90:20:6c/00:00:1b:00:00/40 tag 0 ncq 4096 in
18 Time(s): ata1.00: configured for UDMA/133
13 Time(s): ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
5 Time(s): ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
13 Time(s): ata1.00: failed command: READ DMA EXT
5 Time(s): ata1.00: failed command: READ FPDMA QUEUED
13 Time(s): ata1.00: irq_stat 0x40000001
5 Time(s): ata1.00: irq_stat 0x40000008
18 Time(s): ata1.00: status: { DRDY ERR }
18 Time(s): ata1: EH complete
1 Time(s): sd 0:0:0:0: [sda] Add. Sense: Address mark not found for data field
3 Time(s): sd 0:0:0:0: [sda] CDB: Read(10): 28 00 1b 6c 20 90 00 00 08 00
3 Time(s): sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
3 Time(s): sd 0:0:0:0: [sda] Unhandled sense code
---------------------- Kernel End -------------------------
ATA could be the hard disks. I'd suggest to test them if possible. Or check the filesystem.
Related
As i know global variables(in userspace programs) are stored in coressponding DATA (and BSS) segment. But how and where are they stored in case of kernel code?
For an example let's look at one of Linux's global variable. Basically, Linux is an ELF image compressed with gzip or other compression tools. As stated on Wikipedia (https://en.wikipedia.org/wiki/Vmlinux):
Traditionally, when creating a bootable kernel image, the kernel is also compressed using gzip, or, since Linux 2.6.30,[3] using LZMA or bzip2, which requires a very small decompression stub to be included in the resulting image. The stub decompresses the kernel code, on some systems printing dots to the console to indicate progress, and then continues the boot process. Support for LZO,[4] xz[5] and LZ4[6] compression was added later.
On Ubuntu, there is a special script called extract-vmlinux which comes with your version of the Linux kernel (https://blog.packagecloud.io/eng/2016/03/08/how-to-extract-and-disassmble-a-linux-kernel-image-vmlinuz/). The Linux ELF file is /boot/vmlinuz. To decompress it, you must use the extract-vmlinux script which can be installed with
sudo apt-get install linux-headers-$(uname -r)
The script is safe to use because it is a standard script which is found in the official repository of packages for your distribution (most packages there are open source and safe). Now use
sudo /usr/src/kernels/$(uname -r)/scripts/extract-vmlinux /boot/vmlinuz-$(uname -r) > vmlinux
to extract the Linux kernel's ELF file from the compressed vmlinux image to your home repository. Now we can look at the ELF file of the Linux kernel. Use readelf -a vmlinux to have a summary of the content. The output for me is this:
user#user-System-Product-Name:~$ readelf -a vmlinux
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x1000000
Start of program headers: 64 (bytes into file)
Start of section headers: 46137728 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 5
Size of section headers: 64 (bytes)
Number of section headers: 35
Section header string table index: 34
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .text PROGBITS ffffffff81000000 00200000
0000000000e025f7 0000000000000000 AX 0 0 4096
[ 2] .rodata PROGBITS ffffffff82000000 01200000
00000000004e46f2 0000000000000000 WA 0 0 4096
[ 3] .pci_fixup PROGBITS ffffffff824e4700 016e4700
00000000000032b0 0000000000000000 A 0 0 16
[ 4] .tracedata PROGBITS ffffffff824e79b0 016e79b0
0000000000000078 0000000000000000 A 0 0 1
[ 5] __ksymtab PROGBITS ffffffff824e7a28 016e7a28
0000000000010a34 0000000000000000 A 0 0 4
[ 6] __ksymtab_gpl PROGBITS ffffffff824f845c 016f845c
00000000000121ec 0000000000000000 A 0 0 4
[ 7] __kcrctab PROGBITS ffffffff8250a648 0170a648
00000000000058bc 0000000000000000 A 0 0 4
[ 8] __kcrctab_gpl PROGBITS ffffffff8250ff04 0170ff04
00000000000060a4 0000000000000000 A 0 0 4
[ 9] __ksymtab_strings PROGBITS ffffffff82515fa8 01715fa8
0000000000037bea 0000000000000001 AMS 0 0 1
[10] __init_rodata PROGBITS ffffffff8254dba0 0174dba0
00000000000002a8 0000000000000000 A 0 0 32
[11] __param PROGBITS ffffffff8254de48 0174de48
00000000000038e0 0000000000000000 A 0 0 8
[12] __modver PROGBITS ffffffff82551728 01751728
00000000000005a0 0000000000000000 WA 0 0 8
[13] __ex_table PROGBITS ffffffff82551cd0 01751cd0
0000000000001af4 0000000000000000 A 0 0 4
[14] .notes NOTE ffffffff825537c4 017537c4
00000000000001ec 0000000000000000 A 0 0 4
[15] .data PROGBITS ffffffff82600000 01800000
00000000003657c0 0000000000000000 WA 0 0 8192
[16] __bug_table PROGBITS ffffffff829657c0 01b657c0
00000000000178d4 0000000000000000 WA 0 0 1
[17] .vvar PROGBITS ffffffff8297e000 01b7e000
0000000000001000 0000000000000000 WA 0 0 16
[18] .data..percpu PROGBITS 0000000000000000 01c00000
000000000002f000 0000000000000000 WA 0 0 4096
[19] .init.text PROGBITS ffffffff829ae000 01dae000
00000000000772e5 0000000000000000 AX 0 0 16
[20] .altinstr_aux PROGBITS ffffffff82a252e5 01e252e5
0000000000002d9b 0000000000000000 AX 0 0 1
[21] .init.data PROGBITS ffffffff82a2a000 01e2a000
00000000001c74f0 0000000000000000 WA 0 0 8192
[22] .x86_cpu_dev.init PROGBITS ffffffff82bf14f0 01ff14f0
0000000000000028 0000000000000000 A 0 0 8
[23] .parainstructions PROGBITS ffffffff82bf1518 01ff1518
0000000000022b7c 0000000000000000 A 0 0 8
[24] .altinstructions PROGBITS ffffffff82c14098 02014098
0000000000006660 0000000000000000 A 0 0 1
[25] .altinstr_replace PROGBITS ffffffff82c1a6f8 0201a6f8
0000000000001789 0000000000000000 AX 0 0 1
[26] .iommu_table PROGBITS ffffffff82c1be88 0201be88
00000000000000f0 0000000000000000 A 0 0 8
[27] .apicdrivers PROGBITS ffffffff82c1bf78 0201bf78
0000000000000040 0000000000000000 WA 0 0 8
[28] .exit.text PROGBITS ffffffff82c1bfb8 0201bfb8
0000000000001f4b 0000000000000000 AX 0 0 1
[29] .smp_locks PROGBITS ffffffff82c1e000 0201e000
000000000000a000 0000000000000000 A 0 0 4
[30] .data_nosave PROGBITS ffffffff82c28000 02028000
0000000000001000 0000000000000000 WA 0 0 4
[31] .bss NOBITS ffffffff82c29000 02029000
00000000005d7000 0000000000000000 WA 0 0 4096
[32] .brk NOBITS ffffffff83200000 02029000
000000000002c000 0000000000000000 WA 0 0 1
[33] .init.scratch PROGBITS ffffffff83400000 02800000
0000000000400000 0000000000000000 WA 0 0 32
[34] .shstrtab STRTAB 0000000000000000 02c00000
0000000000000180 0000000000000000 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
l (large), p (processor specific)
There are no section groups in this file.
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000200000 0xffffffff81000000 0x0000000001000000
0x00000000015539b0 0x00000000015539b0 R E 0x200000
LOAD 0x0000000001800000 0xffffffff82600000 0x0000000002600000
0x000000000037f000 0x000000000037f000 RW 0x200000
LOAD 0x0000000001c00000 0x0000000000000000 0x000000000297f000
0x000000000002f000 0x000000000002f000 RW 0x200000
LOAD 0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
0x0000000000e52000 0x0000000000e52000 RWE 0x200000
NOTE 0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
0x00000000000001ec 0x00000000000001ec 0x4
Section to Segment mapping:
Segment Sections...
00 .text .rodata .pci_fixup .tracedata __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __init_rodata __param __modver __ex_table .notes
01 .data __bug_table .vvar
02 .data..percpu
03 .init.text .altinstr_aux .init.data .x86_cpu_dev.init .parainstructions .altinstructions .altinstr_replacement .iommu_table .apicdrivers .exit.text .smp_locks .data_nosave .bss .brk .init.scratch
04 .notes
There is no dynamic section in this file.
There are no relocations in this file.
The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.
No version information found in this file.
Displaying notes found in: .notes
Owner Data size Description
Xen 0x00000006 Unknown note type: (0x00000006)
description data: 6c 69 6e 75 78 00
Xen 0x00000004 Unknown note type: (0x00000007)
description data: 32 2e 36 00
Xen 0x00000008 Unknown note type: (0x00000005)
description data: 78 65 6e 2d 33 2e 30 00
Xen 0x00000008 Unknown note type: (0x00000003)
description data: 00 00 00 80 ff ff ff ff
Xen 0x00000008 Unknown note type: (0x0000000f)
description data: 00 00 00 00 80 00 00 00
Xen 0x00000008 NT_VERSION (version)
description data: c0 e1 9a 82 ff ff ff ff
Xen 0x00000008 NT_ARCH (architecture)
description data: 00 20 00 81 ff ff ff ff
Xen 0x00000029 Unknown note type: (0x0000000a)
description data: 21 77 72 69 74 61 62 6c 65 5f 70 61 67 65 5f 74 61 62 6c 65 73 7c 70 61 65 5f 70 67 64 69 72 5f 61 62 6f 76 65 5f 34 67 62
Xen 0x00000004 Unknown note type: (0x00000011)
description data: 01 88 00 00
Xen 0x00000004 Unknown note type: (0x00000009)
description data: 79 65 73 00
Xen 0x00000008 Unknown note type: (0x00000008)
description data: 67 65 6e 65 72 69 63 00
Xen 0x00000010 Unknown note type: (0x0000000d)
description data: 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
Xen 0x00000004 Unknown note type: (0x0000000e)
description data: 01 00 00 00
Xen 0x00000004 Unknown note type: (0x00000010)
description data: 01 00 00 00
Xen 0x00000008 Unknown note type: (0x0000000c)
description data: 00 00 00 00 00 80 ff ff
Xen 0x00000008 Unknown note type: (0x00000004)
description data: 00 00 00 00 00 00 00 00
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: c0263a3075bc0a9388365ddf35ab5422da3356a9
Linux 0x00000001 OPEN
description data: 00
Xen 0x00000008 Unknown note type: (0x00000012)
description data: 40 07 00 01 00 00 00 00
The important section for the question is the Program headers section:
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000200000 0xffffffff81000000 0x0000000001000000
0x00000000015539b0 0x00000000015539b0 R E 0x200000
LOAD 0x0000000001800000 0xffffffff82600000 0x0000000002600000
0x000000000037f000 0x000000000037f000 RW 0x200000
LOAD 0x0000000001c00000 0x0000000000000000 0x000000000297f000
0x000000000002f000 0x000000000002f000 RW 0x200000
LOAD 0x0000000001dae000 0xffffffff829ae000 0x00000000029ae000
0x0000000000e52000 0x0000000000e52000 RWE 0x200000
NOTE 0x00000000017537c4 0xffffffff825537c4 0x00000000025537c4
0x00000000000001ec 0x00000000000001ec 0x4
I would guess that the RW sections are the data/bss segments of the static and freestanding kernel image. There isn't much bss data (uninitialized data) in this kernel image. You can see that by looking at the filesize versus the memsize of the different segments. In bss data, only the size is written in the executable. The loader will then assign zeroes to these segments. Since we have only memsizes which are equal to the filesizes, then we don't have much bss here.
Also, we can look at the different virtual addresses at which the kernel will load itself after decompression. This will be useful to find the segment in which a certain global variable resides.
Now, to determine where a global variable is loaded one can look at the System.map file in the /boot directory. For example, we can take a random global variable in kernel/sched/core.c called sysctl_sched_rt_period and find its address using
user#user-System-Product-Name:~$ sudo grep "sysctl_sched_rt_period" /boot/System.map-$(uname -r)
ffffffff8266eb24 D sysctl_sched_rt_period
The address is thus ffffffff8266eb24. This address resides in the RW segment loaded at ffffffff82600000 because the size of this segment is 37f000 and because ffffffff82600000 + 37f000 > ffffffff8266eb24. You can clearly see that this global variable is stored in the data segment of the kernel's executable.
I'm trying to analyze a TLS library for Java (not Bouncy Castle). For some reason, Wireshark refuses to recognize a ClientHello fragment as such. Here is a hex dump of the TCP packet. I have broken it up into it's component parts to make it a bit easier to follow.
TCP stuff
020000004502008a000040004006f9ce0aa5001e5db8d822e28d01bbf3b9a9ab8c64a4fe8018081070fa00000101080a4e4e5d5845aa256f
16 TLS handshake record
0301 TLS v 1.0
0052 Fragment length 82 bytes
01 ClientHello message
00004e Message length 78 bytes
0303 TLS v 1.2
Client random
3c88c697bf2b559cc032faff7caccb17475ae76f36ededf279a9d1b9950e7367
00 Session ID length
0024 Cipher suite length 36 bytes 18 cipher suites
1301 1302 1303 c02b c02f cca9 ccaa c02c c030 c00a c009 c013 c014 009c 009d 002f 0035 000a
01 Compression methods length
00 Compression method null
0000 Extensions length
I'm at a loss to understand why Wireshark and several websites don't recognize this as a valid handshake record. Any help is appreciated.
EDIT:
I re-ran the capture to show how Wireshark interprets the network header.
The raw packet
0000 0c ea c9 75 61 30 3c 22 fb 01 07 f3 08 00 45 00
0010 00 8a 00 00 40 00 40 06 42 23 c0 a8 01 c8 5d b8
0020 d8 22 e9 3b 01 bb ee e9 99 55 53 56 a2 a8 80 18
0030 08 0a dc d0 00 00 01 01 08 0a 4e d6 03 2b db 4f
0040 3f fb 16 03 01 00 52 01 00 00 4e 03 03 8a 80 b9
0050 8c 73 ee 40 30 ef 65 1d 8c 51 d2 39 09 34 17 79
0060 d1 af e1 63 96 1a ac b4 ad 96 03 dd 7a 00 00 24
0070 13 01 13 02 13 03 c0 2b c0 2f cc a9 cc aa c0 2c
0080 c0 30 c0 0a c0 09 c0 13 c0 14 00 9c 00 9d 00 2f
0090 00 35 00 0a 01 00 00 00
The Wireshark interpretation
Frame 3616: 152 bytes on wire (1216 bits), 152 bytes captured (1216 bits) on interface en0, id 0
Interface id: 0 (en0)
Encapsulation type: Ethernet (1)
Arrival Time: Nov 3, 2020 09:03:23.957190000 EST
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1604412203.957190000 seconds
[Time delta from previous captured frame: 0.004793000 seconds]
[Time delta from previous displayed frame: 0.011314000 seconds]
[Time since reference or first frame: 49.409611000 seconds]
Frame Number: 3616
Frame Length: 152 bytes (1216 bits)
Capture Length: 152 bytes (1216 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp]
[Coloring Rule Name: TCP]
[Coloring Rule String: tcp]
Ethernet II, Src: Apple_01:07:f3 (3c:22:fb:01:07:f3), Dst: ARRISGro_75:61:30 (0c:ea:c9:75:61:30)
Destination: ARRISGro_75:61:30 (0c:ea:c9:75:61:30)
Source: Apple_01:07:f3 (3c:22:fb:01:07:f3)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.1.200, Dst: 93.184.216.34
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 138
Identification: 0x0000 (0)
Flags: 0x40, Don't fragment
Fragment Offset: 0
Time to Live: 64
Protocol: TCP (6)
Header Checksum: 0x4223 [validation disabled]
[Header checksum status: Unverified]
Source Address: 192.168.1.200
Destination Address: 93.184.216.34
Transmission Control Protocol, Src Port: 59707, Dst Port: 443, Seq: 1, Ack: 1, Len: 86
Source Port: 59707
Destination Port: 443
[Stream index: 18]
[TCP Segment Len: 86]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 4008286549
[Next Sequence Number: 87 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 1398186664
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 2058
[Calculated window size: 131712]
[Window size scaling factor: 64]
Checksum: 0xdcd0 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
[Timestamps]
TCP payload (86 bytes)
TCP segment data (86 bytes)
I'm trying to read an XRef stream object but something doesn't adds up. This is my object:
<<
/DecodeParms << /Columns 5/Predictor 12 >>
/Filter /FlateDecode
/ID [<9597C618BC90AFA4A078CA72B2DD061C><48726007F483D547A8BEFF6E9CDA072F>]
/Index [124332 848]
/Info 124331 0 R
/Length 137
/Prev 8983958
/Root 124333 0 R
/Size 125180
/Type /XRef
/W [1 3 1]
>>
I read the 137 bytes of stream, uncompress them through zlib and I get 5088 bytes. This is the beginning of the uncompressed stream (hexdump -C output):
00000000 02 01 00 00 10 00 02 00 00 27 ec 00 02 00 00 01 |.........'......|
00000010 f4 00 02 00 00 01 f7 00 02 00 00 04 5b 00 02 00 |............[...|
00000020 00 02 68 00 02 00 00 0b ac 00 02 00 00 0f e5 00 |..h.............|
00000030 02 00 00 0e 93 00 02 00 00 0d 14 00 02 00 00 0d |................|
What I don't understand is that I should have 5 bytes per entry: /W [1 3 1] means 1+3+1=5 bytes; but the stream's length of 5088 isn't divisible by 5. Also, I realized that 5088 is divisible by 6: 5088/6=848 and that's the number of entry as the second value of the /Index key confirms. Reading the stream keeping the [1 3 1] scheme is, also, impossible already at the second entry (the byte 0xEC isn't a valid entry type).
Where's my mistake?
Thanks a lot for any help.
After decompression each line has 6 bytes: 1 byte for predictor, 5 for 'predicted' data.
After you apply the predictor you get the actual data.
We have 5 customers running the same WebService from Domino
This weekend we updated the customers servers with Domino 9.01. FP2 and the Poodle fixpack to be able to run TLS 1.0 incomming and outgoing.
4 Customers works perfect
1 Customer gets SSL errors for the outgoing Webservice (same errors as before we updated the servers), the incomming is working for TLS so we guess the updates for Poodle have worked as intended.
After setting som DEBUG_SSL parameters for one working and the failing server we got this logs
The failing row is
S_Read> nti_done return 0 bytes rc = 9
instead of intended
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 00 2E
I have searched google and nothing is there to understand what is missing
My guess is there is some problem with negotiating the cipher, but why and what to do for solving this matter.
I know there is some smart people out there ;-)
Log from failing server handshake
int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
SSLAdvanceHandshake Enter> Processed : SSL_hello_request
SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
S_Write> Enter len = 58
SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 85 B7 4D '....5...1..T%.7M'
SSL_Xmt> 00000010: 15 80 11 80 C7 47 4D 1D 1D B1 89 5F F6 94 18 73 '....GGM..1._v..s'
SSL_Xmt> 00000020: C6 D3 7D 6A 15 92 A9 57 48 19 32 00 00 0A 00 2F 'FS}j..)WH.2..../'
SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00 '.5........'
S_Write> Switching Endpoint to sync
S_Write> Posting a nti_snd for 58 bytes
SSL_EncryptData> SSL not init exit
S_Write> Switching Endpoint to async
SSL_EncryptDataCleanup> SSL not init exit
S_Write> nti_done return 58 bytes rc = 0
S_Write> Exit, wrote 58 bytes
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 0 bytes rc = 9
S_Read> nti_done return 0 bytes rc = 9 Event = 0x100
SSLSendAlert> Sending an alert of 0x0 (close_notify) level 0x2 (fatal)
SSL_Handshake> Changing SSL status from -6989 to -5000 to flush write queue
SSL_Handshake> After handshake state= 2 Status= -5000
SSL_Handshake> Exit Status = -5000
int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
...
Log from working server handshake
int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
SSLAdvanceHandshake Enter> Processed : SSL_hello_request
SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
S_Write> Enter len = 58
SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 89 B3 A0 '....5...1..T%.3 '
SSL_Xmt> 00000010: 2B 75 D1 E9 D4 81 87 C3 5D 91 45 84 6A E2 47 9D '+uQiT..C].E.jbG.'
SSL_Xmt> 00000020: 76 BE 14 A8 A6 10 1C 06 FB 7D 8B 00 00 0A 00 2F 'v>.(&...{}...../'
SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00 '.5........'
S_Write> Switching Endpoint to sync
S_Write> Posting a nti_snd for 58 bytes
SSL_EncryptData> SSL not init exit
S_Write> Switching Endpoint to async
SSL_EncryptDataCleanup> SSL not init exit
S_Write> nti_done return 58 bytes rc = 0
S_Write> Exit, wrote 58 bytes
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 00 2E '.....'
S_Read> Exit, read 5 bytes
S_Read> Enter len = 46
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 46 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 46 bytes rc = 0
SSL_RCV> 00000000: 02 00 00 2A 03 01 54 7C 9D 24 4C B4 AD 62 4E 35 '...*..T|.$L4-bN5'
SSL_RCV> 00000010: 4C C3 B4 AB 34 6D 7D CB 8F 6B CC 80 00 FE 4C 4A 'LC4+4m}K.kL..~LJ'
SSL_RCV> 00000020: 77 87 CD 2E DF 98 04 10 13 29 0B 00 2F 00 'w.M._....)../.'
S_Read> Exit, read 46 bytes
SSLProcessProtocolMessage> Record Content: 22
SSLProcessHandshakeMessage Enter> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 0 Cipher: 0x0000 (Unknown Cipher)
SSLProcessHandshakeMessage Enter> Message: SSL_server_hello
SSLProcessServerHello> Server chose SSL/TLS version TLS1.0 (0x0301)
SSLProcessHandshakeMessage Exit> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 1 Cipher: 0x002F (RSA_WITH_AES_128_CBC_SHA)
SSLAdvanceHandshake Enter> Processed : 2 State: 5 (HandshakeServerHello)
SSLAdvanceHandshake Enter> Processed : SSL_server_hello
SSLAdvanceHandshake Exit> State : 8 (HandshakeCertificate)
SSL_Handshake> After handshake state= 8 Status= -5000
SSL_Handshake> Exit Status = -5000
int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
SSL_Handshake> Enter
SSL_Handshake> Current Cipher 0x002F (RSA_WITH_AES_128_CBC_SHA)
S_Read> Enter len = 5
S_Read> Switching Endpoint to sync
S_Read> Posting a nti_rcv for 5 bytes
SSL_RcvSetup> SSL not init exit
S_Read> Switching Endpoint to async
S_Read> nti_done return 5 bytes rc = 0
SSL_RCV> 00000000: 16 03 01 0E 9D '.....'
S_Read> Exit, read 5 bytes
S_Read> Enter len = 3741
....
/Stefan
PS: Here is the Java errors that come sfter the hand shake error
Error connecting to 'xxxxx' on port '443', SSL IO error. Remote session no longer responding.
at lotus.domino.axis.InternalFault.makeFault(Unknown Source)
at lotus.domino.axis.transport.http.HTTPSender.invoke(Unknown Source)
at lotus.domino.axis.strategies.InvocationStrategy.visit(Unknown Source)
at lotus.domino.axis.SimpleChain.doVisiting(Unknown Source)
at lotus.domino.axis.SimpleChain.invoke(Unknown Source)
at lotus.domino.axis.client.AxisClient.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invokeEngine(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.websvc.client.Call.invoke(Unknown Source)
I'm trying to interpret a system crash that is repeatable, but but not that often. I believe it has do with a third party driver and a optical scanning device. I'm not sure if the driver for scanning device is causing the hang if it's bringing something out in a USB driver that causes the hang or if it's something totally different.
I'm not a driver developer, so I'm fumbling my way through the crash dump while reading the Linux Kernel Crash Book located here - http://rogunix.com/docs/Reversing&Exploiting/Linux_Kernel_Crash_Book.pdf .
I was able to setup a debug kernel with the kexec tools to produce a vmcore. With crash is see the PANIC: "Oops: 0000 [#1]" (check log for details) and in the logs is see the BUG: unable to handle kernel paging request at virtual address 00100100.
Can you help me interpret more of the dump so I can go back to the developer of the driver and tell them the findings or can you tell me is not the driver at all.
Here's more from the dump. Many thanks in advance.
KERNEL: /usr/lib/debug/lib/modules/2.6.18-194.el5debug/vmlinux
DUMPFILE: /root/kernel/vmcore_12032014
CPUS: 4
DATE: Sat Oct 20 16:51:55 2001
UPTIME: 00:03:55
LOAD AVERAGE: 0.79, 0.83, 0.38
TASKS: 128
NODENAME: 3232A
RELEASE: 2.6.18-194.el5debug
VERSION: #1 SMP Fri Apr 2 16:56:41 EDT 2010
MACHINE: i686 (1800 Mhz)
MEMORY: 2 GB
PANIC: "Oops: 0000 [#1]" (check log for details)
PID: 0
COMMAND: "swapper"
TASK: c069d460 (1 of 4) [THREAD_INFO: c075f000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 0 TASK: c069d460 CPU: 0 COMMAND: "swapper"
#0 [c080cdb0] crash_kexec at c044a386
#1 [c080cdf4] die at c04065c3
#2 [c080ce24] do_page_fault at c062d1b7
#3 [c080ce5c] error_code (via page_fault) at c0405bc3
EAX: f7c7de2c EBX: 000ffaec ECX: 00000000 EDX: f4282e8c EBP: f7c665a0
DS: 007b ESI: f7c7ddf8 ES: 007b EDI: c080cea4
CS: 0060 EIP: c05a2ef2 ERR: ffffffff EFLAGS: 00010012
#4 [c080ce90] hiddev_send_event at c05a2ef2
#5 [c080cea0] hiddev_report_event at c05a2fcc
#6 [c080cec8] hid_input_report at c05a0fc3
#7 [c080cf1c] hid_irq_in at c05a261d
#8 [c080cf2c] usb_hcd_giveback_urb at c05964b6
#9 [c080cf3c] uhci_giveback_urb at f883f618 [uhci_hcd]
#10 [c080cf58] uhci_scan_schedule at f883fc14 [uhci_hcd]
#11 [c080cfa4] uhci_irq at f884163d [uhci_hcd]
#12 [c080cfc4] usb_hcd_irq at c0596f9b
#13 [c080cfcc] handle_IRQ_event at c0454d5f
#14 [c080cfe4] __do_IRQ at c0454e2d
--- <hard IRQ> ---
#0 [c075ff80] do_IRQ at c040761f
#1 [c075ff98] common_interrupt at c0405a28
EAX: 00000000 EBX: c075f008 ECX: 00000000 EDX: 00000000 EBP: 00000020
DS: 007b ESI: c06440f8 ES: 007b EDI: ca138790
CS: 0060 EIP: c0403d0a ERR: ffffff4e EFLAGS: 00000246
#2 [c075ffcc] mwait_idle at c0403d0a
#3 [c075ffd4] cpu_idle at c0403cc4
crash> log
Scan: First buffer received = 0x6
Scan: Last packet of command transfer = 6 Command size 6
Scan: 0x2 0x51 0x88 0x80 0x81 0x3
Scan: Read data present
Scan: IOCTL_PS3_PEAK_MSG
Scan: In scanner_read Scan: Copied 1 chunks of data to user space
usb 4-1: new full speed USB device using uhci_hcd and address 3
usb 4-1: configuration #1 chosen from 1 choice
pl2303 4-1:1.0: pl2303 converter detected
usb 4-1: pl2303 converter now attached to ttyUSB0
BUG: unable to handle kernel paging request at virtual address 00100100
printing eip:
c05a2ef2
*pde = 76330067
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:1d.2/usb4/4-1/bConfigurationValue
Modules linked in: vfat fat scan_drv_scanner(U) xt_limit xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables loop dm_mirror dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac lp joydev snd_hda_intel snd_seq_dummy sg snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc parport_pc serio_raw snd_hwdep parport pl2303 snd soundcore i2c_i801 usbserial r8169 i2c_core mii pcspkr dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
CPU: 0
EIP: 0060:[<c05a2ef2>] Tainted: G VLI
EFLAGS: 00010012 (2.6.18-194.el5debug #1)
EIP is at hiddev_send_event+0x65/0x95
eax: f7c7de2c ebx: 000ffaec ecx: 00000000 edx: f4282e8c
esi: f7c7ddf8 edi: c080cea4 ebp: f7c665a0 esp: c080ce94
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, ti=c080c000 task=c069d460 task.ti=c075f000)
Stack: f7cbba78 ca3ec000 c080cebc c05a2fd1 00000001 00000001 ffffffff 00000000
00000000 00000000 00000005 f7cc857c c075ff9c c05a0fc8 f7c57348 00000002
00000002 00000046 c06cbfb0 00000001 ca3ec000 f7cca001 f7cbba78 00000046
Call Trace:
[<c05a2fd1>] hiddev_report_event+0x4b/0x52
[<c05a0fc8>] hid_input_report+0x99/0x399
[<c05a2622>] hid_irq_in+0x49/0xcc
[<c05964b9>] usb_hcd_giveback_urb+0x28/0x53
[<f883f61d>] uhci_giveback_urb+0x104/0x12b [uhci_hcd]
[<f883fc19>] uhci_scan_schedule+0x4f0/0x76b [uhci_hcd]
[<f8841642>] uhci_irq+0x118/0x12e [uhci_hcd]
[<c0596f9e>] usb_hcd_irq+0x23/0x50
[<c0454d61>] handle_IRQ_event+0x3e/0x8a
[<c0454e32>] __do_IRQ+0x85/0xda
[<c0454dad>] __do_IRQ+0x0/0xda
[<c040761f>] do_IRQ+0xa4/0xd1
[<c0405a2d>] common_interrupt+0x25/0x2c
[<c0403d0a>] mwait_idle+0x2a/0x3d
[<c0403cc6>] cpu_idle+0xa4/0xbe
[<c0764a22>] start_kernel+0x39e/0x3a6
=======================
Code: b9 01 00 02 00 ba 1d 00 00 00 40 83 e0 3f 89 83 00 06 00 00 8d 83 0c 06 00 00 e8 93 ac ee ff 8b 9b 14 06 00 00 81 eb 14 06 00 00 <8b> 83 14 06 00 00 0f 18 00 90 8d 93 14 06 00 00 8d 46 34 39 c2
EIP: [<c05a2ef2>] hiddev_send_event+0x65/0x95 SS:ESP 0068:c080ce94
Updated Kernel Tested
I have since updated to the latest CentOS 5.5 released kernel (2.6.18-371.6.1), but still got a kernel panic. However this time the panic occured on the command "khubd" and not on "swapper". I was able to capture the dump and below I have provided what I hope is relevant information to pinpoint this problem. Is it a bug in the kernel or something in the driver?
KERNEL: /usr/lib/debug/lib/modules/2.6.18-371.6.1.el5debug/vmlinux
DUMPFILE: /root/kernel2/HWID-2277-31032014/vmcore
CPUS: 4
DATE: Mon Mar 31 12:50:25 2014
UPTIME: 2 days, 19:29:28
LOAD AVERAGE: 0.07, 0.09, 0.02
TASKS: 181
NODENAME: 32277
RELEASE: 2.6.18-371.6.1.el5debug
VERSION: #1 SMP Wed Mar 12 22:29:36 EDT 2014
MACHINE: i686 (1800 Mhz)
MEMORY: 2 GB
PANIC: "Oops: 0000 [#1]" (check log for details)
PID: 126
COMMAND: "khubd"
TASK: ca2fc5b0 [THREAD_INFO: f7fb2000]
CPU: 3
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 126 TASK: ca2fc5b0 CPU: 3 COMMAND: "khubd"
#0 [f7fb2dfc] crash_kexec at c044da51
#1 [f7fb2e40] die at c0406649
#2 [f7fb2e70] do_page_fault at c0639449
#3 [f7fb2eb0] device_del at c057053e
#4 [f7fb2efc] device_del at c0570552
#5 [f7fb2f10] device_unregister at c0570669
#6 [f7fb2f18] usb_remove_ep_files at c05a3dac
#7 [f7fb2f38] usb_remove_sysfs_dev_files at c05a39d6
#8 [f7fb2f44] usb_disconnect at c059db59
#9 [f7fb2f58] hub_thread at c059e663
#10 [f7fb2fcc] kthread at c0439ef6
#11 [f7fb2fe4] kernel_thread_helper at c0405e0d
crash> log (truncated)
Scan: no error - In command_callback
Scan: In command_callback
Scan: First buffer received = 0x2a
Scan: Last packet of command transfer = 42 Command size 42
Scan: 0x2 0x44 0x20 0x54 0x65 0x73 0x74 0x20 0x4d 0x65 0x73 0x73
Scan: Read data present
Scan: IOCTL_PS3_PEAK_MSG
Scan: In scanner_read Scan: Copied 1 chunks of data to user space
Scan: IOCTL_PS3_SCANNER_COMMAND Busy slots = 0x0 Total_slots = 0x8 Busy image slots = 0x0
Scan: 0x2 0x38 0x3 0x4
Scan: In write_callback status = 0x0
Scan: In image_callback
Scan: Urb status = 0xffffffb9 Urb length = 0x0
Scan: In image_callback_fun - lost connection to scanner.
Scan: urb set to re-submit
Scan: In command_callback
Scan: Urb status = 0xffffffb9 Urb length = 0x0
Scan: In command_callback_fun - lost connection to scanner.
Scan: urb set to re-submit
usb 1-6: USB disconnect, address 11
BUG: unable to handle kernel paging request at virtual address 6b6b6b93
printing eip:
c04bd310
*pde = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:1d.7/usb1/1-6/bConfigurationValue
Modules linked in: vfat fat pdi_ps3_drv_scanner(U) xt_limit xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables loop dm_mirror dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac lp joydev snd_hda_intel snd_seq_dummy sg snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer parport_pc snd_page_alloc snd_hwdep snd parport tpm_tis tpm soundcore r8169 tpm_bios i2c_i801 serio_raw pcspkr pl2303 usbserial mii i2c_core dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
CPU: 3
EIP: 0060:[<c04bd310>] Tainted: G -------------------- VLI
EFLAGS: 00010202 (2.6.18-371.6.1.el5debug #1)
EIP is at sysfs_hash_and_remove+0x18/0x103
eax: 6b6b6b6b ebx: f4be5bf4 ecx: c0570543 edx: 6b6b6b6b
esi: f4be5b0c edi: f3cd936c ebp: f3cd93c4 esp: f7fb2ee0
ds: 007b es: 007b ss: 0068
Process khubd (pid: 126, ti=f7fb2000 task=ca2fc5b0 task.ti=f7fb2000)
Stack: f4be5c50 6b6b6b6b f764d16c f4be5bf4 f4be5b0c f3cd936c f3cd93c4 c0570557
f4be5b0c f3cd93a0 f3cd936c f7c8da2c c057066e f7fb2f28 c05a3db1 f7fb2f28
c06827a6 00000000 305f7065 c06d0030 f3cd936c f3cd93c4 c05a39db f3cd96cc
Call Trace:
[<c0570557>] device_del+0x5c/0x16b
[<c057066e>] device_unregister+0x8/0x10
[<c05a3db1>] usb_remove_ep_files+0x53/0x75
[<c05a39db>] usb_remove_sysfs_dev_files+0xf/0x6c
[<c059db5e>] usb_disconnect+0xa7/0xe0
[<c059e668>] hub_thread+0x327/0x987
[<c0439ed9>] kthread+0xa1/0xec
[<c0439fab>] autoremove_wake_function+0x0/0x2d
[<c059e341>] hub_thread+0x0/0x987
[<c0439ef8>] kthread+0xc0/0xec
[<c0439e38>] kthread+0x0/0xec
[<c0405e0f>] kernel_thread_helper+0x7/0x10
=======================
Code: 8b 40 24 8b 40 40 c3 8b 40 14 8b 00 c3 8b 40 14 8b 00 c3 55 57 56 53 83 ec 0c 85 c0 89 44 24 04 89 14 24 0f 84 e5 00 00 00 89 c2 <8b> 40 28 85 c0 0f 84 d8 00 00 00 8b 52 70 05 90 00 00 00 89 54
EIP: [<c04bd310>] sysfs_hash_and_remove+0x18/0x103 SS:ESP 0068:f7fb2ee0