Multi GPU passthrough failed - gpu

My server has 4 PCI slots, their addresses are 0000:04:00.0, 0000:05:00.0, 0000:41:00.0, 0000:42:00.0.
I want to pass 4 GPUs to virtual machines via vfio, each of the VMs has one GPU.
VMs worked fine when I passed 0000:04:00.0 and 0000:05:00.0 to VMs or passed 0000:41:00.0 and 0000:42:00.0 to VMs.
But VMs which used 0000:41:00.0 and 0000:42:00.0 had no output to the screen and I could not ping VM's ip when I passed 0000:04:00, 0000:41:00.0 and 0000:42:00.0 to VMs, but the VM which used 0000:04:00 worked fine.
I got the same result when I passed 0000:05:00, 0000:41:00.0 and 0000:42:00.0 to VMs: VM with 0000:05:00 worked fine and VMs which used 0000:41:00.0 and 0000:42:00.0 hanged.
When I define 0000:04:00.0 and 0000:05:00.0 as group A, 0000:41:00.0 and 0000:42:00.0 as group B, the result is:
VMs work fine When I just pass group A or group B to vms.
Just group A works fine when the gpus that I pass to vms contain group A and B.
In case 2, VMs work fine when I used '-vga cirrus' instead of '-vga none'. But I get code 12 from graphics card in Device Manager of windows 7. And this is not what I want, I just want to use the graphics card in the host, so I should use '-vga none'.
It also worked fine when I remove VGA passthrough or just pass 0000:41:00.1 (the HDMI audio of graphics card) to VM.
I got no error output in qemu monitor.
How can I solve this probem?
Version of qemu:
QEMU emulator version 2.5.1.1, Copyright (c) 2003-2008 Fabrice Bellard
Qemu command:
/root/qemu25/qemu-system-x86_64 \
-name gputest_41 \
-machine q35,accel=kvm,usb=off,smm=off \
-cpu host \
-m 4096 \
-realtime mlock=off \
-smp 4,sockets=1,cores=4,threads=1 \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-no-shutdown \
-boot strict=on \
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \
-drive file=/root/gpu/gputest_41/win7.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-k en-us \
-device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x4 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=41:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/nvidia/gt710.rom,x-vga=on \
-msg timestamp=on \
-vga none \
-cpu host,hv_vendor_id=1234567890ab,kvm=off \
-net nic,model=virtio \
-net tap,ifname=tap1,script=/root/qemu/qemu-ifup.sh,downscript=/root/qemu/qemu-ifdown.sh \
-monitor stdio
System info:
Linux ns.mqcache.net 4.2.0-1.el7.elrepo.x86_64 #1 SMP Sun Aug 30 21:25:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Output of lspci -vvv:
05:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 15
NUMA node: 0
Region 0: Memory at d8000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]
Region 3: Memory at ce000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at ec80 [size=128]
Expansion ROM at d9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900 v1] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau
05:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 14
NUMA node: 0
Region 0: Memory at d9ffc000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
41:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 15
NUMA node: 1
Region 0: Memory at b8000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Region 1: Memory at 98000000 (64-bit, prefetchable) [disabled] [size=128M]
Region 3: Memory at a6000000 (64-bit, prefetchable) [disabled] [size=32M]
Region 5: I/O ports at bc80 [disabled] [size=128]
Expansion ROM at b9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900 v1] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau
41:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 14
NUMA node: 1
Region 0: Memory at b9ffc000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
After starting vm:
lspci -vvv
05:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 102
NUMA node: 0
Region 0: Memory at d8000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]
Region 3: Memory at ce000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at ec80 [size=128]
Expansion ROM at d9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900 v1] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau
05:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 101
NUMA node: 0
Region 0: Memory at d9ffc000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
41:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 104
NUMA node: 1
Region 0: Memory at b8000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 98000000 (64-bit, prefetchable) [size=128M]
Region 3: Memory at a6000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at bc80 [size=128]
Expansion ROM at b9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900 v1] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau
41:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 103
NUMA node: 1
Region 0: Memory at b9ffc000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

Related

New problem moving from Ubuntu Impish to Jammy; Bluetooth LE Remote pairing/connecting, but not creating /dev/input

Something happened between Impish and Jammy on my device, and my Xiaomi MiTV bluetooth remote control no longer works. After upgrading to 22.04, it just stopped creating a /dev/input, and has continued not working even switching to mainline kernel 5.17 (Jammy's 5.15.x doesn't work either) and the wishful thinking that every update on apt that shows up will fix it. It's been there since the pre-release and has remained ever since.
Device in question:
https://xiaomi-mi.ca/accessories-tv-box/xiaomi-mi-tv-mi-tv-box-bluetooth-remote-control/
I'm running Ubuntu Mate on an Odroid H2+ (j4115-based x86 SBC) as a media player. Bluetooth has been supplied via a BCM20702A-based IOGear BT4.0 adapter (driver installed), but I've also tried a (legit) CSR8510-based BT4.0 adapter with similar results.
It's a bit finicky, but pairs fine if certain steps are followed—but it's been that way with this remote and every ELEC I've had in the past that I've had in the past (PI ArmHF /Odroid ARM64). Note: It pairs and is usable on Windows, MacOS and iOS and pre-Jammy versions of Ubuntu.
Addresses obfuscated to protect the innocent…
REMOTE & CONTROLLER from bluetoothctl:
[XiaoMi RC]# info
Device F4:XX:XX:XX:XX:XX (public)
Name: XiaoMi RC
Alias: XiaoMi RC
Appearance: 0x03c0
Paired: yes
Trusted: yes
Blocked: no
Connected: yes
WakeAllowed: yes
LegacyPairing: no
UUID: Generic Access Profile (00001800-0000-1000-8000-00805f9b34fb)
UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
UUID: Immediate Alert (00001802-0000-1000-8000-00805f9b34fb)
UUID: Link Loss (00001803-0000-1000-8000-00805f9b34fb)
UUID: Tx Power (00001804-0000-1000-8000-00805f9b34fb)
UUID: Device Information (0000180a-0000-1000-8000-00805f9b34fb)
UUID: Battery Service (0000180f-0000-1000-8000-00805f9b34fb)
UUID: Human Interface Device (00001812-0000-1000-8000-00805f9b34fb)
UUID: Scan Parameters (00001813-0000-1000-8000-00805f9b34fb)
Modalias: bluetooth:v2717p3200d0110
Battery Percentage: 0x47 (71)
[XiaoMi RC]# show 5C:XX:XX:XX:XX:XX
Controller 5C:XX:XX:XX:XX:XX (public)
Name: mybox
Alias: mybox
Class: 0x00000104
Powered: yes
Discoverable: no
DiscoverableTimeout: 0x000000b4
Pairable: yes
UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
UUID: Generic Access Profile (00001800-0000-1000-8000-00805f9b34fb)
UUID: PnP Information (00001200-0000-1000-8000-00805f9b34fb)
UUID: A/V Remote Control Target (0000110c-0000-1000-8000-00805f9b34fb)
UUID: A/V Remote Control (0000110e-0000-1000-8000-00805f9b34fb)
UUID: Device Information (0000180a-0000-1000-8000-00805f9b34fb)
Modalias: usb:v1D6Bp0246d0540
Discovering: no
Roles: central
Roles: peripheral
Advertising Features:
ActiveInstances: 0x00 (0)
SupportedInstances: 0x05 (5)
SupportedIncludes: tx-power
SupportedIncludes: appearance
SupportedIncludes: local-name
LSUSB:
Bus 001 Device 005: ID 0a5c:21e8 Broadcom Corp. BCM20702A0 Bluetooth 4.0
busctl:
busctl introspect 'org.bluez' '/org/bluez/hci0/dev_F4_XX_XX_XX_XX_XX'
NAME TYPE SIGNATURE RESULT/VALUE FLAGS
org.bluez.Battery1 interface - - -
.Percentage property y 71 emits-change
org.bluez.Device1 interface - - -
.CancelPairing method - - -
.Connect method - - -
.ConnectProfile method s - -
.Disconnect method - - -
.DisconnectProfile method s - -
.Pair method - - -
.Adapter property o "/org/bluez/hci0" emits-change
.Address property s "F4:XX:XX:XX:XX:XX" emits-change
.AddressType property s "public" emits-change
.Alias property s "XiaoMi RC" emits-change writable
.Appearance property q 960 emits-change
.Blocked property b false emits-change writable
.Class property u - emits-change
.Connected property b true emits-change
.Icon property s - emits-change
.LegacyPairing property b false emits-change
.ManufacturerData property a{qv} - emits-change
.Modalias property s "bluetooth:v2717p3200d0110" emits-change
.Name property s "XiaoMi RC" emits-change
.Paired property b true emits-change
.RSSI property n - emits-change
.ServiceData property a{sv} - emits-change
.ServicesResolved property b true emits-change
.Trusted property b true emits-change writable
.TxPower property n - emits-change
.UUIDs property as 9 "00001800-0000-1000-8000-00805f9b34fb… emits-change
.WakeAllowed property b true emits-change writable
org.freedesktop.DBus.Introspectable interface - - -
.Introspect method - s -
org.freedesktop.DBus.Properties interface - - -
.Get method ss v -
.GetAll method s a{sv} -
.Set method ssv - -
.PropertiesChanged signal sa{sv}as -
a few odd messages in syslog, but nothing that screams "major fail":
:~$ cat /var/log/syslog | grep "Blue"
Apr 5 22:50:24 mybox systemd[1]: Starting Bluetooth management mechanism...
Apr 5 22:50:24 mybox kernel: [ 7.337177] Bluetooth: Core ver 2.22
Apr 5 22:50:24 mybox kernel: [ 7.337217] Bluetooth: HCI device and connection manager initialized
Apr 5 22:50:24 mybox kernel: [ 7.337222] Bluetooth: HCI socket layer initialized
Apr 5 22:50:24 mybox kernel: [ 7.337227] Bluetooth: L2CAP socket layer initialized
Apr 5 22:50:24 mybox kernel: [ 7.337234] Bluetooth: SCO socket layer initialized
Apr 5 22:50:24 mybox kernel: [ 7.575381] Bluetooth: hci0: BCM: chip id 63
Apr 5 22:50:24 mybox kernel: [ 7.576384] Bluetooth: hci0: BCM: features 0x07
Apr 5 22:50:24 mybox kernel: [ 7.592401] Bluetooth: hci0: BCM20702A
Apr 5 22:50:24 mybox kernel: [ 7.592411] Bluetooth: hci0: BCM20702A1 (001.002.014) build 0000
Apr 5 22:50:24 mybox kernel: [ 7.594649] Bluetooth: hci0: BCM20702A1 'brcm/BCM20702A1-0a5c-21e8.hcd' Patch
Apr 5 22:50:24 mybox kernel: [ 8.196379] Bluetooth: hci0: unexpected event 0xff length: 2 > 0
Apr 5 22:50:24 mybox systemd[1]: Started Bluetooth management mechanism.
Apr 5 22:50:24 mybox kernel: [ 8.518390] Bluetooth: hci0: Broadcom Bluetooth Device
Apr 5 22:50:24 mybox kernel: [ 8.518400] Bluetooth: hci0: BCM20702A1 (001.002.014) build 1764
Apr 5 22:50:24 mybox systemd[1]: Starting Bluetooth service...
Apr 5 22:50:24 mybox bluetoothd[970]: Bluetooth daemon 5.64
Apr 5 22:50:24 mybox systemd[1]: Started Bluetooth service.
Apr 5 22:50:24 mybox NetworkManager[699]: <info> [1649213424.6320] Loaded device plugin: NMBluezManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1.36.4/libnm-device-plugin-bluetooth.so)
Apr 5 22:50:24 mybox systemd[1]: Reached target Bluetooth Support.
Apr 5 22:50:24 mybox kernel: [ 8.948829] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
Apr 5 22:50:24 mybox kernel: [ 8.948837] Bluetooth: BNEP filters: protocol multicast
Apr 5 22:50:24 mybox kernel: [ 8.948843] Bluetooth: BNEP socket layer initialized
Apr 5 22:50:24 mybox bluetoothd[970]: Bluetooth management interface 1.21 initialized
Apr 5 22:50:36 mybox kernel: [ 20.341197] Bluetooth: RFCOMM TTY layer initialized
Apr 5 22:50:36 mybox kernel: [ 20.341222] Bluetooth: RFCOMM socket layer initialized
Apr 5 22:50:36 mybox kernel: [ 20.341233] Bluetooth: RFCOMM ver 1.11
Apr 5 22:56:28 mybox systemd[1777]: Reached target Bluetooth.
Apr 5 23:17:03 mybox systemd[1777]: Starting Bluetooth Manager...
Apr 5 23:17:03 mybox systemd[1777]: Started Bluetooth Manager.
Apr 5 23:17:25 mybox kernel: [ 1630.329450] Bluetooth: hci0: unexpected SMP command 0x0b from f4:xx:xx:xx:xx:xx
Apr 5 23:17:52 mybox blueman-manager[115657]: blueman.bluez.errors.BluezDBusException: org.freedesktop.DBus.Error.InvalidArgs No such interface 'org.bluez.Battery1'
Apr 6 06:21:33 mybox NetworkManager[660748]: <info> [1649240493.3391] Loaded device plugin: NMBluezManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1.36.4/libnm-device-plugin-bluetooth.so)
The remote pairs and connects. I can obviously see it bluetoothctl. But no event is created in /dev/input:
me#mybox:~$ sudo evtest
No device specified, trying to scan all of /dev/input/event*
Available devices:
/dev/input/event0: Power Button
/dev/input/event1: Power Button
/dev/input/event2: Video Bus
/dev/input/event3: Logitech K400 Plus
/dev/input/event4: Microsoft X-Box 360 pad
/dev/input/event5: Microsoft X-Box 360 pad
/dev/input/event6: Microsoft X-Box 360 pad
/dev/input/event7: Pulse-Eight CEC Adapter
/dev/input/event8: Microsoft X-Box 360 pad
/dev/input/event9: HDA Intel PCH Mic
/dev/input/event10: HDA Intel PCH Headphone
/dev/input/event11: HDA Intel PCH HDMI/DP,pcm=3
/dev/input/event12: HDA Intel PCH HDMI/DP,pcm=7
/dev/input/event13: HDA Intel PCH HDMI/DP,pcm=8
/dev/input/event14: HDA Intel PCH HDMI/DP,pcm=9
/dev/input/event15: HDA Intel PCH HDMI/DP,pcm=10
Select the device event number [0-15]:
showkey and xev gets no keypresses, but btmon is seeing them:
me#mybox:~$ sudo btmon
Bluetooth monitor ver 5.64
= Note: Linux version 5.17.1-051701-generic (x86_64) 0.026997
= Note: Bluetooth subsystem version 2.22 0.027002
= New Index: 5C:XX:XX:XX:XX:XX (Primary,USB,hci0) [hci0] 0.027004
= Open Index: 5C:XX:XX:XX:XX:XX [hci0] 0.027004
= Index Info: 5C:XX:XX:XX:XX:XX (Broadcom Corporation) [hci0] 0.027005
# MGMT Open: bluetoothd (privileged) version 1.21 {0x0001} 0.027006
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #1 [hci0] 1.698718
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000510000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #2 [hci0] 1.774350
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #3 [hci0] 2.594360
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000520000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #4 [hci0] 2.754407
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #5 [hci0] 3.414342
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000500000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #6 [hci0] 3.574341
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #7 [hci0] 4.014365
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 00004f0000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #8 [hci0] 4.174358
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #9 [hci0] 4.674365
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000660000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #10 [hci0] 4.934380
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #11 [hci0] 5.794429
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000800000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #12 [hci0] 6.034380
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #13 [hci0] 7.194396
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000810000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #14 [hci0] 7.354432
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #15 [hci0] 8.454406
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 00004a0000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #16 [hci0] 8.694405
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #17 [hci0] 10.414431
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000650000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #18 [hci0] 10.654417
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #19 [hci0] 11.134410
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000f10000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #20 [hci0] 11.294427
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #21 [hci0] 12.094482
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000280000000000
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #22 [hci0] 12.234598
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0032
Data: 0000000000000000
This is driving me nuts. Especially because another similar model of the same remote that uses AAA batteries instead of a coin-cell battery works (that one disconnects regularly, and is slow to reconnect however, but pairs fine, creates an input event and all keypresses are recognized).

QEMU how-to allocate specific IRQ number for PCI device?

I'm running qemu-system-x86_64 with my new pci device. And i want to use IRQ 17 (Since driver from kernel listen for IRQ 17). But my PCI device take IRQ 10 or 11. base on interrupt_pin(A,B,C,E).
Then i want to send irq to kernel module by qemu_irq_pulse.
This is how i allocate irq:
pci_config_set_interrupt_pin(pci_dev->config, 1);
d->irq = pci_allocate_irq(pci_dev)
root#hostname:~# cat /proc/interrupts
CPU0
0: 48 IO-APIC 2-edge timer
1: 9 IO-APIC 1-edge i8042
4: 1440 IO-APIC 4-edge ttyS0
8: 1 IO-APIC 8-edge rtc0
9: 0 IO-APIC 9-fasteoi acpi
12: 125 IO-APIC 12-edge i8042
24: 773 PCI-MSI 512000-edge ahci[0000:00:1f.2]
25: 355 PCI-MSI 32768-edge eth0-rx-0
26: 160 PCI-MSI 32769-edge eth0-tx-0
27: 1 PCI-MSI 32770-edge eth0
lspci -nk -vv:
00:1f.3 0880: 10de:0101 (rev 01)
Subsystem: 1af4:1100
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 10
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver failed to request irq since it trying to attach to IRQ 17. I don't want to change kernel side.
This is what i want to see:
root#hostname:~# cat /proc/interrupts
CPU0
0: 2213 IO-APIC
17: 0 IO-APIC 17-fasteoi some_kernel_driver_name
How to allocate interrupt 17 for PCI device in qemu?

Im not sure it is correct answer but for me it helps:
Add to ACPI:
irqs = 17;
aml_append(crs, aml_interrupt(AML_CONSUMER, AML_EDGE,
AML_ACTIVE_HIGH, AML_SHARED,
&irqs, 1));
Also PCI interrupt number looks like somehow depends on PCI vendor_id and device_id.

Tensorflow 1.15 multi worker strategy hangs after graph initialization on multiple machiens

I am running the TF keras_to_estimator example on using two machines, the process hangs after graph initialization when running the start script on each machine.
The messages of console output on worker 0 machine after starting:
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {'worker': ['node4:21111', 'node3:21112']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0',), communication = CollectiveCommunication.AUTO
I0605 17:05:20.218733 139934274328320 collective_all_reduce_strategy.py:310] Multi-worker CollectiveAllReduceStrategy with cluster_spec = {'worker': ['node4:21111', 'node3:21112']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0',
), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Updated config: {'_model_dir': '/node4/jianwang/atp_bert/albert_zh/example_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f44402717b8>, '_device_fn': None, '_protocol': None, '_eval_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f44402755c0>, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.collective_all_reduce_strategy.Collectiv
eAllReduceStrategy object at 0x7f4440275240>, eval_distribute=<tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f44402755c0>, remote_cluster=None), '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f4440275940>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://node4:21111', '_evaluation_master': 'grpc://node4:21111', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 2, '_distribute_coordinator_mode': 'independent_worker'}
I0605 17:05:20.221589 139934274328320 estimator_training.py:228] Updated config: {'_model_dir': '/node4/jianwang/atp_bert/albert_zh/example_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session
_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f44402717b8>, '_device_fn': None, '_proto
col': None, '_eval_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f44402755c0>, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.collective_all_reduce_strategy.Collectiv
eAllReduceStrategy object at 0x7f4440275240>, eval_distribute=<tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f44402755c0>, remote_cluster=None), '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_s
ervice': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f4440275940>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://node4:21111', '_evaluation_master': 'grpc://node4:21111', '_is_chief': True,
'_num_ps_replicas': 0, '_num_worker_replicas': 2, '_distribute_coordinator_mode': 'independent_worker'}
input_fn called
INFO:tensorflow:Calling model_fn.
I0605 17:05:20.358438 139911839606528 estimator.py:1148] Calling model_fn.
...
INFO:tensorflow:Creating chief session creator with config: device_filters: "/job:worker/task:0"
allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
scoped_allocator_optimization: ON
scoped_allocator_opts {
enable_op: "CollectiveReduce"
}
}
}
experimental {
collective_group_leader: "/job:worker/replica:0/task:0"
}
I0605 17:05:20.711247 139934274328320 distribute_coordinator.py:251] Creating chief session creator with config: device_filters: "/job:worker/task:0"
allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
scoped_allocator_optimization: ON
scoped_allocator_opts {
enable_op: "CollectiveReduce"
}
}
}
experimental {
collective_group_leader: "/job:worker/replica:0/task:0"
}
INFO:tensorflow:Graph was finalized.
I0605 17:05:20.870544 139934274328320 monitored_session.py:240] Graph was finalized.
The same message is also print out on the worker 1 machine which also shows that the process is stuck after graph initialization
I0605 17:10:28.616780 140121708521216 collective_all_reduce_strategy.py:310] Multi-worker CollectiveAllReduceStrategy with cluster_spec = {'worker': ['node4:21111', 'node3:21112']}, task_type = 'worker', task_id = 1, num_workers = 2, local_devices = ('/job:worker/task:1',
), communication = CollectiveCommunication.AUTO
INFO:tensorflow:Updated config: {'_model_dir': '/node4/jianwang/atp_bert/albert_zh/example_dir', '_num_ps_replicas': 0, '_tf_random_seed': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_experimental_max_worker_delay_secs': None, '_eval_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f7085a28128>, '_save_checkpoints_secs': 600, '_keep_checkpoint_every_n_hours': 10000, '_is_chief': False, '_keep_checkp
oint_max': 5, '_device_fn': None, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f7085a1eef0>, eval_distribute=<tensorflow.contrib.distribute.pytho
n.mirrored_strategy.MirroredStrategy object at 0x7f7085a28128>, remote_cluster=None), '_session_creation_timeout_secs': 7200, '_master': 'grpc://node3:21112', '_service': None, '_task_type': 'worker', '_task_id': 1, '_protocol': None, '_log_step_count_steps': 100, '_distr
ibute_coordinator_mode': 'independent_worker', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f7085a282e8>, '_global_id_in_cluster': 1, '_evaluation_master': 'grpc://node3:21112', '_train_distribute': <tensorflow.contrib.distribute.python
.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f7085a1e8d0>, '_num_worker_replicas': 2, '_save_checkpoints_steps': None, '_save_summary_steps': 100}
I0605 17:10:28.623507 140121708521216 estimator_training.py:228] Updated config: {'_model_dir': '/node4/jianwang/atp_bert/albert_zh/example_dir', '_num_ps_replicas': 0, '_tf_random_seed': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_experimental_max_worker_delay_secs': None, '_eval_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x7f7085a28128>, '_save_checkpoints_secs': 600, '_keep_checkpoint_every_n_hours': 10000, '_is_chief': False, '_keep_checkp
oint_max': 5, '_device_fn': None, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f7085a1eef0>, eval_distribute=<tensorflow.contrib.distribute.pytho
n.mirrored_strategy.MirroredStrategy object at 0x7f7085a28128>, remote_cluster=None), '_session_creation_timeout_secs': 7200, '_master': 'grpc://node3:21112', '_service': None, '_task_type': 'worker', '_task_id': 1, '_protocol': None, '_log_step_count_steps': 100, '_distr
ibute_coordinator_mode': 'independent_worker', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f7085a282e8>, '_global_id_in_cluster': 1, '_evaluation_master': 'grpc://node3:21112', '_train_distribute': <tensorflow.contrib.distribute.python
.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x7f7085a1e8d0>, '_num_worker_replicas': 2, '_save_checkpoints_steps': None, '_save_summary_steps': 100}
input_fn called
INFO:tensorflow:Calling model_fn.
...
INFO:tensorflow:Creating chief session creator with config: device_filters: "/job:worker/task:1"
allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
scoped_allocator_optimization: ON
scoped_allocator_opts {
enable_op: "CollectiveReduce"
}
}
}
experimental {
collective_group_leader: "/job:worker/replica:0/task:0"
}
I0605 17:10:29.048442 140121708521216 distribute_coordinator.py:251] Creating chief session creator with config: device_filters: "/job:worker/task:1"
allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
scoped_allocator_optimization: ON
scoped_allocator_opts {
enable_op: "CollectiveReduce"
}
}
}
experimental {
collective_group_leader: "/job:worker/replica:0/task:0"
Code related: (1) example.sh (start script running on node4 the worker 0 machine)
export TF_CONFIG='{
"cluster": {
"worker": ["node4:21111", "node3:21112"]
},
"task": {"type": "worker", "index": 0}
}'
export CUDA_VISIBLE_DEVICES=0
export OUTPUT_DIR=/node4/jianwang/atp_bert/albert_zh/example_dir
python example.py $OUTPUT_DIR
(2) example_slave.sh (start script to run on the worker 1 machine)
export TF_CONFIG='{
"cluster": {
"worker": ["node4:21111", "node3:21112"]
},
"task": {"type": "worker", "index": 1}
}'
export CUDA_VISIBLE_DEVICES=7
export OUTPUT_DIR=/node4/jianwang/atp_bert/albert_zh/example_dir
python example.py $OUTPUT_DIR
(3) example.py
"""An example of training Keras model with multi-worker strategies."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import sys
import numpy as np
import tensorflow as tf
def input_fn():
print("input_fn called")
x = np.random.random((1024, 10))
y = np.random.randint(2, size=(1024, 1))
x = tf.cast(x, tf.float32)
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.repeat(100)
dataset = dataset.batch(32)
return dataset
def main(args):
if len(args) < 2:
print('You must specify model_dir for checkpoints such as'
' /tmp/tfkeras_example/.')
return
model_dir = args[1]
print('Using %s to store checkpoints.' % model_dir)
# Define a Keras Model.
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(16, activation='relu', input_shape=(10,)))
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
# Compile the model.
optimizer = tf.train.GradientDescentOptimizer(0.2)
model.compile(loss='binary_crossentropy', optimizer=optimizer)
model.summary()
tf.keras.backend.set_learning_phase(True)
# Define DistributionStrategies and convert the Keras Model to an
# Estimator that utilizes these DistributionStrateges.
# Evaluator is a single worker, so using MirroredStrategy.
config = tf.estimator.RunConfig(
experimental_distribute=tf.contrib.distribute.DistributeConfig(
train_distribute=tf.contrib.distribute.CollectiveAllReduceStrategy(
),
eval_distribute=tf.contrib.distribute.MirroredStrategy(
)))
keras_estimator = tf.keras.estimator.model_to_estimator(
keras_model=model, config=config, model_dir=model_dir)
# Train and evaluate the model. Evaluation will be skipped if there is not an
# "evaluator" job in the cluster.
print("Start train eval")
tf.estimator.train_and_evaluate(
keras_estimator,
train_spec=tf.estimator.TrainSpec(input_fn=input_fn),
eval_spec=tf.estimator.EvalSpec(input_fn=input_fn))
if __name__ == '__main__':
tf.logging.set_verbosity(tf.logging.INFO)
tf.app.run(argv=sys.argv)
lspci output:
lspci |grep PCI
jianwang#node3:~$ lspci |grep PCI
00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
02:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
03:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
03:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
06:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
07:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
07:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0b:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
7f:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
7f:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
80:00.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 0 (rev 01)
80:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
80:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
83:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
84:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
84:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
87:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
88:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
88:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
I have tried to add "chief" to TF_CONFIG, disable IOMMU followed by : disable ioMMU
none worked. Please help on :
(1) how to diagnose the problem on where it hangs
(2) any insights on how to work around this problem

Openstack VM instance SHUTOFF after few minute

For learning purpose i have build openstack on VirtualBox with 2 vCPU and 4GB Memory. It installed successfully and i am able to start VM instances but what happened is guest VM got SHUTOFF status after few minutes. I have google this issue but didn't get proper answer. I have check logs and i didn't find anything suspicious.
How do i check VM console so i can see what is going on there?
Where should i check SHUTOFF specific error logs, i meant in which file?
EDIT:
Following is output of nova console-log but it stuck there not going ahead and i can't see login screen too
openstack#openstack1:~$ nova console-log 970a3722-0fb3-4db6-862b-2aa626cc68a8
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.0.0-12-virtual (buildd#crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 (Ubuntu 3.0.0-12.20-virtual 3.0.4)
[ 0.000000] Command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
[ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000001fffd000 (usable)
[ 0.000000] BIOS-e820: 000000001fffd000 - 0000000020000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI 2.4 present.
[ 0.000000] No AGP bridge found
[ 0.000000] last_pfn = 0x1fffd max_arch_pfn = 0x400000000
[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[ 0.000000] found SMP MP-table at [ffff8800000fdaf0] fdaf0
[ 0.000000] init_memory_mapping: 0000000000000000-000000001fffd000
[ 0.000000] RAMDISK: 1fdf9000 - 1ffed000
[ 0.000000] ACPI: RSDP 00000000000fd990 00014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 000000001fffd7b0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 000000001fffff80 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 000000001fffd9b0 02589 (v01 BXPC BXDSDT 00000001 INTL 20100528)
[ 0.000000] ACPI: FACS 000000001fffff40 00040
[ 0.000000] ACPI: SSDT 000000001fffd910 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: APIC 000000001fffd830 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] ACPI: HPET 000000001fffd7f0 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-000000001fffd000
[ 0.000000] Initmem setup node 0 0000000000000000-000000001fffd000
[ 0.000000] NODE_DATA [000000001fff5000 - 000000001fff9fff]
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000010 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal empty
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000010 -> 0x0000009d
[ 0.000000] 0: 0x00000100 -> 0x0001fffd
[ 0.000000] ACPI: PM-Timer IO Port: 0xb008
[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000
[ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
[ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
[ 0.000000] Allocating PCI resources starting at 20000000 (gap: 20000000:dffc0000)
[ 0.000000] Booting paravirtualized kernel on bare hardware
[ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 27 pages/cpu #ffff88001fa00000 s79296 r8192 d23104 u2097152
[ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 129157
[ 0.000000] Policy zone: DMA32
[ 0.000000] Kernel command line: LABEL=cirros-rootfs ro console=tty0 console=ttyS0 console=hvc0
[ 0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Memory: 497852k/524276k available (6206k kernel code, 460k absent, 25964k reserved, 6907k data, 900k init)
[ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] NR_IRQS:4352 nr_irqs:256 16
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [ttyS0] enabled
[ 0.000000] allocated 4194304 bytes of page_cgroup
[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.000000] Fast TSC calibration failed
[ 0.000000] TSC: Unable to calibrate against PIT
[ 0.000000] TSC: using PMTIMER reference calibration
[ 0.000000] Detected 2486.018 MHz processor.
[ 0.024490] Calibrating delay loop (skipped), value calculated using timer frequency.. 4972.03 BogoMIPS (lpj=9944072)
[ 0.025939] pid_max: default: 32768 minimum: 301
[ 0.029903] Security Framework initialized
[ 0.033041] AppArmor: AppArmor initialized
[ 0.033539] Yama: becoming mindful.
[ 0.037514] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.039560] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.040693] Mount-cache hash table entries: 256
[ 0.054301] Initializing cgroup subsys cpuacct
[ 0.054957] Initializing cgroup subsys memory
[ 0.056108] Initializing cgroup subsys devices
[ 0.056838] Initializing cgroup subsys freezer
[ 0.057341] Initializing cgroup subsys net_cls
[ 0.057824] Initializing cgroup subsys blkio
[ 0.058338] Initializing cgroup subsys perf_event
[ 0.060182] mce: CPU supports 10 MCE banks
[ 0.062116] SMP alternatives: switching to UP code
[ 0.236105] Freeing SMP alternatives: 24k freed
[ 0.237129] ACPI: Core revision 20110413
[ 0.270578] ftrace: allocating 26075 entries in 103 pages
[ 0.289821] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.332667] CPU0: AMD QEMU Virtual CPU version 1.0 stepping 03
[ 0.336020] APIC calibration not consistent with PM-Timer: 103ms instead of 100ms
[ 0.336020] APIC delta adjusted to PM-Timer: 6249961 (6456813)
[ 0.336020] Performance Events: Broken PMU hardware detected, using software events only.
[ 0.341160] Brought up 1 CPUs
[ 0.341596] Total of 1 processors activated (4972.03 BogoMIPS).
[ 0.348508] devtmpfs: initialized
[ 0.370265] print_constraints: dummy:
[ 0.370818] Time: 22:32:35 Date: 07/31/13
[ 0.373184] NET: Registered protocol family 16
[ 0.377862] ACPI: bus type pci registered
[ 0.379805] PCI: Using configuration type 1 for base access
[ 0.394436] bio: create slab <bio-0> at 0
[ 0.441293] ACPI: Interpreter enabled
[ 0.441749] ACPI: (supports S0 S3 S4 S5)
[ 0.442853] ACPI: Using IOAPIC for interrupt routing
[ 0.504949] ACPI: No dock devices found.
[ 0.505458] HEST: Table not found.
[ 0.505922] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[ 0.508456] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.514427] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
[ 0.515222] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB
[ 0.526520] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e)
[ 0.612644] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.614063] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.615312] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.616918] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.618197] ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
[ 0.622888] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.623734] vgaarb: loaded
[ 0.624235] vgaarb: bridge control possible 0000:00:02.0
[ 0.627513] SCSI subsystem initialized
[ 0.629754] usbcore: registered new interface driver usbfs
[ 0.630590] usbcore: registered new interface driver hub
[ 0.632126] usbcore: registered new device driver usb
[ 0.634610] PCI: Using ACPI for IRQ routing
[ 0.640771] NetLabel: Initializing
[ 0.641144] NetLabel: domain hash size = 128
[ 0.641570] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.642769] NetLabel: unlabeled traffic allowed by default
[ 0.744929] AppArmor: AppArmor Filesystem Enabled
[ 0.746522] pnp: PnP ACPI init
[ 0.748377] ACPI: bus type pnp registered
[ 0.761838] pnp: PnP ACPI: found 8 devices
[ 0.762440] ACPI: ACPI bus type pnp unregistered
[ 0.791325] Switching to clocksource acpi_pm
[ 0.791325] NET: Registered protocol family 2
[ 0.792984] Switched to NOHz mode on CPU #0
[ 0.794980] IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.800380] TCP established hash table entries: 16384 (order: 6, 262144 bytes)
[ 0.802008] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[ 0.803089] TCP: Hash tables configured (established 16384 bind 16384)
[ 0.803751] TCP reno registered
[ 0.804373] UDP hash table entries: 256 (order: 1, 8192 bytes)
[ 0.805192] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[ 0.806852] NET: Registered protocol family 1
[ 0.807530] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.808586] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.809327] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.816560] audit: initializing netlink socket (disabled)
[ 0.817591] type=2000 audit(1375309954.816:1): initialized
[ 0.903327] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.928384] VFS: Disk quotas dquot_6.5.2
[ 0.929484] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.938210] fuse init (API version 7.16)
[ 0.940982] msgmni has been set to 972
[ 0.949280] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.950562] io scheduler noop registered
[ 0.951008] io scheduler deadline registered (default)
[ 0.951941] io scheduler cfq registered
[ 0.955245] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 0.956970] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 0.960881] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.962211] ACPI: Power Button [PWRF]
[ 0.979110] ERST: Table is not found!
[ 0.982891] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[ 0.983651] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11
[ 0.986746] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
[ 0.987395] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, high) -> IRQ 10
[ 0.993533] Trying to unpack rootfs image as initramfs...
[ 1.017633] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[ 1.018210] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
[ 1.020389] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 1.052583] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 1.082516] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 1.165489] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 1.244653] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 1.248018] hpet_acpi_add: no address or irqs in _CRS
[ 1.249922] Linux agpgart interface v0.103
[ 1.279474] brd: module loaded
[ 1.287981] loop: module loaded
[ 1.597690] vda: vda1
[ 1.624125] Freeing initrd memory: 2000k freed
[ 1.626790] scsi0 : ata_piix
[ 1.629007] scsi1 : ata_piix
[ 1.629910] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
[ 1.630652] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
[ 1.636489] Fixed MDIO Bus: probed
[ 1.637469] PPP generic driver version 2.4.2
[ 1.638209] tun: Universal TUN/TAP device driver, 1.6
[ 1.638756] tun: (C) 1999-2004 Max Krasnyansky <maxk#qualcomm.com>
openstack#openstack1:~$

You can get guest console on the dashboard or with this command:
nova get-vnc-console <instance id> novnc
If your guest image redirects console messages (like the ubuntu cloud image), you can see boot messages on dashboard or with the command:
nova console-log <instance id>
You may get clues in /var/log/nova/nova-compute.log and in your hypervisor logs (/var/log/libvirt/libvirtd.log for QEMU/KVM).
A possible cause is that your guest can't boot on its primary disk and get stuck on boot sequence. Try other images, like the ones proposed in OpenStack documentation.

How to detect text mode or graphics mode on boot loader?

I want to detect which mode I just using now with BIOS intXX when running bootloader I wrote.
How to detect now is text mode or graphics mode?
Which interrupt function I should use?
Thank you~
I didn't figure out why when I run int 10 ,the value of AL doesn't change.
(0) Breakpoint 1, 0x00007c00 in ?? ()
Next at t=12943079
(0) [0x00007c00] 0000:7c00 (unk. ctxt): mov ah, 0x0f ; b40f
<bochs:3> reg
eax: 0x0000aa55 43605
ecx: 0x00090000 589824
edx: 0x00000000 0
ebx: 0x00000000 0
esp: 0x0000ffd6 65494
ebp: 0x00000000 0
esi: 0x000e476c 935788
edi: 0x0000ffac 65452
eip: 0x00007c00
eflags 0x00000082: id vip vif ac vm rf nt IOPL=0 of df if tf SF zf af pf cf
<bochs:4> n
Next at t=12943080
(0) [0x00007c02] 0000:7c02 (unk. ctxt): mov al, 0xaa ; b0aa
<bochs:5> reg
eax: 0x00000f55 3925
ecx: 0x00090000 589824
edx: 0x00000000 0
ebx: 0x00000000 0
esp: 0x0000ffd6 65494
ebp: 0x00000000 0
esi: 0x000e476c 935788
edi: 0x0000ffac 65452
eip: 0x00007c02
eflags 0x00000082: id vip vif ac vm rf nt IOPL=0 of df if tf SF zf af pf cf
<bochs:6> n
Next at t=12943081
(0) [0x00007c04] 0000:7c04 (unk. ctxt): int 0x0a ; cd0a
<bochs:7> reg
eax: 0x00000faa 4010
ecx: 0x00090000 589824
edx: 0x00000000 0
ebx: 0x00000000 0
esp: 0x0000ffd6 65494
ebp: 0x00000000 0
esi: 0x000e476c 935788
edi: 0x0000ffac 65452
eip: 0x00007c04
eflags 0x00000082: id vip vif ac vm rf nt IOPL=0 of df if tf SF zf af pf cf
<bochs:8> n
Next at t=12943083
(0) [0x00007c06] 0000:7c06 (unk. ctxt): mov dl, al ; 88c2
<bochs:9> reg
eax: 0x00000faa 4010
ecx: 0x00090000 589824
edx: 0x00000000 0
ebx: 0x00000000 0
esp: 0x0000ffd6 65494
ebp: 0x00000000 0
esi: 0x000e476c 935788
edi: 0x0000ffac 65452
eip: 0x00007c06
eflags 0x00000082: id vip vif ac vm rf nt IOPL=0 of df if tf SF zf af pf cf
<bochs:10>

INT10, F
AH = 0F
on return:
AL = mode currently set(page mode)
BH = current display page
Page mode:
AL = 00 40x25 B/W text (CGA,EGA,MCGA,VGA)
= 01 40x25 16 color text (CGA,EGA,MCGA,VGA)
= 02 80x25 16 shades of gray text (CGA,EGA,MCGA,VGA)
= 03 80x25 16 color text (CGA,EGA,MCGA,VGA)
= 04 320x200 4 color graphics (CGA,EGA,MCGA,VGA)
= 05 320x200 4 color graphics (CGA,EGA,MCGA,VGA)
= 06 640x200 B/W graphics (CGA,EGA,MCGA,VGA)
= 07 80x25 Monochrome text (MDA,HERC,EGA,VGA)
= 08 160x200 16 color graphics (PCjr)
= 09 320x200 16 color graphics (PCjr)
= 0A 640x200 4 color graphics (PCjr)
= 0B Reserved (EGA BIOS function 11)
= 0C Reserved (EGA BIOS function 11)
= 0D 320x200 16 color graphics (EGA,VGA)
= 0E 640x200 16 color graphics (EGA,VGA)
= 0F 640x350 Monochrome graphics (EGA,VGA)
= 10 640x350 16 color graphics (EGA or VGA with 128K)
640x350 4 color graphics (64K EGA)
= 11 640x480 B/W graphics (MCGA,VGA)
= 12 640x480 16 color graphics (VGA)
= 13 320x200 256 color graphics (MCGA,VGA)
= 8x EGA, MCGA or VGA ignore bit 7, see below
= 9x EGA, MCGA or VGA ignore bit 7, see below