Two blade servers running Windows 2008 R2 and acting as backup media servers to connect to VTL 6900 are failing to boot up, showing errors.
【Problem description】
Two blade servers running Windows 2008 R2 and acting as Backup Media Servers to connect to VTL 6900 is failing to boot up showing the error . On troubleshooting it was found by RA systems team that Emulex Unknown PCI Slot1,Storport minIPort driver is having a conflict and fails to load the ClassPNP.sys driver. Only by disabling the Emulex Unknown PCI Slot1,Storport minIPort device we are able to boot up the OS as per the troubleshooting done so far by RA Systems team.
【Problem analysis】
Got some message about what was done before the issue occured , the customer installed the latest patch of microsoft , also we can not find out any error on the server. So we collect the os dump file , and create a ticket to microsoft , the analysis from microsoft as bellow.
Analysis:
=============
Both the servers show the same problem.
• Both of the servers rebooted with bugcheck 0x0000003B which is SYSTEM_SERVICE_EXCEPTION. This indicates an exception happened while executing a routine that transitions from non-privileged code to privileged code.
BugCheck 3B, {00000000`c0000005 fffff880`01801198 fffff880`08c77dd0 00000000`00000000}
• Checking further in the dump file I found that there is a trap caused by CLASSPNP.SYS which crashed the server:
12: kd> .cxr 0xfffff88008c77dd0
rax=0000000000000000 rbx=fffffa803d1f88e0 rcx=0000000000000000
rdx=fffffa802a26a010 rsi=fffffa8027cacb90 rdi=fffffa802a26a248
rip=fffff88001801198 rsp=fffff88008c787a0 rbp=fffffa802a26a010
r8=fffffa803d1f8790 r9=0000000000001000 r10=fffffa803d1f88e0
r11=0000000000000000 r12=fffffa802a26a010 r13=fffffa803bdf9010
r14=0000000000001000 r15=fffffa803bdf9010
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010246
CLASSPNP!ServiceTransferRequest+0xa8:
fffff880`01801198 448b502c mov r10d,dword ptr [rax+2Ch] ds:002b:00000000`0000002c=????????
• Here is the call stack during crash.
00 fffff880`08c77508 fffff800`01ccef29 nt!KeBugCheckEx
01 fffff880`08c77510 fffff800`01cce87c nt!KiBugCheckDispatch+0x69
02 fffff880`08c77650 fffff800`01cfb99d nt!KiSystemServiceHandler+0x7c
03 fffff880`08c77690 fffff800`01cfa775 nt!RtlpExecuteHandlerForException+0xd
04 fffff880`08c776c0 fffff800`01d0bcfd nt!RtlDispatchException+0x415
05 fffff880`08c77da0 fffff800`01ccf00e nt!KiDispatchException+0x17d
06 fffff880`08c78430 fffff800`01ccdb7a nt!KiExceptionDispatch+0xce
07 fffff880`08c78610 fffff880`01801198 nt!KiPageFault+0x23a
08 fffff880`08c787a0 fffff880`01801975 CLASSPNP!ServiceTransferRequest+0xa8
(struct _DEVICE_OBJECT * Fdo = 0xfffffa80`3d1f8790 Device for "\Driver\Disk", struct _IRP * Irp = 0xfffffa80`2a26a200)
09 fffff880`08c78840 fffff880`00e620af CLASSPNP!ClassReadWrite+0xd5
(struct _DEVICE_OBJECT * DeviceObject = 0xfffffa80`2a26a010, struct _IRP * Irp = 0xfffff880`08c78b00)
0a fffff880`08c78890 fffff880`0183c063 partmgr!PmGlobalDispatch+0x9f
(struct _DEVICE_OBJECT * DeviceObject = 0xfffffa80`3cf8d5e0, struct _IRP * Irp = 0xfffffa80`2a26a010)
0b fffff880`08c788c0 fffff800`01fbe8ca mfedisk!DEVICEDISPATCH::DispatchPassThrough+0x143
0c fffff880`08c78940 fffff800`01ccec13 nt!NtReadFile+0x718
0d fffff880`08c78a70 00000000`76debd9a nt!KiSystemServiceCopyEnd+0x13
0e 00000000`26a0f4e8 00000000`00000000 0x76debd9a
• We crash on CLASSPNP here because MdlAddres is null. PUCHAR bufPtr = MmGe***lVirtualAddress(Irp->MdlAddress);
12: kd> !irp fffffa802a26a010
Irp is active with 6 stacks 6 is current (= 0xfffffa802a26a248)
No Mdl: No System Buffer: Thread fffffa803e550b50: Irp stack trace.
cmd *** cl Device File Completion-Context
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[IRP_MJ_READ(3), N/A(34)]
0 0 fffffa803d1f8790 fffffa803e302d10 00000000-00000000
\Driver\Disk
Args: 00001000 00000000 00000000 00000000
12: kd> dt _IRP fffffa802a26a010
nt!_IRP
+0x000 Type : 0n6
+0x002 Size : 0x3a0
+0x008 MdlAddress : (null)
+0x010 Flags : 0x60901
+0x018 AssociatedIrp : <unnamed-tag>
+0x020 ThreadListEntry : _LIST_ENTRY [ 0xfffffa80`3e550f40 - 0xfffffa80`3e550f40 ]
+0x030 IoStatus : _IO_STATUS_BLOCK
+0x040 RequestorMode : 1 ''
+0x041 PendingReturned : 0 ''
+0x042 StackCount : 6 ''
+0x043 CurrentLocation : 6 ''
+0x044 Cancel : 0 ''
+0x045 CancelIrql : 0 ''
+0x046 ApcEnvironment : 0 ''
+0x047 AllocationFlags : 0x6 ''
+0x048 UserIosb : 0x00000000`26a0f540 _IO_STATUS_BLOCK
+0x050 UserEvent : (null)
+0x058 Overlay : <unnamed-tag>
+0x068 CancelRoutine : (null)
+0x070 UserBuffer : 0x00000000`27c6e000 Void
+0x078 Tail : <unnamed-tag>
12: kd> !fileobj fffffa803e302d10
Device Object: 0xfffffa803be62cb0 \Driver\nxup
Vpb is NULL
Flags: 0x4000a
Synchronous IO
No Intermediate Buffering
Handle Created
File Object is currently busy and has 0 waiters.
CurrentByteOffset: 0
• Below are all the component involved regarding this IRP.
12: kd> !devstack fffffa803d1f8790
!DevObj !DrvObj !DevExt ObjectName
fffffa803be41550 \Driver\mfedisk fffffa803be416a0
fffffa803bd954a0 \Driver\partmgr fffffa803bd955f0
> fffffa803d1f8790 \Driver\Disk fffffa803d1f88e0 DR1
fffffa803d1635e0 \Driver\iaStorF fffffa803d163730
fffffa803be62cb0 \Driver\nxup fffffa803be62e00 UpioDisk0
!DevNode fffffa80293acd90 :
DeviceInst is "UPIO\DiskHuaweiUltraPath_________________________1.001___\1&146a97f6&0&3666383461626631303036303434383231323066303337333030303030303062"
ServiceName is "disk"
• Here are detail of drivers from the dump file:
12: kd> lmvm classpnp
Browse full module list
start end module name
fffff880`01800000 fffff880`01830000 CLASSPNP
Loaded symbol image file: CLASSPNP.SYS
Image path: \SystemRoot\system32\DRIVERS\CLASSPNP.SYS
Image name: CLASSPNP.SYS
Browse all global symbols functions data
Timestamp: Sat Nov 20 02:19:23 2010 (4CE7929B)
CheckSum: 000318BE
ImageSize: 00030000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Information from resource tables:
12: kd> lmvm iaStorF
Browse full module list
start end module name
fffff880`01a3f000 fffff880`01a48000 iaStorF (no symbols)
Loaded symbol image file: iaStorF.sys
Image path: \SystemRoot\system32\DRIVERS\iaStorF.sys
Image name: iaStorF.sys
Browse all global symbols functions data
Timestamp: Fri Sep 14 03:26:28 2012 (50530654)
CheckSum: 000154AB
ImageSize: 00009000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Information from resource tables:
12: kd> lmvm nxup
Browse full module list
start end module name
fffff880`01000000 fffff880`01049000 nxup (export symbols) nxup.sys
Loaded symbol image file: nxup.sys
Image path: \SystemRoot\system32\DRIVERS\nxup.sys
Image name: nxup.sys
Browse all global symbols functions data
Timestamp: Wed Dec 17 01:33:40 2014 (54913FE4)
CheckSum: 00038A9F
ImageSize: 00049000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Information from resource tables:
And we found that this is an know problem in huawei , if the customer installed Mcafee and used huawei ultrapath on one vm in the same time.
【Root cause】
The customer installed Mcafee and used huawei ultrapath on one vm at the same time.
【Soultion description】
1. Uninstall the McAfee driver mfedisk.
2. Upgrade the Huawei driver nxup to the latest version.
Ultrapath link : http://support.huawei.com/enterprise/en/software/22816044-SW1000279528.
3. Upgrade the Intel RAID driver iaStorF to the latest version.
Tool link : http://support.huawei.com/enterprise/en/software/22870645-SW2000002264.
\Windows-Driver-V329\win2k8r2sp1\onboard_driver_win2k8r2sp1\chipset_10.1.2.10\SetupChipset.exe
4. Install OS convenience update on OS.
