Latest revision as of 09:15, 24 March 2014

Publication from ADAPT'14 workshop.

Notes on reproducibility:

experiment plan:


1. run the detector app until 30 samples.
2. stop the detector app.
3. inject fault (1,2,3,4,5).
4. start the detector app.
5. observe and record the results
5-    repeat until all faults tested.

6. remove the oldest 5 source pairs from the data sub-directory
7. go to 3 -- repeat until all datasets deleted


expected results:

For each DataSet collection, (30, 25, 20, 15, 10, 5, 0)...
    Record the following:
    The Fault Source / Name

    dbl_Precision,
    dbl_Recall,
    dbl_Harmonic_Mean,
    dbl_Accuracy,
    int_False_Negatives,
    int_False_Positives,
    int_True_Negatives,
    int_True_Positives,
    lng_Time_Taken,
    int_Total_Potential_Faults_List_Count
    int_User_Specified_Index_Value.ToString()
    f.Source
    f.Confidence

    Position of the fault in the list - actual -- this is where the REAL fault is in the list
    Position of the fault in the list - program's estimate -- this is what the machine THOUGHT was the fault.

Hypothesis:


My hypothesis will be proven we can detect all faults reliably with either method.
    Reliably means within the top 5% of fault detection lists.
    What about including confidence values here?
    What if the source is something with high variablility? like free disk space?


Validated experiments:

1) We had Windows 7 already installed (with all latest service packs and patches for December 5, 2013)
2) We installed full .NET framework 4.5.1
3) We checked that we have Internet access
4) We unpacked all files from this archive to disk D:\Work
5) We installed and started latest Microsoft IIS (Internet Information Services)
6) We started binary HMM\wmi_data\wmi_data\bin\Debug\wmi_data.exe
7) After 30 minutes we stopped IIS
8) Fault was detected by the running binary wmi_data.exe with the following output:

found 30 existing .xml data files in data subdirectory; attempting to populate.
failure connecting to host http://localhost. Unable to connect to the remote server
failure connecting to host http://localhost. Unable to connect to the remote server
fault detected. verifying result...
Win32_NetworkAdapter\[00000019] Microsoft ISATAP Adapter\Name,0,9999999999
Win32_NetworkAdapter\[00000019] Microsoft ISATAP Adapter\Speed,0,9999999999
Win32_NetworkAdapter\[00000022] Microsoft ISATAP Adapter\Name,0,9999999999
Win32_NetworkAdapter\[00000022] Microsoft ISATAP Adapter\Speed,0,9999999999
Win32_OperatingSystem\00371-OEM-8992671-00524\NumberOfUsers,0,9999999999
Win32_Service\World Wide Web Publishing Service\AcceptStop,0,9999999999
Win32_Service\World Wide Web Publishing Service\Started,0,9999999999
Win32_Service\World Wide Web Publishing Service\State,0,9999999999
Win32_Service\Adobe Flash Player Update Service\ExitCode,0,0118031870684127
Win32_Service\World Wide Web Publishing Service\ProcessId,0,0118031870684127
Win32_Service\Diagnostic System Host\AcceptStop,0,0118031870684127
Win32_Service\Diagnostic System Host\ProcessId,0,0118031870684127
Win32_Service\Diagnostic System Host\Started,0,0118031870684127
Win32_Service\Diagnostic System Host\State,0,0118031870684127
Win32_Service\Portable Device Enumerator Service\AcceptStop,0,00430655980789432
Win32_Service\Portable Device Enumerator Service\ProcessId,0,00430655980789432
Win32_Service\Portable Device Enumerator Service\Started,0,00430655980789432
Win32_Service\Portable Device Enumerator Service\State,0,00430655980789432
Win32_Service\WinHTTP Web Proxy Auto-Discovery Service\ProcessId,0,000685951917059458
Win32_Service\WinHTTP Web Proxy Auto-Discovery Service\AcceptStop,0,000681267101210941
Win32_Service\WinHTTP Web Proxy Auto-Discovery Service\Started,0,000681267101210941
Win32_Service\WinHTTP Web Proxy Auto-Discovery Service\State,0,000681267101210941
Win32_Service\Windows Installer\AcceptStop,0,000589956055791496
Win32_Service\Windows Installer\ProcessId,0,000589956055791496
Win32_Service\Windows Installer\Started,0,000589956055791496
Win32_Service\Windows Installer\State,0,000589956055791496
Win32_LogicalDisk\D:\FreeSpace,0,000136841729281218
Win32_Service\Application Experience\AcceptStop,1,93402061798711E-06
Win32_Service\Application Experience\Started,1,93402061798711E-06
Win32_Service\Application Experience\State,1,93402061798711E-06
Win32_Service\Application Experience\ProcessId,1,32157297932217E-06
Win32_OperatingSystem\00371-OEM-8992671-00524\NumberOfProcesses,1,11838606159631E-08
Win32_Service\Windows Error Reporting Service\ExitCode,3,2349675575024E-10
Win32_Service\Windows Error Reporting Service\AcceptPause,5,37396902422927E-14
Win32_Service\Windows Error Reporting Service\AcceptStop,5,37396902422927E-14
Win32_Service\Windows Error Reporting Service\ProcessId,5,37396902422927E-14
Win32_Service\Windows Error Reporting Service\Started,5,37396902422927E-14
Win32_Service\Windows Error Reporting Service\State,5,37396902422927E-14
Win32_DiskDrive\\\.\PHYSICALDRIVE1\Caption,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\FirmwareRevision,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\Model,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\Partitions,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\PNPDeviceID,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\SerialNumber,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\Signature,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\Size,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\TotalCylinders,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\TotalSectors,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE1\TotalTracks,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\Caption,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\FirmwareRevision,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\Model,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\PNPDeviceID,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\SerialNumber,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\Signature,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\Size,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\TotalCylinders,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\TotalSectors,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE5\TotalTracks,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\Caption,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\FirmwareRevision,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\Model,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\Partitions,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\PNPDeviceID,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\SerialNumber,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\Signature,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\Size,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\TotalCylinders,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\TotalSectors,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE2\TotalTracks,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\Caption,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\Model,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\PNPDeviceID,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\SerialNumber,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\Signature,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\Size,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\TotalCylinders,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\TotalSectors,2,20309881449055E-16
Win32_DiskDrive\\\.\PHYSICALDRIVE4\TotalTracks,2,20309881449055E-16
Win32_NetworkAdapter\[00000000] WAN Miniport (SSTP)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000001] WAN Miniport (IKEv2)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000002] WAN Miniport (L2TP)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000003] WAN Miniport (PPTP)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000004] WAN Miniport (PPPOE)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000005] WAN Miniport (IPv6)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000006] WAN Miniport (Network Monitor)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000008] WAN Miniport (IP)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000009] Bluetooth Device (Personal Area Network)\Speed,2,20309881449055E-16
Win32_NetworkAdapter\[00000009] Bluetooth Device (Personal Area Network)\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000010] RAS Async Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000011] Microsoft ISATAP Adapter\InterfaceIndex,2,20309881449055E-16
Win32_NetworkAdapter\[00000011] Microsoft ISATAP Adapter\Speed,2,20309881449055E-16
Win32_NetworkAdapter\[00000011] Microsoft ISATAP Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000012] Intel(R) Centrino(R) Ultimate-N 6300 AGN\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000014] Microsoft Teredo Tunneling Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000015] Intel(R) 82579LM Gigabit Network Connection\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000017] Microsoft Windows Mobile Remote Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000018] Microsoft Virtual WiFi Miniport Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000019] Microsoft ISATAP Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000020] Cisco AnyConnect VPN Virtual Miniport Adapter for Windows x64\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000021] Microsoft ISATAP Adapter\InterfaceIndex,2,20309881449055E-16
Win32_NetworkAdapter\[00000021] Microsoft ISATAP Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000022] Microsoft ISATAP Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000023] Microsoft ISATAP Adapter\InterfaceIndex,2,20309881449055E-16
Win32_NetworkAdapter\[00000023] Microsoft ISATAP Adapter\Name,2,20309881449055E-16
Win32_NetworkAdapter\[00000023] Microsoft ISATAP Adapter\Speed,2,20309881449055E-16
Win32_NetworkAdapter\[00000023] Microsoft ISATAP Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000026] Remote NDIS based Internet Sharing Device\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000035] Microsoft 6to4 Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_NetworkAdapter\[00000038] Microsoft Windows Mobile Remote Adapter\TimeOfLastReset,2,20309881449055E-16
Win32_OperatingSystem\00371-OEM-8992671-00524\LastBootUpTime,2,20309881449055E-16
Win32_Service\Adobe Acrobat Update Service\ProcessId,2,20309881449055E-16
Win32_Service\Andrea ST Filters Service\ProcessId,2,20309881449055E-16
Win32_Service\Application Host Helper Service\ProcessId,2,20309881449055E-16
Win32_Service\Application Information\ProcessId,2,20309881449055E-16
Win32_Service\Apple Mobile Device\ProcessId,2,20309881449055E-16
Win32_Service\Windows Audio Endpoint Builder\ProcessId,2,20309881449055E-16
Win32_Service\Windows Audio\ProcessId,2,20309881449055E-16
Win32_Service\Bluetooth Driver Management Service\ProcessId,2,20309881449055E-16
Win32_Service\Base Filtering Engine\ProcessId,2,20309881449055E-16
Win32_Service\Background Intelligent Transfer Service\ProcessId,2,20309881449055E-16
Win32_Service\Service Bonjour\ProcessId,2,20309881449055E-16
Win32_Service\Computer Browser\ProcessId,2,20309881449055E-16
Win32_Service\Bluetooth Support Service\ProcessId,2,20309881449055E-16
Win32_Service\Certificate Propagation\ProcessId,2,20309881449055E-16
Win32_Service\Credential Vault Host Control Service\ProcessId,2,20309881449055E-16
Win32_Service\Credential Vault Host Storage\ProcessId,2,20309881449055E-16
Win32_Service\Cryptographic Services\ProcessId,2,20309881449055E-16
Win32_Service\Offline Files\ProcessId,2,20309881449055E-16
Win32_Service\DCOM Server Process Launcher\ProcessId,2,20309881449055E-16
Win32_Service\Dell System Manager Service\ProcessId,2,20309881449055E-16
Win32_Service\DHCP Client\ProcessId,2,20309881449055E-16
Win32_Service\DNS Client\ProcessId,2,20309881449055E-16
Win32_Service\Diagnostic Policy Service\ProcessId,2,20309881449055E-16
Win32_Service\Extensible Authentication Protocol\ProcessId,2,20309881449055E-16
Win32_Service\Encrypting File System (EFS)\ProcessId,2,20309881449055E-16
Win32_Service\Windows Event Log\ProcessId,2,20309881449055E-16
Win32_Service\COM+ Event System\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) PROSet/Wireless Event Log\ProcessId,2,20309881449055E-16
Win32_Service\Function Discovery Provider Host\ProcessId,2,20309881449055E-16
Win32_Service\Function Discovery Resource Publication\ProcessId,2,20309881449055E-16
Win32_Service\Windows Font Cache Service\ProcessId,2,20309881449055E-16
Win32_Service\Group Policy Client\ProcessId,2,20309881449055E-16
Win32_Service\HomeGroup Provider\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) Integrated Clock Controller Service - Intel(R) ICCS\ProcessId,2,20309881449055E-16
Win32_Service\IKE and AuthIP IPsec Keying Modules\ProcessId,2,20309881449055E-16
Win32_Service\IP Helper\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) Identity Protection Technology Host Interface Service\ProcessId,2,20309881449055E-16
Win32_Service\CNG Key Isolation\ProcessId,2,20309881449055E-16
Win32_Service\Server\ProcessId,2,20309881449055E-16
Win32_Service\Workstation\ProcessId,2,20309881449055E-16
Win32_Service\TCP/IP NetBIOS Helper\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) Management and Security Application Local Management Service\ProcessId,2,20309881449055E-16
Win32_Service\Process Monitor\ProcessId,2,20309881449055E-16
Win32_Service\Machine Debug Manager\ProcessId,2,20309881449055E-16
Win32_Service\Windows Firewall\ProcessId,2,20309881449055E-16
Win32_Service\Microsoft Antimalware Service\ProcessId,2,20309881449055E-16
Win32_Service\MySQL55\ProcessId,2,20309881449055E-16
Win32_Service\Net Driver HPZ12\AcceptStop,2,20309881449055E-16
Win32_Service\Net Driver HPZ12\ProcessId,2,20309881449055E-16
Win32_Service\Net Driver HPZ12\Started,2,20309881449055E-16
Win32_Service\Net Driver HPZ12\State,2,20309881449055E-16
Win32_Service\Network Connections\ProcessId,2,20309881449055E-16
Win32_Service\Network List Service\ProcessId,2,20309881449055E-16
Win32_Service\Microsoft Network Inspection\ProcessId,2,20309881449055E-16
Win32_Service\Network Location Awareness\ProcessId,2,20309881449055E-16
Win32_Service\NMSAccessU\ProcessId,2,20309881449055E-16
Win32_Service\Network Store Interface Service\ProcessId,2,20309881449055E-16
Win32_Service\O2FLASH\ProcessId,2,20309881449055E-16
Win32_Service\O2SDIOAssist\ProcessId,2,20309881449055E-16
Win32_Service\Peer Networking Identity Manager\ProcessId,2,20309881449055E-16
Win32_Service\Peer Networking Grouping\ProcessId,2,20309881449055E-16
Win32_Service\Internet Pass-Through Service\ProcessId,2,20309881449055E-16
Win32_Service\Program Compatibility Assistant Service\ProcessId,2,20309881449055E-16
Win32_Service\PDF Architect Helper Service\ProcessId,2,20309881449055E-16
Win32_Service\PDF Architect Service\ProcessId,2,20309881449055E-16
Win32_Service\Plug and Play\ProcessId,2,20309881449055E-16
Win32_Service\Pml Driver HPZ12\AcceptStop,2,20309881449055E-16
Win32_Service\Pml Driver HPZ12\ProcessId,2,20309881449055E-16
Win32_Service\Pml Driver HPZ12\Started,2,20309881449055E-16
Win32_Service\Pml Driver HPZ12\State,2,20309881449055E-16
Win32_Service\Peer Name Resolution Protocol\ProcessId,2,20309881449055E-16
Win32_Service\IPsec Policy Agent\ProcessId,2,20309881449055E-16
Win32_Service\Power\ProcessId,2,20309881449055E-16
Win32_Service\User Profile Service\ProcessId,2,20309881449055E-16
Win32_Service\Windows Mobile-based device connectivity\ProcessId,2,20309881449055E-16
Win32_Service\Remote Access Connection Manager\ProcessId,2,20309881449055E-16
Win32_Service\RealNetworks Downloader Resolver Service\ProcessId,2,20309881449055E-16
Win32_Service\Macrium Reflect Image Mounting Service\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) PROSet/Wireless Registry Service\ProcessId,2,20309881449055E-16
Win32_Service\RPC Endpoint Mapper\ProcessId,2,20309881449055E-16
Win32_Service\Remote Procedure Call (RPC)\ProcessId,2,20309881449055E-16
Win32_Service\Security Accounts Manager\ProcessId,2,20309881449055E-16
Win32_Service\Smart Card\ProcessId,2,20309881449055E-16
Win32_Service\Task Scheduler\ProcessId,2,20309881449055E-16
Win32_Service\System Event Notification Service\ProcessId,2,20309881449055E-16
Win32_Service\Shell Hardware Detection\ProcessId,2,20309881449055E-16
Win32_Service\Simple TCP/IP Services\ProcessId,2,20309881449055E-16
Win32_Service\Skype C2C Service\ProcessId,2,20309881449055E-16
Win32_Service\Print Spooler\ProcessId,2,20309881449055E-16
Win32_Service\SSDP Discovery\ProcessId,2,20309881449055E-16
Win32_Service\Secure Socket Tunneling Protocol Service\ProcessId,2,20309881449055E-16
Win32_Service\Audio Service\ProcessId,2,20309881449055E-16
Win32_Service\Windows Image Acquisition (WIA)\ProcessId,2,20309881449055E-16
Win32_Service\Microsoft Software Shadow Copy Provider\ExitCode,2,20309881449055E-16
Win32_Service\Superfetch\ProcessId,2,20309881449055E-16
Win32_Service\Telephony\ProcessId,2,20309881449055E-16
Win32_Service\Themes\ProcessId,2,20309881449055E-16
Win32_Service\Distributed Link Tracking Client\ProcessId,2,20309881449055E-16
Win32_Service\Windows Modules Installer\ProcessId,2,20309881449055E-16
Win32_Service\Interactive Services Detection\AcceptPause,2,20309881449055E-16
Win32_Service\Interactive Services Detection\AcceptStop,2,20309881449055E-16
Win32_Service\Interactive Services Detection\ExitCode,2,20309881449055E-16
Win32_Service\Interactive Services Detection\ProcessId,2,20309881449055E-16
Win32_Service\Interactive Services Detection\Started,2,20309881449055E-16
Win32_Service\Interactive Services Detection\State,2,20309881449055E-16
Win32_Service\UPnP Device Host\ProcessId,2,20309881449055E-16
Win32_Service\Desktop Window Manager Session Manager\ProcessId,2,20309881449055E-16
Win32_Service\Cisco AnyConnect VPN Agent\ProcessId,2,20309881449055E-16
Win32_Service\Volume Shadow Copy\ExitCode,2,20309881449055E-16
Win32_Service\Windows Process Activation Service\ProcessId,2,20309881449055E-16
Win32_Service\Diagnostic Service Host\ProcessId,2,20309881449055E-16
Win32_Service\Windows Management Instrumentation\ProcessId,2,20309881449055E-16
Win32_Service\WLAN AutoConfig\ProcessId,2,20309881449055E-16
Win32_Service\Windows Live ID Sign-in Assistant\ProcessId,2,20309881449055E-16
Win32_Service\Windows Media Player Network Sharing Service\ProcessId,2,20309881449055E-16
Win32_Service\Security Center\ProcessId,2,20309881449055E-16
Win32_Service\Windows Search\ProcessId,2,20309881449055E-16
Win32_Service\Windows Update\ProcessId,2,20309881449055E-16
Win32_Service\Windows Driver Foundation - User-mode Driver Framework\ProcessId,2,20309881449055E-16
Win32_Service\Intel(R) PROSet/Wireless ZeroConfig Service\ProcessId,2,20309881449055E-16
Win32_LogicalDisk\C:\FreeSpace,5,90483608189918E-17
Win32_Service\Multimedia Class Scheduler\AcceptStop,6,50521303491303E-19
Win32_Service\Multimedia Class Scheduler\ProcessId,6,50521303491303E-19
Win32_Service\Multimedia Class Scheduler\Started,6,50521303491303E-19
Win32_Service\Multimedia Class Scheduler\State,6,50521303491303E-19
Number of faults not detected by the application: (False Negatives)

9) We repeated the same steps from 6 for ANN and fault was also detected