About : standard furniture metro
Title : standard furniture metro
standard furniture metro
all right. >> [applause]>> it's awesome to see a crowd like this. good afternoon.>> good afternoon. come on, how many people hadbeen to one of these before? all right, soyou know good afternoon. >> good afternoon. >> yeah, there we go, much better. how many people have neverbeen to one of these before?
okay, soi can use the same jokes for those people as i've given before. welcome to case of the unexplained. my name is mark russinovich,cto of azure as paid for job. i'm cto of sysinternals as my funjob that i still have on the side to keep myself occupied when idon't have azure things to do. now, here's the outline ofthe talk that i'm gonna be going through today. first, i'm gonna kinda motivatethe reason for this talk.
i think that shouldn't be too hardgiven there's a lot of people already here. so you probably have a good idea forwhy a talk like this is interesting. and then i'm going to gointo different scenarios, different symptoms to show you howto use various tools to go and identify the root causeof these problems. but the reason this talk is still sopopular and it's been almost, i think, 10 years that i've beengiving this talk at every teched and it forums and now ignites,is that every talk is different.
and that is one reason that somany people come back. but the fact is that mostapplications do a really poor job of telling you whenthey run into a problem about what the rootcause of the problem is. and there's a very simple reason forthis. writing software is a complex taskand there is always deadlines for the developers to hit. so what often gets left behind orall of the funny cases in the code where there is unexpectedcircumstances and
making sure that the code dealswith those in a reasonable way. so, for example, if there'sa registry key that's missing or a file that has permission set onlyin a way that the developer didn't anticipate, the code might not evencheck to see if that's the case, continue on as ifeverything was fine and then end up goingoff into the weeds. for example, referencinga pointer that's not valid, or reading contents out ofa buffer that are garbage, because they were never initializedwith the contents of the key.
and so,the way that the code will often react to garbage like this is thatit'll throw up a dialog box that points at a problem that hasnothing to do with the real cause. it will say, when there'sa missing registry key, hey, you don't havean administrator permissions. having nothing to do with the factthat there's a missing registry key. or it my crash or hang, or just hangindefinitely waiting for something, or hang consuming cpu. i've got a collection of some of myfavorite misleading dialogue boxes.
i'm sure you've seen many like this,this isn't my favorite, i think, of all time. and this i still get fromoutlook every now and then. it's unknown error. the response that it asksfrom me for that is ok. and then it has the audacity to askme if the information was helpful. here's another one,this is from visual studio. i didn't know there was sucha thing as zombie processes. have you seen this?
this is pretty frightening. catastrophic failure. it's like my computer is like orthe world is ending or something. and windows has detected it andthen we just kind of give up on it. >> [applause] >> and then,we try to be funny about it. so those are all examples. by the way, this is whatend users see when they. somebody tweeted thisabout six months ago.
something happened and you need toclick ok to get on with things. certificate andthen there's a fine print, certificate mismatch securityidentification administration communication intercept liliputiansnotweasel foxtrot omegaforce. press button for technical crap. and then there's options forlevels of technical crap. so the purpose of this talk is togive you a look behind the scenes of problems like this so you canunderstand and find out, actually, what the root cause is.
we're gonna take a look at a bunchof tools to look at registry activity, file system activity,networking, and dll and process activity. and one of the key troubleshootingtools for getting to the root cause of one of these types of problemsis understanding a call stack. because many cases, the piece ofcode that throws up the dialogue box really is indirectly to blame forthe cause of the problem. some other dll typically or component has fedit some garbage and
it's just responding as it should tothat and then runs into a problem. and it's really the cause ofsomething that's further up in what's called the call stack. and we're gonna look at howto interpret a call stack so you can see when that's the case. the tools that were gonnaare the sysinternals tools. how many people have usedthe sysinternals tools? raise you hands. all right, how many people havenot used the sysinternals tool?
is there anybody? anybody i need to haveescorted out of the room? so we're gonna be using those. is that a train coming through? we're gonna be using those andwe'll also be using a tool called the windows debugger, which isa tool to look at crashes and hangs. so that's the only non sysinternalstool we're gonna be using. this list isn't all of the toolsthat i'm gonna be showing, but like i said, every one ofthese presentation is unique,
everyone has a different setof troubleshooting cases submitted by you or me or others. and so everyone is unique and some presentations use sometools and some use other ones. so i recommend that if you like thispresentation you go take a look at my blog and sysinternals. and you'll find a listof the links to previous years' cases that explains. now this isn't a veryauspicious time for
me because the 20th anniversaryof sysinternals was last week. so i founded sysinternals withbryce cogswell in september of 1996. >> [applause]>> and you can see a screenshot ofthe homepage, so i designed it. yeah, you're like, who the hell designed suchan awesome looking website? that was me. i became proficient at html before, which is a skill that istill use to this day.
and you can see how hip andcool we were. if you remember surfing the web in1996, you did not have a website that was cool and interestingunless it was constantly under construction, so we had the underconstruction logo there. and there was a party thissummer hosted by sammy leo. how many people have heard of thatthe party or anybody been there? so a few of you, yeah. so this party was to celebratethe sysinternals 20th anniversary. i give a remote keynote via skypewhich crashed in the hotel room
halfway through, soi ended up giving about 20 minutes talk on sysinternals tonobody listening which. but here's a collection of some ofthe special speakers that spoke their on various tools includingone aaron margosis there, how many people heard of aaron? so aaron where are you? are you here? >> [inaudible]>> he's here? darn.
so i can't talk abouthim behind his back. but here's aaron giving a talk. and he told me when he came back,mark, i gave a talk. my talk was the best talk. everybody was sointerested in my talk. and then i was surfing the websiteafter the conference and looking at the pictures, and it was like,here's pictures from aaron's talk. and i'm like, yeah,they look really interested aaron. no, actually aarondid give a good talk.
so speaking of aaron, aaron ismy co-author on a book called windows sysinternals administrator'sreference which came out about four years ago. and it was a tool thatwe worked on together. when i say we worked on together, he wrote almost all of it himself,and then i put my name on it. and this book isthe official reference. and now we've just finished thesecond edition of the book in fact aaron talk a really long timewriting this, so i got more and
more involved as the book went on. i went from emailing him once amonth to once a week to everyday, so that was my level of involvementhow i had to step up my game to get this book out the door. and we got it out the door and he actually, he did a prettygood job on this one. so good that i've decided yeah, surei'll put his name on the cover again this time andactually make it the same size. so this book.
>> [laugh]. >> this book is comingout in a month, literally about four weeksyou can preorder it right now in fact i've got 25% offdiscount sheets right here. that you can go downto the bookstore and you can get 25% off andthen aaron's here so we got a stack of littlethings that we can sign for you so that when the book comesyou can slap on our signatures so after the session come on up to thestage and i'll be handing those out.
all right so,let's get into the meat of it. let's start withsluggish performance. how many people have seen a computerwith sluggish performance? and i'm not talkingabout windows vista. >> [laugh]>> okay, so pretty universally applicable. so this case starts with somebodythat noticed that their system was running sluggishly. the mouse was jittery, processes,applications would take awhile,
to launch. and they ran a toolcalled process explorer to try to figure outwhat was going on. i'm not gonna ask how many peoplehave used process explorer because i assume that i'm just gonnaget a lot of hands going up. but we're gonna spend just a fewminutes going through a tour of process explorer. to understand what it shows you andhow that's different from other tools like task manager andresource monitor.
and how it can help you troubleshootparticular problems like this one. one of the key aspectsof it is that it shows you the processes in a indented way. and this indention shows you therelationship within the processes, the parent-child relationship. so anything that is indented and beneath another process isa child of that process. or a descendant if it happensto be indented more than once. and so you can see someinteresting things just by
looking at the indentation. like you can see wininit here launches a bunch ofsystem processes. including services.exe, which is theservice control manager in windows, that's visible righthere in the process tree. and then down hereyou can see explorer, that's the shell process formy logon session. and so just about everything icreate is created by explorer. when i go into explorer andi double-click on an application.
or in the start menu i double clickon a win32 desktop application, this will be a child of explorer. there are some of those modernuap type of applications that will not belaunched from explorer. we'll take a look at wherethey show up here in a second. so that's one ofthe first things you see. another thing that younotice is that there's color highlighting that you don'tsee in these other tools. so you see some blue processes,you see some red processes.
does anybody know whatblue processes are? >> [inaudible]>> male processes. no, there's no such thing as male orfemale processes. [laugh]>> these are democratic processes. i'm just kidding. >> [laugh]>> it's probably not the right time to makea joke like that. >> [laugh]>> so the blue processes are processes that ranin my user context.
so that explains why explorer andeverything underneath explorer, just about everything underneathexplorer, is highlighted that way. because they're all running as mark,and i'll showing you in a second how ican see they're running as mark. and then the red processes are onesthat are hosting windows services. which explain why there's so manyof them that are under services.exe up here that we werejust looking at. like a services host here ishosting windows services. you can see cpu column,
private memory usage column,and working set. working set being physicalmemory assigned to the process. and then you can see a couple ofother columns here that are there by default, the process id, thedescription, and the company name. the description and company namecome directly from the image itself. so it's not like process explorerhas a database inside of it that it's looking these things up in. they're embedded in what's calledthe version resource of the image. and there's apis that a tool likeprocess explorer goes to pull
this stuff out. developer companies put that into identify what the thing is. you want some more information aboutthe process, you double-click on it. and so i'm gonna explore a littlebit in one of these services, this one right here. and you go to the image tab andyou can see information about it. this is the host, this isdescription and company name. you can see the version, you can see when the primaryexecutable is built, compiled.
you can see when it wasstarted down here, so down here is the start time. by the way the tool,anybody know what tool i'm using? >> zoomit. >> this is zoomit andthis is a sysinternals tool. and zoomit, you can do all sortsof things in it like draw lines, draw circles,draw smiley faces like that. you can type text,i just showed you. and this is a drawing pagethat what i can do is,
i'm in what's called live zoom here. and i can go right from livezoom to annotated zoom so i can just highlightthings on the screen. so this is zoomit and that's what i'll been usingto identify things here. so i mentioned that we cansee the start time here. we can see the command line andthe current directory. and so this can be usefulinformation for troubleshooting. in fact i've got a case in this verypresentation that makes use of this
information. oops. now i'm gonna briefly go through the rest of the columns. not spend a whole lotof time on them because we wanna get back to the. but there's a performance column,this will show you cpu, io, virtual memory statistics, physical memorystatistics, handles for the process. you can see some of thesethings in graphical forms, so you can see a history,its timestamps.
so i can see if this thing hasa memory leak, i'd be able to see how it was leaking over time,this graph would be rising. i'd be able to see if there wasa memory spike, or a cpu spike, when exactly it happened byhovering the mouse over it. there's disk andnetwork statistics, for how it's using the disk and network. if it's a process using the gpuwe'd see gpu usage here, including memory usage andthe actual computation on the gpus. this being a service process,there's a tab here called services.
this shows you what servicesare hosted inside this process, including the display name,as well as the path to the dll. in the case of a service host, it'swhat's called a hosting process. there's other examples, like runthe dll32, you've probably seen. it's notorious forbeing the home of malware because when you look at a toollike task manager. you just see run dll32, you don'tsee what it's hosting, what dll. and when you take a lookat process explorer, i'll show you how it shows youwhat it's hosting inside of it.
now if it's a servicehosting process like this, you will see a services tabcuz it's a well defined way of getting that information. then there's the threadstab which shows you the threads in the process andwhere they started. even tagged by which service it is,so if you see a spikein cpu usage here. you can identify which servicein a service hosting process is responsible.
the tcp/ip tab will showyou the network endpoints, udp and tcp that it's got open. you can see this one hasa whole bunch of them and that's because you can see thatit's got various services there. like terminal service and dnscache,network location awareness service, that all have ports open forcommunication. the security tab,remember i told you how you can see what something's running as,here's one way to see it. i've also got it overhere on the image tab so
that network service, here you cansee, as well here, network service. and then you see informationabout the process token, which represents which useraccount it's belonging to. which we just see up there atthe top, as well as which groups it belongs to, andthen what system privileges it has. so is it able to reboot the machine,for example, that would be a privilegein that lower pane. environment variableswhich can be useful for the case where you've got a programwhere it's gonna read some setting.
environment variable setting, and that's only visible throughan interface like this. what environmentvariables does it have, maybe it's set differently thanother processes on the system. and then finally, strings whichis used for malware hunting. so you can look inside ofa process's image as it's loaded into memory or on disk and see whatprintable characters it's got in it. which can give you an ideaof what it's doing, what it's associated with inthe case of an unknown process.
that's a quick look at that. now a couple more thingsbefore we move on. one is this lower pane view. this lower pane view isthe dlls hosted in the process. and so we're gonna see forthis service hosting process. both two types,memory mapped files and executables. the dlls are executables andthen memory mapped files, like these up here. there's a catalogue database,these catalogue database files that
are mapped in this address space,they will show up here as well. and you can see the samekinds of information for these, like here is the ip helper. information about thatsimilar to what we saw on the image properties so youcan dig into the level of detail. and then there'sa handle table view, these are the open operatingsystem resources the process has. the reason this can be interestingis if you've a got handle leak you can go find outwhat's leaking a handle.
but this also useful in other ways. for example i've gota directory here called can't delete me andwhen i try to delete this. i can delete it. >> that's because i didn'tset up the demo properly. i tested it andi forgot to do something. so let's try that again. okay, so now that shows up again and now when i try to deleteit it says no, i can't.
and trying again will justfail cuz the condition that i just created is still there. now what is that condition? if i do a search tosee can't delete me. what process explorer is doing issearching through all the handle tables and dlls mappings. to find which process hasanything with the resource open that refers to that. and you can see that thatcommand.exe open because its current
directory is in that directory. there's a handle to that directorywhich explains why explorer won't let me delete it. so this is a way to go findout those locked files. there's systeminformation views as well. so what i was showing youat process level, but there's system level information. here specific graphs on cpu,memory io, as well as a summary. and then finally, there'sa couple things to be aware of.
some options. here's one option that i recommendyou do, replace task manager. so if you want to use processexplorer instead of task manager as your process investigation utility,you simply check that. and now process explorerbecomes your task manager. what's your favorite wayto launch task manager? ctrl+shift+escape, that'smy way too. ctrl+shift+escape and there we go,that's task manager now. and then you can set it torun when you log in and
there's various otheroptions up here. so that's a quick overviewof process explorer. back to this case. this user noticed that a scriptwas consuming excessive cpu. in fact, it's that powershellnonsense that's being launched here. i say that nonsense becausehopefully jeffrey snover will watch this. >> [laugh]>> and this is being launched by a wmiprovider hosting process.
there's other ways that you cansee what's inside of a process besides what i showed you,by the way. which is to look at the tool tip. so that service host i showed you. here's another way of looking atthe tool tip to see what's in it. and this applies to rundll32,dcomhost, and wmiprovider hosting processeslike the one that we see there so we can see which wmiproviders on in one of those. in this particular case, the usersaw a bunch of that activity,
here's a blow up of it. consuming excessive cpu,you can see it's 20% on one 50% of their processes just coming andgoing. so they looked at the properties,and saw this, something i didn't highlight andthat is that when they looked at that powershell script they lookedat the auto start location and the image properties and see thatit is launched as a scheduled task. the telemetry host production task,and so to fix this problem whichre-occurred on a regular basis,
they went and looked atthe scheduled tasks and here it is. and what they did wasdisable the task. and they could also see, by the way, that the author isa microsoft author. in fact, this is an internalfrom microsoft corporate user running into ms it scripting gonehaywire on our corporate network. and i'm sure none of you haveever seen that happen before on your corporate networks. because you're probably allthe it people that will deny that
that ever happens. >> [laugh]>> but in this case, it did. and so, what they were able to dowas immediately disable that and contact msit andhave them fix that script. this next case isanother interesting one. a little very differenttypes of symptoms. and i've got a video of this one toshow you what this user experienced. and this was on this oneparticular server, server 2008 r2, that they go to tile view,and you can see that
these icons on these filesare really sluggish to refresh. and then, they go to theirwindows 7 machine or windows 10 machine, they go tothe same thing, go to tile view, and you can say they re-drawjust about instantly. so the question was, what's the differencebetween these two machines, it's the same webdav share, so forthat they turn to a different tool. is anybody familiar withprocess monitor see where is the hand for that?
anybody not used process monitorthis one i expect to see some, and i see everybody is afraidto raise his hand and if turn to that question now. but process monitor is, if youask me which is the most useful sysinternals tool besides zoomit,i would say process monitor. and the reason process monitor souseful is that it solves so many problems where you would justnot even think that a file system registry monitoring toolwould lead you to root cause. it'll show you sluggish performance.
root causes will showyou hang root causes. so it's so powerful thatwhen my daughter comes home, and she's got a homework problem,i have her run process monitor- >> [laugh] >> to try to figure out what it is. it's so useful that about 80% ofthe cases i collect are solved with process monitor. and my co-author onwindows internals, dave solomon, he came up with an expressionawhile back called when in doubt,
run process monitor!. we recently updated it tokeep calm and run procmon! because that's the coolmeme these days. so i've got a shirt that'sthe previous iteration of this, keep calm and run procmon! this shirt you can go ithink it's procmon.aka.ms. chris jackson, who's a friend ofours and aaron's at microsoft, he went anddesigns these t-shirts and now you can go buy your own,keep calm and run procman shirt.
in fact, i think if there's onetakeaway from this session, it is keep calm and run procmon! so let's all say that together. and i know i'm asking youto trust me on this one. so this one is just a leap of faith,just to say it. and then, by the end, i hope you'll be cheering itwhen you see what it can do. so ready, one, two, three. >> keep calm and run procmon!
>> all right, see. that was kinda lame. so hopefully, i'll convince you tobe a little more energetic about it. so let's take a look at the actualfiles involved in this case. and this is an awesome example ofa key troubleshooting technique, which is side-by-side comparison. so we've got a tracefrom the good system. actually, you what i didn't evenintroduced process monitor. let me do that first before wedive in to that particular case.
so i'm gonna just launch it here andi'm gonna just leave it running while i talk a littlebit about what it's doing. the first thing you see iscolumn view, time of day, process name process id,operation, path, result in detail. just basic information aboutindividual operations. for a file system operation, you can see a file systemoperations by the icons here. these are the old filemon from theoriginal sysinternals, nt internals, back in 1996.
still preserve those icons there. filemon which is actually the old,what was it called? how many remember what the windows3-1 thing had that icon? >> file manager. >> file manager? yeah, file manager icon. you can see that this particularoperation's a query security operation. you can see the nextone's a create map file.
you can see [inaudible] createdfile and then the result. so you can see success. and then, the details will show you detailed informationabout that thing. like this is an open, as opposedto creating a new file, or overwriting one. you can see the desiredaccess here is read. you can see that it's open forreading and writing share mode. sharing violations are sometimesproblems that you troubleshoot in
this way by looking at this. and then, you can see that thisactually wasn't performed by the process that this, by the userthis process was running under, but rather by me. so this process was impersonatingme when it went and tried to open this file, as opposedto operating as its native identity. that also can be an interestingtroubleshooting signal. so that's a look, you can seethere's, if we scroll down, registry activity, say regopenkeys.
you can see results. but if you want more informationabout a particular event, including which process it, executed it,you go to this event properties tab. and this will show you all ofthe detail about that event. thread id, class, operation, so a bunch of that stuffshown in the main display. and then,you can see the process information. this is similar to what you seein process explorer on purpose. so the version of the executable,the command line of the process,
the parent id. architecture, the user account,the session id and then a list of dll,it's kinda like the dll view. this is interesting though becauseit's a dll view at a point in time. at the point that this processwindows explorer's running these are the dll's loaded into it. if i went and perform an operationthat cause it to load other dll's, we wouldn't see that here butwe would see those further on in the trace we've actuallysee the load dll calls.
in the traces operations and then we'd be able to lookat the properties and see that deal all showup in that lower pane. so that can give you an ideaof what's actually active in the process around the time of theproblem including diversion number because that is alsosomething that it shows for the processes right there. and then, the last column the stackwe'll come back to later. there's also some summaryreviews that are interesting,
that i'll also show you ina later troubleshooting case. but one ofthe fundamental skills for using process monitoreffectively is filtering. how many events have i collectedwhile i've been setting up here and talking? 2, almost 3 million events just ina couple minutes i've been talking. way too many for us to make any sense of ifwe're looking for a prompt. so the key is zooming in on exactlywhat you think is related your
prompt that could be particularprocess like, let see. how many of these eventsare related to explorer? actually i've got 3million events so this take, it's only gonna takeanother 15 minutes which a, so that's gonna, sowhat would you like to talk about? >> [laugh]>> i'm just kidding, we're gonna cancel that andshorten this a little bit. let me start a new trace andthen highlight this capability. so let's say we justwanted to see explorer.
i just filtered just explorer, let's say that if we just wanna seeregopenkeys that explorer does. and we just wanna seereferences to this key. so we've just set a bunch of filtersto zoom in very quickly on to all activity related toa particular registry key. in some cases, we can't easily findthe process that we're interested in, so let me just do actrl+alt+reset, and so we can go and set a filter for a process ora sub-tree of processes like we wanna see everything thatthis service host has done and
it might be nothing at this point,given actually do explorer. so everything that explorer and its children have done inthe trace that i collected here, i can say add process andchildren to include filter. and this, by the way, shows youa list of all the processes. kinda timelines of when theyexisted relative to the trace. and summary information, the companyname, the owner, the command line, the start time and the end time. so a good summary of whatsgoing on in the trace.
and now i've just seen everything. i'm gonna see snap inis a child of this. explorer that is also activeat that point in time. so really great way to zoomin on particular activity. and that is the basicinterface to the filters. if i wanna do somethingmore advanced, i can open up the filter dialog andyou can see when i did that, include process and subprocesses. what this did was add allthe process ids as include filters
to the filtering. and this is oroperation by column and an and across columns, which is why iwas able to quickly filter down in that particular registry path. and you can see that there'ssome exclude filters by default worth knowing. cuz you're not gonna seeactivity from the system process which host a bunchof device drivers. by default you can turn off theadvanced filtering to see that and
you're not gonna see activityfrom process monitor itself, process explorer, autoruns, thebasic sysinternals tools which you presumably aren't interestedin troubleshooting with this. and then you can set up complexfilters like i only wanna see path begins with c:\temp. and then we can seeif there's anything that accesses the temp directory. nothing did, but if i say okay, and then i come back tothis command prompt,
what i just did was generate a bunchof activity in that directory. so really powerfulways to filter and that is if you wanna be processmonitor master the filtering. okay, back to our regularlyscheduled program. remember, we got the slow web appproblem, how we troubleshoot that? run process monitoring andi've got these two traces here. this is the not good one andyou know what, before i show this, i'm gonna get rid of some columns,that we're not interested in. the time of day column as wellas the process id column,
we don't care about. and now,we've kind of just reduce the noise. i'm gonna open upthe good comparison, so we've captured a process monitortraced on this other one. say reset. and now i've got a side by side. so the first thing he did,by the way, was he said, maybe there'ssomething about the left side not working where it ran into someerrors that the right side didn't.
one of the most common sourcesof problems when it comes to errors that in operationsis access denied. permission is being to strict, andthe way that you can quickly see which kinds of errors show up whereis to do count occurrences here. and say result, soyou can count anything the process, unique processes, unique paths, unique sessions, we're interestedin the results and say, count. this'll scan the trace andshow us information about, and this'll just take aboutanother 15 seconds or so.
this will show us information abouthow many different results types did we see in the trace. and this is something alsoworth getting familiar with, is on a running systemthat's looking okay, what does this typically look like? one of the things you typicallydon't see is a lot of access denied like you see in this one. so you can see bad network path,buffer overflows. those are kind of standard events,cannot break oplock.
but access denied, if we double click,we just set a filter just for that. now we're seeing explorer accessprocmon because you ran procmon so we're gonna get rid of that. i'm gonna say exclude that. and now we see a bunch ofcreatefiles from a service host. and service host is whatwebdav operates under. you can see this pathaccesses to temp\tfs or tfs web dav andwere getting access denied on those.
so you we did is take a look atthe trace from the other system and set a filter for path andthe you can say contains and he said tfs_web_dav, d-a-v. and say, all right, now we haveour two traces side by side. you can see that, this one, there's a successoperation on the same thing. so the first thing you did wasgo and set permissions on that dav directory to alloweverybody full access to it. but the problem didn't go away.
so now he needed to diga little bit deeper. what is the differencebetween the right side and the left side when it comesto that particular operation? for that he openedthe properties dialog box that we looked at earlier. and he comparedthe event side by side. and if you take a look at it reallythere's one key difference and that is that this or besides theaccess in that and that is if you look at this it's impersonatinglike we talked about before.
whereas this one isnot impersonating, but that was like okay, soone is impersonating. i wonder which user accountthis launched under? so he went to the process tab. and he saw that theyare both running s. local service, sothat wasn't the clue. so he kept looking, anybody spota difference between these two? >> [laugh]>> yeah, besides the language, okay, that's obvious.
>> [laugh]>> the command line right, the command line. one is launched,the process launched is -k local service the other-k web client group. so he said, wait a minute,that's interesting, i wonder if that might bethe root cause of the problem. so what he did was go intothe registry to where this thing is configured and set itto launch as local service and rebooted, andthe problem was solved.
so that's pretty kind of amazing,isn't it? process monitor webdavis sluggish on. compare side by side traces,you get all the way down to, you know what the problem is? the process launched witha different service host context on one system. fix that and the problem's fixed. and this took him under an hour,is what he told me. you can see it took him longerprobably to get the video set up
to share with me than it did forhim to actually solve the problem. so process monitor to the rescue. all right, let's take a lookat error dialog boxes. this one is one ofmy favorite cases. why is it one of my favorite cases? because it's my own. so i solve this case andi thought it was so cool that i said it to myself. and i said, hey, mark, you shouldinclude this in one of your talks.
and i said yeah that is pretty cool,so i will include that in one of mytalks, thank you for the submission. this one, my laptop wouldgo to sleep after ten minutes when i would rdp into it. and so the way that i use mylaptop is i take it home and i have my game non domain join nonanything join system there and i rdp into my laptopto do work on it. so it started taking mylaptop home after some release of windows 10 came out.
and it will go to sleep after 10minutes, the mouse would freeze. so rdp window would just freeze and the display on the laptopwith blank at the same time. so like okay,there's something interesting here. so i would see that it wasset in low power mode. and i go andi set it to high power mode. and then it would goback to the same setting. by the way i told the rdp guysabout this and they said, yeah that's probably a problem withthe firmware on the gpu that is
triggering the power managementservice to say that even though the system's in use, it reallyshould go into low power mode. in other words,we have no idea what's going on. >> [laugh]>> so you're on your own. so msit corporate policy was beinghelpful in saying you need to save energy on your laptop, sowe're gonna set this policy for you. so my goal is to figure out how toget group policy out of my way. all right, do not do this at home. >> [laugh]>> this is to demonstrate
highlighting troubleshootingtechniques not how to circumvent your company's corporategroup policy settings, right? man, there's flies up here. they're annoyed withme they're msit flies. so let me show you what i did. i said, you know what,there's gotta be. okay cortana,i don't need you right now. that fly is dive bombing me up here. all right so i said there's gotto be some setting here that
sets the power plan in the registry,in the power options. so here's what i did. i set, let me clear this. and let me set a filter forpath contains power, and added that. and then i went backto this thing and i said if i go intopowersaver what happens? ooh, a lot of activity, but there'sa ton of activity you know what i wanna see just changes to that.
so what i did was setcategory is right. now what is category is right? this category isright what it does is only shows us only the modification,and there it is right there, that control panel applet is sendingthis registry key right here. so, i wanted to see how group. and you can see that when i takea look at who's doing this, it is the service host and it's executingin the system account right there. so the question was,that's the key that gets modified,
that sets the power settings. how is msit,how's group policy setting that key? so what i did was set a filter forthis. and i did this. i said, because i knew that wasgonna probably run this for a day cuz i can't figure it out,couldn't predict when group policy would come and slap thatback to what it was before. i went to filter options andi said drop filtered events which has processed drop any eventsthat aren't part of the filter and
then i just let it sit there. i set the power setting and then icame back a little bit later and lo and behold, i saw a modificationevent by that same service host, a group policy which is calling someapi to go set the power setting which is executing this. so my next thing was,how do i stop this from working? so i did this, jump to. and here i am inthe registry underneath the power proof schemes key.
and what i did was i said. >> [laugh]>> [laugh] >> abby's the local user, and then i said disable inheritance andconvert, and then remove. >> [laugh]>> and. >> so what effect does that have? let's go back to this. i just stumbled upon anotherclassic dialogue box. windows can't make the powerplan that you selected active,
chose a different one. >> [laugh]>> no more data's available. >> [laugh]>> obviously changing it to a different one resultsin the same dialog box. so, i used process monitor tofigure out, no i'm just kidding. >> [laugh]>> so, again that's the way that i fixed this problem, do not, actuallyi'm like inception right here. all right, there we go. so, do not use process monitor forthat purpose.
i just wanted to illustrate it. this next case is one that a lot ofyou probably can empathize with. how many of you are tech support foryour families? and friends? so, how many of you have gottena call from your mother or father saying there''sproblem with windows? can you fix it for me? so quite a few of you. i get these periodically,and this one my mom upgraded
to windows 10 because i told heryou have to upgrade to windows 10. cuz we're going fora billion and every one counts. [applause] >> so she updated the windows 10 and she wasn't pleased with me. >> [laugh]>> cuz this is what happened, and i've reproduced this probleminside that same vm. here. and that is that whenshe would go to outlook,
she'd get a dialog box like this. general failure. well actually,this isn't the reproduce system. she got a slightly different one. she got this operation is beingcancelled due to restrictions in effect on this computer. please contact the systemadministrator. so that's why she contacted me. >> [laugh]>> so what i did was,
i got to fix this. it was like saturday morning. emergency. ticket here. troubleshooting ticket, mothercan not access link from outlook. put that on top of the queue and then i went on the web to check ifother people ran into this problem. i found thisknowledge-based article. which was outlook can't openhyperlinks, was the title of it.
and it had in here this fixit. said, you can fix this problem. just redownload and run this fixit. so i ran the fixit and it ran for afew minutes and then it said fixed. so i tried it and it wasn't fixed. so i rebooted and it wasn't fixed. so the fix it was broken. so i followed the manual steps. and then tried it andit was not fixed.
so rebooted. still not fixed. when it comes to family members andfriends asking you to fix their system, especially my family, wheremy mother brags to her friends about how good i am at troubleshootingher windows problems. it's a source of pride for her. the last thing i could do is go,i don't know what's wrong, mom. >> [laugh]>> so i'm in this. so let's go takea look at what i did.
i ran process monitor. >> so let me clear this and capture a trace of this,and i've got filters here. so i've got a whole bunch ofstuff going on, and what i did, is i figured this has todo something with html. so i did searches for html. and i'm gonna quicklytake you to the source of this problem,which is if i set a filter for outlook, here towards the end of it,
and i came across this, last thing,which is, there's a html shell open new command, default, and i lookover at the value that is trying to read, right before the problem,and i what. wait, what did i see? i saw, if i can get my mouse back. where's my mouse? i've lost my mouse. i saw this. and i said, et tu mom.
>> [laugh]>> i told you edge. >> [laugh][applause] >> well, so what i did was, i said okay, so there must besome problem with that key, so i jumped into the registry andi went to open > html.file > open new command here,and i've lost my mouse. so i've right-clicked andwent to rename, because golden rule,the number two of trouble-shooting, is never do something that youcan't undo, if at all possible.
you never know if you're down thewrong path, and what you're doing to fix the problem actually makesthe whole situation worse. and then you're like, i wish i couldgo back and try something different. what's the number onerule of troubleshooting? >> keep calm and>> very good. >> [laugh]>> well, anybody figure it out? keep calm and run procmon. >> [laugh]>> you're not convinced yet? that's the number one rule.
number two is don't doanything you can't undo. so i changed that and i came back. and try this again andi got this fixed and so if i say internet explorer orwhatever, i fixed it back to edge,like it should be and now she was back in business andher links were working. so troubleshooting a few minuteswith process monitor, and i was able to fix something thateven the fix it wasn't able to fix. this next case is that a customerreported to the office team
that every time they ran office2016 the office 2016 installer ran. and it would eventually launch,but of course having the launcher go fire up every time you launchthe tool is really annoying. and i've reproduced this problem,maybe. there we go, right here. and i'm gonna run process monitorhere and have it collect a trace of what we are watching because thisprompt shows up immediately. you can see that bam, the installerjust fires right up and here in a second visiois gonna launch but
if i repeat this operationi get the exact same thing. this launcher starts up. please wait andthen hear a second later visio. it's like okay i finished andvisio fires up. so, what is the problem here? i captured this with process for thetrouble shooting, the administrator, the support person for microsoftcaptured a trace of this, and they went into the process tree andthey saw, they're familiar enough to know that msiexec has somethingto do with the problem.
so they set a filter wherethey said go to event for that launch of that msiexec. and then,you can see that here's msiexec. they scrolled up to visio, there's a lot of crap goingon here besides visio. where's visio? there's visio. and set a filter for visio. and then,they did a search up in visio.
if you take a look at the callstack towards the end, where that event islet's scroll down here. they went to stack, cuz they figuredthe way that msiexec is getting launched, it's from visio itself. and so, that's why they found wherethe msiexec processes started, and they looked at the process stack. now, the way you read this isthat something in visio.exe called something in vislib.exe,a function there, which called another function,which called another function,
which called another function,which called another function. which then called something inmsi.dll, which is the ms installer dll, which you can seemsireinstallproduct was called. those are symbols directlyfrom the function names. and so, what's happening here isvisio thinks something screwed up and it's saying time to reinstallto fix this problem, and it ends up calling this. and so, what he wanted to find outis what is screwed up that has visio concerned?
and he went scanning up untilhe could find call stack that didn't have msi.dll in it. and here's one. here's the first one. i'd skip, save the trouble ofscrolling through this, but there's no msi.dll. so, he figured somethingaround here is the problem. and what's it doing around here? it's referencing this registry key.
so he jumped into the registry here,and renamed this. and then came back to visio. launched it again. voila, problem solved. so, just a few minuteswith process monitor. having this installer dialog box, gofigure out the root cause by losing a tool that i told youwould be indispensable to have narrowed down on the rootcause, and that is the call stack. call stack led him right tothe problem in that trace,
that it allowed himto fix to the issue. so people think that's kind of cool. >> [applause]>> how many people have had toclean malware off a system? off your own system? in the last two months? in the last month? today? anybody have malware ontheir system right now?
>> [laugh]>> cuz my deal is i'd be glad to clean it for you if we can showeverybody your browser history. have not had anybodytake me up on that yet. i don't know why. all right, this case isa relatively simple one, but i'm gonna use this case as theframework for showing you how you can use the system's internaltools to hunt down malware. this one is an internalmicrosoft dl, says last week my nine-year-old son installeda freeware video editing program.
yeah, right. which i believe had someadware embedded in it. unfortunately my,blame it on your son, okay. ever since,when we're browsing in ie, we regularly get a pane showing upon the left, see screenshot below, with related searches andsomething called middle rush. this seems to be hijacking clicks,and then navigating to various randomad betting crapware sites. needless to say,it's driving us crazy.
and then, goes on to say thatthere's an extensions folder, but deletes that and it comes back. so, he can't figure out howto get rid of this thing. and then, if you takea look at the screenshot, here's the related searches. it's related searchesby middle rush. and then, you can see there'snothing for middle rush here. and here's the directorythat he tried to delete that kept coming back.
all right, so already i seethat this guy, it's like what are you doing at microsoftasking this kind of question? cuz what's wrong withwhat we're seeing here? he should be using what tool? no, snipping tool. look he's takingpictures of his screen. >> all right, so he should, let'stalk about some tools that you can use to troubleshootmalware with sysinternals. one is called process explorer.
and the way that you can useprocess explorer to troubleshoot and find malware is to leveragea giant web database of malware. is anybody familiar withthe site called virustotal? so this is a site that, it's got awhole bunch of anti-malware engine. it's actually hosted by google. they bought the company, virustotal, which i'd had a relationshipwith already. and google,in the better interest of everybody, microsoft partners with googleon security things like this.
and so, they allowed me to continuewith the integration of sysinternal tools into virustotal. but what virustotal is, is a sitewhere you can upload in binary or file, and have it scan it using roughly 50antivirus engines, and report back to you what those anti-virusengines say about this thing. this is useful because some engineshave updated signatures that other ones don't. so it's not always the casethat if lots of antimalware
engines say it's fine,that it's not really malicious. it's just that some of the vendorshaven't figured it out yet. there also could be false positiveswhere one vendor thinks it's bad, but it's really not. and so, this integration withvirustotal is as simple as going to options saying virustotal.comcheck virustotal.com. you say agree to the virustotalterms of service. what this does is it launchesthe virustotal website, and you read that andyou say, okay, fine.
and that's what i did. and now you can see thatprocess explorer has added a virustotal column. let me drag this over here. and with the results, andyou can see that there's a bunch of unknowns,we'll take a look at those a second. let's do this, where we can dosubmit unknown executables. so what this does is submitunknown executables. i can also have right-clicked andsaid submit this particular one.
caution on submitting these. if you've got a real targetedattack on your network, and you submit one of these,the attackers are monitoring what's going up into virustotal,and they'll see that you know something's amiss ifthey see their file up there. so that's something tobe aware of in this. now, if you're cleaning home andcommon malware type, that is a train going by, isn't it? that is, so we'll just wait.
it's probably one of those halfan hour long trains, right? >> [laugh]>> where was i? so if you're cleaninga common infestation, then there's no harm to do this. i almost always find,when i do one of these talks and i do another fresh scan ofa latest release of windows, that there's a false-positivetype of malware update. in fact,i happened to catch one here, here. so this will show you, actually,let me show you how you get to this.
if you click on this, this will takeyou to the report in virustotal, which will show you a wholebunch of information about this, including who scanned it last,and if it's been a while, you might wanna rescan it cuzsignatures have been updated. and so, you might have a new signature that identifiesthis thing as malicious. but if it's recent, you can see thiswas last scanned there on july 28th. it's been a while, so i might wannascan it again if it's suspicious. but for the ones that are positive,
they would show up in red in processexplorer and with a non-zero count. and you can sort, and i don'thave any of those in this list, cuz i've got a cleansystem running here. but for one like this, you can see that 1 engine out of57 reported this as malicious. that engine is called thehacker, which, i don't know,doesn't sound too reputable to me. maybe antivirustotal.com hasa piece of malware on it called the hacker that hasidentified this thing.
so this thing is identifiedthis as some trojan or not. it's certainly a positive, sothat gives you an idea of that. the other tool that integrateswith virustotal, and is one that you would use to clean malware offyour system once you've identified it with a tool like process exploreror other means, is autoruns. how many people have used autoruns? so, quite a few of you. autoruns, it's like msconfig,only it actually works. it will show you informationabout hundreds and
hundreds of the place where thirdparties, and even microsoft itself, can extend windows to launch thingsat various automatic points in time. like when you boot the system,the drivers and services all get launched. when you launch explorer,a bunch of things get launched. when you launch ie a bunch ofplug-ins, browser helper objects and other activex controls andthings get launched. when explorer itself is gonnalaunch its own file extensions, print monitors,local security authority,
add-ins, boot execute systems,codex. so this is grown over time. and also office extensions, theoffice team asked me to add this to it, so you can see office extensionsloaded at a particular point. and malware can often loads here,but this is also good for just general troubleshootingyour office extensions. and for these you can see thati've already configured it to do a scan with virustotal. you can go to options, filter,
say scan with virustotal, andit will automatically scan. and i don't think i've got anypositives in the everything tab for virustotal, butlet me just set a filter here. so options,hide virustotal clean entries. and i do. i've got, so i've got lastpass. and lastpass isidentified as malicious. you can see several have theseobviously false positives, including some ms it vbs scriptsare identified as false positives.
slack updaters identified,probably from the hacker again. we click on that,let's see who's, no. rising detects this thing, so that gives you an idea of what you'dsee if you've got potential malware. for each one of these entries,this shows you information about it, including the place in the systemwhere it's configured. so here's startup inmy startup folder, and you can see that informationlike select something, the version, the time stamp, thesize, and the link to where this is
configured if it's not in the placethat it actually happens to be. if it's a configurationthrough registry, for example, you'll see that here. and then you can do jump to. you can do jump to entry orjump to image. jump to entry will take us tothe startup folder in this case. just like you saw, it jumpedto entry in process monitor, took us to the registry wheresomething was configured. and then, if you've identifieda piece of malware in autoruns here,
then again, what i recommend youdo is instead of deleting it, you disable it here by uncheckinglike you do in msconfig. that way you can come back if you'vebroken something, recheck it. there's one other tool related tomalware that i wanna show you before i move on here andthat is called sig check. has anybody run sig check? now, this one i expectto see far fewer hands. sig check. so this tool is a file verificationutility, file version utility.
and for example, let's go take a look at thatlast pass here in programdata. program manage start up. start menu, programs, start up and if i do sigcheck- e -u -vrs *, what this does, whoops. actually, this is gonna be lookingat the links, not the actual files, so that's actually not gonna work. those are links, not files.
so we'd have to go to thisdirectory right here. so let me double-click. jump to image. and we're gonna take a lookat that apollo client. and that is called apolloclient.exe. and this will do the same thing. the way you run this is you runit against your entire system. and with those switches there,which are on a slide, i'll show you in a second,this will automatically, for
any images or files, if you're gonnalook at anything, that are non-zero, detected on virustotal,it'll take you right to the page and open it up automatically,which i just did. so this takes you to the same pagesimilar to what we saw before. and that. where's my slide show here? that entry is right here. so that's the magic sauce. and i've used this to go andfind the executables,
that maybe even aren't configuredin autoruns across the system, to go find these maliciousfiles that are laying around. because every piece of maliciousfile, every malicious registry key, you need to go eradicate because insome cases the malware's complicated and it might stay somewhat activeeven if you disable part of it. so be thorough when yougo do your cleaning. so in this case,they got pointed in autoruns and here in autoruns thisis what they saw. right inside of autoruns, underservices, this thing had services
configured that were watchingthose directories and then recreating them wheneverthe user went and deleted them. so by going and finding thesemalicious registry keys, you can see that the non-zerovirus total scans over here, they were able to disable those andget the problem fixed and, you can see, haven't hada problem since while browsing. so problem solved with autoruns andvirustotal. all right, our final section here,blue screens. how many people have ever seena blue screen of a windows system?
>> [laugh]>> it's late in the day so i wanted everybody toget a good stretch. how many, so this is problemthat we've got under control in microsoft, we've been workingon this for a very long time. we've got cortana intelligenceassigned to this problem, bots working on it, andthis thing is basically, we've machine learning the crapout of it and now it's gone. so how many people have everseen a windows 10 crash? see, nobody raised their hand.
that's just amazing. >> [laugh]>> all right, how many people have seen the bluescreen in public by the way? yep, those of you that have been to this talk before knowthat i like collecting these things. so, let's go take a lookat my latest batch cuz this is always a lot of fun. delta airlines. so everybody's seen oneat the airport right?
i don't wanna askto raise your hand, the airports are, they're bluescreen magnets, airports are. this one is not a blue screen butit's worth including in this. >> [laugh]>> because whenever you have an atm that is sayingit's found new hardware, you probably want togo find another atm. >> right, this is cafe express. here's a billboard, zara store. this is interesting because it'snt four, this is nt four service
control manager, this is the metroauthority in new york city. still running nt four, soif you need free subway tickets, let me know. >> [laugh]>> this is our latest project, which is really to let you in to avr experience with the blue screen. >> [laugh]>> this is at the, aaron sent me this one, this isi think, where is this aaron? some federal agency siting rightthere in like the visitor entrance. so everybody gets tosee the blue screen.
this one is funny cuzsomebody sent me this and if you read the text he's like yeah,i was in the train station here, i think it's in zurich, andit's like i saw this blue screen. i'm like, i'm gonna takea picture and send it to mark. so he goes and he's taking a picture of it and oneof the staff comes up and says, hey. yeah, we've got anotherone over there if you wanna take a look at it. >> [laugh]>> they thought he was
from it, and so. here's the otto bar. >> [laugh]>> that's never a good sign. sam's furniture courtesy phone. this is here,you can see this, right there. so, kind of snippetsof blue screens here. hertz, blue screens, socan't get you a rental car, sorry. this one is one of the coolest ones,it's like artwork. i don't think that theyintended it to be artwork but
it could be artwork,it's like modern cool artwork here. this is a big, long,stretchy one at a mall, you can see fossil behind there,so multi screen blue screens. this one's in a movie theater. >> [laugh]>> nike. this is in another movie,imax movie theater. hard rock cafe. this one is not on butit is somebody running ps kill. at the hard rock hotel and casino.
iris scanner, oid porter. this is not fake. has anybody played this game,it's call of duty? >> [inaudible]>> what is it? >> [inaudible]>> what? >> [inaudible]>> tell it? i can't. >> [inaudible]>> division? it's division.
all right.so it's the wrong game. i don't play this game. but this is division. this is an actual screenshot from in game. they put a blue screen onthe terminal inside the game. >> [laugh]>> so it's made it into popular culture. this one i got to show you. [music]
>> so this is armin van buuren playing here in atlantaabout six months ago. blue screen during his performance. i don't think he did that on purposeas part of his presentation. >> [laugh]>> so this is kind of like the latest thing thatwas happening lately. i'm sure you've probably seenstories of this one showing up. >> [laugh]>> not really blue screens but hey, it's time for you to upgrade.
we need a billion, so here you go. and, >> so that's my latest batch. >> [applause]>> all right, so when you have a blue screen,it actually just takes a few minutes to get a potential lock onexactly what's causing it. it's really straightforward, and the built-in support that's in thedebugging tool makes it possible. you need a few things. you need debugging tools forwindows.
you need where the bluescreen dump is. so when windows crashes, it saves adump file that contains the contents of memory, partial or full, ofthe system at the time of the crash. and you need symbols for windows. and once you havethose three things, the rest is pretty much automatic. maybe a little bit of manuallooking to find a hint, but it's pretty straightforward. so where do you get the symbols?
you can see that windows debug,you can figure the symbols server to point at the microsoft public symbolserver, there is an internal one if you happen to be on microsoft corpnet for the part of it symbols. then the dump file is as simpleas going into memory windows and seeing if there's a memory.dump,if there's not, go into mini dump, and you'll find a dump file there. that's the rule of thumb, never have to memorize when you'regoing to see one or the other. just follow those steps.
and then load the dump. all right, let's talk about a casethat we've got to look at here and how somebody sent this to me, just ran into it when they wereupgrading their windows 10 machine. the machine immediately wentinto a boot blue screen cycle. boot the machine, blue screen. after the dump,the system reboots, blue screen. so they booted successfully, managedto get the system to boot into safe mode, and they startedenabling services one-by-one.
and when they enabled the firewallservice, bam, blue screen. so they contacted me asking forsupport from the windows team, even though i'm not inthe windows team anymore. actually, this was a friend of mine. and i said, you know what? i'll handle this. so what i did was loadthe dump into win debug. and so i'm gonna go load the dump. it's right here.
and what this is gonna dois load the symbols for the images that it sees in the dump. and then, it's gonna have a pointer to whatit think might be the root cause. but i recommend that youignore what that is, and always execute banganalyze-v. and sometimes, this shows upa hyperlink you can just click on. and i don't know why this is takingso long to load the symbols. this should only take a few seconds.
let's give us another few seconds. if this doesn't work, i'm gonna go ahead and show youthe screenshot of what i did. >> run procmon. >> run procmon, yep, there we go. >> [laugh]>> that's good. all right, i don't knowwhat's going on here, so let's switch over to our can view. and when i loaded the dump and i dida !analyze -v, you see a call stack.
and the last items on the callstack are the call to crash the machine here. so you can see righthere kibugcheck, that is the call that saysblue screen the machine. and the way you read it is the sameway that we talked about before. well, these are garbage up hereat the top becauase windows can't understand parse to call stacks, but this called this which called thatwhich called blah, blah, blah. in this case, there's a numberof images in the list.
we see the firewall client,which explains why up here, which explains why when you turnedon the firewall, the crash happened. we see some ndu thing,firewall client, tcpip stack, netio, ndis, which is partof the networking stack, and then we saw this dnelwf64. and so this basic rule of thumb is,if there's a third-party driver on the stack, it's almost alwaysthe cause of the third-party driver because microsoft never makesmistakes with its codes. so- >> it's never a microsoft component
and actually, that's not quite true,but it's almost always true. if there's a third-party driver, it's almost always the causeof third-party driver. and this is one of thoseexamples of a third-party driver. so what he did was take orwhat i did. sorry, it's i did lmkvm to lookat the version information on this thing. do you remember,this dump is from earlier this year. and when i looked at the timestamp inside the debugger,
this information is oftencaptured as part of the dump so i didn't go have to askhim what the version was. you can see that it's from 2011. and when i did look up on the web, i found that this thing isa citrix vpn client driver. so i told him, hey, you've gota old citrix vpn client and your it department hasn't updatedyour laptop for whatever reason or you are on your own on this one,so go upgrade that. he upgraded that, problem solved.
windows upgradeproceeded as he expected. so a few minutes in windbg. steps that anybody can follow. and figure it out, it's a citrix vpn client that'sancient, causing his blue screen. and got him going again. so this one. by the way, one last thingbefore i wrap it up here. this blue screen right here,i posted on twitter.
this is windows 7, and this was back when we were doingthe free windows 10 upgrade. and i posted this, saying, here's what the latest bluescreen in windows 7 looks like. you can see this text atthe bottom says, experience fewer crashes by upgrading free towindows 10 now with a hyperlink. >> [laugh]>> to windows.com. so i thought it was fairly obviousthat this thing was a hoax. so i posted it on twitter, i'm likeha ha this is kind of humorous, and
then i started to get this. >> so that person was suspicious. >> [laugh]>> and this person got it. and then, i got a pr noticeinternally at the company. reporter saw the tweet, i assumeit's a joke but given the source, it's a little confusing. i checked windows 7 andi don't see this behavior. but just wondering if thisin the works or just a joke? i got in trouble with pr forthe tweet.
>> [laugh]>> but that was, i thought, pretty funny. all right, so we're hereat the end of the session. what i showed you is use of thetools, looking beneath the surface of these error messages wasmy goal at the beginning. when you get one of thesefunny error messages or sluggish behavior, a hang, use the tools to go figure outthe root cause, and that root cause might be something where you needsomebody else to fix the problem.
where you can work around it, likeyou saw me work around some problems or you can actuallygo fix the problem. pointing you at some otherreferences, like i said, i'll be handing out those25% discount coupons. if you have a case please, recordscreenshots, save the log file, send them to me so i can put theminto future case of the unexplains. and before you leave, i hope that i've convincedyou the power of procmon and we can all say it together onelast time before we wrap it up.
>> all right, that was pretty good. so have a great rest of the night. >> [applause]>> thank you for