January Cyber Risk Wednesday: Cyber Resilience through Measurement

Transcript

good afternoon everyone thanks for joining um I'm Jay Healey I'm the director of the Cyber statecraft initiative here at the Atlantic Council and this is the next in our series of cyber risk Wednesdays uh we held hold these every third Wednesday um of the month to talk about just different aspects of systemic cyber risk um in partnership with Zurich Insurance uh we're doing a project look at systemic cyber risk uh we feel that when we talk about cyber risk we tend to be um looking at each organization as if it contained its own risk um we look we tend to think of cyber risk one organization at a time ignoring the way it's interconnected uh globally ignoring outsourced ignoring counterparties and partner Upstream infrastructure external shocks disruptive Technologies and the like we think this is very parallel to how we looked at Financial Risk prior to 2008 you'd look One bank at a time how exposed are you to American mortgages uh and we realize that the interconnections were at least as important as the risk profile of each individual bank so we decided to have a regular series every third Wednesday to get people together that was interested in talking about cyber risk uh our last one was on the risks and resilience of the electrical sector our next one which is going to be third Wednesday in February which is February 12 um third third uh Wednesday in February uh and we are hoping that is going to be on Financial Risk and resilience so we'll be uh trying to get some of the leading lights there um like Phil venables or or Charles bowner or Byron collie or or Errol Weiss or others to to come and and help us understand that Brian pretty from treasury today though um we feel like when we talk about electrical risk and resilience Finance risk and resilience um those are interesting topics but they're the obvious topics they're the things that we consider actually kind of surface when we're thinking about the problem and we wanted something for the professionals the people that are trying to that have been in the field longer um because we feel that when we're talking about risk and resilience it's tough to know exactly what we're talking about because we don't know what we're measuring and we haven't had good metrics for this and this has been an ongoing problem for decades that when someone say well how do you measure risk we say well it's either wicked bad or it's wicked wicked bad and and we're we haven't really gotten much farther than that for the past 20 years it sometimes feels like I'm happy to say that that is only the surface that there is good work that's being done and that's what we're here to talk about today today um we're going to start with a uh conversation with Dan gear who is joining us by phone um up in New England I think he's dialing In from from Harvard uh Dan can you hear us okay I sure can perfect thank you so we'll be with you in in one moment Dan um you have the bio we're not going to spend a lot of time on introductions um but once we're done talking uh with Dan for a bit um we're going to go to the um Wade who's going to talk with Verizon has done fantastic work um that many of you I'm sure familiar with in the investigations report that weade overseas um which can get in some of the really good technical stuff of what we're actually seeing um and metrics and measurements on what they've actually been seeing fantastic work um Yodo J JP C and aper um both an interest in hearing from your perspective in what Sears are up to but also you've been doing work with oecd on coming up with Statistics and trying to get the oecd countries to agree on on what are the good metrics that we should follow um and then Kevin Sullivan from Microsoft and I don't know if you're familiar with Kevin's work um but it's wonderful because it brings together the geek and the won it brings together the the facts that we can know on the technical level and bringing that together with social science statistics and things that we learn in polysi classes and it's the a lot of the work that you're going to see um and that we're going to talk about are stuff that you're going to go ah why didn't I think of that and it seems obvious in retrospect but it took leaders like those here and on the phone to really push forward and and make a difference here and we're happy to help them give them a chance to talk about their work and their companies um but to kick off is Dan gear um and uh many of you might be familiar with Dan's work um you know you have read Dan's work if you read it and you know you got smarter but you still are that 5% confused because it's that damn smart um and uh so Dan you've been looking at at at this issue for for a long time um and uh I I've got a couple of questions that I'm interested in um you had talked about how we need to start from a Culture of Fear to a culture of awareness and then a culture of measurement and and I love that that process that you talk about there so maybe can you get can you give us a couple minutes on on how you see metrics and measurement sure uh happy to do so and hello everyone I'm sorry I can't be there in person but um don't have that Star Trek transporter thing up in working yet um the everybody who has been at this for very long um anybody who comes in front of you and says I've been U working in security for 20 or 30 or whatever years there's two things uh you should ask one is well if you've been at it that long why haven't you fixed it and I don't have a good answer for that so you know sorry um and the other is if you've been at it 20 30 40 years kind of thing um you obviously were not trained for at the outset you're trained for something else and uh for what it is worth and probably the reason that uh Jason wanted me here and so forth was that oddly enough I was trained as a statistician um and in that regard um actually a biostatistician so of course that colors how I look at things and you should be aware of that you know word of Statistics advice here all data has bias the question is can you correct for it well mine is um that I tend somewhat naturally to think in terms of disease models in public health and those kinds of things there's certainly a lot of other ways to think about it um if you're a civil engineer why do buildings fall down um if you're a lawyer how do I make policies that are actually enforcable um um and how to enforce the policies that I think I have um you know there's a lot of kinds of prep um and I think one of the things that's fascinating about cyber security is almost any hard thinking um skill that you already have can probably find some application here so nevertheless um my bias and hence the way for you to correct for it is that I come out of the public health World um that was a long time ago but nevertheless um the quote that um Jason gave is one I wrote quite a long time ago um and I stand by in the meantime um progress has happened and in particular um there are very hard questions that require measurement that we are as yet unable uh unable to approach I'll leave those aside for the moment because the ones that we can approach of course are the ones we should be doing um cyber security is is full of of folks who quite naturally um want to boil the ocean on the first go that isn't going to happen the kinds of measurements that I think are relevant here and in fact the whole reason that I would have a regime of of security metrics at all would be for decision support none of us here I suspect are natural scientists like Darwin and the galpagos or somebody trying to identify new species in the jungle um instead why would we measure this stuff and the answer is because if we don't measure it we'll be we'll be Dumber we'll have make Dumber decisions and I'm quite insistent on that um call it a bias if you like but nevertheless I'm quite insistant that the reason we're trying to measure is for decision support So to a degree measurements that don't contribute to decision support um don't interest me as much as ones that do that makes them sometimes idiosyncratic um what is of interest in financial services might not be of interest in Pharma what's an interest in Pharma might not be of interest in um um provision networks Etc um but that's fine that's that's perfectly fine what we do measure um again I'll just be a statistician for a second um whatever you measure U measure it as consistently as you can in other words make the instrument of your measurement be as dis disappear into the woodwork as much as possible um I will point out that if you are indeed interested in decision support uh quite often an ordinal scale is good enough um a nominal scale is fish and bicycle an ordinal scale is ABC um there are other scales but let's just leave it there for the moment an ordinal scale where you you say what which is bigger than what you know is this bigger than this and and that in turn is bigger than something else that's often good enough for the first round of cyber security measurement and sometimes good enough for a long time one of the things that bedevils Us in this field and I'm sure everyone here is aware of it if not uh regularly troubled by it and that is that we don't have really good stable definitions that is to say if I walk into a large room and ask everybody to write on a sheet of paper their definition of a vulnerability um I'll probably get n over two definitions whatever the size of the room is I'll probably get half as many definitions as there are people in the room or something like it even though uh nist for example has done a lot of work on trying to come up with good and stable definitions uh that again encourages me to say that what we're measuring for is decision support not Natural Science um and in particular if you have an in uh an instrument for measuring much of any thing uh and you're not altogether sure how good it is but at least you're sure that it is um measuring a component of what you think you're you're after like for example how many vulnerabilities do we have like for example uh what is the rate at which um our fir walls uh fire Etc um as long as you have something like that where whatever the error is the error is not correlated strongly with what you really think you're trying to get at then I would submit that Trend analysis is generally good enough for trivial example but absolutely trivial example if you walk out on the street and ask the first polican you say how much cocaine is for available for sale in the city there's no way on Earth they can measure measure they report to you a number and there's no way on Earth really that they can measure it but what they can do is tell you whether the price is going up or down and if the police are trying hard and the price is rising you know it may well be that they're having an effect if the police are trying real hard and and the price is falling then it is probable that whatever the police are doing is not um particularly important um that is in a sense what I mean by Trend analysis and decision support and when um I know that Wade um will be speaking in a moment um I look at the dbir um his work product for what are the trend lines I look for a lot of things at what are the trend lines let me suggest a couple of questions that I don't know how to answer but the trend lines would still be good enough and that is for example there is no doubt that in the last 10 to 15 years we that is to say all of us in this field have made um exploits harder uh there were there was a debate only a couple of days ago between a guy who's really good at making exploits and selling them and a guy who's really good at uh cyber security and crypto that is to say Dave Vel and Matt Blaze and one of the questions is are we getting good faster than the position is getting good are they are the exploits that are being put out improving in their um their effectiveness as fast as we are improving in our defensive Effectiveness are they going faster they going slower I'm not quite sure how to measure that but that's an example of where whatever that trend is I would find instructive for the purpose of making decisions you know everybody likes to ask you know how much should I be spending on security that's a wonderful kind of question you know how much should I be spending it is you know a fundamental question the kind of question that you know CFOs want to know the answer to if nobody else but I think lots of people want to know uh the answer well when you think of that in the way that I'm trying to describe you will frequently run into folks who want to do something along the lines of quote cost benefit analysis you know if we do this we get this benefit if we do that we get another benefit the problem with cost benefit analysis is it requires you to to calibrate the benefit in the same units as the cost or vice versa in other words they have to be on a common scale so that you can say I spent let's say $100 for of cost and I got $200 of benefit things things are good that's a kind of ratio that that doesn't tell you what zero is doesn't tell you what Infinity is but it does tell you am I making am I making an impact on the other hand um be there are certainly questions I'll pick one from Healthcare what's a human life worth that where doing a cost benefit analysis is pretty miserable because now you're going to argue about what's a human life worth more than anything else more than mechanisms you're going to argue about the value of instates and and cost benefits suffers from that that's not new news but it does on the other hand cost Effectiveness where you say I'm going to spend $100 what's the best I can get for it in other words you take out of the equation is the benefit worth the cost and what you get instead is I'm going in other words if it's the benefit worth the cost and therefore should I decide to expend the cost instead you say I'm going to expend the cost what's the most I can get for it and again if you look in public health which I am not a practitioner of anymore but if you look in that what you find is something called quality adjusted life years which is to say that sometimes you actually do have choices between would I rather have uh 20 years in bed or five years walking around um you know and and which would you rather have and that idea that we're going to measure Effectiveness on a scale that doesn't require an absolute doesn't require a zero for the scale and doesn't require an Infinity for scale but we're just going to play in the Middle where you know ratios somehow matter um that I think is the kind of thing that we would generally speaking be looking for there are a lot of people doing observational work um you know anybody Netcraft for example anybody who's got a regular body of data they see and and can describe to others in the aggregate what it is they're seeing is a good thing to do I think that consultants in general people who are in a position to know what's going on behind the curtain so to speak any anytime they get big enough they ought to copy what Verizon does namely if you have enough customers where you can say if I pull the data no individual customer is now identifiable pull the data and Report how it is going over time that is not going to be again like discovering a new species in the cagus instead it's going to tell you is the trend going up or is it going down is is is is the ratio of these things going up or going down I mean one of the things that was fun to look at a while ago was for malware how many mechanisms did it have to enter a computer in other words how many mechanisms did it have did it have one tool did it have two tools have 23 tools early on everything had one tool um later on it got bigger and bigger and bigger then there came a stage at which the number of tools that had uh leveled out but the percentage of the tools that were denovo versus the percent percentage of the tools that were oh I know if if it's been broken into by this other tool they leave a back door I'll just use the one that's already there in other words uh piggybacking on anybody else have been doing that's very interesting that is very interesting because it tells you something about our opponents our opponents of course do not have any desire to invest more than that more than is necessary any more than we do they don't need to invest more than we do and one of the things you might ask is is the skill level repres presented by the tools that are currently in circulation is that growing faster or slower than it used to I mean that's the kind of thing it's hard to assess mind you but it is a useful timeline measure let me give you a slightly different example uh Steph J gold uh who was a paleobiologist at at at Harvard invented a phrase early in his career which stuck in his in the language of his of his field and that is punctuated equilibrium the argument was that uh Evolution does not occur as some steady rate it is not a 8% up slope um uh year in and year out rather it's long periods of nothing followed by short periods of Rapid change followed by long periods of nothing followed by short periods of Rapid change you know roughly speaking when a useful mutation occurs everybody else has either got to copy it or they got to die and and that idea of punctuated equilibrium I think applies to us and in fact I'll demonstrate that and again this is measurement kind of demonstration I suspect all of us owe our career to the fact that was it in Windows 31 I forget exactly when but when Microsoft introduced a TCP IP stack into windows that was an excellent thing to do we should all be grateful that they did it as it happens it was introduced into a system that was designed for a single owner operator on a private Network at most all well and good I'm I'm not arguing with them in any way about that it was something that had to it had to happen and it sure beat the other alternatives for networking that were available at the time nevertheless if you go to the CT back when the CT as in cardie melon was the only place that could report uh break-ins to the C was keeping a log of all the break-ins that were reported to it if you look at their timeline which they've since stopped doing by the way but if you look at their timeline what you see is a spike in the second derivative of the rate of reports of of invasions why because the the appearance of a TCP stack created an acceleration much like lighting the solid fuel on the space shuttle you don't feel anything at first and you say is that all there is but pretty soon you pin to your seat and Away you go and you don't turn it off second punctuated El libring punctu like that is there was a period I think it's as much as six years ago now maybe just five and I don't have a way to measure it directly but I can say this what it was was there came a point at which we had actually made exploit too hard to do as a hobby and suddenly the people whose pay if you will for doing exploits was bragging rights were went out of the picture and the people whose pay if you will was actually something more substantial like dollars entered in so we had this kind of seemingly seemingly instantaneous turnover from adventurers and bragat to professionals why do I know that and the answer is I know that because if you're a bragger and you discover something the shelf life is such that you have to demonstrate it in public so you can get the bragging rights while they're still yours before someone else invents it on the other hand if you're professional you discover something you don't share and so the percentage of in of attacks that were in some sense zero day they were unexpected began to rise and why because we had gotten good enough that the amateurs who announced went away and they were replaced by by professionals who don't we're in the middle of I I think we're in the middle of one now it's always hard of course to see the knee of a curve when you're in it but I think we're in one now um and that is where the general purpose computer that we're used to thinking of as a computer at least people my age are used to thinking of as a computer um sort of as likely to go away as a consumer durable and when that happens the question is well where is the data and the answer is well largely not on the smartphone largely somewhere else and so there's some sort of equilibrium punctuation there where here to four how do you protect your data and that is perimeter control around where the data is and you you man the perimeter and it is your wall and it is your that's inside your wall Etc we're going to change the model there and we're going to change the model in a in a way that's not stoppable I'm I'm not here to argue against progress I if I can if I can for a second be political the great conceit of conservatives is that they can stop progress the great conceit of liberals is that they can manage it um both are false um we're going to see what happens I guess is the way to put it but I would like to find measurements yeah yeah no go go ahead and I just get a question in a sec go ahead and finish the point yeah I'd just like to find measurements that allow me to see things like that when is a when is a change that has long-term implications in the meantime if you can measure something measure it Whatever It Is measure it if you find a better way to measure it run both of them side by side for a while so that you can calibrate the new one against the old one and you don't eat your Actuarial tail um and Jason I'll let you I'll let you interrupt sure go ahead well so it was it's probably been about 10 years since you put your slides online on measuring security with which going to recommend if if you're interested in this topic Google gear measuring security it's 400 slides of everything that you'd want to know about this um and it really it's it's it's a textbook on on how to do this and when I first read those I said oh well this is great we've solved this you know there's all these great stats there all these different things that we can borrow from epidemiology from Finance from decision modeling and support um the finance um but I sometimes feel like we're not any further along um there's one a great example when we were when the Morris worm hit in 1988 Cliff stall estimated that a cost anywhere between $100,000 and $10 million um was the cost of that cyber crime two orders of magnitude and when we see estimates of cyber crime today you still see about it seems about two orders of magnitude um how how come you think we're we're not farther along is it is it just our problem is risk managers or is computer Security Professionals that we're not using the tools at our disposal is it too much of this is just unmeasurable or do we just suck at it a great deal of it's unmeasurable um again I'm sort of stealing wage Thunder here but one of the things that they found is that the majority of data thefts for example are discovered by Third parties not the victim mhm why is that possible if you steal my car it's not like I won't notice and the answer is because if you steal my data I still have it um why are there why is there bad stuff embed in good software and the answer is that the good software the bad software still passes the acceptance tests just like the good software it just has extra features you didn't know were there part of it in other words is that kind of complexity but let me try something slightly different if you check into the hospital with um you know if you're sick and you're checking to the hospital you have a great deal of medical privacy unless what you show up with the small pox or the plague or Shaga disease in which case you don't have any medical privacy because there those are reportable diseases socalled and you have to you have to be um you lose your medical privacy and everybody swarms all over you and frankly all your contacts get swarmed all over as well and so forth of course the question is where did this come from and it spreads um not to get carried away but what who does what does Eola kill the answer is the family members of the first infection the here's a question for the group in some sense the very thing that you were getting at there Jason which is one of the examples of we don't know X I don't know how I feel about this and I've thought about it a lot and I can make I can take either side of the debate but do we want or do we need a mandatory reporting regime for cyber break-ins you know if you crash a plane the NTSB shows up if you show up at the hospital like I said with um you know typhoid the Public Health Service shows up do do we want that here because part of the problem with silent failure which is the and I think silent failure is cyber is about the only place that you can get silent failure at the level that you get it we don't learn from it when that unfortunate pilot yanked the tail off the plane a week after or a month after 911 and Christ into into Queens we learned from that nobody's yanked the tail off a plane since then because we learned several things about that um and the forensics involved was important um I think the issue here is that we're having a hard time learning from experience and part of that is indeed measurement and I'm not sure whether it's we don't know what to measure or to measure would require a level of policy change and surveillance and all those things that are frankly in the air right now but we're having a hard time learning from experience I that I just I just don't know what else to say about that yeah and and you've quoted um other source of jti that talks about a good security metric the characteristics of good security metric it has to be consistently measured cheap to gather especially automatability has a unit of measures like dollars expressed as a number rather than as adjective in relevant to decision making yeah that's Andy jakew with and that book is still good um y i I'm I've got nothing to argue with there those are those are great characteristics they're almost metametrics on metrics themselves you know so I I really like that I would submit I would submit that Perfection you know the usual argument that do not let the perfect be the enemy of the good um if you have a way to measure something that is arguably relevant I would suggest that you do it um particularly if it has the characteristics that you just described and um well again speaking as statistician get the data you can always throw it away and uh so we're going to go to turn to a Wade in in in one sec but there's one there's one last point and and that was what you were talking about the mandatory reporting and I think we'll we'll have that'll that we'll work that into the conversation here but it really strikes me because I've I've done work with uh Consulting for dud and and other parts of the government and they love the public health they want to know about the public health model and how that might apply to cyber and by that they you know they love talking about cyber hygiene or other Concepts um but you know having read your slides and and talking with you I I try to get across actually if you're talking about public health um the understanding and measuring and not worrying about causality if you can be effective without understand causality um you and how much measurement and mandatory reporting fit in is being being heart of the public health model and I find that government folks done no we don't mean that part of the public health model we don't mean the part that epidemiologists think is important we want to talk about cyber hygiene and um so I just I found that as an observation I don't know if you have any any counter observation or I think you're absolutely correct I think you're absolutely correct if I can be a public health guy for a second which is this is the first time I've been a public health guy in years if I can be a public health guy for a second part of your problem is that a great deal of infection is occult you know um subclinical infections um of the common coal virus are pretty common that's how it shows up in people who don't seem to know anybody who has a cal now um we have a lot of subclinical infections in this space uh Chris weop at verod uh humorously but I think he might be correct uh notes that in all probability a machine that has become part of a botn net is under better security control than it was before because before it obviously wasn't resisting the botn net entrance and afterwards the botn net owners are better at this than you are because they view it as an asset to a degree that you don't evidently don't by observation um the the occult affection uh aspect maybe is the place that we could begin with that if if you're the dod or whatever now Jason correct me if I'm wrong if you're in the defense industrial base there is a regime that does matter in this regard yes I'm I'm getting I'm getting some yeses some you I mean uh there there might be a regime and whether it's effective uh okay so uh that that's a great that's a great kickoff and Dan we'll be coming back to you not least to hear about the index of cyber security um which is which is one of your one of your efforts on this um but I wanted uh you you did a great introduction for Wade and the and the DB Wade can you can you I would love to hear about the the the dbir and sure and I'll uh I'll try to tailor it to I think just the topic of measurement in general so if you're if you're not familiar with it uh you can find it just Google Verizon or beinging or whatever Verizon and data breach investigations report and in a in a nutshell we take forensic investigations and figure out who did it and how they did it and how systems were affected and how much data was lost and what kind of data was lost and how victims responded and how quickly they responded and all of those things and we take data points hundreds of data points on each each case each incident and we compile those and and share them every year and over the last several years uh the first two years we did it it was just Verizon's forensic practice uh incidents that we were responded to and over the last several years we've we've started collaborating with with other organizations around the world U so the first partner we had was the Secret Service and they do investigations from a law enforcement perspective and sometimes we support them at the forensic angle but so do other forensic providers so uh they provided their their case log uh and we started working with some of the other Global law enforcement agencies Global Sears as well um and in the latest version of the report which we republished last April April it was uh 18 I think organizations in total from around the world and that that included again law enforcement agencies forensic providers um Global uh in-country uh certs and a few other service providers in addition to to Verizon like deoe does forensics as well and they they contributed to the report I think that's important to get to get people that kind of compete to yeah I do too I do too and uh I I was going to talk about that in fact and um um so anyway I went too far into what it is that's that's what it is it looks at at what's going on in the world uh in terms of security incidents and and this this path got started it's it's interesting um Dan was a uh statistics background and and bio statistics I I started out in the field of biology and and I remember the first project that my professor put me on which was I hated it um but he was working on this project where he was trying to figure out if an infestation of zebra muscles which was a um you know a non-native species into the these Lakes were killing off the invertebrate population in these Lakes so my job as the as the research helper was to sit with a macroscope and count invertebrates out of Petri dishes and that's what I did like for hours on in and as terrible as that job was I don't view it as all that different than what I do now strangely enough because you know I'm always staring at my computer and I'm just picking little bugs apart from incidents and you know taking them out of the Petri dishes and Counting them and putting them in other Petri dishes and classifying them and putting them on the shelf and and and so maybe it helped in in some reason but uh I you got it Verizon says hackers are invertebrates you got that you got the headline right there right um so so you know I started looking at this modeling and measuring issue 10 years ago I was I was in uh in grad school and I thought um what I really want to do is optimize security spending you know I would like to come up with a formula that uh someone could apply and in decision support and just take in they have certain amount of dollars to invest in security and they have certain systems and all of these things and you get a formula and it's all pretty and it comes out and it's obvious right well um that was extremely naive uh in retrospect because I found out very quickly that my issue wasn't necessarily the model um it was a the way I solved the model was a nonlinear equation and things like that so it was a little bit messy but but it was solvable the problem was that there was just no data to to drive or validate the model and and so since then I I feel like I've just been trying to collect data and I like the quote that Dan just said get the data you can always throw it away so so in a very real sense the the the DB at Verizon and pretty much everything I've been doing since since then is just trying to get as much data as we could in other words you know we started the dbir because I wanted to understand which kinds of incidents happen more than others it's a very simple question right if I'm going to invest in certain uh security controls I need to know something about the threats that are out there and the rate at which those threats occur and and there were surprisingly little information like that 10 years ago go ahead so danis has said the only security metrics we're interested in are those that support decision-making about risk for the purpose of managing that risk yep so I know which stats are my favorite out of the report what have you found when you're talk when you talk to people that has been the most influ influential statistics or or facts that you're reporting for for risk you know it's it's really changes every year I I I find this interesting so when we started that um there there was no we just asked the question all right we have an excellent source of data here because we we at that time I was not part of the forensic team we just looked across and said hey where could we get this information and and those guys respond to incidents let's let's team up there and see if we can get management to okay uh the overhead involved in taking these data points out so we started asking questions like well what would you want to know about an incident you know let's pretend you could figure out anything about it and save it what would you want to know so we did look at nist had a framework at the time uh and various other things and of course people want to know who did it and and and things of that nature how they did it uh and so getting back to your question you know the the that has always been evolving it in fact my team tells me they they just cut me off because I always want to tweak that and add things to to learn but uh it makes ongoing measurement hard um but anyway the first year we published that I remember some people were Furious that uh sorry what year 2008 was the first year that we published that report and at that that time it was a 5year retros perspective so 2004 through 2007 data um but the thing that really seemed to resonate with people then was the fact that we showed um about an 8020 split between external uh attacks and internal so in other words we found that 80% of all incidents were tied to external actors instead of internal actors and man I mean some people were really mad at that because they said no you know we've known in security forever that 80% of risk uh is Insiders and and you know maybe that's true maybe that's just a false belief we've had for for a long time but you know I also said well be careful here because what I'm counting is not risk you know I'm counting frequency which is part of risk and and if we said that 80% of attacks come from Outsiders it could still be true that 80% of risk comes from insiders you know so we haven't studied this question well enough but anyway that that year that was very uh interesting also the things about uh you I think you already mentioned them about uh um 34 of attacks at least are found by someone other than the victim has been um consistently interesting yeah and it's been in fact I think the the slides that were handed out there's a a graphic of that over over time and if you view how much change in the sample set has been evident in the time we've been doing the dbir the fact that we've always found that is pretty remarkable and that's really interesting you know Dan was talking about the trends and sometimes even if you don't know the number knowing the trend can be really important and I think that's so interesting is you've gone from your very small set to a much larger set um that because because companies that might use your services are already a specific kind of company I mean it's already self- selected but as you Bren that out to to Secret Service and others I think that would stop I'll tell you my favorite stats in those were um the especially the ones that said how long they were in your system before they got detected how long before once they were detected it took to it took uh to get booted out and how many of these were difficult to attacks when I come back and write about those those stats are consistently the ones that I come back and talk about can you can you talk about those those for a bit Yeah so that that was um in fact I think that was a late edition when we were collecting those we just said you know it'd be really interesting to know how long these things broke out into and uh so we started we started collecting that but um yeah you know the the average is months um five or six months on average between the time when a victim is is compromised initially and then they discover that compromise and so you that brings up so many questions about you know why is this and I think Dan hit it on the on the um the nail on the head when he said well it's because their data is still there uh there was nothing wrong with the operations of the system it's not like a denial a service attack or or an outage where all of a sudden your web server is not working anymore or taking uh clients can't get to the site um the problem here is the data is still there it still works everything looks pretty um and you don't realize it until maybe fraud starts showing up or other other things like that um so so I think those are some really interesting findings and I and I you know I think that it has helped us to realize in the security space in general which is always been prevention minded you know keep bad things from happening keep bad things from happening uh as as other companies have found this trustwave does a similar report they found similar findings and that it takes a long time uh mandiant has found that as well it takes a long time to discover incidents every report I've ever read says that and I think it started to shift the focus from just prevention to to detection and response and I I I think we have at least as many products on the market in the security space now in the detection and response side as the prevention side which is new to my mind in the last five years and and in my brain I don't think this is that probably happened this way but but when I think about your stats about how long they were in before they got detected how long they were after they were detected before they get k out and in my brain connects that directly to locked Martin and killchain sure you know CU locked Martin was saying we've got to interrupt that that chain of how they get in and and to me that ties in together um so I've got uh one of the thing that came in here is is Richard bck if he was here I love cuz he comes down to a single metric yep um he says you know what sometimes a lot of the security metrics we talk about are comparing the height of different basketball teams or how um or how fast they can run you know 40 yards or when the only the main statistic is are there bad guys in your system so when it comes to the mandatory reporting he's thinking what boards you know what we ought to require boards is not to see how good their firewalls are and if they've they've got a a strong password policy but go to look if there are Intruders in the system I mean that's one obvious metric that can actually tell you um so what's next what's next for 2014 and then we'll go to Yuri so uh in in 2014 we've we've put a lot of effort into continuing to expand the number of partners that are contributing because I mean we I'm very thankful that that Verizon has let us continue to do this and and so we have 18 contributors in the last report and I think we're going to have over 40 oh my go this next time so and and the the base of contributors is getting really interesting um we've got um Sears from every major re region like we have we've had no Latin America um contributors to this point in time but we've got two in this upcoming year um and we've got a phenomenal number of other security service providers which I I think is fascinating in its own right when you start thinking about hey you know we're we're comfortable enough in information sharing and and enough companies desire this kind of thing that that we can actually do this so uh you know uh fire eye and McAfee uh aami uh threat grid and kasperski are all contributing to so you've got like a common you've got a standard format for everyone to talk about these things right uh we not everybody has a standard format but we are kind of post retrofitting a standard format as best as we can so yeah that that is a topic on its own but um um great yeah and that's that's that's what's coming and the and the different measurements that we're trying to do there's there's a couple uh things in there maybe we can talk about it later but we're trying to get past just counting um here are the bad things that have happened to uh really starting to do some more uh statistical analysis and clustering and things like that you know like for instance a finding that I we realized a few months ago is that when you when you look at organizations and their threat profile if you really break it down um a financial service organization that's an investment bank they are have a closer threat profile to a telecommunications company than to a consumer Bank you know so the the the investment bank and the consumer Bank are more different than an investment bank and a telecommunications company and and stuff like that I think we're just starting to scratch the surface oh that sounds fascinating looking forward to that thanks well you talked about having the regional certs and so let's turn to regional Sears um y there there is we heard a lot from Dan and Wade and so I think both from oecd and from sure um good afterno my name is Yuri um I'm working for Japan's national s teams JP set and I'm also a chair of the Asia Pacific Regional s um Group which which is called um AP s um we started looking at this challenge with actually a wcd and apto which is the policy group um and um it's a little different um motivation and the background we started to um focusing on this measurement so I would like to introduce um quickly about our background and motivation so last couple years it's actually becoming really hard difficult for the national start or maybe Global um security operations or Global um companies to share information um about threat and um risk conditions um really because most of the Security operation especially in the National um level is focused on too much focused focusing on the National Security national competition defense um and intelligence so it's really getting hard to um collaborate or share information now um to avoid that we really have to focus and especially in Edge Pacific region we focus um on how to collaborate to improving the environment itself not instead of you know pointing each other who's conducting attacks and who possesses the risks and taking side and everything that's not going to help for the collaboration internationally so um long time long story short we are focusing on clean up malware and cooperate on in removing bot Nets that's our Focus as a regional collaboration insert um so when we starting doing that um you know of course you need to measure the impact um we're cleaning up those environment we improving try to improve the environment are we really reducing the risk by cleaning up the botn net by cleaning up um and removing the bot nuts so going further um we're trying to measure the prence of the malware and the success of remediation approaches can I jump can I jump in there because this just reminds me so much of what Dan kicked us off with is the metrics that we most care about are those that support risk management and decision support and I think one of the reasons why we're so bad at metrics sometime is that we don't set what is our goal what is it we're actually trying to accomplish because we just say oh well we're going to do computer security and we don't support that to measurable goals right and this is really important that you guys did we said we're going to we're going to get rid of malware we want to reduce malware and botn Nets and so of course the metrics that come out of that are obvious and and measurable right yes as we do new as as any country does a new cyber security Strate I think that's really important because if you don't have goals that can be measured you're not going to have you're not going to be able to do this that's brilliant thank you and by having that actually it's raising the transparency of the sources of the risks so you know we can have an agreement where is the source of the risks and then how we can invest to lowering those risks so it's really giving a good it's one of those things that you say well that's kind of obvious when you look back at it but I haven't heard such a big grouping that that's done this well we're starting there's a lot of challenges so um we need um all the experts help now um so we start taking a look at this one of the things um of course we all now know is we're missing um in you know we're missing is a strong sources of data cross comparable and robust enough to develop the um statistics to measure the risk level nationally and and um globally so um and then this type of you know metrics in Risk measurement is really essential for policy makers to evaluate the um policy impact you know potential policy um um approaches um but at the same times you know as you mentioned Dan mentioned the decision making for operations it's really important to prioritize where is that you know significant threat and we can manage our limited resources um to you know start looking at you know which thre we need to really work on um quick so um you know that helps for the policies and that helps for the um s um and really we can work together not just driven by fear but driven by the fact um you know data driven fact driven um decision making and approach so um and then you know more more importantly more probably significantly this type of metrics can probably serve as a Common Language um between policy and Technical you know operation I think that Kevin is going to talk a lot about it um I sometimes play aason role between operations and policy and you know we talk very different language we see the threat very differently um so metrics um can be a great tool to create agreement around um the sources of the risks and how to invest lowering those risks now so oecd and start um start um looking at this challenge again um and then we're quickly realized that we can learn a lot from the you know again Don you mentioned um but uh we also realize uh that we can learn a lot from the how the public Healthcare organization working internationally together um to respond Global level of risks to the health and underlying um environmental um conditions so we're trying to learn the you know approach we can you know learn I mean they have wh and CDC type of model um we try to learn the approach um how they gather data and use layers of those metrics to um Drive policy and um International coloration so that's really the backgroundd took a look at the um start as like a CDC position um National starts are probably in a good position because we have trust from the policy trust with the industry working closely with the industry and access to that uh risk the data on the risk conditions but s itself does not have a good data um those datas are starts are publishing all the um statistics um annual statistics but these are really about performance metrics you know these starts are receiving how many inent reports how many vulnerabilities how many um uh threads um we are seeing we've been reported so this is not really indicator of the risk so we're going to have to work with the actual industry um players that looking at the sources uh looking at the threat environment and develop mechanism to collect those data and with the help of the oecd institions how to make it robust enough to be you know we can do this um cross comparable so that's a challenge and that's where we are um started and are you finding so you've got this this very specific goal of malware and botet um are you finding those to be relatively e easily measurable I mean it seems seem to be in my my brain that you can actually count things you know you can count the things like the invertebrates that that way talked about is that the case or or is it actually more difficult than that there are many organization who's counting the infected number of the PCS um so and then starts are receiving those data from those global Soul statas including Microsoft including um um uh security operations and our job is to remediate iate that um disinfect Machines working with isps to um clean up um so we know there is a data sources we know there's a Global Security operations looking at and measuring that um but the data is not static um it's changing and um so there's some challenges how to make the statistics um I would love to hear from Microsoft how you you have those like a heat map type of a global um M um so we probably good to hear from uh hey I'll take that um and so we're going to listen to Kevin for a bit then we're going to go back to Dan gear to hear about the index of cyber security and then and then we'll start taking the questions so yeah I think it's been a a great discussion so far um in addition to some of the characteristics we've already talked about for metrics I guess I start to think of a couple of motivations that I hear when my team is thinking about this excuse me or we're talking with customers a big one we've talked about already is kind of how am I doing today you know where do I compare a lot of people want to know that we could debate the usefulness of that uh there's certainly an aspect of what are the things I need to be thinking about going into the future what can I look out for and watch and how is that likely going to affect my cyber security posture and then the other area we've talked quite a bit about is you know what are those activities whether they're operational or policy based that are demonstrably effective and how do we start to measure those uh and I think this also ties into a bit of a larger discussion that has been starting to to unravel around capacity building as organization ations or nations are building capacity around cyber security I think they want to answer those questions they're not creating something that's just static for today's issues um and this is really where it becomes interesting to my team we get asked these questions quite often uh so we started a couple years ago a line of research to try to answer some of these questions uh one of the things we quickly realized it wasn't uh was no longer s uh sufficient to just look at the technical measures we wanted to figure out how do we get Beyond some of the usual advice about patching fire walls antivirus uh but interestingly our research does start with one of those technical measures so for the last8 years now Microsoft has produced the security intelligence report and that's kind of our worldwide view of what we see in terms of vulnerabilities exploits and specifically infection Trends malare infection Trends around the world uh when you step back from that and I'll say it's it's a very good source for if you're thinking you know what's really the top threats in my country what are the most vulnerable applications you can really get some good tactical information there when you step back and look at it perhaps at an executive level you get the heat maps that Yuri mentioned and I look at those and I say okay we got a bunch of countries that are yellow red in between well what's the difference there you know what are the things that would lead to one country say Finland which has one of the world's lowest malware rates versus a turkey or another country who has one of the highest ones and why is that and that's the type of questions that we set out uh to answer and that was what led to our first paper the linking policy and performance that came out about a year ago when we started to dig into that data I said you know I want to look at some non-technical pieces because I had to believe there was something else going on there that might explain part of this and we started do the analysis you did see that you could do some correlations and see that um increases in malware you know better performance which is actually a lower malware rate was correlated with things like Broadband penetration uh PCS per PE per individual person uh even GDP uh but that wasn't necessarily the most actionable thing that I could go and say well if you just increase this number we would see we think we would see malware go down so we continue to push through with it and we're able to actually predict what which components led to the malware rate in a particular country uh and from that we could kind of see how countries performed relative to our prediction and you had these three clusters that come out you've got countries that are outperforming what we would predict they have relatively low levels of infection you have countries that are right on par with what we predict and then you have countries that are have a higher level of infection than we predicted and so was those outliers that were interesting to me I said there's probably something going on there that's not fully represented in the data and that let us step back again and start to test that against some of the other policy measures that we saw uh so it's interesting to see that in that high- performing group uh the vast majority of them had done things like signed or ratified the Council of Europe cyber crime convention uh they had also signed or ratified or participated in in the London action plan and they had significantly lower rates of piracy and that was exciting to us because we looked at those as things that we could go and show policy makers and say these are the things that we're seeing in these countries that are higher performing so I mentioned that was about a year ago uh we have a follow-up to that paper they're actually going to make available online tomorrow I have copies here with me today um we thought a lot about how do you follow something up follow up something like that and a lot of those measures like GDP and even Broadband penetration they don't change that much year to interestingly the malware rate does uh so we looked into a little bit more time I think you just said something important there I the malware rate that we're seeing is changing and it's rather volatile huh uh and specifically when we looked at the years 2011 to 2012 we saw a rate trending down as M as Microsoft is measuring in the security intelligence report but not evenly so we again said well why are some countries going down and others still going up um um and and I want to make sure we leave plenty of time for uh questions but the one really interesting interesting thing that came out in that data in the analysis was if you look at the top performing countries the things that are indicative of them continuing to Trend down things that we would expect like institutional stability Economic Development regulatory quality if we look at those who had the higher rates of malare infection we saw something very different and that was that as they added technical capacity whether it's Broadband capability mobile usage just internet users per capita they saw an increase in the malware rate so take the takeaway for us there is that don't stop adding those things certainly we want to see that happen but recognize that there's potentially increased risk in those early days yeah that's fascinating um and especially you know you can you can start with by having good data it not only allows you to ask good questions but it helps you to discover new questions as you see the trends that come out I mean that malware Trends you saw going down but not evenly that's I mean we could hold an hour and a half event just just exploring that as a topic that's really fascinating yeah that was one of those ones that sounds cliche it's one of those big data type problems you kind of have to sit and swim in the data and see what comes to you you don't necessarily know what questions ask up front and and it reminds me of a point a point from Dan you know and when when he had measuring security slides you know he talks about let's steal wherever we can I mean there are these good mindsets of how to look at data and interesting questions and and one of the ones that that that I've always liked it comes to you we know that if we have a security mindset then it's not in our interest to ever wire up Africa and Latin America I mean it just it's just not going to be in our security um uh and that's also obviously the absolutely wrong thing to do of course it's it's going to be a better world for us to do that but thinking about computer security doesn't get you there um but we know that for example China and the United States are going to pollute but we want the greatest GDP per unit pollution um and to me it's like well we want Africa Latin America and these other centers to connect but we want the lowest number of malware per and so we can think about emissions density and and these other statistics yeah kind of relative to to where they are in that point of maturity yeah perfect um okay Dan if if you're still with us um then can you talk a little about the index for cyber security three two one uh Dan if you're with us uh you might be on mute um no I'm I'm not on mute but this is a this uh we have a new I'm using a phone system I'm not shall we say up to date with I got it now yeah no the uh I kind of love the old days when when you could understand phone you know you didn't get understand phones okay uh yeah do you have a couple minutes on index for cyber security and then we'll start taking questions for for for everybody sure and uh did you distribute something in that regard we did distribute something in that regard yes okay the annual 2013 annual report I'm not going to walk my way through that because we don't have the time but I will say to everybody here my with a colleague in New York at a at a at a bank in New York um we decided we wanted to do an index in this space space and one of the difficulties is we don't have stable definitions of things so how do we deal with that another thing of course is that people are loathed to reveal anything about cyber insecurity either with respect to the firm and themselves and you know all the cavat that anybody here could come up with so what we have here is a thing very much like the consumer confidence index that is to say we ask people's opinions we do it once a month we have a stable group of people whose opinion we ask in that case it's more like the purchasing manager index than it is um consumer confidence and the stable set of people which by the way may I say anybody here who qualifies I would love to have your help because we never have enough just sort of by definition um everybody who answers this is well not necessarily the title ceso but somebody who's Frontline in other words who has operational responsibility for cyber and therefore knows the state of the world without having to be told just can say well this is what I'm seeing so we're looking for that class of people they all answer uh 20 24 I forget I think it's 24 questions the numbers changes slightly over time but not by much the questions are pretty stable I mean really stable over time and we can swap a new question in or swap an old question out just in the same way with the Dow Jones Industrial Average you can add a you can replace one firm with another and the index doesn't fall all over itself there's a there's a way to do that and we're in fact trying to be as boring as possible that is to say we're trying to make the it such that no one can argue with our result by saying well this is an artifact of your method so everything I'm telling you is completely prosaic boring survey research type stuff the 20 Questions we ask every one of them is a fivepoint lyer scale meaning in the last month has malware gotten better gotten worse gotten a lot better gotten a lot worse stayed the same every one of them is the fivepoint scale from a lot worse some worse static better a lot better that all of them are that form and the advantage of that is that the individual's own definition of what is malware what's a vulnerability what's you know whatever we're speaking of as long as their own personal definition is stable they they are not lost in space and having a new definition every third day as long as they have a stable definition what I'm getting in is a change from month over month and I can average those together in a way again is completely boring and at the end of the month on the at 6 p.m. on the last calendar day of the month we publish the number and you we publish it to the world everybody can see it uh people who are respondents we give them a detailed analysis every month of how the uh answers were were and so forth and frankly because they're giving us a little of our time a little of their data the only way I have to pay them is to give them if you want to call it pay I don't know if you want to call that or the only way to be polite um is to exchange a gift with them in this case it's a detail report they get it we don't keep that secret they can give it to their peers and so forth we just sort of rather that no one publish it simply because then what do I have to give to people who are nice enough to play and the answer would be nothing now the thing you have as an annual report is an attempt to write a lot a lot of that up and I'm hoping it self-evident but if it's not uh my contact information is probably on it somewhere certainly Jason knows how to find me uh and we can go on from there the thing that you might find interesting is that month over month the net the total risk as we measure it um and this is a Risk Index so up is is bad like in golf up is bad um that the total index has been rising reasonably steadily for ever since we began and you might say that can't be true it's got to rise and fall doesn't it well the group of ceso type folks that we're dealing with are not saying that they're saying that a steady rise is what they see now they say it in a way where in any given month which which of the 20 questions or which of the three of the 20 questions or whatever most persuasive have the biggest impact on the result does change and change a lot my guess is by next month there'll be a whole lot of of changes in the way they see uh uh you know risk of nation states for example I'm suspecting people will you know to a degree react to the current news that's all right every month in addition to the 20 questions and calculating the index all of which again is intended to be born um we also ask one special question every month we ask a special question uh one of them which I think was last September but I can't remember for sure was in fact inspired by dbir we asked have you ever discovered a data breach not involving your firm but somewhere else have you ever stumbled across someone else's data breach and what we got uh weighed was 55% yes confirmed and 15% yes unconfirmed that adds to 70% that's pretty close to your number Y and um I would suggest that that is corroboration we'd be happy to do other things like that if people have special questions that they wish we would ask or they know somebody who should be playing this game with us or whatever um I'd love to have you in touch by the way we offer as close to anonymity as we can as we can offer um and I can describe all that I don't have time but we we put a lot of effort into making it non-traceable which includes for example it doesn't matter if there's more than one person at one firm who's doing this they'll never know that both of them are and there are two air gaps involved um in our my and my colleagues respective kitchens um it's it's manual labor here and there as well we're doing the best we can in that regard um H you know if you if you attach a bug to my forehead I suppose that this will all fail but we're doing pretty good in that regard and so folks who uh worry about revealing anything um no all we want your opinion we want it once a month and uh we cannot tell uh who answered what and we can't tell whether all we can tell is whether someone hasn't answered answered for three months in a row uh cuz then we'll ask you know you still want to play but other than that um I'd be interested in feedback and we don't have time right now but um you've got something in your hand that you can stare at and then yell about later great yeah I've got questions on it but I'm much more interested in hearing the questions of the audience um if we got metrics our goal was to talk about 45 minutes so I think by that metric yeah we hit that um okay so uh really curious about about questions for Dan on the phone or or the other panelists okay um while we're while we're collecting the questions Dan the I also know you were doing work on decision market and uh Curious yes curious you have update on decision market for anybody who doesn't know what that term of art is it's also called a prediction Market it's where you have a a market in the same sense as a stock market only by which he means what I meant to say was prediction market so I use the wrong term yeah no called decision markets as well um in fact Robin Hansen who I view as the leading expert on this in fact calls him decision markets um he's a George Mason down the street there you should probably talk to him um uh prediction Market is where you have synthetic um if you want to call it that Securities and they're traded in a way where by um a couple of suitable assumptions you can say that the current price is a reading on the probability of the event so the most famous ones of these uh most famous one of these hands down probably is I University of Iowa does a presidential uh prediction Market every four years in which people um if you want to call it bet that's fine um where they bet if you will on who will be the next president and these are treated like they ref Futures contracts if you will um these typically have a defined endpoint um it would be something like um what what can I think of as a sort of a good neutral example in um calendar 20 13 um Cisco will have to uh issue uh three critical emergency patches you know that would be the kind of thing and at the start of the year um at the start of the year the price of that well let me put this way all the contracts would be a dollar that is to say at the end if the event has occurred or when the event occurs then the people who hold that contract get a dollar for every contract they hold if the event does not Cur by the end of the year then the house if you want to call it that um gets it so at any given point if you're saying well I think that's worth 80 cents it's like saying that I have an 80% guess that my my estimate of the probability that's happening is uh 80 uh is 80 per. now you might say well what good does that do for you and the answer is well if you believe markets and rational decision-making and all of that um a large pool of people who are trading anonymously and um making a market for whether it's a a security in the ordinary sense or Security in this sense um in principle has the same characteristics and so a decision Market a decision um Market a prediction Market in this regard um would be I think a very useful thing to have now I have tried mightily uh to get one going I have three times honestly um had what I thought was um a handshake agreement on funding and have it not happen um I don't really want to get that um why it didn't happen it has nothing to do with me it has to do with uh frankly a certain kind of tarity of the people who are making who deciding whether to fund it or not now you might say why does this need to be funding why can't you just do it with uh funny money you know with with bragging rights chips or something and and you can but the um the academic uh literature says that if you're using real money you get better answers you get answers that are both tighter with respect to narrower variant and more accurate and I want that want real money answers so if you're going to do real money what do you have to do and then the problem here is the American commodity Futures Trading uh um act and commission uh get in the way here um Iowa gets away with it because they have a u a dispensation from the pope so to speak they have the um they they were given a no action letter that says we know what you're doing and we're not going to do anything um but for anybody else to do it is a bit of a difficulty and the cftc uh recently had intrade shut down it had uh uh uh b bonked a binary shut down and so forth on the grounds that those are investments well in fact if you're using your own money that you are the the way the reason I was looking for funding and anybody here wants to help please be in touch the reason I was looking for funding if you give the player the money if Jason's going to be for example a player here suppose I give him $500 and he can then play $500 in the market that'll be 30 contracts to choose from and so forth it isn't his money so it's not an investment hence it escapes that issue if the if the entity running the market was a not for-profit meaning couldn't be argued that it was a for-profit casino it escapes the other problem which is internet gambling law that that all being said that's all hypothetical because I haven't gotten off the ground despite five years of trying and three what I consider to be um near misses of people pulling out at the wrong enties pulling out at the right I had fun when we did the last beta okay we're go to Mark Soxs and then to Matt and then we're going to bump over here to Matt flaming here thanks Jay Mark Soxs also from Verizon thanks Wade doing a good job U listening to all this and thinking in terms of resiliency security Etc and taking measurements everything I'm hearing so far tends to be a retroactive look in other words we're measuring what we were we're not measuring what we are and we're certainly not measuring what we could be or will be that's that predictive nature why is it because all we can measure is what we've been is there no way to measure what we are something that as a leader I can look at and say here like like the the concept of Health as an individual I can look at my medical records and I can see whether I was healthy a year ago and I can look at C but I can use those same sorts of things like body weight mass index whether I've got a cold you know I can use that right now to determine whether I'm healthy at the moment and I could even adjust my lifestyle so that I can be healthier in the future because I'm measuring very specific things about my body why can't we do that with cyber why is it only a retroactive look and it seems to be things we're measuring that we can't measure right now that could give us a current state upon which I can as a leader make decisions about my future you see where I'm going with that yeah yeah I I I have a few thoughts on that um my perspective is that for a very long time we've at least been attempting to measure where we are in other words um I I I think probably the longest running measurement in cyber security is some form of a checklist on how you're doing on certain controls and audit or what whatever form it takes in other words we've been measuring where you are at in terms of your security assessment for a very long time and the how are you fairing in terms of in in and all of those kinds of things we've been doing that for a while but I don't think as long as the other one so um and I think one of the issues is that if we take if we take time to look back and see well what has happened in the recent past I think we can do a better job assessing where we are now instead of the 150 questions of an audit where everybody gets the same question we start tailoring questions to certain issues that we we know to be true in the past and therefore can assume they're true in the so our assessment of now gets better and also if we compile enough data about the past we start to see Trends and I think we can start making that prediction jump as well so I think it's kind of all tied together um and the environments are so instrumented you'd think this would be possible you know one of my um this is a different kind of example but when I um one of the most places where having data and experience let us be forward-looking was back with doer and uh probably 1999 and getting ready for Christmas of 1999 and I think I was talking to Reggie or Sean and um and and I saw in the the stats from duder that they saw this big decrease in December into January um from DOD CT and and I was curious about it and they said oh we always see a dip because college students don't have the the bandwidth at the University they go home for the holidays and they stop hacking you know they've got other things to do hang out with their friends and they don't have B you know high bandwidth and at that time I was the predictive warning guy at the Joint Task Force computer network defense and so this this casually dropped fact of of course do sees fewer attacks during during these two months I was like oh my God like that's really important um because the decision support was having a general that says is it okay for me to send people home for holidays um or you know can I Min Min man Min minimally staff during the holidays or am I more likely to get attacked during this time and once we had that data and that experience we were able to put it together and say General it is you know you're probably going to be safe because generally we we see a dip and and I know that's not exactly that's not where you were going with that but understanding the trend helped us under you know until look forward Atlantic Council right now the Enterprise systems that support the Atlantic Council right now are they secure are they resilient they are secure cuz we've all sorts of duct tape on them how do you yeah okay so you're wearing lots of Band-Aids you got a face mask on in the human thing but you see where I'm getting at how could you can you make an assertion of the current right now 2:30 in the afternoon State of Affairs of the Atlantic Council and what would you measure to make that assertion that's where I'm coming from you know how how do you do that right that's the challenge right yeah and and it you know Dan gave five questions how secure am I am I better off this year than last year am I spending the right amount of money how do I compare to my peers and what risk transfer options do I have and I think those are really great questions that you're getting at and you know different organizations are certainly going to fit in with different places and the you know their ability to to answer any of those um Can can I respond to Marcus for a second yeah yeah yeah this is Dan uh Marcus do you think what do you think of for example static analysis of software and I say that what do you think of in the sense of does is that a predict I you know they would the people who do that very much want that to be a predictor right hi Dan good and good to hear from you um there's been a lot of thinking and work over the years too about State like any machine that's currently running what state is it in states of transition introduce insecurity states of stability tend to be more secure so that static analysis of software could introduce at least if you know where you are in program execution where are you in the current state of the running of that thing can you predict states of instability for examp example by doing that static analysis knowing that software is not you know the whole thing running at once the the program instruction pointer is actually going through it in executing lines of code can you predict at some point when I hit this line of code I'm going to be less secure than when I was before I executed that line of code can I take protective measures as I enter enter that piece of software but and they come out on the other side of it so absolutely I think there's value in it but we don't think that way I'm not aware of anybody that's that's designing security with that kind of mindset well um you should take something offline with me that I can't do here and I don't mean that as a secret handshake it's just there's something I should talk to you about um but what I I do think that there's a question sir and that is suppose for the moment we are getting better left right and sideways is the attack surface getting bigger faster than we're getting better very likely absolutely like yeah that's the sense and I know there's been some some work done on this right on the Improvement of tax surface you every new L that I know from Reading Dan's work but you know each new line of code every new feature is you know increases attack surface and um actually I've got one last question for you for your mark sorry um do you know has has nstac been doing anything on that I mean has NST been looking any of these kinds of issues okay I'm getting I'm getting a no okay um so I'm going to go one and then two and Matt I'm gonna I'm gonna keep you warm F your D okay hi um danetta Magano Covenant security Solutions and the question I had um and I think you touched on it very briefly was how much of the metrics include nontechnical controls because in a lot of the work that we do um as a company we tend to find that the big issue is the goal do we really understand our organization do we understand how information moves what's critical what's not critical and without having that understanding it's very hard to assess where your risk is so I just wanted to kind of get the opinion of where is that going to fit into the larger picture of coming up with global metrics thank you yeah I mean for our perspective we started with a very wide search for potential non-technical metrics that could inform uh our research started with about 80 narrowed that down to 30 or so and kind of stepped it down from there uh but I think you're right that you need to take it not just to the inputs what are the outputs and how are you tracking those uh the other comment I'll make around that is an area that I think we as a community could do more on is how are we measuring some of the policy decisions that are made and those another category of non-technical decisions we had a very small set that we could apply worldwide and it was binary did they do it or not and I think that's an area where we can start to build that that list up and then we'll know a little bit more about what's effective and what's not the um one of the great stats that I saw when we were at Goldman Sachs because we were gone to the bosses and we had this very difficult decision of cutting off web mail huge huge issue um because you can imagine everyone wanted everyone wanted to to to be checking their personal emails um those days before you B with of smartphones but the one decision so we did we did a test and you saw the amount of malware I mean the curve just fell off a table it was an incredible elbow in this curve and so I mean this went to the risk committee I mean this went to the partners that are overall judging the risk of the firm and it was incredibly convincing to them to say yes absolutely this was was the right decision and it was a one chart one graph um we were here it dropped like this and I was astounded I'd be very curious to see if password strength if um BYOD I mean there's got to be I I'm wondering how many of those others have such a simple chart where you can see the impact of one policy decision yeah it's interesting tie back to the that question as well what are the other Downstream effects of that have you moved employees on other systems I don't think in that environment but is there a morale or you know people change their hours because they can't get into what they need to do it would be interesting to study yeah and it's and it strikes me especially because to me a lot of the Pentagon that doesn't allow smartphones to come in people don't have another way to check their email they have to use web mail you've now got Port 80 open to all sorts of malware that can come in over Port 80 that you're not able to as easily Monitor and I think that's a very different environment than the average business today and the things that they fix with with bod in Open Access yeah great good question sir hello I'm Alan Blackburn I'm a army officer at army cyber command and obviously I'm not a statistician U my background as an army officer I've seen us try to implement these type of me measuring Effectiveness um in Iraq and Afghanistan I've seen a struggle really um really struggle with it in very complex environments so my question kind of goes to how do we validate causality um between what we're measuring and the conclusions we're drawing going back to the original uh uh example that Dan gave if the police are working really hard in a city and the drug price um goes up is it because of their actions or is it because there's a drought in Colombia for example that may have may be influencing the drug price so if uh Dan or Wade or anyone could um help me understand how we validate that that causality yeah the validation of causality Dan can you um did you hear the question okay Dan yeah I could the well the gold standard in the medical world is dose response so-called meaning if you double the uh whatever it is the dose the input um do you get double the response so if you can show proportionality like with the police example you you double the number of officers on the street doing X if the price changes if you double it again it changes again now you've got something so in a way you've got to perform an experiment I I think causality requires experimentation I could be wrong but I think causality is going to require experimentation the now mind you sometimes they're natural experiments you know um I mean not to not to be morbid but the FAA crash investigators they don't cause crashes they treat them as natural experiments that's the best I can offer uh Wade did you have yeah I was just going to say that I I am not aware of a a study in the security industry I've wanted to to see something like this for a long time but you know if you took a 100 organizations or so over several year period and you monitored all of their controls their expenditures all of their incidents and things like that I think you could start to get to to what you're doing until we get there I think the best we we can do is the is the natural experiment approach where I'm I'm observing but you even even in the very the limited amount of data and and observational field that we see you know there there's a lot of things where we have to sit back and say okay you know that's gone up um let's let's see if we can figure out why it looks like it's it means something but maybe it doesn't maybe it's just because we have a new partner giving us data this year and it's absolutely meaningless maybe it's just because um uh you know there's so many things there and and every now and then you know we can put pieces to together where I think we get something like Dan was mentioning some trends when you start looking at how the criminal movement has sort of switched from the amateur brag uh kind of thing to criminal you really can see changes that okay now I see the way that malware is being created um and distributed changes along those same trajectories you know I think you can kind of say well there's probably some causality going there but it's a it's it's a really interesting question and one we need to look at in the end wait way do you remember when was it Gard Asel back at qualus did some stuff like this I think so yep do do you do you I can't remember his his rules but he came up with a bunch of rules about you know if you have this input you get a half the output kind of proportionality constant I mean in a way causality we're not going to get but we can might we might maybe get proportionality constants which are pretty good yep no I I I I remember what you're talking about but I don't remember what those rules were well that's that's very interesting but going back to what you're talking about way it sounds like it in the end it's a judgment call based on experience hopefully we can get a little better than a judgment call but I think I think you're right and I think part of our problem is there's just not enough um I don't think we've done a good enough job yet collecting enough data where we can really really figure that out and and um you know you guys are doing some really interesting things with the the country by country you know and trying to pick out what are the what really are these factors you know and do our predictions work and and are they holding true and holding true over time so yeah I think in this measurement business and security we're sort of fairly early on I've got a couple comments Kevin do you want to jump in okay the um so one is you can you can also do some overlapping I mean that's one of the things that I liked about about Kevin's work I mean you expect there's going to be some correlation between where a country stands on the failed State index and the Transparency International but I mean the um uh you know I think by having those overlapping stats you can help control for that the um also you to some degree you even gave your own example I mean you said you know if the price of cocaine goes One Direction or another you can look and you can get some smart people around the table and you can say what are the factors that might go into this and then you can see if you can rule them out I mean and I think Kevin a little bit talked about hey we got these exceptions we have some outliers that um you know if we start thinking what might have driven this down can go and see if you can control for those factors and last is to dance point we might not have to depend on um on what life gives us when it comes to this and because we've got things like honeynet project and I'd be really concerned you know really interested you know the the folks at Honey net have done some interesting work and it might be interesting to see right as we can look at these things can you get honey Nets that are that are sophisticated enough that you could play these different variables you know you've got three different honey Nets and you control for things like password strength and and other areas and just see um how they what happens when they're when they're put into real world conditions so um uh Forest Ian pleas so as I was listening to this um a question started to form and I'll try my best to put this in the form of a provocative question um but hearing the different analyses or attempts of analyses different tempts at measurement I've been hearing different levels of aggregation levels of analysis here and so I'm wondering if we reach any type of consensus or convention on what are actually the the more appropriate levels of aggregation such that we can get to some you know as Dan was mentioning some some type of better level of you know relative level of security you know on one end you'd have the individual computer on the other hand you'd have all the computers that are interconnected somewhere in cyberspace probably the appropriate Lev somewhere in between is it the nation um is it by industrial sector within a nation or across Nations is it um somehow related to physical interconnectivity uh you know in the Cyber in the index of cyber security I think that's just you know who who wants a volunteer to contribute so have we been thinking all about what really is the the levels of agregation aggregation that would be provide us some effective effective Insight such that we can kind of work towards that for more um precise measurements um I I I'll throw out some thoughts here and and interested what others have to say I I I don't have a I think a clean answer to the question but I will say that we are struggling with it in in what we do so for instance we we collect a lot of data and then we try to report on the data and if if I just tell you well standing back and everything I can collect across all Industries coming from anywhere any kind of I can give you some statistics but then if I start to compare industry by industry and I think there's a chart it may be the last second to last third chart in there it it takes um eight different incident patterns we call them denial of service um uh point of sale intrusion and things like that and then Industries across the bottom um and that story changes quite remarkably industry by industry and even within Industries um it begins to change if you look at individual organizations or if you split Finance up into into different types of financial companies so so like I said I don't I don't have a clean answer but I can tell you that the level of aggregation really does matter because you can come away with very different um you know interpretations depending on at what level you're looking at and very different sort of marching orders if you will okay when I look at it at a high level here here's how I need to fix things well if I look at just at it you know layer down it's kind of a different different tact and I think some might be to the goals that we're we're aiming to get at Kevin did you yeah we struggle with the same thing obviously looking at it the national level that Maps well to the other indicators that we can look at but most nations are not homogeneous in that way and I think if we had the ability to zoom in even further we'd see some other differences the same is likely true of a lot of organizations uh if you look at different parts of a company uh public facing internal we'd see difference there so you know maybe we start to to your it's the goals right if you're trying to tie to National development indicators then of course countries make sense if you're trying to tie to a control set then it's probably within a cio's purview or how we break that up so I'm curious and so when oecd looks at this I mean it's a it's are they are the stats in the process you've been going through looking at the level of country are you looking at the level of Internet are you looking at the level of individual companies right now it is focusing on the National level um and there's a lot of data a lot of metrics but again it is you know sometimes for Microsoft case it's a product biased um data for the national s is a constituency reported biased data it is um the client's biased data so we need to put all together and that's great that's that's the challenge perect like like Dan like Dan started you know it's it's okay having a bias as long as you under understand the bias that's great I'm going to go to to Matt for a second then get the question from Ian cuz I know Matt so I know you've done some work for Department of Homeland Security and looking looking at Stats and I want to give you a chance to kind of talk about what you have done here sure uh so thanks Jay so my name is uh Matt Fleming and and I'm a fellow at a a think tank that works for DHS it's the homeland security studies and Analysis Institute which is a mouthful um and first of all thanks this has been a great session you know little plug these are all great sessions so it's it's great that you guys are doing this stuff but um so I've spent a reasonable amount of time thinking about measuring information sharing and I think you know some folks in this room have helped peer reviewed and I'm grateful for all that but but a couple things that came up today I thought were really fascinating and and sort of obviously music to my ears but um one is that um the word purpose or goal you know was just used and it's been used a couple times and I think sometimes we sort of lose the plot with you know metrics and we all love the word metrics in the same sense that people love information sharing but they sometimes forget well why what are we trying to measure why do we need to measure it and there's some great you know there's a great article out there a guy named Bob Bean at Harvard talks about seven or eight different reasons for measuring performance but I think sometimes we lose that and therefore we just end up measuring things because we can not necessarily because we perhaps want to improve a process or rather we want to just celebrate small victories or large victories and I think um you know to to Riff Off something that Dan said earlier there was a when he was talking about Trends and understanding Trends you know we often dismiss metrics that measure silly things like the number of pieces of paper we used or pencil sharpened but if over time there's a trend in you know we used to sharpen a 100 pencils a year and all of the sudden we don't sharpen any more pencils anymore it certainly makes you think why did that Trend change and so in information sharing you know that's one of the things that we can think about is lots of folks are sharing you know logging on to some sharing website and all of a sudden they stop doing that why right and so I think those Trends even though they're not necessarily you know sexy you know did we reach cyber Nirvana just studying some of these Trends on really readily available available and measurable things can help us understand if we're achieving a particular goal and then just really quickly to the to the um army cybercom um comment uh be happy to talk offline but I've done some thinking about how do interpret metrics on information sharing with this very problem of causality uh and Steve levit who many know is the author of fre economics with uh Steven Dubner uh in his pure academic work he's an economist at the University of Chicago and levit and many others in the field have done a lot of work on how do we interpret um uh the goodness or not of Criminal Justice interventions for example so you know if we have more police on the streets we may see more crime but if we have more police on the streets hopefully we're going to see less crime as well because we're stopping more crime and so Steve I can talk to whoever's interested but um levit and various others have thought a lot about interesting statistical techniques to sort of untangle that and I've done some work on information relating to that information sharing perfect and that and that report's available online and that one's coming there is one out that's been out for a little while on uh on metrics for information sharing but yeah great and and I want to jump in because we've been so in our work with glob aggregations of cyber risk again we've been working with Zurich Insurance on this and it's really struck us because the more we've been talking with u people that look at Cyber Insurance you know we've been talking about these big systemic cyber risks and these things that are built in the system and and you know these big picture interconnections and we found the closer people were to cyber Insurance itself the more they wanted to talk about data breaches and to a large degree we think that's because data breaches are so measurable um you know you didn't have you have 100,000 or you know if your target you know you've got 40 million or 70 million or 100 million or there's some number you can associate some value with those um you know $2 or $260 per record or or or or the like and and it's easy to asso associate numbers with it's easier to understand that risk it's easier to transfer that risk and so we think you know Dan talked about silent failure and mandatory reporting um the more that we can track towards um getting better numbers on the rest of this the better that we can risk manage it the better we can risk transfer it Ian uh Ian Wallis from the brookens institution I haven't done a great deal of thinking about this this is a question of genuine ignorance um the but but it is one that's been alluded to as we've gone through um how does the changing nature of the environment the the internet affect the the process of drawing up metrics over time so you have the the emergence of the internet of things or at least more and more things with with IP addresses physical cyber convergence uh which both changes um I guess what you have to measure and also what's important to measure and I'd love your thoughts on that I'm gonna go to Kevin first I'm G ask Dan for comment and then see if anyone else because Kevin I mean you a lot of your data I think comes off of yeah Microsoft products I tend to think of desktop computers for that hugely in the short background there is we're getting data from 600 million plus PCS Windows devices that are getting malware cleaned off of them on a monthly basis as part of the pro patching process and both valid as well as PIR anybody who we're giving patches to but as you rightly mentioned the universe of things that are connected and potentially vulnerable is getting much much bigger than that and we don't have that instrumentation there so I think Yuri made this point earlier we need to find a way to properly bring together those data sets and figure out how do we match them up yeah any anything uh so just a few examples on some things we've done over the past few years um uh just kind of in the realm lifetime of when we've been looking at data breaches Cloud came up and a lot of concerns about Security in the cloud so you know we added a few metrics trying to get at uh we used to track well what kind of asset was compromised was it a server was it a desktop um well then when people started asking about the cloud we added well who who manages that device um where is it located is it located in an external environment or a so so we've added things like that with h the BYOD thing um as well we started talking about well who owns the device you know now is it a corporate owned device or does it personally owned device maybe a partner owned device um I've heard a lot of questions now about intern of things I I don't know that we necessarily have any any metrics that we're tracking on that but you know we'll start to think well how do we change our metrics and really figure out if if if this is is an issue or becoming an issue uh Dan any thoughts I was at a event last week where Dave Clark as in if the name Rings any Bell Clark reeden also the original indn principal folks and so forth and if that doesn't you can ignore it uh but an internet if you want call it that founder uh was talking about what defines an internet and he said if you define the internet as a set of protocols that permit communication between parties that didn't have previous relationships then we actually have two of we have two internets now and we will probably have more the two internets are the ones that in ordinary vernacular conversation you'd think of as the internet you know just what we all think of the other one is he his in his Viewpoint voice over IP is a second network has a different set of protocols have a different set of people in charge has a different model of uh uh connectivity has a you know blah blah blah everything's different that's a second model um I suggest that if the Internet of Things is to progress and not be a hazard maybe it's another model you know I could well argue that it'd be it'd be great to connect all the world skated devices to things so long as you can't route to them from the arbitrary botnet infected PC in your grandmother's uh front room you know maybe it is routability that is the issue in which case what is the metrics that we might use to decide whether or not something should be segmented obviously the internet is about to be segmented one way or another is that an issue should I if I run a large company should I be derting China I mean I'm not making fun here I'm just you know should I be doing that um the whole question of naming and of routing and of uh what to do about content inspection whether you demand it or whether you forbid it all those things in a sense could have measures associated with them and since this is the moment of Maximum in of Maximum change maybe this is the time to bring it up and so I don't know quite what the answer is um but if but we could well look back and say why didn't we think of that then talking about today which is if we are going to have you know the itu and not the uh and not I can in charge of naming that that is something and what should we have about that that I mean is is there a number of unique addresses per member of the population at large a useful metric in which case the United States wins hands down I mean MIT and Xerox and so forth all have a class A address all by themselves um I don't know I I proportionality con that when you can't measure two things sometimes you can measure a proportionality between them and maybe maybe that's the answer is is the attack space getting big bigger faster than we're getting better if I had the proportionality constant I sort of would be okay with that and and so forth um and and for me one of my if anybody here can be clever some proportionality constants would be mighty mighty timely and you know a lot of this to to to conclude out um I I really what out of the stats that we talked about here I really like Yuri you know you see different people that are looking at it different different ways you know we have data we look at what's important we have data we want to explain it and we look for for interesting metrics um and I like for a lot of the organizations that's here you know the hey we have a goal what is it that we actually want to accomplish and then and then find the metrics um that are going to help us support that my goal is to get defense the advantage uphill and the attackers have the harder job which flips things around so I like I'm going to come out of this and start go ahead Dan Jason I would I would like to suggest that we don't have and it would be swell to get margin of safety calculations like our civil engineering friends have yeah margin of safety you know how much more weight can the building stand than we're ever expecting to see in it Etc or how much wind load you know could we get a margin of safety here Y and I also think about because I tend to like environmental models also is things like emissions density carrying capacity I mean how many predators can you have in a system you know relative to how much prey there is and um and so I like a lot of the stats that that we've been hearing here cuz that's my goal that's the end state that I want and so uh as we all go off um some of you're going to go back to government some you're going to go back to your companies think about you know what is it the goals that your boss wants or that you want um in that end State uh we've heard lots of lots of not just wisdom here but actual practical facts that and and things that you that you can come and touch so I'd recommend the work that uh the panelists here are putting together um for for Yuri Wade Kevin um for Dan the index of cyber security if you want to learn more again I uh check out Dan Gear's measuring security um just Google Bing that and uh it'll give up lots of great examples and things that you can apply um our next event is going to be on the 20th of February and we do want to thank Zer Insurance um for their work on global aggregations of cyber risk that make this all possible thank you very much oh I'm sorry one last um we are the Cyber statecraft initiative and um you can't have cyber statecraft uh without cyber States men and women and so we have our one of our cyber Statesman mugs for all of our panelists and we'll be giving one to each of you so thank you very much please a round of applause uh for Dan Wade Deary and [Applause] Kevin