IQT Explains: Generative AI for Security

Transcript

[Music] today we're going to talk with Will Howton and Rob Mills about the intersection of generative AI or Genai and security gen holds great potential to transform Computing across a wide range of Industries including the National Security Community but brings with it unique security challenges investing in technologies that Leverage naai to unlock and enable new cyber security capabilities can unlock significant value and strategic advantages for US National Security welcome to iqt explains a series on the iqt podcast where we explore global technology Trends and their impact on National Security to provide you our listeners insights and unique perspectives I'm your host Katie gray and today we're going to discuss the importance of securing generative Ai and how this ties into the greater National Security mission to get things started I'll introduce today's guests will Howton and Rob Mills will is a principal on the investment team at iqt and has been with the team for two and a half years prior to joining iqt will was on the corporate development and strategy teams at cyber security companies zscaler and paloalto networks helping lead those companies Investments and m&a efforts at iqt he focuses primarily on cyber security investments will thank you so much for being here today thanks so much Katy great to be here Rob has been a technology architect in iqt cyber practice for about one and a half years he spent most of his career at Sandia National Laboratories supporting government customers as an embedded systems developer and reverse engineer most recently he worked in Hardware offensive security at Nvidia now he focuses on cyber security startup tech for iqt and has been researching the use of geni as it relates to cyber Rob thank you for joining us thanks Katie also happy to be here great well we're going to kick things off Rob with you um and maybe just at a very high level can you start by explaining what generative what generative AI is and how it relates to security then I'm going to turn over to will for a little bit of a market overview well before I start talking about what we think of as generative AI I want to take a step back and talk about more traditional Ai and ml systems and those traditional systems have been in all our lives for years think really simple examples like Siri or Alexa or your Netflix recommendation queue these started as uh rules-based systems and grew over time to enable machines to make decisions based on data some of these systems were great from a cyber security perspective and have been prevalent for quite some time machine learning systems are often good at detecting things that are like things we've seen before anomalous Network traffic malware signatures things like that but maybe not the greatest at finding new and novel threads but generative Ai and when I say generative AI I mean Transformer based large language models provides is a predictive system trained on mountains of historical data that allow a machine to generate new and novel content stuff we've all seen like chat gbt and Dolly in the public sphere and the story I always relate this to is that the cyber security industry has historically done a great job of one particular thing and that's generating terabytes and terabytes of logs and data we're at a possible industry inflection point which we can train generative a generative AI models excuse me on all this data and historical learnings and have them help us with big cyber security problems at scale being able to make sense of the mountains of data humans don't necessarily have time for or helping analysts quickly make sense of alerts incidents or even helping them their mediation of issues and this is where we uh we're excited to dive into great yeah there's been a long history of AI and cyber security and uh you know certainly this feels like a an inflection point well maybe you could walk us through some of the major market trends that you're seeing um over the last few years and sort of how the market has expanded as the technology has advanced sure yeah so with respect to G AI I think we've seen rapid growth and Enterprise adoption of the technology so estimates here vary but you know by and large the consensus view is that Enterprise spending on gen will be in the billions to tens of billions of dollars in the coming years there are plenty of plenty of examples uh from recent earnings calls of public uh company management teams highlighting their company's use of gen and llms to advance their business um at the same time kind of following a classic Trend as this is an emerging technology uh spending to secure it kind of lags adoption right so total spending on securing gen use of gen is estimated to be somewhere in the neighborhood of maybe 200 million or so today um but this figure is only going to grow uh as the use of gen scales in future years um and so it's against this backdrop that we've seen healthy investor appetite for providing security for AI funding those companies um by our Reckoning we've observed somewhere between 400 500 million invested in new companies uh here in 2024 Alone um over double the amount that we saw last year in 2023 um at the same time we've also seen healthy investor appetite uh for companies um creating uh new Solutions using Genera AI for security um all cyber security companies from the brand new startups to the largest incumbents are looking to build gen based capability into their products um investment in new companies built from from day one so to speak with Gen enabled capability as around 300 to 400 million um in 2024 according to our research again overd doubling uh from 2023 yeah it definitely feels like you know when we were at black hat earlier this year uh you know the on every single sign was incorporation of llms and Genai and AI generally into security products as well as sort of all these new startups that we're seeing sort of has replaced zero trust as the new buzzword but um Rob maybe you can take walk us through some of the specifics like where do you actually see some of the applications of gen in security um where are you seeing you know kind of that activity and that applicability really making sense well to back to my example before about just having terabytes and terabytes of log data in the Cyber secur industry uh I think we're seeing a lot of applicability in areas where we've just got lots of historical data um an interesting area we've seen a lot of startup activity in is around the idea of uh software or specifically binary compiled software build of materials and vulnerability Discovery um think back to for example the uh the solar winds incident from a few years ago that as an industry uh cisos and their organizations are given binaries from trusted vendors that they're intended to use to update their systems in that particular example they were given a binary that was uh certified good by the vendor but turns out was not good how can as an industry how can we adopt tools that might be able to discover that sort of vulnerability before we introduce it into our systems uh another area where we've seen a ton of activity is the around security operations that specifically relates to my log example from earlier so if we have a lot of tier one stock analysts out there who are often charged with uh looking through mountains of alerts and log data how can we use these systems to help triage the types of alerts they're seeing help them uh draft responses that are seeing or even help them remediate the kinds of problems that they're seeing we've seen quite a bit of activity kind of in and around that space and another one that probably lots of people have seen in public at least is kind of maybe Rob if you don't mind I'm going to jump in there just really quickly because I want to maybe pull some threads on some of the things you've talked about here um when you talk about the sock analysts like do you think that this is something that ultimately is going to replace the sock analyst is it going to be human augmentation where do you see that Trend going right now we're really trending towards seeing argumentation and not replacement uh you know in industrywide not just in the National Security space these positions especially at the lower levels are very hard to fill and have high burnout and turnover raos so a lot of these companies are really looking to bring in products that help enhance the productivity of the analyst that we do have and help them just perform at a higher level if you're a tier one analyst can we make a tool that helps you perform more like a tier 2 analyst um and so so so on and so forth up the stack we haven't really gotten a lot of of talk from from the industry about using these to actually replace people and I think a lot of that goes to still there's not there's not complete trust in all these systems uh if I'm a sock analyst and one of these tools gives me triages alert the things I need to look at and perhaps gives me suggested remediation we aren't seeing a lot of companies or in or organizations be willing to just blanket trust the the suggestions that are given to them by the tools we still need humans in the loop on almost all of these things so I think both with industry and the National Security Community we really thinking about these as augmenting the staff that we do have rather than looking to replace them yeah I think certainly at this point these Technologies are pretty early and there still are you know errors and hallucinations and things like that that may affect their um reliability but you know who knows as as the tools get better where will where we'll end up with that um so sorry I was I interrupted you but go on you had a few other examples I think of where geni u u might augment security use cases I think will had something he'd like to say there as well okay no problem yeah I I was just gonna add also that um you know these are the things that Rob's enumerating here are also like defined budgets with known buyers and um known problems uh also vendors uh have been selling into these spaces for quite some time and users are really looking for uh sort of value to be proven out before they fully commit to any kind of new technology um and so to Rob's Point earlier you know specifically on the analyst augmentation piece we're not seeing it's it's again it's augmentation it's not necessarily replacement and even on that augmentation piece um people are watching closely to make sure that uh these things the vendor claims about you know enhancing efficacy and enhancing efficiency and all the rest are actually being able to be can actually be born out um before they sort of commit to broader ad op option or even more autonomy given to these to these systems yeah that makes sense and actually I want to go back rob you mentioned sort of the binary analysis side of things and that feels a little bit less um intuitive to me as to how llms might apply when it comes to sort of software binary analysis is there more that you can dig into there sure I think a lot of that ties into the word language in the large language model um at at a fundamental level what what to me at least is code and that's language and that refers to higher level code uh like scripted languages like python or lower level languages like C even all the way down to the binary level those are still regular organized series of instructions and it turns out we have tons and tons of binaries from over the years that we can use for analysis also so I think the ability for kind of these predictive systems to be able to walk through and recognize pieces of binary that are similar or or look kind of like things they've seen in the past open up a great opportunity uh in the area space like like I said a binary bill of materials or even uh kind of the potential for zero day vulnerability Discovery yeah interesting so we've talked about a couple of use cases one um around binary analysis another sock analyst augumentation um another area where we have certainly seen some activity is around the use of geni in red teaming um Rob is that something you can talk to sure but I before I answer that I want to be careful to kind of Define what we mean by Red teaming there's there's two different ways we can think about that one is how do we red team these new generative AI systems that behave a little bit differently than the software applications we've traditionally seen or conversely how do we leverage these tools to perform the the actions of or augment the capabilities of traditional offensive red teamers in an organization security organization I'll start with the second one there which is of how can we leverage these new machines to augment the capabilities of our red teamers kind of like a stock analyst red teaming is a specialized skill we don't we don't have a plethora of them in the industry or within government so there's been a lot of interest around how do we how do we take these Technologies and use use gener AI to augment them primarily I'd say we've seen companies looking at the idea of breaking up red teaming tasks into fundamental task and and giving giving each individual kind of piece of the puzzle to llms and chaining them all together in order to allow red teamers to be more efficient and effective if we think about more from the from the opposite side is how do we use how do we red team the gener of AI models themselves it's it's a completely different Paradigm than we're used to thinking about red teaming traditional software traditional software you kind of have constrained inputs and outputs you can do things like fuzzing traditional software with gener AI and llms you've pretty much got an infinite set of inputs and outputs so how do you determine what you're going to do how do you determine what you're going to Red Team what are you even trying to red team against are you trying to prevent uh noxious responses from a from a chat bot are you trying to prevent an image generator from generating inappropriate material you really got a lot to think about um from a security standpoint uh how are you preventing these machines from leaking pii or leaking sensitive company data or in the case of some of the National Security Community classified data uh these are Big questions that the entire industry not just the National Security Community is grappling with yeah yeah when I think about red teaming you know part of that is kind of thinking like the adversary and how's the you know how how's the adversary potentially going to use AI uh or generative AI you know from an offensive standpoint so great yeah so you know we've talked about a number of use cases again anomaly detection uh sock automation red teaming will is there anything else that you're seeing in the market around use cases of generative AI in security so I think those probably cover the most uh obvious ones where we've seen the most amount of activity to date I'm sure other ones will pop up um uh in coming years but this is where we've seen the most amount of activity and the greatest promise to help give relief to overburdened security teams some of the things that I want to make sure we highlight about these use case areas right the greatest immediate applicability of these things are segments dealing with high volumes of data processing and repeatable tasks these are also places where uh we've seen the greatest amount of challenges that security teams have cited you know since time in Memorial and uh you know leveraging some of these generative AI enabled products could hopefully uh yield some some positive results there um but again coming back to what will what will ultimately uh kind of shift the tide or or or you know prove these things out is buyers are looking for things that can actually solve a problem and we will start to see evidence of this hopefully um incoming uh in the coming year coming years um particularly as companies that have been funded uh you know actually show that they can acquire customers that they can renew customers once they go through these renewal Cycles that's when we'll really start to have some good data on um which segments uh these Technologies appear to be um best suited to help uh address today or soon yeah there's no question there's a lot of enthusiasm in the market but we need to see kind of what translates into actual dollars contracts uh before we know exactly what's what's going to stick for sure um just kind of Switching gears a little bit so Rob we talked about some of the potential for enhancing cyber defense using geni but we should also think about some of the risks that the technology introduces in the hands of our adversaries uh what new challenges are we facing on that front and how how are they being handled right now well when you say adversary there I'm going to think of it from two different angles one is kind of the the traditional National Security adversary of foreign Nations but I also want to think about it from a cyber crime standpoint because I think that's a lot of where we're seeing generative AI applied from an offensive standpoint right now uh anecdotally when we were doing industry research for this in this general of AI and cyber security space we spoke with a Fortune 50 ciso and asked him if they'd seen anything interesting from from an adversarial cyber standpoint that correlated with J of AI usage and he said right after wide GPT availability they they saw this interesting pattern of new variations into the into password spray attacks different than they'd ever seen before and they really thought that might be correlated with an attacker using GPT or some other gener of AI technology in their attack and of course everyone's aware of the implication of deep fakes and and deep fakes being enabled by generative AI um there's significant reporting around criminals using both audio and video and scams like Pig butchering and along with the concept of you know fishing J AI can help you know non-native speakers like a lot of these cyber criminals are sound more convincing to Targets in different languages and you know this this general idea of deep is something that countries around the world are going to have to Grapple with when it comes to not only you know normal cyber crime but also things like malign influence campaigns and something that really concerns right I have to stop you for just a second Rob because you said something Pig butchering Pig butchering what does that mean I apologize that is a that is a colloquial name for a type of uh cyber crime for lack of a better term in which uh victims are scammed by cyber criminals in order to typically uh contribute large amounts of money or cryptocurrency to fraudulent activity around deep fakes really concerns me and something that I haven't honestly seen a lot of reporting on in the public yet but I expect we will see quite a bit more on over the next year or so is what happens when some of these deep fake and other Technologies go multimodal meaning what happens if for example uh I get a convincing fishing email that reports to be from my manager or my CEO asking me to call you know to take care of some urgent task and then when I call that number I listen to a deep fake audio of this alleged person uh telling me to do something I'm not supposed to be doing how as kind of a cyber security industry do we do we grapple with that I think we've developed reasonable ideas around things like training against fishing attacks right training against emails that look fishy or training against taking calls from strange numbers but as all of these things come together and become more and more convincing I'm not sure as an industry we have a great a great thought process on how we're going to tackle that problem yeah I would I would agree with that and um one thing that I would also highlight there is just taking the that multimodal example um and kind of zooming out a little bit it's a great it's a great way to highlight some of the some of the unique challenges that are posed by um the adoption or the rise of these uh generative AI tools right so we know what fraud looks like we know what fishing looks like but we haven't necessarily seen it in these types of combinations before uh which means that the tools that um Enterprises and and governments rely on to address those challenges may or may not be particularly well suited for this this sort of new types of combinations new types of attacks um so that's what leaves the door open for for Innovation um even if the immediate Market opportunity isn't isn't exactly clear um so uh an exciting time uh to be sure yeah definitely I mean we've talked about some of the ways that Defenders are going to be able to use geni uh to defend their organizations and then also how potential adversaries are going to use gen um in their attacks so I guess maybe just to wrap it up um looking at some of the trend lines and emerging Technologies will and Rob which what predictions do you have for future applications of gen in security Rob maybe we'll go to you first okay sure yeah well as you mentioned earlier Katie it was very striking at black hat to walk the convention floor this year and see the amount of generative AI everywhere that there was literally everyone seemed to be shoehorning it in somehow whether it needed to be or not we probably as industry don't need chat bots on literally everything but I think we are going to see quite a few companies try to use this technology in innovative ways um like I said some of these some of these implementations are clearly better thought out than others but it's not clear to me that anyone has yet come up with a true killer app in the cyber secur space for gener of AI so I think there's going to be a bit of Fallout down the road as the market picks winners and losers and some companies try truly Innovative uses of the technology in my personal opinion I think some of the early wins are going to be in areas that I spoke about earlier uh areas that let us process data at scale where it's been difficult to before like the uh security operations center example I gave early or even the binary analysis problem but you know long term I think we're a little bit of wait and see to re see what really sticks in terms of success for G of AI implementations yeah that makes sense will what what are your thoughts yeah I I agree broadly with what Rob said I think from a market perspective um I think we're we're through that initial burst of enthusiasm and attendant funding that we've alluded to uh earlier in the conversation investors as I mentioned are definitely now going to start to look for um evidence that these companies can execute right acquiring customers prove out value and ultimately make it through through renewal Cycles um I think there'll certainly be some wash out as Rob just said and I think it's still an open question as to whether uh incumbents or startups have the advantage with respect to these applying generative AI for security use cases um I think it'll be certainly use case dependent um but in any case it's it's uh it's never ad moment in the cyber security industry and and this this moment is is certainly no exception great well I appreciate those answers um maybe just to kind of close out um Rob can you talk it all about you know we you know we've talken We've Spoken Here sort of generally around market trends um what we're seeing on the defensive and offensive side how does this relate to National Security and and what are we seeing if it's if you can talk about it sort of at the uh Federal level sure um you know I think when you're dealing with the the federal level the National Security community in particular you're dealing with a customer in Industry that is typically sigh or hesitant to use new and emerging Technologies especially new and emerging Technologies like generative AI that use data in ways that are a little bit different than we've seen before um in you know unpredictable non-deterministic ways not always having a full understanding of where your data is where it's been going that's something that makes um a lot of the organizations we work with very nervous but I think there's also a lot of optimism that uh some of these Technologies are going to kind of enable us to get more of a leg up than we've had on um on adversaries in quite some time um you know not not to go back to the same example I've gone back to several times but the whole idea of you know we as industry we as a national security Community have uh terabytes of log data um we haven't been able to successfully process this at scale there's probably evidence in there of adversaries being in uh sensitive and less sensitive government networks um is generative AI going to give us the capability to find those things even if they're even if they've been there a while find them identify them and help us figure out how to get rid of them I think there's a lot of really really good optimism in the National Security space that you know this technology is going to enable us some wins in areas like that yeah I would I would also add um you know sort of from a Workforce perspective as well uh it's very well documented you know that Enterprise struggles to hire uh Security Professionals cybercity professionals um you know there's there's certainly more open uh more open cybercity jobs than there are um than there are people to fill them this has even been the topic of of like advertisements and other podcasts I've I've heard recently um and the that Dynamic is certainly felt among um among the the federal partners that we serve and so to the extent that um to the extent that some of these Technologies can help alleviate some of those challenges uh with whether it's you know just helping people do their jobs better faster more efficiently um getting some of that uh efficiency into into their workflows alleviating burnout um alleviating turnover and all the rest that would be a huge one for our our community who feels this challenge perhaps more acutely than than Private Industry yeah that's a great place to wrap things up um in terms of commercial markets certainly feels that there's a scale of data and challenges of the workforce but just given how large uh the defense industrial base and the National Security Community is it's just at another level in terms of the scale of the data and the challenges of the workforce so well thank you will and Rob this has been great really appreciate all of your insights and commentary here um and thank you for tuning in to today's episode of I iqt explains on the iqt podcast please make sure to subscribe to the iqt podcast so you don't miss out on future content content and leave us a review or comment to let us know what you think or what content you might be interested to have us cover on a future podcast I also encourage you to check out iq's website at www.q.org to explore more content and learn about iq's global investment platform that accelerates the introduction of ground groundbreaking Technologies to enhance the National Security and prosperity of America and its allies [Music]