Building an AI Cyber Threat Intelligence Dashboard

So this kind of, you know, every solution has some sort of problem associated with it. And the thing that I saw was, you know, every day, there's new vulnerabilities, attack reports, in threat intel that's coming from a lot of sources. But, you know, behind all of that clutter of data, there might be, you know, 1000 vulnerabilities that are published, but only 1 or 2 are actually gonna be exploited in your modern day. So the issue with that is that, you know, it doesn't come labeled to us. Right? So there's I found a stat that it said, there's over 240,000 publicly known software vulnerabilities, but maybe only 10 to 20 are actually exploited, daily. So the issue is that it's hard to tell which 1 of these are the ones that are gonna be exploited. And by the time people figure these out, it's often too late, and it's already affected their you know, whether that be their computer, their production system, whatever it may be. So a lot of the threat hunting that happens today is manual. So people go they read a lot of sources, a lot of feeds, and, you know, they try to make a mental map of how does this connect to another source and how does this connect to another source. But, again, it's a manual inefficient process. And, essentially, why I created this was instead of just collecting data, this actually makes that data explainable and actionable so people can know exactly which vulnerability to patch on that day rather than focusing on the wrong lead. So, again, the purpose of this presentation and this product is to turn fragmented threat intel intelligence into, prioritized and actionable insights for a user. So I'll kind of walk you through the, dashboard itself, kind of talk about how I got data and all of that. So this is my threat intelligence feed that I have up here, and this, scrapes very popular cybersecurity feed. So you you might have heard of some of these hacker news, crepes on security, if you've been in the cyberspace. But these are live, scrapes of this data, and it updates every 10 minutes, which I've, you know, manually set in. So, essentially, how this data collection itself happens is each source has a different way to scrape. So, for example, Hacker News is scraped with XML files and, you know, direct RSS scraping, while something like bleeping computer might use bleeping BeautifulSoup-based scrape. So 1 of the challenges that I had with scraping was every single news source had a different way of getting information, and I had to account for that. And for the third 1 and there's obviously, the malware samples we see on the next page, that was used, with an API. So the data was, obviously, much easier to get from that. Some of the engineering decisions I made, while getting this data. Well, first 1 is I wanted to make sure that, we can autonomously scrape this data on, you know, a a time that we want to scrape it. So I set that time as every 15 minutes, new live data is gonna come into our dashboard. And the way that I had before was, you know, we click a button to regenerate data, but that didn't really make sense for me because you want to always have the most recent data in your dashboard. So how I did that was every source that I have up here. So for example, you know, I have 5 sources up here. I created a separate thread for each of these sources. So there's a daemon running in the back for each source, and each source collects that data within, you know, 10, 15 minutes, whatever you said. The second engineering decision that I wanted to make was, you know, once you get that data from the back end from your scraping, how can the front end dynamically upload? So the way that I, again, I had it before was you click the button, everything, you know, changes and up updates at the same time. But I was preparing for a system design interview, actually. So I figured out a concept called server System events. And, essentially, what that does is once your back end scrapes and gets that data, it automatically pushes it to the front end, and the front end know, collects that data and updates it lifetime. So this was just some engineering decisions I made just to take that, you know, the bare bone vibe coded model, you can say, and it of to make smarter engineering decisions. So now if we move on to our malware samples, these are live collected hashes of vulnerabilities that are existing currently. So just to kind of cross check, there's a tool called VirusTotal, which says, you know, these are malicious hashes and malicious files that exist in you know, that have been exploited. So, again, this scrapes all of that information, puts it onto 1 clean dashboard. And together, these 2 things can be exported as a CSV, or any form of, you know, data for your, tool. So LLC of, you know, security operation centers for, you know, a college, whether that be, you know, big company, they use a tool called Assayam, which essentially, you know, monitors every all the traffic that hits that endpoint of that company. So what this does is you export all of your indicators of compromise, and you upload that into Assayam. And what a company can do with this is they can see these are active threats that I've generated from a CSV. Do I have those threats in my environment right now? And if I do, how can you know, I I have can proactively see all of these, and I can act upon these. So this is where normal feeds stop. So they collect data, and then they Staff, and they leave the interpretation up to, you know, the the human. But I wanted to take this 1 step further by actually synthesizing that data and using some sort of threat intelligence model. So, again, the biggest thing is LLC of there are a lot of vulnerabilities. People know there's a lot of vulnerabilities. You know, being someone who's defending from attacks, the attackers are using the same AI that defenders are using. So if anything, they should be more, you know, Deep at hacking you than you're adept at defending from those vulnerabilities. But the biggest Project, again, that exist right now is people don't know which vulnerabilities are best to act upon and which vulnerabilities you should prioritize acting upon. So I created a CVE risk Georgia, and a CVE is essentially a common vulnerability event. So essentially says, you know, this is a vulnerability that exists. How likely is it to, you know, be exploited? So how I created the score itself was a combination of different API calls. So the entire workflow of how I got this of. So I get the data, then we scrape the entire body of the article. We scrape the titles. We, you know, try to map out specific, you know, information. So for example, IPs, hashes, domains, CVEs that we can find. And for every CVE, I made 3 separate API calls, which, I used to create 1 sort of composite score. So the 3 things that I called to create the score, first 1 is how, you know, how dangerous is the CVE if it's it's on a scale from 0 to 10 with 10 being the most dangerous. So if we think of something like a 0 click attack, which is extremely dangerous, it would be ranked as 10 on the CVSS scale. But just using that is not really, you know, useful for finding what's urgent. Because although a 0 click attack might be a 10 on the CVSS scale, people have not found a way to exploit it. So it's not urgent in your, you know, system. So I use another API to find how likely that exploit is, you know, likely to be exploited in the next 30 days. So there's a database that stores that information, and then I use that as well. And, yeah. So those are the main ones I used here. Just some other visualizations we can see. I created this with d3.js, which is, I think, the same thing that you used, but, just a little, you know, mind map of how things relate to each other. We can see that certain articles relate to certain part of something called a cyber kill chain. So it groups and clusters them into each of these articles. We also Deep, an aggregated profile of threat actors. So these are people that, you know, have notoriously been known for hacking into, you know, production environments. And just having, you know, consistent profile for them to see how they've been acting upon, you know, time is pretty useful, I thought. 1 of the cool features that I built here was some sort of semantic search to understand, you know, what's going on in your environment. So, for example, if an education LLC security operations Member team wants to use this platform, they could search specifically for how what are security risks in education today. So how this system works is it's fully based on a RAG architecture. So every single article that I have within my database is vectorized, and it's portrayed as little dots in vector space. And when you put a search in, for example, you can search up ransomware targeting health care. Your search itself is also vectorized. So you have your search as a tiny point in vector, and then you have a bunch thousands of other articles that you've scraped, which are also vectors. And we use a model called the KNN algorithm. And what that does, it it says how similar is your query search to the articles you have in your database and returns the most similar articles based on your query. So, again, a lot of chatbots, a lot of rag systems, especially in production systems, use this type of architecture to get, you know, chatbots working with your context and your data. So I, again, employed that idea. Final thing I kind of wanna show is an executive dashboard. 1 thing that I thought was, you know, this might be a project that's very, you know, cybersecurity focused and only people in the industry might kind of understand or find use in it. But just any general person who's interested in cybersecurity, they don't want to be reading all of these articles. They don't want to be, you know, going through numbers and CVEs on a daily unless, you know, you like that. But having an executive dashboard is very useful because you can see a 1 sentence summary of, everything that's going on in your day to day, cyber operations. So as we can see here, it passes all of that context into an LLM. I use, GPT 4 0, I think, for this 1. But you get a situation summary. You get your priority Dives. And this is especially useful for, you know, CISOs in your companies who don't wanna be looking at specific numbers and IDs. They just wanna understand, you know, this is the big picture idea of what's going on in my environment. This is the things I have to patch today based on my urgency Georgia. And, you know, they just want the big picture idea. But, yep, this is, you know, a quick thing that I made. Thank you.