The world of Big Data is here, but how to manage it? That was often the view in the intelligence community. Mountains of information used to be more of a hindrance than a help, simply because there were limited ways to use it and challenges storing it. Thankfully, the rapid evolution of artificial intelligence over the past 20 years has enabled big data to be used effectively, by making use of machine learning, data scanning and AI software algorithms. Now, AI and big data actually complement each other, allowing organisations with investigative functions to make decisions based on data-driven insights. Enter the world of algorithm enabled analysts.
AI and big data actually complement each other, allowing organisations with investigative functions to make decisions based on data-driven insights.
Consider the mechanisms of collection. What used to be coveted but difficult methods have become mainstream. While classified means of collection remain crucial to the intelligence field, the sheer amount of data available from social media, the deep and dark web, and the internet in general, has created an environment in which oversaturation and the paralysis of choice have become real threats to decision making and action. This is combined with the ever-increasing capabilities of privately owned satellites, skilled open-source researchers and intelligence practitioners, information extraction tools, and the willingness of people to generate content.
That last point – citizen generated content - is key. The advance of social media means that indicators and warnings of a coming event, whether political, environmental, civil, or conflict, is most frequently reported by citizens themselves. What this means is that social media platforms are often go-to sources of actionable insight, whether through regular journalists, concerned by-standers or people seeking to draw attention to themselves.
What you end up with, of course, is a huge amount of content, which begs the question: how do analysts, officers and other investigators navigate a world where people communicate almost entirely through content?
Typically, they'd use a funnel-like approach in which large volumes of information is collected, processed, analysed and then disseminated as per the standard intelligence cycle. However, when classified content is combined with open-source data, analysts are then faced with the challenge of gaining actionable insight in a restricted time-frame. Instead of finding the needle in a haystack, it’s trying to parse all the needles to find the right one. And this is without even considering the rise of misinformation, disinformation and the difficulties in verifying content.
However powerful software algorithms become, they will never completely replace analysts - mostly because humans are capable of unique insights based on situational, environmental or cultural factors. The objective should be to combine human intuition with AI and machine learning, so that AI can learn not only from data but from humans directly.
With that said, there's still a need for someone to do the grunt work, and this is where AI's ability to recognise patterns, trends and anomalies in data comes into play. When applied to big data, AI can:
The objective should be to combine human intuition with AI and machine learning, so that AI can learn not only from data but from humans directly.
And that's just the beginning. Analysts can be supported by a team of 1,000 ‘virtual interns’, who can process both structured and unstructured information to generate a list of red flags. These flags act as investigatory pivot points, allowing the analyst to rapidly manoeuvre through huge amounts of data in a more efficient and effective way.
Effectively, we now have shortcuts on the road to insight. Here's a good analogy: imagine a tourist in a foreign city where the AI acts as a guide through the winding and confusing streets, taking the person to sights of interest but even more importantly, imparting local knowledge gained through a complete understanding of the environment.
What this means is not only does the melding of AI and big data provide better insights, but it's much less labour-intensive, so it's a significant time-saver as well. Something that could have taken weeks, now only takes a day or two. Much of that is due to the fact that machine learning can be used to 'cleanse' the data before it's analysed - a process that used to be time-consuming and tedious. Now, machine learning algorithms can detect missing values, duplicate records, detect outlier values, and normalise the data before it's analysed, meaning that the insights gleaned from that data are more trustworthy.
Before AI and machine learning began to work together on the big data problem, data analysis was primarily used to better understand past events and gain insights into what might happen if that event reoccurred. Now, analysts can use data in a predictive way; instead of saying "this is what happened," they can now say "this is what is happening", or most importantly enabling decision makers by understanding “what might happen.” This is critical in the world of intelligence by facilitating proactive responses.
The most exciting thing is that the possibilities of what we can do with AI and big data are endless; we've just scratched the surface. The ability of AI and big data to provide context, insight and answers means huge potential for the intelligence industry. It's the kind of technology that needs to be harnessed now, and updated as it evolves.