Open source intelligence

How public security agencies can use it to tackle crime and disorder

By Pete Snelling, principal technical consultant, SAS

Within the intelligence community, open source intelligence can be defined as any information collected from publicly-available sources that then has value added to it by a process of analysis. Data collected from newspapers, magazines, television and radio; public data, such as budgetary and scientific reports and Internet content, including blogs and social media are just some examples of sources which have the potential to generate open source intelligence.

Why it matters

There is huge potential for public security agencies in optimizing the way they use this intelligence. Using publicly available data to understand the demographics of a town, for example, could help a law enforcement organization decide how police officers could most effectively be deployed.

Agencies need solutions they can own, tune, extend and expand to deal with any issue they may face.

Open source intelligence can also support forward planning. Publicly available datasets like that provided by the Armed Conflict Location & Event Data Project (ACLED) can help enable agencies to understand the tactics of terrorist groups, like Boko Haram in Nigeria. This enhanced insight into the way such groups operate gives agencies greater awareness of their modus operandi and potentially helps them develop plans as to how to intervene. Monitoring open source intelligence generated from social networking is also key for agencies. Police would be interested in tracking details of planned demonstrations on social media, especially if they could detect ‘chatter’ from rival groups about intentions to disrupt.

Unfortunately agencies also face difficulties in optimizing the way they use this intelligence. The sheer volume, variety and velocity of open source data makes it hard for agencies to interpret and act on it quickly. Compounding these challenges, most open source data is unstructured - it could be in multiple different languages and the tone of that language is often informal, colloquial or coded.

Scoping the solutions

So what technologies are available to help enable agencies track and analyze open source information and turn it into actionable intelligence? Once they’ve collected the information they need, using a combination of web crawlers, Application Programming Interfaces and, possibly, aggregators, they can start working out what it all means.

One way of extracting meaning from the information is Natural Language Processing (NLP) which is key in helping agencies make sense out of, and create structure from, documents. Rather than simply saying that a document lists the word drugs, it is possible to work out what the intent of the document is. Is it someone talking about manufacturing drugs or selling them illegally, for example? The next challenge is to pinpoint the relevance of the information to which the agencies have given meaning. A key tool used to do this is sentiment analysis, the objective of which is to try to understand the sentiment of the data and ensure only relevant data is pulled.

Next, agencies can extend the benefits they glean from this ever-expanding pool of information by applying analytics. Indeed, by using analytical techniques on their comprehensive integrated data sets, rather than just open source or operational intelligence in isolation, agencies can gain much greater insight from that data.

As part of the analysis process, agencies need to look at the intelligence they have developed from open sources as part of the broader pool of information that they have at their disposal and then join it back to other forms of intelligence like human intelligence; signals intelligence and geospatial intelligence, for example.

The final part of the process is visualization. There is no point in collecting data if you can’t subsequently turn it into a form that a senior decision-maker or operational user such as a police officer, border guard or intelligence agent can quickly and easily make sense of.

Looking ahead

Open source intelligence is a critically important resource for public security agencies to leverage today in their battle against crime and disorder, providing invaluable insight into the nature of the challenges they face and enabling them to proactively plan ways of tackling them. However, simply buying a black box is not the answer. Agencies need solutions they can own, tune and extend and expand to deal with any issue they may face. After all, with the incredible growth of available data showing no sign of abating, it is imperative that agencies are looking at everything they can put in place to ensure that social media and all open source intelligence remains a valuable tool not just today but long into the future.

open source intelligence

Read More

  • Intelligence professionals need to access all available, appropriate information to keep ahead of potential threats. Learn a more effective way to tackle crime.
  • Read more about how analytics can help agencies sift the big data for the important answers.