All-Source & Artificial Intelligence Analysis
Be like water making its way through cracks. Do not be assertive, but adjust to the object, and you shall find a way around or through it. If nothing within you stays rigid, outward things will disclose themselves.
Empty your mind, be formless. Shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle and it becomes the bottle. You put it in a teapot, it becomes the teapot. Now, water can flow or it can crash. Be water, my friend. — Bruce Lee
TOPICS
- All-Source Intelligence
- AI — Man & Machine
- OSINT Tool
ALL-SOURCE INTELLIGENCE
Formless, shapeless, and without void are the three significant characteristics that the book of Genesis uses to describe the Earth at the beginning of creation. Coincidentally, Bruce Lee also mentions these characteristics as a way to reference the state that an individual must achieve if they want to be a successful mixed martial artist.
In the realm of intelligence analysis, the concept of ‘formless’ is particularly significant. It denotes a state devoid of bias and preconceived notions, allowing for a more objective and accurate analysis. Similarly, ‘shapeless’ represents a relentless determination to achieve a desired outcome, and being ‘without void’ is meticulous, ensuring that all relevant data is considered, and no critical information is overlooked.
Furthermore, mixing intelligence disciplines is similar to mixing martial arts, incorporating the best aspects and forming an entirely new skill set. This is the basis of what All-Source Intelligence is: the culmination of the best analytical aspects of multiple intelligence disciplines that fit together to produce a tradecraft capable of producing proximate realities with a higher degree of likelihood.
AI — MAN & MACHINE
The qualities described above that make an overall successful All-Source Intelligence Analyst take considerable time and effort to master. It takes work to maintain a high level of discipline or have continual exposure to environments where an individual analyst can hone their skills, such as a case officer or individuals working in highly technical fields. Still, analysts who have yet to understand their tradecraft can overcome their shortcomings by utilizing AI (artificial intelligence) to achieve high productivity levels. Synthesizing human ingenuity with artificial intelligence creates a counterbalance
Moreover, it’s equally important to consider that AI is not meant to replace the analytical mind of the analyst but only to enhance productivity through technological means. Focus, drive, and determination are qualities instilled in all human beings, but what separates the analysts is a deeper understanding of themselves, the world around them, and the task they set out to achieve.
I will demonstrate this by breaking down the steps and frameworks I used to build an educational open-source intelligence tool that primarily focuses on teaching individuals with a minimal technical background about coding, data visualization, and achieving hyper productivity.
OSINT TOOL
DataScraperMap is an open-source intelligence tool designed to teach coding, data visualization, and productivity skills. This project combines real-time news data with interactive map visualization, providing an engaging way to learn about global events while developing technical skills.
Throughout the development process, I used Cursor AI IDE to achieve focus and high levels of productivity because it’s great for fixing errors in code, scanning the codebase, and aiding in turning logic into a working product. Incredibly helpful. Cursors generation of intelligent code completion, real-time error checking, and even AI-powered coding assistance, making the learning process smoother.
Python was chosen as the primary language for data scraping because it has a variety of libraries that aid in data gathering and visualization, specifically the newsapi, which was the primary source of information utilized to display on the map from the various countries that I tested and were successful to the map which is an OSM (Open Street Map) displayed using Leaflet.
I chose Leaflet because of its performance and usability. It works efficiently across all platforms, can be integrated with many plugins, and has a beautiful, easy-to-use, well-documented API that will make you feel at ease with its use. Its source code is also simple and readable.
from newsapi import NewsApiClient
from app.models import ScrapedData
import time
# Initialize NewsAPI client
newsapi = NewsApiClient(api_key='your_api_key_here')
def scrape_data():
countries_and_languages = {
'us': 'en',
'br': 'pt',
'cn': 'zh',
'ru': 'ru',
'gb': 'en'
}
categories = ['business', 'entertainment', 'general', 'health', 'science', 'sports', 'technology']
new_items_count = 0
for country, language in countries_and_languages.items():
for category in categories:
try:
top_headlines = newsapi.get_top_headlines(
country=country,
category=category,
language=language,
page_size=100 # Adjust as needed, max is 100
)
print(f"Fetched {len(top_headlines['articles'])} articles for country: {country}, category: {category}")
for article in top_headlines['articles']:
title = article['title']
description = article['description'] or "No description available"
url = article['url']
source = article['source']['name']
location = f"{country.upper()} - {category.capitalize()}"
# Check for existing data
db = ScrapedData.get_db()
cursor = db.execute('SELECT id FROM scraped_data WHERE name = ? AND url = ?', (title, url))
existing_data = cursor.fetchone()
if not existing_data:
ScrapedData.add(title, location, description, url, source)
new_items_count += 1
time.sleep(1) # Respect API rate limits
except Exception as e:
print(f"Error fetching news for country: {country}, category: {category}: {str(e)}")
return new_items_count
def get_all_data():
return ScrapedData.get_all()This Python script utilizes the newsapi, which can retrieve top news headlines from worldwide sources. Still, for testing purposes, only a few countries are inputted. It checks for URL, description, source, and title details. Then, it checks if the article already exists in the database. If it’s new, it is added using the ScrapedDatamodel.
I had an idea: the AI tools to achieve hyper-focus productivity and the human ingenuity to create a tool I wished existed. Although an educational and open-source intelligence tool designed as a hackable playground to educate students and professionals.
It took little time to implement or iterate because of human and AI synthesis, which is a crucial point of understanding that analysts can utilize in other ventures if the states mentioned in the first paragraph that make up a successful All-Source Intelligence Analyst are ever to be achieved.
Don’t let your biases and preconceived notions about AI and technology limit how useful these tools can be in overcoming limitations and allowing analysts to learn how to combine intelligence disciplines and build tools that produce more likely proximate realities.
