Before we look at common sources and applications of open-source intelligence, it’s important to understand what it actually is.
According to U.S. public law, open-source intelligence:
- Is produced from publicly available information
- Is collected, analyzed, and disseminated in a timely manner to an appropriate audience
- Addresses a specific intelligence requirement
The important phrase to focus on here is “publicly available.”
The term “open source” refers specifically to information that is available for public consumption. If any specialist skills, tools, or techniques are required to access a piece of information, it can’t reasonably be considered open source.
Crucially, open source information is not limited to what you can find using the major search engines. Web pages and other resources that can be found using Google certainly constitute massive sources of open source information, but they are far from the only sources.
The Different Types Of OSINT
OSINT includes all publicly accessible sources of information and this information can be found either online or offline, in the airwaves and on paper. You can gather OSINT from: The Internet, including forums, blogs, social networking sites, video-sharing sites like, wikis, Whois records of registered domain names, metadata and digital files, dark web resources, geolocation data, IP addresses, people search engines, and anything that can be found online. Traditional mass media, including television, radio, newspapers, books, magazines, specialised journals, academic publications, dissertations, conference proceedings, company profiles, annual reports, company news, employee profiles, and résumés. Metadata in photos and videos and geospatial information from maps and commercial imagery. OSINT can be gathered from almost anywhere and even the most unlikely of places can provide you with valuable intelligence on the subject of your investigation.
Types Of Specialist OSINT Organisations
A number of specialised organisations provide dedicated OSINT services, some of them are government based and others are private companies that offer their services to their customers, often government agencies and business corporations on a subscription basis. The following are the most publicly well-known OSINT gathering organisations:
Government Organisations — The Open Source Center (https://fas.org/irp/dni/osc/index.html) is one such organisation, one that is controlled and operated by U.S. government. BBC Monitoring (https://monitoring.bbc.co.uk/) is another, a department within the British Broadcasting Corporation (BBC) that monitors foreign media worldwide. They offer their services via subscription to interested parties such as commercial and official bodies.
Private Sector — Jane’s Information Group (https://www.janes.com/) is a British company founded in 1898 and a leading provider of OSINT that specialises in military, terrorism, state stability, serious and organised crime, proliferation and procurement intelligence, aerospace, and transportation subjects. The Economist Intelligence Unit (https://www.eiu.com/home.aspx) is the business intelligence, research, and analysis division of the British Economist Group. Oxford Analytica (http://www.oxan.com) is a relatively small OSINT firm compared with the previous two, it specialises in geopolitics and macroeconomics subjects.
Different Kinds Of OSINT Gathering
OSINT gathering is done by using one of three primary methods, passive, semi-passive, and active. Using one rather than another is dependent on the scenario and the kind of intelligence that you are interested in.
Passive Collection — This is the most used type when collecting OSINT intelligence, by default most OSINT gathering methods should use passive collection because the main aim of OSINT gathering is to collect information about the target via publicly available resources.
Semi-passive — More technical in nature, this type of gathering sends internet traffic to target servers in order to acquire general information about them. This traffic should resemble typical internet traffic to avoid drawing any attention to your reconnaissance activities. In this way, you are not implementing in-depth investigation of the target’s online resources, but only investigating lightly without launching any alarm within the group you are investigating.
Active Collection — In this type, you interact directly with the system to gather intelligence about it, but The target can become aware of the reconnaissance process since the person/entity collecting information will use advanced techniques to harvest technical data about the target IT infrastructure such as accessing open ports, scanning vulnerabilities (unpatched Windows systems), scanning web server applications, and more. This traffic will look like suspicious behaviour and will more than likely leave traces on the target’s intrusion detection system (IDS) or intrusion prevention system (IPS).
Best 25 OSINT Tools used by InfoSec Professionals
Important note before we start: remember that you should never use these tools on external networks/systems without previous authorization. These OSINT tools are presented here in order to help IT security researchers and private/public infosec investigators during the first phase of information gathering, which is one of the most important parts of a cybersecurity investigation.
1. OSINT Framework
While OSINT Framework isn’t a tool to be run on your servers, it’s a very useful way to get valuable information by querying free search engines, resources, and tools publicly available on the Internet. They are focused on bringing the best links to valuable sources of OSINT data.
While this web application was originally created focused on IT security, with the time it has evolved and today you can get other kinds of information from other industries as well. Most of the websites it uses to query the information are free, but some may require paying a low fee.
CheckUserNames is an online OSINT tool that can help you to find usernames across over 170 social networks. This is especially useful if you are running an investigation to determine the usage of the same username on different social networks.
It can be also used to check for brand company names, not only individuals.
HaveIbeenPwned can help you to check if your account has been compromised in the past. This site was developed by Troy Hunt, one of the most respected IT security professionals of this market, and it’s been serving accurate reports since years.
If you suspect your account has been compromised, or want to verify for 3rd party compromises on external accounts, this is the perfect tool. It can track down web compromise from many sources like Gmail, Hotmail, Yahoo accounts, as well as LastFM, Kickstarter, Wordpress.com, Linkedin and many other popular websites.
Once you introduce your email address, the results will be displayed, showing something like:
BeenVerified is another similar tool that is used when you need to search people on public internet records. It can be pretty useful to get more valuable information about any person in the world when you are conducting an IT security investigation and a target is an unknown person.
After done, the results page will be displayed with all the people that match the person’s name, along with their details, geographic location, phone number, etc. Once found, you can build your own reports.
The amazing thing about BeenVerified it’s that it also includes information about criminal records and official government information as well.
BeenVerified background reports may include information from multiple databases, bankruptcy records, career history, social media profiles and even online photos.
Censys is a wonderful search engine used to get the latest and most accurate information about any device connected to the internet, it can be servers or domain names.
You will be able to find full geographic and technical details about 80 and 443 ports running on any server, as well as HTTP/S body content & GET response of the target website, Chrome TLS Handshake, full SSL Certificate Chain information, and WHOIS information.
BuiltWith is a cool way to detect which technologies are used at any website on the internet.
BuiltWith also lets you find which are the most popular technologies running right now, or which ones are becoming trending.
Without any doubt, it is a very good open source intelligence tool to gather all the possible technical details about any website.
7. Google Dorks
While investigating people or companies, a lot of IT security newbies forget the importance of using traditional search engines for recon and intel gathering.
In this case, Google Dorks can be your best friend. They have been there since 2002 and can help you a lot in your intel reconnaissance.
Google Dorks are simply ways to query Google against certain information that may be useful for your security investigation.
Search engines index a lot of information about almost anything on the internet, including individual, companies, and their data.
Some popular operators used to perform Google Dorking:
- Filetype: you can use this dork to find any kind of filetypes.
- Ext: can help you to find files with specific extensions (eg. .txt, .log, etc).
- Intext: can perform queries helps to search for specific text inside any page.
- Intitle: it will search for any specific words inside the page title.
- Inurl: will look out for mentioned words inside the URL of any website.
Log files aren’t supposed to be indexed by search engines, however, they do, and you can get valuable information from these Google Dorks, as you see below:
Now let’s focus on other more practical tools used by the most respected InfoSec professionals:
Is an amazing tool to track down footprints of any target you need to match. This piece of software has been developed by Paterva, and it’s part of the Kali Linux distribution.
Using Maltego will allow you to launch reconnaissance tests against specific targets.
One of the best things this software includes is what they call ‘transforms’. Transforms are available for free in some cases, and on others, you will find commercial versions only. They will help you to run a different kind of tests and data integration with external applications.
In order to use Maltego you need to open a free account on their website, after that, you can launch a new machine or run transforms on the target from an existing one. Once you have chosen your transforms, Maltego app will start running all the transforms from Maltego servers.
Finally, Maltego will show you the results for the specified target, like IP, domains, AS numbers, and much more.
If you need to explore more Kali Linux utilities, check out this article: Top 25 Kali Linux Tools
Recon-ng comes already built in the Kali Linux distribution and is another great tool used to perform quickly and thoroughly reconnaissance on remote targets.
This web reconnaissance framework was written in Python and includes many modules, convenience functions and interactive help to guide you on how to use it properly.
The simple command-based interface allows you to run common operations like interacting with a database, run web requests, manage API keys or standardizing output content.
Fetching information about any target is pretty easy and can be done within seconds after installing. It includes interesting modules like google_site_web and bing_domain_web that can be used to find valuable information about the target domains.
While some recon-ng modules are pretty passive as they never hit the target network, others can launch interesting stuff right against the remote host.
theHarvester is another great alternative to fetch valuable information about any subdomain names, virtual hosts, open ports and email address of any company/website.
This is especially useful when you are in the first steps of a penetration test against your own local network, or against 3rd party authorized networks. Same as previous tools, theHarvester is included inside Kali Linux distro.
theHarvester uses many resources to fetch the data like PGP key servers, Bing, Baidu, Yahoo and Google search engine, and also social networks like Linkedin, Twitter and Google Plus.
It can also be used to launch active penetration test like DNS brute force based on dictionary attack, rDNS lookups and DNS TLD expansion using dictionary brute force enumeration.
Shodan is a network security monitor and search engine focused on the deep web & the internet of things. It was created by John Matherly in 2009 to keep track of publicly accessible computers inside any network.
It is often called the ‘search engine for hackers’, as it lets you find and explore a different kind of devices connected to a network like servers, routers, webcams, and more.
Shodan is pretty much like Google, but instead of showing you fancy images and rich content / informative websites, it will show you things that are more related to the interest of IT security researchers like SSH, FTP, SNMP, Telnet, RTSP, IMAP and HTTP server banners and public information. Results will be shown ordered by country, operating system, network, and ports.
Shodan users are not only able to reach servers, webcams, and routers. It can be used to scan almost anything that is connected to the internet, including but not limited to traffic lights systems, home heating systems, water park control panels, water plants, nuclear power plants, and much more.
Jigsaw is used to gather information about any company employees. This tool works perfectly for companies like Google, Linkedin, or Microsoft, where we can just pick up one of their domain names (like google.com), and then gather all their employee’s emails on the different company departments.
The only drawback is that these queries are launched against Jigsaw database located at jigsaw.com, so, we depend entirely on what information they allow us to explore inside their database. You will be able to find information about big companies, but if you are exploring a not so famous startup then you may be out of luck.
SpiderFoot is one of the best reconnaissance tools out there if you want to automate OSINT and have fast results for reconnaissance, threat intelligence, and perimeter monitoring.
It was written by our friend Steve Micallef, who did a great job building this app and writing the SecurityTrails Addon for Splunk
This recon tool can help you to launch queries over 100 public data sources to gather intelligence on generic names, domain names, email addresses, and IP addresses.
Using Spiderfoot is pretty much easy, just specify the target, choose which modules you want to run, and Spiderfoot will do the hard job for you collecting all the intel data from the modules.
Creepy is a geo-location OSINT tool for infosec professionals. It offers the ability to get full geolocation data from any individuals by querying social networking platforms like Twitter, Flickr, Facebook, etc.
If anyone uploads an image to any of these social networks with geolocation feature activated, then you will be able to see a full active mal where this person has been.
You will be able to filter based on exact locations, or even by date. After that, you can export the results in CSV or KML format.
Nmap is one of the most popular and widely used security auditing tools, its name means “Network Mapper”. Is a free and open source utility utilized for security auditing and network exploration across local and remote hosts.
Some of the main features include:
- Host detection: Nmap has the ability to identify hosts inside any network that have certain ports open, or that can send a response to ICMP and TCP packets.
- IP and DNS information detection: including device type, Mac addresses and even reverse DNS names.
- Port detection: Nmap can detect any port open on the target network, and let you know the possible running services on it.
- OS detection: get full OS version detection and hardware specifications of any host connected.
- Version detection: Nmap is also able to get application name and version number.
WebShag is a great server auditing tool used to scan HTTP and HTTPS protocols. Same as other tools, it’s part of Kali Linux and can help you a lot in your IT security research & penetration testing.
You will be able to launch a simple scan, or use advanced methods like through a proxy, or over HTTP authentication.
Written in Python, it can be one of your best allies while auditing systems.
Main features include:
- Port Scan
- URL scanning
- File fuzzing
- Website crawling
In order to avoid getting blocked by remote server security systems, it uses an intelligent IDS evasion system by launching random requests per HTTP proxy server, so you can keep auditing the server without being banned.
OpenVAS (Open Vulnerability Assessment System) is a security framework that includes particular services and tools for infosec professionals.
This is an open source vulnerability scanner & security manager that was built after the famous Nessus switched from open source to private source. Then, the original developers of the Nessus vulnerability scanner decided to fork the original project and create OpenVAS.
While it is a little bit more difficult to setup than the old Nessus, it’s quite effective while working with it to analyze the security of remote hosts.
The main tool included in OpenVAS is OpenVAS Scanner, a highly efficient agent that executes all the network vulnerability tests over the target machine.
On the other hand, another main component is called OpenVAS Manager, which is basically vulnerability management solution that allows you to store scanned data into an SQLite database, so then you can search, filter and order the scan results in a fancy and easy way.
Fierce is an IP and DNS recon tool written in PERL, famous for helping IT sec professionals to find target IPs associated with domain names.
It was written originally by RSnake along with other members of the old http://ha.ckers.org/. It’s used mostly targetting local and remote corporate networks.
Once you have defined your target network, it will launch several scans against the selected domains and then it will try to find misconfigured networks and vulnerable points that can later leak private and valuable data.
The results will be ready within a few minutes, a little bit more than when you perform any other scan with similar tools like Nessus, Nikto, Unicornscan, etc.
Unicornscan is one of the top intel gathering tools for security research. It has also a built-in correlation engine that aims to be efficient, flexible and scalable at the same time.
Main features include:
- Full TCP/IP device/network scan.
- Asynchronous stateless TCP scanning (including all TCP Flags variations).
- Asynchronous TCP banner detection.
- UDP Protocol scanning.
- A/P OS identification.
- Application and component detection.
- Support for SQL Relational Output
FOCA (Fingerprinting Organizations with Collected Archives) is a tool written by ElevenPaths that can be used to scan, analyze, extract and classify information from remote web servers and their hidden information.
Foca has the ability to analyze and collect valuable data from MS Office suite, OpenOffice, PDF, as well as Adobe InDesign and SVG and GIF files. This security tool also works actively with Google, Bing and DuckDuckGo search engines to collect additional data from those files. Once you have the full file list, it starts extracting information to attempt to identify more valuable data from the files.
In the cybersecurity world, we researchers are used to popular IoT search engines such as Shodan or Censys. For a while, however, a powerful new IoT search engine has been rapidly gaining followers. We’re talking about ZoomEye.
ZoomEye is a Chinese IoT OSINT search engine that allows users to grab public data from exposed devices and web services. In order to build its database it uses Wmap and Xmap, and then runs extensive fingerprinting against all the information found, ultimately presenting it to users in a filtered and curated way for easy visualization.
What information can you find with ZoomEye?
- IPs interacting with networks and hosts
- Open ports on remote servers
- Total number of hosted websites
- Total number of devices found
- Interactive map of users hitting different devices
- Vulnerabilities report
And much more. The public version offers access to a lot of data — but if you want to see what it can really do, we suggest you sign up for a free account. That way you’ll get to test the real power of this OSINT tool.
Wappalyzer is a highly useful service that allows security researchers to quickly identify technologies on websites. With it, you can find a complete list of details for any technology stack running on any website. It also allows you to build lists of websites that use certain technologies, letting you add phone numbers and email addresses as well.
Their free plan includes instant results and up to 50 free monthly lookups. It’s perfect for tracking website technologies, discovering old/vulnerable software, finding organic data about your competitors, and last but not least, can be quickly triggered from the web browser with their Chrome/Firefox extensions.
If that isn’t enough, they also offer a handy API to automate technology lookups, and you can even set up website alerts to monitor your competition.
This infosec tool is frequently overlooked, but it has great potential in boosting your infosec discovery and analysis processes. IVRE is an open source tool that’s built on a base of popular projects like Nmap, Masscan, ZDNS, and ZGrab2.
Its framework uses these popular tools to gather network intelligence on any host, then uses a MongoDB database to store the data.
Its web-based interface makes it easy for both beginning and advanced infosec users to perform the following actions:
- Passive reconnaissance by flow analysis (from Zeek, Argus or nfdump)
- Active reconnaissance by using Zmap and Nmap
- Fingerprinting analysis
- Import data from other 3rd party infosec apps, such as Masscan/Nmap
IVRE can be installed by fetching the source from their official Github repo, or from 3rd-party repositories such as Kali Linux repo.
Metagoofil is another great intel-reconnaissance tool that aims to help infosec researchers, IT managers, and red teams to extract metadata from different types of files, such as:
How does it work? This app performs a deep search on search engines like Google, focusing on these types of files. Once it detects such a file, it will download it to your local storage, then proceed to extract all of its valuable data.
Once the extraction is complete, you’ll see a full report with usernames, software banners, app versions, hostnames and more, a valuable resource for your recon phase.
Metagoofil also includes a number of options to help you filter the types of files to search for, refine the results and tweak the output, among many other useful features.
While a lot of OSINT tools focus on data found on public files such as PDF, .DOC, HTML, .SQL, etc., there are other tools that are specifically designed to extract critical Open Source Intelligence data from image, video and audio files.
Exiftool reads, writes and extracts metadata from the following types of files:
- And many others
It also supports native files from a wide range of cameras, such as: Canon, Casio, FujiFilm, Kodak, Sony, and many others. It’s also conveniently available on multiple platforms including Linux, Windows and MacOS.
As you can see, there are a lot of recon and intel gathering tools out there. On this particular post, we mentioned only 20 of the most popular OSINT tools, but there is much more to discover. Start digging around and testing other useful infosec tools.
Top 25 OSINT Tools for Penetration Testing (securitytrails.com)
A Guide To Open Source Intelligence (OSINT) (itsec.group)
What Is Open Source Intelligence and How Is it Used? (recordedfuture.com)