A compilation of links to datajournalism & OSINT tools, guides and resources I find useful to keep at hand. PRs welcomed!
by r3mlab | License: CC-BY-NC 4.0
- π = online tool/service/database
- π» = software
- π = guide/tutorial
- π = list of tools/resources
- π = Python module
- π² = paid or paid-only tool/service
- APIs
- Archival
- Breached Data
- Companies
- Data Analysis & Manipulation
- Lists of tools & resources
- Location, Maps, Satellite Imagery
- Military/Weapons
- Multi-purpose tools
- News
- Phone numbers
- Pictures, Photos, Videos
- Social Networks
- Text & Documents
- Transportation
- Visualization
- Weather
- Websites
- Misc
- Postman π» - API development environment offering useful tools for crafting and debugging API requests.
- ProgrammableWeb π - A good API directory.
- Public APIs π - A categorized list of APIs.
- archive.today π - Saves pages as screenshots, useful for websites the WayBack Machine can't handle.
- Firefox Screenshots π» - Firefox can take a screenshot of a full page (i.e. 'scrolling' screenshot).
- How to Archive Open Source Materials π (Bellingcat)
- Hunch.ly ππ² - Web capture tool designed for online investigations ($129.99/y).
- Internet Archive Wayback Machine π
- waybackpack π»π - Command-line utility & Python library to download content from the Wayback Machine. See this example.
- view-page-archive π» - Browser extension to search for a page's archives on 15+ web archival/caching sites.
- Breach Data Search Engines Comparison π (IntelTechniques)
- CardPwn π» - Find out if a credit card number appears in a breach.
- Dehashed ππ² - Find cleartext & hashed password from data breaches (paid, $4/week, $11/mo).
- GhostProject π - Check if an email appears in a breach. Shows the first 3 characters of the password for free.
- h8mail π» - Find passwords through different breach and reconnaissance services. Can also search the BreachedCompilation torrent.
- Have I Been Pwned? π - Check if an email appears in a breach, set up alerts.
- pwndb.py π» - Command-line tool for searching leaked credentials using the Onion service with the same name.
- WhatBreach π» - Search for breached emails and their corresponding database.
- CompaniesHouse Short Guide π (Bellingcat) - A guide about the UK online company registry.
- DocumentCloud Search π - Search public documents uploaded to DocumentCloud, a publishing plateform used by many journalists and media.
- ICIJ's Offshore Leaks Database π - Data on offshore companies, foundations and trusts from the Panama Papers, the Offshore Leaks, the Bahamas Leaks and the Paradise Papers investigations.
- List of company registers π (Wikipedia) - A list of all companies registers, by country.
- OCCRP Data π - Fantastic search tool & resources made available by OCCRP. Public records, leaks, scraped business registers, and more.
- OCCRP Investigative Dashboard π - Collection of the most useful public data sources for investigative reporting. Many business registries listed.
- OpenCorporates π - A very comprehensive companies database. Has an API.
- Open Ownership Register π - Explore beneficial ownership data. Aggregates many datasets.
See also: Visualization
- csvkit π» - A suite of command-line tools for converting to and working with CSV files.
- OpenRefine π» - Clean & transform messy data.
- pandas π - Powerful Python data analysis library. Best used in a Jupyter notebook.
See also: Breached Data
- emailrep.io π - Public email reputation search & API. Can find social media profiles.
- Infoga π» - Gather email accounts information (ip, hostname, country, etc) from different public sources.
- theHarvester π» - Python command-line tool to search several search engines for mail addresses from a particular domain.
- The most complete guide to finding anyone's email π (Blurbiz)
- Trumail π - Free email verification API.
- Citadel π» - A library of OSINT tools.
- IntelTechniques.com π - Blog, podcast, and paid OSINT/privacy training courses.
- Guides π (Bellingcat) - OSINT & Datajournalism how-tos.
- Online Investigation Toolkit π (Bellingcat)
- awesome-osint π - A curated list of open source intelligence tools and resources.
- OSINT framework π - Tree list of OSINT tools & resources.
- OSINT Collection π - Collection of OSINT related resources.
- I-Intelligence's Open Source Intelligence Tools and Resources Handbook 2018 π - Very complete list of OSINT tools & resources, organized by category. No descriptions.
- AutomatedOSINT.com π - A Blog about automating OSINT techniques using Python.
- netbootcamp π - Custom search forms and lists of resources by theme.
- Week in OSINT π - Fresh links to OSINT tools, resources and investigations every week.
- How To Use Google Earthβs Three Dimensional View π (Bellingcat)
- Identify Burnt Villages on Satellite Imagery π (Bellingcat)
- Photo Interpretation Student Handbook π (US Defense Mapping Agency, 1996) - Old unclassified handbook on analyzing aerial & satellite imagery. General principles & specifics for buildings, industries, transportation & communication facilities.
- Using Time Lapse Satellite Imagery to Detect Infrastructure Changes π (Bellingcat)
- Baidu Maps π - Streetview = Panorama (ηΎεΊ¦ε ¨ζ―)
- Bing Maps π
- GeoNames π - Geographical database.
- GoogleMaps π
- Google Earth π
- Google Earth Outreach - Advanced Google Earth tutorials. Example: Image & Photos Overlay
- Google Earth Engine - Datasets, case studies, etc.
- GEarth Blog - Resources & how-tos about Google Earth
- Satellite imagery providers:
- Copernicus Open Access Hub π - Free access to imagery from the European Sentinel satellites.
- Descartes Labs ππ²
- DigitalGlobe Discover π - Search for satellite imagery of a particular location. Ability to download images (low-resolution compared to Google Earth).
- EOS Landviewer ππ²
- NASA EarthData & EarthViewer π
- USGS Earth Explorer π - NASA Landsat imagery
- Here WeGo π
- SentinelHub π - Satellite imagery, historical data from several sources, vegetation infrared & index, image exports & comparison. 2 products:
- Playground - Data discovery, playing around
- EO Browser - Compare full resolution images from several sources (Landsat, Sentinel), make time lapses & export to GIF (free signup required).
- See also the custom scripts to highlight fire, snow, metals, type of terrain, etc.
- Zoom Earth π - NASA satellite and aerial images of the Earth.
- Yandex Maps π - Has a "Streetview" feature.
- Geographic Bounding Box Drawing Tool π - Draw a rectangle over a map and get the coordinates of its points & center.
- PeakFinder π - Show names of all mountains and peaks from any coordinates with a 360Β° panoramic mountain view.
- Shadows and Angles: Measuring Object Heights from Satellite Imagery π (GISLounge)
- Shadows and Suncalc π - Great tutorial on using Google Earth & Suncalc to calculate time based on shadows.
- SunCalc π - Historical solar data (sun orientation & elevation, shadow length, etc).
- TerraPattern π - Scan large geographical areas for specific visual features using machine learning. Only available for 7 cities.
See also: Social Networks
- EchoSec ππ² - Search and analyze social media data based on location. ($499/mo)
- GeoCreepy π» - Geolocation information gathering through social networking platforms (discontinued).
- Kamerka π»- Create an interactive map of cameras, printers, tweets and photos based on your coordinates.
- OpenStreetMap π - User generated locations & maps. Use taginfo and/or overpass-turbo.eu to search a location by key/value tags (see OSM's Wiki)
- Mapillary π - Interactive map of crowdsourced geotagged photographs.
- OpenStreetCam π - Map of crowdsourced street-level photographs.
- Social networks (see category)
- Surveillance under Surveillance π - User-contributed map of cameras and guards.
- Tourism & review websites: Foursquare, TripAdvisor, Yelp, etc. π
- Vkontakte π - Use
near:<coordinates>
in a search. - Wikimapia π - User-generated locations & descriptions. Has an API.
- CalibreObscura π - A blog about weapons & their uses in Middle East conflicts.
- CamoPedia π - Camouflage encyclopedia. Search & compare camouflage patterns.
- ENAAT Data Browser π - Browse EU Arms Export Data.
- How to Digitally Verify Combatant Affiliation in Middle East Conflicts π (Bellingcat)
- ICUS Camouflage Index π
- International Encyclopedia of Uniform Insignia π
- Investigating and Tracking the Global Arms Trade π (Corruption Watch UK) - Good presentation, full of resources of all types on Arms Trade.
- List of Comparative Military Ranks π (Wikipedia)
- Omega Research Foundation's Identification & Documentation guides π - Guides on identifying and documenting police & military equipment.
- SEESAC Reports & Map π - Database of firearms-related incidents in South East Europe.
- SIPRI Arms Transfers Database π - Information on all transfers of major conventional weapons from 1950 to the current year.
- Sketchfab π - User-made 3D models sharing platform with lots of weapons. Useful to compare, check different angles, etc.
- Small Arms Surveyβs Weapon ID database π - Search for small arms by caliber, type, location, etc.
- Small Arms Survey: Documenting Small Arms and Light Weapons π - International policy recap & identification guide.
- UN Comtrade Database π - Official international trade statistics, including arms trade.
- UNROCA π - UN Register of Conventional Arms. Country-level data on arms exports.
- World Army Pictures π - Pictures of armies from all over the world.
- Buscador π» - A very handy VM with plenty of pre-installed & pre-configured OSINT tools.
- DataSploit π» - A collection of python scripts which automates open source intelligence searches about domain names, email addresses, IP addresses and usernames.
- IntelligenceX Tools π - Various search, email and domain tools.
- Maltego CE π» - Interactive data mining & mapping tool.
- Spiderfoot π» - Open source intelligence automation tool. Gathers intelligence about a given target, which may be an IP address, domain name, hostname, network subnet, ASN, e-mail address or person's name.
- AllYouCanRead π - Database of news outlets by country.
- NewsLookup π - News search engine with useful filters.
- NewsNow π - News search engine with useful filters.
- NewspaperMap π - Newspapers world map with feeds and automatic translation.
- NumberWay π - International directory of white pages and yellow pages phone books.
- PhoneInfoga π» - Information gathering & OSINT reconnaissance tool for phone numbers.
- Using Phone Contact Book Apps For Digital Research π (Bellingcat)
- Exiftool π» - Read and edit metadata. Linode Tutorial
- Exif Viewer (Firefox/Chrome) π»
- FotoForensics π - Online pictures metadata viewer.
- Ghiro π» - Automated image forensics tool.
- Jeffrey's Image Metadata Viewer π
- mat2 π» - Metadata removal tool.
- mat2-web π - Online version of mat2.
- StolenCameraFinder π - Search the web for pictures with a specific camera serial number.
- Bing Images π - Can search part of an image by resizing on the fly.
- CitizenEvidence π - Google Images reverse search on Youtube thumbnails.
- EagleEye π» - Find Instagram, FB and Twitter profiles using image recognition and reverse image search.
- Google Images π
- Search by Image π» - Browser extension to quickly reverse-search an image on 20+ search engines.
- TinEye π
- Yandex Images π
- How to Conduct Comprehensive Video Collection (Bellingcat) π
- PimEyes π - Face-recognition matching search engine.
- SearchFace.ru π - Face recognition search engine for the Russian VK social network. See this guide from Bellingcat for a tutorial.
- SocialMapper π - Social Media Mapping Tool that correlates profiles via facial recognition. Supports LinkedIn, Facebook, Twitter, Instagram, VKontakte, Weibo, Douban.
- Advanced Guide on Verifying Video Content π (Bellingcat)
- face_recognition π»π - Command-line tool and python library for recognizing known faces on a batch of pictures.
- How to verify photos and videos on social media networks π (France24)
- InVID Verification Plugin π» - Verification βSwiss army knifeβ Firefox extension.
- Photo Verification Cheatsheet & Video Verification Cheatsheet π (FirstDraftNews)
- Verification 101 π - Storyfulβs advice for checking out material from social media, and putting it into practice.
- Verification Handbook π - Handbook by the European Journalism Centre about verifying digital content in emergency coverage.
- EagleEye π» - Find Instagram, FB and Twitter profiles using image recognition and reverse image search.
- HashAtIt π - Hashtag search across Twitter, Instagram, Pinterest, Facebook and Youtube.
- Sherlock π» - Search for a username across 135 social media sites.
- SocialMapper π - Social Media Mapping Tool that correlates profiles via facial recognition. Supports LinkedIn, Facebook, Twitter, Instagram, VKontakte, Weibo, Douban.
- WhatsMyName π» - Search for usernames on 180+ web sites.
- dis.cool π - Discord search engine.
- fb-search π - Simple Graph query crafter. Made after Facebook sudden closure of Graph Search.
- FFFF Finds Facebook Friends π» - Builds a relationship graph of a target user. Partially reconstructs hidden friend lists. π₯.
- gitrob π» - Find potentially sensitive files pushed to public repositories on Github. Requires a GitHub access token.
- Zen π» - Find emails of Github users.
- instaloader π» - Download pictures (or videos) along with their captions and other metadata from Instagram.
- instagram-scraper π» - Scrape a user's photos and videos.
- searchmybio π - Search Instagram users biographies.
- An Investigative Guide To LinkedIn π (Bellingcat)
- CrossLinked π» - LinkedIn enumeration tool to extract valid employee names from an organization.
- LinkedIn Operators Tip Sheet π
- raven π» - Linkedin information gathering tool. Extracts employee data for a given company.
- The Endorser π» - Draw out relationships between people on LinkedIn via endorsements/skills.
- Reddit Comment Search π - Search through comments of a particular reddit user.
- Reddit Insight π - Collect info on a Reddit profile, list all posts & comments.
- Reddit Investigator π - Collect info on a Reddit profile.
- Reddit Search π - Reddit search engine with filters.
- ReSavr π - Search deleted comments.
- Buzz.im π - Search in open telegram messages.
- Lyzem π - Telegram search engine.
- Telegago π - Google Custom Search Engine for Telegram users & content. Can discover private groups.
- tlgrm.eu π - Search for Telegram channels.
- tgstat.ru π - Telegram analytics & seach tool.
- DMI-TCAT π» - PHP web interface to retrieve and analyze tweets.
- SocialBearing π - Statistics on keywords, hashtags, users.
- SpoonBill π - Track changes in Twitter profiles & bios. Requires a Twitter account.
- tinfoleak π» - Very complete open-source tool for Twitter intelligence analysis. Needs API credentials.
- twarc π»π - A command line tool and Python library for archiving Twitter in JSON format.
- Tweetdeck π
- Tweetdeck Location Search Tutorial π
- Tweet Map π - Explore the world and find geo-tagged tweets.
- Tweets Analyzer π» - Twitter profile analyzer with tweet activity charts, locations, most used hashtags, etc. Can save tweets to JSON. Requires a Twitter API key.
- tweetsmapper π» - Generates a Leaflet map for a given user or from an existing collection of tweets. Can retrieve full timelines.
- TWINT (Twitter Intelligence Tool) π» - Advanced Twitter scraping tool, no API key needed. Can export to text, CSV, JSON, SQLite, Elasticsearch. Can detect emails, phone numbers, profiles.
- Who Tweeted It First? π - Find out who was the first person who tweeted a link, video, quote or any piece of text.
- SnRadar π - Search VKontakte content by location.
- Unlisted Videos π - Search & submit unlisted YouTube videos. No registration required.
- Apache Tika π» - Extract metadata and text from over a thousand different file types.
- FOCA ππ» - Find metadata and hidden information in Microsoft Office, Open Office, or PDF files.
- ICIJ Extract π» - A command line tool for parallelized, distributed content-extraction.
- Aleph π» - A toolkit for data search, management and analysis in investigative reporting.
- Blacklight π» - Open source Solr user interface discovery platform.
- Datashare π» - Index & search documents on your computer, automatically detect people, organizations and locations with NLP.
- DumpsterDiver π» - Analyze big volumes of various file types in search of secrets, credentials, etc.
- ICIJ Extract π» - A command line tool for parallelized, distributed content-extraction.
- searchbox π» - A simple out-of-the-box web interface to search through thousands of unstructured documents using Solr.
- NewOCR.com π - Recognizes several languages. Can resize images & has shortcuts to Google & Bing Translate.
- Tesseract π» - Open-source OCR engine.
- PDF Text Extraction with PyPDF2, Tika & PDF Miner. π»
- tabula π» - Tool for liberating data tables trapped inside PDF files.
- topia π - Python module to determine important terms within a given piece of content.
- TXM π» - Lexicometry and text statistical analysis for large bodies of text.
- BIC Code Register π - Business Identifier Codes lookup. The website also has other search tools and useful information on container markings.
- Prefix List π - Find the owner of a container from its prefix.
- track-trace π - Track parcels/shipments, air cargo, containers and post.
- Flights tracking:
- FlightAware π
- FlightRadar24 π
- PlaneFinder π
- RadarBox π
- PlaneMapper π - Flights, airports, airlines and aircrafts databases.
- Inmarsat Ships Directory π - Find contact details from a ship's name or number.
- Maritime Connector π - Maritime jobs listings & search.
- Maritime Database π - Lists and details of shipping-related businesses and ports of the world.
- Ship search & track:
- VesselsFinder π
- MyShipTracking π
- Fleetmon π
- Shipfinder π
- Marine Traffic π
- CruiseMapper π
- Data Visualisation Catalogue π - Find which visualisation is right for what you want to show. Plenty of tips & resources.
- DataWrapper ππ² - Easy to use graph & map tool. Free plan available.
- Google Fusion Tables - Create maps & charts from data. Will shut down on Dec. 2019.
- Matplotlib π - Python 2D plotting library. Best used with pandas in a Jupyter notebook.
- RawGraph ππ» - Generate static graphs through a very user-friendly interface. Can be run locally.
- ArcGIS π»π² - Mapping & analysis software (proprietary, paid, 21-day trial)
- Folium π - Python library to create Leaflet.js maps. Can be used in a Jupyter Notebook to map data from pandas.
- Geopy π - Python geocoding library. Supports OSM Nominatim, Google, Bing, GeoNames & many more.
- Google:
- MyMaps π
- Earth π
- Earth Proπ»
- Earth Studio ππ»
- Humanitarian Data Exchange π - Useful resources of shapefiles, especially for administrative boundaries.
- KML Interactive Sampler π - Lots of KML templates.
- QGIS π» - Free & open-source alternative to ArcGis.
- Draw.io ππ» - Open-source diagramming tool. Can be run locally.
- Gephi π» - Powerful visualization and exploration software
- Visual Investigative Scenarios π (OCCRP)
- yEd Graph Editor π»
- Tik Tok π» - Javascript tool to easily create simple, mobile-friendly, vertical timelines. Open-source.
- TimelineJS π»
- timeanddate.com π - Weather history.
- Ventusky π - Live & past wind, rain and temperature maps.
- Wolfram Alpha π - Weather history. What was the weather in New York on January 1st 2017?
- Wunderground History π - Weather history
See also: Archival
- DarkSearch π - Dark web search engine.
- OnionScan π»
- OSINT Tools for the Dark Web (Jake Creps) π - Presentation of several tools to help investigate the dark web.
- Photon π» - Crawl a website (or its archive from the WayBack machine) and extract URLs, emails, social media accounts, files, keys, subdomains, etc.
- Python scraping libraries:
- BeautifulSoup π
- cloudflare-scrape π
- Selenium π
- Scrapy π
- Scrape Interactive Geospatial Data π (Bellingcat)
- Advanced Google searches
- Google Search Operators π (moz.com)
- Mastering Google Search Operators in 67 steps π (moz.com)
- Google Hacking Database π (Exploit.db)
- Google Search Operators: The Complete List π (ahrefs.com)
- CarbonDate π» - Estimate the age of web resources. Has an non-HTTPS online version
- crt.sh π - Certificates search.
- Domain_OSINT π - Ph055a's list of tools to investigate domains & IoT devices.
- DNSDumpster π - Domain research tool that can discover hosts related to a domain.
- FinalRecon π» - All-in-one tool : whois, headers, SSL certificates details, image & links crawling.
- NerdyData Search π - Source code search engine.
- OpenLinkProfiler - Search & analyze the links of a website. Good replacement for Google's defunct
link:
operator. - PublicWWW π - Search the source code of pages.
- pymeta π» - Find document files on a domain, download them and extract metadata.
- SpyOnWeb π - Search by URL, IP address, analytics codes. API with free plan. See this Belligcat how-to for automation.
- Sublist3r π» - Subdomains enumeration tool.
- Unveiling hidden site connections with Google Analytics IDs π (Bellingcat)
- awesome-selfhosted π - A list of Free Software network services and web applications which can be hosted locally
- grayhatwarfare π - Search open Amazon S3 buckets content.
- Shodan π - Internet of Things search engine
- World License Plates π - Pictures of license plates from all around the world.
This list is under the Creative Commons Attribution-NonCommercial 4.0 International Public License License.