A List of Some Web Data Sources

Well I needed to pull together a listing of publicly available data sources for a project, so I figured I’d post them here as well. Some descriptions and tag lines have been taken directly from the website, and some I quickly created on my own. This list is by no means comprehensive (I probably have about 100 links in the “Data” folder of my bookmarks…) but it’s a quick snapshot at some useful data sources on the web. That being said, there are a lot of considerations when targeting a data set and tomorrow’s need for data will most likely differ from today’s need for data. Build and execute a target data strategy using the vast sets of search engines, libraries, and social networks on the web and you’ll be just fine.

AggData – The advantage of AggData is that the data is collected into one file that is very raw and portable, which makes it easy to integrate into any application or website. You can browse free data sets or purchase any of the many data sets from public and private organizations for a relatively small fee.

The Association of Religion Data Archives – The ARDA Data Archive is a collection of surveys, polls, and other data submitted by researchers and made available online by the ARDA. There are nearly 500 data files included in the ARDA collection. You can browse files by category, alphabetically, view the newest additions, most popular files, or search for a file. Once you select a file you can preview the results, read about how the data were collected, review the survey questions asked, save selected survey questions to your own file, and/or download the data file.

Census.gov American FactFinder – In American FactFinder you can obtain data in the form of maps, tables, and reports from a variety of Census Bureau sources. Click here for a good listing of available data sets, visualizations, and search functionalities.

CIA World Factbook – Contains a lot of country-level metrics/statistics, although they are not very easily exportable and/or available in table format.

City Population – Gazetteer of global geographic data and limited demographic statistics per location.

Data360 – This is essentially a wiki for data. Data360 is an open-source, collaborative and free website.  The site hosts a common and shared database, which any person or organization, committed to neutrality and non-partisanship (meaning “let the data speak”), can use for presentation of reports and visualizations about the data.

Data.gov – The purpose of Data.gov is to increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government. Although the initial launch of Data.gov provides a limited portion of the rich variety of Federal datasets presently available, we invite you to actively participate in shaping the future of Data.gov by suggesting additional datasets and site enhancements to provide seamless access and use of your Federal data. Visit today with us, but come back often. With your help, Data.gov will continue to grow and change in the weeks, months, and years ahead. For more information, view our How to Use Data.gov guide.

Data Marketplace – Buy and/or sell data. You can request data sets for others to build and provide for a small fee.

DBpedia – DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data.

EconoMagic – A directory of data sets specific to US states.

Factual – Factual is a platform where anyone can share and mash open data on any subject. Factual was founded to provide open access to better structured data.

FedStats – Provides access to all federal statistical agencies (by geographic scope or listed alphabetically) with a search function to discover available data sets across all US federal statistical agencies.

GapminderA non-profit venture that, through a interactive viz tool accompanied by a listing of available data tables, aims to “unveal the beauty of statistics for a fact based world view”.

GeoCommons Finder! – Upload, organize and share your Geographic Data. Then you can use their built in application called Maker! to map/visualize it.

GeoNames – The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge.

Global Airport Database – Comprehensive set of global airport data (download available for free).

Global Health Facts – Search global data by health topic and/or country. You can also interactively compare data for up to five countries at a time.

Google Public Data – In addition to plainly using the main Google search engine to search for a specific data set, Google has a public data library with some valuable sets available for free.

Guardian.co.uk Data Store – Governments around the globe are opening up their data vaults – allowing you to check out the numbers for yourself. This is the Guardian’s gateway to that information. Search for government data here from the UK (including London), USA, Australia and New Zealand – and look out for new countries and places as we add them. Read more about this on the Datablog. Full list of government data sites here.

Harvard Geographic Information Systems – Contains a highly credible listing of various national and international data providers and data sources, with a strong focus on geographic data.

International Civil Aviation Organization (ICAO) – Global air traffic data available for a fee.

Infochimps – Request data sets, search for existing data sets, or post and sell your own data sets.

International Statistical Agencies
US Census Bureau: http://www.census.gov/aboutus/stat_int.html
US Bureau of Labor Statistics: http://www.bls.gov/bls/other.htm
United Nations: http://unstats.un.org/unsd/methods/inter-natlinks/sd_intstat.htm

MelissaData – Buy comprehensive zip code data for about $150. Tailored for businesses with use in marketing.

NationMaster – NationMaster is a massive central data source and a handy way to graphically compare nations. NationMaster is a vast compilation of data from such sources as the CIA World Factbook, UN, and OECD. Using the form above, you can generate maps and graphs on all kinds of statistics with ease.

National Association of Counties (NACO) – Includes a US county data library.

Numbrary – Numbrary is a free online service dedicated to finding, using and sharing numbers on the web.

OECD Stat Extracts – OECD.Stat includes data and metadata for OECD (Organization for Economic Cooperation and Development) countries and selected non-member economies.

QuickFacts (US Census Bureau Site) – Quick, easy access to facts about people, business, and geography.

StateMaster – StateMaster is a unique statistical database which allows you to research and compare a multitude of different data on US states. We have compiled information from various primary sources such as the US Census Bureau, the FBI, and the National Center for Educational Statistics. More than just a mere collection of various data, StateMaster goes beyond the numbers to provide you with visualization technology like pie charts, maps, graphs and scatterplots. We also have thousands of map and flag images, state profiles, and correlations.

United Nations Development Programme (UNDP) – Includes UN Human Development reports and statistics such as the Human Development Index.

USA Counties (US Census Bureau Site) – A directory of data tables for US states and individual counties. Includes over 6,500 data items.

Weather Underground – Provides free access to historical weather data for cities around the globe.

Wolfram|Alpha – Deemed a “computational knowledge engine”, the W|A search and discovery tool is mathematically-based and tries to turn queries (term-based or data-driven) into actionable knowledge with visualization of in-house data sets and information relevant to your query.

World Gazetteer – The World Gazetteer provides a comprehensive set of population data and related statistics.

World Port Source – Contains extensive data on global sea ports, characterized by size and searchable by shipping liners and other various data fields.

Future Search Requires Human Intervention

The business of organizing information is a tricky one in that it’s a moving target. It’s hard to tell what will be the next hot topic or consumer need, and the contribution of digital information is not necessarily bound by the structure and rules of those needing that information. Second, the mechanisms by which we wish to access and use information are constantly changing. The world in 2050 won’t be PowerPoint presentations, heat maps and csv exports, just as 1950’s world wasn’t all emails, tweets, and search engines. Lastly, with the emergence and growth of social media and networking, information contribution has dramatically increased. Without providing the statistics, it’s well known that more people are online, and more people are hitting a “Submit”, “Post”, “Send”, or “Tweet” button than ever before. As a result, there are more pieces of information, more individual topics, more ways to categorize information, more uses for information, more ways to see information, and more unknowns about that information than ever before.

That being said, it’s important to not be unnerved by this. Collectively, we are smart enough to guide search technology in the right direction, or at least put it in the right place so that its path is optimized for future generations. And collectively, we are smart enough to ensure that the ways we use information will publicly provide guidance to the development of a optimal suite of available tools for effective visualization and communication. That being said, by focusing on the relative near-term, emerging trends can be detected, understood, and leveraged for advancement in the business of organizing information. Gaining an edge on future trends brings tremendous value and exactly that – edge. Among other methods, the detection of such trends requires quantitative analysis mixed with psychology, history, and good ol’ intuition.

In terms of finding information (and more importantly, relevant information), most people think simple search. Or maybe more specifically, they think Boolean search with some available advanced search options. It’s Google, Bing, Wolfram|Alpha, or some other engine – typing in some logical query, occasionally setting some supporting filters, scrolling through the first ten results, and settling on what looks best by some meta data. This certainly works today, and these search engines may be sufficiently wrapping their arms around the influx of new and changing information. But soon technology reaches a limit. The responsibility must shift back to the human. As the informational requirements become more complex and the underlying data become more specialized, relevancy must become more human-driven. We can’t rely completely on technology to provide us with answers.

This notion is two-fold:

  1. Future queries needs more human input. This means “advanced search” needs to become part of “normal search”. Spending 3 seconds to check a box could save 3 minutes in scanning results. You can obviously set defaults for your regular search, but I’d be interested to see stats on the use of type of search vs time spent scanning results. The distance between the two will only grow with the ever-accelerating growth of internet content.
  2. Future content needs more human organization. About.com, Wikipedia.org, and howstuffworks.com are great examples of this. They are sites that show up in search results that organize information. Wired Magazine had a good article last month on About.com as an “answer factory” (in fact, their motto is “Guidance. Not Guesswork” – I love that). These sites organize thought and mimic the human mind looking for an answer to a question, not a query. Websites and blogs need to organize information about information, and as a result we can increase transferable knowledge and decrease search time.

It’s obvious but important to note that people collectively drive relevancy through thought. Trend topics in Twitter, News Feed on Facebook, Google Hot Trends, etc. are driven by fingers on a keyboard, which are driven by synapses in the brain, which are driven by actions and observations, which are driven by circumstance – and maybe even fate. Technology should not drift too far away or we’ll lose access to the very knowledge we create.