วันนี้จะพาไปดูแหล่งศูนย์รวม Open Data สำหรับลองทำ Data Science Project กันครับ
- Publically Traded Market Data: Quandl is an amazing source of finance data. Google Finance and Yahoo Finance are additional good sources of data. Corporate filings with the SEC are available on Edgar.
- Housing Price Data: You can use the Trulia API or the Zillow API.
- Lending data: You can find student loan defaults by university and the complete collection of peer-to-peer loans from Lending Cluband Prosper, the two largest platforms in the space.
- Home mortgage data: There is data made available by the Home Mortgage Disclosure Act and there’s a lot of data from the Federal Housing Finance Agency available here.
- Review Content: You can get reviews of restaurant and physical venues from Foursquare and Yelp (see geodata). Amazon has a large repository of Product Reviews. Beer reviews from Beer Advocate can be found here. Rotten Tomatoes Movie Reviews are available from Kaggle.
- Web Content: Looking for web content? Wikipedia providesdumps of their articles. Common Crawl has a large corpus of the internet available. ArXiv maintains all their data available via Bulk Download from AWS S3. Want to know which URLs are malicious? There’s a dataset for that. Music data is available from the Million Songs Database. You can analyze the Q&A patterns on sites likeStack Exchange (including Stack Overflow).
- Media Data: There’s open annotated articles form the New York Times, Reuters Dataset, and GDELT project (a consolidation of many different news sources). Google Books has published NGrams for books going back to past 1800.
- Communications Data: There’s access to public messages of theApache Software Foundation and communications amongst former execs Enron
- Municipal Data: Crime Data is available for City of Chicago, andWashington DC. Restaurant Inspection Data is available forChicago and New York City.
- Transportation Data: NYC Taxi Trips in 2013 are available courtesy of the Freedom of Information Act. There’s bikesharing data from NYC, Washington DC, and SF. There’s also Flight Delay Data from the FAA
- Census Data: Japanese Census Data. US Census data from 2010,2000, 1990. From census data, the government has also derivedtime use data. EU Census Data. Checkout popular male / female baby names going back to the 19th Century from the Social Security Administration.
- World Bank: they have a lot of data available on their website.
- Election Data: Political contribution data for the last few US elections can be downloaded from the FEC here and here. Polling data is available from Real Clear Politics.
Data With a Cause:
- Environmental Data: Data on household energy usage is availableas well as NASA Climate Data.
- Medical and biological Data: You can get anything fromanonymous medical records, to remote sensor reading for individuals, to data of the Genomes of 1000 individuals.
- Geo Data: Try looking at these Yelp Datasets for venues near major universities and one for major cities in the Southwest. TheFoursquare API is another good source. Open Street Map has open data on venues as well.
- Twitter Data: you can get access to Twitter Data used for sentiment analysis, network Twitter Data, social Twitter data, on top of their API.
- Games Data: Datasets for games, including a large dataset ofPoker hands, dataset of online Domion Games, and datasets ofChess Games are available.
- Web Usage Data: Web usage data is a common dataset that companies look at to understand engagement. Available datasets include Anonymous usage data for MSNBC, Amazon purchase history (also anonymized), and Wikipedia traffic.
Metasources: these are great sources for other web pages.
- Stanford Network Data: http://snap.stanford.edu/index.html
- Every year, the ACM holds a competition for machine learning called the KDD Cup. Their data is available online.
- UCI maintains archives of data for machine learning.
- US Census Data
- Amazon is hosting Public Datasets on s3
- Kaggle hosts machine-learning challenges and many of their datasets are publicly available
- The cities of Chicago, New York, Washington DC, and SF maintain public data warehouses.
- Yahoo maintains a lot of data on its web properties which can be obtained by writing them.
- BigML is a blog that maintains a list of public datasets for the machine learning community.
- Finally, if there’s a website with data you are interested in, crawl for it!
Data Science Technology Evangelist at datascience.in.th
I'm a data science lover. I believe that data science could bring a huge benefit to your organization. I'm a hard problem solver. I'm crazy and expert in statistical modelling. I always talk with R and Python. I'm hungry in data. Check out my full CV at http://chakkrit.com
Latest posts by Chakkrit Tantithamthavorn (see all)
- Open Data สามารถปราบคอร์รัปชันได้อย่างไร โดย ดร. วรากรณ์ สามโกเศศ - February 26, 2015
- แนะนำหนังสือ An Introduction to Statistical Learning with Applications in R - December 23, 2014
- Statistical Analysis และ Data Mining เป็นทักษะที่ร้อนแรงและเป็นที่ต้องการมากที่สุดในปี 2014 - December 20, 2014