A data set is a grouping of data on a particular topic. Many academics gather their own, especially in the sciences. On occasion, however, it is more useful to analyze the data already gathered and provide new insights into a topic.
Data sets which have been gathered and curated by someone else have a couple of advantages: they've been gone over several times and the errors are likely to be reduced, they're likely to have been gathered in a consistent and competent manner, and they're likely to be available electronically.
From Data Sets by Laura Trude, University of North Dakota Libraries
Federal Researve Economic Data (FRED)
American Religion Data Archive (ARDA)
Data and Program Library Service
UCLA Social Science Data Archive Dataverse
Inter-University Consortium for Political and Social Research
National Center for Education Statistics
National Center for Health Statistics
National Longitudinal Study of Adolescent Health (Add Health)
Sociometrics Corporation Data Archives
WHOSIS: World Health Organization
Pikes Peak State College Libraries
Information ∞ Inquiry ∞ Learning