Description

Project 01: Exploratory Data Analysis
Partner work is allowed on this project. Browse through the UCI Machine Learning Repository to find a data set that is interesting to you, and has both categorical and numerical data (but is small enough to work with). Note: If your chosen data set is not small enough to download, take a sample of the data set that contains at least 200 entities (instances/rows).
Problem 1: Write the Introduction
In a well-written paragraph, answer the following questions about the data:
• (4 points) What was the data used for?
• (2 points) Who (or what organization) uploaded the data?
• (5 points) How many attributes and how many entities are represented in the data?
– How many numerical attributes?
– How many categorical attributes?
– Would you suggest that each categorical attribute be label-encoded or one-hot-encoded? – Why did you choose the encoding?
• (7 points) Why is this data set interesting to you?
• (6 points) Of the attributes used to describe this data, which do you think are the most descriptive of the data and why (before doing any data analysis)?
Part 2: Write Python code for data analysis
• (5 points) A function that will compute the multi-dimensional mean of a numerical, multidimensional data set input as a 2-dimensional numpy array
• (5 points) A function that will compute the estimated covariance between two attributes that are input as one-dimensional numpy vectors
• (5 points) A function that will compute the correlation between two attributes that are input as two numpy vectors.
• (5 points) A function that will normalize the attributes in a two-dimensional numpy array using range normalization.
• (5 points) A function that will normalize the attributes in a two-dimensional numpy array using standard normalization.
• (5 points) A function that will compute the covariance matrix of a data set.
• (5 points) A function that will label-encode a two-dimensional categorical data array that is passed in as input.
Part 3: Analyze the data with your code and write up the results
Use your code from Part 2 to answer the following questions in a well-written paragraph, and create the following plots from the numerical portion of the data. Use your functions to compute the multi-dimensional mean and covariance matrix of the numerical portion of your data set. Before answering the questions:
• (5 points) Convert all categorical attributes using label encoding or one-hot-encoding
• (2 points) If your data has missing values, fill in those values with the attribute mean.
Questions to answer:
• (2 points) What is the multi-dimensional mean of the numerical data matrix (where categorical data have been converted to numerical values)?
• (4 points) What is the covariance matrix of the numerical data matrix (where categorical data have been converted to numerical values)?
• (5 points) Choose 5 pairs of attributes that you think could be related. Create scatter plots of all 5 pairs and include these in your report, along with a description and analysis that summarizes why these pairs of attributes might be related, and how the scatter plots do or do not support this intuition.
• (3 points) Which range-normalized numerical attributes have the greatest estimated covariance? What is their estimated covariance? Create a scatter plot of these range-normalized attributes.
• (3 points) Which Z-score-normalized numerical attributes have the greatest correlation? What is their correlation? Create a scatter plot of these Z-score-normalized attributes.
• (3 points) Which Z-score-normalized numerical attributes have the smallest correlation? What is their correlation? Create a scatter plot of these Z-score-normalized attributes.
• (3 points) How many pairs of features have correlation greater than or equal to 0.5?
• (3 points) How many pairs of features have negative estimated covariance?
• (2 points) What is the total variance of the data?
• (2 points) What is the total variance of the data, restricted to the five features that have the greatest estimated variance?
Tips and Acknowledgements
Make sure to submit your answer as a PDF on Gradscope and Brightspace. Make sure to show your work. Include any code snippets you used to generate an answer, using comments in the code to clearly indicate which problem corresponds to which code.
Acknowledgements: Project adapted from assignments of Veronika Strnadova-Neeley.