Data Preparation & Visualization

End-to-end preprocessing of the Telco Customer Churn dataset — from raw CSV inspection through cleaning, exploratory visualization, statistical testing, feature engineering, encoding, scaling, and train/test split.

---

Learning Objectives

---

Task-by-Task Reference

---

Task 0 — Describe Data (0-describe_data.py)

Challenge: Get a quick structural overview of a new dataset without visual inspection.

Approach: Load CSV with pd.read_csv(), then chain .shape, .dtypes, .head(), .isna().sum(), and .duplicated().sum() to inspect structure, types, preview, missing counts, and duplicates.

New techniques introduced:

Key takeaway: The first step of any data project is a high-level structural scan — shape, types, missingness, duplicates — before any cleaning or modeling.

---

Task 1 — Plot Missingness (1-plot_missingness.py)

Challenge: Visually identify which columns and rows contain missing values, at a glance.

Approach: Use np.where(df.isnull().values) to get (row, col) coordinates of missing values, then plt.scatter() them on a grid with y-tick labels as column names.

New techniques introduced:

Key takeaway: A missingness scatter plot reveals the shape of incompleteness — vertical streaks indicate whole-column issues, horizontal streaks indicate problematic rows.

---

Task 2 — Convert Columns (2-convert_columns.py)

Challenge: Fix improperly typed columns — a numeric column stored as string, and a 0/1 column that should be categorical 'Yes'/'No'.

Approach: pd.to_numeric(…, errors='coerce') converts strings to floats (invalid → NaN). .replace({0: 'No', 1: 'Yes'}) remaps numeric codes to readable labels.

New techniques introduced:

Key takeaway: Data read from CSV often arrives with wrong dtypes; to_numeric and replace are the primary tools for type correction.

---

Task 3 — Clean Total Charges (3-clean_total_charges.py)

Challenge: Handle missing values in a critical column with three different strategies, selectable by parameter.

Approach: dropna(subset=[…]) removes rows where TotalCharges is NaN. fillna(median()) replaces NaN with the column median. Formula imputation fills NaN as MonthlyCharges * tenure.

New techniques introduced:

Key takeaway: Missing value handling is not one-size-fits-all; different imputation strategies (drop, median, formula) have different trade-offs and should be parameterized.

---

Task 4 — Remove Duplicates (4-remove_duplicates.py)

Challenge: Eliminate redundant identical rows from a DataFrame.

Approach: df.drop_duplicates() removes all rows that are exact duplicates of preceding rows.

New techniques introduced:

Key takeaway: Duplicate rows inflate counts and bias statistical tests. One line of pandas fixes it.

---

Task 5 — Drop customerID (5-drop_customerID.py)

Challenge: Remove a column that uniquely identifies each row but has no predictive value.

Approach: df.drop("customerID", axis=1) removes the column by name.

New techniques introduced:

Key takeaway: Unique identifiers (customerID, row index) must be dropped before modeling — they carry zero signal and can cause data leakage.

---

Task 6 — Plot Churn Distribution (6-plot_churn_distribution.py)

Challenge: Check whether the target variable is imbalanced (far more No than Yes).

Approach: value_counts().reindex(['No', 'Yes']) ensures consistent bar order, then plt.bar() with per-bar colors.

New techniques introduced:

Key takeaway: Always check target class balance first — severe imbalance determines whether you need stratified sampling, class weights, or resampling.

---

Task 7 — Plot Categorical Distributions (7-plot_categorical_distributions.py)

Challenge: Visualize the frequency of every categorical feature in a single figure.

Approach: Use plt.subplots(n_rows, n_cols) for a grid layout, enumerate over axes and columns, plot each with ax.bar(), rotate labels 45°, and hide unused subplots with .axis('off').

New techniques introduced:

Key takeaway: A subplot grid is the standard pattern for batch-visualizing multiple features; flatten() + zip makes the loop trivial.

---

Task 8 — Plot Continuous Distributions (8-plot_continuous_distributions.py)

Challenge: Show both the shape (histogram + KDE) and outliers (boxplot) of each numeric column in one figure.

Approach: plt.subplots(n, 2) — left column = ax.hist(density=True) + stats.gaussian_kde() overlay as a dashed red line; right column = ax.boxplot().

New techniques introduced:

Key takeaway: Histogram + KDE shows distribution shape; boxplot exposes outliers, skew, and quartiles. Together they give a complete picture of a continuous variable.

---

Task 9 — Plot Correlation Heatmap (9-plot_correlation_heatmap.py)

Challenge: Identify which numeric features move together, using a single visual.

Approach: df.select_dtypes(include='number').corr() computes the pairwise Pearson correlation matrix. sns.heatmap() with annot=True, cmap='coolwarm', and vmin=-1, vmax=1 creates an annotated diverging heatmap.

New techniques introduced:

Key takeaway: Correlation heatmaps quickly reveal multicollinearity — features that carry redundant information — critical for linear models.

---

Task 10 — Plot Categorical vs Churn (10-plot_categorical_vs_churn.py)

Challenge: Quantify how churn rate varies across categories of a categorical feature.

Approach: df.groupby(col)['Churn'].apply(lambda x: (x == 'Yes').mean()) computes churn rate per category as the proportion of 'Yes', then plt.bar() plots the result.

New techniques introduced:

Key takeaway: Groupby + apply is the pandas idiom for per-group statistics; (x == 'Yes').mean() directly gives the proportion without explicit counting.

---

Task 11 — Plot Numeric vs Churn (11-plot_numeric_vs_churn.py)

Challenge: Compare the distribution of a continuous feature between churners and non-churners in a single plot — without overlapping.

Approach: Bin the numeric column with pd.cut(bins=30), create a cross-tabulation with groupby().value_counts().unstack(), then ct.plot.bar() for side-by-side grouped bars per bin.

New techniques introduced:

Key takeaway: pd.cut + unstack + plot.bar turns a continuous-vs-categorical comparison into an intuitive side-by-side grouped bar chart — no manual bin shifting needed.

---

Task 12 — Chi-Square Tests (12-chi_square_tests.py)

Challenge: Quantify, with p-values, whether each categorical feature is statistically associated with Churn.

Approach: For each categorical column, build a contingency table with pd.crosstab(col, Churn), then feed it to stats.chi2_contingency() and extract the p-value.

New techniques introduced:

Key takeaway: crosstab + chi2_contingency is the direct Python equivalent of Stata's tabulate x y, chi2 — test independence between two categorical variables.

---

Task 13 — Welch's t-test (13-ttest_numeric.py)

Challenge: Test whether the mean of each numeric feature differs significantly between churners and non-churners.

Approach: For each numeric column, split into Yes/No groups, then stats.ttest_ind(yes, no, equal_var=False) computes Welch's t-test p-value.

New techniques introduced:

Key takeaway: Welch's t-test is preferred over Student's t-test when group variances are likely unequal; equal_var=False switches to Welch-Satterthwaite degrees of freedom.

---

Task 14 — Create Features (14-create_features.py)

Challenge: Engineer two new features — a count-based composite (NumServices) and a binned categorical (TenureGroup) — then drop the source columns used to create them.

Approach: (df[svc_cols] == 'Yes').sum(axis=1) counts subscribed services row-wise, plus 1 if InternetService is DSL or Fiber optic. pd.cut(tenure, bins, labels, right=True, include_lowest=False) creates tenure interval groups.

New techniques introduced:

Key takeaway: Feature engineering transforms raw columns into more predictive representations — counting services reduces dimensionality; binning tenure captures non-linear relationships.

---

Task 15 — Encode Features (15-encode_features.py)

Challenge: Convert all categorical columns to numeric representations suitable for ML models.

Approach: LabelEncoder for target Churn (single column, 1D). OrdinalEncoder(categories=[['No', 'Yes']]) for binary Yes/No columns, each encoded individually to control the repr. pd.get_dummies(drop_first=True, dtype=int) for multi-category columns (Contract, PaymentMethod) with k−1 columns. Separate OrdinalEncoder for TenureGroup (alphabetical order).

New techniques introduced:

Key takeaway: Three encoding strategies map to three column types: LabelEncoder for target (1D), OrdinalEncoder for ordered/binary features, One-hot (get_dummies) for nominal multi-category features.

---

Task 16 — Scale Numeric (16-scale_numeric.py)

Challenge: Standardize numeric features to mean=0, std=1 so they're on a common scale for distance-based models.

Approach: StandardScaler().fit_transform(df[['MonthlyCharges', 'TotalCharges']]) computes z-scores and replaces the columns in-place.

New techniques introduced:

Key takeaway: Standardization (z-score normalization) is essential for models sensitive to feature magnitude — linear/logistic regression, SVM, KNN, neural networks. Without it, large-magnitude features dominate.

---

Task 17 — Split Data (17-split_data.py)

Challenge: Split data into train/test sets while preserving the class distribution of the target variable.

Approach: df.drop(columns=['Churn']) separates X (features) from y (target). model_selection.train_test_split(X, y, test_size=0.2, stratify=y, random_state=42) splits with stratification.

New techniques introduced:

Key takeaway: stratify=y ensures the train and test sets have the same target distribution — critical for imbalanced datasets where random splitting could skew evaluation metrics.

---

Technique Inventory

---

Resources

• pandas Documentation
• matplotlib Documentation
• seaborn Documentation
• SciPy stats Documentation
• scikit-learn preprocessing Documentation
• scikit-learn model selection Documentation
• Telco Customer Churn Dataset