40 · DataFrame: 2D labeled tables
A DataFrame is a dictionary of Series that share an index — the universal container for tabular ML data. Master creation, inspection, filtering, and the SettingWithCopyWarning footgun that trips every new Pandas user.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
40 · DataFrame: 2D labeled tables
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.A **DataFrame** is a collection of Series that share the same row index — like a spreadsheet or an SQL table, but living in memory and directly interoperable with NumPy. The row index and column labels together mean every cell has a two-coordinate address: `df.loc[row_label, col_name]`. Understanding that a DataFrame is fundamentally a **dict of aligned Series** demystifies every DataFrame operation: selecting a column returns a Series; a boolean filter creates a mask Series; adding a column is just assigning a new key to the dict.
Without this:
Without DataFrames (or equivalent tooling), every preprocessing step — filtering rows, computing derived columns, selecting feature subsets — would require writing explicit loops over raw NumPy arrays and manually tracking column names via separate lists. The DataFrame gives you named columns, row-label alignment, and SQL-style operations (select, filter, join, group) in a single object. It is the primary reason Python displaced R as the default ML language.
A DataFrame is the 2D workhorse of pandas. Internally it is a dict of Series objects all sharing the same row index. You can think of it as a spreadsheet in RAM.
The most common ways to create a DataFrame:
# From a dict of lists — each key becomes a column name
pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
# From a list of dicts — each dict is one row
pd.DataFrame([{'name': 'alice', 'age': 30}, {'name': 'bob', 'age': 25}])
Key inspection methods to run on any new dataset:
.head(n)/.tail(n)— first/last n rows (default 5).shape—(n_rows, n_cols)tuple.info()— column names, dtypes, non-null counts, memory usage.describe()— summary statistics for numeric columns.dtypes— Series of dtype per column.columns— Index of column names
Column access:
df['col']— returns a Series (always safe)df[['col']]— returns a single-column DataFrame (double brackets)df.col— attribute-style access — avoid it for new code: it fails when column names contain spaces or clash with method names likedf.count.
Python (in browser)
`.describe()` is the first command to run on any new numeric dataset. It shows count, mean, std, min, quartiles, and max in one call — enough to catch outliers (max very far from mean), scale mismatches (age in years vs purchase in millions), and constant columns (std ≈ 0) before spending time on model training.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`df['age'] > 30` produces a boolean Series (the mask). Passing that mask back to `df[mask]` returns only the rows where the mask is `True`. Chaining `[['name', 'purchases']]` (double brackets) selects two columns as a DataFrame. This pattern — filter rows, then select columns — is the foundation of every data subset operation in a preprocessing pipeline.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Assigning to `df['new_col']` adds (or overwrites) a column in-place. The right-hand side can be a scalar, a list, a NumPy array, or a Series — pandas aligns by index automatically. This is the standard way to engineer new features before model training.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
`.loc[row_mask, col_name] = value` is the single safe idiom for conditional assignment. It resolves both the row selection and column assignment in one indexing operation, so pandas always modifies the original DataFrame. Chained indexing (`df[mask]['col'] = ...`) is the most common Pandas footgun — it silently fails or warns depending on whether the intermediate result is a view or a copy.
What is the difference between `df['col']` and `df[['col']]`?
- A DataFrame is a dict of aligned Series sharing the same row index. Each column is a Series; the shape is `(n_rows, n_cols)`.
- `.head()`, `.describe()`, `.info()`, `.dtypes` are the first four inspection commands on any new dataset.
- `df['col']` returns a Series; `df[['col']]` returns a single-column DataFrame. `df[bool_mask]` filters rows.
- Use `df.loc[row_mask, 'col'] = value` for conditional assignment — never chain `df[mask]['col'] = value` (SettingWithCopyWarning).
Every supervised ML dataset starts life as a DataFrame. `df[features]` (double brackets, list of column names) extracts the 2D `X` matrix; `df[target]` (single bracket) extracts the 1D `y` vector. `df.describe()` is the mandatory first step of any EDA. Filtering with boolean masks is how you split train/validation sets before `train_test_split`.
If you remove it: Without DataFrames, maintaining the correspondence between feature values and their column names would require a parallel list of strings — easily broken by sorting, filtering, or adding columns. Passing unlabelled arrays to scikit-learn is legal but removes all interpretability: `coef_[3]` means nothing; `coef_[df.columns[3]]` means 'the coefficient for age'. DataFrames keep metadata alive throughout the pipeline.