Chapter 1: Data Structures and Methods of Series
Chapter Introduction
pandas works with indexed data. A Series is one column of values with an index and dtype; a DataFrame is a table of Series. They support reading, cleaning, features and summaries. Chapter 1 develops columns; Chapter 2 develops tables. These structures supply the inputs for regression, Bayesian and time-series models in later chapters.
A Series adds statistical methods, vectorised calculation and observation index values to a Python list. Arithmetic between Series aligns their index values; unmatched index values can produce NaN. Direct comparisons such as left > right require identical indexes, so align them first (§1.3). Vectorisation speeds calculation; valid index values establish which observations belong together.
Three rules organise the chapter: index identifies observations and controls alignment; missingness affects calculations and requires a stated policy; return type distinguishes a new Series from a scalar summary. Select index values with .loc and positions with .iloc. To change the original, use one explicit .loc[...] = ... assignment.
The teaching files contain 20 000 inventory products, 3 084 US public companies, AAPL closes (2015–2024), MSFT closes (2015–2018), S&P 500 closes, AEP hourly load for 2017, Delhi weather, one day of AAPL minute bars and two years of NVDA closes. Each example states its observation, index, measurement and task. Slides and Book use the same data and outputs.
By the end, you can construct and inspect Series, align observations, interpret summaries and missingness, transform values, select rows and build dated features. §1.5 combines these skills with chart selection and Figure/Axes. Chapter 2 extends the workflow to tables.
Chapter 2 combines these Series operations into validated tables and feature pipelines.
Table of Contents
- 1.1 Series Foundations
- 1.2 Value Transformations and Cleaning
- 1.3 Selection and Index Alignment
- 1.4 Time-series Features and Resampling
- 1.5 Figures and Statistical Charts
Use the numbered parts within each section to locate concepts and worked examples. Read a part in sequence before attempting its application.
Series Foundations
| Part | Question |
|---|---|
| 1.1.1 pandas structures and indexed observations | What does one observation represent? |
| 1.1.2 Arithmetic, broadcasting and alignment | Which values will pandas combine? |
| 1.1.3 Summaries, masks and type conversion | What does the summary count? |
Pandas structures and indexed observations
Question: What does one observation represent?
Result: A Series with explicit index values, units and dtype.
pandas, DataFrame and Series
df names the three-row teaching table. Its index contains index values and its columns contain the two variable names. Its shape is (3, 2): the index is not counted as a data column. Selecting one named column preserves the index and returns a Series of shape (3,).
Header and preview are different. A displayed header shows column names. head(n) is a method that returns the first n observations, retaining their index values and all columns. head() defaults to up to five rows; it does not modify the source.
Read shape: rows, columns and dimensions.
Each row is one shop. Select units; follow its index, name and shape.
Read revealed values as a table
Load the library with import pandas as pd. The alias pd is the conventional short name used throughout this course. A function such as pd.read_csv(...) reads a file; a constructor such as pd.Series(...) creates a data object. You do not need to write a file reader yourself to begin analysing a supplied dataset.
| Structure | Dimensions and index values | Example |
|---|---|---|
| Series | One axis: index values; one dtype and an optional name | A product-price column |
| DataFrame | Two axes: row index and column names; each column has its own dtype | Product identifiers and prices together |
The diagram uses the first three records of the course’s inventory file. Switch between SKU_number and PriceReg: the selected values, Series name and dtype change, while index values 0, 1 and 2 remain. A column of integer identifiers and a column of decimal prices can coexist in one DataFrame. Here SKU_number is a business identifier stored in a column; pandas has not automatically made it the row index. Always distinguish the index from an identifier column.
Extract one column with an index from a DataFrame
Read, select and inspect a small table.
pd.read_csv reads a CSV file; usecols chooses its columns. A column name in square brackets selects one Series.
DataFrame: three rows, two columns
Series: one column, same index
Read a DataFrame, then extract a Series. Column names are unique in this example. type(obj).__name__ reports the structure’s name; .shape reports its dimensions.
The table has shape (3, 2): three rows and two columns. The Series has shape (3,): three entries along one axis. Its observed values are 44.99, 24.81 and 46.00, attached to index values 0, 1 and 2. The comma in (3,) is Python’s notation for a one-element tuple; it does not denote an unknown second dimension. Statistical methods follow the structure explanation.
Using a list of column names keeps a DataFrame: intro_table[["PriceReg"]] has shape (3, 1). The outer brackets perform selection; the inner brackets create a list containing one column name. The numerical values are unchanged, but the returned structure differs. This matters when choosing a method or supplying an object to another function.
Compare the structures side by side. .name identifies the Series, .shape gives its dimensions, and .index.equals(...) checks whether two indexes contain the same values in the same order.
Series: name and shape
DataFrame: one column
The Series has name PriceReg, dtype float64 and shape (3,). The single-column DataFrame has shape (3, 1), and its index comparison with the source returns True. Both retain index values 0, 1 and 2. In the later price examples, the same selection pattern takes Close from a file-loaded DataFrame. You can also construct a Series directly from a list or dictionary, as below. The pandas introductory tutorial illustrates the same structures.
The first returns a Series with one axis. The second selects a list of columns and returns a DataFrame with one column. Both retain the source index values.
A list cannot take its own mean
Attributes describe; methods calculate.
pd.Series(values, index=months, name="units") constructs a column with an index; omitted index values default to 0, 1, 2, … . The list [300, 100, 0, 200, 1000] supports positional indexing but has no .mean() method or month index values. Wrapping it in a Series supplies both statistical methods and an explicit index.
A Series has values, an index and one dtype. Without index values, this example uses RangeIndex(start=0, stop=5, step=1); explicit index values can be months, tickers or dates. Integer inputs infer int64, decimal inputs usually infer float64, and mixed numeric/text inputs can infer object. Methods such as mean, std, rank and rolling operate on the column.
Construct a Series from a list, dictionary or NumPy array. A list accepts optional index= and name=; dictionary entry names become index values; an array supplies its dtype. Thus np.arange(5) * 10.0 gives float64. The example compares .loc["Jan"] with .iloc[0].
The list raises AttributeError; the Series mean is 320.0. Dictionary entry names give Index(['Jan', 'Feb', 'March'], dtype='object'). The array example is float64, the integer-list example int64. Both sales.loc["Jan"] and sales.iloc[0] return 300; sales.loc["May"] and sales.iloc[-1] return 1 000. .loc names an observation; .iloc names its position.
sales.index, sales.dtype, sales.shape, sales.name and sales.values are attributes — written without parentheses, they describe the object. sales.mean(), sales.sort_values(), sales.plot() are methods — written with parentheses, they compute something. The rule holds throughout pandas: intrinsic properties are attributes, work is a method. Confusing the two is the source of TypeError: 'RangeIndex' object is not callable (you wrote sales.index()) and of <bound method Series.mean of ...> printed instead of a number (you forgot the parentheses).
Select by index with loc and by position with iloc
For Series selection, always use .loc[...] for index values or .iloc[...] for positions. DataFrame column selection still uses df["column"]. An index value identifies an entry; a position describes its current order, counting from zero. Use sales.loc["Jan"] to select January by its index value, and sales.iloc[0] to select the entry currently displayed first. Both return 300 in the month-indexed example below. A negative position counts from the end: sales.iloc[-1] selects the last entry, May, whose value is 1000.
| Selector | Input | Meaning |
|---|---|---|
.loc[...] |
An index value, a list of index values or an index value slice | Find entries with these index values |
.iloc[...] |
An integer position, a list of positions or a positional slice | Find entries in these positions |
sales.loc[0] searches for index value 0 and raises KeyError here because the index values are month names. sales.iloc["Jan"] cannot use a month name as an integer position. An integer index value is still an index value: if the index were [10, 20, 30], .loc[10] would find the entry with index 10, while .iloc[0] would find the first entry. The two address schemes must remain distinct even when index values happen to be 0, 1 and 2.
The outputs are 300, 300, 1000 and 1000. A single index value or position returns one value; a list such as sales.loc[["Jan", "May"]] returns a Series. Use square brackets with these selectors, not function-call parentheses. After sorting by value, the first position may refer to a different month, whereas the index value Jan still identifies January’s observation.
For slices in this ordered example, .loc["Jan":"March"] includes March; .iloc[0:3] includes positions 0, 1 and 2 and excludes position 3. Both select three entries. Section 1.3 develops these endpoint rules with dates, reordering and a diagram. Learn the selection rule before trying the matching question in the slides.
Build a date index one step at a time
index_col="Date" makes Date the index; parse_dates=True reads it as dates. sort_index() puts the dates in chronological order.
1. Read and preview the data
2. Use dates as the index
Date moves from a column to the index. The next example selects Close as a Series and chooses a date range.
Time-series data and cross-sectional data
Identify the observation unit before choosing an operation. A time series measures one entity on different dates; a cross-section measures different entities on one date. AAPL closes across trading dates are a time series; AAPL and MSFT closes on 5 February 2018 form a cross-section. Repeated entity/date observations form panel data, whose two-part index values appear in Chapter 2.
| Question | Time-series data | Cross-sectional data |
|---|---|---|
| What stays fixed? | Entity: AAPL | Observation date: 5 February 2018 |
| What changes across rows? | Trading date: 1 Feb → 2 Feb → 5 Feb | Entity: AAPL ↔︎ MSFT |
| What should the index identify? | The observation’s date | The observation’s ticker |
| What comparison is meaningful? | Changes or windows within that entity’s history | Differences or ranks among peers on that date |
The same distinction applies outside trading: a shop’s monthly sales are a time series, while sales across shops in the same month form a cross-section. A time series has a temporal order: lagged observations can enter a rolling calculation. A cross-section has entity identities; alphabetically ordering shop names or tickers does not create time. Neither structure guarantees independent observations. Successive days may be dependent, and stocks in one snapshot may share a market or sector exposure.
The first cell uses the existing historical AAPL catalogue. Read the supplied CSV-loading step as infrastructure that returns one column with an index. It sorts that column by date and selects three observed trading sessions.
The stored closes are approximately 41.945, 40.125 and 39.123 USD, on 1, 2 and 5 February. The entity is AAPL on every row and the index is named Date. The weekend contributes no observed trading row; these three rows are not three consecutive calendar days. In §1.4, a shift refers to the preceding observed row unless a calendar has explicitly been reconstructed.
The next example changes the question. First inspect a DataFrame with dates in its index and one Close column per stock, starting at 5 February. Then comparison_prices.loc[comparison_date] selects one row as a Series: its index contains the stock column names. The Series name records the date.
1. Prices from the comparison date
2. One date → one Series
The snapshot contains approximately AAPL 39.123 and MSFT 88.000 USD. The MSFT file ends on this date, so subsequent rows in the preview contain NaN (missing observations). These are the catalogue’s Close values as stored, rather than live quotes or comparable measures of company value. Share units differ across firms: a lower per-share price does not establish a cheaper company or a better investment. For fundamental cross-sections, also distinguish the common decision date from each value’s measurement period and actual availability date; Chapter 2’s as-of rules address that admission problem.
No. The previous row is AAPL on the same date. Dividing the adjacent values would compare MSFT’s stored price with AAPL’s stored price, not calculate MSFT’s daily return. Calculate each stock’s change within its own dated history, then select a common return date for a peer comparison. The return method is introduced in Section 1.2. State entity, date, variable and units before choosing the operation.
Trace the observation unit
Arithmetic, broadcasting and alignment
Question: Which values will pandas combine?
Result: A calculation aligned by index with checked matches and missing partners.
A real column: 20 000 products
The retail file records PriceReg (list price), SoldFlag (sale in the past six months), ReleaseNumber (re-releases) and ReleaseYear. Its 198 917 rows cover historical and active products. These examples use the first 20 000 rows, all Historical. This excerpt is neither the current active catalogue nor a random sample.
Reading a CSV returns a DataFrame; df["PriceReg"] selects a Series with the same index values.
The source table has 20,000 rows and 14 columns; price is float64.
Operators broadcast
A scalar is one number. Multiplying price by 0.9 applies the same 10% discount to every price.
1. Original prices
2. Apply the same rule to every value
The values change from 44.99, 24.81 and 46.00 to 40.491, 22.329 and 41.4. The index remains 0, 1, 2; the result is still a Series. This is an illustrative discount, not an observed sale.
Multiply two Series: price × quantity
With two Series, multiplication matches the index values. .sum() then adds the resulting line values to obtain the order total.
1. Unit price
2. Quantity
3. Line value and total
A: 10 × 2 = 20; B: 20 × 1 = 20; C: 30 × 3 = 90. The total is 130. Each result stays attached to its product index.
Index alignment: how Series arithmetic matches observations
Series arithmetic matches index values. Match the index first, then add the values.
The right input is deliberately in a different order. Match index values before adding.
Read revealed values as a table
The coloured rows show which values belong together. At index 0, 1 + 100 = 101; at index 1, 2 + 200 = 202; at index 2, 3 + 300 = 303. The left Series also contains index 3, but the right Series has no observation there: the result is NaN, not 4 or zero. With unique indexes, the result includes every index present in either input. The later repeated-index example covers duplicate matches.
Two consequences follow immediately. First, pd.Series([1, 2, 3, 4]) + pd.Series([100, 200, 300]) does not fail on a length mismatch: index values 0–2 get sums and index value 3 gets NaN. Second, the result’s dtype is float64 even though every input was an integer, because these NumPy-backed integer operands use floating-point NaN; nullable integer dtypes instead support pd.NA. If you would rather treat the missing side as zero, the method form series1.add(series2, fill_value=0) does exactly that, then / 10 scales the resulting Series while preserving its index values.
The same rule applies to a real calendar. appl.csv holds 2 516 Apple closes from 2015-01-02 to 2024-12-31; microsoft.csv holds 780 Microsoft closes from 2014-12-31 to 2018-02-05. Add them and pandas aligns on the date index value: only dates present in both files receive a sum. In a hand-written loop you would build a lookup dictionary, test membership and decide a policy for misses — one + folds all three steps in. Use >, < and == for comparisons, and &, | and ~ for Boolean Series aligned by index. Put comparisons in parentheses when combining them.
The length-mismatched sum is [101.0, 202.0, 303.0, nan] with dtype float64; with fill_value=0 the last entry becomes 4.0. The chained (Sold.add(New_Release, fill_value=0) / 10) gives [0.1, 0.1, 0.0, 0.2, 0.2] for the first five products. For the two stocks the union holds 2 517 date index values — 2 516 Apple dates plus the one Microsoft date, 2014-12-31, that Apple’s file does not contain — of which 779 carry a sum and 1 738 are NaN. The first two matched days are 2015-01-02 (74.09) and 2015-01-05 (72.89), each the sum of an Apple close near 27 and a Microsoft close near 47. The AND mask selects {'Jan': 300, 'April': 200}; ~s1 flips the mask to {'Feb': 100, 'March': 0}; .between is inclusive at both ends by default and so counts three months.
Alignment prevents you from adding Apple’s Monday to Microsoft’s Tuesday. But it never warns. If you expected 780 sums and got 779, or expected two entries and got three with two NaN, alignment is the cause. The habit that protects you: after any binary operation on Series from different sources, print len() and .isna().sum(). Section 3 introduces reindex, the tool for aligning two calendars deliberately rather than by accident.
The illustration uses the existing small arithmetic example. Reverse the right Series and observe that sums by index are unchanged: 101, 202 and 303 for index values 0, 1 and 2, with NaN at index value 3. A missing partner is different from a zero-valued partner. Inspect unique index values and unexpected missing results before treating a binary operation as a meaningful business calculation.
Repeated arithmetic index values can multiply the result
Alignment matches index values, including repeated ones. Predict how many pairs share index value A before adding.
A duplicate value is harmless when two genuine observations happen to agree. A duplicate index value is different: the index value no longer identifies one observation. With unequal repeated indexes, pandas forms every matching pair. Do not solve this by deleting observations blindly; add the missing identity column or aggregate with a stated rule. Chapter 2 extends this check to composite-index value joins.
Two left observations match three right observations: six results, [11, 21, 31, 12, 22, 32]. Check that index values are unique before interpreting aligned arithmetic.
How two repeated index values produce six sums
For these unequal indexes, each left A matches all three right A entries. Each cell in the grid is one matching pair; reading across the first row and then the second gives the result’s order.
Summaries, masks and type conversion
Question: What does the summary count?
Result: An interpreted statistic and a record of parsing failures.
Comparisons return masks
Mean, median and the selected sample.
The mean is the observed sum divided by the observed count. The median is the middle sorted value, or the average of the two middle values.
For monthly units [300, 100, 0, 200, 1000], the mean is 320 and the median is 200. The unusually large 1000 raises the mean more than the median.
A scalar comparison returns a Boolean Series, or mask, with the original index and length. sales > sales.median() marks observations above the median; sales.loc[mask] selects them. sales.median() returns a scalar. State whether a summary uses all observations or the selected sample.
Two masks combine with & (and), | (or) and ~ (not). They must not be combined with Python’s keywords and, or, not: those keywords ask for a single truth value, and a five-element Series has five, so pandas raises ValueError: The truth value of a Series is ambiguous. And because & binds more tightly than >=, each comparison must sit inside its own parentheses; Section 3 shows the error you get without them.
The median of sales is 200. Before you run the cell, predict which month index values survive sales.loc[sales > sales.median()], and the dtype of the mask itself.
The mask has dtype bool and reads [True, False, False, False, True]; only January (300) and May (1 000) exceed the median of 200, so the filtered Series is {'Jan': 300, 'May': 1000}. The compound mask keeps the three months with at least 100 and fewer than 500 units — January, February and April. Notice that the filtered result kept its index values: filtering never renumbers, which is what lets you line the result up with other Series later.
Scores below the mean
Compare the original scores with the Series of scores strictly below the mean. The condition inside .loc[...] compares each score with the same overall mean.
1. Original scores
2. Select scores below the mean
The mean is 72.5. Only Stats 20 satisfies the condition. The result is a Series with index Stats, name Score and dtype int64.
Calculate standard deviation on three values
For illustrative prices 10, 20, 30, the mean is 20. Deviations are −10, 0, 10; squared deviations sum to 200. Sample variance is 200 / (3 − 1) = 100, so sample SD is 10 in the original price units.
Dividing 200 by 3 instead gives variance 66.667 and SD 8.165. ddof controls the divisor, not the units. Larger SD describes greater spread around the mean; the next example uses real catalogue prices.
Descriptive statistics and the ddof question
What each descriptive statistic measures.
| Method | Meaning |
|---|---|
.min(), .max() |
Smallest and largest observed values |
.mode() |
Most frequent value(s), returned as a Series |
.quantile(q) |
Cut point at fraction q of the sorted sample |
.var(ddof=1) |
Average squared deviation with divisor n−1 |
.std(ddof=1) |
Square root of variance, in the original units |
Summaries describe centre (mean, median, mode), spread (var, std, range, IQR) and shape (skew, kurt). Range is max minus min; IQR is \(Q_3-Q_1\). Quantiles may interpolate between observations, and ties affect exact fractions. Missing values are excluded by default. For variance and SD, also state ddof.
The sample standard deviation divides the sum of squared deviations by \(n-1\); the population standard deviation divides by \(n\):
\[ s = \sqrt{\frac{\sum_{i=1}^{n}(x_i - \bar x)^2}{n-1}}, \qquad \sigma = \sqrt{\frac{\sum_{i=1}^{n}(x_i - \mu)^2}{n}}. \]
Estimating \(\bar x\) from the same sample reduces the squared deviations. Bessel’s correction divides by \(n-1\), making sample variance unbiased; sample SD remains only approximately unbiased. ddof sets the divisor to \(n-\text{ddof}\). pandas defaults to ddof=1; NumPy’s np.std defaults to ddof=0. A smaller divisor gives a larger SD for non-constant data with at least two observations.
The SD ratio is \(\sqrt{n/(n-1)}\): \(1.000025\) for 20 000 prices, but \(\sqrt{5/4}=1.118\) for five sales figures—an 11.8% difference. State the observed sample and convention. Use ddof=1 for sample variance estimation and ddof=0 when describing the complete population.
Mean 109.852, median 89.95, mode 0.0. The mean sits well above the median because the distribution has a long right tail — the maximum is 2 800 while three quarters of the products cost less than 147.40 — and the mode is zero because 607 products have a recorded price of zero. That record alone does not establish whether they are free, unavailable or coded this way for another reason. price.std() is 85.9252 and price.std(ddof=0) is 85.9230, and np.std(price) reproduces the latter: NumPy’s default is the population divisor even when handed a pandas Series. The range is 2 800 and the inter-quartile range 95.41, a robust spread that ignores the tails the standard deviation is inflated by. For the five sales figures the sample standard deviation is 396.232 against a population value of 354.401, ratio 1.118 as computed above.
Leave .std() and .var() at their pandas defaults. ddof=1 is what Excel’s STDEV.S, statsmodels, and every statistics textbook mean by “the standard deviation of a sample”. When two standard deviations differ, compare their observed sample, missingness policy and ddof. When you must call NumPy, write np.std(x, ddof=1) explicitly.
Trace the denominator of a proportion
The mean of a mask, agg, and value_counts
A Boolean mask contains one answer per observation. Its sum counts True entries and its mean is their share among the entries counted. For the fully observed inventory prices, 15,146 of 20,000 are above 50, giving 0.7573. Report unknown values separately rather than treating them as observed failures.
Use price.mean() for one statistic and price.agg(["mean", "std"]) for a named collection. idxmax() returns the index value of the maximum. describe() bundles count, mean, standard deviation, minimum, quartiles and maximum.
The IQR is the 75th minus 25th percentile. The coefficient of variation is standard deviation divided by the mean, a ratio unstable near zero. skew() describes asymmetry and kurt() reports excess kurtosis against the normal reference of zero. These are descriptive summaries of this recorded sample; Chapter 3 examines distributional assumptions.
Mean 109.852, SD 85.925, IQR 95.41 and coefficient of variation 0.7822 measure different features of the same price column. Skewness 3.777 and excess kurtosis 70.075 indicate strong asymmetry and tail concentration; they do not establish a predictive probability model. The maximum price, 2,800, belongs to row index value 15511.
For a categorical Series, value_counts() returns one count per observed category; normalize=True returns shares among counted entries. nunique() counts distinct observed values. Missing values are excluded by default, so state that denominator or include missing explicitly.
2010 leads with 1,796 products, followed by 2008, 2009 and 2007. There are 61 distinct recorded years, including zero, which is a disguised missing year requiring investigation. The largest category contains about 9.0% of the catalogue.
Custom aggregation functions
s.agg(function) passes the whole Series to a named summary function. Use it when the same custom statistic belongs in several grouped summaries; Chapter 2 develops that use.
The answer remains 0.7822. A custom function is an alternative implementation of the statistic already taught.
Conversion: astype and to_datetime
Invalid dates become NaT; §1.3 applies date slicing.
The three text digits sum to 6 after conversion; the original Series still contains text.
Text dates are just strings — until you convert
pd.to_datetime() parses date text. Here format="%m/%d/%Y" declares month/day/four-digit year.
1. Before conversion: text index
2. Convert the index to dates
The index changes from text (object) to dates (datetime64[ns]). The four values and their order stay the same. §1.3 uses the parsed dates for selection; §1.4 develops calendar features.
Arithmetic aligns on the union of the two indexes before operating. An index value with no partner receives NaN; nothing is raised or truncated. Because NaN is a float, an integer result becomes float64. Use .add(other, fill_value=0) to treat the missing side as zero.
Larger. pandas divides by \(n-1\) (ddof=1, the sample standard deviation); NumPy divides by \(n\) (ddof=0, population). The ratio is \(\sqrt{n/(n-1)}\): negligible at \(n = 20\,000\) (85.9252 vs 85.9230) but 11.8 % at \(n = 5\).
True counts as 1 and False as 0, so the mean of a mask is the proportion of elements satisfying the condition — (price > 50).mean() = 0.757. The same trick computes missing-value rates (isna().mean()), hit rates, default rates and bootstrap p-values.
Parse dirty numbers and count conversion failures
A conversion can create missing values. This deliberately damaged copy of four inventory prices makes the failure visible. Predict how many values remain numeric after parsing.
Predict before running: How many conversion failures does this damaged four-row copy contain?
errors="coerce" gives one missing value and three numbers. Count newly missing values separately from values that were already missing. Use errors="raise" when invalid input must stop the pipeline.
An integer identifier can still be missing
SKU numbers are identifiers, not measurements. The capital-I dtype Int64 retains whole-number identifiers together with pd.NA. The missing ID below is introduced only to demonstrate the representation.
The identifier stays Int64; it does not become a fractional measurement. Comparisons with pd.NA can remain unknown. Filling the selection mask with False is an explicit decision to exclude unknown IDs, not to replace an ID with zero.
Section 1: concepts to retain
- pandas supplies Series (one axis) and DataFrame (two axes);
.shapereports axis sizes. - A named column selection returns a Series; a list of column names retains a table.
- Index values identify observations;
.locselects index values and.iloccounts positions. Arithmetic matches index values. - Time series vary dates for one entity; cross-sections vary entities at one date.
- Summaries need observed counts, units and a missingness policy; parse failures remain evidence.
Value Transformations and Cleaning
| Part | Question |
|---|---|
| 1.2.1 Conditional transformations and lookups | Should values change, or should rows disappear? |
| 1.2.2 Missing values and bounded repairs | What evidence supports a replacement? |
| 1.2.3 Ordering, ranks and bins | How should observations be compared or categorised? |
Conditional transformations and lookups
Question: Should values change, or should rows disappear?
Result: A transformed Series whose index values and length have been checked.
Apply one rule to the whole Series with an index
The first five mask values are False, False, False, True and True. The full Boolean Series has 20,000 entries, of which 15,146 are True. It remains a Series; reducing it with sum() returns a single count.
Trace the output structure
Trace selection and replacement on the same three rows
Here mask means a Boolean Series; .mask() is a method that replaces values where that condition is True.
| Index value | Value | Value > 50 | Selection | where(condition, 0) | mask(condition, 0) |
|---|---|---|---|---|---|
| A | 20 | False | removed | 0 | 20 |
| B | 60 | True | 60 | 60 | 0 |
| C | 80 | True | 80 | 80 | 0 |
Selection returns two rows; both replacement methods return three. Change the threshold to 70 and predict which index values and values change.
where and mask: if-else without a loop
Filter rows or replace values.
A Boolean mask gives one True/False answer per index value.
| Expression | Result |
|---|---|
s.loc[condition] |
Keep only qualifying entries; length can shrink |
s.where(condition, other=0) |
Keep values where True; replace the rest with zero |
s.mask(condition, other=0) |
Replace values where True; keep the rest |
where and mask preserve the index and length. Without other=, replacement is missing. Choose between row selection and value replacement before writing the condition.
From prices to fractional changes.
For prices in chronological order, the observed price return is
\[r_t = P_t/P_{t-1}-1.\]
close_prices.pct_change(fill_method=None) calculates this change from the previous row and retains the original index values. Despite its name, it returns a fraction: 0.01 means 1%. The first row has no preceding price and is missing; gaps are not filled.
s.where(cond, other) keeps values where cond is True; .mask(cond, other) replaces them. The default replacement is NaN. Both retain index values and length; Boolean selection drops unselected rows. The next exercise calculates mean return divided by SD as a daily return-to-risk ratio with a zero risk-free rate, excluding dividends and costs. It is not annualised.
The second is NumPy’s np.where(cond, if_true, if_false), which returns an array (wrap it in pd.Series with the original index to get a Series back) and reads like the sentence you meant. Labelling every product “Expensive” or “Cheap” is an if-else on every row; use np.where to choose between the two index values and restore the original index in the resulting Series. One warning about where with a string other: the moment a string enters a numeric Series the whole Series becomes object dtype, and arithmetic on it stops working.
Predict the first five values of price.where((price >= 50), other=0) given that the first five prices are 44.99, 24.81, 46.0, 100.0, 121.95, and the length of price.where((price >= 50)).
where keeps the two prices at or above 50 and zeroes the other three: [0.0, 0.0, 0.0, 100.0, 121.95]. mask does the opposite: [44.99, 24.81, 46.0, 0.0, 0.0]. The "Big" version works — [44.99, 24.81, 46.0, 'Big', 'Big'] — but the dtype is now object. The length is 20 000 either way. The new index value Series contains 15,146 Expensive and 4,854 Cheap values, with the original index values retained.
Compute returns before combining them with a condition
Illustrative history: one stock, four successive observed dates; prices in currency units are 100, 110, 99, 108.9. Returns are missing, 0.10, −0.10, 0.10: 110 / 100 − 1 = 0.10; 99 / 110 − 1 = −0.10.
where retains all four index values but leaves non-winning entries missing. std() uses the two observed winning returns; with fewer than two, sample SD is missing. The following real-data exercise combines these already visible steps.
Worked example: daily return-to-risk ratio. Use the archived AAPL prices for 2015–2024. A daily mean divided by daily sample standard deviation assumes a zero risk-free rate and is not annualised. The calculation excludes dividends and costs.
The daily ratio is 0.0581 and winning-day SD is 0.0127. where leaves other days missing, and std() uses the remaining observed values. This descriptive ratio does not establish a profitable strategy.
Trace filtering and conditional replacement
Map a value to its reference information
map attaches lookup information.
s.map(reference) looks up each value in a dictionary or applies a function to each value. The result retains the Series index.
For an ordinary dictionary, an uncovered index value becomes missing. This is useful evidence of incomplete reference coverage; do not silently turn it into a known category.
s.replace(mapping) recodes specified values and normally leaves other values unchanged. Use map to attach reference information and replace to recode selected values. Inspect coverage after a lookup.
Follow one row through both branches. SI and CCCC have matching sector index values, so both methods return the mapped code. LESL and CPB have uncovered sectors: map returns a missing value while replace keeps the original sector name. In every case, the ticker index is preserved; the dictionary is matched against the sector value, not the ticker index value.
Lookup is different from recoding. Here a deliberately incomplete sector dictionary covers two of the sectors in the first four company records. Predict what happens to an uncovered sector.
Predict before running: An ordinary dictionary has no entry for Consumer Cyclical. What does map return?
map produces ["FIN", NaN, NaN, "HLT"] and preserves the caller’s index. replace leaves uncovered sector names unchanged. Check unmatched lookup values before using the result as a feature.
Membership and text cleaning produce Boolean Series
Use isin to ask whether each value belongs to a permitted set. Clean whitespace and case before matching text; do not infer a permanent security identity by stripping parts of a ticker.
The selected company index values are SI and CCCC. The four cleaned ticker strings match the reference table. .all() asks whether every check passed; .any() asks whether at least one did. These are scalar checks, unlike element-wise & and |.
apply supplies each value to a function
For the scalar function used here, price.apply(compare) calls compare(value) for each price and keeps one returned result per input index value.
price > 50 applies the comparison to every price and returns a Boolean Series with the original index. A built-in numerical operation usually avoids repeated Python function calls.
Timing measures this computer and run, not a universal speed ratio. First verify that the results agree; then compare runtime. Prefer the clearest suitable built-in method.
Time them
| Tool | Purpose and return value |
|---|---|
perf_counter |
time.perf_counter() reads a high-resolution clock; subtract two readings for elapsed seconds. |
apply |
For this scalar function, call it on each value and retain its returned result and input index value. |
All three agree (True True); the recorded times compare three equal-result implementations on this run. Rule: reach for apply only when no vectorised method exists.
Missing values and bounded repairs
Question: What evidence supports a replacement?
Result: Missingness counts, a stated repair policy and a valid denominator.
Missing data: how NaN behaves
Missing values and numerical operations.
Choose a missing-value policy before filling.
| Method | Policy |
|---|---|
s.dropna() |
Remove unknown observations |
s.fillna(value) |
Replace them with a declared value |
s.ffill() |
Carry an earlier observed value forward |
Forward filling carries the last observed value; its meaning and timing must justify that policy because it can become stale. Preserve a missingness flag when useful.
company = isom5650.data.publicCompany() returns the same table (3 084 tickers, sector, marketCap, dividendYield, …) with the ticker as the index.
isna() marks gaps; notna() marks observations. isna().sum() counts gaps and isna().mean() gives their share. count() counts observations; len() counts rows. Most aggregations skip NaN, while element-wise arithmetic propagates it: NaN + 1 and NaN * 0 remain missing. Thus the yield mean describes 1 474 observed companies, not all 3 084.
Choose the policy from the variable’s meaning. dropna() changes the represented sample; many remaining rows do not remove selection bias. fillna(value) supplies an assumption: zero for a confirmed non-payer, or a justified median/mode. ffill() can implement last-known-price valuation, but does not recover a tradable quote or an unknown outcome. The toy compares these policies, then fills dividend yields with their median.
1 610 of 3 084 dividend yields are missing — a missing rate of 0.522, more than half. 1 474 are present, and the three spellings (notna().sum(), the ~isna() mask, dropna()) agree. The sum is 48.277 and the mean 0.0328, both over the 1 474 observed yields; count() says 1 474 while len() says 3 084. (dy + 1) still has 1 610 holes, and (dy * 0).sum() is 0.0 — the multiplication left NaN in the holes, and the sum then skipped them. On the toy: dropna leaves [1.0, 4.0]; fillna(0) gives [1.0, 0.0, 0.0, 4.0]; ffill gives [1.0, 1.0, 1.0, 4.0]. After filling with the median, zero dividend yields are missing.
The lecture notebook asks you to fill the gaps in a 0/1 SoldFlag with the mode. The cell below runs without error and prints a result — but it is wrong. Find the bug before reading on.
mode() returns a Series because modes can tie. Here it returns pd.Series([0.0]), indexed by 0. Passing that Series to fillna aligns index values; index value 0 is already observed, so two gaps remain. Extract the scalar with sold.mode().iloc[0] or .loc[0]: the result is [0.0, 1.0, 0.0, 0.0, 0.0, 0.0], with no gaps and total 1. Distinguish scalar replacement from index value-aligned replacement.
Trace missingness and its denominator
Choose a policy, then trace the replacement and the number of observed or retained values.
Read revealed values as a table
A valid ratio needs a valid denominator
Compare list price with the inventory file’s low user price. A zero denominator is not a very large valid ratio. Keep invalid results missing and count them.
Three of four ratios are valid. The second low user price is zero, so its ratio remains missing. notna() alone does not reject infinity; np.isfinite does. Filling that missing ratio with zero would change its meaning.
Ordering, ranks and bins
Question: How should observations be compared or categorised?
Result: An ordered or categorised Series with ties and cut points explained.
Work out a quantile before clipping
Sorting, clipping, and duplicates
Sorting and clipping preserve observation index values.
s.sort_values() orders entries by their values; s.sort_index() orders them by their index values. Both return a new Series unless explicitly asked to modify in place. Index values travel with their values.
s.clip(lower=a, upper=b) replaces values below a by a and values above b by b; it retains the rows. A Boolean filter instead removes rows.
sort_values() orders values and retains their index values; ascending=False reverses that order. sort_index() orders index values. Sorting returns a new Series; the original is unchanged. Clipping also returns transformed values, but does not establish that extremes were errors. Fit predictive thresholds on training data.
s.quantile(0.8) == 21 estimates an 80th-percentile cut point; interpolation and ties prevent interpreting it as an exact count below 21. Clipping retains index values and replaces values outside its bounds. Count the changed rows: 5th/95th-percentile clipping need not change 10%. Investigate the 2,800 price and preserve the original. Predictive thresholds must come from training data, not held-out observations.
drop_duplicates() removes repeated values, and the keep argument decides which copy survives: "first" (the default) keeps the earliest position, "last" the latest, and False keeps neither — every value that appears more than once is removed entirely.
Ascending order puts three recorded zero prices at the top, with index values 4725, 17419 and 5531 — index values that tell you which products have recorded zero prices. Descending order puts {15511: 2800.0, 4071: 2083.09, 2218: 1580.0} first, and the subsequent sort_index(ascending=False) reverts to index value order, showing rows 19999, 19998 and 19997. The clip cut points are 12.50 and 271.667; exactly 1 000 prices lie below the first and 1 000 above the second. Clipping raises the minimum from 0 to 12.50, lowers the maximum from 2 800 to 271.67, moves the mean only from 109.85 to 106.73, and cuts the standard deviation from 85.93 to 71.16 — a 17 % reduction in the dispersion measure from touching 10 % of the rows, which tells you how much of that standard deviation was tail. For dlist, keep="first" retains the 20 at position 1, keep="last" retains the one at position 3, and keep=False returns {0: 40, 2: 30, 4: 10} with no 20 at all.
Ranking and replacing
The eleven prices, in index value order, are:
| Index value | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Price | 44.99 | 24.81 | 46.00 | 100.00 | 121.95 | 132.00 | 95.95 | 207.80 | 119.81 | 49.95 | 36.95 |
Ranks, ties and percentile ranks.
Compare a common date and horizon; a price rank is not investment value.
replace maps values to values. Given a list and a single replacement, every listed value becomes that replacement; given a dictionary, each index value maps to its own value. It is the tool for recoding — turning ranks 1–4 into the index value "lowest", turning a survey’s "Y"/"N" into 1/0 — and, as with where, the moment a string enters a numeric Series the dtype becomes object. The cell works on partPrice = price.loc[:10]: the first eleven prices, because .loc is inclusive of its end index value — a fact Section 3 makes much of.
Eleven prices. The first four, 44.99, 24.81, 46.0, 100.0, receive ranks 3.0, 1.0, 4.0, 7.0: 24.81 is the smallest of the eleven, 44.99 the third smallest, and the 207.80 at position 7 gets rank 11. The list form of replace relabels ranks 1–4 as "lowest" (four entries), the dictionary form only ranks 1–3 (three entries), and in both cases the remaining entries stay floats inside an object Series. The percentile ranks run from 0.091 for the minimum to 1.0 for the maximum, in steps of \(1/11\).
Rank a dated cross-section, not a mixture of dates
§1.1 compared two data shapes using stored closes. To compare this session’s price performance, first compute each stock’s change along its time-series index and only then assemble a cross-section for the common date. The loading and return calculation are supplied here; §1.4 explains pct_change in detail. The observed changes are AAPL −2.498% and MSFT −4.119%. Predict which receives the larger ascending rank, then verify both calculations.
AAPL’s change is −2.498% and MSFT’s −4.119%, both measured from 2 to 5 February 2018. With default ascending ranks, the larger, less negative return receives the larger rank: percentile 1.0 for AAPL and 0.5 for MSFT in this two-stock snapshot. Sorting the ticker index values would not change those ranks. Ranking all dates together would instead compare a stock-date against a mixture of peers and market sessions, answering a different question.
These are historical stored-close price changes excluding dividends, not a forecast or an executable strategy account. A higher realised rank in one session provides no evidence of future outperformance. Chapter 2 repeats the operation within each date using groupby and introduces sector-relative peer groups. Keep the common horizon, availability clock, eligible population and missing-value policy explicit when turning a cross-sectional rank into a model feature.
Binning: equal width or equal count?
Binning: cut uses boundaries; qcut uses quantiles.
Inventory prices span 0–2 800, so four equal-width bins are 700 wide. With median 89.95 and 95th percentile 271.67, most products occupy the first bin. Quantile bins hold about 5 000 products each. Explicit edges can encode price tiers and separate the 607 recorded zero prices. Intervals are right-closed by default: (0, 1000] includes 1 000 but excludes 0. Inspect edges, counts and missing assignments before adopting a business rule.
Symmetry does not make equal widths equal counts
For −4, −1, 0, 1, 4, equal-width bins have inner boundaries −2, 0, 2; sample quartiles are −1, 0, 1. The data are symmetric, but the boundaries differ.
retbins=True returns two objects: the assignments and the numerical boundaries. Unpack them to inspect how the groups were formed.
Now apply the same distinction to the inventory prices. Inspect counts as well as boundaries; ties can make quantile groups unequal.
There are 19,990 prices at or below 700 among 20,000 prices ranging from 0 to 2,800. Predict the share in the first of four equal-width bins, whose upper boundary is 700, then compare with quantile bins.
Equal width: 19 990 of 20 000 products in (-2.8, 700], then 7, 2 and 1 in the three upper bins — the leftmost edge is pushed slightly below zero so that the minimum is included. The notebook’s explicit edges give [607, 19389, 4, 0]: 607 recorded zero prices, 19 389 priced up to 1 000, four between 1 000 and 4 000, none above. qcut produces four bins of 5 007, 5 111, 4 884 and 4 998 products with edges at 51.99, 89.95 and 147.40 — the quartiles you met in Section 1. The rule: cut when the width of a bin has a meaning (age bands, price tiers), qcut when you want equal-sized groups (deciles of a signal, quartiles of income).
The notebook’s practice — rlist = pd.Series([100, 120, 50, 200, 40]), index value "strong" where weight > 50 else "slim" — is pd.Series(np.where(rlist > 50, "strong", "slim")), giving ['strong', 'strong', 'slim', 'strong', 'slim']. Keep the original index when wrapping the selected values in a Series.
where keeps the length: values where cond is True are kept, the rest replaced by 0 (or NaN by default). A boolean filter s.loc[cond] drops the False rows and returns a shorter Series. Use where when the result must still align with other columns; use a filter when you want the rows gone. mask is where with the condition inverted.
s.mean() returns a number — the mean over the 1 474 present values — because aggregations skip NaN by default. (s + 1).isna().sum() returns 1 610, because element-wise operations propagate NaN. Same missing value, opposite treatments; count() (1 474) is not len() (3 084).
Call it a correction only after establishing an error from source evidence. Otherwise it is a declared transformation whose effect on the question must be assessed. A high-priced product may be genuine, and extreme returns matter for risk. In this inventory sample, exactly 1,000 prices fall below the lower cut point and 1,000 above the upper one: 10% change here. Ties or another sample can change that fraction. Report the bounds, the number changed and the original result alongside the transformed result.
Section 2: concepts to retain
- Choose whether to change values, membership or order; built-ins often express the rule directly.
wherekeeps True values;maskreplaces them.mapperforms a lookup;replacerecodes.- Dropping, filling and carrying forward apply different assumptions; report the denominator.
- Sort, clip, deduplicate, rank and bin for distinct purposes; inspect ties, cut points and index values.
- Calculate returns within a sorted entity history, then rank a common dated cross-section.
Selection and Index Alignment
| Part | Question |
|---|---|
| 1.3.1 Index, positions and slices | Which entries does this selector address? |
| 1.3.2 Inspection, sampling and deliberate alignment | Which identities survive this operation? |
| 1.3.3 Selection and safe assignment | Which object receives the change? |
Index, positions and slices
Question: Which entries does this selector address?
Result: A selection with the intended index values and endpoints.
The index is the address book
Use .loc, .iloc and deliberate reindex/align to address observations. The following examples distinguish changing index values, changing the index header and matching index value text. Chapter 2 develops moving identifiers between columns and a table index.
The supplied cell prepares a small Series with months as its index and the AEP hourly-load Series. read_csv uses Datetime as a parsed index, and .loc["2017"] selects the entire year. You will apply the endpoint rules to these same observations in the next examples.
The small Series has index values jan, feb and march with values 100, 200 and 50. The AEP extract contains 8,759 hourly readings in 2017; inspect its actual dates instead of assuming every calendar hour was recorded.
Edit index values, index names or the index structure
rename takes a function, a dict, or nothing
Before running: Read each rename operation. Which index values change, and which data values stay the same?
A function maps every index value; a dict maps only the ones you list; assigning .index replaces the whole thing (and must have the right length).
rename_axis renames what, exactly?
| Tool | Purpose and return value |
|---|---|
reset_index |
Move Series index values into a DataFrame column; drop=True discards them and retains a Series. |
rename_axis |
Set the index name, rather than its individual index values. |
aep.rename_axis(“FancyName”) changes …
Answer: The index’s name — the header above the index values
rename_axis changes the name of the index (the header printed above the index values), not the index values themselves. It matters at reset_index time: the index name becomes the new column’s name.
reset_index() turns a Series into a two-column DataFrame whose first column is named after the index (FancyName, or Datetime). drop=True throws the timestamps away and leaves a plain RangeIndex — useful after filtering, dangerous for time series.
energy = isom5650.data.energy() returns the same PJM hourly load table (2004–2018) with Datetime as the index; aep = energy["AEP"].
.loc is right-inclusive, .iloc is right-exclusive
.loc selects index values; .iloc selects positions. Both accept single values, lists and slices; .loc also accepts Boolean Series aligned by index. .loc[a:b] includes the named end index value, while .iloc[i:j] excludes position j. Half-open positional slices make s.iloc[:k] and s.iloc[k:] a partition with no gap or overlap. Check the actual selected index values, even when row counts agree.
aep.loc["2017-01-01 01:00":"2017-01-01 05:00"] selects five hours, 01:00–05:00. aep.iloc[0:5] selects five positions, 00:00–04:00. Equal counts do not imply equal rows. .loc[["label"]] retains a one-row Series; scalar lookup returns a scalar. int(len(aep) * 0.7) defines the position of a chronological 70% split.
Five rows ending at 2017-01-01 05:00:00 by index; five rows ending at 2017-01-01 04:00:00 by position. The single-index value list returns a one-element Series showing 12 876 MW at 01:00 with its Datetime header intact. The 70 % training slice holds 6 131 hours and ends at 11:00 on 13 September 2017.
Trace index value and position endpoints
The table is ordered by its numeric index. The index values are not row positions.
Read revealed values as a table
Partial-date strings and the slicing trap
A DatetimeIndex accepts partial date strings in .loc, and they slice whole periods: "2020" means every row in 2020, "2020-03" every row in March 2020, and "2020-01-01":"2020-12-31" the same year spelled out. This is one of the most pleasant features of pandas — a backtest window reads like a sentence — and it is inclusive at both ends, as every .loc slice is. The cell uses the S&P 500 daily closes from sp500.csv.
Within jan = sp500.loc["2020"], positions 0–4 are 2, 3, 6, 7 and 8 January. .loc["2020-01-06":"2020-01-08"] and .iloc[2:5] select 6–8 January. .iloc[2:4] excludes 8 January. When mixing index value and position slices, check boundary dates as well as counts.
"2020" returns 253 trading days, "2020-03" the 22 trading days of March, and the spelled-out range the same 253. The March 2020 low was 2 237.40 on the 23rd, the bottom of the pandemic crash. The first five 2020 index values are 2, 3, 6, 7 and 8 January. The index value slice and iloc[2:5] both give [3246.28, 3237.18, 3253.05]; iloc[2:4] gives only the first two.
In Growth in a Time of Debt (2010) Reinhart and Rogoff reported that countries with public debt above 90 % of GDP averaged −0.1 % real growth, a figure cited by the UK Treasury, the European Commission and the US Congress to justify austerity. In 2013 Thomas Herndon found that the spreadsheet’s averaging formula covered rows 30–44 instead of 30–49: Australia, Austria, Belgium, Canada and Denmark were silently excluded, and with other corrections the figure became +2.2 %. The pandas equivalent is an .iloc end one row short, or a NaN skipped by mean(). Print len(), print isna().sum(), and never trust an aggregate whose denominator you have not seen.
Conditional selection and index versus position
.loc accepts a Boolean Series aligned by index. Build the condition with familiar comparisons, inspect it, then select the matching observations. Compare it with where from Section 2: price.loc[price > Q3] drops the rows below the third quartile, while price.where((price > Q3)).dropna() reaches the same 4 998 rows in two steps.
Parenthesise each comparison: (r > 0.02) & (r < 0.05). Without parentheses, & binds before > and can raise TypeError. Python’s and, or and not require one truth value; they raise ValueError on a Series. Use &, | and ~ for masks.
Sorting changes positions while preserving index values. After sorting prices, top.loc[0] selects index value 0 (44.99); top.iloc[0] selects the first position (2 800, index value 15511). The next index values are 4071 and 2218. The integer 0 has two meanings here: .loc[0] names index value 0, while .iloc[0] names the first position.
top = price.sort_values(ascending=False) has index 15511, 4071, 2218, …. Compare the printed results of top.loc[0] and top.iloc[0].
The third quartile is 147.40 and 4 998 products lie above it, both ways. price.loc[price > 50] returns the 15 146 products above 50 from Section 1, and the 99th-percentile mask picks out the top one per cent, whose three largest members are 1 580, 2 083.09 and 2 800. After sorting, top.loc[0] is 44.99 and top.iloc[0] is 2 800.0. The .loc selector matched the index value, and the first three index values of the sorted Series are [15511, 4071, 2218] — the products with the three highest prices.
Of Microsoft’s 779 trading days from 2015 to early 2018, 41 had a return between +2 % and +5 % and 83 moved more than 2 % in either direction. Without the parentheses the expression raises TypeError. The lecture notebook’s practice — find the worst days, those with a loss beyond 5 % — is one .loc with a mask: three days, the worst being 27 January 2015 at −9.25 %, the day after a disappointing earnings report.
Inspection, sampling and deliberate alignment
Question: Which identities survive this operation?
Result: A reproducible sample or aligned result with index values retained.
head, tail, sample, and reindex
Inspect a Series or draw a reproducible sample.
s.head(n) takes the first n entries; s.tail(n) takes the last n. Neither is a random sample.
s.sample(n, random_state=0) draws entries reproducibly. Without replacement, each selected entry appears at most once; replace=True allows repeated draws.
head(n) and tail(n) select the first and last rows; sample draws rows. Set size with n or frac: 0.002 of 20 000 is 40. random_state reproduces the draw; it does not establish representativeness or independence. replace=True permits repeats, so 500 draws may contain fewer than 500 distinct rows—the basis of bootstrap sampling in Chapter 3. Preserve chronological order for time-based training splits.
reindex follows the supplied index values and order, introducing NaN for absent index values; numeric dtype may change to represent missingness. .loc with an unknown index value instead raises KeyError. Reindexing can place sparse observations on a common calendar, after which each gap needs a variable-specific policy. A missing held-asset return is not zero; an unknown future target need not exclude a prediction candidate.
With seed 0 the five sampled index values are [19134, 4981, 16643, 19117, 5306]; frac=0.002 gives 40 rows; and of 500 draws with replacement only 492 are distinct — eight products came up twice. reindex returns {'jan': 100.0, 'march': 50.0, 'Tom': nan, 'Katy': nan} — the two real index values selected, the two invented ones NaN, the integers now floats — while .loc with the same unknown index values raises KeyError.
Repeated values and repeated index values are different defects
Two products may legitimately share a price. Two observations sharing one supposed unique product index value need investigation. Predict which rows each check marks.
The value mask is [False, True, False]; the index value mask is [False, False, True]. Sorted index values can still be duplicated. Decide what each index value represents before dropping rows. Chapter 2 uses ticker and date together to identify rows in a stock panel.
Choose the common index values before comparing Series
Four original products become three matched pairs; the correlation is 1.0 in this proportional repricing example. corr aligns index values and uses non-missing pairs. Report the paired count with the correlation; high correlation does not mean equal prices.
Arithmetic alignment and comparison have different contracts
A comparison with one number keeps the Series index. A direct comparison between two Series requires the same index values in the same order. Addition can align a different order automatically; comparison does not silently choose that policy.
The sum is A = 20, B = 40; after explicit alignment the comparison is False for both index values. join="inner" keeps common index values: inspect which observations were excluded before interpreting the result.
Sorting retains the product index values
Sorting changes positions while the product index values stay attached to their prices. Apply a transparent 10% price reduction to the sorted Series, then compare index-based and position-based selection.
The first original SKU still has its reduced price of 40.491 when selected with .loc. The first position in the sorted Series belongs to a different product and gives 22.329. Use index values to identify products; use reindex when the result must follow a specified row order.
Your turn: restore the row order
revised_price retains its product index values but follows the sorted price order. Use reindex to put it in the original order given by sku_price.index.
Check both the index values and the values. Matching lengths alone cannot prove that a revised price belongs to the correct product.
filter selects index value text; a mask selects values
| Tool | Purpose and return value |
|---|---|
filter |
Select by index text: items lists index values, like matches a substring, regex matches a pattern. |
Use items for named index values, like for a substring, or regex for a pattern. These arguments inspect the index.
like="Ma" keeps March and May, including the zero. The value mask keeps January and May. Choose one of items, like, and regex per call.
Selection and safe assignment
Question: Which object receives the change?
Result: One explicit selection or assignment, checked against its source.
Select values with a Boolean condition
Use .loc[condition] to select observations by their values. A index-text filter answers a different question; both retain the selected observation index values. Assignment then specifies which object receives a change.
Writing into a slice: the SettingWithCopy trap
Everything so far has read from a Series. Writing into part of one is where the most common pandas bug lives. Suppose you want to add 100 to every price above 50 among the first five products. The correct form is a single .loc on the left-hand side: p.loc[p > 50] = p.loc[p > 50] + 100. pandas evaluates the mask, locates the rows in p itself, and writes the new values into them.
Chained assignment such as q.loc[q > 50].loc[3] = 0 writes through an intermediate object. Its behaviour has historically depended on the selection and pandas settings; copy-on-write prevents it from updating the original. Use one .loc[...] = ... on the intended Series, and keep warnings visible. The example creates p and q with explicit .copy() so edits to those examples do not alter price.
A means units > 2; B means units < 6. True retains the original observation index.
Read revealed values as a table
On the course file nvda_spy_daily_2023_2024.csv (501 trading days), the mask selects 102 Friday closes. The next observed trading session is Monday for 91 Fridays and Tuesday for 11, following Monday holidays; for example, 13 January 2023 is followed by 17 January. shift(-1) — introduced in §1.4 — puts the next session’s close on the Friday row. Retain its timestamp; call a pair Friday–Monday only when that timestamp is Monday:
requests downloads the public CSV; StringIO lets pandas read the downloaded text. These supplied loading helpers support the DataFrame reading covered in §2.1. ::::: {.course-data-example} :::: {.input-code-pair} ::: {.input-static-code}
Complete Colab code
import pandas as pd
import requests
from io import StringIO
URL = "https://statpython-book.pages.dev/data/"
response = requests.get(URL + "nvda_spy_daily_2023_2024.csv", timeout=30)
response.raise_for_status()
nvda = pd.read_csv(StringIO(response.text), index_col="Date", parse_dates=True)["NVDA"]
fridays = nvda.loc[nvda.index.day_name() == "Friday"]
next_session = pd.Series(nvda.index, index=nvda.index).shift(-1)
pairs = pd.DataFrame({"friday": fridays, "next": nvda.shift(-1).loc[fridays.index],
"next_session": next_session.loc[fridays.index]})
print(pairs.head(3))
print(pairs["next_session"].dt.day_name().value_counts())| Date | NVDA | SPY |
|---|---|---|
| 2023-01-03 00:00:00 | 14.299908638000488 | 365.0720520019531 |
| 2023-01-04 00:00:00 | 14.733452796936035 | 367.8904724121094 |
| 2023-01-05 00:00:00 | 14.24996566772461 | 363.6915893554687 |
| 2023-01-06 00:00:00 | 14.843337059020996 | 372.0317687988281 |
| 2023-01-09 00:00:00 | 15.611525535583496 | 371.8209228515625 |
501 rows × 2 columns · index shown on the left
Column types
NVDA: float64; SPY: float64:::: :::::
:::
.loc slices by index and includes the end index value; .iloc slices by position and excludes the end position (Python’s half-open convention). They agree here only because positions 1, 2, 3 happen to be Tue, Wed, Thu. Change to .iloc[1:3] and only Tue and Wed remain. Mixing the two conventions in one backtest is off by one row at every boundary.
44.99 — the price of the product whose index value is 0, not the maximum. The .loc[0] selector looks up index value 0; top.iloc[0] is what returns the 2 800 maximum. Always write .loc or .iloc.
q.loc[q > 50] is an expression that returns a new, temporary Series (a boolean filter cannot be a view). .loc[3] = 0 writes into that temporary, which is then discarded. Nothing reaches q, and pandas may or may not warn. The fix is one .loc on the left-hand side: q.loc[(q > 50) & (q.index == 3)] = 0, or simply q.loc[3] = 0.
Section 3: concepts to retain
- Use
.locfor index values (inclusive stop),.ilocfor positions (exclusive stop). Sorting changes positions. - Keep observation index values through filtering, sorting and numerical transformations; index value editing changes names without changing the recorded values.
sampledraws observations;reindexconforms to index values and can introduce missing entries.- Repeated values and repeated index values differ; align index values before comparing or calculating correlation.
- Keep index values attached during calculations, use
reindexfor a required row order, and assign through one.loc.
Time-series Features and Resampling
| Part | Question |
|---|---|
| 1.4.1 Dates, lags and observed returns | What was observed before this row? |
| 1.4.2 Rolling and cumulative features | Which observations enter the statistic? |
| 1.4.3 Calendar bins and resampling | What belongs in each time bin? |
Dates, lags and observed returns
Question: What was observed before this row?
Result: A chronological Series with amount changes and fractional changes distinguished.
A date column is text until you say otherwise
Series dates use .dt for year, month, dayofweek (Monday = 0), day_name() and is_month_end; a DatetimeIndex exposes those properties directly. strftime formats dates as text, for example 2013-01-01 as "01/01/13". Returns, rolling windows and calendar summaries use this dated structure.
The cell uses the Delhi daily-weather table, 1 462 rows from 2013-01-01 to 2017-01-01, whose date column arrives as text. It then applies the same tools to a finance question the lecture notebook poses: the weekend effect, the folklore that Monday returns are systematically lower than Friday’s because investors sell on Mondays after digesting bad weekend news. With a DatetimeIndex the test is a mask on index.day_name() and two means.
weather = isom5650.data.weather() returns the same Delhi table. For the weekend effect the notebook downloads ten years of a ticker with yfinance; appl.csv holds the same decade for Apple.
The table is 1 462 × 5; the first date is a str before conversion and a Timestamp after, and the column’s dtype is datetime64[ns]. 1 January 2013 was a Tuesday, so dayofweek is 1. The data touch five calendar years but contain 48 month-ends — twelve for each of 2013–2016; the single day of 2017 is not a month-end. strftime("%d/%m/%y") gives "01/01/13" and "02/01/13", and parsing the %m/%d/%y layout back recovers 2013-01-01.
In these 2 515 Apple daily returns, Mondays average about +0.23% and Fridays about +0.01%, opposite to the stated folklore. This is a descriptive comparison for one stock and one period. Comparing a difference in means with the standard deviation of individual returns does not establish statistical significance. Chapter 3 introduces uncertainty and testing; a trading recommendation would also need a clearly defined holding period, costs and out-of-sample evaluation.
shift: from prices to returns
shift(1) moves every value one row later, so that the value on today’s row is yesterday’s. The first row has no predecessor and becomes NaN; the last value falls off the end. shift(-1) moves the other way — tomorrow’s value sits on today’s row, and the NaN appears at the end. That one method turns a price series into a return series, because the simple return is
\[ r_t = \frac{P_t - P_{t-1}}{P_{t-1}}, \]
today minus yesterday, over yesterday — in pandas, (P - P.shift(1)) / P.shift(1). pct_change() is the same line pre-written and gives an identical result. The cell also computes the forward return P.shift(-1) relative to P: tomorrow versus today. Its last entry is NaN because tomorrow is not yet known, and that is exactly why the forward return is the target \(y\) of every forecasting model in this course and never a feature — a model that sees shift(-1) on its input side has been shown the answer.
The notebook downloads Microsoft with yfinance (isom5650.data.getStock("MSFT")); microsoft.csv holds the same ticker from 2014-12-31 to 2018-02-05, and .loc["2015":] keeps the 779 trading days from 2 January 2015.
In the head, shift(1) is NaN on 2 January 2015 and carries 46.76 — the first close — onto the 5 January row; shift(-1) carries 46.33, the second close, back onto the first row. In the tail, shift(-1) is NaN on the last day, 5 February 2018. The manual return and pct_change agree (True); the first two returns are −0.92 % and −1.47 %. Over 2015–2018 Microsoft’s mean daily return was 0.091 % with a standard deviation of 1.42 %. The forward return ends [-0.0412, nan]: the 4.12 % fall from 2 to 5 February 2018 is known on the 2nd only in hindsight.
Trace the previous price into the return row
Prices are ordered by date. Positive shift brings past prices; negative shift brings future prices.
Read revealed values as a table
diff measures an amount; pct_change measures a rate
The first three MSFT closes show the difference. Both methods compare successive observations, which need not be successive calendar days.
Friday, Monday, Tuesday: differences are [NaN, -0.43, -0.68] dollars; fractional changes are [NaN, -0.0092, -0.0147]. 0.01 means 1%, not 0.01%. Position-value differences become P&L only after units and cash flows are handled.
A missing price does not establish a zero return
Damage the middle of the three observed prices to inspect the missing-data rule. Filling the price first answers a different question from leaving it unknown.
The unfilled series has no observable adjacent return. Forward filling creates a zero on the missing date and a later multi-observation move. Use this only under an explicit valuation policy; it does not recover an executable historical price.
Rolling and cumulative features
Question: Which observations enter the statistic?
Result: A dated feature with a valid window and decision-time cutoff.
Rolling windows
Rolling windows and calendar resampling.
s.rolling(k).mean() calculates a mean from the most recent k rows at each position. For an integer window, the default requires k observed values; min_periods changes that requirement.
s.resample("ME").mean() instead groups a DatetimeIndex into calendar-month bins and summarises each bin. The month-end index value identifies the bin boundary, which need not be an observed trading date.
Sort chronologically and check coverage. rolling(k).mean() uses the latest \(k\) rows, including the current one. Complete windows leave \(k-1\) leading gaps: nine for rolling(10), thirty-nine for rolling(40). The first valid mean equals the mean of the first \(k\) closes. Short windows react faster; long windows smooth and lag. A crossover defines a candidate rule, not evidence of predictive value. Use built-in mean, std, median, min, max and sum before custom callbacks.
plt is the short name for Matplotlib’s pyplot module, imported with import matplotlib.pyplot as plt. It creates and displays charts. §1.5 explains plotting in detail; here, compare the moving averages.
For the supplied chart, fig, ax = plt.subplots(...) creates the canvas and one plotting panel. ax.plot(series, label=...) draws against its index, ax.legend() identifies the lines, ax.set(title=...) sets the panel title, and plt.show() displays it.
Nine and thirty-nine leading NaN; the tenth value of the fast average, 46.41, is the mean of the first ten closes. In the figure the 40-day line barely notices the dips of mid-2016 that the 10-day line follows down and back up.
Compound two returns by following the money
The sum is zero, but wealth lost 1. Gross growth, cumulative return and the sum of simple returns are three different quantities. The next historical example applies this same calculation to all observed returns.
Rolling variability and historical compounded growth
r.rolling(21).std() measures realised return volatility. Multiplying by \(\sqrt{252}\) annualises it under variance scaling across about 252 trading days; serial dependence can invalidate that assumption. Chapter 6 models volatility.
cumsum, cumprod, cummax and cummin calculate running sums, products, highs and lows. Gross returns telescope: \(\prod(1+r_t)=P_T/P_0\). This is price growth, not a complete portfolio account; dividends, cash flows and costs also matter. Summing simple returns does not compound them.
The 21-day realised volatility peaked at 43.2 % annualised on 18 September 2015, in the aftermath of the August 2015 China devaluation shock, and averaged 14.3 % through the calm of 2017. Compounded growth is 1.882 — exactly the ratio of the last close (88.00) to the first (46.76): one dollar became $1.88. The running sum of simple returns says 0.711, a different and wrong number — it corresponds to a 71 % gain, not 88 %. cummax ends at the overall maximum, and cummin never moves once the 40.29 low of early 2016 is in; in the figure cummax is the staircase of record highs the price keeps returning to.
Trace a rolling window
Advance the observation date and compare which values are included in the mean.
Read revealed values as a table
min_periods counts valid observations inside the window
A three-row window can contain fewer than three valid numbers. This damaged four-price excerpt makes the validity choice visible.
Strict windows produce no valid mean; relaxed windows produce two. The second rule accepts two observed prices, not three. Document the minimum coverage and inspect it before treating a rolling statistic as a feature.
Put the completed statistic on the decision row
The mean of the first three closes is 46.247. It first becomes usable on the fourth row under this before-close decision rule. After-close decisions have a different cutoff; the feature timestamp and execution rule must agree.
Drawdown
The drawdown at time \(t\) is how far the price sits below its running record high, as a fraction of that high:
\[ \text{drawdown}_t = \frac{\text{cummax}_t - P_t}{\text{cummax}_t}, \]
and the maximum drawdown is the worst peak-to-trough loss in the sample — the number an investor would have suffered had they bought at the worst possible moment and held to the bottom. Later chapters use it alongside volatility to describe risk; this example is preparation practice. Note that the formula needs the running maximum, cummax(), not the overall maximum max(): a peak reached after the trough cannot have been drawn down from.
The lecture notebook asks for the maximum drawdown of ten years of Apple. The cell runs and prints a drawdown — but the number is wrong, and so is the date. Find the bug before reading on.
Using the decade’s overall maximum—a 2024 close near 259—creates look-ahead and an apparent drawdown of about 0.91 at the 2016 low. aapl.cummax() uses only highs observed so far: maximum drawdown is 0.3873 on 3 January 2019, a 38.7% fall from 58.02 on 3 October 2018 to 35.55. Future information cannot determine a historical decision or peak.
expanding applies a statistic to all observations so far
min_periods requires a minimum valid count.
The means are missing, 3, 2.333… and 3. The running sums are 2, 6, 7 and 12. These summaries include the current observation; shift the completed statistic when the decision precedes that observation.
Custom rolling summaries
rolling(k).apply(function) passes each complete window to a function returning one scalar. Prefer a built-in statistic when it expresses the rule.
Both give the same Series result. A callback adds no benefit for a built-in median; keep it for a genuinely custom window rule.
Calendar bins and resampling
Question: What belongs in each time bin?
Result: A calendar summary with meaningful aggregation and coverage checks.
Resampling
Resampling groups a DatetimeIndex into calendar bins. resample("ME").mean() returns one mean per month; resample("W").last() returns the last observed value in each week with its Sunday boundary as the index. The boundary need not be an observation date. Rolling windows instead contain a declared number of recent rows and overlap.
Choose the summary to match the measurement: a mean describes a level, a last value gives the closing observation, and a sum can describe a flow such as sales or volume. Adding daily closing prices does not give a useful monthly price. Use count() to identify partial or poorly covered bins.
Monthly means begin 45.51, 43.08 and 42.13 for January–March 2015. There are 38 monthly bins. The first weekly counts are 1, 5 and 5: the first bin contains only Friday 2 January. Weekly closes start at 46.76 and 47.19, with index dates 4 and 11 January, with 163 weekly bins overall.
The following examples cover broadcasting full-month summaries back to daily rows, intraday close-based OHLC and up-sampling. A full-month statistic uses later days and therefore cannot be a feature known at the month’s start.
Check a calendar summary before using it
Inspect the bin index value, observation count and units together. A month-end index value does not prove complete coverage; an empty bin remains unknown. Chapter 2 extends this grouping logic to entity and date index values.
Connect the time-series operations
For Week 1, use chronological prices, observed returns, past-only rolling statistics, cumulative paths and monthly/weekly summaries. Intraday bar construction and inserting a finer time grid follow the calendar-bin examples before the recap.
agg collapses, transform broadcasts
| Tool | Purpose and return value |
|---|---|
transform |
Broadcast a group statistic back to each input row; a complete-month mean uses later days within that month. |
resample("ME").agg("mean") returns one row per month. .transform("mean") returns one row per day, each carrying its month’s mean — so you can subtract it.
ax.step(...) draws the repeated monthly mean as a staircase.
38 rows versus 779. The step line is the monthly mean broadcast back to every day — the same idea as groupby().transform in Chapter 2. The rolling mean is the smooth cousin of the same idea.
Trace calendar bins: aggregation and transformation
Daily units are grouped by calendar month. agg reduces each group; transform retains every date.
Read revealed values as a table
Intraday: 390 one-minute bars → five-minute bars
| Tool | Purpose and return value |
|---|---|
ohlc |
Return first/max/min/last within each bin of the selected Series; close-only inputs cannot recover full intraminute highs/lows. |
Before running: Compare the first and last timestamps with the five-minute output. Are the first and last bins complete?
Not 78 but 79: bins are anchored on the clock (09:30, 09:35 …), so the first bar holds only 09:31–09:34 and the 16:00 print sits alone in the last one. .ohlc() summarises the selected close observations; full candles require the original open, high and low columns too; .sum() is right for volume, .last() for a close, .mean() for a level. The aggregation must match what the number means.
Bins are anchored on the clock (09:30, 09:35, …). The first bin holds only 09:31–09:34, and the 16:00 print falls alone into a bin of its own at the end. Partial first and last bins are the rule for every resample — check count() on both before trusting the aggregate.
Up-sampling invents rows — you decide how to fill them
resample("1min").asfreq() puts observed values on a one-minute grid. It introduces missing rows. .ffill() carries an earlier observed value into those rows under an explicit replacement policy; it does not create new measurements.
| Tool | Purpose and return value |
|---|---|
asfreq |
Return values on the new frequency grid, leaving new timestamps missing. |
Three 5-minute closes become eleven 1-minute rows, eight of them empty. ffill carries the last observed close forward. Those carried values are assumptions, not newly observed prices. Up-sampling adds rows, not measurements.
Section 4: concepts to retain
- Parse and sort timestamps;
.dtaccesses date values while a DatetimeIndex has date properties directly. shiftis a row lag;diffmeasures an amount;pct_changemeasures a fractional change.- Rolling counts recent observations; expanding uses the history so far; min_periods requires valid values.
- Shift completed statistics when their current observation is unavailable at the decision time.
- Compounding, drawdown and resampling need valid inputs, meaningful bins and declared fill/accounting rules.
shift(-1) puts tomorrow’s price on today’s row; its last entry is NaN because tomorrow is unknown. A model given it as an input has been shown the answer — look-ahead bias — and its backtest will be spectacular and worthless. The same sin, in a different costume, is computing drawdown with max() instead of cummax().
The product of gross returns telescopes: \(\prod_t P_t / P_{t-1} = P_T / P_0\), the growth of one dollar — 1.882 for Microsoft 2015–18, exactly the last close over the first. The cumulative sum of simple returns (0.711) ignores compounding and is not a quantity anyone earns.
Figures and Statistical Charts
A chart is the final step of a data transformation. Its inputs still need the index values, missing-value policy and denominator established in Sections 1.1–1.4. This lab develops the original Topic 1 plotting material around the same Microsoft prices and public-company sectors. Prepare it before class and use it to explain evidence in the Week 1 case; it does not add another lecture.
| Part | Question |
|---|---|
| 1.5.1 Figure and Axes foundations | Where will pandas draw the result? |
| 1.5.2 Distributions and paired observations | What does this chart represent? |
Figure and Axes foundations
Question: Where will pandas draw the result?
Result: A labelled chart on the intended Axes.
A Figure contains Axes; an Axes contains the chart
Load a sorted Series, calculate observed returns and draw into an explicitly named Axes.
The pyplot state interface, such as plt.title, acts on the current axes. That is convenient for one chart and ambiguous when several axes exist. Naming fig and ax makes the drawing destination explicit. A Matplotlib Axes is one plotting panel; its x-axis and y-axis are components of that panel. Price levels have units USD per share, whereas returns are dimensionless rates; the y-axis must say which you have drawn.
plt.subplots(...) returns a Figure and Axes. plot_close.plot(ax=ax) draws into that panel, ax.set(...) assigns the title and units, and fig.tight_layout() adjusts spacing to reduce index value overlap.
fig controls the whole canvas; ax controls this chart. plot_close.plot(ax=ax) uses the pandas index as the horizontal axis.
Trace Figure, Axes and chart labels
Distributions and paired observations
Question: What does this chart represent?
Result: A distribution or paired comparison with units and denominator stated.
Plotting a Series
Use one pattern: fig, ax = plt.subplots(...), draw on ax, index value the measurement with ax.set(...), then plt.show(). A line chart uses time or another meaningful ordered index. A histogram groups a numerical distribution into bins; a bar chart compares category counts.
Six observations; unequal bin widths. Density height × bin width gives the probability in a bin.
Read revealed values as a table
The total bar area is 1.0. The fixed-seed normal example is a declared simulation with 5,000 observations, not a claim that market returns are normal. Chapter 3 develops distributional assumptions. The following examples explain KDE, multiple panels, boxplots and pies using the same observations.
Healthcare has 500 companies, Technology 486 and Financial Services 447 in the archived sample. The labelled horizontal bars make close counts easier to compare. Missing category values, if present, are counted as Unknown so the population remains explicit.
A scatterplot needs paired observations
There are 778 complete pairs. The figure describes an association in this historical sample; it does not establish causality or an out-of-sample trading rule.
KDE smooths a distribution; it does not add observations
The histogram depends on bins; the KDE depends on bandwidth. Neither establishes normality.
Bars preserve category comparisons; pies require a complete whole
| Tool | Purpose and return value |
|---|---|
set_xlabel |
Name the horizontal axis and state its units. |
Count all sector index values. Collapse the remaining sectors into Other so the pie accounts for the complete sample.
The categories account for all 3,084 companies. Bars make close counts easier to compare; a top-five-only pie would show shares of those five sectors.
State the pie denominator
Method reference: purpose before use
Use this lookup alongside the worked examples, not as a memorisation list. Each method is introduced before its first executable use in the chapter slides. Dot attributes such as .shape, .dtype, .name and .index have no parentheses; methods do.
| Tool | Purpose and return value |
|---|---|
head |
Return the first n rows or entries: s.head(n). |
read_csv |
Read CSV into a DataFrame; usecols selects columns, index_col chooses index values, parse_dates=True parses a date index. |
equals |
Compare index values or whole objects in order and return one True/False answer. |
mean |
Observed arithmetic mean: sum divided by non-missing count. |
Series |
Construct one-dimensional data with an index: pd.Series(values, index=dates, name=“…”). |
arange |
np.arange(5) constructs positions 0–4; its stop is excluded. |
sort_index |
Return entries ordered by index values; dates must first be parsed. |
median |
Return the middle sorted observed value (average two middle values for even n). |
sum |
Add observed values; for a Boolean mask count True entries. |
add |
Index value-aligned addition; fill_value declares a replacement for missing partners. |
notna |
Return a Boolean Series: True where a value is present. |
isna |
Return a Boolean Series: True where a value is missing. |
dropna |
Return entries with missing values removed; length can shrink. |
between |
Return a mask for a value interval: s.between(a, b, inclusive=“both”). |
mode |
Return all most frequent values as a Series; iloc[0] explicitly chooses its first value. |
std |
Return standard deviation; pandas uses divisor n−1 unless ddof=0. |
max |
Return the largest observed value. |
min |
Return the smallest observed value. |
quantile |
Return a sample cut point at fraction q, with linear interpolation by default. |
agg |
Apply named or custom summaries; one summary gives a scalar, a list gives named statistics. |
idxmax |
Return the index value of the first maximum, rather than its value. |
describe |
Return count, mean, standard deviation, min, quartiles and max for numeric data. |
value_counts |
Count observed categories; normalize=True gives shares among counted entries. |
nunique |
Count distinct observed values; missing values are excluded by default. |
astype |
Return a Series with a requested dtype; it does not change the original. |
to_datetime |
Parse date text; supply format to resolve ambiguous layouts. A date Series uses the .dt accessor. |
day_name |
Return weekday names: date_series.dt.day_name() or datetime_index.day_name(). |
copy |
Make a separate data object before intentionally damaging or editing an example. |
to_numeric |
Parse numeric text; errors=“coerce” creates missing values for invalid text. |
count |
Count non-missing values; len(s) instead counts all entries. |
fillna |
Replace missing entries with a declared scalar or aligned values. |
perf_counter |
time.perf_counter() reads a high-resolution clock; subtract two readings for elapsed seconds. |
apply |
For this scalar function, call it on each value and retain its returned result and input index value. |
where |
Keep values where a condition is True; replace the others, with missing by default. |
mask |
Replace values where a condition is True; keep the others. |
pct_change |
Return previous-row fractional changes, P_t/P_(t−1)−1; fill_method=None keeps gaps. 0.01 means 1%. |
map |
Look up each value in a dictionary, or transform it with a function; keep input index values. |
replace |
Recode specified values; keep uncovered values unchanged. |
isin |
Return a mask testing membership in a supplied set/list of values. |
upper |
With .str.upper(), return upper-case text for each value. |
strip |
With .str.strip(), remove leading and trailing whitespace from each text value. |
all |
Reduce checks to one Boolean: did every retained check pass? Handle unknown checks explicitly. |
ffill |
Carry a preceding observed value forward in the current row order. |
isfinite |
np.isfinite(values) marks finite numbers; infinity and missing values are not finite. |
any |
Reduce checks to one Boolean: did at least one retained check pass? |
sort_values |
Return entries ordered by value; ascending=False puts larger values first. |
clip |
Cap/floor values at supplied bounds while retaining index values and length. |
drop_duplicates |
Remove repeated values; keep=“first”, “last”, or False selects the survivor policy. |
rank |
Assign ordered ranks; ties average by default, pct=True divides by observed count. |
cut |
pd.cut(values, bins) maps values to numerical intervals; integer bins uses equal widths. |
qcut |
pd.qcut(values, q) uses sample quantiles to seek roughly equal bin counts. |
rename |
Use a dictionary or function to relabel the index; rename(“text”) instead sets the Series name. |
reset_index |
Move Series index values into a DataFrame column; drop=True discards them and retains a Series. |
rename_axis |
Set the index name, rather than its individual index values. |
date |
Timestamp.date() returns its calendar date without the time for display. |
idxmin |
Return the index value of the first minimum, rather than its value. |
tail |
Return the last n rows or entries: s.tail(n). |
sample |
Draw n or frac entries; random_state fixes the draw, replace=True permits repeated draws. |
reindex |
Conform to requested index values and their order; absent index values create missing entries. |
duplicated |
Mark repeated values on a Series, or repeated index values on its index. |
align |
Return two objects with matching ordered index values; join=“inner” retains common index values. |
corr |
Return correlation of index value-matched, non-missing pairs; report their count. |
filter |
Select by index text: items lists index values, like matches a substring, regex matches a pattern. |
strftime |
Format datetimes into text; %d day, %m month, %y two-digit year, %a short weekday. |
DataFrame |
Construct a table; dictionary entry names become column names and Series values align by index values. |
shift |
Move values by row positions while retaining the index; positive periods uses earlier rows. |
allclose |
np.allclose(a, b) checks approximate numerical agreement in array order, not index value identity. |
diff |
Return current minus previous-row values, in the original units. |
rolling |
Define trailing row windows; an integer window defaults to requiring that many observed values. |
subplots |
fig, ax = plt.subplots(figsize=(width, height)) creates a Figure canvas and one Axes panel; sizes are inches. |
plot |
ax.plot(series, index value=…) draws on the named panel, using the Series index as its horizontal coordinate. |
legend |
Show index values assigned to chart elements. |
show |
Display the completed Matplotlib figure. |
sqrt |
np.sqrt(x) returns the square root; multiplying daily volatility by sqrt(252) assumes annual variance scales with trading time. |
cumprod |
Return running products; multiplying valid gross returns (1+r) compounds growth. |
cumsum |
Return running sums; summing simple returns does not compound them. |
cummax |
Return the maximum observed so far at each row. |
cummin |
Return the minimum observed so far at each row. |
expanding |
Define the entire observed prefix at each row; min_periods requires a minimum valid count. |
resample |
Group a DatetimeIndex into clock bins, then aggregate: ME month-end, QE quarter-end, W week ending Sunday. |
last |
Return the last observed value in each resampling bin. |
transform |
Broadcast a group statistic back to each input row; a complete-month mean uses later days within that month. |
step |
ax.step(x, y) draws a staircase between supplied points. |
ohlc |
Return first/max/min/last within each bin of the selected Series; close-only inputs cannot recover full intraminute highs/lows. |
asfreq |
Return values on the new frequency grid, leaving new timestamps missing. |
default_rng |
Create a NumPy random generator with a fixed seed for reproducible teaching draws. |
normal |
rng.normal(mean, sd, size) draws a simulated sample with the declared parameters. |
hist |
Draw a histogram; density=True makes total bar area one, not the sum of heights. |
kde |
Draw a kernel density estimate, smoothing the observed distribution; its shape depends on bandwidth. |
line |
s.plot.line(ax=ax) draws the Series values against index values on the chosen panel. |
tight_layout |
Adjust subplot spacing to reduce index value overlap. |
barh |
Draw category counts as horizontal bars on a chosen Axes. |
pie |
Draw supplied values as shares of their own total; omitted categories change the denominator. |
set |
ax.set(title=…, xlabel=…, ylabel=…) names the chart and its units. |
linspace |
np.linspace(start, stop, n) returns n equally spaced points, including both endpoints by default. |
abs |
Return absolute magnitudes; use to check both negative and positive extremes. |
concat |
pd.concat([…], axis=1) places Series side by side, matching index values into columns. |
scatter |
ax.scatter(x, y) plots paired observations; s sets marker area, alpha transparency. |
axhline |
Draw a horizontal reference line at the supplied y value. |
axvline |
Draw a vertical reference line at the supplied x value. |
set_ylabel |
Name the vertical axis and state its units. |
box |
Draw median, quartiles and whiskers; beyond-whisker points are flags, not proven errors. |
set_xlabel |
Name the horizontal axis and state its units. |
Section 5: concepts to retain
- A Figure contains Axes; send each plotting call to the intended panel.
- Choose line, histogram, scatter or bar to match the question and label the units.
- Pair scatter inputs by identity and remove incomplete pairs together.
- Histogram probability is area; shared bins make comparison clearer.
- Category counts need a declared population and an explicit missing-category policy.
Chapter Wrap-up
Chapters 0 and 1 prepare Week 1’s ungraded company projects: Harbour Wealth for L1 and Meridian Manufacturing for L2. Read the complete chapter and bring a compact note. Each 3.5-hour class uses ten group-discussion, reporting, lecture and individual-response cycles. Use your class-specific notebook for the team investigation; the instructor acts as an external consultant. Chapter practice examples support preparation and do not add another submission. The course records classroom marks in Weeks 2–7 under the outline’s 5% compact-note attendance, 15% individual participation, 20% group mini-project and 60% final exam scheme.
The chapter’s rules connect:
- A Series combines values, an index and a dtype. Arithmetic aligns index values; direct comparison needs matching indexes.
- Aggregations usually skip
NaN; arithmetic propagates it. Count missingness and state the denominator.std()defaults to \(n-1\); at \(n=5\) the two SD conventions differ by 11.8%. - Transformations such as
where,clip,rankandreplacepreserve indexed observations. Use one.locassignment to change the original. .locincludes an end index value;.ilocexcludes an end position. Verify the selected rows.shiftandrollingbuild dated features;cumprodcompounds price growth;cummaxidentifies historical peaks. Resampling needs an aggregation appropriate to the variable.
The professional habits are shorter still. Print len() and isna().sum() after every binary operation and every filter. Print the first and last bin after every resample. Test any slicing rule you are unsure of on a five-element Series before trusting it on a backtest. Never let shift(-1) or a global max() see the future. When an AI copilot proposes code, ask which index values, units, observed sample and decision-time information support its result. Prefer the familiar operations when they express the task clearly.
Chapter 2 puts many Series side by side. A DataFrame has two axes, and many table reductions use an axis= argument; alignment happens on rows and columns; groupby groups rows by column values or index levels; pivot, melt and concat reshape tables; Chapter 3 then develops distributions, hypothesis testing, association and extreme-value methods on top of that toolkit.
The companion slide deck for this chapter, with predict-then-run exercises for every cell above, is at statpython.pages.dev.
← Chapter 0: Basics of Python · Contents · Chapter 2: DataFrames →
Further reading
The pandas guides on indexing, conditional replacement and window calculations provide API details for the rules illustrated here.