{ "cells": [ { "cell_type": "markdown", "id": "1095db4d", "metadata": {}, "source": [ "# Optional 7.7: A Revisable Course Reference for Coding\n", "\n", "For ISOM5650 students who can use DataFrames and functions. This optional lab adds no required assignment. You will retrieve source-linked evidence, filter retired/unfinished examples, detect edits, rebuild a small index and test an adaptation.\n", "\n", "Run cells in order. Python with pandas and scikit-learn is required (Colab works). This lab uses six public cards, not the instructor index. TF–IDF is an interpretable retrieval baseline; the full local course tool additionally uses semantic embeddings and keyword search.\n", "\n", "1. Inspect the sources and rank candidates.\n", "2. Filter eligibility and test the selected pattern.\n", "3. Build a revision-aware index; correct, rebuild and search again.\n", "4. Practise filtering and describe a real-task test.\n" ] }, { "cell_type": "markdown", "id": "d11bc530", "metadata": {}, "source": [ "## 1. Inspect the collection\n", "The `source`, `status` and `revision` fields make each card traceable. Four are current; two are retired or unfinished." ] }, { "cell_type": "code", "execution_count": 1, "id": "1281dcad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " id status revision\n", "mutable-default-v1 retired 1\n", "mutable-default-v2 current 2\n", " validated-merge current 1\n", " past-features current 1\n", " group-transform current 1\n", "unfinished-example exercise 1\n" ] } ], "source": [ "import pandas as pd\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.metrics.pairwise import cosine_similarity\n", "rag_cards = pd.read_json(\"https://statpython.pages.dev/data/rag-reference-cards.json\")\n", "print(rag_cards[[\"id\", \"status\", \"revision\"]].to_string(index=False))\n" ] }, { "cell_type": "markdown", "id": "c4d47f98", "metadata": {}, "source": [ "## 2. Compare relevant and eligible evidence\n", "TF–IDF encodes weighted word counts. Cosine similarity measures shared weighted vocabulary; it is not a correctness probability. Inspect the status of every candidate." ] }, { "cell_type": "code", "execution_count": 2, "id": "ca0c50c6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " id status similarity\n", "unfinished-example exercise 0.423768\n", "mutable-default-v1 retired 0.314940\n", "mutable-default-v2 current 0.235668\n", " validated-merge current 0.000000\n", " past-features current 0.000000\n", " group-transform current 0.000000\n" ] } ], "source": [ "rag_query = \"independent calls default list\"\n", "rag_vectoriser = TfidfVectorizer()\n", "rag_matrix = rag_vectoriser.fit_transform(rag_cards[\"text\"])\n", "rag_query_vector = rag_vectoriser.transform([rag_query])\n", "rag_scores = cosine_similarity(rag_query_vector, rag_matrix).ravel()\n", "rag_ranked = rag_cards.assign(similarity=rag_scores)\n", "rag_ranked = rag_ranked.sort_values(\"similarity\", ascending=False)\n", "print(rag_ranked[[\"id\", \"status\", \"similarity\"]].to_string(index=False))\n" ] }, { "cell_type": "markdown", "id": "4aca7f58", "metadata": {}, "source": [ "Filter current cards before selecting evidence. Read the qualification about a supplied list, not just the code pattern." ] }, { "cell_type": "code", "execution_count": 3, "id": "3f394393", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " id revision similarity\n", "mutable-default-v2 2 0.235668\n", " validated-merge 1 0.000000\n", " past-features 1 0.000000\n", "Source: CourseSlides/topic0.qmd · Functions, the mutable-default trap\n", "Independent calls using a default list need a fresh object per call. Use def collect(value, items=None): then if items is None: items = []; then items.append(value); return items. None is a sentinel. A deliberately supplied list is still mutated; copy it if the task requires isolation.\n" ] } ], "source": [ "rag_current = rag_ranked.query(\"status == 'current'\").head(3)\n", "print(rag_current[[\"id\", \"revision\", \"similarity\"]].to_string(index=False))\n", "rag_selected = rag_current.iloc[0]\n", "print(\"Source:\", rag_selected.loc[\"source\"])\n", "print(rag_selected.loc[\"text\"])\n" ] }, { "cell_type": "markdown", "id": "fde03c27", "metadata": {}, "source": [ "## 3. Test the adaptation\n", "Two calls with no list supplied must be independent. A deliberately supplied list is mutated. These assertions test those precise requirements; change the function and tests if your task needs input isolation." ] }, { "cell_type": "code", "execution_count": 4, "id": "bde1768d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Independent defaults and supplied-list behaviour: PASS\n" ] } ], "source": [ "def collect(value, items=None):\n", " if items is None:\n", " items = []\n", " items.append(value)\n", " return items\n", "rag_first = collect(\"A\")\n", "rag_second = collect(\"B\")\n", "rag_supplied = [\"existing\"]\n", "rag_returned = collect(\"C\", rag_supplied)\n", "assert rag_first == [\"A\"] and rag_second == [\"B\"]\n", "assert rag_returned is rag_supplied and rag_supplied == [\"existing\", \"C\"]\n", "print(\"Independent defaults and supplied-list behaviour: PASS\")\n" ] }, { "cell_type": "markdown", "id": "01fc0247", "metadata": {}, "source": [ "## 4. A small index that refuses stale sources\n", "The following functions contain necessary state checks. `build_demo_index` copies current sources and fits their vectors. `search_demo_index` checks the complete source fingerprint before using that snapshot. An edit or deletion requires rebuilding. Retrieved text is evidence; it is never executed." ] }, { "cell_type": "code", "execution_count": 5, "id": "69d3b9d7", "metadata": {}, "outputs": [], "source": [ "from hashlib import sha256\n", "\n", "def source_fingerprint(frame):\n", " return sha256(frame.to_json(orient=\"records\").encode()).hexdigest()\n", "\n", "def build_demo_index(frame):\n", " eligible = frame.query(\"status == 'current'\").copy()\n", " vectoriser = TfidfVectorizer()\n", " matrix = vectoriser.fit_transform(eligible[\"text\"])\n", " return frame.copy(), eligible, vectoriser, matrix\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "6e4fab51", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idrevisionsource
1mutable-default-v22CourseSlides/topic0.qmd · Functions, the mutab...
2validated-merge1CourseSlides/topic2.qmd · DataFrame merge and ...
3past-features1CourseSlides/topic2.qmd · feature engineering;...
\n", "
" ], "text/plain": [ " id ... source\n", "1 mutable-default-v2 ... CourseSlides/topic0.qmd · Functions, the mutab...\n", "2 validated-merge ... CourseSlides/topic2.qmd · DataFrame merge and ...\n", "3 past-features ... CourseSlides/topic2.qmd · feature engineering;...\n", "\n", "[3 rows x 3 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def search_demo_index(index, sources, query, limit=3):\n", " snapshot, eligible, vectoriser, matrix = index\n", " if source_fingerprint(snapshot) != source_fingerprint(sources):\n", " raise ValueError(\"STALE_INDEX: rebuild before retrieving\")\n", " query_vector = vectoriser.transform([query])\n", " scores = cosine_similarity(query_vector, matrix).ravel()\n", " ranked = eligible.assign(similarity=scores)\n", " return ranked.sort_values(\"similarity\", ascending=False).head(limit)\n", "\n", "lab_sources = rag_cards.copy()\n", "lab_index = build_demo_index(lab_sources)\n", "search_demo_index(lab_index, lab_sources, rag_query)[[\"id\", \"revision\", \"source\"]]\n" ] }, { "cell_type": "markdown", "id": "7fc33397", "metadata": {}, "source": [ "## 5. Correct a source, then try an old index\n", "We add a clarification and increment that card's revision. The search must refuse the stale snapshot. The exception below is expected and is caught deliberately." ] }, { "cell_type": "code", "execution_count": 7, "id": "a4c2109d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "STALE_INDEX: rebuild before retrieving\n" ] } ], "source": [ "lab_sources.loc[lab_sources[\"id\"] == \"mutable-default-v2\", \"text\"] += \" Test two independent calls.\"\n", "lab_sources.loc[lab_sources[\"id\"] == \"mutable-default-v2\", \"revision\"] = 3\n", "try:\n", " search_demo_index(lab_index, lab_sources, rag_query)\n", "except ValueError as error:\n", " print(error)\n" ] }, { "cell_type": "markdown", "id": "b4beb30d", "metadata": {}, "source": [ "Rebuild from corrected sources and repeat the same query. The default-list card must now show revision 3. A full corpus tool also checks source files, new/deleted files and configuration/issue changes, then atomically switches the active revision." ] }, { "cell_type": "code", "execution_count": 8, "id": "a3d201ab", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idrevisionsource
1mutable-default-v23CourseSlides/topic0.qmd · Functions, the mutab...
2validated-merge1CourseSlides/topic2.qmd · DataFrame merge and ...
3past-features1CourseSlides/topic2.qmd · feature engineering;...
\n", "
" ], "text/plain": [ " id ... source\n", "1 mutable-default-v2 ... CourseSlides/topic0.qmd · Functions, the mutab...\n", "2 validated-merge ... CourseSlides/topic2.qmd · DataFrame merge and ...\n", "3 past-features ... CourseSlides/topic2.qmd · feature engineering;...\n", "\n", "[3 rows x 3 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lab_index = build_demo_index(lab_sources)\n", "lab_result = search_demo_index(lab_index, lab_sources, rag_query)\n", "assert lab_result.loc[lab_result[\"id\"] == \"mutable-default-v2\", \"revision\"].iloc[0] == 3\n", "lab_result[[\"id\", \"revision\", \"source\"]]\n" ] }, { "cell_type": "markdown", "id": "b974daa0", "metadata": {}, "source": [ "## 6. Exercise: current references only\n", "Complete the same filter as the slide practice. Predict the number of rows before running it. The next cell shows a worked answer for self-study." ] }, { "cell_type": "code", "execution_count": 9, "id": "cb57e8b3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Current-only check: False\n" ] } ], "source": [ "exercise_pool = rag_cards.copy() # Starter: replace with a status filter.\n", "print(\"Current-only check:\", len(exercise_pool) == 4 and exercise_pool[\"status\"].eq(\"current\").all())\n" ] }, { "cell_type": "markdown", "id": "9a121eb0", "metadata": {}, "source": [ "### Worked answer\n", "The status rule is independent of vocabulary similarity." ] }, { "cell_type": "code", "execution_count": 10, "id": "4b0eba04", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idstatus
1mutable-default-v2current
2validated-mergecurrent
3past-featurescurrent
4group-transformcurrent
\n", "
" ], "text/plain": [ " id status\n", "1 mutable-default-v2 current\n", "2 validated-merge current\n", "3 past-features current\n", "4 group-transform current" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exercise_pool = rag_cards.query(\"status == 'current'\").copy()\n", "assert len(exercise_pool) == 4 and exercise_pool[\"status\"].eq(\"current\").all()\n", "exercise_pool[[\"id\", \"status\"]]\n" ] }, { "cell_type": "markdown", "id": "f19cfe7f", "metadata": {}, "source": [ "## Transfer to a real task\n", "\n", "Write a query that includes your schema and decision time. For a merge, test duplicate keys, row count, missing-key policy and unmatched rows. For a time-series feature, test that the value only uses observations available at the declared prediction time. Ask the assistant to cite its retrieved source, explain assumptions and test its adaptation.\n", "\n", "A good retrieval score does not certify correct code. A retrieved course passage can itself contain an error, and a current API may differ from an older example. Consult current official documentation when needed.\n", "\n", "## Optional full local tool\n", "\n", "The instructor repository contains `KnowledgeBase/course_rag.py` and a read-only MCP server. It keeps canonical sources, revisions and private materials local during indexing. Use `sync` after editing/reviewing the originals; use `report` to quarantine an unresolved source. A cloud assistant may send selected retrieved context to its model provider. This public lab does not contain the private index or credentials.\n", "\n", "Further reading: [CodeRAG-Bench](https://aclanthology.org/2025.findings-naacl.176/), [Sentence Transformers](https://sbert.net/examples/sentence_transformer/applications/semantic-search/README.html), [SQLite FTS5](https://sqlite.org/fts5.html).\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" } }, "nbformat": 4, "nbformat_minor": 5 }