{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "_A2SZPmbD4Pk" }, "source": [ "

Chapter 8 - Semantic Search and Retrieval-Augmented Generation

\n", "Exploring a vital part of LLMs, search.\n", "\n", "\n", "\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-Models/blob/main/chapter08/Chapter%208%20-%20Semantic%20Search.ipynb)\n", "\n", "---\n", "\n", "This notebook is for Chapter 8 of the [Hands-On Large Language Models](https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961) book by [Jay Alammar](https://www.linkedin.com/in/jalammar) and [Maarten Grootendorst](https://www.linkedin.com/in/mgrootendorst/).\n", "\n", "---\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [OPTIONAL] - Installing Packages on \n", "\n", "If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to **uncomment and run** the following codeblock to install the dependencies for this chapter:\n", "\n", "---\n", "\n", "💡 **NOTE**: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to\n", "**Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4**.\n", "\n", "---\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# %%capture\n", "# !pip install langchain==0.2.5 faiss-cpu==1.8.0 cohere==5.5.8 langchain-community==0.2.5 rank_bm25==0.2.2 sentence-transformers==3.0.1\n", "# !pip install llama-cpp-python==0.2.78 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124" ] }, { "cell_type": "markdown", "metadata": { "id": "Ye0HbBr3EV0P" }, "source": [ "# Dense Retrieval Example\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Svgdo3y3F741" }, "source": [ "## 1. Getting the text archive and chunking it\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uOFFg7YWFoaf" }, "outputs": [], "source": [ "import cohere\n", "\n", "# Paste your API key here. Remember to not share publicly\n", "api_key = ''\n", "\n", "# Create and retrieve a Cohere API key from os.cohere.ai\n", "co = cohere.Client(api_key)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_Dcq1j_xFxIr" }, "outputs": [], "source": [ "text = \"\"\"\n", "Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan.\n", "It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine.\n", "Set in a dystopian future where humanity is struggling to survive, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for mankind.\n", "\n", "Brothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007.\n", "Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar.\n", "Cinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm.\n", "Principal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles.\n", "Interstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects.\n", "\n", "Interstellar premiered on October 26, 2014, in Los Angeles.\n", "In the United States, it was first released on film stock, expanding to venues using digital projectors.\n", "The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014.\n", "It received acclaim for its performances, direction, screenplay, musical score, visual effects, ambition, themes, and emotional weight.\n", "It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics. Since its premiere, Interstellar gained a cult following,[5] and now is regarded by many sci-fi experts as one of the best science-fiction films of all time.\n", "Interstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades\"\"\"\n", "\n", "# Split into a list of sentences\n", "texts = text.split('.')\n", "\n", "# Clean up to remove empty spaces and new lines\n", "texts = [t.strip(' \\n') for t in texts]" ] }, { "cell_type": "markdown", "metadata": { "id": "krDDOpcZF5qo" }, "source": [ "## 2. Embedding the Text Chunks\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1117, "status": "ok", "timestamp": 1718963354789, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "xooetZg0Fz4K", "outputId": "1f105f8c-e6a9-4cc1-b620-1f9a03bec288" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(15, 4096)\n" ] } ], "source": [ "import numpy as np\n", "\n", "# Get the embeddings\n", "response = co.embed(\n", " texts=texts,\n", " input_type=\"search_document\",\n", ").embeddings\n", "\n", "embeds = np.array(response)\n", "print(embeds.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "fLVdFg1PF4GG" }, "source": [ "## 3. Building The Search Index\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JyqzN2-JF24N" }, "outputs": [], "source": [ "import faiss\n", "\n", "dim = embeds.shape[1]\n", "index = faiss.IndexFlatL2(dim)\n", "index.add(np.float32(embeds))" ] }, { "cell_type": "markdown", "metadata": { "id": "T6qRFo8dGGrJ" }, "source": [ "## 4. Search the index\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "o83pxM5sGHxp" }, "outputs": [], "source": [ "import pandas as pd\n", "\n", "def search(query, number_of_results=3):\n", "\n", " # 1. Get the query's embedding\n", " query_embed = co.embed(texts=[query],\n", " input_type=\"search_query\",).embeddings[0]\n", "\n", " # 2. Retrieve the nearest neighbors\n", " distances , similar_item_ids = index.search(np.float32([query_embed]), number_of_results)\n", "\n", " # 3. Format the results\n", " texts_np = np.array(texts) # Convert texts list to numpy for easier indexing\n", " results = pd.DataFrame(data={'texts': texts_np[similar_item_ids[0]],\n", " 'distance': distances[0]})\n", "\n", " # 4. Print and return the results\n", " print(f\"Query:'{query}'\\nNearest neighbors:\")\n", " return results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 180 }, "executionInfo": { "elapsed": 403, "status": "ok", "timestamp": 1718963357199, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "Rq_2knm_GLgR", "outputId": "eaee1a34-a690-4f6c-ecb1-53974ae6f319" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query:'how precise was the science'\n", "Nearest neighbors:\n" ] }, { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"results\",\n \"rows\": 3,\n \"fields\": [\n {\n \"column\": \"texts\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics\",\n \"Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\",\n \"Interstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distance\",\n \"properties\": {\n \"dtype\": \"float32\",\n \"num_unique_values\": 3,\n \"samples\": [\n 10757.3798828125,\n 11566.1318359375,\n 11922.8330078125\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "results" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textsdistance
0It has also received praise from many astronom...10757.379883
1Caltech theoretical physicist and 2017 Nobel l...11566.131836
2Interstellar uses extensive practical and mini...11922.833008
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "text/plain": [ " texts distance\n", "0 It has also received praise from many astronom... 10757.379883\n", "1 Caltech theoretical physicist and 2017 Nobel l... 11566.131836\n", "2 Interstellar uses extensive practical and mini... 11922.833008" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "query = \"how precise was the science\"\n", "results = search(query)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "EkkDh12ZGRhY" }, "outputs": [], "source": [ "from rank_bm25 import BM25Okapi\n", "from sklearn.feature_extraction import _stop_words\n", "import string\n", "\n", "def bm25_tokenizer(text):\n", " tokenized_doc = []\n", " for token in text.lower().split():\n", " token = token.strip(string.punctuation)\n", "\n", " if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:\n", " tokenized_doc.append(token)\n", " return tokenized_doc" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 4, "status": "ok", "timestamp": 1718963358455, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "cHl8HnvgGXHG", "outputId": "0defa2a0-8e6b-436b-f6d2-6c5ce94f1636" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 15/15 [00:00<00:00, 38908.20it/s]\n" ] } ], "source": [ "from tqdm import tqdm\n", "\n", "tokenized_corpus = []\n", "for passage in tqdm(texts):\n", " tokenized_corpus.append(bm25_tokenizer(passage))\n", "\n", "bm25 = BM25Okapi(tokenized_corpus)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZlyGXye4GRj0" }, "outputs": [], "source": [ "def keyword_search(query, top_k=3, num_candidates=15):\n", " print(\"Input question:\", query)\n", "\n", " ##### BM25 search (lexical search) #####\n", " bm25_scores = bm25.get_scores(bm25_tokenizer(query))\n", " top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]\n", " bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]\n", " bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)\n", "\n", " print(f\"Top-3 lexical search (BM25) hits\")\n", " for hit in bm25_hits[0:top_k]:\n", " print(\"\\t{:.3f}\\t{}\".format(hit['score'], texts[hit['corpus_id']].replace(\"\\n\", \" \")))\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 3, "status": "ok", "timestamp": 1718963358455, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "jV-V_mhRGRmS", "outputId": "0957579e-0de7-4646-949e-cd9750178dcf" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input question: how precise was the science\n", "Top-3 lexical search (BM25) hits\n", "\t1.789\tInterstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan\n", "\t1.373\tCaltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n", "\t0.000\tIt stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine\n" ] } ], "source": [ "keyword_search(query = \"how precise was the science\")" ] }, { "cell_type": "markdown", "metadata": { "id": "ehyhfd7NG5kw" }, "source": [ "## Caveats of Dense Retrieval\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 180 }, "executionInfo": { "elapsed": 280, "status": "ok", "timestamp": 1718963358733, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "NxYwEfYRGpNe", "outputId": "dfaf50a0-f500-4160-8126-1b3a825fe750" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query:'What is the mass of the moon?'\n", "Nearest neighbors:\n" ] }, { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"results\",\n \"rows\": 3,\n \"fields\": [\n {\n \"column\": \"texts\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"Cinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm\",\n \"The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014\",\n \"It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distance\",\n \"properties\": {\n \"dtype\": \"float32\",\n \"num_unique_values\": 3,\n \"samples\": [\n 12854.458984375,\n 13301.0302734375,\n 13332.01171875\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "results" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textsdistance
0Cinematographer Hoyte van Hoytema shot it on 3...12854.458984
1The film had a worldwide gross over $677 milli...13301.030273
2It has also received praise from many astronom...13332.011719
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "text/plain": [ " texts distance\n", "0 Cinematographer Hoyte van Hoytema shot it on 3... 12854.458984\n", "1 The film had a worldwide gross over $677 milli... 13301.030273\n", "2 It has also received praise from many astronom... 13332.011719" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "query = \"What is the mass of the moon?\"\n", "results = search(query)\n", "results" ] }, { "cell_type": "markdown", "metadata": { "id": "V_RalLmuG0jw" }, "source": [ "# Reranking Example\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 3, "status": "ok", "timestamp": 1718963358733, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "HulOxkW_Focv", "outputId": "06e05541-4383-452e-d41a-04c3dc5521fb" }, "outputs": [ { "data": { "text/plain": [ "[RerankResponseResultsItem(document=RerankResponseResultsItemDocument(text='It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics'), index=12, relevance_score=0.1698185),\n", " RerankResponseResultsItem(document=RerankResponseResultsItemDocument(text='The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014'), index=10, relevance_score=0.07004896),\n", " RerankResponseResultsItem(document=RerankResponseResultsItemDocument(text='Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar'), index=4, relevance_score=0.0043994132)]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "query = \"how precise was the science\"\n", "results = co.rerank(query=query, documents=texts, top_n=3, return_documents=True)\n", "results.results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 2, "status": "ok", "timestamp": 1718963358733, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "SUrmMW8LFofP", "outputId": "9724bf94-cf9d-45ff-eb80-50ededa34275" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 0.1698185 It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics\n", "1 0.07004896 The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014\n", "2 0.0043994132 Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n" ] } ], "source": [ "for idx, result in enumerate(results.results):\n", " print(idx, result.relevance_score , result.document.text)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rqYJaq2CFohv" }, "outputs": [], "source": [ "def keyword_and_reranking_search(query, top_k=3, num_candidates=10):\n", " print(\"Input question:\", query)\n", "\n", " ##### BM25 search (lexical search) #####\n", " bm25_scores = bm25.get_scores(bm25_tokenizer(query))\n", " top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]\n", " bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]\n", " bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)\n", "\n", " print(f\"Top-3 lexical search (BM25) hits\")\n", " for hit in bm25_hits[0:top_k]:\n", " print(\"\\t{:.3f}\\t{}\".format(hit['score'], texts[hit['corpus_id']].replace(\"\\n\", \" \")))\n", "\n", " #Add re-ranking\n", " docs = [texts[hit['corpus_id']] for hit in bm25_hits]\n", "\n", " print(f\"\\nTop-3 hits by rank-API ({len(bm25_hits)} BM25 hits re-ranked)\")\n", " results = co.rerank(query=query, documents=docs, top_n=top_k, return_documents=True)\n", " for hit in results.results:\n", " print(\"\\t{:.3f}\\t{}\".format(hit.relevance_score, hit.document.text.replace(\"\\n\", \" \")))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1, "status": "ok", "timestamp": 1718963359073, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "9FITOXqkHONy", "outputId": "c4e81e12-b19c-4fa0-8ce6-7907002df7ca" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input question: how precise was the science\n", "Top-3 lexical search (BM25) hits\n", "\t1.789\tInterstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan\n", "\t1.373\tCaltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n", "\t0.000\tInterstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects\n", "\n", "Top-3 hits by rank-API (10 BM25 hits re-ranked)\n", "\t0.004\tCaltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar\n", "\t0.004\tSet in a dystopian future where humanity is struggling to survive, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for mankind\n", "\t0.003\tBrothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007\n" ] } ], "source": [ "keyword_and_reranking_search(query = \"how precise was the science\")" ] }, { "cell_type": "markdown", "metadata": { "id": "ugdnTs_VHV25" }, "source": [ "# Retrieval-Augmented Generation" ] }, { "cell_type": "markdown", "metadata": { "id": "-iqKQ7F0HZh-" }, "source": [ "## Example: Grounded Generation with an LLM API\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 2275, "status": "ok", "timestamp": 1718963362077, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "VeHX0D8DHaim", "outputId": "364b75ea-0b36-4a01-b16f-aa407768dde8" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query:'income generated'\n", "Nearest neighbors:\n", "The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.\n" ] } ], "source": [ "query = \"income generated\"\n", "\n", "# 1- Retrieval\n", "# We'll use embedding search. But ideally we'd do hybrid\n", "results = search(query)\n", "\n", "# 2- Grounded Generation\n", "docs_dict = [{'text': text} for text in results['texts']]\n", "response = co.chat(\n", " message = query,\n", " documents=docs_dict\n", ")\n", "\n", "print(response.text)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 362, "status": "ok", "timestamp": 1718963362438, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "E9YmHEOHHpUW", "outputId": "f8670a77-8f69-4870-8887-060fdf195978" }, "outputs": [ { "data": { "text/plain": [ "NonStreamedChatResponse(text='The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.', generation_id='bebc32bd-d620-42cf-bd13-f8d1f96d4aa6', citations=[ChatCitation(start=21, end=57, text='worldwide gross of over $677 million', document_ids=['doc_0']), ChatCitation(start=62, end=103, text='$773 million with subsequent re-releases.', document_ids=['doc_0'])], documents=[{'id': 'doc_0', 'text': 'The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014'}], is_search_required=None, search_queries=None, search_results=None, finish_reason='COMPLETE', tool_calls=None, chat_history=[Message_User(message='income generated', tool_calls=None, role='USER'), Message_Chatbot(message='The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases.', tool_calls=None, role='CHATBOT')], prompt=None, meta=ApiMeta(api_version=ApiMetaApiVersion(version='1', is_deprecated=None, is_experimental=None), billed_units=ApiMetaBilledUnits(input_tokens=106, output_tokens=26, search_units=None, classifications=None), tokens=ApiMetaTokens(input_tokens=797, output_tokens=95), warnings=None), response_id='319002db-505d-4d3c-903f-41eefbf0f856')" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 2, "status": "ok", "timestamp": 1718963362438, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "YLwaGXM2Hg7b", "outputId": "03f052d6-0723-45d9-8b24-fbc1195664c4" }, "outputs": [ { "data": { "text/plain": [ "[ChatCitation(start=21, end=57, text='worldwide gross of over $677 million', document_ids=['doc_0']),\n", " ChatCitation(start=62, end=103, text='$773 million with subsequent re-releases.', document_ids=['doc_0'])]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response.citations" ] }, { "cell_type": "markdown", "metadata": { "id": "D_25ztzEHuWX" }, "source": [ "## Example: RAG with Local Models\n" ] }, { "cell_type": "markdown", "metadata": { "id": "jNZ5gUoWIYhp" }, "source": [ "### Loading the Generation Model\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 33056, "status": "ok", "timestamp": 1718963395761, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "E4LNwOWTHvOv", "outputId": "945a6fa3-511d-48b7-d305-fe6d1dd23be9" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2024-06-21 09:49:22-- https://huggingface.co/lmstudio-community/Phi-3-mini-4k-instruct-GGUF/resolve/main/Phi-3-mini-4k-instruct-Q8_0.gguf\n", "Resolving huggingface.co (huggingface.co)... 18.164.174.118, 18.164.174.23, 18.164.174.17, ...\n", "Connecting to huggingface.co (huggingface.co)|18.164.174.118|:443... connected.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://cdn-lfs-us-1.huggingface.co/repos/8e/3f/8e3fafa0351929e621a3db9a53b131a9d7f4b222332208032555bb92f11ab100/8d2f3732e31c354e169cd81dcde9807a1c73b85b9a0f9b16c19013e7a4bb151c?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-Q8_0.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-Q8_0.gguf%22%3B&Expires=1719222562&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxOTIyMjU2Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhlLzNmLzhlM2ZhZmEwMzUxOTI5ZTYyMWEzZGI5YTUzYjEzMWE5ZDdmNGIyMjIzMzIyMDgwMzI1NTViYjkyZjExYWIxMDAvOGQyZjM3MzJlMzFjMzU0ZTE2OWNkODFkY2RlOTgwN2ExYzczYjg1YjlhMGY5YjE2YzE5MDEzZTdhNGJiMTUxYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=DFqGVAGZBHhpPlWt2pSH5nbTgskxr6FrSK19FhopbgIYnqujHL3LDgnleMTDHrtRvflyX-6QE88ZAYZc1wIZA7ZXgBVoFdYMNPbHixfv4hqGdKiddWcF4QYi5JCYChS3Z8oZPUkcbyX6KNTqMR1nls2KTZ3K0Xl3E7nGmlTXo85mRdRKQojZLkuLOa28pG2z9jrs1wJ1B2W3Ed%7E%7EK1E-BXhjKsK4zUR1Ch-3KfqAqe0q0XmnCcF1Ml2xosujvlA%7EZuVxflmRf8wRVZBZsbZNXaUFAb3oiyNTuyr9g1fvSxJPmAJocs9jIMUZwU88Sa4s%7EVdfytD0s6YruNH1GOAfoA__&Key-Pair-Id=K2FPYV99P2N66Q [following]\n", "--2024-06-21 09:49:22-- https://cdn-lfs-us-1.huggingface.co/repos/8e/3f/8e3fafa0351929e621a3db9a53b131a9d7f4b222332208032555bb92f11ab100/8d2f3732e31c354e169cd81dcde9807a1c73b85b9a0f9b16c19013e7a4bb151c?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-Q8_0.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-Q8_0.gguf%22%3B&Expires=1719222562&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxOTIyMjU2Mn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzhlLzNmLzhlM2ZhZmEwMzUxOTI5ZTYyMWEzZGI5YTUzYjEzMWE5ZDdmNGIyMjIzMzIyMDgwMzI1NTViYjkyZjExYWIxMDAvOGQyZjM3MzJlMzFjMzU0ZTE2OWNkODFkY2RlOTgwN2ExYzczYjg1YjlhMGY5YjE2YzE5MDEzZTdhNGJiMTUxYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=DFqGVAGZBHhpPlWt2pSH5nbTgskxr6FrSK19FhopbgIYnqujHL3LDgnleMTDHrtRvflyX-6QE88ZAYZc1wIZA7ZXgBVoFdYMNPbHixfv4hqGdKiddWcF4QYi5JCYChS3Z8oZPUkcbyX6KNTqMR1nls2KTZ3K0Xl3E7nGmlTXo85mRdRKQojZLkuLOa28pG2z9jrs1wJ1B2W3Ed%7E%7EK1E-BXhjKsK4zUR1Ch-3KfqAqe0q0XmnCcF1Ml2xosujvlA%7EZuVxflmRf8wRVZBZsbZNXaUFAb3oiyNTuyr9g1fvSxJPmAJocs9jIMUZwU88Sa4s%7EVdfytD0s6YruNH1GOAfoA__&Key-Pair-Id=K2FPYV99P2N66Q\n", "Resolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 18.65.25.71, 18.65.25.64, 18.65.25.113, ...\n", "Connecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|18.65.25.71|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 4061221024 (3.8G) [binary/octet-stream]\n", "Saving to: ‘Phi-3-mini-4k-instruct-Q8_0.gguf’\n", "\n", "Phi-3-mini-4k-instr 100%[===================>] 3.78G 66.7MB/s in 33s \n", "\n", "2024-06-21 09:49:55 (119 MB/s) - ‘Phi-3-mini-4k-instruct-Q8_0.gguf’ saved [4061221024/4061221024]\n", "\n" ] } ], "source": [ "!wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "a2Qgnc5OHvRQ" }, "outputs": [], "source": [ "from langchain import LlamaCpp\n", "\n", "# Make sure the model path is correct for your system!\n", "llm = LlamaCpp(\n", " model_path=\"Phi-3-mini-4k-instruct-q4.gguf\",\n", " n_gpu_layers=-1,\n", " max_tokens=500,\n", " n_ctx=2048,\n", " seed=42,\n", " verbose=False\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "H7ahQtlvIZjS" }, "source": [ "### Loading the Embedding Model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 539, "referenced_widgets": [ "138cc674d550408d92f2793bb8d8feef", "83f7665ab87042798e1b8e94ef3275f7", "3c73236c809f45e49293deb47b55de0a", "5d2bcf6e2b874522ae07d65d59bac90b", "6058adc08c9e43adb214dc157980ccd5", "94f720fd05bd42d2a3e80456516511ad", "05e80b09ac484cc5a580d8d23bab4727", "acd7c3193cd849d2927811bcc7c3ab86", "d0db0682e7bb48b3b195dec91dfceea5", "d189e49a470a4fe5ad2b8374cd1c3422", "bdacebab45304f47b732c33ac4639e9e", "5cff7f41f95d47dbb1c296647c967bfc", "ae492efb01234d4fbfb928087d05f094", "a453e6e3da5548d3a0d3336ba397e505", "4dc38e826b7f4982ba36b77a4bf2db2e", "255c1e532ee048e286e1d5c89fe3015d", "19b25464d66945a2a0e40284b0f278ec", "d7b688b70a2d48e3b20b7fa1d90e8e97", "7f75961fa15b443fb4d528116b925922", "b6a72b7b8e864691996256146e91ef48", "1adb2c1d93004a4aaec5842eb07dd5df", "89b93e5618f64fbca249f7f3a0eaae49", "a8e14699f76a49fc95bf572c33e89b9e", "ba3157d275b845edb92df2ebc4ea9bc1", "6a4f72cbd2b94be5af4d3ffcad6b1e87", "71815245763045838934063bd772d3ab", "0b8f820e5efe4e10b33e9693c3eef9bf", "e702233daf474196a5eb7a6b9500e75a", "db1c9245b86946ac9dde32c29604f215", "e61d6b43f4144e6eaa43716261a85516", "ef4669fd011241c2ab048399a5ad89c4", "5d9802b14082481e93aa3650e7a14cec", "c4de82afef0f40af8147f0d93f978174", "25d46b46b26a48d7a4810dd691dbfa86", "d3aee1810b1c40e19a184d2d6af8eefa", "715ed5ce1cd243d09d19ade841be1a7c", "1a8e6ef4564e43c3b4631223491d1138", "277df89d75b0440e99c7c38304de709d", "a95a3d6cff4f4b1fa9d716099789cb9f", "f366405bc2af4d5e97a179300fe267b9", "00fc8b78e0c2440d9b514378c6f95099", "68e451cf20454e86aec8160ae82f3670", "c74ad52fe8bb44da8d3b2a7b1b10c4f1", "a7a538d9ec814c9e8fd50eb15712de8f", "3b990fafeff24fe8adb29a3d24a89565", "e39c103ec551423aa126540b6afa7d10", "29e90bb4d1fa4c11a3a8d563bdbeb160", "82e4cb20b3724261b1c59d8841325bb9", "8cc39fab52804fa1867d6395a51900f9", "fe5b5da0ccad4bd2b5dedbbb25d0817d", "f94579b834ee41d9b12a0759e8206763", "525c9e8a0d834bbd83e23bc381cf3ecd", "7372effad89243fb8d7dbbd87fa3b413", "b935647ac06f4e20bfa62de8a1ddf05e", "681ea5b1d73b4c3580f4dcca25b9d9d0", "89c99f9a11164857a367d22af469cf85", "d4604e6f553941fab0d2c583fcbd593a", "eda119b60c004a1bb00b7ca0e583188d", "2b8114428e8d484a9c0c0638477a31d1", "ad192a0f6d9d42f49e3e24caaee42e26", "f9af1e04b2144d9f9b104cfa87455109", "368731d085ed4eb5bc1c78d73725df39", "430ff548ae1d4edeb14b91c12fbeb9dc", "3038cc2ac17f4a0397840b7104d2fd7c", "15d6c1ab1e0f4908b2a7584159bb0105", "65a247e214364094a8d6fcefdbce1c3a", "326669dae98648b2b55e6fcb45ed4f43", "30cee3a3d2684bd59ec661e67090b7ea", "196331e7f282454fb4ea32de0410a495", "6e5aa6453a5741aa9696ce0444b7fca4", "14f8aa41b5574d6f855ddcde71c1e59b", "4930e2e16041457580e15399bb983c34", "c3180ac3bc61416abc7fe10db1171bd9", "a1f95710663b4bdeb7b1c7dc6f4f4eb7", "6328742a36d54f619a1a7803b6be790e", "cd87abc413db4d65be32fd47389b46ed", "a78c6111af5541e0893f3775f3992a6a", "2d6c6545fc9e446f882e1b0eadd03ffc", "b574a5cb83164992a2dff90c17587acb", "05f0d4f09ba84f7287178b1dd43fdd25", "0b5c0e31175b4b2f9f1c1c3cdc15d34e", "31f1a648bf5d4c7baf88de71dcc6d3c1", "fc5cc6536106431d8515731c0e682acc", "653c8fe00d5d4996b155df26d0a0598d", "1eb0802651cb4560bacfb9be6e23f335", "8518599e1b1144fda9f208956f8f1eff", "ed0677d112e5469bb42068f0e3b14adf", "cb6d2cb161c347aab0b3a06727d776df", "e7fdfec1d8b341d7a1c228c069632400", "32d56ae437da4d9cbcdecc6942a54ce8", "23ed3163e67f4a10bc626b3440de7553", "53f46138cc2c4e74a3cbe1f77ac2f92e", "dac714e2ccba4635a1aceb46ebdc33ac", "b3001c1847394d08ae53cf68ec697e0f", "bb434d424fa144b0b0211603d30b5c72", "e4cb2ade41d4466d99149cee70c1e7e9", "dddb4584c34f40c6bc66df30c1c3c4c0", "23169e2b0cb34fa388a7c7f7a140335c", "c3aacf3d8c7b4b4b9b98ca123e087593", "44d38a0ba97444c6bfd80c8c9553c070", "dd7f4f080941498397dc51d459bbc519", "5c5c32897b19438dbba56f72c577fe64", "e049e3a92fbd422198c16a94e46a5455", "18588efdadbc457a8940facc0aa96511", "ae97a2366ac944d1a089b4f1d8269822", "23dc7b6255cc479c89799dc10653008c", "f19fe22a899e4fa19e607c16c94cf16c", "28f6bcec0d6448e18fb894e2dae4c2ae", "3fd02e49e5f64baf95596b82e1c5d9fb", "9e0b0f814ad6487fb5ee842e723a0b52" ] }, "executionInfo": { "elapsed": 23624, "status": "ok", "timestamp": 1718963519389, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "ODkBMgsIIddp", "outputId": "2ca552cb-565a-459b-e9eb-0aed4481d492" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/sentence_transformers/cross_encoder/CrossEncoder.py:11: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", " from tqdm.autonotebook import tqdm, trange\n", "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n", "The secret `HF_TOKEN` does not exist in your Colab secrets.\n", "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n", "You will be able to reuse this secret in all of your notebooks.\n", "Please note that authentication is recommended but still optional to access public models or datasets.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "138cc674d550408d92f2793bb8d8feef", "version_major": 2, "version_minor": 0 }, "text/plain": [ "modules.json: 0%| | 0.00/385 [00:00\n", "Relevant information:\n", "{context}\n", "\n", "Provide a concise answer the following question using the relevant information provided above:\n", "{question}<|end|>\n", "<|assistant|>\"\"\"\n", "prompt = PromptTemplate(\n", " template=template,\n", " input_variables=[\"context\", \"question\"]\n", ")\n", "\n", "# RAG Pipeline\n", "rag = RetrievalQA.from_chain_type(\n", " llm=llm,\n", " chain_type='stuff',\n", " retriever=db.as_retriever(),\n", " chain_type_kwargs={\n", " \"prompt\": prompt\n", " },\n", " verbose=True\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 90250, "status": "ok", "timestamp": 1718963614201, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "x2p2pJPfIp16", "outputId": "3d284ce5-d35d-429d-fcec-a6a4a427dc05" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n" ] }, { "data": { "text/plain": [ "{'query': 'Income generated',\n", " 'result': \" Interstellar grossed over $677 million worldwide in 2014 and had additional earnings from subsequent re-releases, totaling approximately $773 million. The film's release utilized both traditional film stock and digital projectors across various venues to maximize its income generation potential.\"}" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "rag.invoke('Income generated')" ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "T4", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 4 }