897 lines
64 KiB
Plaintext
897 lines
64 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "adFzzFsB-Ofl"
|
|
},
|
|
"source": [
|
|
"<h1>Chapter 3 - Looking Inside Transformer LLMs</h1>\n",
|
|
"<i>An extensive look into the transformer architecture of generative LLMs</i>\n",
|
|
"\n",
|
|
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961\"><img src=\"https://img.shields.io/badge/Buy%20the%20Book!-grey?logo=amazon\"></a>\n",
|
|
"<a href=\"https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/\"><img src=\"https://img.shields.io/badge/O'Reilly-white.svg?logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMzQiIGhlaWdodD0iMjciIHZpZXdCb3g9IjAgMCAzNCAyNyIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPGNpcmNsZSBjeD0iMTMiIGN5PSIxNCIgcj0iMTEiIHN0cm9rZT0iI0Q0MDEwMSIgc3Ryb2tlLXdpZHRoPSI0Ii8+CjxjaXJjbGUgY3g9IjMwLjUiIGN5PSIzLjUiIHI9IjMuNSIgZmlsbD0iI0Q0MDEwMSIvPgo8L3N2Zz4K\"></a>\n",
|
|
"<a href=\"https://github.com/HandsOnLLM/Hands-On-Large-Language-Models\"><img src=\"https://img.shields.io/badge/GitHub%20Repository-black?logo=github\"></a>\n",
|
|
"[](https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-Models/blob/main/chapter03/Chapter%203%20-%20Looking%20Inside%20LLMs.ipynb)\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"This notebook is for Chapter 3 of the [Hands-On Large Language Models](https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961) book by [Jay Alammar](https://www.linkedin.com/in/jalammar) and [Maarten Grootendorst](https://www.linkedin.com/in/mgrootendorst/).\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961\">\n",
|
|
"<img src=\"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/images/book_cover.png\" width=\"350\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### [OPTIONAL] - Installing Packages on <img src=\"https://colab.google/static/images/icons/colab.png\" width=100>\n",
|
|
"\n",
|
|
"If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to **uncomment and run** the following codeblock to install the dependencies for this chapter:\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"💡 **NOTE**: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to\n",
|
|
"**Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4**.\n",
|
|
"\n",
|
|
"---\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# %%capture\n",
|
|
"# !pip install transformers>=4.41.2 accelerate>=0.31.0"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "W_23Z_do-faF"
|
|
},
|
|
"source": [
|
|
"# Loading the LLM"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 759,
|
|
"referenced_widgets": [
|
|
"5ecafba0f8f04685a56d2b1495baea24",
|
|
"53945d03a26044878ddb7fc6eadfd8db",
|
|
"99db408011ea43c79adbd4a880839484",
|
|
"fd181746067945febc55f94e2dcf6f67",
|
|
"c5acd3e22bbe4a1897f7a12051e8eae9",
|
|
"446d084fde5f422a8c0525e8c5b47f93",
|
|
"6a0dabe02c874ecabd257c5da5f4a7c5",
|
|
"1ee8996717864f79af6cf0314cd27c59",
|
|
"7c719ce3694845b384ad4ae7207d31cc",
|
|
"989fc06dfb25420eaf155bfc0174c692",
|
|
"443850bc37d94a90aa037e73a77f9369",
|
|
"184bbd6daf424ddeba7f8cbe0b5b34d9",
|
|
"a6ee07658a234a54b38d849ff2017d6c",
|
|
"bf63e0a23fa147778dd88b48075611e3",
|
|
"acf8b6868c484885871964cc538359c6",
|
|
"32cc3b668de84e8a9f8cde25d3846822",
|
|
"b1115e5097c4404694e5a7c228e25234",
|
|
"6ca20198bc144510a14b795aaf813940",
|
|
"982a35743f51448cba583e54e7d3987d",
|
|
"17250e2791ba42099c50efa594e229ce",
|
|
"a4c438b7029f47d9a54d5a0ce16541c0",
|
|
"fafd3d7714d4466ea00f354b20dc954a",
|
|
"0a18c83d2645496797d74aef5e84dafa",
|
|
"bfbbea9912104cb1803c7c827c1cea7d",
|
|
"18c23f29824e4cb58e461bf89811a32a",
|
|
"74105769e6a541098769921b84d1f1cd",
|
|
"52109f76852e4388b6df4e5441c43a48",
|
|
"4f39386aea7c4ac9abf2d66291cbcb4e",
|
|
"4893546c2d6f4eec9cb5b7f6853013f0",
|
|
"88616761593a4b2b9365294427a3136d",
|
|
"d3ab3ad192f54d5b9f9e08db62884f0c",
|
|
"6be54d3b31864134bc36d3b9d997530a",
|
|
"27a1df02460948e19cbd40b69a89bec6",
|
|
"6b81bad9c639454980de2a67b414b988",
|
|
"1e8224a73a724058b28a13db2c2197c2",
|
|
"3cbbd1ae4e4d4bf1a34c9d960ee7d26d",
|
|
"e9c0309e4af44726b180c977692e2469",
|
|
"97421a19ce25438e846a84b45321f9d0",
|
|
"c6aa0ad56ce44b1eab1356d3bc56706b",
|
|
"87d0758fdaaa4d90b21b1291f4d20039",
|
|
"8c85660c9e9f4027bd64d596160a7d7f",
|
|
"fce8b8595211452f8a759a6e98410f6a",
|
|
"4249df3055d4427493e2a2e775dc1a93",
|
|
"e42e39002e66410b9d714656a632adea",
|
|
"3cf79ac9541c4b7bbf686b604ff73b81",
|
|
"b709537463c3485dbbcef93e3636af2d",
|
|
"cf7c24d7629b4362ae0905f8e4bbd997",
|
|
"d9f9f98cb9dd465daed1da06d7d4084b",
|
|
"cb20a4ccd9ea4548bc7f694068bb30f6",
|
|
"b91d534a7a5c4863a9e858cedbda9fd5",
|
|
"a65601c4918c4c2cb2020837cd1e1f85",
|
|
"99505d538b894a359bc3791cd95423d5",
|
|
"1a48245eddfc4923a5f267cd799ba9c7",
|
|
"6c50ddb227b4421da5cd391e4d6ec94c",
|
|
"7d62b5c9c90644378b0ca96cee430419",
|
|
"9ec56d025f8446a08184b055fe11598e",
|
|
"54894d44acce4fe6b0ef749d5c02e3cd",
|
|
"e2dd956536a0407fb9b9c4a01c01ba9c",
|
|
"54370883565644c5a2529e092db7f259",
|
|
"7ec24f1bff5f4ca78d7c36a637cfa294",
|
|
"36ff492b5c7c4ff3bcf72cc574d03a38",
|
|
"8279cee867884166bb09df4e02635e2a",
|
|
"a2644dee82b14cfbab069a484f3841e2",
|
|
"e761a9afe1b847579fb51eb0eddd4488",
|
|
"c59d9e04e5964a8891cc3ddce33d6f86",
|
|
"a9007e7552ab4634ac44577279b242ac",
|
|
"2cff15834149418c81eee5239a5e275a",
|
|
"6623f077d95f4faf883fa2ca4397169d",
|
|
"98529c941229460da8563b94d3419c37",
|
|
"3bef1bf002c94340a3323592d616e7ac",
|
|
"47c8fd6b7c9844d983014e07f1999cd2",
|
|
"aa24ef9c6e1d49708b7bb4755a9adb78",
|
|
"28261270cd0449f8ac85dc4b0efdac57",
|
|
"a231962de8584f69a6a107c275d1cda5",
|
|
"2d35dfe75a744987ab210f5fb0118301",
|
|
"1cbaa3bcf20b4af099f6d4310dd071dd",
|
|
"4f7088db853e47e6b6a6ccd654e379dc",
|
|
"7e0710dc5b5c4002b4bcc189bf5514cf",
|
|
"66dd13ca8234409eab15ecbf7a009914",
|
|
"8f021e2bfb7246248baa49b08f4d3358",
|
|
"0a91788829fc49ef95a69efd3256a8e3",
|
|
"b42a56f9b2e44b6992413e023ed44b0e",
|
|
"f9fc98e3d8ed4338bd763d152f8cc5f9",
|
|
"f786e0117562476697092ef828ceb1b2",
|
|
"07fed042c0894ca5aebe717eca6f3018",
|
|
"3bea08fef3ef456e9f180d8a23ded5bc",
|
|
"35f3727f37c44640986d8416141f8069",
|
|
"0dd1ce8a2306431c9b0412e5992f4f84",
|
|
"dcea31213c9f421abc7ffabc3499ecb9",
|
|
"61643fa2fcb54ebaad8da5106dec9ea0",
|
|
"0c4f8c58d213493494120c070d86ac76",
|
|
"e15e9978db794891b9fa0d8ce096c983",
|
|
"b15098d69b5f418da3f81aba8fb79de0",
|
|
"f7cf40042a2e4c8cb8b87444893d8ec1",
|
|
"6dcbc6ec46d44a55a1b38267201926b8",
|
|
"6de4dd6e36b444638b2f65e0ca80bc9a",
|
|
"60febb04a12447c192e1b8eb2aa5ba28",
|
|
"6d200e6b1bdc4e918093670df8e37dc8",
|
|
"934e950a3c0c48f08802d551ee1bd429",
|
|
"34058533e3cf46a88db8927372102b9f",
|
|
"2a7ba0f87814436386e66f4ef7f1111d",
|
|
"d3e41286c5b747a8bb5cf326f9f80ad3",
|
|
"fe92dc2d5e8c449bbb36dafbd6c9935f",
|
|
"615d4ec0fb194688a392b71b99cf3621",
|
|
"e2056e3dba884242807a98b9b3837843",
|
|
"6ca50fb814b64fccaf0e1c6c11d8f4d8",
|
|
"9ec1d173921748b2af19a3a21df9ed40",
|
|
"f8c3d566e98d47239ef2b823544b75a5",
|
|
"3618dcec35a740c485ecafa5589e0c91",
|
|
"f66dd730f3364e35974a3918d12ff51d",
|
|
"dcb9240335394bfa8d3949ef1cdbcdf8",
|
|
"2df88b2fd5e242eca8b1d3f6cea1349b",
|
|
"1b103c69baf74fceb551ebcb5a0ac5e8",
|
|
"6847b7b6b3854d6e9b72f40040f84c8f",
|
|
"ac3e67f03f604883ab4787930cd316f3",
|
|
"6d7e012a7fbf4d788054ead5020e9314",
|
|
"a432e2e32c7c4e56b138ebaefad76c93",
|
|
"352fe4ea215240149d73478e34cd9b66",
|
|
"bf9bc31d7f99477bb391f69df41b8dbe",
|
|
"41b8e463309e4013a50268677f44d4b5",
|
|
"989326b74cd146e3b5b2f2d5f19bdf41",
|
|
"36b7269d8eb849b084694fd1f3b177b9",
|
|
"2d09f9ed15fa452ba8d9ce9aba9f61ac",
|
|
"53818082d62749a28552a1eebd304d88",
|
|
"766f67cb6a8a4a58965edb671ce624e8",
|
|
"a86d22308dbf4c11bf3d6f6515aef561",
|
|
"e9e7d944715b402ab149f86862b92259",
|
|
"4aaa732bb1b94b4895ca3f00f93cd762",
|
|
"9fb02b3bbe79434a93f32291c208aaad",
|
|
"dd9b3a5e84ba44cb9717ded470c258b4",
|
|
"fa89524d446b480aa50d203d01ec7bb7",
|
|
"c8b77256d5fc436fbfdcc150843a6b5b",
|
|
"f5f5b592768048169676e09cca453645",
|
|
"77f40b8bf30c437ba987b71178d0e9f6",
|
|
"3f39ec300bd84852a2388dadaafd8c4b",
|
|
"250cac43e6da47dd8d732ea57d8c50ec",
|
|
"f929d12aad68458b98c21e0669da3d8e",
|
|
"44ddbcadcc4c477c80daf278122de46d",
|
|
"eb606db4125e4eb097d5b7d3cdb90976",
|
|
"b6c177de60b54edd887d1ca983ea7546",
|
|
"6db56d7c52244a3984a0638e060a81cb",
|
|
"11d17ae63dd44ecc8813d482ee17dd95",
|
|
"ff3733c6a1f34580b037e296e3abed6b",
|
|
"2d31d51641e945f695f7315b68e0ad2e",
|
|
"1b694930328e46bd9e0d61063b9141d4",
|
|
"b544b6f2c2bb4f36bf9a983005a8bdb8",
|
|
"afb8b0e602b649fcb92634c8aab4caf7",
|
|
"24fc703c916f43aab5900288b8aa5aca",
|
|
"16cffa93ab234718a8ede1044596e8b2",
|
|
"2d7999217424413d988cd29d41ed5ace",
|
|
"0d1546581c90418fa1cfc37491339134",
|
|
"0e233853a3b74211a0b65dcdd001feed",
|
|
"a1d2163af40a4aaab8a765b148845807",
|
|
"2aaa722b303b4e1f823cee828fc7958c"
|
|
]
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 130259,
|
|
"status": "ok",
|
|
"timestamp": 1718959891215,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "-5RLd6dI-Ytm",
|
|
"outputId": "fb085ff7-e06f-4142-8e95-5ff98b212e37"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
|
|
"The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
|
|
"To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
|
|
"You will be able to reuse this secret in all of your notebooks.\n",
|
|
"Please note that authentication is recommended but still optional to access public models or datasets.\n",
|
|
" warnings.warn(\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "5ecafba0f8f04685a56d2b1495baea24",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"tokenizer_config.json: 0%| | 0.00/3.17k [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "184bbd6daf424ddeba7f8cbe0b5b34d9",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"tokenizer.model: 0%| | 0.00/500k [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "0a18c83d2645496797d74aef5e84dafa",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"tokenizer.json: 0%| | 0.00/1.84M [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "6b81bad9c639454980de2a67b414b988",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"added_tokens.json: 0%| | 0.00/293 [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "3cf79ac9541c4b7bbf686b604ff73b81",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"special_tokens_map.json: 0%| | 0.00/568 [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "9ec56d025f8446a08184b055fe11598e",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"config.json: 0%| | 0.00/931 [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "2cff15834149418c81eee5239a5e275a",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"configuration_phi3.py: 0%| | 0.00/10.4k [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3-mini-4k-instruct:\n",
|
|
"- configuration_phi3.py\n",
|
|
". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "7e0710dc5b5c4002b4bcc189bf5514cf",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"modeling_phi3.py: 0%| | 0.00/73.8k [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3-mini-4k-instruct:\n",
|
|
"- modeling_phi3.py\n",
|
|
". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n",
|
|
"WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.ff07dc01615f8113924aed013115ab2abd32115b.modeling_phi3:`flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'.\n",
|
|
"WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.ff07dc01615f8113924aed013115ab2abd32115b.modeling_phi3:Current `flash-attention` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "dcea31213c9f421abc7ffabc3499ecb9",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"model.safetensors.index.json: 0%| | 0.00/16.3k [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "34058533e3cf46a88db8927372102b9f",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "dcb9240335394bfa8d3949ef1cdbcdf8",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"model-00001-of-00002.safetensors: 0%| | 0.00/4.97G [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "36b7269d8eb849b084694fd1f3b177b9",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"model-00002-of-00002.safetensors: 0%| | 0.00/2.67G [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "f5f5b592768048169676e09cca453645",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "2d31d51641e945f695f7315b68e0ad2e",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"generation_config.json: 0%| | 0.00/172 [00:00<?, ?B/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
|
|
"\n",
|
|
"# Load model and tokenizer\n",
|
|
"tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n",
|
|
"\n",
|
|
"model = AutoModelForCausalLM.from_pretrained(\n",
|
|
" \"microsoft/Phi-3-mini-4k-instruct\",\n",
|
|
" device_map=\"cuda\",\n",
|
|
" torch_dtype=\"auto\",\n",
|
|
" trust_remote_code=False,\n",
|
|
")\n",
|
|
"\n",
|
|
"# Create a pipeline\n",
|
|
"generator = pipeline(\n",
|
|
" \"text-generation\",\n",
|
|
" model=model,\n",
|
|
" tokenizer=tokenizer,\n",
|
|
" return_full_text=False,\n",
|
|
" max_new_tokens=50,\n",
|
|
" do_sample=False,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "REqcz-ID_XgV"
|
|
},
|
|
"source": [
|
|
"# The Inputs and Outputs of a Trained Transformer LLM\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 4955,
|
|
"status": "ok",
|
|
"timestamp": 1718959896168,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "17h6TPHluJ-i",
|
|
"outputId": "18727eeb-ccd6-40f8-aab1-25c8d9a03cbe"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.ff07dc01615f8113924aed013115ab2abd32115b.modeling_phi3:You are not running the flash-attention implementation, expect numerical differences.\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Solution 1:\n",
|
|
"\n",
|
|
"Subject: My Sincere Apologies for the Gardening Mishap\n",
|
|
"\n",
|
|
"\n",
|
|
"Dear Sarah,\n",
|
|
"\n",
|
|
"\n",
|
|
"I hope this message finds you well. I am writing to express my deep\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"prompt = \"Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.\"\n",
|
|
"\n",
|
|
"output = generator(prompt)\n",
|
|
"\n",
|
|
"print(output[0]['generated_text'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 1,
|
|
"status": "ok",
|
|
"timestamp": 1718959898745,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "eoFkdTd6_g5o",
|
|
"outputId": "bdcfde9f-28b7-4f43-ec0c-32c16677a776"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Phi3ForCausalLM(\n",
|
|
" (model): Phi3Model(\n",
|
|
" (embed_tokens): Embedding(32064, 3072, padding_idx=32000)\n",
|
|
" (embed_dropout): Dropout(p=0.0, inplace=False)\n",
|
|
" (layers): ModuleList(\n",
|
|
" (0-31): 32 x Phi3DecoderLayer(\n",
|
|
" (self_attn): Phi3Attention(\n",
|
|
" (o_proj): Linear(in_features=3072, out_features=3072, bias=False)\n",
|
|
" (qkv_proj): Linear(in_features=3072, out_features=9216, bias=False)\n",
|
|
" (rotary_emb): Phi3RotaryEmbedding()\n",
|
|
" )\n",
|
|
" (mlp): Phi3MLP(\n",
|
|
" (gate_up_proj): Linear(in_features=3072, out_features=16384, bias=False)\n",
|
|
" (down_proj): Linear(in_features=8192, out_features=3072, bias=False)\n",
|
|
" (activation_fn): SiLU()\n",
|
|
" )\n",
|
|
" (input_layernorm): Phi3RMSNorm()\n",
|
|
" (resid_attn_dropout): Dropout(p=0.0, inplace=False)\n",
|
|
" (resid_mlp_dropout): Dropout(p=0.0, inplace=False)\n",
|
|
" (post_attention_layernorm): Phi3RMSNorm()\n",
|
|
" )\n",
|
|
" )\n",
|
|
" (norm): Phi3RMSNorm()\n",
|
|
" )\n",
|
|
" (lm_head): Linear(in_features=3072, out_features=32064, bias=False)\n",
|
|
")\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(model)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "RTrwzB67BYVY"
|
|
},
|
|
"source": [
|
|
"# Choosing a single token from the probability distribution (sampling / decoding)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "sEcxYgJxBYbJ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"The capital of France is\"\n",
|
|
"\n",
|
|
"# Tokenize the input prompt\n",
|
|
"input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n",
|
|
"\n",
|
|
"# Tokenize the input prompt\n",
|
|
"input_ids = input_ids.to(\"cuda\")\n",
|
|
"\n",
|
|
"# Get the output of the model before the lm_head\n",
|
|
"model_output = model.model(input_ids)\n",
|
|
"\n",
|
|
"# Get the output of the lm_head\n",
|
|
"lm_head_output = model.lm_head(model_output[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 36
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 421,
|
|
"status": "ok",
|
|
"timestamp": 1718960391623,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "68YUSS4GBf9Q",
|
|
"outputId": "2dc25e8d-03b6-4bca-b46c-fec3e3a4a492"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"application/vnd.google.colaboratory.intrinsic+json": {
|
|
"type": "string"
|
|
},
|
|
"text/plain": [
|
|
"'Paris'"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"token_id = lm_head_output[0,-1].argmax(-1)\n",
|
|
"tokenizer.decode(token_id)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 901,
|
|
"status": "ok",
|
|
"timestamp": 1718960415287,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "cWWrfC5oBjwp",
|
|
"outputId": "c2fdeab7-e787-466f-88f4-988cd5f939a6"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"torch.Size([1, 6, 3072])"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"model_output[0].shape"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 1079,
|
|
"status": "ok",
|
|
"timestamp": 1718960424560,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "nC1PdOnTBnxZ",
|
|
"outputId": "1fd5f482-7046-4536-b745-4e681d6ecdaf"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"torch.Size([1, 6, 32064])"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"lm_head_output.shape"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Of2_rP4QBqrZ"
|
|
},
|
|
"source": [
|
|
"# Speeding up generation by caching keys and values\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "B0n6JhNHBrin"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"Write a very long email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.\"\n",
|
|
"\n",
|
|
"# Tokenize the input prompt\n",
|
|
"input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n",
|
|
"input_ids = input_ids.to(\"cuda\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 47155,
|
|
"status": "ok",
|
|
"timestamp": 1718960517928,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "BwIvt6jSByAF",
|
|
"outputId": "e71c4141-2ca3-488a-fdfb-8d9357af0125"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"6.66 s ┬▒ 2.22 s per loop (mean ┬▒ std. dev. of 7 runs, 1 loop each)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%%timeit -n 1\n",
|
|
"# Generate the text\n",
|
|
"generation_output = model.generate(\n",
|
|
" input_ids=input_ids,\n",
|
|
" max_new_tokens=100,\n",
|
|
" use_cache=True\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"executionInfo": {
|
|
"elapsed": 152674,
|
|
"status": "ok",
|
|
"timestamp": 1718960670601,
|
|
"user": {
|
|
"displayName": "Maarten Grootendorst",
|
|
"userId": "11015108362723620659"
|
|
},
|
|
"user_tz": -120
|
|
},
|
|
"id": "dFb1dcvJByCW",
|
|
"outputId": "0aba6a01-9bc7-40b7-e2e1-e064f13b4c88"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"21.9 s ┬▒ 94.6 ms per loop (mean ┬▒ std. dev. of 7 runs, 1 loop each)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%%timeit -n 1\n",
|
|
"# Generate the text\n",
|
|
"generation_output = model.generate(\n",
|
|
" input_ids=input_ids,\n",
|
|
" max_new_tokens=100,\n",
|
|
" use_cache=False\n",
|
|
")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"accelerator": "GPU",
|
|
"colab": {
|
|
"gpuType": "T4",
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.14"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|