nandigama/Hands-On-Large-Language-Models
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4beb5e1f93fa81c3a75403dd4feea7c47000dbac
Hands-On-Large-Language-Models/chapter02/Chapter 2 - Tokens and Token Embeddings.ipynb
MaartenGr a1fe246954 Colab/Github rendering
2025-04-13 07:42:43 +02:00

3374 lines
125 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "g_a9QvUFVCUR"
},
"source": [
"<h1>Chapter 2 - Tokens and Token Embeddings</h1>\n",
"<i>Exploring tokens and embeddings as an integral part of building LLMs</i>\n",
"\n",
"\n",
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961\"><img src=\"https://img.shields.io/badge/Buy%20the%20Book!-grey?logo=amazon\"></a>\n",
"<a href=\"https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/\"><img src=\"https://img.shields.io/badge/O'Reilly-white.svg?logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMzQiIGhlaWdodD0iMjciIHZpZXdCb3g9IjAgMCAzNCAyNyIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPGNpcmNsZSBjeD0iMTMiIGN5PSIxNCIgcj0iMTEiIHN0cm9rZT0iI0Q0MDEwMSIgc3Ryb2tlLXdpZHRoPSI0Ii8+CjxjaXJjbGUgY3g9IjMwLjUiIGN5PSIzLjUiIHI9IjMuNSIgZmlsbD0iI0Q0MDEwMSIvPgo8L3N2Zz4K\"></a>\n",
"<a href=\"https://github.com/HandsOnLLM/Hands-On-Large-Language-Models\"><img src=\"https://img.shields.io/badge/GitHub%20Repository-black?logo=github\"></a>\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-Models/blob/main/chapter02/Chapter%202%20-%20Tokens%20and%20Token%20Embeddings.ipynb)\n",
"\n",
"---\n",
"\n",
"This notebook is for Chapter 2 of the [Hands-On Large Language Models](https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961) book by [Jay Alammar](https://www.linkedin.com/in/jalammar) and [Maarten Grootendorst](https://www.linkedin.com/in/mgrootendorst/).\n",
"\n",
"---\n",
"\n",
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961\">\n",
"<img src=\"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/images/book_cover.png\" width=\"350\"/></a>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [OPTIONAL] - Installing Packages on <img src=\"https://colab.google/static/images/icons/colab.png\" width=100>\n",
"\n",
"If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to **uncomment and run** the following codeblock to install the dependencies for this chapter:\n",
"\n",
"---\n",
"\n",
"💡 **NOTE**: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to\n",
"**Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4**.\n",
"\n",
"---"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# %%capture\n",
"# !pip install transformers>=4.41.2 sentence-transformers>=3.0.1 gensim>=4.3.2 scikit-learn>=1.5.0 accelerate>=0.31.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oQHfpqT_t9-K"
},
"source": [
"# Downloading and Running An LLM\n",
"\n",
"The first step is to load our model onto the GPU for faster inference. Note that we load the model and tokenizer separately and keep them as such so that we can explore them separately."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 753,
"referenced_widgets": [
"851b6e59cc2e4eb8961cb5fa4906c47c",
"fd5b6ec0a82c493a92a2235635ea5ac0",
"2bc5005713ba4b61a392935e1d83994c",
"607bb5b3f27d463dbdb5ad04583a1c4f",
"0d0c6c47fbb34090a73ed8bf20597ee5",
"083149fa68934cce90cd03b6c8f90dd4",
"73263751695843ada347c6c694afd0cd",
"79c1152a61dd4a87a6a4e700096d92b4",
"27fb45533f134c308cc0c27797c127f3",
"78eb9f6c02e841ad9ecf8ed809c724bb",
"1ade569c5cec4764af86e7eec6bc64ed",
"dbf8268e613e4e1498133a42a48a58f8",
"66c3d79c9411447eabae228b5d125c09",
"46ba964cc4c84bbb87eeccc817a3354c",
"150625fb48b740f6bce66fd4919357a9",
"974be84551a54deb82051b947ff013af",
"22e2cbe577e6475aa80dd94f4b704f1b",
"83daaa4a1a5d437e8345d4740c3625cd",
"6e1c0e41b31b4b0b83aa3736e386aa49",
"bf1d39ed8ee84a6b95e68a11962032b4",
"7476eb73c9544435b07256f028168a11",
"f33bc31f0cb842388057b33dd107f2e7",
"9b9f8a8f0eb14dd9a2c82461f22c4636",
"214eb5ac9a5048c586b3418452017991",
"dd1b7306ad4641799d4947a94aa0f088",
"d58e84330cff499c87e03827d9727d19",
"2809ac7769af482a86d5e763f5f69211",
"746054d4ac0b442e85b75959b66ceb34",
"9b77017c3e764c97b44c56fb5ad3bc77",
"3dc073c6257143748849311e1c08bdbe",
"3117fa7446394cb6b6b56427be0a3290",
"5c3c4612f65f40808f8ef1d0e31f9836",
"f66983763d454728b86d881621948b8d",
"a855727c923648ea8d6c7b9ec117c94f",
"6c9350120fec4f4e9b80e063279b459c",
"28283e52317e437c8e118d052d93419c",
"5556b8b2f22548109153146ef66702df",
"dc1457b4d1d344c9863fbb6b0d38e2cf",
"029a23c145604f209b12044a6b367802",
"252ccf08bd52433a888d2bac9ed1b64c",
"893af8dae5874b55a582f180652e36e3",
"27e14bb1e4aa467c97ac0c6a6eb44108",
"87812873ff2d4514aca02c64a3509bd9",
"13ecb25d0d6541b1a4f2b7dea92dee61",
"2107a7e7d40f416ca4bc46c90725e0ca",
"1cfc631c1d7d4fb087753e9e34fd1aa2",
"238f094ef6744b31ba4444d47c17558c",
"b11bf972b98645e79f98cdec2c1440f3",
"baa7c53a02cd42688d768a4595611f64",
"6a63559431934a4aae4ce79e8b2e76c0",
"81652b39152145748580c9667b2554de",
"4dc8d9b1a2d248f9a30bc8e985db4568",
"adfcbd50473c4ea9bb414e33926cc33b",
"52a9a5aed89a4294882c8c55b2078b84",
"83fb238d4eba47a390233ecb5e870ed4",
"8da9bab6ee214504a187e5b3c9bf1b80",
"77922a918c1b4bb08f5304a34044e9ba",
"ac36ee2d4df04399b11f57bb56930a52",
"df2f73dc04d64bf6adb4df0eae207c27",
"3cbc759c16b24d63971ba5ef53ff9ba3",
"ea5d84bf35754e71a628755c0cff49be",
"3ed30ef053324b3f9a5d3258d2a88a86",
"f8bae10ac5b54f1b99f7b595f123be4d",
"dfb5b5a357ca41f88efa53288ebeb5b3",
"b917827c07344e018543fcbb9c7a6fe8",
"bad199037f6548749f4dfe12724e3e62",
"0e30dc616d9c4c50adb9cd292fc3d89e",
"570a9d836e48456d8c301184d670c50e",
"471cdd69adc24003b14680e8dbe7b183",
"c11552c4f423434aa2c41dfa22b8f227",
"687c61c78b044690a91e34affe570613",
"82dcc51143ac46349c13d2f5bee380da",
"a9c8a9b099aa416d9015ee48006db324",
"1fcc7e4cd65449c49956981ec1b46843",
"5fc8310895a64815b15af53e3583f60d",
"ffa0950ea21a4689b5804b0285049ccd",
"da9ec5fd7acd43b7a11ac860771732df",
"961d4e8f3a1845e393e25d21de3cd6d3",
"e7506426cc424dfdb28fa6c35cb74c24",
"d9b26784fc7c4e6b8abe9490caf53afc",
"e96d58589fcc4bd7a469c4281baa01ee",
"dd8fa47d73774a7684cda73e6675c0bd",
"41843fb3c92c425b9f39cb124b74e3ad",
"2ab54f26cc504dc09db592039c02c210",
"cab531e67dd3452f9a4fcd17e324d76a",
"58de29a54c91401d92cc73a96721a3e9",
"0eabbb5ac0784a1c89ad58cca37a38fe",
"362724ce1c5944bd9335b66a14f89844",
"bd2749d8ba7240408be56e26c796e00a",
"b54e7ab6e22345bdae3484cab6dab985",
"77d3a6cc940a4dd7a38c47dfc2b7041b",
"13071c70f17144b891a1f0fde6d9b21e",
"205628f7d62d4e54add8ee90b418ebdf",
"037c74818af74266b2bce5a454b99ab8",
"bba33c0c581d415bba86aebfc8642196",
"294ee2f27b4c4ee1ad6ca2a121a9e66d",
"5bdf1b7758624e379c71da5296929973",
"6cae5555dce346dc9867edd130c840c9",
"0e37d9ed7d3142a3b80f188ddee54a4d",
"d077b2c0e6a142cb905e38a7c7855997",
"2a47328eace84efbb8fcf6644d4e469d",
"c1634b24795243f5b8384957ce11b9c0",
"b8285e7e6db348c385694d2fd63b514e",
"da9d77e8fa0641dc97f7610aa59374a2",
"7209a02303ad46f3880799f0b92171f0",
"770f06c5ff714c87af1d0d7536da12c9",
"f59c537df5684195b1b5c44754ed2d07",
"6dd8dd0a99bd4cbbb7401a3fa3893128",
"f8c4e2a7f11f47fcbd574ceaf828aa66",
"193e7cde826d43bd894362498a217888",
"febedf9dd6d248baa984ac215697968d",
"0bf43b40190141d5bf8e0be3fdb1e529",
"4070e2510e1549c3879e2458e42bb269",
"9ba44d309f684c98991889b992767ab3",
"b47b04d69b33402f80ea75ed472d3c29",
"19a091b24a5d44b9a98569a7e124c6f8",
"0d1cea03253c426b84814936c93e5279",
"7c1b3811b42549569cc83423bbeb2163",
"55bc2055b3c5426f8ba9afc9f80ccd58",
"61098d22dd294891afdd34b6208906ba",
"0a0db951df42424084b531ebbd6cfa98",
"3f1dc764fb2c48bd9fd8797a86b6c59f",
"95146eb05e964e32bfa6f5078a66fab7",
"8c7e09509c524cf29782df69988eef6d",
"5237587cfb9c4e86a82aaf72cc9db39d",
"22fd14ac3ccf4e1d8e4782d35e7ec91c",
"138a3cb6ba5b494c8a237af0c6306a4f",
"e88a1d50d7f940d793f4dcd76a5f5929",
"4182011b06304005a6e8922d338c65fb",
"16ae1f6d9d844d60b55da9e36a5792cd",
"56ea52f3a1ea485ca52eebaf4f97c3a3",
"c81c6fe3a8ed4466bf95c53dfe415fba",
"157b7540b5a3404e82d7e86b19906c7a",
"48d180eca04341d692d5fb83a8e0f76d",
"c84b966b98144e22884183cd2a03e2a8",
"9a6496f9037149289e64e66a0f46cbd3",
"ecd15644266b43678dbe7b067b553833",
"43a6de1c960341eda9e154d47e0142c9",
"6e8fa3e12a8b4262bbca73c57c7f4eb9",
"86445e2d62b44847a4be7419a6000305",
"3837338134c240caa0c51501fae402fb",
"5e2d4a77710e428ebef6bae0854c8527",
"63d81ace73634bcdaef6f5a103c35dcf",
"dd1af9490bc24d5ea2804adefd442fdb",
"cf18ae8dd5bb4d378f017f9eaff967b4",
"461fde63ac0a41ed8ad9aab2d60769a7",
"3c5bb3c7c77f4ca393dc05b12fb69dd0",
"bd707eacc9914dc69a1fe4e35320a9a4",
"d7cb5720fbbb4ae0858dd3070c712615",
"bf390e3a7172415089b2cce2223dd8b6",
"72ae6f3dcf3d4567a4a60f627a51f1f4",
"026141ba5e404e4e86c1d1c083ab186d",
"fcd5f49ec07f401fb8a2b403516bbd17",
"22d378959eb941c5bbfe77702fb426e6"
]
},
"executionInfo": {
"elapsed": 95520,
"status": "ok",
"timestamp": 1723034396041,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": -60
},
"id": "jjU8NBHnwA4j",
"outputId": "286bdccb-f25d-4b0e-bda3-44d2a4be45cd"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"`flash-attention` package not found, consider installing for better performance: No module named 'flash_attn'.\n",
"Current `flash-attention` does not support `window_size`. Either upgrade or use `attn_implementation='eager'`.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "dd45dd8837f94b38ae6f4ffd205d9ea6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
]
}
],
"source": [
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"\n",
"# Load model and tokenizer\n",
"model = AutoModelForCausalLM.from_pretrained(\n",
" \"microsoft/Phi-3-mini-4k-instruct\",\n",
" device_map=\"cuda\",\n",
" torch_dtype=\"auto\",\n",
" trust_remote_code=False,\n",
")\n",
"tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 5750,
"status": "ok",
"timestamp": 1719641447389,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "_iVl5yePuq3B",
"outputId": "4ce629bf-3897-4ab0-8cf1-8f55e2040155"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:transformers_modules.microsoft.Phi-3-mini-4k-instruct.ff07dc01615f8113924aed013115ab2abd32115b.modeling_phi3:You are not running the flash-attention implementation, expect numerical differences.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"<s> Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|> Subject: My Sincere Apologies for the Gardening Mishap\n",
"\n",
"Dear\n"
]
}
],
"source": [
"prompt = \"Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>\"\n",
"\n",
"# Tokenize the input prompt\n",
"input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(\"cuda\")\n",
"\n",
"# Generate the text\n",
"generation_output = model.generate(\n",
" input_ids=input_ids,\n",
" max_new_tokens=20\n",
")\n",
"\n",
"# Print the output\n",
"print(tokenizer.decode(generation_output[0]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 4,
"status": "ok",
"timestamp": 1719641447389,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "JmzgbbdKuvHt",
"outputId": "82511d5b-7949-49a0-e3a6-c128564575c8"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[ 1, 14350, 385, 4876, 27746, 5281, 304, 19235, 363, 278,\n",
" 25305, 293, 16423, 292, 286, 728, 481, 29889, 12027, 7420,\n",
" 920, 372, 9559, 29889, 32001]], device='cuda:0')\n"
]
}
],
"source": [
"print(input_ids)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1719641447389,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "W4vsjbxwu1K1",
"outputId": "506f32d1-f058-4cfd-a9cd-13c4dabe80e6"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<s>\n",
"Write\n",
"an\n",
"email\n",
"apolog\n",
"izing\n",
"to\n",
"Sarah\n",
"for\n",
"the\n",
"trag\n",
"ic\n",
"garden\n",
"ing\n",
"m\n",
"ish\n",
"ap\n",
".\n",
"Exp\n",
"lain\n",
"how\n",
"it\n",
"happened\n",
".\n",
"<|assistant|>\n"
]
}
],
"source": [
"for id in input_ids[0]:\n",
" print(tokenizer.decode(id))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1719641447389,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "A9wRZ3J3u4z1",
"outputId": "7efaa49c-7a5a-41d7-f000-7aace16007e5"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[ 1, 14350, 385, 4876, 27746, 5281, 304, 19235, 363, 278,\n",
" 25305, 293, 16423, 292, 286, 728, 481, 29889, 12027, 7420,\n",
" 920, 372, 9559, 29889, 32001, 3323, 622, 29901, 1619, 317,\n",
" 3742, 406, 6225, 11763, 363, 278, 19906, 292, 341, 728,\n",
" 481, 13, 13, 29928, 799]], device='cuda:0')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"generation_output"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 275,
"status": "ok",
"timestamp": 1723034447362,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": -60
},
"id": "7QlHLof3u8A3",
"outputId": "c2315e1b-91b4-4a1b-9bcc-084f16ac8db1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sub\n",
"ject\n",
"Subject\n",
":\n"
]
}
],
"source": [
"print(tokenizer.decode(3323))\n",
"print(tokenizer.decode(622))\n",
"print(tokenizer.decode([3323, 622]))\n",
"print(tokenizer.decode(29901))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T9nRducW48bd"
},
"source": [
"# Comparing Trained LLM Tokenizers\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7W0xFIVo5A0S"
},
"outputs": [],
"source": [
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"\n",
"colors_list = [\n",
" '102;194;165', '252;141;98', '141;160;203',\n",
" '231;138;195', '166;216;84', '255;217;47'\n",
"]\n",
"\n",
"def show_tokens(sentence, tokenizer_name):\n",
" tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n",
" token_ids = tokenizer(sentence).input_ids\n",
" for idx, t in enumerate(token_ids):\n",
" print(\n",
" f'\\x1b[0;30;48;2;{colors_list[idx % len(colors_list)]}m' +\n",
" tokenizer.decode(t) +\n",
" '\\x1b[0m',\n",
" end=' '\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Gcc3JjwX5DK-"
},
"outputs": [],
"source": [
"text = \"\"\"\n",
"English and CAPITALIZATION\n",
"🎵 鸟\n",
"show_tokens False None elif == >= else: two tabs:\" \" Three tabs: \" \"\n",
"12.0*50=600\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 354,
"status": "ok",
"timestamp": 1725544666773,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "fCDGSXP75Hv-",
"outputId": "f2c26835-a857-41db-ff2d-d930d06e512e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m[CLS]\u001b[0m \u001b[0;30;48;2;252;141;98menglish\u001b[0m \u001b[0;30;48;2;141;160;203mand\u001b[0m \u001b[0;30;48;2;231;138;195mcapital\u001b[0m \u001b[0;30;48;2;166;216;84m##ization\u001b[0m \u001b[0;30;48;2;255;217;47m[UNK]\u001b[0m \u001b[0;30;48;2;102;194;165m[UNK]\u001b[0m \u001b[0;30;48;2;252;141;98mshow\u001b[0m \u001b[0;30;48;2;141;160;203m_\u001b[0m \u001b[0;30;48;2;231;138;195mtoken\u001b[0m \u001b[0;30;48;2;166;216;84m##s\u001b[0m \u001b[0;30;48;2;255;217;47mfalse\u001b[0m \u001b[0;30;48;2;102;194;165mnone\u001b[0m \u001b[0;30;48;2;252;141;98meli\u001b[0m \u001b[0;30;48;2;141;160;203m##f\u001b[0m \u001b[0;30;48;2;231;138;195m=\u001b[0m \u001b[0;30;48;2;166;216;84m=\u001b[0m \u001b[0;30;48;2;255;217;47m>\u001b[0m \u001b[0;30;48;2;102;194;165m=\u001b[0m \u001b[0;30;48;2;252;141;98melse\u001b[0m \u001b[0;30;48;2;141;160;203m:\u001b[0m \u001b[0;30;48;2;231;138;195mtwo\u001b[0m \u001b[0;30;48;2;166;216;84mtab\u001b[0m \u001b[0;30;48;2;255;217;47m##s\u001b[0m \u001b[0;30;48;2;102;194;165m:\u001b[0m \u001b[0;30;48;2;252;141;98m\"\u001b[0m \u001b[0;30;48;2;141;160;203m/\u001b[0m \u001b[0;30;48;2;231;138;195mt\u001b[0m \u001b[0;30;48;2;166;216;84m/\u001b[0m \u001b[0;30;48;2;255;217;47mt\u001b[0m \u001b[0;30;48;2;102;194;165m\"\u001b[0m \u001b[0;30;48;2;252;141;98mthree\u001b[0m \u001b[0;30;48;2;141;160;203mtab\u001b[0m \u001b[0;30;48;2;231;138;195m##s\u001b[0m \u001b[0;30;48;2;166;216;84m:\u001b[0m \u001b[0;30;48;2;255;217;47m\"\u001b[0m \u001b[0;30;48;2;102;194;165m\"\u001b[0m \u001b[0;30;48;2;252;141;98m12\u001b[0m \u001b[0;30;48;2;141;160;203m.\u001b[0m \u001b[0;30;48;2;231;138;195m0\u001b[0m \u001b[0;30;48;2;166;216;84m*\u001b[0m \u001b[0;30;48;2;255;217;47m50\u001b[0m \u001b[0;30;48;2;102;194;165m=\u001b[0m \u001b[0;30;48;2;252;141;98m600\u001b[0m \u001b[0;30;48;2;141;160;203m[SEP]\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"bert-base-uncased\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 219,
"referenced_widgets": [
"76ff072348d0471abaa566d7de6b8e93",
"5323bca1afe64a2a9bc9c14ac39ad230",
"bf9fbc21d76a424dab0c8687bd0874c9",
"f67d099d1e0343d9a822523223d21a75",
"1dc884bbe68e4bf4ad1e14ca44b5c2fd",
"b57a5357a7334d47b39f14e07e2d5708",
"e608cc17958347c19615577b73926f45",
"1695867ecb3f402ea8e10920adf651ee",
"9fb081ecc7374f73b32cb203c7ed042d",
"2fdb8c6afd2647b695f249b4f8122e52",
"6e12c295923644ab930dcb51de932fd9",
"d3316f8df2804c2ba34504b196aed6be",
"893c6036041346ab9486cb7bde06ad0d",
"42f7a33f1b554c289a434300bcc19f70",
"d75a253a2ef648e88e35d765be5d4c34",
"3b5a28c840ee4ba78676b4c3dbcd7af6",
"e1175aef0e2e4b17afdb6a62f68887a7",
"8774144d2cbc44f4bb39305e20cd5093",
"139f7c4f547f4d08a6126897617989d5",
"61bafb93125042f5bb7cc1195b459d45",
"8053c516a8df40498085843ff07a2884",
"81aa45723ec44d18a0e37844b9c70c4e",
"33541a7c0d664fa2bd104fc9bc91f1bd",
"580f25a7f04943e6a5100bdf584f8c97",
"1848a0b868254c848115fc59b2cdd639",
"47e387c2bcdc43329354304e5358c224",
"883b562fa1c7409fbf43d2ac90c29955",
"9798a6e28f56466f9582a40064ad4c4f",
"4f31fd12d7c04233856206304b2a1bc7",
"1460bc4aca764def9120218a963ae183",
"3933809fb01c43bca62dc220cc94f217",
"e4caf964309345bf82ad65411c1a7f3f",
"ea6b6957c38d469abbe07bb20c811d2d",
"eabd5498553646f18eada3542254cb0b",
"206f377162614c72a74e73557fece973",
"deabe5ef1f1f473f81403df9d8923846",
"4096283e990a47a7bd4c00416fa71788",
"3598675602e34600ae5c719d67778d24",
"9ca5334888b249b2bd318be5a97fae7e",
"7cce748a6f264cb69921fc97d4b8c946",
"449aaf8bc9a64f6483eff88cc4678f6c",
"e983503abea646a68dfe45652f7e78d1",
"06b9fe5f3e644ddab47a17a3276e1a67",
"4128b1818d30497f9a3d6869e02addaa"
]
},
"executionInfo": {
"elapsed": 1520,
"status": "ok",
"timestamp": 1719589575187,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "0Ay_NX3K5HyP",
"outputId": "4a32ab93-75f2-4b70-a55b-b643283c8270"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "76ff072348d0471abaa566d7de6b8e93",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/49.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d3316f8df2804c2ba34504b196aed6be",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/570 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "33541a7c0d664fa2bd104fc9bc91f1bd",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.txt: 0%| | 0.00/213k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "eabd5498553646f18eada3542254cb0b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/436k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m[CLS]\u001b[0m \u001b[0;30;48;2;252;141;98mEnglish\u001b[0m \u001b[0;30;48;2;141;160;203mand\u001b[0m \u001b[0;30;48;2;231;138;195mCA\u001b[0m \u001b[0;30;48;2;166;216;84m##PI\u001b[0m \u001b[0;30;48;2;255;217;47m##TA\u001b[0m \u001b[0;30;48;2;102;194;165m##L\u001b[0m \u001b[0;30;48;2;252;141;98m##I\u001b[0m \u001b[0;30;48;2;141;160;203m##Z\u001b[0m \u001b[0;30;48;2;231;138;195m##AT\u001b[0m \u001b[0;30;48;2;166;216;84m##ION\u001b[0m \u001b[0;30;48;2;255;217;47m[UNK]\u001b[0m \u001b[0;30;48;2;102;194;165m[UNK]\u001b[0m \u001b[0;30;48;2;252;141;98mshow\u001b[0m \u001b[0;30;48;2;141;160;203m_\u001b[0m \u001b[0;30;48;2;231;138;195mtoken\u001b[0m \u001b[0;30;48;2;166;216;84m##s\u001b[0m \u001b[0;30;48;2;255;217;47mF\u001b[0m \u001b[0;30;48;2;102;194;165m##als\u001b[0m \u001b[0;30;48;2;252;141;98m##e\u001b[0m \u001b[0;30;48;2;141;160;203mNone\u001b[0m \u001b[0;30;48;2;231;138;195mel\u001b[0m \u001b[0;30;48;2;166;216;84m##if\u001b[0m \u001b[0;30;48;2;255;217;47m=\u001b[0m \u001b[0;30;48;2;102;194;165m=\u001b[0m \u001b[0;30;48;2;252;141;98m>\u001b[0m \u001b[0;30;48;2;141;160;203m=\u001b[0m \u001b[0;30;48;2;231;138;195melse\u001b[0m \u001b[0;30;48;2;166;216;84m:\u001b[0m \u001b[0;30;48;2;255;217;47mtwo\u001b[0m \u001b[0;30;48;2;102;194;165mta\u001b[0m \u001b[0;30;48;2;252;141;98m##bs\u001b[0m \u001b[0;30;48;2;141;160;203m:\u001b[0m \u001b[0;30;48;2;231;138;195m\"\u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47mThree\u001b[0m \u001b[0;30;48;2;102;194;165mta\u001b[0m \u001b[0;30;48;2;252;141;98m##bs\u001b[0m \u001b[0;30;48;2;141;160;203m:\u001b[0m \u001b[0;30;48;2;231;138;195m\"\u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m12\u001b[0m \u001b[0;30;48;2;102;194;165m.\u001b[0m \u001b[0;30;48;2;252;141;98m0\u001b[0m \u001b[0;30;48;2;141;160;203m*\u001b[0m \u001b[0;30;48;2;231;138;195m50\u001b[0m \u001b[0;30;48;2;166;216;84m=\u001b[0m \u001b[0;30;48;2;255;217;47m600\u001b[0m \u001b[0;30;48;2;102;194;165m[SEP]\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"bert-base-cased\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 284,
"referenced_widgets": [
"77d1608ca87f4bc1b7a731c896d86db9",
"6e2c83d85af0419c81df10e85e31d29d",
"5d3e86f8d3f949aeacabace1f7640d81",
"6fd2baf1fc1244d38fc41cde100d7b6e",
"41965c378ba243339352ad3926d48862",
"873384e3c478450ea4a5a9e061c87133",
"dcce05965068457fb764ae1c04066d88",
"41f2087fd27a4c018087063e8e7629d3",
"905ac86fe3294497b1722e955d63ed4c",
"131c47105d324eddaea5c241829e878a",
"22f955573c2845cfba5c314b59d26739",
"41301a0547754ecfbd6044f6eacb0b8d",
"eff43edf235a4b92b0e26d5bc21fc909",
"bbdc2b4c70a0426aa3d2043d0b91b839",
"405476ddfe634ad793e28474dfe30ecc",
"8575a84785714069921bbfdc13fb957e",
"bad64205077a496f96e6d03d927140ba",
"ca152f8ec99e48b39ee5267269eeaca0",
"461e3c04697641359924ae0902b13db0",
"f15e240b2d01488698caa3275e0bacc1",
"4e2491afa9fc4d65b95e9471af782e4d",
"ff1f8b630ceb449e910ca34d969fdafd",
"fa60038dc7c547b8b1d9c54f88fd6b39",
"e69fe73aa3d44de0bddbe1711269bd8e",
"5ff700af61664f0eaebe580c8a49a910",
"e901ed75738f4e41b79651dc012003a6",
"b115c5c5193f489f87209bb5c6d788f9",
"3bce3be7198c4917a6cb2183e1344e2c",
"dedac5ecd5844ee29e346f465074d3fd",
"4521cc909b2942de889a37dbec1f0277",
"645a2e3dba2f49fb9ce9c0e2b2a8e73f",
"198833f7fa2f4ff8ab064b4671461830",
"66acf835e274473f84ccd486a99e71d2",
"db2934af14274fe78ffc85f7d03fd1c8",
"9be5d1e096934134a00974cf8e3fa63c",
"fc2c07f1eeee43e3aad438206929f5df",
"08d6a11cebf840748261e0ba6970092b",
"f8912b0da8aa4f7499ad3f4e5ccca84b",
"c87ad6d49a054fc8850bebf87c444ee3",
"17b6d632e333476b99f4315fe737d359",
"2661e810b7084f93a4dcd454ea7665a0",
"042557f8b84b422882c651f910a9fce0",
"8f95639d6dc946f18f14d8a16e73b4a4",
"89224986b13645d9a9dbedb038e795bb",
"2fcd1b9c380f422291096e57a6c7f85e",
"142133a41f664fdf82a1b16d87a68ae5",
"d1c2b3aac5cc4f3fad1413f8cfdc04e3",
"bce267a75b9946ae8b0db42dd7f925d1",
"3b46d5e1b7fe4427909d1c82debd7ba7",
"3dc1a66c56fb428aad53f5221ed1ae18",
"23cda173696645b6955515990b6834ec",
"2141c20003154d8dab2855deb44d3aad",
"2213d4aa55eb4ef384eaf879552dcd7d",
"56009a7bf6fc4d7c8300fc9dc4d6ad14",
"551cdd6ea1d94ba8a6cce7b00798c63b"
]
},
"executionInfo": {
"elapsed": 2010,
"status": "ok",
"timestamp": 1719589579935,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "K_k5QduY5H0u",
"outputId": "2e844f23-3dee-4078-8d51-4c250d2c2f3e"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "77d1608ca87f4bc1b7a731c896d86db9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/26.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "41301a0547754ecfbd6044f6eacb0b8d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/665 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "fa60038dc7c547b8b1d9c54f88fd6b39",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.json: 0%| | 0.00/1.04M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "db2934af14274fe78ffc85f7d03fd1c8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2fcd1b9c380f422291096e57a6c7f85e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/1.36M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mEnglish\u001b[0m \u001b[0;30;48;2;141;160;203m and\u001b[0m \u001b[0;30;48;2;231;138;195m CAP\u001b[0m \u001b[0;30;48;2;166;216;84mITAL\u001b[0m \u001b[0;30;48;2;255;217;47mIZ\u001b[0m \u001b[0;30;48;2;102;194;165mATION\u001b[0m \u001b[0;30;48;2;252;141;98m\n",
"\u001b[0m \u001b[0;30;48;2;141;160;203m<33>\u001b[0m \u001b[0;30;48;2;231;138;195m<35>\u001b[0m \u001b[0;30;48;2;166;216;84m<34>\u001b[0m \u001b[0;30;48;2;255;217;47m <20>\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m\n",
"\u001b[0m \u001b[0;30;48;2;231;138;195mshow\u001b[0m \u001b[0;30;48;2;166;216;84m_\u001b[0m \u001b[0;30;48;2;255;217;47mt\u001b[0m \u001b[0;30;48;2;102;194;165mok\u001b[0m \u001b[0;30;48;2;252;141;98mens\u001b[0m \u001b[0;30;48;2;141;160;203m False\u001b[0m \u001b[0;30;48;2;231;138;195m None\u001b[0m \u001b[0;30;48;2;166;216;84m el\u001b[0m \u001b[0;30;48;2;255;217;47mif\u001b[0m \u001b[0;30;48;2;102;194;165m ==\u001b[0m \u001b[0;30;48;2;252;141;98m >=\u001b[0m \u001b[0;30;48;2;141;160;203m else\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m two\u001b[0m \u001b[0;30;48;2;255;217;47m tabs\u001b[0m \u001b[0;30;48;2;102;194;165m:\"\u001b[0m \u001b[0;30;48;2;252;141;98m \u001b[0m \u001b[0;30;48;2;141;160;203m \u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m \"\u001b[0m \u001b[0;30;48;2;255;217;47m Three\u001b[0m \u001b[0;30;48;2;102;194;165m tabs\u001b[0m \u001b[0;30;48;2;252;141;98m:\u001b[0m \u001b[0;30;48;2;141;160;203m \"\u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m \u001b[0m \u001b[0;30;48;2;255;217;47m \u001b[0m \u001b[0;30;48;2;102;194;165m \u001b[0m \u001b[0;30;48;2;252;141;98m \u001b[0m \u001b[0;30;48;2;141;160;203m \u001b[0m \u001b[0;30;48;2;231;138;195m \"\u001b[0m \u001b[0;30;48;2;166;216;84m\n",
"\u001b[0m \u001b[0;30;48;2;255;217;47m12\u001b[0m \u001b[0;30;48;2;102;194;165m.\u001b[0m \u001b[0;30;48;2;252;141;98m0\u001b[0m \u001b[0;30;48;2;141;160;203m*\u001b[0m \u001b[0;30;48;2;231;138;195m50\u001b[0m \u001b[0;30;48;2;166;216;84m=\u001b[0m \u001b[0;30;48;2;255;217;47m600\u001b[0m \u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"gpt2\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 183,
"referenced_widgets": [
"63772475e9234672994f2a8edf89b192",
"fa45fdb364444208b2693760809e3c60",
"5a92b1072a834d69ad33060ae1b0fdf0",
"b8a3d22de4964f369ecabc31b6cdca57",
"274983c981654d1eb25d630d8d5e47e3",
"eebaba8a83ed48f8afe2ca129f3a73c9",
"2d7a3c68a2e246059a93fea280f4c2c8",
"22983afeb04d4a06b8b01527d869585e",
"f7492077d2ba4ccf8df039473057321d",
"fb255528c7754047aeb29863dd642b19",
"cc7f7f3ef40042458ac7565b070af032",
"bc67bb86f76e482ab703f3f403f4cc76",
"a7605351b83941698348ee84cd99f955",
"88af753262344cfe9a88b133540bcaf7",
"8f92316b17a24623b8646747f5ecc7d6",
"8153c7e3f21f44f09a3222da6312137d",
"22bee52a95d243868b40b3f8ce5ca7d9",
"f09f8925fc34454dbb970c31d5d82707",
"354c2db5dbd34284baf62a7529537b8b",
"028db37ce29c45939adeed5ae311583c",
"dd534b3f89c64de9b9aa4f7a95a05f34",
"30ed305df62c45329f24a2f64499d490",
"36e3e6b45fca44c8b01a729189b1bdab",
"4727da03ed724b8dafff24856652fe95",
"b580d17fa1134425a837a48aba06dbf4",
"e71172ffd8d74f189cea18e4898c4c2d",
"ce0031117cf347f48d027cd70e87193b",
"9ee79c669c6641d887fa284286435f57",
"9885aaa3e0be4052af4848b92c642cdd",
"ce6eee6121334479a72e023b252124d2",
"3adda22d2bca47a886da662621ec9a9d",
"6a1a34996ae24a14972fd425be48dd7c",
"2c323975d4454113a863e3ec0b56f4fb",
"4b9fab6416924d509bbb6361f63797e1",
"a721b2fba975474c8a3d9384cf998228",
"1f76260c0f6c46598bee13eb4a0f8b65",
"fd95f752cc684613b0f6c6db12af874f",
"3c0a4cbec1bf4c7886a1a9271c1b0832",
"2489776fa74e4dccb4154368f3861623",
"c8506a9393604d119ff71264e27b6734",
"93187864ee6d4d41b49df1c84f35e6e2",
"d7cbd635afec493ebcc4973f8d98c58b",
"d09ad050869c47fb9fbc558f8b6d47d7",
"b4e35a7edeea4b089182f4e9b15dc12e"
]
},
"executionInfo": {
"elapsed": 1618,
"status": "ok",
"timestamp": 1719589589160,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "EJn5nf3c5H2_",
"outputId": "607c38ff-9425-4371-f5e0-1f8ee9449eee"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "63772475e9234672994f2a8edf89b192",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/2.54k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "bc67bb86f76e482ab703f3f403f4cc76",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"spiece.model: 0%| | 0.00/792k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "36e3e6b45fca44c8b01a729189b1bdab",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/2.42M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4b9fab6416924d509bbb6361f63797e1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"special_tokens_map.json: 0%| | 0.00/2.20k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165mEnglish\u001b[0m \u001b[0;30;48;2;252;141;98mand\u001b[0m \u001b[0;30;48;2;141;160;203mCA\u001b[0m \u001b[0;30;48;2;231;138;195mPI\u001b[0m \u001b[0;30;48;2;166;216;84mTAL\u001b[0m \u001b[0;30;48;2;255;217;47mIZ\u001b[0m \u001b[0;30;48;2;102;194;165mATION\u001b[0m \u001b[0;30;48;2;252;141;98m\u001b[0m \u001b[0;30;48;2;141;160;203m<unk>\u001b[0m \u001b[0;30;48;2;231;138;195m\u001b[0m \u001b[0;30;48;2;166;216;84m<unk>\u001b[0m \u001b[0;30;48;2;255;217;47mshow\u001b[0m \u001b[0;30;48;2;102;194;165m_\u001b[0m \u001b[0;30;48;2;252;141;98mto\u001b[0m \u001b[0;30;48;2;141;160;203mken\u001b[0m \u001b[0;30;48;2;231;138;195ms\u001b[0m \u001b[0;30;48;2;166;216;84mFal\u001b[0m \u001b[0;30;48;2;255;217;47ms\u001b[0m \u001b[0;30;48;2;102;194;165me\u001b[0m \u001b[0;30;48;2;252;141;98mNone\u001b[0m \u001b[0;30;48;2;141;160;203m\u001b[0m \u001b[0;30;48;2;231;138;195me\u001b[0m \u001b[0;30;48;2;166;216;84ml\u001b[0m \u001b[0;30;48;2;255;217;47mif\u001b[0m \u001b[0;30;48;2;102;194;165m=\u001b[0m \u001b[0;30;48;2;252;141;98m=\u001b[0m \u001b[0;30;48;2;141;160;203m>\u001b[0m \u001b[0;30;48;2;231;138;195m=\u001b[0m \u001b[0;30;48;2;166;216;84melse\u001b[0m \u001b[0;30;48;2;255;217;47m:\u001b[0m \u001b[0;30;48;2;102;194;165mtwo\u001b[0m \u001b[0;30;48;2;252;141;98mtab\u001b[0m \u001b[0;30;48;2;141;160;203ms\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m\"\u001b[0m \u001b[0;30;48;2;102;194;165mThree\u001b[0m \u001b[0;30;48;2;252;141;98mtab\u001b[0m \u001b[0;30;48;2;141;160;203ms\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m\"\u001b[0m \u001b[0;30;48;2;102;194;165m12.\u001b[0m \u001b[0;30;48;2;252;141;98m0\u001b[0m \u001b[0;30;48;2;141;160;203m*\u001b[0m \u001b[0;30;48;2;231;138;195m50\u001b[0m \u001b[0;30;48;2;166;216;84m=\u001b[0m \u001b[0;30;48;2;255;217;47m600\u001b[0m \u001b[0;30;48;2;102;194;165m\u001b[0m \u001b[0;30;48;2;252;141;98m</s>\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"google/flan-t5-small\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 714,
"status": "ok",
"timestamp": 1723035784494,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": -60
},
"id": "1ymhAsTg5H5e",
"outputId": "7827a535-4f33-4620-f4e7-4a2b622a78c2"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mEnglish\u001b[0m \u001b[0;30;48;2;141;160;203m and\u001b[0m \u001b[0;30;48;2;231;138;195m CAPITAL\u001b[0m \u001b[0;30;48;2;166;216;84mIZATION\u001b[0m \u001b[0;30;48;2;255;217;47m\n",
"\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m<33>\u001b[0m \u001b[0;30;48;2;231;138;195m <20>\u001b[0m \u001b[0;30;48;2;166;216;84m<34>\u001b[0m \u001b[0;30;48;2;255;217;47m<37>\u001b[0m \u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mshow\u001b[0m \u001b[0;30;48;2;141;160;203m_tokens\u001b[0m \u001b[0;30;48;2;231;138;195m False\u001b[0m \u001b[0;30;48;2;166;216;84m None\u001b[0m \u001b[0;30;48;2;255;217;47m elif\u001b[0m \u001b[0;30;48;2;102;194;165m ==\u001b[0m \u001b[0;30;48;2;252;141;98m >=\u001b[0m \u001b[0;30;48;2;141;160;203m else\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m two\u001b[0m \u001b[0;30;48;2;255;217;47m tabs\u001b[0m \u001b[0;30;48;2;102;194;165m:\"\u001b[0m \u001b[0;30;48;2;252;141;98m \u001b[0m \u001b[0;30;48;2;141;160;203m \"\u001b[0m \u001b[0;30;48;2;231;138;195m Three\u001b[0m \u001b[0;30;48;2;166;216;84m tabs\u001b[0m \u001b[0;30;48;2;255;217;47m:\u001b[0m \u001b[0;30;48;2;102;194;165m \"\u001b[0m \u001b[0;30;48;2;252;141;98m \u001b[0m \u001b[0;30;48;2;141;160;203m \"\n",
"\u001b[0m \u001b[0;30;48;2;231;138;195m12\u001b[0m \u001b[0;30;48;2;166;216;84m.\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m*\u001b[0m \u001b[0;30;48;2;252;141;98m50\u001b[0m \u001b[0;30;48;2;141;160;203m=\u001b[0m \u001b[0;30;48;2;231;138;195m600\u001b[0m \u001b[0;30;48;2;166;216;84m\n",
"\u001b[0m "
]
}
],
"source": [
"# The official is `tiktoken` but this the same tokenizer on the HF platform\n",
"show_tokens(text, \"Xenova/gpt-4\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 284,
"referenced_widgets": [
"770da5f1b8b24972b2018e1cadd3ec8a",
"e2f255083a1b4d8f9992f27b4d21c676",
"4fac840948a048d6b1ca7c0dc4f4c5d5",
"c2ac158eb3f0469ca90f7247c546c70f",
"4a8cfc4637124995866810ef1b750fe4",
"409169ca000a484ca4472750cfe63f30",
"8663795c551e457dafc93d02cf0026c3",
"5639fe0c03e7451db316579356290e3d",
"46e73023abe1465382771e9af87f36fc",
"5feaeba2e88a4a7fb83a85f3200e2639",
"76a8030af7794356b5c2daa891d789e2",
"37c46ab78fc64eb98923b24d6a0de37e",
"3bb4b5235ef74c7d89489f8fa8cded17",
"a720bb387fca45968c75352398935382",
"683b85afadb744e4bd7164c51f01d3f9",
"00dd050102674a1ab3fd8d8f9caec4b0",
"cf6a7c6ada024f8e9f106428d506b078",
"33744d7c827e4784a91955159a47e337",
"f6dce141c94d4c8494f75a7387b65331",
"48738fb1cf8e4f0fb70b74a5896669cd",
"1b10141545cb489fa3a58d4939cc4d9b",
"99b9c874e58c4db9b596b6ca1699e666",
"07bf43728198472997c8b59b9343adfe",
"74d33e70d8af43148fd3a618b5d3c5dd",
"8008a03780b24639abce64498b1d832e",
"82ad72412e1343b983679e625c85f47d",
"0d3aa270949048a5886de118b1a3b1f1",
"cc568e7a8ca84810ab878e601fae557a",
"cb57c7f3455f4a34b59ae39d0b599b8b",
"577a22cb6c7549ff96f367bd6f4f8b12",
"d0d4c92c9a0f4bd29255d8ff47d18c11",
"e6d9b96a5cb9487d90136d097e716a5a",
"e878edbee8ea48178b424e56417b7fa5",
"e227f5f6bb3b4580b0ea4304d34ad556",
"36863dd97aa04c48831d1fb455557adc",
"ece59919873646f9bbf41c7547e802a3",
"7ff1f54520324b3e9462062ecd87ce69",
"2e8d55afb3fc4e2fa6b0b887a09b7ca9",
"e527a040f0be4d43830ae6d4335771b0",
"32f18ab0328146b5aac38b4c7ef8029d",
"5d5d4e02a6724861aa36d9af5ea70ea7",
"c8637134a894493093654456f2a9763b",
"8a1023f076f34f34ab0b091d7f62c172",
"3537e0361ab5475282b7f34adbfc70dc",
"12d4df348cd34dc2b3d7ffc41f0561e8",
"65326149d62e4404ba49a8d2d505adac",
"b84c1bff2fcc48ed8fee636e1bdb16f9",
"a8ae6ec72f2744b4991de0a961e1b142",
"98182c58a56343e482ed44935be4fd31",
"15719135708047a49a19887989dac12d",
"a43ccd6d114644ae86c3129786f8105a",
"90cc27a829e54b16850552e04f718bc3",
"2be39bb4eacf41e381432b37febfd788",
"df4dd594480d446da1523bd2d016c1cb",
"11ea292f268d481396192b158814e6b1"
]
},
"executionInfo": {
"elapsed": 9948,
"status": "ok",
"timestamp": 1719590292199,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "3_vAyeTy5H7_",
"outputId": "ad3f759f-19b7-4880-cbf8-9ed7cb25d627"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "770da5f1b8b24972b2018e1cadd3ec8a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/7.88k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "37c46ab78fc64eb98923b24d6a0de37e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.json: 0%| | 0.00/777k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "07bf43728198472997c8b59b9343adfe",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"merges.txt: 0%| | 0.00/442k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e227f5f6bb3b4580b0ea4304d34ad556",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/2.06M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "12d4df348cd34dc2b3d7ffc41f0561e8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"special_tokens_map.json: 0%| | 0.00/958 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mEnglish\u001b[0m \u001b[0;30;48;2;141;160;203m and\u001b[0m \u001b[0;30;48;2;231;138;195m CAPITAL\u001b[0m \u001b[0;30;48;2;166;216;84mIZATION\u001b[0m \u001b[0;30;48;2;255;217;47m\n",
"\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m<33>\u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m<34>\u001b[0m \u001b[0;30;48;2;255;217;47m<37>\u001b[0m \u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mshow\u001b[0m \u001b[0;30;48;2;141;160;203m_\u001b[0m \u001b[0;30;48;2;231;138;195mtokens\u001b[0m \u001b[0;30;48;2;166;216;84m False\u001b[0m \u001b[0;30;48;2;255;217;47m None\u001b[0m \u001b[0;30;48;2;102;194;165m elif\u001b[0m \u001b[0;30;48;2;252;141;98m ==\u001b[0m \u001b[0;30;48;2;141;160;203m >=\u001b[0m \u001b[0;30;48;2;231;138;195m else\u001b[0m \u001b[0;30;48;2;166;216;84m:\u001b[0m \u001b[0;30;48;2;255;217;47m two\u001b[0m \u001b[0;30;48;2;102;194;165m tabs\u001b[0m \u001b[0;30;48;2;252;141;98m:\"\u001b[0m \u001b[0;30;48;2;141;160;203m \u001b[0m \u001b[0;30;48;2;231;138;195m \"\u001b[0m \u001b[0;30;48;2;166;216;84m Three\u001b[0m \u001b[0;30;48;2;255;217;47m tabs\u001b[0m \u001b[0;30;48;2;102;194;165m:\u001b[0m \u001b[0;30;48;2;252;141;98m \"\u001b[0m \u001b[0;30;48;2;141;160;203m \u001b[0m \u001b[0;30;48;2;231;138;195m \"\u001b[0m \u001b[0;30;48;2;166;216;84m\n",
"\u001b[0m \u001b[0;30;48;2;255;217;47m1\u001b[0m \u001b[0;30;48;2;102;194;165m2\u001b[0m \u001b[0;30;48;2;252;141;98m.\u001b[0m \u001b[0;30;48;2;141;160;203m0\u001b[0m \u001b[0;30;48;2;231;138;195m*\u001b[0m \u001b[0;30;48;2;166;216;84m5\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m=\u001b[0m \u001b[0;30;48;2;252;141;98m6\u001b[0m \u001b[0;30;48;2;141;160;203m0\u001b[0m \u001b[0;30;48;2;231;138;195m0\u001b[0m \u001b[0;30;48;2;166;216;84m\n",
"\u001b[0m "
]
}
],
"source": [
"# You need to request access before being able to use this tokenizer\n",
"show_tokens(text, \"bigcode/starcoder2-15b\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 220,
"referenced_widgets": [
"6a6efb1d66ea423a9b5ff4b2f1f1194c",
"1283ee793a40405aa9763e1b88d6d7a3",
"11df3b517fd94524be18cff070e273a8",
"86b01695df4b42d3a6602d704243b6ee",
"b0b1039168a74b8fa79271592f29f0b7",
"877ff6d25f524779a10df09be5fc6093",
"276ec5fb636b49bcb933a4ce96cb900e",
"ac88f9025a0e44b38fc14720d810b5ab",
"4ceb8ce8b67a44b4b7505cd7e589dec1",
"b0d45aec56fd4219b9224dbe31fad3a3",
"bf3a8980a70547f5b853390235a37592",
"f11120105af24fe1b40b6490897e2e2e",
"8109bb974a3e4b12bd7534b62be20940",
"f1d6a31870da4e27bc482bb84953b165",
"1128f56169ac4376ab5fbb46d44b01dc",
"3da38a2294a145bf86124d0fda8b3255",
"85113d3b53fb47bb8593c3a21e37142c",
"1e4a17f723d14694b5aeb673db7394cc",
"87a4011d2d0a4076b027f7068b244dda",
"48ca9047fc7d424f99b40c54e6d732f4",
"3b169b44c1814ed5a7feba7bab0f3ce6",
"dff4ee8d0bd74822a0adf204e21521b8",
"5d3f3b08ec5044e3acf4414703e579d9",
"900ddbabea1846a3a0dfd8380668bec1",
"7da6e29f0349438494ff83975d48f02a",
"01f1433d221f437eb0692d25948ce080",
"9f189b7e32c94a3a84ffd40beed7d1fd",
"3e5f406442df4b848d324ff584eea75c",
"575b63bdd98047a4934d556423dd9ce6",
"c9662398b8ad4d4fa1a59d44f6205769",
"37379e486478437f9fa2f8eac7f9fd60",
"2b83154cf934484da54fe0d0b08fe3d3",
"759faaea712c4a8abb0ece5217e1c470"
]
},
"executionInfo": {
"elapsed": 1388,
"status": "ok",
"timestamp": 1719589605088,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "KeWcUdxY6I3u",
"outputId": "f39c8f56-1e71-44bb-bade-75bfb33b581c"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6a6efb1d66ea423a9b5ff4b2f1f1194c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/166 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f11120105af24fe1b40b6490897e2e2e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/2.14M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5d3f3b08ec5044e3acf4414703e579d9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"special_tokens_map.json: 0%| | 0.00/3.00 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98mEnglish\u001b[0m \u001b[0;30;48;2;141;160;203m and\u001b[0m \u001b[0;30;48;2;231;138;195m CAP\u001b[0m \u001b[0;30;48;2;166;216;84mITAL\u001b[0m \u001b[0;30;48;2;255;217;47mIZATION\u001b[0m \u001b[0;30;48;2;102;194;165m\n",
"\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m<33>\u001b[0m \u001b[0;30;48;2;231;138;195m<35>\u001b[0m \u001b[0;30;48;2;166;216;84m<34>\u001b[0m \u001b[0;30;48;2;255;217;47m <20>\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m\n",
"\u001b[0m \u001b[0;30;48;2;231;138;195mshow\u001b[0m \u001b[0;30;48;2;166;216;84m_\u001b[0m \u001b[0;30;48;2;255;217;47mtokens\u001b[0m \u001b[0;30;48;2;102;194;165m False\u001b[0m \u001b[0;30;48;2;252;141;98m None\u001b[0m \u001b[0;30;48;2;141;160;203m elif\u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m==\u001b[0m \u001b[0;30;48;2;255;217;47m \u001b[0m \u001b[0;30;48;2;102;194;165m>\u001b[0m \u001b[0;30;48;2;252;141;98m=\u001b[0m \u001b[0;30;48;2;141;160;203m else\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m two\u001b[0m \u001b[0;30;48;2;255;217;47m t\u001b[0m \u001b[0;30;48;2;102;194;165mabs\u001b[0m \u001b[0;30;48;2;252;141;98m:\u001b[0m \u001b[0;30;48;2;141;160;203m\"\u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m Three\u001b[0m \u001b[0;30;48;2;102;194;165m t\u001b[0m \u001b[0;30;48;2;252;141;98mabs\u001b[0m \u001b[0;30;48;2;141;160;203m:\u001b[0m \u001b[0;30;48;2;231;138;195m \u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m \u001b[0m \u001b[0;30;48;2;102;194;165m\"\u001b[0m \u001b[0;30;48;2;252;141;98m\n",
"\u001b[0m \u001b[0;30;48;2;141;160;203m1\u001b[0m \u001b[0;30;48;2;231;138;195m2\u001b[0m \u001b[0;30;48;2;166;216;84m.\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m*\u001b[0m \u001b[0;30;48;2;252;141;98m5\u001b[0m \u001b[0;30;48;2;141;160;203m0\u001b[0m \u001b[0;30;48;2;231;138;195m=\u001b[0m \u001b[0;30;48;2;166;216;84m6\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m0\u001b[0m \u001b[0;30;48;2;252;141;98m\n",
"\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"facebook/galactica-1.3b\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 374,
"status": "ok",
"timestamp": 1719589632350,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "__QNj2Cohzz2",
"outputId": "17ffab73-b07c-44a9-c482-64ab9f4c45a4"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0;30;48;2;102;194;165m<s>\u001b[0m \u001b[0;30;48;2;252;141;98m\u001b[0m \u001b[0;30;48;2;141;160;203m\n",
"\u001b[0m \u001b[0;30;48;2;231;138;195mEnglish\u001b[0m \u001b[0;30;48;2;166;216;84mand\u001b[0m \u001b[0;30;48;2;255;217;47mC\u001b[0m \u001b[0;30;48;2;102;194;165mAP\u001b[0m \u001b[0;30;48;2;252;141;98mIT\u001b[0m \u001b[0;30;48;2;141;160;203mAL\u001b[0m \u001b[0;30;48;2;231;138;195mIZ\u001b[0m \u001b[0;30;48;2;166;216;84mATION\u001b[0m \u001b[0;30;48;2;255;217;47m\n",
"\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m<33>\u001b[0m \u001b[0;30;48;2;231;138;195m<35>\u001b[0m \u001b[0;30;48;2;166;216;84m\u001b[0m \u001b[0;30;48;2;255;217;47m<37>\u001b[0m \u001b[0;30;48;2;102;194;165m<35>\u001b[0m \u001b[0;30;48;2;252;141;98m<38>\u001b[0m \u001b[0;30;48;2;141;160;203m\n",
"\u001b[0m \u001b[0;30;48;2;231;138;195mshow\u001b[0m \u001b[0;30;48;2;166;216;84m_\u001b[0m \u001b[0;30;48;2;255;217;47mto\u001b[0m \u001b[0;30;48;2;102;194;165mkens\u001b[0m \u001b[0;30;48;2;252;141;98mFalse\u001b[0m \u001b[0;30;48;2;141;160;203mNone\u001b[0m \u001b[0;30;48;2;231;138;195melif\u001b[0m \u001b[0;30;48;2;166;216;84m==\u001b[0m \u001b[0;30;48;2;255;217;47m>=\u001b[0m \u001b[0;30;48;2;102;194;165melse\u001b[0m \u001b[0;30;48;2;252;141;98m:\u001b[0m \u001b[0;30;48;2;141;160;203mtwo\u001b[0m \u001b[0;30;48;2;231;138;195mtabs\u001b[0m \u001b[0;30;48;2;166;216;84m:\"\u001b[0m \u001b[0;30;48;2;255;217;47m \u001b[0m \u001b[0;30;48;2;102;194;165m\"\u001b[0m \u001b[0;30;48;2;252;141;98mThree\u001b[0m \u001b[0;30;48;2;141;160;203mtabs\u001b[0m \u001b[0;30;48;2;231;138;195m:\u001b[0m \u001b[0;30;48;2;166;216;84m\"\u001b[0m \u001b[0;30;48;2;255;217;47m \u001b[0m \u001b[0;30;48;2;102;194;165m\"\u001b[0m \u001b[0;30;48;2;252;141;98m\n",
"\u001b[0m \u001b[0;30;48;2;141;160;203m1\u001b[0m \u001b[0;30;48;2;231;138;195m2\u001b[0m \u001b[0;30;48;2;166;216;84m.\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m*\u001b[0m \u001b[0;30;48;2;252;141;98m5\u001b[0m \u001b[0;30;48;2;141;160;203m0\u001b[0m \u001b[0;30;48;2;231;138;195m=\u001b[0m \u001b[0;30;48;2;166;216;84m6\u001b[0m \u001b[0;30;48;2;255;217;47m0\u001b[0m \u001b[0;30;48;2;102;194;165m0\u001b[0m \u001b[0;30;48;2;252;141;98m\n",
"\u001b[0m "
]
}
],
"source": [
"show_tokens(text, \"microsoft/Phi-3-mini-4k-instruct\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9Tu7OY4HvBEm"
},
"source": [
"# Contextualized Word Embeddings From a Language Model (Like BERT)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265,
"referenced_widgets": [
"761c3e6c7f26453bba2f463f39f3ae73",
"a5c45eeafcf4456bbb0fe1bb43ef2497",
"1f35082ee7ec425ea801321112d48db1",
"618c1c5ae3ea4650a412a3e009d9fb49",
"a158991ec46e4587aac2111e02153f4c",
"51af2c28e34245ed83f02b424a6a640a",
"56a95848a2814195b334c31f0a961cd8",
"c1e09c1869f7410ba328e6efe56a0460",
"17777d8eae4742aaa78d7854a52e102f",
"5916af8bb7ed4efcb05c8f1cdf826149",
"b1778299eac04302ac136872dfb0a359",
"8ee98d9d017542609881a1e16a8f393f",
"c4a8f08da3f64fb287f7a940c4a9408f",
"d66ee44b7f5b4c6eac1536235a627441",
"c44feccdd01949c1af40fa69f6e03dd0",
"84682eb9cfff444da7633f4ca9360f77",
"332810f458df4e23bc034852706bcc6f",
"4999e7d8e2384bf4adfbf1777587a65f",
"a8d4f19ff4554165a5e78d3783928fe1",
"29d9e7a799ea402fb5a28b2817156838",
"222d489bf1664763babf0e377e45f4d8",
"2ae57535c39541fe98d6a8ae22bcd7d4",
"fd134a05028c447a994166eccc557806",
"69101d935ae841e59aa7f30e40789496",
"3ef63f93e424409192edd1b1364aba48",
"1d204aedfeb14df5ae7e27eb88a87018",
"e74b57abfaa2487aa8e369103be5d00d",
"94609e349c8b43b5b74d2c059623f9f0",
"febce6a7c96e4a42a9c6faa0bf1763c0",
"18a4603fa6a040c0acd792243510562a",
"c04ce575bf624bd1a80113d1eff1ae94",
"6a9de4aaed054608b800820831aec87f",
"46cd179c1c09474a80dc4cea39b759d5",
"20331d1d457143719fe732325e79877e",
"2a5055c8fc03457eb29390312c555ec5",
"6de0874e33c146bba06131aa452c403f",
"0f1d2e4c312d4ab38e359f96c9a760b7",
"f946c53a81f34d64b25e331fc4b4c7a1",
"2dc553ef192c4002b818de6736367fe0",
"665b4085199a4ec5890dba773f07d4b7",
"de74a5af1ef9462e82e5622232346a79",
"34df4fe808174f80a89a765f6ce2f28f",
"56c082332310434fa5ec791b728fe82e",
"9ef54c7a15d1400f91598b367ac6552e",
"ed19532a8eda4925a4014a3f11517ff9",
"3859f0311ade4d278190924f0107ba0e",
"6ea963b64fd642b29021110d77827021",
"e3efc5b43bf2417faaea2e44a306fa75",
"4975826de21547fca434e1fb492a216a",
"19b547738c8e45bda7879dc527229019",
"6469ef3a7ba6465c911a22060d54ff95",
"657b6ce5f0804bc78d64f9b7b6a27777",
"301363b755a24bc9bf413fc3b0ffd8b2",
"50fc50f975294a23a1d5bccf64efc872",
"3c722f92f6c2479a91cf957d1d18fbee",
"7570614d79184ab2b44700df2342b294",
"543ae93b2c5f4339979b8c64eff33f62",
"72b54fe7c88540488adee31c15595c89",
"3fbdaadec9f545b99057f7c55c6a6df1",
"7656d1b978554f4383160b93a80ee7c6",
"4d55d194e62942df8f1c32b9bb244e9b",
"403f03c3a2434fd2ac6f143c1972c62e",
"cc68d46e46e7487484d366f77ea863ca",
"3dee319ce0aa4ce589ce84033bba8d9d",
"a280161408504bb894ab78526f67750b",
"72d18d21ca2a45ac868d390ead3ac086"
]
},
"executionInfo": {
"elapsed": 5049,
"status": "ok",
"timestamp": 1719641476949,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "nsjz-VsYu9bB",
"outputId": "03ea124b-c6de-449d-ea6f-f5e5b84c2c97"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "761c3e6c7f26453bba2f463f39f3ae73",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/52.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8ee98d9d017542609881a1e16a8f393f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/474 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "fd134a05028c447a994166eccc557806",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.json: 0%| | 0.00/899k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "20331d1d457143719fe732325e79877e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ed19532a8eda4925a4014a3f11517ff9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/578 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7570614d79184ab2b44700df2342b294",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"pytorch_model.bin: 0%| | 0.00/241M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from transformers import AutoModel, AutoTokenizer\n",
"\n",
"# Load a tokenizer\n",
"tokenizer = AutoTokenizer.from_pretrained(\"microsoft/deberta-base\")\n",
"\n",
"# Load a language model\n",
"model = AutoModel.from_pretrained(\"microsoft/deberta-v3-xsmall\")\n",
"\n",
"# Tokenize the sentence\n",
"tokens = tokenizer('Hello world', return_tensors='pt')\n",
"\n",
"# Process the tokens\n",
"output = model(**tokens)[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 567,
"status": "ok",
"timestamp": 1719641482036,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "lQly_KcbvDce",
"outputId": "fe2cc467-2a5a-4111-8d23-4da9aa799b79"
},
"outputs": [
{
"data": {
"text/plain": [
"torch.Size([1, 4, 384])"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719641482353,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "8GcRrpPV0kVj",
"outputId": "93766ff1-1ae5-4e90-dba0-286d9e721c3d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[CLS]\n",
"Hello\n",
" world\n",
"[SEP]\n"
]
}
],
"source": [
"for token in tokens['input_ids'][0]:\n",
" print(tokenizer.decode(token))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 1,
"status": "ok",
"timestamp": 1719641482353,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "e8oHVC7B0lkk",
"outputId": "f7dd1e0c-a2db-4ae4-8ccb-c97fa150071a"
},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[[-3.4816, 0.0861, -0.1819, ..., -0.0612, -0.3911, 0.3017],\n",
" [ 0.1898, 0.3208, -0.2315, ..., 0.3714, 0.2478, 0.8048],\n",
" [ 0.2071, 0.5036, -0.0485, ..., 1.2175, -0.2292, 0.8582],\n",
" [-3.4278, 0.0645, -0.1427, ..., 0.0658, -0.4367, 0.3834]]],\n",
" grad_fn=<NativeLayerNormBackward0>)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"output"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DdEDuLWa0r4L"
},
"source": [
"# Text Embeddings (For Sentences and Whole Documents)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 425,
"referenced_widgets": [
"a50156f59d8548b683982af06d5bda09",
"7cbfb80418f14068bb4327a3823140fb",
"ef1e2b9a1e694eaa8d5b371caa66277e",
"ebf8ef569e374c17aa15472ee6ea98d8",
"75f5174b66c14f5480960c79a44283ad",
"05504faf760d43ea9082ff5a91ff82f7",
"50c6cf2be63b4be29524188d79e8cd53",
"9088f66d46f44030af46aee55005a939",
"7af424dbfd864b8ba6f7661f2204302c",
"44aae8dd70da4be98fd86a970373753b",
"49dd6490203640d1821196fe28a08732",
"c40e76c5e9bd42bea7903e811bce53a4",
"8f7d3eea82614e4ca93482ad7cb637a6",
"05eb94a4253544648654f24268e8b6da",
"01b9535a72c64c95b482640b2bd3c5fa",
"b954ab8edd66487cb9a5502879f0e1c5",
"e6e103b2c53a4b5ca71f988b7804aeaf",
"77f25a7ee7f843b0a24a1d360a1df211",
"a0e30058096941dcaab8f24235ee1c49",
"b26ac1747c884e3a913d90b8c05a991b",
"e5db8326887a48ae97341cf216d35893",
"c76f0b056c1f40f3ac37df24c449e9ec",
"c4c7962b94674cd6898980cea6483595",
"7fc0e3e3c75e47f88410a2774695081e",
"6c96c43dd36a4b08ad94a9cad642b1ab",
"73b403ec39fa4a16883ad08a50c7204b",
"6feee2b015ef4fa6b94c905ec91f5146",
"ab68f3b41d954ec992b9339a91def6a8",
"fb9054b0d27c4a8a9b8241f9d5910e51",
"78b6d1098f754eb491a2a729e00d2335",
"e84f055fc0c04c4983b162d1d8c67147",
"13455551484542ea93d4dbbd937288d5",
"99db2aef29ba4717866072f20f1acf61",
"10f9441d843c44e5b11cfb2e21b5d89e",
"5c987bfb44d14b1ea822c99cb7dde071",
"52ac1a973f7140e8b49dbf58dc0c8b21",
"37117814cae9440c9c54f63def546c4b",
"e019010a35ea4f3a9b9236a626b34760",
"ba03fe6450b742c99e8b8836f585232c",
"47e9e2ed70c84023ae3a1dbf0cb27328",
"99e53690e1c940cea12f581b625c3b3d",
"f53653a9050042649df9d91114ba39bb",
"737c25a604ee4bcd982487f79450d3ea",
"cea92b3b96f24087b494487bf3f4c0f9",
"784748de51254ba18128af8df30b8a93",
"11890f81eceb41f7be6a2d52c9a9e55e",
"f7adb025a07a4aca8b7a3a174304666f",
"7f7a5bfc6073495da65dcfd4b2d49309",
"fa6483d07ae54fb2904fc117aa9a3d5b",
"2cbcf7d0b1384b8cb320e1b30c124d71",
"94ea21e19acf4eb7a9010510b226db86",
"f399e1a91e7d4b7eaddfb910bd81d750",
"f26297a84f224c4da8afd4316c8e7477",
"e4bef8778ddc46e5a8d0756eb27c3e7c",
"66943967c327428a9796fcd38c36b24d",
"f0793746dff34da8858758fb55284b97",
"1bc7b31eddc54588979f3fff14a0e12e",
"75f54f7a8e5b4965b6f0ad28e5f3bf26",
"3cb9178f0568448fa839d5cccc7973d7",
"40bde45ac20f48ab93c4fd9e8284eac8",
"09b701e83e3844fab97ff237b06a1238",
"c1bbe572b8324ea48d42c40a5128bb8e",
"a4bd81ed4d9d498a983c13ea79265819",
"d8d65b5ac8914792b82460cf0bae980d",
"136cd465bac246f2ac2454eea2f0484d",
"59030392bbde468aae6c62aecddd499e",
"ca7eb54b296c4a1fa91678c0e3d65f5d",
"35521a33c1324a928fd2c9f7fce2ce69",
"80c82a58f0924a578bcae9d3c6537c11",
"589d24a92b3d4e5c99460cd609c8a230",
"0edd8bfe5bba47d59c5b195841cc4228",
"2aed08160bef4a7189821c560c01d6bb",
"01ca9f66804048a9a75475ab9c49a24e",
"880c7bdcf3174a78892aa7d0cd11dca7",
"1d4f85ce80d841eab27b411b3e61e9be",
"9960dd3bff70458abc148f1a153175ec",
"0bdc012004ad445fa527fafee0ab55c2",
"2895fad80e754b4e8158c6dd8db69058",
"4a2c467901414bf0afc5b310aa959dae",
"a465317d0f3b4106bbf8fd6c7a3caf6a",
"7c93b5df16a64e1981e350459b05852d",
"66d5c0087bd141b3baa502e3aa8bd408",
"d26ec2f29c154da1ac7cb49cb9729113",
"7385eea9ed2a438c8dae350fe2328162",
"c861634df7524e7cbd7fdca030a0b663",
"9904b80c37c44ed2a6b3c21786016e26",
"df424fcea3e84c8084dbf8e146d1231f",
"a1e116cf62d74d4e8e33c99379e924ed",
"652ddbc085994a36b553ea04359943a1",
"b340ba4de77043dcbedbcbdf6033d0c7",
"10c89678b42b4cf0b8e95b74ad346fba",
"fb22220311094fe2b3d245ff080ca4d7",
"fc1358383bba4e5ebe2edfca57473002",
"342adaf9c12548a4af25f5361ca869ca",
"374287b3d21a427fabd82dc1e0710d62",
"427e31cb6fcc4687937d807116e5e581",
"8195fa9ad4e3487c90cbc0860361a336",
"82bf559e6997425aba4245c44531f762",
"6bc8887963d744a7a6c15930844f5513",
"4688efe8ab954510b30df18f8daa74a5",
"27c697f872b64fb5a43deb255e27fb50",
"08f2dd1dd0a742eb8c9e11a41dbe69a3",
"94b1fee85a034b49aba0c50fcdfc0fdb",
"c21e510111f5425bad28afd9b723f9d1",
"fea69d561fb94e8c98af4526f8d4b33e",
"0cec3fa3672b4bbaa20e3d4aae6fd575",
"2b09f8d112ea43a5b60c010f2bf0bbdb",
"60ea33df89a449ad9e0f90a0bca672ad",
"48230299285241a28e2c8e03db6fce4d",
"3229d1aa3fe74299a1d4e3f917cc2ca6",
"1f5849878621437397efe2de7b7a43fe",
"2a554904b31c4ed8a4269d2c26ea4e91",
"a29c289ad83d4c718d5854e5d3eff48a",
"07584d1f220f4fb6b82a42090f3818f7",
"04e248de88ac4a50ad20272d31549304",
"57bb0b0d061d4e778493adb47482f234",
"2a24003cc0d54e568706cc6fc77d2831",
"849583682f034d3d8b8887cbafb3daaf",
"8ab0464a810e4b208d4c0fc481c58b54",
"cab50f86e61d43df90643acaf98670ab",
"d4180478ac134757bcfe6c4f0ff4990e"
]
},
"executionInfo": {
"elapsed": 7006,
"status": "ok",
"timestamp": 1719641491724,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "TQHWioIc0pQ8",
"outputId": "87112ec7-bee0-4894-d850-8dd5e0f4e38c"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a50156f59d8548b683982af06d5bda09",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"modules.json: 0%| | 0.00/349 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c40e76c5e9bd42bea7903e811bce53a4",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config_sentence_transformers.json: 0%| | 0.00/116 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c4c7962b94674cd6898980cea6483595",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"README.md: 0%| | 0.00/10.6k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "10f9441d843c44e5b11cfb2e21b5d89e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"sentence_bert_config.json: 0%| | 0.00/53.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "784748de51254ba18128af8df30b8a93",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"config.json: 0%| | 0.00/571 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f0793746dff34da8858758fb55284b97",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model.safetensors: 0%| | 0.00/438M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ca7eb54b296c4a1fa91678c0e3d65f5d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/363 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2895fad80e754b4e8158c6dd8db69058",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "652ddbc085994a36b553ea04359943a1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/466k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4688efe8ab954510b30df18f8daa74a5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"special_tokens_map.json: 0%| | 0.00/239 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1f5849878621437397efe2de7b7a43fe",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"1_Pooling/config.json: 0%| | 0.00/190 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from sentence_transformers import SentenceTransformer\n",
"\n",
"# Load model\n",
"model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')\n",
"\n",
"# Convert text to text embeddings\n",
"vector = model.encode(\"Best movie ever!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719641491724,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "PDwfmBiC0uER",
"outputId": "db6755ce-92b2-45d1-85aa-9b53baee446e"
},
"outputs": [
{
"data": {
"text/plain": [
"(768,)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xnuGRjo80yKj"
},
"source": [
"# Word Embeddings Beyond LLMs\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 44634,
"status": "ok",
"timestamp": 1719641543423,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "sKgNdnwe0vfK",
"outputId": "180bbb09-b030-4fa0-9198-085b0eb54c7b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[==================================================] 100.0% 66.0/66.0MB downloaded\n"
]
}
],
"source": [
"import gensim.downloader as api\n",
"\n",
"# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)\n",
"# Other options include \"word2vec-google-news-300\"\n",
"# More options at https://github.com/RaRe-Technologies/gensim-data\n",
"model = api.load(\"glove-wiki-gigaword-50\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719641543423,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "u_vj5NVn01aD",
"outputId": "73c3edd8-0185-494d-a842-d78cbe100642"
},
"outputs": [
{
"data": {
"text/plain": [
"[('king', 1.0000001192092896),\n",
" ('prince', 0.8236179351806641),\n",
" ('queen', 0.7839043140411377),\n",
" ('ii', 0.7746230363845825),\n",
" ('emperor', 0.7736247777938843),\n",
" ('son', 0.766719400882721),\n",
" ('uncle', 0.7627150416374207),\n",
" ('kingdom', 0.7542161345481873),\n",
" ('throne', 0.7539914846420288),\n",
" ('brother', 0.7492411136627197),\n",
" ('ruler', 0.7434253692626953)]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.most_similar([model['king']], topn=11)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QMSgyKKS4xUx"
},
"source": [
"# Recommending songs by embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3dJdWzT67nDL"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from urllib import request\n",
"\n",
"# Get the playlist dataset file\n",
"data = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')\n",
"\n",
"# Parse the playlist dataset file. Skip the first two lines as\n",
"# they only contain metadata\n",
"lines = data.read().decode(\"utf-8\").split('\\n')[2:]\n",
"\n",
"# Remove playlists with only one song\n",
"playlists = [s.rstrip().split() for s in lines if len(s.split()) > 1]\n",
"\n",
"# Load song metadata\n",
"songs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')\n",
"songs_file = songs_file.read().decode(\"utf-8\").split('\\n')\n",
"songs = [s.rstrip().split('\\t') for s in songs_file]\n",
"songs_df = pd.DataFrame(data=songs, columns = ['id', 'title', 'artist'])\n",
"songs_df = songs_df.set_index('id')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1724598630488,
"user": {
"displayName": "Jay Alammar جهاد العمار",
"userId": "14617748739431919458"
},
"user_tz": 240
},
"id": "Q3zirG-lo3H8",
"outputId": "e3b4269e-dd42-428e-8b28-46c27d0231af"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Playlist #1:\n",
" ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '2', '42', '43', '44', '45', '46', '47', '48', '20', '49', '8', '50', '51', '52', '53', '54', '55', '56', '57', '25', '58', '59', '60', '61', '62', '3', '63', '64', '65', '66', '46', '47', '67', '2', '48', '68', '69', '70', '57', '50', '71', '72', '53', '73', '25', '74', '59', '20', '46', '75', '76', '77', '59', '20', '43'] \n",
"\n",
"Playlist #2:\n",
" ['78', '79', '80', '3', '62', '81', '14', '82', '48', '83', '84', '17', '85', '86', '87', '88', '74', '89', '90', '91', '4', '73', '62', '92', '17', '53', '59', '93', '94', '51', '50', '27', '95', '48', '96', '97', '98', '99', '100', '57', '101', '102', '25', '103', '3', '104', '105', '106', '107', '47', '108', '109', '110', '111', '112', '113', '25', '63', '62', '114', '115', '84', '116', '117', '118', '119', '120', '121', '122', '123', '50', '70', '71', '124', '17', '85', '14', '82', '48', '125', '47', '46', '72', '53', '25', '73', '4', '126', '59', '74', '20', '43', '127', '128', '129', '13', '82', '48', '130', '131', '132', '133', '134', '135', '136', '137', '59', '46', '138', '43', '20', '139', '140', '73', '57', '70', '141', '3', '1', '74', '142', '143', '144', '145', '48', '13', '25', '146', '50', '147', '126', '59', '20', '148', '149', '150', '151', '152', '56', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '60', '176', '51', '177', '178', '179', '180', '181', '182', '183', '184', '185', '57', '186', '187', '188', '189', '190', '191', '46', '192', '193', '194', '195', '196', '197', '198', '25', '199', '200', '49', '201', '100', '202', '203', '204', '205', '206', '207', '32', '208', '209', '210']\n"
]
}
],
"source": [
"print( 'Playlist #1:\\n ', playlists[0], '\\n')\n",
"print( 'Playlist #2:\\n ', playlists[1])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EaUz3E0P7sJs"
},
"outputs": [],
"source": [
"from gensim.models import Word2Vec\n",
"\n",
"# Train our Word2Vec model\n",
"model = Word2Vec(\n",
" playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 314,
"status": "ok",
"timestamp": 1719642095066,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "9EFGWesO8rOJ",
"outputId": "1e46ce56-7b14-4268-a38a-c328e0f52943"
},
"outputs": [
{
"data": {
"text/plain": [
"[('2849', 0.9979680776596069),\n",
" ('2640', 0.9964019060134888),\n",
" ('3167', 0.9963980317115784),\n",
" ('5549', 0.9959008693695068),\n",
" ('2715', 0.9958351850509644),\n",
" ('3117', 0.9954560995101929),\n",
" ('2987', 0.9953479766845703),\n",
" ('2881', 0.9951083660125732),\n",
" ('2886', 0.9950577616691589),\n",
" ('3094', 0.994985044002533)]"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"song_id = 2172\n",
"\n",
"# Ask the model for songs similar to song #2172\n",
"model.wv.most_similar(positive=str(song_id))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 321,
"status": "ok",
"timestamp": 1719642762615,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "AMiY6isXqKk4",
"outputId": "0f465f20-ada8-4fa8-92d6-f72966d03aa4"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"title Fade To Black\n",
"artist Metallica\n",
"Name: 2172 , dtype: object\n"
]
}
],
"source": [
"print(songs_df.iloc[2172])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 237
},
"executionInfo": {
"elapsed": 556,
"status": "ok",
"timestamp": 1719642918281,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "aOzWENxr2Fl3",
"outputId": "0b1ac29a-14f7-4e30-e153-e8f35ca97d7e"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"summary": "{\n \"name\": \"print_recommendations(2172)\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"id\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"2640 \",\n \"2715 \",\n \"3167 \"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Red Barchetta\",\n \"Rainbow In The Dark\",\n \"Unchained\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"artist\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Rush\",\n \"Dio\",\n \"Van Halen\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
"type": "dataframe"
},
"text/html": [
"\n",
" <div id=\"df-94b64d84-06f0-49f5-a721-a51ab661e5c4\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>artist</th>\n",
" </tr>\n",
" <tr>\n",
" <th>id</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2849</th>\n",
" <td>Run To The Hills</td>\n",
" <td>Iron Maiden</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2640</th>\n",
" <td>Red Barchetta</td>\n",
" <td>Rush</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3167</th>\n",
" <td>Unchained</td>\n",
" <td>Van Halen</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5549</th>\n",
" <td>November Rain</td>\n",
" <td>Guns N' Roses</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2715</th>\n",
" <td>Rainbow In The Dark</td>\n",
" <td>Dio</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-94b64d84-06f0-49f5-a721-a51ab661e5c4')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-94b64d84-06f0-49f5-a721-a51ab661e5c4 button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-94b64d84-06f0-49f5-a721-a51ab661e5c4');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
"<div id=\"df-66b2f5cc-45c9-44ee-b044-69f0e59b123a\">\n",
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-66b2f5cc-45c9-44ee-b044-69f0e59b123a')\"\n",
" title=\"Suggest charts\"\n",
" style=\"display:none;\">\n",
"\n",
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <g>\n",
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
" </g>\n",
"</svg>\n",
" </button>\n",
"\n",
"<style>\n",
" .colab-df-quickchart {\n",
" --bg-color: #E8F0FE;\n",
" --fill-color: #1967D2;\n",
" --hover-bg-color: #E2EBFA;\n",
" --hover-fill-color: #174EA6;\n",
" --disabled-fill-color: #AAA;\n",
" --disabled-bg-color: #DDD;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-quickchart {\n",
" --bg-color: #3B4455;\n",
" --fill-color: #D2E3FC;\n",
" --hover-bg-color: #434B5C;\n",
" --hover-fill-color: #FFFFFF;\n",
" --disabled-bg-color: #3B4455;\n",
" --disabled-fill-color: #666;\n",
" }\n",
"\n",
" .colab-df-quickchart {\n",
" background-color: var(--bg-color);\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: var(--fill-color);\n",
" height: 32px;\n",
" padding: 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-quickchart:hover {\n",
" background-color: var(--hover-bg-color);\n",
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: var(--button-hover-fill-color);\n",
" }\n",
"\n",
" .colab-df-quickchart-complete:disabled,\n",
" .colab-df-quickchart-complete:disabled:hover {\n",
" background-color: var(--disabled-bg-color);\n",
" fill: var(--disabled-fill-color);\n",
" box-shadow: none;\n",
" }\n",
"\n",
" .colab-df-spinner {\n",
" border: 2px solid var(--fill-color);\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" animation:\n",
" spin 1s steps(1) infinite;\n",
" }\n",
"\n",
" @keyframes spin {\n",
" 0% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" border-left-color: var(--fill-color);\n",
" }\n",
" 20% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 30% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 40% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 60% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 80% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" 90% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" }\n",
"</style>\n",
"\n",
" <script>\n",
" async function quickchart(key) {\n",
" const quickchartButtonEl =\n",
" document.querySelector('#' + key + ' button');\n",
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
" try {\n",
" const charts = await google.colab.kernel.invokeFunction(\n",
" 'suggestCharts', [key], {});\n",
" } catch (error) {\n",
" console.error('Error during call to suggestCharts:', error);\n",
" }\n",
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
" }\n",
" (() => {\n",
" let quickchartButtonEl =\n",
" document.querySelector('#df-66b2f5cc-45c9-44ee-b044-69f0e59b123a button');\n",
" quickchartButtonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
" })();\n",
" </script>\n",
"</div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"text/plain": [
" title artist\n",
"id \n",
"2849 Run To The Hills Iron Maiden\n",
"2640 Red Barchetta Rush\n",
"3167 Unchained Van Halen\n",
"5549 November Rain Guns N' Roses\n",
"2715 Rainbow In The Dark Dio"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"\n",
"def print_recommendations(song_id):\n",
" similar_songs = np.array(\n",
" model.wv.most_similar(positive=str(song_id),topn=5)\n",
" )[:,0]\n",
" return songs_df.iloc[similar_songs]\n",
"\n",
"# Extract recommendations\n",
"print_recommendations(2172)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 310
},
"executionInfo": {
"elapsed": 681,
"status": "ok",
"timestamp": 1719642181255,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "xqrzQQ-m1EJ5",
"outputId": "3cf4967d-f510-4772-cb11-4166d16c6956"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"title Fade To Black\n",
"artist Metallica\n",
"Name: 2172 , dtype: object\n",
"['2849' '2640' '3167' '5549' '2715']\n"
]
},
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"summary": "{\n \"name\": \"print_recommendations(2172)\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"id\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"2640 \",\n \"2715 \",\n \"3167 \"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Red Barchetta\",\n \"Rainbow In The Dark\",\n \"Unchained\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"artist\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Rush\",\n \"Dio\",\n \"Van Halen\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
"type": "dataframe"
},
"text/html": [
"\n",
" <div id=\"df-c38e0eb4-9c39-45f5-aa32-dbd65ad89576\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>artist</th>\n",
" </tr>\n",
" <tr>\n",
" <th>id</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2849</th>\n",
" <td>Run To The Hills</td>\n",
" <td>Iron Maiden</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2640</th>\n",
" <td>Red Barchetta</td>\n",
" <td>Rush</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3167</th>\n",
" <td>Unchained</td>\n",
" <td>Van Halen</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5549</th>\n",
" <td>November Rain</td>\n",
" <td>Guns N' Roses</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2715</th>\n",
" <td>Rainbow In The Dark</td>\n",
" <td>Dio</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c38e0eb4-9c39-45f5-aa32-dbd65ad89576')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-c38e0eb4-9c39-45f5-aa32-dbd65ad89576 button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-c38e0eb4-9c39-45f5-aa32-dbd65ad89576');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
"<div id=\"df-dbb90c85-6dc6-4ec9-a4c5-ebcbdb0a5897\">\n",
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dbb90c85-6dc6-4ec9-a4c5-ebcbdb0a5897')\"\n",
" title=\"Suggest charts\"\n",
" style=\"display:none;\">\n",
"\n",
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <g>\n",
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
" </g>\n",
"</svg>\n",
" </button>\n",
"\n",
"<style>\n",
" .colab-df-quickchart {\n",
" --bg-color: #E8F0FE;\n",
" --fill-color: #1967D2;\n",
" --hover-bg-color: #E2EBFA;\n",
" --hover-fill-color: #174EA6;\n",
" --disabled-fill-color: #AAA;\n",
" --disabled-bg-color: #DDD;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-quickchart {\n",
" --bg-color: #3B4455;\n",
" --fill-color: #D2E3FC;\n",
" --hover-bg-color: #434B5C;\n",
" --hover-fill-color: #FFFFFF;\n",
" --disabled-bg-color: #3B4455;\n",
" --disabled-fill-color: #666;\n",
" }\n",
"\n",
" .colab-df-quickchart {\n",
" background-color: var(--bg-color);\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: var(--fill-color);\n",
" height: 32px;\n",
" padding: 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-quickchart:hover {\n",
" background-color: var(--hover-bg-color);\n",
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: var(--button-hover-fill-color);\n",
" }\n",
"\n",
" .colab-df-quickchart-complete:disabled,\n",
" .colab-df-quickchart-complete:disabled:hover {\n",
" background-color: var(--disabled-bg-color);\n",
" fill: var(--disabled-fill-color);\n",
" box-shadow: none;\n",
" }\n",
"\n",
" .colab-df-spinner {\n",
" border: 2px solid var(--fill-color);\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" animation:\n",
" spin 1s steps(1) infinite;\n",
" }\n",
"\n",
" @keyframes spin {\n",
" 0% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" border-left-color: var(--fill-color);\n",
" }\n",
" 20% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 30% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 40% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 60% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 80% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" 90% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" }\n",
"</style>\n",
"\n",
" <script>\n",
" async function quickchart(key) {\n",
" const quickchartButtonEl =\n",
" document.querySelector('#' + key + ' button');\n",
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
" try {\n",
" const charts = await google.colab.kernel.invokeFunction(\n",
" 'suggestCharts', [key], {});\n",
" } catch (error) {\n",
" console.error('Error during call to suggestCharts:', error);\n",
" }\n",
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
" }\n",
" (() => {\n",
" let quickchartButtonEl =\n",
" document.querySelector('#df-dbb90c85-6dc6-4ec9-a4c5-ebcbdb0a5897 button');\n",
" quickchartButtonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
" })();\n",
" </script>\n",
"</div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"text/plain": [
" title artist\n",
"id \n",
"2849 Run To The Hills Iron Maiden\n",
"2640 Red Barchetta Rush\n",
"3167 Unchained Van Halen\n",
"5549 November Rain Guns N' Roses\n",
"2715 Rainbow In The Dark Dio"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print_recommendations(2172)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 310
},
"executionInfo": {
"elapsed": 316,
"status": "ok",
"timestamp": 1719642205517,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "TIHiN62g1NMi",
"outputId": "c548f528-6e2e-4a46-89e0-6599395d6419"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"title California Love (w\\/ Dr. Dre & Roger Troutman)\n",
"artist 2Pac\n",
"Name: 842 , dtype: object\n",
"['5668' '413' '5661' '330' '886']\n"
]
},
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"summary": "{\n \"name\": \"print_recommendations(842)\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"id\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"413 \",\n \"886 \",\n \"5661 \"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"If I Ruled The World (Imagine That) (w\\\\/ Lauryn Hill)\",\n \"Heartless\",\n \"Sweet Dreams\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"artist\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 4,\n \"samples\": [\n \"Nas\",\n \"Kanye West\",\n \"The Game\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}",
"type": "dataframe"
},
"text/html": [
"\n",
" <div id=\"df-1afa899d-2db1-434a-a095-9b7ade3d2589\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>title</th>\n",
" <th>artist</th>\n",
" </tr>\n",
" <tr>\n",
" <th>id</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>5668</th>\n",
" <td>How We Do (w\\/ 50 Cent)</td>\n",
" <td>The Game</td>\n",
" </tr>\n",
" <tr>\n",
" <th>413</th>\n",
" <td>If I Ruled The World (Imagine That) (w\\/ Laury...</td>\n",
" <td>Nas</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5661</th>\n",
" <td>Sweet Dreams</td>\n",
" <td>Beyonce</td>\n",
" </tr>\n",
" <tr>\n",
" <th>330</th>\n",
" <td>Hate It Or Love It (w\\/ 50 Cent)</td>\n",
" <td>The Game</td>\n",
" </tr>\n",
" <tr>\n",
" <th>886</th>\n",
" <td>Heartless</td>\n",
" <td>Kanye West</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1afa899d-2db1-434a-a095-9b7ade3d2589')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-1afa899d-2db1-434a-a095-9b7ade3d2589 button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-1afa899d-2db1-434a-a095-9b7ade3d2589');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
"<div id=\"df-a8ceaf3a-b291-4c01-adfc-895ceccda974\">\n",
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a8ceaf3a-b291-4c01-adfc-895ceccda974')\"\n",
" title=\"Suggest charts\"\n",
" style=\"display:none;\">\n",
"\n",
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <g>\n",
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
" </g>\n",
"</svg>\n",
" </button>\n",
"\n",
"<style>\n",
" .colab-df-quickchart {\n",
" --bg-color: #E8F0FE;\n",
" --fill-color: #1967D2;\n",
" --hover-bg-color: #E2EBFA;\n",
" --hover-fill-color: #174EA6;\n",
" --disabled-fill-color: #AAA;\n",
" --disabled-bg-color: #DDD;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-quickchart {\n",
" --bg-color: #3B4455;\n",
" --fill-color: #D2E3FC;\n",
" --hover-bg-color: #434B5C;\n",
" --hover-fill-color: #FFFFFF;\n",
" --disabled-bg-color: #3B4455;\n",
" --disabled-fill-color: #666;\n",
" }\n",
"\n",
" .colab-df-quickchart {\n",
" background-color: var(--bg-color);\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: var(--fill-color);\n",
" height: 32px;\n",
" padding: 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-quickchart:hover {\n",
" background-color: var(--hover-bg-color);\n",
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: var(--button-hover-fill-color);\n",
" }\n",
"\n",
" .colab-df-quickchart-complete:disabled,\n",
" .colab-df-quickchart-complete:disabled:hover {\n",
" background-color: var(--disabled-bg-color);\n",
" fill: var(--disabled-fill-color);\n",
" box-shadow: none;\n",
" }\n",
"\n",
" .colab-df-spinner {\n",
" border: 2px solid var(--fill-color);\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" animation:\n",
" spin 1s steps(1) infinite;\n",
" }\n",
"\n",
" @keyframes spin {\n",
" 0% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" border-left-color: var(--fill-color);\n",
" }\n",
" 20% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 30% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 40% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 60% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 80% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" 90% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" }\n",
"</style>\n",
"\n",
" <script>\n",
" async function quickchart(key) {\n",
" const quickchartButtonEl =\n",
" document.querySelector('#' + key + ' button');\n",
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
" try {\n",
" const charts = await google.colab.kernel.invokeFunction(\n",
" 'suggestCharts', [key], {});\n",
" } catch (error) {\n",
" console.error('Error during call to suggestCharts:', error);\n",
" }\n",
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
" }\n",
" (() => {\n",
" let quickchartButtonEl =\n",
" document.querySelector('#df-a8ceaf3a-b291-4c01-adfc-895ceccda974 button');\n",
" quickchartButtonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
" })();\n",
" </script>\n",
"</div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"text/plain": [
" title artist\n",
"id \n",
"5668 How We Do (w\\/ 50 Cent) The Game\n",
"413 If I Ruled The World (Imagine That) (w\\/ Laury... Nas\n",
"5661 Sweet Dreams Beyonce\n",
"330 Hate It Or Love It (w\\/ 50 Cent) The Game\n",
"886 Heartless Kanye West"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print_recommendations(842)"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"gpuType": "T4",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink