{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "g_a9QvUFVCUR" }, "source": [ "

Chapter 11 - Fine-tuning Representation Models for Classification

\n", "Exploring the performance in classification of representation models.\n", "\n", "\n", "\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-Models/blob/main/chapter11/Chapter%2011%20-%20Fine-Tuning%20BERT.ipynb)\n", "\n", "---\n", "\n", "This notebook is for Chapter 11 of the [Hands-On Large Language Models](https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961) book by [Jay Alammar](https://www.linkedin.com/in/jalammar) and [Maarten Grootendorst](https://www.linkedin.com/in/mgrootendorst/).\n", "\n", "---\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [OPTIONAL] - Installing Packages on \n", "\n", "If you are viewing this notebook on Google Colab (or any other cloud vendor), you need to **uncomment and run** the following codeblock to install the dependencies for this chapter:\n", "\n", "---\n", "\n", "💡 **NOTE**: We will want to use a GPU to run the examples in this notebook. In Google Colab, go to\n", "**Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4**.\n", "\n", "---\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# %%capture\n", "# !pip install datasets>=2.18.0 transformers>=4.38.2 sentence-transformers>=2.5.1 setfit>=1.0.3 accelerate>=0.27.2 seqeval>=1.2.2" ] }, { "cell_type": "markdown", "metadata": { "id": "UBeVnXxQWy7-" }, "source": [ "## **Data**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 365, "referenced_widgets": [ "e3c168a65bd34a64b2c4d13129f9d750", "5ad968464c934a15b19e3924663e5ebf", "31756f672e60432da8aa12b033a2ea20", "63adb81b029744fb82e31b3f1190bb9b", "77511fda45a940d1a6664438b07712b2", "677310bbba174f3a828d7f625a483f92", "bbb27e92d9e848e9ada7ba0c9866f87a", "d6299b347ade47bba93df3eff231be6f", "49541355ed224e748754ed4bd89792d8", "7cc8919e7f9e4db48a876619edff0952", "fdb9b32fdccf4156b537df8e000ec7d2", "0cd7314417ab48f99adf352f0cab863f", "bbb68b7249674351abfa8661b4645e2e", "285d6e493dcb462598190862b186b17e", "29419f20fad94db88b6da3b1ecd04725", "3d58f1665e0949d1baa1feaffd52aa29", "82c561fe82084baaa9eedfb78ec476da", "10165d819bea43f1ad531ee0a314d002", "ed2ca51dff8847648010e7b28f03d41e", "9f216034e74a4f0a9794a812addc82c5", "d3748aec876a4ddbb377f16a43b3c23e", "081b5fcad38b4627b1d553244a1bed0c", "3a4ca5b37d6f426a9d294ade901f6984", "673f06ed6138437680a01409cbce3a13", "625db697c4084a4fa7f78905c5c830b2", "a6c837a7a4fb45bcb035537cf41796e3", "0f53e4afed4a42a5a63d8a64f847c27e", "127978e95632464fb3b7396de66ff240", "a7fa1b4de3b6423783861a6d8213f430", "c185b6d9a8514c71b36918027a424204", "9fc58f20135e48b897cb6a2b8a67863d", "24b94b5fc6fd4aa7ae113d9eeb3a4d6f", "7c386e049403493997772cf4aca4a234", "09a021dba22548f3bd101e237c5ce51d", "36fdfe575de440458c4fe7cd572d2a22", "7ef0c97a8b774e57a041a210c3e23ce6", "d9379ba58d8e4de9b4ba3cc12be87b4c", "ee301aff657e405c8cdfd2e0f5d7d19e", "87d2c368a3d44c1994c8fe31fea59212", "0db4bd167da44ee7b9fc044d9bb5834d", "ebc2ddfef4cd4129997138c615cd15a1", "1b0fa0592da147fcab20f0c17f487542", "0a5befced7554f0a92523b5dca953ed4", "ec515b2a14ff4414baddca0cc672ef14", "2eaded0fbbd74c67b802fbb92973fa3a", "d1e9c32d66004c70b0e3fa96ccb1e4e3", "d27a323324cd45789808c22917fa5d36", "7d827c0ee22b47c4b46fd4f59ac3ec23", "aba270b810d6461887633f62cf35195c", "f43737300b254f2bb3aef90fabf652e2", "ff95c667621d46b89412c0638ce91673", "5f810926256a4a3aa5271c504cdad278", "117e9997a5db42e59e1f73eddd345b25", "c807a7269c2a4f8daf8022a177175213", "c123b5dae69741d6a10b0b783f5e54ac", "5a5d72b8d2484c54977e5ec692a00004", "fd5ff63e2c204836b1c6924633880077", "1c96b599ae6d466e94062f21e7d3a54b", "690d9fa381b74085af8025af1dcffdcf", "95ab40e005a9419aa7827dbcbd8a2415", "aeffa7363e144c0a9b830c7b532167cb", "59cd08daef304e78be3574b4345084b2", "1700973179db4c91a9c00fe5e84de1a9", "5a22333fd16d46429e16f8b1ed9276a1", "34e7d842cf364b9f8b377b17befa20b2", "84454ba3262b48d489215b874bd78842", "91552574464449e48e54a6939f0a5d18", "0f4c3ac40bf84e6293419447a7fbdd15", "c82ed37a975149d8895adc331aae34eb", "96252dca0e9049fea03228e7ceba066b", "25ed22abc1934a3393593d5b10331988", "b883ce44ce9e47d6b5ac70e36e049559", "17afecdfb20b4e49a79aea97d20a10dd", "c6b1edc0dece4354b7a43314c419cb71", "19b3c0979b3f45499faa57f341d6d53a", "ef4d27b4292240928439104f38c610b9", "ce07f53ddea3469282b20581a4f4e4da" ] }, "executionInfo": { "elapsed": 21399, "status": "ok", "timestamp": 1719386835780, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "5phRS_z2U_3T", "outputId": "439a9900-6d9a-4a53-ea30-4774d3bf4de7" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n", "The secret `HF_TOKEN` does not exist in your Colab secrets.\n", "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n", "You will be able to reuse this secret in all of your notebooks.\n", "Please note that authentication is recommended but still optional to access public models or datasets.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e3c168a65bd34a64b2c4d13129f9d750", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading readme: 0%| | 0.00/7.46k [00:00\n", " \n", " \n", " [534/534 01:00, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
5000.424000

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "TrainOutput(global_step=534, training_loss=0.4183677966228585, metrics={'train_runtime': 61.6658, 'train_samples_per_second': 138.326, 'train_steps_per_second': 8.66, 'total_flos': 227605451772240.0, 'train_loss': 0.4183677966228585, 'epoch': 1.0})" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.train()" ] }, { "cell_type": "markdown", "metadata": { "id": "xkBUVlUYbUnn" }, "source": [ "Evaluate results." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 173, "referenced_widgets": [ "d656514d509441548af1dd6c7326d18a", "05ce4f6b47354c628c0a41544b9419f5", "d55ba50d84d7412daeb6ccc73be98694", "d5d54eddd21a4999aec6ea6a7782e223", "e8da2d90a93b4ce9a93c2ee93f75d59b", "b94e1b170ccc4ff7abbb9ed87ad2aedd", "3c38333cafb446bab3f3f75824075927", "960641d3e97f4a538a135861ab058bca", "fc4d3609cada4615924d5e92ee1ffdf4", "f914214414b14945a72ac57ac93dcb5e", "8cb0531c591c43f098564ce043eb036f" ] }, "executionInfo": { "elapsed": 3959, "status": "ok", "timestamp": 1719386999026, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "wCI9uYDObWU8", "outputId": "5c2d54d2-0c81-4a10-d46d-9d7c4c9405d3" }, "outputs": [ { "data": { "text/html": [ "\n", "

\n", " \n", " \n", " [67/67 00:01]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d656514d509441548af1dd6c7326d18a", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading builder script: 0%| | 0.00/6.77k [00:00\n", " \n", " \n", " [534/534 00:15, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
5000.697000

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "TrainOutput(global_step=534, training_loss=0.6962381677234664, metrics={'train_runtime': 15.234, 'train_samples_per_second': 559.931, 'train_steps_per_second': 35.053, 'total_flos': 227605451772240.0, 'train_loss': 0.6962381677234664, 'epoch': 1.0})" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from transformers import TrainingArguments, Trainer\n", "\n", "# Trainer which executes the training process\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=tokenized_train,\n", " eval_dataset=tokenized_test,\n", " tokenizer=tokenizer,\n", " data_collator=data_collator,\n", " compute_metrics=compute_metrics,\n", ")\n", "trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 141 }, "executionInfo": { "elapsed": 2623, "status": "ok", "timestamp": 1719387017125, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "eCPpixB1HCsI", "outputId": "3d77ed38-0565-492b-b488-09eb525fc316" }, "outputs": [ { "data": { "text/html": [ "\n", "

\n", " \n", " \n", " [67/67 00:01]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'eval_loss': 0.6823198795318604,\n", " 'eval_f1': 0.637704918032787,\n", " 'eval_runtime': 2.7203,\n", " 'eval_samples_per_second': 391.865,\n", " 'eval_steps_per_second': 24.629,\n", " 'epoch': 1.0}" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.evaluate()" ] }, { "cell_type": "markdown", "metadata": { "id": "Uw729mLhIQL6" }, "source": [ "### Freeze blocks 1-5" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 640, "status": "ok", "timestamp": 1719387017762, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "RbsLR561Kje-", "outputId": "ae1b3e9b-443d-4a06-94bb-6792d1751e27" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parameter: 0bert.embeddings.word_embeddings.weight ----- False\n", "Parameter: 1bert.embeddings.position_embeddings.weight ----- False\n", "Parameter: 2bert.embeddings.token_type_embeddings.weight ----- False\n", "Parameter: 3bert.embeddings.LayerNorm.weight ----- False\n", "Parameter: 4bert.embeddings.LayerNorm.bias ----- False\n", "Parameter: 5bert.encoder.layer.0.attention.self.query.weight ----- False\n", "Parameter: 6bert.encoder.layer.0.attention.self.query.bias ----- False\n", "Parameter: 7bert.encoder.layer.0.attention.self.key.weight ----- False\n", "Parameter: 8bert.encoder.layer.0.attention.self.key.bias ----- False\n", "Parameter: 9bert.encoder.layer.0.attention.self.value.weight ----- False\n", "Parameter: 10bert.encoder.layer.0.attention.self.value.bias ----- False\n", "Parameter: 11bert.encoder.layer.0.attention.output.dense.weight ----- False\n", "Parameter: 12bert.encoder.layer.0.attention.output.dense.bias ----- False\n", "Parameter: 13bert.encoder.layer.0.attention.output.LayerNorm.weight ----- False\n", "Parameter: 14bert.encoder.layer.0.attention.output.LayerNorm.bias ----- False\n", "Parameter: 15bert.encoder.layer.0.intermediate.dense.weight ----- False\n", "Parameter: 16bert.encoder.layer.0.intermediate.dense.bias ----- False\n", "Parameter: 17bert.encoder.layer.0.output.dense.weight ----- False\n", "Parameter: 18bert.encoder.layer.0.output.dense.bias ----- False\n", "Parameter: 19bert.encoder.layer.0.output.LayerNorm.weight ----- False\n", "Parameter: 20bert.encoder.layer.0.output.LayerNorm.bias ----- False\n", "Parameter: 21bert.encoder.layer.1.attention.self.query.weight ----- False\n", "Parameter: 22bert.encoder.layer.1.attention.self.query.bias ----- False\n", "Parameter: 23bert.encoder.layer.1.attention.self.key.weight ----- False\n", "Parameter: 24bert.encoder.layer.1.attention.self.key.bias ----- False\n", "Parameter: 25bert.encoder.layer.1.attention.self.value.weight ----- False\n", "Parameter: 26bert.encoder.layer.1.attention.self.value.bias ----- False\n", "Parameter: 27bert.encoder.layer.1.attention.output.dense.weight ----- False\n", "Parameter: 28bert.encoder.layer.1.attention.output.dense.bias ----- False\n", "Parameter: 29bert.encoder.layer.1.attention.output.LayerNorm.weight ----- False\n", "Parameter: 30bert.encoder.layer.1.attention.output.LayerNorm.bias ----- False\n", "Parameter: 31bert.encoder.layer.1.intermediate.dense.weight ----- False\n", "Parameter: 32bert.encoder.layer.1.intermediate.dense.bias ----- False\n", "Parameter: 33bert.encoder.layer.1.output.dense.weight ----- False\n", "Parameter: 34bert.encoder.layer.1.output.dense.bias ----- False\n", "Parameter: 35bert.encoder.layer.1.output.LayerNorm.weight ----- False\n", "Parameter: 36bert.encoder.layer.1.output.LayerNorm.bias ----- False\n", "Parameter: 37bert.encoder.layer.2.attention.self.query.weight ----- False\n", "Parameter: 38bert.encoder.layer.2.attention.self.query.bias ----- False\n", "Parameter: 39bert.encoder.layer.2.attention.self.key.weight ----- False\n", "Parameter: 40bert.encoder.layer.2.attention.self.key.bias ----- False\n", "Parameter: 41bert.encoder.layer.2.attention.self.value.weight ----- False\n", "Parameter: 42bert.encoder.layer.2.attention.self.value.bias ----- False\n", "Parameter: 43bert.encoder.layer.2.attention.output.dense.weight ----- False\n", "Parameter: 44bert.encoder.layer.2.attention.output.dense.bias ----- False\n", "Parameter: 45bert.encoder.layer.2.attention.output.LayerNorm.weight ----- False\n", "Parameter: 46bert.encoder.layer.2.attention.output.LayerNorm.bias ----- False\n", "Parameter: 47bert.encoder.layer.2.intermediate.dense.weight ----- False\n", "Parameter: 48bert.encoder.layer.2.intermediate.dense.bias ----- False\n", "Parameter: 49bert.encoder.layer.2.output.dense.weight ----- False\n", "Parameter: 50bert.encoder.layer.2.output.dense.bias ----- False\n", "Parameter: 51bert.encoder.layer.2.output.LayerNorm.weight ----- False\n", "Parameter: 52bert.encoder.layer.2.output.LayerNorm.bias ----- False\n", "Parameter: 53bert.encoder.layer.3.attention.self.query.weight ----- False\n", "Parameter: 54bert.encoder.layer.3.attention.self.query.bias ----- False\n", "Parameter: 55bert.encoder.layer.3.attention.self.key.weight ----- False\n", "Parameter: 56bert.encoder.layer.3.attention.self.key.bias ----- False\n", "Parameter: 57bert.encoder.layer.3.attention.self.value.weight ----- False\n", "Parameter: 58bert.encoder.layer.3.attention.self.value.bias ----- False\n", "Parameter: 59bert.encoder.layer.3.attention.output.dense.weight ----- False\n", "Parameter: 60bert.encoder.layer.3.attention.output.dense.bias ----- False\n", "Parameter: 61bert.encoder.layer.3.attention.output.LayerNorm.weight ----- False\n", "Parameter: 62bert.encoder.layer.3.attention.output.LayerNorm.bias ----- False\n", "Parameter: 63bert.encoder.layer.3.intermediate.dense.weight ----- False\n", "Parameter: 64bert.encoder.layer.3.intermediate.dense.bias ----- False\n", "Parameter: 65bert.encoder.layer.3.output.dense.weight ----- False\n", "Parameter: 66bert.encoder.layer.3.output.dense.bias ----- False\n", "Parameter: 67bert.encoder.layer.3.output.LayerNorm.weight ----- False\n", "Parameter: 68bert.encoder.layer.3.output.LayerNorm.bias ----- False\n", "Parameter: 69bert.encoder.layer.4.attention.self.query.weight ----- False\n", "Parameter: 70bert.encoder.layer.4.attention.self.query.bias ----- False\n", "Parameter: 71bert.encoder.layer.4.attention.self.key.weight ----- False\n", "Parameter: 72bert.encoder.layer.4.attention.self.key.bias ----- False\n", "Parameter: 73bert.encoder.layer.4.attention.self.value.weight ----- False\n", "Parameter: 74bert.encoder.layer.4.attention.self.value.bias ----- False\n", "Parameter: 75bert.encoder.layer.4.attention.output.dense.weight ----- False\n", "Parameter: 76bert.encoder.layer.4.attention.output.dense.bias ----- False\n", "Parameter: 77bert.encoder.layer.4.attention.output.LayerNorm.weight ----- False\n", "Parameter: 78bert.encoder.layer.4.attention.output.LayerNorm.bias ----- False\n", "Parameter: 79bert.encoder.layer.4.intermediate.dense.weight ----- False\n", "Parameter: 80bert.encoder.layer.4.intermediate.dense.bias ----- False\n", "Parameter: 81bert.encoder.layer.4.output.dense.weight ----- False\n", "Parameter: 82bert.encoder.layer.4.output.dense.bias ----- False\n", "Parameter: 83bert.encoder.layer.4.output.LayerNorm.weight ----- False\n", "Parameter: 84bert.encoder.layer.4.output.LayerNorm.bias ----- False\n", "Parameter: 85bert.encoder.layer.5.attention.self.query.weight ----- False\n", "Parameter: 86bert.encoder.layer.5.attention.self.query.bias ----- False\n", "Parameter: 87bert.encoder.layer.5.attention.self.key.weight ----- False\n", "Parameter: 88bert.encoder.layer.5.attention.self.key.bias ----- False\n", "Parameter: 89bert.encoder.layer.5.attention.self.value.weight ----- False\n", "Parameter: 90bert.encoder.layer.5.attention.self.value.bias ----- False\n", "Parameter: 91bert.encoder.layer.5.attention.output.dense.weight ----- False\n", "Parameter: 92bert.encoder.layer.5.attention.output.dense.bias ----- False\n", "Parameter: 93bert.encoder.layer.5.attention.output.LayerNorm.weight ----- False\n", "Parameter: 94bert.encoder.layer.5.attention.output.LayerNorm.bias ----- False\n", "Parameter: 95bert.encoder.layer.5.intermediate.dense.weight ----- False\n", "Parameter: 96bert.encoder.layer.5.intermediate.dense.bias ----- False\n", "Parameter: 97bert.encoder.layer.5.output.dense.weight ----- False\n", "Parameter: 98bert.encoder.layer.5.output.dense.bias ----- False\n", "Parameter: 99bert.encoder.layer.5.output.LayerNorm.weight ----- False\n", "Parameter: 100bert.encoder.layer.5.output.LayerNorm.bias ----- False\n", "Parameter: 101bert.encoder.layer.6.attention.self.query.weight ----- False\n", "Parameter: 102bert.encoder.layer.6.attention.self.query.bias ----- False\n", "Parameter: 103bert.encoder.layer.6.attention.self.key.weight ----- False\n", "Parameter: 104bert.encoder.layer.6.attention.self.key.bias ----- False\n", "Parameter: 105bert.encoder.layer.6.attention.self.value.weight ----- False\n", "Parameter: 106bert.encoder.layer.6.attention.self.value.bias ----- False\n", "Parameter: 107bert.encoder.layer.6.attention.output.dense.weight ----- False\n", "Parameter: 108bert.encoder.layer.6.attention.output.dense.bias ----- False\n", "Parameter: 109bert.encoder.layer.6.attention.output.LayerNorm.weight ----- False\n", "Parameter: 110bert.encoder.layer.6.attention.output.LayerNorm.bias ----- False\n", "Parameter: 111bert.encoder.layer.6.intermediate.dense.weight ----- False\n", "Parameter: 112bert.encoder.layer.6.intermediate.dense.bias ----- False\n", "Parameter: 113bert.encoder.layer.6.output.dense.weight ----- False\n", "Parameter: 114bert.encoder.layer.6.output.dense.bias ----- False\n", "Parameter: 115bert.encoder.layer.6.output.LayerNorm.weight ----- False\n", "Parameter: 116bert.encoder.layer.6.output.LayerNorm.bias ----- False\n", "Parameter: 117bert.encoder.layer.7.attention.self.query.weight ----- False\n", "Parameter: 118bert.encoder.layer.7.attention.self.query.bias ----- False\n", "Parameter: 119bert.encoder.layer.7.attention.self.key.weight ----- False\n", "Parameter: 120bert.encoder.layer.7.attention.self.key.bias ----- False\n", "Parameter: 121bert.encoder.layer.7.attention.self.value.weight ----- False\n", "Parameter: 122bert.encoder.layer.7.attention.self.value.bias ----- False\n", "Parameter: 123bert.encoder.layer.7.attention.output.dense.weight ----- False\n", "Parameter: 124bert.encoder.layer.7.attention.output.dense.bias ----- False\n", "Parameter: 125bert.encoder.layer.7.attention.output.LayerNorm.weight ----- False\n", "Parameter: 126bert.encoder.layer.7.attention.output.LayerNorm.bias ----- False\n", "Parameter: 127bert.encoder.layer.7.intermediate.dense.weight ----- False\n", "Parameter: 128bert.encoder.layer.7.intermediate.dense.bias ----- False\n", "Parameter: 129bert.encoder.layer.7.output.dense.weight ----- False\n", "Parameter: 130bert.encoder.layer.7.output.dense.bias ----- False\n", "Parameter: 131bert.encoder.layer.7.output.LayerNorm.weight ----- False\n", "Parameter: 132bert.encoder.layer.7.output.LayerNorm.bias ----- False\n", "Parameter: 133bert.encoder.layer.8.attention.self.query.weight ----- False\n", "Parameter: 134bert.encoder.layer.8.attention.self.query.bias ----- False\n", "Parameter: 135bert.encoder.layer.8.attention.self.key.weight ----- False\n", "Parameter: 136bert.encoder.layer.8.attention.self.key.bias ----- False\n", "Parameter: 137bert.encoder.layer.8.attention.self.value.weight ----- False\n", "Parameter: 138bert.encoder.layer.8.attention.self.value.bias ----- False\n", "Parameter: 139bert.encoder.layer.8.attention.output.dense.weight ----- False\n", "Parameter: 140bert.encoder.layer.8.attention.output.dense.bias ----- False\n", "Parameter: 141bert.encoder.layer.8.attention.output.LayerNorm.weight ----- False\n", "Parameter: 142bert.encoder.layer.8.attention.output.LayerNorm.bias ----- False\n", "Parameter: 143bert.encoder.layer.8.intermediate.dense.weight ----- False\n", "Parameter: 144bert.encoder.layer.8.intermediate.dense.bias ----- False\n", "Parameter: 145bert.encoder.layer.8.output.dense.weight ----- False\n", "Parameter: 146bert.encoder.layer.8.output.dense.bias ----- False\n", "Parameter: 147bert.encoder.layer.8.output.LayerNorm.weight ----- False\n", "Parameter: 148bert.encoder.layer.8.output.LayerNorm.bias ----- False\n", "Parameter: 149bert.encoder.layer.9.attention.self.query.weight ----- False\n", "Parameter: 150bert.encoder.layer.9.attention.self.query.bias ----- False\n", "Parameter: 151bert.encoder.layer.9.attention.self.key.weight ----- False\n", "Parameter: 152bert.encoder.layer.9.attention.self.key.bias ----- False\n", "Parameter: 153bert.encoder.layer.9.attention.self.value.weight ----- False\n", "Parameter: 154bert.encoder.layer.9.attention.self.value.bias ----- False\n", "Parameter: 155bert.encoder.layer.9.attention.output.dense.weight ----- False\n", "Parameter: 156bert.encoder.layer.9.attention.output.dense.bias ----- False\n", "Parameter: 157bert.encoder.layer.9.attention.output.LayerNorm.weight ----- False\n", "Parameter: 158bert.encoder.layer.9.attention.output.LayerNorm.bias ----- False\n", "Parameter: 159bert.encoder.layer.9.intermediate.dense.weight ----- False\n", "Parameter: 160bert.encoder.layer.9.intermediate.dense.bias ----- False\n", "Parameter: 161bert.encoder.layer.9.output.dense.weight ----- False\n", "Parameter: 162bert.encoder.layer.9.output.dense.bias ----- False\n", "Parameter: 163bert.encoder.layer.9.output.LayerNorm.weight ----- False\n", "Parameter: 164bert.encoder.layer.9.output.LayerNorm.bias ----- False\n", "Parameter: 165bert.encoder.layer.10.attention.self.query.weight ----- False\n", "Parameter: 166bert.encoder.layer.10.attention.self.query.bias ----- False\n", "Parameter: 167bert.encoder.layer.10.attention.self.key.weight ----- False\n", "Parameter: 168bert.encoder.layer.10.attention.self.key.bias ----- False\n", "Parameter: 169bert.encoder.layer.10.attention.self.value.weight ----- False\n", "Parameter: 170bert.encoder.layer.10.attention.self.value.bias ----- False\n", "Parameter: 171bert.encoder.layer.10.attention.output.dense.weight ----- False\n", "Parameter: 172bert.encoder.layer.10.attention.output.dense.bias ----- False\n", "Parameter: 173bert.encoder.layer.10.attention.output.LayerNorm.weight ----- False\n", "Parameter: 174bert.encoder.layer.10.attention.output.LayerNorm.bias ----- False\n", "Parameter: 175bert.encoder.layer.10.intermediate.dense.weight ----- False\n", "Parameter: 176bert.encoder.layer.10.intermediate.dense.bias ----- False\n", "Parameter: 177bert.encoder.layer.10.output.dense.weight ----- False\n", "Parameter: 178bert.encoder.layer.10.output.dense.bias ----- False\n", "Parameter: 179bert.encoder.layer.10.output.LayerNorm.weight ----- False\n", "Parameter: 180bert.encoder.layer.10.output.LayerNorm.bias ----- False\n", "Parameter: 181bert.encoder.layer.11.attention.self.query.weight ----- False\n", "Parameter: 182bert.encoder.layer.11.attention.self.query.bias ----- False\n", "Parameter: 183bert.encoder.layer.11.attention.self.key.weight ----- False\n", "Parameter: 184bert.encoder.layer.11.attention.self.key.bias ----- False\n", "Parameter: 185bert.encoder.layer.11.attention.self.value.weight ----- False\n", "Parameter: 186bert.encoder.layer.11.attention.self.value.bias ----- False\n", "Parameter: 187bert.encoder.layer.11.attention.output.dense.weight ----- False\n", "Parameter: 188bert.encoder.layer.11.attention.output.dense.bias ----- False\n", "Parameter: 189bert.encoder.layer.11.attention.output.LayerNorm.weight ----- False\n", "Parameter: 190bert.encoder.layer.11.attention.output.LayerNorm.bias ----- False\n", "Parameter: 191bert.encoder.layer.11.intermediate.dense.weight ----- False\n", "Parameter: 192bert.encoder.layer.11.intermediate.dense.bias ----- False\n", "Parameter: 193bert.encoder.layer.11.output.dense.weight ----- False\n", "Parameter: 194bert.encoder.layer.11.output.dense.bias ----- False\n", "Parameter: 195bert.encoder.layer.11.output.LayerNorm.weight ----- False\n", "Parameter: 196bert.encoder.layer.11.output.LayerNorm.bias ----- False\n", "Parameter: 197bert.pooler.dense.weight ----- False\n", "Parameter: 198bert.pooler.dense.bias ----- False\n", "Parameter: 199classifier.weight ----- True\n", "Parameter: 200classifier.bias ----- True\n" ] } ], "source": [ "# We can check whether the model was correctly updated\n", "for index, (name, param) in enumerate(model.named_parameters()):\n", " print(f\"Parameter: {index}{name} ----- {param.requires_grad}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 285 }, "executionInfo": { "elapsed": 24489, "status": "ok", "timestamp": 1719387042249, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "QyleqOHICBjj", "outputId": "f2dc589c-1b8f-40e7-ef93-3ba0f3d3a3a3" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [534/534 00:21, Epoch 1/1]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
5000.474600

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "

\n", " \n", " \n", " [67/67 00:01]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'eval_loss': 0.4092540740966797,\n", " 'eval_f1': 0.8141086749285034,\n", " 'eval_runtime': 2.7437,\n", " 'eval_samples_per_second': 388.523,\n", " 'eval_steps_per_second': 24.419,\n", " 'epoch': 1.0}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load model\n", "model_id = \"bert-base-cased\"\n", "model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)\n", "tokenizer = AutoTokenizer.from_pretrained(model_id)\n", "\n", "# Encoder block 10 starts at index 165 and\n", "# we freeze everything before that block\n", "for index, (name, param) in enumerate(model.named_parameters()):\n", " if index < 165:\n", " param.requires_grad = False\n", "\n", "# Trainer which executes the training process\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=tokenized_train,\n", " eval_dataset=tokenized_test,\n", " tokenizer=tokenizer,\n", " data_collator=data_collator,\n", " compute_metrics=compute_metrics,\n", ")\n", "trainer.train()\n", "trainer.evaluate()" ] }, { "cell_type": "markdown", "metadata": { "id": "HJRMWsLdA913" }, "source": [ "### [BONUS] Freeze blocks" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ADvKJjNaFAot" }, "outputs": [], "source": [ "# scores = []\n", "# for index in range(12):\n", "# # Re-load model\n", "# model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=2)\n", "# tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n", "\n", "# # Freeze encoder blocks 0-index\n", "# for name, param in model.named_parameters():\n", "# if \"layer\" in name:\n", "# layer_nr = int(name.split(\"layer\")[1].split(\".\")[1])\n", "# if layer_nr <= index:\n", "# param.requires_grad = False\n", "# else:\n", "# param.requires_grad = True\n", "\n", "# # Train\n", "# trainer = Trainer(\n", "# model=model,\n", "# args=training_args,\n", "# train_dataset=tokenized_train,\n", "# eval_dataset=tokenized_test,\n", "# tokenizer=tokenizer,\n", "# data_collator=data_collator,\n", "# compute_metrics=compute_metrics,\n", "# )\n", "# trainer.train()\n", "\n", "# # Evaluate\n", "# score = trainer.evaluate()[\"eval_f1\"]\n", "# scores.append(score)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 212, "status": "ok", "timestamp": 1712321357732, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "dWYlHFNdLQtk", "outputId": "a28d645b-7633-4bdf-cc51-b7a937d36508" }, "outputs": [ { "data": { "text/plain": [ "[0.8541862652869239,\n", " 0.8525519848771267,\n", " 0.8514664143803217,\n", " 0.8506616257088847,\n", " 0.8398104265402844,\n", " 0.8391345249294448,\n", " 0.8377358490566037,\n", " 0.8433962264150944,\n", " 0.8258801141769743,\n", " 0.816247582205029,\n", " 0.7917485265225934,\n", " 0.7019400352733686]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# scores" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 410 }, "executionInfo": { "elapsed": 1383, "status": "ok", "timestamp": 1712388601684, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "bf3PIvKhOBJ-", "outputId": "aadf0ff7-e4ab-4d62-87b5-cbf1ecdea8da" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAGJCAYAAABo5eDAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACUfklEQVR4nOzdd1iT19sH8G8SSMJGmU5w4hYFQXGvupU66qiK+2fVaktrK3VVW7XT2m2tqK21Ql2tEwdK3WJxD3CLWgUXWxJIzvuHJa8RkCEkIXw/15VLc/KM+86TcfPkPOdIhBACRERERERmSmrsAIiIiIiIShMLXiIiIiIyayx4iYiIiMisseAlIiIiIrPGgpeIiIiIzBoLXiIiIiIyayx4iYiIiMisseAlIiIiIrPGgpeIiIiIzBoLXjKKtLQ0jBs3Du7u7pBIJHjrrbcAAAkJCRg4cCCcnJwgkUiwZMkSo8ZZFPnlRGVTVFQUJBIJoqKijB1KLp6enhg1alSpbPvGjRuQSCT44osvSmX75mTUqFHw9PQs1roffvghJBJJyQZkgvi5SKaCBS+VmFWrVkEikeR7O3r0qG7ZhQsXYtWqVXjjjTewevVqjBgxAgDw9ttvY+fOnQgJCcHq1avRvXv3Eo9z4cKF+PPPP0tlu3nllBdPT898n6fMzMwSj82YcgrH/G5hYWHGDtHo8nqOKlasiJYtW2LNmjXGDq/MedHr7dmbKf4xYwijRo3Sex7s7e3RtGlTfPnll1CpVCW6r6J8LhKVJgtjB0DmZ/78+ahRo0au9tq1a+v+v3fvXrRs2RJz587VW2bv3r3o168f3n333VKLb+HChRg4cCACAwNLdLv55ZQfb29vvPPOO7na5XJ5icZlKqZOnYoWLVrkam/VqpURojFNzz5HDx8+RHh4OIYPH46kpCRMnjzZyNGVHatXr9a7/+uvv2L37t252uvXr/9S+/n555+h1WqLte6sWbMwY8aMl9r/y1AoFFi+fDkAICkpCRs2bMC7776L48ePl+gfoUX9XCQqLSx4qcT16NEDvr6+L1wmMTERDRo0yLPd0dGxlCIrXfnllJ8qVapg+PDhhV4+IyMD1tbWxQnNJLRt2xYDBw40dhhGk56eDhsbmxcu8/xz9MYbb6BmzZr4/fffWfAWwfPvq6NHj2L37t0Fvt+K+h6ztLQsVnwAYGFhAQsL430FW1hY6D0fkyZNgr+/P8LDw7F48WJUrly52NvWarVQq9VQKpVF/lwsSHZ2NrRardmeGKDSwy4NZFA5P91ev34d27Zt0/2kltMdQgiB77//XteeIykpCW+99RaqVasGhUKB2rVr49NPP811dkWr1eLrr79G48aNoVQq4eLigu7du+Off/4B8PSnzvT0dPzyyy+6fRTUFzIxMRFjx46Fm5sblEolmjZtil9++aXAnG7cuFHs56lDhw5o1KgRYmJi0K5dO1hbW+ODDz4oVDw56+f3M+6qVauK9Lw+26dz2bJlqFWrFhQKBVq0aIHjx48XO8e8SCQSTJkyBX/++ScaNWoEhUKBhg0bIiIiIteyd+7cwdixY1G5cmUoFArUqFEDb7zxBtRqtW6Za9euYdCgQahYsSKsra3RsmVLbNu2Lde2bt++jcDAQNjY2MDV1RVvv/12vj/tHjt2DN27d4eDgwOsra3Rvn17HDp0SG+ZnP6ZFy5cwLBhw1ChQgW0adOmyM+HXC5HhQoVClUYFTbXzMxMfPjhh6hbty6USiUqVaqE/v374+rVq/luWwiBCRMmQC6XY+PGjQCArKwszJs3D3Xq1IFSqYSTkxPatGmD3bt3l0isOe+rP/74AwsWLEDVqlWhVCrRuXNnXLlypcB9FORF77G//voLvXr10r22atWqhY8++ggajUZvG8/34S3KeyWvPrxFef1HRUXB19cXSqUStWrVwk8//fRS/YKlUik6dOigywMAVCoV5s6di9q1a0OhUKBatWp47733cr03cuJes2YNGjZsCIVCgYiIiBd+Lhbmc+zZ53PJkiW65/PChQu6XC9duoThw4fDwcEBLi4umD17NoQQuHXrFvr16wd7e3u4u7vjyy+/1Nu2Wq3GnDlz4OPjAwcHB9jY2KBt27bYt29fvjEU5vMvNjYWr732GlxcXGBlZQUvLy/MnDlTb5k7d+5gzJgxcHNz0x3jFStWFPWQURHxDC+VuOTkZDx48ECvTSKRwMnJCfXr18fq1avx9ttvo2rVqrqf9Js1a6br39W1a1eMHDlSt25GRgbat2+PO3fu4H//+x+qV6+Ow4cPIyQkBHfv3tW7sG3s2LFYtWoVevTogXHjxiE7OxsHDhzA0aNH4evri9WrV2PcuHHw8/PDhAkTAAC1atXKN5cnT56gQ4cOuHLlCqZMmYIaNWpg3bp1GDVqFJKSkjBt2rR8c3JxcXnh85SVlZXrebK2ttadYXr48CF69OiBIUOGYPjw4XBzcytUPAAwc+ZMjBs3Tm/bv/32G3bu3AlXV9ciP68A8PvvvyM1NRX/+9//IJFI8Nlnn6F///64du1aoc50paam5soXgO4CxRwHDx7Exo0bMWnSJNjZ2eGbb77BgAEDEB8fDycnJwDAv//+Cz8/PyQlJWHChAmoV68e7ty5g/Xr1yMjIwNyuRwJCQkICAhARkYGpk6dCicnJ/zyyy/o27cv1q9fj1dffRXA02PcuXNnxMfHY+rUqahcuTJWr16NvXv35op179696NGjB3x8fDB37lxIpVKsXLkSnTp1woEDB+Dn56e3/KBBg1CnTh0sXLgQQogiPUePHj3C77//jnPnziE0NPSF6xU2V41Gg969eyMyMhJDhgzBtGnTkJqait27d+PcuXN5vhc0Gg3GjBmD8PBwbNq0Cb169QLwtGBbtGiR7v2UkpKCf/75BydOnEDXrl1fOtYcn3zyCaRSKd59910kJyfjs88+w+uvv45jx44V+HwWJK/3GPD0egRbW1sEBwfD1tYWe/fuxZw5c5CSkoLPP/+8wO2+zHulMK//kydPonv37qhUqRLmzZsHjUaD+fPnF/iZU5CcP3qcnJyg1WrRt29fHDx4EBMmTED9+vVx9uxZfPXVV7h06VKu6yD27t2LP/74A1OmTIGzszMqVaqU7+diYT/HcqxcuRKZmZmYMGECFAoFKlasqHts8ODBqF+/Pj755BNs27YNH3/8MSpWrIiffvoJnTp1wqeffoo1a9bg3XffRYsWLdCuXTsAQEpKCpYvX46hQ4di/PjxSE1NRWhoKLp164bo6Gh4e3vrxVCYY3rmzBm0bdsWlpaWmDBhAjw9PXH16lVs2bIFCxYsAPD09d+yZUvdHwkuLi7YsWMHxo4di5SUFF7UV5oEUQlZuXKlAJDnTaFQ6C3r4eEhevXqlWsbAMTkyZP12j766CNhY2MjLl26pNc+Y8YMIZPJRHx8vBBCiL179woAYurUqbm2q9Vqdf+3sbERQUFBhcppyZIlAoD47bffdG1qtVq0atVK2NraipSUlAJzyouHh0eez9PcuXOFEEK0b99eABBLly4tdjzPOnTokLC0tBRjxozRtRX2eb1+/boAIJycnMSjR490y/31118CgNiyZcsLc923b1++rwsA4u7du7plAQi5XC6uXLmiazt9+rQAIL799ltd28iRI4VUKhXHjx/Ptb+cY/3WW28JAOLAgQO6x1JTU0WNGjWEp6en0Gg0Qoj/f07/+OMP3XLp6emidu3aAoDYt2+fbrt16tQR3bp103s9ZWRkiBo1aoiuXbvq2ubOnSsAiKFDh77wuSnoOZJKpWLBggW5lvfw8NB7DRc21xUrVggAYvHixfk+bznH+/PPPxdZWVli8ODBwsrKSuzcuVNv+aZNmxb69f6swsaa85zUr19fqFQq3bJff/21ACDOnj1b6H1OnjxZPP91l997TIinx/R5//vf/4S1tbXIzMzUtQUFBQkPDw/d/aK8V3JeI88q7Ou/T58+wtraWty5c0fXdvnyZWFhYZFrm3kJCgoSNjY24v79++L+/fviypUrYuHChUIikYgmTZoIIYRYvXq1kEqlesdJCCGWLl0qAIhDhw7pxS2VSsX58+dz7Suvz8XCfo7lPJ/29vYiMTFRbxs5z9+ECRN0bdnZ2aJq1apCIpGITz75RNf++PFjYWVlpfeeyc7O1ntd5Szn5uam9zlZlGParl07YWdnJ27evKm33Wc/L8aOHSsqVaokHjx4oLfMkCFDhIODQ56vPSoZ7NJAJe7777/H7t279W47duwo9vbWrVuHtm3bokKFCnjw4IHu1qVLF2g0Guzfvx8AsGHDBkgkkjwvjijuz3zbt2+Hu7s7hg4dqmuztLTE1KlTkZaWhr///rt4SQHw9/fP9Tw9e2ZboVBg9OjRLx3PvXv3MHDgQHh7e+OHH37QtRf2ec0xePBgVKhQQXe/bdu2AJ7+PF0Yc+bMyZXv7t279c7WAECXLl30zjQ2adIE9vb2uv1otVr8+eef6NOnT559xXOO9fbt2+Hn56fXlcDW1hYTJkzAjRs3cOHCBd1ylSpV0us7a21trfsFIMepU6dw+fJlDBs2DA8fPtQ9X+np6ejcuTP279+fq4vNxIkTC/Xc5PUchYeHY+jQoZg5cya+/vrrF65X2Fw3bNgAZ2dnvPnmm7m28fx7RK1WY9CgQdi6dSu2b9+OV155Re9xR0dHnD9/HpcvXy5SjoWNNcfo0aP1+msW9XX3Inm9xwDAyspK9/+cs+5t27ZFRkYGYmNjC9zuy7xXCnr9azQa7NmzB4GBgXr9bGvXro0ePXoUuP0c6enpcHFxgYuLC2rXro0PPvgArVq1wqZNmwA8/XyoX78+6tWrp/f50KlTJwDI9dN/+/btC91Xt6ifYwMGDMj37PWzv2TJZDL4+vpCCIGxY8fq2h0dHeHl5aX3/MtkMt3rSqvV4tGjR8jOzoavry9OnDiRaz8FHdP79+9j//79GDNmDKpXr663bs57SwiBDRs2oE+fPhBC6D2v3bp1Q3Jycp77ppLBLg1U4vz8/Aq8aK0oLl++jDNnzuT7gZeYmAjg6c9xlStXzlVAvYybN2+iTp06kEr1/zbMubr75s2bxd62s7MzunTpku/jVapUyXVhRlHjyc7OxmuvvQaNRoONGzdCoVDoHivs85rj+Q/xnA//x48f55vDsxo3bvzCfPPbT86+cvZz//59pKSkoFGjRi/czs2bN+Hv75+r/dnnqlGjRrh58yZq166dq+Dz8vLSu59T2AUFBeW7z+TkZL0vxbxGK3mR55+j1157DcnJyZgxYwaGDRuW77EqbK5Xr16Fl5dXofoEL1q0CGlpadixY4eub+ez5s+fj379+qFu3bpo1KgRunfvjhEjRqBJkyYv3G5hY83xsq+7F8nrPQYA58+fx6xZs7B3716kpKToPZacnFzgdl8m5oJe/4mJiXjy5IneqDc58mrLj1KpxJYtWwBA1we+atWquscvX76MixcvFvrzoSiv9aJ+jr1o288/Xw4ODlAqlXB2ds7V/vDhQ722X375BV9++SViY2ORlZX1wv0VdExzCt8XfS7dv38fSUlJWLZsGZYtW5bnMs8/r1RyWPCSydNqtejatSvee++9PB+vW7eugSMyjGfPMhXX9OnTceTIEezZs0fvywwo+vMqk8nyXE4Uom9qURhqP0WVc/b2888/z9W/L4etra3e/ZI4hp07d8bWrVsRHR2t6z9rCN26dUNERAQ+++wzdOjQAUqlUu/xdu3a4erVq/jrr7+wa9cuLF++HF999RWWLl2aq//4yyjN10NexycpKQnt27eHvb095s+fj1q1akGpVOLEiRN4//33CzUM2cvEbMj32Yv+ANVqtWjcuDEWL16c5+PVqlXTu18Sr/X8vGjbeT1fhXkOf/vtN4waNQqBgYGYPn06XF1dIZPJsGjRojwv4CyJ45Lz2hk+fHi+fzgX9AcjFR8LXjJ5tWrVQlpaWoFnB2vVqoWdO3fi0aNHLzzLW5TuDR4eHjhz5gy0Wq3e2YicnzU9PDwKva2SUJR4wsLCsGTJEixZsgTt27fPta3CPq+mxsXFBfb29jh37twLl/Pw8EBcXFyu9uefKw8PD5w7dw5CCL3XxvPr5vzMbG9vb9DnLDs7G8DTGavyU9hca9WqhWPHjiErK6vAi6datmyJiRMnonfv3hg0aBA2bdqU68xwxYoVMXr0aIwePRppaWlo164dPvzwwxcWvIWN1ViioqLw8OFDbNy4UXeBEwBcv37diFH9P1dXVyiVyjxHqiiJ0Sty1KpVC6dPn0bnzp1LfEY4U/hcXb9+PWrWrImNGzfq5Vfc8YJr1qwJAC/8XHJxcYGdnR00Gk2Z+9w1B+zDSybvtddew5EjR7Bz585cjyUlJekKggEDBkAIgXnz5uVa7tm/wm1sbJCUlFSofffs2RP37t1DeHi4ri07OxvffvstbG1t8ywkS1Nh4zl37hzGjRuH4cOH57riOUdhn1dTI5VKERgYiC1btuiGm3tWzrHu2bMnoqOjceTIEd1j6enpWLZsGTw9PXX9DXv27Il///0X69ev1y2XkZGR6ydHHx8f1KpVC1988UWexef9+/dLJL/nbd26FQDQtGnTfJcpbK4DBgzAgwcP8N133+XaRl5nqrp06YKwsDBERERgxIgRemc3n/952NbWFrVr1y5wpq7CxmosOWfynn0+1Gq1Xv93Y8o5M/vnn3/i33//1bVfuXLlpa6VeN5rr72GO3fu4Oeff8712JMnT5Cenl7sbZvC52pex/nYsWN6r8uicHFxQbt27bBixQrEx8frPZazD5lMhgEDBmDDhg15Fsal9RlCT/EML5W4HTt25HlhR0BAgO6v4KKYPn06Nm/ejN69e2PUqFHw8fFBeno6zp49i/Xr1+PGjRtwdnZGx44dMWLECHzzzTe4fPkyunfvDq1WiwMHDqBjx46YMmUKgKeFy549e3SDq9eoUSPPPoUAMGHCBPz0008YNWoUYmJi4OnpifXr1+PQoUNYsmQJ7OzsipzPyyhsPDkX4rRr1w6//fab3jZyjkNhn9eScuDAgTynTW7SpEmRf8ZbuHAhdu3ahfbt2+uGTLp79y7WrVuHgwcPwtHRETNmzMDatWvRo0cPTJ06FRUrVsQvv/yC69evY8OGDbozS+PHj8d3332HkSNHIiYmRjec0vMTEEilUixfvhw9evRAw4YNMXr0aFSpUgV37tzBvn37YG9vr+sTWVzPPkePHj3C5s2b8ffff2PIkCGoV69evusVNteRI0fi119/RXBwMKKjo9G2bVukp6djz549mDRpEvr165dr24GBgVi5ciVGjhwJe3t7/PTTTwCABg0aoEOHDvDx8UHFihXxzz//YP369br32cvGaiwBAQGoUKECgoKCMHXqVEgkEqxevdroXWqe9eGHH2LXrl1o3bo13njjDWg0Gnz33Xdo1KgRTp06VSL7GDFiBP744w9MnDgR+/btQ+vWraHRaBAbG4s//vgDO3fuLPa1Gqbwudq7d29s3LgRr776Knr16oXr169j6dKlaNCgwQt/TXmRb775Bm3atEHz5s0xYcIE1KhRAzdu3MC2bdt0x+WTTz7Bvn374O/vj/Hjx6NBgwZ49OgRTpw4gT179uDRo0clmCXpMfCoEGTGXjQsGQCxcuVK3bJFGZZMiKfDFoWEhIjatWsLuVwunJ2dRUBAgPjiiy+EWq3WLZednS0+//xzUa9ePSGXy4WLi4vo0aOHiImJ0S0TGxsr2rVrJ6ysrASAAocoS0hIEKNHjxbOzs5CLpeLxo0b6+VSUE55KWjZ9u3bi4YNGxY7nvyGPXv+OBTmeX12mKrn4Zmh1PJT0LBkz66f3/F/fhguIYS4efOmGDlypHBxcREKhULUrFlTTJ48WW+ooatXr4qBAwcKR0dHoVQqhZ+fn9i6dWuu7d+8eVP07dtXWFtbC2dnZzFt2jQRERGhNyxZjpMnT4r+/fsLJycnoVAohIeHh3jttddEZGSkbpmcIZPu37//wufmRc+RXC4X9erVEwsWLNB7jef3fBQ214yMDDFz5kxRo0YNYWlpKdzd3cXAgQPF1atXhRD5H+8ffvhBABDvvvuuEEKIjz/+WPj5+QlHR0dhZWWVb6x5KUysOc/JunXr9Npz4svrPZif/IYly+89dujQIdGyZUthZWUlKleuLN577z2xc+fOXK+H/IYlK8x7Jb9hyQr7+o+MjBTNmjUTcrlc1KpVSyxfvly88847QqlU5vMs/L+cYckKolarxaeffioaNmwoFAqFqFChgvDx8RHz5s0TycnJBcadE3ten3WF+Rx70fOZ33ssv9yeP95arVYsXLhQeHh4CIVCIZo1aya2bt36UsdUCCHOnTsnXn31Vd1r28vLS8yePTtX7pMnTxbVqlXTvQc7d+4sli1blmsfVHIkQpjQn61ERERULIGBgcUaKo6oPGAfXiIiojLmyZMnevcvX76M7du35zmEHBEBPMNLRERUxlSqVAmjRo1CzZo1cfPmTfz4449QqVQ4efIk6tSpY+zwiEwOL1ojIiIqY7p37461a9fi3r17UCgUaNWqFRYuXMhilygfPMNLRERERGaNfXiJiIiIyKyx4CUiIiIis2b0Przff/89Pv/8c9y7dw9NmzbFt99+Cz8/v3yXX7JkCX788UfEx8fD2dkZAwcOxKJFi/Tmeb9z5w7ef/997NixAxkZGahduzZWrlxZ6EGytVot/v33X9jZ2ZX4lIpERERE9PKEEEhNTUXlypULnrTGiGMAi7CwMCGXy8WKFSvE+fPnxfjx44Wjo6NISEjIc/k1a9YIhUIh1qxZI65fvy527twpKlWqJN5++23dMo8ePRIeHh5i1KhR4tixY+LatWti586d4sqVK4WO69atWy8cKJ833njjjTfeeOONN9O43bp1q8DazqgXrfn7+6NFixa6ed21Wi2qVauGN998EzNmzMi1/JQpU3Dx4kVERkbq2t555x0cO3YMBw8eBPB02spDhw7hwIEDxY4rOTkZjo6OuHXrFuzt7Yu9ncLSCoGk5FQ4OthBagZnlM0pH+ZimswpF8C88mEupom5mC5zysfQuaSkpKBatWpISkqCg4PDC5c1WpcGtVqNmJgYhISE6NqkUim6dOmCI0eO5LlOQEAAfvvtN0RHR8PPzw/Xrl3D9u3bMWLECN0ymzdvRrdu3TBo0CD8/fffqFKlCiZNmoTx48fnG4tKpYJKpdLdT01NBQDY2tnB1gBzegshoBES2NnZmkUXCnPKh7mYHpGpRsobP8JOq4Xt0kmQWimMHdJLM5djAzAXU8VcTJc55WPoXLT/nbMtzL6MVvA+ePAAGo0Gbm5ueu1ubm6IjY3Nc51hw4bhwYMHaNOmDYQQyM7OxsSJE/HBBx/olrl27Rp+/PFHBAcH44MPPsDx48cxdepUyOVyBAUF5bndRYsWYd68ebnak5JToRGGe/ElpaQZbF+GYE75MBcTkqkGbt4HACSnpAFqtZEDKjll/tg8g7mYJuZiuswpH0PlkpqSWuhljX7RWlFERUVh4cKF+OGHH+Dv748rV65g2rRp+OijjzB79mwAT7tF+Pr6YuHChQCAZs2a4dy5c1i6dGm+BW9ISAiCg4N193NOkTs62MHe3jBneJNS0uBoX/b/ugPMKx/mYnqEXI2k//7vYGcLqbV5nOE1h2MDMBdTxVxMlznlY+hcZJLC98o1WsHr7OwMmUyGhIQEvfaEhAS4u7vnuc7s2bMxYsQIjBs3DgDQuHFjpKenY8KECZg5cyakUikqVaqEBg0a6K1Xv359bNiwId9YFAoFFIrcX5pSicQgfVC0//0rMdD+Sps55cNcTI94JvaynksOczk2AHMxVczFdJlTPobOpSj7MNo4vHK5HD4+PnoXoGm1WkRGRqJVq1Z5rpORkZFr2AmZTAbg6V8VANC6dWvExcXpLXPp0iV4eHiUZPhEREREVEYYtUtDcHAwgoKC4OvrCz8/PyxZsgTp6ekYPXo0AGDkyJGoUqUKFi1aBADo06cPFi9ejGbNmum6NMyePRt9+vTRFb5vv/02AgICsHDhQrz22muIjo7GsmXLsGzZMqPlSURERETGY9SCd/Dgwbh//z7mzJmDe/fuwdvbGxEREboL2eLj4/XO6M6aNQsSiQSzZs3CnTt34OLigj59+mDBggW6ZVq0aIFNmzYhJCQE8+fPR40aNbBkyRK8/vrrBs+PiIiIiIzPqOPwmqqUlBQ4ODggOTnZYOPwPk5ORQUzGIMPMK98mIvpEZlqJI36GlqtFo6/vAWZGQxLZi7HBmAupoq5mC5zysfQuRSlXitTozQQEUmUcjisfRePk1MhUcqNHQ4REZUBRrtojYiIiIjIEHiGl4iIiIiKRKPRIDU1FcnJybrb46QkJN5/gDGjRxk7vFxY8BJRmSJUWUiZ9jOg0UB8+z+A3RqIiIpEo9EgJSVFr1gt6i01Ne9ZziQSCUYFjYT0v9GzTAULXiIqW7QCmrg7uv8TEZUn2dnZL12spqWV3NS/CoUCDg4Oupu1jQ1UKhUsrK1LbB8lgQUvERERUSkRQiAzMxMZGRl53tLS03HvXiKys1SFKmRLslhVKpV6xWpxbs/OVJszSoOVlVWJxVhSWPASERFRuSOEQFZWFjIyMvDkyZN8C9KSuJWGgopVR0fHAotVubz8dAljwUtERERGJ4SASqVCZmYmVCqV3v+fbct48gSPHidBAoHMlyxUNRqNQXOUy+WwtrbWu1lZWcHK2hpOTk5wLMKZ1fJUrJYEFrxERETllBAC2dnZBRaZhmhTq9VGex6kUilsbGxyFaLPF6cvc7OysoKFRe6yy5wmnjBlLHiJiIjMlBAChw4dwvLQUBw6dAhZanWuwtNUJ1xVKBRQKpVQKBR6/1cqlZDJLGBnbwebEipGLS0tIWGxadZY8BJRmSNxsDbZL2kiU5CYmIhff/0Vy5cvR1xcXKHXs7CwyLfINGSbXC7PtwDlGVEqDha8RFSmSKzkcNz0wdOpha3Yh40oh0ajwa5du7B8+XJs3rwZ2dnZAABra2sMHjwY3Xv2RvVqVWClVOZZZCoUCshMbOxUopLCgpeIiKgMu3HjBlasWIGVK1fi9u3bunZ/f3+MHTsWgwcPhq2dHc+KUrnGgpeIiKiMUalU+PPPPxEaGoo9e/bouvhUrFgRI0aMwNixY9G4cWPd8lp2AaJyjgUvEZUpQpWF1PdXAdkaiC/GcGphKlfOnTuH0NBQrF69Gg8fPtS1d+nSBePGjUO/fv2gVCqNGCGRaWLBS0Rli1Yg+/QN3f+JzF1qairCw8OxfPlyHDt2TNdepUoVjBkzBqNHj0aNGjWMGCGR6WPBS0REZGKEEDh69CiWL1+O8PBwpKenA3g6ikLfvn0xduxYdOvWjReZERUSC14iIiITcf/+faxevRqhoaG4cOGCrr1u3boYN24cRo4cCTc3NyNGSFQ2seAlIiIyIq1Wiz179mD58uX4888/kZWVBQCwsrLCa6+9hnHjxqF169acGIHoJbDgJSIiMoL4+HisXLkSK1asQHx8vK7d19cX48aNw5AhQ+Dg4GDECInMBwteIiIiA1Gr1di8eTOWL1+OXbt26YYTc3R0xPDhwzF27Fh4e3sbN0giM8SCl4jKHqUlwAEaqAy5cOECQkND8euvv+LBgwe69o4dO2LcuHF49dVXYWVlZcQIicwbC14iKlMkVnLc+WwAdu3eg+bHDsPXxwf29vbGDosol7S0NPzxxx8IDQ3F4cOHde2VKlXC6NGjMWbMGNSqVcuIERKVHyx4iajMSE9Px4cffoivvvoKGo1G1+7l5QVfX1/dzdvbG7a2tkaMlMorIQSOHz+O5cuXY+3atUhLSwMAyGQy9O7dG+PGjUP37t1hYcGvXyJD4juOiMqEHTt2YNKkSbhx4wYAwLdFC9xPTMTNmzcRFxeHuLg4rFmzBgAglUpRv359vSK4adOm/MmYSs3Dhw/x22+/Yfny5Th37pyuvXbt2hg7diyCgoJQqVIlI0ZIVL6x4CUik3bv3j289dZbCA8PBwBUq1YNn/sNQBfPRnBcMAKPUpIx7+gpPLkUi/sH9+HE8WjcuXMH58+fx/nz5/HLL78AeHqGrVGjRnpFcOPGjaFQKIyZHpVhWq0We/fuRWhoKDZu3Ai1Wg0AUCqVGDhwIMaNG4d27dpxODEiEyA1dgAA8P3338PT0xNKpRL+/v6Ijo5+4fJLliyBl5cXrKysUK1aNbz99tvIzMzMc9lPPvkEEokEb731VilETkSlRavVYtmyZahfvz7Cw8MhlUrx9ttv43zMKXR5aA3EXAM0Ak7OzvhBrcAKz6bYOfJt1PhtK948cgaz/9yB6fM/Qq9eveDq6gqNRoPTp08jNDQUb7zxBlq0aAE7Ozv4+vpi4sSJWL58OU6dOqUbA5UoP7dv38bHH3+MWrVqoWvXrggLC4NarUazZs3w/fff4+7du1i9ejXat2/PYpfIRBj9DG94eDiCg4OxdOlS+Pv7Y8mSJejWrRvi4uLg6uqaa/nff/8dM2bMwIoVKxAQEIBLly5h1KhRkEgkWLx4sd6yx48fx08//YQmTZoYKh0iKgEXLlzAhAkTcOjQIQBA8+bNsWzZMvj4+EA8UePRM8tKJRIMquKMP+48gForcPBhCo5KJMgW1pA2aIdDk6bCv6Id7ty5g3/++Ufv9vDhQ8TExCAmJgY//fQTAEChUMDb21vvTHC9evXY57Kcy8rKwtatW7F8+XJERERAq9UCABwcHPD6669j7NixaN68uZGjJKL8GP0TfPHixRg/fjxGjx4NAFi6dCm2bduGFStWYMaMGbmWP3z4MFq3bo1hw4YBADw9PTF06FAcO3ZMb7m0tDS8/vrr+Pnnn/Hxxx+XfiJE9NIyMzOxYMECfPrpp8jKyoKNjQ0++ugjvPnmmy8sOOc18MQfd/5/qKdsISAFUM1aAS87K0gkElStWhVVq1ZFYGAggKcXF928eTNXEZycnIxjx47pfaZYW1ujWbNmekVw3bp1IZWaxI9kVIouX76MDX+E4ddff0ViYqKuvX379hg7diwGDBgAa2trI0ZIRIVh1IJXrVYjJiYGISEhujapVIouXbrgyJEjea4TEBCA3377DdHR0fDz88O1a9ewfft2jBgxQm+5yZMno1evXujSpUuBBa9KpYJKpdLdT0lJAQBohYBWlP5gnzkDjwshoC31vZU+c8qHuRjO3r17MemNN3D58mUAQK/evfHdd9+hevXqAKB7L4pn3pPiv/doXTsr9HCrgF2Jj6H572GpBNjUsgEcLC3yfR9X9/BAdQ8P9B8wQLe9q1ev4p9//kHMP//ozv6mpaXh0KFDujPOAGBrawsfH5+nt/+K4Fq1ahXrJ2xTPzZF8TK5CCGQnZ0NlUoFtVqtuz1/v6C2rIKWe+6+Wq2GOo99ZmZm6s2A5ubmhqCgIIweMwZ169bVtRvie+Jl8TVmuswpH0PnUpT3nlEL3gcPHkCj0cDNzU2v3c3NDbGxsXmuM2zYMDx48ABt2rTRfThOnDgRH3zwgW6ZsLAwnDhxAsePHy9UHIsWLcK8efNytSclp0IjDNf/KiklzWD7MgRzyoe5lJ6HDx9i7uxZCFv7OwDAzd0diz75FH369oNEIsHj5FT9FTLVuv8mp6YBWU/vT6hcETsSHv//cgKYdOISVjWuAWd54T/qnFzc0K1HL3Tr0QvA077EV69cwcmTJ3H61EmcOnkSZ8+eQVpaGv7++2/8/fffunUdHBzQ1Nsb3t7N4N3s6a1ateqFLoJN7dgAT08IpKWlIT09DWmpaUhLS0NaWup//z69pT/7//R0ZGZm6grHrCw11OosZGU9LSKz1FlQZ6l1/6pVKqizsnRFqqmRSqXo0vUVDB85El27vgJLS0sAyP26LCNM8TVWXOaUC2Be+Rgql9SUwr8Pjd6loaiioqKwcOFC/PDDD/D398eVK1cwbdo0fPTRR5g9ezZu3bqFadOmYffu3VAqlYXaZkhICIKDg3X3U1JSUK1aNTg62MHe3q60UtERQiApJQ2O9rZmcYGDOeXDXEo3nl9//RXT330XDx8+hEQiwf/+9z8sXLQIDg4O+a8nVyPpv/872NlCav10lIU+9rZofPUuzqZkYFFDT7RxdsDAoxfQLeYyNrVsgKaOxR+X16mFD/xa+OjuZ2dnIzY2Vu9M8KlTp5CcnIz9f/+N/c8UwU5OTk/PAD9zJrhKlSp6x6Akj41arUZq6tOCNDU1Ndct/fn2Z+7neiw11agX8UkkEigUCsjlcr1bXm3y59oUCgXklpYvXM8yn23ltFlaWqKCkwvq1q5pEu+Zl2Fq7/+XYU65AOaVj6FzkUnKyBleZ2dnyGQyJCQk6LUnJCTA3d09z3Vmz56NESNGYNy4cQCAxo0bIz09HRMmTMDMmTMRExODxMREvYsHNBoN9u/fj++++w4qlQoymUxvmwqFIs+hiaQSCaQGOGA5p/0lBtpfaTOnfJhL6bh8+TImTpyIvXv3AgAaNWqEZcuWoVWrVgWuK56JXS8XiQTLmtdF1IMkvF+3GiQSCY53aobAIxfQ5u/T+LWFFwZUcSmR+OWWlmjSuDGaNG6MMf9df5CVlYXz58/r9Qc+c+YMHj58iF07d2LXzp269d3c3PSHR2vSBEnJqfhXInIVnfkVrvk9VlpnSa2srGBnZwdbW1vY2dnluuW029raQkikcLC3g/KZYvL5wrIwhaxMJjNqAaAVAo+TU03iPfOyTOn9/7LMKRfAvPIxdC5F2YdRC165XA4fHx9ERkbqLiTRarWIjIzElClT8lwnIyMj14UiOQWsEAKdO3fG2bNn9R4fPXo06tWrh/fffz9XsUtEhqNWq/HZZ5/h448/hkqlglKpxNy5c/HOO+/ofiouiMRKjgp7P35aiFjJ9R5r6WSPlk7/P81wNWslDrRvitExlzDw6EV8WD8Ds+tXL5UPYktLS3h7e8Pb21v3B7lKpcLZs2f1iuBz584hISEB27Ztw7Zt20o8jhxKpfKFBWphitdn7xd2lIqcIrGCg12Z//ImIvNh9C4NwcHBCAoKgq+vL/z8/LBkyRKkp6frRm0YOXIkqlSpgkWLFgEA+vTpg8WLF6NZs2a6Lg2zZ89Gnz59IJPJYGdnh0aNGuntw8bGBk5OTrnaichwDh06hAkTJuDChQsAgK5du+LHH39ErVq1SnW/1hYyhPnVQxMHG8w6fwNnU9Lxi68XbCxK/49fhUKhO4ub48mTJzh9+rReEXzx4kVYWloWuRDN7zFbW9tC/wFBRFQeGL3gHTx4MO7fv485c+bg3r178Pb2RkREhO5Ctvj4eL0zurNmzYJEIsGsWbNw584duLi4oE+fPliwYIGxUiCiF0hKSsL777+PZcuWAQBcXFzw1VdfYdiwYQb7uVoikWBmvepoZG+N4cfj0DrqFP5q1RAeNoXr51+SrKys0LJlS7Rs2VLXlq3RIDk1nWdFiYhKiUSIMjCeioGlpKTAwcEBycnJsLe3L3iFl2RuPwGaUz7MpfiEEPjjjz8wbdo0XT/9MWPG4LPPPoOTk1Pxt6vOQuqCdcjKyobj3CGQKeQFr/SMs8np6Hv4HNI1Wmxo2QBtnfO/QM5Q+DozTczFNJlTLoB55WPoXIpSr3HUdCIqcTdu3ECvXr0wZMgQJCQkwMvLC1FRUQgNDX2pYhcAoBHI2n8eOBIH3aC7RdDYwQbHOzVHAztrdN5/Bj9fv/ty8RARkcljwUtEJSY7OxtffPEFGjZsiB07dkAul2Pu3Lk4ffo02rdvb+zwdJwVltjdtjHG1XDHhBOX8eapK8jW8scuIiJzZfQ+vERkHo4fP44JEybg1KlTAIB27drhp59+Qr169YwbWD4spVL80KwOGtvbYOrpq7iYkoE/WtZHRTkv9iIiMjc8w0tELyU1NRXTpk1Dy5YtcerUKVSoUAHLly/Hvn37TLbYfdYbtSpjd9vGOJWcBr+9J3EhJd3YIRERUQljwUtExfbXX3+hQYMG+Oabb6DVajFs2DDExsZi7NixucbLNmUdXBxxvFMzWMmkaLnvFLbefWjskIiIqASVnW8kIjIZd+7cQf/+/REYGIjbt2+jZs2a2LlzJ9asWQNXV1djh1csNWyscLiDNzq7OqLv4fP4NO4WOIgNEZF5YMFLRIWm0Wjw3XffoX79+ti0aRMsLCwwY8YMnD17Fq+88oqxw3tpdpYW2NCyAWbVq44Z565j+PFYPNFojB0WERG9JF60RkSFcvr0aUyYMAHR0dEAgJYtW2LZsmVo3LixYQNRWsJx2xwkpaQCypK/wEwqkWB+Q080crDBqH/icCntNP5s1RBVrBQlvi8iIjIMnuElohfKyMjA+++/Dx8fH0RHR8Pe3h7ff/89Dh48aPhiF09nTZNYyQGlvFRnanutqgsOdfBGQmYWfPeexLFHKaW2LyIiKl0seIkoXzt37kSjRo3w2WefQaPRYMCAAbh48SImTZoEmUxm7PBKXTNHWxzv1Aw1bZRo//dp/HozwdghERFRMbDgJaJcEhISMGzYMHTv3h3Xr19HtWrVsHnzZqxfvx6VK1c2amxCnY30TzcA32yDUGeX+v7clHLsbdsEr1d3RdA/cZh+5ho0vJiNiKhMYcFLRDparRbLly9HvXr1sHbtWkilUrz11lu4cOEC+vTpY+zwntJood55Eth3DtBoDbJLhUyK5c3r4uumtbD48m30PnQOSQYotomIqGSw4CUiAMDFixfRoUMHjB8/HklJSWjWrBmio6Px1VdfwdbW1tjhGZ1EIsHU2lUQ0aYxjj5KRct9J3EpNcPYYRERUSGw4CUq5zIzMzFnzhw0bdoUBw4cgLW1Nb788ktER0fDx8fH2OGZnK5uFRDdsRkkEsBv30nsvPfI2CEREVEBWPASlWP79u1D06ZN8dFHHyErKwu9evXChQsXEBwcDAsLjlqYnzp2VjjasRlaOzmg56Fz+OrybU5SQURkwljwEpVDDx8+xOjRo9GpUydcunQJ7u7u+OOPP7BlyxZ4eHgYO7wywcHSApsDGuLdulURfOYaxsRcgspAfYqJiKhoWPASlSNCCKxevRr16tXDqlWrIJFI8MYbbyA2NhaDBg0q1XFtzZFMIsGnjWtidQsvrL2ViI77z+BeptrYYRER0XNY8BKVE7GxsRjYPxCjgoLw4MEDNGrUCIcOHcIPP/wABwcHY4dXpg2v7ob97ZviRkYmWuw9gZjHqcYOiYiInsGCl8iM3blzB1999RX8/f3RsEED/B0VBaVSiYULFyImJgatWrUydohFp7SEw8YQYNWbpTK1cHH5VbTH8U7NUEmpQJuo0wi7lWjskIiI6D+8KoXIzDx48AAbNmzA2rVrsX//ft3FVFKpFF1f6YZvvvkadevUMXKUxSeRSCB1tAEkWpPrglHFSoG/2zfB+JjLGBodi7PJ6fiooSekJhYnEVF5w4KXyAykpKTgzz//RFhYGHbv3o3s7P+fFKF169YYOnQo+g8YALnSGhUc7IwYqfmzksmwuoUXmjjYYMa56zifkoHVLbxgZ8mPWyIiY+EnMFEZ9eTJE2zbtg1r167Ftm3boFKpdI81b94cQ4YMweDBg1G9enUAgFYIPE4u+31LhTobGT9sB1RqiLf6AQrT6daQQyKR4D2vamhob42h0bFoFXUKm1s1RE1bK2OHRkRULrHgJSpD1Go1du/ejbCwMPz5559IS0vTPVavXj0MHToUgwcPhpeXlxGjLGUaLVR/HXv6/zdNZLrjfPSq5IRjHZuh7+FzaLHvJNb7N0BHV0djh0VEVO6w4CUycRqNBvv378fatWuxYcMGPHr0/zN7eXh4YMiQIRg6dCiaNGlicn1aCahvb41jnZph8LGL6HrwDL5pWhuTalU2dlhEROUKC14iEySEwLFjxxAWFoY//vgDd+/e1T3m5uaGwYMHY8iQIWjZsiWL3DKgotwSO1o3xrtnr2HyqSs4m5yOr71rQS7lQDlERIbAgpfIRAghcPbsWaxduxZhYWG4ceOG7rEKFSpgwIABGDp0KNq3bw+ZTGa8QKlYLKQSLGlaC43trfHGySu4kJqB9S3rw0UhN3ZoRERmzyROL3z//ffw9PSEUqmEv78/oqOjX7j8kiVL4OXlBSsrK1SrVg1vv/02MjMzdY8vWrQILVq0gJ2dHVxdXREYGIi4uLjSToOoWC5fvoyPPvoIDRs2RNOmTfHJJ5/gxo0bsLGxweuvv44tW7bg3r17+Pnnn9GpUycWu2Xc2BqVsK9dE8SmZsBv70mcSU4reCUiInopRj/DGx4ejuDgYCxduhT+/v5YsmQJunXrhri4OLi6uuZa/vfff8eMGTOwYsUKBAQE4NKlSxg1ahQkEgkWL14MAPj7778xefJktGjRAtnZ2fjggw/wyiuv4MKFC7CxsTF0ikS53Lp1C+Hh4Vi7di1OnDiha1coFOjZsyeGDh2KXr16wdra2ohRUmlp7eyA452aod/h8wjYdwq/+Hqhg63C2GEREZktoxe8ixcvxvjx4zF69GgAwNKlS7Ft2zasWLECM2bMyLX84cOH0bp1awwbNgwA4OnpiaFDh+LYsWO6ZSIiIvTWWbVqFVxdXRETE4N27dqVYjZE+UtMTMT69euxdu1aHDx4UNcuk8nQtWtXDBkyBIGBgZzmt5yobq3EwQ7eGP1PHAYeu4gZNd3xcVNbgH2yiYhKnFELXrVajZiYGISEhOjapFIpunTpgiNHjuS5TkBAAH777TdER0fDz88P165dw/bt2zFixIh895OcnAwAqFixYp6Pq1QqvTFMU1JSADwdt1T73yxVpSlnJiwhBLSlvrfSZ075vGwuSUlJ2LRpE8LCwrA3MhJa7dOtSCQStG3bFoOHDMGAAQPg4uKiW6e0XnPmclyEXAa7NcFITcuAkMsM8h4tLVYyKdb61UOj2HjMvRiPK6psrPCpCxuLstttxVxeZwBzMVXmlAtgXvkYOpeifP4bteB98OABNBoN3Nzc9Nrd3NwQGxub5zrDhg3DgwcP0KZNGwghkJ2djYkTJ+KDDz7Ic3mtVou33noLrVu3RqNGjfJcZtGiRZg3b16u9qTkVGiE4c62JKWYV18+c8qnKLmkp6dj184IbNywAZF7dkOtVuse827WDP0HDERg4KuoXKWKrt2QE0KYxXGxtgSsHZCclmHsSErE5MoV4WkhxaQL8Wi99yRWN62BqsqyfTGbWbzO/sNcTJM55QKYVz6GyiU1pfDfnUbv0lBUUVFRWLhwIX744Qf4+/vjypUrmDZtGj766CPMnj071/KTJ0/GuXPn9H5Cfl5ISAiCg4N191NSUlCtWjU4OtjB3r70p2EVQiApJQ2O9rZmMcSUOeVT2FxUKhV27tyJ8PBwbP7rL2Rk/H8h1rBhQwwZMgSvDR6M2rVrGyLsPJXH41JWCCHQC8BBlwp49egFdP3nMtb710cb57LXvcWcjg1zMU3mlAtgXvkYOheZpIyc4XV2doZMJkNCQoJee0JCAtzd3fNcZ/bs2RgxYgTGjRsHAGjcuDHS09MxYcIEzJw5E9JnxrWcMmUKtm7div3796Nq1ar5xqFQKKBQ5L5gRCqRQGqAA5Zz2l9ioP2VNnPK50W5ZGdnIyoqCmvXrsXGjRuRlJSke6xmzZq6CSHy+2XB0MzluIisbGQs3w2o1MCknpDKTW9q4aLKOTZNHG1xvFNzDDx6AV0OnMWPzWpjbI1KRo2tqMzldQYwF1NlTrkA5pWPoXMpyj6MWvDK5XL4+PggMjISgYGBAJ52QYiMjMSUKVPyXCcjI0OvqAWgG6bp2b4jb775JjZt2oSoqCjUqFGj9JKgckWr1eLIkSO6CSESExN1j1WuXFk3IUSLFi3K/F/qJitbC9Uf//1iM6E7ULZ/+c/FWWGJXW0bY+qpqxh34jLOJKfjyya1YCHl64mIqLiM3qUhODgYQUFB8PX1hZ+fH5YsWYL09HTdqA0jR45ElSpVsGjRIgBAnz59sHjxYjRr1kzXpWH27Nno06ePrvCdPHkyfv/9d/z111+ws7PDvXv3AAAODg6wsrIyTqJUZgkhcPLkSYSHhSE8PBzx8fG6x5ycnDBw4EAMHToUbdq04Ri5VCLkUimWNq+Dpg42ePP000kqwv3ro6IZnM0mIjIGoxe8gwcPxv379zFnzhzcu3cP3t7eiIiI0F3IFh8fr3dGd9asWZBIJJg1axbu3LkDFxcX9OnTBwsWLNAt8+OPPwIAOnTooLevlStXYtSoUaWeE5mHrKwsfP7551ixciWuXrmia7ezs0NgYCCGDh2KLl26wNKSRQiVjjdqVUY9O2sMPHYBfntPYnNAQzSw51jiRERFJRGiDI/pU0pSUlLg4OCA5ORk2Nvbl/r+tELgcXIqKjjYlfn+O4B55JOSkoJBgwZh165dAAClUonevXtjyJAh6NmzZ5n8pcAcjgsAiCdqPOr5dFQVx21zILMu+xM2FHRsrqU9Qb8j53EzQ4Xf/eqhdyUnI0RZOObyOgOYi6kyp1wA88rH0LkUpV4ziamFiUzJv//+i3bt2mHXrl2wtrbG4iVf4+69e1i3bh0GDBhQJotdKttq2lrhcAdvdHRxRN/D5/Fp3C3wXAURUeEZvUsDkSk5f/48evTogVu3bsHV1RVbtm5FrTpeBhmejuhF7CwtsKlVA8y5cAMzzl3HmeQ0LPepCyv2GyciKhDP8BL9JyoqCq1bt8atW7dQt25dHDlyBL6+vsYOi0hHKpHg44Y1EOZXD5v+fYh2f5/GnSeqglckIirnWPASAVi7di26deuG5ORktG7dGocPH0bNmjWNHRblRWEB+9A3ga/HAIry+SPV4GquONC+Ke5lqtFi70lEP0oxdkhERCaNBS+Va0IIfPrppxg2bBjUajUGDhyIPXv2wMnJdC8KKu8kUilkNdyA6i6QSMvvR5hPBTsc79QcntZKtPv7NH6LTyh4JSKicqr8fltQuafRaDB58mTMmDEDAPD2228jPDwcSqXSyJERFY67Uo597ZpgaDVXjDgeh/fOXoOGF7MREeVSPn8PpHIvPT0dQ4cOxZYtWyCRSPDVV19h2rRpxg6LCkFkZePJb1GASg0xpitQzidjUMikWOFTF00dbPDOmWs4n5KB3/3qwcGSH+9ERDl4hpfKncTERHTs2BFbtmyBUqnEunXrWOyWJdlaZP66Dwg/BGRrC16+HJBIJHirTlXsaNMIhx+moOW+k7ic+sTYYRERmQwWvFSuXLp0Ca1atcLx48dRsWJFREZGYsCAAcYOi6hEvOJWEcc6ekMIwG/fSexOeGzskIiITAILXio3Dh8+jICAAFy7dg01atTQ3ScyJ3XtrHGsUzO0qmiH7gfP4uvLdzhJBRGVeyx4qVzYtGkTOnfujIcPH6JFixY4cuQIvLy8jB0WUalwsLTAltaNEFynKt46cxXjTlyCSsPuH0RUfrHgJbP3zTffYMCAAcjMzESfPn2wb98+uLm5GTssolIlk0jweZOa+MXXC7/FJ6LTgTNIyFQbOywiIqNgwUtmS6vV4p133sG0adMghMAbb7yBjRs3wsbGxtihERnMSA83/N2uKa6lZ8J37wmceJxq7JCIiAyOBS+ZpczMTAwZMgSLFy8GACxatAjff/89LCw4VBOVPy2d7PFPp2ZwV8rR5u/T+OP2fWOHRERkUPz2J7Pz6NEj9OvXDwcPHoSlpSVWrVqFYcOGGTssKilyC9j9MBGpaRmAnB9hhVXFSoH97ZtiXMwlDD52EWeT0zGvgQekEomxQyMiKnX8tiCzcv36dfTo0QNxcXFwcHDApk2b0LFjR2OHRSVIIpPCol5VIDkVEhl/pCoKK5kMv7WohyYOtgg5dx1nk9OxuoUX7DhJBRGZOX5bkNn4559/0KpVK8TFxaFatWo4ePAgi12i50gkErzvVQ2bAxpi7/0kBESdwvV0TlJBROaNBS+Zhe3bt6N9+/ZISEhA06ZNceTIETRq1MjYYVEpEFnZyAw7AGw6BpGVbexwyqzelZxwtKM3nmi0aLH3JKLuJxk7JCKiUsOCl8q8n3/+GX379kVGRga6du2K/fv3o0qVKsYOi0pLthZPlu0Efo3i1MIvqYG9DaI7NYO3gy26HjiLH6/+a+yQiIhKBQteKrOEEJg1axYmTJgAjUaDUaNGYdu2bbC3tzd2aERlRkW5JXa0aYQ3albCpFNX8MaJy8jS8g8JIjIvvFKByiS1Wo1x48Zh9erVAIC5c+di7ty5kPCKc6Iis5RK8Y13bTR2sMHkk1dwMTUD61s2gLPC0tihERGVCJ7hpTInOTkZPXv2xOrVqyGTybB8+XJ8+OGHLHaJXtL4GpUQ2a4JLqRmoMXeEziTnGbskIiISgQLXipTbt++jbZt2yIyMhK2trbYunUrxo4da+ywiMxGW2cHHO/YDA6WFgjYdwp/3nlg7JCIiF4aC14qM86cOYOWLVvi7NmzcHd3x/79+9G9e3djh0VkdjxslDjUwRvd3Svi1aMX8PHFmxBCGDssIqJiY8FLZUJkZCTatm2LO3fuoH79+jh69CiaNWtm7LCIzJaNhQx/+NfHvAYemH3hJgYfu4j0bI2xwyIiKhaTKHi///57eHp6QqlUwt/fH9HR0S9cfsmSJfDy8oKVlRWqVauGt99+G5mZmS+1TTJdq1evRvfu3ZGSkoL27dvj0KFD8PDwMHZYZCxyC9guHgN8NJRTC5cyqUSCOfU9sL5lfWy79wid9p+Blmd6iagMMnrBGx4ejuDgYMydOxcnTpxA06ZN0a1bNyQmJua5/O+//44ZM2Zg7ty5uHjxIkJDQxEeHo4PPvig2Nsk0ySEwIIFCzBy5EhkZ2djyJAh2LlzJypUqGDs0MiIJDIpLL1rAo2qc2phAxlQxQWHO3ijk6sjtKx3iagMMvq3xeLFizF+/HiMHj0aDRo0wNKlS2FtbY0VK1bkufzhw4fRunVrDBs2DJ6ennjllVcwdOhQvTO4Rd0mmZ7s7Gz873//w6xZswAA7733HtasWQOFQmHkyIjKp6aOtljUqAYspBwNhYjKHqP+HqhWqxETE4OQkBBdm1QqRZcuXXDkyJE81wkICMBvv/2G6Oho+Pn54dq1a9i+fTtGjBhR7G2qVCqoVCrd/ZSUFACAVgiD/HyXczGIEALmMNz7y+aTlpaGIUOGYMf27ZBKpfj6668xafJkADD4z6nmdGzMJReRrYFqy3EgUwXtwNaAZdnv1mAuxwZgLqaKuZguc8rH0LkUpSYw6jfFgwcPoNFo4Obmptfu5uaG2NjYPNcZNmwYHjx4gDZt2kAIgezsbEycOFHXpaE421y0aBHmzZuXqz0pORUaYbizGUkp5jXmZXHySUhIwNDBg3Dm9GlYWVnhp+Wh6NmzFx4np5ZChIVnTsemzOeSqQa+3QoASO7UCFDKjRxQySnzx+YZzMU0MRfTZU75GCqX1JTC1wZl7tRIVFQUFi5ciB9++AH+/v64cuUKpk2bho8++gizZ88u1jZDQkIQHBysu5+SkoJq1arB0cEO9vZ2JRV6voQQSEpJg6O9rVlMnlDcfGJjY9GrZ0/cuHEDLi4u+GvzZvj7+5dipAUzp2NjLrkIuRpJ//3fwc4WUuuy383FXI4NwFxMFXMxXeaUj6FzkUnKyBleZ2dnyGQyJCQk6LUnJCTA3d09z3Vmz56NESNGYNy4cQCAxo0bIz09HRMmTMDMmTOLtU2FQpFn31CpRAKpAQ5Yzml/iYH2V9qKk8+BAwfQr18/PH78GLVr10ZERARq1apVekEWkjkdG3PJRTwTe1nPBQA+/PBD/Pjjj0hMTMSGjRvR/9VXjR3SSzHl15lEIsGmTZsQGBiY5+M3btxAjRo1cPLkSXh7e2NvVBQ6d+qEh48eoWKFCli1ahXeeustJCUlFbiv57dlbKZ8XIrKnHIBzCsfQ+dSlH0Y9aI1uVwOHx8fREZG6tq0Wi0iIyPRqlWrPNfJyMiAVKoftkwmA/D0L4vibJOMa926dejatSseP36Mli1b4vDhwyZR7BLlGDVqFCQSCSQSCeRyOWrXro358+cjOzv7pbZ78eJFzJs3Dz8uXYrzsZfQo0ePEorYfN24cQMSiQSnTp0q8W1Xq1YNd+/eRaNGjfJ8fPDgwbh06VKJbIuIDMvoXRqCg4MRFBQEX19f+Pn5YcmSJUhPT8fo0aMBACNHjkSVKlWwaNEiAECfPn2wePFiNGvWTNelYfbs2ejTp4+u8C1om2QahBD46quv8M477wAAAgMDsWbNGlhbWxs5MqLcunfvjpUrV0KlUmH79u2YPHkyLC0t9S6QLSyNRgOJRIKrV68CAPr164eklLRij0KSlZUFS0vLYq1L/08mk+X7SyAAWFlZwcrKqkS2RUSGZfRhyQYPHowvvvgCc+bMgbe3N06dOoWIiAjdRWfx8fG4e/eubvlZs2bhnXfewaxZs9CgQQOMHTsW3bp1w08//VTobZLxaTQavPXWW7pi980338T69etZ7JLJUigUcHd3h4eHB9544w106dIFmzdvBvB0pJd3330XVapUgY2NDfz9/REVFaVbd9WqVXB0dMTmzZvRoEEDKBQKjBkzBn369AEAWMhkcK7gAODpL1Lz589H1apVoVAo4O3tjYiICN22cs5whoeHo3379lAqlVizZg1GjRqFwMBALFy4EG5ubnB0dNSdhZ4+fToqVqyIqlWrYuXKlXp5vf/++6hbty6sra1Rs2ZNzJ49G1lZWbrHP/zwQ3h7e2P16tXw9PSEg4MDhgwZgtTU/79YRKvV4rPPPkPt2rVhpVSiaaOGWLhgge7xW7du4bXXXoOjoyMqVqyIfv364caNG/k+148fP8brr78OFxcXWFlZoU6dOrq4a9SoAQBo1qwZJBIJOnToAAA4fvw4unbtCmdnZzg4OKB9+/Y4ceJErm3fvXsXPXr0gJWVFWrWrIn169fnem7zO3uccxxzeHp66s78P3vLa1tRUVGQSCSIjIyEr68vrK2tERAQgLi4OL19fPzxx3B1dYWdnR3GjRuHGTNmmESXCKIyT1AuycnJAoBITk42yP40Wq148DhZaLRag+yvtBWUT0ZGhnj11VcFAAFAfPHFF0Jrormb07Exl1y0GSrxoMMH4kGHD0R2eqZB9hkUFCT69eun19a3b1/RvHlzIYQQ48aNEwEBAWL//v3iypUr4vPPPxcKhUJcunRJCCHEypUrhaWlpQgICBCHDh0SsbGxIjk5WaxcuVIAEHf+/Vecj70kNFqtWLx4sbC3txdr164VsbGx4r333hOWlpa6bV2/fl0AEJ6enmLDhg3i2rVr4t9//xVBQUHCzs5OTJ48WcTGxorQ0FABQHTr1k0sWLBAXLp0SXz00UfC0tJS3Lp1S5fHRx99JA4dOiSuX78uNm/eLNzc3MSnn36qe3zu3LnC1tZW9O/fX5w9e1bs379fuLu7iw8++EC3zHvvvScqVKggVq1aJS5dviy2bo8QPy1bJoQQQq1Wi/r164sxY8aIM2fOiAsXLohhw4YJLy8voVKp8ny+J0+eLLy9vcXx48fF9evXxe7du8XmzZuFEEJER0cLAGLPnj3i7t274uHDh0IIISIjI8Xq1avFxYsXxYULF8TYsWOFm5ubSElJ0W0XgHBychI///yziIuLE7NmzRIymUxcuHBB77k9efLk023u3SsAiIePHumOo4ODg257iYmJ4u7du+Lu3bvi9u3bomXLlqJt27Z5bmvfvn0CgPD39xdRUVHi/Pnzom3btiIgIEC3vd9++00olUqxYsUKERcXJ+bNmyfs7e1F06ZN83yeisJc3v9CmFcuQphXPobOpSj1GgvePLDgfTkvyuf+/fuiZcuWAoCQy+UiLCzMCBEWnjkdG3PJRZudLTIPXxQPdp8Q2VlZBtnnswWvVqsVu3fvFgqFQrz77rvi5s2bQiaTiTt37uit07lzZxESEiKEELrC9tSpU3rLbNq0SQDQOzaVK1cWCxYs0FuuRYsWYtKkSUKI/y+klixZkitGDw8PodFodG1eXl66AkwIIbKzs4WNjY1Yu3Ztvrl+/vnnwsfHR3d/7ty5wtraWq9wnD59uvD39xdCCJGSkiIUCoX4+eefhRC5X2erV68WXl5een/UqlQqYWVlJXbu3JlnDH369BGjR4/O87HnC8n8aDQaYWdnJ7Zs2aJrAyAmTpyot5y/v79444038tx2QQXvs6ZOnSo8PDxEYmJintvKKXj37NmjW2fbtm0CgHjy5IkulsmTJ+ttt3Xr1ix4n2NOuQhhXvmYcsFr9D68VH5cvXoVPXr0wOXLl1GhQgX89ddfaNu2rbHDojJGIpPBsqUXkJwKyX/99g1h69atsLW1RVZWFrRaLYYNG4YPP/wQUVFR0Gg0qFu3rt7yKpUKTk5OuvtyuRxNmjR54T5SUlLw77//onXr1nrtrVu3xunTp/XafH19c63fsGFDvYt63dzc9C6akslkcHJy0ptmPTw8HN988w2uXr2KtLQ0ZGdnw97eXm+7np6esLP7/yEaK1WqpNvGxYsXoVKp0Llz5zxzOn36NK5cuaK3PgBkZmbq+jA/74033sCAAQNw4sQJvPLKKwgMDERAQECey+ZISEjArFmzEBUVhcTERGg0GmRkZCA+Pl5vuecvXm7VqtVLXwC3bNkyhIaG4vDhw3BxcXnhss++BipVqgQASExMRPXq1REXF4dJkybpLe/n54e9e/e+VHxEZAIXrVH5EB0djd69e+P+/fvw8PDAjh07UL9+fWOHRVRoHTt2xI8//gi5XI7KlSvDwuLpx2daWhpkMhliYmJ0F87msLW11f3fysqqRMeltLGxydX2/IVrEokkzzat9ungQUeOHMHrr7+OefPmoVu3bnBwcEBYWBi+/PLLArebs42CLuJKS0uDj48P1qxZk+ux/IrDHj164ObNm9i+fTt2796Nzp07Y/Lkyfjiiy/y3U9QUBAePnyIr7/+Gh4eHlAoFGjVqhXUavUL43tZ+/btw5tvvom1a9cW+AcNoP9c5rwecp5LIio9Rr9ojczf5s2b0aFDB9y/fx/NmzfH0aNHWexSsYlsDVQRJ4C9ZyGyNQbbr42NDWrXro3q1avril3g6cVTGo0GiYmJqF27tt6tqFfp29vbo3Llyjh06JBe+6FDh9CgQYMSyeNZhw8fhoeHB2bOnAlfX1/UqVMHN2/eLNI26tSpAysrK72hIJ/VvHlzXL58Ga6urrmeHwcHh3y36+LigqCgIPz2229YsmQJli1bBuDpmXLg6YWvzzp06BCmTp2Knj17omHDhlAoFHjw4EGu7R49ejTX/eJ+Hl25cgUDBw7EBx98gP79+xdrG8/y8vLC8ePH9dqev09ExcMzvFSqfvjhB7z55pvQarXo0aMH/vjjD72zXkRFlqVBxmcbn/6/uy9gadyPsbp16+L111/HyJEj8eWXX6JZs2a4f/8+IiMj0aRJE/Tq1atI25s+fTrmzp2LWrVqwdvbGytXrsSpU6fyPEP6surUqYP4+HiEhYWhRYsW2LZtGzZt2lSkbSiVSrz//vt47733IJfL0SogANeu30T8zesYP24cXn/9dXz++efo16+fbvSJmzdvYuPGjXjvvfdQtWrVXNucM2cOfHx80LBhQ6hUKmzdulVXlLq6usLKygoRERGoWrUqlEolHBwcUKdOHaxevRq+vr5ISUnB9OnT8zz7vG7dOvj6+qJNmzZYs2YNoqOjERoaWuTn7smTJ+jTpw+aNWuGCRMm4N69e7rHijsc2Ztvvonx48fD19cXAQEBCA8Px5kzZ1CzZs1ibY+I/h/P8FKp0Gq1CJkxA5MnT4ZWq8W4ceOwefNmFrtkllauXImRI0finXfegZeXFwIDA3H8+HFUr169yNuaOnUqgoOD8c4776Bx48aIiIjA5s2bUadOnRKPu2/fvnj77bcxZcoUeHt74/Dhw8Waon327Nl45513MGfOHDRs0ADjxozG/f/6+FpbW2P//v2oXr06+vfvj/r162Ps2LHIzMzM1Vc4h1wuR0hICJo0aYJ27dpBJpMhLCwMAGBhYYFvvvkGP/30EypXrox+/foBAEJDQ/H48WM0b94cI0aMwNSpU+Hq6ppr2/PmzUNYWBiaNGmCX3/9FWvXri3W2fOEhATExsYiMjISlStXRqVKlXS34nr99dcREhKCd999F82bN8f169cxatQoKJXKYm+TiJ6SCCEKPxFxOZGSkgIHBwckJyfn+4FckrRC4HFyKio42JX5aQUB4ElmJoYPH4GNG56Ob/nRRx9h5syZZXKOcHM6NuaSi3iixqOe8wAAjtvmQGZdvMkaTIm5HBuAuZS0rl27wt3dHatXr36p7ZhCLiXFnHIBzCsfQ+dSlHqNXRqoRKWmpqJv376IioqChYUFQkNDMXLkSGOHRURk8jIyMrB06VJ069YNMpkMa9euxZ49e7B7925jh0ZU5r1UwXvlyhVcvXoV7dq1g5WVFYQQZfIsHpWcr776ClFRUbC1s8OGDRvwSteuxg6JiKhMkEgk2L59OxYsWIDMzEx4eXlhw4YN6NKli7FDIyrzilXwPnz4EIMHD8bevXshkUhw+fJl1KxZE2PHjkWFChVyDWlD5cfGjU8vJlq46BN+SBMRFYGVlRX27Nlj7DCIzFKxLlp7++23YWFhgfj4eFhbW+vaBw8erDfnO5Uv169fx+nTpyGTydC9R09jh0NEREQEoJhneHft2oWdO3fmGk6mOGM4kvnYvHkzAKBt27aoWLGikaMhsyWXwWbOEKRnPAHkhptpjYiIyq5iFbzp6el6Z3ZzPHr0CApF2b9imornzz//BPB0qCOi0iKRySDv0AjpBp5amIiIyq5idWlo27Ytfv31V939nGkmP/vsM3Ts2LHEgqOy49GjRzhw4AAAoO9/42ISERERmYJineH97LPP0LlzZ/zzzz9Qq9V47733cP78eTx69CjXlJhUPmzbtg0ajQZNmjRBjRo18Dg51dghkZkSGg3U+y8AGU8gujUHLDi6IhERvVixzvA2atQIly5dQps2bdCvXz+kp6ejf//+OHnyJGrVqlXSMVIZkNOdoR/P7lJpU2uQPj8M+OIvQK0xdjRERFQGFPnUSFZWFrp3746lS5di5syZpRETlTFPnjzBzp07AQCBgYHGDYaIiIjoOUU+w2tpaYkzZ86URixURkVGRiI9PR3VqlVDs2bNjB0OERERkZ5idWkYPnw4QkNDSzoWKqP++usvAE9HZ+BMe0RERGRqinW1R3Z2NlasWIE9e/bAx8cHNjY2eo8vXry4RIIj06fRaHTj77I7AxEREZmiYhW8586dQ/PmzQEAly5d0nuMZ/jKl2PHjiExMREODg5o3769scMhIiIiyqVYBe++fftKOg4qo3K6M/Tq1QuWlpZGjoaIiIgot5cewPL27dsAkGuaYSofOBwZGZylDNbv9UfGk0zAkjOtERFRwYp10ZpWq8X8+fPh4OAADw8PeHh4wNHRER999BG0Wm1Jx0gmKjY2FpcuXYJcLkf37t2NHQ6VExILGRTdmwOdGkNiwYKXiIgKVqwzvDNnzkRoaCg++eQTtG7dGgBw8OBBfPjhh8jMzMSCBQtKNEgyTTlndzt16gR7e3vjBkNERESUj2IVvL/88guWL1+Ovn376tqaNGmCKlWqYNKkSSx4y4mc/rvszkCGJDQaZEVfBtKfQHRozKmFiYioQMXq0vDo0SPUq1cvV3u9evXw6NGjIm/v+++/h6enJ5RKJfz9/REdHZ3vsh06dIBEIsl169Wrl26ZtLQ0TJkyBVWrVoWVlRUaNGiApUuXFjkuyt/du3dx7NgxAND7w4eo1Kk1SPtgNbBgPacWJiKiQilWwdu0aVN89913udq/++47NG3atEjbCg8PR3BwMObOnYsTJ06gadOm6NatGxITE/NcfuPGjbh7967udu7cOchkMgwaNEi3THBwMCIiIvDbb7/h4sWLeOuttzBlyhTdeLH08rZs2QIhBPz8/FC5cmVjh0NERESUr2L9FvjZZ5+hV69e2LNnD1q1agUAOHLkCG7duoXt27cXaVuLFy/G+PHjMXr0aADA0qVLsW3bNqxYsQIzZszItXzFihX17oeFhcHa2lqv4D18+DCCgoLQoUMHAMCECRPw008/ITo6mmcjS0hOdwZONkFERESmrlgFb/v27REXF4cffvgBsbGxAID+/ftj0qRJRTrbp1arERMTg5CQEF2bVCpFly5dcOTIkUJtIzQ0FEOGDNGb7S0gIACbN2/GmDFjULlyZURFReHSpUv46quv8tyGSqWCSqXS3U9JSQEAaIWAVohC51Nc4r99CCFQFsa4SE1NxZ49ewAAffr2zfUclbV8XoS5mB7xzOtNGOg9WtrM5dgAzMVUMRfTZU75GDqXonz+F/tqjypVqrz0xWkPHjyARqOBm5ubXrubm5uukH6R6OhonDt3DqGhoXrt3377LSZMmICqVavCwsICUqkUP//8M9q1a5fndhYtWoR58+blak9KToVGGG7muKSUNIPt62Vs/utPqNVq1KhZE+6Vq+Jxcmqey5WVfAqDuZiQTLXuv8mpaUCW+gULly1l/tg8g7mYJuZiuswpH0PlkpqSd/2Rl2IVvCtXroStra1eNwIAWLduHTIyMhAUFFSczRZZaGgoGjduDD8/P732b7/9FkePHsXmzZvh4eGB/fv3Y/LkyahcuTK6dOmSazshISEIDg7W3U9JSUG1atXg6GAHe3u7Us9DCIGklDQ42tuWiamZ9+zeBQDo/+qrqOiYeziyspbPizAX0yPkaiT9938HO1tIrRXGDKdEmMuxAZiLqWIupsuc8jF0LjJJKZ/hXbRoEX766adc7a6urpgwYUKhC15nZ2fIZDIkJCTotSckJMDd3f2F66anpyMsLAzz58/Xa3/y5Ak++OADbNq0STdyQ5MmTXDq1Cl88cUXeRa8CoUCCkXuL02pRAKpAQ5Yzml/iYH29zKysrKwfds2AE/77+YVb1nKpyDMxfSIZ2Iv67nkMJdjAzAXU8VcTJc55WPoXIqyj2KN0hAfH48aNWrkavfw8EB8fHyhtyOXy+Hj44PIyEhdm1arRWRkpO5iuPysW7cOKpUKw4cP12vPyspCVlYWpFL91GQyGWeBKwEHDhxAUlISXFxcCjxGRKXCUgarqb2B8V05tTARERVKsc7wurq64syZM/D09NRrP336NJycnIq0reDgYAQFBcHX1xd+fn5YsmQJ0tPTdaM2jBw5ElWqVMGiRYv01gsNDUVgYGCu/dnb26N9+/aYPn06rKys4OHhgb///hu//vorFi9eXPRkSU/O7Gp9+vSBTMZigwxPYiGDMrAlniSncmphIiIqlGIVvEOHDsXUqVNhZ2enuxDs77//xrRp0zBkyJAibWvw4MG4f/8+5syZg3v37sHb2xsRERG6C9ni4+Nzna2Ni4vDwYMHsWvXrjy3GRYWhpCQELz++ut49OgRPDw8sGDBAkycOLEY2VIOIQSHIyMiIqIyRyJE0cf0UavVGDFiBNatWweL/6b11Gq1GDlyJJYuXQq5XF7igRpSSkoKHBwckJycDHv73BdllTStEHicnIoKDnYm3X/n5MmTaN68OaytrfHgwQNYWVnluVxZyacwmIvpERot1GeuIy39CRxb1ofMDM7ymsuxAZiLqWIupsuc8jF0LkWp14p1hlculyM8PBwff/wxTp06BSsrKzRu3BgeHh7FCpjKhpyzu6+88kq+xS5RqVNnIy14xdP/b5sDmEHBS0REpavY4/ACQJ06dVCnTh1oNBqcPXsW9vb2qFChQknFRiaG3RmIiIioLCrWKA1vvfWWbrIHjUaD9u3bo3nz5qhWrRqioqJKMj4yETdu3MCpU6cglUp1w70RERERlQXFKnjXr1+Ppk2bAgC2bNmCa9euITY2Fm+//TZmzpxZogGSadi8eTMAoG3btnB2djZyNERERESFV6yC98GDB7qJIbZv347XXnsNdevWxZgxY3D27NkSDZBMQ85wZP369TNuIERERERFVKyC183NDRcuXIBGo0FERAS6du0KAMjIyODYrGbo0aNH2L9/PwAWvERERFT2FOuitdGjR+O1115DpUqVIJFIdNP1Hjt2DPXq1SvRAMn4tm/fDo1Gg8aNG6NmzZrGDoeIiIioSIpV8H744Ydo1KgRbt26hUGDBkGhUAB4On3vjBkzSjRAMj52ZyCTYiGF1YRueJKpAiyK9SMVERGVM8UelmzgwIEAgNu3b0Or1UIqlSIoKKjEAiPTkJmZiYiICAAcjoxMg8TSAsohbZ9OLWz5UiMrEhFROfHSp0caNGiAGzdulEAoZIoiIyORnp6OqlWronnz5sYOh4iIiKjIXrrgLcbMxFSG5Ew20a9fP0jK+JSHZB6ERovs2NvA5bsQGq2xwyEiojKAvwdSvrRarW78XfbfJZOhzkbqpKVP/8+phYmIqBBe+gzvBx98gIoVK5ZELGRijh07hoSEBNjb26N9+/bGDoeIiIioWF76DG9ISEhJxEEmKKc7Q69evSCXy40cDREREVHxlOiYPrdu3cKYMWNKcpNkRByOjIiIiMxBiRa8jx49wi+//FKSmyQjiY2NRVxcHCwtLdGjRw9jh0NERERUbEXq0pBzAVN+rl279lLBkOnI6c7QqVMn2NvbGzkaIiIiouIrUsEbGBgIiUTywqHIOHSVeXh2ODIiIiKisqxIXRoqVaqEjRs3QqvV5nk7ceJEacVJBnTv3j0cPXoUANC3b18jR0P0HAsplCM7AoNbc2phIiIqlCJ9W/j4+CAmJibfxws6+0tlw5YtWyCEQIsWLVClShVjh0OkR2JpAatRnYEhbTi1MBERFUqRvi2mT5+O9PT0fB+vXbs29u3b99JBkXHldGcIDAw0biBEREREJaBIBW+VKlVQo0aNfB+3sbHhBAVlXFpaGvbs2QOA/XfJNAmtFpobiUBaOkRDG0DGmdaIiOjFitSloU6dOrh//77u/uDBg5GQkFDiQZHx7Ny5EyqVCrVr10aDBg2MHQ5RbqpspIz9Fpi2AlBlGzsaIiIqA4pU8D7fP3f79u0v7OJAZc+zk01wxA0iIiIyB7zEmXSysrKwbds2AOzOQEREROajSAWvRCLJddaPZwHNx8GDB/H48WM4OzsjICDA2OEQERERlYgid2kYNWoU+vfvj/79+yMzMxMTJ07U3c+5FdX3338PT09PKJVK+Pv7Izo6Ot9lO3TooCu8n7316tVLb7mLFy+ib9++cHBwgI2NDVq0aIH4+Pgix1ae5HRn6NOnD2S8EIiIiIjMRJFGaQgKCtK7P3z48JcOIDw8HMHBwVi6dCn8/f2xZMkSdOvWDXFxcXB1dc21/MaNG6FWq3X3Hz58iKZNm2LQoEG6tqtXr6JNmzYYO3Ys5s2bB3t7e5w/fx5KpfKl4zVXQggOR0ZERERmqUgF78qVK0s8gMWLF2P8+PEYPXo0AGDp0qXYtm0bVqxYgRkzZuRavmLFinr3w8LCYG1trVfwzpw5Ez179sRnn32ma6tVq1aJx25OTp8+jZs3b8LKygpdunQxdjhEREREJcao0xSp1WrExMQgJCRE1yaVStGlSxccOXKkUNsIDQ3FkCFDYGNjAwDQarXYtm0b3nvvPXTr1g0nT55EjRo1EBISku+ZS5VKBZVKpbufkpLydFtCQGuAmeNyRr8QQkBb6nvLW053hq6vvAKlldVL5W0K+ZQU5mJ6hEwCxWutoVJlQcgkBnmPljZzOTYAczFVzMV0mVM+hs6lKJ//Ri14Hzx4AI1GAzc3N712Nzc3xMbGFrh+dHQ0zp07h9DQUF1bYmIi0tLS8Mknn+Djjz/Gp59+ioiICPTv3x/79u3Lc2KMRYsWYd68ebnak5JToRGGuygvKSXNYPt63sZNmwAAXV/phsfJqSWyTWPmU9KYi4kZ2gYAkPwkE3iSaeRgSo5ZHJv/MBfTxFxMlznlY6hcUlMKX6+U6YnoQ0ND0bhxY/j5+enatNqnf1P069cPb7/9NgDA29sbhw8fxtKlS/MseENCQhAcHKy7n5KSgmrVqsHRwQ729nalnMXTv4SSUtLgaG9rlFEvbt68ibNnzkAqleK1QQNRweHlcjZ2PiWJuZgmc8oFMK98mItpYi6my5zyMXQuMkkZOcPr7OwMmUyWa7a2hIQEuLu7v3Dd9PR0hIWFYf78+bm2aWFhkWuWsPr16+PgwYN5bkuhUEChUORql0okkBrggOWc9pcYaH/P27plCwCgTZs2cHVxeentGTufksRcTI/QaqFJSAJS0wE7G0ilZX84cXM5NgBzMVXMxXSZUz6GzqUo+zDqN4VcLoePjw8iIyN1bVqtFpGRkWjVqtUL1123bh1UKlWukSLkcjlatGiBuLg4vfZLly7Bw8Oj5II3I8/OrkZk8lTZSBn2JfC/pZxamIiICsXoXRqCg4MRFBQEX19f+Pn5YcmSJUhPT9eN2jBy5EhUqVIFixYt0lsvNDQUgYGBcHJyyrXN6dOnY/DgwWjXrh06duyIiIgIbNmyBVFRUYZIqUx5/Pgx/v77bwAseImIiMg8Gb3gHTx4MO7fv485c+bg3r178Pb2RkREhO5Ctvj4+Fw/WcbFxeHgwYPYtWtXntt89dVXsXTpUixatAhTp06Fl5cXNmzYgDZt2pR6PmXNtm3boNFo0KhRIw7dRkRERGbJ6AUvAEyZMgVTpkzJ87G8zsp6eXnphr7Iz5gxYzBmzJiSCM+s5Uw2wbO7REREZK7K/tUeVGyZmZmIiIgAwNnViIiIyHyx4C3H9u7di7S0NFSpUgU+Pj7GDoeIiIioVLDgLcee7c5Q1sf+IyIiIsqPSfThJcPTarXYvHkzAPbfpTJGJoWinz9UKjUg49/sRERUMBa85VR0dDTu3bsHe3t7dOjQwdjhEBWaRG4B62l9oEpOhUTOjzAiIioYT4+UUzndGXr27Am5XG7kaIiIiIhKDwvecoqzq1FZJYSANikdSM4ocHhCIiIigAVvuRQXF4fY2FhYWlqiR48exg6HqGgys5DcfxEw6lsgM8vY0RARURnAgrccyunO0LFjRzg4OBg5GiIiIqLSxYK3HMopeDnZBBEREZUHLHjLmYSEBBw5cgQA0LdvXyNHQ0RERFT6WPCWM1u2bIEQAr6+vqhSpYqxwyEiIiIqdSx4yxl2ZyAiIqLyhgVvOZKWlobdu3cD4HBkREREVH5wmqJyZNeuXVCpVKhVqxYaNmxo7HCIikcmhbxbM6jVWZxamIiICoUFbzny7GQTEonEuMEQFZNEbgGb9wdAzamFiYiokHh6pJzIzs7Gtm3bALA7AxEREZUvLHjLiYMHD+LRo0dwdnZGQECAscMhKjYhBMQTNZCp5tTCRERUKCx4y4mc7gy9e/eGhQV/BqYyLDMLSb3mA0O/4tTCRERUKCx4ywEhBIcjIyIionKLBW85cObMGdy4cQNWVlbo2rWrscMhIiIiMigWvOVAztndrl27wtra2sjREBERERkWC95yIKf/LrszEBERUXnEgtfMxcfH4+TJk5BKpejdu7exwyEiIiIyOBa8Zm7z5s0AgNatW8PFxcXI0RAREREZHsenMnPPzq5GZBZkEli2a4isrGxAxhkDiYioYCZxhvf777+Hp6cnlEol/P39ER0dne+yHTp0gEQiyXXr1atXnstPnDgREokES5YsKaXoTdfjx4/x999/A2DBS+ZDIreE7YdDgfcCIZFbGjscIiIqA4xe8IaHhyM4OBhz587FiRMn0LRpU3Tr1g2JiYl5Lr9x40bcvXtXdzt37hxkMhkGDRqUa9lNmzbh6NGjqFy5cmmnYZK2b9+O7OxsNGzYELVr1zZ2OERERERGYfSCd/HixRg/fjxGjx6NBg0aYOnSpbC2tsaKFSvyXL5ixYpwd3fX3Xbv3g1ra+tcBe+dO3fw5ptvYs2aNbC0LJ9ngXKGI+PZXSIiIirPjNqHV61WIyYmBiEhIbo2qVSKLl264MiRI4XaRmhoKIYMGQIbGxtdm1arxYgRIzB9+nQ0bNiwwG2oVCqoVCrd/ZSUlKfbEQJaIQqbTrGJ//YhhIC2hLapUqmwY8cOAEDffv0MkkeO0sjHWJiL6RFP1E+nFgag3TobsFYYOaKXZy7HBmAupoq5mC5zysfQuRSltjFqwfvgwQNoNBq4ubnptbu5uSE2NrbA9aOjo3Hu3DmEhobqtX/66aewsLDA1KlTCxXHokWLMG/evFztScmp0AjDXRSTlJJWYtvas3s30tLS4F6pEmrWrovHyakltu3CKsl8jI25mJBMte6/yalpQJb6BQuXLWX+2DyDuZgm5mK6zCkfQ+WSmlL42qZMj9IQGhqKxo0bw8/PT9cWExODr7/+GidOnIBEUrhiNSQkBMHBwbr7KSkpqFatGhwd7GBvb1ficT9PCIGklDQ42tsWOuaC7N2zCwAQ2K8fnCo4lMg2C6s08jEW5mJ6hFyNpP/+72BnC6mZnOE1h2MDMBdTxVxMlznlY+hcZJIycobX2dkZMpkMCQkJeu0JCQlwd3d/4brp6ekICwvD/Pnz9doPHDiAxMREVK9eXdem0WjwzjvvYMmSJbhx40aubSkUCigUub80pRIJpAY4YDmn/SUltD+tVqsbfzcwMNAgOejt/79/SyofY2Iupkc8E3tZzyWHuRwbgLmYKuZiuswpH0PnUpR9GPWiNblcDh8fH0RGRuratFotIiMj0apVqxeuu27dOqhUKgwfPlyvfcSIEThz5gxOnTqlu1WuXBnTp0/Hzp07SyUPU3P8+HHcu3cPdnZ26NChg7HDISIiIjIqo3dpCA4ORlBQEHx9feHn54clS5YgPT0do0ePBgCMHDkSVapUwaJFi/TWCw0NRWBgIJycnPTanZyccrVZWlrC3d0dXl5epZuMiciZbKJnz555nrkmIiIiKk+MXvAOHjwY9+/fx5w5c3Dv3j14e3sjIiJCdyFbfHw8pFL9E9FxcXE4ePAgdu3aZYyQTR6HIyMiIiL6f0YveAFgypQpmDJlSp6PRUVF5Wrz8vLSDX1RGHn12zVXly5dwsWLF2FpaYmePXsaOxyikieTwMK/LrKzObUwEREVjkkUvFRycs7udujQAQ4Ohh2dgcgQJHJL2C0aicfJqZxamIiICsXoM61RycopeAMDA40bCBEREZGJYMFrRhISEnD48GEAQN++fY0cDREREZFpYMFrRrZu3QohBHx8fFC1alVjh0NUKsQTNR73nAcMWQzxxHxmWSMiotLDPrxmhN0ZqNzIzDJ2BEREVIbwDK+ZSE9Px+7duwFwODIiIiKiZ7HgNRO7du1CZmYmatasiUaNGhk7HCIiIiKTwYLXTOTMrtavXz9Iyvhc3EREREQliQWvGcjOzsbWrVsBsDsDERER0fNY8JqBQ4cO4dGjR3ByckLr1q2NHQ4RERGRSeEoDWYgpztD7969YWHBQ0pmTiqBRVNPZGdrACm77xARUcFYHZVxQggOR0blikRhCbuvxj2dWljBqYWJiKhg7NJQxp09exbXr1+HUqlE165djR0OERERkclhwVvG5ZzdfeWVV2BjY2PkaIiIiIhMDwveMu7Z4ciIygPxRI2kVxcCQd9wamEiIioU9uEtw27duoUTJ05AIpGgd+/exg6HyGBEcoaxQyAiojKEZ3jLsM2bNwMAWrduDVdXVyNHQ0RERGSaWPCWYezOQERERFQwFrxlVFJSEqKiogCw4CUiIiJ6ERa8ZdT27duRnZ2NBg0aoE6dOsYOh4iIiMhkseAto3KGI+PZXSIiIqIX4ygNZZBKpcKOHTsAcHY1KoekEsi8qkCj4dTCRERUOCx4y6B9+/YhNTUVlSpVgq+vr7HDITIoicIS9j++wamFiYio0NiloQx6tjuDVMpDSERERPQirJbKGK1Wy/67REREREXAgreM+eeff3D37l3Y2dmhY8eOxg6HyOBEphrJQ78AJvwIkcmphYmIqGAmUfB+//338PT0hFKphL+/P6Kjo/NdtkOHDpBIJLluvXr1AgBkZWXh/fffR+PGjWFjY4PKlStj5MiR+Pfffw2VTqnKmWyiR48eUCgUxg2GyBgEoE1IAu6nAMLYwRARUVlg9II3PDwcwcHBmDt3Lk6cOIGmTZuiW7duSExMzHP5jRs34u7du7rbuXPnIJPJMGjQIABARkYGTpw4gdmzZ+PEiRPYuHEj4uLi0LdvX0OmVWrYnYGIiIioaIw+SsPixYsxfvx4jB49GgCwdOlSbNu2DStWrMCMGTNyLV+xYkW9+2FhYbC2ttYVvA4ODti9e7feMt999x38/PwQHx+P6tWrl1Impe/y5cu4cOECLCws0LNnT2OHQ0RERFQmGLXgVavViImJQUhIiK5NKpWiS5cuOHLkSKG2ERoaiiFDhsDGxibfZZKTkyGRSODo6Jjn4yqVCiqVSnc/JSUFAKAVAlpR+r+Ziv/2IYSA9gXL5XRnaN+hA+wdHAwSW3EUNp+ygLmYHvHM614Y6D1a2szl2ADMxVQxF9NlTvkYOpeifP4bteB98OABNBoN3Nzc9Nrd3NwQGxtb4PrR0dE4d+4cQkND810mMzMT77//PoYOHQp7e/s8l1m0aBHmzZuXqz0pORUaYbiB7ZNS0l74+MaNmwAAr7zSHY+TUw0R0kspKJ+yhLmYkGcuVEtOTQOyzOfCtTJ/bJ7BXEwTczFd5pSPoXJJTSl8LWT0Lg0vIzQ0FI0bN4afn1+ej2dlZeG1116DEAI//vhjvtsJCQlBcHCw7n5KSgqqVasGRwc72NvblXjczxNCICklDY72tpBI8i6wExMTcezYUQDA4MGDUMGh9OMqrsLkU1YwF9Mj5Gok/fd/BztbSK3L/sWb5nJsAOZiqpiL6TKnfAydi0xSRs7wOjs7QyaTISEhQa89ISEB7u7uL1w3PT0dYWFhmD9/fp6P5xS7N2/exN69e/M9uwsACoUizxEPpBIJpAY4YDmn/SUv2N/2bdsghEDz5s3hYeL9kAuTT1nBXEyPkEog9XCFVquBRFq2c8lhLscGYC6mirmYLnPKx9C5FGUfRh2lQS6Xw8fHB5GRkbo2rVaLyMhItGrV6oXrrlu3DiqVCsOHD8/1WE6xe/nyZezZswdOTk4lHruh5YzOEBgYaNxAiIxMopTDYeVU4JtxkCjlxg6HiIjKAKN3aQgODkZQUBB8fX3h5+eHJUuWID09XTdqw8iRI1GlShUsWrRIb73Q0FAEBgbmKmazsrIwcOBAnDhxAlu3boVGo8G9e/cAPB3hQS4ve1+Q6enp2LVrFwAOR0ZERERUVEYveAcPHoz79+9jzpw5uHfvHry9vREREaG7kC0+Ph5Sqf6J6Li4OBw8eFBXBD7rzp072Lx5MwDA29tb77F9+/ahQ4cOpZJHadq9ezcyMzNRo0YNNG7c2NjhEBEREZUpRi94AWDKlCmYMmVKno9FRUXlavPy8tIbmuhZnp6e+T5WVuUMR9avX78y36Gd6GWJTDWSJ/4IaDUQP00GrMr+RWtERFS6TKLgpfxlZ2dj69atANh/lwjA06mFbybq/k9ERFQQo08tTC926NAhPHz4EBUrVkTr1q2NHQ4RERFRmcOC18TljM7Qu3dvWFjwhDwRERFRUbHgNWFCCA5HRkRERPSSWPCasHPnzuHatWtQKpV45ZVXjB0OERERUZnEgteE5Zzd7dq1K2xsbIwcDREREVHZxE6hJuzZ4ciI6D8SQOrmCK1WC3CUPiIiKgQWvCbq9u3biImJgUQiQe/evY0dDpHJkCjlcFj7Lh4np3JqYSIiKhR2aTBRObPFBQQE6GadIyIiIqKiY8FrotidgYiIiKhksOA1QUlJSdi3bx8ADkdG9DyhykLKGz8C03+BUGUZOxwiIioD2IfXBO3YsQPZ2dmoX78+6tSpY+xwiEyLVkATd0f3fyIiooLwDK8J4mQTRERERCWHBa+JUalU2L59OwD23yUiIiIqCSx4TUxUVBRSU1NRqVIltGjRwtjhEBEREZV5LHhNTE53hr59+0Iq5eEhIiIielmsqEyIVqvVFbzszkBERERUMjhKgwmJiYnBv//+C1tbW3Tq1MnY4RCZLImDNYTgCA1ERFQ4LHhNyF//TTbRo0cPKBQK4wZDZKIkVnI4bvrg6dTCVpxamIiICsYuDSYkZzphdmcgIiIiKjkseE3EtWtXcf78eVhYWKBnz57GDoeIiIjIbLDgNRE7/ht7t3379qhQoYKRoyEyXUKVhdS3lwOzfufUwkREVCjsw2sicgpezq5GVACtQPbpG7r/ExERFYRneE3A/fv3EX3sKICn4+8SERERUclhwWsCtm7dCq1Wi2bNmqF69erGDoeIiIjIrJhEwfv999/D09MTSqUS/v7+iI6OznfZDh06QCKR5Lr16tVLt4wQAnPmzEGlSpVgZWWFLl264PLly4ZIpVg42QQRERFR6TF6wRseHo7g4GDMnTsXJ06cQNOmTdGtWzckJibmufzGjRtx9+5d3e3cuXOQyWQYNGiQbpnPPvsM33zzDZYuXYpjx47BxsYG3bp1Q2ZmpqHSKpJGDRvCw8MDfVnwEhEREZU4oxe8ixcvxvjx4zF69Gg0aNAAS5cuhbW1NVasWJHn8hUrVoS7u7vutnv3blhbW+sKXiEElixZglmzZqFfv35o0qQJfv31V/z777/487+JHUzNxwsW4J+Tp9GkSRNjh0JERERkdow6SoNarUZMTAxCQkJ0bVKpFF26dMGRI0cKtY3Q0FAMGTIENjY2AIDr16/j3r176NKli24ZBwcH+Pv748iRIxgyZEiubahUKqhUKt39lJQUAIBWCGgNMH2pEAISiUS3z7IuZ8pXIQS0Ro7lZTEX0yOEAJSWgPgvF75nTApzMU3MxXSZUz6GzqUon/9GLXgfPHgAjUYDNzc3vXY3NzfExsYWuH50dDTOnTuH0NBQXdu9e/d023h+mzmPPW/RokWYN29ervak5FRohKTAOEpKUkqawfZlCOaUD3MxMWuDAQDJWWogWW3kYEqOWRyb/zAX08RcTJc55WOoXFJTUgu9bJkehzc0NBSNGzeGn5/fS20nJCQEwcHBuvspKSmoVq0aHB3sYG9v97JhFkgIgaSUNDja2+rO9JZl5pQPczFN5pQLYF75MBfTxFxMlznlY+hcZJIycobX2dkZMpkMCQkJeu0JCQlwd3d/4brp6ekICwvD/Pnz9dpz1ktISEClSpX0tunt7Z3nthQKBRQKRa52qUQCqQEOWM5pf4mB9lfazCkf5mKazCkXwLzyYS6mibmYLnPKx9C5FGUfRr1oTS6Xw8fHB5GRkbo2rVaLyMhItGrV6oXrrlu3DiqVCsOHD9drr1GjBtzd3fW2mZKSgmPHjhW4TSIyfUKdhdSQX4GP10GoObUwEREVzOhdGoKDgxEUFARfX1/4+flhyZIlSE9Px+jRowEAI0eORJUqVbBo0SK99UJDQxEYGAgnJye9dolEgrfeegsff/wx6tSpgxo1amD27NmoXLkyp+0lMgcagexjl3T/JyIiKojRC97Bgwfj/v37mDNnDu7duwdvb29EREToLjqLj4+HVKp/IjouLg4HDx7Erl278tzme++9h/T0dEyYMAFJSUlo06YNIiIioFQqSz0fIiIiIjItRi94AWDKlCmYMmVKno9FRUXlavPy8tINfZEXiUSC+fPn5+rfS0RERETlj9EnniAiIiIiKk0seImIiIjIrLHgJSIiIiKzZhJ9eE1NTv/gnCmGS5tWCKSmpEImEWV+DD7AvPJhLqZHPFEjNfvpVODSlBTIsnOPoV3WmMuxAZiLqWIupsuc8jF0Ljl12ouu68ohEYVZqpy5ffs2qlWrZuwwiIiIiKgAt27dQtWqVV+4DAvePGi1Wvz777+ws7MzyNR4OVMZ37p1C/b29qW+v9JmTvkwF9NkTrkA5pUPczFNzMV0mVM+hs5FCIHU1FRUrlw51xC2z2OXhjxIpdIC/1IoDfb29mX+xf4sc8qHuZgmc8oFMK98mItpYi6my5zyMWQuDg4OhVqOF60RERERkVljwUtEREREZo0FrwlQKBSYO3cuFIqyf7U5YF75MBfTZE65AOaVD3MxTczFdJlTPqacCy9aIyIiIiKzxjO8RERERGTWWPASERERkVljwUtEREREZo0FLxERERGViKioKEgkEiQlJQEAVq1aBUdHR6PGBLDgLTGjRo2CRCLBJ598otf+559/GmS2NkP6/vvv4enpCaVSCX9/f0RHR79w+QULFiAgIADW1tYm8aJ/XlHyuXHjBsaOHYsaNWrAysoKtWrVwty5c6FWqw0Ycf6Kemz69u2L6tWrQ6lUolKlShgxYgT+/fdfA0X7YkXNJYdKpYK3tzckEglOnTpVukEWUlFz8fT0hEQi0bs9/9liLMU5Ltu2bYO/vz+srKxQoUIFBAYGln6ghVSUfHK+yPO6HT9+3IBR562ox+bSpUvo168fnJ2dYW9vjzZt2mDfvn0GivbFiprLiRMn0LVrVzg6OsLJyQkTJkxAWlqagaLVVxrfkfHx8ejVqxesra3h6uqK6dOnIzs7uxSiL7wjR45AJpOhV69eRo2j0ASViKCgIKFUKoWjo6N49OiRrn3Tpk3CnJ7msLAwIZfLxYoVK8T58+fF+PHjhaOjo0hISMh3nTlz5ojFixeL4OBg4eDgYLhgC6Go+ezYsUOMGjVK7Ny5U1y9elX89ddfwtXVVbzzzjsGjjy34hybxYsXiyNHjogbN26IQ4cOiVatWolWrVoZMOq8FSeXHFOnThU9evQQAMTJkydLP9gCFCcXDw8PMX/+fHH37l3dLS0tzYBR5604uaxfv15UqFBB/PjjjyIuLk6cP39ehIeHGzDq/BU1H5VKpXdM7t69K8aNGydq1KghtFqtgaPXV5xjU6dOHdGzZ09x+vRpcenSJTFp0iRhbW0t7t69a8DIcytqLnfu3BEVKlQQEydOFLGxsSI6OloEBASIAQMGGDjy0vmOzM7OFo0aNRJdunQRJ0+eFNu3bxfOzs4iJCSkFDMp2NixY8W0adOEra2tuHPnjq593759AoB4/PixEEKIlStXmsR3v/lUYkYWFBQkevfuLerVqyemT5+ua3++4F2/fr1o0KCBkMvlwsPDQ3zxxRd62/Hw8BALFiwQo0ePFra2tqJatWrip59+0lsmPj5eDBo0SDg4OIgKFSqIvn37iuvXr5dqfjn8/PzE5MmTdfc1Go2oXLmyWLRoUYHrmsqL/lkvk0+Ozz77TNSoUaM0wiuSksjlr7/+EhKJRKjV6tIIsdCKm8v27dtFvXr1xPnz502m4C1OLh4eHuKrr74yQHRFU9RcsrKyRJUqVcTy5csNFWKRvOx7Rq1WCxcXFzF//vzSCrHQiprL/fv3BQCxf/9+XVtKSooAIHbv3l3q8b5IUXP56aefhKurq9BoNLq2M2fOCADi8uXLpR7vs0rjO3L79u1CKpWKe/fu6dp+/PFHYW9vL1QqVYnEXVSpqanC1tZWxMbGisGDB4sFCxboHjPVgpddGkqQTCbDwoUL8e233+L27du5Ho+JicFrr72GIUOG4OzZs/jwww8xe/ZsrFq1Sm+5L7/8Er6+vjh58iQmTZqEN954A3FxcQCArKwsdOvWDXZ2djhw4AAOHToEW1tbdO/evdR/Vler1YiJiUGXLl10bVKpFF26dMGRI0dKdd+loaTySU5ORsWKFUsjxEIriVwePXqENWvWICAgAJaWlqUVaoGKm0tCQgLGjx+P1atXw9ra2hChFuhljssnn3wCJycnNGvWDJ9//rnRf74sTi4nTpzAnTt3IJVK0axZM1SqVAk9evTAuXPnDBV2vkriPbN582Y8fPgQo0ePLq0wC6U4uTg5OcHLywu//vor0tPTkZ2djZ9++gmurq7w8fExVOi5FCcXlUoFuVwOqfT/SxorKysAwMGDB0s34GeU1nfkkSNH0LhxY7i5uenaunXrhpSUFJw/f/6lYi6uP/74A/Xq1YOXlxeGDx+OFStWQJj4tA4seEvYq6++Cm9vb8ydOzfXY4sXL0bnzp0xe/Zs1K1bF6NGjcKUKVPw+eef6y3Xs2dPTJo0CbVr18b7778PZ2dnXb+q8PBwaLVaLF++HI0bN0b9+vWxcuVKxMfHIyoqqlRze/DgATQajd6bDgDc3Nxw7969Ut13aSiJfK5cuYJvv/0W//vf/0ojxEJ7mVzef/992NjYwMnJCfHx8fjrr79KM9QCFScXIQRGjRqFiRMnwtfX1xBhFkpxj8vUqVMRFhaGffv24X//+x8WLlyI9957r7TDfaHi5HLt2jUAwIcffohZs2Zh69atqFChAjp06IBHjx6VeswvUhLv/9DQUHTr1g1Vq1YtjRALrTi5SCQS7NmzBydPnoSdnR2USiUWL16MiIgIVKhQwRBh56k4uXTq1An37t3D559/DrVajcePH2PGjBkAgLt375Z6zDlK6zvy3r17eW4z5zFjCA0NxfDhwwEA3bt3R3JyMv7++2+jxFJYLHhLwaeffopffvkFFy9e1Gu/ePEiWrdurdfWunVrXL58GRqNRtfWpEkT3f8lEgnc3d2RmJgIADh9+jSuXLkCOzs72NrawtbWFhUrVkRmZiauXr1ailkVbOLEibqYbG1tjRpLSSgonzt37qB79+4YNGgQxo8fb4QIC+9FuUyfPh0nT57Erl27IJPJMHLkSJP+Sz2vXL799lukpqYiJCTEyNEVTX7HJTg4GB06dECTJk0wceJEfPnll/j222+hUqmMGO2L5ZWLVqsFAMycORMDBgyAj48PVq5cCYlEgnXr1hkz3AIV9P6/ffs2du7cibFjxxohuqLJKxchBCZPngxXV1ccOHAA0dHRCAwMRJ8+fQxaJBZVXrk0bNgQv/zyC7788ktYW1vD3d0dNWrUgJubm95ZX2Mzl+/IuLg4REdHY+jQoQAACwsLDB48GKGhoUaO7MUsjB2AOWrXrh26deuGkJAQjBo1qsjrP/9zskQi0X1xpKWlwcfHB2vWrMm1nouLS7HiLSxnZ2fIZDIkJCTotSckJMDd3R3z58/Hu+++W6oxlKSXyefff/9Fx44dERAQgGXLlhki3Bd6mVycnZ3h7OyMunXron79+qhWrRqOHj2KVq1aGSL0POMpai579+7FkSNHcs3f7uvri9dffx2//PJLqcedl5J6z/j7+yM7Oxs3btyAl5dXaYX7QsXJpVKlSgCABg0a6NoUCgVq1qyJ+Pj40g/6BV722KxcuRJOTk7o27dvaYdaoOK+Z7Zu3YrHjx/D3t4eAPDDDz9g9+7d+OWXX3RnSA2tuMdl2LBhGDZsGBISEmBjYwOJRILFixejZs2ahgq91L4j3d3dc430kLMPd3f34gdcTKGhocjOzkblypV1bUIIKBQKfPfddwaPp7BM508fM/PJJ59gy5Ytev126tevj0OHDuktd+jQIdStWxcymaxQ223evDkuX74MV1dX1K5dW+/m4OBQojk8Ty6Xw8fHB5GRkbo2rVaLyMhItGrVKldMpq64+dy5cwcdOnTQna0yhTMIJXVscv6wMuaZxOLk8s033+D06dM4deoUTp06he3btwN42gVowYIFRskDKLnjcurUKUilUri6uhoi7DwVJxcfHx8oFArdNQjA0+sQbty4AQ8PD4Pn8KyXOTZCCKxcuRIjR440an/3HMXJJSMjAwByfX5JpVLd54AxvOx7xs3NDba2tggPD4dSqUTXrl3LTOz5adWqFc6ePav7pRcAdu/eDXt7e70/Jg0hOzsbv/76K7788kvd5+2pU6dw+vRpVK5cGWvXrjVoPEVizCvmzElQUJDo16+fXtuIESOEUqnUjdIQExMjpFKpmD9/voiLixOrVq0SVlZWYuXKlbp18ro6u2nTpmLu3LlCCCHS09NFnTp1RIcOHcT+/fvFtWvXxL59+8Sbb74pbt26VYoZPhUWFiYUCoVYtWqVuHDhgpgwYYJwdHTUu3r0eTdv3hQnT54U8+bNE7a2tuLkyZPi5MmTIjU1tdTjLUhR87l9+7aoXbu26Ny5s7h9+7be8ETGVtRcjh49Kr799ltx8uRJcePGDREZGSkCAgJErVq1RGZmpoGj11ec19mzrl+/bjKjNBQ1l8OHD4uvvvpKnDp1Sly9elX89ttvwsXFRYwcOdLAkedWnOMybdo0UaVKFbFz504RGxsrxo4dK1xdXfWGbzSW4r7O9uzZIwCIixcvGijSghU1l/v37wsnJyfRv39/cerUKREXFyfeffddYWlpKU6dOmXg6PUV57h8++23IiYmRsTFxYnvvvtOWFlZia+//tqAUT9VGt+ROcOSvfLKK+LUqVMiIiJCuLi4GGVYsk2bNgm5XC6SkpJyPfbee+8JX19fkx2lgQVvCcmr4L1+/bqQy+V5DktmaWkpqlevLj7//HO9dQoqeIUQ4u7du2LkyJHC2dlZKBQKUbNmTTF+/HiRnJxc0mnl6dtvvxXVq1cXcrlc+Pn5iaNHj75w+aCgIAEg123fvn0GibcgRcln5cqVeeZiKn87FiWXM2fOiI4dO4qKFSsKhUIhPD09xcSJE8Xt27cNGHH+ivo6e5YpFbxCFC2XmJgY4e/vLxwcHIRSqRT169cXCxcuNPofITmKelzUarV45513hKurq7CzsxNdunQR586dM1C0BSvO62zo0KEiICDAANEVTVFzOX78uHjllVdExYoVhZ2dnWjZsqXYvn27gaJ9saLmMmLECFGxYkUhl8tFkyZNxK+//mqgSHMrje/IGzduiB49eggrKyvh7Ows3nnnHZGVlVXKmeTWu3dv0bNnzzwfO3bsmAAgvv76a5MseCVCmPDVKUREREREL8n4nQ+JiIiIiEoRC14iIiIiMmsseImIiIjIrLHgJSIiIiKzxoKXiIiIiMwaC14iIiIiMmsseImIiIjIrLHgJSIiIiKzxoKXiMoNT09PLFmypES3OWrUKAQGBr5wmQ4dOuCtt94q0f2aisLkXxw3btyARCLBqVOn8l0mKioKEokESUlJJbLPkt4eEZkOFrxEZHIkEskLbx9++GGxtnv8+HFMmDChZIMlIiKTZ2HsAIiInnf37l3d/8PDwzFnzhzExcXp2mxtbXX/F0JAo9HAwqLgjzMXF5eSDZReWlGOHxFRcfEMLxGZHHd3d93NwcEBEolEdz82NhZ2dnbYsWMHfHx8oFAocPDgQVy9ehX9+vWDm5sbbG1t0aJFC+zZs0dvu893aZBIJFi+fDleffVVWFtbo06dOti8ebPucY1Gg7Fjx6JGjRqwsrKCl5cXvv766zxjnjdvHlxcXGBv/3/t3X1M1dUfwPE3D8Xzg0LLOx/ABZiYIEwioHADKVKZMRQkSghcOcIxAyRJHowaNxJo9GDWUqYrDdKcE5lDFqxInWOITG7MTKUpDiYiXURI7vn90fzOK2D86vdg7PP675zP+X7OOd/vPx++91yuM+vXr2dkZGTC/Q0PD5Odnc3MmTNxcHAgODiYxsbG+96T/v5+1q1bp80RERFBW1ubFi8qKmLRokXs2bMHT09PXFxcWLNmDb/99ps2xmQyUVpaipeXFzY2NsyZM4d3331Xi7e3txMREYGdnR1ubm68+uqrGI1Gs/vxxhtv4OrqipubG5s2bUIpZbZOk8lESUmJds/8/f355ptvtPidYwP3Pr+J/PTTT4SGhmJra8sTTzxBU1PTfe/T/v37WbBgATY2Nnh6elJWVmYWHx4eJjc3l9mzZ2NjY4OXlxdffPHFuLlu3rzJ888/T1hYGP39/YyMjJCRkYFOp8PW1hYPDw9KSkruux4hxINBCl4hxD/Sm2++iV6vx2Aw4Ofnh9FoZNmyZTQ0NNDa2kp0dDQxMTF0dXXdN8/WrVuJj4/nzJkzLFu2jKSkJPr6+oA/irdZs2ZRU1NDR0cHBQUF5OXlUV1dbZajoaEBg8FAY2Mje/fu5cCBA2zdunXCOTMyMjh+/Dj79u3jzJkzrF69mujoaM6dOzfhNatXr6anp4e6ujpaWloIDAwkMjJSWyvA+fPnOXjwIIcPH+bw4cM0NTWh1+u1+ObNm9Hr9eTn59PR0cFXX33Fo48+CsDg4CDPPfcc06ZN49SpU9TU1HDs2DEyMjK068vKyqiqqmLnzp388MMP9PX18e2335qts6SkhN27d/Ppp59y9uxZNm7cyEsvvTSmUL33+U0kJyeHrKwsWltbCQkJISYmhmvXro07tqWlhfj4eNasWUN7eztFRUXk5+dTVVWljVm7di179+6lsrISg8HAjh07zD4xuKO/v5+oqChMJhP19fW4urpSWVnJoUOHqK6uprOzky+//BJPT88J1y6EeIAoIYR4gO3atUu5uLho7e+++04B6uDBg3967YIFC9SHH36otT08PFRFRYXWBtSWLVu0ttFoVICqq6ubMOfrr7+u4uLitHZycrKaPn26Ghwc1Pq2b9+uHB0d1ejoqFJKqSVLlqjMzEyllFKXLl1SVlZW6vLly2Z5IyMj1ebNm8ed8/vvv1fOzs7q1q1bZv2PPfaY2rFjh1JKqcLCQmVvb68GBga0eE5OjgoODlZKKTUwMKBsbGzU559/Pu4cn332mZo2bZoyGo1aX21trbK0tFRXr15VSiml0+lUaWmpFv/999/VrFmz1MqVK5VSSt26dUvZ29urH3/80Sx3WlqaSkxMVEpN/vlduHBBAUqv14+Z77333jPLdf36daWUUi+++KKKiooyy5OTk6N8fX2VUkp1dnYqQNXX14875518BoNB+fn5qbi4ODU8PKzFN2zYoCIiIpTJZLrv2oUQDx45NCWE+EdavHixWdtoNFJUVERtbS3d3d3cvn2boaGhP33De/fbRQcHB5ydnenp6dH6Pv74Y3bu3ElXVxdDQ0OMjIywaNEisxz+/v7Y29tr7ZCQEIxGI7/++iseHh5mY9vb2xkdHcXHx8esf3h4GDc3t3HX2NbWhtFoHBMfGhri/PnzWtvT0xMnJyetrdPptL0YDAaGh4eJjIwcdw6DwYC/vz8ODg5aX1hYGCaTic7OTmxtbenu7iY4OFiLW1tbs3jxYu1Yw88//8zNmzeJiooyyz0yMkJAQIBZ373PbyIhISFj5jMYDBPuYeXKlWZ9YWFhfPDBB4yOjnL69GmsrKxYsmTJfeeMioriySef5Ouvv8bKykrrT0lJISoqinnz5hEdHc2KFSt49tlnJ7UPIcT/lxS8Qoh/pLsLM4Ds7Gzq6+vZtm0bXl5e2NnZsWrVqvuepQV46KGHzNoWFhaYTCYA9u3bR3Z2NmVlZYSEhODk5MT777/PyZMn//K6jUYjVlZWtLS0mBVTwLgfrd+5RqfTjXvO19XVdVJ7sbOz+8trnqw7531ra2uZOXOmWczGxsasfe/z+1+Y7D1Yvnw5+/fvp6Ojg4ULF2r9gYGBXLhwgbq6Oo4dO0Z8fDxLly41O6MshHgwScErhJgSmpubSUlJITY2Fvij+Lp48eLfzhkaGkp6errWd/cb1Tva2toYGhrSCqoTJ07g6OjI7Nmzx4wNCAhgdHSUnp4ennnmmUmtIzAwkKtXr2Jtbf2Xz4x6e3tjZ2dHQ0MD69atGxOfP38+VVVVDA4OasVoc3MzlpaWzJs3DxcXF3Q6HSdPniQ8PByA27dva+eJAXx9fbGxsaGrq+tP36JO1okTJ8bMd/e54nv30NzcbNbX3NyMj48PVlZWLFy4EJPJRFNTE0uXLp1wTr1ej6OjI5GRkTQ2NuLr66vFnJ2dSUhIICEhgVWrVhEdHU1fXx/Tp0//D+xWCPHfIgWvEGJK8Pb25sCBA8TExGBhYUF+fr72dvPv5Ny9ezdHjx5l7ty57Nmzh1OnTjF37lyzcSMjI6SlpbFlyxYuXrxIYWEhGRkZWFqO/V6wj48PSUlJrF27lrKyMgICAujt7aWhoQE/Pz+WL18+5pqlS5cSEhLCCy+8QGlpKT4+Ply5coXa2lpiY2MndTzA1taW3NxcNm3axMMPP0xYWBi9vb2cPXuWtLQ0kpKSKCwsJDk5maKiInp7e9mwYQMvv/yy9sW2zMxM9Ho93t7ePP7445SXl5v9SIOTkxPZ2dls3LgRk8nE008/zY0bN2hubsbZ2Znk5OR/8wn8caTE29ub+fPnU1FRwfXr10lNTR13bFZWFkFBQRQXF5OQkMDx48f56KOP+OSTT4A/jnwkJyeTmppKZWUl/v7+XLp0iZ6eHuLj481ybdu2jdHRUSIiImhsbNT2q9PpCAgIwNLSkpqaGmbMmGH2ll0I8WCSglcIMSWUl5eTmppKaGgo7u7u5ObmMjAw8Ldyvvbaa7S2tpKQkICFhQWJiYmkp6dTV1dnNi4yMhJvb2/Cw8MZHh4mMTHxvj+OsWvXLt555x2ysrK4fPky7u7uPPXUU6xYsWLc8RYWFhw5coS33nqLV155hd7eXmbMmEF4eLhWjE5Gfn4+1tbWFBQUcOXKFXQ6HevXrwfA3t6eo0ePkpmZSVBQEPb29sTFxVFeXq5dn5WVRXd3N8nJyVhaWpKamkpsbCw3btzQxhQXF/PII49QUlLCL7/8gqurK4GBgeTl5U16nXfT6/Xo9XpOnz6Nl5cXhw4dwt3dfdyxgYGBVFdXU1BQQHFxMTqdjrfffpuUlBRtzPbt28nLyyM9PZ1r164xZ86cCddWUVFhVvQ6OTlRWlrKuXPnsLKyIigoiCNHjoz7h40Q4sFiodQ9/0RRCCGEEEKIKUT+LBVCCCGEEFOaFLxCCCGEEGJKk4JXCCGEEEJMaVLwCiGEEEKIKU0KXiGEEEIIMaVJwSuEEEIIIaY0KXiFEEIIIcSUJgWvEEIIIYSY0qTgFUIIIYQQU5oUvEIIIYQQYkqTglcIIYQQQkxp/wL7Z6Nzq42pRwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# import matplotlib.pyplot as plt\n", "# import numpy as np\n", "\n", "# # Create Figure\n", "# plt.figure(figsize=(8,4))\n", "\n", "# # Prepare Data\n", "# x = [f\"0-{index}\" for index in range(12)]\n", "# x[0] = \"None\"\n", "# x[-1] = \"All\"\n", "# y = [\n", "# 0.8541862652869239,\n", "# 0.8525519848771267,\n", "# 0.8514664143803217,\n", "# 0.8506616257088847,\n", "# 0.8398104265402844,\n", "# 0.8391345249294448,\n", "# 0.8377358490566037,\n", "# 0.8433962264150944,\n", "# 0.8258801141769743,\n", "# 0.816247582205029,\n", "# 0.7917485265225934,\n", "# 0.7019400352733686\n", "# ][::-1]\n", "\n", "# # Stylize Figure\n", "# plt.grid(color='#ECEFF1')\n", "# plt.axvline(x=4, color=\"#EC407A\", linestyle=\"--\")\n", "# plt.title(\"Effect of Frozen Encoder Blocks on Training Performance\")\n", "# plt.ylabel(\"F1-score\")\n", "# plt.xlabel(\"Trainable encoder blocks\")\n", "\n", "# # Plot Data\n", "# plt.plot(x, y, color=\"black\")\n", "\n", "# # Additional Annotation\n", "# plt.annotate(\n", "# 'Performance stabilizing',\n", "# xy=(4, y[4]),\n", "# xytext=(4.5, y[4]-.05),\n", "# arrowprops=dict(\n", "# arrowstyle=\"-|>\",\n", "# connectionstyle=\"arc3\",\n", "# color=\"#00ACC1\")\n", "# )\n", "# plt.savefig(\"multiple_frozen_blocks.png\", dpi=300, bbox_inches='tight')" ] }, { "cell_type": "markdown", "metadata": { "id": "sf785lzMjwiy" }, "source": [ "## Few-shot Classification" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8ybeQ3j6kOk4" }, "outputs": [], "source": [ "from setfit import sample_dataset\n", "\n", "# We simulate a few-shot setting by sampling 16 examples per class\n", "sampled_train_data = sample_dataset(tomatoes[\"train\"], num_samples=16)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 3418, "status": "ok", "timestamp": 1719390247822, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "1Y55TDrmSqHm", "outputId": "84ec7a9c-92ed-4277-dfff-f0cb4cc918d6" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n", " and should_run_async(code)\n", "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n", " warnings.warn(\n", "model_head.pkl not found on HuggingFace Hub, initialising classification head with random weights. You should TRAIN this model on a downstream task to use it for predictions and inference.\n" ] } ], "source": [ "from setfit import SetFitModel\n", "\n", "# Load a pre-trained SentenceTransformer model\n", "model = SetFitModel.from_pretrained(\"sentence-transformers/all-mpnet-base-v2\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 49, "referenced_widgets": [ "630de7830edb4104938bef1c304129c0", "736c3004e2ec48b2b3a8974e465eb838", "54c6443169d84a3c9983d8c5b125c3e7", "bbee47201aac4b8f96aa772cc751d1d8", "9ad6e36197894003809d3ae6fa1b6a2c", "4708f67e67194377a94ec08aa07fe688", "c03a1bf3d2214c3680ab0b983dd582ef", "e777041b66fa484090714094d9516f35", "c5d50176a0a049769731d1c6fdb37f0f", "aa6d83c964d741178534cd0fae6ccec9", "318f91cb3cdd4f6e88681121de392207" ] }, "executionInfo": { "elapsed": 2, "status": "ok", "timestamp": 1719390248689, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "zZ10SpAXdNvC", "outputId": "f9930b86-e077-4233-f7d7-16c5ec1e640d" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "630de7830edb4104938bef1c304129c0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map: 0%| | 0/32 [00:00\n", " \n", " \n", " [240/240 00:37, Epoch 3/0]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Training loop\n", "trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 2455, "status": "ok", "timestamp": 1719390291303, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "PyRxiY32R3Jd", "outputId": "51d00a55-8086-4456-b3c2-d8720342bcc8" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "***** Running evaluation *****\n" ] }, { "data": { "text/plain": [ "{'f1': 0.8363988383349468}" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Evaluate the model on our test data\n", "trainer.evaluate()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 129 }, "executionInfo": { "elapsed": 10, "status": "ok", "timestamp": 1719390291304, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "-aKIHJpCQdAm", "outputId": "e2f786a4-78a4-4b92-d464-e3336d92edb2" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n", " and should_run_async(code)\n" ] }, { "data": { "text/html": [ "

LogisticRegression()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "LogisticRegression()" ] }, "execution_count": 76, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.model_head" ] }, { "cell_type": "markdown", "metadata": { "id": "L7NbUeSn-QSe" }, "source": [ "## MLM" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1601, "status": "ok", "timestamp": 1719387317479, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "Z35PD47AXXnv", "outputId": "45667c3b-9421-4406-e6e1-a406d803b8fd" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n", "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" ] } ], "source": [ "from transformers import AutoTokenizer, AutoModelForMaskedLM\n", "\n", "# Load model for Masked Language Modeling (MLM)\n", "model = AutoModelForMaskedLM.from_pretrained(\"bert-base-cased\")\n", "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 81, "referenced_widgets": [ "5333a40e5cbc4bb5b89628ecaa6608b2", "c7a67733b49e48df88a34329bd090349", "40f95e431f264d3eb034f09f3f2e9e09", "827d0cdd85c14d13b78da5c0d7f054c7", "e8403bc0d63b4ec0aa679505a962dc34", "bb38873892e847c6b646c9e3cc4c753d", "496cfba4c0d04e399c6fde17772fa62c", "e21c06a4d4ac4f5987b8e4f34c4b9ac8", "c39ac7f5c60e4bf98e64174f21959d9a", "1caf892f43c94d50bb31f12b830462c6", "526eb0a0a0d246578aa2fb726c7385e4", "83160c120ff64c42af6d38bb3fb932b5", "aa3c27c5a1254b19a7cfd04f28aee5d0", "c350f4d70d5646fc973aa9e68711f830", "1ca86de4b7174993942d43b0222f698f", "cac994c5cc274c14ba2163c73fa4ef8c", "4a1c759888cf472aab34c96be48e95b1", "76aeb09a52f04f97aa006927b60bad84", "60f082d7f4234bfa985c9241a7e1e9c0", "8d84f304aca74bd99d7d624b29630495", "71ea5effe952447fa40b578e64e1b868", "933e2bdff863421abba4ba875a817dc0" ] }, "executionInfo": { "elapsed": 3, "status": "ok", "timestamp": 1719387318933, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "zgLardIvEFTG", "outputId": "f40b1e95-9a82-437e-ffd8-5e0fdb4be138" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5333a40e5cbc4bb5b89628ecaa6608b2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map: 0%| | 0/8530 [00:00\n", " \n", " \n", " [5340/5340 12:10, Epoch 10/10]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
5002.601700
10002.377500
15002.313100
20002.187500
25002.150400
30002.096100
35002.059500
40001.990300
45001.986100
50001.958500

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Save pre-trained tokenizer\n", "tokenizer.save_pretrained(\"mlm\")\n", "\n", "# Train model\n", "trainer.train()\n", "\n", "# Save updated model\n", "model.save_pretrained(\"mlm\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 1511, "status": "ok", "timestamp": 1719388054975, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "HfxN1p8TOg2v", "outputId": "e9bcca59-cebb-4297-e675-b3d2a555d36e" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n", " and should_run_async(code)\n", "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n", "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ ">>> What a horrible idea!\n", ">>> What a horrible dream!\n", ">>> What a horrible thing!\n", ">>> What a horrible day!\n", ">>> What a horrible thought!\n" ] } ], "source": [ "from transformers import pipeline\n", "\n", "# Load and create predictions\n", "mask_filler = pipeline(\"fill-mask\", model=\"bert-base-cased\")\n", "preds = mask_filler(\"What a horrible [MASK]!\")\n", "\n", "# Print results\n", "for pred in preds:\n", " print(f\">>> {pred['sequence']}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 3, "status": "ok", "timestamp": 1719388054975, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "ogk1hJ4zOlAU", "outputId": "316b16d0-065b-41e8-c35c-3ce3be481469" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ">>> What a horrible movie!\n", ">>> What a horrible film!\n", ">>> What a horrible mess!\n", ">>> What a horrible comedy!\n", ">>> What a horrible story!\n" ] } ], "source": [ "# Load and create predictions\n", "mask_filler = pipeline(\"fill-mask\", model=\"mlm\")\n", "preds = mask_filler(\"What a horrible [MASK]!\")\n", "\n", "# Print results\n", "for pred in preds:\n", " print(f\">>> {pred['sequence']}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "1sDqoG2NyeJO" }, "source": [ "## Named Entity Recognition\n", "\n", "Here are a number of interesting datasets you can also explore for NER:\n", "* tner/mit_movie_trivia\n", "* tner/mit_restaurant\n", "* wnut_17\n", "* conll2003" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "KGDvXU-ZzJ3J" }, "outputs": [], "source": [ "from transformers import AutoModelForTokenClassification, AutoTokenizer\n", "from transformers import DataCollatorWithPadding\n", "from transformers import TrainingArguments, Trainer\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 298, "referenced_widgets": [ "e8567cb28aff4974acf098b8fc4184ea", "a0960ca186e9491fbff044109a197da2", "1a56e5d0d26e49ad9bd6a919515de403", "2fe1e77e27014f62964c94b62b66f049", "2df3bbada30f48068cf8bd414f40d895", "4457b446302442e4bcab06fc10eb15ae", "23db3d47ec284ebfa6da24d9a58d0881", "a2795e523b5a436d9f94b059e7252597", "027f9ea43b6647ab91915020a2e25771", "15c3f91714a74762989e35950d6c12a5", "19b10fadedbd48288026ff8b21148c8f", "e4c076fd3de24dba84f4d566d4fca48a", "79c5f352cf7f4bf7ace576348e3c55d6", "690b323dd52e48b4b130e63d377b4831", "dbb5c772fc674bedaee90064d8e4163b", "4fe4a9b1182e4383b9ffa8b0b0c0f321", "6d12abf106954c2f9008c8505ed6fce3", "ff5ebeb9078d4f8b975a8aa59977fae3", "befc22d4789e49eb81793e4ce1210f41", "18c3194ae3b8497fb325a424dfb95901", "848c37b4a0224b4e994896ab75d695e4", "bc720c483d74483790aa184e927bd742", "7c9beb946034455e8f4e68cf9d3ed451", "18f06f815052464aa05c788246507c1b", "b58c221705ed490f801fe1cea0a50b60", "c51e7b34c1c049c99753132ddf06a8bf", "ca05b822e80640728b7b98d75fc0a569", "e70e0557dffe4ac4883151f464d88fb3", "caab4f8ab71343dc9d949a4b387ff97d", "114e90e1dc0b49149e94d9352a389352", "95784022cda24c588e9986f5267e3922", "1309c3d2eb8d4bd7b8aa8d7176b225d3", "67d416d3edb747598eefad5de3969330", "ec5ff2eceeaf425bb25a1580b1c86f35", "322b02e7719b4795a0afcbbe99074678", "376dbc09d30742cf9ea209d51b10bb46", "453b539e6093481dabbd4bacc7b746e6", "bb87ce6727e349df960b276b31477d42", "ddf0589272894ee8bc63371aebae6c47", "ce8a5082960345b28883dc1128550675", "69a2c842c8e04a00bccfd9534ead0a3d", "e4b0a87f816c4762b01e6952a4fb91f3", "4956e10177f74c30a4dcec1aee5d84d1", "930cc9ef722e4ad09fc5aec2bdb58f51", "d133524c37da40a688ae41d0be4fb608", "5068f7a72b444b368e3064b523d37316", "20fc8f2be16b48a3a330cb07558c4a7d", "b6871ea8d9ee464ca3d68d62c0264f65", "2ba02b04fbee4db093abeffdc880ce1b", "f4bce89187a74fda9f6b8412a886be56", "2118f618461a4f45b864b8ca51561ecd", "ca9ae83245b3466d961f8741b5b5ca22", "c9aa5057f4c64210a50738ab0ae8d92f", "bbc7a519dee04caca31b77c03b599cc1", "5f33680650354398bd53b210141050ad", "ed3c3a380c3a46e9b6eabe8e0df7f7a1", "a1cfaa8574e34145a95b170954a6af0f", "c94da6e530da46b284bee469474890cf", "9e0395ec942f4b42b7983147701806f8", "4730ed0d0d9a43d0b8b6a46f3351a91e", "6eaad3c842a14b5b86fa908a18d86f20", "9437374ca4e44034a7852cc05a345c18", "31d743d1493746cdae66dff187dee185", "4a2604352c6946e5b6ce02da0e82f00c", "243f45f71e3a4385ac447112e2778915", "20c7a8103eb44b61b2f221cc5ab5527b" ] }, "executionInfo": { "elapsed": 19287, "status": "ok", "timestamp": 1719388740030, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "YOQlMioIGY33", "outputId": "3f5e52d8-e736-411c-e468-d63b48828109" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e8567cb28aff4974acf098b8fc4184ea", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading builder script: 0%| | 0.00/9.57k [00:00\n", " \n", " \n", " [878/878 02:49, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
5000.047500

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "TrainOutput(global_step=878, training_loss=0.04094860494001037, metrics={'train_runtime': 169.4752, 'train_samples_per_second': 82.85, 'train_steps_per_second': 5.181, 'total_flos': 351240792638148.0, 'train_loss': 0.04094860494001037, 'epoch': 1.0})" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Training arguments for parameter tuning\n", "training_args = TrainingArguments(\n", " \"model\",\n", " learning_rate=2e-5,\n", " per_device_train_batch_size=16,\n", " per_device_eval_batch_size=16,\n", " num_train_epochs=1,\n", " weight_decay=0.01,\n", " save_strategy=\"epoch\",\n", " report_to=\"none\"\n", ")\n", "\n", "# Initialize Trainer\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=tokenized[\"train\"],\n", " eval_dataset=tokenized[\"test\"],\n", " tokenizer=tokenizer,\n", " data_collator=data_collator,\n", " compute_metrics=compute_metrics,\n", ")\n", "trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 141 }, "executionInfo": { "elapsed": 14521, "status": "ok", "timestamp": 1712753153056, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "ds5osPr9T0pq", "outputId": "28e6836f-5eae-43c2-a37f-b92acb9a42b6" }, "outputs": [ { "data": { "text/html": [ "\n", "

\n", " \n", " \n", " [216/216 00:09]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'eval_loss': 0.16888542473316193,\n", " 'eval_f1': 0.9180087380808113,\n", " 'eval_runtime': 14.5731,\n", " 'eval_samples_per_second': 236.943,\n", " 'eval_steps_per_second': 14.822,\n", " 'epoch': 1.0}" ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Evaluate the model on our test data\n", "trainer.evaluate()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "executionInfo": { "elapsed": 2674, "status": "ok", "timestamp": 1712753204334, "user": { "displayName": "Maarten Grootendorst", "userId": "11015108362723620659" }, "user_tz": -120 }, "id": "Q0PAXyzT-N45", "outputId": "d29825c0-2be2-48dd-c685-fad114afb369" }, "outputs": [ { "data": { "text/plain": [ "[{'entity': 'B-PER',\n", " 'score': 0.99534035,\n", " 'index': 4,\n", " 'word': 'Ma',\n", " 'start': 11,\n", " 'end': 13},\n", " {'entity': 'I-PER',\n", " 'score': 0.9928328,\n", " 'index': 5,\n", " 'word': '##arte',\n", " 'start': 13,\n", " 'end': 17},\n", " {'entity': 'I-PER',\n", " 'score': 0.9954301,\n", " 'index': 6,\n", " 'word': '##n',\n", " 'start': 17,\n", " 'end': 18}]" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from transformers import pipeline\n", "\n", "# Save our fine-tuned model\n", "trainer.save_model(\"ner_model\")\n", "\n", "# Run inference on the fine-tuned model\n", "token_classifier = pipeline(\n", " \"token-classification\",\n", " model=\"ner_model\",\n", ")\n", "token_classifier(\"My name is Maarten.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "MAfBhqVwC61e" }, "outputs": [], "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "authorship_tag": "ABX9TyOzKODTW4KPEpkvmaAMgqlD", "gpuType": "T4", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 4 }