925 lines
110 KiB
Plaintext
925 lines
110 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"colab_type": "text",
|
|
"id": "jBasof3bv1LB"
|
|
},
|
|
"source": [
|
|
"<h1><center>How to export 🤗 Transformers Models to ONNX ?<h1><center>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[ONNX](http://onnx.ai/) is open format for machine learning models. It allows to save your neural network's computation graph in a framework agnostic way, which might be particulary helpful when deploying deep learning models.\n",
|
|
"\n",
|
|
"Indeed, businesses might have other requirements _(languages, hardware, ...)_ for which the training framework might not be the best suited in inference scenarios. In that context, having a representation of the actual computation graph that can be shared accross various business units and logics across an organization might be a desirable component.\n",
|
|
"\n",
|
|
"Along with the serialization format, ONNX also provides a runtime library which allows efficient and hardware specific execution of the ONNX graph. This is done through the [onnxruntime](https://microsoft.github.io/onnxruntime/) project and already includes collaborations with many hardware vendors to seamlessly deploy models on various platforms.\n",
|
|
"\n",
|
|
"Through this notebook we'll walk you through the process to convert a PyTorch or TensorFlow transformers model to the [ONNX](http://onnx.ai/) and leverage [onnxruntime](https://microsoft.github.io/onnxruntime/) to run inference tasks on models from 🤗 __transformers__"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"colab_type": "text",
|
|
"id": "yNnbrSg-5e1s"
|
|
},
|
|
"source": [
|
|
"## Exporting 🤗 transformers model to ONNX\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"Exporting models _(either PyTorch or TensorFlow)_ is easily achieved through the conversion tool provided as part of 🤗 __transformers__ repository. \n",
|
|
"\n",
|
|
"Under the hood the process is sensibly the following: \n",
|
|
"\n",
|
|
"1. Allocate the model from transformers (**PyTorch or TensorFlow**)\n",
|
|
"2. Forward dummy inputs through the model this way **ONNX** can record the set of operations executed\n",
|
|
"3. Optionally define dynamic axes on input and output tensors\n",
|
|
"4. Save the graph along with the network parameters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Collecting git+https://github.com/huggingface/transformers\n",
|
|
" Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-9rvbp9p8\n",
|
|
" Running command git clone -q https://github.com/huggingface/transformers /tmp/pip-req-build-9rvbp9p8\n",
|
|
"Requirement already satisfied, skipping upgrade: numpy in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (1.18.1)\n",
|
|
"Requirement already satisfied, skipping upgrade: tokenizers==0.8.1.rc2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (0.8.1rc2)\n",
|
|
"Requirement already satisfied, skipping upgrade: packaging in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (20.4)\n",
|
|
"Requirement already satisfied, skipping upgrade: filelock in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (3.0.12)\n",
|
|
"Requirement already satisfied, skipping upgrade: requests in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (2.23.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: tqdm>=4.27 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (4.46.1)\n",
|
|
"Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (2020.6.8)\n",
|
|
"Requirement already satisfied, skipping upgrade: sentencepiece!=0.1.92 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (0.1.91)\n",
|
|
"Requirement already satisfied, skipping upgrade: sacremoses in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers==3.0.2) (0.0.43)\n",
|
|
"Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from packaging->transformers==3.0.2) (2.4.7)\n",
|
|
"Requirement already satisfied, skipping upgrade: six in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from packaging->transformers==3.0.2) (1.15.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers==3.0.2) (3.0.4)\n",
|
|
"Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers==3.0.2) (2.9)\n",
|
|
"Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers==3.0.2) (1.25.9)\n",
|
|
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers==3.0.2) (2020.6.20)\n",
|
|
"Requirement already satisfied, skipping upgrade: click in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from sacremoses->transformers==3.0.2) (7.1.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: joblib in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from sacremoses->transformers==3.0.2) (0.15.1)\n",
|
|
"Building wheels for collected packages: transformers\n",
|
|
" Building wheel for transformers (setup.py) ... \u001b[?25ldone\n",
|
|
"\u001b[?25h Created wheel for transformers: filename=transformers-3.0.2-py3-none-any.whl size=883063 sha256=5f2caef76450921ae2e5b10abbbaab436e9c87c83486114fa08d305e4396d4cd\n",
|
|
" Stored in directory: /tmp/pip-ephem-wheel-cache-kftypcjz/wheels/42/68/45/c63edff61c292f2dfd4df4ef6522dcbecc603e7af82813c1d7\n",
|
|
"Successfully built transformers\n",
|
|
"Installing collected packages: transformers\n",
|
|
" Attempting uninstall: transformers\n",
|
|
" Found existing installation: transformers 3.0.2\n",
|
|
" Uninstalling transformers-3.0.2:\n",
|
|
" Successfully uninstalled transformers-3.0.2\n",
|
|
"Successfully installed transformers-3.0.2\n",
|
|
"Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
|
|
"Requirement already up-to-date: torch==1.6.0+cpu in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.6.0+cpu)\n",
|
|
"Requirement already up-to-date: torchvision==0.7.0+cpu in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (0.7.0+cpu)\n",
|
|
"Requirement already satisfied, skipping upgrade: numpy in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from torch==1.6.0+cpu) (1.18.1)\n",
|
|
"Requirement already satisfied, skipping upgrade: future in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from torch==1.6.0+cpu) (0.18.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: pillow>=4.1.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from torchvision==0.7.0+cpu) (7.2.0)\n",
|
|
"Requirement already up-to-date: onnxruntime==1.4.0 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.4.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: protobuf in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime==1.4.0) (3.12.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: numpy>=1.16.6 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime==1.4.0) (1.18.1)\n",
|
|
"Requirement already satisfied, skipping upgrade: setuptools in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->onnxruntime==1.4.0) (47.1.1.post20200604)\n",
|
|
"Requirement already satisfied, skipping upgrade: six>=1.9 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->onnxruntime==1.4.0) (1.15.0)\n",
|
|
"Looking in indexes: https://test.pypi.org/simple/\n",
|
|
"Requirement already satisfied: ort-nightly in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.4.0.dev202008262)\n",
|
|
"Requirement already satisfied: protobuf in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from ort-nightly) (3.12.2)\n",
|
|
"Requirement already satisfied: numpy>=1.16.6 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from ort-nightly) (1.18.1)\n",
|
|
"Requirement already satisfied: setuptools in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->ort-nightly) (47.1.1.post20200604)\n",
|
|
"Requirement already satisfied: six>=1.9 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->ort-nightly) (1.15.0)\n",
|
|
"Requirement already up-to-date: onnxruntime-tools in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.4.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: numpy in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (1.18.1)\n",
|
|
"Requirement already satisfied, skipping upgrade: coloredlogs in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (14.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: py3nvml in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (0.2.6)\n",
|
|
"Requirement already satisfied, skipping upgrade: psutil in /home/mfuntowicz/.local/lib/python3.8/site-packages/psutil-5.7.0-py3.8-linux-x86_64.egg (from onnxruntime-tools) (5.7.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: packaging in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (20.4)\n",
|
|
"Requirement already satisfied, skipping upgrade: py-cpuinfo in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (5.0.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: onnx in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-tools) (1.7.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: humanfriendly>=7.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from coloredlogs->onnxruntime-tools) (8.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: xmltodict in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from py3nvml->onnxruntime-tools) (0.12.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: six in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from packaging->onnxruntime-tools) (1.15.0)\n",
|
|
"Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from packaging->onnxruntime-tools) (2.4.7)\n",
|
|
"Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.2.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnx->onnxruntime-tools) (3.7.4.2)\n",
|
|
"Requirement already satisfied, skipping upgrade: protobuf in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnx->onnxruntime-tools) (3.12.2)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Requirement already satisfied, skipping upgrade: setuptools in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->onnx->onnxruntime-tools) (47.1.1.post20200604)\r\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import sys\n",
|
|
"!{sys.executable} -m pip install --upgrade git+https://github.com/huggingface/transformers\n",
|
|
"!{sys.executable} -m pip install --upgrade torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html\n",
|
|
"!{sys.executable} -m pip install --upgrade onnxruntime==1.4.0\n",
|
|
"!{sys.executable} -m pip install -i https://test.pypi.org/simple/ ort-nightly\n",
|
|
"!{sys.executable} -m pip install --upgrade onnxruntime-tools"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"metadata": {
|
|
"colab": {},
|
|
"colab_type": "code",
|
|
"id": "PwAaOchY4N2-"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /home/mfuntowicz/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.9da767be51e1327499df13488672789394e2ca38b877837e52618a67d7002391\n",
|
|
"Model config BertConfig {\n",
|
|
" \"architectures\": [\n",
|
|
" \"BertForMaskedLM\"\n",
|
|
" ],\n",
|
|
" \"attention_probs_dropout_prob\": 0.1,\n",
|
|
" \"gradient_checkpointing\": false,\n",
|
|
" \"hidden_act\": \"gelu\",\n",
|
|
" \"hidden_dropout_prob\": 0.1,\n",
|
|
" \"hidden_size\": 768,\n",
|
|
" \"initializer_range\": 0.02,\n",
|
|
" \"intermediate_size\": 3072,\n",
|
|
" \"layer_norm_eps\": 1e-12,\n",
|
|
" \"max_position_embeddings\": 512,\n",
|
|
" \"model_type\": \"bert\",\n",
|
|
" \"num_attention_heads\": 12,\n",
|
|
" \"num_hidden_layers\": 12,\n",
|
|
" \"pad_token_id\": 0,\n",
|
|
" \"type_vocab_size\": 2,\n",
|
|
" \"vocab_size\": 28996\n",
|
|
"}\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"ONNX opset version set to: 11\n",
|
|
"Loading pipeline (model: bert-base-cased, tokenizer: bert-base-cased)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /home/mfuntowicz/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1\n",
|
|
"loading model card file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-modelcard.json from cache at /home/mfuntowicz/.cache/torch/transformers/72b46f187c40a666d54782e06684c2870e109350a3efe9aa5027253dec2e671d.455d944f3d1572ab55ed579849f751cf37f303e3388980a42d94f7cd57a4e331\n",
|
|
"Model card: {\n",
|
|
" \"caveats_and_recommendations\": {},\n",
|
|
" \"ethical_considerations\": {},\n",
|
|
" \"evaluation_data\": {},\n",
|
|
" \"factors\": {},\n",
|
|
" \"intended_use\": {},\n",
|
|
" \"metrics\": {},\n",
|
|
" \"model_details\": {},\n",
|
|
" \"quantitative_analyses\": {},\n",
|
|
" \"training_data\": {}\n",
|
|
"}\n",
|
|
"\n",
|
|
"loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /home/mfuntowicz/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.9da767be51e1327499df13488672789394e2ca38b877837e52618a67d7002391\n",
|
|
"Model config BertConfig {\n",
|
|
" \"architectures\": [\n",
|
|
" \"BertForMaskedLM\"\n",
|
|
" ],\n",
|
|
" \"attention_probs_dropout_prob\": 0.1,\n",
|
|
" \"gradient_checkpointing\": false,\n",
|
|
" \"hidden_act\": \"gelu\",\n",
|
|
" \"hidden_dropout_prob\": 0.1,\n",
|
|
" \"hidden_size\": 768,\n",
|
|
" \"initializer_range\": 0.02,\n",
|
|
" \"intermediate_size\": 3072,\n",
|
|
" \"layer_norm_eps\": 1e-12,\n",
|
|
" \"max_position_embeddings\": 512,\n",
|
|
" \"model_type\": \"bert\",\n",
|
|
" \"num_attention_heads\": 12,\n",
|
|
" \"num_hidden_layers\": 12,\n",
|
|
" \"pad_token_id\": 0,\n",
|
|
" \"type_vocab_size\": 2,\n",
|
|
" \"vocab_size\": 28996\n",
|
|
"}\n",
|
|
"\n",
|
|
"loading weights file https://cdn.huggingface.co/bert-base-cased-pytorch_model.bin from cache at /home/mfuntowicz/.cache/torch/transformers/d8f11f061e407be64c4d5d7867ee61d1465263e24085cfa26abf183fdc830569.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2\n",
|
|
"All model checkpoint weights were used when initializing BertModel.\n",
|
|
"\n",
|
|
"All the weights of BertModel were initialized from the model checkpoint at bert-base-cased.\n",
|
|
"If your task is similar to the task the model of the checkpoint was trained on, you can already use BertModel for predictions without further training.\n",
|
|
"/home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages/transformers/modeling_bert.py:201: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
|
|
" position_ids = self.position_ids[:, :seq_length]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Creating folder onnx\n",
|
|
"Using framework PyTorch: 1.6.0\n",
|
|
"Found input input_ids with shape: {0: 'batch', 1: 'sequence'}\n",
|
|
"Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}\n",
|
|
"Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}\n",
|
|
"Found output output_0 with shape: {0: 'batch', 1: 'sequence'}\n",
|
|
"Found output output_1 with shape: {0: 'batch'}\n",
|
|
"Ensuring inputs are in correct order\n",
|
|
"position_ids is not present in the generated input list.\n",
|
|
"Generated inputs order: ['input_ids', 'attention_mask', 'token_type_ids']\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages/transformers/modeling_utils.py:1570: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
|
|
" input_tensor.shape == tensor_shape for input_tensor in input_tensors\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"!rm -rf onnx/\n",
|
|
"from pathlib import Path\n",
|
|
"from transformers.convert_graph_to_onnx import convert\n",
|
|
"\n",
|
|
"# Handles all the above steps for you\n",
|
|
"convert(framework=\"pt\", model=\"bert-base-cased\", output=Path(\"onnx/bert-base-cased.onnx\"), opset=11)\n",
|
|
"\n",
|
|
"# Tensorflow \n",
|
|
"# convert(framework=\"tf\", model=\"bert-base-cased\", output=\"onnx/bert-base-cased.onnx\", opset=11)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## How to leverage runtime for inference over an ONNX graph\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"As mentionned in the introduction, **ONNX** is a serialization format and many side projects can load the saved graph and run the actual computations from it. Here, we'll focus on the official [onnxruntime](https://microsoft.github.io/onnxruntime/). The runtime is implemented in C++ for performance reasons and provides API/Bindings for C++, C, C#, Java and Python.\n",
|
|
"\n",
|
|
"In the case of this notebook, we will use the Python API to highlight how to load a serialized **ONNX** graph and run inference workload on various backends through **onnxruntime**.\n",
|
|
"\n",
|
|
"**onnxruntime** is available on pypi:\n",
|
|
"\n",
|
|
"- onnxruntime: ONNX + MLAS (Microsoft Linear Algebra Subprograms)\n",
|
|
"- onnxruntime-gpu: ONNX + MLAS + CUDA\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"metadata": {
|
|
"pycharm": {
|
|
"name": "#%%\n"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Requirement already satisfied: transformers in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (3.0.2)\n",
|
|
"Requirement already satisfied: onnxruntime-gpu in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.3.0)\n",
|
|
"Requirement already satisfied: onnx in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (1.7.0)\n",
|
|
"Requirement already satisfied: psutil in /home/mfuntowicz/.local/lib/python3.8/site-packages/psutil-5.7.0-py3.8-linux-x86_64.egg (5.7.0)\n",
|
|
"Requirement already satisfied: matplotlib in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (3.3.1)\n",
|
|
"Requirement already satisfied: tqdm>=4.27 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (4.46.1)\n",
|
|
"Requirement already satisfied: numpy in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (1.18.1)\n",
|
|
"Requirement already satisfied: sacremoses in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (0.0.43)\n",
|
|
"Requirement already satisfied: regex!=2019.12.17 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (2020.6.8)\n",
|
|
"Requirement already satisfied: filelock in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (3.0.12)\n",
|
|
"Requirement already satisfied: sentencepiece!=0.1.92 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (0.1.91)\n",
|
|
"Requirement already satisfied: requests in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (2.23.0)\n",
|
|
"Requirement already satisfied: packaging in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (20.4)\n",
|
|
"Requirement already satisfied: tokenizers==0.8.1.rc2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from transformers) (0.8.1rc2)\n",
|
|
"Requirement already satisfied: protobuf in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnxruntime-gpu) (3.12.2)\n",
|
|
"Requirement already satisfied: six in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnx) (1.15.0)\n",
|
|
"Requirement already satisfied: typing-extensions>=3.6.2.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from onnx) (3.7.4.2)\n",
|
|
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (2.4.7)\n",
|
|
"Requirement already satisfied: kiwisolver>=1.0.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (1.2.0)\n",
|
|
"Requirement already satisfied: python-dateutil>=2.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (2.8.1)\n",
|
|
"Requirement already satisfied: cycler>=0.10 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (0.10.0)\n",
|
|
"Requirement already satisfied: pillow>=6.2.0 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (7.2.0)\n",
|
|
"Requirement already satisfied: certifi>=2020.06.20 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from matplotlib) (2020.6.20)\n",
|
|
"Requirement already satisfied: click in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from sacremoses->transformers) (7.1.2)\n",
|
|
"Requirement already satisfied: joblib in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from sacremoses->transformers) (0.15.1)\n",
|
|
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers) (1.25.9)\n",
|
|
"Requirement already satisfied: chardet<4,>=3.0.2 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers) (3.0.4)\n",
|
|
"Requirement already satisfied: idna<3,>=2.5 in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests->transformers) (2.9)\n",
|
|
"Requirement already satisfied: setuptools in /home/mfuntowicz/miniconda3/envs/pytorch/lib/python3.8/site-packages (from protobuf->onnxruntime-gpu) (47.1.1.post20200604)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"!pip install transformers onnxruntime-gpu onnx psutil matplotlib"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"colab_type": "text",
|
|
"id": "-gP08tHfBvgY"
|
|
},
|
|
"source": [
|
|
"## Preparing for an Inference Session\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"Inference is done using a specific backend definition which turns on hardware specific optimizations of the graph. \n",
|
|
"\n",
|
|
"Optimizations are basically of three kinds: \n",
|
|
"\n",
|
|
"- **Constant Folding**: Convert static variables to constants in the graph \n",
|
|
"- **Deadcode Elimination**: Remove nodes never accessed in the graph\n",
|
|
"- **Operator Fusing**: Merge multiple instruction into one (Linear -> ReLU can be fused to be LinearReLU)\n",
|
|
"\n",
|
|
"ONNX Runtime automatically applies most optimizations by setting specific `SessionOptions`.\n",
|
|
"\n",
|
|
"Note:Some of the latest optimizations that are not yet integrated into ONNX Runtime are available in [optimization script](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers) that tunes models for the best performance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"metadata": {
|
|
"pycharm": {
|
|
"name": "#%%\n"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# # An optional step unless\n",
|
|
"# # you want to get a model with mixed precision for perf accelartion on newer GPU\n",
|
|
"# # or you are working with Tensorflow(tf.keras) models or pytorch models other than bert\n",
|
|
"\n",
|
|
"# !pip install onnxruntime-tools\n",
|
|
"# from onnxruntime_tools import optimizer\n",
|
|
"\n",
|
|
"# # Mixed precision conversion for bert-base-cased model converted from Pytorch\n",
|
|
"# optimized_model = optimizer.optimize_model(\"bert-base-cased.onnx\", model_type='bert', num_heads=12, hidden_size=768)\n",
|
|
"# optimized_model.convert_model_float32_to_float16()\n",
|
|
"# optimized_model.save_model_to_file(\"bert-base-cased.onnx\")\n",
|
|
"\n",
|
|
"# # optimizations for bert-base-cased model converted from Tensorflow(tf.keras)\n",
|
|
"# optimized_model = optimizer.optimize_model(\"bert-base-cased.onnx\", model_type='bert_keras', num_heads=12, hidden_size=768)\n",
|
|
"# optimized_model.save_model_to_file(\"bert-base-cased.onnx\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# optimize transformer-based models with onnxruntime-tools\n",
|
|
"from onnxruntime_tools import optimizer\n",
|
|
"from onnxruntime_tools.transformers.onnx_model_bert import BertOptimizationOptions\n",
|
|
"\n",
|
|
"# disable embedding layer norm optimization for better model size reduction\n",
|
|
"opt_options = BertOptimizationOptions('bert')\n",
|
|
"opt_options.enable_embed_layer_norm = False\n",
|
|
"\n",
|
|
"opt_model = optimizer.optimize_model(\n",
|
|
" 'onnx/bert-base-cased.onnx',\n",
|
|
" 'bert', \n",
|
|
" num_heads=12,\n",
|
|
" hidden_size=768,\n",
|
|
" optimization_options=opt_options)\n",
|
|
"opt_model.save_model_to_file('bert.opt.onnx')\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"metadata": {
|
|
"pycharm": {
|
|
"name": "#%%\n"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from os import environ\n",
|
|
"from psutil import cpu_count\n",
|
|
"\n",
|
|
"# Constants from the performance optimization available in onnxruntime\n",
|
|
"# It needs to be done before importing onnxruntime\n",
|
|
"environ[\"OMP_NUM_THREADS\"] = str(cpu_count(logical=True))\n",
|
|
"environ[\"OMP_WAIT_POLICY\"] = 'ACTIVE'\n",
|
|
"\n",
|
|
"from onnxruntime import GraphOptimizationLevel, InferenceSession, SessionOptions, get_all_providers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"metadata": {
|
|
"colab": {},
|
|
"colab_type": "code",
|
|
"id": "2k-jHLfdcTFS"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from contextlib import contextmanager\n",
|
|
"from dataclasses import dataclass\n",
|
|
"from time import time\n",
|
|
"from tqdm import trange\n",
|
|
"\n",
|
|
"def create_model_for_provider(model_path: str, provider: str) -> InferenceSession: \n",
|
|
" \n",
|
|
" assert provider in get_all_providers(), f\"provider {provider} not found, {get_all_providers()}\"\n",
|
|
"\n",
|
|
" # Few properties that might have an impact on performances (provided by MS)\n",
|
|
" options = SessionOptions()\n",
|
|
" options.intra_op_num_threads = 1\n",
|
|
" options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL\n",
|
|
"\n",
|
|
" # Load the model as a graph and prepare the CPU backend \n",
|
|
" session = InferenceSession(model_path, options, providers=[provider])\n",
|
|
" session.disable_fallback()\n",
|
|
" \n",
|
|
" return session\n",
|
|
"\n",
|
|
"\n",
|
|
"@contextmanager\n",
|
|
"def track_infer_time(buffer: [int]):\n",
|
|
" start = time()\n",
|
|
" yield\n",
|
|
" end = time()\n",
|
|
"\n",
|
|
" buffer.append(end - start)\n",
|
|
"\n",
|
|
"\n",
|
|
"@dataclass\n",
|
|
"class OnnxInferenceResult:\n",
|
|
" model_inference_time: [int] \n",
|
|
" optimized_model_path: str"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"colab_type": "text",
|
|
"id": "teJdG3amE-hR"
|
|
},
|
|
"source": [
|
|
"## Forwarding through our optimized ONNX model running on CPU\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"When the model is loaded for inference over a specific provider, for instance **CPUExecutionProvider** as above, an optimized graph can be saved. This graph will might include various optimizations, and you might be able to see some **higher-level** operations in the graph _(through [Netron](https://github.com/lutzroeder/Netron) for instance)_ such as:\n",
|
|
"- **EmbedLayerNormalization**\n",
|
|
"- **Attention**\n",
|
|
"- **FastGeLU**\n",
|
|
"\n",
|
|
"These operations are an example of the kind of optimization **onnxruntime** is doing, for instance here gathering multiple operations into bigger one _(Operator Fusing)_."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 34
|
|
},
|
|
"colab_type": "code",
|
|
"id": "dmC22kJfVGYe",
|
|
"outputId": "f3aba5dc-15c0-4f82-b38c-1bbae1bf112e"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /home/mfuntowicz/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Sequence output: (1, 6, 768), Pooled output: (1, 768)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers import BertTokenizerFast\n",
|
|
"\n",
|
|
"tokenizer = BertTokenizerFast.from_pretrained(\"bert-base-cased\")\n",
|
|
"cpu_model = create_model_for_provider(\"onnx/bert-base-cased.onnx\", \"CPUExecutionProvider\")\n",
|
|
"\n",
|
|
"# Inputs are provided through numpy array\n",
|
|
"model_inputs = tokenizer(\"My name is Bert\", return_tensors=\"pt\")\n",
|
|
"inputs_onnx = {k: v.cpu().detach().numpy() for k, v in model_inputs.items()}\n",
|
|
"\n",
|
|
"# Run the model (None = get all the outputs)\n",
|
|
"sequence, pooled = cpu_model.run(None, inputs_onnx)\n",
|
|
"\n",
|
|
"# Print information about outputs\n",
|
|
"\n",
|
|
"print(f\"Sequence output: {sequence.shape}, Pooled output: {pooled.shape}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Benchmarking PyTorch model\n",
|
|
"\n",
|
|
"_Note: PyTorch model benchmark is run on CPU_"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 51
|
|
},
|
|
"colab_type": "code",
|
|
"id": "PS_49goe197g",
|
|
"outputId": "0ef0f70c-f5a7-46a0-949a-1a93f231d193"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json from cache at /home/mfuntowicz/.cache/torch/transformers/b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.9da767be51e1327499df13488672789394e2ca38b877837e52618a67d7002391\n",
|
|
"Model config BertConfig {\n",
|
|
" \"architectures\": [\n",
|
|
" \"BertForMaskedLM\"\n",
|
|
" ],\n",
|
|
" \"attention_probs_dropout_prob\": 0.1,\n",
|
|
" \"gradient_checkpointing\": false,\n",
|
|
" \"hidden_act\": \"gelu\",\n",
|
|
" \"hidden_dropout_prob\": 0.1,\n",
|
|
" \"hidden_size\": 768,\n",
|
|
" \"initializer_range\": 0.02,\n",
|
|
" \"intermediate_size\": 3072,\n",
|
|
" \"layer_norm_eps\": 1e-12,\n",
|
|
" \"max_position_embeddings\": 512,\n",
|
|
" \"model_type\": \"bert\",\n",
|
|
" \"num_attention_heads\": 12,\n",
|
|
" \"num_hidden_layers\": 12,\n",
|
|
" \"pad_token_id\": 0,\n",
|
|
" \"type_vocab_size\": 2,\n",
|
|
" \"vocab_size\": 28996\n",
|
|
"}\n",
|
|
"\n",
|
|
"loading weights file https://cdn.huggingface.co/bert-base-cased-pytorch_model.bin from cache at /home/mfuntowicz/.cache/torch/transformers/d8f11f061e407be64c4d5d7867ee61d1465263e24085cfa26abf183fdc830569.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2\n",
|
|
"All model checkpoint weights were used when initializing BertModel.\n",
|
|
"\n",
|
|
"All the weights of BertModel were initialized from the model checkpoint at bert-base-cased.\n",
|
|
"If your task is similar to the task the model of the checkpoint was trained on, you can already use BertModel for predictions without further training.\n",
|
|
"Warming up: 100%|██████████| 10/10 [00:00<00:00, 39.30it/s]\n",
|
|
"Tracking inference time on PyTorch: 100%|██████████| 100/100 [00:02<00:00, 41.09it/s]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers import BertModel\n",
|
|
"\n",
|
|
"PROVIDERS = {\n",
|
|
" (\"cpu\", \"PyTorch CPU\"),\n",
|
|
"# Uncomment this line to enable GPU benchmarking\n",
|
|
"# (\"cuda:0\", \"PyTorch GPU\")\n",
|
|
"}\n",
|
|
"\n",
|
|
"results = {}\n",
|
|
"\n",
|
|
"for device, label in PROVIDERS:\n",
|
|
" \n",
|
|
" # Move inputs to the correct device\n",
|
|
" model_inputs_on_device = {\n",
|
|
" arg_name: tensor.to(device)\n",
|
|
" for arg_name, tensor in model_inputs.items()\n",
|
|
" }\n",
|
|
"\n",
|
|
" # Add PyTorch to the providers\n",
|
|
" model_pt = BertModel.from_pretrained(\"bert-base-cased\").to(device)\n",
|
|
" for _ in trange(10, desc=\"Warming up\"):\n",
|
|
" model_pt(**model_inputs_on_device)\n",
|
|
"\n",
|
|
" # Compute \n",
|
|
" time_buffer = []\n",
|
|
" for _ in trange(100, desc=f\"Tracking inference time on PyTorch\"):\n",
|
|
" with track_infer_time(time_buffer):\n",
|
|
" model_pt(**model_inputs_on_device)\n",
|
|
"\n",
|
|
" # Store the result\n",
|
|
" results[label] = OnnxInferenceResult(\n",
|
|
" time_buffer, \n",
|
|
" None\n",
|
|
" ) "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"colab_type": "text",
|
|
"id": "Kda1e7TkEqNR"
|
|
},
|
|
"source": [
|
|
"## Benchmarking PyTorch & ONNX on CPU\n",
|
|
"\n",
|
|
"_**Disclamer: results may vary from the actual hardware used to run the model**_"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 170
|
|
},
|
|
"colab_type": "code",
|
|
"id": "WcdFZCvImVig",
|
|
"outputId": "bfd779a1-0bc7-42db-8587-e52a485ec5e3"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Tracking inference time on CPUExecutionProvider: 100%|██████████| 100/100 [00:01<00:00, 63.62it/s]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"PROVIDERS = {\n",
|
|
" (\"CPUExecutionProvider\", \"ONNX CPU\"),\n",
|
|
"# Uncomment this line to enable GPU benchmarking\n",
|
|
"# (\"CUDAExecutionProvider\", \"ONNX GPU\")\n",
|
|
"}\n",
|
|
"\n",
|
|
"\n",
|
|
"for provider, label in PROVIDERS:\n",
|
|
" # Create the model with the specified provider\n",
|
|
" model = create_model_for_provider(\"onnx/bert-base-cased.onnx\", provider)\n",
|
|
"\n",
|
|
" # Keep track of the inference time\n",
|
|
" time_buffer = []\n",
|
|
"\n",
|
|
" # Warm up the model\n",
|
|
" model.run(None, inputs_onnx)\n",
|
|
"\n",
|
|
" # Compute \n",
|
|
" for _ in trange(100, desc=f\"Tracking inference time on {provider}\"):\n",
|
|
" with track_infer_time(time_buffer):\n",
|
|
" model.run(None, inputs_onnx)\n",
|
|
"\n",
|
|
" # Store the result\n",
|
|
" results[label] = OnnxInferenceResult(\n",
|
|
" time_buffer,\n",
|
|
" model.get_session_options().optimized_model_filepath\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 31,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABRoAAAPeCAYAAABjjKazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAA9hAAAPYQGoP6dpAABezklEQVR4nOzdeZjd8/3//8fJNklkEyIJQmxFbPWh1dDGFiL2rYQuUVsX1Fof+WgtpVJUpVW7fkI1GruqfuxLKtaKWtuqqBBrbEkkYZA5vz/6zfyMLDJeM2aG2+26znXlvM/7nPfznFmucfdeKtVqtRoAAAAAgALtWnoAAAAAAKDtExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAYA2ZeDAgdlnn30+8fNPP/30rLzyymnfvn2++MUvNtlcnyV33XVXKpVK7rrrrpYepYFZs2ZlmWWWybhx4z7V7R5zzDHZaKONGvWcz/r32bzvkauuuqqlR2kyF198cSqVSqZMmfKx65b+HgKAzyqhEQBaoXPOOSeVSqXRcYNFu+WWW3L00Udnk002ydixY3PKKae09Egt6pxzzsnFF1/c0mMstl/96lfp3r17RowY8alu97DDDsujjz6a66+/frHW930GAHxedWjpAQCA+Y0bNy4DBw7Mgw8+mMmTJ2fVVVdt6ZFajaeeeirt2n2y/1d6xx13pF27dvntb3+bTp06NfFkbc8555yTpZdeer49s4YMGZJ33nmnVX1G77//fn71q1/l8MMPT/v27T/Vbffr1y877bRTfvGLX2THHXf82PV9n7VN3/rWtzJixIjU1NS09CgA0GbZoxEAWplnn3029957b375y1+mT58+n/phoklSV1eXd99991Pf7uKoqalJx44dP9Fzp02bli5dujRp/JkzZ06TvVZr0a5du3Tu3PkTB93mcMMNN+S1117LHnvs0SLb32OPPTJx4sT8+9///th1m/r7rFqt5p133mmS1/osaK7fT+3bt0/nzp1TqVSa/LUX5YMPPsh77733qW4TAJpL6/nrEQBI8p+9GZdccslst9122X333RuExvfffz+9e/fOd77znfmeN3PmzHTu3DlHHXVU/bLa2tocf/zxWXXVVVNTU5MBAwbk6KOPTm1tbYPnViqVHHzwwRk3blzWWmut1NTU5KabbkqS/OIXv8jGG2+cpZZaKl26dMkGG2ywwPOyvfPOO/nhD3+YpZdeOt27d8+OO+6YF198MZVKJSeccEKDdV988cXsu+++6du3b2pqarLWWmvlf//3fxfr8/noudHmnVftnnvuyRFHHJE+ffpkiSWWyC677JLXXnutwXscO3ZsZs+enUqlkkql0uCw4d///vfZYIMN0qVLl/Tu3TsjRozI1KlTG2x7s802y9prr51JkyZlyJAh6dq1a/7nf/7nE33W1113XdZee+369z/v8/7o57Tffvtl2WWXTU1NTVZaaaV8//vfbxAlpk+fnsMOOywDBgxITU1NVl111Zx66qmpq6v72M/xySefzIQJE+o/j8022yzJgs/ROO+9P/bYY9l0003TtWvXrLrqqvXfCxMmTMhGG22ULl26ZPXVV89tt922wPfzSb/u1113XQYOHJhVVlmlwfJ99tkn3bp1y/PPP5/tt98+3bp1y3LLLZezzz47SfL4449niy22yBJLLJEVV1wxl112WYPnv//++znxxBOz2mqrpXPnzllqqaXy1a9+NbfeemuD9YYOHZok+eMf/7jIORf1ffbBBx/kpJNOyiqrrJKampoMHDgw//M//zPf98jAgQOz/fbb5+abb86GG26YLl265Pzzz1/kdh944IFss8026dmzZ7p27ZpNN90099xzT4N1nnvuufzgBz/I6quvni5dumSppZbK17/+9QWek3D69Ok5/PDDM3DgwNTU1GT55ZfPt7/97bz++usN1qurq8vPfvazLL/88uncuXO23HLLTJ48eZGzJskJJ5yQSqWSf/7zn9ljjz3So0ePLLXUUjn00EPni4iL+v30t7/9LcOHD0+PHj3SrVu3bLnllrn//vvrn/vQQw+lUqnkkksumW+Gm2++OZVKJTfccEOSBZ+jsVqt5uSTT87yyy+frl27ZvPNN8+TTz65wPe0OD+LU6ZMSaVSyS9+8YuMGTOm/nvh73//+8d+ZgDQFjh0GgBamXHjxmXXXXdNp06dstdee+Xcc8/NX//613zpS19Kx44ds8suu+Saa67J+eef32CPqeuuuy61tbX156+rq6vLjjvumIkTJ+bAAw/Mmmuumccffzxnnnlm/vWvf+W6665rsN077rgjV1xxRQ4++OAsvfTSGThwYJL/nBdvxx13zDe+8Y289957GT9+fL7+9a/nhhtuyHbbbVf//H322SdXXHFFvvWtb+UrX/lKJkyY0ODxeV599dV85StfqY8Hffr0yY033pj99tsvM2fOzGGHHfaJPrdDDjkkSy65ZI4//vhMmTIlY8aMycEHH5zLL788SXLppZfmggsuyIMPPpiLLrooSbLxxhsnSX72s5/lJz/5SfbYY4/sv//+ee2113LWWWdlyJAh+dvf/pZevXrVb+eNN97I8OHDM2LEiHzzm99M3759G/1ZT5w4Mddcc01+8IMfpHv37vn1r3+d3XbbLc8//3yWWmqpJMlLL72UL3/5y5k+fXoOPPDArLHGGnnxxRdz1VVXZc6cOenUqVPmzJmTTTfdNC+++GK++93vZoUVVsi9996bUaNG5eWXX86YMWMW+nmNGTMmhxxySLp165Zjjz02SdK3b99FfsZvvfVWtt9++4wYMSJf//rXc+6552bEiBEZN25cDjvssHzve9/L3nvvndNPPz277757pk6dmu7duycp/7rfe++9+a//+q8FPjZ37twMHz48Q4YMyWmnnZZx48bl4IMPzhJLLJFjjz023/jGN7LrrrvmvPPOy7e//e0MHjw4K620UpL/BK/Ro0dn//33z5e//OXMnDkzDz30UB5++OFstdVW9dvo2bNnVlllldxzzz05/PDDFzrnor7P9t9//1xyySXZfffdc+SRR+aBBx7I6NGj849//CPXXnttg9d56qmnstdee+W73/1uDjjggKy++uoL3eYdd9yR4cOHZ4MNNsjxxx+fdu3aZezYsdliiy1y991358tf/nKS5K9//WvuvffejBgxIssvv3ymTJmSc889N5tttln+/ve/p2vXrkn+c9Gdr33ta/nHP/6RfffdN//1X/+V119/Pddff31eeOGFLL300vXb/vnPf5527drlqKOOyowZM3LaaaflG9/4Rh544IGFzvthe+yxRwYOHJjRo0fn/vvvz69//eu89dZb+d3vfjffe/zo76cnn3wyX/va19KjR48cffTR6dixY84///xsttlm9eF7ww03zMorr5wrrrgiI0eObPCal19+eZZccskMGzZsofMdd9xxOfnkk7Pttttm2223zcMPP5ytt956vj0QG/uzOHbs2Lz77rs58MADU1NTk969ey/W5wUArV4VAGg1HnrooWqS6q233lqtVqvVurq66vLLL1899NBD69e5+eabq0mqf/rTnxo8d9ttt62uvPLK9fcvvfTSart27ap33313g/XOO++8apLqPffcU78sSbVdu3bVJ598cr6Z5syZ0+D+e++9V1177bWrW2yxRf2ySZMmVZNUDzvssAbr7rPPPtUk1eOPP75+2X777Vft379/9fXXX2+w7ogRI6o9e/acb3sfteKKK1ZHjhxZf3/s2LHVJNWhQ4dW6+rq6pcffvjh1fbt21enT59ev2zkyJHVJZZYosHrTZkypdq+ffvqz372swbLH3/88WqHDh0aLN90002rSarnnXdeg3Ub+1l36tSpOnny5Ppljz76aDVJ9ayzzqpf9u1vf7varl276l//+tf5PoN57/Okk06qLrHEEtV//etfDR4/5phjqu3bt68+//zz8z33w9Zaa63qpptuOt/yO++8s5qkeuedd8733i+77LL6Zf/85z/rv3fuv//++uXzvkfHjh1bv6zk6/7+++9XK5VK9cgjj5zvsZEjR1aTVE855ZT6ZW+99Va1S5cu1UqlUh0/fvx88374+3G99darbrfddgvd9odtvfXW1TXXXPNj11vQ99kjjzxSTVLdf//9Gyw/6qijqkmqd9xxR/2yFVdcsZqketNNN33sturq6qqrrbZaddiwYQ2+/+fMmVNdaaWVqltttVWDZR913333VZNUf/e739UvO+6446pJqtdcc80Ct1et/v/fI2uuuWa1tra2/vFf/epX1STVxx9/fJFzH3/88dUk1R133LHB8h/84AfVJNVHH320ftnCfj/tvPPO1U6dOlWfeeaZ+mUvvfRStXv37tUhQ4bULxs1alS1Y8eO1TfffLN+WW1tbbVXr17Vfffdt37ZvN8lzz77bLVarVanTZtW7dSpU3W77bZr8Nn+z//8TzVJg99Di/uz+Oyzz1aTVHv06FGdNm3aIj8jAGiLHDoNAK3IuHHj0rdv32y++eZJ/nPI4J577pnx48dn7ty5SZItttgiSy+9dP2eesl/9jS79dZbs+eee9Yvu/LKK7PmmmtmjTXWyOuvv15/22KLLZIkd955Z4Ntb7rpphk0aNB8M3Xp0qXBdmbMmJGvfe1refjhh+uXzzuM8Qc/+EGD5x5yyCEN7ler1Vx99dXZYYcdUq1WG8w1bNiwzJgxo8HrNsaBBx7Y4NxqX/va1zJ37tw899xzi3zeNddck7q6uuyxxx4N5unXr19WW221+T6nmpqa+Q5db+xnPXTo0AaHAK+77rrp0aNH/fn/6urqct1112WHHXbIhhtuON/M897nlVdema997WtZcsklG2x36NChmTt3bv7yl7983MfWKN26dWtwxefVV189vXr1ypprrtngCunz/j3v/ZR+3d98881Uq9UsueSSC11n//33r/93r169svrqq2eJJZZocE7HefN++DyLvXr1ypNPPpmnn376Y9//vM/5k/i///u/JMkRRxzRYPmRRx6ZJPnzn//cYPlKK620yD3t5nnkkUfy9NNPZ++9984bb7xR/7nOnj07W265Zf7yl7/UH7r74Z/l999/P2+88UZWXXXV9OrVq8Hnf/XVV2e99dbLLrvsMt/2Pnr+wu985zsN9qz+2te+liSLdS7LJDnooIMa3J/3O2Pe5zXPR38/zZ07N7fcckt23nnnrLzyyvXL+/fvn7333jsTJ07MzJkzkyR77rln3n///VxzzTX1691yyy2ZPn16g9+ZH3XbbbflvffeyyGHHNLgfS9o79vG/izutttu6dOnz0K3DQBtlUOnAaCVmDt3bsaPH5/NN988zz77bP3yjTbaKGeccUZuv/32bL311unQoUN22223XHbZZamtrU1NTU2uueaavP/++w3+o/npp5/OP/7xj4X+x+y0adMa3J93KOlH3XDDDTn55JPzyCOPNDiX3If/w/u5555Lu3bt5nuNj14t+7XXXsv06dNzwQUX5IILLlisuRbXCius0OD+vCj11ltvLfJ5Tz/9dKrValZbbbUFPv7RC88st9xy813ko7Gf9UdnnTfvvFlfe+21zJw5M2uvvfbHzv7YY48t9nZLLb/88vOFpp49e2bAgAHzLUvS4P00xde9Wq0ucHnnzp3n+wx69uy50Hk//D3x05/+NDvttFO+8IUvZO21184222yTb33rW1l33XUXuP1PeqGQeT8jH/2Z6NevX3r16jVfEF/Yz+NHzQukHz0s+MNmzJiRJZdcMu+8805Gjx6dsWPH5sUXX2zwec6YMaP+388880x22223xdr+J/25m+ejP3errLJK2rVrN995Iz/6ebz22muZM2fOAg8pX3PNNVNXV5epU6dmrbXWynrrrZc11lgjl19+efbbb78k/zlseumll67/nwELMu9r8tEZ+/TpM1/0buzP4uJ+fQGgrREaAaCVuOOOO/Lyyy9n/PjxGT9+/HyPjxs3LltvvXWSZMSIETn//PNz4403Zuedd84VV1yRNdZYI+utt179+nV1dVlnnXXyy1/+coHb+2gc+vDeTvPcfffd2XHHHTNkyJCcc8456d+/fzp27JixY8fOd1GNxTFvz6pvfvObCw0jCwo8i6N9+/YLXL6wOPXhmSqVSm688cYFvka3bt0a3F/Q59TYz/qTzrqg7W611VY5+uijF/j4F77whUa93sdZ2Nwf935Kv+69e/dOpVJZaLz6pHMlyZAhQ/LMM8/kj3/8Y2655ZZcdNFFOfPMM3Peeec12Esy+U88+/D5CT+JxQ2VC/o+W5B5n+3pp5+eL37xiwtcZ9738CGHHJKxY8fmsMMOy+DBg9OzZ89UKpWMGDHiYy8etDBN9b08z8I+n8X9PBZmzz33zM9+9rO8/vrr6d69e66//vrstdde6dChaf5zqLE/i6XvBwBaK6ERAFqJcePGZZlllqm/Wu6HXXPNNbn22mtz3nnnpUuXLhkyZEj69++fyy+/PF/96ldzxx131F/QY55VVlkljz76aLbccstPvBfW1Vdfnc6dO+fmm29OTU1N/fKxY8c2WG/FFVdMXV1dnn322QZ7/3z06rN9+vRJ9+7dM3fu3Pqr+La0VVZZJdVqNSuttNInDnNN8Vl/WJ8+fdKjR4888cQTH7vdWbNmfeLPsilmXRylX/cOHTpklVVWabCnb1OadyX373znO5k1a1aGDBmSE044Yb7Q+OyzzzaI+Y0x72fk6aefzpprrlm//NVXX8306dOz4oorfqLXnXcIfo8ePT72s73qqqsycuTInHHGGfXL3n333UyfPn2+1/y4772m8vTTTzfYu2/y5Mmpq6urvxjVwvTp0yddu3bNU089Nd9j//znP9OuXbsGgX/PPffMiSeemKuvvjp9+/bNzJkzG5wGYEHmfU2efvrpBodnv/baa/NF79KfRQD4rHCORgBoBd55551cc8012X777bP77rvPdzv44IPz9ttv5/rrr0+StGvXLrvvvnv+9Kc/5dJLL80HH3ww37nG9thjj7z44ou58MILF7i92bNnf+xc7du3T6VSqT8/ZJJMmTJlvqsozzuX3DnnnNNg+VlnnTXf6+222265+uqrFxgyXnvttY+dqantuuuuad++fU488cT59sKqVqt54403PvY1muKz/rB27dpl5513zp/+9Kc89NBD8z0+b8499tgj9913X26++eb51pk+fXo++OCDRW5niSWWmC8yNYem+LoPHjx4gZ9FqY9+fbt165ZVV121wWkCkv8cWvzMM8/UX0G6sbbddtskme/qw/P2gl3QFdoXxwYbbJBVVlklv/jFLzJr1qz5Hv/wZ9u+ffv5vsfPOuusBj/fyX/OH/joo4/OdyXs5JPvqbgwH/0fK/N+ZwwfPnyRz2vfvn223nrr/PGPf2xwmPWrr76ayy67LF/96lfTo0eP+uVrrrlm1llnnVx++eW5/PLL079//wwZMmSR2xg6dGg6duyYs846q8H7XtDV3Et/FgHgs8IejQDQClx//fV5++23s+OOOy7w8a985Svp06dPxo0bVx8U99xzz5x11lk5/vjjs8466zTYSypJvvWtb+WKK67I9773vdx5553ZZJNNMnfu3Pzzn//MFVdckZtvvnmBFxr5sO222y6//OUvs80222TvvffOtGnTcvbZZ2fVVVfNY489Vr/eBhtskN122y1jxozJG2+8ka985SuZMGFC/vWvfyVpuOfcz3/+89x5553ZaKONcsABB2TQoEF588038/DDD+e2227Lm2+++Yk+w09qlVVWycknn5xRo0ZlypQp2XnnndO9e/c8++yzufbaa3PggQfmqKOOWuRrNMVn/VGnnHJKbrnllmy66aY58MADs+aaa+bll1/OlVdemYkTJ6ZXr1750Y9+lOuvvz7bb7999tlnn2ywwQaZPXt2Hn/88Vx11VWZMmXKIg/13WCDDXLuuefm5JNPzqqrrppllllmkeesK1H6dd9pp51y6aWX5l//+leTHhI+aNCgbLbZZtlggw3Su3fvPPTQQ7nqqqty8MEHN1jvtttuS7VazU477fSJtrPeeutl5MiRueCCCzJ9+vRsuummefDBB3PJJZdk5513rr8AVGO1a9cuF110UYYPH5611lor3/nOd7LccsvlxRdfzJ133pkePXrkT3/6U5Jk++23z6WXXpqePXtm0KBBue+++3LbbbdlqaWWavCaP/rRj3LVVVfl61//evbdd99ssMEGefPNN3P99dfnvPPO+8R7dS7Is88+mx133DHbbLNN7rvvvvz+97/P3nvvvVjbOPnkk3Prrbfmq1/9an7wgx+kQ4cOOf/881NbW5vTTjttvvX33HPPHHfccencuXP222+/tGu36H0u+vTpk6OOOiqjR4/O9ttvn2233TZ/+9vfcuONN873c1X6swgAnxVCIwC0AuPGjUvnzp2z1VZbLfDxdu3aZbvttsu4cePyxhtvZKmllsrGG2+cAQMGZOrUqQu8cmq7du1y3XXX5cwzz8zvfve7XHvttenatWtWXnnlHHrooYsVa7bYYov89re/zc9//vMcdthhWWmllXLqqadmypQpDUJjkvzud79Lv3798oc//CHXXntthg4dmssvvzyrr756OnfuXL9e37598+CDD+anP/1prrnmmpxzzjlZaqmlstZaa+XUU09t5CfXNI455ph84QtfyJlnnpkTTzwxyX/Oq7j11lsvNP5+WFN81h+13HLL5YEHHshPfvKTjBs3LjNnzsxyyy2X4cOHp2vXrkmSrl27ZsKECTnllFNy5ZVX5ne/+1169OiRL3zhCznxxBPrL8qyMMcdd1yee+65nHbaaXn77bez6aabNltoLP2677DDDll66aVzxRVX5Mc//nGTzfXDH/4w119/fW655ZbU1tZmxRVXzMknn5wf/ehHDda78sor89WvfrXB1cIb66KLLsrKK6+ciy++ONdee2369euXUaNG5fjjjy96D5tttlnuu+++nHTSSfnNb36TWbNmpV+/ftloo43y3e9+t369X/3qV2nfvn3GjRuXd999N5tsskluu+22+a5u3a1bt9x99905/vjjc+211+aSSy7JMsssky233DLLL7980awfdfnll+e4447LMccckw4dOuTggw/O6aefvljPXWuttXL33Xdn1KhRGT16dOrq6rLRRhvl97//fYOroM+z55575sc//nHmzJmzyKtNf9jJJ5+czp0757zzzqsP5bfccst8e6CW/iwCwGdFpdrUxz8AAPw/jzzySNZff/38/ve/zze+8Y2WHoc27qSTTsrYsWPz9NNPL/QiJM3hlVdeyUorrZTx48d/4j0aaeiEE07IiSeemNdee82efgDwGeIcjQBAk3jnnXfmWzZmzJi0a9fuY8+FBovj8MMPz6xZsxZ4VfbmNGbMmKyzzjoiIwDAx3DoNADQJE477bRMmjQpm2++eTp06JAbb7wxN954Yw488MAGV3+FT6pbt26ZNm3ap77dn//855/6NgEA2iKhEQBoEhtvvHFuvfXWnHTSSZk1a1ZWWGGFnHDCCTn22GNbejQAAOBT0KLnaDz33HNz7rnnZsqUKUn+c0Ln4447LsOHD0+SvPvuuznyyCMzfvz41NbWZtiwYTnnnHPSt2/flhoZAAAAAFiAFg2Nf/rTn9K+ffusttpqqVarueSSS3L66afnb3/7W9Zaa618//vfz5///OdcfPHF6dmzZw4++OC0a9cu99xzT0uNDAAAAAAsQKu76nTv3r1z+umnZ/fdd0+fPn1y2WWXZffdd0+S/POf/8yaa66Z++67L1/5yldaeFIAAAAAYJ5Wc47GuXPn5sorr8zs2bMzePDgTJo0Ke+//36GDh1av84aa6yRFVZYoVGhsa6uLi+99FK6d++eSqXSXOMDAAAAwGdStVrN22+/nWWXXTbt2rVb6HotHhoff/zxDB48OO+++266deuWa6+9NoMGDcojjzySTp06pVevXg3W79u3b1555ZWFvl5tbW1qa2vr77/44osZNGhQc40PAAAAAJ8LU6dOzfLLL7/Qx1s8NK6++up55JFHMmPGjFx11VUZOXJkJkyY8Ilfb/To0TnxxBPnWz516tT06NGjZFQAAAAA+NyZOXNmBgwYkO7duy9yvVZ3jsahQ4dmlVVWyZ577pktt9wyb731VoO9GldcccUcdthhOfzwwxf4/I/u0Tjvg5gxY4bQCAAAAACNNHPmzPTs2fNj+9rCD6puIXV1damtrc0GG2yQjh075vbbb69/7Kmnnsrzzz+fwYMHL/T5NTU16dGjR4MbAAAAANC8WvTQ6VGjRmX48OFZYYUV8vbbb+eyyy7LXXfdlZtvvjk9e/bMfvvtlyOOOCK9e/dOjx49csghh2Tw4MGuOA0AAAAArUyLhsZp06bl29/+dl5++eX07Nkz6667bm6++eZstdVWSZIzzzwz7dq1y2677Zba2toMGzYs55xzTkuODAAAAAAsQKs7R2NTW9xjyAEAAACA+bXZczQCAAAAAG2P0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAK3c7NmzU6lUUqlUMnv27JYeBwAAABZIaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIq1aGgcPXp0vvSlL6V79+5ZZpllsvPOO+epp55qsM5mm22WSqXS4Pa9732vhSYGAAAAABakRUPjhAkTctBBB+X+++/Prbfemvfffz9bb711Zs+e3WC9Aw44IC+//HL97bTTTmuhiQEAAACABenQkhu/6aabGty/+OKLs8wyy2TSpEkZMmRI/fKuXbumX79+n/Z4AAAAAMBialXnaJwxY0aSpHfv3g2Wjxs3LksvvXTWXnvtjBo1KnPmzGmJ8QAAAACAhWjRPRo/rK6uLocddlg22WSTrL322vXL995776y44opZdtll89hjj+W///u/89RTT+Waa65Z4OvU1tamtra2/v7MmTObfXYAAAAA+LxrNaHxoIMOyhNPPJGJEyc2WH7ggQfW/3udddZJ//79s+WWW+aZZ57JKqusMt/rjB49OieeeGKzzwsAAAAA/P9axaHTBx98cG644YbceeedWX755Re57kYbbZQkmTx58gIfHzVqVGbMmFF/mzp1apPPCwAAAAA01KJ7NFar1RxyyCG59tprc9ddd2WllVb62Oc88sgjSZL+/fsv8PGamprU1NQ05ZgAAAAAwMdo0dB40EEH5bLLLssf//jHdO/ePa+88kqSpGfPnunSpUueeeaZXHbZZdl2222z1FJL5bHHHsvhhx+eIUOGZN11123J0QEAAACAD6lUq9Vqi228Ulng8rFjx2afffbJ1KlT881vfjNPPPFEZs+enQEDBmSXXXbJj3/84/To0WOxtjFz5sz07NkzM2bMWOznALQms2fPTrdu3ZIks2bNyhJLLNHCEwEAAPB5srh9rcUPnV6UAQMGZMKECZ/SNAAAAADAJ9UqLgYDAAAAALRtQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQrENLD0C5gcf8uaVHAJpR3Xvv1v97zZ/clHadOrfgNEBzmvLz7Vp6BAAA+MTs0QgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKtWhoHD16dL70pS+le/fuWWaZZbLzzjvnqaeearDOu+++m4MOOihLLbVUunXrlt122y2vvvpqC00MAAAAACxIi4bGCRMm5KCDDsr999+fW2+9Ne+//3623nrrzJ49u36dww8/PH/6059y5ZVXZsKECXnppZey6667tuDUAAAAAMBHdWjJjd90000N7l988cVZZpllMmnSpAwZMiQzZszIb3/721x22WXZYostkiRjx47Nmmuumfvvvz9f+cpXWmJsAAAAAOAjWtU5GmfMmJEk6d27d5Jk0qRJef/99zN06ND6ddZYY42ssMIKue+++1pkRgAAAABgfi26R+OH1dXV5bDDDssmm2yStddeO0nyyiuvpFOnTunVq1eDdfv27ZtXXnllga9TW1ub2tra+vszZ85stpkBAAAAgP9oNXs0HnTQQXniiScyfvz4otcZPXp0evbsWX8bMGBAE00IAAAAi2/27NmpVCqpVCoNrkUA8FnVKkLjwQcfnBtuuCF33nlnll9++frl/fr1y3vvvZfp06c3WP/VV19Nv379Fvhao0aNyowZM+pvU6dObc7RAQAAAIC0cGisVqs5+OCDc+211+aOO+7ISiut1ODxDTbYIB07dsztt99ev+ypp57K888/n8GDBy/wNWtqatKjR48GNwAAAACgebXoORoPOuigXHbZZfnjH/+Y7t271593sWfPnunSpUt69uyZ/fbbL0cccUR69+6dHj165JBDDsngwYNdcRoAAAAAWpEWDY3nnntukmSzzTZrsHzs2LHZZ599kiRnnnlm2rVrl9122y21tbUZNmxYzjnnnE95UoCW065T56z43ze09BgAAACwSC0aGqvV6seu07lz55x99tk5++yzP4WJAAAAAIBPolVcDAYAAAAAaNuERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKBYh8asPH369Fx77bW5++6789xzz2XOnDnp06dP1l9//QwbNiwbb7xxc80JAAAAALRii7VH40svvZT9998//fv3z8knn5x33nknX/ziF7Pllltm+eWXz5133pmtttoqgwYNyuWXX97cMwMAAAAArcxi7dG4/vrrZ+TIkZk0aVIGDRq0wHXeeeedXHfddRkzZkymTp2ao446qkkHBQAAAABar8UKjX//+9+z1FJLLXKdLl26ZK+99spee+2VN954o0mGAwAAAADahsU6dPrjImPp+gAAAABA29boq05fcskl+fOf/1x//+ijj06vXr2y8cYb57nnnmvS4QAAAACAtqHRofGUU05Jly5dkiT33Xdfzj777Jx22mlZeumlc/jhhzf5gAAAAABA67dY52j8sKlTp2bVVVdNklx33XXZbbfdcuCBB2aTTTbJZptt1tTzAQAAAABtQKP3aOzWrVv9xV5uueWWbLXVVkmSzp0755133mna6QAAAACANqHRezRutdVW2X///bP++uvnX//6V7bddtskyZNPPpmBAwc29XwAAAAAQBvQ6D0azz777AwePDivvfZarr766vorTE+aNCl77bVXkw8IAAAAALR+jd6jsVevXvnNb34z3/ITTzyxSQYCAAAAANqeRofGJHn33Xfz2GOPZdq0aamrq6tfXqlUssMOOzTZcAAAAABA29Do0HjTTTflW9/6Vv0FYT6sUqlk7ty5TTIYAAAAANB2NPocjYccckj22GOPvPzyy6mrq2twExkBAAAA4POp0aHx1VdfzRFHHJG+ffs2xzwAAAAAQBvU6NC4++6756677mqGUQAAAACAtqrR52j8zW9+k69//eu5++67s84666Rjx44NHv/hD3/YZMMBAAAAAG1Do0PjH/7wh9xyyy3p3Llz7rrrrlQqlfrHKpWK0AgAAAAAn0ONDo3HHntsTjzxxBxzzDFp167RR14DAAAAAJ9BjS6F7733Xvbcc0+REQAAAACo1+haOHLkyFx++eXNMQsAAAAA0EY1+tDpuXPn5rTTTsvNN9+cddddd76Lwfzyl79ssuEAAAAAgLah0aHx8ccfz/rrr58keeKJJxo89uELwwAAAAAAnx+NDo133nlnc8wBAAAAALRhrugCAAAAABRbrND4ve99Ly+88MJiveDll1+ecePGFQ0FAAAAALQti3XodJ8+fbLWWmtlk002yQ477JANN9wwyy67bDp37py33norf//73zNx4sSMHz8+yy67bC644ILmnhsAAAAAaEUWKzSedNJJOfjgg3PRRRflnHPOyd///vcGj3fv3j1Dhw7NBRdckG222aZZBgUAAAAAWq/FvhhM3759c+yxx+bYY4/NW2+9leeffz7vvPNOll566ayyyiquOA0AAAAAn2ONvup0kiy55JJZcsklm3oWAAAAAKCNctVpAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUOwThcYPPvggt912W84///y8/fbbSZKXXnops2bNatLhAAAAAIC2odFXnX7uueeyzTbb5Pnnn09tbW222mqrdO/ePaeeempqa2tz3nnnNcecAAAAAEAr1ug9Gg899NBsuOGGeeutt9KlS5f65bvssktuv/32Jh0OAAAAAGgbGr1H491335177703nTp1arB84MCBefHFF5tsMAAAAACg7Wj0Ho11dXWZO3fufMtfeOGFdO/evUmGAgAAAADalkaHxq233jpjxoypv1+pVDJr1qwcf/zx2XbbbZtyNgAAAACgjWj0odNnnHFGhg0blkGDBuXdd9/N3nvvnaeffjpLL710/vCHPzTHjAAAAABAK9fo0Lj88svn0Ucfzfjx4/PYY49l1qxZ2W+//fKNb3yjwcVhAAAAAIDPj0aHxiTp0KFDvvnNbzb1LAAAAABAG/WJQuNLL72UiRMnZtq0aamrq2vw2A9/+MMmGQwAAAAAaDsaHRovvvjifPe7302nTp2y1FJLpVKp1D9WqVSERgAAAAD4HGp0aPzJT36S4447LqNGjUq7do2+aDUAAAAA8BnU6FI4Z86cjBgxQmQEAAAAAOo1uhbut99+ufLKK5tjFgAAAACgjWr0odOjR4/O9ttvn5tuuinrrLNOOnbs2ODxX/7yl002HAAAAADQNnyi0HjzzTdn9dVXT5L5LgYDAAAAAHz+NDo0nnHGGfnf//3f7LPPPs0wDgAAAADQFjX6HI01NTXZZJNNmmMWAAAAAKCNanRoPPTQQ3PWWWc1xywAAAAAQBvV6EOnH3zwwdxxxx254YYbstZaa813MZhrrrmmyYYDAAAAANqGRofGXr16Zdddd22OWQAAAACANqrRoXHs2LHNMQcAAAAA0IY1+hyNAAAAAAAftVh7NP7Xf/1Xbr/99iy55JJZf/31U6lUFrruww8/3GTDAQAAAABtw2KFxp122ik1NTX1/15UaAQAAAAAPn8WKzQef/zx9f8+4YQTmmsWAAAAAKCNavQ5GldeeeW88cYb8y2fPn16Vl555SYZCgAAAABoWxodGqdMmZK5c+fOt7y2tjYvvPBCkwwFAAAAALQti3XodJJcf/319f+++eab07Nnz/r7c+fOze23356VVlqpaacDAIDPuIHH/LmlRwCaSd1779b/e82f3JR2nTq34DRAc5vy8+1aeoQWt9ihceedd06SVCqVjBw5ssFjHTt2zMCBA3PGGWc06XAAAAAAQNuw2KGxrq4uSbLSSivlr3/9a5ZeeulmGwoAAAAAaFsWOzTO8+yzzzbHHAAAAABAG9boi8EAAAAAAHyU0AgAAAAAFBMaAQAAAIBiQiMAAAAAUOwThcZnnnkmP/7xj7PXXntl2rRpSZIbb7wxTz75ZJMOBwAAAAC0DY0OjRMmTMg666yTBx54INdcc01mzZqVJHn00Udz/PHHN/mAAAAAAEDr1+jQeMwxx+Tkk0/Orbfemk6dOtUv32KLLXL//fc36XAAAAAAQNvQ6ND4+OOPZ5dddplv+TLLLJPXX3+9SYYCAAAAANqWRofGXr165eWXX55v+d/+9rcst9xyTTIUAAAAANC2NDo0jhgxIv/93/+dV155JZVKJXV1dbnnnnty1FFH5dvf/nZzzAgAAAAAtHKNDo2nnHJK1lhjjQwYMCCzZs3KoEGDMmTIkGy88cb58Y9/3KjX+stf/pIddtghyy67bCqVSq677roGj++zzz6pVCoNbttss01jRwYAAAAAmlmHxj6hU6dOufDCC3Pcccfl8ccfz6xZs7L++utntdVWa/TGZ8+enfXWWy/77rtvdt111wWus80222Ts2LH192tqahq9HQAAAACgeTU6NM4zYMCADBgwoGjjw4cPz/Dhwxe5Tk1NTfr161e0HQAAAACgeTX60Onddtstp5566nzLTzvttHz9619vkqE+7K677soyyyyT1VdfPd///vfzxhtvNPk2AAAAAIAyjQ6Nf/nLX7LtttvOt3z48OH5y1/+0iRDzbPNNtvkd7/7XW6//faceuqpmTBhQoYPH565c+cu9Dm1tbWZOXNmgxsAAAAA0Lwafej0rFmz0qlTp/mWd+zYscmj3ogRI+r/vc4662TdddfNKquskrvuuitbbrnlAp8zevTonHjiiU06BwAAAACwaI3eo3GdddbJ5ZdfPt/y8ePHZ9CgQU0y1MKsvPLKWXrppTN58uSFrjNq1KjMmDGj/jZ16tRmnQkAAAAA+AR7NP7kJz/JrrvummeeeSZbbLFFkuT222/PH/7wh1x55ZVNPuCHvfDCC3njjTfSv3//ha5TU1PjytQAAAAA8ClrdGjcYYcdct111+WUU07JVVddlS5dumTdddfNbbfdlk033bRRrzVr1qwGeyc+++yzeeSRR9K7d+/07t07J554Ynbbbbf069cvzzzzTI4++uisuuqqGTZsWGPHBgAAAACaUaNDY5Jst9122W677Yo3/tBDD2XzzTevv3/EEUckSUaOHJlzzz03jz32WC655JJMnz49yy67bLbeeuucdNJJ9lgEAAAAgFbmE4XGJHnvvfcybdq01NXVNVi+wgorLPZrbLbZZqlWqwt9/Oabb/6k4wEAAAAAn6JGh8ann346++67b+69994Gy6vVaiqVSubOndtkwwEAAAAAbUOjQ+M+++yTDh065IYbbkj//v1TqVSaYy4AAAAAoA1pdGh85JFHMmnSpKyxxhrNMQ8AAAAA0Aa1a+wTBg0alNdff705ZgEAAAAA2qhGh8ZTTz01Rx99dO6666688cYbmTlzZoMbAAAAAPD50+hDp4cOHZok2XLLLRssdzEYAAAAAPj8anRovPPOO5tjDgAAAACgDWt0aNx0002bYw4AAAAAoA1r9Dkak+Tuu+/ON7/5zWy88cZ58cUXkySXXnppJk6c2KTDAQAAAABtQ6ND49VXX51hw4alS5cuefjhh1NbW5skmTFjRk455ZQmHxAAAAAAaP0aHRpPPvnknHfeebnwwgvTsWPH+uWbbLJJHn744SYdDgAAAABoGxodGp966qkMGTJkvuU9e/bM9OnTm2ImAAAAAKCNaXRo7NevXyZPnjzf8okTJ2bllVdukqEAAAAAgLal0aHxgAMOyKGHHpoHHngglUolL730UsaNG5ejjjoq3//+95tjRgAAAACglevQ2Cccc8wxqaury5Zbbpk5c+ZkyJAhqampyVFHHZVDDjmkOWYEAAAAAFq5RoXGuXPn5p577slBBx2UH/3oR5k8eXJmzZqVQYMGpVu3bs01IwAAAADQyjUqNLZv3z5bb711/vGPf6RXr14ZNGhQc80FAAAAALQhjT5H49prr51///vfzTELAAAAANBGNTo0nnzyyTnqqKNyww035OWXX87MmTMb3AAAAACAz59GXwxm2223TZLsuOOOqVQq9cur1WoqlUrmzp3bdNMBAAAAAG1Co0PjnXfe2RxzAAAAAABtWKND46abbtoccwAAAAAAbVijz9GYJHfffXe++c1vZuONN86LL76YJLn00kszceLEJh0OAAAAAGgbGh0ar7766gwbNixdunTJww8/nNra2iTJjBkzcsoppzT5gAAAAABA6/eJrjp93nnn5cILL0zHjh3rl2+yySZ5+OGHm3Q4AAAAAKBtaHRofOqppzJkyJD5lvfs2TPTp09vipkAAAAAgDam0aGxX79+mTx58nzLJ06cmJVXXrlJhgIAAAAA2pZGh8YDDjgghx56aB544IFUKpW89NJLGTduXI466qh8//vfb44ZAQAAAIBWrkNjn3DMMcekrq4uW265ZebMmZMhQ4akpqYmRx11VA455JDmmBEAAAAAaOUWKzQ+9thjWXvttdOuXbtUKpUce+yx+dGPfpTJkydn1qxZGTRoULp169bcswIAAAAArdRiHTq9/vrr5/XXX0+SrLzyynnjjTfSqVOnDBo0KF/+8pdFRgAAAAD4nFus0NirV688++yzSZIpU6akrq6uWYcCAAAAANqWxTp0erfddsumm26a/v37p1KpZMMNN0z79u0XuO6///3vJh0QAAAAAGj9Fis0XnDBBdl1110zefLk/PCHP8wBBxyQ7t27N/dsAAAAAEAbsdhXnd5mm22SJJMmTcqhhx4qNAIAAAAA9RY7NM4zduzY5pgDAAAAAGjDGh0aZ8+enZ///Oe5/fbbM23atPkuDOMcjQAAAADw+dPo0Lj//vtnwoQJ+da3vlV/cRgAAAAA4POt0aHxxhtvzJ///OdssskmzTEPAAAAANAGtWvsE5Zccsn07t27OWYBAAAAANqoRofGk046Kccdd1zmzJnTHPMAAAAAAG1Qow+dPuOMM/LMM8+kb9++GThwYDp27Njg8YcffrjJhgMAAAAA2oZGh8add965GcYAAAAAANqyRofG448/vjnmAAAAAADasEafoxEAAAAA4KMWe4/GJZdcMpVK5WPXe/PNN4sGAgAAAADansUOjWPGjGnGMQAAAACAtmyxQ+PIkSObcw4AAAAAoA1zjkYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIot9lWn5zniiCMWuLxSqaRz585ZddVVs9NOO6V3797FwwEAAAAAbUOjQ+Pf/va3PPzww5k7d25WX331JMm//vWvtG/fPmussUbOOeecHHnkkZk4cWIGDRrU5AMDAAAAAK1Pow+d3mmnnTJ06NC89NJLmTRpUiZNmpQXXnghW221Vfbaa6+8+OKLGTJkSA4//PDmmBcAAAAAaIUaHRpPP/30nHTSSenRo0f9sp49e+aEE07Iaaedlq5du+a4447LpEmTmnRQAAAAAKD1anRonDFjRqZNmzbf8tdeey0zZ85MkvTq1Svvvfde+XQAAAAAQJvwiQ6d3nfffXPttdfmhRdeyAsvvJBrr702++23X3beeeckyYMPPpgvfOELTT0rAAAAANBKNfpiMOeff34OP/zwjBgxIh988MF/XqRDh4wcOTJnnnlmkmSNNdbIRRdd1LSTAgAAAACtVqNDY7du3XLhhRfmzDPPzL///e8kycorr5xu3brVr/PFL36xyQYEAAAAAFq/Rh86/fvf/z5z5sxJt27dsu6662bddddtEBkBAAAAgM+fRofGww8/PMsss0z23nvv/N///V/mzp3bHHMBAAAAAG1Io0Pjyy+/nPHjx6dSqWSPPfZI//79c9BBB+Xee+9tjvkAAAAAgDag0aGxQ4cO2X777TNu3LhMmzYtZ555ZqZMmZLNN988q6yySnPMCAAAAAC0co2+GMyHde3aNcOGDctbb72V5557Lv/4xz+aai4AAAAAoA1p9B6NSTJnzpyMGzcu2267bZZbbrmMGTMmu+yyS5588smmng8AAAAAaAMavUfjiBEjcsMNN6Rr167ZY4898pOf/CSDBw9ujtkAAAAAgDai0aGxffv2ueKKKzJs2LC0b9++wWNPPPFE1l577SYbDgAAAABoGxodGseNG9fg/ttvv50//OEPueiiizJp0qTMnTu3yYYDAAAAANqGT3SOxiT5y1/+kpEjR6Z///75xS9+kS222CL3339/U84GAAAAALQRjdqj8ZVXXsnFF1+c3/72t5k5c2b22GOP1NbW5rrrrsugQYOaa0YAAAAAoJVb7D0ad9hhh6y++up57LHHMmbMmLz00ks566yzmnM2AAAAAKCNWOw9Gm+88cb88Ic/zPe///2sttpqzTkTAAAAANDGLPYejRMnTszbb7+dDTbYIBtttFF+85vf5PXXX2/O2QAAAACANmKxQ+NXvvKVXHjhhXn55Zfz3e9+N+PHj8+yyy6burq63HrrrXn77bebc04AAAAAoBVr9FWnl1hiiey7776ZOHFiHn/88Rx55JH5+c9/nmWWWSY77rhjc8wIAAAAALRyjQ6NH7b66qvntNNOywsvvJA//OEPTTUTAAAAANDGFIXGedq3b5+dd945119/fVO8HAAAAADQxjRJaAQAAAAAPt+ERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAinVo6QEAAADgs6hdp85Z8b9vaOkxAD419mgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKtWho/Mtf/pIddtghyy67bCqVSq677roGj1er1Rx33HHp379/unTpkqFDh+bpp59umWEBAAAAgIVq0dA4e/bsrLfeejn77LMX+Phpp52WX//61znvvPPywAMPZIkllsiwYcPy7rvvfsqTAgAAAACL0qElNz58+PAMHz58gY9Vq9WMGTMmP/7xj7PTTjslSX73u9+lb9++ue666zJixIhPc1QAAAAAYBFa7Tkan3322bzyyisZOnRo/bKePXtmo402yn333bfQ59XW1mbmzJkNbgAAAABA82q1ofGVV15JkvTt27fB8r59+9Y/tiCjR49Oz549628DBgxo1jkBAAAAgFYcGj+pUaNGZcaMGfW3qVOntvRIAAAAAPCZ12pDY79+/ZIkr776aoPlr776av1jC1JTU5MePXo0uAEAAAAAzavVhsaVVlop/fr1y+23316/bObMmXnggQcyePDgFpwMAAAAAPioFr3q9KxZszJ58uT6+88++2weeeSR9O7dOyussEIOO+ywnHzyyVlttdWy0kor5Sc/+UmWXXbZ7Lzzzi03NAAAAAAwnxYNjQ899FA233zz+vtHHHFEkmTkyJG5+OKLc/TRR2f27Nk58MADM3369Hz1q1/NTTfdlM6dO7fUyAAAAADAArRoaNxss81SrVYX+nilUslPf/rT/PSnP/0UpwIAAAAAGqvVnqMRAAAAAGg7hEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAECxVh0aTzjhhFQqlQa3NdZYo6XHAgAAAAA+okNLD/Bx1lprrdx222319zt0aPUjAwAAAMDnTquvdh06dEi/fv1aegwAAAAAYBFa9aHTSfL0009n2WWXzcorr5xvfOMbef755xe5fm1tbWbOnNngBgAAAAA0r1YdGjfaaKNcfPHFuemmm3Luuefm2Wefzde+9rW8/fbbC33O6NGj07Nnz/rbgAEDPsWJAQAAAODzqVWHxuHDh+frX/961l133QwbNiz/93//l+nTp+eKK65Y6HNGjRqVGTNm1N+mTp36KU4MAAAAAJ9Prf4cjR/Wq1evfOELX8jkyZMXuk5NTU1qamo+xakAAAAAgFa9R+NHzZo1K88880z69+/f0qMAAAAAAB/SqkPjUUcdlQkTJmTKlCm59957s8suu6R9+/bZa6+9Wno0AAAAAOBDWvWh0y+88EL22muvvPHGG+nTp0+++tWv5v7770+fPn1aejQAAAAA4ENadWgcP358S48AAAAAACyGVn3oNAAAAADQNgiNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYkIjAAAAAFBMaAQAAAAAigmNAAAAAEAxoREAAAAAKCY0AgAAAADFhEYAAAAAoJjQCAAAAAAUExoBAAAAgGJCIwAAAABQTGgEAAAAAIoJjQAAAABAMaERAAAAACgmNAIAAAAAxYRGAAAAAKCY0AgAAAAAFBMaAQAAAIBiQiMAAAAAUExoBAAAAACKCY0AAAAAQDGhEQAAAAAoJjQCAAAAAMWERgAAAACgmNAIAAAAABQTGgEAAACAYm0iNJ599tkZOHBgOnfunI022igPPvhgS48EAAAAAHxIqw+Nl19+eY444ogcf/zxefjhh7Peeutl2LBhmTZtWkuPBgAAAAD8P60+NP7yl7/MAQcckO985zsZNGhQzjvvvHTt2jX/+7//29KjAQAAAAD/T4eWHmBR3nvvvUyaNCmjRo2qX9auXbsMHTo099133wKfU1tbm9ra2vr7M2bMSJLMnDmzeYdtQXW1c1p6BACgCXyW/15h4fwtBwCfDZ/lv+XmvbdqtbrI9Vp1aHz99dczd+7c9O3bt8Hyvn375p///OcCnzN69OiceOKJ8y0fMGBAs8wIANBUeo5p6QkAAPikPg9/y7399tvp2bPnQh9v1aHxkxg1alSOOOKI+vt1dXV58803s9RSS6VSqbTgZACf3MyZMzNgwIBMnTo1PXr0aOlxAABYTP6OAz4LqtVq3n777Sy77LKLXK9Vh8all1467du3z6uvvtpg+auvvpp+/fot8Dk1NTWpqalpsKxXr17NNSLAp6pHjx7+QAUAaIP8HQe0dYvak3GeVn0xmE6dOmWDDTbI7bffXr+srq4ut99+ewYPHtyCkwEAAAAAH9aq92hMkiOOOCIjR47MhhtumC9/+csZM2ZMZs+ene985zstPRoAAAAA8P+0+tC455575rXXXstxxx2XV155JV/84hdz0003zXeBGIDPspqamhx//PHznRoCAIDWzd9xwOdJpfpx16UGAAAAAPgYrfocjQAAAABA2yA0AgAAAADFhEYAAAAAoJjQCNDGXHzxxenVq1dLjwEAAAANCI3A594+++yTSqWSSqWSTp06ZdVVV81Pf/rTfPDBBx/73Isvvrj+uQu7TZkypfnfxGK6+uqrs9lmm6Vnz57p1q1b1l133fz0pz/Nm2++maTh+2nXrl2WX375fOc738m0adOSJFOmTEmlUskjjzwy32tvttlmOeywwz7FdwMAsHBTp07Nvvvum2WXXTadOnXKiiuumEMPPTRvvPFGg/U222yzVCqVjB8/vsHyMWPGZODAgfX35/2dtM022zRYb/r06alUKrnrrruSJI8++mg6deqU66+/vsF6V199dTp37pwnnnhikXP7ew1oy4RGgCTbbLNNXn755Tz99NM58sgjc8IJJ+T000//2Oftueeeefnll+tvgwcPzgEHHNBg2YABAxZ7jvfee6/kbSzSsccemz333DNf+tKXcuONN+aJJ57IGWeckUcffTSXXnpp/Xo9evTIyy+/nBdeeCEXXnhhbrzxxnzrW99qtrkAAJrav//972y44YZ5+umn84c//CGTJ0/Oeeedl9tvvz2DBw+uj3bzdO7cOT/+8Y/z/vvvL/J1O3TokNtuuy133nnnQtdZb731ctxxx+XAAw+sj5rTpk3L9773vZx44olZe+21F/pcf68BbZ3QCJCkpqYm/fr1y4orrpjvf//7GTp0aK6//vrMnj07PXr0yFVXXdVg/euuuy5LLLFEPvjgg/Tr16/+1qlTp3Tt2rX+/nvvvZddd9013bp1S48ePbLHHnvk1VdfrX+dE044IV/84hdz0UUXZaWVVkrnzp2T/Of/jH/3u99N375907lz56y99tq54YYbGsxw8803Z80110y3bt3qQ+nCPPjggznllFNyxhln5PTTT8/GG2+cgQMHZquttsrVV1+dkSNH1q9bqVTSr1+/LLvsshk+fHh++MMf5rbbbss777zTFB81AECzO+igg9KpU6fccsst2XTTTbPCCitk+PDhue222/Liiy/m2GOPbbD+XnvtlenTp+fCCy9c5OsuscQS2XfffXPMMccscr1Ro0ZlhRVWyEEHHZQk+e53v5vVVlstRx111EKf4+814LNAaARYgC5duuS9997LEksskREjRmTs2LENHh87dmx23333dO/efaGvUVdXl5122ilvvvlmJkyYkFtvvTX//ve/s+eeezZYb/Lkybn66v+vvTsLifJt4zj+G61wy60kK0sYzVLJqDTPFEMTqSOlJJWiFAMpUkNDPEjFSkOCJKwgyj08SBCiHNQioyRbEAyFbDOjIqNM1MpI34NoyNTRsqh/7/dzNHPfz73MHF1c9/KcV21trdra2jQyMqKoqChdv35dlZWV6ujoUEFBgaytrc1thoaGVFRUpIqKCjU3N+vp06cWA9eqqio5ODgoJSVlwnpLdz7a2tpqZGRkWkfJAQAA/rQ3b97IZDIpJSVFtra2Y+rc3d0VHx+vmpoajY6OmssdHR2VnZ2tvLw8DQ4OWuw/JydH7e3t4xaiv2Vtba2ysjLV1dUpLi5OJpNJpaWlY+K57xGvAfgXkGgEgG+Mjo6qsbFRJpNJ69evlyQlJSXJZDKZdwy+evVKFy9e1M6dOy321dTUpPb2dlVXV2vt2rUKDg5WeXm5rl69qlu3bpmfGx4eVnl5uVavXq2AgAA1NjaqtbVVtbW1ioiIkNFo1KZNmxQVFWVu8+nTJ508eVKBgYFas2aNdu/eraampknn0tXVJaPRqNmzZ//Q/9HV1WUex1JSFQAA4G/R1dWl0dFR+fr6Tljv6+urt2/fqre3d0x5SkqKbGxsdPToUYv9L1q0SHv37lV2drbFxJ6vr69SU1N17tw55eTkyMfHZ8p5E68B+K8j0QgAki5cuCAHBwfZ2NgoKipKsbGxysnJkSStW7dO/v7+KisrkyRVVlbK09NTISEhFvvs7OzUkiVLxtzR6OfnJ2dnZ3V2dprLPD095ebmZv7e1tYmDw8Pi8GonZ2dvLy8zN8XLlxovgB8It+u2E/l3bt3cnBwkJ2dnZYvX64FCxaoqqpq2u0BAAD+Bj8S/0hfrtLJy8tTUVGRXr9+bfHZ/fv3q7e3V2fOnJn0mYGBAdXU1MjOzk7Xrl37pfMlXgPwtyLRCACSwsLC1NbWpq6uLr1//15lZWWyt7c31yclJam0tFTSl2PTO3bskMFg+CVjfzuOpHFHfCby/Uq3wWCwGJz6+Pjo0aNHU15wLklz585VW1ub7t27p8HBQTU3N5uTno6OjpK+BLff6+vrk5OT05T9AwAA/E7e3t4yGAxjFna/1dnZKRcXlzELvV8lJCTI09NT+fn5FsdwdnZWVlaWcnNzNTQ0NOEzGRkZsrGx0Y0bN9TY2Kjy8nKLfRKvAfgXkGgEAH1J9nl7e2vp0qWaNWvWuPqEhAR1d3eruLhYHR0dYy7jnoyvr696enrU09NjLuvo6FBfX5/8/PwmbRcQEKBnz57p/v37P/djJhAXF6eBgQGVlJRMWN/X12f+bGVlJW9vbxmNxnFJT1dXV82fP1937twZU97f368HDx5MeSQIAADgd5s3b54iIiJUUlIy7uUoL1++VFVVlWJjYydcNLaystLhw4d14sQJPXnyxOI4e/bskZWVlY4dOzaurqGhQadPn1ZZWZlWrVql/Px8paamWnx5H/EagH8BiUYAmAYXFxdFR0crIyNDGzZskIeHx5RtwsPDtXLlSsXHx+vu3btqbW3Vtm3bFBoaqsDAwEnbhYaGKiQkRDExMWpoaNDjx4916dIl1dfX//T8g4ODlZmZqX379ikzM1MtLS3q7u5WU1OTNm/ebD4WPh3p6ek6dOiQqqqq9PDhQ7W2tio+Pl5ubm6Kjo7+6TkCAAD8KsePH9fHjx8VGRmp5uZm9fT0qL6+XhEREVq8eLEOHjw4aduNGzcqODhYp06dsjiGjY2NcnNzVVxcPKa8v79fiYmJysjIUFBQkCQpLS1Nfn5+Sk5OnrQ/4jUA/wISjQAwTYmJiRoeHp7yJTBfGQwG1dXVycXFRSEhIQoPD5fRaFRNTc2Ubc+fP6+goCBt3bpVfn5+yszM1OfPn2c0/8LCQlVXV+vmzZuKjIyUv7+/0tPTFRAQMK0dml9lZmbqwIEDKiwsVEBAgGJiYmRvb68rV65M69g3AADA77Zs2TLdvn1bRqNRW7ZskZeXl5KTkxUWFqaWlha5urpabF9YWKgPHz5MOc727dtlNBrHlKWmpsrJycl837f0ZQfi2bNndfnyZYtHqInXAPzXGUZ/9IZcAPg/VVFRobS0ND1//lxz5sz509MBAAAAAOCvMv4iMgDAGENDQ3rx4oUKCgq0a9cukowAAAAAAEyAo9MAMIUjR45oxYoVcnd3V1ZW1p+eDgAAAAAAfyWOTgMAAAAAAACYMXY0AgAAAAAAAJgxEo0AAAAAAAAAZoxEIwAAAAAAAIAZI9EIAAAAAAAAYMZINAIAAAAAAACYMRKNAAAAAAAAAGaMRCMAAAAAAACAGSPRCAAAAAAAAGDGSDQCAAAAAAAAmLH/AdVxUJyGIWHGAAAAAElFTkSuQmCC\n",
|
|
"text/plain": [
|
|
"<Figure size 1600x1200 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"import matplotlib\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"import os\n",
|
|
"\n",
|
|
"\n",
|
|
"# Compute average inference time + std\n",
|
|
"time_results = {k: np.mean(v.model_inference_time) * 1e3 for k, v in results.items()}\n",
|
|
"time_results_std = np.std([v.model_inference_time for v in results.values()]) * 1000\n",
|
|
"\n",
|
|
"plt.rcdefaults()\n",
|
|
"fig, ax = plt.subplots(figsize=(16, 12))\n",
|
|
"ax.set_ylabel(\"Avg Inference time (ms)\")\n",
|
|
"ax.set_title(\"Average inference time (ms) for each provider\")\n",
|
|
"ax.bar(time_results.keys(), time_results.values(), yerr=time_results_std)\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Quantization support from transformers\n",
|
|
"\n",
|
|
"Quantization enables the use of integers (_instead of floatting point_) arithmetic to run neural networks models faster. From a high-level point of view, quantization works as mapping the float32 ranges of values as int8 with the less loss in the performances of the model.\n",
|
|
"\n",
|
|
"Hugging Face provides a conversion tool as part of the transformers repository to easily export quantized models to ONNX Runtime. For more information, please refer to the following: \n",
|
|
"\n",
|
|
"- [Hugging Face Documentation on ONNX Runtime quantization supports](https://huggingface.co/transformers/master/serialization.html#quantization)\n",
|
|
"- [Intel's Explanation of Quantization](https://nervanasystems.github.io/distiller/quantization.html)\n",
|
|
"\n",
|
|
"With this method, the accuracy of the model remains at the same level than the full-precision model. If you want to see benchmarks on model performances, we recommand reading the [ONNX Runtime notebook](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/notebooks/Bert-GLUE_OnnxRuntime_quantization.ipynb) on the subject."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Benchmarking PyTorch quantized model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 32,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 100/100 [00:01<00:00, 90.15it/s]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch \n",
|
|
"\n",
|
|
"# Quantize\n",
|
|
"model_pt_quantized = torch.quantization.quantize_dynamic(\n",
|
|
" model_pt.to(\"cpu\"), {torch.nn.Linear}, dtype=torch.qint8\n",
|
|
")\n",
|
|
"\n",
|
|
"# Warm up \n",
|
|
"model_pt_quantized(**model_inputs)\n",
|
|
"\n",
|
|
"# Benchmark PyTorch quantized model\n",
|
|
"time_buffer = []\n",
|
|
"for _ in trange(100):\n",
|
|
" with track_infer_time(time_buffer):\n",
|
|
" model_pt_quantized(**model_inputs)\n",
|
|
" \n",
|
|
"results[\"PyTorch CPU Quantized\"] = OnnxInferenceResult(\n",
|
|
" time_buffer,\n",
|
|
" None\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Benchmarking ONNX quantized model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 33,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"As of onnxruntime 1.4.0, models larger than 2GB will fail to quantize due to protobuf constraint.\n",
|
|
"This limitation will be removed in the next release of onnxruntime.\n",
|
|
"Quantized model has been written at bert.onnx: ✔\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Tracking inference time on CPUExecutionProvider with quantized model: 100%|██████████| 100/100 [00:00<00:00, 237.49it/s]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from transformers.convert_graph_to_onnx import quantize\n",
|
|
"\n",
|
|
"# Transformers allow you to easily convert float32 model to quantized int8 with ONNX Runtime\n",
|
|
"quantized_model_path = quantize(Path(\"bert.opt.onnx\"))\n",
|
|
"\n",
|
|
"# Then you just have to load through ONNX runtime as you would normally do\n",
|
|
"quantized_model = create_model_for_provider(quantized_model_path.as_posix(), \"CPUExecutionProvider\")\n",
|
|
"\n",
|
|
"# Warm up the overall model to have a fair comparaison\n",
|
|
"outputs = quantized_model.run(None, inputs_onnx)\n",
|
|
"\n",
|
|
"# Evaluate performances\n",
|
|
"time_buffer = []\n",
|
|
"for _ in trange(100, desc=f\"Tracking inference time on CPUExecutionProvider with quantized model\"):\n",
|
|
" with track_infer_time(time_buffer):\n",
|
|
" outputs = quantized_model.run(None, inputs_onnx)\n",
|
|
"\n",
|
|
"# Store the result\n",
|
|
"results[\"ONNX CPU Quantized\"] = OnnxInferenceResult(\n",
|
|
" time_buffer, \n",
|
|
" quantized_model_path\n",
|
|
") "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Show the inference performance of each providers "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 34,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 676
|
|
},
|
|
"colab_type": "code",
|
|
"id": "dj-rS8AcqRZQ",
|
|
"outputId": "b4bf07d1-a7b4-4eff-e6bd-d5d424fd17fb"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABRoAAAPeCAYAAABjjKazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnA0lEQVR4nOzdd3iV9fn48fuEEWaCDBmCshwgai22igNQUcSJooKruFvrVmqlDkSpOKrSWhVHi6MouKBoiwsRxVmx7lZBQVFQFIVA0KDk/P7ol/yMAUz4JCbR1+u6znVxnvOc89w5SY7y5hmZbDabDQAAAACABDnVPQAAAAAAUPsJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AQK3SsWPHOProo9f7+VdeeWV07tw56tSpEz/5yU8qba4fkieeeCIymUw88cQT1T1KKcuXL48NN9wwxo8f/71u99xzz43tt9++Qs/5of+crf4Zuffee6t7lEpz6623RiaTiXnz5n3nuqmfQwDwQyU0AkANdP3110cmk6lw3GDdHnnkkTjnnHNip512inHjxsWll15a3SNVq+uvvz5uvfXW6h6j3P74xz9G06ZNY8iQId/rds8444x45ZVXYsqUKeVa388ZAPBjVbe6BwAAyho/fnx07NgxXnjhhZgzZ0507dq1ukeqMd56663IyVm/fyt9/PHHIycnJ/7yl79E/fr1K3my2uf666+Pli1bltkzq3fv3vHFF1/UqPfoq6++ij/+8Y9x5plnRp06db7Xbbdp0yYOOOCA+MMf/hD777//d67v56x2Ouqoo2LIkCGRm5tb3aMAQK1lj0YAqGHmzp0bzzzzTFx99dXRqlWr7/0w0YiI4uLi+PLLL7/37ZZHbm5u1KtXb72eu2jRomjYsGGlxp8VK1ZU2mvVFDk5OdGgQYP1DrpV4cEHH4xPPvkkDj300GrZ/qGHHhozZ86Md9999zvXreyfs2w2G1988UWlvNYPQVV9PtWpUycaNGgQmUym0l97Xb7++utYuXLl97pNAKgqNef/HgGAiPjf3owbbLBB7LPPPnHwwQeXCo1fffVVNG/ePI455pgyzysoKIgGDRrEsGHDSpYVFRXFiBEjomvXrpGbmxsdOnSIc845J4qKiko9N5PJxCmnnBLjx4+PLbfcMnJzc+Ohhx6KiIg//OEPseOOO0aLFi2iYcOG0bNnzzWel+2LL76I0047LVq2bBlNmzaN/fffPz788MPIZDJx0UUXlVr3ww8/jGOPPTZat24dubm5seWWW8Zf//rXcr0/3z432urzqj399NNx1llnRatWraJx48Zx4IEHxieffFLqaxw3blwUFhZGJpOJTCZT6rDhv/3tb9GzZ89o2LBhNG/ePIYMGRLz588vte2+fftGjx49YtasWdG7d+9o1KhR/O53v1uv93ry5MnRo0ePkq9/9fv97ffpuOOOi3bt2kVubm506tQpTjrppFJRYsmSJXHGGWdEhw4dIjc3N7p27RqXX355FBcXf+f7+MYbb8SMGTNK3o++fftGxJrP0bj6a3/11VejT58+0ahRo+jatWvJz8KMGTNi++23j4YNG8bmm28ejz322Bq/nvX9vk+ePDk6duwYXbp0KbX86KOPjiZNmsT7778f++67bzRp0iQ22mijuO666yIi4rXXXovddtstGjduHJtssknceeedpZ7/1VdfxciRI2PTTTeNBg0aRIsWLWLnnXeORx99tNR6/fr1i4iIv//97+ucc10/Z19//XVccskl0aVLl8jNzY2OHTvG7373uzI/Ix07dox99903Hn744dhuu+2iYcOGceONN65zu88//3zstddekZ+fH40aNYo+ffrE008/XWqd9957L37961/H5ptvHg0bNowWLVrEIYccssZzEi5ZsiTOPPPM6NixY+Tm5kb79u3jF7/4RXz66ael1isuLo7f//730b59+2jQoEHsvvvuMWfOnHXOGhFx0UUXRSaTif/+979x6KGHRl5eXrRo0SJOP/30MhFxXZ9P//73v2PAgAGRl5cXTZo0id133z2ee+65kue++OKLkclk4rbbbiszw8MPPxyZTCYefPDBiFjzORqz2WyMGjUq2rdvH40aNYpdd9013njjjTV+TeX5XZw3b15kMpn4wx/+EGPGjCn5WXjzzTe/8z0DgNrAodMAUMOMHz8+DjrooKhfv34cdthhccMNN8S//vWv+NnPfhb16tWLAw88MO6///648cYbS+0xNXny5CgqKio5f11xcXHsv//+MXPmzDjxxBOjW7du8dprr8U111wTb7/9dkyePLnUdh9//PG4++6745RTTomWLVtGx44dI+J/58Xbf//944gjjoiVK1fGhAkT4pBDDokHH3ww9tlnn5LnH3300XH33XfHUUcdFTvssEPMmDGj1OOrffzxx7HDDjuUxINWrVrF1KlT47jjjouCgoI444wz1ut9O/XUU2ODDTaIESNGxLx582LMmDFxyimnxMSJEyMi4o477oibbropXnjhhbjlllsiImLHHXeMiIjf//73ccEFF8Shhx4axx9/fHzyySdx7bXXRu/evePf//53NGvWrGQ7ixcvjgEDBsSQIUPiyCOPjNatW1f4vZ45c2bcf//98etf/zqaNm0af/rTn2LQoEHx/vvvR4sWLSIiYsGCBfHzn/88lixZEieeeGJsscUW8eGHH8a9994bK1asiPr168eKFSuiT58+8eGHH8Yvf/nL2HjjjeOZZ56J4cOHx8KFC2PMmDFrfb/GjBkTp556ajRp0iTOO++8iIho3br1Ot/jzz//PPbdd98YMmRIHHLIIXHDDTfEkCFDYvz48XHGGWfEr371qzj88MPjyiuvjIMPPjjmz58fTZs2jYj07/szzzwTP/3pT9f42KpVq2LAgAHRu3fvuOKKK2L8+PFxyimnROPGjeO8886LI444Ig466KAYO3Zs/OIXv4hevXpFp06dIuJ/wWv06NFx/PHHx89//vMoKCiIF198MV566aXYY489SraRn58fXbp0iaeffjrOPPPMtc65rp+z448/Pm677bY4+OCD4+yzz47nn38+Ro8eHf/5z39i0qRJpV7nrbfeisMOOyx++ctfxgknnBCbb775Wrf5+OOPx4ABA6Jnz54xYsSIyMnJiXHjxsVuu+0WTz31VPz85z+PiIh//etf8cwzz8SQIUOiffv2MW/evLjhhhuib9++8eabb0ajRo0i4n8X3dlll13iP//5Txx77LHx05/+ND799NOYMmVKfPDBB9GyZcuSbV922WWRk5MTw4YNi6VLl8YVV1wRRxxxRDz//PNrnfebDj300OjYsWOMHj06nnvuufjTn/4Un3/+edx+++1lvsZvfz698cYbscsuu0ReXl6cc845Ua9evbjxxhujb9++JeF7u+22i86dO8fdd98dQ4cOLfWaEydOjA022CD69++/1vkuvPDCGDVqVOy9996x9957x0svvRR77rlnmT0QK/q7OG7cuPjyyy/jxBNPjNzc3GjevHm53i8AqPGyAECN8eKLL2YjIvvoo49ms9lstri4ONu+ffvs6aefXrLOww8/nI2I7AMPPFDquXvvvXe2c+fOJffvuOOObE5OTvapp54qtd7YsWOzEZF9+umnS5ZFRDYnJyf7xhtvlJlpxYoVpe6vXLky26NHj+xuu+1WsmzWrFnZiMieccYZpdY9+uijsxGRHTFiRMmy4447Ltu2bdvsp59+WmrdIUOGZPPz88ts79s22WST7NChQ0vujxs3LhsR2X79+mWLi4tLlp955pnZOnXqZJcsWVKybOjQodnGjRuXer158+Zl69Spk/39739favlrr72WrVu3bqnlffr0yUZEduzYsaXWreh7Xb9+/eycOXNKlr3yyivZiMhee+21Jct+8YtfZHNycrL/+te/yrwHq7/OSy65JNu4cePs22+/Xerxc889N1unTp3s+++/X+a537Tllltm+/TpU2b59OnTsxGRnT59epmv/c477yxZ9t///rfkZ+e5554rWb76Z3TcuHEly1K+71999VU2k8lkzz777DKPDR06NBsR2UsvvbRk2eeff55t2LBhNpPJZCdMmFBm3m/+PG6zzTbZffbZZ63b/qY999wz261bt+9cb00/Zy+//HI2IrLHH398qeXDhg3LRkT28ccfL1m2ySabZCMi+9BDD33ntoqLi7Obbrpptn///qV+/lesWJHt1KlTdo899ii17NueffbZbERkb7/99pJlF154YTYisvfff/8at5fN/v+fkW7dumWLiopKHv/jH/+YjYjsa6+9ts65R4wYkY2I7P77719q+a9//etsRGRfeeWVkmVr+3waOHBgtn79+tl33nmnZNmCBQuyTZs2zfbu3btk2fDhw7P16tXLfvbZZyXLioqKss2aNcsee+yxJctWf5bMnTs3m81ms4sWLcrWr18/u88++5R6b3/3u99lI6LU51B5fxfnzp2bjYhsXl5edtGiRet8jwCgNnLoNADUIOPHj4/WrVvHrrvuGhH/O2Rw8ODBMWHChFi1alVEROy2227RsmXLkj31Iv63p9mjjz4agwcPLll2zz33RLdu3WKLLbaITz/9tOS22267RUTE9OnTS227T58+0b179zIzNWzYsNR2li5dGrvssku89NJLJctXH8b461//utRzTz311FL3s9ls3HfffbHffvtFNpstNVf//v1j6dKlpV63Ik488cRS51bbZZddYtWqVfHee++t83n3339/FBcXx6GHHlpqnjZt2sSmm25a5n3Kzc0tc+h6Rd/rfv36lToEeOutt468vLyS8/8VFxfH5MmTY7/99ovtttuuzMyrv8577rkndtlll9hggw1Kbbdfv36xatWqePLJJ7/rbauQJk2alLri8+abbx7NmjWLbt26lbpC+uo/r/56Ur/vn332WWSz2dhggw3Wus7xxx9f8udmzZrF5ptvHo0bNy51TsfV837zPIvNmjWLN954I2bPnv2dX//q93l9/POf/4yIiLPOOqvU8rPPPjsiIv7xj3+UWt6pU6d17mm32ssvvxyzZ8+Oww8/PBYvXlzyvhYWFsbuu+8eTz75ZMmhu9/8Xf7qq69i8eLF0bVr12jWrFmp9/++++6LbbbZJg488MAy2/v2+QuPOeaYUntW77LLLhER5TqXZUTEySefXOr+6s+M1e/Xat/+fFq1alU88sgjMXDgwOjcuXPJ8rZt28bhhx8eM2fOjIKCgoiIGDx4cHz11Vdx//33l6z3yCOPxJIlS0p9Zn7bY489FitXroxTTz211Ne9pr1vK/q7OGjQoGjVqtVatw0AtZVDpwGghli1alVMmDAhdt1115g7d27J8u233z6uuuqqmDZtWuy5555Rt27dGDRoUNx5551RVFQUubm5cf/998dXX31V6i/Ns2fPjv/85z9r/cvsokWLSt1ffSjptz344IMxatSoePnll0udS+6bf/F+7733Iicnp8xrfPtq2Z988kksWbIkbrrpprjpppvKNVd5bbzxxqXur45Sn3/++TqfN3v27Mhms7Hpppuu8fFvX3hmo402KnORj4q+19+edfW8q2f95JNPoqCgIHr06PGds7/66qvl3m6q9u3blwlN+fn50aFDhzLLIqLU11MZ3/dsNrvG5Q0aNCjzHuTn56913m/+TFx88cVxwAEHxGabbRY9evSIvfbaK4466qjYeuut17j99b1QyOrfkW//TrRp0yaaNWtWJoiv7ffx21YH0m8fFvxNS5cujQ022CC++OKLGD16dIwbNy4+/PDDUu/n0qVLS/78zjvvxKBBg8q1/fX9vVvt2793Xbp0iZycnDLnjfz2+/HJJ5/EihUr1nhIebdu3aK4uDjmz58fW265ZWyzzTaxxRZbxMSJE+O4446LiP8dNt2yZcuSfwxYk9Xfk2/P2KpVqzLRu6K/i+X9/gJAbSM0AkAN8fjjj8fChQtjwoQJMWHChDKPjx8/Pvbcc8+IiBgyZEjceOONMXXq1Bg4cGDcfffdscUWW8Q222xTsn5xcXFstdVWcfXVV69xe9+OQ9/c22m1p556Kvbff//o3bt3XH/99dG2bduoV69ejBs3rsxFNcpj9Z5VRx555FrDyJoCT3nUqVNnjcvXFqe+OVMmk4mpU6eu8TWaNGlS6v6a3qeKvtfrO+uatrvHHnvEOeecs8bHN9tsswq93ndZ29zf9fWkft+bN28emUxmrfFqfeeKiOjdu3e888478fe//z0eeeSRuOWWW+Kaa66JsWPHltpLMuJ/8eyb5ydcH+UNlWv6OVuT1e/tlVdeGT/5yU/WuM7qn+FTTz01xo0bF2eccUb06tUr8vPzI5PJxJAhQ77z4kFrU1k/y6ut7f0p7/uxNoMHD47f//738emnn0bTpk1jypQpcdhhh0XdupXz16GK/i6mfj0AUFMJjQBQQ4wfPz423HDDkqvlftP9998fkyZNirFjx0bDhg2jd+/e0bZt25g4cWLsvPPO8fjjj5dc0GO1Ll26xCuvvBK77777eu+Fdd9990WDBg3i4Ycfjtzc3JLl48aNK7XeJptsEsXFxTF37txSe/98++qzrVq1iqZNm8aqVatKruJb3bp06RLZbDY6deq03mGuMt7rb2rVqlXk5eXF66+//p3bXb58+Xq/l5Uxa3mkft/r1q0bXbp0KbWnb2VafSX3Y445JpYvXx69e/eOiy66qExonDt3bqmYXxGrf0dmz54d3bp1K1n+8ccfx5IlS2KTTTZZr9ddfQh+Xl7ed7639957bwwdOjSuuuqqkmVffvllLFmypMxrftfPXmWZPXt2qb375syZE8XFxSUXo1qbVq1aRaNGjeKtt94q89h///vfyMnJKRX4Bw8eHCNHjoz77rsvWrduHQUFBaVOA7Amq78ns2fPLnV49ieffFImeqf+LgLAD4VzNAJADfDFF1/E/fffH/vuu28cfPDBZW6nnHJKLFu2LKZMmRIRETk5OXHwwQfHAw88EHfccUd8/fXXZc41duihh8aHH34YN9988xq3V1hY+J1z1alTJzKZTMn5ISMi5s2bV+YqyqvPJXf99deXWn7ttdeWeb1BgwbFfffdt8aQ8cknn3znTJXtoIMOijp16sTIkSPL7IWVzWZj8eLF3/kalfFef1NOTk4MHDgwHnjggXjxxRfLPL56zkMPPTSeffbZePjhh8uss2TJkvj666/XuZ3GjRuXiUxVoTK+77169Vrje5Hq29/fJk2aRNeuXUudJiDif4cWv/POOyVXkK6ovffeOyKizNWHV+8Fu6YrtJdHz549o0uXLvGHP/whli9fXubxb763derUKfMzfu2115b6/Y743/kDX3nllTJXwo5Y/z0V1+bb/7Cy+jNjwIAB63xenTp1Ys8994y///3vpQ6z/vjjj+POO++MnXfeOfLy8kqWd+vWLbbaaquYOHFiTJw4Mdq2bRu9e/de5zb69esX9erVi2uvvbbU172mq7mn/i4CwA+FPRoBoAaYMmVKLFu2LPbff/81Pr7DDjtEq1atYvz48SVBcfDgwXHttdfGiBEjYquttiq1l1RExFFHHRV33313/OpXv4rp06fHTjvtFKtWrYr//ve/cffdd8fDDz+8xguNfNM+++wTV199dey1115x+OGHx6JFi+K6666Lrl27xquvvlqyXs+ePWPQoEExZsyYWLx4ceywww4xY8aMePvttyOi9J5zl112WUyfPj223377OOGEE6J79+7x2WefxUsvvRSPPfZYfPbZZ+v1Hq6vLl26xKhRo2L48OExb968GDhwYDRt2jTmzp0bkyZNihNPPDGGDRu2zteojPf62y699NJ45JFHok+fPnHiiSdGt27dYuHChXHPPffEzJkzo1mzZvGb3/wmpkyZEvvuu28cffTR0bNnzygsLIzXXnst7r333pg3b946D/Xt2bNn3HDDDTFq1Kjo2rVrbLjhhus8Z12K1O/7AQccEHfccUe8/fbblXpIePfu3aNv377Rs2fPaN68ebz44otx7733ximnnFJqvcceeyyy2WwccMAB67WdbbbZJoYOHRo33XRTLFmyJPr06RMvvPBC3HbbbTFw4MCSC0BVVE5OTtxyyy0xYMCA2HLLLeOYY46JjTbaKD788MOYPn165OXlxQMPPBAREfvuu2/ccccdkZ+fH927d49nn302HnvssWjRokWp1/zNb34T9957bxxyyCFx7LHHRs+ePeOzzz6LKVOmxNixY9d7r841mTt3buy///6x1157xbPPPht/+9vf4vDDDy/XNkaNGhWPPvpo7LzzzvHrX/866tatGzfeeGMUFRXFFVdcUWb9wYMHx4UXXhgNGjSI4447LnJy1r3PRatWrWLYsGExevTo2HfffWPvvfeOf//73zF16tQyv1epv4sA8EMhNAJADTB+/Pho0KBB7LHHHmt8PCcnJ/bZZ58YP358LF68OFq0aBE77rhjdOjQIebPn7/GK6fm5OTE5MmT45prronbb789Jk2aFI0aNYrOnTvH6aefXq5Ys9tuu8Vf/vKXuOyyy+KMM86ITp06xeWXXx7z5s0rFRojIm6//fZo06ZN3HXXXTFp0qTo169fTJw4MTbffPNo0KBByXqtW7eOF154IS6++OK4//774/rrr48WLVrElltuGZdffnkF37nKce6558Zmm20W11xzTYwcOTIi/ndexT333HOt8febKuO9/raNNtoonn/++bjgggti/PjxUVBQEBtttFEMGDAgGjVqFBERjRo1ihkzZsSll14a99xzT9x+++2Rl5cXm222WYwcObLkoixrc+GFF8Z7770XV1xxRSxbtiz69OlTZaEx9fu+3377RcuWLePuu++O888/v9LmOu2002LKlCnxyCOPRFFRUWyyySYxatSo+M1vflNqvXvuuSd23nnnUlcLr6hbbrklOnfuHLfeemtMmjQp2rRpE8OHD48RI0YkfQ19+/aNZ599Ni655JL485//HMuXL482bdrE9ttvH7/85S9L1vvjH/8YderUifHjx8eXX34ZO+20Uzz22GNlrm7dpEmTeOqpp2LEiBExadKkuO2222LDDTeM3XffPdq3b58067dNnDgxLrzwwjj33HOjbt26ccopp8SVV15ZruduueWW8dRTT8Xw4cNj9OjRUVxcHNtvv3387W9/K3UV9NUGDx4c559/fqxYsWKdV5v+plGjRkWDBg1i7NixJaH8kUceKbMHaurvIgD8UGSylX38AwDA/3n55Zdj2223jb/97W9xxBFHVPc41HKXXHJJjBs3LmbPnr3Wi5BUhY8++ig6deoUEyZMWO89GintoosuipEjR8Ynn3xiTz8A+AFxjkYAoFJ88cUXZZaNGTMmcnJyvvNcaFAeZ555ZixfvnyNV2WvSmPGjImtttpKZAQA+A4OnQYAKsUVV1wRs2bNil133TXq1q0bU6dOjalTp8aJJ55Y6uqvsL6aNGkSixYt+t63e9lll33v2wQAqI2ERgCgUuy4447x6KOPxiWXXBLLly+PjTfeOC666KI477zzqns0AADge+AcjQAAAABAMudoBAAAAACSCY0AAAAAQLIf/Dkai4uLY8GCBdG0adPIZDLVPQ4AAAAA1CrZbDaWLVsW7dq1i5ycte+3+IMPjQsWLHClSwAAAABINH/+/Gjfvv1aH//Bh8amTZtGxP/eiLy8vGqeBgAAAABql4KCgujQoUNJZ1ubH3xoXH24dF5entAIAAAAAOvpu05L6GIwAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExohBqosLAwMplMZDKZKCwsrO5xAAAAAL6T0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgWbWGxhtuuCG23nrryMvLi7y8vOjVq1dMnTq15PEvv/wyTj755GjRokU0adIkBg0aFB9//HE1TgwAAAAArEm1hsb27dvHZZddFrNmzYoXX3wxdttttzjggAPijTfeiIiIM888Mx544IG45557YsaMGbFgwYI46KCDqnNkAAAAAGANMtlsNlvdQ3xT8+bN48orr4yDDz44WrVqFXfeeWccfPDBERHx3//+N7p16xbPPvts7LDDDuV6vYKCgsjPz4+lS5dGXl5eVY4OlaawsDCaNGkSERHLly+Pxo0bV/NEAAAAwI9VeftajTlH46pVq2LChAlRWFgYvXr1ilmzZsVXX30V/fr1K1lniy22iI033jieffbZapwUAAAAAPi2utU9wGuvvRa9evWKL7/8Mpo0aRKTJk2K7t27x8svvxz169ePZs2alVq/devW8dFHH6319YqKiqKoqKjkfkFBQVWNDgAAAAD8n2rfo3HzzTePl19+OZ5//vk46aSTYujQofHmm2+u9+uNHj068vPzS24dOnSoxGkBAAAAgDWp9tBYv3796Nq1a/Ts2TNGjx4d22yzTfzxj3+MNm3axMqVK2PJkiWl1v/444+jTZs2a3294cOHx9KlS0tu8+fPr+KvAAAAAACo9tD4bcXFxVFUVBQ9e/aMevXqxbRp00oee+utt+L999+PXr16rfX5ubm5kZeXV+oGAAAAAFStaj1H4/Dhw2PAgAGx8cYbx7Jly+LOO++MJ554Ih5++OHIz8+P4447Ls4666xo3rx55OXlxamnnhq9evUq9xWnAQAAAIDvR7WGxkWLFsUvfvGLWLhwYeTn58fWW28dDz/8cOyxxx4REXHNNddETk5ODBo0KIqKiqJ///5x/fXXV+fIAAAAAMAaZLLZbLa6h6hKBQUFkZ+fH0uXLnUYNbVGYWFhNGnSJCIili9fHo0bN67miQAAAIAfq/L2tRp3jkYAAAAAoPYRGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEhWraFx9OjR8bOf/SyaNm0aG264YQwcODDeeuutUuv07ds3MplMqduvfvWrapoYAAAAAFiTag2NM2bMiJNPPjmee+65ePTRR+Orr76KPffcMwoLC0utd8IJJ8TChQtLbldccUU1TQwAAAAArEnd6tz4Qw89VOr+rbfeGhtuuGHMmjUrevfuXbK8UaNG0aZNm+97PAAAAACgnGrUORqXLl0aERHNmzcvtXz8+PHRsmXL6NGjRwwfPjxWrFhRHeMBAAAAAGtRrXs0flNxcXGcccYZsdNOO0WPHj1Klh9++OGxySabRLt27eLVV1+N3/72t/HWW2/F/fffv8bXKSoqiqKiopL7BQUFVT47AAAAAPzY1ZjQePLJJ8frr78eM2fOLLX8xBNPLPnzVlttFW3bto3dd9893nnnnejSpUuZ1xk9enSMHDmyyucFAAAAAP6/GnHo9CmnnBIPPvhgTJ8+Pdq3b7/OdbfffvuIiJgzZ84aHx8+fHgsXbq05DZ//vxKnxcAAAAAKK1a92jMZrNx6qmnxqRJk+KJJ56ITp06fedzXn755YiIaNu27Rofz83Njdzc3MocEwAAAAD4DtUaGk8++eS488474+9//3s0bdo0Pvroo4iIyM/Pj4YNG8Y777wTd955Z+y9997RokWLePXVV+PMM8+M3r17x9Zbb12dowMAAAAA35DJZrPZatt4JrPG5ePGjYujjz465s+fH0ceeWS8/vrrUVhYGB06dIgDDzwwzj///MjLyyvXNgoKCiI/Pz+WLl1a7udAdSssLIwmTZpERMTy5cujcePG1TwRAAAA8GNV3r5W7YdOr0uHDh1ixowZ39M0AAAAAMD6qhEXgwEAAAAAajehEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkq1vdA5Cu47n/qO4RqGTFK78s+XO3Cx6KnPoNqnEaqsK8y/ap7hEAAACgUtmjEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAFAhhYWFkclkIpPJRGFhYXWPAwAA1BBCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAklVraBw9enT87Gc/i6ZNm8aGG24YAwcOjLfeeqvUOl9++WWcfPLJ0aJFi2jSpEkMGjQoPv7442qaGAAAAABYk2oNjTNmzIiTTz45nnvuuXj00Ufjq6++ij333DMKCwtL1jnzzDPjgQceiHvuuSdmzJgRCxYsiIMOOqgapwYAAAAAvq1udW78oYceKnX/1ltvjQ033DBmzZoVvXv3jqVLl8Zf/vKXuPPOO2O33XaLiIhx48ZFt27d4rnnnosddtihOsYGAAAAAL6lRp2jcenSpRER0bx584iImDVrVnz11VfRr1+/knW22GKL2HjjjePZZ5+tlhkBAAAAgLKqdY/GbyouLo4zzjgjdtppp+jRo0dERHz00UdRv379aNasWal1W7duHR999NEaX6eoqCiKiopK7hcUFFTZzAAAAADA/9SYPRpPPvnkeP3112PChAlJrzN69OjIz88vuXXo0KGSJgQAAAAA1qZGhMZTTjklHnzwwZg+fXq0b9++ZHmbNm1i5cqVsWTJklLrf/zxx9GmTZs1vtbw4cNj6dKlJbf58+dX5egAAAAAQFRzaMxms3HKKafEpEmT4vHHH49OnTqVerxnz55Rr169mDZtWsmyt956K95///3o1avXGl8zNzc38vLySt0AAAAAgKpVredoPPnkk+POO++Mv//979G0adOS8y7m5+dHw4YNIz8/P4477rg466yzonnz5pGXlxennnpq9OrVyxWnAQAAAKAGqdbQeMMNN0RERN++fUstHzduXBx99NEREXHNNddETk5ODBo0KIqKiqJ///5x/fXXf8+TAgAAAADrUq2hMZvNfuc6DRo0iOuuuy6uu+6672EiAAAAAGB91IiLwQAAAAAAtZvQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQCAH4TCwsLIZDKRyWSisLCwuscBgB8doREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMnqVmTlJUuWxKRJk+Kpp56K9957L1asWBGtWrWKbbfdNvr37x877rhjVc0JAAAAANRg5dqjccGCBXH88cdH27ZtY9SoUfHFF1/ET37yk9h9992jffv2MX369Nhjjz2ie/fuMXHixKqeGQAAAACoYcq1R+O2224bQ4cOjVmzZkX37t3XuM4XX3wRkydPjjFjxsT8+fNj2LBhlTooAAAAAFBzlSs0vvnmm9GiRYt1rtOwYcM47LDD4rDDDovFixdXynAAAAAAQO1QrkOnvysypq4PAAAAANRuFb7q9G233Rb/+Mc/Su6fc8450axZs9hxxx3jvffeq9ThAAAAAIDaocKh8dJLL42GDRtGRMSzzz4b1113XVxxxRXRsmXLOPPMMyt9QAAAAACg5ivXORq/af78+dG1a9eIiJg8eXIMGjQoTjzxxNhpp52ib9++lT0fAAAAAFALVHiPxiZNmpRc7OWRRx6JPfbYIyIiGjRoEF988UXlTgcAAAAA1AoV3qNxjz32iOOPPz623XbbePvtt2PvvfeOiIg33ngjOnbsWNnzAQAAAAC1QIX3aLzuuuuiV69e8cknn8R9991XcoXpWbNmxWGHHVbpAwIAAAAANV+F92hs1qxZ/PnPfy6zfOTIkZUyEAAAAABQ+1Q4NEZEfPnll/Hqq6/GokWLori4uGR5JpOJ/fbbr9KGAwAAAABqhwqHxoceeiiOOuqokgvCfFMmk4lVq1ZVymAAAAAAQO1R4XM0nnrqqXHooYfGwoULo7i4uNRNZAQAAACAH6cKh8aPP/44zjrrrGjdunVVzAMAAAAA1EIVPnT64IMPjieeeCK6dOlSFfMAEZFTv0Fs8tsHq3sMAAAAgHKrcGj885//HIccckg89dRTsdVWW0W9evVKPX7aaadV2nAAAAAAQO1Q4dB41113xSOPPBINGjSIJ554IjKZTMljmUxGaAQAAACAH6EKh8bzzjsvRo4cGeeee27k5FT4FI8AAAAAwA9QhUvhypUrY/DgwSIjAAAAAFCiwrVw6NChMXHixKqYBQAAAACopSp86PSqVaviiiuuiIcffji23nrrMheDufrqqyttOAAAAACgdqhwaHzttddi2223jYiI119/vdRj37wwDAAAAADw41Hh0Dh9+vSqmAMAAAAAqMVc0QUAAAAASFau0PirX/0qPvjgg3K94MSJE2P8+PFJQwEAAAAAtUu5Dp1u1apVbLnllrHTTjvFfvvtF9ttt120a9cuGjRoEJ9//nm8+eabMXPmzJgwYUK0a9cubrrppqqeGwAAAACoQcoVGi+55JI45ZRT4pZbbonrr78+3nzzzVKPN23aNPr16xc33XRT7LXXXlUyKAAAAABQc5X7YjCtW7eO8847L84777z4/PPP4/33348vvvgiWrZsGV26dHHFaQAAAAD4EavwVacjIjbYYIPYYIMNKnsWAAAAAKCWctVpAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkGy9QuPXX38djz32WNx4442xbNmyiIhYsGBBLF++vFKHAwAAAABqhwpfdfq9996LvfbaK95///0oKiqKPfbYI5o2bRqXX355FBUVxdixY6tiTgAAAACgBqvwHo2nn356bLfddvH5559Hw4YNS5YfeOCBMW3atEodDgAAAACoHSq8R+NTTz0VzzzzTNSvX7/U8o4dO8aHH35YaYMBAAAAALVHhfdoLC4ujlWrVpVZ/sEHH0TTpk0rZSgAAAAAoHapcGjcc889Y8yYMSX3M5lMLF++PEaMGBF77713Zc4GAAAAANQSFT50+qqrror+/ftH9+7d48svv4zDDz88Zs+eHS1btoy77rqrKmYEAAAAAGq4CofG9u3bxyuvvBITJkyIV199NZYvXx7HHXdcHHHEEaUuDgMAAAAA/HhUODRGRNStWzeOPPLIyp4FAAAAAKil1is0LliwIGbOnBmLFi2K4uLiUo+ddtpplTIYAAAAAFB7VDg03nrrrfHLX/4y6tevHy1atIhMJlPyWCaTERoBAAAA4EeowqHxggsuiAsvvDCGDx8eOTkVvmg1AAAAAPADVOHQuGLFihgyZIjICEC5dDz3H9U9ApWseOWXJX/udsFDkVO/QTVOQ2Wbd9k+1T0CAAC1VIVr4XHHHRf33HNPVcwCAAAAANRSFd6jcfTo0bHvvvvGQw89FFtttVXUq1ev1ONXX311pQ0HAAAAANQO6xUaH3744dh8880jIspcDAYAAAAA+PGpcGi86qqr4q9//WscffTRVTAOAAAAAFAbVfgcjbm5ubHTTjtVxSwAAAAAQC1V4dB4+umnx7XXXlsVswAAAAAAtVSFD51+4YUX4vHHH48HH3wwttxyyzIXg7n//vsrbTgAAAAAoHaocGhs1qxZHHTQQVUxCwAAAABQS1U4NI4bN64q5gAAAAAAarEKn6OxMj355JOx3377Rbt27SKTycTkyZNLPX700UdHJpMpddtrr72qZ1gAAAAAYK3KtUfjT3/605g2bVpssMEGse2220Ymk1nrui+99FK5N15YWBjbbLNNHHvssWs9HHuvvfYqtRdlbm5uuV8fAAAAAPh+lCs0HnDAASWB74ADDlhnaKyIAQMGxIABA9a5Tm5ubrRp06ZStgcAAAAAVI1yhcYRI0aU/Pmiiy6qqlnW6IknnogNN9wwNthgg9htt91i1KhR0aJFi+91BgAAAABg3Sp8jsbOnTvH4sWLyyxfsmRJdO7cuVKGWm2vvfaK22+/PaZNmxaXX355zJgxIwYMGBCrVq1a63OKioqioKCg1A0AAAAAqFoVvur0vHnz1hj6ioqK4oMPPqiUoVYbMmRIyZ+32mqr2HrrraNLly7xxBNPxO67777G54wePTpGjhxZqXMAAAAAAOtW7tA4ZcqUkj8//PDDkZ+fX3J/1apVMW3atOjUqVPlTvctnTt3jpYtW8acOXPWGhqHDx8eZ511Vsn9goKC6NChQ5XOBQAAAAA/duUOjQMHDoyIiEwmE0OHDi31WL169aJjx45x1VVXVepw3/bBBx/E4sWLo23btmtdJzc315WpAQAAAOB7Vu7QWFxcHBERnTp1in/961/RsmXL5I0vX7485syZU3J/7ty58fLLL0fz5s2jefPmMXLkyBg0aFC0adMm3nnnnTjnnHOia9eu0b9//+RtAwAAAACVp8LnaJw7d26lbfzFF1+MXXfdteT+6kOehw4dGjfccEO8+uqrcdttt8WSJUuiXbt2seeee8Yll1xij0UAAAAAqGEqHBorU9++fSObza718Ycffvh7nAYAAAAAWF851T0AAAAAAFD7CY0AAAAAQDKhEQAAAABItl6h8Z133onzzz8/DjvssFi0aFFEREydOjXeeOONSh0OAAAAAKgdKhwaZ8yYEVtttVU8//zzcf/998fy5csjIuKVV16JESNGVPqAAAAAAEDNV+HQeO6558aoUaPi0Ucfjfr165cs32233eK5556r1OEAAAAAgNqhwqHxtddeiwMPPLDM8g033DA+/fTTShkKAAAAAKhdKhwamzVrFgsXLiyz/N///ndstNFGlTIUAAAAAFC7VDg0DhkyJH7729/GRx99FJlMJoqLi+Ppp5+OYcOGxS9+8YuqmBEAAAAAqOEqHBovvfTS2GKLLaJDhw6xfPny6N69e/Tu3Tt23HHHOP/886tiRgAAAACghqtb0SfUr18/br755rjwwgvjtddei+XLl8e2224bm266aVXMBwAAAADUAhUOjat16NAhOnToUJmzAAAAAAC1VIUPnR40aFBcfvnlZZZfccUVccghh1TKUAAAAABA7VLh0Pjkk0/G3nvvXWb5gAED4sknn6yUoQAAAACA2qXCoXH58uVRv379Msvr1asXBQUFlTIUAAAAAFC7VDg0brXVVjFx4sQyyydMmBDdu3evlKEAAAAAgNqlwheDueCCC+Kggw6Kd955J3bbbbeIiJg2bVrcddddcc8991T6gAAAAABAzVfh0LjffvvF5MmT49JLL4177703GjZsGFtvvXU89thj0adPn6qYEQAAAACo4SocGiMi9tlnn9hnn30qexYAAAAAoJZar9AYEbFy5cpYtGhRFBcXl1q+8cYbJw8FAAAAANQuFQ6Ns2fPjmOPPTaeeeaZUsuz2WxkMplYtWpVpQ0HAAAAANQOFQ6NRx99dNStWzcefPDBaNu2bWQymaqYCwAAAACoRSocGl9++eWYNWtWbLHFFlUxDwAAAABQC+VU9Andu3ePTz/9tCpmAQAAAABqqQqHxssvvzzOOeeceOKJJ2Lx4sVRUFBQ6gYAAAAA/PhU+NDpfv36RUTE7rvvXmq5i8EAAAAAwI9XhUPj9OnTq2IOAAAAAKAWq3Bo7NOnT1XMAQAAAADUYhU+R2NExFNPPRVHHnlk7LjjjvHhhx9GRMQdd9wRM2fOrNThAAAAAIDaocKh8b777ov+/ftHw4YN46WXXoqioqKIiFi6dGlceumllT4gAAAAAFDzVTg0jho1KsaOHRs333xz1KtXr2T5TjvtFC+99FKlDgcAAAAA1A4VDo1vvfVW9O7du8zy/Pz8WLJkSWXMBAAAAADUMhUOjW3atIk5c+aUWT5z5szo3LlzpQwFAAAAANQuFQ6NJ5xwQpx++unx/PPPRyaTiQULFsT48eNj2LBhcdJJJ1XFjAAAAABADVe3ok8499xzo7i4OHbfffdYsWJF9O7dO3Jzc2PYsGFx6qmnVsWMAAAAAEANV6HQuGrVqnj66afj5JNPjt/85jcxZ86cWL58eXTv3j2aNGlSVTMCAAAAADVchUJjnTp1Ys8994z//Oc/0axZs+jevXtVzQUAAAAA1CIVPkdjjx494t13362KWQAAAACAWqrCoXHUqFExbNiwePDBB2PhwoVRUFBQ6gYAAAAA/PhU+GIwe++9d0RE7L///pHJZEqWZ7PZyGQysWrVqsqbDgAAAACoFSocGqdPn14VcwAAAAAAtViFQ2OfPn2qYg4AAAAAoBar8DkaIyKeeuqpOPLII2PHHXeMDz/8MCIi7rjjjpg5c2alDgcAAAAA1A4VDo333Xdf9O/fPxo2bBgvvfRSFBUVRUTE0qVL49JLL630AQEAAACAmm+9rjo9duzYuPnmm6NevXoly3faaad46aWXKnU4AAAAAKB2qHBofOutt6J3795llufn58eSJUsqYyYAAAAAoJapcGhs06ZNzJkzp8zymTNnRufOnStlKAAAAACgdqlwaDzhhBPi9NNPj+effz4ymUwsWLAgxo8fH8OGDYuTTjqpKmYEAAAAAGq4uhV9wrnnnhvFxcWx++67x4oVK6J3796Rm5sbw4YNi1NPPbUqZgQAAAAAarhyhcZXX301evToETk5OZHJZOK8886L3/zmNzFnzpxYvnx5dO/ePZo0aVLVswIAAAAANVS5Dp3edttt49NPP42IiM6dO8fixYujfv360b179/j5z38uMgIAAADAj1y5QmOzZs1i7ty5ERExb968KC4urtKhAAAAAIDapVyHTg8aNCj69OkTbdu2jUwmE9ttt13UqVNnjeu+++67lTogAAAAAFDzlSs03nTTTXHQQQfFnDlz4rTTTosTTjghmjZtWtWzAQAAAAC1RLmvOr3XXntFRMSsWbPi9NNPFxoBAAAAgBLlDo2rjRs3rirmAAAAAABqsQqHxsLCwrjsssti2rRpsWjRojIXhnGORgAAAAD48alwaDz++ONjxowZcdRRR5VcHAYAAAAA+HGrcGicOnVq/OMf/4iddtqpKuYBAAAAAGqhnIo+YYMNNojmzZtXxSwAAAAAQC1V4dB4ySWXxIUXXhgrVqyoinkAAAAAgFqowodOX3XVVfHOO+9E69ato2PHjlGvXr1Sj7/00kuVNhwAAAAAUDtUODQOHDiwCsYAAAAAAGqzCofGESNGVMUcAAAAAEAtVuFzNAIAAAAAfFu592jcYIMNIpPJfOd6n332WdJAAAAAAEDtU+7QOGbMmCocAwAAAACozcodGocOHVqVcwAAAAAAtZhzNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkKzcV51e7ayzzlrj8kwmEw0aNIiuXbvGAQccEM2bN08eDgAAAACoHSocGv/973/HSy+9FKtWrYrNN988IiLefvvtqFOnTmyxxRZx/fXXx9lnnx0zZ86M7t27V/rAAAAAAEDNU+FDpw844IDo169fLFiwIGbNmhWzZs2KDz74IPbYY4847LDD4sMPP4zevXvHmWeeWRXzAgAAAAA1UIVD45VXXhmXXHJJ5OXllSzLz8+Piy66KK644opo1KhRXHjhhTFr1qxKHRQAAAAAqLkqHBqXLl0aixYtKrP8k08+iYKCgoiIaNasWaxcuTJ9OgAAAACgVlivQ6ePPfbYmDRpUnzwwQfxwQcfxKRJk+K4446LgQMHRkTECy+8EJtttlllzwoAAABQKxUWFkYmk4lMJhOFhYXVPQ5UiQpfDObGG2+MM888M4YMGRJff/31/16kbt0YOnRoXHPNNRERscUWW8Qtt9xSuZMCAAAAADVWhUNjkyZN4uabb45rrrkm3n333YiI6Ny5czRp0qRknZ/85CeVNiAAAAAAUPNV+NDpv/3tb7FixYpo0qRJbL311rH11luXiowAAAAAwI9PhUPjmWeeGRtuuGEcfvjh8c9//jNWrVpVFXMBAAAAALVIhUPjwoULY8KECZHJZOLQQw+Ntm3bxsknnxzPPPNMVcwHAAAAANQCFQ6NdevWjX333TfGjx8fixYtimuuuSbmzZsXu+66a3Tp0qUqZgQAAAAAargKXwzmmxo1ahT9+/ePzz//PN577734z3/+U1lzAQAAAAC1SIX3aIyIWLFiRYwfPz723nvv2GijjWLMmDFx4IEHxhtvvFHZ8wEAAAAAtUCF92gcMmRIPPjgg9GoUaM49NBD44ILLohevXpVxWwAAAAAQC1R4dBYp06duPvuu6N///5Rp06dUo+9/vrr0aNHj0obDgAAAACoHSocGsePH1/q/rJly+Kuu+6KW265JWbNmhWrVq2qtOEAAAAAgNphvS8G8+STT8Zf/vKXuO+++6Jdu3Zx0EEHxXXXXVeZswEAQJXqeO4/qnsEKlHxyi9L/tztgocip36DapyGqjDvsn2qewQA1qFCofGjjz6KW2+9Nf7yl79EQUFBHHrooVFUVBSTJ0+O7t27V9WMAAAAAEANV+6rTu+3336x+eabx6uvvhpjxoyJBQsWxLXXXluVswEAAAAAtUS592icOnVqnHbaaXHSSSfFpptuWpUzAQAAAAC1TLn3aJw5c2YsW7YsevbsGdtvv338+c9/jk8//bQqZwMAAAAAaolyh8Yddtghbr755li4cGH88pe/jAkTJkS7du2iuLg4Hn300Vi2bFlVzgkAAAAA1GDlDo2rNW7cOI499tiYOXNmvPbaa3H22WfHZZddFhtuuGHsv//+VTEjAAAAAFDDVTg0ftPmm28eV1xxRXzwwQdx1113VdZMAAAAAEAtkxQaV6tTp04MHDgwpkyZUhkvBwAAAADUMpUSGgEAAACAHzehEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJqjU0Pvnkk7HffvtFu3btIpPJxOTJk0s9ns1m48ILL4y2bdtGw4YNo1+/fjF79uzqGRYAAAAAWKtqDY2FhYWxzTbbxHXXXbfGx6+44or405/+FGPHjo3nn38+GjduHP37948vv/zye54UAAAAAFiXutW58QEDBsSAAQPW+Fg2m40xY8bE+eefHwcccEBERNx+++3RunXrmDx5cgwZMuT7HBUAAAAAWIcae47GuXPnxkcffRT9+vUrWZafnx/bb799PPvss9U4GQAAAADwbdW6R+O6fPTRRxER0bp161LLW7duXfLYmhQVFUVRUVHJ/YKCgqoZEAAAAAAoUWP3aFxfo0ePjvz8/JJbhw4dqnskAAAAAPjBq7GhsU2bNhER8fHHH5da/vHHH5c8tibDhw+PpUuXltzmz59fpXMCAAAAADU4NHbq1CnatGkT06ZNK1lWUFAQzz//fPTq1Wutz8vNzY28vLxSNwAAAACgalXrORqXL18ec+bMKbk/d+7cePnll6N58+ax8cYbxxlnnBGjRo2KTTfdNDp16hQXXHBBtGvXLgYOHFh9QwMAAAAAZVRraHzxxRdj1113Lbl/1llnRUTE0KFD49Zbb41zzjknCgsL48QTT4wlS5bEzjvvHA899FA0aNCgukYGAAAAANagWkNj3759I5vNrvXxTCYTF198cVx88cXf41QAAAAAQEXV2HM0AgAAAAC1h9AIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGR1q3sAAKB2yanfIDb57YPVPQYAAFDD2KMRAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyWp0aLzooosik8mUum2xxRbVPRYAAAAA8C11q3uA77LlllvGY489VnK/bt0aPzIAAAAA/OjU+GpXt27daNOmTXWPAQAAAACsQ40+dDoiYvbs2dGuXbvo3LlzHHHEEfH++++vc/2ioqIoKCgodQMAAAAAqlaNDo3bb7993HrrrfHQQw/FDTfcEHPnzo1ddtklli1bttbnjB49OvLz80tuHTp0+B4nBgAAAIAfpxodGgcMGBCHHHJIbL311tG/f//45z//GUuWLIm77757rc8ZPnx4LF26tOQ2f/7873FiAAAAAPhxqvHnaPymZs2axWabbRZz5sxZ6zq5ubmRm5v7PU4FAAAAANToPRq/bfny5fHOO+9E27Ztq3sUAAAAAOAbanRoHDZsWMyYMSPmzZsXzzzzTBx44IFRp06dOOyww6p7NAAAAADgG2r0odMffPBBHHbYYbF48eJo1apV7LzzzvHcc89Fq1atqns0AAAAAOAbanRonDBhQnWPAAAAAACUQ40+dBoAAAAAqB2ERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJLVre4BAAAAgNI6nvuP6h6BSla88suSP3e74KHIqd+gGqehKsy7bJ/qHqHa2aMRAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMnqVvcAAAAAlSGnfoPY5LcPVvcYAPCjZY9GAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJCsVoTG6667Ljp27BgNGjSI7bffPl544YXqHgkAAAAA+IYaHxonTpwYZ511VowYMSJeeuml2GabbaJ///6xaNGi6h4NAAAAAPg/NT40Xn311XHCCSfEMcccE927d4+xY8dGo0aN4q9//Wt1jwYAAAAA/J8aHRpXrlwZs2bNin79+pUsy8nJiX79+sWzzz67xucUFRVFQUFBqRsAAAAAULUy2Ww2W91DrM2CBQtio402imeeeSZ69epVsvycc86JGTNmxPPPP1/mORdddFGMHDmyzPKlS5dGXl5elc4LAAAAsCaFhYXRpEmTiIhYvnx5NG7cuJongvIrKCiI/Pz87+xrNXqPxvUxfPjwWLp0aclt/vz51T0SAAAAAPzg1a3uAdalZcuWUadOnfj4449LLf/444+jTZs2a3xObm5u5Obmfh/jAQAAAAD/p0bv0Vi/fv3o2bNnTJs2rWRZcXFxTJs2rdSh1AAAAABA9arRezRGRJx11lkxdOjQ2G677eLnP/95jBkzJgoLC+OYY46p7tEAAAAAgP9T40Pj4MGD45NPPokLL7wwPvroo/jJT34SDz30ULRu3bq6RwMAAAAA/k+Nvup0ZSjvVXEAAAAAqoqrTlOb/WivOg0AAAAAfP+ERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACS1a3uAQAAAAB+6Bo3bhzZbLa6x4AqZY9GAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJhEYAAAAAIJnQCAAAAAAkExoBAAAAgGRCIwAAAACQTGgEAAAAAJIJjQAAAABAMqERAAAAAEgmNAIAAAAAyYRGAAAAACCZ0AgAAAAAJBMaAQAAAIBkQiMAAAAAkExoBAAAAACSCY0AAAAAQDKhEQAAAABIJjQCAAAAAMmERgAAAAAgmdAIAAAAACQTGgEAAACAZEIjAAAAAJBMaAQAAAAAkgmNAAAAAEAyoREAAAAASCY0AgAAAADJ6lb3AFUtm81GRERBQUE1TwIAAAAAtc/qrra6s63NDz40Llu2LCIiOnToUM2TAAAAAEDttWzZssjPz1/r45nsd6XIWq64uDgWLFgQTZs2jUwmU93jQLkVFBREhw4dYv78+ZGXl1fd4wCU4jMKqKl8PgE1mc8oaqtsNhvLli2Ldu3aRU7O2s/E+IPfozEnJyfat29f3WPAesvLy/MfIKDG8hkF1FQ+n4CazGcUtdG69mRczcVgAAAAAIBkQiMAAAAAkExohBoqNzc3RowYEbm5udU9CkAZPqOAmsrnE1CT+Yzih+4HfzEYAAAAAKDq2aMRAAAAAEgmNAIAAAAAyYRGAAAAACCZ0Ai1wK233hrNmjWr7jEAANbK/69UrXnz5kUmk4mXX365yrbhewhAKqGRH6Wjjz46MplMZDKZqF+/fnTt2jUuvvji+Prrr7/zubfeemvJc9d2mzdvXtV/EeV03333Rd++fSM/Pz+aNGkSW2+9dVx88cXx2WefRUTprycnJyfat28fxxxzTCxatCgi1v0/tX379o0zzjjje/xqgPnz58exxx4b7dq1i/r168cmm2wSp59+eixevLjUen379o1MJhMTJkwotXzMmDHRsWPHkvurPwP22muvUustWbIkMplMPPHEExER8corr0T9+vVjypQppda77777okGDBvH666+vc26fRVBx/n/l+/uM+Oyzz+KMM86ITTbZJOrXrx/t2rWLY489Nt5///3K/lLL5eijj46BAweWWtahQ4dYuHBh9OjRo1pmAvAPHpSH0MiP1l577RULFy6M2bNnx9lnnx0XXXRRXHnlld/5vMGDB8fChQtLbr169YoTTjih1LIOHTqUe46VK1emfBnrdN5558XgwYPjZz/7WUydOjVef/31uOqqq+KVV16JO+64o2S9vLy8WLhwYXzwwQdx8803x9SpU+Ooo46qsrmA9fPuu+/GdtttF7Nnz4677ror5syZE2PHjo1p06ZFr169Sv5CvlqDBg3i/PPPj6+++mqdr1u3bt147LHHYvr06WtdZ5tttokLL7wwTjzxxJKouWjRovjVr34VI0eOXOdffH0Wwfrz/ytV/xnx2WefxQ477BCPPfZYjB07NubMmRMTJkyIOXPmxM9+9rN49913k7dRGerUqRNt2rSJunXrVvcosE7+UfTlMq/tHzz4MREa+dHKzc2NNm3axCabbBInnXRS9OvXL6ZMmRKFhYWRl5cX9/6/9u49KKq6jQP4FwhmYRfUkAS5rYokhKhlURowWoRM5C2VEpXxMjmDNyogHWYSHcFmREfRTCcNQTGxF8wZU8mlAicMGnIHCgwQEErJ9YIOYqLwvH/4emK5LAJab/r9zOwMe87v/M5vD/Ds7zzP2bP/+Y9R+y+//BJqtRp37tyBo6Oj8rCysoKNjY3yvLm5GdOnT4dGo4GdnR1mzZqFP/74Q+knPj4eo0ePxq5duzBkyBCoVCoAd98oFy9ejEGDBkGlUsHHxwdHjhwxGkN2dja8vLyg0WiUE4+uFBYWIjExERs3bsSGDRswbtw4aLVaBAUFITMzExEREUpbMzMzODo6YvDgwQgJCcHy5cuh0+lw8+bNB3GoiegBWbJkCaysrPD1118jMDAQbm5uCAkJgU6nw++//464uDij9m+//TYaGhrw6aefmuxXrVZjwYIFWLlypcl2q1atgpubG5YsWQIAWLx4MYYPH47o6Ogut2EsIuobzlcefoyIi4vD+fPnodPpEBISAjc3NwQEBCA7OxuWlpZKzAMArVaLzZs3G20/evRoxMfHK883bdqEkSNHQq1Ww9XVFZGRkWhsbFTW37tap6vjFB8fj9TUVBw+fFhJanz33Xcdkhhtr3ht+7iXdLl16xaio6Ph7OwMtVoNPz8/ZV3bsbi5ucHGxgbTpk3rkAgi6ikWRXuHBQ96lDDRSPQ/1tbWaG5uhlqtxltvvYWUlBSj9SkpKZgxYwZsbW277KO1tRVTpkzBlStXkJubixMnTqCqqgphYWFG7SorK5GZmYmsrCzo9Xq0trYiJCQE33//Pfbt24fS0lJ89NFHsLCwULZpampCUlIS9u7di7y8PNTW1po8uU9PT4dGo0FkZGSn601djm5tbY3W1tb7+mgWEf09rly5guzsbERGRsLa2tponaOjI8LDw5GRkQERUZbb2dkhLi4Oa9euxY0bN0z2Hx8fj5KSkg5Ji7YsLCyUk9/Zs2cjOzsbe/bsMYpV7TEWET1YnK8YH4u+xojW1lYcOHAA4eHhcHR07NB/ZGQksrOzOyRHTDE3N0dycjJ++eUXpKam4ptvvkFsbKxRG1PHKTo6GrNmzVKSjxcuXMC4ceM67GfLli1GV6iuWLECTz31FEaMGAEAWLp0KU6dOoUDBw6guLgYM2fOxKRJk1BRUQEAKCgowMKFC7F06VLo9XpMmDAB69at69HxI2qPRdHeYcGDBY9HCRON9NgTEeh0OmRnZ2PixIkAgEWLFiE7O1sJtBcvXsTRo0exYMECk33l5OSgpKQE+/fvx3PPPQc/Pz+kpaUhNzcXP/74o9KuubkZaWlpGDNmDHx9faHT6VBYWIisrCwEBQVh6NChCA0NRUhIiLLN7du3sWPHDowdOxbPPvssli5dipycnC7HUlFRgaFDh8LS0rJHx6OiokLZj6mTFCL6e1VUVEBE4OXl1el6Ly8vXL16FQaDwWh5ZGQkVCoVNm3aZLL/wYMHY8WKFYiLizN50u7l5YWoqCh8/vnniI+Ph6enZ7fjZiwi6jvOVzpu9yBihMFgQENDg8nYKiKorKy87z6joqIwYcIEaLVaTJw4EevWrcPBgweN2pg6ThqNBtbW1srVrPeuSG2vX79+yvr8/Hzs3LkTWVlZcHR0RG1tLVJSUvDFF1/A398fw4YNQ3R0NF5++WUlOb1lyxZMmjQJsbGx8PT0xPLlyxEcHHzfr5OoPRZFe4cFDxY8HjVMNNJj68iRI9BoNFCpVAgJCUFYWJhSBXrhhRfwzDPPIDU1FQCwb98+uLu7IyAgwGSfZWVlcHV1Nbrnkbe3N/r374+ysjJlmbu7OxwcHJTner0eLi4uJk/YbWxsMGzYMOW5k5OTcj+QzrR9A+/OtWvXoNFoYGNjg6effhqDBg1Cenr6fW9PRH+fnvxvA3c/drl27VokJSXh0qVLJtt+8MEHMBgM+Oyzz7ps09jYiIyMDNjY2ODkyZMPdLyMRUQdcb7yl4cZI7obR2eJvq7odDq88sorcHZ2hq2tLebOnYvLly+jqalJadPT42TK6dOnMXfuXGzbtg3jx48HAJSUlKClpQWenp7QaDTKIzc3F2fPngVw9+/Az8/PqK+XXnqpV2MgAlgU7S0WPFjweNQw0UiPrQkTJkCv16OiogI3b95Eamoq1Gq1sn7RokXYs2cPgLsfQ5o/fz7MzMweyL7b7gdAh4pfZ9q/8ZmZmZmcFHt6eqKqqqrb+50AgK2tLfR6PX7++WfcuHEDeXl5yhuynZ0dgLuT+/YaGhrQr1+/bvsnor7z8PCAmZmZURKgrbKyMgwYMMAoKXDPnDlz4O7u3m2FuH///li1ahXWrFljdELcVkxMDFQqFfLz86HT6ZCWlmayT8Yior7hfOUvDyNGODg4dEiwtlVWVoYnnngCQ4YMAXD3KqH2r6ft2GtqahAaGgpfX19kZmaiqKgIH3/8MQDjL9Tp6XHqSn19PSZPnoxFixZh4cKFyvLGxkZYWFigqKgIer1eeZSVlWHLli093g9RT7Ao2jsseNzFgse/HxON9NhSq9Xw8PCAm5tbpzeznTNnDs6dO4fk5GSUlpYa3ZujK15eXqirq0NdXZ2yrLS0FA0NDfD29u5yO19fX/z2228oLy/v3YvpxOzZs9HY2Ijt27d3ur6hoUH52dzcHB4eHhg6dGiHk4gnn3wSAwcORFFRkdHy69evo7KystsKIRE9GPb29ggKCsL27ds73Aeovr4e6enpCAsL6zTBYG5ujvXr1+OTTz5BTU2Nyf0sW7YM5ubmnZ6InjhxArt27UJqaipGjRqFdevWISoqyuQXPTAWEfUN5ysNys8PI0aYm5tj1qxZ2L9/P+rr643W3bx5E9u3b8e0adOURKWDg4NRzLt+/Tqqq6uV50VFRWhtbcXGjRvx4osvwtPTE+fPn+/+QLRjZWWFlpYWk23+/PNPTJkyBSNGjOhwJdiYMWPQ0tKCixcvwsPDw+hx76OZXl5eKCgoMNruhx9+6PFYie5hUZQFDxY8CGCikahLAwYMwPTp0xETE4PXXnsNLi4u3W7z6quvYuTIkQgPD8dPP/2EwsJCzJs3D4GBgRg7dmyX2wUGBiIgIABvvvkmTpw4gerqahw7dgzHjx/v9fj9/PwQGxuL999/H7GxsTh16hTOnTuHnJwczJw5U/mY1f147733kJiYiPT0dJw9exaFhYUIDw+Hg4MDpk+f3usxElHPbNu2Dbdu3UJwcDDy8vJQV1eH48ePIygoCM7OzkhISOhy29dffx1+fn7YuXOnyX2oVCqsWbMGycnJRsuvX7+OhQsXIiYmBs8//zwA4N1334W3tzfeeeedLvtjLCJ6uDhf+UtvY0RCQgIcHR0RFBSEY8eOoa6uDnl5eQgODu5QeJk4cSL27t2LkydPoqSkBBEREUb3fvPw8MDt27exdetWVFVVYe/evdixY0ePj4tWq0VxcTF+/fVXXLp0qdMEyOLFi1FXV4fk5GQYDAbU19ejvr4ezc3N8PT0RHh4OObNm4esrCxUV1ejsLAQ69evx1dffQUAWL58OY4fP46kpCRUVFRg27ZtffpdErEoyoJHWyx4PMaE6DEUEREhU6ZM6bZdTk6OAJCDBw922SYwMFBWrFihPD937pxMnjxZ1Gq12NraysyZM6W+vl5Zv3r1ahk1alSHfi5fvizz588Xe3t7UalU4uPjI0eOHBERkZSUFOnXr59R+0OHDsn9/AtnZGRIQECA2NrailqtFl9fX1m7dq1cvXq1y77bu3PnjiQnJ8vIkSPFxsZGXFxcJCwsTKqrq7vdPxE9WDU1NRIRESGDBg0SS0tLcXV1lWXLlsmlS5eM2rWPTSIi+fn5AkDc3d2VZZ3FgDt37oi3t7cAkG+//VZERObPny8+Pj5y69Yto7bl5eViY2MjqampJsfNWETUc5yv/H0xwmAwyLJly8TV1VUsLCwEgIwbN04uX75s1O7atWsSFhYmdnZ24urqKnv27JFRo0bJ6tWrlTabNm0SJycnsba2luDgYElLSxMAJl9L++N08eJFCQoKEo1Go8Ti6upqASCnT58WERF3d3cB0OFxL243NzfLhx9+KFqtViwtLcXJyUmmTZsmxcXFyn52794tLi4uYm1tLW+88YYkJSV1e5yJTCkvL5eBAweKv7+/5ObmSm1trRw7dkx8fHxk+PDhRv9Tnc1V/P39RaVSdTtX2b17t6hUKqO/+WvXromrq6usWrVKadfS0iLjx4+X0NBQk+OOjY0VCwsLiYmJkfz8fKmpqRGdTiczZsyQzZs3dzmO9hITE8Xe3l727dsnlZWVUlBQIKGhoaLVaqWpqanL7QwGgwwbNkx8fHzk6NGjUltbK7m5ueLv7y+urq5y/vx5pe3KlSvF0dFR8vLypLi4WKZOnSoajUaJQ3q9XgDI5s2b5ezZs5KWlibOzs49jkMJCQni5uYmZ86cEYPBIM3NzR3i0Lx588TJyUlKS0vlwoULyuPefDE8PFy0Wq1kZmZKVVWVFBQUSGJiovK+cerUKTE3N5cNGzZIeXm5bN26Vfr378849C/HRCORCWlpaWJvb9/hxJqIiIjo/wXnKw/erl27xMrKSg4dOvRPD4XoX4dFURY8WPB4vJmJ9OJD+ESPuKamJly4cAGTJ0/G1KlTTX4ckYiIiOifwPnKw3Xo0CGcOXMGUVFR9/VFOERED9Lu3bsRGRmJjIwMTJ069Z8eDtF9Y6KRqBPx8fFISEhAQEAADh8+DI1G808PiYiIiMgI5ytERI82Fjzo34iJRiIiIiIiIiIiIuozfus0ERERERERERER9RkTjURERERERERERNRnTDQSERERERERERFRnzHRSERERERERERERH3GRCMRERERERERERH1GRONRERERERERERE1GdMNBIREREREREREVGfMdFIREREREREREREfcZEIxEREREREREREfXZfwH+3hof9wmK1wAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<Figure size 1600x1200 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"import matplotlib\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"import os\n",
|
|
"\n",
|
|
"\n",
|
|
"# Compute average inference time + std\n",
|
|
"time_results = {k: np.mean(v.model_inference_time) * 1e3 for k, v in results.items()}\n",
|
|
"time_results_std = np.std([v.model_inference_time for v in results.values()]) * 1000\n",
|
|
"\n",
|
|
"plt.rcdefaults()\n",
|
|
"fig, ax = plt.subplots(figsize=(16, 12))\n",
|
|
"ax.set_ylabel(\"Avg Inference time (ms)\")\n",
|
|
"ax.set_title(\"Average inference time (ms) for each provider\")\n",
|
|
"ax.bar(time_results.keys(), time_results.values(), yerr=time_results_std)\n",
|
|
"plt.show()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"accelerator": "GPU",
|
|
"colab": {
|
|
"collapsed_sections": [],
|
|
"name": "ONNX Overview",
|
|
"provenance": [],
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.0"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 1
|
|
}
|