1_RNN_LSTM_Kaspersky_patent_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Stateful model with Keras - long-term dependencies understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LIbraries import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division, print_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.random import choice\n",
    "from keras.utils import * \n",
    "from keras.models import Sequential\n",
    "from keras.layers import TimeDistributed, Activation, Dropout, Dense, Embedding, LSTM, BatchNormalization\n",
    "from keras import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data prepatation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://psv4.userapi.com/c834700/u7402511/docs/d10/ca6298a74692/Kaspersky.txt\n",
      "49152/45212 [================================] - 0s 1us/step\n",
      "Corpus lengh:  45115\n"
     ]
    }
   ],
   "source": [
    "path = get_file('kaspersky.txt', origin=\"https://psv4.userapi.com/c834700/u7402511/docs/d10/ca6298a74692/Kaspersky.txt\")\n",
    "text = open(path).read()\n",
    "print('Corpus lengh: ', len(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Flexible fingerprint for detection of malware\n",
      "US 8955120 B2\n",
      "ABSTRACT\n",
      "System and method for analyzing a target object for similarity to classes of reference objects. A first and a second set of attributes of the target object is identified composed respectively of attributes having values that are common, and variable, among a class of similar objects. A first hash is computed representing the first set of attributes according to a first hashing algorithm that is sensitive to variations in the first set of attributes among the class of similar objects. A second hash representing the second set of attributes is computed according to a second hashing algorithm that is insensitive to variations in the second set of attributes among the class of similar objects. An aggregate representation of the target object that is based on the first hash and the second hash is generated.\n",
      "DESCRIPTION\n",
      "CLAIM TO PRIORITY\n",
      "The Application claims priority to Russian Federation Patent Application No. 2013129552\n"
     ]
    }
   ],
   "source": [
    "print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting unique characters, i.e. creating vocabulary size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total unique chars:  80\n",
      "['\\n', ' ', \"'\", '(', ')', '*', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '\\\\', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '«', '®', '—', '“', '”']\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(chars)\n",
    "print(\"Total unique chars: \", vocab_size)\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating dictionary of chars and indices, and vice versa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'\\n': 0, ' ': 1, \"'\": 2, '(': 3, ')': 4, '*': 5, ',': 6, '-': 7, '.': 8, '/': 9, '0': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9': 19, ':': 20, ';': 21, 'A': 22, 'B': 23, 'C': 24, 'D': 25, 'E': 26, 'F': 27, 'G': 28, 'H': 29, 'I': 30, 'J': 31, 'K': 32, 'L': 33, 'M': 34, 'N': 35, 'O': 36, 'P': 37, 'R': 38, 'S': 39, 'T': 40, 'U': 41, 'V': 42, 'W': 43, 'X': 44, 'Y': 45, 'Z': 46, '\\\\': 47, '_': 48, 'a': 49, 'b': 50, 'c': 51, 'd': 52, 'e': 53, 'f': 54, 'g': 55, 'h': 56, 'i': 57, 'j': 58, 'k': 59, 'l': 60, 'm': 61, 'n': 62, 'o': 63, 'p': 64, 'q': 65, 'r': 66, 's': 67, 't': 68, 'u': 69, 'v': 70, 'w': 71, 'x': 72, 'y': 73, 'z': 74, '«': 75, '®': 76, '—': 77, '“': 78, '”': 79} \n",
      " {0: '\\n', 1: ' ', 2: \"'\", 3: '(', 4: ')', 5: '*', 6: ',', 7: '-', 8: '.', 9: '/', 10: '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: '9', 20: ':', 21: ';', 22: 'A', 23: 'B', 24: 'C', 25: 'D', 26: 'E', 27: 'F', 28: 'G', 29: 'H', 30: 'I', 31: 'J', 32: 'K', 33: 'L', 34: 'M', 35: 'N', 36: 'O', 37: 'P', 38: 'R', 39: 'S', 40: 'T', 41: 'U', 42: 'V', 43: 'W', 44: 'X', 45: 'Y', 46: 'Z', 47: '\\\\', 48: '_', 49: 'a', 50: 'b', 51: 'c', 52: 'd', 53: 'e', 54: 'f', 55: 'g', 56: 'h', 57: 'i', 58: 'j', 59: 'k', 60: 'l', 61: 'm', 62: 'n', 63: 'o', 64: 'p', 65: 'q', 66: 'r', 67: 's', 68: 't', 69: 'u', 70: 'v', 71: 'w', 72: 'x', 73: 'y', 74: 'z', 75: '«', 76: '®', 77: '—', 78: '“', 79: '”'}\n"
     ]
    }
   ],
   "source": [
    "char_indices = dict((c,i) for i,c in enumerate(chars))\n",
    "indices_char = dict((i,c) for i,c in enumerate(chars))\n",
    "print(char_indices, \"\\n\", indices_char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector creation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Every character of a whole text into list of int (indices) according to dictionary indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27, 60, 53, 72, 57, 50, 60, 53, 1, 54, 57, 62, 55, 53, 66, 64, 66, 57, 62, 68, 1, 54, 63, 66, 1, 52, 53, 68, 53, 51, 68, 57, 63, 62, 1, 63, 54, 1, 61, 49, 60, 71, 49, 66, 53, 0, 41, 39, 1, 18, 19, 15, 15, 11, 12, 10, 1, 23, 12, 0, 22, 23, 39, 40, 38, 22, 24, 40, 0, 39, 73, 67, 68, 53, 61, 1, 49, 62, 52, 1, 61, 53, 68, 56, 63, 52, 1, 54, 63, 66, 1, 49, 62, 49, 60, 73, 74, 57, 62, 55]\n"
     ]
    }
   ],
   "source": [
    "idx = [char_indices[c] for c in text]\n",
    "print(idx[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Splitting whole idx into the chunks of 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequence of 40 with first element as second of previous:  [[27, 60, 53, 72, 57, 50, 60, 53, 1, 54, 57, 62, 55, 53, 66, 64, 66, 57, 62, 68, 1, 54, 63, 66, 1, 52, 53, 68, 53, 51, 68, 57, 63, 62, 1, 63, 54, 1, 61, 49], [60, 53, 72, 57, 50, 60, 53, 1, 54, 57, 62, 55, 53, 66, 64, 66, 57, 62, 68, 1, 54, 63, 66, 1, 52, 53, 68, 53, 51, 68, 57, 63, 62, 1, 63, 54, 1, 61, 49, 60]] \n",
      "\n",
      "Sequence of 40 where last element is a missed int from related sequence [[60, 53, 72, 57, 50, 60, 53, 1, 54, 57, 62, 55, 53, 66, 64, 66, 57, 62, 68, 1, 54, 63, 66, 1, 52, 53, 68, 53, 51, 68, 57, 63, 62, 1, 63, 54, 1, 61, 49, 60], [53, 72, 57, 50, 60, 53, 1, 54, 57, 62, 55, 53, 66, 64, 66, 57, 62, 68, 1, 54, 63, 66, 1, 52, 53, 68, 53, 51, 68, 57, 63, 62, 1, 63, 54, 1, 61, 49, 60, 71]] \n",
      "\n",
      "Number sequences:  45076\n"
     ]
    }
   ],
   "source": [
    "maxlen = 40\n",
    "sentences = [idx[E:E+maxlen] for E in range(len(idx)-maxlen+1)] \n",
    "#each element is a sequence of 40, where each first is a second of previous\n",
    "next_char = [idx[E+1:E+maxlen+1] for E in range(len(idx)-maxlen+1)]\n",
    "#shift on one position which gives a training data, i.e. one char\n",
    "print('Sequence of 40 with first element as second of previous: ', sentences[:2], '\\n')\n",
    "print('Sequence of 40 where last element is a missed int from related sequence', next_char[:2], '\\n')\n",
    "print('Number sequences: ', len(sentences))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Numpy array of inputs from chuncked idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array of sentences:  [[27 60 53 72 57 50 60 53  1 54 57 62 55 53 66 64 66 57 62 68  1 54 63 66\n",
      "   1 52 53 68 53 51 68 57 63 62  1 63 54  1 61 49]\n",
      " [60 53 72 57 50 60 53  1 54 57 62 55 53 66 64 66 57 62 68  1 54 63 66  1\n",
      "  52 53 68 53 51 68 57 63 62  1 63 54  1 61 49 60]] \n",
      "\n",
      "Array of next_char:  [[60 53 72 57 50 60 53  1 54 57 62 55 53 66 64 66 57 62 68  1 54 63 66  1\n",
      "  52 53 68 53 51 68 57 63 62  1 63 54  1 61 49 60]\n",
      " [53 72 57 50 60 53  1 54 57 62 55 53 66 64 66 57 62 68  1 54 63 66  1 52\n",
      "  53 68 53 51 68 57 63 62  1 63 54  1 61 49 60 71]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "((45074, 40), (45074, 40))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = np.concatenate([[np.array(element)] for element in sentences[:-2]]) #2D array (similar to list of lists)\n",
    "next_char = np.concatenate([[np.array(element)] for element in next_char[:-2]])\n",
    "print('Array of sentences: ', sentences[:2], '\\n')\n",
    "print('Array of next_char: ', next_char[:2])\n",
    "sentences.shape, next_char.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_fac = 24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Model creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential([\n",
    "    Embedding(vocab_size, n_fac, input_length=maxlen), \n",
    "    BatchNormalization(),\n",
    "    LSTM(512, return_sequences=True), \n",
    "    Dropout(0.2),\n",
    "    TimeDistributed(Dense(vocab_size)),\n",
    "    Activation('softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 40, 24)            1920      \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 40, 24)            96        \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 40, 512)           1099776   \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 40, 512)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_2 (TimeDist (None, 40, 80)            41040     \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 40, 80)            0         \n",
      "=================================================================\n",
      "Total params: 1,142,832\n",
      "Trainable params: 1,142,784\n",
      "Non-trainable params: 48\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and test "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_example():\n",
    "    seed_string=\"malware detection can be implemented in \"\n",
    "    for i in range(320):\n",
    "        x=np.array([char_indices[c] for c in seed_string[-40:]])[np.newaxis,:]\n",
    "        preds = model.predict(x, verbose=0)[0][-1] \n",
    "        preds = preds/np.sum(preds)\n",
    "        next_char = choice(chars, p=preds)\n",
    "        seed_string = seed_string + next_char\n",
    "    print(seed_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "45072/45072 [==============================] - 3275s 73ms/step - loss: 1.4191\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x3fab2f8ac8>"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(sentences, np.expand_dims(next_char, -1), batch_size=64, epochs=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "malware detection can be implemented in k4csn«.lr/pztLwF:1rMJ73o8g;,“)-jUFgNc6z4KnffJ«pdF(Hft8I3(Smx'®H“—6//MTOZ-RW\\TC/Thsyf\n",
      "I(TO_XFc“\\Og/069C;UGf®J/dJ”922_wZhfrXGunYjH7®M00jCy6-3pJ«:mZgf)3K« JcWh-0:«yMHIGm7(86LB)Ef.x\\bDto“XsFn)-E_8dDq )8jpOV_Wi_cS,(LxG“YH.:covHr«I/xAt,\n",
      "4* xh“/v22bnX9o“Ci78VWZrc.jS F;Cl*kmsZ3o:sC2GLZ4xXtyeu06Z1v3”NDNv«R3ePPW)drkOudN83wiJ/\n",
      "y2\n"
     ]
    }
   ],
   "source": [
    "print_example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.optimizer.lr=0.001 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/8\n",
      "45074/45074 [==============================] - 3280s 73ms/step - loss: 1.3759\n",
      "Epoch 2/8\n",
      "45074/45074 [==============================] - 3272s 73ms/step - loss: 0.5003\n",
      "Epoch 3/8\n",
      "45074/45074 [==============================] - 3286s 73ms/step - loss: 0.3483\n",
      "Epoch 4/8\n",
      "45074/45074 [==============================] - 3320s 74ms/step - loss: 0.3080\n",
      "Epoch 5/8\n",
      "45074/45074 [==============================] - 3288s 73ms/step - loss: 0.2887\n",
      "Epoch 6/8\n",
      "45074/45074 [==============================] - 3279s 73ms/step - loss: 0.2771\n",
      "Epoch 7/8\n",
      "45074/45074 [==============================] - 3260s 72ms/step - loss: 0.2687\n",
      "Epoch 8/8\n",
      "45074/45074 [==============================] - 3286s 73ms/step - loss: 0.2630\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x3f92482e10>"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(sentences, np.expand_dims(next_char, -1), batch_size=64, epochs=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "malware detection can be implemented in this manner, as be an application server, an administrative server, client computers, or a network appliance.\n",
      "A user may enter information to the computer 2 using input devices connected to the user input interface 14 such as a mouse 68 and keyboard 70. Additionally, the input device may be a trackpad, fingerprint scan\n"
     ]
    }
   ],
   "source": [
    "print_example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "malware detection can be implemented in software application executing on a compres for alaly analysis will be deemed malicious at 500. If, at stage 490, it is found that a cluster of similar nimit and am ollow hoaving respectively different values for a common type of attribute, when processed by hash generation module 130 for the subset of fixed attributes\n"
     ]
    }
   ],
   "source": [
    "print_example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "malware detection can be implemented in to hashes of file attribute subsets. Being generated on the basis of instructions that limit access to the aforementioned data stored in the files.\n",
      "In another embodiment, the attribute hash generation module creates a hash of a file attribute subset, which includes at least one variable attribute. In another embodiment\n"
     ]
    }
   ],
   "source": [
    "print_example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('C:/Users/Gavrilov/My Projects/LSTM_weights_1.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.load_weights('C:/Users/Gavrilov/My Projects/LSTM_weights_1.h5', by_name=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['mae', 'acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "45074/45074 [==============================] - 3120s 69ms/step - loss: 0.2602 - mean_absolute_error: 48.2810 - acc: 0.9153\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x72e8adeb38>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(sentences, np.expand_dims(next_char, -1), batch_size=64, epochs=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.save_weights('C:/Users/Gavrilov/My Projects/LSTM_weights_1.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "malware detection can be implemented in accordance with the above-described file hash generation method. During the generation of a selection of hashes for the hash database 170 consists of a set of clusters of similar files which are malicious files. \n",
      "In one embodiment, the set of attributes is computed according to a second hashing algorithm that is insens\n"
     ]
    }
   ],
   "source": [
    "print_example()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}