text:
# [GloVe as a TensorFlow Embedding layer](https://github.com/guillaume-chevalier/GloVe-as-a-TensorFlow-Embedding-Layer)

In this tutorial, we'll see how to convert GloVe embeddings to TensorFlow layers. This could also work with embeddings generated from word2vec.

First, we'll download the embedding we need.

Second, we'll load it into TensorFlow to convert input words with the embedding to word features. The conversion is done within TensorFlow, so it is GPU-optimized and it could run on batches on the GPU. It is also possible to run this tutorial with just a CPU. We'll play with word representations once the embedding is loaded.

What you'll need:
- A working installation of TensorFlow.
- 4 to 6 GB of disk space to download embeddings.

## First, some theory

### Representations

We need a way to represent content in neural networks. For audio, it's possible to use a [spectrogram](https://github.com/guillaume-chevalier/filtering-stft-and-laplace-transform). For images, it's possible to directly use the pixels and then get features maps from a convolutional neural network. For text, analyzing every letter is costly, so it's better to use word representations to embed words or documents as vectors into Artificial Neural Networks and other Machine Learning algorithms.
> ![Features from content](https://www.tensorflow.org/images/audio-image-text.png)
> https://www.tensorflow.org/tutorials/word2vec

As described by Keras, an embedding:

> "Turns positive integers (indexes) into dense vectors of fixed size".

That's it. It's to extract features from words. An embedding is a huge matrix for which each row is a word, and each column is a feature from that word. To summarize, it's possible to convert a word to a vector of a certain length, such as 25, or 100, 200, 1000, and on. In practice, a length of 100 to 300 features is acceptable. With less than 100, we would risk underfitting our linguistic dataset. Word embeddings can eat a lot of RAM, so in this tutorial we'll download and use dimensions of size 25, however changing that to 200 would be a breeze with the actual code.

### You can compute word analogies

The word representations (features) are linear, therefore it's possible to add and substract words with word embeddings. For example, here's the most known word analogy example:

 For example, it's possible to change from:
- Masculine and feminine
- Country and capital
- Singular and plural
- Verb tenses
- And the list goes on...

> ![Word features from embeddings](https://www.tensorflow.org/images/linear-relationships.png)
> https://www.tensorflow.org/tutorials/word2vec

It's also possible to compute the [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) between a word A and a word B, which is the cosine of the angle between the two words. A cosine similarity of -1 would mean the words are complete opposites, while a cosine similarity of 1 would mean that the words are the same. Here's the formula to compare two words:

Here, the norm (such as |A|₂) is the **L2 norm**, the radius in space from the origin, but in a higher dimensional space such as with $n=300$:

### How does it looks like concretely?

For example, here are some cosine similarities to the word "king", computed from the code explained below:

| Other Word | Cosine Similarity |
| ---------- | ----------------- |
| prince:    |   0.933741,       |
| queen:     |   0.9202421,      |
| aka:       |   0.91769224,     |
| lady:      |   0.9163239,      |
| jack:      |   0.91473544,     |
| 's:        |   0.90668976,     |
| stone:     |   0.8982374,      |
| mr.:       |   0.89194083,     |
| the:       |   0.88934386,     |
| star:      |   0.88920873,     |

Finally, notice how similar words are close in space:
> ![](https://www.tensorflow.org/images/embedding-nearest-points.png)
> https://www.tensorflow.org/programmers_guide/embedding

Note: in the image above, the embedding have been subsampled to a lower 3D space with a PCA (Princial Component Analysis) or t-SNE (t-distributed Stochastic Neighbor Embedding) to be explorable. This is possible with TensorBoard for inspection. 300 dimensions can't be visualised easily.

### But how are those word representations obtained?

Before continuing to the practical part where we'll use pretrained embeddings, it's a good thing to know that embeddings can be obtained from unsupervised training on large datasets of text. That's at least a way we can use the text off the internet! To perform this training to get an embedding, it's possible to go with the word2vec approach, or also with the GloVe (Global word Vectors). GloVe is a more recent approach that builds upon the theory of word2vec. Here, we'll use GloVe embeddings. To summarize how the unsupervised training happens, let's see what John Rupert Firth has to say:

> You shall know a word by the company it keeps (Firth, J. R., 1957)

It's amazing that by comparing words and trying to guess the surrounding words, it's possible to find their meaning. To learn more on that, I'd recommend you the [5th course of the Deep Learning Specialization](https://www.coursera.org/learn/nlp-sequence-models) on coursera by Andrew Ng, a course which can lead to the Deep Learning Specialization [certificate](https://www.coursera.org/account/accomplishments/specialization/U7VNC3ZD9YD8).

## Let's get practical!

### First, download the pretrained embeddings with the code below

Careful, the download will take 4-6 GB on disks. If you have already downloaded the embeddings, they will be located under the `./embeddings/` folder relative to here, and won't be downloaded again.

Note: several embeddings were downloaded with different dimension sizes in the zip file, but we only need one.

```python

import numpy as np
import tensorflow as tf
import chakin

import json
import os
from collections import defaultdict

```

```python
chakin.search(lang='English')
```

                       Name  Dimension                     Corpus VocabularySize  \
    2          fastText(en)        300                  Wikipedia           2.5M
    11         GloVe.6B.50d         50  Wikipedia+Gigaword 5 (6B)           400K
    12        GloVe.6B.100d        100  Wikipedia+Gigaword 5 (6B)           400K
    13        GloVe.6B.200d        200  Wikipedia+Gigaword 5 (6B)           400K
    14        GloVe.6B.300d        300  Wikipedia+Gigaword 5 (6B)           400K
    15       GloVe.42B.300d        300          Common Crawl(42B)           1.9M
    16      GloVe.840B.300d        300         Common Crawl(840B)           2.2M
    17    GloVe.Twitter.25d         25               Twitter(27B)           1.2M
    18    GloVe.Twitter.50d         50               Twitter(27B)           1.2M
    19   GloVe.Twitter.100d        100               Twitter(27B)           1.2M
    20   GloVe.Twitter.200d        200               Twitter(27B)           1.2M
    21  word2vec.GoogleNews        300          Google News(100B)           3.0M

          Method Language    Author
    2   fastText  English  Facebook
    11     GloVe  English  Stanford
    12     GloVe  English  Stanford
    13     GloVe  English  Stanford
    14     GloVe  English  Stanford
    15     GloVe  English  Stanford
    16     GloVe  English  Stanford
    17     GloVe  English  Stanford
    18     GloVe  English  Stanford
    19     GloVe  English  Stanford
    20     GloVe  English  Stanford
    21  word2vec  English    Google

```python

# Downloading Twitter.25d embeddings from Stanford:

CHAKIN_INDEX = 17
NUMBER_OF_DIMENSIONS = 25
SUBFOLDER_NAME = "glove.twitter.27B"

DATA_FOLDER = "embeddings"
ZIP_FILE = os.path.join(DATA_FOLDER, "{}.zip".format(SUBFOLDER_NAME))
ZIP_FILE_ALT = "glove" + ZIP_FILE[5:]  # sometimes it's lowercase only...
UNZIP_FOLDER = os.path.join(DATA_FOLDER, SUBFOLDER_NAME)
if SUBFOLDER_NAME[-1] == "d":
    GLOVE_FILENAME = os.path.join(UNZIP_FOLDER, "{}.txt".format(SUBFOLDER_NAME))
else:
    GLOVE_FILENAME = os.path.join(UNZIP_FOLDER, "{}.{}d.txt".format(SUBFOLDER_NAME, NUMBER_OF_DIMENSIONS))

if not os.path.exists(ZIP_FILE) and not os.path.exists(UNZIP_FOLDER):
    # GloVe by Stanford is licensed Apache 2.0:
    #     https://github.com/stanfordnlp/GloVe/blob/master/LICENSE
    #     http://nlp.stanford.edu/data/glove.twitter.27B.zip
    #     Copyright 2014 The Board of Trustees of The Leland Stanford Junior University
    print("Downloading embeddings to '{}'".format(ZIP_FILE))
    chakin.download(number=CHAKIN_INDEX, save_dir='./{}'.format(DATA_FOLDER))
else:
    print("Embeddings already downloaded.")

if not os.path.exists(UNZIP_FOLDER):
    import zipfile
    if not os.path.exists(ZIP_FILE) and os.path.exists(ZIP_FILE_ALT):
        ZIP_FILE = ZIP_FILE_ALT
    with zipfile.ZipFile(ZIP_FILE,"r") as zip_ref:
        print("Extracting embeddings to '{}'".format(UNZIP_FOLDER))
        zip_ref.extractall(UNZIP_FOLDER)
else:
    print("Embeddings already extracted.")

```

    Embeddings already downloaded.
    Embeddings already extracted.

### Let's read the embedding from disks here

First, we load the embeddings, then we demonstrate their usage.

```python

def load_embedding_from_disks(glove_filename, with_indexes=True):
    """
    Read a GloVe txt file. If `with_indexes=True`, we return a tuple of two dictionnaries
    `(word_to_index_dict, index_to_embedding_array)`, otherwise we return only a direct
    `word_to_embedding_dict` dictionnary mapping from a string to a numpy array.
    """
    if with_indexes:
        word_to_index_dict = dict()
        index_to_embedding_array = []
    else:
        word_to_embedding_dict = dict()

    with open(glove_filename, 'r') as glove_file:
        for (i, line) in enumerate(glove_file):

            split = line.split(' ')

            word = split[0]

            representation = split[1:]
            representation = np.array(
                [float(val) for val in representation]
            )

            if with_indexes:
                word_to_index_dict[word] = i
                index_to_embedding_array.append(representation)
            else:
                word_to_embedding_dict[word] = representation

    _WORD_NOT_FOUND = [0.0]* len(representation)  # Empty representation for unknown words.
    if with_indexes:
        _LAST_INDEX = i + 1
        word_to_index_dict = defaultdict(lambda: _LAST_INDEX, word_to_index_dict)
        index_to_embedding_array = np.array(index_to_embedding_array + [_WORD_NOT_FOUND])
        return word_to_index_dict, index_to_embedding_array
    else:
        word_to_embedding_dict = defaultdict(lambda: _WORD_NOT_FOUND)
        return word_to_embedding_dict

```

```python

print("Loading embedding from disks...")
word_to_index, index_to_embedding = load_embedding_from_disks(GLOVE_FILENAME, with_indexes=True)
print("Embedding loaded from disks.")

```

    Loading embedding from disks...
    Embedding loaded from disks.

### Unknown words have representations with values of zero, such as [0, 0, ..., 0]

```python
vocab_size, embedding_dim = index_to_embedding.shape
print("Embedding is of shape: {}".format(index_to_embedding.shape))
print("This means (number of words, number of dimensions per word)\n")
print("The first words are words that tend occur more often.")

print("Note: for unknown words, the representation is an empty vector,\n"
      "and the index is the last one. The dictionnary has a limit:")
print("    {} --> {} --> {}".format("A word", "Index in embedding", "Representation"))
word = "worsdfkljsdf"
idx = word_to_index[word]
embd = list(np.array(index_to_embedding[idx], dtype=int))  # "int" for compact print only.
print("    {} --> {} --> {}".format(word, idx, embd))
word = "the"
idx = word_to_index[word]
embd = list(index_to_embedding[idx])  # "int" for compact print only.
print("    {} --> {} --> {}".format(word, idx, embd))
```

    Embedding is of shape: (1193515, 25)
    This means (number of words, number of dimensions per word)

    The first words are words that tend occur more often.
    Note: for unknown words, the representation is an empty vector,
    and the index is the last one. The dictionnary has a limit:
        A word --> Index in embedding --> Representation
        worsdfkljsdf --> 1193514 --> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
        the --> 13 --> [-0.010167, 0.020194, 0.21473, 0.17289, -0.43659, -0.14687, 1.8429, -0.15753, 0.18187, -0.31782, 0.06839, 0.51776, -6.3371, 0.48066, 0.13777, -0.48568, 0.39, -0.0019506, -0.10218, 0.21262, -0.86146, 0.17263, 0.18783, -0.8425, -0.31208]

### The L2 norm of some words can vary
Notice how more common words have a longer embedding norm, how some text on Twitter was in French, and how it deals with punctuation

```python
words = [
    "The", "Teh", "A", "It", "Its", "Bacon", "Star", "Clone", "Bonjour", "Intelligence",
    "À", "A", "Ça", "Ca", "Été", "C'est", "Aujourd'hui", "Aujourd", "'", "hui", "?", "!", ",", ".", "-", "/", "~"
]

for word in words:
    word_ = word.lower()
    embedding = index_to_embedding[word_to_index[word_]]
    norm = str(np.linalg.norm(embedding))
    print((word + ": ").ljust(15) + norm)
print("Note: here we printed words starting with capital letters, \n"
      "however to take their embeddings we need their lowercase version (str.lower())")
```

    The:           6.825211375610675
    Teh:           5.168743789022242
    A:             6.697466132368121
    It:            6.6026557827711265
    Its:           5.815954107190023
    Bacon:         5.061203156401844
    Star:          4.377212317550177
    Clone:         3.318821469851497
    Bonjour:       4.569226609068979
    Intelligence:  4.978693160848336
    À:             6.757003390719884
    A:             6.697466132368121
    Ça:            6.447278498352788
    Ca:            5.757383694621195
    Été:           5.7824019165330425
    C'est:         6.6648048720694
    Aujourd'hui:   0.0
    Aujourd:       5.28318283424017
    ':             5.146860627039459
    hui:           4.813765207599868
    ?:             5.291857611164723
    !:             5.145156971946508
    ,:             5.401075354278071
    .:             4.965766197438418
    -:             5.10790151248338
    /:             5.063642432429447
    ~:             4.910993053437055
    Note: here we printed words starting with capital letters,
    however to take their embeddings we need their lowercase version (str.lower())

### Let's load the embedding in TensorFlow

We simply create a non-trainable (frozen) tf.Variable() which we set to hold the value of the big embedding matrix.

First, let's define the variables and graph:

```python
batch_size = None  # Any size is accepted

tf.reset_default_graph()
sess = tf.InteractiveSession()  # sess = tf.Session()

# Define the variable that will hold the embedding:
tf_embedding = tf.Variable(
    tf.constant(0.0, shape=index_to_embedding.shape),
    trainable=False,
    name="Embedding"
)

tf_word_ids = tf.placeholder(tf.int32, shape=[batch_size])

tf_word_representation_layer = tf.nn.embedding_lookup(
    params=tf_embedding,
    ids=tf_word_ids
)
```

Sending the embedding to TensorFlow below. It will be located in the GPU from now (or on CPU if GPU is unavailable):

```python
tf_embedding_placeholder = tf.placeholder(tf.float32, shape=index_to_embedding.shape)
tf_embedding_init = tf_embedding.assign(tf_embedding_placeholder)
_ = sess.run(
    tf_embedding_init,
    feed_dict={
        tf_embedding_placeholder: index_to_embedding
    }
)

print("Embedding now stored in TensorFlow. Can delete numpy array to clear some CPU RAM.")
del index_to_embedding

```

    Embedding now stored in TensorFlow. Can delete numpy array to clear some CPU RAM.

Now we can use or fetch representations, for example:

```python
batch_of_words = ["Hello", "World", "!"]
batch_indexes = [word_to_index[w.lower()] for w in batch_of_words]

embedding_from_batch_lookup = sess.run(
    tf_word_representation_layer,
    feed_dict={
        tf_word_ids: batch_indexes
    }
)
print("Representations for {}:".format(batch_of_words))
print(embedding_from_batch_lookup)
```

    Representations for ['Hello', 'World', '!']:
    [[-0.77069    0.12827    0.33137    0.0050893 -0.47605   -0.50116
       1.858      1.0624    -0.56511    0.13328   -0.41918   -0.14195
      -2.8555    -0.57131   -0.13418   -0.44922    0.48591   -0.6479
      -0.84238    0.61669   -0.19824   -0.57967   -0.65885    0.43928
      -0.50473  ]
     [ 0.10301    0.095666  -0.14789   -0.22383   -0.14775   -0.11599
       1.8513     0.24886   -0.41877   -0.20384   -0.08509    0.33246
      -4.6946     0.84096   -0.46666   -0.031128  -0.19539   -0.037349
       0.58949    0.13941   -0.57667   -0.44426   -0.43085   -0.52875
       0.25855  ]
     [ 0.4049    -0.87651   -0.23362   -0.34844   -0.097002   0.40895
       1.6928     1.7058    -1.293      0.70091   -0.12498   -0.75998
      -3.1586     0.14081    0.57255   -0.46097   -0.75721   -0.72414
      -1.4071    -0.17224    0.0099324 -0.45711    0.074886   1.2035
       1.1614   ]]

### To avoid loading the embedding twice in RAM, make TensorFlow able to load them from disks directly

```python
prefix = SUBFOLDER_NAME + "." + str(NUMBER_OF_DIMENSIONS) + "d"
TF_EMBEDDINGS_FILE_NAME = os.path.join(DATA_FOLDER, prefix + ".ckpt")
DICT_WORD_TO_INDEX_FILE_NAME = os.path.join(DATA_FOLDER, prefix + ".json")

variables_to_save = [tf_embedding]
embedding_saver = tf.train.Saver(variables_to_save)
embedding_saver.save(sess, save_path=TF_EMBEDDINGS_FILE_NAME)
print("TF embeddings saved to '{}'.".format(TF_EMBEDDINGS_FILE_NAME))
sess.close()

with open(DICT_WORD_TO_INDEX_FILE_NAME, 'w') as f:
    json.dump(word_to_index, f)
print("word_to_index dict saved to '{}'.".format(DICT_WORD_TO_INDEX_FILE_NAME))

```

    TF embeddings saved to 'embeddings/glove.twitter.27B.25d.ckpt'.
    word_to_index dict saved to 'embeddings/glove.twitter.27B.25d.json'.

```python
words_B = "like absolutely crazy not hate bag sand rock soap"
r = [word_to_index[w.strip()] for w in words_B]
print(words_B)
print(r)
```

    like absolutely crazy not hate bag sand rock soap
    [293, 10, 151, 49, 1193514, 11, 76, 137, 50, 293, 51, 187, 49, 293, 47, 1193514, 210, 73, 11, 1016, 47, 1193514, 36, 50, 187, 1193514, 369, 11, 187, 49, 1193514, 76, 11, 456, 1193514, 137, 11, 36, 199, 1193514, 73, 50, 210, 151, 1193514, 137, 50, 11, 351]

## Build a model to get word similarities from word A to a list of many words B

This is for demo purposes. With a GPU, we can fetch many words quickly and compute on them.

### Restarting from scratch: resetting the Jupyter notebook and loading embeddings from disks, the good way

Now that we have a TensorFlow checkpoint, let's load the embedding without having to parse the txt file into NumPy in CPU:

```python
# Magic iPython/Jupyter command to delete variables and restart the Python kernel
%reset
```

    Once deleted, variables cannot be recovered. Proceed (y/[n])? y

```python
import numpy as np
import tensorflow as tf

import os
import json
from string import punctuation
from collections import defaultdict

batch_size = None  # Any size is accepted
word_representations_dimensions = 25  # Embedding of size (vocab_len, nb_dimensions)

DATA_FOLDER = "embeddings"
SUBFOLDER_NAME = "glove.twitter.27B"
TF_EMBEDDING_FILE_NAME = "{}.ckpt".format(SUBFOLDER_NAME)
SUFFIX = SUBFOLDER_NAME + "." + str(word_representations_dimensions)
TF_EMBEDDINGS_FILE_PATH = os.path.join(DATA_FOLDER, SUFFIX + "d.ckpt")
DICT_WORD_TO_INDEX_FILE_NAME = os.path.join(DATA_FOLDER, SUFFIX + "d.json")

def load_word_to_index(dict_word_to_index_file_name):
    """
    Load a `word_to_index` dict mapping words to their id, with a default value
    of pointing to the last index when not found, which is the unknown word.
    """
    with open(dict_word_to_index_file_name, 'r') as f:
        word_to_index = json.load(f)
    _LAST_INDEX = len(word_to_index) - 2  # Why - 2? Open issue?
    print("word_to_index dict restored from '{}'.".format(dict_word_to_index_file_name))
    word_to_index = defaultdict(lambda: _LAST_INDEX, word_to_index)

    return word_to_index

def load_embedding_tf(word_to_index, tf_embeddings_file_path, nb_dims):
    """
    Define the embedding tf.Variable and load it.
    """
    # 1. Define the variable that will hold the embedding:
    tf_embedding = tf.Variable(
        tf.constant(0.0, shape=[len(word_to_index)-1, nb_dims]),
        trainable=False,
        name="Embedding"
    )

    # 2. Restore the embedding from disks to TensorFlow, GPU (or CPU if GPU unavailable):
    variables_to_restore = [tf_embedding]
    embedding_saver = tf.train.Saver(variables_to_restore)
    embedding_saver.restore(sess, save_path=tf_embeddings_file_path)
    print("TF embeddings restored from '{}'.".format(tf_embeddings_file_path))

    return tf_embedding

def cosine_similarity_tensorflow(tf_word_representation_A, tf_words_representation_B):
    """
    Returns the `cosine_similarity = cos(angle_between_a_and_b_in_space)`
    for the two word A to all the words B.
    The first input word must be a 1D Tensors (word_representation).
    The second input words must be 2D Tensors (batch_size, word_representation).
    The result is a tf tensor that must be fetched with `sess.run`.
    """
    a_normalized = tf.nn.l2_normalize(tf_word_representation_A, axis=-1)
    b_normalized = tf.nn.l2_normalize(tf_words_representation_B, axis=-1)
    similarity = tf.reduce_sum(
        tf.multiply(a_normalized, b_normalized),
        axis=-1
    )

    return similarity

# In case you didn't do the "%reset":
tf.reset_default_graph()
sess = tf.InteractiveSession()  # sess = tf.Session()

# Load the embedding matrix in tf
word_to_index = load_word_to_index(
    DICT_WORD_TO_INDEX_FILE_NAME)
tf_embedding = load_embedding_tf(
    word_to_index,
    TF_EMBEDDINGS_FILE_PATH,
    word_representations_dimensions)

# Input to the graph where word IDs can be sent in batch. Look at the "shape" args:
tf_word_A_id = tf.placeholder(tf.int32, shape=[1])
tf_words_B_ids = tf.placeholder(tf.int32, shape=[batch_size])

# Conversion of words to a representation
tf_word_representation_A = tf.nn.embedding_lookup(
    params=tf_embedding, ids=tf_word_A_id)
tf_words_representation_B = tf.nn.embedding_lookup(
    params=tf_embedding, ids=tf_words_B_ids)

# The graph output are the "cosine_similarities" which we want to fetch in sess.run(...).
cosine_similarities = cosine_similarity_tensorflow(
    tf_word_representation_A,
    tf_words_representation_B)

print("Model created.")

```

    word_to_index dict restored from 'embeddings/glove.twitter.27B.25d.json'.
    INFO:tensorflow:Restoring parameters from embeddings/glove.twitter.27B.25d.ckpt
    TF embeddings restored from 'embeddings/glove.twitter.27B.25d.ckpt'.
    Model created.

Testing the fetch:

```python
def sentence_to_word_ids(sentence, word_to_index):
    """
    Note: there might be a better way to split sentences for GloVe.
    Please look at the documentation or open an issue to suggest a fix.
    """
    # Separating punctuation from words:
    for punctuation_character in punctuation:
        sentence = sentence.replace(punctuation_character, " {} ".format(punctuation_character))
    # Removing double spaces and lowercasing:
    sentence = sentence.replace("  ", " ").replace("  ", " ").lower().strip()
    # Splitting on every space:
    split_sentence = sentence.split(" ")
    # Converting to IDs:
    ids = [word_to_index[w.strip()] for w in split_sentence]
    return ids, split_sentence

def predict_cosine_similarities(sess, word_A, words_B):
    """
    Use the model in sess to predict cosine similarities.
    """

    word_A_id, _ = sentence_to_word_ids(word_A, word_to_index)
    words_B_ids, split_sentence = sentence_to_word_ids(words_B, word_to_index)

    evaluated_cos_similarities = sess.run(
        cosine_similarities,
        feed_dict={
            tf_word_A_id: word_A_id,
            tf_words_B_ids: words_B_ids
        }
    )
    return evaluated_cos_similarities, split_sentence

word_A = "Science"
words_B = "Hello internet, a vocano erupt like the bitcoin out of the blue and there is an unknownWord00!"

evaluated_cos_similarities, splitted = predict_cosine_similarities(sess, word_A, words_B)

print("Cosine similarities with \"{}\":".format(word_A))
for word, similarity in zip(splitted, evaluated_cos_similarities):
    print("    {}{}".format((word+":").ljust(15), similarity))
```

    Cosine similarities with "Science":
        hello:         0.4928313195705414
        internet:      0.6569848656654358
        ,:             0.49928268790245056
        a:             0.4862615764141083
        vocano:        0.0
        erupt:         0.16974501311779022
        like:          0.6019276976585388
        the:           0.7420801520347595
        bitcoin:       0.5125284194946289
        out:           0.6307196617126465
        of:            0.7674074172973633
        the:           0.7420801520347595
        blue:          0.4629560112953186
        and:           0.6889371275901794
        there:         0.7172714471817017
        is:            0.7015751600265503
        an:            0.6883363723754883
        unknownword00: 0.0
        !:             0.3991800844669342

## Getting the top k most similars words to a word with the embedding matrix

Let's take an input word and compare it to every other words in the embedding matrix to return the most similar words.

```python
tf.reset_default_graph()

# Transpose word_to_index dict:
index_to_word = dict((val, key) for key, val in word_to_index.items())

# New graph
tf.reset_default_graph()
sess = tf.InteractiveSession()

# Load the embedding matrix in tf
tf_word_to_index = load_word_to_index(
    DICT_WORD_TO_INDEX_FILE_NAME)
tf_embedding = load_embedding_tf(
    tf_word_to_index,
    TF_EMBEDDINGS_FILE_PATH,
    word_representations_dimensions)

# An input word
tf_word_id = tf.placeholder(tf.int32, shape=[1])
tf_word_representation = tf.nn.embedding_lookup(
    params=tf_embedding, ids=tf_word_id)

# An input
tf_nb_similar_words_to_get = tf.placeholder(tf.int32)

# Dot the word to every embedding
tf_all_cosine_similarities = cosine_similarity_tensorflow(
    tf_word_representation,
    tf_embedding)

# Getting the top cosine similarities.
tf_top_cosine_similarities, tf_top_word_indices = tf.nn.top_k(
    tf_all_cosine_similarities,
    k=tf_nb_similar_words_to_get+1,
    sorted=True
)

# Discard the first word because it's the input word itself:
tf_top_cosine_similarities = tf_top_cosine_similarities[1:]
tf_top_word_indices = tf_top_word_indices[1:]

# Get the top words' representations by fetching
# tf_top_words_representation = "tf_embedding[tf_top_word_indices]":
tf_top_words_representation = tf.gather(
    tf_embedding,
    tf_top_word_indices)

```

    word_to_index dict restored from 'embeddings/glove.twitter.27B.25d.json'.
    INFO:tensorflow:Restoring parameters from embeddings/glove.twitter.27B.25d.ckpt
    TF embeddings restored from 'embeddings/glove.twitter.27B.25d.ckpt'.

```python
# Fetch 10 similar words:
nb_similar_words_to_get = 10

word = "king"
word_id = word_to_index[word]

top_cosine_similarities, top_word_indices, top_words_representation = sess.run(
    [tf_top_cosine_similarities, tf_top_word_indices, tf_top_words_representation],
    feed_dict={
        tf_word_id: [word_id],
        tf_nb_similar_words_to_get: nb_similar_words_to_get
    }
)

print("Top similar words to \"{}\":\n".format(word))
loop = zip(top_cosine_similarities, top_word_indices, top_words_representation)
for cos_sim, word_id, word_repr in loop:
    print(
        (index_to_word[word_id]+ ":").ljust(15),
        (str(cos_sim) + ",").ljust(15),
        np.linalg.norm(word_repr)
    )
```

    Top similar words to "king":

    prince:         0.933741,       4.1957383
    queen:          0.9202421,      4.1540723
    aka:            0.91769224,     3.5418782
    lady:           0.9163239,      4.256068
    jack:           0.91473544,     4.2670665
    's:             0.90668976,     6.276691
    stone:          0.8982374,      4.2092624
    mr.:            0.89194083,     4.1567283
    the:            0.88934386,     6.8252115
    star:           0.88920873,     4.3772125

Notice the bad quality of the similar words, embeddings with more dimensions than 25 would make it better.

Reminder: we chose 25 dimensions for tutorial purposes not to eat all our RAM. There are better embeddings out there.

##  What's next?

I think getting the embeddings into TensorFlow is a good step into building a language model. You may want to grab some data, such as [here](https://github.com/awesomedata/awesome-public-datasets#naturallanguage) and [here](https://github.com/niderhoff/nlp-datasets). You may also want to learn more about how recurrent neural networks can read features such as sentences or signal of varying length, such as [an LSTM (RNN) encoder reading signal](https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition) or an [signal predictor from a seq2seq GRU (RNN)](https://github.com/guillaume-chevalier/seq2seq-signal-prediction) which could be used in practice to [predict next words in a sentence](https://blog.openai.com/unsupervised-sentiment-neuron/). Since signal is closely related to sentences with embedded words, RNNs can be applied on both.

## References

The pretrained word vectors can be found there:
- Repo https://github.com/stanfordnlp/GloVe
- Manual download: http://nlp.stanford.edu/data/glove.twitter.27B.zip

Chakin was used to download those word embeddings:
- https://github.com/chakki-works/chakin

Some images in this notebook are references/links from the TensorFlow website:
- https://www.tensorflow.org/

To cite my work, point to the URL of the GitHub repository:
- https://github.com/guillaume-chevalier/GloVe-as-TensorFlow-Embedding

My code is available under the [MIT License](https://github.com/guillaume-chevalier/GloVe-as-TensorFlow-Embedding/blob/master/LICENSE).

## Connect with me

- https://ca.linkedin.com/in/chevalierg
- https://twitter.com/guillaume_che
- https://github.com/guillaume-chevalier/

```python
# Let's convert this notebook to a README for the GitHub project's title page:
!ipython3 nbconvert --to markdown "GloVe-as-TensorFlow-Embedding-Tutorial.ipynb"
!mv "GloVe-as-TensorFlow-Embedding-Tutorial.md" README.md
```

    [TerminalIPythonApp] WARNING | Subcommand `ipython nbconvert` is deprecated and will be removed in future versions.
    [TerminalIPythonApp] WARNING | You likely want to use `jupyter nbconvert` in the future
    [NbConvertApp] Converting notebook GloVe-as-TensorFlow-Embedding-Tutorial.ipynb to markdown
    [NbConvertApp] Writing 30873 bytes to GloVe-as-TensorFlow-Embedding-Tutorial.md


text:
# Prettycom ![Node.js CI](https://github.com/hyldmo/prettycom/workflows/Node.js%20CI/badge.svg)
----
Inspired by [CuteCom](https://gitlab.com/cutecom/cutecom/), Prettycom is a cross-platform serial monitor written in electron.

![Screenshot](./screenshot.png)

## Installation
Binaries can be found [here](https://github.com/hyldmo/prettycom/releases).

## Development
- Clone repository: `git clone https://github.com/hyldmo/prettycom.git`
- Run `yarn`
- Run `yarn dev`, or if you're using VSCode, press `F5`

### Notes
- Code in `app/server` server folder lives inside electron, and is not automatically reloaded.
You will have to restart the electron shell for changes to be reflected.
- Build files (such as `.js` and `.css`) are automatically hidden in VS Code. Go to `.vscode\settings.json` to unhide them.
- If you are switching branches and you're getting errors, try running `yarn clean` to clear old build files

license: MIT

repository: hyldmo/prettycom

branch_name: refs/heads/master

extension: md

gha_language: TypeScript

gha_license_id: MIT

license_type: permissive

repo_name: hyldmo/prettycom


text:
# Version

* Docker：20.10.7
* Kubectl：v1.20.1
* Minikube：v1.20.0
* kustomize:v3.2.3
* Kubeflow：v1.3.1

# Check Version

* Docker
```Bash
docker version
```
* Kubectl
```Bash
kubectl version
```
* Minikube
```Bash
minikube version
```
* Kustomize
```Bash
kustomize version
```
* Kubeflow
> Only the version of kubeflow has no relevant instructions to query. \
> The version of kubeflow can be found in 1. Installed kubeflow.v13.sh. \
> When kubeflow.v13.sh is executed, v1.3.1.tar will be downloaded, v1.3.1 is the version number of kubeflow.

# Relevant part

* [About Install](https://github.com/dfm871002/examples/blob/master/Natural-Language-Processing/1.%20Install/Install.md)
* [About Docker](https://github.com/dfm871002/examples/blob/master/Natural-Language-Processing/2.%20Docker/Docker.md)
* [About Jupyter Notebook](https://github.com/dfm871002/examples/blob/master/Natural-Language-Processing/3.%20Jupyter%20Notebook/Jupyter%20Notebook.md)

repository: kubeflow/examples

gha_language: Jsonnet

repo_name: kubeflow/examples


text:
# project Summary
----------------------------------------------------
- Project Title:
	- Graduation Rate for Florida university

- Team Members:
	- Cheikh Sidi El Moctar
	- Bridgette Sanders
	- Lisette Cruz
	- Ricardo Kenny

- Research question to Answer:
	- What affect the graduation rate within Florida state?

- Dataset to be Used:
	- Federal education datasets.

- Factor Attributes:
	- School size,
	- affordablity,
	- Public or private,
	- Class size.

## Analysis Approach and Tools

to answer the research quation, we collected dataset from 2019 Acceptance Rate Comparison between Colleges in Florida ,
datasource : https://www.collegetuitioncompare.com/compare/tables/?state=fl&factor=acceptance-rate

- Set Clear Measurement Priorities
- Data acquisition by scraping the datasource above using Pandas.
- Data Cleaning, messaging, and visualization using Pandas, Matplotlib and Seaborn
- the following tables were cleaned and processed:
	- Graduation Rate
	- Acceptance Rate
 	- SAT & ACT test Scores
	- Tuition Fees
	- Student Population

- Filtering Florida from the dataset using Pandas
- Filtering the Florida dataset for target universities (top 5 universities).
- Analysis and features correlation


text:
magic-property
==============

*Objective-C data records made easy.*

magic-property generates immutable Objective-C classes with init, copy, isEqual, hash,
and description methods from a simple header file with just a list of properties.

Features
--------
 * Automatically generates immutable Objective-C classes
 * correct `isEqual:` and `hash` implementation
 * init, new and copy methods for/with all properties
 * `description` method with human-readable output
 * all property attributes (e.g. nonatomic, strong) supported
 * ARC support
 * NSCopying, NSCoding and NSSecureCoding support
 * support for extra method implementations and custom property setters and
   accessors
 * support for custom typedefs and enums
 * support for generating correct `isEqual:`, `hash`, and `description` for
   handwritten classes
 * no special compiler features needed

Usage
-----

### Sample Header ####

A simple header like this is enough to generate the immutable Objective-C class:
```objective-c
#import <Foundation/Foundation.h>

#import "Address.h"

@interface PhonebookEntry : NSObject
@property (nonatomic, strong, readonly) NSString *name;
@property (nonatomic, strong, readonly) NSString *email;
@property (nonatomic, strong, readonly) NSArray *phoneNumbers;
@property (nonatomic, strong, readonly) NSString *birthday;
@property (nonatomic, strong, readonly) Address *address;
@end
```

Just process the header file with the magic-property script and the
actual class header and implementation file will be generated by the script.

### Generated Code ###

magic-property expands the interface to the code shown below.
The generated code in the .m file also contains meaningful
implementations of `isEqual:`, `hash`, and `description`.

```
                                     // added support for NSCopying and NSCoding
@interface PhonebookEntry : NSObject <NSCopying, NSCoding>
@property (nonatomic, strong, readonly) NSString *name;
@property (nonatomic, strong, readonly) NSString *email;
@property (nonatomic, strong, readonly) NSArray *phoneNumbers;
@property (nonatomic, strong, readonly) NSString *birthday;
@property (nonatomic, strong, readonly) Address *address;

// added init and new methods
- (id)initWithName:(NSString *)name
             email:(NSString *)email
      phoneNumbers:(NSArray *)phoneNumbers
          birthday:(NSString *)birthday
           address:(Address *)address;
+ (PhonebookEntry *)newWithName:(NSString *)name
                          email:(NSString *)email
                   phoneNumbers:(NSArray *)phoneNumbers
                       birthday:(NSString *)birthday
                        address:(Address *)address;

// copy methods for all properties.
- (PhonebookEntry *)copyWithName:(NSString *)name;
- (PhonebookEntry *)copyWithEmail:(NSString *)email;
- (PhonebookEntry *)copyWithPhoneNumbers:(NSArray *)phoneNumbers;
- (PhonebookEntry *)copyWithBirthday:(NSString *)birthday;
- (PhonebookEntry *)copyWithAddress:(Address *)address;

// meaningful implementations of isEqual:, hash, and description
// are placed in the .m file
@end
```

### Commandline Options ###

The magic-property script supports the following commandline options, see below
for Xcode integration.

```
usage: magic-property [-h] -o DIR [--no-arc] [--force] [--custom-ints FILE]
                      [--enums FILE] [--only-eq-hash-description]
                      [--eq-hash-description-outfile FILE]
                      [--eq-hash-macro-base-name NAME]
                      [--description-macro-base-name NAME]
                      [--eq-hash-description-macro-base-name NAME]
                      [INPUT_FILE [INPUT_FILE ...]]

Objective-C data records made easy

positional arguments:
  INPUT_FILE            input files for case-class generation (.h and .m
                        files)

optional arguments:
  -h, --help            show this help message and exit
  -o DIR, --output-dir DIR
                        output directory
  --no-arc              generate code without ARC support
  --force               force code generation
  --custom-ints FILE    file with custom integer types (newline separated)
  --enums FILE          file with enum definitions
  --only-eq-hash-description
                        generate only the IsEqualHashDescription macros
  --eq-hash-description-outfile FILE
                        output file for the IsEqualHashDescription macros,
                        default: MPEqHashDescription.h
  --eq-hash-macro-base-name NAME
                        basename for the IsEqualHash macro, default:
                        MPGenerateIsEqualHash
  --description-macro-base-name NAME
                        basename for the Description macro, default:
                        MPGenerateDescription
  --eq-hash-description-macro-base-name NAME
                        basename for the IsEqualHashDescription macro,
                        default: MPGenerateIsEqualHashDescription
```

### Custom Methods ###

magic-property can use custom and extra methods to override the default
behavior. Simply add the methods to override or to add in an m-file. There is no
need to manually add any of the generated methods.

```objective-c
@implementation PhonebookEntry
+ (PhonebookEntry *)parseFromString:(NSString *)
{
   ... // the method is added to the final class by the script
}

- (NSString *)serializeToString
{
    ... // this becomes a method of the immutable class
}
@end
```

### Custom Integers ###

TODO

Enumerations
------------

magic-property also simplifies definition of enumeration types. To define
an enumeration type `NetworkState` with two states `Offline` and `Online`,
just place the following line in some file and pass the file to magic-property
via the `--enums` flag.

```
NetworkState: Offline Online
```

magic-property then generates the following .h file and a suitable .m file:

```objective-c
// the definition of the enumeration
typedef enum {
  NetworkStateOffline = 1,
  NetworkStateOnline = 2,
} NetworkState;
// a function for converting enum values to strings
NSString *stringFromNetworkState(NetworkState x);
// support for meaningful description methods if the enumeration type is used
// in a property in classes processed by magic-property (see below under "Custom Integers")
@interface MPIntegerNetworkState : MPInteger
+ (NSString *)descriptionOf:(NSInteger)i;
@end
```

`isEqual:`, `hash`, `description` for Handwritten Classes
-----------------

TODO

XCode Integration
------------

magic-property can be used as standalone script but XCode integration can be setup with these simple steps:

### Step 1: Copy magic-property script ###

Copy the `scripts` folder to your project.

### Step 2: Create folder and property files ###

Create a folder where the generated code will be placed (e.g. `gen`) and a folder
where all your property header files will go (e.g. `property-classes`). Place the
property header files (*.h) you want to use to this folder.

### Step 3: Add the script as a build phase ###

Add a "Run script" build phase to XCode.
![Add build target](screenshot_buildtarget.png "Add build target")

Make sure the script build phase is before compiling sources and add the
following code:
```
scripts/magic-property -o gen property-classes/*.h property-classes/*.m
```
Make sure to change the paths to the correct location you chose for the
magic-property script, generated code folder and input folder. Add the
`property-classes/*.m` argument only if you have at least one `.m` file in
`property-classes`.

![Script](screenshot_script.png "Add script")

### Step 4: Add generated files ###

Add the files from the `objective-c` folder to your project. Build your project
once. The build will generate all the necessary files from your property class
headers. Add the generated headers and implementation files to your XCode
project. Note that you have to add the files again for each new property class
header file you create, but not if you change the contents of a property header
file.

![Add files](screenshot_addfiles.png "Add files")

repository: skogsbaer/magic-property

gha_language: Python

repo_name: skogsbaer/magic-property


text:
# perry-game
Simple game about eating smaller platypus and avoid being eaten by bigger ones.

## Requirements:
- Python 2.7
- Pygame
- All project files (code and images) should be located in the same folder. Run main.py to start the game.

## Files
A little explanation about what is inside each file:
- ```main.py```: Main loop of the game. It renders the background, the player character and the enemies. It also checks for collisions between player and enemies and reads player input.
- ```perry.py```: File that holds the class of the player's character. It just contains the class, which inherits from pygame.sprite.Sprite, with its attributes and methods.
- ```enemy.py```: File that holds the class of the enemies. It just contains the class, which inherits from pygame.sprite.Sprite, with its attributes, methods, and enemy AI.
- ```helper.py```: Contains several functions that are called in the main loop. It mainly improves readability

## TODO
- *fix bug* enemy spawns. From time to time the number of enemies that appear goes wild.
- *fix bug* enemy speed. When the direction of movement of the enemies is close to +-90º the enemy displacement along the y-axis is huge.
- *Improve* the enemy movement and the ratio of appearance and total number of enemies in the screen.

repository: juangallostra/perry-game

repo_name: juangallostra/perry-game


text:
# Installing git

## For Mac or Linux

See git's [official installation guide](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).

## For Windows

There are several ways to install git on Windows. For the workshop, you should install
"Git for Windows". First, [download it](https://git-for-windows.github.io/), and run the installer.

During installation, you will encounter the following screen. Select "checkout as-is, commit unix-style endings":

![install screen](gitForWindowsInstallScreen.png)

Don't worry if you've already installed Git for Windows and accepted the default setting; it should not cause
any problems.

After installing Git for Windows, you will have an application called Git Bash, which we will be using for the workshop.

During the hands-on, you will be editing files. Instead of using Notepad, you should use an editor that is
more appropriate for code, such as [Notepad++](https://notepad-plus-plus.org/) (free) or
[Sublime Text](https://www.sublimetext.com/) (free to try).

There are tools for Windows that provide GUI support for git, such [TortoiseGit](https://tortoisegit.org/).
The workshop will not cover those tools.

repository: WHOIGit/whoi-git-workshop

repo_name: WHOIGit/whoi-git-workshop


text:
# PEPE CANNON
This PEPE CANNON deploys RARE PEPE to your Discord server via webhooks.

To run this properly, follow the instructions listed on DockerHub [here](https://hub.docker.com/repository/docker/cryptobiff/pepe-launcher/general)

## Requirements
You must meet the following requirements for this ancient tool to be useful to your memeing efforts:
* Docker Installed & Running
* DockerHub Account
* Requires the end user to be based.  Special snowflakes will experience strange errors when running this tool.

## Pepe Discord Bombardment Cannon
Set up a Webhook URL for the Discord channel you want to deploy the pepe bombardment cannon to by following the instructions below:
* Go to **Server Settings** --> **Integrations** --> **Webhooks** --> **Create New Webhook**
* Select the name for the webhook bot user to display
* Add an avatar if desired to your webhook PEPE User
* Select the destination channel for your pepe cannon
* Copy the "Webhook URL" after you have configured it and paste it below where it says **`PASTE_YOUR_DISCORD_WEBHOOK_URL_HERE`** and the rare pepe will be deployed after you run the docker command shown below.

```
 docker run -e DISCORD_WEBHOOK_URL="PASTE_YOUR_DISCORD_WEBHOOK_URL_HERE" cryptobiff/pepe-launcher

```

repository: thearchive-win/pepe-launcher

repo_name: thearchive-win/pepe-launcher


text:
# 자바 디자인 패턴

- ### [Iterator, 순서대로 지정해서 처리하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/01_Iterator.md)

- ### [Adapter, 바꿔서 재이용하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/02_Adapter.md)

## 하위 클래스에게 위임

- ### [Template Method, 하위 클래스에서 처리하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/03_Template_Method.md)

- ### [Factory Method, 하위클래스에 위임하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/04_Factory_Method.md)

## 인스턴스 생성 관련 패턴

- ### [Singleton, 하나만 생성하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/05_Singleton.md)

- ### [Prototype, 복사해서 인스턴스 만들기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/06_Prototype.md)

- ### [Builder, 복잡한 객체 조립하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/07_Builder.md)

- ### [Abstract Factory, 객체 조합하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/08_Abstract_Factory.md)

## 분리 관련 패턴

- ### [Bridge, 기능 구현과 추가 분리하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/09_Bridge.md)

- ### [Strategy, 알고리즘 동채로 바꾸기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/10_Strategy.md)

## 동일시 하기

- ### [Composite, 그릇과 내용물 동일하게 하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/11_Composite.md)

- ### [Decorator, 장식과 내용물 동일하게 하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/12_Decorator.md)

## 구조를 돌아다니기

- ### [Vistor, 데이터 구조 돌아다니기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/13_Visitor.md)

- ### [Chain of responsibility, 책임 떠넘기기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/14_Chain_Of_Responsibility.md)

## 단순화하기

- ### [Facade, 처리하는 창구 만들기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/15_Facade.md)

- ### [Mediator, 책임 떠넘기기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/16_Mediator.md)

### 상태 관리

- ### [Observer , 상태 관찰하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/17_Observer.md)

- ### [Memento, 상태 저장](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/18_Memento.md)

- ### [State, 상태 관리](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/19_State.md)

## 낭비 없애기

- ### [Flyweight, 책임 떠넘기기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/20_Flyweight.md)

- ### [Proxy, 책임 떠넘기기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/21_Proxy.md)

## 클래스로 표현하기

- ### [Command, 명령을 커맨드로 표현하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/22_Command.md)

- ### [Interpreter, 명령을 커맨드로 표현하기](https://github.com/banziha104/GOF_Design_Pattern/blob/master/markdown/23_Interpreter.md)


text:
FastDtools
===

Bioinformatic tools to work on FastQ and FastA files.
Bioplayground for working on bioinformatic data files in the Dlang.

Tools
---

    `fastq-seqstat`: Summarize the fastq file with sequence metrics like n-reads, n-bases, reads with N's.
        Quality metrics on the phred qualities and autodetection of the encoding.

About
---

The aim of this project is to provide the essential tools to work with sequencing data stored in the FastQ format.
We are aiming to provide an extensive API to the fileformat, file-mangling as conversion, extraction, validation.

On the FastA part we invite you to submit a ticket to tell us what to support.

Licence
---

MIT licenced

repository: wyleung/FastDtools

repo_name: wyleung/FastDtools


text:
### **Class 05 - Linked List Implementation**

### Implementation of Singly Linked Lists

### Challenge
- Create a Node class that has properties for the value stored in the Node, and a pointer to the next Node.
- Create a LinkedList class that includes a head property, which creates an empty LinkedList when instantiated.
- Define insert, includes, and toString methods for the Linked List.
- Add tests

## Approach & Efficiency

- Insert Method
  - Create new node with a value
  - Set the head to the new node's next value
  - Set the new node to the head value

- Includes Method
  - Start with the head and check to see if it matches value passed in to method.
    - if it matches, return true.
    - if it does not match, set the current to current.next and check for a match
    - if no matches are found, return false

- toString Method
  - start with current value, check to ensure it does not equal null
    - if it equals null, return null.
  - return the readable contents of the LinkedList.

gha_language: JavaScript


text:
# DROP Sample Funding Feed Package

DROP Sample Funding Feed illustrates the Smooth and Shape Preserving Reconstitution of the Funding Feed.

## Class Components

 * [***AUDShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/AUDShapePreservingReconstitutor.java)
 AUDShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the AUD Input Marks.

 * [***AUDSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/AUDSmoothReconstitutor.java)
 AUDSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the AUD Input
 Marks.

 * [***CADShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/CADShapePreservingReconstitutor.java)
 CADShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the CAD Input Marks.

 * [***CADSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/CADSmoothReconstitutor.java)
 CADSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the CAD Input
 Marks.

 * [***CHFShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/CHFShapePreservingReconstitutor.java)
 CHFShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the CHF Input Marks.

 * [***CHFSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/CHFSmoothReconstitutor.java)
 CHFSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the CHF Input
 Marks.

 * [***CZKShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/CZKShapePreservingReconstitutor.java)
 CZKShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the CZK Input Marks.

 * [***DKKShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/DKKShapePreservingReconstitutor.java)
 DKKShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the DKK Input Marks.

 * [***EURShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/EURShapePreservingReconstitutor.java)
 EURShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the EUR Input Marks.

 * [***EURSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/EURSmoothReconstitutor.java)
 EURSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the EUR Input
 Marks.

 * [***GBPShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/GBPShapePreservingReconstitutor.java)
 GBPShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the GBP Input Marks.

 * [***GBPSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/GBPSmoothReconstitutor.java)
 GBPSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the GBP Input
 Marks.

 * [***HKDShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/HKDShapePreservingReconstitutor.java)
 HKDShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the HKD Input Marks.

 * [***HUFShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/HUFShapePreservingReconstitutor.java)
 HUFShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the HUF Input Marks.

 * [***ILSShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/ILSShapePreservingReconstitutor.java)
 ILSShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the ILS Input Marks.

 * [***JPYShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/JPYShapePreservingReconstitutor.java)
 JPYShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the JPY Input Marks.

 * [***JPYSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/JPYSmoothReconstitutor.java)
 JPYSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the JPY Input
 Marks.

 * [***MXNShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/MXNShapePreservingReconstitutor.java)
 MXNShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the MXN Input Marks.

 * [***NOKShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/NOKShapePreservingReconstitutor.java)
 NOKShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the NOK Input Marks.

 * [***NOKSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/NOKSmoothReconstitutor.java)
 NOKSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the NOK Input
 Marks.

 * [***NZDShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/NZDShapePreservingReconstitutor.java)
 NZDShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the NZD Input Marks.

 * [***NZDSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/NZDSmoothReconstitutor.java)
 NZDSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the NZD Input
 Marks.

 * [***PLNShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/PLNShapePreservingReconstitutor.java)
 PLNShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the PLN Input Marks.

 * [***SEKShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/SEKShapePreservingReconstitutor.java)
 SEKShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the SEK Input Marks.

 * [***SEKSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/SEKSmoothReconstitutor.java)
 SEKSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the SEK Input
 Marks.

 * [***SGDShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/SGDShapePreservingReconstitutor.java)
 SGDShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the SGD Input Marks.

 * [***TRYShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/TRYShapePreservingReconstitutor.java)
 TRYShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the TRY Input Marks.

 * [***SEKShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/SEKShapePreservingReconstitutor.java)
 SEKShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the SEK Input Marks.

 * [***UnifiedShapePreserving1YStart***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/UnifiedShapePreserving1YStart.java)
 UnifiedShapePreserving1YStart demonstrates the unified re-constitution and Metrics Generation.

 * [***USDSmoothReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/USDSmoothReconstitutor.java)
 USDSmoothReconstitutor Demonstrates the Cleansing and the Smooth Re-constitution of the SEK Input
 Marks.

 * [***USDShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/USDShapePreservingReconstitutor.java)
 USDShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the USD Input Marks.

 * [***ZARShapePreservingReconstitutor***](https://github.com/lakshmiDRIP/DROP/tree/master/src/main/java/org/drip/sample/fundingfeed/ZARShapePreservingReconstitutor.java)
 ZARShapePreservingReconstitutor Demonstrates the Cleansing and the Shape Preserving Re-constitution
 of the ZAR Input Marks.

## DROP Specifications

 * Main                     => https://lakshmidrip.github.io/DROP/
 * Wiki                     => https://github.com/lakshmiDRIP/DROP/wiki
 * GitHub                   => https://github.com/lakshmiDRIP/DROP
 * Repo Layout Taxonomy     => https://github.com/lakshmiDRIP/DROP/blob/master/Taxonomy.md
 * Javadoc                  => https://lakshmidrip.github.io/DROP/Javadoc/index.html
 * Technical Specifications => https://github.com/lakshmiDRIP/DROP/tree/master/Docs/Internal
 * Release Versions         => https://lakshmidrip.github.io/DROP/version.html
 * Community Credits        => https://lakshmidrip.github.io/DROP/credits.html
 * Issues Catalog           => https://github.com/lakshmiDRIP/DROP/issues
 * JUnit                    => https://lakshmidrip.github.io/DROP/junit/index.html
 * Jacoco                   => https://lakshmidrip.github.io/DROP/jacoco/index.html

repository: lakshmiDRIP/DROP

gha_language: HTML

repo_name: lakshmiDRIP/DROP


text:
# ionic-android-builder:1.0
Docker image include Android SDK for building Ionic framework application.

## About this project
In my work, I want to build the android application develop by ionic framework via [gitlab-runner](https://gitlab.com/gitlab-org/gitlab-runner) with Docker driver.

## Example `.gitlab-ci` file
```Dockerfile
build:
  image: josefd8/ionic-android-builder:1.0
  script:
    - npm install
    - mkdir www
    - yes | /opt/android-sdk/tools/bin/sdkmanager --licenses || true
    - ionic cordova build android
  artifacts:
    paths:
    - platforms/android/build/outputs/apk/*.apk
```
## Extra helper command
If you want to run or build the ionic project in computer but doesn't have Android Studio, Android SDK or Ionic Framework and this computer installed Docker. You can use helper command

- Restore npm package
```
docker run --rm -v $(pwd):/ionicapp josefd8/ionic-android-builder:1.0 npm install
```
- Preview Ionic web app in your web browser
```
docker run --rm -v $(pwd):/ionicapp -p 8100:8100 josefd8/ionic-android-builder:1.0 ionic serve
```
- Build android apk output file
```
docker run --rm -v $(pwd):/ionicapp josefd8/ionic-android-builder:1.0 ionic cordova build android
```

## ADB Support
```
docker run --privileged -v /dev/bus/usb:/dev/bus/usb -P -v $(pwd):/ionicapp josefd8/ionic-android-builder:1.0 /opt/android-sdk/platform-tools/adb devices
```


text:
# IPFS-RFP0
RFP 0: Internet per peer

> NOTE: Very draft state!
>
> Some philosophy on Russian (translation comming soon)

[philosophy](https://github.com/stels-community/IPFS-RFP0/blob/master/README.RU.md) | [theory](https://github.com/stels-community/IPFS-RFP0/blob/master/IDEA.RU.md) | [practic](https://stels-community.github.io/)

## Summary

Most important concept in any practical activity of any thinking creature is addressing
Any word is address, word combinations, even point by finger and means "look here" is addressing.
We meet addressing in discontinuos forms, and our practic said: "unified addressing very useful", but application cases of addressing is different too. In the technologies sphere, We are use addressing based on location (IP, RAM LFB, FUSE, NTFS, ...), context (IPFS, BitTorrent, ...), access (encryption, signatures, DC consensus, multyside cryptopazzl, ...), naming (DNS, IPNS, ...), structured (Foreign keys, databases, Merkle treas, hash chains, ...)

...

1. register ipfs gtld
2. IEEE standartization
3. //

raw

`nodehash.ipfs/contenthash`

ipns

`nodename.ipfs/contentname`


text:
# Brain-Observatory-Toolbox
A Matlab toolbox for interacting with the [Allen Brain Observatory](http://observatory.brain-map.org/visualcoding)

Refactored and eanhanced by Mathworks at https://github.com/emeyers/Brain-Observatory-Toolbox

All data accessed by this toolbox comes from the Allen Brain Observatory © 2016 Allen Institute for Brain Science. Allen Brain Observatory. Available from: http://observatory.brain-map.org/visualcoding

This engineering work was supported by the Foundation of Psychocultural Research and Sherman Fairchild Award at Hampshire College. We are also grateful to Mathworks for their advice.


text:
#AATTOOLS

##Info

The goal of the project is to provide some common tools used by Swift Projects

Tipically: Extensions, Protocols, CustomViewComponents

If the project grows I will create a framework/pod

Feel free to comment / add functionalities ;)

##DOCS

There's a script generateJazzyDocs.sh to generate the Doc with Jazzy. @see [Jazzy](https://github.com/realm/jazzy)

You can see the [doc](http://www.albertarroyo.com/ios/swift/AATTools/docs/swift_output/index.html) generated. Included in the repository in doc folder.

##Introduction

The app will use different codes that I found interesting.

Some inspiration fonts:

- [EZSwiftExtensions](https://github.com/goktugyil/EZSwiftExtensions) -> good extensions with a mantained repository

##Requirements

- iOS 9.0+
- Xcode 8
- Swift 3

##License

This project is released under the [MIT license](https://github.com/jimmyaat10/AATTools/blob/master/LICENSE).


text:
# Usage examples
> Note: The following examples assume the use of Kotlin. If you are working with Java, please note the [small differences](./JAVA_API.md) in usage.
## Creating a user

Creating a user should be the first step when using Tink Link.

### Creating a user with access token

If you have an existing `accessToken`, you can create a `User` instance by passing the `accessToken` as input.

```kotlin
val user = User.fromAccessToken("yourAccessToken")
// Set this user to Tink.
Tink.setUser(user)
```

### Authenticating a permanent user

Creating and authenticating permanent users is limited to our Enterprise customers. A permanent user has the option to add multiple credentials and also refresh manually-refreshable credentials.
If you wish to support these features in your application, you will need to authenticate your permanent users through Tink Link, as shown below:

1. Create a permanent user and generate a code for the created user by following the steps mentioned in the [permanent user guide](https://docs.tink.com/resources/tutorials/permanent-users).
2. Authenticate your permanent user with the generated code.

```kotlin
Tink.authenticateUser(
    authenticationCode = "yourCode",
    resultHandler = ResultHandler(
        onSuccess = { user: User ->
            // Set this user to the TinkLink instance.
            Tink.setUser(user)
        },
        onError = { error ->
            // Handle error
        }
    )
)
```
### Setting a user to TinkLink instance

Once a permanent user is authenticated, the user must be set passed to `Tink`. Once this is done, you can perform operations related to the user through the `UserContext` object.
You can access the `UserContext` object through the `Tink.getUserContext()`. Please note that this will return `null` if no user has been set.

```kotlin
Tink.setUser(user)
val userContext = Tink.getUserContext()
```

## Handling user consent

### Getting links to Terms and Conditions and Privacy Policy

In order to show Tink Terms and Conditions and Privacy Policy to the user, you are going to need to:

1. Get the links:

```kotlin
val consentContext = Tink.getConsentContext()
val locale = Locale.ENGLISH // Use the locale of your choice

val termsUri = consentContext.termsAndConditions(locale)
val privacyPolicyUri = consentContext.privacyPolicy(locale)
```

2. Make the contents of the `Uri`'s available to the users, and require the users to give their consent in order to move forward.

### Showing scope descriptions

If aggregating under Tink’s license the user must be informed and fully understand what kind of data will be aggregated before aggregating any data:

```kotlin
val consentContext = Tink.getConsentContext()
consentContext.scopeDescriptions(
    setOf(Scope.AccountsRead, Scope.TransactionsRead), // See the [Scope] class for more scopes
    ResultHandler(
        { list ->
            // Show the list of scope descriptions to the user.
            // The list items contain a title and a description and are of type [ScopeDescription]
            // See the [ScopeDescription] class for further info.
        },
        { error ->
            // Handle error
        }
    )
)
```

## Listing providers

```kotlin
Tink.getUserContext()?.providerRepository.listProviders(
    ResultHandler(
        onSuccess = { list ->
            // Proceed with using the provider list, for example:
            yourAdapter.items = list
        },
        onError = { error ->
            // Handle error
        }
    )
)
```

## Field setup and validation

The Tink platform will specify how certain values are supposed to be filled by the user. For a good UX, make sure to validate input on the frontend and return helpful hints.

### Setup inputs for field specifications from the `Provider`
```kotlin
// Choose a provider from the list
val provider = providerList.first()

// Setup fields
val inputFields =
    provider.fields.map { field ->
        val editText = TextInputEditText(context).apply {

            // Prefill text
            setText(field.value)

            // Determine input type
            if (field.attributes.inputType.isNumeric) {
                inputType = InputType.TYPE_CLASS_NUMBER
            }

            if (field.attributes.inputType.isMasked) {
                transformationMethod = PasswordTransformationMethod.getInstance()
            }

            // Set maximum length
            filters = arrayOf(InputFilter.LengthFilter(field.validationRules.maxLength))

            // Fields might be immutable, meant for displaying data only
            enabled = !(field.attributes.inputType.isImmutable && field.value.isNotEmpty())
        }

        val helpView = TextView(context).apply {
            text = field.attributes.helpText
        }

        TextInputLayout(context).apply {
            hint = field.attributes.description
            helperText = field.attributes.hint +
                field.attributes.helpText.takeUnless { it.isEmpty() }?.let { "\n\n$it" }.orEmpty()
            //or create your own view for the help text

            addView(editText) // [1]
        }
    }
```
`[1]` _Note: This is for demonstration purpose only. Programmatically creating and adding views like here is not advisable. A real implementation should use a `RecyclerView` or similar best practise._

### Validate fields

```kotlin
// Fill fields with user input
val filledField = field.copy(value = inputField.text.toString())

// Validate fields
val result = filledField.validate()

when(result) {
    ValidationResult.Valid -> print(result) // Valid field input
    is ValidationResult.ValidationError.MinLengthLimit -> showError(result.errorMessage)
    is ValidationResult.ValidationError.MaxLengthLimit -> showError(result.errorMessage)
    is ValidationResult.ValidationError.Invalid -> showError(result.errorMessage)
}

fun showError(message: String) {
    inputField.error = message
}
```

## Creating credentials
```kotlin

// Convert your field list to a map
val fieldsMap = yourFilledFieldsList.associate { it.name to it.value }

Tink.getUserContext()?.credentialRepository.create(
    provider.name,
    provider.credentialType,
    fieldsMap,
    ResultHandler(
        onSuccess = { credential ->
            // Start observing the credential status
        },
        onError = { error ->
            // Handle error
        }
    )
)
```

## Observing credentials
```kotlin
Tink.getUserContext()?.credentialRepository.listStream().subscribe(
    object : StreamObserver<List<Credential>> {
        override fun onNext(value: List<Credential>) {
            // Handle list updates. For example:
            value
                .find { it.id = "idOfYourCredential" }
                ?.status?.let {
                    when(it) {
                        Credential.Status.UPDATED -> { /* Done, credential is added */ }
                        Credential.Status.UPDATING -> { /* Updating, statusPayload will contain more info */ }

                        Credential.Status.AWAITING_MOBILE_BANKID_AUTHENTICATION,
                        Credential.Status.AWAITING_THIRD_PARTY_APP_AUTHENTICATION -> { /* [1] */ }

                        Credential.Status.TEMPORARY_ERROR,
                        Credential.Status.AUTHENTICATION_ERROR,
                        Credential.Status.PERMANENT_ERROR -> { /* Something went wrong, handle error */ }
                        else -> {}
                    }
                }
        }
    }
)
```
`[1]` Follow the [third party authentication guide](/third-party-authentication.md) for handling this flow.

repository: maggud/tink-link-android

gha_language: Kotlin

repo_name: maggud/tink-link-android


text:
---
published: true
layout: post
title: EC2 LAMP Stack Implementation Options and Considerations
---

I was just starting to look into ways to improve our LAMP stack implementation on EC2, but I need to clean up and concentrate on some other tasks. This is what I found so far:

* [Install LAMP Stack, Download LAMP Stack](https://bitnami.com/stack/lamp/installer)
* [Bitnami AMP Stacks - Bitnami documentation](https://wiki.bitnami.com/Infrastructure_Stacks/Bitnami_AMP_Stacks)
* [Bitnami Sign in](https://bitnami.com/sign_in)
* [(2) Can anyone recommend a good, public AMI for a LAMP stack on Amazon's EC2? - Quora](https://www.quora.com/Can-anyone-recommend-a-good-public-AMI-for-a-LAMP-stack-on-Amazons-EC2)
* [(2) What are some tips for deploying and managing a LAMP stack on Amazon EC2? - Quora](https://www.quora.com/What-are-some-tips-for-deploying-and-managing-a-LAMP-stack-on-Amazon-EC2)
* [(2) What's a good, clean, Linux base AMI for AWS EC2? - Quora](https://www.quora.com/What-s-a-good-clean-Linux-base-AMI-for-AWS-EC2)
* [(2) What is the checklist for hosting a website on Amazon EC2 using LAMP stack? - Quora](https://www.quora.com/What-is-the-checklist-for-hosting-a-website-on-Amazon-EC2-using-LAMP-stack)
* [LAMP Stack Cloud Hosting, LAMP Stack Hosting - Installers and VM](https://bitnami.com/stack/lamp)

---

I also went back to basics a bit and started reading some amazon literature which does seem to have been updated since years ago when I started fiddling with EC2. I should read this stuff again to make sure I'm not making silly mistakes of oversight.

* [Tutorial: Installing a LAMP Web Server on Amazon Linux - Amazon Elastic Compute Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html)
* [Getting Started with Amazon EC2 Linux Instances - Amazon Elastic Compute Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html#ec2-launch-instance_linux)

---

Now, to actually set up an ec2 instance with apache and php, using mysql from an rds instance, I followed these guides:

* [Tutorial: Installing a LAMP Web Server on Amazon Linux - Amazon Elastic Compute Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html) (but didn't install or configure mysql server)

* [Tutorial: Configure Apache Web Server on Amazon Linux to use SSL/TLS - Amazon Elastic Compute Cloud](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-an-instance.html) (but just using a self-signed cert and amazon's default public dns name)


text:
:include-markdown: installation.md

# Bare Minimum

:include-file: examples/scenarios/rest/simpleGet.groovy {title: "examples/scenarios/rest/simpleGet.groovy"}

To run test, navigate to `examples` dir and

:include-cli-command: webtau scenarios/rest/simpleGet.groovy --url=https://my-server {paramsToHighlight: "url"}

:include-markdown: common/note-package-import.md

# Config File

Url parameter can be moved to a `webtau.cfg` file.

:include-file: examples/scenarios/rest/urlOnly.cfg {title: "examples/scenarios/rest/webtau.cfg"}

[Specify multiple environments](configuration/environments) to streamline test execution.

repository: shubicus/webtau

repo_name: shubicus/webtau


text:
### [CVE-2016-0704](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0704)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

An oracle protection mechanism in the get_client_master_key function in s2_srvr.c in the SSLv2 implementation in OpenSSL before 0.9.8zf, 1.0.0 before 1.0.0r, 1.0.1 before 1.0.1m, and 1.0.2 before 1.0.2a overwrites incorrect MASTER-KEY bytes during use of export cipher suites, which makes it easier for remote attackers to decrypt TLS ciphertext data by leveraging a Bleichenbacher RSA padding oracle, a related issue to CVE-2016-0800.

### POC

#### Reference
- http://www.oracle.com/technetwork/security-advisory/cpujan2018-3236628.html
- http://www.oracle.com/technetwork/topics/security/bulletinapr2016-2952098.html
- http://www.oracle.com/technetwork/topics/security/linuxbulletinjan2016-2867209.html
- https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA40168

#### Github
- https://github.com/ARPSyndicate/cvemon
- https://github.com/Live-Hack-CVE/CVE-2016-0704
- https://github.com/chnzzh/OpenSSL-CVE-lib
- https://github.com/imhunterand/hackerone-publicy-disclosed

repository: trickest/cve

branch_name: refs/heads/main

repo_name: trickest/cve


text:
Heroku self ping
================

To avoid the one-hour time limit on your dyno, use this npm module:

```js
require('heroku-self-ping')("http://your-app-url");
```

This will register a timer self-pinging your app every 45 minutes.

This timer will only activate when your app is running on Heroku platform. The script is a no-op on any other environment, including local and CI.

## Parameters
* `your-app-url` can be any URL to access your app -- an `herokuapp.com` subdomain or any custom domain defined with your app.

> Heroku doesn't allow an app to retrieve its current name, which is why you need to manually specify the URL.

*  `options`, optional object with the following keys:
    * `interval` number of ms between two heartbeats calls. Default to `45 * 60 * 1000` (45 minutes).
    * `logger` function to use for logging. Default to `console.log`
    * `verbose` send more information to logger function. Default to `false`.

## Return
The function returns an [interval id](https://developer.mozilla.org/en/docs/Web/API/window.setInterval) when running on Heroku and the URL is not falsy, and `false` in other cases.

## Usage
We recommend you set a custom env variable, for instance `heroku config:set APP_URL http://yourapp.herokuapp.com` and access it later from `process.env.APP_URL`.

The lib won't do anything when passed an empty `url`.


text:
# kOps addons

kOps supports two types of addons:

 * Managed addons, which are configurable through the [cluster spec](cluster_spec.md)
 * Static addons, which are manifest files that are applied as-is

## Managed addons

The following addons are managed by kOps and will be upgraded following the kOps and kubernetes lifecycle, and configured based on your cluster spec. kOps will consider both the configuration of the addon itself as well as what other settings you may have configured where applicable.

### Available addons

#### AWS Load Balancer Controller
{{ kops_feature_table(kops_added_default='1.20') }}

AWS Load Balancer Controller offers additional functionality for provisioning ELBs.

```yaml
spec:
  awsLoadBalancerController:
    enabled: true
```

Read more in the [official documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/).

#### Cluster autoscaler
{{ kops_feature_table(kops_added_default='1.19') }}

Cluster autoscaler can be enabled to automatically adjust the size of the kubernetes cluster.

```yaml
spec:
  clusterAutoscaler:
    enabled: true
    expander: least-waste
    balanceSimilarNodeGroups: false
    scaleDownUtilizationThreshold: 0.5
    skipNodesWithLocalStorage: true
    skipNodesWithSystemPods: true
    newPodScaleUpDelay: 0s
    scaleDownDelayAfterAdd: 10m0s
    image: <the latest supported image for the specified kubernetes version>
    cpuRequest: "100m"
    memoryRequest: "300Mi"
```

Read more about cluster autoscaler in the [official documentation](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler).

##### Disabling cluster autoscaler for a given instance group
{{ kops_feature_table(kops_added_default='1.20') }}

You can disable the autoscaler for a given instance group by adding the following to the instance group spec.

```yaml
spec:
  autoscale: false
```

#### Cert-manager
{{ kops_feature_table(kops_added_default='1.20', k8s_min='1.16') }}

Cert-manager handles x509 certificates for your cluster.

```yaml
spec:
  certManager:
    enabled: true
```

**Warning: cert-manager only supports one installation per cluster. If you are already running cert-manager, you need to remove this installation prior to enabling this addon. As long as you are using v1 versions of the cert-manager resources, it is safe to remove existing installs and replace it with this addon**

Read more about cert-manager in the [official documentation](https://cert-manager.io/docs/)

#### Metrics server
{{ kops_feature_table(kops_added_default='1.19') }}

Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.

```yaml
spec:
  metricsServer:
    enabled: true
```

Read more about Metrics Server in the [official documentation](https://github.com/kubernetes-sigs/metrics-server).

##### Secure TLS

{{ kops_feature_table(kops_added_default='1.20') }}

By default, API server will not verify the metrics server TLS certificate. To enable TLS verification, set the following in the cluster spec:

```yaml
spec:
  certManager:
    enabled: true
  metricsServer:
    enabled: true
    insecure: false
```

This requires that cert-manager is installed in the cluster.

#### Node local DNS cache
{{ kops_feature_table(kops_added_default='1.18', k8s_min='1.15') }}

NodeLocal DNSCache can be enabled if you are using CoreDNS. It is used to improve the Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet.

`memoryRequest` and `cpuRequest` for the `node-local-dns` pods can also be configured. If not set, they will be configured by default to `5Mi` and `25m` respectively.

If `forwardToKubeDNS` is enabled, kubedns will be used as a default upstream

```yaml
spec:
  kubeDNS:
    provider: CoreDNS
    nodeLocalDNS:
      enabled: true
      memoryRequest: 5Mi
      cpuRequest: 25m
```

#### Node termination handler

{{ kops_feature_table(kops_added_default='1.19') }}

Node Termination Handler ensures that the Kubernetes control plane responds appropriately to events that can cause your EC2 instance to become unavailable, such as EC2 maintenance events, EC2 Spot interruptions, ASG Scale-In, ASG AZ Rebalance, and EC2 Instance Termination via the API or Console. If not handled, your application code may not stop gracefully, take longer to recover full availability, or accidentally schedule work to nodes that are going down.

```yaml
spec:
  nodeTerminationHandler:
    enabled: true
```

## Static addons

The command `kops create cluster` does not support specifying addons to be added to the cluster when it is created. Instead they can be added after cluster creation using kubectl. Alternatively when creating a cluster from a yaml manifest, addons can be specified using `spec.addons`.
```yaml
spec:
  addons:
  - manifest: kubernetes-dashboard
  - manifest: s3://kops-addons/addon.yaml
```

This document describes how to install some common addons and how to create your own custom ones.

### Available addons

#### Ambassador

The [Ambassador API Gateway](https://getambassador.io/) provides all the functionality of a traditional ingress
controller (i.e., path-based routing) while exposing many additional capabilities such as authentication, URL rewriting,
CORS, rate limiting, and automatic metrics collection.

Install using:
```
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/ambassador/ambassador-operator.yaml
```

Detailed installation instructions in the [addon documentation](https://github.com/kubernetes/kops/blob/master/addons/ambassador/README.md).
See [Ambassador documentation](https://www.getambassador.io/docs/) on configuration and usage.

#### Dashboard

The [dashboard project](https://github.com/kubernetes/dashboard) provides a nice administrative UI:

Install using:
```
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.10.1.yaml
```

And then follow the instructions in the [dashboard documentation](https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above) to access the dashboard.

The login credentials are:

* Username: `admin`
* Password: get by running `kops get secrets kube --type secret -oplaintext` or `kubectl config view --minify`

##### RBAC

It's necessary to add your own RBAC permission to the dashboard. Please read the [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/) docs before applying permissions.

Below you see an example giving **cluster-admin access** to the dashboard.

```yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system
```

### Monitoring with Heapster - Standalone

**This addons is deprecated. Please use metrics-server instead**

Monitoring supports the horizontal pod autoscaler.

Install using:
```
kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/monitoring-standalone/v1.11.0.yaml
```
Please note that [heapster is retired](https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md). Consider using [metrics-server](https://github.com/kubernetes-incubator/metrics-server) and a third party metrics pipeline to gather Prometheus-format metrics instead.

### Monitoring with Prometheus Operator + kube-prometheus

The [Prometheus Operator](https://github.com/coreos/prometheus-operator/) makes the Prometheus configuration Kubernetes native and manages and operates Prometheus and Alertmanager clusters. It is a piece of the puzzle regarding full end-to-end monitoring.

[kube-prometheus](https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus) combines the Prometheus Operator with a collection of manifests to help getting started with monitoring Kubernetes itself and applications running on top of it.

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/prometheus-operator/v0.26.0.yaml
```

### Route53 Mapper

**This addon is deprecated. Please use [external-dns](https://github.com/kubernetes-sigs/external-dns) instead.**

Please note that kOps installs a Route53 DNS controller automatically (it is required for cluster discovery).
The functionality of the route53-mapper overlaps with the dns-controller, but some users will prefer to
use one or the other.
[README for the included dns-controller](https://github.com/kubernetes/kops/blob/master/dns-controller/README.md)

route53-mapper automates creation and updating of entries on Route53 with `A` records pointing
to ELB-backed `LoadBalancer` services created by Kubernetes. Install using:

The project is created by wearemolecule, and maintained at
[wearemolecule/route53-kubernetes](https://github.com/wearemolecule/route53-kubernetes).
[Usage instructions](https://github.com/kubernetes/kops/blob/master/addons/route53-mapper/README.md)

```
kubectl apply -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/route53-mapper/v1.3.0.yml
```

### Custom addons

The docs about the [addon management](contributing/addons.md#addon-management) describe in more detail how to define a addon resource with regards to versioning.
Here is a minimal example of an addon manifest that would install two different addons.

```yaml
kind: Addons
metadata:
  name: example
spec:
  addons:
  - name: foo.addons.org.io
    version: 0.0.1
    selector:
      k8s-addon: foo.addons.org.io
    manifest: foo.addons.org.io/v0.0.1.yaml
  - name: bar.addons.org.io
    version: 0.0.1
    selector:
      k8s-addon: bar.addons.org.io
    manifest: bar.addons.org.io/v0.0.1.yaml
```

In this example the folder structure should look like this;

```
addon.yaml
  foo.addons.org.io
    v0.0.1.yaml
  bar.addons.org.io
    v0.0.1.yaml
```

The yaml files in the foo/bar folders can be any kubernetes resource. Typically this file structure would be pushed to S3 or another of the supported backends and then referenced as above in `spec.addons`. In order for master nodes to be able to access the S3 bucket containing the addon manifests, one might have to add additional iam policies to the master nodes using `spec.additionalPolicies`, like so;
```yaml
spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject"
          ],
          "Resource": ["arn:aws:s3:::kops-addons/*"]
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket"
          ],
          "Resource": ["arn:aws:s3:::kops-addons"]
        }
      ]
```
The masters will poll for changes in the bucket and keep the addons up to date.

repository: ahmeddrawy/kops

repo_name: ahmeddrawy/kops


text:
---
title: "Configuration"
description: ""
weight: 3
---

{{% tts %}} can be started for development without passing any configuration. However, there are a number of things you need to configure for a production deployment. In this guide we'll set environment variables in the `docker-compose.yml` file to configure {{% tts %}} as a private deployment on `thethings.example.com`.

## Databases

We need to configure an SQL database, in this guide we'll use a single instance of [CockroachDB](https://www.cockroachlabs.com/). Make sure to find a recent tag of the [cockroachdb/cockroach image on Docker Hub](https://hub.docker.com/r/cockroachdb/cockroach/tags) and update it in the `docker-compose.yml` file. Make sure that the `volumes` are set up correctly so that the database is persisted on your server's disk.

The simplest configuration for CockroachDB will look like this:

```yaml
cockroach:
  image: 'cockroachdb/cockroach:<the tag>'
  command: 'start --http-port 26256 --insecure'
  restart: 'unless-stopped'
  volumes:
    - './data/cockroach:/cockroach/cockroach-data'
```

Read the CockroachDB documentation for setting up a highly available cluster.

We also need to configure [Redis](https://redis.io/), in this guide we'll use a single instance of Redis. Just as with the SQL database, find a recent tag of the [redis image on Docker Hub](https://hub.docker.com/_/redis?tab=tags) and update it in the `docker-compose.yml` file. Again, make sure that the `volumes` are set up correctly so that the datastore is persisted on your server's disk.

The simplest configuration for Redis will look like this:

```yaml
redis:
  image: 'redis:<the tag>'
  command: 'redis-server --appendonly yes'
  restart: 'unless-stopped'
  volumes:
    - './data/redis:/data'
```

Read the Redis documentation for setting up a highly available cluster.

## {{% tts %}}

Before we go into the details of {{% tts %}}'s configuration, we'll take a look at the basics. Below you see part the configuration of the `stack` service in the `docker-compose.yml` file. As with the databases, you need to find a recent tag of the [thethingsnetwork/lorawan-stack image on Docker Hub](https://hub.docker.com/r/thethingsnetwork/lorawan-stack/tags) and update the `docker-compose.yml` file with that. We tell Docker Compose to start the container with `ttn-lw-stack start`, we indicate that it depends on CockroachDB and Redis, and we configure a volume for storing blobs (such as profile pictures).

The `ports` section exposes {{% tts %}}'s ports to the world. Port `80` and `443` are mapped to the internal HTTP and HTTPS ports. The other ports have a direct mapping. If you don't need support for gateways and applications that don't support TLS, you can remove ports starting with `188`.

In the `environment` section, we configure the databases used by the stack. We will set these to the CockroachDB and Redis instances that are defined in the `docker-compose.yml` above.

```yaml
stack:
  image: 'thethingsnetwork/lorawan-stack:<the tag>'
  entrypoint: 'ttn-lw-stack -c /config/ttn-lw-stack.yml'
  command: 'start'
  restart: 'unless-stopped'
  depends_on:
    - 'cockroach'
    - 'redis'
  volumes:
    - './data/blob:/srv/ttn-lorawan/public/blob'
    - './config/stack:/config:ro'
    # If using Let's Encrypt
    - './acme:/var/lib/acme'
  ports:
    - '80:1885'
    - '443:8885'
    - '1881:1881'
    - '8881:8881'
    - '1882:1882'
    - '8882:8882'
    - '1883:1883'
    - '8883:8883'
    - '1884:1884'
    - '8884:8884'
    - '1887:1887'
    - '8887:8887'
    - '1700:1700/udp'
  environment:
    TTN_LW_BLOB_LOCAL_DIRECTORY: '/srv/ttn-lorawan/public/blob'
    TTN_LW_REDIS_ADDRESS: 'redis:6379'
    TTN_LW_IS_DATABASE_URI: 'postgres://root@cockroach:26257/ttn_lorawan?sslmode=disable'

  # If using (self) signed certificates
  # secrets:
  #   - cert.pem
  #   - key.pem

# If using (self) signed certificates
# secrets:
#   cert.pem:
#     file: ./cert.pem
#   key.pem:
#     file: ./key.pem
```

Next, we'll have a look at the configuration options for your private deployment. We'll set these options in the `ttn-lw-stack.yml` file that is mounted as a volume on the `stack` service in `docker-compose.yml`.

First is TLS with Let's Encrypt. Since we're deploying {{% tts %}} on
`thethings.example.com`, we configure it to only request certificates for that
host, and also to use it as the default host.

We also configure HTTP server keys for encrypting and verifying cookies, as well
as passwords for endpoints that you may want to keep for internal use.

{{% tts %}} sends emails to users, so we need to configure how these are sent.
You can use Sendgrid or an SMTP server. If you skip setting up an email provider,
{{% tts %}} will print emails to the stack logs.

Finally, we also need to configure the URLs for the Web UI and the secret used
by the console client.

For more detailed configuration, refer to [the relevant documentation]({{< ref "/reference/configuration" >}})

Below is an example `ttn-lw-stack.yml` file:

```yaml
# Example ttn-lw-stack configuration file

# Identity Server configuration
is:
  # Email configuration for "thethings.example.com"
  email:
    sender-name: 'The Things Stack'
    sender-address<EMAIL_ADDRESS>    network:
      name: 'The Things Stack'
      console-url: 'https://thethings.example.com/console'
      identity-server-url: 'https://thethings.example.com/oauth'

  # Web UI configuration for "thethings.example.com":
  oauth:
    ui:
      canonical-url: 'https://thethings.example.com/oauth'
      is:
        base-url: 'https://thethings.example.com/api/v3'

# HTTP server configuration
http:
  cookie:
    # generate 32 bytes (openssl rand -hex 32)
    block-key: '0011223344556677001122334455667700112233445566770011223344556677'
    # generate 64 bytes (openssl rand -hex 64)
    hash-key: '00112233445566770011223344556677001122334455667700112233445566770011223344556677001122334455667700112233445566770011223344556677'
  metrics:
    password: 'metrics'               # choose a password
  pprof:
    password: 'pprof'                 # choose a password

# Let's encrypt for "thethings.example.com"
tls:
  source: 'acme'
  acme:
    dir: '/var/lib/acme'
    email<EMAIL_ADDRESS>    hosts: ['thethings.example.com']
    default-host: 'thethings.example.com'

# If Gateway Server enabled, defaults for "thethings.example.com":
gs:
  mqtt:
    public-address: 'thethings.example.com:1882'
    public-tls-address: 'thethings.example.com:8882'
  mqtt-v2:
    public-address: 'thethings.example.com:1881'
    public-tls-address: 'thethings.example.com:8881'

# If Gateway Configuration Server enabled, defaults for "thethings.example.com":
gcs:
  basic-station:
    default:
      lns-uri: 'wss://thethings.example.com:8887'
  the-things-gateway:
    default:
      mqtt-server: 'mqtts://thethings.example.com:8881'

# Web UI configuration for "thethings.example.com":
console:
  ui:
    canonical-url: 'https://thethings.example.com/console'
    is:
      base-url: 'https://thethings.example.com/api/v3'
    gs:
      base-url: 'https://thethings.example.com/api/v3'
    ns:
      base-url: 'https://thethings.example.com/api/v3'
    as:
      base-url: 'https://thethings.example.com/api/v3'
    js:
      base-url: 'https://thethings.example.com/api/v3'
    qrg:
      base-url: 'https://thethings.example.com/api/v3'
    edtc:
      base-url: 'https://thethings.example.com/api/v3'

  oauth:
    authorize-url: 'https://thethings.example.com/oauth/authorize'
    token-url: 'https://thethings.example.com/oauth/token'
    client-id: 'console'
    client-secret: 'console'          # choose or generate a secret (*)
```

(*) You will need the `console.oauth.client-secret` again in a later step.

repository: astonix/lorawan-stack

repo_name: astonix/lorawan-stack


text:
# WebAssembly DOOM fire

An implementation of the Doom fire effect described in
[Fabien Sanglard's blog][1], using hand-written WebAssembly.

[demo!](https://binji.github.io/doomfirewasm/)

[1]: http://fabiensanglard.net/doom_fire_psx/index.html

repository: edgardec/Nastya

repo_name: edgardec/Nastya