text:
Join us on  `slack <https://join.slack.com/t/pyopenclopt/shared_invite/enQtMjQyNjU3NDk1NTA5LWRiODY5NTY3N2Y4ODY3YmI5Zjg2NGI4Nzg1NTMzZDYyNjBkOTdhNzU4MzAxNDYwNmE2YzEzODMzNzQ5YWQyYzU>`_  (歡迎上 `slack <https://join.slack.com/t/pyopenclopt/shared_invite/enQtMjQyNjU3NDk1NTA5LWRiODY5NTY3N2Y4ODY3YmI5Zjg2NGI4Nzg1NTMzZDYyNjBkOTdhNzU4MzAxNDYwNmE2YzEzODMzNzQ5YWQyYzU>`_ 交流 !!)
===============
.. figure:: https://img.shields.io/badge/slack-n.n-pink.svg
    :target: https://pyopenclopt.slack.com/

OpenCLGA
===============
OpenCLGA is a python library for running genetic algorithm among Open CL devices, like GPU, CPU, DSP, etc. In the best case, you can run your GA parallelly at all of your Open CL devices which give you the maximum computing power of your machine. In the worse case, which you only have CPU, you still can run the code at parallel CPU mode.

We had implemented OpenCLGA at ocl_ga.py which encapsulate GA follow (population, crossover, mutation, fitness calculation). User could solve their problem by providing the fitness calculation function and run the code.

Please note that OpenCLGA is implemented at Python 3.5 or above. It should work at Python 2.x but it is not guaranteed.

Demo Video
==============================
Taiwan Travel example: https://youtu.be/4pqmuV8RkMg

Prerequisite: install PYOPENCL
==============================
Option A. Please refer to https://wiki.tiker.net/PyOpenCL to find your OS or

Option B. Install by ourself

- Windows 10 (just to install all required stuff in a quick way)

  * Step 1. Install platform opencl graphic driver, e.g. Intel CPU or Intel HD Grahics (https://software.intel.com/en-us/intel-opencl), NVIDIA GPU Driver (https://developer.nvidia.com/opencl)

  * Step 2. Install the following `*.whl` for python from

     1. http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy

     2. http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyopencl

- Ubuntu 16.04

  * Step 1. Install platform OpenCL graphic driver, i.e.

     1. Intel CPU or Intel HD Graphics.

        a. `OpenCL™ 2.0 GPU/CPU driver package(SRB_4.1) for Linux* (64-bit) <https://software.intel.com/en-us/articles/opencl-drivers>`_

        b. `Installation instructions. <https://software.intel.com/en-us/articles/sdk-for-opencl-gsg-srb41>`_

  * Step 2. Download `Intel SDK for Application (2016 R3)& ICDs <https://software.intel.com/en-us/intel-opencl/download>`_ and install, if you have Intel devices : ::

      $> sudo apt-get install libnuma1 alien
      $> tar -xvf ./intel_sdk_for_opencl_2016_ubuntu_<IP_ADDRESS>4_x64.tgz
      $> cd ./intel_sdk_for_opencl_2016_ubuntu_<IP_ADDRESS>4_x64/rpm
      $> sudo alien opencl-2.1-intel-cpu-exp-<IP_ADDRESS>4-1.x86_64.rpm
      $> sudo dpkg -i opencl-2.1-intel-cpu-exp_<IP_ADDRESS>4-2_amd64.deb
      $> sudo apt-get install clinfo
      // To verify platform information from OpenCL
      $> clinfo

    You can verify the installed OpenCL driver is located in /opt/intel/ and the ICD loader is located in /etc/OpenCL/vendors.

  * Step 3. Create a virtual environment for pyopencl. ::

      // Make sure dependencies for building wheel is available on system.
      $> sudo apt-get install build-essential libssl-dev libffi-dev python-dev
      $> sudo apt-get install python3-pip python3-venv python3-tk ocl-icd-*
      $> python3 -m venv [NameOfEnv]
      $> source ./NameOfEnv/bin/activate
      <NameOfEnv>$> pip3 install --upgrade pip
      <NameOfEnv>$> pip3 install pyopencl

  * Step 4. Verification ::

      <NameOfEnv>$> python3
      > import pyopencl as cl
      > cl.create_some_context()

- Mac OS X

  * Step 1.
    Install Python3: since OpenCL drivers had already included in Mac OS X, we don't need to install any OpenCL driver by ourself. So, we can start from Python3. ::

      $> brew update
      $> brew install python3
      $> pip3 install virtualenv

    *Note that you may not need to install virtualenv if you already installed it with python 2.7.*

  * Step 2. Create a virtual environment for pyopencl: before install pyopencl, we may need to install XCode developer console tool with `xcode-select --install` command. If you already had it, you don't need to run it. ::

      $> python3 -m venv [NameOfEnv]
      $> source ./NameOfEnv/bin/activate
      <NameOfEnv>$> pip3 install --upgrade pip
      <NameOfEnv>$> pip3 install pycparser cffi numpy wheel
      <NameOfEnv>$> pip3 install pyopencl

  * Step 3. Verification. ::

      <NameOfEnv>$> python3
      > import pyopencl as cl
      > cl.create_some_context()

Run OpenCLGA examples
==============================
1. Enter virtual env (optional):

* For Windows with MinGW environment. ::

      $> source <NameOfEnv>/Scripts/activate

* For Linux/Mac OS X environment. ::

      $> source ./NameOfEnv/bin/activate

2. Download the code from `Github <https://github.com/PyOCL/OpenCLGA/archive/master.zip>`_ or git clone the repository via the following command. ::

      <NameOfEnv>$> git clone https://github.com/PyOCL/OpenCLGA.git

3. Install Extra package (optional) :

* For Linux environment. ::

      // To make matplotlib display correctly.
      $> sudo apt-get install python3-tk

4. Execute the code. ::

      <NameOfEnv>$> pip3 install git+git://github.com/PyOCL/OpenCLGA.git
      <NameOfEnv>$> unzip OpenCLGA-master.zip
      <NameOfEnv>$> cd OpenCLGA-master
      <NameOfEnv>$> python3 examples/tsp/simple_tsp.py

  *NOTE : In external process mode, if "no device" exception happen during create_some_context(), Please set PYOPENCL_CTX=N (N is the device number you want by default) at first.*

  *NOTE : Since we didn't publish this project to pipa. We need to install this project with source, `pip3 install .`.*

repository: PyOCL/OpenCLGA

extension: rst

repo_name: PyOCL/OpenCLGA


text:
# Flicker Fusion Threshold
### Embedded system assignment 0

## AIM

Set up a prototype of an Embedded System for the measurement of a person’s subjective flicker fusion threshold using Teensy 3.2 board. An LED with varying blinking frequency is used as stimulus and when it reaches the threshold flicker fusion frequency, the current frequency should be displayed in a serial monitor on a connected PC.

## INTRODUCTION

*"The **flicker fusion threshold**, is a concept in the psycho-physics of vision. It is defined as the frequency at which an intermittent light stimulus appears to be completely steady to the average human observer. Flicker fusion threshold is related to persistence of vision."*
##### Flicker Fusion Threshold https://en.wikipedia.org/wiki/Flicker_fusion_threshold
In this project I used a Teensy 3.2, a push button, 390 ohm resistor and LED to create a prototype. The frequency of LED blinking is continuously increasing in my prototype.  The flicker fusion frequency is displayed using serial monitor of VSCode when the push button is pressed.

## METHODS
### Hardware Description

#### Components

 1. Teensy 3.2
 2. 390 ohm resistor
 3. Push button
 4. Wires
 5. Breadboard

#### Pull up resistor calculation

The equation to find the resistance of a pullup resistor
R = (Vs-Vf)/If
According to the [Teensy 3.2 data card](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Datasheet/Teensy%203_2%20card%20%20.pdf)the input voltage of Teensy board is 3.3 - 6 Volts.The output voltage of Teensy 3.2 board is 3.3V. The typical Vf of a red LED is 1.8 to 2.1Volts. The safe operating forward current through this led is taken as 10mA.

 Then,

(3.3-1.8)/10mA = 150ohm

Therefore for the safe operation of LED the pullup resistance should be greater than 150 ohm. The pullup resistor used in this circuit is 390 ohm.

#### Push Button

The mechanical push button may produce errors in switching therefore we have written codes for debouncing and enabled input pullup resistor internally in teensy 3.2.

### Circuit
The push button is connected to the D5 pin of Teensy 3.2 and GND pin. Led is connected to the D12 pin and GND.
It is also explained in this Youtube video:
https://youtu.be/GUNod7JRGfk

![ Breadboard representation of circuit](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Circuit/Flickerfusion%20breadboard.png)
![Schematic Diagram](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Circuit/Flickerfusion%20Schematic.png)
### [Software](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Flicker_fusion/src/main.cpp)
 VSCode with Platform IO extension is used to code in this project. The pin 12 and 5 is defined as *ledpin* and *buttonpin* respectively. Internal pullup resistor is enabled by `pinMode(buttonpin, INPUT_PULLUP)` for the button pin. The frequency of led is varied from 0.5Hz to 90Hz using if loop. When the Flicker fusion frequency reaches, the person will press the button and it will trigger the interrupt. In the interrupt routine function, frequency at that time is displayed on the serial monitor along with time period in millisecond. Interrupt is triggered on the falling edge of the switching action.

### Output Waveform Display
The waves generated is, by serial printing the led state and plotting it using [Serialplot](https://hackaday.io/project/5334-serialplot-realtime-plotting-software) software.
## [RESULT](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/tree/master/Output)
The flicker fusion threshold frequency is printed on the serial monitor when the button is pressed.
![Screenshot of VSCode serial monitor](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Output/Serial%20monitor%20Screenshot.jpg)

The output is verified by plotting the led state by using Serial plot software.
![Screenshot of Serialplot software at iption hereherehere](https://github.com/jomaljose/Flicker_Fusion_Assignment_0/blob/master/Output/Serial%20Plot%20output_waveform.png)
This waveform is plotted at 35Hz frequency. The plotted waveform consist of 4 samples in every wave cycle. In the above figure at right corner of the serial plot window SPS (samples per second) is shown. By dividing the SPS by 4 we get the frequency of the plotted waveform.
In this case the SPS is 138
Therefore the frequency of the plotted waveform will be 138/4 = **34.5Hz**.
## FUTURE SCOPE & LIMITATIONS
The current prototype is giving flicker fusion threshold frequency value almost similar to the theoretical value of human flicker fusion threshold frequency. Eventhough,the plotted waveform have some error with the actual serial printed frequency. In future I like to eliminate this error and make it available for primary school teachers, because it will be helpful for them to teach persistence of vision.

## CONCLUSION

Human  flicker fusion threshold was tested successfully using this prototype. My conclusion after several tests is that my flicker fusion threshold frequency is between 37-39.5Hz.

This project gave me the opportunity to get into GitHub, Teensy 3.2 and different IDEs etc.

## REFERENCES

 1. Wikipedia contributors. (2020, January 29). Flicker fusion threshold. Retrieved from https://en.wikipedia.org/wiki/Flicker_fusion_threshold
 2. Teensyduino: Using the UART (real serial) with Teensy on the Arduino IDE. (2020). Retrieved from https://www.pjrc.com/teensy/td_uart.html
 3. Vishay Semiconductors. (2019). _TLUR6400, TLUR6401 datasheet_ (2019 revised ed.), Vishay Semiconductors.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# Report

## Learning Algorithm

To solve Collaboration and Competition project, I used as a start point the code from my https://github.com/jigor/DRLND_Project2_Continuous_Control project that implements Deep Deterministic Policy Gradients (DDPG) algorithm. Original code is customized to work with multi agent environment and implement MADDPG algorithm. Main reference for customization was paper https://arxiv.org/pdf/1706.02275.pdf.

DDPG is using the actor-critic architecture, Actor is used to decide the best action for a specific state and the Critic is used for evaluating action. DDPG is also using target network with soft update for both Actor and Critic and experience replay buffer.

Ornstein-Uhlenbeck random process is used for adding noise to action so the agent has more exploration of action space. Parameters of Ornstein-Uhlenbeck process are

```Python
seed = 2
mu = 0.0
theta = 0.25
sigma = 0.35
```
In this project I am using Tennis environment with 2 agents. The main part of the program is in following files

```Python
ddpg_agent.py
model.py
Tennis.ipynb
```

*ddpg_agent.py* code implements an environment-aware agent, while in *model.py* are the neural network models of Actor and Critic.
*Tennis.ipynb* contains a code that trains a agent and displays results.

#### MADDPG  algorithm with Actor - Critic networks
The model of the neural network used is shown in the picture:

![Network model](./images/model_agent.png  "Network model")

I used a two layers network for the Actor

The size of the hidden layers are:
FC1 size = 400
FC2 size = 400

The input parameter is the state of the environment (size 24) and the output is action (size 2).

For the Critic, I also used a two layers network
FCS1 size = 400
FC2 size = 400

Input parameters are states (size 48) in first layer and actions (size 4)  in second layer and output is Q  value (size 1).

The activation function is *relu* on output of all hidden layers of Actor, except on the last layer of Actor where I used tanh (because the action values are in range [-1.1]). Critic is using *leaky_relu* activation function in all layers except the last layer, where I did not use any activation function because the output is Q value.

The hyper parameters are:

```Python
BUFFER_SIZE = int(1e5)  # replay buffer size
BATCH_SIZE  = 128       # minibatch size
GAMMA = 0.99            # discount factor
TAU = 1e-3              # for soft update of target parameters
LR_ACTOR = 1e-4         # learning rate of the actor
LR_CRITIC = 1e-4        # learning rate of the critic
WEIGHT_DECAY = 0        # L2 weight decay
WARMUP_TIME = 1024     # initial experience buffer fill time
```

Learning rates of Actor and Critic are decreased for this project to stabilize learning. *WARMUP_TIME* is much lower than in base project. I was experimenting with changing TAU hyperparameter and with using hard update for target networks, but that did not give a result.
Other hyper parameters have the same values as in the original project.

As the paper [1706.02275](https://arxiv.org/pdf/1706.02275.pdf) propose, I have changed DDPG's agent, so Actor only has insight into its observation, while Critic has insight into the state and action of both agents, which can be seen in the picture underneath from mentioned paper.

![Plot of rewards](./images/model_from_1706.02275.PNG  "Overview of multi-agent decentralized actor, centralized critic approach")

The Critic in MADDPG learns a centralized action-value function
$$
Qi(o⃗ ,a1,…,aN)
$$
for the i-th agent, where
$$
a1∈A1,…,aN∈AN
$$

$$
o⃗ =o1,…,oN
$$

are actions and observations of all agents. Each Qi is learned separately for i=1,…,N and therefore multiple agents can have arbitrary rewards, including conflicting rewards in a competitive setting. At the same time multiple actors, one for each agent, are exploring and upgrading the policy on their own.

Algorithm solves environment in 3025 episodes. Partial log and plot of rewards can be seen below:

	Episode 100	Average Score: 0.00	 Score: [ 0.   -0.01]
	Episode 200	Average Score: 0.01	 Score: [-0.01  0.  ]
	.
	.
	.
	Episode 1000	Average Score: 0.11	 Score: [0.1  0.19]
	.
	.
	.
	Episode 2000	Average Score: 0.17	 Score: [0.2  0.09]
	.
	.
	.
	Episode 3025	Average Score: 0.50	 Score: [1.89000003 2.00000003]
	Environment solved in 3025 episodes!	Average Score: 0.50

![Plot of rewards](./images/scores.png  "Plot of rewards")

## Ideas for Future Work
In the next step I planed to try to optimize and speed up learning using network model with less parameters and better hyperparameters optimizations. Implementation of the Prioritized Experience Replay algorithm [https://arxiv.org/pdf/1511.05952](https://arxiv.org/pdf/1511.05952), I think will improve results also. After MADDPG optimization I will try to implement Proximal Policy Optimization (PPO) algorithm and to compare results with MADDPG.

## References

1. [https://arxiv.org/pdf/1706.02275.pdf](https://arxiv.org/pdf/1706.02275.pdf)
2. [https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#maddpg](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#maddpg)
3. [https://medium.com/@scitator/run-skeleton-run-3rd-place-solution-for-nips-2017-learning-to-run-207f9cc341f8](https://medium.com/@scitator/run-skeleton-run-3rd-place-solution-for-nips-2017-learning-to-run-207f9cc341f8)


text:
---
layout: post
title: "A note on patch-based low-rank minimization for fast image denoising"
date: 2018-02-21 03:14:36
categories: arXiv_CV
tags: arXiv_CV Sparse Attention
author: Haijuan Hu, Jacques Froment, Quansheng Liu
mathjax: true
---

* content
{:toc}

##### Abstract
Patch-based low-rank minimization for image processing attracts much attention in recent years. The minimization of the matrix rank coupled with the Frobenius norm data fidelity can be solved by the hard thresholding filter with principle component analysis (PCA) or singular value decomposition (SVD). Based on this idea, we propose a patch-based low-rank minimization method for image denoising. The main denoising process is stated in three equivalent way: PCA, SVD and low-rank minimization. Compared to recent patch-based sparse representation methods, experiments demonstrate that the proposed method is rather rapid, and it is effective for a variety of natural grayscale images and color images, especially for texture parts in images. Further improvements of this method are also given. In addition, due to the simplicity of this method, we could provide an explanation of the choice of the threshold parameter, estimation of PSNR values, and give other insights into this method.

##### Abstract (translated by Google)
基于补丁的低级别图像处理最小化近年来备受关注。可以通过具有主成分分析（PCA）或奇异值分解（SVD）的硬阈值滤波器来解决与Frobenius范数据保真度相结合的矩阵秩的最小化。基于这个思想，我们提出了一种基于块的低秩最小化图像去噪方法。主要的去噪过程以三种等效方式表示：PCA，SVD和低秩最小化。与最近基于补丁的稀疏表示方法相比，实验表明所提出的方法相当快速，并且对于各种自然灰度图像和彩色图像尤其是图像中的纹理部分是有效的。还给出了这种方法的进一步改进。此外，由于此方法的简单性，我们可以提供对阈值参数的选择，PSNR值的估计以及对该方法的其他见解的解释。

##### URL
[http://arxiv.org/abs/1506.08353](http://arxiv.org/abs/1506.08353)

##### PDF
[http://arxiv.org/pdf/1506.08353](http://arxiv.org/pdf/1506.08353)


text:
---
title: Sister Cities
author: L3viathan
profile_picture: /images/l3viathan.png
about_stub: Recreational linguistics, distributional gastronomics, and applied galettalogy.
author_site: https://github.com/L3viathan
---

This post is on so called "sister cities", also called "twin towns"; where
towns form symbolic partnerships with other towns. I was wondering how
connected towns are through this bond: Is it possible to start at one town, go
to one of its sister cities, then one of _its_ sister cities, and so on, to
eventually reach any other city.

To test this, I used [WikiData](https://www.wikidata.org), which conveniently
has a property type for sister cities
([P190](https://www.wikidata.org/wiki/Property:P190)).  I ignored the API
documentation, but found the [basic entity request URL for my
hometown](https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q3955&format=json).

In it, I searched for `P190` and found a list of entities, all refering to
partner cities of Weimar. Then I stopped with doing things manually and wrote a
short [Python
script](https://github.com/L3viathan/sister-cities/blob/master/get_graph.py) to
recursively crawl all towns "reachable" from my starting point.

A few hours later, I had a list of about 34k links between towns (in both
directions).  I also saved the names of the cities, so I didn't have to look at
the WikiData IDs. Afterwards, I wrote a [quick&dirty
script](https://github.com/L3viathan/sister-cities/blob/master/interactive.py)
to explore sister cities interactively.

The network seems to be pretty well connected, probably because:

- Horizontally: Big cities are linked to eachother. They also often have a high
  amount of connections, making them link smaller subnetworks.
- Vertically: Links aren't always between exqually sized cities, which ensures
  that also the smallest towns have a connection to the big ones.

Obviously, the graph that I collected is fully connected because of the method
of collection, but since the network is so big I assume it got all linked towns
or at least most of them.

You can look at the code and the data
[here](https://github.com/L3viathan/sister-cities/){:.capsule}, I appreciate
input or further work.

extension: markdown


text:
# urho

Using clojure 1.9.0 beta for specs.

http://clojure.org/guides/spec

## Prerequisites

You will need [Leiningen][] 2.0.0 or above installed.

[leiningen]: https://github.com/technomancy/leiningen

## Running

To start a web server for the application, run:

   lein run
   # or (better for development)
   lein ring server-headless

## Testing

    lein test

## Run local docker

### Build

    ./build.sh

### Run

    docker run -e urho

## License

Copyright © 2016 FIXME

repository: tanankoodarit/urho

repo_name: tanankoodarit/urho


text:
# preventing-exceptions-in-destructors

Prevent exceptions from leaving destructors. If you fear that a
destructor may have to call a function that may throw an error, provide
the user with an explicit function to “close” or “destruct” the object
where the user

can handle exceptions. Inside the destructor, should an exception arise
when calling that closing function (given the user didn’t do so before,
so have a private boolean member), either:

1.  Abort the program after logging some information. If the program
    can’t continue to function after an error was emitted inside the
    constructor, e.g. some vital things couldn’t be closed, terminate
    the program by calling std::abort().
2.  Silence, or “swallow”, the exception, by having a try-catch block
    inside the destructor for that closing function and then just
    logging that it threw an exception, like “Closing failed!”.

These two things above don’t actually solve the problem of handling
exceptions and giving the user options of how to react, but we already
do so by giving the user the chance to catch exceptions with the
explicit “close” function, so he or she

can’t complain if we don’t handle the exception in the destructor.

```
Provide a public interface for release or special work that throws exception.
Let client handle the exception. Keep destrucor simple.
```

There are two situations in which a destructor is called. The first is when an object is destroyed under "normal" conditions, e.g., when it goes out of scope or is explicitly deleted. The second is when an object is destroyed by the exception-handling mechanism during the stack-unwinding part of exception propagation. ¤

That being the case, an exception may or may not be active when a destructor is invoked. Regrettably, there is no way to distinguish between these conditions from inside a destructor.4 As a result, you must write your destructors under the conservative assumption that an exception is active, because if control leaves a destructor due to an exception while another exception is active, C++ calls the terminate function.

repository: CJDevWorks/cpp-notes

repo_name: CJDevWorks/cpp-notes


text:
Alerterator
===========

[![Github Actions](https://github.com/nais/alerterator/workflows/Build%20and%20deploy/badge.svg)](https://github.com/nais/alerterator/actions?query=workflow%3A%22Build+and+deploy%22)
[![Go Report Card](https://goreportcard.com/badge/github.com/nais/alerterator)](https://goreportcard.com/report/github.com/nais/alerterator)

Alerterator is a Kubernetes operator for managing Prometheus Alertmanager alerts. With this resource you can easily get
notified (either via Slack or email) when somethings is happening with your app.

As alerts are namespace agnostic, you don't have to have different files for each namespace you are running (although we
don't recommend running you application in different namespaces). You can even make your own personal alert-resources
that only notifes you!

The documentation for how to start using alerts are over at https://doc.nais.io/observability/alerts.

## Local testing

```
kind create cluster --image kindest/node:v1.17.11
kubeclt apply -f ./config/local-test/
make run

# different terminal
kubectl apply -f ./config/samples/alerts.yaml
kubectl get alerts
```

repository: nais/alerterator

gha_language: Go

gha_license_id: MIT

repo_name: nais/alerterator


text:
---
layout: post
title: Decision Tree classification
tags:
- F#
- Machine-Learning
- Decision-Tree
- Entropy
- ID3
- Classification
---

{% include ml-in-action-series.html %}

Today's topic will be Chapter 3 of "[Machine Learning in Action](http://www.manning.com/pharrington/)", which covers Decision Trees.

*Disclaimer: I am new to Machine Learning, and claim no expertise on the topic. I am currently reading [Machine Learning in Action](http://www.manning.com/pharrington/), and thought it would be a good learning exercise to convert the book's samples from Python to F#.*

The idea behind Decision Trees is similar to the [Game of 20 Questions](http://en.wikipedia.org/wiki/Twenty_Questions): construct a set of discrete Choices to identify the Class of an item. We will use the following dataset for illustration: imagine that we have 5 cards, each with a major masterpiece of contemporary cinema, classified by genre. Now I hide one - and you can ask 2 questions about the genre of the movie to identify the Thespian luminary in the lead role, in as few questions as possible:

  | Action | Sci-Fi | Actor
--- | --- | --- | ---
Cliffhanger | Yes | No | Stallone
Rocky | Yes | No | Stallone
Twins | No | No | Schwarzenegger
Terminator | Yes | Yes | Schwarzenegger
Total Recall | Yes | Yes | Schwarzenegger

The questions you would likely ask are:

* Is this a Sci-Fi movie? If yes, Arnold is the answer, if no,
* Is this an Action movie? if yes, go for Sylvester, otherwise Arnold it is.

![Deciding actor in 2 questions]({{ site.url }}/assets/2012-08-05-image_thumb_32.png)

That's a Decision Tree in a nutshell: we traverse a Tree, asking about features, and depending on the answer, we draw a conclusion or recurse deeper into more questions. The goal today is to let the computer build the right tree from the dataset, and use that tree to classify "subjects".

<!--more-->

## Defining a Tree

Let's start with the end - the `Tree`. A common and convenient way to model Trees in F# is to use a discriminated union like this:

``` fsharp
type Tree =
    | Conclusion of string
    | Choice of string * (string * Tree) []
```

A **`Tree`** is composed of either a **`Conclusion`**, described by a string, or a **`Choice`**, which is described by a string, and an Array of multiple options, each described by a string and its own `Tree`, "tupled".

For instance, we can manually create a tree for our example like this:

``` fsharp
let manualTree =
    Choice
        ("Sci-Fi",
         [|("No",
            Choice
              ("Action",
               [|("Yes", Conclusion "Stallone");
                 ("No", Conclusion "Schwarzenegger")|]));
           ("Yes", Conclusion "Schwarzenegger")|])
```

Our tree starts with a `Choice`, labeled "Sci-Fi", with 2 options in an Array, "No" or "Yes". "Yes" leads to a `Conclusion` (a Leaf node), Arnold, while "No" opens another `Choice`, "Action", with 2 Conclusions.

So how can we use this to Classify a "Subject"? We need to traverse down the Tree, check what branch corresponds to the Subject for the current Choice, and continue until we reach a Decision node, at what point we can return the contents of the Conclusion. To that effect, we'll represent a "Subject" (the thing we are trying to classify) as an collection of Tuples, each Tuple being a key/value pair, representing a Feature and its value:

``` fsharp
let test = [| ("Action", "Yes"); ("Sci-Fi", "Yes") |]
```

We are ready to write a classification function now:

``` fsharp
let rec classify subject tree =
    match tree with
    | Conclusion(c) -> c
    | Choice(label, options) ->
        let subjectState =
            subject
            |> Seq.find(fun (key, value) -> key = label)
            |> snd
        options
        |> Array.find (fun (option, tree) -> option = subjectState)
        |> snd
        |> classify subject
```

**`classify`** is a recursive function: given a subject and a tree, if the Tree is a Conclusion, we are done, otherwise, we retrieve the label of the next Choice, find the value of the Subject for that Choice, and use it to pick the next level of the Tree.
At that point, using the Tree to classify our subject is as simple as:

``` fsharp
> let actor = classify test manualTree;;

val actor : string = "Schwarzenegger"
```

Not bad for 14 lines of code. The most painful part is the manual construction of the Tree - let's see if we can get the computer to build that for us.

## So what...s a good question?

So how could we go about constructing a "good" tree based on the data, that is, a good sequence of questions?

The first consideration here is how informative the answer to a question is. An answer is very informative if it helps us separate the dataset into very differentiated groups, groups which are different from each other and internally homogeneous. A question which helps us break the dataset into 2 groups, the first containing only "Schwarzenegger" movies, the other only "Stallone" movies, would be perfect. A question which splits the dataset into groups with a half/half mix of movies by each actor would be totally useless.

To quantify the "internally homogeneous" notion, we will use [Shannon Entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory)), which is defined as

![Shannon Entropy]({{ site.url }}/assets/2012-08-05-efdf8c905c0f9dfd78002df6f20edb5d.png)

*Shannon Entropy formula, courtesy of Wikipedia*

I won't go into a full discussion on Entropy today (if you want to know more, you may find this [older post]({{ site.url }}/2012/05/20/Infinite-Monkeys-and-their-typewriters/) of interest) - the short version is, it provides a measure for how "disorganized" a set is. The lower the Entropy, the more predictable the content: a set where all results are identical would have a 0 entropy, while a set where all results are equally probably will result in an entropy value that is maximal.

The second consideration is, how much information do we gain by asking one question versus the other. If we look back at our dataset, there are 2 questions we could ask first: Sci-Fi or Action. If we asked "is it an Action movie" as an opener, there is a 20% chance that we are done and can conclude "Schwarzenegger", but in 80% of the cases, we are left with a 50/50 chance of guessing the right actor. By contrast, if we ask "is it a Sci-Fi movie" first, there is a 40% chance that we are done, with a perfect Entropy group of Schwarzenegger movies, and only a 60% chance of a not-so-useful answer.

We can quantify this - how informative is each question - by considering the expected gain in Entropy: for a question, consider the conclusion of each possible answer and compute its Entropy, and average out the Entropy of receiving each answer, weighted by the probability of obtaining that answer.

Let's illustrate on our dataset. Asking "Action?" can produce 2 outcomes:

* Yes, with 20% probability: we have now a 100% chance of Schwarzenegger, with a 0 Entropy
* No, with 80% probability: we have now a 50/50 chance of either actor, with a 0.69 Entropy

By contrast, asking "Sci-Fi?" first can produce 2 outcomes:

* Yes, with 40% probability: we have now a 100% chance of Schwarzenegger, with a 0 Entropy
* No, with 60% probability: we have now a 33.3%/66.7% chance of either actor, with a 0.64 Entropy

Case 1 gives us 0.8 x 0.69 value, case 2 0.6 x 0.64 - a much lower Entropy for a much better question.

The final consideration is whether the question is informative at all, which we can translate as "is our Entropy better after we ask the question"? For instance, if we know it's a Sci-Fi movie, asking whether it's an Action movie brings us nothing at all: the expected Entropy after the question isn't better than our initial Entropy. If a Question doesn't result in a higher expected Entropy, we shouldn't even bother to ask the Question.

Another way to consider what we just discussed is from a Decision Theory standpoint. In that context, information is considered valuable if having the information available would result in making a different decision. If for any answer to the question, the decision remains identical to what it would have been without it, the information is worthless - and if offered the choice between multiple pieces of information, one should pick the most likely to produce a better decision.

## Automatic Creation of the Tree using Entropy

Let's put these concepts into action, and figure out how to construct a good decision tree from the dataset. The outline of the algorithm is the following:

* Given a dataset, consider each Feature/Question available, split the dataset according to that Feature, and compute the Entropy that would be gained by splitting on that Feature,
* Pick the Feature that results in the highest Entropy gain, and split the dataset accordingly into subsets corresponding to each possible "answer",
* For each of the subsets, repeat the procedure, until no information is gained,
* When no Entropy is gained by asking further questions, return the most frequent Conclusion in the subset.

On our example, this would result in the following sequence:

* Sci-Fi has a better Entropy gain than Action: break into 2 subsets, Sci-Fi and non-Sci-Fi movies,
* For the Sci-Fi group, do we gain Entropy by asking about Action? No , stop,
* For the non-Sci-Fi group, asking about Action still produces information; after that, stop - there is no question left.

Rather than build up step-by-step, I'll dump the whole tree-building code here and comment afterwards:

``` fsharp
let prop count total = (float)count / (float)total

let inspect dataset =
    let header, (data: 'a [][]) = dataset
    let rows = data |> Array.length
    let columns = header |> Array.length
    header, data, rows, columns

let h vector =
    let size = vector |> Array.length
    vector
    |> Seq.groupBy (fun e -> e)
    |> Seq.sumBy (fun e ->
        let count = e |> snd |> Seq.length
        let p = prop count size
        - p * log p)

let entropy dataset =
    let _, data, _, cols = inspect dataset
    data
    |> Seq.map (fun row -> row.[ cols-1 ])
    |> Seq.toArray
    |> h

let remove i vector =
    let size = vector |> Array.length
    Array.append vector.[ 0 .. i-1 ] vector.[ i+1 .. size-1 ]

let split dataset i =
    let hdr, data, _, _ = inspect dataset
    remove i hdr,
    data
    |> Seq.groupBy (fun row -> row.[i])
    |> Seq.map (fun (label, group) ->
        label,
        group |> Seq.toArray |> Array.map (remove i))

let splitEntropy dataset i =
    let _, data, rows, cols = inspect dataset
    data
    |> Seq.groupBy(fun row -> row.[i])
    |> Seq.map (fun (label, group) ->
        group
        |> Seq.map (fun row -> row.[cols - 1])
        |> Seq.toArray)
    |> Seq.sumBy (fun subset ->
        let p = prop (Array.length subset) rows
        p * h subset)

let selectSplit dataset =
    let hdr, data, _, cols = inspect dataset
    if cols < 2
    then None
    else
        let currentEntropy = entropy dataset
        let feature =
            hdr.[0 .. cols - 2]
            |> Array.mapi (fun i f ->
                (i, f), currentEntropy - splitEntropy dataset i)
            |> Array.maxBy (fun f -> snd f)
        if (snd feature > 0.0) then Some(fst feature) else None

let majority dataset =
    let _, data, _, cols = inspect dataset
    data
    |> Seq.groupBy (fun row -> row.[cols-1])
    |> Seq.maxBy (fun (label, group) -> Seq.length group)
    |> fst

let rec build dataset =
    match selectSplit dataset with
    | None -> Conclusion(majority dataset)
    | Some(feature) ->
        let (index, name) = feature
        let (header, groups) = split dataset index
        let trees =
            groups
            |> Seq.map (fun (label, data) -> (label, build (header, data)))
            |> Seq.toArray
        Choice(name, trees)
```

**`prop`** is a simple utility function which converts the count of items from a total in a float proportion.

**`inspect`** is a utility function which extracts useful information from a dataset. I noticed I was doing variations of the same operations everywhere, so I ended up extracting it in a function. The dataset is modeled as a Tuple, where the first element is an array of headers, representing the names of each Feature, and the second is an Array of Arrays, representing a list of observations. The feature we are classifying on is expected to be in the last column of the dataset.

**`h`** computes the Shannon Entropy of a vector (an Array of values). **`entropy`** extracts the vector we are classifying on, and computes its Shannon Entropy.

**`remove`** is a simple utility function, which removes the ith component of a vector. It is used in **`split`**, a function which takes a dataset and splits it according to feature i. split returns a Tuple, where the first element is the updated header (with the split-feature removed), and a Sequence containing each value of the feature, with the corresponding reduced dataset. For instance, splitting the movies dataset on feature 1 ("Sci-Fi") produces the following:

``` fsharp
> split movies 1;;
val it : string [] * seq<string * string [] []> =
  ([|"Action"; "Actor"|],
   seq
     [("No",
       [|[|"Yes"; "Stallone"|]; [|"Yes"; "Stallone"|];
         [|"No"; "Schwarzenegger"|]|]);
      ("Yes", [|[|"Yes"; "Schwarzenegger"|]; [|"Yes"; "Schwarzenegger"|]|])])
```

Sci-Fi is now gone from the Headers, and 2 groups have been produced, corresponding to Non-Sci-Fi and Sci-Fi movies, with the corresponding reduced dataset.

**`splitEntropy`** computes the expected Entropy that would result from splitting on feature i. It splits the dataset on the value in column i (the feature we would split on), retrieves the value of the last column (the feature we are classifying on) and computes the sum of the entropies, weighted proportionally to the size of each sub-group (the probability to get that answer). **`selectSplit`** builds on it, and computes the entropy gain achieved by splitting on each of the available features (if there is any feature left), and picks the feature with maximum gain, if it is higher than the current situation.

**`majority`** is a simple utility function, which returns the most-frequent value in a dataset.

Finally, **`build`** puts all of this together and recursively builds the Decision Tree. If there is no feature to split on (no information gain left), it produces a Conclusion, containing the majority of elements in the current dataset. Otherwise, the dataset is split on the best feature, and a Choice node is created, which is populated with a Tree corresponding to the action taken for each possible outcome of the Feature we are splitting on.

That was a lot of code to go through (actually, under 100 lines, not too bad) - let's get our reward, and try it out:

``` fsharp
> let movies =
    [| "Action"; "Sci-Fi"; "Actor" |],
    [| [| "Yes"; "No";  "Stallone" |];
        [| "Yes"; "No";  "Stallone" |];
        [| "No";  "No";  "Schwarzenegger"  |];
        [| "Yes"; "Yes"; "Schwarzenegger"  |];
        [| "Yes"; "Yes"; "Schwarzenegger"  |] |]

let tree = build movies
let subject = [| ("Action", "Yes"); ("Sci-Fi", "No") |]
let answer = classify subject tree;;

val movies : string [] * string [] [] =
  ([|"Action"; "Sci-Fi"; "Actor"|],
   [|[|"Yes"; "No"; "Stallone"|]; [|"Yes"; "No"; "Stallone"|];
     [|"No"; "No"; "Schwarzenegger"|]; [|"Yes"; "Yes"; "Schwarzenegger"|];
     [|"Yes"; "Yes"; "Schwarzenegger"|]|])
val tree : Tree =
  Choice
    ("Sci-Fi",
     [|("No",
        Choice
          ("Action",
           [|("Yes", Conclusion "Stallone");
             ("No", Conclusion "Schwarzenegger")|]));
       ("Yes", Conclusion "Schwarzenegger")|])
val subject : (string * string) [] = [|("Action", "Yes"); ("Sci-Fi", "No")|]
val answer : string = "Stallone"
```

Looks like we are doing something right!

## Real data

The book demonstrates the algorithm on the [lenses dataset](http://archive.ics.uci.edu/ml/datasets/Lenses), which focuses on what contact lenses to prescribe to patients in various conditions, and can be found at the wonderful collection of test datasets maintained at the [UC Irvine Machine Learning Repository](http://archive.ics.uci.edu/ml/index.html). I tried it out, and got the same tree as the author. I also tested it on the [Nursery](http://archive.ics.uci.edu/ml/datasets/Nursery) dataset (I am still not 100% clear on the details of that dataset, but I have to admit I cracked up when I read the description for attribute 1: "parents: usual, pretentious, great_pret"...), which is significantly larger (12,960 records); the algorithm went through it like a champ, and produced an ungodly tree which I am not even going to try to reproduce here.

I haven't found a good way to plot the resulting Decision Trees yet, so I'll leave it at that - running the algorithm on these datasets is pretty straightforward.

## Conclusion

The code conversion for this chapter was interesting. The part where F# really shines is the Tree representation as a Discriminated Union, which combined with pattern-matching works wonders in manipulating Trees, and seems to me cleaner than the equivalent Python code using nested dictionaries.

I spent quite a bit of time second-guessing myself on how to organize the dataset itself - one array, labels + data, headers, data and "feature of interest", 2-d array or array of arrays... There is a lot of unpleasant grouping and array manipulations going on in the splitting / feature selection part, and I have a nagging feeling that there has to be a representation that allows for clearer transformations - and clearer return types. However, the code remains reasonably readable, and `Seq.groupBy` does in one line all the unique keys identification and counting that is more involved in the Python code.

The piece I have completely left out from Chapter 3 is persisting and plotting Decision Trees. I may go back to the plotting part at some later time, but my first impression is that this will be no piece of cake, and I...d rather focus on the algorithms at that point in time. If you know of a good and free F# tree-plotting library, let me know!

Finally, I decided to put this code on GitHub. It...s my first repository there, so please be patient and let me know if there are things I can do to make that repository better!

[Machine Learning in Action, in F#, on GitHub](https://github.com/mathias-brandewinder/Machine-Learning-In-Action)

Questions, comments? Let me know your feedback!

gha_language: CSS


text:
# Modeling on SMD dataset

## Finetune GPT2 model on SMD training set

We provide the [**checkpoint**](https://drive.google.com/open?id=1v7P-UulLBTwdW2EIk2Jz0vdPKlpZlLUI) of GPT-2 model fine-tuned on SMD training set. Download the checkpoint and put it under `./modeling/smd` folder.

```console
❱❱❱ cd ./modeling/smd
❱❱❱ mkdir ./runs
❱❱❱ unzip FOLDER/TO/CHECKPOINT/SMD_gpt2_graph_False_adj_False_edge_False_unilm_False_flattenKB_False_historyL_1000000000_lr_6.25e-05_epoch_10_weighttie_False_kbpercentage_0_layer_12.zip -d ./runs
```

You can also choose to train the model by yourself using the following command.

``` console
❱❱❱  python main.py --dataset SMD --lr 6.25e-05 --n_epochs 10 --kbpercentage 0 --layers 12
```

## Adapt fine-tuned GPT-2 model on the test set

``` console
❱❱❱ python evaluate_finetune.py --dataset SMD --model_checkpoint runs/SMD_gpt2_graph_False_adj_False_edge_False_unilm_False_flattenKB_False_historyL_1000000000_lr_6.25e-05_epoch_10_weighttie_False_kbpercentage_0_layer_12 --top_k 1 --eval_indices 0,303 --filter_domain ""
```

You can also speed up the finetuning process by running experiments parallelly. Please modify the GPU setting in #L14 of the [code](https://github.com/HLTCHKUST/ke-dialogue/blob/master/modeling/smd/runner_expe_SMD.py#L14).

``` console
❱❱❱ python runner_expe_SMD.py
```

repository: mehrdad-shokri/ke-dialogue

repo_name: mehrdad-shokri/ke-dialogue


text:
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/a2a3b4b68222433a8215f1ef8b39a02d)](https://www.codacy.com/app/solosTec/node?utm_source=github.com&utm_medium=referral&utm_content=solosTec/node&utm_campaign=Badge_Grade)
![GitHub](https://img.shields.io/github/license/solosTec/node.svg?logo=solosTec)

# node
**node** is a Smart Metering Framework (SMF) build upon the [CYNG](https://github.com/solosTec/cyng) library.

## Introduction ##

* Current version is 0.7. Core functions are ready for productive use:
  * master
  * setup
  * ipt
  * store
  * dash
  * e350/iMega
  * csv
* Linux (32/64 bit) are supported
* Windows 7 (64 bit) or higher are supported.
* Crosscompiling for Raspberry 3 is supported
* Requires [g++](https://gcc.gnu.org/) >= 4.8 or cl 19.00.24215.1 (this is VS 14.0), [boost](http://www.boost.org/) >= 1.66.0 and [pugixml](https://pugixml.org/) 1.8

## How do I get set up? ##

* Buildsystem is based on [cmake](http://www.cmake.org/) >= 3.5
* Download or clone from [github](https://github.com/solosTec/node.git)

To build the SMF first install the [CYNG](https://github.com/solosTec/cyng) library, then run:

### on Linux ###

```
#!shell

git clone https://github.com/solosTec/node.git
mkdir build && cd build
cmake -DNODE_BUILD_TEST:bool=ON ..
make -j4 all
sudo make install

```

It is recommended to use the latest boost library version. It is best to create them by hand.
Some hints to build Boost on Linux:

* Download the latest [version](https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.bz2)
* For unicode support install ICU: (sudo apt install libicu-dev) and call bootstrap with option --with-icu
* Build the library with b2. Depending on your machine this may take some time. Choose option -j depending on the CPU count.
* Instruct CMake to use the path to the Boost library specified with the --prefix option
* Make sure to update your linker configuration to find this path. To make changes permanent add a .conf file below /etc/ld.so.conf.d/.

```

wget -c https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.bz2
tar -xvjf boost_1_68_0.tar.bz2
cd boost_1_68_0/
./bootstrap.sh --prefix=install --with-toolset=gcc
./b2 --architecture=x86 --address-model=64 install -j2

```

### on Windows ###

```

git clone https://github.com/solosTec/node.git
mkdir build
cd build
cmake -DNODE_BUILD_TEST:bool=ON -G "Visual Studio 15 2017" -A x64 ..

```

Open _NODE.sln_ with Visual Studio.

## License ##

SMF is free software under the terms of the [MIT License](https://github.com/solosTec/node/blob/master/LICENSE).

repository: solosTec/node

repo_name: solosTec/node


text:
<EMAIL_ADDRESS>
OPEN HTTP, WebSocket, MQTT, & CoAP COMMUNICATIONS NETWORK & API FOR THE INTERNET OF THINGS (IoT)!

[![Build Status](https://travis-ci.org/octoblu/meshblu.svg?branch=master)](https://travis-ci.org/octoblu/meshblu)
[![Code Climate](https://codeclimate.com/github/octoblu/meshblu/badges/gpa.svg)](https://codeclimate.com/github/octoblu/meshblu)
[![Test Coverage](https://codeclimate.com/github/octoblu/meshblu/badges/coverage.svg)](https://codeclimate.com/github/octoblu/meshblu)
[![npm version](https://badge.fury.io/js/meshblu.svg)](http://badge.fury.io/js/meshblu)
[![Gitter](https://badges.gitter.im/octoblu/meshblu.svg)](https://gitter.im/octoblu/meshblu)

Visit [developer.octoblu.com](http://developer.octoblu.com) for up-to-the-latest documentation and screencasts.

Introduction
------------

Meshblu is an open source machine-to-machine instant messaging network and API. Our API is available on HTTP REST, realtime Web Sockets via RPC (remote procedure calls), [MQTT](http://mqtt.org), and [CoAP](http://en.wikipedia.org/wiki/Constrained_Application_Protocol).  We seamlessly bridge all of these protocols. For instance, an MQTT device can communicate with any CoAP or HTTP or WebSocket connected device on Meshblu.

Meshblu auto-assigns 36 character UUIDs and secret tokens to each registered device connected to the network. These device "credentials" are used to authenticate with Meshblu and maintain your device's JSON description in the device directory.

Meshblu allows you to discover/query devices such as drones, hue light bulbs, weemos, insteons, raspberry pis, arduinos, server nodes, etc. that meet your criteria and send IM messages to 1 or all devices.

You can also subscribe to messages being sent to/from devices and their sensor activities.

Meshblu offers a Node.JS NPM module called [Meshblu](https://www.npmjs.org/package/meshblu) and a [meshblu.js](http://meshblu.octoblu.com/#javascript) file for simplifying Node.JS and mobile/client-side connectivity to Meshblu.

Press
-----
[AllSeen Alliance](https://allseenalliance.org/announcement/allseen-alliance-reaches-50-members-expands-smart-home-connected-car-and-security-focus) - Allseen Alliance reaches 50 members; expands smart home, connected car and security focus

[Sys-Con](http://iot.sys-con.com/node/3125944) - Exclusive Octoblu Interview @ThingsExpo Silicon Valley

[Onalytica](http://www.onalytica.com/blog/posts/the-internet-of-things-top-100-organizations) - The Internet of Things - Top 100 Organizations

[GigaOm](http://gigaom.com/2014/07/21/octoblu-launches-to-make-skynet-internet-of-things-tools-professional-grade/) - Octoblu launches to make Skynet internet of things tools professional grade

[VentureBeat](http://venturebeat.com/2014/07/22/internet-of-things-startup-octoblu-designs-a-platform-that-translates-protocols/) - Internet of things startup Octoblu designs a platform that translates protocols

[Forbes](http://www.forbes.com/sites/benkepes/2014/07/22/octoblu-rolls-out-its-internet-of-things-thing/) - Octoblu Peels Back The Covers On Its Internet Of Things Platform

[GigaOm](http://gigaom.com/2014/02/04/podcast-meet-skynet-an-open-source-im-platform-for-the-internet-of-things/) - Listen to Stacey Higginbotham from GigaOm interview Chris Matthieu, the founder of SkyNet, about our capabilities, uses, and future direction.

[Wired](http://www.wired.com/wiredenterprise/2014/02/skynet/) - ‘Yes, I’m trying to build SkyNet from Terminator.’

[Wired](http://www.wired.com/2014/05/iot-report/) - Why Tech’s Best Minds Are Very Worried About the Internet of Things

[LeapMotion](https://labs.leapmotion.com/46/) - Developer newsletter covers flying drones connected to SkyNet with LeapMotion sensor!

[The New Stack](http://thenewstack.io/a-messaging-network-for-drones-called-skynet/) - Drones Get A Messaging Network Aptly Called SkyNet

Roadmap
-------

* Phase 1 - Build a network and realtime API for enabling machine-to-machine communications.
* Phase 2 - Connect all of the thingz.
* Phase 3 - Become self-aware!

Installing/Running Meshblu private cloud
----------

Clone the git repository, then:

```bash
$ npm install
```

[Meshblu](http://meshblu.octoblu.com) uses Mongo, Redis, ElasticSearch, and Splunk; however, we have made this infrastructure optional for you.  Meshblu falls back to file system and memory storage if these services are not configured allowing you to deploy a private Meshblu cloud to a Raspberry Pi or other mini-computer!

If you want to include these services for a scalable infrastructure, you can make the following changes to your `config.js` file.

Modify `config.js` with your MongoDB connection string. If you have MongoDB running locally use:

```
mongo: {
  databaseUrl: "mongodb://localhost:27017/skynet"
},
```

You can also modify `config.js` with your Redis connection information. If you have Redis running locally use:

```
redis: {
  host: "<IP_ADDRESS>",
  port: "6379",
  password: "abcdef"
},
```

You can also modify `config.js` with your ElasticSearch connection information. If you have ES running locally use:

```
elasticSearch: {
  host: "localhost",
  port: "9200"
},
```

If you would like to connect your private Meshblu cloud to [meshblu.octoblu.com](http://meshblu.octoblu.com) or another private Meshblu cloud, register a UUID on the parent cloud (i.e. meshblu.octoblu.com) using the POST /Devices REST API and then add the returned UUID and token to the following section to your private cloud's config.js file:

```
parentConnection: {
  uuid: 'xxxx-my-uuid-on-parent-server-xxxx',
  token: 'xxx ---  my token ---- xxxx',
  server: 'meshblu.octoblu.com',
  port: 80
},
```

Start the Meshblu server running HTTP and WebSocket protocols use:

```bash
$ node server.js --http
```

You may also run something like [forever](https://www.npmjs.org/package/forever) to keep it up and running:

```bash
$ forever start server.js --http
```

MQTT Broker
-----------

MQTT is an optional Meshblu protocol.  If you would like to run our MQTT broker with your private Meshblu cloud, open another console tab and run:

```bash
$ node server.js --http --mqtt
```

 or using forever

 ```bash
 $ forever start server.js --http --mqtt
 ```

Note: Our MQTT Broker defaults to using Mongo; however, you can run it in memory by removing the databaseUrl from the config.js.

```
mqtt: {
  port: 1883,
  skynetPass: "Very big random password 34lkj23orfjvi3-94ufpvuha4wuef-a09v4ji0rhgouj"
}
```

CoAP
----

CoAP is an optional Meshblu protocol.  If you would like to run our CoAP protocol with your private Meshblu cloud, open another console tab and run:

```bash
$ node server.js --http --coap
```

 or using forever

 ```bash
 $ forever start server.js --http --coap
 ```

Note: Our CoAP protocol defaults to using Mongo; however, you can run it in memory by removing the databaseUrl from the config.js.

```
coap: {
  port: 5683,
  host: "localhost"
}
```

Heroku
------

[![Deploy](https://www.herokucdn.com/deploy/button.png)](https://heroku.com/deploy)

Use the button above to deploy to the [Heroku](http://heroku.com/) cloud for Free! Starts out with a basic implementation of Meshblu with only MongoDB. The app.json / config.js can be modified to allow for more protocols and storage systems.

[Heroku](http://heroku.com/) allows you scale with a touch of a button.

DigitalOcean
------------

[![Install on DigitalOcean](http://installer.71m.us/button.svg)](http://installer.71m.us/install?url=https://github.com/octoblu/meshblu)

Use the button above to deploy to the [DigitalOcean](https://www.digitalocean.com/) cloud for as little as $5/month! The app.json / config.js can be modified to allow for more protocols and storage systems.

Docker
------

The default Dockerfile will run Meshblu a container to make quick experiments easier. The easiest way to get started is to use our image from the docker registry:

```bash
docker run -e TOKEN=something_secret octoblu/meshblu
```

*A note about secrets: We require that whomever runs Meshblu to supply a text secret, in the form of the `TOKEN` environmnet variable, that we can use as a salt when hashing tokens. Not knowing this secret hinders a malicious user when creating a rainbow table to reverse the hashed tokens.*

If you want to build your own image, then run the following from the Meshblu project root:

```
docker build -t local/meshblu .
```

Then run it like above, but with the local image:

```
docker run -e TOKEN=something_secret local/meshblu
```

This will run Meshblu (formerly skynet) and expose port 80 from the container and map it to your port 80. If you'd like to map it to a different port, use `-p`. For example, to map it to local port 3000:

```shell
docker run -p 3000:80 -e TOKEN=something_secret local/meshblu
```

Nodeblu Developer Toolkit
--------------------------------
Play with [Meshblu](http://meshblu.octoblu.com) IoT platform in Chrome! [Nodeblu](https://chrome.google.com/webstore/detail/nodeblu/aanmmiaepnlibdlobmbhmfemjioahilm) helps you experiment with the [Octoblu](http://octoblu.com) and [Meshblu](http://meshblu.octoblu.com) Internet of Things platforms by dragging, dropping, and wiring up various nodes connected to Meshblu!

Nodeblu is Octoblu's fork of the popular [NodeRED](https://github.com/node-red/node-red) application from IBM.  Since our app is deployed as a Chrome extension, we decided to add extra local features such as speech recognition and text-to-speech, a WebRTC webcam, HTML5 notifications, access to ChromeDB, a gamepad controller, and access to local and remote Arduino and Spark devices via our [Microblu](https://github.com/octoblu/microblu_mqtt) OS.

Gateblu
-------

We have an open source Octoblu Gateway also available on [GitHub](https://github.com/octoblu/gateblu). Our Gateway allows you to connect devices with or *without* IP addresses to Meshblu and communicate bi-directionally!  The Gateway uses WebSockets to connect to Meshblu so it can traverse NAT firewalls and send/receive messages to/from Meshblu.

Gateblu has an extensible [plugin](https://github.com/octoblu/gateblu/blob/master/plugins.md) architecture allowing you to create plugins for devices that we have not had a chance to connect yet.  You can search [NPMJS.org](https://www.npmjs.org/search?q=skynet-plugin) for a list of active Gateblu plugins.

Microblu Operating System
-----------------------

Meshblu includes a micro-controller operating system that is compatible with [Arduino](http://www.arduino.cc/), [Spark](https://www.spark.io/), and [Pinoccio](https://pinocc.io/)!  The OS is available on [GitHub](https://github.com/octoblu/microblu_mqtt) and comes with firmata built-in.

On power-on, the Microblu OS connects to Meshblu, obtains a UUID, and awaits your instructions! You can message the micro-controller to turn on/off pins and servos as well as request streaming analog sensor data from various pins.

HTTP(S) REST API
----------------

Most of our API endpoints require authentication credentials (UUID and secret token) passed in the HTTP headers as meshblu_auth_uuid and meshblu_auth_token respectively. These credentials are generated by registering a device or user with Meshblu via the POST /Devices API (see below). If you would like to associate additional Meshblu devices with the UUID and Token that you created (as a user), you can add an "owner" property to your other devices with the user's UUID as its value; otherwise, you can use the device's UUID and token in the headers to control the device itself.

We support the following device permissions: Discover, Send Messages, Receive Messages (subscribe) and Configure (Update). These permissions are manageable by adding UUIDs to whitelists and blacklists arrays with the following names: discoverWhitelist, discoverBlacklist, sendWhitelist, sendBlacklist, receiveWhitelist, receiveBlacklist, updateWhitelist, updateBlacklist. Note: If your UUID is the same as the "owner" UUID, these permissions are not enforced (you are the owner).

GET /status

Returns the current status of the Meshblu network

```
curl http://localhost:3000/status

=> {"skynet":"online","timestamp":1380480777750,"eventCode":200}
```

GET /devices

Returns an array of all devices available to you on Meshblu. Notice you can query against custom properties i.e. all drones or light switches and online/offline etc.

```
curl "http://localhost:3000/devices" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl "http://localhost:3000/devices?key=123" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl "http://localhost:3000/devices?online=true" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl "http://localhost:3000/devices?key=123&online=true" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> ["ad698900-2546-11e3-87fb-c560cb0ca47b","2f3113d0-2796-11e3-95ef-e3081976e170","9c62a150-29f6-11e3-89e7-c741cd5bd6dd","f828ef20-29f7-11e3-9604-b360d462c699","d896f9f0-29fb-11e3-a27c-614201ddde6e"]
```

GET /devices/uuid

Returns all information on a given device by its UUID

```
curl "http://localhost:3000/devices/01404680-2539-11e3-b45a-d3519872df26" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"_id":"5241d9140345450000000001","channel":"main","deviceDescription":"this is a test","deviceName":"hackboard","key":"123","online":true,"socketid":"pG5UAhaZa_xXlvrItvTd","timestamp":1380340661522,"uuid":"ad698900-2546-11e3-87fb-c560cb0ca47b"}b
```

POST /devices

Registers a device on the Meshblu network. You can add as many properties to the device object as desired. Meshblu returns a device UUID and token which needs to be used with future updates to the device object

Note: You can pass in a token parameter to overide skynet issuing you one

```
curl -X POST -d "name=arduino&description=this+is+a+test" "http://localhost:3000/devices" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl -X POST -d "name=arduino&token=123" "http://localhost:3000/devices" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"name":"arduino","description":"this is a test","uuid":"8220cff0-2939-11e3-88cd-0b8e5fdfd7d4","timestamp":1380481272431,"token":"1yw0nfc54okcsor2tfqqsuvnrcf2yb9","online":false,"_id":"524878f8cc12f0877f000003"}
```

PUT /devices/uuid

Updates a device object.

```
curl -X PUT -d "online=true" "http://localhost:3000/devices/01404680-2539-11e3-b45a-d3519872df26" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"uuid":"8220cff0-2939-11e3-88cd-0b8e5fdfd7d4","timestamp":1380481439002,"online":true}
```

DELETE /devices/uuid

Unregisters a device on the Meshblu network. Token is required for security.

```
curl -X DELETE -d "token=123" "http://localhost:3000/devices/01404680-2539-11e3-b45a-d3519872df26" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"uuid":"8220cff0-2939-11e3-88cd-0b8e5fdfd7d4","timestamp":1380481567799}
```

GET /localdevices

Returns a list of unclaimed devices that are on the same network as the requesting resource.

```
curl -X GET http://meshblu.octoblu.com/localdevices --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"devices":[{"autoRegister":true,"online":false,"timestamp":"2014-08-05T20:38:31.139Z","ipAddress":"<IP_ADDRESS>","protocol":"websocket","secure":false,"uuid":"76537331-1ce0-11e4-861d-89322229e557","channel":"main"},{"autoRegister":true,"online":true,"timestamp":"2014-08-05T16:50:52.492Z","ipAddress":"<IP_ADDRESS>","protocol":"websocket","secure":false,"uuid":"a92350c1-1cc0-11e4-861d-89322229e557","channel":"main"}]}
```

PUT /claimdevice/:uuid

Adds the meshblu_auth_uuid as the owner of this device UUID allowing a user or device to claim ownership of another device.

```
curl -X PUT http://meshblu.octoblu.com/claimdevice/{:uuid} --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"updatedExisting":true,"n":1,"connectionId":232,"err":null,"ok":1}
```

GET /mydevices

Returns all information (excluding tokens) of all devices or nodes belonging to a user's UUID (identified as "owner")

```
curl -X GET "http://meshblu.octoblu.com/mydevices" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"devices":[{"owner":"0d1234a0-1234-11e3-b09c-1234e847b2cc","name":"SMS","phoneNumber":"16025551234","uuid":"1c1234e1-xxxx-11e3-1234-671234c01234","timestamp":1390861609070,"token":"1234eg1234zz1tt1234w0op12346bt9","channel":"main","online":false,"_id":"52e6d1234980420c4a0001db"}}]}
```

POST /messages

Sends a JSON message to all devices or an array of devices or a specific device on the Meshblu network.

```
curl -X POST -d '{"devices": "all", "message": {"yellow":"off"}}' "http://localhost:3000/messages" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl -X POST -d '{"devices": ["ad698900-2546-11e3-87fb-c560cb0ca47b","2f3113d0-2796-11e3-95ef-e3081976e170"], "message": {"yellow":"off"}}' "http://localhost:3000/messages" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

curl -X POST -d '{"devices": "ad698900-2546-11e3-87fb-c560cb0ca47b", "message": {"yellow":"off"}}' "http://localhost:3000/messages" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"devices":"ad698900-2546-11e3-87fb-c560cb0ca47b","message":{"yellow":"off"},"timestamp":1380930482043,"eventCode":300}
```

Note: If your Meshblu cloud is connected to meshblu.octoblu.com or other private Meshblu clouds, you can send messages across Meshblu clouds by chaining UUIDs together separated by slashes (/) where the first UUID is the target cloud and the second UUID is the device on that cloud.

```
curl -X POST -d '{"devices": "ad698900-2546-11e3-87fb-c560cb0ca47b/2f3113d0-2796-11e3-95ef-e3081976e170", "message": {"yellow":"off"}}' "http://localhost:3000/messages" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"
```

GET /events/uuid?token=token

Returns last 10 events related to a specific device or node

```
curl -X GET "http://meshblu.octoblu.com/events/ad698900-2546-11e3-87fb-c560cb0ca47b?token=123" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"events":[{"uuid":"0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc","socketid":"lnHHS06ijWUXEzb01ZRy","timestamp":1382632438785,"eventCode":101,"_id":"52694bf6ad11379eec00003f"},{"uuid":"0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc","socketid":"BuwnWQ_oLmpk5R3m1ZRv","timestamp":1382561240563,"eventCode":101,"_id":"526835d8ad11379eec000017"}]}
```

GET /subscribe

This is a streaming API that returns device/node mesages as they are sent and received. Notice the comma at the end of the response. Meshblu doesn't close the stream.

```
curl -X GET "http://meshblu.octoblu.com/subscribe" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> [{"devices":"0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc","message":{"red":"on"},"timestamp":1388768270795,"eventCode":300,"_id":"52c6ec0e4f67671e44000001"},{"devices":"0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc","message":{"red":"on"},"timestamp":1388768277473,"eventCode":300,"_id":"52c6ec154f67671e44000002"},
```

GET /subscribe/uuid

This is a streaming API that returns device/node broadcast mesages as they are sent. Notice the comma at the end of the response. Meshblu doesn't close the stream.

```
curl -X GET "http://meshblu.octoblu.com/subscribe/ad698900-2546-11e3-87fb-c560cb0ca47b" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> [{"fromUuid":"ad698900-2546-11e3-87fb-c560cb0ca47b","devices":"*",message":{"red":"on"},"timestamp":1388768270795,"eventCode":300,"_id":"52c6ec0e4f67671e44000001"},{"fromUuid":"ad698900-2546-11e3-87fb-c560cb0ca47b","devices":"*",,"message":{"red":"on"},"timestamp":1388768277473,"eventCode":300,"_id":"52c6ec154f67671e44000002"},
```

GET /authenticate/uuid?token=token

Returns UUID and authticate: true or false based on the validity of uuid/token credentials

```
curl -X GET "http://meshblu.octoblu.com/authenticate/81246e80-29fd-11e3-9468-e5f892df566b?token=5ypy4rurayktke29ypbi30kcw5ovfgvi" --header "meshblu_auth_uuid: {my uuid}" --header "meshblu_auth_token: {my token}"

=> {"uuid":"81246e80-29fd-11e3-9468-e5f892df566b","authentication":true} OR {"uuid":"81246e80-29fd-11e3-9468-e5f892df566b","authentication":false}
```

GET /ipaddress

Returns the public IP address of the request.   You can use [Matteo Collina's](https://twitter.com/matteocollina) [CoAP CLI](https://www.npmjs.org/package/coap-cli) for testing CoAP REST API calls.  Here are a few examples:

coap get coap://coap.octoblu.com/status

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/devices?type=drone

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/devices/ad698900-2546-11e3-87fb-c560cb0ca47b

coap post -p "type=drone&color=black" -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/devices

coap put -p "token=123&color=blue&online=true" -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/devices/ad698900-2546-11e3-87fb-c560cb0ca47b

coap delete -p "token=123" -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/devices/ad698900-2546-11e3-87fb-c560cb0ca47b

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/mydevices/0d1234a0-1234-11e3-b09c-1234e847b2cc?token=1234glm6y1234ldix1234nux41234sor

coap post -p '{"devices": "*", "payload": {"yellow":"off"}}' -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/messages

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/events/ad698900-2546-11e3-87fb-c560cb0ca47b?token=123

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/subscribe -o

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/subscribe/ad698900-2546-11e3-87fb-c560cb0ca47b -o

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/ipaddress

coap post -p "token=123&temperature=78" -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/data/0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc

coap get -H "meshblu_auth_uuid={:UUID}&meshblu_auth_token={:TOKEN}" coap://coap.octoblu.com/data/0d3a53a0-2a0b-11e3-b09c-ff4de847b2cc?token=123

WEBSOCKET API
-------------

Request and receive system status

```js
socket.emit('status', function (data) {
  console.log(data);
});
```

Request and receive an array of devices matching a specific criteria

```js
socket.emit('devices', {"key":"123"}, function (data) {
  console.log(data);
});
```

Request and receive information about a specific device

```js
socket.emit('whoami', {"uuid":"ad698900-2546-11e3-87fb-c560cb0ca47b"}, function (data) {
  console.log(data);
});
```

Request and receive a device registration

```js
socket.emit('register', {"key":"123"}, function (data) {
  console.log(data);
});
```

Request and receive a device update

```js
socket.emit('update', {"uuid":"ad698900-2546-11e3-87fb-c560cb0ca47b", "token": "zh4p7as90pt1q0k98fzvwmc9rmjkyb9", "key":"777"}, function (data) {
  console.log(data);
});
```

Request and receive a device unregistration

```js
socket.emit('unregister', {"uuid":"b5535950-29fd-11e3-9113-0bd381f0b5ef", "token": "2ls40jx80s9bpgb9w2g0vi2li72v5cdi"}, function (data) {
  console.log(data);
});
```

Store sensor data for a device uuid

```js
socket.emit('data', {"uuid":"b5535950-29fd-11e3-9113-0bd381f0b5ef", "token": "2ls40jx80s9bpgb9w2g0vi2li72v5cdi", "temperature": 55}, function (data) {
  console.log(data);
});

```

Request and receive an array of sensor data matching a specific criteria

```js
socket.emit('getdata', {"uuid":"b5535950-29fd-11e3-9113-0bd381f0b5ef", "token": "2ls40jx80s9bpgb9w2g0vi2li72v5cdi", "limit": 1}, function (data) {
  console.log(data);
});
```

Request and receive a message broadcast

```js
// sending message to all devices
socket.emit('message', {"devices": "all", "message": {"yellow":"on"}});

// sending message to a specific devices
socket.emit('message', {"devices": "b5535950-29fd-11e3-9113-0bd381f0b5ef", "message": {"yellow":"on"}});

// sending message to an array of devices
socket.emit('message', {"devices": ["b5535950-29fd-11e3-9113-0bd381f0b5ef", "ad698900-2546-11e3-87fb-c560cb0ca47b"], "message": {"yellow":"on"}});
```

Websocket API commands include: status, register, unregister, update, whoami, devices, subscribe, unsubscribe, authenticate, and message. You can send a message to a specific UUID or an array of UUIDs or all nodes on SkyNet.

MQTT API
--------

Our MQTT API works similar to our WebSocket API. In fact, we have a [skynet-mqtt](https://www.npmjs.org/package/skynet-mqtt) NPM module for to simplify client-side MQTT connects with SkyNet.

```bash
$ npm install skynet-mqtt
```

Here are a few examples:

```javascript
var skynet = require('skynet-mqtt');

var conn = skynet.createConnection({
  "uuid": "xxxxxxxxxxxx-My-UUID-xxxxxxxxxxxxxx",
  "token": "xxxxxxx-My-Token-xxxxxxxxx",
  "qos": 0, // MQTT Quality of Service (0=no confirmation, 1=confirmation, 2=N/A)
  "host": "localhost", // optional - defaults to meshblu.octoblu.com
  "port": 1883  // optional - defaults to 1883
});

conn.on('ready', function(){

  console.log('UUID AUTHENTICATED!');

  //Listen for messages
  conn.on('message', function(message){
    console.log('message received', message);
  });

  // Send a message to another device
  conn.message({
    "devices": "xxxxxxx-some-other-uuid-xxxxxxxxx",
    "payload": {
      "skynet":"online"
    }
  });

  // Broadcast a message to any subscribers to your uuid
  conn.message({
    "devices": "*",
    "payload": {
      "hello":"skynet"
    }
  });

  // Subscribe to broadcasts from another device
  conn.subscribe('xxxxxxx-some-other-uuid-xxxxxxxxx');

  // Log sensor data to skynet
  conn.data({temperature: 75, windspeed: 10});

});

```

Event Codes
-----------

If `log: true` in config.js, all transactions are logged to skynet.txt.  Here are the event codes associated with Meshblu transactions.

* 100 = Web socket connected
* 101 = Web socket identification
* 102 = Authenticate
* 200 = System status API call
* 201 = Get events
* 204 = Subscribe
* 205 = Unsubscribe
* 300 = Incoming message
* 301 = Incoming SMS message
* 302 = Outgoing SMS message
* 303 = Incoming Yo message
* 304 = Outgoing Yo message
* 305 = Outgoing Push Notification message
* 400 = Register device
* 401 = Update device
* 402 = Delete device
* 403 = Query devices
* 500 = WhoAmI
* 600 = Gateway Config API call
* 700 = Write sensor data
* 701 = Read sensor data

FOLLOW US!
----------

* [Twitter/Octoblu](http://twitter.com/octoblu)
* [Facebook/Octoblu](http://facebook.com/octoblu)
* [Google Plus](https://plus.google.com/communities/106179367424841209509)

repository: CESARBR/knot-cloud-source

gha_language: CoffeeScript

repo_name: CESARBR/knot-cloud-source


text:
# OpenFaaS Cloud

## Conceptual diagram

This diagram shows the interactions between the functions that make up the OpenFaaS Cloud

![](https://pbs.twimg.com/media/DZiif9QXcAEd8If.jpg:large)

## Development

* Before you start

The public trial of OpenFaaS Cloud is running on Docker Swarm, so deploy OpenFaaS on Docker Swarm using [the documentation](https://docs.openfaas.com/deployment/).

You will also need to extend all timeouts for the gateway and the queue-worker.

The `ack_timeout: "300s"` field should have a value of around `300s` or 5 minutes to allow for a Docker build of up to 5 minutes.

OpenFaaS Cloud leverages OpenFaaS functions so will work with Kubernetes, but a complete configuration has not been provided yet. Some minor tweaks may be needed to configuration YAML files such as URLs and memory limits for the of-builder and/or buildshiprun function.

* Create secrets for the API Gateway

The API gateway uses secrets to enable basic authentication, even if your gateway doesn't have auth enabled you should create some temporary secrets or comment out the lines that bind secrets to functions in stack.yml.

Comment out these lines in stack.yml:

```
      - basic-auth-user
      - basic-auth-password
```

Or create temporary secrets:

```
echo "admin" > basic-auth-user
echo "admin" > basic-auth-password
docker secret create basic-auth-user basic-auth-user
docker secret create basic-auth-password basic-auth-password
```

If you already have secrets set up for your gateway version 0.8.2 or later then there is no need to do the steps above.

* Create a GitHub app

Create a GitHub app from your GitHub profile page.

Before you start you'll need to create a free GitHub app and select these OAuth permissions:

- "Repository contents" read-only
- "Commit statuses" read and write

Now select only the "push" event.

Now download the private key for your GitHub App which we will use later in the guide for allowing OpenFaaS Cloud to write to commit statuses when a build passes or fails.

The GitHub app will deliver webhooks to your OpenFaaS Cloud instance every time code is pushed in a user's function repository. Make sure you provide the public URL for your OpenFaaS gateway to the GitHub app. Like:
`http://my.openfaas.cloud/function/gh-push`

* Create `github.yml` and populate it with your secrets as configured on the GitHub App:

```
environment:
    github_webhook_secret: "Long-Password-Goes-Here"
```
The shared secret is used to securely verify each message came from GitHub and not a third party.

* Add github appId to `github.yml`
```
environment:
    github_webhook_secret: "Long-Password-Goes-Here"
    github_app_id: "<app_id>"
```

### Status updates

OpenFaaS Cloud can update the statuses of your commits when a build passes or fails.

To enable this set `report_status: "true"` in `github.yml` before deploying the stack.

A private key must also be mounted as a Docker / Kubernetes secret so that the code can authenticate and update statuses.

* Find the .pem file from the GitHub App page

Create Docker secret
```
docker secret create derek-private-key <your_private_key_file>.pem
```

* Update the remote gateway URL in `stack.yml` or set the `OPENFAAS_URL` environmental variable.

```
provider:
  name: faas
  gateway: http://localhost:8080

```

* Deploy the registry and of-builder

Using the instructions given in the repo deploy of-builder (buildkit as a HTTP service) and the registry

https://github.com/openfaas/openfaas-cloud/tree/master/of-builder

* Build/deploy

> Before running this build/push/deploy script change the Docker Hub image prefix from `alexellis2/` to your own.

```
$ faas-cli build --parallel=4 \
  && faas-cli push --parallel=4 \
  && faas-cli deploy
```

* Test it out

Simply install your GitHub app to one of your OpenFaaS function repos and push a commit.

Within a few seconds you'll have your function deployed and live with a prefix of your username.

* Find out more

For more information get in touch directly for a private trial of the public service.

### Appendix for Kubernetes

The functions which make up OpenFaaS Cloud are compatible with Kubernetes but some additional work is needed for the registry to make it work as seamlessly as it does on Swarm. The of-builder should also be brought up using a Kubernetes YAML file instead of `docker run` / `docker service create`.

You will need to edit `buildshiprun_limits_swarm.yml` and change the unit value from `30m` to `30Mi` so that Kubernetes can make use of the value.

The way Kubernetes accesses the Docker registry will need a NodePort and a TLS cert (or an insecure-registry setting in the kubelet), so bear this in mind when working through the `of-builder` README.

repository: alexcourouble/openfaas-cloud

repo_name: alexcourouble/openfaas-cloud


text:
# Star-Counter

A simple Github star counter using Redis

## Endpoints

 * Healthcheck: *<IP_ADDRESS>:/healthcheck*
    * Response:
    ```js
    { message: 'The API is working correctly!' }
    ```

  * Healthcheck: *<IP_ADDRESS>:/:username*
    * GitHub API Response:
    ```js
    { totalStars: x, source: 'GitHub API' }
    ```
    * Redis Response:
    ```js
    { totalStars: x, source: 'Redis Cache' }
    ```
    * Error Response:
    ```js
    { message: 'The GitHub username start could not be tracked. Try "dougtq" as an example!' }
    ```

  * Check the response-time diference from the GitHub API Response to the Redis Response

## Built With

* [Express](http://expressjs.com/)
* [Redis](redis.io)
* [Axios](https://github.com/axios/axios)

repository: dougtq/star-counter

repo_name: dougtq/star-counter


text:
# EnhancedDiscord
A simple way to load plugins and themes on Discord without token stealing and lag.

Support server:

(https://discord.gg/XAvh9tq)

By default, this loads my theme, with some settings applied. More info [on its repo](https://github.com/joe27g/Discord-Theme). To change this, edit `plugins/style.css` or change the theme location in settings.

You can download more quality plugins from the following repos:
- [joe27g/EnhancedDiscord-Plugins](https://github.com/joe27g/EnhancedDiscord-Plugins)
- [rauenzi/EnhancedDiscordPlugins](https://github.com/rauenzi/EnhancedDiscordPlugins)
- [jakuski/ed_plugins](https://github.com/jakuski/ed_plugins)

### Installing the easy way:

*Currently, only Windows and Ubuntu 18.04+ are supported for the new installer, due to lack of devices to compile on*

Go to https://enhanceddiscord.com/ and hit the 'Download' button, or if you're lazy, click [here](https://enhanceddiscord.com/EnhancedDiscord.exe).

If you're having errors (especially related to `InteropServices`), you may need to download [.NET Framework 4.7.1](https://www.microsoft.com/en-us/download/details.aspx?id=56116).

If you're on Linux or MacOS consider using [this install script](https://github.com/Cr3atable/LinuxED) that is specifically made for Unix.
LinuxED is unaffiliated with EnhancedDiscord so please do not request support for this script in an issue.
### ~~Installing the old way:~~

~~You can also use the console installer (no GUI.)~~

~~*Currently, only Windows is supported for this installer as well.*~~

~~Download: https://enhanceddiscord.com/ED_cmdline_BETA.exe~~

*(this installer is outdated and will most likely not work.)*

### Installing the hard way:

See the [advanced installation guide](/advanced_installation.md).

### Explanation of files:

From now on, please refer to https://enhanceddiscord.com for explanations of included plugins.

### Custom plugins

To create your own plugin, check out [the plugin readme](/plugins.md).

### Contributing

Feel free to make pull requests or make your own plugins repository. If you do make your own plugins, request the 'Plugin Developer' role in the support server so you can announce your releases!

repository: legendary-bro/EnhancedDiscord

repo_name: legendary-bro/EnhancedDiscord


text:
# StackViews
Small library for assigning auto layout constraints using declarative Stack View / Flex-Box semantics.

## Motivation
There are many reasons for UI Reach production strength maintainable IOS applications to lean towards code based layouting and use storyboards sparingly.
But programmatic constraints assignment is not very intuitive.
There are many libraries that are trying to address this issue. [Some](https://github.com/robb/Cartography) have more intuitive syntax for assigning layout constraints, [some](https://github.com/linkedin/LayoutKit) completely bypassing auto layout engine and render views using there own engine.

This library consists of one major function: *stackViews(...)* and it is trying to simplify very common scenario of stacking collection of child views inside the parent view. It is similar to the IOS UIStackView but strives to have few advantages:
- it treats **any** view as a stacking container
- it sets in one place **all** constraints necessary for stack layout, including dimensions and/or proportional detentions of all children, individual alignments and stack container insets
- if has additional *justify* (*distribute*) options "borrowed" from the css flex-box layout: justify-start, justify-end, justify-center

## Sample application
In addition to the StackViews library, workspace includes StackViewExamples reference application.
Reference application uses StackView library to replicate Example View Controller from Apple's [Auto Layout Cookbook](https://developer.apple.com/library/content/documentation/UserExperience/Conceptual/AutolayoutPG/LayoutUsingStackViews.html).

In addition, Sample Application includes interactive view controller where different parameters of the stackViews function can be changed interactively and layout changes observed in the real time.

## Usage Example
Here is an annotated code example that generates this form header ([check out source file for more details](./StackViewsExamples/StackViewsExamples/FormExapmle/FormExampleHeaderViewController.swift)):

```Swift
    let image = UIImageView(image: UIImage(named: "PersonProfile"))
    let firstName = UITextField()
    let middleName = UITextField()
    let lastName = UITextField()

    //Stack Fields vertically
    let fieldsStackView = stackViews(
            orientation: .vertical, // vertical stack of 3 rows
            justify: .fill, // because heights of all 3 rows set explicitly (no nil height row) this will make
                            // fieldsStackView height determined by the sum of rows heights and spaces between them

            align: .fill,  // fill all available space across stack axis (horizontally)
            spacing: 10, //vertical spacing between rows
            views: [
                    applyLabel("First Name", ofWidth: 110, toField: firstName),
                    applyLabel("Middle Name", ofWidth: 110, toField: middleName),
                    applyLabel("Last Name", ofWidth: 110, toField: lastName)
            ],
            heights: [25, 25, 25]) // height of each row
        .container // stackViews function returns StackingResult structure that returns generated container as one of it's properties.

    //set image view to be a square
    image.widthAnchor.constraint(equalTo: image.heightAnchor, multiplier: 1).isActive = true

    //Stack imageView and fieldsStackView horizontally
    //We do not set any explicit widths or heights because: 1)height of the header
    //is determined by the height of the fieldsStackView and vertical insets.
    //2) Height of the image is determined by the height of the header. 3) Width of the image is determined by image being square
    //4) fieldsStackView view we want to be stretched horizontally to fill all the available space
    _ = stackViews(
            container: self.view, // use ViewController's view as a stack container
            orientation: .horizontal, //stack image and fieldsStackView horizontally
            justify: .fill, // fill all available space along stack axis (horizontally
            align: .fill, // fill all available space across stack axis (vertically)
            insets: Insets(horizontal: 5, vertical: 5), // set space between container view boundaries and
                                                        // and it's children
            spacing: 10, // space between image and fieldsStackView
            views: [image, fieldsStackView])

```
## API

Api consists of ***stackViews*** function and few small convenience functions. All public functions and types are in [StackViews.swift](./StackViews/StackViews/StackViews.swift).
```Swift
public func stackViews(
        container: UIView? = nil,
        orientation: Orientation,
        justify: Justify?,
        align: Alignment?,
        insets: Insets = Insets.zero,
        spacing: CGFloat = 0,
        views: [UIView],
        widths: [CGFloat?]? = nil,
        proportionalWidths: [CGFloat?]? = nil,
        heights: [CGFloat?]? = nil,
        proportionalHeights: [CGFloat?]? = nil,
        individualAlignments: [Alignment?]? = nil,
        activateConstrains: Bool = true) -> StackingResult
```
- **container:** Container view that acts as a stack panel.
If view is not provided a new view will be created and returned as a part of the StackingResult structure
- **orientation:** defines the stacking axes as horizontal or vertical.
- **justify:** describes the stacking option along the stacking axis.
supported options _{ **.fill .start .end .spaceBetween .center** }_
If justify parameter is set to nil no justification will be applied
- **align:** describes the stacking option across the stacking axes for all views.
supported options _{ **.fill .start .center .end** }_
Can be overwritten for specific views by the individualAlignments parameter.
If align set to nil and no individualAlignments provided - no views will be aligned
- **insets:** specify insets applied to the edges of the container view.
(Gap between the container view edges and it's children) Default = 0
- **spacing:** spacing between neighboring views along the stacking axis. Default = 0
- **views:** array of view that should be arranged (stacked) inside the container view
- **widths:** array of width values corresponding to each view in views array. nil value indicates that width constraint will not be set
If width array is provided it's length should be equal to the views array length
- **proportionalWidths:** can be used to specify relative width value for views. If bot absolute width value and relative value provided
than the firs view with both values specified will be used as a 'key' view and all other proportionalWidths will be set in relation to this view.
For example if we have 3 views with width: [nil, 50, nil] and proportionalWidths: [1,2,3] then resulting constraint width of the views will be: [25, 50, 75]
- **heights:** array of heights values corresponding to each view in views array. nil value indicates that height constraint will not be set
If heights array is provided it's length should be equal to the views array length
- **proportionalHeights:** can be used to specify relative height value for views. If bot absolute height value and relative value provided
than the firs view with both values specified will be used as a 'key' view and all other proportionalHeights will be set in relation to this view.
For example if we have 3 views with height: [nil, 50, nil] and proportionalHeights: [1,2,3] then resulting constraint height of the views will be: [25, 50, 75]
- **individualAlignments:** Alignment for individual views.

## License
[MIT](./LICENSE)

repository: pmabiala/StackViews

repo_name: pmabiala/StackViews


text:
Tempy
===
Python HTML template engine and yet another meta-language for python

Tempy is still under construction. Contribute to me!

Installation
==
In the root of project directory, type
```bash
python setup.py install
```

Basic example
==

Let’s assume that working directory is composed like the following structure.
```
run.py
tempy-templates/
  index.tpy
```
As shown above, name of tempy files ends with “.tpy”. Write below code into index.tpy.
```
MainTemplate = html:
  head:
    title: "First example" # comment starts with sharp(#), as in Python.
  body:
    h2: 'Hello World!' # You can wrap content with either single quotation mark or double one.
    h3:
      "This is content in h3 element" # You can also specify content at the next line with one-indent deeper
    p(id="main-content", class="foo bar"):
	    '''
	    It is the first example of Tempy library, HTML template-engine.
	    With a sequence of three single or double quotation marks, you can write multi-line content.
	    '''
    a(href="#"): "external link"
```

And, in run.py, write the following stub
```python
from tempy import Environment
env = Environment("tempy-templates") # specify the directory where tpy files are present

index_module = env.module("index")
with open("result.html", "w") as f:
    f.write(str(index_module.MainTemplate))
```
Note that “Environment.module(..)” method doesn’t accept file name with extension. In other words, you should specify filename without its extention (“index” in this case) instead of full filename(“index.tpy” in this case)

Then if you execute run.py, you will see the below HTML code genereted in result.html.
```html
<html>
  <head>
    <title>
      First example
    </title>
  </head>
  <body>
    <h2>
      Hello World!
    </h2>
    <h3>
      This is content in h3 element
    </h3>
    <p class="foo bar" id="main-content">

      It is the first example of Tempy library, HTML template-engine.
      With a sequence of three single or double quotation marks, you can write multi-line content.

    </p>
    <a href="#">
      external link
    </a>
  </body>
</html>
```

Many template engines provide methods to fill up template file with data so that user generates different page on various data. In Tempy, user can use "def" statement to define templates to which data is passed as arguments.

```
def MainTemplate(user_name):
	head:
		title: "First example" # comment starts with sharp(#), as in Python.
	body:
		h2: 'Hello World!' # You can wrap content with either single quotation mark or double one.
		h3:
			"This is content in h3 element" # You can also specify content at the next line with one-indent deeper
		p:
			'''
			It is the first example of Tempy library, HTML template-engine.
			With a sequence of three single or double quotation marks, you can write multi-line content.
			'''
		p:
			"Hello!"
			user_name
			"How have you been?"

```

You may notice that it's very simillar to define a function in Python. In fact, there's no difference between function and template. Thus, user just calls MainTemplate in python with argument "user_name".
```python
from tempy import Environment
env = Environment("tempy-templates")

user_name = raw_input("Your name")

index_module = env.module("index")
result = index_module.MainTemplate(user_name)
with open("result.html", "w") as f:
    f.write(str(result))
```

You can also access elements of a dict, access attributes of a list, or iterate a list in Tempy.
```
def MainTemplate(author_list, footer_dict):
  table:
		thead:
			tr:
				td: "Name"
				td: "Country"
				td: "Note"
		tbody:
			$each(author_obj, in = author_list):
				tr:
					td: author_obj.name
					td: author_obj.country
					td: author_obj.note
	footer:
	    p: "Maintainer: " + footer_dict["maintainer"]
	    p: "Tel. " + footer_dict["tel"]
	    p:
	        "Contact us at "
	        a(href=footer_dict["github_url"])
```

run.py:
```python
from tempy import Environment
env = Environment("tempy-templates")

class Author:
    def __init__(self, name, country, note):
        self.name = name
        self.country = country
        self.note = note

authors = [
    Author('Alchan "Algy" Kim', 'Korea', 'Maintainer'),
    Author('You', 'Somewhere', '')
]

footer_dict = {
    "maintainer": "Kim",
    "tel": "82-10-xxxx-xxxx",
    "github_url": "https://github.com/Algy/tempy"
}

index_module = env.module("index")
result = index_module.MainTemplate(authors, footer_dict)
with open("result.html", "w") as f:
    f.write(str(result))
```
Execute run.py and you will see result.html.

Many template engines have extension system. For example, there could be a base template with common header and footer so that concrete templates could fill up detail contents into the base. In Tempy, all you need to do that is defining "base function" and call it with appropriate html fragments.
```
def Base(_title, Fragment):
  html:
    header:
      title: _title # A preceeding underline is used to avoid name conflict with the HTML tag, "title"
      Fragment

Fragment1 = div:
  h3: "Heading3"
  p: "Paragraph content"

Fragment2 = table:
  tr:
    td: "1"
    td: "2"
  tr:
    td: "3"
    td: "4"

def Template1():
  Base("H3 And Paragraph", Fragment1)
  # Base(_title="H3 And Paragraph", Fragment=Fragment1) is possible as well.

def Template2():
  Base("A table", Fragment2)

```
Or, use ">" next to the callee and you can directly specify non-inline elements as arguments vertically.
```
# ...

def Template1():
  Base>
    # First argument
    "H3 And Paragraph"
    # Second argument
    div:
      h3: "Heading3"
      p: "Paragraph content"

# ...
```
Or, you can use "<argument name> -> <content>" notation for the sake of readability. It is like calling with keyword argument in python. Ordering of arguments doesn't matter in this case.

```
def Template1():
  Base>
    Fragment -> div:
      h3: "Heading 3"
      p: "Paragraph content"
    _title -> "H3 And Paragraph"
```

You can freely use various standard libaries of python in Tempy. So you can do funny thing like this!
```
pyimport urllib2

def MainTemplate():
  # it allows content to be written as it is, without being escaped.
	rawstring:
		urllib2.urlopen("http://canrailsscale.com").read()\
		       .replace("Rails", "Flask").replace("NO", "YES")
```

The following document about LISN parser is outdated.
LISN
====
General-purpose expression-oriented syntax parser

What is LISN?
--
Originally, its name is originated from "LIdl Syntax Notation", in which LIDL is a functional programming language with python-like grammar(though it's being designed and not implemented yet :). LISN is designed to have a flexible grammar enough to represent any kind of expression-oriented syntax where syntax block is represented by indentation.

Currently, source codes consist of two part, parser written in C and its wrapper written in python.

Dependency
--
In compilation of C sources, it only depends on Lemon parser, which is provided inside the repo. No need to install other library beforehand.

Python wrapper
--
TODO

C native API
--
clisn/ directory has sources for LISN parser written in C.
TODO

Compilation & Installation
--
cd clisn/
make
cd ..
./setup.py install

TODO

repository: Algy/tempy

repo_name: Algy/tempy


text:
---
layout: page
title: "Literature And Rhetoric"
date: 2005-03-09T21:59:36+00:00
comments: true
sharing: true
footer: true
group: LiteratureAndRhetoric
---

## Scholar/ChangSingh
* [Myths of the Decline](/literature-and-rhetoric/myths-decline) - Collected by the famed scholar Scholar/ChangSingh, this important work traces many of the legends, myths and fables that grew during the dark period that followed the failure of FTL travel.

## Macropedia/JanoosWelkred
* The RepublicanPapers
* [A Tale of Counter-revolution](/literature-and-rhetoric/tale-counter-revolution)

## Macropedia/NazirAlShamuth
* [The Life and Times of Furnoklin Taschera](/literature-and-rhetoric/life-times-furnoklin-taschera) [-(see also Macropedia/FurnoklinTaschera)-]
* [A Loyal Reformer's Essays](/literature-and-rhetoric/loyal-reformer-essays)

## Hans Richter Kuber (see Macropedia/KuberianMovement)
* [Universe Union and Empires of the Mind and Soul](/literature-and-rhetoric/universe-union-empires-mind-soul)

This is a manually updated list of topics that fit into the **Literature and Rhetoric** Category.

If you visit the site often, you might want to check out this category's {$Group}/RecentChanges.

----

(These are all pages in this category. If you find one here that is not above, please correct the list above.

gha_language: HTML