text:
# Overview

Harbor is an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management.

For more information, see the Harbor [GitHub](https://github.com/goharbor/harbor) and [official documentation](https://goharbor.io/docs/2.2.0/).

## About Google Click to Deploy

Popular open stacks on Kubernetes packaged by Google.

## Architecture

The application offers the Harbor application deployed as Deployment with all dependencies required to fully operate.

![Architecture diagram](resources/harbor-k8s-app-architecture.png)

Harbor configuration file `/etc/core/app.conf` is automatically generated in the application
through a Kubernetes ConfigMap. It's good to know that to modify ConfigMaps you wish you should remove the corresponding Pod to reload the Harbor configuration.

# Installation

## Quick install with Google Cloud Marketplace

Get up and running with a few clicks! Install this Harbor app to a Google
Kubernetes Engine cluster using Google Cloud Marketplace. Follow the
[on-screen instructions](https://console.cloud.google.com/marketplace/details/google/harbor).

## Command line instructions

You can use [Google Cloud Shell](https://cloud.google.com/shell/) or a local
workstation to complete these steps.

[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.svg)](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/GoogleCloudPlatform/click-to-deploy&cloudshell_open_in_editor=README.md&cloudshell_working_dir=k8s/harbor)

### Prerequisites

#### Set up command-line tools

You'll need the following tools in your development environment. If you are
using Cloud Shell, `gcloud`, `kubectl`, Docker, and Git are installed in your
environment by default.

-   [gcloud](https://cloud.google.com/sdk/gcloud/)
-   [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
-   [docker](https://docs.docker.com/install/)
-   [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
-   [helm](https://helm.sh/)

Configure `gcloud` as a Docker credential helper:

```shell
gcloud auth configure-docker
```

#### Create a Google Kubernetes Engine cluster

Create a new cluster from the command line:

```shell
export CLUSTER=harbor-cluster
export ZONE=us-west1-a

gcloud container clusters create "$CLUSTER" --zone "$ZONE"
```

Configure `kubectl` to connect to the new cluster:

```shell
gcloud container clusters get-credentials "$CLUSTER" --zone "$ZONE"
```

#### Clone this repo

Clone this repo and the associated tools repo.

```shell
git clone --recursive https://github.com/GoogleCloudPlatform/click-to-deploy.git
```

#### Install the Application resource definition

An Application resource is a collection of individual Kubernetes components,
such as Services, Deployments, and so on, that you can manage as a group.

To set up your cluster to understand Application resources, run the following
command:

```shell
kubectl apply -f "https://raw.githubusercontent.com/GoogleCloudPlatform/marketplace-k8s-app-tools/master/crd/app-crd.yaml"
```

You need to run this command once.

The Application resource is defined by the
[Kubernetes SIG-apps](https://github.com/kubernetes/community/tree/master/sig-apps)
community. The source code can be found on
[github.com/kubernetes-sigs/application](https://github.com/kubernetes-sigs/application).

### Install the Application

Navigate to the `harbor` directory:

```shell
cd click-to-deploy/k8s/harbor
```

#### Configure the app with environment variables

Choose an instance name and
[namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
for the app. In most cases, you can use the `default` namespace.

```shell
export APP_INSTANCE_NAME=harbor-1
export NAMESPACE=default
```

Enable Stackdriver Metrics Exporter:

> **NOTE:** Your GCP project must have Stackdriver enabled. If you are using a
> non-GCP cluster, you cannot export metrics to Stackdriver.

By default, the application does not export metrics to Stackdriver. To enable
this option, change the value to `true`.

```shell
export METRICS_EXPORTER_ENABLED=false
```

Set up the image tag:

It is advised to use a stable image reference, which you can find on:
- [Harbor](https://hub.docker.com/r/goharbor/harbor-core/tags).
- [Metrics exporter](https://marketplace.gcr.io/google/harbor/prometheus-to-sd).

```shell
export TAG="v2.5.4"
export TAG_METRICS_EXPORTER="2.5"
```

Configure the container images:

```shell
export HARBOR_SOURCE_REGISTRY=goharbor
export IMAGE_HARBOR_CORE="${HARBOR_SOURCE_REGISTRY}/harbor-core:${TAG}"
export IMAGE_HARBOR_CHARTMUSEUM="${HARBOR_SOURCE_REGISTRY}/chartmuseum-photon:${TAG}"
export IMAGE_HARBOR_DATABASE="${HARBOR_SOURCE_REGISTRY}/harbor-db:${TAG}"
export IMAGE_HARBOR_JOBSERVICE="${HARBOR_SOURCE_REGISTRY}/harbor-jobservice:${TAG}"
export IMAGE_HARBOR_NOTARY_SERVER="${HARBOR_SOURCE_REGISTRY}/notary-server-photon:${TAG}"
export IMAGE_HARBOR_NOTARY_SIGNER="${HARBOR_SOURCE_REGISTRY}/notary-signer-photon:${TAG}"
export IMAGE_HARBOR_PORTAL="${HARBOR_SOURCE_REGISTRY}/harbor-portal:${TAG}"
export IMAGE_HARBOR_REDIS="${HARBOR_SOURCE_REGISTRY}/redis-photon:${TAG}"
export IMAGE_HARBOR_REGISTRY="${HARBOR_SOURCE_REGISTRY}/registry-photon:${TAG}"
export IMAGE_HARBOR_REGISTRYCTL="${HARBOR_SOURCE_REGISTRY}/harbor-registryctl:${TAG}"
export IMAGE_HARBOR_TRIVY="${HARBOR_SOURCE_REGISTRY}/trivy-adapter-photon:${TAG}"
export IMAGE_HARBOR_NGINX="${HARBOR_SOURCE_REGISTRY}/nginx-photon:${TAG}"
export IMAGE_METRICS_EXPORTER="marketplace.gcr.io/google/harbor/prometheus-to-sd:${TAG_METRICS_EXPORTER}"
```

Set how to expose the service - you can choose the type as `ingress` or `clusterIP`.

```shell
export HARBOR_EXPOSE_TYPE="clusterIP"
```

Set the initial password of Harbor admin. You can change it from portal after launching Harbor.

```shell
export HARBOR_ADMIN_PASSWORD="Harbor12345"
```

#### Create namespace in your Kubernetes cluster

If you use a different namespace than `default`, run the command below to create
a new namespace:

```shell
kubectl create namespace "$NAMESPACE"
```

#### Expand the manifest template

Use `helm template` to expand the template.

# Application metrics

## Prometheus metrics

The application is configured to natively expose its metrics in the
[Prometheus format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md).

You can access the metrics at `[EXPORTER_SVC_URL]:8001/metrics`. For example, you can access the metrics at the
[http://localhost:8080/metrics](http://localhost:8080/metrics) endpoint using port forwarding as below:

```shell
kubectl port-forward svc/${APP_INSTANCE_NAME}-exporter 8080:8001
```

## Configuring Prometheus to collect the metrics

To configure Prometheus to automatically collect metrics, follow the steps in
[Configuring Prometheus](https://prometheus.io/docs/introduction/first_steps/#configuring-prometheus).
You configure the metrics in the
[`scrape_configs` section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).

## Exporting metrics to Stackdriver

The deployment includes a
[Prometheus to Stackdriver (`prometheus-to-sd`)](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/prometheus-to-sd)
container. If you enabled the option to export metrics to Stackdriver, the
metrics are automatically exported to Stackdriver and visible in
[Stackdriver Metrics Explorer](https://cloud.google.com/monitoring/charts/metrics-explorer).

The name of each metric starts with the application's name, which you define in
the `APP_INSTANCE_NAME` environment variable.

The exporting option might not be available for GKE on-prem clusters.

> Note: Stackdriver has [quotas](https://cloud.google.com/monitoring/quotas) for
> the number of custom metrics created in a single GCP project. If the quota is
> met, additional metrics might not show up in the Stackdriver Metrics Explorer.

You can remove existing metric descriptors using
[Stackdriver's REST API](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/delete).

# Scaling

You can use the `kubectl scale` commands to scale all Harbor components.

For example:

```shell
$ kubectl scale deployment harbor-1-core --replicas=3
deployment.apps/harbor-1-core scaled

$ k get pods | grep harbor-1-core
harbor-1-core-749ffb546c-824bp            1/1     Running
harbor-1-core-749ffb546c-9wlcr            1/1     Running
harbor-1-core-749ffb546c-ccfwh            1/1     Running
```

# Backup and restore

To backup Harbor application you need to create a PostgreSQL backup - you can follow the instructions from the [official documentation](https://www.postgresql.org/docs/current/continuous-archiving.html#BACKUP-LOWLEVEL-BASE-BACKUP).

You can also back up a specific database with the `pg_dump` command.

# Uninstalling the Application

## Using the Google Cloud Platform Console

1.  In the GCP Console, open
    [Kubernetes Applications](https://console.cloud.google.com/kubernetes/application).
1.  From the list of applications, click **Falco**.
1.  On the Application Details page, click **Delete**.

## Using the command line

### Delete the resources

> **NOTE:** We recommend that you use a `kubectl` version that is the same as
> the version of your cluster. Using the same versions of `kubectl` and the
> cluster helps avoid unforeseen issues.

To delete the resources, use the expanded manifest file used for the
installation.

Run `kubectl` on the expanded manifest file:

```shell
kubectl delete -f ${APP_INSTANCE_NAME}_manifest.yaml --namespace ${NAMESPACE}
```

If you don't have the expanded manifest, delete the resources using types and a
label:

```shell
kubectl delete daemonset,service,configmap,application \
  --namespace ${NAMESPACE} \
  --selector app.kubernetes.io/name=${APP_INSTANCE_NAME}
```

### Delete the GKE cluster

Optionally, if you don't need the deployed application or the GKE cluster,
delete the cluster using this command:

```shell
gcloud container clusters delete "${CLUSTER}" --zone "${ZONE}"
```

repository: GoogleCloudPlatform/click-to-deploy

gha_language: Ruby

repo_name: GoogleCloudPlatform/click-to-deploy


text:
# Changes and improvements from AlexaBeagleBone

Dependency on VLC has been removed.  VLC is great, however on the BeagleBone
Black, it is difficult to build from source if you're on
Yocto filesystem (such as the TI Linux Processor SDK).  Mplayer is another
capable media player available on ARM Linux, consumes low CPU resources
while streaming audio, and builds natively on TI ARM platforms running a
recent Yocto filesystem without much fuss (probably the case with other
platforms, but I've only worked with TI processors).

Mplayer can be put into slave mode and controlled through special files (fifo).
The pymplb wrapper by Stefan Parviainen is a clean and elegant solution for
programatically controlling Mplayer from Python.

Voice triggering is  supported using PocketSphinx and Kitt.AI Snowboy.
The Pocket Sphinx trigger word is "Pal" - fans of the movie "Uncle Buck" might
see the humor in using that name.  Kitt.AI is using the trained phrase
"Hey buddy" at the moment.  Other voice triggering libraries will be tested as
well.

CPU usage seems reasonable, even while streaming audio and performing a
voice triggering session.  Right now it looks like about 20-30% CPU utilization
while streaming audio over WiFi, and waiting for trigger phrase, running at
600MHz.  Since the AM335x can go up to 1GHz, there is lots of headroom for
more activities!

Seamless music and AVS responses are now supported.  This scenario arrises
when a user is playing a streaming audio station, and asks AVS a question.
Instead of muting the music track to playback the AVS response, the music
volume is lowered, and we playback the AVS response at normal volume, then
restore the music volume to the previous level.  More about this below in the
ALSA confiuration section.

# Build Instructions

For now things are not terribly user friendly, and are more focused towards my
own development.  Probably the most ornerious task will be creating an AVS
credentials file.

These instructions are written using Debian 8.7, and kernel version 4.4.30
on a Beaglebone Black Wireless (out of box image on eMMC).  Yes, I did rant
above about Yocto, but for now I am starting with Debian, then will switch
over to Yocto filesystem and add instructions for building dependencies.

The following information below is a collection of my own notes, and hopefully
is complete.

## Adding Required Linux System Libraries

Begin with updating apt

    $ sudo apt-get update

Install missing headers (alsa/asoundlib.h) for pyalsaaudio.  apt-file is
a nice helper to determine which package provides a library or header file.

    $ apt-file search alsa/asoundlib.h

    (sudo apt-get install apt-file if you don't have it)

    $ sudo apt-get install libasound2-dev

Install memcached (pronunciation: mem-cash-dee) is a general-purpose distributed
memory caching system. It is often used to speed up dynamic database-driven
websites by caching data and objects in RAM to reduce the number of times an
external data source (such as a database or API) must be read.

    $ sudo apt-get install memcached

Mplayer used for basic mp3 playback

    $ sudo apt-get install mplayer

Evtest helps finding input devices associated file descriptor so we can use a
keyboard for initiating a voice recording

    $ sudo apt-get install evtest

## Adding Python Packages

Install PIP to install python package dependencies.  Using 'get-pip' script
because this will be used on Yocto builds too.

Note: must be done with root permissions, or will fail

    $ wget https://bootstrap.pypa.io/get-pip.py
    $ sudo python ./get-pip.py

    Collecting pip
      Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
        100% |████████████████████████████████| 1.3MB 107kB/s
    Installing collected packages: pip
      Found existing installation: pip 9.0.0
        Uninstalling pip-9.0.0:
          Successfully uninstalled pip-9.0.0
    Successfully installed pip-9.0.1

Update PIP

    $ sudo pip install update
    Collecting update
      Downloading update-0.4.4.tar.gz
    Building wheels for collected packages: update
      Running setup.py bdist_wheel for update ... done
      Stored in directory: /root/.cache/pip/wheels/f0/35/57/6d8a5c1...
    Successfully built update
    Installing collected packages: update
    Successfully installed update-0.4.4

Install Python package dependencies
    pyalsaaudio
    python-memcached
    webrtcvad
    cherrypy

    $ sudo pip install pyalsaaudio python-memcached webrtcvad cherrypy

## Optional Voice Triggering

### KITT.AI snowboy

Snowboy is promising because we are going for a more targeted approach for
the wake word, which is fine.  We don't necessarily need an entire natural
language processing library - that's what AVS is for!

We can even define several "trigger words" to embed into the program to trigger
different actions, whether AVS, or something else on the web or even locally.

Information in the section is lacking - still learning and will fill in more as
it comes to me.

#### Install python bindings

Go over to the snowboy page on github for the latest information on installation
on your system [https://github.com/Kitt-AI/snowboy]

This project also uses the Python package webrtcvad like Pocket Sphinx to while
recording audio for AVS to detect when speech ends.  Unfortunately it appears
to not be working at the moment, and seems to be recording to the maximum
quiet time before giving up, which is currently set at 4 seconds.  I will be
working on that to speed up latency for short requests, and not cut off long
requests.

#### Train wake word

Create an account and train your keyword here: [https://snowboy.kitt.ai/dashboard]

Download the model and apply in the AlexaBeagleBone2 resources folder

Included are three models:
    "Hey Buddy"
    "Alexa"
    "Snowboy"

"Alexa" is the default.

### Pocket Sphinx

I stripped out the basics from the AlexaPi project to get PocketSphinx working
with this project.  To install the Python package for PocketSphinx, first
install the dependencies (these were what was missing on my system):

	$ sudo apt-get install swig libpulse-dev

	# --no-cache-dir helped turn up above dependenices for pocketsphinx
	$ sudo pip --no-cache-dir install --upgrade pocketsphinx

	$ sudo pip install webrtcvad

# Freeing space on eMMC

You might be ruinng low on eMMC space.  I removed several things, including the
Chromium browser package (please use care when deleting things!).

On Debian systems, you can use dpkg to see who are taking the most space:

	$ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n

# Features

* Supports hot phrase detection (Kitt.AI snowboy)
* Supports voice triggering (PocketSphinx)
* Supports using /dev/input device to initiate voice recording from ALSA input device
* Supports streaming audio (e.g. "Play 106.1")
* Supports asking questions while streaming audio playing - stream resumes after response
* Supports Alexa Skills Kit using natural language responses

# Configuring ALSA

ALSA is a beast.  I prefer not to deal with it, but there are times when it's required, such
as allowing streams to be mixed.  Just a forewarning, these changes were for my system, using
a USB audio card for both input and output.  If you're mixing onboard audio and USB audio
inputs and outputs, things will be different.

## Does your system support multiple streams?

You might be surprised what you find :)

Try playing two audio files simultaneously on the command line.  If your second player instance
gives errors, or is never heard, keep reading the next section.

## Supporting multiple streams

After a bunch of Googling, the answer was found here:
https://discuss.mopidy.com/t/setting-up-c-media-usb-headphone-set-in-mixer-mode/1409/4

In my settings below, notice I'm using hw:1,0 for both dmixed and dsnooped pcm blocks-
this is because I'm using my external USB audio card for input and output.

my /etc/asound.conf settings:
```
	pcm.dmixed {
	    type dmix
	    ipc_key 1024
	    ipc_key_add_uid 0
	    ipc_perm 0666
	    slave.pcm "hw:1,0"
	}
	pcm.dsnooped {
	    type dsnoop
	    ipc_key 1025
	    slave.pcm "hw:1,0"
	}

	pcm.duplex {
	    type asym
	    playback.pcm "dmixed"
	    capture.pcm "dsnooped"
	}

	pcm.!default {
	    type plug
	    slave.pcm "duplex"
	}

	ctl.!default {
	    type hw
	    card 1
	}
```

Old settings for external USB audio card (or your card can handle mixing):

```
	pcm.!default {
	    type hw
	    card 1
	}

	ctl.!default {
	    type hw
	    card 1
	}
```

## alsamixer ##

You may get better results tweaking the microphone settings on your system.  Strike a good
balance between reliable detection of the trigger word/ phrase and false detection.  This
will be likely specific to the audio capture device and mic used.

To save your settings:

```
    ~ $ sudo alsactl store
```

### Playback settings ###

You may wish to mute the microphone in the playback stream (notice 'MM' buried just above the mic gain)

```
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqq AlsaMixer v1.0.28 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Card: USB Audio Device                               F1:  Help               x
x Chip: USB Mixer                                      F2:  System information x
x View: F3:[Playback] F4: Capture  F5: All             F6:  Select sound card  x
x Item: Speaker [dB gain: -30.00, -30.00]              Esc: Exit               x
x                                                                              x
x                  lqqk               lqqk                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               x  x                                     x
x                  x  x               xaax                                     x
x                  x  x               xaax                                     x
x                  x  x               xaax                                     x
x                  xaax               xaax                                     x
x                  tqqu               tqqu               lqqk                  x
x                  xOOx               xMMx               xOOx                  x
x                  mqqj               mqqj               mqqj                  x
x                 10<>10               37                                      x
x          <     Speaker      >       Mic         Auto Gain Control            x
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
```

### Capture settings ###

Notice on the capture tab the mic is not muted.  Some terminal programs will not
switch tabs using the function keys.  Use TAB instead.

As mentioned previously, adjust the mic volume to strike a balance between reliable
and false detection.

```
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqq AlsaMixer v1.0.28 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Card: USB Audio Device                               F1:  Help               x
x Chip: USB Mixer                                      F2:  System information x
x View: F3: Playback  F4:[Capture] F5: All             F6:  Select sound card  x
x Item: Mic [dB gain: 18.00]                           Esc: Exit               x
x                                                                              x
x                                     lqqk                                     x
x                                     x  x                                     x
x                                     x  x                                     x
x                                     x  x                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                     xaax                                     x
x                                    LmqqjR                                    x
x                                   CAPTURE                                    x
x                                      76                                      x
x                                  <  Mic   >                                  x
x                                                                              x
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
```

# Instructions and Usage

* Replace cloned version of creds.py with your known-good credentials file
* Optionally add execute permissions to alexa-mplayer2.py

	$ ./alexa-mplayer2.py

Note: voice triggering is currenting the default interface to AVS.  Going back to a key
press event currently only requires uncommenting the alsabuttonrecord call in main().
Right now I do not support multiple triggering mechanisms.  In the future, the triggering
and utterance recording mechanisms will be separated.

# Future work

* Use YAML for configuration and holding credentials, trigger source, trigger word, ALSA settings, etc
* Add support for GPIO triggering (keyboard adds latency due to key repeats)
* Overlaying primary and secondary audio streams (simultaneous playback) requires more work
* Separate out triggering and audio logic
* Move global volume control to ALSA layer
* The two goals above will make it easier to lower volume while a request is being made

"Ever heard of the band 999 MegaBytes?  Probably not, they never made it to a gig" -- Alexa Joke

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
---
layout: post
author: [Phil]
title: The Tale of Tom, Dick and Harry
excerpt: This article describes three fictional characters as they all learn how various
  tools and methodologies work in the world of technology, internets and programming.
date: '2014-05-07 21:21:00'
# tags: TODO: ?
comments: true
disqus_identifier: the-tale-of-tom-dick-and-harry
alias: blog/2014/05/the-tale-of-tom-dick-and-harry/
alias_1: 2014/05/07/the-tale-of-tom-dick-and-harry/
---

This article describes three fictional characters as they all learn how various tools and methodologies work in the world of technology, internets and programming.

## Tom

Tom works with some massive clients these days and has been in the industry for 10+ years, seeing various tools, frameworks and methodologies come and go. He is very used to the "right tool for the job" idea and is happy to switch between tools, languages and workflows based on the project he is working on, and the team he is working on that project with.

Tom has recently decided to learn about InternetThingX, after he saw somebody give a great talk about it. In part his interest is down to his love of learning new things, but he also thinks it might solve a problem he has run into on a few other projects, and thinks it will help in some cases in the future.

Tom is an eager learner, an explorer and is motivated by playing with new things. He also has a real penchant  for teaching, and after learning and successfully implementing InternetThingX in a project or two Tom starts to recommend that others take a look too.

## Harry

Harry - like Tom - is interested in learning new things too, but is a little less experienced. He is also incredibly busy, being a respected Symfony and Drupal developer. He makes enough money to get all of his bills paid, and does not have much time outside of that due to his clients _never ending_ demands.

Harry only knows PHP but never saw a problem with that. He sometimes uses Drupal when he needs a CMS, Symfony when he needs a framework, and his web stack uses XAMPP because he is on Windows.

Harry saw Tom give a talk about InternetThingX, and loved it. Harry struggled for a while with InternetThingX because it was quite new, but after finding some more documentation he eventually realized it's a really useful tool for some of his projects.

He has added this to his reasonably small tool-belt for occasional use, and will actively recommend it to others because he is proud about learning a new thing.

## Dick

Dick has only been a programmer for about a year. He is very confident in his skill-set and has built a few sites, but his tool-belt is restricted to just one of each type of thing, which he has learned as he needed to use it. Dick is actively looking for things to learn so that he can become a better developer.

Dick sees Harry suggesting that InternetThingX is good, so InternetThingX goes straight on Dicks todo list.

It takes Dick a _really_ long time to get the hang of InternetThingX. InternetThingX is quite a bit more complicated than InternetThingA but Dick keeps at it because he saw Harry saying that it was good. Harry missed out some of the "InternetThingX in useful in this context..." because he was just proud about learning the tool, and so Dick just keeps on chugging away trying to learn it. He never used InternetThingA anyway, so he can't give up now after spending so much time to wrap his brain around it.

When Dick has finally learned all about InternetThingX he feels like a superstar for mastering it, and now uses it for absobloodyutelyeverygoddamnthing.

After spending that long trying to learn it, why wouldn't he? It has more features than that other thing and it can theoretically help Dick avoid some big scary problems that happen at large scale. Problems sound bad, so... cool!

Dick now goes around recommending it to everyone who mentions trying to do something _near_ the problem space, not even necessarily in it. He doesn't have many options available so he proudly shouts about the closest one and it will _technically_ work. If Dick spots anyone talking negatively about InternetThingX he pounces, telling them they're wrong.  They don't understand.

Dick screams at Tom when Tom suggests at a conference that InternetThingX might not always be the right tool for the job.  Dick remembers that it was complex to learn and took him a while, but maybe that was just because he was fairly new to things. Really it was good for Dick jumping in at the deep end. If others drown in missed deadlines due to not learning the tool quick enough then... well that is their fault. It's make or break. _In the long run_ Dick feels better off for for doing things the complicated way.

## "Conclusion"

Dick is well intentioned, but has been mislead to believe a meal must _always_ be a four-course culinary masterpiece, when sometimes a basic meat and two veg will satisfy the criteria.  He is likely to run around forcing beginners to get confused by overly complex solutions to problems that they (and he) do not have yet, and might not have for years.

Harry is a good lad, but he should have explained the use-cases a little better in the blogs he released.

Maybe Tom is just a know-it-all with too much time on his hands due to charging his clients through the nose.

Nobody is perfect.

In this story, InternetThingX could have been:

* Vagrant
* PaaS
* Laravel
* The "Repository" Pattern
* TDD

_These two were mentioned on Reddit as a joke, but they fit in here perfectly too:_

* NodeJS
* MongoDB

Learn what you reasonably can in the time you have available. You can't learn everything, but you should learn more.

Try to learn **when** something is useful, but more importantly learn **when not** to use something. You don't need to use Vagrant for every project ever.

For example, Vagrant is excellent. I wrote about [how excellent Vagrant is](https://code.tutsplus.com/tutorials/vagrant-what-why-and-how--net-26500) and how it can avoid obscure problems on production. I forgot to write that I use MAMP for basic CMS projects because the CMSs have all been built to abstract away those differences, and most of the time it doesn't matter.

I also forgot to mention that sometimes clients don't really feel like paying you the day/week it takes to get their architecture moved over, and all of their other staff upgraded and switched over.

Vagrant is cool. Use it when it fits. Don't bother if its trouble.

PaaS is cool. Use it when it fits. Don't bother if its trouble.

Laravel is cool. Use it when it fits. Don't bother if its trouble.

The "Repository" Pattern is cool. Use it when it fits. Don't bother if its trouble.

TDD is cool. Use it when it fits. Don't bother if its trouble.

branch_name: refs/heads/main

gha_license_id: MIT


text:
It uses [lerna](https://github.com/lerna/lerna) for monorepo's packages release to NPM.

### Options

* `--canary` Publish a canary version of the packages using incremental patch number and commit hash.
* `--no-git` Skip Git commit and tag (NPM only release).
* `--no-npm` Skip NPM release (Git tag only).

### Usage
```sh
# Publish a canary version of your monorepo's packages
$ rna publish --canary

# Publish a new version of the packages
$ rna publish

# Publish a single package (just use npm)
$ npm publish
```

repository: fquffio/rna-cli

gha_language: JavaScript

repo_name: fquffio/rna-cli


text:
# Ménneu Markdown Notebook

#### Edit markdown locally w/o installing any apps.

Using [Ménneu](https://github.com/shellyln/menneu) document processor to render the markdowns.

## Getting started

1. Download this repository as zip archive, and extract it.
1. Initialize the git repository on the extracted folder.
    ```bash
    $ git init
    ```
1. Copy article template from `./articles/*.html` and open from the editor you like.
    * `Article-1.html` : Normal markdown format.
    * `ScriptableArticle-1.html` : Markdown format that Lisp [LSX](https://github.com/shellyln/liyad#what-is-lsx) script expansion is enabled.
    * `Notebook-1.html` : JS Notebook markdown format.
        * Lisp LSX script expansion is enabled.
1. Preview the markdown by opening the browser.
1. Save your contents and commit to the repository.

## Playground

https://shellyln.github.io/menneu/playground.html

## Use via CDN

Replace `*.js` URLs in the `<script>` tags of `./articles/*.html` .

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

* `Article-1.html`
```html
<!DOCTYPE html><head><meta charset="UTF-8"></head><body><script type="text/markdown">

# Hello!

</script>
<script<EMAIL_ADDRESS><script<EMAIL_ADDRESS>onload="start({title: 'My Notebook 1'})"></script>
</body>
```

* `ScriptableArticle-1.html`
```html
<!DOCTYPE html><head><meta charset="UTF-8"></head><body><script type="text/markdown">

# Hello!

</script>
<script<EMAIL_ADDRESS><script<EMAIL_ADDRESS>onload="start({title: 'My Notebook 1'})"></script>
</body>
```

* `Notebook-1.html`
```html
<!DOCTYPE html><head><meta charset="UTF-8">
<!-- <script src="https://cdn.jsdelivr.net/npm/<EMAIL_ADDRESS>-->
</head><body><script type="text/markdown">

# JS Notebook

%%%(script (@ (src "https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML") (crossorigin "anonymous") (async)))
</script>
<script<EMAIL_ADDRESS><script<EMAIL_ADDRESS>onload="start({title: 'My Notebook 1'})"></script>
</body>
```

## Notes

* To save the repository size, move `articles/js/menneu.min.js` and `articles/js/menneu.min.js.map` outside the repository.
* You should escape some characters if script expansion is enabled.
  See [this document](https://github.com/shellyln/menneu/#lisp-block-expansion).

## Gallery

![notebook1](https://shellyln.github.io/menneu/assets/image/notebook1.png)

![notebook2](https://shellyln.github.io/menneu/assets/image/notebook2.png)

![notebook3](https://shellyln.github.io/menneu/assets/image/notebook3.png)

![notebook4](https://shellyln.github.io/menneu/assets/image/notebook4.png)

----

## License
[ISC](https://github.com/shellyln/menneu-md-notebook/blob/master/LICENSE.md)
Copyright (c) 2018, Shellyl_N and Authors.

license: ISC

repository: shellyln/menneu-md-notebook

repo_name: shellyln/menneu-md-notebook


text:
---
title: "Book Notes: Clojure - Learn Clojure"
date: 2020-03-15T21:05:32+03:00
draft: false
description: ""
tags:
categories: clojure
type: post
url:
author: "Mert Nuhoglu"
output:
  html_document:
    css: styles.css
state: wip

---

# Clojure - Learn Clojure

## Clojure - Learn Clojure - Syntax

https://clojure.org/guides/learn/syntax

### Strings (character types) id=g11315

``` clojure
"hello"         ; string
\e              ; character
  #"[0-9]+"       ; regular expression
```

### Symbols and idents: id=g11316

		map             ; symbol
		+               ; symbol - most punctuation allowed
		clojure.core/+  ; namespaced symbol
		nil             ; null value
		true false      ; booleans
		:alpha          ; keyword
		:release/alpha  ; keyword with namespace

Literal collections:

``` clojure
'(1 2 3)     ; list
[1 2 3]      ; vector
  #{1 2 3}     ; set
{:a 1, :b 2} ; map
```

### Structure vs Semantics

A clojure expression:

/Users/mertnuhoglu/Pictures/screenshots/20191105131739.png

Syntax in green (data structure produced by the Reader)

Semantics in blue (how that data is understood by clojure runtime)

Most literal clojure forms evaluate to themselves, except symbols and lists.

Symbols refer to something else. They return what they refer to when evaluated.

Lists are evaluated as invocation.

`(+ 3 4)` is read as a list. the first element is called "function position". It is invoked.

- `3` and `4` evaluate to themselves
- `+` evaluates to a function that implements `+`
- evaluating list invokes `+` function

Some languages have both statements and expressions. Statements have stateful effect but don't return a value. In clojure, everything is an expression that evaluates to a value.

### Delaying evaluation with quoting id=g11317

Sometimes, symbols should just be a symbol without looking up what it refers to. Then we suspend evaluation with quoting:

``` clojure
'x
```

Sometimes, a list should just be a list of data values:

``` clojure
'(1 2 3)
```

### REPL id=g11318

Clojure is always compiled to JVM bytecode. There is no clojure interpreter.

History of values of last expressions: `*1` `*2`...

``` clojure
(+ 3 4)
;; => 7
(+ 10 *1)
;; => 17
```

#### Helper functions id=g11313

Helper functions: `doc`, `find-doc`, `apropos`, `source`, `dir`

``` clojure
(require '[clojure.repl :refer :all])
;; => (doc +)
```

You can use the `apropos` command to find functions that match a particular string or regular expression.

``` clojure
(apropos "+")
;; => (clojure.core/+ clojure.core/+')
```

`find-doc` search inside documentation

`dir` list all functions in a namespace

``` clojure
(dir clojure.repl)
```

### Clojure basics

#### def var

``` clojure
(def x 7)
  ##> #'user/x
```

`def` associates symbol `x` with value `7`.

This association is called a `var`

`#'user/x` is the return value. It is literal representation for a var `#'`

`user` is the default namespace.

Symbols are evaluated by looking up what they refer to:

``` clojure
(+ x x)
  ##> 14
```

#### Printing id=g11319

`println` and `print` for human-readable printing.

`prn` and `pr` for machine-readable printing. It puts quotes around strings.

``` clojure
(println "What is this:" (+ 1 2))
;; => What is this: 3
```

`print` and `println` functions have side-effects and returns nil as result.

## Clojure - Learn Clojure - Functions

`defn` defines a named function:

``` clojure
;;    name   params         body
;;    -----  ------  -------------------
(defn greet  [name]  (str "Hello, " name) )
```

### Multi-arity functions

A function can take different numbers of parameters:

``` clojure
(defn messenger
  ([]     (messenger "Hello world!"))
  ([msg]  (println msg)))
```

### Variadic functions

A function can define a variable number of parameters.

The variable parameters must occur at the end of the parameter list.

``` clojure
(defn hello [greeting & who]
  (println greeting who))
```

Additional parameters are collected in a list `who`

``` clojure
user=> (hello "Hello" "world" "class")
Hello (world class)
```

### Anonymous Functions

``` clojure
;;    params         body
;;   ---------  -----------------
(fn  [message]  (println message) )
```

Usually it is passed to another function.

### defn vs fn

`defn` is both `def` and `fn`. `fn` defines the function. `def` binds it to a name

``` clojure
(defn greet [name] (str "Hello, " name))

(def greet (fn [name] (str "Hello, " name)))
```

### Anonymous function syntax

``` clojure
;; Equivalent to: (fn [x] (+ 6 x))
  #(+ 6 %)

;; Equivalent to: (fn [x y] (+ x y))
  #(+ %1 %2)

;; Equivalent to: (fn [x y & zs] (println x y zs))
  #(println %1 %2 %&)
```

### Gotcha

How to define an anonymous function that takes an element and wrapsi it in a vector?

3 options:

``` clojure
  #(vector %)
(fn [x] [x])
vector
```

### Applying Functions

`apply` is useful when arguments are handed to you as a sequence but you must invoke the function with the values in the sequence

For example, you can use apply to avoid writing this:

``` clojure
(defn plot [shape coords]   ;; coords is [x y]
  (plotxy shape (first coords) (second coords)))
```

Instead you can simply write:

``` clojure
(defn plot [shape coords]
  (apply plotxy shape coords))
```

### Locals and Closures

#### let

`let` binds symbols to values in a lexical scope.

``` clojure
;;      bindings     name is defined here
;;    ------------  ----------------------
(let  [name value]  (code that uses name))
```

``` clojure
(let [x 1
      y 2]
  (+ x y))
```

#### Closures

``` clojure
(defn messenger-builder [greeting]
  (fn [who] (println greeting who))) ; closes over greeting

;; greeting provided here, then goes out of scope
(def hello-er (messenger-builder "Hello"))

;; greeting value still available because hello-er is a closure
(hello-er "world!")
;; Hello world!
```

### Java Interop

#### invoking java code id=g11324

``` clojure
	| Task            | Java              | Clojure          |
	| Instantiation   | new Widget("foo") | (Widget. "foo")  |
	| Instance method | rnd.nextInt()     | (.nextInt rnd)   |
	| Instance field  | object.field      | (.-field object) |
	| Static method   | Math.sqrt(25)     | (Math/sqrt 25)   |
	| Static field    | Math.PI           | Math/PI          |
```

## Clojure - Learn Clojure - Sequential Collections

https://clojure.org/guides/learn/sequential_colls

### Vectors

``` clojure
[1 2 3]
user=> (get ["abc" false 99] 0)
  ##> "abc"
user=> (get ["abc" false 99] 1)
  ##> false
user=> (count [1 2 3])
3
user=> (vector 1 2 3)
[1 2 3]
user=> (conj [1 2 3] 4 5 6)
[1 2 3 4 5 6]
```

### Lists

Quote a list to prevent evaluation.

``` clojure
(def cards '(10 :ace :jack 9))
(first cards)
  ##> 10
(rest cards)
  ##> '(:ace :jack 9)
(conj cards :queen)
  ##> (:queen 10 :ace :jack 9)
(def stack '(:a :b))
  ##> #'user/stack
(peek stack)
  ##> :a
(pop stack)
  ##> (:b)
```

## Hashed Collections

https://clojure.org/guides/learn/hashed_colls

### Sets

``` clojure
(def players #{"Alice", "Bob", "Kelly"})
```

Adding to a set: `conj`

``` clojure
(conj players "Fred")
  ##> #{"Alice", "Bob", "Kelly", "Fred"}
```

Removing from a set: `disj`

``` clojure
(disj players "Fred")
  ##> #{"Alice", "Bob", "Kelly"}
```

Checking containment: `contains?`

``` clojure
(contains? players "Fred")
  ##> true
```

Concatenate two collections: `into`

``` clojure
(into players new-players)
```

### Maps

Two uses:

1. association of keys to values (like dictionaries in python)
2. domain application data

Creating a literal map:

``` clojure
(def scores {"Fred" 1400
  "Bob" 1240})
```

Adding new key-value pairs: `assoc`

``` clojure
(assoc scores "Sally" 0)
```

Removing: `dissoc`

``` clojure
(dissoc scores "Bob")
```

#### Look up:

opt01: `get`

``` clojure
(get scores "Fred")
  ##> 1400
```

opt02: invoke map itself like a function

``` clojure
(def directions {:north 0
                        :east 1
                        :south 2
                        :west 3})
  #'user/directions
directions :north)
  ##> 0
```

Default value:

``` clojure
(get scores "Sam" 0)
  ##> 0
```

Checking contains:

``` clojure
(contains? scores "Fred")
  ##> true
(find scores "Fred")
  ##> ["Fred" 1400]
```

Get keys or values:

``` clojure
(keys scores)
  ##> ("Fred" "Bob" "Angela")
(vals scores)
  ##> (1400 1240 1024)
```

### Representing application domain information

Map with keyword keys:

``` clojure
(def person
	{:first-name "Kelly"
	 :last-name "Keen"
	 :age 32})
```

#### Field accessors

Looking up by key:

``` clojure
(get person :age)
  ##> 32
(person :age)
  ##> 32
(:age person) ; invoking keyword as function
  ##> 32
(:age person 30) ; default value
	##> 32
```

When a keyword is invoked, it looks itself up in the associative data structure that it was passed.

Keywords are also functions like maps and sets.

Updating fields: `assoc`

Removing fields: `dissoc`

#### Nested entities

``` clojure
(def company
  {:name "WidgetCo"
   :address {:street "123 Main St"
             :city "Springfield"
             :state "IL"}})
```

Access using `get-in`

``` clojure
(get-in company [:address :city])
```

Add new field: `assoc-in`
Removing: `dissoc-in`

### Records

## Clojure - Learn Clojure - Flow Control

https://clojure.org/guides/learn/flow

``` clojure
(str "2 is " (if (even? 2) "even" "odd"))
  ##> 2 is even

(if (true? false) "impossible!") ;; else is optional
  ##> nil
```

Only false values are `false` and `nil`

Use `do` if there are multiple expressions.

``` clojure
(if (even? 5)
  (do (println "even")
      true)
  (do (println "odd")
      false))
```

`when` is an `if` with only a `then` branch:

``` clojure
(when (neg? x)
  (throw (RuntimeException. (str "x must be positive: " x))))
```

`cond` is multiple if:

``` clojure
(let [x 5]
  (cond
    (< x 2) "x is less than 2"
    (< x 10) "x is less than 10"))
```

`case` is switch:

``` clojure
user=> (defn foo [x]
         (case x
           5 "x is 5"
           10 "x is 10"))
  #'user/foo

user=> (foo 10)
x is 10

```

#### Iteration for Side Effects

`dotimes`

``` clojure
(dotimes [i 3]
         (println i))
;; 0
;; 1
;; 2
;; => nil
```

`for` is list comprehension

``` clojure
(for [letter [:a :b]
             number (range 3)] ; list of 0, 1, 2
         [letter number])
;; => ([:a 0] [:a 1] [:a 2] [:b 0] [:b 1] [:b 2])
```

repository: mertnuhoglu/study

gha_language: HTML

repo_name: mertnuhoglu/study


text:
# For Developers

We strongly encourage you to review the project's scope described in the `README.md` file before working on new features. For large changes, consider writing a design document using [this template](https://goo.gl/YCQttR).

## Testing changing downstream

By default, downstream projects use both an `http_archive` of `npm_bazel_typescript` and the released `@bazel/typescript` and `@bazel/karma` npm packages. `postinstall` steps in these npm packages check that the version of the `npm_bazel_typescript` is compatible with the version of the npm package(s).

For example, if a downstream `WORKSPACE` contain:

```python
http_archive(
    name = "npm_bazel_typescript",
    url = "https://github.com/bazelbuild/rules_typescript/archive/0.21.0.zip",
    strip_prefix = "rules_typescript-0.21.0",
)
```

that that project's `package.json` would contain the matching:

```json
"@bazel/typescript": "0.21.0",
"@bazel/karma": "0.21.0",
```

When authoring changes and testing downstream, depending on the `@bazel/typescript` and `@bazel/karma` npm packages makes the workflow confusing and difficult.
To make authoring and testing changes downstream easier, it is recommended that you override the default `compiler` attribute of `ts_library` if making changes
to `ts_library` and the default `karma` attribute of `ts_web_test_suite`/`ts_web_test` if making changes to those rules.

For example, in `/internal/build_defs.bzl`, change

```python
"compiler": attr.label(
    default = Label(_DEFAULT_COMPILER),
```

to

```python
"compiler": attr.label(
    default = Label("@npm_bazel_typescript//internal:tsc_wrapped_bin"),
```

The correct defaults to use so that you are not depending on the npm package downstream are in `/internal/defaults.bzl`. Note, your downstream
workspace will also need the correct `@npm` dependencies available to build these targets (see `internal/e2e/typescript_3.1/package.json`).
In the case of the `angular` workspace, some `@npm` dependencies in this repository will also need to be changed to `@ngdeps` since `angular` does not have
an `@npm` workspace with npm dependencies.

Note, with this workflow the downstream version of `@npm//typescript` will be used to compile the `ts_library` targets in `npm_bazel_typescript`.
An example of this can be found under `internal/e2e/typescript_3.1`.

## Releasing

Start from a clean checkout at master/HEAD. Check if there are any breaking
changes since the last tag - if so, this will be a minor, if not, it's a patch.
(This may not sound like semver - but since our major version is a zero, the
rule is that minors are breaking changes and patches are new features).

1. Re-generate the API docs: `yarn skydoc`
1. May be necessary if Go code has changed though probably it was already necessary to run this to keep CI green: `bazel run :gazelle`
1. If we depend on a newer rules_nodejs, update the `check_rules_nodejs_version` in `ts_repositories.bzl`
1. `git commit -a -m 'Update docs for release'`
1. `npm config set tag-version-prefix ''`
1. `npm version minor -m 'rel: %s'` (replace `minor` with `patch` if no breaking changes)
1. Build npm packages and publish them: `TMP=$(mktemp -d -t bazel-release.XXXXXXX); bazel --output_base=$TMP run //:npm_package.publish && cd internal/karma && bazel --output_base=$TMP run //:npm_package.publish`
1. `git push && git push --tags`
1. (Temporary): submit a google3 CL to update the versions in package.bzl and package.json

[releases]: https://github.com/bazelbuild/rules_typescript/releases


text:
# C-Sharp-Matrix-Library
A simple matrix library written in pure C#.

# API Documentation
https://tristanbatchler.github.io/C-Sharp-Matrix-Library/

repository: tristanbatchler/C-Sharp-Matrix-Library

repo_name: tristanbatchler/C-Sharp-Matrix-Library


text:
# elixirforgirls.github.io
Static Site for [ElixirGirls Stockholm](https://elixirforgirls.github.io)

## Running the app in development

### install
This site is built using GitHub pages
- requires [Ruby 2.x and bundler](https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/#requirements)
- Clone this repo from GitHub
- Ensure you have the `jekyll` gem available to you.
- Inside the `elixirforgirls.github.io` directory:

    `bundle install`

### Running the site locally
- `bundle exec jekyll serve`

### Publish website

First you need to update the `GITHUB_REPONAME` variable in the `/Rakefile`
file. If you have a project GH Page you need to define the `baseurl` as
your project name and set `url` as empty in `_config.yml`.

- Organization/User GH Page
  - `rake org:publish`

- Project GH Page
  - `rake project:publish`


text:
# EL-SEC: ELastic Management of SECurity Applications on Virtualized Infrastructure

### Overview:
The concept of Virtualized Network Functions (VNFs) aims to move Network Functions (NFs) out of dedicated hardware devices into software that runs on commodity hardware. A single NF consists of multiple VNF instances, usually running on virtual machines in a cloud infrastructure. The elastic management of an NF refers to load management across the VNF instances and the autonomic scaling of the number of VNF instances as load on the NF changes. Here provide EL-SEC, an autonomic framework to elastically manage security NFs on a virtualized infrastructure.

As a use case, we deploy the Snort Intrusion Detection System as the NF on the GENI testbed. Concepts from control theory are used to create an Elastic Manager, which implements various controllers – in this use case, Proportional Integral (PI) and Proportional Integral Derivative (PID) – to direct traffic across the VNF Snort instances by monitoring the current load. RINA (a clean-slate Recursive InterNetwork Architecture) is used to build a distributed application that monitors load and collects Snort alerts, which are processed by the Elastic Manager and an Attack Analyzer, respectively. Software Defined Networking (SDN) is used to steer traffic through the VNF instances, and to block attack traffic. Our results shows that virtualized security NFs can be easily deployed using our EL-SEC framework. With the help of real-time graphs, we show that PI and PID controllers can be used to easily scale the system, which leads to quicker detection of attacks.

## Tutorial on GENI Testbed:
[Link](https://github.com/akhtarnabeel/ELSEC/blob/master/Tutorial.md) to EL-SEC Tutorial

## Video Demonstration:
[Link](https://www.youtube.com/watch?v=2T_d_J4AVn8) EL-SEC Elastic Management on GENI testbed

## Source Code:
[Source code](https://github.com/akhtarnabeel/ELSEC) of our GENI implementation is available on GitHub

## Experiment Traces:
[Link](https://github.com/akhtarnabeel/ELSEC/tree/master/ExperimentalTraces) to experiment traces.

## Publications:

- Nabeel Akhtar, Ibrahim Matta, Ali Raza and Yuefeng Wang

  **EL-SEC: ELastic Management of SECurity Applications on Virtualized Infrastructure**

  *IEEE INFOCOM International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT), 2018, Honolulu, Hawaii, USA, April 2018*

    [\[PDF\]](http://cs-people.bu.edu/nabeel/papers/ELSEC.pdf)[\[Bibtex\]](http://cs-people.bu.edu/nabeel/papers/bib/ELSEC.txt)[\[ProjetPage\]](https://github.com/akhtarnabeel/ELSEC)

- Nabeel Akhtar, Ibrahim Matta and Yuefeng Wang

  **Managing NFV using SDN and Control Theory**

  *IEEE/IFIP NOMS International Workshop on Management of the Future Internet (ManFI) 2016, Istanbul, Turkey, April 2016*

  [\[PDF\]](http://cs-people.bu.edu/nabeel/papers/NOMS2016.pdf) [\[Bibtex\]](http://cs-people.bu.edu/nabeel/papers/bib/NOMS2016.txt)

repository: akhtarnabeel/ELSEC

gha_language: Python

repo_name: akhtarnabeel/ELSEC


text:
# Cuttle

An embedded job scheduler/executor for your Scala projects.

# Concepts

Embedded means that cuttle is not an hosted service where you submit jobs to schedule/execute. Instead it is
a Scala library that you embed into your own project to schedule and execute a DAG of jobs. The DAG and the jobs
definitions are all written using the cuttle Scala API. The scheduling mechanism can be customized.

## Jobs

A cuttle project is composed of many [Jobs](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Job.html) to execute.

Each job is defined by a set of metadata (_such as the job identifier, name, etc._) and most importantly by a side effect function. This function handles the actual job execution, and its Scala signature is something like `Context => Future[Completed]` (_which can be read as “execute the job for this input parameter and signal me the completion or failure with the returned Future value”_).

The side effect function is opaque for cuttle, so it can't exactly know what will happen there (_it can be any Scala code_), but it assumes that the function:

- Is asynchronous and non-blocking. It will immediately return a [Future](https://www.scala-lang.org/api/current/scala/concurrent/Future.html) value that will be resolved upon execution success or failure.
- Produces a side effect, so calling it actually will do some job and mutate some state somewhere.
- Is idempotent, so calling it twice for the same input (_context_) won't be a problem.

Being idempotent is important because cuttle is an __at least once__ executor. It will ensure that the job has been successfully executed at least once for a given input. In case of failure or crash it may have to execute it again and so it may happen that the side effect function will succeed more that once. It would be very brittle otherwise.

## Scheduler

Executions of these jobs are planned by a [Scheduler](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Scheduler.html). Actually a job is always configured for a specific [Scheduling](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Scheduling.html) and this is the type `S` you usually see in the Scala API. This scheduling information allows to provide more information to the scheduler about how the jobs must be triggered.

The scheduler gets the list of job (a scheduling specific [Workload](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Workload.html)) as input and starts producing [Executions](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Execution.html). A basic scheduler can for example run a single execution for each job.

But of course more sophisticated schedulers can exist. Cuttle comes with a [TimeSeries](https://criteo.github.io/cuttle/api/com/criteo/cuttle/timeseries/TimeSeries.html) scheduler that executes a whole job workflow (a Directed Acyclic Graph of jobs) across time partitions. For example it can execute the graph hourly or daily. And it can even execute it across different time partitions such as a daily job depending on several executions of an hourly job.

The input context given to the side effect function depends of the scheduling. For example the input for a time series job is [TimeSeriesContext](https://criteo.github.io/cuttle/api/com/criteo/cuttle/timeseries/TimeSeriesContext.html) and contains basically the start and end time for the partition for which the job is being executed.

## Executor

The cuttle [Executor](https://criteo.github.io/cuttle/api/com/criteo/cuttle/Executor.html) handles the job executions triggered by the scheduler. When it has to execute a job for a given [SchedulingContext](https://criteo.github.io/cuttle/api/com/criteo/cuttle/SchedulingContext.html) it creates and execution, and then invoke the job's side effect function for it.

As soon as the execution starts, it is in the __Started__ state. Started executions are displayed in the UI with a special status indicating if they are __Running__ or __Waiting__. This actually indicates if the Scala code being currently executed is waiting for some external resources (_the permit to fork an external process for example_). But as soon as the execution is __Started__ it means that the Scala lambda behind is running!

An execution can also be in the __Stuck__ state. It happens when a given execution keeps failing: Let's say the scheduler wants to execute the job _a_ for the _X_ context. So it asks the executor which eventually executes the job side effect. If the function fails, the returned [Future](https://www.scala-lang.org/api/current/scala/concurrent/Future.html) fails and the scheduler is notified of that failure. Because the scheduler really wants that job to be executed for the _X_ context, it will submit it again. When the executor sees this new execution coming back after a failure it will apply a [RetryStrategy](https://criteo.github.io/cuttle/api/com/criteo/cuttle/RetryStrategy.html). The default strategy is to use an exponential backoff to delay the retry of these failing executions. While being in this state __Stuck__ executions are displayed in a special tab of the UI and it means that it is something you should take care of.

An execution can also be in __Paused__ state. It happens when the job itself has been paused. Note that this is a temporary state; eventually the job has to be unpaused and so the executions will be triggered, otherwise more and more paused executions will stack forever.

Finally executions can be __Finished__ either with a __Success__ or __Failed__ state. You can retrieve these old executions in the log for finished executions.

## Execution Platforms

The way to manage external resources in cuttle is via [ExecutionPlatform](https://criteo.github.io/cuttle/api/com/criteo/cuttle/ExecutionPlatform.html). An execution platforms defines the contract about how to use the resources. They are configured at project bootstrap and usually set limits on how resources will be used (_for example to only allow 10 external processes to be forked at the same time_).

This is necessary because potentially thousands of concurrent executions can happen in cuttle. These executions will fight for shared resources via these execution platforms. Usually a platform will use a priority queue to prioritize access to these shared resources, and the priority is based on the [SchedulingContext](https://criteo.github.io/cuttle/api/com/criteo/cuttle/SchedulingContext.html) of each execution (_so the executions with highest priority get access to the shared resources first_). For example the [TimeSeriesContext](https://criteo.github.io/cuttle/api/com/criteo/cuttle/timeseries/TimeSeriesContext.html) defines its [Ordering](https://www.scala-lang.org/api/current/scala/math/Ordering.html) in such way that oldest partitions take priority.

## Flow scheduling
Flow scheduling is a sub api based on cuttle core API, the goal is to provide a DAG Workflow which use the Core Cuttle `Job`.

To declare a job :
```scala
val job1 = Job("job1", FlowScheduling()) { implicit execution =>
    execution.streams.info("Just writing logged info in job execution");

    Future.successful(Finished); // Every job finish with a future
}
```
### Simple example

The declaration of the dag is supposed to be simple, let's say we have 5 jobs declared like this code.
```scala
def myWorkflow : FlowWorkflow = ((job1 --> job2) && (job3 --> job4)) --> jobfinal
```
 Flow allow you to compose easily to help you recompose DAG, let's say you have `job1` and `job2` that work together and you want to reuse them in other DAG.
 Then nothing block you to write them elsewhere and reuse them later and build them in subgraph.
 ```scala
 val subgraph1 = job1 --> job2
 val subgraph2 = job3 --> job4
 val finalDag = (subgraph1 && subgraph2) --> jobfinal
 ```
![alt Example 01](graphviz02.png)

`job1` wil be executed in parallel with `job3` then if both successfully finished then `job2` and `job4` will run.
`jobfinal` will run only if both subgraph are successfully runned.

### Path
If one of the job fail it will wait before all the current job **Running** to finish and then handle the error by looking for
an **error job** if there is no error job attached to the job **Failing** then it will just fail without recovery.

Error Job are basically normal job just there are attached in a different manner onto the graph.

If we want to attach errors job for the example 01 like this:

![alt Example 02](graphviz03.png)

**You have to do this**
```scala
val subgraph1 = job1.error(errorJob1) --> job2.error(errorJob2)
val subgraph2 = job3.error(errorJob3) --> job4.error(errorJob4)
val finalDag : FlowWorkflow = (subgraph1 && subgraph2) --> jobfinal
```
Doing something like this is practical if you want an error management different from one job to another, but if you want
to manage all error job the same way, then nothing block you to do this :
```scala
val subgraph1 = job1.error(errorJob) --> job2.error(errorJob)
val subgraph2 = job3.error(errorJob) --> job4.error(errorJob)
val finalDag : FlowWorkflow = (subgraph1 && subgraph2) --> jobfinal
```
Looking like
![alt Example 03](graphviz04.png)
Flow will rename the error job `job.id` with `${job.id}-from-${parent-job-id}` where *parent-job-id* is the id of
the job where it comes from, so `errorJob-from-job1`, `errorJob-from-job2` etc.

### Scheduler
The Workflow is backed by a scheduler that will read the DAG and run step by step the workflow.
You use the workflow by creating a `FlowExecutor` object and running one of its methods `start`, `parStart` or `runSingleJob`.

It is advised to use `FlowManager` and passing the `FlowExecutor` to it, the reasons are :
1. `FlowManager` use a *Semaphore* to serialize his graph, if you run two workflow based on the same workflow it will avoid both workflow trying to serialize at the same time the same graph.
2. `FlowManager` needs a `SignalManager` (explained after) to run, so if you use a signal system for your graph, it will create before execution and remove after execution the topic to received the signal for this particular **workflow execution**
3.  It can also run a single job based on a **workflow execution id** so if you want to rerun a job from an old execution by providing the FlowWorkflow and others informations.

**Note 1** : If your workflow has failed at one step for whatever reason, you can recall the method `start`
to rerun the workflow from last step failed. (One example in the test file `FlowTestsSpec.scala`).

**Note 2** : If a job fail because of an exception, then it will just fail the job without triggering the exception, the exception will still be readable in the database execution table.
### Signals
Flow provide a signal system based on Kafka and Fs2, the class `SignalManager` take a `KafkaConfig` and Key/Value Types matching the
pattern of Kafka Key/Value topic of Kafka.

`FlowManager` use `SignalManager[String, _]` to send message to workflow, it uses the Key of the Kafka Topic as *workflow id*
and the value can be anything you want, in our case we have created a simple trait called `EventSignal` that can be used as a value
in those Topic.

Here you have the only test made for signal in `FlowSignalTestsSpec.scala`

```scala
    val simpleWorkflow : WFSignalBuilder[String] = topic : SignalManager => {

      lazy val job1 =  Job("step-one", FlowScheduling(inputs = Json.obj("audience" -> "step is one".asJson))) { implicit e =>
        IO(Finished).unsafeToFuture()
      }

      lazy val job1bis = Job("step-one-bis", FlowScheduling()) { implicit execution =>
        execution.streams.info("On step bis")
        val receive = for {
          value <- topic
            .subscribeOnTopic(execution.context.workflowId)
            .evalTap(m => IO(println(s"received $m")))
            .head
            .compile
            .last
        } yield { Output(Json.obj("aud" -> s"step bis is done and received ${value.get._2}".asJson)) }

        receive.unsafeToFuture()
      }

      lazy val job2 = Job("step-two", FlowScheduling(inputs = Json.obj("test" -> "final test".asJson))) { implicit e =>
        val in = e.job.scheduling.inputs
        val x = e.optic.aud.string.getOption(in).get + " passed to step three"
        IO(Output(x.asJson)).unsafeToFuture()
      }

      (job1 && job1bis) --> job2
    }

    val kafkaCf = KafkaConfig("signal_cuttle", "signals", List("localhost:9092"))

    import cats.implicits._
    import scala.concurrent.duration._

    val program = for {
      transactor    <- xa
      stopping      <- Deferred[IO, Unit]
      signalManager <- SignalManager[String, String](kafkaCf)
      scheduler     <- FlowManager(transactor, signalManager)
      workflowWithTopic = simpleWorkflow(signalManager)

      graph1 <- FlowExecutor(transactor, "Run jobs with signal")(workflowWithTopic)
      graph2 <- FlowExecutor(transactor, "Run jobs with signal 2")(workflowWithTopic)

      flow1 <- scheduler.runOne(graph1).start
      flow2 <- scheduler.runOne(graph2).start

      runFlowAndStop = (flow1.join, flow2.join).parMapN { (resWorkflow1, resWorkflow2) =>
        (resWorkflow1, resWorkflow2)
      }.flatMap { res => stopping.complete(()).map(_ => res) }

      finalResult <- (runFlowAndStop,
            signalManager.broadcastTopic.interruptWhen(stopping.get.attempt).compile.drain,
            fs2.Stream.awakeEvery[IO](4 seconds).head.compile.drain *> signalManager.pushOne(graph2.workflowId, "step-one-bis").compile.drain,
            fs2.Stream.awakeEvery[IO](8 seconds).head.compile.drain *> signalManager.pushOne(graph1.workflowId, "step-one-bis").compile.drain)
        .parMapN { (bothWorkflow, _, _, _) => bothWorkflow }
    } yield finalResult

    val bothWorkflow = program.unsafeRunSync()
```
In this quit big example we declared a transcator postgresql, create a `Deferred` to stop the consumer when both workflow are done,
and then create the signal manager based on a config Kafka.

Then we use the type `WFSignalBuilder` which is basically a shortcut for `SignalManager[String, _] => FlowWorkflow`, meaning
you create a workflow based on a Signal Manager.

We run two workflow based on the same workflow waiting for them to finish, these workflow are waiting for an Event corresponding to their workflow id.
The Event in this case is a String but we don't care what value it is.

Then at the end we received the two workflow finished which means we receive either a *Left* which will be a **Throwable** or *Right* a tuple **(FlowWorkflow, JobState)**.
The tuple is actually the workflow you give plus all the inputs and outputs of the job, so perfect is you want to
see what happened to a job and the `JobState` which is a map of `FlowJob` and `JobFlowState` corresponding to the state of the job,
this is useful if you want to see if the job is running, failed or done.

### Input/Output

![alt Example Input output](graphviz05.png)

Input and output are managed with Json, in this example, `Job1` might take default input in this case do it like this at the declaration
```scala
val job1 = Job("job1", FlowScheduling(inputs = Json.obj(...))) {...}
```
You can do it with outputs too but it is useless.
In the job you will be able to access to optic by doing this in the side effect :
(Exemple I/O)
```scala
val job1 = Job("job1", FlowScheduling(inputs = Json.obj("myvalue" -> "something".asJson))) { implicit e =>
 val myvalue : Option[String] = e.optic.myvalue.string.getOption(e.job.scheduling.inputs)
 // ...
 Future.successful(Output(Json.obj("newval" -> "my new val".asJson)))
}
```
*It is not really convenient to use Json but due to too much problem with Core Cuttle too implement static type then Json as been favoured.*

`Job1` return obviously a Future but with an Output this time, therefore the scheduler will automatically
take the json and put it as an input in the `Job2`, the value `newval` will be available in `Job2`.

**Note :** in the case of `jobfinal` it will receive both `Job2` and `Job4` outputs but if both of them have common value
it will automatically create a map in the json where the key is the job id and the value corresponding to the value of this particular job.
If both job return a newval key ``jobfinal`` will receive in his inputs :
```json
"newval" : {
    "job2" : "value_in_job2",
    "job4" : "value_in_job4"
}
```
**Note 2** : The key are formatted following the rules in the file `FlowSchedulerUtils.scala`

**Note 3** : If you want to trigger an error job but still want him to send data then you can use `OutputErr(data : io.circe.Json)`

Possible `Future[Completed]` for the job are :
- `Finished` : Is like `Unit`, a success but nothing to return
- `Output(data : io.circe.Json)` : Return a json and successfully finish the job
- `OutputErr(data : io.circe.Json)` : Fail the job and return json
- `Fail` : Just fail the job

### Examples and more

See the folder `examples` or `tests` in `flow` module for more example.

You can render the graph by installing graphviz `brew install graphviz`
and then run the method `renderDot` from you `FlowWorkflow` object to get a png file of your graph.
# Usage

The library is cross-built for __Scala 2.11__ and __Scala 2.12__.

The core module to use is `"com.criteo.cuttle" %% "cuttle" % "0.9.2"`.

You also need to fetch one __Scheduler__ implementation:
- __Flow__: `"com.criteo.cuttle" %% "flow" % "0.9.2""`.
- __Cron__: `"com.criteo.cuttle" %% "cron" % "0.9.2""`.

# License

This project is licensed under the Apache 2.0 license.

# Copyright

Copyright © Criteo, 2019.

gha_language: Scala


text:
# AWS Elastic Container Service (ECS) ARM templates

We host our stateless Laravel applications containers in ECS.
We recently moved our applications from Intel-based images to ARM-based images.

We used these CloudFormation templates to build the ARM-based ECS cluster
and each ECS service. Refer to repos from past presentations, such as
[sgf-aws/aws-ecs-demo-laravel](https://github.com/sgf-aws/aws-ecs-demo-laravel)
for sample ``Dockerfile`` and ``buildspec.yml`` files.

We are publishing these CloudFormation templates because we could NOT find ANY
CloudFormation templates for ECS on ARM (as of December 2020).
Most templates are based on "Amazon Linux 1", which does NOT offer an ARM image.
These templates are derived from various existing CloudFormation templates
published by AWS as well as original research required to launch an ECS cluster
on ARM with "Amazon Linux 2".

## Limitations

The ONLY known limitation with this template is the inability to update outdated
ECS agents in the EC2 instances from the ECS console. It is unclear if the ECS
console is unable to interact with the newer Amazon Linux 2 OS image or if this
is an ARM-specific issue.

My workaround was to add a "LaunchTemplateCounter" value to the template that,
when incremented, forces CloudFormation to deploy new EC2 instances with the
latest available "Amazon Linux 2" image and latest ECS agent. This can be a slow
process, since CF will replace (start new, stop old) a single EC2 instance at a time.

## Create the ECS Cluster

The following CloudFormation template will create an ECS cluster and ARM-based
EC2 hosts for the cluster.
You can override various options, including the default EC2 instance type
(t4g.small) and the number of EC2 instances that will host this ECS cluster.

```
cloudformation/ecs-cluster.yml
```
NOTE: You must specify the following existing resources. This template does NOT create these resources:
* Existing VPC ID
* Existing Subnet IDs
* Optional Existing EC2 SSH Key Pair

## Create an ECS Web Service

The following CloudFormation template will create a web service designed to run
a Laravel web application. If you have multiple Laravel-based web services (e.g.
microservices), you will use this template to create a separate CloudFormation
stack for each web service.

```
cloudformation/ecs-service-web.yml
```

NOTE: You must specify the following existing resources. This template does NOT create these resources:
* Existing VPC ID
* Existing ECS Cluster ID (see first template above)
* Existing Task Container Image Path (e.g. ECR Path)
* Existing ALB Listener Path
* Existing Primary Hostname (hostname must already be pointed to ALB)
* Optional Existing Secondary Hostname (hostname must already be pointed to ALB)

## Create an ECS Worker Service

The following CloudFormation template will create a worker service designed to
run either a Laravel Cron worker, a Laravel Queue worker, or a
[Laravel Plain Queue](https://github.com/dusterio/laravel-plain-sqs) worker.

```
cloudformation/ecs-service-worker.yml
```

NOTE: You must specify the following existing resources. This template does NOT create these resources:
* Existing VPC ID
* Existing ECS Cluster ID (see first template above)
* Existing Task Container Image Path (e.g. ECR Path)
* Existing SQS Queue Names

## Cleanup

You must delete the following items BEFORE you can delete your CloudFormation
Stacks:

* None?

When you delete your CloudFormation stacks, most items will be automatically
deleted. The following items must be deleted manually:

* CloudWatch Log Groups
* Any items you created manually must be deleted manually.

repository: gonuxxiv/aws-ecs-arm

repo_name: gonuxxiv/aws-ecs-arm


text:
This folder contains the source of UML diagrams which are being used within the
sabi architecture documentation (see projects wiki on github)

To modify or generate the diagrams you require to install the PlantUML Plugin.
Examples or language reference can be found here: http://de.plantuml.com/sequence.html

If you are using a MAC you may also need the graphiz library as prerequisite, meaning
http://www.tekkie.ro/processes/create-beautiful-uml-diagrams-in-minutes-from-the-jetbrains-ide/

brew install graphviz

repository: StefanSchubert/sabi

gha_language: Java

repo_name: StefanSchubert/sabi


text:
---
layout: post
title: Nanocalligraphy paper combines art with advanced nanolithography
sub-title:
image: /img/nanocalligraphy.jpg
tags: [nanocalligraphy, nanofabrication, art]
comments: false
---

Congratulations to Nikolaos Farmakidis from the Advanced Nanoscale Engineering group at Oxford for his new paper entitled, "[Exploiting rotational asymmetry for sub-50 nm mechanical nanocalligraphy](https://doi.org/10.1038/s41378-021-00300-y)." This novel approach to nanolithography enabled Nik and co-lead author Jacob Swett to create complex patterns at the nanoscale inspired by the artwork of calligraphy artist [Majid Alyousef](https://majidalyousef.com/). Read more about their work which is featured on Oxford's [news blog](https://www.ox.ac.uk/news/features/mark-making-artist-s-impression-inside-oxford-s-bhaskaran-lab).

gha_license_id: NOASSERTION


text:
# Redeem a domain name

After a domain name expires, a 30-day grace period is provided for renewal. When the grace period ends, the domain name enters the redemption period. Domain names with different suffixes have different redemption periods. You need to pay a higher price to redeem your domain name within the redemption period. This topic describes how to redeem a domain name.

The renewal grace period of the domain name ends, and the domain name is in its redemption period.

## Procedure

To make sure that your domain name can be properly used, renew your domain name before it expires. This prevents extra payment for redemption. If you do not renew a domain name within the renewal grace period, the domain name enters the redemption period. Perform the following steps to redeem a domain name:

**Note:**

-   When a domain name enters the redemption period, the costs of redeeming the domain name from the domain name registry are much higher than the costs of registering or renewing the domain name. For more information about the redemption prices of domain names, see [t14309.md\#section\_n3z\_znv\_12b](/intl.en-US/Pricing/Domain name fees.md).
-   During the redemption period, DNS resolution is suspended. DNS resolution is immediately resumed when the domain name is redeemed. However, you must reset the DNS records of the domain name. The resolution recovery usually takes 24 to 48 hours to take effect.

1.  Log on to the [Alibaba Cloud Domains console](https://dc.console.aliyun.com).

2.  On the **Domain Names** page, click the **To Be Redeemed** tab.

    **Note:** If you cannot find your domain name on the **To Be Redeemed** tab, the domain name may have been deleted by the domain name registry because the redemption period has ended.

3.  Select the domain names that you want to redeem. In the lower part of the domain name list, select **Bulk Redeem**

4.  Complete the domain name redemption as prompted.

url: https://raw.githubusercontent.com/UMUTBAYGUT/domain/6155e06208e8b14f65a8c412c2b864763e611c51/intl.en-US/Domain name management/Redeem a domain name.md

repository: UMUTBAYGUT/domain

filename: Redeem a domain name.md

repo_name: UMUTBAYGUT/domain


text:
---
layout: page
title: APIs Worth Noting
permalink: /api/
---

Develop an app require many components. Fortunately, there are many APIs for different job, here are a list of note worthy API service.

#### 1. Location
- [iOS Location Service]()
- [Google Maps]()
- [Apple Maps]()
- [Foursquare]()
- [Yelps]()
- [Mapbox](http://mapbox.com)

#### 2. Activities
  - [Under Armour]()
  - [Fitbit]()
  - [Microsoft Health]()
  - [Google Fit]()

#### 3. Payment

  - [Braintree]()
  - [Stripe]()
  - [Paddle]()
  - [Apple Pay]()

#### 4. Weather

  - [Weather Channel]()
  - [Aerie Weather](http://www.aerisweather.com/develop/)

#### 5. Sign in

  - [Digits]()
  - [Facebook]()
  - [Twitter]()
  - [Goolge]()
  - [Microsoft]()

#### 6. Usage Tracking

   - [Answers]()

#### 7. Log Management

  - [Papertrail](https://papertrailapp.com)

#### 8. Issue Tracking

  - [Crashlytics]()

#### 9. Continuous Integration

  - [Jenkins]()
  - [CodeShip]()

#### 10. Source Code Management

  - [GitHub]()
  - [GitLab]()

#### 11. Wiki

  - [dokuWiki]()

#### 12. Machine Learning

  - [Google TensorFlow]()
  - [Microsoft Cognitive Services]()

Updated: April 4, 2016

---
Original: March 21, 2016


text:
---
title       : Introduction to Cluster Analysis
description : This course will teach you the basics of cluster analysis.
attachments :

--- type:NormalExercise lang:r xp:100 skills:1  key:211e1551ba
## Hierarchical Clustering

One of the most basic forms of cluster analysis is the hierarchical cluster. Hierarchical clustering measures disimilarity. Disimilarity can be thought of as distance from one observation to another.

To execute a hierarchical cluster we will first change our data to matrix form, calculate distance, preform the analysis, and the plot the cluster analysis.

A dataset with a selection of cars, `mtcars`, is available in the workspace.

*** =instructions
- View the structure of your data.
- Convert the data set to matrix format with `as.matrix()`, assign it to the object `cars`.
- Calculate the distance of the `cars` object with `dist()` and assign it to `distance`.
- Use `hclust()` to preform your cluster of the `distance` data, assign it to `h_clust`.
- Plot `h_clust`

*** =hint
- Use `str()` for the first instruction.
- Did you assign the matrix to `cars`?
- Did you calculate the distance?
- Did you preform `hclust()` on the right data?
- Did you plot `h_clust`?

*** =pre_exercise_code
```{r}
# Pre-load dplyr to access mtcars
library(dplyr)

# load mtcars:
mtcars <- data(mtcars)
```

*** =sample_code
```{r}
# mtcars is available in your workspace

# Check out the structure of mtcars

# Create a new matrix called "cars"

# Calculate the distance

# Perform a hierarchical cluster

# Plot h_clust

```

*** =solution
```{r}
# mtcars is available in your workspace

# Check out the structure of mtcars
str(mtcars)

# Create a new matrix called "cars"
cars <- as.matrix(mtcars)

# Calculate the distance
distance <- dist(cars)

# Perform a hierarchical cluster
h_clust <- hclust(distance)

# Plot h_clust
plot(h_clust)
```

*** =sct
```{r}
# Test whether the function str is called with the correct argument, object
# If it is not called, print something informative
# If it is called, but called incorrectly, print something else

test_function("plot", args = "x",
              incorrect_msg = "You didn't call `plot(___)` with the correct argument, `h_clust`")

test_function("str", args = "object",
              incorrect_msg = "You didn't call `str(___)` with the correct argument, `mtcars`.")

test_function("as.matrix", args ="x",
              incorrect_msg = "You didn't call `as.matrix(___)` with the correct object.")

test_output_contains("h_clust")

# Test the objects
test_object("distance")
test_object("cars")

test_error()

# Final message the student will see upon completing the exercise
success_msg("Good work!")
```

--- type:MultipleChoiceExercise lang:r xp:50 skills:1  key:a6e6db2517
## Quiz Time: Dissimilarity

How do we measure dissimilarity?

*** =instructions
- Standard Deviations
- RMSE
- Variance
- Distance

*** =hint
Have a look at the solution, what did we calculate from the matrix?

*** =pre_exercise_code
```{r}
# mtcars is available in your workspace

# Check out the structure of mtcars
str(mtcars)

# Create a new matrix called "cars"
cars <- as.matrix(mtcars)

# Calculate the distance
dist <- dist(cars)

# Perform a hierarchical cluster
h_clust <- hclust(dist)

#Plot h_clust
plot(h_clust)
```

*** =sct
```{r}
# The sct section defines the Submission Correctness Tests (SCTs) used to
# evaluate the student's response. All functions used here are defined in the
# testwhat R package

msg_bad <- "That is not correct!"
msg_success <- "Exactly! We calculate distance and use that as a measure of dissimilarity."

# Use test_mc() to grade multiple choice exercises.
# Pass the correct option (Action, option 2 in the instructions) to correct.
# Pass the feedback messages, both positive and negative, to feedback_msgs in the appropriate order.
test_mc(correct = 4, feedback_msgs = c(msg_bad, msg_bad, msg_bad, msg_success))
```
--- type:NormalExercise lang:r xp:100 skills:1  key:c0b56cff17
## K-Means Cluster

K-Means clustering is one of the most basic, and most used unsupervised algorithims. The main idea is to generate **k** number of clusters, or groups based on a mean center. We want each cluster to be as far from each other as possible. If clustered, we assume that each cluster is different from another.

Creating these clusters is fairly easy and requires you to indicate how many clusters you are searching for. K-means is performed using the `kmeans()`function. The function requires 2 arguments. The first argument passed is the matrix, and the second is the number (**k**) of clusters.

*** =instructions
- Preform a k-means cluster with 5 clusters and assign it to object `fit`.
- Subset data by cluster with `aggregate()` function.
- Create a new data frame with `cars` and `fit$cluster`, assign it to `cluster_cars`.

*** =hint
- Did you assign the correct number of clusters?
- Did you provide the correct argument for `aggregate()`
- Did you call the `data.frame()` with the correct arguments?

*** =pre_exercise_code
```{r}
set.seed(0)
library(dplyr)
data(mtcars)
cars <- as.matrix(mtcars)
```

*** =sample_code
```{r}
# The cars data is preloaded into your workspace.

# Preform your k-means cluster

# Aggregate your cars data by cluster fits
aggregate(___, by=list(fit$cluster), FUN = mean)

# Create a new data frame with the cars and fit data
```

*** =solution
```{r}
# The cars data is preloaded into your workspace.

# Preform your k-means cluster
fit <- kmeans(cars, 5)

# Aggregate your cars data by cluster fits
aggregate(cars, by=list(fit$cluster), FUN = mean)

# Create a new data frame with the cars and fit data
cluster_cars <- data.frame(cars, fit$cluster)
```

*** =sct
```{r}
#First Instruction
test_object("fit")

#Second Instruction
test_student_typed("aggregate(cars, by=list(fit$cluster), FUN = mean)",
                  not_typed_msg = "Something is wrong with `aggregate()`. Take another look at the instruction.")

#Third Instruction
test_function("data.frame",
              incorrect_msg = "Did you call cars & fit$cluster?")
test_object("cluster_cars")

test_error()

# Final message the student will see upon completing the exercise
success_msg("Good work!")
```

--- type:MultipleChoiceExercise lang:r xp:50 skills:1 key:36e8b9e8b4
##Understanding your cluster.

Take a look at your console, and try running the code.

This graph was created using ggvis. Check out the course on ggvis!

Which cluster has the best mileage?

*** =instructions
- 1
- 2
- 3
- 4
- 5

*** =hint
Look at the legend and the color of the dots!

*** =pre_exercise_code
```{r}
library(ggvis)
library(dplyr)

set.seed(0)

cars <- mtcars

fit <- kmeans(cars, 5)

# Aggregate your cars data by cluster fits
aggregate(cars, by=list(fit$cluster), FUN = mean)

# Create a new data frame with the cars and fit data
cluster_cars <- data.frame(cars, fit$cluster)
cluster_cars %>% ggvis(~mpg, ~disp, fill = ~factor(fit.cluster)) %>% group_by(fit.cluster)
```

*** =sct
```{r}
msg_bad <- "That is not correct, guess again."
msg_success <- "Wonderful! he 4th cluster has the best mileage."

# Pass the correct option (Action, option 2 in the instructions) to correct.
# Pass the feedback messages, both positive and negative, to feedback_msgs in the appropriate order.
test_mc(correct = 5, feedback_msgs = c(msg_bad, msg_bad, msg_bad, msg_bad, msg_success))
```

repository: norberello/clusterplaster

repo_name: norberello/clusterplaster


text:
# Tool for dividin up Shard space evenly

# Building

If you want a docker image then use the [Makefile](../Makefile) in the root:

```shell
make radixdlt-shard-allocator
```

# Deployment

Deployed to the "Explorer" node by the [mtps.py](../libcloud/mtps.py) script.

repository: radixdlt/mtps

repo_name: radixdlt/mtps


text:
# Constants and Formulas

Parameter           | Value                | Description
--------------------|----------------------|--------------------
Boost Amount        | 300                  |
Boost Consumption   | 1                    | consumption per one simulation step
Bullet Damage       | 10*power + 3*power^2 | power is a value between 0.1 and 1
Bullet Speed        | 4                    |
Gun Reloading Time  | 70*power             | power is a value between 0.1 and 1
Gun Turning Speed   | 3°                   | in degrees per simulation step
Gun Length          | 25                   | length of the gun (from the center of the tank to end of the barrel)
Hit Enemy Damage    | 0.2                  | damage received when colliding with another tank
Hit Wall Damage     | 0.2                  | damage received when colliding with a wall
Initial Energy      | 100                  |
Radar Beam Focal    | 6°                   | angle that radar beam covers
Radar Beam Range    | 300                  | how far away from the tank the radar scan
Radar Turning Speed | 6°                   | in degrees per simulation step
Ram Damage          | 0.1 + speed * 0.4    | damage dealt to an opponent when you ram him
Tank Moving Speed   | 2                    | enabling boost will double this value
Tank Turning Speed  | 2                    | in degrees per simulation step

repository: Gary-Ascuy/jsbattle

branch_name: refs/heads/develop

repo_name: Gary-Ascuy/jsbattle


text:
# What is this package?
This package was created to help decrease the time it takes to build a project
which needs to create, delete, and edit the project's [Laravel Model](http://laravel.com/docs/master/eloquent)s.

# Warden?
The project is named Warden because much like a prison warden, it can bring in an inmate (create a new item), take out an inmate (delete an item), or make changes to what the inmate can do
(edit an item). By no means is it limited to just a User's model. It (so far) can work an any model.

# How can I install this package?
Take a look at [our documentation](https://github.com/austinkregel/warden/blob/master/docs/readme.md)

# Questions?
Email me, my email is on [my github page](http://github.com/austinkregel), or you can make an issue. :)

repository: austinkregel/Warden

gha_language: PHP

repo_name: austinkregel/Warden


text:
# OCIO Configs

# [Modular ACES OCIO Config](/ocio-configs/config_modular-aces.ocio)

An OCIO config based on the [ACES 1.2 OCIO config](https://github.com/colour-science/OpenColorIO-Configs/tree/feature/aces-1.2-config/aces_1.2) from the [colour-science](https://github.com/colour-science) [OpenColorIO-Configs git repo](https://github.com/colour-science/OpenColorIO-Configs).

This description is going to be technical and wordy, so apologies in advance. Here's a pretty picture to keep you interested.

![Nuke OCIO Screenshot - DRT](/images/modular-aces_screenshot_nuke-drt.jpg)

### Terms
A very important thing when talking about color science topics is to define your terms and make sure they are clear. To whit, here is a list of terms I'll use below!

- **DRT** - Display Rendering Transform : This is not an ACES term, but I'm borrowing it [from Filmlight](https://vimeo.com/119143638) to describe the color transformation from scene linear wide gamut to display device. This transform generally contains some type of look, a tonemapping transform to map from scene linear to display linear, gamut transformations to map from working colorspace to display colorspace, and an EOTF for the display device.
- **RRT** - The ACES Reference Rendering Transform : Transforms from ACES to OCES. Contains the segmented_spline_c5 tonescale algorithm.
- **OCES** - The ACES Output Color Encoding Specification : A display-referred colorspace targeted for an idealized wide gamut hdr display device with AP1 primaries, and a 10000 nit dynamic range, in floating point candelas per meter squared encoding. (Some more information in [these](https://acescentral.com/t/odt-tonescale/387/11) [threads](https://acescentral.com/t/color-appearance-models-interpretation-options-and-odts/1257/4) on [acescentral.com](https://acescentral.com)).
- **ODT** - The ACES Output Device Transform : Transforms from OCES to a specific display device. Contains the segmented_spline_c9 tonescale algorithm.
- **SSTS** - The ACES Single Stage Tonescale : This is a newer tonescale algorithm which combines the segmented_spline_c5 and segmented_spline_c9 tonescales into a single parameterized tonescale that is simpler for HDR display transforms. In ACES 1.2, this tonescale is used for all HDR display transforms. The SDR display transforms still use the older RRT+ODT tonescale algorithms, but this [might change](https://acescentral.com/t/nuke-aces-output-transform-and-a-question/2554/3) with the ACES 2.0 Output Transforms.
- **LMT** - [Look Modification Transform](https://github.com/ampas/aces-dev/tree/master/transforms/ctl/lmt) : A color transform to apply a creative look to an image. This is an ACES -> ACES' transform.
- **CTL** - Color Transformation Language : The programmiing language that the source ACES code is written in.
- **EOTF** - Electrical to Optical Transfer Function : Transfer function to transform from electrical domain to optical domain, for example from pixels to a monitor.
- **Shaper** - A shaper space is a transfer function that transforms the input data into a more convenient representation for performing certain operations. Commonly, it is a linear to log transfer function which remaps scene linear into a 0 to 1 range, so that the available precision of a color transform like a 3d lut might be better utilized.
- **Code Values** - A term used to refer to the integer encoding common in display-referred images. For example an 8 bit jpeg image from the internet has 256 code values or discreet steps between 0 (black) and 1 (white).

![Nuke OCIO Screenshot - DRT](/images/modular-aces_screenshot_nuke-idt.jpg)

### Explanation

With the increasing amount of display rendering transforms (DRT), ACES OCIO configs have gotten increasingly large in file size. The [luts](https://github.com/colour-science/OpenColorIO-Configs/tree/feature/aces-1.2-config/aces_1.2/luts) folder of the latest config is 429.5MiB. This filesize is mostly from the fact that each DRT is baked out into a 3d lut. This lut describes the color transformation from the shaper log encoding of scene linear to the final code values for display on a monitor. The ACES config uses 65x65x65 luts, so each lut is about 9-10MiB.

When building the [Nuke ACES Output Transforms](https://github.com/jedypod/nuke-colortools), I started wondering if it had to be this way. The ACES Output transform has different stages, many of which are the same for each display render transform. Here's a breakdown of the basic stages of a typical ACES display rendering transform:
1. All have the default ACES LMT, described as the `rrt_sweeteners` in the [AMPAS CTL](https://github.com/ampas/aces-dev/blob/master/transforms/ctl/lib/ACESlib.RRT_Common.ctl). Technically, the rrt_sweeteners are included in the Reference Rendering Transform (RRT), but in my opinion, the function of this component of the RRT is to modify the look of the image to render the colors in a more visually appealing way, and thus is should be thought of as an LMT. It's also useful to conceptualize it this way because the `rrt_sweeteners` is actually the only component of the typical ACES display render transform that actually needs a 3d lut. (!)
2. A tonescale or tonemapping operation to compress the high dynamic range scene linear working space into low dynamic range display linear code values. This operation can be described by a single channel 1d lut since it is only a curve.
3. A gamut transformation to convert from the working colorspace gamut to the display gamut (with optional gamut limiting and whitepoint compensation). This operation can be described by a matrix transform.
4. An EOTF, which compensates for the transfer function of the display.

So instead of doing this in the OCIO config:
```
          |--------|      |-------|
          |        |      |       |
ACES ---->| SHAPER |----->|3D LUT |--> code values
          |        |      |       |
          |--------|      |-------|
```

We could do something like this:

```

          |--------|      |-------|      |-------|      |-------|      |-------|
          |        |      |       |      |       |      |       |      |       |
ACES ---->| SHAPER |----->|  LMT  |----->| SSTS  |----->|  PRI  |----->| EOTF  |--> code values
          |  (1D)  |      |  (3D) |      | (1D)  |      | (MTX) |      | (1D)  |
          |--------|      |-------|      |-------|      |-------|      |-------|

```

It's a bit more verbose to write out, but the advantage is that we only need a single 3D LUT to describe all display rendering transforms.

That takes our filesize from 429.5MiB to around 30MiB. Interesting!

Of course filesize in an OCIO config is not the primary concern. Easily understanding what is going on is probably the most important thing, and I don't claim this approach is better. It does make it more clear what is going on in the ACES output transform though, and I think this is useful for learning.

Another thing I don't like about the standard ACES OCIO configs is how much duplication of information there is. For example the color matrix to transform from ACES to ACEScg is duplicated many many times. To try to avoid this, I've taken the approach of building each transform as a modular colorspace. Each gamut transformation is defined as a colorspace. Each transfer function for each log space is defined as a colorspace. Then each Input Device Transform (IDT) is defined as a combination of the other colorspaces.

To keep things simple I'm using a single shaper log space for all output transforms: ACEScct. I'm sure there is a good reason that there are multiple shaper spaces being used in the official ACES config, probably optimization, but I haven't found a visual difference strong enough to warrant the increased complexity.

![Blender OCIO Screenshot](/images/modular-aces_screenshot_blender.png)

### Show Config
I've also been doing some thinking about color pipelines for balance grades (see [CalibrateMacbeth](https://gist.github.com/jedypod/798b365ea64e8121999e7036ae7e0217) and [BalanceGrade](https://gist.github.com/jedypod/d13595f856976869fe4cacd265a2b15e)). I've included a sketch of an idea for how a VFX company might set up a show. Generally in VFX you receive plates in camera colorspcae. These might be scene-linear exr files with Alexa Wide Gamut primaries, or DPX files encoded in RedLog3G10 / RedWideGamutRGB, or sometimes even weirder formats.

It is an advantage for streamlining and simplifying a color pipeline to have a single common working colorspace company wide that all images are transformed into on ingest. The most common emerging practice seems to be ACEScg.

To that end, this config includes a few colorspace definitions to describe those image states:
- work_lin: The working scene-linear colorspace.
- work_log: The working log colorspace.
- work_dmp: uint16 display-referred hdr wide gamut working colorspace for digital matte painting.
- src_lin: The external source image scene-linear colorspace.
- src_log: The external source image log colorspace.
- balance: The per-shot internal balance grade. Applied in work_lin.
- shot_grade: The external source DI shot grade.
- shot_grade_internal: The internal shot grade.
- view_show: The show view transform.
- view_log: The show log transform.
- view_shotgrade: The shot di grade view transform. Balance grade inverted if there is one.
- view_balance: The balance view transform: work_lin -> shot_grade_internal -> view_show.

# [Minimal ACEScg OCIO Config](/ocio-configs/config_minimal-acescg.ocio)
This is a minimal ACES OCIO config that uses ACEScg as the reference colorspace instead of ACES 2065-1 / AP0. It includes only the less obscure IDT transforms, and includes only a subset of the display render transforms.

# [Truelight ACES OCIO Config](/ocio-configs/config_truelight-acescg.ocio)
This config utilizes the Truelight CAM display rendering transforms instead of the ACES output transforms. It also includes the Filmlight scene look transforms (basically LMTs) as 3D LUTs. For more information, check out the [trulight lut creation folder](/lut-creation/truelight/).

repository: leegoonz/colortools

repo_name: leegoonz/colortools


text:
# Catedra
Catedra is an application used by an University in order to get access to certain information

# Screenshoots

| Main| Menu| Contact|
|-----|-----|-----|
|![Main](Screenshots/main.png)|![Menu](Screenshots/menu.png)|![Main](Screenshots/contact.png)|