text:
---
layout: post
title:  tidy resampling
date:   2017-10-26

---

I went through a phase a couple weeks ago where I became obsessed with knowing everything I could about permutation / Monte Carlo / bootstrapping-based statistical methods. But besides the excellent [Tidy bootstrapping with dplyr+broom](https://cran.r-project.org/web/packages/broom/vignettes/bootstrapping.html) I couldn't find many resources on doing randomization-based tests in keeping with the [tidy data](http://vita.had.co.nz/papers/tidy-data.html) philosophy and more modern tools for R (e.g. `purrr` and `modelr`).

At the end of the week I ended up with a lengthy R Markdown document which I figured I share in piecemeal over the coming weeks.

But first:

# why? And some definitions

**bootstrapping**: sampling _with replacement_ from your data. Mostly useful for getting a confidence interval or seeing the variation inherent some test statistic of interest.

**permutation testing:** creating all possible permutations of your data, calculating some test statistic each time to generate a null distribution of test statistics, and comparing the observed results to this distribution. Because permutation testing is often not feasible for larger datasets, I focus on the two other strategies.

**Monte-Carlo sampling:** Similar to permutation testing, but uses random or pseudo-random numbers to randomly permute the dataset to generate a null distribution and compare the distribution to the observed test statistic.

#### advantages:
- makes few assumptions, especially about the form of the distribution from which the data are draw (e.g. normally distributed)
- p-values are intuitive to calculate and understand
- seems like magic

#### disadvantages:
- p-values can change when you re-run the analysis (but can change less if you increase the number of times you permute the dataset)
- fewer resources on running permutation tests / less entrenched in the culture of hypothesis testing

## comparing two groups

Comparing some quantitative variable of interest in two groups is a common problem. For instance, many websites perform A/B testing where two different forms of a website are presented to users, data are collected, and the company then wants to know whether some user behavior differed between the two forms of the website.

Typically a two-sample t-test is used in this situation. But t-tests make a couple of assumptions that might not be true for our data. For instance, a two-sample t-test assumes that the two groups are independent, that the variance within each group is the same, and that the data within each group are normally distributed.

If we're uncomfortable with that last assumption, it usually makes sense to use a randomization or Monte-Carlo-based approach to testing. The approach works like this:

(1) Destroy any relationship between an observation and which of the two groups it belongs to. You can think of this as taking the labels that tell you whether a given datum belongs to group 1 or group 2, shuffling them, then randomly assigning the labels to the data points.

(2) Using this new dataset with shuffled labels, compute some statistic of interest, like the difference means between the two groups.

(3) Repeat many times.

Because we're randomly assigning the labels to the data points, it's equally likely that each data point gets a given label; this means that this particular method _assumes equal variance of the two groups_. There's a way around this assumption according to Good's _Resampling Methods: A practical guide to data analysis_ which I'll cover in another post.

The difference in means in all of our permuted datasets can be thought of as our _null distribution_. If there's no difference in the means between the two groups, we'd expect the difference in means we calculate from the data to fall within the middle 95% of these permuted means (assuming alpha = 0.05). We can calculate a p-value by finding the proportion of estimates in our null distribution that are more extreme than the test statistic we calculated on our dataset.

Below I show how to carry this out using a tidy data approach in R.

I'll use the `mtcars` dataset and test whether manual and automatic cars differ in their weights:

```r
library(tidyverse)
library(modelr)
library(broom)

mtcars %>%
  ggplot(aes(x = factor(am), y = wt)) +
  geom_boxplot() +
  theme_minimal()
```

![]({{site.baseurl}}/images/post9/plot1.png)

We can first run a traditional two-sample t-test:

```r

t.test(wt ~ factor(am), data = mtcars)

```
>   Welch Two Sample t-test
  ##
  ## data:  wt by factor(am)
  ## t = 5.4939, df = 29.234, p-value = 6.272e-06
  ## alternative hypothesis: true difference in means is not equal to 0
  ## 95 percent confidence interval:
  ##  0.8525632 1.8632262
  ## sample estimates:
  ## mean in group 0 mean in group 1
  ##        3.768895        2.411000

Next we can do the randomization-based approach outlined above:

```r

# number of permutations
n <- 999

permuted <- mtcars %>%
  mutate(am = factor(am)) %>% # factor am
  modelr::permute(n, am) %>% # permute the am column `n` times
  mutate(models = map(perm, ~ t.test(wt ~ am, data = .))) %>% # do a t-test for each permutation
  mutate(tidy = map(models, broom::glance)) %>% # extract useful statistics from the t-test
  unnest(tidy) # unnest the tidy column

```

`permuted` look like this:

```r
# A tibble: 999 x 13
perm   .id      models     estimate estimate1 estimate2   statistic    p.value parameter    conf.low  conf.high
<list> <chr>      <list>        <dbl>     <dbl>     <dbl>       <dbl>      <dbl>     <dbl>       <dbl>      <dbl>
 1 <S3: permutation>   001 <S3: htest>  0.677473684  3.492474  2.815000  2.00328522 0.05595078  25.34094 -0.01854899  1.3734964
 2 <S3: permutation>   002 <S3: htest>  0.278704453  3.330474  3.051769  0.85140972 0.40129861  29.94857 -0.38987074  0.9472796
 3 <S3: permutation>   003 <S3: htest>  0.316016194  3.345632  3.029615  0.97431917 0.33773747  29.81439 -0.34655899  0.9785914
```

Each row represents a different permutation. Using `broom::glance` allowed us to easily pull out a bunch of useful information from the t-test, and `unnest`ing the `tidy` column added these useful statistics as columns to the data frame.

The _estimate_ column here is the most useful to us: it represents the difference in the means between each group. If we've effectively destroyed any correlation between transmission type and weight, the average difference between the two groups should be zero. We can verify this is the case. I'll also find the observed difference between the groups in the non-randomized dataset and plot that as a red line:

```r

observed <- broom::glance(t.test(wt ~ factor(am), data = mtcars))$estimate

ggplot(permuted, aes(x = estimate)) +
  geom_histogram() +
  geom_vline(xintercept = observed, color = "red") +
  theme_minimal()

```

![]({{site.baseurl}}/images/post9/plot2.png)

We can also calculate the average difference and show that it's about 0 in the randomized datasets:

```r
mean(permuted$estimate)
```
> [1] 0.001492764

Looking at the plot above, it's clear the the observed difference is not what we've expect if there is no relationship between transmission type and weight, but we can go ahead and calculate a p-value:

```r

(sum(abs(permuted$estimate) > ifelse(observed > 0, observed, -observed)) + 1) / (n+1)

```
> [1] 0.001

This way of calculating p-values is based on [this](http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12102/full) paper and ensures p-values can never by 0, which wouldn't make any sense.

__Aside:__ To create randomized datasets that you can re-generate at some point in the future, use `set.seed(some_number)` before randomizing your data. This sets the seed of the random number generator in R, ensuring the same pseudo-random numbers are producing the next time you run your analysis.

---------------

In future posts, I'll look at bootstrapping regressions, correlations, and using Monte-Carlo-based resampling for ANOVA-like data.


text:
---
layout: post
title: Metric Compression Part 1
excerpt: "A review of research on compression of metrics"
categories: [computer science, machine learning, curse of dimensionality, metric spaces, math]
usemathjax: true
useemoji: true
---

# Prologue: What is space?

In the field of machine learning, there's a lot of focus on dealing with
high-dimensional vector spaces. Usually, though not always, the standard
Euclidean space $$\mathbb{R}^d$$, and if we want to measure the distance between
two vectors in $$\mathbb{R}^d$$, we typically choose the familiar Euclidean
distance: $$d(x, y) = \sqrt{(x_1-y_1)^2 + ... + (x_d-y_d)^2}$$. The combination
of an ambient "space" and distance "metric" is aptly referred to as a *metric
space*.[^space]

I would be missing out on audience engagement if I didn't somehow mention Deep
Learning :tm:, so here's the tie-in: an *artificial neural network* is just a
bunch of "dumb" *differentiable* transformations composed together. Some are
linear, some are non-linear, and the "deep" part just means we're composing a
lot of 'em in a big chain like $$g(x) = f_1(f_2(f_3(...f_{1000}(x)...)))$$. The
way we train them is with *backpropagation*, which is just a technical term for
a way of efficiently calculating the derivative and optimizing a loss function
$$L(x, g(x))$$.

The requirement here is that our functions are all differentiable - which means
we need to be able to do calculus somehow. The bog-standard metric space to do
calculus in is, you guessed it, Euclidean space.

Now let's say we chop a trained neural network in half, perhaps one that
classifies images, to see what the intermediate function outputs are. We get
some points which don't resemble the inputs or the final outputs. The idea of
*representation learning* is that these intermediate outputs do mean something -
the neural network has learned to "see" things like different lines, shapes, and
colors patches in the image. Our intermediate vector might have coordinates
which are $$1$$ if it "sees" a vertical line in the image, and $$0$$ if not. But
when we try to interpret these intermediate network outputs in general, there is
a simple question which is very difficult to answer concretely. What does the
Euclidean distance between two points in this intermediate space
*mean*?[^embedding]

Even though we rely on a distance metric for the idea of backpropagation to even
make sense, the question is unanswered. Doing the math to outline the algorithm
is one thing. Doing the math (and experiments!) to figure out what are the
*structures* that result in the "representation space" when we work with
real-world data, and explaining the *meaning* of any such structures, appears to
be orders of magnitude more difficult. I'll leave it at this: the question "what
is space?" might seem like it's only for mathematicians or physicists, or maybe
even philosophers - but it can actually be a hard question even in applications
of machine learning.

That said, we'll sweep that under the rug and pretend that as long as we're
working with Euclidean space, everything is cozy and well-understood.

# Introduction

One commonly encountered fundamental problem is *nearest neighbor search* (NNS).
In an "Intro to Machine Learning" course the idea of nearest neighbors might be
discussed as part of a simple classifier called *k-nearest neighbors* (k-NN),
which is simple in concept: if you have a bunch of training data $$(x_i, y_i)$$,
and you see a new input $$x'$$ then you can classify $$x'$$ by looking at the
$$k$$ nearest $$x_i$$ points in your training data and seeing what their
corresponding labels $$y_i$$ are. Related applications are in recommendations
and information retrieval, where the goal is just to find some of the closest
$$x_i$$ without any particular labels required.

However, to actually implement these, well, you have to *find* those $$k$$
nearest neighbors. In other words, you need a solution to the NNS problem. The
naive solution is to do an exhaustive search every time - compute the distances
between $$x'$$ and *all* $$x_i$$. Problem is, if you have $$d$$-dimensional data
and $$n$$ training points, that's $$O(nd)$$ scaling. If you're looking at more
than a thousand dimensions and a billion points, things quickly get out of hand.

In this post, we won't be looking at direct solutions to the NNS problem.
Instead I'll explore some background and theory for a related problem,
compressing metric spaces, concluding with a powerful data structure and
algorithm with both practical effectiveness and strong theoretical
justification.[^original]

# Fundamentals

Let's outline our goal a bit more clearly: what we want is a data structure and
algorithm, AKA an index, that lets us query for the distance between any two
data points in given data set $$X$$, and scales up to thousands of dimensions
and billions or even trillions of points.

That means we have to be really efficient - both constructing and
querying the index have to be close to linear in computation and storage. In
terms of main memory (RAM) we would like it if, similar to classic
databases[^db], we could operate with much less RAM than the whole index.[^mem]
The storage size is of particular interest, as with all DBs, because reading
from storage and the network is slow but cheap, while RAM is fast but expensive.

To achieve this, we're generally willing to accept a reasonable *approximation*
to the true distances, as long as it's still practically useful.

## JL Lemma

A famous classical result in this space is the *Johnson-Lindenstrauss (JL)
lemma* first published in 1984. Since that first publishing, there have been
various different proofs of the lemma, as well as variants of it applicable to
different contexts. We'll just look at applying the original result in Euclidean
space.

Essentially, the JL lemma tells us that if we have $$n$$ points in
$$\mathbb{R}^d$$, we can project those points down to a lower dimensional space
$$\mathbb{R}^k$$, while preserving the distances between any pair of the $$n$$
points.

Specifically, we can choose any "tolerance" $$\epsilon$$ - normally referred to
in this context as the *distortion* - which is basically a percentage delta
between $$0\%$$ and $$50\%$$. Then there exists a $$k \times d$$ matrix $$A$$
that projects our $$d$$-dimensional data into $$k$$-dimensional space, while
preserving distances up to a relative error of $$\pm \epsilon$$. Crucially,
$$k$$ scales *logarithmically* with $$n$$, and is completely *independent* of
the original dimensionality $$d$$.[^tight]

Here's the mathematical statement:

Let $$\epsilon \in (0, 1/2)$$, and let $$X = \{x_1,...,x_n\} \subset \mathbb{R}^d$$ be
a set of high-dimensional points. For some positive integer $$k = O(\log
(n)/\epsilon^2)$$, there exists a matrix $$A \in \mathbb{R}^{k \times d}$$ such that for any
$$x, y \in X$$, we have

$$ (1 - \epsilon) \|x - y\|_2^2 \leq \|Ax - Ay\|_2^2 \leq (1 + \epsilon)\|x - y\|_2^2$$

### Intuition

To review some linear algebra, $$A$$ is a $$k \times d$$ matrix which is
projecting a higher-dimensional space into a lower-dimensional one. There are
two interesting views of this matrix: the column view and the row view.

The column vectors of $$A$$ are what the standard basis vectors of
$$\mathbb{R}^d$$ get mapped to, forming a new basis in $$\mathbb{R}^k$$. You can
see how that works by multiplying $$A$$ by the standard basis vectors, or in
aggregate, the $$d$$-dimensional identity $$I$$. Since we want $$d$$ to be much
larger than $$k$$, it's impossible for the column vectors to be linearly
independent. In other words, a projection to lower dimensions is inherently
*lossy* - some pairs of distinct points in the original higher-dimensional space
will get mapped to the same lower-dimensional points.

The row vectors of $$A$$ are $$k$$ different $$d$$-dimensional vectors that we
are projecting each data point $$x_i$$ onto. In this case, it's possible for
these row vectors to be linearly independent (geometrically,
orthogonal/perpendicular). In fact, the larger $$d$$ gets, the more likely that
a "small" set of random "almost"-unit vectors will be "nearly" linearly
independent. This is good for us - although the mapping is lossy, we are
extracting the maximum amount of information from the points as we can get for a
given $$k$$.

## JL Lemma Applied

OK, so conceptually we could maybe turn our 10,000-dimensional data, even with a
trillion data points, into 100-dimensional data. We haven't answered some
crucial questions, though. How do we actually compute this projection matrix
$$A$$? What are the constants involved here - will we actually end up with
10-dimensional data, or 1000-dimensional data? Does this projection require each
individual coordinate to be more precise (e.g. by going from 32-bit
single-precision to 64-bit double precision), nullifying some of the
compression?

I won't review the entire body of work here, but there is a paper published 2003
from Dimitris Achlioptas from Microsoft Research [^randomproj] which provides us
with a particularly nice result in applying the JL lemma. The basic idea, which
is common across most applications of the JL lemma, is to construct $$A$$ by
random sampling. The interesting contribution of this result is that we can
sample from very simple distributions, and with high probability get a "good"
$$A$$.

More formally:

Let $$\beta > 0$$ and

$$k_0 = \frac{4 + 2\beta}{\epsilon^2/2-\epsilon^3/3} \log n$$

Let $$k > k_0$$ and $$A$$ be constructed by randomly sampling from either one of
these following two distributions: $$a_{ij} = \{\pm 1\}$$ uniformly, or
$$a_{ij}=\pm 1$$ with $$1/6$$ probability each and $$a_{ij} = 0$$ with $$2/3$$
probability.

Then after some scaling by $$\sqrt{k}$$, and $$\sqrt{3}$$ for the latter
probability distribution, this $$A$$ projects $$X$$ with $$(1 \pm
\epsilon)$$-distortion with probability $$> 1 - n^{-\beta}$$.

Now we have a pretty well-specified algorithm with some nice properties, we can
take a look at the practical consequences.

### Computation

Since the distributions are pretty simple, we need very few random bits. In the
distribution with only $$\pm 1$$, we only need a single bit of randomness per
matrix entry. For the distribution that includes $$0$$, we only need $$\approx
2.6$$ bits per entry.

The original paper suggests an interesting possibility for specializing the
required computation: for each of the $$k$$ coordinates, we can randomly throw
away $$2/3$$ of the $$d$$ inputs. Then we randomly partition the $$d$$ inputs
into two halves, sum each half, and then subtract the difference. Originally
they intended this to exploit the performance of SQL databases, but in the
modern day we might consider whether this has efficient implementations on GPUs
or even further specialized hardware.

Just on the face of it, I find it incredibly interesting that we can get away
with avoiding the *multiplication* part of matrix multiplication, and have this
elegant construction of $$A$$ which takes as little as a single "coin flip" per
entry. You don't even need to do anything special to maintain the length of the
vectors.

### Choice of $$\beta$$

For any $$n$$ big enough that we don't want to do brute force search, $$\beta =
1$$ is good enough -- the chance of getting a bad $$A$$ would be $$1/n$$. In
fact, if $$n > 10^8$$, we're probably even fine with $$\beta = \frac{1}{2}$$. Of
course, the $$2 \beta$$ term only accounts for $$1/3$$ of $$k$$, so while going
down to $$\beta = \frac{1}{2}$$ reduces $$k$$ by $$\approx 17\%$$, trying to get
it down much further will reduce our probability of success for tiny marginal
savings.

### Values of $$k$$

Let's say we have $$n = 10^8$$, and we choose $$\beta = \epsilon =
\frac{1}{2}$$. Then $$k_0 \approx 1105$$. OK, that might be interesting if the
original data is 10,000-dimensional, but if it was under 1,000 dimensions to
begin with, we're not achieving much. This is already with a fairly small
$$\beta$$ and large $$\epsilon$$, so it seems like this is a limitation of the
JL transform...

### Unluckiness

One trade-off we have to consider is whether it's necessary to maintain $$(1 \pm
\epsilon)$$-distortion across all points simultaneously. In other words, a
single pair of points which gets mapped to a too-wrong distance is considered a
violation of the JL lemma. If we're willing to accept that *some* pairs of
points will have distances that are slightly outside the $$\epsilon$$ error
range, what exactly is that trade-off? That is, if we intentionally choose a
too-small $$k$$, how often do we get bad pairs of points, and how off is the
mapped distance between them?

If the answer is not too often and not too far off, then maybe we could do a lot
better (smaller $$k$$), if we're willing to accept that, say, $$1\%$$ of the
time, a pair of points would have an error between $$\epsilon$$ and
$$2\epsilon$$. More generally, can we quantify the probability distribution of
the relative error between randomly chosen points?

I'm not aware of any published results in this context, but I'm fairly certain
that it's a straightforward exercise[^exercise] to compute this distribution, so
it's likely been done.

An advantage of this construction compared to other methods for nearest neighbor
indices is that we get a smooth fall-off of the error, so in that sense the
output (probabilistically) gets gradually worse with smaller $$k$$. However,
using the JL lemma has a marked disadvantage in that, if many of the distances
in the metric are close to each other, $$\epsilon$$ relative error could give
wrong neighbors frequently.

### Precision

Since this approach only involves a handful of multiplications by constants,
it's pretty clear that the coordinates we end up with have the same precision as
the ones we started with. Plus with only additions and subtractions (and scaling
by relatively small constants) we don't have to worry about numerical stability
or anything like that.

# Conclusion

In this post, we covered the idea of *metric spaces* and *nearest neighbor
search*, particularly in a high-dimensional context, to motivate the need for a
compressed index that does a good job of approximately preserving distances.
Next time, we'll dig deeper into a concrete analysis of the storage
requirements, and a more sophisticated data structure which is provably
near-optimal as a compressed index.

[^space]: Many of the results referenced in this post apply generally to all sorts of metric spaces. For simplicity we'll only cover the "standard" Euclidean $$\mathbb{R}^n$$.

[^embedding]: This also applies for various models which produce *embeddings* or any form of *latent representation*.

[^original]: Most of the content of this post series I originally wrote for [one of my graduate topics reports](https://github.com/rationalis/cse-291-final-report/blob/master/report_postsubmit.pdf)

[^db]: Nowadays databases solving this particular problem are referred to as *vector databases*.

[^mem]: If this system is part of a major feature which makes a ton of money, like say Amazon's recommendation engine, you can probably justify keeping everything in RAM. The rest of us probably can't.

[^randomproj]: Beyond just the result, [this paper](https://www.sciencedirect.com/science/article/pii/S0022000003000254) is also quite readable, outlines the motivations and intuitions quite well, and reviews specific instances of prior work. Technically it was published in 2001 but officially in a journal only in 2003.

[^exercise]: The original proof of the JL lemma involves starting from the *distributional JL lemma*, which is a consequence of the concentration of measure (AKA [the soap bubble](https://www.inference.vc/high-dimensional-gaussian-distributions-are-soap-bubble/)), and taking a union bound over pairs of points.

[^tight]: One important note here is that the JL bound is tight in the sense that, for any set size $$n$$, there's some pathological cases where you can't use any fewer than $$k$$ dimensions and still keep a low distortion. In fact, [the bound even applies to *nonlinear* embeddings](https://arxiv.org/abs/1609.02094), not just the linear ones guaranteed by the JL lemma. To do any better in terms of compression, we have to give up the full distortion guarantee, or we have to look outside of pure dimensionality reduction.

*[JL]: Johnson-Lindenstrauss

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
Here's a bunch of stuff made for [Finch](https://www.birdbraintechnologies.com/finch/).
---
When you import the projects and stop there, you'll end up with broken build paths and code consequently refusing to compile. To fix that:

1. Go to [Releases](https://github.com/dorukayhan/finch/releases)
2. Download and extract finch.zip
3. Find finch.jar and import it as an Eclipse user library named "Finch"

repository: dorukayhan/finch

repo_name: dorukayhan/finch


text:
title: Leaderboard
description: Shows a top ten leaderboard of who has the most oats.
path: tree/master/src/commands/economy
source: leaderboard.js

# Leaderboard

## Description

Shows a top ten leaderboard of who has the most oats.

## Aliases

* `top-10`
* `top-ten`
* `chart`
* `top`
* `top10`
* `topten`

## Usage

### Format

`leaderboard`

repository: BasilBug/docs

repo_name: BasilBug/docs


text:
# todo

[![CircleCI](https://circleci.com/gh/sjeandeaux/todo.svg?style=svg)](https://circleci.com/gh/sjeandeaux/todo)
[![Coverage Status](https://coveralls.io/repos/github/sjeandeaux/todo/badge.svg?branch=master)](https://coveralls.io/github/sjeandeaux/todo?branch=master)
[![Go Report Card](https://goreportcard.com/badge/github.com/sjeandeaux/todo)](https://goreportcard.com/report/github.com/sjeandeaux/todo)
[![](https://images.microbadger.com/badges/image/sjeandeaux/todo.svg)](https://microbadger.com/images/sjeandeaux/todo)

> Manage your todos with a daemon *todod* and a client *todo-cli*.

## Tools and development

### Structure

> The folder `todo-grpc` contains the schema of the microservice. The schema describes a simple CRUD of todos.
> The folder `todod` contains the default implementation of todo management based on mongo.
> The folder `todo-cli` contains the client which calls the daemon **todod**.
> The folder `pkg` contains the source code.
> The file `Dockerfile` use the multi-staging to generate an image docker with the binaries (todod and todo-cli).
> The folder `helm` contains the deployment.

### Tools

* make helps to manage common command lines.
* go the go module must be activated GO111MODULE=on.
* docker-compose for local tests.
* docker for the containerization.
* protoc to generate code from protobuff.

### Development

```sh
make help
clean                          clean the target folder
cover-html                     show the coverage in an HTML page
docker-compose-build           builds the application image with docker-compose.
docker-compose-up              spawns the containers.
fmt                            go fmt
generate                       generate the go from protobuf
gocyclo                        check cyclomatic
help                           prints help.
it-test                        integration test
lint                           go lint on packages
misspell                       misspellpackages
test                           test
tools                          download tools
vet                            go vet on packages
```

### Run the unit tests and integration tests

The tests are written in a Behavior Driven Development way and most of the time in TDD.

```sh
make tools #install the requirements for the test
make dependencies #download the go dependencies
make test
make it-test
```

The integration tests are not considered as [short test](https://golang.org/pkg/testing/#hdr-Skipping).

### Client

```bash
#Create
➜  todo-cli git:(develop) ✗ todo-cli create --title=ori --description='12factor apps' --state=NOT_STARTED --tags="job" --reminder=$(date +%s)
INFO[0000] ID:"5dc58ee5d954d9bc69be5523"
➜  todo-cli git:(develop) ✗ todo-cli create --title=ori --description='12factor eventstore' --state=NOT_STARTED --tags="job" --reminder=$(date +%s)
INFO[0000] ID:"5dc58f08d954d9bc69be5524"

#Read
➜  todo-cli git:(develop) ✗ todo-cli read --id 5dc58f08d954d9bc69be5524
INFO[0000] Todo:&{5dc58f08d954d9bc69be5524 ori 12factor eventstore NOT_STARTED [job golang]<PHONE_NUMBER>}

#Update
➜  todo-cli git:(develop) ✗ todo-cli update --id 5dc58f08d954d9bc69be5524 --title=ori --description='12factor eventstore' --state=NOT_STARTED --tags="job,golang" --reminder=$(date +%s)
INFO[0000] Updated:true

#Delete
➜  todo-cli git:(develop) ✗ todo-cli delete --id 5dc587ecd954d9bc69be5522                                                                                INFO[0000] Deleted:true

#Search
➜  todo-cli git:(develop) ✗ todo-cli search
INFO[0000] Todo:{5dc586b0a766c6e6404f01b5 Read - Challenge - todo Read - Should create a micro service with 12factor DONE [golang 12factor k8s]<PHONE_NUMBER>}
INFO[0000] Todo:{5dc58ee5d954d9bc69be5523 ori 12factor apps NOT_STARTED [job]<PHONE_NUMBER>}
INFO[0000] Todo:{5dc58f08d954d9bc69be5524 ori 12factor eventstore NOT_STARTED [job]<PHONE_NUMBER>}

➜  todo-cli git:(develop) ✗ todo-cli search --pattern '.event.*'
INFO[0000] Todo:{5dc58f08d954d9bc69be5524 ori 12factor eventstore NOT_STARTED [job]<PHONE_NUMBER>}
```

## CI/CD

The stack for the CI/CD:
* circle ci
* coveralls
* docker registry

The circle ci uses:
 * CODECOV_TOKEN
 * COVERALLS_REPO_TOKEN
 * DOCKER_LOGIN
 * DOCKER_PASSWORD

## Questions

### Prove how it aligns to 12factor app best practices

|Factor|Why does it fit|
|:----------:|:----------:|
|Codebase|The SCM follows the gitflow pattern. The `master` is for the production, `develop` for the staging. |
|Dependencies|It uses go mod which manages dependencies. Same happens with helm where we can spevify the dependencies the requirements.yaml file|
|Config|The `todod` application can be configured with environment variables|
|Backing services|The `todod` service has a resource which is the mongo database and configuration with `MONGO_URL`|
|Build, release, run|Build: docker image, Release: kubernetes|
|Processes|The `todod` application is stateless|
|Port binding|The `todod` application use the port 8080|
|Concurrency|The `todod` application is easy to scale|
|Disposability|If the database is up and ready, the `todod` application will start fast |
|Dev/Prod parity|The gitflow pattern allows to have the parity between dev and prod|
|Logs|The logs are redirected in stdout for the container and in a json file.|
|Admin processes|The adminstrotator can use the metrics from `/metrics` and the logs|

### Prove how it fits and uses the best cloud native understanding

* Containerized: The application `todod` is deployed in a container.
* Dynamically orchestrated: A load-balancer (traefik, nginx, ...) can be used above the services of todod.
* Microservices-oriented: The `todod` application manages only todos
* Statelessness: The `todod` can die. Nothing is lost.

### How would you expand on this service to allow for the use of an eventstore?

It should:
* use a event source such as Kafka, Pulser or Rabbit MQ.
* create the events Create, Update and Remove.
* create a projection which manages the events to be able to query them.

### How would this service be accessed and used from an external client from the cluster?

Multiple solutions exist:
* Nodeport.
* Ingress used here.
* Load-balancer.

## Todos

- [ ] better documentation in go.
- [ ] better coverage (ex: cli).
- [ ] deployment in CI/CD, give the version (image) in parameter.
- [ ] to keep it simple see [KO](https://github.com/google/ko) to run go project in kubernetes.
- [ ] Kubernates operator instead of helm chart for mongodb.
- [ ] create release for the client.

### Useful command lines

## Minikube

```bash
# Install the release
cd helm
helm dep update

export NAME=todo

helm install stable/nginx-ingress --name nginx --namespace dev
helm install --name ${NAME} --namespace dev .

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=${NAME}-todod.io"
kubectl create secret tls ${NAME}-todod-secret --key /tmp/tls.key --cert /tmp/tls.crt  -n dev
```

license: Unlicense

repository: sjeandeaux/todo

repo_name: sjeandeaux/todo


text:
# Inquirer.js

[![npm](https://badge.fury.io/js/inquirer.svg)](https://www.npmjs.com/package/inquirer)
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FSBoudrias%2FInquirer.js.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FSBoudrias%2FInquirer.js?ref=badge_shield)

A collection of common interactive command line user interfaces.

## Table of Contents

1.  [Documentation](#documentation)
    1.  [Installation](#installation)
    2.  [Examples](#examples)
    3.  [Methods](#methods)
    4.  [Objects](#objects)
    5.  [Question](#question)
    6.  [Answers](#answers)
    7.  [Separator](#separator)
    8.  [Prompt Types](#prompt-types)
2.  [User Interfaces and Layouts](#user-interfaces-and-layouts)
    1.  [Reactive Interface](#reactive-interface)
3.  [Support](#support)
4.  [Known issues](#issues)
5.  [News](#news)
6.  [Contributing](#contributing)
7.  [License](#license)
8.  [Plugins](#plugins)

## Goal and Philosophy

**`Inquirer.js`** strives to be an easily embeddable and beautiful command line interface for [Node.js](https://nodejs.org/) (and perhaps the "CLI [Xanadu](https://en.wikipedia.org/wiki/Citizen_Kane)").

**`Inquirer.js`** should ease the process of

- providing _error feedback_
- _asking questions_
- _parsing_ input
- _validating_ answers
- managing _hierarchical prompts_

> **Note:** **`Inquirer.js`** provides the user interface and the inquiry session flow. If you're searching for a full blown command line program utility, then check out [commander](https://github.com/visionmedia/commander.js), [vorpal](https://github.com/dthree/vorpal) or [args](https://github.com/leo/args).

## [Documentation](#documentation)

### Installation

```shell
npm install --save inquirer
```

```javascript
import inquirer from 'inquirer';

inquirer
  .prompt([
    /* Pass your questions in here */
  ])
  .then((answers) => {
    // Use user feedback for... whatever!!
  })
  .catch((error) => {
    if (error.isTtyError) {
      // Prompt couldn't be rendered in the current environment
    } else {
      // Something else went wrong
    }
  });
```

Inquirer v9 and higher are native esm modules, this mean you cannot use the commonjs syntax `require('inquirer')` anymore. If you want to learn more about using native esm in Node, I'd recommend reading [the following guide](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c). Alternatively, you can rely on an older version until you're ready to upgrade your environment:

```sh
npm install --save<EMAIL_ADDRESS>```

This will then allow import inquirer with the commonjs `require`:

```js
const inquirer = require('inquirer');
```

### Examples (Run it and see it)

Check out the [`packages/inquirer/examples/`](https://github.com/SBoudrias/Inquirer.js/tree/master/packages/inquirer/examples) folder for code and interface examples.

```shell
node packages/inquirer/examples/pizza.js
node packages/inquirer/examples/checkbox.js
# etc...
```

### Methods

#### `inquirer.prompt(questions, answers) -> promise`

Launch the prompt interface (inquiry session)

- **questions** (Array) containing [Question Object](#question) (using the [reactive interface](#reactive-interface), you can also pass a `Rx.Observable` instance)
- **answers** (object) contains values of already answered questions. Inquirer will avoid asking answers already provided here. Defaults `{}`.
- returns a **Promise**

#### `inquirer.registerPrompt(name, prompt)`

Register prompt plugins under `name`.

- **name** (string) name of the this new prompt. (used for question `type`)
- **prompt** (object) the prompt object itself (the plugin)

#### `inquirer.createPromptModule() -> prompt function`

Create a self contained inquirer module. If you don't want to affect other libraries that also rely on inquirer when you overwrite or add new prompt types.

```js
const prompt = inquirer.createPromptModule();

prompt(questions).then(/* ... */);
```

### Objects

#### Question

A question object is a `hash` containing question related values:

- **type**: (String) Type of the prompt. Defaults: `input` - Possible values: `input`, `number`, `confirm`, `list`, `rawlist`, `expand`, `checkbox`, `password`, `editor`
- **name**: (String) The name to use when storing the answer in the answers hash. If the name contains periods, it will define a path in the answers hash.
- **message**: (String|Function) The question to print. If defined as a function, the first parameter will be the current inquirer session answers. Defaults to the value of `name` (followed by a colon).
- **default**: (String|Number|Boolean|Array|Function) Default value(s) to use if nothing is entered, or a function that returns the default value(s). If defined as a function, the first parameter will be the current inquirer session answers.
- **choices**: (Array|Function) Choices array or a function returning a choices array. If defined as a function, the first parameter will be the current inquirer session answers.
  Array values can be simple `numbers`, `strings`, or `objects` containing a `name` (to display in list), a `value` (to save in the answers hash), and a `short` (to display after selection) properties. The choices array can also contain [a `Separator`](#separator).
- **validate**: (Function) Receive the user input and answers hash. Should return `true` if the value is valid, and an error message (`String`) otherwise. If `false` is returned, a default error message is provided.
- **filter**: (Function) Receive the user input and answers hash. Returns the filtered value to be used inside the program. The value returned will be added to the _Answers_ hash.
- **transformer**: (Function) Receive the user input, answers hash and option flags, and return a transformed value to display to the user. The transformation only impacts what is shown while editing. It does not modify the answers hash.
- **when**: (Function, Boolean) Receive the current user answers hash and should return `true` or `false` depending on whether or not this question should be asked. The value can also be a simple boolean.
- **pageSize**: (Number) Change the number of lines that will be rendered when using `list`, `rawList`, `expand` or `checkbox`.
- **prefix**: (String) Change the default _prefix_ message.
- **suffix**: (String) Change the default _suffix_ message.
- **askAnswered**: (Boolean) Force to prompt the question if the answer already exists.
- **loop**: (Boolean) Enable list looping. Defaults: `true`
- **waitUserInput**: (Boolean) Flag to enable/disable wait for user input before opening system editor - Defaults: `true`

`default`, `choices`(if defined as functions), `validate`, `filter` and `when` functions can be called asynchronously. Either return a promise or use `this.async()` to get a callback you'll call with the final value.

```javascript
{
  /* Preferred way: with promise */
  filter() {
    return new Promise(/* etc... */);
  },

  /* Legacy way: with this.async */
  validate: function (input) {
    // Declare function as asynchronous, and save the done callback
    const done = this.async();

    // Do async stuff
    setTimeout(function() {
      if (typeof input !== 'number') {
        // Pass the return value in the done callback
        done('You need to provide a number');
        return;
      }
      // Pass the return value in the done callback
      done(null, true);
    }, 3000);
  }
}
```

### Answers

A key/value hash containing the client answers in each prompt.

- **Key** The `name` property of the _question_ object
- **Value** (Depends on the prompt)
  - `confirm`: (Boolean)
  - `input` : User input (filtered if `filter` is defined) (String)
  - `number`: User input (filtered if `filter` is defined) (Number)
  - `rawlist`, `list` : Selected choice value (or name if no value specified) (String)

### Separator

A separator can be added to any `choices` array:

```
// In the question object
choices: [ "Choice A", new inquirer.Separator(), "choice B" ]

// Which'll be displayed this way
[?] What do you want to do?
 > Order a pizza
   Make a reservation
   --------
   Ask opening hours
   Talk to the receptionist
```

The constructor takes a facultative `String` value that'll be use as the separator. If omitted, the separator will be `--------`.

Separator instances have a property `type` equal to `separator`. This should allow tools façading Inquirer interface from detecting separator types in lists.

### Prompt types

---

> **Note:**: _allowed options written inside square brackets (`[]`) are optional. Others are required._

#### List - `{type: 'list'}`

Take `type`, `name`, `message`, `choices`[, `default`, `filter`, `loop`] properties.
(Note: `default` must be set to the `index` or `value` of one of the entries in `choices`)

![List prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/list.svg)

---

#### Raw List - `{type: 'rawlist'}`

Take `type`, `name`, `message`, `choices`[, `default`, `filter`, `loop`] properties.
(Note: `default` must be set to the `index` of one of the entries in `choices`)

![Raw list prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/rawlist.svg)

---

#### Expand - `{type: 'expand'}`

Take `type`, `name`, `message`, `choices`[, `default`] properties.
Note: `default` must be the `index` of the desired default selection of the array. If `default` key not provided, then `help` will be used as default choice

Note that the `choices` object will take an extra parameter called `key` for the `expand` prompt. This parameter must be a single (lowercased) character. The `h` option is added by the prompt and shouldn't be defined by the user.

See `examples/expand.js` for a running example.

![Expand prompt closed](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/expand-y.svg)
![Expand prompt expanded](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/expand-d.svg)

---

#### Checkbox - `{type: 'checkbox'}`

Take `type`, `name`, `message`, `choices`[, `filter`, `validate`, `default`, `loop`] properties. `default` is expected to be an Array of the checked choices value.

Choices marked as `{checked: true}` will be checked by default.

Choices whose property `disabled` is truthy will be unselectable. If `disabled` is a string, then the string will be outputted next to the disabled choice, otherwise it'll default to `"Disabled"`. The `disabled` property can also be a synchronous function receiving the current answers as argument and returning a boolean or a string.

![Checkbox prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/checkbox.svg)

---

#### Confirm - `{type: 'confirm'}`

Take `type`, `name`, `message`, [`default`, `transformer`] properties. `default` is expected to be a boolean if used.

![Confirm prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/confirm.svg)

---

#### Input - `{type: 'input'}`

Take `type`, `name`, `message`[, `default`, `filter`, `validate`, `transformer`] properties.

![Input prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/input.svg)

---

#### Input - `{type: 'number'}`

Take `type`, `name`, `message`[, `default`, `filter`, `validate`, `transformer`] properties.

---

#### Password - `{type: 'password'}`

Take `type`, `name`, `message`, `mask`,[, `default`, `filter`, `validate`] properties.

![Password prompt](https://cdn.rawgit.com/SBoudrias/Inquirer.js/28ae8337ba51d93e359ef4f7ee24e79b69898962/assets/screenshots/password.svg)

---

Note that `mask` is required to hide the actual user input.

#### Editor - `{type: 'editor'}`

Take `type`, `name`, `message`[, `default`, `filter`, `validate`, `postfix`, `waitUserInput`] properties

Launches an instance of the users preferred editor on a temporary file. Once the user exits their editor, the contents of the temporary file are read in as the result. The editor to use is determined by reading the $VISUAL or $EDITOR environment variables. If neither of those are present, notepad (on Windows) or vim (Linux or Mac) is used.

The `postfix` property is useful if you want to provide an extension.

### Use in Non-Interactive Environments

`prompt()` requires that it is run in an interactive environment. (I.e. [One where `process.stdin.isTTY` is `true`](https://nodejs.org/docs/latest-v12.x/api/process.html#process_a_note_on_process_i_o)). If `prompt()` is invoked outside of such an environment, then `prompt()` will return a rejected promise with an error. For convenience, the error will have a `isTtyError` property to programmatically indicate the cause.

## User Interfaces and layouts

Along with the prompts, Inquirer offers some basic text UI.

#### Bottom Bar - `inquirer.ui.BottomBar`

This UI present a fixed text at the bottom of a free text zone. This is useful to keep a message to the bottom of the screen while outputting command outputs on the higher section.

```javascript
const ui = new inquirer.ui.BottomBar();

// pipe a Stream to the log zone
outputStream.pipe(ui.log);

// Or simply write output
ui.log.write('something just happened.');
ui.log.write('Almost over, standby!');

// During processing, update the bottom bar content to display a loader
// or output a progress bar, etc
ui.updateBottomBar('new bottom bar content');
```

## Reactive interface

Internally, Inquirer uses the [JS reactive extension](https://github.com/ReactiveX/rxjs) to handle events and async flows.

This mean you can take advantage of this feature to provide more advanced flows. For example, you can dynamically add questions to be asked:

```js
const prompts = new Rx.Subject();
inquirer.prompt(prompts);

// At some point in the future, push new questions
prompts.next({
  /* question... */
});
prompts.next({
  /* question... */
});

// When you're done
prompts.complete();
```

And using the return value `process` property, you can access more fine grained callbacks:

```js
inquirer.prompt(prompts).ui.process.subscribe(onEachAnswer, onError, onComplete);
```

## Support (OS Terminals)

You should expect mostly good support for the CLI below. This does not mean we won't
look at issues found on other command line - feel free to report any!

- **Mac OS**:
  - Terminal.app
  - iTerm
- **Windows ([Known issues](#issues))**:
  - [Windows Terminal](https://github.com/microsoft/terminal)
  - [ConEmu](https://conemu.github.io/)
  - cmd.exe
  - Powershell
  - Cygwin
- **Linux (Ubuntu, openSUSE, Arch Linux, etc)**:
  - gnome-terminal (Terminal GNOME)
  - konsole

## Known issues

- **nodemon** - Makes the arrow keys print gibrish on list prompts.
  Workaround: Add `{ stdin : false }` in the configuration file or pass `--no-stdin` in the CLI.
  Please refer to [this issue](https://github.com/SBoudrias/Inquirer.js/issues/844#issuecomment-736675867)

- **grunt-exec** - Calling a node script that uses Inquirer from grunt-exec can cause the program to crash. To fix this, add to your grunt-exec config `stdio: 'inherit'`.
  Please refer to [this issue](https://github.com/jharding/grunt-exec/issues/85)

- **Windows network streams** - Running Inquirer together with network streams in Windows platform inside some terminals can result in process hang.
  Workaround: run inside another terminal.
  Please refer to [this issue](https://github.com/nodejs/node/issues/21771)

## News on the march (Release notes)

Please refer to the [GitHub releases section for the changelog](https://github.com/SBoudrias/Inquirer.js/releases)

## Contributing

**Unit test**
Please add a unit test for every new feature or bug fix. `npm test` to run the test suite.

**Documentation**
Add documentation for every API change. Feel free to send typo fixes and better docs!

We're looking to offer good support for multiple prompts and environments. If you want to
help, we'd like to keep a list of testers for each terminal/OS so we can contact you and
get feedback before release. Let us know if you want to be added to the list (just tweet
to [@vaxilart](https://twitter.com/Vaxilart)) or just add your name to [the wiki](https://github.com/SBoudrias/Inquirer.js/wiki/Testers)

## License

Copyright (c) 2023 Simon Boudrias (twitter: [@vaxilart](https://twitter.com/Vaxilart))

Licensed under the MIT license.

## Plugins

### Prompts

[**autocomplete**](https://github.com/mokkabonna/inquirer-autocomplete-prompt)

Presents a list of options as the user types, compatible with other packages such as fuzzy (for search)

![autocomplete prompt](https://github.com/mokkabonna/inquirer-autocomplete-prompt/raw/master/inquirer.gif)

[**checkbox-plus**](https://github.com/faressoft/inquirer-checkbox-plus-prompt)

Checkbox list with autocomplete and other additions

![checkbox-plus](https://github.com/faressoft/inquirer-checkbox-plus-prompt/raw/master/demo.gif)

[**inquirer-date-prompt**](https://github.com/haversnail/inquirer-date-prompt)

Customizable date/time selector with localization support

![Date Prompt](https://github.com/haversnail/inquirer-date-prompt/raw/master/examples/demo.gif)

[**datetime**](https://github.com/DerekTBrown/inquirer-datepicker-prompt)

Customizable date/time selector using both number pad and arrow keys

![Datetime Prompt](https://github.com/DerekTBrown/inquirer-datepicker-prompt/raw/master/example/datetime-prompt.png)

[**inquirer-select-line**](https://github.com/adam-golab/inquirer-select-line)

Prompt for selecting index in array where add new element

![inquirer-select-line gif](https://media.giphy.com/media/xUA7b1MxpngddUvdHW/giphy.gif)

[**command**](https://github.com/sullof/inquirer-command-prompt)

Simple prompt with command history and dynamic autocomplete

[**inquirer-fuzzy-path**](https://github.com/adelsz/inquirer-fuzzy-path)

Prompt for fuzzy file/directory selection.

![inquirer-fuzzy-path](https://raw.githubusercontent.com/adelsz/inquirer-fuzzy-path/master/recording.gif)

[**inquirer-emoji**](https://github.com/tannerntannern/inquirer-emoji)

Prompt for inputting emojis.

![inquirer-emoji](https://github.com/tannerntannern/inquirer-emoji/raw/master/demo.gif)

[**inquirer-chalk-pipe**](https://github.com/LitoMore/inquirer-chalk-pipe)

Prompt for input chalk-pipe style strings

![inquirer-chalk-pipe](https://github.com/LitoMore/inquirer-chalk-pipe/blob/main/screenshot.gif)

[**inquirer-search-checkbox**](https://github.com/clinyong/inquirer-search-checkbox)

Searchable Inquirer checkbox

![inquirer-search-checkbox](https://github.com/clinyong/inquirer-search-checkbox/blob/master/screenshot.png)

[**inquirer-search-list**](https://github.com/robin-rpr/inquirer-search-list)

Searchable Inquirer list

![inquirer-search-list](https://github.com/robin-rpr/inquirer-search-list/blob/master/preview.gif)

[**inquirer-prompt-suggest**](https://github.com/olistic/inquirer-prompt-suggest)

Inquirer prompt for your less creative users.

![inquirer-prompt-suggest](https://user-images.githubusercontent.com/5600126/40391192-d4f3d6d0-5ded-11e8-932f-4b75b642c09e.gif)

[**inquirer-s3**](https://github.com/HQarroum/inquirer-s3)

An S3 object selector for Inquirer.

![inquirer-s3](https://github.com/HQarroum/inquirer-s3/raw/master/docs/inquirer-screenshot.png)

[**inquirer-autosubmit-prompt**](https://github.com/yaodingyd/inquirer-autosubmit-prompt)

Auto submit based on your current input, saving one extra enter

[**inquirer-file-tree-selection-prompt**](https://github.com/anc95/inquirer-file-tree-selection)

Inquirer prompt for to select a file or directory in file tree

![inquirer-file-tree-selection-prompt](https://github.com/anc95/inquirer-file-tree-selection/blob/master/example/screenshot.gif)

[**inquirer-tree-prompt**](https://github.com/insightfuls/inquirer-tree-prompt)

Inquirer prompt to select from a tree

![inquirer-tree-prompt](https://github.com/insightfuls/inquirer-tree-prompt/blob/main/example/screenshot.gif)

[**inquirer-table-prompt**](https://github.com/eduardoboucas/inquirer-table-prompt)

A table-like prompt for Inquirer.

![inquirer-table-prompt](https://raw.githubusercontent.com/eduardoboucas/inquirer-table-prompt/master/screen-capture.gif)

[**inquirer-interrupted-prompt**](https://github.com/lnquy065/inquirer-interrupted-prompt)

Turning any existing inquirer and its plugin prompts into prompts that can be interrupted with a custom key.

![inquirer-interrupted-prompt](https://raw.githubusercontent.com/lnquy065/inquirer-interrupted-prompt/master/example/demo-menu.gif)

[**inquirer-press-to-continue**](https://github.com/leonzalion/inquirer-press-to-continue)

A "press any key to continue" prompt for Inquirer.js

![inquirer-press-to-continue](https://raw.githubusercontent.com/leonzalion/inquirer-press-to-continue/main/assets/demo.gif)

gha_language: JavaScript

gha_license_id: MIT


text:
---
name: Bug report
about: Create a report to help us improve

---

**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the **Bug** tag.

**Describe the bug:**
A clear and concise description of what the bug is.

**To Reproduce:**
Steps to reproduce the behavior:
  * Step 1: ...
  * Step 2: ...

**Expected behavior:**
A clear and concise description of what you expected to happen.

**Setup Details:**
Any details about your local setup that might be helpful in troubleshooting. Include the details about the versions of:
 - OS type and version:
 - Python:
 - scikit-learn:
 - numpy:
 - keras:
 - scipy:
 - tensorflow:

**Additional context:**
Add any other context about the problem here.

repository: yitzuchen/autokeras

repo_name: yitzuchen/autokeras


text:
# Calendar Skill (Productivity)

The Calendar Skill provides Calendar related capabilities to a Virtual Assistant.
The most common scenarios have been implemented in this beta release, with additional scenarios in development.

## Table of Contents

- [Supported Sources](#supported-sources)
- [Supported Scenarios](#supported-scenarios)
- [Scenario Configurations](#scenario-configurations)
- [Skill Deployment](#skill-deployment)
- [Language Model](#language-model)

## Supported Sources

> Office 365 and Outlook.com through the Microsoft Graph is supported along with support for Google accounts.

To use Google account in skill you need to follow these steps:
1. Create your calendar API credential in [Google developers console](https://console.developers.google.com).
2. Create an OAuth connection setting in your Web App Bot.
    - Connection name: `googleapi`
    - Service Provider: `Google`
    - Client id and secret are generated in step 1
    - Scopes: `https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/contacts`.
3. Add the connection name, client id, secret and scopes in appsetting.json file.

## Supported Scenarios

The following scenarios are currently supported by the Skill:

- Accept a Meeting
  - *I'll attend the meeting this afternoon*
  - *Accept the event sent by Yolanda Wong*
- Change an Event
  - *Bring forward my 4:00 appointment two hours*
  - *Reschedule my interview on monday to 1 pm*
- Connect to a Meeting
  - *Connect me to conference call*
  - *Connect me with my 2 o'clock meeting*
- Create a Meeting
  - *Create a meeting tomorrow at 9 AM with Lucy Chen*
  - *Put anniversary on my calendar*
- Delete a Meeting
  - *Cancel my meeting at 3 PM today*
  - *Drop my appointment for monday*
- Find a Meeting
  - *Do I have any appointments today?*
  - *Get to my next event*
- Find an Event by Time
  - *What day is Lego Land scheduled for?*
  - *What time is my next appointment?*
- Find an Event by Location
  - *Where is my meeting with Kayla?*
  - *Where do I need to be next?*
- Find an Event by Attendee
  - *Who am I meeting at 10 AM tomorrow?*
  - *Who is in my next meeting?*
- Find an Event's Duration
  - *How long will the next meeting last?*
  - *What's the duration of my 4 PM meeting?*
- Time Remaining
  - *How long until my next meeting?*
  - *How many days are there until Thanksgiving?*

## Scenario Configurations
In dialogs such as `Create Meeting`, the user needs to provide information such as title and meeting content before able to create a meeting. Depending on the user context (e.g. Speech driven and whilst driving) you can configure default slot-filling to minimise the number of questions - e.g. `Schedule a project review meeting with alex tomorrow at 8pm` would create the meeting with a default length of 30 minutes and no meeting contents.

This behaviour can be configured in the Skill `appsettings.json` by setting `isSkipByDefault` to true, and modify `EventTitle`, `EventContent` and `EventLocation` default values in `CalendarCommonStrings.resx` to set default value of different locales.

The `EventStartDate` default value should be a integer that means date difference with today. For example, if you want to set default start date as next day of meeting created date, the default value should be `1`.

The `EventDuration` default value is the default duration expressed in minutes (number).

```json
"defaultValue": {
    "createMeeting": [
        {
            "name": "EventTitle",
            "isSkipByDefault": false
        },
        {
            "name": "EventContent",
            "isSkipByDefault": false
        },
        {
            "name": "EventStartDate",
            "isSkipByDefault": false,
            "defaultValue": ""
        },
        {
            "name": "EventDuration",
            "isSkipByDefault": false,
            "defaultValue": ""
        },
        {
            "name": "EventLocation",
            "isSkipByDefault": false
        }
    ]
}
```

## Skill Deployment

The Calendar Skill require the following dependencies for end to end operation which are created through an ARM deployment script which you can modify as required.

- Azure Web App
- Azure Storage Account (Transcripts)
- Azure Application Insights (Telemetry)
- Azure CosmosDb (State)
- Azure Cognitive Services - Language Understanding

> Review the pricing and terms for the services and adjust to suit your scenario.

**To deploy your services using the default configuration, follow the steps in this common [deployment documentation page](/docs/tutorials/assistantandskilldeploymentsteps.md) from the folder where your have cloned the GitHub repo.**

### Authentication Connection Settings

If you plan to use the skill as part of a Virtual Assistant the process of registering a skill with your Virtual Assistant will create the supporting authentication connection information automatically for your Virtual Assistant. This skill uses the following authentication scopes which are registered automatically:

- `User.ReadBasic.All`
- `Calendars.ReadWrite`
- `People.Read`
- `Contacts.Read`

**However**, if you wish to use the Skill directly without using a Virtual Assistant please use the following steps to manually configure Authentication for the Calendar Skill. This is **not** required when using the Skill with a Virtual Assistant.

Follow the general instructions [here](/docs/reference/skills/manualauthsteps.md) to configure this using the scopes shown above.

## Language Model

LUIS models for the Skill are provided in .LU file format as part of the Skill. Further languages are being prioritized.

|Supported Languages|
|-|
|English|
|French|
|Italian|
|German|
|Spanish|
|Chinese (simplified)|

### Intents

|Name|Description|
|-|-|
|AcceptEventEntry| Matches queries to accept an event|
|ChangeCalendarEntry| Matches queries to change an event|
|CheckAvailability| Matches queries to check a contact's availability |
|ConnectToMeeting| Matches queries to connect to a meeting|
|ContactMeetingAttendees| Matches queries to contact the attendees of a meeting|
|CreateCalendarEntry| Matches queries to create a calendar entry|
|DeleteCalendarEntry| Matches queries to delete a calendar entry|
|FindCalendarDetail| Matches queries to find a calendar entry with details|
|FindCalendarEntry| Matches queries to find a calendar entry|
|FindCalendarWhen| Matches queries to get the time of a meeting|
|FindCalendarWhere| Matches queries to get the location of a meeting |
|FindCalendarWho| Matches queries  to get the attendees of a meeting|
|FindDuration| Matches queries to find out how long a meeting is|
|FindMeetingRoom| Matches queries to find a meeting room|
|GoBack| Matches queries to return to the previous step of a dialog|
|NoLocation| Matches queries to not specify a location|
|ReadAloud| Matches queries to read a calendar entry aloud |
|TimeRemaining| Matches queries to get the time until a meeting begins|

### Entities

|Name|Description|
|-|-|
|AskParameter| Simple entity|
|ContactName| Simple entity|
|DestinationCalendar| Simple entity|
|Duration| Simple entity|
|FromTime| Simple entity|
|Location| Simple entity|
|MeetingRoom| Simple entity|
|MoveEarlierTimeSpan| Simple entity|
|MoveLaterTimeSpan| Simple entity|
|OrderReference| Simple entity|
|PositionReference| Simple entity|
|RelationshipName| Simple entity|
|SlotAttribute| Simple entity|
|Subject| Simple entity|
|ToDate| Simple entity|
|ToTime| Simple entity|
|datetimeV2| Prebuilt entity|
|number| Prebuilt entity|
|ordinal| Prebuilt entity|

repository: lilydawn/botframework-solutions

repo_name: lilydawn/botframework-solutions


text:
# The MIR type-check

A key component of the borrow check is the
[MIR type-check](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/nll/type_check/index.html).
This check walks the MIR and does a complete "type check" -- the same
kind you might find in any other language. In the process of doing
this type-check, we also uncover the region constraints that apply to
the program.

TODO -- elaborate further? Maybe? :)

repository: stupidhuman/rustc-guide

gha_license_id: NOASSERTION

repo_name: stupidhuman/rustc-guide


text:
---
layout: article
title: EE 210 - Control Theory
aside:
  toc: true
sidebar:
  nav: docs-en
categories: minor elec ee-210
permalink: /course_reviews/:categories
tags: elec theory
---

# Autumn 2019
### Prof. Madhu N. Belur
**Author**: K T Prajwal Prathiksh

**Pre-requisite courses**: MA 108 (Not a hard pre-requisite)

**Pre-requisite skills**: None. The concepts required can be picked up on the go.

**Course Content**:
* Continuous-time signals and systems - signal characteristics, properties of continuous-time systems
* Continuous linear time-invariant systems
* Impulse response & Convolution
* Linear constant-coefficient differential equations
* Fourier series, Fourier transform, Laplace transform
* System analysis frequency response
* Analog filters
* State-space analysis for continuous-time systems
* Discrete-time signals and systems
* Discrete-time LTI systems - convolution, difference equations
* Sampling

**Motivation to take up the course**: Wanted to pursue an Electrical minor, and was keen on starting it off with a theoretical course as opposed to a course that had lab-work.

---

**Information about Projects/Assignments**: No assignments or projects were conducted. Just quizzes, Mid-Sem & End-Sem were used for the evaluation.

**Quizzes/Midsem/Endsem papers Difficulty**: 3/5

**Overall Course Difficulty**: Moderate

**Average Time Commitment**:
(Apart from lectures and tutorials)
Less than 3 hrs

**Attendance Policy**: 80% Attendance was told, however, it wasn't strictly enforced.

---

**General funda**: Go through the standard reference book by Alan V. Oppenheim. Work out the problems given in the book, and tutorials. This is enough to crack the course.

**Grading stats**: Normally distributed about 'BC'.

**Professor's Teaching Style**: The method of teaching was good. However towards the end of the course, content was being repeated more often than not, and hence classes became a bit boring.

**Should you do this course?**: Someone who is interested in understanding what the theoretical aspects of an Electrical minor comprises of can take up this course.
Even students who are interested in a SysCon minor, but have not yet made up their mind and want instead want to try out related courses before ca pick this one up.

---

branch_name: refs/heads/sources

gha_language: SCSS


text:
# Python Bytes 226

Special guests:

- [**Kelly Schuster-Pared**](https://twitter.com/kellypared)[**es**](https://twitter.com/kellypared)
- [**Sean Tibor**](https://twitter.com/smtibor)

**Brian #1:** [**DataClass vs NamedTuple vs Object: A Battle of Performance in Python**](https://medium.com/@jacktator/dataclass-vs-namedtuple-vs-object-for-performance-optimization-in-python-691e234253b9)

- Jack Song
- I’ve been using dataclass for a while now and love it. For some reason, I thought I heard there was some performance hit from them, so I was a bit worried before reading this.
- Jack came up with “a performance tests to compare the different *size* and *speed* when creating, reading and executing functions for `Object`, `NamedTuple` and the new `DataClass` introduced in Python 3.7”

|                  | **Object**   | **NamedTuple** | **DataClass** |
| ---------------- | ------------ | -------------- | ------------- |
| create           | 2.94 µs      | **2.01 µs**    | 2.34 µs       |
| read property    | **24.7 ns**  | 26.9 ns        | **24.7 ns**   |
| nested property  | **48.1 ns**  | 75.8 ns        | 52.1 ns       |
| execute function | 829 ns       | 946 ns         | **821 ns**    |
| size             | **56 bytes** | 80 bytes       | **56 bytes**  |

- Marvelous. Dataclass is still awesome. At the very least, it’s on the same order of size and speed as other structures.
- Further questions:
	- This was a limited bit of code, and performance metrics always depend on what kind of example code was used. If anyone has info about different performance examples that give wildly different results or even similar results, regarding DataClass, I’d love to hear about it.

**Michael #2:** [**Can My Water Cooled Raspberry Pi Cluster Beat My MacBook?**](https://www.the-diy-life.com/can-my-water-cooled-raspberry-pi-cluster-beat-my-macbook/)

- Did you know there were Raspberry Pi clusters? Amazing.
- Two primary things of interest
	- Pi clusters exist!
	- All the tools and setup to make the pi cluster nodes run in a grid computing scenario
- Compared against a mediocre HP laptop:
	- 10,000 → 1.69 sec
	- 100,000 → 74 sec
	- 200,000 → 268 sec
- Compared against an Intel MB Air:
	- 10,000 → .88 sec
	- 100,000 → 83 sec
	- 200,000 → 355 sec
- **Pi cluster, one node**:
	- 10,000 → 1.57 sec
	- 100,000 → 148 sec
	- 200,000 → 646 sec
- **Pi cluster, grid computed**:
	- 10,000 → .65 sec
	- 100,000 → 22 sec
	- 200,000 → 85 sec
- Ran the [**same script**](https://www.the-diy-life.com/can-my-water-cooled-raspberry-pi-cluster-beat-my-macbook/#test_script) on my **Mac Mini M1** (on Python 3.9.2 Rosetta, **single threaded**)
	- 10,000 → .33 sec
	- 100,000 → 24 sec
	- 200,000 → 91 sec
- Ran the [**my parallel script**](https://gist.github.com/mikeckennedy/ffe8a65473e974d0582da0c74a8f1c67) on my **Mac Mini M1** (on Python 3.9.2 Rosetta, **multi-threaded**)
	- 10,000 → .28 sec
	- 100,000 → 7 sec
	- 200,000 → 26 sec
	- 500,000 → 2m (vs. 9m on the cluster)
![](https://trello-attachments.s3.amazonaws.com/58e3f7c543422d7f3ad84f33/6052ec2a4b948a1595ea0114/f279fa64f7d2ebf93201318397726e26/Screen_Shot_2021-03-21_at_12.31.33_PM.png)

- Ran the [**my parallel script**](https://gist.github.com/mikeckennedy/ffe8a65473e974d0582da0c74a8f1c67) on my **Alienware sim racing machine** (**16 core, multi-threaded**)
	- 10,000 → .27 sec
	- 100,000 → 2.7 sec
	- 200,000 → 10 sec
	- 500,000 → 1m
	- **[Video of the Alienware sound](https://www.youtube.com/watch?v=rkRcvehsP6g)**

**Kelly #3:** [**There is an app for that!**](https://www.programiz.com/learn-python)
I am always trying to find time to practice Python and learn more ways to teach Python Basics. I am always on the go, so having Python Practice online and an online editor is always a plus.

- Programiz a company out of Nepal, has an IOS and Android app for beginner pythonistas
	- From “Hello World” to Generators and Decorators
	- Original Owners of Programiz: Ranjit Bhatta, Aswin Shrestha and Punit Jajodia
	- Swipe, Learn, and Repeat
    - Has an editor built in
    - Leaderboards
    - Quizzes and “Interview Challenges”  (paid)
        - Newest update with 200+ programs
    - Interactive IDE
    - FREE, with minimal app advertising
    - Paid version as well
	- Learn anywhere availability, helps to reduce the Digital Divide
    - 826 million learners online due to Covid19
    - 50% of those learners do not have a household computer
    - 706 million have no at home internet
	- The engineering behind the app and its editor function: [https://www.programiz.com/blog/online-python-compiler-engineering/](https://www.programiz.com/blog/online-python-compiler-engineering/)

**Brian #4:** [**New packaging security funding & NYU**](https://discuss.python.org/t/new-packaging-security-funding-nyu/7792)

- Sumana Harihareswara
- “New York University – specifically Professor Justin Cappos – and I (Sumana) have successfully asked the US National Science Foundation for a grant to improve Python packaging security. The NSF is awarding NYU $800,000 over two years – from mid-2021 to mid-2023 – to further improve the pip dependency resolver and to integrate The Update Framework further into the packaging toolchain.”
- “NYU researchers and developers will
	- Further assess and improve pip’s dependency resolver, following up on the work done in 2020 and making [ResolveLib](https://github.com/sarugaku/resolvelib/) more reusable by other tools in the packaging ecology
	- Secure the PyPI-to-user pipeline by integrating TUF support for signed packages throughout PyPI’s clients (we’re targeting conda, pip, and bandersnatch initially)”
- TUF is [The Update Framework (TUF) specification,](https://www.theupdateframework.com/)
	- From [theupdateframework.io](https://theupdateframework.io/):
		- “The Update Framework (**TUF**) helps developers maintain the security of software update systems, providing protection even against attackers that compromise the repository or signing keys. TUF provides a flexible framework and specification that developers can adopt into any software update system.
- [ResolveLib](https://github.com/sarugaku/resolvelib/)  “provides a `Resolver` class that includes dependency resolution logic.”. It’s an independent project, but vendored by pip
- [bandersnatch](https://pypi.org/project/bandersnatch/) “is a PyPI mirror client”

**Michael #5: Extra x8, hear all about it**

1. We are on [**amazon**](https://music.amazon.com/podcasts/7dc269ab-93df-4a32-b382-9e3e8c41d016/Python-Bytes) and [**audible**](https://www.audible.com/pd/Python-Bytes-Podcast/B08JJRD7D9?qid=1616362521&sr=1-3&ref=a_search_c3_lProduct_1_3&pf_rd_p=83218cca-c308-412f-bfcf-90198b687a2f&pf_rd_r=HCG8XTQVENWMJWPNVSH2) **(so is** [**Teaching Python**](https://music.amazon.com/podcasts/search/python)**)**
2. Sourcery added [**skip refactorings**](https://github.com/sourcery-ai/sourcery/wiki/Skipping-refactorings). I did major refactoring since, favorite:
	1. `url = request.url if request.url else ''` to
	2. `url = request.url or '``'`
3. Via [**Matthew Feickert**](https://twitter.com/HEPfeickert/status/1372425321203568642), it’s easy to become a PSF member and support the PSF
4. [**Beanie adds indexes**](https://roman-right.github.io/beanie/#indexes)!
5. Pycharm (and other JetBrains IDEs) in the browser via Dave Kirby
	1. [**lp.jetbrains.com/projector**](https://lp.jetbrains.com/projector/)
	2. [**github.com/JetBrains/projector-installer**](https://github.com/JetBrains/projector-installer)
6. [**SQLite as a file format comes to Audacity**](https://www.theregister.com/2021/03/19/audacity_3/) (via Jon Bultmeyer)
7. **Prayson Daniel** shared [**his Neo4J examples**](https://github.com/Proteusiq/graphs) using Neomodel.
8. [**Call for proposals for PyCon Latam just went live**](https://twitter.com/ChekosWH/status/1371992445093744641)

**Sean #6:** [**Using Development Containers with VS Code for Students**](https://code.visualstudio.com/blogs/2020/07/27/containers-edu)
* One of the big chores at the start of a course is getting everyone set up in the same development environment. Usually takes at least a few classes and nearly impossible to get everyone with the same environment at the end.
* **Solution -** use Docker images and the Remote-Containers extension to allow students to run a pre-configured container right in VS Code.
* Needs VS Code & Docker Desktop (maybe a remote Docker host could make it even easier?)
* Can deploy directly from a Github repo with persistent storage volumes, default VS Code settings, and even pre-installed packages and environment variables.
* Seems well suited for universities, experienced learners, etc. Also makes it easy to create different environments by assignment or course unit. What if every Talk Python training course had a GitHub container repo with the same starting environment?

Extras:

**Kelly**:
- [https://us.pycon.org/2021/summits/education-training/](https://us.pycon.org/2021/summits/education-training/)

**Sean:**

- [Repl.it adds Python package caching](https://blog.replit.com/python-package-cache) (and gets a 40% speed boost)
- [Home Assistant (actually Nabu Casa) buys ESPHome](https://www.home-assistant.io/blog/2021/03/18/nabu-casa-has-acquired-esphome/) (makes DIY IoT devices cheap & easy to integrate)
- [It’s good to know a Python programmer if you need a vaccine appointment](https://www.nbcnews.com/tech/security/want-vaccination-appointment-helps-know-python-programmer-rcna457)

Joke (in honor of our teachers today):

![](https://trello-attachments.s3.amazonaws.com/6041d3db4ddead109a997194/912x670/888520d608537dd14fdd724eb01fe754/Screen_Shot_2021-03-04_at_10.34.37_PM.png)

gha_language: Python


text:
# Distributed Patricia Trie - DPT

[![Build Status](https://travis-ci.org/kurpicz/dpt.svg?branch=master)](https://travis-ci.org/kurpicz/dpt)

## What is it?
The Distributed Patricia Trie (DPT) is a distributed full-text index implemented in C++.
A detailed description can be found [here](https://doi.org/10.4230/LIPIcs.CPM.2016.26) ([arXiv preprint](https://arxiv.org/abs/1610.03332)).

    @inproceedings{DBLP:conf/alenex/0001K017,
        author    = {Johannes Fischer and
                     Florian Kurpicz and
                     Peter Sanders},
        title     = {Engineering a Distributed Full-Text Index},
        booktitle = {Proceedings of the Ninteenth Workshop on Algorithm Engineering and
               Experiments, {ALENEX}},
        pages     = {120--134},
        year      = {2017},
        doi       = {10.1137/1.9781611974768.10},
    }

## How to get it?
First clone this repository, then build all executables.
```sh
git clone https://github.com/kurpicz/dpt.git
cd dpt
git submodule update --init
mkdir build
cd build
cmake ..
make
```

## Dependencies
- [SDSL](https://github.com/simongog/sdsl-lite): Giving us access to a variety of succinct data structures.

When cloning this project, you will also clone [googletest](https://github.com/google/googletest).

repository: patflick/dpt

repo_name: patflick/dpt


text:
core
====

Core services for Nimblestratus.

The idea is to provide a scalable set of core services which are HA and resilient.

The identified services include:

* etcd discovery service
* DNS
* Docker Registry
* Node Management
* Fleet Cluster Management
* Routing & Service Discovery
* Metrics & Logs Gathering & Aggregation

## Background ##

In order to resolve the chicken and egg problem of services needed to
bootstrap a PAAS cluster, some services need to be "pets" —
handcrafted. However, they need to be recreatable on demand; whether
due to expansion or a failure the services need to be well defined and
maintainable.

It is assumed that the services will be HA in order to prevent
a service failure from causing an entire cluster to fail.

## Philosophy and Motivators ##

In general and as much as possible, the driving motivator is to have a
basis upon which to build a stable, fault tolerant platform.  To this
end, the following design criteria are assumed:

1. Simple
2. Reuse
3. Well defined
4. Repeatable
5. Highly Available — there is no one copy of a service.

In order to support reuse, etcd is leveraged as a backing service
throughout the design.

## Services ##

### Etcd Discovery Service###

[Etcd](https://coreos.com/using-coreos/etcd/) is an open-source
distributed key-value store used by a number of
technologies including:

* [CoreOS](http://coreos.com)
* [Fleet](https://github.com/coreos/fleet) — a distributed init
  system

CoreOS provides a global cluster discovery service at
[https://discovery.etcd.io/new](https://discovery.etcd.io/new) which
returns something like:
[https://discovery.etcd.io/6c1f17ed3e6c866406857f90690a3d43](https://discovery.etcd.io/6c1f17ed3e6c866406857f90690a3d43).
This returns a json formatted object which lists the members of the
cluster.  An empty cluster looks like:

`{"action":"get","node":{"key":"/_etcd/registry/6c1f17ed3e6c866406857f90690a3d43","dir":true,"modifiedIndex":86931009,"createdIndex":86931009}}`

In order to simplify the use of the etcd discovery service and to
remove dependencies on outside links, a discovery service will be run
at a canonical name such as `http://discovery.local` which
mimics the behaviour of the global service.  The source for the global
repository is at
[https://github.com/coreos/discovery.etcd.io](https://github.com/coreos/discovery.etcd.io);
it has `https://discovery.etcd.io` hard coded in its source.  In order
to make it useful and be more [12 Factor](http://12factor.net)
compliant, the code should be modified to take an argument for the host and preferably read from the environment.

Additionally, DNS should maintain service records for the core etcd
servers and discovery servers.

### DNS ###

skydns is built upon etcd.

It needs to be started in "discovery" mode.

### Docker Registry ###

* Using a local registry eliminates the need to connect to outside
  resources for starting docker containers.  Additionally it allows
  for the use of flattened containers — a container in which all of
  the layers have been flattened and consequently it is
  deterministic.  Version 1.0 of the Gnomemobile container contains
  the entire image of Gnomemobile and its dependencies.  This
  alleviates potential issues with dependencies on the "latest"
  version of container.
* Runs on a known DNS location such as `registry.DOMAIN`.
  This assumes that the DNS search defaults to `DOMAIN`
* Exposes port 80.  This simplifies matters — instead of being on the
  "default" port of 5000 whereby containers are referenced by
  `registry:5000/redis:latest` the `:5000` can be left off, simplifying matters.
* Backed by [S3](http://en.wikipedia.org/wiki/Amazon_S3) instead of
  local storage.  If a server goes down, then
  the images will still be available.  The S3 storage can be provided
  by any service which understands the S3 protocol, such as [riak](http://basho.com/riak/). In particular riak-cs needs to be running. The [hectcastro/riak-cs](https://registry.hub.docker.com/u/hectcastro/riak-cs/) container works well. However, the docker daemon needs to be started with `-H tcp://<IP_ADDRESS>:4243 -H unix:///var/run/docker.sock` for the container to work as the makefile advertises.

### Fleet Cluster Management ###

* flit

### Routing ###

(eg haproxy)

### Service Discovery ###

(skydns supports this via service records)

## Core Servers OS ##

CoreOS will be used for the core services OS.  Services will either be
controlled by systemd or docker containers spawned by fleet.

Host metrics will be collected by metrix.

## Bootstrap Order ##

0. Have S3 store available for docker registry.
1. Bring up first etcd instance telling it that it is a "master" and
   set for auto-discovery. (cloud-config)
2. Bring up other etcd instances, telling them about the master and
   set them for auto-discovery. (cloud-config)
3. Bring up first docker registry instance on etcd master (it is a
   docker container).  It will need to be bootstrapped as it is a
   docker container and the registry does not yet exist.  Save its
   details to put into DNS.  This can be accomplished via cloud-config.
4. Use fleet to bring up DNS & register etcd instances in DNS.
5. Bring up additional docker registries, bootstrapped off of the
   initial registry instance.
6. Bring up metrics collector & register in DNS
7. Start metrics service on the core hosts
8. Use fleet to bring up Etcd Discovery service instances &
   register in DNS.
9. Fleet cluster management tool

repository: nimblestratus/core

repo_name: nimblestratus/core


text:
# Themes

The application comes pre-packaged with a couple of built-in themes. Users can add their own custom themes on the "Settings" page. For that, a user should supply either a URL of a `.css` file or just paste [CSS](https://developer.mozilla.org/docs/Learn/CSS/Introduction_to_CSS/How_CSS_works) code.

Themes should mostly use the available [CSS Variables](https://github.com/catamphetamine/captchan/blob/master/docs/themes/variables.md). If a customization can't be achieved using the available CSS Variables consider contacting the repo author through [issues](https://github.com/catamphetamine/captchan/issues) for a discussion of adding new CSS Variables, or just use CSS selectors directly instead.

Each theme should provide both "Light" mode and "Dark" mode.

See the [built-in themes](https://github.com/catamphetamine/captchan/tree/master/src/styles/theme) as an example.

### Default theme

#### Light

[View in full resolution](https://raw.githubusercontent.com/catamphetamine/captchan/master/docs/images/default-theme-light-mode-3605x1955.png)

#### Dark

[View in full resolution](https://raw.githubusercontent.com/catamphetamine/captchan/master/docs/images/default-theme-dark-mode-3605x1955.png)

[style-variables.css](https://github.com/catamphetamine/captchan/blob/master/src/styles/style-variables.css)

### Neon Genesis Evangelion

#### Light

[View in full resolution](https://raw.githubusercontent.com/catamphetamine/captchan/master/docs/images/eva-theme-light-mode-3605x1955.png)

#### Dark

[View in full resolution](https://raw.githubusercontent.com/catamphetamine/captchan/master/docs/images/eva-theme-dark-mode-3605x1955.png)

[neon-genesis-evangelion.css](https://github.com/catamphetamine/captchan/blob/master/src/styles/theme/neon-genesis-evangelion.css)

repository: forklifters/captchan

repo_name: forklifters/captchan


text:
# lightcouch
Library to access CouchDB built on LightSpeed and JsonRPCServer (uses httpclient)

## compile

To compile the library, you have to put **lightspeed** and **jsonrpcserver** into the same directory. Then you can perform **make runtests** on it

```
/project_root
  + -- lightspeed
  + -- jsonrpcserver
  + -- lightcouch
```

If you need to use the library in your project, put it along with other libraries into the libs folder

```
/project
 + -- libs
 |     + -- lightspeed
 |     + -- jsonrpcserver
 |     + -- lightcouch
 ```

repository: ondra-novak/lightcouch

repo_name: ondra-novak/lightcouch


text:
Name: Sanjana Raj; Datadog: Solutions Engineer-San Francisco- Hiring Challenge.
I used Vagrant on my Mac OS, as I felt using a VM is much better for resolving bugs and it is faster.

Prerequisites - Setup the environment
1.	I downloaded Vagrant from the official website, along with the VirtualBox by Oracle for hypervisor.
2.	Then, instead of using just vagrant init, I found it better and more efficient to install it with the required OS (in this case-Ubuntu 16.04).
3.	I used ‘vagrant init bento/ubuntu-16.04’ on the terminal to start my vagrant.
4.	Then I used the command ‘vagrant up’ which installed the vagrant file in my local directory. I then ssh’ed into the VM using ‘vagrant ssh’.

  ![image](https://user-images.githubusercontent.com/32622982/53146155-ff1c9180-3557-11e9-8898-7a3f97a5afb6.png)

5.	I signed up for Datadog as recruiting candidate with my gmail address and the required fields.
6.	As a next step, I installed the datadog-agent for Ubuntu by following the steps in the UI. I used the command given by datadog in my terminal.

  ![image](https://user-images.githubusercontent.com/32622982/53146105-d1374d00-3557-11e9-914c-4bc473ae03a1.png)

7.	The above screenshot shows that the agent was successfully installed and running.
8.	To confirm if the agent was running, I used ‘sudo service datadog-agent status’ command

  ![image](https://user-images.githubusercontent.com/32622982/53146062-a947e980-3557-11e9-8e6a-345744ee0cfc.png)

Collecting Metrics:
1.	Adding tags to the agent makes it easier to monitor an agent when the server and infrastructure become more complex. Ex: region, role, environment, name, field, etc. usually in the key-value pair.
2.	I added tags by navigating to the directory which has the datadog-yaml file. It is in /etc/datadog-agent/datadog.yaml.

  ![image](https://user-images.githubusercontent.com/32622982/53146208-30955d00-3558-11e9-8434-fe20662ed81b.png)

3.	Any editor can be used to change the tags in the file, I used the nano editor.
4.	In the file, I looked for the tags section, uncommented it and added ‘region: west’ tag.

  ![image](https://user-images.githubusercontent.com/32622982/53146247-5c184780-3558-11e9-9b9b-8aa9f1660a80.png)

5.	I viewed my host on the datadog UI as soon as it started reporting along with the tags.

  ![image](https://user-images.githubusercontent.com/32622982/53146315-9b469880-3558-11e9-8c25-19677a486b0b.png)

6.	I navigated to the Host Map page to view my host and also see the tags that I added earlier. The results can also be filtered out based on the tags. Here, I have used ‘region – west’ tag.

  ![image](https://user-images.githubusercontent.com/32622982/53147000-0c874b00-355b-11e9-93e3-a477b846136f.png)

7.	I Installed PostgreSQL on my vagrant machine from the official website

  ![image](https://user-images.githubusercontent.com/32622982/53147029-21fc7500-355b-11e9-85ab-46d0e7aeb5b5.png)

8.	Then, I installed and configured the appropriate PostgreSQL integration in the datadog UI, the screenshot below elaborates the configuration.
9.	I created a postgres.yaml file in the datadog-agent/conf.d/postgres.d directory to configure the integration with PostgreSQL on my local machine.
10.	I restarted the agent from terminal and verified that the file exists in checks.d directory in etc/datadog-agent.

  ![image](https://user-images.githubusercontent.com/32622982/53147186-9d5e2680-355b-11e9-8caf-2eb244b16f99.png)

Custom Agent Check.

1.	To create a custom agent check, I created two files in two different directories. The first file I created is a yaml file for configuration in etc/datadog-agent/conf.d directory.
2.	I named the file mymetric.yaml in which I wrote the following script.

  ![image](https://user-images.githubusercontent.com/32622982/53147068-47897e80-355b-11e9-861b-4ed898efdb04.png)

3.	The second file is a python script in the etc/datadog-agent/checks.d directory. The python script and the yaml file should always have the same filename.
4.	In mymetric.py file, I wrote a simple function to create a metric called hello world and used a rand() function to generate a value between 0 and 1000.
        *The mymetric.py script is attached in the scripts folder*
5.	To change the interval, I modified the mymetric.yaml file by changing the instances object to 45 seconds. The above screenshot shows the changes to the interval.

Visualizing Data:
1.	Datadog has its own API to create features in any language of your choice. I used the API in python to create a Timeboard which will display 3 different graphs.

     *The custom_timeboard.py script is attached in the scripts folder*

2.	The script has three functions:
•	To show the metrics over the custom-metric I created in the previous step called ‘hello.world’.
•	To show the anomaly for the postgres-bgwriter-write-time metric from the PostgreSQL integration installed earlier.
•	To show the metrics for the rollup to sum all points.

  ![image](https://user-images.githubusercontent.com/32622982/53147294-fcbc3680-355b-11e9-95d1-67aca3af1216.png)

3.	The timeboard generated was displayed on the Dashboard list menu in the datadog UI.

  ![image](https://user-images.githubusercontent.com/32622982/53147330-21b0a980-355c-11e9-9984-7f41cdb482ec.png)

What is the anomaly graph displaying?
Answer: When the PSQL makes commits, the graph shows the activity.

Monitoring Data
1.	To avoid monitoring the data constantly, I set up alerts for warning threshold, alerting threshold and also for alerts for no data reported.
2.	I used the monitor menu in the UI and set up alerts and wrote custom alert messages to get notifications on my email address when the alerts are triggered.

  ![image](https://user-images.githubusercontent.com/32622982/53147364-3e4ce180-355c-11e9-931f-26d807eb1fd6.png)

  ![image](https://user-images.githubusercontent.com/32622982/53147397-56246580-355c-11e9-84b6-80c91af50bb7.png)

3.	The screenshot below shows the alert notification.

  ![image](https://user-images.githubusercontent.com/32622982/53147468-81a75000-355c-11e9-8f99-389e97c98b3c.png)

  ![image](https://user-images.githubusercontent.com/32622982/53147512-9edc1e80-355c-11e9-84e3-aa013730da24.png)

Bonus Question: Scheduled downtime for after work-hours from 7 PM to 9PM M-F.

  ![image](https://user-images.githubusercontent.com/32622982/53147559-bd421a00-355c-11e9-9746-0a415cea7771.png)

Scheduled downtime for weekends all day - Saturday and Sunday.

  ![image](https://user-images.githubusercontent.com/32622982/53147580-cb903600-355c-11e9-93be-0d598c48ac33.png)

Collecting APM Data:
1.	Datadog lets you use APM data to monitor your application and collect metrics.
2.	I used the given Flask application in a new file called app.py in a new datadog directory.
3.	I installed Flask on my machine using ‘pip install flask’. To run the flask app, I could either use virtualenv or ‘flask run’ with the host ip or port number.
4.	The mandatory step before running the application is to set an environment variable to the config file for which I used ‘export FLASK_APP=config.cfg” to set my env variable.
5.	To collect the APM data, I modified my app.py file to install the datadog tracer to trace my functions.
6.	I could either use ddtrace-run or tracer middleware. I tried both but I faced challenges in ddtrace-run as no metrics were reported back.
7.	I tried troubleshooting by reinstalling and modifying my program and verified for any bugs in the app but there were no metrics reported.
8.	Then, I chose to use another custom tracer which is the middleware. The tracer started running on the host, reporting APM services and collecting metrics.

  ![image](https://user-images.githubusercontent.com/32622982/53147657-0d20e100-355d-11e9-981e-66cca5132693.png)

9.	The application’s infrastructure metrics are shown in the graph and below is the screenshot

  ![image](https://user-images.githubusercontent.com/32622982/53147689-29bd1900-355d-11e9-9c1b-e6fb9e56ec09.png)

Question: What is the difference between a Service and a Resource?
Answer: A service can be defined as a function that does a task. It can be a unit of design and implementation. It can be a database, interceptor, API endpoint etc. A resource describes the implementation or action of a service like a query to a database. We can think of service as a verb and resource as a noun.
Final Question:
Is there anything creative you would use Datadog for?

1.	Stock Market Prediction - Based on daily highs and lows of the stock price.
2.	Personal Health Tracking
                - Sleep Monitoring and PSQI Sleep score computation.
                - Weight loss and gain based on calorie intake and diet.
3.	Traffic Monitoring based on your saved routes on freeways – Can monitor traffic on your routes and alert you/notify you about the optimal time to avoid traffic.

Bonus application – Minitwit

I wanted to try collecting metrics via APM with a custom application I wrote in python flask. It is a clone of Twitter and it uses sqlite3 as the database.

Challenges faced during APM for this custom application – I could install the required dependencies and run the application perfectly. I used ddtrace for this and it was showing one service reporting and it was collecting metrics. The only problem was with the UI, when I navigate to APM services, there was no metrics shown. My application was sending data, but it was not shown in the UI.

I have attached a few screenshots to show a glimpse of my application, while I ran it locally on port 5000 from my Mac.

Log in Page.

  ![image](https://user-images.githubusercontent.com/32622982/53147793-802a5780-355d-11e9-8bc0-4359affac270.png)

Sending a tweet and glimpse of your timeline.

  ![image](https://user-images.githubusercontent.com/32622982/53147821-933d2780-355d-11e9-8ba1-ac5bbd8ccc39.png)

Public Timeline- shows tweets from all followers.

  ![image](https://user-images.githubusercontent.com/32622982/53147858-abad4200-355d-11e9-9dbb-586842d73e37.png)


text:
# Esther 09 General Notes

## Special concepts in this chapter

### Purim

The events of this chapter were so significant, the Jews celebrated these events every year after this. It is known as “Purim.”

## Important stylistic devices in this chapter

### Ironic Situation

The day that was supposed to bring great victory to the enemies of the Jews became a day of great victory for the Jews. This is a type of irony.

### Opening Summary

Verse 1 of chapter 9 provides a summary of everything that happens in this chapter. If using an opening summary is not normally used in your language, be sure to mark this as a summary either in the text or in a footnote. An opening summary like this is a characteristic device of Hebrew storytelling. So even though it describes the final outcome, we do not recommend that you move it to the end of the chapter unless it is very confusing to leave it here. To make it clear that verse 1 is only a summary and not the entire story, you could say something like this at the end of the verse: “This is what happened” or “Here are more details about what happened.”

repository: mandolyte/uw-obsidian

branch_name: refs/heads/main

repo_name: mandolyte/uw-obsidian


text:
{% extends "/docs.md" %}
{% block content %}
# StopPoint/Departures
The following [Straming](https://en.wikipedia.org/wiki/WebSocket) API channels offer access to live data of train and bus departures flowing through Transport for Luxembourg in real time.
Currently we're providing data from the [Verkéiersverbond](https://data.public.lu/en/organizations/mobiliteitszentral/) only.

## License
The data from [Verkéiersverbond](https://data.public.lu/en/organizations/mobiliteitszentral/) is licensed under [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

## Available channels
- [/StopPoint/Departures](/Streaming_APIs/StopPoint/Departures/index.md)
- [/StopPoint/Departures/{id}](/Streaming_APIs/StopPoint/Departures/id.md)
{% endblock %}

repository: TfLlu/TfL-API

repo_name: TfLlu/TfL-API


text:
# Naming conventions for classes, methods and variables

## Motivation

The purpose of this ADR is to standardize the naming convention through the project to avoid inconsistencies.

## Decision

- Prefix abstract classes with `Abstract`
- Suffix interfaces with `Interface`
- Suffix traits with `Trait`
- Suffix exceptions with `Exception`
- Use `PascalCase` for class names
- Use `camelCase` for PHP variables, function and method names, arguments

repository: phel-lang/phel-lang

gha_language: PHP

repo_name: phel-lang/phel-lang