text:
BAP Tutorial
============

## Introduction

In this tutorial we will develop a non-trivial plugin that will verify
that in a program a certain sequence of calls doesn't happen. We will
develop the analysis in both OCaml and Python. You can choose either
path, or even both paths.

## Takeaways

By passing this tutorial you will learn how to use BAP basic
capabilities and how to extend BAP using our plugin system. You will
learn how to examine programs, by looking into their intermediate
representations (IR) or disassembly. You will also learn how to run
binaries in the emulated environment.

### OCaml Path Takeaways

1. how to build and install plugins
2. how to use command line arguments in a plugin
3. how to work with IR and different graph representations.

### Python Path Takeaways

1. how to interact with bap from the Python universe
2. how to use IR visitors
3. how to create graphs

## Preparation

To complete the OCaml path of the tutorial you will need an OCaml
development environment, bap, and all the necessary libraries. We have
prepared a Vagrantfile, that will setup your virtual machine using
Vagrant. If you have `vagrant` installed on your machine the following
will setup and prepare the necessary development environment.

```
wget http://tiny.cc/Vagrantfile
vagrant up
vagrant ssh
```

The last command will ssh you into the installed virtual machine, that
has BAP and all the necessary libraries readily available. It also
have Emacs preinstalled and configured to work with OCaml.

If you're going to take only the Python part of the tutorial and do
not want to install any virtual machines, then you can use BAP's
binary distribution. If you're using some Debian derivative, then the
following will install BAP on your system.

```
wget https://github.com/BinaryAnalysisPlatform/bap/releases/download/v2.0.0/{bap,libbap,libbap-dev}_2.0.0.deb
sudo dpkg -i {bap,libbap,libbap-dev}_2.0.0.deb
sudo install pip
sudo pip install bap networkx
```

## First Steps

For OCaml, create a new folder somewhere in your home folder, and
enter into it, e.g.,

```
mkdir check_path
cd check_path
```

Then use Emacs to create an empty file

```
emacs check_path.ml
```

Note: it is mandatory to develop each BAP plugin in a separate
folder.

After Emacs has started, use `M-x merlin-use <RET> bap` command to tell
[Merlin][1] that you're going to use the `bap` library. See [our wiki][2] for more
instructions about Merlin.

If you're developing in Python then do the same but create a file with
the `'.py` extension, e.g.,

```
emacs check_path.py
```

[1]: https://github.com/ocaml/merlin
[2]: https://github.com/BinaryAnalysisPlatform/bap/wiki/Emacs#configuring-merlin

## First steps with bap

We will need some binary to play with. For the purpose of this
tutorial, we created a special binary that is small, but still
non-trivial. You can get it using `wget`

```
wget tiny.cc/bap-echo -O exe
```

If you're running on an x86 machine you can run the binary by

```
chmod a+x exe
./exe
```

The binary will parse and print some verses from Shakespeare. It will
also echo all passed arguments.

You can also run the binary using `bap`. The binary will not be
run on an actual CPU, but emulated using the Primus Framework.

To output the executable in the default assembler use `-dasm` option, e.g.,

```
bap ./exe -dasm
```

ProTip: if you prefer the intel syntax use `--llvm-x86-syntax=intel`.

To get the IR just use the `-d` option, e.g.,

```
bap ./exe -d
```

You can also use `-dcfg` to get the control-flow graphs, and
`-dcallgraph` to dump the program call graph. To get the full list of
supported formats for the `project` data structure use the
`bap list formats -f project` command, e.g.,

```
$ bap list formats -f project
  Bap.Std.Project:
    symbols (1.0.0)        print symbol table
    ogre (1.0.0)           print the file specification in the OGRE format
    marshal (2.0.0)        OCaml standard marshaling format
    knowledge (1.0.0)      dumps the knowledge base
    graph (1.0.0)          print unlabeled CFG for each procedure
    cfg (1.0.0)            print rich CFG for each procedure
    callgraph (1.0.0)      print program callgraph in DOT format
    bir (1.0.0)            print program in IR
    bil.sexp (1.0.0)       print BIL instructions in Sexp format
    bil.adt (1.0.0)        print BIL instructions in ADT format
    bil (1.0.0)            print BIL instructions
    asm.sexp (1.0.0)       print assembly instructions as it was decoded
    asm.decoded (1.0.0)    print assembly instructions as it was decoded
    asm.adt (1.0.0)        print assembly instruction endcoded in ADT format
    asm (1.0.0)            print assembly instructions
    adt (1.0.0)            print program IR in ADT format

```

For more options see `bap --help` or `man bap`. To get information about a plugin
named `<PLUGIN` use the `bap --<PLUGIN>-help>` option. e.g., for the `optimization`
plugin, `bap --optimization-help`.

ProTip: if you have a GUI setup, then you can use `xdot` for interactive
visualization of control flow graphs:
```sh
bap exe -dcfg --print-symbol=main | xdot -
```

## Theoretical Background

The purpose of our analysis is to verify that for all execution paths
in a program a call to the function `f` is not followed by a call to
the function `g`, where `f` and `g` are specified by a user. We can
express this property concisely using Linear Temporal Logic:

    f -> G not(g)

where `f` and `g` are propositions that are true if a call to a
the corresponding function occurred, and `G x` is the LTL operator that
requires the predicate `x` to be true on the entire subsequent path.

We will write a policy checker, that will verify that this property
holds for all execution paths, or will provide a counter-example,
proving the opposite. The checker will be static and sound (wrt to the
soundness of the CFG).

Now, let's figure out how we will prove this property:

1. If there are no calls to `f` or to `g` or to both, then the
   verification property is trivially true, i.e., `not f \/ not g`
   This property has a strong "must" modality. Since the call indeed can't happen,
   so the property `must` hold for all paths.

2. If `f` calls `g`, either directly or by calling a function that
   calls `g`, then `f` occurs before `g`. The call to `g` still may not occur,
   as we ignore path constraints, so this gives us the may modality.
   We will express this as `calls(f,g)`.

3. If there is a subroutine that has two calls that lead to `f` and `g`
   correspondingly, and these calls are reachable in the control flow graph
   of the subroutine, then `g` may be called after `f` with the may modality.
   (Again, we do not consider the feasibility of the path constraint).
   More formally, this property can be expressed as:

        sites(p,q) := calls(p,f) /\ calls(q,g) /\ reaches(p,q)

   where `reaches(x,y)` is true if there exists a control flow graph with
   a path between `x` and `y`.

To summarize, in order to prove that `f -> G not(g)` holds for the static model of our program,
we need to prove that the following doesn't hold

    f /\ g \/ calls(f,g) \/ ∃p∃q, sites(p,q)

## OCaml implementation

### Boilerplate Code

We will start with several `open` and `include` statements that will
prepare our development context. Add the following to the top of the
`check_path.ml` file.

```ocaml
open Core_kernel
open Bap.Std
open Graphlib.Std
open Format
include Self()
```

After you saved the file with `C-x C-s`, the `(No errors)` message
should appear in the mini-buffer. If it's not true, then make sure
that you have set up Merlin and didn't make any errors in your code.

### Defining the Command Line Interface

We will create a `Cmdline` module with the defintion of our command line
interface. It is not required to put the code in a separate
module, but it looks cleaner and makes it easier for a reader to
identify that this particular section of code deals with the command
line interface

```ocaml
let main src dst proj = ()

module Cmdline = struct
  open Config
  let src = param string "src" ~doc:"Name of the source function"
  let dst = param string "dst" ~doc:"Name of the sink function"

  let () = when_ready (fun {get=(!!)} ->
      Project.register_pass' (main !!src !!dst))

  let () = manpage [
      `S "DESCRIPTION";
      `P
        "Checks whether there is an execution path that contains a call
    to the $(v,src) function before a call to the $(v,dst)
    parameter. Outputs a possible (shortest) path that can
    lead to the policy violation."
    ]
end
```

The code above will specify that our plugin has two command line parameters,
one is named `src` and another `dst`, both having type
`string`. It will also register the `main` function as a pass in the
analysis framework. The following stanza probably needs some
explanations:

```ocaml
  let () = when_ready (fun {get=(!!)} ->
      Project.register_pass' (main !!src !!dst))
```

The `when_ready` function takes one argument, that is a function that
will be called as soon as system configuration is processed and
command line arguments are parsed. The function will take a record,
that has only one field `get`, that is a function on itself, that will
take a value of type `'a param` and will extract a value of type '`a`
from it. The `{get=(!!)}` syntax, binds the `get` function to the
`(!!)` unary operator that can be used nicely to extract values from
parameters.

To summarize, a user specifies the command line interface, but using
the `param` function. Then the `Config.when_ready` function is used to
get the `get` function, that is a key that will unlock the parameters,
and extract the values, that are written there by the command-line
parser. The interface is described thoroughly in the reference
[documentation][3].

[3]: http://binaryanalysisplatform.github.io/bap/api/master/Bap.Std.Self.Config.html

### Building, Installing, and Running

The following three commands will build, install, and run our
analysis:

```
bapbuild check_path.plugin
bapbundle install check_path.plugin
bap ./exe --check-path-src=malloc --check-path-dst=free --pass=check-path
```

The `bapbuild check_path.plugin` command will compile the `check_path.plugin`
from the `check_path.ml`. The `bapbundle install check_path.plugin`
command will install the plugin into the place, where BAP will
automatically load (but not run) it.

The last command will run `bap` on a binary and instruct it to process
the binary with a pass named `check-path`. By default, a pass name has
the same name as the plugin name, except that all underscores are
substituted with dashes.

The `--check-path-src=malloc` will pass `malloc` as an argument to the
`src` parameter. Note, that in order to pass an argument to a plugin
parameter you need to prefix the parameter with the plugin name.

When developing, it is convenient to run all three commands at once,
with one keystroke. Fortunately, Emacs allows us to do this. To
compile for the first time, type

```
M-x compile <RET> bapbuild check_path.plugin && bapbundle install check_path.plugin && bap ./exe --check-path-src=malloc --check-path-dst=free --pass=check-path
```

This will run all three commands. You can also create a Makefile, or
store this command in some build script.

Whenever you need to recompile, just hit `C-c C-c` and Emacs will
recompile using the last compilation command.

See the [Emacs manual][4] for more information about compilation.

[4]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Compilation.html#Compilation

### Trivial Proof

We will first check the trivial proof, i.e., when one of the functions
or both are not present. For that we will write a function, that will
translate a name of a function, as specified by a user, to the term
identifier of this function.

```ocaml
let find_sub prog name =
  Term.enum sub_t prog |>
  Seq.find_map ~f:(fun s ->
      Option.some_if (Sub.name s = name) (Term.tid s))
```

Now we can extend the `main` function:

```ocaml
let main src dst proj =
  let prog = Project.program proj in
  match Option.both (find_sub prog src) (find_sub prog dst) with
  | None -> printf "satisfied (trivially)\n"
  | Some (src,dst) -> printf "not implemented\n"
```

### The Skeleton Of Our Proof

We will now start writing a `verify` function, that will try to prove
that the relation doesn't hold by providing a concrete
counter-example. The `verify` function will take the program in the
Intermediate Representation (IR) and two term identifiers, one for the source function
and another for the destination function.

The function will try to find a path in the program call graph, that leads from
the source function to the destination function. If there is no such
path then for each control flow graph we will try to find a path
between a callsite that reaches the source function and a
callsite that reaches the destination function.

We will work with two different graph representations, one for the
call graph, and another for the control flow graph, let's introduce
the following abbreviations:

```ocaml
module CG = Graphs.Callgraph
module CFG = Graphs.Tid
```

Our `verify` function return type will be `proof option`, i.e., it
will return `Some proof` if the proof was found, otherwise an absence of
a proof is the proof that the relation will hold, unless there is a not yet
discovered indirect call.

Let's define the `proof` type as a variant type with two branches that
correspond to the two cases in our proof strategy:

```ocaml
type proof =
  | Calls of CG.edge path
  | Sites of CFG.edge path
```

We will use the `path` data structure from the Graphlib library, as a
particular representation of the proof term. Since we would like to output
these paths, we will start with the implementation of two pretty-printing functions for both
variants. Since they share the same polymorphic data
structure, we will first write a polymorphic function that will print
all types of paths, and then we will specialize this function to our
two concrete types:

```ocaml
let pp_path get_src get_dst ppf path =
  let print_node n = fprintf ppf "%s" (Tid.name n) in
  print_node (get_src (Path.start path));
  Seq.iter (Path.edges path) ~f:(fun e ->
      pp_print_string ppf " -> ";
      print_node (get_dst e))

let pp_calls = pp_path CG.Edge.src CG.Edge.dst
let pp_sites = pp_path CFG.Edge.src CFG.Edge.dst
```

The `pp_path` function is parametrized with the `get_src` and
`get_dst` functions that will extract, correspondingly, the source node and
the destination node of an edge. Then it will print the source of the first
edge, and iterate over all all edges and print their
destinations. Recall, that an edge consists of two endpoints and a
label (we will ignore the label for now). Thus a path is a
sequence of `m` edges, connecting `m+1` nodes, e.g.,

    (s, n0); (n0, n1); ...; (sm, d)
      /\       /\              /\
      ||       ||              ||
      e0       e1              em

Since, we want our output to be concise, we will print only `m+1`
nodes, i.e., in our case `s,n0,n1,...,d`.

Now, let's update the `main` function:

```ocaml
let main src dst proj =
  let prog = Project.program proj in
  match Option.both (find_sub prog src) (find_sub prog dst) with
  | None -> printf "satisfied (trivially)@\n"
  | Some (src,dst) -> match verify src dst prog with
    | None -> printf "satisfied (no counter-example was found)@\n"
    | Some p ->
      printf "unsatisfied by ";
      match p with
      | Calls p -> printf "calls via %a@\n" pp_calls p
      | Sites p -> printf "callsites via %a@\n" pp_sites p
```

We will also provide a stub for the `verify` function, so that we can
compile and run our program (and keep Merlin happy):

```ocaml
let verify src dst prog : proof option = None
```

### Implementing `verify`

First of all, let's try to prove, that there is a path in the program
call graph from the source function to the destination function. It
would be even easier then the trivial proof:

```ocaml
let verify src dst prog : proof option =
  let cg = Program.to_graph prog in
  match Graphlib.shortest_path (module CG) cg src dst with
  | Some path -> Some (Calls path)
  | None -> None
```

Proving that for all control flow graphs there is no such callsite to
the destination function that is reachable from a callsite to the source,
would be a little bit harder. We need to enumerate all subroutines,
obtain their control flow graphs, and then check the connectivity of each pair of
calls, that lead to the destination and source function respectively.

We will address this problems using the bottom-up approach. We will
start with the supporting code, that will enumerate all interesting
callsites from a subroutine term. For that we need to identify whether
a callsite, that is a call term, may lead to an invocation of a target
function. That would be the `reaches` predicate:

```ocaml
let reaches cg callee target =
  Graphlib.is_reachable (module CG) cg callee target
```

Now we are ready to write the `callsites` function, that will take the
call graph `cg`, the `target` function, and the subroutine term `sub`,
and return a sequence of calls that has a destination function, that
reaches `target` in the call graph.

```ocaml
let callsites cg target sub =
  Term.enum blk_t sub |>
  Seq.concat_map ~f:(fun blk ->
      Term.enum jmp_t blk |> Seq.filter_map ~f:(fun j ->
          match Jmp.kind j with
          | Goto _ | Ret _ | Int (_,_) -> None
          | Call dst -> match Call.target dst with
            | Direct tid when reaches cg tid target ->
              Some (Term.tid blk)
            | _ -> None))
```

In our implementation, we are ignoring local jumps, as well as return
statements and CPU interrupts. We also ignore indirect jumps and
calls, expecting that it is the task for another analysis to make all
possible control flow explicit in the graph.

Now we have all the necessary tools to finish our analysis, we will
update the `verify` function:

```ocaml
let verify src dst prog : proof option =
  let cg = Program.to_graph prog in
  match Graphlib.shortest_path (module CG) cg src dst with
  | Some path -> Some (Calls path)
  | None ->
    Term.enum sub_t prog |> Seq.find_map ~f:(fun sub ->
        let g = Sub.to_graph sub in
        Seq.find_map (callsites cg src sub) ~f:(fun sc ->
            Seq.find_map (callsites cg dst sub) ~f:(fun dc ->
                if Tid.equal sc dc then None
                else Graphlib.shortest_path (module CFG) g sc dc))) |>
    Option.map ~f:(fun p -> Sites p)
```

That's it, for each subroutine in the program, we generate the control
flow graph, then we iterate over all callsites that reach the source
function, and over all callsites that reach the destination function,
and check that if these two callsites are not equal, then there should
not be a path between them, and if there is, then it is the proof of
the policy violation. We added the equality test, because if the same
callsite (and we denote a callsite by the basic block that hosts the
call) reaches both the destination and the source, then it means that
we have two distinct calls in the basic block that are mutually
exclusive by the invariant of a control flow graph, or that we have
one call where the destination function is invoked strictly before the
source function. The only other case, is ruled out by the first phase
of our analysis, that proved that there is no path between the source
function and the destination function in the call graph.

## Python Implementation

The Python implementation will follow closely the OCaml
implementation, so it is recommended to read the latter, to get some
insights. The main difference is that in Python we need to build
graphs manually. It is quite possible, that we will introduce the code
for graph building in our next release of Python bindings, but so far
(as of version 1.3) we need to build them manually. Anyway, building a
graph is an excellent showcase of how to work with the complex IR
structures. But we will start from the easy stuff first.

### Running bap from Python

Unfortunately, currently we do not provide an interface for writing
real plugins in Python and so far, the only way to use BAP from Python
is to run the `bap` executable, and to parse its output. Fortunately,
we provide some supporting code for this, so running `bap` is as easy
as:

    import bap
    proj = bap.run('./echo')

We recommend to run bap in some interactive REPL, such as IPython,
that makes it easy to explore the data structures that are returned
from the bap module functions.

### Setting up the user interface

We will use the standard `argparse` module for argument parsing, and
the Networkx library to work with graphs. So let's start with writing
some boilerplate code:

```python
import bap
import networkx as nx
import argparse

def verify(prog, src_name, dst_name):
    pass

description = """
Verifies the safety property that the DST function is never called
after the SRC function is called
"""

def main():
    parser = argparse.ArgumentParser(description=description)
    parser.add_argument('filename', help='target filename')
    parser.add_argument('-s', '--src', required=True, help='the source function')
    parser.add_argument('-d', '--dst', required=True, help='the sink function')
    args = parser.parse_args()
    proj = bap.run(args.filename)
    result = verify(proj.program, args.src, args.dst)
    if result:
        print('unsatisfied ({} are reachable via {})'.format(str(result[0]), str(result[1])))
    else:
        print('satisfied')

if __name__ == '__main__':
    main()
```

We specify one positional argument for the target binary, as well as
two mandatory arguments, the names of the source and destination
functions. Our verification function, is expected to return `None`, if
no counter-example was found, or a tuple if we found a proof. The
first element of the tuple represents the kind of the proof, i.e.,
whether we found a relation in a call graph or in a control flow
graph, and the second element is the path itself.

### Building Graphs

BAP represents graphs using [Algebraic Data Types][5]. It is easier
to work with ADT using the Visitor pattern. The idea is simple. A
complex ADT, like the IR in our case, is a tree with many different
kinds of branches and leaves. The base visitor class will traverse the
tree, and every time an element of type `Xxx` is entered, a method
named `enter_Xxx` will be called (if it exists). For example, when a
basic block is entered, the `enter_Blk` method will be invoked with
the particular instance of the block.

We will implent `GraphsBuilder` that will descent
through the program IR and build two kinds of graphs simultaneously -
the program callgraph, and a list of control flow graphs (for each
subroutine). Our builder state (other than graphs itself) will include
the current subroutine and the current basic block:

```python
class GraphsBuilder(bap.adt.Visitor):
    def __init__(self):
        self.callgraph = nx.DiGraph()
        self.cfgs = []
        self.sub = None
        self.blk = None
```

Every time we enter into a new subroutine term we create an empty
digraph and append it to the list of control flow graphs. We also add
a node to the call graph and label this node with the
reference to the newly created control flow graph - this will make it
easier for us to get the CFG of a function, the functionality that we
will need later. We also update the currently processed subroutine
(we do not really store the subroutine number, as we only interested
in the term identifier of the subroutine).

```python
    def enter_Sub(self, sub):
        cfg = nx.DiGraph()
        self.callgraph.add_node(sub.id.number, name=sub.name, sub=sub, cfg=cfg)
        self.sub = sub.id.number
        self.cfgs.append(cfg)
```

When we enter a basic block, we push its term identifier as a node to
the currently built control flow graph and update the current block
number:

```python
    def enter_Blk(self, blk):
        self.cfgs[-1].add_node(blk.id.number, blk=blk)
        self.blk = blk.id.number
```

When we see a call we need to add an edge to the call graph and an edge
to the current control flow graph to represent the fallthrough edge
(if the call is not marked as a non-return call).

```python
    def enter_Call(self, jmp):
        callee = direct(jmp.target[0])
        if callee:

        fall = direct(jmp.target[1]) if len(jmp.target) == 2 else None
        if fall:
            self.cfgs[-1].add_edge(self.blk, fall.number, jmp=jmp)
```

Note, that we are ignoring indirect calls, and using the following
helper function to extract the destination of a direct jump:

```python
def direct(jmp):
    return jmp.arg if jmp is not None and jmp.constr == 'Direct' else None
```

Whenever we enter the `Goto` statement, that represent a transfer of
the control flow between two basic block in a function, we just add a
new edge, if this transfer is explicit (i.e., direct)

```python
    def enter_Goto(self, jmp):
        dst = direct(jmp.target)
        if dst:
            self.cfgs[-1].add_edge(self.blk, dst.number, jmp=jmp)
```

Finally, when a CPU exception occurs, we just add a fallthrough edge
to the currently built control flow graph

```python
    def enter_Exn(self, exn):
        fall = exn.target[1]
        if fall:
            self.cfgs[-1].add_edge(self.blk, fall.number, exn=exn)
```

[5]: http://pythonhosted.org/bap/toc-bap.adt-module.html

### Implementing the Callsites Collector

As in the OCaml version, we will collect all callsites that reach
targets in which we are interested. The implementation is rather
straight-forward, whenever we see a call, we look at the call
destination, and if it is known and reaches either the source
function, or the destination function, we add it to the corresponding
list.

```python
class CallsitesCollector(bap.adt.Visitor):
    def __init__(self, callgraph, src, dst):
        self.callgraph = callgraph
        self.caller = None
        self.src = src
        self.dst = dst
        self.srcs = []
        self.dsts = []

    def clear(self):
        self.srcs = []
        self.dsts = []

    def enter_Blk(self, blk):
        self.caller = blk.id.number

    def enter_Call(self,jmp):
        callee = direct(jmp.target[0])
        if callee:
            if nx.has_path(self.callgraph, callee.number, self.src):
                self.srcs.append(self.caller)
            if nx.has_path(self.callgraph, callee.number, self.dst):
                self.dsts.append(self.caller)
```

We also added the `clear` method that will drop both lists, as we are
planning to reuse the same collector for all subroutines.

### Implementing The Verification Procedure

Now we have a full arsenal of tools to implement the verification
procedure. It is rather straightforward, we will start by finding
target functions, using the [find][6] method that is provided by all
instances of the `Seq` class. If one of the targets is not present,
then the verification condition is satisfied trivially and we bail out
of the function. Otherwise, we instantiate `GraphsBuilder` and run it
to build the graphs. We then look at the call graph to see if there
is a path between the source and destination functions. If there is no
such path, then we are looking at each CFG for a pair of
callsites to the target functions that has a path. And if there is one,
then it is returned.

```python
def verify(prog, src_name, dst_name):
    src = prog.subs.find(src_name)
    dst = prog.subs.find(dst_name)
    if src is None or dst is None:
        return None

    graphs = GraphsBuilder()
    graphs.run(prog)
    cg = graphs.callgraph

    if nx.has_path(cg, src.id.number, dst.id.number):
        return ('calls', nx.shortest_path(cg, src.id.number, dst.id.number))

    calls = CallsitesCollector(graphs.callgraph, src.id.number, dst.id.number)

    for sub in prog.subs:
        calls.run(sub)
        cfg = graphs.callgraph.nodes[sub.id.number]['cfg']
        for src in calls.srcs:
            for dst in calls.dsts:
                if src != dst and nx.has_path(cfg, src, dst):
                    return ('sites', nx.shortest_path(cfg, src, dst))
        calls.clear()

    return None
```

[6]: http://pythonhosted.org/bap/bap.adt.Seq-class.html#find

repository: BwRy/bap-tutorial

repo_name: BwRy/bap-tutorial


text:
# Grafana Dashboard

We have created a custom detailed dashboard for monitoring the stats of Redis Cluster.

- [Redis Operator Cluster Dashboard for Prometheus](https://github.com/OT-CONTAINER-KIT/redis-operator/blob/master/dashboards/redis-operator-cluster.json)

![](./images/grafana1.png)

![](./images/grafana2.png)

repository: ankeshdave/redis-operator

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: ankeshdave/redis-operator


text:
## Post Pandemic Bucket List

Nearly a year into the pandemic, everyone is eager for a little normalcy. This site allows users to dream of their post-pandemic life by creating their personalized bucketlists. It also allows the user to update their bucketlist with the "carpe diem" button which switches items from the "to-do" column to the "done" column.

![Alt Text](bucketlist.gif)
* [Deployed Link](https://cryptic-falls-03491.herokuapp.com/)

## **Built With**
* [HTML](https://developer.mozihlla.org/en-US/docs/Web/HTML)
* [CSS](https://developer.mozilla.org/en-US/docs/Web/CSS)
* [Git Hub](https://github.com/)
* [JavaScript](https://www.javascript.com/)
* [MySQL](https://www.mysql.com/products/workbench/)
* [Express](https://expressjs.com/)
* [Node](https://nodejs.org/en/)
*  Git - used to track changes to code
______________________________________________________________________________

#### **Code Snippet**
This application uses the handlebars npm package as well as the express package. Handlebars is only equipped to read data in the form of an object so in order to read it we must first make it an object. In the code below, the handlebars object is represented by the const hbsObject. This hbsObject is then passed to the res.render function which is a built in express function. It renders to the 'index' page which in turn defaults to the main page.

```
     router.get('/', function (req, res) {
    bucketlist.all(function (data) {
        const hbsObject = {
            bucketItem: data
        };
        console.log(hbsObject);
        res.render('index', hbsObject);
    });
});

```
The code below is what enables us to render to the index page without having the expressly write index.handlebars and pushes that content to the main page.
```
app.engine('handlebars', exphbs({ defaultLayout: 'main' }));
app.set('view engine', 'handlebars');
```
______________________________________________________________________________

### **Author Links**

* **PAMELA GUTIERREZ**
- [Link to Portfolio Site](https://pamela-gutierrez.github.io/updated-portfolio/)
- [Link to Github](https://github.com/pamela-gutierrez)
- [Link to LinkedIn](www.linkedin.com/in/pamela-gutierrez)

* **UC Berkeley Coding Bootcamp**

______________________________________________________________________________

#### **License**

This project is licensed under the MIT License

license: MIT

repository: pamela-gutierrez/post-pan-bucketlist

branch_name: refs/heads/main

repo_name: pamela-gutierrez/post-pan-bucketlist


text:
---
title: "Replacements"
date: 2021-01-31T14:14:11-08:00
draft: false
weight: 6
summary: Automate version values in your entire project with find and replace options.
---

### replacements
type: `[]Replacement` | default: `empty` | optional

List of replacements.

## Replacement
type: `struct`

When working in projects that include the version directly in the source code
replacements can be used to replace those values.
This works similar to the find and replace from IDE tools but also includes the
file path of the file.

### path
type: `string` | default: `""` | required

Path of the file to find and replace in.

### find
type: `string` | default: `""` | required

Regular expression to search for in the file.

### replace
type: `string` | default: `""` | required

Template string to replace the line with.

**Replace Arguments**

| Field | Type | Description |
| --- | --- | --- |
| **Version** | `string` | Semantic version of the release, includes `v` suffix if used |
| **VersionNoPrefix** | `string` | Semantic version of the release without the suffix if used |

repository: developgo/changie

repo_name: developgo/changie


text:
---
title: Using Android Support Library
keywords: android, mobile, java, support, library, bug, developer, froyo, gingerbread, honeycomb, ice cream sandwhich, jelly beans
description: |
  Supporting older version of Android is relatively easy, since Google gave us the support library. But, unfortunately, installing it properly is a bit problematic. Check here for the proper installation way.
tags: java, android
---

Despite Google attempt to keep pushing newer version of Android to the market. Device manufacturer is still lacking behind in conveying the updates to user for various reasons. Geeks around the world has been benefiting from community projects such as [CyanogenMod](http://cyanogenmod.com/) for alternative in getting the updates. On the other hands, non-tech-savvy user is still desperately locked on the decision made by their device manufacturer. This results in, by the time of writing, Gingerbread (API Level 10) still holding on 28.5% of the available device. You can see the updated distribution [here](http://developer.android.com/about/dashboards/index.html) btw.

Now, for us developer, this means that not supporting those devices equal a big lost in market opportunity. Luckily Android have backward compatibility support for their newest features covered in the form of [Support Library](http://developer.android.com/tools/support-library/setup.html). The guidelines for using it has also been covered there, but.. I encountered a couple of slight bump when trying to use the v7 support library with eclipse.

What they tell you to do, to set it up is basically :

1. Download the library,
2. Import the library project into eclipse and configure it,
3. Tell your project to use the support library.
4. ...
5. Profit!!

What they don't tell you though is, since Android uses Ant, it has the same limitation that Ant have. So for the support library to be installed correctly, you want to make sure that :

1. The path leading to the support library must have no space. I find it work best to just copy the whole project that I use (e.g: `appcompat`), and put it alongside the project using it, then target it using relative path. Example, If your project is at `/path/to/my/project` then put the library at `/path/to/my/library` So that you can refer to it using `../library`

2. When they tell you to "Add the library to your application project", what they meant is actually to:

   1. Open the project properties
   2. Go to Android
   3. Under "Library", click Add and choose the intended library.
   4. See that there is a green checkmark. If you get a redcross instead, it means that ant can't find it, and it usually means the path leading to the library have invalid character (like space).

repository: ignatiusreza/blog

gha_language: JavaScript

gha_license_id: MIT

repo_name: ignatiusreza/blog


text:
Title: Classification — Statistical Learning
Date: 2018-08-29 14:46
Author: ankitrathi
Category: Uncategorized
Tags: Classification, Data Science, Datadeft, Logistic Regression, Machine Learning
Slug: classification-statistical-learning
Status: published

![](https://cdn-images-1.medium.com/max/1200/1*WE7ndSyOB2lTp4e9OVD5RA.png)

*This is the 3rd post of blog post series ‘*[*Statistical Learning Notes*](https://medium.com/@rathi.ankit/statistical-learning-notes-series-c8c218102ae0)*’, this post is my notes on ‘Chapter 4 — Classification’ of ‘Introduction to Statistical Learning (ISLR)’. Here I have tried to give an intuitive understanding of key concepts and how these concepts are related to Classification.*

\[embed\]http://www-bcf.usc.edu/\~gareth/ISL/\[/embed\]

*Note: I suggest the reader to refer ISLR book in case he/she wants to dig further or wants to look for examples.*

------------------------------------------------------------------------

Predicting a *qualitative response* for an observation can be referred to as *classifying* that observation, since it involves *assigning* the observation to a *category* or *class*.

Given a feature vector X and a qualitative response Y taking values in the set *C*, the classification task is to build a function C(X) that takes as input the feature vector X and predicts its value for Y. Often we are more interested in *estimating the probabilities* that X belongs to *each category* in *C*.

![](https://cdn-images-1.medium.com/max/800/1*McStH0zjIgyg6N2O5uQUhg.png)

We might think that *regression* is perfect for *classification* task as well. However, linear regression might produce *probabilities* *less than zero* or *bigger than one*. Logistic regression is more appropriate.

![](https://cdn-images-1.medium.com/max/800/1*bLw4dtlQGDsnhMOkOH534g.png)

Now suppose we have a *response variable* with *three* possible values. If we *code the categories*, it will suggest an ordering, and in fact implies that the *difference between categories* is the same. So, we can see that *Linear Regression* is not appropriate here. *Multi-class Logistic Regression* or *Discriminant Analysis* are more appropriate.

**Logistic Regression**

*Logistic regression* models the *probability* that y belongs to a particular category rather than modeling the response itself. It uses the *logistic function* to ensure a prediction between 0 and 1. The logistic function takes the form:

![](https://cdn-images-1.medium.com/max/800/1*JXY8s-7rc9M_QD47iuuTmw.png)

**Estimating Regression Coefficients**

*Logistic regression* uses a strategy called *maximum likelihood* to estimate regression coefficients.

We use maximum likelihood to estimate the parameters.

![](https://cdn-images-1.medium.com/max/800/1*fqC_j-VdT4upIQ-gRMha-w.png)

This *likelihood* gives the probability of the observed 0s and 1s in the data. We pick 0 and 1 to maximize the likelihood of the observed data.

Logistic regression measures the accuracy of coefficient estimates using a quantity called the *z-statistic*. The z-statistic is similar to the *t-statistic*. The *z-statistic* for *β1* is represented by:

![](https://cdn-images-1.medium.com/max/800/1*F3lg10fXNDkvZp7w56AgkA.png)

A *large z-statistic* offers *evidence against* the *null hypothesis*.

In logistic regression, the null hypothesis:

Null hypothesis (*H0*): *β1=0* implies that

![](https://cdn-images-1.medium.com/max/800/1*x9_O-tXN7fzB46GPMqOC9g.png)

which means *p(X)* does not depend on *X*.

**Making Predictions**

Once coefficients have been estimated, *predictions* can be made by plugging the coefficients into the *model equation*.

In general, the *estimated intercept* (*β0*) is of limited interest since it mainly captures the *ratio* of *positive* and *negative* classifications in the given data set.

Similar to linear regression, *dummy variables* can be used to accommodate *qualitative predictors*.

**Multiple Logistic Regression**

Using a strategy similar to that employed for linear regression, *multiple logistic regression* can be generalized as:

![](https://cdn-images-1.medium.com/max/800/1*LHkrh4jd2HDHtew5m8jEag.png)

*Maximum likelihood* is also used to estimate *β0,β1,…,βp* in the case of multiple logistic regression.

In general, the scenario in which the result obtained with a single predictor does not match the result with multiple predictors, especially when there is correlation among the predictors, is referred to as *confounding*. More specifically, *confounding* describes situations in which the *experimental controls* do not adequately allow for ruling out *alternative explanations* for the observed relationship between the predictors and the response.

**Logistic Regression For More Than Two Classes**

Though multi-class logistic regression is possible, *discriminant analysis* tends to be the preferred means of handling multi-class classification.

**Linear Discriminant Analysis (LDA)**

While *logistic regression* models the *conditional distribution* of the response Y given the predictor(s) X, *linear discriminant analysis* (LDA) is an approach to model the *distribution* of X in each of the *classes separately*, and then use *Bayes’ theorem* to flip things around and obtain *Pr(Y/X)*.

*Linear discriminant analysis* is popular when there are *more than two response* classes, because it also provides *low-dimensional views* of the data. Beyond its popularity, linear discriminant analysis also benefits from not being susceptible to some of the problems that logistic regression suffers from:

-   The *parameter estimates* for logistic regression can be *surprisingly unstable* when the *response classes are well separated*. Linear discriminant analysis does not suffer from this problem.
-   *Logistic regression* is *more unstable* than *linear discriminant analysis* when *n is small* and the *distribution* of the predictors X is *approximately normal* in each of the response classes.

When we use *normal (Gaussian) distributions* for each class, this leads to *linear or quadratic discriminant analysis*. However, this approach is quite general, and other distributions can be used as well.

**Classification With Bayes’ Theorem**

Assuming a qualitative variable YY that can take on K≥2 distinct, unordered values, the *prior probability* describes the probability that a given observation is associated with the *kth* class of the response variable Y.

The *density function* of X for an observation that comes from the *kth* class is defined as:

![](https://cdn-images-1.medium.com/max/800/1*qU7XHSk5R_uYSAC0WfTd1g.png)

This means that *fk(X)* should be *relatively large* if there’s a *high probability* that an observation from the *kth* class features X=x. Conversely, *fk(X)* will be *relatively small* if it is unlikely that an observation in class k would feature X=x.

Following this intuition, *Bayes’ theorem* states:

![](https://cdn-images-1.medium.com/max/800/1*Dv0fY6GfoWq1YmUceSBIlw.png)

where *πk* denotes the *prior probability* that the chosen observation comes from the *kth* class. This equation is sometimes abbreviated as *pk(x)*.

*pk(x)=Pr(Y=k\|X)* is also known as the *posterior probability*, or the probability that an observation belongs to the *kth* class, given the predictor value for that observation.

![](https://cdn-images-1.medium.com/max/800/1*war7jDX-ccPt-ip09kAA0Q.png)

Estimating the *prior probability* (*πk*) is easy given a random sample of responses from the population.

Estimating the *density function fk(X)* tends to be harder, but making some assumptions about the form of the densities can simplify things. A good estimate for *fk(X)* allows for developing a classifier that approximates the Bayes’ classifier which has the *lowest possible error rate* since it always selects the class for which *pk(x)* is largest.

**Linear Discriminant Analysis For One Predictor**

When only considering one predictor, if we assume that fk(X)fk(X) has a *normal or Gaussian distribution*, the normal density is described by:

![](https://cdn-images-1.medium.com/max/800/1*fODhv6dp0ujSn2EnRHuLgg.png)

where *μk* is the *mean parameter* for the *kth* class and *σ²k* is the *variable parameter* for the *kth class*.

The *density function* can be further simplified by assuming that the *variance* *terms*, *σ²1,…,σ²k*, are all equal in which case the variance is denoted by σ².

Plugging the simplified normal density function into *Bayes’ theorem* yields:

![](https://cdn-images-1.medium.com/max/800/1*3JNK-NntqmlgI8tlAZkA6g.png)

It can be shown that by taking a log of both sides and removing terms that are not class specific, a simpler equation can be extracted:

![](https://cdn-images-1.medium.com/max/800/1*NJ38XF-ReqaTvbXXYGCoYQ.png)

Using above equation, an observation can be classified by taking the class yields the largest value.

Linear discriminant analysis uses the following estimated values for μ\^k and σ²:

![](https://cdn-images-1.medium.com/max/800/1*4ySexeFyqgCvC-NKGNWfRw.png)

where n is the total number of training observations and *nk* is the number of training observations in class k. The estimate of *μ\^k* is the average value of x for all training observations in class k. The estimate of σ² can be seen as a weighted average of the sample variance for all k classes.

When the class prior probabilities, *π1,…,πk,*is not known, it can be estimated using the proportion of training observations that fall into the *kth* class:

*πk*=*nk/n*

Plugging the estimates for *μ\^k* and *σ²k* into the modified Bayes’ theorem yields the *linear discriminant analysis classifier*:

![](https://cdn-images-1.medium.com/max/800/1*4Xf-yf2oqIWMU6VxibjWRw.png)

which assigns an observation *X=x* to whichever class yields the largest value.

This classifier is described as linear because the discriminant function *δ\^k(x)* is linear in terms of *x* and not a more complex function.

The *linear discriminant analysis classifier* assumes that the observations from *each class follow a normal distribution* with a class specific average vector and constant variance (σ²), and uses these simplifications to build a Bayes’ theorem based classifier.

**Linear Discriminant Analysis with Multiple Predictors**

*Multivariate linear discriminant analysis* assumes that *X=(X1,X2,…,Xp)* comes from a *multivariate normal distribution* with a *class-specific mean vector* and a *common covariance matrix*.

The *multivariate Gaussian distribution* used by *linear discriminant analysis* assumes that each predictor follows a *one-dimensional normal distribution* with *some correlation* between the predictors. The *more correlation* between predictors, the *more* the bell shape of the normal distribution will be *distorted*.

A p-dimensional variable X can be indicated to have a multivariate Gaussian distribution with the notation X∼N(μ,Σ) where E(x)=μ is the mean of XX (a vector with p components) and Cov(X)=Σ is the p x p covariance matrix of X.

As was the case for one-dimensional linear discriminant analysis, it is necessary to estimate the unknown parameters μ1,…,μk, π1,…,πk, and Σ. The formulas used in the multi-dimensional case are similar to those used with just a single dimension.

Since, even in the multivariate case, the *linear discriminant analysis* decision rule *relates to X in a linear fashion*, the name linear discriminant analysis holds. And as with other methods, the *higher the ratio of parameters* p, to number of samples n, the *more likely overfitting* will occur.

In general, binary classifiers are subject to two kinds of error: *false positives* and *false negatives*. A *confusion matrix* can be a useful way to display these error rates. *Class-specific performance* is also important to consider because in some cases a particular class will contain the *bulk of the error*.

The term *sensitivity* refers to the percentage of observations correctly positively classified (true positives) and *specificity* refers to the percentage of observations correctly negatively classified (true negatives).

A *ROC curve* is a useful graphic for displaying the two types of error rates for all possible thresholds. ROC is a historic acronym that comes from communications theory and stands for *receiver operating characteristics*.

![](https://cdn-images-1.medium.com/max/800/0*Gg3ixBvG35uHg-Th.jpg)

The *overall performance* of a classifier summarized over all possible thresholds is quantified by the *area under the ROC curve*.

A more ideal ROC curve will hold more tightly to the top left corner which, in turn, will increase the area under the ROC curve. A classifier that performs no better than chance will have an area under the ROC curve less than or equal to 0.5 when evaluated against a test data set.

In summary, varying the *classifier threshold* changes its *true positive* and *false positive* rate, also called *sensitivity* and (*1−specificity*).

**Quadratic Discriminant Analysis**

*Quadratic discriminant analysis* offers an *alternative approach* to linear discriminant analysis that makes most of the same assumptions, except that quadratic discriminant analysis assumes that *each class* has its own *covariance matrix*.

The quadratic discriminant analysis Bayes classifier gets its name from the fact that it is a *quadratic function* in terms *of x*.

The choice between a *shared covariance matrix* (like that assumed in *linear discriminant analysis*) and a *class-specific covariance matrix* (like that assumed in *quadratic discriminant analysis*) amounts to a *bias-variance trade-off*.

**Comparing Classification Methods**

Since *logistic regression* and *linear discriminant analysis* are both linear in terms of *x*,the *primary difference* between the two methods is *their fitting procedures*. Linear discriminant analysis assumes that observations come from a *Gaussian distribution* with a *common covariance matrix*, and as such, out performs logistic regression in cases where these assumptions hold true.

*K-nearest neighbors* can outperform *logistics regression* and *linear discriminant analysis* when the *decision boundary* is *highly non-linear*, but *at the cost of* a *less interpretable model*.

*Quadratic discriminant analysis* falls somewhere between the linear approaches of *linear discriminant analysis* and *logistic regression* and the non-parametric approach of *K-nearest neighbors*. Since quadratic linear analysis models a *quadratic decision boundary*, it has more capacity for modeling a wider range of problems. *Quadratic discriminant analysis* is not as flexible as *K-nearest neighbors*, however it can perform better than *K-nearest neighbors* when there are *fewer training observations* due to its high bias.

*Linear discriminant analysis* and *logistic regression* will perform well when the true *decision boundary* is *linear*. *Quadratic discriminant analysis* may give *better results* when the *decision boundary* is *moderately non-linear*. *Non-parametric* approaches like *K-nearest neighbors* may give better results when the *decision boundary* is *more complex* and the right level of smoothing is employed.

As was the case in the *regression* setting, it is possible to apply *non-linear transformations* to the predictors to better accommodate non-linear relationships between the response and the predictors. The effectiveness of this approach will depend on whether or not the *increase in variance* introduced by the *increase in flexibility* is offset by the *reduction in bias*.

It is possible to add *quadratic terms* and *cross products* to the *linear discriminant analysis* model such that it has the same form as *quadratic discriminant analysis*, however the *parameter estimates* for each of the models would be *different*. In this fashion, it’s possible to build a model that falls somewhere between *linear discriminant analysis* and *quadratic discriminant analysis*.

**References:**

\[embed\]http://www-bcf.usc.edu/\~gareth/ISL/\[/embed\]\
<https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/classification.pdf>

------------------------------------------------------------------------

*Thank you for reading my post. I regularly write about Data & Technology on* [*LinkedIn*](https://www.linkedin.com/today/posts/ankitrathi) *&* [*Medium*](https://medium.com/@rathi.ankit)*. If you would like to read my future posts then simply ‘Connect’ or ‘Follow’. Also feel free to listen to me on* [*SoundCloud*](https://soundcloud.com/ankitrathi)*.*


text:
# IAR-stm8-NightLight

NightLight

IAR project for STM8S103F3P6 mcu.
Photodiode is connected to internal ADC and used for light level monitoring.

When light level is low, LED will be turned on via logic-level MOSFET transistor.


text:
# Further Setup
(Nginx with PHP, Symlinks, Permissions)

## Dependencies
```sh
 sudo apt install php5-common webp imagemagick gphoto2 rsync php5-fpm
```

[Enable PHP for Nginx](http://askubuntu.com/questions/134666/what-is-the-easiest-way-to-enable-php-on-nginx) and [Enable symlinks](http://unix.stackexchange.com/questions/157022/make-nginx-follow-symlinks) for Nginx
```sh
sudo nano /etc/nginx/sites-available/default
```

```sh
sudo service php5-fpm restart
sudo service nginx restart
```
## Symbolic links
```sh
sudo ln -s ~/git/photo photo
sudo ln -s ~/git/gal gal
```
## Permissions
Permissions for `www-data`
```sh
sudo chown www-data:www-data -R photo
sudo chown www-data:www-data -R gal
```

Allow gphoto2 to be used by `www-data`
```sh
sudo chmod +s $(which gphoto2)
```
## SCP (upload to server)
Upload albums without a password (mandatory for UI uploads)
```sh
# copy ssh directory of user that server already trusts
sudo cp /home/trusted_username/.ssh/* /var/www/.ssh/
```
# Optional

[Https (SSL)](https://www.digitalocean.com/community/tutorials/how-to-create-an-ssl-certificate-on-nginx-for-ubuntu-14-04) for Nginx
[Redirect HTTP to HTTPS](https://www.digitalocean.com/community/questions/http-https-redirect-positive-ssl-on-nginx) for Nginx

Disable root login
```sh
sudo nano /etc/ssh/sshd_config
```

Here are some lines from my Nginx config that show where log files are
```sh
#        access_log /var/log/nginx/access.log;
#        error_log /var/log/nginx/error.log;
```

### Additional information
Photos
Place photos in a sibling directory of the project named `photo` (can be symbolic link). This can be customized in `config.php`.

repository: zvakanaka/php-gal

repo_name: zvakanaka/php-gal


text:
# Mac'n'Minos

Test Case For Dominos ®

## Requirements

- Node > 13.2
- Yarn > ~

## Installation

    yarn

#### OR

    yarn install

## Usage

I used json-server for API. For raise API

    yarn api

Note: App needs API. It will not work without API

For Development:

    yarn start

For SSR:

    yarn server

repository: yusufilkeroguz/mac-n-minos

repo_name: yusufilkeroguz/mac-n-minos


text:
#Changelog

## 1.2.0 (2022-12-22)

### Added
* compatibility with EXAM REST-API 1.2.0
* dokumentation is moved to Github
* execute TCG of EXAM

### removed
* Test Model Connection with Soap

## 1.1.1 (2020-05-05)

### Added
* [JENKINS-58411](https://issues.jenkins-ci.org/browse/JENKINS-58411) make connection timeout configurable
* [JENKINS-61057](https://issues.jenkins-ci.org/browse/JENKINS-61057) execute a GroovyScript (since EXAM 4.8)

### Fixed
* [JENKINS-58412](https://issues.jenkins-ci.org/browse/JENKINS-58412) Job connot be aborted

## 1.0.8 (2019-05-09)

### Added
 * [JENKINS-57100](https://issues.jenkins-ci.org/browse/JENKINS-57100) Improve Communication with REST-API

### Fixed
 * [JENKINS-57099](https://issues.jenkins-ci.org/browse/JENKINS-57099) Configuration of EXAM Installation not loaded

## 1.0.3 (2019-03-20)
first release

repository: jenkinsci/exam-plugin

branch_name: refs/heads/develop

gha_language: Java

gha_license_id: NOASSERTION

repo_name: jenkinsci/exam-plugin


text:
## Phoenix

### Officer tackles and arrests person walking home from protest | May 30th

An officer tackles a protestor that is walking home. The officer screams "put your hands behind your head" twice and then "stop resisting" twice, the protestor responds "I'm not".

repository: devinus/police-brutality

repo_name: devinus/police-brutality


text:
# LYRA Block Lattice

LYRA is DPoS/DAG technology that provides out of the box real time authorizations, multi-token support, built-in decentralized exchange,
superlight clients, and many other features required for real world financial transaction processing.

DAG-style block lattice ledger and DPoS consensus algorithm enable a lot of unique and interesting features that make it a perfect fit
for many real life applications:

* Super Light Clients (Wallets)
* Very High Scalability (tps)
* User-defined fungible and non-fungible tokens
* Pre-programmed Smart Contracts
* Instant Transactions (Real Time Authorizations and Settlements)
* Built-in Instant Decentralized Token Exchange
* Flexible Transaction Fee Structure
* Non-locking wallet balances

[**LYRA Documentation Site**](https://docs.lyra.live)

[**LYRA on Twitter**](https://twitter.com/LYRAblockchain)

[**LYRA on Telegram**](https://t.me/joinchat/F25OCR1H3Fdq_IUNJfclSQ)

repository: wizd/LYRA-Docs

repo_name: wizd/LYRA-Docs


text:
RemotePrinterController

  About   |
  Technologies   |
  Starting   |
  License   |
  Author

## About ##

Using a M5StackCore2 you will be able to see the status of your 3D printer. To develop this project we used the Octoprint APIs to request the data that comes from the printer. The M5Stack displays some data: at the moment we are asking for printer status, bed temperature and tool temperature. In the futher releases we will implement more control from the M5Stack to the printer like cancel a print or control the axes. We can do this because the M5Stack have integrated the ESP32 and a touch display which we will use to select printer options.
Octoprint is used as interface between the M5Stack and the printer: through its APIs we can interact with the printer and take the full control. The APIs are REST based so we use HTTP requests to take the data.

## Technologies ##

The following tools were used in this project:

- [PlatformIO](https://platformio.org/)
- [VSCode](https://code.visualstudio.com/)
- [ArduinoJson](https://arduinojson.org/)
- [M5Core2](https://github.com/m5stack/M5Core2)

## Starting ##

```bash
# Clone this project
$ git clone https://github.com/S7Industries/RemotePrinterController

# Access
$ cd remoteprintercontroller
```

Now you need to install PlatformIO into VSCode and than open the project.

You can even use Atom instead of VScode.
See PlatformIO site for supported IDEs.

## License ##

This project is under license from MIT. For more details, see the [LICENSE](LICENSE.md) file.

Made with :heart: by MarcS7

Back to top


text:
# [Pinboard](https://pinboard.in) Poster for Kotlin/Java

[![License (3-Clause BSD)](https://img.shields.io/badge/license-BSD%203--Clause-blue.svg?style=flat-square)](http://opensource.org/licenses/BSD-3-Clause) [![Release](https://img.shields.io/github/release/ethauvin/pinboard-poster.svg)](https://github.com/ethauvin/pinboard-poster/releases/latest) [![Maven Central](https://img.shields.io/maven-central/v/net.thauvin.erik/pinboard-poster.svg?label=maven%20central)](https://search.maven.org/search?q=g:%22net.thauvin.erik%22%20AND%20a:%22pinboard-poster%22)

[![Known Vulnerabilities](https://snyk.io/test/github/ethauvin/pinboard-poster/badge.svg?targetFile=pom.xml)](https://snyk.io/test/github/ethauvin/pinboard-poster?targetFile=pom.xml) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=ethauvin_pinboard-poster&metric=alert_status)](https://sonarcloud.io/dashboard?id=ethauvin_pinboard-poster) [![Build Status](https://travis-ci.com/ethauvin/pinboard-poster.svg?branch=master)](https://travis-ci.com/ethauvin/pinboard-poster) [![CircleCI](https://circleci.com/gh/ethauvin/pinboard-poster/tree/master.svg?style=shield)](https://circleci.com/gh/ethauvin/pinboard-poster/tree/master)

A small Kotlin/Java/Android library for posting to [Pinboard](https://pinboard.in).

## Examples

### Kotlin

```kotlin

val poster = PinboardPoster("user:TOKEN")

poster.addPin("http://www.example.com/foo", "This is a test")
poster.deletePin("http:///www.example.com/bar")

```
[View Example](https://github.com/ethauvin/pinboard-poster/blob/master/samples/kotlin/src/main/kotlin/net/thauvin/erik/pinboard/samples/KotlinExample.kt)

### Java
```java

final PinboardPoster poster = new PinBboardPoster("user:TOKEN");

poster.addPin("http://www.example.com/foo", "This is a test");
poster.deletePin("http:///www.example.com/bar");
```
[View Example](https://github.com/ethauvin/pinboard-poster/blob/master/samples/java/src/main/java/net/thauvin/erik/pinboard/samples/JavaExample.java)

Your API authentication token is available on the [Pinboard settings page](https://pinboard.in/settings/password).

## Usage with Gradle, Maven, etc.

To install and run from Gradle, add the following to the build.gradle file:

```gradle
dependencies {
    compile 'net.thauvin.erik:pinboard-poster:1.0.3'
}
```
[View Example](https://github.com/ethauvin/pinboard-poster/blob/master/samples/java/build.gradle)
[View Kotlin DSL Example](https://github.com/ethauvin/pinboard-poster/blob/master/samples/kotlin/build.gradle.kts)

Instructions for using with Maven, Ivy, etc. can be found on [Maven Central](https://search.maven.org/artifact/net.thauvin.erik/pinboard-poster/1.0.3/jar).

## Adding

The `addPin` function support all of the [Pinboard API parameters](https://pinboard.in/api/#posts_add):

```kotlin
poster.addPin(url = "http://www.example.com",
              description = "This is the title",
              extended = "This is the extended description.",
              tags = "tag1 tag2 tag3",
              dt = "2010-12-11T19:48:02Z",
              replace = true,
              shared = true,
              toRead = false)
```

`url` and `description` are required.

It returns `true` if the bookmark was added successfully, `false` otherwise.

## Deleting

The `deletePin` function support all of the [Pinboard API parameters](https://pinboard.in/api/#posts_delete):

```kotlin
poster.deletePin(url = "http://www.example.com/")
```

It returns `true` if the bookmark was deleted successfully, `false` otherwise.

## Logging

The library used [`java.util.logging`](https://docs.oracle.com/javase/8/docs/api/java/util/logging/package-summary.html) to log errors. Logging can be configured as follows:

#### Kotlin
```kotlin
with(poster.logger) {
    addHandler(ConsoleHandler().apply { level = Level.FINE })
    level = Level.FINE
}
```
#### Java
```java
final ConsoleHandler consoleHandler = new ConsoleHandler();
consoleHandler.setLevel(Level.FINE);
final Logger logger = poster.getLogger();
logger.addHandler(consoleHandler);
logger.setLevel(Level.FINE);
```

or using a logging properties file.

## API Authentication Token

The token can also be located in a [properties file](https://en.wikipedia.org/wiki/.properties) or environment variable.

### Local Property

For example, using the default `PINBOARD_API_TOKEN` key value from a `local.properties` file:

```ini
# local.properties
PINBOARD_API_TOKEN=user\:TOKEN
```

```kotlin
val poster = PinboardPoster(Paths.get("local.properties"))
```

or by specifying your own key:

```ini
# my.properties
my.api.key=user\:TOKEN
```

```kotlin
val poster = PinboardPoster(Paths.get("my.properties"), "my.api.key")
```

or even specifying your own property:

```kotlin
val p = Properties()
p.setProperty("api.key", "user:TOKEN")

val poster = PinboardPoster(p, "api.key")
```

_In all cases, the value of the `PINBOARD_API_TOKEN` environment variable is used by default if the specified property is invalid or not found._

### Environment Variable

If no arguments are passed to the constructor, the value of the `PINBOARD_API_TOKEN` environment variable will be used, if any.

```sh
export PINBOARD_API_TOKEN="user:TOKEN"
```

```kotlin
val poster = PinboardPoster()
```

## API End Point

The API end point is automatically configured to `https://api.pinboard.in/v1/`. Since Pinboard uses the `del.ico.us` API, the library could potentially be used with another compatible service. To configure the API end point, use:

```kotlin
poster.apiEndPoint = "https://www.example.com/v1"
```

repository: ethauvin/pinboard-poster

gha_language: Kotlin

repo_name: ethauvin/pinboard-poster


text:
# PgJobs
Simple ActiveJob worker for PostgreSQL using LISTEN/NOTIFY and
SKIP LOCKED.

Supports most ActiveJob features like multiple queues, priorities
and wait times.

Needs PostgreSQL 9.5 to use SKIP LOCKED.

## Usage
Just schedule your work with ActiveJob, then run one or multiple
workers for the default queue with
```bash
$ bin/rails runner PgJobs.work
```
or for other queues with
```bash
$ bin/rails runner "PgJobs.work(:my_queue)"
```

## Installation
Add this line to your application's Gemfile:

```ruby
gem 'pg_jobs'
```

And then execute:
```bash
$ bundle
```

Then copy the migrations and migrate your database:
```bash
$ bin/rails railties:install:migrations
$ bin/rails db:migrate
```

To configure the ActiveJob adapter add this to your environment
configuration (config/environments/production.rb):
```ruby
config.active_job.queue_adapter = :pg_jobs
```

## Contributing
Use Github issues and pull requests.

## License
The gem is available as open source under the terms of the
[MIT License](http://opensource.org/licenses/MIT).


text:
# Use apache to act as a reverse proxy for a local server running on another port and make it secure.
## Ubuntu 20.04 and Apache 2.4
```
ServerName example.com
<VirtualHost *:80>
    ServerName example.com
    Redirect permanent / https://www.example.com/
</VirtualHost>

<VirtualHost *:443>
    #SSL Config
    ServerName example.com
    DocumentRoot /var/www/html/
    SSLEngine on
    #SSLOptions +StrictRequire
    SSLVerifyClient none
    SSLProxyEngine on
    <IfModule mime.c>
        AddType application/x-x509-ca-cert .crt
        AddType application/x-pkcs7-crl    .crl
    </IfModule>
            SetEnvIf User-Agent ".*MSIE.*"nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    SSLProtocol TLSv1.2
    SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM

    #Do not put Directory stuff in here, does not work!
    SSLCertificateFile /etc/letsencrypt/live/www.example.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/www.example.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateChainFile /etc/letsencrypt/live/www.example.com/chain.pem
</VirtualHost>

#Require SSL everytwhere
<Directory /var/www/html>
    SSLRequireSSL
</Directory>

# proxy a local server running on port 8080 to www.example.com/program/
#   for qbittorrent-nox
#   https://github.com/qbittorrent/qBittorrent
ProxyRequests off
<Proxy *>
    allow from all
</Proxy>
RewriteEngine On
RewriteRule ^/gui/?(.*)$ /program/$1 [P]
<Location /torrent/>
    ProxyPass http://<IP_ADDRESS>:8080/
    ProxyPassReverse http://<IP_ADDRESS>:8080/
</Location>
```

## CentOS 7 and Apache 2.4

```
<VirtualHost *:80>
    Redirect permanent / https://www.example.com/
</VirtualHost>

<VirtualHost *:443>
    DocumentRoot /var/www/html/
    SSLEngine on
    SSLProxyEngine On
    SSLCertificateFile /etc/httpd/conf/apache_ssl_cert.crt
    SSLCertificateKeyFile /etc/httpd/conf/apache_ssl_cert.key
    SSLVerifyClient none
    SSLProxyEngine off
    <IfModule mime.c>
        AddType application/x-x509-ca-cert .crt
        AddType application/x-pkcs7-crl    .crl
    </IfModule>
    SetEnvIf User-Agent ".*MSIE.*"  nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    SSLProtocol TLSv1.2
    SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
</VirtualHost>

# Section for ruTorrent
#   https://github.com/Novik/ruTorrent
<Location /subdir/>
    SSLRequireSSL
    AuthType Basic
    AuthName "Please Enter Password"
    AuthBasicProvider file
    AuthUserFile /etc/httpd/conf/.htpasswd
    Require valid-user
    CacheDisable on
    ProxyPass         http://localhost:8080/ retry=0
    ProxyPassReverse  http://localhost:8080/
    SetOutputFilter   proxy-html
    ProxyHTMLURLMap   http://localhost:8080/
</Location>
RewriteRule ^/subdir$ /subdir/ [R]
```

repository: LRomandine/Notes

repo_name: LRomandine/Notes


text:
# TravelCard

You are given a stack of boarding cards for various transportations that will take you from a point A to point B via several stops on the way. All of
the boarding cards are out of order and you don't know where your journey starts, nor where it ends. Each boarding card contains information
about seat assignment, and means of transportation (such as flight number, bus number etc)

Write an API that lets you sort this kind of list and present back a description of how to complete your journey.
For instance the API should be able to take an unordered set of boarding cards, provided in a format defined by you

## Description

+ To write the program was used Domain-driven design (DDD) approach.
+ The Laravel command bus provides a convenient method of encapsulating tasks your application needs to perform into simple, easy to understand "commands".
+ In order to reduce the number of requests on the server, caching of query results was used.
+ Basically, a .env file is an easy way to load custom configuration variables that your application needs without having to modify .htaccess files or Apache/nginx virtual hosts.
+ Monolog sends your logs to files, sockets, inboxes, databases and various web services.
+ The EventDispatcher component use for dispatching events and listening to them.
+ Use a PHP HTTP client Guzzle to send HTTP requests and trivial to integrate with web services.
+ Use a PHPUnit testing framework for cover application tests

## Server

For run application use a Docker - is an open platform for developers and sysadmins to build, ship, and run distributed applications.

## Client
 - **[POST client/index/:data](https://github.com/dykyi-roman/TravelCard/blob/master/client/README.md)**

## Install

+ composer install && composer dumpautoload -o
+ change URL path to client API in .env file

## Usage

```php
 php index.php
```

```
**************************************************
************ Welcome to Application **************
**************************************************

Usage:
   command [options]

Available commands:
   Version    - Application version
   BuildRoute - Build Route
```

```php
 php index.php BuildRoute
```
```
Take train 78A From Madrid to Barcelona. Sit in seat 45B.
Take airport bus From Barcelona to Gerona.No seat assignment.
Take flight SK455, Gate 45B. From Gerona to Stockholm. Sit in seat 3A. Baggage drop at ticket counter 344.
Take flight SK22, Gate 22. From Stockholm to New York. Sit in seat 7B. Baggage will we automatically transferred from your last leg.
You have arrived at your final destination

```

## Author
[Dykyi Roman](https://www.linkedin.com/in/roman-dykyi-43428543/), e-mail: [mr.dukuy<EMAIL_ADDRESS>

repository: dykyi-roman/TravelCard

repo_name: dykyi-roman/TravelCard


text:
---
layout: story
category: short story
issue: 81, July 2020
issue-cover: issue-81-cover.png
issue-buy-link: https://store.firesidefiction.com/products/fireside-magazine-issue-81-july-2020
title: Even the Clearest Water
subtitle:
author: andi-c-buchanan
author-2:
author-3:
editor: l-d-lewis
illustrator:
copyeditor: chelle-parker
selector:
translator:
narrator: victor-ramos
letterer:
date: 2020-07-28 01:00:01 -0500
permalink: even-the-clearest-water
audiourl: https://traffic.libsyn.com/secure/firesidemagazine/Even_the_Clearest_Water.mp3
teaser: A story about water fae, autistic families, bargains, and love.
art:
artcaption:
spanish-language:
spanish-date:
english-url:
spanish-url:
serial-name:
serial-url:
part:
previous-url:
next-url:
contentnote: This story contains images of self-harm/suicidal ideation and harm to a child.
generalnote:
published: true
---
From water, white-capped, rushing over pebbles and stones, chasing round rocks, my flesh emerges; limbs stick thin and translucent like the freshwater they came from, wings glinting in the late-summer sun. I perch on the river edge, soft grass and slipping mud, listening for footsteps.

There’s water in the ground, even after summer brightness, and through it I can hear the child before she comes into view. I hear soft footsteps on the grass and bare toes curling round the damp soil, soaking in the grainy texture as if it was a sensation she’d been deprived of all her life and yet was as essential to her as breathing.

She wanders over the hill towards the stream, where she has been headed all along even if she doesn’t know it. Her pigtails are pulled so tight they must hurt and yet she shows no sign of distress. Her clothes are all green; her t-shirt and patchwork skirt, and likely her socks and shoes before she lost them, far behind her. She walks in small steps, her feet spread apart, her eyes wide as if this is a new world waiting for her. It's not magic, just observation; paying attention to these things about her tells me instantly she's autistic. Many humans would give her less understanding labels.

She sees me. I hold still, careful not to scare her. I pull my wings back, feeling them tremble in the breeze. Like many children, she moves forward with wonderment and curiosity rather than fear. I flutter a little way into the grass to entice her away from the water.

It works for a few moments, but then the pull of the water is too strong. She falls and the undercurrent pulls her down. I fly down, slipping into water, becoming water, and I speak to her.

“Do you want me to save you?” I ask. It’s the beginning of a negotiation. A bargain. Many humans would criticise me, but I am here and they are not.

This child is face-down in the water and does not respond. This is a child who does not speak, perhaps not ever or perhaps not now. I need her agreement if I’m going to save her. I look at her more closely.

I see a child who wants to live. A child who loves her family and the soil beneath her feet and, yes, the water. The fact that she can’t say any of that doesn’t make it any less clear. It does not mean she is not communicating.

Her will to live is a bargain with me, and I will be back one day to claim my part of it. I pull her choking, spluttering, from the river, turn her over, and let her vomit into the grass. There are other footsteps approaching, quick and frantic. I hear the child’s name on the softly-moving summer air, and it sounds like bird-call: _Cora, Cora_.

Then there’s a woman running, then kneeling beside the child, checking her for breath, for a heartbeat, sobbing with relief or sobbing with fear. Normally I would disappear back into the stream, letting myself flow into it. But this time is different. I look up at the woman holding her daughter tightly, and realise that I recognise her. I know her name.

Rosalind. The woman who paid me a debt before I came to collect.

----

The first time I saw Rosalind, she was fourteen years old. I’d known she was on her way, sensed her trudging, determined steps through the river to the waterfall. I pulled myself out of the pool below, crashing water refracting rainbows into the winter air. The water was bone cold and the ground cracked with ice, but neither would stop her.

Like so many children, Rosalind had always been drawn to the water, but she grew up playing in parks without fountains and swam in chlorinated pools under the watchful eye of an instructor, learning to keep a panicked child’s head above water or to tow a patient in a rescue tube. But this was later, and she was older. She wasn’t making her way through the river because she loved water. Her face was calligraphed with thick eyeliner, because Rosalind had always liked to make a statement, but the barely-clotted blood on her arms was not art.

Rosalind had her reasons, even if I didn’t know them.

She continued towards the waterfall, clothes heavy-laden with water. I didn’t think she’d do it, thought she’d back out just in time, but she crashed down through the water, bouncing against the rocks, and I swooped down and caught her from the waterfall before she hit the pool below.

“Do you want me to save you?” I asked over and over.

“No,” she said, thrashing against my hold.  Let me die.”

I pulled her from the water anyway, carried her all the way to the tourist car park, and left her for others to find.

That was my mistake. She still owed me but, because she had not consented to the bargain, she could fulfil it any way she chose.

----

“Her debt is mine,” Rosalind says. “But you will not take her. You will not take my daughter.”  It’s a bold statement, especially to one of the fair folk. But she’s right. We don’t ask for firstborn these days; when we did, it was almost a blessing. Better to lose a child to us than to the factories that wrench skin from bone, to the blistering wars of imperialism, to illness, or to starvation.

“You?” I say, edging scepticism onto my tongue. “What do you have to offer?”

But the woman gazing back at me could be my equal — one who can play at my own games. She’s in her thirties now, a lawyer, tying knots with a clever turn of phrase as well as I ever could. Cover-up tattoos on her arms have twisted her scars into swaying strands of seaweed, fish swimming between them. Those are arms that have pulled her daughter from water before. At work they are always beneath a blazer, but now it is summer and she has a daughter within them, tapping out a rhythm on her mother’s shoulder.

Rosalind has not been one to cheat me. She could have chosen so many ways to fulfil her debt, but she did not choose the easy ones, the perfunctory ones. She copied out poems on pages torn from her school books, wrapped them in plastic, and dropped them into rivers or off wharfs. She collected coins when she travelled and threw them into the sea as if she were skimming stones. When her grandmother died, Rosalind gave me a necklace she inherited of blue-green stones that glint even through water.

“Time,” she says. She’s gazing just to the right of my face and into the distance, a trick she’s learned to give the impression of eye contact. I don’t mind. “I can give you time.”

I let the silence hang between us and wait for her to continue. I have saved so many from the water — mostly children, some older. I once stayed away from all humans, but those I saved were those who fascinated me. I have emerged in rips at beaches, in lakes and rivers and waterfalls, to drag them from drowning.

It is not that I have no feelings for them. But I understand, as so few of them do, that to exist among others is a series of exchanges, of bargains, and I am making another one. It is true that I would prefer these mortals to not drown, not so young, but so many mortals are lost already. Death is simply part of their existence.

“I am told,” she says, when she’s realised I’m not going to respond. “I am told that your kind sometimes want us to give up something of value more than you want to receive it.

Her voice softens as I listen. She continues.

“Come and spend some time with us. Come and be with us, with me, in the places we go. Let me give you these few short years. They will barely register for you, but for me they are everything.”

I smile despite myself. She’s doing well at making it sound like she’s doing me a favour, and yet I am oddly pleased. I remember that I have kept everything she gave me, hidden behind rocks in a waterfall. I did not know why I kept it until now.

I look up at her and she’s smiling back, standing now. She has no powers of glamour, and yet it’s hard to look away.

The years of mortals are so short. But they can be so beautiful.

I pull myself out from the river and take her hand, feeling the flow of water in the soil below and the summer breeze upon my wings as Cora, now recovered, runs rings around us as we walk.

gha_language: HTML


text:
# Hashing

<https://www.hackerearth.com/practice/data-structures/hash-tables/basics-of-hash-tables/tutorial>

## Hash Function

A **hash function** is any [function](https://en.wikipedia.org/wiki/Function_(mathematics)) that can be used to map [data](https://en.wikipedia.org/wiki/Data_(computing)) of arbitrary size to data of fixed size. The values returned by a hash function are called *hash values*,*hash codes*, *digests*, or simply *hashes*. The values are used to index a fixed-size table called a *hash table*. Use of a hash function to index a hash table is called *hashing* or *scatter storage addressing*.

## Consistent Hashing

Consistent Hashing is a special kind of hashing such that when a hash table is resized, only K/n keys need to be remapped on average, where K is the number of the keys, and n is the number of slots. In contrast, in most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped because the mapping between the keys and slots are defined by a modular operation.

<https://www.akamai.com/es/es/multimedia/documents/technical-publication/consistent-hashing-and-random-trees-distributed-caching-protocols-for-relieving-hot-spots-on-the-world-wide-web-technical-publication.pdf>

[The Ultimate Guide to Consistent Hashing | Toptal®](https://www.toptal.com/big-data/consistent-hashing)

## Modular based hashing

## Applications

- **Associative arrays:** Hash tables are commonly used to implement many types of in-memory tables. They are used to implement associative arrays (arrays whose indices are arbitrary strings or other complicated objects).
- **Database indexing:** Hash tables may also be used as disk-based data structures and database indices (such as in dbm).
- **Caches:** Hash tables can be used to implement caches i.e. auxiliary data tables that are used to speed up the access to data, which is primarily stored in slower media.
- **Object representation:** Several dynamic languages, such as Perl, Python, JavaScript, and Ruby use hash tables to implement objects.
- Hash Functions are used in various algorithms to make their computing faster
- De-Duplication
  - Remove duplicates
    - Report unique visitors to web site
    - avoid duplicates in search results
    - Do not crawl same page twice
- Symbol tables in compilers
- Blocking network traffic
- Search algorithms (e.g. game tree exploration)
  - use hash table to avoid exploring any configuration (e.g. arrangement of chess pieces) more than once

## How to choose n = # of buckets (When using modulus to hash values to buckets)

1. Choose n to be prime (should be few factors, within constant factor of # of objects in table)

2. not too close to a power of 2

3. not too close to a power of 10

## The load of a hash table

## alpha = # of objects in hash table / # of buckets of hash table

Question - which hash table implementation strategy is feasible for load factors larger than 1 (Answer - Only chaining, because # of objects are greater than # of buckets, so only using linked list for collision can be used)
Therefore load factor for a hash table must be << 1, for constant time operations, since if chaining is used, than we have to use exhaustive search for getting values whose key hash to same place
So if load factor increases, we increase the number of buckets.
A super hash function, that can handle all types of data **does not exists** (for every hash function there exists a pathological data set)

## Solution to pathological data set

- Use a cryptographic hash function (e.g. SHA-2)

This works because it's infeasible to reverse engineer a pathological data set

- Use randomization

Deisgn a family of hash function, and choose randomly. Such that, for all data sets S, "almost all" functions spread out "pretty evenly"

## Universal Hashing

## Perfect Hash Function

A hash function that maps each item into a unique slot is referred to as a perfect hash function.

## Hashing Integer Data Types

- **Identity Hash Function**
- **Trivial Hash Function**
- **Folding Method**
  - The folding method for constructing the hash functions begins by dividing the item into equal-size pieces (the last piece may not be of equal size)
  - These pieces are then added together to give the resulting hash value.- **Mid-square method**
  - We first square the item, and then extract some portion of the resulting digits.
  - For example, if the item were 44, we would first compute 44^2^ = 1936
  - By extracting the middle two digits, 93, and performing the remainder step, we get 93 % 11 = 5- **Division hashing**
- **Algebraic coding**
- **Unique permuatation hashing**
- **Multiplicative hashing**
- **Fibonacci hashing**
- **Zobrist hashing**

## Hashing Variable Length Data

- **Middle and ends**
- **Character folding**
- **Word length folding**
- **Radix conversion hashing**
- **Rolling hash**

## 2-choice Hashing

Explain the 2-choice hashing algorithm for exact-match hash tables.

In the 2-choice hashing algorithm, we use two independent hash functions to compute two separate indices into the hash table. Then, we look at both indices and find whichever one has lower occupancy.We insert a new element at this index. To lookup an element, we again hash the element using both hash functons to find 2 indices, and compare the element against the elements stored at each of the 2 indices.
On what metric is the 2-choice algorithm better than the standard hashing algorithm? Why?

The 2-choice algorithm is better on the metric of likelihood of overflowing the hash tables given a certain number of inserts into the hash table (equivalently, given a certain occupancy ratio). This is because the 2-choice algorithm explicitly favors the hash table location that has lower occupancy, which means that it is less likely to overflow an already full location in the hash table. The standard hashing algorithm, on the other hand, pays no attention to how occupied a location is.
<https://en.wikipedia.org/wiki/2-choice_hashing>

## 2-left Hashing

A [dictionary](https://xlinux.nist.gov/dads/HTML/dictionary.html) implemented with two [hash tables](https://xlinux.nist.gov/dads/HTML/hashtab.html) of equal size, T1and T2, and two different [hash functions](https://xlinux.nist.gov/dads/HTML/hash.html), h1and h2. A new [key](https://xlinux.nist.gov/dads/HTML/key.html) is put in table 2 only if there are fewer (colliding) keys at T2 [h2(key)] than at T1 [h1(key)], otherwise it is put in table 1. With n keys and two tables of size n/2, the most collisions is 0.69... log2ln n +[O](https://xlinux.nist.gov/dads/HTML/bigOnotation.html)(1) with high probability.

## Hash Table Implementations

- Using Array
- Alternatively, we can implement the hash table with a binary search tree. We can then guarantee an 0(log n) lookup time, since we can keep the tree balanced. Additionally, we may use less space, since a large array no longer needs to be allocated in the very beginning.

## Locality Sensitive Hashing (LSH)

In computer science, locality-sensitive hashing(LSH) is an algorithmic technique that hashes similar input items into the same "buckets" with high probability.(The number of buckets are much smaller than the universe of possible input items.)Since similar items end up in the same buckets, this technique can be used for [data clustering](https://en.wikipedia.org/wiki/Cluster_analysis) and [nearest neighbor search](https://en.wikipedia.org/wiki/Nearest_neighbor_search). It differs from [conventional hashing techniques](https://en.wikipedia.org/wiki/Hash_function) in that hash collisions are maximized, not minimized. Alternatively, the technique can be seen as a way to [reduce the dimensionality](https://en.wikipedia.org/wiki/Dimension_reduction) of high-dimensional data; high-dimensional input items can be reduced to low-dimensional versions while preserving relative distances between items.
Hashing-based approximate [nearest neighbor search](https://en.wikipedia.org/wiki/Nearest_neighbor_search) algorithms generally use one of two main categories of hashing methods: either data-independent methods, such as locality-sensitive hashing (LSH); or data-dependent methods, such as [Locality-preserving hashing](https://en.wikipedia.org/wiki/Locality-preserving_hashing)(LPH).
<https://en.wikipedia.org/wiki/Locality-sensitive_hashing>

<https://towardsdatascience.com/understanding-locality-sensitive-hashing-49f6d1f6134>

## Further Reading

- Birthday Paradox
- Pigeonhole Principle
<https://en.wikipedia.org/wiki/Hash_function>


text:
[![Build][build-badge]][build-url]
[![Issues][issues-badge]][issues-url]
[![Gitter][gitter-badge]][gitter-url]

PCS Authentication and Authorization Overview
=============================================

This service allows to manage the users authorized to access Azure IoT
Solutions. Users management can be done using any identity service
provider supporting OpenId Connect.

Dependencies
============

The service depends on:

* [Azure Active Directory][aad-url] used to store users and providing
  the certificates to validate JWT tokens signature. Any identity
  provider supporting OpenId Connect should work though.
* Configuration settings to define the trusted Issuer and expected
  Audience.

How to use the microservice
===========================

## Quickstart - Running the service with Docker

1. Create an instance of [Azure Active Directory][aad-url] or simply
   reuse the instance coming with your Azure subscription
1. [Register][aad-register-app] an application in AAD
1. Get the Application ID and Issuer URL and store them in the
   [service configuration](WebService/appsettings.ini).
1. [Install Docker][docker-install-url]
1. Start the Auth service using docker compose:
   ```
   cd scripts
   cd docker
   run
   ```
1. Use an HTTP client such as [Postman][postman-url], to exercise the
   RESTful API.

## Running the service with Visual Studio

1. Install any edition of [Visual Studio 2017][vs-install-url] or Visual
   Studio for Mac. When installing check ".NET Core" workload. If you
   already have Visual Studio installed, then ensure you have
   [.NET Core Tools for Visual Studio 2017][dotnetcore-tools-url]
   installed (Windows only).
1. Create an instance of [Azure Active Directory][aad-url] or simply
   reuse the instance coming with your Azure subscription
1. Get the Application ID and Issuer URL and store them in the
   [service configuration](WebService/appsettings.ini).
1. Open the solution in Visual Studio
1. In Visual Studio, start the WebService project
1. Use an HTTP client such as [Postman][postman-url], to exercise the
   RESTful API.

## Project Structure

The solution contains the following projects and folders:

* **WebService**: ASP.NET Web API exposing a RESTful API for Authentication
  functionality, e.g. show the current user profile.
* **Services**: Library containing common business logic for interacting with
  Azure Active Directory.
* **WebService.Test**: Unit tests for the ASP.NET Web API project.
* **Services.Test**: Unit tests for the Services library.
* **scripts**: a folder containing scripts from the command line console,
  to build and run the solution, and other frequent tasks.

## Build and Run from the command line

The [scripts](scripts) folder contains scripts for many frequent tasks:

* `build`: compile all the projects and run the tests.
* `compile`: compile all the projects.
* `run`: compile the projects and run the service. This will prompt for
  elevated privileges in Windows to run the web service.

## Building a customized Docker image

The `scripts` folder includes a [docker](scripts/docker) subfolder with the
scripts required to package the service into a Docker image:

* `Dockerfile`: Docker image specifications
* `build`: build a Docker image and store the image in the local registry
* `run`: run the Docker container from the image stored in the local registry
* `content`: a folder with files copied into the image, including the entry
  point script

## Configuration and Environment variables

The service configuration is stored using ASP.NET Core configuration
adapters, in [appsettings.ini](WebService/appsettings.ini) and
[appsettings.ini](SimulationAgent/appsettings.ini). The INI format allows to
store values in a readable format, with comments. The application also
supports references to environment variables, which is used to import
credentials and networking details.

The configuration files in the repository reference some environment
variables that need to be created at least once. Depending on your OS and
the IDE, there are several ways to manage environment variables:

* Windows: the variables can be set [in the system][windows-envvars-howto-url]
  as a one time only task. The
  [env-vars-setup.cmd](scripts/env-vars-setup.cmd) script included needs to
  be prepared and executed just once. The settings will persist across
  terminal sessions and reboots.
* Visual Studio: the variables can be set in the projects's settings, both
  WebService and SimulationAgent, under Project Propertie -> Configuration
  Properties -> Environment
* For Linux and OSX environments, the [env-vars-setup](scripts/env-vars-setup)
  script needs to be executed every time a new console is opened.
  Depending on the OS and terminal, there are ways to persist values
  globally, for more information these pages should help:
  * https://stackoverflow.com/questions/13046624/how-to-permanently-export-a-variable-in-linux
  * https://stackoverflow.com/questions/135688/setting-environment-variables-in-os-x
  * https://help.ubuntu.com/community/EnvironmentVariables

Contributing to the solution
============================

Please follow our [contribution guidelines](CONTRIBUTING.md).  We love PRs too.

Troubleshooting
===============

{TODO}

Feedback
==========

Please enter issues, bugs, or suggestions as GitHub Issues here:
https://github.com/Azure/pcs-auth-dotnet/issues.

[build-badge]: https://img.shields.io/travis/Azure/pcs-auth-dotnet.svg
[build-url]: https://travis-ci.org/Azure/pcs-auth-dotnet
[issues-badge]: https://img.shields.io/github/issues/azure/pcs-auth-dotnet.svg
[issues-url]: https://github.com/azure/pcs-auth-dotnet/issues
[gitter-badge]: https://img.shields.io/gitter/room/azure/iot-solutions.js.svg
[gitter-url]: https://gitter.im/azure/iot-solutions

[aad-url]: https://azure.microsoft.com/services/active-directory
[aad-register-app]: https://docs.microsoft.com/azure/active-directory/develop/active-directory-integrating-applications
[docker-install-url]: https://docs.docker.com/engine/installation/
[postman-url]: https://www.getpostman.com
[vs-install-url]: https://www.visualstudio.com/downloads
[dotnetcore-tools-url]: https://www.microsoft.com/net/core#windowsvs2017
[windows-envvars-howto-url]: https://superuser.com/questions/949560/how-do-i-set-system-environment-variables-in-windows-10

gha_language: C#


text:
## hmx-protocol

Library for working with HMX messages and blockchain

### private message

```
GET /jaMHiVNo_MpTmVUD

HTTP/1.1 200 OK
Date: Tue, 15 Nov 1994 08:12:31 GMT
Content-Length: 3473
Content-Type: application/hmx
Link: </eoiFDR1nq2libsxZvF7yiYUrfjVVyvl5iICffStv2kf>; rel="prev", </2h-92i_KZEb05u2_48JYCSnSbRZd_Hio32ZaJGJH3Vc>; rel="next"
HMX: 1.0/private
HMX-Key: igkopauD8k0GzuG8b0LLgfKMtpEOswSuqwW1OIHFpfv (hash of a peer key)
HMX-Digest: BQ53nHce1X-0XTlGgSIbd-kS5hF2rrRNHwtm2lpde3Y (hash of encrypted body)
HMX-Id: jaMHiVNo_MpTmVUDQH-VDDKR6x00r1Ejf1AKx-W98Ak (hash of public HMX* headers, unique id)

(encrypted body)
```

#### unencrypted body

```
HMX-File-Location: https://mybucket.s3.amazonaws.com/myobject.hmx
HMX-File-Name: encrypted.tar.gz
HMX-File-Digest: kkO7HiRigl-RUfNRNXqv-1y4qkBJCx0e-1hKleYjqsE
HMX-Options: markdown
HMX-From<EMAIL_ADDRESS>"6f:2a:cd:0b:02:19:35:ab:62:65:58:c7:91:be:32:64"
HMX-To<EMAIL_ADDRESS>"6f:2a:cd:0b:02:19:35:ab:62:65:58:c7:91:be:32:64"
HMX-Subject: Testing HMX
HMX-Time:<PHONE_NUMBER>998
HMX-Signature: BQ53nHce1X-0XTlGgSIbd-kS5hF2rrRNHwtm2lpde3Y (signature of all public+private HMX* headers)

(plain message here)
```

### public message board

- posts are limited in byte size
- content-type MUST be `text/plain` or `text/x-markdown`.
- message MUST be signed by a registered public key
- group/thread is chosen by client but may be whitelisted/blackslisted by server

```
GET /public/comp.hackers/jaMHiVNo_MpTmVUD

HTTP/1.1 200 OK
Date: Tue, 15 Nov 1994 08:12:31 GMT
Content-Type: text/x-markdown
Content-Length: 3473
Link: </eoiFDR1nq2libsxZvF7yiYUrfjVVyvl5iICffStv2kf>; rel="prev", </2h-92i_KZEb05u2_48JYCSnSbRZd_Hio32ZaJGJH3Vc>; rel="next"
HMX: 1.0/public
HMX-Digest: nYmlRrhDTkc2SlDokGktVl9xucIUNlb1BChMUvXp1oy (hash of body)
HMX-From<EMAIL_ADDRESS>"6f:2a:cd:0b:02:19:35:ab:62:65:58:c7:91:be:32:64"
HMX-To<EMAIL_ADDRESS>HMX-Options: markdown, replies (allow subsequent replies to this post)
HMX-Thread: igkopauD8k0GzuG8b0LLgfKMtpEOswSuqwW1OIHFpfv (id of a previous post, for replies)
HMX-Subject: Testing HMX
HMX-Agent: node-hmx/1.0; https://h-mx.org/agent
HMX-Time:<PHONE_NUMBER>998
HMX-Id: jaMHiVNo_MpTmVUDQH-VDDKR6x00r1Ejf1AKx-W98Ak (hash of HMX* headers, unique id)
HMX-Signature: oqQz5gwGap7axNtsQsceTVbRtQj74wj0zptal21BpYn (signature of id)

(body)
```

- - -

### Developed by [Terra Eclipse](http://www.terraeclipse.com)
Terra Eclipse, Inc. is a nationally recognized political technology and
strategy firm located in Aptos, CA and Washington, D.C.