text:
# DataStreams

[![DataStreams](http://pkg.julialang.org/badges/DataStreams_0.4.svg)](http://pkg.julialang.org/?pkg=DataStreams&ver=0.4)
[![DataStreams](http://pkg.julialang.org/badges/DataStreams_0.5.svg)](http://pkg.julialang.org/?pkg=DataStreams)

Linux and OS X: [![Build Status](https://travis-ci.org/JuliaData/DataStreams.jl.svg?branch=master)](https://travis-ci.org/JuliaData/DataStreams.jl)

Windows: [![Build Status](https://ci.appveyor.com/api/projects/status/github/JuliaData/DataStreams.jl?branch=master&svg=true)](https://ci.appveyor.com/project/JuliaData/datastreams-jl/branch/master)

[![codecov.io](http://codecov.io/github/JuliaData/DataStreams.jl/coverage.svg?branch=master)](http://codecov.io/github/JuliaData/DataStreams.jl?branch=master)

The `DataStreams.jl` packages defines a data processing framework based on Sources, Sinks, and the `Data.stream!` function.

`DataStreams` defines the common infrastructure leveraged by individual packages to create systems of various
data sources and sinks that talk to each other in a unified, consistent way.

The workflow enabled by the `DataStreams` framework involves:
 * constructing new `Source` types to allow streaming data from files, databases, and other "sources" of data.
 * `Data.stream!` those datasets to newly created or existing `Sink` types
 * convert `Sink` types that have received data into new `Source` types
 * continue to `Data.stream!` from `Source`s to `Sink`s

The typical approach for a new package to "satisfy" the DataStreams interface is to:
 * Define a `Source` type that wraps a "true data source" (i.e. a file, database table/query, etc.) and fulfills the `Source` interface (see `?Data.Source`)
 * Define a `Sink` type that can create or write data to a "true data source" and fulfills the `Sink` interface (see `?Data.Sink`)
 * Define appropriate `Data.stream!(::Source, ::Sink)` methods as needed between various combinations of Sources and Sinks;
   i.e. define `Data.stream!(::NewPackage.Source, ::CSV.Sink)` and `Data.stream!(::CSV.Source, ::NewPackage.Sink)`

#### Sources
A `Data.Source` type holds data that can be read/queried/parsed/viewed/streamed; i.e. a "true data source"
To clarify, there are two distinct types of "source":
  1) the "true data source", which would be the file, database, API, structure, etc; i.e. the actual data
  2) the `Data.Source` julia object that wraps a "true source" and provides the `DataStreams` interface

`Source` types have two different types of constructors:
  1) "independent constructors" that wrap "true data sources"
  and 2) "sink constructors" where a `Data.Sink` object that has received data is turned into a `Source`

`Source`s also have a, currently implicit, notion of state:
  * `BEGINNING`: a `Source` is in this state immediately after being constructed and is ready to be used; i.e. ready to read/parse/query/stream data from it; any necessary setup, construction, or query execution has taken place during construction
  * `READING`: the ingestion of data from this `Source` has started and has not finished yet
  * `DONE`: the ingestion process has exhausted all data expected from this `Source` instance

The `Data.Source` interface includes the following:
 * `Data.schema(::Data.Source) => Data.Schema`; typically the `Source` type will store the `Data.Schema` directly, but this isn't strictly required
 * `Data.reset!(::Data.Source)`; used to reset a `Source` type from `READING` or `DONE` to the `BEGINNING` state, ready to be read from again
 * `Data.isdone(::Data.Source)` => Bool; indicates whether the `Source` type is in the `DONE` state; i.e. all data has been exhausted from this source

#### Sinks
A `Data.Sink` type represents a data destination; i.e. a "true data source" such as a database, file, API endpoint, etc.

There are two broad types of `Sink`s:
  1) "new sinks": an independent `Sink` constructor creates a *new* "true data source" that can be streamed to
  2) "existing sinks": the `Sink` wraps an already existing "true data source" (or `Source` object that wraps a "true data source").
    Upon construction of these Sinks, there is no new creation of "true data source"s; the "true data source" is simply wrapped to replace or append to

`Sink`s also have notions of state:
  * `BEGINNING`: the `Sink` is freshly constructed and ready to stream data to; this includes initial metadata like column headers
  * `WRITING`: data has been streamed to the `Sink`, but is still open to receive more data
  * `DONE`: the `Sink` has been closed and can no longer receive data

The `Data.Sink` interface includes the following:
 * `Data.schema(::Data.Sink) => Data.Schema`; typically the `Sink` type will store the `Data.Schema` directly, but this isn't strictly required
 * `Data.reset!(::Data.Sink;append::Bool=false)`; used to reset a `Sink` type from `WRITING` or `DONE` to the `BEGINNING` state, ready to receive data again
 * `Data.isdone(::Data.Sink)`; indicates whether the `Sink` type is in the `DONE` state; i.e. if it can receive any more data


text:
# Skillcrush HTML/CSS Milestone project

1. [Description](##Description)
2. [Credits](##Credits)
3. [Licenses](##Licenses)

## Description

This is a Milestone project for Skillcrush classes 101 and 105, which pulls together coursework covering HTML, CSS, Git, and GitHub, culminating in a portfolio project.

## Credits

### Original Design Comps

Original Design Comp for main page was from Skillcrush Milestone class.
Original Design Comp for Jubilee Austen page and Rogue Pickings page were from SkillCrush 101 class. The concept of the manifesto challenge was also taken from the SkillCrush 101 class.

### Images

Images were provided by Skillcrush, but many images were likely from [Unsplash](https://unsplash.com/).

Further detail will be provided as updates are made.

## Licenses

My code for this project is licensed under the MIT license.

The [HTML5 boilerplate](https://html5boilerplate.com/) v.6.1.0 was used in this project. The license for the boilerplate can be found in [LICENSE-boilerplate.txt](LICENSE-boilerplate.txt) which is also referred to as an MIT license.

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: CSS

gha_license_id: MIT

license_type: permissive


text:
# bdo
bdo utilities

Built on: Python3 and Docker (alpine)

Maintained by: Chris Lee<EMAIL_ADDRESS>
## Getting Started
- Additional Python3 dependencies can be added to requirements.txt

- Tests are located in ./tests

- To run the docker container with the basic requirements, dependencies, and the package installed:
    ```bash
    $ docker-compose up --build
    ```


text:
# The safegcd implementation in libsecp256k1 explained

This document explains the modular inverse and Jacobi symbol implementations in the `src/modinv*.h` files.
It is based on the paper
["Fast constant-time gcd computation and modular inversion"](https://gcd.cr.yp.to/papers.html#safegcd)
by Daniel J. Bernstein and Bo-Yin Yang. The references below are for the Date: 2019.04.13 version.

The actual implementation is in C of course, but for demonstration purposes Python3 is used here.
Most implementation aspects and optimizations are explained, except those that depend on the specific
number representation used in the C code.

## 1. Computing the Greatest Common Divisor (GCD) using divsteps

The algorithm from the paper (section 11), at a very high level, is this:

```python
def gcd(f, g):
    """Compute the GCD of an odd integer f and another integer g."""
    assert f & 1  # require f to be odd
    delta = 1     # additional state variable
    while g != 0:
        assert f & 1  # f will be odd in every iteration
        if delta > 0 and g & 1:
            delta, f, g = 1 - delta, g, (g - f) // 2
        elif g & 1:
            delta, f, g = 1 + delta, f, (g + f) // 2
        else:
            delta, f, g = 1 + delta, f, (g    ) // 2
    return abs(f)
```

It computes the greatest common divisor of an odd integer *f* and any integer *g*. Its inner loop
keeps rewriting the variables *f* and *g* alongside a state variable *δ* that starts at *1*, until
*g=0* is reached. At that point, *|f|* gives the GCD. Each of the transitions in the loop is called a
"division step" (referred to as divstep in what follows).

For example, *gcd(21, 14)* would be computed as:
- Start with *δ=1 f=21 g=14*
- Take the third branch: *δ=2 f=21 g=7*
- Take the first branch: *δ=-1 f=7 g=-7*
- Take the second branch: *δ=0 f=7 g=0*
- The answer *|f| = 7*.

Why it works:
- Divsteps can be decomposed into two steps (see paragraph 8.2 in the paper):
  - (a) If *g* is odd, replace *(f,g)* with *(g,g-f)* or (f,g+f), resulting in an even *g*.
  - (b) Replace *(f,g)* with *(f,g/2)* (where *g* is guaranteed to be even).
- Neither of those two operations change the GCD:
  - For (a), assume *gcd(f,g)=c*, then it must be the case that *f=a c* and *g=b c* for some integers *a*
    and *b*. As *(g,g-f)=(b c,(b-a)c)* and *(f,f+g)=(a c,(a+b)c)*, the result clearly still has
    common factor *c*. Reasoning in the other direction shows that no common factor can be added by
    doing so either.
  - For (b), we know that *f* is odd, so *gcd(f,g)* clearly has no factor *2*, and we can remove
    it from *g*.
- The algorithm will eventually converge to *g=0*. This is proven in the paper (see theorem G.3).
- It follows that eventually we find a final value *f'* for which *gcd(f,g) = gcd(f',0)*. As the
  gcd of *f'* and *0* is *|f'|* by definition, that is our answer.

Compared to more [traditional GCD algorithms](https://en.wikipedia.org/wiki/Euclidean_algorithm), this one has the property of only ever looking at
the low-order bits of the variables to decide the next steps, and being easy to make
constant-time (in more low-level languages than Python). The *δ* parameter is necessary to
guide the algorithm towards shrinking the numbers' magnitudes without explicitly needing to look
at high order bits.

Properties that will become important later:
- Performing more divsteps than needed is not a problem, as *f* does not change anymore after *g=0*.
- Only even numbers are divided by *2*. This means that when reasoning about it algebraically we
  do not need to worry about rounding.
- At every point during the algorithm's execution the next *N* steps only depend on the bottom *N*
  bits of *f* and *g*, and on *δ*.

## 2. From GCDs to modular inverses

We want an algorithm to compute the inverse *a* of *x* modulo *M*, i.e. the number a such that *a x=1
mod M*. This inverse only exists if the GCD of *x* and *M* is *1*, but that is always the case if *M* is
prime and *0 < x < M*. In what follows, assume that the modular inverse exists.
It turns out this inverse can be computed as a side effect of computing the GCD by keeping track
of how the internal variables can be written as linear combinations of the inputs at every step
(see the [extended Euclidean algorithm](https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm)).
Since the GCD is *1*, such an algorithm will compute numbers *a* and *b* such that a x + b M = 1*.
Taking that expression *mod M* gives *a x mod M = 1*, and we see that *a* is the modular inverse of *x
mod M*.

A similar approach can be used to calculate modular inverses using the divsteps-based GCD
algorithm shown above, if the modulus *M* is odd. To do so, compute *gcd(f=M,g=x)*, while keeping
track of extra variables *d* and *e*, for which at every step *d = f/x (mod M)* and *e = g/x (mod M)*.
*f/x* here means the number which multiplied with *x* gives *f mod M*. As *f* and *g* are initialized to *M*
and *x* respectively, *d* and *e* just start off being *0* (*M/x mod M = 0/x mod M = 0*) and *1* (*x/x mod M
= 1*).

```python
def div2(M, x):
    """Helper routine to compute x/2 mod M (where M is odd)."""
    assert M & 1
    if x & 1: # If x is odd, make it even by adding M.
        x += M
    # x must be even now, so a clean division by 2 is possible.
    return x // 2

def modinv(M, x):
    """Compute the inverse of x mod M (given that it exists, and M is odd)."""
    assert M & 1
    delta, f, g, d, e = 1, M, x, 0, 1
    while g != 0:
        # Note that while division by two for f and g is only ever done on even inputs, this is
        # not true for d and e, so we need the div2 helper function.
        if delta > 0 and g & 1:
            delta, f, g, d, e = 1 - delta, g, (g - f) // 2, e, div2(M, e - d)
        elif g & 1:
            delta, f, g, d, e = 1 + delta, f, (g + f) // 2, d, div2(M, e + d)
        else:
            delta, f, g, d, e = 1 + delta, f, (g    ) // 2, d, div2(M, e    )
        # Verify that the invariants d=f/x mod M, e=g/x mod M are maintained.
        assert f % M == (d * x) % M
        assert g % M == (e * x) % M
    assert f == 1 or f == -1  # |f| is the GCD, it must be 1
    # Because of invariant d = f/x (mod M), 1/x = d/f (mod M). As |f|=1, d/f = d*f.
    return (d * f) % M
```

Also note that this approach to track *d* and *e* throughout the computation to determine the inverse
is different from the paper. There (see paragraph 12.1 in the paper) a transition matrix for the
entire computation is determined (see section 3 below) and the inverse is computed from that.
The approach here avoids the need for 2x2 matrix multiplications of various sizes, and appears to
be faster at the level of optimization we're able to do in C.

## 3. Batching multiple divsteps

Every divstep can be expressed as a matrix multiplication, applying a transition matrix *(1/2 t)*
to both vectors *[f, g]* and *[d, e]* (see paragraph 8.1 in the paper):

```
  t = [ u,  v ]
      [ q,  r ]

  [ out_f ] = (1/2 * t) * [ in_f ]
  [ out_g ] =             [ in_g ]

  [ out_d ] = (1/2 * t) * [ in_d ]  (mod M)
  [ out_e ]               [ in_e ]
```

where *(u, v, q, r)* is *(0, 2, -1, 1)*, *(2, 0, 1, 1)*, or *(2, 0, 0, 1)*, depending on which branch is
taken. As above, the resulting *f* and *g* are always integers.

Performing multiple divsteps corresponds to a multiplication with the product of all the
individual divsteps' transition matrices. As each transition matrix consists of integers
divided by *2*, the product of these matrices will consist of integers divided by *2N* (see also
theorem 9.2 in the paper). These divisions are expensive when updating *d* and *e*, so we delay
them: we compute the integer coefficients of the combined transition matrix scaled by *2N*, and
do one division by *2N* as a final step:

```python
def divsteps_n_matrix(delta, f, g):
    """Compute delta and transition matrix t after N divsteps (multiplied by 2^N)."""
    u, v, q, r = 1, 0, 0, 1 # start with identity matrix
    for _ in range(N):
        if delta > 0 and g & 1:
            delta, f, g, u, v, q, r = 1 - delta, g, (g - f) // 2, 2*q, 2*r, q-u, r-v
        elif g & 1:
            delta, f, g, u, v, q, r = 1 + delta, f, (g + f) // 2, 2*u, 2*v, q+u, r+v
        else:
            delta, f, g, u, v, q, r = 1 + delta, f, (g    ) // 2, 2*u, 2*v, q  , r
    return delta, (u, v, q, r)
```

As the branches in the divsteps are completely determined by the bottom *N* bits of *f* and *g*, this
function to compute the transition matrix only needs to see those bottom bits. Furthermore all
intermediate results and outputs fit in *(N+1)*-bit numbers (unsigned for *f* and *g*; signed for *u*, *v*,
*q*, and *r*) (see also paragraph 8.3 in the paper). This means that an implementation using 64-bit
integers could set *N=62* and compute the full transition matrix for 62 steps at once without any
big integer arithmetic at all. This is the reason why this algorithm is efficient: it only needs
to update the full-size *f*, *g*, *d*, and *e* numbers once every *N* steps.

We still need functions to compute:

```
  [ out_f ] = (1/2^N * [ u,  v ]) * [ in_f ]
  [ out_g ]   (        [ q,  r ])   [ in_g ]

  [ out_d ] = (1/2^N * [ u,  v ]) * [ in_d ]  (mod M)
  [ out_e ]   (        [ q,  r ])   [ in_e ]
```

Because the divsteps transformation only ever divides even numbers by two, the result of *t [f,g]* is always even. When *t* is a composition of *N* divsteps, it follows that the resulting *f*
and *g* will be multiple of *2N*, and division by *2N* is simply shifting them down:

```python
def update_fg(f, g, t):
    """Multiply matrix t/2^N with [f, g]."""
    u, v, q, r = t
    cf, cg = u*f + v*g, q*f + r*g
    # (t / 2^N) should cleanly apply to [f,g] so the result of t*[f,g] should have N zero
    # bottom bits.
    assert cf % 2**N == 0
    assert cg % 2**N == 0
    return cf >> N, cg >> N
```

The same is not true for *d* and *e*, and we need an equivalent of the `div2` function for division by *2N mod M*.
This is easy if we have precomputed *1/M mod 2N* (which always exists for odd *M*):

```python
def div2n(M, Mi, x):
    """Compute x/2^N mod M, given Mi = 1/M mod 2^N."""
    assert (M * Mi) % 2**N == 1
    # Find a factor m such that m*M has the same bottom N bits as x. We want:
    #     (m * M) mod 2^N = x mod 2^N
    # <=> m mod 2^N = (x / M) mod 2^N
    # <=> m mod 2^N = (x * Mi) mod 2^N
    m = (Mi * x) % 2**N
    # Subtract that multiple from x, cancelling its bottom N bits.
    x -= m * M
    # Now a clean division by 2^N is possible.
    assert x % 2**N == 0
    return (x >> N) % M

def update_de(d, e, t, M, Mi):
    """Multiply matrix t/2^N with [d, e], modulo M."""
    u, v, q, r = t
    cd, ce = u*d + v*e, q*d + r*e
    return div2n(M, Mi, cd), div2n(M, Mi, ce)
```

With all of those, we can write a version of `modinv` that performs *N* divsteps at once:

```python3
def modinv(M, Mi, x):
    """Compute the modular inverse of x mod M, given Mi=1/M mod 2^N."""
    assert M & 1
    delta, f, g, d, e = 1, M, x, 0, 1
    while g != 0:
        # Compute the delta and transition matrix t for the next N divsteps (this only needs
        # (N+1)-bit signed integer arithmetic).
        delta, t = divsteps_n_matrix(delta, f % 2**N, g % 2**N)
        # Apply the transition matrix t to [f, g]:
        f, g = update_fg(f, g, t)
        # Apply the transition matrix t to [d, e]:
        d, e = update_de(d, e, t, M, Mi)
    return (d * f) % M
```

This means that in practice we'll always perform a multiple of *N* divsteps. This is not a problem
because once *g=0*, further divsteps do not affect *f*, *g*, *d*, or *e* anymore (only *δ* keeps
increasing). For variable time code such excess iterations will be mostly optimized away in later
sections.

## 4. Avoiding modulus operations

So far, there are two places where we compute a remainder of big numbers modulo *M*: at the end of
`div2n` in every `update_de`, and at the very end of `modinv` after potentially negating *d* due to the
sign of *f*. These are relatively expensive operations when done generically.

To deal with the modulus operation in `div2n`, we simply stop requiring *d* and *e* to be in range
*[0,M)* all the time. Let's start by inlining `div2n` into `update_de`, and dropping the modulus
operation at the end:

```python
def update_de(d, e, t, M, Mi):
    """Multiply matrix t/2^N with [d, e] mod M, given Mi=1/M mod 2^N."""
    u, v, q, r = t
    cd, ce = u*d + v*e, q*d + r*e
    # Cancel out bottom N bits of cd and ce.
    md = -((Mi * cd) % 2**N)
    me = -((Mi * ce) % 2**N)
    cd += md * M
    ce += me * M
    # And cleanly divide by 2**N.
    return cd >> N, ce >> N
```

Let's look at bounds on the ranges of these numbers. It can be shown that *|u|+|v|* and *|q|+|r|*
never exceed *2N* (see paragraph 8.3 in the paper), and thus a multiplication with *t* will have
outputs whose absolute values are at most *2N* times the maximum absolute input value. In case the
inputs *d* and *e* are in *(-M,M)*, which is certainly true for the initial values *d=0* and *e=1* assuming
*M > 1*, the multiplication results in numbers in range *(-2NM,2NM)*. Subtracting less than *2N*
times *M* to cancel out *N* bits brings that up to *(-2N+1M,2NM)*, and
dividing by *2N* at the end takes it to *(-2M,M)*. Another application of `update_de` would take that
to *(-3M,2M)*, and so forth. This progressive expansion of the variables' ranges can be
counteracted by incrementing *d* and *e* by *M* whenever they're negative:

```python
    ...
    if d < 0:
        d += M
    if e < 0:
        e += M
    cd, ce = u*d + v*e, q*d + r*e
    # Cancel out bottom N bits of cd and ce.
    ...
```

With inputs in *(-2M,M)*, they will first be shifted into range *(-M,M)*, which means that the
output will again be in *(-2M,M)*, and this remains the case regardless of how many `update_de`
invocations there are. In what follows, we will try to make this more efficient.

Note that increasing *d* by *M* is equal to incrementing *cd* by *u M* and *ce* by *q M*. Similarly,
increasing *e* by *M* is equal to incrementing *cd* by *v M* and *ce* by *r M*. So we could instead write:

```python
    ...
    cd, ce = u*d + v*e, q*d + r*e
    # Perform the equivalent of incrementing d, e by M when they're negative.
    if d < 0:
        cd += u*M
        ce += q*M
    if e < 0:
        cd += v*M
        ce += r*M
    # Cancel out bottom N bits of cd and ce.
    md = -((Mi * cd) % 2**N)
    me = -((Mi * ce) % 2**N)
    cd += md * M
    ce += me * M
    ...
```

Now note that we have two steps of corrections to *cd* and *ce* that add multiples of *M*: this
increment, and the decrement that cancels out bottom bits. The second one depends on the first
one, but they can still be efficiently combined by only computing the bottom bits of *cd* and *ce*
at first, and using that to compute the final *md*, *me* values:

```python
def update_de(d, e, t, M, Mi):
    """Multiply matrix t/2^N with [d, e], modulo M."""
    u, v, q, r = t
    md, me = 0, 0
    # Compute what multiples of M to add to cd and ce.
    if d < 0:
        md += u
        me += q
    if e < 0:
        md += v
        me += r
    # Compute bottom N bits of t*[d,e] + M*[md,me].
    cd, ce = (u*d + v*e + md*M) % 2**N, (q*d + r*e + me*M) % 2**N
    # Correct md and me such that the bottom N bits of t*[d,e] + M*[md,me] are zero.
    md -= (Mi * cd) % 2**N
    me -= (Mi * ce) % 2**N
    # Do the full computation.
    cd, ce = u*d + v*e + md*M, q*d + r*e + me*M
    # And cleanly divide by 2**N.
    return cd >> N, ce >> N
```

One last optimization: we can avoid the *md M* and *me M* multiplications in the bottom bits of *cd*
and *ce* by moving them to the *md* and *me* correction:

```python
    ...
    # Compute bottom N bits of t*[d,e].
    cd, ce = (u*d + v*e) % 2**N, (q*d + r*e) % 2**N
    # Correct md and me such that the bottom N bits of t*[d,e]+M*[md,me] are zero.
    # Note that this is not the same as {md = (-Mi * cd) % 2**N} etc. That would also result in N
    # zero bottom bits, but isn't guaranteed to be a reduction of [0,2^N) compared to the
    # previous md and me values, and thus would violate our bounds analysis.
    md -= (Mi*cd + md) % 2**N
    me -= (Mi*ce + me) % 2**N
    ...
```

The resulting function takes *d* and *e* in range *(-2M,M)* as inputs, and outputs values in the same
range. That also means that the *d* value at the end of `modinv` will be in that range, while we want
a result in *[0,M)*. To do that, we need a normalization function. It's easy to integrate the
conditional negation of *d* (based on the sign of *f*) into it as well:

```python
def normalize(sign, v, M):
    """Compute sign*v mod M, where v is in range (-2*M,M); output in [0,M)."""
    assert sign == 1 or sign == -1
    # v in (-2*M,M)
    if v < 0:
        v += M
    # v in (-M,M). Now multiply v with sign (which can only be 1 or -1).
    if sign == -1:
        v = -v
    # v in (-M,M)
    if v < 0:
        v += M
    # v in [0,M)
    return v
```

And calling it in `modinv` is simply:

```python
   ...
   return normalize(f, d, M)
```

## 5. Constant-time operation

The primary selling point of the algorithm is fast constant-time operation. What code flow still
depends on the input data so far?

- the number of iterations of the while *g ≠ 0* loop in `modinv`
- the branches inside `divsteps_n_matrix`
- the sign checks in `update_de`
- the sign checks in `normalize`

To make the while loop in `modinv` constant time it can be replaced with a constant number of
iterations. The paper proves (Theorem 11.2) that *741* divsteps are sufficient for any *256*-bit
inputs, and [safegcd-bounds](https://github.com/sipa/safegcd-bounds) shows that the slightly better bound *724* is
sufficient even. Given that every loop iteration performs *N* divsteps, it will run a total of
*⌈724/N⌉* times.

To deal with the branches in `divsteps_n_matrix` we will replace them with constant-time bitwise
operations (and hope the C compiler isn't smart enough to turn them back into branches; see
`ctime_tests.c` for automated tests that this isn't the case). To do so, observe that a
divstep can be written instead as (compare to the inner loop of `gcd` in section 1).

```python
    x = -f if delta > 0 else f         # set x equal to (input) -f or f
    if g & 1:
        g += x                         # set g to (input) g-f or g+f
        if delta > 0:
            delta = -delta
            f += g                     # set f to (input) g (note that g was set to g-f before)
    delta += 1
    g >>= 1
```

To convert the above to bitwise operations, we rely on a trick to negate conditionally: per the
definition of negative numbers in two's complement, (*-v == ~v + 1*) holds for every number *v*. As
*-1* in two's complement is all *1* bits, bitflipping can be expressed as xor with *-1*. It follows
that *-v == (v ^ -1) - (-1)*. Thus, if we have a variable *c* that takes on values *0* or *-1*, then
*(v ^ c) - c* is *v* if *c=0* and *-v* if *c=-1*.

Using this we can write:

```python
    x = -f if delta > 0 else f
```

in constant-time form as:

```python
    c1 = (-delta) >> 63
    # Conditionally negate f based on c1:
    x = (f ^ c1) - c1
```

To use that trick, we need a helper mask variable *c1* that resolves the condition *δ>0* to *-1*
(if true) or *0* (if false). We compute *c1* using right shifting, which is equivalent to dividing by
the specified power of *2* and rounding down (in Python, and also in C under the assumption of a typical two's complement system; see
`assumptions.h` for tests that this is the case). Right shifting by *63* thus maps all
numbers in range *[-263,0)* to *-1*, and numbers in range *[0,263)* to *0*.

Using the facts that *x&0=0* and *x&(-1)=x* (on two's complement systems again), we can write:

```python
    if g & 1:
        g += x
```

as:

```python
    # Compute c2=0 if g is even and c2=-1 if g is odd.
    c2 = -(g & 1)
    # This masks out x if g is even, and leaves x be if g is odd.
    g += x & c2
```

Using the conditional negation trick again we can write:

```python
    if g & 1:
        if delta > 0:
            delta = -delta
```

as:

```python
    # Compute c3=-1 if g is odd and delta>0, and 0 otherwise.
    c3 = c1 & c2
    # Conditionally negate delta based on c3:
    delta = (delta ^ c3) - c3
```

Finally:

```python
    if g & 1:
        if delta > 0:
            f += g
```

becomes:

```python
    f += g & c3
```

It turns out that this can be implemented more efficiently by applying the substitution
*η=-δ*. In this representation, negating *δ* corresponds to negating *η*, and incrementing
*δ* corresponds to decrementing *η*. This allows us to remove the negation in the *c1*
computation:

```python
    # Compute a mask c1 for eta < 0, and compute the conditional negation x of f:
    c1 = eta >> 63
    x = (f ^ c1) - c1
    # Compute a mask c2 for odd g, and conditionally add x to g:
    c2 = -(g & 1)
    g += x & c2
    # Compute a mask c for (eta < 0) and odd (input) g, and use it to conditionally negate eta,
    # and add g to f:
    c3 = c1 & c2
    eta = (eta ^ c3) - c3
    f += g & c3
    # Incrementing delta corresponds to decrementing eta.
    eta -= 1
    g >>= 1
```

A variant of divsteps with better worst-case performance can be used instead: starting *δ* at
*1/2* instead of *1*. This reduces the worst case number of iterations to *590* for *256*-bit inputs
(which can be shown using convex hull analysis). In this case, the substitution *ζ=-(δ+1/2)*
is used instead to keep the variable integral. Incrementing *δ* by *1* still translates to
decrementing *ζ* by *1*, but negating *δ* now corresponds to going from *ζ* to *-(ζ+1)*, or
*~ζ*. Doing that conditionally based on *c3* is simply:

```python
    ...
    c3 = c1 & c2
    zeta ^= c3
    ...
```

By replacing the loop in `divsteps_n_matrix` with a variant of the divstep code above (extended to
also apply all *f* operations to *u*, *v* and all *g* operations to *q*, *r*), a constant-time version of
`divsteps_n_matrix` is obtained. The full code will be in section 7.

These bit fiddling tricks can also be used to make the conditional negations and additions in
`update_de` and `normalize` constant-time.

## 6. Variable-time optimizations

In section 5, we modified the `divsteps_n_matrix` function (and a few others) to be constant time.
Constant time operations are only necessary when computing modular inverses of secret data. In
other cases, it slows down calculations unnecessarily. In this section, we will construct a
faster non-constant time `divsteps_n_matrix` function.

To do so, first consider yet another way of writing the inner loop of divstep operations in
`gcd` from section 1. This decomposition is also explained in the paper in section 8.2. We use
the original version with initial *δ=1* and *η=-δ* here.

```python
for _ in range(N):
    if g & 1 and eta < 0:
        eta, f, g = -eta, g, -f
    if g & 1:
        g += f
    eta -= 1
    g >>= 1
```

Whenever *g* is even, the loop only shifts *g* down and decreases *η*. When *g* ends in multiple zero
bits, these iterations can be consolidated into one step. This requires counting the bottom zero
bits efficiently, which is possible on most platforms; it is abstracted here as the function
`count_trailing_zeros`.

```python
def count_trailing_zeros(v):
    """
    When v is zero, consider all N zero bits as "trailing".
    For a non-zero value v, find z such that v=(d<<z) for some odd d.
    """
    if v == 0:
        return N
    else:
        return (v & -v).bit_length() - 1

i = N # divsteps left to do
while True:
    # Get rid of all bottom zeros at once. In the first iteration, g may be odd and the following
    # lines have no effect (until "if eta < 0").
    zeros = min(i, count_trailing_zeros(g))
    eta -= zeros
    g >>= zeros
    i -= zeros
    if i == 0:
        break
    # We know g is odd now
    if eta < 0:
        eta, f, g = -eta, g, -f
    g += f
    # g is even now, and the eta decrement and g shift will happen in the next loop.
```

We can now remove multiple bottom *0* bits from *g* at once, but still need a full iteration whenever
there is a bottom *1* bit. In what follows, we will get rid of multiple *1* bits simultaneously as
well.

Observe that as long as *η ≥ 0*, the loop does not modify *f*. Instead, it cancels out bottom
bits of *g* and shifts them out, and decreases *η* and *i* accordingly - interrupting only when *η*
becomes negative, or when *i* reaches *0*. Combined, this is equivalent to adding a multiple of *f* to
*g* to cancel out multiple bottom bits, and then shifting them out.

It is easy to find what that multiple is: we want a number *w* such that *g+w f* has a few bottom
zero bits. If that number of bits is *L*, we want *g+w f mod 2L = 0*, or *w = -g/f mod 2L*. Since *f*
is odd, such a *w* exists for any *L*. *L* cannot be more than *i* steps (as we'd finish the loop before
doing more) or more than *η+1* steps (as we'd run `eta, f, g = -eta, g, -f` at that point), but
apart from that, we're only limited by the complexity of computing *w*.

This code demonstrates how to cancel up to 4 bits per step:

```python
NEGINV16 = [15, 5, 3, 9, 7, 13, 11, 1] # NEGINV16[n//2] = (-n)^-1 mod 16, for odd n
i = N
while True:
    zeros = min(i, count_trailing_zeros(g))
    eta -= zeros
    g >>= zeros
    i -= zeros
    if i == 0:
        break
    # We know g is odd now
    if eta < 0:
        eta, f, g = -eta, g, -f
    # Compute limit on number of bits to cancel
    limit = min(min(eta + 1, i), 4)
    # Compute w = -g/f mod 2**limit, using the table value for -1/f mod 2**4. Note that f is
    # always odd, so its inverse modulo a power of two always exists.
    w = (g * NEGINV16[(f & 15) // 2]) % (2**limit)
    # As w = -g/f mod (2**limit), g+w*f mod 2**limit = 0 mod 2**limit.
    g += w * f
    assert g % (2**limit) == 0
    # The next iteration will now shift out at least limit bottom zero bits from g.
```

By using a bigger table more bits can be cancelled at once. The table can also be implemented
as a formula. Several formulas are known for computing modular inverses modulo powers of two;
some can be found in Hacker's Delight second edition by Henry S. Warren, Jr. pages 245-247.
Here we need the negated modular inverse, which is a simple transformation of those:

- Instead of a 3-bit table:
  - *-f* or *f ^ 6*
- Instead of a 4-bit table:
  - *1 - f(f + 1)*
  - *-(f + (((f + 1) & 4) << 1))*
- For larger tables the following technique can be used: if *w=-1/f mod 2L*, then *w(w f+2)* is
  *-1/f mod 22L*. This allows extending the previous formulas (or tables). In particular we
  have this 6-bit function (based on the 3-bit function above):
  - *f(f2 - 2)*

This loop, again extended to also handle *u*, *v*, *q*, and *r* alongside *f* and *g*, placed in
`divsteps_n_matrix`, gives a significantly faster, but non-constant time version.

## 7. Final Python version

All together we need the following functions:

- A way to compute the transition matrix in constant time, using the `divsteps_n_matrix` function
  from section 2, but with its loop replaced by a variant of the constant-time divstep from
  section 5, extended to handle *u*, *v*, *q*, *r*:

```python
def divsteps_n_matrix(zeta, f, g):
    """Compute zeta and transition matrix t after N divsteps (multiplied by 2^N)."""
    u, v, q, r = 1, 0, 0, 1 # start with identity matrix
    for _ in range(N):
        c1 = zeta >> 63
        # Compute x, y, z as conditionally-negated versions of f, u, v.
        x, y, z = (f ^ c1) - c1, (u ^ c1) - c1, (v ^ c1) - c1
        c2 = -(g & 1)
        # Conditionally add x, y, z to g, q, r.
        g, q, r = g + (x & c2), q + (y & c2), r + (z & c2)
        c1 &= c2                     # reusing c1 here for the earlier c3 variable
        zeta = (zeta ^ c1) - 1       # inlining the unconditional zeta decrement here
        # Conditionally add g, q, r to f, u, v.
        f, u, v = f + (g & c1), u + (q & c1), v + (r & c1)
        # When shifting g down, don't shift q, r, as we construct a transition matrix multiplied
        # by 2^N. Instead, shift f's coefficients u and v up.
        g, u, v = g >> 1, u << 1, v << 1
    return zeta, (u, v, q, r)
```

- The functions to update *f* and *g*, and *d* and *e*, from section 2 and section 4, with the constant-time
  changes to `update_de` from section 5:

```python
def update_fg(f, g, t):
    """Multiply matrix t/2^N with [f, g]."""
    u, v, q, r = t
    cf, cg = u*f + v*g, q*f + r*g
    return cf >> N, cg >> N

def update_de(d, e, t, M, Mi):
    """Multiply matrix t/2^N with [d, e], modulo M."""
    u, v, q, r = t
    d_sign, e_sign = d >> 257, e >> 257
    md, me = (u & d_sign) + (v & e_sign), (q & d_sign) + (r & e_sign)
    cd, ce = (u*d + v*e) % 2**N, (q*d + r*e) % 2**N
    md -= (Mi*cd + md) % 2**N
    me -= (Mi*ce + me) % 2**N
    cd, ce = u*d + v*e + M*md, q*d + r*e + M*me
    return cd >> N, ce >> N
```

- The `normalize` function from section 4, made constant time as well:

```python
def normalize(sign, v, M):
    """Compute sign*v mod M, where v in (-2*M,M); output in [0,M)."""
    v_sign = v >> 257
    # Conditionally add M to v.
    v += M & v_sign
    c = (sign - 1) >> 1
    # Conditionally negate v.
    v = (v ^ c) - c
    v_sign = v >> 257
    # Conditionally add M to v again.
    v += M & v_sign
    return v
```

- And finally the `modinv` function too, adapted to use *ζ* instead of *δ*, and using the fixed
  iteration count from section 5:

```python
def modinv(M, Mi, x):
    """Compute the modular inverse of x mod M, given Mi=1/M mod 2^N."""
    zeta, f, g, d, e = -1, M, x, 0, 1
    for _ in range((590 + N - 1) // N):
        zeta, t = divsteps_n_matrix(zeta, f % 2**N, g % 2**N)
        f, g = update_fg(f, g, t)
        d, e = update_de(d, e, t, M, Mi)
    return normalize(f, d, M)
```

- To get a variable time version, replace the `divsteps_n_matrix` function with one that uses the
  divsteps loop from section 5, and a `modinv` version that calls it without the fixed iteration
  count:

```python
NEGINV16 = [15, 5, 3, 9, 7, 13, 11, 1] # NEGINV16[n//2] = (-n)^-1 mod 16, for odd n
def divsteps_n_matrix_var(eta, f, g):
    """Compute eta and transition matrix t after N divsteps (multiplied by 2^N)."""
    u, v, q, r = 1, 0, 0, 1
    i = N
    while True:
        zeros = min(i, count_trailing_zeros(g))
        eta, i = eta - zeros, i - zeros
        g, u, v = g >> zeros, u << zeros, v << zeros
        if i == 0:
            break
        if eta < 0:
            eta, f, u, v, g, q, r = -eta, g, q, r, -f, -u, -v
        limit = min(min(eta + 1, i), 4)
        w = (g * NEGINV16[(f & 15) // 2]) % (2**limit)
        g, q, r = g + w*f, q + w*u, r + w*v
    return eta, (u, v, q, r)

def modinv_var(M, Mi, x):
    """Compute the modular inverse of x mod M, given Mi = 1/M mod 2^N."""
    eta, f, g, d, e = -1, M, x, 0, 1
    while g != 0:
        eta, t = divsteps_n_matrix_var(eta, f % 2**N, g % 2**N)
        f, g = update_fg(f, g, t)
        d, e = update_de(d, e, t, M, Mi)
    return normalize(f, d, Mi)
```

## 8. From GCDs to Jacobi symbol

We can also use a similar approach to calculate Jacobi symbol *(x | M)* by keeping track of an
extra variable *j*, for which at every step *(x | M) = j (g | f)*. As we update *f* and *g*, we
make corresponding updates to *j* using
[properties of the Jacobi symbol](https://en.wikipedia.org/wiki/Jacobi_symbol#Properties):
* *((g/2) | f)* is either *(g | f)* or *-(g | f)*, depending on the value of *f mod 8* (negating if it's *3* or *5*).
* *(f | g)* is either *(g | f)* or *-(g | f)*, depending on *f mod 4* and *g mod 4* (negating if both are *3*).

These updates depend only on the values of *f* and *g* modulo *4* or *8*, and can thus be applied
very quickly, as long as we keep track of a few additional bits of *f* and *g*. Overall, this
calculation is slightly simpler than the one for the modular inverse because we no longer need to
keep track of *d* and *e*.

However, one difficulty of this approach is that the Jacobi symbol *(a | n)* is only defined for
positive odd integers *n*, whereas in the original safegcd algorithm, *f, g* can take negative
values. We resolve this by using the following modified steps:

```python
        # Before
        if delta > 0 and g & 1:
            delta, f, g = 1 - delta, g, (g - f) // 2

        # After
        if delta > 0 and g & 1:
            delta, f, g = 1 - delta, g, (g + f) // 2
```

The algorithm is still correct, since the changed divstep, called a "posdivstep" (see section 8.4
and E.5 in the paper) preserves *gcd(f, g)*. However, there's no proof that the modified algorithm
will converge. The justification for posdivsteps is completely empirical: in practice, it appears
that the vast majority of nonzero inputs converge to *f=g=gcd(f0, g0)* in a
number of steps proportional to their logarithm.

Note that:
- We require inputs to satisfy *gcd(x, M) = 1*, as otherwise *f=1* is not reached.
- We require inputs *x &neq; 0*, because applying posdivstep with *g=0* has no effect.
- We need to update the termination condition from *g=0* to *f=1*.

We account for the possibility of nonconvergence by only performing a bounded number of
posdivsteps, and then falling back to square-root based Jacobi calculation if a solution has not
yet been found.

The optimizations in sections 3-7 above are described in the context of the original divsteps, but
in the C implementation we also adapt most of them (not including "avoiding modulus operations",
since it's not necessary to track *d, e*, and "constant-time operation", since we never calculate
Jacobi symbols for secret data) to the posdivsteps version.

repository: bitcoin/bitcoin

repo_name: bitcoin/bitcoin


text:
# jBPM Case Management Showcase

Showcase application for managing Case instances backed by a running [Kie Server](https://docs.jboss.org/drools/release/6.5.0.Final/drools-docs/html_single/index.html#d0e24201).

## Getting Started

You can clone the repository and build all the projects using: mvn clean install

In order to start the showcase for development you should execute the following Maven commands:
```
cd jbpm-wb-case-mgmt/
mvn clean install
cd jbpm-wb-case-mgmt-showcase/
mvn clean gwt:run
```
The default login for the showcase app is admin/admin

## Connecting to the KIE Execution Server

There are a few supported ways for you to connect and authenticate to a running Kie Server.
These settings are provded using system variables to the running application server where the Case Management application is runnig.
If running on Wildfly, please check the different ways you can provide these settings:
- [General configuration concepts](https://docs.jboss.org/author/display/WFLY10/General+configuration+concepts)
- [Command line parameters](https://docs.jboss.org/author/display/WFLY10/Command+line+parameters)

### General settings

To get started, you need to provide the url for the running Kie Server instance. This is done via the system property *org.kie.server.location*.

  Example:

  *org.kie.server.location*=http://localhost:8230/kie-server/services/rest/server

For some administrative operations to be executed on the Kie Server, i.e. provisioning of example project, it is also required that a pre-defined user is set.
This can be done via two different methods:

- Using a pre-defined user name and password (default)

    Example:

    *org.kie.server.user*=kieserver

    *org.kie.server.pwd*=kieserver1!

- Using a Keyclock token

    For more information on how to set up Keyclock using token based authentication, please read the [Keycloak SSO integration](https://docs.jboss.org/drools/release/6.5.0.Final/drools-docs/html_single/index.html#kie.KeycloakSSOIntegration) documentaion.

    Example:

    *org.kie.server.location*=http://localhost:8230/kie-server/services/rest/server

    *org.kie.server.token*=kieserver1!

For more information about how to set up and operate the Kie Server, please read the upstream [documentation](https://docs.jboss.org/drools/release/6.5.0.Final/drools-docs/html_single/index.html#d0e24201).

### Authentication setup

To authenticate using the current logged in user and forward the user credentials to the Kie Server, a new login module should be added to the WildFly instance running the showcase app.
  Please add this new login module in the security domain *other*.
  ```
  <login-module code="org.kie.security.jaas.KieLoginModule" flag="required" module="deployment.jbpm-wb-case-mgmt-showcase.war"/>
  ```
  When using this mode, users must co-exist in both servers where the showcase app and Kie Server are running in order for the authentication to suceed.

  Users should also have the following roles:
  - user
  - kie-server

## Development mode

The development mode allows you to manage Case instances using an in memory database, whithout connecting to a running Kie Server.
To start the showcase in development mode, please make sure the system property *org.kie.server.location* is not provided.
In order to create case instances, a list of case definitions is set via JSON format [case_definitions.json](./src/main/resources/case_definitions.json).

## Deploying to Wildfly 10

Download and unzip the WildFly 10 distribution. Let's call the root of the distribution WILDFLY_HOME. This directory is named after the WildFly version, so for example wildfly-10.0.0.Final.

Build the jbpm-wb-case-mgmt-showcase project using the following Maven command
```
cd jbpm-wb-case-mgmt/
mvn clean install -Dfull
```
Copy the newly create war file (jbpm-wb-case-mgmt-showcase/jbpm-wb-case-mgmt-showcase.war) into WILDFLY_HOME/standalone/deployments.

Configure user(s) and role(s). Execute the following command:
```
WILDFLY_HOME/bin/add-user.[sh|bat] -a -u 'kieserver' -p 'kieserver1!' -ro 'user'
```
You can of course choose different username and password, just make sure that the user has role user.

Start the server by running:
```
./standalone.sh -Dorg.kie.server.location=http://localhost:8230/kie-server/services/rest/server -Dorg.kie.server.user=kieserver -Dorg.kie.server.pwd=kieserver1!
```

Verify the application is running. Go to http://SERVER:PORT/jbpm-wb-case-mgmt-showcase/ and type the specified username and password. You should see simple XML message with basic information about the server.

For more details, please check the [Getting Started Guide](https://docs.jboss.org/author/display/WFLY10/Getting+Started+Guide).


text:
# FCC-EL1: Recognize existing VM templates provided on a particular cloud infrastructure.

Supporting materials

* [GCCFM3](../../../Materials/GCCFM3.md): Use GCP to Build Your Apps (Slides, Lecture Notes, Lab)
* [CloudAdmin_Azure_M02]():()
* [IBM-ST-MVMK](../../../Materials/IBM-ST-MVMK.md): Moving a VM based app to Kubernetes(Tutorial)
* [AWSACF]():()
* [AWSACA]():()


text:
# SharpGather [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

C# client for Phase 1 - a client application that lets you play the game just like you would on the web.

##Quick Start (for local development and testing)
1. You need a development environment like visual studio.
2. You need a copy of the code.  If you just want to check it out you can goto https://github.com/mrdatawolf/SharpGather and use the "Clone or Download" button.  Otherwise you will need to clone it so you can begin working on it and make pull requests.

##Current Roadmap
Road to 1.0 MVP
1. ~~Allow login using the API on the web~~
2. ~~Save data locally so the access token is saved.~~
2. Populate the storage balances.
3. Populate the bank balance.
4. Populate the gather page so it shows the same data as the web version.
5. On gather click send API request.
6. On add worker click send API request.
7. On add tool click send API request.
8. On add foreman click send API request.
9. On enable click send API request.
10. On automate click send API request.

##Future Roadmap
1. On sell click send API request.
2. On My storage click send API request.
3. On Team Storage click send API request.

##Helping
Feel free to make pull requests. And to message me if you have other ways to help!

repository: mrdatawolf/SharpGather

gha_language: C#

repo_name: mrdatawolf/SharpGather


text:
1498\. Number of Subsequences That Satisfy the Given Sum Condition

Medium

You are given an array of integers `nums` and an integer `target`.

Return _the number of **non-empty** subsequences of_ `nums` _such that the sum of the minimum and maximum element on it is less or equal to_ `target`. Since the answer may be too large, return it **modulo** 109 + 7.

**Example 1:**

**Input:** nums = [3,5,6,7], target = 9

**Output:** 4

**Explanation:** There are 4 subsequences that satisfy the condition.

[3] -> Min value + max value <= target (3 + 3 <= 9)

[3,5] -> (3 + 5 <= 9)

[3,5,6] -> (3 + 6 <= 9)

[3,6] -> (3 + 6 <= 9)

**Example 2:**

**Input:** nums = [3,3,6,8], target = 10

**Output:** 6

**Explanation:** There are 6 subsequences that satisfy the condition. (nums can have repeated numbers). [3] , [3] , [3,3], [3,6] , [3,6] , [3,3,6]

**Example 3:**

**Input:** nums = [2,3,3,4,6,7], target = 12

**Output:** 61

**Explanation:** There are 63 non-empty subsequences, two of them do not satisfy the condition ([6,7], [7]). Number of valid subsequences (63 - 2 = 61).

**Constraints:**

*   1 <= nums.length <= 105
*   1 <= nums[i] <= 106
*   1 <= target <= 106

repository: javadev/LeetCode-in-Java

branch_name: refs/heads/main

gha_language: Java

repo_name: javadev/LeetCode-in-Java


text:
## run environment

```sh
docker-compose up -d
```

## enter on docker image bash

```sh
docker exec -it code_pix_go bash
```

## run application and start a gRPC server

```sh
go run main.go grpc
```


text:
mms-agent-init-file
===================

An init.d file for MongoDB's MMS Agent on CentOS (probably works for Fedora and RHEL too, but I haven't tested it). I couldn't find an init script for MMS and I didn't want to run the agent in a screen, so I decided to write this. Fixes and updates welcome.

Installation
===
Drop this file in /etc/init.d and make sure it's 0755 (executable). Also, this script assumes that your mms-agent files are in /opt/mms-agent.

Usage
====

[root@mongo-node-01 willnewby]# service mms-agent start
Starting MMS Daemon: OK
[root@mongo-node-01 willnewby]#

[root@mongo-node-01 willnewby]# service mms-agent stop
Stopping MMS Daemon: OK
[root@mongo-node-01 willnewby]#

[root@mongo-node-01 willnewby]# service mms-agent restart
Stopping MMS Daemon: OK
Starting MMS Daemon: OK
[root@mongo-node-01 willnewby]#

[root@mongo-node-01 willnewby]# service mms-agent status
MMS Agent running
[root@mongo-node-01 willnewby]#

Assumptions
====
There are a number of assumptions that the script makes, that you can change if you so desire
 - Your mms-agent files live in /opt/mms-agent
 - Your mongodb log directory is /var/log/mongo (which is the RPM default)
 - /var/lock/subsys exists as a directory. (If this is missing, you likely have other problems)

repository: willnewby/mms-agent-init-file

repo_name: willnewby/mms-agent-init-file


text:
Dedupe API
==========

[![Build
Status](https://travis-ci.org/datamade/dedupe-api.svg?branch=master)](https://travis-ci.org/datamade/dedupe-api)

[![Coverage Status](https://coveralls.io/repos/datamade/dedupe-api/badge.png?branch=master)](https://coveralls.io/r/datamade/dedupe-api?branch=master)

Repository for Enterprise Dedupe API

### Setup

**Install OS level dependencies:**

* [Python 2.7](https://www.python.org/download/releases/2.7/)
* [Redis](http://redis.io/)

**Install app requirements**

```bash
git clone git@github.com:datamade/dedupe-api.git
cd dedupe-api
pip install "numpy>=1.6"
pip install -r requirements.txt
```

Create a PostgreSQL database for dedupeapi. (If you aren't
  already running [PostgreSQL](http://www.postgresql.org/), we recommend
  installing version 9.3 or later.)

```
createdb dedupeapi
```

Create your own `app_config.py` file:

```
cp api/app_config.py.example api/app_config.py
```

You will want to change, at the minimum, the following `app_config.py` fields:

* `DB_CONN`: edit this field to reflect your PostgreSQL
  username, server hostname, port, and database name.

* `DEFAULT_USER`: change the username, email and password on the administrator account you will use on Plenario locally.

Before running the server, [Redis](http://redis.io/).

* To start Redis locally (in the background):
```bash
redis-server &
```

Initialize the dedupe-api database:

```bash
python init_db.py
```

Finally, run the queue and server:

```bash
nohup python run_queue.py &
python runserver.py
```

Once the server is running, navigate to http://localhost:5000/

## Community
* [Dedupe Google group](https://groups.google.com/forum/?fromgroups=#!forum/open-source-deduplication)
* IRC channel, #dedupe on irc.freenode.net

repository: dmkoch/dedupe-api

gha_language: Python

repo_name: dmkoch/dedupe-api


text:
Viewer component for 3dbionotes.

## Setup

```shell
$ yarn install
```

## Build

```shell
$ yarn build

# On main Rails app
$ RAILS_ENV=production bundle exec rake assets:precompile
```

## Development

```shell
$ PORT=3001 yarn start
```

If you need to make change in protvista-pdb or pdbe-molstar, run:

```shell
$ git clone https://github.com/EyeSeeTea/protvista-pdb
$ (cd protvista-pdb && npm install && npm run build && yarn link)

$ git clone https://github.com/EyeSeeTea/pdbe-molstar
$ (cd pdbe-molstar && npm install && npm run build && yarn link)

$ yarn link @3dbionotes/protvista-pdb
$ yarn link @3dbionotes/pdbe-molstar
```

And, finally, change the `<script src="..."` tags in `public/index.html` to the version you are using in the cloned repos. Run `yarn run watch` on the repositories your are making changes.

gha_language: TypeScript


text:
---
layout: post
rel_link:  "9_3_14_patch1"
title:  "CMSSW_9_3_14_patch1"
date:   2019-03-13 18:06:10
categories: cmssw
relmajor: 9
relminor: 3
relsubminor: 14
relpatch: _patch1
---

# CMSSW_9_3_14_patch1
#### Changes since CMSSW_9_3_14:
[compare to previous](https://github.com/cms-sw/cmssw/compare/CMSSW_9_3_14...CMSSW_9_3_14_patch1)

1. [26123](http://github.com/cms-sw/cmssw/pull/26123){:target="_blank"}  from **@Andrej-CMS**: herwig 7 settings `generators`  created: 2019-03-08 17:51:06 merged: 2019-03-13 15:55:09

#### CMSDIST Changes between Tags REL/CMSSW_9_3_14/slc6_amd64_gcc630 and REL/CMSSW_9_3_14_patch1/slc6_amd64_gcc630:
[compare to previous](https://github.com/cms-sw/cmsdist/compare/REL/CMSSW_9_3_14/slc6_amd64_gcc630...REL/CMSSW_9_3_14_patch1/slc6_amd64_gcc630)

repository: cms-sw/ReleaseNotes

repo_name: cms-sw/ReleaseNotes


text:
- https://github.com/webpack/webpack/tree/master/examples/dll
- https://robertknight.github.io/posts/webpack-dll-plugins/
- https://gist.github.com/robertknight/058a194f45e77ff95fcd
-<EMAIL_ADDRESS>http://engineering.invisionapp.com/post/optimizing-webpack/
- https://engineering.bitnami.com/2016/11/15/optimizing-your-webpack-builds.html
http://odetocode.com/blogs/scott/archive/2016/12/01/building-vendor-and-feature-bundles-with-webpack.aspx
- https://bendyworks.com/blog/optimizing-ci-and-webpack
- https://webpack.js.org/guides/caching/
- https://github.com/danethurber/webpack-manifest-plugin
- https://github.com/danethurber/webpack-manifest-plugin/tree/master/spec
- https://webpack.js.org/guides/code-splitting-libraries/#manifest-file
- https://medium.com/connect-the-dots/caching-assets-long-term-with-webpack-5ad24a4c39bd#.58yunf3an
- http://stackoverflow.com/questions/36986460/selecting-webpack-dll-bundle-via-scope-mode
- http://www.developersite.org/101-35917-webpack
- https://github.com/shlomiassaf/webpack-dll-bundles-plugin
- https://www.npmjs.com/package/webpack-build-dll-plugin
- https://github.com/webpack/webpack/blob/9cf5901f37c43e53e7adc831ff30a5a60d12c94e/lib/DllPlugin.js#L17
- https://github.com/webpack/webpack/blob/93ac8e9c3699bf704068eaccaceec57b3dd45a14/lib/DllReferencePlugin.js

repository: fliphub/d-l-l

gha_language: JavaScript

repo_name: fliphub/d-l-l


text:
[![HourMonitor Logo](http://kylerassweiler.com/media/img/HourMonitor/logo.png)](http://hourmonitor.kylerassweiler.ca)

[![Build Status](https://travis-ci.org/rassweiler/HourMonitor-MEAN.svg?branch=master)](https://travis-ci.org/rassweiler/HourMonitor-MEAN)

HourMonitor is a simple web application for tracking hours worked, designed to be mobile friendly and multiplatform.

## Prerequisites
Make sure you have installed all these prerequisites on your development machine.
* Node.js - [Download & Install Node.js](https://nodejs.org/en/download/) and the npm package manager, if you encounter any problems, you can also use this [GitHub Gist](https://gist.github.com/isaacs/579814) to install Node.js.
* MongoDB - [Download & Install MongoDB](http://www.mongodb.org/downloads), and make sure it's running on the default port (27017).
* Bower - You're going to use the [Bower Package Manager](http://bower.io/) to manage your front-end packages, in order to install it make sure you've installed Node.js and npm, then install bower globally using npm:

```
$ npm install -g bower
```

* Yeoman - Install the generator with npm:

```
$ npm install -g yo
```

* Grunt - You're going to use the [Grunt Task Runner](http://gruntjs.com/) to automate your development process, in order to install it make sure you've installed Node.js and npm, then install grunt globally using npm:

```
$ sudo npm install -g grunt-cli
```

* Python - [Download & Install](https://www.python.org/downloads/)
* GIT - [Download & Install](https://git-scm.com/downloads)
* JDK - [Download & Install](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)

## Quick Install

###Windows
- Install python 2.7 (not 3) and select to add to path
- Install git
- Install Node.js
- Install MongoDB (also add path to env var e.g. C:\Program Files\MongoDB\Server\3.0\bin to make starting it easier)
- Create dirs C:\data\db\
- clone this repo
- Open new powershell(now called ps2) and cd to repo dir
- Enter in ps2:
```
npm install -g bower
```
- Enter in ps2:
```
$ npm install -g grunt-cli
```
- Enter in ps2:
```
$ npm install -g generator-meanjs
```
- Install the rest of the projects dependencies. enter in ps2:
```
$ npm install
```
This command does a few things:
* First it will install the dependencies needed for the application to run.
* If you're running in a development environment, it will then also install development dependencies needed for testing and running your application.
* Finally, when the install process is over, npm will initiate a bower install command to install all the front-end modules needed for the application

###Linux
TODO: Add linux install instructions

## Running Your Application
After the install process is over, you'll be able to run your application using Grunt, just run grunt default task:

```
$ grunt
```

Your application should run on the 3000 port so in your browser just go to [http://localhost:3000](http://localhost:3000)

You also need mongodb running in a seperate shell:

```
$ mongod
```
Your db should run on the 27017 port so in your browser just go to [http://localhost:27017](http://localhost:27017)

That's it! your application should be running by now, to proceed with your development check the other sections in this documentation.
If you encounter any problem try the Troubleshooting section.

## Credits
- HourMonitor - Created by [Kyle Rassweiler](http://www.kylerassweiler.com)
- MEAN - Inspired by the great work of [Madhusudhan Srinivasa](https://github.com/madhums/)
The MEAN name was coined by [Valeri Karpov](http://blog.mongodb.org/post/49262866911/the-mean-stack-mongodb-expressjs-angularjs-and)

## License
(The MIT License)

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: rassweiler/HourMonitor-MEAN

gha_license_id: NOASSERTION

repo_name: rassweiler/HourMonitor-MEAN


text:
# Client Sync

This is the mail sync engine that runs within the Nylas Mail client

It is symlinked in as an `internal_package` of Nylas Mail via the `postinstall`
script of the root repo.

## Important Usage Notes:

Since this is symlinked in as an `internal_package` of Nylas Mail, there are a
handulf of considerations when developing in client-sync. Some common gotchas:

- You MAY use `NylasEnv`, `NylasExports` and other injected libraries in the
  Nylas Mail client environment.
- You MAY use any 3rd party library declared in `client-app/package.json`.
  Since this gets added as a plugin of the Nylas Mail client, you'll have
  access to all libraries. This works because the `client-app/node_modules` was
  added to the global require paths. That lets us access client-app plugins
  without being a file directory decendent of client-app (client-sync is now a
  sibling of client-app)
- You may NOT add "dependencies" to the `client-sync/package.json`. If you need
  a 3rd party library, add it to the main `client-app/package.json`. All Nylas
  Mail plugins (those inside of `internal_packages`), may no longer declare
  their own dependencies.
- You should be aggressive at moving generic mail methods to `isomorphic-core`.
  We may eventually want to make large chunks of client-sync work in a cloud
  environment as well.

repository: Microsoft-Android/nylas-mail

repo_name: Microsoft-Android/nylas-mail


text:
---
title: "Loading older todos"
metaTitle: "Loading older todos | GraphQL React Native Apollo Tutorial"
metaDescription: "You will learn how to load older todos added by other people to see an example of pagination in GraphQL."
---

It might not be always ideal to load all the todos in the database. You would want to have some kind of pagination to enforce fetching data incrementally.

In this app, we will implement the basic functionality of loading older todos on a button click.

The ideal workflow to implement this feature would be:

1. Start with loading only "n" items
2. On "load more", fetch "n" more items older than the oldest local item
3. Append the newly loaded items to apollo cache

You will learn:

1. Ideas behind implementing pagination in GraphQL
2. To update apollo cache whenever required

Lets do it.

repository: hasura/learn-graphql

repo_name: hasura/learn-graphql


text:
https://www.youtube.com/watch?v=CK6h48zOY9k

Doug Terry FAST 19

# Architecture changes

1. Introduced Transaction coordinator Node after Request coordinator Node, and before Storage Node
1. Added Ledger and Recovery

# Latency

1. TransactWriteItem : is indep of db size, but depend on number of participants.  Two round protocol added factor of 4 latency.
1. TransactGetItem : extra hop to Transaction Coordinator node
1. Get/Put have no additional latency

# design

timestamps given to transactions by Transaction Coordinators - it is some unique ID.

time sync not requirement for correctness

why not MVCC : DDB does not store multiple versions; you would have to keep versions and retire them.

why not explicit multi-step transactions and 2-phase locking : too costly

use timestamp ordering and not locks

1. transactions can arrive at each storage node in different order
2. each storage node reasons independently about transaction acceptance
3. but each node eventually reaches same state of data

use knowledge of already accepted transactions to reason about acceptance or rejection of incoming (but older) transaction

check conditions attached to each transaction

1. can accept old Transaction which came later (reordered) if the newer transaction was an unconditional Put (overwrite)
2. can accept newer transaction unless condition is not met
3. Unconditional PUT and DELETE can be accepted even if out of order

Time 45 min in video
1. Tx2 (update b = b + 1) comes before Tx1 (Put b=1)
2. wait for Tx1 before applying Tx2

## handling readonly

Reject (or retry) Readonly transaction if its in the future

## Mix of Non-transcational with transactions

PUT with no timestamp comes to storage node

If either PUT and Transaction has condition, then PUT has to be aborted.  This is new failure mode

non-transactional reads : ddb never has dirty data since writes done in phase 2 (accept + write)

## timestamps

easily correct serializability : based on TS (no locks)

independent serializability : storage nodes need not talk to each other - makes it easier to test

## global tables

repository: sanjosh/csdocs

repo_name: sanjosh/csdocs


text:
# final semester project
Underwater Communication using OFDM techniques

Designing an underwater communication for modulation and demodulation of acoustic waves using OFDM technique. We are simulating it using MATLAB and Simulink. The channels considered for transmission are
###### Marine water
###### River water
###### Deep underwater
###### Shallow underwater

repository: nestorwheelock/finalsemproject

repo_name: nestorwheelock/finalsemproject


text:
---
title: Adding Javascript
taxonomy:
    category: docs
---

![Javascript Compressor](javascript-compressor.jpg)

## Add external javascript files to the template javascript
If you would like to compress external or additional javascript files inside the single template javascript file then you can add the relative path to the javascript file in the text area here. Each file needs to be entered on a new line.

In order to compress the new file into the template.js file you will need to also click the compress scripts button as seen in the screen shot and then proceed to save the template by clicking the save or save and close button at the top of the page.

## Compress js after updates
If you have recently updated your template either manually or updating the template package using one of our official releases you may need to re-compress scripts to have any updated javascript display oin your page.

After clicking on the compress scripts button a javascript file is created that references the id for this specific instance of the template. As an example if the id for the template is 78 (as specified in the Joomla installation) then a file called template-78.js will be written to the js folder of the template.

repository: bamboothemes/buildr-documentation

gha_language: PHP

repo_name: bamboothemes/buildr-documentation


text:
# MDLP

This is an sklearn-transformer compliant MDLP Discretizer that discretizes numeric features according to the MDLPC criterion.

## Reference
```
Fayyad and Irani. Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning. 1993. IJCAI-93.
```

repository: paulbkoch/mdlp

repo_name: paulbkoch/mdlp


text:
Thus, you can take care of those issues, sending them to your developers by email, but also use some hooks of your own, as adding them to your Agile board, to only name a few.

### What you'll receive in the server-side

- `message`: The error message. Basically.
- `file`: The target file where the bug has been found.
- `line`: The error's line number, only making sense when the script is not minified.
- `column`: The column's number.
- `date`: The date when the errors happened.
- `ismobile`: Whether it happened on a mobile device.
- `isretina`: Whether the device is retina.
- `os`: The operating system on which the error was thrown.
- `browser`: The target browser.
- `version`: The browser's version.
- `language`: The browser's main language.
- `resolution`: The browser's resolution (widthxheight).
- `orientation`: The browser's orientation.
- `url`: The URL where the error happened.

### How to settle it

```javascript
var options = {
  serverUrl: "http://domain.tld/errors",
  method: "POST",
  maxRequests: 10,
  token: {
    'key':'X-CSRF-Token',
    'value': document.querySelector('meta[name="csrf-token"]').content
  },
  shouldLog: true,
  cb: function (serverResponse) {
    console.info(serverResponse);
  }
};

new Sioux(options).watch();

// Now we throw a dumb exception
console.log(new UnknownObject());
```

### Available parameters

- `serverUrl`: Your server-side error API route.
- `method`: The method to apply, default to POST.
- `maxRequests`: The maximum number of error logs to send, avoiding spam.
- `token`: If you have a CSRF token to add to the request's header, that's here.
- `shouldLog`: Whether of not it should log the requests in the front-end console. Default to false.
- `cb`: The callback function to run when we get an answer from the server.

### Example

Take a look at the `/example` folder where you will find a basic Ruby on Rails app that uses the library. It will raise an error in front, log its parameters in backend and send back a json.
__How to try it:__
`$ cd example`
`$ gem install`
`$ rails s`
From this point, you can do whatever you want with those errors. For instance you can send an email, add a [Trello](http://trello.com) card, a [Github](http://github.com) issue, use [IFTTT](http://ifft.com) to use thousand of hooks, actually the only limit is your imagination!

### Credits

Special thanks to [Simon Child](http://thenounproject.com/Simon%20Child/) for the visual.

repository: Pygocentrus/sioux

gha_language: Ruby

repo_name: Pygocentrus/sioux


text:
---
site: freiburg
title: Enough talking - time to act!
tags: [s4f, society]
author_github: bgruening
---

September the 25th will see the next global climate strike!
As last time, the Freiburg Galaxy Team is going to express its support more directly:

The team will be striking and **we will not schedule new jobs on the European Galaxy server for the whole day**.

Running jobs will not be interrupted, but no new jobs will start to run. They will stay grey
for the entire day.

Again, we encourage all of you to join one of the huge number of strikes
[that will be taking place world-wide](https://fridaysforfuture.org/september25/) on this day!

We will be striking in Freiburg, if you are nearby join us - and maybe we will find time
to talk about research as well :)

repository: usegalaxy-eu/website

gha_language: TeX

repo_name: usegalaxy-eu/website


text:
# pantrydb

**A Django Project to manage the Heights Interfaith Ministry Food Pantry volunteers and clients. It's built with Python and the Django Web Framework**

This project has the following apps:

* accounts - contains shared information used in the rest of the apps
* volunteers - has volunteer profile and the hours log
* clients - clients of the pantry
* delivery_clients - clients that are delivered to

## Volunteers

### Models
* Volunteer
    * proxy model of user
    * Creates a group called Volunteers if their isn't one when creating a new Volunteer.
    * Volunteer queries will only return users who are in the volunteers group.
* VolunteerProfile
    * Inherits baseProfile
* VolunteerLog
    * Dates with sign and out times
    * One Volunteer
    * Custom queryset that gets todays logs without a signout time

### Views
#### CreateVolunteerView
 * Create volunteer
 * Name and a zipcode
 * Nice to show username for future reference

#### SignInView
 * Two forms handled with prefixes
  * Sign out
  * Sign In
  * Link to CreateVolunteerView if volunteer not found.

#### Forms
* CreateSignOutForm
* Sign In
* LogSignOutForm

Each user has extras fields based on if they are staff, volunteer, or client.

A specific user is created to be signed in to allow volunteers to be able to access the volunteer page.

Cronjabs are run once a day at 11 pm to clean up and users without a sign in time.

repository: HIM-Food-Pantry/pantrydb

branch_name: refs/heads/dev

repo_name: HIM-Food-Pantry/pantrydb


text:
---
title: IMS Support for Adobe Experience Manager as a Cloud Service
description: IMS Support for Adobe Experience Manager as a Cloud Service
exl-id: fb563dbd-a761-4d83-9da1-58f8e462b383
---
# IMS Support for Adobe Experience Manager as a Cloud Service {#ims-support-for-aem-as-a-cloud-service}

## Introduction {#introduction}

* AEM as a Cloud Service includes Admin Console support for AEM instances and Adobe Identity Management System (IMS for short) based authentication.
* The Admin Console allows administrators to centrally manage all Experience Cloud users.
* Users and Groups can be assigned to product profiles associated with AEM as a Cloud Service instances, allowing them to log in to that instance.

>[!TIP]
>
>See our Experience League course [Configure Access to AEM for Administrators](https://experienceleague.adobe.com/?recommended=ExperienceManager-A-1-2020.1.aem) for an introduction to how users authenticate using Adobe IMS to AEM as a Cloud Service and how Adobe IMS Users, User Groups, and Product Profiles are used to control access to AEM and its features and functionalities. Adobe ID required.

## Key Highlights {#key-highlights}

AEM as a Cloud Service offers IMS authentication support only for Author, Admin and Dev users. It does not offer support for external end users of customer sites like site visitors.

* The Admin Console will represent customers as IMS Organizations, Author and Publish Instances in an environment as Product Context Instances. This will allow System and Product administrators to manage access to instances.
* Product Profiles in the Admin Console will determine which Instances a user can access.
* Customers will be able to use their own SAML 2 compliant Identity Providers (IDP for short) for Single Sign On.
* Only Enterprise or Federated IDs for customer Single Sign On will be supported, no personal Adobe IDs.

## Architecture {#architecture}

IMS Authentication works using OAuth protocol between AEM and the Adobe IMS endpoint. Once a user has been added to IMS and has an Adobe Identity, they can log in to AEM author service using IMS credentials.

The user login flow is shown below, the user will be redirected to IMS and optionally to the customer IDP for SSO and then redirected back to AEM.

![IMS Architecture](/help/security/assets/ims1.png)

## How to Set Up {#how-to-set-up}

### Onboarding Organizations to Adobe Admin Console {#onboarding-orgs-to-adobe-admin-console}

The customer onboarding to Adobe Admin Console is a prerequisite to using Adobe IMS for AEM authentication.

As the first step, customers need to have an Organization provisioned in Adobe IMS. Adobe Enterprise customers are represented as IMS Organizations in the [Adobe Admin Console](https://helpx.adobe.com/enterprise/using/admin-console.html) This is the portal used by Adobe customers to manage their product entitlements for their users and groups.

AEM customers should already have an Organization provisioned, and as part of the IMS provisioning, the customer instances will be made available in Admin Console for managing user entitlements and access.

Once a customer exists as an IMS Organization, they will have to configure their system as summarized below:

![IMS Onboarding](/help/security/assets/ims2.png)

1. The designated System Administrator receives an invite to log in to Cloud Manager. After logging into Cloud manager, the System Administrators can choose to provision AEM programs and environments or navigate to Admin Console for Administrative tasks.
1. The System Administrator claims a domain to confirm the ownership of the respective domain (for example acme.com)
1. The System Administrator sets up User Directories
1. The System Administrator does IDP configuration in Admin Console in order to set up Single Sign On.
1. The AEM Administrator manages the local groups and permissions and privileges as usual.

The Adobe Identity Management basics including IDP configuration are covered [here](https://helpx.adobe.com/enterprise/using/set-up-identity.html).

Enterprise Administration and Admin Console usage is covered [here](https://helpx.adobe.com/enterprise/managing/user-guide.html).

### Onboarding Users in Admin Console {#onboarding-users-in-admin-console}

There are three ways to onboard users depending on the size of the customer and their preference: manually create users in Admin Console, upload a .csv file or sync users from the customer's enterprise Active Directory.

**Manual Addition through Admin Console UI**

Users and Groups can be manually created in the Admin Console UI. This method can be used if you do not have a large number of users to manage. For example, less than 50 AEM users or if the you are already using this method for administering other Adobe products like Analytics, Target or Creative Cloud applications.

![User Onboarding](/help/security/assets/ims3.png)

**File Upload in Admin Console UI**

For easy handling of user creation, a `.csv` file can be uploaded for adding users in bulk.

![File Upload](/help/security/assets/ims4.png)

**User Sync Tool**

User Sync Tool (UST in short) enables our enterprise customers to create and manage Adobe users utilizing Active Directory. This also works for other tested OpenLDAP directory services. The target users are IT Identity Administrators (Enterprise Directory or System Admins) who will be able to install and configure the tool. The open source tool is customizable so that customers you modify it to suit your own particular requirements.

When User Sync runs, it fetches a list of users from the organization’s Active Directory and compares it with the list of users within the Admin Console.  It then calls the Adobe User Management API so that the Admin Console is synchronized with the organization’s directory. The change flow is entirely one way. Any edits made in the Admin Console do not get pushed out to the directory.

The tool allows the system admin to map user groups in the customer’s directory with product configuration and user groups in the Admin Console.

To set up User Sync, the organization needs to create a set of credentials in the same way they would use the [User Management API](https://www.adobe.io/apis/experienceplatform/umapi-new.html).

![User Sync Tool](/help/security/assets/ims5.png)

User Sync Tool is distributed through the Adobe Github repository [at this location](https://github.com/adobe-apiplatform/user-sync.py/releases/latest).

>[!NOTE]
>
>A prerelease version **2.4RC1** is available with dynamic group creation support and can be found [here](https://github.com/adobe-apiplatform/user-sync.py/releases/tag/v2.4rc1).

The major features for this release are the ability to dynamically map new LDAP groups for user membership in the Admin Console, as well as dynamic user group creation.

More information about the new group features can be found [at this location](https://github.com/adobe-apiplatform/user-sync.py/blob/v2/docs/en/user-manual/advanced_configuration.md#additional-group-options).

**User Sync Documentation**

Refer to the [UST documentation](https://adobe-apiplatform.github.io/user-sync.py/en/) for more details.

The User Sync Tool needs to register as an Adobe I/O client UMAPI using the procedure [here](https://adobe-apiplatform.github.io/umapi-documentation/en/UM_Authentication.html).

Adobe I/O Console Documentation can be found [here](https://www.adobe.io/apis/cloudplatform/console.html).

The User Management API that is used by the User Sync Tool is covered [here](https://www.adobe.io/apis/cloudplatform/umapi-new.html).

## Adobe Experience as a Cloud Service Configuration {#aem-configuration}

>[!NOTE]
>
>The AEM IMS configuration required will be automatically configured when the AEM environments and instances are provisioned. However, the administrator may modify it as per their requirements using the method described [here](/help/implementing/deploying/overview.md).

The AEM IMS configuration required will be auto-configured when the AEM environments and instances are provisioned.  Customer administrators may modify part of the configuration as per their requirements

The overall approach is to configure Adobe IMS as an OAuth provider. The **Apache Jackrabbit Oak Default Sync Handler** can be modified just like for LDAP synchronization.

Below are the key OSGI configurations that need to be modified in order to change properties like User Auto Membership or Groups Mappings.

<!-- Arun to provide list of osgi configs -->

## How to Use {#how-to-use}

### Managing Products and User Access in Admin Console {#managing-products-and-user-access-in-admin-console}

When the Product Administrator logs in to Admin Console, they will see multiple instances of the AEM as a Cloud Service Product Context, as shown below. For example, select any of the the products from the **Overview** page:

![Instances login](/help/security/assets/ims6.png)

You will see a list of existing instances:

![Instances login2](/help/security/assets/ims7.png)

Under each Product Context instance, there will be instances spanning Author or Publish services across Production, Stage, or Development environments. Each instance will be associated to Product Profiles or Cloud Manager roles. These product profiles are used for assigning access to Users and Groups with the required privileges.

The **Administrator_xxx** profile will be used to grant Administrator privileges in the associated AEM instance while the **User_xxx** profile is used to add regular users.

Any users and groups added under this product profile will be able to login to that particular instance as shown in the example below:

![Product Profile](/help/security/assets/ims8.png)

### Logging into Adobe Experience Manager as a Cloud Service {#logging-in-to-aem}

**Local Administrator Login**

AEM can continue to support local logins for Admin users. The login screen has an option to log in locally:

![Local Login](/help/security/assets/ims9.png)

<!-- the above image needs to be updated for skyline -->

**IMS Based Login**

For other users, the IMS based login can be used once IMS is configured on the instance. The user will first click on the Sign in with Adobe button as shown below:

![IMS Login](/help/security/assets/ims10.png)

>[!NOTE]
>
>Any user created in IMS can be created using Adobe ID or Federated ID. If a user is setup using Federated ID, they are authenticated using their Company's Identity Provider to login.

They will then be redirected to the IMS login screen and will need to enter their credentials:

![IMS Login2](/help/security/assets/ims11.png)

![IMS Login3](/help/security/assets/ims12.png)

If a federated IDP is configured during initial Admin Console setup, then the user will be redirected to the customer IDP for SSO:

![IMS Login4](/help/security/assets/ims13.png)

After authentication is complete, the user will be redirected back to AEM and logged in:

![IMS Login5](/help/security/assets/ims14.png)

### Managing Permissions and ACLs in Adobe Experience Manager as a Cloud Service {#managing-permissions-in-aem}

The ACLs and permissions will continue to be managed in AEM. The User Groups that are synced from IMS can be assigned to local groups where ACLs and privileges are defined.

In the example below, we are adding synced groups to the local **Dam_Users** group as an example.

The user is part of the following Groups in IMS:

![ACL1](/help/security/assets/ims15.png)

When the user logs in, their Group Memberships are synced, as shown below:

![ACL2](/help/security/assets/ims16.png)

In AEM, the User Groups synced from IMS can be added as members to existing local groups, like **DAM Users**.

![ACL3](/help/security/assets/ims17.png)

As shown below, the group **AEM-GRP_008** inherits the permissions and privileges of **DAM Users**, this is an effective way of managing Permissions for synced groups and is commonly used in the LDAP based Authentication method as well.

![ACL3](/help/security/assets/ims18.png)

### Accessing Cloud Manager {#accessing-cloud-manager}

To be able to access Cloud Manager or to AEM as a Cloud Service environments, you must be assigned to Profiles of the Cloud Manager Product.

Refer to Role Definitions to learn more about roles for users which govern the availability of specific features in Cloud Manager.

>[!NOTE]
>Cloud Manager has pre-configured roles with appropriate permissions. To learn about each of the roles with specific permissions, pre-configured tasks, or permissions, associated with each role, refer to [Role Based Permissions](https://experienceleague.adobe.com/docs/experience-manager-cloud-service/onboarding/what-is-required/role-based-permissions.html).

**Steps for Adding a User**

1. Add a user to a particular profile either from an existing user's screen or from a new user screen.

1. Alternatively, you can also add a user from the **Overview** screen, as shown in the figure below.

   ![ACL3](/help/security/assets/ims23.png)

   >[!NOTE]
   >You can assign more than one profile to a user as shown in the figure below.

   ![ACL3](/help/security/assets/ims22.png)

1. Once you have been added to the appropriate profile, you should be able to access the respective tenants in Cloud Manager via [Adobe Experience Cloud](http://my.cloudmanager.adobe.com) using the top right corner from the user interface.

### Accessing an Instance in AEM as a Cloud Service {#accessing-instance-cloud-service}

>[!IMPORTANT]
>The steps mentioned in the preceding section must have already been completed before you are granted access to an instance in AEM as a Cloud Service.

In order to have access to an AEM instance within the **Admin Console**, you should see the Cloud Manager Program and the environments within the program in the product list on the **Admin Console**.

For example, in the screenshot below, you will see two available environments namely *dev author* and a *publish*.

![ACL3](/help/security/assets/ims19.png)

To get access to AEM instances the user will need to be added to a group of the appropriate Cloud Service Product.

Every author instance will have an AEM Administrators and AEM Users Profile and every publish instance will have an AEM Users Profile. You can add other profiles as needed.

To get admin level access to the AEM instance, add the user to the AEM Administrators Profile for that particular Product.


text:
# Good Practices to Use a KafkaConsumer

If we want to achieve high performance/availability, here're some rules of thumb.

## How to distribute the messages (for the same topics) to different KafkaConsumers

* Use a consumer group for these KafkaConsumers, thus they will work together -- each one deals with different partitions.

* Besides `subscribe` (topics), users could also choose to explicitly `assign` certain partitions to a `KafkaConsumer`.

## How to enhance the throughput

* Try with a larger `QUEUED_MIN_MESSAGES`, especially for small messages.

* Use multiple KafkaConsumers to distribute the payload.

## How to avoid polling duplicated messages

* A `KafkaManualCommitConsumer` could help to commit the offsets more frequently (e.g, always do commit after finishing processing a message).

repository: alex-q-chen/modern-cpp-kafka

repo_name: alex-q-chen/modern-cpp-kafka


text:
---
layout: post
title:      "Understanding Join Tables and Relational Databases"
date:       2020-04-29 17:31:50 -0400
permalink:  understanding_join_tables_and_relational_databases
---

Boy, do I love SQL. After 8 grueling weeks of Procedural and Object Oriented Ruby (which was equivalent to quite literally learning a new langauge for me), I finally feel like I am cruising through SQL and databases. I, for one, love data. I've always enjoyed working in spreadsheets, collecting data, analyzing data, and using it to solve business problems (no, I am not a data scientist).

I think part of the reason I enjoy learning about SQL and relational databases is that it is a much more visual process than building methods in OO Ruby. Not to mention, the language is more literal (e.g. "CREATE TABLE," "INSERT INTO," "SELECT * FROM" etc...). Relational concepts are also much easier for me to understand when I can create a visual structure prior to building a database.

When building databases, the "belongs_to/has_many" relationship is established with the use of foreign keys. When building a table, the child entity (i.e. the one that belongs to another entity) has a foreign key column. Here's an example:

![](https://i.imgur.com/7698X3H.png)

As you can see above, Cujo and Charlie both have an owner_id of 1, and therefore belong to Beth. Luna has an owner_id of 2, belonging to Steven. And finally, Dakota has an owner_id of 3 and belongs to Ruth.

So how do we go about storing data consisting of many-to-many relationships? This is where things get fun, and a little complex (IMHO). Enter: *join tables*. Join tables allow you to store data with many-to-many associations by having a field of common data from multiple tables (typically foreign keys). A great example of this is one of the labs in our Learn.co curriculum, SQL Library Lab. In this lab, I had to create a table for the following: Characters, Books, Series, Authors, and Sub-Genres. In order to create complex SQL queries and create proper associations, I had to create a join table. Before getting started, I went ahead and physically illustrated the tables I was building and their relationships with one another.

![](https://i.imgur.com/KZ09AFP.jpg)

As you can see, I built the join table to connect both books and characters, as books have many characters and characters have many books. The join table is called character_books and needs two foreign key columns, like so:

```
CREATE TABLE character_books (
id INTEGER PRIMARY KEY,
book_id INTEGER,
character_id INTEGER);
```

Once all of the tables were built, I simply had to write queries that would complete the tests. Here's an example of one a query involving the join table:

```
def select_character_names_and_number_of_books_they_are_in
     "SELECT characters.name, COUNT(*) as book_count
     FROM characters
     JOIN character_books
     ON character_books.character_id = characters.id
     GROUP BY characters.name
     ORDER BY book_count DESC"
end
```

Join tables make these complex relationships and complicated database queries possible! The biggest takeaway of this week for me is this: *when it doubt, draw it out.*

extension: markdown


text:
# Docker compose + Consul (core & template) + Registrator + Nginx + NodeJs + Blue/Green deployment
To test blue green deployment with docker compose

We add this extra configuration:

1st: We build our docker-compose.
```yaml
sudo docker-compose up build
```

2nd: We start our docker-compose, but, only start consul app
```yaml
sudo docker-compose up -d consul
```

3rd: Create our key value that we are going to use to change dynamically to bluee / green deployment.
Put the key/value for our purpose:
```yaml
curl -X PUT -d 'blue' http://localhost:8500/v1/kv/bluegreen/deploy
```
Check if this value was stored or not:
```yaml
curl http://localhost:8500/v1/kv/bluegreen/deploy?raw
```

4.- Start all our app throughout:
```yaml
sudo docker-compose up -d --no-recreate
```

5.- This line have the main idea:  proxy_pass http://{{key "bluegreen/deploy"}};
```yaml
upstr```yamleam blue {
  least_conn;
  {{range service "production.appblue"}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
  {{else}}server <IP_ADDRESS>:65535; # force a 502{{end}}
}
upstream green {
  least_conn;
  {{range service "production.appgreen"}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
  {{else}}server <IP_ADDRESS>:65535; # force a 502{{end}}
}
server {
  listen 80 default_server;

  location / {
    proxy_pass http://{{key "bluegreen/deploy"}};
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}
```

6.- Check the nginx configuration (it should be generated with blue app_stream):
```yaml

root@8c64c9853f47:/# cat /etc/nginx/conf.d/app.conf
upstream blue {
  least_conn;
  server <IP_ADDRESS>:32768 max_fails=3 fail_timeout=60 weight=1;

}

upstream green {
  least_conn;
  server <IP_ADDRESS>:32769 max_fails=3 fail_timeout=60 weight=1;

}

server {
  listen 80 default_server;

  location / {
    proxy_pass http://green;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}
```

7.- Testing:
```yaml
curl -X PUT -d 'green' http://localhost:8500/v1/kv/bluegreen/deploy
curl http://localhost:8500/v1/kv/bluegreen/deploy?raw
```


text:
C# Language Design Review, Mar 25, 2015
=======================================

Discussion thread for these notes can be found at https://github.com/dotnet/roslyn/issues/1921.

We've recently changed gears a little on the C# design team. In order to keep a high design velocity, part of the design team meets one or two times each week to do detailed design work. Roughly monthly the full design team gets together to review and discuss the direction. This was the first such review.

Agenda
------

1. Overall direction
2. Nullability features
3. Performance and reliability features
4. Tuples
5. Records
6. Pattern matching
1. Overall direction
====================

In these first two months of design on C# 7 we've adopted a mix of deep dive and breadth scouring. There's agreement that we should be ambitious and try to solve hard problems, being willing to throw the result away if it's not up to snuff. We should keep an open mind for a while still, and not lock down too soon on a specific feature set or specific designs.

2. Nullability features
=======================

Non-nullable types are the number one request on UserVoice. We take it that the *underlying problem* is trying to avoid null reference exceptions. Non-nullable types are at best only part of the solution to this. We'd also need to help prevent access when something is *nullable*.

We've looked at this over a couple of design meetings (#1303, #1648). Ideally we could introduce non-null types, such as `string!` that are guaranteed never to be null. However, the problems around initialization of fields and arrays, etc., simply run too deep. We can never get to full guarantees.

We've been mostly looking at implementation approaches that use type erasure, and that seems like a promising approach.

However, the thing we need to focus on more is this: when you get a nullability warning from this new feature, how do you satisfy the compiler? If you need to use unfamiliar new language features or significant extra syntax to do so, it probably detracts from the feature.

Instead we should at least consider an flow-based approach, where the "null-state" of a variable is tracked based on tests, assignments etc.

``` c#
if (x == null) return;
// not null here
```

It's an open question how far we would go. Would we track only locals and parameters, or would we also keep track of fields?

``` c#
if (foo.x == null) ...
```

This is more problematic, not just because of the risk of other threads changing the field, but also because other code may have side effects on the field or property.

TypeScript uses information about type guards to track union types in if branches, but it's not full flow analysis, and works only for local variables. Google Closure is more heuristics based, and is happy to track e.g. `foo.bar.baz` style patterns.

A core nuisance with nullability checking is that it raises a wealth of compat questions that limit the design in different ways. There may need to be some sort of opt-in to at least some of the diagnostics you'd get, since you wouldn't want them if you were just recompiling old code that used to "work".

3. Performance and reliability
==============================

The list produced at a recent design meeting (#1898) looks sensible.

val / readonly
--------------

We should cross check with Scala on their syntax.

ref return / locals
-------------------

Lots of compexitity - we question whether it is worth the cost?

Never type
----------

The type system approach is interesting, and allows throw expressions.

Method contracts
----------------

`requires` / `ensures`, show up in docs, etc. This looks great. The biggest question is what happens on failure: exceptions? fail fast?

Slices
------

We got strong feedback that array slices are only interesting if we unify them with arrays. Otherwise there's yet another bifurcation of the world. There's *some* value to have a `Slice<T>` struct type just show up in the Framework. But it really doesn't seem worth it unless it's a runtime feature. That unification is really hard to achieve, and would require CLR support. It's valuable enough to try to pursue even with high likelihood of failure.

Slicing as a language syntax could also be "overloadable" - on IEnumerables for instance.

In Go, if you treat a slice as an object, it gets boxed.

Lambda capture lists
--------------------

Not interesting as a feature, but the idea of allowing attributes on lambdas might fly.

Immutable types
---------------

General concern that this doesn't go far enough, is lying to folks, etc. It tries to have strong guarantees that we can't make.

But a lot of people would appreciate *something* here, so the scenario of immutability should continue to motivate us.

Destructible types
------------------

The scenario is good, not the current proposal.

4. Tuple types
==============

There's agreement on wanting the feature and on the syntax (#347, #1207).

We probably prefer a value type version of Tuple<T>. Of course those would be subject to tearing, like all structs. We're willing to be swayed.

There are performance trade offs around allocation vs copying, and also around generic instantiation. We could do some experiments in F# source code, which already has tuples.

5. Records
==========

See #180, #206, #396, #1303, #1572.

In the current proposal, we should just give up on the ability to name constructor parameters and members differently. The motivation was to be able to upgrade where parameter names start with lower case and member names upper case, but it's not worth the complexity.

Should it have `==` and `!=` that are value based? Clashes a little with the ability to make them mutable.

If I introduce extra state, then I have to write my own GetHashCode. That seems unfortunate.

All the gunk today is part of why Roslyn uses XML to generate its data structure. A test of success would be for the Roslyn syntax trees to be concise to write in source code.

A big issue here is incremental non-destructive modification. Roslyn follows the pattern of "Withers", a method for each property that takes a new value for that property and returns a new object that's a copy of the old one except for that property. Withers are painfully verbose to declare, and ideally this feature would offer a solution to that.

Serialization has to work *somehow*, even though many of the members will be generated.

We should not be *too* concerned about the ability to grow up to represent all kinds of things. Start from it being the POD feature, and work from there.

6. Pattern matching
===================

See #180, #206, #1572.

Whether introduced variables are mutable or not is not a key question: we can go with language uniformity or scenario expectation.

Integral value matching is an opportunity to generalize. The pattern 3 may match the value 3 of all integral types rather than just the int 3.

Named matching against all objects and positional against ones that define a positional match. Are recursive patterns necessary? No, but probably convenient and there's no reason not to have them.

Pattern matching could be shoe-horned into current `switch` statements as well as `is` expressions. And we could have a switching *expression* syntax as well:

``` c#
var x = match(e) { p => e, p => e, * => e }
```
An expression version would need to be checked by the compiler for completeness. A little clunky, but much more concise than using a switch.

Similar to a current pattern in the Roslyn code base:

``` c#
Match<T1, T2, TResult>(Func<T1, TResult> f1, Func<T2, TResult> f2) { ... }
var x = Match((string s) => e, (int i) => e);
```

Maybe the fat arrow is not right. We need to decide on syntax. Another option is to use the case keyword instead.

path: /docs/designNotes/2015-03-25 C# Design Review.md

filename: 2015-03-25 C# Design Review.md