text:
**Note: for updates please also check the [fork](https://github.com/tzaeschke/phtree-cpp) by the original PH-Tree developer.**

# PH-Tree C++

The PH-Tree is an ordered index on an n-dimensional space (quad-/oct-/2^n-tree) where each dimension is (by default)
indexed by a 64bit integer. The index order follows z-order / Morton order. The default implementation is effectively
a 'map', i.e. *each key is associated with at most one value.*
Keys are points or boxes in n-dimensional space.

Two strengths of PH-Trees are fast insert/removal operations and scalability with large datasets. It also provides fast
window queries and _k_-nearest neighbor queries, and it scales well with higher dimensions. The default implementation
is limited to 63 dimensions.

The API ist mostly analogous to STL's `std::map`, see function descriptions for details.

Theoretical background is listed [here](#theory).

More information about PH-Trees (including a Java implementation) is available [here](http://www.phtree.org).

----------------------------------

## User Guide

### API Usage

[Key Types](#key-types)

[Basic operations](#basic-operations)

[Queries](#queries)

* [for_each](#for-each-example)

* [Iterators](#iterator-examples)

* [Filters](#filters)

* [Distance Functions](#distance-functions)

[Converters](#converters)

[Custom Key Types](#custom-key-types)

[Restrictions](#restrictions)

[Troubleshooting / FAQ](#troubleshooting-faq)

### Performance

[When to use a PH-Tree](#when-to-use-a-ph-tree)

[Optimising Performance](#optimising-performance)

### Compiling / Building

[Build system & dependencies](#build-system-and-dependencies)

[bazel](#bazel)

[cmake](#cmake)

## Further Resources

[Theory](#theory)

----------------------------------

## API Usage

#### Key Types

The **PH-Tree Map** supports out of the box five types:

- `PhTreeD` uses `PhPointD` keys, which are vectors/points of 64 bit `double`.
- `PhTreeF` uses `PhPointF` keys, which are vectors/points of 32 bit `float`.
- `PhTreeBoxD` uses `PhBoxD` keys, which consist of two `PhPointD` that define an axis-aligned rectangle/box.
- `PhTreeBoxF` uses `PhBoxF` keys, which consist of two `PhPointF` that define an axis-aligned rectangle/box.
- `PhTree` uses `PhPoint` keys, which are vectors/points of `std::int64`

The **PH-Tree MultiMap** supports out of the box three types:

- `PhTreeMultiMapD` uses `PhPointD` keys, which are vectors/points of 64 bit `double`.
- `PhTreeMultiMapBoxD` uses `PhBoxD` keys, which consist of two `PhPointD` that define an axis-aligned rectangle/box.
- `PhTreeMultiMap` uses `PhPoint` keys, which are vectors/points of `std::int64`

Additional tree types can be defined easily analogous to the types above, please refer to the declaration of the tree
types for an example. Support for custom key classes (points and boxes) as well as custom coordinate mappings can be
implemented using custom `Converter` classes, see below. The `PhTreeMultiMap` is by default backed
by `std::unordered_set` but this can be changed via a template parameter.

The `PhTree` and `PhTreeMultiMap` types are available from `phtree.h` and `phtree_multimap.h`.

#### Basic Operations

```C++
class MyData { ... };
MyData my_data;

// Create a 3D point tree with floating point coordinates and a value type of `MyData`.
auto tree = PhTreeD<3, MyData>();

// Create coordinate
PhPointD<3> p{1.1, 1.0, 10.};

// Some operations
tree.emplace(p, my_data);
tree.emplace_hint(hint, p, my_data);
tree.insert(p, my_data);
tree[p] = my_data;
tree.count(p);
tree.find(p);
tree.erase(p);
tree.erase(iterator);
tree.size();
tree.empty();
tree.clear();

// Multi-map only
tree.relocate(p_old, p_new, value);
tree.estimate_count(query);
```

#### Queries

* For-each over all elements: `tree.fore_each(callback);`
* Iterator over all elements: `auto iterator = tree.begin();`
* For-each with box shaped window queries: `tree.fore_each(PhBoxD(min, max), callback);`
* Iterator for box shaped window queries: `auto q = tree.begin_query(PhBoxD(min, max));`
* Iterator for _k_ nearest neighbor queries: `auto q = tree.begin_knn_query(k, center_point, distance_function);`
* Custom query shapes, such as spheres: `tree.for_each(callback, FilterSphere(center, radius, tree.converter()));`

##### For-each example

```C++
// Callback for counting entries
struct Counter {
    void operator()(PhPointD<3> key, T& t) {
        ++n_;
    }
    size_t n_ = 0;
};

// Count entries inside of an axis aligned box defined by the two points (1,1,1) and (3,3,3)
Counter callback;
tree.for_each({{1, 1, 1}, {3, 3, 3}}, callback);
// callback.n_ is now the number of entries in the box.
```

##### Iterator examples

```C++
// Iterate over all entries
for (auto it : tree) {
    ...
}

// Iterate over all entries inside of an axis aligned box defined by the two points (1,1,1) and (3,3,3)
for (auto it = tree.begin_query({{1, 1, 1}, {3, 3, 3}}); it != tree.end(); ++it) {
    ...
}

// Find 5 nearest neighbors of (1,1,1)
for (auto it = tree.begin_knn_query(5, {1, 1, 1}); it != tree.end(); ++it) {
    ...
}
```

##### Filters

All queries allow specifying an additional filter. The filter is called for every key/value pair that would normally be
returned (subject to query constraints) and to every node in the tree that the query decides to traverse (also subject
to query constraints). Returning `true` in the filter does not change query behaviour, returning `false` means that the
current value or child node is not returned or traversed. An example of a geometric filter can be found
in `phtree/common/filter.h` in `FilterAABB`.

```C++
template <dimension_t DIM, typename T>
struct FilterByValueId {
    [[nodiscard]] constexpr bool IsEntryValid(const PhPoint<DIM>& key, const T& value) const {
        // Arbitrary example: Only allow values with even values of id_
        return value.id_ % 2 == 0;
    }
    [[nodiscard]] constexpr bool IsNodeValid(const PhPoint<DIM>& prefix, int bits_to_ignore) const {
        // Allow all nodes
        return true;
    }
};

// Iterate over all entries inside of an axis aligned box defined by the two points (1,1,1) and (3,3,3).
// Return only entries that suffice the filter condition.
for (auto it = tree.begin_query({1, 1, 1}, {3, 3, 3}, FilterByValueId<3, T>())); it != tree.end(); ++it) {
    ...
}
```

##### Distance function

Nearest neighbor queries can also use custom distance metrics, such as L1 distance. Note that this returns a special
iterator that provides a function to get the distance of the current entry:

```C++
#include "phtree/phtree.h"

// Find 5 nearest neighbors of (1,1,1) using L1 distance
for (auto it = tree.begin_knn_query(5, {1, 1, 1}, DistanceL1<3>())); it != tree.end(); ++it) {
    std::cout << "distance = " << it.distance() << std::endl;
    ...
}

```

#### Converters

The PH-Tree can internally only process integer keys. In order to use floating point coordinates, the floating point
coordinates must be converted to integer coordinates. The `PhTreeD` and `PhTreeBoxD` use by default the
`PreprocessIEEE` & `PostProcessIEEE` functions. The `IEEE` processor is a loss-less converter (in terms of numeric
precision) that simply takes the 64bits of a double value and treats them as if they were a 64bit integer
(it is slightly more complicated than that, see discussion in the papers referenced above). In other words, it treats
the IEEE 754 representation of the double value as integer, hence the name `IEEE` converter.

The `IEEE` conversion is fast and reversible without loss of precision. However, it has been shown that other converters
can result in indexes that are up to 20% faster. One useful alternative is a `Multiply` converter that convert floating
point to integer by multiplication and casting:

```C++
double my_float = ...;
// Convert to int
std::int64_t my_int = (std::int64_t) my_float * 1000000.;

// Convert back
double resultung_float = ((double)my_int) / 1000000.;
```

It is obvious that this approach leads to a loss of numerical precision. Moreover, the loss of precision depends on the
actual range of the double values and the constant. The chosen constant should probably be as large as possible but
small enough such that converted values do not exceed the 64bit limit of `std::int64_t`. Note that the PH-Tree provides
several `ConverterMultiply` implementations for point/box and double/float.

```C++
template <dimension_t DIM>
struct MyConverterMultiply : public ConverterPointBase<DIM, double, scalar_64_t> {
    explicit MyConverterMultiply(double multiplier)
    : multiplier_{multiplier}, divider_{1. / multiplier} {}

    [[nodiscard]] PhPoint<DIM> pre(const PhPointD<DIM>& point) const {
        PhPoint<DIM> out;
        for (dimension_t i = 0; i < DIM; ++i) {
            out[i] = point[i] * multiplier_;
        }
        return out;
    }

    [[nodiscard]] PhPointD<DIM> post(const PhPoint<DIM>& in) const {
        PhPointD<DIM> out;
        for (dimension_t i = 0; i < DIM; ++i) {
            out[i] = ((double)in[i]) * divider_;
        }
        return out;
    }

    [[nodiscard]] auto pre_query(const PhBoxD<DIM>& query_box) const {
        return PhBox{pre(query_box.min()), pre(query_box.max())};
    }

    const double multiplier_;
    const double divider_;
};

template <dimension_t DIM, typename T>
using MyTree = PhTreeD<DIM, T, MyConverterMultiply<DIM>>;

void test() {
    MyConverterMultiply<3> converter{1000000};
    MyTree<3, MyData> tree(converter);
    ...  // use the tree
}
```

It is also worth trying out constants that are 1 or 2 orders of magnitude smaller or larger than this maximum value.
Experience shows that this may affect query performance by up to 10%. This is due to a more compact structure of the
resulting index tree.

##### Custom key types

With custom converters it is also possible to use your own custom classes as keys (instead of `PhPointD` or `PhBoxF`).
The following example defined custom `MyPoint` and `MyBox` types and a converter that allows using them with a `PhTree`:

```c++
struct MyPoint {
    double x_;
    double y_;
    double z_;
};

using MyBox = std::pair<MyPoint, MyPoint>;

class MyConverterMultiply : public ConverterBase<3, 3, double, scalar_64_t, MyPoint, MyBox> {
    using BASE = ConverterPointBase<3, double, scalar_64_t>;
    using PointInternal = typename BASE::KeyInternal;
    using QueryBoxInternal = typename BASE::QueryBoxInternal;

  public:
    explicit MyConverterMultiply(double multiplier = 1000000)
    : multiplier_{multiplier}, divider_{1. / multiplier} {}

    [[nodiscard]] PointInternal pre(const MyPoint& point) const {
        return {static_cast<long>(point.x_ * multiplier_),
                static_cast<long>(point.y_ * multiplier_),
                static_cast<long>(point.z_ * multiplier_)};
    }

    [[nodiscard]] MyPoint post(const PointInternal& in) const {
        return {in[0] * divider_, in[1] * divider_, in[2] * divider_};
    }

    [[nodiscard]] QueryBoxInternal pre_query(const MyBox& box) const {
        return {pre(box.first), pre(box.second)};
    }

  private:
    const double multiplier_;
    const double divider_;
};

void test() {
    MyConverterMultiply tm;
    PhTree<3, Id, MyConverterMultiply> tree(tm);
    ... // use the tree
}
```

#### Restrictions

* **C++**: Supports value types of `T` and `T*`, but not `T&`
* **C++**: Return types of `find()`, `emplace()`, ... differ slightly from `std::map`, they have function `first()`
  , `second()` instead of fields of the same name.
* **General**: PH-Trees are **maps**, i.e. each coordinate can hold only *one* entry. In order to hold multiple values
  per coordinate please use the `PhTreeMultiMap` implementations.
* **General**: PH-Trees order entries internally in z-order (Morton order). However, the order is based on the (
  unsigned) bit representation of keys, so negative coordinates are returned *after* positive coordinates.
* **General**: The current implementation support between 2 and 63 dimensions.
* **Differences to std::map**: There are several differences to `std::map`. Most notably for the iterators:
    * `begin()`/`end()` are not comparable with `<` or `>`. Only `it == tree.end()` and `it != tree.end()` is supported.
    * Value of `end()`: The tree has no linear memory layout, so there is no useful definition of a pointer pointing _
      after_ the last entry or any entry. This should be irrelevant for normal usage.

### Troubleshooting / FAQ

**Problem**: The PH-Tree appears to be losing updates/insertions.

**Solution**: Remember that the PH-Tree is a *map*, keys will not be inserted if an identical key already exists. The
easiest solution is to use one of the `PhTreeMultiMap` implementations. Alternatively, this can be solved by turning the
PH-Tree into a multi-map, for example by using something like `std::map` or `std::set` as member type:
`PhTree<3, std::set<MyDataClass>>`. The `set` instances can then be used to handle key conflicts by storing multiple
entries for the same key. The logic to handle conflicts must currently be implemented manually by the user.

----------------------------------

## Performance

### When to use a PH-Tree

The PH-Tree is a multi-dimensional index or spatial index. This section gives a rough overview how the PH-Tree compares
to other spatial indexes, such as *k*D-trees, R-trees/BV-hierarchies or quadtrees.

Disclaimer: This overview cannot be comprehensive (there are 100s of spatial indexes out there) and performance depends
heavily on the actual dataset, usage patterns, hardware, ... .

**Generally, the PH-Tree tends to have the following advantages:**

* Fast insertion/removal times. While some indexes, such as *k*-D-trees, trees can be build from scratch very fast, they
  tend to be be much slower when removing entries or when indexing large datasets. Also, most indexes require
  rebalancing which may result in unpredictable latency (R-trees) or may result in index degradation if delayed
  (*k*D-trees).

* Competitive query performance. Query performance is generally comparable to other index structures. The PH-Tree is
  fast at looking up coordinates but requires more traversal than other indexes. This means it is especially efficient
  if the query results are 'small', e.g. up to 100 results per query.

* Scalability with large datasets. The PH-Tree's insert/remove/query performance tends to scale well to large datasets
  with millions of entries.

* Scalability with the number of dimensions. The PH-Tree has been shown to deal "well" with high dimensional data (
  1000k+ dimensions). What does "well" mean?
    * It works very well for up to 30 (sometimes 50) dimensions. **Please note that the C++ implementation has not been
      optimised nearly as much as the Java implementation.**
    * For more dimensions (Java was tested with 1000+ dimensions) the PH-Tree still has excellent insertion/deletion
      performance. However, the query performance cannot compete with specialised high-dim indexes such as cover-trees
      or pyramid-trees (these tend to be *very slow* on insertion/deletion though).

* Modification operations (insert/delete) in a PH-Tree are guaranteed to modify only one Node (potentially
  creating/deleting a second one). This guarantee can have advantages for concurrent implementations or when serializing
  the index. Please note that this advantage is somewhat theoretical because this guarantee is not exploited by the
  current implementation (it doesn't support concurrency or serialization).

**PH-Tree disadvantages:**

* A PH-Tree is a *map*, not a *multi-map*. This project also provides `PhTreeMultiMap` implementations that store a
  hash-set at each coordinate. In practice, the overhead of storing sets appears to be usually small enough to not
  matter much.

* PH-Trees are not very efficient in scenarios where queries tend to return large result sets in the order of 1000 or
  more.

### Optimising Performance

There are numerous ways to improve performance. The following list gives an overview over the possibilities.

1) **Use `for_each` instead of iterators**. This should improve performance of queries by 5%-10%.

2) **Use `emplace_hint` if possible**. When updating the position of an entry, the naive way is to use `erase()`
   /`emplace()`. With `emplace_hint`, insertion can avoid navigation to the target node if the insertion coordinate is
   close to the removal coordinate.
    ```c++
    auto iter = tree.find(old_position);
    tree.erase(iter);
    tree.emplace_hint(iter, new_position, value);
    ```

3) **Store pointers instead of large data objects**. For example, use `PhTree<3, MyLargeClass*>` instead of
   `PhTree<3, MyLargeClass>` if `MyLargeClass` is large.
    * This prevents the PH-Tree from storing the values inside the tree. This should improve cache-locality and thus
      performance when operating on the tree.
    * Using pointers is also useful if construction/destruction of values is expensive. The reason is that the tree has
      to construct and destruct objects internally. This may be avoidable but is currently still happening.

4) **Use non-box query shapes**. Depending on the use case it may be more suitable to use a custom filter for queries.
   For example:

   `tree.for_each(callback, FilterSphere(center, radius, tree.converter()));`

5) **Use a different data converter**. The default converter of the PH-Tree results in a reasonably fast index. Its
   biggest advantage is that it provides lossless conversion from floating point coordinates to PH-Tree coordinates
   (integers) and back to floating point coordinates.
    * The `ConverterMultiply` is a lossy converter but it tends to improve performance by 10% or more. This is not
      caused by faster operation in the converter itself but by a more compact tree shape. The example shows how to use
      a converter that multiplies coordinates by 100'000, thus preserving roughly 5 fractional digits:

      `PhTreeD<DIM, T, ConverterMultiply<3, 100 * 1000, 1>>`

6) **Use custom key types**. By default, the PH-Tree accepts only coordinates in the form of its own key types, such
   as `PhPointD`, `PhBoxF` or similar. To avoid conversion from custom types to PH-Tree key types, custom classes can
   often be adapted to be accepted directly by the PH-Tree without conversion. This requires implementing a custom
   converter as described in the section about [Custom Key Types](#custom-key-types).

7) Advanced: **Adapt internal Node representation**. Depending on the dimensionality `DIM`, the PH-Tree uses internally
   in
   `Nodes` different container types to hold entries. By default, it uses an array for `DIM<=3`, a vector for `DIM<=8`
   and an ordered map for `DIM>8`. Adapting these thresholds can have strong effects on performance as well as memory
   usage. One example: Changing the threshold to use vector for `DIM==3` reduced performance of the `update_d` benchmark
   by 40%-50% but improved performance of `query_d` by 15%-20%. The threshold is currently hardcoded.
   The effects are not always easy to predict but here are some guidelines:
    * "array" is the fastest solution for insert/update/remove type operations. Query performance is "ok". Memory
      consumption is
      **O(DIM^2)** for every node regardless of number of entries in the node.
    * "vector" is the fastest for queries but has for large nodes **worst case O(DIM^2)** insert/update/remove
      performance.
    * "map" scales well with `DIM` but is for low values of `DIM` generally slower than "array" or "vector".

----------------------------------

## Compiling the PH-Tree

This section will guide you through the initial build system and IDE you need to go through in order to build and run
custom versions of the PH-Tree on your machine.

### Build system & dependencies

PH-Tree can be built with *cmake 3.14* or [Bazel](https://bazel.build) as build system. All code is written in C++
targeting the C++17 standard. The code has been verified to compile on Linux with Clang 9, 10, 11, 12, and GCC 9, 10,
11, and on Windows with Visual Studio 2019.

#### Ubuntu Linux

* Installing [clang](https://apt.llvm.org/)

* Installing [bazel](https://docs.bazel.build/versions/main/install-ubuntu.html)

* To install [cmake](https://launchpad.net/~hnakamur/+archive/ubuntu/cmake):

```
sudo add-apt-repository ppa:hnakamur/libarchive
sudo add-apt-repository ppa:hnakamur/libzstd
sudo add-apt-repository ppa:hnakamur/cmake
sudo apt update
sudo apt install cmake
```

#### Windows

To build on Windows, you'll need to have a version of Visual Studio 2019 installed (likely Professional), in addition to
[Bazel](https://docs.bazel.build/versions/master/windows.html) or
[cmake](https://cmake.org/download/).

### Bazel

Once you have set up your dependencies, you should be able to build the PH-Tree repository by running:

```
bazel build ...
```

Similarly, you can run all unit tests with:

```
bazel test ...
```

### cmake

```
mkdir build
cd build
cmake ..
cmake --build .
./example/Example
```

## Further Resources

### Theory

The PH-Tree is discussed in the following publications and reports:

- T. Zaeschke, C. Zimmerli, M.C. Norrie:
  "The PH-Tree -- A Space-Efficient Storage Structure and Multi-Dimensional Index", (SIGMOD 2014)
- T. Zaeschke: "The PH-Tree Revisited", (2015)
- T. Zaeschke, M.C. Norrie: "Efficient Z-Ordered Traversal of Hypercube Indexes" (BTW 2017).

repository: improbable-eng/phtree-cpp

repo_name: improbable-eng/phtree-cpp


text:
---
title: Bind a service to a Namespace
type: Tutorials
---

This guide shows how to bind a service provided by an external solution to a Namespace in Kyma. To execute the binding, provision a Service Instance for the registered service in the Namespace. Follow the instructions to bind your service to the desired Namespace.

> **NOTE:** To learn more about Service Instances and the Service Catalog domain in Kyma, see the [Service Catalog](/components/service-catalog/) topic.

## Prerequisites

- An Application created in your cluster and bound to the desired Namespace.
- A service provided by an external solution registered in the Application.

## Steps

1. Export the name of the desired Namespace and the name of the Service Instance.

   ```bash
   export NAMESPACE={DESIRED_NAMESPACE}
   export INSTANCE_NAME={SERVICE_INSTANCE_NAME}
   ```

2. Expose the `externalName` of the Service Class of the registered service.

   ```bash
   export EXTERNAL_NAME=$(kubectl -n $NAMESPACE get serviceclass {SERVICE_ID} -o jsonpath='{.spec.externalName}')
   ```

   > **NOTE:** `{SERVICE_ID}` is the identifier of the registered service.

3. Create a Service Instance for the registered service.

   ```bash
   cat <<EOF | kubectl apply -f -
   apiVersion: servicecatalog.k8s.io/v1beta1
   kind: ServiceInstance
   metadata:
     name: $INSTANCE_NAME
     namespace: $NAMESPACE
   spec:
     serviceClassExternalName: $EXTERNAL_NAME
   EOF
   ```

4. Check if the Service Instance was created successfully. The last condition in the CR status should state `Ready True`:

   ```bash
   kubectl get serviceinstance $INSTANCE_NAME -n $NAMESPACE -o=jsonpath="{range .status.conditions[*]}{.type}{'\t'}{.status}{'\n'}{end}"
   ```

repository: Rider-io/kyma

branch_name: refs/heads/main

extension: md

license_type: permissive

repo_name: Rider-io/kyma


text:
# Kotlin Examples
Miscellaneous examples ported to Kotlin as standalone Gradle projects.

## graphql-ktor
A mock GraphQL service running on Ktor.

Uses graphql-java for the underlying engine without Spring or any dependency injection.
Showcases how Kotlin's builder and extension syntax can be used to simplify wiring of resolvers.

## opencl-lwjgl
Porting the OpenCL example from LJWGL to Kotlin.

This demonstrates how you can take advantage of GPU acceleration for data parallel calculations.

I would recommend taking advantage of Kotlin's coroutine API to work with OpenCL asynchronously.

license: MIT

repository: patonw/kotlin-examples

branch_name: refs/heads/master

repo_name: patonw/kotlin-examples


text:
Title: Apache Lucene 6.4.2 and Apache Solr 6.4.2 Available
category: core/news
URL:
save_as:
slug: core-apache-lucene-642-and-apache-solr-642-available

The Lucene PMC is pleased to announce the release of Apache Lucene 6.4.2

Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for nearly
any application that requires full-text search, especially cross-platform.

This release contains numerous bug fixes, optimizations, and
improvements, some of which are highlighted below.
The release is available for immediate download at:

  <https://www.apache.org/dyn/closer.lua/lucene/java/6.4.2>

See the [CHANGES.txt](/core/6_4_2/changes/Changes.html) file included with the
release for a full list of changes and further details.

### Highlights of this Lucene release include:

  * Fixed: CommonGramsQueryFilter was producing a disconnected token graph, messing up phrase queries during query parsing

repository: apache/lucene-site

gha_language: CSS

repo_name: apache/lucene-site


text:
# notion-repackaged

Notion Repackaged is an effort to bring the Notion app to all major Linux distributions as well as providing a variant with [notion-enhancer](https://github.com/notion-enhancer/notion-enhancer) embedded for both Windows, Linux and MacOS.

> This project is heavily inspired on [notion-linux](https://github.com/davidbailey00/notion-linux) and [AUR notion-app-enhanced](https://aur.archlinux.org/packages/notion-app-enhanced/) but extending on the idea of embedding notion-enhancer, providing an easy way to use notion-enhancer on all operating systems.

## Installation

> :warning: Make sure you don't run the official (or vanilla) and enhanced versions at the same time. You should only install one or the other.

Downloads are available at the repository releases section.
You can go directly to [the latest release](https://github.com/notion-enhancer/notion-repackaged/releases/latest) and download the build for the platform you are using.

### Windows

Download the `exe` file from the release and install it, should be a pretty straightforward experience.

You can also download the `zip` build for Windows which provides a portable way to run the application.

### Linux

For linux there are two variants, `vanilla` and `enhanced`. The vanilla is a direct port of the official sources and the enhanced includes `notion-enhancer`.

Currently, we target the following package types:
- AppImage (any distribution, not actively tested)
- deb (Debian, Ubuntu, Pop!_OS, Linux Mint)
- rpm (Fedora)
- pacman (Arch Linux, Manjaro)
- zip (any, portable)

#### AppImage

AppImages are binary installers that are compatible with any Linux distribution, to install them you can just run them.

#### deb

You can add our repository to your package manager by running

```
echo "deb [trusted=yes] https://apt.fury.io/notion-repackaged/ /" | sudo tee /etc/apt/sources.list.d/notion-repackaged.list
```

> :information_source: For the new repository to be able to provide our packages, you will have to run `sudo apt update` first

With that you will be able to install `notion-app` or `notion-app-enhanced` using `sudo apt install <package name>`

Alternatively, download manually and run `sudo dpkg -i <downloaded file>.deb` to install the package.

#### rpm

> :warning: These instructions assume you are using dnf/yum in Fedora or alikes. Instructions might vary depending on the distribution you are using.

You can add our repository to your package manager by creating the file `/etc/yum.repos.d/notion-repackaged.repo` with the following contents:

```
[notion-repackaged]
name=Notion Repackaged Repo
baseurl=https://yum.fury.io/notion-repackaged/
enabled=1
gpgcheck=0
```

With that you will be able to install `notion-app` or `notion-app-enhanced` using `sudo dnf install <package name>`

Alternatively, download manually and run `sudo rpm -i <downloaded file>.rpm` to install the package.

#### pacman

You can install the [notion-app](https://aur.archlinux.org/packages/notion-app/) or [notion-app-enhanced](https://aur.archlinux.org/packages/notion-app-enhanced) packages for the vanilla or enhanced versions respectively.

Alternatively, download manually and run `sudo pacman -U <downloaded file>.pacman` to install the package.

### MacOS

> :warning: As of now, the M1 (arm64) build is non-functional due to [electron-userland/electron-builder#5850](https://github.com/electron-userland/electron-builder/issues/5850), you can try using the regular build thanks to Rosetta. You can also try using the arm64 zip build that might work.

We also build the enhanced variant for MacOS but there are no concrete instructions as I have no way of trying the builds for myself.

In theory, you should be able to download the `dmg` build and drag it to your dock to install it.

The dmg archive produced is not signed, so chances are you'll not be able to use it, or even install it.

Please, take look at the official instructions from Apple on how to install unsigned apps: https://support.apple.com/en-us/HT202491

## Issues

If you find yourself not being able to use our builds, please create an issue with any and all information you can provide on how to reproduce and debug this issue. Screenshots and terminal logs are vital to being able to reproduce and fix the issues.

If you have an issue with the enhanced variant, please try using the vanilla variant. This step is crucial on the way to figuring our the root of the issue. Do not create issues in notion-enhancer before making sure it's not our fault, we modify notion-enhancer's internals heavily.

## Contributing

Any contribution is welcome, be it documentation, code improvements or fixes.
We don't plan on improving notion-enhancer, we aim to be compatible with it and track their changes.

## Disclaimer

This project extracts the source code of the publicly-available binaries on the [official Notion website](https://www.notion.so/desktop), meaning all code contained within our packages is owned by Notion.

Additionally, in the enhanced variant, we embed [notion-enhancer](https://github.com/notion-enhancer/notion-enhancer) which is MIT licensed. All code contained in the `embedded_enhancer` directory and the icons in the `assets` directory are owned by them.

This project is made as a best-effort, integrating and supporting all Linux distributions as well as other OSes like Mac and Windows is very complicated.

We are not liable for any problems caused by this (or included) software.

repository: Arecsu/notion-repackaged

gha_license_id: MIT

repo_name: Arecsu/notion-repackaged


text:
# Pyxidiophora marchalii (Sacc.) N. Lundq., 1980 SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Bot. Notiser 133(2): 139 (1980)

#### Original name
Rhynchomyces marchalii Sacc. & Marchal, 1885

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
Unimod-mapper
=============

![RTD](https://readthedocs.org/projects/unimod-mapper/badge/?version=latest)
![Travis-CI](https://travis-ci.org/computational-ms/unimod-mapper.svg?branch=master)

Unimod is the public domain database for amino acid modifications. For more detail, please see [here](http://www.unimod.org/unimod_help.html).
This Python module offers helper functions to query the unimod.xml from [unimod.org](http://www.unimod.org/modifications_list.php?)

Originally, the unimod-mapper was part of different Python packages from the fufezan-lab, but we decided to create its own package for the sake of modularity.

Contributors
------------

* J. Leufken
* S. Schulze
* M. Koesters
* C. Fufezan

repository: Seb-Leb/unimod-mapper

repo_name: Seb-Leb/unimod-mapper


text:
An attempt to create a basic liquid simulation using cellular-autonoma like rules

Rules:
- Cells have 3 states: background (0), wall (1), and liquid (2)
- Only liquid cells have rules applied to them
- A liquid cells rules per generation happens like this:
  + If a cell has a BG cell below it, move there, if not, look at next statement
  + If a cell has both south east and south west BG cells, pick a random one and move there, if not, look at next statement
  + If a cell has either a south east or south west BG cell, move there, if not, look at next statement
  + If a cell has both east and west BG cells, pick a random one and move there, if not, look at next statement
  + If a cell has either an east BG cell or west BG cell, move there, if not, look at next statement
  + Do nothing


text:
Slugify Core
=======

### This is a fork of the original project here: [https://github.com/fcingolani/Slugify](https://github.com/fcingolani/Slugify). This has been updated for .NET Standard 2.0 support (older versions support .NET Standard down to 1.3).

[![Build Status](https://dev.azure.com/chadtolkien/Slugify/_apis/build/status/ctolkien.Slugify?branchName=master)](https://dev.azure.com/chadtolkien/Slugify/_build/latest?definitionId=6&branchName=master)
[![codecov](https://codecov.io/gh/ctolkien/Slugify/branch/master/graph/badge.svg)](https://codecov.io/gh/ctolkien/Slugify)
![Version](https://img.shields.io/nuget/v/slugify.core.svg?maxAge=2000)
[![license](https://img.shields.io/github/license/ctolkien/Slugify.svg?maxAge=2592000)]()

Simple [Slug / Clean URL](http://en.wikipedia.org/wiki/Slug_%28web_publishing%29#Slug) generator helper for Microsoft .NET framework.

With default settings, you will get an **hyphenized**, **lowercase**, **alphanumeric** version of any string you please, with any [diacritics](http://en.wikipedia.org/wiki/Diacritic) removed and collapsed whitespace, collapsed dashes and trimmed whitespace.

In example, having:

> a ambição cerra o coração

You'll get:

> a-ambicao-cerra-o-coracao

Installation
------------

You can get the [Slugify NuGet package](https://www.nuget.org/packages/Slugify.Core/) by running the following command in the [Package Manager Console](http://docs.nuget.org/docs/start-here/using-the-package-manager-console):

```
PM> Install-Package Slugify.Core
```

Upgrading from 2.x to 3.x
-------------------------

* 3.0 is a significantly faster and less memory intensive version of the Slugifier. Whilst effort has been made to maintain backwards compatability, there may be some breaking changes.
* The `SlugHelper.Config` nested class has been renamed to just `SlugHelperConfiguration`.

Basic Usage
-----------

It's really simple! Just instantiate _SlugHelper_ and call its _GenerateSlug_ with the **string** you want to convert; it'll return the URL Safe version:

```csharp
using Slugify;

public class MyApp
{
   public static void Main()
   {
      SlugHelper helper = new SlugHelper();

      String title = "OLA ke ase!";

      String slug = helper.GenerateSlug(title); // "ola-ke-ase"

      Console.WriteLine(slug);
   }
}

```

Configuration
-------------

You can provide a _SlugHelperConfiguration_ instance to _SlugHelper_'s constructor to customize the helper's behavior:

```csharp
// Creating a configuration object
var config = new SlugHelperConfiguration();

// Replace spaces with a dash
config.StringReplacements.Add(" ", "-");

// We want a lowercase Slug
config.ForceLowerCase = true;

// Will collapse multiple seqential dashes down to a single one
config.CollapseDashes = true;

// Will trim leading and trailing whitespace
config.TrimWhitespace = true;

// Colapse consecutive whitespace chars into one
config.CollapseWhiteSpace = true;

// Remove everything that's not a letter, number, hyphen, dot, or underscore
config.DeniedCharactersRegex = @"[^a-zA-Z0-9\-\._]";

// Create a helper instance with our new configuration
SlugHelper helper = new SlugHelper(config);
```

In fact, the above values are so common they're the default ones! So last code could be rewritten as:

```csharp
var config = new SlugHelperConfiguration();
SlugHelper helper = new SlugHelper(config);
```

One more thing: _SlugHelperConfiguration_ is used when you call the parameterless _SlugHelper_ constructor. Then ...

```csharp
SlugHelper helper = new SlugHelper();
```

... is the same as running the code we had in first place.

### Options

#### CharacterReplacements

Type: _Dictionary<String, String>_. Default: [" ": "-"].

Will replace the specified keys with their associated value.

By default, will replace spaces with hyphens.

#### ForceLowerCase

Type: _Boolean_. Default: **true**.

Setting it to true will convert output string to be LowerCase. If false, original casing will be preserved.

#### CollapseWhiteSpace

Type: _Boolean_. Default: **true**.

Setting it to true will replace consecutive whitespace characters by just one space (" ").

#### DeniedCharactersRegex

Type: _String_. Default: **[^a-zA-Z0-9\-\._]**.

Any character matching this Regular Expression will be deleted from the resulting string.

#### CollapseDashes

Type: _Boolean_. Default: **true**

This will condense multiple dashes (e.g. `foo---bar`) down to a single dash (`foo-bar`). This is useful to avoid scenarios like `foo & bar` becoming `foo--bar`.

License
-------

The MIT License (MIT)

Copyright (c) 2013 Federico Cingolani

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

repository: Peadey/Slugify

repo_name: Peadey/Slugify


text:
::: tip

[Code-Pop/Intro-to-Vue-3: Code for Vue Mastery's Intro to Vue 3 course: (github.com)](https://github.com/Code-Pop/Intro-to-Vue-3)

[Request a discount - GitHub Education](https://education.github.com/discount_requests/student_application) 1193

<EMAIL_ADDRESS>Ro

[Educative: Interactive Courses for Software Developers](https://www.educative.io/learn)

[Q10 VIKING's Frontend Masters](https://frontendmasters.com/dashboard/)<PHONE_NUMBER>@qq :star:

[Coding Interview Practice | Interview Cake](https://www.interviewcake.com/table-of-contents) :star:

:::

win+G 菜单

win+alt+r开始录制

```
I am a junior student from China, and I used Github to share my code with my classmates and friends since I entered university.
```

path: /docs/vue3/18 vuemaster学习.md

filename: 18 vuemaster学习.md


text:
This sample is for API-M 4.0.0

This sample will write analytics event details to a log.

Build the project using Maven:

        mvn clean install

Copy the generated JAR file from the target folder and place it in wso2am-4.0.0/repository/components/lib.

Edit apim.analytics configurations in the deployment.toml located inside wso2am-4.0.0/repository/conf with the
following configuration.

        [apim.analytics]
        enable = true
        properties."publisher.reporter.class" = "org.wso2.am.analytics.publisher.sample.reporter.CustomReporter"
        logger.reporter.level = "INFO"

To enable logging for reporter, edit log4j2.properties file located inside wso2am-4.0.0/repository/conf

Add reporter to the loggers list

        loggers = reporter, ...(list of other available loggers)

Add bellow configurations after the loggers

        logger.reporter.name = org.wso2.am.analytics.publisher.sample.reporter
        logger.reporter.level = INFO

Once this is successfully deployed a sample log will look like bellow,

        INFO - LogCounterMetric Metric Name: apim:response Metric Value: {apiName=PizzaShackAPI, proxyResponseCode=200,
        errorType=null, destination=https://localhost:9443/am/sample/pizzashack/v1/api/,
        apiCreatorTenantDomain=carbon.super, platform=Linux, apiMethod=GET, apiVersion=1.0.0, gatewayType=SYNAPSE,
        apiCreator=admin, responseCacheHit=false, backendLatency=700,
        correlationId=f2566443-04f9-4865-9612-40a4d8bd6318, requestMediationLatency=102, keyType=PRODUCTION,
        apiId=3076a598-65f0-4b8e-a36e-50ecd797aa1b, applicationName=internal-key-app, targetResponseCode=200,
        requestTimestamp=2021-07-13T09:23:35.311Z, applicationOwner=internal-key-app, userAgent=Chrome,
        eventType=response, apiResourceTemplate=/menu, responseLatency=804, regionId=default,
        responseMediationLatency=2, userIp=<IP_ADDRESS>, applicationId=118fd1a4-73aa-3969-8c72-4f5cc06b1b0a, apiType=HTTP}


text:
# [mml.html](https://mathjax.github.io/MathJax-demos-web/custom-tex-extension/mml.html)

This example shows how to create a custom TeX extension that defines new TeX commands implemented by javascript functions.

The commands implemented by this example provide the ability to generate MathML token elements from within TeX by hand.  This allows more control over the content and attributes of the elements produced.  The macros are `\mi`, `\mo`, `\mn`, `\ms`, and `\mtext`, and they each take an argument that is the text to be used as the content of the corresponding MathML element.  The text is not further processed by TeX, but the extension does convert sequences of the form `\uNNNN` (where the N's are hexadecimal digits) into the corresponding unicode character; e.g., `\mi{\u2460}` would produce U+2460, a circled digit 1.

The main code for the extension is

* [mml.js](mml.js)

which contains comments describing it in detail.  In order to use the extension in your web pages, you must turn this into a MathJax component file, which you do by first defining the component using the file

* [webpack.config.js](webpack.config.js)

which gives the name of the component, the other components that this extension will draw on (so that only the needed files will be included in this component), along with some other data about directories.  The final directory being set to `'.'` means that the component will be placed in the directory with the source file, but with `.min.js` as the extension rather than '.js'.

To make the final component, use the commands

```
npm install
npm run make-custom-tex-extension
```

from the main directory of this repository.  That will create the `mml.min.js` file in the `custom-tex-extension` directory.

To use this in your own web page, the key lines are

``` html
  <script>
  MathJax = {
    loader: {
      load: ['[custom]/mml.min.js'],
      paths: {custom: '.'}
    },
    tex: {
      packages: {'[+]': ['mml']}
    }
  };
  </script>
  <script id="MathJax-script" async<EMAIL_ADDRESS>```

This asks the loader to load the custom extension file, and defines the `[custom]` path to be the directory relative to the `mml.html`  example file (`mml.min.js` is in the same directory).  The tex component is configured to add the `mml` package to the other packages already there by default.  Finally, the `tex-chtml` component is loaded, and MathJax does the rest.

The expressions in the example file illustrate the use of the custom macros.  For example

``` latex
$$abc + \mi{abc} \mo{++} x ++ y$$
```

shows the difference between `abc`, which produces three separate `mi` elements internally, each containing one letter, and `\mi{abc}`, which produces one `mi` element containing three letters.  The latter will be in upwrite letters (as the default variant for `mi` elements depends on the number of characters in the content).  View the MathML source to see the difference.

Similarly the difference between `\mo{++}` and `++` is the number of `mo` elements and the length of their content.  `\mo{++}` produces `<mo>++</mo>` while `++` produces `<mo>+</mo><mo>+</mo>`.  Note the difference in output for these two: the TeX spacing rules for `x ++ y` means that it will be spaced as `x + +y`, while `x \mo{++} y` will be `x ++ y`.

The expression

``` latex
\mi[mathvariant="bold"]{A}
```

produces a bold A via `<mi mathvariant="bold">A</mi>`.

There are additional example equations in the HTML file.

[Run the example](https://mathjax.github.io/MathJax-demos-web/custom-tex-extension/mml.html)

repository: mathjax/MathJax-demos-web

gha_language: HTML

repo_name: mathjax/MathJax-demos-web


text:
fluxex
======

An extended flux implement for both server/client side.

[![npm version](https://img.shields.io/npm/v/fluxex.svg)](https://www.npmjs.org/package/fluxex) [![Dependency Status](https://david-dm.org/zordius/fluxex.png)](https://david-dm.org/zordius/fluxex)  [![Build Status](https://travis-ci.org/zordius/fluxex.svg?branch=master)](https://travis-ci.org/zordius/fluxex) [![Test Coverage](https://codeclimate.com/github/zordius/fluxex/badges/coverage.svg)](https://codeclimate.com/github/zordius/fluxex) [![Code Climate](https://codeclimate.com/github/zordius/fluxex/badges/gpa.svg)](https://codeclimate.com/github/zordius/fluxex) [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE.txt)

See the Fluxex Magic
--------------------

Check example projects you can see how fluxex do server side rendering + context deliver + Full HTML react rendering!

So far we do not include these in the example, but ....coming soon!

* api call cross server/client
* complex action creator

Difference with Flux
--------------------

Fluxex is context based flux implemention. Server side react rendering can be done easy when the flux is scoped under a request based context. Store and dispatcher are singletons in Facebook flux, but in fluxex they are not.

**Fluxex**
* Fluxex is an instance, it is constructed by provided context.
* Fluxex can be serialized by `.toString()` and reconstructed by the serialized string. All server side store status can be transfered to client side by this way.

**Action**
* An action creator should return a promise.
* An action creator function can be executed with `.executeAction()` method under a Fluxex instance.
* When the action be `.executeAction()`, the Fluxex instance can be referenced by `this`.
* `.executeAction()` will return a promise, so you can manage asynchronous actions in promise way.

**Dispatcher**
* the Fluxex instance itself is a dispatcher with `.dispatch()` method.
* the dispatched `FOOBAR` action will trigger all stores with `handleFOOBAR` method under the Fluxex instance.

**Store**
* Store is an instance, it is constructed by serialized status.
* Store is created by a Fluxex object automatically.
* Store naming and prototype information are provided by the `.stores` property of Fluxex instance.
* Use `.getStore(name)` to get the store by name from an Fluxex instance.

Usage
-----

**Define your app**

Provide store `{name: implementation}` pairs and Html.jsx location (example):
```javascript
'use strict';

var commonStores = require('fluxex/extra/commonStores');

module.exports = require('fluxex').createApp({
    page: commonStores.page,
    productStore: require('./stores/product')
}, process.cwd() + '/components/Html.jsx');
```

.... to be continue ...

repository: brianhph/fluxex

repo_name: brianhph/fluxex


text:
---
layout: default
---

# 1993-03-02 - Future of anonymity (short-term vs. long-term)

## Header Data

From: Eric Hughes<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: e4aae2780bfbca661958bbb82dba9c19f8fa3a65007cb2ddd553535daef6339d

Message ID<EMAIL_ADDRESS>Reply To: \<9303020037.AA26459@SOS\>

UTC Datetime: 1993-03-02 01:47:45 UTC

Raw Date: Mon, 1 Mar 93 17:47:45 PST

## Raw message

```
{% raw  %}From: Eric Hughes<EMAIL_ADDRESS>Date: Mon, 1 Mar 93 17:47:45 PST
To<EMAIL_ADDRESS>Subject: Future of anonymity (short-term vs. long-term)
In-Reply-To: <9303020037.AA26459@SOS>
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

Re: authenticated news software

>Also, keep in mind that netnews is run
>by lots of real sites, who have real assets which could be reached if
>sued by RSA.

Why is there a presumption that any such authenticated news software
would be used without license?  RSADSI is not trying to sit on their
patents, but to make money from them.

>There doesn't seem to be a lot of realism in these discussions, which is
>really bothering me.

What you believe to be real and what I believe to be real may be
different.  To claim that another is being unrealistic is to mask
what is foremost a difference in belief.

What assumptions here do you disagree with?  If you are explicit,
perhaps we can forge an agreement.

Eric

{% endraw %}
```

## Thread

+ Return to [March 1993](/archive/1993/03)

+ Return to "[Eli Brandt <ebrandt@jarthur.Claremont.EDU>](/authors/eli_brandt_ebrandt_at_jarthur_claremont_edu_)"
+ Return to "[Eric Hughes <hughes@soda.berkeley.edu>](/authors/eric_hughes_hughes_at_soda_berkeley_edu_)"
+ Return to<EMAIL_ADDRESS>(Perry E. Metzger)](/authors/pmetzger_at_shearson_com_perry_e_metzger_)"
+ Return to "[Theodore Ts'o <tytso@Athena.MIT.EDU>](/authors/theodore_tso_tytso_at_athena_mit_edu_)"
+ Return to<EMAIL_ADDRESS>(E. Dean Tribble)](/authors/tribble_at_xanadu_com_e_dean_tribble_)"

+ 1993-03-01 (Mon, 1 Mar 93 14:06:12 PST) - [Re: Future of anonymity (short-term vs. long-term)](/archive/1993/03/887be117d81c786db6f6cd011fabb918e373b9eb7b0cc8b392e118043a7ebeef) -<EMAIL_ADDRESS>(Perry E. Metzger)_
  + 1993-03-02 (Mon, 1 Mar 93 16:39:05 PST) - [Re: Future of anonymity (short-term vs. long-term)](/archive/1993/03/883e696ff5ba9a84e457bca1f94367b4c806659b2382a4c1c526ce9b7dc6e529) - _Theodore Ts'o<EMAIL_ADDRESS>    + 1993-03-02 (Mon, 1 Mar 93 17:47:45 PST) - Future of anonymity (short-term vs. long-term) - _Eric Hughes<EMAIL_ADDRESS>      + 1993-03-02 (Mon, 1 Mar 93 20:54:18 PST) - [Re: Future of anonymity (short-term vs. long-term)](/archive/1993/03/5343a585c28d083a402a2921e602b09d2b8ea2dd4134b2f12a10f41407d671f1) - _Theodore Ts'o<EMAIL_ADDRESS>        + 1993-03-03 (Tue, 2 Mar 93 23:51:12 PST) - [Re: Future of anonymity (short-term vs. long-term)](/archive/1993/03/41464dc51864d19564b6231259f32d4fc5e818d1f25712a2e09e1f4581074cf7) - _Eli Brandt<EMAIL_ADDRESS>        + 1993-03-03 (Wed, 3 Mar 93 01:29:30 PST) - [Future of anonymity (short-term vs. long-term)](/archive/1993/03/83fbff656c47eea5b42c3be58322339e1bf459de18c5f1a6ae2789b38341ca7f) -<EMAIL_ADDRESS>(E. Dean Tribble)_


text:
# PBS 151 of X — Bash: Printf and More

In the previous instalment we dug deep into how Bash interacts with the three standard POSIX data streams (`/dev/stdin`, `/dev/stdout` & `/dev/stderr`), and we finished with a look at the special file `/dev/tty` which provides access to the terminal our script is running in, regardless of where the three standard streams have been redirected to. The plan for this instalment was to revisit scope, and to look at better string outputs with `printf` rather than `echo`, but thanks to some excellent feedback from listener Jill of Kent (her choice of title), we'll postpone scope until the next instalment, and start this instalment with an excellent tip related to the `[[` builtin and `/dev/tty`.

## Matching Podcast Episode

Listen along to this instalment on [episode 768 of the Chit Chat Across the Pond Podcast](https://www.podfeet.com/blog/2023/05/ccatp-768/).

<audio controls src="https://media.blubrry.com/nosillacast/traffic.libsyn.com/nosillacast/CCATP_2023_05_27.mp3?autoplay=0&loop=0&controls=1">Your browser does not support HTML 5 audio 🙁</audio>

You can also Download the MP3

Read an unedited, auto-generated transcript:  CCATP_2023_05_27

## Episode Resources

* The instalment ZIP file — [pbs151.zip](https://github.com/bartificer/programming-by-stealth/raw/master/instalmentZips/pbs151.zip)

## PBS 150 Challenge Solution

The challenge set at the end of the previous instalment was to update the menu script we've been working on for the previous few challenges to ensure all user interaction is via the terminal, even when standard in or standard out are redirected, and to allow the user control the source of the menu based on the following simple rules:

1. By default, the menu will be read from `menu.txt` in the same folder as the script.
2. The menu can be read from standard in with the option `-m -`.
3. The menu can be read from any file with the option `-m FILE`.

To simplify things, there was the option to remove the snark (`-s`) flag.

I chose to remove the `-s` flag to minimise the distractions in my sample solution. You'll find my full solution in the instalment ZIP as `pbs150-challengeSolution.sh`.

The first thing I want to draw your attention to is the use of redirects to and from `/dev/tty` to force the user interaction via the terminal. I added `>/dev/tty` after all echo statements related the accepting the order, and I added both a `>/dev/tty` and `</dev/tty` to the end of the `select` statement so it both writes to the terminal and reads from the keyboard no matter where standard in and out have been redirected:

```bash
# present the menu, with a done option
if [[ -z $limit ]]
then
    echo 'Choose your breakfast (as many items as you like)' >/dev/tty
else
    echo "Choose up to $limit breakfast items" >/dev/tty
fi
select item in done "${menu[@]}"
do
    # skip invalid selections ($item is empty)
    [[ -z $item ]] && continue

    # exit if done
    [[ $item == done ]] && break

    # store and print the item
    order+=("$item")
    echo "Added $item to your order" >/dev/tty

    # if we're limiting, check the limit
    if [[ -n $limit ]]
    then
        [[ ${#order[@]} -ge $limit ]] && break
    fi
done </dev/tty >/dev/tty
```

Finally, lets look at how I chose to initialise the menu.

To avoid code duplication, or, to avoid adding a lot of inline logic to the end of the `while` loop for processing the menu I chose to break the task into two parts:

1. slurp the raw menu into a string
2. process the raw menu line-by-line to built up the array of options

Looking at the first part, I chose to initialise the menu source to the default, then use a simple `if` statement to read from standard in or a file:

```bash
# initialise the options to their default values
# …
menuSource=$(dirname "$BASH_SOURCE")/menu.txt # menu.txt in script dir

# …

# slurp the menu into a string
menuString=''
if [[ $menuSource == '-' ]]
then
    menuString=$(cat)
else
    menuString=$(cat "$menuSource")
fi
```

With the menu loaded into a string named `$menuString`, I then loop through it line-by-line using a here string (`<<<`). As a reminder, this operator redirects the contents of a variable to standard in:

```bash
# process the menu
declare -a menu
while read -r menuLine
do
    # skip comment lines
    echo "$menuLine" | egrep -q '^[ ]*#' && continue

    # skip empty lines
    echo "$menuLine" | egrep -q '^[ ]*$' && continue

    # store the menu item
    menu+=("$menuLine")
done <<<"$menuString"
```

The logic inside the loop is un-changed from the previous version of the script.

The only other change was updating `getopts` to remove support for the `-s` flag, and add the `-m` optional argument.

We can verify that a menu can be loaded from an alternative file by using the `-m`  optional argument to load the menu from the file `menu-montyPython.txt` which offers just one choice, `spam` 😉:

```bash
./pbs150-challengeSolution.sh -m menu-montyPython.txt
```

We can verify the correct handling of redirecting both standard out and standard in with the command:

```bash
echo -e "bacon\neggs\ntoast" | ./pbs150-challengeSolution.sh -m - > order.txt
```

This command redirects standard out and standard in, but the menu is still presented in the terminal, and the keyboard can still be used to enter the options. Finally, the chosen items are correctly written to `order.txt`, which you can verify with:

```bash
cat ./order.txt
```

## Followup — Detecting Terminals

As mentioned in the introduction, listener Jill of Kent sent me a wonderful message after we posted the previous instalment with a related tip, and I thought it was just too good not to include in the series.

One of the things you may have noticed over the years on the terminal is that some commands seem to somehow know when they're outputting to a terminal rather than a redirected stream, and behave differently in each case.

Many of the Git commands do this, for example, `git log` will show you the full Git commit history of your currently checked out branch, and when your output is going to a terminal, it will run the output through `less` for you so you get it one page at a time, waiting in your input to move to the next page. But, if you redirect the identical command to a file, the information gets written directly to the file without being run through `less` which would block the operation, e.g. `git log > log.txt`.

How does that work? Presumably, the POSIX spec provides an API for figuring out if a given stream is connected to a terminal, so the real question is, how does Bash expose this functionality?

Back in [instalment 145](./pbs145) we learned to use the `[[` builtin command to perform boolean tests, and we learned about some of the operators `[[` supports. As I was at pains to point out, we only looked at a subset of the available operators, and one of the ones we ignored, `-t`, lets us test if a given file handle is a terminal or not. The syntax is a little abstract looking, because the operator works on the numeric ID for the file descriptor (`0` for standard input, `1` for standard output, and `2` for standard error). If a stream is a terminal, then `-t` will give a successful exit code, and if not, it will give an error code, so we can test if standard out is a terminal with `[[ -t 1 ]]`.

As a more practical example, let's look at `pbs151a-terminalTester.sh` from the instalment ZIP:

```bash
#!/usr/bin/env bash

# check standard input
if [[ -t 0 ]]
then
    echo 'STDIN is a terminal' >/dev/tty
else
    echo 'STDIN is NOT a terminal' >/dev/tty
fi

# check standard output
if [[ -t 1 ]]
then
    echo 'STDOUT is a terminal' >/dev/tty
else
    echo 'STDOUT is NOT a terminal' >/dev/tty
fi

# check standard error
if [[ -t 2 ]]
then
    echo 'STDERR is a terminal' >/dev/tty
else
    echo 'STDERR is NOT a terminal' >/dev/tty
fi
```

We can see this little script in action with the following commands:

```bash
./pbs151a-terminalTester.sh
echo 'pancakes' | ./pbs151a-terminalTester.sh
./pbs151a-terminalTester.sh >/dev/null
./pbs151a-terminalTester.sh 2>/dev/null
echo 'pancakes' | ./pbs151a-terminalTester.sh >/dev/null 2>&1
```

# Better String Formatting with `printf`

One of the most important programming languages in the history of computer programming is C. We've not written a single line of C code in this series, but C's impact on what we have learned has been huge — semi colons to end lines, curly braces to wrap code blocks, `if` and `while` statements with conditions in round brackets, function calls with parameter lists wrapped in round brackets — all that is from C. Apart from its general style, C's other biggest export is probably the `printf` function, which stands for *'print format'* because it prints strings to standard out that can contain formatted versions of the values of variables. Do do this, `printf` uses a rich placeholder syntax, and it's that syntax even more than the function itself that has spread far and wide.

The syntax for the actual `printf` command is trivial. The first argument is a format string, and subsequent arguments provide the values for the placeholders in the format string. If you have a format string with two placeholders, you'd call `printf` with three arguments — the format string, the value for the first placeholder, and the value for the second.

```text
printf FORMAT_STRING [VAL …]
```

For example:

```bash
printf 'I like to have %s %d times a week!\n' pancakes 5
```

Bash's `printf` command only supports one optional argument, `-v`, which allows you to specify a variable name to save the output to rather than printing it, so the following works:

```bash
printf -v dessert 'some %s please!\n' waffles
echo $dessert
```

It's the format string syntax where both the power and the complexity lie!

## `printf` Format Strings

The format string syntax looks simple at first glance, but it's deceptively complex and powerful. The good news is that all the complexity is optional. The syntax for doing simple things is really simple so you can get started with `printf` very quickly, but you'll probably never learn everything it can do, even in a lifetime of programming!

As is our custom, we'll be cherry picking the most interesting aspects of the spec, so if you want all the details, you'll need to spend some time with the documentation.

Let's start with the basics — a format string consists of a series of zero or more of the following:

1. escape sequences
2. format specifications
3. plain text

Escape sequences are pre-fixed with a backslash (`\`), format specifications start with a percentage sign (`%`), and everything that's not part of an escape sequence or format  spec is plain text, and is included in the output unchanged.

### Escape Sequences

There are just three escape sequences we need to know about:

| Sequence | Description |
| :-- --: | :--- |
| `\n` | Add a new line character |
| `\t` | Add a tab character |
| `\\`| Add an actual backslash character |

For the most part, Bash is smart enough to make these works as expected in both interpolated (`""`) and uninterpolated strings (`''`):

```bash
printf 'ho\tho\nho\thum\n'
printf "ho\tho\nho\thum\n"
```

However, the one exception to that is `\\`. When possible, I would advise avoiding using this escape sequence in an interpolated string, because you need to double-escape it!

```bash
printf 'ho\\hum\n'
printf "ho\\\\hum\n"
```

Finally note that  **`printf` does not end lines, so you need to add your own `\n` if you want to move to a new line**!

### Format Specifications

Format specifications always start with a `%` symbol and always contain a type, but they can optionally contain more details. For our purposes the following is the full syntax for format specifiers (there are actually even more options):

```text
%[FLAGS][MIN_WIDTH][.PRECISION]TYPE
```

Since types are the only required part of the format spec, let's start there. From our POV there are just three types and a special case:

| Type | Description |
| :--: | :-- |
| `d` | Format as as whole number (think *digits*). |
| `f` | Format as a floating-point number. |
| `s`| Format as a string. |
| `%` | Output an actual percentage symbol (i.e. `%%` is the format string for a percentage symbol) |

This gives us the following simple usage:

```bash
printf '%d %s cost $%f\n' 5 pancakes 5.55
# outputs: 5 pancakes costs $5.550000
```

Straight away we see our first problem — we need to control the **precision**!

We can do that by pre-fixing the type with a period followed by the number of decimal places we'd like, so for a currency that would be `%.2f`:

```bash
printf '%d %s cost $%.2f\n' 5 pancakes 5.55
# outputs: 5 pancakes cost $5.55
```

Much better!

Now let's look at some bigger numbers:

```bash
earthDiameter=40075.017
printf 'The earth is %fkm around the equator.\n' $earthDiameter
# outputs: The earth is 40075.017000km around the equator.
```

OK, let's lose the decimals completely:

```bash
printf 'The earth is %.0fkm around the equator.\n' $earthDiameter
# outputs: The earth is 40075km around the equator.
```

Better, but we're missing the thousand separators! How can we add those in? This is where we meet our first optional flag, the `'` (single quote) character indicates that thousand separators should be used (think of it as a comma in the air 😉). As well as being weird, this is awkward, because a single quote ends an uninterpolated string. The simplest fix is to use interpolated strings when your format needs to specify the thousand separator:

```bash
printf "The earth is %'.0fkm around the equator.\n" $earthDiameter
# outputs: The earth is 40,075km around the equator.
```

Note that the flags are the first thing after the `%` symbol.

### Aligning Data with `printf`

If you're just going to insert values into lines of text what we've covered so far is all you're likely to need, but if you need to align your data, say in some kind of table-like output, you need to learn a few more things.

Firstly, you'll need to specify the required **minimum width** for each format spec by adding a number between the flags and the precision (both of which are optional). **By default the rendered text is right-aligned and padded with spaces** as needed.

For example:

```bash
rowFormat='%20s %8.2f\n'
printf "$rowFormat" Waffles 4.5; printf "$rowFormat" Pancakes 5.4
```

This prints:

```text
             Waffles     4.50
            Pancakes     5.40
```

There's a few things to note here. Firstly, to make the output look right when typed into a terminal, I placed multiple `printf` commands on one line by using the `;` separator. If I was running this code in a script I could simply have those two statements as separate lines.

Secondly, note that I used a variable to store the format - this is particularly important of you'll be outputting a lot of lines of information! Finally, also note that **you need to quote variables passed as format strings**.

To left-align our description we use the `-` flag:

```bash
rowFormat='%-20s $%8.2f\n'
printf "$rowFormat" Waffles 4.5; printf "$rowFormat" Pancakes 5.4
```

This prints:

```text
Waffles              $    4.50
Pancakes             $    5.40
```

In some situations you might want to pad numeric fields with `0` rather than spaces, you can do that with the `0` flag:

```bash
rowFormat='%03d\n'
printf "$rowFormat" 1; printf "$rowFormat" 20; printf "$rowFormat" 300
```

Prints:

```text
001
020
300
```

If you have a mix of positive and negative numbers you might want to either add `+` symbols to the positive ones which you can do with the `+` flag:

```bash
rowFormat='%+d\n'
printf "$rowFormat" -1; printf "$rowFormat" 0; printf "$rowFormat" 1
```

Which prints:

```text
-1
+0
+1
```

A nicer solution might be to add a space before positive numbers, which you can do by using a space as a flag, no, seriously, **a space is a flag**!

```bash
rowFormat='% d\n'
printf "$rowFormat" -1; printf "$rowFormat" 0; printf "$rowFormat" 1
```

Prints:

```text
-1
 0
 1
```

You can of course use multiple flags in the same format spec:

```bash
rowFormat="% '.2f\n"
printf "$rowFormat" -1234.5678; printf "$rowFormat" 9876.5432
```

Prints:

```text
-1,234.57
 9,876.54
```

One final tip — **you can use the precision operator to truncate strings**. Remember that the width property specifies a minimum width, so it won't stop long strings breaking a table. If you want to set a maximum width you can combine the precision operator (`.`) with the string type, e.g.:

```bash
rowFormat='%-3.3s %2d\n'
printf "$rowFormat" monday 5; printf "$rowFormat" tuesday 11
```

Prints:

```text
mon  5
tue 11
```

## Final Thoughts

Firstly, Jill's message serves as a good reminder that this series is very much community driven, if you'd like to join in, check out Allison's Slack at [podfeet.com/slack](https://www.podfeet.com/slack)!

Secondly, I hope you can appreciate just how powerful `printf` is for formatting strings. What's extra useful is that the `printf` format syntax is very portable. Like Perl's Regular Expression syntax, `printf` formatting is used very widely, so having at least the basics mastered is very empowering!

### An Optional Challenge

Write a script to render multiplication tables in a nicely formatted table. Your script should:

1. Require one argument — the number to render the table for
2. Default to multiplying the number given by 1 to 10 inclusive
3. Accept the following two optional arguments:
	* `-m` to specify a minimum value, replacing the default of 1
	* `-M` to specify a maximum value, replacing the default of 10

For bonus credit, pipe the output through `less` if and only if standard out is a terminal.

repository: bartificer/programming-by-stealth

repo_name: bartificer/programming-by-stealth


text:
# merkletree
A Python implementation of the Merkle/Hash Tree Algorithm

This implementation uses distinct node objects which are designed to be easily traversible.  Each node has references to it's children, parent, and sibling nodes.  The tree has a convenience method for accessing the path from any node to the Merkle root. By using the add_adjust() method, new leaves can be added to
an already built tree, without rebuilding.

Installation:

    pip install merkle

The MIT License (MIT)

Copyright (c) 2015 Jamie Steiner

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


text:
# Distribution

## Properties
Name | Type | Description | Notes
------------ | ------------- | ------------- | -------------
**cpeUri** | **String** | The cpe_uri in [cpe format](https://cpe.mitre.org/specification/) denoting the package manager version distributing a package. |  [optional]
**architecture** | [**ArchitectureEnum**](#ArchitectureEnum) | The CPU architecture for which packages in this distribution channel were built |  [optional]
**latestVersion** | [**Version**](Version.md) | The latest available version of this package in this distribution channel. |  [optional]
**maintainer** | **String** | A freeform string denoting the maintainer of this package. |  [optional]
**url** | **String** | The distribution channel-specific homepage for this package. |  [optional]
**description** | **String** | The distribution channel-specific description of this package. |  [optional]

## Enum: ArchitectureEnum
Name | Value
---- | -----
UNKNOWN | "UNKNOWN"
X86 | "X86"
X64 | "X64"

repository: pombredanne/client-java

repo_name: pombredanne/client-java


text:
# How to use code in this folder

This folder generates materialized views for a variety of severity of illness scores (detailed below).
The queries make use of materialized views which aggregate data from the first day of a patient's ICU stay. To run the code, it is necessary to:

1. Run the scripts which generate the constituent materialized views
    * echodata - generated by /concepts/echo-data.sql
    * ventdurations - generated by /concepts/durations/ventilation-durations.sql - (needed for subsequent view)
    * vitalsfirstday - generated by /concepts/firstday/vitals-first-day.sql
    * uofirstday - generated by /concepts/firstday/urine-output-first-day.sql
    * ventfirstday - generated by /concepts/firstday/ventilated-first-day.sql
    * gcsfirstday - generated by /concepts/firstday/gcs-first-day.sql
    * labsfirstday - generated by /concepts/firstday/labs-first-day.sql
    * bloodgasfirstday - generated by /concepts/firstday/blood-gas-first-day.sql - (needed for subsequent view)
    * bloodgasfirstdayarterial - generated by /concepts/firstday/blood-gas-first-day-arterial.sql
2. Run the script for the severity of illness score you are interested in
    * OASIS - oasis.sql
    * SAPS - saps.sql
    * SOFA - sofa.sql

The scripts were written and tested in PostgreSQL 9.4.4. The scripts are written to follow the SQL standard as close as possible, and there should be minimal changes necessary for them to operate under different languages. Key changes likely involve:
  * Changing "with" clause to subqueries if using MySQL
  * Modifying the regular expression in the *echo* view in sofa.sql as appropriate

If you modify a script to operate in another RDBMS, we would welcome a pull request!

# Severity of illness scores

This section lists the severity scores available here.

## OASIS

The Oxford Acute Severity of Illness Score (OASIS) is a parsimonious severity score developed using a hybrid genetic algorithm and particle swarm optimization approach which allowed direct optimization of a severity score in a clinically relevant form with simultaneous multivariate feature selection (Johnson, 2013). The new score discriminated better than the Acute Physiology Score III (APS III) when evaluated in a univariate fashion and equivalently when evaluated as a covariate in a larger risk adjustment model similar to APACHE IV. OASIS was designed to have an extremely low burden for data collection and quality control, requiring only 10 features, and not requiring laboratory measurements, diagnosis or comorbidity information.

## SAPS

The Simplified Acute Physiology Score (SAPS) was intended as a simplification of the Acute Physiology Score (APS), reducing the number of physiological parameters required from the original 34 to 13 plus age (LeGall, 1984). The variables chosen were present for 90\% of patients in the initial survey used to develop the APS (Knaus, 1981). Though the original publication only provided ROC curves and not the AUROC, trapezoidal integration showed that SAPS had an AUROC of 0.7697 and APS had an AUROC of 0.7661.

## SOFA

The Sepsis-related Organ Failure Assessment score was first developed by a consensus meeting of the ESICM in October 1994, though it eventually became known as the Sequential Organ Failure Assessment (SOFA) score as it was applied outside of septic populations (Vincent, 1996).
The purpose of the score was to provide the clinical community with an objective measure of the severity of organ dysfunction in a patient.
It is stressed that the score is not meant as a direct predictor of mortality but rather a measure of morbidity, or the level of the diseased state, in a patient.
The score is evaluated for 6 organ systems: pulmonary, renal, hepatic, cardiovascular, haematologic and neurologic.
Note that the organ-specific morbidity scores are highlighted as a useful compliment to the overall score.

# Future work

The following severity scores have yet to be implemented, but are desirable, and contributions are welcome.

## SAPS II

The Simplified Acute Physiology Score II (SAPS II) published in 1993 (Le Gall, 1993), aimed to rectify two issues with SAPS. First, the variable selection process in SAPS was done by clinical judgement, whereas SAPS II utilised univariate feature selection to filter out features uncorrelated with hospital mortality. Second, there was no model for calculating a probability of mortality from SAPS. SAPS II was published with calibration coefficients which allowed conversion of the integer score into a risk of mortality which ranged between zero and one.

### SAPS III

The Simplified Acute Physiology Score III (SAPS III) is the latest model published and attempts to account for poor calibration of SAPS II found in later studies (Metnitz, 2005).
While SAPS II was developed on ICUs in western Europe, SAPS III included ICUs worldwide.
The parameters of the model were identified using a stepwise logistic regression, followed by statistical hypothesis testing to ensure the parameters were significantly related to patient hospital mortality (Moreno, 2005).
The final prediction used 61 binary indicator features which dichotomised various ranges of 20 distinct variables (e.g. the highest heart rate variable was split into three binary features, one for heart rates $<$ 120, one for 120 <= heart rates $<$ 160 and a final feature for heart rates >= 160).
A hierarchical model was used, specifying patient characteristics as fixed effects and different ICUs as a random effect. A logistic regression equation was then used to calculate the probability of mortality.

## APACHE IV

APACHE IV is the most recent iteration in the Acute Physiology, Age, and Chronic Health Evaluation system. The model was published in 2006 and coefficients are freely available online.
Overall, APACHE IV tends to have the best discriminative performance among the most recent generation of general purpose severity of illness models.

APACHE IV's components include: the Acute Physiology Score (APS) III, comorbidities, diagnosis, admission source, surgical status, mechanical ventilation flags and existence of thrombolytic therapy.
Unfortunately, the diagnostic component of APACHE IV is very difficult to code.

An implementation of APACHE IV would use structured data for the APS III, and either:

* search of the discharge notes to provide comorbidity information
* Non-primary ICD-9 codes to define comorbidities

Admission diagnosis would also need to be defined, likely using a search of the discharge notes. While this method would be reasonably accurate, it is unlikely it would be as accurate as manual classification of patients (a burdensome task).

### References

Vincent JL, de Mendonça A, Cantraine F, Moreno R, Takala J, Suter PM, Sprung CL, Colardyn F, Blecher S. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Working group on "sepsis-related problems" of the European Society of Intensive Care Medicine. Crit Care Med 1998 Nov;26(11):1793-800. [PMID 9824069](https://www.ncbi.nlm.nih.gov/pubmed/9824069?dopt=Abstract).

Moreno R, Vincent JL, Matos R, Mendonça A, Cantraine F, Thijs L, Takala J, Sprung C, Antonelli M, Bruining H, Willatts S. The use of maximum SOFA score to quantify organ dysfunction/failure in intensive care. Results of a prospective, multicentre study. Working Group on Sepsis related Problems of the ESICM. Intensive Care Med 1999 Jul;25(7):686-96. [PMID 10470572](https://www.ncbi.nlm.nih.gov/pubmed/10470572?dopt=Abstract).

repository: MIT-LCP/mimic-code

gha_language: Jupyter Notebook

repo_name: MIT-LCP/mimic-code


text:
# Package Sharpener

I created this commandline tool to remove duplicate files in the packages and bin child folders to the working directory. The publish command
below also demonstrates how to compile a self contained exe from a .NET Core project. Note that this project doesn't actually deduplicate successfully because the packages folder is actually arranged by type of architecture, so I would need to add some additional logic. Luckily, I am no longer cleaning up after people using manual references instead of Nuget.

Build an exe into .\bin\Release\netcoreapp2.0\win10-x64\

```
dotnet publish -c Release --self-contained --runtime win10-x64
```

Add to path for this session

```
$env:Path = (Get-Item -Path ".\" -Verbose).FullName + "\bin\Release\netcoreapp2.0\win10-x64;$env:Path"
```

Or add `"cwd": "<path of a .net project>"` to launch.json

[← Back to Project List](../README.md)

repository: PaluMacil/misc

repo_name: PaluMacil/misc


text:
# Hiding Services & Runtime Discovery with Spring Cloud Gateway

[Ben Wilcock][30] – Spring Marketing, Pivotal.
[Brian McClain][31] – Technical Marketing, Pivotal.

It's rare for a company to want _every_ API to be publicly accessible. Most prefer to keep their services secret by default, only exposing APIs publicly when absolutely necessary.

[Spring Cloud Gateway][14] can help. Spring Cloud Gateway allows you to route traffic to your APIs using simple Java™ instructions (which we saw [in the last article][15]) or with YAML configuration files (which we’ll demonstrate in this one). To hide your services, you set up your network so that the only server accessible from the outside is the gateway. The gateway then becomes a gate-keeper, controlling ingress and egress from outside. It’s a very popular pattern.

Cloud-based services also have a habit of changing location and granularity without much warning. To cope better with this, you can combine a gateway with a service registry to allow the applications on your network to find each other dynamically at runtime. If you do this, your applications will be much more resilient to changes. [Spring Cloud Netflix Eureka Server][13] is one such service registry.

In this post, we'll look at Spring Cloud’s gateway and registry components and illustrate how you can use them together to make your applications more secure and more reliable.

Because this arrangement requires a particular setup, we’ve provided ready-to-run code which you can download and run. We’ll be using Docker to orchestrate our services and emulate a private network. We’ll then talk to our running services using HTTPie.

## Things You'll Need

* Java (version 8 is assumed), plus your favorite web browser and terminal applications.

* [The Source Code][3] – There’s no need to write any code this time, simply `git clone` (or download and `unzip`) [this project's source code repository][3] from GitHub.

* [Docker Desktop][1] – Docker will provide our pseudo-production environment. We'll use it to hide our services in a private network.

* [Cloud Native Buildpacks][2] – We'll use Cloud Native Buildpacks to build Docker container images for us. Buildpacks embody several DevOps best practices, including hardened open-source operating systems and free to use OpenJDK distributions.

* About 10-15 minutes, depending on the speed of your internet, etc.

## Let’s Get Started...

#### Step 1:

Download and install all the "things you'll need" from the list above. Then change to the runtime-discovery folder in the source code as follows:

```bash
cd runtime-discovery
```

#### Step 2:

Build & package the gateway, registry, and service into JARs using Maven, and then create Docker containers for each of them. We have provided a handy [pack-images][21] script to do this:

```bash
./pack-images.sh
```

#### Step 3:

Start up the Docker test environment in the background. We’re using docker-compose here as it can start multiple containers and create a private network for them to communicate on:

```bash
docker-compose up
```

#### Step 4:

Wait. Docker will start all the containers (using the configuration provided in the [docker-compose.yml][8] file). Waiting a couple of extra minutes here is advised. It gives Docker time to start everything and gives the applications a chance to communicate and settle down. If you do wait, you should see the Gateway and the Greeting Service register themselves with the Registry. There will be lots of logs, but within them will be lines like these from the registry:

```bash
registry    | 2019-06-28 12:19:01.780  INFO 1 --- [nio-8761-exec-2] c.n.e.registry.AbstractInstanceRegistry  : Registered instance REGISTRY/db1d80613789:registry:8761 with status UP (replication=false)
registry    | 2019-06-28 12:19:02.380  INFO 1 --- [nio-8761-exec-6] c.n.e.registry.AbstractInstanceRegistry  : Registered instance GATEWAY/9c0c0c9ba027:gateway:8760 with status UP (replication=true)
registry    | 2019-06-28 12:19:02.382  INFO 1 --- [nio-8761-exec-6] c.n.e.registry.AbstractInstanceRegistry  : Registered instance SERVICE/fe7e38b21cac:service:8762 with status UP (replication=true)
```

## Let's Try It...

#### First, Check that the Greeting Service is Hidden:

The Greeting Service operates on port `8762` and is hidden inside the Docker network. Let's try to call it from your favorite browser using [http://localhost:8762/greeting](http://localhost:8762/greeting). You should be told that "the site can't be reached" by your browser. This is because the Greeting Service is hidden inside the Docker network (as if it were behind a company firewall). It shouldn't be possible for us to talk to the greeting service directly. Instead, you’ll see an error page similar to the one below.

![Screenshot from the browser window showing that the service is unreachable][unreachable]

#### Next, Access the Greeting Service via the Gateway:

Now, Navigate your browser to [http://localhost:8080/service/greeting][11]. You should now get a valid response with content similar to the "Hello, World" JSON shown below:

```json
{ "id": 1, "content": "Hello, World!"}
```

When you issued this new HTTP request from your browser, it was sent to, and handled by, the Gateway. The Gateway service _is_ publicly accessible (it’s mapped to port `8080`). Your request was forwarded by the Gateway to the Greeting Service on your behalf, and the response was then routed back to you by the Gateway.

#### Now, View the Registry of Services:

The microservices on the Docker network are each registering themselves with the Registry server (this may take a couple of minutes, so be patient). The Registry server acts an address book for the services. If the services move, or if new instances are created, they will add themselves to the registry.

To view the current list of registered services, point your browser at [http://localhost:8080/registry][10]. You should see a screen similar to the one below.

![Screenshot from the Registry console, listing several services][registry]

#### Finally, Shutting Down:

When you're done, use `ctrl-C` in your terminal to shut down the Docker services. If for any reason this fails to work, you can also use `docker-compose down` from the `runtime-discovery` folder. Using either technique, you should see output similar to this:

```bash
Stopping gateway  ... done
Stopping service  ... done
Stopping registry ... done
```

Further clean-up can be achieved using `docker-compose rm -f`.

## How It Works

In this demo, we have 3 servers. These servers are all running inside a "hidden" network which is provided by [Docker Compose][20]. Only the [Gateway server][5] is exposed to the outside world, so all HTTP traffic must go via this Gateway.

Here's a description of the 3 servers and what each does...

1. [The Gateway][5] – acts as our gatekeeper for all HTTP traffic. All inbound and outbound traffic flows through this portal – it acts as the bridge between the outside world (your browser) and the services on the internal Docker network. The Gateway has a configuration that specifies routes which can be used to talk to the services inside the network. These routes use the 'logical' names of the target services. These logical names are turned into real addresses by the Registry server.

2. [The Registry][6] – acts as a phone book of all the services on the hidden network. It allows the Gateway to find the Greeting Service using only its logical name.

3. [The Greeting Service][7] – is a simple REST service based on the popular [Spring.io](spring.io) guide "[Building a RESTful Web Service][4]".

As you can see in the [`docker-compose.yml`][8], Docker is configured to only allow external calls to reach the Gateway – on port `80`. The other servers (the Registry, and the Greeting Service), cannot be reached directly from outside.

The Gateway's configured URL passthrough paths can be seen in the Gateway's [application.yml file][9]. This configuration is using the "logical" names of these servers and the `lb:` (load balancer) protocol as shown in the extract below:

```yaml
spring:
  application:
    name: gateway
  cloud:
    gateway:
      routes:
      - id: service
        uri: lb://service
        predicates:
        - Path=/service/**
        filters:
        - StripPrefix=1
...
```

By using these 'logical' server names, the Gateway can use the Registry to discover the true location of these services at runtime.

## Wrapping Up

With the entire Spring toolkit at your disposal, it quickly becomes apparent how flexible and powerful Spring Cloud Gateway can be. If you dig into [the source code][3], you’ll notice that with just a few lines of Java and a few dependencies, we can easily integrate Spring Boot microservices with Eureka, and control access to our service’s APIs.

Before you finish, why not sign up for [SpringOne Platform 2019][18] – the premier conference for building scalable applications that delight users. Join thousands of like-minded Spring developers to learn, share, and have fun in Austin, TX from October 7th to 10th.  Use the discount code **S1P_Save200** when registering to save money on your ticket. If you need help convincing your manager use [this page][19]. See you there.

[1]: https://www.docker.com/products/docker-desktop
[2]: https://buildpacks.io/docs/app-journey/
[3]: https://github.com/benwilcock/spring-cloud-gateway-demo.git
[4]: https://spring.io/guides/gs/rest-service/
[5]: https://github.com/benwilcock/spring-cloud-gateway-demo/tree/master/runtime-discovery/gateway
[6]: https://github.com/benwilcock/spring-cloud-gateway-demo/tree/master/runtime-discovery/registry
[7]: https://github.com/benwilcock/spring-cloud-gateway-demo/tree/master/runtime-discovery/service
[8]: https://github.com/benwilcock/spring-cloud-gateway-demo/blob/master/runtime-discovery/docker-compose.yml
[9]: https://github.com/benwilcock/spring-cloud-gateway-demo/blob/master/runtime-discovery/gateway/src/main/resources/application.yml
[10]: http://localhost:8080/registry
[11]: http://localhost:8080/service/greeting

[13]: https://spring.io/guides/gs/service-registration-and-discovery/
[14]: https://spring.io/guides/gs/gateway/
[15]: https://content.pivotal.io/practitioners/getting-started-with-spring-cloud-gateway-3
[16]: https://httpie.org/
[17]: https://github.com/OlgaMaciaszek/spring-cloud-netflix-demo
[18]: https://springoneplatform.io/
[19]: https://springoneplatform.io/2019/convince-your-manager
[20]: https://docs.docker.com/compose/
[21]: https://github.com/benwilcock/spring-cloud-gateway-demo/blob/master/runtime-discovery/pack-images.yml

[unreachable]: https://static.spring.io/blog/bwilcock/20190701/unreachable.png "Screenshot from the browser window showing that the service is unreachable"
[registry]: https://static.spring.io/blog/bwilcock/20190701/registry.png "Screenshot showing the Eureka registry console in a browser"

[30]: https://twitter.com/benbravo73
[31]: https://twitter.com/BrianMMcClain

repository: Gaven-Yeh/spring-cloud-gateway-demo

repo_name: Gaven-Yeh/spring-cloud-gateway-demo


text:
# SndVolHWMixer

A real mixer board-styled volume control for Windows.

What makes this application special is the ability to retreive active streams from the audio endpoint device and send them to the hardware.
The end goal is to make it as practical as the Windows SndVol application.

The Windows application is built in VC++ with VS2017.

The Arduino end is built in a Arduino Mega2560
Use either the Arduino IDE or Platform.IO.
The Arduino program requires the Adafruit GFX library and the Adafruit SSD1306 library.

If using anything else than the Arduino IDE, get the libraries by using git submodule:

git submodule init

git submodule update --remote

![Breadboard build](cover.jpg)

repository: rondadon/SndVolHWMixer

repo_name: rondadon/SndVolHWMixer


text:
#Alexa Skill - Guess the animal sound.

Alexa plays animal sounds and you have to guess what animal made the sound.
A simple game for kids.

##Notes

This is how I generated the list of animals for the animal-sounds.js file.
There is a list of built in animal sounds at https://developer.amazon.com/docs/custom-skills/animal-sounds.html.
I took the source and extracted the animals and the sound location from the source.

Copied the source into https://regexr.com/

Used the following regex expressions to extract the names and the sound locations.

Regex: <p>([a-z]*?) .*<\/p>/g
Tools: {"name":"$1",\n to just show matching group.
Result: Multiple lines with {"name":"bear",

Extract sounds
Regex: <code .+>(.*)<\/code>
Tools: "sound": `$1`},\n to just show matching group. Can specify text to any output you want.
Result: Multiple lines with "sound": `<audio src='soundbank://soundlibrary/animals/amzn_sfx_bear_groan_roar_01'/>`},

Saved the results to two files.

Take the two files and use paste from the command line to merge them together.
paste animal.txt sound.txt > combined.txt

Copy result into the json object in animal-sounds.js.

To get the list of unique animal names I did the following:
Use the first regex to get just the animal names from the source code file. Save the result to a file.
Then used awk to remove all the diblicates.
https://stackoverflow.com/questions/1444406/how-can-i-delete-duplicate-lines-in-a-file-in-unix
awk '!seen[$0]++' animals.txt > all_animals.txt

repository: anthonyoconnor/guess-the-animal-sound

repo_name: anthonyoconnor/guess-the-animal-sound


text:
# TFNE

TFNE is a Tensorflow based Neuroevolution package. It provides a modular grammar based Neuroevolution algorithm that is very simple to modify and enhance, along with multiple Fitness Landscapes analysis methods.

Full file documentation comming soon.

TF includes the following features:

  * Modular JSON grammar for layers and parameters
  * Genome encoding structure with phenotype decoding and multiple other useful operation
  * Evolution strategies (Currently only GA approach, Grammatical evolution to be added)
  * Metropolis-Hastings sampling
  * Selective walks over the landscape
  * 3 independent mutation operators (Topological, Layer parameters and Learning parameters)
  * Generalization analysis
  * Multiple fitness landscape analysis methods:
    * Autocorrelation
    * Overfitting measure
    * Density Clouds
    * Entropic Measure of Ruggedness
    * Fitness Clouds
    * Negative Slope Coefficient
  * Saving module with function to save network topologies as well as csvs containing data from evolution/samples/walks

## Requirements

```
Python 3
Tensorflow 2.x
numpy
scipy
matplotlib
seaborn
pandas
json
```
### Optional requirements
These requirements optional.
```
tensorflow_datasets
```
Makes for testing and running the algorithm easy since there is no need to build a data pipeline. Used in provided examples.

```
Cuda 10.1
CUDNN
```
Both only needed if the user intend to run experiments on GPU. The included examples include the GPU device selection and call, but they can be simply removed if the user wants to run on CPU.

## Example & Notebooks

The following example gives a quick overview of how to run experiments using TFNE,in this case, doing neuroevolution on MNIST.  Full example on run.py, and another example using the overfitting artificial dataset similar to the one described in the paper [citation or link] on runoverfit.py.

```python

"""
##################################################################
Imports and environment settings such as loggers and runtime
##################################################################
"""
from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf
import tensorflow_datasets as tfds
import time
from selective_walk import SelectiveWalk
from evolution import Evolution
from sample import Sample
import logging
import numpy as np

logger = tf.get_logger()
logger.setLevel(logging.ERROR)

gpu_devices = tf.config.experimental.list_physical_devices('GPU')
for device in gpu_devices:
    tf.config.experimental.set_memory_growth(device, True)

start_time = time.time()

"""
##################################################################
Import the dataset and pre process the data
##################################################################
"""
# Importing MNIST from tf datasets
train_dataset, metadata = tfds.load('mnist:3.*.*', split='train', as_supervised=True, with_info=True)
test_dataset = tfds.load('mnist:3.*.*', split='test', as_supervised=True)

# Getting the number of examples from each partition
num_train_examples = int(metadata.splits['train'].num_examples)
num_test_examples = int(metadata.splits['test'].num_examples)

# Getting the number of classes
num_classes = metadata.features['label'].num_classes

# Normalizing the pizel values to be in the [0,1] range
def normalize(images, labels):
    print(labels)
    images = tf.cast(images, tf.float32)
    images /= 255
    return images, labels

train_dataset =  train_dataset.map(normalize)
test_dataset = test_dataset.map(normalize)

# Defining a data pipeline to be fed into the model
BATCH_SIZE = 64
NUM_EPOCHS = 8
train_dataset = train_dataset.cache().shuffle(num_train_examples).batch(BATCH_SIZE).repeat(NUM_EPOCHS)
test_dataset = test_dataset.cache().shuffle(num_test_examples).batch(BATCH_SIZE).repeat(1)

"""
##################################################################
Create a data dictionary that will be passed to the algorithms
##################################################################
"""

data_information = {
    'train_data': trd,
    'test_data': td,
    'nt_examples': num_train_examples,
    'nT_examples': num_test_examples,
    'nclasses': num_classes,
    'shape': metadata.features['image'].shape,
    'epochs': 6,
    'batch': 64
}

"""
##################################################################
Call and run the desired algorithm
##################################################################
"""
# Creating a neuroevolution object with the following parameters:
# population of 7 individuals with a tournament size 2 and 20 generations
evo = Evolution(7,2,20,data_information)
# Calling the GPU
if tf.test.is_gpu_available():
    with tf.device("GPU:0"):
        # Performing 10 neuroevolution runs
        evo.evolve(10)

print("--- %s seconds ---" % (time.time() - start_time))
```
In the following example we show how simple it is to add new layers to the grammar, byt adding a batch normalization layer.

```xml
{
    "Conv": {
        "filters": [32,64,128,256],
        "kernel_size": [2,3,4,5],
        "stride": [1,2,3],
        "activation": ["relu", "elu", "sigmoid"],
        "use_bias": [true, false]
    },
    "Pool": {
        "type": ["Max", "Avg"],
        "pool_size": [2,3,4,5],
        "stride": [1,2,3]
    },
    "Dense": {
        "units": [8,16,32,64,128,256,512],
        "activation": ["relu", "elu", "sigmoid"],
        "use_bias": [true, false]
    },
    "Drop": {
        "active": [true, false]
    },
    "BatchNorm": {
        "momentum": [0.999, 0.99, 0.9],
        "epsilon": [0.01, 0.001, 0.0001]
    },
    "Optimizer": {
        "lr": [0.01, 0.001, 0.0001, 0.00001],
        "decay": [0.01, 0.001, 0.0001, 0.00001],
        "momentum": [0.99, 0.9, 0.5, 0.1],
        "nesterov": [true, false]
    }
}
```

## How to run
Simply clone the repo and run the given examples or make new scripts in the same format.
`run.py` is the same as the example provided in this readme.
`overfit.py` is a script that replicates the generation of the SM (overfit) dataset described in the paper.

## How to cite TFNE
Papers that include results generated using TFNE should cite the following paper:

```xml
@ARTICLE{9113453,
  author={N. M. {Rodrigues} and S. {Silva} and L. {Vanneschi}},
  journal={IEEE Access},
  title={A Study of Generalization and Fitness Landscapes for Neuroevolution},
  year={2020},
  volume={8},
  number={},
  pages={108216-108234},}
```

repository: YoussefBushra/TFNE

repo_name: YoussefBushra/TFNE