text:
---
author: dealingwith
date: '2006-01-17 16:34:00'
layout: post
slug: confidence-in-the-idea
status: publish
title: Confidence in the Idea
wordpress_id: '886'
categories:
 - ideas
 - thoughts
---

_It doesn’t really matter what you’re developing, as long as you have the
right kind of confidence. Self-confidence is wonderful, but it’s far less
important than confidence in the idea. Honestly, I’ve never been the most
self-assured person. But I’ve always been unwavering in my confidence that
every concept I’ve helped launch—standalone IA consulting services, the Polar
Bear book, enterprise IA seminars, the IAI, UXnet, and now Rosenfeld
Media—made absolute sense, despite how I might have felt about myself at the
time.

I bring up this obvious point because it’s not always so obvious.  You’ll find
that smart people often won’t understand your concept, and others will find
ways to pick it apart just because. Will you be prepared to deal with
criticism from both the people you respect and, perhaps, from those who you
don’t? Yes, if you have the right kind of confidence.

How will you know if you have it? See how you feel about your Big Idea on your
absolutely worst day in recent memory, the day you stepped on the cat’s tail,
dozed off during your daughter’s recital, forgot your mother’s birthday, and
lost out on your dream job.

repository: dealingwith/poole

gha_language: CSS

repo_name: dealingwith/poole


text:
# TDS
A Distributed System Simulator for a single node. It has been developed as a platform for analyzing and benchmarking a novel approach on Fault-Tolerant Termination Detection based on an optimized  version of Safra's Termination Detection algorithm.

More: TBA

Ref:

[1] https://www.cs.utexas.edu/users/EWD/ewd09xx/EWD998.PDF

[2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>.4105&rep=rep1&type=pdf

# Usage
 - Build
```sh
$ ant jar
```
 - Run

```sh
$ java -jar TDS-0.1.jar [OPTION]...
```

# Options

   Option   |                  Description                   | Params                                                  | Default |
------------|------------------------------------------------|---------------------------------------------------------|---------|
 -h 	    | Print help                                     |    -                                                    |off      |
 -ver {val} | Algorithm version                              |1,2,3 or any combination. 0 for all versions |All        |0        |
 -n {val}   | Number of nodes.                               |4 - 1024                                                 |4        |
 -l {val}   | Level of activity                              |2-4                                                      |2        |
 -w {val}   | Maximum wait time in ms.                       |>0                                                       |100000   |
 -v 	    | Verbose                                        |    -                                                    |off      |
 -csv	    | Write performance metrics to csv               |    -                                                    |off      |
 -f	    | flush the csv file                             |    -                                                    |off      |
 -log	    | Log ouput in files    (N/A yet)                |    -                                                    |off      |

# Verbose Output
This is a typical output when running with option **-v** and default **-ver**

# Issues
[1] There is currently a bug introduced after the migration. When all versions run at the same time, occasionally, one will hang with no activity taking place for that version.

license: MIT

repository: KristinnArdal/TDS

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: KristinnArdal/TDS


text:
# truffled

 * [Cucumber Radish And Watercress Sandwiches With Truffled Goat Cheese](../../index/c/cucumber-radish-and-watercress-sandwiches-with-truffled-goat-cheese-107900.json)
 * [Parsley Root Soup With Truffled Chestnuts](../../index/p/parsley-root-soup-with-truffled-chestnuts-240568.json)
 * [Truffled Lobster Risotto](../../index/t/truffled-lobster-risotto-4622.json)
 * [Truffled Mac N Cheese](../../index/t/truffled-mac-n-cheese-51186230.json)
 * [Truffled Red Wine Risotto With Parmesan Broth](../../index/t/truffled-red-wine-risotto-with-parmesan-broth-235854.json)
 * [Truffled Toast With Radicchio And Egg](../../index/t/truffled-toast-with-radicchio-and-egg-236003.json)
 * [Truffled Mushroom Pizza](../../index/t/truffled-mushroom-pizza.json)
 * [Truffled Egg Salad Tea Sandwiches](../../index/t/truffled-egg-salad-tea-sandwiches.json)
 * [Truffled Fillet Of Beef Sandwiches Recipe](../../index/t/truffled-fillet-of-beef-sandwiches-recipe.json)
 * [Chef Johns Truffled Potato Gratin](../../index/c/chef-johns-truffled-potato-gratin.json)
 * [Four Cheese Truffled Macaroni And Che](../../index/f/four-cheese-truffled-macaroni-and-che.json)
 * [Truffled Cauliflower Gratin](../../index/t/truffled-cauliflower-gratin.json)
 * [Truffled Mashed Sweet Potatoes](../../index/t/truffled-mashed-sweet-potatoes.json)
 * [Beef Tenderloin With Red Wine Sauce Creamed Spinach And Truffled French Fries 230635](../../index/b/beef-tenderloin-with-red-wine-sauce-creamed-spinach-and-truffled-french-fries-230635.json)
 * [Braised Truffled Turkey 20132](../../index/b/braised-truffled-turkey-20132.json)
 * [Cucumber Radish And Watercress Sandwiches With Truffled Goat Cheese 107900](../../index/c/cucumber-radish-and-watercress-sandwiches-with-truffled-goat-cheese-107900.json)
 * [Parsley Root Soup With Truffled Chestnuts 240568](../../index/p/parsley-root-soup-with-truffled-chestnuts-240568.json)
 * [Truffled Lobster Risotto 4622](../../index/t/truffled-lobster-risotto-4622.json)
 * [Truffled Mac N Cheese 51186230](../../index/t/truffled-mac-n-cheese-51186230.json)
 * [Truffled Red Wine Risotto With Parmesan Broth 235854](../../index/t/truffled-red-wine-risotto-with-parmesan-broth-235854.json)
 * [Truffled Toast With Radicchio And Egg 236003](../../index/t/truffled-toast-with-radicchio-and-egg-236003.json)


text:
## Compiler APIs

> This API is unstable. Learn more about
> [unstable features](../runtime/stability.md).

Deno supports runtime access to the built-in TypeScript compiler. There are
three methods in the `Deno` namespace that provide this access.

### `Deno.compile()`

This works similar to `deno cache` in that it can fetch and cache the code,
compile it, but not run it. It takes up to three arguments, the `rootName`,
optionally `sources`, and optionally `options`. The `rootName` is the root
module which will be used to generate the resulting program. This is like the
module name you would pass on the command line in
`deno run --reload example.ts`. The `sources` is a hash where the key is the
fully qualified module name, and the value is the text source of the module. If
`sources` is passed, Deno will resolve all the modules from within that hash and
not attempt to resolve them outside of Deno. If `sources` are not provided, Deno
will resolve modules as if the root module had been passed on the command line.
Deno will also cache any of these resources. All resolved resources are treated
as dynamic imports and require read or net permissions depending on if they're
local or remote. The `options` argument is a set of options of type
`Deno.CompilerOptions`, which is a subset of the TypeScript compiler options
containing the ones supported by Deno.

The method resolves with a tuple. The first argument contains any diagnostics
(syntax or type errors) related to the code. The second argument is a map where
the keys are the output filenames and the values are the content.

An example of providing sources:

```ts
const [diagnostics, emitMap] = await Deno.compile("/foo.ts", {
  "/foo.ts": `import * as bar from "./bar.ts";\nconsole.log(bar);\n`,
  "/bar.ts": `export const bar = "bar";\n`,
});

assert(diagnostics == null); // ensuring no diagnostics are returned
console.log(emitMap);
```

We would expect map to contain 4 "files", named `/foo.js.map`, `/foo.js`,
`/bar.js.map`, and `/bar.js`.

When not supplying resources, you can use local or remote modules, just like you
could do on the command line. So you could do something like this:

```ts
const [diagnostics, emitMap] = await Deno.compile(
<EMAIL_ADDRESS>);
```

In this case `emitMap` will contain a `console.log()` statement.

### `Deno.bundle()`

This works a lot like `deno bundle` does on the command line. It is also like
`Deno.compile()`, except instead of returning a map of files, it returns a
single string, which is a self-contained JavaScript ES module which will include
all of the code that was provided or resolved as well as exports of all the
exports of the root module that was provided. It takes up to three arguments,
the `rootName`, optionally `sources`, and optionally `options`. The `rootName`
is the root module which will be used to generate the resulting program. This is
like module name you would pass on the command line in `deno bundle example.ts`.
The `sources` is a hash where the key is the fully qualified module name, and
the value is the text source of the module. If `sources` is passed, Deno will
resolve all the modules from within that hash and not attempt to resolve them
outside of Deno. If `sources` are not provided, Deno will resolve modules as if
the root module had been passed on the command line. All resolved resources are
treated as dynamic imports and require read or net permissions depending if
they're local or remote. Deno will also cache any of these resources. The
`options` argument is a set of options of type `Deno.CompilerOptions`, which is
a subset of the TypeScript compiler options containing the ones supported by
Deno.

An example of providing sources:

```ts
const [diagnostics, emit] = await Deno.bundle("/foo.ts", {
  "/foo.ts": `import * as bar from "./bar.ts";\nconsole.log(bar);\n`,
  "/bar.ts": `export const bar = "bar";\n`,
});

assert(diagnostics == null); // ensuring no diagnostics are returned
console.log(emit);
```

We would expect `emit` to be the text for an ES module, which would contain the
output sources for both modules.

When not supplying resources, you can use local or remote modules, just like you
could do on the command line. So you could do something like this:

```ts
const [diagnostics, emit] = await Deno.bundle(
<EMAIL_ADDRESS>);
```

In this case `emit` will be a self contained JavaScript ES module with all of
its dependencies resolved and exporting the same exports as the source module.

### `Deno.transpileOnly()`

This is based off of the TypeScript function `transpileModule()`. All this does
is "erase" any types from the modules and emit JavaScript. There is no type
checking and no resolution of dependencies. It accepts up to two arguments, the
first is a hash where the key is the module name and the value is the content.
The only purpose of the module name is when putting information into a source
map, of what the source file name was. The second argument contains optional
`options` of the type `Deno.CompilerOptions`. The function resolves with a map
where the key is the source module name supplied, and the value is an object
with a property of `source` and optionally `map`. The first is the output
contents of the module. The `map` property is the source map. Source maps are
provided by default, but can be turned off via the `options` argument.

An example:

```ts
const result = await Deno.transpileOnly({
  "/foo.ts": `enum Foo { Foo, Bar, Baz };\n`,
});

console.log(result["/foo.ts"].source);
console.log(result["/foo.ts"].map);
```

We would expect the `enum` would be rewritten to an IIFE which constructs the
enumerable, and the map to be defined.

### Referencing TypeScript library files

When you use `deno run`, or other Deno commands which type check TypeScript,
that code is evaluated against custom libraries which describe the environment
that Deno supports. By default, the compiler runtime APIs which type check
TypeScript also use these libraries (`Deno.compile()` and `Deno.bundle()`).

But if you want to compile or bundle TypeScript for some other runtime, you may
want to override the default libraries. To do this, the runtime APIs support the
`lib` property in the compiler options. For example, if you had TypeScript code
that is destined for the browser, you would want to use the TypeScript `"dom"`
library:

```ts
const [errors, emitted] = await Deno.compile(
  "main.ts",
  {
    "main.ts": `document.getElementById("foo");\n`,
  },
  {
    lib: ["dom", "esnext"],
  },
);
```

For a list of all the libraries that TypeScript supports, see the
[`lib` compiler option](https://www.typescriptlang.org/docs/handbook/compiler-options.html)
documentation.

**Don't forget to include the JavaScript library**

Just like `tsc`, when you supply a `lib` compiler option, it overrides the
default ones, which means that the basic JavaScript library won't be included
and you should include the one that best represents your target runtime (e.g.
`es5`, `es2015`, `es2016`, `es2017`, `es2018`, `es2019`, `es2020` or `esnext`).

#### Including the `Deno` namespace

In addition to the libraries that are provided by TypeScript, there are four
libraries that are built into Deno that can be referenced:

- `deno.ns` - Provides the `Deno` namespace.
- `deno.shared_globals` - Provides global interfaces and variables which Deno
  supports at runtime that are then exposed by the final runtime library.
- `deno.window` - Exposes the global variables plus the Deno namespace that are
  available in the Deno main worker and is the default for the runtime compiler
  APIs.
- `deno.worker` - Exposes the global variables that are available in workers
  under Deno.

So to add the Deno namespace to a compilation, you would include the `deno.ns`
lib in the array. For example:

```ts
const [errors, emitted] = await Deno.compile(
  "main.ts",
  {
    "main.ts": `document.getElementById("foo");\n`,
  },
  {
    lib: ["dom", "esnext", "deno.ns"],
  },
);
```

**Note** that the Deno namespace expects a runtime environment that is at least
ES2018 or later. This means if you use a lib "lower" than ES2018 you will get
errors logged as part of the compilation.

#### Using the triple slash reference

You do not have to specify the `lib` in the compiler options. Deno also supports
[the triple-slash reference to a lib](https://www.typescriptlang.org/docs/handbook/triple-slash-directives.html#-reference-lib-)
which can be embedded in the contents of the file. For example, if you have a
`main.ts` like:

```ts
/// <reference lib="dom" />

document.getElementById("foo");
```

It would compile without errors like this:

```ts
const [errors, emitted] = await Deno.compile("./main.ts", undefined, {
  lib: ["esnext"],
});
```

**Note** that the `dom` library conflicts with some of the default globals that
are defined in the default type library for Deno. To avoid this, you need to
specify a `lib` option in the compiler options to the runtime compiler APIs.

repository: piscisaureus/deno

gha_language: Rust

gha_license_id: MIT

repo_name: piscisaureus/deno


text:
# What is monolish? {#monolish_intro}

## Introduction
monolish is a linear equation solver library that monolithically fuses variable data type, matrix structures, matrix data format, vendor specific data transfer APIs, and vendor specific numerical algebra libraries.

monolish is a vendor-independent open-source library written in C++ that aims to be a grand unified linear algebra library on any hardware.

BLAS has 150+ functions and LAPACK has 1000+ functions.
These are complete software for dense matrix operations.
However, these are not enough due to the growing needs of users.
In particular, sparse matrix operations and machine learning kernels are not implemented.

To solve this, the libraries (MKL, cusolver, etc.) by each hardware vendor extended functions.
Sadly, these software APIs are not unified. Moreover, These are language or vendor specific.

The BLAS library, including MKL and CUDA libraries, is the perfect library in terms of performance, but the function names depend on the data type.
Python numpy, Julia, Matlab, etc. define APIs that eliminate these dependencies and call BLAS in them.
Fast Sparse BLAS is implemented in MKL and CUDA Libraries.
Sadly, there is no open-source Sparse BLAS library that works on all hardware.
However, due to the needs of application users, the Python and Julia libraries have implemented Sparse BLAS on their own.
The performance of these functions is not clear.

We think it is necessary to develop C/C++/Fortran libraries for dense and sparse matrices, which are the base of numerical computation.
monolish is a monolithic C++ numerical library designed to bridge the gap between HPC engineers and application engineers.

monolish provides an API that integrates the numerical algebra libraries of each vendor.
monolish calls the vendor-developed numerical algebra libraries whenever possible.
monolish implements and provides functions that are not implemented in these libraries.

We dream that monolish will be used as a backend for python, Julia, etc. libraries in the future.

monolish solves cumbersome package management by Docker.

monolish uses OpenMP Offloading for GPU acceleration. Currently, only NVIDIA GPUs are supported.
By using OpenMP Offloading, it is possible to support AMD Radeon and Intel Xe in the future.

## Switching libraries {#call_lib}

The first goal of monolish is to implement the basic operations that allow the BLAS, Sparse BLAS, and VML functions of the MKL and CUDA libraries to work on any hardware environment.

On Intel CPUs and NVIDIA GPUs, MKL and CUDA libraries are the fastest.
The monolish uses these libraries as much as possible and implements the missing functions by itself.

When compiling, monolish switches the function to be called if the MKL or CUDA libraries are available or not.
monolish wwitch dependency libraries at compile time.

The current monolish has four branches, `MKL`, `OSS`, `MKL + NVIDIA`, and `OSS + NVIDIA`, as shown in the following figure.

The branch for the case where MKL or CUDA libraries are not available is called `OSS`.
`OSS` is an implementation for architectures such as AMD, ARM, Power, etc.
For example, in the case of Intel, monolish uses MKL, and in the case of OSS it uses OpenBLAS.

In `OSS`, we assume that only CBLAS compatible BLAS libraries and LAPACK can be used.

The functions of MKL and CUDA libraries that are not implemented in CBLAS are implemented in monolish.
We plan to increase the number of libraries switching branches for AMD, ARM, and others.

![](img/call_blas.png)

## Development policy for high performance

monolish has five development policies.

### 1. Don't require users to change programs due to changes in data types

Override functions and do not implement datatype-dependent functions.

In the future, imaginary numbers and high precision data types will be supported.

### 2. Don't require users to change their programs due to changes in hardware architecture.

monolish integrates vendor-implemented BLAS libraries and device communication APIs for each architecture.
It provides a new layer to BLAS/LAPACK/Sparse BLAS that monolithically integrates types, matrix formats, and vendor-specific APIs.

To support AMD GPUs and Intel Xe with our own implementation, the internal device programs are implemented using OpenMP Offloading.

### 3. Don't require users to change their programs due to changes in the matrix storage format.

In monolish, all matrix formats, including Dense matrix, are defined as sparse matrix format.

The same interface is designed for all classes to minimize changes in user program due to the matrix storage format changes.

### 4. Don't implement functions that do not provide performance.

It is important to have the same functionality in all classes.
However, there are operations that cannot be made faster in principle.
For example, operations on columns of a matrix in CRS format.

If a useful function is implemented, many users will use it even if its performance is low.
In monolish, even if there is a bias in functionality between classes, functions that are not fast enough in principle are not implemented.

We guarantee, as much as possible, that programs implemented with a combination of monolish functions will run fast.

If a function is absolutely necessary, warn the user by writing the number of computation in the documentation.

### 5. Don't allocate memory in ways that users cannot anticipate.

Compound functions and operator overloading are useful.
However, they often require to allocate memory in the function for the return vector or matrix internally.
users cannot anticipate this allocation.

In monolish, we do not implement functions that allocate memory in a way that is not intuitively expected by the user.

repository: termoshtt/monolish

repo_name: termoshtt/monolish


text:
# OONI Network Extensions

| Author       | Simone Basso |
|--------------|--------------|
| Last-Updated | 2019-09-30   |
| Status       | approved     |

## Introduction

OONI experiments send and/or receive network traffic to
determine if there is blocking. We want the implementation
of OONI experiments to be as simple as possible.

At the same time, we want an experiment to collect as much
low-level data as possible. For eample, we want to know
whether and when the TLS handshake completed; what certificates
were provided by the server; what TLS version was selected;
and so forth. These bits of information are very useful
to analyze a measurement and better classify it.

We also want to be able to change some configuration properties
and repeat the measurement; e.g., we may want to configure DNS
over HTTPS (DoH) and then attempt to fetch again an URL. Or
we may want to force TLS to use a specific SNI.

In this document we design a Go library that solves all the
above problems by exposing to the user an API with simple replacements
for standard Go interfaces, e.g. `http.RoundTripper`.

## Rationale

As we observed [in a recent ooni/probe-engine issue](
https://github.com/ooni/probe-engine/issues/13), every
experiment consists of two separate phases:

1. measurement gathering

2. measurement analysis

During measurement gathering, we perform specific actions
that cause network data to be sent and/or received. During
measurement analysis, we process the measurement on the
device. For some experiments (e.g., Web Connectivity), this
second phase also entails contacting OONI backend services
that provide data useful to complete the analysis.

In [Measurement Kit](https//github.com/measurement-kit/measurement-it),
we implement measurement gathering by combining _nettest
templates_. These are special APIs that perform a single
low-level action (e.g. connecting to a TCP endpoint, resolving
a domain name). So, for example, Web Connectivity's
measurement gathering is obtained by combining the DNS
template, the TCP template, and the HTTP template.

This approach based on combining low-level test helpers
has two problems. First, the implementation of an
experiment is rather low level, because you need to
invoke the test helpers in sequence, to populate the
measurement result object. Second, the test helpers API is likely
to eventually change when new measurement techniques
are added to the measurement engine.

Because Go has powerful interfaces, we propose in this
document to use an alternative approach where we provide
OONI-measurements-aware replacements for Go standard
library interfaces, e.g., `http.RoundTripper`.

This repository is separate from `ooni/probe-engine`
because they solve different problems. Here we provide
replacements for standard Go library interfaces that
allow us to perform measurements. In `probe-engine` we
implement OONI tests and clients for OONI backend
services. Putting all the code into the same repository
would have put too many concerns into the same repo.

## Design

We want to provide moral replacements for the following
interfaces in the Go standard library:

1. `http.RoundTripper`

2. `http.Client`

3. `net.Dialer`

4. `net.Resolver`

Where possible (e.g. for `http.RoundTripper`) we will
provide structures implementing the interface. Where
instead this is not possible (e.g. for `net.Dialer`) we
will provide structures implementing methods that are
compatible with the originals. For example, in the
case of `net.Dialer`, we will provide compatible
functions, such as `Dial`, `DialContext`, and `DialTLS`.

This make it possible to use our `net.Dialer`
replacement with other libraries. Both `http.Transport`
and `gorilla/websocket`'s `websocket.Dialer` have
functions like `Dial` and `DialContext` that can be
overriden. Therefore, we will be able to use our
replacements to collect measurements.

There will be a mechanism for gathering such low
level measurements as they occur, for logging
and/or storing purposes.

A OONI experiment is expected to create instances of
our replacement objects, configure them properly,
then use our replacements, which are compatible with
standard library mechanisms to perform their task,
e.g. fetching a URL. After the measurement task
is completed, the experiment code will include the
low-level events into the measurement result object, and will walk
through the stream of events to determine in a more
precise way what could have gone wrong.

## Implementation

The actual implementation must follow this spec. It may include more
methods or interfaces. The exact structure of measurements events
is left unspecified, as they are likely to change. That said, we will
be careful to not remove existing fields and/or change the meaning
of existing fields unless that is necessary.

### The github.com/ooni/netx/model package

This package will contain the definition of low-level
events. We are interested in knowing the following:

1. the timing and result of each I/O operation.

2. the timing of HTTP events occurring during the
lifecycle of an HTTP request.

3. the timing and result of the TLS handshake including
the negotiated TLS version and other details such as
what certificates the server has provided.

4. DNS events, e.g. queries and replies, generated
as part of using DoT and DoH.

Hence, this package should define measurement events
representing each of the above. We will use types
as close as possible to standard Go types, e.g. we
will use `time.Duration` to represent the elapsed
time since a specific "zero", because this will allow
for easy further processing of events.

This package will also contain the definition of the
following interface:

```Go
type Handler interface {
    OnMeasurement(Measurement)
}
```

Every replacement that we write will call the
`OnMeasurement` method of the handler wherever
there is a measurement event.

In turn, the `Measurement` event will be defined
as follows:

```Go
type Measurement struct {
    Close                   *CloseEvent
    Connect                 *ConnectEvent
    DNSQuery                *DNSQueryEvent
    DNSReply                *DNSReplyEvent
    HTTPConnectionReady     *HTTPConnectionReadyEvent
    HTTPRequestStart        *HTTPRequestStartEvent
    HTTPRequestHeadersDone  *HTTPRequestHeadersDoneEvent
    HTTPRequestDone         *HTTPRequestDoneEvent
    HTTPResponseStart       *HTTPResponseStartEvent
    HTTPResponseHeadersDone *HTTPResponseHeadersDoneEvent
    HTTPResponseDone        *HTTPResponseDoneEvent
    Read                    *ReadEvent
    Resolve                 *ResolveEvent
    TLSHandshake            *TLSHandshakeEvent
    Write                   *WriteEvent
}
```

That is, it will contain a pointer for every event
that we support. The events processing code will
check what pointer or pointers are not `nil` to
known which event or events have occurred.

The following network-level events will be defined:

1. `CloseEvent`, indicating when a socket is closed
2. `ConnectEvent`, indicating the result of connecting
3. `Read`, indicating when a `read` completes
4. `Resolve`, indicating when a name resolution completes
5. `Write`, indicating when a `write` completes

The following DNS-level events will be defined:

1. `DNSQueryEvent`, containing the query data
2. `DNSReplyEvent`, containing the reply data

The following HTTP-level events will be defined:

1. `HTTPConnectionReadyEvent`, indicating when the connection
is ready to be used by HTTP code

2. `HTTPRequestStartEvent`, indicating when we start sending the request

3. `HTTPRequestHeadersDoneEvent`, indicating when we have sent the
request headers, and containing the sent headers

4. `HTTPRequestDoneEvent`, indicating when the whole request has been sent

5. `HTTPResponseStartEvent`, indicacting when we receive the first
byte of the HTTP response

6. `HTTPResponseHeadersDoneEvent`, indicating when we have received the
response headers, and containing headers and status code

7. `HTTPResponseDoneEvent`, indicating when we have received the
response body

Every event will include at the minimum this field:

```Go
    Time     time.Duration
```

This will be the time when the event occurred, relative to
a configured "zero" time. If an event pertains to a blocking
operation (i.e. `Read`), it will also contain this field:

```Go
    Duration time.Duration
```

This will be the amount time we have been waiting for
the event to occur. That is, in the case of `Read` the
amount of time we've been blocking waiting for the `Read`
operation to return a value or an error.

Every operation that can fail will have a field

```Go
    Error    error
```

This will indicate the error that occurred.

Measurement events will also contain contextual information
that is meaningful to the event itself. Since this is likely
to change as we improve our understanding of what could
be measured, as stated above, please see the current documentation
for more information on the structure of each event.

Every network event will be additionally identified by

```Go
    ConnID   int64
```

Where `ConnID` is the identifier of the connection and is
unique within a specific set of measurements.

Likewise, HTTP events will have their

```Go
    TransactionID int64
```

which will uniquely identify the round trip within a specific
set of measurements.

Because in this first PoC it has been deemed complex to access
the `ConnID` from the HTTP code, we have determined that we
will be using the five-tuple to join network and HTTP
events. Accordingly, both the `ConnectEvent` and
`HTTPConnectionReadyEvent` structures will thus include:

```Go
    LocalAddress  string
    Network       string
    RemoteAddress string
```

The problem of joining together network and HTTP level
measurements is currently not solved by this library. If
we perform a measurement at a time, however, this may
not be a big issue, because all the low-level events will
necessarily pertain to a single measurement, e.g., to
the fetching of a specific URL.

A subsequent revision of this specification will see
whether we can join measurements in a better way.

(As a contextual note, the problem of knowing the ID
of a connection is that we cannot wrap `*tls.Conn`
with a ConnID-aware-replacement that is compatible with `net.Conn`,
because that will confuse `net/http` and prevent using
`http2`. We could solve the problem to join automatically network
and lower-level events by implementing a goroutine
safe cache mapping the five tuple to a `ConnID`.)

### The github.com/ooni/netx/httpx package

This package will contain HTTP extensions. The core
structure that we will provide is as follows:

```Go
type Client struct {
  HTTPClient *http.Client
  Transport  Transport
}
```

Client code is expected to create a `*Client` instance
using the `NewClient` constructor, configure it, and
then pass to code that needs it `HTTPClient` as the real
`*http.Client` instance.

To configure our `*Client` instance, one could use the
`ConfigureDNS`, `SetCABundle` and `ForceSpecificSNI`
methods. They should all be called before using the
`HTTPClient` field, as they'll not be goroutine safe.

```Go
func (c *Client) SetCABundle(path string) error
```

The `SetCABundle` forces using a specific CA bundle,
which is what we already do in OONI Probe.

```Go
func (c *Client) ForceSpecificSNI(sni string) error
```

The `ForceSpecificSNI` forces the TLS code to use a
specific SNI when connecting. This allows us to check
whether there is SNI-based blocking.

```Go
func (c *Client) ConfigureDNS(network, address string) error
```

The `ConfigureDNS` method will behave exactly like the
`ConfigureDNS` method of `netx.Resolver` (see below).

```Go
func (c *Client) SetProxyFunc(f func(*Request) (*url.URL, error) error
```

The `SetProxyFunc` will allow us to configure
a specific proxy. This is useful to have precise
measurements of requests over, say, Psiphon.

Lastly, one will construct an `http.Client` using:

```Go
func NewClient(handler model.Handler) *Client
```

The `handler` shall point to a structure implementing the
`model.Handler` interface. Also, this constructor will
automatically record the current time as the "zero" time
used to compute the `Time` field of every event.

### The github.com/ooni/netx package

This package will contain a replacement for `net.Dialer`,
called `netx.Dialer`, that exposes the following API:

```Go
func (d *Dialer) Dial(network, address string) (net.Conn, error)
```

```Go
func (d *Dialer) DialContext(
    ctx context.Context, network, address string,
) (net.Conn, error)
```

```Go
func (d *Dialer) DialTLS(network, address string) (conn net.Conn, err error)
```

These three functions will behave exactly as the same
functions in the Go standard library, except that they
will perform measurements. A `Dialer` replacement will be
constructed like:

```Go
func NewDialer(handler model.Handler) *Dialer
```

This function is like `httpx.NewClient` and, specifically, it also
uses the current time as "zero" for subsequent events.

The `netx.Dialer` will also feature the following functions, to
be called before using the dialer:

```Go
func (c *Client) ConfigureDNS(network, address string) error
```

```Go
func (c *Client) SetCABundle(path string) error
```

```Go
func (c *Client) ForceSpecificSNI(sni string) error
```

`SetCABundle` and `ForceSpecificSNI` behave exactly like the same
methods of `httpx.Client`.

As far as `ConfigureDNS` is concerned it will work as follows:

* when `network` is `"system"`, the system resolver will be
used and no low-level events pertaining to the DNS will be
emitted to the configured `handler`. This will be the default.

* when `network` is `"netgo"`, we will try to use the DNS
resolver written in Go within the standard library (which is
know to work only on Unix), and we will use a bunch of
hacks to observe the events occurring during name resolutions,
such as `Read` and `Write` events. We will also be able to
record the DNS messages sent and received.

* when `network` is `"udp"`, `address` must be a valid
string following the `"<ip>:<port>"` pattern. This will
indicate the code to use the selected DNS server using
UDP transport. We will be able to observe all events including
DNS messages sent and received.

* when `network` is `"tcp"`, everything will be like when
`network` is `"udp"`, except that we will speak the DNS
over TCP protocol with the configured server.

* when `network` is `"dot"`, `address` must be a valid
domain name of a DNS over TLS server to use. We will
observe all events, which of course include the results
of the TLS handshake with the server, the DNS messages
sent and received, etc.

* when `network` is `"doh"`, `address` must be a valid
URL of a DNS over HTTPS server to use. We will observe all
events, including the TLS handshake and HTTP events, the
DNS messages sent and received, etc.

Lastly, `netx.Dialer` will expose this API:

```Go
func (d *Dialer) NewResolver(network, address string) (dnsx.Client, error)
```

The arguments have the same meaning of `ConfigureDNS` and
the will return an interface replacement for `net.Resolver`
as described below.

### The github.com/ooni/netx/dnsx package

This package will define an interface compatible with the
`net.Resolver` struct, such that its methods can be used
as replacaments for the golang stdlib `net.Resolver` methods:

```Go
type Client interface {
    LookupAddr(ctx context.Context, addr string) (names []string, err error)
    LookupCNAME(ctx context.Context, host string) (cname string, err error)
    LookupHost(ctx context.Context, hostname string) (addrs []string, err error)
    LookupMX(ctx context.Context, name string) ([]*net.MX, error)
    LookupNS(ctx context.Context, name string) ([]*net.NS, error)
}
```

## Future work

The current revision of this specification does not specify a
programmatic way of joining measurements occurring at different
levels (e.g. network and HTTP). This has been done under the
assumption that we will probably be able to understand the
sequence of events anyway, by looking at the timing, if we're
measuring a single URL at a time. We will implement and
deploy code conformant with this specification and see whether
this assumption is correct, or we need something else.

repository: limkokholefork/netx

repo_name: limkokholefork/netx


text:
# libmustache

[![Build Status](https://travis-ci.org/jbboehr/libmustache.png?branch=master)](https://travis-ci.org/jbboehr/libmustache)

C++ implementation of Mustache intended mainly for use as a PHP extension.
See [php-mustache](https://github.com/jbboehr/php-mustache) and [mustache](https://mustache.github.com/)

## License

This project is licensed under the [MIT license](http://opensource.org/licenses/MIT).

repository: cgvarela/libmustache

repo_name: cgvarela/libmustache


text:
# Organizing observers

You can organize your signals and observers in any way you like and link them freely with
`Sea.Signal.emit_to/1` macro calls. Sea doesn't limit you in this regard as it'll depend on specific
use cases how best to name and split observers mapped to the signals. In this guide, we'll review
some of possible approaches and propose technical solutions for applying them efficiently.

For instance, you could choose to share observer logic for handling multiple signals by implementing
single observer module with multiple `c:Sea.Observer.handle_signal/1` clauses, target it from
multiple signals and pattern-match specific signal or group of signals there. The
`AnalyticsObserver` from the [getting started guide] could become such multi-signal observer, ie:

    defmodule AnalyticsObserver do
      use Sea.Observer

      @impl true
      def handle_signal(%InvoiceCreatedSignal{invoice_number: invoice_number}) do
        IO.puts("Adding invoice #{invoice_number} to analytics accumulators")
        increase_count(:invoice)
      end

      @impl true
      def handle_signal(%UserRegisteredSignal{email: email}) do
        IO.puts("Adding user #{email} to analytics accumulators")
        increase_count(:user)
      end

      defp increase_count(type) do
        # common logic here
      end
    end

This approach was applied in exemplary [`InvoicingApp.Analytics.Observer`] module.

An opposite approach would be to implement observers dedicated to specific signals and perhaps map
signal and observer names across context modules in order to simplify working with a growing number
of signals and observers scattered across project modules, ie:

    defmodule InvocingApp.Sales.InvoiceCreatedSignal do
      use Sea.Signal

      emit_to InvocingApp.Customers.InvoiceCreatedObserver
      emit_to InvocingApp.Inventory.InvoiceCreatedObserver

      # ...
    end

    defmodule InvocingApp.Customers.InvoiceCreatedObserver do
      use Sea.Observer

      # ...
    end

    defmodule InvocingApp.Inventory.InvoiceCreatedObserver do
      use Sea.Observer

      # ...
    end

This is indeed how exemplary [`InvoicingApp.Customers.InvoiceCreatedObserver`] and
[`InvoicingApp.Inventory.InvoiceCreatedObserver`] were organized.

Regardless of the approach, you may want to avoid a fishy practice from the above example - forcing
signals nested in `InvoicingApp.Sales` module to pick specific observer modules internal to
`InvoicingApp.Customers` or `InvoicingApp.Inventory` - they should be a private implementation
detail of these modules. This can be achieved by delegating `handle_signal` from entry module to one
of its nested observers. For the "single observer" approach, this could be as simple as:

    def InvoicingApp.Analytics do
      use Sea.Observer

      defdelegate handle_signal(signal), to: __MODULE__.Observer
    end

For "one signal one observer" approach we could pick observer dynamically based on signal name:

    def InvoicingApp.Customers do
      use Sea.Observer

      @impl true
      def handle_signal(signal = %{__struct__: signal_mod}) do
        signal_name = signal_mod |> Module.split() |> List.last()
        observer_name = String.replace(signal_name, ~r/Signal$/, "Observer")
        observer_mod = :"#{__MODULE__}.#{observer_name}"

        observer_mod.handle_signal(signal)
      end
    end

Here, the original signal name was taken and the `Signal` suffix was replaced with `Observer` suffix
in order to infer the name of child observer module within specific entry module.

Finally, you could put one (or both) of these observer organization strategies into reusable module
that you would subsequently `use` in all entry modules that include observers. A basic version of
such router that covers routing strategies described above is available in `Sea.SignalRouter`
module. You can easily create your own as well.

This way all of your observers will be organized in consistent and predictable way, with a single
module that describes the rules governing this aspect of your application and with explicit yet
compact references to it all over the project.

[getting started guide]: getting_started.html
[`InvoicingApp.Analytics.Observer`]: https://github.com/surgeventures/sea-elixir/tree/master/examples/invoicing_app/lib/invoicing_app/analytics/observer.ex
[`InvoicingApp.Customers.InvoiceCreatedObserver`]: https://github.com/surgeventures/sea-elixir/tree/master/examples/invoicing_app/lib/invoicing_app/customers/invoice_created_observer.ex
[`InvoicingApp.Inventory.InvoiceCreatedObserver`]: https://github.com/surgeventures/sea-elixir/tree/master/examples/invoicing_app/lib/invoicing_app/inventory/invoice_created_observer.ex

repository: surgeventures/sea-elixir

gha_language: Elixir

repo_name: surgeventures/sea-elixir


text:
```@meta
Author = "Robert Rosca"
```

# [Constants](@id lib_constants)

Constants contains a number of constant arrays and dictionaries used to define
which keys and attributes are expected to be present in a GRIB message, how to
map grid types to correct representations of the data, etc...

These constants are shown in the documentation for ease of use and reference
purposes.

---

```@index
Modules = [CfGRIB]
Pages   = ["constants.jl"]
```

```@autodocs
Modules = [CfGRIB]
Pages   = ["constants.jl"]
```

branch_name: refs/heads/dev


text:
This module is not yet tested. This module is an implementation of my advice from [this forum post](https://devforum.roblox.com/t/example-code-on-developer-product-purchase-handling-misses-a-key-point-that-can-cause-devs-to-lose-revenue/268980/5?u=corecii).

# PurchaseProcessor

*PurchaseProcessor* is a module for Roblox that tracks purchase state to safely award and save purchases.

Most example ProcessReceipt code does not handle edge-cases such that leaving too early, data save failing, or buying another product can lead to lost money or double of a product. *PurchaseProcessor* tracks product purchase state to prevent these things.

*PurchaseProcessor* will only track products in player save data for four days. After four days, Roblox will not call ProcessReceipt with those purchase ids anymore.

## Scenarios

*PurchaseProcessor* handles the following scenarios:

| Scenario | Action |
| :------- | :----- |
| ProcessReceipt called on `purchaseId` that's already being processed | Only process the first time (prevent double awarding) |
| Player leaves during processing | Retry on next join (or refund after 3 days of not playing) |
| Data Stores failing or server crash before saving | Retry on next join (or refund after 3 days of not playing) |
| Data saves very late after awarding product | Return PurchaseGranted for the existing award (prevent double awarding) |
| Awarding product fails | Retry later (or refund after 3 days) (prevent accidental "scams" due to errors) |

## How-to

To use *PurchaseProcessor*, you implement a *bridge*. This bridge lets *PurchaseProcessor* save relevant data and trigger your award-product code.

Here is a [Template Bridge](./Example/TemplateBridge.lua).
Here is an [Example Bridge](./Example/ExamplePurchaseProcessorBridge.lua).
Here is an [Example startup script](./Example/ExampleStartup.lua).

repository: Corecii/PurchaseProcessor

repo_name: Corecii/PurchaseProcessor


text:
## Lid-driven cavity flow at Re=100

The input files for this example are located in the directory `$PETIBM_DIR/examples/2d/lidDrivenCavity/Re100`.

The fluid, with kinematic viscosity `0.01` is enclosed in a two-dimensional box that spans `[0.0, 1.0]` in both directions. All solid walls are fixed except the top one that moves in the x-direction at velocity speed `1.0`. The Reynolds number of the flow, based on the box-length and the top-wall speed, is `100`. The fluid is initially at rest and the top-wall impulsively starts moving.

The computational domains discretized using a uniform `32x32` Cartesian grid. We choose a time-increment of `0.02`.

To run the case in parallel using `4` MPI processes, change directory to your PetIBM build directory, then:

    > cd examples
    > make examples
    > make cavity2dRe100Parallel

The program runs for `1000` time-steps, and saves the numerical solution at the final time-step (at which we consider the flow has reach a steady-state).

We may now post-process the results:

    > python $PETIBM_DIR/scripts/python plotFields2d.py --case 2d/lidDrivenCavity/Re100

The command-line above, generates the contours of the velocity components, the pressure field and the vorticity field after `1000` time-steps. The figures are saved as `.png` files in the sub-folder `images` of the simulation directory.

Here are the figures generated:

[[images/lidDrivenCavity2dRe100/pressure0001000.png]]

[[images/lidDrivenCavity2dRe100/uVelocity0001000.png]]

[[images/lidDrivenCavity2dRe100/vVelocity0001000.png]]

[[images/lidDrivenCavity2dRe100/vorticity0001000.png]]

## flow over a circular cylinder at Re=40

A circular cylinder of diameter `1.0` is placed at the center of a two-dimensional domain spanning `[-15,15]x[-15,15]`. The initial velocity of the fluid in the domain is `(1, 0)`. Dirichlet boundary conditions for the velocity are on all edges (velocity set to `(1.0, 0.0)`), except at the outlet where a the fluid is convective outside the domain in the x-direction at speed `1.0`. The Reynolds number, based on the freestream speed and the diameter of the circular cylinder, is `40`.

The computational domain is discretized using an stretched grid with `186x186` cells. The uniform region consists of the box `[-0.6, 0.6]x[-0.6, 0.6]` that contains `48x48` cells, giving a grid-resolution of `0.025` in both directions. Then, a stretching-ratio of `1.05` is applied in all directions from the edges of the uniform region to the domain limits.

We run the simulation for `300` time-steps with time-increment of `0.01`, saving the numerical solution every `50` iterations.

To run this example using `2` MPI processes, navigate to your PetIBM build directory, then:

    > cd examples
    > make examples
    > make cylinder2dRe40

You may now post-process the results:

    > python $PETIBM_DIR/scripts/python/plotFields2d.py --case 2d/cylinder/Re40 \
                  -- bottom-left -2.0 -2.0 \
                  --top-right 4.0 2.0 \
                  --no-velocity \
                  --pressure-range -1.0 0.5 101 \
                  --vorticity-range -1.0 1.0 101

The command-line above generates the contours within the box `[-2.0, 4.0]x[-2.0, 2.0]` of the pressure field, the vorticity field (between `-1.0` and `1.0` using `101` contours), excluding the velocity fields, at each time-step saved.

Here are the figures generated:

[[images/cylinder2dRe40/pressure0000300.png]]

[[images/cylinder2dRe40/vorticity0000300.png]]

branch_name: refs/heads/develop


text:
# General Attachments Worker

This worker proesses attachments.

# Deprication Note
This is depricated in favor of [couchmagick](https://github.com/jo/couchmagick)!

## Installation

    npm install worker-attachment

## Create a new Worker

Define an object with two functions: _check_ and _process_.

Look at examples/

## Per Database Configuration

Configuration is done in a worker configuration document inside the target database.
The worker looks at all databases and only process if there exists such a configuration file.

A Worker Configuration File might look like this:

    {
      "_id": "worker-config/attachments",
      "_rev": "9-a653b27246b01cf9204fa9f5dee7cc64",
      "versions": {
        "thumbnail": {
          "size": "135x135"
        }
      }
    }

You can update the config live so that all future processings will take the new configuration.

Note that attachment names which starts with a version name, followed by a '/' are ignored.

## Worker Status Document

The worker stores a status document inside the target database.
The worker stores its last update seq here and can resume at the point it stopped.

    {
      "_id": "worker-status/attachments",
      "_rev": "1-1922b5623d07453a753ab6ab2c634d04",
      "last_update_seq": 34176
    }

## Document Status Object

The worker updates a status object inside the document.
This makes it supereasy to monitor worker status as well as
it keeps an atomic lock when many workers listen to the same database.
Images are prozessed only once by comparing the attachments _revpos_ property
with the revpos property of the status object.

The status object of the worker looks like this:

    "worker_status": {
      "worker-name": {
        "attachment-name": {
          "status": "completed",
          "revpos": 160
        }
      }
    }

The status field can be _triggered_, _completed_ or _error_.

The worker status is scoped by the worker name in order to have many workers
processing the same document.
The worker status is also per attachment to process only attachments which had changed.

## Running the Worker

To start, this needs either the following environment variables set:

    export HOODIE_SERVER=http://example.org
    npm start

or pass them to the commandline:

    HOODIE_SERVER=http://example.org npm start

## Testing

Testing is done with Mocha. Run the tests with

    mocha

## License & Copyright

(c) 2012 Johannes J. Schmidt, null2 GmbH, Berlin

Licensed under the Apache License 2.0.Licensed under the Apache License 2.0.

repository: jo/worker-attachments

repo_name: jo/worker-attachments


text:
# Prediction Metrics

This project implements several metrics to measure the prediction
quality of probabilistic models that are implemented using TensorFlow.

There are two uses of predictive metrics:

1.  _Model evaluation_: assessing the quality of your probabilistic model on a
    validation or test set.
2.  _Model training_: you can use the loss functions defined in this project to
    train probabilistic classification and regression models.

To use it, import the desired submodule. For example, in order to use the
regression scores, use

```python
from uncertainty_metrics import regression
```

## Table of Contents

[TOC]

## Key Concepts

### Proper Scoring Rules {#proper-scoring-rules}

_Proper scoring rules_ are loss functions for probabilistic predictions.
Formally, a proper scoring rule is a function which assign a numerical score
$$S(P,y)$$ to a predicted distribution $$P$$ and a realized value $$y$$.
Assuming that data instances are generated according to an unknown distribution,
$$(x_i,y_i) \sim Q$$, we would evaluate $$S(P(y|x_i),y_i)$$.

Common examples of proper scoring rules include the log-likelihood, the Brier
score, and the continuous ranked probability score (CRPS).

__Reference__

* _Tilmann Gneiting_ and _Adrian E. Raftery_, "Strictly Proper Scoring Rules,
  Prediction, and Estimation", Journal of the American Statistical Association
  (JASA), 2007.
  [(PDF)](https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf)

### Calibration {#calibration}

_Calibration_ refers to a frequentist property of probabilistic predictions being correct _on average_.  Intuitively, when predicting a binary outcome using a model, if we group all predictions where the outcome is believed to be 80 percent probable, then within this group we should, on average, see this outcome become true 80 percent of the time.  A model with this property is said to be well-calibrated.

Formally, calibration can best be defined in terms of measuring the difference between a conditional predictive distribution and the true conditional distribution, where
the conditioning is done with respect to a set defined as a function of the prediction.
For $$(x,y) \sim Q$$, with $$x \in \mathcal{X}$$, we consider $$\gamma = f(x)$$ a function of the prediction.
For example, $$\gamma$$ could be the vector of logits of a classifier, or a function thereof.  (In practice it will often be a discretization of a continuous quantity.)

Then, the _true conditional distribution_ $$q(y|\gamma)$$ corresponds to the true distribution over $$y$$ over the subset
$$\mathcal{X}_{\gamma} := \{x \in \mathcal{X} \,|\, f(x) = \gamma\}$$, i.e.

$$q(y|\gamma) =
\frac{\mathbb{E}_{x \sim Q}[\mathbb{1}_{\{f(x)=\gamma\}} \, q(y|x)]}{Q(\mathcal{X}_{\gamma})}.$$

The _model predictive conditional distribution_ $$p(y|\gamma)$$ is likewise given as

$$p(y|\gamma) =
\frac{\mathbb{E}_{x \sim Q}[\mathbb{1}_{\{f(x)=\gamma\}} \, p(y|x)]}{Q(\mathcal{X}_{\gamma})}.$$

The _reliability_ of a probabilistic prediction system is now defined as an expected difference between the two quantities,

$$\textrm{Reliability} = \mathbb{E}_{(x,y) \sim Q}[
D(q(y|\gamma(x)) \,\|\, p(y|\gamma(x)))].$$

A reliability of zero means that the model is perfectly calibrated: the predictions are on average correct.
In practice the reliability needs to be estimated: none of the expectations are available analytically.  Most importantly, to estimate the true conditional distribution requires discretization of the set of predictions.

__Reference__

* _Jochen Brocker_, "Reliability, sufficiency, and the decomposition of proper scores",
  Quarterly Journal of the Royal Meteorological Society, 2009.
  [(PDF)](https://rmets.onlinelibrary.wiley.com/doi/pdf/10.1002/qj.456)

### Mutual Information {#mutual-information}

Mutual information is a way to measure the spread or _disagreement_ of an
ensemble $$\{p(y|x; \theta^{(m)})\}^M_{m=1}$$. These metrics are used after or
during model training to estimate the predictive performance on the
hold-out/test set. We differ between two mutual information based metrics,
_model uncertainty_ and _knowledge uncertainty_. Model uncertainty estimates the
mutual information between the categorical label $$y$$ and the model parameters
$$\theta$$, whereas knowledge uncertainty is the mutual information between the
categorical label $$y$$ and parameters $$\pi$$ of the categorical distribution.
Both model and knowledge uncertainty can be expressed as the difference of the
total uncertainty and the expected data uncertainty, where total uncertainty is
the entropy of expected predictive distribution and expected data uncertainty is
the expected entropy of individual predictive distribution.

Formally, given an ensemble model $$\{p(y|x^*,\theta^{(m)})\}_{m=1}^M$$ trained
on a finite dataset $$D$$, model uncertainty for a test input $$x^*$$ is defined
as:

$$ MI[y, \theta|x^*, D] =
\mathcal{H}(\mathbb{E}_{p(\theta|D)}[p(y|x^*,\theta)]) -
\mathbb{E}_{p(\theta|D)}[\mathcal{H}(p(u|x^*, \theta))] $$

The total uncertainty will be high whenever the model is uncertain - both in
regions of severe class overlap and out-of-domain. However, model uncertainty,
the difference between total and expected data uncertainty, will be non-zero iff
the ensemble disagrees.

Knowledge uncertainty estimates the mutual information between between the
categorical output $$y$$ and the parameters $$\pi$$ of the categorical
distribution. Malinin et al. introduced this metric specifically for ensemble
distribution distillation. It behaves exactly the same way as model uncertainty,
but the spread is now explicitly due to distributional uncertainty, rather than
model uncertainty. Given $$p(\pi|x^*, \hat{\theta}) = \textrm{Dir}(\pi|\alpha)$$
which models a Dirichlet distribution over the categorical output distribution
$$\pi$$, knowledge uncertainty is defined as:

$$
MI[y, \pi|x^*, \hat{\theta}] =
\mathcal{H}(\mathbb{E}_{p(\pi| x^*, \hat{\theta})}[p(y|\pi)]) -
\mathbb{E}_{p(\pi|x^*,\hat{\theta})}[\mathcal{H}(p(y|\pi))]
$$

__Reference__

*   _Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and
    Steffen Udluft_, "Decomposition of uncertainty for active learning and
    reliable reinforcement learning in stochastic systems", stat 1050, 2017.
    [(PDF)](https://proceedings.mlr.press/v80/depeweg18a/depeweg18a.pdf)

*   _Andrey Malinin, Bruno Mlodozeniec and Mark Gales_, "Ensemble Distribution
    Distillation.", arXiv:1905.00076, 2019.
    [(PDF)](https://arxiv.org/pdf/1905.00076.pdf)

### Information Criteria {#information-criteria}

_Information criteria_ are used after or during model training to estimate the predictive performance on future holdout data.  They can be useful for selecting among multiple possible models or to perform hyperparameter optimization.  There are also strong connections between [cross validation estimates](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) and some information criteria.

__Reference__

* _Sumio Watanabe_, "Mathematical Theory of Bayesian Statistics",
  CRC Press, 2018.

## API: Scoring Rules

### Brier score {#brier-score}

The Brier score for discrete labels is defined as follows: given a predicted
probability vector $$P = (p_1,p_2,\dots,p_L)$$ and given a realized value $$y
\in \{1,2,\dots,L\}$$ the brier score is

$$S_{\textrm{Brier}}(P,y) = -2 p_y + \sum_{i=1}^L p_i^2.$$

Here is an example of how to use the Brier score as loss function for a
classifier. Suppose you have a classifier implemented in Tensorflow and your
current training code looks like

```python
  per_example_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
    labels=target_labels, logits=logits)
  loss = tf.reduce_mean(per_example_loss)
```

Then you can alternatively use the API-compatible [_Brier loss_](#brier-score)
as follows:

```python
  per_example_loss = predictive_metrics.calibration.brier_score(
    labels=target_labels, logits=logits)
  loss = tf.reduce_mean(per_example_loss)
```

The Brier score penalizes low-probability predictions which do occur less than
the cross-entropy loss.

### Continuous Ranked Probability Score (CRPS) {#crps}

The continuous ranked probability score (CRPS) has several equivalent
definitions.

_Definition 1_: CRPS measures the integrated squared difference between an
arbitrary cummulative distribution function $$F$$ and a
[Heaviside distribution function](https://en.wikipedia.org/wiki/Heaviside_step_function)
at the realization, $$y$$,

$$S_{\textrm{CRPS}}(F,y) = \int_{-\infty}^{\infty}
(F(z) - \mathbb{1}_{\{z \geq y\}})^2 \,\textrm{d}z.$$

_Definition 2_: CRPS measures the expected distance to the realization minus one
half the expected distance between samples,

$$S_{\textrm{CRPS}}(F,y) = \mathbb{E}_{z \sim F}[|z-y|]
- \frac{1}{2} \mathbb{E}_{z,z' \sim F}[|z-z'|].$$

CRPS has two desirable properties:

1. It generalizes the absolute error loss and recovers the absolute error if a predicted distribution $$F$$ is deterministic.
2. It is reported in the same units as the predicted quantity.

To compute CRPS we either need to make an assumption regarding the form of $$F$$
or need to approximate the expectations over $$F$$ using samples from the
predictive model. In the current code we implement one analytic solution to CRPS
for predictive models with univariate Normal predictive distributions, and one
generic form for univariate predictive regression models that uses a sample
approximation.

__Normal model__

For a regression model which predicts Normal distributions with mean `means` and
standard deviation `stddevs`, we can compute CRPS as follows,

```python
  squared_errors = tf.square(target_labels - pred_means)
  per_example_crps = predictive_metrics.regression.crps_normal_score(
    labels=target_labels,
    means=pred_mean,
    stddevs=pred_stddevs)
```

__Sample-based approximation__

As long as we can sample predictions from our model, we can construct a Tensor
`predictive_samples` of size `(ninstances, npredictive_samples)` and evaluate the
Monte Carlo CRPS against the true targets `target_labels` using the following
code,

```python
  per_example_crps = predictive_metrics.regression.crps_score(
    labels=target_labels,
    predictive_samples=predictive_samples)
```

## API: Calibration Metrics

### Brier decomposition {#brier-decomposition}

Here is an example of how to compute calibration metrics for a classifier.
Suppose you evaluate the accuracy of your classifier on the validation set,

```python
  logits = model(validation_data)
  class_prediction = tf.argmax(logits, 1)
  accuracy = tf.metrics.accuracy(validation_labels, class_prediction)
```

You can compute additional metrics using the so called
[Brier decomposition](#brier-decomposition) that quantify prediction
_uncertainty_, _resolution_, and _reliability_ by appending the following code,

```python
  uncert, resol, reliab = predictive_metrics.calibration.brier_decomposition(
    labels=validation_labels, logits=logits)
```

In particular, the reliability (`reliab` in the above line) is a measure of
calibration error, with a value between 0 and 2, where zero means perfect
calibration.

### Expected Calibration Error {#ece}

The expected calibration error (ECE) is a scalar summary statistic between zero
(perfect calibration) and one (fully miscalibrated).
It is widely reported in the literature as a standard summary statistic
for classification.

The ECE metric works by binning the probability of the _decision label_ into a
fixed number of `num_bins` bins.  Typically `num_bins` is chosen to be 5 or 10.
For example, a binning into 5 bins would yield a partition,

$$\{(-\infty,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], (0.8,\infty)\},$$

and all counting and frequency estimation of probabilities is done using this
binning.
For $$M$$ being `num_bins`, and a total of $$n$$ samples, the ECE is defined as

$$\textrm{ECE} := \sum_{m=1}^{M}
\frac{|B_m|}{n} |\textrm{acc}(B_m) - \textrm{conf}(B_m)|,$$

where $$B_m$$ is the set of instances with predicted probability
$$\hat{p}_i(\hat{y}_i)$$ of the decision label $$\hat{y}_i$$ assigned to
bin $$m$$, and with the true label $$y_i$$, using

$$\textrm{acc}(B_m) :=
\frac{1}{|B_m|} \sum_{i \in B_m} \mathbb{1}_{\{\hat{y}_i=y_i\}},$$

and

$$\textrm{conf}(B_m) :=
\frac{1}{|B_m|} \sum_{i \in B_m} \hat{p}_i(\hat{y}_i).$$

To compute the ECE metric you can pass in the decision labels $$\hat{y}_i$$
using the `labels_predicted` keyword argument.
In case you do not use `labels_predicted`,
the argmax label will be automatically inferred from the `logits`.  (Therefore,
in the following code we could remove `labels_predicted`.)

```python
  # labels_true is a tf.int32 Tensor
  logits = model(validation_data)
  labels_predicted = tf.argmax(logits, axis=1)
  ece = predictive_metrics.calibration.expected_calibration_error(
      10, logits=logits, labels_true=labels_true,
      labels_predicted=labels_predicted)
```

Reference:

* _Chuan Guo_, _Geoff Pleiss_, _Yu Sun_, and _Kilian Q. Weinberger_,
  ["On Calibration of Modern Neural Networks"](https://arxiv.org/pdf/1706.04599.pdf),
  _Proceedings of the 34th International Conference on Machine Learning_
  (ICML 2017), [arXiv:1706.04599](https://arxiv.org/abs/1706.04599)

### Bayesian Expected Calibration Error {#bayesian-ece}

The [ECE metric](#ece) is a scalar summary statistic of miscalibration
evaluated on a finite sample of validation data.  Resulting in a single scalar,
the sampling variation due to the limited amount of validation data is hidden,
and this can result in significant over- or under-estimation of the ECE as well
as wrongly concluding significant differences in ECE between multiple models.

To address these issues, a Bayesian estimator of the ECE can be used.  The
resulting estimate is not a single scalar summary but instead a probability
distribution over possible ECE values.

Using the probability distribution of ECE values we can make statistical
conclusions.
For example, we can assess whether the ECE really is significantly different
between two or more models by reasoning about the overlap of the
ECE distributions of each model, using for example a Wilcoxon signed rank test.

The Bayesian ECE can be [used like the normal ECE](#ece), as in the following
code:

```python
  # labels_true is a tf.int32 Tensor
  logits = model(validation_data)
  ece_samples = predictive_metrics.calibration.bayesian_expected_calibration_error(
      10, logits=logits, labels_true=labels_true)

  ece_quantiles = tensorflow_probability.stats.percentile(
    ece_samples, [10,50,90])
```

The above code also includes an example of using the samples to infer
10%/50%/90% quantiles of the distribution of possible ECE values.

[Further details regarding the Bayesian model used to estimate ECE are
available](bayesian-ece.md).

## API: Mutual Information

### Model Uncertainty

Model uncertainty measures the mutual information between the categorical output
$$y$$ and the model parameters $$\theta$$. It is the difference of the total
uncertainty and expected data uncertainty between $$y$$ and $$\theta$$. The
implementation uses the logits to calculate model uncertainty. It returns
knowledge uncertainty as well as total uncertainty and expected data
uncertainty. An example code is given below:

```python
  logits = model(validation_data)
  model_uncert, total_uncert, avg_data_uncert = mutual_information.model_uncertainty(
      logits)
```

## API: Information Criteria

We estimate information criteria using log-likelihood values on training samples.
In particular, for both the WAIC and the ISCV criteria we assume that we have an ensemble of models with equal weights, such that the average predictive distribution over the ensemble is a good approximation to the true Bayesian posterior predictive distribution.

For a set of $$n$$ training instances and $$m$$ ensemble models we will use a
Tensor of shape $$(n,m)$$, such that the `[i,j]` element contains
$$\log p(y_i | x_i, \theta_j)$$, where $$\theta_j$$ represents the `j`'th ensemble element.  For example, the $$\theta_j$$ values can be obtained by MCMC sampling or as samples from a variational posterior distribution over parameters.

If the $$\theta$$ samples closely approximate the true posterior over parameters, then the WAIC and ISCV information criteria have strong theoretical properties in that they consistently predict the generalization loss with low variance.

### Widely/Watanabe Applicable Information Criterion (WAIC)

The negative WAIC criterion estimates log-likelihood on future observables.
It is given as

$$\textrm{nWAIC}_1 :=
\frac{1}{n} \sum_{i=1}^n \left(
  \log \frac{1}{m} \sum_{j=1}^m p(y_i|x_i,\theta_j)
  - \hat{\mathbb{V}}_i
\right),$$

where $$\hat{\mathbb{V}}_i := \frac{1}{m-1} \sum_{j=1}^m \left(
\log p(y_i|x_i,\theta_j) - \frac{1}{m} \sum_{k=1}^m \log p(y_i|x_i,\theta_k)\right)^2$$.

An alternative form of the WAIC, called type-2 WAIC, is computed as

$$\textrm{nWAIC}_2 :=
\frac{1}{n} \sum_{i=1}^n \left(
  \frac{2}{m} \sum_{j=1}^m \log p(y_i|x_i,\theta_j)
  - \log \frac{1}{m} \sum_{j=1}^m p(y_i|x_i,\theta_j)
\right).$$

Both nWAIC criteria have comparable properties, but Watanabe recommends $$\textrm{nWAIC}_1$$.
To estimate the negative WAIC, we use the following code.

```python
  import predictive_metrics.posterior_predictive_criteria as ppc

  # logp has shape (n,m), n instances, m ensemble members
  neg_waic, neg_waic_sem = ppc.negative_waic(logp, waic_type="waic1")
```

You can select the type of nWAIC to estimate using `waic_type="waic1"` or
`waic_type="waic2"`.  The method returns the scalar estimate as well as the
standard error of the mean of the estimate.

### Importance Sampling Cross Validation Criterion (ISCV)

Like the negative WAIC, the ISCV criterion estimates the holdout log-likelihood on future observables using the training data $$(x_i,y_i)_i$$ and a sample
$$(\theta_j)_j$$ from the posterior distribution over parameters.

The ISCV is defined as follows.

$$\textrm{ISCV} := -\frac{1}{n} \sum_{i=1}^n
  \log \frac{1}{m} \sum_{j=1}^m \frac{1}{p(y_i|x_i,\theta_j)}$$

We can estimate the ISCV using the following code:

```python
  import predictive_metrics.posterior_predictive_criteria as ppc

  # logp has shape (n,m), n instances, m ensemble members
  iscv, iscv_sem = ppc.importance_sampling_cross_validation(logp)
```

__References__

* _Alan E. Gelfand_, _Dipak K. Dey_, and _Hong Chang_.
  "Model determination using predictive distributions with implementation via
  sampling-based methods",
  Technical report No. 462, Department of Statistics,
  Stanford university, 1992.
  [(PDF)](http://citeseerx.ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>0.3702&rep=rep1&type=pdf)
* _Aki Vehtari_, _Andrew Gelman_, and _Jonah Gabry_.
  "Practical Bayesian model evaluation using leave-one-out cross-validation and
  WAIC",
  arXiv:1507.04544,
  [(PDF)](https://arxiv.org/pdf/1507.04544.pdf)

repository: kashif/uncertainty-metrics

repo_name: kashif/uncertainty-metrics


text:
<!-- README.md is generated from README.Rmd via `devtools::build_readme()`. Please edit README.Rmd -->

# thesisdown <img src="man/figures/thesisdown_hex.png" align="right" width=200 />

This project was inspired by the
[bookdown](https://github.com/rstudio/bookdown) package and is an
updated version of my Senior Thesis template in the `reedtemplates`
package [here](https://github.com/ismayc/reedtemplates). It was
originally designed to only work with the Reed College LaTeX template,
but has since been adapted to work with many different institutions by
many different individuals. Check out the [**Customizing thesisdown to
your
institution**](https://github.com/ismayc/thesisdown#customizing-thesisdown-to-your-institution)
section below for examples.

Currently, the PDF and gitbook versions are fully-functional. The word
and epub versions are developmental, have no templates behind them, and
are essentially calls to the appropriate functions in bookdown.

If you are new to working with `bookdown`/`rmarkdown`, please read over
the documentation available in the `gitbook` template at
<https://ismayc.github.io/thesisdown_book>.

The current output for the four versions is here:

-   [PDF](https://github.com/ismayc/thesisdown_book/blob/master/thesis.pdf)
    (Generating LaTeX file is available
    [here](https://github.com/ismayc/thesisdown_book/blob/master/thesis.tex)
    with other files in the [book
    directory](https://github.com/ismayc/thesisdown_book/tree/master).)
-   [Word](https://github.com/ismayc/thesisdown_book/blob/master/thesis.docx)
-   [ePub](https://github.com/ismayc/thesisdown_book/blob/master/thesis.epub)
-   [gitbook](https://ismayc.github.io/thesisdown_book)

Under the hood, the Reed College LaTeX template is used to ensure that
documents conform precisely to submission standards. At the same time,
composition and formatting can be done using lightweight
[markdown](https://rmarkdown.rstudio.com/authoring_basics.html) syntax,
and **R** code and its output can be seamlessly included using
[rmarkdown](https://rmarkdown.rstudio.com).

## Customizing thesisdown to your institution

In an ideal world, this package would support a variety of different
LaTeX templates from a wide range of institutions and we’d love to get
it there at some point. Until that time, realize that this was designed
to only work with the Reed College LaTeX template but others have
adapted it to work with their institutions. Here are some that have
customized it to fit their needs. It is recommended you review how they
changed the files by comparing their repositories to this one and then
make tweaks to yours as needed. Feel free to file an issue on this repo
if you have questions/troubles.

Have you created a thesisdown template for your institution and would
like to have it included here? Make a PR [similar to the commit done to
include
`jayhawkdown`](https://github.com/ismayc/thesisdown/commit/760113a076767cf67b6e22339e398bd3f15305c5).
I’ll review it and merge it in. Let’s keep the list going!

| College/University                                          | Repository                                                                                | Based on                                                        |
|:------------------------------------------------------------|:----------------------------------------------------------------------------------------------|:----------------------------------------------------------------|
| American University                                         | [SimonHeuberger/eagledown](https://github.com/SimonHeuberger/eagledown)                       | [benmarwick/huskydown](https://github.com/benmarwick/huskydown) |
| Brock University                                            | [brentthorne/brockdown](https://github.com/brentthorne/brockdown)                             | [zkamvar/beaverdown](https://github.com/zkamvar/beaverdown)     |
| École Doctorale de Mathématiques Hadamard                   | [abichat/hadamardown](https://github.com/abichat/hadamardown)                                 | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Drexel University                                           | [tbradley1013/dragondown](https://github.com/tbradley1013/dragondown)                         | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Duke University                                             | [mine-cetinkaya-rundel/thesisdowndss](https://github.com/mine-cetinkaya-rundel/thesisdowndss) | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Graduate Institute of International and Development Studies | [jhollway/iheiddown](https://github.com/jhollway/iheiddown)                                   | [ulyngs/oxforddown](https://github.com/ulyngs/oxforddown)       |
| Heidelberg University, Faculty of Biosciences               | [nkurzaw/heididown](https://github.com/nkurzaw/heididown)                                     | [phister/huwiwidown](https://github.com/phister/huwiwidown)     |
| Humboldt University of Berlin                               | [phister/huwiwidown](https://github.com/phister/huwiwidown)                                   | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Kansas State University                                     | [emraher/wildcatdown](https://github.com/emraher/wildcatdown)                                 | [benmarwick/huskydown](https://github.com/benmarwick/huskydown) |
| McMaster University                                         | [paezha/macdown](https://github.com/paezha/macdown)                                           | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Massachusetts Institute of Technology                       | [ratatstats/manusdown](https://github.com/ratatstats/manusdown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Monash University | [masiraji/monashthesisdown](https://github.com/masiraji/monashthesisdown) | [ismayc/thesisdown](https://github.com/ismayc/thesisdown) |
| Oregon State University                                     | [zkamvar/beaverdown](https://github.com/zkamvar/beaverdown)                                   | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Oxford University                                           | [davidplans/oxdown](https://github.com/davidplans/oxdown)                                     | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Queen's University | [eugenesit/gaelsdown](https://github.com/eugenesit/gaelsdown)| [ismayc/thesisdown](https://github.com/ismayc/thesisdown) |
| Smith College                                               | [SmithCollege-SDS/pioneerdown](https://github.com/SmithCollege-SDS/pioneerdown)               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Southampton University                                      | [dr-harper/sotonthesis](https://github.com/dr-harper/sotonthesis)                             | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Stanford University                                         | [mhtess/treedown](https://github.com/mhtess/treedown)                                         | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Universidade Federal do Ceará                               | [damarals/ufcdown](https://github.com/damarals/ufcdown)                                       | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Universidade Federal do Rio de Janeiro                      | [COPPE-UFRJ/coppedown](https://github.com/COPPE-UFRJ/coppedown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Université Paris-Saclay                                     | [abichat/hadamardown](https://github.com/abichat/hadamardown)                                 | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University College London                                   | [benyohaiphysics/thesisdownUCL](https://github.com/benyohaiphysics/thesisdownUCL)             | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Arizona                                       | [kelseygonzalez/beardown](https://github.com/kelseygonzalez/beardown)                         | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Auckland                                      | [d-scanzi/UOAdown](https://github.com/d-scanzi/UOAdown)                                       | [ismayc/thesisdown](https://github.com/ismayc/thesisdown) |
| University of California, Davis                             | [ryanpeek/aggiedown](https://github.com/ryanpeek/aggiedown)                                   | [DanOvando/gauchodown](https://github.com/DanOvando/gauchodown) |
| University of California, Santa Barbara                     | [DanOvando/gauchodown](https://github.com/DanOvando/gauchodown)                               | [benmarwick/huskydown](https://github.com/benmarwick/huskydown) |
| University of Florida                                       | [ksauby/thesisdownufl](https://github.com/ksauby/thesisdownufl)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Freiburg                                      | [vivekbhr/doctorRbite](https://github.com/vivekbhr/doctorRbite)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Kansas                                        | [wjakethompson/jayhawkdown](https://github.com/wjakethompson/jayhawkdown)                     | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Manchester                                    | [juliov/uomthesisdown](https://github.com/JulioV/uomthesisdown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Maryland, College Park                          | [ImNotaGit/thesisdown](https://github.com/ImNotaGit/thesisdown/tree/umd)                    | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
|University of Minnesota |[zief0002/gopherdown](https://github.com/zief0002/gopherdown) |[wjakethompson/jayhawkdown](https://github.com/wjakethompson/jayhawkdown) |
| University of New South Wales                               | [rensa/unswthesisdown](https://github.com/rensa/unswthesisdown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Salzburg                                      | [irmingard/salzburgthesisdown](https://github.com/irmingard/salzburgthesisdown)               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Toronto                                       | [mattwarkentin/torontodown](https://github.com/mattwarkentin/torontodown)                     | [zkamvar/beaverdown](https://github.com/zkamvar/beaverdown)     |
| University of Washington                                    | [benmarwick/huskydown](https://github.com/benmarwick/huskydown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| TU Wien                                                     | [ben-schwen/robotdown](https://github.com/ben-schwen/robotdown)                               | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| University of Bristol                                       | [mattlee821/bristolthesis](https://github.com/mattlee821/bristolthesis)                       | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Universidade Federal de Santa Catarina                      | [lfpdroubi/ufscdown](https://github.com/lfpdroubi/ufscdown)                                   | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |
| Universiteit van Amsterdam                                  | [lcreteig/amsterdown](https://github.com/lcreteig/amsterdown)                                 | [benmarwick/huskydown](https://github.com/benmarwick/huskydown) |
| Youngstown State University                                 | [gjkerns/penguindown](https://github.com/gjkerns/penguindown)                                 | [ismayc/thesisdown](https://github.com/ismayc/thesisdown)       |

### Using thesisdown from Chester’s GitHub

Special thanks to [Ben Marwick](https://github.com/benmarwick) for
helping to add a lot more clarity to the directions below from the
[README of his spin-off `huskydown`
package](https://github.com/benmarwick/huskydown/blob/master/README.md).

Using {thesisdown} has some prerequisites which are described below. To
compile PDF documents using **R**, you are going to need to have LaTeX
installed. By far the easiest way to install LaTeX on any platform is
with the [tinytex](https://yihui.name/tinytex/) R package:

``` r
install.packages(c('tinytex', 'rmarkdown'))
tinytex::install_tinytex()
# after restarting RStudio, confirm that you have LaTeX with
tinytex:::is_tinytex()
```

You may need to install a few extra LaTeX packages on your first attempt
to knit as well. Here is one such example of how to do so:

``` r
tinytex::tlmgr_install("babel-portuges")
```

To use {thesisdown} from
[RStudio](https://www.rstudio.com/products/rstudio/download/):

1.  Ensure that you have already installed LaTeX and the fonts described
    above, and are using the latest version of
    [RStudio](https://www.rstudio.com/products/rstudio/download/). You
    can use `thesisdown` without RStudio. For example, you can write the
    Rmd files in your favorite text editor
    (e.g. [Atom](https://atom.io/),
    [Notepad++](https://notepad-plus-plus.org/)). But RStudio is
    probably the easiest tool for writing both R code and text in your
    thesis. It also provides a nice way to build your thesis while
    editing. We’ll proceed assuming that you have decided to use the
    RStudio workflow.

2.  Install the {bookdown} and {thesisdown} packages. Note that
    {thesisdown} is not available on CRAN at the moment and that’s why
    `install.packages("thesisdown")` won’t work. Use
    `remotes::install_github()` as shown below instead to install the
    package.

    ``` r
    if (!require("remotes"))
      install.packages("remotes", repos = "https://cran.rstudio.org")
    remotes::install_github("rstudio/bookdown")
    remotes::install_github("ismayc/thesisdown")
    ```

          Note that you may need to restart RStudio at this point for
the following dialog to show up.

3.  Get started with the {thesisdown} template. There are two options
    for doing so.

-   3a) **RECOMMENDED** Create a new RStudio project with a {thesisdown}
    template.

    In RStudio, click on **File** > **New Project** > **New Directory**.
    Then select **Thesis Project using thesisdown** from the dropdown
    that will look something like the image below. You’ll see the
    graduation cap as the icon on the left for the appropriate project
    type.

    ![](https://raw.githubusercontent.com/ismayc/thesisdown/master/docs/reference/figures/thesis_proj.png)

    Next, give your project a name and specify where you’d like the
    files to appear. In the screenshot below, the project name is
    `my_thesis` and it will appear as a new folder on my Desktop.

    ![](https://raw.githubusercontent.com/ismayc/thesisdown/master/docs/reference/figures/thesis_proj_name.png)

    If you got this far, skip over step 3b which is the older version of
    getting the template. It might force you to change some of the
    directories to get knitting to work and has some other limitations
    as well. That’s why step 3a is recommended.

-   3b) Use the **New R Markdown** dialog to select **Thesis**:

    ![](https://raw.githubusercontent.com/ismayc/thesisdown/master/docs/reference/figures/thesis_rmd.png)

    Note that this will currently only **Knit** if you name the
    directory `index` as shown above. This guarantees that `index.html`
    is generated correctly for the Gitbook version of the thesis.

4.  After choosing which type of output you’d like in the YAML at the
    top of `index.Rmd`, **Knit** the `index.Rmd` file to get the book in
    PDF or HTML formats.

### Day-to-day writing of your thesis

You need to edit the individual chapter R Markdown files to write your
thesis. It’s recommended that you version control your thesis using
GitHub if possible. RStudio can also easily sync up with GitHub to make
the process easier. While writing, you should `git commit` your work
frequently, after every major activity on your thesis. For example,
every few paragraphs or section of text, and after major step of
analysis development. You should `git push` at the end of each work
session before you leave your computer or change tasks. For a gentle,
novice-friendly guide to getting starting with using Git with R and
RStudio, see <https://happygitwithr.com/>.

## Rendering

To render your thesis into a PDF, open `index.Rmd` in RStudio and then
click the “knit” button. To change the output formats between PDF,
gitbook and Word, look at the `output:` field in `index.Rmd` and
comment-out the formats you don’t want.

The PDF file of your thesis will be deposited in the `_book/` directory,
by default.

## Components

The following components are ones you should edit to customize your
thesis:

### `_bookdown.yml`

This is the main configuration file for your thesis. You can change the
name of your outputted file here for your thesis and other options about
your thesis here.

### `index.Rmd`

This file contains all the meta information that goes at the beginning
of your document. You’ll need to edit the top portion of this file (the
YAML) to put your name on the first page, the title of your thesis, etc.
Note that you need to have at least one chapter start in the `index.Rmd`
file for the build to work. For the template, this is done with
`# Introduction` in the example from the template.

### `01-chap1.Rmd`, `02-chap2.Rmd`, etc.

These are the Rmd files for each chapter in your dissertation. Write
your thesis in these. If you’re writing in RStudio, you may find the
[wordcount addin](https://github.com/benmarwick/wordcountaddin) useful
for getting word counts and readability statistics in R Markdown
documents.

### `bib/`

Store your bibliography (as bibtex files) here. We recommend using the
[citr addin](https://github.com/crsh/citr) and
[Zotero](https://www.zotero.org/) to efficiently manage and insert
citations.

### `csl/`

Specific style files for bibliographies should be stored here. A good
source for citation styles is
<https://github.com/citation-style-language/styles#readme>.

### `figure/` and `data/`

Store your figures and data here and reference them in your R Markdown
files. See the [bookdown book](https://bookdown.org/yihui/bookdown/) for
details on cross-referencing items using R Markdown.

repository: ismayc/thesisdown

gha_language: TeX

gha_license_id: NOASSERTION

repo_name: ismayc/thesisdown


text:
# General/About

## What is the Internet Union?
The Internet Union is a non-profit organization to give the people of the Internet a voice against governments worldwide.
As the Internet continues to grow, namely the World Wide Web, governments are introducing unfair restrictions upon it, often against the views of the users and general populace.

## Why is it needed?
Many different websites have user-created content that promotes the same message we do. However, this is not a universal way, and will only be seen by a small number of people.
The Internet Union allows for an easy way to join together to protect the Internet we all use.

## Should we trust you?
We're not going to say that you should trust us, that's up to you to decide. We will do our best to be transparent and democratic in many processes.

repository: TheInternetUnion/Information

repo_name: TheInternetUnion/Information


text:
# Useful packages

## Day.js

::: tip
Official documentation: [https://day.js.org/](https://day.js.org/en)
:::

Day.js is a powerful package that parses, validates, manipulates, and displays dates and times in JavaScript.
It allows you to quickly and easily format dates in any way you want or parse strings back into JavaScript Date objects.
There are many plugins for it that allow you to work with durations and more.

For example if you wanted to ask your users to give you a date,
you can use Day.js to turn it into a Date object you can use in your code:

<!-- eslint-skip -->
```js
const input = await interaction.channel.awaitMessages({
	filter: m => m.author.id === interaction.user.id,
	max: 1,
	time: 10e3,
	errors: ['time'],
});
const date = dayjs(input.first().content).toDate();
```

Using the [duration plugin](https://day.js.org/docs/en/durations/durations), you could tell the user if the date is in the future or the past:

```js
if (date.isValid()) {
	const now = dayjs();
	const duration = date - now;
	const formatted = dayjs.duration(duration, 'ms').format();

	if (duration > 0) {
		interaction.reply(`The date you gave me is ${formatted} into the future.`);
	} else {
		interaction.reply(`The date you gave me is ${formatted} into the past.`);
	}
} else {
	interaction.reply('You didn\'t give me a valid date.');
}
```

## ms

::: tip
Official documentation: [https://github.com/vercel/ms](https://github.com/vercel/ms)
:::

Ms is another tool for working with times in JavaScript. However, ms specializes on durations.
It allows you to convert times in milliseconds into human-readable formats and vice versa.

Example:

<!-- eslint-skip -->
```js
await interaction.reply('Send two messages and I\'ll tell you how far apart you sent them.');
const messages = await interaction.channel.awaitMessages({
	filter: m => m.author.id === interaction.user.id,
	max: 2,
	time: 30e3,
	errors: ['time'],
});

const difference = messages.last().createdTimestamp - messages.first().createdTimestamp;
const formatted = ms(difference);

await interaction.followUp(`You sent the two messages ${formatted} apart.`);
```

## common-tags

::: tip
Official documentation: [https://github.com/zspecza/common-tags](https://github.com/zspecza/common-tags)
:::

Common-tags is a library all about working with template literals.
So far, you have probably only used them for interpolating variables into your strings, but they can do a whole lot more.
If you got time, you should check out [the MDN's documentation about *tagged literals*.](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#Tagged_templates).

Ever got annoyed your multi-line strings had nasty bits of indentation in them,
but you did not want to remove the indentation in your source code?
common-tags got you covered:

```js
const packageName = 'common-tags';

if (someCondition) {
	const poem = stripIndents`
		I like ${packageName}.
		It makes my strings so pretty,
		you should use it too.
	`;

	console.log(poem);
}
```

This will print your little poem like expected, but it will not have any tabs or other whitespace on the left.

But this is just the start! Another set of useful functions are the list-related functions:
`inlineLists`, `commaLists`, etc.
With those, you can easily interpolate arrays into your strings without them looking ugly:

```js
const options = ['add', 'delete', 'edit'];

// -> Do you want me to add, delete or edit the channel?
interaction.reply(oneLineCommaListsOr`
	Do you want me to ${options} the channel?
`);
```

Check the the documentation to find more useful functions.

## chalk

::: tip
Official documentation: [https://www.npmjs.com/package/chalk](https://www.npmjs.com/package/chalk)
:::

Chalk is not exactly useful for Discord bots themselves, but it will make your terminal output a lot prettier and organized.
This package lets you color and style your `console.log`s in many different ways; No more simple white on black.

Let's say you want your error messages to be easily visible; Let us give them a nice red color:

```js
console.error(chalk.redBright('FATAL ERROR'), 'Something really bad happened!');
```

![image of code above](./images/chalk-red.png)

You can also chain multiple different multipliers.
If you wanted to have green text, a grey background, and have it all underlined, that is possible:

```js
console.log(chalk.green.bgBrightBlack.underline('This is so pretty.'));
```

![image of code above](./images/chalk-ugly.png)

## pino

::: tip
Official documentation: [getpino.io](https://getpino.io)
:::

Pino is a Node.js logger with a very low overhead. But why does that even matter, if `console.log()` exists?  Whenever you make a call to `console.log()` your program halts and cannot do anything until the logging is finished.

To get started, install the package:

:::: code-group
::: code-group-item npm
```sh:no-line-numbers
npm install pino@next
npm install -g pino-pretty
```
:::
::: code-group-item yarn
```sh:no-line-numbers
yarn add pino@next
yarn global add pino-pretty
```
:::
::: code-group-item pnpm
```sh:no-line-numbers
pnpm add pino@next
pnpm add --global pino-pretty
```
:::
::::

Pino is highly configurable, so we heavily recommend you take a look at their documentation yourself.

To use the same logger across the project you can put the following code into it's own file, for example `logger.js` and import it when needed:

```js
const pino = require('pino');
const logger = pino();
module.exports = logger;
```

```js
const { Client, Events, GatewayIntentBits } = require('discord.js');
const client = new Client({ intents: [GatewayIntentBits.Guilds] });
const logger = require('./logger');

client.on(Events.ClientReady, () => logger.info('The bot is online'));
client.on(Events.Debug, m => logger.debug(m));
client.on(Events.Warn, m => logger.warn(m));
client.on(Events.Error, m => logger.error(m));

client.login('your-token-goes-here');
```

Pino logs in a json format, so other programs and services like log aggregators can easily parse and work with the output. This is very useful for production systems, but quite tedious to read during development. This is why you installed `pino-pretty` earlier. Instead of formatting the log output itself the developers recommended that you [pipe](https://en.wikipedia.org/wiki/Pipeline_(Unix)) the log output to other services instead. `pino-pretty` is a formatter you can use for easy-to-read logs during development.

We recommend you set `pino-pretty` up in a package script in your `package.json` file, rather than typing the pipeline out every time. Please read our [guide section on package scripts](/improving-dev-environment/package-json-scripts), if you are not sure what we're talking about here.

```json {10}
{
	"name": "my-bot",
	"version": "1.0.0",
	"description": "A Discord bot!",
	"main": "index.js",
	"scripts": {
		"test": "echo \"Error: no test specified\" && exit 1",
		"start": "node .",
		"lint": "eslint .",
		"dev": "node . | pino-pretty -i pid,hostname -t 'yyyy-mm-dd HH:MM:ss'"
	},
	"keywords": [],
	"author": "",
	"license": "ISC"
}
```

:::warning
If you are using powershell, you have to use a package script for `pino-pretty`. Powershell handles pipelines in a way that prevents logging. The cmd commandline interface is not affected.
:::

In the example above, further arguments are passed to `pino-pretty` to modify the generated output. `-i pid,hostname` hides these two elements from logged lines and `-t yyyy-mm-dd HH:MM:ss` formats the timestamp into an easy to use format. Try out what works for you! The official [pino-pretty documentation](https://github.com/pinojs/pino-pretty) explains all possible arguments.

To start your bot with prettified input you run the `dev` script via your package manager of choice:

:::: code-group
::: code-group-item npm
```sh:no-line-numbers
npm run dev
```
:::
::: code-group-item yarn
```sh:no-line-numbers
yarn run dev
```
:::
::: code-group-item pnpm
```sh:no-line-numbers
pnpm run dev
```
:::
::::

Pino is very flexible, supports custom log levels, worker threads and many more features. Please check out the [official documentation](https://getpino.io) if you want to up your pino game! Below we show an alternative for a production setup. Using this code, you will be logging the raw json objects into a file, instead of printing to your console:

```js {2-6}
const pino = require('pino');
const transport = pino.transport({
	target: 'pino/file',
	options: { destination: './log.json' },
});
const logger = pino(transport);
module.exports = logger;
```

## i18next

::: tip
Official documentation: [https://www.i18next.com](https://www.i18next.com)
:::

i18next is an internationalization-framework for JavaScript. It is beneficial to translate your bot's user-facing messages into various languages based on the server it is used in.

Covering an entire use case example for internationalization would be out of this guide's scope and requires some more explanation as to how the system operates. Please refer to the official documentation linked above for an in-depth usage guide.

repository: discordjs/guide

branch_name: refs/heads/main

gha_language: JavaScript

repo_name: discordjs/guide