text:
# The Things Stack for LoRaWAN Development

The Things Stack components are primarily built in Go, we use React for web front-ends. It is assumed that you have decent knowledge and experience with these technologies. If you want to get more familiar with Go, we strongly recommend to take [A Tour of Go](https://tour.golang.org/).

## Table of contents

- [Development Environment](#development-environment)
- [Cloning the Repository](#cloning-the-repository)
- [Getting Started](#getting-started)
- [Running a development build of The Things Stack](#running-a-development-build-of-the-things-stack)
  - [Pre-requisites](#pre-requisites)
  - [Steps](#steps)
- [Using the CLI with the Development Environment](#using-the-cli-with-the-development-environment)
- [Managing the Development Databases](#managing-the-development-databases)
  - [PostgreSQL](#PostgreSQL)
  - [Redis](#redis)
- [Building the Frontend](#building-the-frontend)
- [Starting The Things Stack](#starting-the-things-stack)
- [Project Structure](#project-structure)
  - [API](#api)
  - [Documentation](#documentation)
  - [Web UI](#web-ui)
- [Code Style](#code-style)
  - [Code Formatting](#code-formatting)
  - [Line Length](#line-length)
  - [Formatting and Linting](#formatting-and-linting)
  - [Documentation Site](#documentation-site)
- [Naming Guidelines](#naming-guidelines)
  - [API Method Naming](#api-method-naming)
  - [Variable Naming](#variable-naming)
  - [Event Naming](#event-naming)
  - [Error Naming](#error-naming)
  - [Log Field Keys, Event Names, Error Names, Error Attributes and Task Identifiers](#log-field-keys-event-names-error-names-error-attributes-and-task-identifiers)
  - [Comments](#comments)
- [JavaScript Code Style](#javascript-code-style)
  - [Code Formatting](#code-formatting)
  - [Code Comments](#code-comments)
  - [Import Statement Order](#import-statement-order)
  - [React Component Syntax (Functional, Class Components and Hooks)](#react-component-syntax-functional-class-components-and-hooks)
  - [React Component Types](#react-component-types)
  - [Frontend Related Pull Requests](#frontend-related-pull-requests)
- [Translations](#translations)
  - [Backend Translations](#backend-translations)
  - [Frontend Translations](#frontend-translations)
- [Events](#events)
- [Testing](#testing)
  - [Unit Tests](#unit-tests)
  - [End-to-end Tests](#end-to-end-tests)
- [Building and Running](#building-and-running)
- [Releasing](#releasing)
  - [Release From Master](#release-from-master)
  - [Release Backports](#release-backports)
- [Troubleshooting](#troubleshooting)
  - [Console](#console)

## Development Environment

The Things Network's development tooling uses [Mage](https://magefile.org/). Under the hood, `mage` calls other tools such as `git`, `go`, `yarn`, `docker` etc. Recent versions are supported; Node v18.x and Go v1.18.x.

- Follow [Go's installation guide](https://golang.org/doc/install) to install Go.
- Download Node.js [from their website](https://nodejs.org) and install it.
- Follow [Yarn's installation guide](https://yarnpkg.com/en/docs/install) to install Yarn.
- Follow the guides to [install Docker](https://docs.docker.com/install/#supported-platforms) and to [install Docker Compose](https://docs.docker.com/compose/install/#install-compose).

## Cloning the Repository

If you are unfamiliar with forking projects on GitHub or cloning them locally, please [see the GitHub documentation](https://help.github.com/articles/fork-a-repo/).

## Getting Started

The first step to get started is to initialize tooling and some dependencies:

```bash
$ make init
```

You may want to run this commands from time to time to stay up-to-date with changes to tooling and dependencies.

## Running a development build of The Things Stack

This section explains how to get a bare-bones version of The Things Stack running on your local machine. This will build whatever code is present in your local repository (along with local changes) and run in it using the default ports.

If you want to just run a docker image of The Things Stack, then check the [Installation](https://thethingsstack.io/getting-started/installation/) section of the documentation.

### Pre-requisites

1. This section requires that the required tools from [Development Environment](##Development-Environment) are installed.
2. This repository must be cloned inside the `GOPATH`. Check the [official documentation](https://golang.org/doc/gopath_code.html) on working with `GOPATH`.
3. Make sure that you've run `$ make init` before continuing.
4. If this is not the first time running the stack, make sure to clear any environment variables that you've been using earlier. You can do check what variables are set currently by using

```
$ printenv | grep "TTN_LW_*"
```

### Steps

1. Build the frontend assets

```bash
$ tools/bin/mage js:build
```

This will build the frontend assets and place it in the `public` folder.

2. Start the databases

```bash
$ tools/bin/mage dev:dbStart # This requires Docker to be running.
```

This will start one instance each of `postgres` and `Redis` as Docker containers. To verify this, you can run

```bash
$ docker ps
```

3. Initialize the database with defaults.

```bash
$ tools/bin/mage dev:initStack
```

This creates a database, migrates tables and creates a user `admin` with password `admin`.
- An API Key for the admin user with `RIGHTS_ALL` is also created and stored in `.env/admin_api_key.txt`.

4. Start a development instance of The Things Stack

```bash
$ go run ./cmd/ttn-lw-stack -c ./config/stack/ttn-lw-stack.yml start
```

5. Login to The Things Stack via the Console

In a web browser, navigate to `http://localhost:1885/` and login using credentials from step 3.

6. Customizing configuration

To customize the configuration, copy the configuration file `/config/stack/ttn-lw-stack.yml` to a different location (ex: the `.env` folder in your repo). The configuration is documented in the [Configuration Reference](https://thethingsstack.io/reference/configuration/).

You can now use the modified configuration with

```bash
$ go run ./cmd/ttn-lw-stack -c <custom-location>/ttn-lw-stack.yml start
```

## Using the CLI with the Development Environment

In order to login, you will need to use the correct OAuth Server Address. `make init` uses CFSSL to generate a `ca.pem` CA certificate to support https:

```bash
$ export TTN_LW_CA=./ca.pem
$ export TTN_LW_OAUTH_SERVER_ADDRESS=https://localhost:8885/oauth
$ go run ./cmd/ttn-lw-cli login
```

## Managing the Development Databases

You can use the following commands to start, stop and erase databases.

```bash
$ tools/bin/mage dev:dbStart # Starts all databases in a Docker container.
$ tools/bin/mage dev:dbStop  # Stops all databases.

# The contents of the databases will be saved in .env/data

$ tools/bin/mage dev:dbErase # Stops all databases and erase storage.
```

### PostgreSQL

PostgreSQL is a SQL database that we use in the Identity Server.

You can use `tools/bin/mage dev:dbSQL` to enter an SQL shell.

### Redis

Redis is an in-memory data store that we use as a database for "hot" data.

You can use `tools/bin/mage dev:dbRedisCli` to enter a Redis-CLI shell.

## Building the Frontend

You can use `tools/bin/mage js:build` to build the frontend.

## Starting The Things Stack

You can use `go run ./cmd/ttn-lw-stack start` to start The Things Stack.

#### Codec

Most data is stored as base64-encoded protocol buffers. For debugging purposes it is often useful to inspect or update the stored database models - you can use Redis codec tool located at `./pkg/redis/codec` to decode/encode them to/from JSON.

##### Examples

###### Get and Decode

```bash
$ redis-cli get "ttn:v3:ns:devices:uid:test-app:test-dev" | go run ./pkg/redis/codec -type 'ttnpb.EndDevice'
```

###### Get, Decode, Modify, Encode and Set

```
$ redis-cli get "ttn:v3:ns:devices:uid:test-app.test-dev" \
  | go run ./pkg/redis/codec -type 'ttnpb.EndDevice' \
  | jq '.supports_join = false' \
  | go run ./pkg/redis/codec -type 'ttnpb.EndDevice' -encode \
  | redis-cli -x set "ttn:v3:ns:devices:uid:test-app.test-dev"
```

## Project Structure

The folder structure of the project looks as follows:

```
.
├── .editorconfig       configuration for your editor, see editorconfig.org
├── CODEOWNERS          maintainers of folders who are required to approve pull requests
├── CONTRIBUTING.md     guidelines for contributing: branching, commits, code style, etc.
├── DEVELOPMENT.md      guide for setting up your development environment
├── docker-compose.yml  deployment file (including databases) for Docker Compose
├── Dockerfile          formula for building Docker images
├── LICENSE             the license that explains what you're allowed to do with this code
├── Makefile            dev/test/build tooling
├── tools               dev/test/build tooling
├── README.md           general information about this project
│   ...
├── api                 contains the protocol buffer definitions for our API
├── cmd                 contains the different binaries that form The Things Stack for LoRaWAN
│   ├── internal        contains internal files shared between the different binaries
│   │   ...
│   ├── ttn-lw-cli      the command-line-interface for The Things Stack for LoRaWAN
│   └── ttn-lw-stack    bundles the server binaries that form The Things Stack for LoRaWAN
├── config              configuration for our JavaScript SDK and frontend
├── data                data from external repositories, such as devices, frequency plans and webhook templates
├── doc                 detailed documentation on the workings of The Things Stack for LoRaWAN
├── pkg                 contains all libraries used in The Things Stack for LoRaWAN
│   ├── component       contains the base component; all other components extend this component
│   ├── config          package for configuration using config files, environment and CLI flags
│   ├── console         package that provides the web server for the console
│   ├── errors          package for rich errors that include metadata and cross API boundaries
│   ├── log             package for logging
│   ├── messages        contains non-proto messages (such as the messages that are sent over MQTT)
│   ├── metrics         package for metrics collection
│   ├── ttnpb           contains generated code from our protocol buffer definitions and some helper functions
│   ├── types           contains primitive types
│   ├── webui           contains js code for the console and oauth provider
│   └── ...
├── public              frontend code will be compiled to this folder - not added to git
├── release             binaries will be compiled to this folder - not added to git
└── sdk                 source code for our SDKs
    └── js              source code for our JavaScript SDK
```

### API

Our APIs are defined in `.proto` files in the `api` folder. These files describe the messages and interfaces of the different components of The Things Stack. If this is the first time you hear the term "protocol buffers" you should probably read the [protocol buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3) before you continue.

From the `.proto` files, we generate code using the `protoc` compiler. As we plan to compile to a number of different languages, we decided to put the compiler and its dependencies in a [Docker image](https://github.com/TheThingsIndustries/docker-protobuf). The actual commands for compilation are handled by our tooling, so the only thing you have to execute when updating the API, is:

```bash
$ tools/bin/mage proto:clean proto:all jsSDK:definitions
```

### Documentation

The documentation site for The Things Stack is built from the [`lorawan-stack-docs`](https://github.com/TheThingsIndustries/lorawan-stack-docs) repository.

### Web UI

The Things Stack for LoRaWAN includes two frontend applications: the **Console** and **Account App**. Both applications use [React](https://reactjs.org/) as frontend framework. The `console` and `account` packages of the backend expose their respective web servers and handle all logic that cannot be done in the browser. Otherwise both applications are single page applications (SPA) that run entirely in the browser.

The folder structure of the frontend looks as follows:

```
./pkg/webui
├── assets            assets (eg. vectors, images) used by the frontend
├── components        react components shared throughout the frontend
├── console           root of the console application
│   ├── api           api definitions to communicate with The Things Stack
│   ├── containers    container components
│   ├── lib           utility classes and functions
│   ├── store         redux actions, reducers and logic middlewares
│   ├── views         whole view components of the console (~pages)
├── containers        global react container components
├── lib               global utility classes and functions
├── locales           frontend and backend locale jsons used for i18n
├── oauth             root of the oauth application
│   ├── api           api definitions to communicate with The Things Stack
│   ├── store         redux actions, reducers and logic middlewares
│   ├── views         whole view components of the oauth provider (~pages)
├── styles            global stylus (~css) styles and mixins
├── console.js        entry point of the console app
├── oauth.js          entry point of the oauth app
├── template.go       go template module used to render the frontend HTML
```

For development purposes, the frontend can be run using `webpack-dev-server`. After following the [Getting Started](#getting-started) section to initialize The Things Stack and doing an initial build of the frontend via `tools/bin/mage js:build`, it can be served using:

```bash
$ export NODE_ENV=development
$ tools/bin/mage js:serve
```

The development server runs on `http://localhost:8080` and will proxy all api calls to port `1885`. The serve command watches any changes inside `pkg/webui` and refreshes automatically.

#### Development Configuration

In order to set up The Things Stack to support running the frontend via `webpack-dev-server`, the following environment setup is needed:

```bash
# .dev.env
export NODE_ENV="development"
export TTN_LW_LOG_LEVEL="debug"
export TTN_LW_CONSOLE_UI_CANONICAL_URL="http://localhost:8080/console"
export TTN_LW_CONSOLE_OAUTH_AUTHORIZE_URL="http://localhost:8080/oauth/authorize"
export TTN_LW_CONSOLE_OAUTH_LOGOUT_URL="http://localhost:8080/oauth/logout"
export TTN_LW_CONSOLE_OAUTH_TOKEN_URL="http://localhost:8080/oauth/token"
export TTN_LW_IS_OAUTH_UI_CANONICAL_URL="http://localhost:8080/oauth"
export TTN_LW_IS_EMAIL_NETWORK_IDENTITY_SERVER_URL="http://localhost:8080/oauth"
export TTN_LW_IS_EMAIL_PROVIDER="dir"
export TTN_LW_IS_EMAIL_DIR=".dev/email"
export TTN_LW_CONSOLE_UI_ASSETS_BASE_URL="http://localhost:8080/assets"
export TTN_LW_IS_OAUTH_UI_CONSOLE_URL="http://localhost:8080/console"
export TTN_LW_CONSOLE_UI_ACCOUNT_URL="http://localhost:8080/oauth"
```

We recommend saving this configuration as an `.dev.env` file and sourcing it like `source .dev.env`. This allows you to easily apply development configuration when needed.

> Note: It is important to **source these environment variables in all terminal sessions** that run The Things Stack or the `tools/bin/mage` commands. Failing to do so will result in erros such as blank page renders. See also [troubleshooting](#troubleshooting).

#### Optional Configuration

##### Disable [Hot Module Replacement](https://webpack.js.org/concepts/hot-module-replacement/)

If you experience trouble seeing the WebUIs updated after a code change, you can also disable hot module replacement and enforce a hard reload on code changes instead. This method is a bit slower but more robust. To do so apply the following variable:

```bash
WEBPACK_DEV_SERVER_DISABLE_HMR="true"
```

> Note: Webpack-related configuration can be loaded from environment variables only. It cannot be sourced from a config file.

##### Enable TLS in `webpack-dev-server`

```bash
WEBPACK_DEV_SERVER_USE_TLS="true"
```
This option uses the key and certificate set via `TTN_LW_TLS_KEY` and `TTN_LW_TLS_CERTIFICATE` environment variables. Useful when developing functionalities that rely on TLS.

> Note: To use this option, The Things Stack for LoRaWAN must be properly setup for TLS. You can obtain more information about this in the **Getting Started** section of the The Things Stack for LoRaWAN documentation.

## Code Style

### Code Formatting

We want our code to be consistent across our projects, so we'll have to agree on a number of formatting rules. These rules should usually usually be applied by your editor. Make sure to install the [editorconfig](https://editorconfig.org) plugin for your editor.

Our editorconfig contains the following rules:

- We use the **utf-8** character set.
- We use **LF** line endings.
- We have a **final newline** in each file.
- We we **trim whitespace** at the end of each line (except in Markdown).
- All `.go` files are indented using **tabs**
- The `Makefile` and all `.make` files are indented using **tabs**
- All other files are indented using **two spaces**

### Line Length

We don't have strict rules for line length, but in our experience the following guidelines result in nice and readable code:

- If a line is longer than 80 columns, try to find a "natural" break
- If a line is longer than 120 columns, insert a line break
- In very special cases, longer lines are tolerated

### Formatting and Linting

Go code can be automatically formatted using tools such as [`gofmt`](https://godoc.org/github.com/golang/go/src/cmd/gofmt) and [`goimports`](https://godoc.org/golang.org/x/tools/cmd/goimports). The [Go language server](https://github.com/golang/tools/tree/master/gopls) can also help with formatting code. There are many editor plugins that automatically format your code when you save your files. We highly recommend using those.

We use [`golangci-lint`](https://golangci-lint.run/) to lint Go code and [`eslint`](https://eslint.org) to lint JavaScript code. These tools should automatically be installed when initializing your development environment.

### Documentation Site

Please respect the following guidelines for content in our documentation site. A copy and paste template for creating new documentation can be found [here](doc/content/example-template).

- Use the `{{< new-in-version "3.8.5" >}}` shortcode to tag documentation for features added in a particular version. For documentation that targets `v3.n`, that's the next patch bump, e.g `3.8.x`. For documentation targeting `v3.n+1` that's the next minor bump, e.g `3.9.0`.
- The title of a doc page is already rendered by the build system as a h1, don't add an extra one.
- Use title case for headings.
- A documentation page starts with an introduction, and then the first heading. The first paragraph of the introduction is typically a summary of the page. Use a `<!--more-->` to indicate where the summary ends.
- Divide long documents into separate files, each with its own folder and `_index.md`.
- Use the `weight`tag in the [Front Matter](https://gohugo.io/content-management/front-matter/) to manually sort sections if necessary. If not, they will be sorted alphabetically.
- Since the title is a `h1`, everything in the content is at least `h2` (`##`).
- Paragraphs typically consist of at least two sentences.
- Use an empty line between all blocks (headings, paragraphs, lists, ...).
- Prefer text over bullet lists or enumerations. For bullets, use `-`, for enumerations `1.` etc.
- Explicitly call this product "The Things Stack". Not "the stack" etc. You can use the shortcode `{{% tts %}}` which will expand to "The Things Stack".
- Avoid shortening, i.e. write "it is" instead of "it's".
- Write guides as a goal-oriented journey.
- Unless already clear from context, use a clearer term than "user", especially if there are multiple kinds (network operator, gateway owner, application developer, ...).
- The user does not have a gender (so use they/them/their).
- Taking screenshots is done as follows:
  - In Chrome: activate the **Developer Tools** and toggle the **Device Toolbar**. In the **Device Toolbar**, select **Laptop with HiDPI screen** (add it if not already there), and click **Capture Screenshot** in the menu on the right.
  - In Firefox: enter **Responsive Design Mode**. In the **Device Toolbar**, select "Laptop with HiDPI screen" (add it if not already there) and **Take a screenshot of the viewport**.
- Use `**Strong**` when referring to buttons in the Console.
- Use `>Note:`to add a note.
- Use fenced code blocks with a language:
  - `bash` for lists of environment variables: `SOME_ENV="value"`.
  - `bash` for CLI examples. Prefix commands with `$ `. Wrap strings with double quotes `""` (except when working with JSON, which already uses double quotes).
  - Wrap large CLI output with `<details><summary>Show CLI output</summary> ... output here ... </details>`.
  - `yaml` (not `yml`) for YAML. Wrap strings with single quotes `''` (because of frequent Go templates that use double quotes).

## Naming Guidelines

### API Method Naming

All API method names should follow the naming convention of `VerbNoun` in upper camel case, where the verb uses the imperative mood and the noun is the resource type.

The following snippet defines the basic CRUD definitions for a resource named `Type`.
Note also that the order of the methods is defined by CRUD.

```
CreateType
GetType
ListTypes (returns slice)
UpdateType
DeleteType

AddTypeAttribute
SetTypeAttribute
GetTypeAttribute
ListTypeAttributes (returns slice)
RemoveTypeAttribute
```

### Variable Naming

Variable names should be short and concise.

We follow the [official go guidelines](https://github.com/golang/go/wiki/CodeReviewComments#variable-names) and try to be consistent with Go standard library as much as possible, everything not defined in the tables below should follow Go standard library naming scheme. In general, variable names are English and descriptive, omitting abbreviations as much as possible (except for the tables below), as well as putting adjectives and adverbs before the noun and verb respectively.

#### Single-word Entities

| entity               | name    | example type                                                  |
| :------------------: | :-----: | :-----------------------------------------------------------: |
| context              | ctx     | context.Context                                               |
| mutex                | mu      | sync.Mutex                                                    |
| configuration        | conf    | go.thethings.network/lorawan-stack/v3/pkg/config.Config       |
| logger               | logger  | go.thethings.network/lorawan-stack/v3/pkg/log.Logger          |
| message              | msg     | go.thethings.network/lorawan-stack/v3/api/gateway.UplinkMessage  |
| status               | st      | go.thethings.network/lorawan-stack/v3/api/gateway.Status         |
| server               | srv     | go.thethings.network/lorawan-stack/v3/pkg/networkserver.Server|
| ID                   | id      | string                                                        |
| unique ID            | uid     | string                                                        |
| counter              | cnt     | int                                                           |
| gateway              | gtw     |                                                               |
| application          | app     |                                                               |
| end device           | dev     |                                                               |
| user                 | usr / user |                                                            |
| transmit             | tx / Tx |                                                               |
| receive              | rx / Rx |                                                               |

The EUI naming scheme can be found in the well-known variable names section bellow.

#### 2-word Entities

In case both of the words have an implementation-specific meaning, the variable name is the combination of first letter of each word.

| entity                                                  | name    |
| :-----------------------------------------------------: | :-----: |
| wait group                                              | wg      |
| Application Server                                      | as      |
| Gateway Server                                          | gs      |
| Identity Server                                         | is      |
| Join Server                                             | js      |
| Network Server                                          | ns      |

In case one of the words specifies the meaning of the variable in a specific language construct context, the variable name is the combination of abbrevations of the words.

#### Well-known Variable Names

These are the names of variables that occur often in the code. Be consistent in naming them, even when their
meaning is obvious from the context.

| entity                          | name    |
| :-----------------------------: | :-----: |
| gateway ID                      | gtwID   |
| gateway EUI                     | gtwEUI  |
| application ID                  | appID   |
| application EUI                 | appEUI  |
| join EUI                        | joinEUI |
| device ID                       | devID   |
| device EUI                      | devEUI  |
| user ID                         | usrID / userID |

### Event Naming

Events are defined with

```go
events.Define("event_name", "event description")
```

The event name is usually of the form `component.entity.action`. Examples are `ns.up.receive_duplicate` and `is.user.update`. We have some exceptions, such as `ns.up.join.forward`, which is specifically used for join messages.

The event description describes the event in simple English. The description is capitalized by the frontend, so the message should be lowercase, and typically doesn't end with a period. Event descriptions will be translated to different languages.

### Error Naming

Errors are defined with

```go
errors.Define<Type>("error_name", "description with `{attribute}`", "other", "public", "attributes")
```

Error definitions must be defined as close to the return statements as possible; in the same package, and preferably above the concerning function(s). Avoid exporting error definitions unless they are meaningful to other packages, i.e. for testing the exact error definition. Keep in mind that error definitions become part of the API.

Prefer using a specific error type, i.e. `errors.DefineInvalidArgument()`. If you are using a cause (using `WithCause()`), you may use `Define()` to fallback to the cause's type.

The error name in snake case is a short and unique identifier of the error within the package. There is no need to append `_failed` or `_error` or prepend `failed_to_` as an error already indicates something went wrong. Be consistent in wording (i.e. prefer the more descriptive `missing_field` over `no_field`), order (i.e. prefer the more clear `missing_field` over `field_missing`) and avoid entity abbreviations.

The error description in lower case, with only names in title case, is a concise plain English text that is human readable and understandable. Do not end the description with a period. You may use attributes, in snake case, in the description defined between backticks (`` ` ``) by putting the key in curly braces (`{ }`). See below for naming conventions. Only provide primitive types as attribute values using `WithAttributes()`. Error descriptions will be translated to different languages.

### Log Field Keys, Event Names, Error Names, Error Attributes and Task Identifiers

Any `name` defined in the following statements:

- Logging field key: `logger.WithField("name", "value")`
- Event name: `events.Define("name", "description")`
- Error name: `errors.Define("name", "description")`
- Error attribute: ``errors.Define("example", "description `{name}`")``
- Task identifier: `c.RegisterTask("name", ...)`

Shall be snake case, optionally having an event name prepended with a dotted namespace, see above. The spacer `_` shall be used in LoRaWAN terms: `DevAddr` is `dev_addr`, `AppSKey` is `app_s_key`, etc.

### Comments

All comments should be English sentences, starting with a capital letter and ending with a period.

Every Go package should have a package comment. Every top-level Go type, const, var and func should have a comment. These comments are recognized by Go's tooling, and added to the [Godoc for our code](https://godoc.org/go.thethings.network/lorawan-stack). See [Effective Go](https://golang.org/doc/effective_go.html#commentary) for more details.

Although Go code should typically explain itself, it's sometimes important to add additional comments inside funcs to communicate what a block of code does, how a block of code does that, or why it's implemented that way.

Comments can also be used to indicate steps to take in the future (*TODOs*). Such comments look as follows:

```go
// TODO: Open the pod bay doors (https://github.com/TheThingsNetwork/lorawan-stack/issues/<number>).
```

In our API definitions (`.proto` files) we'd like to see short comments on every service, method, message and field. Code that is generated from these files does not have to comply with guidelines (such as Go's guideline for starting the comment with the name of the thing that is commented on).

## JavaScript Code Style

For our frontend development, we use a syntax based on ES6 with a couple of extensions from later standards. The code is transpiled via [`webpack`](https://webpack.js.org/) using [`babel`](https://babeljs.io/) to be interpreted by the browser or Node.JS.

### Code Formatting

We use [`prettier`](https://prettier.io/) and [`eslint`](https://eslint.org/) to conform our code to our guidelines as far as possible. Committed code that violates these rules will cause a CI failure. For your convenience, it is hence recommended to set up your development environment to apply autoformatting on every save. We usually don't enforce any formatting or styles that go beyond of what we can ensure using our linting setup. You can check the respective configuration in `/config/eslintrc.yaml`, `/config/.prettierrc.yaml` as well as the global `.editorconfig`.

To run the linter, you can use `mage js:lint` and to format all JavaScript files, you can run `mage js:fmt`.

### Code Comments

Additionally to the overall code comment rules outlined above, we use [JSDoc](https://jsdoc.app)-conform documentation of classes and functions. We also use full English sentences and ending sentence periods here.

Please make sure that these multi-line comments follow the correct format, especially leaving the first line of this multiline JSDoc comments empty:

```js
// Bad.

/** Converts a byte string from hex to base64.
 * @param {string} bytes - The bytes, represented as hex string.
 * @returns {string} The bytes, represented as base64 string.
 */

// Good.

/**
 * Converts a byte string from hex to base64.
 * @param {string} bytes - The bytes, represented as hex string.
 * @returns {string} The bytes, represented as base64 string.
 */
```

It also makes sense to wrap code bits, variable names and URLs in \`\`  quotes, so they can easily be recognized and do not clash with our capitalization rules enforced by eslint, when they are at the beginning of a sentence:

```js
// Bad. This will get flagged by the linter.

// devAddr is a hex string.
const devAddr = '270000FF'

// Good.

// `devAddr` is a hex string.
const devAddr = '270000FF'
```

### Import Statement Order

Our `import` statements use the following order, each separated by empty newlines:

1. Node "builtin" modules (e.g. `path`)
2. External modules (e.g. `react`)
3. Internal modules (e.g. `@ttn-lw/*`, `@console`, etc.)
  1. Constants
  2. API module
  3. Components
    1. Global presentational components (`@ttn-lw/components/*`)
    2. Global container components (`@ttn-lw/containers/*`)
    3. Global utility components (`@ttn-lw/lib/components/*`)
    4. Local presentational components (`@{console|oauth}/components/*`)
    5. Local container components (`@{console|oauth}/containers/*`)
    6. Local utility components (`@{console|oauth}/lib/components/*`)
    7. View components (`@{console|oauth}/views/*`)
  4. Utilities
    1. Global utilities (`@ttn-lw/lib/*`)
    2. Local utilities (`@{console|oauth}/lib/*`)
  5. Store modules
    1. Actions
    2. Reducers
    3. Selectors
    4. Middleware and logics
  6. Assets and styles
4. Parent modules (e.g. `../../../module`)
5. Sibling modules (e.g. `./validation-schema`, `./button.styl`)
6. Index of the current directory (`.`)

Note that this order is enforced by our linter and will cause a CI fail when not respected. Again, settting up your development environment to integrate linting will assist you greatly here.

### React Component Syntax (Functional, Class Components and Hooks)

Lately, we have been embracing [react hooks](https://reactjs.org/docs/hooks-overview.html) and write all new components using this approach. However, there are a lot of class components from the time before react hooks which we will try to refactor successively.

#### A note on decorators and HOCs

Decorators provided an easy syntax to wrap Classes around functions and we have used this syntax extensively during early stages of development. We now consider decorators and HOCs as hindrance with regards to our aim to adopt hooks. As a result, we refrain from introducing new higher order components and implement hooks instead. This will help us avoiding decorators as well as literal (concatenated) wrappers for function components.

### React Component Types

We differentiate four different component scopes:
- Presentational Components (global and application level)
- Container Components (global and application level)
- View Components
- Utility Components (global and application level)

The differentiation is not always 100% clear and we tend not to be too dogmatic about it. Additionally, the introduction of react hooks tends too break up these traditional categorizations even more and might necessitate a review of these in the near future.

Generally we understand these component types as follows:

#### Presentational Components

These are UI elements that primarily serve a presentational purpose. They implement the basic visual interface elements of the application, focusing on interaction and plain UI logic. They never connect to the store or perform any data fetching or have any other side effects and render rich DOM trees which are also styled according to our design guidelines.

Examples for presentational components are simple UI elements such as buttons, input elements, navigations, breadcrumbs. They can also combine and extend functionality of other presentational components by composition to achieve more complex elements, such as forms. We also regard our application specific forms as such components, as long as they don't connect to the store or perform the data fetching themselves.

To decide whether a component is a presentational component, ask yourself:
- Is this component more concerned with how things look, rather than how things work?
- Does this component use no state or only UI state?
- Does this component not fetch or send data?
- Does this component render a lot of (nested and styled) DOM nodes?

If you answered more than 2 questions with yes, then you likely have a presentational component.

Presentational components should **always** define storybook stories, to provide usage information for other developers.

#### Container Components

Container components focus more on state logic, data fetching, store management and similar concerns. They usually perform business logic and eventually render results using presentational components.

An example for a container components are our table components, that manage the fetching and preparation of the respective entity and render the result using our `<Table />` component.

To decide whether a component is a container component, ask yourself:
- Is this component more concerned with how things work, rather than how things look?
- Does this component connect to the store?
- Does this component fetch or send data?
- Is the component generated by higher order components (e.g. `withFeatureRequirement`)?
- Does this component render simple nodes, like a single presentational component?

If you can answer more than 2 questions with yes, then you likely have a container component.

#### View components

View components always represent a single view of the application, represented by a single route. They structurize the overall appearance of the page, obtain global state information, fetch necessary data and pass it down (implicitly via the store or explicitly as props) mostly to container components, but also to presentational components. Usually, these components also define submit and error handlers of the forms that they render. Otherwise, these components should not employ excessive (stateful) logic which should rather be handled by container components. It should focus on globally structurizing the page using the grid system and respective containers.

##### View component checklist

- Conciseness and no stateful logic (use containers instead)
- Uses `<PageTitle />` component to define heading and page title
- Uses breadcrumbs (if within breadcrumb view)
- Fetching necessary data (via `withRequest` HOC), if not done by a container
- Unavailable "catch-all"-routes are caught by `<NotFoundRoute />` component, including subviews
- Errors should be caught by the `<ErrorView />` error boundary component
- `withFeatureRequirement` HOC is used to prevent access to routes that the user has no rights for
- Ensured responsiveness and usage of the grid system

#### Utility components

These components do not render any DOM elements and are hence not *visible* by themselves. Utility components can be higher order components or similar components, that modify their children or introduce a side effect to the render tree.

To decide whether a component is a utility component ask yourself:
- Is this component a higher order component?
- Is this component invisible on its own?
- Is this component an abstraction layer on top of another component?

If you can answer at least one of those questions with yes, then you likely have a container component.

#### Global or Application Scope?

Components can be categorized as either local (e.g. `pkg/webui/{console|oauth}/{components|containers}`) or global (e.g. `pkg/webui/{components|containers}`). The distinction should come naturally: Global components are ones that can be used universally in every application. Local components are tied to a specific use case inside the respective application.

Sometimes, you might find that during implementing an application specific component that it can actually be generalized without much refactoring and hence be a useful addition to our global component library.

### Frontend Related Pull Requests

Pull requests for frontend related changes generally follow our overall pull request scheme. However, in order to assist reviewers, a browser screenshot of the changes is included in the PR comment, if applicable.

It might help you to employ the following checklist before opening the pull request:

* [ ] All visible text is using [i18n messages](#frontend-translations)?
* [ ] Assets minified or compressed if possible (e.g. SVG assets)?
* [ ] Screenshot in PR description?
* [ ] Responsiveness checked?
* [ ] New components [categorized correctly](#global-or-application-scope) (global/local, container/component)?
* [ ] Feature flags added (if applicable)?
* [ ] All views use  `<PageTitle />` or `<IntlHelmet />` properly?
* [ ] Storybook story added / updated?
* [ ] Prop types / default props added?

## Translations

We do our best to make all text that could be visible to users available for translation. This means that all text of the console's user interface, as well as all text that it may forward from the backend, needs to be defined in such a way that it can be translated into other languages than English.

### Backend Translations

In the API, the enum descriptions, error messages and event descriptions available for translation. Enum descriptions are defined in `pkg/ttnpb/i18n.go`. Error messages and event descriptions are defined with `errors.Define(...)` and `events.Define(...)` respectively.

These messages are then collected in the `config/messages.json` file, which will be processed in the frontend build process, but may also be used by other (native) user interfaces. When you define new enums, errors or events or when you change them, the messages need to be updated into the `config/messages.json` file.

```bash
$ tools/bin/mage go:messages
```

If you forget to do so, this will cause a CI failure.

Adding translations of messages to other languages than English is a matter of adding key/value pairs to `translations` in `config/messages.json`.

### Frontend Translations

The frontend uses [`react-intl`](https://github.com/yahoo/react-intl), which helps us greatly to define text messages used in the frontend.

The workflow for defining messages is as follows:

1. Add a `react-intl` message using `intl.defineMessages({…})`
  * This can be done either inline, or by adding it to `pkg/webui/lib/shared-messages.js`
2. Use the message in components (e.g. `sharedMessage.myMessage`)

After adding messages this way, it needs to be added the locales file `pkg/webui/locales/*.js` by using:

```bash
$ tools/bin/mage js:translations
```

> Note: When using `tools/bin/mage js:serve`, this command will be run automatically after any change.

The message definitions in `pkg/webui/locales` can be used to provide translations in other languages (e.g. `fr.js`). Keep in mind that locale files are checked in and committed, any discrepancy in the locales file with the defined messages will lead to a CI failure.

## Events

In addition to the previously described translation file that we generate, we also generate a data file that contains all event definitions. This file is then loaded by the documentation system so that we can generate documentation for our events.

After adding or changing events, regenerate this file with:

```bash
$ tools/bin/mage go:eventData
```

## Testing

### Unit Tests

To run unit tests, use the following mage targets:

```bash
$ tools/bin/mage go:test js:test jsSDK:test
```

### End-to-end Tests

We use [Cypress](https://cypress.io) for running frontend-based end-to-end tests. The tests specifications are located at `/cypress/integration`.

#### Running frontend end-to-end tests locally

Make sure to [build the frontend assets](#building-the-frontend) and run `tools/bin/mage dev:initStack dev:sqlDump` to create a seed database which the tests can reset the database to in between runs. To run the stack when working on end-to-end tests, use the `tools/bin/mage dev:startDevStack` command. This will run the runs The Things Stack with proper configuration for the end-to-end tests. Note: this command does not output anything, but logs are written to `.cache/devStack.log`.

[Cypress](https://www.cypress.io/) provides two modes for running tests: headless and interactive.
- **Headless mode** will not display any browser GUI and output test progress into your terminal instead. This is helpful when one just needs see the results of the tests.
- **Interactive mode** will run an Electron based application together with the full-fledged browser. This is helpful when developing frontend applications as it provides hot reload, time travelling, browser extensions and DOM access.

> Note: Currently, we test our frontend only in Chromium based browsers.

You can run Cypress in the headless mode by running the following command:

```bash
$ tools/bin/mage js:cypressHeadless
```

You can run Cypress in the interactive mode by running the following command:

```bash
$ tools/bin/mage js:cypressInteractive
```

#### JavaScript based tests

We find the [JS Unit Testing Guide](https://github.com/mawrkus/js-unit-testing-guide) a good starting point for informing our testing guidelines and we recommend reading through this guide. Note, that we employ some different approaches regarding [Grammar and Capitalization](#Grammar-and-capitalization).

We have extracted and adapted the most important parts below.

##### Pattern

The goal of naming our tests is to have a concise and streamlined description helping us to understand what a test is testing specifically. In order to do that, we follow a **"unit of work - scenario/context - expected behaviour"** pattern:
```js
// Schema.
describe('[unit of work]', () => {
  it('[expected behaviour] when [scenario/context]', () => {
    …
  });
});

// Example.
describe('Login', () => {
  it('succeeds when using correct credentials', () => {
    …
  });
});
```

This pattern will also help you organizing your tests better.

##### Grammar and Capitalization

Avoid using the modal verb `should` when describing tests. This will add redundancy and unnecessary verbosity to the test description. Instead, use a simple present tense sentence without any modality and with `when` as conjunction. Don't use end of sentence periods.

The `[unit of work]` bit, as part of the outermost `describe()` function is always capitalized, whereas the `[expected behavior]` part of the `it()` function is always lowercase. This way, the suit will generate proper english sentences when concatenating the test descriptions.

```js
// Bad: using `should`.
describe('Login', () => {
  it('should succeed when using correct credentials', () => {
    …
  });
});

// Bad: wrong capitalization.
describe('login', () => {
  it('Succeeds when using correct credentials', () => {
    …
  });
});

// Good: No should and proper capitalization.
describe('Login', () => {
  it('succeeds when using correct credentials', () => {
    …
  });
});

```

##### React Components

When testing react components, the name of the component is written as `<ReactComponent />`.

```js
// Bad: not using JSX syntax.
describe('MyComponent', () => {
  it('matches snapshot', () => {
    …
  });
});

// Bad: describing the component instead of naming it.
describe('My component', () => {
  it('matches snapshot', () => {
    …
  });
});

// Good
describe('<MyComponent />', () => {
  it('matches snapshot', () => {
    …
  });
});

```

##### Structurizing tests

We always use the `describe() / it()` hooks to write all tests, even if there's only one test in the suite. This keeps our tests streamlined and allows for easy extension of the test suite.

```js
// Bad: using `test()` hook.
test('flattens the object', () => {
  …
});

// Good: using `describe() / it()` hooks
describe('Get by path', () => {
  it('succeeds when using correct credentials', () => {
    …
  });
});

```

It's fine to use multiple hierarchies of `describe()` to group related tests more accurately:

```js
describe('User registration', () => {
  it('succeeds when using valid inputs', () => {
  });

  describe('when using invalid input values', () => {
    it('shows an error notification', () => {
    });

    it('does not perform a redirect', () => {
    });
  });

  describe('when using an already registered email', () => {
    it('shows an error notification', () => {
    });
  });
});
```

##### Test Driven Development (TDD)

Test Driven Development is a development philosophy that puts tests at the core of development. At The Things Industries, we don't enforce this method but we strongly encourage to adopt a process that emphasizes testing. Since adding fontend-based end-to-end tests to our codebase, we plan to do the following:

1. Writing end-to-end tests for all newly added features
2. Writing end-to-end tests for each (significant) bug that was resolved
3. Gradually adding coverage to existing features

Currently, we only employ frontend-based end-to-end tests, meaning that these tests can only be written if they are also operable through the frontend.

#### Writing End-to-End Tests

It is highly suggested to read [Cypress documentation](https://docs.cypress.io/guides) before starting to write tests.

##### Guiding Principle

We follow the following principle for writing useful end-to-end tests:

> The more your tests resemble the way your software is used, the more confidence they can give you.

This means that when writing tests, we always consider the real-life equivalent of the test scenario to design the test setup. This means:

##### Selecting elements

In line with the principle mentioned above, we have also included [`Testing Library`](https://testing-library.com/) to use advanced testing utilities. `Testing Library` has a good guide for [how to select elements](https://testing-library.com/docs/guide-which-query). We try to follow this guide for our end-to-end tests.

In some cases it can be necessary to select DOM elements using a special selection data attribute. We use `data-test-id` for this purpose. Use this attribute to select DOM elements when more realistic means of selection are not sufficient. Use meaningful but concise ID values, such as `error-notification`.

- Select DOM elements using text captions and labels when possible.
  - Select form fields by its label via `cy.findByLabelText`, e.g. `cy.findByLabelText('User ID')`. Same for field errors,warnings and descriptions, use `cy.findErrorByLabelText`, `cy.findWarningByLabelText` and `cy.findDescriptionByLabelText`.
  - Select buttons, links, tabs and other elements that are described by [ARIA roles](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques#Roles) via `cy.findByRole`, e.g. `cy.findByRole('button', {name: 'Submit'})`.
  - Select text elements via `cy.findByText`.
- Assert that selected elements are visible.

  ```html
  <!-- Instead of `visibility: hidden` it could be `display:none` or `z-index: -1` as well. -->
  <div data-test-id="test" style="visibility: hidden">
    Test content
  </div>
  ```

  ```js
  // Bad. This assertions will pass while not being visible to the user.
  cy.findByTestId('test').should('exists')

  // Good. This assertion will rightfully fail.
  cy.findByTestId('test').should('be.visible')
  ```

##### Test runner globals

Cypress uses [Mocha](https://mochajs.org/) as the test runner internally, while for unit tests we use [Jest](https://jestjs.io/). To keep our tests consistent we prefer using globals from `Jest` when possible.

Jest globals | Mocha globals | Used for
--- | --- | ---
`describe` | `describe` | Group together related tests
`it` | `it` | Define a single test
`beforeEach`/`afterEach` | `beforeEach`/`AfterEach` | Hook before/after each test (`it`)
`beforeAll`/`afterAll` | `before`/`after` | Hook before/after test block (`describe`)

##### End-to-end tests file structure

```bash
./pkg/cypress
|-- fixtures                    Cypress mocks
|-- integration                 frontend end-to-end specifications (1)
|   |-- console                 Console related end-to-end tests
|   |   |-- users               tests related to user entity
|   |   |-- ...
|   |   |-- shared              tests that are not directly related to a specific entity (2)
|   |-- oauth                   OAuth related end-to-end tests
|   |   `-- ...
|   `--smoke                    smoke tests (3)
|-- plugins                     Cypress plugins
|-- support                     Cypress commands and test utilities
|-- screenshots                 screenshots generated when running tests (4)
`-- videos                      videos generated when running tests (5)
```

1. `pkg/cypress/integration` contains all test specifications.
  - Each test file must be placed into corresponding folder (`console`/`oauth`/`smoke`).
  - Each test must follow the following naming: `{context}.spec.js`.
  - One test file must have end-to-end tests dedicated only to a specific entity or view.
2. `pkg/cypress/{console|oauth}/shared` contains all test specification not directly related to a single entity or a view. For example, `side-navigation.spec.js` or `header.spec.js` must be placed into the `cypress/console/shared` folder because both components are present on multiple views and are partially related to the stack entities. Make sure to scope cypress selections within the tested component using `cy.within`.
3.  For example, a typical smoke test can verify that the user is able to register, login, create application and register The Things Uno. For more details and diffeence between regular end-to-end and smoke tests see the [End-to-end tests structure](#organizing-end-to-end-tests) section.
4. and 5. Cypress stores screenshots and videos to the appropriate folder after running end-to-end tests. These should not be added to the repository.

##### Organizing end-to-end tests

When writing end-to-end tests we comply with the following guidelines:

- Tests are grouped by views for a specific entity. For example, when testing creation of application API keys:
  1. Add test file `cypress/integration/console/applications/api-keys/create.spec.js`.
  2. Test the behavior of the API key create view independently from any other specification.
- Do not repeat actions via the UI that are not related to the current test context. Consider adding reusable [Cypress commands](https://docs.cypress.io/api/cypress-api/custom-commands.html) that do necessary test setup programmatically. This means that when testing any UI that is not the login specification and requires the user to be logged in, there is no need to log in through the login page, while we simply fetch the access token. Note: this does not mean that one cannot create a cypress command that performs actions via UI.
- Extract components that appear on various views and test them separately instead of making assertions in each test where this component is used. For example, such components could be the page header and side navigation.
- Dedicate at least one test to assert that the view displays its UI elements in place on initial load. Assert on UI changes in tests that trigger these changes.
- Prefer duplicating entities with non-conflicting ids in tests instead of executing database teardown before each test. For example, when testing various scenarios for registering gateway, consider creating gateways with different ID's and EUI's instead of using a single gateway and drop the database before each test. Note: try to use this approach when possible, otherwise do not hesitate to restore database state before each test.
- Consider various stack configurations when writing end-to-end tests. Some views have different UI depending on availability of different stack components. For example, the end device wizard looks different for deployments with complete cluster (NS+JS+AS) and for JS-only configuration. Likewise, sections and entire views can be enabled or disabled based on our feature toggles. If your test scenario differs based on different feature toggle conditions, make sure to probe these preconditions in your tests.

##### Smoke tests

We distinguish between regular end-to-end tests and smoke tests. While regular end-to-end tests are scoped to a specific view or component and tests those in depth, smoke tests are testing complete user stories that are critical to the overall integrity of the application and usually comprise multiple components and views, e.g. login flow, user registration or creation of applications. When writing smoke tests we comply with the following guidelines:

- Smoke tests are testing complete user stories in a **wide and shallow** manner, meaning:
  - performing some complex and critical flow that touches multiple components, APIs and/or views
  - not testing different configurations or preconditions of the same flow in depth
  - For example, when testing registration of The Things Uno:
    1. Add test file `cypress/integration/smoke/devices/create.js`
    2. Describe the whole user story to register the device including creating an application (or using an existing one), link the application and create the end device.
- One smoke test should be encapsulated into a single `describeSmokeTest` declaration.

## Building and Running

There is a single binary for the server, `ttn-lw-stack`, as well as a binary for the command-line interface `ttn-lw-cli`. The single binary contains all components start one or multiple components. This allows you to run The Things Stack with one command in simple deployment scenarios, as well as distributing micro-services for more advanced scenarios.

We provide binary releases for all supported platforms, including packages for various package managers at https://github.com/TheThingsNetwork/lorawan-stack/releases. We suggest you use the compiled packages we provide in production scenarios.

Before the binaries are built, the frontend needs to be built. You can control whether to build the frontend for production or development by setting the `NODE_ENV` environment variable to either `development` or `production`.

The difference of a development build includes:

- Including source maps
- Using DLL bundle for modules to reduce build time
- A couple of special build options to improve usage with `webpack-dev-server`

The frontend can then be built using:

```bash
$ tools/bin/mage js:build
```

For development/testing purposes we suggest to run the binaries directly via `go run`:

```bash
$ go run ./cmd/ttn-lw-stack start
```

It is also possible to use `go build`, or release snapshots, as described below.

## Releasing

The Things Stack uses [GoReleaser](https://goreleaser.com/) for releases. If you want to build a release (snapshot), you first need to install GoReleaser:

```bash
$ go install<EMAIL_ADDRESS>```

The command for building a release snapshot is:

```bash
$ goreleaser --snapshot -f .goreleaser.snapshot.yml --rm-dist
```

The command for building a full release is:

```bash
$ goreleaser -f .goreleaser.release.yml --rm-dist
```

> Note: Goreleaser is configured to sign binaries, as per GitHub Action in `.github/workflows/release-*.yml`. If you're doing a release locally, you will need key's passphrase, or need to skip the signing step.

> Note: You will at least need to have [`rpm`](http://rpm5.org/) and [`snapcraft`](https://snapcraft.io/) in your `PATH` if you want to build a full release.

This will compile binaries for all supported platforms, `deb`, `rpm` and Snapcraft packages, release archives in `dist`, as well as Docker images.

> Note: The operating system and architecture represent the name of the directory in `dist` in which the binaries are placed.
> For example, the binaries for Darwin x64 (macOS) will be located at `dist/darwin_amd64`.

A new version is released from the `v3.n` branch. The necessary steps for each are detailed below.

> Note: To get the target version, you can run `version=$(tools/bin/mage version:bumpXXX version:current)`, where xxx is the type of new release (minor/patch/RC). Check the section [Version Bump](#version-bump) for more information.

### Release From Master

Create a `Release` issue in this repository and follow the steps.

### Release Backports

Create a `Backport Release` issue in this repository and follow the steps.

## Troubleshooting

### Console

#### Problem: Assets are not found

The Console will render a blank page and you can see backend logs like e.g.:
```
INFO Request handled                          duration=40.596µs error=error:pkg/errors/web:unknown (Not Found) message=Not Found method=GET namespace=web remote_addr=[::1]:50450 request_id=01DZ2CJDWKAFS10QD1NKZ1D56H status=404 url=/assets/console.36fcac90fa2408a19e4b.js
```

You might also see error messages in the Console such as:
```
Uncaught ReferenceError: libs_472226f4872c9448fc26 is not defined
    at eval (eval at dll-reference libs_472226f4872c9448fc26 (console.js:26130), <anonymous>:1:18)
    at Object.dll-reference libs_472226f4872c9448fc26 (console.js:26130)
    at …
```

#### Possible causes

##### Using incorrect or no configs

If you plan to run the Console / Account App in development mode, it is important to ensure that the right configuration is loaded both for running The Things Stack itself, as well as the development tooling (e.g. `tools/bin/mage serve`).

##### Possible solution

Make sure that you source the environment variables as [described above](#development-configuration) **in all terminal sessions** that run The Things Stack or the `tools/bin/mage` commands.

##### Missing restart

The stack has not been restarted after the Console bundle has changed. In production mode, The Things Stack will access the bundle via a filename that contains a content-hash, which is set during the build process of the Console. The hash cannot be updated during runtime and will take effect only after a restart.

##### Possible solution

  1. Restart the The Things Stack

##### Accidentally deleted bundle files

The bundle files have been deleted. This might happen e.g. when a mage target encountered an error and quit before running through.

##### Possible solution

  1. Rebuild the Console `tools/bin/mage js:clean js:build`
  2. Restart The Things Stack

##### Mixing up production and development builds

If you switch between production and development builds of the Console, you might forget to re-run the build process and to restart The Things Stack. Likewise, you might have arbitrary config options set that are specific to a respective build type.

##### Possible solution

  1. Double check whether you have set the correct environment: `echo $NODE_ENV`, it should be either `production` or `development`
  2. Double check whether [your The Things Stack config](#development-configuration) is set correctly (especially `TTN_LW_CONSOLE_UI_JS_FILE`, `TTN_LW_CONSOLE_UI_CANONICAL_URL` and similar settings). Run `ttn-lw-stack config --env` to see all environment variables
  3. Make sure to rebuild the Console `tools/bin/mage js:clean js:build`
  4. Restart The Things Stack

#### Problem: Console rendering blank page and showing arbitrary error message in console logs, e.g.:

```
console.4e67a17c1ce5a74f3f50.js:104 Uncaught TypeError: m.subscribe is not a function
    at Object../pkg/webui/console/api/index.js (console.4e67a17c1ce5a74f3f50.js:104)
    at o (console.4e67a17c1ce5a74f3f50.js:1)
    at Object../pkg/webui/console/store/middleware/logics/index.js (console.4e67a17c1ce5a74f3f50.js:104)
    at o (console.4e67a17c1ce5a74f3f50.js:1)
    at Object.<anonymous> (console.4e67a17c1ce5a74f3f50.js:104)
    at Object../pkg/webui/console/store/index.js (console.4e67a17c1ce5a74f3f50.js:104)
    at o (console.4e67a17c1ce5a74f3f50.js:1)
    at Module../pkg/webui/console.js (console.4e67a17c1ce5a74f3f50.js:104)
    at o (console.4e67a17c1ce5a74f3f50.js:1)
    at Object.0 (console.4e67a17c1ce5a74f3f50.js:104)
```

#### Possible causes

##### Bundle using old JS SDK

The bundle integrates an old version of the JS SDK. This is likely a caching/linking issue of the JS SDK dependency.

##### Possible solutions

- Re-establish a proper module link between the Console and the JS SDK
  - Run `tools/bin/mage js:cleanDeps js:deps`
  - Check whether the `ttn-lw` symlink exists inside `node_modules` and whether it points to the right destination: `lorawan-stack/sdk/js/dist`
    - If you have cloned multiple `lorawan-stack` forks in different locations, `yarn link` might associate the JS SDK module with the SDK on another ttn repository
  - Rebuild the Console and (only after the build has finished) restart The Things Stack

#### Problem: Console rendering blank page and showing `Module not found` message in console logs, e.g.:

```
ERROR in ./node_modules/redux-logic/node_modules/rxjs/operators/index.js Module not found: Error: Can't resolve '../internal/operators/audit' in '/lorawan-stack/node_modules/redux-logic/node_modules/rxjs/operators'
```

##### Possible cause: Broken yarn or npm cache

##### Possible solution: Clean package manager caches

- Clean yarn cache: `yarn cache clean`
- Clean npm cache: `npm cache clean`
- Clean and reinstall dependencies: `tools/bin/mage js:cleanDeps js:deps`

#### Problem: The build crashes without showing any helpful error message

##### Cause: Not running mage in verbose mode

`tools/bin/mage` runs in silent mode by default. In verbose mode, you might get more helpful error messages

##### Solution

Run mage in verbose mode: `tools/bin/mage -v {target}`

#### Problem: Browser displays error:
`Cannot GET /`

##### Cause: No endpoint is exposed at root

##### Solution:

Console is typically exposed at `http://localhost:8080/console`,
API at `http://localhost:8080/console`,
OAuth at `http://localhost:8080/oauth`,
etc

#### Problem: Browser displays error:
`Error occurred while trying to proxy to: localhost:8080/console`

##### Cause: Stack is not available or not running

##### Solution:

For development, remember to run the stack with `go run`:

```bash
$ go run ./cmd/ttn-lw-stack start
```

#### General advice

A lot of problems during build stem from fragmented, incomplete runs of mage targets (due to arbitrary errors happening during a run). Oftentimes, it then helps to build the entire Web UI from scratch: `tools/bin/mage jsSDK:cleanDeps jsSDK:clean js:cleanDeps js:clean js:build`, and (re-)start The Things Stack after running this.

repository: TheThingsNetwork/lorawan-stack

gha_language: Go

repo_name: TheThingsNetwork/lorawan-stack


text:
# Online Search Relevance Metrics

A simulation and set of configurations for calculating and exploring online search relevance metrics.

For a high-level introduction, please see the accompanying blog post, [Analyzing online search relevance metrics with the Elastic Stack](https://www.elastic.co/blog/analyzing-online-search-relevance-metrics-with-the-elastic-stack).

![Kibana dashboard](https://user-images.githubusercontent.com/181622/85378369-c40ae380-b53a-11ea-9d0c-5a97d1c00d24.png)

## Contents

- [Online Search Relevance Metrics](#online-search-relevance-metrics)
  - [Setup](#setup)
     - [Prerequisites](#prerequisites)
     - [Get started](#get-started)
  - [Simulating events and visualizing metrics](#simulating-events-and-visualizing-metrics)
     - [Run simulation](#run-simulation)
     - [Kibana visualizations](#kibana-visualizations)
  - [Implementation details](#implementation-details)
     - [Event schema](#event-schema)
        - [Generate a custom ECS schema](#generate-a-custom-ecs-schema)
     - [Transforming events into per-query metrics](#transforming-events-into-per-query-metrics)
        - [Ingest pipeline](#ingest-pipeline)
        - [Output index](#output-index)
     - [Aggregate metrics](#aggregate-metrics)
        - [Counting with queries](#counting-with-queries)
        - [Single-metric aggregations](#single-metric-aggregations)
        - [Using Kibana for visualisation](#using-kibana-for-visualisation)
        - [Using eland for deeper exploration](#using-eland-for-deeper-exploration)

## Setup

### Prerequisites

To run the simulation, you will first need:

 - Elasticsearch and Kibana 7.7+ **
   - Cloud [Elasticsearch Service](https://www.elastic.co/elasticsearch/service) (free trials available)
   - [Local installations](https://www.elastic.co/start)
 - `make`
 - Python 3.7+ (try [pyenv](https://github.com/pyenv/pyenv) to manage multiple Python versions)
 - `virtualenv` (installed with `pip install virtualenv`)

** Instructions and code have been tested on versions: 7.7.0, 7.8.0. Instructions reference Kibana and Cloud pages as of June 2020.

### Get started

Use the `Makefile` for all setup, building, testing, etc. Common commands (and targets) are:

 - `make init`: install project dependencies (from requirements.txt)
 - `make clean`: cleanup environment
 - `make test`: run tests
 - `make jupyter`: run Jupyter Lab (notebooks)

Most operations are performed using scripts in the `bin` directory. Use `-h` or `--help` on the commands to explore their functionality and arguments.

## Simulating events and visualizing metrics

### Run simulation

To simulate and index events then run all transforms, use the `bin/simulate` script.

```bash
bin/simulate elasticsearch
```

The simulation will ensure that the right configurations in Elasticsearch are setup. You can also alter dimensions of the simulation such as the number of users. For example:

```bash
bin/simulate --num-users 1000 elasticsearch
```

If you are running a Cloud Elasticsearch Service instance you can set the Elasticsearch endpoint in `simulate`. To find the Elasticsearch endpoint, open your [deployments](https://cloud.elastic.co/deployments), select a deployment, then under "Applications" you will find a link beside Elasticsearch to "Copy endpoint". Now use the `url` option to set the endpoint URL for your instance. Since security is enabled, you will need to also add the username and password in the URL. For example, the user `elastic` with password `changeme` can be set as follows:

```bash
bin/simulate elasticsearch --url https://elastic:changeme@YYY.us-central1.gcp.cloud.es.io:9243
```

Use the `-h` or `--help` arguments to explore more functionality and arguments.

### Kibana visualizations

To recreate the visualisations in Kibana, you need to first make sure you have data in your Elasticsearch instance using the above `simulate` command. Then you just need to run `kibana` to create the dashboard, index pattern and visualizations.

```bash
bin/kibana
```

As with `simulate`, if you are running on Cloud, ensure that your credentials are set and the correct Kibana URL is used (don't use the Elasticsearch URL!). You can find your Kibana endpoint just below where you found the Elasticsearch endpoint, and your credentials should be the same as with Elasticsearch.

```bash
bin/kibana --url https://elastic:changeme@YYY.us-central1.gcp.cloud.es.io:9243
```

You're all set! Have a look at the Dashboard and Visualizations pages now and you should see a large set of ready-made visualizations. Make sure that your time range is set to the entire day of 15 Nov 2019 UTC.

## Implementation details

In [the blog post](http://elastic.co/blog/), we cover the motivation, set some context and outline the high-level approach to collecting and exploring online search relevance metrics in the stack. The remainder of this README contains implementation details which, should you want to implement the approach yourself, provide you will everything you need to get up-and-running fast!

### Event schema

The first thing to setup is the index in Elasticsearch that will store the user behaviour events: `query`, `page` and `click`. To describe the schema we'll be using the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS). ECS is a good fit for this use-case since it already describes a large set of common attributes for events. We can reuse attributes like basic categorical, timestamp and user information, instead of having to describe them ourselves. When using ECS, we also get some features for free in Kibana such as the Logs and Maps apps.

We don't need all of the ECS field sets, so we'll stick to a [subset](config/ecs/subset.yml) for now. If you have more ECS fields that you can or want to use, you are welcome to do so. A snippet from the [schema](config/indices/ecs-search-metrics.json) shows some of the ECS fields we need.

```json
{
    "@timestamp": {
      "type": "date"
    },
    "ecs": {
      "properties": {
        "version": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "event": {
      "properties": {
        "action": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "dataset": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "duration": {
          "type": "long"
        },
        "id": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    }
}
```

Some attributes of our events are not described in ECS, but we can easily extend ECS with custom fields for our specific solution. Here's some fields that we will need. The purpose of each field is described in the [custom field set](config/ecs/custom).

```json
{
  "SearchMetrics": {
    "properties": {
      "click": {
        "properties": {
          "result": {
            "properties": {
              "id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "rank": {
                "type": "long"
              },
              "reciprocal_rank": {
                "type": "float"
              }
            }
          }
        }
      },
      "query": {
        "properties": {
          "id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "page": {
            "type": "long"
          },
          "value": {
            "ignore_above": 4096,
            "type": "keyword"
          }
        }
      },
      "results": {
        "properties": {
          "ids": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "size": {
            "type": "long"
          },
          "total": {
            "type": "long"
          }
        }
      }
    }
  }
}
```

With the extended ECS schema we can create the event index `ecs-search-metrics`. Make sure that you configure the index settings appropriate for your environment as the settings that are generated are not for production purposes (e.g. number of shards and replicas).

Have a look at the simulation's `ecs-search-metrics` index configuration for a complete example of this step: [config/indices/ecs-search-metrics.json](config/indices/ecs-search-metrics.json)

#### Generate a custom ECS schema

(Optional reading! Feel free to skip this if you are OK using the ECS schema already generated in this project.)

Given a subset of ECS fields that we need, plus custom fields that we want to add, we can generate an ECS compatible schema from a clone of the ECS repository. We want to stick to a specific commit in the HEAD of master for now, so we also checkout a specific commit.

``` bash
git clone git@github.com:elastic/ecs.git && \
  cd ecs && \
  git checkout 994f7777b53ba31fe55dffd0667b347fb8afdfee
```

Now run the `generator.py` script to generate a new schema. This will use the configuration in `config/ecs` to define which ECS fields to use and the definition of custom fields.

Please make to set the `SIM_DIR` variable before running this to reference this codebase on your local drive.

```bash
export SIM_DIR=...
echo $SIM_DIR
```

```bash
make ve && \
build/ve/bin/python scripts/generator.py \
  --ref v1.5.0 \
  --subset $SIM_DIR/config/ecs/subset.yml \
  --include $SIM_DIR/config/ecs/custom \
  --out generated_custom
```

After generating the custom schema, you can find it in `generated_custom/generated/elasticsearch/7/template.json`. This will have to be changed again, manually, to set the index settings appropriate for the simulation. The easiest thing to do is just copy the `properties` of the `mappings` and replace that in the existing schema. All the index settings don't need to be copied.

### Transforming events into per-query metrics

The heart of calculating per-query metrics is in this step. Once you have events indexed in `ecs-search-metrics`, you can apply a transform to group events by their search query ID and calculate per-query metrics. This step requires three components: a transform, an ingest pipeline, and an output index.

As mentioned above, the transform is where we define the group-by strategy, as well as the basic aggregations we want to do on our events in each group. This includes any basic counting of events like total clicks, as well as descriptive statistics like the average number of clicks, etc.

Here's a snippet of the [transform configuration](config/transforms/ecs-search-metrics_transform_queryid.json) showing some of the calculations performed on `click` events.

```json
{
    "metrics.clicks": {
      "filter": { "term": { "event.action": "SearchMetrics.click" } },
      "aggregations": {
        "count": { "value_count": { "field": "event.id" } },
        "count_at_3": { "filter": { "range": { "SearchMetrics.click.result.rank": { "lte": 3 } } } },
        "max_reciprocal_rank": { "max": { "field": "SearchMetrics.click.result.reciprocal_rank" } },
        "mean_reciprocal_rank": { "avg": { "field": "SearchMetrics.click.result.reciprocal_rank" } },
        "first_click_time": { "min": { "field": "@timestamp" } },
        "last_click_time": { "max": { "field": "@timestamp" } }
      }
    }
}
```

The transform can be set up as either a [continuous or batch transform](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transform-overview.html). Continuous transforms are necessary when you want to calculate metrics on an ongoing basis from events that are being ingested on a continuous basis (e.g. in real time or hourly batches from an external source). Batch transforms can be used for experimentation as they are only able to be run once (the simulation uses batch transforms).

#### Ingest pipeline

The ingest pipeline is used to take documents from the transform and calculate additional metrics that are easier to calculate in two steps, rather than in a complicated script in the transform. For example, any metrics that require a comparison between the original `query` event and the statistics from the transform group are best performed this way, such as time to first click which needs to find the difference between the `query` event and the first `click` event.

In the transform, we calculate the minimum click timestamp:

```json
{
    "metrics.clicks": {
      "filter": { "term": { "event.action": "SearchMetrics.click" } },
      "aggregations": {
        "first_click_time": { "min": { "field": "@timestamp" } }
      }
    }
}
```

In the pipeline, we take the difference between the query and the minimum click timestamp to find the time to first click:

```json
{
  "script": {
    "source": "ctx.metrics.clicks.time_to_first_click = ChronoUnit.MILLIS.between(ZonedDateTime.parse(ctx.query_event['@timestamp']), ZonedDateTime.parse(ctx.metrics.clicks.first_click_time))",
    "if": "ctx.metrics.clicks.first_click_time != null"
  }
}
```

#### Output index

What comes out of the ingest pipeline is now ready to be stored in an index that will be our primary metrics index. We're giving it the super creative yet transparent name of `ecs-search-metrics_transform_queryid` (also matches the transform name) and it will contain all of the per-query documents that describe the original `query` event plus the metrics we calculated in the previous steps.

The schema for this index should include all the fields from the original `query` event, as well as the metrics we've calculated in the previous steps. Have a look at the schema for a complete example: [config/indices/ecs-search-metrics_transform_queryid.json](config/indices/ecs-search-metrics_transform_queryid.json)

### Aggregate metrics

Now that we have metrics calculated on a per-query basis and stored in an index, we can use various methods of aggregation to calculate metrics spanning over a period time and multiple users.

#### Counting with queries

Let's start with the simplest way of calculating and showing metrics by executing an Elasticsearch Query to do some simple counting. There's no time bucketing, user filtering or aggregating — we'll save those for the next steps! For our CTR@3 example, we just need to find the two components of calculating CTR@3: the number of queries with clicks at 3 and the total number of queries with clicks.

The number of queries with results and clicks in any position:

```
POST /ecs-search-metrics_transform_queryid/_search?size=0
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "query_event.SearchMetrics.results.size": {
              "gt": 0
            }
          }
        },
        {
          "range": {
            "metrics.clicks.count": {
              "gt": 0
            }
          }
        }
      ]
    }
  }
}
```

The number of queries with results and clicks at 3:

```
POST /ecs-search-metrics_transform_queryid/_search?size=0
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "query_event.SearchMetrics.results.size": {
              "gt": 0
            }
          }
        },
        {
          "term": {
            "metrics.clicks.exist_at_3": true
          }
        }
      ]
    }
  }
}
```

Dividing the former by the latter gives us CTR@3, simple as that.

#### Single-metric aggregations

Some of the metrics we calculated per-query are even simpler to aggregate into single metrics. Take for example the time to first click. If we want to get a high-level understanding of the time it takes for users to click on the first result after executing the query, we simply take the average over a timeframe. If we want to include all of time, we can use the following [average aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/search-aggregations-metrics-avg-aggregation.html).

```
POST /ecs-search-metrics_transform_queryid/_search?size=0
{
  "aggs": {
    "avg_time_to_first_click": {
      "avg": {
        "field": "metrics.clicks.time_to_first_click"
      }
    }
  }
}
```

Returns:

```
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : { ... },
  "hits" : { ... },
  "aggregations" : {
    "avg_time_to_first_click" : {
      "value" : 21222.222222222223
    }
  }
}
```

The same applies to any other per-query metric that we want to aggregate to a higher level. Typically they are just averages over a time period, but you can also explore other descriptive statistics such as variance, median, or extreme values such as 90th percentile. Check out the [extended stats aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/search-aggregations-metrics-extendedstats-aggregation.html) as a way to collect a set of descriptive statistics for a field.

```
POST /ecs-search-metrics_transform_queryid/_search?size=0
{
  "aggs": {
    "descriptive_stats": {
      "extended_stats": {
        "field": "metrics.clicks.time_to_first_click"
      }
    }
  }
}
```

Returns:

```
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : { ... },
  "hits" : { ... },
  "aggregations" : {
    "descriptive_stats" : {
      "count" : 27,
      "min" : 3000.0,
      "max" : 49000.0,
      "avg" : 21222.222222222223,
      "sum" : 573000.0,
      "sum_of_squares" : 1.7223E10,
      "variance" : 1.8750617283950615E8,
      "std_deviation" : 13693.289336003463,
      "std_deviation_bounds" : {
        "upper" : 48608.80089422915,
        "lower" : -6164.356449784704
      }
    }
  }
}
```

#### Using Kibana for visualisation

Now that we've seen some examples of calculating aggregate metrics directly with the Elasticsearch Query DSL, we know that we can use Kibana to visualize metrics and show more useful metrics that are bucketed by time. For example, plotting our CTR@3 metric can be achieved using a simple [TSVB time series chart](https://www.elastic.co/guide/en/kibana/7.8/TSVB.html), bucketing out metrics every hour:

When we configure this time series chart for CTR@3 we only want to include queries that have clicks, so we'll set a filter on this time series using the [KQL](https://www.elastic.co/guide/en/kibana/7.8/kuery-query.html) clause `metrics.clicks.count > 0`:

We'll use then a ratio filter to calculate ratio of the number of queries with clicks at 3 (KQL: `metrics.clicks.exists_at_3 : true`) over the number of queries with clicks in any position (KQL: `*`):

In this simulation project, we've created a large number of visualizations and a dashboard. These can be imported into a fresh Elasticsearch using Kibana's import saved objects feature. Instructions for this can be found above in the [Kibana Visualizations](#kibana-visualizations) section.

#### Using `eland` for deeper exploration

Sometimes we want to dig deeper into our datasets and metrics and using Kibana or the Elasticsearch Query DSL statements can be cumbersome or insufficient. Many Data Scientists and Analysts are more comfortable with using Python, Jupyter notebooks and Pandas to do this type of exploration. Using the [eland](https://eland.readthedocs.io/) library provides a Pandas interface to explore large datasets natively in Elasticsearch, by executing Elasticsearch queries and aggregations behind the scene. This allows you to explore all the metrics using a familiar interface.

For a full example of calculating the same set of online search relevance metrics as in our Kibana example, check out the [Jupyter notebook](notebooks/Metrics%20with%20eland.ipynb).

url: https://raw.githubusercontent.com/rwaight/elastic-examples/2da957327d77c3284641ecf3bb2d624516fc7543/Machine Learning/Online Search Relevance Metrics/README.md

repository: rwaight/elastic-examples

branch_name: refs/heads/master

extension: md

gha_language: JavaScript

license_type: permissive

repo_name: rwaight/elastic-examples


text:
# Monitoring solution

Deploy monitoring solution with graphana and prometheus and redis

## Accessing the solution

Go to the marketplace https://marketplace.grid.tf/marketplace and click on Monitoring

## Steps

![Step1](./img/monitoring_1.png)

## Solution name

![Step2](./img/monitoring_2.png)

## Select the network to deploy the solution on

![Step3](./img/monitoring_3.png)

## Add your sshkey to be able to access the containers

![Step4](./img/monitoring_4.png)

## Select the solution resources

![Step5](./img/monitoring_5.png)

## Select prometheus volume size

![Step6](./img/monitoring_6.png)

## Select Graphana container resources

![Step7](./img/monitoring_7.png)

## Select the redis container resources

![Step8](./img/monitoring_8.png)

## Optional, select the node id you want to deploy prometheus on

![Step9](./img/monitoring_9.png)

## Optional, select the node id you want to deploy Graphana on

![Step10](./img/monitoring_10.png)

## Optional, select the node id you want to deploy Redis on

![Step11](./img/monitoring_11.png)

## Optional, select the farm you want to deploy promethus on

![Step12](./img/monitoring_12.png)

## Optional, select the farm you want to deploy graphana on

![Step13](./img/monitoring_13.png)

## Optional, select the farm you want to deploy Redis on

![Step14](./img/monitoring_14.png)

## Select prometheus container ip

![Step15](./img/monitoring_15.png)

## Select graphana container ip

![Step16](./img/monitoring_16.png)

## Select Redis container ip

![Step17](./img/monitoring_17.png)

## Select expiration time for the solution

![Step18](./img/monitoring_18.png)

## Summary for the solution values

![Step19](./img/monitoring_19.png)

## Deploy the solution

![Step20](./img/monitoring_20.png)

## Pay for the solution using threebot connect staging app wallet

![Step21](./img/monitoring_21.png)

## Wait for deployment

![Step22](./img/monitoring_22.png)

## Congratulations your solution has been deployed

![Step23](./img/monitoring_23.png)

## Acccess container via web or ssh

![Step24](./img/monitoring_24.png)


text:
### Description

This problem was asked by Google.

The area of a circle is defined as `πr^2`. Estimate `π` to `3` decimal places using a Monte Carlo method.

Hint: The basic equation of a circle is `x^2 + y^2 = r^2`.

### Solution

```swift
/*
 * For a circle of 2 units of diameter,
 * surface = πr^2 = π(1)^2 = π
 *
 * Imagine a square surface of 2x2 units
 * Imagine a circle of 2 unit of diameter centered in this square
 * Imagine printing enough points in this square to totally cover the area
 * For a large amount of random points then, we can say that:
 *
 * (amount of points in the circle) / (amount of points in the square) = (area of circle) / (area of square)
 *
 * We know that (area of square) = 2x2 = 4:
 *
 * (amount of points in the circle) / (amount of points in the square) = (area of circle) / 4
 *
 * We also know that (area of circle) = surface = π:
 *
 * (amount of points in the circle) / (amount of points in the square) = π/4
 *
 * π = 4 * (amount of points in the circle) / (amount of points in the square)
 *
 * Now, the equation of a circle is x^2 + y^2 = r^2
 *
 * x^2 + y^2 = 1
 *
 * A point is in the circle if x^2 + y^2 <= 1
 * A point is in the square if x <= 1 & y <= 1
 */

struct Maths {
    func estimatePi(iteration: Int) -> Double {
        var pointsInSquare = 0
        var pointsInCircle = 0

        for _ in 0...iteration {
            let xRandom = Double.random(in: 0...1)
            let yRandom = Double.random(in: 0...1)

            pointsInSquare += 1

            if pow(xRandom, 2) + pow(yRandom, 2) <= 1 {
                pointsInCircle += 1
            }
        }

        return 4 * Double(pointsInCircle) / Double(pointsInSquare)
    }
}
```

license: MIT

repository: Krisloveless/daily-coding-challenges

path: /swift/Problem 14/README.md

gha_license_id: MIT

repo_name: Krisloveless/daily-coding-challenges


text:
---
layout: engineering-education
status: publish
published: true
url: /graphs-in-data-structure-using-cplusplus/
title: Graphs in Data structure (using C++)
description: In this articles the reader will learn about Graphs in Data structure (using C++). We will briefly discuss undirected, directed, weighted, and Cyclic graphs.
author: anubhav-bansal
date: 2021-08-11T00:00:00-13:00
topics: []
excerpt_separator:
images:
  - url: /engineering-education/graphs-in-data-structure-using-cplusplus/hero.jpg
    alt: graph data structure c++ example image
---
Graphs are mathematical structures that reflect the pairwise relationship between things. A graph is a type of flow structure that displays the interactions of several objects. It may be represented by utilizing the two fundamental components, nodes and edges.

- **Nodes**: These are the most crucial elements of every graph. Edges are used to represent node connections. For example, a graph with two nodes connected using an undirected edge shows a bi-directional connection between those two nodes.

- **Edges**: Edges are part of a graph showing the connections between nodes. An edge represents the connection between two nodes.

### An application in real life
- **Google Maps**: link your journey from the start to the end.
- **Social Networks**: friends are connected with each other using an edge where each user represents a vertex.
- **Recommendation System**: relationship data between user's recommendations uses graphs for connection.

#### Type of graphs
- **Undirected**: All edges in an undirected graph are bidirectional and do not point in any particular direction.

![Undirected-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_1.png)

- **Directed**: In a directed graph, all edges are unidirectional; they point in a single direction.

![Directed-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph__2.JPG)

- **Weighted**: Weighted graph, with each edge, allocated a weight or cost. Consider a network with four nodes, shown in the picture. Every edge of a graph has some weight with it.

If you wish to get from vertex 2 to vertex 1, you can choose one of three routes:
```bash
2->3->1
2->1
2->3->4->1
```
As a result, the total cost of each path is as follows:

```bash
The total cost of 2->3->1 is (2+1), or three units.

The total cost of getting from 2->1 is one unit.

The total cost of 2->3->4->1 is (2+2+3), or seven units.
```

![Weighted-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_3.JPG)

- **Cyclic**: If a graph contains a path that starts and finishes at the same vertex, it is said to be cyclic. This is referred to as a cycle. An acyclic graph does not have a cycle.

![Cyclic-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_4.JPG)

The diagram below depicts a cycle (0->1->2). A tree is a graph that has just one path connecting any two vertices. Every node in a graph may have one or more parents. In a tree as each node has precisely one parent node.

A tree cannot have any loops or cycles, whereas graphs may.

![tree-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_5.JPG)

### Graph representation
We can represent a graph in several ways.

The following are the two most frequent ways of expressing a graph:

1. **Adjacency Matrix**
A V*V binary matrix is an adjacency matrix. There is an edge that is connecting vertex i and vertex j, element Ai,j is 1, otherwise Ai,j is 0.

>**Note**: A binary matrix has cells that can only have one of two possible values: 0 or 1.

We can store the cost of the edge at Ai,j in the adjacency matrix of a weighted graph rather than storing 0 or 1.

If Ai,j is 1 in the undirected graph then Ai,j will also be 1.

If Ai,j is 1 in the directed graph, then it may or may not be 1.

The adjacency matrix offers constant-time access (O(1)) to detect if two nodes have an edge. Therefore, the adjacency matrix has a space complexity of O(V)2.

2. **Adjacency List**
This is the most common way of representing a graph when we implement it in our code. An adjacency list is a collection of distinct lists in an array A. For example, every node of a graph Ai is a list that holds all the vertices' values next to vertex i.

If we have a weighted graph, we store the cost with the vertex using pairs. You will get a better understanding when we implement it in our code. If vertex j is in list Ai, vertex i will be in list Aj in an undirected graph.

Since that information in the adjacency list is only stored for edges that exist in the graph, its space complexity is O(V + E).
Representation of an undirected graph,

- G is an undirected graph with 5 vertices and 7 edges.

![undirected-graph](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_6.JPG)

- Adjacency-list representation of G.

![graphList](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_7.JPG)

- The adjacency-matrix representation of G.

![graphMartrix](/engineering-education/graphs-in-data-structure-using-cplusplus/graph_8.JPG)

### Code for Adjacency list representation of a graph

``` C++
#include <iostream>
#include<list>
using namespace std;

class graph{
public:
    list<int> *adjlist;
    int n;
    graph(int v){
        adjlist=new list<int> [v];
        n=v;
    }

    void addedge(int u,int v,bool bi){
        adjlist[u].push_back(v);
        if(bi){
            adjlist[v].push_back(u);
        }
    }

    void print(){
        for(int i=0;i<n;i++){
            cout<<i<<"-->";
            for(auto it:adjlist[i]){
                cout<<it<<" ";
            }
            cout<<endl;
        }
        cout<<endl;
    }

};

int main() {
    graph g(5);
    g.addedge(1,2,true);
    g.addedge(4,2,true);
    g.addedge(1,3,true);
    g.addedge(4,3,true);
    g.addedge(1,4,true);

    g.print();
}

```

You can find the complete execution of the above code [here](https://replit.com/@AnubhavBansal1/Gaphs#main.cpp).

#### Output

![output](/engineering-education/graphs-in-data-structure-using-cplusplus/output.JPG)

- In the output, you can see the implementation of graphs in the adjacency list view.

#### Conclusion
By learning graphs from the basics you will think more analytically while solving data structures questions. Next steps now, would be to practice some of the graphs questions, since we now have a better understanding with the foundation of graphs.

### Resources

- [Leet Code Problem 1](https://leetcode.com/problems/find-center-of-star-graph/)

- [Leet Code Problem 2](https://leetcode.com/problems/find-the-town-judge/)

Happy coding!

---
Peer Review Contributions by: [Okelo Violet](/engineering-education/authors/okelo-violet/)

repository: mbuguac/engineering-education

gha_language: SCSS

repo_name: mbuguac/engineering-education


text:
# mapsme Module

## Description

Titanium module to open the Maps.ME app from within your app

## Accessing the mapsme Module

To access this module from JavaScript, you would do the following:

    var mapsme = require("nl.vanvianen.mapsme");

The mapsme variable is a reference to the Module object.

## Usage

```js
var ret = require("nl.vanvianen.mapsme").openMapsMe({
	lat: 48.8582,
	lon: 2.2945
	name: 'Eiffel Tower'
});
if (!ret) {
	alert('Maps.ME is not installed');
}
```

## Author

Jeroen van Vianen

## License

Copyright 2016  Jeroen van Vianen

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

repository: morinel/mapsme

repo_name: morinel/mapsme


text:
# minio-node

## Dockerfile

This image is based on **node:8-alpine** with the minio-js npm's installed

Build
```bash
docker build -t lukepatrick/minio-node:1.0 .
```
Push
```bash
docker push lukepatrick/minio-node:1.0
```

repository: lukepatrick/minio-brigade

repo_name: lukepatrick/minio-brigade