text:
# Installing NixOS {#sec-installation}

## Booting the system {#sec-installation-booting}

NixOS can be installed on BIOS or UEFI systems. The procedure for a UEFI
installation is by and large the same as a BIOS installation. The
differences are mentioned in the steps that follow.

The installation media can be burned to a CD, or now more commonly,
"burned" to a USB drive (see [](#sec-booting-from-usb)).

The installation media contains a basic NixOS installation. When it's
finished booting, it should have detected most of your hardware.

The NixOS manual is available by running `nixos-help`.

You are logged-in automatically as `nixos`. The `nixos` user account has
an empty password so you can use `sudo` without a password:
```ShellSession
$ sudo -i
```

If you downloaded the graphical ISO image, you can run `systemctl
start display-manager` to start the desktop environment. If you want
to continue on the terminal, you can use `loadkeys` to switch to your
preferred keyboard layout. (We even provide neo2 via `loadkeys de
neo`!)

If the text is too small to be legible, try `setfont ter-v32n` to
increase the font size.

To install over a serial port connect with `115200n8` (e.g.
`picocom -b 115200 /dev/ttyUSB0`). When the bootloader lists boot
entries, select the serial console boot entry.

### Networking in the installer {#sec-installation-booting-networking}

The boot process should have brought up networking (check `ip
a`). Networking is necessary for the installer, since it will
download lots of stuff (such as source tarballs or Nixpkgs channel
binaries). It's best if you have a DHCP server on your network.
Otherwise configure networking manually using `ifconfig`.

On the graphical installer, you can configure the network, wifi
included, through NetworkManager. Using the `nmtui` program, you can do
so even in a non-graphical session. If you prefer to configure the
network manually, disable NetworkManager with
`systemctl stop NetworkManager`.

On the minimal installer, NetworkManager is not available, so
configuration must be perfomed manually. To configure the wifi, first
start wpa_supplicant with `sudo systemctl start wpa_supplicant`, then
run `wpa_cli`. For most home networks, you need to type in the following
commands:

```plain
> add_network
0
> set_network 0 ssid "myhomenetwork"
OK
> set_network 0 psk "mypassword"
OK
> set_network 0 key_mgmt WPA-PSK
OK
> enable_network 0
OK
```

For enterprise networks, for example *eduroam*, instead do:

```plain
> add_network
0
> set_network 0 ssid "eduroam"
OK
> set_network 0 identity<EMAIL_ADDRESS>OK
> set_network 0 password "mypassword"
OK
> set_network 0 key_mgmt WPA-EAP
OK
> enable_network 0
OK
```

When successfully connected, you should see a line such as this one

```plain
<3>CTRL-EVENT-CONNECTED - Connection to 32:85:ab:ef:24:5c completed [id=0 id_str=]
```

you can now leave `wpa_cli` by typing `quit`.

If you would like to continue the installation from a different machine
you can use activated SSH daemon. You need to copy your ssh key to
either `/home/nixos/.ssh/authorized_keys` or
`/root/.ssh/authorized_keys` (Tip: For installers with a modifiable
filesystem such as the sd-card installer image a key can be manually
placed by mounting the image on a different machine). Alternatively you
must set a password for either `root` or `nixos` with `passwd` to be
able to login.

## Partitioning and formatting {#sec-installation-partitioning}

The NixOS installer doesn't do any partitioning or formatting, so you
need to do that yourself.

The NixOS installer ships with multiple partitioning tools. The examples
below use `parted`, but also provides `fdisk`, `gdisk`, `cfdisk`, and
`cgdisk`.

The recommended partition scheme differs depending if the computer uses
*Legacy Boot* or *UEFI*.

### UEFI (GPT) {#sec-installation-partitioning-UEFI}

Here\'s an example partition scheme for UEFI, using `/dev/sda` as the
device.

::: {.note}
You can safely ignore `parted`\'s informational message about needing to
update /etc/fstab.
:::

1.  Create a *GPT* partition table.

    ```ShellSession
    # parted /dev/sda -- mklabel gpt
    ```

2.  Add the *root* partition. This will fill the disk except for the end
    part, where the swap will live, and the space left in front (512MiB)
    which will be used by the boot partition.

    ```ShellSession
    # parted /dev/sda -- mkpart primary 512MiB -8GiB
    ```

3.  Next, add a *swap* partition. The size required will vary according
    to needs, here a 8GiB one is created.

    ```ShellSession
    # parted /dev/sda -- mkpart primary linux-swap -8GiB 100%
    ```

    ::: {.note}
    The swap partition size rules are no different than for other Linux
    distributions.
    :::

4.  Finally, the *boot* partition. NixOS by default uses the ESP (EFI
    system partition) as its */boot* partition. It uses the initially
    reserved 512MiB at the start of the disk.

    ```ShellSession
    # parted /dev/sda -- mkpart ESP fat32 1MiB 512MiB
    # parted /dev/sda -- set 3 esp on
    ```

Once complete, you can follow with
[](#sec-installation-partitioning-formatting).

### Legacy Boot (MBR) {#sec-installation-partitioning-MBR}

Here\'s an example partition scheme for Legacy Boot, using `/dev/sda` as
the device.

::: {.note}
You can safely ignore `parted`\'s informational message about needing to
update /etc/fstab.
:::

1.  Create a *MBR* partition table.

    ```ShellSession
    # parted /dev/sda -- mklabel msdos
    ```

2.  Add the *root* partition. This will fill the the disk except for the
    end part, where the swap will live.

    ```ShellSession
    # parted /dev/sda -- mkpart primary 1MiB -8GiB
    ```

3.  Finally, add a *swap* partition. The size required will vary
    according to needs, here a 8GiB one is created.

    ```ShellSession
    # parted /dev/sda -- mkpart primary linux-swap -8GiB 100%
    ```

    ::: {.note}
    The swap partition size rules are no different than for other Linux
    distributions.
    :::

Once complete, you can follow with
[](#sec-installation-partitioning-formatting).

### Formatting {#sec-installation-partitioning-formatting}

Use the following commands:

-   For initialising Ext4 partitions: `mkfs.ext4`. It is recommended
    that you assign a unique symbolic label to the file system using the
    option `-L label`, since this makes the file system configuration
    independent from device changes. For example:

    ```ShellSession
    # mkfs.ext4 -L nixos /dev/sda1
    ```

-   For creating swap partitions: `mkswap`. Again it's recommended to
    assign a label to the swap partition: `-L label`. For example:

    ```ShellSession
    # mkswap -L swap /dev/sda2
    ```

-   **UEFI systems**

    For creating boot partitions: `mkfs.fat`. Again it's recommended
    to assign a label to the boot partition: `-n label`. For
    example:

    ```ShellSession
    # mkfs.fat -F 32 -n boot /dev/sda3
    ```

-   For creating LVM volumes, the LVM commands, e.g., `pvcreate`,
    `vgcreate`, and `lvcreate`.

-   For creating software RAID devices, use `mdadm`.

## Installing {#sec-installation-installing}

1.  Mount the target file system on which NixOS should be installed on
    `/mnt`, e.g.

    ```ShellSession
    # mount /dev/disk/by-label/nixos /mnt
    ```

2.  **UEFI systems**

    Mount the boot file system on `/mnt/boot`, e.g.

    ```ShellSession
    # mkdir -p /mnt/boot
    # mount /dev/disk/by-label/boot /mnt/boot
    ```

3.  If your machine has a limited amount of memory, you may want to
    activate swap devices now (`swapon device`).
    The installer (or rather, the build actions that it
    may spawn) may need quite a bit of RAM, depending on your
    configuration.

    ```ShellSession
    # swapon /dev/sda2
    ```

4.  You now need to create a file `/mnt/etc/nixos/configuration.nix`
    that specifies the intended configuration of the system. This is
    because NixOS has a *declarative* configuration model: you create or
    edit a description of the desired configuration of your system, and
    then NixOS takes care of making it happen. The syntax of the NixOS
    configuration file is described in [](#sec-configuration-syntax),
    while a list of available configuration options appears in
    [](#ch-options). A minimal example is shown in
    [Example: NixOS Configuration](#ex-config).

    The command `nixos-generate-config` can generate an initial
    configuration file for you:

    ```ShellSession
    # nixos-generate-config --root /mnt
    ```

    You should then edit `/mnt/etc/nixos/configuration.nix` to suit your
    needs:

    ```ShellSession
    # nano /mnt/etc/nixos/configuration.nix
    ```

    If you're using the graphical ISO image, other editors may be
    available (such as `vim`). If you have network access, you can also
    install other editors -- for instance, you can install Emacs by
    running `nix-env -f '<nixpkgs>' -iA emacs`.

    BIOS systems

    :   You *must* set the option [](#opt-boot.loader.grub.device) to
        specify on which disk the GRUB boot loader is to be installed.
        Without it, NixOS cannot boot.

    :   If there are other operating systems running on the machine before
        installing NixOS, the [](#opt-boot.loader.grub.useOSProber)
        option can be set to `true` to automatically add them to the grub
        menu.

    UEFI systems

    :   You *must* set the option [](#opt-boot.loader.systemd-boot.enable)
        to `true`. `nixos-generate-config` should do this automatically
        for new configurations when booted in UEFI mode.

        You may want to look at the options starting with
        [`boot.loader.efi`](#opt-boot.loader.efi.canTouchEfiVariables) and
        [`boot.loader.systemd-boot`](#opt-boot.loader.systemd-boot.enable)
        as well.

    If you need to configure networking for your machine the
    configuration options are described in [](#sec-networking). In
    particular, while wifi is supported on the installation image, it is
    not enabled by default in the configuration generated by
    `nixos-generate-config`.

    Another critical option is `fileSystems`, specifying the file
    systems that need to be mounted by NixOS. However, you typically
    don't need to set it yourself, because `nixos-generate-config` sets
    it automatically in `/mnt/etc/nixos/hardware-configuration.nix` from
    your currently mounted file systems. (The configuration file
    `hardware-configuration.nix` is included from `configuration.nix`
    and will be overwritten by future invocations of
    `nixos-generate-config`; thus, you generally should not modify it.)
    Additionally, you may want to look at [Hardware configuration for
    known-hardware](https://github.com/NixOS/nixos-hardware) at this
    point or after installation.

    ::: {.note}
    Depending on your hardware configuration or type of file system, you
    may need to set the option `boot.initrd.kernelModules` to include
    the kernel modules that are necessary for mounting the root file
    system, otherwise the installed system will not be able to boot. (If
    this happens, boot from the installation media again, mount the
    target file system on `/mnt`, fix `/mnt/etc/nixos/configuration.nix`
    and rerun `nixos-install`.) In most cases, `nixos-generate-config`
    will figure out the required modules.
    :::

5.  Do the installation:

    ```ShellSession
    # nixos-install
    ```

    This will install your system based on the configuration you
    provided. If anything fails due to a configuration problem or any
    other issue (such as a network outage while downloading binaries
    from the NixOS binary cache), you can re-run `nixos-install` after
    fixing your `configuration.nix`.

    As the last step, `nixos-install` will ask you to set the password
    for the `root` user, e.g.

    ```plain
    setting root password...
    New password: ***
    Retype new password: ***
    ```

    ::: {.note}
    For unattended installations, it is possible to use
    `nixos-install --no-root-passwd` in order to disable the password
    prompt entirely.
    :::

6.  If everything went well:

    ```ShellSession
    # reboot
    ```

7.  You should now be able to boot into the installed NixOS. The GRUB
    boot menu shows a list of *available configurations* (initially just
    one). Every time you change the NixOS configuration (see [Changing
    Configuration](#sec-changing-config)), a new item is added to the
    menu. This allows you to easily roll back to a previous
    configuration if something goes wrong.

    You should log in and change the `root` password with `passwd`.

    You'll probably want to create some user accounts as well, which can
    be done with `useradd`:

    ```ShellSession
    $ useradd -c 'Eelco Dolstra' -m eelco
    $ passwd eelco
    ```

    You may also want to install some software. This will be covered in
    [](#sec-package-management).

## Installation summary {#sec-installation-summary}

To summarise, [Example: Commands for Installing NixOS on `/dev/sda`](#ex-install-sequence)
shows a typical sequence of commands for installing NixOS on an empty hard
drive (here `/dev/sda`). [Example: NixOS Configuration](#ex-config) shows a
corresponding configuration Nix expression.

::: {#ex-partition-scheme-MBR .example}
::: {.title}
**Example: Example partition schemes for NixOS on `/dev/sda` (MBR)**
:::
```ShellSession
# parted /dev/sda -- mklabel msdos
# parted /dev/sda -- mkpart primary 1MiB -8GiB
# parted /dev/sda -- mkpart primary linux-swap -8GiB 100%
```
:::

::: {#ex-partition-scheme-UEFI .example}
::: {.title}
**Example: Example partition schemes for NixOS on `/dev/sda` (UEFI)**
:::
```ShellSession
# parted /dev/sda -- mklabel gpt
# parted /dev/sda -- mkpart primary 512MiB -8GiB
# parted /dev/sda -- mkpart primary linux-swap -8GiB 100%
# parted /dev/sda -- mkpart ESP fat32 1MiB 512MiB
# parted /dev/sda -- set 3 esp on
```
:::

::: {#ex-install-sequence .example}
::: {.title}
**Example: Commands for Installing NixOS on `/dev/sda`**
:::
With a partitioned disk.

```ShellSession
# mkfs.ext4 -L nixos /dev/sda1
# mkswap -L swap /dev/sda2
# swapon /dev/sda2
# mkfs.fat -F 32 -n boot /dev/sda3        # (for UEFI systems only)
# mount /dev/disk/by-label/nixos /mnt
# mkdir -p /mnt/boot                      # (for UEFI systems only)
# mount /dev/disk/by-label/boot /mnt/boot # (for UEFI systems only)
# nixos-generate-config --root /mnt
# nano /mnt/etc/nixos/configuration.nix
# nixos-install
# reboot
```
:::

::: {#ex-config .example}
::: {.title}
**Example: NixOS Configuration**
:::
```ShellSession
{ config, pkgs, ... }: {
  imports = [
    # Include the results of the hardware scan.
    ./hardware-configuration.nix
  ];

  boot.loader.grub.device = "/dev/sda";   # (for BIOS systems only)
  boot.loader.systemd-boot.enable = true; # (for UEFI systems only)

  # Note: setting fileSystems is generally not
  # necessary, since nixos-generate-config figures them out
  # automatically in hardware-configuration.nix.
  #fileSystems."/".device = "/dev/disk/by-label/nixos";

  # Enable the OpenSSH server.
  services.sshd.enable = true;
}
```
:::

## Additional installation notes {#sec-installation-additional-notes}

```{=docbook}
<xi:include href="installing-usb.section.xml" />
<xi:include href="installing-pxe.section.xml" />
<xi:include href="installing-virtualbox-guest.section.xml" />
<xi:include href="installing-from-other-distro.section.xml" />
<xi:include href="installing-behind-a-proxy.section.xml" />
```

repository: obsidiansystems/nixpkgs

gha_language: Nix

repo_name: obsidiansystems/nixpkgs


text:
# Domoticz-DSCEnvisalink-Plugin
Full version of DSC Envisalink Plugin for Domoticz home automation

Controls a single DSC Alarm (PC1616) Envisalink on your network.

## Key Features

* Creates a Contact device per zone that show Open/Closed status.  These can be changed to 'Motion' devices in the Device Edit page and they will show On/Off (recommend setting an Off Delay otherwise activity is rarely seen in the Web UI)
* Creates an Alert per partition that shows partition state, useful if you don't want to use the Security Panel integration or you have more than one partition.
* Creates a Contact device for each Command Output/Partition combination seen. The DSC only reports activation so an Off Delay must exist for the device to reset).
* Optionally creates a Security Panel device that allows arming and disarming via Domoticz.
* When network connectivity is lost the Domoticz UI will optionally show the device(s) with Red banner

## Example

Example of a single partition with six zones. Zone Front Door is bypassed so the partition is showing Ready but forced arming would be required.  Optional Security Panel integration has been selected, clicking the padlock will bring up the Domoticz alarm keypad to allow Arm Home, Arm Away and Disarm functionality.
Three zones have been converted to PIRs within Domoticz.

![picture](DSCAlarm.jpg)

## Installation

Python version 3.4 or higher required & Domoticz version 3.9446 or greater.

To install:
* Go in your Domoticz directory using a command line and open the plugins directory.
* Run: ```git clone https://github.com/dnpwwo/Domoticz-DSCEnvisalink-Plugin.git```
* Restart Domoticz.

In the web UI, navigate to the Hardware page.  In the hardware dropdown there will be an entry called "DSC Alarm via EnvisaLink 4".

Devices are created in the 'Devices' tab, to use them you need to click the green arrow icon and 'Add' them to Domoticz.

To use the integrated Security Panel this must be configured through the 'Settings' page in Domoticz. In the 'Security Panel' widget:
* Enter a numeric password, I used the same code as my DSC Alarm
* Set the 'Delay' to match the exit delay in the DSC
This will create a 'Domoticz Internal' Security Panel device in the 'Devices' page, just ignore it.

To use the Security Panel functionality, either select 'Security Panel' from the menu system or click the padlock icon on the Security Panel device (which will be in the 'Switches' tab)

For security reasons the DSC Alarm can only be Armed or Disarmed via the Security Panel.

## Updating

To update:
* Go in your Domoticz directory using a command line and open the plugins directory then the Domoticz-DSCEnvisalink-Plugin directory.
* Run: ```git pull```
* Restart Domoticz.

## Configuration

### DSC

DSC must have an Envisalink 3 or 4 attached as an additional keypad.

### Domoticz

| Field | Information|
| ----- | ---------- |
| IP Address | Will handle DNS names and IP V4 addresses (e.g 192.168.xxx.xxx) |
| Port | The port that the Envisalink is listening on. Default 4025. |
| Password | Envisalink password, as used on the device's website on your LAN |
| Max Partitions | The number of partitions you have set up |
| Max Zones | The number of Zones you have defined |
| Integrated Security Panel | If true the a Domoticz security panel device is created for partition 1. |
| Alarm Passcode | A numeric PIN that can disarm the DSC Alarm panel |
| Time Out Lost Devices | When true, the devices in Domoitcz will have a red banner when network connectivity is lost to the Envisalink |
| Debug | Debug logging options |

## Change log

| Version | Information|
| ----- | ---------- |
| 1.0.0 | Initial upload version |
| 1.1.0 | Added Command Output support |
| 2.1.7 | Added integration with Security Panel |
| 2.1.8 | Bugfix: time synchronisation error message |

license: MIT

repository: dnpwwo/Domoticz-DSCEnvisalink-Plugin

branch_name: refs/heads/master

extension: md

gha_language: Python

gha_license_id: MIT

license_type: permissive

repo_name: dnpwwo/Domoticz-DSCEnvisalink-Plugin


text:
# Browser {#concept_64056_zh .concept}

This topic describes the application of browsers.

## Supported browser {#section_dmt_1bm_lfb .section}

-   IE \(\>=10\) and Edge
-   Mainstream Chrome/Firefox/Safari versions
-   Mainstream Android/iOS/WindowsPhone versions

**Note:** For IE, you must introduce the promise library before introducing the SDK:

## Support for browsers of earlier versions { .section}

The OSS SDK perform operations through File APIs so errors may occur in browsers of earlier versions. You can use third-party plug-ins to implement OSS APIs to perform operations, such as upload and download. For detailed examples, see [Overview of direct transfer on Web client](../../../../../reseller.en-US/Best Practices/Direct upload to OSS from Web/Javascript client signature pass-through.md#).

-   Settings
    -   Bucket settings

        To access OSS through browser, you must set the [CORS](../../../../../reseller.en-US/Developer Guide/Buckets/Cross-origin resource sharing.md#) of the bucket as follows:

        -   Set Source to `*`.
        -   Set Method to `GET, POST, PUT, DELETE, HEAD`.
        -   Set Allowed Headers to `*`.
        -   Set Expose Header to `ETag`.
        **Note:** Set the preceding CORS rule to the first option.

        ![](http://static-aliyun-doc.oss-cn-hangzhou.aliyuncs.com/assets/img/22580/155186587813704_en-US.png)

    -   STS settings

        To prevent exposing the AccessKeyId and AccessKeySecret in the webpage, Alibaba Cloud adopts STS for temporary authorization. The authorization server is maintained by users. Only authenticated users are allowed to apply for a temporary permission from the authorization server. For more information on how to set RAM users and STS policies, see Overview. For information about how to establish your own authorization servers, see [Node.js STS AppServer](https://github.com/rockuw/node-sts-app-server).

-   For example:

    Here, we use SDK to develop a webpage app which offers four features:

    -   Upload objects
    -   Upload text
    -   List objects
    -   Download objects
    For complete code, see [oss-in-browser](https://github.com/rockuw/oss-in-browser). The expected result is as follows:

    Let’s take upload objects as an example in the following text:

    1.  Create a webpage

        -   An object choice box to select the object to be uploaded
        -   A textbox to input the name of the object to be uploaded to the OSS.
        -   A button to start the upload.
        -   A progress bar to display the upload progress
        The following code creates four controls. `class="xxx"` serves to apply the [Bootstrap](http://getbootstrap.com/) style and can be ignored here.

        ```language-html
        <div class="form-group">
          <label>Select file</label>
          <input type="file" id="file" />
        </div>
        <div class="form-group">
          <label>Store as</label>
          <input type="text" class="form-control" id="object-key-file" value="object" />
        </div>
        <div class="form-group">
          <input type="button" class="btn btn-primary" id="file-button" value="Upload" />
        </div>
        <div class="form-group">
          <div class="progress">
            <div id="progress-bar"
                 class="progress-bar"
                 role="progressbar"
                 aria-valuenow="0"
                 aria-valuemin="0"
                 aria-valuemax="100" style="min-width: 2em;">
              0%
            </div>
          </div>
        </div>

        ```

    2.  Add a style

        You can add some styles to the webpage to enhance its look. Here, we use [Bootstrap](http://getbootstrap.com/). Add the style in the `<head>` tag as follows:

        ```language-html
        <head>
          <title>OSS in Browser</title>
          <link rel="stylesheet" href="bootstrap.min.css" />
        </head>

        ```

        `bootstrap.min.css` can be downloaded from the official website of Bootstrap.

    3.  Add a script

        Before adding a script, the webpage is static and does not respond to any click. We can add some JavaScript so that it can upload/download/list objects.

        First, introduce the SDK in the `<head>` tag:

        ```language-html
        <head>
          <title>OSS in Browser</title>
          <link rel="stylesheet" href="bootstrap.min.css" />
          <script type="text/javascript" src="http://gosspublic.alicdn.com/aliyun-oss-sdk-x.x.x.min.js"></script>
          <script type="text/javascript" src="app.js"></script>
        </head>

        ```

        `app.js` contains the code that implements the object uploading. The content is as follows:

        ```language-js
        const appServer = 'http://localhost:3000';
        const bucket = 'js-sdk-bucket-sts';
        const region = 'oss-cn-hangzhou';

        const urllib = OSS.urllib;

        const applyTokenDo = function (func) {
          const url = appServer;
          return urllib.request(url, {
            method: 'GET'
          }).then(function (result) {
            const creds = JSON.parse(result.data);
            const client = new OSS({
              region: region,
              accessKeyId: creds.AccessKeyId,
              accessKeySecret: creds.AccessKeySecret,
              stsToken: creds.SecurityToken,
              bucket: bucket
            });

            return func(client);
          });
        };

        let currentCheckpoint;
        const progress = async function progress(p, checkpoint) {
          currentCheckpoint = checkpoint;
          const bar = document.getElementById('progress-bar');
          bar.style.width = `${Math.floor(p * 100)}%`;
          bar.innerHTML = `${Math.floor(p * 100)}%`;
        };

        let uploadFileClient;
        const uploadFile = function (client) {
          if (! uploadFileClient || Object.keys(uploadFileClient).length === 0) {
            uploadFileClient = client;
          }

          const file = document.getElementById('file').files[0];
          const key = document.getElementById('object-key-file').value.trim() || 'object';
          console.log(`${file.name} => ${key}`);

          const options = {
            progress,
            partSize: 100 * 1024,
            meta: {
              year: 2017,
              people: 'test',
            },
          };

          return client.multipartUpload(key, file, options).then( (res) => {
            console.log('upload success: %j', res);
        	currentCheckpoint = null;
            uploadFileClient = null;
          }).catch((err) => {
            if (uploadFileClient && uploadFileClient.isCancel()) {
              console.log('stop-upload!') ;
            } else {
              console.error(err);
            }
          });
        };

        window.onload = function () {
          document.getElementById('file-button').onclick = function () {
            applyTokenDo(uploadFile);
          }
        };

        ```

        The following are the steps to uploading an object:

        1.  Apply for a temporary permission from the authorization server. The authorization server is a service built by the website developer to provide temporary authorization for end users. Developers can provide different permissions for different users during authorization \(see [Overview](../../../../../reseller.en-US/Developer Guide/Hide/Access control/Overview.md#)\). For authorization server information, see [GitHub](https://github.com/rockuw/node-sts-app-server). To make it more convenient, the authorization server in the example returns the temporary credential to the user directly.
        2.  Create an OSS client using a temporary key.
        3.  Upload the selected files through the `multipartUpload` interface and set the progress bar through the `progress` parameter.
        4.  Other features

            For text content uploading, object list retrieval, and object downloading features, see the example code: [OSS in Browser](https://github.com/rockuw/oss-in-browser).

repository: Hello-hao/oss

repo_name: Hello-hao/oss


text:
# ![NDART LOGO](/72x72.png?raw=true)
# NDART
## http://ndart.ca
##
## RNSaveAudio Module [![npm version](https://badge.fury.io/js/rnsaveaudio.svg)](https://badge.fury.io/js/rnsaveaudio)
### React-Native | [Home](README.md) | [Useful Links](UsefulLinks.md) | [Custom Module Tutorial](Tut.md)
###
### Purpose
The purpose of this module is to take an array of int16 data from javascript, and create a .wav file from that data, on Android and iOS devices natively. This module is essentially suppose to provide a sink when working with the `react-native-recording` module (https://www.npmjs.com/package/react-native-recording). More information can be found on the projects respective github homepage page -> https://github.com/qiuxiang/react-native-recording#readme...
### Installation
#### Android
1) To install, run `npm install rnsaveaudio`
2) Edit `/android/settings.gradle` and add the following lines...
``` java
include ':rnsaveaudio'
project(':rnsaveaudio').projectDir = new File(rootProject.projectDir, '../node_modules/rnsaveaudio/android')
```
3) Edit `/android/app/build.gradle` and add the following line inside of `dependencies`...
``` java
compile project(':rnsaveaudio')
```
4) Edit `/android/app/src/main/java/.../MainApplication.java` with the follow lines...
- Import the module at the top of the file
```java
import com.navraj.rnsaveaudio.RNSaveAudioPackage;
```
- Instantiate the package inside of the `getPackages()` method
```java
new RNSaveAudioPackage()
```
5) run the command `cd android && gradlew clean && cd ../` for windows, or `cd android && ./gradlew clean && cd ../` for OSX, inside the root directory of your react-native project
#### iOS
1) Open your xCode project and add the `RNSaveAudio.xcodeproj` file to your project
2) Go to your project's build phases, and add the `libRNSaveAudio.a` file to the list labeled `Link Binary With Libraries`, using the `+` under the list
- If you get an error saying xCode cannot find `<RCTEventEmitter.h>`, open `RNSaveAudio.h` inside of the `ios` project folder, and change the the header to `<React/RCTEventEmitter.h>`
- For more information, look at https://github.com/maxs15/react-native-spinkit/wiki/Manual-linking---IOS
### Notes
- Version 1.0.6 is current
- Works on Android and iOS (Tested)
- Sends a promise when complete
- Creates a .wav file based on an array of signed short (SInt16) values at a frequency of 44.1kHz (mono)
- Meant to work with react-native-recording node package and react-native-fs package
### Useage
- Import the module using `import RNSaveAudio from 'rnsaveaudio';`
- Export audio using `RNsaveAudio.saveWav(PATH+'/filename.wav',dataArray);`
- Make sure to include the keywords `await` and `async` if used inside of a function, eg. `async func(prop){ const variable = await RNsaveAudio.saveWav(PATH+'/filename.wav',dataArray); }`
### Contact Information
- My email is<EMAIL_ADDRESS>use the module name in the subject line.
- Submit an issue to the github page. http://github.com/navrajkambo/RNSaveAudio/issues
### Contribution
- If someone would like to add methods for exporting stereo wav files, or files of other formats and sampling frequencies from signed short arrays, feel free to create a PR, it would be useful to have
### Other Information
- If you feel like contributing, send me a message or open an issue! Any help is appreciated :)
- For more information on bridging native modules to react-native, have a look at https://gist.github.com/chourobin/f83f3b3a6fd2053fad29fff69524f91c, https://eastcodes.com/packaging-and-sharing-react-native-modules, and http://matthewsessions.com/2017/10/27/developing-rn-module.html. They are very informative!

repository: navrajkambo/RNSaveAudio

gha_language: Objective-C

repo_name: navrajkambo/RNSaveAudio


text:
# Traffic Source

## Description

You are given a web log consisting of a list of events for each user.
Each event consists of userId, url visited and timestamp of activity.
Using this, output the traffic source distribution for a given url.

##### Input
###### WebLog: (UserId, URL, Timestamp)
eg
- 1 ABC T0
- 1 XYZ T2
- 1 PQR T1
- 2 DEF T2
- 3 PQR T3

###### TargetUrl:
- PQR

##### Output
###### (TrafficSource, count)
- (ABC,1)
- (DEF,1)

repository: hemantgupta-cs/spark-practice-master

repo_name: hemantgupta-cs/spark-practice-master


text:
---
layout: post
title: Normal Mapping & Parallax Mapping
tags: [normals, normalmapping, parallax, parallaxmapping, technique, opengl]
---

## Normal Mapping

  To be able to add some extra detail to quads or meshes in our scene, in this engine we decided to add normal mapping, this technique in fact adds an extra layer of realism due to lighting being a very important factor in any engine.

  The technique in itself is just a direct interaction with **lighting and the normals of a given mesh** to emphasize on the depth of a surface so it does not look like a flat surface. In essence this technique basically converts the normals from the mesh into **per-fragment** (_per-pixel_) normals, this way we can have little variations on the normals of a mesh and trick the lighting system in the engine to think it has some kind of change in light in specific parts of the mesh otherwise if it was a per-surface it is going to be pretty much the same for each fragment in the screen and it is not going to have any variation, in essence we give the **illusion** that the mesh is way more complex than it appears without adding any extra geometry.

  ![Example Normal Mapping](https://learnopengl.com/img/advanced-lighting/normal_mapping_surfaces.png)

  Giving this micro-level details of normals per pixel, we in essence create an illusion where have for example a flat brick plane to look like it is indented where it needs to be like in the following image:

  ![Brick Wall Example](https://learnopengl.com/img/advanced-lighting/normal_mapping_compare.png)

  This way we avoid doing an **interpolation** between surface normals and instead of that we utilize a specific normal per pixel in the screen, the lighting does the rest of the job to create this effect.

  Now, to be able to introduce in the engine we need to follow the usual **art pipeline**, usually the normals of a geometry are stored in what we know as a **normal map**, this will hold all the per-fragment information for the normals for given geometry and will allow us to add that realism to the geometry.

  This means that we will be loading a normal texture in the engine through our **texture builder** and usually a normal map looks very blue-ish in color with reds and greens in between, indicating that the colors correspond to the axis, `Blue (0, 0, 1)`, `Red (1, 0, 0)` and lastly `Top (0, 1, 0)` this corresponds to what we see in the texture, greens appear in the top-most part of geometry whereas reds tend to appear in the left-right most parts and in general the faces of the geometry are blue because they are pointing to `+Z`.

  ![Brick Normal Map Example](https://learnopengl.com/img/advanced-lighting/normal_mapping_normal_map.png)

  We do not need to forgive that whenever we are processing the texture through the shader we need to normalize in the `[0, 1]` space instead of the space in which they come `[-1, 1]` to be able to utilize them correctly. _Remember that we can represent the information from a normal texture through the **x, y, z** coordinates instead of the usual **RGB** that we do whenever we want to get information from the **diffuse**_.

  This means that to be able to normalize the information from the texture in the given limits we do the following shader code:

```glsl
  vec3 transformed_normal = normal * 0.5 + 0.5; // ( This essentially transforms from [-1,1] to [0, 1] )
```
  This allows us to transform from a normal space coordinates to an RGB coordinate space, essentially allowings us to read it without having too much trouble with random normal artifacts. Now, to be able to utilize the normal map we need to sample it and use them for lighting calculations and this is done the following way:

 ```glsl
 uniform sampler2D normalMap;

 in vec2 uvs;

 void main()
 {
    normal = texture(normalMap, uvs).rgb
    normal = normalize(normal * 2.0 - 1.0) // We convert them back to range [-1, 1]

    // [......] Proceed with the lighting calculations (Pointlight, Directional, Spotlight)

 }

```

  This all will work perfectly if we have surface sof the geometry or a simple flat plane with normal mapping that is pointing to the `+Z` direction, in case we rotate it or make it point in any other direction, normal mapping gets ruined, because the **normals** will still be pointing to the `+Z` coordinates, think of it as normals being absolute towards the `+Z` direction no matter how much you rotate the geometry or mesh, this implies that the lighting is going to be done incorrectly and we have officially broken the technique.

  ![Z Positive Normals Issue](https://learnopengl.com/img/advanced-lighting/normal_mapping_ground_normals.png)

  Luckily we have a solution for this, to be able to keep the normals relative to the geometry / mesh we utilize what we call `tangent space`, this in essence brings the normals to a local space where they are relative to the individual triangles of the geometry, of course always pointing `+Z` but this time in local coordinates, implying that even if we rotate it, it is going to rotate in the **positive Z** coordinate taking into account the rotation of the mesh.

  To be able to transform from **tangent space** to **world space** and viceversa we need to be able to construct what we call a **TBN Matrix** to be able to utilize the inverse and the matrix itself to do the transformations of normals comfortably.

  As we know, the normals in **tangent space** point towards the direction of the surface, this leaves us knowing one piece of information, **the normal**, we are left out with more information that we need to discover, in this case the **bitangent** and the **tangent**.

  ![TBN Coordinates](https://learnopengl.com/img/advanced-lighting/normal_mapping_tbn_vectors.png)

  The issue here is that we do not have any other piece of information other than the normal, it is not like we can magically do a **cross product** and get it done, this means that we will need to do some simple math and it is the following one:

  ![Normal Mapping TBN Math](https://learnopengl.com/img/advanced-lighting/normal_mapping_surface_edges.png)

  You first need to calculate the edges and delta UV coordinates so you can get the desired vectors, to ease out the mathematics we will do a manual approach so it feels more integral, if we had the following positions, normals and texture coordinates:

 ```cpp
  glm::vec3 pos1(-1.0f, 1.0f, 0.0f);
  glm::vec3 pos2(-1.0f, -1.0f, 0.0f);
  glm::vec3 pos3(1.0f, -1.0f, 0.0f);
  glm::vec3 pos4(1.0f, 1.0f, 0.0f);

  // tex coords
  glm::vec2 uv1(0.0f, 1.0f);
  glm::vec2 uv2(0.0f, 0.0f);
  glm::vec2 uv3(1.0f, 0.0f);
  glm::vec2 uv4(1.0f, 1.0f);

 // normal vector +Z
 glm::vec3 nm(0.0f, 0.0f, 1.0f);

 ```

  We need to calculate the delta UV coordinates which correspond to `ΔU2` and `ΔV2` which is doing a substraction to the positions:

 ```cpp
  glm::vec2 deltaUV1 = uv2 - uv1;
  glm::vec2 deltaUV2 = uv3 - uv1;

```

  You can see that `deltaUV1` corresponds to the horizontal `ΔU2` delta and the `deltaUV2` corresponds to the vertical `ΔV2` delta, afterwards the next calculations we will need to do is the edges to be able to calculate the triangle, in this case `E1` and `E2`.

 ```cpp
  glm::vec3 edge1 = pos2 - pos1;
  glm::vec3 edge2 = pos3 - pos1;

```

  Once again we can correlate the horizontal and vertical edges with `E1` (_edge1_) and `E2` (_edge2_) respectively, this way we can see that with the edge vectors we essentially have constructed a triangle, having the delta for UVs and the edges of the triangle we can start calculating the vectors we want which in this case are the **tangents** and **bitangents**.

```cpp
float f = 1.0f / (deltaUV1.x * deltaUV2.y - deltaUV2.x * deltaUV1.y);

tangent1.x = f * (deltaUV2.y * edge1.x - deltaUV1.y * edge2.x);
tangent1.y = f * (deltaUV2.y * edge1.y - deltaUV1.y * edge2.y);
tangent1.z = f * (deltaUV2.y * edge1.z - deltaUV1.y * edge2.z);

bitangent1.x = f * (-deltaUV2.x * edge1.x + deltaUV1.x * edge2.x);
bitangent1.y = f * (-deltaUV2.x * edge1.y + deltaUV1.x * edge2.y);
bitangent1.z = f * (-deltaUV2.x * edge1.z + deltaUV1.x * edge2.z);

```
  With this you would already have all the necessary vectors to calculate the **TBN matrix**.

  {: .box-note}
**Note:** You need to do the same process for the tangent and bitangent calculation for each triangle in your geometry, usually you would incorporate this piece of code in your **object loader** in your engine and store them in a **vector** so you can later **build** the **TBN matrix** and **pass it to the respective shader**.

  You have two options here, you can construct the TBN here in the **CPU** or you can do it in the respective **shader**, in our case we have it built in the shader itself and you would pass the information through vertex attributes to the vertex shader as following:

```glsl

layout (location = 0) in vec4 a_position;
layout (location = 1) in vec4 a_normal;
layout (location = 2) in vec4 a_uvs;
layout (location = 3) in vec4 a_tangent;
layout (location = 4) in vec4 a_bitangent;

void main()
{
  vec3 T = normalize(vec3(u_m_matrix * a_tangent));
  vec3 B = normalize(vec3(u_m_matrix * a_bitangent));
  vec3 N = normalize(vec3(u_m_matrix * a_normal));
  mat3 TBN = mat3(T, B, N);
}

```

  {: .box-note}
**Note:** We do not need to calculate all the three vectors, with only two of the vectors, one of them being the **normals** we can do a **cross product** `vec3 B = cross (N, T)`.

 Now after we have the TBN matrix we can take two approaches, either taking the sampled normal from tangent space to world space using the TBN matrix or the opposite, using the inverse of the TBN to transform any vector to tangent space, the rule that we need to abide by is that **the calculations need to be in the same coordinate space** to avoid any kind of artifacts or weird / unexpected behavior.

 In our specific case in Metro Engine we utilize the second option, we bring all the vectors to tangent space to do all the necessary calculations, this means that we first need to inverse the matrix before sending it to the fragment shader.

```glsl

in mat3 TBN_in;

void main()
{
  mat3 inv_tbn = transpose(TBN_in);

}

```

 Of course, we get the TBN from the vertex shader as input into the fragment shader to later transpose it to be able to transform any vector in the fragment shader to tangent space for the lighting calculations.

{: .box-note}
  **Note:** The transpose of an orthogonal matrix equals the inverse of it.

  And now we need to transform the necessary vectors `lighDir` and `viewDir` to tangent space.

```glsl

in mat3 TBN_in;
in vec2 in_uvs;

void main()
{
  mat3 inv_tbn = transpose(TBN_in);
  vec3 normal = texture(normalMap, in_uvs).rgb;
  normal = normalize(normal * 2.0 - 1.0);

  vec3 lightDir = inv_tbn * normalize(lightPos - fragPos);
  vec3 viewDir = inv_tbn * normalize(viewPos - fragPos);
  // ...
}

```

  And with this, we can calculate all the **lighting** variables in tangent space given that the normal is in tangent space too, it is a perfect fit for normal mapping to shine and show the technique fully working without caring about the rotation of the geometry/mesh.

![Rotated Plane with Normal Mapping](https://learnopengl.com/img/advanced-lighting/normal_mapping_correct_tangent.png)

  We extracted this technique from **[LearnOpenGL](https://learnopengl.com/Advanced-Lighting/Normal-Mapping)** and all the images and mathematics done here are extracted from this website.

## Parallax Mapping

As previously mentioned, normal mapping was a technique that added extra realism to a flat surface, using the **lighting** system we gave the illusion that there is depth. This does not only stop there, we have another technique called "**Parallax Mapping**" and it is the next step after doing normal mapping, it is specifically related to the **displacement mapping** family and it is also used to give an illusion or effect that there is a given displacement in a flat surface through **height maps**.

 Usually you would utilize a plane with `1000` vertices and use a height map to tesselate and displace each one of the vertex to add that extra depth and make a geometry even more realistic regardless of the lighting.

 We can notice that this can get out of control pretty easily, given that each geometry needs to have around `1000` vertices that will be displaced just to make it look more realistic, it quickly turns out horrendous for performance because it is expected that there is **not** going to be only that geometry, there are a lot more things running in an engine in a fully fleshed scene, implying that we need to go for an optimized approach to avoid reduction in performance.

 That is why this technique is here, instead of using actual **geometry vertices** to displace and create depth for our geometries, we will be utilizing the texture alongside the **height map** to create the illusion that there is depth in a flat surface with little to no geometry / vertices like for example a quad that consists of two triangles.

 This is what a normal height map would look like:

 ![Height Map Example](https://learnopengl.com/img/advanced-lighting/parallax_mapping_height_map.png)

  Now that we have the information about the height at which the geometry shouldbe displaced, we need to alter the texture coordinates of the quad in such a way to make it look like there is depth, of course we need to take into consideration the **camera** formerly known as the **view direction** to be able to do the calculations correctly like in the following image:

![Example Displacement Texture](https://user-images.githubusercontent.com/48097484/118874719-80a42b00-b8eb-11eb-8484-485949e62621.png)

  We calculate the vector `V` that ends up colliding with the plane in the point `A` and if we had the height map projected onto the quad, given the specific **direction** we would be colliding in point `B`.

  Now, you might be thinking, we have the `viewDir` and the point `A` from which we get the `viewDir`, how do we calculate the point B, because we do not have an apparent way to calculate this point B to create an effect of displacement, we should be technically drawing the pixel which corresponds to the vertical of point **B**.

  To be able to calculate this point we need to do a **texture coordinate offset** it is as simple as offseting the UVs with which we are reading with a small offset that takes into consideration the **height map** values what we have available to us.

  As we have point `A` we can know the projected value onto the height map which in this case is `H(A)`, after we get that projected point and its length, we can project it onto the `viewDir` and get an **approximation** on the desired point `B`, the following image shows how its done:

![Texture Coordinate Offset Example](https://user-images.githubusercontent.com/48097484/118876205-476cba80-b8ed-11eb-8a7e-39affa44a206.png)

  Essentially what we are doing is scaling the vector `V` by the height at the fragment position `H(A)` so we can get the approximated point `H(P)` because it is scaled onto the `viewDir` which is essentially close enough to the point `B`.

  Although this is the rough **approximation**, it is still very crude and in some edge cases of displacement we might end up **overpassing** the desired point and project an undesired height and we might end up with artifacts. This next image exemplifies it:

  ![Crude Height Map Projection Case](https://user-images.githubusercontent.com/48097484/118876960-2789c680-b8ee-11eb-9a4a-576bc569227a.png)

  As we can see the projected point `H(P)` in the texture plane is overpassing the desired height `B`, proving the point that we can end up with an undesired effect depending on the height map. Another **edge case** is whenever the plane is **rotated**, this practically causes the `viewDir` to not be correct and relative to the forward of the plane to be able to project the height correctly.

### Correct Way to do Parallax Mapping

  To be able to do this technique correctly and avoiding the previously mentioned edge case of rotation and sometimes the height map doing a weird **overpass** we need to do two things.

  - To be able to fix the overpass that happens depending on the height map, we need to utilize the **inverse** of the height map, it is in general easier to fake depth than doing it the other way around and the example looks like the following now:

![Inverted Height Map](https://user-images.githubusercontent.com/48097484/118878819-4721ee80-b8f0-11eb-837b-ec598ed59fd1.png)

  We can see that we still have a point `A` and desired point `B` with a direction vector `P`, this time for us to be able to calculate the **texture offset** we just **substract** the vector `V` in this case it is the `viewDir` from the camera to the fragment with the point `A`, this means that the calculations are even simpler because this time we are not obtaining a **height** value, we are doing it the other way around, we are obtaining **depth** values and this is done by substracting the sampled heightmap value from `1.0f` and on or by simply **inverting** the texture values in an image-editing software such as `GIMP`, `PHOTOSHOP`, etc.

  Now to be able to fix the other issue we were talking about, the **rotation**, we need to be able to convert the `viewDir` to a local coordinate space that takes into consideration the rotation of the geometry and normals, doesn't this sound **familiar** to you ?

  (_I mean, you essentially read about normal mapping like just a tad bit ago, you should know the answer..._)

  Yeah, of course! **Tangent Space**! Correct, we need to also utilize the **bitangent** and **tangent** to be able to bring the `viewDir` to tangent space to be able to do the calculations in the same coordinate space and so they are relative to the given rotation, just like normal mapping but this time without doing any lighting calculations in between!

  As far as we can remember, we did pass the **TBN Matrix** to the **fragment shader**, so we already have a template from which we can expand and work on parallax mapping to make the geometry look even more **realistic** and appealing! Given the following fragment shader code we will be doing the following:

  ```glsl

in mat3 TBN_in;
in vec2 in_uvs;

// NEW FUNCTION!
vec2 ParallaxMapping(vec2 texCoords, vec3 viewDir)
{
  float height = texture(heightMap, in_uvs).r;
  vec2 p = viewDir.xy / viewDir.z * (height * height_scale);
  return in_uvs - p; // Remember the texture offset we were talking about?
}

void main()
{
  mat3 inv_tbn = transpose(TBN_in);
  vec3 normal = texture(normalMap, in_uvs).rgb;
  normal = normalize(normal * 2.0 - 1.0);

  vec3 lightDir = inv_tbn * normalize(lightPos - fragPos);
  vec3 viewDir = inv_tbn * normalize(viewPos - fragPos);
  // ...
}

```
  Some stuff needs to be explained here in the previous newly added function, we can see that we are clearly **sampling** from the height map texture that we have previously mentioned, considering that it is the **inverted** version of the height map. Afterwards we do what we have previosly have talked about, given the tangent space coordinates of the converted viewDir, we take the **x** and **y** components and then divide them by its own **z** component to later scale it by `H(A)` which from the previous drawings we know is the projected height map onto the displacement map.

  Lastly we introduce a new variable called **height_scale** which essentially servers as a control variable for the parallax effect to tone up/down the height of the effect we are trying to achieve. With a `height_scale` of `0.1f` we achieve an effect like the following for parallax mapping:

  ![image](https://user-images.githubusercontent.com/48097484/118885262-d67ed000-b8f7-11eb-9a60-05425c46bb51.png)

  As you can observe in the image in the right-most part we have the combined effect of normal mapping + parallax mapping, but you might notice that it is not perfectly placed, we need to trim the edges a bit so it has a perfect quad shape, this often happens because we **oversample** the texture given the **displacement map** so we need to constraint it in the `[0, 1]` range and it is done the following way:

```glsl
in_uvs = ParallaxMapping(in_uvs, viewDir);
if (in_uvs.x > 1.0 || in_uvs.y > 1.0 || in_uvs.x < 0.0 || in_uvs.y < 0.0)
  discard;
```

### Stepped Parallax Mapping

  To be able to do a more precise calculation on the previous parallax mapping technique we can do what we call a **steep parallax mapping** so we can approach the `P` value towards the desired point `B`  we already know.

  Instead of doing only one calculation of `P` we take various samples of the same point towards the direction `V` which is the `viewDir` towards depth, because remember, we are utilizing an **inverted height map** and we are now speaking in terms of depth instead of height, essentially you do a "_linear interpolation_" on the vector `P` until the sampled depth value is less than the depth value of the current layer, becuase we cut the depth in different depth heights like in the image shown here:

  ![Split depth for stepped parallax mapping](https://user-images.githubusercontent.com/48097484/118889547-899df800-b8fd-11eb-8a81-31de3f74f3dc.png)

  As you can see, the depth is split in interpolations of 0.2 and what we basically do is traverse from top to bottom depth-wise for each layer we compare the **sampled depth value** to the depth value stored in the **depthmap**, if it is less  than the current depth map value, it means that the layer's segment of the `P` vector is not below the surface, this process is continued until you reach a point as previously mentioned where the sampled depth is less than the value of the current layer.

  To implement this technique we only have to change a few things in the `ParallaxMapping` function that we created in our last **fragment shader** because we have all the information needed, we just need to make the code a bit more complex.

```glsl
vec2 ParallaxMapping(vec2 texCoords)
{
	//Clculate tangent space coords
	vec3 tangentSpaceCameraPos = transpose(TBN) * u_camerapos.xyz;
	vec3 tangentSpaceFragPos = transpose(TBN) * worldSpacePosition.xyz;
	vec3 viewDir =  normalize(tangentSpaceCameraPos - tangentSpaceFragPos); // In this case we opted to move the calculation of the tangent space view dir inside the function

    const float minLayers = 8.0;
	const float maxLayers = 32.0;
	float numLayers = mix(minLayers, maxLayers, max(dot(vec3(0.0, 0.0, 1.0), viewDir), 0.0));
	// calculate the size of each layer
    float layerDepth = 1.0 / numLayers;
    // depth of current layer
    float currentLayerDepth = 0.0;
    // the amount to shift the texture coordinates per layer (from vector P)
    vec2 P = viewDir.xy * u_parallax_height_multiplier;
    vec2 deltaTexCoords = P / numLayers;

	// get initial values
	vec2  currentTexCoords     = texCoords;
	float currentDepthMapValue = 1 - texture(u_texture_height, currentTexCoords).r;

	while(currentLayerDepth < currentDepthMapValue)
	{
		// shift texture coordinates along direction of P
		currentTexCoords -= deltaTexCoords;
		// get depthmap value at current texture coordinates
		currentDepthMapValue = 1 - texture(u_texture_height, currentTexCoords).r;
		// get depth of next layer
		currentLayerDepth += layerDepth;
	}

	return currentTexCoords;

}

```
  We should get a similar effect to something like this:

![Stepped Parallax Mapping](https://user-images.githubusercontent.com/48097484/118890267-af77cc80-b8fe-11eb-8f11-97998e29908b.png)

 {: .box-note}
**Note:** Notice how the actual samples of the texture start repeating until they reach the specified goal and then they stop, this gives a false illusion of **depth**.

  This technique is the preferred approach whenever you want to improve the realism of your objects without adding extra geometry and we recommend it, you learn a lot about **tangent space** and how you can bring vectors to tangent space and world space back and forth to do the necessary calculations, it is a worth the try.

  Once again, all the resources and images were taken from [LearnOpenGL](https://learnopengl.com/Advanced-Lighting/Parallax-Mapping) thanks to this website we were able to explain the technique in an easier manner and hopefully this was helpful!


text:
TVCaption
===
PyTorch implementation of MultiModal Transformer (MMT), a method for multimodal (video + subtitle) captioning.

[TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://arxiv.org/abs/2001.09099)

[Jie Lei](http://www.cs.unc.edu/~jielei/), [Licheng Yu](http://www.cs.unc.edu/~licheng/),
[Tamara L. Berg](http://tamaraberg.com/), [Mohit Bansal](http://www.cs.unc.edu/~mbansal/)

### TVC Dataset and Task

We extended TVR by collecting extra captions
for each annotated moment. This dataset, named TV show Captions (TVC),
is a large-scale multimodal video captioning dataset,
contains 262K captions paired with 108K moments.
We show our annotated captions and model generated captions below.
Similar to TVR, the TVC task requires systems to gather information
from both video and subtitle to generate relevant descriptions.
![tvc example](./imgs/caption_prediction.jpg)

### Method: MultiModal Transformer (MMT)

we designed a MultiModal Transformer (MMT) captioning model which
follows the classical encoder-decoder transformer architecture. It takes both
video and subtitle as encoder inputs to generate the captions from the decoder.

## Resources
- Data: [TVC dataset](./data/)
- Website (with leaderboard): [https://tvr.cs.unc.edu/tvc.html](https://tvr.cs.unc.edu/tvc.html)
- Submission: [codalab evaluation server](https://competitions.codalab.org/competitions/23109)
- Related works: [TVR (Moment Retrieval)](https://github.com/jayleicn/TVRetrieval), [MART (Video Paragraph Captioning)](https://github.com/jayleicn/recurrent-transformer), [TVQA (Localized VideoQA)](https://github.com/jayleicn/TVQA), [TVQA+ (Grounded VideoQA)](https://github.com/jayleicn/TVQAplus)

## Getting started
### Prerequisites
0. Clone this repository
```
git clone --recursive https://github.com/jayleicn/TVCaption.git
cd TVCaption
```

1. Prepare feature files
Download [tvc_feature_release.tar.gz ](https://drive.google.com/file/d/1bSjxbKSxp1qEBCSwAmk8YlkRl1ztgrWO/view?usp=sharing) (23GB).
After downloading the file, extract it to the `data` directory.
```
tar -xf path/to/tvc_feature_release.tar.gz -C data
```
You should be able to see `video_feature` under `data/tvc_feature_release` directory.
It contains video features (ResNet, I3D, ResNet+I3D), these features are the same as the video features
we used for [TVR/XML](https://github.com/jayleicn/TVRetrieval).
Read the code to learn details on how the features are extracted:
[video feature extraction](https://github.com/jayleicn/TVRetrieval/tree/master/utils/video_feature).

2. Install dependencies:
- Python 2.7
- PyTorch 1.1.0
- nltk
- easydict
- tqdm
- h5py
- tensorboardX

3. Add project root to `PYTHONPATH`
```
source setup.sh
```
Note that you need to do this each time you start a new session.

### Training and Inference

1. Build Vocabulary
```
bash baselines/transformer_captioning/scripts/build_vocab.sh
```
Running this command will build vocabulary `cache/tvc_word2idx.json` from TVC train set.

2. MMT training
```
bash baselines/multimodal_transformer/scripts/train.sh CTX_MODE VID_FEAT_TYPE
```
`CTX_MODE` refers to the context (video, sub, video_sub) we use.
`VID_FEAT_TYPE` video feature type (resnet, i3d, resnet_i3d).

Below is an example of training MMT with both video and subtitle, where we use
the concatenation of ResNet and I3D features for video.
```
bash baselines/multimodal_transformer/scripts/train.sh video_sub resnet_i3d
```
This code will load all the data (~30GB) into RAM to speed up training,
use `--no_core_driver` to disable this behavior.

Training using the above config will stop at around epoch 22, around 3 hours with a single 2080Ti GPU.
You should get ~45.0 CIDEr-D and ~10.5 BLEU@4 scores on val split.
The resulting model and config will be saved at a dir: `baselines/multimodal_transformer/results/video_sub-res-*`

3. MMT inference
After training, you can inference using the saved model on val or test_public split:
```
bash baselines/multimodal_transformer/scripts/translate.sh MODEL_DIR_NAME SPLIT_NAME
```
`MODEL_DIR_NAME` is the name of the dir containing the saved model,
e.g., `video_sub-res-*`.  `SPLIT_NAME` could be `val` or `test_public`.

### Evaluation and Submission

We only release ground-truth for train and val splits, to get results on test-public split,
please submit your results follow the instructions here:
[standalone_eval/README.md](standalone_eval/README.md)

## Citations
If you find this code useful for your research, please cite our paper:
```
@inproceedings{lei2020tvr,
  title={TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval},
  author={Lei, Jie and Yu, Licheng and Berg, Tamara L and Bansal, Mohit},
  booktitle={ECCV},
  year={2020}
}
```

## Acknowledgement
This research is supported by grants and awards from NSF, DARPA, ARO and Google.

This code borrowed components from the following projects:
[recurrent-transformer](https://github.com/jayleicn/recurrent-transformer),
[OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py),
[transformers](https://github.com/huggingface/transformers),
[coco-caption](https://github.com/tylin/coco-caption),
we thank the authors for open-sourcing these great projects!

## Contact
jielei [at] cs.unc.edu

repository: jayleicn/TVCaption

repo_name: jayleicn/TVCaption


text:
Description
===========

Installs pngquant.

Requirements
============

## Platform:

Tested on:

* Ubuntu 14.04

Should also function on RHEL but not tested

Usage
=====

  include_recipe "pngquant"

repository: vgadreau/demo

repo_name: vgadreau/demo


text:
# A minimal elastic agent example
In this example, we show how to use the PyTorch Elastic Trainer launcher to start a distributed application in an elastic and fault tolerant manner. The application is intentionally kept "bare bones" since the objective is to show how to create a `torch.distributed.ProcessGroup` instance. Once a `ProcessGroup` is created, you can use any functionality needed from the `torch.distributed` package.

This application can be run on practically any machine that supports Docker containers and does not require installing additional software or modifying your existing Python environment.

> The `docker-compose.yml` file is based on the example provided with the [Bitnami ETCD container image](https://hub.docker.com/r/bitnami/etcd/).

## Prerequisites
We assume you have a recent version of Docker (version 18.03 or above) and Docker Compose installed on your machine. Verify the version by running
```
docker --version
```
and
```
docker-compose --version
```
which should print something like
```
Docker version 19.03.8, build afacb8b
```
and
```
docker-compose version 1.25.4, build 8d51620a
```
respectively.
## Obtaining the example repo
Clone the PyTorch Elastic Trainer Git repo using
```
git clone https://github.com/pytorch/elastic.git
```
make an environment variable that points to the elastic repo, e.g.
```
export TORCHELASTIC_HOME=~/elastic
```

# Building the samples Docker container
While you can run the rest of this example using a pre-built Docker image, you can also build one for yourself. This is especially useful if you would like to customize the image. To build the image, run:
```
cd $TORCHELASTIC_HOME && docker build -t hello_elastic:dev .
```

# Running an existing sample
This example uses `docker-compose` to run two containers: one for the ETCD service and one for the sample application itself. Docker compose takes care of all aspects of establishing the network interfaces so the application container can communicate with the ETCD container.

To start the example, run
```
cd $TORCHELASTIC_HOME/examples/multi_container && docker-compose up
```
You should see two sets of outputs, one from ETCD starting up and one from the application itself. The output from the application looks something like this:

```
example_1      | INFO 2020-04-03 17:36:31,582 Etcd machines: ['http://etcd-server:2379']
example_1      | *****************************************
example_1      | Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
example_1      | *****************************************
example_1      | INFO 2020-04-03 17:36:31,922 Attempting to join next rendezvous
example_1      | INFO 2020-04-03 17:36:31,929 New rendezvous state created: {'status': 'joinable', 'version': '1', 'participants': []}
example_1      | INFO 2020-04-03 17:36:32,032 Joined rendezvous version 1 as rank 0. Full state: {'status': 'frozen', 'version': '1', 'participants': [0], 'keep_alives': []}
example_1      | INFO 2020-04-03 17:36:32,032 Waiting for remaining peers.
example_1      | INFO 2020-04-03 17:36:32,033 All peers arrived. Confirming membership.
example_1      | INFO 2020-04-03 17:36:32,116 Waiting for confirmations from all peers.
example_1      | INFO 2020-04-03 17:36:32,118 Rendezvous version 1 is complete. Final state: {'status': 'final', 'version': '1', 'participants': [0], 'keep_alives': ['/torchelastic/p2p/run_None/rdzv/v_1/rank_0'], 'num_workers_waiting': 0}
example_1      | INFO 2020-04-03 17:36:32,118 Creating EtcdStore as the c10d::Store implementation
example_1      | ======================================================
example_1      | Environment variables set by the agent on PID 51:
example_1      | {'GROUP_RANK': '0',
example_1      |  'LOCAL_RANK': '1',
example_1      |  'MASTER_ADDR': '6002aeb7c496',
example_1      |  'MASTER_PORT': '46289',
example_1      |  'RANK': '1',
example_1      |  'TORCHELASTIC_MAX_RESTARTS': '100',
example_1      |  'TORCHELASTIC_RESTART_COUNT': '0',
example_1      |  'WORLD_SIZE': '2'}
example_1      | ======================================================
example_1      |
example_1      | ======================================================
example_1      | Environment variables set by the agent on PID 52:
example_1      | {'GROUP_RANK': '0',
example_1      |  'LOCAL_RANK': '0',
example_1      |  'MASTER_ADDR': '6002aeb7c496',
example_1      |  'MASTER_PORT': '46289',
example_1      |  'RANK': '0',
example_1      |  'TORCHELASTIC_MAX_RESTARTS': '100',
example_1      |  'TORCHELASTIC_RESTART_COUNT': '0',
example_1      |  'WORLD_SIZE': '2'}
example_1      | ======================================================
example_1      |
example_1      | On PID 51, after init process group, rank=1, world_size = 2
example_1      |
example_1      | On PID 52, after init process group, rank=0, world_size = 2
example_1      |
lib_example_1 exited with code 0
```
As you can see above, the application starts a process group with two peers and all information needed to initialize the group is provided by the elastic agent via environment variables.

## Customizing parameters
The application Docker container can run an arbitrary user-provided script and run . To do this, simply mount the path containing your script into the application container by creating a volume mount in the `docker-compose.yaml` file as shown below:

```yaml
  example:
    image: 'hello_elastic:dev'
    volumes:
       - /path/to/your/app:/workspace
    command: --nnode=1 --nproc_per_node=2 --rdzv_endpoint=etcd-server /workspace/your_app.py
    networks:
      - app-tier
```
Note that the path to your application (`/path/to/your/app` in the example above) must be an absolute path to the directory containing your script (`your_app.py` above)

## Conclusions
In this simple example, we illustrated the following principles when using PyTorch Elastic Trainer:
1. How to launch a PyTorch distributed application using the elastic launcher.
2. How to obtain parameters such as the world size, local rank and the master URL within an application to establish the process group.
3. How to configure parameters for an elastic job such as the number of workers per node and the number of times your application should be restarted in the event of failures.

In the next set of samples, we will cover more advanced topics such as checkpointing state in your application and deploying it to an orchestrator such as Kubernetes.

repository: pytorch/elastic

repo_name: pytorch/elastic


text:
# OpenMediaLegalHack

> BLS Node

Brooklyn Law School Legal Hackers Chapter Node of the distributed OpenMedia Legal Hack Event Series October 2018.

We look forward to co-hosting the NY Node at the Bushwick Generator on October 20, 2018. We invite programmers, lawyers, students, artists, and thinkers to join in this international initiative. We will explore and develop creative solutions for the ownership, protection, distribution, and exchange of property rights to data and digital assets from artists and creators of artistic mediums and the music and media industries.

Some of the pressing issues that we would like our participants to engage with include: limiting the copy of a creative release, preventing a copy of creative content from being reproduced (and other DRM topics), developing creative accounting tools, projects inspired by band-in-a-box, creative solutions for trademark protection, trademark and copyright using VR, and more to come.

If you believe that you can offer topics/legal issues that musicians and content creators face on a digital landscape that you would like to be addressed at the event – please join our Telegram channel and share those perspectives with the team: https://t.me/joinchat/HT4a1hCyAtquAMVYBvj_1g

https://blslegalhackers.github.io/OpenMediaLegalHack/

repository: BLSLegalHackers/OpenMediaLegalHack

repo_name: BLSLegalHackers/OpenMediaLegalHack


text:
MSF-Dashboard v2.3
====================

The latest released version of MSF's Surveillance Dashboard with data management tools integrated. Fully configurable by the user.

|Description                                   |Size    |Links                                                 |
|----------------------------------------------|--------|------------------------------------------------------|
|Blank template of the dashboard that can be fully configured |56.2 MB| [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/2.3/MSF_Surveillance_Dashboard_v2.3_blank.7z)|
|NERU configuration (with demo data) |56.8 MB| [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/2.3/MSF_Surveillance_Dashboard_v2.3_NERU.7z)|
|CERU configuration (with demo data) |60.5 MB| [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/2.3/MSF_Surveillance_Dashboard_v2.3_CERU.7z)|
|User Manual v2.3 (English)|2.59 MB| [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/2.3/MSF_Surveillance_Dashboard_v2.3_User_Manual_EN.pdf)|
|User Manual v2.3 (French)|2.59 MB| [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/2.3/MSF.Surveillance.Dashboard.v2.3.-.Guide.Utilisateur.FR.20210224.pdf)|

MSF-Dashboard v1.3
====================

A lightweight and modular data visualization micro-website that can be offlined and adapted to various data-feeds.

A new release (v1.3) is now ready and is available for [download](https://github.com/MSF-UK/MSF-Dashboard/releases/tag/v1.3). This includes various examples of configuration files.

Want to contribute to MSF-Dashboard? Please read `CONTRIBUTING.md`.

Sample Dashboards
------------------

Here are some example use cases of the MSF-Dashboard, including online demo versions. If you are interested, they can be a good start to help you generate and customize your own MSF-Dashboard.

|Description                                           |Links                                                 |
|------------------------------------------------------|------------------------------------------------------|
|**Surveillance: MSF DRC KERE** (Oct 2018) - offline, data parsed via excel spreadsheets|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_eru-drc/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.3.1/drc_kere_dashboard_v1.3_demo_data.zip)|
|**Surveillance: MSF Nigeria NERU** (Jul 2018) - offline, data parsed via dedicated datamanager|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_eru-nigeria/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.3/neru_dashboard_datamanager_v1.3_fake_data.zip)|
|**Surveillance: MSF Chad CERU** (updated Jul 2018) - offline, data parsed via dedicated datamanager|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_eru-chad/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.3/ceru_dashboard_datamanager_v1.3_fake_data.zip)|
|**Outbreak: MSF Lubumbashi Mission** (December 2015) - offline, data parsed via tsv|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_msf-lubumbashi) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.0.0_repackaged/msf-lubumbashi_v1.0.0.zip)|
|**Surveillance: MSF Katanga ERU** (updated May 2017) - offline, data parsed via excel-spreadsheets|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_eru-katanga/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.1/eru-katanga_v1.1.zip)|
|**Surveillance: SRE MOH Tonkolili DHMT** (updated May 2017) - offline, data parsed via dedicated datamanager|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_dhmt-tonkolili/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.1/dhmt-tonkolili_v1.1.zip)|
|**Surveillance: MSF Sud-Kivu ERU** (updated May 2017) - offline, data parsed via excel-spreadsheets|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_eru-sudkivu/) / [download](https://github.com/MSF-UK/MSF-Dashboard/releases/download/v1.1/eru-sudkivu_v1.1.zip)|
|**Outbreak: MSF Bentiu Mission** (November 2016) - offline, data parsed via csv|[online-demo](http://MSF-UK.github.io/MSF-Dashboard/ver_demos/cfg_msf-bentiu)|
|More to come...| |

Installation and Usage
----------------------

The MSF Dashboard can be deployed either on- or off-line. As an online tool it is a micro-website, requiring only a web browser (Chrome, Firefox or equivalent; IE support is not guaranteed). The offline version uses nwjs, which includes a node instance and allows for more advanced file manipulation.

The technical side of a MSF-Dashboard deployment is generally a 4-step process:
1) Establish the parsing and data formatting workflow
2) Configure the dashboard
3) Deploy the dashboard
4) Support the end users

The main constraints for installation are the connection with the data source and specific data visualization and interaction needs that are not yet implemented.

Please get in touch [here](https://github.com/MSF-UK/MSF-Dashboard/issues) with any issues or suggestions.

<!-- **Ready to try? You can start with the ERU versions:**
+ Download the release including configuration files for the Sud-Kivu ERU here -> [X](https://github.com/MSF-UK/MSF-Dashboard/releases/tag/v1.1_eru-sudkivu)
+ Here is one page guide for quick deployement (included in the release) -> [X](https://github.com/MSF-UK/MSF-Dashboard/blob/master/cfg_eru-sudkivu/doc-dev/eru_dashboard-deployement_onepage.pdf)
+ Here is a simple user guide (included in the release) -> [X](https://github.com/MSF-UK/MSF-Dashboard/blob/master/cfg_eru-sudkivu/doc-user/eru_doc-user_draft.pdf) -->

Customize the Dashboard: Get started!
-------------------------------------

The Dashboard relies on two types of files:
+ source files */src_core/* : These include both third-party libraries (in */src_core/lib/*) and libraries developed specifically for this MSF-Dashboard (other files in */src_core/*). These files should not be changed for a 'simple' customization ('simple' relates to changes in the layout or text, and the reuse of existing charts and functions).
+ configuration files */cfg_[name-your-custom-version-here]/* : There are 3 configuration files: *index.html* for the layout, *lang/module_lang.js* for the texts and their translations, and *dev/dev-defined.js* for defining specific parameters relating to the data, the data files (geojson, population and medical data), and the charts to be implemented.

Current documentation status:
+ A complete documentation of [dev/dev-defined.js](http://msf-uk.github.io/MSF-Dashboard/doc_getting-started/docs/dev-defined.html) for v1.0.0 is available.
+ The documentation for *index.html* (eg. [index.html](https://github.com/MSF-UK/MSF-Dashboard/blob/master/cfg_msf-bentiu/index.html)) for the layout and *lang/module_lang.js* (eg. [lang/module_lang.js](https://github.com/MSF-UK/MSF-Dashboard/blob/master/cfg_msf-bentiu/lang/module-lang.js)) for the texts and their translations, are not yet available.
+ Developer documentation for MSF-Dashboard v0.9.9 is available [here](https://msf-uk.github.io/MSF-Dashboard_doc/). It is currently being updated for v1.1.

Please get in touch [here](https://github.com/MSF-UK/MSF-Dashboard/issues) if you have any problems or suggestions.

General Architecture
--------------------

```
index.html                                - Defines the layout of the dashboard (charts position and
|                                           identifiers).
|                                           @see: Bootstrap grid system: http://getbootstrap.com/css/#grid.
|
└─ js/main-getlibs.js                     - Loads third-party ibrairies and dashboard sources.
   |                                        @see: Folders are defined in 'index.html'.
   |
   ├─ dev/dev-defined.js                  - Parameters to get and check medical and geometry data, define
   |                                        charts and maps.
   |                                        @see: All the defined parameters are global and members of 'module:g'.
   |
   ├─ lang/module-lang.js                 - Stores texts visible to the end user in available languages.
   |                                        @see: 'module:module_lang'
   |
   ├- js/module-chartwarper.js            - [Optional] Warp charts/containers under tabs.
   |                                        @see 'module:module_chartwarper'
   |
   └─ js/main-getdata.js                  - Loads datafiles.
      |                                     @see: 'module:module_getdata'
      |
      └── js/main-loadfiles.js            - Combines functions of: requesting data from sources, reading
          |                                 and formatting data, connecting with other components.
          |                                 @see: 'module:main_loadfiles'
          |
          ├── js/module-datacheck.js      - Medical data quality check and processing (runs through all
          |                                 the medical records).
          |                                 @see: 'module:module_datacheck'
          |
          └── js/main-core.js             - Central component: setup and pulls the data in the charts and
              |                             maps. Define the interactions. Connects with other components.
              |                             @see 'module:main_core'
              |
              ├── js/module-epitime.js    - Calculates and processes all time-related data based on
              |                             epidemiological weeks.
              |                             @see 'module:module_epitime'
              |
              ├── js/module-population.js - Processes all population-related data.
              |
              |                             @see 'module:module_population'
              |
              ├── js/module-interface.js  - A lateral menu and exta functionalities buttons for charts/maps.
              |   |                         @see 'module:module_interface'
              |   |
              |   └-- js/module-intro.js  - [Optional] intro.js to access quick interactive help: setup.
              |                             @see 'module:module_intro'
              |
              ├── js/module-datatable.js  - [Optional] datatables.js to display data: setup, display and
              |                             interactions.
              |                             @see 'module:module_datatable'
              |
              ├── js/module-multiadm.js   - [Optional] Maps warper (tabs) and 'Goto' dropdown lists: setup,
              |                             display and interactions.
              |                             @see 'module:module_multiadm'
              |
              ├── js/module-colorscale.js - [Optional] Provide functions to deal with maps colorscale
                                            (automatic adaptation, diffrent computing methods - geostats.js -
                                            or color tones...).
                                            @see 'module:module_colorscale'

```

For More Information
--------------------

+ Developer Documentation is available at [Contribute to MSF-Dashboard](https://MSF-UK.github.io/MSF-Dashboard_doc) but is not not complete for v1.1 yet, bear with us!
+ Report issues or ask for help  at [Issues MSF-Dashboard](https://github.com/MSF-UK/MSF-Dashboard/issues).

License
-------

MSF-Dashboard is copyright (c) 2015-present, [Contributors to MSF-Dashboard](https://github.com/MSF-UK/MSF-Dashboard/graphs/contributors) for MSF.

MSF-Dashboard is free software, licensed under the MIT License. See the file `LICENSE.md` in this distribution for more details on the license and see `LICENSES-DEP.md` to know more about the dependencies.

repository: MSF-UK/MSF-Dashboard

gha_language: JavaScript

repo_name: MSF-UK/MSF-Dashboard


text:
# TravelBucketList

    Travel Bucket List is a website to show people your top travel destinations and why!

## Installation

- Clone this project to your local desired folder.
- run the following commands:
- `npm i`
- `npm run dev`
- Open your browser at address `http://localhost:3000`

## Support

    This app supports adding and removing of new locations
    except of those chosen by the creators

## Credits

    This app created by Nativ Sibony & Shlomi Dori as part of our 'computer tech' course.

repository: NativSibony/TravelBucketList

branch_name: refs/heads/main

repo_name: NativSibony/TravelBucketList


text:
# Zabbix
[Zabbix](https://www.zabbix.com) is a mature and effortless enterprise-class open source monitoring solution for network monitoring and application monitoring of millions of metrics. This integration extends the alerting capabilities to use xMatters.

# Pre-Requisites
* [Zabbix](https://www.Zabbix.com) (tested with version 3.2 but should support others)
* xMatters account - If you don't have one, [get one](https://www.xmatters.com)!
* xMatters Integration Agent (IA) - Installed on the Zabbix server and configured. The IA is available [here](https://support.xmatters.com/hc/en-us/articles/201463419-Integration-Agent-for-xMatters-5-x-xMatters-On-Demand).

# Files
* [zabbix_comm_plan.zip](zabbix_comm_plan.zip) - The Communication Plan that receives Zabbix alerts
* [zabbix_files.zip](zabbix_files.zip) - Files required by the Integration Agent and Zabbix

**Note**: To download, follow each link above and then click the **Download** button. Do not try to download by right-clicking the links.

# Installation

## xMatters set up
### Add Event Domain
To add the Event Domain in xMatters:
1. In xMatters, go to the **Developer** tab and then select **Event Domains** from the left hand menu.
2. Click on the **applications** link.
3. Click **Add New** next to INTEGRATION SERVICES.
4. Enter `Zabbix` as the name and click **Save**.

### Import the communication plan
To import the communication plan into xMatters:
1. From the xMatters **Developer** tab, select **Communication Plans** from the left hand menu.
2. Click **Import Plan**.
3. Click **Choose File**, and then locate the downloaded communication plan (.zip file).
4. Click **Import Plan**.
	* Importing the plan will automatically enable it, and enable its forms for web services.

### Access URL
The Comm Plan has a URL that is required when configuring the Integration Agent.
To get the URL:
1. On the Zabbix Comm Plan, click **Edit**, then **Integration Builder**.
2. On the Integration Builder tab, expand the list of Inbound integrations.
3. Click the gear icon beside Zabbix inbound, and then select **Integration URL**.

Copy the URL displayed in the dialog box and save it for use below.

## Setup the Integration Agent (IA)
To add the Zabbix integration to your Integration Agent:
1. Extract zabbix_files.zip on the Zabbix server.
2. Under the integration-agent folder, copy the zabbix folder to <IA_HOME>/integrationservices/
3. Edit <IA_HOME>/integrationservices/zabbix/configuration.json
	* Set WEB_SERVICE_URL to the URL copied earlier.
	* Set ZABBIX_API_URL with the correct URL to your Zabbix API.
4. Edit <IA_HOME>/conf/IAConfig.xml
    * Add `<path>zabbix/zabbix.xml</path>` under the <service-configs> node.
5. From <IA_HOME>/bin, execute ``./iadmin.sh reload all``
    * This loads the new configuration into the Integration Agent.

## Zabbix Setup
### Add xMatters API User
In order for xMatters to Acknowledge and add comments to a Zabbix event, it needs to be able to use the Zabbix API. In order to use the API, an xMatters user needs to be created in Zabbix for authentication:
1. In Zabbix, go to **Administration**, then **Users** and click **Create User**.
2. On the **User** tab, enter the following:
    * **Alias**: `xMatters`
    * **Name**: `xMatters`
    * **Surname**: `API`
    * **Groups**: <select any group available in your instance of Zabbix>
    * **Password**: <set a password for this user>
    * **Password (once again)**: <repeat the previous password>
        * Remember this password as it will be needed later
3. On the **Permissions** tab, enter the following:
    * **User Type**: Zabbix Super Admin
4. Click **Add**.

### Create the xMatters Media Type
Media types are used for sending notifications from Zabbix. To create an xMatters Media Type:
1. In Zabbix, go to **Administration**, then **Media Types** and click **Create Media Type**.
2. Enter the following:
    * **Name**: `xMatters`
    * **Type** Script
    * **Script Name**: `xMattersEvent.sh`
    * **Script Parmeters**:
        * `{ALERT.SENDTO}`
        * `{ALERT.SUBJECT}`
        * `{ALERT.MESSAGE}`
3. Click the **Add** button.

### Create/Update Recipients
If you intend to send notifications directly to specific users:
1. In Zabbix, go to **Administration**, then **Users** and select your user.
2. In the **Media** tab, click the **Add** link.
3. Enter the following:
    * **Type**: xMatters
    * **Send to**: <the user's xMatters User ID>
    * **When active**: `1-7,00:00-24:00`
    * **Use if severity**: <select the severity levels you want to notify on>
4. Click **Add**.
5. Repeat these steps for each of your users.

If you intend to send notifications to groups in xMatters, you will need to create a **user** (not a group) in Zabbix to represent the xMatters group. You cannot use Zabbix groups as Zabbix expands those groups before calling xMatters, sending it to each user individually instead of following group shifts and escalations.

To send to notifications to xMatters groups:
1. In Zabbix, go to **Administration**, then **Users** and click **Create User**.
2. On the **User** tab, enter the following:
    * **Alias**: <the name of your xMatters group>
    * **Groups**: <either select a group that has access to all hosts that you'll want this xMatters group to get notifications on OR select any group and set the User Type to Zabbix Super Admin (see below)>
    * **Password**: <set a password for this user>
    * **Password (once again)**: <repeat the previous password>
3. In the **Media** tab, click the **Add** link.
4. Enter the following:
    * **Type**: xMatters
    * **Send to**: <the name of your xMatters group>
    * **When active**: `1-7,00:00-24:00`
    * **Use if severity**: <select the severity levels you want to notify on>
5. Click the **Add** button.
6. On the **Permissions** tab, enter the following:
    * User Type: <depending on the Groups setting in step 2, set this to Zabbix User or Zabbix Super Admin)
7. Click **Add**.
8. Repeat these steps for each of your xMatters groups.

### Create Actions
In Zabbix, an Action is used to when you want to do something (such as send a notification) based on an event.
To create an Action that sends a notification via xMatters:
1. In Zabbix, go to **Configuration**, then **Actions** and click **Create Action**.
2. In the **Action** tab, set a Name and Conditions for your Action.
3. In the **Operations** tab, under the Operations section click the **New** link.
4. Enter the following:
    * **Send to Groups**: <leave empty>
    * **Send to Users**: <select your users and/or xMatters groups>
    * **Send only to**: xMatters
    * **Default message**: <uncheck>
    * **Message**: <enter the following in order, one per line>
```
{EVENT.ID}
{HOST.CONN1}
{HOST.DESCRIPTION1}
{HOST.IP1}
{HOST.NAME1}
{ITEM.DESCRIPTION1}
{ITEM.ID1}
{ITEM.KEY1}
{ITEM.NAME1}
{ITEM.VALUE1}
{TRIGGER.NAME}
{TRIGGER.SEVERITY}
{TRIGGER.STATUS}
{TRIGGER.URL}
```
5. Click the **Add** link (not the button).
6. Click the **Add** button.

## Setup Alert Script
To add the Zabbix integration to your Integration Agent:
1. Extract zabbix_files.zip on the Zabbix server.
2. Under the zabbix-alertscripts folder, copy xMattersEvent.sh to the Zabbix AlertScripts folder.
    * If you don't know the location of the Zabbix AlertScripts folder, check your Zabbix configuration file.
3. In the Zabbix AlertScripts folder, execute ``chmod 755 xMattersEvent.sh``

## Setup Encrypted Password File
In the Add xMatters API User step above, you created a user called xMatters in Zabbix. The Integration Agent will log in as this user when accessing the Zabbix API. To encrypt the password for xMatters to use when it logs in, perform the following:
1. Change directory to <IA_HOME>/bin
2. Execute ``./iapassword.sh --new <PASSWORD> --file integrationservices/zabbix/.zabbixpasswd``
    * Replace <PASSWORD> with the password for the xMatters user in Zabbix.
    * Note that there are two dashes in front of ``new`` and ``file``.

# Testing
To test the integration, create a Zabbix event by causing the conditions required to trigger it. The associated Action will call xMatters to send notifications.

When you receive the xMatters notification, respond with **Acknowledge** and, if using the xMatters Mobile App, add a comment. In Zabbix, go to **Monitoring**, then **Problems** to verify that the event is acknowledged and that any comment is shown when you mouse over the Ack.

repository: tkouhsari/xm-labs-zabbix

repo_name: tkouhsari/xm-labs-zabbix


text:
## Introduction

**msgpack-tools** contains simple command-line utilities for converting from [MessagePack](http://msgpack.org/) to [JSON](http://json.org/) and vice-versa. They support options for lax parsing, lossy conversions, pretty-printing, and base64 encoding.

- `msgpack2json` -- Convert MessagePack to JSON
- `json2msgpack` -- Convert JSON to MessagePack

They can be used for dumping MessagePack from a file or web API to a human-readable format, or for converting hand-written or generated JSON to MessagePack. The lax parsing mode supports comments and trailing commas in JSON, making it possible to hand-write your app or game data in JSON and convert it at build-time to MessagePack.

## Build Status

| [Travis-CI](https://travis-ci.org/) |
| :-------: |
| [![Build Status](https://travis-ci.org/ludocode/msgpack-tools.svg?branch=master)](https://travis-ci.org/ludocode/msgpack-tools/branches) |

## Examples

To view a MessagePack file in a human-readable format for debugging purposes:

    msgpack2json -di file.mp

To convert a hand-written JSON file to a MessagePack file, ignoring comments and trailing commas, and allowing embedded base64 with a `base64:` prefix:

    json2msgpack -bli file.json -o file.mp

To fetch MessagePack from a web API and view it in a human-readable format:

    curl 'http://example/api/url' | msgpack2json -d

To view the MessagePack-equivalent encoding of a JSON string:

    $ echo '{"compact": true, "schema": 0}' | json2msgpack | hexdump -C
    00000000  82 a7 63 6f 6d 70 61 63  74 c3 a6 73 63 68 65 6d  |..compact..schem|
    00000010  61 00                                             |a.|
    00000012

## Installation

msgpack-tools currently must be built from source. The latest version of the msgpack-tools source archive can be downloaded from the releases page. This includes the library dependencies and has pre-generated man pages:

[https://github.com/ludocode/msgpack-tools/releases](https://github.com/ludocode/msgpack-tools/releases)

msgpack-tools uses CMake. A `configure` wrapper is provided that calls CMake, so on Linux and Mac OS X, you can simply run the usual:

    ./configure && make && sudo make install

If you are building from the repository, you will need to fetch the dependencies and generate the man pages. These are done with the scripts under `tools/fetch.sh` and `tools/man.sh`. [md2man](https://github.com/sunaku/md2man) is required to generate the man pages.

## Differences between MessagePack and JSON

MessagePack is intended to be very close to JSON in supported features, so they can usually be transparently converted from one to the other. There are some differences, however, which can complicate conversions.

These are the differences in what objects are representable in each format:

- JSON keys must be strings. MessagePack keys can be any type, including maps and arrays.

- JSON supports "bignums", i.e. integers of any size. MessagePack integers must fit within a 64-bit signed or unsigned integer.

- JSON real numbers are specified in decimal scientific notation and can have arbitrary precision. MessagePack real numbers are in IEEE 754 standard 32-bit or 64-bit binary.

- MessagePack supports binary and extension type objects. JSON does not support binary data. Binary data is often encoded into a base64 string to be embedded into a JSON document.

- The top-level object in a JSON document must be either an array or object (map). The top-level object in MessagePack can be any type.

- A JSON document must be entirely in one encoding. MessagePack does not specify encoding, and different strings in the same document can technically be in different encodings (though this is a really bad idea.)

By default, `msgpack2json` and `json2msgpack` convert in strict mode. If an object in the source format is not representable in the destination format, the converter aborts with an error. A lax mode is available which performs a "lossy" conversion, and base64 conversion modes are available to support binary data in JSON.

repository: dybekm/msgpack-tools

repo_name: dybekm/msgpack-tools


text:
[<<< Previous question <<<](0171.md)   Question ID#0172.md   [>>> Next question >>>](0173.md)
---

Consider the PHP program (which includes a file specified by request):

```php
<?php
$color = 'blue';
if (isset( $_GET['COLOR'] ) )
       $color = $_GET['COLOR'];
require( $color . '.php' );
?>
<form method="get">
     <select name="COLOR">
        <option value="red">red</option>
        <option value="blue">blue</option>
     </select>
     <input type="submit">
</form>
```
A malicious user injects the following command:
```php
/vulnerable.php?COLOR=C:\\notes.txt%00
```
where vulnerable.php is a remotely hosted file containing an exploit. What does the malicious user want to do?

- [ ] A) Remove the .php suffix, allowing access to files other than .php.
- [ ] B) Perform a cross-site scripting attack.
- [ ] C) Inject a remotely hosted file containing an exploit.
- [ ] D) Execute the malicious code that exists in the file vulnerable.php.

Answer

  Answer: A

repository: alceanicu/zce

gha_language: TypeScript

repo_name: alceanicu/zce


text:
[Public Git Archive](https://pga.sourced.tech/) is an archive of all github projects with more than 50 stars.
It is downloaded and stored as index2.csv

Firstly, GetJavaProjects.py is executed to create a file JavaProjectsData.txt. It finds all projects in the archive that have at least
1 line of code in Java and saves the following information for them:
* author of the repository;
* name of the repository;
* licensing information.

After that, DownloadProjects.py downloads the projects and saves them in the desired directory. In the end, each project will be located
in the subdirectory with the name of the author of the repository, and will consist of three items:
* git-cloned directory with the entire history of changes;
* zip-file of the last commit to be processed by SourcererCC;
* txt-file with the licensing information.

The download happens in several steps. The reason for that is that certain projects are deleted or made private after the compilation of the archive, so some of the repositories will not be cloned. The script deletes extra files and also checks that the zip-file contains the main branch of the repository. During its operation, the script creates temporary files ListChecks.txt (that checks the existence of the project files) and ListBranches.txt (that stores the main brach of each repository).

The last stage of data collection is running CreateListOfProjects.py that generates project-list.txt, a file that must be copied to
SourcererCC/tokenizers/block-level/.

This marks the end of Data Collection and the beginning of Clone Detection with SourcererCC.

repository: ml-in-programming/PlagiarismStudy

repo_name: ml-in-programming/PlagiarismStudy