text:
SHIELD Operator's Manual
========================

Hello, and welcome to the **SHIELD Operator's Manual**, an in-depth look at
all things SHIELD.  This guide aims to be an exhaustive guide to the
installation and operation of the SHIELD Data Protection solution.

If you are looking for a more easy-going start-up guide, you may want to
check out the [Getting Started]($docs/ops/getting-started) guide.

If you are interested in contributing to SHIELD itself, or wish to write a
plugin to extend the capabilities of your SHIELD installation, head on over
to the [SHIELD Developer Documentation]($docs/dev).

What is SHIELD?
---------------

SHIELD is a _data protection_ solution.  It is designed to run scheduled
tasks to backup your important data systems to off-site cloud storage
solutions, and facilitate the restoration of backup archives in the event of
outages or data loss.

SHIELD supports lots of different data systems, through its flexible and
modular plugin architecture.  We currently support:

  - **PostgreSQL** databases, via the [`postgres` plugin][postgres-plugin].
  - **MySQL** / **MariaDB**, via the [`mysql` plugin][mysql-plugin], or the
    [`xtrabackup` plugin][xtrabackup-plugin].
  - **Redis** key-value store, via the [`redis` plugin][redis-plugin].
  - ... and [many more][target-plugins].

[postgres-plugin]:   $docs/ops/plugins/postgres
[mysql-plugin]:      $docs/ops/plugins/mysql
[xtrabackup-plugin]: $docs/ops/plugins/xtrabackup
[redis-plugin]:      $docs/ops/plugins/redis
[target-plugins]:    $docs/ops/plugins#target-plugins

Cloud Storage systems are likewise pluggable.  Out of the box, SHIELD
supports:

  - **Amazon S3** (and S3 work-alikes) via the [`s3` plugin][s3-plugin].
  - **Microsoft Azure Blobstore** via the [`azure` plugin][azure-plugin].
  - **GCP Blobstore** via the [`google` plugin][google-plugin].
  - On-premise **WebDAV** endpoints, via the [`webdav` plugin][webdav-plugin].
  - ... and [several others][store-plugins].

[s3-plugin]:     $docs/ops/plugins/s3
[azure-plugin]:  $docs/ops/plugins/azure
[google-plugin]: $docs/ops/plugins/google
[webdav-plugin]: $docs/ops/plugins/webdav
[store-plugins]: $docs/ops/plugins/#store-plugins

SHIELD is a distributed system.  The SHIELD _core_ leverages a network of
_agents_ to do the heavy lifting of data backup and restore.  When you
deploy SHIELD into your infrastructure, yet get to choose how many agents
you want to provision, and where in the network they sit.

// FIXME: link to more in-depth docs on the architecture of SHIELD

_Multi-tenancy_ is baked right into SHIELD via a robust role-based access
control (RBAC) system in place to help isolate different subsets of users
from one another.  People in one tenant are unable to see configurations
made by people in another tenant.  This allows a single SHIELD to support
multiple, independent teams.

// FIXME: link to more in-depth docs on MT and RBAC

SHIELD also supports a sophisticated authentication system.  You can hook
up to your external Cloud Foundry UAA server, a BOSH UAA instance, or even
Github (both public and on-premise).  As users log in via their external
credentials, SHIELD will automatically create the necessary tenants and
assign roles based on the SHIELD configuration.

// FIXME: link to more in-depth docs on how Authn works

We believe strongly in encryption.  Whenever SHIELD communicates across the
network, it does so over encrypted channels (SSH, TLS/HTTPS), with endpoint
identity verification (host keys, mutual TLS, etc.).  All backup archives
are encrypted with unique key material, to ensure that data at-rest is also
resistent to snooping and tampering.

// FIXME: link to more in-depth docs on how SHIELD encryption works

Installation
------------

Before you can begin to use SHIELD to protect your important data, you're
going to need to install it.  You have two options: via [BOSH][bosh]
(ideal for Cloud Foundry users), or via [Docker][docker].

[bosh]:   https://bosh.io
[docker]: https://docker.com

### Planning Your Installation

Before you start installing the software, it's worthwhile to take a step
back and plan out your installation

#### Network Topology

SHIELD operates best on flat network topologies, without <abbr>NAT</abbr>
devices or HTTP(S) proxy services.

//![A Flat Network Topology]($docs/ops/manual/flat-topology.png)

SHIELD requires _mutual network visibility_ between the core and all
cooperating agents.  Each agent issues a small HTTP request to the core,
to inform the core that it is alive, and ready to be inventoried.  This is
called the _registration ping_.  For each registration ping received, the
core records the name and port given and the remote address of the
connecting TCP socket.  At some later time, the SHIELD core will initiate an
SSH connection to the recorded agent IP address, and gather agent
information.

Because of this, NAT devices tend to confound SHIELD.  The registration ping
originates (at the TCP level) from the NAT gateway, not the host running the
SHIELD agent software.  When the core attempts to connect back to the agent,
it initiates a connection to the _NAT device_ on the agent port, which
generally fails.

Flat networks with HTTP(S) proxies are not impossible, but they can be
unruly.  When configuring proxy clients (via `http_proxy` / `https_proxy`,
`no_proxy` environment variables, or similar mechanisms), you will want to
be especially cognizant of the HTTP(S) connections needed by the SHIELD
software itself.  Often, these connections will need to bypass am
Internet-bound proxy (i.e. one in a DMZ) in order to function.

//![A Proxied Network Topology]($docs/ops/manual/proxied-topology.png)

Likewise, if your cloud storage solution is to be dealt with over HTTP(S),
you will need to make sure that either your proxy server can contact it on
behalf of each SHIELD agent, or that each agent blacklists the domains
and/or IP addresses of the storage endpoints in something like `no_proxy`.

#### Where to Colocate SHIELD Agents

Depending on the data systems you wish to backup, and their configuration
with respect to access control, you may be able to get away without
colocating _any_ SHIELD agents in your infrastructure.

There are really only two reasons for colocating a SHIELD agent on a data
system installation: plugin requirements and host access control
configuration.

Most SHIELD plugins stream their data through, without relying on any
temporary local storage.  This removes a throughput bottleneck (the disk),
as well as a capacity concern (how much temporary space do you need?).  Some
plugins, however, require local disk.  If the target system doesn't require
local access (more on that in a moment), you may want to spin up some
machines with large ephemeral disks just to handle these backup / restore
operations.  The conversation for backing up data then looks like this:

![A Dedicated SHIELD Agent]($docs/ops/manual/dedicated-shield-agent.png)

Some plugins absolutely cannot be executed across the network.  The
[Filesystem Plugin][fs], for example, can only deal with files on the local
filesystem (networked filesystems notwithstanding).  Therefore, if you need
to back up files on a host, you will need to deploy a SHIELD agent to run on
that host.

[fs]: $docs/ops/plugins/fs

### Using BOSH

[BOSH][bosh] is a cloud-agnostic deployment and orchestration tool that
excels at lifecycle management of software at all scales.  SHIELD has a
_BOSH release_ that can be used to deploy both the SHIELD core, and SHIELD
agents into new and existing BOSH deployments.

If you're already using BOSH (for example, if you are deploying Cloud
Foundry), adding SHIELD into your infrastructure should be easy.  If you are
still looking for a great release engineering framework, you can get your
feet wet with a SHIELD deployment or three.

The SHIELD BOSH release can be found [on Github][release].

[release]: https://github.com/starkandwayne/shield-boshrelease/releases

#### Deploying the SHIELD Core

Usually, the SHIELD core is a standalone, self-contained deployment.  To
deploy, you'll need to find or create a deployment manifest.    You may
need to consult your BOSH [cloud-config][bosh-cc] to find a suitable IP in
the `default` network.  Optionally, you may modify the deployment manifest
to specify a different network.

[bosh-cc]: https://bosh.io/docs/cloud-config

**NOTE**: The provided deployment manifest assumes that your BOSH director
has been deployed with a _config-server_ that can generate the necessary
certificates and keys for securing SHIELD's communications.  If that is not
the case, you will need to provide additional command-line options to the
`bosh deploy` command to store the generated credentials locally.  See
[the BOSH documentation][bosh-vars] for more information.

[bosh-vars]: https://bosh.io/docs/cli-int/#vars-store

Once BOSH has finished deploying SHIELD, you should be able to access the
SHIELD management console at [https://$IP](#).  The default login will be
`admin` (username) and `password` (password).

#### Deploying SHIELD Agents

If you need to colocate agents on other BOSH deployments, you have a few
options.  The fastest method is to modify those deployment manifests to
include the `shield-agent` job in the appropriate BOSH instance groups, like
this:

    instance_groups:
      - name: some-database
        jobs:
          # ... other jobs ....
          :::- name:    shield-agent
            release: shield:::

        # ... rest of configuration ...

This can get out of hand fast.  A more elegant solution is to use [BOSH
runtime configs][bosh-rcs] and _inject_ the SHIELD agent job into other
deployments without mucking about with their deployment manifests.

[bosh-rcs]: https://bosh.io/docs/runtime-config

Here's a working runtime config:

    ---
    releases:
      - name:    shield
        version: 8.4.0

    addons:
      - name: shield
        jobs:
          :::- name:    shield-agent
            release: shield:::

To use this, update your existing runtime-config:

    $ bosh update-runtime-config addons.yml

Then, `bosh deploy` your pre-existing manifests, without changing them.
For more information, including how to limit the `shield` addon to just
specific deployments / VMs, read the
[BOSH runtime-configs documentation][bosh-rcs].

### Using Docker

#### The SHIELD Image

The SHIELD project builds Docker images for all releases, and
hosts them on [Docker Hub][project-dockerhub].

To cut down on space used in image registries, and on
containerization hosts, we bundle both components of a typical
SHIELD deployment — the core and the agent — into a
single Docker image: `shieldproject/shield`.  If you want a SHIELD
core, run the container with the `/shield/init/core` command.  For
agents, run `/shield/init.agent`.

#### Configuring the SHIELD Core

A (containerized) SHIELD Core relies on two directories that you
probably want to bind-mount in from a persistent backing store:

  - `/var/shield` stores the SHIELD database, which houses all of
    the live configuration and metadata, including pointers to all
    backup archives.  It's kind of a big deal.

  - `/etc/shield` stores the SHIELD static configuration.

The latter directory is trickier to get right, since it contains
some specific files that you may want to specify:

  1. `/etc/shield/shieldd.conf` - A full SHIELD core YAML
     configuration file.  If not present, one will be created for
     you during container boot, pursuant to a set of environment
     variables (detailed next).

  2. `/etc/shield/ssh.key` - A private key to be used for core
     ⇆ communication (initiated by the core).  If you don't
     specify a `shieldd.conf` file, the image will create one for
     you, and pull the SSH key from this file.  _Otherwise, this
     file is **ignored**._

If you don't want to provide files, you _can_ configure your new
SHIELD core via environment variables.  See _SHIELD Core
Configuration Reference_, below.

[project-dockerhub]: https://cloud.docker.com/u/shieldproject/repository/list

#### Embedding the SHIELD Agent

TBD

### Configuration Reference

This section contains detailed descriptions of all configuration options for
the SHIELD core, and SHIELD agents.

#### SHIELD Core Configuration Reference

The SHIELD core configuration file is a YAML file, read at startup by the
`shieldd` binary.

Here's an example configuration file:

    ---
    # comments start with the octothorpe (#) and continue
    # until the end of the line

    api:
      env: Production
      color: green
      bind: <IP_ADDRESS>:80
      session:
        timeout: 300         # (hours) - about 2 weeks
      failsafe:
        username: admin
        password: super-sekrit-password

    scheduler:
      threads: 20

    limit:
      retention:
        min: 4
        max: 90

Here's the full set of configuration directives (in dot-notation)
that are currently supported:

- **api.bind** - The IP address and TCP port that the SHIELD core daemon
  should bind to and listen on for incoming API (HTTP) requests.  Defaults
  to `*:8888`.  `*` is interpreted to mean _all interfaces_.

- **api.color** - You can color code your SHIELD Web User Interfaces!  Set a hex
  value here (i.e. `##003300`) or other CSS-compatible color identifier, and
  the web UI will use it to colorize the environment name.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_COLOR` environment variable.

- **api.env** - An name for the environment, that SHIELD will pass
  through to clients accessing its API and web management console.  This can
  be useful for differentiating your staging SHIELD from your production
  SHIELD.  By default, no environment is set.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_ENV_NAME` environment variable.

- **api.failsafe.username** - When the SHIELD core starts up, it
  checks the local users table.  If it is empty (there are no
  local users), it creates a _failsafe_ account, using these
  parameters.  This is designed to assist in a safe and secure
  bootstrap.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_FAILSAFE_USERNAME` environment variable.

- **api.failsafe.password** - The cleartext password to assign the
  failsafe user, upon creation.  You can change this later,
  without needing to reconfigure or re-deploy SHIELD.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_FAILSAFE_PASSWORD` environment variable.

- **api.motd** - A (hopefully) short message to display to operators on the
  login screen.  You can use this for compliance messages, important
  notices, an explanation of which authentication method people should use,
  who to contact for help, etc.  By default, there is no MOTD.

- **api.session.clear-on-boot** - Whether or not to clear
  interactive sessions from the database when the SHIELD core
  reboots.  This is set to `yes` by default, which will log
  everyone out when the core process restarts.

- **api.session.timeout** - How long (in hours) before idle
  authenticated sessions are invalidated.  Defaults to 720 hours
  (about a month).

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_SESSION_TIMEOUT` environment variable.

- **auth** - A list of non-local authentication backends.

- **auth[].backend** - What backend to use for this authentication
  provider.  The backend determines how authentication is carried
  out, what external systems are involved, etc.

  The following backends are supported:

  - **github** - GitHub, either public (github.com) or private
    (Github Enterprise).  Tenant membership is mapped based on
    GitHub organization / team membership.

  - **uaa** - Cloud Foundry UAA, to allow Cloud Foundry users to
    log into SHIELD without needing another account.  Tenant
    membership is mapped based on scim rights.

- **auth[].identifer** - A unique (and unchanging) internal
  identifier for this authentication provider.

- **auth[].name** - The human-readable name for this
  authentication providers, to be displayed in command-line and
  web UI login flows.

- **auth[].properties** - A set of backend-specific configuration
  items.

- **cipher** - Which encryption algorithm and chaining mode to use
  for encrypting backup archives.  Supported values are:

  - `aes256-ctr` - 256-bit AES, in Counter CBC mode.

  We plan to introduce more types as the need arises.

  Each backup archive tracks which encryption type was in force when it was
  taken, to allow operators to change this value without rendering previous
  backup archives unusable.

- **data-dir** - The absolute path to the directory where SHIELD will
  store all of its persistent data.  Important files stored here include:

  - `$datadir/shield.db` - The SHIELD metadata database.
  - `$datadir/vault/*` - The encrypted files that back the vault.
  - `$datadir/vault.crypt` - The encrypted file which stores the seal
    keys to the SHIELD Vault.  This file is encrypted with the
    SHIELD master password.
  - `$datadir/bootstrap.log` - A log of what occurred during a SHIELD
    _from-nothing_ recovery.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_DATA` environment variable.

- **debug** - Whether or not to enable verbose debug logging.  This is a
  boolean, and it defaults to `no`, which is a sane choice for any
  production or staging environment.  Debug logging is verbose, and very
  low-level.  It is of primary value to SHIELD developers.

- **fabrics** - A list of SHIELD Agent Fabric configurations.

- **fabrics[].name** - The name of this fabric.  For the Legacy
  SSH fabric, this must be `legacy`.

- **fabrics[].ssh-key**  - The literal, PEM-encoded RSA private
  key that the SHIELD core should use when connecting to remote
  SHIELD agent processes.

  This is **required** for the Legacy SSH fabric.

- **limit.retention.max** - The maximum number of days that any
  backup archive may be retained for.  This can be useful to
  ensure that tenants don't overrun storage with excessively long
  retention periods.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_MAXIMUM_RETENTION` environment variable.

- **limit.retention.min** - The minimum number of days that all
  backup archives must be retained for.  This allows SHIELD
  operators to enforce organization-wide compliance for archive
  retention.  Defaults to 1 (day).

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_MINIMUM_RETENTION` environment variable.

- **scheduler.fast-loop** - The frequency, in seconds, of the
  SHIELD scheduler's "fast loop."  On every iteration of the fast
  loop, SHIELD will schedule backup jobs that ought to run, execute
  pending tasks (if it has workers available), and handle inbound
  agent registration pings.

  By default, the fast loop executes once a second.  Unles you have an
  urgent need otherwise, you shouldn't change this.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_FAST_LOOP` environment variable.

- **scheduler.slow-loop** - The frequency, in seconds, of the
  SHIELD scheduler's "slow loop."  The slow loop handles
  administrative tasks for the SHIELD core, including archive
  expiration and purgation, session clearing, data analytics, and
  cloud storage testing>

  By default, the slow loop exucutes once every 300 seconds (5 minutes).
  Turning up the frequency will result in higher load on external cloud
  storage systems.  Decreasing the frequency will cause expired archives to
  remain in cloud storage for longer.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_SLOW_LOOP` environment variable.

- **scheduler.threads** - How many worker threads should the
  SHIELD core spin.  This defaults to `5`, but you should increase
  the 1.5 times the number of concurrent backup tasks you expect to
  see, at peak.

  Low worker counts can cause the SHIELD scheduler to "stall out"
  and not execute scheduled tasks in a timely fashion.  The 1.5x
  multiplier accounts for purge operations, cloud storage tests, and
  other background tests.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_THREADS` environment variable.

- **scheduler.timeout** - How long (in hours) should the SHIELD
  core allow a single task to execute before considering it
  _hung_, and terminate it.

  In the Docker image (under automatic configuration), this can be
  set by the `$SHIELD_TASK_TIMEOUT` environment variable.

- **vault.address** - The URL of the SHIELD Vault.  This should almost
  always be `https://<IP_ADDRESS>:8200`.  If you are using the BOSH release,
  this cannot be configured.

  In the Docker image (under automatic configuration), this can be
  set by the `$VAULT_ADDR` environment variable.

- **vault.ca** - The path to the X.509 Certificate Authority certificate,
  PEM-encoded, for validating the Vault certificate.  If you are using the
  BOSH release, this cannot be configured (nor does it need to be).

- **web-root** - The root path to the SHIELD web management UI assets.
  Defaults to the relative path `web`, which is probably not what you want.

#### SHIELD Agent Configuration File Reference

The SHIELD agent configuration file is a YAML file, read at startup by the
`shield-agent` binary.

- **name** (required) - The name of this agent, for registration with the
  SHIELD core.  This name will appear in web and CLI interfaces, to people
  configuring backup jobs, and should describe the role this agent
  installation plays in the overall topology.

- **authorized\_keys\_file** (required) - The path to an SSH _authorized
  keys_ file, which should contain the public component of the agent private
  key that the SHIELD core will use to authenticate to the agent for remote
  orchestration.

- **listen\_address** - The IP address and TCP port that the SHIELD agent
  should bind to and listen on for incoming orchestration (via SSH).
  Defaults to `*:5444`.  `*` is interpreted to mean _all interfaces_.

- **plugin\_paths** (required) - A YAML list of paths that the agent will
  use when attempting to resolve plugin names to binaries.  This is kind of
  like the canonical UNIX `$PATH` environment variable, except it does not
  apply to any programs that the plugins themselves attempt to execute.

  You should list all of your plugin binary directories here.

- **registration** - This subsection governs how this agent will register
  with its SHIELD core.  While technically optional, registration is highly
  recommended, from an ease-of-use standpoint.

  The following keys exist underneath `registration:`:

  - **url** - The HTTP(S) URL of the upstream SHIELD core.  This will
    normally be something like `https://$ip_or_hostname/`.

  - **interval** - How often (in seconds) should the agent ping the SHIELD
    core and provide reigstration details.  The SHIELD core determines when
    it _validates_ agent registrations and extracts metadata information, so
    this setting cannot be used to increase the frequency of such updates.

  - **shield\_ca\_cert** - Path to a file containing the PEM-encoded CA
    certificate that issued the SHIELD core's X.509 TLS certificate.  This
    allows operators to validate self-signed certificates, or custom,
    in-house CA-issued certificates.

    This has no effect of `skip_verify` is set to true.

  - **skip\_verify** - Whether or not to disable verification of the SHIELD
    core X.509 TLS certificate.  This defaults to _false_, since certificate
    verification is generally A Good Thing ™

Using SHIELD
------------

SHIELD features a beautiful web user interface and a robust command-line
interface.  We like to think of the web UI as providing more visibility into
the configuration of SHIELD, while the CLI provides more flexibility in
terms of automation.

### The Web UI

You can access the web UI by pointing your browser at the IP address of your
SHIELD core installation.  SHIELD forces all HTTP traffic over TLS, via port
443, for security reasons.

#### Logging In

Before you can interact with SHIELD, you must log in.

![The SHIELD Web UI Login Screen]($docs/ops/manual/ui/login-screen.png)

On the right is the login form for local authentication.  On the left is a
list of the configured _authentication providers_.  These allow SHIELD
administrators to integrate SHIELD authentication with 3rd-party, external
identity systems like Github, or Cloud Foundry UAA.

Note: You may not have any authentication providers listed.

#### The Heads-up Display

All logged in?  Great!

At the top of the screen, you should see the _heads-up display_:

![The SHIELD Heads-Up Display]($docs/ops/manual/ui/hud.png)

To the left is identifying information about this SHIELD core, including the
configured SHIELD environment name, the IP address and/or fully-qualified
domain, and the version of SHIELD.

The first pane summarizes the overall health of SHIELD and the current
tenant's configuration.

  - **SHIELD is ...** - Reports the current status of the SHIELD API.
    If all is well, this will say _SHIELD is up_, in a reassuring green hue.
    If the SHIELD core is not responding to API calls, this will say _SHIELD
    is DOWN_, in red.  Sometimes, it may report that the SHIELD is locked,
    in which case an administrator needs to intervene to unlock it.

  - **Cloud Storage is ...** - Reports the health of all global cloud
    storage systems, as well as the health of all cloud storage specific to
    the currently selected tenant.

  - **Jobs are ...** - Reports the status of all backup jobs for the current
    tenant.  It considers only the most recent execution of each job,
    whether it was scheduled or run manually (ad hoc).

The second pane, titled _Data Protection Summary_, provides some numbers for your
consideration.  All of these are per-tenant.

  - **Scheduled Backup Jobs** - How many total jobs are scheduled to run.

  - **Backup Archives** - How many backup archives exist.

  - **Cloud Storage Used** - How much of cloud storage is being used by the
    backup archives for this tenant's jobs.

  - **Daily Storage Increase** - A simple linear projection of the amount of
    additional cloud storage that will be used, each day, given the current
    schedules, retention policies, and archive sizes.

The heads-up display is partially dependent on the current tenant, so if you
switch to a different tenant, you might get different numbers / statuses.

#### The Task Sidebar

To the left of the screen is a sidebar with links to the common tasks you
may want to perform:

  - Run an ad hoc backup
  - Restore data from a backup
  - Configure a new backup job

#### The Top Bar

TBD

#### The Navigation Bar

The black navigation bar (immediately under the heads-up display) will stick
to the top of the viewport as you scroll.  It provides top-level navigation,
including:

- **Systems** - Your data systems are the things that SHIELD protects, by
  making copies of the important data contained within them, on a scheduled
  and recurring basis.  This page lets you review and manage those systems.

- **Storage** - Cloud storage is where SHIELD keeps the backup copies of
  your data.  You can configure however many storage systems you want, in
  whatever configuration you deem appropriate.  This page lets you access
  both global (shared) cloud storage systems, and those specific to your
  tenant.

- **Admin** - This one is for SHIELD administrators only.  It provides
  access to a host of administrative functions like tenant and user
  management, global cloud storage management, etc.

### The CLI

If you prefer life in the terminal, you're in luck — SHIELD
has a command-line interface packed with just as much
functionality as the web interface.  To get started with the CLI,
you first must configure access to the SHIELD core by identifying
the API endpoint and logging in:

    $ shield api https://:::shield-ip::: my-shield

    $ shield -c my-shield login
    Username: admin
    Password:
    logged in successfully!

To see what more the CLI can do, check out the following helpful
commands:

    $ shield -h
    $ shield commands

Note: the remainder of this guide will focus on the web user
interface.  Terminal fans are encouraged to explore the SHIELD CLI
to see how those same concepts can be applied in a textual world.

Configuring Backups
-------------------

TBD

### Wizard Walkthrough

TBD

### The Systems Page

TBD

### Adding a second schedule

TBD

Running Backups / Restores
--------------------------

The primary point of running SHIELD is to run regularly-scheduled
backup jobs, and then (when necessary) restore backups to their
original data system.

### Ad hoc vs. Scheduled

SHIELD can run backup jobs in one of two ways.

**Ad hoc** backup jobs are executed in one-off scenarios, at the
direction of a SHIELD operator.

**Scheduled** backup jobs are run at specific, cyclical times.
Operators can specify that backups should be taken hourly, daily,
weekly, or monthly.  Inside SHIELD, the _scheduler_ keeps track of
the next scheduled run of each job, and executes accordingly.

For maximum value, you will probably want to schedule most of your
backups.  The two modes of operation are not mutually exclusive;
you can trigger an ad hoc run of a scheduled job.  This comes in
handy when carrying out maintenance or change operations against a
system and you want the freshest _pre-change_ backup archives as
you can get.

### The Ad hoc backup Wizard

TBD

### The Timeline View

TBD

### Annotating Tasks

TBD

### Restoring from the Timeline Page

TBD

### The Restore Wizard

TBD

Cloud Storage
-------------

TBD

### How SHIELD Uses Cloud Storage

TBD

### Retention Policies

TBD

### How the HUD interacts

TBD

### Storage Thresholds

TBD

### The Storage Display Page

TBD

### Shared Storage

TBD

Multi-Tenancy
-------------

TBD

### What is a Tenant?

TBD

### Switching Tenants

TBD

### Role Assignments (and what they mean)

TBD

### Authentication Providers and Tenants

TBD

#### UAA

TBD

#### Github

TBD

### The default tenant

TBD

Encryption
----------

TBD

### What is Encryption

TBD

### Why do I Care?

TBD

### How does SHIELD use encryption?

TBD

### At-rest vs. in-flight encryption

TBD

Administration
--------------

All Administrative tasks are done from the admin panel

![The Shield Admin Panel](admin-panel.png)

### Initializing A SHIELD Core

TBD

### The Master Password

TBD

### The Administrative Backend

TBD

#### Tenants

Under `Tenants` admins have the ability to see all current teneants as well as create new tenants.
Users can be invited to tenants by editing the tenant or during tenant creation.

![Tenants](tenant-panel.png)

#### Shared Storage

Shared storage can be found under `Global Storage Systems`, from this page you can view all existing
global stores as well as create new ones. Shared or global storage can be configured for use across
multiple tenants.

![Global Storage](storage-panel.png)

#### Retention Policy Templates

TBD

#### Managing Agents

Information on Shield Agents can be found under `Agents of Shield`. This can be helpful for keeping
inventory on agents throughout your deployments or to resync any disconnected agents.

![Shield Agents](agents-panel.png)

#### Authentication Providers

You can view the configuration of your shield authentication providers under `Authemtication Providers`.
Please note that configuring an additonal auth provider is done from deploying shield itself.

![Authentication](auth-panel.png)

#### Local User Management

Shield users can be managed under `Local User Management`. Here admins can view a list of all Shield users,
edit permissions of users, and onboard new users.

![Users](user-panel.png)

#### Rekeying SHIELD

If you wish to Change your shield master password, you can find this option under `Rekeying SHIELD`

**Note** You need your current master password to complete this.

![Panel](rekey-panel.png)

#### Session Management

Admins can manage user sessions under `Manage User Sessions`. Here You can get information on all current sessons
and expire session tokens if desired.

![Sessions](session-panel.png)

How Do I Backup _X_?
--------------------

### SHIELD Itself

In a disaster recovery situation, it's important to get shield up and running to begin recovering your other data systems
as soon as possible. It requires backups of SHIELD itself.

#### How do SHIELD backups work?

The SHIELD state exists as 3 entities found in /var/vcap/store/shield (for a shield deployed by bosh).
They Include:
  - The SHIELD database which exists as a sqlite database name shield.db
  - The vault.crpyt file
  - The vault directory

We use the file system plugin to backup these 3 items and store them as our SHIELD backup. When using the fs plugin to backup
shield make sure `Strict Mode` is unchecked. We do this to prevent failures with missing files which can happen with the presence
or absence of the `shield.db-journal`.

#### The Problem With Encryption

SHIELD backups *must* be taken with fixed key encryption. In this situation of restoring shield, we cannot use generated
encryption keys because there is no way to restore our vault with keys that exist in that sealed vault. This is why it's
important to keep the fixed key generated when you initialize SHIELD in a safe and retrievable place.

#### Configuring the `fs` Plugin

Lets walk through creating a SHIELD backup from the shield webui.

#### For the properties make sure you're targeting the agent on the SHIELD box and the correct path.

  ![Recover](shield-recover2.png)

#### Make sure to de-select `Randomize Encryption Keys`

  ![Recover](shield-recover4.png)

#### Lets double check our configuration is correct

  ![Recover](shield-recover5.png)

#### "Normal-mode" Restores

TBD

### Cloud Foundry UAA

### Cloud Foundry CCDB

### BOSH

Monitoring SHIELD
-----------------

TBD

### Using the HUD

TBD

### API Access for Monitoring

TBD

### Metrics of interest

TBD

### Log messages to watch

TBD

Glossary
--------
Shield has a number of terms and abstractions that are important for an operator to know when working with the tool.

- **Shield Core**: The shield core servers as the main deployed shield box that includes the shield daemon and api.

- **Shield Agents**: A shield agent is software that is co-loacted on a deployment you wish to back up that leverages plugins to back up given data system

- **Tenants**: Shield provides tenancy to allow the seperation of teams/environments by allowing tenants to be created. Under a tenant users can be assigned
               and given roles, stores can be created, and indivdual systems can be backed up.

- **Targets**: A target is the name of a configured data system. They can be under the data systems tab.

repository: shieldproject/shield

gha_language: Go

repo_name: shieldproject/shield


text:
---
layout: default
---

# 1995-01-19 - Re: copyright and data havens

## Header Data

From<EMAIL_ADDRESS>(Roy M. Silvernail)

To<EMAIL_ADDRESS>(Mats Bergstrom)

Message Hash: 9b825ae9f283ae9c9944a06f7ac963cd673810c1cd20e3a373b52cce27553d0f

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1995-01-19 04:38:55 UTC

Raw Date: Wed, 18 Jan 95 20:38:55 PST

## Raw message

```
{% raw  %}From<EMAIL_ADDRESS>(Roy M. Silvernail)
Date: Wed, 18 Jan 95 20:38:55 PST
To<EMAIL_ADDRESS>(Mats Bergstrom)
Subject: Re: copyright and data havens
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

-----BEGIN PGP SIGNED MESSAGE-----

In list.cypherpunks<EMAIL_ADDRESS>writes:

>            (by the way, how long might it take to IDEA-encrypt a 2 MB
> .zip file?  I never tried)

Me, neither.  Let's find out....

[1] d:\doom>dir tt.zip

 Volume in drive D is unlabeled      Serial number is 0000:13FD
 Directory of  d:\doom\tt.zip

tt.zip        2292723   1-18-95  22:06
   2,292,723 bytes in 1 file(s)          2,293,760 bytes allocated
   4,853,760 bytes free

[1] d:\doom>timer^pgp +armor=off +compress=off +textmode=off -c tt.zip ^ timer
Timer 1 on: 22:16:16
Pretty Good Privacy(tm) 2.6.2 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 11 Oct 94
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Distributed by the Massachusetts Institute of Technology.
Export of this software may be restricted by the U.S. government.
Current time: 1995/01/19 04:16 GMT

You need a pass phrase to encrypt the file.
Enter pass phrase:
Enter same pass phrase again: Just a moment...
Ciphertext file: tt.pgp
Timer 1 off: 22:18:42  Elapsed: 0:02:26.05

'+textmode=off' is probably overkill, since PGP recognizes the non-text
content.  But my config.txt has armor and compress turned on.
Compressing almost doubled the time to encrypt, and armor gave me 39
*.asc files.
- --
Roy M. Silvernail --<EMAIL_ADDRESS>will do just fine, thanks.
          "Does that not fit in with your plans?"
                      -- Mr Wiggen, of Ironside and Malone (Monty Python)
          PGP public key available upon request (send yours)

-----BEGIN PGP SIGNATURE-----
Version: 2.6.1

iQCVAwUBLx3pkBvikii9febJAQFfgwP+MkpU2xv9zUPyYIYtzvsTUzSSdjcVi4Dd
8PW8AmVFCu2xHI0Zce8CCh3/i+ZMK15E6xvoFdESwS3mkz9DIBZZ/JfyPu0kQ4Lc
H76HHoBQNyW00K4Alzfa1rkvyiot8j3KjagpOcCazMlCsKfbZ/xa8PSt8ae8H2k5
RpoZ3o8H87k=
=FGL8
-----END PGP SIGNATURE-----

{% endraw %}
```

## Thread

+ Return to [January 1995](/archive/1995/01)

+ Return to "[Mats Bergstrom <asgaard@sos.sll.se>](/authors/mats_bergstrom_asgaard_at_sos_sll_se_)"
+ Return to "[rishab@dxm.ernet.in](/authors/rishab_at_dxm_ernet_in)"
+ Return to "[Robert Rothenberg <rrothenb@libws4.ic.sunysb.edu>](/authors/robert_rothenberg_rrothenb_at_libws4_ic_sunysb_edu_)"
+ Return to<EMAIL_ADDRESS>(Roy M. Silvernail)](/authors/roy_at_cybrspc_mn_org_roy_m_silvernail_)"
+ Return to<EMAIL_ADDRESS>(Robert Seymour)](/authors/rseymour_at_reed_edu_robert_seymour_)"
+ Return to<EMAIL_ADDRESS>(Timothy C. May)](/authors/tcmay_at_netcom_com_timothy_c_may_)"

+ 1995-01-17 (Tue, 17 Jan 95 14:26:56 PST) - [Known data havens for pirates? Doubtful](/archive/1995/01/8921b008594671410d53a0582047d7c8dc32a5d14d0583dcd8a59d3e7c7d193a) -<EMAIL_ADDRESS>  + 1995-01-18 (Tue, 17 Jan 95 19:49:47 PST) - [Re: Known data havens for pirates? Doubtful](/archive/1995/01/aeb0620eec40b3a59c98bc4971640633843e0a3ace1e0e53507985ea122c486c) -<EMAIL_ADDRESS>(Robert Seymour)_
  + 1995-01-18 (Tue, 17 Jan 95 23:30:20 PST) - [Re: Known data havens for pirates? Doubtful](/archive/1995/01/af0bc1f520c60e13b7a213b5029d8e0122a96d77b961ca09e2544eaea4560f2b) -<EMAIL_ADDRESS>(Timothy C. May)_
    + 1995-01-18 (Wed, 18 Jan 95 03:27:49 PST) - [Re: Known data havens for pirates? Doubtful](/archive/1995/01/83af420d1522f91a937580160797d3d7e53af373419d64463842aada9d904e6f) - _Robert Rothenberg<EMAIL_ADDRESS>  + 1995-01-19 (Wed, 18 Jan 95 17:45:03 PST) - [copyright and data havens](/archive/1995/01/6509f2cdad78b8fceffe85ef2ab404a5197f77618a3b102e1a8434efe5476974) - _Mats Bergstrom<EMAIL_ADDRESS>    + 1995-01-19 (Wed, 18 Jan 95 20:38:55 PST) - Re: copyright and data havens -<EMAIL_ADDRESS>(Roy M. Silvernail)_

license: MIT

branch_name: refs/heads/master

extension: md

gha_license_id: MIT

license_type: permissive


text:
# Photos13

Photos13 is a sample app that requires iOS 13 and Xcode 11. It is a photo app that illustrates the use of: a) scrolling view using DisplayLink, b)UICollectionView that uses UICollectionViewDiffableDataSource and UICollectionViewCompositionalLayout, c)properly handling getting access to the camera roll and using your photos, d) use of the Coordinator pattern, among other things. For me, the coolest thing is UICollectionViewCompositionalLayout which lets you create interesting layouts that would be very time-consuming otherwise.

In the screenshot below the top row is the view that scrolls horizontally using CADisplayLink. Below it is the collection view that uses UICollectionViewCompositionalLayout. The section with the Koala bear scrolls horizontally, while the other section use layouts made easy by the new layout.

|![Screenshot](Photos13.gif)

## License

Photos13 is licensed under the Unlicense. See the LICENSE file for more information, but basically this is sample code and you can do whatever you want with it.

license: Unlicense


text:
# rammon
Displays a bar graph of RAM usage.

The graph goes from 0% at the bottom to 100% at the top, showing combined RAM use of all cores/threads. It's 100 px high, so each pixel vertically equals one percent.

Horizontally, the graph is 200px wide, so being set to 4 FPS, it shows the last 50 s.

Data is read directly ``/proc/meminfo`` (only works on Linux), using ``MemTotal`` and ``MemAvailable`` entries.  This is forked from cpumon, and shares the majority of its code with it.

# Author

Based heavily on cpumon Martin C. Doege

+ github: https://github.com/mdoege

+ date: 5 Jun 2020

Change from CPU usage to RAM made by pavo

+ github: https://github.com/callmepavo

+ date: 9 July 2020


text:
# Module 5: Processing big data using Azure Data Lake Analytics

# Lab: Processing big data using Data Lake Analytics

## Exercise 1: Create and test a Data Lake Analytics job

#### Task 1: Prepare data files for a local Data Lake Analytics instance
1.  Ensure that the **MT17B-WS2016-NAT**, **20776A-LON-DC**, and **20776A-LON-DEV** virtual machines are running, and then log on to **20776A-LON-DEV** as **ADATUM\\AdatumAdmin** with the password **Pa55w.rd**.
2.  On the taskbar, click **File Explorer**.
3.  In File Explorer, go to **E:\\Labfiles\\Lab05\\Exercise1**, and copy the **speeds** folder tree to the clipboard.
4.  In File Explorer, go to **C:\\Users\\AdatumAdmin\\AppData\\Local\\USQLDataRoot**, and paste the speeds folder tree.
    Note that this folder contains sample speed camera data for three hours—each hour's data is in a separate folder.
5.  In File Explorer, go to **C:\\Users\\AdatumAdmin\\AppData\\Local\\USQLDataRoot\\speeds\\2017\\08\\30\\16**, and double-click **speeds1**.
    Note that, for each hour, the speeds1 file contains the camera ID, vehicle registration, recorded speed, speed limit at the camera location, camera location (latitude and longitude), and time for each record.
6.  Close Microsoft Excel, without saving any changes.

#### Task 2: Create a new Data Lake project
1.  On the Start menu, type **Visual Studio 2017**, and then press Enter.
2.  In Visual Studio, on the **File** menu, point to **New**, and then click **Project**.
3.  In the **New Project** dialog box, in the **Templates** list, expand **Azure Data Lake**, and then click **U-SQL (ADLA)**.
4.  In the **Name** box, type **CameraSpeeds**, and then click **OK**.
5.  In Solution Explorer, double-click **Script.usql**.
6.  In the **Script.usql** pane, type the following code (this code can be copied from **E:\\Labfiles\\Lab05\\Exercise1\\SpeedsJob1.usql**):
	```
	@cameraData =
	EXTRACT CameraID string,
	VehicleRegistration string,
	Speed int,
	SpeedLimit int,
	LocationLatitude double,
	LocationLongitude double,
	Time DateTime
	FROM "/speeds/2017/08/30/16/speeds1.csv"
	USING Extractors.Csv(skipFirstNRows: 1);

    @avgspeeds =
	SELECT CameraID, AVG(Speed) AS AvgSpeed
	FROM @cameraData
	GROUP BY CameraID;

    OUTPUT @avgspeeds
	TO "/AverageSpeeds.csv"
	USING Outputters.Csv(outputHeader: true);
	```
Note that, in the first statement (@cameraData), because the source data first line contains headers, you need a command to skip over it. In the second statement (@avgspeeds), the code creates a rowset that reads the file, and calculates the average speed reported by each speed camera. In the third statement (@avgspeeds), the code saves the results to a CSV file named AverageSpeeds.csv, and includes the column header data.

#### Task 3: Run a Data Lake Analytics job locally
1.  To submit the job using the (Local) ADLA account, click **Submit**.
2.  The job will open a command window; when the job has completed, note the **Success** status, and then press Spacebar to close the command window.
3.  Examine the simple job graph by hovering the mouse over the SV1 Extract step.
4.  In the **Local Run Results** pane, double-click **AverageSpeeds.csv**.
    Note that this file contains the data generated by the job; the file should contain 50 lines (one for each camera), displaying the camera ID and the average speed reported.

#### Task 4: Create a Data Lake Analytics account
1.  On the Start menu, type **Internet Explorer**, and then press Enter.
2.  In Internet Explorer, go to **http://portal.azure.com**, and sign in using the Microsoft account that is associated with your Azure Learning Pass subscription.
3.  In the Azure portal, click **+ New**, click **Data + Analytics**, and then click **Data Lake Analytics**.
4.  On the **New Data Lake Analytics Account** blade, in the **Name** box, type **speedsdla*<your name><date>***.
5.  Under **Resource Group**, click **Use existing**, and then click **CamerasRG**.
6.  In the **Location** list, select the same location as you used for the Data Lake Store that you created in Lab 2.
7.  Click **Data Lake Store**.
8.  On the **Select Data Lake Store** blade, click **adls*<your name><date>***.
9.  Leave all other settings at their defaults, and click **Create**.
10. Wait until the account has deployed before continuing with the lab.

#### Task 5: Prepare data files for a cloud Data Lake Analytics instance
1.  In the Azure portal, click **All resources**, and then click **adls*<your name><date>***.
2.  On the **adls*<your name><date>*** blade, click **Data Explorer**, and then click **New Folder**.
3.  In the **Create new folder** dialog box, type **speeds**, and then click **OK**.
4.  On the **adls*<your name><date>*** blade, click **speeds**, and then click **New Folder**.
5.  In the **Create new folder** dialog box, type **2017**, and then click **OK**.
6.  On the **adls*<your name><date>*** blade, click **2017**, and then click **New Folder**.
7.  In the **Create new folder** dialog box, type **08**, and then click **OK**.
8.  On the **adls*<your name><date>*** blade, click **08**, and then click **New Folder**.
9.  In the **Create new folder** dialog box, type **30**, and then click **OK**.
10. On the **adls*<your name><date>*** blade, click **30**, and then click **New Folder**.
11. In the **Create new folder** dialog box, type **16**, and then click **OK**.
12. On the **adls*<your name><date>*** blade, click **New Folder**.
13. In the **Create new folder** dialog box, type **17**, and then click **OK**.
14. Repeat steps 12-13 to create a new folder called **18**.
15. On the **adls*<your name><date>*** blade, click **16**, and then click **Upload**.
16. On the **Upload files** blade, click the **file** button.
17. In the **Choose File to Upload** dialog box, go to **E:\\Labfiles\\Lab05\\Exercise1\\speeds\\2017\\08\\30\\16**, click **speeds1.csv**, and then click **Open**.
18. On the **Upload files** blade, click **Add selected files**.
19. When the file has uploaded, close the **Upload files** blade.
20. In the breadcrumb trail, click **30**.
21. Repeat steps 15-20 to upload the **speeds2.csv** and **speeds3.csv** files to folders **17** and **18** respectively.

#### Task 6: Run a Data Lake Analytics job in the cloud
1.  Switch to Visual Studio.
2.  In the **Script.usql** pane, in the toolbar, in the **(Local)** list, click **speedsdla*<your name><date>***.
    Note, if your Azure Data Lake Analytics account is not shown, on the **View** menu, click **Server Explorer**, and then enter the credentials of your Azure account.
3.  Click **Submit**.
4.  When the job has completed, examine the simple job graph by hovering the mouse over the **SV1 Extract** step to see the processing details.
5.  Switch to the Azure portal.
6.  In Azure Data Lake Explorer, in the breadcrumb trail, click your root folder (**adls*<your name><date>***).
7.  Verify that **AverageSpeeds.csv** has been created.
8.  Click **AverageSpeeds.csv** and verify that it contains the same data as before.
9.  Close the **File Preview** blade.

#### Task 7: Edit a Data Lake Analytics job to add rolling averages
1.  Switch to Visual Studio.
2.  In the **Script.usql** pane, at the end of the existing script, type the following code (this code can be copied from **E:\\Labfiles\\Lab05\\Exercise1\\SpeedsJob2.usql**):
	```
	@rollingAvgSpeeds =
	SELECT CameraID, Time, Speed, AVG(Speed) OVER (PARTITION BY CameraID ORDER BY Time ROWS 25 PRECEDING) AS RollingAverage
	FROM @cameraData;

    OUTPUT @rollingAvgSpeeds
	TO "/RollingAverageSpeeds.csv"
	USING Outputters.Csv(outputHeader: true);
	```
Note that the first statement (@rollingAvgSpeeds) creates another rowset that calculates the average speed over the last 25 observations for each speed camera.
The second statement (OUTPUT @rollingAvgSpeeds) outputs the CameraID, observation time, and speed observed at that time—in addition to the rolling average—and writes the results out to another CSV file (with header).

#### Task 8: Run the Data Lake Analytics updated job
1.  In the second list box, ensure that **speedsdla*<your name><date>*** is shown, and then click **Submit**.
2.  Note that, this time, the job graph is more complex, and that the two rowsets are generated in parallel.
3.  When the job has completed, examine the job graph; you will now see two parallel processes.
4.  Hover the mouse over the **SV2 Process** and **SV3 Process** steps to see the processing details.
5.  Close the **Job View** pane.
6.  On the **File** menu, click **Close Solution**.
7.  Switch to the Azure portal.
8.  On the **adls*<your name><date>*** blade, verify that **RollingAverageSpeeds.csv** has been created.
9.  Click **RollingAverageSpeeds.csv** and view the file contents.
10. Close the **File Preview** blade.

>**Result**: At the end of this exercise, you will have prepared data files for a local Data Lake Analytics instance, created a new Data Lake project, and run this Data Lake Analytics job locally. Also, you will have created a Data Lake Analytics account, prepared data files for a cloud Data Lake Analytics instance, run this Data Lake Analytics job in the cloud—then edited the job to add rolling averages, and run the updated job.

## Exercise 2: Use Data Lake Analytics with data joins

#### Task 1: Prepare data files for a local Data Lake Analytics instance
1.  In File Explorer, go to **E:\\Labfiles\\Lab05\\Exercise2**, and copy **ownerdata.csv** to the clipboard.
2.  In File Explorer, go to **C:\\Users\\AdatumAdmin\\AppData\\Local\\USQLDataRoot**, create a new folder called **vehicles**, and paste **ownerdata.csv** into the vehicles folder.
3.  Double-click **ownerdata.csv**, and note that this file contains the names and addresses of the owner for each vehicle.
4.  Close Microsoft Excel, without saving any changes.

#### Task 2: Create a new Data Lake project

1.  Switch to Visual Studio.
2.  On the **File** menu, point to **New**, and then click **Project**.
3.  In the **New Project** dialog box, in the **Templates** list, expand **Azure Data Lake**, and then click **U-SQL (ADLA)**.
4.  In the **Name** box, type **SpeedAnalyzer**, and then click **OK**.
5.  In the **Script.usql** pane, type the following statement to create a new database named **VehicleData** (if it doesn't already exist); all the code for this exercise can be copied from **E:\\Labfiles\\Lab05\\Exercise2\\VehiclesJobCmds.txt**:
	```
	CREATE DATABASE IF NOT EXISTS VehicleData;
	```
6.  In the **Script.usql** pane, add the following statement to create a table for holding speed camera information, and to index and hash the data by the speed camera ID:
	```
	CREATE TABLE IF NOT EXISTS VehicleData.dbo.Speeds
	(
	CameraID string,
	VehicleRegistration string,
	Speed int,
	SpeedLimit int,
	LocationLatitude double,
	LocationLongitude double,
	Time DateTime,
	INDEX speedidx
	CLUSTERED (CameraID ASC)
	DISTRIBUTED BY HASH (CameraID)
	);
	```
7.  In the **Script.usql** pane, add the following statement to remove any existing data from the table:
	```
	TRUNCATE TABLE VehicleData.dbo.Speeds;
	```
8.  In the **Script.usql** pane, add the following statement to declare a variable that uses pattern matching to specify the file name:
	```
	DECLARE @speedCameraData string = @"/speeds/{Date:yyyy}/{Date:MM}/{Date:dd}/{Hour}/{Filename}";
	```
9.  In the **Script.usql** pane, add the following statement to create an extractor to retrieve speed data from the selected CSV file; this requires defining virtual fields for the date, hour, and filename components of the file, in addition to the fields within the file (the first line contains headers, so skip over it):
	```
	@speedData =
	EXTRACT CameraID string,
	VehicleRegistration string,
	Speed int,
	SpeedLimit int,
	LocationLatitude double,
	LocationLongitude double,
	Time DateTime,
	Date DateTime,
	Hour int,
	Filename string
	FROM @speedCameraData
	USING Extractors.Csv(skipFirstNRows: 1);
	```
10.  In the **Script.usql** pane, add the following statement to declare variables that specify the date and hour for the data to process (there are three hours of data—16-18, for August 30):
	```
	DECLARE @selectedDate DateTime = new DateTime(2017, 8, 30);
	DECLARE @hour int = 16; // Could also be 17 or 18
	```
11.  In the **Script.usql** pane, add the following statement to create a rowset that fetches the data for the specified date and time, and to ensure that only CSV files are read (this is specified using the Filename variable):
	```
	@speedRecords =
	SELECT CameraID, VehicleRegistration, Speed, SpeedLimit, LocationLatitude, LocationLongitude, Time
	FROM @speedData
	WHERE Date == @selectedDate AND Hour == @hour AND Filename LIKE "%.csv";
	```
12.  In the **Script.usql** pane, add the following statement to save the data to the Speeds table in the database for further analysis later:
	```
	INSERT INTO VehicleData.dbo.Speeds (CameraID, VehicleRegistration, Speed, SpeedLimit, LocationLatitude, LocationLongitude, Time)
	SELECT CameraID, VehicleRegistration, Speed, SpeedLimit, LocationLatitude, LocationLongitude, Time
	FROM @speedRecords;
	```

#### Task 3: Test the Data Lake Analytics job locally
1.  Click **Submit**, to submit the job using the (Local) ADLA account in Visual Studio.
2.  The job will open a command window; when the job has completed, note the **Success** status, and then press Spacebar to close the command window.
3.  Examine the job graph by hovering the mouse over the **SV1** – **SV3** steps to see the processing details.
4.  In Server Explorer, expand **Azure**, expand **Data Lake Analytics**, expand **(Local)**, expand **U-SQL Databases**, expand **VehicleData**, expand **Tables**, right-click **dbo.Speeds**, and then click **Preview**; you might need to click **Refresh** to see **VehicleData**.
5.  Note that the preview has run another Data Lake Analytics job that fetches the data in the table and returns it as a TSV file.
6.  Close the **Table Preview** pane.

#### Task 4: Modify the Data Lake Analytics job to use joins

1.  In the **Script.usql** pane, add the following statement to create a table for recording fines and summonses:
	```
	CREATE TABLE IF NOT EXISTS VehicleData.dbo.Fines
	(
	CameraID string,
	VehicleRegistration string,
	Speed int,
	SpeedLimit int,
	TimeCaught DateTime,
	PenaltyType string, // "P" for a Fixed Penalty (£100), "S" for a Summons (for excessive speeds, fine amount to be determined by a magistrate)
	PenaltyIssuedWhen DateTime,
	PenaltyIssuedTo string, // Full name of owner, including title
	OffenderAddress string, // Full address of the owner
	INDEX finesidx
	CLUSTERED (CameraID ASC)
	DISTRIBUTED BY HASH (CameraID)
	);
	```
2.  In the **Script.usql** pane, add the following statement to define a rowset that fetches the vehicle owner data from the ownerdata.csv file (and skip the headers in the first line):
	```
	@vehicleOwnerData =
	EXTRACT VehicleRegistration string,
	Title string,
	Forename string,
	Surname string,
	AddressLine1 string,
	AddressLine2 string,
	AddressLine3 string,
	AddressLine4 string
	FROM "/vehicles/ownerdata.csv"
	USING Extractors.Csv(skipFirstNRows: 1);
	```
3.  In the **Script.usql** pane, add the following statement to create a rowset that joins the speed data with the vehicle owner data, so that the application will know where to send fines/summonses. The code combines owner title and names into a single string, and concatenates the address information into another single string (with semicolons between each line of the address):
	```
	@speedsAndOwners =
	SELECT S.CameraID AS CameraID, S.VehicleRegistration AS VehicleRegistration, S.Speed AS Speed, S.SpeedLimit AS SpeedLimit,
	S.Time AS TimeCaught, O.Title + " " + O.Forename + " " + O.Surname AS PenaltyIssuedTo,
	O.AddressLine1 + "; " + O.AddressLine2 + "; " + O.AddressLine3 + "; " + O.AddressLine4 AS OffenderAddress
	FROM @speedRecords AS S
	JOIN @vehicleOwnerData AS O
	ON S.VehicleRegistration == O.VehicleRegistration;
	```
4.  In the **Script.usql** pane, add the following statement to process each row of data from the speed camera as follows:
 -   If the speed shown is less than the speed limit plus 5 percent, do not raise a fixed penalty or summons.
 -   If the speed shown is between the speed limit plus 5 percent and the speed limit plus 30 mph, generate a fixed penalty fine.
 -   If the speed is greater than the speed limit plus 30 mph, then issue a summons to appear in court.
	```
	@fines =
	SELECT *, DateTime.UtcNow AS PenaltyIssuedWhen,
	Speed >= SpeedLimit * 1.05 && Speed <= SpeedLimit + 30 ? "P" :
	Speed >= SpeedLimit + 30 ? "S" : "" AS PenaltyType
	FROM @speedsAndOwners;
	```
5.  In the **Script.usql** pane, add the following statement to save the details for fines and summonses to the Fines table in the database, and do not record data for legal traffic speeds:
	```
	INSERT INTO VehicleData.dbo.Fines (CameraID, VehicleRegistration, Speed, SpeedLimit, TimeCaught, PenaltyType, PenaltyIssuedWhen, PenaltyIssuedTo, OffenderAddress)
	SELECT CameraID, VehicleRegistration, Speed, SpeedLimit, TimeCaught, PenaltyType, PenaltyIssuedWhen, PenaltyIssuedTo, OffenderAddress
	FROM @fines
	WHERE !String.IsNullOrEmpty(PenaltyType);
	```
6.  In the **Script.usql** pane, add the following statement to copy the data to an output file; note that you cannot read data from the VehicleData.dbo.Fines table because it could be mutating:
	```
	@results =
	SELECT CameraID, VehicleRegistration, Speed, SpeedLimit, TimeCaught, PenaltyType, PenaltyIssuedWhen, PenaltyIssuedTo, OffenderAddress
	FROM @fines
	WHERE !String.IsNullOrEmpty(PenaltyType);

    OUTPUT @results
	TO "/FinesAndSummonses.csv"
	USING Outputters.Csv(outputHeader: true);
	```

#### Task 5: Test the modified Data Lake Analytics job locally
1.  Click **Submit**, to submit the job using the (Local) ADLA account.
2.  The job will open a command window; when the job has completed, note the **Success** status, and then press Spacebar to close the command window.
3.  Examine the job graph and note that there are now more processing steps shown.
4.  In Server Explorer, under **Azure\\Data Lake Analytics\\(Local)\\U-SQL Databases\\VehicleData**, right-click the **Tables** folder, and then click **Refresh**.
5.  Right-click **dbo.Fines**, and then click **Preview**.
6.  Verify that the **Fines** table contains the details of any fines and summonses.
7.  Close the **Table Preview** pane.

#### Task 6: Run the updated Data Lake Analytics job in the cloud
1.  Switch to the Azure portal.
2.  Click **All resources**, click **adls*<your name><date>***, and then click **Data Explorer**.
3.  In Data Explorer, ensure you are at the root folder, and then click **New Folder**.
4.  In the **Create new folder** dialog box, type **vehicles**, and then click **OK**.
5.  Click **vehicles**, and then click **Upload**.
6.  On the **Upload files** blade, click the **file** button.
7.  In the **Choose File to Upload** dialog box, go to **E:\\Labfiles\\Lab05\\Exercise2**, click **ownerdata.csv**, and then click **Open**.
8.  On the **Upload files** blade, click **Add selected files**.
9.  When the file has uploaded, close the **Upload files** blade.
10. Switch to Visual Studio.
11. In the **Script.usql** pane, in the toolbar, in the **(Local)** list, click **speedsdla*<your name><date>***, and then click **Submit**.
12. When the job has completed, examine the job graph by hovering the mouse over the processing steps.
13. Switch to the Azure portal.
14. In Azure Data Lake Explorer, in the breadcrumb trail, click **adls*<your name><date>***.
15. Verify that **FinesAndSummonses.csv** has been created.
16. Click **FinesAndSummonses.csv** and verify that it contains the same data as before.
17. Close the **File Preview** blade.
18. Switch to Visual Studio.
19. In Server Explorer, under **Azure**, under **Data Lake Analytics**, under **speedsdla*<your name><date>***, expand **U-SQL Databases**, expand **VehicleData**, and then expand **Tables**; note that you might need to click **Refresh** to see **VehicleData**.
20. In the **Tables** folder, right-click **dbo.Speeds**, and then click **Preview by Running a Job**.
21. In the **Preview by Running a Job** dialog box, click **Submit**.
22. When the preview job has completed, click the **Data** tab, and in the **Job Outputs** pane, double-click **Speeds\_Preview.tsv**.
23. Verify that the Speeds table contains the same data as when the job was run locally.
24. Close the **File Preview** and **Job View** panes.
25. In Server Explorer, right-click **dbo.Fines**, and then click **Preview by Running a Job**.
26. In the **Preview by Running a Job** dialog box, click **Submit**.
27. When the preview job has completed, click the **Data** tab, and in the **Job Outputs** pane, double-click **Fines\_Preview.tsv**.
28. Verify that the table contains the same data as when the job was run locally.
29. Verify that the **Fines** table contains the same details of fines and summonses.
30. Close Visual Studio without saving any changes.

>**Result**: At the end of this exercise, you will have prepared data files for a local Data Lake Analytics instance, created a new Data Lake project and tested this job locally, then modified the job to use joins, and tested the job locally before running it in the cloud.

## Exercise 3: Use Data Lake Analytics with SQL Database

#### Task 1: Create a SQL Database
1.  In the Azure portal, click **+ New**, click **Databases**, and then click **SQL Database**.
2.  On the **SQL Database** blade, in the **Database name** box, type **VehicleDB**.
3.  Under **Resource group**, click **Use existing**, and then click **CamerasRG**.
4.  In the **Location** list, select the same location as you used for the Data Lake Store.
5.  In **Select source**, click **Blank database**.
6.  Click **Server**.
7.  On the **New server** blade, enter the following details, and then click **Select**:
 -   **Server name**: camerasdbs*<your name><date>*
 -   **Server admin login**: student
 -   **Password**: Pa55w.rd
 -   **Confirm password**: Pa55w.rd
 -   **Location**: select the same location as you used for the Data Lake Store that you created in Exercise 1
 -   Ensure that the **Allow azure services to access server** check box is selected
8.  On the **SQL Database** blade, click **Pricing tier**.
9.  On the **Configure performance** blade, click **Basic**, and then click **Apply**.
10.  On the **SQL Database** blade, leave all other settings at their defaults, and click **Create**.
11.  Wait until the database and server have deployed before continuing with the lab.
12.  In the Azure portal, click **All resources**, and then click **camerasdbs*<your name><date>***.
13.  On the **camerasdbs*<your name><date> ***blade, under **SETTINGS**, click **Firewall/Virtual Networks**, and then click **Add client IP**.
14.  Ensure that **Allow access to Azure services** in set to **ON**, and then click **Save**.
15.  In the **Success** dialog box, click **OK**.

#### Task 2: Upload data to SQL Database and add an index

1.  On the Start menu, type **Microsoft SQL Server Management Studio**, and then press Enter.
2.  In the **Connect Server** dialog box, in the **Server name** box, type **camerasdbs*<your name><date>*.database.windows.net**.
3.  In the **Authentication** list, click **SQL Server Authentication**.
4.  In the **Login** box, type **student**.
5.  In the **Password** box, type **Pa55w.rd**, and then click **Connect**.
6.  In SQL Server Management Studio, in Object Explorer, expand **Databases**, right-click **VehicleDB**, point to **Tasks**, and then click **Import Data**.
7.  In the **SQL Server Import and Export Wizard** dialog box, on the **Welcome to SQL Server Import and Export Wizard** page, click **Next**.
8.  On the **Choose a Data Source** page, in the **Data source** list, click **Flat File Source**, and then click **Browse**.
9.  In the **Open** dialog box, go to **E:\\Labfiles\\Lab05\\Exercise3**, change the file type to **CSV files (\*.csv)**, click **ownerdata.csv**, and then click **Open**.
10. On the **Choose a Data Source** page, leave the **Column names in the first data row** check box selected, and then click **Next**.
11. On the **Choose a Data Source** page, click **Next**.
12. On the **Choose a Destination** page, in the **Destination** list, select **SQL Server Native Client 11.0**.
13. Ensure that the **Server name** is set to **camerasdbs*<your name><date>*.database.windows.net**, and then click **Use SQL Server Authentication**.
14. In the **User name** box, type **student**, in the **Password** box, type **Pa55w.rd**, and then click **Next**.
15. On the **Select Source Tables and Views** page, in the **Destination** column, accept the default table name **[dbo].[ownerdata]**, and then click **Next**.
16. On the **Save and Run Package** page, leave the settings at the defaults, and then click **Next**.
17. On the **Complete the Wizard** page, click **Finish**.
18. Verify that the data has been uploaded successfully (the operation should upload 1782 records to the database). If you get data truncation warnings, you can ignore them.
19. On the **The execution was successful** page, click **Close**.
20. In Object Explorer, under **Databases**, expand **VehicleDB**, expand **Tables**, right-click **dbo.ownerdata**, and then click **Select Top 1000 Rows**.
21. Verify that the data has been uploaded to the table.
22. In Object Explorer, expand **dbo.ownerdata**, right-click **Indexes**, and then click **New Index**.
23. In the SQL command pane, replace the template index with the following SQL command (the SQL code can be copied from **E:\\Labfiles\\Lab05\\Exercise3\\Index.txt**):
	```
	CREATE INDEX RegIdx
	ON [dbo].[ownerdata]
	(
	[Vehicle Registration]
	)
	GO
	```
24.  Click **Execute**.
25.  Close SQL Server Management Studio, without saving any scripts or SQL files.

#### Task 3: Store a SQL Database credential in the Data Lake Analytics catalog using PowerShell
1.  In File Explorer, go to **E:\\Labfiles\\Lab05\\Exercise3**, right-click **Setup.cmd,** and then click **Run as administrator**.
2.  In the **User Account Control** dialog box, click **Yes**.
3.  On the Start menu, type **Windows PowerShell ISE**, right-click **Windows PowerShell ISE**, and then click **Run as administrator**.
4.  In the **User Account Control** dialog box, click **Yes**.
5.  In Windows PowerShell ISE, on the **File** menu, click **Open**.
6.  In the **Open** dialog box, go to **E:\\Labfiles\\Lab05\\Exercise3**, click **CreateCredential.ps1**, and then click **Open**, to open the following script:
	```
	# Install Azure PowerShell modules
	Install-Module AzureRM -AllowClobber;
	Import-Module AzureRM;

    # Login to Azure
	Login-AzureRmAccount;

    # Create credential
	New-AzureRmDataLakeAnalyticsCatalogCredential -AccountName "<name of your ADLA account>" -DatabaseName "VehicleData" -CredentialName "VehicleOwnerDataCredential" -Credential (Get-Credential) -DatabaseHost "<name of your Azure SQL Database Server>.database.windows.net" -Port 1433;

    # Remove credential
	# Remove-AzureRmDataLakeAnalyticsCatalogCredential -AccountName "<name of your ADLA account>" -DatabaseName "VehicleData" -Name "VehicleOwnerDataCredential";
	```
7.  Edit the script, replacing **<name of your ADLA account>** with **speedsdla*<your name><date>***.
8.  Edit the script, replacing **<name of your Azure SQL Database Server>** with **camerasdbs*<your name><date>***.
9.  On the **File** menu, click **Save**.
10.  Highlight lines 1-3, and then on the toolbar, click **Run Selection**, to install the latest AzureRM cmdlets.
11.  If you get a warning message about installing modules from an untrusted repository, click **Yes to All**.
12.  Highlight lines 5-6, and then on the toolbar, click **Run Selection**, to log in to Azure.
13.  In the **Sign in to your account** dialog box, enter the details of the Microsoft account that is associated with your Azure Learning Pass subscription, and then click **Sign in**.
14.  Highlight lines 8-9, and then on the toolbar, click **Run Selection**, to create a credential in your ADLA catalog that contains the connection information for connecting to your Azure SQL Database.
15.  In the **cmdlet** **Get-Credential at command pipe** dialog box, in the **User name** box, type **student**, in the **Password** box type **Pa55w.rd**, and then click **OK**. These are the credentials for accessing the database server—these credentials are stored, encrypted, as part of the credential in the catalog.
16. Note that the **-DatabaseName** parameter is the name of the database in the ADLA catalog that will hold the credential, and is *not* the name of the SQL database to which the credential connects!
17. Note also that the script shows the **Remove-AzureRmDataLakeAnalyticsCatalogCredential** command (commented out); this is how you remove a credential from the catalog. You could use this command if you make a mistake when creating the credential; you can then recreate the credential with the correct information.

#### Task 4: Configure a Visual Studio-based Data Lake Analytics job to use stored credentials
1.  On the Start menu, type **Visual Studio 2017**, and then press Enter.
2.  In Visual Studio, click **Open Project / Solution**.
3.  In the **Open Project** dialog box, go to the **E:\\Labfiles\\Lab05\\Exercise3\\Starter-SpeedAnalyzer** folder, click **SpeedAnalyzer.sln**, and then click **Open**.
    Note that this project is a sample solution for the U-SQL job created in Exercise 2.
4.  If the **Select Azure Account** dialog box appears, select your Azure subscription, and then click **OK**.
5.  In Solution Explorer, double-click **Script.usql**.
6.  In the **Script.usql** pane, locate the comment **// TODO: Switch the context to the VehicleData database in the catalog**, then on the next line, add the following USE statement for the credential store (the U-SQL commands can be copied from **E:\\Labfiles\\Lab05\\Exercise3\\UsqlCmds.txt**):
	```
	USE DATABASE VehicleData;
	```
7.  In the **Script.usql** pane, locate the comment **// TODO: Drop the VehicleOwner Data Source if it already exists**, then on the next line, add the following statement:
	```
	DROP DATA SOURCE IF EXISTS VehicleOwner;
	```
8.  In the **Script.usql** pane, locate the comment **// TODO: Create the VehicleOwner data source**, then on the next line, add the following statement to create the data source:
	```
	CREATE DATA SOURCE VehicleOwner
	FROM AZURESQLDB
	WITH (
	PROVIDER_STRING = "Database=VehicleDB;Trusted_Connection=False;Encrypt=True;",
	CREDENTIAL = VehicleOwnerDataCredential ,
	REMOTABLE_TYPES = (bool, byte, sbyte, short, int, long, decimal, float, double, string, DateTime)
	);
```
9.  In the **Script.usql** pane, locate the comment **// TODO: Fetch the vehicle owner data from the SQL database**, then on the next line, add the following statement to create a rowset from the dbo.ownerdata table held by the SQL database, using the data source you just created:
	```
	@vehicleOwnerData =
	SELECT [Vehicle Registration] AS VehicleRegistration, [Title] AS Title, [Forename] AS Forename, [Surname] AS Surname,
	[Address Line 1] AS AddressLine1, [Address Line 2] AS AddressLine2, [Address Line 3] AS AddressLine3, [Address Line 4] AS AddressLine4
	FROM EXTERNAL VehicleOwner
	LOCATION "dbo.ownerdata";
	```
10.  Leave the remainder of the script as it is; it is unchanged from Exercise 2, and performs the same tasks, but now uses the data retrieved from the SQL database instead of a flat file.

#### Task 5: Run a Data Lake Analytics job using stored credentials to access data in SQL Database
1.  In the **Script.usql** pane, in the toolbar, click the **(Local)** list, click **speedsdla*<your name><date>***, and then click **Submit**.
    Note that this script must be run in the cloud, because the credential was created in the ADLA account, not locally.
2.  When the job has completed, examine the job graph by hovering the mouse over the processing steps.
3.  Switch to the Azure portal.
4.  Click **All resources**, click **adls*<your name><date>***, and then click **Data Explorer**.
5.  In Data Lake Explorer, in the root folder, verify that **FinesAndSummonses.csv** has been created.
6.  Click **FinesAndSummonses.csv** and verify that it contains the same data as before.
7.  Close the **File Preview** blade.
8.  Switch to Visual Studio.
9.  In Server Explorer, expand **Azure**, expand **Data Lake Analytics**, then expand **speedsdla*<your name><date>***, expand **U-SQL Databases**, expand **VehicleData**, and then expand **Tables**; note that you might need to click **Refresh** to see **VehicleData**.
10. In the **Tables** folder, right-click **dbo.Speeds**, and then click **Preview by Running a Job**.
11. In the **Preview by Running a Job** dialog box, click **Submit**.
12. When the preview job has completed, on the **Data** tab, in the **Job Outputs** section, double-click **Speeds\_Preview.tsv**.
13. Verify that the Speeds table contains the same data as when the job was run locally.
14. Close the **File Preview** and **Job View** panes.
15. In Server Explorer, right-click **dbo.Fines**, and then click **Preview by Running a Job**.
16. In the **Preview by Running a Job** dialog box, click **Submit**.
17. When the preview job has completed, on the **Data** tab, in the **Job Outputs** section, double-click **Fines\_Preview.tsv**.
18. Verify that the table contains the same data as when the job was run locally.
19. Verify that the **Fines** table contains the same details of fines and summonses.
20. Close Visual Studio without saving any changes.

>**Result**: At the end of this exercise, you will have created a SQL Database, uploaded data to SQL Database and added an index. You will also have stored a SQL Database credential in the Data Lake Analytics catalog using PowerSheII, configured a Data Lake Analytics job to use stored credentials, and run this job using stored credentials to access data in SQL Database.

## Exercise 4: Use Data Lake Analytics to categorize data and present results using Power BI

#### Task 1: Create a new Data Lake project

1.  On the Start menu, type Visual Studio 2017, and then press Enter.
2.  In Visual Studio, on the **File** menu, point to **New**, and then click **Project**.
3.  In the **New Project** dialog box, in the **Templates** list, expand **Azure Data Lake**, then click **U-SQL (ADLA)**.
4.  In the **Name** box, type **SpeedCameraAnalytics**, and then click **OK**.
5.  In Solution Explorer, double-click **Script.usql**.
6.  You will now define a rowset variable that extracts the data from the speed camera data files (uploaded in Exercise 1) under the /speeds folder. These files are organized by date and time, but you should ensure that you read every available file. Remember that each file is in CSV format, and has a header.
7.  In the **Script.usql** pane, type the following code (all the code in this exercise can be copied from **E:\\Labfiles\\Lab05\\Exercise4\\CameraJobCmds.txt**):
	```
	// Capture the historical and current data from all available files containing speed data
	DECLARE @speedCameraData string = @"/speeds/{Date:yyyy}/{Date:MM}/{Date:dd}/{Hour}/{*}.csv";

    // Specify the format of the data in the CSV files
	@speedData =
	EXTRACT CameraID string,
	VehicleRegistration string,
	Speed int,
	SpeedLimit int,
	LocationLatitude double,
	LocationLongitude double,
	Time DateTime,
	Date DateTime,
	Hour int
	FROM @speedCameraData
	USING Extractors.Csv(skipFirstNRows: 1);
	```
8.  You will now define a rowset that retrieves the data in the CameraID and Speed fields from these files; in the **Script.usql** pane, type the following code:
	```
	// Generate a rowset that reads the relevant data from the CSV files
	@cameraData =
	SELECT CameraID, Speed, LocationLatitude, LocationLongitude
	FROM @speedData;
	```
9.  You will now define a rowset that, for each camera, calculates the number of vehicles that were travelling at less than 10 mph (this is the first "bucket"); in the **Script.usql** pane, type the following code:
	```
	// Define a rowset that calculates the number of vehicles for each camera that were travelling at less than 10 mph (this is the first "bucket")
	@lessThan10 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS LessThan10
	FROM @cameraData
	WHERE Speed < 10
	GROUP BY CameraID, LocationLatitude, LocationLongitude;
	```
10.  You will now define another rowset that, for each camera, calculates the number of vehicles that were travelling at between 10 and 29 mph (this is the second "bucket); in the **Script.usql** pane, type the following code:
	```
	// Define a rowset that calculates the number of vehicles for each camera that were travelling at between 10 and 29 mph (this is the second "bucket)
	@between10And29 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS Between10And29
	FROM @cameraData
	WHERE Speed BETWEEN 10 AND 29
	GROUP BY CameraID, LocationLatitude, LocationLongitude;
	```
11.  You will now repeat the previous code for the following speed ranges—30-49 mph, 50-69 mph, 70-99 mph, 100 mph and more; in the **Script.usql** pane, type the following code:
	```
	// Define rowsets for the following speed ranges; 30-49 mph, 50-69 mph, 70-99 mph, 100 mph and over
	@between30And49 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS Between30And49
	FROM @cameraData
	WHERE Speed BETWEEN 30 AND 49
	GROUP BY CameraID, LocationLatitude, LocationLongitude;

    @between50And69 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS Between50And69
	FROM @cameraData
	WHERE Speed BETWEEN 50 AND 69
	GROUP BY CameraID, LocationLatitude, LocationLongitude;

    @between70And99 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS Between70And99
	FROM @cameraData
	WHERE Speed BETWEEN 70 AND 99
	GROUP BY CameraID, LocationLatitude, LocationLongitude;

    @over99 =
	SELECT CameraID, LocationLatitude, LocationLongitude, COUNT(*) AS Over99
	FROM @cameraData
	WHERE Speed > 99
	GROUP BY CameraID, LocationLatitude, LocationLongitude;
	```
12.  You will now define another rowset that joins the buckets together over the CameraID column. You must use an OUTER join in case the bucket for a particular speed camera is empty (you don't want to discard all the data in the other buckets for that camera); in the **Script.usql** pane, type the following code:
	```
	// Join the speed buckets together, this requires an OUTER join, as one or more buckets for a camera could be null (empty)
	@speedAnalysis =
	SELECT a.CameraID, a.LocationLatitude, a.LocationLongitude, a.LessThan10, b.Between10And29, c.Between30And49, d.Between50And69, e.Between70And99, f.Over99
	FROM @lessThan10 AS a
	LEFT JOIN @between10And29 AS b
	ON a.CameraID == b.CameraID
	LEFT JOIN @between30And49 AS c
	ON a.CameraID == c.CameraID
	LEFT JOIN @between50And69 AS d
	ON a.CameraID == d.CameraID
	LEFT JOIN @between70And99 AS e
	ON a.CameraID == e.CameraID
	LEFT JOIN @over99 AS f
	ON a.CameraID == f.CameraID;
	```
13.  Finally, you will save the results to a CSV file named **/SpeedAnalysis.csv**, and include a header; in the **Script.usql** pane, type the following code:
	```
	OUTPUT @speedAnalysis
	TO "/SpeedAnalysis.csv"
	USING Outputters.Csv(outputHeader: true);
	```

#### Task 2: Test the Data Lake Analytics job locally then run it in the cloud
1.  Click **Submit**, to submit the job using the (Local) ADLA account in Visual Studio.
2.  The job will open a command window; when the job has completed, note the **Success** status, and then press Spacebar to close the command window.
3.  Examine the job graph by hovering the mouse over the various processing steps.
4.  In File Explorer, go to **C:\\Users\\AdatumAdmin\\AppData\\Local\\USQLDataRoot**, then double-click **SpeedAnalysis.csv** to open the file in Excel.
5.  Verify that the results show a tabular list of cameras, their locations, and the number of cars detected in each speed bucket.
6.  Close Microsoft Excel, without saving any changes.
7.  Switch to Visual Studio.
8.  In the **Script.usql** pane, in the toolbar, in the **(Local)** list, click **speedsdla*<your name><date>***, and then click **Submit**, to now run the job in the cloud.
9.  When the job has completed, examine the job graph by hovering the mouse over the various processing steps.
10. Close Visual Studio, without saving any changes.

#### Task 3: Use Power BI to visualize the data
1.  On the Start menu, type **Power BI Desktop**, and then press Enter.
2.  In the **Power BI Desktop welcome** pane, click **Get data**.
3.  In the **Get Data** dialog box, click **Azure**, click **Azure Data Lake Store**, and then click **Connect**.
4.  In the **Azure Data Lake Store** dialog box, type the following URL, replacing **<name of your Data Lake Store>** with **adls*<your name><date>***, and then click **OK**:
	```
	adl://<name of your Data Lake Store>.azuredatalakestore.net/SpeedAnalysis.csv
	```
5.  In the **Azure Data Lake Store** dialog box, if there is a **You aren't signed in** message, click **Sign in**, and authenticate using the Microsoft account that is associated with your Azure Learning Pass subscription, and then click **Connect**.
6.  In the **adl://adls*<your name><date>*.azuredatalakestore.net/SpeedAnalysis.csv** dialog box, click **Edit**.
7.  In the Query Editor, in the **Content column**, click **Binary**; if the data in the file does not appear, click **Binary** again.
8.  In the Query Editor, click **Close & Apply**.
9.  In the **Visualizations** pane, click **ArcGIS Maps for Power BI**.
10.  In the **Welcome to ArcGIS Maps for Power BI** dialog box, click **OK**.
11.  In the **Fields** pane, drag the **LocationLatitude** field to the **Latitude** box in the **Visualizations** pane.
12.  In the **Fields** pane, drag the **LocationLongitude** field to the **Longitude** box in the **Visualizations** pane.
13.  In the **Fields** pane, drag the **CameraID** field to the **Location** box in the **Visualizations** pane.
14. To make the names more meaningful, in the **Fields** pane, point to **LessThan10**, click the ellipses (**...**), and then click **Rename**.
15. Type **Less than 10 mph**, and then press Enter.
16. Repeat steps 14 and 15 to rename each of the **BetweenXAndY** fields to **Between X and Y mph** (for example, Between10And29 will be renamed to Between 10 and 29 mph).
17. Repeat steps 14 and 15 to rename **Over99** to **Over 99 mph**.
18. In the **Fields** pane, drag the **Less than 10 mph** field to the **Tooltips** box in the **Visualizations** pane.
19. In the **Fields** pane, drag the **Between 10 and 29 mph** field to the bottom of the **Tooltips** box in the **Visualizations** pane.
20. Repeat the previous step for each of the remaining speed fields, finishing with the **Over 99 mph** field.
21. You should now have all the speed fields listed under **Tooltips**, and in ascending speed order.
22. On the map object, click **Focus mode**, and view the map.
23. Zoom in and hover over any speed camera to see the detected speed values for that camera.
24. On the **File** menu, click **Save**.
25. In the **Save As** dialog box, go to **E:\\Labfiles\\Lab05\\Exercise4**, in the **File name** box, type **SpeedCameraAnalysis.pbix**, and then click **Save**.
26. Close Power BI Desktop.

#### Task 4: Lab closedown
1.  Close Windows PowerShell ISE.
2.  Do not remove the Azure resources (resource group, Stream Analytics jobs, event hub, IoT hub, and storage); these resources will be used in later labs.
3.  Close Internet Explorer.

>**Result**: At the end of this exercise, you will have created a new Data Lake project, tested the job locally, run it in the cloud, and then used Power BI to visualize the data.

©2017 Microsoft Corporation. All rights reserved.

The text in this document is available under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/legalcode), additional terms may apply. All other content contained in this document (including, without limitation, trademarks, logos, images, etc.) are **not** included within the Creative Commons license grant. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

This document is provided "as-is." Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. Some examples are for illustration only and are fictitious. No real association is intended or inferred. Microsoft makes no warranties, express or implied, with respect to the information provided here.


text:
What does puts do?
	Puts writes on the screen whatever comes after it. So that could be the solution to a math problem or a string.

What is an integer? What is a float?
	An integer is a whole number. A float is a number with a decimal point.

What is the difference between float and integer division? How would you
explain the difference to someone who doesn't know anything about programming?
	When you use division with integers, ruby will only return integers. So 9/2 will return 4 instead of the mathematically correct answer of 4.5.
```ruby
puts 24*365
puts (((60*24)*365)*10)
```

How does Ruby handle addition, subtraction, multiplication, and division of numbers?
  Ruby addition, subtraction and multiplication commands works as expected. You input the +, -, * sign, respectively, and it computes the answer. Division in ruby is different because it can recieve two answers. If dividing integers (whole numbers) ruby will only return an integer. WHen dividing floats ruby is capable of give correct answers with decimal points.

What is the difference between integers and floats?
  An integer is a whole number (a number without a decimal point). A float is a number with a decimal point.

What is the difference between integer and float division?
  When dividing a integer ruby can only return an integer. When dividing floats ruby returns answers with decimal point.

What are strings? Why and when would you use them?
  Strings are groups of letters, numbers,punctuation and essentially anything else you want thrown into it. You can use strings to make sentences, recieve informaiton for a program, or return information to a user.

What are local variables? Why and when would you use them?
  Local variables are variables that live inside a method and cannot leave. You would use them when you needed to define a method that you do not want to interfere with other parts of your code.

How was this challenge? Did you get a good review of some of the basics?
  The most difficult part was understanding how to work ruby using the terminal.

gha_language: Ruby


text:
# MXfold2
RNA secondary structure prediction using deep learning with thermodynamic integration

## Installation

### System requirements
* python (>=3.7)
* pytorch (>=1.4)
* C++17 compatible compiler (tested on Apple clang version 12.0.0 and GCC version 7.4.0) (optional)

### Install from wheel

We provide the wheel python packages for several platforms at [the release](https://github.com/mxfold/mxfold2/releases). You can download an appropriate package and install it as follows:

    % pip3 install mxfold2-0.1.2-cp310-cp310-manylinux_2_17_x86_64.whl

### Install from sdist

You can build and install from the source distribution downloaded from [the release](https://github.com/mxfold/mxfold2/releases) as follows:

    % pip3 install mxfold2-0.1.2.tar.gz

To build MXfold2 from the source distribution, you need a C++17 compatible compiler.

## Prediction

You can predict RNA secondary structures of given FASTA-formatted RNA sequences like:

    % mxfold2 predict test.fa
    >DS4440
    GGAUGGAUGUCUGAGCGGUUGAAAGAGUCGGUCUUGAAAACCGAAGUAUUGAUAGGAAUACCGGGGGUUCGAAUCCCUCUCCAUCCG
    (((((((........(((((..((((.....))))...)))))...................(((((.......)))))))))))). (24.8)

By default, MXfold2 employs the parameters trained from TrainSetA and TrainSetB (see our paper).

We provide other pre-trained models used in our paper. You can download [``models-0.1.0.tar.gz``](https://github.com/mxfold/mxfold2/releases/download/v0.1.0/models-0.1.0.tar.gz) and extract the pre-trained models from it as follows:

    % tar -zxvf models-0.1.0.tar.gz

Then, you can predict RNA secondary structures of given FASTA-formatted RNA sequences like:

    % mxfold2 predict @./models/TrainSetA.conf test.fa
    >DS4440
    GGAUGGAUGUCUGAGCGGUUGAAAGAGUCGGUCUUGAAAACCGAAGUAUUGAUAGGAAUACCGGGGGUUCGAAUCCCUCUCCAUCCG
    (((((((.((....))...........(((((.......))))).(((((......))))).(((((.......)))))))))))). (24.3)

Here, ``./models/TrainSetA.conf`` specifies a lot of parameters including hyper-parameters of DNN models.

## Training

MXfold2 can train its parameters from BPSEQ-formatted RNA sequences. You can also download the datasets used in our paper at [the release](https://github.com/mxfold/mxfold2/releases/tag/v0.1.0).

    % mxfold2 train --model MixC --param model.pth --save-config model.conf data/TrainSetA.lst

You can specify a lot of model's hyper-parameters. See ``mxfold2 train --help``. In this example, the model's hyper-parameters and the trained parameters are saved in ``model.conf`` and ``model.pth``, respectively.

## Web server

A web server is working at http://www.dna.bio.keio.ac.jp/mxfold2/.

## References

* Sato, K., Akiyama, M., Sakakibara, Y.: RNA secondary structure prediction using deep learning with thermodynamic integration. *Nat Commun* **12**, 941 (2021). https://doi.org/10.1038/s41467-021-21194-4


text:
---
layout: docs
title: Continuous Tasks
section: core
---

# Continuous Tasks
Continuous tasks are used to represent operations that can continue indefinitely but do not have a defined condition for when they should stop. Actions such as spinning a wheel at a given velocity or continuously shooting balls can be represented by a continuous event. The largest difference between continuous and finite tasks is that continuous tasks cannot be sequenced, since there is no definite point where a continuous task ends.

## Creating a Continuous Task
To create a continuous task, extend the `ContinuousTask` abstract class and implement the `onStart` and `onEnd` methods.

```scala
class DriveAtVelocity(velocity: Double) extends ContinuousTask {
  def onStart(): Unit = { ... }
  def onEnd(): Unit = { ... }
}
```

## Parallel Continuous Tasks
As continuous tasks have no defined end, there is no way to chain continuyous tasks to run one after the other. Instead, the only continuous task combination method is `and`, which allows two continuous tasks to be run in parallel.

```scala
val collectAndShoot: ContinuousTask = new CollectItems().and(new ShootIntoGoal())
```

## Conversion to Finite Tasks
Sometimes, a continuous task needs to be integrated into a finite task sequence by having the continuous task only run for a specified duration. To convert a continuous task to a finite task that runs the action for the specified duration, `ContinuousTask` has a method `forDuration(time)` that produces a new finite task that wraps the continuous task.

```scala
val fiveSecondShoot: FiniteTask = new ShootIntoGoal().forDuration(Seconds(5))
```

gha_language: Scala


text:
# brew cask

> Package manager for macOS applications distributed as binaries.
> Homepage: <https://github.com/Homebrew/homebrew-cask>.

- Search for formulas and casks:

`brew search {{text}}`

- Install a cask:

`brew cask install {{cask_name}}`

- List all installed casks:

`brew cask list`

- List installed casks that have newer versions available:

`brew cask outdated`

- Upgrade an installed cask to its latest version:

`brew cask upgrade {{cask_name}}`

- Uninstall a cask:

`brew cask uninstall {{cask_name}}`

- Uninstall a cask and remove related settings and files:

`brew cask zap {{cask_name}}`

- Display information about a given cask:

`brew cask info {{cask_name}}`

repository: chrillebile/tldr

repo_name: chrillebile/tldr


text:
# Midotis lingua Fr. SPECIES

#### Status
ACCEPTED

#### According to
Index Fungorum

#### Published in
Elench. fung. (Greifswald) 2: 29 (1828)

#### Original name
Midotis lingua Fr.

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
Chainer with Speech Emotion Recognition
====

This tool is the Speech Emotion Recognition

## Japanese Reference Page

http://tech-sketch.jp/2015/10/chainer.html

## Description
This tool is recognition emotion by humam voice

If you see the detail about it, you see the below

##Requirements

    Python 3.4+
    NumPy
    chainer
    PyYAML
    OpenSMILE http://www.audeering.com/research/opensmile
#
### Install

You have to install OpenSMILE

http://www.audeering.com/research-and-open-source/files/openSMILE-book-latest.pdf

```
apt-get install direnv
apt-get install virtualenv
apt-get install libhdf5-dev
```

You edit the enviroment.sh

1: You choose python version

2: You set the "Your Chainer Path"

3: You set the "Your OpenSMILE Path"

4: You set the "Your OpenSMILE Config Path"

5: You set the "Your VirtualEnv Path"

```
virtualenv -p python3.xx
echo numpy > requeriment.txt
pip install -r requeriment.txt
echo export OPENSMILE_PYTHON=`pwd` > .envrc
#You set the openSMILE-x.x.x/bin/linux_x64_standalone_static/SMILExtract PATH
echo export OPENSMILE_PATH=OPENSMILE_PATH >>.envrc
#You set the openSMILE-x.x.x/config/IS09_emotion.conf PATH
#or
#You set the openSMILE-x.x.x/config/IS10_paraling.conf PATH
echo export OPENSMILE_CONFIG=OPENSMILE_CONFIG_PATH >> .envrc
#You set the Data Path for Parsing Info file
echo export OPENSMILE_DATA=OPENSMILE_DATA >> .envrc
#You set the arff output Path
echo export OPENSMILE_OUTPUT=OPENSMILE_OUTPUT >> .envrc
#You set the ARFF Path
echo export OPENSMILE_ARFF=OPENSMILE_ARFF >> .envrc
direnv allow
source $OPENSMILE_PYTHON/my_env/bin/activate
```

```
sh enviroment.sh
```
#
### Usage
#
Prepare arff file
If you don't know how to make a arff file please read below repository
https://github.com/tech-sketch/Make-arff-for-Emotion-reconize-by-voice
```
*You execute python example
**prepare arff file
python make_arff/test/test_arff.py --config OPENSMILE_CONFIG
**chainer learn
python emotion_voice_recognize/test/test_emotion_voice.py --arff_label arff_emo_list_change_label_sort --feature IS2009
```
#
### Licence
#
```
The MIT License (MIT)

Copyright (c) 2015 Masaya Ogushi

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
```
#
### Author
#
[SnowMasaya](https://github.com/SnowMasaya)
### References
#
>[Chainer]http://chainer.org/

[OpenSMILE]http://www.audeering.com/research/opensmile

[Baby Picture]https://www.pakutaso.com/