text:
# Storage Providers

Connecting storage and platforms...

---

## Overview
This page reviews the storage providers and platforms supported by `libStorage`.

### Client/Server Configuration
Regarding the examples below, please
[read the provision](./config.md#clientserver-configuration) about
client/server configurations before proceeding.

## Amazon
libStorage includes support for multiple Amazon Web Services (AWS) storage
services.

### Elastic Block Storage
The AWS EBS driver registers a storage driver named `ebs` with the
libStorage service registry and is used to connect and manage AWS Elastic Block
Storage volumes for EC2 instances.

!!! note
    For backwards compatibility, the driver also registers a storage driver
    named `ec2`. The use of `ec2` in config files is deprecated but functional.
    The `ec2` driver **will be removed in 0.7.0**, at which point all instances
    of `ec2` in config files must use `ebs` instead.

!!! note
    The EBS driver does not yet support snapshots or tags, as previously
    supported in Rex-Ray v0.3.3.

The EBS driver is made possible by the
[official Amazon Go AWS SDK](https://github.com/aws/aws-sdk-go.git).

#### Requirements

* AWS account
* VPC - EBS can be accessed within VPC
* AWS Credentials

#### Configuration
The following is an example with all possible fields configured.  For a running
example see the [Examples](./storage-providers.md#aws-ebs-examples) section.

```yaml
ebs:
  accessKey:      XXXXXXXXXX
  secretKey:      XXXXXXXXXX
  region:         us-east-1
  maxRetries:     10
  kmsKeyID:       arn:aws:kms:us-east-1:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef
  statusMaxAttempts:  10
  statusInitialDelay: 100ms
  statusTimeout:      2m
```

##### Configuration Notes
- The `accessKey` and `secretKey` configuration parameters are optional and
should be used when explicit AWS credentials configuration needs to be provided.
EBS driver uses official golang AWS SDK library and supports all other ways of
providing access credentials, like environment variables or instance profile IAM
 permissions.
- `region` represents AWS region where EBS volumes should be provisioned.
See official AWS documentation for list of supported regions.

- `maxRetries` is the number of retries that will be made for failed operations
  by the AWS SDK.
- If the `kmsKeyID` field is specified it will be used as the encryption key for
all volumes that are created with a truthy encryption request field.
- `statusMaxAttempts` is the number of times the status of a volume will be
  queried before giving up when waiting on a status change
- `statusInitialDelay` specifies a time duration used to wait when polling
  volume status. This duration is used in exponential backoff, such that the
  first wait will be for this duration, the second for 2x, the third for 4x,
  etc. The units of the duration must be given (e.g. "100ms" or "1s").
- `statusTimeout` is a maximum length of time that polling for volume status can
  occur. This serves as a backstop against a stuck request of malfunctioning API
  that never returns.

For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

#### Activating the Driver
To activate the AWS EBS driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `ebs` as the driver name.

#### Troubleshooting
- Make sure that AWS credentials (user or role) has following AWS permissions on
  `libStorage` server instance that will be making calls to AWS API:
    - `ec2:AttachVolume`,
    - `ec2:CreateVolume`,
    - `ec2:CreateSnapshot`,
    - `ec2:CreateTags`,
    - `ec2:DeleteVolume`,
    - `ec2:DeleteSnapshot`,
    - `ec2:DescribeAvailabilityZones`,
    - `ec2:DescribeInstances`,
    - `ec2:DescribeVolumes`,
    - `ec2:DescribeVolumeAttribute`,
    - `ec2:DescribeVolumeStatus`,
    - `ec2:DescribeSnapshots`,
    - `ec2:CopySnapshot`,
    - `ec2:DescribeSnapshotAttribute`,
    - `ec2:DetachVolume`,
    - `ec2:ModifySnapshotAttribute`,
    - `ec2:ModifyVolumeAttribute`,
    - `ec2:DescribeTags`

#### Examples
Below is a working `config.yml` file that works with AWS EBS.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: ebs
  server:
    services:
      ebs:
        driver: ebs
        ebs:
          accessKey:      XXXXXXXXXX
          secretKey:      XXXXXXXXXX
          region:         us-east-1
```

### Elastic File System
The AWS EFS driver registers a storage driver named `efs` with the
libStorage service registry and is used to connect and manage AWS Elastic File
Systems.

#### Requirements

* AWS account
* VPC - EFS can be accessed within VPC
* AWS Credentials

#### Configuration
The following is an example with all possible fields configured.  For a running
example see the [Examples](./storage-providers.md#aws-efs-examples) section.

```yaml
efs:
  accessKey:      XXXXXXXXXX
  secretKey:      XXXXXXXXXX
  securityGroups:
  - sg-XXXXXXX
  - sg-XXXXXX0
  - sg-XXXXXX1
  region:              us-east-1
  tag:                 test
  disableSessionCache: false
  statusMaxAttempts:  6
  statusInitialDelay: 1s
  statusTimeout:      2m
```

##### Configuration Notes
- The `accessKey` and `secretKey` configuration parameters are optional and
should be used when explicit AWS credentials configuration needs to be provided.
EFS driver uses official golang AWS SDK library and supports all other ways of
providing access credentials, like environment variables or instance profile IAM
 permissions.
- `region` represents AWS region where EFS should be provisioned. See official
AWS documentation for list of supported regions.
- `securityGroups` list of security groups attached to `MountPoint` instances.
If no security groups are provided the default VPC security group is used.
- `tag` is used to partition multiple services within single AWS account and is
used as prefix for EFS names in format `[tagprefix]/volumeName`.
- `disableSessionCache` is a flag that can be used to disable the session cache.
If the session cache is disabled then a new AWS connection is established with
every API call.
- `statusMaxAttempts` is the number of times the status of a volume will be
  queried before giving up when waiting on a status change
- `statusInitialDelay` specifies a time duration used to wait when polling
  volume status. This duration is used in exponential backoff, such that the
  first wait will be for this duration, the second for 2x, the third for 4x,
  etc. The units of the duration must be given (e.g. "100ms" or "1s").
- `statusTimeout` is a maximum length of time that polling for volume status can
  occur. This serves as a backstop against a stuck request of malfunctioning API
  that never returns.

For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

#### Runtime Behavior

AWS EFS storage driver creates one EFS FileSystem per volume and provides root
of the filesystem as NFS mount point. Volumes aren't attached to instances
directly but rather exposed to each subnet by creating `MountPoint` in each VPC
subnet. When detaching volume from instance no action is taken as there isn't
good way to figure out if there are other instances in same subnet using
`MountPoint` that is being detached. There is no charge for `MountPoint`
so they are removed only once whole volume is deleted.

By default all EFS instances are provisioned as `generalPurpose` performance
mode. `maxIO` EFS type can be provisioned by providing `maxIO` flag as
`volumetype`.

Its possible to mount same volume to multiple container on a single EC2 instance
as well as use single volume across multiple EC2 instances at the same time.

!!! note
    Each EFS FileSystem can be accessed only from single VPC at the time.

#### Activating the Driver
To activate the AWS EFS driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `efs` as the driver name.

#### Troubleshooting
- Make sure that AWS credentials (user or role) has following AWS permissions on
  `libStorage` server instance that will be making calls to AWS API:
    - `elasticfilesystem:CreateFileSystem`
    - `elasticfilesystem:CreateMountTarget`
    - `ec2:DescribeSubnets`
    - `ec2:DescribeNetworkInterfaces`
    - `ec2:CreateNetworkInterface`
    - `elasticfilesystem:CreateTags`
    - `elasticfilesystem:DeleteFileSystem`
    - `elasticfilesystem:DeleteMountTarget`
    - `ec2:DeleteNetworkInterface`
    - `elasticfilesystem:DescribeFileSystems`
    - `elasticfilesystem:DescribeMountTargets`

#### Examples
Below is a working `config.yml` file that works with AWS EFS.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: efs
  server:
    services:
      efs:
        driver: efs
        efs:
          accessKey:      XXXXXXXXXX
          secretKey:      XXXXXXXXXX
          securityGroups:
          - sg-XXXXXXX
          - sg-XXXXXX0
          - sg-XXXXXX1
          region:         us-east-1
          tag:            test
```

### Simple Storage Service
The AWS S3FS driver registers a storage driver named `s3fs` with the
libStorage service registry and provides the ability to mount Amazon Simple
Storage Service (S3) buckets as filesystems using the
[`s3fs`](https://github.com/s3fs-fuse/s3fs-fuse) FUSE command.

Unlike the other AWS-related drivers, the S3FS storage driver does not need
to deployed or used by an EC2 instance. Any client can take advantage of
Amazon's S3 buckets.

#### Requirements
* AWS account
* The [`s3fs`](https://github.com/s3fs-fuse/s3fs-fuse) FUSE command must be
present on client nodes.

#### Configuration
The following is an example with all possible fields configured.  For a running
example see the [Examples](./storage-providers.md#aws-s3fs-examples) section.

#### Server-Side Configuration
```yaml
s3fs:
  region:           us-east-1
  accessKey:        XXXXXXXXXX
  secretKey:        XXXXXXXXXX
  disablePathStyle: false
```

* The `accessKey` and `secretKey` configuration parameters are optional and
should be used when explicit AWS credentials configuration needs to be provided.
The S3FS driver uses the official Golang AWS SDK library and supports all other
ways of  providing access credentials, like environment variables or instance
profile IAM permissions.
* `region` represents AWS region where S3FS buckets should be provisioned.
Please see the official AWS documentation for list of
[supported regions](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region).
* The `disablePathStyle` property disables the use of the path style for
bucket endpoints. The path style is more stable with regards to regions
than bucket URI FQDNs, but the path style is also less performant.

#### Client-Side Configuration
```yaml
s3fs:
  cmd:            s3fs
  options:
  - XXXX
  - XXXX
  accessKey:      XXXXXXXXXX
  secretKey:      XXXXXXXXXX
```

* The `cmd` property defaults simply to `s3fs` with the assumption that the
`s3fs` binary will be in the path. This value can also be the absolute path
to the `s3fs` binary.
* `options` is a list of options to pass to the `s3fs` command. Please see the
[official](https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon)
documentation for a full list of CLI options. The `-o` prefix should not be
provided in the configuration file.
* The credential properties can be defined on the client via the configuration
file and will be supplied to the `s3fs` process via environment variables.
However, the `s3fs` command will also look in all the
[usual places](https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon)
for the credentials if they're not in this file.

For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

#### Runtime Behavior
The AWS S3FS storage driver can create new buckets as well as remove existing
ones. Buckets are mounted to clients as filesystems using the
[`s3fs`](https://github.com/s3fs-fuse/s3fs-fuse) FUSE command. For clients
to correctly mount and unmount S3 buckets the `s3fs` command should be in
the path of the executor or configured via the `s3fs.cmd` property in the
client-side REX-Ray configuration file.

The client must also have access to the AWS credentials used for mounting and
unmounting S3 buckets. These credentials can be stored in the client-side
REX-Ray configuration file or via
[any means avaialble](https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon)
to the `s3fs` command.

#### Activating the Driver
To activate the AWS S3FS driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `s3fs` as the driver name.

#### Examples
Below is a working `config.yml` file that works with AWS S3FS.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: s3fs
  server:
    services:
      s3fs:
        driver: s3fs
        s3fs:
          accessKey:      XXXXXXXXXX
          secretKey:      XXXXXXXXXX
```

## Ceph
libStorage includes support for the following Ceph storage technologies.

### RADOS Block Device
The Ceph RBD driver registers a driver named `rbd` with the `libStorage` driver
manager and is used to connect and mount RADOS Block Devices from a Ceph
cluster.

#### Requirements

* The `ceph` and `rbd` binary executables must be installed on the host
* The `rbd` kernel module must be installed
* A `ceph.conf` file must be present in its default location
  (`/etc/ceph/ceph.conf`)
* The ceph `admin` key must be present in `/etc/ceph/`

#### Configuration
The following is an example with all possible fields configured. For a running
example see the [Examples](./storage-providers.md#ceph-rbd-examples) section.

```yaml
rbd:
  defaultPool: rbd
  testModule: true
```

##### Configuration Notes

* The `defaultPool` parameter is optional, and defaults to "rbd". When set, all
  volume requests that do not reference a specific pool will use the
  `defaultPool` value as the destination storage pool.
* The `testModule` parameter is optional, and defaults to "true". This setting
  indicates whether the libStorage client should test if the `rbd` kernel module
  is loaded, with the side-effect of loading it if is not already loaded. This
  setting should be disabled when the driver is executing inside of a container.

#### Runtime behavior

The Ceph RBD driver only works when the client and server are on the same node.
There is no way for a centralized `libStorage` server to attach volumes to
clients, therefore the `libStorage` server must be running on each node that
wishes to mount RBD volumes.

The RBD driver uses the format of `<pool>.<name>` for the volume ID. This allows
for the use of multiple pools by the driver. During a volume create, if the
volume ID is given as `<pool>.<name>`, a volume named *name* will be created in
the *pool* storage pool. If no pool is referenced, the `defaultPool` will be
used.

Both *pool* and *name* may only contain alphanumeric characters, underscores,
and dashes.

When querying volumes, the driver will return all RBDs present in all pools in
the cluster, prefixing each volume with the appropriate `<pool>.` value.

All RBD creates are done using the default 4MB object size, and using the
"layering" feature bit to ensure greatest compatibility with the kernel clients.

#### Activating the Driver
To activate the Ceph RBD driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers), using `rbd` as the
driver name.

#### Troubleshooting

* Make sure that `ceph` and `rbd` commands work without extra parameters for
  ID, key, and monitors. All configuration must come from `ceph.conf`.
* Check status of the ceph cluster with `ceph -s` command.

#### Examples

Below is a full `config.yml` that works with RBD

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: rbd
  server:
    services:
      rbd:
        driver: rbd
        rbd:
          defaultPool: rbd
```

#### Caveats
* Snapshot and copy functionality is not yet implemented
* libStorage Server must be running on each host to mount/attach RBD volumes
* There is not yet options for using non-admin cephx keys or changing RBD create
  features
* Volume pre-emption is not supported. Ceph does not provide a method to
  forcefully detach a volume from a remote host -- only a host can attach and
  detach volumes from itself.
* RBD advisory locks are not yet in use. A volume is returned as "unavailable"
  if it has a watcher other than the requesting client. Until advisory locks are
  in place, it may be possible for a client to attach a volume that is already
  attached to another node. Mounting and writing to such a volume could lead to
  data corruption.

## OpenStack

Another welcome community contribution, libStorage includes support for
OpenStack storage.

### Cinder
The Cinder driver registers a storage driver named `cinder` with the
`libStorage` driver manager and is used to connect and manage storage on
Cinder-compatible instances.

#### Configuration
The following is an example configuration with most fields populated for
illustration. For a running example see the
[Examples](./storage-providers.md#openstack-cinder-examples) section.

```yaml
cinder:
  authURL:              https://domain.com/openstack
  userID:               0
  userName:             myusername
  password:             mypassword
  tenantID:             0
  tenantName:           customer
  domainID:             0
  domainName:           corp
  regionName:           USNW
  availabilityZoneName: Gold
```

##### Configuration Notes
- `regionName` is optional, it should be empty if you only have one region.
- `availabilityZoneName` is optional, the volume will be created in the default
availability zone if not specified.

For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

#### Activating the Driver
To activate the Cinder driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `cinder` as the driver name.

#### Examples
Below is a full `config.yml` file that works with Cinder.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: cinder
  server:
    services:
      cinder:
        driver: cinder
cinder:
  authUrl: https://keystoneHost:35357/v2.0/
  username: username
  password: password
  tenantName: tenantName
```

## Dell EMC
libStorage includes support for several Dell EMC storage platforms.

### Isilon
The Isilon driver registers a storage driver named `isilon` with the
libStorage service registry and is used to connect and manage Isilon NAS
storage. The driver creates logical volumes in directories on the Isilon
cluster. Volumes are exported via NFS and restricted to a single client at a
time. Quotas can also be used to ensure that a volume directory doesn't exceed
a specified size.

#### Configuration
The following is an example configuration of the Isilon driver. For a running
example see the [Examples](./storage-providers.md#dell-emc-isilon-examples)
section.

```yaml
isilon:
  endpoint: https://endpoint:8080
  insecure: true
  username: username
  group: groupname
  password: password
  volumePath: /libstorage
  nfsHost: nfsHost
  dataSubnet: subnet
  quotas: true
```

For information on the equivalent environment variable and CLI flag names
please see the section on how configuration properties are
[transformed](./config.md#configuration-properties).

#### Extra Parameters
The following items are configurable specific to this driver.

 * `volumePath` represents the location under `/ifs/volumes` to allow volumes to
   be created and removed.
 * `nfsHost` is the configurable NFS server hostname or IP (often a
   SmartConnect name) used when mounting exports
 * `dataSubnet` is the subnet the REX-Ray driver is running on. This is used
   for the NFS export host ACLs.

#### Optional Parameters
The following items are not required, but available to this driver.

 * `insecure` defaults to `false`.
 * `group` defaults to the group of the user specified in the configuration.
   Only use this option if you need volumes to be created with a different
   group.
 * `volumePath` defaults to "". This will have all new volumes created directly
   under `/ifs/volumes`.
 * `quotas` defaults to `false`. Set to `true` if you have a SmartQuotas
   license enabled.

#### Activating the Driver
To activate the Isilon driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `isilon` as the driver name.

#### Examples
Below is a full `config.yml` file that works with Isilon.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: isilon
  server:
    services:
      isilon:
        driver: isilon
        isilon:
          endpoint: https://endpoint:8080
          insecure: true
          username: username
          password: password
          volumePath: /libstorage
          nfsHost: nfsHost
          dataSubnet: subnet
          quotas: true
```

#### Instructions
It is expected that the `volumePath` exists already within the Isilon system.
This example would reflect a directory create under `/ifs/volumes/libstorage`
for created volumes. It is not necessary to export this volume. The `dataSubnet`
parameter is required so the Isilon driver can restrict access to attached
volumes to the host that REX-Ray is running on.

If `quotas` are enabled, a SmartQuotas license must also be enabled on the
Isilon cluster for the capacity size functionality of `libStorage` to work.

A SnapshotIQ license must be enabled on the Isilon cluster for the snapshot
functionality of `libStorage` to work.

#### Caveats
The Isilon driver is not without its caveats:

 * The account used to access the Isilon cluster must be in a role with the
  following privileges:
    * Namespace Access (ISI_PRIV_NS_IFS_ACCESS)
    * Platform API (ISI_PRIV_LOGIN_PAPI)
    * NFS (ISI_PRIV_NFS)
    * Restore (ISI_PRIV_IFS_RESTORE)
    * Quota (ISI_PRIV_QUOTA)          (if `quotas` are enabled)
    * Snapshot (ISI_PRIV_SNAPSHOT)    (if snapshots are used)

you can set the RBAC rights on the Isilon console:

```bash
# create RBAC group
isi auth roles create --name libstorage_roles
# asign privileges to role
isi auth roles modify libstorage_roles --add-priv  ISI_PRIV_NS_IFS_ACCESS
isi auth roles modify libstorage_roles --add-priv  ISI_PRIV_LOGIN_PAPI
isi auth roles modify libstorage_roles --add-priv  ISI_PRIV_NFS
isi auth roles modify libstorage_roles --add-priv  ISI_PRIV_IFS_RESTORE
isi auth roles modify libstorage_roles --add-priv  ISI_PRIV_QUOTA
isi auth roles modify libstorage_roles  --add-priv  ISI_PRIV_SNAPSHOT
# add user to RBAC group
isi auth roles modify libstorage_roles --add-user libstorage
```

### ScaleIO
The ScaleIO driver registers a storage driver named `scaleio` with the
libStorage service registry and is used to connect and manage ScaleIO storage.

#### Requirements
 - The ScaleIO `REST Gateway` is required for the driver to function.
 - The `libStorage` client or application that embeds the `libStorage` client
   must reside on a host that has the SDC client installed. The command
   `/opt/emc/scaleio/sdc/bin/drv_cfg --query_guid` should be executable and
   should return the local SDC GUID.
 - The [official](http://www.oracle.com/technetwork/java/javase/downloads/index.html)
   Oracle Java Runtime Environment (JRE) is required. During testing, use of the
   Open Java Development Kit (JDK) resulted in unexpected errors.

#### Configuration
The following is an example with all possible fields configured.  For a running
example see the [Examples](./storage-providers.md#dell-emc-scaleio-examples)
section.

```yaml
scaleio:
  endpoint:             https://host_ip/api
  apiVersion:           "2.0"
  insecure:             false
  useCerts:             true
  userName:             admin
  password:             mypassword
  systemID:             0
  systemName:           sysv
  protectionDomainID:   0
  protectionDomainName: corp
  storagePoolID:        0
  storagePoolName:      gold
  thinOrThick:          ThinProvisioned
```

##### Configuration Notes
- The `apiVersion` can optionally be set here to force certain API behavior.
The default is to retrieve the endpoint API, and pass this version during calls.
- `insecure` should be set to `true` if you have not loaded the SSL
certificates on the host.  A successful wget or curl should be possible without
SSL errors to the API `endpoint` in this case.
- `useCerts` should only be set if you want to leverage the internal SSL
certificates.  This would be useful if you are deploying the REX-Ray binary
on a host that does not have any certificates installed.
- `systemID` takes priority over `systemName`.
- `protectionDomainID` takes priority over `protectionDomainName`.
- `storagePoolID` takes priority over `storagePoolName`.
- `thinkOrThick` determines whether to provision as the default
`ThinProvisioned`, or `ThickProvisioned`.

For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

#### Runtime Behavior
The `storageType` field that is configured per volume is considered the
ScaleIO Storage Pool.  This can be configured by default with the `storagePool`
setting.  It is important that you create unique names for your Storage Pools
on the same ScaleIO platform.  Otherwise, when specifying `storageType` it
may choose at random which `protectionDomain` the pool comes from.

The `availabilityZone` field represents the ScaleIO Protection Domain.

#### Configuring the Gateway
- Install the `EMC-ScaleIO-gateway` package.
- Edit the
`/opt/emc/scaleio/gateway/webapps/ROOT/WEB-INF/classes/gatewayUser.properties`
file and append the proper MDM IP addresses to the following `mdm.ip.addresses=`
parameter.
- By default the password is the same as your administrative MDM password.
- Start the gateway `service scaleio-gateway start`.
 - With 1.32 we have noticed a restart of the gateway may be necessary as well
after an initial install with `service scaleio-gateway restart`.

#### Activating the Driver
To activate the ScaleIO driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `scaleio` as the driver name.

#### Troubleshooting
- Verify your parameters for `system`, `protectionDomain`, and
`storagePool` are correct.
- Verify that have the ScaleIO SDC service installed with
`rpm -qa EMC-ScaleIO-sdc`
- Verify that the following command returns the local SDC GUID
`/opt/emc/scaleio/sdc/bin/drv_cfg --query_guid`.
- Ensure that you are able to open a TCP connection to the gateway with the
address that you will be supplying below in the `gateway_ip` parameter.  For
example `telnet gateway_ip 443` should open a successful connection.  Removing
the `EMC-ScaleIO-gateway` package and reinstalling can force re-creation of
self-signed certs which may help resolve gateway problems.  Also try restarting
the gateway with `service scaleio-gateway restart`.
- Ensure that you have the correct authentication credentials for the gateway.
This can be done with a curl login. You should receive an authentication
token in return.
`curl --insecure --user admin:XScaleio123 https://gw_ip:443/api/login`
- Please review the gateway log at
`/opt/emc/scaleio/gateway/logs/catalina.out` for errors.

#### Examples
Below is a full `config.yml` file that works with ScaleIO.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: scaleio
  server:
    services:
      scaleio:
        driver: scaleio
        scaleio:
          endpoint: https://gateway_ip/api
          insecure: true
          userName: username
          password: password
          systemName: tenantName
          protectionDomainName: protectionDomainName
          storagePoolName: storagePoolName
```

## DigitalOcean
Thanks to the efforts of our tremendous community, libStorage also has built-in
support for DigitalOcean!

### DO Block Storage
The DigitalOcean Block Storage (DOBS) driver registers a driver named `dobs`
with the libStorage service registry and is used to attach and mount
DigitalOcean block storage devices to DigitalOcean instances.

#### Requirements
The DigitalOcean block storage driver has the following requirements:

* Valid DigitalOcean account
* Valid DigitalOcean [access token](https://goo.gl/iKoAec)

#### Configuration
The following is an example with all possible fields configured. For a running
example see the [Examples](./storage-providers.md#dobs-examples) section.

```yaml
dobs:
  token:  123456
  region: nyc1
  statusMaxAttempts: 10
  statusInitialDelay: 100ms
  statusTimeout: 2m
  convertUnderscores: false
```

##### Configuration notes
- The `token` contains your DigitalOcean [access token](https://goo.gl/iKoAec)
- `region` specifies the DigitalOcean region where volumes should be created
- `statusMaxAttempts` is the number of times the status of a volume will be
  queried before giving up when waiting on a status change
- `statusInitialDelay` specifies a time duration used to wait when polling
  volume status. This duration is used in exponential backoff, such that the
  first wait will be for this duration, the second for 2x, the third for 4x,
  etc. The units of the duration must be given (e.g. "100ms" or "1s").
- `statusTimeout` is a maximum length of time that polling for volume status can
  occur. This serves as a backstop against a stuck request of malfunctioning API
  that never returns.
- `convertUnderscores` is a boolean flag that controls whether the driver will
  automatically convert underscores to dashes during a volume create request.
  Digital Ocean does not allow underscores in the volume name, but some
  container orchestrators (e.g. Docker Swarm) automatically prefix volume names
  with a string containing a dash. This flag enables such requests to proceed,
  but with the volume name modified.

!!! note
    The DigitalOcean service currently only supports block storage volumes in
    specific regions. Make sure to use a [suuported region](https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean#what-is-digitalocean-block-storage).

    The standard environment variable for the DigitalOcean access token is
    `DIGITALOCEAN_ACCESS_TOKEN`. However, the environment variable mapped to
    this driver's `dobs.token` property is `DOBS_TOKEN`. This choice was made
    to ensure that the driver must be explicitly configured for access instead
    of detecting a default token that may not be intended for the driver.

#### Examples
Below is a full `config.yml` that works with DOBS

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: dobs
  server:
    services:
      dobs:
        driver: dobs
        dobs:
          token: 123456
          region: nyc1
```

## FittedCloud
Another example of the great community shared by the libStorage project, the
talented people at FittedCloud have provided a driver for their EBS optimizer.

### EBS Optimizer
The FittedCloud EBS Optimizer driver registers a storage driver named
`fittedcloud` with the libStorage service registry and provides the ability to
connect and manage thin-provisioned EBS volumes for EC2 instances.

!!! note
    This version of the FittedCloud driver only supports configurations where
    client and server are on the same host.  The libStorage server must be
    running on each node along side with the FittedCloud Agent.

!!! note
    This version of the FittedCloud driver does not support co-existing with the
    ebs driver on the same host. As a result it also doesn't support optimizing
    existing EBS volumes. See the [Examples](#fittedcloud-examples) section
    below for a running example.

!!! note
    The FittedCloud driver does not yet support snapshots or tags.

#### Requirements
This driver has the following requirements:

* AWS account
* VPC - EBS can be accessed within VPC
* AWS Credentials
* FittedCloud Agent software

#### Getting Started
Before starting, please make sure to register as a user by visiting the
FittedCloud [customer website](https://customer.fittedcloud.com/register).
Once an account is activated it will be assigned a user ID, which can be found
on the Settings page after logging into the web site.

The following commands will download and install the latest FittedCloud Agent
software. The flags `-o S -m` enable new thin volumes to be created via the
docker command instead of optimizing existing EBS volumes.
Please replace the `<User ID>` with a FittedCloud user ID.

```sh
$ curl -skSL 'https://customer.fittedcloud.com/downloadsoftware?ver=latest' \
  -o fcagent.run
$ sudo bash ./fcagent.run -- -o S -m -d <User ID>
```

Please refer to FittedCloud
[website](https://customer.fittedcloud.com/download) for more details.

#### Configuration
The following is an example with all possible fields configured.  For a running
example see the [Examples](./storage-providers.md#fittedcloud-examples) section.

```yaml
ebs:
  accessKey:      XXXXXXXXXX
  secretKey:      XXXXXXXXXX
  kmsKeyID:       abcd1234-a123-456a-a12b-a123b4cd56ef
  statusMaxAttempts:  10
  statusInitialDelay: 100ms
  statusTimeout:      2m
```

##### Configuration Notes
- FittedCloud driver shares the ebs driver's configuration
parameters.
- The `accessKey` and `secretKey` configuration parameters are optional and
should be used when explicit AWS credentials configuration needs to be provided.
FittedCloud driver uses official golang AWS SDK library and supports all other
ways of providing access credentials, like environment variables or instance
profile IAM permissions.
- If the `kmsKeyID` field is specified it will be used as the encryption key for
all volumes that are created with a truthy encryption request field.
- `statusMaxAttempts` is the number of times the status of a volume will be
  queried before giving up when waiting on a status change
- `statusInitialDelay` specifies a time duration used to wait when polling
  volume status. This duration is used in exponential backoff, such that the
  first wait will be for this duration, the second for 2x, the third for 4x,
  etc. The units of the duration must be given (e.g. "100ms" or "1s").
- `statusTimeout` is a maximum length of time that polling for volume status can
  occur. This serves as a backstop against a stuck request of malfunctioning API
  that never returns.

#### Examples
The following example illustrates how to configured the FittedCloud driver:

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: fittedcloud
  server:
    services:
      fittedcloud:
        driver: fittedcloud
ebs:
  accessKey:  XXXXXXXXXX
  secretKey:  XXXXXXXXXX
```

Additional information on configuring the FittedCloud driver may be found
at [this](https://goo.gl/I6mf20) location.

## Google
libStorage ships with support for Google Compute Engine (GCE) as well.

### GCE Persistent Disk
The Google Compute Engine Persistent Disk (GCEPD) driver registers a driver
named `gcepd` with the libStorage service registry and is used to connect and
mount Google Compute Engine (GCE) persistent disks with GCE machine instances.

#### Requirements
* GCE account
* The libStorage server must be running on a GCE instance created with a Service
  Account with appropriate permissions, or a Service Account credentials file
  in JSON format must be supplied. If not using the Compute Engine default
  Service Account with the Cloud Platform/"all cloud APIs" scope, create a new
  Service Account via the [IAM Portal](https://console.cloud.google.com/iam-admin/serviceaccounts).
  This Service Account requires the `Compute Engine/Instance Admin`,
  `Compute Engine/Storage Admin`, and `Project/Service Account Actor` roles.
  Then create/download a new private key in JSON format. see
  [creating a service account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount)
  for details. The libStorage service must be restarted in order for permissions
  changes on a service account to take effect.

#### Configuration
The following is an example with all possible fields configured. For a running
example see the [Examples](./storage-providers.md#gce-persistent-disk-examples)
section.

```yaml
gcepd:
  keyfile: /etc/gcekey.json
  zone: us-west1-b
  defaultDiskType: pd-ssd
  tag: rexray
  statusMaxAttempts:  10
  statusInitialDelay: 100ms
  statusTimeout:      2m
  convertUnderscores: false
```

##### Configuration Notes
* The `keyfile` parameter is optional. It specifies a path on disk to a file
  containing the JSON-encoded Service Account credentials. This file can be
  downloaded from the GCE web portal. If `keyfile` is specified, the GCE
  instance's service account is not considered, and is not necessary. If
  `keyfile` is *not* specified, the application will try to lookup
  [application default credentials](https://developers.google.com/identity/protocols/application-default-credentials).
  This has the effect of looking for credentials in the priority described
  [here](https://godoc.org/golang.org/x/oauth2/google#FindDefaultCredentials).
* The `zone` parameter is optional, and configures the driver to *only* allow
  access to the given zone. Creating and listing disks from other zones will be
  denied. If a zone is not specified, the zone from the client Instance ID will
  be used when creating new disks.
* The `defaultDiskType` parameter is optional and specifies what type of disk
  to create, either `pd-standard` or `pd-ssd`. When not specified, the default
  is `pd-ssd`.
* The `tag` parameter is optional, and causes the driver to create or return
  disks that have a matching tag. The tag is implemented by using the GCE
  label functionality available in the beta API. The value of the `tag`
  parameter is used as the value for a label with the key `libstoragetag`.
  Use of this parameter is encouraged, as the driver will only return volumes
  that have been created by the driver, which is most useful to eliminate
  listing the boot disks of every GCE disk in your project/zone. If you wsih to
  "expose" previously created disks to the `GCEPD` driver, you can edit the
  labels on the existing disk to have a key of `libstoragetag` and a value
  matching that given in `tag`.
* `statusMaxAttempts` is the number of times the status of a volume will be
  queried before giving up when waiting on a status change
* `statusInitialDelay` specifies a time duration used to wait when polling
  volume status. This duration is used in exponential backoff, such that the
  first wait will be for this duration, the second for 2x, the third for 4x,
  etc. The units of the duration must be given (e.g. "100ms" or "1s").
* `statusTimeout` is a maximum length of time that polling for volume status can
  occur. This serves as a backstop against a stuck request of malfunctioning API
  that never returns.
* `convertUnderscores` is a boolean flag that controls whether the driver will
  automatically convert underscores to dashes during a volume create request.
  GCE does not allow underscores in the volume name, but some container
  orchestrators (e.g. Docker Swarm) automatically prefix volume names
  with a string containing a dash. This flag enables such requests to proceed,
  but with the volume name modified.

#### Runtime behavior
* The GCEPD driver enforces the GCE requirements for disk sizing and naming.
  Disks must be created with a minimum size of 10GB. Disk names must adhere to
  the regular expression of `[a-z]([-a-z0-9]*[a-z0-9])?`, which means the first
  character must be a lowercase letter, and all following characters must be a
  dash, lowercase letter, or digit, except the last character, which cannot be a
  dash.
* If the `zone` parameter is not specified in the driver configuration, and a
  request is received to list all volumes that does not specify a zone in the
  InstanceID header, volumes from all zones will be returned.
* By default, all disks will be created with type `pd-ssd`, which creates an SSD
  based disk. If you wish to create disks that are not SSD-based, change the
  default via the driver config, or the type can be changed at creation time by
  using the `Type` field of the create request.

#### Activating the Driver
To activate the GCEPD driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers), using `gcepd` as the
driver name.

#### Troubleshooting
* Make sure that the JSON credentials file as specified in the `keyfile`
  configuration parameter is present and accessible, or that you are running in
  a GCE instance created with a Service Account attached. Whether using a
  `keyfile` or the Service Account associated with the GCE instance, the Service
  Account must have the appropriate permissions as described in
  `Configuration Notes`

#### Examples
Below is a full `config.yml` that works with GCE

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: gcepd
  server:
    services:
      gcepd:
        driver: gcepd
        gcepd:
          keyfile: /etc/gcekey.json
          tag: rexray
```

#### Caveats
* Snapshot and copy functionality is not yet implemented
* Most GCE instances can have up to 64 TB of total persistent disk space
  attached. Shared-core machine types or custom machine types with less than
  3.75 GB of memory are limited to 3 TB of total persistent disk space. Total
  persistent disk space for an instance includes the size of the root persistent
  disk. You can attach up to 16 independent persistent disks to most instances,
  but instances with shared-core machine types or custom machine types with less
  than 3.75 GB of memory are limited to a maximum of 4 persistent disks,
  including the root persistent disk. See
  [GCE Disks](https://cloud.google.com/compute/docs/disks/) docs for more
  details.
* If running libStorage server in a mode where volume mounts will not be
  performed on the same host where libStorage server is running, it should be
  possible to use a Service Account without the `Service Account Actor` role,
  but this has not been tested. Note that if persistent disk mounts are to be
  performed on *any* GCE instances that have a Service Account associated with
  the, the `Service Account Actor` role is required.

## Microsoft
Microsoft Azure support is included with libStorage as well.

### Azure Unmanaged Disk
The Microsoft Azure Unmanaged Disk (Azure UD) driver registers a driver
named `azureud` with the libStorage service registry and is used to connect and
mount Azure unmanaged disks from Azure page blob storage with Azure virtual
machines.

#### Requirements
* An Azure account
* An Azure subscription
* An Azure storage account
* An Azure resource group
* Any virtual machine where disks are going to be attached must have the
  `lsscsi` utility installed. You can install this with `yum install lsscsi` on
  Red Hat based distributions, or with `apt-get install lsscsi` on Debian based
  distributions.

#### Configuration
The following is an example with all possible fields configured. For a running
example see the [Examples](./storage-providers.md#azure-ud-examples) section.

```yaml
azureud:
  subscriptionID: abcdef01-2345-6789-abcd-ef0123456789
  resourceGroup: testgroup
  tenantID: usernamehotmail.onmicrosoft.com
  storageAccount: username
  storageAccessKey: XXXXXXXX
  clientID: 123def01-2345-6789-abcd-ef0123456789
  clientSecret: XXXXXXXX
  certPath:
  container: vhds
  useHTTPS: true
```

##### Configuration Notes
* `subscriptionID` is required, and is the UUID of your Azure subscription
* `resourceGroup` is required, and is the name of the resource group for your
  VMs and storage.
* `tenantID` is required, and is either the domain or UUID for your active
  directory account within Azure.
* `storageAccount` is required, and is the name of the storage account where
  your disks will be created.
* `storageAccessKey` is required, and is a valid access key associated with the
  `storageAccount`.
* `clientID` is required, and is the UUID of your client, which was created as
  an App Registration within your Azure active directory account. When creating
  an App Registration, this ID is shown as the Application ID.
* `clientSecret` is required if `certPath` is not provided instead. It is a
  valid access key associated with `clientID`, and is managed as part of the App
  Registration.
* `certPath` is an alternative to `clientSecret`, contains the location of a
  PKCS encoded RSA private key associated with `clientID`.
* `container` is optional, and specifies the name of an existing container
  within `storageAccount`. This container must already exist and is not created
  automatically.
* `useHTTPS` is optional, and is a boolean value on whether to use HTTPS when
  communicating with the Azure storage endpoint.

#### Runtime Behavior
* The `container` config option defaults to `vhds`, and this container is
  present by default in Azure. Changing this option is only necessary if you
  want to use a different container within your storage account.
* Volume Attach/Detach operations in Azure take a long time, sometimes greater
  than 60 seconds, which is libStorage's default task timeout. When the timeout
  is hit, libStorage returns information to the caller about a queued task, and
  that the task is still running. This may cause issues for upstream callers.
  It is *highly* recommended to adjust this default timeout to 120 seconds by
  setting the `libstorage.server.tasks.exeTimeout` property. This is done in
  the `Examples` section below.

#### Activating the Driver
To activate the Azure UD driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers), using `azureud` as
the driver name.

#### Troubleshooting
* For help creating App Registrations, the steps in
  [this guide](https://www.terraform.io/docs/providers/azurerm/index.html#creating-credentials)
  cover creating a new Registration using the Azure portal and CLI.
* After creating your app registration, you must go into the
  `Required Permissions` tab and grant access to "Windows Azure Service
  Management API". Choose the delegated permission for accessing as organization
  users.
* You must also grant your app registration access to your subscription, by
  going to Subscriptions->Your `subscriptionID`->Access Control (IAM). From
  there, add your app registration as a user, which you will have to search for
  by name. Grant the role of "Owner".
* You should carefully check that your VM is compatible with the storage account
  you want to use. For example, if you need Azure Premium storage your machine
  should be of a compatible size (e.g. DS_V2, FS). For more details see the
  available VM [sizes](https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes).

#### Examples
Below is a full `config.yml` that works with Azure UD

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: azureud
  server:
    tasks:
      exeTimeout: 120s
    services:
      azure:
        driver: azureud
        azureud:
          subscriptionID: abcdef01-2345-6789-abcd-ef0123456789
          resourceGroup: testgroup
          tenantID: usernamehotmail.onmicrosoft.com
          storageAccount: username
          storageAccessKey: XXXXXXXX
          clientID: 123def01-2345-6789-abcd-ef0123456789
          clientSecret: XXXXXXXX
```

#### Caveats
* Snapshot and Copy functionality is not yet implemented
* The number of disks you can attach to a Virtual Machine depends on its type.
* Good resources for reading about disks in Azure are
  [here](https://docs.microsoft.com/en-us/azure/storage/storage-standard-storage)
  and [here](https://docs.microsoft.com/en-us/azure/storage/storage-about-disks-and-vhds-linux).

## VirtualBox
The VirtualBox driver registers a storage driver named `virtualbox` with the
libStorage service registry and is used by VirtualBox's VMs to connect and
manage volumes provided by VirtualBox.

### Prerequisites
In order to leverage the `virtualbox` driver, the `libStorage` client or must
be located on each VM that you wish to be able to consume external volumes.
The driver leverages the `vboxwebserv` HTTP SOAP API which is a process that
must be started from the VirtualBox *host* (ie OS X) using
`vboxwebsrv -H <IP_ADDRESS> -v` or additionally with `-b` for running in the
background. This allows the VMs running `libStorage` to remotely make calls to
the underlying VirtualBox application. A test for connectivity can be done with
`telnet virtualboxip 18083` from the VM. The `virtualboxip` is what you
would put in the `endpoint` value.

Leveraging authentication for the VirtualBox webserver is optiona.. The HTTP
SOAP API can have authentication disabled by running
`VBoxManage setproperty websrvauthlibrary null`.

Hot-Plugging is required, which limits the usefulness of this driver to `SATA`
only.  Ensure that your VM has *pre-created* this controller and it is
named `SATA`.  Otherwise the `controllerName` field must be populated
with the name of the controller you wish to use.  The port count must be set
manually as it cannot be increased when the VMs are on.  A count of `30`
is suggested.

VirtualBox 5.0.10+ must be used.

!!! note
    For a VirtualBox VM to work successfully, the following three points are of
    the utmost importance:

    1. The SATA controller should be named `SATA`.
    2. The SATA controller's port count must allow for additional
    connections.
    3. The MAC address must not match that of any other VirtualBox VM's
    MAC addresses or the MAC address of the host.

    The REX-Ray `Vagrantfile` has
    [a section](https://github.com/codedellemc/rexray/blob/master/Vagrantfile#L166-L179)
    that automatically configures these options.

### Configuration
The following is an example configuration of the VirtualBox driver.
The `localMachineNameOrId` parameter is for development use where you force
`libStorage` to use a specific VM identity.  Choose a `volumePath` to store the
volume files or virtual disks.  This path should be created ahead of time.

```yaml
virtualbox:
  endpoint: http://virtualboxhost:18083
  userName: optional
  password: optional
  tls: false
  volumePath: $HOME/VirtualBox/Volumes
  controllerName: name
  localMachineNameOrId: forDevelopmentUse
```
For information on the equivalent environment variable and CLI flag names
please see the section on how non top-level configuration properties are
[transformed](./config.md#configuration-properties).

### Activating the Driver
To activate the VirtualBox driver please follow the instructions for
[activating storage drivers](./config.md#storage-drivers),
using `virtualbox` as the driver name.

### Examples
Below is a working `config.yml` file that works with VirtualBox.

```yaml
libstorage:
  # The libstorage.service property directs a libStorage client to direct its
  # requests to the given service by default. It is not used by the server.
  service: virtualbox
  server:
    services:
      virtualbox:
        driver: virtualbox
        virtualbox:
          endpoint:       http://<IP_ADDRESS>:18083
          tls:            false
          volumePath:     $HOME/VirtualBox/Volumes
          controllerName: SATA
```

### Caveats
- Snapshot and create volume from volume functionality is not available yet
  with this driver.
- The driver supports VirtualBox 5.0.10+

repository: thecodeteam/libstorage

repo_name: thecodeteam/libstorage


text:
---
id: 382ee14c
title: Networking Issues
description: Systems are currently down.
date: 2020-03-12T21:33:18.362Z
modified: 2020-03-12T23:33:18.362Z
severity: degraded-performance
resolved: true
affectedsystems:
  - api
  - cdn
---

Systems are currently down.

::: update Resolved | 2020-03-12T23:33:18.362Z
The Conversion Process is back to normal and we'll continue to monitor.
:::

::: update Monitoring | 2020-03-12T22:33:18.362Z
The Files Conversion System is back to normal and we'll continue to monitor.
:::

::: update Resolved | 2020-03-12T22:03:18.362Z
We detected a networking problem that caused temporary issues for our API and origin servers. All systems are back to normal now, but we're monitoring closely.
:::

repository: ncs-bbrunner/NCS-Technology-Status-Page

extension: md

gha_language: TypeScript

license_type: permissive

repo_name: ncs-bbrunner/NCS-Technology-Status-Page


text:
# Week 7 - Class 2:
## Lab Assignment
* Using core location, display the users location on the map
* Add a long press gesture to the map view
* On long press,  add an annotation onto the map view, which has a callout with an accessory button
* Upon pressing the accessory button, segue to a AddReminderDetailViewController
* **Code Challenge:**
	* Write a method that detects if two strings are anagrams.

### Reading Assignment (read **before** Class 33):
* Programming in Objective-C
  * **Chapter 10:** More on Variables and Data Types
* Apple Documentation
  * Notification Center

## Other Resources
* [Grading Rubrics](../../resources/)
* [Lecture Slides](https://www.icloud.com/keynote/0002PSxT891quC_xyaHmdrCYg#Week7_Day2)

license: MIT

branch_name: refs/heads/master


text:
# Developer Guide

## Build
To build the addon (incl. auto code generation), simply do the following in console:
```
cd src/object-recognition
npm install
```

## Run

To test the addon
1. Build `librealsense`
1. Connect R200 camera into your system
1. Setup LD_LIBRARY_PATH pointing to the path of all the following
  1. `realsense_sdk` libraries
  1. `opencv` libraries
  1. `librealsense_object_recognition` (middleware) libraries
  1. `librealsense` libraries
1. Copy `classifiers` directory from `librealsense_object_recognition` and put them into ``<project_root>/src` (i.e. the same level with `object-recognition` directory)
1. Run `node examples/quick-test.js`

Please refer to librealsense_object_recognition docs for the required version of `opencv`/`librealsense`/`realsense_sdk`.

## Known Issues
- Hard-coded include & library directories in source (`binding.gyp`)
- LD_LIBRARY_PATH must be set to run the addon
- To enable run addon inside `src/object-recognition` directory, have to put files in `src/classifiers` directory -- expecting native middleware to have an function to configure this runtime behavior


text:
---
title: "Install K8ssandra on EKS"
linkTitle: "Amazon EKS"
weight: 2
description: >
  Complete **production** ready environment of K8ssandra on Amazon Elastic Kubernetes Service (EKS).
---

Amazon [Elastic Kubernetes Service](https://aws.amazon.com/eks/features/) or "EKS" is a managed Kubernetes service that makes it easy for you to run Kubernetes on AWS and on-premises. EKS is certified Kubernetes conformant, so existing applications that run on upstream Kubernetes are compatible with EKS. AWS automatically manages the availability and scalability of the Kubernetes control plane nodes responsible scheduling containers, managing the availability of applications, storing cluster data, and other key tasks.

{{% alert title="Tip" color="success" %}}
Also available in followup topics are post-install steps and role-based considerations for [developers]({{< relref "/quickstarts/developer">}}) or [site reliability engineers]({{< relref "/quickstarts/site-reliability-engineer">}}) (SREs).
{{% /alert %}}

## Minimum deployment

This topic covers provisioning the following infrastructure resources as a minimum for production. See the [next section]({{< relref "#infrastructure-and-cassandra-recommendations" >}}) for additional considerations discovered during performance benchmarks.

* 1x Virtual Private Cloud
* 10x Subnets
* 3x Security Groups (& Rules)
* 1x NAT Gateway
* 1x Internet Gateway
* 3x Elastic IP
* 6x Route Table
* 4x Route Table Association
* 1x EKS cluster with instances spread across multiple Availability Zones.
* 1x EKS Node Group
  * 6x Kubernetes workers
    * 8 vCPUs
    * 64 GB RAM
* 3x 2TB EBS Volumes (provisioned automatically during installation of K8ssandra)
* 1x Amazon S3 bucket for K8ssandra Medusa backups

On this infrastructure the K8ssandra installation will consist of the following workloads.

* 3x node Apache Cassandra cluster
* 3x node Stargate deployment
* 1x node Prometheus deployment
* 1x node Grafana deployment
* 1x node Reaper deployment

Feel free to update the parameters used during this guide to match your target deployment. This should be considered a minimum for production workloads.

## Infrastructure and Cassandra recommendations

While the section above includes infrastructure settings for **minimum** production workloads, performance benchmarks reveal a wider range of recommendations that are important to consider. The performance benchmark report, available in this [detailed blog post](https://k8ssandra.io/blog/articles/k8ssandra-performance-benchmarks-on-cloud-managed-kubernetes/), compared the throughput and latency between:

* The baseline performance of a Cassandra cluster running on AWS EC2 instances -- a common setup for enterprises operating Cassandra clusters
* The performance of K8ssandra running on Amazon EKS, Google GCP GKE, and Microsoft Azure AKS.

It's important to note the following additional AWS infrastructure settings and observations from the benchmark:

* 8 to 16 vCPUs
  * r5 instances: Intel Xeon Platinum 8000 series. Cassandra workloads are mostly CPU bound and the core speed made a difference in the throughput benchmarks.
* 32 GB to 128 GB RAM (we used 64 GB RAM during the benchmark)
* 2 to 4 TB of disk space
  * In the benchmark, we used 1x 3.4 TB EBS gp2 volume
* 10k IOPS (observed)

For the disk performance, the benchmark used [Cassandra inspired fio profiles](https://github.com/ibspoof/cassandra-fio) that attempt to emulate Leveled Compaction Strategy and Size Tiered Compaction Strategy behaviors.

Regarding the Cassandra version and settings:

* The benchmark used Cassandra 4.0-beta4.
* Cassandra default settings were applied with the exception of garbage collection (GC) settings. This used G1GC with 31GB of heap size, along with a few GC related JVM flags:

  ```
  -XX:+UseG1GC
  -XX:G1RSetUpdatingPauseTimePercent=5
  -XX:MaxGCPauseMillis=300
  -XX:InitiatingHeapOccupancyPercent=70 -Xms31G -Xmx31G
  ```

To summarize the findings, running Cassandra in Kubernetes using K8ssandra didn't introduce any notable performance impacts in throughput or latency, all while K8ssandra simplified the deployment steps. See the [blog post](https://k8ssandra.io/blog/articles/k8ssandra-performance-benchmarks-on-cloud-managed-kubernetes/) for more detailed settings, results, and the measures taken to ensure fair production comparisons.

## Terraform

As a convenience we provide reference [Terraform](https://www.terraform.io/) modules for orchestrating the provisioning of cloud resources necessary to run K8ssandra.

### Prerequisite tools

First, these steps assume you already have an AWS account. If not, see [Create and activate a new AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) in the AWS documentation.

Next, ensure you have the prerequisite tools installed (or subsequent versions), including the Terraform binary. Links below go to download resources:

| Tool | Version |
|------|---------|
| [Terraform](https://www.terraform.io/downloads.html) | 1.0.0 or higher |
| [Terraform EKS provider](https://learn.hashicorp.com/collections/terraform/aws-get-started) | ~>N.n |
| [Helm](https://helm.sh/) | 3 |
| [Amazon AWS SDK](https://aws.amazon.com/cli/)  | 2.2.0 |
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | 1.17.17 |
| [Python](https://www.python.org/) | 3 |
| [aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html) | 0.5.0 |

#### Install the Terraform binary

If you haven't already, install Terraform. Refer to the helpful Terraform installation video on this [hashicorp.com page](https://learn.hashicorp.com/tutorials/terraform/infrastructure-as-code?in=terraform/aws-get-started). Follow the instructions for your OS type, then return here.

Terraform install example for Ubuntu Linux:

```bash
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform
```

Verify the installation:

```bash
terraform version
```

**Output**:

```bash
Terraform v1.0.0
on darwin_amd64

Your version of Terraform is out of date! The latest version
is 1.0.1. You can update by downloading from https://www.terraform.io/downloads.html
```

#### Set up the AWS CLI v2

If you haven't already, set up the AWS CLI v2. The steps assume you already have an AWS account.

Follow the instructions in [Installing, updating, and uninstalling the AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).

Next, FYI, refer to [Installing aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html). As explained on that page,
if you're running the AWS CLI version 1.16.156 or later, you don't need to install the authenticator.

#### Install kubectl

If you haven't already, install `kubectl`. You'll use `kubectl` commands to interact with your K8ssandra resources.

One option to get `kubectl` is described in this AWS topic, [Installing kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html). See the OS-specific examples. Here's an example on Linux and the 1.17 Kubernetes version:

```bash
curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.17.12/2020-11-02/bin/linux/amd64/kubectl
```

Verify the `kubectl` install:

```bash
kubectl version --short --client
```

**Output**:

```bash
Client Version: v1.17.12
```

#### Install helm v3

If you haven't already, install Helm v3. See this EKS topic, [Using Helm with Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/helm.html). Note the  prerequisite: before you can install Helm charts on your Amazon EKS cluster, you must configure `kubectl` to work for Amazon EKS. If you have not already done this, see [Create a kubeconfig for Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html).

Once you've completed the prerequisites, see the [Helm install] steps for your OS. Here's a Linux example:

```bash
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
```

#### Install Python v3

If you haven't already, install Python version 3.x for your OS. See the [python downloads](https://www.python.org/downloads/) page.

### Checkout the `k8ssandra-terraform` project

Each of our reference deployment may be found in the GitHub [k8ssandra/k8ssandra-terraform](https://github.com/k8ssandra/k8ssandra-terraform) project. Download the latest release or current `main` branch locally.

```bash
git clone git@github.com:k8ssandra/k8ssandra-terraform.git
```

**Output**:

```bash
Cloning into 'k8ssandra-terraform'...
remote: Enumerating objects: 273, done.
remote: Counting objects: 100% (273/273), done.
remote: Compressing objects: 100% (153/153), done.
remote: Total 273 (delta 145), reused 233 (delta 112), pack-reused 0
Receiving objects: 100% (273/273), 71.29 KiB | 1.30 MiB/s, done.
Resolving deltas: 100% (145/145), done.
```

```bash
cd k8ssandra-terraform/aws
```

### Configure `aws` CLI

Ensure you have authenticated your `aws` client by running the following command:

```console
$ aws configure
AWS Access Key ID [None]: ....
AWS Secret Access Key [None]: ....
Default region name [None]: us-east-1
Default output format [None]:
```

### Setup Environment Variables

Set up the following environment variables for Terraform's use. Be sure to specify the region you're using in AWS.

```bash
export TF_VAR_environment=prod
export TF_VAR_name=k8ssandra
export TF_VAR_region=us-east-1
```

### Provision Infrastructure

We begin this process by initializing our environment and configuring a workspace. To start we run `terraform init` which handles pulling down any plugins required and configures the backend.

```console
cd env
terraform init
```

**Output**:

```console
Initializing modules...
- eks in ../modules/eks
- iam in ../modules/iam
- s3 in ../modules/s3
- vpc in ../modules/vpc

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.37.0...
- Installed hashicorp/aws v3.37.0 (self-signed, key ID 34365D9472D7468F)

# Output reduced for brevity

Terraform has been successfully initialized!
```

Now we configure a workspace to hold our terraform state information.

```console
terraform workspace new my-workspace
```

**Output**:

```bash
Created and switched to workspace "my-workspace"!

 Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
```

With the workspace configured we now instruct terraform to `plan` the required changes to our infrastructure (in this case creation).

```console
terraform plan
```

**Output**:

```console
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

# Output reduced for brevity

Plan: 50 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bucket_id        = (known after apply)
  + cluster_Endpoint = (known after apply)
  + cluster_name     = (known after apply)
  + cluster_version  = "1.19"
```

After planning we tell terraform to `apply` the plan. This command kicks off the actual provisioning of resources for this deployment.

```console
terraform apply
```

**Output**:

```console
# Output reduced for brevity

Plan: 50 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bucket_id        = (known after apply)
  + cluster_Endpoint = (known after apply)
  + cluster_name     = (known after apply)
  + cluster_version  = "1.19"

Do you want to perform these actions in workspace "my-workspace"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

# Output reduced for brevity

Apply complete! Resources: 50 added, 0 changed, 0 destroyed.

Outputs:

bucket_id = "prod-k8ssandra-s3-bucket"
cluster_Endpoint = "https://....us-east-1.eks.amazonaws.com"
cluster_name = "prod-k8ssandra-eks-cluster"
cluster_version = "1.19"
```

With the EKS cluster deployed you may now continue with [installing K8ssandra](#install-k8ssandra).

## Validate Kubernetes Cluster Connectivity

After provisioning the EKS cluster with `terraform apply` the local Kubeconfig will automatically be updated with the appropriate entries. Let's test this connectivity with `kubectl`.

```console
kubectl cluster-info
```

**Output**:

```console
Kubernetes control plane is running at https://.....
CoreDNS is running at https://..../api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```

```console
kubectl version
```

**Output**:

```console
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1
```

## Install K8ssandra

With all of the infrastructure provisioned we can now focus on installing K8ssandra. This will require configuring a service account for the backup and restore service, creating a set of Helm variable overrides, and setting up EKS specific ingress configurations.

### Create Backup / Restore Service Account Secrets
As part of deploying infrastructure with Terraform an IAM policy is created allowing the EKS cluster workers to access S3 for backup and restore operations. At this time as part of deploying Medusa we _must_ provide a secret for the pods to successfully get scheduled. In this case we will create an empty secret to bypass this limitation until [k8ssandra/k8ssandra#712](https://github.com/k8ssandra/k8ssandra/issues/712) is resolved.

```console
kubectl create secret generic prod-k8ssandra-medusa-key
```

**Output**:

```console
secret/prod-k8ssandra-medusa-key created
```

### Generate `eks.values.yaml`

Here is a reference Helm `values.yaml` file with configuration options for running K8ssandra in EKS.

{{< readfilerel file="eks.values.yaml"  highlight="yaml" >}}

{{% alert title="Important" color="primary" %}}
Take note of the comments in this file. If you have changed the name of your secret, are deploying in a different region, or have tweaked any other values it is imperative that you update this file before proceeding.
{{% /alert %}}

### Deploy K8ssandra with Helm

If you haven't already, add the latest K8ssandra repo:

```bash
helm repo add k8ssandra https://helm.k8ssandra.io
```

**Output**:

```bash
"k8ssandra" has been added to your repositories
```

To ensure you have the latest from all your repos:

```bash
helm repo update
```

**Output**:

```bash
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "k8ssandra" chart repository
Update Complete. ⎈Happy Helming!⎈
```

Now install K8ssandra and specify the `eks.values.yaml` file that you customized in a prior step:

```console
helm install prod-k8ssandra k8ssandra/k8ssandra -f eks.values.yaml
```

### Retrieve K8ssandra superuser credentials {#superuser}

You'll need the K8ssandra superuser name and password in order to access Cassandra utilities and do things like generate a Stargate access token.

{{% alert title="Tip" color="success" %}}
In `kubectl get secret` commands, be sure to prepend the environment name. In this topic's examples, we have used `prod-k8ssandra`. Notice how it's prepended in the examples below. Also, save the displayed superuser name and the generated password for your environment. You will need the credentials when following the
[Quickstart for developers]({{< relref "/quickstarts/developer" >}}) or [Quickstart for Site Reliability Engineers]({{< relref "/quickstarts/site-reliability-engineer" >}}) post-install steps.
{{% /alert %}}

To retrieve K8ssandra superuser credentials:

1. Retrieve the K8ssandra superuser name:

    ```bash
    kubectl get secret prod-k8ssandra-superuser -o jsonpath="{.data.username}" | base64 --decode ; echo
    ```

    **Output**:

    ```bash
    prod-k8ssandra-superuser
    ```

1. Retrieve the K8ssandra superuser password:

    ```bash
    kubectl get secret prod-k8ssandra-superuser -o jsonpath="{.data.password}" | base64 --decode ; echo
    ```

    **Output**:

    ```bash
    PGo8kROUgAJOa8vhjQrE49Lgruw7s32HCPyVvcfVmmACW8oUhfoO9A
    ```

## Cleanup Resources

If this cluster is no longer needed you may optionally uninstall K8ssandra or delete all of the infrastructure.

### Uninstall K8ssandra

```console
$ helm uninstall prod-k8ssandra
release "prod-k8ssandra" uninstalled
```

### Destroy EKS Cluster

```console
terraform destroy
```

**Output**:

```console
# Output omitted for brevity

Plan: 0 to add, 0 to change, 50 to destroy.

Do you really want to destroy all resources in workspace "my-workspace"?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

# Output omitted for brevity

Destroy complete! Resources: 50 destroyed.
```

## Next steps

With a freshly provisioned cluster on Amazon EKS, consider visiting the [developer]({{< relref "/quickstarts/developer" >}}) and [site reliability engineer]({{< relref "/quickstarts/site-reliability-engineer" >}}) quickstarts for a guided experience exploring your cluster.

Alternatively, if you want to tear down your Amazon EKS cluster and / or infrastructure, refer to the sections above that cover [cleaning up resources]({{< relref "#cleanup-resources" >}}).

branch_name: refs/heads/main


text:
# CocosNeat
Neat Games Bundle, Written in Cocos2d-x for iOS, Android and Win32

This repository contains full C++ source codes for:
* Game of Drones
* Neat Bundle
* Mesh
* Mesh X
* Contrast
* Contrastic
* Raid
* Snake
* Tree Massacre
* Hello Blobs
* Neat Platformer Engine
* Neat Tiley Engine
* Eggplant Hunter
* Matryoshka
* Fall
* + Some unreleased work-in-progress games!

Cocos2d-x version: cocos2d-x-3.3 Dec.12
CocosNeat version: Jan-17-2015

repository: samafshari/CocosNeat

repo_name: samafshari/CocosNeat


text:
# Amazon SQS for the AWS Service Broker
Provision, manage and connect to [Amazon SQS](https://aws.amazon.com/sqs/).

## Prerequisites

**IAM resources** - see the [AWS Service Broker Requirements](https://github.com/awslabs/aws-servicebroker-documentation/blob/master/Overview.md#requirements) for details

## Plans

### sqs-fifo
Creates a FIFO SQS Queue.

### sqs-standard
Creates an SQS Queue.

## Retained resources

No resources are retained. The SQS queue, data and all associated resources will be fully removed if the Service Instance is deleted.

## Resources

[Getting Started With OCP and the AWS Service Broker](https://github.com/awslabs/aws-servicebroker-documentation/blob/master/getting-started.md)
[AWS Service Broker Overview](https://github.com/awslabs/aws-servicebroker-documentation/blob/master/Overview.md)
[FAQ](https://github.com/awslabs/aws-servicebroker-documentation/blob/master/FAQ.md)
[Troubleshooting](https://github.com/awslabs/aws-servicebroker-documentation/blob/master/Troubleshooting.md)

## License

This library is licensed under the Apache 2.0 License.


text:
# AWS SSM Command using Lacework Provider

This example shows how to use the Terraform Provider for Lacework to create
a new Lacework Agent Token and use it to deploy an AWS System Manager Command
that can be used to install the Lacework Agent on a Linux EC2 instance.

```hcl
provider "aws" {
  region = "us-east-1"
}

provider "lacework" {}

resource "lacework_agent_access_token" "ssm_deployment" {
  name        = "ssm-deployment"
  description = "Used to deploy agents using AWS System Manager"
}

module "lacework_aws_ssm_agents_install" {
  source  = "lacework/ssm-agent/aws"
  version = "~> 0.4"

  lacework_agent_tags = {
    env = "dev"
  }

  aws_resources_tags = {
    billing = "testing"
    owner   = "myself"
  }

  lacework_access_token = lacework_agent_access_token.ssm_deployment.token
}

resource "aws_resourcegroups_group" "testing" {
  name = "Testing"

  resource_query {
    query = jsonencode({
      ResourceTypeFilters = [
        "AWS::EC2::Instance"
      ]

      TagFilters = [
        {
          Key    = "environment"
          Values = ["Testing"]
        }
      ]
    })
  }

  tags = {
    billing = "testing"
    owner   = "myself"
  }
}

resource "aws_ssm_association" "lacework_aws_ssm_agents_install_testing" {
  association_name = "install-lacework-agents-testing-group"

  name = module.lacework_aws_ssm_agents_install.ssm_document_name

  targets {
    key = "resource-groups:Name"
    values = [
      aws_resourcegroups_group.testing.name,
    ]
  }

  compliance_severity = "HIGH"
}
```


text:
# vanillajs-handlebars-boilerplate
A no-nonsense boilerplate for VanillaJS projects that uses Handlebars as a templating engine.

## 🚀 Quick start

To run this locally:

1. Clone project `git clone https://github.com/marknorrapscm/vanillajs-handlebars-boilerplate.git .`
2. Install dependencies: `npm install`
3. Run the app: `npm run start`
4. If your browser doesn't open it automatically, navigate to: [http://localhost:3000](http://localhost:3000)

## Maintenace
I will perform periodic package maintenance (i.e. every few months) so the repo doesn't end up with horrific `npm audit` results.


text:
## The thought process (development, approach)

### The process
1. Get familiar with Kotlin and his developing ecosystem
2. Understand well the requirements, explore the project and make a couple of requests to the existing resources
3. Think about the design of the solution
  - The system should be: performant, reliable, should handle transparently the different error types, the operations should be idempotent (ideally)
  - All the events should be registered (for traceability and reporting)
4. Code a very basic solution (just to have a working application)
5. Learn about the coroutines (I want to make the app fast, process invoices in parallel)
6. Find a reasonable library for dependency injection in Kotlin
7. Find a reasonable library for the scheduler (I don't want to reinvent the wheel)
8. Implement dependency injection (Dagger2)
9. Implement the scheduler (In-App)
10. Designing a pipeline that process invoices in parallel (channels, agents, commands) using kotlin coroutines
11. Do some refactorings
12. Implement an event service to keep track of all the events in the system (for traceability)
13. Write some tests
14. Refactor the code

### General principle
For the scope of this exercices I wanted to create a system based on passing messages between components (I want a solution that is microservice-fiendly).
The idea is that the app is small and is focused on doing one thing well (UNIX like philosophy). I created a pipeline (using coroutines) that uses channels
to pass messages (commands) between components and to distribute the main flow to different agents for parallel processing.
The system should be scalable and it should be relatively easy to add new features, that's why I introduced a couple of abstractions.
The pipeline can be used for other computations as we wrap the action in a command object. The pipeline expects a stream of commands and as long as these
commands respect the contract (Interfaces: Command and CommandResult) the pipeline will work (I added an image at the root of the project for the general principle).
I want to be able to scale (horizontally) the application

### The scheduler
I choosed to use a library and to make the scheduler in-app (we can also implement the scheduler as a cron job outside of the main application). The scheduling pattern is configurable in properties file. I started to implement
some methods so we can change the scheduler pattern without closing the app. Didn't finished that just to save time and to work more on
the parallel processing of invoices but the implementation of this mechanism is pretty straightforward. We can expose a new
resource that can change the scheduling pattern or use another method.

### Dependency Injection
I used Dagger2 for this purpose. It does a lot of magic and it was relatively difficult to bootstrap with dagger. A lot of difficulties when I started to
write some tests. Once you have a setup that works, dagger is very powerful and useful. Adding new services is very easy. It took me time to get confortable
with but it is an investment that pays off really fast after that and the code becomes more clean and elegant.

### The tests
This challenge was not coded using TDD as I did a lot of exploration in the same time, so TDD was not a good option in this particular case.

I made the error to leave writing the tests at the end. As I didn't had much time, I don't have a good coverage of tests for the 'Billing Service'.
I didn't do a lot of unit testing. I used the real components for the integration tests. I lost a lot of time setting up the tests with dagger.
It was difficult to test for the parallelism (the pipeline) but I can refactor a little bit the solution so it becomes more test friendly (ex: make the retry
strategy parametrable)

### Possible new features
- Generate reports from events
- Setting timeouts for coroutines
- Different retry mechanisms (parametrable)
- Add a notification system (to send automatically messages to clients and system administrators, for example when a payment fails)
- Apply discounts for specific clients
- and many more...

### Time Estimation
It took me a couple of days to learn the basics about Kotlin, the coroutines and some useful libraries that could be helpful to beat Antaeus.
For writing the code I made 4-5 sessions of coding (3 hours each) so in total it was roughly 12-15 hours of coding.

### Improvements (not coded do to self-imposed time limit) - getting closer to a production grade app
1. Dynamic number of workers (adapt the number of agents in function of the number of invoices to process and delay)
2.  Consider TimeZone
4. Mechanism of limiting the blast radius
5. Generating reports (for specific client, for specific date intercal)
6. Being able to pay just one invoice (another rest resource)
7. Validate the scheduling pattern
8. Make some methods to have less responsabilities (ideally just one)
9. Make some commands undoable

### Libraries
- kotlin coroutines
- dagger2
- www.sauronsoftware.it/projects/cron4j

### Notes
- I used properties file to make configurable some parameters of the application. I used it for the scheduling pattern but there are plenty of other
possibilities that can be configurable (ex: the retry mechanism, dynamic number of agents(workers) etc.)
- Some methods may appear too big (wall of text) (and some of them truly are :) ) but I leaved a lot of logging and comments in the code
so the person who will review will understand faster my intent. If we remove all the unneeded comments and logs the code will be much cleaner.
- As it is the first time that I use Kotlin, sometimes I felt like I didn't used fully the features he offers.
- This solution needs more refactoring and testing but for the scope of the exercices I'll let it like that for instant
- I'm open to discuss different suggestions and explain some of my decisions during a review

### Final thoughts
It was my first experience ever with Kotiln. I enjoyed doing the challenge and I learned a lot of new things. I'll consider using Kotlin for some
personal projects in the future and explore more of the powerful features he has to offer. It took me more time that I thought at the beginning of the challenge.

 He compelled all strangers who were passing through the country to wrestle with him. Whenever Antaeus touched the Earth (his mother), his strength was renewed, so that even if thrown to the ground, he was invincible. Heracles, in combat with him, discovered the source of his strength and, lifting him up from Earth, crushed him to death.

Welcome to our challenge.

## The challenge

As most "Software as a Service" (SaaS) companies, Pleo needs to charge a subscription fee every month. Our database contains a few invoices for the different markets in which we operate. Your task is to build the logic that will schedule payment of those invoices on the first of the month. While this may seem simple, there is space for some decisions to be taken and you will be expected to justify them.

## Instructions

Fork this repo with your solution. Ideally, we'd like to see your progression through commits, and don't forget to update the README.md to explain your thought process.

Please let us know how long the challenge takes you. We're not looking for how speedy or lengthy you are. It's just really to give us a clearer idea of what you've produced in the time you decided to take. Feel free to go as big or as small as you want.

## Developing

Requirements:
- \>= Java 11 environment

### Building

```
./gradlew build
```

### Running

There are 2 options for running Anteus. You either need libsqlite3 or docker. Docker is easier but requires some docker knowledge. We do recommend docker though.

*Running through docker*

Install docker for your platform

```
make docker-run
```

*Running Natively*

Native java with sqlite (requires libsqlite3):

If you use homebrew on MacOS `brew install sqlite`.

```
./gradlew run
```

### App Structure
The code given is structured as follows. Feel free however to modify the structure to fit your needs.
```
├── pleo-antaeus-app
|       main() & initialization
|
├── pleo-antaeus-core
|       This is probably where you will introduce most of your new code.
|       Pay attention to the PaymentProvider and BillingService class.
|
├── pleo-antaeus-data
|       Module interfacing with the database. Contains the database models, mappings and access layer.
|
├── pleo-antaeus-models
|       Definition of the "rest api" models used throughout the application.
|
├── pleo-antaeus-rest
|        Entry point for REST API. This is where the routes are defined.
└──
```

### Main Libraries and dependencies
* [Exposed](https://github.com/JetBrains/Exposed) - DSL for type-safe SQL
* [Javalin](https://javalin.io/) - Simple web framework (for REST)
* [kotlin-logging](https://github.com/MicroUtils/kotlin-logging) - Simple logging framework for Kotlin
* [JUnit 5](https://junit.org/junit5/) - Testing framework
* [Mockk](https://mockk.io/) - Mocking library
* [Sqlite3](https://sqlite.org/index.html) - Database storage engine

Happy hacking 😁!

repository: marcelbernic/antaeus

repo_name: marcelbernic/antaeus


text:
# WooliesChallengeApi
Created Open APIs for the challeges.

Implemented CQRS pattern for the API by using MediatoR (https://github.com/jbogard/MediatR).

Using simplified "Clean Architecture" for structure.

## Swagger Docs:
https://woolieschallengeapi2019.azurewebsites.net/swagger

## API Base URL:
https://woolieschallengeapi2019.azurewebsites.net/api/answers

## Getting Started

Just Run it ~~😆.

Use Swagger UI for testing.

## Technology
ASP.NET CORE,
MediatoR,
NSwag,
AutoMapper

### Prerequisites
```
ASP.NET CORE 2.1 Runtime
```

repository: mariusDing/WooliesChallengeApi

repo_name: mariusDing/WooliesChallengeApi


text:
pgmock2
=======

[![Build Status](https://travis-ci.com/jfavrod/pgmock2.svg?branch=master)](https://travis-ci.com/jfavrod/pgmock2)
![Bundle Size](https://img.shields.io/bundlephobia/min/pgmock2.svg)
[![Requirements Status](https://requires.io/github/jfavrod/pgmock2/requirements.svg?tag=v1.0.1)](https://requires.io/github/jfavrod/pgmock2/requirements/?tag=v1.0.1)

An NPM module for mocking a connection to a PostgreSQL database.

The module mocks a [pg](https://www.npmjs.com/package/pg) module
connection to a PostgreSQL database. Both the `pg.Client` and `pg.Pool`
classes have a `query` method, therefore the mock connection can be
used to simulate an instance of either class.

Installation
------------
Installation via `npm`.
```
npm i --dev-save pgmock2
```

Use
---
The idea is to simulate a connection to a database. To enable that
simulation, we need to first `add` data.

```javascript
const
pgmock = require('pgmock2'),
pgMock = pgmock();

pgMock.add('SELECT * FROM employees WHERE id = $1', ['number'], {
    rowCount: 1,
    rows: [
        { id: 0, name: 'John Smith', position: 'application developer' }
    ]
});
```

Now we can create a mock connection and query for this data.

```javascript
const conn = pgMock.connect();

conn.query('select * from employees where id=$1;', [0])
.then(data => console.log(data))
.catch(err => console.log(err.message));
```

Since the query is valid and the values passed are correct in number
and type, we should see the expected output.

```
{ rowCount: 1,
  rows:
   [ { id: 0, name: 'John Smith', position: 'application developer' } ] }
```

gha_license_id: MIT


text:
# Example using Rust

If you want to combine the **Web Fundamentals & Modern UX** of Remix together with the **Reliability, Performance & Efficiency** of Rust, you can use functions built with Rust on your server. Useful for intensive computations such as on the fly machine learning tasks, fibonacci etc.

## Preview

Open this example on [CodeSandbox](https://codesandbox.com):

[![Open in CodeSandbox](https://codesandbox.io/static/img/play-codesandbox.svg)](https://codesandbox.io/s/github/remix-run/examples/tree/main/rust)

## Example

This example uses Rust compiled to WASM.

If you don't have Rust installed on your computer the first thing to do is to get this set up

Installing Rust:

```sh
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

Here we use `wasm-pack` to comile our Rust code down to WASM

Installing [wasm-pack](https://github.com/rustwasm/wasm-pack):

```sh
cargo install wasm-pack
```

In this example, the Rust library is already built with the associated code. But if you wanted to set up your own library you could do so by running the following command:

```sh
cargo new --lib <library-name>
```

Then build the library using:

```sh
cd <library-name>
wasm-pack build --target nodejs
```

After succesfully bulding the library you can add this to your dependencies by running inside your remix project:

```sh
npm install ./<library-name>/pkg
```

This will add the dependency to your `package.json` file.

```json
    ...
    "@remix-run/serve": "1.1.3",
    "<library-name>": "file:<library-name>/pkg"
    ...
```

## Notes:

_To prevent remix from including the rust-functions library in the client build we can re-export the functions using a `.server.ts` file, e.g. [rust.server.ts](app/rust.server.ts)_

## Related Links

[Rust](https://rust-lang.org/)

[Wasm-pack](https://github.com/rustwasm/wasm-pack)

[Tensorflow & WebAssembly](https://blog.tensorflow.org/2020/03/introducing-webassembly-backend-for-tensorflow-js.html)

repository: lyquocnam/examples

repo_name: lyquocnam/examples


text:
# Triodos Manager CSV converter
This simple Python script converts transactions in a format provided by the `Triodos Bank` to a format usable by `Manager.io`.
My company uses Triodos banking and the free [Manager.io](https://www.manager.io) desktop client for bookkeeping.
Using this converter I am able to download transaction records in csv and import them in `Manager.io`.

# Usage

1. Download the `csv` from the online banking Triodos.
2. Run `python convert-csv.py <triodos.csv>`.
3. Import the csv file into `Manager.io`.

# Example
python convert-csv.py example_data/example-triodos.csv


text:
---
layout: post
title: 'Rails 5.0.0.beta2: Slashed Action Cable dependencies, fixes galore'
category: releases
author: dhh
published: true
date: 2016-02-02 17:00:00 +01:00
---
Progress waits for nobody: The second beta release of Rails 5 is out, and it's packed with a six weeks worth of fixes and upgrades.

The big news is that Action Cable no longer depends on Celluloid, Redis, or even EventMachine! We've doubled down on concurrent-ruby, added a PostgreSQL alternative adapter to Redis for pubsub, and added a non-EventMachine Redis adapter too. So if you were freaking out over the new dependencies in beta1, you can breathe easy again. The Rails community has, as per-usual, stepped up and Made It Better. Special thanks to Mike Perham, Jon Moss, and Matthew Draper for their work on this!

Beyond that, there's literally [25 pages of commits on GitHub](https://github.com/rails/rails/commits/v5.0.0.beta2?page=25) detailing the work done since the beta1 release on December 18. It's a good stroll through the work it takes to go from beta to beta.

The release targets from here are RC1 on February 16 and then final on February 23. All depends on how much stuff pops up from beta2 and RC1, though. So don't order cake or champagne for delivery on those dates just yet!

If you missed the announcement on what's new in Rails 5, [checkout the beta1 story](http://weblog.rubyonrails.org/2015/12/18/Rails-5-0-beta1/). Oh, and thanks to Sean Griffin for coordinating this release.

repository: rails/weblog

branch_name: refs/heads/gh-pages

extension: markdown

gha_language: HTML

repo_name: rails/weblog