text:
#Messaging System
***

Message bases are sometimes referred to as "subs".  These are areas on the BBS where messages related to particular subject areas may be kept.  Any topic you can imagine may be used for a sub. Common examples include politics, science fiction, entertainment, sports, trivia, etc.  WWIV based networks have subs on every imaginable topic.  You may want to join such a network to offer a more diverse array of topics in your message bases.

The very first sub on your BBS should be one where you alone can post messages.  This is your place to make announcements concerning your system etc.  You can force users to read this sub any time there are new messages using the USE_FORCE_SCAN and FORCE_SCAN_SUBNUM INI settings.  See WWIV.INI for more information on these settings. Users who have the appropriate SL and ARs can read the message bases and post on them. These messages are not private and may be read by anyone else who has the appropriate SL and ARs.

**NOTE:** Because the more message bases and directories that you create, the more memory is required for pointers and so forth, it is advisable to set the number of message bases to the next highest multiple of 32 over the number that you actually intend to use. This will help conserve memory.

Message bases are setup by using the BOARDEDIT feature. This feature may be accessed in the following ways:

1. From WFC, enter B and then enter the system password when prompted for it.
2. From the main menu, enter //BOARDEDIT and then enter the system password when prompted for it.
3. From the main menu, enter //BE (short form for BOARDEDIT and the system password when prompted for it.

**BoardEdit**
Using BoardEdit, you can delete, insert, and modify subs.  The default installation sets up one sub-board named General.

The first thing that is seen when BoardEdit is accessed is a Substring? prompt. If you wish to go directly to an existing subboard, enter a portion of the name, otherwise hit enter for a complete listing.

You will see a screen that looks like the following after you have chosen to insert or modify your listing.  A description of each setting is included below:
```
ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
³   Subboard      : 0                              ³
³   A. Name       : General                        ³
³   B. Filename   : GENERAL                        ³
³   C. Key        : None.                          ³
³   D. Read SL    : 10                             ³
³   E. Post SL    : 20                             ³
³   F. Anony      : No                             ³
³   G. Min. Age   : 0                              ³
³   H. Max Msgs   : 50                             ³
³   I. AR         : None.                          ³
³   J. Net info   : Not networked.                 ³
³   K. Storage typ: 2                              ³
³   L. Val network: No                             ³
³   M. Req ANSI   : No                             ³
³   N. Disable tag: No                             ³
³   O. Description: None.                          ³
³                                                  ³
³   Which (A-O,[,],Q) ?                            ³
ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
```
**A)  Name**
Each sub-board needs a name, describing what type of
messages are expected on that sub-board. If your BBS is a
member of a WWIV network, the software will insert either <Net>
or [Net] preceding the sub-board name when the sub listing is
obtained.  The difference between these two uses of NET is
discussed below under the Network Validation Option.

**B.  Filename**
Each sub-board also needs a UNIQUE filename. If
two subs have the same filename, BOTH SUBS WILL REFER TO THE
SAME MESSAGES.  The filename can be whatever you want (up to 8
letters in length), but it must not contain an extension.  For
example, WSSYSOP would be a legal name to enter, but
WSSYSOP.SUB would be illegal because it uses an extension.  It
is recommended that you develop a naming convention for your
networked message bases to identify the file that belongs to a
particular sub.  Using the sub above, if you subscribed via
WWIVnet, you may want to name the file WWSSYSOP.  The prepended
W identifies the file as subtype WSSYSOP on WWIVnet.

**C) Key**
Normally, all sub-boards are assigned a number (in
ascending order, starting at one for your first sub-board)
which callers can press to immediately access them. If you
would prefer that a sub-board be referred to by a key (such
as %,&,#, etc) instead of a number, setting the key to that
character will do so. This option will not normally be used.

**D) Read SL**
This setting determines the minimum SL required to
allow a caller to read messages on the sub-board. If the read
SL for a sub is 40, and a caller's SL is 39, that caller will
never know that the sub-board exists.

**E) Post SL**
This setting determines the minimum SL required
for a caller to post on the sub. The post SL should be greater
or equal to the read SL. Having the post SL greater than the
read SL would allow callers to read messages on the sub, but
not post any.  This type of setting is often useful on some
network subs where anyone may be permitted to read the sub, but
posting may be only by certain persons (perhaps the sysop
and/or the co-sysops.)

**F)  Anony**
Setting the anony (no, yes, forced, dear abby)
determines whether or not messages posted on that sub can be
anonymous. No means no anonymous (unless the caller has a
security level which permits it); yes means the caller has the
option of posting anonymously, regardless of security level;
forced means the post is always anonymous; dear abby means that
the caller will be presented a choice of "signatures" for
his/her post: their name (alias), Abby, or Problemed Person.
In effect, the "dear abby" option does the same thing as
setting anony to yes, but it gives the caller two options on
which name will represent the author of the post.  This option
is ignored on all networked message bases.

**G) Min age**
This will allow you to specify the minimum age of
callers on that sub.  If, for example, you set the minimum age
to 18 (years), anyone under 18 would not know the sub existed.
This setting is normally used for adult-oriented subjects.

**H)  Max msgs**
This sets the maximum number of messages that
can be on the sub at one time. This is usually set to 50 or
100, but can be up to 999.  When the maximum number is reached,
the oldest post in the sub is rotated off (deleted) as each new
post is entered, unless a post has been given permanent status.
A post with permanent status means that it will not be erased.
This is useful if you have posted the sub rules or something as
the first message on the sub.  To give permanent status to a
post, you hit N after reading the post.  You will see the
message, "This post will NOT be auto-purged.

**I)  AR**
This allows you to specify the AR for that sub. AR is
used to allow specific people to access specific subs when
other sub limitations (age, post SL, etc) are not sufficient
for your needs. As an example, suppose you have a group of
Amiga callers on your BBS. You want to give them their own area
to talk about Amigas. Age and SL restrictions would be
cumbersome and inefficient.  That is where AR comes into play.
There are 16 different AR flags that can be used, A through P.
To give the Amiga callers their own sub, you would use this
setting to assign an AR to the sub.  You can use any letter
(A-P) you wish, as long as it isn't in conflict with another
sub which doesn't concern the Amiga people (you can use the
same AR on different subs, but realize that anybody with the
same AR flag set would be able to access all subs with the
identical AR).  For this example, you could set the AR flag to
A. Then you would go into UEDIT and activate each Amiga
caller's A flag.  Now, all Amiga callers would be able to
access the Amiga sub (as long as SL and age requirements, if
any, were met) and nobody else would know the sub exists.

**J)  Net info**
Used with WWIV network software on network subs
to properly allocate messages. If the sub you are editing is not
a WWIV network sub, leave this blank.  If you select this
option, you will get a sub-menu asking for information
regarding Network, Type, Host, and Flags. As you enter the
information, you will receive helpful prompts.  The Networks
that are recognized are those that you have setup in the INIT
program.

**NOTE:** When setting up a networked message base, this is the
first  setting you should enter.  Some of the fields will be
filled out for you if you have a current SUBS.LST file for
the specified network.

TYPE refers to the Subtype.  Under WWIV v4.21a and NET31 or
earlier, these subtypes had to be integer values; under NET32
and WWIV v4.22, you may use 7 letter character names for the
subtype.  This option is referred to in WWIV literature as
"sub-by-name".

HOST refers to the network host of the sub; if you host it,
the information will display HERE; otherwise, the host's
network node number will be shown.

SUBS refers to the number of subscribers on the sub and is
displayed only when you are the sub host.

FLAGS refers to whether the sub is auto-requestable and whether
or not the information about the sub should be reported in the
SUBS.LST distributed on each network. For more information on
this, refer to the WWIVnet Documentation that accompanies the
Network Software.

There will also be an appended category on subs you host.  If
this is chosen, a list of  available categories (distributed
by the NC for the network) will be given.  The purpose of this
is to allow the sysop to select the most appropriate category
(for that sub information) for listing in the Subs List that
is distributed by each network.

**K) Storage type**
There are 2 main ways messages can be
stored, either with each message in its own file, or with all
messages in that message base in one large file. 0 means each
message will be in its own (text) file, 2 means all messages
in that sub will be in one large (non-text) file.

**L) Val Network**
This option may be toggled on or off.  If
toggled on, sub-board that are network subs, will have the
network name enclosed in brackets (i.e. [WWIVnet]) in front of
the name and the sysop will need to validate the messages
received there before the messages will be sent over the
network.  If the option is toggled off, the sub will have
the network name enclosed in v-brackets (i.e. <WWIVnet> in
front of the Sub-Board name.  Because many network subs have
various restrictions (such as no ANSI posts, or posts from #1
on the BBS only, etc), network validation is a means of
insuring that posts originating from this BBS are in compliance
with the sub-host's rules.

**M) Req. ANSI**
The sysop can require that the user have ANSI
enabled in order to view this sub.  This option is particularly
useful if the sub is one of ANSI pictures drawn with The Draw or
some other ANSI editor.  An example of an all ANSI sub is the
WWIVnet Sub known as The ANSI Art Gallery.

**N) Disable Tag**
This option, if toggled on, will disable
taglines in the messages. This may be useful if a network sub
host does not permit posts which have taglines added to them.

**O) Description**
This option permits you to supply a
description of a sub that you host.  This description will
be used in the SUBS.LST file maintained by the network. The
entry in this field makes it possible for subs you host to
be included in the SUBS.LST without your intervention; the
only requirement is that the auto-info option in FLAGS be set
when the sub is set up.

repository: BloodStoneBBS/docs

repo_name: BloodStoneBBS/docs


text:
### Status
[![Build Status](https://travis-ci.org/chad-autry/alpine-rdiff-backup.svg?branch=master)](https://travis-ci.org/chad-autry/alpine-rdiff-backup)
[![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://registry.hub.docker.com/u/chadautry/alpine-rdiff-backup/)

## Synopsis

Containerized rdiff-backup. Mount the volume(s) required, and otherwise call docker alpine-rdiff-backup as if it were the rdiff-backup executable.

license: MIT

repository: chad-autry/alpine-rdiff-backup

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: chad-autry/alpine-rdiff-backup


text:
---
layout: default
---

# 1993-10-13 - Re: Spread-spectrum net (vulnerability of)

## Header Data

From: Matthew J Ghio<EMAIL_ADDRESS>To<EMAIL_ADDRESS>(Stuart W. Card)

Message Hash: 03564c14d82af1b6ddae14bc8e816172b12b5f44ddbf6acad99eb4d4e4ef8468

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1993-10-13 03:09:57 UTC

Raw Date: Tue, 12 Oct 93 20:09:57 PDT

## Raw message

```
{% raw  %}From: Matthew J Ghio<EMAIL_ADDRESS>Date: Tue, 12 Oct 93 20:09:57 PDT
To<EMAIL_ADDRESS>(Stuart W. Card)
Subject: Re: Spread-spectrum net (vulnerability of)
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

> Has anyone pointed out that the ideal location for these
> packet repeaters is in the cars of as many people as possible?

This would defeat many detection methods, but would complicate the
software further, because it would need to be able to quickly compensate
for repeaters which travelled out of range.

{% endraw %}
```

## Thread

+ Return to [October 1993](/archive/1993/10)

+ Return to "[Matthew J Ghio <mg5n+@andrew.cmu.edu>](/authors/matthew_j_ghio_mg5n_at_andrew_cmu_edu_)"
+ Return to<EMAIL_ADDRESS>(Stuart W. Card)](/authors/swc_at_uc1_ucsu_edu_stuart_w_card_)"

+ 1993-10-13 (Tue, 12 Oct 93 19:16:39 PDT) - [Re: Spread-spectrum net (vulnerability of)](/archive/1993/10/b65bb6af6f5859fa84dcf1f255a96592305f3484b242c24ce96d87784fb26362) -<EMAIL_ADDRESS>(Stuart W. Card)_
  + 1993-10-13 (Tue, 12 Oct 93 20:09:57 PDT) - Re: Spread-spectrum net (vulnerability of) - _Matthew J Ghio<EMAIL_ADDRESS>

gha_license_id: MIT


text:
<pre>
npm install
cake build

#--> lib/js-chart-widgets.min.js
</pre>

## Calendar

    new Calendar({
      append_to: 'some_id',
      num_weeks: 51,
      day_size: 11,
      data: [
        ['2012-03-01', 123],
        ['2012-03-02', 456.7],
        // gaps are fine
        ['2012-03-04', 234]
      ],
      color: '#005500'
    });

![](https://github.com/loule/js-chart-widgets/blob/master/doc/calendar.png)

## The Means

- Wrap or borrow from other open-source widgets when it makes sense
- Get our NIH on when it doesn't

## The End

- Make a sensible abstraction with nice defaults for each of the world's widget types

![](http://imgs.xkcd.com/comics/standards.png)


text:
## Migration

In this section, we will describe how to to configure and use Gohan
migrations.

Gohan migrations are based on goose library fork 'cloudwan/goose'
(see http://github.com/cloudwan/goose). The library is used internally
so user does not need to have an external working goose binary.
All migration tasks are exposed with a single 'gohan migrate' command.

Gohan migrate command has different subcommands which are similar
to goose subcommands. Each command takes "--config-file" parameter
to specify a configuration file. For example:

```
  gohan migrate status --config-file etc/gohan.yaml
```

will display current migration status. Remaining commands include:

##### up: Migrate to the most recent version

This subcommand will walk though all pending migrations and apply
each pending migration until there is an error or all migrations
were applied.

##### up-by-one: Migrate one version up

Migrate only one pending migration. This is useful to separately test
if each pending migration does not introduce any new problems to
the system.

##### create: Create a template for a new migration
Complete command line is:
```bash
gohan migrate create <name> [sql/go]
```
This subcommand will create an empty template for migrations either in GO or in SQL. By default
a new migration is created in GO if no migration type parameter is given.

##### init: Create an initial migration

This subcommand is used to create an initial migration from an empty
database to the current version of all schemas.

##### down: Migrate to the oldest version

This subcommand reverts all applied migrations.

##### redo: Migrate one version back

This subcommand reverts the last applied migration.

##### status: Display migration status

This command displays a complete summary of available migrations and their
status. A migration can be in one of the two states:
- applied: a migration has been applied and the corresponding status contains
the time/date when the migration was applied
- pending: a migration has not yet been applied

##### version: Display migration version

This subcommand will display the current version of database.

### Configuration

Migrations in SQL and in GO are read from a common 'migrations' directory
which is defined as key 'database/migrations' in Gohan configuration file.
By default it is set to "db/migrations":
```yaml
database:
    migrations: "db/migrations"
```

### SQL migrations

SQL migrations are the same as in goose. Please refer to the original
documentation for the details.

### Precompiled GO migrations

Gohan uses precompiled shared libraries to store migrations in go.
In order to create a migrations library, one can use the following command
in migrations directory:

```
  go build -buildmode=plugin -o gohan_go_migrations.so `find . -maxdepth 1 -type f -name "*.go"`
```

Note that each migration can be precompiled as separate libraries
or alternatively, all migrations can be stored in a single library.
Both solutions are handled by Gohan.

repository: cloudwan/gohan

gha_language: Go

gha_license_id: NOASSERTION

repo_name: cloudwan/gohan


text:
# SAP HANA Submodule

This module handles opinionated SAP HANA configuration and deployment.

## Usage

The resources/services/activations/deletions that this module will create/trigger are:

- Create a Compute Instance that will host SAP HANA
- Create a Static IP Adresse for the Compute Instance
- Create a 2 Persistent Disks to host SAP HANA's File systems

You can go in the [examples](../../examples) folder complete working example. However, here's an example of how to use the module in a main.tf file.

```hcl
provider "google" {
  version = "~> 2.6.0"
}

module "gcp_sap_hana" {
  source                 = "terraform-google-modules/sap/google/modules/sap_hana"
  subnetwork                 = "${var.subnetwork}"
  linux_image_family         = "${var.linux_image_family}"
  linux_image_project        = "${var.linux_image_project}"
  instance_name              = "${var.instance_name}"
  instance_type              = "${var.instance_type}"
  project_id                 = "${var.project_id}"
  region                     = "${var.region}"
  zone                       = "${var.zone}"
  service_account_email      = "${var.service_account_email}"
  boot_disk_type             = "${var.boot_disk_type}"
  boot_disk_size             = "${var.boot_disk_size}"
  autodelete_disk            = "true"
  pd_ssd_size                = "${var.pd_ssd_size}"
  pd_hdd_size                = "${var.pd_hdd_size}"
  sap_hana_deployment_bucket = "${var.sap_hana_deployment_bucket}"
  sap_deployment_debug       = "false"
  post_deployment_script     = "${var.post_deployment_script}"
  startup_script             = "${var.startup_script}"
  sap_hana_sid               = "${var.sap_hana_sid}"
  sap_hana_instance_number   = "${var.sap_hana_instance_number}"
  sap_hana_sidadm_password   = "${var.sap_hana_sidadm_password}"
  sap_hana_system_password   = "${var.sap_hana_system_password}"
  network_tags               = "${var.network_tags}"
  sap_hana_sidadm_uid        = 900
  sap_hana_sapsys_gid        = 900
  address_name               = "${var.address_name}"
}
```
## Requirements

Make sure you've gone through the root [Requirement Section](../../README.md#requirements)

### SAP HANA Software
 Follow instructions [here](https://cloud.google.com/solutions/sap/docs/sap-hana-deployment-guide#creating_a_cloud_storage_bucket_for_the_sap_hana_installation_files) to properly Download SAP HANA from the SAP Marketplace, and upload it to a GCS bucket.

### Configure Service Account for identifying the Compute instance
The compute instance created by this submodule will need to download SAP HANA from a GCS bucket in order install it. Follow the instructions below to ensure a successful installation:

 1. [Create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts)
 2. Grant this new service account the following permissions on the bucket where you uploaded SAP HANA installation file:
    - Stackdriver Metadata writer: `roles/stackdriver.resourceMetadata.writer`
    - Compute Network admin: `roles/compute.networkAdmin`
    - Log writer: `roles/logging.logWriter`
    - Storage viewer: `roles/storage.objectViewer`

  You may use the following gcloud command:

```shell
ROLES=("roles/stackdriver.resourceMetadata.writer", "roles/compute.networkAdmin", "roles/logging.logWriter", "roles/storage.objectViewer")
for ROLE in ${ROLES[*]}; do
  gcloud projects add-iam-policy-binding <project-id> \
    --member=serviceAccount:<service-account-email> --role=${ROLE}
done
```

3. When configuring the module, use this newly created service account's email, to set the `service_account_email` input variable.

### Post deployment script
If you need to run a post deployment script, the script needs to be accessible via a **https:// or gs:// URl**.
The recommended way is to use a GCS Bucket in the following way.:

1. Upload the post-deployment-script to a GCS bucket.
2. Grant the following role on the bucket to the service account attached to the instance if the bucket is not in the same project as the service account:
   - Storage Object Viewer: `roles/storage.objectViewer`

 3. Set the value of the `post_deployment_script` input to the URI of the post-deployment script storage bucket object, like `gs://<bucket_name>/<script_name>`.

[^]: (autogen_docs_start)

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|:----:|:-----:|:-----:|
| address\_name | Name of static IP adress to add to the instance's access config. | string | n/a | yes |
| autodelete\_disk | Whether the disk will be auto-deleted when the instance is deleted. | string | `"false"` | no |
| boot\_disk\_size | Root disk size in GB. | string | n/a | yes |
| boot\_disk\_type | The GCE boot disk type. May be set to pd-standard (for PD HDD) or pd-ssd. | string | n/a | yes |
| device\_name\_pd\_hdd | device name for standard persistant disk | string | `"backup"` | no |
| device\_name\_pd\_ssd | device name for ssd persistant disk | string | `"pdssd"` | no |
| disk\_name\_0 | Name of first disk. | string | `"sap-hana-pd-sd-0"` | no |
| disk\_name\_1 | Name of second disk. | string | `"sap-hana-pd-sd-1"` | no |
| disk\_type\_0 | The GCE data disk type. May be set to pd-ssd. | string | `"pd-ssd"` | no |
| disk\_type\_1 | The GCE data disk type. May be set to pd-standard (for PD HDD). | string | `"pd-standard"` | no |
| instance\_name | A unique name for the resource, required by GCE. Changing this forces a new resource to be created. | string | n/a | yes |
| instance\_type | The GCE instance/machine type. | string | n/a | yes |
| linux\_image\_family | GCE image family. | string | n/a | yes |
| linux\_image\_project | Project name containing the linux image. | string | n/a | yes |
| network\_tags | List of network tags to attach to the instance. | list | n/a | yes |
| pd\_hdd\_size | Persistent disk size in GB. | string | `""` | no |
| pd\_ssd\_size | Persistent disk size in GB. | string | `""` | no |
| post\_deployment\_script | SAP HANA post deployment script. Must be a gs:// or https:// link to the script. | string | `""` | no |
| project\_id | The ID of the project in which the resources will be deployed. | string | n/a | yes |
| region | Region to deploy the resources. Should be in the same region as the zone. | string | n/a | yes |
| sap\_deployment\_debug | Debug flag for SAP HANA deployment. | string | `"false"` | no |
| sap\_hana\_deployment\_bucket | SAP hana deployment bucket. | string | n/a | yes |
| sap\_hana\_instance\_number | SAP HANA instance number | string | n/a | yes |
| sap\_hana\_sapsys\_gid | SAP HANA SAP System GID | string | n/a | yes |
| sap\_hana\_sid | SAP HANA System Identifier. When using the SID to enter a user session, like this for example, `su - [SID]adm`, make sure that [SID] is in lower case. | string | n/a | yes |
| sap\_hana\_sidadm\_password | SAP HANA System Identifier Admin password | string | n/a | yes |
| sap\_hana\_sidadm\_uid | SAP HANA System Identifier Admin UID | string | n/a | yes |
| sap\_hana\_system\_password | SAP HANA system password | string | n/a | yes |
| service\_account\_email | Email of service account to attach to the instance. | string | n/a | yes |
| startup\_script | Startup script to install SAP HANA. | string | n/a | yes |
| subnetwork | The name or self_link of the subnetwork where the isntance will be deployed. The subnetwork must exist in the same region this instance will be created in. | string | n/a | yes |
| zone | The zone that the instance should be created in. | string | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| instance\_name |  |
| machine\_type |  |
| zone |  |

[^]: (autogen_docs_end)

repository: arvindpai/terraform-google-sap

gha_language: HCL

repo_name: arvindpai/terraform-google-sap


text:
# An overview of C-Programming Language

* C++ is based on C programming Language.

* **ANSI/ISO C Standard** is a base document for International C++ Standard.

* All C programs are also C++ programs but C++ does not use everything defined in CPP:
	* File I/O in CPP is object oriented and rarely uses C type I/O
	* Pre-processors are rarely given as much importance as in C language.

## C is a Middle Level Language

### Middle Level Language

C combines the best elements of high-level languages with the control and flexibility of assembly languages.

### Portability

C programs are portable and cross-platform.

### Support for Data Types

* Data Types define a set of values and a set of operations that can be performed.

* C has five in-built data types - int, char, float, long and double.

* C is not strongly typed - it permits type conversions.

### C does not provide Exception handling

Runtime error checking is not supported in C.

### C does not demand strict type compatibility between formal and actual arguments

Automatic conversion are supported.

### System Level Programming

C allows direct manipulation of bits, bytes, words and address pointers.

### Keywords

C only has 32 keywords (27 of original specification and 5 added by ANSI).

## C is a Structured Language

* C and C++ are not block structured.

* Allows **compartmentalization** - differentiation between **code** and **data**.

* Local temportary variables are better for isolation and tight control. Global variables might lead to bugs spread entire code.

* Use of **Goto** is discouraged

* Allows placing statement anywhere on the line!

```C
printf      ("This is valid")   ;
```

* Main structural component for C is **functions** which are also the only sub-routine allowed in C

* **Code blocks** can also be used for compartmentalization

```C
x = 1; //1
{
x = 10; x++; //11
}
x++; //2
```

## C is a programmer's language

* C was created by the programmers and is made for the programmers.

* Allows efficiency, flexibility and direct system access like Assembly language

* Allows structured code and portability, which were drawbacks in Assembly Language.

## Keywords in C

* There are 32 Keywords in C

* All keywords are in complete lowercase.

* 5 out of 32 keywords were not in original C keyword list - enum, cosnt, void, volatile and signed.

![C Keywords](img/C_Keywords.png)

## Basic Form of C program

![C programming form](img/BasicProgram.png)

* Most compiler for C and C++ come with standard library.

* C++ standard library can be divided into two: **Standard Library Function** (C-library functions) and **Class Library** including OOPS based functions and **Standard Template Library (STL)**

* All the functions stored in C library are in **relocatable** format. Their absolute address is not stored, only offset. **Linking and loading process**

**Note:** C and C++ allows separate compilation. We can compile different codes and link them into one object code.

repository: karangurtu/CPP-Handbook

repo_name: karangurtu/CPP-Handbook


text:
# WAVIG
The Waterborne Vessel Identification Generator
http://wavig.kbarkevich.com

WAVIG is a web application built to simplify the task of coming up with a name for your watery means of transportation.

#### If nautical nonsense is something you wish, WAVIG is the tool for you.

*This project uses the GitFlow model for all source control contributions.*

repository: kbarkevich/WAVIG

gha_language: PHP

repo_name: kbarkevich/WAVIG


text:
# Ping

This handler sends one ICMP echo request to the specified host and reports an error if it is unsuccessful.

> Note that this may require some sort of elevated privilege to be able to run. For example, on Linux, it needs either to be run as `root` (not recommended), or to have the `CAP_NET_RAW` capability.
>
> To set `CAP_NET_RAW`, you can execute the following command on Defcon's binary:
>
> ```shell
> setcap cap_net_raw+ep defcon
> ```

## Attributes

| Attribute | Type   | Example     | Description                                 |
| --------- | ------ | ----------- | ------------------------------------------- |
| `kind`    | string | `"ping"`    | -                                           |
| `host`    | string | `"<IP_ADDRESS>"` | Host to which to send the ICMP echo request |

repository: apognu/defcon

gha_language: Rust

repo_name: apognu/defcon


text:
AQSWhatsAppActivity
===============

![](http://img.shields.io/cocoapods/v/AQSLINEActivity.svg?style=flat) [![Build Status](https://travis-ci.org/AquaSupport/AQSLINEActivity.svg)](https://travis-ci.org/AquaSupport/AQSLINEActivity)

[iOS] UIActivity for WhatsApp

Usage
---

```objc
UIActivity *whatsAppActivity = [[AQSWhatsAppActivity alloc] init];
NSArray *items = @[[UIImage<EMAIL_ADDRESS>
UIActivityViewController *activityViewController = [[UIActivityViewController alloc] initWithActivityItems:items withApplicationActivities:@[whatsAppActivity]];

[self presentViewController:activityViewControoler animated:YES completion:NULL];
```

Installation
---

```
pod "AQSWhatsAppActivity"
```

Related Projects
---

- [AQSShareService](https://github.com/AquaSupport/AQSShareService) - UX-improved share feature in few line

repository: mohsinalimat/AQSWhatsAppActivity

repo_name: mohsinalimat/AQSWhatsAppActivity


text:
---
layout: post
title: "Semi-automatic System for Title Construction"
date: 2019-05-01 19:49:41
categories: arXiv_CL
tags: arXiv_CL
author: Swagata Duari, Vasudha Bhatnagar
mathjax: true
---

* content
{:toc}

##### Abstract
In this paper, we propose a semi-automatic system for title construction from scientific abstracts. The system extracts and recommends impactful words from the text, which the author can creatively use to construct an appropriate title for the manuscript. The work is based on the hypothesis that keywords are good candidates for title construction. We extract important words from the document by inducing a supervised keyword extraction model. The model is trained on novel features extracted from graph-of-text representation of the document. We empirically show that these graph-based features are capable of discriminating keywords from non-keywords. We further establish empirically that the proposed approach can be applied to any text irrespective of the training domain and corpus. We evaluate the proposed system by computing the overlap between extracted keywords and the list of title-words for documents, and we observe a macro-averaged precision of 82%.

##### Abstract (translated by Google)

##### URL
[http://arxiv.org/abs/1905.00470](http://arxiv.org/abs/1905.00470)

##### PDF
[http://arxiv.org/pdf/1905.00470](http://arxiv.org/pdf/1905.00470)


text:
# Service Agreement

This service agreement is agreed upon by the following parties:

* {{client.name}}, located at {{client.address}}
* {{supplier.name}}, located at {{supplier.address}}

## Terms of Service

1. Some
2. Terms
3. Will
4. Go
5. Here

## Hourly Rate

The hourly rate will be {{supplier.currency}} {{supplier.hourlyRate}}.

## Signatures

Signed,

{{#signatories}}
![Signature]({{signature}}) \
{{name}}

Signed on: {{signed_date}}

{{/signatories}}


text:
# Efficient Algorithms for Device Placement of DNN Graph Operators - code

This code package contains algorithms and input files (DNN workloads) from the paper "Efficient Algorithms for Device Placement of DNN Graph Operators" published at NeurIPS 2020.
It allows one to reproduce the results in the paper, as well as run the partitioning algorithms on other workloads.

Throughout this package (code and input files), "FPGA" is used is to refer to an arbitrary accelerator. In particular, for layer-granularity graphs we actually use profiled processing time (latency) data from a GPU deployment scenario.

## Input format

All our algorithms take as input a JSON file with the following format (all fields are mandatory unless indicated otherwise). This format closely follows our model (see Section "Computational Model" in the paper):
* `maxSizePerFPGA` (floating-point): a memory size limit of a single FPGA/accelerator, in bytes,
* `maxFPGAs` (integer): number of accelerators (`k` from the paper),
* `maxCPUs` (integer): number of CPU cores (`\ell` from the paper),
* `nodes` (array): for each node (layer or operator in the DNN):
    * `id` (integer): unique ID of node,
    * `supportedOnFpga` (boolean or 0/1 integer): whether the node can be placed on an accelerator,
    * `cpuLatency` (floating-point): time it takes to execute this node on a CPU,
    * `fpgaLatency` (floating-point): time it takes to execute this node on an accelerator,
    * `isBackwardNode` (boolean or 0/1 integer): whether this is a backward node (only used in training workloads),
    * `colorClass` (integer, optional): if two nodes share the same color class, then they need to be colocated on the same device,
    * `size` (floating-point): memory taken by this node on an accelerator, in bytes,
* `edges` (array): for each edge:
    * `sourceId` (integer): the ID of the tail of the edge (edge from `sourceId` to `destId`),
    * `destId` (integer): the ID of the head of the edge,
    * `cost` (floating-point): time it takes to transfer the output of node with ID `sourceId` between an accelerator and CPU DRAM. Should be the same for all edges going out of the same node.

Other debug information may be present in the input files, such as `name`s or `layerId`s on nodes and `size`s on edges.

## Throughput maximization (max-load minimization)

Here we give two algorithms: DP (dynamic programming) and IP (integer programming) based. We also give several baselines. Input files are placed in the directory `throughput-inputs`.

### Dynamic programming solution

The solution is implemented in `throughput-dp/dp.cpp`. It is a single C++ file (using one header-only library for JSON parsing) and can be compiled with a recent version of `gcc` by running e.g. `g++ -O3 throughput-dp/dp.cpp -o dp.exe`.

The compiled program takes the input network in the JSON format outlined above via standard input. It can be executed as follows: `./dp.exe < throughput-inputs/OperatorGraphs/bert_l-3_inference.json`. It produces an output JSON file on standard output and diagnostic messages on standard error.

### Integer programming solution

This solution requires the solver Gurobi to be installed and have an active license.
The solution is implemented in Python 3.7.

To run the solution, execute `python throughput-ip/ip.py contig` or `python throughput-ip/ip.py noncontig` depending on whether contiguous splits are desired.
It also takes the input on standard input, so an example run could be `python throughput-ip/ip.py contig < throughput-inputs/OperatorGraphs/bert_l-3_inference.json`. The Gurobi solver will print information about the progress of the optimization to file `gurobi.log`.

Further parameters than can be adjusted include:
* the line `model.setParam("MIPGap", 0.01)` controls the optimality gap at which optimization will stop (currently 1%),
* the line `model.setParam("TimeLimit", 1200)` controls the time limit after which optimization will stop (currently 20 minutes),
* the line `model.setParam("MIPFocus", 1)` controls the solver strategy (currently it should focus on finding as good solutions as possible rather than refining the lower bound),
* a flag `ALSO_FORCE_ON_CPUS` in `ip.py` toggles whether contiguity should also be forced on subgraphs that go onto CPU cores,
* a second command-line argument `diffmaxima` can be used to toggle optimization of the alternative objective function `max_i FW_i + max_i BW_i` (see Appendix C).

### Scotch baseline

First compile the DP solution. Then run it with a command-line parameter `-scotch`, e.g.: `./dp.exe -scotch < throughput-inputs/OperatorGraphs/bert_l-3_inference.json`. For this to work, Scotch needs to be installed and its `bin` directory should be added to the system PATH. To check if this is the case, run `gpart` in the console. Alternatively, you can change the constant `scotchExecutable` in `dp.cpp`.

### Human-expert baseline

We provide human-expert splits in the directory `human-experts`. The `dp` program can compute the max-load (Time-Per-Sample) for a given split file using the command-line parameter `-expert`. The second parameter should be the split file. E.g. `./dp.exe -expert human-experts/resnet50_inference_expert.json < throughput-inputs/LayerGraphs/resnet50_inference.json`. If there is no human expert split for a training workload, use the corresponding inference split - the `dp` program will infer the split for the backward pass from the split for the forward pass.

### Local search baseline

Add an option `-localSearch` and a second parameter specifying the number of restarts (we used 10).

### PipeDream baseline

Use option `-pipeDream`.

### Place everything on a single FPGA

Use option `-oneFpga`.

## Latency minimization

Here we give an IP (integer programming) based algorithm,
as well as two baselines. Input files are placed in the directory `latency-inputs`.

### Integer programming solution

This solution requires the solver `Gurobi` to be installed and have an active license.
The solution is implemented in Python 3.7.

To run the solution, execute `python latency-ip/lip.py`.
It also takes the input on standard input, so an example run could be `python latency-ip/lip.py < latency-inputs/OperatorGraphs/bert_l-3_inference.json`. The Gurobi solver will print information about the progress of the optimization to file `gurobi.log`.

Parameters that can be adjusted include:
* the same three Gurobi optimizer parameters as for throughput maximization above (currently the time limit is set to 1 hour),
* a parameter `MAX_SUBGRAPHS_PER_FPGA` that allows one to obtain non-contiguous splits. This is the `q` parameter from Appendix A.

### Greedy baseline

The solution, described in Appendix F, is implemented in `latency-greedy/greedy.cpp`. It is a single C++ file (using one header-only library for JSON parsing) and can be compiled with a recent version of `gcc` by running e.g. `g++ -O3 latency-greedy/greedy.cpp -o greedy.exe`.

The compiled program takes the input network in the JSON format outlined above via standard input. It can be executed as follows: `./greedy.exe < latency-inputs/OperatorGraphs/bert_l-3_inference.json`. It produces an output JSON file on standard output and diagnostic messages on standard error.

### Max-load DP baseline

The same program can alternatively take any split (in the output JSON format produced by our implementations) and compute the minimum single-sample latency that can be obtained with this split. To do this, run `greedy.exe` with two command-line parameters: `-computeLatency` and the path to the JSON file containing the split. The input workload should still be given via standard input.

Our second baseline for latency minimization, described in Appendix F, consists in running the throughput-maximization (max-load minimization) DP solution and computing the latency that it obtains. To do so, run e.g.:
* `./dp.exe < latency-inputs/OperatorGraphs/bert_l-12_inference.json > bert-l12-split.json`
* `./greedy.exe -computeLatency bert-l12-split.json < latency-inputs/OperatorGraphs/bert_l-12_inference.json`

### Human expert baseline

See "max-load DP baseline" above - the same program can compute the latency for a given split. Run e.g.:
* `./greedy.exe -computeLatency human-experts/bert24_inference_expert.json < latency-inputs/LayerGraphs/bert24_inference.json`

### Scotch baseline

First run `dp` with `-scotch` option (see above) to produce the split. Then use `greedy` to compute the latency of the split. E.g.:
* `./dp.exe -scotch < latency-inputs/OperatorGraphs/bert_l-12_inference.json > bert-l12-split-scotch.json`
* `./greedy.exe -computeLatency bert-l12-split-scotch.json < latency-inputs/OperatorGraphs/bert_l-12_inference.json`

## Legal notices

**Trademarks**
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
or contact [opencode<EMAIL_ADDRESS>with any additional questions or comments.

We use the [JSON for Modern C++](https://github.com/nlohmann/json) library, copyright (c) 2013-2020 Niels Lohmann, licensed under the MIT license.

repository: pinxuezhao/dnn-partitioning

branch_name: refs/heads/main

repo_name: pinxuezhao/dnn-partitioning


text:
---
title:		"Alberto & David"
type:		"photos"
mediatype:		"upload"
location:		"Berlin, Germany"
date:		"2014-04-19 19:44:29+00:00"
album:		"people"
filename:		"alberto-david-unicorns.md"
series:		"unicorns"
cl_public_id:		"people/alberto-david-unicorns"
cl_version:<PHONE_NUMBER>
format:		"tiff"
bytes:		3820968
width:		2158
height:		1440
colours:
- "#0D140A"
- "#1A110C"
- "#040403"
- "#12160B"
- "#080A0F"
- "#010205"
- "#13110B"
- "#130703"
- "#7F4F37"
- "#D79574"
- "#000101"
- "#080D10"
- "#010407"
- "#171418"
exposure_mode:		"Auto"
program:		"Program AE"
aperture:		"2.8"
focal_length:		"112.0 mm"
iso:		"250"
shutter_speed:		"1/320"
metering:		"Spot"
flash:		"Off, Did not fire"
white_balance:		"Custom"
colour_temp:		"4550"
has_crop:		"false"
orientation:		"Horizontal (normal)"
camera_model:		"NIKON D800"
lens_info:		"70-200mm f/2.8"
artist: "Matt Finucane"
x_resolution:		"300"
y_resolution:		"300"
---

repository: matfin/cinematt

gha_language: HTML

repo_name: matfin/cinematt


text:
# Maypop

> Work in Progress: Features are added on a daily basis.

__Maypop__ is a simple implementation of Apect Oriented Programming (AOP) for any ECMAScript5 engine.
Hence it works on Node.js and modern browsers.

# Status

[ ![Codeship Status for xcambar/maypop](https://www.codeship.io/projects/e9fb6f50-ca97-0130-5ff8-0ac4a233f6f3/status?branch=master) ](https://www.codeship.io/projects/4840)

# Aspect oriented programming

Aspect oriented programming allows developers to add behaviour (_aspects_) to functions and
objects by declaring functions that should be run _before_, _after_ and _around_ other methods.

# Example

    var maypop = require('maypop');

    var noAspects = function () {

    };
    var withAspects = maypop(noAspects);

    withAspects.before(function () {/*   */);
    withAspects.before(function () {/*   */);
    withAspects.before(function () {/*   */);

    withAspects.after(function () {/*   */);
    withAspects.after(function () {/*   */);

    //Execute
    withAspects();

    //Need to find better than "just an example"
    //meanwhile, you should read the tests to have a clear view of what's possible

# Features

## Add run points to

* functions

## Runpoints available

* __before__: `before` runpoints are executed on their own context, and take the same parameters as the initial function.
* __after__: `after` runpoints are executed on their own context, and take the return value of the initial function as unique parameter

# Licence

MIT

repository: xcambar/maypop

repo_name: xcambar/maypop


text:
AngularJS
==========

Day 2 is about single page applications with AngularJS.

The presentation is available at:
http://smancke.github.io/angularjs-intro/


text:
Time to make chlodnik soup pink coloured - we need some fresh beet juice. Most probably you can't buy it in grocery. Since beetroot is more like a potato, than a juicy fruit, electrical juicer cannot be overestimated :) I should mention salutary effect, that a red beetroot juice has on your health.

Add some salt to taste, and do not forget a pinch of powdered sugar. Soup should be gently but firmly sour.

Peel one fresh, green cucumber and dice it.  Both diced cucumber and sliced radishes are another necessary addition to chlodnik.

Chlodnik should be left to 'mature' in a cool place for 2-3 hours (put it into your fridge).

When ready, serve chlodnik in bowls with a hard boiled egg, cut into quarters. Both eggs and chlodnik should be cold.

Notes
=====

Some people like to add small pieces of good quality cold cuts, or roast veal. For sure, there's a reason if chlodnik is going to be served as a main dish. Therefore it should has some more sustenance. More exquisite addition is a cooked shrimp, but it isn't Polish tradition at all. And remember that chlodnik should always be served cold (almost directly from your fridge).

repository: JUULShawn/Cooking-Recipes

repo_name: JUULShawn/Cooking-Recipes


text:
+++
title = "Permanent notes in a Zettelkasten"
author = ["Georgi Bojinov"]
date = 2021-03-02T00:00:00+02:00
slug = "20210302231738-permanent-notes-in-a-zettelkasten"
draft = false
+++

## Note {#note}

A permanent note, or a zettel, is the main building block of a zettelkasten. It's a single, atomic,
self-sufficient idea that connects to other, already existing ideas in the slip box. A permanent note can also
be added to at any time as new information surfaces, so it "grows" in a way. It's the main product of the note
taking process, and serves as a basis for writing afterwards.
Permanent notes most commonly appear as a joining of literature notes.

So, what to consider when making a permanent note out of ideas from a literature note - will I use this later in
writing or is this relevant to other already existing notes?

Tag notes by the context in which you wish to find them.

A good idea I had found is notes being sentences or parts of sentences, which you can then reference in other notes.

Permanent notes have:

-   permanent note tag
-   topic tags
-   links to a literature note
-   relevant other permanent notes and explanations why and how they are linked
-   summary of the idea

## Links to this note {#links-to-this-note}

-   [Stop Taking Regular Notes; Use a Zettelkasten Instead]({{< relref "20210302231143-stop_taking_regular_notes_use_a_zettelkasten_instead" >}})

> [Simplistic view of Zettelkasten workflow]({{< relref "20210302232348-simplistic_view_of_zettelkasten_workflow" >}})


text:
# StyleCI Bugsnag

StyleCI Bugsnag was created by, and is maintained by [Graham Campbell](https://github.com/GrahamCampbell), and is a Bugsnag bridge for Laravel 5. Feel free to check out the [change log](CHANGELOG.md), [releases](https://github.com/StyleCI/Bugsnag/releases), [license](LICENSE), and [contribution guidelines](CONTRIBUTING.md).

## WARNING

#### This package is no longer maintained.

![StyleCI Bugsnag](https://cloud.githubusercontent.com/assets/2829600/6657233/3473a988-cb3d-11e4-979c-a96f4ee7ff9f.png)

## Installation

[PHP](https://php.net) 5.5+ or [HHVM](http://hhvm.com) 3.6+, and [Composer](https://getcomposer.org) are required.

To get the latest version of StyleCI Bugsnag, simply add the following line to the require block of your `composer.json` file:

```
"styleci/bugsnag": "0.1.*"
```

You'll then need to run `composer install` or `composer update` to download it and have the autoloader updated.

Once StyleCI Bugsnag is installed, you need to register the service provider. Open up `config/app.php` and add the following to the `providers` key.

* `'StyleCI\Bugsnag\BugsnagServiceProvider'`

## Usage

StyleCI Bugsnag is designed to manage user provided bugsnag and protect against bad input. There is currently no real documentation for this package, but we are open to pull requests.

## License

StyleCI Bugsnag is licensed under [The MIT License (MIT)](LICENSE).

repository: GrahamDeprecated/Bugsnag

repo_name: GrahamDeprecated/Bugsnag


text:
# TopNotch

[![Build Status](https://travis-ci.org/blackrock/TopNotch.svg?branch=master)](https://travis-ci.org/blackrock/TopNotch) [![codecov.io](https://codecov.io/github/blackrock/TopNotch/coverage.svg?branch=master)](https://codecov.io/github/blackrock/TopNotch?branch=master)

## What Is TopNotch?

TopNotch is a system for quality controlling large scale data sets. It addresses the following three problems:

1. How to define and measure data quality
2. How to efficiently ensure data quality across many data sets
3. How to institutionalize existing knowledge of data sets

TopNotch uses rules to verify individual components of a data set. Each rule defines and measures some small component of data quality. The combination of rules provides a complete definition of and metrics for quality in a data set. The rules can be reused on other data sets to maximize efficiency. Finally, the clear definitions and reuseability of these rules allows users to institutionalize knowledge by documenting a data set.

## Getting Started

TopNotch has three requirements:

1. Installing SBT, version 0.13.9 or greater
2. Installing Spark 1.6
3. Setting ```$SPARK_HOME``` to the top-level folder of the Spark installation.

Follow the below steps to run TopNotch:

1. Clone this repo.
2. Get the latest JAR either by building this project or by downloading it from the releases portion of TopNotch's GitHub page. Place it in this project's top level __bin__ folder.
2. Create the configuration files to test your data set.
    * See the __example__ folder for a sample data set and configuration files.
3. Run __bin/TopNotchRunner.sh__ with all the configuration files passed as arguments. Ensure that your plan file is the first argument.
    * To try the example, run the commands ```chmod u+x bin/TopNotchRunner.sh``` and then ```bin/TopNotchRunner.sh example/plan.json example/assertions.json``` . Please note that the example must be run from the root folder of the TopNotch project, as shown in the prior commands.
4. View the resulting report and parquet file in the __topnotch__ folder in your home directory on HDFS.
    * To view the results of the example, look at the JSON file __topnotch/exampleAssertionReport__ and the Parquet file __example/exampleAssertionOutput.parquet__ .

Please note that you must change __bin/TopNotchRunner.sh__ in order to run TopNotch with a master other than local. It is currently recommended that you run TopNotch in local or client mode.

## High-Level Technical Summary

At the top level there are _plans_. These _plans_ group together _commands_ into a series that is run on one or more data sets.
TopNotch quality controls data using three _commands_: _assertions_, _diffs_, and _views_. _Assertions_ encode rules. _Diffs_ and _views_ transform data sets so that they can be processed by rules.

- _Assertion_: define and measure metrics of data quality
- _Diff_: create a new data set by comparing two other data sets with similar schemas
- _View_: create a new data set by joining or selecting a subset of existing data sets to be used in diffs and assertions

The _commands'_ reusable configuration settings are defined in files separate from the _plan_. The setting externalParamsFile in the _command_ references that file. See the section Components for a more complete description.

Here is a basic example of TopNotch. It is a _plan_ which contains one _command_, an _assertion_.

##### The Plan

```javascript
    {
      "topnotch": [ {
        "command": "assertion",
        "externalParamsFile": "testAssertion.json",
        "input": {
          "ref": "viewKey",
          "onDisk": false
        },
        "outputKey": "exampleAssertionKey",
        "outputPath": "/user/durst/topnotch/exampleAssertionOutput.parquet"
      } ]
    }
```

##### The Assertion

```javascript
    {
      "topnotch" : {
        "assertions" : [ {
          "query": "loanBal > 0",
          "description": "Loan balances are positive",
          "threshold": 0.01
        } ]
      }
    }
```

## Full Technical Details

Go to http://blackrock.github.io/TopNotch to see the Scaladocs.

### Terminology
- Dataframe: A nested, tabular data set
- Assertion: A rule _command_. It defines a measure of quality and filters all rows which are invalid according to that metric
    - Invalid: A row is considered "invalid" if it does not pass the query clause of an assertion
    - Failure: An assertion "fails" if the fraction of rows that it declares to be "invalid" in a data set is greater than a user-specified threshold.
- View: A _command_ that transforms one or more data sets into a single data set against which assertions can be run
- Diff: A _command_ that transforms two data sets into one by joining them on a unique key and then comparing user-specified columns
- Plan: A user-defined combination of assertions, views, and diffs
- Row: A single data point in a dataframe

### Components

There are four components to TopNotch. Each component is persisted in JSON format.
The following list describes each component in-depth and demonstrates how to write the JSON files for each of these components:

1. The _Plan_: This runs a series of commands that quality controls data. Subsequent commands in the _plan_ can depend on the outputs of previous ones. The user can have zero, one, or any greater number of operations of each type of views, diffs, and assertions. There must be at least one operation.
    * Each _command_ takes at least one input. The _view command_ takes a list of inputs in the JSON field "inputs". The _diff command_ takes two inputs in the fields "input1" and "input2". The _assertion command_ takes one input in the field "input". Each input is either a character-delimited or Parquet file on HDFS or a dataframe in memory that is the output of a previous _command_ in the _plan_.
        * Set "onDisk" to "true" if the input is a file on disk and "false" if it is the result of a previous _command_.
        * Set "delimiter" only if the input is a character-delimited file. In that case, set the option to the file's delimiter.
    * Each _command_ outputs a dataframe that can be referred to by later commands. Set the name of this with the "outputKey" value. If the user want this value to be cached for faster access, set "cache" to "true". The "cache" flag is optional and defaults to "false" if not specified.
    * Each _command_ can write its output to disk. Set "outputPath" to a path on HDFS if the result of a command is to be persisted. Relative paths will be relative to the user's home directory on HDFS. If "outputPath" is not set, the _command_'s result will not be not persisted.
    * For each "externalParamsFile" entry, enter the path to the file relative to the plan. Currently, it is recommended that users place the plan and all associated command files in the same directory.
    * For each assertion _command_, a report is written to the __~/topnotch__ folder on HDFS in JSON with the name of the command's output key.

    ```javascript
    {
      "topnotch": [
        {
          "command": "view",
          "externalParamsFile": "testView.json",
          "inputs": [
            {
              "ref": "topnotch/viewInput.csv",
              "onDisk": true,
              "delimiter": ","
            }
          ],
          "outputKey": "viewKey",
          "cache": true
        },
        {
          "command": "diff",
          "externalParamsFile": "testDiff.json",
          "input1": {
            "ref": "topnotch/currentLoans.parquet",
            "onDisk": true
          },
          "input1Name": "cur",
          "input2": {
            "ref": "topnotch/oldLoans.parquet",
            "onDisk": true
          },
          "input2Name": "old",
          "outputKey": "diffKey",
          "outputPath": "topnotch/diffOutput.parquet"
        },
        {
          "command": "assertion",
          "externalParamsFile": "testAssertion.json",
          "input": {
            "ref": "viewKey",
            "onDisk": false
          },
          "outputKey": "assertionKey",
          "outputPath": "topnotch/assertionOutput.parquet"
        }
      ]
    }
    ```

2. The _Assertion_ Runner: For each assertion command in a plan, this applies a number of _assertions_ to a data set, produces a dataframe containing all the rows declared invalid by any _assertion_, and creates a summary of how well the  data set abides by the assertions run against it.
    * The query uses syntax from the where clause of a HiveQL query. Each query (where clause) defines rows that are declared valid. Those not selected by the where clause are declared invalid.
    * The format for an assertion json file is:

    ```javascript
    {
      "topnotch" : {
        "assertions" : [
          {
            "query": "loanBal > 0",
            "description": "Loan balances are positive",
            "threshold": 0.01
          },
          {
            "query": "loanBal > 1",
            "description": "Loan balances are greater than 1",
            "threshold": 0.02
          }
        ]
      }
    }
    ```

3. The _Diff_ Creator: For diff each command in a plan, this joins two data sets on columns that form a unique key and then compares the values in other columns of the data sets.
    * Columns in equal locations in the "joinColumns" arrays are the keys to join the two data sets.
    * Columns in equal locations in the "diffColumns" arrays are the columns to be compared.
    * There must be at least one set of join columns and another of diff columns. The "joinColumns" arrays must have the same number of elements and so too must the "diffColumns" arrays.
    * Columns can be joined and compared even if they have different names because comparisons are determined by positions in the "joinColumns" and "diffColumns" arrays.
    * The format for a _diff_ json file is:

    ```javascript
       {
         "topnotch": {
           "input1Columns": {
             "joinColumns": [
               "loanID",
               "poolNum"
             ],
             "diffColumns": [
               "loanBal"
             ]
           },
           "input2Columns": {
             "joinColumns": [
               "loanIDOld",
               "poolNumOld"
             ],
             "diffColumns": [
               "loanBalOld"
             ]
           }
         }
       }
    ```

4. The _View_ Creator: For each view command in a plan, this takes in one or more data sets and produces a single dataframe based on a HiveQL query defined by the user. Use this to transform data into a form against which _diffs_ and _assertions_ can be run .
    * The inputs specified in the _command_ file will be loaded as tables with names specified in the "tableAliases" array for the query.
    * The result of the query is the _command_'s output.
    * The format for a _view_ json file is:

    ```javascript
    {
      "topnotch": {
        "tableAliases": [
          "loanData"
        ],
        "query": "select * from loanData"
      }
    }
    ```

Copyright © 2016 BlackRock, Inc. All Rights Reserved.

repository: David-Durst/TopNotch

repo_name: David-Durst/TopNotch


text:
# Development Guide

## Docker Components

![Docker Dashboard](/site/img/docker-dashboard.png)

## Development Environment Setup

### IDE integration (optional)

Create an RVM gemset so IDEs (like VS Code) will be able to use Ruby integration with Solargraph and Rubocop for formatting and linting.

```console
$ cd docker
```

```console
~/docker $ rvm use 2.6.6@opencspm --create --ruby-version
```

```console
~/docker $ gem install bundler:2.0.2
```

```console
~/docker $ bundle
```

Launch IDE from within `docker` directory:

```console
~/docker $ code .
```

### Launch Full Environment

To just run the full environment locally, run `docker-compose -f docker-compose-development.yml up` from within the `docker` directory. The initial run will take several minutes - at roughly 1 minute to build the Docker image, roughly 2 more minutes to install all Ruby gems (`core` container) and JavaScript packages (`ui` container).

Most of the console output won't be visible while the initial build is happening, though you can view status in the Docker Desktop dashboard or with `docker-compose logs`.

```console
$ cd docker
$ docker-compose -f docker-compose-development.yml up
Creating network "opencspm_default" with the default driver
Creating volume "opencspm_gems" with default driver
Creating volume "opencspm_postgres" with default driver
Creating volume "opencspm_redis" with default driver
Creating volume "opencspm_bundle" with default driver
Creating volume "opencspm_node_modules" with default driver
Creating volume "opencspm_rails_cache" with default driver
Creating opencspm_db_1       ... done
Creating opencspm_redis_1 ... done
Creating opencspm_redis_ui_1 ... done
Creating opencspm_shell_1    ... done
Creating opencspm_core_1     ... done
Creating opencspm_ui_1       ... done
Creating opencspm_worker_1   ... done
Attaching to opencspm_redis_1, opencspm_db_1, opencspm_redis_ui_1, opencspm_core_1, opencspm_shell_1, opencspm_ui_1, opencspm_worker_1
```

The `worker` container depends on the `core` container being fully built and running, so it will always complete last. When `sidekiq` log messages appear, all containers should be up and running.

```console
worker_1    | 2020-10-08T19:10:01.283Z pid=1 tid=gnzrya5pl INFO: Booted Rails <IP_ADDRESS> application in development environment
worker_1    | 2020-10-08T19:10:01.284Z pid=1 tid=gnzrya5pl INFO: Running in ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
worker_1    | 2020-10-08T19:10:01.284Z pid=1 tid=gnzrya5pl INFO: See LICENSE and the LGPL-3.0 for licensing details.
worker_1    | 2020-10-08T19:10:01.284Z pid=1 tid=gnzrya5pl INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
worker_1    | 2020-10-08T19:10:01.284Z pid=1 tid=gnzrya5pl INFO: Starting processing, hit Ctrl-C to stop
```
As such if there are errors in the `worker` container such as:
```
worker_1    | bundler: command not found: sidekiq
worker_1    | Install missing gem executables with `bundle install`
opencspm_worker_1 exited with code 127
```
Just wait for the `core` container to finish initializing.

### Application URLs & Endpoints

* Web UI: http://localhost:8000
* Rails API: http://localhost:5000
* Sidekiq console: http://localhost:5000/sidekiq
* Redis Insight UI: http://localhost:8001

### Development Environment Details

To re-deploy and re-build the Docker images
```console
$ cd docker
$ docker-compose -f docker-compose-development.yml up --build
```

To build or re-build the Docker image, use `docker-compose build`.

```console
$ cd docker
$ docker-compose build
```

A local console/shell is available on the `shell` container. To drop into the shell, use `docker-compose exec`. For example, to run a Rails console in this container:

```console
$ docker-compose exec shell bash
app@shell:~$ bundle exec rails c
Running via Spring preloader in process 24
Loading development environment (Rails <IP_ADDRESS>)
irb(main):001:0>
```

It is also possible to run a Rails console directly from `docker-compose exec`, for example:

```console
$ docker-compose exec shell bundle exec rails c
Running via Spring preloader in process 43
Loading development environment (Rails <IP_ADDRESS>)
irb(main):001:0>
```

### Restarting Containers

Frontend UI (HTML/JavaScript) and Rails changes are reflected in their respective containers in real-time. However, certain changes will require a container restart. For example:

```console
$ docker-compose restart core
```

### Running Rails Rspec Tests

Run tests in a dedicated container (with the correct environment set) and remove it afterwards:

```console
$ docker-compose run core bundle exec rspec -e RAILS_ENV=test
```

### Clean up

To stop the containers but leave the volumes intact:

```console
$ docker-compose down
```

To stop the contianers and remove the volumes:

```console
$ docker-compose down --remove
$ docker volume prune
```

To re-run a clean "initial" setup, remove the `.init` artifacts after deleting volumes, then re-run `docker-compose up`:

```console
$ rm .init ui/.init
$ docker-compose up
```

## Custom Control Pack Development

OpenCSPM supports the addition and modification of the control metadata and control check logic.  Groups of related controls are distributed as git repositories, and they are expected to be kept as subdirectories of the `opencspm-controls` directory.  For example:

```console
|- opencspm-controls
|  |- opencspm-darkbit-community-controls
|  |- your-custom-control-repo-here
|  '- another-custom-control-repo-here
'- opencspm
```

### Control Pack Required Files

* `config.yaml` - holds the index and metadata for the controls
* `controls_spec.rb` - contains the custom control check logic in [RSpec3](https://rspec.info/) format

### Creating a Custom Control Pack

The recommended way to begin writing custom controls is to modify an existing control pack.  Consider forking [https://github.com/OpenCSPM/opencspm-darkbit-community-controls](https://github.com/OpenCSPM/opencspm-darkbit-community-controls) and modifying the `config.yaml` and `controls_spec.rb` as needed.

Notes:

* In `config.yaml`, the `id:` and `controls > id:` fields must be globally unique across all control packs.  The convention is for the top level `id:` to be the [Kebab case](https://en.wikipedia.org/wiki/Letter_case#Special_case_styles) format of the repository name, and the control IDs to be `<orgname>-<groupingname>-<numberidentifier>`.  e.g. `darkbit-aws-42`.
* Tags have two levels.  The top level is meant for a higher order grouping like a compliance framework, and the nested level is meant for listing out all the applicable tags.  Tags are arbitrary strings, but the convention is to list multiple tags that represent the various levels of a compliance requirement.  For example, the top level tag `nist-csf` is meant to indicate that this control has one or more associations with the NIST Cybersecurity Framework.  As subtags, `nist-csf-pr`, `nist-csf-pr.ac` and `nist-csf.pr.ac-1` indicate that this control is in the `Protect` Function, the `Identity Management and Access Control` Category of the `Protect` Function, and that it maps to the specific ID `1` in that Category.  This allows for tag-based grouping in the UI at any level of depth and specificity.
* Currently, the responsibility is on the user for ensuring the contents of the control pack repos stay in sync with the upstream git repo where they are hosted.

### Querying the Redisgraph Database

* Launch the `docker-compose-development.yaml` to also run the [RedisInsight](https://redislabs.com/redis-enterprise/redis-insight/) container from [RedisLabs](https://redislabs.com).
* Visit [http://localhost:8001](http://localhost:8001)
* Accept the EULA if needed
* Click on Redisgraph on the left, and add a new database.  Supply `redis`, `redis`, and `6379` for the name, hostname, and port, respectively.
* Once inside and your data (or the sample data) has been loaded, you can run [Cypher](https://www.opencypher.org/) queries against the `opencspm` "database".


text:
# crunchy

 * [Crunchy Asian Chicken Salad](../../index/c/crunchy-asian-chicken-salad-243126.json)
 * [Crunchy Chicken In Green Sauce](../../index/c/crunchy-chicken-in-green-sauce-107283.json)
 * [Crunchy Chicken Salad](../../index/c/crunchy-chicken-salad-1120.json)
 * [Crunchy Chicken Salad With Shiitake Mushrooms Spicy Sesame Dressing And Peanuts](../../index/c/crunchy-chicken-salad-with-shiitake-mushrooms-spicy-sesame-dressing-and-peanuts-101251.json)
 * [Crunchy Crab Salad](../../index/c/crunchy-crab-salad-56389510.json)
 * [Crunchy Fried Shrimp With Cayenne Aioli](../../index/c/crunchy-fried-shrimp-with-cayenne-aioli-3075.json)
 * [Crunchy String Bean Salad With Red Onion And Prosciutto](../../index/c/crunchy-string-bean-salad-with-red-onion-and-prosciutto-235715.json)
 * [Crunchy Turnip Apple And Brussels Sprout Slaw](../../index/c/crunchy-turnip-apple-and-brussels-sprout-slaw-51260300.json)
 * [Crunchy Veg Bowl With Warm Peanut Sauce](../../index/c/crunchy-veg-bowl-with-warm-peanut-sauce.json)
 * [Crunchy Vegetable Rolls With Soy Dipping Sauce](../../index/c/crunchy-vegetable-rolls-with-soy-dipping-sauce-1227.json)
 * [Crunchy Wasabi Crusted Fish With Red Cabbage Slaw](../../index/c/crunchy-wasabi-crusted-fish-with-red-cabbage-slaw-235465.json)
 * [Crunchy Wasabi Salmon With Lime](../../index/c/crunchy-wasabi-salmon-with-lime-237322.json)
 * [Crunchy Winter Slaw With Asian Pear And Manchego](../../index/c/crunchy-winter-slaw-with-asian-pear-and-manchego.json)
 * [Ginger Yogurt With Berries And Crunchy Caramel](../../index/g/ginger-yogurt-with-berries-and-crunchy-caramel-239055.json)
 * [Grilled Salmon With Crunchy Sweet Mustard Vinaigrette](../../index/g/grilled-salmon-with-crunchy-sweet-mustard-vinaigrette-109542.json)
 * [Marinated Lentils With Crunchy Vegetables](../../index/m/marinated-lentils-with-crunchy-vegetables.json)
 * [Poached Chicken Crunchy Vegetables And Herb Dressing](../../index/p/poached-chicken-crunchy-vegetables-and-herb-dressing.json)
 * [Turkey Shawarma With Crunchy Vegetables](../../index/t/turkey-shawarma-with-crunchy-vegetables.json)
 * [Crunchy Avocado Salad Recipe](../../index/c/crunchy-avocado-salad-recipe.json)
 * [Crunchy Noodle Salad Recipe](../../index/c/crunchy-noodle-salad-recipe.json)
 * [Crunchy Oven Baked Chicken Toes Recipe](../../index/c/crunchy-oven-baked-chicken-toes-recipe.json)
 * [Sunnys Crunchy Peanut Butter Smore Bites Recipe](../../index/s/sunnys-crunchy-peanut-butter-smore-bites-recipe.json)
 * [Amazing Crunchy Tofu Salad](../../index/a/amazing-crunchy-tofu-salad.json)
 * [Carmels Crunchy Pea Salad](../../index/c/carmels-crunchy-pea-salad.json)
 * [Cinnamon Graham Crunchy Cookies](../../index/c/cinnamon-graham-crunchy-cookies.json)
 * [Covered Crunchy Potato Chips](../../index/c/covered-crunchy-potato-chips.json)
 * [Creamy And Crunchy Potato Soup](../../index/c/creamy-and-crunchy-potato-soup.json)
 * [Creamy And Crunchy Spinach Over Chick](../../index/c/creamy-and-crunchy-spinach-over-chick.json)
 * [Creamy And Crunchy Tuna Salad Supreme](../../index/c/creamy-and-crunchy-tuna-salad-supreme.json)
 * [Crispy Crunchy Chicken Strips](../../index/c/crispy-crunchy-chicken-strips.json)
 * [Crunchy Almond Biscotti](../../index/c/crunchy-almond-biscotti.json)
 * [Crunchy And Delicious Granola](../../index/c/crunchy-and-delicious-granola.json)
 * [Crunchy Baked French Toast](../../index/c/crunchy-baked-french-toast.json)
 * [Crunchy Berry Parfait](../../index/c/crunchy-berry-parfait.json)
 * [Crunchy Bok Choy Salad](../../index/c/crunchy-bok-choy-salad.json)
 * [Crunchy Brunch Wrap](../../index/c/crunchy-brunch-wrap.json)
 * [Crunchy Caramel Tart](../../index/c/crunchy-caramel-tart.json)
 * [Crunchy Carrot Ball](../../index/c/crunchy-carrot-ball.json)
 * [Crunchy Catfish Nuggets](../../index/c/crunchy-catfish-nuggets.json)
 * [Crunchy Cauliflower And Tomato Salad](../../index/c/crunchy-cauliflower-and-tomato-salad.json)
 * [Crunchy Cheese Sandwiches](../../index/c/crunchy-cheese-sandwiches.json)
 * [Crunchy Cheesy Fish And Spinach Casse](../../index/c/crunchy-cheesy-fish-and-spinach-casse.json)
 * [Crunchy Chicken Casserole](../../index/c/crunchy-chicken-casserole.json)
 * [Crunchy Chicken Fingers](../../index/c/crunchy-chicken-fingers.json)
 * [Crunchy Chicken Salad](../../index/c/crunchy-chicken-salad.json)
 * [Crunchy Chip Cookies](../../index/c/crunchy-chip-cookies.json)
 * [Crunchy Chocolate Bars](../../index/c/crunchy-chocolate-bars.json)
 * [Crunchy Chocolate Chip Peanut Butter](../../index/c/crunchy-chocolate-chip-peanut-butter.json)
 * [Crunchy Chocolate Chippers](../../index/c/crunchy-chocolate-chippers.json)
 * [Crunchy Coconut And Lime Burgers](../../index/c/crunchy-coconut-and-lime-burgers.json)
 * [Crunchy Corn Chip Tacos](../../index/c/crunchy-corn-chip-tacos.json)
 * [Crunchy Curry Tuna Sandwich](../../index/c/crunchy-curry-tuna-sandwich.json)
 * [Crunchy Date Rounds](../../index/c/crunchy-date-rounds.json)
 * [Crunchy Egg Salad](../../index/c/crunchy-egg-salad.json)
 * [Crunchy Eggplant Parmesan](../../index/c/crunchy-eggplant-parmesan.json)
 * [Crunchy Fish Tacos](../../index/c/crunchy-fish-tacos.json)
 * [Crunchy Freeze](../../index/c/crunchy-freeze.json)
 * [Crunchy French Onion Chicken](../../index/c/crunchy-french-onion-chicken.json)
 * [Crunchy French Toast Sticks](../../index/c/crunchy-french-toast-sticks.json)
 * [Crunchy Fried Shrimp](../../index/c/crunchy-fried-shrimp.json)
 * [Crunchy Fun Drops](../../index/c/crunchy-fun-drops.json)
 * [Crunchy Granola Breakfast Cereal](../../index/c/crunchy-granola-breakfast-cereal.json)
 * [Crunchy Green Bean Casserole](../../index/c/crunchy-green-bean-casserole.json)
 * [Crunchy Honey Wheat Bread](../../index/c/crunchy-honey-wheat-bread.json)
 * [Crunchy Lemon Quinoa And Asparagus Bo](../../index/c/crunchy-lemon-quinoa-and-asparagus-bo.json)
 * [Crunchy Lettuce](../../index/c/crunchy-lettuce.json)
 * [Crunchy Mung Bean Snack](../../index/c/crunchy-mung-bean-snack.json)
 * [Crunchy No Fry Chicken](../../index/c/crunchy-no-fry-chicken.json)
 * [Crunchy Nutty Granola](../../index/c/crunchy-nutty-granola.json)
 * [Crunchy Oven Fried Tilapia](../../index/c/crunchy-oven-fried-tilapia.json)
 * [Crunchy Pea And Water Chestnut Salad](../../index/c/crunchy-pea-and-water-chestnut-salad.json)
 * [Crunchy Pea Salad With Bacon](../../index/c/crunchy-pea-salad-with-bacon.json)
 * [Crunchy Peanut Butter Chocolate Coc](../../index/c/crunchy-peanut-butter-chocolate-coc.json)
 * [Crunchy Peanut Butter Swirl Brownies](../../index/c/crunchy-peanut-butter-swirl-brownies.json)
 * [Crunchy Pepperoni Sticks](../../index/c/crunchy-pepperoni-sticks.json)
 * [Crunchy Pineapple Smoothie](../../index/c/crunchy-pineapple-smoothie.json)
 * [Crunchy Pumpkin Pie Granola](../../index/c/crunchy-pumpkin-pie-granola.json)
 * [Crunchy Ramen Coleslaw](../../index/c/crunchy-ramen-coleslaw.json)
 * [Crunchy Ranch Chicken Tenders](../../index/c/crunchy-ranch-chicken-tenders.json)
 * [Crunchy Romaine Salad](../../index/c/crunchy-romaine-salad.json)
 * [Crunchy Romaine Toss](../../index/c/crunchy-romaine-toss.json)
 * [Crunchy Sesame Lemon Asparagus](../../index/c/crunchy-sesame-lemon-asparagus.json)
 * [Crunchy Spiced Chickpeas](../../index/c/crunchy-spiced-chickpeas.json)
 * [Crunchy Spinach Dip With Hearts Of Pa](../../index/c/crunchy-spinach-dip-with-hearts-of-pa.json)
 * [Crunchy Turkey Salad](../../index/c/crunchy-turkey-salad.json)
 * [Crunchy Walnut Coated Chicken Strips](../../index/c/crunchy-walnut-coated-chicken-strips.json)
 * [Crunchy Walnut Crusted Salmon Filets](../../index/c/crunchy-walnut-crusted-salmon-filets.json)
 * [Crunchy Wasabi Cheese Balls](../../index/c/crunchy-wasabi-cheese-balls.json)
 * [Crunchy Wheat And Rye](../../index/c/crunchy-wheat-and-rye.json)
 * [Easy Crunchy Berry Parfaits](../../index/e/easy-crunchy-berry-parfaits.json)
 * [Easy Crunchy Hot Dogs](../../index/e/easy-crunchy-hot-dogs.json)
 * [Fiery Fish Tacos With Crunchy Corn Sa](../../index/f/fiery-fish-tacos-with-crunchy-corn-sa.json)
 * [Ghirardelli Crispy Crunchy Chocolate](../../index/g/ghirardelli-crispy-crunchy-chocolate.json)
 * [Ham And Cheese Skewers With Crunchy M](../../index/h/ham-and-cheese-skewers-with-crunchy-m.json)
 * [Ham Sandwich With Crunchy Maille C](../../index/h/ham-sandwich-with-crunchy-maille-c.json)
 * [Julies Crunchy Potato Salad](../../index/j/julies-crunchy-potato-salad.json)
 * [Loris Crunchy Peanut Butter Fudge Sa](../../index/l/loris-crunchy-peanut-butter-fudge-sa.json)
 * [Munchy Crunchy Cookies](../../index/m/munchy-crunchy-cookies.json)
 * [Perfectly Crunchy Granola](../../index/p/perfectly-crunchy-granola.json)
 * [Potato Crunchy Tenders](../../index/p/potato-crunchy-tenders.json)
 * [Quinoa And Steel Cut Oats Crunchy Gra](../../index/q/quinoa-and-steel-cut-oats-crunchy-gra.json)
 * [Squash Casserole With Crunchy Pecan T](../../index/s/squash-casserole-with-crunchy-pecan-t.json)
 * [Sweet And Crunchy Pea Salad](../../index/s/sweet-and-crunchy-pea-salad.json)
 * [Sweet And Crunchy Popcorn Snack Mix](../../index/s/sweet-and-crunchy-popcorn-snack-mix.json)
 * [Sweet And Crunchy Salad](../../index/s/sweet-and-crunchy-salad.json)
 * [Toms Crunchy Salad](../../index/t/toms-crunchy-salad.json)
 * [Apricot Pie With Candied Ginger And Crunchy Topping 109813](../../index/a/apricot-pie-with-candied-ginger-and-crunchy-topping-109813.json)
 * [Brown Rice Salad With Crunchy Sprouts And Seeds 395931](../../index/b/brown-rice-salad-with-crunchy-sprouts-and-seeds-395931.json)
 * [Butternut Squash Kale And Crunchy Pepitas Taco 51249020](../../index/b/butternut-squash-kale-and-crunchy-pepitas-taco-51249020.json)
 * [Crunchy Asian Chicken Salad 243126](../../index/c/crunchy-asian-chicken-salad-243126.json)
 * [Crunchy Chicken In Green Sauce 107283](../../index/c/crunchy-chicken-in-green-sauce-107283.json)
 * [Crunchy Chicken Salad 1120](../../index/c/crunchy-chicken-salad-1120.json)
 * [Crunchy Chicken Salad With Shiitake Mushrooms Spicy Sesame Dressing And Peanuts 101251](../../index/c/crunchy-chicken-salad-with-shiitake-mushrooms-spicy-sesame-dressing-and-peanuts-101251.json)
 * [Crunchy Crab Salad 56389510](../../index/c/crunchy-crab-salad-56389510.json)
 * [Crunchy Fried Shrimp With Cayenne Aioli 3075](../../index/c/crunchy-fried-shrimp-with-cayenne-aioli-3075.json)
 * [Crunchy String Bean Salad With Red Onion And Prosciutto 235715](../../index/c/crunchy-string-bean-salad-with-red-onion-and-prosciutto-235715.json)
 * [Crunchy Turnip Apple And Brussels Sprout Slaw 51260300](../../index/c/crunchy-turnip-apple-and-brussels-sprout-slaw-51260300.json)
 * [Crunchy Veg Bowl With Warm Peanut Sauce](../../index/c/crunchy-veg-bowl-with-warm-peanut-sauce.json)
 * [Crunchy Vegetable Rolls With Soy Dipping Sauce 1227](../../index/c/crunchy-vegetable-rolls-with-soy-dipping-sauce-1227.json)
 * [Crunchy Wasabi Crusted Fish With Red Cabbage Slaw 235465](../../index/c/crunchy-wasabi-crusted-fish-with-red-cabbage-slaw-235465.json)
 * [Crunchy Winter Slaw With Asian Pear And Manchego](../../index/c/crunchy-winter-slaw-with-asian-pear-and-manchego.json)
 * [Ginger Yogurt With Berries And Crunchy Caramel 239055](../../index/g/ginger-yogurt-with-berries-and-crunchy-caramel-239055.json)
 * [Grilled Salmon With Crunchy Sweet Mustard Vinaigrette 109542](../../index/g/grilled-salmon-with-crunchy-sweet-mustard-vinaigrette-109542.json)
 * [Marinated Lentils With Crunchy Vegetables](../../index/m/marinated-lentils-with-crunchy-vegetables.json)
 * [Poached Chicken Crunchy Vegetables And Herb Dressing](../../index/p/poached-chicken-crunchy-vegetables-and-herb-dressing.json)
 * [Turkey Shawarma With Crunchy Vegetables](../../index/t/turkey-shawarma-with-crunchy-vegetables.json)


text:
# Nano snake game
> A nano project aimed to learn more about the [p5.js](http://p5js.org) library and game development.

Based on the work of Daniel Shiffman: http://codingtra.in

You can play with it [here](https://rawgit.com/conradoqg/snake/master/index.html).

## Development

    $ git clone git@github.com:conradoqg/study-nano-projects.git
    $ cd study-nano-projects/snake
    $ npm install

### Start task
It serves a realtime http that bundles the sources and do LiveReloading.

    $ npm start

### Dist task
It generates the bundled javascript and write to dist/

    $ npm run dist

repository: conradoqg/snake

repo_name: conradoqg/snake


text:
# Missing `MessageEvent.origin` verification in `postMessage` handlers
If you use cross-origin communication between Window objects and do expect to receive messages from other sites, always verify the sender's identity using the origin and possibly source properties of the recevied \`MessageEvent\`.

Unexpected behaviours, like \`DOM-based XSS\` could occur, if the event handler for incoming data does not check the origin of the data received and handles the data in an unsafe way.

## Recommendation
Always verify the sender's identity of incoming messages.

## Example
In the first example, the \`MessageEvent.data\` is passed to the \`eval\` function withouth checking the origin. This means that any window can send arbitrary messages that will be executed in the window receiving the message

```javascript
function postMessageHandler(event) {
    let origin = event.origin.toLowerCase();

    console.log(origin)
    // BAD: the origin property is not checked
    eval(event.data);
}

window.addEventListener('message', postMessageHandler, false);

```
In the second example, the \`MessageEvent.origin\` is verified with an unsecure check. For example, using \`event.origin.indexOf('www.example.com') > -1\` can be bypassed because the string \`www.example.com\` could appear anywhere in \`event.origin\` (i.e. \`www.example.com.mydomain.com\`)

```javascript
function postMessageHandler(event) {
    let origin = event.origin.toLowerCase();

    let host = window.location.host;

    // BAD
    if (origin.indexOf(host) === -1)
        return;

    eval(event.data);
}

window.addEventListener('message', postMessageHandler, false);
```
In the third example, the \`MessageEvent.origin\` is properly checked against a trusted origin.

```javascript
function postMessageHandler(event) {
    console.log(event.origin)
    // GOOD: the origin property is checked
    if (event.origin === 'www.example.com') {
        // do something
    }
}

window.addEventListener('message', postMessageHandler, false);
```

## References
* [CWE-020: Improper Input Validation](https://cwe.mitre.org/data/definitions/20.html)
* [Window.postMessage()](https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
* [Web-message manipulation](https://portswigger.net/web-security/dom-based/web-message-manipulation)
* [The pitfalls of postMessage](https://labs.detectify.com/2016/12/08/the-pitfalls-of-postmessage/)
* Common Weakness Enumeration: [CWE-20](https://cwe.mitre.org/data/definitions/20.html).


text:
# aXe Angular1 Test Demo

This project is based on the [angular-seed](https://github.com/angular/angular-seed) application skeleton for a typical [AngularJS](http://angularjs.org/) web app.

You can use it to quickly bootstrap your angular webapp projects and dev environment for these projects with accessibility testing built-in.

The seed contains a sample AngularJS application and is preconfigured to install the Angular framework and a bunch of development and testing tools for instant web development gratification, including the [axe-core](https://github.com/dequelabs/axe-core) accessibility testing library and [axe-webdriverjs](https://github.com/dequelabs/axe-webdriverjs) helper.

## Getting Started

To get you started you can simply clone the axe-angular1-demo repository and install the dependencies:

### Prerequisites

You need git to clone the axe-angular1-demo repository. You can get git from
[http://git-scm.com/](http://git-scm.com/).

We also use a number of node.js tools to initialize and test axe-angular1-demo. You must have node.js and
its package manager (npm) installed.  You can get them from [http://nodejs.org/](http://nodejs.org/).

### Clone axe-angular1-demo

Clone the axe-angular1-demo repository using [git][git]:

```
git clone https://github.com/marcysutton/axe-angular1-demo.git
cd axe-angular1-demo
```

If you just want to start a new project without the angular-seed commit history then you can do:

```bash
git clone --depth=1 https://github.com/marcysutton/axe-angular1-demo.git <your-project-name>
```

The `depth=1` tells git to only pull down one commit worth of historical data.

### Install Dependencies

We have two kinds of dependencies in this project: tools and angular framework code.  The tools help
us manage and test the application.

* We get the tools we depend upon via `npm`, the [node package manager][npm].
* We get the angular code via `bower`, a [client-side code package manager][bower].

We have preconfigured `npm` to automatically run `bower` so we can simply do:

```
npm install
```

Behind the scenes this will also call `bower install`.  You should find that you have two new
folders in your project.

* `node_modules` - contains the npm packages for the tools we need
* `app/bower_components` - contains the angular framework files

*Note that the `bower_components` folder would normally be installed in the root folder but
axe-angular1-demo changes this location through the `.bowerrc` file.  Putting it in the app folder makes
it easier to serve the files by a webserver.*

### Run the Application

We have preconfigured the project with a simple development web server.  The simplest way to start
this server is:

```
npm start
```

Now browse to the app at `http://localhost:8000/index.html`.

## Directory Layout

```
app/                    --> all of the source files for the application
  app.css               --> default stylesheet
  components/           --> all app specific modules
    version/              --> version related components
      version.js                 --> version module declaration and basic "version" value service
      version_test.js            --> "version" value service tests
      version-directive.js       --> custom directive that returns the current app version
      version-directive_test.js  --> version directive tests
      interpolate-filter.js      --> custom interpolation filter
      interpolate-filter_test.js --> interpolate filter tests
  view1/                --> the view1 view template and logic
    view1.html            --> the partial template
    view1.js              --> the controller logic
    view1_test.js         --> tests of the controller
  view2/                --> the view2 view template and logic
    view2.html            --> the partial template
    view2.js              --> the controller logic
    view2_test.js         --> tests of the controller
  app.js                --> main application module
  index.html            --> app layout file (the main html template file of the app)
  index-async.html      --> just like index.html, but loads js files asynchronously
karma.conf.js         --> config file for running unit tests with Karma
e2e-tests/            --> end-to-end tests
  protractor-conf.js    --> Protractor config file
  scenarios.js          --> end-to-end scenarios to be run by Protractor
```

## Testing

There are two kinds of tests in the axe-angular1-demo application: Unit tests and end-to-end tests. The aXe example can be found in the [end-to-end test](./e2e-tests/scenarios.js).

### Running Unit Tests

The axe-angular1-demo app comes preconfigured with unit tests. These are written in
[Jasmine][jasmine], which we run with the [Karma Test Runner][karma]. We provide a Karma
configuration file to run them.

* the configuration is found at `karma.conf.js`
* the unit tests are found next to the code they are testing and are named as `..._test.js`.

The easiest way to run the unit tests is to use the supplied npm script:

```
npm test
```

This script will start the Karma test runner to execute the unit tests. Moreover, Karma will sit and
watch the source and test files for changes and then re-run the tests whenever any of them change.
This is the recommended strategy; if your unit tests are being run every time you save a file then
you receive instant feedback on any changes that break the expected code functionality.

You can also ask Karma to do a single run of the tests and then exit.  This is useful if you want to
check that a particular version of the code is operating as expected.  The project contains a
predefined script to do this:

```
npm run test-single-run
```

### End to end testing

The axe-angular1-demo app comes with end-to-end tests, again written in [Jasmine][jasmine]. These tests
are run with the [Protractor][protractor] End-to-End test runner.  It uses native events and has
special features for Angular applications.

* the configuration is found at `e2e-tests/protractor-conf.js`
* the end-to-end tests are found in `e2e-tests/scenarios.js`

Protractor simulates interaction with our web app and verifies that the application responds
correctly. Therefore, our web server needs to be serving up the application, so that Protractor
can interact with it.

```
npm start
```

In addition, since Protractor is built upon WebDriver we need to install this.  The axe-angular1-demo
project comes with a predefined script to do this:

```
npm run update-webdriver
```

This will download and install the latest version of the stand-alone WebDriver tool.

Once you have ensured that the development web server hosting our application is up and running
and WebDriver is updated, you can run the end-to-end tests using the supplied npm script:

```
npm run protractor
```

This script will execute the end-to-end tests against the application being hosted on the
development server.

**Note:**
Under the hood, Protractor uses the [Selenium Stadalone Server][selenium], which in turn requires
the [Java Development Kit (JDK)][jdk] to be installed on your local machine. Check this by running
`java -version` from the command line.

If JDK is not already installed, you can download it [here][jdk-download].

### Using axe-core

To make the most of accessibility tests, it might be helpful to reference the results object from the aXe API: https://github.com/dequelabs/axe-core/blob/master/doc/API.md#results-object

The integration test set up in this application returns the aXe results with two violations. To fix them, you would review the aXe results output, make changes that address those violations, and get the test passing.

### Running the App during Development

The angular-seed project comes preconfigured with a local development webserver.  It is a node.js
tool called [http-server][http-server].  You can start this webserver with `npm start` but you may choose to
install the tool globally:

```
sudo npm install -g http-server
```

Then you can start your own development web server to serve static files from a folder by
running:

```
http-server -a localhost -p 8000
```

Alternatively, you can choose to configure your own webserver, such as apache or nginx. Just
configure your server to serve the files under the `app/` directory.


text:
# Outliner-ObjC

I want to learn Apple Development, and don't know what to create. Then, John Gruber's tweet inspired me:

> [@gruber - 2:55 PM - Jun 28, 2020][gruber]
>
> I would kill for a good simple outliner that synced across iOS and Mac, and had a great \*simple\* UI on all three platforms.

## Why Objective-C?

Some reasons:
1. I want to learn the "new" and "old" way of Apple Development. I know that many SwiftUI's features are great. I just wanted to _learn bottom-up_, while if I use Swift first, I think that is a _top-down_ approach.
2. I develop with React Native too. Most of [React Native modules][rnmodules] use Objective-C for their iOS Native Module.

---

Let's learn! :D

[LICENSE](./LICENSE.md)

[gruber]: https://twitter.com/gruber/status/1277329886080905219?s=20
[rnmodules]: https://github.com/topics/react-native

repository: ybbond/Outliner-ObjC

repo_name: ybbond/Outliner-ObjC


text:
weaveJ
:	Programatic aspect weaver for Java.

To use weaveJ in Java:

1. Download `java-agent.jar` from `source-code\lib` folder, and add it to your system classpath.

2. Employ the API in your code to use pointcuts, see [documentation](http://htmlpreview.github.com/?https://github.com/ComputationalReflection/weaveJ/blob/master/documentation/index.html).

3. Specify the following arguments to the VM :
  * The java agent: `-javaagent:[path_to_the_jar]/java-agent.jar`.
  * The classes or packages to be instrumented: `-Dformatting=[packages_or_classes_you_want_instrument]`  (`';'` as separator)

Notice:
weaveJ works with Java 8. The `setField` joinpoint is only supported for non-final fields.

Directories:

 * Benchmarks: Code and binaries of all the programs used in the evaluation.
   * Micro Benchmark
   * Real Applications
 * binaries: Binaries and scripts to use weaveJ and run the example.
 * documentation: Files containing the documentation of weaveJ (javadoc).
 * example: Binaries, scripts and code of the example.
 * source-code: Code and binaries of weaveJ.

repository: ComputationalReflection/weaveJ

repo_name: ComputationalReflection/weaveJ


text:
---
layout: page
permalink: /stretchexercise
---

# Stretch Exercise: Improving the accuracy of your model
**Unlike MNIST, the cats and dogs dataset has much more variation and is therefore harder to for the model to make a prediction. Use one or a combination of techniques described in the following sections to maximise your accuracy. What's the highest accuracy that you can achieve?**

## Customising your network layers

It is possible to customise your network by editing the network definition file. The network is composed of a series of layers connected to each other. For example, here's a `Convolution` and `Pooling` layer:

```
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 50
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
```
The connection between layers are indicated by the `top` and `bottom` parameters. The `top` parameter is the output of the layer, and `bottom` parameter is the input. From the above example, it can be seen that the `Convolution` layer named `conv2` has the input `pool1` and outputs `conv2`.

To edit your model, on model creation page after chossing `LeNet`, click `Customize`.

This will take you to the network editor. You can click on the `Visualize` button to see the visulisation of the network graph.

The network visulisation is shown below:

Once you've started training your customised model, it is possible to clone your model in order to keep the settings and changes in the model definition file so that you can make further changes. Just click on the `Clone Job` button of the model that's training or has already finished training.

## Use more filters in your convolution layers

Adding more filters in your `Convolution` layers allows it to detect more features which can allow the model to make more complex predictions. You can do this by increasing the `num_output` parameter. In this case, for `conv2` layer, the `50` filters has changed to `100`.

```
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 100 #<-- Change the number of feature maps here
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
```

## Add activation functions

Activation functions such as `ReLU` adds non-linearity to our network and allows it to express more complexity. Try adding the `ReLU` layer after every `Pooling` layer in your network:

```
layer {
  name: "relu"
  type: "ReLU"
  bottom: "pool1"
  top: "pool1"
}
```

## Add more layers

Deeper networks also allows for more complex predictions. Try adding more `Convolution` layers to your model.
* Don't forget to add a `Pooling` and optionally `ReLU` layer after your new `Convolution` layer.
* Don't forget to change the `bottom` prameter of the `ip1` layer.

## More training

As the more feature maps and layers are added, the model will have to be trained for longer. This can be done by changing the number of training `epochs` while creating your model:

## Reduce overfitting

It is also important to observe the `validation loss` (green line) along with the `accuracy` (orange line). The graph below shows an example of `training loss` (blue line) diverging from the `validation loss`:

This is a clear sign of `overfitting` and it means that your model will more likely to misclassify images that it wasn't trained on. In order to reduce overfitting, a technique called `Dropout` where a certain percentage of random neurons are excluded from training. The neurons selected to be excluded change at every `forward propagation` step.

The technique can be used in your model by introducing the `Dropout` layer and any layer below it will experience the dropout. The `dropout_ratio` determines the percentage of excluded neurons in the layer e.g. `0.6` means 60% of random neurons will not be used.  Try putting `Dropout` layers after the `Pooling` or `ReLU` layers.

```
layer {
  name: "drop1"
  type: "Dropout"
  top: "pool1"
  bottom: "pool1"
  dropout_param{
    dropout_ratio: 0.5
  }

}
```