text:
# JSON and Swift

## Minute-by-Minute

| **Elapsed** | **Time**  | **Activity**              |
| ----------- | --------- | ------------------------- |
| 0:00        | 0:05      | Objectives                |
| 0:05        | 0:05      | Initial Exercise          |
| 0:10        | 0:10      | JSON review               |
| 0:20        | 0:10      | In Class Activity I       |
| 0:30        | 0:10      | Reading data from a file  |
| 0:40        | 0:15      | Encoding and Decoding     |
| 0:55        | 0:10      | BREAK                     |
| 1:05        | 0:05      | In Class Activity II      |
| 1:10        | 0:10      | In Class Activity III     |
| 1:20        | 0:25      | Challenge                 |
| 1:45        | 0:05      | Wrap Up                   |
| TOTAL       | 1:50      |                           |

# Why you should know this

Every client application receives information from a common server. It doesn't matter how the implementation of handling the data happens, it has a standard format that can be recognized by every client app. Years ago XML(1996) was widely used for this purpose. But recent trends point that JSON(2002) has become the most popular data-interchange format and that it will continue to dominate the web.

## Class Learning Objectives/Competencies (5 min)

1. Identify JSON structures and create your own
1. Reading data from a file
1. Deserialize a JSON file
1. Turning data into objects using the Codable protocol

## Initial Exercise (5 min)

### Swift 5 Release Notes
- Raw strings
- Handling future enum cases
- Flattening nested optionals
- Checking for integer multiples
- compactMapValues()

[What's new in Swift 5?](https://www.hackingwithswift.com/articles/126/whats-new-in-swift-5-0)

## JSON quick intro/review (10 min)

JSON stands for JavaScript Object Notation. It is a lightweight data-interchange format that allows us to read and write data that machines can also generate and parse easily.

Two main structures:
- **Object:** Collection of name/value pairs.
- **Array:** Ordered list of values.

An object begins with a left brace and ends with a right brace. Each name is followed by colon and the pairs are separated by commas.

![object](assets/object.gif)

An array begins with a left bracket and ends with a right bracket. Values are separated by commas.

![array](assets/array.gif)

Values can be strings in double quotes, numbers, true, false, null, objects or arrays. Also, structures can be nested.

![value](assets/value.gif)

A string is a sequence of 0 or more characters in double quotes.

![string](assets/string.gif)

Numbers are regular numbers like any language.
![number](assets/number.gif)

In the end, JSON is a way of communicating using very specific rules that every entity involved can agree on.

### Example
```
{
   "festivals":[
      {
         "name":"Austin City Limits"
      },
      {
         "name":"Riot Fest"
      },
      {
         "name":"Coachella"
      }
   ]
}
```

## In Class Activity I (10 min)

Without using a format validator, create your own JSON file to represent a list of festivals. Include two festival entries. They do not have to be real.

Consider you have this structs as reference:

```Swift
struct Participant{
    let name: String
    let id: String
}

struct City{
    let name: String
    let id: String
}

enum FestivalType{
    case music
    case food
    case cinema
}

struct Festival{
    let date: Date
    let name: String
    let city: City
    let lineup: [Participant]
    let type: FestivalType
}
```

After your'e done writing the structure use the [JSON validator](https://jsonformatter.curiousconcept.com) to make sure your format is correct. If not, do the necessary adjustments.

## Reading data from a file (10 min)

Data can sometimes be stored in text files inside our app bundle. To retrieve them there is a particular way of doing it. We can either look for a path or a URL. Today we're going to use the path. The call is the following:

```Swift
func path(forResource name: String?, ofType extension: String?) -> String?
```
`name` is the name of the resource file
`extension` is the filename extension of the files to locate

The return value will be the full pathname for the resource or `nil` if the file could not be located.

```Swift
let path = Bundle.main.path(forResource: "fileName", ofType: ".json")
if let path = path {
    let url = URL(fileURLWithPath: path)
    print(url)
}
```

## Getting the data from JSON

```Swift
let contents = try? Data(contentsOf: url, options: .alwaysMapped)
let jsonResult = try JSONSerialization.jsonObject(with: data, options: .allowFragments) as ...
// print(jsonResult)
```
We can now see all of the data we retrieved from the file.
If we were to create an object for each entry and save them in an array, before Swift 4, we would need to retrieve properties one by one. This is why external libraries like [SwiftyJSON](https://github.com/SwiftyJSON/SwiftyJSON) became popular, they made the job less complicated.

## Encoding and Decoding (15 min)

“Serializing a program’s internal data structures into some kind of data interchange format and vice versa is one of the most common programming tasks. Swift calls these operations encoding and decoding. One of the headline features of Swift 4 is a standardized design for encoding and decoding data that all custom types can opt into.”

Excerpt From: Chris Eidhof. “Advanced Swift.”

### The Codable system has 3 main goals:

#### Universality
Should work with structs, enums and classes.

#### Type safety
JSON is often weakly typed, but we're working with Swift so our code works with strongly typed data structures.

#### Reducing boilerplate
Writing less code, while the compiler generates what we need automatically.

Types conform to the **Encodable** and/or **Decodable** protocols to state their ability to encode itself and creating and instance from serialized data.

Most types will adapt both, so the standard library came up with a typealias for both: **Codable** protocol.

```
public typealias Codable = Encodable & Decodable
```

When we have a value that conforms to the Codable protocol we can:
- Create an *encoder* to serialize the value into JSON. (JSONEncoder is built-in in Swift)
- Create a *decoder* that takes the serialized data and turns it into an instance of the original type. . (JSONDecoder is built-in in Swift)

Note: Error handling is important during decoding. Many things can go wrong (missing fields, type mismatches, corrupted data, etc.)

```Swift
struct Location: Codable {
    var name: String
    var distance: Double
}

struct Park: Codable{
    var name: String
    var location: Location
}

let parks:[Park] = [Park(name: "Universal Studios", location: Location(name: "Florida", distance: 100.0)), Park(name: "Disneyland", location: Location(name: "California", distance: 200.0)), Park(name: "Cedar Point", location: Location(name: "Ohio", distance: 300.0))
]

//Encoding

do {
    let encoder = JSONEncoder()
    let jsonData = try encoder.encode(parks)
    let jsonString = String(decoding: jsonData, as: UTF8.self)
    print(jsonString)
} catch {
    print(error.localizedDescription)
}

//Decoding

do {
    let encoder = JSONEncoder()
    let jsonData = try encoder.encode(parks)

    let decoder = JSONDecoder()
    let decoded = try decoder.decode([Park].self, from: jsonData)

    for park in decoded{
        print(park.name)
    }

} catch {
    print(error.localizedDescription)
}
```
### Encoding
1. Initiate the encoding process
1. Encoder calls `encode(to:Encoder)` method passing itself as the argument.

   Ex. `let jsonData = try encoder.encode(parks)`
 which in result calls `parks.encode(to:self)``
1. In order to encode itself in the correct format, it will use *encoding containers*
The Encoder protocol returns these containers.
 ##### Types of containers
 - Keyed container: Encodes key value pairs. Will turn into JSON objects.
 - Unkeyed container: Encodes multiple values sequentially, omitting keys. Will turn into JSON arrays.
 - Single value container: Encodes a single value. Will turn into numbers, strings, booleans or null.
1. The compiler generates an enum named `CodingKeys`. It contains one case for every stored property of the struct.

Strongly typed keys lead to safety but they must eventually be able to turn into strings/integers. The CodingKey protocol handles this task.

### Decoding
1. call `try decoder.decode([Park].self, from: jsonData)`
1. Decoder creates an instance of the type passed in. What is it in this case?
1. Decoder manages a tree of *decoding continers* where each value travels down recursively in a container hierarchy while initializing values.

If at some point in the hierarchy, an error occurs, tje entire process fails.

### Coding Keys
We can control how a type encodes itself by writing a custom CodingKeys enum.

- Rename fields by giving string values
- Skip fields by omitting keys

## In Class Activity II (5 min)

How will the same `Park` struct look like if we're using custom Coding Keys and we're expecting the key of the `name` value to come with the label `state` in the JSON object?

Write the struct.

How will the same `Park` struct look like if we're using custom Coding Keys and want to ignore the value for `distance`?
Write the struct.

## In Class Activity III (10 min)
Go back to the film locations example and musing the Codable protocol make sure to save this properties only:
```Swift
var firstActor: String
var location: String
var productionCompany: String
var releaseYear: String
var title: String
```

## Challenge (20 min)
Take the JSON file [Festivals](assets/festivals.json) and create a project where you decode the contents to display The name of the Festival, the date and the number of artists in the lineup in a TableViewCell.

## Wrap Up (5 min)
**Q:** How does the encoding process work?

**Q:** What does the compiler handle when using Codable?

**Q:** How does the decoding process work?

- Start Product Hunt tutorial

## Additional Resources
1. [Slides](https://docs.google.com/presentation/d/1hrtAoznelyYGna7CwhStlU_46rH4JLiMV5DNTowEHYc/edit?usp=sharing)
1. [XML vs JSON](https://www.cs.tufts.edu/comp/150IDS/final_papers/tstras01.1/FinalReport/FinalReport.html)
1. [JSON documentation](http://www.json.org)
1. [JSON format validator](https://jsonformatter.curiousconcept.com)
1. [Film locations](https://data.sfgov.org/Culture-and-Recreation/Film-Locations-in-San-Francisco/yitu-d5am)
1. [Codable sheet](https://www.hackingwithswift.com/articles/119/codable-cheat-sheet)
1. (Book) “Advanced Swift.” by Chris Eidhof


text:
# bone-oauth2
[![Latest Stable Version](https://poser.pugx.org/delboy1978uk/bone-oauth2/v/stable)](https://packagist.org/packages/delboy1978uk/bone-oauth2) [![Total Downloads](https://poser.pugx.org/delboy1978uk/bone-oauth2/downloads)](https://packagist.org/packages/delboy1978uk/bone-oauth2) [![License](https://poser.pugx.org/delboy1978uk/bone-oauth2/license)](https://packagist.org/packages/delboy1978uk/bone-oauth2)<br />
OAuth2 Authorization and Resource Server functionality for Bone MVC Framework

## installation
Install via composer from the root of your Bone Framework project
```
composer require delboy1978uk/bone-oauth2
```
## configuration
Simply add the Package to Bone's packages config
```php
<?php

// use statements here
use Bone\OAuth2\BoneOAuth2Package;
use Bone\User\BoneUserPackage;

return [
    'packages' => [
        // packages here (order is important)...,
        BoneUserPackage::class,
        BoneOAuth2Package::class,
    ],
    // ...
];
```
#### generate a public and private key
Firstly go into the `data/keys` directory.
```
cd data/keys
```
Use openssl to generate a private key, then extract the public key from the private key:
```
openssl genrsa -out private.key 2048
openssl rsa -in private.key -pubout -out public.key
chmod 660 public.key
chmod 660 private.key
```
If you want to provide a passphrase for your private key run these commands instead:
```
openssl genrsa -passout pass:_passphrase_ -out private.key 2048
openssl rsa -in private.key -passin pass:_passphrase_ -pubout -out public.key
chmod 660 public.key
chmod 660 private.key
```
If a passphrase has been used to generate private key it must be provided to the authorization server.

The public key should be distributed to any services (for example resource servers) that validate access tokens.
#### generate an encryption key
Go back to the project root.
```
cd ../..
```
Run this command and add to your config.
```
vendor/bin/generate-defuse-key
```
Run database migrations to generate the tables
```
vendor/bin/bone migrant:diff
vendor/bin/bone migrant:migrate
vendor/bin/bone migrant:generate-proxies
````
### required config values
Keys can be stored out of the config array and fetched as an environment variable for better security, but these are the config settings you need.
```php
<?php

return [
    'oauth2' => [
        'clientCredentialsTokenTTL' => 'PT1H', // 1hour
        'authCodeTTL' => 'PT1M', // 1 minute
        'accessTokenTTL' => 'PT5M', // 5 minutes
        'refreshTokenTTL' => 'P1M', // 1 month
        'privateKeyPath' => '/path/to/private.key',
        'publicKeyPath' => '/path/to/private.key',
        'encryptionKey' => 'generatedKeyString',
    ],
];
```
## usage
#### server side
You can create a client using the `vendor/bin/bone` command. You can also create scopes, and grant scopes to clients.

To lock down an endpoint to require an access token, simply add the `ResourceServerMiddleware` to the route or route
group in your Bone Framework Package class
```php
$router->map('GET', '/ping', [ExampleController::class, 'pingAction'])->middleware($c->get(ResourceServerMiddleware::class));
```
In your controller, you will have access to the user, which is now an instance of `OAuthUser`. You can also get the
scopes granted for the request.
```php
    /**
     * @param $request
     * @param array $args
     * @return ResponseInterface
     * @throws \Exception
     */
    public function someAction(ServerRequestInterface $request, array $args) : ResponseInterface
    {
        /** @var \Bone\OAuth2\Entity\OAuthUser $user */
        $user = $request->getAttribute('user');

        if (!in_array('email', $request->getAttribute('oauth_scopes'))) {
            throw new Exception('How dare you!', 403);
        }

        return new JsonResponse(['random' => 'data']);
    }
```
#### client side
Clients connect using the standard OAuth2 flow described in RFC6749, the two endpoints in your Bone App are
- /oauth2/authorize
- /oauth2/token
#### site users
Logged in users now have an additional end point which they can go to, `/user/api-keys`, where they can get a new API key, or delete existing ones.
#### console
You also have access to more `bone` command options:
```
bone client:create
bone client:create
bone scope:create
bone scope:list
```

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# README Generator
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

# Table of Contents
1. [Description](#description)
2. [Installation Instructions](#installation-instructions)
3. [Usage Information](#usage-information)
4. [License](#license)
5. [Contribution Guidelines](#contribution-guidelines)
6. [Test Instrucions](#test-instructions)
7. [Questions](#questions)

## Description
README Generator uses the readme template and inquirer to create a README for your project.

## Installation Instructions
To install ensure that in the integrated terminal you have run 'npm init -y' to initialise the JSON package, then run 'npm install inquirer'. If you have previously run the above commands, run 'npm install' install to ensure the dependencies are up to date.
![Alt text](/screenshots/init.JPG?raw=true "init")
![Alt text](/screenshots/install.JPG?raw=true "install")

## Usage Information
To use the application in the integrated terminal run 'node .\index.js' and follow the command prompts.
![Alt text](/screenshots/run.JPG?raw=true "run")

## License
This project is licensed under the MIT license.

## Contribution Guidelines
Please refer to each project's style and contribution guidelines for submitting patches and additions. In general, we follow the "fork-and-pull" Git workflow.
Fork the repo on GitHub.
Clone the project to your own machine.
Commit changes to your own branch.
Push your work back up to your fork. Submit a Pull request so that we can review your changes.
NOTE: Be sure to merge the latest from "upstream" before making a pull request!

## Test Instructions
Test the project at your own will, please be aware that the project provides a template README for further formatting and use of screenshots please add manually.

## Questions
Github:
https://github.com/willygodwin
Email:
<EMAIL_ADDRESS>
## Link to Video
https://drive.google.com/file/d/1v-1pc8Jcq4e0fNXzL4tihdCaT57YYZ6p/view

repository: willygodwin/README-Generator

repo_name: willygodwin/README-Generator


text:
# Calonectria brongniartii (P. Crouan & H. Crouan) Sacc., 1883 SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Syll. fung. (Abellini) 2: 549 (1883)

#### Original name
Nectria brongniartii P. Crouan & H. Crouan, 1867

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Ascomycota/Sordariomycetes/Hypocreales/Bionectriaceae/Bryocentria/Bryocentria brongniartii/ Syn. Calonectria brongniartii/README.md

repo_name: mdoering/backbone


text:
---
layout: blog
title: "COVID Vaccine Information Hotline"

---

[Health Department Release](https://storage.googleapis.com/static.rutherford-nj.com/covid/Health%20Dept%20VaccineHotline.pdf)

**The text below can also be heard at the Health Department’s 24-Hour Hotline at<PHONE_NUMBER> x3154**

This message is designed to give residents some detailed information about the state’s vaccination program regarding COVID-19.

At this time, the state is vacating the following individuals: health care workers, long-term care facility residents and staff, Police, Fire, EMS, individuals at high risk, such as those aged 65 years of age and older and individuals ages 16 to 64 with medical conditions that increase the risk of severe illness from the virus.

These conditions include: Cancer, chronic kidney disease, Down syndrome, heart conditions such as heart failure, coronary artery disease or cardiomyopathies, obesity, sickle cell disease, smoking, type 2 diabetes.

The state plans on setting up six mega vaccination clinics throughout the state, one of which will be at the Meadowlands Race Track in East Rutherford, NJ. It is also anticipated that pharmacies, such as Walgreens, CVS, Shop Rite, etc., will also be vaccinating people. It is not anticipated that our department will be vaccinating residents against COVID-19.

Timing of the progression among the groups noted is dependent on the supply of the vaccine provided to the state and balanced by the demands in each group noted above.

More detailed information is available off the State Health Department COVID-19 information hub web site, which can be accessed at www.covid19.nj.gov .

Residents who have questions not answered by the 24-Hour Hotline Number can
call the Health Department weekdays at<PHONE_NUMBER>.

Thank you.

branch_name: refs/heads/beta

gha_language: HTML

gha_license_id: MIT


text:
sqlish - tutorial
=================

# Getting started

The basic idea of sqlish is to mimic the typical SQL vocabulary
as a serious of function calls. Here's an example of a typical
SQL statement followed by how to render the same thing with
in native JavaScript with sqlish.

```SQL
    SELECT id, name, email FROM email_list;
```

Now let's write that with sqlish running under NodeJS-

```JavaScript
    // Create an sql object
    var sqlish = require("sqlish"),
        sql = new sqlish.Sqlish();

    sql.select(["id", "name", "email"]).from("email_list").toString();
```

That's the basic idea. The nice thing about sqlish is this also
works in the MongoDB shell though you'll need to load it a little
differently. MongoDB's shell doesn't provide a module system like
NodeJS. It will through errors when it encounters undefined
variables. If you look in extras folder you'll see and example
load script for using sqlish.js with Mongo. Here's the
basic idea (I've assumed that sqlish.js is in the same directory
as the load script in this example).

```JavaScript
    // We're going to declare exports just
    // to keep the MongoDB shell happy
    var exports;
    load("./sqlish.js");

    var sql = new sqlish.Sqlish();

    sql.select(["id", "name", "email"]).from("test").toString();
```

Each function produces phrases or clauses of SQL. _select()_ is a verb
and takes an array of column names. If no field names are provided then the
an asterisk is used in its place. _from()_ adds the FROM SQL clause
and accepts a list of table names as arguments.  The finally toString()
job is to return an assemble SQL statement as a string.

# Creating a sqlish object

A sqlish object is created with the new operator after it has been
imported. The constructor can take an object as a parameter. The
attributes supported are dialect name, use_UTC, and eol (e.g. ';')
which will be used by toString() to terminate the SQL statement
generated.

```JavaScript
    var sql = new sqlish.Sqlish("PostgreSQL 9.2");
    sql.use_UTC = true;
    sql.eol = ";\n";
```

# SQL statements and clauses

## Overview

Chaining some sqlish functions together will let you
assemble an SQL statement rendered by toString().
sqlish groups SQL phrases into four general categories
verbs, supporting clauses, expressions and definitions.
The verbs always start the chain. Chaining two verbs
together will cause the second verb to replace the first.
Clauses refine the actions of the verb. They be describe
the object to work on (e.g. _from()_) or filter results
(e.g. _where()_) or effect ordering (e.g. _order()_
or _group()_). Some

## Details

Verb phrases include SQL statements starting with _CREATE_,
_ALTER_, _DROP_, _INSERT_, _UPDATE_, _REPLACE_, _DELETE_,
_SELECT_, and _SET_. These generalally describe an action
you will take on a database, table or environment variable.
Sqlish provides a corresponding set of functions to generate
these phrases including - _createTable()_, _alterTable()_,
_dropTable()_, _insert()_, _update()_, _replace()_, _deleteFrom()_,
_select()_, _set()_ and _union()_.  Where the SQL term would collide
with a JavaScript keyword (e.g. delete, create) a similar
compound word is used (e.g. deleteFrom, createTable). To
keep a modicum of similarity compound words are used for all
actions relating to table, index and view -

Supporting clauses modify the action in some way. Sometimes
this is to filter the effect of the verb, modified or limit
the results or to set the target of the action. They are
typically used after the verb phrase.  Supporting
clauses include _FROM_, _INTO_, _JOIN_, _WHERE_, _ORDER BY_,
_GROUP BY_, _LIMIT_, _OFFSET_, _HAVING_ and _SET_. You will
see these defined later as _from()_, _into()_, _join()_,
_where()_, _order()_, _group()_, _limit()_, _offset()_,
_having()_ and _set()_

Expressions find themselves sprinkled through out SQL
statements.  Much of the power of SQL comes from its expressions.
Expressions implement logic, e.g._email =<EMAIL_ADDRESS>or _title LIKE "%Wild Things%"_.  They also can define, e.g.
_id INTEGER AUTO_INCREMENT PRIMARY KEY_. Expressions in sqlish
usually are  formed by a JavaScript object literals inspired
by those used in MongoDB. From the three previous examples-

```JavaScript
    // email =<EMAIL_ADDRESS>    email_johndoe_expr = {email<EMAIL_ADDRESS>    // (title LIKE "%Wild Things%")
    titles_like_expr = {title: {$like: /Wild Things/}};
    // id INTEGER AUTO_INCREMENT PRIMARY KEY
    def_auto_increment_id = {
        type: "int",
        auto_increment: true,
        primary_key: true
    };
```

If you need to group phrases in SQL to assert a precedence
in evaluation you do so with parenthesis. Unfortunately a simple
wrapper of Parenthesis will not work in JavaScript since that
delineates a functions' parameters and functions can be assigned
inside object literals.  To allow expression composition we
a the _$p_ operator (p is short for parenthesis).

A sqlish's object literal integrating -

```JavaScript
    // generating something like:
    // order_id > 3 AND (cnt = 2 OR name = "fred")
    nestedOrExpression = {$and: [
        {user_id: {$gt: 3}},
        {$p:
            {$or: [
                {cnt: 2},
                {name: "fred"}
             ]}
        }
    ]};
```

Here's the SQL expression rendered -

```SQL
    order_id > 3 AND (cnt = 2 OR name = "fred")
```

Here's putting that statement into action with in a where clause -

```JavaScript
    var sql = new sqlish.Sqlish("MySQL 5.5"),
        qry;

    qry = sql.select()
        .from("mytable")
        .where({$and: [
        {user_id: {$gt: 3}},
        {$p:
            {$or: [
                {cnt: 2},
                {name: "fred"}
             ]}
        }
    ]}).toString();
    console.log(qry);
```

This would output something like -

```SQL
    SELECT * FROM mytable WHERE order_id > 3 (cnt = 2 OR name = "fred");
```

The downside of the Sqlish approach is sometimes you'll type more JavaScript/JSON
than raw the SQL. The upside is that Sqlish is that it will attempt to prevent
SQL injection by apply quoting and escaping to values and restricting the
characters used for SQL function names, table names, column names
and environment variable references.

### IMPORTANT WARNING!

> sqlish is not a replacement for best practices when writing
> software. It remains important to defensively validate input from users,
> web browsers or any data source.

## Verbs

This is a list of the verbs implemented so far as sqlish
functions. More may be added overtime.

### createTable()

This generates a full create statement for the targeted
dialect of SQL. createTable takes two parameters -
a table name and an object who's property names are
the column names you want to create and who's values
are also an object which attributes describe types
and properties of the column

```JavaScript
	sql.createTable("story_book_characters", {
		id: {
			type: "integer",
			auto_increment: true,
			primary_key: true
		},
		name: {
			type: "varchar",
			length: 255,
			not_null: true
		},
		tunnel_id: {
			type: "integer",
			not_null: true,
			default: 100
		},
		start_date: {
			type: "datetime",
            not_null: true
		},
		poem: { type: "text" },
		modified: { type: "timestamp" }
	});
```

### dropTable()

Generates a drop table statement. It takes a single parameter
of the table name.

```JavaScript
    sql.dropTable("story_book_characters");
```

### createIndex()

Generate a index. Parameters expected are index name and
object with attributes describing the index.

```JavaScript
	sql.createIndex("i_character_names", {
		unique: true,
		table: "story_book_characters",
		columns: ["name"]
	});
```

### droptIndex()

Generate a drop index statement. It takes a single parameter
of the index name to drop.

```JavaScript
	sql.dropIndex("i_character_names");
```

### createView()

Generate a create view statement. It takes a two parameters
a view name and a SQL object that will be rendered with
toString().

```JavaScript
    sql_subquery = sqlish.Sqlish();
    sql_subquery.select(["email", "name"],
    	{unique: true}).from("my_test").limit(10);
    sql.createView("my_view", sql_subquery);
```

### dropView()

Generate a drop view statement. It takes a single parameter
of the view name to drop.

```JavaScript
    sql.dropView("my_view");
```

### insert()

This generates a row insert statement. Parameters are
table name and an object which has attribute names corresponding
to column names and attribute values containing the values to
be inserted.

```JavaScript
	sql.insert("test1", {
		name: "Fred",
		tunnel_id: 12,
		start_date: new Date("01/22/1803"),
		modified: new Date()
	});
```

### replace()

Replace is support for SQLite3 and MySQL55 dialects. It takes
the same parameters as insert.

```JavaScript
	sql.replace("test1", {
		id: 231,
		name: "Fred",
		tunnel_id: 2,
		start_date: new Date("01/22/2003"),
		modified: new Date()
	});
```

### update()

Generate an update clause. It is usually combined with
a set() and where(). Update takes the table name as the only parameter.

```JavaScript
	sql.update("story_book_characters").set([{name: "Albert"},
        {email: al@example.mf"}]).where({id: 1});
```

### select()

This creates the clause _SELECT *FIELD NAMES*_, _SELECT UNIQUE *FIELD_NAMES*_
or _SELECT ALL *FIELD_NAMES*_. The Select takes two parameter, the second
is optional.  The first parameter accepts either a string that is a
single column identifier, an array of strings as a list of columns or number
if you want the SQL wild card for all columns. A SQL function name supported
can also be used as a column identifier. Columns can be
aliased with the AS operator if they are in an array of of column names. The alias
is specified with a name key/value object literal _{unique: true}_ would be
used to generate _SELECT UNIQUE_.

```JavaScript
	// SELECT myId
	sql.select("myId");
	// SELECT COUNT()
	sql.select("COUNT()");
	// SELECT COUNT() AS subtotal
    sql.select([{"COUNT()": "subtotal"}]);
    // SELECT id, name, email
    sql.select(["id", "name", "email"]);
    // SELECT UNIQUE email, name
    sql.select(["email", "name"], {unique: true});
    // SELECT ALL email, name
    sql.select(["email", "name"], {all: true});
```

### union()

Render a union clause. Takes one parameters each an Sql object. Not supported
in SQLite3.

```JavaScript
sql1 = sqlish.Sqlish();
sql2 = sqlish.Sqlish();

sql1.select().from("test1");
sql2.select().from("test2");
// render (SELECT * FROM test1) UNION (SELECT * FROM test2)
sql.union(sql1, sql2);
```

### transaction()

Generate a transaction bound set of SQL statements.

### savepoint()

Sets marker among the array of SQL statements in the
transaction block.

### commit()

Sets a marker amount the array of SQL statements in the
transaction block.

### rollback()

Sets a marker for rollback in the array of SQL statements
in the transaction block.

# Supporting clauses

### from()

Generates a from clause. Usually used with select. Takes
a single table name or an array of table names.

```JavaScript
    // SQL: SELECT id, name FROM story_book_characters
	sql.select(["id", "name"]).from("story_book_characters")
```

### where()

Generating a where clause. Usually used with select.
Takes an expression as its parameter.

```JavaScript
    // SQL: SELECT id, name FROM story_book_characters
    // WHERE name = "fred";
	sql.select(["id", "name"]).from("story_book_characters")
        .where({name: "fred"});
```

If you are comparing two columns then you need to define some
validation rules with the _on()_.

```JavaScript
	sql.on("test1", {
		id: function (value) {
			if (value === "test1.id" || parseInt(value, 10) === value) {
				return value;
			}
			throw "injection error, test1.id: " + value;
		}
	});
	sql.on("test2", {
		id: function (value) {
			if (value === "test2.test_id" || parseInt(value, 10) === value) {
				return value;
			}
			throw "injection error, test.test_id: " + value;
		}
	});
	sql.select([{"test1.id": "id"}, {"test1.name": "name"}, {"test2.id": "test2_id"}])
		.from(["test1", "test2"])
        .where({"test1.id": {$eq: "test2.test1_id"}});
````

### into()

Into is usually used with _select()_. It is not supported
by the SQLite 3 dialect. Takes a string or array of variable
or column names.

```JavaScript
    // MySQL 5.5: SELECT 2 INTO @number;
    // PostgreSQL 9.2: SELECT 2 INTO number;
    sql.select(2).into("number");
```

### join()

Renders a JOIN clause. Takes two parameters the table to join to
and the expression to join with.

```JavaScript
    sql.select().from("test").joinOn("something", {"test.id" : "something.test_id"}));
```

# Expressions

### =, !=, >, >=, <, <=

Equal, not equal, greater than, greater than or equal, less than,
and less than or equal are expressed using _$eq_, _$ne_, _$gt_, _$gte_,
_$le_, and _$lte_. Equals can be expressed two ways. Implicitly
where the value assigned to the attribute is a number, string,
or date. Explicitly it can be expression where the value is an
object who's attribute name is _$eq_.

```JavaScript
    // SQL: cnt > 3
    expr = {cnt: {$gt: 3}};
    // SQL: cnt >= 3
    expr = {cnt: {$gte: 3}};
    // SQL: cnt = 3
    expr = {cnt: 3};
    expr = {cnt: {$eq: 3}};
    // SQL: cnt != 3
    expr = {cnt: {$ne: 3}};
    // SQL: cnt < 3
    expr = {cnt: {$lt: 3}};
    // SQL: cnt <= 3
    expr = {cnt: {$lte: 3}};
```

### OR, AND

The conjunction operators OR are expressioned using _$or_ and _$and_.
Both take an erray literal as its value or the function _P()_.

```JavaScript
    // SQL: cnt = 3 OR cnt = 6
    expr = {$or: [{cnt: 3}, {cnt: 6}]};
    // SQL: cnt = 3 OR cnt = 6 OR cnt = 9
    expr = {$or: [{cnt: 3}, {cnt: 6}, {cnt: 9}]};
    // SQL: cnt > 3 AND cnt < 9
    expr = {$and: [{cnt: {$gt: 3}}, {cnt: {$lt: 9}}]};
    // SQL: (cnt = 3 OR cnt < 9)
    expr = sql.P({$or: [{cnt: 3}, {cnt: {$lt: 9}}]})
    // SQL: (cnt = 3) OR (cnt < 9)
    expr = {$or: [sql.P({cnt: 3}), sql.P({cnt: {$lt: 9})}]})
```

### (), parenthesis

    Generating parenthesis in expressions is a little different than
    in MonogDB. Group with parenthesis is accomplish by a function
    named _P_. It evaluates the objet literal passed to it, converts
    that literal to a string and wraps the results in parenthesis.

```JavaScript
    // SQL: (cnt = 3)
    expr = sql.P({cnt: 3});
    // SQL: (cnt = 3) or (cnt = 9)
    expr = {$or: [sql.P({cnt: 3}), sql.P({cnt: 9})]};
```

# Adding or modifying a dialect

This is hypathetical psuedo code at this point. Will definately change before
implemented.

```JavaScript
    // Just some sketches of what it might look like.

    sqlish.Dialect.define("ReadOnlySQL", {
    	verbs: {
            // Defining schema
            createTable: false,
            dropTable: false,
            createIndex: false,
            dropIndex: false,
            createView: false,
            dropView: false,

            // Manimulating rows
            insert: false,
            update: false,
            replace: false,
            select: true,
            union: true,
            set: false,
            deleteFrom: false,
        },
    	clauses: {
            // Modifing a verb
            set: false,
            from: true,
            join: true,
            where: true,
            limit: true,
            order: true,
            group: true,
            into: false
        },
    	// Replaces default toString()
    	toString: function (someArgs) {
    		... Code to render inner sql Model
    	},
    	// Optional method bind a db connection
    	// e.g. could transform the active verb
    	// and clauses into a MongoDB query.
    	// Should support steams of results if
    	// no callback is provided
    	execute: function (...someArgs, callback) {
    	}
    });

    // Now set the dialect of a sql object.
    sql = new sqlish.Sqlish("SQL Read Only", {
    	createTable: function () {
    		throw "createTable not supported"
    	},
    	createIndex: function () {
    		throw "createIndex not supported"
    	},
    	createView: function () {
    		throw "createView not supported"
    	},
    	dropTable: function () {
    		throw "dropTable not supported"
    	},
    	dropIndex: function () {
    		throw "dropIndex not supported"
    	},
    	dropView: function () {
    		throw "dropView not supported"
    	},
    	deleteFrom: function () {
    		throw "deleteFrom not supported"
    	},
    	insert: function () {
    		throw "insert not supported"
    	},
    	update: function () {
    		throw "update not supported"
    	},
    	replace: function () {
    		throw "replace not supported"
    	}
    });

    // This is Ok in SQLReadOnly dialect
    sql.select("id").from("test").where({id: 203});
    // This will throw an error because
    // SQLReadOnly does not support "deleteFrom", so
    // it will throw an error
    sql.deleteFrom("test").where({id: 203});
```

repository: rsdoiel/sqlish

gha_language: JavaScript

repo_name: rsdoiel/sqlish


text:
MongooseJS models for NowJS clients
--------------------------------------

To enable:

var now_mongoose = require('now-mongoose');

To activate, when you're ready:
-------------------------------

everyone.now.MODEL_NAME = now_mongoose.model( MODEL_CLASS );

Parameters are:

* the MongooseJS model class you created using the mongoose.model() api

For example, let's say you have a mongoose model named Adventure which you want to expose to connected clients via a real-time API:

SERVER SIDE

// connect to db
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/usefuldb');

// create server
var express = require('express');
var mongoose = require('mongoose');

var nowjs = require('now');
var everyone = nowjs.initialize(app);

var now_mongoose = require('now-mongoose');

var AdventureSchema = new mongoose.Schema({
		title: { type: String, required: true },
		description: { type: String, required: true },
		rsvps: [{ type: mongoose.Schema.ObjectId, ref: 'User' }]
	});

AdventureSchema.statics.rsvp = function(adventure_id, user_id, callback){
	this.findById( adventure_id, function(err,adventure){
		adventure.rsvps.push(user_id);
		adventure.save(function(err){
			callback(err,adventure);
		});
	});
};

var Adventure = mongoose.model( 'Adventure', AdventureSchema );

everyone.now.Adventures = now_mongoose.model( Adventure )

Now on the CLIENT SIDE, you can do this:

now.Adventure.search(function(err,adventures){ console.log(adventures) });
> Retrieves all Adventures in the DB and returns them to the callback

now.Adventure.search( {title: 'International Skeeball'}, function(err,adventures){ console.log( adventures ); });
> Retrieves Adventures with the title 'International Skeeball' and returns them to the callback

now.Adventure.save(	{ 	title: 'Irish Coffee Marathon',
						description: 'Beats the hell outta actually running'
					}, function(err,adventure){
							console.log( adventure )
						}
					);
> Creates a new Adventure with the given title and description and returns it to the callback

now.Adventure.save(	{ 	_id: '4efadce3223444adf',
						title: 'Irish Coffee Marathon of Doom'
					}, function(err,adventure){
							console.log( adventure )
						}
					);
> Updates the given Adventure with the parameters you pass, in this case an updated title

now.Adventure.get( '43edaff3243242a334423',
					function(err, adventure){
						console.log( adventure )
					}
				);
> Retrieves the Adventure with the given id and returns it to the callback

In addition, now-mongoose also exposes all static methods defined on your Mongoose model to the NowJS client, so using the example above, calling:

var adventure_id = '43ee432429898432iadfsdf';
var user_id = '33334524543523242ff';
now.Adventure.rsvp( adventure_id, user_id, function( err, adventure ) );

would work and return you the updated adventure after the new rsvp was saved for the given user id.

NOTE: Model instance level methods are more tricky and I haven't figured out how to proxy those yet, any suggestions welcome!

repository: zseti/now-mongoose

repo_name: zseti/now-mongoose


text:
# ArRollout

ArRollout is the ActiveRecord version of [Rollout](https://github.com/jamesgolick/rollout), written
after the wonderful Railscast [315-rollout-and-degrade](http://railscasts.com/episodes/315-rollout-and-degrade)
by Ryan Bates (thank you).
Originally released as https://github.com/doochoo-inc/ar_rollout but now maintained at https://github.com/markpundsack/ar_rollout.

This project rocks and uses MIT-LICENSE.

## HowTo

To use this gem, add on your gemfile

```ruby
gem 'ar_rollout'
```

and run the generator:

    rails generate ar_rollout

This will create for you the initializer and the migration.

The migration looks like this:

```ruby
class CreateRollout < ActiveRecord::Migration
  def up
    create_table :rollouts do |t|
      t.string :name
      t.string :group
      t.integer :user_id
      t.integer :percentage
      t.integer :failure_count
    end
  end

  def down
    drop_table :rollouts
  end
end
```

The initializer like this:

```ruby
ArRollout.configure do |configure|
  # # Here you can define the logic for your own groups
  # # For example, if you have a admin? method for your User class
  # # you can define an :admin group:

  configure.define_group(:all) do |user|
    true
  end

  # configure.define_group :admin do |user|
  #   user.admin?
  # end

  # What if you want to add your 3 developers using email to a :dev group?
  # configure.define_group :devs do |user|
  #<EMAIL_ADDRESS><EMAIL_ADDRESS>user.email
  # end
end
```

You have to run the migration to have the table rollouts created in database:

    rake db:migrate

And that's it. You're now ready to use ArRollout.

If you want to add a feature to a group you define in `config/initializer/ar_rollout.rb`, in your rails console
you can do:

```ruby
ArRollout.activate_group  :my_new_amazing_feature, :tester
ArRollout.activate_user   :my_new_amazing_feature, current_user
```

Or via rake with:

```
rake rollout:activate_group[my_new_amazing_feature,all]
```

So, in your controller/view you can use the helper method `rollout? :my_new_amazing_feature`, which will test if the
`current_user` is enabled to that feature.

In your *_controller.rb:

```ruby
before_filter :rollout
around_filter :degrade

private

def rollout
  redirect_to root_url, alert: "Feature unavailable" unless rollout? :phone
end

def degrade
  degrade_feature(:phone) { yield }
end
```

List known features with:

`ArRollout.features`

## Versions and ToDo

## ToDo

- Optimize user lookup
- Optimize database structure
- Merge ArRollout and Rollout class methods
- Autodetection of new features from code, before rollout begins

## Version 0.0.20 - 04 Dec 2012
- Add Group, Membership, OptOut

## Version 0.0.17 - 29 Nov 2012
- ArRollout caches groups

## Version 0.0.16 - 29 Nov 2012
- Add rake task 'rollout:summary'

## Version 0.0.14 - 28 Nov 2012
- Add rake task 'rollout:list'

## Version 0.0.7 - 16 Aug 2012
- Add `ArRollout.info`

## Version 0.0.4 - 16 Aug 2012
- Add percentage support
- Add feature list with `ArRollout.features`

## Version 0.0.3 - 16 Aug 2012
- Add `ArRollout.deactivate_all` method and rake task

## Version 0.0.2 - 16 Aug 2012
- Add `:all` to default initializer

## Version 0.0.1 - 13 Jan 2012
- Generator which create both migration and initializer
- `activate_group` and `deactivate_group` methods are available
- `activate_user_` and `deactivate_user` methods are available


text:
# nand2tetris

## 01

- [ ] Windows annoyances - executable files and Windows EOLs.

## 02

- [ ] Is there a more efficient FullAdder based on HalfAdders?

``` hdl
Mux16(a=out0, b=out1, sel=no, out=out,
                              out[0..7]=out07,
                              out[8..15]=out815,
                              out[15]=ng);
```

repository: kejadlen/katas

gha_language: Ruby

repo_name: kejadlen/katas


text:
# AWS

AWS Certified Cloud Practitioner

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Security.html

### RDS services:

```
VPC - Virtual Private Cloud.
AS  -  Auto scaling
AL  -  Load balancing
```

repository: sandysanthosh/AWS

branch_name: refs/heads/main

repo_name: sandysanthosh/AWS


text:
# Cloudmesh Pi Burner for SD Cards

**WARNING:** *This program is designed for a Raspberry Pi and must not
be executed on your laptop or desktop. An earlier version that could
be run on **Linux, macOS, and Windows 10 is no longer supported**. If
you want to help us porting them on any of these OSes, please contact
<EMAIL_ADDRESS>

[![image](https://img.shields.io/travis/TankerHQ/cloudmesh-pi-burn.svg?branch=main)](https://travis-ci.org/TankerHQ/cloudmesn-pi-burn)
[![image](https://img.shields.io/pypi/pyversions/cloudmesh-pi-burn.svg)](https://pypi.org/project/cloudmesh-pi-burn)
[![image](https://img.shields.io/pypi/v/cloudmesh-pi-burn.svg)](https://pypi.org/project/cloudmesh-pi-burn/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

<!--TOC-->

- [Cloudmesh Pi Burner for SD Cards](#cloudmesh-pi-burner-for-sd-cards)
  - [Introduction](#introduction)
  - [Nomenclature](#nomenclature)
  - [Quickstart for Bridged WiFi](#quickstart-for-bridged-wifi)
    - [Requirements](#requirements)
    - [Master Pi](#master-pi)
    - [Single Card Burning](#single-card-burning)
    - [Burning Multiple SD Cards with a Single Burner](#burning-multiple-sd-cards-with-a-single-burner)
    - [Connecting Pis to the Internet via Bridge](#connecting-pis-to-the-internet-via-bridge)
  - [Set up of the SSH keys and SSH tunnel](#set-up-of-the-ssh-keys-and-ssh-tunnel)
  - [Manual Pages](#manual-pages)
    - [Manual Page for the `burn` command](#manual-page-for-the-burn-command)
    - [Manual Page for the `bridge` command](#manual-page-for-the-bridge-command)
    - [Manual Page for the `host` command](#manual-page-for-the-host-command)
    - [Manual Page for the `pi` command](#manual-page-for-the-pi-command)
  - [FAQ and Hints](#faq-and-hints)
    - [Can I use the LEDs on the PI Motherboard?](#can-i-use-the-leds-on-the-pi-motherboard)
    - [How can I use pycharm, to edit files or access files in general from my Laptop on the PI?](#how-can-i-use-pycharm-to-edit-files-or-access-files-in-general-from-my-laptop-on-the-pi)
    - [How can I enhance the `get` script?](#how-can-i-enhance-the-get-script)
    - [Can I use a Mesh Network for the setup?](#can-i-use-a-mesh-network-for-the-setup)
    - [Can I use cms burn on Linux?](#can-i-use-cms-burn-on-linux)

<!--TOC-->

## Introduction

`cms burn` is a program to burn many SD cards for the preparation of
building clusters with Raspberry Pi's. It allows users to create
readily bootable SD cards that have the network configured, contain a
public ssh key from your machine that you used to configure the
cards. Thus not much additional setup is needed for a cluster. Another
unique feature is that you can burn multiple cards in a row, each with
their individual setup such as hostnames and ipadresses.

## Nomenclature

* Commands proceeded with `pi@red:$` are to be executed on the
  Rasperry Pi with the name red.

* Commands with `(ENV3) pi@red:$` are to be executed in a virtula ENV
  using Python 3 on the Raspberry Pi with the name red

## Quickstart for Bridged WiFi

To provide you with a glimpse on what you can do with cms burn, we
have provided this quickstart guide that will create one master PI and
several workers.

This setup is intended for those who have restricted access to their
home network (ie. cannot access router controls).  For example, those
on campus WiFis or regulated apartment WiFis.

The Figure 1 describes our network configuration. We have 5
Raspberry Pi 4s: 1 master and 4 workers. We have WiFi access, but we
do not necessarily have access to the router's controls.

We also have a network switch, where the master and workers can
communicate locally, but we will also configure the master to provide
internet access to devices on the network switch via a "network
bridge".

![](https://github.com/cloudmesh/cloudmesh-pi-burn/raw/main/images/network-bridge.png)

Figure 1: Pi Cluster setup with bridge network

### Requirements

For the quickstart we have the following requirements:

* SD Cards and Raspberry Pis

* Master Pi: You will need at least **1 Raspberry Pi** SD Card burned
  using [Raspberry Pi imager](https://www.raspberrypi.org/software/).
  You can use your normal operating system to burn such a card
  including Windows, macOS, or Linux.  Setting up a Raspberry Pi in
  this manner should be relatively straightforward as it is nicely
  documented online (For example,
  [how to setup SSH](https://www.raspberrypi.org/documentation/remote-access/ssh/)).
  All you will need for this guide is an internet connection for your
  Pi. It might also be of use to change the hostname of this Pi.

* You will need an SD card writer (USB tends to work best) to burn new
  cards We recommend that you invest in a USB3 SDCard writer as they
  are significantly faster and you can resuse them on PI'4s

### Master Pi

First we need to configure the Master Pi

**Step 1.** Installing Cloudmesh on the Master Pi

Update pip and the simple curl command below will generate an ssh-key,
update your system, and install cloudmesh.

```
pi@masterpi:~ $ pip install pip -U
pi@masterpi:~ $ curl -Ls http://cloudmesh.github.io/get/pi | sh
                # see note use different link for now
```

Note: at present we are still improving the pi script and thus you
should for now use the command

```
pi@masterpi:~ $  curl -Ls https://raw.githubusercontent.com/cloudmesh/get/main/pi/index.html | sh
```

This will take a moment...

**Step 2.** Reboot

The installation script updates your system. Reboot for effect.

```
pi@masterpi:~ $ sudo reboot
```

**Step 3.** Download the latest Raspberry Pi Lite OS

TODO: does this still work. E.g. do w not have to get the list of
newest images first?

The following command will download the latest images for Raspberry
Lite OS.

```
(ENV3) pi@masterpi:~ $ cms burn image get latest-lite
```

We can verify our image's downloaded with the following.

```
(ENV3) pi@masterpi:~ $ cms burn image ls
```

**Note.** We can use the following command to list the current
  Raspberry Pi OS versions (full and lite)

```
(ENV3) pi@masterpi:~ $ cms burn image versions --refresh
```

This will list the Tags and Types of each available OS. We can then
modify the `image get` command for versions we are interested in. For
example,

```
(ENV3) pi@masterpi:~ $ cms burn image get full-2020-05-28
```

**Step 4**. Setup SD Card Writer

Plug your SD Card Writer into the Pi. Ensure you have an SD Card
inserted into your writer. Run the following command to find the path
to your SD Card.

```
(ENV3) pi@masterpi:~ $ cms burn info
...
# ----------------------------------------------------------------------
# SD Cards Found
# ----------------------------------------------------------------------

+----------+----------------------+----------+-----------+-------+------------------+---------+-----------+-----------+
| Path     | Info                 | Readable | Formatted | Empty | Size             | Aaccess | Removable | Writeable |
+----------+----------------------+----------+-----------+-------+------------------+---------+-----------+-----------+
| /dev/sda | Generic Mass-Storage | True     | True      | False | 64.1 GB/59.7 GiB | True    | True      |           |
+----------+----------------------+----------+-----------+-------+------------------+---------+-----------+-----------+
```

> `cms burn info` has other useful information, but for the purposes
> of this guide we omit it.

We can see from the information displayed that our SD card's path is
`/dev/sda`. Of course, this may vary.

### Single Card Burning

Step 0. Ensure the SD card is inserted.

We can run `cms burn info` again as we did above to verify our
SD card is connected.

Step 1. Burning the SD Card

Choose a hostname for your card. We will use `red001` with ip
`<IP_ADDRESS>`. The IP address `<IP_ADDRESS>` is reserved for the burner pi
(ie. `masterpi`).

> Note we are using the subnet `<IP_ADDRESS>/24` in this guide. We
> currently recommend you do the same, otherwise the WiFi bridge will
> not configure correctly. We will change this in the future to
> support other
> [Private IP Ranges](https://www.arin.net/reference/research/statistics/address_filters/)

```
(ENV3) pi@masterpi:~ $ cms burn create --hostname=red001 --ip=<IP_ADDRESS> --device=/dev/sda --tag=latest-lite
```

Wait for the card to burn. Once the process is complete, it is safe
to remove the SD card.

We can now proceed to [the bridge setup](#connecting-pis-to-the-internet-via-bridge )

### Burning Multiple SD Cards with a Single Burner

Step 0. Ensure the first SD card is inserted into the burner.

We can run `cms burn info` again as we did above to verify our SD
card is connected.

Step 2. Burning the Cards

`cms burn` supports logical incremenation of numbers/characters.

For example, `red00[1-2]` is interpreted by cms burn as `[red001, red002]`.
Similarly, `red[a-c]` is interpreted by cms burn as `[reda, redb, redc]`.

We can burn 2 SD cards as follows:

```
(ENV3) pi@masterpi:~ $ cms burn create --hostname=red00[1-2] --ip=10.1.1.[2-3] --device=/dev/sda --tag=latest-lite
```

The user will be prompted to swap the SD cards after each card burn if
there are still remaining cards to burn.

We can now proceed to the next section where we configure our bridge.

### Connecting Pis to the Internet via Bridge

Figure 1 depicts how the network is set up with the help of the bridge command.

![](https://github.com/cloudmesh/cloudmesh-pi-burn/raw/main/images/network-bridge.png)

Figure 1: Networking Bridge

**Step 0.** Review and Setup

At this point we assume that you have used `cms burn` to create all SD
cards for the Pi's with static IP addresses in the subnet range
`<IP_ADDRESS>/24` (excluding `<IP_ADDRESS>`. See step 1 for details)

We are also continuing to use `masterpi` (which is where we burn the
worker SD cards).

We will now use `cms bridge` to connect the worker Pis to the
internet. Let us again reference the diagram of our network setup. You
should now begin connecting your Pis together via network switch
(unmanaged or managed) if you have not done so already. Ensure that
`masterpi` is also connected into the network switch.

**Step 1.** Configuring our Bridge

We can easily create our bridge as follows.

```
(ENV3) pi@masterpi:~ $ cms bridge create --interface='wlan0'
```

We should now reboot.

```
(ENV3) pi@masterpi:~ $ sudo reboot
```

> Note the `--interface` option indicates the interface used by the
> master pi to access the internet.  In this case, since we are using
> WiFi, it is likely `wlan0`. Other options such as `eth0` and `eth1`
> exist for ethernet connections.

**Step 2.** Verifying internet connection

At this point, our workers should have internet access. Let us SSH
into one and ping google.com to verify.

```
(ENV3) pi@masterpi:~ $ ssh red001

pi@red001:~ $ ping google.com
PING google.com (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from mia07s57-in-f14.1e100.net (<IP_ADDRESS>): icmp_seq=1 ttl=106 time=48.2 ms
64 bytes from mia07s57-in-f14.1e100.net (<IP_ADDRESS>): icmp_seq=2 ttl=106 time=48.3 ms
64 bytes from mia07s57-in-f14.1e100.net (<IP_ADDRESS>): icmp_seq=3 ttl=106 time=47.9 ms
64 bytes from mia07s57-in-f14.1e100.net (<IP_ADDRESS>): icmp_seq=4 ttl=106 time=47.10 ms
64 bytes from mia07s57-in-f14.1e100.net (<IP_ADDRESS>): icmp_seq=5 ttl=106 time=48.5 ms
^C
--- google.com `ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 9ms
rtt min/avg/max/mdev = 47.924/48.169/48.511/0.291 ms
```

Note how we are able to omit the pi user and .local extension. We have
successfuly configured our bridge.

## Set up of the SSH keys and SSH tunnel

One important aspect of a cluster is to setup authentication via ssh
in a convenient way, so we can easily login from the laptop to each of
the PI workers and the PI manager. Furthermore, we like to be able to
login from the PI manager to each of the workers. In addition, we like
to be able to login between the workers.

We have chosen a very simple setup while relying on ssh tunnel.

To simplify the setup of this we have developed a command `cms host`
that gathers and scatters keys onto all machines, as well as, sets up
the tunnel.

It is essential that that the key on the laptop must not be password
less. This is also valid for any machine that is directly added to the
network such as in the mesh notwork.

To avoid password less keys we recommend you to use `ssh-add`
or `ssh-keychain` which will ask you for one.

> Note: More information and a concrete example will be documented
> here shortly.

The manual page for `cms host` is provided in the Manual Pages
section.

**Step 1.** On the manager create ssh keys for each of the workers.

```
(ENV3) pi@managerpi:~ $ cms host key create red00[1-3]
```

**Step 2.** On the manager gather the worker, manager, and your laptop
public ssh keys into a file.

```
(ENV3) pi@managerpi:~ $ cms host key gather<EMAIL_ADDRESS>keys.txt
```

**Step 3.** On the manager scatter the public keys to all the workers
and manager ~/.ssh/authorized_hosts file

```
(ENV3) pi@managerpi:~ $ cms host key scatter red00[1-3],localhost keys.txt
```

**Step 4.** Remove undeeded keys.txt file

```
(ENV3) pi@managerpi:~ $ rm keys.txt
```

**Step 5.** Verify SSH reachability from worker to manager and worker to worker.

```
(ENV3) pi@managerpi:~ $ ssh red001
pi@red001:~ $ ssh managerpi
```

TODO: check this comment

BUG: if the manager is still named raspberrypi then the
worker might resolve it as <IP_ADDRESS>. Use raspberrypi.local instead.

```
(ENV3) pi@managerpi:~ $ exit
pi@red001:~ $ ssh red002
pi@red002:~ $ exit
pi@red001:~ $ exit
```

**Step 6.** (For Bridge setup) Create SSH tunnels on the manager
to enable ssh acces from your laptop to the workers

For now we manually install autossh, to test the new cms host tunnel
program. Later we add it to the main master setup script.

```
(ENV3) pi@managerpi:~ $ yes y | sudo apt install autossh
```

```
(ENV3) pi@managerpi:~ $ cms host tunnel create red00[1-3]
```

**Step 7.** (For Bridge setup) Copy the specified command output to
your `~/.ssh/config` file on your laptop

```
host tunnel create red00[1-3]

Using wlan0 IP = <IP_ADDRESS>
Using cluster hostname = managerpi

Tunnels created.

Please place the following in your remote machine's (i.e. laptop)
`~/.ssh/config` file to alias simple ssh access (i.e. `ssh red001`).

# ----------------------------------------------------------------------
# copy to ~/.ssh/config on remote host (i.e laptop)
# ----------------------------------------------------------------------

Host red001
     HostName managerpi.local
     User pi
     Port 8001

Host red002
     HostName managerpi.local
     User pi
     Port 8002

Host red003
     HostName managerpi.local
     User pi
     Port 8003
```

> Note: We will in future provide an addition to the command so you
> can remove and add
> them directly from the commandline
>
> ```
> cms host tunnel config create red00[1-3]
> cms host tunnel config delete red00[1-3]
> ```

**Step 8.** (For Bridge setup) Verify SSH reachability from the laptop
  to workers

```
you@yourlaptop:~ $ ssh red001
```

## Manual Pages

### Manual Page for the `burn` command

Note to execute the command on the commandline you have to type in
`cms burn` and not jsut `burn`.

<!--MANUAL-BURN-->
```
  burn firmware check
  burn install
  burn load --device=DEVICE
  burn format --device=DEVICE
  burn mount [--device=DEVICE] [--os=OS]
  burn unmount [--device=DEVICE] [--os=OS]
  burn network list [--ip=IP] [--used]
  burn network
  burn info [--device=DEVICE]
  burn image versions [--refresh] [--yaml]
  burn image ls
  burn image delete [--image=IMAGE]
  burn image get [--url=URL] [TAG...]
  burn backup [--device=DEVICE] [--to=DESTINATION]
  burn copy [--device=DEVICE] [--from=DESTINATION]
  burn shrink [--image=IMAGE]
  burn create [--image=IMAGE]
              [--device=DEVICE]
              [--hostname=HOSTNAME]
              [--ip=IP]
              [--sshkey=KEY]
              [--blocksize=BLOCKSIZE]
              [--dryrun]
              [--passwd=PASSWD]
              [--ssid=SSID]
              [--wifipassword=PSK]
              [--format]
              [--tag=TAG]
  burn sdcard [TAG...] [--device=DEVICE] [--dryrun]
  burn set [--hostname=HOSTNAME]
           [--ip=IP]
           [--key=KEY]
  burn enable ssh
  burn wifi --ssid=SSID [--passwd=PASSWD] [-ni]
  burn check [--device=DEVICE]

Options:
  -h --help              Show this screen.
  --version              Show version.
  --image=IMAGE          The image filename,
                         e.g. 2019-09-26-raspbian-buster.img
  --device=DEVICE        The device, e.g. /dev/mmcblk0
  --hostname=HOSTNAME    The hostname
  --ip=IP                The IP address
  --key=KEY              The name of the SSH key file
  --blocksize=BLOCKSIZE  The blocksise to burn [default: 4M]

Arguments:
    TAG                  Keyword tags to identify an image
                         [default: latest]
Files:
  This is not fully thought through and needs to be documented
  ~/.cloudmesh/images
    Location where the images will be stored for reuse

Description:
    cms burn create --passwd=PASSWD

         if the passwd flag is added the default password is
         queried from the commandline and added to all SDCards

         if the flag is ommitted login via the password is disabled
         and only login via the sshkey is allowed

  Network

    cms burn network list

        Lists the ip addresses that are on the same network

         +------------+---------------+----------+-----------+
         | Name       | IP            | Status   | Latency   |
         |------------+---------------+----------+-----------|
         | Router     | <IP_ADDRESS>   | up       | 0.0092s   |
         | iPhone     | <IP_ADDRESS>   | up       | 0.061s    |
         | red01      | <IP_ADDRESS>  | up       | 0.0077s   |
         | laptop     | <IP_ADDRESS>  | up       | 0.058s    |
         | unkown     | <IP_ADDRESS> | up       | 0.14s     |
         | red03      | <IP_ADDRESS> | up       | 0.0037s   |
         | red02      | <IP_ADDRESS> | up       | 0.0046s   |
         | red        | <IP_ADDRESS> | up       | 0.00021s  |
         +------------+----------------+----------+-----------+

    cms burn network list [--used]

        Lists the used ip addresses as a comma separated parameter
        list

           <IP_ADDRESS>,<IP_ADDRESS>,...

    cms burn network address

        Lists the own network address

         +---------+----------------+----------------+
         | Label   | Local          | Broadcast      |
         |---------+----------------+----------------|
         | wlan0   | <IP_ADDRESS>   | <IP_ADDRESS>  |
         +---------+----------------+----------------+

    cms burn firmware check

        checks if the firmware on the Pi is up to date

    cms burn install

        installs a program to shring img files. THis is useful, after
        you created a backup to make the backup smaller and allow
        faster burning in case of recovery

    cms burn load --device=DEVICE

        loads the sdcard into the USB drive. Thi sis similar to
        loading a cdrom drive. It s the oposite to eject

    cms burn format --device=DEVICE

        formats the SDCard in the specified device. Be careful it is
        the correct device.  cms burn info will help you to identifying it

    cms burn mount [--device=DEVICE] [--os=OS]

        mounts the file systems available on the SDCard

    cms burn unmount [--device=DEVICE] [--os=OS]

        unmounts the mounted file systems from the SDCard

    cms burn info [--device=DEVICE]

        provides useful information about the SDCard

    cms burn image versions [--refresh] [--yaml]

        The images that you like to burn onto your SDCard can be cached locally with the image command.
        The available images for the PI can be found when using the --refresh option. If you do not
        specify it it reads a copy of the image list from our cache

    cms burn image ls

        Lists all downloaded images in our cache. You can download
        them with the cms burn image get command

    cms burn image delete [--image=IMAGE]

        deletes the specified image. The name can be found with the image ls command

    cms burn image get [--url=URL] [TAG...]

        downloads a specific image or the latest image. The tag are a number of words
        separated by a space that must occur in the tag that you find in the versions command

    cms burn backup [--device=DEVICE] [--to=DESTINATION]

        backs up a SDCard to the given location

    cms burn copy [--device=DEVICE] [--from=DESTINATION]

        copies the file form the destination on the SDCard
        this is the same as the SDCard command. we will in future remove one

    cms burn shrink [--image=IMAGE]

        shrinks the size of a backoup or image file that is on
        your local file system. It can only be used for .img files

    cms burn create [--image=IMAGE]
                    [--device=DEVICE]
                    [--hostname=HOSTNAME]
                    [--ip=IP]
                    [--sshkey=KEY]
                    [--blocksize=BLOCKSIZE]
                    [--dryrun]
                    [--passwd=PASSWD]
                    [--ssid=SSID]
                    [--wifipassword=PSK]
                    [--format]

        This is a comprehensif cuntion that not only can format the SDCard, but also
        initializes it with specific falues

    cms burn sdcard [TAG...] [--device=DEVICE] [--dryrun]

        this burns the sd card, see also copy and create

    cms burn set [--hostname=HOSTNAME]
                 [--ip=IP]
                 [--key=KEY]
                 [--mount=MOUNTPOINT]

        this sets specific values on the sdcard after it has ben created
        with the creat, copy or sdcard command

        a --ssh is missing from this command

    cms burn enable ssh [--mount=MOUNTPOINT]

        this enables the ssh server once it is booted

    cms burn wifi --ssid=SSID [--passwd=PASSWD] [-ni]

        this sets the wifi ssid and password afterthe card is created,
        copies, or sdcard is used

    cms burn check [--device=DEVICE]

        this command lists the parameters that were set with the set or create command

Examples: ( \ is not shown)

   > cms burn create --image=2019-09-26-raspbian-buster-lite
   >                 --device=/dev/mmcblk0
   >                 --hostname=red[5-7]
   >                 --ip=192.168.1.[5-7]
   >                 --sshkey=id_rsa

   > cms burn image get latest

   > cms burn image get https://downloads.raspberrypi.org/
   >   raspbian_lite/images/
   >   raspbian_lite-2018-10-11/2018-10-09-raspbian-stretch-lite.zip

   > cms burn image delete 2019-09-26-raspbian-buster-lite

```
<!--MANUAL-BURN-->

### Manual Page for the `bridge` command

Note to execute the command on the commandline you have to type in
`cms bridge` and not jsut `bridge`.

<!--MANUAL-BRIDGE-->
```
Options:
    --interface=INTERFACE  The interface name [default: eth1]
                           You can also specify wlan0 if you wnat
                           to bridge through WIFI on the master
                           eth0 requires a USB to WIFI adapter

Description:

  Command used to set up a bride so that all nodes route the traffic
  trough the master PI.

  bridge create [--interface=INTERFACE]
      creates the bridge on the current device.
      A reboot is required.

```
<!--MANUAL-BRIDGE-->

### Manual Page for the `host` command

Note to execute the command on the commandline you have to type in
`cms host` and not jsut `host`.

<!--MANUAL-HOST-->
```
    host scp NAMES SOURCE DESTINATION [--dryrun]
    host ssh NAMES COMMAND [--dryrun] [--output=FORMAT]
    host config NAMES [IPS] [--user=USER] [--key=PUBLIC]
    host check NAMES [--user=USER] [--key=PUBLIC]
    host key create NAMES [--user=USER] [--dryrun] [--output=FORMAT]
    host key list NAMES [--output=FORMAT]
    host key gather NAMES [--authorized_keys] [FILE]
    host key scatter NAMES FILE
    host tunnel create NAMES [--port=PORT]

This command does some useful things.

Arguments:
    FILE   a file name

Options:
    --dryrun   shows what would be done but does not execute
    --output=FORMAT  the format of the output
    --port=PORT starting local port for tunnel assignment

Description:

    host scp NAMES SOURCE DESTINATION

      TBD

    host ssh NAMES COMMAND

      runs the command on all specified hosts
      Example:
           ssh red[01-10] "uname -a"

    host key create NAMES
      create a ~/.ssh/id_rsa and id_rsa.pub on all hosts specified
      Example:
          ssh key create "red[01-10]"

    host key list NAMES

      list all id_rsa.pub keys from all hosts specifed
       Example:
           ssh key list red[01-10]

    host key gather HOSTS FILE

      gathers all keys from file FILE including the one from localhost.

          ssh key gather "red[01-10]" keys.txt

    host key scatter HOSTS FILE

      copies all keys from file FILE to authorized_keys on all hosts,
      but also makes sure that the users ~/.ssh/id_rsa.pub key is in
      the file.

      1) adds ~/.id_rsa.pub to the FILE only if its not already in it
      2) removes all duplicated keys

      Example:
          ssh key scatter "red[01-10]"

    host key scp NAMES FILE

      copies all keys from file FILE to authorized_keys on all hosts
      but also makes sure that the users ~/.ssh/id_rsa.pub key is in
      the file and removes duplicates, e.g. it calls fix before upload

      Example:
          ssh key list red[01-10] > pubkeys.txt
          ssh key scp red[01-10] pubkeys.txt

    host config NAMES IPS [--user=USER] [--key=PUBLIC]

      generates an ssh config file tempalte that can be added to your
      .ssh/config file

      Example:
          cms host config "red,red[01-03]" "198.168.1.[1-4]" --user=pi

    host check NAMES [--user=USER] [--key=PUBLIC]

      This command is used to test if you can login to the specified
      hosts. It executes the hostname command and compares it.
      It provides a table  with a sucess column

      cms host check "red,red[01-03]"

          +-------+---------+--------+
          | host  | success | stdout |
          +-------+---------+--------+
          | red   | True    | red    |
          | red01 | True    | red01  |
          | red02 | True    | red02  |
          | red03 | True    | red03  |
          +-------+---------+--------+

    host tunnel create NAMES [--port=PORT]

      This command is used to create a persistent local port
      forward on the host to permit ssh tunnelling from the wlan to
      the physical network (eth). This registers an autossh service in
      systemd with the defualt port starting at 8001.

      Example:
          cms host tunnel create red00[1-3]

```
<!--MANUAL-HOST-->

### Manual Page for the `pi` command

Note to execute the command on the commandline you have to type in
`cms pi` and not jsut `pi`.

**Note**: Please note that the command `hadoop`, `spark`, and `k3` are
experimental and do not yet work.  In fact the hadoop and spark
deployment are not fullfilling our standard and should not be
used. They will be put into a different command soon so they are not
confusing the used in this README. The command is likely to be called
`pidev`. Once the command is graduated it will be moved into the main
command pi.

There is some very usefull aditional information about how to use the
LED and temperature monitoring programs at

* <https://github.com/cloudmesh/cloudmesh-pi-cluster/blob/main/README.md>

<!--MANUAL-PI-->
```
  pi led reset [NAMES]
  pi led (red|green) VALUE
  pi led (red|green) VALUE NAMES [--user=USER]
  pi led list NAMES [--user=USER]
  pi led blink (red|green) NAMES [--user=USER] [--rate=SECONDS]
  pi led sequence (red|green) NAMES [--user=USER] [--rate=SECONDS]
  pi temp NAMES [--rate=RATE] [--user=USER] [--output=FORMAT]
  pi free NAMES [--rate=RATE] [--user=USER] [--output=FORMAT]
  pi load NAMES [--rate=RATE] [--user=USER] [--output=FORMAT]
  pi script list SERVICE [--details]
  pi script list SERVICE NAMES
  pi script list
  pi wifi SSID [PASSWORD] [--dryrun]

This command does some useful things.

Arguments:
    FILE   a file name

Options:
    -f      specify the file

Description:

  This command switches on and off the LEDs of the specified PIs. If
  the hostname is ommitted. IT is assumed that the code is executed on
  a PI and its LED are set. To list the PIs LED status you can use the
  list command

  Examples:

      cms pi led list  "red,red[01-03]"

          lists the LED status of the given hosts

      cms pi led red off  "red,red[01-03]"

          switches off the led of the given PIs

      cms pi led red on  "red,red[01-03]"

          switches on the led of the given PIs

      cms pi led red blink  "red,red[01-03]"

          switches on and off the led of the given PIs

      cms pi led red sequence  "red,red[01-03]"

          goes in sequential order and switches on and off the led of
          the given PIs

```
<!--MANUAL-PI-->

## FAQ and Hints

Here, we provide some usefule FAQs and hints.

### Can I use the LEDs on the PI Motherboard?

Typically this LED is used to communicate some system related
information. However `cms pi` can controll it to switch status on
and off. This is helpful if you like to showcase a particular state
in the PI. Please look at the manual page.  For more options see the
manual page

### How can I use pycharm, to edit files or access files in general from my Laptop on the PI?

This is easily possible with the help of SSHFS. To install it we
refer you to See also: <https://github.com/libfuse/sshfs> SSHFS: add
master to `.ssh/config` onlocal machine

Let us assume you like to edit fles on a PI that you named `red`

Please craete a `./.ssh/config file that containes the following:

```
 Host red
      HostName xxx.xxx.xxx.xxx
      User pi
      IdentityFile ~/.ssh/id_rsa.pub
```

Now let us create a directory in which we mount the remote PI directories

```
mkdir master
sshfs master: master -o auto_cache
```

### How can I enhance the `get` script?

Instead of using the link

* <http://cloudmesh.github.io/get/pi>

please use

* <https://raw.githubusercontent.com/cloudmesh/get/main/pi>

This allows us to test also modifications to the get script before we
push them to the official community repository.

You can create a pull request at

* <https://github.com/cloudmesh/get/blob/main/pi/index.html>

### Can I use a Mesh Network for the setup?

This section is still under development.

In case you have a Mesh Network, the setup can typically be even more
simplifies as we can attach the unmanaged router directly to a Mesh
node via a network cable. IN that case the node is directly connected
to the internet and uses the DHCP feature from the Mesh router (see
Figure 2).

![](https://github.com/cloudmesh/cloudmesh-pi-burn/raw/main/images/network-mesh.png)

Figure 2: Networking with Mesh network

You will not need the bridge command to setup the network.

### Can I use cms burn on Linux?

Nlt everything is supported.

To download the latest Raspberry Pi OS Lite image use

```
cms burn image get latest-lite
```

To see what SDCard writers you have attached, you can use the command

```
cms burn info
```

It will issue a probe of USB devices and see if SDCards can be found.

Identify the `/dev/sdX`, where X is a letter such as b,c,d, ...

### What is the status of the implementation?

| Feature         | PI    | Ubuntu | Mac   | Windows |
| --------------- | ----- | ------ | ----- | ------- |
| image versions  |    +  |    +   |    +  |         |
| image ls        |    +  |    +   |    +  |         |
| image delete    |    +  |    +   |    +  |         |
| image get       |    +  |    +   |    +  |         |
| info            |    +  |    +   | g  +  |         |
| network         |    +  |    +   | TODO  |         |
| backup          |    +  |    +   |    -  |         |
| copy            |    +  |    +   |    -  |         |
| shrink install  |    +  |    +   | TODO  |         |
| shrink          |    +  |    +   | TODO  |         |
| sdcard          |    +  |    +   | TODO  |         |
| mount           |    +  |    +   | TODO  |         |
| unmount         |    +  |    +   | g  +  |         |
| enable ssh      |    +  |    +   | g  +  |         |
| wifi            |    +  |    +   | g  +  |         |
| set             |    +  |    +   | TODO1 |         |
| create          |  TODO |  TODO  | TODO  |         |
| check           |    +  |    +   | g  +  |         |
| format          |    +  |    +   | TODO  |         |
| firmware        | a  +  | NA     |  NA   | NA      |

* + verified throug unit test either by ANthony or Gregor

* TODO1 = todo for boot fs, rootfs not supported

* g = gregor
* r = richie
* a = anthony
* ad = adam
* as = asuri
* ar = arjun

* d = diffrent implementation between Linux and PI (compare)
* 2 = change and add --ssd so its uniform
* ? = needs test
* - = broken
1 = get needs to use the image versions refresh cache
3 = does not report when the USB card is found
t = has a unit test

### What packages do I need to run the info command on macOS

```
brew install libusb
``````

### Are there any unit tests?

As `cms burn` may delete format, delete, and remove files during unit
testing users are supposed to first review the tests before running
them. Please look at the source and see if you can run a test.

we have the following tests:

* `pytest -v --capture=no tests/test_01_image.py`
  * This test removes files forom ~/.cloudmesh/cmburn/images
  * See also:  [test_01_image.py](https://github.com/cloudmesh/cloudmesh-pi-burn/blob/main/tests/test_01_image.py)

repository: adamint/cloudmesh-pi-burn

repo_name: adamint/cloudmesh-pi-burn


text:
# sbt-ci-release

This is an sbt plugin to help automate publishing from Travis CI to Sonatype.

- tag pushes are published to Maven Central
- merge into master commits are published as -SNAPSHOT

Beware that publishing from Travis CI requires you to expose Sonatype
credentials as secret environment variables in Travis CI jobs. However, note
that secret environment variables are not accessible during pull requests.

Let's get started!

First, follow the instructions in
https://central.sonatype.org/pages/ossrh-guide.html to create a Sonatype account
and make sure you have publishing rights for a domain name.

Next, install this plugin in `project/plugins.sbt`

```scala
addSbtPlugin("com.geirsson" % "sbt-ci-release" % "1.0.0-M1")
```

By installing `sbt-ci-release` the following sbt plugins are also brought in:

- [sbt-dynver](https://github.com/dwijnand/sbt-dynver): sets the version number
  based on your git history
- [sbt-pgp](https://github.com/sbt/sbt-pgp): to cryptographically sign the
  artifacts before publishing
- [sbt-sonatype](https://github.com/xerial/sbt-sonatype): to publish artifacts
  to Sonatype
- [sbt-git](https://github.com/sbt/sbt-git): to automatically populate `scmInfo`

Next, define publishing settings at the top of `build.sbt`

```scala
inThisBuild(List(
  organization := "com.geirsson",
  homepage := Some(url("https://github.com/scalameta/sbt-scalafmt")),
  licenses := Seq("Apache-2.0" -> url("http://www.apache.org/licenses/LICENSE-2.0")),
  developers := List(
    Developer(
      "olafurpg",
      "Ólafur Páll Geirsson",
<EMAIL_ADDRESS>      url("https://geirsson.com")
    )
  )
))
```

Next, copy-paste the following to the bottom of `build.sbt`

```scala
inScope(Global)(Seq(
  PgpKeys.pgpPassphrase := sys.env.get("PGP_PASSPHRASE").map(_.toCharArray())
))
```

Next, create a fresh gpg key that you will share with Travis CI and only use for
this particular project.

```
gpg --gen-key
```

- For real name, use "$PROJECT_NAME bot". For example, in Scalafmt I use
  "Scalafmt bot"
- For email, use your own email address
- For passphrase, generate a random password with a password manager

At the end you'll see output like this

```
pub   rsa2048 2018-06-10 [SC] [expires: 2020-06-09]
      $LONG_ID
uid                      $PROJECT_NAME bot<EMAIL_ADDRESS>```

Take note of the id `$LONG_ID`, make sure to replace this ID from the code
examples below. The ID will look something like
`6E8ED79B03AD527F1B281169D28FC818985732D9`.

Next, copy the public gpg signature

```
# macOS
gpg --armor --export $LONG_ID | pbcopy
# linux
gpg --armor --export $LONG_ID | xclip
```

and post the signature to a keyserver: https://pgp.mit.edu/

![mit_pgp_key_server](https://user-images.githubusercontent.com/1408093/41208114-b8c89dce-6d1f-11e8-9280-9ab2b70bb0d7.jpg)

Next, open the "Settings" panel for your project on Travis CI, for example
https://travis-ci.org/scalameta/sbt-scalafmt/settings.

Define four secret variables

![](https://user-images.githubusercontent.com/1408093/41207402-bbb3970a-6d15-11e8-8772-000cc194ee92.png)

- `PGP_PASSPHRASE`: The randomly generated password you used to create a fresh
- `PGP_SECRET`: The base64 encoded secret of your private key that you can
  export from the command line like here below

```
# macOS
gpg --armor --export-secret-keys $LONG_ID | base64 | pbcopy
# linux
gpg --armor --export-secret-keys $LONG_ID | base64 | xclip
```

- `SONATYPE_PASSWORD`: The password you use to log into
  https://oss.sonatype.org/
- `SONATYPE_USERNAME`: The email you use to log into https://oss.sonatype.org/
  gpg key.

Next, update `.travis.yml` to trigger `ci-release` on successful merge into
master and on tag push. We use
[Travis "build stages"](https://docs.travis-ci.com/user/build-stages/) to
implement this. It's not necessary to use build stages. The reason we use build
stages is to prevent that `ci-release` runs multiple times in parallel jobs.

- define `test` and `release` build stages

```yml
stages:
  - name: test
  - name: release
    if: (branch = master AND type = push) OR (tag IS present)
```

- define your build matrix with `ci-release` at the bottom, for example:

```yml
jobs:
  include:
    # default stage is "test"
    - env: TEST="compile"
      script: sbt compile
    - env: TEST="formatting"
      script: ./bin/scalafmt --test
    # run ci-release only if all previous stages passed.
    - stage: release
      script: sbt ci-release
```

If we don't use Travis build stages but for example `after_success` instead, we
would run `ci-release` after both `TEST="formatting"` and `TEST="compile"`. As
long as you make sure you don't publish the same module multiple times, you can
use any Travis configuration you like.

We're all set! Time to manually try out the new setup

- Merge a PR to your project and watch the CI release a -SNAPSHOT for your
  project.
- Push a tag and watch the CI release a -SNAPSHOT for your project.

It's is normal that something fails on the first attempt to publish from CI.
Even if it takes 10 attempts to get it right, it's still worth it because it's
so nice to have automatic CI releases.


text:
## To Train
N.txt is used to track the number of trail. Please do not keep other .txt file in this folder.

        python3 run.py

## Hyper Parameters
modify config.py, (optionally) set sanity = True to train on only 10 data
* fc_scale  : dividing number of parameters in fully-connected layers
* conv_scale: dividing number of parameters in convolutional layers
* weights   : weighting factor for unbalanced dataset
* lr	    : learning rate
* batch size


text:
---
templateKey: blog-post
title: Wrong Question
date: 2019-06-17T11:51:27.223Z
description: |-
  [how to ask the right questions]
  [Photo by Lorca Wiles on Unsplash]
featuredpost: true
featuredimage: /img/dog_ask.jpg
tags:
  - '#questions'
---
If you're talking to somebody and they can't help you, you're asking the wrong questions of the wrong person. Or, simply asking the wrong questions.

Can you help me?

"Yes, I can help you."

Great, I want to exchange this product for a different one.

"Oh, we don't do exchanges."

I see. Thank you.

At this point, an experienced asker can detect the unwillingness to help immediately after the person said they could help.

They didn't even notice what they just did. That means low awareness. Low awareness people won't be resourceful.

A highly trained and highly resourced person doesn't respond with: "We can't do that." ever. The worst case scenario might be a slow down in the conversation and a response of something like. "Sir, I don't know how to do that, but my goal here is to make sure you leave feeling good about our organization. Would you give me a couple of minutes to get more information?"

This person has been trained to say, what Improv comedians use as their goto skill: Yes and.

Yes. and... means you don't ever say no. "No" or a variation of it is a deal killer. It stops the action. You affirm the offer and figure out a way to re-frame it so you can take action on it.

If you ever notice you are interacting with an un-resourceful person, you can help them. Help by asking more questions.

What is their experience?

What have they been able to do for others?

What is currently working?

The answers to these questions will help you become a better asker and you can quickly follow up with a better ask once you get a clue as to what might work.


text:
# Django example: Task form

This example shows:

- Usage of Django generic views
- Usage of Django generic forms
- Django with Bootstrap
- How to use pagination with generic views
- Default fixtures usage

## What does it do?

It will render a list of tasks. Some of them are provided as fixtures.

The list of tasks is provided as a Bootstrap table, and allows to add new tasks. If there are more than 5, they will be paginated.

You are allowed to create, edit or remove tasks.

## Usage

First of all, create the database. This will install the fixtures automatically:

    $ ./manage.py syncdb
    Creating tables ...
    Creating table django_admin_log
    Creating table auth_permission
    Creating table auth_group_permissions
    Creating table auth_group
    Creating table auth_user_groups
    Creating table auth_user_user_permissions
    Creating table auth_user
    Creating table django_content_type
    Creating table django_session
    Creating table todolist_task

    You just installed Django's auth system, which means you don't have any superusers defined.
    Would you like to create one now? (yes/no): no
    Installing custom SQL ...
    Installing indexes ...
    Installed 42 object(s) from 1 fixture(s)

The user admin/admin will be created and some example tasks too.

Then run the server:

    $ ./manage.py runserver 7500
    Validating models...

    0 errors found
    January 06, 2014 - 11:12:25
    Django version 1.6.1, using settings 'example.settings'
    Starting development server at http://<IP_ADDRESS>:7500/
    Quit the server with CONTROL-C.

And it will be listening on localhost:7500

## To be done

- Render Date fields as dates.
- Add tests.

repository: magmax/examples-django

repo_name: magmax/examples-django


text:
Making a Release
================

Follow semantic versioning and make sure the changelog is up-to-date.

For non prerelease level also update [README.md](README.md) and [CHANGELOG.md](CHANGELOG.md) and make a release on github including copying the relevant info from the changelog file there.

We use [Versioneer](https://github.com/warner/python-versioneer) to automatically update the version string (of a release but also in development). This means for a release a new git tag should be created. The tag should be of the form vX.Y or vX.Y.Z and generally follow [pep440](https://www.python.org/dev/peps/pep-0440/) with a prefixed "v".

```bash
git tag  -a vX.Y -m "version X.Y"
git push
git push origin --tags
python setup.py sdist upload -r pypi  # better use twine for uploading, see below
```

To ensure a secure upload use `twine`:
```bash
# Create some distributions in the normal way:
python setup.py sdist
# Upload with twine:
twine upload dist/*
```

Development
===========

Development of the MDsrv is coordinated through the repository on [github](http://github.com/arose/mdsrv). Please use the [issue tracker](https://github.com/arose/mdsrv/issues) there to report bugs or suggest improvements.

To participate in developing for the MDsrv you need a local copy of the source code, which you can obtain by forking the [repository](https://github.com/arose/mdsrv) on github. Read about how to [fork a repository](https://help.github.com/articles/fork-a-repo/), [sync a fork](https://help.github.com/articles/syncing-a-fork/) and [start a pull request](https://help.github.com/articles/using-pull-requests/).

repository: nglviewer/mdsrv

gha_language: Python

gha_license_id: NOASSERTION

repo_name: nglviewer/mdsrv


text:
# DeepFLR
DeepFLR is a deep learning-based framework that accurately predicts phosphopeptides spectra and effectively controls false localization rates in phosphoproteomics.

# Requirements
Model construction was performed using python (3.8.3, Anaconda distribution version 5.3.1, https://www.anaconda.com/) with the following packages: FastNLP (0.6.0), pytorch (1.8.1), transformers (4.12.5), bidict (0.22.0) and pyteomics (4.5.5).

Data analysis for FLR estimation was performed using python (3.8.3) with the following packages: pandas (1.0.5) and numpy (1.18.5).

Users can install packages using the command provided in the User Guide, or use the “pip install -r requirements.txt” command to install all the required packages.

# Tutorial
Tutorials are avaliable in [User guide.docx](https://github.com/yuz2011/DeepFLR/blob/main/User%20guide.docx).
# Model
Users can either use the model parameters used in DeepFLR or finetune the model or retrain a model following the  [User guide.docx](https://github.com/yuz2011/DeepFLR/blob/main/User%20guide.docx). The Model parameters "best__2deepchargeModelms2_bert_mediancos_2021-09-20-01-17-50-729399" (329 MB) can be downloaded from [Model](https://github.com/yuz2011/DeepFLR/releases/tag/Model) and put it in `phosT` folder.
# Publications
Waiting for publication...([paper]())

# License
DeepFLR is distributed under a BSD license. See the LICENSE file for details.

# Contacts
Please report any problems directly to the github issue tracker. Also, you can send feedback to<EMAIL_ADDRESS>

repository: lmsac/DeepFLR

repo_name: lmsac/DeepFLR


text:
---
layout: post
title: How many models do we need to predict 4 categories using Logistic Regression?
---

We use Logistic Regression (LR) to solve binary or 2-class classification problems. LR produces a single value between 0 to 1.

In order to find four different categories with the help of LR, we need to create four different models (i.e. One-vs-All technique). One model tries to predict Class 1 vs [Class 2, Class 3, Class 4]. Second model finds Class 2 vs [Class 1, Class 3, Class 4]. Third model is for Class 3 vs [Class 1, Class 2, Class 4] and the last model is to predict Class 4. Once we train four models, we can find the prediction result from the models with the highest score.


text:
# Java / Spring Demo of idQ Authentication

This repository demonstrates how to integrate idQ authentication into your Java / Spring web application.

## Prerequisites
1. You need to have an idQ Developer account (https://beta.idquanta.com)
2. Login to your Account Portal and issue OAuth2 credentials for the demo app. See <https://docs.idquanta.com> for instructions on issuing OAuth2 credentials.
3. Specify http://localhost:8080/redirect as the Callback URL when creating your OAuth2 credentials.

## Usage
1. Clone this repository
2. Configure your OAuth2 Credentials in `src/main/resources/oauth2.properties`

```
clientID = "YOUR_CLIENT_ID"
clientSecret = "YOUR_CLIENT_SECRET"
redirectUrl = "YOUR_REDIRECT_URI"
baseUrl = "idQ Trust as a Service OAuth 2.0 endpoint, eg. https://beta.idquanta.com/idqoauth"
```

3. Execute `mvn jetty:run`
4. Open <http://localhost:8080>

repository: inbaytech/example-java-spring

repo_name: inbaytech/example-java-spring


text:
---
layout: default
---

# 1996-11-29 - Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE

## Header Data

From<EMAIL_ADDRESS>(Anonymous)

To<EMAIL_ADDRESS>Message Hash: 710adb5e89cffa389ec8d19ab922d25463b4278df59d636f633887354658f8e2

Message ID<EMAIL_ADDRESS>Reply To<EMAIL_ADDRESS>UTC Datetime: 1996-11-29 01:02:48 UTC

Raw Date: Thu, 28 Nov 1996 17:02:48 -0800 (PST)

## Raw message

```
{% raw  %}From<EMAIL_ADDRESS>(Anonymous)
Date: Thu, 28 Nov 1996 17:02:48 -0800 (PST)
To<EMAIL_ADDRESS>Subject: Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE
In-Reply-To<EMAIL_ADDRESS>Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

> From: Ben Laurie<EMAIL_ADDRESS>>
> I think I would discuss this with the author before going public, to give
> him the usual opportunity to clean up before all hell breaks loose. However,
> that is what I'd call "work" rather than "fun", so I'd want paying for it.

Translation:  You don't really know what you are talking about.

> My impression is that Eric is more interested in speed and functionality than
> strict security (and considering the incredible vulnerability that is more or
> less inherent in an SSL implementation, I feel the same). I could be wrong, of
> course.

How is any security hole inherent in an SSL implementation?  The
protocol itself may not give you everything you need, but regardless
of whether or not the protocol is useable for any given task (or any
task at all), nothing precludes a secure implementation.

{% endraw %}
```

## Thread

+ Return to [November 1996](/archive/1996/11)

+ Return to "[Ben Laurie <ben@gonzo.ben.algroup.co.uk>](/authors/ben_laurie_ben_at_gonzo_ben_algroup_co_uk_)"
+ Return to "[geeman@best.com](/authors/geeman_at_best_com)"
+ Return to<EMAIL_ADDRESS>(Anonymous)](/authors/lucifer_at_dhp_com_anonymous_)"

+ 1996-11-28 (Thu, 28 Nov 1996 09:10:01 -0800 (PST)) - [Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE](/archive/1996/11/0632a88e866e10975bb06976812f044f53b27130ed5d6e5672362e3634999ebb) -<EMAIL_ADDRESS>  + 1996-11-28 (Thu, 28 Nov 1996 12:10:57 -0800 (PST)) - [Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE](/archive/1996/11/0ce292ad664fbad7dcf74e8c381cc15da7c40326e5fde2bda24bc38f1d518b91) - _Ben Laurie<EMAIL_ADDRESS>    + 1996-11-29 (Thu, 28 Nov 1996 17:02:48 -0800 (PST)) - Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE -<EMAIL_ADDRESS>(Anonymous)_
      + 1996-11-29 (Fri, 29 Nov 1996 01:52:55 -0800 (PST)) - [Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE](/archive/1996/11/29c8860fb0356dfbacd00151549adda07bc8b63b887462e5a15b35d32d878672) - _Ben Laurie<EMAIL_ADDRESS>      + 1996-11-29 (Fri, 29 Nov 1996 07:40:51 -0800 (PST)) - [Re: SAFEPASSAGE BRINGS STRONG CRYPTO TO WEB BROWSERS WORLDWIDE](/archive/1996/11/4d99f995778ac24c6e39968fae765c9cc673d39845b87e1ae2f98c7a66851796) - _Ben Laurie<EMAIL_ADDRESS>


text:
# `measure-export-cli`

> Command line for measure export

## Usage

```bash
npm i measure-export-cli -g
# Run the online server
measure-export start path/to/psdDir
# Build assets in $PWD/dist
measure-export build path/to/psdDir

measure-export --help
measure-export start --help
measure-export build --help
```

repository: dearwayne/measure

repo_name: dearwayne/measure


text:
# Java-Effekt
An Implementation of effect handlers using bytecode manipulation.

## Examples
- The world famous [Drunkflip](tests/src/test/java/run/amb/DrunkFlip.java) example.

An example of some instrumented bytecode can be found [in this gist](https://gist.github.com/b-studios/28b9ed229369b962e0083989343d5ede). The source-file is [Simple.java](tests/src/test/java/run/Simple.java).

## Project Organization
The project currently consists of the following subprojects:

- [core](core) The runtime system of jvm-effekt -- contains necessary runtime classes (like Stack).
- [instrumentation](instrumentation) The implementation of byte code instrumentation, depends on core.
- [sbtplugin](sbtplugin) An sbtplugin for offline instrumentation of classfiles.
- [tests](tests) A [separate](tests/build.sbt) project illustrating the use of Java Effekt.

## Dependencies
The project currently depends on version `2.0.1` of the static analysis and bytecode generation framework [OPAL](https://bitbucket.org/delors/opal/overview).

To run the tests, execute the following steps:
```
# If you haven't already, clone the repo
git clone git@github.com:b-studios/java-effekt.git
cd java-effekt

# clean all previous builds, if any to avoid sbt conflicts (only necessary once in a while ;) )
rm -r **/target
# Start sbt and publish the effekt instrumentation locally
sbt publishLocal

# now switch to the effekt core library and publish this locally
cd core
sbt publishLocal

# switch to tests subproject and run tests
cd ..
git clone git@github.com:b-studios/java-effekt-tests.git
cd java-effekt-tests
sbt
> test -- -oDF
```

Running the tests will instrument the class files and dump them for inspection under `target/instrumented`.
To inspect bytecode, I highly recommend the
[OPAL - Bytecode Disassembler Plugin](https://atom.io/packages/java-bytecode-disassembler) for Atom.

## Background
[Scala-Effekt](https://github.com/b-studios/scala-effekt) is a library based implementation of
effect handlers and uses a monad to implement multiprompt delimited continuations (which are
necessary for effect handlers).
Similarly, some experiments in Java show that this is in fact a viable approach. However,
the monadic style might impose usability as well as performance issues.

This project implements stack manipulation (a la [DelimCC](http://okmij.org/ftp/continuations/implementations.html)) on the JVM by
only using the JVM stack for pure (non effectful) operations. For effectful operations, instead a
user-level [`Stack`](core/src/main/java/effekt/Effekt.java)
is used. Before each effectful operation, the continuation is pushed as a first-class frame to this stack.

These first-class frames are instances of [`Frame`](core/src/main/java/effekt/runtime/Frame.java)
and exactly represent the continuation after the effectful call. The method state is stored in the closure of the frame and restored before the original body is executed.

This is very similar to the bytecode instrumentation performed by [Quasar](http://docs.paralleluniverse.co/quasar/). In particular,
we also use Java checked exceptions to mark "effectful" functions.
However, Quasar does not have a notion of first-class frames and still uses the JVM-stack for effectful functions. Thus resuming a
continuation takes O(n) in the depth of the JVM stack.

repository: b-studios/java-effekt

gha_language: Scala

repo_name: b-studios/java-effekt


text:
# Arc

![Arc layer](https://d1a3f4spazzrp4.cloudfront.net/kepler.gl/documentation/image37.png "Arc layer")

Arc layers draw an arc between two points. They’re useful for visualizing the distance between two points as well as comparing distances in 3D. Note that arc layers don’t show routes between points, but simply the distance between the two points. The tallest arc represents the greatest distance.

To draw arcs, your dataset must contain the latitude and longitude of two different points for each arc.

[Back to table of contents](../README.md)

gha_language: TypeScript