text:
# :octocat: 🕷 🕸 GitHub Scraper

Learn how to parse the DOM of a web page
by using your favourite coding community as an example.

[![Build Status](https://img.shields.io/travis/nelsonic/github-scraper/master.svg?style=flat-square)](https://travis-ci.org/nelsonic/github-scraper)
[![codecov.io](https://img.shields.io/codecov/c/github/nelsonic/github-scraper/master.svg?style=flat-square)](http://codecov.io/github/nelsonic/github-scraper?branch=master)
[![Dependencies: None!](https://david-dm.org/nelsonic/github-scraper/status.svg?style=flat-square)](https://david-dm.org/nelsonic/github-scraper)
[![devDependencies Status](https://david-dm.org/nelsonic/github-scraper/dev-status.svg?style=flat-square)](https://david-dm.org/nelsonic/github-scraper?type=dev)
[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat-square)](https://github.com/nelsonic/github-scraper/issues)
[![HitCount](http://hits.dwyl.io/nelsonic/github-scraper.svg)](http://hits.dwyl.io/nelsonic/github-scraper)

# ⚠  Disclaimer / Warning!

This repository/project is intended for
***Educational Purposes*** **ONLY**.

The project and corresponding NPM module should not
be used for any purpose other than *learning*.
Please do not use it for any other reason
than to learn _about_ DOM parsing
and _definitely_ don't _depend_ on it for anything important!

The nature of DOM parsing is that when the HTML/UI changes,
the parser will inevitably fail ...
GitHub have every right to change/improve their UI as they see fit.
When they do change their UI the scraper will _inevitably_ "_break_"!
We have Travis-CI tests precisely
to _check_ that parsers for the various pages are working.
You can run the tests locally too,
see "Run The Tests" section below.

## Why?

Our _initial reason_ for writing this set of scrapers was to satisfy the _curiosity_ / _question_:
> _How_ can we ***discover*** which are the ***interesting people and projects
on GitHub***
(_without **manually** checking *dozens* of GitHub profiles/repositories each day_) ?

Our _second reason_ for scraping data from GitHub is so that we can show people a "*summary view*" of all their issues in our [Tudo](https://github.com/dwyl/tudo) project (which helps people track/manage/organise/prioritise their GitHub issues).
See: https://github.com/dwyl/tudo/issues/51

We needed a _simple_ way of systematically getting data from GitHub (_before people authenticate_) and scraping is the only way we could think of.

We _tried_ using the [GitHub ***API***](https://developer.github.com/v3/)
to get records from GitHub, but sadly,
it has quite a few limitations (see: "_Issues with GitHub API_" section below) the biggest limitation being the [_rate-limiting_](https://developer.github.com/v3/#rate-limiting) on API requests.

Thirdly we're building this project to [***scratch our own itch***](https://gettingreal.37signals.com/ch02_Whats_Your_Problem.php)
... scraping the _pages_ of GitHub has given us a _unique_ insight into the features of the platform which has leveled-up our skills.

> Don't *you* want to know ***what's "Hot" right now on GitHub***...?

## What (*Problem* are we _trying_ to Solve)?

Having a way of extracting the *essential* data from GitHub
is a solution to a _surprisingly **wide array of problems**_, here are a few:

+ ***Who*** are the up-and-comming people (_worth following_) on GitHub?
+ ***Which*** are the ***interesting projects*** (*and why?!*)
+ ***What*** is the average age of an issue for a project?
+ Is a project's ***popularity growing*** or *plateaued*?
+ Are there (_already_) any ***similar projects*** to what I'm trying to build? (_reduce duplication of effort which is rampant in Open Source!!_)
+ How many projects get started but never finished?
+ ***Will*** my **Pull Request** *ever* get *merged* or is the module maintainer *too busy* and did I just [***waste 3 hours***](https://twitter.com/nelsonic/status/621984170353524736)?
+ _insert **your idea/problem** here_ ...
+ **Associative Lists** e.g: People who starred `abc` also liked `xyz`

# How?

This module fetches (_public_) pages from GitHub, "[_scrapes_](https://en.wikipedia.org/wiki/Web_scraping)" the html to extract raw data and returns a JSON Object.

# Usage

## install from NPM

install from npm and save to your `package.json`:

```sh
npm install github-scraper --save
```

## Use it in your script!

```js
var gs = require('github-scraper');
var url = '/iteles' // a random username
gs(url, function(err, data) {
  console.log(data); // or what ever you want to do with the data
})
```

## Example URLs and Output

### Profile Page

User profile has the following format `https://github.com/{username}`
example: [https://github.com/**iteles**](https://github.com/iteles)

![iteles-github-profile-april-2019-annotated](https://user-images.githubusercontent.com/194400/56076833-3deafd00-5dcd-11e9-87b0-693341a0ff64.png)

```js
var gs = require('github-scraper'); // require the module
var url = 'alanshaw' // a random username (of someone you should follow!)
gs(url, function(err, data) {
  console.log(data); // or what ever you want to do with the data
})
```

Sample output:

```json
{
  "type": "profile",
  "url": "/iteles",
  "avatar": "https://avatars1.githubusercontent.com/u/4185328?s=400&v=4",
  "name": "Ines Teles Correia",
  "username": "iteles",
  "bio": "Co-founder @dwyl | Head cheerleader @foundersandcoders",
  "uid": 4185328,
  "worksfor": "@dwyl",
  "location": "London, UK",
  "website": "http://www.twitter.com/iteles",
  "orgs": {
    "bowlingjs": "https://avatars3.githubusercontent.com/u/8825909?s=70&v=4",
    "foundersandcoders": "https://avatars3.githubusercontent.com/u/9970257?s=70&v=4",
    "docdis": "https://avatars0.githubusercontent.com/u/10836426?s=70&v=4",
    "dwyl": "https://avatars2.githubusercontent.com/u/11708465?s=70&v=4",
    "ladiesofcode": "https://avatars0.githubusercontent.com/u/16606192?s=70&v=4",
    "TheScienceMuseum": "https://avatars0.githubusercontent.com/u/16609662?s=70&v=4",
    "SafeLives": "https://avatars2.githubusercontent.com/u/20841400?s=70&v=4"
  },
  "repos": 28,
  "projects": 0,
  "stars": 453,
  "followers": 341,
  "following": 75,
  "pinned": [
    { "url": "/dwyl/start-here" },
    { "url": "/dwyl/learn-tdd" },
    { "url": "/dwyl/learn-elm-architecture-in-javascript" },
    { "url": "/dwyl/tachyons-bootstrap" },
    { "url": "/dwyl/learn-ab-and-multivariate-testing" },
    { "url": "/dwyl/learn-elixir" }
  ],
  "contribs": 878,
  "contrib_matrix": {
    "2018-04-08": { "fill": "#c6e48b", "count": 1, "x": "13", "y": "0" },
    "2018-04-09": { "fill": "#c6e48b", "count": 2, "x": "13", "y": "12" },
    "2018-04-10": { "fill": "#7bc96f", "count": 3, "x": "13", "y": "24" },
    ...etc...
    "2019-04-11": { "fill": "#c6e48b", "count": 1, "x": "-39", "y": "48" },
    "2019-04-12": { "fill": "#7bc96f", "count": 5, "x": "-39", "y": "60"}
  }
}
```

### Followers

How many people are following a given person on Github.
Url format: `https://github.com/{username}/followers`
example: [https://github.com/iteles/**followers**](https://github.com/iteles/followers)

```js
var gs = require('github-scraper'); // require the module
var url = 'iteles/followers' // a random username (of someone you should follow!)
gs(url, function(err, data) {
  console.log(data); // or what ever you want to do with the data
})
```

Sample output:

```js
{ entries:
   [ 'tunnckoCore', 'OguzhanE', 'minaorangina', 'Jasonspd', 'muntasirsyed', 'fmoliveira', 'nofootnotes',
    'SimonLab', 'Danwhy', 'kbocz', 'cusspvz', 'RabeaGleissner', 'beejhuff', 'heron2014', 'joshpitzalis',
    'rub1e', 'nikhilaravi', 'msmichellegar', 'anthonybrown', 'miglen', 'shterev', 'NataliaLKB',
    'ricardofbarros', 'boymanjor', 'asimjaved', 'amilvasishtha', 'Subhan786', 'Neats29', 'lottie-em',
    'rorysedgwick', 'izaakrogan', 'oluoluoxenfree', 'markwilliamfirth', 'bmordan', 'nodeco', 'besarthoxhaj',
    'FilWisher', 'maryams', 'sofer', 'joaquimserafim', 'vs4vijay', 'intool', 'edwardcodes', 'hyprstack',
    'nelsonic' ],
  url: 'https://github.com/iteles/followers' }
ok 1 iteles/followers count: 45
```

If the person has ***more than 51 followers*** they will have multiple pages of followers.
The data will have a **next_page** key with a value such as:
[/nelsonic/followers?**page=2**](https://github.com/nelsonic/followers?page=2)
If you want to keep fetching these subsequent pages of followers,
simply keep running the scraper:
e.g:

```js
var url = 'alanshaw/followers' // a random username (of someone you should follow!)
gs(url, function(err, data) {
  console.log(data); // or what ever you want to do with the data
  if(data.next_page) {
    gs(data.next_page, function(err2, data2) {
      console.log(data2); // etc.
    })
  }
})
```

### **Following**
Want to know the list of people this person is `following` that's *easy* too!
The url format is: `https://github.com/{username}/following`
e.g: [https://github.com/iteles/**following**](https://github.com/iteles/following) or
[https://github.com/nelsonic/following?**page=2**](https://github.com/nelsonic/following?page=2)
(_where the person is following more than 51 people_ ...)

Usage format is *identical* to `followers` (above) so here's an example
of fetching page 3 of the results:

```js
var gs = require('github-scraper'); // require the module
var url = 'nelsonic/following?page=3' // a random dude
gs(url, function(err, data) {
  console.log(data); // or what ever you want to do with the data
})
```

Sample output:

```js
{
  entries:
   [ 'kytwb', 'dexda', 'arrival', 'jinnjuice', 'slattery', 'unixarcade', 'a-c-m', 'krosti',
   'simonmcmanus', 'jupiter', 'capaj', 'cowenld', 'FilWisher', 'tsop14', 'NataliaLKB',
   'izaakrogan', 'lynnaloo', 'nvcexploder', 'cwaring', 'missinglink', 'alanshaw', 'olizilla',
   'tancredi', 'Ericat', 'pgte' 'hyprstack', 'iteles' ],
  url: 'https://github.com/nelsonic/following?page=3',
  next_page: 'https://github.com/nelsonic/following?page=4'
}
```

### Starred Repositories

The list of projects a person has *starred* a fascinating source of insight.
url format: https://github.com/stars/{username}
e.g: [/stars/iteles](https://github.com/stars/iteles)

```js
var gs = require('github-scraper'); // require the module
var url = 'stars/iteles';           // starred repos for this user
gs(url, function(err, data) {
  console.log(data);                // or what ever you want to do with the data
})
```

Sample output:

```js
{
  entries:
   [ '/dwyl/repo-badges', '/nelsonic/learn-testling', '/joshpitzalis/testing', '/gmarena/gmarena.github.io',
    '/dwyl/alc', '/nikhilaravi/fac5-frontend', '/foundersandcoders/dossier', '/nelsonic/health', '/dwyl/alvo',
    '/marmelab/gremlins.js', '/docdis/learn-saucelabs', '/rogerdudler/git-guide', '/tableflip/guvnor',
    '/dwyl/learn-redis', '/foundersandcoders/playbook', '/MIJOTHY/FOR_FLUX_SAKE', '/NataliaLKB/learn-git-basics',
    '/nelsonic/liso', '/dwyl/learn-json-web-tokens', '/dwyl/hapi-auth-jwt2', '/dwyl/start-here',
    '/arvida/emoji-cheat-sheet.com', '/dwyl/time', '/docdis/learn-react', '/dwyl/esta', '/alanshaw/meteor-foam',
    '/alanshaw/stylist', '/meteor-velocity/velocity', '/0nn0/terminal-mac-cheatsheet',
    '/bowlingjs/bowlingjs.github.io' ],
  url: 'https://github.com/stars/iteles?direction=desc&page=2&sort=created',
  next_page: 'https://github.com/stars/iteles?direction=desc&page=3&sort=created'
}
```

### Repositories

The second tab on the personal profile page is "Repositories"
this is a **list** of the ***personal projects*** the person is working on, e.g: https://github.com/iteles?tab=repositories

We crawl this page and return an array containing the repo properties:

```js
var url = 'iteles?tab=repositories';
gs(url, function(err, data) {
  console.log(data);  // or what ever you want to do with the data
})
```

sample output:

```js
{
  entries: [
    { url: '/iteles/learn-ab-and-multivariate-testing',
      name: 'learn-ab-and-multivariate-testing',
      lang: '',
      desc: 'Tutorial on A/B and multivariate testing',
      info: '',
      stars: '4',
      forks: '0',
      updated: '2015-07-08T08:36:37Z' },
    { url: '/iteles/learn-tdd',
      name: 'learn-tdd',
      lang: 'JavaScript',
      desc: 'A brief introduction to Test Driven Development (TDD) in JavaScript',
      info: 'forked from dwyl/learn-tdd',
      stars: '0',
      forks: '4',
      updated: '2015-06-29T17:24:56Z' },
    { url: '/iteles/practical-full-stack-testing',
      name: 'practical-full-stack-testing',
      lang: 'HTML',
      desc: 'A fork of @nelsonic\'s repo to allow for PRs',
      info: 'forked from nelsonic/practical-js-tdd',
      stars: '0',
      forks: '36',
      updated: '2015-06-06T14:40:43Z' },
    { url: '/iteles/styling-for-accessibility',
      name: 'styling-for-accessibility',
      lang: '',
      desc: 'A collection of \'do\'s and \'don\'t\'s of CSS to ensure accessibility',
      info: '',
      stars: '0',
      forks: '0',
      updated: '2015-05-26T11:06:28Z' },
    { url: '/iteles/Ultimate-guide-to-successful-meetups',
      name: 'Ultimate-guide-to-successful-meetups',
      lang: '',
      desc: 'The ultimate guide to organizing successful meetups',
      info: '',
      stars: '3',
      forks: '0',
      updated: '2015-05-19T09:40:39Z' },
    { url: '/iteles/Javascript-the-Good-Parts-notes',
      name: 'Javascript-the-Good-Parts-notes',
      lang: '',
      desc: 'Notes on the seminal "Javascript the Good Parts: byDouglas Crockford',
      info: '',
      stars: '41',
      forks: '12',
      updated: '2015-05-17T16:39:35Z' }
  ],
  url: 'https://github.com/iteles?tab=repositories' }
```

### Activity feed

Every person on GitHub has an RSS feed for their recent activity;
this is the 3rd and final tab of the person's profile page.

it can be viewed online by visiting:
```sh
https://github.com/{username}?tab=activity
```
e.g: [/iteles?tab=activity](https://github.com/iteles?tab=activity)

#### Parsing the Feed

The activity feed is published as an [**.atom**](https://en.wikipedia.org/wiki/RSS)
xml string which contains a list of entries.

We use [**xml2js**](https://www.npmjs.com/package/xml2js)
(which in turn uses the [**sax**](https://www.npmjs.com/package/sax) xml parser) to parse the xml stream. This results in a object similar to the following example:

```js
{ '$':
   { xmlns: 'http://www.w3.org/2005/Atom',
     'xmlns:media': 'http://search.yahoo.com/mrss/',
     'xml:lang': 'en-US' },
  id: [ 'tag:github.com,2008:/iteles' ],
  link: [ { '$': [Object] }, { '$': [Object] } ],
  title: [ 'iteles’s Activity' ],
  updated: [ '2015-07-22T23:31:25Z' ],
  entry:
   [ { id: [Object],
       published: [Object],
       updated: [Object],
       link: [Object],
       title: [Object],
       author: [Object],
       'media:thumbnail': [Object],
       content: [Object] },
     { id: [Object],
       published: [Object],
       updated: [Object],
       link: [Object],
       title: [Object],
       author: [Object],
       'media:thumbnail': [Object],
       content: [Object] }
    ]
}
```
Each call to the atom feed returns the latest 30 enties.
We're showing 2 here for illustration (_so you get the idea..._)

From this we _extract_ only the relevant info:

```sh
'2015-07-22T12:33:14Z alanshaw pushed to master at alanshaw/david-www',
'2015-07-22T12:33:14Z alanshaw created tag v9.4.3 at alanshaw/david-www',
'2015-07-22T09:23:28Z alanshaw closed issue tableflip/i18n-browserify#6',
'2015-07-21T17:08:19Z alanshaw commented on issue alanshaw/david#71',
'2015-07-21T08:24:13Z alanshaw pushed to master at tableflip/score-board',
'2015-07-20T17:49:59Z alanshaw deleted branch refactor-corp-events at tableflip/sow-api-client',
'2015-07-20T17:49:58Z alanshaw pushed to master at tableflip/sow-api-client',
'2015-07-20T17:49:58Z alanshaw merged pull request tableflip/sow-api-client#2',
'2015-07-20T17:49:54Z alanshaw opened pull request tableflip/sow-api-client#2',
'2015-07-18T07:30:36Z alanshaw closed issue alanshaw/md-tokenizer#1',
'2015-07-18T07:30:36Z alanshaw commented on issue alanshaw/md-tokenizer#1',
```
Instead of _wasting_ (_what will be **Giga**_) ***Bytes*** of space with key:value pairs by storing the entries as JSON, we are storing the activity feed entries as strings in an array.
Each item in the array can be broken down into:
```sh
{date-time} {username} {action} {link}
```

As we can see from this there are several event types:

+ **pushed to master** at
+ **created tag** v9.4.3 at
+ **opened issue**
+ **commented on issue**
+ **closed issue**
+ **deleted branch**
+ **opened pull request**
+ **merged pull request**
+ **starred** username/repo-name

For now we are *not* going to parse the event types, we are simply going to store them in our list for later analysis.

We have a good pointer when its time to start interpreting the data:
https://developer.github.com/v3/activity/events/types/

One thing worth noting is that RSS feed is ***Not Real-Time*** ...
sadly, it only gets updated periodically so we cannot rely on it to
have the *latest* info.

### Organization

Organization pages have the following url pattern: `https://github.com/{orgname}`
example: [https://github.com/**dwyl**](https://github.com/dwyl)

```js
var url = 'dwyl';
gs(url, function(err, data) {
  console.log(data); // or do something way more interesting with the data!
});
```

sample data (`entries` _truncated for brevity_):
```js
{
  entries:
   [ { name: 'hapi-auth-jwt2',
       desc: 'Secure Hapi.js authentication plugin using JSON Web Tokens (JWT)',
       updated: '2015-08-04T19:30:50Z',
       lang: 'JavaScript',
       stars: '59',
       forks: '11' },
     { name: 'start-here',
       desc: 'A Quick-start Guide for People who want to DWYL',
       updated: '2015-08-03T11:04:14Z',
       lang: 'HTML',
       stars: '14',
       forks: '9' },
     { name: 'summer-2015',
       desc: 'Probably the best Summer Sun, Fun & Coding Experience in the World!',
       updated: '2015-07-31T11:02:29Z',
       lang: 'CSS',
       stars: '16',
       forks: '1' },
  ],
  website: 'http://dwyl.io',
  url: 'https://github.com/dwyl',
  name: 'dwyl - do what you love',
  desc: 'Start here: https://github.com/dwyl/start-here',
  location: 'Your Pocket',
  email<EMAIL_ADDRESS>  pcount: 24,
  avatar: 'https://avatars3.githubusercontent.com/u/11708465?v=3&s=200',
  next_page: '/dwyl?page=2'
}
```
Note #1: *sadly*, this has the ***identical*** url format to *Profile*
this gets handled by the `switcher` which infers what is an org vs. profile page
by checking for an known element on the page.

Note #2: when an organization has *multiple pages* of repositories you will see a `next_page`
key/value in the `data` e.g: [/dwyl?**page=2**](/dwyl?page=2) (for the second page of repos)

### Repository Stats

This is where things start getting interesting ...

![github-repo-page](https://cloud.githubusercontent.com/assets/194400/8930109/d8a76ab8-3522-11e5-8e07-95596a889fde.png)

example: https://github.com/nelsonic/adoro

```js
var url = 'nelsonic/adoro';
gs(url, function(err, data) {
  console.log(data); // or do something way more interesting with the data!
});
```

sample data:

```js
{
  url: 'https://github.com/nelsonic/adoro',
  desc: 'The little publishing tool you\'ll love using. [work-in-progress]',
  website: 'http://www.dwyl.io/',
  watchers: 3,
  stars: 8,
  forks: 1,
  commits: 12,
  branches: 1,
  releases: 1,
  langs: [ 'JavaScript 90.7%', 'CSS 9.3%' ]
}
```

> Annoyingly the number of issues and pull requests, contributors and issues
 are only rendered *after* the page has loaded (via XHR) so we do not get
 these three stats on page load.

 ### 7. Issues

 Clicking on the issues icon/link in any repository takes us to the list of all the issues.

 Visiting a project with more than a page worth of issues has pagination at the bottom of the page:

 ![tudo-issues-list-showing-pagination](https://cloud.githubusercontent.com/assets/194400/8942419/27b9446a-356d-11e5-84f9-5de2eaae506b.png)

 Which has a link to: https://github.com/dwyl/tudo/issues?page=2&q=is%3Aissue+is%3Aopen

 ![tudo-issues-second-page](https://cloud.githubusercontent.com/assets/194400/8942423/33bf0a2e-356d-11e5-82b8-1bd142fb2302.png)

 List of issues for a repository:

 ```js
 var gs  = require('github-scraper');
 var url = '/dwyl/tudo/issues';
 gs(url, function (err, data) {
   console.log(data); // use the data how ever you like
 });
 ```

 sample output:

 ```sh
 { entries:
    [
      {
        url: '/dwyl/tudo/issues/46',
        title: 'discuss components',
        created: '2015-07-21T15:34:22Z',
        author: 'benjaminlees',
        comments: 3,
        assignee: 'izaakrogan',
        milestone: 'I don\'t know what I\'m doing',
        labels: [ 'enhancement', 'help wanted', 'question' ]
      },
      {
        url: '/dwyl/tudo/issues/45',
        title: 'Create riot components from HTML structure files',
        created: '2015-07-21T15:24:58Z',
        author: 'msmichellegar',
        comments: 2,
        assignee: 'msmichellegar',
        labels: [ 'question' ]
      }
   ], // truncated for brevity
   open: 30,
   closed: 20,
   next: '/dwyl/tudo/issues?page=2&q=is%3Aissue+is%3Aopen',
   url: '/dwyl/tudo/issues'
 }
 ```

 Each issue in the list would create a entry in the crawler (worker) queue:

 ```sh
 2015-07-22T12:33:14Z issue /dwyl/tudo/issues/77
 ```

 > Should we include the "all issues by this author" link?
 + **created_by** https://github.com/dwyl/tudo/issues/created_by/iteles
 + **assignee** (assigned to): https://github.com/dwyl/tudo/issues?q=assignee%3Aiteles+is%3Aopen

 ### Issue (_individual_)

 The result of scraping https://github.com/dwyl/tudo/issues/51

 ```js
 var gs  = require('github-scraper');
 var url = '/dwyl/tudo/issues/51';
 gs(url, function (err, data) {
   console.log(data); // use the data how ever you like
 });
 ```

 sample output:

 ```js
 { entries:
    [ { id: 'issue-96442793',
        author: 'nelsonic',
        created: '2015-07-22T00:00:45Z',
        body: 'instead of waiting for people to perform the steps to authorise Tudo (to access their GitHub orgs/issues we could request their GitHub username on the login page and initiate the retrieval of their issues while they are authenticating... That way, by the time they get back to Tudo their issues dashboard is already pre-rendered and loaded! This is a wow-factor people won\'t be expecting and thus our app immediately delivers on our first promise!\n\nThoughts?' },
      { id: 'issuecomment-123807796',
        author: 'iteles',
        created: '2015-07-22T17:54:12Z',
        body: 'I\'d love to test this out, this will be an amazing selling point if we can get the performance to work like we expect!' },
      { id: 'issuecomment-124048121',
        author: 'nelsonic',
        created: '2015-07-23T10:20:15Z',
        body: '@iteles have you watched the Foundation Episode featuring Kevin Systrom (instagram) ?\n\n\nhttps://www.youtube.com/watch?v=nld8B9l1aRE\n\n\nWhat were the USPs that contributed to instagram\'s success (considering how many photo-related-apps were in the app store at the time) ?\n\ncc: @besarthoxhaj' },
      { id: 'issuecomment-124075792',
        author: 'besarthoxhaj',
        created: '2015-07-23T11:59:31Z',
        body: '@nelsonic love the idea! Let\'s do it!' } ],
   labels: [ 'enhancement', 'help wanted', 'question' ],
   participants: [ 'nelsonic', 'iteles', 'besarthoxhaj' ],
   url: '/dwyl/tudo/issues/51',
   title: 'Pre-fetch people\'s issues while they are authenticating with GitHub',
   state: 'Open',
   author: 'nelsonic',
   created: '2015-07-22T00:00:45Z',
   milestone: 'Minimal Usable Product',
   assignee: 'besarthoxhaj' }
 ```

 By contrast using the GitHub API to fetch this issue
 see: https://developer.github.com/v3/issues/#get-a-single-issue

 format:
 ```sh
 /repos/:owner/:repo/issues/:number
 ```

 ```sh
 curl https://api.github.com/repos/dwyl/tudo/issues/51
 ```

 ### Milestones

 Milestones are used to group issues into logical units.

 ![dwyl-tudo-milestones](https://cloud.githubusercontent.com/assets/194400/9010055/b3e4da72-379c-11e5-8fd3-680bf928a389.png)

 ```js
 var gs  = require('github-scraper');
 var url = '/dwyl/tudo/milestones';
 gs(url, function (err, data) {
   console.log(data); // use the data how ever you like
 });
 ```

Sample output:

 ```js
 { entries:
    [ { name: 'Test Milestone - Please Don\'t Close!',
        due: 'Past due by 16 days',
        updated: 'Last updated 5 days ago',
        desc: 'This Milestone in used in our e2e tests to check for an over-due milestone, so please don\'t close it!',
        progress: '0%',
        open: 1,
        closed: 0 },
      { name: 'Minimal Usable Product',
        due: 'Due by July  5, 2016',
        updated: 'Last updated 2 days ago',
        desc: 'What is the absolute minimum we can do to deliver value to people using the app?\n(and thus make them want to come back and use it!)',
        progress: '0%',
        open: 5,
        closed: 0 } ],
   url: 'https://github.com/dwyl/tudo/milestones',
   open: 2,
   closed: 1 }
 ```

 ### Labels (for a repository)

 All repositories have a set of standard labels (built-in to GitHub)
 e.g: https://github.com/dwyl/tudo/labels is (_currently_) only using the "*standard*" labels.

 Whereas the RethinkDB (which uses GitHub for all their project tracking) uses _several **custom labels**_:
 https://github.com/rethinkdb/rethinkdb/labels

 We need to crawl these for each repo.

```js
var gs  = require('github-scraper');
var url = '/dwyl/time/labels';
gs(url, function (err, data) {
  console.log(data); // use the data how ever you like
});
```

 Here's the extraction of the standard labels:
 ```js
 [
   { name: 'bug',
     style: 'background-color: #fc2929; color: #fff;',
     link: '/dwyl/tudo/labels/bug',
     count: 3 },
   { name: 'duplicate',
     style: 'background-color: #cccccc; color: #333333;',
     link: '/dwyl/tudo/labels/duplicate',
     count: 0 },
   { name: 'enhancement',
     style: 'background-color: #84b6eb; color: #1c2733;',
     link: '/dwyl/tudo/labels/enhancement',
     count: 11 },
   { name: 'help wanted',
     style: 'background-color: #159818; color: #fff;',
     link: '/dwyl/tudo/labels/help%20wanted',
     count: 21 },
   { name: 'invalid',
     style: 'background-color: #e6e6e6; color: #333333;',
     link: '/dwyl/tudo/labels/invalid',
     count: 1 },
   { name: 'question',
     style: 'background-color: #cc317c; color: #fff;',
     link: '/dwyl/tudo/labels/question',
     count: 10 }
 ]
 ```

 or a repo that has ***custom labels***:

 ```js
 { entries:
   [ { name: '[alpha]',
       style: 'background-color: #79CDCD; color: #1e3333;',
       link: '/dwyl/time/labels/%5Balpha%5D',
       count: 2 },
     { name: 'API',
       style: 'background-color: #006b75; color: #fff;',
       link: '/dwyl/time/labels/API',
       count: 11 },
     { name: 'bug',
       style: 'background-color: #fc2929; color: #fff;',
       link: '/dwyl/time/labels/bug',
       count: 5 },
     { name: 'chore',
       style: 'background-color: #e11d21; color: #fff;',
       link: '/dwyl/time/labels/chore',
       count: 9 },
     { name: 'discuss',
       style: 'background-color: #bfe5bf; color: #2a332a;',
       link: '/dwyl/time/labels/discuss',
       count: 43 },
     { name: 'Documentation',
       style: 'background-color: #eb6420; color: #fff;',
       link: '/dwyl/time/labels/Documentation',
       count: 2 },
     { name: 'duplicate',
       style: 'background-color: #cccccc; color: #333333;',
       link: '/dwyl/time/labels/duplicate',
       count: 0 },
     { name: 'enhancement',
       style: 'background-color: #84b6eb; color: #1c2733;',
       link: '/dwyl/time/labels/enhancement',
       count: 27 },
     { name: 'external dependency',
       style: 'background-color: #D1EEEE; color: #2c3333;',
       link: '/dwyl/time/labels/external%20dependency',
       count: 1 },
     { name: 'FrontEnd',
       style: 'background-color: #f7c6c7; color: #332829;',
       link: '/dwyl/time/labels/FrontEnd',
       count: 26 },
     { name: 'help wanted',
       style: 'background-color: #009800; color: #fff;',
       link: '/dwyl/time/labels/help%20wanted',
       count: 42 },
     { name: 'invalid',
       style: 'background-color: #e6e6e6; color: #333333;',
       link: '/dwyl/time/labels/invalid',
       count: 0 },
     { name: 'investigate',
       style: 'background-color: #fbca04; color: #332900;',
       link: '/dwyl/time/labels/investigate',
       count: 18 },
     { name: 'MVP',
       style: 'background-color: #207de5; color: #fff;',
       link: '/dwyl/time/labels/MVP',
       count: 27 },
     { name: 'NiceToHave',
       style: 'background-color: #fbca04; color: #332900;',
       link: '/dwyl/time/labels/NiceToHave',
       count: 7 },
     { name: 'Post MVP',
       style: 'background-color: #fef2c0; color: #333026;',
       link: '/dwyl/time/labels/Post%20MVP',
       count: 24 },
     { name: 'question',
       style: 'background-color: #cc317c; color: #fff;',
       link: '/dwyl/time/labels/question',
       count: 25 },
     { name: 'UI',
       style: 'background-color: #bfdadc; color: #2c3233;',
       link: '/dwyl/time/labels/UI',
       count: 13 } ],
  url: 'https://github.com/dwyl/time/labels' }
 ```

 ### Issues > *Search* (*Bonus Feature*)

 A ***much*** more *effective* way of collating all the issues relevant to a person is to search for them!

 example:
  https://github.com/search?type=Issues&q=author%3Aiteles&state=open&o=desc&s=created

 ```js
 {
   entries:
    [
      { title: 'Remove flexbox from CSS',
        url: '/dwyl/dwyl.github.io/issues/29',
        desc: 'To ensure the site works across all devices, particularly Kindle/e-readers.',
        author: 'iteles',
        created: '2015-07-25T22:57:20Z',
        comments: 2 },
      { title: 'CSS | Add indentation back into main.css (disappeared from master)',
        url: '/dwyl/tudo/issues/77',
        desc: 'All indentation has been removed from main.css in the latest commit.     \n\nThis needs to be put back in as originally written by @msmichellegar and @iteles.',
        author: 'iteles',
        created: '2015-07-25T16:27:59Z' },
      { title: 'CSS | Investigate styling of issue label colours',
        url: '/dwyl/tudo/issues/72',
        desc: 'Labels can be given any colour so there is no predictable set that we can code into the CSS file.\n\nWe need to investigate what the best way to ensure we can provide the right colour of background to the ...',
        author: 'iteles',
        created: '2015-07-23T17:49:02Z',
        comments: 4 }
   ],
   next: '/search?o=desc&p=2&q=author%3Aiteles&s=created&state=open&type=Issues'
 }
 ```

 #### Owner

 For the issues created across all their *personal* repositories
 use a search query of the form:
 ```sh
 https://github.com/search?q=user%3A{username|org}
 &state={state}
 &type=Issues&s={relevance}
 &o={order}
 ```
 e.g:
 https://github.com/search?q=user%3Aiteles&state=open&type=Issues&s=updated&o=asc

 #### Author (_created by_)

 Or to find ***all*** the issues where the person is the ***author***
 use a query of the following format:

 ```sh
 https://github.com/search?q=author%3A{username|org}
 &state={state}
 &type=Issues&s={relevance}
 &o={order}
 ```

 #### Assignee (_issues assigned to this person_)

 Or to find ***all*** the issues *assigned* to the person use a query of the following format:

 ```sh
 https://github.com/search?q=assignee%3A{username|org}
 &state={state}
 &type=Issues&s={relevance}
 &o={order}
 &s={filter}
 ```

 #### Mentions

 We can use a ***mentions*** (search) query to discover all the
 issues where a given person (_username_) was mentioned:

 ```sh
 https://github.com/search?q=mentions%3A{username}&type=Issues&state={state}
 ```

 e.g: https://github.com/search?q=mentions%3Aiteles&type=Issues&state=open

 This _could_ be more than the issues in the person's (_own_) repos *or* the repos the person has access to (_via org_). e.g:
 if [_Sally_](http://www.imdb.com/title/tt1483013/quotes?item=qt1905812)
   axks a clarifying question on a project she has not yet contributed to,
   the issue will not appear when we crawl the repos on her profile or orgs she has access to ...

 #### Issues Filters

 There are *many* filters we can use to find issues, here are a few:

 + **created** https://github.com/search?q=author%3Aiteles&s=created&type=Issues&o=desc&state=open
 + **updated**: https://github.com/search?q=author%3Aiteles&s=updated&type=Issues&o=desc&state=open
 + **date range**: https://github.com/dwyl/time/issues?q=is%3Aissue+is%3Aopen+updated%3A%3C2015-06-28

 ##### Further Reading on Searching+Filters

 For *way* more details on searching & filters see:

 + https://help.github.com/articles/searching-issues/
 + https://help.github.com/articles/searching-github/#types-of-searches
 + https://help.github.com/articles/search-syntax/

## Want More Examples?

If you want ***even more*** examples of the pages you can scrape,
take a look at our end-to-end tests where we *test* all the scrapers!

## Future Features / Road Map ?

### Crawl the List of commits

Would it be interesting to see/track:
+ **who** makes the most commits to the project
+ **when** (***what time*** of day/night) people do their work
+ **what** did the person contribute? (docs, code improvement, tests, typo, dependency update?)

Show your interest in this feature: https://github.com/nelsonic/github-scraper/issues/17

# Contributing?

Contributions are _always_ welcome!
We have a backlog of features (_many pages we want to parse_)

please see: https://github.com/nelsonic/github-scraper/issues

If anything interests you, please lave a comment on the issue.

Your first step to _contributing_ to this project
is to run it on your **`localhost`**.

### 1. Clone the Repository

In your terminal, clone the repository from GitHub:

```sh
git clone https://github.com/nelsonic/github-scraper.git && cd github-scraper
```

### 2. Install the Dependencies

Ensure you have Node.js installed, see https://nodejs.org

Then run the following command to install the project dependencies:

```sh
npm install
```

### 3. Run the Tests

In your terminal execute the following command:

```sh
npm test
```
You should see output in your terminal similar to the following:

```
added 162 packages from 177 contributors and audited 265 packages in 4.121s
```

You should see output similar to the following:

```
><EMAIL_ADDRESS>test /Users/n/code/github-scraper
> istanbul cover ./node_modules/tape/bin/tape ./test/*.js | node_modules/tap-spec/bin/cmd.js

  read list of followers for @jupiter (single page of followers)

      - - - GitHub Scraper >> /jupiter/followers >> followers  - - -
    ✔ jupiter/followers data.type: followers
    ✔ @jupiter/followers has 34 followers
    ✔ Nelson in jupiter/followers
    ✔ @jupiter/followers only has 1 page of followers

  read list of followers for @iteles (multi-page)

      - - - GitHub Scraper >> /iteles/followers >> followers  - - -
    ✔ "followers": 51 on page 1
    ✔ iteles/followers multi-page followers

... etc ...

=============================================================================
Writing coverage object [/Users/n/code/github-scraper/coverage/coverage.json]
Writing coverage reports at [/Users/n/code/github-scraper/coverage]
=============================================================================
    =============================== Coverage summary ===============================
    Statements   : 100% ( 192/192 )
    Branches     : 100% ( 63/63 )
    Functions    : 100% ( 22/22 )
    Lines        : 100% ( 192/192 )
    ================================================================================

  total:     102
  passing:   102
  duration:  31.6s
```

The tests take around 30 seconds to run on _my_ `localhost`,
but your test execution time will vary depending on your location
(_the further you are from GitHub's servers the slower the tests will run..._).

Don't panic if you see some red in your terminal while the tests are running.
We have to simulate failure `404` and `403` errors
to ensure that we can handle them.

And our script needs to not freak out when that happens.
This is good practice in DOM parsing, the web changes a _lot_!

When the tests _pass_ on your `localhost`,
you know everything is working as expected.
Time to move on to the fun bit!

> **Note**: This project follows Test Driven Development (TDD)
because it's the only way we can maintain our sanity ...
If we didn't have tests it would be _chaos_
and _everything_ would "break" all the time.
If you are contributing to the project,
please be aware that tests are required
and any Pull Requests without tests will not be considered.
(_please don't take it personally, it's just a rule we have_).

If you are new to TDD, please see:
[github.com/dwyl/**learn-tdd**](https://github.com/dwyl/learn-tdd)

### 4. Pick an Issue and Write Some Code!

Once you have the project running on your `localhost`,
it's time to pick a page to parse!

### Travis-CI?

We use Travis-CI (Continuous Integration),
to ensure that our code works
and all tests _pass_ whenever a change is made to the code.
This is _essential_ in _any_ project and even more so in a DOM parsing one.

If you are new to Travis-CI, please see:
[github.com/dwyl/**learn-travis**](https://github.com/dwyl/learn-travis)

### Pre-Commit Hook?

This is a precaution to ensure that the code we write is _always tested_.
There is no point writing code that is not being tested
as it will "break" almost immediately and be unmaintainable.

Simply wait a few seconds for the tests to pass
and then push your work to GitHub.

If you are new to pre-commit hooks, please see:
[github.com/dwyl/**learn-pre-commit**](https://github.com/dwyl/learn-pre-commit)

## tl;dr

If you are the kind of person that likes to *understand* how something works,
this is *your* section.

### Inferring Which Scraper to use from the URL

`lib/switcher.js` handles inference.
We wanted to use a `switch > case` construct but, ended up using `if/else`
because there are two types of checks we need to do so `if/else` seemed simpler.

## Interesting Facts

- GitHub has 10.3 Million users (_at last count_)
- yet the most followed person [Linus Torvalds](https://github.com/torvalds)
"_only_" has **28k followers** (_so its a **highly distributed network**_ )
+ https://www.githubarchive.org/ attempts to archive all of GitHub
+ http://octoboard.com/ shows stats for the past 24h

## Research

> Must read up about http://en.wikipedia.org/wiki/Inverted_index
> so I understand how to use: https://www.npmjs.org/package/level-inverted-index

- GitHub stats (node module): https://github.com/apiengine/ghstats
(no tests or recent work/activity, but interesting functionality)

- Hard Drive reliability stats:
https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014
(useful when selecting which drives to use in the storage array -
  Clear Winner is Hitachi 3TB)
- RAID explained in layman's terms:
http://uk.pcmag.com/storage-devices-reviews/7917/feature/raid-levels-explained
- RAID Calculator:
https://www.synology.com/en-global/support/RAID_calculator
(if you don't already know how much space you get)
- SQLite limits: https://www.sqlite.org/limits.html

## Useful Links

- Summary of ***Most Active*** GitHub users: http://git.io/top
- Intro to web-scraping with cheerio:
https://www.digitalocean.com/community/tutorials/how-to-use-node-js-request-and-cheerio-to-set-up-simple-web-scraping
- GitHub background info: http://en.wikipedia.org/wiki/GitHub
+ GitHub Event Types:
https://developer.github.com/v3/activity/events/types/

### GitHub Stats API

- Github Stats API: https://developer.github.com/v3/repos/statistics/
- GitHub Followers API: https://developer.github.com/v3/users/followers/

Example:

```sh
curl -v https://api.github.com/users/pgte/followers
```

```js
[
  {
    "login": "methodmissing",
    "id": 379,
    "avatar_url": "https://avatars.githubusercontent.com/u/379?v=2",
    "gravatar_id": "",
    "url": "https://api.github.com/users/methodmissing",
    "html_url": "https://github.com/methodmissing",
    "followers_url": "https://api.github.com/users/methodmissing/followers",
    "following_url": "https://api.github.com/users/methodmissing/following{/other_user}",
    "gists_url": "https://api.github.com/users/methodmissing/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/methodmissing/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/methodmissing/subscriptions",
    "organizations_url": "https://api.github.com/users/methodmissing/orgs",
    "repos_url": "https://api.github.com/users/methodmissing/repos",
    "events_url": "https://api.github.com/users/methodmissing/events{/privacy}",
    "received_events_url": "https://api.github.com/users/methodmissing/received_events",
    "type": "User",
    "site_admin": false
  },

etc...]
```

#### Issues (with using the) GitHub API:

- The API only returns 30 results per query.
- **X-RateLimit-Limit**: **60** (can only make 60 requests per hour) ...
1440 queries per day (60 per hour x 24 hours) sounds like *ample* on the surface.
But, if we assume the average person has at least 2 pages worth of followers (30<)
it means on a single instance/server we can only track 720 people.
Not really enough to do any sort of trend analysis. :disappointed:
If we are tracking people with hundreds of followers (and *growing fast*)
e.g. 300< followers. the number of users we can track comes down to
1440 / 10 = 140 people...
(10 requests to fetch complete list of followers) we burn through 1440 requests
pretty quickly.
- There's no guarantee which order the followers will be in
(e.g. most recent first?)
- **Results** are ***Cached*** so they are not-real time like they are in the
Web. (seems daft, but its true.) Ideally they would have a ***Streaming API***
but sadly, [GitHub is built in Ruby-on-Rails](http://builtwith.com/github.com)
which is "**RESTful**" (***not real-time***).

#### *But*...

Once we know _who_ we *should* be following, we can use

- https://developer.github.com/v3/users/followers/#follow-a-user
- https://developer.github.com/v3/users/followers/#check-if-one-user-follows-another

e.g:
```sh
curl -v https://api.github.com/users/pgte/following/visionmedia
```

# FAQ?

## Is *Crawling* a Website *Legal*...?

The fact that scraping or "crawling" is Google's Business Model suggests that scraping is at least "OK" ...

Started typing this into google and saw:

I read a few articles and was not able to locate a definitive answer ...

+ Legal Issues: https://en.wikipedia.org/wiki/Web_scraping#Legal_issues
+ It depends: http://resources.distilnetworks.com/h/i/53822104-is-web-scraping-illegal-depends-on-what-the-meaning-of-the-word-is-is/181642
+ Screen scraping: How to profit from your rival's data:
http://www.bbc.com/news/technology-23988890
+ Web Scraping For Fun and Profit: https://blog.hartleybrody.com/web-scraping/


text:
![CDL 2020 Cohort Project](../figures/CDL_logo.jpg)
# Quantum Cohort Project Business Application

## Step 1: Explain the technical problem you solved in this exercise

High-level problem: Using VQE to obtain potential energy surfaces (PSEs) for small molecules.

1. We employ a variety of different classical algorithms for calculating the molecular bond energy. The choice of algorithm is based on a tradeoff between speed and accuracy. These methods give us an anchor to help us calibrate and understand our smaller scale quantum experiments.

2. Generating the qubit hamiltonian requires mapping the fermionic states to qubit states through Jordan-Wigner transformation. The qubit hamiltonian produces the number of orbitals needed to represent the molecule and the number of qubits required. We then tapper the hamiltonian to reduce the number of qubits required to represent the molecule. The elements of the tapered hamiltonian generate a matrix. The eigenvalues are from this matrix.

3. We generate elementary unitary operations with UCC and QCC . Unitary Coupled-Cluster seemed very resource-heavy and took too long to generate any useful results. The Qubit Coupled-Cluster was instead leveraged as it returned results within reasonable time frames. The calculations were made with STO-3G basis because the size of the calculations was reasonable and achievable with our available machines. Ideally, we would use a 6-31g basis however, we were unable to determine the process and or theory required to restrict the active space. Freezing all other orbitals beyond the necessary few would have allowed for simpler calculations. The same problem occurred when we attempted to generate unitaries with a sto-3g basis when it came to larger molecules like LiH.

4. Measuring the outcome of each trial run actually requires a large number of physical measurements because quantum hardware can only measure the system in one quantum dimension at a time. We use novel techniques to reduce the number of measurements required, by grouping components of the quantum system together into configurations which require only one measurement for the whole group. By forming these groups, we can reduce the number of measurements required by factors of 3 to several 100. Again, the tradeoff is between speed and accuracy. Coarser grouping reduces accuracy but results in less measurements.

5. We provide a simple framework to calculate PES energy using quantum hardware devices. Our quantumPES module allows your scientists to select a molecule, prepare the circuits for deployment and finally launch the jobs with three lines of code. Our optimization of the the molecule energy function for submission to the Quantum computers ensures the smallest job footprint is created to minimize the number of circuits to be executed.

## Step 2: Explain or provide examples of the types of real-world problems this solution can solve

We know about one main problem/use case from our readings:

Use Case: An R&D scientist is trying to understand the potential energy surfaces (PSE) of molecules so that he can develop better pharmaceutical drugs.

Problem: Finding the lowest energy state of a molecular Hamiltonian cannot be solved exactly for more than a few atoms. For large molecules this problem is intractable even on today's largest supercomputers.

In addition to this understanding, our team decided to conduct primary interviews with QC companies who interact with customers on a daily basis. To that end, we set up calls with experts at IBM and Zapata.

Gavin Jones, Team Lead, Quantum Applications, IBM (PhD Chemistry, UCLA)

Pain points

This was a very insightful call with Gavin. I started off by asking him generally about customer articulated pain points within chemistry/pharma. Gavin said that people in industry aren’t very interested in the latest research, e.g. PES bond stretching. Yes, they may be interested in PES from a benchmarking standpoint, but they want to focus on applications that industry cares about. They want to better understand 2-3 molecule reactions. One area of interest are catalysts performing reactions with one molecule and then another, thereby accelerating the union of those two molecules. Also, can we rearrange reactions?

Areas of interest outside of PSE

Outside of reactions, another area of interest are general properties of molecules. Examples include dipole moments, excited states (this is one property, he says, that all industries care about!), electro-chemical processes - Can we make OLEDs, photoacid generators, and better batteries out of these things? Scientists also care about the interaction of molecules with light. On the materials side, they care about periodic solids and crystalline materials. And, one thing scientists care about that is cross-cutting is quantum machine learning to improve anything that they are doing classically.
I asked him how receptive the business side is to quantum experimentation. He said that they are in very early stages, so it is still very scientist driven. He said that they still have to be convinced that they need to be ready for a quantum future. But, he said that one thing that surprised him is that there is quite a bit of C-level interest in quantum. He believes that one reason that they are excited is because they realize that they don’t have to invest that much to get started (i.e. they don’t have to build a quantum computer themselves). Also, sometimes C-level executives are interested in areas outside of simulation, like optimization. CEOs are asking their teams to look into the benefits of quantum computing.

Great anecdote

I asked Gavin if he knows of cases where VQE has been used in industries outside of chemistry/pharmaceuticals. He said that VQE has actually been adapted and used in finance for the use case of CVAR (conditional value-at-risk). Then they took this method from finance and used it back in the chemistry world!
Also he said people have thought about how to use VQE QAOA for optimization algorithms within chemistry.

Large molecules

I asked Gavin if there large molecules that currently cannot be studied, but that are very important/impactful. His immediate answer was the iron complex that is used for nitrogen fixation (1-2% of the world’s energy is consumed for this process). But he said we are probably years away from fully understanding that.

Personal excitement

Gavin would love to see the day where quantum computers are used routinely in chemistry. He remembers roles that he had where he was modeling different materials and reactions. He never knew how believable his results were, and the methods could be quite wrong. He said that sometimes he didn’t have time for benchmarking. So, he would have to go back to experimentalists in the lab to verify his results. He’s excited about having more accurate methods.
I asked him what’s the impact of something that’s not so accurate. He gave an example of trying to design a new catalyst. Experimentalists may come to him with a problem - we have seen X happening and we don’t know why. Then he would try to model the problem on a classical computer. They try your results in the lab, and it’s quite embarrassing if it does not work! If it was more accurate you could help them improve yields for desired products, reaction rates.

James Ma, Product Manager, Zapata (Computer Science background)

James isn’t a chemistry expert, so he didn’t have much to say about the technical components of VQE, molecules etc. But, he did mention that quantum advantage hasn’t been proven in VQE so the benefits to customers is an open question, Here are some highlights from our conversation:

Hedging: The first is what is already commonly known - We haven’t been able to simulate something or access research because of the limits of classical computing. We can't execute algorithms in a reasonable amount of time. So we view this as a hedge since this is an active area of research. It’s on us to allocate resources to this.

Software best practices 101: The more interesting finding is that these groups have basic problems related to software engineering principles and practices that are really stifling their progress. One example is that we have seen that they email code around instead of using repositories on Github! They don’t use microservices. One big problem is reproducibility of code - if someone develops something and leaves, they have a lot of loss related to that. We help our clients with these basics and they really appreciate it; it creates stickiness for us.
Shift from algorithms to tooling: Generally James sees a shift from a focus on algorithms to tooling. Scientists need to build the capabilities to access quantum computing, and we are creating that new abstraction level. We want the barrier to using quantum computing to be lower.

Quantum inspired: Clients are exploring the area of quantum inspired algorithms. They want to know if exploring quantum can help them improve their classical methods.

Costs: Executives want to cut costs, but they don’t care if it comes from AI, blockchain or quantum. But, James believes that the cost advantages from quantum come from increased reliability, less asynchronous computing, and time savings.

Four levels of customers:

1. Proof of concept: These customers don’t really get quantum yet, so they want to have workshops and learn and perhaps design initial experiments or collaborations
2. Resource estimation: These clients understand the benefits of quantum computing. They want to know about resource estimation - how many qubits do we need and what stability do we need before I need to care/invest in this?
3. Workflow: How do I implement quantum and integrate it into my existing workflow pipeline? How do I productize it?
4. Novel IP: We want to develop the quantum version of X algorithm before anyone else, or maybe it’s something specific to a chemical they are developing. They want patents.

## Step 3: Identify at least one potential customer for this solution - ie: a business who has this problem and would consider paying to have this problem solved

Executive summary: We would initially focus on pharmaceutical companies and maybe the top two chemicals companies (Dupont, BASF). For pharma companies, we would focus on companies who have shown significant quantum interest. This list seems pretty short though (Merck, Abbvie, Bayer, Novartis). We would also cast a wider net (i.e. look at pharma firms beyond the top 10) and take a more educational/workshop (increase awareness) approach as a “foot-in-the-door” strategy.

We are going to target the pharmaceutical industry. Below you see a chart of the revenue rankings for pharmaceutical companies in 2020. For these firms, we also want to understand if they are already interested/investing in quantum computing. The idea is that if they are already interested/investing, then they will be more receptive to our pitch.

Source: https://www.bizvibe.com/blog/largest-pharmaceutical-companies/

Here is the evidence that we found that these 10 firms are quantum interested:

Johnson & Johnson
Supply chain executive believes quantum can have an impact in the next few years
https://www.cxotalk.com/video/johnson-johnson-creates-intelligent-supply-chain
Neil Ackerman, Global Supply Chain Digital Executive:
https://www.linkedin.com/in/neilackerman/

Roche
Found a number of articles, social media posts:
https://www.roche.com/quantum-computing.htm
https://www.roche.com/future-quantum-computing.htm

Martin Strahm,Head of Data Science:
https://www.linkedin.com/in/martin-strahm-56a67b12a/
https://www.facebook.com/RocheCareers/posts/2458483347511552

Pfizer
Some evidence that Pfizer has partnerships with AI companies that are experimenting with quantum computing:
https://www.breakthroughs.com/health-tomorrow/how-quantum-physics-and-ai-disrupting-drug-discovery-development

Bruno Hancock, Global Head of Materials Science:
https://www.linkedin.com/in/bruno-hancock-20a3b210/
https://pharmaphorum.com/news/pfizer-ai-drug-discovery-xtalpi/

Bayer
Bayer is hiring for the role, Quantum Computing Manager. The posting says that Bayer has been working on quantum since 2016.
https://career.bayer.us/en/job/quantum-computing-manager--SF142341
This article mentions that in 2018, Bayer was working with Atos to explore quantum computing applications for human disease patterns:
https://atos.net/en/2018/press-release_2018_11_07/atos-bayer-rwth-aachen-university-use-atos-quantum-learning-machine-study-human-disease-patterns
Ulf Hengstmann, IT Manager:
https://www.linkedin.com/in/ulf-hengstmann-bbb479/

Novartis
They have an existing partnership with Microsoft that they would like to expand. This may include quantum computing.
Forbes article mentions that new CEO is keen on using quantum computing
https://www.forbes.com/sites/matthewherper/2018/03/26/ai-telemedicine-quantum-new-novartis-boss-says-tech-will-finally-change-the-drug-biz/#308a58656b54

Merck
Earlier this year, Merck’s venture arm invested in a quantum startup, Seeqc:
https://www.wsj.com/articles/merck-venture-arm-invests-in-quantum-computing-startup-seeqc-11586454285
Announcement of partnership with HQS Quantum Simulations, a quantum startup:
https://www.emdgroup.com/en/news/quantum-computing-04-06-2019.html
Philip Harbach, Head of In Silico Research: https://www.linkedin.com/in/philipp-harbach-579670172/
Kam Chana, Director Computational Platforms, attended a quantum conference (QTech) in Boston last September.
https://www.linkedin.com/in/kam-chana/

GlaxoSmithKline
GSK has invested in a company called Cloud Pharmaceuticals that is exploring quantum applications:
https://www.biopharmatrend.com/post/99-8-startups-applying-quantum-calculations-for-drug-discovery/
It looks like GSK has partnered with industry groups to explore quantum:
https://quantumlab.info/2019-1
https://admin.ktn-uk.co.uk/app/uploads/2019/04/QC_Pharma_140319.pdf
https://gtr.ukri.org/projects?ref=EP%2FT001062%2F1
Potential lead - Bonnie Kruft, Director of Data Science:
https://www.linkedin.com/in/bonniekruft/?originalSubdomain=uk

Sanofi
Did not find any evidence of quantum interest/investment

AbbVie
Brian Martin, Head of AI in R&D Information Research:
https://www.linkedin.com/in/brianm1028/
Tweet: https://twitter.com/abbvie/status/1228695498191757312

Brian attends several quantum conferences
https://www.quantumtechcongress.com/agenda/conference-day-one
https://www.bio-itworld.com/2020/02/11/abbvies-fresh-look-at-how-ai-and-quantum-computing-will-transform-biotech.aspx

Abbott
Did not find any evidence of quantum interest/investment

Largest chemical companies by sales - here we would approach the top two, DuPont and BASF, and then expand beyond these two.

## Step 4: Prepare a 90 second video explaining the value proposition of your innovation to this potential customer in non-technical language

Many of our clients in the chemicals and pharmaceutical industry are interested and have invested in quantum computing for several reasons:

Simulation: There is a potential to learn about molecules in a way that simply isn’t possible on classical computers. These learnings, for example with catalysts, can help scientists speed up chemical reactions and yields for specific products. This will have revenue-side benefits for organizations.

Machine learning/Optimization: Outside of molecular simulation, clients also care about how QC can help with optimization, and accelerate machine learning.

Learning curve: While perfect quantum hardware does not exist today, many clients realize that the learning curve is very steep, so they need to start the learning process today so they do not miss out on the significant benefits. Also, C-level executives have realized that they do not have to invest that much to get started (i.e. they don’t have to build their own quantum computer).

Costs: Compared to classical computing, quantum computing outcomes will be more reliable/accurate, and much less time consuming. All of this has cost-side benefits for our clients.

Quantum inspired: Finally, we often hear the critique - useful quantum computers are years away. However, and here is what many people are not aware of - many of our clients that are experimenting and learning about quantum algorithms now are using these learnings to improve their classical algorithms and workflows! That means that investing in quantum computing today can have benefits TODAY.

If you are interested in learning more, please reach out to us at Creative Destruction Lab.

[![Introduction](../figures/youtube.png)](https://youtu.be/hcUpBK2gFis)

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: HTML

gha_license_id: MIT

license_type: permissive


text:
# adventofcode-d-2015

These are my solutions to http://adventofcode.com

All solutions are written in D.
Originally written in July 2017, during which point 2.074.1 would have been available.
Have been checked for deprecation warnings up through 2.098.0, though other possible code improvements in the meantime have not necessarily been noticed.

[![Build Status](https://travis-ci.org/petertseng/adventofcode-d-2015.svg?branch=master)](https://travis-ci.org/petertseng/adventofcode-d-2015)

# Input

In general, all solutions can be invoked in both of the following ways:

* Without command-line arguments, takes input on standard input.
* With 1+ command-line arguments, reads input from the first, which must be the path to an input file.
  Arguments beyond the first are ignored.

Some may additionally support other ways:

* 4 (Advent Coins): Pass the secret key in ARGV.
* 10 (Look and Say): Pass the seed sequence in ARGV.
* 11 (Passwords): Pass the initial password in ARGV.
* 20 (Factors): Pass the target number of gifts in ARGV.
* 21 (RPG): Pass the Boss's HP, damage, and armor in ARGV, in that order.
* 22 (Wizard): Pass the Boss's HP and damage in ARGV, in that order. Pass `-d` flag to see example battles.
* 25 (Triangular): Pass the row and column number in ARGV, in that order.

## DUB

DUB is used to build the library functions common to all days, but each day is its own [single-file package](https://code.dlang.org/getting_started#single-file-packages).
The alternative would have to been to have the entire thing be a single DUB library, taking advantage of D's native `unittest` functionality.

Advantages of doing it this way:

  * It can be useful to have example code to do file I/O (turns out it's pretty simple in D).
  * Fairer runtime timing comparisons against other languages. If inputs are known at compile-time, the compiler may pre-compute some values before runtime.
  * Failure messages of D's native `unittest`, being `assert`s, are not particularly helpful in saying what failed (no expected/observed values, etc.); `diff` makes it clear what failed.
  * Flexibility: A built binary may be run against many users' inputs, instead of only some specific hard-coded inputs.

Diadvantages:

  * Can't take advantage of `dub test`.
  * Extra boilerplate per file (`dub.sdl` comment header, `main` that reads input file).

## Language notes

### Slurp

Slurp is quite good.
Impressed how it can return whatever type you specify, and also that it can return tuples if there's more than one per line, and scalars if there's just one.
It's not all-powerful though: See day 06 where I had to hack around its eager match.
Also note that for some reason what it returns can't become immutable.

### Style: const?

Should I make args const if they are pass-by-value? This includes strings.
This serves as assurance that I don't accidentally reassign internally.
But if I were writing an API, such an assurance is not particularly helpful to my clients.
It only helps the programmer.
I decided not to.
Select things in main (such as my input) are made const or immutable though.

### Regexes

Not impressed. See day 05.

### Tuples

Tuples with named fields are nice.

### map!

I initially kept forgetting I have to say `map!` instead of `map`, and the compiler error is very unhelpful in this case.
As of 2.074.1:

    day06_light_grid.d(58,23): Error: template std.algorithm.iteration.map cannot deduce function from argument types !()(int[1000][], void), candidates are:
    /usr/include/dlang/dmd/std/algorithm/iteration.d(456,1):        std.algorithm.iteration.map(fun...) if (fun.length >= 1)

As of 2.094.1, a little better, but still not sure it's clear I'm suppoesd to use `map!`

    day06_light_grid.d(58,23): Error: template std.algorithm.iteration.map cannot deduce function from argument types !()(uint[1000][], void), candidates are:
    /usr/include/dlang/dmd/std/algorithm/iteration.d(482,1):        map(fun...)
      with fun = ()
      must satisfy the following constraint:
           fun.length >= 1

### Static arrays

See day 06: static arrays can't be used as inputs to functions that want ranges.
<EMAIL_ADDRESS>
### algebraic

Very useful, see day 07.
`visit!` makes sure all variants are handled.

### UFCS

Very useful; I routinely use it.
I seem to not use it for `array` though, whereas the code I lifted off of Rosetta Code (day 24) did.

### No destructuring

Not being able to destructure the iteratee in `foreach` is a little unfortunate.
Would have been useful in day 09 and day 21.
At least you can omit the type (explicitly saying `auto` doesn't work!).

### recurrence

Pretty cool. See day 10.

### min of a range

I wish you could do `things.min` to get the minimum thing out of things, rather than `things.fold!(min)`.

### Style: template args?

Unsure about best style for template args, such as for map.
Double quotes? Backticks? Function literal? If I use either of the first two, parens or not?

### Style: imports?

Unsure whether all imports should go at top of function, or as close as possible to use.
I think I'll settle on "top of scopes only, narrowest scope possible"?


text:
* Create the front-end (visuals) for home page, reservation form, and reservation views.
    * Create 3 html files
        * home.html
        * tables.html
        * reserve.html

* Create a basic server using Express.JS

* Create a few array variables that will hold the data

* Create a set of routes for getting and posting table data

* Create a set of routes for displaying the HTML pages

* Use jQuery to run AJAX calls to GET and POST data from users to the Express server

repository: HillaryBLavin/hot-restaurant

repo_name: HillaryBLavin/hot-restaurant


text:
# eGo
Generates e with Go. Serial/parallel versions included
https://github.com/NinjaJc01/eGo

Instructions for use:
  1.
Use the website, not the package manager otherwise it's outdated
https://golang.org/
  1. Install dependency for my code: ```go get -u github.com/ericlagergren/decimal```
  1. Build my project: ```go build ./parallel/egoParallel.go```
  To build for different architectures or operating systems:
  ```GOARCH=arm64 GOOS=linux go build ./parallel/egoParallel.go```
  Would build for arm64, linux (raspi etc, sometimes you may need just arm)
  See entries in bold for supported OS and Architectures out of the box: https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63

New: try -hard as a flag, sets the iterations and precision higher to give your CPU a workout! Overrides any set precision or iterations!

gha_language: Go


text:
# Informal

Informal is a small gem that enhances a Plain Old Ruby Object so it can be used
with Rails 3 form helpers in place of an ActiveRecord model. It works with the
Rails `form_for` helper, and `simple_form` as well.

Here's a quick (and slightly insane) example:

    # models/command.rb
    require "informal"
    class Command
      include Informal::Model
      attr_accessor :command, :args
      validates_presence_of :command
      def run; `#{command} #{args}`; end
    end

    # views/commands/new.html.erb
    <%= form_for @command do |form| %>
      <%= form.text_field :command %>
      <%= form.text_field :args %>
      <%= form.submit "Do It!" %>
    <% end %>

    # controllers/commands_controller.rb
    def create
      command = Command.new(params[:command])
      if command.valid?
        command.run
      end
    end

## Installation

It's a Ruby gem, so just install it with `gem install informal`, add it to your
bundler Gemfile, or do whatever you like to do with gems. There is nothing to
configure.

## Usage

The insanity of the above example aside, Informal is pretty useful for creating
simple RESTful resources that don't map directly to ActiveRecord models. It
evolved from handling login credentials to creating model objects that were
stored in a serialized attribute of a parent resource.

In many ways using an informal model is just like using an AR model in
controllers and views. The biggest difference is that you don't `save` an
informal object, but you can add validations and check if it's `valid?`. If
there are any validation errors, the object will have all the usual error
decorations so that error messages will display properly in the form view.

### Initialization, #super and attributes

If you include `Informal::Model`, your class automatically gets an
`#initialize` method that takes a params hash and calls setters for all
attributes in the hash. If your model class inherits from a class that has its
own `#initialize` method that needs to get the super call, you should instead
include `Informal::ModelNoInit`, which does not create an `#initialize` method.
Make your own `#initialize` method, and in that you can assign the attributes
using the `#attributes=` method and also call super with whatever args are
needed.

## License

Released under the MIT License.  See the LICENSE file.

repository: Epictetus/informal

repo_name: Epictetus/informal


text:
---
title: There-s finally a portable battery pack you can-t forget at home
author: PipisCrew
date: 2020-02-29
categories: [news]
toc: true
---

TL;DR: You can carry around a charger with you wherever you go for just $39.99, a 20% savings as of Feb. 29, thanks to this deal on the AtomXS emergency keychain charger. Even if you have the latest and (arguably) greatest iPhone 11, the battery isn't going to last if you're on it 24/7.

https://mashable.com/shopping/atomxs-keychain-charger-on-sale/

origin - https://www.pipiscrew.com/2020/02/theres-finally-a-portable-battery-pack-you-cant-forget-at-home/ theres-finally-a-portable-battery-pack-you-cant-forget-at-home


text:
---
layout: post
title: "Radix"
excerpt: ""
categories: knownotes
tags: [mathematics]
comments: true
share: true
author: chungyu
---
> [Radix](https://en.wikipedia.org/wiki/Radix)

- The radix or base is the number of unique digits, including zero, used to represent numbers in a positional numeral system.
- The decimal system (the most common system in use today) the radix is ten, because it uses the ten digits from 0 through 9.
- In a system of radix $$b$$, a string of digits $$d_1, d_2, ..., d_n$$ denotes the number $$d_1b^{n-1} + d_2b^{n-2} + ... + d_nb^0$$

### [Excel Sheet Column Title](https://leetcode.com/problems/excel-sheet-column-title/)
- [Difference between this counting system and the normal base-10 system](https://leetcode.com/discuss/45534/note-difference-between-counting-system-normal-base-system)


text:
# 3.0.2 Smirk

- Fixed a bug that prevented kanye from filtering on a common ancestor

# 3.0.1 Mug Face

- Changed the public API in favor of a single `options` object

# 2.0.4 Mug Look

- Expose `handlers` so that you can manipulate them directly

# 2.0.3 Mug Lock

- Fixed a bug where events would be incorrectly filtered

# 2.0.2 Teapot

- Simplified event binding logic into a single event listener

# 2.0.0 Walkie Talkie

- Introduced ability to filter on DOM elements
- Removed ability to manipulate keyboard event listeners directly

# 1.0.0 IPO

- Initial Public Release

repository: bevacqua/kanye

extension: markdown

gha_language: JavaScript

repo_name: bevacqua/kanye


text:
##crisis.js
##A library for websites that encounter individuals experiencing crisis

###Warning

This is a **proof of concept** library that is **not** intended for production environments. If you wish to participate in
this project, or you wish to use this library, you, and your users, agree not to hold this library, its author, or any of
its contributors liable for its use. We're trying to potentially save lives, not be penalized for doing the right thing.

That being said, because this project deals with highly sensitive topics, the terminology contained within this project may
not be suitable for all audiences. It may trigger overwhelming emotions, pain, or past trauma.  Please exercise caution before continuing.

###Purpose

Crisis is an every day occurrence. It may be possible, for a few kb per page, to intervene and prevent a death, prevent abuse,
or otherwise save a life. This library is a proof of concept to see if it's possible to build a system that can detect pain
in users and help them wherever possible.

###How It Works

JavaScript is typically used for client-side interaction. One of the benefits of that, is being able to detect user input and
react to that input. This application works by taking a definition file that outlines sensitive combinations of phrases, and
processes them into regular expressions. When a user enters any number of sensitive strings defined by those definitions,
they will be greeted with a basic JavaScript confirmation prompt asking them if they need help. If they choose to select "OK",
they will be sent to a one of the accompanying online resources assigned to a particular category of crisis.

There is a consideration that monitoring the input of users may be seen as an invasion of privacy. This application **will not
and should not ever store user input or identify the person beyond location.** In order to alleviate a factor of annoyance,
whether the user selects "OK" or "Cancel", the pop-up will not reactive on the input that was used until the page is refreshed.

###Regular Expressions

Combinations of phrases are designed to use regular expressions to appropriately trigger help. For instance, if someone writes
`I am thinking about committing suicide` it will trigger just as much as `I will commit suicide` or `he cuts himself`.

###Settings

There are various settings that can be defined for the application, or passed in on instantiation. While the application is in
a transition phase, please refer to the `settings.js` file in the `src` directory for a complete list of settings that can be
passed in.

####Languages

Crisis happens in all languages. Therefore, this application is designed to support multiple languages. A language template
file can be found in the `src/langs` directory. If you plan to contribute a new language to Crisis, please follow the existing
design therein. You will need to modify the `gulpfile.js` and create an appropriate `index-xx.html` file for demonstration.
Please see the contribution section below.

###Contribution

Contribution is welcomed, but because of the nature of this project, must be performed with the best intentions. The primary
goal is to use our programming technique to potentially save lives. Next to that, we want to make sure it's an attractive
application. This means keeping the file-size low, accessibility high, and compatibility across many devices and browsers.

####Installation, Compilation, and Modification

Node is used extensively to handle streamlining the build process for crisis.js. Please become familiar with node, npm, and
the technologies used here before contributing.

Gulp is being used exclusively for task running. Baked into the system is various commands that can be run to manipulate the
resulting program. Here's a list of common commands and explanations

* `gulp` - Compile all languages, all files, and minify the content for release.
* `gulp --dev` - Compile all languages, all files, and reserve minification for development work.
* `gulp lang-xx` - Compile the language specified, for instance, `gulp lang-en` will compile and minify English for release.
* `gulp lang-xx --dev` - Compile the language specified, and reserve minification for development work.
* `gulp compress` - Compresses all compiled files for release.

Included in the gulp build is BrowserSync. Please read the BrowserSync documentation if you are unfamiliar with how it helps
development.

###Demonstration

The crisis.js application, compiled with the latest, most stable version, can be viewed from the accompanying GitHub Pages
repository located here: [http://mookman288.github.io/crisis.js/](http://mookman288.github.io/crisis.js/).

###License
####The MIT License (MIT)

Copyright (c) 2016 PxO Ink

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


text:
# Automated Chef Client Installation Using Chef-Provisioning

1. [Overview](#overview)
2. [Pre-Install Tasks](#pre-install)
3. [Workstation Setup](#workstation)
4. [Target Node Setup](#node)
5. [Chef Provisioning Recipe](#recipe)
6. [Running the Chef Provisioning Recipe](#running)
7. [Troubleshooting](#troubleshooting)

## Overview

This document describes automated Chef client installation and setup on Cisco Nexus switches using the [Chef Provisioning](https://github.com/chef/chef-provisioning/) and [Chef Provisioning SSH](https://github.com/chef/chef-provisioning-ssh) Ruby gems.

## Pre-Install Tasks

### Platform and Software Support

See [Requirements](../README.md#requirements) for details on platform and software requirements.

Additionally, all of the following software versions are required:

* Chef Provisioning 1.3.0 or later
* Chef Provisioning SSH 0.0.10 or later

### Disk space

400MB of free disk space on bootflash is recommended before installing the
Chef client software on the target agent node.

### Environment
NX-OS supports three possible environments for running third party software:
`bash-shell`, `guestshell` and the `open agent container (OAC)`.

Environment                  | Supported Platforms       |
-----------------------------|---------------------------|
`bash-shell`                 | Cisco Nexus N3k, N9k, N9k-F |
`guestshell`                 | Cisco Nexus N3k, N9k, N9k-F |
`open agent container (OAC)` | Cisco Nexus N5k, N6k, N7k |

You may run chef-client in any of the three supported environments but not simultaneously.

 Environment                 | Description |
-----------------------------|------------------------------------------|
`bash-shell` | This is the native WRL Linux environment underlying NX-OS. It is disabled by default.
`guestshell` | This is a secure Linux container environment running CentOS. It is enabled by default in most platforms that support it.
`open agent container (OAC)` 1 |
 This is a 32-bit CentOS-based container created specifically for running Puppet Agent software.

 OAC containers are created for specific platforms and must be downloaded from Cisco.

 The OAC must be installed before the Puppet Agent can be installed.

Access the following [link](README-agent-install.md) for more information on enabling these environments.

1*Automated chef-client installation is currently not supported in the `open agent container (OAC)` environment. This may change in future releases.*

## Workstation Setup

This document assumes you already have a Chef workstation set up. The below are additional steps needed to use your Chef workstation for Chef Provisioning.

### Gem installation

Install both gems on your Chef workstation. You may need to use `sudo` depending on your permissions setup.

```bash
chefws$ sudo /opt/chef/embedded/bin/gem install chef-provisioning chef-provisioning-ssh
```

### Chef server certificate

Fetch a copy of the certificate for your Chef server and mark it as trusted:

```bash
chefws$ sudo knife ssl fetch -c /etc/chef/client.rb
WARNING: Certificates from chefserver.example.com will be fetched and placed in your trusted_cert
directory (/etc/chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for chefserver.example.com in /etc/chef/trusted_certs/chefserver_example_com.crt
chefws$
```

## Target Node Setup

The SSH feature needs to be enabled and a devops user with sudo access will be needed for the Chef workstation to access and install the Chef client software into the `bash-shell` or `guestshell` environment.

**Example:**

```bash
configure terminal
  feature ssh
  username devops password devopspassword role network-admin
  username devops shelltype bash
end
```

Additional setup needed for `bash-shell` or `guestshell` (whether using Chef Provisioner or a manual installation) is documented [here](README-agent-install.md).

## Chef Provisioning Recipe

For example, `chef_provisioning.rb`:

```ruby
require 'chef/provisioning/ssh_driver'

with_driver 'ssh'

with_chef_server 'https://chefserver.example.com/organizations/chef',
  :client_name => 'chefuser',
  :signing_key_filename => '/etc/chef/chefuser.pem'

with_machine_options :transport_options => {
  :ip_address => '<IP_ADDRESS>',
  :username => 'devops',
  :options => {
    # BASH-SHELL options - uncomment if installing to bash-shell
    # :prefix => 'sudo ip netns exec management ',

    # GUESTSHELL options - uncomment if installing to guestshell
    # :prefix => '/isan/bin/guestshell sudo ip netns exec management ',
    # :scp_temp_dir => '/bootflash',
  },
  :ssh_options => {
    :password => 'devopspassword',
  },
}

machine "n3k_100_1.example.com" do
  action [:ready, :setup, :converge]
  # Copy the trusted certificate to the newly provisioned node
  file '/etc/chef/trusted_certs/chefserver_example_com.crt', '/etc/chef/trusted_certs/chefserver_example_com.crt'
  converge true
end
```

### Bash-shell versus guestshell

As can be seen above, the difference between a `bash-shell` recipe and a `guestshell` recipe is quite small and only exists in the `transport_options` `options` hash.

* `bash-shell`
  * `:prefix => 'sudo ip netns exec management '`
     * The `prefix` string is prepended to all commands executed on the node by Chef Provisioning SSH. This prefix causes all commands to be run as root in the `management` VRF namespace.
* `guestshell`
  * `:prefix => '/isan/bin/guestshell sudo ip netns exec management '`
     * As above, but the addition of `/isan/bin/guestshell` causes commands to be executed in `guestshell` instead of `bash-shell`.
  * `:scp_temp_dir => '/bootflash'`
     * Because `guestshell` does not have access to the `bash-shell` `/tmp` directory, but does have access to `/bootflash`, we upload files via SCP to `/bootflash` in order to make them visible to `guestshell`.

## Running The Chef Provisioning Recipe

```bash
chefws$ chef-client -z chef_provisioning.rb
```

## Troubleshooting

### "Unable to connect"

If you get an error like the following:

```
RuntimeError
------------
UNABLE to Connect to <IP_ADDRESS> using devops and {:user=>"devops", :password=>"devopspassword", :timeout=>30, :keys=>[]}
```

then you may have a stale SSH key in your `~/.ssh/known_hosts` - remove it and rerun.

### Net::HTTPServerException: 401 "Unauthorized"

You can get this error if your Chef workstation's clock is not in sync with the Chef server. Ensure that their clocks are synchronized, preferably with NTP.

repository: cisco/cisco-network-chef-cookbook

branch_name: refs/heads/develop

gha_language: Ruby

repo_name: cisco/cisco-network-chef-cookbook


text:
Anyplace Clients
================
Different clients for Anyplace.

## [android](android): Anyplace Navigator and Logger
Developed by Timotheos.

## [android-new](android-new)
In development by Christakis Achilleos

#### [android-new/lib](android-new/lib)
Gradle-based Android library for anyplace.
Uses core and implements new features specific for Android.

#### [android-new/demo](android-new/demo)
Thin Demo Android client that uses the [android-new/lib](android-new/lib).

## [core](core)
#### [core/lib](core/lib)
Gradle-based Java core library that can be included in both Android and desktop JVMs.

#### [core/cli](core/cli)
Command line interface for fully-fledged desktops.
Is used to create CLI clients for Windows, Linux, macOS.

## [robotos](robotos)
Robot OS client demo.
Developed by Mike Karamousadakis.

## [macos](macos)
Demo bash script for macOS.

## [linux](linux)
Demo bash script for Linux.

## [windows](windows)
PowerShell script for Windows (placeholder).

## [web](web)
Web clients:
- architect
- viewer
- viewer campus

---
## [_deprecated](deprecated)
Clients that won't be further developed.

#### [_deprecated/windows-phone](deprecated/windows-phone)
Legacy Windows client code.

#### [_deprecated/ios](deprecated/ios)
Legacy iOS client.

---
TEAM
---
- **Leader:** Christakis Achilleos
- **Co-Advisor:** Paschalis Mpeis

* Past Members:
    - Marcos Antonios Charalambous
    - Constandinos Demetriou
    - Christakis Achilleos

## [About us](https://anyplace.cs.ucy.ac.cy/#team)

repository: dmsl/anyplace

repo_name: dmsl/anyplace


text:
## About Claravue

Claravue is short for "Clean Laravel Vue", which implements a clean instance of Laravel and VueJs to make deployment and management of websites significantly easier. Features are constantly being developed and improved upon to make management of a service significantly easier.

## Features

- Full Screen Backend Display
- User Management
- Role Management [In Development]
- Icon Reference List
- Menu Management
- Page Management
- File Management
- Interest Check
- Recording JSON Lists [In Development]
- Blog
- Shop
- Checkout
- Stripe Integration [In Development]
- Todo List
- Frontend/Backend Settings Management
- Internal Chat Service

## Installation

- Make sure you have a Database and the credentials on hand.
- Make sure you have your email sftp information on hand.
- Run the following command in the directory of where you pulled the project.

```sh
$ ./install.sh
```

### If you wish to know more about me...

Feel free to email me at<EMAIL_ADDRESS>
Please give me a follow or subscribe in the following:

|Twitter|Github|Youtube|Twitch|Linkedin|Personal Site|
| ----- | ---- | ----- | ---- | ------ | ----------- |
|[MathisonProject](https://twitter.com/MathisonProject)|[MathisonProjects](https://github.com/MathisonProjects)|[Jacob Mathison](https://www.youtube.com/channel/UCNNxB1TRbdJxE_y51sJb9DA)|[MathisonProjects](http://twitch.tv/mathisonprojects)|[Jacob Mathison](https://www.linkedin.com/in/jacob-a-mathison-62359912/)|[Mathison Projects](http://mathisonprojects.com)|

repository: duanecarbullido/Claravue

repo_name: duanecarbullido/Claravue


text:
---
title: "Draft: Greedy Algorithms"
date: "2020-08-22"
description: "In which we learn how greed can be good...for problem solving"
---

What is a greedy algorithm?

I wondered this myself. What subjects an algorithm to avarice? As it
turns out, the adjective "greedy" has nothing to do what one might
consider greed, and everything to do with taking a chance on a solution
that might be incredibly efficient if everything goes well.

In a way, it reminds me of how the Houston Rockets play basketball.
Could they theoretically have James Harden shoot 3-pointers every time they have
possession and have it turn out well?

Sure, maybe.

But there are edge cases, like when really good defenders guard him.

>Nobody can defend his step back 3 though... ;)

In reality, a greedy algorithm is more about:
- picking a first approach,
- proving that that approach is safe to use,
- using that same approach to solve any remaining subproblem(s).

This, of course, can be problematic if you don't  verify that the chosen
approach is actually safe to use, e.g., that it won't break due to edge cases.

This differs from brute force algorithms, in which each and every
possible approach is attempted.

What's an example?

// Example to be added

**Resources**
- https://towardsdatascience.com/algorithms-in-c-62b607a6131d
https://www.freecodecamp.org/news/brute-force-algorithms-explained/
- https://book.huihoo.com/data-structures-and-algorithms-with-object-oriented-design-patterns-in-c++/html/page441.html
- https://medium.com/self-training-data-science-enthusiast/brute-force-algorithm-and-greedy-algorithm-13195d48e9bf


text:
---
title: Culture of Reading
description: ''
date: '2010-05-10T22:55:00.000Z'
categories: ''
keywords: ''
slug: /@kuppurao/culture-of-reading-7715eccae791
---

Update on 5/16: Have made changes to the original post to ensure correct interpretation of this article. Thanks to those that posted comments, that helped highlight the problem in my original articulation.

By now, you must have heard me several times that I recently converted to the book-reading cult. The same was true of Hollywood movies as well (never watched one until 5 years ago), but with the latter I was atleast genuinely ignorant.

But with books, it is another matter. I always knew what I was missing, I always had a certain respect for people devoted to reading, yet I did not pick up a book. I happened to have a fierce discussion on this subject with my sister and few of my friends and hence this post.

 It should be cultivated early on in everyone’s life and parents are obliged to inculcate this habit as well. And I squarely blame parents if kids don’t read. Not for the consequence, but for not creating an ambience conducive for learning, for not exposing them to reading and creating awareness of the benefits of reading.

The people I brainstormed this with, came up with various responses. My comments in _Italics_ separated by Tilde (~)

*   **Reading is a taste** like it is for music or painting and therefore cannot be enforced. People read or they don’t based upon their individual taste. ~_Disagree. Unlike music or painting, reading affects all facets of life._
*   **Parents’ reading habits** singularly affect kids’ reading habits. ~_Non-reading parents have an uphill task. Tough luck, but no excuses._
*   **Gender plays a role** — Males go out, socialize and had other avenues of learning. But girls were home-bound and took to reading as a means of being in touch with the world. This was particularly true of India of the past, but not anymore. ~_I reserve my comments on this one_.
*   **Affordability plays a role**. Financially secure parents can afford more books and better books. ~_With public libraries and internet, reading has never been cheaper. Even membership libraries are affordable and will cost a fraction of cable TV subscription costs. Added to this, we have roadside shops as well for used books_.
*   **School culture plays a role** — some schools encourage exploration and encourage reading outside of the prescribed books. ~_Sad but true. Parent have to work harder_.
*   **Peers play a role** in influencing the reading habits. _~Sad but true. Parents have to work harder._

_But what if parents themselves are completely ignorant as to the benefits of reading? This was brought up by_ [_Vidya_](http://vidyab.blogspot.com/) _through her comments and my subsequent discussion with her. I dont have an answer for this._

I would like to hear from you, the readers.

PS: Also, I use “parents” as a placeholder for anybody who provides for the kid — sending to school, food, cloth, shelter etc…

repository: kuppurao/website

repo_name: kuppurao/website


text:
# Stretch Cluster with an Arbiter

Target version: 1.5

## Summary

In a production environment it is generally expected to have three failure domains where three replicas
of the data are stored. If any individual failure domain goes down, the data is still available for
reads and writes in the other two failure domains.

For environments that only have two failure domains available where data can be replicated, we need to
also support the case where one failure domain is lost and the data is still fully available in the
remaining failure domain.

To support this scenario, Ceph has integrated support for stretch clusters with an arbiter mon
as seen in [this PR](https://github.com/ceph/ceph/pull/35906) and in the following design docs:
- [Stretch clusters](https://github.com/ceph/ceph/blob/master/doc/rados/operations/stretch-mode.rst)
- Mon election [strategies](https://github.com/ceph/ceph/blob/master/doc/rados/operations/change-mon-elections.rst) and [design](https://github.com/ceph/ceph/blob/master/doc/dev/mon-elections.rst)

Rook will enable the stretch clusters when requested by the admin through the usual Rook CRDs.

## Architecture

To enable the stretch cluster based on the Ceph architecture:
- Rook requires three zones
- Two zones (A and B) will each run all types of Rook pods. We call these the "data" zones.
    - Two mons run in each data zone for two reasons:
      - The OSDs can only connect to the mon in their own zone so we need more than one mon in the data zones.
      - A zone is considered down by Ceph when the mons in the zone are all unavailable.
- The third zone (Arbiter):
    - Runs a single mon, called the "tiebreaker" or "arbiter"
    - No other Rook or Ceph daemons will be run in the arbiter zone

![Stretch Cluster](stretchcluster.png)

The arbiter zone will commonly contain just a single node that is also a K8s master node,
although the arbiter zone may certainly contain more nodes.

The type of failure domain used for stretch clusters is commonly "zone", but can be set to a different failure domain.

### Latency

Distributed systems are impacted by the network latency between critical components.
In a stretch cluster, the critical latency is in the Etcd servers configured with Kubernetes.
K8s only supports latency of up to 5ms (10ms round trip), which is lower than the latency
requirements for any Ceph components. Ceph mons can handle higher latency, designed for up to
700ms round trip.

## Design

### Domain Failure Configuration

The topology of the K8s cluster is to be determined by the admin, outside the scope of Rook.
Rook will simply detect the topology labels that have been added to the nodes.

If the desired failure domain is a "zone", the `topology.kubernetes.io/zone` label should
be added to the nodes. Any of the [topology labels](https://rook.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology)
supported by OSDs can be used.

In the minimum configuration, two nodes in each data zone would be labeled, while one
node in the arbiter zone is required.

For example:

```yaml
topology.kubernetes.io/zone=a
topology.kubernetes.io/zone=a
topology.kubernetes.io/zone=b
topology.kubernetes.io/zone=b
topology.kubernetes.io/zone=arbiter
```

### Rook Cluster Design

The core changes to rook are to associate the mons with the required zones.

- Five mons are required. No other count of mons is supported for stretch clusters.
- One mon will be assigned to the arbiter zone
- Two mons will each be assigned to the data zones
  - The two mons within each zone will have node antiaffinity

The new configuration is found under the `stretchCluster` configuration where the
three zones must be listed and the arbiter zone is identified.

```yaml
  mon:
    count: 5
    allowMultiplePerNode: false
    stretchCluster:
      # The cluster is most commonly stretched over zones, but could also be stretched over
      # another failure domain such as datacenter or region. Must be one of the labels
      # used by OSDs as documented at https://rook.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology.
      failureDomainLabel: topology.kubernetes.io/zone
      zones:
      # There must be exactly three zones in this list, with one of them being the arbiter
      - name: arbiter
        arbiter: true
      - name: a
      - name: b
```

The operator will track which mons belong to which zone. The zone assignment for each mon
will be stored in the `rook-ceph-mon-endpoints` configmap in the same data structure where the host
assignment is stored for mons with node affinity.

For example, if the zones are called
"arbiter", "zone1", and "zone2", the configmap would contain:

```
data:
  data: a=<IP_ADDRESS>:6789,b=<IP_ADDRESS>:6789,c=<IP_ADDRESS>:6789,d=<IP_ADDRESS>:6789,e=<IP_ADDRESS>:6789
  mapping: '{"node":{"a":"arbiter","b":"zone1","c":"zone1","d":"zone2","e":"zone2"}}'
  maxMonId: "4"
```

If no zones are listed in the `stretchCluster.zones`, it is not considered a stretch cluster.
If anything other than 3 zones, it would be an error and the cluster would not be configured.

### Mon Failover

Mon failover will replace a mon in the same zone where the previous mon failed.

### Mon Storage

The mons can all be backed by a host path or a PVC, similar to any other Rook cluster.

In this example, all the mons will be backed by the same PVC:

```yaml
  mon:
    count: 5
    allowMultiplePerNode: false
    stretchCluster:
      zones:
      - name: arbiter
        arbiter: true
      - name: a
      - name: b
    volumeClaimTemplate:
      spec:
        storageClassName: gp2
        resources:
          requests:
            storage: 10Gi
```

It is possible that the mons will need to be backed by a different type of storage in different zones.
In that case, a volumeClaimTemplate specified under the zone will override the default storage setting for
the mon.

For example, the arbiter could specify a different backing store in this manner:

```yaml
  mon:
    count: 5
    allowMultiplePerNode: false
    stretchCluster:
      zones:
      - name: arbiter
        arbiter: true
        volumeClaimTemplate:
          spec:
            storageClassName: alternative-storage
              resources:
                requests:
                  storage: 10Gi
      - name: a
      - name: b
    volumeClaimTemplate:
      spec:
        storageClassName: gp2
        resources:
          requests:
            storage: 10Gi
```

Question: Is there a need for a dataDirHostPath to be specified for the mons in one zone
when the other mons are using the PVCs? For now, we assume all the mons are either using
a PVC or a host path, but not a mix.

### Ceph Config

Rook will set the mon election strategy to the new "connectivity" algorithm.

```console
mon election default strategy: 3
```

The mons will be configured so Ceph will associate the mons with the correct failure domains.
Rook will set each mon location to the zone name

```console
$ ceph mon set_location <mon> <zone>
```

The stretch mode will be enabled with the command:

```console
$ ceph mon enable_stretch_mode tiebreaker_mon <mon> new_crush_rule <rule> dividing_bucket zone
```

### Pools

For data protection in the stretch cluster, all pools should be created with the following configuration, including
pools for an rbd storage class (CephBlockPool), shared filesystem (CephFilesystem), or object store (CephObjectStore).
- Replica: 4
- Failure domain: zone
- Two replicas of data are stored in each zone using a special CRUSH rule specified by `replicasPerFailureDomain: 2`

```yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: stretchedreplica
  namespace: rook-ceph
spec:
  failureDomain: zone
  replicated:
    size: 4
    replicasPerFailureDomain: 2
```

See [this issue](https://github.com/rook/rook/issues/5591) for more details on implementation of the CRUSH rule.

Erasure coded pools are not supported currently in Ceph stretch clusters.

repository: rook/rook

repo_name: rook/rook