text:
---
title: Git, the Engine Room Tour
---
=id="title"

# Git: The Engine Room Tour

![an engine](images/RR_Merlin_labeled.jpg)

*Ash Wilson, Rackspace, DRG*

<!--
intended for people who use git, but are somewhat mystified by it
not an introduction: that would be a half-day workshop or so
not a laundry list of git commands
-->

---
=data-y="1000" id="weird"

## Git is *weird*.

No, really.

* Lots of **strange nomenclature**.
* Odd **division of labor** among commands.
* Strange **error messages**.

Especially if you learned it by rote!

<!--
devs have two learning modes: "quick as possible" and "in-depth"
git is pretty elegant if you learn it in-depth;
 it's ugly and weird if you learn it as quick as possible!
you're not alone!
-->

---
=data-y="2000" id="outline"

# Outline

1. The Object Database
2. The Index and The Working Copy
3. Remotes
4. The Git Way

<!--
here's the plan: start with git's internals, its data structures and concepts;
then introduce how git sees your code;
then cover shipping code around the network;
and finally touch on some more philosophic points.
-->

---
=data-x="2000" data-y="1500" data-scale="2" id="object-database-title"

# The Object Database

<!--
this is how git stores things internally.
everything else in git works by interacting with these constructs.
-->

---
=data-x="2000" data-y="2500" id="dotgit"

# .git

git stores *everything* in a single `.git/` directory.

* local **configuration**
* your **full history**
* **branches**, **tags**, the works

<!--
git is distributed, so every repository is independent and self-contained.
just about everything is plain-text, maybe compressed.
if you're brave, you can drive git using nothing but a text editor!
-->

---
=data-x="2000" data-y="3500" id="objects"

# .git/objects

at git's heart is a filesystem-based **content-addressable database**.
if you peek inside, here's what you'll see:

```bash
$ find .git/objects/ -type f
.git/objects/05/b6bc38da385b07652d2d09123719193e77b1b8
.git/objects/0f/3a0b8da1742a091c822376291abceb01cda3d6
.git/objects/13/a6fbc30e1e85a2641083427bc4b2075a45ebf3
.git/objects/15/796067f1a8d405eb728206d06fef3c100ce496
.git/objects/1b/72f857b356aa5622374593fa78bff9a585a42e
.git/objects/25/9e166d36e8a9e8a0235af3a1aa2d3a2592ed1a
```

<!--
"things" in regular files. each name 40 hex chars, a sha1 checksum of its contents
deflate-compressed.
-->

---
=data-x="2000" data-y="4500" id="blobs"

# .git/objects: blobs

### files you store in git become *blobs*

![blob diagram](images/blob.png)

<!--
format: "blob\t{size in bytes}\n{content}"
no metadata: no filename, mode, whatever
-->

---
=data-x="2000" data-y="5500" id="trees"

# .git/objects: trees

### directories are stored as *trees*

![tree diagram](images/tree.png)

<!--
list of sha1 checksums of other tree objects or blobs
stored as simple, \0-separated records: kind, type, sha1, name
-->

---
=data-x="2000" data-y="6500" id="commits"

# .git/objects: commits

### *commits* are snapshots of the root tree at a particular point in time

![commit diagram](images/commit.png)

<!--
commits point to:
0..n parent commits;
a root tree;
your name, email;
and your commit message.
then these form a big graph. all git does is work with this graph!
-->

---
=data-x="2000" data-y="7500" id="refs"

# .git/refs: branches and tags

### *branches* and *tags* store a commit's address

![branch and tag diagram](images/branches.png)

### `.git/HEAD` is a *pointer to a branch*

## ... and that's basically everything

<!--
literally a text file holding a sha1 hash in hex
branch: "moves along" when you make new commits
tag: just a marker
HEAD tells git the "current branch" so it knows which one to move!
it is literally a text file that holds "refs/heads/master"
-->

---
=data-x="3000" data-y="1500" data-scale="2" id="index-working-copy-title"

# the Index and the Working Copy

---
=data-x="3000" data-y="2500" id="three-trees"

# the Index and the Working Copy

while you're working, there are three places code can be:

![three trees](images/three-trees.png)

`git status` will tell you what's in each.

<!--
git has this thing called the "index" between HEAD and your code on disk.
basically, this lets you make partial commits.
-->

---
=data-x="3000" data-y="3500" id="add"

# Working Copy -> Index

![git add](images/git-add.png)

`git add -- path/to/file`: stage one file at a time

`git add -p`: stage diffs

<!--
"git add" moves changes from your working copy to the index.
you can stage parts of files, the whole directory
-->

---
=data-x="3000" data-y="4500" id="committing"

# Index -> .git/objects

![git commit](images/git-commit.png)

`git commit` persists the index in `.git/objects`, makes a new commit object,
and points the `HEAD` branch to it.

<!--
"git commit" makes a new commit out of whatever you've staged with "git add".
notice that it drags `HEAD` along with it.
-->

---
=data-x="3000" data-y="5500" id="checkout"

# .git/objects --> Working Copy

![git checkout](images/git-checkout.png)

`git checkout -- path/to/file`: check out a single file

`git checkout -p`: check out individual diffs

<!--
"git checkout" turns your working copy (and index) into another state, like
  a branch.
if the state you name is a branch, it also moves HEAD!
"detached HEAD" means "you checked out something that can't move"
"git reset --hard" is similar: but it yanks HEAD along with you.
-->

---
=data-x="2500" data-y="6500" id="naming"

# *Sidenote:* How to Name Things

git gives you a bunch of ways to name commits.

* giant 40-character sha1 checksums: `ca0d1ce34c23e9c2c875ae7c33a742e12ad1ff82`
* any unique prefix: `ca0d1c`
* branches or tags: `master`, `my-feature`
* parent-of operator: `my-feature^`, `master^^`, `some-branch~5`
* by time, absolute or relative: `master@{1 day ago}`
* regexp search on commit messages: `:/broken`

`git help rev-parse` has the full list!

<!--
almost all of the time you use ref names... but these other ones are handy too.
-->

---
=data-x="3000" data-y="7500" id="merge"

# Merging

a **merge** is how you unify work from two *(or more)* branches.

in essence, `git merge` means:

> "Make the branch I'm on right now also include the work of this other thing I name."

what actually happens during the merge depends entirely on **the topography** of history between
*where you are* and *what you're merging in*.

<!--
git is really good at merging things.
a lot of the time, it Just Works. it's still helpful to know what's going on to reason about it.
commit parent pointers == "this work builds on this other work"

does *not* depend on:
- timestamps
- in which clone commits happen
-->

---
=data-x="3000" data-y="8500" id="nothing-to-merge"

# Merging: Trivial Cases

## "Nothing to merge"

when the branch you're on is already a **direct descendant** of the branch you're merging in.

![nothing to merge](images/nothing-to-merge.png)

<!--
because your parent pointers already lead to the commit you're trying to merge,
  your work *already includes* the other work.
so, there's nothing to do.
-->

---
=data-x="3000" data-y="9500" id="fast-forward"

# Merging: Trivial Cases

## "Fast-forward merge"

when the branch you're on is a **direct ancestor** of the branch you're merging in.

![fast-forward merge](images/fast-forward-merge.png)

<!--
you can suppress this with the `--no-ff` flag (and always make a new merge commit)
  pull requests on GitHub do this.
  important sometimes because you *want* to say "i merged this" in your history.
you can force *only* this to happen with `--ff-only`.
  merge will fail if it's not trivial.
  say, if you're on "master" and you never want to have any local work.
sidenote: these are the only merges that can happen in a bare clone!
-->

---
=data-x="3000" data-y="10500" id="recursive"

# Merging: Recursive

this is the most common non-trivial case: a **three-way merge**.

the end result is a **new commit** with two parents: the previous `HEAD` and the commit that you
gave to `git merge`.

<!--
the new commit is important:
- it memorializes that you did this merge in the logs.
- it makes the history graph correct for further merges!
other "merge strategies" are available: ours, theirs, octopus
-->

---
=data-x="3000" data-y="11500" id="recursive-img"

# Merging: Recursive

![recursive merge](images/recursive-merge.png)

<!--
here's how a three-way merge works:
- walk up commit parent pointers to find the common ancestor
- compare diffs from the common ancestor to the top, bottom
- where only one side did a thing: apply that thing
- where both sides did a thing: conflict time
then the new commit goes on the end.
files are tracked by *content*, not *path* (80% similarity)
- seamless rename detection!
"recursive" because, if the common ancestor is ambiguous, you do a "shadow merge" with the
  candidates first
-->

---
=data-x="3000" data-y="12500" id="conflict"

# Merging: Conflicts

if both lines of work modify the same part of the same file, you get a conflict:

```php
one
one
<<<<<<< HEAD
one, changed on master
=======
one, changed on branch
>>>>>>> branch
one
one
```

to *resolve* it:

1. edit the file to the state you want
2. `git add path/to/resolved-file`
3. `git commit` to finalize the merge

<!--
you can just eye it up, or there are a bunch of tools that make this easier.
kdiff3, meld, ...
(or my Atom package ;-) )
"git status" will remind you how to continue if you forget!
-->

---
=data-x="3000" data-y="13500" id="rebase"

# Rebasing

an alternative to merging is to **rebase**.

`git rebase` means:

> "Make it look like this other work was already done before I started mine."

<!--
merge v. rebase is a purely aesthetic workflow choice.
  i'll get into when either is "appropriate" later.
lots of bikeshedding about this.
why would you ever do this?
- it keeps your history legible! a straight line instead of a bunch of back-and-forth.
- no extra merge commit.
-->

---
=data-x="3000" data-y="14500" id="rebase-img"

# Rebasing

![git rebase](images/rebasing.png)

<!--
here's how a rebase works:
- check out the target branch
- "play back" (meaning create new) commits from ancestor -> former HEAD, one at a time
- if any conflict: stop; stage; git rebase --continue
notice that the new commits have different parents than the old ones.
  so, they have *different names* and are *different commits*
  rebase *rewrites history!*
`rebase -i` opens $EDITOR with a list of commits.
- you can delete some, move them around, smash them together
- basically a Swiss army knife for mucking with history
- very very powerful
- also a good way to check that you're rebasing the right stuff
-->

---
=data-x="4000" data-y="1500" data-scale="2" id="remotes-title"

# Remotes

<!--
now we're going to actually touch the network.
-->

---
=data-x="4000" data-y="3500" id="network-of-clones"

# A Network of Clones

git clones can shuttle work to and from other clones by URL.

![network of clones](images/clone-network.png)

some clones are **bare** clones, meaning they have no working copy.

a **remote** is just shorthand for the URL of another clone somewhere.

<!--
git provides a way to get eventual consistency around this whole graph.
"work" here means commits and trees and blobs and refs.
clones with a working copy are for getting work done;
  bare clones are access points for sharing.
every git clone carries a pointer to the one that it was created from called "origin."
the clones in the network don't *have* to share history, but it's helpful when they do!
-->

---
=data-x="4000" data-y="4500" id="remote-tracking"

# remote-tracking branches

the last-known state of branches in other clones is tracked by special refs
called **remote-tracking branches**.

```php
.git/refs/remotes/origin/master
.git/refs/remotes/origin/otherbranch
.git/refs/remotes/upstream/master
.git/refs/remotes/upstream/work-it
```

*(usually you can name these as `origin/master`)*

<!--
these are just refs like your own branches.
they are read-only! if you check one out directly, you'll see a "detached HEAD" message.
instead, you can:
- merge them in
- rebase on top of them
-->

---
=data-x="4000" data-y="5500" id="fetch"

# git fetch

`git fetch <somewhere> <something>`

![git-fetch](images/git-fetch.png)

> "download anything new from another clone and update my remote-tracking branches."

does **not** touch local branches until you `git merge origin/master`.

<!--
"somewhere" is a remote name or a full URL.
"something" is a remote branch name or a "refspec".
- usually you leave it off to get "everything".
git fetch will never trash your local work in any way.
all it does is pull more data in.
then, you can diff it, merge it, rebase it, whatever your workflow is.
-->

---
=data-x="4000" data-y="6500" id="pull"

# git pull

`git pull <somewhere> <something>`

convenient shortcut that combines a fetch and merge.

local branches can also **track** another branch, to always pull from the same place.

<!--
some people will tell you to never pull and always fetch + merge.
really though, it's just a way to save keystrokes for something you're going to do anyway.
i would say: don't pull until you understand it :-)
you can also do `git pull --rebase`, `git pull --ff-only`, `git pull --no-ff`, ...
there is one difference: remote tracking branches are not updated when you pull directly.
  instead FETCH_HEAD is.
tracking is a big source of confusion. really it's just a configuration switch that says:
"what do i do when someone runs `git pull` without args on this branch?"
-->

---
=data-x="4000" data-y="7500" id="push"

# git push

`git push <somewhere> <something>`

![git-push](images/git-push.png)

> "send everything new to another clone."

generally, you only push to *bare clones*.

<!--
push new work somewhere else, so someone else can pull it later.
it'll also update a remote ref.
"-u" or "--upstream" marks this as the new "upstream" and starts tracking it.
-->

---
=data-x="5000" data-y="1500" data-scale="2" id="git-way-title"

# the Git Way

<!--
a few more philosophical points about effective git usage.
-->

---
=data-x="5000" data-y="2500" id="small-commits"

# 1. many small commits

in git, `commit` *records* a state, while `push` *shares* a state

 * don't wait to be "done" before you commit!

 * don't "shotgun commit"

 * commit *complete thoughts*

 * *communicate* through history

<!--
beginners tend to ignore the index, work for a month, and jam everything in one giant commit.
all of git's tooling works better when you commit in smaller chunks.
- following someone's work through a PR
- git bisect
- merging works more reliably
- lots of git commands only work with a clean working directory
think about committing as "making changes to advance state", not "this state is good"
also: .git/objects is *write-only* in normal operations.
the *only way* i have seen lost data in git is when someone is too afraid to commit.
-->

---
=data-x="5000" data-y="3500" id="public-history"

# 2. don't change public history

a variety of git commands will *rewrite history*.

don't rewrite history that's "public".

"public" may mean different things depending on your team's workflow!

<!--
anything that changes existing commits changes history:
- git commit --amend, git rebase, git filter-branch are the usual suspects.
a simple, conservative approach is "don't rebase it if it's pushed".
really it's "don't rebase it if someone else is using it".
- rebase "my-feature-branch" all you want; don't rebase "dev"
- on your own private fork? do whatever you want
-->

---
=data-x="5000" data-y="4500" id="branch-like-mad"

# 3. branches are your friends

git branches are **cheap** and **disposable**.

do separate work on separate branches!

pay attention to *where you are* when you commit.

<!--
want to try something quick? make a local branch. doesn't work? kill it
most git workflows isolate features to individual branches. this is helpful!
don't make pull requests from master -> master; do my-feature -> master instead.
- that way you can easily make more before you're accepted!
- make sure you start new ones at `upstream/master`.
-->

---
=data-x="5000" data-y="5500" id="github-flow"

# 4. pick a workflow
## ...and roll with it

I'm personally fond of [the GitHub flow](https://guides.github.com/introduction/flow/index.html). it's
great for continuous delivery or continuous deployment.

alternatives exist! pick one that meets your needs and run with it. things to consider:

* what does it mean for code to be in different branches?
* merging or rebasing when it's time to integrate?
* which is more important: clear history, or easy process?
* how hard is it for newcomers to contribute?

<!--
prime opportunity for bikeshedding, just like tabs vs. spaces.
GitHub flow is roughly:
* master is always ready for deployment. or, actually deployed.
* feature branch.
* make pull requests early, for discussion.
* merge when they're ready.
"git flow" is another, which is more suited for doing the tagged release thing.
(much heavier-weight though.)
-->

---
=data-x="6000" data-scale="3" data-rotate-y="180" id="questions"

# Questions?

## resources and links

* [Pro Git](http://git-scm.com/book/en/) by Scott Chacon
* [GitHub help](https://help.github.com/) for git and GitHub help
* [git ready](http://gitready.com/) for bite-sized git tips and tricks
* [try git](https://try.github.io/) for an interactive step-by-step tutorial

send me issues or pull requests at:

https://github.com/smashwilson/git-engineroom-tour

repository: smashwilson/git-engineroom-tour

repo_name: smashwilson/git-engineroom-tour


text:
<img src="icon.png" alt="Icon" data-linktype="relative-path" height="100" width="100">

# Eventbrite (Preview)

Eventbrite is a self-service ticketing platform used to create and discover local events.

# Actions

## Create event
Create a new Eventbrite event.

**Operation Id:** CreateEvent

#### Parameters
| Summary | Type | Description | Required |
|---------|------|-------------|----------|
| Name | string | The name of the event. | true |
| Description | string | The description on the event page. | true |
| Start | string | Start time (UTC format Ex. "2017-01-12T13:45:30Z"). | true |
| End | string | End time (UTC format Ex. 2017-01-12T16:45:30Z). | true |
| Start timezone | string | Timezone of the event. | true |
| End timezone | string | Timezone of the event. | true |
| Event currency | string | Currency for this Event's transactions. | true |
| Organizer | string | The ID of the organizer of this event. | false |
| Venue | string | A previously-created venue to associate with this event. | false |
| Category | string | The category of the event. | false |
| Password | string | Password needed to see the event in unlisted mode. | false |
| Capacity | string | Set specific capacity (if omitted, sums ticket capacities). | false |
| Shareable | boolean | If users can share the event on social media. | false |
| Invite Only | boolean | Only invited users can see the event page. | false |
| Online Event | boolean | Whether the event is online-only (no venue). | false |
| Listed | boolean | If the event is publicly listed and searchable. | false |
| Hide start date | boolean | Whether the start date should be hidden. | false |
| Hide end date | boolean | Whether the end date should be hidden. | false |
| Show Remaining | boolean | If the remaining number of tickets is publicly visible on the event page. | false |

#### Returns
| Summary | Type | Description |
|---------|------|-------------|
| [CreateEvent_Response](#createeventresponse) | object |  |

___

## Update event
Update an existing event.

**Operation Id:** UpdateEvent

#### Parameters
| Summary | Type | Description | Required |
|---------|------|-------------|----------|
| Event | string | Event to update. | true |
| Name | string | The name of the event. | false |
| Description | string | The description on the event page. | false |
| Start | string | Start time (UTC format Ex. "2017-01-12T13:45:30Z"). | false |
| End | string | End time (UTC format Ex. 2017-01-12T16:45:30Z). | false |
| Start timezone | string | Timezone of the event. | false |
| End timezone | string | Timezone of the event. | false |
| Event currency | string | Currency for this Event's transactions. | false |
| Organizer | string | The ID of the organizer of this event. | false |
| Venue | string | A previously-created venue to associate with this event. | false |
| Category | string | The category of the event. | false |
| Password | string | Password needed to see the event in unlisted mode. | false |
| Capacity | string | Set specific capacity (if omitted, sums ticket capacities). | false |
| Shareable | boolean | Set specific capacity (if omitted, sums ticket capacities). | false |
| Invite Only | boolean | Only invited users can see the event page. | false |
| Online Event | boolean | Whether the event is online-only (no venue). | false |
| Listed | boolean | If the event is publicly listed and searchable. | false |
| Hide start date | boolean | Whether the start date should be hidden. | false |
| Hide end date | boolean | Whether the start date should be hidden. | false |
| Show Remaining | boolean | If the remaining number of tickets is publicly visible on the event page. | false |

#### Returns
| Summary | Type | Description |
|---------|------|-------------|
| [CreateEvent_Response](#createeventresponse) | object |  |

___

# Triggers

## When an event is created
Triggers a flow when an organizer has a new event.

**Operation Id:** OnNewEvent

#### Parameters
| Summary | Type | Description | Required |
|---------|------|-------------|----------|
| Organizer | string | Organizer to trigger on new events. | true |

#### Returns
| Summary | Type | Description |
|---------|------|-------------|
| [GetEventsForOrganizer_Response](#geteventsfororganizerresponse) | array |  |

___

## When an order changes
Trigger a flow when an order changes.

**Operation Id:** OnOrderChanged

#### Parameters
| Summary | Type | Description | Required |
|---------|------|-------------|----------|
| Event | string | Event to trigger on changed order. | true |

#### Returns
| Summary | Type | Description |
|---------|------|-------------|
| [GetOrders_Response](#getordersresponse) | array |  |

___

## Objects

### GetOrganizers_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| Organizer | array of object |  | organizers |
| Id | string | The id of the organizer. | organizers.id |
| Name | string | The name of the organizer. | organizers.name |

### GetMyVenues_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| Venue | array of object |  | venues |
| Id | string | The id of the venue. | venues.id |
| Name | string | The name of the venue. | venues.name |

### GetEventsForOrganizer_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| Name | string | The name of the event (in text form). | name.text |
| Description | string | The description of the event (in text form). | description.text |
| Id | string | The id of the event. | id |
| URL | string | The URL of the event. | url |
| Timezone | string | The timezone of the start of the event. | start.timezone |
| local | date-time | The start date-time of the event in local time. | start.local |
| UTC | date-time | The start date-time of the event in UTC. | start.utc |
| Timezone | string | The timezone of the end of the event. | end.timezone |
| local | date-time | The end date-time of the event in local time. | end.local |
| UTC | date-time | The end date-time of the event in UTC. | end.utc |
| Created Date-Time | date-time | The date-time the event was created at. | created |
| Chaged Date-Time | date-time | The date-time the event was changed at. | changed |
| Capacity | int32 | The capacity of the event. | capacity |
| Is Online Event | boolean | A boolean indicating whether or not the event is online. | online_event |
| Organizer Id | string | The id of the organizer of the event. | organizer_id |
| Venue Id | string | The id of the venue of the event. | venue_id |
| Category Id | string | The id of the category of the event. | category_id |

### GetOrders_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| Id | string | The id of the order. | id |
| Name | string | The name of the order. | name |
| First Name | string | The first name on the order. | first_name |
| Last Name | string | The last name on the order. | last_name |
| Email | string | The email on the order. | email |
| Status | string | The status of the order. | status |

### GetMyEvents_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| events | array of object |  | events |
| Name | string | The name of the event (in text form). | events.name.text |
| Id | string | The id of the event. | events.id |

### GetCategories_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| categories | array of object |  | categories |
| Id | string | The id of the category. | categories.id |
| Name | string | The name of the category. | categories.name |

### CreateEvent_Response

| Summary | Type | Description | Path |
|---------|------|-------------|------|
| Name | string | The name of the event (in text form). | name.text |
| Description | string | The description of the event. | description.text |
| Id | string | The id of the event. | id |
| URL | string | The URL of the event. | url |
| Timezone | string | The timezone of the start of the event. | start.timezone |
| local | date-time | The start date-time of the event in local time. | start.local |
| UTC | date-time | The start date-time of the event in UTC. | start.utc |
| Timezone | string | The timezone of the end of the event. | end.timezone |
| local | date-time | The end date-time of the event in local time. | end.local |
| UTC | date-time | The end date-time of the event in UTC. | end.utc |
| Created Date-Time | date-time | The date-time the event was created at. | created |
| Chaged Date-Time | date-time | The date-time the event was changed at. | changed |
| Capacity | int32 | The capacity of the event. | capacity |
| Is Online Event | boolean | A boolean indicating whether or not the event is online. | online_event |
| Organizer Id | string | The id of the organizer of the event. | organizer_id |
| Venue Id | string | The id of the venue of the event. | venue_id |
| Category Id | string | The id of the category of the event. | category_id |

## Limits
| Name | Calls | Renewal Period |
|------|-------|----------------|
| API calls per connection | 100 | 60 seconds |
| Frequency of trigger polls | 1 | 120 seconds |

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# README Generator

## Description
This app generates a README for your project

## Table of Contents
- [Installation](#installation)
- [Usage](#usage)
- [License](#license)
- [Contributing](#contributing)
- [Tests](#tests)
- [Questions](#questions)

## Installation
Provide a detailed list of instructions for install

## Usage
Provide instructions on how to use the project

## License
![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)

## Contributing
Provide external resources that you used for your project

## Tests
Writes tests and explain how to run tests for your project

## Questions
Having an issue? Email me with any questions about the app<EMAIL_ADDRESS>

branch_name: refs/heads/main


text:
---
layout: post
title: "Web Scraping and Analyzing Home Rentals (Python + R)"
date: 2020-12-11 6:30:00
excerpt: "Scraping Long Beach Island Summer Rentals with Python and the R reticulate package."
author: "David Lucey"
categories: [Business]
tags: [R-Blogger, Learn-Python, Python, Scraping, reticulate]
image: /assets/2020-12-10-python-web-scraping/python-scraping-cover.jpg
image_preview: /assets/2020-12-10-python-web-scraping/python-scraping-preview.jpg
canonical: https://redwallanalytics.com/2020/06/07/scraping-long-beach-island-summer-rentals-with-python/
---

👉 Each month, we release tons of great content on R for Business. [__Register for our blog to get fresh R Tips straight to your inbox.__](https://mailchi.mp/business-science/blog-registration)

# Introduction

This tutorial showcases how to use Python with R `reticulate`.

When I started this mini-project, I hoped to use `datatable` as our main data frame in conjunction with the Python libraries like `BeautifulSoup` and data structures not available in R, like dictionaries. I soon learned that `datatable` doesn’t support dates yet. In [Python and R - Part 1: Exploring Data with Datatable](/code-tools/2020/10/30/python-and-r-exploring-data.html), we noted that `datatable` is still in alpha stage, and worked around the lack of reshaping capability and the inability to pipe our data directly into plots, but this really a deal breaker for this project. As a result, though we were able to keep using `plotnine` and pandas.

{% include cta_rtrack.html %}

## Code Setup

{% highlight r %}
# R Libraries
library("reticulate")

knitr::opts_chunk$set(
  fig.width = 15,
  fig.height = 8,
  out.width = '100%')
# Choose Python 3.7 miniconda
reticulate::use_condaenv(
  condaenv = "r-reticulate",
  required = TRUE
  )
# Install Python packages
lapply(c("bs4", "requests", "plotnine", "mizani"), function(package) {
       conda_install("r-reticulate", package, pip = TRUE)
})

from bs4 import BeautifulSoup
import requests
import itertools
import plotnine as p9
from mizani.breaks import date_breaks
from mizani.formatters import date_format
import pandas as pd
import numpy as np
import re
import time
import json as JSON
import pickle
import datetime
import pprint
{% endhighlight %}

# Scraping Process

We found vacationrentalslbi.com has an extensive list of rentals, and doesn’t restrict to the particular agency maintaining the listing, so this is the perfect place to gather our data. Naturally, we would like to be polite, so **check that we are allowed to scrape**. We can see from the R `robotstxt` library, which indicates that `paths_allowed` is `TRUE`, that we are good to go with our intended link.

{% highlight r %}
# Robots.txt says okay to scrape
robotstxt::paths_allowed('https://www.vacationrentalslbi.com/search/for.rent/sleeps_min.4/')
{% endhighlight %}

{% highlight r %}
## [1] TRUE
{% endhighlight %}

Scraping the website for listings is a two-step process:

1. Go through and extract the links to all of the listings
2. Navigate back to all of the links and extract the listing details

We set our requests to the website to vary at an average 5-second delay, and build a list of the ‘href’ links from the returned ‘a’ tags. We are not running our scraping code now for the blog post, but the results is shown in the link below loaded from disc.

{% highlight r %}
# Extract links for min sleep of 4 from vacationrentalslbi.com website
links = []
for i in range(0, 1250, 25):
  if i == 0:
     url = 'https://www.vacationrentalslbi.com/search/for.rent/sleeps_min.4/'
  else:
      url = 'https://www.vacationrentalslbi.com/search/for.rent/sleeps_min.4/' + str(i)
  html_content = requests.get(url).text

  # Parse the html content
  soup = BeautifulSoup(html_content, 'lxml')
  time.sleep(5)
  links.append([link['href'] for link in soup.find_all('a', href=True) if 'listing.' in link['href']])

# Extract links
links = list(set(itertools.chain(*links)))
listings = [link for link in links if not '#' in link]
{% endhighlight %}

{% highlight r %}
# Load pre-scraped listing links from disc
path = '/Volumes/davidlucey/aDL/data/lbi/'
file = 'listings.txt'
# Pickle load listings
with open(path + file, "rb") as fp:   # Unpickling
  listings = pickle.load(fp)

# First 5 listings
listings[0:4]
{% endhighlight %}

{% highlight r %}
## ['https://www.vacationrentalslbi.com/listing.464', 'https://www.vacationrentalslbi.com/listing.165', 'https://www.vacationrentalslbi.com/listing.353', 'https://www.vacationrentalslbi.com/listing.453']
{% endhighlight %}

## Get Listings from Home

Now, we build a second scraper to take the list of listings, extract the key elements of each and return a dictionary which we store in a list. We won’t go into detail here, but the way to find the desired classes is to navigate to the vacationrentalslbi.com on Google Chrome, select Ctrl-Alt-I, choose the ‘Select Element’ option in the ‘Elements’ pane, and then navigate to the desired spot on the page.

We selected the title, content, description, location, and calendar sub-element tables for ‘booked’ and ‘available’ from the ‘month_box’. It took some work to get the calendar. We then returned a dictionary with all of these elements from our `get_dict` function.

{% highlight r %}
# Function to take listing, scrape and return key elements as dictionary
def get_dict(listing):

  # Extract html text
  html_content = requests.get(listing).text
  # Parse the html content
  soup1 = BeautifulSoup(html_content)

  # Title and attributes
  title = soup1.find_all("div", class_= "col-md-3 title")
  title = [item.text for item in title]
  content = soup1.find_all("div", class_= "col-md-9 content")
  content = [item.text for item in content]
  d = {title[i]: content[i] for i in range(len(title))}

  # Description and location
  try:
    description = soup1.find("p").get_text()
  except:
    description = None
  d['description'] = description
  try:
    location = soup1.find("div", attrs={"class" : "ld_location"}).get_text()
  except:
    location = None
  d['location'] = location

  # Extract full calendar
  availability = soup1.find_all("div", class_ = "month_box col-sm-4 col-xs-12")
  table_rows = [item.table for item in availability]

  # Extract booked
  l = []
  for tr in table_rows:
    td = tr.find_all('td', class_ = "booked")
    rows = [tr.text for tr in td if tr]
    for row in rows:
      if row != '':
          rows.remove(row)
      l.append(rows)
  l = list(itertools.chain.from_iterable(l))
  df1 = pd.DataFrame(l)
  df1["status"] = "booked"

  # Extract available
  l = []
  for tr in table_rows:
    td = tr.find_all('td', class_ = "available")
    rows = [tr.text for tr in td if tr]
    for row in rows:
      if row != '':
          rows.remove(row)
      l.append(rows)
  l = list(itertools.chain.from_iterable(l))
  df = pd.DataFrame(l)
  df["status"] = "available"

  # Combine 'booked' and 'available' in calendar
  calendar = pd.concat(list([df,df1]))
  calendar[["start_day","start_date","hyphen", "end_date","period"]] = calendar[0].str.split(expand=True)
  calendar[["end_date", "rate"]] = calendar["end_date"].str.split("$", expand=True)

  # Clean up
  del calendar['hyphen']
  del calendar['start_day']
  del calendar[0]
  calendar = calendar.drop_duplicates()

  # Convert to dictionary for return
  d['calendar'] = calendar.to_dict()
  return d
{% endhighlight %}

When an element of a listing is not present, we were having breaks, so we put in exception handling for those cases. Although we think we have handled most of the likely errors in get_dict, the full scraping process takes a couple of hours, so we thought best to save to disc after each request. It took us a while how to figure this out, because it turns out not to be so straight-forward to save and append a json to disc. We were able to write to disc as txt as we do in `append_record` below.

{% include cta_rtrack.html %}

## Scrape All Listings

With our `get_dict` function, we scrape each listing, create a dictionary entry and append it to disc with `append_record`.

{% highlight r %}
# Loop through listings with get_dict and add to Google Drive
for listing in listings:

  # Initiate dictionaries
  entry = {}
  details = {}

  # Extract listing details with `get_dict` function above
  try:
    details = get_dict(listing)
  except:
    details = None

  # Take break
  time.sleep(5)

  # Get `listing_id` to add as dictionary key
  listing_id = re.findall('\d+', listing)[0]

  # Create dictionary `entry`
  # Then append to lbi.txt file on disc with `append_record` function above
  entry = {listing_id:  details }
  try:
    append_record(entry)
  except:
    print(listing_id + ' Failed')
{% endhighlight %}

Again, we wanted to avoid re-running the code, we are showing our saved data from disc. We load the saved data from our text file as a list of 1231 Python dictionaries. The dictionary for a sample listing of ‘464’ is shown in the chunk below. The attributes of the listing are deeply nested and not easy to filter and sort. However, we learned that it is easy to extract the desired elements using the dictionary keys, which we do in the `get_calendar` function below.

{% highlight r %}
# Load lbi.txt back into notebook as list of dictionaries
filename = 'lbi.txt'
with open(path + filename) as fh:
  lbi = [JSON.loads(line) for line in fh]

# Show listing '464' dictionary
pp = pprint.PrettyPrinter(depth=4)
pp.pprint(lbi[0])
{% endhighlight %}

{% highlight r %}
## {'464': {' Payment options': 'Cash, Checks',
##          'Access': ' Stairs',
##          'Bathroom(s)': ' 2 Bathroom(s)Toilet / Shower: 1Toilet / Tub / '
##                         'Shower: 1',
##          'Bedroom(s)': ' 3 Bedroom(s),\n'
##                        '8 SleepsBunk Beds (twin / twin): 1King: 1Queen: 1Sleep '
##                        'Sofa (queen): 1Trundle: 1Twin / Single: 1',
##          'Entertainment': ' DVD Player Game Room Ping Pong Table Satellite '
##                           '/Cable Television',
##          'Indoor Features': ' Blender Central Air Coffee Maker Cooking '
##                             'Utensils Dining Area Dishes & Utensils Dishwasher '
##                             'Internet Keurig Kitchen Living Room Microwave '
##                             'Oven Refrigerator Stove Toaster Vacuum',
##          'Local Activities & Adventures': ' Basketball Cycling Deep Sea '
##                                           'Fishing Fishing Golf Jet Skiing '
##                                           'Paddle Boating Photography Pier '
##                                           'Fishing Rafting Roller Blading '
##                                           'Sailing Scenic Drives Sight Seeing '
##                                           'Snorkeling Surf Fishing Surfing '
##                                           'Swimming Tennis Walking Water '
##                                           'Skiing Water Tubing Wind Surfing',
##          'Location Type': ' Ocean Block Oceanfront Oceanside',
##          'Outdoor Features': ' Balcony / Terrace Beach Badges Community Pool '
##                              'Deck / Patio Heated Pool Outdoor Grill Sun Deck',
##          'Popular Amenities': ' Air Conditioning Pool Washer / Dryer WiFi',
##          'Property Type': ' Condo',
##          'Security deposit': '$300',
##          'Suitability': '  Pets Welcome: No  Smoking Allowed: No smoking  '
##                         'GREAT for Kids: Yes  Winter/Seasonal Rentals: No  Not '
##                         'Many Stairs: Two or more Flights  Wheelchair '
##                         'Accessible: No Parties/events not allowed',
##          'Themes': ' Beach Vacation Family Vacations',
##          'calendar': {'end_date': {'0': '06/26/2020',
##                                    '12': '10/02/2020',
##                                    '202': '08/07/2020',
##                                    '205': '08/14/2020',
##                                    '208': '08/21/2020',
##                                    '21': '07/10/2020',
##                                    '211': '09/11/2020',
##                                    '24': '07/17/2020',
##                                    '27': '07/24/2020',
##                                    '3': '07/03/2020',
##                                    '30': '07/31/2020',
##                                    '376': '09/18/2020',
##                                    '380': '09/25/2020'},
##                       'period': {'0': '/Week',
##                                  '12': '/Week',
##                                  '202': '/Week',
##                                  '205': '/Week',
##                                  '208': '/Week',
##                                  '21': '/Week',
##                                  '211': None,
##                                  '24': '/Week',
##                                  '27': '/Week',
##                                  '3': '/Week',
##                                  '30': '/Week',
##                                  '376': '/Week',
##                                  '380': '/Week'},
##                       'rate': {'0': '2,600',
##                                '12': '1,500',
##                                '202': '3,395',
##                                '205': '3,395',
##                                '208': '3,395',
##                                '21': '3,395',
##                                '211': None,
##                                '24': '3,395',
##                                '27': '3,395',
##                                '3': '3,195',
##                                '30': '3,395',
##                                '376': '1,500',
##                                '380': '1,500'},
##                       'start_date': {'0': '06/20/2020',
##                                      '12': '09/26/2020',
##                                      '202': '08/01/2020',
##                                      '205': '08/08/2020',
##                                      '208': '08/15/2020',
##                                      '21': '07/04/2020',
##                                      '211': '08/22/2020',
##                                      '24': '07/11/2020',
##                                      '27': '07/18/2020',
##                                      '3': '06/27/2020',
##                                      '30': '07/25/2020',
##                                      '376': '09/12/2020',
##                                      '380': '09/19/2020'},
##                       'status': {'0': 'booked',
##                                  '12': 'available',
##                                  '202': 'booked',
##                                  '205': 'booked',
##                                  '208': 'booked',
##                                  '21': 'booked',
##                                  '211': 'booked',
##                                  '24': 'booked',
##                                  '27': 'booked',
##                                  '3': 'booked',
##                                  '30': 'booked',
##                                  '376': 'booked',
##                                  '380': 'booked'}},
##          'description': '"FISHERY FUN" The Fishery is an oceanfront complex '
##                         'that offers so much for the LBI Vacationer. What to '
##                         'do first? Take a swim in the heated pool, enjoy the '
##                         'game room or head to the beach, so many options! '
##                         'After a day of fun in the sun take a stroll into town '
##                         '(close to Ship Bottom and Surf City) to enjoy a meal '
##                         'or do some shopping. This 3 bedroom first floor unit '
##                         'offers a lot of value and fun for the money.  About '
##                         'Sand Dollar Real Estate, Manager  Since 1983, Sand '
##                         'Dollar Real Estate has been successfully assisting '
##                         'LBI vacationers find their perfect summer beach '
##                         'house. We represent over 100 LBI vacation homes and '
##                         'offer secure accounting procedures (including '
##                         'security) in accordance with NJ State Statues. Our '
##                         'local office is open 7 days a week to respond to any '
##                         'issues or questions you might have about your '
##                         'vacation home. All of our guests receive a FREE Goody '
##                         'Bag upon arrival, filled with sweet treats, useful '
##                         'commodities and information on what to do, where to '
##                         'eat and a lot more on how best to enjoy your stay on '
##                         'LBI.  Sand Dollar Real Estate’s experienced staff is '
##                         'here to help you select the best vacation home for '
##                         'you and your family. Our Concierge Service will '
##                         'listen to what you are looking for, ask questions and '
##                         'then search to find the perfect match based on our '
##                         'years of experience. Our staff knows our homes well '
##                         'and can offer you very specific information about our '
##                         'properties. We update our listings weekly to be sure '
##                         'our information is accurate.  It is nice to know that '
##                         'once you book, Sand Dollar Real Estate is available 7 '
##                         'days a week to respond to any issues or questions you '
##                         'might about your vacation home or your vacation stay '
##                         'in general.  We look forward to sharing our Island '
##                         'with your family this Summer.',
##          'location': ' Condo in Long Beach Island, Ship Bottom'}}
{% endhighlight %}

## Parse Listings

Using `get_calendar`, we extract the dictionary key for the listing, and then the value desired value elements including ‘rate’, ‘start_date’, ‘end_date’, ‘location’, ‘location_type’ and ‘beds’. We have to clean and transform the ‘rate’ variable to float and the date fields to datetime, and in our case, we are looking for the first two weeks of August, so we filter for just those two weeks. We also add the url back in so it is easy to take a look at an interesting listing in more detail.

We also manufactured some variables for our graphs below. For example, we generated a ‘month-year’ variable so we could aggregate weekly average prices and number of homes available. There were too many different sleep capacities, so we aggregated into just four levels (sleeps 4 or under, 8 or under, 12 or under and more than 12). Beach Haven has 7-8 separate small sections, so we changed to just one.

{% highlight r %}
# Extract availability calendar from disc
def get_calendar(listing):
  key, value = list(listing.items())[0]
  if value is not None:
    data = listing.get(key)['calendar']
    df = pd.DataFrame.from_dict(data)

    # Parse variables in pandas
    df['rate'] = df['rate'].str.replace(',', '').astype(float)
    df['date'] = pd.to_datetime(df['start_date'], errors='ignore')
    df['listing'] = 'https://www.vacationrentalslbi.com/listing.' + key
    try:
      df['location'] = listing[key]['location']
    except:
      df['location'] = None
    df['type'] = df['location'].str.extract('([A-Z][a-z]+)\s')
    df['city'] = df['location'].str.extract('\,\s(.*)')
    df['city'] = df['city'].str.replace('.*Beach Haven.*', 'Beach Haven')
    try:
      df['location_type'] = listing[key]['Location Type']
    except:
      df['location_type'] = None
    try:
      df['beds'] = listing[key]['Bedroom(s)']
    except:
      df['beds'] = None
    df['bedrooms'] = df['beds'].str.extract('(\d+)').astype(int)
    df['sleeps'] = df['beds'].str.split('\,\s').str[1].str.extract('(\d+)').astype(int)
    df['sleeps_bin'] = pd.cut(df['sleeps'], [0, 4, 8, 12, 100])
    return(df)
{% endhighlight %}

We loop through our dictionary and use our `get_calendar` function to extract all of our listings.

{% highlight r %}
data = pd.DataFrame()
for i in range(0,len(lbi)):
  data_new = get_calendar(lbi[i])
  data = data.append(data_new, ignore_index = True)
{% endhighlight %}

In the table below, we can see the mean rental rate and number of units available by month. July has the fewest available among the months of the peak period, and also the highest rates. We can also that the average size of houses rented is higher outside the peak period.

{% highlight r %}
# Summary table of 2020 rental average rates and counts by month
table = data.set_index('date')['2020'].resample('M').agg(['mean', 'count'])
table[table.notnull()]
{% endhighlight %}

{% highlight r %}
##                    rate        bedrooms           sleeps
##                    mean count      mean count       mean count
## date
## 2020-01-31  1396.875000     8  3.750000     8   9.500000     8
## 2020-02-29  2600.000000     1  3.000000     1   6.000000     1
## 2020-03-31  5300.000000     4  4.000000     4   9.500000     4
## 2020-04-30  4298.461538    13  4.307692    13  11.615385    13
## 2020-05-31  3327.782407   216  3.842593   216   9.685185   216
## 2020-06-30  4548.799439  1426  3.851049  1430   9.606294  1430
## 2020-07-31  5714.932432   888  3.884400   891   9.653199   891
## 2020-08-31  5079.022660  1015  3.822439  1025   9.477073  1025
## 2020-09-30  3056.289474  1064  3.815299  1072   9.542910  1072
## 2020-10-31  2267.789683   252  4.019841   252   9.888889   252
## 2020-11-30  1471.580000    50  4.500000    50  11.240000    50
## 2020-12-31  1493.181818    22  4.363636    22  11.000000    22
{% endhighlight %}

## Location Inflation

We had hoped to segment and consider the prices for Oceanside, Ocean block, Bayside block and Bayfront, but landlords interpret the meaning of “Oceanside” liberally. We tend to think of that term as looking at the water from your deck, but ~60% of rentals are designated in this category, when true “Oceanside” can’t be more than 10%. In most cases, landlords probably mean Ocean block, but there is not a lot we can do to pick this apart. We also don’t have the exact addresses, so we are probably out of luck to find anything useful in this regard.

{% highlight r %}
data.location_type.value_counts(normalize = True)[1:10]
{% endhighlight %}

{% highlight r %}
##  Bayside                             0.216226
##  Oceanside                           0.174151
##  Ocean Block Oceanfront Oceanside    0.146604
##  Oceanfront                          0.061321
##  Bayfront Bayside                    0.048868
##  Ocean Block                         0.044906
##  Bayside Lagoon                      0.020377
##  Bayfront                            0.017736
##  Oceanfront Oceanside                0.010189
## Name: location_type, dtype: float64
{% endhighlight %}

## Biggest Rental Towns by Volume

By far the most rental action is in the Beach Haven sections in July and August (shown in orange), but those sections also have more year-round availability than the the other towns. If the plan is to go with less than 8 people, there is not a lot of options. In fact, it was surprising to see that there was more available in the largest sleeps >12 than the family of four category. As mentioned in our previous post about `plotnine`, the lack of support for plotly hovering is a bit of a detraction here, because it can be hard to tell which color denotes which city. This might mean we have to learn `seaborn` in the future, just as we have been forced to learn `pandas` for this post.

{% highlight r %}
filtered_data = data.set_index('date')['2020'].groupby([pd.Grouper(freq='M'), 'city', 'sleeps_bin'])['rate'].count().reset_index()
(p9.ggplot(filtered_data,
      p9.aes(x = 'date',
             y = 'rate',
             group='factor(city)',
             color = 'factor(city)')) +
      p9.geom_smooth() +
      p9.theme_bw() +
      p9.labs(
        title = 'Most Listings by Far in Aggregated Beach Haven Sections',
        subtitle = 'Listed from Smallest to Largest Sleep Capacity',
        x = 'Month',
        y = 'Monthly Rental Volume',
        color = 'City'
        ) +
      p9.scale_x_datetime(breaks=date_breaks('1 month'), labels=date_format('%m-%Y')) +
      p9.theme(
        axis_text_x=p9.element_text(rotation=45, size=6),
        subplots_adjust={'bottom': 0.20},
        figure_size=(10, 3), # inches
        aspect_ratio=1/1.5,    # height:width
        legend_position='bottom',
        legend_direction='horizontal') +
      p9.facet_wrap('~sleeps_bin', ncol = 2)
)
{% endhighlight %}

{% highlight r %}
## <ggplot: (-9223363274524820999)>
##
## /Users/davidlucey/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/plotnine/stats/smoothers.py:168: PlotnineWarning: Confidence intervals are not yet implementedfor lowess smoothings.
##   "for lowess smoothings.", PlotnineWarning)
{% endhighlight %}

![](/assets/2020-12-10-python-web-scraping/available-by-town-1.png)

## Availability vs Booked by City

Beach Haven has more B&B’s and some of the only hotels on the Island, so smaller size properties on average and somewhat less consistent visitors. More rentals outside of Beach Haven are probably renewed annually, so it might be more impacted by delayed plans due to COVID-19 than other towns. Coupled with it being about as big as all the other towns put together, this may help explain why also shows a lot more relatively more red at this stage.

{% highlight r %}
filtered_data = data[data['city'].notnull()].set_index('date')['2020']
(p9.ggplot(filtered_data,
    p9.aes('rate',
    group = 'status',
    fill = 'status')) +
    p9.geom_histogram(position ='stack') +
    p9.theme_bw() +
    p9.labs(
      title = "Most Rentals Booked Across Range of Prices for Early August",
      x = 'Weekly Rate ($)',
      y = 'Number of Bookings',
      fill = 'Status'
      ) +
    p9.theme(
      axis_text_x=p9.element_text(rotation=45, hjust=1),
      subplots_adjust={'right': 0.75},
      figure_size=(10, 4), # inches
      aspect_ratio=1/1.5,    # height:width
      legend_position='right',
      legend_direction='vertical') +
    p9.facet_wrap('~city')
)
{% endhighlight %}

{% highlight r %}
## <ggplot:<PHONE_NUMBER>746)>
##
## /Users/davidlucey/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/plotnine/stats/stat_bin.py:93: PlotnineWarning: 'stat_bin()' using 'bins = 104'. Pick better value with 'binwidth'.
##   warn(msg.format(params['bins']), PlotnineWarning)
## /Users/davidlucey/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/plotnine/layer.py:360: PlotnineWarning: stat_bin : Removed 25 rows containing non-finite values.
##   data = self.stat.compute_layer(data, params, layout)
{% endhighlight %}

![](/assets/2020-12-10-python-web-scraping/rental-histogram-1.png)

# Prices for Booked Properties Peaking in July

2020 might not be a typical year with the uncertainty around COVID-19, but the fall off in prices starting in August, when there appears to be more supply, is shown here. Landlords may have pulled supply for July when things looked uncertain and then put it back on at the last minute. It also looks like the available properties might be in that category, because they are priced higher than comparable properties. At least for the bigger properties, the posted prices of available properties are clearly higher than for the booked ones. Let’s face it, if you haven’t booked your property sleeping more than 8 by now, it might be tough for most groups of that size to organize at this late stage.

{% highlight r %}
filtered_data = data.set_index('date')['2020'].groupby([pd.Grouper(freq='M'), 'status', 'sleeps_bin'])['rate'].mean().reset_index()
(p9.ggplot(filtered_data,
      p9.aes(x = 'date',
             y = 'rate',
             group = 'factor(sleeps_bin)',
             color = 'factor(sleeps_bin)'
             )) +
      p9.geom_smooth() +
      p9.theme_bw() +
      p9.scale_x_datetime(breaks=date_breaks('1 month'), labels=date_format('%m-%Y')) +
      p9.labs(
        title = "Prices for Available Rentals Falling Off Steadily After July",
        x = 'Month',
        y = 'Average Weekly Rate ($)',
        color = 'Sleep Bin'
        ) +
      p9.facet_wrap('~status') +
      p9.theme(
        axis_text_x=p9.element_text(rotation=45, hjust=1, size=6),
        subplots_adjust={'bottom': 0.20},
        figure_size=(10, 3), # inches
        aspect_ratio=1/1.4,    # height:width
        legend_position='bottom',
        legend_direction='horizontal')
)
{% endhighlight %}

{% highlight r %}
## <ggplot: (-9223363274522058934)>
##
## /Users/davidlucey/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/plotnine/stats/smoothers.py:168: PlotnineWarning: Confidence intervals are not yet implementedfor lowess smoothings.
##   "for lowess smoothings.", PlotnineWarning)
{% endhighlight %}

![](/assets/2020-12-10-python-web-scraping/rate-by-sleeps-1.png)

## Homegenous Prices Across Cities

For anyone who has been to LBI, it is pretty much nice everywhere. Accept for maybe Loveladies, there aren’t really premium towns in the sense of the NYC suburbs. Loveladies shown in light blue can be seen towards the higher end, but still among the pack. The main distinction is if the house is beachfront or not, but unfortunately, we don’t have a good source of that data at this stage. The rents for the largest homes does show quite a bit more divergence among towns than the other three categories.

{% highlight r %}
filtered_data = data.set_index('date')['2020'].groupby([pd.Grouper(freq='M'), 'city', 'sleeps_bin'])['rate'].mean().reset_index()
(p9.ggplot(filtered_data,
      p9.aes(x = 'date',
             y = 'rate',
             group = 'factor(city)',
             color = 'factor(city)'
             )) +
      p9.geom_smooth() +
      p9.theme_bw() +
      p9.scale_x_datetime(breaks=date_breaks('1 month'), labels=date_format('%m-%Y')) +
      p9.labs(
        title = "All Sized Rental Prices Peak in July for Most Towns",
        x = 'Month',
        y = 'Average Weekly Rate ($)',
        color = 'City'
        ) +
      p9.facet_wrap('~sleeps_bin') +
      p9.theme(
        axis_text_x=p9.element_text(rotation=45, hjust=1, size=6),
        subplots_adjust={'bottom': 0.30},
        figure_size=(10, 4), # inches
        aspect_ratio=1/1.5,    # height:width
        legend_position='bottom',
        legend_direction='horizontal')
)
{% endhighlight %}

{% highlight r %}
## <ggplot:<PHONE_NUMBER>201)>
##
## /Users/davidlucey/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/plotnine/stats/smoothers.py:168: PlotnineWarning: Confidence intervals are not yet implementedfor lowess smoothings.
##   "for lowess smoothings.", PlotnineWarning)
{% endhighlight %}

![](/assets/2020-12-10-python-web-scraping/rate-town-sleeps-1.png)

# Conclusion

Most families are constrained to July and early August, but for those with the freedom to go at other times, there is a lot of opportunity to have a great vacation at an affordable price! We also know that vacationrentalslbi.com also operates sites for Wildwood, North Wildwood, Wildwood Crest and Diamond Beach, so it our scraper would probably work the same for all of those. Now that we have the code, we can parse listings whenever considering a vacation at the Jersey Shore.

Author: David Lucey, Founder of [Redwall Analytics](https://redwallanalytics.com)

David spent 25 years working with institutional global equity research with several top investment banking firms.

{% include cta_rtrack.html %}


text:
# Pre-work - Todo

*Todo* is an android app that allows building a todo list and basic todo items management functionality including adding new items, editing and deleting an existing item.

Submitted by: Emilie Brisseau (github user: kri55is)

Time spent: 14 hours spent in total

## User Stories

The following **required** functionality is completed:

* [x] User can **successfully add and remove items** from the todo list
* [x] User can **tap a todo item in the list and bring up an edit screen for the todo item** and then have any changes to the text reflected in the todo list.
* [x] User can **persist todo items** and retrieve them properly on app restart

The following **optional** features are (not yet) implemented:

* [x] Persist the todo items [into SQLite](http://guides.codepath.com/android/Persisting-Data-to-the-Device#sqlite) instead of a text file
* [ ] Improve style of the todo items in the list [using a custom adapter](http://guides.codepath.com/android/Using-an-ArrayAdapter-with-ListView)
* [ ] Add support for completion due dates for todo items (and display within listview item)
* [ ] Use a [DialogFragment](http://guides.codepath.com/android/Using-DialogFragment) instead of new Activity for editing items
* [ ] Add support for selecting the priority of each todo item (and display in listview item)
* [x] Tweak the style improving the UI / UX, play with colors, images or backgrounds

The following **additional** features *could be* implemented:

Customize style with:
* [x] splashscreen
* [x] original icon

Have a way to tell the item has been done by one of the following solutions:
* [ ] have a check button with every item in the list view
* [ ] being able to "cross" an item by swiping it (as a user would do to cross a liine on a paper with a pen). The item would be displayed but striked through.

* [ ] Being able to diplay or not the item that have been done
* [ ] Being able to rearrange items in the list with drag n'drop
* [ ] Being able to create different kind of lists: work/personnal/...
* [ ] Better handling of errors and exceptions
* [ ] Better logging system using more TAGs with LogCat

## Video Walkthrough

Here's a walkthrough of implemented user stories:

<img src='http://i.imgur.com/qURwgdi.gif' width="600" />

GIF created with [LiceCap](http://www.cockos.com/licecap/).

## Project Analysis

As part of your pre-work submission, please reflect on the app and answer the following questions below:

**Question 1:** "What are your reactions to the Android app development platform so far? Compare and contrast Android's approach to layouts and user interfaces in past platforms you've used."

**Answer:** I had used Android Studio in the past to build small apps and I enjoyed the different approaches it gives: we can either go with the UI or edit the xml manually. In the past I didn't use Genymotion to emulate a mobile use, I find it very easy to use and a lot faster
than the built-in emulator. I also use a real android phone because even if I can only verify if all is all right on this particulat phone and Android version it gives the real impression because in the real life users don't use mouse.
I find it very confortable to use the tools you recommanded when they are finally all installed (some packaged on Android Studio take a very long time).

Except the Android apps I already built in the past I have mostly wrote code without interface so it is difficult for me to compare. I can say that I really enjoy building Android app also because we can see results very fast and that it can be used by everybody.

**Question 2:** "Take a moment to reflect on the `ArrayAdapter` used in your pre-work. How would you describe an adapter in this context and what is its function in Android? Why do you think the adapter is important? Explain the purpose of the `convertView` in the `getView` method of the `ArrayAdapter`."

**Answer:** The ArrayAdapter is in between a ListView and the data to be seen on the screen. In order to have a data displayed we need a view but there is no need to create a view for all the data if they are not always displayed. The adapter is ther for that: to create and handle the necessary views efficiently and populate them with the data. Then returning the view to the ListView.
The convertView is very usefull to not create views that were already created, we give the old view as parameter so we don't create an un-necessary object.

## Notes
I had to debug the app because it crashed only when it was launched the first time and we were trying to read items in an empty file. My list of items was null so I had to add checks on that.
I plan on adding more features :)

## License

    Copyright 2017 Emilie Brisseau

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.


text:
👮🏻👮🏻👮🏻 !!!!  DO NOT FORGET !!!! 👮🏻👮🏻👮🏻

# PR Checklist:

- [ ] Have you read the [CONTRIBUTING.md](https://github.com/crypto-com/chain-nodelib/blob/master/CONTRIBUTING.md)?
- [ ] Does your PR follow the [C4 patch requirements](https://rfc.zeromq.org/spec:42/C4/#23-patch-requirements)?
- [ ] Have you rebased your work on top of the latest master?
- [ ] Have you checked your code compiles? (`npm run build`)
- [ ] Have you checked your code formatting is correct? (`npm run lint`)
- [ ] If you are contributing for the first time, please read the agreement in [CONTRIBUTING.md](https://github.com/crypto-com/sample-chain-wallet/blob/master/CONTRIBUTING.md) now and add a comment to this pull request stating that your PR is in accordance with the [Developer's Certificate of Origin](https://github.com/crypto-com/sample-chain-wallet/blob/master/CONTRIBUTING.md#developer-certificate-of-originn).

Thank you for your code, it's appreciated! :)

repository: foreseaz/cro-nodelib

gha_license_id: NOASSERTION

repo_name: foreseaz/cro-nodelib


text:
# Lohwagia intermedia (Speg.) Petr., 1942 SPECIES

#### Status
ACCEPTED

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Bot. Arch. 43: 205 (1942)

#### Original name
Phyllachora intermedia Speg., 1885

### Remarks
null

repository: mdoering/backbone

repo_name: mdoering/backbone


text:
# NOTE:May 13-2018: Composer 0.19.3 and above are breaking the code due to incompatibility
# with latest version of grpc (0.11.1) I have hardcoded the version of grpc
# to (0.10.1) ... will fix once the issue has been resolved by the developers

# NOTE: Composer 0.19.0 have introduced breaking changes.
# April 2018 : I am in the process of updating code
#              If you see an issue please send me a note raj at ACloudFan.com
#              Files that have been updated/tested for 0.19.0 has a comment ontop

How to use this?
Most of the files may be executed against a local fabric insallation.
Some samples are coded to work with embedded runtime.
Lectures in the course refer to the samples and describe how to launch the samples

0. Install    > npm install
1. Launch your local fabric runtime
2. Deploy the App against which you would like to test the API (airlinev7 by default)
       1.   > composer network start -a<EMAIL_ADDRESS>-c PeerAdmin@hlfv1 -A admin -S adminpw
       2.   > composer network start -a<EMAIL_ADDRESS>-c PeerAdmin@hlfv1 -A admin -S adminpw
       3.   > composer card delete admin@airlinev7
       4.   > composer card import -f<EMAIL_ADDRESS>3. Samples are using airlinev7. If you would like to use some other you need to change the code
4. On terminal prompt   >  node code-sample-file.js

license: Unlicense

repository: oamat/hyperledgerFabric

repo_name: oamat/hyperledgerFabric


text:
# qgis-geo-png-db
A QGIS plugin for generating [geo-png-db](https://github.com/sasakiassociates/geo-png-db) tiles.

## Using
Copy the `geo_png_tiler` folder to your QGIS plugins folder.

## Building
in Makefile, update this with your plugin dir
`QGISDIR=[ENTER YOU QGIS PLUGIN DIR HERE]`

follow instructions in geo_png/README.txt

repository: sasakiassociates/qgis-geo-png-db

repo_name: sasakiassociates/qgis-geo-png-db


text:
# Harp2hugo
[![Gem Version](https://badge.fury.io/rb/harp2hugo.svg)](http://badge.fury.io/rb/harp2hugo)

Gem for converting posts from [Harp](https://github.com/sintaxi/harp) [metadata json](http://harpjs.com/docs/development/metadata) into [Hugo](https://github.com/spf13/hugo) [Front Matter](https://gohugo.io/content/front-matter/) format.

Used in [https://www.yunspace.com/](https://www.yunspace.com/)

## Usage

    $ gem install harp2hugo
    $ harp2hugo [input_file] [output_dir]

Where input_file defaults to `_data.json` and output_dir defaults to `output`.
NOTE: the output dir will be created relative to your input file directory.

    $ harp2hugo public/posts/_data.json output

Logic is to loop through each post specified in `_data.json` file:

 1. Reads in the metadata in JSON format
 2. Convert metadata to YAML format and prepend to the post
 3. Write the new post with prepended metadata into output directory.

## Disclaimer

Currently still work in progress. Written with plenty of help from [@nagliyvred](https://github.com/nagliyvred)

## Contributing

1. Fork it ( https://github.com/yunspace/harp2hugo/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request


text:
---
title: February 20, 2020
date: 2020-02-20
slug: february-20-2020

---
### Fixed

* Locations: The "Help" text button for "Location Type" drop-down field doesn't match the screen size, overlaps the field
* Marketplace: When adding a Custom product, on step 3, the "Next" button is staying gray - you need to select another unit type, then the default "Unit" and the "Next" button turns blue

### Updated

* Inventory: When in the "grid" mode, hide "eye" button on the far right on each product line

### New

repository: bcllns/uptracker-releases

gha_language: Vue

gha_license_id: MIT

repo_name: bcllns/uptracker-releases


text:
## Vernacular resources database
*[temporary name]*

Epigraphy, signs, vernacular letters, etc.
*Collecting the collectors...*

Backup repository and project planning.

Ressources:
* [online](./online.md) [websites, blogs, articles, Flickrs, Tumblrs, ...]
* [printed](./printed.md) [books, manuals, albums, ...]
* [afk](./afk.md) [museums, galleries, places, archives, ...]

## Contributors

[@pascalduez](https://twitter.com/pascalduez),
[@hardwig](https://twitter.com/hardwig),
[@fritzgroegel](https://twitter.com/fritzgroegel)

## License

[![CC0](http://mirrors.creativecommons.org/presskit/buttons/88x31/svg/cc-zero.svg)](https://creativecommons.org/publicdomain/zero/1.0/)

To the extent possible under law, [Pascal Duez](https://github.com/pascalduez) has waived all copyright and related or neighboring rights to this work.

repository: anatoletype/vdb

repo_name: anatoletype/vdb


text:
## Blackspace Core Metrics Library
---
```bash
go get github.com/Lens-Platform/Platform/src/libraries/core/core-metrics
```

Requires:

* Go >= 1.12

This document outlines how to effectively make use of this library.

```go
// define a core engine registry object to which the version info would be tied to
CoreEngine := NewCoreMetricsEngineInstance("blackspace_platform",nil)

// Define a
LoginRequestCounter = NewGaugeVec(&GaugeOpts{
        Namespace: "blackspace_platform"
		Subsystem:  "authenticationservice",
		Name:      "login_request_counter",
		Help:      "Number of log in requests",
	}, []string{"Request"})

CoreEngine.RegisterMetric(LoginRequestCounter)

LoginRequestCounter.WithLabelValues("Request").Observe(1.0)
```


text:
# WMLAction

WMLAction is WML parser and modifier. WML modifications described as simple extension to WML. If you have many WML files and want to do some modifications to them all, then you can give it a try.

 Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request


text:
# Time Complexity

When there is more than one way to solve a problem. We need to learn how to compare
their performances to choose the best one.
The exact time needed for a program to finish executing depends on lots of things like hardware, operating system, processors, etc. However, we don't consider any of these factors while analyzing an algorithm. We will only consider the execution time of the algorithm with respect to the size of input fed to the program. This measurement is called "Time Complexity" or "Order of Growth".

## Lecture

What you will learn:
- What is time complexity?
- How to measure time complexity?
- Big O.
- Examples on the most common time complexity classes.
- Introduction to the brute force technique through some practice problems.

Watch the following video.

[![](https://img.youtube.com/vi/sOYBdcsMePE/mq1.jpg)](https://www.youtube.com/watch?v=sOYBdcsMePE)

## Practice Problems:

Cut Ribbon: [https://codeforces.com/contest/189/problem/A]

QAQ:  [https://codeforces.com/problemset/problem/894/A]

Worms Evolution: [https://codeforces.com/problemset/problem/31/A]

Hotelier: [https://codeforces.com/problemset/problem/1200/A]

Choosing Laptop: [https://codeforces.com/problemset/problem/106/B]

Two-gram: [https://codeforces.com/problemset/problem/977/B]

**Authors:**
* Osama Fathy


text:
Creating a repository on Github is as simple as clicking the "+" icon and "New
Repository", naming it under "Repository Name". Make sure to check the box next
to "Initialize this repository with a README" to be able to clone it to your local
computer. Copy the HTTPS link in the repository page (it's almost like the URL of
the repository page, but with a ".git" at the end of it).

To clone the repository, that is bring the repository from Github online to your
own computer's drive, open up your terminal/command line. Once you navigate to
the folder you want to put your repository in, type "git clone [insert HTTPS link
here]"

####Describe what open source means.

Open source means that the source code of a program, the code that makes up the
program, is available for anyone to view and hold a copy of. Anyone with a copy
can do various things with the source code, depending on the kind of licence
attached to it.

####What do you think about Open Source? Does it make you nervous or protective? Does it feel like utopia?

Anyone that is putting their code out there has made the choice to do so, so I
don't think there is any reason to be nervous or protective. Open source is great
to get other developers to contribute to your work, build on it, and ultimately
make it better. The first thing that usually comes to mind when I think about
open source after "free program" is "community effort."

####Assess the importance of using licenses.

While open source programs and their source code are generally free to the public,
licenses may offer varying degrees of protection to the source code and/or the
original creator. Some licenses require that credit is given to the creator, others
restrict anyone from making money on the source code.

####What concepts were solidified in the challenge? Did you have any "aha" moments? What did you struggle with?

I was able to get a stronger hold on the git workflow. In the previous challenge,
it was more me following directions and hoping everything worked out. With more
practice, I was able to actually tell myself what was actually going on with each
step in the process.

####Did you find any resources on your own that helped you better understand a topic? If so, please list it.

N/A

gha_language: Ruby


text:
---
title: Adjacency and Neighbors
---

# Adjacency <!-- Comment to get header to display properly, because docfx wont' accept a header of Adjacency properly. -->
GoRogue provides convenient methods to calculate and retrieve neighboring grid locations via the `AdjacencyRule` class. This class also provides easy methods to get the `Direction` instances that lead to those neighbors.

Instances of the `AdjacencyRule` class cannot be created by the user.  Instead, three instances are included as static members of the `AdjacencyRule` class, which represent all possible ways of defining adjacency on a 2d integer-based grid.  Each instance, and how it defines adjacency, is covered below.  Functions of these instances may be used to retrieve the `Direction` instances that lead to adjacent squares, as well as the coordinates of each square adjacent to a given point.

# Methods of Determining Adjacency
The three instances included are `AdjacencyRule,CARDINALS`, `AdjacencyRule.DIAGONALS`, and `AdjacencyRule.EIGHT_WAY`.  These instances and how they define adjacency is covered below.

## AdjacencyRule.CARDINALS
This rule is effectively standard 4-way connectivity -- a location is "adjacent" to another if it is within 1 cardinal step away. In the image below, the red grid location is the starting cell, and the blue grid locations are those that are considered adjacent by this rule:

![cardinal neighbors](~/images/adjacency_and_neighbors/cardinals.PNG)

## AdjacencyRule.DIAGONALS
This rule considers a location "adjacent" to another if it is within one diagonal step away.  Again, in the image below, the red grid location represents the starting cell, and the blue grid locations are those that are considered adjacent by this rule:

![diagonal neighbors](~/images/adjacency_and_neighbors/diagonals.PNG)

## AdjacencyRule.EIGHT_WAY
This rule is effectively standard 8-way connectivity -- a location is "adjacent" to another if it is within 1 step in any direction.  Like in the previous images, in the image below the red grid location represents the starting cell, and the blue grid locations are those that are considered adjacent by this rule:

![eight-way neighbors](~/images/adjacency_and_neighbors/eight_way.PNG)

# Getting Directions of Neighbors
`AdjacencyRule` provides a few functions to allow you to get the directions that lead to neighbors of a location.  The API documentation covers these in detail, however this may serve as a quick overview of their functionality.

The `AdjacencyRule.DirectionsOfNeighbors` function is the most straightforward; it returns you an `IEnumerable<Direction>` with all the directions that lead to neighbors:

```CSharp
foreach (Direction d in AdjacencyRule.EIGHT_WAY.DirectionOfNeighbors())
    System.Console.WriteLine(d);
```

The `AdjacencyRule.DirectionsOfNeighborsClockwise` and `AdjacencyRule.DirectionsOfNeighborsCounterclockwise` functions work the same way, except that they return the directions in clockwise and counter-clockwise order, respectively.  In addition, both of these functions optionally take a `Direction` instance that will be the first direction returned.  Directions are returned in clockwise or counter-clockwise order as appropriate, starting from the given direction:

```CSharp
// Returns directions in order Direction.LEFT, Direction.RIGHT, Direction.DOWN, Direction.UP.
// AdjacencyRule.DirectionsOfNeighborsCounterclockwise functions similarly.
foreach (Direction d in AdjacencyRule.CARDINALS.DirectionsOfNeighborsClockwise(Direction.LEFT))
    System.Console.WriteLine(d);
```

# Getting Locations of Neighbors
`AdjacencyRule` also provides a few functions to allow you to get the coordinates that represent neighbors of a given location. This may prove to be more convenient than using the functions which return `Directions` and adding those `Direction` instances to a position. Again, the API documentation covers these in detail, however this may serve as a quick overview of their functionality.

These functions work very similarly to their `Direction`-based equivalents.  `AdjacencyRule.Neighbors` returns an `IEnumerable<Coord>` for each neighboring position:

```CSharp
Coord position = (1, 2);
foreach (Coord neighbor in AdjacencyRule.EIGHT_WAY.Neighbors(position))
    System.Console.WriteLine(neighbor);
```

The `AdjacencyRule.NeighborsClockwise` and `AdjacencyRule.NeighborsCounterclockwise` functions work the same way, except that they return the coordinates in clockwise and counter-clockwise order, respectively.  In addition, both of these functions optionally take a `Direction` instance, which indicates which position to return first.  Positions are returned in clockwise or counter-clockwise order as appropriate, starting from the position that is located in the given direction:

```CSharp
Coord position = (1, 2);

// Returns directions in clockwise order, starting with the position located in the direction Direction.LEFT.
// AdjacencyRule.NeighborsCounterclockwise functions similarly.
foreach (Direction d in AdjacencyRule.CARDINALS.NeighborsClockwise(position, Direction.LEFT))
    System.Console.WriteLine(d);
```

# Implicit Conversion for Related Types
For any given `Distance` or `Radius` class instance, a method of determining adjacency is implied due to the mathematical concepts those classes represent.  Therefore, both `Distance` and `Radius` instances may be implicitly converted to their corresponding `AdjacencyRule` instance.  As a result of this, any function that accepts a parameter of type `AdjacencyRule` may also simply be passed a `Distance` or `Radius` instance, and that instance will be converted to the corresponding `AdjacencyRule` automatically.

The correspondance between `Distance` instances and `AdjacencyRule` instances is as follows:
- `Distance.MANHATTAN` -> `AdjacencyRule.CARDINALS`
- `Distance.CHEBYSHEV` -> `AdjacencyRule.EIGHT_WAY`
- `Distance.EUCLIDEAN` -> `AdjacencyRule.EIGHT_WAY`

The correspondance between `Radius` instances and `AdjacencyRule` instances is similar:
- `Radius.CIRCLE` or `Radius.SPHERE` -> `AdjacencyRule.EIGHT_WAY`
- `Radius.SQUARE` or `Radius.CUBE` -> `AdjacencyRule.EIGHT_WAY`
- `Radius.DIAMOND` or `Radius.OCTAHEDRON` -> `AdjacencyRule.CARDINALS`

A demonstration of some of these conversions can be seen via the following code:

```CSharp
// Both of these function without compiler error, since implicit conversions are defined.
AdjacencyRule rule = Distance.MANHATTAN;
rule = Radius.SQUARE;

// Assuming a function MyFunc(AdjacencyRule rule) is defined
MyFunc(Distance.CHEBYSHEV);
```

repository: rwindegger/GoRogue

repo_name: rwindegger/GoRogue


text:
# Megami Tensei Fusion Tools - How to Use
## Navigation
* [All Fusion Tools](https://aqiu384.github.io/megaten-fusion-tool/home)
* [Fusion Theory](https://aqiu384.github.io/docs-megaten/fusion-theory)
* [Report Issue](https://github.com/aqiu384/megaten-fusion-tool/issues)

## Save Offline
### iOS and Android
1. Navigate to the [Megami Tensei Fusion Tools homepage](https://aqiu384.github.io/megaten-fusion-tool/home)
2. Add page to Home Screen
    * iOS Safari: Safari > Action (Share) > Add to Home Screen > Add
    * Android Chrome: Chrome > Settings > Add to Home Screen > Add Automatically
3. Open the downloaded page from the Home Screen
4. Navigate to the skills page of each desired game (e.g. [List of Skills - Persona 5 Royal](https://aqiu384.github.io/megaten-fusion-tool/p5r/skills))
5. Close the downloaded page and disable Wi-Fi and Cellular Data
6. Reopen the page from the Home Screen and ensure each downloaded game is still accessible
7. Reenable Wi-Fi and repeat steps 4-6 if game does not load offline


text:
# Testing BCP-003-01 TLS

Testing of certain aspects of BCP-003-01 makes use of an external tool 'testssl.sh'. Please see [testssl/README.md](../testssl/README.md) for installation instructions.

In order to ease testing of TLS with the various specifications, sample certificates are provided in this repository. Please see [test_data/BCP00301/README.md](../test_data/BCP00301/README.md) for their details and installation guidance.

url: https://raw.githubusercontent.com/cristian-recoseanu/nmos-testing/733fb28e98999bf6eae2d537e392c84020e29a14/docs/2.2. Usage - Testing BCP-003-01 TLS.md

repository: cristian-recoseanu/nmos-testing

filename: 2.2. Usage - Testing BCP-003-01 TLS.md

repo_name: cristian-recoseanu/nmos-testing


text:
---
layout: post
title: "Wednesday 01.31.18 -  CrossFit"
teaser: "A.  Power Snatch 6 x 2
 B.  Weighted Pull-Up 3 x 5
 C.  21-15-9 Power Snatches and Pull-Ups
 D. Alternating Sets: GHD Hip Extension, Hollow Flutters"
date: 2018-01-31 00:00:00
categories: wodup
tags: wods wednesday
header: no
---

A.  Power Snatch 6 x 2

Every 90 seconds, for 9 minutes (6 sets):
2 Power Snatches

B.  Weighted Pull-Up 3 x 5

Weighted Pull-Up
– 3 sets of 5 reps
Rest as needed

C.  21-15-9 Power Snatches and Pull-Ups

21-15-9 Reps for Time:
– Power Snatch @ 35/25 kg
– Pull-Up

D. Alternating Sets: GHD Hip Extension, Hollow Flutters

3 sets, for quality,  of:
D1. 6-8 GHD Hip Extensions (2010 tempo)Hold a bar or plate to perform theseD2. Hollow Flutters for 30-45 seconds
View on WodUp

#### Compare to other Wednesdays
{: .t60 }

{% include list-posts tag='wednesday' %}

branch_name: refs/heads/gh-pages

gha_language: JavaScript


text:
![Corda](https://www.corda.net/wp-content/uploads/2016/11/fg005_corda_b.png)

# Corda Training Template

This repo contains all of the instructions and class templates for the practical exercises of the Corda two day
training course.

This repository is divided into two parts: Java templates, and Kotlin templates. You may complete the training in whichever
language you prefer.

# Setup

### Tools
* JDK 1.8 latest version
* IntelliJ latest version (2017.1 or newer)
* git

After installing the required tools, clone or download a zip of this repository, and place it in your desired
location.

### IntelliJ setup
* From the main menu, click `open` (not `import`!) then navigate to where you placed this repository.
* Click `File->Project Structure`, and set the `Project SDK` to be the JDK you downloaded (by clicking `new` and
nagivating to where the JDK was installed). Click `Okay`.
* Next, click `import` on the `Import Gradle Project` popup, leaving all options as they are.
* If you do not see the popup: Navigate back to `Project Structure->Modules`, clicking the `+ -> Import` button,
navigate to and select the repository folder, select `Gradle` from the next menu, and finally click `Okay`,
again leaving all options as they are.

# Instructions
Once you are all set up, you can start expanding on the class templates. This project follows a test-based
development style - the unit tests for each class contain all the information you will need to complete this CorDapp.

All the tests are commented out by default - to complete this training, you will uncomment them one at a time, building up
the CorDapp until it passes everything.

You will begin by opening `IOUState` and `IOUStateTests` in your language of choice and uncommenting the first test. Then, use the TODO and
hints to modify `IOUState` to pass the test.

In order to issue IOU's using Corda - You will first fix `IOUState`, then `IOUContract`, and lastly `IOUIssueFlow`. Then you can move on to
more challenging exercises specified in Transfer and Settle testing files.

### Running the tests
* Kotlin: Select `Kotlin - Unit tests` from the dropdown run configuration menu, and click the green play button.
* Java: Select `Java - Unit tests` from the dropdown run configuration menu, and click the green play button.
* Individual tests can be run by clicking the green arrow in the line number column next to each test.
* When running flow tests you must add the following to your run / debug configuration in the VM options field. This enables us to use
* Quasar - a library that provides high-performance, lightweight threads.
* "-javaagent: /PATH_TO_FILE_FROM_ROOT_DIR/quasar.jar"

# Template Files

### Kotlin
State:

* Template: `kotlin-source/src/test/kotlin/net/corda/training/state/IOUState.kt`
* Tests: `kotlin-source/src/main/kotlin/net/corda/training/state/IOUStateTests.kt`

Contract:

* Template: `kotlin-source/src/main/kotlin/net/corda/training/contract/IOUContract.kt`
* Issue Tests: `kotlin-source/src/test/kotlin/net/corda/training/contract/IOUIssueTests.kt`
* Transfer Tests: `kotlin-source/src/test/kotlin/net/corda/training/contract/IOUTransferTests.kt`
* Settle Tests: `kotlin-source/src/test/kotlin/net/corda/training/contract/IOUSettleTests.kt`

Flow:

* Issue flow template: `kotlin-source/src/main/kotlin/net/corda/training/flow/IOUIssueFlow.kt`
* Issue flow tests: `kotlin-source/src/test/kotlin/net/corda/training/flow/IOUIssueFlowTests.kt`
* Transfer flow template `kotlin-source/src/main/kotlin/net/corda/training/flow/IOUTransferFlow.kt`
* Transfer flow tests: `kotlin-source/src/test/kotlin/net/corda/training/flow/IOUTransferFlowTests.kt`
* Settle flow template `kotlin-source/src/main/kotlin/net/corda/training/flow/IOUSettleFlow.kt`
* Settle flow tests: `kotlin-source/src/test/kotlin/net/corda/training/flow/IOUSettleFlowTests.kt`

The code in the following files was already added for you:

* `kotlin-source/src/main/kotlin/net/corda/training/plugin/IOUPlugin.kt`
* `kotlin-source/src/test/kotlin/net/corda/training/Main.kt`
* `kotling-source/src/main/kotlin/net/corda/training/plugin/IOUPlugin.kt`
* `kotling-source/src/main/java/kotlin/corda/training/flow/SelfIssueCashFlow.kt`

### Java
State:

* Template: `java-source/src/main/java/net/corda/training/state/IOUState.java`
* Tests: `java-source/src/test/java/net/corda/training/state/IOUStateTests.java`

Contract:

* Template: `java-source/src/main/java/net/corda/training/contract/IOUContract.java`
* Issue Tests: `java-source/src/test/java/net/corda/training/contract/IOUIssueTests.java`
* Transfer Tests: `java-source/src/test/java/net/corda/training/contract/IOUIssueTests.java`
* Settle Tests: `java-source/src/test/java/net/corda/training/contract/IOUIssueTests.java`

Flow:

* Issue template: `java-source/src/main/java/net/corda/training/flow/IOUIssueFlow.java`
* Issue tests: `java-source/src/test/java/net/corda/training/flow/IOUIssueFlowTests.java`
* Transfer template: `java-source/src/main/java/net/corda/training/flow/IOUTransferFlow.java`
* Transfer tests: `java-source/src/test/java/net/corda/training/flow/IOUTransferFlowTests.java`
* Settle template: `java-source/src/main/java/net/corda/training/flow/IOUSettleFlow.java`
* Settle tests: `java-source/src/test/java/net/corda/training/flow/IOUSettleFlowTests.java`

The code in the following files was already added for you:

* `java-source/src/main/java/net/corda/training/plugin/IOUPlugin.java`
* `java-source/src/test/java/net/corda/training/NodeDriver.java`
* `java-source/src/main/java/net/corda/training/plugin/IOUPlugin.java`
* `java-source/src/main/java/net/corda/training/flow/SelfIssueCashFlow.java`

# Running the CorDapp
Once your application passes all tests in `IOUStateTests`, `IOUIssueTests`, and `IOUIssueFlowTests`, you can run the application and
interact with it via a web browser. To run the finished application, you have two choices for each language: from the terminal, and from IntelliJ.

### Kotlin
* Terminal: Navigate to the root project folder and run `./gradlew kotlin-source:deployNodes`, followed by
`./kotlin-source/build/node/runnodes`
* IntelliJ: With the project open, select `Kotlin - Node driver` from the dropdown run configuration menu, and click
the green play button.

### Java
* Terminal: Navigate to the root project folder and run `./gradlew java-source:deployNodes`, followed by
`./java-source/build/node/runnodes`
* IntelliJ: With the project open, select `Java - NodeDriver` from the dropdown run configuration menu, and click
the green play button.

### Interacting with the CorDapp
Once all the three nodes have started up (look for `Webserver started up in XXX sec` in the terminal or IntelliJ ), you can interact
with the app via a web browser.
* From a Node Driver configuration, look for `Starting webserver on address localhost:100XX` for the addresses.

* From the terminal: Node A: `localhost:10009`, Node B: `localhost:10012`, Node C: `localhost:10015`.

To access the front-end gui for each node, navigate to `localhost:XXXX/web/iou/`

## Troubleshooting:
When running the flow tests, if you get a Quasar instrumention error then add:

```-ea -javaagent:lib/quasar.jar```

to the VM args property in the default run configuration for JUnit in IntelliJ.

Solutions are available [here](https://github.com/corda/corda-training-solutions).

repository: etaroid/corda-training-template

gha_language: Java

repo_name: etaroid/corda-training-template


text:
# BMI 203 - HW 1

[![Build
Status](https://travis-ci.org/david-joy/bmi203-hw1.svg?branch=master)](https://travis-ci.org/david-joy/bmi203-hw1)

David Joy
1/24/2017

## Implement BubbleSort and Quicksort for integers

BubbleSort is implemented in [bubblesort()](https://github.com/david-joy/bmi203-hw1/blob/master/hw1/algs.py#L12)

Quicksort is implemented in [quicksort()](https://github.com/david-joy/bmi203-hw1/blob/master/hw1/algs.py#L69), with auxilary functions [_quicksort()](https://github.com/david-joy/bmi203-hw1/blob/master/hw1/algs.py#L61) and [_quicksort_partition()](https://github.com/david-joy/bmi203-hw1/blob/master/hw1/algs.py#L36)

## Count the number of assignments

* BubbleSort:
    * `(n outer loops) * (n-1 + n-2 + ... 1 inner loops) * (2 assignments worst case)`
    * `n * (n - 1)/2 * 2 = n**2 + n`
    * `O(n**2)` assignments
* Quicksort:
    * `(2 * (r - p) assigns per partition worst case)`
    * p starts at 0, r at n - 1, so `2 * (r - p) = 2n - 2`
    * `O(n)` total assignments

## Count the number of conditionals

* BubbleSort:
    * `(n outer loops) * (n-1 + n-2 + ... 1 inner loops) * (2 compares)`
    * `n * (n - 1)/2 * 2 = n**2 + n`
    * `O(n**2)` conditionals
* Quicksort:
    * `(r - p compares per partition)`
    * p starts at 0, so `r - p = n`
    * Average case:
        * A random partition splits the array into roughly equal pieces. Assuming all partitions are roughly equal, we recurse `O(log(n))` times.
        * Total conditions `O(n*log(n))`
    * Worst case:
        * A bad partition splits the array into splits of size 1, n - 1. Since each split reduces the size of the array by 1, we recurse `n` times.
        * Total conditions `O(n**2)`

## Test some typical edge cases for sorting and run them on Travis

All tests are implemented in `test/test_algs.py`

Some cases to think of:

* Empty vector - [test_bubblesort_zero_elements()](https://github.com/david-joy/bmi203-hw1/blob/master/test/test_algs.py#L23)
* Single element vector - [test_bubblesort_one_element()](https://github.com/david-joy/bmi203-hw1/blob/master/test/test_algs.py#L32)
* Duplicated elements - [test_bubblesort_many_elements()](https://github.com/david-joy/bmi203-hw1/blob/master/test/test_algs.py#L61)
* Odd vs even length of input vector - [test_bubblesort_many_elements_even()](https://github.com/david-joy/bmi203-hw1/blob/master/test/test_algs.py#L71)

Tests for quicksort are also provided with the same naming convention.

## Test the time complexity of your algorithms as follows:

* For sizes of 100, 200, 300, ... 1000
* Generate 100 random vectors
* Sort them using your code

The sorting simulation is implemented in [plot_time_complexity.py](https://github.com/david-joy/bmi203-hw1/blob/master/plot_time_complexity.py)

The output plot is written to `images/time_complexity.png`

Fitting `O(n**2)` and `O(n log(n))` to each data set, BubbleSort most closely fits `O(n**2)` while Quicksort most closely fits `O(n log(n))` on average.

## Installing

To use the package, first run

```
conda install --yes --file requirements.txt
```

to install all the dependencies in `requirements.txt`. Then the package's
main function (located in `hw1/__main__.py`) can be run as follows

```
python -m hw1
```

## Testing

Testing is as simple as running

```
python -m pytest
```

from the root directory of this project.