text:
# FeynmanDNA Linode VPS for Nginx/ufw/Gunicorn/Flask Guide

## 0. Deployment Note

Workflow: make the change in master branch, then switch to deployment branch and do the merge. **UPLOAD THE DEPLOYMENT BRANCH!** since deployment branch has the configs set up in `__init__`, `run_cpp`, and `generate_path`.

Also remember to copy the PublicationPdfs folder to the build/static/ after `npm run build`

## 1. Useful commands

`hostnamectl`
```
  Operating System: Debian GNU/Linux 9 (stretch)
            Kernel: Linux 4.9.0-6-amd64
      Architecture: x86-64
```

`lscpu`
```
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
Model name:            Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
CPU MHz:               2299.968

```

### Compress an Entire Directory or a Single File

```tar -czvf name-of-archive.tar.gz /path/to/directory-or-file```
Here’s what those switches actually mean:

* `-c`: Create an archive.
* `-z`: Compress the archive with gzip.
* `-v`: Display progress in the terminal while creating the archive, also known as “verbose” mode. The v is always optional in these commands, but it’s helpful.
* `-f`: Allows you to specify the filename of the archive.

### Extract an Archive
```tar -xzvf archive.tar.gz```

### Upgrade pip3
`sudo -H pip3 install --upgrade setuptools pip`

## 2. Firefall config (ufw)

`sudo apt-get install ufw` for debian. UFW is included in Ubuntu by default but must be installed in Arch and Debian. Debian will start UFW’s systemd unit automatically and enable it to start on reboots. Then `sudo ufw enable` => 'Firewall is active and enabled on system startup'

first allow for ssh(22/tcp):
`sudo ufw allow ssh`

and normalize the defaults:
```
sudo ufw default deny incoming
sudo ufw default allow outgoing
```

Perform `sudo ufw status verbose` to see if you're even logging in the first place.
If you're not, perform `sudo ufw logging on` if it isn't.
If you are logging, but there are no `/var/log/ufw*` files, check to see if `rsyslog` is running: `sudo service rsyslog status`
ufw logging file (need sudo to view content):
`/var/log/ufw.log`

To see the list of apps ufw knows about and can use:
`sudo ufw app list`
The app profiles should be stored in `/etc/ufw/applications.d`
create a `nginx` file, and write (if you do not see the `Ngin *` in the `app list` in ufw:
```
[Nginx HTTP]
title=Web Server (Nginx, HTTP)
description=Small, but very powerful and efficient web server
ports=80/tcp
```
then the `ufw app list` will display an entry:
`Nginx HTTP`

Enable Nginx in ufw by typing:
`sudo ufw allow 'Nginx HTTP'`

after the edit:
`sudo ufw --force enable`

to delete ufw rules:
`sudo ufw delete allow ssh`

for the yjg-calculator, need to open port 7717:
`sudo ufw allow 7717`

generally, port 22 (ssh), 80 (http) and 443 (https).

in a typical ufw.log:
- TOS, for Type of service,
- DST is destination ip,
- SRC is source ip
- TTL is time to live, a small counter decremented each time a packet is passed through another router (so if there is a loop, the package destroy itself once to 0)
- DF is "don't fragment" bit, asking to packet to not be fragmented when sent
- PROTO is the protocol (mostly TCP and UDP)
- SPT is the source port
- DPT is the destination port

## 3. NGINX config

All NGINX configuration files are located in the `/etc/nginx/` directory. The **primary** configuration file is `/etc/nginx/nginx.conf`.

first preserve the default `nginx.conf` file so you have something to restore to if your customizations get so convoluted that NGINX breaks.
`sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup-original`

Now let's look at the main `nginx.conf`:

```
user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
```
Usual way of Nginx configuration is to have one worker per CPU core. For a basic virtual private server (VPS), the default value of 1 is a good choice.

the `warn` in `error_log` indicates the minimal severity to enter the log

**Gzip**

gzip on; - enables gzip compression
gzip_vary on: - tells proxies to cache both gzipped and regular versions of a resource
gzip_min_length 1024; - informs NGINX to not compress anything smaller than the defined size
gzip_proxied - compress data even for clients that are connecting via proxies (here we're enabling compression if: a response header includes the "expired", "no-cache", "no-store", "private", and "Authorization" parameters)
gzip_types - enables the types of files that can be compressed, NOTE: do not compress images, as they are usually already compressed
gzip_disable "MSIE [1-6]\."; - disable compression for Internet Explorer versions 1-6

---

Multiple site configuration files should be stored in `/etc/nginx/conf.d/` as `example.com.conf`, or `example.com.disabled`. Do not add server blocks directly to `/etc/nginx/nginx.conf`

*Virtual Host* is an Apache term, *Server Block* is the NGINX equivalent.

NGINX **site-specific** configuration files are kept in `/etc/nginx/conf.d/`. Generally you will want a separate file in this directory for each domain or subdomain you will be hosting.

**Note on conf.d vs sites-available**: ubuntu/debian tend to have `/etc/nginx/sites-available/` and `../sites-enabled/`. The default nginx package for debian 9 is 1.10, which has those configurations. But we want to use Nginx configurations not Apache configurations. So try to get Nginx 1.15 to see `/nginx/conf.d/.

Open `/etc/apt/sources.list` in a text editor and add the following line to the bottom:

`deb http://nginx.org/packages/mainline/debian/ stretch nginx`

Import the repository’s package signing key and add it to apt:

```
sudo wget http://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key
```
Then install Nginx with apt update and apt install nginx. This will make the debian get at least 1.15 (mainline) nginx.

**Gunicorn+Flask**

Nginx pass web requests to Gunicorn.
Open a `server block`, use this block for requests for our server's domain name or IP address, and pass the `proxy-params` to a `location block` in the `server block`.

*Proxy Setups*
If you deploy your application using one of these servers behind an HTTP proxy you will need to rewrite a few headers in order for the application to work. The two problematic values in the `WSGI` environment usually are `REMOTE_ADDR` and `HTTP_HOST`. You can configure your httpd to pass these headers, or you can fix them in middleware. Here we will configure the Nginx httpd to provide these Headers.
**Note: httpd = http daemon**
**Note: daemon is Greek myth, a good or benevolent "supernatural beings between mortals and gods"; demon is Christian myth, a bad spirit thing. Daemon, not demon!**

For the flask+gunicorn app, change the config in the conf.d's site:
```
server {
    listen       80;
    server_name  your_public_dnsname_here;
    # do not use the default root for site data, as it will be removed during nginx upgrade
    root filepath_here;

    # added the proxy_pass so the nginx listens for port N for the flask app
	# added the proxy_set_header for IP and HOST
	# Proxy pass directive must be the same port on which the gunicorn process is listening.
	location / {
        proxy_pass         http://<IP_ADDRESS>:Port_GUNICORN;
        proxy_redirect     off;

        # Redefine the header fields that NGINX sends to the upstream server
        proxy_set_header   Host                 $host;
        proxy_set_header   X-Real-IP            $remote_addr;
        proxy_set_header   X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $scheme;
    }

    location /static {
        # handle static files directly, without forwarding
        alias /home/N/app/static;
        expires 30d;
    }

}
```
In Nginx, `$host` vs `$http_host`:
`$host` is equal to line Host in the header of request or name of the server processing the request if the Host header is not available, or your `server_name` in the `server block`.

`location` block serves the routes.
Do not use `root` inside Location Block, use `alias`.

The `try_files` directive exists for an amazing reason: It tries files in a specific order. NGINX can first try to serve the static content, and if it can’t, it moves on.
```
    location / {
        try_files $uri $uri/ /index.php;
    }
```

**React-webpack-build**
In order for webpack's index.html to work, the parameters to `try_files` should be URIs and not pathnames. This will tell nginx to forward other requests to the index.html file too:
```
location / {
  try_files $uri $uri/ /index.html;
}
```

Besides the Gzip, you can also Configuring Browser Caching:
```
location ~*  \.(jpg|jpeg|png|gif|ico|css|js|pdf)$ {
    expires 7d; # or 30 days for long-term build
}
```

Restart the nginx web server.
`sudo nginx -t`
`sudo service nginx restart`

---

Test Nginx configuration for errors:
`sudo nginx -t`

check if Nginx is running:
`systemctl status nginx`

To stop your web server, you can type:
`sudo systemctl stop nginx`

To start the web server when it is stopped, type:
`sudo systemctl start nginx`

To stop and then start the service again, type:
`sudo systemctl restart nginx`

**NOTE!**
- in order for nginx config to take effect, restart it by calling `sudo service nginx restart` - if there are errors in the config, they will be brought up at this step
- so many people don't restart the nginx service and wonder why nothing has taken effect - don't fall for it, restart it alway

We can make sure that our web server will restart automatically when the server is rebooted by typing:
`sudo systemctl enable nginx`

See Server Logs
`/var/log/nginx/`

## 4. Gunicorn/Flask

`sudo pip3 install gunicorn flask`

Gunicorn is a pure python production-level server. Flask is for fast development.

To start the flask under Gunicorn:
`gunicorn -b localhost:7717 -w 1 <module that contains the application>:<name of this gunicorn app>`
`-b` option gunicorn listen for requests at port N, By default Gunicorn runs on port 8000. We can change the port by adding the `-b` bind option.
`-w` option configures how many workers gunicorn will run. Having four workers allows the application to handle up to four clients concurrently.

**NOTE on -w in this app**: the global cal_procs_dict in the flask app cannot be processed correctly when the worker is more than 1. So we just have to use 1 worker in gunicorn.

Number of workers in Gunicorn, usually 4 is enough. Gunicorn should only need 4-12 worker processes to handle hundreds or thousands of requests per second.
```
def number_of_workers():
    return (multiprocessing.cpu_count() * 2) + 1
```
The above formula is based on the assumption that for a given core, one worker will be reading or writing from the socket while the other worker is processing a request.

We will not run Gunicorn/Flask from the command-line. We will have it under constant monitoring, and launch with `Supervisor`.

After flask/gunicorn is changed, do:
`sudo service supervisor restart`
to see the changes implemented
OR for the gunicorn/flask app specifically:
```
sudo supervisorctl stop yjg_calculator_webapp_React     # stop the current server
sudo supervisorctl start yjg_calculator_webapp_React    # start a new server
```

OR! just do this:
`sudo supervisorctl reload` :D

## 5. Supervisor

`sudo apt-get install supervisor`

the configuration for Supervisor is in `/etc/supervisor/conf.d/`
create a new `calculatorapp.conf`
```
[program:yjg_calculator_webapp]
command=gunicorn -b localhost:8000 -w 4 flask:app
directory=/home/deploy/webapp
user=<user name>
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
```

To enable the configuration, run the following commands:
`sudo supervisorctl reread`
then `sudo service supervisor restart`

This should start a new process. To check the status of all monitored apps:
`sudo supervisorctl status`

## References:
1. https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux
2. http://flask.pocoo.org/docs/1.0/deploying/wsgi-standalone/#gunicorn
3. https://www.linode.com/docs/web-servers/nginx/nginx-installation-and-basic-setup/
4. https://www.linode.com/docs/web-servers/nginx/slightly-more-advanced-configurations-for-nginx/
5. https://www.techrepublic.com/article/how-to-configure-gzip-compression-with-nginx/
6. https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-14-0

repository: FeynmanDNA/MakeTheWorldABetterPlace

repo_name: FeynmanDNA/MakeTheWorldABetterPlace


text:
---
layout: post
title: July 29, 2007 - Jesus Every Day
categories: [Verse of the Day]
alias: [/VerseOfTheDay/2007/07/29/]
---

_I John 1:5-7
5 This then is the message which we have heard of him, and declare
unto you, that God is light, and in him is no darkness at all.
6 If we say that we have fellowship with him, and walk in darkness,
we lie, and do not the truth:
7 But if we walk in the light, as he is in the light, we have
fellowship one with another, and the blood of Jesus Christ his Son
cleanseth us from all sin._

To Him who is both Light and the Creator of light, there cannot
possibly be any darkness at all, for if there was, He would cease to
be Light! In Him is no ignorance, no imperfection, no sinfulness, no
darkness at all; and from Him the light of wisdom, knowledge,
holiness, happiness, and life, temporal and eternal, are received by
every believing soul.

 --

Jesus Every Day is a ministry of Rodgers Baptist Church in Garland, Texas.

license: MIT

branch_name: refs/heads/master

extension: markdown

gha_language: JavaScript

license_type: permissive


text:
# Google Books

Application that allows the user to search for books (using Google Books API) and favorite them. Easily keep track of your favorite books!

## Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

### Prerequisites

- NodeJS
- git
- nodemon

### Installing

in terminal:

```
git clone https://github.com/6055093/google-books-project.git
```

```
npm install //installs all dependencies
```

```
nodemon server.js
```

## Screenshots

![Alt text](./public/screenshots/Signin.png 'Login page')

![Alt text](./public/screenshots/Search.png 'Search Books')

![Alt text](./public/screenshots/Description.png 'Book Description')

## Built With

- [ReactJS](https://reactjs.org/) - The web framework used
- [Redux](https://redux.js.org/) - Used for state management
- [NodeJS](https://nodejs.org/en/) - Used for Express (Login/Signup feature)

## Authors

- **Victor Petrescu** - Creator - [vicpetrescu](https://github.com/vicpetrescu)

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

repository: vicpetrescu/google-books

extension: md

repo_name: vicpetrescu/google-books


text:
Bifrost PostgreSQL Schema
=======================

![alt text](doc/assets/authz-schema.png "schema diagram")

# Modifying the Schema

* [Set up and Use Sqitch](doc/sqitch_background.md)

# Testing

We use [pgTAP][] to test both the schema and the stored procedures in
the database.

* [Developer Setup for Local Testing](doc/system_setup_for_testing.md)
* [Running pgTAP Tests for Authz](doc/running_pgtap.md)
* [Writing pgTAP Tests for Authz](doc/writing_tests.md)

# Debugging

Support for debugging a live Authz system.  Currently consists of
special SQL views that can be queried using `psql`.

* [SQL Debug Schema](doc/debug_schema.md)

[pgTAP]:http://pgtap.org

repository: vladuemilian/chef-server

repo_name: vladuemilian/chef-server


text:
## Houdini-SubdivIsoline-View
![Alt Text](https://media.giphy.com/media/LPxL71hXmRAzPhqPcI/giphy.gif)
![Alt Text](https://media.giphy.com/media/YPbn7xlFftblgubcN7/giphy.gif)

Shows the subdivision surface isolines in the viewport for a geometry and highlights crease weights. Could be useful for SDS modeling. This is an experimental code, which uses OpenSubdiv HDK API, suffers with FPS drops when working with a heavy geometry.
## Requiremenets
 - cmake
 - Xcode/Visual Studio (version depeds on the Houdini installation).
## Building
 - Set the **HOUDINI_HFS_DIR** environment variable to the Houdini
   installation directory (e.g. *C:\Program Files\Side Effects
   Software\Houdini 17.5.229*).
 - Build with cmake.


text:
---
title: "Laser cut pattern: Sample rotator."
date: 2018-10-31
tags: [laser cutter, sample rotator]
categories: [science, excel]
header:
  image: "images/header_image2.png"
excerpt: "Template for an ergonomic, inexpensive system that rotates microcentrifuge tubes. This can be used to mix biological and chemical samples."
---

This laser cut file is for rotating laboratory samples on a stepper motor. Tubes are held in place by a bar with magnets, making easy extraction. Other indents hold rubber bands to mount larger tubes such as 15 and 50 ml. Depending on the load, a simple DC motor with good torque can power it.

[Download](https://github.com/scotttmoen/Science)


text:
---
aliases:
- ../../configuration-language/standard-library/coalesce/
canonical: https://grafana.com/docs/agent/latest/flow/reference/stdlib/coalesce/
title: coalesce
---

# coalesce

`coalesce` takes any number of arguments and returns the first one that isn't null, an empty string, empty list, or an empty object.
It is useful for obtaining a default value, such as if an environment variable isn't defined.
If no argument is non-empty or non-zero, the last argument is returned.

## Examples

```
> coalesce("a", "b")
a
> coalesce("", "b")
b
> coalesce(env("DOES_NOT_EXIST"), "c")
c
```

repository: grafana/agent

branch_name: refs/heads/main

gha_language: Go

repo_name: grafana/agent


text:
# fillcache

[![Documentation](https://pkg.go.dev/badge/github.com/mccutchen/fillcache)](https://pkg.go.dev/github.com/mccutchen/fillcache)
[![Build status](https://github.com/mccutchen/fillcache/actions/workflows/test.yaml/badge.svg)](https://github.com/mccutchen/fillcache/actions/workflows/test.yaml)
[![Code coverage](https://codecov.io/gh/mccutchen/fillcache/branch/main/graph/badge.svg)](https://codecov.io/gh/mccutchen/fillcache)
[![Go report card](http://goreportcard.com/badge/github.com/mccutchen/fillcache)](https://goreportcard.com/report/github.com/mccutchen/fillcache)

An in-process cache with single-flight filling semantics.

In short: Given a function that computes the value to be cached for a key, it
will ensure that the function is called only once per key no matter how many
concurrent cache gets are issued for a key.

This might be useful if, say, you find yourself reaching for the
[`singleflight` package][singleflight] _and_ you want to cache the resulting
values in memory.

## Usage

See [example_test.go](/example_test.go) for example usage.

## Testing

```bash
make test
```

## Credits

If you like this package, all credit should go to [@jphines][jphines], who
suggested the initial design as we were working through an in-process DNS
caching mechanism.

If you don't like its design or its implementation, all blame lies with
[@mccutchen][mccutchen].

[singleflight]: https://godoc.org/golang.org/x/sync/singleflight
[jphines]: https://github.com/jphines
[mccutchen]: https://github.com/mccutchen

repository: mccutchen/fillcache

gha_license_id: MIT

repo_name: mccutchen/fillcache


text:
# SimplePosition

Automatically increments the position field for an ActiveRecord model in a `before_save` callback.

## Installation

Add this line to your application's Gemfile:

    gem 'simple_position'

## Usage

Add an integer `position` column to your model:

```ruby
add_column :pages, :position, :integer
```

Include `SimplePosition` class into your model:

```ruby
class Page < ActiveRecord::Base
  include SimplePosition
  default_scope order(:position)
end
```

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Added some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

gha_language: Ruby


text:
Replacer
========

Replacer is a python script that you can use inorder to replace some text in your file and save the result into another
file.

Installation
------------

You need [Python3](https://Python.org) inorder to run the script. [INSTALL PYTHON](https://Python.org)

#### Required Packages

- traceback (Comes with python core.)
- [log21](https://github.com/MPCodeWriter21/log21) (Install using `python3 -m pip install log21`)

_You can use `pip install -r requirements.txt` to install the required packages._

Usage
-----

```
usage: replacer.py [-h] FileName SavePath Old New

positional arguments:
  FileName          File path to the target file.
  SavePath          File path to save the output.
  Old               The Old word to be replaced.
  New               The New word to replace.

options:
  -h, --help
                        show this help message and exit
```

About
-----
Author: CodeWriter21 (Mehrad Pooryoussof)

GitHub: [MPCodeWriter21](https://github.com/MPCodeWriter21)

Telegram Channel: [@CodeWriter21](https://t.me/CodeWriter21)

Aparat Channel: [CodeWriter21](https://www.aparat.com/CodeWriter21)


text:
---
title: 'Command/Control: giving OS aware keybinding hints'
---

As someone who writes a lot of [web-based](https://extemporelang.github.io)
[documentation for](https://cs.anu.edu.au/courses/comp1720/)
[using](https://cs.anu.edu.au/courses/comp2300/)
[computers](https://comp.anu.edu.au/courses/laptop-ensemble/), it's often useful to
give people hints about keybindings which could make their lives easier.
However, this is tricky when it comes to the control key/command key keybinding
convention on macOS vs Windows/Linux.

In the past, I've just written things like:

> To save the file, hit CTRL+S to save
> (or ⌘+S if you're on macOS)

which gets _really tedious_ when you have to repeat it every time.

So, I wrote a little bit of javascript which

1. scans the document for any control elements
2. if it detects[^detection] you're viewing the site on macOS, changes it to
   ⌘

It actually searches for all of control, command, ctrl, ctl or cmd and "normalises"
them to ⌘ (macOS) or CTRL (otherwise).

On the off chance that I actually don't want it to do this, the script will skip
any `<kbd>` element with a `nopretty` class.

[^detection]: it'll work in _most_ cases; foolproof OS autodetection is really hard

If you're interested, the script looks like this:

```javascript
window.addEventListener('DOMContentLoaded', (event) => {

  // replace all these with the appropriate modifier for the platform
  let modKeys = ["control", "command", "ctrl", "ctl", "cmd"];

  let modifier = navigator.appVersion.indexOf("Mac")!=-1 ? "⌘" : "CTRL";

  for (const kbdElement of document.querySelectorAll("kbd")) {
	if (modKeys.includes(kbdElement.textContent.toLowerCase()) && !kbdElement.classList.contains("nopretty")){
	  kbdElement.textContent = modifier;
	}
  }
});
```

In case I make any tweaks, you can always find the latest version in my [blog
source on
GitHub](https://github.com/benswift/benswift.github.io/blob/source/assets/js/os-aware-modifiers.js).

branch_name: refs/heads/source


text:
---
layout: post
title: "Deep Learning with Python"
date: "2019-06-02"
categories:
- Textbooks
book: "deep-learning-with-python"
book_authors: "François Chollet"
excerpt: "An introduction to practical deep learning with Python and Keras."
---

Computer science is no more about computers than astronomy is about telescopes.

Unknown

# Part I: Fundamentals of Deep Learning

## Chapter 1: What is deep learning?

- Repository with author’s code
    - <https://github.com/fchollet/deep-learning-with-python-notebooks>
- Download my Google Colaboratory code
    - [Google Colaboratory/Jupyter Notebook](google-colaboratory.ipynb)
    - [Raw Python](raw-python.py)
- Deep learning → Machine learning → Artificial intelligence
![Figure 1.2](figure1-2.png)
- *Representations*: different ways to look at data; to represent or encode the data.
- *Learning*: in machine learning, describes the automatic search for better representations.
- *Hypothesis space*: the set of possible operations that tests whether the operation on the input data generates the correct output.
- E.g. Coordinate changes, translations, linear projections.
- The high level steps of machine learning
    - Search for useful representations of some input data
    - Search within a predefined space of possibilities (hypothesis space)
    - Search using guidance from a feedback signal
- *Deep learning*: the idea of successive layers of representations.
- You can think of a deep network as a multistage information-distillation operation, where information goes through successive filters and comes out increasingly purified.
- However, don’t mistake deep learning for deeper understanding.
- The term “neural network” comes from biology but the relationship to the brain isn’t there.
- Don’t link artificial neural networks with biological neural networks.
- *Loss function*: the error between the actual output of network and the expected output.
![Figure 1.9](figure1-9.png)
- Other types of machine learning
    - Probabilistic modeling
    - Support vector machines
    - Decision trees and random forests
- Deep learning is different due to
    - Not needing feature engineering anymore
    - Incremental, layer-by-layer way in which increasingly complex representations are developed
    - Intermediate incremental representations are learned jointly
- Three technical forces driving deep learning
    - Hardware
    - Data
    - Algorithms

## Chapter 2: Before we begin: the mathematical building blocks of neural networks

- *Tensor*: a container for data that generalizes matrices to any arbitrary dimension.
- The tensor’s dimension is also called its axis.
- E.g. A 5D vector of [1, 2, 3, 4, 5] has one axis but five dimensions.
- Dimensionality can denote either the number of entries along a specific axis (as in the case of our 5D vector) or the number of axes in a tensor (such as a 5D tensor).
- A tensor has three key attributes
    - *Number of axes/rank*: how many dimensions it takes up
    - *Shape*: the description of the axes
    - *Data type*: float32, float64, or uint8
- E.g. A tensor with shape (60000, 28, 28) means 60000 matrices of size 28 by 28.
- Shape is NOT the same as the tensor’s dimensions.
- E.g. A tensor with shape (6, 1) means 6 vectors of size 1 x 1.
- Seems like shape tends to ignore the last dimension (1) in 1 x 1 for vectors.
- *Tensor slicing*: selecting specific elements from a tensor.
- To slice the dataset

```python
my_slice = train_images[10:100]
```

- To slice each tensor in the dataset (E.g. To get the 14 x 14 pixels on the bottom-right)

```python
my_slice = train_images[:, 14:, 14:]
```

- The first axis (or axis 0) is often called the batch axis or batch dimension since it’s usually used to control the batch size.
- Real world data tensors
    - *Vector data*: 2D tensors of (samples, features)
        - E.g. Storing age, ZIP code, and income of each person. (num_people, num_features=3)
    - *Timeseries/sequence data*: 3D tensors of (samples, timesteps, features)
![Figure 2.3](figure2-3.png)
        - E.g. Stock prices. (num_days, num_mins_per_day, num_features)
    - *Images*: 4D tensors of (samples, height, width, channels)
![Figure 2.4](figure2-4.png)
        - E.g. Batch of 128 grayscale images of 256 x 256. (128, 256, 256, 1)
    - *Videos*: 5D tensors of (samples, frames, height, width, channels)
        - E.g. 4 Youtube videos that’s 60 seconds long, 5 frames per second, and has a resolution of 1920 x 1080. (num_videos=4, num_frames=300, 1920, 1080, 3)
- Similar to binary operations on binary inputs (E.g. AND, OR, NOT), there are tensor operations.
- We can think of each layer in the network as performing a function.

```python
keras.layers.Dense(512, activation='relu')
// This layer is doing the same as the following line of code
output = relu(dot(w, input) + b)
```

- The steps each layer does
    - Dot product of the weights and input
    - Add the result with a bias
    - Pass the result through a rectified linear function
- A dense layer is the same as a fully connected layer.
- *Element-wise operations*: an operation on a vector’s individual elements.
- Since element-wise operations are independent of each other, it can be parallelized.
- When adding two tensors of different shapes, the smaller tensor is broadcasted to match the shape of the larger tensor.
- Steps to broadcast a smaller tensor into a larger one
    - Add axes to the smaller tensor to match the number of dimensions of the larger tensor.
    - Repeat the smaller tensor alongside the new axes to match the full shape of the larger tensor.
- E.g. Let X have shape (32, 10) and Y have shape (10, ). To broadcast Y to X, first add another axis to Y to get (1, 10). Then repeat Y 32 times in that new axis to get (32, 10).
- *Tensor dot*: the same as vector dot product. Multiply each corresponding component and then sum the results.
![Figure 2.5](figure2-5.png)
- *Tensor reshaping*: change the shape of a tensor while maintaining its data.
- E.g. Changing from (3, 2) to (6, 1).

```python
x = np.array([0, 1], [2, 3], [4, 5])
x = x.reshape((6, 1))
print(x)
[[0], [1], [2], [3], [4], [5]]
```

- A common tensor reshaping operation is transposing which switches the rows and columns.
- Neural network → chain of tensor operations → geometric transformations of the input data.
- So an ANN can be interpreted as a very complex geometric transformation in a high-dimensional space.
- The analogy is that machine learning takes a crumpled paper ball and uncrumples it which is like finding the neat representations for complex, highly folded data manifolds.
- ML uncrumples the ball by incrementally decomposing a complicated geometric transformation into a long chain of elementary ones.
- It’s a reductionist approach.
- The weights of the neural network contain information learned by the network from exposure to training data.
- The training loop
    - Draw a batch of training samples.
    - Run the network on those samples to predict the target.
    - Compute the loss/error of the network on the batch.
    - Update all weights of the network to slightly reduce the loss on this batch.
- The intuition behind a continuous function is that a small change in x must result in a small change to y.
- An ANN can be thought of as moving towards the lowest point on the error map where the gradient points which direction to head down and how steep the slope it.
- *Stochastic gradient descent (SGD)*: randomly sampling an input and using its error to descent the error map.
- Stochastic refers to how each batch of data is randomly drawn.
- Mini-batch stochastic gradient descent versus Stochastic gradient descent: each iteration uses a batch of data instead of a single sample.

|                                 | SGD    | Mini-batch SGD  | Batch SGD  |
| ------------------------------- | ------ | --------------- | ---------- |
| Input/Sample Size per Iteration | Single | Batch or subset | Entire set |

- Momentum address two issues with SGD
    - Convergence speed
    - Local minima
- Momentum draws inspiration from physics. The mental image is to think of the optimization process as a small ball rolling down the loss curve. If the ball has enough momentum, it won’t get stuck in the local minimum.
- Momentum is implemented by moving the ball at each step based not only on the current slope value (current acceleration) but also on the current velocity (resulting from past acceleration).

```python
past_velocity = 0
momentum = 0.1
while loss > 0.01:
  w, loss, gradient = get_current_parameters()
  velocity = past_velocity * momentum + learning_rate * gradient
  w = w + momentum * velocity - learning_rate * gradient
  past_velocity = velocity
  update_parameter(w)
```

- *Backpropagation*: applying the calculus chain rule to the computation of gradients in an ANN.
- *Symbolic differentiation*: given a chain of operations with a known derivative, compute a gradient function for the chain that maps network parameter values to gradient values.
- *Epoch*: each iteration over all of the training data.

**Summary**
![Chapter 2](chapter2.png)

## Chapter 3: Getting started with neural networks

- Neural networks have four components
    - Layers
    - Input data and corresponding targets
    - Loss function to define the feedback signal used for learning
    - Optimizer to determine how learning proceeds
- The fundamental data structure in neural networks is the layer.
- Different layers are used for different input formats.
    - Use dense/fully connected layers for 2D tensors (basic)
    - Use recurrent layers for 3D tensors (time series)
    - Use 2D convolution layers for 4D tensors (images)
- *Layer compatibility*: layers only accept input tensors of a certain shape and will return output tensors of a certain shape.
- Keras automatically handles layer compatibility.
- The loss function is a measure of success for the task.
- The optimizer determines how the network will be updated.
- Guidelines for choosing the loss function
    - Two-class classification → Binary crossentropy
    - Many-class classification → Categorical crossentropy
    - Regression → Mean-squared error
    - Sequence-learning → Connectionist temporal classification
- Keras is a model-level library that uses a backend engine for tensor calculations.
![Figure 3.3](figure3-3.png)
- There are two ways to define a model
    - Sequential for linear stacks of layers.
    - Functional API for arbitrary architectures.
- Use separate sets of data for training and testing. Reasons being
    - Performing well on the training set doesn’t mean the model will perform well on new data.
    - You only care about the model’s performance on new data.
    - The model might end up memorizing a mapping between the training samples and the target instead of actually using the training data.
- You can’t feed lists of integers into a neural network, you have to turn them into tensors. There are two methods to do this.
- First method is to pad the lists so that they’re the same length

```python
y_train = np.asarray(train_labels).astype('float32')
# Or use
y_train = np.array(train_labels)
```

- Second method is to one-hot encode

```python
def vectorize_sequences(sequences, dimensions=10000):
  results = np.zeros((len(sequences), dimensions))
  for i, sequence in enumerate(sequences):
    results[i, sequence] = 1
  return results
```

- *Hidden unit*: a dimension in the representation space of the layer.
- Dimensionality can intuitively be thought of as how much freedom you’re allowing the network to have when learning internal representations.
- More hidden units allows for learning more complex representations but it increases the use of resources and may lead to learning unwanted patterns.
- The two key architecture decisions to make for dense layers are
    - How many layers to use
    - How many hidden units to choose for each layer
- *Rectified linear unit (ReLU)*: zeros out negative values and maintains positive values.
- *Sigmoid*: squashes arbitrary values to between 0 and 1.
- Activation functions are required to get access to a richer hypothesis space for deep representations. It must be non-linear as `output = dot(W, input) + b` is a linear transformation.
- Crossentropy is usually the best loss function when dealing with models that output probabilities.
- Crossentropy measures the distance between probability distributions.
- Example of custom optimizer, loss, and metric

```python
model.compile(
  optimizer=optimizers.RMSprop(lr=0.001),
  loss=losses.binary_crossentropy,
  metrics=[metrics.binary_accuracy]
)
```

- If the training loss and accuracy improve but the validation metrics don’t, then the model is over-fitted.

```python
results = model.evaluate(x_test, y_test)
model.predict(x_test)
```

- *Single-label, multiclass classification*: each label can only belong to one category.
- E.g. Each picture can only be labeled as either cat or dog. There are multiple classes (cat, dog) but each picture can only get one label.
- *Multi-label, multiclass classification*: each label can belong to multiple categories.
- E.g. Each picture can have both a cat and dog so there each picture can get multiple labels.
- If you’re trying to predict data with N classes, then end the network with a dense layer of size N.
- If a layer drops information relevant to classification, then that information can never be recovered as each layer can become an information bottleneck.
- If not using categorical encoding/one-hot such as using integer labels, use `sparse_categorical_crossentropy` instead of `categorical_crossentropy`. It’s mathematically equivalent.
- As a rule of thumb, hidden layers shouldn’t have fewer units than the output layer.
- Predicting discrete labels is categorization, predicting continuous labels is regression.
- Logistic regression (a classification method) =/= regression.
- It’s problematic to feed an ANN with feature values that vary widely as it will slow down learning due to large gradient updates preventing convergence.
- E.g. House features such as money, house square feet, year built all have different ranges.
- To deal with this, normalize the features using a z-score so that the feature has a mean of 0 and a standard deviation of 1.

```python
# Calculate the mean and std to normalize the training data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /=std

# Apply the normalization to the test data
test_data -= mean
test_data /= std
```

- Apply the computed values to the test data but never calculate the values using the test data, even data normalization.
- In general, the less training data there is, the worse overfitting will be. Mitigate this by using a smaller network.
- When doing regression, use a linear activation layer for the last layer.
- *Mean squared error (mse)*: the square of the difference between the predictions and the targets.
- *Mean absolute error (mae)*: the absolute value of the difference between the predictions and the targets.
- A small dataset leads to challenges with picking the validation and testing set. The validation set might have high variance which makes evaluating the model difficult.
- *K-fold cross-validation*: splitting the available data into K partitions, starting K identical models, and training each one on K - 1 partitions.
![Figure 3.11](figure3-11.png)
- Once you’re finished tuning the parameters of the model, you can train the final production model on the combination of the training and validation dataset.

**Summary**
![Chapter 3](chapter3.png)

## Chapter 4: Fundamentals of machine learning

- The central problem of deep learning is overfitting.
- Four branches of machine learning
    - *Supervised learning*: learning to map input data to known targets.
        - Classification
        - Regression
        - Sequence generation
        - Object detection
        - Image segmentation
    - *Unsupervised learning*: finding interesting transformation of the input data without the help of any targets.
        - Dimensionality reduction
        - Clustering
    - *Self-supervised learning*: a specific instance of supervised learning without human-annotated labels.
        - Autoencoders
        - Temporally supervised learning
    - *Reinforcement learning*: an agent received information about its environment and learns to choose actions that will maximize some reward.
- These four branches are sometimes blurry and aren’t well defined. It’s more of a continuum than distinct.
![Classification and regression glossary 1](class-and-regress-gloss-1.png)
![Classification and regression glossary 2](class-and-regress-gloss-2.png)
![Classification and regression glossary 3](class-and-regress-gloss-3.png)
- We split the input data into a
    - Training set
    - Validation set
    - Testing set
- Because if we use all of the data, the model starts overfitting.
- The goal of machine learning is to achieve models that generalize to perform well on never-before-seen data.
- So a measure of generalization power of the model is important.
- The hyperparameters of the model are the controllable factors of the model.
- E.g. Number of neurons, number of layers, learning rate, optimization method.
- The parameters of the model are the network’s weights.
- Be careful to not tune the hyperparameters too much to the validation set as you could be overfitting to the validation set.
- Three ways to split the train, valid, and test sets
- Simple hold-out validation
    - Set apart some fraction of the data as the test set.
    - However, if little data is available, then the validation and test sets may contain too few samples.
    - Recognize this problem if the model shows variance based on the train-test split.
![Figure 4.1](figure4-1.png)

```python
num_validation_sample = 10000
np.random.shuffle(data)

# Define the validation set
validation_data = data[:num_validation_samples]
data = data[num_validation_samples:]

# Define the training set
training_data = data[:]

model = get_model()
model.train(training_data)
validation_score = model.evaluate(validation_data)

# At this point, you can tune your model,
# retrain it, evaluate it, and tune it again.

# Train the final model on all non-testing data
model = get_model()
model.train(np.concatenate([training_data, validation_data]))
test_score = model.evaluate(test_data)
```

- K-fold validation
    - Split the data into K partitions of equal size.
    - For each partition i, train the model on the remaining K - 1 partitions and evaluate on partition i.
    - Your final score is the averages of the K scores.
![Figure 3.11](figure3-11.png)

```python
# The number of folds
k = 3
num_validation_samples = len(data) // k
np.random.shuffle(data)

validation_scores = []
for fold in range(k):
  validation_data = data[num_validation_samples * fold:
    num_validation_samples * (fold + 1)]
  training_data = data[:num_validation_samples * fold] +
    data[num_validation_samples * (fold + 1):]

    model = get_model()
    model.train(training_data)
    validation_score = model.evaluate(validation_data)
    validation_scores.append(validation_score)

# Calculate the average validation score
validation_scores = np.average(validation_scores)

# Train the final model on all non-testing data
model = get_model()
model.train(data)
test_score = model.evaluate(test_data)
```

- Iterated K-fold validation with shuffling
    - Use this when there is very little data
    - Apply K-fold validation multiple times, shuffling the data every time before splitting it K-ways.
    - The final score is the average of the scores obtained from each run of K-fold validation.
- Things to keep in mind
    - *Data representativeness*: ensure to shuffle your data before splitting it into the training and test sets.
        - E.g. If taking the first 80% of a digits data set for training, then the training set only contains classes of 0-7 where the test set contains only classes 8-9.
    - *Arrow of time*: don’t shuffle the data if you’re trying to predict future events from the past because the order matters.
        - E.g. Stock market, weather, music.
    - *Redundancy in the data*: if duplicate data ends up in both the training and testing set, then the network will train on the testing set which introduces bias.
- Data preprocessing for neural networks
    - *Vectorization*: convert all data into tensors.
        - Done because neural networks only take in tensors as input.
    - *Value normalization*: convert all data into the same range with a mean of 0 and a standard deviation of 1.
        - Done because a large variations in the input data can cause large gradient updates preventing the model from converging.
    - *Missing values*: set missing data to zero so the model learns zero to mean missing data.
        - Done because neural networks can’t handle missing values.
- *Feature engineering*: the process of using your own knowledge of the data to make the model work better.
![Figure 4.3](figure4-3.png)
- E.g. Instead of inputting the raw clock face, convert the time into Cartesian coordinates or, even better, into polar coordinates.
- Make the problem easier by expressing it in a simpler way. However, this requires understanding the problem in depth.
- Benefits of feature engineering
    - Reduces the amount of resources used
    - Reduces the amount of data used
- Learning how to deal with overfitting is essential to mastering machine learning.
- The fundamental issue in machine learning is the tension between optimization and generalization.
- The challenge in machine learning is to generalize from the data.
- *Optimization*: adjusting a model to get the best performance.
- *Generalization*: adjusting a model to perform well on any data.
- *Regularization*: adjusting a model to prevent overfitting.
- Regularization methods
    - Get more training data
    - Reduce the capacity of the network
    - Add weight regularization
    - Add dropout
- Constraining the model
    - *Capacity*: the number of learn-able parameters in a model.
    - A model with more parameters has a higher learning capacity.
    - However, it’ll start overfitting if it has too much room to learn.
    - There’s a tradeoff between too much capacity and too little capacity.
- The general workflow to find an appropriate model is to start small and to slowly grow the model until you see diminishing returns.
- The larger the model, the quicker it can model the training data but the more susceptible it is to overfitting.
- Simpler models are less likely to overfit than complex ones.
- One way of making the model simpler is to reduce the range of the weights.
- *Weight regularization*: forcing the weights to take only small values.
- Weight regularization penalizes the size of the weights of a network so that large output values that signify strong confidence can no longer be achieved from a single large weight, but instead require several medium-sized weights.
- Conceptually, it penalizes strong opinions from single units and encourages taking into account the opinion of multiple units, thus reducing bias.
- This penalty is added to the error function value thus increasing the error if larger weights are used. As a result, the network is driven to solve the problem with small weights.
- The cost comes in two flavors
    - *L1 regularization*: penalizes the absolute size of the weight.
    - *L2 regularization*: penalizes the squared size of the weight.
- Code implementation of regularizers

```python
from keras import layers, models, regularizers

model = models.Sequential()
# Use L2 regularization with penalty of 0.001 * weight
model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),
    activation ='relu', input_shape=(10000,)))
# Use both L1 and L2 regularization
model.add(layers.Dense(16,
    kernel_regularizer=regularizers.l1_l2(l1=0.001, l2=0.001),
    activation ='relu', input_shape=(10000,)))
```

- The `l2(0.001))` means to use L2 regularization and for every coefficient in the weight matrix of the layer, add `0.001 * weight_coefficient_value` to the total loss of the network.
- Note that this penalty is only added at training and not during testing.
- Another way to prevent overfitting is dropout.
- *Dropout*: randomly dropping out (or setting to zero) a number of output features of the layer during training.
![Dropout](dropout.png)
- E.g. A normal layer returns `[0.2, 0.5, 1.3, 0.8, 1.1]` but a dropout layer would return `[0, 0.5, 1.3, 0, 1.1]`.
- <https://stats.stackexchange.com/questions/241645/how-to-explain-dropout-regularization-in-simple-terms>
- *Dropout rate*: the fraction of features that are zeroed out.
- During testing, no units are dropped out but are instead scaled down by a factor equal to the dropout rate to balance the increased number of active units.
- E.g. `[0.2, 0.5, 1.3, 0.8, 1.1]` with a dropout rate of `0.2` at testing would turn into `[0, 0.5, 1.3, 0.6, 1.1]`
- When applying dropout, you need to compensate for the fact that at training time, a portion of the neurons are deactivated. To do so, there are two strategies.
    - *Inverted dropout*: divide the output by the dropout rate during training and leave the testing unchanged.
    - *Scaled activation*: leave the training unchanged but multiply the output by the dropout rate during testing.
- Code implementation of dropout

```python
p = 0.5  # Dropout rate or the probability of keeping a unit active.
## Method 1: Inverted Dropout
# Training: randomly nuke 50% of the output and multiply it by 2
layer_output *= np.random.randint(0, high=2, size=layer_output.shape)
layer_output /= p
# Testing: do nothing

## Method 2: Scaled Activation
# Training: randomly nuke 50% of the output
layer_output *= np.random.randint(0, high=2, size=layer_output.shape)
# Testing: scale output down by 50%
layer_output *= p
```

![Figure 4.8](figure4-8.png)

- <https://stats.stackexchange.com/questions/241645/how-to-explain-dropout-regularization-in-simple-terms>
- Adding dropout to a neural network

```python
model = models.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(16, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))
```

**Universal Machine Learning Workflow**

1. Define the problem and get the data
   - What will your input data be? What are you trying to predict? What type of problem are you facing? (binary, multiclass, regression).
   - You believe that your outputs can be predicted given your inputs.
   - You believe that the input contains sufficient information to predict the output.
   - Machine learning can only recognize what it’s already seen. You’re predicting the future given the past.
2. Choose a measure of success
   - Success must be measurable for it to be improvable.
   - The metric for success will inform which loss function to choose.
3. Decide an evaluation protocol
   - Once you know what you’re aiming for, you have to figure out how to get there.
   - E.g. A validation set, K-fold cross validation.
4. Prepare your data
   - Format the data as tensors.
   - Scale the values to be in a small range.
   - Normalize features with different ranges.
   - Feature engineering.
5. Develop a model to beat the baseline
   - The baseline is random guessing.
   - E.g. For guessing digits between 0-9, the baseline is 10% for random guessing.
   - The key choices are for the last-layer activation, loss function, and optimization configuration.
![Table 4.1](table4-1.png)
6. Scale up the model so it overfits
   - Once the model is better than the baseline, optimize the model.
   - The universal tension in machine learning is between optimization and generalization.
   - The model should stand on the border between under-fitting and overfitting; between under-capacity and overcapacity.
   - To determine the border, you must first cross it.
   - To cross the border, develop a model that overfits by adding more layers, making the layer bigger, and training for more epochs.
   - When the validation loss increases, you’ve achieved overfitting.
7. Regularize the model and tune the hyperparameters
   - Now that the model overfits, move it closer to the border by
       - adding dropout
       - adding L1/L2 regularization
       - modifying the architecture
       - modifying the hyperparameters
   - Be mindful of leaking information from the validation set to the model. You don’t want the hyperparameters to be overspecialized to the validation set.
   - Once the neural network looks good, train on both the training and validation set and test it on the testing set.

**Summary**
![Chapter 4](chapter4.png)

# Part II: Deep Learning in Practice

## Chapter 5: Deep learning for computer vision

- *Convolutional neural networks (convnets)*: a class of neural networks that uses the convolution operation on images for recognition and classification.
- Convnets take in as input tensors of shape `(image_height, image_width, image_channels)` not including the batch dimension.
- Since the output of the conv layer is 3D, we need to flatten it to 1D using it in a dense layer.
- The fundamental difference between dense and convolution layer
    - Dense layers learn global patterns in the data.
    - Convolution layer learn local patterns in the data.
![Figure 5.1](figure5-1.png)
- Since conv layer learn local patterns, this gives them two interesting properties
    - The patterns they learn are translation invariant.
    - They can learn spatial hierarchies of patterns.
- Convolutions operate over 3D tensors, call feature maps, with two spatial axes (height and width) as well as a depth axis (channels).
- RGB has three channels: red, green, blue. Greyscale has one channel: levels of grey.
- The convolution operation outputs a feature map with an arbitrary depth since the depth axis no longer stands for channels, but rather filters.
- *Filters*: encodes specific aspects of the input data.
![Figure 5.3](figure5-3.png)
- Convolutions are defined by two key parameters
    - Size of the patches extracted from the inputs
        - E.g. 3 x 3 or 5 x 5
    - Depth of the output feature map
        - The number of filters computed by the convolution.
        - E.g. 32, 64
![Figure 5.4](figure5-4.png)
- The output width and height may differ from the input width and height due to
    - Border effects
    - Strides
- Border effects and padding
    - If you use a sliding window on the raw input, the output will decrease in dimensions due to not being able to use the edge.
![Figure 5.5](figure5-5.png)
    - To get the output feature map to have the same spatial dimensions as the input, you can use padding.
    - *Padding*: adding an appropriate number of rows/columns to the input to fit center convolution windows around every input tile.
![Figure 5.6](figure5-6.png)
    - There are two padding arguments in Keras
        - *Valid*: no padding (only valid window locations are used).
        - *Same*: pad so that the output will have the same dimensions as the input.
- *Stride*: the distance between two successive windows.
![Figure 5.7](figure5-7.png)
- Stride can be used to downsample a feature map however, max-pooling is more commonly used.
- The role of max pooling is to aggressively downsample feature maps.
- *Max pooling*: extracting windows from the input feature maps and outputting the max value of each channel.
- It’s similar to convolution but instead of transforming local patches using the convolution kernel, they’re transformed using the max tensor operation.
- Max pooling is typically done with 2 x 2 windows and stride 2.
- Use max pooling to
    - Learn a spatial hierarchy of features.
    - Reduce the final feature map over time instead of all at the end to reduce overfitting.
- Alternatives to max pooling
    - Strides
    - Average pooling
- The depth of feature maps progressively increases whereas the size of the feature maps decreases. This is a common pattern in almost all convnets.
- Data preprocessing steps for images
    1. Read the picture file.
    2. Decode the JPEG into RGB grids of pixels.
    3. Convert these to floating-point tensors.
    4. Rescale the pixel values from between 0 and 255 to between 0 and 1.
- To fit a model to a generator instead of an array of tensors, use `fit_generator()` instead of `fit()`.
- Since the data is being generated endlessly, use the `steps_per_epoch` argument to set the number of batches to draw per epoch.
- Use data augmentation to generate more training data from existing samples to prevent overfitting.
- *Data augmentation*: augmenting the training samples by using random transformations.
- Keras supports
    - Rotation
    - Translation
    - Shear
    - Zoom
    - Flip
- *Pretrained network*: a saved network that was previously trained on a large dataset.
- The two ways to use a pretrained network
    - Feature extraction
    - Fine-tuning
- *Feature extraction*: using representations learned by a previous network to extract interesting features from new samples.
- Convnets consist of two parts
    - the pooling and convolution layers (called the convolution base)
    - the dense layers
![Figure 5.14](figure5-14.png)
- The reason to discard the classifier is because the representations learned by the convolution base are more general and thus more reusable.
- The more general a model is, the more reusable it is.
- After getting a pretrained network, there are two ways you can proceed
    - Running the conv base on your dataset and storing the result on disk. Then feeding the results into dense layers. This is fast and cheap but can’t use data augmentation since the conv base is only run once per image.
    - Extending the pretrained network by adding dense layers on top. Can use data augmentation since the image goes through the conv base every time it’s seen by the model.
- In Keras, models behave just like layers which means you can do something like this: `model.add(conv_base)`.
- *Freezing*: preventing a layer’s weights from updating.
- Always recompile the model after freezing layers.
- *Fine-tuning*: unfreezing a few of the top layers of a frozen model and training the thawed layers along with the classifier.
![Figure 5.19](figure5-19.png)
- Steps for fine-tuning a network
    1. Add your custom network on top of the pre-trained base network.
    2. Freeze the base network.
    3. Train your custom network.
    4. Unfreeze some layers in the base network.
    5. Jointly train both the unfrozen layer and your custom network.
- Three ways of visualizing what a convnet learn
    - *Intermediate activations*: for understanding how convnet layers transform their input.
    - *Convnet filters*: for understanding what visual patterns/concepts each filter is receptive to.
    - *Heatmaps of class activation*: for understanding which parts of an image lead to the activation of that neuron.
- To grab the intermediate activations from a model for certain inputs, use the following code.

```python
from keras import models

# Extract the outputs of the top eight layers
layer_outputs = [layer.output for layer in model.layers[:8]]
activation_model = models.Model(
    inputs=model.input, outputs=layer_outputs)
activations = activation_model.predict(img_tensor)
```

- Interesting notes from displaying the activations
    - The first layer acts like an edge detector.
    - The higher you go, the more the abstract the activations become.
    - The higher you go, the more sparse the activations hence why some of the filters are blank. This means the pattern encoded by the filter isn’t found in the input image.
![Figure 5.25](figure5-25.png)
- This channels appears to encode a diagonal edge detector.
- Gradient ascent in input space: applying gradient descent to the input of a convnet to maximize the response of a specific filter.
- Implementing gradient ascent in the input space
    - Build a loss function that maximizes the value of a given filter in a given layer.
    - Use SGD to adjust the values of the input to maximize the activation value.
- Similar to the activations, the filters get more abstract the deeper into the network we go.
- *Class activation heatmap (CAM)*: a 2D grid of scores associated with a specific output class, computed for every location in any input image, indicating how important each location is with respect to the class under consideration.
![Figure 5.36](figure5-36.png)
- You’re weighting a spatial map of “how intensely the input image activates different channels” by “how important each channel is with regard to the class”, resulting in a spatial map of “how intensely the input image activates the class.”

**Summary**
![Chapter 5](chapter5.png)

## Chapter 6: Deep learning for text and sequences

- None of the deep learning models in this chapter understand text in the human sense.
- Rather, these models can map the statistical structure of written language.
- Deep learning for natural language processing is pattern recognition applied to words, sentences, and paragraphs.
- *Vectorizing text*: the processing of transforming text into vectors/tensors.
- Vectorize text in multiple ways
    - Segment text into words and transform each word into a vector.
    - Segment text into characters and transform each character into a vector.
    - Extract n-grams of words/characters and transform each n-gram into a vector.
- *N-grams*: overlapping groups of multiple consecutive words/characters.
![Understanding n-grams and bag-of-words](n-grams.png)
- *Tokens*: the different units you can break down text (characters, words, n-grams).
- *Tokenization*: breaking text into tokens.
![Figure 6.1](figure6-1.png)
- There are multiple ways of associating a vector with a token
    - *One-hot encoding*: associating a unique integer index to each unique word.
    - *Word embedding*: associating a vector to a word.
![Figure 6.2](figure6-2.png)
- Two ways to obtain word embeddings
    - Learn word embeddings jointly with the main task.
    - Load into your model word embeddings that were precomputed on a different task.
- The geometric relationship between word vectors should reflect the semantic relationship between those words.
- Word embeddings are meant to map human language into a geometric space with words closer in meaning mapped closer together and specific directions to be meaningful.
![Figure 6.3](figure6-3.png)
- The transformation “from pet to wild animal” goes from bottom to top.
- The transformation “from canine to feline” goes from left to right.
- Learn the embeddings using backprop.

```python
from keras.layers import Embedding

# The 1000 is the number of possible tokens (1 + maximum word index).
# The 64 is the dimensionality of the embeddings.
embedding_layer = Embedding(1000, 64)
```

- Think of the embedding layer as a dictionary that maps words to vectors.
- Word index → Embedding layer → Corresponding word vector.
- The embedding layer takes a 2D tensor of shape `(samples, sequence_length)` and returns a 3D tensor of shape `(samples, sequence_length, embedding_dimensionality`.
- All sequences in a batch must have the same length so pad the shorter ones and chop the longer ones.
- Use pretrained word embeddings when you don’t have a lot of data.
- *Recurrent neural network (RNN)*: a class of neural network that maintains previously seen information; a network with memory.
- RNNs feed their previous output alongside the current input.
![Figure 6.9](figure6-9.png)
- The state of the RNN is reset between processing two different, independent sequences.
- However, the processing of those sequences is no longer a single step as the network internally loops over it.
- An RNN is a `for` loop that reuses quantities computed during the previous iteration of the loop.
![Figure 6.10](figure6-10.png)
- Usually, you don’t need the full sequence of outputs from an RNN because the last step already has the information about the entire sequence.
- The `SimpleRNN` in Keras takes inputs of shape `(batch_size, timesteps, input_features)`.
- Use the `return_sequences` argument to specify whether you want the full sequences of successive outputs for each timestamp or if you only want the last output.
- Sample of stacking RNNs

```python
model = Sequential()
model.add(Embedding(10000, 32))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32, return_sequences=True))
model.add(SimpleRNN(32))
model.summary()
```

- In practice, you’ll never use `SimpleRNN` because it’s too simple. Use either `LSTM` or `GRU`.
- SimpleRNN doesn’t work as well due to the vanishing gradient problem.
- *Vanishing gradient problem*: if the network as too many layers, it becomes untrainable.
- *Long Short-Term Memory (LSTM)*: an extension of RNNs with a carry track to carry information forwards in time.
![Figure 6.14](figure6-14.png)
- An LSTM saves information for later thus preventing older signals from gradually vanishing.
- Psuedocode for an LSTM

```python
output_t = activation(dot(state_t, Uo) +
    dot(input_t, Wo) + dot(C_t, Vo) + bo)

i_t = activation(dot(state_t, Ui) + dot(input_t, Wi) + bi)
f_t = activation(dot(state_t, Uf) + dot(input_t, Wf) + bf)
k_t = activation(dot(state_t, Uk) + dot(input_t, Wk) + bk)

c_t+1 = i_t * k_t + c_t * f_t
```

- Three advance RNN techniques
    - *Recurrent dropout*: uses a temporally constant dropout pattern.
    - *Stacking recurrent layers*: recurrent layers that are stacked on one another.
    - *Bidirectional recurrent layers*: recurrent layers that learn using both the forward and reverse order of the dataset.
- The common-sense, non-ML baseline is either to guess randomly or to guess the class that appears most often.
- E.g. If the data contains 90% of class A and 10% of class B, then the baseline is 90% because if we always guess A, then we will be right 90% of the time.
- If you don’t have a baseline to beat, you can’t tell whether you’re making real progress.
- *Gated recurrent unit (GRU)*: a streamlined version of the LSTM.
- ML is about the tradeoff between computational expensiveness and representational power.
- The proper way to apply dropout to RNNs is to apply the same pattern of dropped units at every timestep instead of applying a different pattern at every timestep.
- Vary the dropout pattern between sequences, not within sequences.
- To stack recurrent layers on top of each other, all intermediate layers should return their full sequence of outputs instead of just the last output.
- A bidirectional RNN processes input sequences chronologically and anti-chronologically. It trains in both forward and reverse time.
- Interestingly, reverse-order processing on text works just as well as forward-order processing in some cases.
- Although the word order does matter in understanding language, which order is used doesn’t matter.
![Figure 6.25](figure6-25.png)
- When it comes to markets, past performance is not a good predictor of future returns; looking in the rear-view mirror is a bad way to drive.
- *1D convolution*: a convolution in one dimension where the window is the slice of time.
![Figure 6.26](figure6-26.png)
- Like 2D convnets, 1D convnets find local patterns in the sequence so one pattern learned at one time can later be recognized at a different time.
- This makes 1D convnets translation invariant for temporal translations.
- E.g. If it sees that after it’s cloudy it rains, then the next time it’s cloudy, it’ll predict that it’ll rain.
- Just like max pooling, 1D sequences used 1D pooling.
- 1D convnets can’t learn longer-term patterns as well as an RNN can due to only focusing on local patterns.
- However, for very long sequences, you can use a 1D convnet to reduce the sequence’s length and feed that into an RNN.
![Figure 6.30](figure6-30.png)

**Summary**
![Chapter 6](chapter6.png)

## Chapter 7: Advanced deep-learning best practices

- Sequential models are limited in that they have exactly one input and exactly one output.
- However, you may want a model that
    - takes in multiple inputs
    - returns multiple outputs
    - has a graph-like structure
![Figure 7.1](figure7-1.png)
![Figure 7.2](figure7-2.png)
![Figure 7.3](figure7-3.png)
![Figure 7.5](figure7-5.png)

- The Keras functional API lets you use layers as functions.
- E.g. The `Sequential` model converted into the functional API.

```python
from keras.models import Sequential, Model
from keras import layers
from keras import Input

seq_model = Sequential()
seq_model.add(layers.Dense(32, activation='relu', input_shape=(64,)))
seq_model.add(layers.Dense(32, activation='relu'))
seq_model.add(layers.Dense(10, activation='softmax'))

input_tensor = Input(shape=(64,))
x = layers.Dense(32, activation='relu')(input_tensor)
x = layers.Dense(32, activation='relu')(x)
output_tensor = layers.Dense(10, activation='softmax')(x)
model = Model(input_tensor, output_tensor)
```

- The compiling, training, and evaluating is the same for the functional API.
- Multi-output models require different loss functions for the different outputs.
- However, SGD requires a scalar to work.
- *Residual connection*: a connection from shallow layers to deeper layers that allows re-injecting previous representations into the deeper layers.
- Resnets are built using residual connections and tackle the following problems
    - vanishing gradients
    - representational bottlenecks
![Vanishing gradients in deep learning](vanishing-gradients.png)
- You can use models as you use layers. Think of models as a “bigger layer”.
- E.g. You can use a model to evaluate a single input: `y = model(x)`.
- When training a model, we currently let the model run until it starts overfitting. However, this is wasteful and inefficient.
- A better way is to stop training when the validation loss stops improving. Use a Keras callback to implement this.
- Examples of ways to use callbacks during training
    - Model check-points by saving the current weights
    - Early stopping when validation loss doesn’t improve
    - Dynamically adjusting the value of certain parameters
    - Logging training/validation metrics
![Figure 7.9](figure7-9.png)
- Use Tensorboard to visualize
    - Metrics during training
    - Model architecture
    - Histograms of activations and gradients
    - Embeddings in 3D
- Advance neural network design patterns
    - Residual connections
    - Normalization
    - Depthwise separable convolution
- Normalization makes different samples seem more similar.
- `BatchNormalization` is a type of layer that can continuously normalize data throughout the network.
- *Depthwise seperable convolution*: separating the learning of spatial features and the learning of channel-wise features.
![Figure 7.16](figure7-16.png)
- *Hyperparameters*: the architecture-level parameters.
- E.g. How many layers/neurons to use, how many epochs, use dropout.
- *Model ensembling*: pooling together predictions of a set of different models to produce better predictions.
- E.g. The group of blind men exploring an elephant story.
- Ensembling assumes that different models are good for different reasons hence combining them makes them more effective. The models must be different and diverse.
- The whole is greater than the sum of its parts.
- You want ensemble models that are as good as possible but that are also as different as possible.

**Summary**
![Chapter 7](chapter7.png)

## Chapter 8: Generative deep learning

- *Generative recurrent networks*: using an RNN to predict the next token(s) in a sequence using the previous tokens as input.
- E.g. Given “the sky is bl-” predict “ue”.
- Once the model is trained, sample from it by feeding the generated output as input.
![Figure 8.1](figure8-1.png)
- Example sampling strategies
    - *Greedy sampling*: always choosing the most likely token.
    - *Stochastic sampling*: randomly choosing the next token based on the token’s probability.
- Stochastic sampling is better than greedy sampling as it provides more variety in the generated output.
- However, we want to control the randomness or “creativity” of the network to tune how predictable or surprising the output is.
![Figure 8.2](figure8-2.png)
- Skipping over the DeepDream content.
- Skipping over the neural style transfer content.
- The key idea for image generation is to develop a latent space that can be sampled from to generate images.
![Figure 8.9](figure8-9.png)
- *Latent space*: the space that holds all possible abstractions obtained from the environment.
- Latent space is the same idea as the embedded space for words where certain directions encode certain features.
![Figure 8.10](figure8-10.png)
- *Variational autoencoders (VAE)*: a variant of autoencoders that turns the input image into a mean and a variance.
- The three steps of VAE
    - Encode the image into two parameters in the latent space, a mean and a variance.
    - Randomly sample a point from the latent space to generate the input image.
    - Decode the point in latent space back into the original input image.
- *Generative adversarial networks (GAN)*: consists of two parts
    - *Generator network*: takes a random vector as input and decodes it into a synthetic image.
    - *Discriminator network*: takes an image (real/synthetic) and predicts whether the image is real or synthetic.
![Figure 8.15](figure8-15.png)
- GANs don’t seek to minimize like SGD, but instead they seek to reach equilibrium.
- GANs are notoriously difficult to train.
- It’s important to freeze the discriminator’s weights while training the GAN. This way the discriminator won’t always predict “real”.
- Steps to train a GAN (per epoch)
    1. Get random points from the latent space.
    2. Generate images using the random points and the `generator`.
    3. Mix the generated images with the real ones.
    4. Train the `discriminator` using the mixed images with corresponding targets (real/fake).
    5. Get new random points from the latent space.
    6. Train `gan` using these random points with targets that all say “real”. Freeze the discriminator’s weights and update the generator’s weights.

**Summary**
![Chapter 8](chapter8.png)

## Chapter 9: Conclusions

- A review on everything learnt in this textbook.
- Anything requiring reasoning is beyond deep learning.
- Neural networks doesn’t understand the task they perform.
![Figure 9.4](figure9-4.png)
- The only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data.
- To make something more efficient, there’s one universal recipe: introduce modularity and hierarchy.
- So please go on learning, questioning, and researching.
- Never stop. Because even given the progress made so far, most of the fundamental questions in AI remain unanswered. Many haven’t even been properly asked yet.

gha_language: Jupyter Notebook


text:
# Phase 0 GPS 1.1

## Terminal Commands

1. mkdir  -Create a folder for repo to live in
2. cd -Navigate into the new directory
3. git clone <URL> -Clone repo into directory
4. touch -Create file (awesome_page.md)
5. git checkout -b -Create and checkout new feature branch
6. subl -Open file in Sublime (README.md)


text:
# Rust-SDL2_mixer

[![Build Status](https://travis-ci.org/andelf/rust-sdl2_mixer.svg?branch=master)](https://travis-ci.org/andelf/rust-sdl2_mixer)

Rust bindings for SDL2_mixer.

## Installation

Requirements

* [Rust-sdl2](https://github.com/AngryLawyer/rust-sdl2)
* sdl_mixer 2.0 development libraries
* Rust nightly
* Cargo nightly

### normal

```
git clone https://github.com/andelf/rust-sdl2_mixer
cargo build
# TODO: OR if you are using the mac framework version
rustc --cfg mac_framework src/sdl2_mixer/lib.rs
```

### cargo

```
[dependencies]
sdl2_mixer = "$version-here$"
```

or

```
[dependencies.sdl2_mixer]
git = "https://github.com/andelf/rust-sdl2_mixer"
```

## Demo

```
cargo run /path/to/wav_file.wav
```

gha_language: Rust


text:
+++
author = "Kendall Clark"
categories = ""
date = "2017-04-26"
discourseUsername = "kendall"
draft = false
series = ["release-preview"]
title = "9 Reasons Stardog 5 is Awesome"
heroShot = "http://blog.stardog.com/img/pinkdog.jpg"
+++

We're prepping Stardog 5 beta for release. If data siloes rule everything around
you, then you need to build a Knowledge Graph with Stardog 5.

**TLDR**: We've made Stardog easier, faster, more resilient, and more useful.

{{% figure src="/img/pinkdog.jpg" class="inliner" %}}

## 1. AWS Deployments are Trivial

Let's start with push-button AWS deployments, i.e., the stuff that gets you home
from work on time. It's now *dead simple easy* to deploy Stardog.

[Stardog Graviton](https://github.com/stardog-union/stardog-graviton)---our
first Apache-licensed, Golang project---compiles to a single binary executable
that lives on a client machine and provides a one-click virtual appliance
installation of Stardog HA Cluster. It leverages the power
of [Amazon Web Services](https://aws.amazon.com/) to deploy, configure, and
launch all of the virtual hardware and software needed for an optimal Stardog
cluster deployment. All you have to provide is the AWS account.

Now, as the man once said, watch this video!

Say thanks to John Bresnahan who wrote Graviton because he lives in Hawaii and
ain't nobody got time for manual deployments when they could be paddling!

## 2. Tableau for a Great Good

A knowledge graph relates data from anywhere, in any format, to data from
everywhere else. Knowledge graph is to data as Cloud is to compute and SANs are
to storage. And one of the uses for a knowledge graph is better understanding
faster, including analytics and visualizations.

<a href="/img/connector.png" class="no-underline"
{{% figure src="/img/connector.png"
           class="inliner no-underline"
           caption="Stardog Data Connector 1.1.1"
           %}}

<a href="/img/locations.png" class="no-underline"
{{% figure src="/img/locations.png"
           class="inliner no-underline"
           caption="Stardog Data Connector 1.1.1"
           %}}

For Stardog 5 we've implemented [Tableau](http://www.tableau.com/) support so
that you can use Tableau's visualization powers to take advantage of the
knowledge graph.

You can get the Stardog Data Connector for
Tableau [here](http://tableau.stardog.com/).

Thanks to the plucky Javascript-focused law firm of Bretz,
Soehngen, & Ellis for this work.

## 3. Queries, Search, and Geospatial are Faster

More or less everything in Stardog 5 is faster, more scalable, and faster, but
especially query, search, and geospatial.

### New Statistics → Faster Queries

In the query eval game, lots of small changes can add up to non-trivial gains,
but we also made some bigger changes, too. For example, statistics. Statistics
are important for query costing and, hence, for query planning and optimization.
But they can be prohibitively expensive to compute and are one of the dark
arts of database design and programming.

We not only wanted stats to be more accurate but we also wanted them to be
cheaper, too.
The [count-min sketch](https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch)
turned out to be just the thing. Stardog 5 computes statistics in a single
streaming pass through one index and doesn't build any auxiliary persistent data
structures. Stardog 5 stats include *two probabilistic data structures* to
replace the previous persistent maps to store object counts for each
characteristic set. *First*, we build a count-median-min sketch to answer
frequency queries for objects. This one is compact and very accurate for
frequently occurring objects. *Second*, we use a Bloom filter to answer
membership queries for rarely occurring objects.

Bloom filters have linear space requirements which make them expensive to
build and manage. Also their size is difficult to know ahead of time since we
don't know the size of each characteristic set before we build it. Therefore, we
take a fixed-size uniform sample of objects occurring in each set using the
reservoir sampling technique and build the Bloom filter accordingly.

For some graphs more accurate statistics make a huge difference in
performance. For example, on
[Yago](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/yago/demo/)
Stardog 5 is **1,600% faster** than Stardog 4.

{{% figure src="/img/frame.jpg" class="inliner" %}}

Pavel Klinov did this work, and he's so modest that he basically forgot it,
stashed in a branch for more time than any of us would like to admit. He's also
an avid cyclist and refuses any Stardog stickers on his new, hand-built frame
because "excess weight kills". At scale, in databases and bike races, everything
matters.

### Faster Search, Too

We improved performance for Stardog's semantic search capability significantly
by optimizing the way we use Lucene. We reduced search times in some hard cases
from several seconds to milliseconds. And by "we", I mean Pedro Oliveira.

The improvements involved using `SearchManager` more cleverly and deleting lots
of unnecessary code. It also involved recognzing our interactions with Lucene
are batch and transaction-oriented, in which case it made more sense to use a
new `IndexWriter` with each transaction. They occupy a fair bit of memory and
are only optimized on `close()`.

* A union of 100 `textMatch`, Stardog 5 is **1,300%** faster than Stardog 4
* A join of 30 `textMatch`, Stadog 5 is **416% faster** than Stardog 4.

### Geospatial is bigly faster...Not sad!

We also improved the performance of spatial queries significantly, largely by
following some of the same tracks laid down in the base Lucene case. Some of
these numbers are, frankly, a little hard to believe, but they're real.

* Queries with `geo:nearby`, from **13 seconds to 10 milliseconds**
* Queries for objects in vicinity of several locations, from **12 seconds to 21 milliseconds**
* Queries with `geo:wktLiteral`, from **10 seconds to 100 milliseconds**
* Queries for a point `geo:within` a polygon, from **15 seconds to 2.5 seconds**...a mere 600% increase, maybe a little sad?!

Some other geospatial performance improvements go beyond query answering:

* Bulk-loading of spatial data is about **80% faster**
* Deletes that involve the search index are roughly **100% faster overall**; when
  there are several named graphs, the improvement is even better---transactions that took minutes now take milliseconds
* Indexing of search data with named graphs is up to **300% faster**

To be honest these search and geospatial numbers are sick, and I blame Pedro
Oliveira for them 100%.

## 4. Query Hints are a Thing

Our query planner is state of the art for graph databases, but query hints are
real and they can be fabulous. So we've added them to Stardog 5. We will add new
hint types in the 5.x release cycle, including for (1) *reasoning*: define the
reasoning mode for a scope; (2) *virtual graphs*: supply information on
cardinality; (3) *query eval*: suggest specific join algorithms for scopes with
two patterns; (3a) whether or not to reorder filters in a scope; (3b) whether or
not to use parallel unions, etc.

{{% figure src="/img/qhint.jpg" caption="Image courtesy of Duncan Hull" class="inliner" %}}

In this release we support two hints. First, `assume.iri` tells Stardog that the
variables `?o` and `?o2` will only be bound to a graph node (i.e, IRIs) and not
to a node property (or its value).

```sql
select ?s where {
         #pragma assume.iri ?o, ?o2
                 <ns:a> ?p  ?o .
                 <ns:b> ?p2 ?o2
}
```

Second, the `group.joins` hint introduces an explicit scoping mechanism to help
with join order optimization. Finding the optimal order of joins for an
arbitrary set of triples patterns is no easy task, especially if the query
contains many triple patterns. Stardog does a very good job most of the time.
The new statistics really help.

But in cases where Stardog fails to find the optimal join order you can guide
the optimizer to use a specific join order using a hint:

```sql
select ?s where {
          ?s1 :p ?o1 .
            { #pragma group.joins
              #these patterns will be joined before being joined with anything else
                ?s1 :p ?o2 .
                ?o1 :p ?o3 .
          }
}
```

Our hints are safely ignored as comments by other systems, but really why would
you even think of doing that? Again, thanks to Pedro and Pavel for the query
hints work. More of this to come throughout 5.x.

## 5. Garbage Collection (almost) Never Happens

Alexander Toktarev already blogged
(["Saying Goodbye to Garbage"]({{< ref "2017-mm.md" >}})) about how memory
management in Stardog 5 goes from the conventional Java garbage collection to
*some other completely different thing*. No, no, go read that post. I'll wait,
it's fascinating stuff. Alexander is, needless to say, a very serious JVM engineer.

Stardog 5 memory management uses some real cutting-edge JVM techniques---which
you'll also find in things like Apache Flink and the like---to allocate memory
in collections using byte arrays in such a way that no JVM GC occurs. See Alex's
post for some of the details. Since that post we've been working on using
Stardog 5 MM ("memory management") in the mapping dictionary and in other
places, too.

Saying goodbye to JVM GC in Stardog 5 means more predictability, more stability,
better performance at scale, and no irritating stop-the-world pauses. Stardog 5
should never OOM and sees dramatic speedups on some very complex queries.

{{% figure src="/img/nog.jpg" caption="Image courtesy of jlwelsh" class="inliner" %}}

For example, an financial services customer has a very complex forensics query
that Stardog 4 just couldn't evaluate. Stardog 5 evaluates this query in about 2
minutes. In some limited cases we see a small slowdown by a constant factor, but
we expect that to be negligible and rare by Stardog 5 final release.

## 6. Core Server is Better in Every Way

Stardog core server has always been a fairly complex pile of Netty code. Mike
Grove wrote all of that code, more or less, so he knew how hard it was to reason
about. Sometimes customers saw puzzling deadlocks that were hard to reproduce
and even harder to fix.

So Mike decided to replace Netty with [Undertow](http://undertow.io/). A couple
of benefits accrued to us almost immediately. First, Mike got to delete **about
60,000 lines of code** net. Be jealous. That doesn't happen often. The new
server is also simpler, easier to extend and reason about, as well as measurably
faster.

The new server has a *simplified threading model*, which means that it's easier
to segment requests by type into separate thread pools and to partition reads
and writes and control the resources they use. It also means that what thread or
thread pool something runs in is much more obvious. The new server is also much
*less prone to thread starvation* by overloading a pool or inadvertently
blocking in an I/O thread.

The new server is easier to develop against for us internally and easier for
everyone else to extend. The API is more concise, with a smaller, flatter
learning curve. It also allows for greater reuse of services via composition.
For example, our implementation of SPARQL Protocol is now **40% less code.**

User-defined extensions of the new server are really easy. You just have to
implement a single interface, declare it such that the JDK service loader picks
it up, put it in the classpath, and restart the server. See
the [Stardog 5 docs](https://stardog.com/docs) for details.

{{% figure src="/img/rock-core.jpg" class="inliner" %}}

All that and **no performance penalty**. In fact, we consistently measure **12%
better throughput.**

## 7. Virtual Graphs are Faster and Easier to Extend

We will publish a blog post from Jess Balint---who built the new Virtual Graph
engine in Stardog 5---soon after the beta release. It will include lots of
details. For now here's a quick preview. Stardog 5's Virtual Graph service is
rewritten from scratch:

* new optimizations to minimize joins
* new optimizations for SPARQL `OPTIONAL`
* pushes aggregates and subqueries down to source systems
* heavy use of Apache Calcite
* create-time validation of mappings
* Apache Hive support

Jess previously wrote about Stardog's Virtual Graph capability
in
[Virtual Graphs: Relational Data in Stardog]({{< ref "2017-bring-relational.md" >}}).
Jess heavily recruited us before we closed our seed round last year and we're
very glad we said yes.

## 8. Cluster is More Resilient

We've completely reworked how HA Cluster works, including how it uses Zookeeper.
This work was done by Evren Sirin, Paul Marshall, Mike Grove, John Bresnahan,
and Scott Fines. As Paul Marshall blogged
(in ["Blockading Stardog"]({{< ref "2017-blockade-stardog-1.md" >}})), we've
added a new kind of testing to Stardog Cluster ("controlled chaos") using the
Blockade open source tool. This means adding extensive test suite, torture
testing, chaos testing to the rest of our testing regime.

Test coverage for the Cluster has increased dramatically. We also improved
Cluster sync performance at new node join time, which helps Stardog customers in
production.

### Improved Performance

Data loading performance in the Cluster, which is critical to uninterrupted
operations, has also dramatically improved in Stardog 5.

{{% figure src="/img/s5cluster.png"
           caption="Data load speed performance: 4.2.4, 5.0, 5.0 server side"
           class="inliner"
           %}}

Before Stardog 5 bulk load in the cluster (`db create` command) first created
the database in one node and then replicated the data to the other nodes in the
cluster. This approach makes error-catching eager, but when there are no errors
in the data it increases loading time.

We've changed this behavior so that the first node that receives data starts
replicating the data to other nodes immediately. All nodes will bulk load the
data concurrently. This change nearly doubles the bulk loading speed in the
cluster. The data transfer time between the client and the server remains
unchanged, but if your files are already stored on the server machines, then you
can omit the `--copy-server-side` option and increase the loading speed even
more.

## 9. Pivotal Cloud Foundry? Also Trivial!

Finally, the work on Stardog Graviton was so much fun that John Bresnahan
thought he'd also do the same thing for the *other* cloud environment we love,
Pivotal's Cloud Foundry. Thanks to our good
friend, [Stuart Charlton](https://twitter.com/svrc), for help on this one.
Stardog PCF Service Broker paves the way for Stardog on Pivotal Network as a
tiled service. **Watch this space for an announcement, which should be in time
for Stardog 5 final release.**

## Conclusion

Finally, let's look at some software engineering stats really fast:

Release cycle    | Months      | +/- lines changed   | Net          | PRs & Issues
--------         | -------     | ---------           | --------     | ------------
3.0 to 4.0       | `8`         | `152,231/-112,788`  | `39,443`     | `419`
4.0 to 5.0       | `18`        | `204,476/-230,341`  | `-25,865`    | `905`

Stardog is net negative KLOC changed over the 4.0 to 5.0 release cycle. For 5.0
final we expect another chunk of deletion, which will put us at a net growth
since 3.0 release of about 10,00 lines of code.

I described 9 reasons Stardog 5 is awesome. But the truth is that there's only
*one* actual reason why Stardog 5 is awesome: **my coworkers are an unstoppable
force of software power.**

Which is to say that I'm only telling you about the amazing work of other
people. It's *these* amazing people who did the work---Jess Balint, John
Bresnahan, Adam Bretz, Mark Ellis, Scott Fines, Michael Grove, Pavel Klinov,
Paul Marshall, Stephen Nowell, Pedro Oliveira, Evren Sirin, Ty Soehngen, and
Alexander Toktarev.

Stardog 5 will be the fastest, most scalable Knowledge Graph platform available
anywhere.

**[Download Stardog today](http://stardog.com/) to start your free 30-day
evaluation.**

repository: stardog-union/stardog-blog

gha_language: HTML

repo_name: stardog-union/stardog-blog


text:
# Warning
1. Before submitting the issue, read the [CONTRIBUTING guidelines](https://github.com/huawei-noah/streamDM/blob/master/CONTRIBUTING.md).
2. Use only one of the templates below to create either an Enhancement or Bug Report issue.

-----------------------------------------------

# Enhancement template

## Description
_<< What is the purpose of this enhancement >>_

## Resources
_<< Is it implemented in another machine learning library? Are there academic papers describing the method? >>_

-----------------------------------------------

# Bug Report template

## Expected behavior
_<< What were you trying to do? In other words, what did you expected to achieve? Describe it in here. >>_

## Observed behavior
_<< What results have you obtained? Is there an error message? If so, please paste it in here (not all issues produce error messages) >>_

## Steps to reproduce the issue

**Command line**
_<< What is the command line to reproduce the behavior? >>_

**Data source**
_<< What data was used? Did you use Kafka? >>_

_<< Does the problem applies to one specific data source only? If so, Include more information about it, such as the features' domain and dimensionality, the data format, and a sample of data. >>_

**Infrastructure details**
* **Java Version:**
* **Scala Version:**
* **Spark version:**
* **OS version:**
* **Cluster mode or local mode?**

repository: QiuQiuDongDong/streamDM

repo_name: QiuQiuDongDong/streamDM


text:
# My-PR-Contribution-Practice

### This space is created for you to make your first pull request/ contribution as a firstimer in open source, also this might not be your first time but involve yourself in this, why?
- To get an addtional 🟩 to your GitHub profile
- To add to your pull request count
- To add to your contributions count
- To make connections, how?
 In this PR/contribution practice, these can be included:
  - Your name
  - Your GitHUb profile link
  - What you do

By these data provided, connections can be made!

- For example ,

  ```- [Full Name](https://github.com/your-username)```

  ```<p>Bio:_________________________________________</p>```

  On a new line in alphabetical order, add your name, bio and the link to your Github profile

____________________________________________________________________________________________________________________

### START CONTRIBUTING BY ADDING YOUR DATA
____________________________________________________________________________________________________________________

- [Alaba Olanrewaju](https://github.com/chryzcodez)

Bio: A developer, technical writer/ blogger, an open-source personnel and an aspiring data scientist

- [Naman Gupta](https://github.com/namangupta1399)

Bio: Computer Science Undergraduate, Aspiring Web Developer

- [Tanimowo Emmanuel](https://github.com/mannuel25)

Bio: I'm an aspiring back-end engineer


text:
# Build guide

## Platform preparation

* GoLang 1.14+

    **Mac OSX**
    ```
    brew install go
    ```

* Python 3.7+ Virtual Environment

    **Mac OSX**
    ```
    brew install python
    pip install virtualenv setuptools wheel
    ```

* Rocksdb 6.22+

    **Mac OSX**
    ```
    brew install rocksdb
    ```

## Environment

### Source checkout

First of all, you need to check out the source.
```bash
git clone $REPOSITORY_URL goloop
```

### Prepare virtual environment
```bash
cd $HOME/goloop
virtualenv -p python3 venv
. venv/bin/activate
```

### Install required packages
```bash
pip install -r pyee/requirements.txt
```

## Build

### Build executables

```bash
make
```

Output binaries are placed under `bin/` directory.

### Build python package

```bash
make pyexec
```

Output files are placed under `build/pyee/dist/` directory.

## Quick start

First step, you need to make a configuration for the node.

```bash
./bin/gochain --save_key_store wallet.json --save config.json
```

It generates a configuration file, `config.json`, along with a wallet keystore file, `wallet.json`.
If you don't specify any password, it uses `gochain` as a default password for the keystore.
You may apply more options while it generates.
Please run `./bin/gochain --help` for more information.

Now, you may start the server with it.

```bash
./bin/gochain --config config.json
```

You may send transactions with the wallet, `wallet.json`, for the initial balance of other wallets.

Note that this is a single node configuration.  If you want to make a network with multiple nodes,
you need to make your own genesis and node configurations.


text:
*Language: [한국어](README.ko.md)*

## Project Details
In Natural Language Processing (NLP), QA, the task of generating answer of the question with the given context, is one of the important tasks to solve. However, the datasets for QA tasks are hard to be human-generated since they should include all context, questions, and answers, causing lots of efforts. Thus, here comes the question generation task, which generates question when the context is given, alleviating the labour taken to generate dataset.<br>

So, the objective of this project is to make Korean-version model of both question generation and answer generation. For this, we used MRC data of [KLUE dataset](https://github.com/KLUE-benchmark/KLUE), and used Korean-pretrained T5 model [KE-T5](https://github.com/AIRC-KETI/ke-t5).


text:
# GeneticOptimize
 Genetic algorithm implements

## Requirement
.NET 5.0

## Install

[Download DLL](https://github.com/tk-yoshimura/GeneticOptimize/releases)

- To install, just import the DLL.
- This library does not change the environment at all.

## Usage
1. Override the optimizer to define the problem.
-> [Source](https://github.com/tk-yoshimura/GeneticOptimize/blob/master/GeneticOptimize/Optimizers/NapsackOptimizer.cs)

2. Describe the parameters in problem and update the generation of the gene pool.
-> [Source](https://github.com/tk-yoshimura/GeneticOptimize/blob/master/GeneticOptimizeTest/Optimizers/NapsackOptimizerTest.cs)

## Licence
[MIT](https://github.com/tk-yoshimura/GeneticOptimize/blob/master/LICENSE)

## Author

[tk-yoshimura](https://github.com/tk-yoshimura)

repository: tk-yoshimura/GeneticOptimize

repo_name: tk-yoshimura/GeneticOptimize