text:
# Rickshaw

Testing out the D3-based Rickshaw time series graph library

## Notes

[Rickshaw](https://tech.shutterstock.com/rickshaw/) is:

* a JavaScript toolkit for creating interactive time series graphs
* built on d3.js
* open source (MIT license)
* developed at Shutterstock
* available on [CDN (cdnjs)](https://cdnjs.com/libraries/rickshaw)

See also:

* [online examples](https://tech.shutterstock.com/rickshaw/examples/)
* [tutorial](https://tech.shutterstock.com/rickshaw/tutorial/introduction.html)

### Installation Options

* download compiled sources from [github](https://github.com/shutterstock/rickshaw)
* install with bower `bower install rickshaw`
* install with npm `npm install --save rickshaw`
* direct from [CDN (cdnjs)](https://cdnjs.com/libraries/rickshaw)

### Using a CDN

Requires:

* [rickshaw](https://cdnjs.com/libraries/rickshaw) css and js
* [d3](https://cdnjs.com/libraries/d3/3.5.16) - but latest versions don't work
* optionally jquery ~>1.8.1

e.g.

```
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/rickshaw/1.6.6/rickshaw.css" integrity="sha256-3vAiod7sePtPgQYpPMFvmDNLWVbCeYiEjb9p4lt3PXQ=" crossorigin="anonymous" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.16/d3.min.js" integrity="sha256-uB5nPcWK8vr5e83snqtMUYJ2n/5TZ3PV9CCRk1pzob4=" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rickshaw/1.6.6/rickshaw.js" integrity="sha256-F4WIKaQsZ/uUDfd1G2srG7DoLrveN3Q4LoaQ+4RI/PQ=" crossorigin="anonymous"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.1/jquery.min.js"></script>
```

### Quick Demo

See [demo.html](./demo.html) for a quick test of:

* applying color palettes
* loading data with AJAX

[![demo](./assets/demo.png?raw=true)](./demo.html)

## Credits and References

* [Rickshaw](https://tech.shutterstock.com/rickshaw/) - home
* [Rickshaw](https://cdnjs.com/libraries/rickshaw) - cdnjs
* [Rickshaw](https://github.com/shutterstock/rickshaw) - github

repository: tardate/LittleCodingKata

branch_name: refs/heads/main

repo_name: tardate/LittleCodingKata


text:
# password-generator
No more need to remember all of your password , you can generate all of they in my programm.
In my program you can generate passwords.
In first line write the length of password,
In second you write your personal key (come up with yourself),
In third you write the link on the site or name of program for which you need password.
This programm generates same passwords if you insert same keys and links,that means that you can always get your password.

license: MIT

extension: md

license_type: permissive


text:
# Heroku Buildpack for Flutter
![header]
Automate your deployments on Heroku easily.

## 🔨 Setup

#### Add this Buildpack in your Heroku app
   In dashboard:

   You can copy the [link](https://github.com/diezep/heroku-buildpack-flutter) of this repository and paste it in buildpacks or write **diezep/flutter** and will be added automatically.

   In Heroku CLI:
   ```shellscript
   $ heroku buildpacks:set diezep/flutter -a <Heroku App name>
   ```

#### Optional

   You can add optional [environment variables](#-environment-variables) to customize the deployment of your project but nothing is required, all are optional.

## 🚧 Environment variables:

All variables were built following the structure **FLUTTER_VARNAME** to identified easily in Heroku configurations. If you want to use some variable, remember **use the structure** following to the variable.

| Variable |   Type  |   Default        |  Description
|----------|---------|------------------| -------------------|
| CLEANUP  | *boolean* |  true |Remove **uneccessary files** of your project after the compile. |
|  VERSION | *string* | *Last version in [stable channel](https://flutter.dev/docs/development/tools/sdk/releases?tab=linux).* | The **name of the version** you want to use to compile the project.
|  BUILD | *string* | flutter build web --release --quiet | Customize the **command to compile** the project.|

### Set variable
   Example of setting CLEANUP var in Heroku CLI:
   ```shellscript
   $ heroku config:set FLUTTER_CLEANUP=true -a project-name
   ```
### Example

The next image is an example of setting environment variables following the structure mentioned above :

![example_use_in_variables](https://user-images.githubusercontent.com/38699812/89090700-42447c00-d36a-11ea-8148-84af7cddfa21.PNG)

## 📝 Final note
   This buildpack is unofficial that means i don't have any conection with Heroku or Google from Flutter developer team . This repository was made as a hobby, i'm always interested in learn new things, this is one demostration of that :)

Recently, [@ludwiktrammer](https://github.com/ludwiktrammer) developed a lighter version of this buildpack with some improvements, if you are having problems with size of project or you are using Heroku CI, i invite you to test it. [here's the repo](https://github.com/EE/heroku-buildpack-flutter-light)

repository: yellowgraysu/my-test-buildpack

branch_name: refs/heads/master

repo_name: yellowgraysu/my-test-buildpack


text:
# NoTest

NoTest is a test automation framework that requires NO testing code.

Its basic idea is to action method-level input/output as test vectors by running your application in real environment, and then playback the test vectors in the future (in either development environment or production environment) to do regression testing.

## Build

run gradle task `shadowJar`, then the output jar will be placed at `<project folder>/build/libs/no-test.jar`

## Demo Usage

### Only support linux & macOS

#### Recording

1. prepare an json file for which method you would like to record, using `com.github.zszlly.util.MethodUtils.toNoTestMethodDescription(Method)` to generate your method description and fill them to a json list.

    Here is an example:

        [
            "Lcom/github/zszlly/Tester;.add(II)I",
            "Lcom/github/zszlly/Tester;.addB(ILcom/github/zszlly/Tester$GetB;)I",
            "Lcom/github/zszlly/Tester;.addArray([Ljava/lang/Integer;)I",
            "Lcom/github/zszlly/Tester;.addAAndB(Lcom/github/zszlly/Tester$GetB;)I"
        ]

2. build the project as build section described.

3. copy the json properties file and built jar file `<project folder>/build/libs/no-test.jar` into the host where the demo program will run at.

4. run demo program in first terminal at the host like:

        java -cp "no-test.jar" com.github.zszlly.DummyMain

5. run `jps` to get your java program pid in second terminal.

6. run command in second terminal:

        java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar -DrecordMethod=<your configuration file path> -DcaseSavingPath=<the path where recorded files will be saved to> -jar no-test.jar <your process PID>

7. return to first terminal and press any key to continue the demo program.

8. you will find the latest 10 method calls for which you filled in your configuration.

#### Playback

you can now just add your project libraries as classpath and run the following command to mock and run the test cases:

    java -cp "no-test.jar:<your project classpath>" com.github.zszlly.player.NoTestPlayer <recorded file path>

repository: zszlly/NoTest

gha_language: Java

repo_name: zszlly/NoTest


text:
# Kibana

An opinionated Docker implementation for Kibana based on the upstream official image.

<!-- TOC depthFrom:2 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 -->

- [Features](#features)
- [Getting Started](#getting-started)
	- [Prerequisites](#prerequisites)
	- [Installing](#installing)
	- [Environmental Variables](#environmental-variables)
- [Deployment](#deployment)
- [Components](#components)
- [Contributing](#contributing)
- [Versioning](#versioning)
- [Authors](#authors)
- [License](#license)
- [Acknowledgments](#acknowledgments)

<!-- /TOC -->

## Features

There is currently only one difference between this implementation and the upstream image.

* Enables Prometheus plugin for metrics.

## Getting Started

This implementation is designed to be used with a Helm chart. However, there is a docker compose file included for testing this implementation locally. Bringing up a test cluster and discovering the client endpoint is easily done with just 2 commands.

```
docker-compose up
```

It will likely take a few minutes for the containers to start and you can verify when it is available using `curl`.

```
curl $(docker-compose port kibana 5601)
```

### Prerequisites

The only requirements for bringing up this implementation locally is Docker.

```
Docker 17.12.0+
```

### Installing

It is expected that the Helm chart will be used to install this container. For local development, there are a number of ways to interact with Kibana.

The Kibana endpoint is the main way applications interact with Kibana.

```
docker-compose port kibana 5601
```

The Elasticsearch endpoint can be found with the following command.

```
docker-compose port elasticsearch 9200
```

### Environmental Variables

See the [official documentation](https://www.elastic.co/guide/en/kibana/current/docker.html) for Kibana running in Docker.

## Deployment

For use on live systems, see the documentation for the Helm chart.

## Components

* [Elasticsearch](https://github.com/elastic/elasticsearch) - Open Source, Distributed, RESTful Search Engine
* [Elasticsearch-Docker](https://github.com/elastic/elasticsearch-docker/tree/6.5) - Official Elasticsearch Docker image from Elastic.
* [Kibana](https://github.com/elastic/kibana) - Kibana analytics and search dashboard for Elasticsearch
* [Kibana-Docker](https://github.com/elastic/kibana-docker/tree/6.5) - Official Kibana Docker image from Elastic.
* [Kibana Prometheus Exporter](https://github.com/pjhampton/kibana-prometheus-exporter) - A Kibana plugin that exposes metrics in the Prometheus format.

## Contributing

Please read [CONTRIBUTING.md](https://gist.github.com/PurpleBooth/b24679402957c63ec426) for details on our code of conduct, and the process for submitting pull requests to us.

## Versioning

We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/fairfaxmedia/elasticsearch/tags).

## Authors

* **Michael Lorant** - *Initial work* - [Nine](https://github.com/mikelorant)

See also the list of [contributors](https://github.com/mikelorant/kibana/contributors) who participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

## Acknowledgments

* Elastic
* Nine

repository: fairfaxmedia/kibana

gha_language: Dockerfile

repo_name: fairfaxmedia/kibana


text:
# tracker-scraper

This is some kind-of-nasty code I cobbled together, to do two things:

1. Whittle a huge list of known trackers down to a smaller list of those that actually work.
2. Search across them for a given torrent hash.

Please note that the tracker-list file is mostly bad trackers, and that this is
not a functional torrent client. It just scrapes a list of trackers for a given hash.

I play with `work_queue` doing 25 (number from a hat) concurrent workers here.

### Finding good trackers

This is slow. It'll query about 300 trackers, most of them defunct, 5 times, once
per hash. (The hashes are random things found on public trackers.) It then prints
whether each is good or bad, and what percentage of queries succeeded.

I don't _think_ a hash unknown to the tracker counts as a failure?

```ruby
require 'tracker_scraper'
t = TrackerScraper.new
t.find_working_trackers
```

### Scraping trackers for a hash

```ruby
require 'tracker_scraper'
t = TrackerScraper.new
t.print_find('3F19B149F53A50E14FC0B79926A391896EABAB6F')
````

It'll print a crude little table of seeders/leechers/downloads for each tracker
that returns something.

Everything except my implementation supports taking an array of hashes, but I wasn't
using it that way so I didn't add it.


text:
---
layout: post
title: Titanic Machine learning from Disaster
subtitle: Python implementation of Titanic Machine learning competition from Kaggle
bigimg: /img/head.png
---

# Titanic Kaggle Challenge
The Titanic challenge on Kaggle is a competition in which the task is to predict the survival or the death of a given passenger based on a set of variables describing him such as his age, his sex, or his passenger class on the boat.

```python
#pandas
import pandas as pd
from pandas import Series,DataFrame

#numpy,matplotlib
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import os
pd.set_option('display.notebook_repr_html',False)
pd.set_option('display.max_columns',12)
pd.set_option('display.max_rows',12)
plt.style.use = 'default'

# machine learning
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
import sklearn
```

## Reading and Exploring the data

http://hamelg.blogspot.in/2015/11/python-for-data-analysis-part-14.html
https://chrisalbon.com/#Kaggle

```python
os.chdir('C:\\Users\\SSKS\\Music\\kaggle\\kaggle-titanic\\data')
df=pd.read_csv('train.csv') ##index_col='PassengerId' can be used to apply it as index label in our dataframe
df_test=pd.read_csv('test.csv')
print(df.head(5))
print('-----------------------')
print(df_test.head(5))
```

       PassengerId  Survived  Pclass  \
    0            1         0       3
    1            2         1       1
    2            3         1       3
    3            4         1       1
    4            5         0       3

                                                    Name     Sex   Age  SibSp  \
    0                            Braund, Mr. Owen Harris    male  22.0      1
    1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1
    2                             Heikkinen, Miss. Laina  female  26.0      0
    3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1
    4                           Allen, Mr. William Henry    male  35.0      0

       Parch            Ticket     Fare Cabin Embarked
    0      0         A/5 21171   7.2500   NaN        S
    1      0          PC 17599  71.2833   C85        C
    2      0  STON/O2. 3101282   7.9250   NaN        S
    3      0            113803  53.1000  C123        S
    4      0            373450   8.0500   NaN        S
    -----------------------
       PassengerId  Pclass                                          Name     Sex  \
    0          892       3                              Kelly, Mr. James    male
    1          893       3              Wilkes, Mrs. James (Ellen Needs)  female
    2          894       2                     Myles, Mr. Thomas Francis    male
    3          895       3                              Wirz, Mr. Albert    male
    4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female

        Age  SibSp  Parch   Ticket     Fare Cabin Embarked
    0  34.5      0      0   330911   7.8292   NaN        Q
    1  47.0      1      0   363272   7.0000   NaN        S
    2  62.0      0      0   240276   9.6875   NaN        Q
    3  27.0      0      0   315154   8.6625   NaN        S
    4  22.0      1      1  3101298  12.2875   NaN        S

The variables/features provided in the dataset are described below:-

PassengerId: id given to each traveler on the boat.
Pclass: the passenger class. It has three possible values: 1,2,3.
The Name
The Sex
The Age
SibSp: number of siblings and spouses traveling with the passenger
Parch: number of parents and children traveling with the passenger
The ticket number
The ticket Fare
The cabin number
The embarkation. It has three possible values S,C,Q

```python
df.describe()
```

           PassengerId    Survived      Pclass         Age       SibSp  \
    count   891.000000  891.000000  891.000000  714.000000  891.000000
    mean    446.000000    0.383838    2.308642   29.699118    0.523008
    std     257.353842    0.486592    0.836071   14.526497    1.102743
    min       1.000000    0.000000    1.000000    0.420000    0.000000
    25%     223.500000    0.000000    2.000000   20.125000    0.000000
    50%     446.000000    0.000000    3.000000   28.000000    0.000000
    75%     668.500000    1.000000    3.000000   38.000000    1.000000
    max     891.000000    1.000000    3.000000   80.000000    8.000000

                Parch        Fare
    count  891.000000  891.000000
    mean     0.381594   32.204208
    std      0.806057   49.693429
    min      0.000000    0.000000
    25%      0.000000    7.910400
    50%      0.000000   14.454200
    75%      0.000000   31.000000
    max      6.000000  512.329200

```python
df_test.describe()
```

           PassengerId      Pclass         Age       SibSp       Parch        Fare
    count   418.000000  418.000000  332.000000  418.000000  418.000000  417.000000
    mean   1100.500000    2.265550   30.272590    0.447368    0.392344   35.627188
    std     120.810458    0.841838   14.181209    0.896760    0.981429   55.907576
    min     892.000000    1.000000    0.170000    0.000000    0.000000    0.000000
    25%     996.250000    1.000000   21.000000    0.000000    0.000000    7.895800
    50%    1100.500000    3.000000   27.000000    0.000000    0.000000   14.454200
    75%    1204.750000    3.000000   39.000000    1.000000    0.000000   31.500000
    max    1309.000000    3.000000   76.000000    8.000000    9.000000  512.329200

Non-numeric columns are dropped from the statistical summary provided by df.describe().Therefore for categorical variables we pass only those columns to the describe() method as shown below

```python
print(df.dtypes)
print('--------------------------------')
print(df.info())
```

    PassengerId      int64
    Survived         int64
    Pclass           int64
    Name            object
    Sex             object
    Age            float64
    SibSp            int64
    Parch            int64
    Ticket          object
    Fare           float64
    Cabin           object
    Embarked        object
    dtype: object
    --------------------------------
    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 12 columns):
    PassengerId    891 non-null int64
    Survived       891 non-null int64
    Pclass         891 non-null int64
    Name           891 non-null object
    Sex            891 non-null object
    Age            714 non-null float64
    SibSp          891 non-null int64
    Parch          891 non-null int64
    Ticket         891 non-null object
    Fare           891 non-null float64
    Cabin          204 non-null object
    Embarked       889 non-null object
    dtypes: float64(2), int64(5), object(5)
    memory usage: 66.2+ KB
    None

```python
categorical=df.columns[df.dtypes=='object']
print(categorical)
df[categorical].describe()
```

    Index(['Name', 'Sex', 'Ticket', 'Cabin', 'Embarked'], dtype='object')

                          Name   Sex  Ticket Cabin Embarked
    count                  891   891     891   204      889
    unique                 891     2     681   147        3
    top     Svensson, Mr. Olof  male  347082    G6        S
    freq                     1   577       7     4      644

```python
print('first 12 Cabin values\n\n',df['Cabin'][0:12])
print('-------------------------------------------')
print('first 12 Ticket values\n\n',df['Ticket'][0:12])
```

    first 12 Cabin values

     0      NaN
    1      C85
    2      NaN
    3     C123
    4      NaN
    5      NaN
    6      E46
    7      NaN
    8      NaN
    9      NaN
    10      G6
    11    C103
    Name: Cabin, dtype: object
    -------------------------------------------
    first 12 Ticket values

     0            A/5 21171
    1             PC 17599
    2     STON/O2. 3101282
    3               113803
    4               373450
    5               330877
    6                17463
    7               349909
    8               347742
    9               237736
    10             PP 9549
    11              113783
    Name: Ticket, dtype: object

This shows the statistical summary of only the categorical variables in our dataset.As we can see the Variable 'Name' has unique values throughout and is not useful in our prediction analysis,therefore we can drop these values.

Furthermore the 'Cabin' variable has only 204 values present and a lot of missing values,so we could drop this column as well from the dataset(we could also fill value by fillna() method,but it is not useful in this case or as the names of the levels for the cabin variable seem to have a regular structure: each starts with a capital letter followed by a number. We could use that structure to reduce the number of levels to make categories large enough that they might be useful for prediction).So we keep the Cabin value in the dataset.

"PassengerId" is just a number assigned to each passenger. It is nothing more than an arbitrary identifier; we could keep it for identification purposes, but let's remove it anyway

"Ticket" has 680 unique values: almost as many as there are passengers. Categorical variables with almost as many levels as there are records are generally not very useful for prediction. We could try to reduce the number of levels by grouping certain tickets together, but the ticket numbers don't appear to follow any logical pattern we could use for grouping. Let's remove it:

```python
df=df.drop(['Ticket','PassengerId','Name'],axis=1)
df_test=df_test.drop(['Ticket','PassengerId','Name'],axis=1)
```

Now we have removed features which are not helpful in our prediction model.Next we will select,transform,impute/fill feature values by exploring the data

## Transform variables

A few variable which are categorical in nature have been encoded as integer types in the Python DataFrame such as "Survived" and "Pclass".

In case of Survived we could convert into Categorical value as below
pd.Categorical(df["Survived"]) but we wont be doing this as when submitting predictions for the competition, the predictions need to be encoded as 0 or 1.

But the "Pclass" seems to be encoded as integer that indicated a passenger class:-
1:First Class
2:Second Class
3:Third Class
These can be converted to Categorical variables as Passenger class is a category,furthermore 1st class would be considered "above" or "higher" than second class, but when encoded as an integer, 1 comes before 2. We can fix this by transforming Pclass into an ordered categorical variable:

```python
##Train data
new_Pclass=pd.Categorical(df['Pclass'],ordered=True)
new_Pclass=new_Pclass.rename_categories(['Class1','Class2','Class3'])
df['Pclass']=new_Pclass
print(new_Pclass.describe())
print('---------------------')
print(df['Pclass'].describe())
print('######################\n\n')
##Test data
new_Pclass=pd.Categorical(df_test['Pclass'],ordered=True)
new_Pclass=new_Pclass.rename_categories(['Class1','Class2','Class3'])
df_test['Pclass']=new_Pclass
print(new_Pclass.describe())
print('---------------------')
print(df_test['Pclass'].describe())

```

                counts     freqs
    categories
    Class1         216  0.242424
    Class2         184  0.206510
    Class3         491  0.551066
    ---------------------
    count        891
    unique         3
    top       Class3
    freq         491
    Name: Pclass, dtype: object
    ######################

                counts     freqs
    categories
    Class1         107  0.255981
    Class2          93  0.222488
    Class3         218  0.521531
    ---------------------
    count        418
    unique         3
    top       Class3
    freq         218
    Name: Pclass, dtype: object

Now lets look at "Cabin" variable,to check if we can combine the different levels based on first letter A,B,C,D.If we grouped cabin just by this letter, we could reduce the number of levels while extracting some useful information.

Refer:https://www.analyticsvidhya.com/blog/2015/11/easy-methods-deal-categorical-variables-predictive-modeling/

```python
##Train data
df["Cabin"].unique()
```

    array([nan, 'C85', 'C123', 'E46', 'G6', 'C103', 'D56', 'A6', 'C23 C25 C27',
           'B78', 'D33', 'B30', 'C52', 'B28', 'C83', 'F33', 'F G73', 'E31',
           'A5', 'D10 D12', 'D26', 'C110', 'B58 B60', 'E101', 'F E69', 'D47',
           'B86', 'F2', 'C2', 'E33', 'B19', 'A7', 'C49', 'F4', 'A32', 'B4',
           'B80', 'A31', 'D36', 'D15', 'C93', 'C78', 'D35', 'C87', 'B77',
           'E67', 'B94', 'C125', 'C99', 'C118', 'D7', 'A19', 'B49', 'D',
           'C22 C26', 'C106', 'C65', 'E36', 'C54', 'B57 B59 B63 B66', 'C7',
           'E34', 'C32', 'B18', 'C124', 'C91', 'E40', 'T', 'C128', 'D37',
           'B35', 'E50', 'C82', 'B96 B98', 'E10', 'E44', 'A34', 'C104', 'C111',
           'C92', 'E38', 'D21', 'E12', 'E63', 'A14', 'B37', 'C30', 'D20',
           'B79', 'E25', 'D46', 'B73', 'C95', 'B38', 'B39', 'B22', 'C86',
           'C70', 'A16', 'C101', 'C68', 'A10', 'E68', 'B41', 'A20', 'D19',
           'D50', 'D9', 'A23', 'B50', 'A26', 'D48', 'E58', 'C126', 'B71',
           'B51 B53 B55', 'D49', 'B5', 'B20', 'F G63', 'C62 C64', 'E24', 'C90',
           'C45', 'E8', 'B101', 'D45', 'C46', 'D30', 'E121', 'D11', 'E77',
           'F38', 'B3', 'D6', 'B82 B84', 'D17', 'A36', 'B102', 'B69', 'E49',
           'C47', 'D28', 'E17', 'A24', 'C50', 'B42', 'C148'], dtype=object)

```python
cabin_char=df['Cabin'].astype(str)
new_cabin=np.array([a[0] for a in cabin_char])
new_cabin=pd.Categorical(new_cabin)
new_cabin.describe()
```

                counts     freqs
    categories
    A               15  0.016835
    B               47  0.052750
    C               59  0.066218
    D               33  0.037037
    E               32  0.035915
    F               13  0.014590
    G                4  0.004489
    T                1  0.001122
    n              687  0.771044

```python
df['Cabin']=new_cabin
```

```python
###Test data
df_test["Cabin"].unique()
cabin_char=df_test['Cabin'].astype(str)
new_cabin=np.array([a[0] for a in cabin_char])
new_cabin=pd.Categorical(new_cabin)
new_cabin.describe()
df_test['Cabin']=new_cabin
```

### Finding and treating NA values,Outliers and strange values

EDA,plotting,treating Outliers and missing values
https://chrisalbon.com/python/pandas_missing_data.html
https://chartio.com/resources/tutorials/how-to-check-if-any-value-is-nan-in-a-pandas-dataframe
https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration
http://www.bogotobogo.com/python/scikit-learn/scikit_machine_learning_Data_Preprocessing-Missing-Data-Categorical-Data.php

Data sets are often littered with missing data, extreme data points called outliers and other strange values. Missing values, outliers and strange values can negatively affect statistical tests and models and may even cause certain functions to fail.

```python
df.isnull().sum()
```

    Survived      0
    Pclass        0
    Sex           0
    Age         177
    SibSp         0
    Parch         0
    Fare          0
    Cabin         0
    Embarked      2
    dtype: int64

Detecting missing values is the easy part: it is far more difficult to decide how to handle them. In cases where you have a lot of data and only a few missing values, it might make sense to simply delete records with missing values present. On the other hand, if you have more than a handful of missing values, removing records with missing values could cause you to get rid of a lot of data. Missing values in categorical data are not particularly troubling because you can simply treat NA as an additional category. Missing values in numeric variables are more troublesome, since you can't just treat a missing value as number. As it happens, the Titanic dataset has some NA's in the Age variable:

```python
df['Age'].describe()
```

    count    714.000000
    mean      29.699118
    std       14.526497
    min        0.420000
    25%       20.125000
    50%       28.000000
    75%       38.000000
    max       80.000000
    Name: Age, dtype: float64

```python
np.where(df['Age'].isnull()==True)
```

    (array([  5,  17,  19,  26,  28,  29,  31,  32,  36,  42,  45,  46,  47,
             48,  55,  64,  65,  76,  77,  82,  87,  95, 101, 107, 109, 121,
            126, 128, 140, 154, 158, 159, 166, 168, 176, 180, 181, 185, 186,
            196, 198, 201, 214, 223, 229, 235, 240, 241, 250, 256, 260, 264,
            270, 274, 277, 284, 295, 298, 300, 301, 303, 304, 306, 324, 330,
            334, 335, 347, 351, 354, 358, 359, 364, 367, 368, 375, 384, 388,
            409, 410, 411, 413, 415, 420, 425, 428, 431, 444, 451, 454, 457,
            459, 464, 466, 468, 470, 475, 481, 485, 490, 495, 497, 502, 507,
            511, 517, 522, 524, 527, 531, 533, 538, 547, 552, 557, 560, 563,
            564, 568, 573, 578, 584, 589, 593, 596, 598, 601, 602, 611, 612,
            613, 629, 633, 639, 643, 648, 650, 653, 656, 667, 669, 674, 680,
            692, 697, 709, 711, 718, 727, 732, 738, 739, 740, 760, 766, 768,
            773, 776, 778, 783, 790, 792, 793, 815, 825, 826, 828, 832, 837,
            839, 846, 849, 859, 863, 868, 878, 888], dtype=int32),)

714 values count of age is much lesser than the total count of values 891,indicating a case of missing data.We could fill the values with mean/median but it is much safer to do some visualization to identify the distribution of the values and later decide how to treat the missing values.

Here are a few ways we could deal with them:
Replace the null values with 0s
Replace the null values with some central value like the mean or median
Impute values (estimate values using statistical/predictive modeling methods.).
Split the data set into two parts: one set with where records have an Age value and another set where age is null.

```python
df.hist(column='Age',
       figsize=(9,6),
       bins=20)
```

    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x07830CB0>]], dtype=object)

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_27_1.png)

On plotting the histogram for the 'Age' variable we are able to see it is slightly right skewed and therefore we can use the median value 20-30 to fill the missing values

```python
###Test data
df_test['Age'].isnull().sum()
```

    86

```python
df_test.hist(column='Age',
           figsize=(9,6),
            bins=20)
```

    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x07830CF0>]], dtype=object)

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_30_1.png)

```python
df['Age'].fillna(df['Age'].median(),inplace=True)
df_test['Age'].fillna(df_test['Age'].median(),inplace=True)
```

```python
df['Age'].describe()
```

    count    891.000000
    mean      29.361582
    std       13.019697
    min        0.420000
    25%       22.000000
    50%       28.000000
    75%       35.000000
    max       80.000000
    Name: Age, dtype: float64

Now lets look at the histogram after filling median values for just a sanity check and its distribution.

```python
df.hist(column='Age',
       figsize=(9,6),
       bins=20)
```

    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x0794DCB0>]], dtype=object)

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_34_1.png)

```python
df_test.hist(column='Age',
       figsize=(9,6),
       bins=20)
```

    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x079E8290>]], dtype=object)

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_35_1.png)

Filling the missing values with median is better than deleting entire rows with missing values, even though the median value 28 might be off the actual values.In practice imputing the missing data (estimating age based on other variables) might have been a better option, but we'll stick with this for now.

Outliers

Outliers are extreme numerical values: values that lie far away from the typical values a variable takes on. Creating plots is one of the quickest ways to detect outliers. For instance, the histogram above shows that 1 or 2 passengers were near age 80. Ages near 80 are uncommon for this data set, but in looking at the general shape of the data seeing one or two 80 year olds doesn't seem particularly surprising.
Now let's investigate the "Fare" variable. This time we'll use a boxplot, since boxplots are designed to show the spread of the data and help identify outliers:

```python
df['Fare'].plot(kind='box',
               figsize=(9,9))
```

    <matplotlib.axes._subplots.AxesSubplot at 0x7b9af90>

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_38_1.png)

```python
df_test['Fare'].plot(kind='box',
               figsize=(9,9))
```

    <matplotlib.axes._subplots.AxesSubplot at 0x7bcadd0>

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_39_1.png)

In a boxplot, the central box represents 50% of the data and the central bar represents the median. The dotted lines with bars on the ends are "whiskers" which encompass the great majority of the data and points beyond the whiskers indicate uncommon values. In this case, we have some uncommon values that are so far away from the typical value that the box appears squashed in the plot: this is a clear indication of outliers. Indeed, it looks like one passenger paid almost twice as much as any other passenger. Even the passengers that paid between 200 and 300 are far higher than the vast majority of the other passengers.
For interest's sake, let's check the name of this high roller:

```python
maxfare=np.where(df['Fare']==max(df['Fare']))
df.loc[maxfare]
```

         Survived  Pclass     Sex   Age  SibSp  Parch      Fare Cabin Embarked
    258         1  Class1  female  35.0      0      0  512.3292     n        C
    679         1  Class1    male  36.0      0      1  512.3292     B        C
    737         1  Class1    male  35.0      0      0  512.3292     B        C

In the graph there appears to be one passenger who paid more than all the others, but the output above shows that there were actually three passengers who all paid the same high fare.
Similar to NA values, there's no single cure for outliers. You can keep them, delete them or transform them in some way to try to reduce their impact.  Let's keep the three high rollers unchanged.
Data sets can have other strange values beyond missing values and outliers that you may need to address. Sometimes data is mislabeled or simply erroneous; bad data can corrupt any sort of analysis so it is important to address these sorts of issues before doing too much work.

Creating new variables:

The variables present when you load a data set aren't always the most useful variables for analysis. Creating new variables that are derivations or combinations existing ones is a common step to take before jumping into an analysis or modeling task.
For example, imagine you are analyzing web site auctions where one of the data fields is a text description of the item being sold. A raw block of text is difficult to use in any sort of analysis, but you could create new variables from it such as a variable storing the length of the description or variables indicating the presence of certain keywords.
Creating a new variable can be as simple as taking one variable and adding, multiplying or dividing by another. Let's create a new variable, Family, that combines SibSp and Parch to indicate the total number of family members (siblings, spouses, parents and children) a passenger has on board:

```python
df.isnull().sum()
```

    Survived    0
    Pclass      0
    Sex         0
    Age         0
    SibSp       0
    Parch       0
    Fare        0
    Cabin       0
    Embarked    2
    dtype: int64

```python
df_test.isnull().sum()
```

    Pclass      0
    Sex         0
    Age         0
    SibSp       0
    Parch       0
    Fare        1
    Cabin       0
    Embarked    0
    dtype: int64

```python
nofare=np.where(df_test['Fare'].isnull()==True)
df_test.loc[nofare]
```

         Pclass   Sex   Age  SibSp  Parch  Fare Cabin Embarked
    152  Class3  male  60.5      0      0   NaN     n        S

```python
df_test['Fare'].fillna(df_test.groupby('Pclass')['Fare'].transform('mean'),inplace=True)###filling NA value with mean of fare grouped by Pclass
df_test.isnull().sum()
```

    Pclass      0
    Sex         0
    Age         0
    SibSp       0
    Parch       0
    Fare        0
    Cabin       0
    Embarked    0
    dtype: int64

```python
df['Embarked'].describe()
```

    count     889
    unique      3
    top         S
    freq      644
    Name: Embarked, dtype: object

```python
###Filling missing embarked values with most frequently occuring value --'S'
df['Embarked']=np.where(df['Embarked'].isnull(),
               'S',
               df['Embarked'])

```

```python
df['Embarked'].describe()
```

    count     891
    unique      3
    top         S
    freq      646
    Name: Embarked, dtype: object

```python
df.isnull().sum()
```

    Survived    0
    Pclass      0
    Sex         0
    Age         0
    SibSp       0
    Parch       0
    Fare        0
    Cabin       0
    Embarked    0
    dtype: int64

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 9 columns):
    Survived    891 non-null int64
    Pclass      891 non-null category
    Sex         891 non-null object
    Age         891 non-null float64
    SibSp       891 non-null int64
    Parch       891 non-null int64
    Fare        891 non-null float64
    Cabin       891 non-null category
    Embarked    891 non-null object
    dtypes: category(2), float64(2), int64(3), object(2)
    memory usage: 43.6+ KB

```python
df.describe()
```

             Survived         Age       SibSp       Parch        Fare
    count  891.000000  891.000000  891.000000  891.000000  891.000000
    mean     0.383838   29.361582    0.523008    0.381594   32.204208
    std      0.486592   13.019697    1.102743    0.806057   49.693429
    min      0.000000    0.420000    0.000000    0.000000    0.000000
    25%      0.000000   22.000000    0.000000    0.000000    7.910400
    50%      0.000000   28.000000    0.000000    0.000000   14.454200
    75%      1.000000   35.000000    1.000000    0.000000   31.000000
    max      1.000000   80.000000    8.000000    6.000000  512.329200

```python
df_test.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 418 entries, 0 to 417
    Data columns (total 8 columns):
    Pclass      418 non-null category
    Sex         418 non-null object
    Age         418 non-null float64
    SibSp       418 non-null int64
    Parch       418 non-null int64
    Fare        418 non-null float64
    Cabin       418 non-null category
    Embarked    418 non-null object
    dtypes: category(2), float64(2), int64(2), object(2)
    memory usage: 17.2+ KB

```python
#category=['Pclass','Sex','Cabin','Embarked']
#from sklearn.preprocessing import LabelEncoder
#lbl_enc=LabelEncoder()
#for label in category:
#    df[label]=lbl_enc.fit_transform(df[label])
```

```python
#df[category].describe()
```

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 9 columns):
    Survived    891 non-null int64
    Pclass      891 non-null category
    Sex         891 non-null object
    Age         891 non-null float64
    SibSp       891 non-null int64
    Parch       891 non-null int64
    Fare        891 non-null float64
    Cabin       891 non-null category
    Embarked    891 non-null object
    dtypes: category(2), float64(2), int64(3), object(2)
    memory usage: 43.6+ KB

## Dealing with Categorical variables

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality.

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.

One-Hot-Encoding has a the advantage that the result is binary rather than ordinal and that everything sits in an orthogonal vector space. The disadvantage is that for high cardinality, the feature space can really blow up quickly and you start fighting with the curse of dimensionality. In these cases, I typically employ one-hot-encoding followed by PCA for dimensionality reduction. I find that the judicious combination of one-hot plus PCA can seldom be beat by other encoding schemes. PCA finds the linear overlap, so will naturally tend to group similar features into the same feature.

Many a times, you have a machine learning problem with a data set where you have one ore more categorical features/columns. Now, there are generally three parts to a machine learning problem, prepare/clean the data, do feature selection, fit models and predict.

In feature selection phase, if you plan to use things like chi square, variance (note if you have extremely skewed data set, say with 95% false/0 target values and 5% true/>0 target values, a very low variance feature might also be an important feature), L1/Lasso regularized Logistic Regression or Support Vector (with Linear Kernel), Principal component analysis etc, you will need to convert your categorical values to one/against all in each column. If you have only categorical values, or a mixture, and your target is a class, and you are using trees, information gain etc to do the feature selection phase, then you will not need this conversion.

Similarly in the fit models and predict phase, if you are using any algorithm other than trees/clustering where your feature values will be multiplied by co-efficients, then you will need to covert your categorical values into one/against all in each column. It is possible though that the library you are using in R or Matlab or Python or R or SPSS, may already have this option inbuilt. So, do check before doing the conversion yourself.

If you need to do the conversion, this is how you do it in Python using OneHotEncoder, LabelEncoder

http://www.bogotobogo.com/python/scikit-learn/scikit_machine_learning_Data_Preprocessing-Missing-Data-Categorical-Data.php

```python
type(df['Pclass'])
```

    pandas.core.series.Series

```python
##Label Encoder

label_encoder = sklearn.preprocessing.LabelEncoder()
category=['Pclass','Sex','Cabin','Embarked']
for col in category:
    data_label_encoded_train = label_encoder.fit_transform(df[col])
    df[col] = data_label_encoded_train

    data_label_encoded_test = label_encoder.fit_transform(df_test[col])
    df_test[col] = data_label_encoded_test
#data_feature_one_hot_encoded = encoder.fit_transform(df[['Pclass']].as_matrix())

```

```python
##One Hot Encoder
from sklearn.preprocessing import OneHotEncoder
enc=OneHotEncoder(sparse=False)
df_train1=df
df_test1=df_test
category=['Pclass','Sex','Cabin','Embarked']
for col in category:
    # creating an exhaustive list of all possible categorical values
    data=df[[col]].append(df_test[[col]])
    enc.fit(data)
    # Fitting One Hot Encoding on train data
    temp = enc.transform(df[[col]])
    # Changing the encoded features into a data frame with new column names
    temp=pd.DataFrame(temp,columns=[(col+"_"+str(i)) for i in data[col].value_counts().index])
    # In side by side concatenation index values should be same
    # Setting the index values similar to the X_train data frame
    temp=temp.set_index(df.index.values)
    # adding the new One Hot Encoded varibales to the train data frame
    df_train1=pd.concat([df_train1,temp],axis=1)
    # fitting One Hot Encoding on test data
    temp = enc.transform(df_test[[col]])
    # changing it into data frame and adding column names
    temp=pd.DataFrame(temp,columns=[(col+"_"+str(i)) for i in data[col].value_counts().index])
    # Setting the index for proper concatenation
    temp=temp.set_index(df_test.index.values)
    # adding the new One Hot Encoded varibales to test data frame
    df_test1=pd.concat([df_test1,temp],axis=1)
```

```python
df_train1['Cabin']
```

    0      8
    1      2
    2      8
    3      2
    4      8
    5      8
          ..
    885    8
    886    8
    887    1
    888    8
    889    2
    890    8
    Name: Cabin, dtype: int32

```python
df_train1.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 26 columns):
    Survived      891 non-null int64
    Pclass        891 non-null int32
    Sex           891 non-null int32
    Age           891 non-null float64
    SibSp         891 non-null int64
    Parch         891 non-null int64
    Fare          891 non-null float64
    Cabin         891 non-null int32
    Embarked      891 non-null int32
    Pclass_2      891 non-null float64
    Pclass_0      891 non-null float64
    Pclass_1      891 non-null float64
    Sex_1         891 non-null float64
    Sex_0         891 non-null float64
    Cabin_8       891 non-null float64
    Cabin_7       891 non-null float64
    Cabin_2       891 non-null float64
    Cabin_1       891 non-null float64
    Cabin_3       891 non-null float64
    Cabin_4       891 non-null float64
    Cabin_0       891 non-null float64
    Cabin_5       891 non-null float64
    Cabin_6       891 non-null float64
    Embarked_2    891 non-null float64
    Embarked_0    891 non-null float64
    Embarked_1    891 non-null float64
    dtypes: float64(19), int32(4), int64(3)
    memory usage: 167.1 KB

```python
df_train=df_train1.drop(['Sex','Cabin','Embarked','Pclass'],axis=1)
df_test=df_test1.drop(['Sex','Cabin','Embarked','Pclass'],axis=1)
```

```python
df_train.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 22 columns):
    Survived      891 non-null int64
    Age           891 non-null float64
    SibSp         891 non-null int64
    Parch         891 non-null int64
    Fare          891 non-null float64
    Pclass_2      891 non-null float64
    Pclass_0      891 non-null float64
    Pclass_1      891 non-null float64
    Sex_1         891 non-null float64
    Sex_0         891 non-null float64
    Cabin_8       891 non-null float64
    Cabin_7       891 non-null float64
    Cabin_2       891 non-null float64
    Cabin_1       891 non-null float64
    Cabin_3       891 non-null float64
    Cabin_4       891 non-null float64
    Cabin_0       891 non-null float64
    Cabin_5       891 non-null float64
    Cabin_6       891 non-null float64
    Embarked_2    891 non-null float64
    Embarked_0    891 non-null float64
    Embarked_1    891 non-null float64
    dtypes: float64(19), int64(3)
    memory usage: 153.2 KB

## Feature Scaling and Feature Standardization

https://www.analyticsvidhya.com/blog/2015/11/8-ways-deal-continuous-variables-predictive-modeling/

```python
df_train[['Age','SibSp','Parch','Fare']].hist(figsize =[11,11])
```

    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x07C305B0>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x080D1AF0>],
           [<matplotlib.axes._subplots.AxesSubplot object at 0x0810C370>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x081485D0>]], dtype=object)

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_68_1.png)

As we can see the continuous variable features are not having the same range of values and are not properly scaled,it is very important to implement feature scaling to improve the performance of the model.

But before putting your raw data into the model, it’s important that it’s in the right format. This means ensuring that your predictor variables are numerical and properly scaled.

Scaling methods

Standard scale

Features can be scaled to have mean 0 and variance 1, which is also known as calculating Z-scores for every observation. This will usually transform the data so that most of your data falls between [-3, 3].

Min-max scale

Another way of scaling is to subtract each value by the minimum value and divide by the range. This is called min-max scaling and it transforms your feature to a scale of [0, 1]. For better or worse, this scaling technique does not change the distribution of your data.

http://pythonhow.com/accessing-dataframe-columns-rows-and-cells

```python
X=df_train.values
Y=df_train['Survived'].values
#X has still Survived values in it, which should not be there. So we drop in numpy column which is the 1st column.
X=np.delete(X,0,axis=1)
X_tst=df_test.values

```

```python
plt.figure(figsize=(10,8))
plt.hist(X[:,0])
np.var(X[:,0])
```

<PHONE_NUMBER>6193815

![png](/kaggle_titanic_notebook_files/kaggle_titanic_notebook_71_1.png)

```python
std_scalar = sklearn.preprocessing.StandardScaler()
std_scalar.fit(X)
X_train_std = std_scalar.transform(X)
Kaggle_test_std = std_scalar.transform(X_tst)
```

```python
mean_vec=np.mean(X_train_std,axis=0)
print(mean_vec)###these values are close to zero

var_vec=np.var(X_train_std,axis=0)
var_vec## has unit variance after standardization
```

    [  2.27277979e-16   4.38606627e-17   5.38289951e-17   3.98733297e-18
      -7.57593265e-17   1.99366649e-17  -6.77846605e-17   3.98733297e-17
      -1.15632656e-16   5.98099946e-18   4.38606627e-17  -6.77846605e-17
      -2.99049973e-17  -4.98416622e-17  -1.99366649e-18   3.98733297e-18
       0.00000000e+00  -9.56959913e-17  -1.99366649e-17   0.00000000e+00
      -8.37339924e-17]

    array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])

```python

```

```python
from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X_train_std,Y,test_size=0.3,random_state=0)
```

    D:\Windows.old\Users\Public\Documents\programs\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)

```python
print('Class label frequencies')

print('\nTraining Dataset:')
for l in range(0,2):
    print('Class {:} samples: {:.2%}'.format(l, list(y_train).count(l)/y_train.shape[0]))

print('\nTest Dataset:')
for l in range(0,2):
    print('Class {:} samples: {:.2%}'.format(l, list(y_test).count(l)/y_test.shape[0]))
```

    Class label frequencies

    Training Dataset:
    Class 0 samples: 61.16%
    Class 1 samples: 38.84%

    Test Dataset:
    Class 0 samples: 62.69%
    Class 1 samples: 37.31%

As we can see after the train test split of our data,the class label frequencies are equally distributed in equal proportion across the train/test split which is whats expected - balanced class distribution in both.

Feature selection http://machinelearningmastery.com/feature-selection-machine-learning-python/

## Model Building

### 1) simple Decision Tree Classifier machine learning algorithm

http://dni-institute.in/blogs/decision-tree-entropy-and-information-gain/
https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/

```python
clf1=DecisionTreeClassifier(max_depth=5)
clf1.fit(X_train,y_train)
clf1.score(X_test,y_test)
```

    0.82089552238805974

Not bad it gives score of 81.34%.Decision trees compute entropy in the information system. If you peform a decision tree on dataset, the variable importances_ contains important information on what columns of data has large variances thus contributing to the decision. Lets see the output.We can see 9th element 0.53 indicating Sex_0 which is female class having 53% importance in deciding target class as well as 1st element age having 12% importance

```python
clf1.feature_importances_
```

    array([ 0.13744537,  0.04472946,  0.00938927,  0.07390082,  0.00629757,
            0.        ,  0.11238557,  0.        ,  0.53247143,  0.        ,
            0.        ,  0.        ,  0.        ,  0.01524026,  0.        ,
            0.        ,  0.        ,  0.05036617,  0.        ,  0.        ,
            0.01777407])

```python
df_train.columns[1:]
```

    Index(['Age', 'SibSp', 'Parch', 'Fare', 'Pclass_2', 'Pclass_0', 'Pclass_1',
           'Sex_1', 'Sex_0', 'Cabin_8', 'Cabin_7', 'Cabin_2', 'Cabin_1', 'Cabin_3',
           'Cabin_4', 'Cabin_0', 'Cabin_5', 'Cabin_6', 'Embarked_2', 'Embarked_0',
           'Embarked_1'],
          dtype='object')

## 2) RandomForestClassifier

using GridSearchCV to tune and select the hyperparameters of model

conduct the grid search using scikit-learn's GridSearchCV which stands for grid search cross validation. By default, the GridSearchCV's cross validation uses 3-fold KFold or StratifiedKFold depending on the situation.

http://blog.datadive.net/selecting-good-features-part-iii-random-forests/
https://www.analyticsvidhya.com/blog/2014/06/introduction-random-forest-simplified/

```python

from sklearn.model_selection import GridSearchCV
param_grid = {'n_estimators': range(100,500,100), 'max_features': ['sqrt','log2'],'min_samples_leaf':range(20,80,20)}

clf2=RandomForestClassifier(random_state=1)
#clf.fit(X_train,y_train)
#clf.score(X_test,y_test)

model=GridSearchCV(estimator=clf2,param_grid=param_grid,n_jobs=10,cv=10)
model.fit(X_train,y_train)
```

    GridSearchCV(cv=10, error_score='raise',
           estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                max_depth=None, max_features='auto', max_leaf_nodes=None,
                min_impurity_split=1e-07, min_samples_leaf=1,
                min_samples_split=2, min_weight_fraction_leaf=0.0,
                n_estimators=10, n_jobs=1, oob_score=False, random_state=1,
                verbose=0, warm_start=False),
           fit_params={}, iid=True, n_jobs=10,
           param_grid={'n_estimators': range(100, 500, 100), 'max_features': ['sqrt', 'log2'], 'min_samples_leaf': range(20, 80, 20)},
           pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
           scoring=None, verbose=0)

```python
model.best_params_
```

    {'max_features': 'sqrt', 'min_samples_leaf': 20, 'n_estimators': 300}

We tune the hyperparameters of our RandomForest model to find best set of parameters by a process called grid search. Actually what it does is simply iterating through all the possible combinations and find the best one.We can see that the GridSearchCV has found the best parameter for max_features as sqrt of total features and n_estimators as 300,so we create our model with these parameters and fit to our training data.We get an accuracy score of 83.9% with RandomForest algorithm.

```python
clf2=RandomForestClassifier(max_features='sqrt',n_estimators=300,min_samples_leaf=2,random_state=10)
clf2.fit(X_train,y_train)
clf2.score(X_test,y_test)
```

    0.83955223880597019

## 3)GradientBoostingClassifier.

```python
from sklearn.model_selection import GridSearchCV
#clf3 = GradientBoostingClassifier(learning_rate=0.05,min_samples_split=500,min_samples_leaf=50,max_depth=8,max_features='sqrt',subsample=0.8,random_state=10)
clf3=GradientBoostingClassifier(learning_rate=0.1,random_state=10)
param_grid={'n_estimators':range(20,81,10)}
model=GridSearchCV(clf3,param_grid=param_grid,n_jobs=4,iid=False, cv=5)
model.fit(X_train,y_train)
#clf.fit (X_train, y_train)
#clf.score (X_test, y_test)
```

    GridSearchCV(cv=5, error_score='raise',
           estimator=GradientBoostingClassifier(criterion='friedman_mse', init=None,
                  learning_rate=0.1, loss='deviance', max_depth=3,
                  max_features=None, max_leaf_nodes=None,
                  min_impurity_split=1e-07, min_samples_leaf=1,
                  min_samples_split=2, min_weight_fraction_leaf=0.0,
                  n_estimators=100, presort='auto', random_state=10,
                  subsample=1.0, verbose=0, warm_start=False),
           fit_params={}, iid=False, n_jobs=4,
           param_grid={'n_estimators': range(20, 81, 10)},
           pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
           scoring=None, verbose=0)

```python
model.best_params_,model.grid_scores_
clf3=GradientBoostingClassifier(learning_rate=0.1,random_state=10,n_estimators=40,max_depth=9,max_features='sqrt',min_samples_leaf=5)
clf3.fit(X_train,y_train)
clf3.score(X_test,y_test)
```

    D:\Windows.old\Users\Public\Documents\programs\lib\site-packages\sklearn\model_selection\_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20
      DeprecationWarning)

    0.83955223880597019

##  gbm https://www.analyticsvidhya.com/blog/2016/02/complete-guide-parameter-tuning-gradient-boosting-gbm-python/

## xgboost http://machinelearningmastery.com/tune-number-size-decision-trees-xgboost-python/

```python
y_results=clf2.predict(Kaggle_test_std)
```

```python
y_results.shape
```

    (418,)

```python
a=pd.read_csv('test.csv')
output=np.column_stack((a['PassengerId'].values,y_results))
df_results=pd.DataFrame(output.astype('int'),columns=['PassengerID','Survived'])
df_results.to_csv('titanic_results.csv',index=False)
```

```python

```

```python

```

```python

```

```python

```


text:
TVWeb is the web interface for TiledWiz.

# Structure

```

app/
	static/
		dist/ # Bootstrap, jQuery, other frameworks files (css, js)
			css/
			js/
			webfonts/ # To use with fontawesome (css in ../css)
		css/ # My CSS files
		js/ # My JS files
		images/
			icons/
		favicon.ico # To be displayed
	templates/
		base.html # Imports (js/css) + block "additional scripts"
		base_simple.html # Classic page
		base_forms.html # Page suited for forms templates
		base_grid.html # Page with menu on the left, legend space at the top, grid space in the middle
		main.html # Filling
		grid.html # Filling
	__init__.py # SocketIO(app, host) is *here*
	forms.py # Implementation of various forms to be later embedded (as imports)
	routes.py # Routes, template rendering, event processing (socketio)
	models.py # Data models (ie database content management)
tvweb.py # Only imports app (no code here), main file in regard to variable definition: export FLASK_APP=tvweb.py
config.py # Non-critical parameters (critical ones should never be written in this repo)
requirements.txt # Packages listed to be installed through pip on the web server
```

# Conventions
## Naming
Socket.io communications:
* Server-side: data received from the client will be named `cdata`, data sent to the client will be named `sdata`
* Client-side: data received from the server will be named `sdata`, data sent to the server will be named `cdata`

# Logging
A FIFO is created in Flaskdock to modify TiledViz.log output level for clicks in web clients :
LOGLEV=WARNING by default.
To change that, connect to the container as flaskusr and give new level (ERROR, WARNING, INFO, DEBUG)
> echo INFO > /tmp/logfifo

repository: mmancip/TiledViz

repo_name: mmancip/TiledViz


text:
# BUSINESS MESSAGES: BM-GOTCHI-BOT / LIT-PET

This sample demonstrates an interactive experience between a user and a bot using
the Business Messages pltaform and the Node.js SDK.

This digital agent is a virtual pet simulation game. The pet starts off in a
happy state and over time needs play time, needs to be fed, and be taken care
of.

This sample is setup to run on the Google App Engine.

See the Google App Engine (https://cloud.google.com/appengine/docs/nodejs/) standard environment
documentation for more detailed instructions.

## PREREQUISITES

You must have the following software installed on your development machine:

* [Google Cloud SDK](https://cloud.google.com/sdk/) (aka gcloud)
* [Node.js](https://nodejs.org/en/) - version 10 or above

## SETUP

Register with Business Messages:

    1.  Open [Google Cloud Console](https://console.cloud.google.com) with your
        Business Messages Google account and create a new project for your agent.

        Note the **Project ID** and **Project number** values.

    2.  Open the
        [Business Messages API](https://console.developers.google.com/apis/library/businessmessages.googleapis.com)
        in the API Library.

    3.  Click **Enable**.

    4.  [Register your project](https://developers.google.com/business-communications/business-messages/guides/set-up/register)
        with Business Messages.

    5.  Create a service account.

        1.   Navigate to [Credentials](https://console.cloud.google.com/apis/credentials).

        2.   Click **Create service account**.

        3.   For **Service account name**, enter your agent's name, then click **Create**.

        4.   For **Select a role**, choose **Project** > **Editor**, the click **Continue**.

        5.   Under **Create key**, choose **JSON**, then click **Create**.

             Your browser downloads the service account key. Store it in a secure location.

    6.  Click **Done**.

    7.  Copy the JSON credentials file into this sample's /resources
        folder and rename it to "bm-agent-service-account-credentials.json".

Create a Business Messages agent:

    *   Open the [Create an agent](https://developers.google.com/business-communications/business-messages/guides/set-up/agent)
        guide and follow the instructions to create a Business Messages agent.

Create a Firebase Realtime Database:

    1.  Open [Firebase Console](https://console.firebase.google.com/) with your Google account
        and create new Firebase project.

    2.  Select the GCP Project you used for enabling the Business Messages API above. Accept the
        Terms and Conditions, select billing type, click Continue and Add Firebase.

    3.  After the resources are provisionsed, headover to your Firebase Project. On the left
        navigation, select Develop > Realtime Database and then click **Create Database**
        and start your database in **Locked Mode**.

    4.  Your Realtime Database will then be created and a URL will be provided for you to
        reference this database. Update ./lib/firebase_helper.js `https://FIREBASE_REALTIME_DATABASE_URL`
        to match the URL provided by Realtime Database in Firebase.

Create a Firebase Service Account.

    1.  Return to the [GCP Developer Console](https://console.cloud.google.com) and open the
        respective project used for Business Messages.

    2.  Navigate to [Credentials](https://console.cloud.google.com/apis/credentials).

    3.  In Service Accounts, a new entry should have been added for Firebase named
        `firebase-adminsdk`.
        Click it to open the service account details.

    4.  Under **Add Key** > **Create new key**, choose **JSON**, then click **Create**.

    5.  Your browser downloads the service account key. Store it in a secure location.

    6.  Click **Done**.

    7.  Copy the JSON credentials file into this sample's /resources
        folder and rename it to "firebase-credentials.json"

## RUN THE SAMPLE

    1.  In a terminal, navigate to this sample's root directory.

    2.  Run the following commands:

        gcloud config set project PROJECT_ID

        Where PROJECT_ID is the project ID for the project you created when you registered for
        Business Messages.

        gcloud app deploy

    3.  On your mobile device, use the test business URL associated with the
        Business Messages agent you created.


text:
---
layout: fluid/pro_docs_base
category: pro
id: pro-live-deploy-plugin-api
title: Deploy PLugin API - Ionic Pro Documentation
body_class: 'pro-docs'
hide_header_search: true
dark_header: true
---

# Deploy Plugin API

When using `UPDATE_METHOD="auto"` or `UPDATE_METHOD="background"`, the plugin manages updates for you. If you'd like to manually control updates, the Ionic plugin has a number of functions available to manage the update lifecycle.

See the Deploy manual example for an example of using the plugin directly:

`IonicCordova.deploy.init(app_id, server_host, success, failure)`

Initializes the plugin with an app ID and API host specified in js-land.  Can be used to change these variables at runtime

`IonicCordova.deploy.check(app_id, channel_tag, success, failure)`

Check for updates from a specified channel, will change the saved channel from the install step

`IonicCordova.deploy.download(app_id, success, failure)`

If an update is present, download it

`IonicCordova.deploy.extract(app_id, success, failure)`

If an update has been downloaded, extract it and set the default redirect location for next app start.

`IonicCordova.deploy.redirect(app_id, success, failure)`

Redirect to the latest version of the app on this device.

`IonicCordova.deploy.info(app_id, success, failure)`

Get info on current version for this device.

`IonicCordova.deploy.getVersions(app_id, success, failure)`

List downloaded versions on this device.

`IonicCordova.deploy.deleteVersion(app_id, uuid, success, failure)`

Delete a downloaded version by UUID from this device.


text:
# WPF

Windows Presentation Foundation

## Introduction

* Framework for building UIs for Windows
* Works for old Windows versions (e.g. Windows 7), too
* *XAML* for defining views
  * Like *HTML* being the language for web views
  * See also [XAML Standard](https://github.com/Microsoft/xaml-standard/blob/staging/docs/faq.md)
* WPF is "done"
  * Stable
  * Backwards compatible
  * No huge investments for future enhancements

## Alternatives

* Consider [*Xamarin*](https://www.xamarin.com/) for mobile apps written in C#

  Also uses XAML
* Consider [*UWP*](https://developer.microsoft.com/en-us/windows/apps) for native Windows 10 apps written in C#

  Also uses XAML
* Consider *Cordova*-based frameworks (e.g. *Ionic*) for mobile apps written with web technology
* Consider *Progressive Web Apps* for apps that should also work in the browser

## WPF Material

* See slides and samples in [GitHub repository](https://github.com/rstropek/Samples/tree/master/WpfTraining)
* [*Cash Register* hands-on-lab](https://github.com/rstropek/htl-csharp/tree/master/wpf/9020-register)
* [*Taxi Manager* exercise](https://github.com/rstropek/htl-csharp/tree/master/wpf/9030-taxi-manager)

## Further Readings

* [WPF in Microsoft documentation](https://docs.microsoft.com/en-us/dotnet/framework/wpf/getting-started/introduction-to-wpf-in-vs)
* [Step-by-step guide for creating a WPF application](https://docs.microsoft.com/en-us/dotnet/framework/wpf/getting-started/walkthrough-my-first-wpf-desktop-application)
* [Implementing the MVVM Pattern Using the Prism Library for WPF](http://prismlibrary.github.io/docs/wpf/Implementing-MVVM.html)

gha_license_id: MIT


text:
# Wallet Package
The Wallet package contains all the things needed to create a `wallet` from a mnemonic phrase and use it
to sign a tx.

This package can also be compiled to WASM and used on the browser.

### Import wallet from mnemonic
````rust
let cosmos_dp = "m/44'/118'/0'/0/0";
let mnemonic = "battle call once stool three mammal hybrid list sign field athlete amateur cinnamon eagle shell erupt voyage hero assist maple matrix maximum able barrel";

let wallet = MnemonicWallet::new(mnemonic, cosmos_dp).unwrap();
````

### Wallet from random mnemonic
````rust
let cosmos_dp = "m/44'/118'/0'/0/0";

let (wallet, mnemonic) = MnemonicWallet::random(cosmos_dp).unwrap();
````

### Sign tx
````rust
fn sign_tx_example() {
    let cosmos_dp = "m/44'/118'/0'/0/0";

    let (wallet, mnemonic) = MnemonicWallet::random(cosmos_dp).unwrap();

    // ... Prepare the transaction

    let serialized_transaction: Vec<u8> = ...;

    let signature = wallet.sign(&serialized_transaction).unwrap();
}
````

repository: andynog/cosmos-rust-wallet

repo_name: andynog/cosmos-rust-wallet


text:
---
title: X-marks the spot
summary: The X-chromosome plays a major role in many diseases and disorders.
tags:
- X-linked disorders
date: 2019-11-12

# Optional external URL for project (replaces project detail page).
external_link: ""

image:
  caption:
  focal_point: Smart

# Slides (optional).
#   Associate this project with Markdown slides.
#   Simply enter your slide deck's filename without extension.
#   E.g. `slides = "example-slides"` references `content/slides/example-slides.md`.
#   Otherwise, set `slides = ""`.
slides: example
---

Differences in penetrance and severity of X-linked disorders stem from both genetic and transcriptional variation. How X-linked variants affect functional pathways remains a knowledge gap, along with the impact of X-inactivation on penetrance. Due to the stochasticity of the silencing - differences in timing of silencing in cell lineages, survival, selection and other unknown influences - an imbalance is typically observed. X-skewing is a driver of phenotypic variation and mosaicism although in disease contexts the breadth of this is not entirely understood. Severe mutations can often be obscured from presenting in females or appear in milder forms as the ‘unhealthy’ X may be preferentially silenced during XCI. Males, having no compensatory mechanism for such variants, exhibit disease or if embryonically lethal, fail to survive. X-skewing is implicated in atherosclerosis, cardiovascular disease, inflammatory and autoimmune diseases.

---

repository: sarbal/academic-kickstart

repo_name: sarbal/academic-kickstart


text:
# Signals

Signals are "events" that will be triggered in some circumstances and they can have one or more functions attached as receivers of the event.
These functions will be called automatically when the signal is fired.

To subscribe to any of these signals, you just need to import it and add your receiver to it. In the same way, you also can remove an receiver from an signal whenever you want, in case you don't want the event to trigger your receiver again.

```python
>>> from esipy.event import after_token_refresh
>>> after_token_refresh.add_receiver(your_receiver_function)  # from now, your_receiver_function will be triggered with this signal

...

>>> after_token_refresh.remove_receiver(your_receiver_function)  # once you do this, your_receiver_function will never be triggered, unless you add it again
```

## List of signals
####__after_token_refresh__

This signal is triggered as soon as the refresh token is automatically updated by the ESI Client (not triggered if you manually call the EsiSecurity.update() method).

Arguments | Type | Description
--- | --- | ---
access_token | String | The new access token used to log in
refresh_token | String | The refresh token used to refresh
expires_in | int | The token validity time in seconds
token_type | String | The token type returned

####__api_call_stats__

This signal is triggered after each requests done. The only purpose of this request is to monitor api calls.

__IMPORTANT:__ Please keep in mind that, if you do time consuming tasks using this event, your calls will be slowed ! Consider using async / parallel tasks here.

Arguments | Type | Description
--- | --- | ---
url | String | The url called
status_code | int | The status code of the request
elapsed_time | int | The elapsed time to get the response in seconds
message | String | JSON response string only if status_code is not 200, else None.

repository: a-tal/EsiPy

gha_language: Python

repo_name: a-tal/EsiPy


text:
---
parent: crystal.script
grand_parent: API Reference
nav_exclude: true
---

# Thread:block

Blocks the currently running thread until this thread runs to completion or stops. If the blocking thread is already dead, this function returns immediately.

## Usage

```lua
thread:block()
```

### Returns

| Name        | Type      | Description                                                                                |
| :---------- | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `completed` | `boolean` | True if the thread ran to completion, false if it was stopped prematurely.                                                                                |
| `...`       | `any`     | Return values (if any) of the thread we were waiting for. These return values are populated even if the thread had already finished before `block` was called. |

## Examples

```lua
local script = crystal.Script:new();

local my_thread = script:run_thread(function(self)
  self:wait_for("my_signal");
  return "hello", "world";
end);

script:run_thread(function(self)
  local success, hello, world = my_thread:block();
  print(success);
  print(hello);
  print(world);
end);

script:signal("my_signal"); -- Prints "true", "hello", "world"
```

repository: agersant/crystal

gha_language: Lua

repo_name: agersant/crystal


text:
# Film Buff - A Ruby wrapper for IMDb's JSON API

## Description

Film Buff provides a Ruby wrapper for IMDb's JSON API, which is the fastest and easiest way to get information from IMDb.

Film Buff supports IMDb's different locales, so information can be retrieved in different languages. See [Locales](#locales).

## Installation

### Bundler

You can install Film Buff with Bundler simply by adding it to your Gemfile

    gem 'filmbuff'

Bundler will install the gem when you use the `bundle install` command

    $ bundle install

### RubyGems

You can install the latest Film Buff gem using RubyGems

    $ gem install filmbuff

### GitHub

Alternatively you can check out the latest code directly from Github

    $ git clone https://github.com/sachse/filmbuff.git

## Usage

Film Buff 0.2.x provides two ways to return information on a movie or TV show. First, set up an IMDb instance:

    require 'filmbuff'
    imdb = FilmBuff::IMDb.new

### find_by_id

If you know the movie's IMDb ID you can return an object with the IMDb information:

    movie = imdb.find_by_id('tt0032138')

    movie.title => "The Wizard of Oz"
    movie.rating => 8.3
    movie.genres => ["Adventure", Family", "Fantasy", "Musical"]

Accessible information for an object returned by `find_by_id` is:

- Title
- Tagline
- Plot
- Runtime
- Rating
- Amount of votes
- Poster URL
- Genres
- Release date
- IMDb ID

### find_by_title

You can then search for a movie by its title. This will return an array with results from IMDb's search feature:

    results = imdb.find_by_title('The Wizard of Oz')

    results => [
                    {
                        :type=>"title_popular",
                        :imdb_id=>"tt0032138",
                        :title=>"The Wizard of Oz",
                        :release_year=>"1939"
                    },

                    {
                        :type=>"title_exact",
                        :imdb_id=>"tt0016544",
                        :title=>"The Wizard of Oz",
                        :release_year=>"1925"
                    },

                    {
                        :type=>"title_exact",
                        :imdb_id=>"tt0001463",
                        :title=>"The Wonderful Wizard of Oz",
                        :release_year=>"1910"
                    },

                    etc.
               ]

This behavior is different from Film Buff 0.1.x which returned a single object when searching by title, the same way `find_by_id` does. This was changed due to a change in IMDb's API and the fact that the new solution offers more flexibility.

`find_by_title` takes an option hash where you can set the following options

- limit
- types

`limit` limits the amount of results returned.

`types` decides the types of titles IMDb will search. Valid settings are:

- title_popular
- title_exact
- title_approx
- title_substring

These can be passed in an array inside the hash.

    results = imdb.find_by_title('The Wizard of Oz', types: %w(title_popular))

    results => [
                    {
                        :type=>"title_popular",
                        :imdb_id=>"tt0032138",
                        :title=>"The Wizard of Oz",
                        :release_year=>"1939"
                    }
                ]

### Configuration

Film Buff offers three configuration settings:

- locale
- legacy_mode
- ssl

`ssl` is used by default, but it can be disabled when setting up an instance of FilmBuff::IMDb by passing a hash with configuration options.

    imdb = FilmBuff::IMDb.new({ ssl: false })

`locale` can also be set in this a hash, but this options can also be changed as necessary during runtime by changing the instance variable value.

#### Locales

To retrieve information in a different language, either pass locale in the options hash when setting up the `imdb` instance or set the instance variable locale to your wanted locale once the instance has already been created:

    imdb.locale = 'de_DE'
    movie = imdb.find_by_id('tt0032138')

    movie.title => "Das zauberhafte Land"
    movie.rating => 8.3
    movie.genres => ["Abenteuer", "Familie", "Fantasy", "Musical"]

Supported locales are

- de_DE (German)
- en_US (English) (Default)
- es_ES (Spanish)
- fr_FR (French)
- it_IT (Italian)
- pt_PT (Portuguese)

## Authors

- [Kristoffer Sachse](https://github.com/sachse)

## Contribute

You can contribute either with code by forking the project, implementing your changes in its own branch, and sending a pull request, or you can report issues and ideas for changes [on the issues page](https://github.com/sachse/filmbuff/issues).

### Contributors
- [Jon Maddox](https://github.com/maddox) inspired the 0.1.0 rewrite through his imdb_party gem.

repository: arafatm/filmbuff

repo_name: arafatm/filmbuff


text:
#libsdnbc

This is a c library containing various data structures and modules that I use in my personal projects.

So far this library contains implementations for:

-   automatic reference counting pointer
-   dictionary
-   gap buffer

This project is not meant to be installed as a system library (yet). It's pretty small. I just include the entire project as a dependency in other projects.

###How To Build

As a shared library

clone into new directory and cd into it, then run:

__Unix__
```
./gradlew releaseSdnbcSharedLibrary
```
__Windows__
```
gradlew.bat releaseSdnbcSharedLibrary
```

The library will be output as <project root>/build/binaries/sdnbcsharedlibrary/release/libsdnbc.so

>   extension may be different depending on your system

As a static library

__Unix__
```
./gradlew releaseSdnbcStaticLibrary
```
__Windows__
```
gradlew.bat releaseSdnbcStaticLibrary
```

The library will be output as <project root>/build/binaries/sdnbcStaticLibrary/release/libsdnbc.a

>   extension may be different depending on your system

To see how to run tests or build debug variants run:

__Unix__
```
./gradlew tasks
```
__Windows__
```
gradlew.bat tasks
```

repository: mrbroll/libsdnbc

repo_name: mrbroll/libsdnbc


text:
Description
===========

Installs Tor and optionally sets up a hidden service or configures as a relay

Requirements
============
## Ohai and Chef:

* Ohai: 6.14.0+

This cookbook makes use of `node['platform_family']` to simplify platform
selection logic. This attribute was introduced in Ohai v0.6.12.

## Platform:

The following platform families are supported:

* Debian
* RHEL
* Fedora

## Cookbooks:

* apt (for Debian installation)
* yum (for RHEL 5 installation)

## Attributes

### General config section
* `node['tor']['DataDirectory']` - The directory for keeping all the keys/etc
* `node['tor']['MinLogLevel']` - The minimum log level to log. Possible values include debug, info, notice, warn, and err.
* `node['tor']['LogDestination']` - Where logs should be written. Valid values include a path to a file or "syslog"
* `node['tor']['SocksPorts']` - List of 'address:port' to open tor socks proxy on. Defaults to disabled

### Hidden Services config section

* `node['tor']['HiddenServices']` - hidden services Tor should expose
```ruby
# Example
default['tor']['hiddenServices'] = {
  'HIDDEN_SERVICE_NAME':{
  'HiddenServiceDir' => '/var/lib/tor/some_service/', #default is /var/lib/tor/HIDDEN_SERVICE_NAME/
  'HiddenServicePorts' => ['80 <IP_ADDRESS>:80'] #x y:z says to redirect requests on port x to the address y:z
}
```

### Relay config section

* `node['tor']['relay']['enabled']` - if true tor will act as a relay
* `node['tor']['relay']['ORPort']` - What port to advertise for incoming Tor connections
* `node['tor']['relay']['Address']` - The IP address or full DNS name for incoming connections to your relay
* `node['tor']['relay']['OutboundBindAddress']` - If you have multiple network interfaces, you can specify one for outgoing traffic to use
* `node['tor']['relay']['Nickname']` - A handle for your relay, so people don't have to refer to it by key
* `node['tor']['relay']['RelayBandwidthRate']` - Limit how much relayed traffic you will allow in kilobytes (not bits)
* `node['tor']['relay']['RelayBandwidthBurst']` - Limit how much relayed traffic you will allow for bursts in kilobytes (not bits)
* `node['tor']['relay']['ContactInfo']` - ContactInfo you can be reached at
  * Example: `"0xFFFFFFFF Random Person nobody AT example dot com"`
* `node['tor']['relay']['ExitPolicy']` - Sets the exit node policy for tor defaults to no exit
  * Exampe: `['accept *:6660-6667','reject *:*'] # allow irc ports but no more`
* `node['tor']['relay']['BridgeRelay']` - Set to 1 to run a bridge relay
* `node['tor']['relay']['PublishServerDescriptor']` - Set to 0 to run a private bridge relay
* `node['tor']['relay']['Directory']` - If true tor relay will server as a directory mirror
* `node['tor']['relay']['DirPort']` - "address:port" from which to mirror directory information
* `node['tor']['relay']['DirPortFrontPage']` - If true a blob of html will be returned on your DirPort explaining Tor.
  * To send a custom HTML blob specify its full path, example `"/etc/tor/tor-exit-notice.html"`
* `node['tor']['relay']['MyFamily']` - If you run more than one tor node add keyids for other tor nodes

Recipes
=======

## tor-full::default

Installs Tor and enables Tor service. By default it will not open a socks proxy, offer a hidden service,
or act as a relay.

## tor-full::relay

Installs Tor and configs Tor to be a relay. By default the relay will not be an exit or directory.
Make sure to read through the attributes section for relays above.

Usage
=====

This cookbook primarily installs Tor core packages. It can also be
used to run a Tor relay or a hidden service.

To install tor client (all supported platforms):

    include_recipe 'tor'

To install tor relay:

    include_recipe "tor::relay"

Examples
-----
### Open local socks port
The example role below opens a Tor socks proxy on port 9050 available to localhost only

```ruby
name "torproxy"
run_list("recipe[tor-full]")
override_attributes(
  "tor" => {
    "SocksPorts" => ["<IP_ADDRESS>:9050"]
  }
)
```

### Hidden service on port 80
The example role below serves a website on port 80 as a hidden service.

```ruby
name "torservice"
run_list("recipe[tor-full]")
override_attributes(
  "tor" => {
    "hiddenServices" => {
      "hidden_web_service" => {
       "HiddenServicePorts" => ["80 <IP_ADDRESS>:8080"]
       #requests on port 80 are redirected to localhost port 8080
      }
    }
  }
)
```

Note: The `tor-full` recipe will write the hidden service's hostname to the attribute `node.tor.hiddenServices.HIDDEN_SERVICE_NAME.hostname` after node convergence.

### Tor Relay
The node config below sets up a Tor relay. The relay is a directory and an exit
for IRC (ports 6660 & 6667).

```json
{
  "run_list": [
    "recipe[tor-full::relay]"
  ],
  "tor": {
    "relay": {
      "Address":"tor.icyego.com",
      "Nickname":"IcyEgo",
      "RelayBandwidthRate":"1000",
      "RelayBandwidthBurst":"1100",
      "ContactInfo":"ContactInfo 0x04FAC2E9CC21424A Richard Klafter<EMAIL_ADDRESS>      "Directory":true,
      "ExitPolicy":["accept *:6660-6667","reject *:*"]
    }
  }
}

```

Note: you can make `recipe[tor-full]` behave like `recipe[tor-full::relay]` by
setting the attribute `tor.relay.enabled = true`.

License and Author
==================

- Author:: Richard Klafter<EMAIL_ADDRESS>- License:: MIT

repository: bchrobot/tor-full

repo_name: bchrobot/tor-full


text:
Caloosahatchee River Estuary - Optical Model Evaluation
================

### Purpose

The objective of this project is to evaluate existing optical or
spectral models developed for the tidal Caloosahatchee River Estuary and
surrounding areas. Optical models are useful in Colored dissolved
organic matter (CDOM) dominated estuaries to evaluate changes in light
attenuation and potential impacts to benthic ecology.

### Study Area

The Caloosahatchee River and its estuary is located on the lower west
coast of Florida, USA (Fig 1). The historic Caloosahatchee River was
originally shallow, meandering river with its headwaters located
approximately 100 km inland at Lake Hicpochee and tidally influences
approximately 70 km upstream. To accommodate navigation, flood control
and land reclamation needs the river was straightened and extended
eastward and connected to Lake Okeechobee in the late 1800s essentially
converting the river to a canal, resulting in the C-43 canal
designation. In the early 1900s three lock-and-dam structures were
constructed to control flow and stage elevation along the river. Of the
three lock-and-dam structures installed, the final downstream structure
identified as the S-79 (Franklin Lock and Dam) regulates discharge from
the canal to the estuary. The estuary portion of the Caloosahatchee
River (hereafter Caloosahatchee River Estuary or CRE) extends
approximately 48 km from the S-79 to San Carlos Bay, where it empties
into the Gulf of Mexico and Pine Island Sound

<img src="./Plots/otherSource/JulianOsborne_Map.png" alt="The C-43 and Caloosahatchee River Estuary (CRE) with water quality; River, Estuary, and Coastal Observing Network (RECON); and flow, weather, and rainfall monitoring locations identified. Upper left inset identifies study location relative to the rest of peninsula Florida, USA. Upper right inset identifies the C-43 canal, C-43 basin, and estuary relative to Lake Okeechobee and associated water control structures. Source: Julian and Osborne (2018)" width="75%" />

The C-43 and Caloosahatchee River Estuary (CRE) with water quality;
River, Estuary, and Coastal Observing Network (RECON); and flow,
weather, and rainfall monitoring locations identified. Upper left inset
identifies study location relative to the rest of peninsula Florida,
USA. Upper right inset identifies the C-43 canal, C-43 basin, and
estuary relative to Lake Okeechobee and associated water control
structures. Source: Julian and Osborne (2018)

### Literature

Some literature with general summaries specific to this project.

-   Chen Z, Doering PH, Ashton M, Orlando BA (2015) Mixing Behavior of
    Colored Dissolved Organic Matter and Its Potential Ecological
    Implication in the Caloosahatchee River Estuary, Florida. Estuaries
    and Coasts 38:1706–1718. <https://doi.org/10.1007/s12237-014-9916-0>

    -   CDOM is one of the most important water quality constituents
        impacting light attenuation in estuaries; its concentration and
        distribution influence light quality and quantity available to
        phytoplankton and submerged aquatic vegetation.

    -   Light attenuation in the CRE was controlled primarily by CDOM in
        the upper estuary and by turbidity in the lower estuary. The
        contribution of chlorophyll a (Chl a) to light attenuation was
        less than both CDOM and turbidity.

-   Corbett CA, Hale JA (2006) Development of water quality target for
    Charlotte Harbor, Florida using seagrass light requirements. Florida
    Scientist 69:36–50
    [Link](https://www.jstor.org/stable/24322131?seq=1#metadata_info_tab_contents)

    -   Provides an optical model to set water quality targets for
        color, turbidity and chlorophyll a that maintain
        percent-light-at-depth requirements to achieve the maximum
        seagrass depth distribution presently observed in seagrass
        transect monitoring. Analysis of recently collected water
        quality data show that in all regions of the harbor, dry season
        water quality in general met the percent-light-at-depth goals
        proposed in this effort but less than half the data met the
        goals during the wet season

-   Dixon LK, Wessel MR (2016) A spectral optical model and updated
    water clarity reporting tool for Charlotte Harbor seagrasses.
    Florida Scientist 79:69–92.
    [Link](https://www.jstor.org/stable/44113166)

    -   Quantify how primary light attenuation parameters affect the
        amount and quality of light reaching seagrass target depths
        through the development of a spectrally explicit optical model,
        based on partitioned absorption and scattering, and
        parameterized as a function of color, chlorophyll, and
        turbidity.

-   McPherson BF, Miller RL (1987) The vertical attenuation of light in
    Charlotte Harbor, a shallow, subtropical estuary, south-western
    Florida. Estuarine, Coastal and Shelf Science 25:721–737.
    <https://doi.org/10.1016/0272-7714(87)90018-7>

    -   The relative contribution of different components to the
        attenuation of PAR as determined in the Charlotte Harbor
        estuarine system based on laboratory and *in-situ* measurements.

    -   Suspended Solids, DOM (color), chlorophyll and water accounted
        for as much as 99% of kt

------------------------------------------------------------------------


text:
---
layout: post
title: Python learning frustration
subtitle: how to proceed
---

  It's been over a month since I started this journey towards web development.
I've learned how to modify static templates in order to create a website like this.
Creation would be an overstatement as I actually don't know any of the languages that are used to create this template. I have learned how to use [**Markdown**,](https://markdowntutorial.com/) in order to create posts like this. Learning basic git commands was also very fun, primarily because the terminal made me feel like a true l33t coder. I also realized that people who create GUI, are angels.

My mentor was pleased with my ability to follow novice instructions. So he appointed me with a new task to learn coding with Python. He recommended that I start with [Automating Boring Stuff with Python](https://automatetheboringstuff.com/2e/chapter3/)

Although the first chapter seems to be have an easy flow going. I've been unable to retain information in the next two chapters. I've been taking notes and reading extra materials online whenever I stumbled upon a term I couldn't understand. This dramatically lengthened the learning process. Honestly, the lack of comprehensive capability on my end kind of discouraging.Perhaps because I'm expecting results much faster.I'm not sure if this book really is noob friendly considering that problems you are asked to solve aren't taught prior to that request. Hopefully, I achieve a breakthrough soon, because for now, my brain feels like its thermal throttling attempting to comprehend Python.

This is what it feels like learning Python using [Automating Boring Stuff with Python](https://automatetheboringstuff.com/2e/chapter3/)

![Comparison](https://i.imgur.com/cb206IF.jpg)


text:
## Rebase

Sometimes you have the following situation:
Your branch has changes, but master also has changes.
You don't want to merge master into your branch, but you want your branch to build on the newest commits on master.

With `rebase` you can take your branch and put it on the latest commits on master.

```
git rebase master
```

There is also an interactive mode for rebasing. With that, you can make even more things with it, e.g. rewrite the commit history.

```
git rebase HEAD~3 -i
```
This opens a vim editor, that looks something like this:

```
pick f7f3f6d changed my name a bit
pick 310154e updated README formatting and added blame
pick a5f4a0d added cat-file

# Rebase 710f0f8..a5f4a0d onto 710f0f8
# ...
```

There, you can, for example, squash multiple commits to one.

For more information on this, refer to the git documentation:
https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

If there are conflicts, you need to solve them, add the files, and do
```
git rebase --continue
```

---

[Next](15-resolving-conflicts.md)

repository: hamburgcodingschool/git-and-github

repo_name: hamburgcodingschool/git-and-github


text:
# Greedy Algorithm for the Set Cover Problem
An implementation of a greedy algorithm for the set cover optimization problem (Chvatal, V 1979)

## Greedy Algorithm
Set cover optimization problem is NP-hard.
The greedy algorithm finds a solution at most H(n) times as large as the minimum cover within O(n^3) time,
where H(n) is the n-th harmonic number and n be the size of the universe.

repository: jinnaiyuu/Set-Cover

repo_name: jinnaiyuu/Set-Cover


text:
---
layout: documentation
title: RT Communication Data Model
category: datamodel
---

### RT Communication Data Model

<!--
@startuml "RTCommunication-Data-Object-Model.png"

package Stream <<Rect>> {

	interface MediaStreamTrack : EventTarget {
    readonly    attribute DOMString             kind;
    readonly    attribute DOMString             id;
    readonly    attribute DOMString             label;
                attribute boolean               enabled;
    readonly    attribute boolean               muted;
                attribute EventHandler          onmute;
                attribute EventHandler          onunmute;
    readonly    attribute boolean               _readonly;
    readonly    attribute boolean               remote;
    readonly    attribute MediaStreamTrackState readyState;
                attribute EventHandler          onended;
	};

dictionary MediaTrackSupportedConstraints {
    boolean width;
    boolean height;
    boolean aspectRatio;
    boolean frameRate;
    boolean facingMode;
    boolean volume;
    boolean sampleRate;
    boolean sampleSize;
    boolean echoCancellation;
    boolean deviceId;
    boolean groupId;
};

	dictionary MediaTrackCapabilities {
    (long or LongRange)     width;
    (long or LongRange)     height;
    (double or DoubleRange) aspectRatio;
    (double or DoubleRange) frameRate;
    DOMString               facingMode;
    (double or DoubleRange) volume;
    (long or LongRange)     sampleRate;
    (long or LongRange)     sampleSize;
    sequence<boolean>       echoCancellation;
    DOMString               deviceId;
    DOMString               groupId;
	};

enum MediaStreamTrackState {
    "live",
    "ended"
};

enum SourceTypeEnum {
    "camera",
    "microphone"
};
	dictionary MediaTrackSettings {
    long      width;
    long      height;
    double    aspectRatio;
    double    frameRate;
    DOMString facingMode;
    double    volume;
    long      sampleRate;
    long      sampleSize;
    boolean   echoCancellation;
    DOMString deviceId;
    DOMString groupId;
	}

enum MediaDeviceKind {
    "audioinput",
    "audiooutput",
    "videoinput"
};
}

package Connection <<Rect>> {

	class	Receiver {

	}

	class	Sender {

	}

}

@enduml
-->

![Communication Data Object Model](RTCommunication-Data-Object-Model.png)

Connection Receiver and Sender classes are compliant with ORTC spec.
Sender is an Observer Object (changes made remotely) while Receiver is a Reporter object (changes made locally).

**Open Issue:** does it work with current SDP based peerconnection API?

repository: reTHINK-project/specs

gha_language: JavaScript

repo_name: reTHINK-project/specs


text:
# passport-authentiq

[Passport](http://passportjs.org/) strategy for authenticating with [AuthentiqID](https://authentiq.com/)
using the OAuth 2.0 API.

This module lets you authenticate using AuthentiqID in your Node.js applications.
By plugging into Passport, AuthentiqID authentication can be easily and
unobtrusively integrated into any application or framework that supports
[Connect](http://www.senchalabs.org/connect/)-style middleware, including
[Express](http://expressjs.com/).

## Install

```bash
$ npm install passport-authentiqid
```

## Usage

#### Create an Application

Before using `passport-authentiqid`, you must register an application with Authentiq.
If you have not already done so, a new application can be created at
[developer applications](https://dashboard.authentiq.com/).
Your application will be issued a client ID and client
secret, which need to be provided to the strategy.  You will also need to
configure a callback URL which matches the route in your application.

#### Configure Strategy

The AuthentiqID authentication strategy authenticates users that use the AuthentiqID mobile application.
The client ID and secret obtained when creating an application are supplied as options when creating the strategy.
The strategy also requires a `verify` callback, which receives the access token.
The `verify` callback must call `cb` providing a user to complete authentication.

```js
var AuthentiqIDStrategy = require('passport-authentiqid').Strategy;

passport.use(new AuthentiqIDStrategy({
    clientID: AUTHENTIQ_CLIENT_ID,
    clientSecret: AUTHENTIQ_CLIENT_SECRET,
    callbackURL: "http://<IP_ADDRESS>:3000/auth/authentiq/callback"
  },
  function(accessToken, refreshToken, profile, cb) {
    User.findOrCreate({ authentiqId: profile.id }, function (err, user) {
      return cb(err, user);
    });
  }
));
```

#### Authenticate Requests

Use `passport.authenticate()`, specifying the `'authentiqid'` strategy, to
authenticate requests.

For example, as route middleware in an [Express](http://expressjs.com/)
application:

```js
app.get('/auth/authentiq',
  passport.authenticate('authentiq'));

app.get('/auth/authentiq/callback',
  passport.authenticate('authentiq', { failureRedirect: '/login' }),
  function(req, res) {
    // Successful authentication, redirect home.
    res.redirect('/');
  });
```

## Examples

## Contributing

#### Tests

The test suite is located in the `test/` directory.  All new features are
expected to have corresponding test cases.  Ensure that the complete test suite
passes by executing:

```bash
$ make test
```

#### Coverage

```bash
$ make test-cov
$ make view-cov
```

## License

[The MIT License](http://opensource.org/licenses/MIT)

Copyright (c) 2017 Authentiq

repository: alexkeramidas/passport-authentiq

repo_name: alexkeramidas/passport-authentiq