text:
---
title: "Data Pre Processing for Machine Learning with Python"
date: 2019-11-02
tags: [data cleaning, missing values, feature encoding, ml]
---

This article is the first of a series dedicated to the machine learning workflow using a dataset retrieved from a Kaggle competition entitled **House Prices: Advanced Regression Techniques** and available [here](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/).

The dataset contains information from the Ames Assessor’s Office on residential property sales that occurred in Ames, Iowa from 2006 to 2010.
{: style="text-align: justify;"}

The aim is to build a model to predict the price of a home based on 81 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa.

The code is written in Python and it is available [here](https://github.com/DataStream101/Machine-Learning/blob/master/Machine%20Learning%20Workflow/Data%20Pre%20Processing%20for%20Machine%20Learning%20with%20Python.ipynb).

{: style="text-align: justify;"}

## The Machine Learning Worflow
{: style="text-align: center;"}

Any machine learning project should follow those steps (often in an iterative way):

{: style="text-align: justify;"}

* Step 1  –  State the question and determine required data

* Step 2  –  Acquire the data in an accessible format

* Step 3  –  Loading the required libraries and modules

* Step 4  –  Loading the data

* Step 5  –  Exploratory Data Analysis (EDA)

* Step 6  –  Data Cleaning and Data Pre-Processing

* Step 7  –  Split data into training and test set

* Step 8  –  Build and training the model

* Step 9  –  Test the model: making Predictions

* Step 10 –  Evaluate the model

{: style="text-align: justify;"}

In this part we will focus on the tasks required to prepare the data for analysis and any further operation (i.e. we will cover steps from 1 to 7).
{: style="text-align: justify;"}

Since the research question has been already stated in the introduction, let's start from the second step.
{: style="text-align: justify;"}

## Loading the required libraries and modules
{: style="text-align: center;"}

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```
The *Numpy* package is for performing Mathematical operations, the *Pandas* package is for manipulating dataset, *pyplot* and *seaborn* for data visualization.
Throughout the project we'll use different packages from *sklearn* (a Python library dedicated to machine learning).

{: style="text-align: justify;"}

## Loading the data
{: style="text-align: center;"}
From the source, data have already been split into train and test set. Therefore, at this stage, we load only the train set.
{: style="text-align: justify;"}

```python
df = pd.read_csv("train.csv")
df.sample(3)
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img1.jpeg)

The dataframe *.shape* attribute allows to view the number of samples and features we’re dealing with.
{: style="text-align: justify;"}

```python
df.shape
```
Our dataset contains 1460 observation and 81 explanatory variables.
{: style="text-align: justify;"}

## Exploratory Data Analysis
{: style="text-align: center;"}

Exploratory data analysis is a crucial step in any data workflow. The key purpose of EDA is to examine the dataset in order to:
{: style="text-align: justify;"}
•   map out the underlying structure of that data and identify potential patterns

•     generate hypotheses for further analysis

•     identify missing values

•     highlight outliers

{: style="text-align: justify;"}

Different techniques can be used to perform Exploratory Data Analysis:

{: style="text-align: justify;"}
 - **Univariate analysis and visualization**, which provides summary statistics for each feature in the dataset

 - **Bivariate analysis and visualization**, which allows to explore the relationship between each variable in the dataset and the target variable of interest

 - **Multivariate analysis and visualization**, which allows to understand interactions between different features in the dataset

 - **Dimensionality reduction**,  which helps to understand which features account for the most variance between observations and to feature selection

{: style="text-align: justify;"}

### Becoming familiar with the features
It's important to understand what feature each column represents, in order to avoid to make mistakes in the data analysis and the modeling process.

{: style="text-align: justify;"}

The dataset comes with a data dictionary containing a description of each feature along with the list of allowed values.

{: style="text-align: justify;"}

We can create a function to display its contents as follows

{: style="text-align: justify;"}

```python
def read_description(file_name):
    f = open(file_name, "r")

    lines = f.readlines()

    for i in lines:
        print(i)
```
We can also take advantage of the pandas *.info()* method which displays for each column:

{: style="text-align: justify;"}

•	  its name

•	  the number of non-null observation

•	  the data type.

{: style="text-align: justify;"}

An overview of the output is displayed below

{: style="text-align: justify;"}

![image-center]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img2.jpeg){: .align-center}

As we can see from the output, the dataset contains a miscellany of float, integer and object data types as well as several columns having missing values (we'll come back to this issue later).

{: style="text-align: justify;"}

In order to understand which feature each column actually represents (and if the current data type is suitable to describe it) we have to refer to the data dictionary by calling the function *read_description()* created above.

{: style="text-align: justify;"}

```python
read_description("data_description.txt")
```
The output looks like this
{: style="text-align: justify;"}

![image-center]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/imgx.jpeg){: .align-center}

Based on this information, we can split up the features into 4 different sets according to their nature.

{: style="text-align: justify;"}

+ Continuous features (such as SalePrice, LotFrontage, GarageArea, etc)

+ Discrete features (such as BsmtFullBath, BsmtFullBathAbvGrd, Fireplaces, etc)

+ Nominal Categorical features (such as MSSubClass, Neighborhood, Heating, etc)

+ Ordinal Categorical features (such as OverallQual, HeatingQC, YearBuilt)

{: style="text-align: justify;"}

We consider nominal and ordinal features separately for make easier the encoding step at the pre-processing stage.

{: style="text-align: justify;"}

The MSSubClass variable, which identifies the type of dwelling involved in the sale, is coded as an integer but it is actually a categorical variable. For that, we recode its as a cateogorical variable.

{: style="text-align: justify;"}

```python
df['MSSubClass'] = df['MSSubClass'].astype('category')
```

We remove the 'Id' column and reorganize the other according to the following order:

{: style="text-align: justify;"}

-	first continuous and discrete features

-	then categorical and ordinal features

{: style="text-align: justify;"}

For that, we start creating a list of features for each type of variable. Then, we concatenate these lists in one single list called *new_columns_order* and finally we pass this list as argument to the dataframe *.reindex()* method.

{: style="text-align: justify;"}

```python
continuous_features = ['SalePrice','LotFrontage', 'LotArea',\
'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2','BsmtUnfSF',\
'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',\
'GrLivArea', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF',\
'EnclosedPorch','3SsnPorch', 'ScreenPorch','PoolArea',\
'MiscVal']
discrete_features = ['BsmtFullBath', 'BsmtHalfBath',\
'FullBath', 'HalfBath', 'BedroomAbvGr','KitchenAbvGr',\
'TotRmsAbvGrd','Fireplaces', 'GarageCars']
nominal_features = ['MSSubClass', 'MSZoning', 'Street',\
'Alley','LotShape', 'LandContour', 'Utilities','LotConfig',\
'LandSlope', 'Neighborhood','Condition1', 'Condition2',\
'BldgType','HouseStyle','RoofStyle', 'RoofMatl','Exterior1st',\
'Exterior2nd', 'MasVnrType','Foundation', 'Heating', 'CentralAir',\
'Electrical', 'SaleType','SaleCondition','Functional', 'GarageType',\
'GarageFinish', 'PavedDrive','MiscFeature']
ordinal_features = ['OverallQual','OverallCond','ExterQual','ExterCond',\
'BsmtQual', 'BsmtCond','BsmtExposure', 'BsmtFinType1', 'BsmtFinType2',\
'HeatingQC','KitchenQual', 'FireplacesQu', 'GarageQual', 'GarageCond',\
'PoolQC', 'Fence', 'YearBuilt', 'YearRemodAdd', 'GarageYrBlt', 'MoSold',\
'YrSold' ]

new_columns_order = continuous_features + discrete_features +\
					nominal_features +  ordinal_features

df = df.reindex(columns=new_columns_order)
```

### Univariate Exploratory Analysis on numeric features

{: style="text-align: center;"}
After understanding what kind of information are available, we have to get an idea of how these features are distributed inside our dataset.

{: style="text-align: justify;"}

We start from the continuous features which relate measurements of area dimensions for each observation. For each of them we compute some summary statistics and we plot their distribution.

{: style="text-align: justify;"}

```python
df[continuous_features].describe()

df[continuous_features].hist(figsize = (30,30), bins = 15);
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/continDistrib.jpeg)

Any of our continuous features follows a normal distribution, except for the TotalBsmtSF variable which is quite close to a gaussian.
 Some of them are right skewed like 1stFlrSF, 2ndFlrSF. Also the 2ndFlrSF feature for most of the observation takes value zero, meaning that many properties in our sample have no a second floor.

{: style="text-align: justify;"}
Others have most of the observations concentrated at the lower end of the distributions and few at the other end, resulting in a distribution with a lot of empty space (like 3SsnPorch, ScreenPorch or PoolArea). This may be due to the existence of outliers.

{: style="text-align: justify;"}

The first impression about our sample is that most of the properties sold are one-floor houses, with a prevalence of open porch and wood deck solutions to extend outdoor living areas, and a prevalence of unfinished over finished basements area. Also, 50% of properties has a living area above ground between 1130 and 1777 square feets and a garage area between 335 and 576 square feets. Very few of them have additional features such as pool or other facilities like elevator, a second garage etc.

{: .notice--info}
{: style="text-align: justify;"}

Let’s take a closer look to our target variable 'SalePrice'

```python
plt.hist(df['SalePrice'], bins = 40);
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/target.jpeg)

It's clear that SalePrice doesn't follow normal distribution, so before performing regression it has to be transformed.

{: style="text-align: justify;"}

```python
df['SalePrice'] = np.log10(df['SalePrice'])
```

The resulting distribution is displayed below
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/logNorm.png)
{: style="text-align: cente;"}

Next, we consider discrete features.

```python
df[discrete_features].describe()
df[discrete_features].hist(figsize = (30,30), bins = 30);

```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/discreteDistr.jpeg)

The discrete variables mostly have to do with the number and kind of rooms, as well as the number of fireplaces or the size the garage, in cars capacity, that a given property has.

{: style="text-align: justify;"}

The majority of the properties sold have between 5 and 7 rooms above the ground, with 2 or 3 bedrooms and 1 or 2 full bathrooms (and any half or full basement baths). The median size of garages is of 2 cars capacity and more than 50% of the properties has a fireplace.

{: style="text-align: justify;"}

### Univariate Exploratory Analysis on categorical features

{: style="text-align: center;"}

We define a function *bar_plot()* to plot multiple barplot at once. Then, we just need to call this function and pass the different subset of our feature we want to visualize.

{: style="text-align: justify;"}

```python
def bar_plot (df, features, rows, cols, hfig, wfig):
    fig = plt.figure(figsize = (hfig,wfig))
    for i, feature_name in enumerate(features):
        ax = fig.add_subplot(rows, cols, i +1)
        sns.countplot(x = feature_name, data = df)
        plt.xticks(rotation = 90);
    fig.tight_layout()
```
Starting from **Nominal variables**, we can split them up into different groups, based on property’s features they represent:

{: style="text-align: justify;"}
 •   **A first group relate property location**. Variables belonging to this group are ‘MSZoning', which identifies the general zoning classification of the sale (such as commercial, residential, etc), 'Neighborhood' (i.e. the physical location of a property within Ames city limits), 'Condition1' and 'Condition2' (which describes if the property is located near to various conditions such as arterial street, railroad, park, etc.).

 {: style="text-align: justify;"}

We store those variables in a list and we pass that list to the *bar_plot* function.

{: style="text-align: justify;"}
```python
location = ['MSZoning','Condition1', 'Condition2', 'Neighborhood']
bar_plot (df, location, int(len(location)/2)+1, 2, 12, 10)
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/nominalLocation.jpeg)

Considering this first group, we see that 94% of the properties in our sample are located in a residential (medium and lower density) zone and 86% of properties have a normal proximity to various city conditions. The 'Neighborhood' variable will be explored later with reference to the SalePrice.

{: style="text-align: justify;"}

•    **The second group of the nominal variables identifies**:
 - **structural characteristics of the property**, including lot configuration, general shape and flatness of the property, type of access to the property (street, alley and whether they are paved), the garage location as well as if its interior is finished.

 - **structural characteristics of the dwelling**, such as the type of building, the house style, the type of foundation, which is the exterior siding, which is the type of the roof and the material is made of, the masonry veneer type (if any).

 - **property’s utilities**, such as which kind of public utilities are available, the type of heating and electrical system, if there is central air conditioning and other miscellaneous features.

 {: style="text-align: justify;"}

We store those variables in 3 different lists and we pass each of them to the *bar_plot* function.

{: style="text-align: justify;"}

```python
strucCaractProp = ['LotShape','LandContour', 'LandSlope', 'LotConfig',\
                         'Street', 'Alley', 'PavedDrive', 'GarageType', 'GarageFinish']

strucCaractDwel = ['MSSubClass','BldgType','HouseStyle','Foundation','RoofStyle','RoofMatl',\
                         'MasVnrType', 'Exterior1st', 'Exterior2nd']

utilities = ['Utilities','Heating', 'CentralAir', 'Electrical', 'MiscFeature','Functional']
```
```python
bar_plot (df, strucCaractProp, int(len(strucCaractProp)/2)+1, 2, 12, 18)
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/strucCaractProp.jpeg)

Looking at the lot shape we can see that 64% of properties were a regular shape whereas 33% were slightly irregular. We may consider to regroup the other two categories of the variable lot of shape into a new category “IR+”.

{: style="text-align: justify;"}

```python
df.loc[(df.LotShape == 'IR2'),'LotShape'] ='IR+'
df.loc[(df.LotShape == 'IR3'),'LotShape'] ='IR+'
```

Also, most of the properties are located on a flat land and the most common type of lot configuration is the interior lot (i.e. a property which lies between homes on the left and right side, facing the street on only the frontal side). When considering the variables related to the garage we can see that the majority of the properties has an attached garage with an unfinished interior.

{: style="text-align: justify;"}

```python
bar_plot (df, strucCaractDwel, int(len(strucCaractDwel)/2)+1, 2, 12, 20);
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/strucCaractDwel.jpeg)

The HouseStyle and MSSubClass variables provide pretty much the same information: most of the properties sold were one or two-level single-family detached dwelling.

{: style="text-align: justify;"}
Most of the properties had foundation made of poured Contrete or cinder Block.

{: style="text-align: justify;"}

Since 98% of roof material is Standard (Composite) Shingle we may regroup the other seven categories to into a NotCompShg category.

{: style="text-align: justify;"}

```python
df.loc[df['RoofMatl'] != 'CompShg', 'RoofMatl'] = 'NotCompShg'
```

```python
bar_plot (df, utilities, int(len(utilities)/2)+1, 2, 9, 10);
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/utilities.jpeg)

Since 99% of the observations in our sample have all public utilities available, the feature ‘Utilities’ is not informative and we can drop it.

{: style="text-align: justify;"}

```python
df.drop('Utilities', axis = 1, inplace = True)
```

Since 98% of heating types are either GasA (Gas forced warm air furnace) or GasW (Gas hot water or steam heat) we keep those categories and group the other into an ‘other’ class.
{: style="text-align: justify;"}

```python
other = {'Grav': 'Other', 'Wall': 'Other', 'OthW': 'Other', 'Floor': 'Other'}
df['Heating'].replace(other, inplace = True)
```

•    **The third group relate sale characteristics**, such as the type of sale and the sale conditions

{: style="text-align: justify;"}

```python
sale = ['SaleType','SaleCondition']

bar_plot (df,sale,2, 2, 12, 7)
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/sale.jpeg)

In our sample, 87% of properties were purchased from a former owner through a warranty deed whereas 8% were new homes just constructed and sold.
{: style="text-align: justify;"}
We regroup the other categories into a single category.

{: style="text-align: justify;"}

```python
otherType = {'COD': 'Other', 'ConLD': 'Other', 'ConLw': 'Other', 'ConLI': 'Other', 'CWD': 'Other',\
             'Oth': 'Other', 'Con': 'Other'}
df['SaleType'].replace(otherType, inplace = True)
```

When considering the condition of sale, we see that 82% of properties were sold through a normal sale, it was a partial sale for 9% of the properties (this is may be related to the new homes) and 7% are sold under abnormal sale conditions (e.g. foreclosure).
{: style="text-align: justify;"}

After checking the median sale price across the different categories as follows

{: style="text-align: justify;"}

```python
df.groupby('SaleCondition')['SalePrice'].median()
```
we regroup the remaining categories into a the ‘normal’ category.

{: style="text-align: justify;"}

```python
otherCond = {'AdjLand':'Normal', 'Alloca':'Normal', 'Family':'Normal'}
df['SaleCondition'].replace(otherCond, inplace = True)
```

**Ordinal variables**

Ordinal variables are ratings of the quality and condition of the properties as a whole and of single rooms or utilities. In this category were also considered the year in which the property was built or remodelled and the month and year when the sale was made.

{: style="text-align: justify;"}

We start by examining the month of sale, to figure out if there is a seasonality in sales.

```python
sns.countplot(df['MoSold']);
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/month.jpeg)

65.35% of the properties were sold between april and august.

{: style="text-align: justify;"}

A useful information can be provide by the age of the house at the time it was sold, that can be calculated as follows

{: style="text-align: justify;"}

```python
df['HouseAge'] = df['YrSold'] - df['YearBuilt']

plt.figure(figsize = (20,5))
sns.countplot(df['HouseAge'])
plt.xticks(rotation = 90);
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/houseAge.jpeg)
Most of the properties were sold one year after building.

{: style="text-align: justify;"}

We can also visualize the variability of the median price of sale based on the house age.

```python
df.groupby("HouseAge")['SalePrice'].median().plot(kind = 'bar', figsize = (20,8))
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/SalePriceHouseAge.jpeg)

### Bivariate Exploratory Analysis on numeric features

{: style="text-align: center;"}

The next step is to understand which features are the most correlated with the target variable (in order to find out which feature can be use as predictors in our model) and which are correlated each other (in order to prevent multicollinearity).

{: style="text-align: justify;"}

For that we build a function to visualize how numeric features are correlated to the target variable through a scatter plot. We use the *.regplot(*) method in seaborn.

{: style="text-align: justify;"}

```python
def scatters (df, features, rows, cols):
    fig = plt.figure(figsize = (10,30))
    for i, feature_name in enumerate(features):
        ax = fig.add_subplot(rows, cols, i +1)
        sns.regplot(x = feature_name, y= 'SalePrice', data = df);
    fig.tight_layout()
```
We analyse continuous and discrete numeric features separately.

{: style="text-align: justify;"}

```python
scatters(df, continuous_features[1:], int(len(continuous_features[1:])/2)+1,2)
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/correlPlots.jpeg)

From the plots above, it seems that the target variable is linearly correlated with the GrLivArea, 1stFlrSF and GarageArea features. However, they don't provide information about the strenght of these correlations.

{: style="text-align: justify;"}

For that we can plot an *heatmap*, available in the seaborn package.

{: style="text-align: justify;"}

```python
corr_continuous = round(df[continuous_features].corr(),2)
plt.figure(figsize= (15, 15))
mask = np.zeros_like(corr_continuous, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True;
sns.heatmap(corr_continuous, mask=mask, annot = True, linewidths=.5);
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/corrMatrix.jpeg)

The correlation matrix confirms the scatterplots' results: the continuous variables most correlated to our target variable (in order of strenght of correlation) are GrLivArea, GarageArea, 1stFlrSF, TotalBsmtSF.

However, GrLivArea and 1stFlrSF are also correlated each other, like 1stFlrSF and TotalBsmtSF.

{: style="text-align: justify;"}

```python
scatters(df, discrete_features, int(len(discrete_features)/2)+1,2);
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/corrDiscrete.jpeg)

From the plot above, it seems that the target variable is linearly correlated withFullBath, TotRmsAbcGrd, Fireplaces and GarageCars featues.

{: style="text-align: justify;"}

```python
corr_discrete = round(df[['BsmtFullBath','BsmtHalfBath','FullBath','HalfBath','BedroomAbvGr',\
                          'KitchenAbvGr','TotRmsAbvGrd','Fireplaces','GarageCars','SalePrice']].corr(),2)
plt.figure(figsize= (8, 8))
mask = np.zeros_like(corr_discrete, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True;
sns.heatmap(corr_discrete, mask=mask, annot = True, linewidths=.5);
```
![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/corrMatrDisc.jpeg)

The correlation matrix confirms the initial thoughts. However, FullBath and TotRmsAbcGrd are also correlated each other.

The variable with the stronger correlation is GarageCars (for the other the correlation is weaker).

{: style="text-align: justify;"}

### Visualizing relationships between categorical features and target variable

{: style="text-align: center;"}

The same way, we build a function to visualize the relationship between categorical features and to the target variable through boxplots as follows

{: style="text-align: justify;"}

```python
def boxplots (df, features, rows, cols):
    fig = plt.figure(figsize = (15,70))
    for i, feature_name in enumerate(features):
        ax = fig.add_subplot(rows, cols, i +1)
        sns.boxplot(x = feature_name, y= 'SalePrice', data = df)
        plt.xticks(rotation = 90);
    fig.tight_layout()
```

We start plotting the target variable's distribution across nominal features

{: style="text-align: justify;"}

```python
boxplots(df, nominal_features, int(len(nominal_features)/2)+1 ,2)
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/boxplotNom.jpeg)

Moving to ordinal features

{: style="text-align: justify;"}

```python
ordinal_features = ['OverallQual','OverallCond','ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond',\
 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'GarageQual', 'GarageCond',\
 'Fence', 'YearBuilt', 'YearRemodAdd', 'MoSold', 'YrSold']

boxplots(df, ordinal_features, int(len(ordinal_features)/2)+1 ,2)
```

The categorical features for which the SalePrice shows the highest variability are:
{: style="text-align: justify;"}
-   *OverallQuality*

-   *MSZoning*

Properties sold in Floating Village Residential (FV) zone have the highest median SalePrice compared to other zones. However, for properties sold in the Residential Low Density (RL) zone the SalePrice distribution is right skewed with many observations having high SalePrice.
{: style="text-align: justify;"}
-   *Neighborhood*

When considering Neighborhood, we see that the median SalePrice is highly variable depending on the neighbourhood. Also, Northridge and Northridge Heights are those with the highest median SalePrice followed by the Stone Brook neighbourhood.
{: style="text-align: justify;"}
-   *Condition1 and Condition2*

Proximity to off-site feature like park, greenbelt, etc. as well as to railroad leads to a higher median SalePrice.
{: style="text-align: justify;"}
-   *BldgType*

When considering the type of building, the Single-family Detached and the Townhouse End Unit dwelling are those with the highest median SalePrice.
{: style="text-align: justify;"}
-   *HouseStyle*

Two-level properties as well as split-level house have the highest median SalePrice
{: style="text-align: justify;"}
-   *HouseAge*

It is one of the features determining the highest variability in the SalePrice
{: style="text-align: justify;"}
-   *Exterior1st and ExternalQual*

Among the different materials for external siding, Imitation stucco, stone and cement board are those which are related to the highest median SalePrice. This may be due to the fact that they ensure high durability.
{: style="text-align: justify;"}
-   *GarageType*

Built-In garages (which are part of the house) have the highest median SalePrice. This may be due to the fact that they typically have a room above them, allowing to increase the total living area of a property.
{: style="text-align: justify;"}
-   *GarageFinish*

-   *Heating and HeatingQC*

Based on type of heating, the GasA type has the highest median SalePrice. Also the quality of the heating system has a high impact on SalePrice
{: style="text-align: justify;"}
-   *CentralAir*

Also, the presence of central air conditioning seems to be an important feature in determing SalePrice level.
{: style="text-align: justify;"}
-   *Electrical*

When considering the Electrical system, Standard Circuit Breakers & Romex is what ensure the highest SalePrice
{: style="text-align: justify;"}
-   *MiscFeature*

Among the Miscellaneous Features, the availability of a tennis court seems to have the highest impact on the median selling price, followed by the availability of a second garage.
{: style="text-align: justify;"}
-   *SaleType*

New home are sold at a highest price than others.
{: style="text-align: justify;"}
-   *SaleCond*

Properties sold under partial condition (Home not completed yet) have the highest median SalePrice (it is related to the fact that are new properties).
{: style="text-align: justify;"}

# Data Cleaning and Pre-Processing.

{: style="text-align: justify;"}
It's now time to preparing the data for analysis. It mainly means

{: style="text-align: justify;"}

•	  Deal with missing data

•     Deal with outliers

•	  Deal with categorical data

•	  Scaling the features

## Dealing with missing data
{: style="text-align: center;"}

We can check how many missing value a column has by concatenatig the *.isnull()* and the *.sum()* methods. We perform this operation on the 4 subsets of features by using indexing selection on the dataframe.

{: style="text-align: justify;"}

```python
df[continuous_features].isnull().sum()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img3.jpeg)

```python
df[discrete_features].isnull().sum()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img4.jpeg)

```python
df[nominal_features].isnull().sum()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img5.jpeg)

```python
df[ordinal_features].isnull().sum()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img6.jpeg)

Let's examine one feature having missing values at a time.

{: style="text-align: justify;"}

•	We know, from the data dictionary that **GarageType**,  **Fence**, **MiscFeature** and **Alley** variables have 'nan' values when the property lacks those features.

{: style="text-align: justify;"}

We can create an additional category *"No feature"* and encoding those missing values to the new category.

{: style="text-align: justify;"}

```python
df['GarageType'].fillna("No Garage", inplace = True)
df['Fence'].fillna("No Fence", inplace = True)
df['MiscFeature'].fillna("No MiscFeature", inplace = True)
df['Alley'].fillna("No Alley", inplace = True)
```
Missing values for other categorical/ordinal variables related to GarageType such as **GarageQual**, **GarageCond** and **GarageFinish** can be handled in the same way.

{: style="text-align: justify;"}

However, for the **GarageYrBlt** variable, this approach is not suitable and since no valid value can be inputated (since it is not logical to have a year of built for something never been built) we may consider to drop those feature.

{: style="text-align: justify;"}

```python
df.drop('GarageYrBlt', axis=1, inplace = True)
```
•	As for the previous variables, **BsmtQual**, **BsmtCond**, **BsmtExposure**, **BsmtFinType1**, **BsmtFinType2** have missing values when a property has no basement. Therefore, we can use the same encoding and create an additional category *"None"*.

{: style="text-align: justify;"}

However, one observation (*id 948*) has a missing value for the **BsmtExposure** variable and another one (*id 332*) has a missing value for the BsmtFinType2 variable even if the value of the **TotalBsmtSF** variable (Total square feet of basement area) for those observations is different from zero (meaning that they actually have a basement).

{: style="text-align: justify;"}

We can check which values other basement related variables take for those observations and which is the most common value for the variables having missing values for other observations having the same values of related variables.

{: style="text-align: justify;"}
Starting with the observation having Id 948

```python
df[df.index == 948][["BsmtExposure","BsmtCond","BsmtFinSF1",\
"BsmtFinType1","BsmtFinSF2", "BsmtFinType2", "BsmtUnfSF",\
"TotalBsmtSF"]]
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img7.jpeg)

```python
df[(df["BsmtCond"] == 'TA')& (df['BsmtFinType1'] == "Unf")&\
	(df["BsmtFinType2"] == 'Unf')]\
	['BsmtExposure'].value_counts()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img8.jpeg)

We can assume that the Id 948 has an 'Normal' basement exposure and impute this value.

{: style="text-align: justify;"}
```python
df.at[948, "BsmtExposure"] = "No"
```
Moving to the observation having Id 332

```python
df[df.index == 332][["BsmtExposure","BsmtCond",\
					"BsmtFinSF1","BsmtFinType1",\
                     "BsmtFinSF2", "BsmtFinType2",\
                     "BsmtUnfSF", "TotalBsmtSF"]]
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img9.jpeg)

```python
df[(df["BsmtExposure"] == 'No')&(df["BsmtCond"] == 'TA')\
	& (df['BsmtFinType1'] == "GLQ")]['BsmtFinType2'].value_counts()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img10.jpeg)

We can assume that the Id 332 has an Unfinished basement and impute this value.

```python
df.at[332, "BsmtFinType2"] = "Unf"
```
Finally, we encoding the missing values for the other basement variables to the  *"None"* category.
{: style="text-align: justify;"}
```python
df["BsmtQual"].fillna("None",inplace = True)
df["BsmtCond"].fillna("None",inplace = True)
df["BsmtExposure"].fillna("None",inplace = True)
df["BsmtFinType1"].fillna("None",inplace = True)
df["BsmtFinType2"].fillna("None",inplace = True)
```
•	For the variable **Electrical**, which has only one missing value, we can impute a value using deduction. We can check the year this property has been built and which kind of electrical systems the other properties built the same year have.

{: style="text-align: justify;"}

```python
df[df['Electrical'].isna()]['YearBuilt']
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img11.jpeg)

```python
df[df['YearBuilt'] == 2006]['Electrical'].value_counts()
```

![alt]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img12.jpeg)

Since all those properties have a Standard Circuit Breakers & Romex, we can assume that our record missing this value has the same kind of electrical circuit.

{: style="text-align: justify;"}
```python
df['Electrical'].fillna("SBrkr", inplace = True)
```

•	For **MasVnrType** column we impute **"No MasVnr"** and for the related MasVnrArea we impute 0.0

{: style="text-align: justify;"}

```python
df['MasVnrType'].fillna("No MasVnr", inplace = True)
```
```python
df['MasVnrArea'].fillna(0.0, inplace = True)
```

•	For the **LotFrontage** variable, we can impute missing values using the pandas *.interpolate()* method. Interpolation is a mathematical method that adjusts a function to the data and uses this function to extrapolate the missing data.

In this case a good approximation may be a linear method.

{: style="text-align: justify;"}

```python
df['LotFrontage'].interpolate(method='linear', inplace = True)
```

•	For **FireplacesQu** and **PoolQC** columns, since almost all values are missing we can drop the whole column.

```python
df.drop(["FireplacesQu", "PoolQC"], axis = 1, inplace = True)
```
After these operations, the former list of ordinal features in our dataset has changed (we have removed GarageYrBlt, FireplacesQu and PoolQC variables) and we need to update it since we will use it again in the next step.

{: style="text-align: justify;"}

```python
ordinal_features = ['OverallQual','OverallCond','ExterQual',\
'ExterCond', 'BsmtQual','BsmtCond','BsmtExposure',\
'BsmtFinType1', 'BsmtFinType2','HeatingQC',\
'KitchenQual', 'GarageQual', 'GarageCond', 'Fence',\
'YearBuilt', 'YearRemodAdd', 'MoSold', 'YrSold' ]
```

## Dealing with outliers
{: style="text-align: center;"}

Outliers can impact the results of analysis and statistical modelling, by altering the relationship between variables.
However, unusual observations are not necessarily outliers; in some cases, they are legitimate observations that accurately describe the variability in the data and may contain valuable information.
{: style="text-align: justify;"}

For that, at this stage, we only remove extreme observations for the target variable and we keep all the observations (even those having unusual values) for the predictors.

For the SalePrice we drop observation which are below and above 1st and 99th percentile, i.e. observations having a log10 value below 4.79 or above 5.64.

```python
# defining a function returning for a feature the 1st and the 99th percentile
def outlier_detection(datacolumn):
    global Q1
    global Q99
    sorted(datacolumn)
    Q1,Q99 = np.percentile(datacolumn , [1,99])
    return Q1,Q99
```

```python
# Printing out the IQR's boundaries and the nb of observations having values above or below
print("SalePrice:", outlier_detection(df['SalePrice']),
          "\nNb of outliers: ", len(df[(df["SalePrice"] < Q1) | (df["SalePrice"] > Q99)]))
```

![image-center]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/print.png){: .align-center}

```python
# Removing outliers from the dataset
df.drop(df[(df.SalePrice > Q99) | (df.SalePrice < Q1) ].index , inplace=True)
```

## Dealing with categorical data
{: style="text-align: center;"}

Even if some machine learning algorithms (like tree-based models) can deal with categorical variables, many other which are algebric (like linear/logistic regression, SVM, KNN) require that their input must be numerical values.
Therefore, when the dataset contains categorical variables, we need to map categorical feature values to integer representations.

{: style="text-align: justify;"}

Before that, we need to be aware of some issues that we may cope with:

•     we may deal with a feature having **high cardinality** (feature having a very large number of levels where most of the levels appear in a relatively small number of instances). In this case, we need to use specific encoding approach.

•     we may deal with a categorical variable having **levels which rarely occur**. Many of these levels have minimal chance of making a real impact on model fit. To avoid redundant levels in a categorical variable and to deal with rare levels, we can combine the different levels. For that we can look at the frequency distribution of of each level and combine levels having similar frequency.

•     there is **one level which always occurs**, i.e. for most of the observations in data set there is only one level. Variables with such levels fail to make a positive impact on model performance due to very low variation.
{: style="text-align: justify;"}

Let's start by examining the distribution of categories for our nominal features.

{: style="text-align: justify;"}

```python
for v in nominal_features:
    print(df[v].value_counts())
```

A sample of the output is shown below.

{: style="text-align: justify;"}

![image-center]({{ site.url }}{{ site.baseurl }}/images/2019-11-02-data-pre-processing-for-machine-learning/img13.jpeg){: .align-center}

We are not dealing with features having high cardinality. Even if some features have low frequency categories, it may be worth keep them instead of grouping with other.

{: style="text-align: justify;"}

We can make the same analysis for ordinal features.

{: style="text-align: justify;"}

Since no issue arises, we can move to the encoding part, taking into account that a different approach is required for ordinal and nominal categorical features.

{: style="text-align: justify;"}

Also, it's a good practice to typecast categorical features to a **category** dtype because they make the operations much faster than the object dtype. This can do by using the *.astype()* method on each columns like shown below:

{: style="text-align: justify;"}

```python
df[nominal_features] = df[nominal_features].astype('category')
```

```python
df[ordinal_features] = df[ordinal_features].astype('category')
```

We can check the current datatype by using the *.dtypes()* method

```python
df[nominal_features].dtypes.head(3)
```

```python
df[ordinal_features].dtypes.head(3)
```

***Encoding Ordinal Features***

When encoding ordinal features (the categories of which have an intrinsec order) we need to make sure that the learning algorithm takes into account the order of the categories. For that we have to map the categorical values to integer values manually.

{: style="text-align: justify;"}

We can use a dictionary-mapping approach along with the *replace()* function.

{: style="text-align: justify;"}

Note: Some of the ordinal features ('OverallQual','OverallCond', 'YearBuilt','YearRemodAdd','MoSold' and 'YrSold') do not need encoding since they are already in a numeric format.

{: style="text-align: justify;"}

For the other we proceed as shown below
{: style="text-align: justify;"}

```python
mapping_dict = {
"ExterQual": {'Ex':4, 'Gd':3, 'TA':2, 'Fa':1},\
"ExterCond": {'Ex':4, 'Gd':3, 'TA':2, 'Fa':1},\
"BsmtQual":{'Ex':4, 'Gd':3, 'TA':2, 'Fa':1,'None':0},\
"BsmtCond":{'Gd':4, 'TA':3, 'Fa':2,'Po':1, 'None':0},\
"BsmtExposure":{'Av':4, 'Gd':3, 'Mn':2, 'No':1, 'None':0},\
"BsmtFinType1":{'GLQ':6,'ALQ':5, 'BLQ':4, 'Rec':3, 'LwQ':2,'Unf':1,'None':0},\
"BsmtFinType2":{'GLQ':6,'ALQ':5, 'BLQ':4, 'Rec':3, 'LwQ':2,'Unf':1,'None':0},\
"HeatingQC" :{'Ex':5, 'Gd':4, 'TA':3, 'Fa':2,'Po':1},\
"KitchenQual":{'Ex':4, 'Gd':3, 'TA':2, 'Fa':1},\
"GarageQual":{'Ex':5, 'Gd':4, 'TA':3, 'Fa':2,'Po':1,'None':0},\
"GarageCond":{'Ex':5, 'Gd':4, 'TA':3, 'Fa':2,'Po':1,'None':0},\
"Fence":{'GdPrv':4,'MnPrv':3, 'GdWo':2,'MnWw':1, 'No Fence':0}
}

df.replace(mapping_dict, inplace=True)
```

***Encoding Nominal Features***

To map nominal features (i.e. variables having multiple categories without any intrinsic ordering to them) we can use a technique called one-hot encoding. The idea behind this approach is to create a new dummy feature for each unique value in the nominal feature column.

{: style="text-align: justify;"}

Many libraries support one-hot encoding (e.g. sklearn), however the fastes way to do that is to use the pandas *.get_dummies()* method.

{: style="text-align: justify;"}
The most important arguments are

 >    -   the DataFrame we want to encode on

 >    -   the columns we want to do encoding on

 >    -   the *prefix* argument which lets us specify the prefix for the new columns that will be created after encoding

 >    -   the *drop_first* argument which, when set to True, leaves the first category of the feature out. This missing category becomes the the reference category for all further interpretation.

 >   This approach allows to avoid a risk of introduce into the regression redundant information, resulting in multicollinearity, and in a poor fit model.

{: style="text-align: justify;"}

```python
nominal_features_dummies = pd.get_dummies(df[nominal_features], drop_first=True)
```

After creating our dummy variables, we concatenate them to the dataframe and we remove the original columns.

{: style="text-align: justify;"}

```python
df = pd.concat([df, nominal_features_dummies], axis = 1)

df.drop(nominal_features, axis = 1, inplace = True)

df.drop(ordinal_features, axis = 1, inplace = True)
```

## Scaling Features
{: style="text-align: center;"}

This operation is needed when certain features in the dataset have a higher ‘magnitude’ than others.
{: style="text-align: justify;"}

In such cases, since most machine learning algorithms take into account only the magnitude of the measurements and not the units of those measurements, if no scaling is performed, features having high magnitude will have a much larger impact on the prediction than features having lower magnitude, regardless their actual relative importance.
{: style="text-align: justify;"}

Scaling features allows for features to contribute relative to their importance rather than their scale, by bringing all of them to an equivalent range of values.
{: .notice--info}
{: style="text-align: justify;"}

Different approaches can be used to scale the features. The most commonly used and implemented in the Python ML package scikit-learn are :
{: style="text-align: justify;"}
+ StandardScaler
+ MinMaxScaler
+ RobustScaler
+ Normalizer
{: style="text-align: justify;"}

We use the *RobustScaler* which is robust to outliers.

Before scaling features, we need to split data into target (the value we want to predict, in this case the SalePrice) and features (all the columns the model uses to make a prediction). We have also to convert the dataframes to Numpy arrays in order to implement the model.
{: style="text-align: justify;"}

```python
y_train = np.array(df['SalePrice'])
X_train = np.array(df.iloc[:,2:])
```
After that, we import the sklearn's module for preprocessing, we create an object of the class 'RobustScaler' and we pass to it our train set as follows
{: style="text-align: justify;"}

```python
from sklearn import preprocessing
scaler = preprocessing.RobustScaler()
X_train_scaled = scaler.fit_transform(X_train)
```

The next article, we'll be focused on building a model to make prediction about home's prices.
{: style="text-align: justify;"}


text:
Based on [j-greig/gulptwigtachyons](https://github.com/j-greig/gulptwigtachyons).

---

This repo is basically just a ripped version of the same concept.

# What's happening?

Twig takes care of the templates, CSS is knocked out of the park with Tachyons, and there's a tiny Express server with LiveReload so it just updates without you having to hit refresh all the time. That's the stuff.

## What do I need to make it work?

- [Node and npm](https://nodejs.org/en/)
- [Gulp](https://gulpjs.com/)

If you didn't already run into it, you'll figure out soon enough that Gulp needs to be installed globally. No, I don't know why either, but here's how:

```
npm install -g gulp
```

## Now, how do I make it work?

Clone this repo:

```
git clone https://github.com/marksdk/gulptwigtachyons.git
```

Install the dependencies:

```
cd gulptwigtachyons
npm install
```

Run the default Gulp command:

```
gulp
```

Now open your browser of choice at `http://localhost:3000` and marvel at the wonder that is something that actually works on your computer.

## Now what?

Make sure you edit the .twig files in `/src`, not the .html files in `/web`. If you do that, it will treat you well, and as long as you have Gulp running, all of your changes in the Twig templates will automatically be compiled to HTML. 🎩 Magic ✨

Place reusable elements in the `_partials` folder, and include them in the other Twig templates. (See the `<nav>` element in `_layout.twig` for an example).

## License

G/T/T is licensed under the MIT License, luckily.

license: MIT

repository: markhougaard/gulptwigtachyons

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: markhougaard/gulptwigtachyons


text:
# vectorforth
SIMD vectorized Forth compiler with CPU based shader application

Vectorforth is a forth compiler that creates executable code, but every instruction is vectorized. For instance, if you compile `1 2 +`, then the result will obviously be `3`, but actually you will get 16 times the result `3` (using AVX-512), as each instruction is vectorized. The main toy application that I am interested in here is to see how fast your CPU can compute pixel shaders a la [ShaderToy](https://www.shadertoy.com). So we make use as much as possible of SIMD (i.e. vectorized) instructions and multithreading. Of course the CPU will not beat the GPU but it was fun to test how large the gap would be.

As the compiler generates machine instructions, this will only work for an Intel architecture, and not for ARM architectures.

## Building

Vectorforth has been tested on Windows 10 using Visual Studio 2017/2019, on Ubuntu 18.04.4 with gcc 7.5.0, and on MacOS 10.15.6 with XCode 11.7.
You best use CMake to generate a solution file or makefile or XCode project.

Vectorforth uses two CMake variables: VECTORFORTH_AVX512 and VECTORFORTH_SINCOS_METHOD.

  - If your computer can handle it, switch on VECTORFORTH_AVX512, as it will perform faster than AVX2. If your computer does not support AVX-512, and you switched VECTORFORTH_AVX512 on, the application will crash.
  - Computation of sine and cosine can be (as good as) exact or approximate. This behaviour is controlled with variable VECTORFORTH_SINCOS_METHOD. If you want to see the shader examples, `approximate` is a good choice as it is faster, and the loss of accuracy is not visible in the shader images. If you want the highest accuracy for sine and cosine, then choose `vectorclass`.

Also of importance is your choice related to multithreading. This is controlled with the CMake variable JTK_THREADING. There are several choices here, but the best (fastest) choice is to use [Intel's TBB library](https://software.intel.com/content/www/us/en/develop/tools/threading-building-blocks.html). This requires some work though, as you will need to install TBB yourself first.

On Windows you can download TBB's binaries from its website, and install them, preferably, in
folder C:\Program Files\TBB. Another folder is also possible, but then you'll need to
change the CMake variables `TBB_INCLUDE_DIR` and `TBB_LIBRARIES` and make them point to the correct location.
On Ubuntu you can simply run

    sudo apt install libtbb-dev

to install TBB.
On MacOS you can run

    brew install tbb

If this gives an error in the sense of `Cannot write to /usr/local/Cellar` then you can solve this probably by updating your write privileges in this folder with the command `sudo chmod a+w /usr/local/Cellar`, and then try `brew` again.

If you don't want to use/install TBB, you can set JTK_THREADING to std, which will use std::thread instead.

Other dependencies, which are delivered with the code, are
  - SDL2 (https://www.libsdl.org/)
  - Agner Fog's vectorclass (https://github.com/vectorclass/version2)
  - ocornut's dear imgui (https://github.com/ocornut/imgui)
  - nlohmann's json (https://github.com/nlohmann/json)

## The shader

I've mainly used vectorforth as a test to see what the performance of a CPU is when you let it do embarrassingly parallel GPU tasks, such as writing a shader [(watch here)](https://youtu.be/X5ZvD8SHmVI). The most basic shader application is `sf` which has only a dependency on `TBB`. The other application is `shaderforth` which, apart from `TBB`, also depends on `SDL2` and `imgui`. Both `sf` and `shaderforth` have the same functionality, but `sf` is a command line tool, and `shaderforth` has a gui.

To run a shader you have to provide the shader code in vectorforth. There are some example vectorforth shaders in subfolder shaderforth/examples. One of the examples that is good to start with (`iq_tutorial.4th`) is based on a tutorial by Inigo Quilez ( see https://www.youtube.com/watch?v=0ifChJ0nJfM). The vectorforth code looks like:

    : qx u 0.33 - ;
    : qy v 0.7 - ;

    : dist-to-center qx dup * qy dup * + sqrt ;

    : r 0.2 0.1 qy qx atan2 10 * qx 20 * + 1 + cos * +;

    : factor r 0.01 r + dist-to-center smoothstep ;

    : r2 0.01 120 qy * cos 0.002 * + -40 v * exp +;

    : factor2 1
          1 r2 0.01 r2 + qx 0.25 2 qy * sin * - abs smoothstep -
          1 0 0.1 qy smoothstep -
          * -  ;

    factor factor2 *
    dup dup
    0.4 0.8 v sqrt mix * swap
    0.1 0.3 v sqrt mix *

The shader generates the following image.
![](images/palmtree.png)

I've also reworked the tutorial by Inigo Quilez on the Happy Jumper shader (https://www.youtube.com/watch?v=Cfe5UQ-1L9Q) to vectorforth. Currently I've processed the first three steps in the tutorial, see scripts `sphere.4th`, `guy.4th`, and `guy2.4th` with corresponding images below. I'll probably not work through steps 4 and 5, as step 3 currently has 10 FPS on my pc, so steps 4/5 will be too slow on the CPU.

![](images/happy_jumper_step1.png)
![](images/happy_jumper_step2.png)
![](images/happy_jumper_step3.png)

You can also [watch the video](https://youtu.be/X5ZvD8SHmVI) on YouTube.

The other examples in subfolder shaderforth/examples are mainly taken from the website https://forthsalon.appspot.com/. The author and link to the original shader are always mentioned in the `4th` script.

## Memory

The memory used by vectorforth is subdivided into 4 batches of memory that each have their own name:
  - the stack
  - the return stack
  - the binding space
  - the data space or heap

The return stack corresponds to the c stack, which, in assembler, is controlled by the rsp register.
The other memories are dynamically aligned on 32 bytes and they are sequential, like this:

\[   ... <== stack | variable space | data space ==> ...   \]

The stack is the workspace for vectorforth. Most operators pop or push values on this stack. The top of the stack resides at the end of the memory batch, as in the scheme above. Adding items on the stack moves the stack pointer to the left, i.e., the addresses have lower values in bytes.

The data space is similar to the heap in c. With "allot" data space memory can be allocated and values can be stored here for later reference.

The binding space is used for binding words to data space locations. This memory is controlled by the compiler. With the "create \<name\>" keyword, the "variable <name>" keyword, or the "value <name>" keyword, the compiler will bind a given name to a data space address. Internally a dictionary is kept that binds \<name\> to a location in the binding space. This binding space location will point to the address in the data space memory where the memory was allocated. This means that the number of named variables/words is restricted by the size of binding space. For instance, if the binding space has size 2048 bytes, then 64 variables can be named ( 64 equals 2048 / 32 ).

## Glossary

### Core vectorforth

`: ;`    Define a new word with the syntax `: <word> <definition ...> ;`. Defined words are always inlined.

`@`    ( #a -- v )  Read the value v at 64-bit memory address #a and put it on the stack.

`!`    ( v #a -- )  Store value v in 64-bit memory address #a.

`+!`    ( v #a -- )  Add value v to the value in 64-bit memory address #a and store the result in #a.

`-!`    ( v #a -- )  Subtract value v from the value in 64-bit memory address #a and store the result in #a.

`*!`    ( v #a -- )  Multiply value v to the value in 64-bit memory address #a and store the result in #a.

`/!`    ( v #a -- )  Divide the value in 64-bit memory address #a by v and store the result in #a.

`(`    ( -- )  A multiline comment until the corresponding ).

`\`    ( -- )  A comment until the end of the line.

`+`    ( a b -- a+b )  Pop the top two values from the stack, and push their sum to the stack.

`-`    ( a b -- a-b )  Pop the top two values from the stack, and push their difference to the stack.

`*`    ( a b -- a\*b )  Pop the top two values from the stack, and push their multiplication to the stack.

`/`    ( a b -- a/b )  Pop the top two values from the stack, and push their quotient to the stack.

`=`    ( a b -- v ) 0xffffffff if the top elements are equal, else 0x00000000.

`<>`    ( a b -- v ) 0xffffffff if the top elements are not equal, else 0x00000000.

`<`    ( a b -- v ) 0xffffffff if a < b, else 0x00000000.

`>`    ( a b -- v ) 0xffffffff if a > b, else 0x00000000.

`<=`    ( a b -- v ) 0xffffffff if a <= b, else 0x00000000.

`>=`    ( a b -- v ) 0xffffffff if a >= b, else 0x00000000.

`,`    ( v -- )  Moves the top stack element to the address pointed to by the data space pointer, and updates the data space pointer so that it points to the next available location.

`abs`    ( a -- v )  Pop the top element of the stack, and put its absolute value on the stack.

`add2`    ( #a #b #r -- )  Adds two vec2 objects whose addresses are given by #a and #b. The result is saved at memory location #r.

`add3`    ( #a #b #r -- )  Adds two vec3 objects whose addresses are given by #a and #b. The result is saved at memory location #r.

`allot`    ( #u -- #a )  Allocates #u bytes of memory on the data space (heap). #u should always be a multiple of 32 for correct alignment with the simd addresses. The address of the memory allocated is pushed on the stack.

`and`    ( a b -- v )  bitwise and operator on a and b.

`atan2`    ( a b -- v )  Pop the top two elements of the stack and push the arctangent of a/b on the stack.

`begin <test> while <loop> repeat`     While loop. The loop is repeated until all vectorized values in the test return false. So it is possible that a vectorized value in the loop already is returning true in the test, but is still going through the loop, as other vectorized values are still returning false.

`ceil`    ( v -- v )  Round the top element of the stack upward.

`cells`    ( #n -- 32*#n)  Puts the size in bytes on the stack of #n cells as a 64-bit number.

`clamp`    ( c a b -- v )  Pops c a b off the stack. Pushes c if a < c and c < b, or a if c < a, or b if c > b on the stack.

`cos`    ( a -- v )  Pop the top element of the stack, and put its cosine on the stack.

`create <name>`    ( #a -- )  Creates a word <name> that returns the address #a.

`cross3`    ( #a #b #r -- )  Computes the cross product of the two vec3 objects whose addresses are given by #a and #b. The cross product is saved at memory location #r.

`depth`    ( -- #d )  Pushes the depth of the stack on the stack as a 64-bit integer.

`div`    ( a b -- quot )  Pop the top two elements a and b of the stack. Divide a by b, giving the floored quotient quot and the remainder rem. Put quot on the stack.

`div2`    ( #a #b #r -- )  Divides two vec2 objects componentwise whose addresses are given by #a and #b. The result is saved at memory location #r.

`div3`    ( #a #b #r -- )  Divides two vec3 objects componentwise whose addresses are given by #a and #b. The result is saved at memory location #r.

`dot2`    ( #a #b -- v )  Computes the dot product of the two vec2 objects whose addresses are given by #a and #b. The dot product is pushed on the stack.

`dot3`    ( #a #b -- v )  Computes the dot product of the two vec3 objects whose addresses are given by #a and #b. The dot product is pushed on the stack.

`drop`    ( v -- )  Pop the top element of the stack.

`2drop`    ( a b -- )  Pop the top two elements of the stack.

`dup`    ( v -- v v )  Duplicate the value on the top of the stack.

`2dup`    ( a b -- a b a b ) Duplicate the top two elements on the stack.

`exp`    ( a -- v )  Pop the top element of the stack, and put its exponential on the stack.

`f=`    ( a b -- v )  1.0 if the top elements are equal, else 0.0.

`f<>`    ( a b -- v )  1.0 if the top elements are not equal, else 0.0.

`f<`    ( a b -- v )  1.0 if a < b, else 0.0.

`f>`    ( a b -- v )  1.0 if a > b, else 0.0.

`f<=`    ( a b -- v )  1.0 if a <= b, else 0.0.

`f>=`    ( a b -- v )  1.0 if a >= b, else 0.0.

`false`    ( -- f )  Pushes false on the stack. False is a 256-bit number with all bits equal to 0.

`fcast`  ( #v -- v )  Pops the top item off the stack, treats this value as 8 vectorized 32-bit integers, casts them to float, and pushes the 8 vectorized float values on the stack.

`floor`    ( v -- v )  Round the top element of the stack downward.

`fm/mod`    ( a b -- rem quot )  Pop the top two elements a and b of the stack. Divide a by b, giving the floored quotient quot and the remainder rem. Put rem and quot on the stack.

`here`    ( -- #a )  Puts the 64-bit address of the data space (heap) pointer on the stack.

`icast`  ( v -- #v )  Pops the top item off the stack, treats this value as 8 vectorized 32-bit floats, casts them to a 32-bit integer, and pushes the 8 vectorized 32-bit integers on the stack.

`<test> if <true> else <false> then`    Conditional statement. The test value on the stack should equal 0xffffffff as true or 0x00000000 as false. The true and false branches should be balanced, meaning that both branches should add an equal amount of items on the stack, and any use of the return stack should be balanced. Furthermore it is not allowed in a branch to pop items off the stack that existed before the branch was initiated.

`length2`    ( #a -- )  Pushes the length of vec2 object with address #a on the stack.

`length3`    ( #a -- )  Pushes the length of vec3 object with address #a on the stack.

`log`    ( a -- v )  Pop the top element of the stack, and put its logarithm with base e on the stack.

`mat2x2 <name>`    ( -- )  Reserves 4 cells on the data space (heap) and binds \<name\> to this memory.

`mat2x2!`    ( a b c d #v -- )  Moves values a, b, c, d to the address #v. Typicall #v is obtained by providing a mat2x2 word. The values are assumed to be in column-major order.

`mat3x3 <name>`    ( -- )  Reserves 9 cells on the data space (heap) and binds \<name\> to this memory.

`mat3x3!`    ( a1 ... a9 #v -- )  Moves values a1, a2, ..., a9 to the address #v. Typicall #v is obtained by providing a mat3x3 word. The values are assumed to be in column-major order.

`max`    ( a b -- v )  Select the larger value of the top two stack elements.

`min`    ( a b -- v )  Select the smaller value of the top two stack elements.

`mix`    ( x y a -- v )  Pops x y a off the stack. Pushes the glsl function mix(x, y, a) on the stack ( x * (1 - a) + y\*a ).

`mix2`    ( #x #y a #r -- )  Assumes #x #y and #r point to the address of a vec2. Computes #r = ( #x * (1 -a ) + #y\*a).

`mix3`    ( #x #y a #r -- )  Assumes #x #y and #r point to the address of a vec3. Computes #r = ( #x * (1 -a ) + #y\*a).

`mod`    ( a b -- rem )  Pop the top two elements a and b of the stack. Divide a by b, giving the floored quotient quot and the remainder rem. Put rem on the stack.

`mul2`    ( #a #b #r -- )  Multiplies two vec2 objects componentwise whose addresses are given by #a and #b. The result is saved at memory location #r.

`mul2x2`    ( #m #v #r -- )  Multiplies a mat2x2 object #m with a vec2 object #v. The result is saved at memory location #r.

`mul3`    ( #a #b #r -- )  Multiplies two vec3 objects componentwise whose addresses are given by #a and #b. The result is saved at memory location #r.

`mul3x3`    ( #m #v #r -- )  Multiplies a mat3x3 object #m with a vec3 object #v. The result is saved at memory location #r.

`negate`    ( a -- -a )  Pop the top element of the stack, and put its negative value on the stack.

`nip`    ( a b -- b )  Drop the first item below the top of the stack.

`normalize2`  ( #a #r -- )  Normalizes the vec2 object with address #a and saves the result at memory location #r.

`normalize3`  ( #a #r -- )  Normalizes the vec3 object with address #a and saves the result at memory location #r.

`not`    ( a -- v ) toggle all bits of a.

`or`    ( a b -- v ) bitwise or operator on a and b.

`over`    ( v w -- v w v )  Duplicate the element under the top stack element.

`pi`    ( -- 3.1415926535897 )  Puts pi on the stack.

`pick`    ( x_u ... x_1 x_0 #u -- x_u ... x_1 x_0 x_u )  Remove #u from the stack and copy x_u to the top of the stack.

`pop`   ( -- v )  Pop a value v from the return stack and push it on the regular stack.

`pow`    ( a b -- a^b )  Pop the top two elements of the stack, and push a to the power b on the stack.

`**`    ( a b -- a^b )  Pop the top two elements of the stack, and push a to the power b on the stack.

`push`   ( v -- )  Move a value v from the stack to the return stack.

`r>`   ( -- v )  Pop a value v from the return stack and push it on the regular stack.

`>r`   ( v -- )  Move a value v from the stack to the return stack.

`r@`    ( -- v )  Push a copy of the top value of the return stack on the regular stack.

`rdepth`    ( -- #d )  Pushes the depth of the return stack on the stack as a 64-bit integer.

`reflect3`    ( #i #n #r -- )  Assumes #i, #n and #r point to the address of a vec3. Computes #r = #i - 2 dot3(#n, #i) * #n.

`rot`    ( a b c -- b c a )  Rotate the top three elements on the stack.

`-rot`    ( a b c -- c a b )  Reverse rotate the top three elements on the stack.

`round`    ( v -- v )  Round the top element of the stack to the nearest integer.

`rp@`    ( -- #v ) Read the 64-bit address of the return stack pointer and put it on the stack.

`rt@`    ( -- #v ) Read the 64-bit address of the return stack top and put it on the stack.

`scalarmul2`    ( t #a #r -- )  Multiplies a vec2 object with address #a with a scalar t. The result is saved at memory location #r.

`scalarmul3`    ( t #a #r -- )  Multiplies a vec3 object with address #a with a scalar t. The result is saved at memory location #r.

`sdcapsule`    ( #p #a #b r -- d )  Computes the signed distance from vec3 #p to a line segment defined by vec3 #a and vec3 #b and radius r. (https://iquilezles.org/www/articles/distfunctions/distfunctions.htm)

`sdellipsoid`    ( #p #r -- d ) Computes the signed distance from vec3 #p to an origin centered ellipsoid with radii defined by vec3 #r.  (https://iquilezles.org/www/articles/distfunctions/distfunctions.htm)

`sdsphere`    ( #p r -- d )  Computes the signed distance from vec3 #p to an origin centered sphere with radius r.  (https://iquilezles.org/www/articles/distfunctions/distfunctions.htm)

`sdstick`    ( #p #a #b r1 r2 -- d )  Computes the signed distance from vec3 #p to a line segment defined by vec3 #a and vec3 #b and varying thickness going from radius r1 to radius r2. (https://iquilezles.org/www/articles/distfunctions/distfunctions.htm)

`sin`    ( a -- v )  Pop the top element of the stack, and put its sine on the stack.

`smax`     ( a b k -- v )  Polynomial smoothed maximum of values a and b with smoothness parameter k.

`smin`    ( a b k -- v )  Polynomial smoothed minimum of values a and b with smoothness parameter k.

`smoothstep`    ( a b x -- v )  Pops a b x off the stack. Pushes the glsl function smoothstep(a, b, x) on the stack.

`sp@`    ( -- #v ) Read the 64-bit address of the stack pointer and put it on the stack.

`sqrt`    ( a -- v )  Pop the top element of the stack, and put its square root on the stack.

`step`    ( edge x -- v )  Generates a step function by comparing x to edge. 0 is returned if x < edge, and 1 is returned otherwise. This is similar to f<=.

`st@`    ( -- #v ) Read the 64-bit address of the stack top and put it on the stack.

`sub2`    ( #a #b #r -- )  Subtracts two vec2 objects whose addresses are given by #a and #b. The result is saved at memory location #r.

`sub3`    ( #a #b #r -- )  Subtracts two vec3 objects whose addresses are given by #a and #b. The result is saved at memory location #r.

`swap`    ( v w -- w v ) Swap the top two elements on the stack.

`2swap`    ( a b c d --- c d a b )  Swap the top two pairs on the stack.

`tan`    ( a -- v )  Pop the top element of the stack, and put its tangent on the stack.

`to <name>`    ( v -- )  The word <name> should be the name of a value previously defined. The top value of the stack is popped, and its value is assigned to the value <name>.

`true`    ( -- t )  Pushes true on the stack. True is a 256-bit number with all bits equal to 1.

`trunc`    ( v -- v )  Truncate the top element of the stack (i.e. round towards zero).

`tuck`    ( a b -- b a b )  Copy the top stack item below the second stack item.

`value <name>`    ( v -- )  Pops the top value off the stack. Assigns this value to the word given by <name>.

`variable <name>`    ( -- )  Reserves/allocates 1 cell on the data space (heap). The address of this cell can be obtained by the provided name.

`vec2 <name>`    ( -- )  Reserves 2 cells on the data space (heap) and binds \<name\> to this memory.

`vec2!`    ( a b #v -- )  Moves values a, bc to the address #v. Typicall #v is obtained by providing a vec2 word.

`vec3 <name>`    ( -- )  Reserves 3 cells on the data space (heap) and binds \<name\> to this memory.

`vec3!`    ( a b c #v -- )  Moves values a, b, c to the address #v. Typicall #v is obtained by providing a vec3 word.

`v8 a b c d e f g h`    ( -- v ) Push a vectorized value on the stack with floating point values given by a,...,h.

`v8 #a #b #c #d #e #f #g #h`    ( -- v ) Push a vectorized value on the stack with 32-bit integer values given by #a,...,#h.

`within`    ( c a b -- <true|false> )  Returns true if a <= c and c < b, false otherwise.

`xor`    ( a b -- v ) bitwise xor operator on a and b.

`z*`    ( a b c d -- a\*c-b\*d a\*d+b\*c )  Pop the top four values from the stack, treat the pairs as complex numbers, and store their multiplication on the stack.

`z+`    ( a b c d -- a+c b+d )  Pop the top four values from the stack, treat the pairs as complex numbers, and store their sum on the stack.

`z-`    ( a b c d -- a-c b-d )  Pop the top four values from the stack, treat the pairs as complex numbers, and store their difference on the stack.

`#+`    ( #a #b -- #a+#b )  Pop two 64-bit addresses from the stack, and push their sum on the stack.

`#+!`    ( #v #a -- )  Assumes that 64-bit address #a points to another address, and adds 64-bit value #v to this address pointed to by #a. The top two elements on the stack are dropped.

`#-`    ( #a #b -- #a-#b )  Pop two 64-bit addresses from the stack, and push their difference on the stack.

`#-!`    ( #v #a -- )  Assumes that 64-bit address #a points to another address, and subtracts 64-bit value #v from this address pointed to by #a. The top two elements on the stack are dropped.

`#*`    ( #a #b -- #a*#b )  Pop two 64-bit addresses from the stack, and push their multiplication on the stack.

`#/`    ( #a #b -- #a*#b )  Pop two 64-bit addresses from the stack, and push their quotient on the stack.

`#<<`    ( #a #b -- #a<<#b )  Pop two 64-bit addresses from the stack, left-shift them, and put the result on the stack.

`#>>`    ( #a #b -- #a>>#b )  Pop two 64-bit addresses from the stack, right-shift them, and put the result on the stack.

`#1+`    ( #a -- #a+1)  Pop the top item (an address typically) from the stack and add one. Push the result on the stack.

`#1-`    ( #a -- #a-1)  Pop the top item (an address typically) from the stack and subtract one. Push the result on the stack.

`#32+`    ( #a -- #a+32)  Pop the top item (an address typically) from the stack and add 32. Push the result on the stack.

`#32-`    ( #a -- #a-32)  Pop the top item (an address typically) from the stack and subtract 32. Push the result on the stack.

`#64+`    ( #a -- #a+64)  Pop the top item (an address typically) from the stack and add 64. Push the result on the stack.

`#64-`    ( #a -- #a-64)  Pop the top item (an address typically) from the stack and subtract 64. Push the result on the stack.

`#96+`    ( #a -- #a+96)  Pop the top item (an address typically) from the stack and add 96. Push the result on the stack.

`#96-`    ( #a -- #a-96)  Pop the top item (an address typically) from the stack and subtract 96. Push the result on the stack.

`#128+`    ( #a -- #a+128)  Pop the top item (an address typically) from the stack and add 128. Push the result on the stack.

`#128-`    ( #a -- #a-128)  Pop the top item (an address typically) from the stack and subtract 128. Push the result on the stack.

`#160+`    ( #a -- #a+160)  Pop the top item (an address typically) from the stack and add 160. Push the result on the stack.

`#160-`    ( #a -- #a-160)  Pop the top item (an address typically) from the stack and subtract 160. Push the result on the stack.

`#192+`    ( #a -- #a+192)  Pop the top item (an address typically) from the stack and add 192. Push the result on the stack.

`#192-`    ( #a -- #a-192)  Pop the top item (an address typically) from the stack and subtract 192. Push the result on the stack.

`#224+`    ( #a -- #a+224)  Pop the top item (an address typically) from the stack and add 224. Push the result on the stack.

`#224-`    ( #a -- #a-224)  Pop the top item (an address typically) from the stack and subtract 224. Push the result on the stack.

`#256+`    ( #a -- #a+256)  Pop the top item (an address typically) from the stack and add 256. Push the result on the stack.

`#256-`    ( #a -- #a-256)  Pop the top item (an address typically) from the stack and subtract 256. Push the result on the stack.

### Specific shader forth definitions

`dt`    ( -- dt )  Put the change in time since the last frame in seconds on the stack.

`frame`    ( -- frame )  Put the frame index on the stack.

`mw`    ( -- mw )  Is negative my when a mouse button is pressed, otherwise positive my.

`mx`    ( -- mx )  Put the mouse x-coordinate on the stack.

`my`    ( -- my )  Put the mouse y-coordinate on the stack.

`mz`    ( -- mz )  Is negative mx when a mouse button is pressed, otherwise positive mx.

`rx`   ( -- rx )  Put the x-component of the screen resolution on the stack.

`ry`   ( -- ry )  Put the y-component of the screen resolution on the stack.

`t`    ( -- t )  Put the current time in seconds since the start of the shader on the stack.

`u`    ( -- u )  Put the normalized (between 0 and 1) x-coordinate on the stack.

`v`    ( -- v )  Put the normalized (between 0 and 1) y-coordinate on the stack.

`x`    ( -- x )  Put the current x-coordinate on the stack.

`y`    ( -- y )  Put the current y-coordinate on the stack.


text:
# Building Modern Services with .NET Core 3 and gRPC

This is the sample code from my session on Building Modern Services with .NET Core 3 and gRPC. [Slides are available here on Speaker Deck](https://speakerdeck.com/gshackles/building-modern-services-with-net-core-3-and-grpc).

Samples:
- [Rangers Playoff Status Checker](./samples/RangersPlayoffStatusChecker): demo that walks through creating a simple worker service and hosting it in different environments
- [Hockey Standings](./samples/HockeyStandings): demo gRPC service for checking NHL team standings

repository: gshackles/aspnetcore-grpc-demo

branch_name: refs/heads/main

gha_license_id: MIT

repo_name: gshackles/aspnetcore-grpc-demo


text:
# [WIP] Symfony CMF Taxonomy Bundle [![Build Status](https://secure.travis-ci.org/symfony-cmf/TagBundle.png)](http://travis-ci.org/symfony-cmf/TaxonomyBundle)

The aim of this bundle is to provide a complete taxonomy solution. A taxonomy
is a system of classification which can be either hierachical (e.g. a family *tree*)
or flat (e.g. a *list* of HTTP codes).

Taxonomies can also be mutable either upon request (e.g. blog post tags) or by
an administrator (e.g. product categories).

Individual categories are known as *taxons*, and depending on context this
can translated variously as "tag", "category", "type", etc.

The bundle is structured in such a way as to allow it to be adapted easily to
different ORMs and Admin implementations.

## Installation

Add a requirement for ``__/__`` to your
composer.json and instantiate the bundle in your AppKernel.php

    new Symfony\Cmf\Bundle\TaxonomyBundle\SymfonyCmfTaxonomyBundle()

## Running the tests

To initialize the test environment run the initialization script (you only need
to do this once):

    ./Tests/Functional/init_travis.sh

Then run all the tests with:

    phpunit -c phpunit.xml.dist

repository: dantleech/TaxonomyBundle

repo_name: dantleech/TaxonomyBundle