text:
# NEROS

<<<<<<< HEAD
A Node-RED fork serving the IoT more securely. A visual tool for wiring the Internet of Things securely.

## Authors

NEROS is based on Node-RED which is a project of the [JS Foundation](http://js.foundation).

It was created by [IBM Emerging Technology](https://www.ibm.com/blogs/emerging-technology/).

## Copyright and license

Copyright JS Foundation and other contributors, http://js.foundation under [the Apache 2.0 license](LICENSE).

Additions made to Node-RED are Copyright 2016 FORSEC project.

repository: SEDARI/neros

repo_name: SEDARI/neros


text:
---
# Documentation: https://sourcethemes.com/academic/docs/managing-content/

title: "Singapore City-AI Meetup 3.0"
event:
event_url: http://www.eventnook.com/event/modelbiasinai
location:
address:
  street:
  city:
  region:
  postcode:
  country:
summary: Singapore AI 3rd Quarterly Meetup
abstract:

# Talk start and end times.
#   End time can optionally be hidden by prefixing the line with `#`.
date: 2017-08-17T17:00:00+08:00
date_end:
all_day: false

# Schedule page publish date (NOT talk date).
publishDate: 2020-03-02T08:40:38+08:00

authors: []
tags: []

# Is this a featured talk? (true/false)
featured: false

# Featured image
# To use, add an image named `featured.jpg/png` to your page's folder.
# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
image:
  caption: ""
  focal_point: ""
  preview_only: false

# Custom links (optional).
#   Uncomment and edit lines below to show custom links.
# links:
# - name: Follow
#   url: https://twitter.com
#   icon_pack: fab
#   icon: twitter

# Optional filename of your slides within your talk's folder or a URL.
url_slides:

url_code:
url_pdf:
url_video:

# Markdown Slides (optional).
#   Associate this talk with Markdown slides.
#   Simply enter your slide deck's filename without extension.
#   E.g. `slides = "example-slides"` references `content/slides/example-slides.md`.
#   Otherwise, set `slides = ""`.
slides: ""

# Projects (optional).
#   Associate this post with one or more of your projects.
#   Simply enter your project's folder or file name without extension.
#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
#   Otherwise, set `projects = []`.
projects: []
---

I was honoured to be invited to present on Model Bias at Singapore City-AI's 3rd Quarterly Meetup.

I presented a case study from Gojek- our drivers ranking model and various ways we can use to dissect the model in order to understand the relationship between features and prediction. I also highlighted the importance of evaluating the fairness of the model and how it impacts the overall system. Later, I used a case study from a credit default prediction challenge to demonstrate how we can use Partial Dependence Plot (PDP) to figure out how each feature affects the prediction. It was a great experience and I enjoyed interacting with other esteemed guest speakers from Nvidia, MSD, Allianz and the audience. It ended with a great panel discussion on bias in AI / ML models and how we can tackle that.

```python
import xgboost as xgb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import xgbfir
pd.options.display.max_columns = 100
pd.options.display.max_rows = 100
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import plotly.graph_objs as go
init_notebook_mode(connected=True)
pd.set_option('max_colwidth',200)
import cPickle
import lime
from sklearn.model_selection import train_test_split
```

<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>

# Peeking Inside the Blackbox
Discover potential bias inside machine learning models

Zane Lim

* Senior data-scientist at Go-jek
* Lead and mentor a sub-team of data-scientists
* Mentor in Udacity's AI nanodegree
* Avid participant in Kaggle / data science hackathons

* https://www.linkedin.com/in/limzane/
* https://www.kaggle.com/zanelim

## AI and Machine Learning: Applications and Pitfalls
Applications
* [AlphaGo beats Ke Jie](https://www.theverge.com/2017/5/25/15689462/alphago-ke-jie-game-2-result-google-deepmind-china)
* [DeepMind training AI to beat human Starcraft players](https://www.newscientist.com/article/2143144-deepmind-dojo-will-train-ai-to-beat-human-starcraft-players/)
* [OpenAI Dota Bot destroys world's best player](https://www.theverge.com/2017/8/11/16137388/dota-2-dendi-open-ai-elon-musk)
* https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html
* https://www.theguardian.com/technology/self-driving-cars

Pitfalls
* http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender
* https://www.forbes.com/sites/mariyayao/2017/05/01/dangers-algorithmic-bias-homogenous-thinking-ai/#2228a48270b3
* https://techcrunch.com/2016/12/10/5-unexpected-sources-of-bias-in-artificial-intelligence/

![title](./news.png)
Critical as more and more decisions are made using algorithms

## Case study on Go-Jek

<img src="./gojek.png" width="1000" height="100" align="middle"/>

* On-demand mobile platform that provides a variety of complete services
    * transportation (300,000 *ojek* and car driver partners)
    * logistics / food delivery (100,000 food vendors)
    * payment
    * other on-demand services (masseurs, cleaners etc.)
* As of July 2017, downloaded more than 44 million times
* Operates in 35 cities in Indonesia- Jakarta, Bandung, Surabaya, Bali, Medan, Yogyakarta etc.

### Drivers Ranking Model
![title](./drivers-gojek.jpg)
* Real-time machine learning model that handles **hundreds** of driver assignment (per second) during peak hours
* Rank drivers who are available in the vicinity of customer whenever a booking is requested
* Based on certain criteria eg. *probability of completion*
* Assign the job to the top ranked driver

### Features of Model
* basic features- transactional data (time of day, location, gps accuracy)
* derived features- drivers' attributes (statistics, performance)
* re-trained everyday on a rolling window of historical data to ensure freshness
* deployed and a/b tested

### Results

* Very successful, improved conversion rate and acceptance rate against control
* *How do we assess biasness and fairness of the model?*
* Walking on fine line, affects the livelihood of drivers, any perceived mistreatment will trigger riots

**Dissect the model and identify potential bias**
* Plot and visualize the "model" directly (individual trees in gradient boosting model)
* Relationship between feature and prediction
    * Partial Dependence Plot (PDP) / Individual Conditional Expectation (ICE) Plot
* "Explains" the prediction
    * Local Interpretable Model-agnostic Explanations (LIME)

**Evaluate the impact of the model**
* Fairness of the system- Gini coefficient of drivers income / jobs assigned
* Average score (probability) / rank for each driver

**Promote transparency**
* Monitor and measure driver complaints on not getting jobs
* Explain and provide suggestions to drivers on how they can improve their prospect of getting jobs (indirectly promote good behaviours)

## Case study on Credit Card Default Payment

* A dataset containing, default payments, history of payments, bill statements and demographics of 30,000 credit card clients in Taiwan from April 2005 to September 2005
* Predict the probability of default payment in next month

https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset

Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

```python
df = pd.read_csv("../data/UCI_Credit_Card.csv")
X = df.drop('default.payment.next.month', axis=1)
y = df['default.payment.next.month']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
```

```python
with open("../data/xgb_clf.pkl") as f:
    xgb_clf = cPickle.load(f)
```

* ID: ID of each client
* LIMIT_BAL: Amount of given credit in NT dollars (includes individual and family/supplementary credit
* SEX: Gender (1=male, 2=female)
* EDUCATION: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)
* MARRIAGE: Marital status (1=married, 2=single, 3=others)
* AGE: Age in years
* PAY_0: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, ... 8=payment delay for eight months, 9=payment delay for nine months and above)
* PAY_2: Repayment status in August, 2005 (scale same as above)
* PAY_3: Repayment status in July, 2005 (scale same as above)
* PAY_4: Repayment status in June, 2005 (scale same as above)
* PAY_5: Repayment status in May, 2005 (scale same as above)
* PAY_6: Repayment status in April, 2005 (scale same as above)
* BILL_AMT1: Amount of bill statement in September, 2005 (NT dollar)
* BILL_AMT2: Amount of bill statement in August, 2005 (NT dollar)
* BILL_AMT3: Amount of bill statement in July, 2005 (NT dollar)
* BILL_AMT4: Amount of bill statement in June, 2005 (NT dollar)
* BILL_AMT5: Amount of bill statement in May, 2005 (NT dollar)
* BILL_AMT6: Amount of bill statement in April, 2005 (NT dollar)
* PAY_AMT1: Amount of previous payment in September, 2005 (NT dollar)
* PAY_AMT2: Amount of previous payment in August, 2005 (NT dollar)
* PAY_AMT3: Amount of previous payment in July, 2005 (NT dollar)
* PAY_AMT4: Amount of previous payment in June, 2005 (NT dollar)
* PAY_AMT5: Amount of previous payment in May, 2005 (NT dollar)
* PAY_AMT6: Amount of previous payment in April, 2005 (NT dollar)
* default.payment.next.month: Default payment (1=yes, 0=no)

### Plotting the "model" directly

```python
features_importance = pd.DataFrame.from_dict(xgb_clf.get_fscore(), orient='index').reset_index().rename(columns={'index': 'features', 0: 'importance'})
features_importance.sort_values('importance', ascending=False, inplace=True)
```

```python
features_importance.plot.barh(x='features', y='importance', figsize=(20,15))
plt.gca().invert_yaxis()
```

![png](./singapore-city-ai_17_0.png)

```python
# plotting the trees directly
xgb.to_graphviz(xgb_clf, num_trees=0, size='150,200')
```

![svg](./singapore-city-ai_18_0.svg)

### Partial Dependence Plot (PDP)

* A graph plotting the relationship of one single feature with the prediction / probability on average
* For each combination of a unique value of the feature with all other features, what is the average prediction / probability
* One caveat of using average is that it does not represent the usual case, as in any single realistic case, it depends on the combination with other features

```python
def plotly_lines(df, x, y):
    data = [
    go.Scatter(
        x=df[x], # assign x as the dataframe column 'x'
        y=df[y],
        name=x
    )
    ]
    return data

def plotly_figure(df, title, x_title, y_title, x, y):
    layout = go.Layout(
        title=title,
        xaxis=dict(
            title=x_title,
            titlefont=dict(
                family='Courier New, monospace',
                size=18,
                color='#7f7f7f'
            )
        ),
        yaxis=dict(
            title=y_title,
            titlefont=dict(
                family='Courier New, monospace',
                size=18,
                color='#7f7f7f'
            )
        )
    )
    return go.Figure(data=plotly_lines(df, x, y), layout=layout)
```

```python
df_plot = pd.read_csv("../data/age.csv")
ax = df_plot.plot(x='AGE', y='yhat', title='Age', figsize=(15,5), legend=False)
ax.set_xlabel("Age")
ax.set_ylabel("Probability of Default")
```

    <matplotlib.text.Text at 0x11d3003d0>

![png](./singapore-city-ai_21_1.png)

```python
df_plot = pd.read_csv("../data/sex.csv")
ax = df_plot.plot(x='SEX', y='yhat', title='Sex', figsize=(15,5), legend=False)
ax.set_xlabel("Sex")
ax.set_ylabel("Probability of Default")
```

    <matplotlib.text.Text at 0x11d5f0290>

![png](./singapore-city-ai_22_1.png)

```python
df_plot = pd.read_csv("../data/marriage.csv")
ax = df_plot.plot(x='MARRIAGE', y='yhat', title='Marital Status', figsize=(15,5), legend=False)
ax.set_xlabel("Marital Status")
ax.set_ylabel("Probability of Default")
```

    <matplotlib.text.Text at 0x11d7bba50>

![png](./singapore-city-ai_23_1.png)

```python
df.MARRIAGE.value_counts(normalize=True, sort=False)
```

    0    0.001800
    1    0.455300
    2    0.532133
    3    0.010767
    Name: MARRIAGE, dtype: float64

```python
df_plot = pd.read_csv("../data/education.csv")
ax = df_plot.plot(x='EDUCATION', y='yhat', title='Education', figsize=(15,5), legend=False)
ax.set_xlabel("Education")
ax.set_ylabel("Probability of Default")
```

    <matplotlib.text.Text at 0x11d9a9a50>

![png](./singapore-city-ai_25_1.png)

```python
df.EDUCATION.value_counts(normalize=True, sort=False)
```

    0    0.000467
    1    0.352833
    2    0.467667
    3    0.163900
    4    0.004100
    5    0.009333
    6    0.001700
    Name: EDUCATION, dtype: float64

Other techniques to dissect the black box
* Individual Conditional Expectation (ICE) Plot
* Local Interpretable Model-agnostic Explanations (LIME)

Reference:
* [Why Should I Trust You?” Explaining the Predictions of Any Classifier](https://arxiv.org/pdf/1602.04938.pdf)
* [Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation](https://arxiv.org/pdf/1309.6392.pdf)
* https://www.datarobot.com/blog/give-me-one-good-reason-to-trust-artificial-intelligence/

"""

    Build AI with conscience, trust and empathy

"""

Thank you!

license: MIT

repository: zyuanlim/academic-kickstart

branch_name: refs/heads/master

extension: md

gha_license_id: MIT

license_type: permissive

repo_name: zyuanlim/academic-kickstart


text:
# Redmine Notifier

[![Code Climate](https://codeclimate.com/github/emsk/redmine-notifier/badges/gpa.svg)](https://codeclimate.com/github/emsk/redmine-notifier)
[![Dependency Status](https://gemnasium.com/emsk/redmine-notifier.svg)](https://gemnasium.com/emsk/redmine-notifier)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Redmine Notifier is a simple updated issues checker that runs in the background.
It sends a desktop notification if there are any updates in issues.

![Notification Mac 10.10](examples/notification_osx_10.10.png?raw=true)
![Notification Win 8.1](examples/notification_win_8.1.png?raw=true)
![Notification Win 7](examples/notification_win_7.png?raw=true)

The number of issues that were updated after previous fetch is shown in parentheses.
Also, the subject is shown only about the latest issue.

## Downloads

Installers for OS X and Windows can be found on the [releases](../../releases) page.

## Settings

![Settings](examples/redmine_notifier_settings.png?raw=true)

The icon appears in the menu bar or task tray when Redmine Notifier is started.
Select "Preferences" in the context menu to open the settings window.

## Development

Redmine Notifier is powered by [Electron](http://electron.atom.io/), so we can develop it with web technologies.

### Dependencies

##### Production

* [node-notifier](https://github.com/mikaelbr/node-notifier)
* [notie](https://github.com/jaredreich/notie.js)

##### Development

* [csslint](https://github.com/CSSLint/csslint)
* [electron-builder](https://github.com/loopline-systems/electron-builder)
* [electron-packager](https://github.com/maxogden/electron-packager)
* [electron-prebuilt](https://github.com/mafintosh/electron-prebuilt)
* [eslint](https://github.com/eslint/eslint)

See `dependencies` and `devDependencies` in [`package.json`](package.json).

### Installing dependencies

```sh
cd /path/to/redmine-notifier
npm run prepare
```

### Starting app

```sh
npm start
```

### Building apps

```sh
npm run build
```

### Building installers

```sh
npm run pack
```

### Linting JavaScript and CSS files

```sh
npm run lint
```

See `scripts` in [`package.json`](package.json) with regard to other commands.

## Contributing

1. Fork it ( https://github.com/emsk/redmine-notifier/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request

## License

[MIT](LICENSE)

repository: konnosena/redmine-notifier

repo_name: konnosena/redmine-notifier


text:
This doc contains information about the synchronous replication module in gluster and has two sections

- Replication logic
- Self-heal logic.

# 1. Replication logic

AFR is the module (translator) in glusterfs that provides all the features that you would expect of any synchronous replication system:

1.  Simultaneous updating of all copies of data on the replica bricks when a client modifies it.
2.  Providing continued data availability to clients when say one brick of the replica set goes down.
3.  Automatic self-healing of any data that was modified when the brick that was down, once it comes back up, ensuring consistency of data on all the bricks of the replica.

1 and 2 are in the I/O path while 3 is done either in the I/O path (in the background) or via the self-heal daemon.

Each gluster translator implements what are known as _File Operations (FOPs)_ which are mapped to the I/O syscalls which the application makes. For example, AFR has _afr_writev_ that gets invoked when application does a _write(2)_. As is obvious, all FOPs fall into one of 2 types:

i) Read based FOPs which only get informtion from and don’t modify the file in any way.

viz: afr_readdir, afr_access, afr_stat, afr_fstat, afr_readlink, afr_getxattr, afr_fgetxattr, afr_readv,afr_seek

ii) Write based FOPs which change the file or its attributes.

viz: afr_create, afr_mknod,afr_mkdir,afr_link, afr_symlink, afr_rename, afr_unlink, afr_rmdir, afr_do_writev, afr_truncate, afr_ftruncate, afr_setattr, afr_fsetattr, afr_setxattr, afr_fsetxattr, afr_removexattr, afr_fremovexattr, afr_fallocate, afr_discard, afr_zerofill, afr_xattrop, afr_fxattrop, afr_fsync.

AFR follows a transaction model for both types of FOPs.

### Read transactions:

For every file in the replica, AFR has an in-memory notion/array called ‘readables’ which indicate whether each brick of the replica is a good copy or a bad one (i.e. in need of a heal). In a healthy state, all bricks are readable and a read FOP will be served from any one of the readable bricks. The read-hash-mode volume option decides which brick is the chosen one.

```console
gluster volume set help | grep read-hash-mode -A7
```

```text
Option: cluster.read-hash-mode
Default Value: 1
Description: inode-read fops happen only on one of the bricks in replicate. AFR will prefer the one computed using the method specified using this option.
0 = first readable child of AFR, starting from 1st child.
1 = hash by GFID of file (all clients use same subvolume).
2 = hash by GFID of file and client PID.
3 = brick having the least outstanding read requests.
```

If the brick is bad for a given file (i.e. it is pending heal), then it won’t be marked readable to begin with. The readables array is populated based on the on-disk AFR xattrs for the file during lookup. These xattrs indicate which bricks are good and which ones are bad. We will see more about these xattrs in the write transactions section below. If the FOP fails on the chosen readable brick, AFR attempts it on the next readable one, until all are exhausted. If the FOP doesn’t succeed on any of the readables, then the application receives an error.

### Write transactions:

Every write based FOP employs a write transaction model which consists of 5 phases:

**1) The lock phase**
Take locks on the file being modified on all bricks so that AFRs of other clients are blocked if they try to modify the same file simultaneously.

**2) The pre-op phase**
Increment the ‘dirty’ xattr (trusted.afr.dirty) by 1 on all participating bricks as an indication of an impending FOP (in the next phase)

**3) The FOP phase**
Perform the actual FOP (say a setfattr) on all bricks.

**4) The post-op phase**
Decrement the dirty xattr by 1 on bricks where the FOP was successful.
In addition, also increment the ‘pending’ xattr (trusted.afr.$VOLNAME-client-x) xattr on the success bricks to ‘blame’ the bricks where the FOP failed.

**5) The unlock phase**
Release the locks that were taken in phase 1. Any competing client can now go ahead with its own write transaction.

**Note**: There are certain optimizations done at the code level which reduce the no. of lock/unlock phases done for a transaction by piggybacking on the previous transaction’s locks. These optimizations (eager-locking, piggybacking and delayed post-op) beyond the scope of this post.

AFR returns sucess for these FOPs only if they meet quorum. For replica 2, this means it needs to suceed on any one brick. For replica 3, it is two out of theree and so on.

### More on the AFR xattrs:

We saw that AFR modifies the dirty and pending xattrs in the pre-op and post-op phases. To be more precise, only parts of the xattr are modified in a given transaction. Which bytes are modified depends on the type of write transaction which the FOP belongs to.

|     Transaction Type     |                                                FOPs that belong to it                                                |
| :----------------------: | :------------------------------------------------------------------------------------------------------------------: |
|   AFR_DATA_TRANSACTION   |             afr_writev, afr_truncate, afr_ftruncate, afr_fsync, afr_fallocate, afr_discard, afr_zerofill             |
| AFR_METADATA_TRANSACTION | afr_setattr, afr_fsetattr, afr_setxattr, afr_fsetxattr, afr_removexattr, afr_fremovexattr, afr_xattrop, afr_fxattrop |
|  AFR_ENTRY_TRANSACTION   |              afr_create, afr_mknod, afr_mkdir, afr_link, afr_symlink, afr_rename, afr_unlink, afr_rmdir              |

Stop here and convince yourself that given a write based FOP, you can say which one of the 3 transaction types it belongs to.

**Note:** In the code, there is also a AFR_ENTRY_RENAME_TRANSACTION (used by afr_rename) but it is safe to assume that it is identical to AFR_ENTRY_TRANSACTION as far as interpreting the xattrs are concerned.

Consider the xttr:
`trusted.afr.dirty=0x000000000000000000000000`
The first 4 bytes of the xattr are used for data transactions, the next 4 bytes for metadata transactions and the last 4 for entry transactions. Let us see some examples of how the xattr would look like for various types of FOPs during a transaction:

| FOP         | Value after pre-op phase                       | Value after post-op phase                      |
| ----------- | ---------------------------------------------- | ---------------------------------------------- |
| afr_writev  | trusted.afr.dirty=0x00000001 00000000 00000000 | trusted.afr.dirty=0x00000000 00000000 00000000 |
| afr_setattr | trusted.afr.dirty=0x00000000 00000001 00000000 | trusted.afr.dirty=0x00000000 00000000 00000000 |
| afr_create  | trusted.afr.dirty=0x00000000 00000000 00000001 | trusted.afr.dirty=0x00000000 00000000 00000000 |

Thus depending on the type of FOP (i.e. data/ metadata/ entry transaction), different set of bytes of the dirty xattr get incremented/ decremented. Modification of the pending xattr also follows the same pattern, execept it is incremented only in the post-op phase if the FOP fails on some bricks.

**Example:**
Let us say a write was performed on a file, say FILE1, on replica 3 volume called ‘testvol’. Suppose the lock and pre-op phase succeeded on all bricks. After that the 3rd brick went down, and the transaction completed successfully on the first 2 bricks.
What will be the state of the afr xattrs on all bricks?

```console
# getfattr -d -m . -e hex /bricks/brick1/FILE1 | grep afr
getfattr: Removing leading '/' from absolute path names
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.testvol-client-2=0x000000010000000000000000
```

```
# getfattr -d -m . -e hex /bricks/brick2/FILE1 | grep afr
getfattr: Removing leading '/' from absolute path names
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.testvol-client-2=0x000000010000000000000000
```

```
# getfattr -d -m . -e hex /bricks/brick3/FILE1 | grep afr
getfattr: Removing leading '/' from absolute path names
trusted.afr.dirty=0x000000010000000000000000
```

So Brick3 will still have the dirty xattr set because it went down before the post-op had a chance to decrement it. Bricks 1 and 2 will have a zero dirty xattr and in addition, a non-zero pending xattr set. The client-2 in trusted.afr.testvol-client-2 indicates that the 3rd brick is bad and has some pending data operations.

# 2. Self-heal logic.

We already know that AFR increments and/or decrements the dirty (i.e. `trusted.afr.dirty`) and pending (i.e. `trusted.afr.$VOLNAME-client-x`) xattrs during the different phases of the transaction. For a given file (or directory), an all zero value of these xattrs or the total absence of these xattrs on all bricks of the replica mean the file is healthy and does not need heal. If any of these xattrs are non-zero even on one of the bricks, then the file is a candidate for heal- it as simple as that.

When we say these xattrs are non-zero, it is in the context of no on-going I/O going from client(s) on the file. Otherwise the non-zero values that one observes might be transient as the write transaction is progressing through its five phases. Of course, as an admin, you wouldn’t need to figure out all of this. Just running the `heal info` set of commands should give you the list of files that need heal.

So if self-heal observes a file with non-zero xattrs, it does the following steps:

1.  Fetch the afr xattrs, examine which set of 8 bytes are non-zero and determine the corresponding heals that are needed on the file – i.e. **data heal/ metadata heal/ entry heal**.
2.  Determine which bricks are good (a.k.a. ‘sources’) and which ones are bad (a.k.a. ‘sinks’) for each of those heals by interpretting the xattr values.
3.  Pick one source brick and heal the file on to all the sink bricks.
4.  If the heal is successful, reset the afr xattrs to zero.

This is a rather simplified description and I have omitted details about various locks that each of these steps need to take because self-heal and client I/O can happen in parallel on the file. Or even multiple self-heal daemons (described later) can attempt to heal the same file.

**Data heal**: Happens only for files. The contents of the file are copied from the source to the sink bricks.

**Entry heal**: Happens only for directories. Entries (i.e. files and subdirs) under a given directory are deleted from the sinks if they are not present in the source. Likewise, entries are created on the sinks if they are present in the source.

**Metadata heal:** Happens for both files and directories. File ownership, file permissions and extended attributes are copied from the source to the sink bricks.

It can be possible that for a given file, one set of bricks can be the source for data heal while another set could be the source for metadata heals. It all depends on which FOPs failed on what bricks and therefore what set of bytes are non-zero for the afr xattrs.

## When do self-heals happen?

There are two places from which the steps described above for healing can be carried out:

### i) From the client side.

Client-side heals are triggered when the file is accessed from the client (mount). AFR uses a monotonically increasing generation number to keep track of disconnect/connect of its children (i.e. the client translators) to the bricks. When this ‘event generation’ number changes, the file’s inode is marked as a candidate for refresh. When the next FOP comes on such an inode, a refresh is triggered to update the readables during which a heal is launched (if the AFR xattrs indicate that a heal is needed, that is). This heal happens in the background, meaning it does not block the actual FOP which will continue as usual post the refresh. Specific client-side heals can be turned off by disabling the 3 corresponding volume options:

```{ .text .no-copy }
cluster.metadata-self-heal
cluster.data-self-heal
cluster.entry-self-heal
```

The number of client-side heals that happen in the background can be tuned via the following volume options:

```{ .text .no-copy }
background-self-heal-count
heal-wait-queue-length
```

See the `gluster volume set help` for more information on all the above options.

**_Name heal_**: Name heal is just healing of the file/directory name when it is accessed. For example, say a file is created and written to when a brick is down and all the 3 client side heals are disabled. When the brick comes up and the next I/O comes on it, the file name is created on it as a part of lookup. Its contents/metadata are not healed though. Name heal cannot be disabled. It is there to ensure that the namespace is consistent on all bricks as soon as the file is accessed.

### ii) By the self-heal daemon.

There is a self-heal daemon process (glutershd) that runs on every node of the trusted storage pool. It is a light weight client process consisting mainly of AFR ant the protocol/client translators. It can talk to all bricks of all the replicate volume(s) of the pool. It periodically crawls (every 10 minutes by default; tunable via the `heal-timeout` volume option) the list of files that need heal and does their healing. As you can see, client side heal is done upon file access but glustershd processes the heal backlog pro-actively.

### Index heal:

But how does glustershd know which files it needs to heal? Where does it get the list from? So in part-1, while we saw the five phases of the AFR write transaction, we left out one detail:

- In the pre-op phase, in addition to marking the dirty xattr, each brick also stores the gfid string of the file inside its `.glusterfs/indices/dirty` directory.
- Likewise, in the post-op phase, it removes the gfid string from its `.glusterfs/indices/dirty` If addition, if the write failed on some brick, the good bricks will stores the gfid string inside the `.glusterfs/indices/xattrop` directory.

Thus when no I/O is happening on a file and you still find its gfid inside `.glusterfs/indices/dirty` of a particular brick, it means the brick went down before the post-op phase. If you find the gfid inside `.glusterfs/indices/xattrop`, it means the write failed on some other brick and this brick has captured it.

The glustershd simply reads the list of entries inside `.glusterfs/indices/*` and triggers heal on them. This is referred to as **index heal**. While this happens automcatically every heal-timeout seconds, we can also manaully trigger it via the CLI using `gluster volume heal $VOLNAME` .

### Full heal:

A full heal, triggered from the CLI with `gluster volume heal $VOLNAME full`, does just what the name implies. It does not process a particular list of entries like index heal, but crawls the whole gluster filesystem beginning with root, examines if files have non zero afr xattrs and triggers heal on them.

### Of missing xattrs and split-brains:

You might now realise how AFR pretty much relies on its xattr values of a given file- from using it to find the good copies to serve a read to finding out the source and sink bricks to heal the file. But what if there is inconsistency in data/metadata of a file and

(a) there are zero/ no AFR xattrs (or)

(b) if the xattrs all blame each other (i.e. no good copy=>split-brain)?

For (a), AFR uses heuristics like picking a local (to that specfic glustershd process) brick, picking the bigger file, picking the file with latest ctime etc. and then does the heal.

For (b) you need to resort to using the gluster split-brain resolution CLI or setting the favorite-child-policy volume option to choose a good copy and trigger the heal.

repository: gluster/glusterdocs

branch_name: refs/heads/main

gha_language: HTML

repo_name: gluster/glusterdocs


text:
# **Traffic Sign Recognition**

## Writeup

**Traffic Sign Recognition Project**

The goals of this project are the following:
* Load the data set
* Explore, summarize and visualize the data set
* Design, train and test a model architecture
* Use the model to make predictions on new images
* Analyze the softmax probabilities of the new images
* Summarize the results with a written report

[//]: # (Image References)

[image1]: ./output_images/frequency_per_class.png "Visualization"
[image2]: ./web_data/class_3.jpg "Traffic Sign 1"
[image3]: ./web_data/class_4.jpg "Traffic Sign 2"
[image4]: ./web_data/class_17.jpg "Traffic Sign 3"
[image5]: ./web_data/class_25.jpg "Traffic Sign 4"
[image6]: ./web_data/class_33.jpg "Traffic Sign 5"

## Rubric Points

### Writeup / README

You're reading it!

### Data Set Summary & Exploration

#### 1. Provide a basic summary of the data set.

I used the numpy library to calculate summary statistics of the traffic
signs data set:

* The size of training set is 34799 images,
* The size of the validation set is 4410 images,
* The size of test set is 12630 images,
* The shape of a traffic sign image is 32x32 pixels
* The number of unique classes/labels in the data set is 43

#### 2. Include an exploratory visualization of the dataset.

Here is an exploratory visualization of the data set. It is a bar chart showing how the data is distributed per class.

![alt text][image1]

### Design and Test a Model Architecture

#### 1. Describe how you preprocessed the image data.

As a first step, I decided to convert the images to grayscale because it reduce the number of parameters that I have to train and for this kind of project is not absolutely necessary.

As a second step, I normalized the image data using this equation `(pixel - 128)/ 128`, this is a quick way to approximately normalize the data, the normalization helps to optimize the learning process, helping the optimization to be faster.

As a last step I reshape the data, getting it ready for the tranning process, this is to get the arrays in shape for the convolutions in the CNN.

It would be a natural next step to generate more data this due to the imbalanced number of images per class, but in this project I got a great result without doing it.

#### 2. Model.

My final model consisted of the following layers:

| Layer         		|     Description	        					|
|:---------------------:|:---------------------------------------------:|
| Input         		| 32x32x1 grayscaled image   							|
| Convolution 5x5     	| 1x1 stride, valid padding, outputs 28x28x6 	|
| RELU					|												|
| Max pooling 2x2      	| 2x2 stride,  outputs 14x14x6 				|
| Convolution 5x5     	| 1x1 stride, valid padding, outputs 10x10x16 	|
| RELU					|												|
| Max pooling 2x2      	| 2x2 stride,  outputs 5x5x16 				|
| Flatten				| Output 1x400				|
| Fully connected		| Output 1x120        									|
| RELU					|												|
| Dropout				|
| Fully connected		| Output 1x84        									|
| RELU					|												|
| Dropout				|
| Fully connected		| Output 1x43        									|
| Softmax				|         									|

#### 3. Training the algorithm

To train the model, I used an **adam optimizer**, a **batch size** of 128
and 30 **epochs** and use this hyperparameters:

* Learning Rate: 0.001
* Keep Probability: 0.6

#### 4. Finding a 95% accuracy in the validation set.

My final model results were:
* training set accuracy of: 99.8%
* validation set accuracy of: 95.2%
* test set accuracy of: 93.4%

First I use the LeNet architecture in the same way we used it in the MNIST dataset, but I notice a 89% in validation accuracy, but always noticing on average a 10% better behavior in the training data than in the test data, then I sense that there is overfitting in my neural network, so I used dropout in the two fully connected layers during training with a 0.6 keep probability, this help a lot in the final outcome.

I tuned the number of epochs because I was still getting better results with more epochs, and leave it in a good accuracy output.

### Testing the Model on New Images

#### 1. Another german traffic signs.

Here are five German traffic signs that I found on the web:

![alt text][image2] ![alt text][image3] ![alt text][image4]
![alt text][image5] ![alt text][image6]

#### 2. Discuss the model's predictions on these new traffic signs.

Here are the results of the prediction:

| Image			        |     Prediction	        					|
|:---------------------:|:---------------------------------------------:|
| Speed limit 60km/h  	| Speed limit 60km/h   									|
| Speed limit 70km/h    | Speed limit 70km/h 										|
| No entry				| No entry											|
| Road work	      		| Road work					 				|
| Turn right ahead		| Ahead only      							|

The model was able to correctly guess 4 of the 5 traffic signs, which gives an accuracy of 80%. This compares favorably to the accuracy on the test set of 93.9% taking in account that the sample is only of 5 images, but that error is an important error for the people in the self driving car, because it can cause an accident.

#### 3. Certain of the model

For the first image, the model is sure that this is a 60 km/h limit sign (probability of 0.78), and the image does contain a 60 km/h limit sign. The top five soft max probabilities were

| Probability         	|     Prediction	        					|
|:---------------------:|:---------------------------------------------:|
| .78         			| Speed limit 60km/h   									|
| .17     				| Speed limit 80km/h 										|
| .04					| Speed limit 50km/h											|
| 1x10^-5^    			| Speed limit 100km/h					 				|
| 1x10^-11^			    | Wild animals crossing      							|

But for the last image where I got the false prediction, the model output a Ahead only sign, and the image really contain a Turn right ahead sign, throwing the next softmax probabilities:

| Probability         	|     Prediction	        					|
|:---------------------:|:---------------------------------------------:|
| .33         			| Ahead only 									|
| .28     				| Turn left ahead										|
| .12					| Beware of ice/snow											|
| .09  		  			| Turn right ahead					 				|
| .04				    | Children crossing      							|

Here we can conclude a lot of things, first the maximum probability is 33%, comparing with the first image it implies that the model wasn't so sure about a prediction, second in the sum 3 most likely predictions of the first image I got a 99% of certainty, but in the sum of the 5 most likely predictions in the last image I only got 86%, that's another sign that the model wasn't so sure, third it's important to understand that the amount of traning images for the turn right ahead sign is almost the half of the most probably one (ahead only sign), where I can get a reason for the bad prediction, but the second most probably prediction has less training images than the ground truth.


text:
### ChangeLog

###### v0.0.13 [2019.08.03|20:50~1]

* add `noProxy` flag & `start:noproxy` task
* add coverage remove on clear
* add `NO_MOCKS` command line flag, add `NO_WEBPACK_MOCKS` flag with js
* add `tinyServer` for serve:compiled task
* add `eslint@7` as linter
  * add `.eslintrc.json` & `.eslintignore` as configs for eslint
* add ability to use `css` with `scss`
* add `rouble` `font` as example
  * add examples of use rouble font icon
* add `__*` mask for `.gitignore`
* add `env.js` to define environment contexts (now prod is deployed to `spa/` folder)
* fix `autoprefixer` use with `postcss-loader`
* fix `webfont` use with webpack
* add wrapper `wrapLinkWithWebpackContextPrefix` for correct fetch paths in different environments
* add wrapper `wrapLinkToLocalApiMocker` for fix mocked data url on different environments
* add use of app `VERSION`
* update node packages: `typescript`, `postcss-loader`, `node-sass`, `mocker-api`, `mocha`, `husky`
* add `postcss`, `eslint`, `eslint-plugin-optimize-regex`, `eslint-plugin-promise`, `eslint-plugin-sonarjs`, `@typescript-eslint/eslint-plugin`, `@typescript-eslint/parser`
* add appropriate tasks to package.json
* add `package-lock.json` versioning

[tested on<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### v0.0.12 [2019.08.03|20:50~1]

* add `typescript` language supporting
* add examples for inclusion of ts into js & back
* tests for `typescript`
* types for `mocha`

###### v0.0.11 [2019.08.02|20:45~1.5]

* add ability of `mocks` writing (for independent `api` development)

###### v0.0.10 [2019.08.02|17:05~2]

* add `async`|`await` support for javascript with `babel` transpiler

###### v0.0.9 [2019.08.02|13:05~4]

* add `sinon` for spies, stubs, fakes
* add simple examples for sinon tests
* remove `prepush` git hook testing

###### v0.0.8 [2019.07.30|16:40~.5]

* add `chai` for asserts library

###### v0.0.7 [2019.07.30|8:40~.5]

* add `husky` package for precommit & prepush git hooks

###### v0.0.6 [2019.07.29|23:24~.5]

* add `ROADMAP.md`
* fixes for `README.md`

###### v0.0.5 [2019.07.29|22:52~4]

* add `istanbul-instrumenter-loader` for coverage reporting
* add preventing coverage generation while watching tests
* minor refactor tests tasks
* add correct tests use (see specs.js)

###### v0.0.4 [2019.07.29|14:52~16?]

* add karma testing engine
* add mocha framework for karma testing
* split tests into dynamically connectable parts (*.spec.js)
* add watch mode for tests monitoring (and for sources changing)
* add sourcemaps for tests
* add chrome launcher for karma

###### v0.0.3 [2019.07.13|2:12~3]

* add images processing (serve && [likely for svg] copy)
* fix for images paths from css (& file-loader is as dep for url-loader)
* fetch example
* copy for jsons

###### v0.0.2 [2019.07.09|5:47~2.5]

Done:
* refactoring syntax of webpack modes configurations
* add stats analyzation
* add babel preloader for es6 (with preset-env && plugin-proposal-object-rest-spread)
* sourcemaps for js, HMR while dev mode
* uglification && remove comments for production js-bundle
* make vendors.js out of 'node_modules' imports
* add scss serving, HMR, source maps
* add postcss autoprefixer + browserlist config
* move index.css into styles.css

all is still good for development && production mode (resources HMR (except html, not very actual) in development, building stuff correctly while production mode)

###### v0.0.1
Initial commit

repository: blindlybright/bb-webpack-frontend

gha_language: JavaScript

repo_name: blindlybright/bb-webpack-frontend


text:
# JSONScript
A compiler written in JavaScript to compile data in JSON Files and run it as code,
Inside the JSON File keywords are recognized by the compiler and run a certain function,
Inside the file there is classes and in each class there are lines that are run one at a time by the compiler.

See the file 'Walk Through.json' for more information or Run the complete example that I have provided.


text:
# Installing Tracker

```bash
pip3 install dti-tracker
```

## From source

We recommend that you either use virtualenv or Docker.

```bash
# Install Tracker Dependencies
sudo apt update
sudo apt install git make python3-pip

# Install Tracker
git clone https://github.com/dti-research/tracker
cd tracker
sudo -H pip3 install -e .

# Enable bash autocompletion for Tracker on boot
echo ""  >> ~/.bashrc
echo "# Bash autocompletion for Tracker"  >> ~/.bashrc
echo 'eval "$(_TRACKER_COMPLETE=source tracker)"'  >> ~/.bashrc
source ~/.bashrc
```

### For contributors

Install developer dependencies

```bash
cd tracker
sudo -H pip3 install -r requirements-dev.txt
```
After that install the pre-commit hooks

```bash
pre-commit install
```

# Installing System Dependencies

### Docker

Refer to the official install guide: https://docs.docker.com/install/linux/docker-ce/ubuntu/

### NVIDIA Drivers and NVIDIA Docker (Optional, required for GPU)

```bash
# Add PPA
sudo add-apt-repository ppa:graphics-drivers
sudo apt update

# Install the driver that fits your GPU
sudo apt install nvidia-driver-XXX

# Verify installation after reboot
nvidia-smi
```

With the graphics card driver installed you must proceed to install NVIDIA docker: https://github.com/NVIDIA/nvidia-docker

repository: dti-research/tracker

repo_name: dti-research/tracker


text:
# The Tengo Language

[![GoDoc](https://godoc.org/github.com/d5/tengo?status.svg)](https://godoc.org/github.com/d5/tengo/script)
[![Go Report Card](https://goreportcard.com/badge/github.com/d5/tengo)](https://goreportcard.com/report/github.com/d5/tengo)
[![Build Status](https://travis-ci.org/d5/tengo.svg?branch=master)](https://travis-ci.org/d5/tengo)
[![](https://img.shields.io/badge/Support%20Tengo-%241-brightgreen.svg)](https://www.patreon.com/tengolang)

**Tengo is a small, dynamic, fast, secure script language for Go.**

Tengo is **[fast](#benchmark)** and secure because it's compiled/executed as bytecode on stack-based VM that's written in native Go.

```golang
/* The Tengo Language */

each := func(seq, fn) {
    for x in seq { fn(x) }
}

sum := func(init, seq) {
    each(seq, func(x) { init += x })
    return init
}

n := sum(0, [1, 2, 3])   // == 6
s := sum("", [1, 2, 3])  // == "123"
```

> Run this code in the [Playground](https://tengolang.com/?s=d01cf9ed81daba939e26618530eb171f7397d9c9)

## Features

- Simple and highly readable [Syntax](https://github.com/d5/tengo/blob/master/docs/tutorial.md)
  - Dynamic typing with type coercion
  - Higher-order functions and closures
  - Immutable values
  - Garbage collection
- [Securely Embeddable](https://github.com/d5/tengo/blob/master/docs/interoperability.md) and [Extensible](https://github.com/d5/tengo/blob/master/docs/objects.md)
- Compiler/runtime written in native Go _(no external deps or cgo)_
- Executable as a [standalone](https://github.com/d5/tengo/blob/master/docs/tengo-cli.md) language / REPL

## Benchmark

| | fib(35) | fibt(35) |  Type  |
| :--- |    ---: |     ---: |  :---: |
| Go | `58ms` | `4ms` | Go (native) |
| [**Tengo**](https://github.com/d5/tengo) | `4,334ms` | `5ms` | VM on Go |
| Lua | `1,740ms` | `3ms` | Lua (native) |
| [go-lua](https://github.com/Shopify/go-lua) | `5,229ms` | `5ms` | Lua VM on Go |
| [GopherLua](https://github.com/yuin/gopher-lua) | `5,486ms` | `5ms` | Lua VM on Go |
| Python | `3,116ms` | `27ms` | Python (native) |
| [starlark-go](https://github.com/google/starlark-go) | `15,414ms` | `5ms` | Python-like Interpreter on Go |
| [gpython](https://github.com/go-python/gpython) | `17,754ms` | `6ms` | Python Interpreter on Go |
| [goja](https://github.com/dop251/goja) | `6,843ms` | `6ms` | JS VM on Go |
| [otto](https://github.com/robertkrimen/otto) | `86,542ms` | `13ms` | JS Interpreter on Go |
| [Anko](https://github.com/mattn/anko) | `98,962ms` | `26ms` | Interpreter on Go |

_* [fib(35)](https://github.com/d5/tengobench/blob/master/code/fib.tengo): Fibonacci(35)_
_* [fibt(35)](https://github.com/d5/tengobench/blob/master/code/fibtc.tengo): [tail-call](https://en.wikipedia.org/wiki/Tail_call) version of Fibonacci(35)_
_* **Go** does not read the source code from file, while all other cases do_
_* See [here](https://github.com/d5/tengobench) for commands/codes used_

## References

- [Language Syntax](https://github.com/d5/tengo/blob/master/docs/tutorial.md)
- [Tengo Objects](https://github.com/d5/tengo/blob/master/docs/objects.md)
- [Runtime Types](https://github.com/d5/tengo/blob/master/docs/runtime-types.md)
- [Builtin Functions](https://github.com/d5/tengo/blob/master/docs/builtins.md)
- [Interoperability](https://github.com/d5/tengo/blob/master/docs/interoperability.md)
- [Tengo CLI](https://github.com/d5/tengo/blob/master/docs/tengo-cli.md)
- [Standard Library](https://github.com/d5/tengo/blob/master/docs/stdlib.md) _(experimental)_

## Roadmap

### v0. _(Current)_

Things are experimental, and, the focus is on the **core language features**, **stability**, **basic interoperability**, and the **performance optimization**.

### [v1. Tengo as a Script Language](https://github.com/d5/tengo/labels/v1.0)

This will be the first _versioned_ release, and, the main goal for v1 is to make Tengo as a _fast_ embeddable script language for Go, which means Tengo will be comparable to other Go-based script languages such as [Starlark](https://github.com/google/starlark-go), [Lua](https://github.com/Shopify/go-lua) [VM](https://github.com/yuin/gopher-lua)s, and [other](https://github.com/robertkrimen/otto) [interpreter](https://github.com/mattn/anko)s.

- Interoperability with Go code
- Sandbox environment
- More language features

### v2. Tengo as a Standalone Language

- Language-level concurrency support
- Tengo Standard Libraries
- Native executables compilation
- More language features

repository: shammishailaj/tengo

repo_name: shammishailaj/tengo


text:
# Stipa californica Merr. & Burtt Davy SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Univ. Calif. Publ. Bot. 1:61. 1902

#### Original name
null

### Remarks
null

repository: mdoering/backbone

path: /life/Plantae/Magnoliophyta/Liliopsida/Poales/Poaceae/Stipa/Stipa occidentalis/ Syn. Stipa californica/README.md

repo_name: mdoering/backbone


text:
---
layout: post
title: "ExtenXLS Working With Charts"
excerpt: "Examples and code for displaying images in posts."
image: "http://farm9.staticflickr.com/8426/7758832526_cc8f681e48_c.jpg"
tags:
  - Spreadsheet
  - charts
  - test
---

## Working With Charts

With ChartHandle, you can add cells to a chart, change a chart’s title, axis labels, and copy charts between other worksheets and workbooks.

Begin by retrieving a ChartHandle from an existing chart in a worksheet. To retrieve a ChartHandle, use the ChartHandle WorkSheetHandle.getChart(String chartname) method.

```java
try{
       ChartHandle performancechart = mysheet.getChart('Performance');
}catch(ChartNotFoundException e){

  // oops!
};
```

With a ChartHandle, you can add cells to the chart using the ChartHandle.changeSeriesRange(String originalrange, String newrange) method.

```java
if(performancechart.changeSeriesRange('Sheet1!C23:E23', 'Sheet1!C23:G23')){
System.out.println('Successfully Added Columns F and G to Series Range');
}
```

To change the chart title, use the Charthandle.setTitle(String title) method.

```java
performancechart.setChartTitle('Athletic Performance Results');
```

To change the axis labels, or other text labels in the chart use the ChartHandle.changeTextValue(String originaltext, String newtext) method.

```java
if(ct.changeTextValue('50 Meter Dash Times", "High Jump    Distance')){
System.out.println('Successfully Changed Categories Label');
}
```

To copy a chart into another sheet or workbook, use:

```java
book.copyChartToSheet("New Chart Title", "Sheet3");
```


text:
# python-keylogger
This is a simple python script that runs in the background and records the keystrokes of that system

Warning: This is just for educational purpose do not use it for any unethical means, *i will not be responsible for your actions*

Steps:

1.Create a folder somewhere in the drive and download the keylogger.pyw there and execute it

2.It will run silently in the background and record the keystrokes of the system

3.A text file named key_logs will be created in the same folder containing the key logs

*You will need pynput library for this to work*


text:
## Authentication
In order to use the Rapid7 InsightIDR integration, you need to get an API key with Read/Write privileges.

### Generate an API Key
1. Log in to Rapid7 InsightIDR.
2. Click **Settings > API Keys**.
3. Generate a new API key with Read/Write privileges.

repository: demisto/content

gha_language: Python

repo_name: demisto/content


text:
---
title: LoRa App Server
menu:
    main:
        parent: overview
        weight: 1
---

## LoRa App Server

LoRa App Server is an open-source LoRaWAN application-server, part of the
[LoRa Server](https://docs.loraserver.io/) project. It is responsible
for the node "inventory" part of a LoRaWAN infrastructure, handling of received
application payloads and the downlink application payload queue. It comes
with a web-interface and API (RESTful JSON and gRPC) and supports authorization
by using JWT tokens (optional). Received payloads are published over MQTT
and payloads can be enqueued by using MQTT or the API.

### Screenshots

![applications](/lora-app-server/img/web_applications.png)
![nodes](/lora-app-server/img/web_nodes.png)
![node details](/lora-app-server/img/web_node_details.png)
![swagger api](/lora-app-server/img/swagger.png)

repository: Frankz/lora-app-server

repo_name: Frankz/lora-app-server


text:
## End-To-End Memory Networks for Question Answering on Tables

For more details see [TableQA: Question Answering on Tabular Data](https://arxiv.org/abs/1705.06504)

## Demo

* The demo is deployed at [https://svakulenko.ai.wu.ac.at/tableqa](https://svakulenko.ai.wu.ac.at/tableqa)

* Command to run it locally:
```
python -m demo.qa
```

## Requirements
* Python 2.7
* Numpy, Flask (only for web-based demo)
* fastText (+cython)

can be installed via pip:
```
$ sudo pip install -r requirements.txt
```

## Pre-requisite

1. Provide the correct path to the pre-trained fastText model (clone fastText Github repository and run ./word-vector-example.sh to train a model on the English Wikipedia)
download pre-trained model
('make clean' and 'unzip enwik9.zip' before)

* To train and evaluate the model use `tableQA_runner.py`

## Experiments

### Data

Synthetic data based on a real table (limiting vocabulary size and producing more training examples)

Cell-based formatting

Dictionary: 65

1 Row1 LAU2_NAME Allhaming

2 Row1 YEAR 2002

3 Row1 INTERNAL_MIG_IMMIGRATION 2

4 Row1 INTERNATIONAL_MIG_IMMIGRATION 4

5 Row2 LAU2_NAME Geretsberg

6 Row2 YEAR 2005

7 Row2 INTERNAL_MIG_IMMIGRATION 3

8 Row2 INTERNATIONAL_MIG_IMMIGRATION 5

9 What is the INTERNAL_MIG_IMMIGRATION for Geretsberg?	3	5 7

### Settings

* linear start

* 2 question templates

* BOW

### Results

1. Simple key

What is the EMIGRATION_TOTAL for Helfenberg?	2	13 20

IMMIGRATION_TOTAL in Burgkirchen?	4	3 7

Number of training examples 5949

20 + 9 epochs

train error: 0 | val error: 0

2. Complex key

What is the INTERNAL_MIG_IMMIGRATION for Grieskirchen in 2004?	4	13 14 15

IMMIGRATION_TOTAL in Burgkirchen for 2002?	10240	23 24 27

Number of training examples 18953

20 + 68 epochs

train error: 0 | val error: 0

## Acknowledgments

* Based on [Vinh Khuc: MemN2N for babi tasks](https://github.com/vinhkhuc/MemN2N-babi-python).
* Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus,
  [End-To-End Memory Networks](http://arxiv.org/abs/1503.08895)

## Related Work
* [YerevaNN: Dynamic-memory-networks-in-Theano](https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano)
* Kumar et al. [Dynamic memory networks](http://arxiv.org/abs/1506.07285)

## Citation

```bibtex
@inproceedings{DBLP:conf/i-semantics/VakulenkoS17,
  author    = {Svitlana Vakulenko and
               Vadim Savenkov},
  title     = {TableQA: Question Answering on Tabular Data},
  booktitle = {Proceedings of the Posters and Demos Track of the 13th International
               Conference on Semantic Systems - SEMANTiCS2017 co-located with the
               13th International Conference on Semantic Systems (SEMANTiCS 2017),
               Amsterdam, The Netherlands, September 11-14, 2017},
  year      = {2017}
}
```


text:
---
slug: /en/sql-reference/aggregate-functions/reference/varpop
sidebar_position: 32
---

# varPop(x)

Calculates the amount `Σ((x - x̅)^2) / n`, where `n` is the sample size and `x̅`is the average value of `x`.

In other words, dispersion for a set of values. Returns `Float64`.

:::note
This function uses a numerically unstable algorithm. If you need [numerical stability](https://en.wikipedia.org/wiki/Numerical_stability) in calculations, use the `varPopStable` function. It works slower but provides a lower computational error.
:::

repository: ClickHouse/ClickHouse

repo_name: ClickHouse/ClickHouse


text:
---
layout: post
title: commonCharacters
date: 2019-07-23
comments: true
categories: [Study, algorithm]
tags: [Javascript]
excerpt: Write a function f(a, b) which takes two strings as arguments and returns a string containing the characters found in both strings (without duplication), in the order that they appeared in a.
hidden: true
---

### 문제

Write a function `f(a, b)` which takes two strings as arguments and returns a string containing the characters found in both strings (without duplication), in the order that they appeared in `a`. Remember to skip spaces and characters you have already encountered!

Example: commonCharacters('acexivou', 'aegihobu')
Returns: 'aeiou'

Extra credit: Extend your function to handle more than two input strings.

### 풀이

```javascript
var commonCharacters = function(string1, string2) {
  let args = Array.from(arguments);
  let commonCharacterString = "";
  const isIncluded = str => {
    for (let i = 1; i < args.length; i++) {
      if (!args[i].includes(str)) {
        return false;
      }
    }
    return true;
  };
  for (let i = 0; i < args[0].length; i++) {
    if (isIncluded(args[0][i]) && !commonCharacterString.includes(args[0][i])) {
      commonCharacterString += args[0][i];
    }
  }
  return commonCharacterString;
};
```

gha_language: CSS


text:
Docker Containers
=================

These Docker containers should make it possible to build runX in
any of the available environments on any system that supports
running Docker. They are organized by distro and tagged with
the version of that distro. They are available from the GitLab
Container Registry under the Xen project at the [registry] and
can be pulled with Docker from the following path:

```
docker pull registry.gitlab.com/lf-edge/runx/DISTRO:VERSION
```

To see the list of available containers run `make` in this
directory. You will have to replace the `/` with a `:` to use
them.

Building runX
-------------

From the top level of the source tree it should be possible to
run the following:

```
./automation/scripts/containerize ./build.sh
```

Which will cause the top level `./build.sh` to execute within the
efault container, which is currently defined as Alpine 4.12. Any
arguments specified to the script will be executed within the container
from the default shell.

There are several environment variables which the containerize script
understands.

- DOCKED_CMD: Whether to use docker or podman for running the containers.
  podman can be used as a regular user (rootless podman), but for that
  to work, /etc/subuid and /etc/subgid needs to containe the proper
  entries, for such user.
  docker is the default, for running with podman, do:

  ```
  DOCKER_CMD=podman ./automation/scripts/containerize ./build.sh
  ```

- CONTAINER: This overrides the container to use. For Alpine 3.12
  aarch64, use:

  ```
  CONTAINER=alpine ./automation/scripts/containerize ./build.sh
  ```

- CONTAINER_PATH: This overrides the path that will be available under the
  `/build` directory in the container, which is the default path.

  ```
  CONTAINER_PATH=/some/other/path ./automation/scripts/containerize ls
  ```

- CONTAINER_ARGS: Allows you to pass extra arguments to Docker
  when starting the container.

- CONTAINER_UID0: This specifies whether root is used inside the container.

- CONTAINER_NO_PULL: If set to 1, the script will not pull from docker hub.
  This is useful when testing container locally.

- XEN_CONFIG_EXPERT: If this is defined in your shell it will be
  automatically passed through to the container.

If your docker host has Linux kernel > 4.11, and you want to use containers
that run old glibc (for example, CentOS 6 or SLES11SP4), you may need to add

```
vsyscall=emulate
```

to the host kernel command line. That enables a legacy interface that is used
by old glibc.

Building a container
--------------------

There is a makefile to make this process easier. You should be
able to run `make DISTRO/VERSION` to have Docker build the container
for you. If you define the `PUSH` environment variable when running the
former `make` command, it will push the container to the [registry] if
you have access to do so and have your Docker logged into the registry.

To login you must run `docker login registry.gitlab.com`. For more
information see the [registry help].

[registry]: https://gitlab.com/lf-edge/runx/container_registry
[registry help]: https://gitlab.com/help/user/project/container_registry

repository: lf-edge/runx

repo_name: lf-edge/runx


text:
# Frontend meeting 24 March 2021

Present: Aapeli, Chris, Darren, Lucas

## Soft beta milestone progress

**Should be able to release soon - hopefully a week**

- Update references to use the new API in references.proto - in review

- User's age is not checked on sign up backend bug - backend issue, move to beta

- Group chat view doesn't scroll to bottom when you send a new message bug frontend -
Josh is working on it - but we can move to beta

- EditUserLocationMap can't edit on mobile bug frontend mobile - in review

- Put the logo in and update the palette - in review

- Host request view doesn't give request info - awaiting merge

- Host Requests - being worked on / review

- Change login/signup picture - Lucas will bug the design team

- Requests Beta UX - Lucas will clarify the issue (and do it?)

### New issues to make

- Hide communities stuff in the soft beta map

- Events page placeholder

- Signup map submits when pressing enter - needs to not be in <form>

## Beta maps ui discussion

- Think about how to split up map

  - Not easy, just one person should start it initially

- Router state can set initial filter (because search may need to be within a community)

- Only user search has community/area filter so far in backend

### Recommended approach

Display everything in a layer like the soft beta map does (big geojson), and when a search is performed, pass the result ids to the layer as a filter.

## Other discussion

- Padding between text fields in login/signup

- We should move radius editing from map to a slider in profile settings (just have 500m default or something in signup)

- Transitions are nice

- We should use milestones more, it's nice

repository: pawlean/couchers

branch_name: refs/heads/develop

repo_name: pawlean/couchers


text:
# iriscli upgrade query-signals

## Description

Query the current information of signals.

## Usage

```
iriscli upgrade query-signals
```

Print help messages:

```
iriscli upgrade query-signals --help
```

## Example

Query the current information of signals.

```
iriscli upgrade query-signals
```

```
[
  "fca1pz46rw68wjxklxp7ptw6wrdvn5hae04jhuhdc5",
  "fca1uvjn440xjgech86avm855v5u5c65dar6gu3ng4"
]
```

repository: Czone-League/irishub

repo_name: Czone-League/irishub


text:
A basic line-oriented chat server built atop [step 4 of the
pony-workshop](https://github.com/aturley/pony-workshop/blob/master/steps/04/main.pony)
used as a learning example.

It features:
- graceful shutdown of the server (triggered by sending `SIGTERM` and `SIGINT`)
- logging which can be controlled via `PONY_LOG_LEVEL` environment variable
- requires clients to provide their names
- clients can send:
    - `/quit` - terminate the client's connection to the server and announces
    that this client has left
    - `/time` - the server sends the current time to the server
    - everything else is sent to all connected clients

## Requirements

- [Pony compiler](https://github.com/ponylang/ponyc/blob/master/README.md#installation)

## Build

    ponyc

## Run

Note: the server listens for connections on `localhost:8989`

    ./main

To run with various log levels:

    PONY_LOG_LEVEL=<log-level> ./main

Where `<log-level>` is one of `fine`, `info`, `warn`, `error`

## Interact

The easiest way to connect to the server is to use `netcat` like so:

    > nc localhost 8989
    welcome! Please enter your name:

license: Unlicense

repository: joncfoo/pony-simple-chat-server

repo_name: joncfoo/pony-simple-chat-server


text:
# Kishmish *in a nutshell*
### *Because everything happens for a raisin*

## Introduction
This is a stripped-down shell implemented in C.

## Functionality and Limitations
Kishmish supports the following features:

- Built-in commands: `ls [al]`, `echo`, `pwd`, `cd`, `quit`, `history`, `pinfo`, `nightswatch`, `setenv`, `unsetenv`, `jobs`, `kjobs`, `overkill`, `fg`, `bg`, `cronjob`, `quit`
- Process management including foreground and background processes and switching between them.
- Piping and redirection
- Signal handling for signals like `SIGINT` and `SIGTSTP`
- A prompt that shows the username, hostname and the current working directory
- Command history and up arrow command recall across sessions
- Appropriate error handling

Kishmish does not support the following yet:

- Glob expansion and tab completion
- Support for quoting and escape sequences
- Aliasing
- User profiles
- Environment variables

and maybe a lot more...

## Make instructions

In the directory where the source files reside, execute:
```shell
$ make
$ ./kishmish
```

To clean up object and binary files, execute:
```shell
$ make clean
```

## File-Wise Code Breakdown

- `shell.c` : Contains the REPL of the shell
- `utils.c` : Contains several utility functions such as those for string processing, `~` expansion, storing and retrieving session history, initialization and cleanup
- `prompt.c` : Contains the function responsible for generating and printing the shell prompt
- `parse.c` : Contains functions that parse and sanitize the entered commands and call appropriate functions
- `external.c` : Contains functions that handle the execution of external programs in the foreground and background
- `exit.c` : Contains functions that implement the `quit` command
- `pwd.c` : Contains functions that implement the `pwd` command
- `cd.c` : Contains functions that implement the `cd` command
- `ls.c` : Contains functions that implement the `ls` command
- `echo.c` : Contains functions that implement the `echo` command
- `pinfo.c` : Contains functions that implement the `pinfo` command
- `history.c` : Contains functions that implement the `history` command
- `nightswatch.c` : Contains functions that implement the `nightswatch` command
- `bg.c` : Contains functions that implement the `bg` command
- `cronjob.c` : Contains functions that implement the `cronjob` command
- `env.c` : Contains functions that implement the `setenv` and `unsetenv` commands
- `fg.c` : Contains functions that implement the `fg` command
- `jobs.c` : Contains functions that implement the `jobs` command
- `kjob.c` : Contains functions that implement the `kjob` command
- `overkill.c` : Contains functions that implement the `overkill` command
- `pipe.c` : Contains function that help in parsing and setting up command pipelines
- `process.c` : Contains utility functions for process management
- `recall.c` : Contains functions that implement up arrow command recall
- `redirection.c` : Contains functions that implement input and output redirection
- `signals.c` : Contains functions that handle signals sent to the shell
- `makefile` : Contains the `make` rules for compiling the files
- `README.md` : The file you are reading now
- `README.pdf` : Another file you could be reading now
- `LICENSE` : This project is open-sourced under an MIT license. Details in this file. WARNING: Legal language ahead.

*A header file accompanies each of the C files listed above*

## Implementation Details

### The UI and the Prompt
- Kishmish indicates that it is waiting for input by printing a prompt that shows the current username, hostname and working directory address.
- The working directory address is shown relative to the *home directory*, which is defined as the directory where the shell executable resides1.

### Input Parsing
- Kishmish accepts *bash-like* commands and is capable of executing a list of semi-colon-separated ones.
- Kishmish is capable of parsing space-separated arguments
- Kishmish tries to handle all reasonable input errors appropriately.

### Command History
- Kishmish stores command history across sessions, capped at a maximum of 20 commands2.
- Command recall on pressing the Up arrow key is supported up to 20 commands3. Kishmish does not currently support editing the recalled command before execution.
- This is achieved by storing history data in a file at the end of this session and reading it at the beginning of the next. During a session, history data is managed internally in an array. This is to reduce the number of file accesses and improve performance.
- The history file, `.kishmish_history.dat`, is a binary file that is created automatically in the home directory, if it doesn't already exist. Deletion of this file is harmless, except the loss of history data from the previous sessions.
- The history data is written to file only when the shell exits normally, that is on receiving an `EOF` signal (Ctrl+D on Linux-based systems) or upon use of the `quit` command.

### Execution of External Programs and Process Management
- Kishmish is capable of launching external programs in the foreground as well as in the background.
- This is achieved by forking the shell process into two, and replacing the child process's program image with that of the desired program.
- Depending on whether the program was invoked in the foreground or in the background, the parent process does or does not wait for the child process to terminate, respectively.
- Invocation in the background is requested by adding an ampersand symbol (`&`) after the complete command. It may or may not be separated by whitespace. Any part of the command after the `&` symbol is ignored.
- Kishmish prints an alert if a background job terminates.
- Kishmish supports switching jobs between foreground and background execution using the `fg`, `bg` commands and the `SIGTSTP` signal (Ctrl+Z) on Linux based systems.
- This is achieved by maintaining an internal list of processes, managing processing groups and manipulating the foreground process group of the attached terminal.

### Piping and Redirection
- Kishmish supports input and output redirection using `<`,  `>` and `>>`.
- This is achieved by duplicating the file descriptors of the standard I/O files.
- Kishmish also supports pipelining of commands using `|`, achieved using the `pipe` syscall.

### Built-in Commands
- All built-in commands listed above behave as expected, mimicking the behaviour of bash.
- The syntax of some peculiar commands is elaborated below:
- - `cronjob -c command -t wait -p time` : Repeatedly run `command` after every `wait` seconds for a total duration of `time` seconds in a subshell in the background. The subshell is however granted access to terminal I/O and runs in the same process group as Kishmish.
- - `nightswatch -n wait [dirty | interrupt]` : This command prints certain system information to the screen every `wait` seconds. The command runs in Kishmish itself, not in a separate subshell, hence it cannot be sent to the background and cannot be interrupted. This is by design, because the command needs to periodically print data to the terminal. It can be exited using the sequence Q+Enter. The information displayed corresponds to the amount of dirty memory in the system cache if `dirty` is selected, or the number of keyboard interrupts per CPU core if `interrupt` is selected.
- - `fg job-number` or `bg job-number` where job number is the job number assigned by the shell, different from the process id
- - for command recall, press the Up arrow the required number of times and press Enter
- - `kjob job-number signal-number` sends signal with `signal-number` to process with `job-number`
- - `overkill` kills all background processes at once
- - `pinfo [pid]` lists information about process with `pid` as the process id. If no process id is specified, it lists information about Kishmish's own process

*Have a grape day!*

___
1: As per requirements

2: As per requirements

3: As per requirements

repository: jiviteshjain/kishmish

repo_name: jiviteshjain/kishmish


text:
Write a program that accepts two words as arguments and concatenates them with a space between them, then prints them.

-----------------
## HINTS

You can access command-line arguments via the global `$argv` array.

The concatenation operator in PHP is `.`

To test your program, you can run it with `php program.php` and two words as arguments, e.x.

```sh
$ php program.php concat world
```

where `program.php` is replaced by the name of your file.

Be aware that concatenation does not automatically add a space between words.

repository: cruxicheiros/phpschool-stringthings

repo_name: cruxicheiros/phpschool-stringthings


text:
Hit Simple
==========

Hit is a reimplementation of some git operations in pure haskell.
Hit Simple is a simplification to give a more user friendly
API and focus on git reading operations

what it does do:

* read loose objects, and packed objects.

Based on the original [hit](https://github.com/vincenthz/hit) repository

repository: Twinside/hit-simple

repo_name: Twinside/hit-simple


text:
#  ![](favicon.png) cbevins/fire-behavior-simulator

[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) ![Status](https://github.com/cbevins/fire-behavior-simulator/actions/workflows/nodejs.yml/badge.svg)

---

*fire-behavior-simulator* is a directed acyclical graph of the variables and equations comprising the *BehavePlus Fire Modeling System* Version 6.  Implemented as a Javascript (ES6) library package, it is designed for creating wildland fire behavior prediction, simulation, training, and management applications that operate within any modern web browser or the Node.js runtime environment.

*fire-behavior-simulator* is a fully independent package and requires no other modules to be installed.

---

If you are reading this **README** from the *fire-behavior-simulator* **GitHub repository**, you can access the docs directly from the table of contents below.  Otherwise, you will need access the docs from the [repository](https://github.com/cbevins/fire-behavior-simulator).

## ![](favicon.png) Contents
- [01 Summary](./docs/01_Summary.md)
- [02 Installation](./docs/02_Installation.md)
- [03 Capabilities](./docs/03_Capabilities.md)
- [04 Basic Usage](./docs/04_BasicUsage.md)
- [05 Client API](./docs/05_ClientAPI.md)
- [06 Example Use with Node.js](./docs/06_ExampleUseWithNodeJs.md)
- [07 Live Web Browser Examples](./docs/07_LiveWebBrowserExamples.md)
- [08 Configuration](./docs/08_Configuration.md)
- [09 Module Linkages](./docs/09_ModuleLinkages.md)
- [10 Design and Implementation](./docs/10_DesignImplementation.md)
- [11 Performance Testing](./docs/11_PerformanceTesting.md)
- [12 Author](./docs/12_Author.md)
- [13 Release Notes](./docs/13_RELEASE_NOTES.md)
- [14 Variable Names](./docs/14_VariableNames.md)


text:
# Image Classification
## A Simple Example
The first step is to prepare your data. Here we use the [MNIST
dataset](https://keras.io/datasets/#mnist-database-of-handwritten-digits) as an
example.

```python
from tensorflow.keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
print(x_train.shape) # (60000, 28, 28)
print(y_train.shape) # (60000,)
print(y_train[:3]) # array([7, 2, 1], dtype=uint8)
```

The second step is to run the [ImageClassifier](/image_classifier).

```python
import autokeras as ak

# Initialize the image classifier.
clf = ak.ImageClassifier(max_trials=10) # It tries 10 different models.
# Feed the image classifier with training data.
clf.fit(x_train, y_train)
# Predict with the best model.
predicted_y = clf.predict(x_test)
# Evaluate the best model with testing data.
print(clf.evaluate(x_test, y_test))
```

## Validation Data
By default, AutoKeras use the last 20% of training data as validation data.
As shown in the example below, you can use `validation_split` to specify the percentage.

```python
clf.fit(x_train,
        y_train,
        # Split the training data and use the last 15% as validation data.
        validation_split=0.15)
```

You can also use your own validation set
instead of splitting it from the training data with `validation_data`.

```python
split = 50000
x_val = x_train[split:]
y_val = y_train[split:]
x_train = x_train[:split]
y_train = y_train[:split]
clf.fit(x_train,
        y_train,
        # Use your own validation set.
        validation_data=(x_val, y_val))
```

## Customized Search Space
For advanced users, you may customize your search space by using
[AutoModel](/auto_model/#automodel-class) instead of
[ImageClassifier](/image_classifier). You can configure the
[ImageBlock](/block/#imageblock-class) for some high-level configurations,
e.g., `block_type` for the type of neural network to search, `normalize` for whether to do
data normalization, `augment` for whether to do data augmentation. You can also
do not specify these arguments, which would leave the different choices to be
tuned automatically.
See the following example for detail.

```python
import autokeras as ak

input_node = ak.ImageInput()
output_node = ak.ImageBlock(
    # Only search ResNet architectures.
    block_type='resnet',
    # Normalize the dataset.
    normalize=True,
    # Do not do data augmentation.
    augment=False)(input_node)
output_node = ak.ClassificationHead()(output_node)
clf = ak.AutoModel(inputs=input_node, outputs=output_node, max_trials=10)
clf.fit(x_train, y_train)
```
The usage of [AutoModel](/auto_model/#automodel-class) is similar to the
[functional API](https://www.tensorflow.org/guide/keras/functional) of Keras.
Basically, you are building a graph, whose edges are blocks and the nodes are intermediate outputs of blocks.
To add an edge from `input_node` to `output_node` with
`output_node = ak.[some_block]([block_args])(input_node)`.

You can even also use more fine grained blocks to customize the search space even
further. See the following example.

```python
import autokeras as ak

input_node = ak.ImageInput()
output_node = ak.Normalization()(input_node)
output_node = ak.ImageAugmentation(percentage=0.3)(output_node)
output_node = ak.ResNetBlock(version='v2')(output_node)
output_node = ak.ClassificationHead()(output_node)
clf = ak.AutoModel(inputs=input_node, outputs=output_node, max_trials=10)
clf.fit(x_train, y_train)
```

## Data Format
The AutoKeras ImageClassifier is quite flexible for the data format.

For the image, it accepts data formats both with and without the
channel dimension. The images in the MNIST dataset do not have the channel dimension.
Each image is a matrix with shape (28, 28).  AutoKeras also accepts images of three
dimensions with the channel dimension at last, e.g., (32, 32, 3), (28, 28, 1).

For the classification labels, AutoKeras accepts both plain labels, i.e. strings or
integers, and one-hot encoded encoded labels, i.e. vectors of 0s and 1s.

So if you prepare your data in the following way, the ImageClassifier should still
work.

```python
from tensorflow.keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Reshape the images to have the channel dimension.
x_train = x_train.reshape(x_train.shape + (1,))
x_test = x_test.reshape(x_test.shape + (1,))

# One-hot encode the labels.
import numpy as np
eye = np.eye(10)
y_train = eye[y_train]
y_test = eye[y_test]

print(x_train.shape) # (60000, 28, 28, 1)
print(y_train.shape) # (60000, 10)
print(y_train[:3])
# array([[0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
#        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]])
```

We also support using [tf.data.Dataset](
https://www.tensorflow.org/api_docs/python/tf/data/Dataset?version=stable) format for
the training data. In this case, the images would have to be 3-dimentional.
The labels have to be one-hot encoded for multi-class
classification to be wrapped into tensorflow Dataset.

```python
import tensorflow as tf
train_set = tf.data.Dataset.from_tensor_slices(((x_train, ), (y_train, )))
test_set = tf.data.Dataset.from_tensor_slices(((x_test, ), (y_test, )))

clf = ak.ImageClassifier(max_trials=10)
# Feed the tensorflow Dataset to the classifier.
clf.fit(train_set)
# Predict with the best model.
predicted_y = clf.predict(test_set)
# Evaluate the best model with testing data.
print(clf.evaluate(test_set))
```

## Reference
[ImageClassifier](/image_classifier),
[AutoModel](/auto_model/#automodel-class),
[ImageBlock](/block/#imageblock-class),
[Normalization](/preprocessor/#normalization-class),
[ImageAugmentation](/preprocessor/#image-augmentation-class),
[ResNetBlock](/block/#resnetblock-class),
[ImageInput](/node/#imageinput-class),
[ClassificationHead](/head/#classificationhead-class).


text:
# Changelog

All notable changes to this project will be documented in this file.

## [0.2.1] - 2020-11-23

### Changed

- Edits in readme and package.json

## [0.2.0] - 2020-11-11

🎊 Initial release

### Added

- Main functions:
  - Get the country name from a given code.
  - Get the code from a given country name.
- Functions to get all the data:
  - Get all country names as array or object.
  - Get all codes as array or object.
- README, CHANGELOG, LICENSE.
- Tests for all the functions.


text:
## Why TED-XML?

[TED](http://ted.europa.eu/) serves tender information in two ways. Notices can be viewed (and downloaded) **in HTML** on their website, and there's their public **[FTP server](ftp://guest:guest@ted.europa.eu/)** where **notice XMLs** can be downloaded in daily/monthly packages. Both approach have advantages and disadvantages:

Feature                      | HTML                         |   | XML
-----------------------------|------------------------------|---|------------------------------
Availability                 | Latest 5 years only          | < | **Full archive of 20+ years**
Crawling                     | Not elegant, harmful         | < | **Packaged for bulk download**
File size1        | ~ 30 KB - 1.3 MB             | < | **~ 7 KB - 200 KB**
Parsing difficulty           | Ugly, non-semantic HTML      | < | **Language independent XML**
Notice completeness          | Not 100% of data2 | < | **Full notice**
Reference to related notices | **All notice of tender**     | > | Only direct parent notice

1 The smallest and largest XML file in the `20171011_2017195.tgz` daily package and their corresponding HTML size on the TED website.

2 E.g. award criteria (AC field) is missing from the "Data" tab for newer notices, but can be found in XMLs.

While [Red Flags](http://www.redflags.eu/) works well with HTML, it seems that in the long run, using XMLs is a better choice.

## TED-XML schemas

There are two XML schemas being currently used by TED<EMAIL_ADDRESS>and [2.0.9](ftp://guest:guest@ted.europa.eu/Resources/XML%20schema%202.0.9/TED_Schema_R209S01.zip). 2.0.9 is the latter one and appeared at some time in 2016, while R2.0.8 can still be found in 2017.

Currently, we **only support schema version 2.0.9** and will extend the parser to other versions later.

## Problems with TED-XML

Though TED-XMLs seem to contain more information than the HTMLs, they are missing a key feature. While TED website has a "Document family" tab for all notices which references all notices of the tender, XMLs **only** contain reference for the **direct parent notice** of the current notice.

The direct parent notice is **not always the previous** notice. Example tender:

Notice ID                                                                       | Document type          | Referenced notice
--------------------------------------------------------------------------------|------------------------|------------------
[409527-2016](http://ted.europa.eu/udl?uri=TED:NOTICE:409527-2016:TEXT:EN:HTML) | contract notice        | -
[418746-2016](http://ted.europa.eu/udl?uri=TED:NOTICE:418746-2016:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[399656-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:399656-2017:TEXT:EN:HTML) | contract award notice  |<PHONE_NUMBER>

The direct parent notice is **not always the first notice** of the tender. Example tender:

Notice ID                                                                       | Document type          | Referenced notice
--------------------------------------------------------------------------------|------------------------|------------------
[346129-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:346129-2017:TEXT:EN:HTML) | contract notice        | -
[353282-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:353282-2017:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[357352-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:357352-2017:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[392136-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:392136-2017:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[401616-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:401616-2017:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[424116-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:424116-2017:TEXT:EN:HTML) | additional information |<PHONE_NUMBER>
[458065-2017](http://ted.europa.eu/udl?uri=TED:NOTICE:458065-2017:TEXT:EN:HTML) | contract award notice  |<PHONE_NUMBER>

As we see, we **cannot determine the tender ID** (first notice ID of the tender) at the time of processing one notice. Though we **can recognize first notices** of tenders: they don't reference any notice.

### Solution

* We have to use daily packages from TED, and we have to import notices in the order of their publication date, from oldest to newest.
* If the current notice doesn't reference any notice, we can save a tender ID. Otherwise, we can fetch the tender ID of the referenced notice from our database.
* If we imported the referenced notice (or its referenced notices transitively) before, we will surely get a tender ID for the current notice, since we flush data into the database after each daily package.
* If we don't have the referenced notice in the database, we can drop the current notice to hide incomplete tenders from the website.

## XML parsing method

The first tool that came to our mind was [JAXB](http://www.oracle.com/technetwork/articles/javase/index-140168.html), because it seemed that it could reduce the work time on XML parsing literally to minutes.

* It can **fill a POJO model** with extracted data.
* There's an official tool (`xjc`, bundled with JDK) which can **generate annotated POJO model** for JAXB from existing XML schema definition. The TED-XML schema is available on TED's FTP server.
* The JAXB parsing itself is just 2 lines of code.

But as we tried JAXB, it turned out that it's not the right tool for this job.

* The `xjc` tool can only generate the POJO for the 2.0.9 schema and **fails for 2.0.8**.
* In dozens of generated files, the generated namespace declaration in `@XmlSchema` and `@XmlElementRef` annotations were incorrect, causing JAXB to throw exception unless we **modify the namespace by hand.**
* In dozens of generated files, possible XML elements were represented by only a **generic list** of `JAXBElement` objects instead of separate fields.
* `xjc` generated 25 `F##_2014` classes, and all of them have more than one field having additional **numbered types**, like `BodyF##`. Since they have similar content, we would want to use them like `Form_2014` or `Body`.
* [XPath](https://www.w3.org/TR/1999/REC-xpath-19991116/) could provide a workaround for some of these problems, but we couldn't make [MOXy](http://www.eclipse.org/eclipselink/documentation/2.4/moxy/advanced_concepts005.htm)'s `@XmlPath` annotation work.

These things led us to **use [Simple XML](http://simple.sourceforge.net/) instead.**

* It also works by filling an annotated POJO model, though there's no tool for generating one.
* It can handle XPath expressions, though not the whole XPath specification.
* It **doesn't care about XML namespaces** at all.
* Handling of `F##_2014` XML elements can be achieved in a **much more elegant way** than in JAXB.
* Extending the model to other TED-XML schema versions seems **much more easier.**

repository: deepdata-ltd/tenderbase

repo_name: deepdata-ltd/tenderbase


text:
## Minimal intro "How to make a test":

1. Files matching pattern test_*.py contain tests.
2. Functions matching pattern test_* are tests.

3. To run the tests in a dials distribution, install dlstbx and run

       py.test

## Advanced stuff

4.  Use asserts, generally no need to create or import other
   assert implementations.
   eg.

   ```python
   def test_assert():
     assert [1,2,3] == [1,2,4]
   ```

   will give you the following output:

   ```python
      def test_assert():
   >    assert [1,2,3] == [1,2,4]
   E    assert [1, 2, 3] == [1, 2, 4]
   E      At index 2 diff: 3 != 4
   E      Use -v to get the full diff
   ```

6. more advanced stuff at https://docs.pytest.org/en/latest/


text:
---
layout: post
title: Just Scratching the Surface of Sydney
date: 2018-04-11
category: australia
comments: true
---

We were lucky enough to ring in the new year down under, and with 2018 still fresh we flew out to the capital of the southern hemisphere: Sydney.  We arrived there with no real expectations and no knowledge of the city, other than that there's a big spiky white building on the harbour that's probably slightly overrated.

{% include postpic.html file="IMG_20180105_203725632.jpg" caption="Or...not overrated at all." %}

We made it to our hostel just in time to not check in, thanks to the weird hours kept by the staff, but managed to get sorted nonetheless. After getting our stuff stowed away, we checked out the rooftop of our hostel, and wandered around for some grub. Not a terribly eventful first day.

The next order of business was Justin's Birthday! Dana did a fair bit of research, and discovered a Bavarian biergarten a short train ride away from the hostel. The beer was delicious and cheap, and and we therefore have a treasure trove of lovely images from the evening.

{% include postpic.html file="IMG_20180105_173607.jpg" caption="This can only end well." %}

{% include postpic.html file="IMG_20180105_180810.jpg" caption="Justin had the only green stuff on the entire table. He must be getting old." %}

{% include postpic.html file="IMG_20180105_200909.jpg" caption="Yes, there are more photos like this. No, you can't see them." %}

{% include postpic.html file="IMG_20180105_202820.jpg" caption="Sydney skyline at dusk." %}

Having thoroughly enjoyed the night before, we were rather slow to get moving the next day. We even managed to miss a walking tour of Sydney that was to take place at two in the afternoon. Classy.

{% include postpic.html file="IMG_20180106_100705.jpg" caption="The morning after.  %}

Undeterred, we did our own tour, which led us on a trip across the harbour to Manly.

{% include postpic.html file="IMG_20180106_150411.jpg" caption="Back of the opera house. It kinda looks like a spaceship from this angle." %}

{% include postpic.html file="IMG_20180106_150613.jpg" caption="A bit further out, and we were treated to a gorgeous view of the skyline." %}

{% include postpic.html file="IMG_20180106_152333.jpg" caption="This boat was cruising along until it capsized a few minutes later. Woops!" %}

Upon landing at Manly Wharf, we grabbed some lunch and started hiking around. This place has A LOT of beaches, and a good amount of hiking trails winding along and through the waterfront. Eventually, the trail we were on meandered back inland, and we finished the day on Manly with a pint at one of the local breweries.

{% include postpic.html file="IMG_20180106_171749618.jpg" caption="No crashing into the bandicoots, no matter how well Nintendo has conditioned you." %}

{% include postpic.html file="IMG_20180106_165414.jpg" caption="One of Manly's many beaches. This particular trail took us all around this beach, and eventually led us passed St. Patrick's Seminary..." %}

{% include postpic.html file="IMG_20180106_174552.jpg" caption="...which has been around since 1889. The seminary part was closed in 1995, and it is now part of the International School of Management. Fun fact: former Australian PM Tony Abbott attended this seminary, and dropped out, before it closed down." %}

{% include postpic.html file="IMG_20180105_2.jpg" caption="On the way back. We were big fans of the skyline." %}

While we were in Sydney, we were lucky(?) enough to witness an historical event - the second hottest day on record in the Sydney metropolitan area. It was about 43C (110F) where we were in the city, and got up to 47C (~120F) in one surrounding town. After trying to brave the outdoors for an hour or two, we gave up and hid in a mall for the remainder of the day - it was actually too hot to get back to our hostel!

{% include postpic.html file="IMG_20180107_2.jpg" caption="Pictured: fanciful floats and general signs of happiness. Not pictured: our shoes melting into the ground." %}

{% include postpic.html file="IMG_20180107_1.jpg" caption="Lunch while we hid in one of the malls. It was delicious, and gave us a much needed respite from the not-quite-but-almost record breaking heat outside." %}

After the heat subsided, we decided to get out of the city. Fortunately, Sydney has a pretty extensive public transit system, and we were able to get to the Blue Mountains National Park via an easy two hour train ride. Once there, we were able to hike to our hearts' content, and were treated to some fantastic views!

{% include postpic.html file="IMG_20180109_113537.jpg" caption="A view from the beginning of the trail near the Three Sisters rock formation. It wove in and around the rock outcropping and the landscape alternated between dense tree canopy and exposed rocks." %}

{% include postpic.html file="IMG_20180109_122639.jpg" caption="A bit further along, this is one of the first views of the valley that we saw on the trail." %}

{% include postpic.html file="IMG_20180109_140821.jpg" caption="A few hours later... The rock in the distance is the rock on the left of the previous picture." %}

{% include postpic.html file="IMG_20180109_1.jpg" caption="This was taken from an outcropping near the end of the trail, which provided a view of where we had just come from. The trail is nearly invisible from here, but if you look near the waterfall in the centre, you can see a tiny glimpse." %}

{% include postpic.html file="IMG_20180109_144448.jpg" caption="After a few hours of hiking up and down and around these beautiful cliffs, we were a bit delirious!" %}

During out last few days, we were able to catch up with some friends, found some interesting food, and checked out some of the animal attractions in the city. Since we were only there a week, we barely scratched the surface of Sydney. However, our months in the South Pacific had come to an end, and our adventure was taking us to our next continent: South America!

{% include postpic.html file="IMG_20180108_141218.jpg" caption="Justin found the most fluffiest of all puppies. We considered stealing him, but since his owner is a cop, we thought better of it." %}

{% include postpic.html file="IMG_20180108_190021.jpg" caption="Meat pie with mushy peas on top, and mustard on top of that. Yes, it was weird. Good, but weird." %}

{% include postpic.html file="IMG_20180110_140717.jpg" caption="Rainbow dumplings!" %}

{% include postpic.html file="IMG_20180111_122415.jpg" caption="Kookaburra sits on the old gum tree..." %}

{% include postpic.html file="IMG_20180111_123426.jpg" caption="Sadly, the only kangaroos we saw the whole trip." %}

{% include postpic.html file="IMG_20180111_124856.jpg" caption="Whatchu lookin at?" %}

{% include postpic.html file="IMG_20180111_125152.jpg" caption="Hahaha, that's so cute and funny..." %}

{% include postpic.html file="IMG_20180111_130755.jpg" caption="Oh, that's the real one. It's <i>bigger</i> than the statue. Nope. Nope nope nope." %}

{% include postpic.html file="IMG_20180111_143350.jpg" caption="This gal just loved coming by to say 'hello!'" %}

{% include postpic.html file="IMG_20180111_145601.jpg" caption="Snaggletoothed nurse shark!" %}

{% include postpic.html file="IMG_20180111_222027.jpg" caption="The view from our hostel window on the last night. The tower is the train station, which we had been walking the long way to and didn't realize till our last day. Whoops." %}

repository: PenTagaris/oldmapsnewtracks

repo_name: PenTagaris/oldmapsnewtracks


text:
# Elixir LCD exercise
LCD like printer exercise. Elixir Version.

## Run with elixir
1. build with `mix escript.build`
2. optionnaly launch tests with `mix test`
3. launch with `./lcd 1234` and modify args as you wish

repository: vlamy/lcd

repo_name: vlamy/lcd