text:
# Agents (Code)

Agents are containers for reinforcement learning algorithms. They consist of a number of different components which are specified in the `experiment_specs`.

>The corresponding source code folder. When composing an agent by choosing components, look at the source code folder to see the list of available `Classes` to use, and look at their `__init__` method to see what parameters are available.

```json
"experiment_spec": "rl/spec/",
"problem": "rl/spec/problems.json",
"Agent": "rl/agent/",
"Policy": "rl/policy/",
"Memory": "rl/memory/",
"Optimizer": "rl/optimizer/",
"HyperOptimizer": "rl/hyperoptimizer/",
"PreProcessor": "rl/preprocessor/",
```

- `problem`: the gym environment the agent is solving.
- `Agent` (Learning algorithm),: decision function for learning from experiences gained by acting in an environment (eg Q-Learning, Sarsa). This is also the main class for agents. All other components of an agent are contained within this class.
- `Policy`: decision function for acting in an environment. Controls exploration vs. exploitation trade off(e.g. epsilon greedy, boltzmann)
- `Memory`: for storing experiences gained by acting in an environment. Controls how experiences are sampled for an agent to learn from. (e.g. random uniform with no forgetting, prioritized sampling with forgetting)
- `Optimizer`: controls how to optimize the function approximators contained within the agent (e.g. Stochatic Gradient Descent, Adam)
- `HyperOptimizer`: hyperparameter optimization algorithms used to vary the agent parameters and run trials with them (e.g grid search, random search)
- `PreProcessor`: controls the transformations made to state representaions before being passed as inputs to the policy and learning algorithm. (e.g. no preprocessing, concatenating current and previous state). Useful for Atari.
- `param`: the default parameters, unified for all agent components. To know what parameters are available, look into the `__init__` method of the components you're using. We have ensured there are no conflicting keys.
- `param_range`: the parameter space specification for the HyperOptimizer to search over in an experiment with many trials.

To define an `Agent` you must specify each of the components. The example below is from the specification for `dqn` in `rl/spec/classic_experiment_specs.json`. The `rl/spec/*.json` files contains lots of other examples.

```json
{
  "dqn": {
    "problem": "CartPole-v0",
    "Agent": "DQN",
    "HyperOptimizer": "GridSearch",
    "Memory": "LinearMemoryWithForgetting",
    "Optimizer": "AdamOptimizer",
    "Policy": "BoltzmannPolicy",
    "PreProcessor": "NoPreProcessor",
    "param": {
      "lr": 0.02,
      "gamma": 0.99,
      "hidden_layers": [64],
      "hidden_layers_activation": "sigmoid",
      "exploration_anneal_episodes": 10
    },
    "param_range": {
      "lr": [0.001, 0.005, 0.01, 0.02],
      "gamma": [0.95, 0.97, 0.99, 0.999],
      "hidden_layers": [
        [16],
        [32],
        [64],
        [16, 8],
        [32, 16]
      ]
    }
  }
}
```

Each of the components with the exception of `Agent` and `Policy` are uncoupled, so can be freely switched in an out for different types of components. Different combinations of components may work better than others. We leave that up to you to experiment with. For some inspiration, see how the Lab users build [the best solutions](#solutions).

For the currently implemented algorithms, the following `Agents` can go with the following `Policies`.

- `DQN, Sarsa, ExpectedSarsa, OffPolSarsa, FreezeDQN: EpsilonGreedyPolicy, OscillatingEpsilonGreedyPolicy, TargetedEpsilonGreedyPolicy, BoltzmannPolicy`
- `DoubleDQN: DoubleDQNPolicy, DoubleDQNBoltzmannPolicy`

## problem

**code location:** `rl/spec/problems.json`

Problems are not part of agent, but they are part of the `experiment_spec` that gets specified with the agent.

We have not added all the OpenAI gym environments AKA problems. If you get to new environments using the lab, please add them in `rl/spec/problems.json`, and it should be clear from those examples.

Moreover, when adding new problems, consider the dependencies setup too, such as Mujoco. Please add these to the `bin/setup` so that other users could run it.

## Agent

**code location:** `rl/agent/`

*See the overall list of agents and their implementation status under the [Agents Fitness Matrix](#agents-matrix). See [algorithms section](#algorithms) for an explanation of the currently implemented agents (learning algorithms).*

The main `Agent` class that will house all the other components. This is where the algorithms like `DQN, DDQN, DDPG` are implemented. This class shall contain only the algorithm-specific logic; the generic components are modularized for reuse across different agents.

Despite the multitude of parameters you can see in the contructor of the Agent classes, `Agent`s always need the following:

- `gamma`: how much to discount the future
- `hidden_layers`: list of hidden layer sizes for neural network function approximators
- `hidden_layers_activation`: activation function for the hidden layers

The input and output layer sizes are autoamtically inferred from the environment specs.

```json
"dqn": {
  "Agent": "DQN",
  "param": {
    "hidden_layers": [32, 16],
    "hidden_layers_activation": "sigmoid",
  },
```

For HyperOptimizer to search over the network architecture (yes this is possible), use the *auto-architecture* mode for building the network. The HyperOptimizer will search over the network depth and width (2 parameters is sufficient to construct a whole network). To do so, set the `auto-architecture` param to true, and specific the `num_hidden_layers` and the `first_hidden_layer_size`.

```json
"dqn": {
  "Agent": "DQN",
  "param": {
    "hidden_layers_activation": "sigmoid",
    "auto_architecture": true,
    "num_hidden_layers": 3,
    "first_hidden_layer_size": 512
  },
```

## Policy

**code location:** `rl/policy/`

A policy is a decision function for acting in an environment. Policies take as input a description of the state space and output an action for the agent to take.

Depending on the algorithm used, agents may directly approximate the policy (policy based algorithms) or have an indirect policy, that depends on the Q-value function approximation (value based algorithms). Algorithms that approximate both the policy and the Q-value function are known as actor-critic algorithms.

All of the algorithms implemented so far are value-based. The policy for acting at each timestep is often a simple epsilon-greedy policy.

![](./images/e_greedy.png)

Alternatively, an indirect policy may use the Q-value to output a probability distribution over actions, and sample actions based on this distribution. This is the approach taken by the Boltzmann policies.

A critical component of the policy is how is balances *exploration* vs. *exploitation*. To learn how to act well in an environment an agent must *explore* the state space. The more random the actions an agent takes, the more it explores. However, to do well in an environment, an agent needs to take the best possible action given the state. It must *exploit* what it has learnt.

Below is a summary of the currently implemented policies. Each takes a slightly different approach to balancing the exploration-exploitation problem.

### EpsilonGreedyPolicy
Parameterized by starting value for epsilon (`init_e`), min value for epsilon (`final_e`), and the number of epsiodes to anneal epsilon over (`exploration_anneal_episodes`). The value of epsilon is decayed linearly from start to min.

```json
"dqn": {
  "Policy": "EpsilonGreedyPolicy",
  "param": {
    "init_e" : 1.0,
    "final_e" : 0.1,
    "exploration_anneal_episodes": 100
  },
```

### DoubleDQNPolicy
When actions are not random this policy selects actions by summing the outputs from each of the two Q-state approximators before taking the max of the result. Same approach as `EpsilonGreedyPolicy` to decaying epsilon and same params.

### BoltzmannPolicy
Parameterized by the starting value for tau (`init_tau`), min value for tau (`final_tau`), and the number of epsiodes to anneal epsilon over (`exploration_anneal_episodes`). At each step this policy selects actions based on the following probability distribution

![](./images/boltzmann.png)

Tau is decayed linearly over time in the same way as in the `EpsilonGreedyPolicy`.

```json
"dqn": {
  "Policy": "BoltzmannPolicy",
  "param": {
    "init_tau" : 1.0,
    "final_tau" : 0.1,
    "exploration_anneal_episodes": 10
  },
```

### DoubleDQNBoltzmannPolicy
Same as the Boltzmann policy except that the Q value used for a given action is the sum of the outputs from each of the two Q-state approximators.

### TargetedEpsilonGreedyPolicy
Same params as epsilon greedy policy. This policy swtches between active and inactive exploration cycles controlled by partial mean rewards and it distance to the target mean rewards.

### DecayingEpsilonGreedyPolicy
Same params as epsilon greedy policy. Epsilon is decayed exponentially.

### OscillatingEpsilonGreedyPolicy
Same as epsilon greedy policy except at episode 18 epsilon is dropped to the max of 1/3 or its current value or min epsilon.

### Creating your own

A policy has to have the following functions. You can create your own by inheriting from Policy or one of its children.

```python
def select_action(self, state):
        '''Returns the action selected given the state'''

def update(self, sys_vars):
    '''Update value of policy params (e.g. epsilon)
    Called each timestep within an episode'''
```

## Memory

**code location:** `rl/memory/`

The agent's memory stores experiences that an agent gains by acting within an environment. An environment is in a particular state. Then the agent acts, and receives a reward from the environment. The agent also receives information about the next state, including a flag indicating whether the next state is the terminal state. Finally, an error measure is stored, indicating how well an agent can estimate the value of this particular transition.

This information about a single step is stored as an experience. Each experience consists of

- Current state
- Action taken
- Reward
- Next state
- Terminal
- Error

```python
(state, action, next_state, reward, terminal, error)
```

Crucially, the memory controls how long experiences are stored for, and which experiences are sampled from it to use as input into the learning algorithm of an agent. Below is a summary of the currently implemented memories.

### LinearMemory
The size of the memory is unbounded and experiences are sampled random uniformly from memory.

### LinearMemoryWithForgetting

```json
"dqn": {
  "Memory": "LinearMemoryWithForgetting",
  "param": {
    "max_mem_len" : 10000
  },
```

Parameterizes by `max_mem_len` param which bounds the size of the memory. Once memory reaches the max size, the oldest experiences are deleted from the memory to make space for new experiences. Experiences are sampled random uniformly from memory.

### LeftTailMemory
Like linear memory with sampling via a left-tail distribution. This has the effect of drawing more from newer experiences.

### PrioritizedExperienceReplay
Experiences are weighted by the error, a measure of how well the learning algorithm currently performs on that experience. Experiences are sampled from memory in proportion to the p value (adjusted error value)

```python
p = (1 + e)** alpha
```

The parameter `e > 0` (if not positive, our implementation will bump it up) is a constant added onto the error to prevent experiences with error 0 never being sampled. `alpha` controls how spiked the distribution is. The lower `alpha` the closer to unform the distribution is. `alpha = 0` corresponds to uniform random sampling.

```json
"dqn": {
  "Memory": "PrioritizedExperienceReplay",
  "param": {
    "e" : 0.01,
    "alpha": 0.6,
    "max_mem_len" : 10000
  },
```

This has the effect of drawing more from experiences that the learning algorithm doesn't perform well on, i.e. the experiences from which is has most to learn. The size of the memory is bounded as in LinearMemoryWithForgetting.

### RankedMemory
Memory sorted by the `mean_rewards` of each episode. Still experimental.

### Creating your own

A memory has to have the following functions. You can create your own by inheriting from Memory or one of its children.

```python
def add_exp(self, action, reward, next_state, terminal, error):
    '''add an experience to memory'''

def get_exp(self, inds):
    '''get a batch of experiences by indices
       helper function called by rand_minibatch'''

def pop(self):
    '''get the last experience (batched like get_exp()'''

def size(self):
    '''returns the size of the memory'''

def rand_minibatch(self, size):
        '''returns a batch of experiences sampled from memory'''
```

## Optimizer

**code location:** `rl/optimizer/`

Controls how to optimize the function approximators contained within the agent. For feedforward and convolutional neural networks, we suggest using Adam with the default parameters for everything except the learning rate as this is widely considered to be the best algorithm for optmizing deep neural network based function approximators. For recurrent neural networks we suggest using RMSprop.

### SGD
Stochastic Gradient Descent. Parameterized by `lr` (learning rate), `momentum`, `decay` and `nestorov`. See [Keras](https://keras.io/optimizers/#sgd) for more details.

```json
"dqn": {
  "Optimizer": "SGDOptimizer",
  "param": {
    "lr": 0.02,
    "momentum" : 0.9
    "decay": 0.00001
    "nesterov": true
  },
```

### Adam
Parameterized by `lr` (learning rate), `beta_1`, `beta_2`, `epsilon`, `decay`. See [Keras](https://keras.io/optimizers/#adam) for more details.

### RMSprop
Parameterized by `lr` (learning rate), `rho`, `epsilon`, `decay`. See [Keras](https://keras.io/optimizers/#rmsprop) for more details.

## HyperOptimizer

**code location:** `rl/hyperoptimizer/`

Controls how to search over the hyperparameter space. We suggest using each of the three hyperoptimizers in the following order when trying to find the optimal parameters for an agent in an environment, to help gradually narrow down the best search space. From rough to fine granularity, `LineSearch > GridSearch > RandomSearch`.

### LineSearch

Varies one parameter per trial while using default values for the rest. This is useful when trying to estimate of what value range should work for a parameter without having to search through the full space blindly.

Given that we have built a proven base of best parameters, this is now seldom used, as we often could know the optimal parameter values to a small range.

### GridSearch

Once the search space is narrowed down, it's time to do a systematic grid search. This takes the cartesian products of the discrete list of options, i.e. every combination, and run trials. This is the most commonly used, and could help us properly see the heatmap when we probe the search space systematically.

### RandomSearch

```
Random Search by sampling on hysphere around a search path:
1. init x a random position in space
2. until termination (max_eval or fitness, e.g. solved all), do:
    2.1 sample new pos some radius away: next_x = x + r
    2.2 if f(next_x) > f(x) then set x = next_x
```

After GridSearch, there are usually intermediate values between the grid points that could yield slightly higher fitness score. Narrow down even further to the best search space as shown in the GridSearch heatmap, and do a random search. This is the final fine-tuning. The RandomSearch algorithm is directional and greedy.

### HyperOptimizer Roadmap

These are the future hyperparameter optimization algorithms we'd like to implement standalone in the Lab. The implementations for them currently exists, but they're too bloated, and their engineering aspects are not ideal for the Lab.

- [TPE](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf) / [hyperopt](https://github.com/hyperopt/hyperopt)
- [Bayesian Optimizer (Spearmint)](https://github.com/HIPS/Spearmint)
- [SMAC](http://www.cs.ubc.ca/labs/beta/Projects/SMAC/#software)

### Implementation Guideline

All implemented hyperoptimizers shall extend the base `HyperOptimizer` class in `rl/hyperoptimizer/base_hyperoptimizer.py` and follow its design for compatibility. Below we show this design to be general theoretically and practically. Moreover, do not use bloated dependencies.

**Theoretical design:**

A hyperoptimizer is a function `h` that takes:

- a trial (objective) function `Trial`
- a parameter space `P` (implemented in `experiment_spec`)

and runs the algorithm:

1. search the next `p` in `P` using its internal search algorithm, add to its internal `param_search_list`.
2. run a (slow) function `Trial(p) = fitness_score` (inside trial data)
3. update search using the feedback `fitness_score`
4. repeat until max steps or fitness condition met

Note that the search space `P` is a tensor space product of `m` bounded real spaces `R` and `n` bounded discrete spaces `N`. The search path in `param_search_list` must also be well-ordered to ensure resumability.

**Implementation requirements:**

1. we want order-preserving and persistence in search for the ability to resume/reproduce an experiment.
2. the search algorithm may have its own internal memory/belief to facilitate search.
3. the `Trial` function shall be treated as a blackbox `Trial(p) = fitness_score` with input/output `(p, fitness_score)` for the generality of implementation/

**Specification of search space:**

1\. for real variable, specify a distribution (an interval is just a uniformly distributed space) in the `experiment_spec.param_range`. Example:

```json
"lr": {
  "min": 0.0005,
  "max": 0.05
}
```

2\. for discrete variable, specify a list of the values to search over (since it is finite anyway) in the `experiment_spec.param_range`. This will automatically be sorted when read into `HyperOptimizer` to ensure ordering. Example:

```json
"lr": [0.01, 0.02, 0.05, 0.1, 0.2]
```

The hyperopt implementation shall be able to take these 2 types of param_range specs and construct its search space.

Note that whether a variable is real or discrete can be up to the user; some variable such as `lr` can be sampled from interval `0.001 to 0.1` or human-specified options `[0.01, 0.02, 0.05, 0.1, 0.2]`. One way may be more efficient than the other depending on the search algorithm.

The experiment will run it as:

```python
# specify which hyperoptimizer class to use in spec for bookkeeping
Hopt = get_module(GREF, experiment_spec['HyperOptimizer'])
hopt = Hopt(Trial, **experiment_kwargs)
experiment_data = hopt.run()
```

## Preprocessor

**code location:** `rl/preprocessor/`

Sometimes preprocessing the states before they are received by the agent can help to simplify the problem or make the agent strong. One example is the pixel preprocessing the removes color channels and rescales image size, in order to reduce unnecessary information overload. The other is to concat states from sequential timesteps to present richer, correlated information that is otherwise sparse.

The change in dimensions after preprocessing is handled automatically, so you can use them without any concerns.

### NoPreProcessor

The default that does not preprocess, but pass on the states as is.

### StackStates

Concat the current and the previous states. Turns out this boosts agent performance in the `LunarLander-v2` environment.

### DiffStates

Take the difference `new_states - old_states`.

### Atari

Convert images to greyscale, downsize, crop, then stack 4 most recent states together. Useful for the Atari environments.


text:
---
title: "Planning Week 41 - HackerNews, Reddit, MVP"
date: 2020-10-05
layout: post.njk
tags:
  - post
  - featured
  - privacy
  - update
---

Today I am going to post this article to HackerNews and Reddit

[Notes on “I've got nothing to hide”](/blog/2020-10-02-Notes-on-I-have-got-nothing-to-hide/).

Let's see how it goes and what feedback I'll receive.

---

In the next days I'll work on a MVP for the Decent.social web client!

If you have feedback about it, reach out on Twitter.

People that are currently subscribed will have access to the beta.

---

I still need to receive feedback on the overall Design of Decent.social from a few friends of mine.

---

Write a new Blog post?

license: MIT

repository: decentsocial/website

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: decentsocial/website


text:
#  Learning from Limited and Imperfect Data (L2ID): Classification Challenges

### Website
#### https://l2id.github.io/challenge_classification.html

### Paper

Please cite the following paper in use of this evaluation framework:

```
@inproceedings{guo2020broader,
  title={A broader study of cross-domain few-shot learning},
  author={Guo, Yunhui and Codella, Noel C and Karlinsky, Leonid and Codella, James V and Smith, John R and Saenko, Kate and Rosing, Tajana and Feris, Rogerio},
  year={2020},
  organization={ECCV}
}
```

## Introduction

We will have three tracks investigating *cross-domain*, *multi-source settings* as well as *discrimination across a larger number of classes*, bridging the gap between few-shot learning, domain adaptation, and semi-supervised learning. All tracks will include multiple sources and the use of unlabeled data to support semi-supervised algorithms.

# Track 1: Cross-domain, *small* scale

## Description
This setting is similar as the previous VL3 workshop challenge (https://www.learning-with-limited-labels.com/challenge), supporting teams that would like to continue development for cross-domain few-shot learning. However, we will have multiple sources rather than relying on ImageNet solely, with no explicit label overlap between sources and target. These additional sources remain consistent with prior literature, allowing results to be directly comparable to prior results (https://arxiv.org/abs/1912.07200)

## Datasets
The following datasets are used for evaluation in this challenge:

### Source domain:

* miniImageNet
* CUB (http://www.vision.caltech.edu/visipedia/CUB-200.html)
* CIFAR100
* Caltech256
* DTD (<https://www.robots.ox.ac.uk/~vgg/data/dtd/>)

### Target domains:

* **EuroSAT**:

    Home: http://madm.dfki.de/downloads

    Direct: http://madm.dfki.de/files/sentinel/EuroSAT.zip

* **ISIC2018**:

    Home: http://challenge2018.isic-archive.com

    Direct (must login): https://challenge.isic-archive.com/data#2018

* **Plant Disease**:

    Home: https://www.kaggle.com/saroz014/plant-disease/

    Direct: command line `kaggle datasets download -d plant-disease/data`

* **ChestX-Ray8**:

    Home: https://www.kaggle.com/nih-chest-xrays/data

    Direct: command line `kaggle datasets download -d nih-chest-xrays/data`

# Track 2: Cross-domain, *Large* scale

## Description
In this track we add additional datasets to both source and target datasets for participants with sufficient compute resources. Importantly, in this task, in addition to the multiple sources, we provide multiple tasks from which to draw source data or models.

## Datasets
The following datasets are used for evaluation in this challenge:

### Source domain:

* miniImageNet
* CUB (http://www.vision.caltech.edu/visipedia/CUB-200.html)
* CIFAR100
* Caltech256
* DTD (<https://www.robots.ox.ac.uk/~vgg/data/dtd/>)
* DomainNet (http://ai.bu.edu/M3SDA/#dataset)
* COCO (2017 Train Images) (https://cocodataset.org/#download)
* PASCAL (http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html#data)
* KITTI (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d)
* Cityscapes (*any version or task)  (https://www.cityscapes-dataset.com/dataset-overview/#class-definitions)

### Target domains:

* **EuroSAT**:

    Home: http://madm.dfki.de/downloads

    Direct: http://madm.dfki.de/files/sentinel/EuroSAT.zip

* **ISIC2018**:

    Home: http://challenge2018.isic-archive.com

    Direct (must login): https://challenge.isic-archive.com/data#2018

* **Plant Disease**:

    Home: https://www.kaggle.com/saroz014/plant-disease/

    Direct: command line `kaggle datasets download -d plant-disease/data`

* **ChestX-Ray8**:

    Home: https://www.kaggle.com/nih-chest-xrays/data

    Direct: command line `kaggle datasets download -d nih-chest-xrays/data`

* **PatchCamelyon**:

    Home: https://github.com/basveeling/pcam (level_2_split_train)

* **KenyanFood13**:

    Home: https://drive.google.com/file/d/1CHnTy4XqGowT2chCBBxZ8rdYxMC1bwTc/view (training image)

* **IP102**:

    Home: https://github.com/xpwu95/IP102 (training image)

* **Bark-101**:

    Home: http://eidolon.univ-lyon2.fr/~remi1/Bark-101/ (training image)

## General information

* **No meta-learning in-domain**
* 5-way classification
* n-shot, for varying n per dataset
* 100 randomly selected few-shot 5-way trials up to 50-shot (scripts provided to generate the trials)
* Average accuracy across all trials reported for evaluation.

* **For generating the trials for evaluation, please refer to finetune.py and the examples below**

## Specific Tasks:

### Tasks for Track 1 and Track 2

**EuroSAT**

  • Shots: n = {5, 20, 50}

**ISIC2018**

  • Shots: n = {5, 20, 50}

**Plant Disease**

  • Shots: n = {5, 20, 50}

**ChestX-Ray8**

  • Shots: n = {5, 20, 50}

### Additional Tasks for Track 2

**PatchCamelyon**

  • Shots: n = {5, 20, 50}

**KenyanFood13**

  • Shots: n = {5, 20, 50}

**IP102**

  • Shots: n = {5, 20, 50}

**Bark-101**

  • Shots: n = {5}

## Enviroment

Python 3.5.5

Pytorch 0.4.1

h5py 2.9.0

## Steps

1. Download the datasets for specific tracks for evaluation using the above links.

2. Download miniImageNet using <https://drive.google.com/file/d/1uxpnJ3Pmmwl-6779qiVJ5JpWwOGl48xt/view?usp=sharing>

3. Change configuration file `./configs.py` to reflect the correct paths to each dataset. Please see the existing example paths for information on which subfolders these paths should point to.

4. Train base models on miniImageNet

    • *Standard supervised learning on miniImageNet*

    ```bash
        python ./train.py --dataset miniImageNet --model ResNet10  --method baseline --train_aug
    ```

    • *Train meta-learning method (protonet) on miniImageNet*

    ```bash
        python ./train.py --dataset miniImageNet --model ResNet10  --method protonet --n_shot 5 --train_aug
    ```

5. Save features for evaluation (optional, if there is no need to adapt the features during testing)

    • *Save features for testing*

    ```bash
        python save_features.py --model ResNet10 --method baseline --dataset CropDisease --n_shot 5 --train_aug
    ```

6. Test with saved features (optional, if there is no need to adapt the features during testing)

    ```bash
        python test_with_saved_features.py --model ResNet10 --method baseline --dataset CropDisease --n_shot 5 --train_aug
    ```

7. Test

    • *Finetune with frozen model backbone*:

    ```bash
        python finetune.py --model ResNet10 --method baseline  --train_aug --n_shot 5 --freeze_backbone
    ```

    • *Finetune*

    ```bash
        python finetune.py --model ResNet10 --method baseline  --train_aug --n_shot 5
    ```

    • *Example output:* 600 Test Acc = 49.91% +- 0.44%

8. Test with Multi-model selection (make sure you have trained models on all the source domains (miniImageNet, CUB, Caltech256, CIFAR100, DTD))

    • *Test Multi-model selection without fine-tuning*:

    ```bash
       python model_selection.py --model ResNet10 --method baseline  --train_aug --n_shot 5
    ```

    • *Test Multi-model selection without fine-tuning*:

     ```bash
       python model_selection.py --model ResNet10 --method baseline  --train_aug --n_shot 5 --fine_tune_all_models
     ```

9. For testing your own methods, simply replace the function **finetune()** in `finetune.py` with your own method. Your method should at least have the following arguments,

    • *novel_loader: data loader for the corresponding dataset (EuroSAT, ISIC2018, Plant Disease, ChestX-Ray8 for track 1)*

    • *n_query: number of query images per class*

    • *n_way: number of shots*

    • *n_support: number of support images per class*

## References

[1] Sharada P Mohanty, David P Hughes, and Marcel Salathe. Using deep learning for image
based plant disease detection. Frontiers in plant science, 7:1419, 2016

[2] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel
dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of
Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217–2226, 2019.

[3] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.
Scientific data, 5:180161, 2018.

[4] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen
Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael
Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge
hosted by the international skin imaging collaboration (isic). arXiv preprint.
arXiv:1902.03368, 2019

[5] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald
M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly
supervised classification and localization of common thorax diseases. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017


text:
---
layout: post
title: "Calling stata from python"
---

{{ page.title }}
================

15 July, 2011 - Dan Hammer

I think this may be useful to some poor souls in graduate school.  It has certainly been useful for me; and I am a poor soul in graduate school.  Suppose you want to scrape a website for data, prep it in Python, and then send it to Stata.  This standalone function will launch a do-file and pass parameters to the Stata process from within a Python script.

{% highlight python %}
## Python function to launch a do-file
def dostata(dofile, *params):
    ## Launch a do-file, given the fullpath to the do-file
    ## and a list of parameters.
    import subprocess
    cmd = ["stata", "do", dofile]
    for param in params:
        cmd.append(param)
    return subprocess.call(cmd)

{% endhighlight %}

Then, within the do-file, you can grab the parameters by setting locals or globals, as shown below.  Be sure to clear and exit Stata at the end of the script (and leave an extra line -- hit enter a couple of times at the end of the do-file) so that you can free up the subprocess and return to Python.

{% highlight clojure %}

## Stata do file, launched by the preceding Python function

clear all
set mem 100m
set more off

local param1 `1'
global param2 `2'

di "first parameter: `param1'"
di "second parameter: $param2"

exit, STATA clear

## Leave an extra line here, since the last line of the do-file
## will not get sent to the process.

{% endhighlight %}

Oh, and if one of the parameters is a list that you want to iterate through, you'll need to prep the Python list with this function before passing it to `dostata()`:

{% highlight python %}

def ls2stata(pylist):
    ## Convert a Python list into an object that can be read as
    ## a list in Stata from the commandline.
    statalist = '"' + " ".join(pylist) + '"'
    return statalist

{% endhighlight %}


text:
# PHP Blade Directives

[![PHP CodeSniffer](https://github.com/Saritasa/php-blade-directives/workflows/PHP%20Codesniffer/badge.svg)](https://github.com/Saritasa/php-blade-directives/actions)
[![Release](https://img.shields.io/github/release/saritasa/php-blade-directives.svg)](https://github.com/Saritasa/php-blade-directives/releases)
[![PHPv](https://img.shields.io/packagist/php-v/saritasa/blade-directives.svg)](http://www.php.net)
[![Downloads](https://img.shields.io/packagist/dt/saritasa/blade-directives.svg)](https://packagist.org/packages/saritasa/blade-directives)

Custom Blade directives for Laravel Applications.

See https://laravel.com/docs/blade

## Laravel 5.x

Install the ```saritasa/blade-directives``` package:

```bash
$ composer require saritasa/blade-directives
```

If you use Laravel 5.4 or less,
or 5.5+ with [package discovery](https://laravel.com/docs/5.5/packages#package-discovery) disabled,
add the BladeDirectivesServiceProvider service provider ``config/app.php``:

```php
'providers' => array(
    // ...
    Saritasa\Laravel\BladeDirectivesServiceProvider::class,
)
```

## Available directives

### @activeIfRoute(routeName)
Output word 'active' if URL, requested by user, matches provided route name

**Example**:
```
<li class="@activeIfRoute('user.settings')">
    <a href="{{ route('user.settings') }}">Settings</a>
</li>
```

### @selectedIf(expression)
Output word 'selected' if passed expression evaluates to true

**Example**:
```
<select>
    @foreach($users as $user)
        <option value='{{ $user->id }}' @selectedIf($user->id == $currentUserId)>{{ $user->name }}</option>
    @endforeach
</select>
```

## Contributing

1. Create fork, checkout it
2. Develop locally as usual. **Code must follow [PSR-1](http://www.php-fig.org/psr/psr-1/), [PSR-2](http://www.php-fig.org/psr/psr-2/)** -
    run [PHP_CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer) to ensure, that code follows style guides
3. Update [README.md](README.md) to describe new or changed functionality
4. Add changes description to [CHANGES.md](CHANGES.md) file. Use [Semantic Versioning](https://semver.org/) convention to determine next version number.
5. When ready, create pull request

### Make shortcuts

If you have [GNU Make](https://www.gnu.org/software/make/) installed, you can use following shortcuts:

* ```make cs``` (instead of ```php vendor/bin/phpcs```) -
    run static code analysis with [PHP_CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
    to check code style
* ```make csfix``` (instead of ```php vendor/bin/phpcbf```) -
    fix code style violations with [PHP_CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
    automatically, where possible (ex. PSR-2 code formatting violations)
* ```make install``` - instead of ```composer install```
* ```make all``` or just ```make``` without parameters -
    invokes described above **install**, **cs** tasks sequentially -
    project will be assembled and checked with linter with one single command

## Resources

* [Bug Tracker](http://github.com/saritasa/php-blade-directives/issues)
* [Code](http://github.com/saritasa/php-blade-directives)
* [Changes History](CHANGES.md)
* [Authors](http://github.com/saritasa/php-blade-directives/contributors)

repository: Saritasa/php-blade-directives

gha_language: PHP

gha_license_id: MIT

repo_name: Saritasa/php-blade-directives


text:
This is intended to be a very fast test suite. On some platform (e.g. Windows) we've found that test suites like this
can run faster than the resolution of the filesystem timestamps. This leads to problems where Python doesn't regenerate
pycs files when necessary, leading to incorrect mutation testing results.

We've modified CR to work around these problems, and this test (as driven by the pytest suite) will hopefully detect
regressions in our workaround.

repository: sixty-north/cosmic-ray

gha_language: Python

repo_name: sixty-north/cosmic-ray


text:
# Analog to Digital Converter

Céu-Arduino provides asynchronous reads of analog values. This driver provides users with events and a data structure for full control of the driver as well as an abstraction which encapsulates the procedure.

## Output Events

### ADC_REQUEST

```ceu
output (pin) ADC_REQUEST
```

- Parameters:
    - `pin` | `integer` : pin to queue read

Queues an analog read for the specified analog pin.

The example below starts a conversion for `A0`:

```ceu
...
emit ADC_REQUEST(0);
...
```

## Input Events

### ADC_DONE

```ceu
input (none) ADC_DONE
```

Marks the end of a read. Can be used for synchronization.

The example below keeps a LED connected to `PIN_01` on while the read is in progress:

```ceu
...
emit ADC_REQUEST(1);
emit PIN_01(on);
await ADC_DONE;
emit PIN_01(off);
...
```

## Data Abstractions

There is a data structure which carries all information about the read state for each pin. The structure is a vector of [`ADC_Channel`](#adc_channel) named `channels`. There is an instance of [`ADC_Channel`](#adc_channel) in `channels` for each analog read pin available on the board (5 for the Arduino Uno).

### ADC_Channel

```ceu
data ADC_Channel with
    var bool is_available = false;
    var bool is_busy      = false;
    var int  value;
end
```

- Fields:
    - `is_available` : Whether or not there is read value available
    - `is_busy` : Whether or not a read is in progress
    - `value` : Value of latest conversion, in case `is_available` is set

This data structure carries all information about a pin read state. It should be used to get the actual value of the read and to monitor the state of the conversion.

The `is_available` flag is only set when a conversion is done for that pin. Since [`ADC_DONE`](#adc_done) is emitted at the end of every conversion, the flag provides a way to differentiate between conversion results.

**Clearing the flag is the user's responsability.**

The `is_busy` flag is for the driver's internal usage. It asserts that there are not more than one [`await Analog()`](#analog) for the same pin at a same given time.

**If the application enters two [`await Analog()`](#analog) simultaneously, the application will raise an error**

The example below reads the conversion value for pin `A2`:

```ceu
...
emit ADC_REQUEST(2);
await ADC_DONE;
var int recv = channels[2].value;
...
```
The code below provides a safer alternative to the code above since it deals with simultaneous reads:

```ceu
...
emit ADC_REQUEST(2);
await ADC_DONE until channels[2].is_available;
var int recv = channels[2].value;
channels[2].is_available = false;
...
```

## Code/Await Abstractions

### Analog

Performs a full analog read.

```ceu
code/await Analog (var int channel) -> int do
    _ceu_dbg_assert(outer.channels[channel].is_busy == false);
    outer.channels[channel].is_busy = true;
    do finalize with
        outer.channels[channel].is_busy = false;
    end
    emit ADC_REQUEST(channel);
    await ADC_DONE until outer.channels[channel].is_available;
    outer.channels[channel].is_available = false;
    escape outer.channels[channel].value;
end
```

- Parameters:
    - `channel` : analog pin for which to perform the read

The example below reads the value from analog pins 3 and 4, concurrently:

```ceu
...
par/and do
    var int recv_3 = await Analog(3);
with
    var int recv_4 = await Analog(4);
end
...
```

repository: fsantanna/ceu-arduino

gha_language: Makefile

repo_name: fsantanna/ceu-arduino


text:
# Robot Resize Images
Robot images editor with progess bar

Robot to resize images and edit with blurred background

## Getting Started

- Clone the repo: `git clone https://github.com/lucassilva15/robot-resize-image.git`.
- [Download from Github](https://github.com/lucassilva15/robot-resize-image/archive/master.zip).

### Prerequisites

```
- Node.JS
- Yarn (Opcional)
```

### Quick Setup
```
- After clone the repository, install dependencies with yarn. `yarn`
- Add all images to images folder.
- Run `yarn start`
- Wait for processing.
- Open the converted folder at the project root
```

## Authors

| [
@lucassilva15](https://github.com/lucassilva15) |
| :---: |

## Licensing

- Licensed under MIT (https://github.com/lucassilva15/robot-resize-image/blob/master/LICENSE)

gha_language: JavaScript


text:
# Sonata Admin Clone Action

[![Latest Stable Version](http://poser.pugx.org/jorrit/sonata-clone-action-bundle/v)](https://packagist.org/packages/jorrit/sonata-clone-action-bundle) [![Total Downloads](http://poser.pugx.org/jorrit/sonata-clone-action-bundle/downloads)](https://packagist.org/packages/jorrit/sonata-clone-action-bundle) [![Latest Unstable Version](http://poser.pugx.org/jorrit/sonata-clone-action-bundle/v/unstable)](https://packagist.org/packages/jorrit/sonata-clone-action-bundle) [![License](http://poser.pugx.org/jorrit/sonata-clone-action-bundle/license)](https://packagist.org/packages/jorrit/sonata-clone-action-bundle) [![PHP Version Require](http://poser.pugx.org/jorrit/sonata-clone-action-bundle/require/php)](https://packagist.org/packages/jorrit/sonata-clone-action-bundle)

Adds a clone action to Sonata Admin. This allows you to add a clone button to
your list action that leads to a create form with the values of the cloned item
prefilled.

The clone action does not create the clone in the database, this happens only
when the create form is submitted.

## Installation

```bash
$ composer require jorrit/sonata-clone-action-bundle
```

## Setup

The extension is registered just like any other Sonata Admin extension.
For more information regarding extensions, see the [Sonata Admin documentation](https://sonata-project.org/bundles/admin/3-x/doc/reference/extensions.html).

### Add to specific admins

Add the following code to services.yml to add the extension to one or more admin classes.

Replace `admin1` and `admin2` with the service names of your admin classes.

```yaml
    admin.clone.extension:
        class: Jorrit\SonataCloneActionBundle\Admin\Extension\CloneAdminExtension
        tags:
            - { name: sonata.admin.extension, target: admin1 }
            - { name: sonata.admin.extension, target: admin2 }
```

### Add to all admins

Add the following code to services.yml to add the extension all admin classes.

```yaml
    admin.clone.extension:
        class: Jorrit\SonataCloneActionBundle\Admin\Extension\CloneAdminExtension
        tags:
            - { name: sonata.admin.extension, global: true }
```

### Add the action to your admin list

Edit your admin class to add `clone` to the list of actions:

```php
    protected function configureListFields(ListMapper $listMapper)
    {
        $listMapper
            ...
            ->add('_action', null, [
                'actions' => [
                    'edit' => [],
                    'delete' => [],
                    'clone' => [],
                ]
            ]);
    }
```

repository: jorrit/SonataCloneActionBundle

branch_name: refs/heads/main

repo_name: jorrit/SonataCloneActionBundle


text:
Title: Implementation of Filter Visitor (JDBC)

# Implementation of Filter Visitor (JDBC)

### How To Guide for implementing a filter tree transformation into a JDBC where clause
The query option $filter can be used to apply a filter query to the result set. This tutorial will be about consuming and working with the filter tree which an application will get from the OData Java library by implementing a transformation of the filter expression into a JDBC where clause. The example explained here will be kept simple to show the mechanism of the visitor pattern. Security problem which occur when using user input (e.g. the filter string of the URI) inside a where clause will be pointed out but not solved for this tutorial. Knowledge about the visitor pattern is not necessary but helpful. If you want to read further please refer to the further information chapter at the end of this tutorial.

### Examples
##### Simple example
If a filter expression is parsed by the OData library it will be transformed into a filter tree. A simple tree for the expression ‘a’ eq ‘b’ would look like this:

![Picture:Simple Filter Expression](/img/FilterExpressionSimple.png)

To visit a filter tree we have to implement the interface `org.apache.olingo.odata2.api.uri.expression.ExpressionVisitor`. For this simple example we will only need the following methods:

- `visitFilter(…)`
- `visitBinary(…)`
- `visitLiteral(…)`

These methods will be called if the `accept(…)` method of the filter expression is called. The visitor will always start with the far left of the tree. First visitLiteral is called for 'a' and 'b'  before the `visitBinary()` and finally the `visitFilter()` is called.

Now lets have a look at the implementation of the `visitLiteral()`:

    @Override
    public Object visitLiteral(LiteralExpression literal, EdmLiteral edmLiteral) {
      if(EdmSimpleTypeKind.String.getEdmSimpleTypeInstance().equals(edmLiteral.getType())) {
        // we have to be carefull with strings due to sql injection
        // TODO: Prevent sql injection via escaping
        return "'" + edmLiteral.getLiteral() + "'";
      } else {
        return "'" + edmLiteral.getLiteral() + "'";
      }
    }

The signature for this method contains the literal expression as an object as well as the edmLiteral representation. In this case the literal would be "a" without the '. Inside this method we just return the literal. Since this literal is coming from the URL it represents user input. In the case of the type Edm.String we would have to escape the String to make sure no SQL Injection is possible.

Next the method visitBinary is called:

    @Override
      public Object visitBinary(BinaryExpression binaryExpression, BinaryOperator operator, Object leftSide, Object rightSide) {
        //Transform the OData filter operator into an equivalent sql operator
        String sqlOperator = "";
        switch (operator) {
        case EQ:
          sqlOperator = "=";
          break;
        case NE:
          sqlOperator = "<>";
          break;
        case OR:
          sqlOperator = "OR";
          break;
        case AND:
          sqlOperator = "AND";
          break;
        case GE:
          sqlOperator = ">=";
          break;
        case GT:
          sqlOperator = ">";
          break;
        case LE:
          sqlOperator = "<=";
          break;
        case LT:
          sqlOperator = "<";
          break;
        default:
          //Other operators are not supported for SQL Statements
          throw new UnsupportetOperatorException("Unsupported operator: " + operator.toUriLiteral());
        }
        //return the binary statement
        return leftSide + " " + sqlOperator + " " + rightSide;
      }

The signature for this method contains the binaryExpression as an object as well as the left and right Side of this binary expression. In this example the left side would be "a" and the right side would be "b" which we got from the visitLiteral() method. In between we set the operator in its SQL syntax.

Finally the `visitFilter()` method is called:

      @Override
      public Object visitFilterExpression(FilterExpression filterExpression, String expressionString, Object expression) {
          return "WHERE " + expression;
      }

Here we just append the WHERE at the beginning and give the whole thing back to the caller of the `accept()` method.

A simple test can show how the expression is transformed. As one can see the `UriParser` can parse a filter expression if necessary. Usually the application will already have the filter expression.

    @Test
    public void printExpression() throws Exception {
        FilterExpression expression = UriParser.parseFilter(null, null, "'a' eq 'b'");
        String whereClause = (String) expression.accept(new JdbcStringVisitor());
        System.out.println("Raw: " + rawExpression + " ------> Whereclause: " + whereClause);
        System.out.println();
      }

The output will be:

    Raw: 'a' eq 'b' ------> Whereclause: 'a' = 'b'

The implementation right now can only transform literals which will not be sufficient if you want to address a property. If an expression contains properties like "EmployeeId" we have to implement the method `visitProperty()`.

    @Override
    public Object visitProperty(PropertyExpression propertyExpression, String uriLiteral, EdmTyped edmProperty) {
      if (edmProperty == null) {
        //If a property is not found it wont be represented in the database thus we have to throw an exception
        throw new PropertyNotFoundException("Could not find Property: " + uriLiteral);
      } else {
        //It is also possible to use the mapping of the edmProperty if the name differs from the databasename
        try {
          return edmProperty.getName();
        } catch (EdmException e) {
          throw new PropertyNotFoundException(e);
        }
      }
    }

This method has an `edmProperty` in its signature which is set if the `uriLiteral` matches a property in the `EntityType`. Now we could transform an expression like `"EmployeeId eq '1'" or "EmployeeId eq '1' and ManagerId eq '2'"`. To get a validation if the property exists we have to give an `EntityType` on which the filter will be applied. In this case we use the `EntityType` "Employee".

    @Test
    public void printExpressionWithProperty() throws Exception {
      //Use a mocked edmProvider for this tutorial
      TestEdmProvider provider = new TestEdmProvider();
      Edm edm = RuntimeDelegate.createEdm(provider);
      EdmEntityType entityType = edm.getEntityType(TestEdmProvider.NAMESPACE_1, TestEdmProvider.ENTITY_TYPE_1_1.getName());

      String rawExpression = "EmployeeId eq '1'";
      FilterExpression expression = UriParser.parseFilter (null, entityType, rawExpression);
      String whereClause = (String) expression.accept(new JdbcSimpleStringVisitor());
      System.out.println("Raw: " + rawExpression + " ------> Whereclause: " + whereClause);
      System.out.println();
    }

The output will be:

    Raw: EmployeeId eq '1' ------> Whereclause: WHERE EmployeeId = '1'

Test in the sources: JdbcSimpleStringVisitorTest.class

##### Advanced Example
The implementation shown in the previous chapter can transform simple expressions. But if the expression gets more complex like `"'a' eq 'b' or ('c' eq 'd' and 'b' eq 'd')"` it won´t produce a correct where clause. The following test shows this.

    @Test
    public void compareSimpleAndAdvancedVisitor() throws Exception{
      String rawExpression = "'a' eq 'b' or ('c' eq 'd' and 'b' eq 'd')";
      FilterExpression expression = UriParser.parseFilter(null, null, rawExpression);

      String whereClauseSimple = (String) expression.accept(new JdbcSimpleStringVisitor());
      String whereClauseAdvanced = (String) expression.accept(new JdbcAdvancedStringVisitor());
      System.out.println("Simple: " + whereClauseSimple + " ------> Advanced: " + whereClauseAdvanced);
    }

The output will be:

    Simple: WHERE 'a' = 'b' OR 'c' = 'd' AND 'b' = 'd' ------> Advanced: WHERE 'a' = 'b' OR ('c' = 'd' AND 'b' = 'd')

In the simple implementation the brackets will not be transformed. To fix this the method `visitBinary()` has to be enhanced.

    @Override
    public Object visitBinary(BinaryExpression binaryExpression, BinaryOperator operator, Object leftSide, Object rightSide) {
      String actualLeftSide = leftSide.toString();
      String actualRightSide = rightSide.toString();
      if (leftSide instanceof Expression) {
        //If something is lower in the tree and is of the type AND or OR it needs brackets to show the higher priority
        if (BinaryOperator.AND.equals(((Expression) leftSide).getOperator()) || BinaryOperator.OR.equals(((Expression) leftSide).getOperator())) {
        actualLeftSide = "(" + leftSide + ")";
        }
      }
      if (rightSide instanceof Expression) {
        //If something is lower in the tree and is of the type AND or OR it needs brackets to show the higher priority
        if (BinaryOperator.AND.equals(((Expression) rightSide).getOperator()) || BinaryOperator.OR.equals(((Expression) rightSide).getOperator())) {
        actualRightSide = "(" + rightSide + ")";
        }
      }
      //Transform the OData filter operator into an equivalent sql operator
      String sqlOperator = "";
      switch (operator) {
      case EQ:
        sqlOperator = "=";
        break;
      case NE:
        sqlOperator = "<>";
        break;
      case OR:
        sqlOperator = "OR";
        break;
      case AND:
        sqlOperator = "AND";
        break;
      case GE:
        sqlOperator = ">=";
        break;
      case GT:
        sqlOperator = ">";
        break;
      case LE:
        sqlOperator = "<=";
        break;
      case LT:
        sqlOperator = "<";
        break;
      default:
        //Other operators are not supported for SQL  Statements
        throw new UnsupportetOperatorException("Unsupported operator: " + operator.toUriLiteral());
      }

      //return the binary statement
      return new Expression(actualLeftSide + " " + sqlOperator + " " + actualRightSide, operator);
    }

Since simple strings cannot show this complexity a new private class "Expression" was introduced. The signature of all `visit()` methods accept any kind of object so we can mix in the new class. Now we only have to check if one side of the tree is of type String or Expression.

Test in the sources: JdbcAdvancedStringVisitorTest.class

##### Example with prepared Statements
Since string concatenation is very vulnerable against SQL Injection a best practice is to use prepared statements. This can be a tough challenge in this case because not only the value of `EmployeeId` is supplied in the filter expression but the field EmployeeId and the operator as well. Prepared Statements don´t allow statements like `"WHERE ? ? ?"` thus we have to find a way to prepare the prepared statements in advance which can be very complex as the following example will show: The filter expression `"EmployeeId eq '1' and ManagerId eq '2'"` is the same as `"ManagerId eq '2' and EmployeeId eq '1'"` but the prepared statement will always look like `"…. WHERE EmployeeId = ? and ManagerId = ?"`.

This tutorial will not solve this problem. Instead it will show a first idea on how to implement such a prepared statement visitor. Again the Where clause will be created using string concatenation. But this time we will replace literals with a "?". These questionmarks can be set in a prepared statement. The methods `visitLiteral` and `visitProperty` will just return their value while the `visitBinary` will contain the logic.

VisitLiteral:

    @Override
    public Object visitLiteral(final LiteralExpression literal, final EdmLiteral edmLiteral) {
      //Sql Injection is not possible anymore since we are using prepared statements. Thus we can just give back the edmLiteral content
      return edmLiteral.getLiteral();
    }

VisitProperty:

    @Override
    public Object visitProperty(final PropertyExpression propertyExpression, final String uriLiteral, final EdmTyped edmProperty) {
      if (edmProperty == null) {
        //If a property is not found it wont be represented in the database thus we have to throw an exception
        throw new PropertyNotFoundException("Could not find Property: " + uriLiteral);
      } else {
        //To distinguish between literals and properties we give back the whole edmProperty in this case
        return edmProperty;
      }
    }

In addition we will use the following class as a container for the where clause and the parameters:

    public class Expression {
      private String preparedWhere;
      private List<Object> parameters;
      private BinaryOperator operator;

      public Expression(final BinaryOperator operator) {
        preparedWhere = "";
        parameters = new ArrayList<Object>();
        this.operator = operator;
      }

      public void addParameter(final Object parameter) {
      parameters.add(parameter);
      }

      public void setPrepeparedWhere(final String where) {
        preparedWhere = where;
      }

      public List<Object> getParameters() {
        return parameters;
      }

      public BinaryOperator getOperator() {
        return operator;
      }

      @Override
      public String toString() {
        return preparedWhere;
      }
    }

Finally the visitBinary method:

    @Override
    public Object visitBinary(final BinaryExpression binaryExpression, final BinaryOperator operator, final Object leftSide, final Object rightSide) {
      //Transform the OData filter operator into an equivalent sql operator
      String sqlOperator = "";
      switch (operator) {
      case EQ:
        sqlOperator = "=";
        break;
      case NE:
        sqlOperator = "<>";
        break;
      case OR:
        sqlOperator = "OR";
        break;
      case AND:
        sqlOperator = "AND";
        break;
      case GE:
        sqlOperator = ">=";
        break;
      case GT:
        sqlOperator = ">";
        break;
      case LE:
        sqlOperator = "<=";
        break;
      case LT:
        sqlOperator = "<";
        break;
      default:
        //Other operators are not supported for SQL Statements
        throw new UnsupportetOperatorException("Unsupported operator: " + operator.toUriLiteral());
      }
      //The idea is to check if the left side is of type property. If this is the case we append the property name and the operator to the where clause
      if (leftSide instanceof EdmTyped && rightSide instanceof String) {
        Expression expression = new Expression(operator);
        try {
          expression.setPrepeparedWhere(((EdmTyped)   leftSide).getName() + " " + sqlOperator + " ?");
        } catch (EdmException e) {
          throw new RuntimeException("EdmException occured");
        }
        expression.addParameter(rightSide);
        return expression;
      } else if (leftSide instanceof Expression && rightSide instanceof Expression) {
        Expression returnExpression = new Expression(operator);
        Expression leftSideExpression = (Expression) leftSide;
        if (BinaryOperator.AND.equals    (leftSideExpression.getOperator()) || BinaryOperator.OR.equals(leftSideExpression.getOperator())) {
        leftSideExpression.setPrepeparedWhere("(" + leftSideExpression.toString() + ")");
        }
        Expression rightSideExpression = (Expression) rightSide;
        if (BinaryOperator.AND.equals(rightSideExpression.getOperator()) || BinaryOperator.OR.equals(rightSideExpression.getOperator())) {
        rightSideExpression.setPrepeparedWhere("(" + rightSideExpression.toString() + ")");
        }
        returnExpression.setPrepeparedWhere(leftSideExpression.toString() + " " + sqlOperator + " " + rightSideExpression.toString());

        for (Object parameter : leftSideExpression.getParameters()) {
        returnExpression.addParameter(parameter);
        }
        for (Object parameter : rightSideExpression.getParameters()) {
          returnExpression.addParameter(parameter);
        }
        return returnExpression;
      } else {
        throw new RuntimeException("Not right format");
      }
    }

This implementation will work if the filter supplies a property on the left side and a literal on the right side. The output of this new implementation could look like this:

    Raw: EmployeeId eq '1' or (ManagerId eq '2' and TeamId eq '3') ------> Whereclause: EmployeeId = ? OR (ManagerId = ? AND TeamId = ?)
    1
    2
    3

Test in the sources: JdbcPreparedStatementVisitorTest.class

### Further Information
Documentation about how to create such a filter expression can be found under [http://www.odata.org](http://www.odata.org "External Link") in the OData protocol specification.

Visitor pattern: [http://en.wikipedia.org/wiki/Visitor_pattern](http://en.wikipedia.org/wiki/Visitor_pattern "External Link")


text:
# Latest Log

## [1.20.7](https://github.com/alibaba-fusion/next/compare/1.20.6...1.20.7) (2020-05-28)

### Bug Fixes

* **Breadcrumb:** separator icon's vertical align should be middle ([6f947a5](https://github.com/alibaba-fusion/next/commit/6f947a5))
* **Drawer:** title should be react node ([080ac7b](https://github.com/alibaba-fusion/next/commit/080ac7b))
* **Input:** IE10 can't find getDerivedStateFromProps ([8ef2c4c](https://github.com/alibaba-fusion/next/commit/8ef2c4c))
* **Typescript:** add types for Button iconSize ([#1882](https://github.com/alibaba-fusion/next/issues/1882)) ([8df9bac](https://github.com/alibaba-fusion/next/commit/8df9bac))
* **Typescript:** update typescript, close [#1875](https://github.com/alibaba-fusion/next/issues/1875) ([b3a3a26](https://github.com/alibaba-fusion/next/commit/b3a3a26))
* **Upload:** enhance robustness ([53b3cd7](https://github.com/alibaba-fusion/next/commit/53b3cd7))
* **Upload:** support upload a directory, close [#1877](https://github.com/alibaba-fusion/next/issues/1877) ([c3a2b7b](https://github.com/alibaba-fusion/next/commit/c3a2b7b))

### Features

* **Overlay:** support render overlay in shadow dom ([#1869](https://github.com/alibaba-fusion/next/issues/1869)) ([ba100cc](https://github.com/alibaba-fusion/next/commit/ba100cc))
* **Transfer:** add API showCheckAll & add demo ([d55c82d](https://github.com/alibaba-fusion/next/commit/d55c82d))

repository: chunlei/next

repo_name: chunlei/next


text:
# Installation

Recommended way to install SixModLoader is using [SixModLoader.Launcher](https://github.com/SixModLoader/SixModLoader.Launcher).

0. Install [.NET Core 3.1+](https://dotnet.microsoft.com/download/dotnet-core) *or not recommended [Mono](https://www.mono-project.com/download/stable/)*
1. Download launcher from [release page](https://github.com/SixModLoader/SixModLoader.Launcher/releases/latest)
   *(recommended .net core `linux` or `win`, or `net472` for mono/windows only, but only if you have to)*
2. Extract launcher executable from zip to server directory
3. Just run `./SixModLoader launch "LocalAdmin 7777"` *(or `SixModLoader.exe launch "LocalAdmin.exe 7777"` if you are winblows masochist)*<br>
It will download/update SixModLoader, configure Doorstop and launch SCP:SL with mod loader injected!

*TIP for people using hosting providers, if your provider allows replacing LocalAdmin/MultiAdmin, you can replace it with launcher and write your arguments in `jumploader.txt`*

Manual installation is a bit more complicated.
1. Extract [SixModLoader zip](https://github.com/SixModLoader/SixModLoader/releases/latest) to zip to server directory
2. Download Doorstop ([Unix](https://github.com/NeighTools/UnityDoorstop.Unix) or [Windows](https://github.com/NeighTools/UnityDoorstop))
3. Configure Doorstop to load `SixModLoader/bin/SixModLoader.dll`
4. Launch and you should be done *but without auto updates :(*

repository: SixModLoader/SixModLoader

repo_name: SixModLoader/SixModLoader


text:
# OpenCloud

[install-php]: https://www.php.net/manual/en/features.file-upload.php#107406
[install-ms]: #link
[install-ios]: #link
[install-android]: #link
[wiki]: https://ilosey14.github.io/wiki/open-cloud

⛅ *Roll your own cloud storage.*

**OpenCloud** aims to be a plug-and-play cloud storage api for your own server space.
The goal is to provide your preferred backend environment with a headless
content management system (CMS) API and one-off session authentication.

Given a compatible frontend, start by authorizing one-time login access on your server.
An authorized session generates login tokens and an optional qr code.
Frontend login flow should prompt these values before passing them to the api,
return valid access tokens on authentication.

## Generating a login session

Generate a one-time login session.
The following registers this to the access control list (ACL) and outputs session info for immediate use.
The ACL session is immediately deleted after successful login.

```bash
sudo bash login
```

---

## Installation

<!--
### Client (Desktop/Mobile)

[💼 Microsoft Store][install-ms] \
[🍎 App Store][install-ios] \
[▶ Play Store][install-android]
-->

### ◻ Apache 2 / PHP 7.X

Configure the `php.ini` for file uploads.

[PHP.net Reference][install-php]

```ini
file_uploads = on
session.gc_maxlifetime = 10800
max_input_time = 10800
max_execution_time = 10800
upload_max_filesize = 110M
post_max_size = 120M
```

<!-- *other languages on apache...* -->

### ◻ Node.js

TODO...

### ◻ Other Backends

TODO...

---

## API

[Wiki: TODO][wiki]

## TODO

- [x] CRUD API
- [x] Database API
- [x] Authentication
- [x] Login session generation
- [ ] Wiki
  - Install/plug-in-play guide
  - Recommendations/config
- [ ] Recycling Bin for deleted items


text:
# Self Study

Author: Jedi Chou, Create at 2019.12.20 8:02 AM

* Daily study
  * Check-in
    * 101WeiQi daily exercise check-in. 9:09
    * Read articles and check interview invitation (MaiMai APP). 9:10
    * NowCoder check-in and do an exercise that contains 20 puzzles. 9:13
    * Don't memorize words APP check-in. 9:13
    * 163 music APP check-in. 9:13
    * Tencent cartoon APP check-in. 9:13
    * Exercise of Vocabulary (vocabulary.com). 9:21

  * Micro habit
    * Read Sina Blog. 9:36
    * Reading 1 minutes at SIMPLE Wiki. 9:40

  * Reading
    * Feedly RSS. 10:06
    * 163.com mail. 11:38

* Weekly review
  * Read a thesis and don't record
  * Open class APP by 163.com check-in

## Other

* VS Code plugin
  * Debugger for Microsoft Edge

* 配色网站
  * [Material Palette](https://www.materialpalette.com/)
  * [Flat UI Color Picker](http://www.flatuicolorpicker.com/)
  * [Flat UI Colors 2](https://flatuicolors.com/)
  * [COOLORS](http://coolors.co/)
  * [brand colors](http://brandcolors.net/)
  * [material ui](https://www.materialui.co/colors)
  * [Adobe Color](https://color.adobe.com/zh/create)

* Icon Type
  * Action
  * Alert
  * AV
  * Cmmunication
  * Content
  * Device
  * Edtior
  * File
  * Hardware
  * Image
  * Maps
  * Navigation
  * Notification
  * Place
  * Social
  * Toggle

repository: JediChou/jedi-md-book

repo_name: JediChou/jedi-md-book


text:
# TCP Benchmarks

This directory contains a standardized TCP benchmark. This helps to evaluate the
performance of netstack and native networking stacks under various conditions.

## `tcp_benchmark`

This benchmark allows TCP throughput testing under various conditions. The setup
consists of an iperf client, a client proxy, a server proxy and an iperf server.
The client proxy and server proxy abstract the network mechanism used to
communicate between the iperf client and server.

The setup looks like the following:

```
 +--------------+  (native)            +--------------+
 | iperf client |[lo @ <IP_ADDRESS>]------>| client proxy |
 +--------------+                      +--------------+
                                    [client.0 @ <IP_ADDRESS>]
                            (netstack)  |            |  (native)
                                        +------+-----+
                                               |
                                             [br0]
                                               |
          Network emulation applied ---> [wan.0:wan.1]
                                               |
                                             [br1]
                                               |
                                        +------+-----+
                            (netstack)  |            |  (native)
                                     [server.0 @ <IP_ADDRESS>]
 +--------------+                      +--------------+
 | iperf server |<------[lo @ <IP_ADDRESS>]| server proxy |
 +--------------+            (native)  +--------------+
```

Different configurations can be run using different arguments. For example:

*   Native test under normal internet conditions: `tcp_benchmark`
*   Native test under ideal conditions: `tcp_benchmark --ideal`
*   Netstack client under ideal conditions: `tcp_benchmark --client --ideal`
*   Netstack client with 5% packet loss: `tcp_benchmark --client --ideal --loss
    5`

Use `tcp_benchmark --help` for full arguments.

This tool may be used to easily generate data for graphing. For example, to
generate a CSV for various latencies, you might do:

```
rm -f /tmp/netstack_latency.csv /tmp/native_latency.csv
latencies=$(seq 0 5 50;
            seq 60 10 100;
            seq 125 25 250;
            seq 300 50 500)
for latency in $latencies; do
  read throughput client_cpu server_cpu <<< \
    $(./tcp_benchmark --duration 30 --client --ideal --latency $latency)
  echo $latency,$throughput,$client_cpu >> /tmp/netstack_latency.csv
done
for latency in $latencies; do
  read throughput client_cpu server_cpu <<< \
    $(./tcp_benchmark --duration 30 --ideal --latency $latency)
  echo $latency,$throughput,$client_cpu >> /tmp/native_latency.csv
done
```

Similarly, to generate a CSV for various levels of packet loss, the following
would be appropriate:

```
rm -f /tmp/netstack_loss.csv /tmp/native_loss.csv
losses=$(seq 0 0.1 1.0;
         seq 1.2 0.2 2.0;
         seq 2.5 0.5 5.0;
         seq 6.0 1.0 10.0)
for loss in $losses; do
  read throughput client_cpu server_cpu <<< \
    $(./tcp_benchmark --duration 30 --client --ideal --latency 10 --loss $loss)
  echo $loss,$throughput,$client_cpu >> /tmp/netstack_loss.csv
done
for loss in $losses; do
  read throughput client_cpu server_cpu <<< \
    $(./tcp_benchmark --duration 30 --ideal --latency 10 --loss $loss)
  echo $loss,$throughput,$client_cpu >> /tmp/native_loss.csv
done
```

repository: google/gvisor

gha_language: Go

repo_name: google/gvisor


text:
election:
--------
- in election at most one master will be selected
- for now we can save the master's identity in a key in redis (it wont be used for now, might find some use later)
- detail of election process is same as we follow in internal codebase
- anyone participating election will register itself in a set in redis in every cycle

master server:
-------------
- this is the server which has won election
- participates election regularly, increases ttl of election key if elected consecutive times
- master will keep pinging nodes from the registration set
- if the registration set changes a redistribution will kick in
- if master does not get response from some worker for consecutive two times, it will remove node from registration list and do a redistribution
redistribution will be done by master node and will be persisted in a redis key

worker node:
-----------
- they will participate in election regularly
- they will listen to the distribution key for any update and based on that they will process work units

when worker dies:
----------------
master's ping check will fail, so master will remove the dead node from registration list and trigger a redistribution

when master dies:
----------------
- until the next election things won't change,
- to avoid work computation loss, we can keep redundant computation meaning same computation can happen in more than one nodes
- in the next election cycle a new master will be selected and new work distribution will happen

if redis connection goes down:
-----------------------------
we have two choices,
 - we can stop everything and crash
 - we can continue doing things that are already assigned,
   once redis comes back up reestablish connections and subscriptions and continue as usual


text:
---
layout: default
title: "Frequently Asked Questions"
permalink: /resources/faq.html
---

How do I integrate payments with my website?

The easiest way to integrate payments on your website is to link to one of our pre-built hosted payment pages. We also have a WooCommerce plugin you can use if you have a WordPress site. If you need to build a custom payment page, refer to our Getting Started Guide.

Can I embed a hosted payment page in my website using an <iframe>?

Yes, but first you have to <a<EMAIL_ADDRESS>AffiniPay support and provide us with your site's domain name. An administrator must enable iframes in your merchant account and send you a code snippet to embed in your site's HTML.

Here's an example code snippet:

<div id="amp-embed"></div>
<script src="https://cdn.affinipay.com/libs/amp-embed.min.js"></script>
<script>
  AMP.Embed.init({
    publicKey: 'm_7LwaBGj6T52TyR60xvjgzg'
    containerID: 'amp-embed'
  });
</script>

<button id="btn" class="btn copy" style="margin: 13px 13px 0 13px;" data-clipboard-target="#iframe" onclick="Materialize.toast('Copied!', 2000)">Copy</button>

The script generates an iframe inside the #amp-embed element. However, you must add a ?page=payment-page-path parameter to the end of the page URL. For example, the page secure.affinipay.com/pages/merchant/payment requires ?page=payment at the end of the URL.

Important: When using WordPress or another Content Management System (CMS), if you're using a text field to embed the iframe script code snippet, make sure the snippet is being added to the bottom of the page and not inside the header element.

How do I integrate payments with my application?

First, you need a registered AffiniPay partner application, which provides a client ID and secret necessary for initiating an OAuth flow. The OAuth flow is what enables users to grant your application access to their AffiniPay merchant account and perform actions on their behalf (e.g., running charges). Refer to the Getting Started Guide for more information.

What do I do after tokenizing payment details with your JS library?

POST the token and payment amount to your server, which must submit the charge to the AffiniPay Payment Gateway using your protected secret key. Refer to Create a Charge for more information.

Where can I find my API keys?

To submit API requests to the AffiniPay Payment Gateway, you must use credentials from an authorized AffiniPay application. By default, AffiniPay creates an application for you when you sign up for a merchant account called My Website. This application contains the credentials you must use to submit charges to the payment gateway. Refer to Obtain Credentials for more information.

How do I update a saved credit card?

Currently, we only support creating a new saved card/bank from an existing card/bank or one-time token and not directly updating or patching an existing saved card/bank.

You can achieve the desired result, however, by first creating a new one-time token using the existing saved card/bank ID (along with the updated properties) and then creating a new saved card/bank from this one-time token. Click here for instructions.

How do I report security issues?

Although we do our best to ensure that our software meets rigorous security requirements, bugs and vulnerabilities are inevitable. If you think you've discovered a vulnerability in our platform, please email us: <a href="mailto:security<EMAIL_ADDRESS>You can also use our PGP key to contact us securely. We ask that you allow us an opportunity to address any issues before disclosing them to the public.

repository: jeff-matthews/jekyll-dev-docs

repo_name: jeff-matthews/jekyll-dev-docs


text:
Blubr
=====

A [logr](https://github.com/go-logr/logr) implementation using [Logrus](https://github.com/sirupsen/logrus).

The Name
--------

It's a small reference to the walrus theme of Logrus. It's also fun to say.

Origin
------

Blubr was originally developed for the purpose of using [Logrus](https://github.com/sirupsen/logrus) to provide logging for the [Mattermost Operator](https://github.com/mattermost/mattermost-operator). In order to do so, we developed the necessary interfaces to meet the [Operator SDK](https://github.com/operator-framework/operator-sdk/blob/master/doc/user/logging.md#operator-sdk-logging) requirements for logging.

Todo
----

A few of the interfaces require additional work to finalize them. Todo comments call out most of these tasks.

repository: mattermost/mattermost-operator

repo_name: mattermost/mattermost-operator


text:
---
layout: post
title: "Disable Right Click Zone In Linux On A Synaptics Touchpad"
quote: "Use that beautiful touchpad the way god intended"
excerpt: "By default the Synaptics driver on Linux will treat the bottom right corner of your touchpad as a right click button, even if you have a macbook like trackpad or Logitech T650 touchpad. I have become used to clicking with 2 fingers for a right click, so the that zone is more annoying than useful. Here's how to disable it."
image: /media/2015-10-08-Touchegg/cover.jpg
video: false
comments: true
categories:
- Tutorial
tags:
- touchegg
- logitech t650 touchpad
- apple magic trackpad
- multitouch gestures
- linux
- rightclick
---
Support for modern touchpads in Linux has been steadily improving in the last few years, but the default settings are still tailored towards yesteryears awkward touchpads rather the glass beauties you find in a macbook or high end dell. In a [previous post]({% post_url 2015-10-08-touchegg %}) I explained how you can enable multitouch gestures. This post explains how you can disable the right click zone, and just rely on a two finger click for the "right click" menu.

The right click zone is configured in the Synaptics driver. You can disable it using the `synclient` utility on a per session basis, or by tweaking `xorg.conf` if you want to save the settings.

## Disable for current session
Using `synclient`, you can disable the right click zone in the current session by running the following commands:

```bash
synclient RightButtonAreaLeft=0
synclient RightButtonAreaTop=0
```

You can run `synclient -l` to see if the settings have been properly applied.

## Disable for all sessions
To disable the right click zone permanently follow these steps:

1. Move the Synaptics configuration to a new file to prevent a package update from overwriting your changes

    ```bash
    mv /usr/share/X11/xorg.conf.d/50-synaptics    /usr/share/X11/xorg.conf.d/40-synaptics
    ```

2. Comment out the `Default clickpad buttons` block in `/usr/share/X11/xorg.conf.d/40-synaptics` to disable the right click zone:

    ```bash
    # Section "InputClass"
    #        Identifier "Default clickpad buttons"
    #        MatchDriver "synaptics"
    #        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
    #        Option "SecondarySoftButtonAreas" "58% 0 0 15% 42% 58% 0 15%"
    # EndSection
    ```

    Alternatively, if you don't have the default ubuntu config, you can add the following block to the file:

    ```bash
    Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "RightButtonAreaLeft" "0"
        Option "RightButtonAreaTop" "0"
    EndSection
    ```

## Further Reading

* [Disable Right Click Synaptics](http://kernpanik.com/geekstuff/2015/01/12/disable-rightclick-synaptics.html)

gha_language: CSS


text:
# _Hair Salon_

#### _Web App to track stylists and clients at a hair salon, 23 September 2016_

#### By _**Joshua Huffman**_

## Description

_This web app allows a user to create a list of stylists at a hair salon and add clients for each stylist.  This project is the Independent Project for week 3 of the Epicodus PHP class._

## Setup/Installation Requirements

* Clone repository to computer
* Run composer install to install project dependencies
* Import hair_salon database into SQL server
* Start a web server with web/ as the root, can run "php -S localhost:8000" from terminal while within hair-salon-ip/web
* Visit localhost:8000 on your web browswer

## Database Setup
If unable to import database run the following SQL commands to recreate database

  * CREATE DATABASE hair_salon;
  * USE hair_salon;
  * CREATE TABLE stylists (id serial PRIMARY KEY, name varchar (255));
  * CREATE TABLE clients (id serial PRIMARY KEY, name varchar (255), stylist_id int);

  Then copy hair_salon structure to hair_salon_test

## Specifications
  Behavior|Input|Output
  --------|-----|------
  create a new Stylist|Sandra|Sandra
  delete a Stylist|Sandra|0
  list all stylists|Stylists|Sandra, Becky
  delete all stylists|Delete Stylists|0
  create a new Client|Jim|Jim
  delete a Client|Jim|0
  associate a client with a stylist|Jim to Sandra|Sandra:Jim
  list all clients for a stylist|Sandra|Jim, Bill
  delete all clients|Delete Clients|0

## Known Bugs
  SQL queries are not sanitized, will refactor project to use PDO prepared statements instead of directly inserting user input into queries. In current state project is vulnerable to SQL injection and some inputs will just cause query to fail.

## Support and contact details

_Contact me at<EMAIL_ADDRESS>with any comments or questions_

## Technologies Used

_This project is writen in PHP and uses the silex framework and twig for templating._

### License

*MIT License*

Copyright (c) 2016 **_Joshua Huffman_**

repository: joshgh/hair-salon-ip

repo_name: joshgh/hair-salon-ip


text:
![Test Coveralls](https://github.com/biothings/biomedical_id_resolver.js/workflows/Test%20Coveralls/badge.svg)
[![Coverage Status](https://coveralls.io/repos/github/biothings/biomedical_id_resolver.js/badge.svg?branch=main)](https://coveralls.io/github/biothings/biomedical_id_resolver.js?branch=main)
![npm](https://img.shields.io/npm/dw/biomedical_id_resolver)
![GitHub issues](https://img.shields.io/github/issues/biothings/biomedical_id_resolver.js)
![NPM](https://img.shields.io/npm/l/biomedical_id_resolver)
![npm](https://img.shields.io/npm/v/biomedical_id_resolver?style=plastic)
![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/biothings/biomedical_id_resolver.js)

# biomedical_id_resolver.js
js library for resolving biological ids to their equivalent ids in batch

## Install

```
$ npm i biomedical_id_resolver
```

## Usage

```js
const resolve = require('biomedical_id_resolver');

// input should be an object, with semantic type as the key, and array of CURIEs as value
let input = {
    "Gene": ["NCBIGene:1017", "NCBIGene:1018", "HGNC:1177"],
    "SmallMolecule": ["CHEBI:15377"],
    "Disease": ["MONDO:0004976"],
    "Cell": ["CL:0002372"]
  };

(async () => {
  const resolver = new resolve();
	console.log(await resolver.resolve(input);
	//=> {'NCBIGene:1017': {...}, 'NCBIGene:1018': {...}, 'HGNC:1177': {...}, 'CHEBI:15377': {...}, 'MONDO:0004976': {...}, 'CL:0002372': {...}}
})();

```

## Output Schema

- Output is a javascript Object

- The root keys are CURIES (e.g. NCBIGene:1017) which are passed in as input

- The values represents resolved identifiers

- Each CURIE will have 4 required fields

  - id: the primary id (selected based on the ranking described in the next section) and label

  - curies: an array, each element represents a resolved id in CURIE format

  - type: the semantic type of the identifier

  - db_ids: original ids from source database, could be curies or non-curies.

- if an ID can not be resolved using the package, it will have an additional field called "flag", with value equal to "failed"

- Example Output

```json
{
  "NCBIGene:1017": {
    "id": {
      "label": "cyclin dependent kinase 2",
      "identifier": "NCBIGene:1017"
    },
    "db_ids": {
      "NCBIGene": [
        "1017"
      ],
      "ENSEMBL": [
        "ENSG00000123374"
      ],
      "HGNC": [
        "1771"
      ],
      "SYMBOL": [
        "CDK2"
      ],
      "UMLS": [
        "C1332733",
        "C0108855"
      ],
      "name": [
        "cyclin dependent kinase 2"
      ]
    },
    "type": "Gene",
    "curies": [
      "NCBIGene:1017",
      "ENSEMBL:ENSG00000123374",
      "HGNC:1771",
      "SYMBOL:CDK2",
      "UMLS:C1332733",
      "UMLS:C0108855"
    ]
  }
}
```

## Query Using [SRI node normalizer](https://nodenormalization-sri-dev.renci.org/1.1/docs#/)
### Usage
```js
const resolver = require('biomedical_id_resolver');

// input must be an object, with semantic type as the key, and array of CURIEs as value
let input = {
    "Gene": ["NCBIGene:1017", "NCBIGene:1018", "HGNC:1177"],
    "SmallMolecule": ["CHEBI:15377"],
    "Disease": ["MONDO:0004976"],
    "Cell": ["CL:0002372"]
};

(async () => {
  let res = await resolver.resolveSRI(input);
  console.log(res);
})();
```

### Example Output
The output contains `id` and `equivalent_identifiers` straight from SRI as well as the same fields as the base resolver to make it backwards compatible with it.
```json
{
  "NCBIGene:1017": [
    {
      "id": {
        "identifier": "NCBIGene:1017",
        "label": "CDK2"
      },
      "equivalent_identifiers": [
        {
          "identifier": "NCBIGene:1017",
          "label": "CDK2"
        },
        {
          "identifier": "ENSEMBL:ENSG00000123374"
        },
        {
          "identifier": "HGNC:1771",
          "label": "CDK2"
        },
        {
          "identifier": "OMIM:116953"
        },
        {
          "identifier": "UMLS:C1332733",
          "label": "CDK2 gene"
        }
      ],
      "type": [
        "biolink:Gene",
        "biolink:GeneOrGeneProduct",
        "biolink:BiologicalEntity",
        "biolink:NamedThing",
        "biolink:Entity",
        "biolink:MacromolecularMachineMixin"
      ],
      "primaryID": "NCBIGene:1017",
      "label": "CDK2",
      "attributes": {},
      "semanticType": "Gene",
      "semanticTypes": [
        "biolink:Gene",
        "biolink:GeneOrGeneProduct",
        "biolink:BiologicalEntity",
        "biolink:NamedThing",
        "biolink:Entity",
        "biolink:MacromolecularMachineMixin"
      ],
      "dbIDs": {
        "NCBIGene": [
          "1017"
        ],
        "ENSEMBL": [
          "ENSG00000123374"
        ],
        "HGNC": [
          "1771"
        ],
        "OMIM": [
          "116953"
        ],
        "UMLS": [
          "C1332733"
        ],
        "name": [
          "CDK2",
          "CDK2 gene"
        ]
      },
      "curies": [
        "NCBIGene:1017",
        "ENSEMBL:ENSG00000123374",
        "HGNC:1771",
        "OMIM:116953",
        "UMLS:C1332733"
      ]
    }
  ]
}
```
## Available Semantic Types & prefixes

> Gene, Transcript, Protein ID resolution is done through MyGene.info API

- Gene
  1. NCBIGene
  2. ENSEMBL
  3. HGNC
  4. MGI
  5. OMIM
  6. UMLS
  7. SYMBOL
  8. UniProtKB
  9. name

- Transcript
  1. ENSEMBL
  2. SYMBOL
  3. name

- Protein
  1. UniProtKB
  2. ENSEMBL
  3. UMLS
  4. SYMBOL
  5. name

> Variant ID resolution is done through MyVariant.info API

- SequenceVariant
  1. CLINVAR
  2. DBSNP
  3. HGVS
  4. MYVARIANT_HG19

> SmallMolecule, Drug ID resolution is done through MyChem.info API

- SmallMolecule
    1. PUBCHEM.COMPOUND
    2. CHEMBL.COMPOUND
    3. UNII
    4. CHEBI
    5. DRUGBANK
    6. MESH
    7. CAS
    8. HMDB
    9. KEGG.COMPOUND
    10. INCHI
    11. INCHIKEY
    12. UMLS
    13. LINCS
    14. name

- Drug
    1. RXCUI
    2. NDC
    3. DRUGBANK
    4. PUBCHEM.COMPOUND
    5. CHEMBL.COMPOUND
    6. UNII
    7. CHEBI
    8. MESH
    9. CAS
    10. HMDB
    11. KEGG.COMPOUND
    12. INCHI
    13. INCHIKEY
    14. UMLS
    15. LINCS
    16. name

> Disease, ClinicalFinding ID Resolution is done through MyDisease.info API

- Disease

  1. MONDO
  2. DOID
  3. OMIM
  4. ORPHANET
  5. EFO
  6. UMLS
  7. MESH
  8. MEDDRA
  9. NCIT
  10. SNOMEDCT
  11. HP
  12. GARD
  13. name

- ClinicalFinding
  1. LOINC
  2. NCIT
  3. EFO
  4. name

> Pathway ID Resolution is done through biothings.ncats.io/geneset API

- Pathway
  1. GO
  2. REACT
  3. KEGG
  4. SMPDB
  5. PHARMGKB.PATHWAYS
  6. WIKIPATHWAYS
  7. BIOCARTA
  8. name

> MolecularActivity ID Resolution is done through BioThings Gene Ontology Molecular Activity API

- MolecularActivity
  1. GO
  2. REACT
  3. RHEA
  4. MetaCyc
  5. KEGG.REACTION
  6. name

> CellularComponent ID Resolution is done through BioThings Gene Ontology Cellular Component API

- CellularComponent
  1. GO
  2. MetaCyc
  3. name

> BiologicalProcess ID Resolution is done through BioThings Gene Ontology Biological Process API

- BiologicalProcess

  1. GO
  2. REACT
  3. MetaCyc
  4. KEGG
  5. name

> AnatomicalEntity ID Resolution is done through BioThings UBERON API

- AnatomicalEntity
  1. UBERON
  2. UMLS
  3. MESH
  4. NCIT
  5. name

> PhenotypicFeature ID Resolution is done through BioThings HPO API

- PhenotypicFeature
  1. HP
  2. EFO
  3. NCIT
  4. UMLS
  5. MEDDRA
  6. MP
  7. SNOMEDCT
  8. MESH
  9. name

> Cell ID Resolution is done through Biothings Cell Ontology API

- Cell
  1. CL
  2. NCIT
  3. MESH
  4. EFO
  5. name

## Development

1. Install Node 12 or later. You can use the [package manager](https://www.npmjs.com/) of your choice.
   Tests need to pass in Node 12 and 14.
2. Clone this repository.
3. Run `npm ci` to install the dependencies.
4. scripts are stored in `/src` folder
5. Add test to `/__tests__` folder
6. run `npm run release` to bump version and generate change log
7. run `npx depcheck` to check for unused packages in package.json

## CHANGELOG

See [CHANGELOG.md](https://github.com/biothings/biomedical_id_resolver.js/blob/main/CHANGELOG.md)


text:
---
site: freiburg
tags: [galaxy, MMOS, devops, society]
title: Massively Multiplayer Online Science in Galaxy to help Ecologists - A Citizen Science Project
author_github: bgruening
---

A few years back the "webhook" concept was contributed to the Galaxy codebase.
Webhooks have enabled custom modifications to the Galaxy User Interface (UI) without changing the Galaxy source code directly.
The advantage is that the maintenance of custom changes is a lot easier. You can think of it as a plugin concept for the Galaxy UI.
Webhooks are powering the so much-loved XKCD or PhD comics that you see when you submit jobs or workflows for example - but this was just the first step...

Today we are proud to announce an extension that enables __you__ to help our ecology community to annotate existing datasets by classifying hoverflies into males and females.
This information will help ecologist in further data analysis, for example, to train future machine learning models. [Try it!](https://usegalaxy.eu/gapars-experiment)

    SPIPOLL datasets and first MMOS task on usegalaxy.eu

#### Crowdsourcing in science is hard

Crowdsourcing in science is hard. Nevertheless, there are tremendous opportunities where we all can advance science and help our colleagues. Here we are trying to overcome the challenges by embedding the crowdsourcing tasks into Galaxy.

The idea is to provide a simple framework to create and manage tasks and data via the [MMOS platform](http://mmos.ch),
together with a simple [UI](https://usegalaxy.eu/gapars-experiment) and integrate this into [Galaxy Europe](https://usegalaxy.eu).

For us, it made a lot of sense to offer the hoverfly images after you execute a job or a workflow. Usually, you need to wait a few seconds until your job is processed and you can study the results. During this time you can now classify marmalade hoverflies as a means to procrastinate - but in a very meaningful way and the
good feeling to have helped our colleagues from ecology :)

The work done by __you__ with this new Galaxy feature allows ecologist to extend metadata on already captured data
and will open new ways to analyse them. For example by
training machine learning models or taking into account new factors (here the sex of the flies) in statistical modelling.

#### How it works and how to create new plugins of this kind

The [SPIPOLL](https://www.spipoll.org) citizen science project collects images from different species, in our case from [Episyrphus balteatus](https://en.wikipedia.org/wiki/Episyrphus_balteatus), also called marmalade hoverfly. In a second step, a project manager prepares certain tasks and seeks for community contribution to help to classify the images.
As a first project, we have chosen to help with the task to classify ~5000 hoverfly images and identify the sex by looking at their relative eyes position.

To make it really simple for __you__ to contribute to such a crowdsourcing project we teamed up with the project [Massively Multiplayer Online Science](http://gapars.mmos.ch), which is a citizen science platform connecting scientific research to citizen through notably video games! :)

On the MMOS side, we had to populate the developer portal with pictures and provide all mandatory metadata. We then had to create a simple webpage that could be embedded into Galaxy. This site displays the task (our hoverfly image) and allows submitting an answer to the MMOS server.

The hoverfly example can be found on [GitHub](https://github.com/galaxyecology/webhook_SPIPOLL_Flash/blob/master/templates/guess.html).
This is a very simple example, with a single picture displayed as a task, but you can imagine many other possibilities and not only pictures!

If you have a task where you need the help of thousands of researchers, talk to us and we will figure out together how we can treat your data in a similar way as our marmalade flies!

#### Have a lot of fun during your research!

Have much fun classifying flies and let us know if you have similar use-cases!
Thanks a lot to all contributors, particularly [Rémi Planel](https://github.com/rplanel/), [Helena Rasche](https://github.com/erasche/),
the MMOS project and [Yvan Le Bras](https://github.com/yvanlebras/)!

This work is a collaborative effort between [French National Museum of Natural History](https://www.mnhn.fr/en), UseGalaxy.eu ([de.NBI](https::/denbi.de/), [ELIXIR](https://elixir-europe.org)) and the [MMOS](http://mmos.ch) company. This project has also received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement Nr 732703”

If you want to read more about crowdsourcing citizen science data on UseGalaxy.eu have a look at the [new blog post by Yvan](https://galaxyproject.org/blog/2020-01-galaxy-ecology-citizen-science/)!

repository: usegalaxy-eu/website

gha_language: TeX

repo_name: usegalaxy-eu/website


text:
# LSB-Steganography
Steganography using LSB

# Requirements

1. Install Python
2. Install Python Dependencies

```
pip install numpy opencv-python
```
# How to Run

## Embedding

```
python stegano.py [original_image] [message_location] [channel_used]
```

## Extraction

```
python stegano.py -e [image path] [location to store message] [channel_used]
```

```
-e: required for extraction
```

# About

Created by:
1. Pritam Chowdhury -<EMAIL_ADDRESS>2. Ishita Jana -<EMAIL_ADDRESS>3. Sukanya Mukherjee -<EMAIL_ADDRESS>4. Jagriti Hazra -<EMAIL_ADDRESS>


text:
# Firehose Datasets

Dataset downloader to recover the text of Firehose dataset.

### Preparing the meta-data

Please download the MetaData for the Firehose Datasets from [here](https://drive.google.com/drive/folders/17IfgC3DV2TkbiFeQjO8IBrnbV_4cAu2U?usp=sharing).

### Contact

If you have any question about the dataset (troubles in downloading the data), please contact Hexiang Hu (hexiang.frank.hu@gmail.com).

repository: firehose-dataset/downloader

repo_name: firehose-dataset/downloader


text:
# Ubuntu 18  Pepper SDK  Android Studio troubleshooting

You might face difficulties running the Pepper emulator on Android Studio under Ubuntu 18.04. Here is how to solve them. Tested OK with API 5.

### 1. Make sure your user is in the kvm group.

You need to first install this:

sudo apt install qemu-kvm

To check the ownership of /dev/kvm use:

ls -al /dev/kvm

The user was root, the group kvm. To check which users are in the kvm group, use

grep kvm /etc/group

This returned

kvm:x:some_number:

on my system: as there is nothing rightwards of the final " : ", there are no users in the kvm group.

To add the user yourname to the kvm group, you could use

sudo adduser yourname kvm

which adds the user to the group, and check once again with _grep kvm /etc/group_.

You might want to log out and back in (or restart) for the permissions to take effect.

### 2. Relink the correct libraries

Add the repo and install the missing lib:

echo "deb http://security.ubuntu.com/ubuntu xenial-security main" | sudo tee --append /etc/apt/sources.list
sudo apt-get update
sudo apt-get install libicu55

Navigate to your lib folder:

cd /home/$USER/.local/share/Softbank Robotics/RobotSDK/API 5/tools/lib

Back up the old libraries:

mv libz.so.1 libz.so.1.bak
mv libicui18n.so.55 libicui18n.so.55.bak

And relink the system's ones:

ln -s /usr/lib/x86_64-linux-gnu/libz.so libz.so.1
ln -s /usr/lib/x86_64-linux-gnu/libicui18n.so.55 libicui18n.so.55

### 3. Restart Android Studio

You should now be able to start the Pepper emulator!


text:
---

date:  "2006-11-06"
aliases: ["/2006/11/06/ms-releases-vhd-virtual-hard-disk-images-for/"]
title: MS releases VHD (Virtual Hard Disk) Images for ...
categories: ["old network blog"]
tags: ["old network blog"]
type: "oldnetblog"
---
OK, there was a new surprise this morning when looking at the Downloads site at Microsoft...

Pre-built Virtual Server VHD Images...

The posted builds are for ISA Server, Exchange Server 2007 + Office LCS 2007, Windows 2003 R2, and SQL Server 2007.  This is a good start.  Now if only they would encourage the formation of a library of pre-built VHD images like what VMWare has on the VMTN site.

System requirements mention the need for Virtual Server 2005 R2, but I am making an "guesstimate" that it would probably work OK in Virtual PC 2007.  I'll test that supposition later tonight if my download finishes (don't get me started on my very sub-standard DSL connection...)

[EDIT]I forgot the link here[/EDIT]


text:
### How to create a server (ubuntu for my case):

0) Need choose deploy and image and chose ubuntu ( or other os )
1) After fill enough information click Boot
2) When connect by ssh if your already added linode to your know key. You may need call ```ssh-keygen -R <your server ip>``` to rebuild that again
3) setting the hostname (unique name - do not start with www or anything specific)
```
echo "linode4228692" > /etc/hostname
hostname -F /etc/hostname
hostname #verify
```

add this to your /etc/hosts: <IP_ADDRESS> myhostname

4) change timezone
```dpkg-reconfigure tzdata```
```date #check ```

### Secure your server

0) Add new user

```
adduser example_user
adduser example_user sudo # add user to sudo group

1) Copy ssh public key to your server:
at local machine ```ssh-copy-id<EMAIL_ADDRESS>
2) Disallow root login over SSH:
Edit file /etc/ssh/sshd_config: PermitRootLogin no

3) Disable ssh password authentication:
Edit file: /etc/ssh/sshd_config: PasswordAuthentication no

4) Listen on only one internet protoco
echo 'AddressFamily inet' | sudo tee -a /etc/ssh/sshd_config # Support ipv4 only

5) Restart: ```	sudo systemctl restart sshd```

6) enable ufw:
```
sudo ufw allow ssh
sudo ufw enable
```

7) Use fail2ban protect login:

  apt-get install fail2ban
  ufw allow ssh
  ufw enable

fail2ban read .local which will override .conf.
  cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
Edit fail2ban.local

  cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local # this file about protect ssh by default (http, ftp are comment out)

Whitelist IP: /etc/fail2ban/jail.local
  ```
  ignoreip = <IP_ADDRESS>/8 <IP_ADDRESS>
  ```

or

  ```fail2ban-client set <name your jail> addignoreip <IP_ADDRESS>```

Ban Time and Retry Amount. Edit file: /etc/fail2ban/jail.local
  ```
  # "bantime" is the number of seconds that a host is banned.
  bantime  = 600

  # A host is banned if it has generated "maxretry" during the last "findtime"
  # seconds.
  findtime = 600
  maxretry = 3
  ```

If you use default ssh port it is ok. If you not must change port at : /etc/fail2ban/jail.local

The idea: Fail2ban work by parsing log. For example:
In file: /etc/fail2ban/jail.local
 ```
 [wordpress]
 enabled  = true
 filter   = wordpress
 logpath  = /var/www/html/andromeda/logs/access.log
 port     = 80,443
 ```

in file: /etc/fail2ban/filter.d/wordpress.conf
```
[Definition]

failregex = <HOST> - - \[(\d{2})/\w{3}/\d{4}:\1:\1:\1 -\d{4}\] "POST /wp-login.php HTTP/1.1" 200
ignoreregex =
```

8) Remove unsed network-facing service
Most service listen for incomming from internets. loopback interface or both.
See list services: ```sudo ss -lnp```

### Problems:

0) Cannot download ipv6
Edit /etc/sysctl.conf
```
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
```
sudo sysctl -p # reload setting of /etc/sysctl.conf

repository: tuanphpvn/playground

repo_name: tuanphpvn/playground