text:
---
layout: doc_page
---
# Working with different versions of Hadoop

## Download Hadoop Dependencies

```hadoop-client:2.3.0``` is already bundled in the Druid release tarball. You can get it by downloading the tarball at [druid.io](http://druid.io/downloads.html).
Unpack the tarball; You will see a ```hadoop-dependencies``` folder that contains all the Hadoop dependencies. Each dependency will have its own folder
that contains Hadoop jars.

You can also use the `pull-deps` tool to download other Hadoop dependencies you want.
See [pull-deps](../operations/pull-deps.html) for a complete example.

## Load Hadoop dependencies

There are two different ways to let Druid pick up your Hadoop version, choose the one that fits your need.

### Load Hadoop dependencies from Hadoop dependencies directory

You can create a Hadoop dependency directory and tell Druid to load your Hadoop dependencies from there.

To make this work, follow the steps below

**Tell Druid where your Hadoop dependencies are**

Specify `druid.extensions.hadoopDependenciesDir` (root directory for Hadoop related dependencies) See [Configuration](../configuration/index.html).

The value for this property should be set to the absolute path of the folder that contains all the Hadoop dependencies.
In general, you should simply reuse the release tarball's ```hadoop-dependencies``` directory.

Example:

Suppose you specify `druid.extensions.hadoopDependenciesDir=/usr/local/druid_tarball/hadoop-dependencies`, and you have downloaded
`hadoop-client` 2.3.0 and 2.4.0.

Then underneath ```hadoop-dependencies```, it should look like this:

```
hadoop-dependencies/
└── hadoop-client
    ├── 2.3.0
    │   ├── activation-1.1.jar
    │   ├── avro-1.7.4.jar
    │   ├── commons-beanutils-1.7.0.jar
    │   ├── commons-beanutils-core-1.8.0.jar
    │   ├── commons-cli-1.2.jar
    │   ├── commons-codec-1.4.jar
    ..... lots of jars
    └── 2.4.0
        ├── activation-1.1.jar
        ├── avro-1.7.4.jar
        ├── commons-beanutils-1.7.0.jar
        ├── commons-beanutils-core-1.8.0.jar
        ├── commons-cli-1.2.jar
        ├── commons-codec-1.4.jar
    ..... lots of jars
```

As you can see, under ```hadoop-client```, there are two sub-directories, each denotes a version of ```hadoop-client```.

**Tell Druid what version of Hadoop to load**

Use `hadoopDependencyCoordinates` in [Hadoop Index Task](../ingestion/batch-ingestion.html) to specify the Hadoop dependencies you want Druid to load.

For example, in your Hadoop Index Task spec file, you have

`"hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.4.0"]`

This instructs Druid to load hadoop-client 2.4.0 when processing the task. What happens behind the scene is that Druid first looks for a folder
called ```hadoop-client``` underneath `druid.extensions.hadoopDependenciesDir`, then looks for a folder called ```2.4.0```
underneath ```hadoop-client```, upon successfully locating these folders, hadoop-client 2.4.0 is loaded.

### Append your Hadoop jars to the Druid classpath

If you don't like the way above, and you just want to use one specific Hadoop version, and don't want Druid to work with different Hadoop versions, you can

(1) Set `druid.indexer.task.defaultHadoopCoordinates=[]`.  `druid.indexer.task.defaultHadoopCoordinates` specifies the default Hadoop coordinates that Druid uses. Its default value is `["org.apache.hadoop:hadoop-client:2.3.0"]`. By setting it to an empty list, Druid will not load any other Hadoop dependencies except the ones specified in the classpath.

(2) Append your Hadoop jars to the classpath, Druid will load them into the system. This mechanism is relatively easy to reason about, but it also means that you have to ensure that all dependency jars on the classpath are compatible. That is, Druid makes no provisions while using this method to maintain class loader isolation so you must make sure that the jars on your classpath are mutually compatible.

#### Hadoop 2.x

The default version of Hadoop bundled with Druid is 2.3.

To override the default Hadoop version, both the Hadoop Index Task and the standalone Hadoop indexer support the parameter `hadoopDependencyCoordinates`(See [Index Hadoop Task](../ingestion/tasks.html)). You can pass another set of Hadoop coordinates through this parameter (e.g. You can specify coordinates for Hadoop 2.4.0 as `["org.apache.hadoop:hadoop-client:2.4.0"]`), which will overwrite the default Hadoop coordinates Druid uses.

The Hadoop Index Task takes this parameter has part of the task JSON and the standalone Hadoop indexer takes this parameter as a command line argument.

If you are still having problems, include all relevant hadoop jars at the beginning of the classpath of your indexing or historical nodes.

#### CDH

Members of the community have reported dependency conflicts between the version of Jackson used in CDH and Druid when running a Mapreduce job like:
```
java.lang.VerifyError: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
```

In order to use the Cloudera distribution of Hadoop, you must configure Mapreduce to
[favor Druid classpath over Hadoop]((https://hadoop.apache.org/docs/r2.7.1/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html))
(i.e. use Jackson version provided with Druid).

This can be achieved by either:
- adding `"mapreduce.job.user.classpath.first": "true"` to the `jobProperties` property of the `tuningConfig` of your indexing task (see the [Job properties section for the Batch Ingestion using the HadoopDruidIndexer page](../ingestion/batch-ingestion.html)).
- configuring the Druid Middle Manager to add the following property when creating a new Peon: `druid.indexer.runner.javaOpts=... -Dhadoop.mapreduce.job.user.classpath.first=true`
- edit Druid's pom.xml dependencies to match the version of Jackson in your Hadoop version and recompile Druid

For more about building Druid, please see [Building Druid](../development/build.html).

Another workaround solution is to build a custom fat jar of Druid using [sbt](http://www.scala-sbt.org/), which manually excludes all the conflicting Jackson dependencies, and then put this fat jar in the classpath of the command that starts overlord indexing service. To do this, please follow the following steps.

(1) Download and install sbt.

(2) Make a new directory named 'druid_build'.

(3) Cd to 'druid_build' and create the build.sbt file with the content [here](./use_sbt_to_build_fat_jar.html).

You can always add more building targets or remove the ones you don't need.

(4) In the same directory create a new directory named 'project'.

(5) Put the druid source code into 'druid_build/project'.

(6) Create a file 'druid_build/project/assembly.sbt' with content as follows.
```
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
```

(7) In the 'druid_build' directory, run 'sbt assembly'.

(8) In the 'druid_build/target/scala-2.10' folder, you will find the fat jar you just build.

(9) Make sure the jars you've uploaded has been completely removed. The HDFS directory is by default '/tmp/druid-indexing/classpath'.

(10) Include the fat jar in the classpath when you start the indexing service. Make sure you've removed 'lib/*' from your classpath because now the fat jar includes all you need.

## Working with Hadoop 1.x and older

We recommend recompiling Druid with your particular version of Hadoop by changing the dependencies in Druid's pom.xml files. Make sure to also either override the default `hadoopDependencyCoordinates` in the code or pass your Hadoop version in as part of indexing.

repository: KeymanHuang/druid

repo_name: KeymanHuang/druid


text:
---
layout: post
comments: true
title: Podcast 23 - The Scale of Giving
image: /assets/posts/scaleofgiving.jpg
tags: [Podcast, Coaching]
---

Extending the concept of evaluating people as Givers or Takers with the Scale of Giving. Make the World a better place by being a Giver and surround yourself with other Givers. This will create ripple effects and become a true pay-it-forward solution.

![](https://isene.org/assets/posts/scaleofgiving.jpg)

Perhaps the most insidious is the people pretending to be givers when all they think about is what's in it for them.

---
Link to this post: <https://isene.org/2021/02/Podcast-ScaleOfGiving.html>

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: JavaScript

gha_license_id: MIT

license_type: permissive


text:
---
layout: post
title:  "Changes to CarpentryCon 2020"
date:   2020-04-02 09:00:00
isStaticPost: false
---

We are very disappointed to announce that CarpentryCon 2020 will not be occurring
as had been planned in June 2020. Here are the changes to know about:

### No in-person (or virtual) conference this year

Due to the worldwide spread of COVID-19, it is not possible or safe to hold an
in-person conference in Madison this year. If you have already registered for
CarpentryCon, **your registration fee will be refunded**. We gave thought to a
virtual conference as an alternative, but that too was not feasible due to a
number of constraints. As the Carpentries team, the CarpentryCon task force, and
the Carpentries community are all juggling new challenges and responsibilities in
the face of the pandemic, we know that people’s capacity to organise and attend
an event like CarpentryCon is far more limited, which informed our decision,
especially as we would like CarpentryCon to be an inclusive, accessible event.

### Online community sessions

We received many great breakout and discussion proposals for CarpentryCon 2020
that we still want to share with the community. We are investigating the best way
to share some of this content virtually in June and July instead of in person.
Once we have been able to contact presenters and agree on a plan, we will
communicate any virtual opportunities to The Carpentries community.

### CarpentryCon in the future

We still want to produce an in-person conference for members of The Carpentries
community to meet, exchange ideas, and grow. Stay tuned for information from The
Carpentries about the proposed schedule for CarpentryCon in 2021 and beyond.

gha_language: HTML


text:
Tempo Demo

Tempo is a demo application based on modern Android application tech-stacks and MVVM architecture.

Fetching data from the network and integrating persisted data in the database via repository pattern.

## Tech stack & Open-source libraries
- Minimum SDK level 21
- [Kotlin](https://kotlinlang.org/) based + [Coroutines](https://github.com/Kotlin/kotlinx.coroutines) for asynchronous.
- JetPack
  - LiveData - notify domain layer data to views.
  - Lifecycle - dispose of observing data when lifecycle state changes.
  - ViewModel - UI related data holder, lifecycle aware.
  - Room Persistence - construct a database using the abstract layer.
- Architecture
  - MVVM Architecture (View - DataBinding - ViewModel - Model)
  - Repository pattern
  - [Koin](https://github.com/InsertKoinIO/koin) - dependency injection.
- [Retrofit2 & Gson](https://github.com/square/retrofit) - construct the REST APIs.
- [OkHttp3](https://github.com/square/okhttp) - implementing interceptor, logging and mocking web server.
- [Sandwich](https://github.com/skydoves/Sandwich) - construct lightweight http API response and handling error responses.
- [Glide](https://github.com/bumptech/glide) - loading images.
- [TransformationLayout](https://github.com/skydoves/transformationlayout) - implementing transformation motion animations.
- [WhatIf](https://github.com/skydoves/whatif) - checking nullable object and empty collections more fluently.
- [AndroidVeil](https://github.com/skydoves/androidveil) - easy way to implement veil skeletons and shimmering effect.
- [DiscreteScrollView](https://github.com/yarolegovich/DiscreteScrollView) - implementing a scrollable list of items.
- [Timber](https://github.com/JakeWharton/timber) - logging.
- [Material-Components](https://github.com/material-components/material-components-android) - Material design components like ripple animation, cardView.

## Architecture
MarvelHeroes is based on MVVM architecture and a repository pattern.

![architecture](https://user-images.githubusercontent.com/24237865/77502018-f7d36000-6e9c-11ea-92b0-1097240c8689.png)


text:
# Line Segment Detector Module

## description

Line detection using Line Segments Detector (LSD) algorithm

see also [IPOL Journal][IPOL]

### sample
todo

## usage
todo

license
----------
Copyright © 2017 wellflat Licensed under the [MIT License][MIT]

[MIT]: http://www.opensource.org/licenses/mit-license.php
[IPOL]: http://www.ipol.im/pub/art/2012/gjmr-lsd/

repository: sekure/imageprocessing-labs

repo_name: sekure/imageprocessing-labs


text:
This package provides a PyBullet simulation environment with ROS topic support for the Wolfgang Robot.

There are different options to use this:
1. Start the simulation with interface `rosrun wolfgang_pybullet_sim simulation_with_gui.py`
2. Start the simulation without interface `rosrun wolfgang_pybullet_sim simulation_headless.py`
3. Use the python class `Simulation` in `simulation.py` to directly run a simulation without using ROS

Shortcuts in gui:

`r` reset simulation

`g` debug interface

`space` pausing

`s` hold to step while pausing

`n` gravity on/off

gha_language: Python


text:
---
layout: default
title: Matrix.TR
description: Translate, Rotate. Creates a transform Matrix using these components!
---
# [Matrix]({{site.url}}/Pages/Reference/Matrix.html).TR

static [Matrix]({{site.url}}/Pages/Reference/Matrix.html) TR([Vec3]({{site.url}}/Pages/Reference/Vec3.html) translation, [Quat]({{site.url}}/Pages/Reference/Quat.html) rotation)

|  |  |
|--|--|
|[Vec3]({{site.url}}/Pages/Reference/Vec3.html) translation|Move an object by this amount.|
|[Quat]({{site.url}}/Pages/Reference/Quat.html) rotation|A Quaternion describing the rotation for              this transform.|
|RETURNS: [Matrix]({{site.url}}/Pages/Reference/Matrix.html)|A Matrix that combines translation and rotation information into a single Matrix!|

Translate, Rotate. Creates a transform Matrix using
these components!

static [Matrix]({{site.url}}/Pages/Reference/Matrix.html) TR(float x, float y, float z, [Quat]({{site.url}}/Pages/Reference/Quat.html) rotation)

|  |  |
|--|--|
|float x|Move an object on the x axis by this amount.|
|float y|Move an object on the y axis by this amount.|
|float z|Move an object on the z axis by this amount.|
|[Quat]({{site.url}}/Pages/Reference/Quat.html) rotation|A Quaternion describing the rotation for              this transform.|
|RETURNS: [Matrix]({{site.url}}/Pages/Reference/Matrix.html)|A Matrix that combines translation and rotation information into a single Matrix!|

Translate, Rotate. Creates a transform Matrix using
these components!

static [Matrix]({{site.url}}/Pages/Reference/Matrix.html) TR([Vec3]({{site.url}}/Pages/Reference/Vec3.html) translation, [Vec3]({{site.url}}/Pages/Reference/Vec3.html) pitchYawRollDeg)

|  |  |
|--|--|
|[Vec3]({{site.url}}/Pages/Reference/Vec3.html) translation|Move an object by this amount.|
|[Vec3]({{site.url}}/Pages/Reference/Vec3.html) pitchYawRollDeg|Pitch (x-axis), yaw (y-axis), and              roll (z-axis) stored as x, y and z respectively in this Vec3.             Units are in degrees.|
|RETURNS: [Matrix]({{site.url}}/Pages/Reference/Matrix.html)|A Matrix that combines translation and rotation information into a single Matrix!|

Translate, Rotate. Creates a transform Matrix using
these components!

static [Matrix]({{site.url}}/Pages/Reference/Matrix.html) TR(float x, float y, float z, [Vec3]({{site.url}}/Pages/Reference/Vec3.html) pitchYawRollDeg)

|  |  |
|--|--|
|float x|Move an object on the x axis by this amount.|
|float y|Move an object on the y axis by this amount.|
|float z|Move an object on the z axis by this amount.|
|[Vec3]({{site.url}}/Pages/Reference/Vec3.html) pitchYawRollDeg|Pitch (x-axis), yaw (y-axis), and              roll (z-axis) stored as x, y and z respectively in this Vec3.             Units are in degrees.|
|RETURNS: [Matrix]({{site.url}}/Pages/Reference/Matrix.html)|A Matrix that combines translation and rotation information into a single Matrix!|

Translate, Rotate. Creates a transform Matrix using
these components!

repository: sc-tec/StereoKit

repo_name: sc-tec/StereoKit


text:
# ember-cli-google-charts

EmberJS add-on for using Google Charts

[![npm version](https://img.shields.io/npm/v/ember-cli-google-charts.svg?maxAge=2592000)](https://www.npmjs.com/package/ember-cli-google-charts)
[![Dependencies](https://david-dm.org/onehilltech/ember-cli-google-charts.svg)](https://david-dm.org/onehilltech/ember-cli-google-charts)
[![devDependencies Status](https://david-dm.org/onehilltech/ember-cli-google-charts/dev-status.svg)](https://david-dm.org/onehilltech/ember-cli-google-charts?type=dev)

## Features

* Designed to support seamless integration into an EmberJS application.
* Proper binding of attributes to options for real-time, dynamic updates.
* Handle events as actions for interactive designs.
* Auto-loading and configuring of scripts that correspond with appropriate lifecycle events.

## Installation

    ember install ember-cli-google-charts

## Quick Example

Here is an example of creating
[the following line chart](https://developers.google.com/chart/interactive/docs/gallery/linechart#curving-the-lines):

```handlebars
{{g-linechart data=lineChartData title="Company Performance" curveType="function" legendPosition="bottom"}}
```

Each of the chart options above can bind to a variable. When the variable is updated,
the chart is redrawn.

repository: onehilltech/ember-cli-google

repo_name: onehilltech/ember-cli-google


text:
# Model agonistic methods

This section discusses a variety of model agnostic methods that can be applied to make machine learning models interpretable.

By model-agnostic, we mean that the methods can be applied to any model. All the techniques under this section can be applied to any model and each technique has its own pros and cons. We explain every technique technically and give a metaphorical example to help understand the underlying concept easily.

![](../.gitbook/assets/image%20%28112%29.png)

repository: abhi-gm/computational-skepticism-book

repo_name: abhi-gm/computational-skepticism-book


text:
Supabase Schema

    Secured & Simple Supabase Schema Visualizer.

    No installations • No Database password required.

    View Demo
    ·
    Report Bug
    ·
    Request Feature

![Supabase Schema](images/main.png)

## 🚀 Features

- 🔨 No installations
- 🔒 Persistent State in LocalStorage
- 🚪 Login-less
- 👀 No Sensitive info required
- 🤚 Moveable Schema for own design

## 📇 About The Project

I look around the internet and found many Database Schema Visualizers that require me to either download the software, or login to view my Database Schema. I feel like there's too many unneccessary steps, just to view the schema.

Thus, I've decided to take this opportunity to build this **Open Source, Free, Login-less and Secured** Supabase Schema Visualizer 🔥

### 🔨 Built With

- [Vue 3](https://v3.vuejs.org/)
- [Vite](https://vitejs.dev/)
- [WindiCSS](https://windicss.org/)

## 🐾 Instructions

1. Go to [Supabase App](https://app.supabase.io/)
2. Select your Project
3. Go to `Settings`
4. In Settings tab, Go to `API`
5. Under `Config`, copy `URL`
6. Under `API Keys`, copy `anon` `public`
7. Paste both value into their respectively slot in [Supabase Schema](https://supabase-schema.vercel.app/)
8. Fetch data
9. Enjoy the Visualization 🎉

Remeber to ⭐ this repo if you ❤ this Apps

## 🌎 Local Development

### Prerequisites

Yarn

- ```sh
  npm install --global yarn
  ```

### Development

1. Clone the repo
   ```sh
   git clone https://github.com/zernonia/supabase-schema.git
   ```
2. Install NPM packages
   ```sh
   yarn install
   ```
3. Run Development instance
   ```sh
   yarn dev
   ```

## ➕ Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3.  Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## 🙏 Acknowledgement

1. Fundamental for this Visualization ([Generate database types from OpenAPI specification](https://supabase.io/docs/reference/javascript/generating-types#generate-database-types-from-openapi-specification))
2. Guide to Construct Dynamic SVG Connector ([Connecting Table using SVG](https://codepen.io/alojzije/pen/ndfrI))
3. [Icones - icon gallery](https://icones.js.org/)

## 📈 Analytics

I'm using [Umami Analytics](https://umami.is/docs/about) because I'm interested in the distributions of user who uses Supabase and this tool.

[This](https://umami-zernonia.vercel.app/share/yzSUulXQ/Supabase%20Schema) is the public URL for the analytics. Enjoy!

## 📜 License

Not Associated with Supabase.

Distributed under the MIT License. See `LICENSE` for more information.

# 📧 Contact

Zernonia - [@zernonia](https://twitter.com/zernonia) -<EMAIL_ADDRESS>
Also, if you like my work, please buy me a coffee ☕😳


text:
# The `setup` stage

So far for all our component storages and resources, we've been adding
them to the `World` manually. In Specs, this is not required if you use
`setup`. This is a manually invoked stage that goes through `SystemData`
and calls `register`, `add_resource`, etc. for all (with some exceptions)
components and resources found. The `setup` function can be found in
the following locations:

* `ReadStorage`, `WriteStorage`, `Read`, `Write`
* `SystemData`
* `System`
* `RunNow`
* `Dispatcher`
* `ParSeq`

During setup, all components encountered will be registered, and all
resources that have a `Default` implementation or a custom `SetupHandler`
will be added. Note that resources encountered in `ReadExpect` and `WriteExpect`
will not be added to the `World` automatically.

The recommended way to use `setup` is to run it on `Dispatcher` or `ParSeq`
after the system graph is built, but before the first `dispatch`. This will go
through all `System`s in the graph, and call `setup` on each.

## Custom `setup` functionality

The good qualities of `setup` don't end here however. We can also use `setup`
to create our non-`Default` resources, and also to initialize our `System`s!
We do this by custom implementing the `setup` function in our `System`.

Let's say we have a `System` that process events, using `shrev::EventChannel`:

```rust,ignore
struct Sys {
    reader: ReaderId<Event>,
}

impl<'a> System<'a> for Sys {
    type SystemData = Read<'a, EventChannel<Event>>;

    fn run(&mut self, events: Self::SystemData) {
        for event in events.read(&mut self.reader) {
            [..]
        }
    }
}
```

This looks pretty OK, but there is a problem here if we want to use `setup`.
The issue is that `Sys` needs a `ReaderId` on creation, but to get a `ReaderId`,
we need `EventChannel<Event>` to be initialized. This means the user of `Sys` need
to create the `EventChannel` themselves and add it manually to the `World`.
We can do better!

```rust,ignore
use specs::prelude::Resources;

#[derive(Default)]
struct Sys {
    reader: Option<ReaderId<Event>>,
}

impl<'a> System<'a> for Sys {
    type SystemData = Read<'a, EventChannel<Event>>;

    fn run(&mut self, events: Self::SystemData) {
        for event in events.read(&mut self.reader.as_mut().unwrap()) {
            [..]
        }
    }

    fn setup(&mut self, res: &mut Resources) {
        use specs::prelude::SystemData;
        Self::SystemData::setup(res);
        self.reader = Some(res.fetch_mut::<EventChannel<Event>>().register_reader());
    }
}
```

This is much better; we can now use `setup` to fully initialize `Sys` without
requiring our users to create and add resources manually to `World`!

**If we override the `setup` function on a `System`, it is vitally important that we
remember to add `Self::SystemData::setup(res);`, or setup will not be performed for
the `System`s `SystemData`.**

repository: Moxinilian/specs

repo_name: Moxinilian/specs


text:
---
layout: post
keywords: [ruby, rails, rspec, minitest-rails]
---

*Ruby 2.1.4 | Rails 4.2.3 | Sep 2015*

When I begin a new Ruby on Rails project, there are a few thing I always like to have in place from the start. This post covers my initial setup, go-to gems, and what they do. I've created a [companion code base] on Github to share everything that is in this article, so you can play with the code in a working application. Just make sure to match up the branch which corresponds to the example.

## Prerequisites

- [RVM] \(Ruby Version Manager) or [rbenv] \(simple Ruby version management) runtime environments
- [Bundler] a gem/extension manager

## Create a new Rails app

```bash
rails new <app_name> -T --database=postgresql --skip-turbolinks
```

This gives us a new Rails app with a few options in place that I find useful for every application.

Option | Description
--- | ---
__`-T`__ | No test framework. Only use this if you're planing on using Rspec
__`--database=postgresql`__ | Setup for a postgrsql database
__`--skip-turbolinks`__ | Turbolinks are more trouble than they are worth

A more complete list of the available [command line options] can be found by entering the `rails -h` command.

>Note: All of the `rails new` options can be set as default by adding them to a `.railsrc` file in your home directory.

>```
># ~/.railsrc
>
>--skip-test-unit
>--database=postgresql
>--skip-turbolinks
>```

## Gems

My next step is to add a few gems I know I'm going to use in the initial setup. I'll go over them in more detail later, as we put them to use. I also remove all the comments and organize the Gemfile a little.

### Example Gemfile ([Rspec])

```ruby
gem 'pry-rails'

group :test, :development do
  gem 'rspec-rails'
  gem 'capybara'
  gem 'launchy'
  gem 'database_cleaner', '~> 1.4.1'
end
```

### Example Gemfile ([Minitest])

This is how I approach setting up [Minitest] in a Rails application.  This is specific to the `minitest-rails` gem. Rails comes with Minitest built in, however setting up an app to make use of Minitest without the the gem requires slightly different code.

```ruby
# Gemfile

gem 'pry-rails'

group :test, :development do
  gem 'minitest-rails-capybara'
  gem 'launchy'
  gem 'database_cleaner', '~> 1.4.1'
end
```

>Note: The Minitest-rails gem is a dependency of `minitest-rails-capybara` so I don't need to list it in the `Gemfile`. If you prefer, you can include it to remind you that it's included.

Run `bin/bundle` with test framework chosen, and gems in place.

## Testing Framework Setup

Now that we have the gems in place, we need to do the initial setup.

### [Rspec] Setup

```bash
rails generate rspec:install
```

This will add a couple configuration files:

```
- .rspec
- - spec/spec_helper.rb
- - spec/rails_helper.rb
```

At this point I remove the comments. Here is the [example Rspec] implementation on Github.

### [Minitest] Setup

The `minitest-rails` gem gives us a generator.

```bash
rails generate minitest:install
```

This will overwrite the current `/test/test_helper.rb` file created by the `rails new ...` command. I remove the comments in the new file and I end up with something that looks like:

```ruby
ENV["RAILS_ENV"] = "test"
require File.expand_path("../../config/environment", __FILE__)
require "rails/test_help"
require "minitest/rails"

class ActiveSupport::TestCase
  ActiveRecord::Migration.check_pending!
  fixtures :all
end
```

### Unit Testing

At this point we are all set up for unit testing. The test framework knows about itself and knows which directory to look in for test files. That will allow us to run the Rails generators and get the corresponding test files.

Next we'll get integration and feature testing up and running.

## [Capybara] Setup

The capybara gem is a very common way to implement integration/feature testing. By using a _headless browser_ to navigate the app, Capybara allows testing of elements on the page that change based on the input. So when you click the logout button, you ras expert the page to display "Good bye {user name}".

To get Capybara up and running in our test framework we need to add the gem(s) and make the gem available.

>One thing to note: Integration tests and feature tests are effectively the same thing. __Integration tests for Minitest land and feature for in Rspec world.__

### Rspec Capybara Setup

```ruby
# Gemfile

group :test, :development do
  gem 'capybara'
  gem 'launchy'
end
```

```ruby
# rails_helper.rb

require 'capybara/rails'

```

Capybara and Rspec play very nicely together. If you place your feature tests in the `spec/features` directory, Rspec will automatically know what type of test to run and you won't need to specify the type.

>A word of caution. This convenience can trip you up if your not aware of it. When you name the feature test directory anything other than `spec/features` (i.e. `spec/feature`), Capybara requires you to explicitly declare the type for each example group with `:type => :feature`

>```ruby
>describe "the signin process", :type => :feature do
>  before :each do
>    User.make(:email =><EMAIL_ADDRESS>:password => 'password')
>  end

>  it "signs me in" do
>    visit '/sessions/new'
>    within("#session") do
>      fill_in 'Email', :with =><EMAIL_ADDRESS>>      fill_in 'Password', :with => 'password'
>    end
>    click_button 'Sign in'
>    expect(page).to have_content 'Success'
>  end
>end
>```

### Minitest Capybara Setup

I approach [Minitest] a little differently from the Rspec example. I use the `minitest-rails-capybara` gem to bring in the `minitest-rails` gem as a dependency, so we have nothing to change in the `Gemfile` at this point.

In the `test/test_helper.rb` we need to add `require "minitest/rails/capybara"`.

At this point I have a `test_helper.eb` that looks like this:

```ruby
ENV["RAILS_ENV"] = "test"
require File.expand_path("../../config/environment", __FILE__)
require "rails/test_help"
require "minitest/rails"
require "minitest/rails/capybara"

class ActiveSupport::TestCase
  fixtures :all
end
```

Here is the [example Minitest] implementation on Github.

Minitest integration tests live in `test/integration`. This folder is automatically created with `rails new ...` when you don't include the `-T` option.

### [Launchy]

When using either Minitest or Rspec the __[Launchy] gem__ is something I always use in conjunction with Capybara. It is not required for Capybara to run, but it is very useful for debugging. To learn more about why it's to useful do a search for [save_and_open_page](http://www.google.com/search?q=save_and_open_page){:target="_blank"}

## [Database Cleaner]

I use the [Database Cleaner] gem to help maintain a clean database after each test:

Add a file to `spec/support/database_cleaner.rb`

```ruby
# spec/support/database_cleaner.rb

require 'database_cleaner'

RSpec.configure do |config|

  config.before(:suite) do
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.clean_with(:truncation)
  end

  config.around(:each) do |example|
    DatabaseCleaner.cleaning do
      example.run
    end
  end

end
```

## [Pry]

[Pry] is an enhanced version of Ruby's IRB (a Ruby shell). The [Pry-rails] gem brings Pry to the Rails console and interactive debugger environments. Pry is packed with additional features not available in IRB or the Rails Console. While I love it, your milage may vary. Season to taste.

## [Minitest-Reporters]

When I'm using Minitest I also add the `minitest/reporters` gem. I like the enhanced test output. Along with adding the gem, it requires a little more setup than the other gems we've added so far. Add the `Minitest::Reporters.use!` method to the `test_helper.rb` file.

```ruby
# Gemfile

  gem 'pry-rails'

group :test, :development do
  gem 'minitest-rails-capybara'
  gem 'minitest-reporters'
  gem 'launchy'
  gem 'database_cleaner', '~> 1.4.1'
end
```

```ruby
# test_helper.rb

ENV["RAILS_ENV"] = "test"
require File.expand_path("../../config/environment", __FILE__)
require "rails/test_help"
require "minitest/rails"
require "minitest/rails/capybara"

require "minitest/reporters"
Minitest::Reporters.use!(
  Minitest::Reporters::SpecReporter.new,
  ENV,
  Minitest.backtrace_filter
)

class ActiveSupport::TestCase
  fixtures :all
end
```

## Wrapping up

I work to keep this article updated, to reflect my current Rails application set-up. I'd love to know what you think and what you use.

Was this article helpful to to you? If you got stuck on any piece or have a question, leave a comment down below.

I want you to be honest and will do my best to update this article to keep it fresh and free of bugs.

And, hey!  Keep it kind.  As much as I value and respect opinions, I will remove any comment that are unkind to anyone.


text:
# EEG_WebServer

Web server built using Spring Boot that creates EEG data analysis jobs, check their status and get the result over a REST API.

---
#### The server listens for requests:
- /jobs/submit/(id)?{qp}
		- to submit a job with configuration specified in the query parameters
- /jobs/check/(id)
		- to check the status of a job with id=={id} -> either RUNNING OR FINISHED
- /jobs/result/(id)
		- to get the result output of a job with id=={id}
- /jobs/log/{id}
		- to get the log of a job with id=={id}
- /jobs/cancel/{id}
		- to cancel a job with id=={id}
- /jobs/configuration/{name}
		- to get the configuration of a job with id=={id}
- /classifiers/list
		- to list all the saved classifiers

where the {id} or {name} are path variables provided by the client,
and {qp} are query parameters that configure the spark application.

---

#### Requirements

Java 6+
Open port: 8990
Place the Data Analysis Package in the same directory as this JAR file

---

#### The supported query parameters are:

 0. Id (REQUIRED - the id of the job)
  - id = {some_integer}*

 1. Input file (REQUIRED - choose what type of input you want to provide)
  - info_file={*path_to_file}
 		OR
  - eeg_file={*path_to_file} AND guessed_num={*number}

 2. Feature extraction (REQUIRED - choose the method used for classification)
  - fe = {dwt-8}

 3. Classification
  - a) (REQUIRED - choose whether to train or load a classifier)
 	train_clf = {svm,dt,logreg,rf}
 			OR
 	load_clf={svm,dt,logreg,rf} AND load_name={src/main/resources/Classifiers/(*name*) }

  - b) (OPTIONAL - choose whether to save the classifier and give him a name)
 	save_clf={true,false}
 		AND
 	save_name={*name}

  - c) (REQUIRED - the specific configuration for each classifier)
 	config_*clf_param*
 	-> can't be condensed as it's very specific for each classifier

 4. Saving the results (REQUIRED - choose where to save the classifier performance)
  - result_path={*path_to_file}

  ### Copyright

  This file is part of the Workflow Designer project

  ==========================================

  Copyright (C) 2018 by University of West Bohemia (http://www.zcu.cz/en/)

 ***********************************************************************************************************************

  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
  the License. You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
  an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
  specific language governing permissions and limitations under the License.


text:
Creating Custom Actions - RotateAroundTo
----------------------------------------

This sample goes along with the forum post of the same name.

Part1 can be found here [Creating Custom Actions - RotateAroundTo - Part 1](http://forums.xamarin.com/discussion/38215/creating-custom-actions-rotatearoundto-part-1)


text:
# GeckoBot
A python script built to scrap latest news from CoinGecko news and post them to a telegram channel without a delay...

## Prerequisite

The script won't just start working automatically. Here's some little nuisance before it start working :

- Get Bot API token from [@BotFather](https://t.me/botfather)
- Get your telegram Channel Username
- Make the bot admin in your channel with posting privilege..
- Make some edit into the code..
- Command `pip install -r requirements.txt`

## Usage

- Command `python gecko.py` ...

🔥 **Boom !  Enjoy !**

repository: ycngmn/GeckoBot

branch_name: refs/heads/main

repo_name: ycngmn/GeckoBot


text:
# Users
Ansible roles to create/configure users on Linux/FreeBSD.

## Variables
| user_groups  |                                               |                |
| ---:         |---                                            |---             |
| name         | name of the group                             | Data type      |
| gid          | Optionally set the group ID                   | int            |
| state        | whether the group shoud be created or removed | present/absent |

| users            |   |    |
| ---:             |---| ---|
| _variable name_    | Description                                   | Data type       |
| name             | username                                      | string          |
| state            | whether the user should be created or removed | present/absent  |
| password         | string of an encrypted value(1)               | string          |
| groups           | additional groups the user should belong to   | list            |
| uid              | optionally specify a user id                  | int             |
| enable_sudo      | Enable passwordless sudo for the given user   | bool            |
| keys             | list of dictionaries                          | list            |
| bash_lines       | configure lines in .bashrc                    | list            |
| bash_blocks      | configure lines in .bashrc                    | list            |
| csh_lines        | configure lines in .cshrc                     | list            |
| csh__blocks      | configure lines in .cshrc                     | list            |

(1) https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module

## Default variables
The default shells depending on the OS are:

- Linux: `/bin/bash`
- FreeBSD: `/bin/cshrc`

This is defined in the `defaults` section of the **users** role

## Example inventory
```
user_groups:
  - name: mygroup
    gid: 700

users:
  - name: remember
    state: present
    password: "blabla"
    groups:
      - mygroup
    uid: 1100
    enable_sudo: true
    keys:
      - file: key1
        state: present
    bash_lines:
      - line: 'export SSH_AUTH_SOCK="${HOME}/.gnupg/S.gpg-agent.ssh"'
        state: present
      - line: "alias ls='ls lah'"
        state: present
    bash_blocks:
      - content: |
          #testing
          #multiline
        state: present
  - name: test
    enable_sudo: false
    keys:
      - file: key2
        state: absent
    csh_lines:
      - line: "ls ls -lah"
        state: absent
```
## Using the Role
### Example Playbook
```
---
- name: Manage user configuration
  hosts: all
  remote_user: root
  roles:
    - users
```
### Configure a user's ssh keys
For every user a directory matching the username should be created under the _keys_ folder in the role's _files_ folder. In this folder the user's ssh keys can be stored.

```
├── files
│   └── keys
│       ├── remember
│       │   └── key1.pub
│       └── test
│           └── key2.pub
```
The name of the file holding the key should match the name in the _users_ variable

```
    keys:
      - file: key1
        state: present
```

### Configure a user's shell
This role allows you to add or remove lines to a user's `.bashrc` or `cshrc` file. Since this is not based on a template that overwrites the complete file, users can still add their own configuration too.

Add items to the **shell_lines** key in the **users** variable. Each item exists of a _line_ and _state_ key.

**lines**

Use _lines_ if you want to make sure a single line is present or not.
Example:
```
shell_lines:
  - line: "testline"
    state: absent
  - line: 'export SSH_AUTH_SOCK="${HOME}/.gnupg/S.gpg-agent.ssh"'
    state: present
  - line: "alias ls='ls lah'"
    state: present
```

**blocks**

use blocks if you want to make sure a number of lines that belong together are
present or not.

Example:
```
bash_blocks:
  - content: |
      if [ condition ]; then
        do something
    state: present
```

repository: stationgroup/ansible-experiments

gha_language: Ruby

repo_name: stationgroup/ansible-experiments


text:
---
title: "Notify Webhooks"
draft: false
weight: 6
---

### Description

Paytrail calls the notify URL with a `GET` request when the refund status changes. The received notification must be acknowledged by responding with an HTTP status in the `200–299` range.

If an update is not acknowledged by the receiving end, the update to that refund is postponed until the next retry round. Updates will be retried after **1, 3, 6, 10, 15, 21, 28, 36, 45** and **55** hours, after which the notification is discarded. This may cause some updates to be skipped if the notification URL endpoint cannot be contacted within 55 hours.

### Parameters

- `refundToken` – Refund token.
- `oldStatus` – Refund's previous status.
- `newStatus` – Refund's new status
- `signature` – Signature to ensure the update comes from Paytrail.

Signature is calculated with the following formula:

```rb
sha256(:refundToken + '|' + :oldStatus + '|' + :newStatus + '|' + :merchantSecret)
```

### Refund Statuses

- `created` – Refund was created. Never used as `newStatus`, since notify is not sent when a refund is created.
- `completed` – Refund paid to customer. For bank or _Collector_ refund, amount is deducted from merchant's settlement.
- `cancelled` – Refund cancelled or failed.

{{% notice note %}}New statuses may be introduced in the future. Implementations should be able to discard any other statuses received.{{% /notice %}}

repository: eeturoponen/documentation

repo_name: eeturoponen/documentation


text:
---
bg: "test.jpg"
layout: post
title:  "Login Script in Python"
crawlertitle: "Dictionary Manipulation"
summary: "A function that takes password and writes to text  file"
date:   2018-06-28 23:09:47 +0700
categories: posts
tags: 'python'
author: Uzoma
---
Exploring the file handling capabilities in python, I complete the first phase of my word-to-meaning match application in pyton. This phase involves building a script that collects and stores a users-defined  password to a textfile and authenticase user login prior to use of the word-to-meaning macher.

See code snippets below:

{% highlight python %}
###### The LOGIN SCRIPT -----> 08/11/2018
######  Login() searches for a file password.txt,reads the file, recieves input from user
######  returns TRUE and print welcome message if input exits in password.txt, Else returns FALSE

def login():
    aba = glob2.glob("*.txt")
    for i in aba:
        if i == 'password.txt':
            with open ("password.txt") as file:
                holder = file.read().split(",")
                hold = input("Please enter Login Password : ")
                if hold in holder:
                    print("Welcome to the Dictionary App")
                    return True
                else:
                    print("Wrong password")
                    return False

###### newpass() takes arguement TRUE of FALSE,
#####  if FALSE, it prompts user to create password, and writes or appends password to password.txt file

def newpass(var):
    if var == True:
        pass
    else:
        with open ("password.txt","r+") as newpass:
            contain = input("You MUST CREATE a login password : ")
            lookup = newpass.read().split(",")
            if contain in lookup :
                print("Password exist Already")
            else:
                newpass.write(contain+",")
                print("New Password Saved")

{% endhighlight %}

extension: markdown


text:
# LecroyStrParser

LeCroyStrParser.py parses binary str produced by LeCroy scopes.

LeCroyStrParser.py is derived from the matlab program
ReadLeCroyBinaryWaveform.m, available at
<https://www.mathworks.com/matlabcentral/fileexchange/26375-readlecroybinarywaveform-m>,
Original version (c)2001 Hochschule fr Technik+Architektur Luzern
Fachstelle Elektronik 6048 Horw, Switzerland Slightly modified by Alan
Blankman, LeCroy Corporation, 2006

Further elements for the code were taken from pylecroy, written by Steve Bian
Slightly modified by Tom Aubier to support binary string decoding.

A useful resource for modifications is the LeCroy Remote Control Manual
available at
<http://cdn.teledynelecroy.com/files/manuals/dda-rcm-e10.pdf>

LecroyStrParser has been tested with and Python 3.7

repository: Tomaubier/leCroyParser

repo_name: Tomaubier/leCroyParser


text:
# demo-firebase-android
End-to-end encrypted, HIPAA-compliant Android chat sample app for Firebase. While this is a chat app, you can reuse it in any other apps to protect user data, documents, images using Virgil's end-to-end encryption. Only HIPAA-compliant for chat use-cases.

## Clone project

Clone the repo to your computer: using your Android Studio, navigate to 'File -> New -> Project from Version Control -> Git' and fill in the 'Git Repository URL' field with:
```
https://github.com/VirgilSecurity/demo-firebase-android
```

## Create Firebase project
Go to the [Firebase console](https://console.firebase.google.com) and if you haven't created a project yet, create one now. If you already have one that you want to use, open it and skip to the **Firebase app setup**

* Select the **Authentication** panel and then click the **Sign In Method** tab.
*  Click **Email/Password** and turn on the **Enable** switch, then click **Save**.
* Select the **Database** panel and then enable **Cloud Firestore**.
  * Click **Rules** and paste:
  ```
  service cloud.firestore {
    match /databases/{database}/documents {
      match /{document=**} {
        allow read, write: if request.auth.uid != null;
      }
    }
  }
  ```
* Click **PUBLISH**.

## Firebase Android app setup
* In your Firebase project (on the Firebase console), click the **gear icon** -> **Project settings**
* Click **Add app** and choose **Add Firebase to your Android app**. Fill in all required fields.
* Download the generated **google-services.json** file from Project Settings and copy it to the 'app' folder, as metioned in the instructions by Firebase. You're good to go!

## Firebase cloud functions setup

> In order for the app to work, you need to deploy a Firebase function that creates JWT tokens for your authenticated users. If you already deployed this function for either the iOS or Android apps, you don't need to do it again.

* Otherwise, [follow the instructions here](https://github.com/VirgilSecurity/demo-firebase-func)
* Once the function is successfully created, go to the Firebase console -> Functions tab and copy your function's url
* Go to Android Studio -> `app/src/main/java/com/android/virgilsecurity/virgilonfire/di/NetworkModule.java` and change variable `BASE_URL` to:
```
https://YOUR_FUNCTION_URL.cloudfunctions.net/api/
```

## Build and Run
At this point you are ready to build and run the application on your real device or emulator (with google services).
* Check out what Firebase sees from your users' chats: Firebase dashboard -> Database -> Channels -> click on the thread -> Messages. This is what the rest of the world is seeing from the chat, without having access to the users' private keys (which we store on their devices).

## Credentials

To build this sample we used the following third-party frameworks:

* [Cloud Firestore](https://firebase.google.com/docs/firestore/) - as a database for messages, users and channels.
* [Cloud Functions](https://firebase.google.com/docs/functions/) - getting jwt.
* [Firebase Authentication](https://firebase.google.com/docs/auth/) - authentication.
* [Virgil SDK](https://github.com/VirgilSecurity/virgil-sdk-x) - managing users' Keys.
* [Virgil Crypto](https://github.com/VirgilSecurity/virgil-foundation-x) - encrypting and decrypting messages.

license: Unlicense

repository: AzriDelta/demo-firebase-android

repo_name: AzriDelta/demo-firebase-android


text:
---
page_type: sample
languages:
- azurepowershell
- azurecli
products:
  - azure
  - azure-load-balancer
  - azure-virtual-network
  - azure-bastion
  - azure-application-gateway
description: This sample deploys simulated Network Virtual Appliances (NVAs) in a zone-aware highly-available configuration. Azure Application Gateway is used to route ingress traffic, and Azure Load Balancers are used for internal and egress traffic routing.
---

# Highly Available Ingress/Egress with layer 7 NVAs

[![Build Status](https://dev.azure.com/ceapex/pnp/_apis/build/status/deployment-validation/mspnp.samples-solutions-hanva?branchName=master)](https://dev.azure.com/ceapex/pnp/_build/latest?definitionId=3667&branchName=master)

## Overview

This sample template can be used to deploy several configurations of simulated Layer 7 NVAs to demonstrate a highly available configuration. For more information on this deployment and other highly available NVA options, see [Deploy highly available NVAs](https://learn.microsoft.com/azure/architecture/reference-architectures/dmz/nva-ha).

**Ingress-only** - Deploy an Azure Application Gateway in front of two simulated network virtual appliance (NVA) VMs.  The NVA VMs will forward requests on port 80 to a web-tier Azure Load Balancer, which will distribute traffic across two back-end VMs running a simple web server. *This deployment takes approximately 21 minutes.*

![Highly Available Egress with layer 7 NVAs architectural diagram.](l7-ingress.png)

**Egress-only** - Deploy an internal Azure Load Balancer in front of two simulated network virtual appliance (NVA) VMs.  The NVA VMs will proxy requests on port 8080 from web-tier VMs to the Internet. *This deployment takes approximately 11 minutes.*

![Highly Available Egress with layer 7 NVAs architectural diagram.](l7-egress.png)

**Ingress-Egress** - Combines the above two options to provide highly available Layer 7 NVA-controlled traffic for both the Ingress path to the web-tier, and the Egress path from the web-tier. *This deployment takes approximately 22 minutes.*

![Highly Available Egress with layer 7 NVAs architectural diagram.](l7-ingress-egress.png)

## Deploy this template to Azure

> Note: If you fork this repository, you will need to modify the deployment links in [README.md](README.md) to point to your repo.  If you create a separate branch for testing, you will have to include a change to this link to point to your branch as well. You must include a URL-encoded link to the raw [azuredeploy.json](azuredeploy.json) file after `/uri/` in the link defined for the deployment button, or you can point to your repo with the `--template-uri`/`-TemplateUri` parameter of the CLI or Powershell commands.

### Azure portal

Use the following buttons to deploy the architecture using the Azure portal.

[![Deploy To Azure](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.svg?sanitize=true)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmspnp%2Fsamples%2Fmaster%2Fsolutions%2Fha-nva%2Fazuredeploy.json)
[![Deploy To Azure US Gov](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazuregov.svg?sanitize=true)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmspnp%2Fsamples%2Fmaster%2Fsolutions%2Fha-nva%2Fazuredeploy.json)

To deploy the **ingress-only** configuration, set the values of the following parameters:

    deployIngressAppGatewayWebLoadBalancer=true
    deployEgressLoadBalancerNva=false

To deploy the **egress-only** configuration, set the values of the following parameters:

    deployIngressAppGatewayWebLoadBalancer=false
    deployEgressLoadBalancerNva=true

To deploy the **ingress-egress** configuration, set the values of the following parameters:

    deployIngressAppGatewayWebLoadBalancer=true
    deployEgressLoadBalancerNva=true

### Azure CLI

Use the following command to create a resource group for the deployment.

```azurecli-interactive
az group create --name ha-nva-l7 --location eastus
```

Run one of the following commands to deploy the example architecture.

#### For ingress-only

```azurecli-interactive
az deployment group create --resource-group ha-nva-l7 \
    --template-uri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json \
    --parameters deployIngressAppGatewayWebLoadBalancer=true deployEgressLoadBalancerNva=false
```

#### For egress-only

```azurecli-interactive
az deployment group create --resource-group ha-nva-l7 \
    --template-uri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json \
    --parameters deployIngressAppGatewayWebLoadBalancer=false deployEgressLoadBalancerNva=true
```

#### For ingress-egress

```azurecli-interactive
az deployment group create --resource-group ha-nva-l7 \
    --template-uri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json
```

### PowerShell

Use the following command to create a resource group for the deployment.

```azurepowershell-interactive
New-AzResourceGroup -Name ha-nva-l7 -Location eastus
```

Run one of the following commands to deploy the example architecture.

#### For ingress-only

```azurepowershell-interactive
New-AzResourceGroupDeployment -ResourceGroupName ha-nva-l7 `
    -TemplateUri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json `
    -deployIngressAppGatewayWebLoadBalancer $true -deployEgressLoadBalancerNva $false
```

#### For egress-only

```azurepowershell-interactive
New-AzResourceGroupDeployment -ResourceGroupName ha-nva-l7 `
    -TemplateUri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json `
    -deployIngressAppGatewayWebLoadBalancer $false -deployEgressLoadBalancerNva $true
```

#### For ingress-egress

```azurepowershell-interactive
New-AzResourceGroupDeployment -ResourceGroupName ha-nva-l7 `
    -TemplateUri https://raw.githubusercontent.com/mspnp/samples/master/solutions/ha-nva/azuredeploy.json
```

### Redeploying the architecture to the same Resource Group

If you redeploy this template to the same Resource Group, you must specify the same admin user name.  If you do not remember the admin user name, you should first delete the VM resources so that they can be recreated. Azure does not allow redeployment of VMs that change the `linuxConfiguration.adminUsername` property and a change to the user name will cause a conflict.

### Redeploying the architecture using the same Resource Group Name

If you are redeploying the template using the same Resource Group name after deleting the Resource Group or deployed resources, you may need to purge the previously created Azure Key Vault.  Azure requires globally unique names for Key Vault resources and this template will generate a unique name based on your subscription and Resource Group. Key Vault implements a soft-delete feature which is enabled by default. If you want to deploy the template to a Resource Group of the same name, you will have to purge the previous instance of the Key Vault following the instructions at [List, recover, or purge a soft-deleted key vault](https://learn.microsoft.com/azure/key-vault/general/key-vault-recovery#list-recover-or-purge-a-soft-deleted-key-vault).  For testing purposes, you may pass in a `false` value for the template parameter `keyVaultEnableSoftDelete`, but this may cease to work in the future.

### Solution deployment parameters

| Parameter | Type | Description | Default |
|---|---|---|--|
|location|string|The Azure Region to use|[resourceGroup().location]|
|availabilityZoneCount|int|The number of Availability Zones to use|2|
|virtualNetworkName|String|Name of the VNET for all resources|vnet-hanva|
|vnetPrefix|String|CIDR address space definition for VNet|<IP_ADDRESS>/16|
|nvaSubnetPrefix|String|CIDR address prefix definition for NVA subnet|<IP_ADDRESS>/27|
|webSubnetPrefix|String|CIDR address prefix definition for web subnet|<IP_ADDRESS>/24|
|appGatewaySubnetPrefix|String|CIDR address prefix definition for AppGateway subnet|<IP_ADDRESS>/27|
|bastionSubnetPrefix|String|CIDR address prefix definition for Bastion subnet|<IP_ADDRESS>/27|
|gatewaySubnetPrefix|String|CIDR address prefix definition for Gateway subnet|<IP_ADDRESS>/27|
|operatingSystemImage|string|OS Image to use for all VMs|CentOS 7 (latest, LVM)|
|adminUserForVMAccess|String|Admin User for all VMs|null|
|nvaNetworkSecurityGroupName|String|Name of the Network Security Group for the NVA resources|nsg-nva|
|nvaVMNameBase|String|Prefix for naming NVA VMs|vm-nva|
|nvaVMCount|Int|How many NVA VMs to provision|2|
|nvaVMSize|String|VM size for NVA VMs|Standard_D2s_v3|
|webNetworkSecurityGroupName|String|Name of the Network Security Group for the web resources|nsg-web|
|webVMNameBase|String|Prefix for naming web VMs|vm-web|
|webVMCount|Int|How many web VMs to provision|2|
|webVMSize|String|VM size for NVA VMs|Standard_D2s_v3|
|gatewayNetworkSecurityGroupName|String|Name of the Network Security Group for the gateway resources|nsg-gateway|
|allowConnectionFromIPOrCIDRBlock|string|Default value of <IP_ADDRESS>/0 allows management and connections from the entire Internet|<IP_ADDRESS>/0|
|deployAzureBastion|bool|Selection to deploy Azure Bastion|true|
|deployIngressAppGatewayWebLoadBalancer|bool|Selection to deploy the Azure Application Gateway for the NVA VMs and Azure Load Balancer for the web VMs|true|
|deployEgressLoadBalancerNva|bool|Selection to deploy Azure Load Balancer for the NVA VMs|true|
|keyVaultEnableSoftDelete|bool|Enable delete protection (soft delete) for Key Vault (should always be enabled for production environments)|true|
|_artifactsLocation|string|The base URI where artifacts required by this template are located including a trailing '/' or `deployment().properties.template.uri` to use detected deployment root location|[deployment().properties.templateLink.uri]|
|_artifactsLocationSasToken|securestring|The sasToken required to access _artifactsLocation||

## Architecture details

This deployment will set up a simulated environment representing the architectures above.  Clients from the Internet will reach the application via the Public IP on the Application Gateway.  The Application Gateway will forward the request to one of the available deployed NVA VMs. In a real deployment, the NVA VMs may perform other analysis, but for this sample they just forward the request to the web-tier Load Balancer.  The web-tier Load Balancer will forward the request to one of the available web VMs.

For any outbound traffic from the web-tier VMs, they will use the locally configured `http_proxy` environment variable to direct http/https traffic to the internal NVA load balancer, which is configured as a forwarding proxy.

### Simulated NVA VMs

The simulated NVA VMs for this sample are CentOS 7 or RedHat 7 virtual machines running Apache httpd configured as a forward proxy on port 8080 to handle requests from the web-tier, and a reverse proxy to forward all requests on port 80 to the back end Load Balancer IP address (<IP_ADDRESS>). The script to set up the proxy is at: [scripts/simpleproxy.sh](scripts/simpleproxy.sh).

### Simulated Web VMs

The simulated web VMs for this sample are CentOS 7 or RedHat 7 virtual machines running Apache httpd with an index.html file generated to include the VM instance name from the Azure Instance Metadata Service. The script to set up the web server is at: [scripts/simpleweb.sh](scripts/simpleweb.sh).

## Connecting to the web endpoint

To test the ingress path, find the public IP address of the Application Gateway (ag-hanva) and enter it in the address bar of your web browser.  You should see a simple page indicating which back-end web server is providing the response. You can stop any of the NVA or web VMs, and the endpoint solution should automatically route requests through and to available VMs.

## Connecting to deployed VMs with Azure Bastion

An Azure Bastion service is deployed to provide access to the VMs in this architecture. The SSH Key required to connect to the VMs is generated at deployment time and stored in Azure Key Vault using the nested deployment script [nested/storekeyinkeyvault.json](nested/storekeyinkeyvault.json), which was adapted from [Create ssh-keys and store in Key Vault](https://azure.microsoft.com/resources/templates/201-deployment-script-ssh-key-gen/).

In order to access the ssh key from the portal to use Azure Bastion, you will need to add an access policy on the Key Vault and grant yourself ***Secret*** **List** and **Get** permissions on `kv-hanva-{uniquestring}`.

To test the egress path, connect to one of the web-tier VMs using Azure Bastion as described above. You can use a curl command like the following to confirm the egress path is working properly:

```bash
curl -v https://ifconfig.me
```

The output should show the connection to the internal NVA loadbalancer as well as the public IP address that is used to connect to the internet.  See the example below:

```
* About to connect() to proxy <IP_ADDRESS> port 8080 (#0) # Confirm proxy address is private IP of NVA Load Balancer
*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8080 (#0)
> GET http://ifconfig.me/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ifconfig.me
> Accept: */*
...
< Via: 1.1 vm-nva-1.internal.cloudapp.net:8080 # Shows which NVA VM serviced the request
<
* Connection #0 to host <IP_ADDRESS> left intact
<IP_ADDRESS> # Should be the Public IP of the NVA VM
```

## Considerations

### Cost Optimization

The primary drivers of cost in this example architecture are Compute, Load Balancers, the Application Gateway, and storage associated with Azure Monitor.

### Operational Excellence

All resources are currently configured to send all available logs and metrics to the provisioned Azure Monitor workspace. This allows visibility into the performance of all resources and provides the ability to generate alerts for abnormal functioning of the system.

### Performance Efficiency

The sample architecture demonstrates the ability to provision a number of like-functioning VMs across multiple Azure Availability Zones. Larger numbers of VMs can be provisioned based on the throughput requirements of the web tier workload, or the performance requirements of the NVA VMs.

### Reliability

This sample architecture shows how to incorporate Azure Application Gateways and Load Balancers with Network Virtual Appliances running on VMs in a Zone-aware fashion to provide resiliency against fault domain related failures.

### Security

Inbound traffic is restricted to expected traffic for all resources using three Network Security Groups.  In the egress and ingress-egress configurations, outbound traffic from the web tier is also restricted to only allow egress via the layer-7 http/https proxy that is running on the NVA VMs.

The SSH key needed to log into all VMs is stored in Azure Key Vault. In order to use Azure Bastion to connect to the VMs, you must grant the user access to LIST and GET secrets in the Key Vault.  This example uses an empty passphrase on the SSH key, relying on Key Vault RBAC to control access to the secret.

Access to the VMs in the NVA and Web subnets is only achieved through Azure Bastion.

## Acknowledgements

These sample architectures leverage concepts from the [Example Parameterized Deployment With Linked Templates](https://azure.microsoft.com/resources/templates/301-parameterized-linked-templates/) and [Create ssh-keys and store in Key Vault](https://azure.microsoft.com/resources/templates/201-deployment-script-ssh-key-gen/) samples.

## Code of conduct

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information, see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode<EMAIL_ADDRESS>with any additional questions or comments.

repository: mspnp/samples

gha_language: Bicep

repo_name: mspnp/samples


text:
# nodeclrhost - Hosting .NET core in node and electron

![Build status master](https://github.com/sanosdole/nodeclrhost/actions/workflows/build.yml/badge.svg)

_Latest release_ __v6.0.4__: ![Build status release](https://github.com/sanosdole/nodeclrhost/actions/workflows/release.yml/badge.svg)

This project enables writing node/electron applications with .NET.
This is achieved by a native node module (`coreclr-hosting`) that runs a .NET application.
The .NET application uses the `NodeHostEnvironment` library to interact with the node runtime.

Besides running .NET in node, we can also run [.NET Blazor apps](https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor) in an [Electron](https://electronjs.org/) renderer process without WebAssembly.
This enables access to the DOM and the full .NET core framework at the same time (including full debugger support).
Instructions on how to set this up can be found [here](docs/electron-blazor-setup.md).

> `ElectronHostedBlazor` only supports the .NET version matching the major version of this project.

## Simple example

To run a .NET application the following JS code is required:

```js
const coreclrHosting = require('coreclr-hosting');

var exitcode = await coreclrHosting.runCoreApp(pathToAssembly, "Hello", "world");
console.log('.NET entry point returned: ' + exitcode);
```

The .NET application can use the `NodeHost.Instance` in its entry point like this:

```cs
class Program
{
    static Task<int> Main(string[] args)
    {
        var tcs = new TaskCompletionSource<int>();
        var host = NodeHost.Instance;
        var console = host.Global.console;
        console.log($"{args[0]} {args[1]}! Starting timeout");
        host.Global.setTimeout(new Action(() =>
                                {
                                    console.log("Timeout from node");
                                    tcs.SetResult(5);
                                }),
                                1500);
        return tcs.Task;
    }
}
```

This application will output:

```console
Hello world! Starting timeout
Timeout from node
.NET entry point returned: 5
```

## Further examples

The [examples folder](examples) contains examples for:

- [node + dotnet](examples/sample)
- [Running dotnet in Electron browser & renderer processes](examples/electron-sample)
- [Using ElectronHostedBlazor to run a Blazor application in an Electron renderer process](examples/electron-blazor)
- [Running a dotnet MVC app with server side blazor as an Electron application](examples/electron-mvc)

repository: sanosdole/nodeclrhost

gha_language: C#

repo_name: sanosdole/nodeclrhost


text:
---
title: |
    205. Blessed Assurance - Christ in Song
metadata:
    description: |
        Christ in Song 205. Blessed Assurance. Blessed assurance, Jesus is mine! O, what a foretaste of glory divine! Heir of salvation, purchase of God, Born of his Spirit, wash'd in his blood. Chorus: This is my story, this is my song, Praising my Saviour all the day long; This is my story, this is my song, Praising my Saviour all the day long.
    keywords:  |
        Christ in Song, adventhymnals, advent hymnals, Blessed Assurance, Blessed assurance, Jesus is mine!. This is my story, this is my song,
    author: Brian Onang'o
---

#### Advent Hymnals
## 205. BLESSED ASSURANCE
####  Christ in Song,

```txt
1.
Blessed assurance, Jesus is mine!
O, what a foretaste of glory divine!
Heir of salvation, purchase of God,
Born of his Spirit, wash'd in his blood.

Chorus:
This is my story, this is my song,
Praising my Saviour all the day long;
This is my story, this is my song,
Praising my Saviour all the day long.

2.
Perfect submission, perfect delight,
Visions of rapture now burst on my sight.
Angels descending bring from above
Echoes of mercy, whispers of love.  [Chorus]

3.
Perfect submission, all is at rest,
I in my Saviour am happy and blest,
Watching and waiting, looking above,
Filled with his goodness, lost in his love.  [Chorus]

```

- |   -  |
-------------|------------|
Title | Blessed Assurance |
Key | D Major |
Titles | This is my story, this is my song, |
First Line | Blessed assurance, Jesus is mine! |
Author | F. J. Crosby
Year | 1908
Composer| Mrs. Jos. F. Knapp |
Hymnal|  - |
Tune| Blessed assurance, Jesus is mine! |
Metrical pattern | |
# Stanzas |  |
Chorus |  |
Chorus Type |  |
Subjects | Living His Life: Assurance |
Texts | Song of Solomon 2:16 |
Print Texts |
Scripture Song |  |

repository: GospelSounders/adventhymnals

repo_name: GospelSounders/adventhymnals


text:
---
layout: post
title: Source Separation in Music
---

Below we present a detailed overview of our approach for source separation of a musical piece. This part is divided into the following sections: First, we describe the pre-processing steps needed to feed the data into the model.  Next, we describe the model architecture and the details of the approach. Finally, we mention some evaluation metrics for our method and the performance of our model.
We build upon the paper (1) and it forms the main backbone of our approach. However, we improve upon the existing model by using LSTMs layers in the model architecture. We further modify the loss such that it also accounts for increasing the difference between the sources.

## Preprecessing

The figure below shows the spectrogram of an audio file we used in our dataset.
Spectograms are a common way of visualizing audio signals. The x-axis represents time, y-axis represents frequency of sounds at that time. The different intensities of colors are used to express the 3rd dimension - amplitude corresponding to a particular frequency.

The very first preprocessing step is to convert our audio files from dual channel to mono channel, which is obtained by taking the average of the two channels. The figure below shows the spectrogram we obtain:

![_config.yml]({{ site.baseurl }}/images/Initialial_img.PNG)

After the initial preprocessing, we also downsample the signal data and then max-normalize the signal data by time.

We use STFT transform function to create a time-frequency representation of our data. Briefly, the STFT transformation is used to compute the sinusoidal frequency and phase content of local sections of a signal as it changes over time. The idea here is to firstly divide the input spectra into equally sized smaller intervals. Then a Fourier transform is applied to each one of them, thus resulting spectrogram is visualized below:
![_config.yml]({{ site.baseurl }}/images/STFT transform.PNG)

## Model Description

Visually the architecture we use is:

![_config.yml]({{ site.baseurl }}/images/source_sep_model_updated.PNG)

The intuition behind using an LSTM is two-fold.
 1. It can capture the neighbouring features in the input spectra.
 2. It has been shown to be more stable than regular RNNs. Since the pre-processing above transforms the data in a sequence of time frames, that further validates the use of LSTMs, to capture the effect of time-sequencing.

We apply 3 LSTMs layers, with number of hidden units = 1000. The number of layers and numbers of hidden units are the hyperparameters for our model. We obtained the best results with these values of hyperparameters after fine tuning them.

The LSTM layers are followed by two dense layers. These are used to identify each source out of the mixed spectra. The activation function used for the dense layers is **ReLU**.

We then have a masking layer which helps to smoothen the output for each source obtained from each dense layer. Mathematically, this is expressed as:

![_config.yml]({{ site.baseurl }}/images/Masking_eq.PNG)

 Here z_t represents the input spectrogram and yhat_1t and yhat_2t are the predictions for the output spectrogram from the neural network. The y^tilde values are the final separated source values, the masking helps in extracting each component out by masking out the values for the other source.

 The loss function corresponds to applying MSE+discriminative training. Mathematically, this is expressed as:

 ![_config.yml]({{ site.baseurl }}/images/source_sep_loss_func.PNG)

 The 1st and the 3rd terms represent the MSE error between the predicted and the actual output and the 2nd and the 4th term are the discrimative learning terms to increase the difference between the predicted sources with respect to the actual sources.

 We set **gamma** to be **0.5**, which is again a model choice.

 The main highlight of this framework is the fact that we not only train w.r.t to the parameters of the model but also w.r.t. the masking layer. This is practically very useful, since our final objective is to improve the quality of the masked output and not the individual predictions for each source.

 The actual output against which we train(represented above as y_1t, y_2t), represents the two channels in the input dual channel audio file.

Finally, an inverse STFT transform is applied to convert the masked spectrogram which a time-frequency magnitude representation back to audio signals, which we store in .wav file. This acts as input for the transcription model.

Shown below is a visual captured from the training of the network:
![_config.yml]({{ site.baseurl }}/images/source_sep_loss_curve.PNG)

We also experimented with different model architectures, with different hyperparameters such as number of iterations, number of LSTM layers, but did not see much variations in the training process from our current specified architecture.

The model achieves separating the music file into two parts: voice and instrument. Separating the input into each instrument adds several layers of complexity in terms of preprocessing, identifying ground truth labels, defining the loss function, etc. and hence is a direction for future work.

## Dataset
We use the MIR-1K dataset (MIR-Music information retrieval).
It comprises of a variety of songs, a total of 1000 musical clips (hence 1K in the name of dataset). This dataset is annotated with several musical features such as pitch, timbre, etc. We use this dataset since it is a popular dataset used in source separation literature. It captures various aspects of music and hence forms a very powerful dataset to be used for training for source separation. We needed a dataset which was rich and diverse in vocal and instrument audio signals and hence we used this dataset.
Since our target transcription is for piano only, we use datasets for evaluation which have vocal and only piano as the background instrument.

## Results
Below we visualization of the two source separated components:
1. Original spectrogram

![_config.yml]({{ site.baseurl }}/images/original_spec.PNG)

2. Only-voice spectrogram

![_config.yml]({{ site.baseurl }}/images/voice_spec.PNG)

3. Only-music spectrogram

![_config.yml]({{ site.baseurl }}/images/music_spec.PNG)

## References
1. Huang, Po-Sen, et al. "Singing-Voice Separation from Monaural Recordings using Deep Recurrent Neural Networks." ISMIR. 2014.

2. Chandna, Pritish, et al. "Monoaural audio source separation using deep convolutional neural networks." International Conference on Latent Variable Analysis and Signal Separation. Springer, Cham, 2017.

3. MIR-1K dataset:
Chao-Ling Hsu,   DeLiang Wang, Jyh-Shing Roger Jang, and Ke Hu, “ A Tandem Algorithm for Singing Pitch Extraction and Voice Separation from Music Accompaniment,” IEEE Trans. Audio, Speech, and Language Processing,  2011

[BACK HOME](https://subhasreesengupta.github.io/project-intro/)
[NEXT: Predominant Music Identification](https://subhasreesengupta.github.io/predominant-instrument/)


text:
---
layout: page
permalink: /about/
---

# About

I am currently working at Lyft as a software engineer. Before I was a data scientist at Lidl. I finished my computer science master at FU Berlin with a specialization in machine learning and deep learning. During my master study, I was a student assistant in workgroup mathematics of machine learning
(Resume).

Python is my primary programming language. Numpy, Pandas, sklearn, Matplotlib are Python libraries that I use every day. As for the deep learning framework I use Keras. I also have experience in backend development using Flask and Fastapi. In my free time, I enjoy doing leetcode in Python. So far I have finished 500 leetcode problems.

I can also program in Javascript. I learned React in my free time. I am planning to learn more in frontend and in the future become a full-stack developer with machine learning and deep learning knowledge equipped.

I would like to use this blog not only to note down tech-problems I encounter and how I solve them but also to share the paper or books I recently read.


text:
# ActionNotFound
This is a Rails plugin to see not found action.

## Usage
In your Gemfile.

```ruby
group :development do
  gem "action_not_found"
end
```

and execute rake task `action_not_found`.

```sh
examples GET  /examples(.:format) examples#index
         POST /examples(.:format) examples#create #=> not found action
   tests GET  /tests(.:format)    tests#index #=> not found controller file
    root      /                   examples#index

4 routes, 1 not found actions, 1 not controller files
```


text:
### [CVE-2018-7544](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-7544)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

** DISPUTED ** A cross-protocol scripting issue was discovered in the management interface in OpenVPN through 2.4.5. When this interface is enabled over TCP without a password, and when no other clients are connected to this interface, attackers can execute arbitrary management commands, obtain sensitive information, or cause a denial of service (SIGTERM) by triggering XMLHttpRequest actions in a web browser. This is demonstrated by a multipart/form-data POST to http://localhost:23000 with a "signal SIGTERM" command in a TEXTAREA element. NOTE: The vendor disputes that this is a vulnerability. They state that this is the result of improper configuration of the OpenVPN instance rather than an intrinsic vulnerability, and now more explicitly warn against such configurations in both the management-interface documentation, and with a runtime warning.

### POC

#### Reference
- http://blog.0xlabs.com/2018/03/openvpn-remote-information-disclosure.html

#### Github
No PoCs found on GitHub currently.

repository: trickest/cve

repo_name: trickest/cve


text:
A large language model (LLM) is a type of machine learning model that can do a lot of cool things with language. It can write text that sounds like it was written by a human, answer questions like a human would, and even translate text from one language to another. LLMs are trained using a lot of data and deep learning algorithms to understand language better. 😎

## Transformer Architecture

- **What is the transformer architecture?** - The transformer architecture is a deep neural network architecture that was introduced in the paper "Attention Is All You Need" by Vaswani et al..
- **How does the transformer architecture work?** - The transformer architecture uses self-attention to compute representations of input sequences.
- **What are some applications of the transformer architecture?** - The transformer architecture has been used for a wide range of applications, including machine translation, text classification, and text generation.

## Stages of building LLM

![gpt_stage](/images/rlhf/gpt-stages.png)
[Image courtesy](https://www.youtube.com/watch?v=bZQun8Y4L2A&pp=ygUPYW5kcmVqIGthcnBhdGh5)

LLM starts with the pre-training stage where the model learns to predict next token given it is an auto-regressive language model. Pre-training is done an internet-scale dataset.

#### Scaling law
Scaling law helps to identify relationship among model parameters, dataset size and compute budget. There is a powerlaw relationship, meaning two quantities are proportional to each other based on power raised to a constant:
- Testloss vs compute budget
- Testloss vs dataset size (number of tokens)
- Testloss vs number of model parameters

Refer to [Chinchilla scaling law: Training Compute-Optimal Large Language Models](https://openreview.net/pdf?id=iBBcRUlOAPR)

Following scaling law was followed during pretraining BloombertGPT.
![bgpt_scaling_polity](/images/rlhf/scaling-bgpt.png)

## Supervised Fine-Tuning

- **What is fine-tuning?** - Fine-tuning is the process of adapting a pre-trained language model to a specific task or domain.
- **How does fine-tuning work?** - Fine-tuning involves training the pre-trained language model on a small amount of task-specific data.
- **What are some applications of fine-tuning?** - Fine-tuning can be used for a wide range of applications, including sentiment analysis, named entity recognition, and text classification.
- **Fine-tuning examples:**

- [Finetune llama2 with HF on sagemaker](https://www.philschmid.de/sagemaker-llama2-qlora)

- [Fine-tuning recipe with Parameter efficient finetuning LoRA and evaluation](https://github.com/MonirZaman/MonirZaman.github.io/blob/master/_posts/generative-ai-with-llm/Lab_2_fine_tune_generative_ai_model.md)

- [Interactively fine-tune Falcon-40B and other LLMs on Amazon SageMaker Studio notebooks using QLoRA](https://aws.amazon.com/blogs/machine-learning/interactively-fine-tune-falcon-40b-and-other-llms-on-amazon-sagemaker-studio-notebooks-using-qlora/)

## RLHF: Reinforcement Learning from Human Feedback
 - [Youtube lecture by Hyung from OpenAI](https://youtu.be/zjrM-MW-0y0) [Slides](https://docs.google.com/presentation/d/13Tylt2SvKvBL2hgILy5CmBtPDv3rXlVrQp01OzAe5Xo/edit#slide=id.g238b2698243_0_791)
 - [Blog by Chip](https://huyenchip.com/2023/05/02/rlhf.html)
 - Reading notes on RLHF

**Reward model (RM)**: RM is used to model human feedback. Reward model is also a language model except for the last layear is the linear layer that outputs a reward value. Given two completion $$y_{i}$$ and $$y_{j}$$, objective to model the probability $$p_{ij}$$ which denotes the confidence for $$y_{i}$$ is better than $$y_{j}$$:

![reward](/images/rlhf/reward.png)

**Proximal policy optimization (PPO)**: Next, we apply reinforcement learning that trains a policy aka language model parameters to generate text with higher reward based on the reward model. It samples many prompts and uses the language model to generate sequence for these prompts. Objective is to maximize the expected reward i.e., weighted summation of completion rewards with weights as the probability of the completion.

Proximal policy optimization (PPO) is used to compute the gradient. Iterative algorithm like gradient ascend is used to solve the optimization objective.

Steps of policy gradient training steps:
- Initialize parameters of the language model from Supervised finetuned mode
- Samples prompt from dataset, generates sequence for the prompts and current policy
- Calculate reward for the prompt and completion
- Calculate gradients and update the parameters of the language model

**Regularization**
Vanilla policy gradient can over optimize to the Reward model. A per-token KL divergence from SFT distribution penalty added as a regularization. This is to keep some of the variation from SFT model.

![reward](/images/rlhf/reg.png)

## Langchain handbooks
- LangChain for LLM Application Development \| [DeepLearning.ai](https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/) \| [Student copy](https://github.com/MonirZaman/MonirZaman.github.io/blob/master/_posts/LangChain-for-LLM-Application-Development/)
- LangChain: Chat with Your Data \| [DeepLearning.ai](https://www.deeplearning.ai/short-courses/langchain-chat-with-your-data/) \| [Student copy](https://github.com/MonirZaman/MonirZaman.github.io/blob/master/_posts/LangChain-Chat-with-Your-Data/)
- [LangChain handbook](https://www.pinecone.io/learn/series/langchain/)

## Online course
- [NYU 2023 spring course on Natural Language Processing](https://nyu-cs2590.github.io/spring2023/calendar/)
- Building Systems with the ChatGPT API \| [DeepLearning.ai](https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/) \| [Student copy](https://github.com/MonirZaman/MonirZaman.github.io/blob/master/_posts/Building-Systems-with-the-ChatGPT-API/LLM.md)
- [Full stack's course on LLM](https://fullstackdeeplearning.com/llm-bootcamp/spring-2023/)

## Evaluation of LLM
- [Holistic evaluation of language model HELM](https://youtu.be/HJGccJh07Os)

## GPT from scratch
Generative pretrained-transformer created by Andrej Karpathy for education purpose
 - [Video](https://www.youtube.com/watch?v=kCc8FmEb1nY)

## Knowledge representation and Hallucination
- Hallucination refers to mistakes in the generated text that are statistically plausible but are in fact incorrect or nonsensical
- [Stanford lecture](https://www.youtube.com/watch?v=4ynrGLIuPv4)

## Application
- [flyte q/a bot with langchain, faiss, gpt](https://flyte.org/blog/building-flytegpt-on-flyte-with-langchain)

## Select technical concepts related to LLM and Transformer

**Byte Pair Encoding (BPE)**
Byte Pair Encoding (BPE) and SentencePiece are both subword tokenization methods that are used in NLP. BPE is a data-driven method that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. Subword tokenization with BPE helps in effectively tackling the concerns of out-of-vocabulary words. SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training

**Positional embedding**
Positional embeddings are added to the word embeddings once before the first layer. Each position t within the sequence gets different embedding when t is even then use sine function if t is odd then consine function.
- [Referrence](https://towardsdatascience.com/understanding-positional-encoding-in-transformers-dc6bafc021ab)


text:
Validate Argument types.
========================

Validate parameters you pass into methods. Can log or throw errors.

Mostly written for the purpose of understanding npm, writing a module, functional programming, etc. There are lots of argument validators I could have used.

## Installation

```
  npm install https://github.com/emileswain/validate-arg-types/tarball/master --save

```

```
  npm install validate-arg-types --save

```

## Usage

##### New instance
```javascript
  var ValidateArgs = require('validate-arg-types');

  var validator = ValidateArgs('MyMethod');

```

##### To Throw or not to Throw
Fail fast?, in which case call throwThem first. All tests will throw at the first opportunity.

```javascript

  validator.throwThem().isString('isAString');

```

or fail last and once.

```javascript

  validator.throwThem(false).isString('isAString').throw();

```

you may want to log all errors detected.
```javascript

  validator.throwThem(false).isString('isAString').log().throw();

  validator.isString('isAString').log().throw();

```

##### Basic tests
```javascript

  validator.throwThem().isString('isAString');
  validator.throwThem().isArray([]);
  validator.throwThem().isObject({});
  validator.throwThem().isNumber(1234);
  validator.throwThem().isFunction(function(){});

```

##### Advanced tests.

```javascript

  validator.throwThem().isObject({name:'firstname'}).has('name');

```

## Contributing

I'm no javascript expert, writing this was in part a foray into writing an npm module on a whim.

## Thanks to
https://quickleft.com/blog/creating-and-publishing-a-node-js-module/

## Pondering

For the purpose of experimentation, how to re-factor to add in <,>,= tests on properties.

## Release History

* 1.0.0 follows semver recommendations.
* 0.2.0 Adding functionality, docs.
* 0.1.0 Initial release

repository: emileswain/validate-arg-types

repo_name: emileswain/validate-arg-types


text:
# Instabot

Languages: [🇬🇧](/docs/en/README.md) [🇷🇺](/docs/ru/README.md)

[![Telegram Chat](https://img.shields.io/badge/chat%20on-Telegram-blue.svg)](https://t.me/instabotproject)
[![Build Status](https://travis-ci.org/instagrambot/instabot.svg?branch=master)](https://travis-ci.org/instagrambot/instabot)

![Instabot is better than other open-source bots!](/docs/img/tag%20instabot.png "Instabot is better than other open-source bots!")

repository: Alligator-bot/instabot

repo_name: Alligator-bot/instabot


text:
---
layout: default
title: Discrete Observation Requirement
nav_order: 3
permalink: /documentation/user-guide/data-requirements
parent: User Guide
grand_parent: Documentation
---

# DAME-FLAME Data Requirements
{: .no_toc }

    Table of contents

  {: .text-delta }
1. TOC
{:toc}

## Discrete Observation Requirement for DAME-FLAME

The main requirements for researchers to justify the use of matching methods also hold for DAME-FLAME. Those are discussed [here](https://almost-matching-exactly.github.io/DAME-FLAME-Python-Package/documentation/user-guide/to-match-or-not). In sum, we require SUTVA, ignorability, and some overlap.

Additionally, we require that all observational covariates be discrete. The outcome data can be continuous. The treatment indicator column must be binary.

We **do not** recommend users bin continous covariates.

The exception that could be made is a scenario where users are confident they are binning variables in a way that is a typical for their research. In this scenario, categories must be pre-defined and considered acceptable in their domain of work.

## Example of Acceptable Binning

In research incorporating infant births, gestation time could be binned and used in `DAME-FLAME` as an observational covariate. Classifications of gestational age are well established norms adhered to in obstetric publications.  *Early term* is considered 37 0/7 weeks of gestation through 38 6/7 weeks of gestation, *full term*  is 39 0/7 weeks of gestation through 40 6/7 weeks of gestation, etc. The American College of Obstetricians and Gynecologists and the Society for Maternal-Fetal Medicine endorse and encourage these categories [1].

## Example using Binned Data

This documentation provides an example [here](https://almost-matching-exactly.github.io/DAME-FLAME-Python-Package/examples/natality_experiment/), where  we aim to examine the effect of “extreme smoking” of a mother during pregancy on her newborn’s birthweight. We discuss using two binned continuous variables, father's age, and mother's education attainment.

## Input Format Example

Below, we provide an example of the format that the `DAME-FLAME` package requires input data to be in. The input data can be either a file, or a **Python Pandas Data Frame**. All covariates in the input data should be categorical covariates. If there are continuous covariates, we only recommend users regroup the data if they are sure they are doing so in a way that is typical of their research question. In addition to input observational data columns, the input data must contain (1) A column indicating the outcome variable as an *integer* or *float* data type, and (2) A column specifying whether a unit is treated or control (treated = 1, control = 0) as an *integer* data type. There are no requirements for input data column names or order of columns. Below is an example of input data with n units and m covariates.

*Column-name / unit-id*  | x_1 | x_2 |...| x_m | outcome | treated
--- | --- | --- | --- | --- | --- | --- |
**1** | 2 | 3 | ... | 4 | 9 | 0
**2** | 1 | 3 | ... | 3 | 5.5 | 1
**3** | 1 | 4 | ... | 5 | -1 | 0
... | ... | ... | ... | ... | ... | ...
**n** | 0 | 5 | ... | 0 | 1 | 1
*Data Type*| *integer* | *integer* | *integer* | *integer* |  *numeric* | *0 or 1* |

The holdout training set, if provided, should also follow the same format.

References

  [1]

    ACOG Committee Opinion No 579: Definition of Term Pregnancy


text:
# Weather-Dashboard-

# Description

A weekly weather forecast application for your sunny self!

# Table of Contents

* [Usage](#Usage)

* [Github](#Github)

* [License](#License)

# Usage

You can use it if you need it, I make no promises, I'm not a witch.

# Github

[Deployed](https://lawriedrew.github.io/Weather-Dashboard-/)

[Repo](https://github.com/LawrieDrew/Weather-Dashboard-.git)

# License

MIT License

Copyright (c) 2021 Lauren Drew

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

repository: LawrieDrew/Weather-Dashboard-

repo_name: LawrieDrew/Weather-Dashboard-


text:
# Build the Graph

Using the information from [01-stringdb](../01-stringdb), [02-mapping](../02-mapping), and [03-metadata](../03-metadata), we can build a more self-describing graph where,

- Each gene is of type `biolink:Gene`
- Each protein is of type `biolink:Protein`
- Each interaction is of type `biolink:GeneToGeneAssociation`
- The predicate/relationship that link two protein nodes is `biolink:interacts_with`
- The relation that further qualifies the interaction is [RO:0002434](https://www.ebi.ac.uk/ols/ontologies/ro/properties?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FRO_0002434)
- The interaction score can be represented as a property on the edge
- Each link between a gene and its protein can be represented via the `biolink:has_gene_product`
- The relation that further qualifies the link is [RO:0002205](https://www.ebi.ac.uk/ols/ontologies/ro/properties?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FRO_0002205)
- Each gene node can have an `xref` property that holds its alternate NCBIGene identifier
- Each genomic location information is of type `biolink:GenomicSequenceLocalization`
- The predicate/relationship that links a genomic entity to a reference sequence is `biolink:has_sequence_location`
- The relation that further qualifies the link is `faldo:location`
- The location information can be represented as properties on the edge

[graph_nodes.tsv](graph_nodes.tsv) represents all the nodes in the final graph in a TSV format.

A snippet of nodes.tsv,

```
id	category	name	symbol	xref	synonym	in_taxon
ENSEMBL:ENSG00000003056	biolink:Gene	mannose-6-phosphate receptor, cation dependent	M6PR	HGNC:6752|NCBIGene:4074	46-kDa mannose 6-phosphate receptor|CD Man-6-P receptor|Mr 46,000 Man6PR|cation-dependent mannose-6-phosphate receptor|small mannose 6-phosphate receptor	NCBITaxon:9606
ENSEMBL:ENSG00000005381	biolink:Gene	myeloperoxidase	MPO	HGNC:7218|NCBIGene:4353	myeloperoxidase	NCBITaxon:9606
...
```

[graph_edges.tsv](graph_edges.tsv) represents all the nodes in the final graph in a TSV format.

A snippet of edges.tsv,

```
subject	predicate	object	relation	association_type	combined_score	start_interbase_coordinate	end_interbase_coordinate	strand
ENSEMBL:ENSG00000003056	biolink:interacts_with	ENSEMBL:ENSG00000005381	RO:0002434	biolink:GeneToGeneAssociation	195
ENSEMBL:ENSG00000003056	biolink:interacts_with	ENSEMBL:ENSG00000166598	RO:0002434	biolink:GeneToGeneAssociation	222
...
```

## Loading into Neo4j

One can easily load this graph into Neo4j using [KGX](https://github.com/biolink/kgx) with the following command,

```sh
kgx neo4j-upload --input-format tsv \
                    --uri http://localhost:7474 \
                    --username neo4j \
                    --password test \
                    graph_nodes.tsv graph_edges.tsv
```

> The above command assumes that you have a Neo4j running locally and is available at http://localhost:7474

repository: biolink/biolink-model-workshop

repo_name: biolink/biolink-model-workshop