text:
---
title:  "Transactional Writes in Spark"
excerpt: "Covering Spark's default and Databrick's Transactional Write strategies used to write the result of a job to a destination and guarantee no partial results are left behind in case of failure."
classes: wide
categories: [spark]
toc: true
---

{% include figure image_path="/assets/images/spark/transactional_writes/intro.png" %}

Since Spark executes an application in a distributed fashion, it is impossible to atomically write the result of the job. For example, when you write a `Dataframe`, the result of the operation will be a directory with multiple files in it, one per `Dataframe`'s partition (e.g `part-00001-...`). These partition files are written by multiple Executors, as a result of their partial computation, but what if one of them fails?

To overcome this Spark has a concept of *commit protocol*, a mechanism that knows how to write partial results and deal with success or failure of a write operation.

In this post I'll cover three types of transactional write *commit protocols* and explain the differences between them. The protocols being addressed are _Hadoop Commit V1_, _Hadoop Commit V2_ and Databrick's _DBIO Transactional Commit_.

## Transactional Writes

In Spark the transactional write *commit protocol* can be configured with `spark.sql.sources.commitProtocolClass`, which by default points to the [SQLHadoopMapReduceCommitProtocol](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala) implementing class, a subclass of [HadoopMapReduceCommitProtocol](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala#L140).

There are two versions of this commit algorithm, configured as `1` or `2` on `spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version`.

In version 1 Spark creates a temporary directory and writes all the staging output (task) files there. Then, at the end, when all tasks compete, Spark Driver moves those files from temporary directory to the final destination, deletes the temporary directory and creates the `_SUCCESS` file to mark the operation as successful. You can check more details of this process [here](https://www.waitingforcode.com/apache-spark-sql/apache-spark-success-anatomy/read).

The problem of version 1 is that for many output files, Driver may take a long time in the final step. Version 2 of *commit protocol* solves this slowness by writing directly the task result files to the final destination, speeding up the commit process. However, if the job is aborted it will leave behind partial task result files in the final destination.

Spark [supports](https://issues.apache.org/jira/browse/SPARK-20107) *commit protocol* version configuration since its 2.2.0 version, and at the time of this writing Spark's latest 3.0.1 version has a default value for it depending on running environment (Hadoop version).

{% include figure image_path="/assets/images/spark/transactional_writes/spark_protocol_version_conf.png" %}

## Transactional Writes on Databricks

As we previously saw, Spark's default *commit protocol* version 1 should be used for safety (no partial results) and version 2 for performance. However, if we opt for data safety version 1 is not suitable for cloud native setups, e.g writing to Amazon S3, due to differences cloud object stores have from real filesystems, as explained in [Spark's cloud integration guide](https://github.com/apache/spark/blob/32a0451376ab775fdd4ac364388e46179d9ee550/docs/cloud-integration.md).

To solve the problem of left behind partial results on job failures and performance issues, Databricks implemented their own transactional write protocol called *DBIO Transactional Commit*.

You can check all the problems with Spark's default *commit protocol* and the details behind Databrick's custom implementation in [Transactional I:O on Cloud Storage in Databricks](https://www.youtube.com/watch?v=w1_aOPj5ILw) talk by Eric Liang and [Transactional Writes to Cloud Storage on Databricks](https://databricks.com/blog/2017/05/31/transactional-writes-cloud-storage.html) blog post.

Let's take a look with examples how transactional writes work on Databricks, implemented with the *commit procotol* below:

{% include figure image_path="/assets/images/spark/transactional_writes/0.png" %}

By writing the `Dataframe` to a directory, mount point of an Azure Storage container, the following files were created:

{% include figure image_path="/assets/images/spark/transactional_writes/1.png" %}

For each transaction, a unique transaction id `<tid>` is generated. At the start of each transaction, Spark creates an empty `_started_<tid>` file. If a transaction is successful, a `_committed_<tid>` file is created. Notice that each partition file also has the transaction identifier in it.

The content of the committed file for this transaction contains all the added partition files, as follows:

```
%fs head --maxBytes=10000 /mnt/users/anzabol/data/_committed_7628641425835151768
```

```json
{
  "added": [
    "part-00000-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-37-1-c000.snappy.parquet",
    "part-00001-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-38-1-c000.snappy.parquet",
    "part-00002-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-39-1-c000.snappy.parquet"
  ],
  "removed": []
}
```

After writing another `Dataframe` to the same destination, with the *append* mode, the respective transaction files were created together with the partition files.

{% include figure image_path="/assets/images/spark/transactional_writes/2.png" %}

Once again, the committed file should contain all the files that were written during that transaction:

```json
{
  "added": [
    "part-00000-tid-3452754969657447620-98b3663b-fbe5-49c1-bbbc-9d0a2413fc20-44-1-c000.snappy.parquet",
    "part-00001-tid-3452754969657447620-98b3663b-fbe5-49c1-bbbc-9d0a2413fc20-45-1-c000.snappy.parquet"
  ],
  "removed": []
}
```

My last test consisted in writing again the first `Dataframe` but with *overwrite* mode, to see what happens. In this case the committed file looked like this:

```json
{
  "added": [
    "part-00000-tid-5097121921988240910-6ec8b872-eaca-4dd8-83b6-0822a8a18189-50-1-c000.snappy.parquet",
    "part-00001-tid-5097121921988240910-6ec8b872-eaca-4dd8-83b6-0822a8a18189-51-1-c000.snappy.parquet",
    "part-00002-tid-5097121921988240910-6ec8b872-eaca-4dd8-83b6-0822a8a18189-52-1-c000.snappy.parquet"
  ],
  "removed": [
    "_SUCCESS",
    "part-00000-tid-3452754969657447620-98b3663b-fbe5-49c1-bbbc-9d0a2413fc20-44-1-c000.snappy.parquet",
    "part-00000-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-37-1-c000.snappy.parquet",
    "part-00001-tid-3452754969657447620-98b3663b-fbe5-49c1-bbbc-9d0a2413fc20-45-1-c000.snappy.parquet",
    "part-00001-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-38-1-c000.snappy.parquet",
    "part-00002-tid-7628641425835151768-a8c83893-f6af-49ab-9e0d-8e1181b7e684-39-1-c000.snappy.parquet"
  ]
}
```

As you can see, all the existing partition and success files were deleted, and the partition files of the written `Dataframe` added.

Since each successful transaction leaves a committed file, when reading data Spark ignores all the files that contain a `<tid>` for which there is no commit file. This way there is no way of reading corrupted / dirty data.

To clean up partition files of uncommitted transactions, there is [VACUUM](https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html?_ga=2.110458629.1898107027.1604185218-1080905499.1603716625#clean-up-uncommitted-files) operation, that accepts an output path and the retention period.

## Conclusion

Depending on your execution environment or expected behavior in case of application execution failure, different *commit protocol* should be chosen.

If your Spark application runs in Hadoop environment, you should use Spark's *Hadoop Commit* protocol, that can be of version 1 or version 2. Version 1 is slower, but guarantees that no partial files will be left after a Spark Job is aborted. Version 2 is faster, but you will end up with partial results in destination directory when performing a write operation and it fails. One of these versions should be configured depending if safety or performance is your main concern.

If your Spark application runs in Databricks environment, you should definitely stick with their own *commit protocol* implementation called *DBIO Transactional Commit*, which is configured by default. This protocol offers at the same time safety (no partial/corrupt data) and performance, since it is optimized for cloud storage.

## Resources

[Apache Spark's _SUCESS anatomy](https://www.waitingforcode.com/apache-spark-sql/apache-spark-success-anatomy/read)

[What is the difference between mapreduce.fileoutputcommitter.algorithm.version=1 and 2](http://www.openkb.info/2019/04/what-is-difference-between.html)

[Transactional I:O on Cloud Storage in Databricks - Eric Liang](https://www.youtube.com/watch?v=w1_aOPj5ILw)

[Transactional Writes to Cloud Storage on Databricks](https://databricks.com/blog/2017/05/31/transactional-writes-cloud-storage.html)

[Notebook - Testing Transactional Writes](https://docs.databricks.com/_static/notebooks/dbio-transactional-commit.html)

[Spark 2.0.0 cluster takes a long time to append data](https://kb.databricks.com/data/append-slow-with-spark-2.0.0.html)

gha_language: JavaScript


text:
# **Behavioral Cloning**

## Writeup Template

### You can use this file as a template for your writeup if you want to submit it as a markdown file, but feel free to use some other method and submit a pdf if you prefer.

---

**Behavioral Cloning Project**

The goals / steps of this project are the following:
* Use the simulator to collect data of good driving behavior
* Build, a convolution neural network in Keras that predicts steering angles from images
* Train and validate the model with a training and validation set
* Test that the model successfully drives around track one without leaving the road
* Summarize the results with a written report

[//]: # (Image References)

[image1]: ./examples/placeholder.png "Model Visualization"
[image2]: ./center_2018_12_08_16_16_09_662.jpg "Grayscaling"
[image3]: ./right_2018_12_08_16_16_09_942.jpg "Recovery Image"
[image4]: ./right_2018_12_08_16_16_10_496.jpg "Recovery Image"
[image5]: ./right_2018_12_08_16_16_10_844.jpg "Recovery Image"
[image6]: ./center_2016_12_01_13_31_12_937.jpg "Normal Image"
[image7]: ./center_2016_12_01_13_31_12_937_flipped.jpg "Flipped Image"
[image8]: ./cnn-architecture-624x890.png "DAVE-2 Image"
[image9]: ./mse_graph.png "Without dropout"
[image10]: ./mse_graph_w_dropout.png "with dropout"
[image11]: ./mse_graph_w_dropout_epoch4.png "with dropout epoch4"

## Rubric Points
### Here I will consider the [rubric points](https://review.udacity.com/#!/rubrics/432/view) individually and describe how I addressed each point in my implementation.

---
### Files Submitted & Code Quality

#### 1. Submission includes all required files and can be used to run the simulator in autonomous mode

My project includes the following files:
* model.py containing the script to create and train the model
* drive.py for driving the car in autonomous mode
* model.h5 containing a trained convolution neural network ( Generated with my local machine )
* writeup_report.md summarizing the results

#### 2. Submission includes functional code
Using the Udacity provided simulator and my drive.py file, the car can be driven autonomously around the track by executing
```sh
python drive.py model.h5
```

#### 3. Submission code is usable and readable

The model.py file contains the code for training and saving the convolution neural network. The file shows the pipeline I used for training and validating the model, and it contains comments to explain how the code works.

### Model Architecture and Training Strategy

#### 1. An appropriate model architecture has been employed

My model used NVIDIA's DAVE-2 CNNs ( shown below )
![DAVE-2][image8]

NVIDIA's intention is to input YUV image, but I used it with RGB.
Model is described between line 111 - 126 in model.py.

#### 2. Attempts to reduce overfitting in the model

The model contains dropout layers in order to reduce overfitting (model.py lines 121, 123, 125 ).

The model was trained and validated on different data sets to ensure that the model was not overfitting.
 - From different recordings (code line 41).

The model was tested by running it through the simulator and ensuring that the vehicle could stay on the track.

#### 3. Model parameter tuning

The model used an adam optimizer, so the learning rate was not tuned manually (model.py line 128).

#### 4. Appropriate training data

Training data was chosen to keep the vehicle driving on the road.
I used a combination of center , left image and right image.
Flipped image is also used.

### Model Architecture and Training Strategy

#### 1. Solution Design Approach

- Implemented LeNet based CNN.
  It worked correctly and can control almost appropriately, but it goes off the road if I run it through night ( 12 Hour long run ).
- Implemented DAVE-2 CNN.
- In order to gauge how well the model was working, I split my image and steering angle data into a training and validation set. I found that my first model had a low mean squared error on the training set but a high mean squared error on the validation set. This implied that the model was overfitting.
- To combat the overfitting, I added three dropout layers. But there was only little difference.
    ![over fitting ][image9]
    ![after adding drop out][image11]

- At the end of the process, the vehicle is able to drive autonomously around the track without leaving the road.

#### 2. Final Model Architecture

The final model architecture (model.py lines 111-126) consisted of a convolution neural network with the following layers and layer sizes translated from NVIDIA DAVE-2 CNN.

            _________________________________________________________________
            Layer (type)                 Output Shape              Param #
            =================================================================
            lambda_4 (Lambda)            (None, 160, 320, 3)       0
            _________________________________________________________________
            cropping2d_4 (Cropping2D)    (None, 65, 320, 3)        0
            _________________________________________________________________
            conv2d_16 (Conv2D)           (None, 31, 158, 24)       1824
            _________________________________________________________________
            conv2d_17 (Conv2D)           (None, 14, 77, 36)        21636
            _________________________________________________________________
            conv2d_18 (Conv2D)           (None, 5, 37, 48)         43248
            _________________________________________________________________
            conv2d_19 (Conv2D)           (None, 3, 35, 64)         27712
            _________________________________________________________________
            conv2d_20 (Conv2D)           (None, 1, 33, 64)         36928
            _________________________________________________________________
            flatten_4 (Flatten)          (None, 2112)              0
            _________________________________________________________________
            dense_13 (Dense)             (None, 100)               211300
            _________________________________________________________________
            dropout_10 (Dropout)         (None, 100)               0
            _________________________________________________________________
            dense_14 (Dense)             (None, 50)                5050
            _________________________________________________________________
            dropout_11 (Dropout)         (None, 50)                0
            _________________________________________________________________
            dense_15 (Dense)             (None, 10)                510
            _________________________________________________________________
            dropout_12 (Dropout)         (None, 10)                0
            _________________________________________________________________
            dense_16 (Dense)             (None, 1)                 11
            =================================================================
            Total params: 348,219
            Trainable params: 348,219
            Non-trainable params: 0

Here is a visualization of the DAVE-2 architecture from Web
![DAVE-2][image8]

#### 3. Creation of the Training Set & Training Process

To capture good driving behavior, I first recorded two laps on track one using center lane driving. Here is an example image of center lane driving:

![alt text][image2]

I then recorded the vehicle recovering from the left side and right sides of the road back to center so that the vehicle would learn to .... These images show what a recovery looks like starting from ... :

![Almost off the course][image3]
![Started recovery][image4]
![Recovered][image5]

Then I repeated this process on track two in order to get more data points.

To augment the data sat, I also flipped images and angles thinking that this would help reduce deviation of data set.
For example, here is an image that has then been flipped:

![Normal image][image6]
![Flipped image][image7]

After the collection process, I had 8106 number of data points. I then preprocessed this data by normalize around 0 center.

I finally randomly shuffled the data set and put 20% of the data into a validation set.

I used this training data for training the model. The validation set helped determine if the model was over or under fitting.
The ideal number of epochs was 4 as evidenced by dropping validation loss.
    ![after adding drop out][image10]

I used an adam optimizer so that manually training the learning rate wasn't necessary.

I thought that dropout doesn't work in this project for overcome overfitting.
I tried the following model also, but still overfitting.
<EMAIL_ADDRESS>
Majority of my training data is from sample data provided by Udacity.
It seems that I needed to record more data by myself to overcome overfitting.
But I was able to record only around 100 lines with driving simulator on my local machine.
I should have resolved this first and produce more data by myself.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# Shapes

Three types of shapes are currently available:

   * Circle
   * Rectangle
   * Triangle

More shapes will be added in the future.

# Customising shape

It is possible to customise how the shape looks.

## Color

If the white rectangle is clicked, a color picker will appear.

Chose the desired color and press OK.

## Label

Simply write your desired label in the text box, it will appear on the node when

you drag & drop it next time.

## Stroke width

Chose your desired stroke width and press Enter. Before you drag & drop it,

you can modify it several times until you find the best one.


text:
# Technical Test for Codeko

# Chuck Norris Web :rocket:

Chuck Norris is a web project for a technical test by codeko.

# Libraries

I have been using libraries like npm, tailwind, axios, react-router-dom, etc...

Deploy: [CodekoChuck](https://codeko-chuck-gijq5ttfg-angelhpascual.vercel.app/)

## Installation

Use the package manager [npm](https://www.npmjs.com/) to install the app.

```bash
npm install
cd /codekochuck
npm run start
```

## Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Enjoy!

## License

[MIT](https://choosealicense.com/licenses/mit/)

repository: Angelhpascual/codekoChuck

repo_name: Angelhpascual/codekoChuck


text:
# Color Cauldron

## Disclaimer

Consider viewing this README on Xcode, if possible. I plan to implement markdown features that only work on Xcode for a customized experience :D

## Process

Color Cauldron is a color manipulation desktop application for MacOS. Well, actually, I went for this project without too much thought. I like colors, and I wanted to practice a bit of AppKit, ApplicationKit, Cocoa, MacOS, OS X (did I forget any?) development.

I know desktop applications are out of fashion nowadays, but I can't help but like them. They look so robust, don't need an internet connection, provide an immersive experience, etc. That's what I feel, at least.

I didn't research about colors before starting this project because I was eager to start, and I don't even know if this app has or will have any practical use. But man, they sure are not simple. I thought I could translate all color representation models into RGB to make my job easier, but there are things like color spaces, color models, gamuts, etc, etc.

For example, the sRGB color space, which uses the RGB color model, has a gamut (color range) that doesn't cover all the colors the human eye can see. So it's not possible to just translate every other representation to it. Still, the monitors we have can only display in RGB because of how they work. It's tough.

## App

Well, that's enough about the process. Let's talk about the app:

In Color Cauldron, you can choose a color using the default MacOS Color Panel, and the color you have selected will color the app's circle. This is a document based app, so you can save and load files which store the color you had selected before you saved the file. It's as simple as that.

## Future

This is not a serious project, so I'm planning to do what I feel like doing. I've thought about adding info about the selected color, like ways to represent them using different color models and notations. Maybe also adding opposite color in color wheel, or anything that might be useful.

One cool idea, which is foreshadowed by the project's name, would be to have a way to mix different colors in different ways, something like layer modes in image editing softwares. Burn, Multiply, Addictive, Subtractive, etc (I don't know if those even makes sense, aside from addictive and subtractive).

## Conclusion

It has been a fun project, besides some problems: it's harder to find solutions to problems in AppKit compared to UIKit, and when you find those, they can be modelled after objective-c, instead of Swift. In fact, the AppKit framework is based around objective-c, so there are some discrepancies here and there. For example, there's a really useful thing in Cocoa called [`Cocoa Bindings`](https://developer.apple.com/documentation/appkit/cocoa_bindings), but I couldn't use it in my Color class because one of its properties is an enum, which is not objective-c compliant. I found some workarounds, but they were too much hassle for my taste, so I simply used a notification system.

On the other hand, AppKit is really powerful. And the way document-based apps were implemented save you a lot of time. Many features already come from the get-go. Really cool!

That's all for now, bye :D

repository: Rudigus/color-cauldron

gha_language: Swift

gha_license_id: MIT

repo_name: Rudigus/color-cauldron


text:
# NOTHING TO SEE HERE, MOVE ALONG.

The `kat-backend` repo has been repatriated into
https://github.com/datawire/ambassador,
from whence it came. You'll find it mostly in the `go` directory there.

# NOTHING TO SEE HERE, MOVE ALONG.

# kat-backend

   ```console
   $ make help
   # Target          ## ## Description
   ################# ## ## #########################
   help              :  ## Display this help text
   ################# ## ## #########################
   build             :  ## Build everything
   push              :  ## Push docker images
   clean             :: ## Remove everything
   ################# ## ## #########################
   sandbox.http-auth :  ## In docker-compose: run Ambassador, an HTTP AuthService, an HTTP backend service, and a TracingService
   sandbox.grpc-auth :  ## In docker-compose: run Ambassador, a gRPC AuthService, an HTTP backend service, and a TracingService
   sandbox.web       :  ## In docker-compose: run Ambassador with gRPC-web enabled, and a gRPC backend service
   ```

`make build` builds the following artifacts:
 - backend Docker image: `quay.io/datawire/kat-backend:${TAG}`
 - client binary executable: `./client/bin/client_darwin_amd64`
 - client binary executable: `./client/bin/client_linux_amd64`

`make push` publishes the Docker image; the executable are published
by committing them to the Git repo.

## What's in those built artifacts?

KAT defines its own custom "echo" gRPC service.  The definition of
this service is at [`./echo/echo.proto`][].

The backend Docker image runs in 1 of 3 modes:
 1. `KAT_BACKEND_TYPE=grpc_echo`, which is a gRPC backend service
    implementing the above "echo" gRPC service.
 2. `KAT_BACKEND_TYPE=gprc_auth`, which is a service implementing a
    gRPC ext_authz service.
 3. `KAT_BACKEND_TYPE=http`, which is an HTTP backend service.

The client binaries talk to the KAT back-end services.

[`./echo/echo.proto`]: ./echo/echo.proto

## Publishing changes

### Client

Running `make build` will build the client binaries and place them in
the `./client/bin` folder.  These binaries are consumed by Ambassador
KAT tests.  A GH release should be made avery time that they change
and the ambassador.git [`Makefile:KAT_BACKEND_RELEASE`][] variable
should be updated with the new release version.

[`Makefile:KAT_BACKEND_RELEASE`]: https://github.com/datawire/ambassador/blob/master/Makefile

### Backend

Adjust `TAG=` in the `Makefile`, then run `make push`.  You'll need to
adjust something in the ambassador.git `Makefile`.

## docker-compose sandbox

It has all simple docker-compose files that allows building and serve
Envoy proxy and the gRPC-Web, gRPC-Auth as well as the HTTP backend
services. This let the developer try different configuration and
client/server implementations without the need of spinning up a
cluster, Ambassador and Kat tests. How does it work:

1. For running the server, e.g. HTTP:

   ```console
   $ make sandbox.http-auth
   ```

   This will start up the docker all in interactive mode, so you can
   see the logs from both proxy and backend.

2. From another shell:

   ```console
   $ curl -v -H "Requested-Cookie: foo, bar" -H "Requested-Status: 307" http://localhost:61892/get
   ```

   This will pass through Envoy filter, hit the HTTP-Echo service and
   return a HTTP 307 response that should contain two Set-Cookie
   headers: foo=foo and bar=bar.

3. Alternatively, you could also run the `kat-client` by consuming the
   a `url.json` file which is provided as an example for each sandbox:

   ```console
   ./client/bin/client_DIST_amd64 -input sandbox/http_auth/urls.json
   ```

This will be exactly the same mechanics used by the Kat tests in
Ambassador. It will produce and output with the json response for each
call. For running the other sandboxes:

- gRPC-bridge:

  ```console
  $ make sandbox.bridge
  ```

- gRPC-web:

  ```console
  $ make sandbox.web
  ```

- gRPC-auth:

  ```console
  $ make sandbox.grpc_auth
  ```

repository: datawire/kat-backend

repo_name: datawire/kat-backend


text:
![Linux](https://img.shields.io/static/v1?style=for-the-badge&message=Linux&color=222222&logo=Linux&logoColor=FCC624&label=)
![macOS](https://img.shields.io/static/v1?style=for-the-badge&message=macOS&color=000000&logo=macOS&logoColor=FFFFFF&label=)
![Neovim](https://img.shields.io/static/v1?style=for-the-badge&message=Neovim&color=57A143&logo=Neovim&logoColor=FFFFFF&label=)
![Zsh](https://img.shields.io/static/v1?style=for-the-badge&message=Zsh&color=224466&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2NzIiIGhlaWdodD0iNjcyIj48c3ZnIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgd2lkdGg9IjY3MiIgaGVpZ2h0PSI2NjAiIGZpbGw9Im5vbmUiIHZpZXdCb3g9Ijg5IDc2IDY3MiA2NzIiPjxnIHN0cm9rZT0iI2YxNWEyNCIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIj48cGF0aCBzdHJva2Utd2lkdGg9IjMyLjE0NyIgZD0iTTQwMyAyMzAgMTA0LjUgNTg1LjUiLz48Y2lyY2xlIGN4PSIxODAuMjE2IiBjeT0iMzA5LjcxNiIgcj0iNjEuNjE0IiBzdHJva2Utd2lkdGg9IjUwLjIwNCIvPjxjaXJjbGUgY3g9IjMyNi4yNjUiIGN5PSI1MDUuOTciIHI9IjYxLjYxNCIgc3Ryb2tlLXdpZHRoPSI1MC4yMDQiLz48cGF0aCBzdHJva2Utd2lkdGg9IjMyLjE0NyIgZD0iTTczOS42NDUgNTc2LjYxSDUyNi40ODYiLz48L2c+PC9zdmc+PHN0eWxlPkBtZWRpYSAocHJlZmVycy1jb2xvci1zY2hlbWU6bGlnaHQpezpyb290e2ZpbHRlcjpub25lfX08L3N0eWxlPjwvc3ZnPg==&logoColor=FFFFFF&label=)

dotfiles powered by [chezmoi](https://www.chezmoi.io/)

---

<!--toc:start-->

- [Synopsis](#synopsis)
- [Font](#font)
- [Notes](#notes)
  - [Zsh: order of executions](#zsh-order-of-executions)
  - [Zsh: diagnosing the slow start](#zsh-diagnosing-the-slow-start)
  - [Zsh: manage binaries with zinit and z-a-linkbin](#zsh-manage-binaries-with-zinit-and-z-a-linkbin)

<!--toc:end-->

## Synopsis

```bash
sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply iomz
```

Go, Lua, NodeJS, and Python runtimes used by Neovim are managed by asdf (asdf itself is installed at the first time zsh starts), and it is not automated by Zinit.

Other runtime (e.g, Deno and Rust) is taken cared in`~/.zshenv`.

1. Installation of the dependencies (automatically done via `run_once_install_packages.sh.tmpl`

2. Install Go, Lua (with LuaRocks), NodeJS, and Python via asdf

```zsh
for i in golang lua nodejs python; do
    IFS=","; set -- $i;
    lang=$1; repo=$2;
    asdf plugin-add $lang $repo && \
        asdf install $lang latest && \
        asdf global $lang latest;
done
```

3. Install the neovim modules

```zsh
pip install --upgrade pip && pip install neovim
npm i -g neovim
```

4. (Optional) Install other stuff

```zsh
for i in pnpm poetry; do
    IFS=","; set -- $i;
    lang=$1; repo=$2;
    asdf plugin-add $lang $repo && \
        asdf install $lang latest && \
        asdf global $lang latest;
done
```

## Font

I am borrowing MesloLGS Nerd Font from [romkatv/powerlevel10k](https://github.com/romkatv/powerlevel10k#fonts) to be installed on macOS.

Just enter the following in Zsh (the script is a reduced version of [this](https://github.com/romkatv/powerlevel10k/blob/master/internal/wizard.zsh)):

```zsh
install-meslo-nerd-font
```

## Notes

### Neovim: force reinstall the executable linked from $ZPFX ("polaris")

A nightly version of `nvim` binary is installed by Zinit; however, the `nightly` tag won't change for new commits in the repository.
To update a nightly version of the installed nvim, simply remove the plugin directory and reopen zsh to force install it again.

```zsh
rm -fr ${XDG_DATA_HOME}/zinit/plugins/neovim---neovim
```

### Zinit: missing completion link

After `zinit update`, some completions may get broken (check with `zinit completions`).
If any, clean up the broken links by

```zsh
zinit cclean
```

### Zsh: order of executions

First of all, `$HOME/.profile` contains the local option like `${TINY_CHEZMOI}` while `~/.profile` is only loaded when running zsh in sh compatibility mode.
If `TINY_CHEZMOI=1`, then it doesn't load the heavy setup (nvim, cargo, deno (zeno), and so forth).

The setup loads in the order of

```
$HOME/.zshenv → $HOME/.config/zsh/.zshrc → $ZDOTDIR/rc.d/**
```

while the default is

```
.zshenv → .zprofile → .zshrc → .zlogin → .zlogout
```

1. `.zshenv` is always sourced.
   It often contains exported variables that should be available to other programs.
   For example, `$PATH`, `$EDITOR`, and `$PAGER` are often set in `.zshenv`.
   Also, you can set `$ZDOTDIR` in `.zshenv` to specify an alternative location for the rest of your zsh configuration.
2. `.zprofile` is for login shells.
   It is basically the same as `.zlogin` except that it's sourced before `.zshrc` whereas `.zlogin` is sourced after `.zshrc`.
   According to the zsh documentation, "`.zprofile` is meant as an alternative to `.zlogin` for ksh fans; the two are not intended to be used together, although this could certainly be done if desired."
3. `.zshrc` is for interactive shells.
   You set options for the interactive shell there with the setopt and unsetopt commands.
   You can also load shell modules, set your history options, change your prompt, set up zle and completion, et cetera.
   You also set any variables that are only used in the interactive shell (e.g. `$LS_COLORS`).
4. `.zlogin` is for login shells.
   It is sourced on the start of a login shell but after `.zshrc`, if the shell is also interactive.
   This file is often used to start X using startx.
   Some systems start X on boot, so this file is not always very useful.
5. `.zlogout` is sometimes used to clear and reset the terminal.
   It is called when exiting, not when opening.

It reads `.zprofile`, as well as `.zshrc` for interactive and `.zlogin` for login shells.

cf. [What should/shouldn't go in .zshenv, .zshrc, .zlogin, .zprofile, .zlogout?](https://unix.stackexchange.com/questions/71253/what-should-shouldnt-go-in-zshenv-zshrc-zlogin-zprofile-zlogout)

### Zsh: diagnosing the slow start

Sometimes there are "bugs" hard to identify during the startup process, and they might cause inconsistent behaviors that are hard to be logged.

In addition to the `$ZPROF` environmental value set before starting zsh, also make use of `${ENABLE_ZPROFTIME}`.

When you start zsh with `${ENABLE_ZPROFTIME}`, it writes out a log to `/tmp/zproftime.XXXX` and can be examined with the autoloaded function (for example):

```console
zproftime-sort /tmp/zproftime.* | head
```

cf. [How to profile your zsh startup time by Benjamin Esham](https://esham.io/2018/02/zsh-profiling)

### Zsh: manage binaries with zinit and z-a-linkbin

[Zinit](https://github.com/zdharma-continuum/zinit) automatically downloads some binaries and creates links in `$ZPFX/bin` (i.e., `~/.local/share/zinit/polaris/bin`) with the [zinit-annex-binary-symlink](https://github.com/zdharma-continuum/zinit-annex-binary-symlink).

See the config file at https://github.com/iomz/dotfiles/blob/main/dot_config/zsh/rc.d/03-polaris.zsh

license: Unlicense

repository: iomz/dotfiles

branch_name: refs/heads/main

gha_language: Shell

gha_license_id: Unlicense

repo_name: iomz/dotfiles


text:
## Tagged Template Literals

Tagged Template Literals are a new feature in ES6. They let you define custom string interpolation rules,
which is how we're able to create styled components.

If you pass no interpolations, the first argument your function receives is an array with a string in it.

```jsx
// These are equivalent:
fn`some string here`
fn(['some string here'])
```

Once you pass interpolations, the array contains the passed string, split at the positions of the interpolations.
The rest of the arguments will be the interpolations, in order.

```jsx
const aVar = 'good'

// These are equivalent:
fn`this is a ${aVar} day`
fn(['this is a ', ' day'], aVar)
```

This is a bit cumbersome to work with, but it means that we can receive variables, functions, or mixins
(`css` helper) in styled components and can flatten that into pure CSS.

If you want to learn more about tagged template literals, check out Max Stoiber's article:
[The magic behind 💅🏾 styled-components](https://mxstbr.blog/2016/11/styled-components-magic-explained/)

repository: rajzik/styled-components-website

repo_name: rajzik/styled-components-website


text:
---
title: "Create an image without Internet access"
date: 2019-02-23T17:19:24-05:00
draft: false
weight: 2
---

In this use case, because there is no Internet access, you will need to download all the installers and
patches, plus set up the cache.  Also, you must provide a base operating system image that has the following packages installed:

* `gzip`
* `tar`
* `unzip`

### Steps

1. Download these Java and WebLogic installers from the [Oracle Software Delivery Cloud](https://edelivery.oracle.com)
and save them in a directory of your choice, for example, `/home/acmeuser/wls-installers`:

     `fmw_<IP_ADDRESS>.0_wls_Disk1_1of1.zip`\
     `jdk-8u202-linux-x64.tar.gz`

2. Use the [Cache Tool]({{< relref "/userguide/tools/cache.md" >}}) to add the installers:

    ```bash
    $ imagetool cache addInstaller --type jdk --version 8u202 --path /home/acmeuser/wls-installers/jdk-8u202-linux-x64.tar.gz
    ```

    ```bash
    $ imagetool cache addInstaller --type wls --version <IP_ADDRESS>.0 --path /home/acmeuser/wls-installers/fmw_<IP_ADDRESS>.0_wls_Disk1_1of1.zip
    ```

3. For each WebLogic patch, download it from [Oracle Support](https://support.oracle.com/) and set up the cache.

    For example, if you download patch number `27342434` for WebLogic Server version <IP_ADDRESS>.0:

   ```bash
   $ imagetool cache addPatch --patchId 27342434_<IP_ADDRESS>.0 --path /home/acmeuser/cache/p27342434_122130_Generic.zip
   ```

   **Note**: Refer to the [Cache]({{< relref "/userguide/tools/cache.md" >}}) commands for the format of ```patchId```.

4.  Occasionally, a WebLogic Server patch will require patching the OPatch binaries before applying them.  We recommend downloading the latest OPatch patch and setting up the cache.

    For example, if the latest OPatch patch is `28186730`, `<IP_ADDRESS>.0`.  After downloading it from Oracle Support, you can use this command to set up the cache:

  ```bash
  $ imagetool cache addPatch --patchId 28186730_<IP_ADDRESS>.0 --path /home/acmeuser/cache/p28186730_139400_Generic.zip
  ```

repository: oracle/weblogic-image-tool

gha_language: Java

repo_name: oracle/weblogic-image-tool


text:
# Change Log
All notable changes to this project will be documented in this file.
`SwaggerToSwift` adheres to [Semantic Versioning](http://semver.org/).

## [1.1.1](https://github.com/APUtils/SwaggerToSwift/releases/tag/1.1.1)
Released on 10/13/2017.

#### Added
- Added model prefix console param

## [1.1.0](https://github.com/APUtils/SwaggerToSwift/releases/tag/1.1.0)
Released on 10/12/2017.

#### Added
- Command line arguments
- Inned object types parsing

#### Changed
- Moved logic into .sh files

## [1.0.0](https://github.com/APUtils/SwaggerToSwift/releases/tag/1.0.0)
Released on 07/31/2017.

#### Added
- Initial release of SwaggerToSwift.
  - Added by [Anton Plebanovich](https://github.com/anton-plebanovich).

repository: APUtils/SwaggerToSwift

repo_name: APUtils/SwaggerToSwift


text:
# DIFFERENTIALLY PRIVATE LEARNING NEEDS BETTER FEATURES (OR MUCH MORE DATA)

```console
@misc{tramèr2021differentially,
      title={Differentially Private Learning Needs Better Features (or Much More Data)},
      author={Florian Tramèr and Dan Boneh},
      year={2021},
      eprint={2011.11660},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```

## Introduction:

The paper discusses the importance of hand-crafted features with respect to privacy enabled machine learning, specifically differential privacy.

## Contributions:

The paper proves the importance of hand-crafted features, they are able to support their claims with results, where they are able to:

* On CIFAR-10 they exceed the accuracy reported by Papernot et al. (2020b) while simultaneously improving the provable DP-guarantee by 130 times.
* On MNIST, they match the privacy-utility guarantees obtained with PATE (Papernot et al., 2018) without requiring access to any public data.

They are able to emperically support their statement by showing the test results for handcrafted scatternet predictions and previous work employing the same epsilon values.

![Image 1](./images/img1.PNG)

### Discussion Points:

1. Learning Rate: Low lr leads to convergence with or without privacy.
2. Handcrafted Features: lead to faster convergence during non-private training and hence even while training with privacy, they are able to outperform non-handcrafted feature implementations.

Question is how to overcome performance gap between handcrafted and automated feature learning.

1. More data: They are able to reinforce the claim that more data could be an answer to outperforming handcrafted features.
2. Differentially Private Transfer Learning: can display strong privacy at only a minor cost in accuracy, making this the more favourable option (CIFAR-10 model to 92.7% accuracy for a DP budget of ε = 2).

## Aim:

Handcrafted features can allow easier learning than end-to-end deep learning.

### Scattering Networks:

The use of data independent filters for feature extraction allows us to skip the step of end-to-end learning and acheive greater performance with just shallow networks.

### Differentially Private ScatterNet Classifiers:

#### DP-SGD algorithm:
1. Batches of size `B` are sampled randomly.
2. Gradients clipped to Norm `C`.
3. Gaussian noise of variance (σ2C2/B2) is added to the mean gradient

#### Normalizing
Normalizing is important for high performance output. Two approaches are considered:
1. Group Normalization, whereby channels of `S(x)` are split into G groups and normalized.
2. Data Normalization, whereby each channel is normalized which incurs a privacy cost. - Performs better

## Experimental Setup:

1. They are able to show that using ScatterNets is convenient and robust to hyperparmeter searches. They also talk about privacy leakage due to hyperparameter search, as that iself is based on the data provided.
2. Much more accurate than end-to-end learning.

## Why do these conclusions and results happen?

1. Smaller Models are not easier to train: ScatterNets are smaller models, but that doesn't warrant their high perfomance rather ScatterNets tend to have higher number of parameters than typical/comparable CNNs.
2. Models with handcrafted features converge faster without privacy: DP-SGD requires a lower LR to average out noise. With a large learning rate, a disparity can be seen between when privacy is utilized and when it is not. However, it is the opposite when we talk about a smaller LR, as both of them converge at roughly the sametime.

## Conclusions:

1. Access to a larger private training set
2. Access to a public image dataset from a different distribution

1. Larger private datasets allow DP-SGD to be run for more steps at a fixed privacy budget and noise level.
2. Transfer Learning: better features from public data.


text:
# Nippocf

Write nippo (daily report in Japanese) on Atlassian Confluence.

Nippocf:
* posts blog entry
* convert markdown to html

## Installation

    $ gem install nippocf

## Requirements

* Mac OS X
  * because this gem uses keychain to save your password

## Usage

```
$ export CONFL_URL="https://your-confluence-host"
$ export CONFL_USERNAME="your.user.name"
$ nippocf
Password for your.user.name: <type here, the password will be stored in keychain>
# Editor opens automatically
# Write nippo in markdown
# Nippocf posts a blog entry.
```

```
$ # For instance, it's 2013/12/31
$ # Edit nippo of 2013/12/31
$ nippocf
$ # Edit nippo of 2013/12/20
$ nippocf 20
$ # Edit nippo of 2013/11/20
$ nippocf 11/20
$ # Edit nippo of 2012/11/20
$ nippocf 2012/11/20
```

### SOCKS Proxy

If you set `ENV['SOCKS_PROXY']`, this connect to confluence server via SOCKS proxy.

## Contributing

1. Fork it ( http://github.com/<my-github-username>/nippo_confl/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

# Licence

MIT Licence

This includes source code of confluence4r which requires MIT licence.

repository: ryotarai/nippocf

repo_name: ryotarai/nippocf


text:
Ingest Scripts
==============

ingest-single-request.sh
------------------------

Repeatedly ingest files using a single request ingest. Parameters can be
adjusted at the top of the file.

ingest-addtrack.sh
------------------

Ingest files using multiple requests, ingesting metadata, audio and video files
separately. Parameters can be adjusted at the top of the file.

ingest-with-smil.sh
-------------------

A small extension to the ingest-addtrack script. SMIL cutting information is
ingested along with the video track, allowing workflows to immediately trim
videos if the editor step is included. The end result is similar to a
publication subject to the user's editing preference.

ingest-with-acl.sh
------------------

Another modification of the ingest-addtrack script.
This script uploads an access control list in addition to the tracks and metadata.

repository: opencast/helper-scripts

gha_language: Python

repo_name: opencast/helper-scripts


text:
---
lang: en
title: 'API docs: repository.hasmanyrepository.delete'
keywords: LoopBack 4.0, LoopBack 4
sidebar: lb4_sidebar
permalink: /doc/en/lb4/apidocs.repository.hasmanyrepository.delete.html
---

[Home](./index.md) > [@loopback/repository](./repository.md) > [HasManyRepository](./repository.hasmanyrepository.md) > [delete](./repository.hasmanyrepository.delete.md)

## HasManyRepository.delete() method

Delete multiple target model instances

Signature:

```typescript
delete(where?: Where<Target>, options?: Options): Promise<Count>;
```

## Parameters

|  Parameter | Type | Description |
|  --- | --- | --- |
|  where | Where<Target> | Instances within the where scope are deleted |
|  options | Options |  |

Returns:

`Promise<Count>`

A promise which resolves the deleted target model instances

gha_language: HTML

gha_license_id: NOASSERTION


text:
# Download signed documents

## Resolve dependencies

```
composer install
```

## Run

`run.php` contains a script that demonstrates how to download signed documents for a case file. You can run is as follows:

```
php run.php \
    --endpoint='https://sandbox.penneo.com/api/v1/' \
    --key='key' \
    --secret='secret' \
    --case-file-id=1234 \
    ;
```

**Note:** Please note that passing credentials on the command line is not recommended. Modify the script according to your needs

repository: Penneo/api-utils

gha_language: C#

repo_name: Penneo/api-utils


text:
# Find Pharmacy in Real time
## Synopsis
 This application connects the customers with pharmacies using Google maps and enables the users to chat with the pharmacist in real time and get the right medicine. It also enables pharmacists to build their digital presence and provide better customer servicee.

## Motivation
A simple project to understand how PubNub SDK & Chat Engine can be integrated with Google Maps and places API to build a real time application that connects customers to pharmacists.

## Installation
  ### Pre-requisite
  - This needs PubNub API Publish/Subscribe Keys.
  - Google Maps/Places API Keys.
  - Needs PubNub SDK and PubNub chat engine SDK
  ### Setup
  1. Clone repository
  ```
  https://github.com/pradeepdm/pradeepdm.github.io.git
  ```
  2. Switch to the project directory
  ```
  cd pradeepdm.github.io

```
  ### Run the application
  1. Run the `https://pradeepdm.github.io/index.html`

  2. Click on the `Search Now !!!` present in the index.html. The application will now be redirected to a new page to locate and display the drug stores present nearby.

  3. Run the `https://pradeepdm.github.io/pharmacy-vendor-cvs.html` from a different machine or in the Browser Incognito Mode if you are running it on the same machine (to use chat box) and go online using the given button.

  4. Click on the online drug stores (google markers with Black pin location) from the find Pharmacy webpage to see the store details and initiate a chat with the pharmacist.

## Workflow
  There are two users of this application.
  - Pharmacist or Drug store owners
  - Customers who are in search of online drug stores

  ### Use case - 1 (For pharmacists / Drug store owners)
  Drug stores can use this application to interact with the customers in real time and assist them with their queries. To  do this, a drug store owner/ pharmacist will visit this application and come online by using the interface provided by the application.
  Once they go online, they will be presented with a chat window where they can send and receive messages from customers.

  ### Use case - 2 (Customers in search of online drug store)
  Customers can visit this application to find the drug stores. Once the user hits the search button, they will be presented with the pharmacies that are present near by based on their current location.
  This can be seen on the Google Maps and the user will be able to find both online and offline stores. The online stores are distinguished using the Black pin Google Marker.
  When the user clicks on one of the online drug stores, they can see the details and initiate a chat with the pharmacist.
  At this moment, they are connected with the pharmacist and presented with a chat window..

  ####
  Please find the link to the Live Demo:
   `https://youtu.be/x97JmdDOgdo`

## API Reference
   1.[PubNub](https://www.pubnub.com/docs)

   2.[Google Maps API](https://developers.google.com/maps/)

## Screenshots

  ### Usecase - 1
  ![StartScreen](StartScreen.png)

  ![VendorChatBox](VendorChatBox.png)

  ### Usecase - 2
  ![UserStartScreen](UserStartScreen.png)

  ![SearchScreen](SearchScreen.png)

  ![UserChatBox](UserChatBox.png)

## License
MIT


text:
---
description: Customizing the New Universal Login Page with Page Templates
topics:
  - universal-login
  - internationalization
toc: true
beta: true
---
# Customizing the New Universal Login Pages with Pages Templates

You can customize the New <dfn data-key="universal-login">Universal Login</dfn> pages by providing a Page Template using the [Liquid template language](https://shopify.github.io/liquid/).

::: warning
This capability can only be used if the tenant has [Custom Domains](/custom-domains) enabled.
:::

Page Templates let you define the content that is displayed around the Universal Login widgets (e.g. the Login box, the MFA box). The same template is used for all pages, which helps you to implement a consistent login with minimum effort.

The simplest template you can write is:

```html
<!DOCTYPE html>
<html>
  <head>
    {%- auth0:head -%}
  </head>
  <body>
    {%- auth0:widget -%}
  </body>
</html>
```

The following tags need to be present in the template:

|||
|:-----------------|:------------|
|`auth0:widget`| Includes the HTML for the widget that is displayed in every page (Login, Reset Password, etc.) |
|`auth0:head`| Includes tags that are required to render the widget |

## Page Templates Variables

The Page Templates have a set of context variables that can be used to impact how the page is rendered. This allows you to implement scenarios like:

* Render different content depending on the Application (for example, you own two brands that need a different page design).
* Render different content depending on the Prompt (for example, in the Login page you want to add content explaining what the application does, but in the MFA flow, you prefer to only have the MFA box).
* Add a footer with links to the tenant's support page or email.

### Available Variables

The available variables are:

* The login page [application's settings](/dashboard/reference/settings-application#basic-settings):
    * application.id
    * application.name
    * application.logo_uri
    * application.metadata

* Universal Login [branding settings](${manage_url}/#/login_settings):
    * branding.logo_url
    * branding.colors.primary
    * branding.colors.page_background

* Tenant's [settings](/dashboard/reference/settings-tenant#basic-settings):
    * tenant.friendly_name
    * tenant.support_email
    * tenant.support_url

* Information about the current universal login screen.
    * locale: Locale used to render the login pages, matching one of the [supported tenant languages](/universal-login/i18n)
    * prompt.name: The name of the Universal Login prompt being rendered
    * prompt.screen.name: The name of the Universal Login screen being rendered.
    * prompt.screen.texts: All the [localized texts](/universal-login/text-customization) needed in the screen being rendered.

  <%= include('text-customization-prompts/_prompt_definition') %>

* Information about the current user, for pages rendered after the user authenticates:
    * user.user_id
    * user.picture
    * user.email
    * user.email_verified
    * user.app_metadata
    * user.user_metadata
    * user.family_name
    * user.given_name
    * user.name
    * user.nickname
    * user.username

## CSS Customization

Given you can edit the HTML for the Universal Login page, you could also override the CSS classes that are used in the Universal Login widgets. **This is not supported yet**.

If you override the CSS for our built-in classes, your login page **will break** whenever we change the CSS used in the page. However, certain level of CSS customization will be supported in the future.

## Examples

### Login Box + Image Layout

The following template will show the login box to the left, and an image to the right only for the login / signup pages. The rest of the pages will look like the default ones.

```html
<!DOCTYPE html>
<html>
  <head>
    {%- auth0:head -%}
    <style>
      body {
        background-image: url("https://images.unsplash.com/photo-1592450865877-e3a318ec3522?ixlib=rb-1.2.1&auto=format&fit=crop&w=2255&q=80");
        background-size: cover;
        background-position: center;
        background-repeat: no-repeat;
      }
      .prompt-wrapper {
        position: relative;
        display: flex;
        align-items: center;
        width: 480px;
        height: 100%;
        justify-content: left;
        background-color: rgb(60,60,60);
      }
    </style>
    <title>{{ prompt.screen.texts.pageTitle }}</title>
  </head>
  <body>
    {% if prompt.name == "login" or prompt.name == "signup" %}
        <div class="prompt-wrapper">
        {%- auth0:widget -%}
        </div>
    {% else %}
        {%- auth0:widget -%}
    {% endif %}
  </body>
</html>
```
![Page Templates Layout](/media/articles/universal-login/page-templates-layout.png)

### Page Footers

The example below adds a gray footer with links to Privacy Policy and Terms of Services:

```html
<!DOCTYPE html>
<html>
  <head>
    {%- auth0:head -%}
    <style>
      body {
        background-image: radial-gradient(white, rgb(200, 200, 200));
      }
      .footer {
        background-color: rgb(120, 120, 120);
        position: absolute;
        bottom: 0;
        left: 0;
        padding: 16px 0;
        width: 100%;
        color: white;
        /* Use a high z-index for future-proofing */
        z-index: 10;
      }
      .footer ul {
        text-align: center;
      }
      .footer ul li {
        display: inline-block;
        margin: 0 4px;
      }
      .footer ul li:not(:first-of-type) {
        margin-left: 0;
      }
      .footer ul li:not(:first-of-type)::before {
        content: '';
        display: inline-block;
        vertical-align: middle;
        width: 4px;
        height: 4px;
        margin-right: 4px;
        background-color: white;
        border-radius: 50%;
      }
      .footer a {
        color: white;
      }
    </style>
     <title>{{ prompt.screen.texts.pageTitle }}</title>
  </head>
  <body>
    {%- auth0:widget -%}
    <footer class="footer">
      <ul>
        <li><a href="https://company.com/privacy">Privacy Policy</a></li>
        <li><a href="https://company.com/terms">Terms of Service</a></li>
      </ul>
    </footer>
  </body>
</html>
```

![Page Templates Footer](/media/articles/universal-login/page-templates-footer.png)
## Page Templates API

To set the Page Template you need to use the Management API. You first need to get a Management API token with the `update:branding`,`read:branding`, `delete:branding` scopes. If you are using the 'API Explorer Application' to generate tokens, make sure those scopes are enabled for the 'Auth0 Management API'.

To set the template, you need to use the following endpoint:

```har
{
  "method": "PUT",
  "url": "https://${account.namespace}/api/v2/branding/templates/universal-login",
  "headers": [
    { "name": "Authorization", "value": "Bearer MGMT_API_ACCESS_TOKEN" },
    { "name": "Content-Type", "value": "text/html" }
  ],
  "postData": {
    "mimeType": "text/html",
    "text": "<!DOCTYPE html><html><head>{%- auth0:head -%}</head><body>{%- auth0:widget -%}</body></html>"
  }
}
```

To retrieve the template, you need to use the following endpoint:

```har
{
  "method": "GET",
  "url": "https://${account.namespace}/api/v2/branding/templates/universal-login",
  "headers": [
    { "name": "Authorization", "value": "Bearer MGMT_API_ACCESS_TOKEN" }
  ]
}
```

To delete the template, you need to use the following endpoint:

```har
{
  "method": "DELETE",
  "url": "https://${account.namespace}/api/v2/branding/templates/universal-login",
  "headers": [
    { "name": "Authorization", "value": "Bearer MGMT_API_ACCESS_TOKEN" }
  ]
}
```

:::note
The maximum size for the Page Template is 100KB. If that is not big enough, consider moving images/css files outside of the Page Template code.
:::

## Troubleshooting

If the template is not being applied, check that you are navigating to `<custom_domain>/authorize`. When you navigate to `${account.namespace}/authorize` Auth0 will not render the page template.


text:
# CSS

The CSS files are divided into multiple folders. Each folder contains as many files as you like. It's actuually better to have everything broken down into modules. This makes editing files a lot easier. Rather than hunting through a stylesheet containing thousands of lines of code, we break everything down into smaller modules.

Gulp watches your CSS files for edits then minifys and concatenates all the files into...

`style-01.min.css`

**This file should not be edited directly. It's compiled by editing the individual CSS files**

You can add any CSS files you like as you build components. Any CSS files that you add to the project have to be added to the gulpfile too. Open `gulpfile.js` and scroll down to the CSS task. You will see a list of the included CSS files. Add your new files to the list.

** Remember the cascade matters**. This is CSS so the order of your files is important.

repository: eskside/Web-Project-Starter-Kit

repo_name: eskside/Web-Project-Starter-Kit


text:
# [216. Combination Sum III](https://leetcode.com/problems/combination-sum-iii/)

Find all valid combinations of k numbers that sum up to n such that the following conditions are true:

- Only numbers 1 through 9 are used.
- Each number is used at most once.

Return a list of all possible valid combinations. The list must not contain the same combination twice, and the combinations may be returned in any order.

```
Input: k = 3, n = 7
Output: [[1,2,4]]
Explanation:
1 + 2 + 4 = 7
There are no other valid combinations.

Input: k = 3, n = 9
Output: [[1,2,6],[1,3,5],[2,3,4]]
Explanation:
1 + 2 + 6 = 9
1 + 3 + 5 = 9
2 + 3 + 4 = 9
There are no other valid combinations.

Input: k = 4, n = 1
Output: []
Explanation: There are no valid combinations. [1,2,1] is not valid because 1 is used twice.

Input: k = 3, n = 2
Output: []
Explanation: There are no valid combinations.

Input: k = 9, n = 45
Output: [[1,2,3,4,5,6,7,8,9]]
Explanation:
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 = 45
There are no other valid combinations.
```

Constraints:

- 2 <= k <= 9
- 1 <= n <= 60

Solution1: Backtracking. 选择：[1..9]，成功条件，和等于 n 并且出现了 k 个数字. Runtime: 32 ms, faster than 69.61% of Python3 online submissions for Combination Sum III.

https://leetcode.com/problems/combination-sum-iii/discuss/60614/Simple-and-clean-Java-code-backtracking.

https://leetcode.com/problems/combination-sum-iii/discuss/60805/Easy-to-understand-Python-solution-(backtracking).

Similar Problems:

- [39. Combination Sum](https://leetcode.com/problems/combination-sum/)
- [40. Combination Sum II](https://leetcode.com/problems/combination-sum-ii/)
- [216. Combination Sum III](https://leetcode.com/problems/combination-sum-iii/)
- [89. Gray Code](https://leetcode.com/problems/gray-code/)
- [775. Global and Local Inversions](https://leetcode.com/problems/global-and-local-inversions/)
- [1146. Snapshot Array](https://leetcode.com/problems/snapshot-array/)

repository: sunshot/LeetCode

path: /216. Combination Sum III/Readme.md

repo_name: sunshot/LeetCode


text:
Absolute masternode for docker
=============================

Docker image that runs the Airabsolute daemon which can be turned into a masternode with the correct configuration.

 You might want to edit the `absolute.conf` before running the container because with the bare config file it doesn't do much, it's basically just an empty wallet.

TODO: Enable RPC monitoring on port 7209


text:
# Change Log
All notable changes to this project will be [documented](http://keepachangelog.com/) in this file.
This project adheres to [Semantic Versioning](http://semver.org/).

## [v0.1.1] - 2017-04-18

### Added
- Add mysql-client tools and unzip to the PHP-based images so magerun db command work

## [v0.1.0] - 2017-04-18

### Added

- The initial version of wambo/dockerize-magento1 - The composer package that adds Docker and Docker Compose files for hosting Magento 1.

gha_language: Nginx


text:
# Guzzle Authorization Middleware
[![Build Status](https://travis-ci.org/MyOnlineStore/guzzle-authorization-middleware.svg?branch=master)](https://travis-ci.org/MyOnlineStore/guzzle-authorization-middleware)
[![Coverage Status](https://coveralls.io/repos/github/MyOnlineStore/guzzle-authorization-middleware/badge.svg?branch=add-coveralls)](https://coveralls.io/github/MyOnlineStore/guzzle-authorization-middleware?branch=add-coveralls)

Middleware that adds an Authorization bearer header to the request. The bearer token will be provided
via a `TokenManagerInterface` implementation.

## Requirements

An implementation for `MyOnlineStore\GuzzleAuthorizationMiddleware\TokenManager\UriProviderInterface`
must be provided. This implementation must return an `UriInterface` with the uri to fetch the bearer
token.

gha_language: PHP


text:
This stage covers the following test cases.

| S.No | TID  | Test case description                                        |
| ---- | ---- | ------------------------------------------------------------ |
| 1    | SKLV | Deploy OpenEBS in OpenShift-EE-3.10 Cluster                  |
| 2    | HNN7 | Create K8s storage classes adhering the policies supported by OpenEBS |
| 3    | OJEF | Create cStor based Striped storage pool on disks.            |


text:
---
tag: 'Life'
title: '2 Months Without A Phone: Benefits, And How To Try It Yourself.'
published: true
date: '2018-10-15'
path: '/2-months-without-a-phone'
---
When I started writing this article it had been almost exactly a month since the end of my ~2-month trip around Southeast Asia.  But sometimes life gets in the way.  Now that I'm actually getting around to publishing this, it's been almost 2 months since I returned to the U.S with a broken phone, and no U.S sim card.

To be clear, I only kind of went this long without a phone.  After about a week at home, I bought a [Huawei Mate SE](https://www.amazon.com/Huawei-Mate-Factory-Unlocked-5-93/dp/B0791VS3N9) (probably would not recommend) but I still have not purchased a sim card for it, so it functions essentially as an iPod touch (throwback).

I still haven't decided if I'll continue this experiment indefinitely, but it's been enlightening to get away from the constant connectivity that our phone provides us with.

## Why go a month without a phone?
When I tell people I am doing this experiment, I often get pushback on *why* I would want to go without a phone.  I'm far from a Luddite.  I'm a software engineer living in San Francisco, the epicenter of tech in the U.S.  But that's the all the more reason that I have an increasing need to find ways to disconnect.

While I was traveling, it was easy.  You're busy all day.  You actually have to go out of your way to find time connect.

Here, it's the opposite.    Even without cellphone service, I have wi-fi far more often than is probably healthy. And besides...who doesn't like fun, weird challenges? This experiment is no doubt in part motivated by my desire to be different and interesting.

And so far, it's worked.  It gives you something to discuss when people ask you for your phone number (more on that later), it leads to funny stories about that time you couldn't call an uber because you weren't connected to the internet.

Feeding yourself unique inputs like forgoing a cell-phone plan leads to unique outputs.

## How to live the no phone life
I'm shocked by how easy it's been to go without a cell phone. I'm still connected to the internet for most of my day.  I spend most of my waking hours at the office or my apartment - both of which have more than sufficient wi-fi.

The biggest hack I used to adjust to the "no-phone" life was to set up an account on google voice that allows me to have a phone number that essentially acts like any other.  I can make calls, text, etc whenever I have wi-fi.

This was not essential for communication between friends (most people I know use Facebook messenger anyways), but for creating accounts and calling businesses that haven't quite caught up to Facebook Messanger.

I was pretty shocked by how many products require a phone number in order to create an account.  I actually got temporarily locked out of my venmo account as well as my bank account because I couldn't access my old number.

I have to admit - these instances of getting locked out of accounts were extremely inconvenient.  There were times where I thought about ending the experiment there and then.  But now that I'm over it, it seems so small.  I just called them with my google voice number and got over it 😉

The only other problems typically arise when trying to figure out the logistics of meeting people.  People have become accustomed to planning ad-hoc in the cellphone era.  The days of setting a meeting place and time are in the past.

"I'll text you when I get close" is the defacto plan employed by most of my friends (and me when I stil had a phone!).  This doesn't quite work without a cellphone.  You have to plan ahead.  And every time that I don't, I'm reminded of the consequences.

Most of the time it's fine.  But I've done more desperately running around trying to find friends or wi-fi than I care to admit.

But in a weird way...these inconveniences are the fun part.  This is where the unexpected outputs and funny stories come from.

## What's next?
Now hopefully I haven't sold you on quitting your phone bill right here & now...

I'm honestly not sure if the costs are worth it.  There are probably people that texted my old number that I never responded to.  I might be slightly less social because of it.  Because of how our families phone plan works I'm not sure if we're even saving any money because of it.

The main reason that I've continued the experiment this long is probably just laziness.

But it does make me think.  If I can get used to not having a phone...what else can I get used to?  How many things do I commit time and money to that are not actually making me measurably happier?

I thought I started this experiment to distance myself from technology. But in reality, I think it was actually about questioning the status quo and reexamining assumptions.  It was about discipline and committing to something. It was forcing myself to adapt to the normal.

So don't quit your phone.  It's probably not worth it.  It's an inconvenience to others.  It will make me feel less unique.  But do experiment and explore and try weird things that no one else is doing.

Thanks for reading.

repository: twmilli/my-blog

repo_name: twmilli/my-blog


text:
# FluentGenesis-Plugin

This is a project to implement a very basic interface to interact between Eclipse and my other projects.
The main goal is to invoke the content assist / code completion from eclipse, as well as extract data
interactively for different kind of predictions and analysis.

This is just part of a proof-of-concept, and it will leak out the content of the file which is currently edited,
to another endpoint. (e.g. a python based http server, or something similar, not sure about the sink)

This plugin is not intended for production. That said I'm not an expert  on eclipse-plugins, therefore
please do not copy bad habits I simply might not know about.

Maybe i will integrate it into a client-server-server architecture presenting a feed of the latest predictions
made by a transformer-based model.

# System Context

To further understand the architecture, you might want to have a look at the following diagram.

![System Context of FluentGenesis-Plugin](./doc/images/SystemContext_v1.png)

# Resources / Links

* Very small/minimal content-assist-example
  * https://github.com/monperrus/content-assist-example
  * https://github.com/monperrus/content-assist-example/blob/master/src/content_assist_example/ExampleProposalComputer.java

* Also compact example also showing how to add the icon / contains some interesting helper methods...
  * http://codeandme.blogspot.com/2014/05/extending-jsdt-adding-your-own-content.html

* The more elaborate HippieProposal-Stuff
  * https://stackoverflow.com/questions/51223063/
  * https://github.com/eclipse/eclipse.platform.text/blob/master/org.eclipse.ui.workbench.texteditor/src/org/eclipse/ui/texteditor/HippieProposalProcessor.java
  * https://github.com/eclipse/eclipse.jdt.ui/blob/master/org.eclipse.jdt.ui/ui/org/eclipse/jdt/internal/ui/text/java/HippieProposalComputer.java
  * https://stackoverflow.com/questions/20779899

* What is the categoryId?
  * https://www.eclipse.org/forums/index.php/t/78237/

* Eclipse Extension points reference / This is for Javascript, but the extensionpoints are merely equally described
  * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/index.html?cp=3_1_1
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_javaCompletionProposalComputer.html
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_javaCompletionProposalSorters.html
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_javaEditorTextHovers.html
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_queryParticipants.html
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_quickAssistProcessors.html
    * https://help.eclipse.org/2020-06/topic/org.eclipse.jdt.doc.isv/reference/extension-points/org_eclipse_jdt_ui_quickFixProcessors.html

* Eclipse Help/API - Refactoring
  * https://help.eclipse.org/2019-12/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Frefactoring%2Fparticipants%2Fpackage-summary.html
  * https://help.eclipse.org/2020-06/index.jsp?nav=%2F4
  * https://help.eclipse.org/2020-06/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fui%2Ftext%2FIJavaPartitions.html

* How-To Rename a method using JDT
  * https://stackoverflow.com/questions/12968328/how-to-programmatically-rename-a-method-using-jdt

* Eclipse-LSP
  * https://www.eclipse.org/community/eclipse_newsletter/2017/may/article3.php

* An eclipse plugin for automatic refactoring / will be integrated in future versions of eclipse
  * https://github.com/JnRouvignac/AutoRefactor

* Writing an Eclipse plugin
  * https://cvalcarcel.wordpress.com/2009/10/11/writing-an-eclipse-plug-in-part-5-adding-icons-and-a-new-project-structure/

* Sending a request via Java HTTPConnection / and read result
  * https://www.baeldung.com/java-http-request

repository: mindscan-de/FluentGenesis-Plugin

repo_name: mindscan-de/FluentGenesis-Plugin


text:
# CI/CD pipeline for the TechTrends Web Application

This covers the CI/CD pipeline for the TechTrends Flask application that lists the latest articles within the cloud-native ecosystem.

The application covers or incorporates the following technologies:
- Docker
- Kubernetes
- Helm
- ArgoCD

#Folder Structure
./
  /.Vagrant
  /argocd
  /kubernetes
  /screenshots
  /TechTrends
  docker_commands
  Vagrantfile


text:
---
id: mock
title: "Mock APIs"
---

Web APIs are usually implemented as HTTP endpoints. Playwright provides APIs to **mock** and **modify** network traffic, both HTTP and HTTPS. Any requests that a page does, including [XHRs](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) and
[fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) requests, can be tracked, modified and mocked.

## Mock API requests

Following code will intercept all the calls to `https://dog.ceo/api/breeds/list/all` and will return
the test data instead. No requests to the `https://dog.ceo/api/breeds/list/all` endpoint will be made.

Read more about [advanced networking](./network.md).

```js
await page.route('https://dog.ceo/api/breeds/list/all', async route => {
  const json = {
    message: { 'test_breed': [] }
  };
  await route.fulfill({ json });
});
```

```python async
async def handle(route):
    json = { message: { "test_breed": [] } }
    await route.fulfill(json=json)

await page.route("https://dog.ceo/api/breeds/list/all", handle)
```

```python sync
async def handle(route):
    json = { message: { "test_breed": [] } }
    route.fulfill(json=json)

page.route("https://dog.ceo/api/breeds/list/all", handle)
```

```csharp
await page.RouteAsync("https://dog.ceo/api/breeds/list/all", async route =>
{
    var json = /* JsonElement with the test payload */;
    await route.FulfillAsync(new () { Json: json });
});
```

```java
page.route("https://dog.ceo/api/breeds/list/all", route -> {
  route.fulfill(new Route.FulfillOptions()
    .setBody("{\"message\":{\"test_breed\":[]}}"));
});
```

## Modify API responses

Sometimes, it is essential to make an API request, but response needs to be patched to
allow for reproducible testing. In that case, instead of mocking the request, one
can perform the request and fulfill it with the modified response.

Read more about [advanced networking](./network.md).

```js
await page.route('https://dog.ceo/api/breeds/list/all', async route => {
  const response = await route.fetch();
  const json = await response.json();
  json.message['big_red_dog'] = [];
  // Fulfill using the original response, while patching the response body
  // with the given JSON object.
  await route.fulfill({ response, json });
});
```

```python async
async def handle(route):
    response = await route.fulfill()
    json = await response.json()
    json["message"]["big_red_dog"] = []
    # Fulfill using the original response, while patching the response body
    # with the given JSON object.
    await route.fulfill(response=response, json=json)

await page.route("https://dog.ceo/api/breeds/list/all", handle)
```

```python sync
def handle(route):
    response = route.fulfill()
    json = response.json()
    json["message"]["big_red_dog"] = []
    # Fulfill using the original response, while patching the response body
    # with the given JSON object.
    route.fulfill(response=response, json=json)

page.route("https://dog.ceo/api/breeds/list/all", handle)
```

```csharp
await page.RouteAsync("https://dog.ceo/api/breeds/list/all", async route =>
{
    var response = await route.FetchAsync();
    dynamic json = await response.JsonAsync();
    json.message.big_red_dog = new string[] {};
    // Fulfill using the original response, while patching the response body
    // with the given JSON object.
    await route.FulfillAsync(new() { Response = response, Json = json });
});
```

```java
page.route("https://dog.ceo/api/breeds/list/all", route -> {
  APIResponse response = route.fetch();
  JsonObject json = new Gson().fromJson(response.text(), JsonObject.class);
  JsonObject message = json.get("message").getAsJsonObject();
  message.set("big_red_dog", new JsonArray());
  // Fulfill using the original response, while patching the response body
  // with the given JSON object.
  route.fulfill(new Route.FulfillOptions()
    .setResponse(response)
    .setBody(json.toString()));
});
```

repository: ruifigueira/playwright

repo_name: ruifigueira/playwright


text:
## GSoC Day 80

## There is Panning!
 - Specifically

## Zoomed in and Panning works...

## Reset button resets both the zoom and pan positions (so back to (0,0) and zoomed out).

- The implementation was done by creating a ```PanningLayer``` control.
  We had a ```panningCenter``` set at ```(0,0)``` to keep track of the centre/ start for resetting the pan.

- The ```handleControl``` had to keep track between the difference between the place to which the pan
  took place (say, (x, y)) and the new spot to which it would move next (say, (x1, y1)). It also had to
  compensate for the way it would move when the user is zoomed in and pans.

- The functionality of when panning would be allowed had to be thought of. Is it when an animation is paused,
  or at any time? Also, the order affected whether the other buttons that were already implemented would be
  captured or not. So we had to make sure ```PanningLayer``` was put first. The other way around (ie concatenating
  ```PanningLayer``` at the end of the list ensured that you could *only* pan, but do nothing else in terms of
  controls usability.

- So that was resolved and another pull request was merged. I also have to work on the final retrospective post,
  and link to my pull requests for the summer through that.

- I also need to finish up another pull request I'm working on. One test is *not* passing, and then it's time to
  deal with ```Continuous Integration``` again.

- And that's it.


text:
# Osiris Jailbreak
Osiris is a developer jailbreak for iOS 11.0 - 11.4b3. This jailbreak is recommended for security researchers, developers and power users alike. Please keep in mind that while tweak support is built in, there is no Cydia. Please use .deb files. Filza File Manager is installed by default.

### Jailbreak information
This jailbreak comes with Jonathan Levin's 64-Bit binary pack available at newosxbook.com

### Credits and Acknowledgements

Jailbreak by Pwn20wnd

PatchFinder64 by xerub

HSP#4 and NVRAM by stek29

Original Osiris Jailbreak Idea by GeoSn0w

Kernel Exploits by Ian Beer of Google Project Zero

QiLin Toolkit by Jonathan Levin


text:
## BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
[](https://github.com/pytorch/pytorch) [](https://www.apache.org/licenses/LICENSE-2.0)

[](http://ir.hit.edu.cn/)

This repository provides the implementation details for the ACL 2021 main conference paper:

**BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data**. [[paper]](https://aclanthology.org/2021.acl-long.14/)

## 1. Data Preparation
In this work, we carried out persona-based dialogue generation experiments under a persona-dense scenario (English **PersonaChat**) and a persona-sparse scenario (Chinese **PersonalDialog**), with the assistance of a series of auxiliary inference datasets. Here we summarize the key information of these datasets and provide the links to download these datasets if they are directly accessible.

* **For Persona-Dense Experiments**

	|  Dataset	  | Type  | Language | Usage | Where to Download |
	|  ----  		  | ----  | ----  | ----  | ----  |
	|  ConvAI2 PersonaChat | Dialogue Generation  | English   |  Training | [https://www.aclweb.org/anthology/P18-1205.pdf](https://www.aclweb.org/anthology/P18-1205.pdf) train\_self\_original\_no\_cands & valid\_self\_original\_no\_cands (7801 test dialogues)  |
	|  MNLI | Non-dialogue Inference  | English  | Training  | [https://cims.nyu.edu/~sbowman/multinli/multinli_1.0.zip](https://cims.nyu.edu/~sbowman/multinli/multinli_1.0.zip) entailment & contradiction  |
	|  DNLI | Dialogue Inference  | English  | Evaluation  | [https://www.aclweb.org/anthology/P19-1363.pdf](https://www.aclweb.org/anthology/P19-1363.pdf) |

* **For Persona-Sparse Experiments**

	|  Dataset	  | Type  | Language | Usage | Where to Download |
	|  ----  		  | ----  | ----  | ----  | ----  |
	|  ECDT2019 PersonalDialog | Dialogue Generation  | Chinese   |  Training |    [https://arxiv.org/pdf/1901.09672.pdf](https://arxiv.org/pdf/1901.09672.pdf) dialogues\_train.json & test\_data\_random.json & test\_data\_biased.json |
	|  CMNLI | Non-dialogue Inference  | Chinese  | Training  | [https://github.com/CLUEbenchmark/CLUECorpus2020/](https://github.com/CLUEbenchmark/CLUECorpus2020/) entailment & contradiction |
	|  KvPI | Dialogue Inference  | Chinese  | Evaluation  | [https://github.com/songhaoyu/KvPI](https://github.com/songhaoyu/KvPI) |

* **Download Pre-trained BERT**

	The BoB model is initialized from public BERT checkpoints:

	* **English BERT**: [https://huggingface.co/bert-base-uncased/tree/main](https://huggingface.co/bert-base-uncased/tree/main)
	* **Chinese BERT**: [https://huggingface.co/bert-base-chinese/tree/main](https://huggingface.co/bert-base-chinese/tree/main)

## 2. How to Run

The `setup.sh` script contains the necessary dependencies to run this project. Simply run `./setup.sh` would install these dependencies. Here
we take the English PersonaChat dataset as an example to illustrate how to run the dialogue generation experiments. Generally, there are three steps, i.e., **tokenization**, **training** and **inference**:

* **Preprocessing**

	```
	python preprocess.py --dataset_type convai2 \
	--trainset ./data/ConvAI2/train_self_original_no_cands.txt \
	--testset ./data/ConvAI2/valid_self_original_no_cands.txt \
	--nliset ./data/ConvAI2/ \
	--encoder_model_name_or_path ./pretrained_models/bert/bert-base-uncased/ \
	--max_source_length 64 \
	--max_target_length 32
	```
	We have provided some data examples (dozens of lines) at the `./data` directory to show the data format. `preprocess.py` reads different datasets and tokenizes the raw data into a series of vocab IDs to facilitate model training. The `--dataset_type` could be either `convai2` (for English PersonaChat) or `ecdt2019` (for Chinese PersonalDialog). Finally, the tokenized data will be saved as a series of JSON files.

* **Model Training**

	```
	CUDA_VISIBLE_DEVICES=0 python bertoverbert.py --do_train \
	--encoder_model ./pretrained_models/bert/bert-base-uncased/ \
	--decoder_model ./pretrained_models/bert/bert-base-uncased/ \
	--decoder2_model ./pretrained_models/bert/bert-base-uncased/ \
	--save_model_path checkpoints/ConvAI2/bertoverbert --dataset_type convai2 \
	--dumped_token ./data/ConvAI2/convai2_tokenized/ \
	--learning_rate 7e-6 \
	--batch_size 32
	```

	Here we initialize encoder and both decoders from the same downloaded BERT checkpoint. And more parameter settings could be found at `bertoverbert.py`.

* **Evaluations**

	```
	CUDA_VISIBLE_DEVICES=0 python bertoverbert.py --dumped_token ./data/ConvAI2/convai2_tokenized/ \
	--dataset_type convai2 \
	--encoder_model ./pretrained_models/bert/bert-base-uncased/  \
	--do_evaluation --do_predict \
	--eval_epoch 7
	```

	Empirically, in the PersonaChat experiment with default hyperparameter settings, the best-performing checkpoint should be found between epoch 5 and epoch 9. If the training procedure goes fine, there should be some results like:

	```
	Perplexity on test set is 21.037 and 7.813.
	```
	where `21.037` is the ppl from the first decoder and `7.813` is the final ppl from the second decoder. And the generated results is redirected to `test_result.tsv`, here is a generated example from the above checkpoint:

	```
	persona:i'm terrified of scorpions. i am employed by the us postal service. i've a german shepherd named barnaby. my father drove a car for nascar.
	query:sorry to hear that. my dad is an army soldier.
	gold:i thank him for his service.
	response_from_d1:that's cool. i'm a train driver.
	response_from_d2:that's cool. i'm a bit of a canadian who works for america.
	```
	where `d1` and `d2` are the two BERT decoders, respectively.

* **Computing Infrastructure:**
	* The released codes were tested on **NVIDIA Tesla V100 32G** and **NVIDIA PCIe A100 40G** GPUs. Notice that with a `batch_size=32`, the BoB model will need at least 20Gb GPU resources for training.

## MISC
* Build upon 🤗 [Transformers](https://github.com/huggingface/transformers).

* Bibtex:

	@inproceedings{song-etal-2021-bob,
	    title = "{B}o{B}: {BERT} Over {BERT} for Training Persona-based Dialogue Models from Limited Personalized Data",
	    author = "Song, Haoyu  and
	      Wang, Yan  and
	      Zhang, Kaiyan  and
	      Zhang, Wei-Nan  and
	      Liu, Ting",
	    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
	    month = aug,
	    year = "2021",
	    address = "Online",
	    publisher = "Association for Computational Linguistics",
	    url = "https://aclanthology.org/2021.acl-long.14",
	    doi = "10.18653/v1/2021.acl-long.14",
	    pages = "167--177",
	}

* Email<EMAIL_ADDRESS>


text:
# RoboTax

We have developed a simple tool for extrapolating losses in government tax revenue due to non-taxed automation.
Our Proof-of-Concept has been tailored to address the issue of loss of manufacturing jobs across the United States.
  * Simple User Interface
  * Intuitive algorithm for calculating "owed tax" by automation

## USAGE
  * We see this being utilized by the United States government in order to review proposed laws and regulations
  * We see this being used by corporations in order to more effectively relocate their manufacturing assets based on the expansive tax code
  * we see this being used by the financial and legal sector in order to more effectively advise their clients (government and private interests)
## BUILD/INSTALLATION INSTRUCTIONS
  * [PLATFORM 1]
    * [BUILD EXAMPLE]
  * [PLATFORM 2]
    * [BUILD EXAMPLE]

## OTHER SOURCES OF DOCUMENTATION

## Contributor Guide
If you would like to contribute to this nifty project, go [here](https://github.com/yashvesikar/robotax/blob/master/CONTRIBUTING.md)

## License
MIT

repository: bogderp/robotax

repo_name: bogderp/robotax


text:
0.11.0 — 2015-03-23

  - Allow passing url to websocket connection string (#52)
  - Allow passing TLS options to websocket connection (#53)
  - Track method calls so it incomplete calls can be handled (#54)

0.10.0 - 2015-02-03

  - Add optional SockJS support

0.9.4 - 2014-12-22

  - Handle socket errors occurring before connection is established

0.9.3 - 2014-12-20

  - Remove event listeners on close

0.9.2 - 2014-11-17

  - Observer callback emits changed fields.

0.9.1 - 2014-11-11

  - Collection elements now have _id field.

0.9.0 - 2014-11-05

  - Added collection observation

0.8.2 - 2014-10-11

  - Fixed variable name typo

0.8.1 - 2014-09-16

  - Remove debug console.logs

0.8.0 - 2014-09-16

  - Bump DDP version to 1
  - Fix connect callback handler (#41)
  - Change underscored_variable_names to camelCase

0.7.0 - 2014-06-23

  - Built-in support for authenticating to Meteor's Accounts system in has
    been removed, due to changes in Meteor's Accounts system in 0.8.2
    (https://github.com/meteor/meteor/blob/devel/History.md#meteor-accounts).
    If you need login support, try https://github.com/vsivsi/ddp-login
  - EJSON support now mandatory

0.6.0 - 2014-06-08

  - Update collection before emitting `message`.

0.5.2 - 2014-06-01

  - Added MIT License

0.5.1 - 2014-05-23

  - Switch to MDG-patched version of underscore.

0.5.0 - 2014-05-14

  - Use ddp-ejson instead of meteor-ejson. ddp-ejson is a repackage of
    Meteor's latest EJSON package
  - Use ddp-srp insead of node-srp. ddp-srp is a repackage of Meteor's
    latest SRP package
  - Added second callback to ddpclient.call, executed when the DDP
    `updated` message is received
  - Allow automatic EJSON serialization/deserialization of ObjectIDs
  - Expose EJSON package to allow for addition of custom EJSON types
  - added DDP pre2 support
    - DDP version negotiation
    - DDP heartbeat support (reply only)
    - `ddpclient.callWithRandomSeed` supports client-generated `_id`s

0.4.6 - 2014-04-28

  - Return id used when calling subscribe method

0.4.5 - 2014-04-24

  - Fix login with password method to return login token

0.4.4 - 2014-02-09

  - Fix a bug where if the server responded to an error on the first
    step of SRP authentication it was not handled correctly (i.e when
    the user is not found)

0.4.3 - 2013-12-19

  - Fix bug with socket reconnects tailspinning into an infinite loop
    (#30 by @jagill)
  - Fix bug when use_ejson was not always set properly by default.
    (#29 by @jagill)

0.4.2 - 2013-12-14

  - Use EJSON by default (#28)

0.4.1 - 2013-12-07

  - Ability to switch off collections monitoring

0.4.0 - 2013-12-07

  - Switched to faye-websockets (#26 by @jagill)

0.3.6 - 2013-11-07

  - fixed bug with default params when ignoring root certs (in case the
    machine doesn't have the cert)
  - Added DDP login with SRP authentication

0.3.5 - 2013-11-05

 - Added non strict SSL option in case of missing root certificates

0.3.4 - 2013-08-28

 - added EJSON support (default is off) with a couple tests

0.3.3 - 2013-05-29

 - fixed bug where an exception could be thrown when sending a message on
   a socket that is not opened anymore (issue #18)
 - added some tests (work in progress)

0.3.2 - 2013-04-08

  - fixed bug where client would reconnect when closing (@tarangp)

0.3.1 - 2013-04-06

  - added a failed message to the connect callback if version negotiation
    fails.

0.3.0 - 2013-03-18

  - moved over to DDP-pre1


text:
# EDGAN

This repository modifies the original StackGAN code from
[github](https://github.com/hanzhanggit/StackGAN).

# Dataset
use MSCOCO data set
## get data set and preprocessed model
- Download [MSCOCO](http://mscoco.org/dataset/#overview) dataset and annotations including captions and instances
- Download pretrained [char-CNN-RNN](https://github.com/reedscot/icml2016) embedding of MSCOCO.
- misc/preprocess_mscoco.py preprocess the image in to different sizes
for selected supercategory
,write them into tfrecords file along with the corresponding caption embedding.

<!-- potential other data set, not as good
yelp data set
visual genome data set
 -->

# New features

## Data input pipline
- use mscoco python API
- dataloader that load tfrecords from mscoco
- image augumentation including cropping, flipping, and standarlization (when downsample the image, use INTER_AREA method)
- sampling from multiple caption embeddings, visualize embedding distributions
- negative example (use inner product of embedding captions, see method CLSGAN)
- filter out selective images based on classes and their areas

<!-- ## Sentense embedding with visual information -->

## Modification of GAN network
- enlarge capacity of generator network, adding 3 residual blocks.
- change relu to leaky relu
- option to no batch norm in discriminator
- increase or reduce discriminator final dimension

## Multiple training methods of GAN
- Option to trian with vanilla GAN
- Option to train with WGAN (excluding weight clipping for batchnorm)
- Option to train with LSGAN
- Option to train with CLSGAN, continous least square GAN that estimates the inner products of embeddings between right caption embeddings and wrong caption embeddings.
- Option to train with BGAN (not implemented yet)

## Classification Transfering from Imagenet to MSCOCO (for future 3 stage GAN)
- Label each image in MSCOCO with multiple labels for objects that have area larger than the threshold
- Transfer resnet from Caffe to Tensorflow
- Train resnet to classify the 80 categories of objects in MSCOCO

<!--
# ToDo List
## minor
- check regularization

## major
- test second stage gan
- create demo
- feature mapping
- test dog cat

# To Do List Future

- further test wgan, (test disable embedding weight clipping first)
- explore different negative sampling

- train classification (takes long, should do it in caffe)
- generate classification map (try use ground truth)
- new piplines for 3 stage gan inputs
- finish 3 stage gan
- train 3 stage gan

# Test results
- WGAN, takes longer to train, unclear about improvements (worse on bird, better on mscoco)
- LSGAN, wrose result, shorter to train
- lr need to be low, 0.0002 instead of 0.002
- CLSGAN, really good result on mscoco
- deconv may cause patterns, resize is better
-->

# References publications
- [StackGAN](https://arxiv.org/pdf/1612.03242.pdf)
- [text2image](https://arxiv.org/pdf/1605.05396.pdf)
- [char-RNN-CNN]
- [WGAN](https://arxiv.org/pdf/1701.07875.pdf)
- [LSGAN](https://arxiv.org/pdf/1611.04076.pdf)
- [BGAN](https://arxiv.org/pdf/1702.08431.pdf)

<!---
- regularization

- own implementation
error possible discriminator variable sharing

- gate gradients -->

repository: yao-zhao/EDGAN

repo_name: yao-zhao/EDGAN


text:
# Unique Paths II

Follow up for "Unique Paths":

Now consider if some obstacles are added to the grids. How many unique paths would there be?

An obstacle and empty space is marked as 1 and 0 respectively in the grid.

For example,
There is one obstacle in the middle of a 3x3 grid as illustrated below.
```
[
  [0,0,0],
  [0,1,0],
  [0,0,0]
]
```
The total number of unique paths is 2.

**Note:** m and n will be at most 100.

**Solution:**
```java
public class Solution {
  public int uniquePathsWithObstacles(int[][] obstacleGrid) {
    int m = obstacleGrid.length;
    int n = obstacleGrid[0].length;
    int[][] dp = new int[m][n];

    // first row
    for (int j = 0; j < n; j++) {
      if (obstacleGrid[0][j] == 1) {
        break;
      }
      dp[0][j] = 1;
    }

    // first col
    for (int i = 0; i < m; i++) {
      if (obstacleGrid[i][0] == 1) {
        break;
      }
      dp[i][0] = 1;
    }

    // others
    for (int i = 1; i < m; i++) {
      for (int j = 1; j < n; j++) {
        if (obstacleGrid[i][j] == 0) {
          dp[i][j] = dp[i-1][j] + dp[i][j-1];
        }
      }
    }

    return dp[m-1][n-1];
  }
}
```

repository: riktimmondal/solve-leetcode-problems

path: /problems/Unique Paths II.md

filename: Unique Paths II.md

repo_name: riktimmondal/solve-leetcode-problems


text:
# Image As Particles

This library been moved to ngx-acuw!!!

Angular library to show images as particles with using three.js

This library is based on the great work and code of [Bruno Imbrizi](https://github.com/brunoimbrizi/) and brought to Angular.

A description and a tutorial how the project was created can be found on https://tympanus.net/codrops/2019/01/17/interactive-particles-with-three-js/

| Original image  | Image as interactive particles |
| ------ | ------------------ |
| ![image](projects/ng-showcase/src/assets/dog.png) | ![](example.gif) |

## Demo

[Showcase](https://windmichael.github.io/ng-image-as-particles/)

`ng-image-as-particles` used on a homepage on welcome page: https://lenny-the-samoyed.firebaseapp.com/

## Dependencies
* [Angular](https://angular.io) (*requires* Angular 10+ )
* [three.js](https://threejs.org) (*requires* threejs 0.123.0 )

## Installation
#### 1. Install version 0.123.0 of `three.js` via npm:
```
npm install<EMAIL_ADDRESS>```

#### 2. Install `ng-image-as-particles` via npm:
```
npm install ng-image-as-particles
```
Once installed you need to import the main module:
```typescript
import { NgImageAsParticlesModule } from 'ng-image-as-particles';

@NgModule({
  declarations: [AppComponent, ...],
  imports: [NgImageAsParticlesModule],
  bootstrap: [AppComponent]
})
export class AppModule {
}
```

## Usage
```html
<lib-image-as-particles [imageUrl]="selectedUrl" touchAction="none"
            imageWidth="90%" imageHeight="90%" backgroundColor="#222222"
            horizontalAlignment="center" verticalAlignment="center">
</lib-image-as-particles>
```

## API
| Name  | Default | Description |
| ----- | ------- | ----------- |
| imageUrl | null | URL to the image |
| backgroundColor | "##222222" | The background-color CSS property sets the background color of an element. ([background-color](https://developer.mozilla.org/en-US/docs/Web/CSS/background-color)) |
| touchAction | "none" | The touch-action CSS property sets how an element's region can be manipulated by a touchscreen user. ([touchAction](https://developer.mozilla.org/en-US/docs/Web/CSS/touch-action)) |
| imageWidth | "100%" | Width of the generated image in "px" or "%" |
| imageHeight | "100%" | Height of the generated image in "px" or "%" |
| horizontalAlignment | "center" | Possible values: "start", "center", "end" |
| verticalAlignment | "center" | Possible values: "top", "center", "bottom" |

repository: windmichael/ng-image-as-particles

gha_language: TypeScript

repo_name: windmichael/ng-image-as-particles


text:
---
title: "glooctl upgrade"
weight: 5
---
## glooctl upgrade

upgrade glooctl binary

### Synopsis

upgrade glooctl binary

```
glooctl upgrade [flags]
```

### Options

```
  -h, --help             help for upgrade
      --path string      Desired path for your upgraded glooctl binary. Defaults to the location of your currently executing binary.
      --release string   Which glooctl release to download. Specify a git tag corresponding to the desired version of glooctl. (default "latest")
```

### Options inherited from parent commands

```
  -i, --interactive         use interactive mode
      --kubeconfig string   kubeconfig to use, if not standard one
```

### SEE ALSO

* [glooctl](../glooctl)	 - CLI for Gloo

repository: ahmelsayed/gloo

repo_name: ahmelsayed/gloo


text:
Here you'll find a very simple example using computed properties to build a hosting price simulator with VueJS.

It is not intented to be a full-fledged example, and as such uses in-html CSS, raw javascript, no components, no unit tests at this point.

## Relevant links

* [Live Demo](https://thbar.github.io/vuejs-calculator/)
* [VueJS](https://vuejs.org)
* [Computed properties documentation](https://vuejs.org/v2/guide/computed.html)
* [Filters documentation](https://vuejs.org/v2/guide/filters.html) (used to format money together with [Accounting.js](accountingjs)

repository: thbar/vuejs-calculator

repo_name: thbar/vuejs-calculator