text:
# ThenKit
[![Build Status](https://travis-ci.org/rodrigo-lima/ThenKit.svg?branch=master)](https://travis-ci.org/rodrigo-lima/ThenKit) [![codecov](https://codecov.io/gh/rodrigo-lima/ThenKit/branch/master/graph/badge.svg)](https://codecov.io/gh/rodrigo-lima/ThenKit)

Promises/A+ implementation in Swift 2 inspired by Pinky - http://lazd.github.io/Pinky/

When researching [Promises/A+](https://promisesaplus.com), I was looking for a really simple implementation of the spec to better understand the data structures being using and how callbacks/closures/blocks were managed and fired during fulfilled / rejected.

There are [several implementations](https://promisesaplus.com/implementations) of the spec and I look at several of them and ended up using [Pinky](http://lazd.github.io/Pinky/) as a starting point.

From their page - [Pinky](http://lazd.github.io/Pinky/) is a no-nonsense Promises/A+ 1.1 implementation. Pinky is written to be very readable and easy to follow, with references to the relevant sections of the spec for each operation. *As such, Pinky can be used as an academic example of a promises implementation.* -- exactly what I was looking for!

## Keeping It Simple

There are several existing frameworks for Promises/A+, Reactive, RX, you name it. I've used [ReactiveCocoa](https://github.com/ReactiveCocoa/ReactiveCocoa) before and it's very powerful. But again, the existing frameworks were either *too big* for my use-case, or not yet fully ported to *Swift 2*.

My main interest was to orchestrate asynchronous into synchronous steps. Example is a upload media service, where you have a sequence of networks requests that need to happen in specific sequence.

**Promises/A+** seemed like a good option, as *"a __promise__ represents the eventual result of an asynchronous operation."*

**ThenKit** implements *Promises/A+* and adds a *Completion* block to be executed after all blocks are fulfilled/rejected. It's supposed to be very simple so it might not resolve all your uses cases, i.e. no cancellable or scheduled promises, hot/cold signals, etc... There are more complete frameworks for that.

## Usage

TBD. Add Carthage support.
In the meantime, copy **ThenKit.swift** somewhere in our project.

### API

As this implementation is based on **Pinky**, the API is very similar:

Create a `Promise` that will eventually be *Fulfilled* or *Rejected*:

- *Promises* could have a name -- which helps a lot during debug

```swift
let p = Promise()
p.name = "Promise_1"    // look for me in the debug logs
return p.promise
```

After executing some asynchronous work, this promise will be either *fulfilled* or *rejected* -- say, depending on the network operation result:

```swift
func httpGetPromise(someURL: String) -> Thenable {
    let p = Promise()
    p.name = "HTTP_GET_PROMISE"    // look for me in the debug logs

    guard let url = NSURL(string: someURL) else {
        let badURLErr = NSError(domain: "ThenKitTests.Error.BadURL", code: 100, userInfo: nil)
        p.reject(badURLErr)
        return p.promise // rejected promise
    }
    let request = NSURLRequest(URL: url)
    let session = NSURLSession.sharedSession()
    let task = session.dataTaskWithRequest(request) { (data, response, error) in
        if error != nil {
            p.reject(error!)
        }
        else {
            // wraps NSData & NSHTTPURLResponse to return
            let responseWrapper = HTTPResponseWrapper(data: data, urlResponse: response)
            p.fulfill(responseWrapper)
        }
    }
    task.resume()
    return p.promise    // this is *Thenable* object you'll be working with
}

// somewhere else in the code, let's GET some URL
httpGetPromise("http://google.com")
.then({ httpResponse in
    print("got this response \(httpResponse)")

    // optionally, we could return a value to chain promises together

}, onRejected: { error in
    print("some error ocurred \(error)")
    return error

}) { success in
    print("and we're done. -- success ? \(success)")
}
```

These are the additional helper *then* methods of *Thenable* protocol:

##### `then(onFulfilled:Function?) -> Thenable`
- simple call with optional *success/fulfill* block

##### `then(onFulfilled:Function?, onRejected:RejectedFunction?) -> Thenable`
- provides *success/fulfill* and *failure/reject* blocks

##### `then(onFulfilled:Function?, onCompleted: CompleteFunction?) -> Thenable`
- adds *onCompleted* block, which returns true/false indicating fulfilled / rejected state

##### `then(onFulfilled:Function?, onRejected:RejectedFunction?, onCompleted: CompleteFunction?) -> Thenable`
- complete call with the 3 blocks - success, failure, complete. Note that *onCompleted* is invoked whether it's fulfilled or rejected.

### Chaining

The really interesting functionality is to be able to synchronize asynchronous calls:

```swift
func fetchRandom(name:String) -> Thenable {
    let p = Promise()
    p.name = name
    dispatch_after(2.seconds) {
        print(".... FULFILL fetchRandom \(p)")
        p.fulfill(random()/100000)
    }
    return p.promise
}

func testChainPromises() {
    let readyExpectation = expectationWithDescription("testChainPromises")

    // FUTURE
    let step0 = fetchRandom("__step0__")
    step0.then { someResult in
        print("Step 0 -- complete with \(someResult)")

        // notice that we are returning another promise here!
        let step1 = fetchRandom("__step1__")
        return step1
    }
    .then { someResult in
        print("Step 1 -- complete with \(someResult)")

        // and another promise
        let step2 = someOtherPromise("__step2__")
        return step2
    }
    .then({ someResult in
        print("Step 2 -- complete with \(someResult)")
        // last step - does not need to return anything if we don't want to
    },
    // handle failures (in any step!!)
    onRejected: { someError
        print("Error \(someError)")
    }) { success in
        // on complete
        print("All steps completed -- success? \(success)")
        readyExpectation.fulfill()
    }

    // WAIT
    waitForExpectationsWithTimeout(10) { error in
        XCTAssertNil(error)
    }
}
```

### Resources
- https://promisesaplus.com
- https://github.com/promises-aplus/promises-spec
- http://lazd.github.io/Pinky/
- http://www.drewag.me/posts/practical-use-for-curried-functions-in-swift

### Additional
- https://github.com/antitypical/Result
- https://gist.github.com/softwaredoug/9044640
- http://robnapier.net/functional-wish-fulfillment
- http://robnapier.net/flatmap
- https://github.com/thoughtbot/FunctionalJSON-swift/tree/d3fcf771c20813e57cb54472dd8c55ee33e87ae4/FunctionalJSON
- http://www.sunsetlakesoftware.com/2015/06/12/swift-2-error-handling-practice
- https://medium.com/@robringham/promises-in-swift-66f377c3e403
- https://github.com/rringham/swift-promises/blob/master/promises/Promise.swift
- https://www.promisejs.org/implementing/
- http://eamodeorubio.github.io/tamingasync/#/
- https://robots.thoughtbot.com/efficient-json-in-swift-with-functional-concepts-and-generics
- https://robots.thoughtbot.com/real-world-json-parsing-with-swift

### Other Promises Frameworks in Swift
- https://github.com/ReactiveX/RxSwift
- https://github.com/ReactiveCocoa/ReactiveCocoa
- https://github.com/mxcl/PromiseKit
- https://github.com/ReactKit/ReactKit
- https://github.com/supertommy/craft
- https://github.com/Thomvis/BrightFutures

repository: rodrigo-lima/ThenKit

repo_name: rodrigo-lima/ThenKit


text:
LocalCI

A continuous integration(CI) solution for local application or service which is behind a firewall. No need to share code with commonly CI tools like CircleCI, TravisCI and Jenkin, because you are hosting your the whole CI cycle.

## History
I am hosting few web applications locally in development server and will be built many times a day due to provide a touch edge preview to user. The repeatly process can be done by CI tool. However, none of CI tool can deploy to my server due to the company's firewall. That's why I developed LocalCI.

1) Edit and push the code to GitHub.

2) GitHub will fire a Push event(trigger) to registered webhook (*the webhook should be set by you. Just point to the LocalCI server script is OK!).

3) LocalCI client will pull events by polling the LocalCI server script every 5 seconds (*you should provide the repository url and branch name which you want to watch).
4) Running your script if Push event is received.

## Installation + Use
1. Install the npm globally
``` bash
npm install -g localci
```

2. Start versioning for directory
```bash
localci
```

## License
MIT

license: MIT

repository: ssmak/LocalCI

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: ssmak/LocalCI


text:
---
services: media-services,logic-app
platforms: dotnetcore
author: raffertyuy
---

# Media Services: Integrating Azure Media Services with Azure Functions & Azure Logic Apps

This project contains Logic Apps that converts-to-media-asset, encodes and publishes media files that you upload in an Azure Storage.

The project includes Logic Apps ARM templates that is dependent on the [advanced-vod-workflow](../advanced-vod-workflow) functions. There are 2 projects, one for uploading and encoding and another for publishing.

This Media Services Functions example is based on AMS REST API v3 on Azure Functions v3.

See [here](https://raffertyuy.com/raztype/creating-an-azure-media-services-upload-workflow-using-azure-storage-and-logic-apps) to read more about how this logic app is implemented.

## Prerequisites for a sample Logic Apps deployments

### 1. Deploy the Advanced VOD Workflow Azure Functions

Follow the instructions and deploy the azure functions found in [advanced-vod-workflow](../advanced-vod-workflow).

### 2. Edit the ARM Template Parameters

Update the parameters found in the LogicApp.json of both projects

### 3. Deploy the 2 Logic App Projects

See [here](https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-azure-resource-manager-templates-overview) for instructions.

branch_name: refs/heads/main

gha_language: C#

gha_license_id: MIT


text:
# Tornado Scaffold

## Installation

```bash
git clone https://github.com/zxzhang5/tornado_scaffold.git
cd tornado_scaffold
python setup.py develop
```

## Run

```bash
python app.py
```

## License

This project is licensed under the MIT License.


text:
# Papers for stringologist (2017)

- [ALENEX(2017)](#alenex-2017) [[dblp]](https://dblp.org/db/conf/alenex/alenex2017.html)
- [COCOA (2)(2017)](#cocoa-(2)-2017) [[dblp]](https://dblp.org/db/conf/cocoa/cocoa2017-2.html)
- [COCOON(2017)](#cocoon-2017) [[dblp]](https://dblp.org/db/conf/cocoon/cocoon2017.html)
- [CPM(2017)](#cpm-2017) [[dblp]](https://dblp.org/db/conf/cpm/cpm2017.html)
- [CSR(2017)](#csr-2017) [[dblp]](https://dblp.org/db/conf/csr/csr2017.html)
- [CiE(2017)](#cie-2017) [[dblp]](https://dblp.org/db/conf/cie/cie2017.html)
- [DCC(2017)](#dcc-2017) [[dblp]](https://dblp.org/db/conf/dcc/dcc2017.html)
- [DLT(2017)](#dlt-2017) [[dblp]](https://dblp.org/db/conf/dlt/dlt2017.html)
- [EANN(2017)](#eann-2017) [[dblp]](https://dblp.org/db/conf/eann/eann2017.html)
- [ESA(2017)](#esa-2017) [[dblp]](https://dblp.org/db/conf/esa/esa2017.html)
- [FCT(2017)](#fct-2017) [[dblp]](https://dblp.org/db/conf/fct/fct2017.html)
- [ICALP(2017)](#icalp-2017) [[dblp]](https://dblp.org/db/conf/icalp/icalp2017.html)
- [ISAAC(2017)](#isaac-2017) [[dblp]](https://dblp.org/db/conf/isaac/isaac2017.html)
- [IWOCA(2017)](#iwoca-2017) [[dblp]](https://dblp.org/db/conf/iwoca/iwoca2017.html)
- [LATA(2017)](#lata-2017) [[dblp]](https://dblp.org/db/conf/lata/lata2017.html)
- [MFCS(2017)](#mfcs-2017) [[dblp]](https://dblp.org/db/conf/mfcs/mfcs2017.html)
- [SEA(2017)](#sea-2017) [[dblp]](https://dblp.org/db/conf/wea/sea2017.html)
- [SISAP(2017)](#sisap-2017) [[dblp]](https://dblp.org/db/conf/sisap/sisap2017.html)
- [SODA(2017)](#soda-2017) [[dblp]](https://dblp.org/db/conf/soda/soda2017.html)
- [SOFSEM(2017)](#sofsem-2017) [[dblp]](https://dblp.org/db/conf/sofsem/sofsem2017.html)
- [SPIRE(2017)](#spire-2017) [[dblp]](https://dblp.org/db/conf/spire/spire2017.html)
- [STACS(2017)](#stacs-2017) [[dblp]](https://dblp.org/db/conf/stacs/stacs2017.html)
- [STOC(2017)](#stoc-2017) [[dblp]](https://dblp.org/db/conf/stoc/stoc2017.html)
- [Stringology(2017)](#stringology-2017) [[dblp]](https://dblp.org/db/conf/stringology/stringology2017.html)
- [WABI(2017)](#wabi-2017) [[dblp]](https://dblp.org/db/conf/wabi/wabi2017.html)
- [WADS(2017)](#wads-2017) [[dblp]](https://dblp.org/db/conf/wads/wads2017.html)
- [WALCOM(2017)](#walcom-2017) [[dblp]](https://dblp.org/db/conf/walcom/walcom2017.html)
- [WORDS(2017)](#words-2017) [[dblp]](https://dblp.org/db/conf/cwords/words2017.html)

- [ACM J. Exp. Algorithmics(2017)](#acm-j-exp-algorithmics-2017) [[dblp]](https://dblp.org/db/journals/jea/jea22.html)
- [Algorithmica(2017)](#algorithmica-2017) [[dblp]](https://dblp.org/db/journals/algorithmica/algorithmica79.html)
- [Algorithms Mol. Biol.(2017)](#algorithms-mol-biol-2017) [[dblp]](https://dblp.org/db/journals/almob/almob12.html)
- [BMC Bioinform.(2017)](#bmc-bioinform-2017) [[dblp]](https://dblp.org/db/journals/bmcbi/bmcbi18S.html)
- [Bioinform.(2017)](#bioinform-2017) [[dblp]](https://dblp.org/db/journals/bioinformatics/bioinformatics33.html)
- [CoRR(2017)](#corr-2017) [[dblp]](https://dblp.org/db/journals/corr/corr1705.html)
- [IEEE ACM Trans. Comput. Biol. Bioinform.(2017)](#ieee-acm-trans-comput-biol-bioinform-2017) [[dblp]](https://dblp.org/db/journals/tcbb/tcbb14.html)
- [IEICE Trans. Fundam. Electron. Commun. Comput. Sci.(2017)](#ieice-trans-fundam-electron-commun-comput-sci-2017) [[dblp]](https://dblp.org/db/journals/ieicet/ieicet100a.html)
- [Inf. Process. Lett.(2017)](#inf-process-lett-2017) [[dblp]](https://dblp.org/db/journals/ipl/ipl128.html)
- [Inf. Retr. J.(2017)](#inf-retr-j-2017) [[dblp]](https://dblp.org/db/journals/ir/ir20.html)
- [Inf. Syst.(2017)](#inf-syst-2017) [[dblp]](https://dblp.org/db/journals/is/is69.html)
- [J. Comput. Syst. Sci.(2017)](#j-comput-syst-sci-2017) [[dblp]](https://dblp.org/db/journals/jcss/jcss86.html)
- [J. Discrete Algorithms(2017)](#j-discrete-algorithms-2017) [[dblp]](https://dblp.org/db/journals/jda/jda43.html)
- [Math. Comput. Sci.(2017)](#math-comput-sci-2017) [[dblp]](https://dblp.org/db/journals/mics/mics11.html)
- [Math. Struct. Comput. Sci.(2017)](#math-struct-comput-sci-2017) [[dblp]](https://dblp.org/db/journals/mscs/mscs27.html)
- [Pattern Recognit. Lett.(2017)](#pattern-recognit-lett-2017) [[dblp]](https://dblp.org/db/journals/prl/prl88.html)
- [SIAM J. Comput.(2017)](#siam-j-comput-2017) [[dblp]](https://dblp.org/db/journals/siamcomp/siamcomp46.html)
- [Theor. Comput. Sci.(2017)](#theor-comput-sci-2017) [[dblp]](https://dblp.org/db/journals/tcs/tcs698.html)

## Contents
#### [ALENEX 2017](https://dblp.org/db/conf/alenex/alenex2017.html)
  1. [CSA++: Fast Pattern Search for Large Alphabets.](https://doi.org/10.1137/1.9781611974768.6)
  2. [Compact Dynamic Rewritable (CDRW) Arrays.](https://doi.org/10.1137/1.9781611974768.9)
  3. [Elias-Fano meets Single-Term Top-k Document Retrieval.](https://doi.org/10.1137/1.9781611974768.11)
  4. [Engineering External Memory Induced Suffix Sorting.](https://doi.org/10.1137/1.9781611974768.8)
  5. [Engineering a Distributed Full-Text Index.](https://doi.org/10.1137/1.9781611974768.10)

#### [COCOA (2) 2017](https://dblp.org/db/conf/cocoa/cocoa2017-2.html)
  1. [Faster Algorithms for 1-Mappability of a Sequence.](https://doi.org/10.1007/978-3-319-71147-8_8)

#### [COCOON 2017](https://dblp.org/db/conf/cocoon/cocoon2017.html)
  1. [Efficient Enumeration of Non-Equivalent Squares in Partial Words with Few Holes.](https://doi.org/10.1007/978-3-319-62389-4_9)

#### [CPM 2017](https://dblp.org/db/conf/cpm/cpm2017.html)
  1. [A Family of Approximation Algorithms for the Maximum Duo-Preservation String Mapping Problem.](https://doi.org/10.4230/LIPIcs.CPM.2017.10)
  2. [Approximate Cover of Strings.](https://doi.org/10.4230/LIPIcs.CPM.2017.26)
  3. [Beyond Adjacency Maximization: Scaffold Filling for New String Distances.](https://doi.org/10.4230/LIPIcs.CPM.2017.27)
  4. [Can We Recover the Cover?.](https://doi.org/10.4230/LIPIcs.CPM.2017.25)
  5. [Clique-Based Lower Bounds for Parsing Tree-Adjoining Grammars.](https://doi.org/10.4230/LIPIcs.CPM.2017.12)
  6. [Communication and Streaming Complexity of Approximate Pattern Matching.](https://doi.org/10.4230/LIPIcs.CPM.2017.13)
  7. [Computing All Distinct Squares in Linear Time for Integer Alphabets.](https://doi.org/10.4230/LIPIcs.CPM.2017.22)
  8. [Deterministic Indexing for Packed Strings.](https://doi.org/10.4230/LIPIcs.CPM.2017.6)
  9. [Document Listing on Repetitive Collections with Guaranteed Performance.](https://doi.org/10.4230/LIPIcs.CPM.2017.4)
  10. [Dynamic Elias-Fano Representation.](https://doi.org/10.4230/LIPIcs.CPM.2017.30)
  11. [Fast and Simple Jumbled Indexing for Binary Run-Length Encoded Strings.](https://doi.org/10.4230/LIPIcs.CPM.2017.19)
  12. [Faster STR-IC-LCS Computation via RLE.](https://doi.org/10.4230/LIPIcs.CPM.2017.20)
  13. [From LZ77 to the Run-Length Encoded Burrows-Wheeler Transform, and Back.](https://doi.org/10.4230/LIPIcs.CPM.2017.17)
  14. [Front Matter, Table of Contents, Preface, Conference Organization, External Reviewers.](https://doi.org/10.4230/LIPIcs.CPM.2017.0)
  15. [Gapped Pattern Statistics.](https://doi.org/10.4230/LIPIcs.CPM.2017.21)
  16. [Lempel-Ziv Compression in a Sliding Window.](https://doi.org/10.4230/LIPIcs.CPM.2017.15)
  17. [Longest Common Extensions with Recompression.](https://doi.org/10.4230/LIPIcs.CPM.2017.18)
  18. [On the Weighted Quartet Consensus Problem.](https://doi.org/10.4230/LIPIcs.CPM.2017.28)
  19. [On-Line Pattern Matching on Similar Texts.](https://doi.org/10.4230/LIPIcs.CPM.2017.9)
  20. [Optimal Omnitig Listing for Safe and Complete Contig Assembly.](https://doi.org/10.4230/LIPIcs.CPM.2017.29)
  21. [Palindromic Length in Linear Time.](https://doi.org/10.4230/LIPIcs.CPM.2017.23)
  22. [Path Queries on Functions.](https://doi.org/10.4230/LIPIcs.CPM.2017.5)
  23. [Position Heaps for Parameterized Strings.](https://doi.org/10.4230/LIPIcs.CPM.2017.8)
  24. [Recompression of SLPs.](https://doi.org/10.4230/LIPIcs.CPM.2017.2)
  25. [Representing the Suffix Tree with the CDAWG.](https://doi.org/10.4230/LIPIcs.CPM.2017.7)
  26. [Revisiting the Parameterized Complexity of Maximum-Duo Preservation String Mapping.](https://doi.org/10.4230/LIPIcs.CPM.2017.11)
  27. [Shortest Superstring.](https://doi.org/10.4230/LIPIcs.CPM.2017.3)
  28. [Synergistic Solutions on MultiSets.](https://doi.org/10.4230/LIPIcs.CPM.2017.31)
  29. [The Longest Filled Common Subsequence Problem.](https://doi.org/10.4230/LIPIcs.CPM.2017.14)
  30. [Tight Bounds on the Maximum Number of Shortest Unique Substrings.](https://doi.org/10.4230/LIPIcs.CPM.2017.24)
  31. [Time-Space Trade-Offs for Lempel-Ziv Compressed Indexing.](https://doi.org/10.4230/LIPIcs.CPM.2017.16)
  32. [Wheeler Graphs: Variations on a Theme by Burrows and Wheeler.](https://doi.org/10.4230/LIPIcs.CPM.2017.1)

#### [CSR 2017](https://dblp.org/db/conf/csr/csr2017.html)
  1. [Palindromic Decompositions with Gaps and Errors.](https://doi.org/10.1007/978-3-319-58747-9_7)

#### [CiE 2017](https://dblp.org/db/conf/cie/cie2017.html)
  1. [Flexible Indexing of Repetitive Collections.](https://doi.org/10.1007/978-3-319-58741-7_17)

#### [DCC 2017](https://dblp.org/db/conf/dcc/dcc2017.html)
  1. [A Compact Index for Order-Preserving Pattern Matching.](https://doi.org/10.1109/DCC.2017.35)
  2. [A Succinct Data Structure for Multidimensional Orthogonal Range Searching.](https://doi.org/10.1109/DCC.2017.47)
  3. [Complementary Contextual Models with FM-Index for DNA Compression.](https://doi.org/10.1109/DCC.2017.50)
  4. [Compressed Dynamic Range Majority Data Structures.](https://doi.org/10.1109/DCC.2017.9)
  5. [Content Adaptive Embedded Compression.](https://doi.org/10.1109/DCC.2017.13)
  6. [Full Compressed Affix Tree Representations.](https://doi.org/10.1109/DCC.2017.39)
  7. [Improved Parallel Construction of Wavelet Trees and Rank/Select Structures.](https://doi.org/10.1109/DCC.2017.85)
  8. [Improvements on Re-Pair Grammar Compressor.](https://doi.org/10.1109/DCC.2017.52)
  9. [LZ-End Parsing in Compressed Space.](https://doi.org/10.1109/DCC.2017.73)
  10. [Making Compression Algorithms for Unicode Text.](https://doi.org/10.1109/DCC.2017.58)
  11. [Marlin: A High Throughput Variable-to-Fixed Codec Using Plurally Parsable Dictionaries.](https://doi.org/10.1109/DCC.2017.74)
  12. [Optimize Genomics Data Compression with Hardware Accelerator.](https://doi.org/10.1109/DCC.2017.49)
  13. [Space-Efficient Re-Pair Compression.](https://doi.org/10.1109/DCC.2017.24)
  14. [Stabbing Colors in One Dimension.](https://doi.org/10.1109/DCC.2017.44)
  15. [Streaming K-Mismatch with Error Correcting and Applications.](https://doi.org/10.1109/DCC.2017.14)
  16. [Symmetry-Compressible Graphs.](https://doi.org/10.1109/DCC.2017.20)

#### [DLT 2017](https://dblp.org/db/conf/dlt/dlt2017.html)
  1. [On the Number of Rich Words.](https://doi.org/10.1007/978-3-319-62809-7_26)

#### [EANN 2017](https://dblp.org/db/conf/eann/eann2017.html)
  1. [Efficient Computation of Palindromes in Sequences with Uncertainties.](https://doi.org/10.1007/978-3-319-65172-9_52)
  2. [Efficient Identification of k-Closed Strings.](https://doi.org/10.1007/978-3-319-65172-9_49)

#### [ESA 2017](https://dblp.org/db/conf/esa/esa2017.html)
  1. [A Space-Optimal Grammar Compression.](https://doi.org/10.4230/LIPIcs.ESA.2017.67)
  2. [An Encoding for Order-Preserving Matching.](https://doi.org/10.4230/LIPIcs.ESA.2017.38)
  3. [Dynamic Space Efficient Hashing.](https://doi.org/10.4230/LIPIcs.ESA.2017.58)
  4. [Fast Dynamic Arrays.](https://doi.org/10.4230/LIPIcs.ESA.2017.16)
  5. [LZ-End Parsing in Linear Time.](https://doi.org/10.4230/LIPIcs.ESA.2017.53)
  6. [Real-Time Streaming Multi-Pattern Search for Constant Alphabet.](https://doi.org/10.4230/LIPIcs.ESA.2017.41)

#### [FCT 2017](https://dblp.org/db/conf/fct/fct2017.html)
  1. [Minimal Absent Words in a Sliding Window and Applications to On-Line Pattern Matching.](https://doi.org/10.1007/978-3-662-55751-8_14)

#### [ICALP 2017](https://dblp.org/db/conf/icalp/icalp2017.html)
  1. [Dynamic Time Warping and Geometric Edit Distance: Breaking the Quadratic Barrier.](https://doi.org/10.4230/LIPIcs.ICALP.2017.25)
  2. [String Inference from Longest-Common-Prefix Array.](https://doi.org/10.4230/LIPIcs.ICALP.2017.62)

#### [ISAAC 2017](https://dblp.org/db/conf/isaac/isaac2017.html)
  1. [Almost Linear Time Computation of Maximal Repetitions in Run Length Encoded Strings.](https://doi.org/10.4230/LIPIcs.ISAAC.2017.33)
  2. [Fast Compressed Self-Indexes with Deterministic Linear-Time Construction.](https://doi.org/10.4230/LIPIcs.ISAAC.2017.57)
  3. [On-the-Fly Array Initialization in Less Space.](https://doi.org/10.4230/LIPIcs.ISAAC.2017.44)
  4. [Structural Pattern Matching - Succinctly.](https://doi.org/10.4230/LIPIcs.ISAAC.2017.35)
  5. [Succinct Color Searching in One Dimension.](https://doi.org/10.4230/LIPIcs.ISAAC.2017.30)

#### [IWOCA 2017](https://dblp.org/db/conf/iwoca/iwoca2017.html)
  1. [A Faster Implementation of Online Run-Length Burrows-Wheeler Transform.](https://doi.org/10.1007/978-3-319-78825-8_33)
  2. [Computing Abelian String Regularities Based on RLE.](https://doi.org/10.1007/978-3-319-78825-8_34)
  3. [How to Answer a Small Batch of RMQs or LCA Queries in Practice.](https://doi.org/10.1007/978-3-319-78825-8_28)
  4. [Shortest Unique Palindromic Substring Queries in Optimal Time.](https://doi.org/10.1007/978-3-319-78825-8_32)

#### [LATA 2017](https://dblp.org/db/conf/lata/lata2017.html)
  1. [Efficient Pattern Matching in Elastic-Degenerate Texts.](https://doi.org/10.1007/978-3-319-53733-7_9)
  2. [Integrated Encryption in Dynamic Arithmetic Compression.](https://doi.org/10.1007/978-3-319-53733-7_10)
  3. [Two-Dimensional Palindromes and Their Properties.](https://doi.org/10.1007/978-3-319-53733-7_11)

#### [MFCS 2017](https://dblp.org/db/conf/mfcs/mfcs2017.html)
  1. [Binary Search in Graphs Revisited.](https://doi.org/10.4230/LIPIcs.MFCS.2017.20)
  2. [Small-Space LCE Data Structure with Constant-Time Queries.](https://doi.org/10.4230/LIPIcs.MFCS.2017.10)
  3. [The Hardness of Solving Simple Word Equations.](https://doi.org/10.4230/LIPIcs.MFCS.2017.18)

#### [SEA 2017](https://dblp.org/db/conf/wea/sea2017.html)
  1. [A Framework of Dynamic Data Structures for String Processing.](https://doi.org/10.4230/LIPIcs.SEA.2017.11)
  2. [Compression with the tudocomp Framework.](https://doi.org/10.4230/LIPIcs.SEA.2017.13)
  3. [Engineering External Memory LCP Array Construction: Parallel, In-Place and Large Alphabet.](https://doi.org/10.4230/LIPIcs.SEA.2017.17)
  4. [Fast and Scalable Minimal Perfect Hashing for Massive Key Sets.](https://doi.org/10.4230/LIPIcs.SEA.2017.25)
  5. [Online Construction of Wavelet Trees.](https://doi.org/10.4230/LIPIcs.SEA.2017.16)
  6. [Practical Range Minimum Queries Revisited.](https://doi.org/10.4230/LIPIcs.SEA.2017.12)
  7. [The Quantile Index - Succinct Self-Index for Top-k Document Retrieval.](https://doi.org/10.4230/LIPIcs.SEA.2017.15)

#### [SISAP 2017](https://dblp.org/db/conf/sisap/sisap2017.html)
  1. [Practical Space-Efficient Data Structures for High-Dimensional Orthogonal Range Searching.](https://doi.org/10.1007/978-3-319-68474-1_16)
  2. [Scalable Similarity Search for Molecular Descriptors.](https://doi.org/10.1007/978-3-319-68474-1_14)
  3. [Succinct Quadtrees for Road Data.](https://doi.org/10.1007/978-3-319-68474-1_18)

#### [SODA 2017](https://dblp.org/db/conf/soda/soda2017.html)
  1. [Hardness of Permutation Pattern Matching.](https://doi.org/10.1137/1.9781611974782.24)
  2. [Space-Efficient Construction of Compressed Indexes in Deterministic Linear Time.](https://doi.org/10.1137/1.9781611974782.26)
  3. [Sparse Suffix Tree Construction in Optimal Time and Space.](https://doi.org/10.1137/1.9781611974782.27)
  4. [pBWT: Achieving Succinct Data Structures for Parameterized Pattern Matching and Related Problems.](https://doi.org/10.1137/1.9781611974782.25)

#### [SOFSEM 2017](https://dblp.org/db/conf/sofsem/sofsem2017.html)
  1. [Computing Longest Single-arm-gapped Palindromes in a String.](https://doi.org/10.1007/978-3-319-51963-0_29)
  2. [Edit-Distance Between Visibly Pushdown Languages.](https://doi.org/10.1007/978-3-319-51963-0_30)
  3. [Longest Common Subsequence in at Least k Length Order-Isomorphic Substrings.](https://doi.org/10.1007/978-3-319-51963-0_28)

#### [SPIRE 2017](https://dblp.org/db/conf/spire/spire2017.html)
  1. [A Self-index on Block Trees.](https://doi.org/10.1007/978-3-319-67428-5_24)
  2. [Constructing a Consensus Phylogeny from a Leaf-Removal Distance (Extended Abstract).](https://doi.org/10.1007/978-3-319-67428-5_12)
  3. [Counting Palindromes in Substrings.](https://doi.org/10.1007/978-3-319-67428-5_25)
  4. [Detecting One-Variable Patterns.](https://doi.org/10.1007/978-3-319-67428-5_22)
  5. [Distinct Squares in Circular Words.](https://doi.org/10.1007/978-3-319-67428-5_3)
  6. [Efficient Compression and Indexing of Trajectories.](https://doi.org/10.1007/978-3-319-67428-5_10)
  7. [Fast Construction of Compressed Web Graphs.](https://doi.org/10.1007/978-3-319-67428-5_11)
  8. [Fast Label Extraction in the CDAWG.](https://doi.org/10.1007/978-3-319-67428-5_14)
  9. [Faster Practical Block Compression for Rank/Select Dictionaries.](https://doi.org/10.1007/978-3-319-67428-5_20)
  10. [Greedy Shortest Common Superstring Approximation in Compact Space.](https://doi.org/10.1007/978-3-319-67428-5_1)
  11. [LZ78 Compression in Low Main Memory Space.](https://doi.org/10.1007/978-3-319-67428-5_4)
  12. [Lightweight BWT and LCP Merging via the Gap Algorithm.](https://doi.org/10.1007/978-3-319-67428-5_15)
  13. [Linear-Size CDAWG: New Repetition-Aware Indexing and Grammar Compression.](https://doi.org/10.1007/978-3-319-67428-5_26)
  14. [Listing Maximal Independent Sets with Minimal Space and Bounded Delay.](https://doi.org/10.1007/978-3-319-67428-5_13)
  15. [Longest Common Factor After One Edit Operation.](https://doi.org/10.1007/978-3-319-67428-5_2)
  16. [Mining Bit-Parallel LCS-length Algorithms.](https://doi.org/10.1007/978-3-319-67428-5_18)
  17. [On Suffix Tree Breadth.](https://doi.org/10.1007/978-3-319-67428-5_6)
  18. [On Two LZ78-style Grammars: Compression Bounds and Compressed-Space Computation.](https://doi.org/10.1007/978-3-319-67428-5_5)
  19. [Optimal Skeleton Huffman Trees.](https://doi.org/10.1007/978-3-319-67428-5_21)
  20. [Order Preserving Pattern Matching on Trees and DAGs.](https://doi.org/10.1007/978-3-319-67428-5_23)
  21. [Pattern Matching on Elastic-Degenerate Text with Errors.](https://doi.org/10.1007/978-3-319-67428-5_7)
  22. [Practical Evaluation of Lempel-Ziv-78 and Lempel-Ziv-Welch Tries.](https://doi.org/10.1007/978-3-319-67428-5_16)
  23. [Practical Implementation of Space-Efficient Dynamic Keyword Dictionaries.](https://doi.org/10.1007/978-3-319-67428-5_19)
  24. [Regular Abelian Periods and Longest Common Abelian Factors on Run-Length Encoded Strings.](https://doi.org/10.1007/978-3-319-67428-5_17)
  25. [Succinct Partial Sums and Fenwick Trees.](https://doi.org/10.1007/978-3-319-67428-5_8)
  26. [Tight Bounds for Top Tree Compression.](https://doi.org/10.1007/978-3-319-67428-5_9)

#### [STACS 2017](https://dblp.org/db/conf/stacs/stacs2017.html)
  1. [On Long Words Avoiding Zimin Patterns.](https://doi.org/10.4230/LIPIcs.STACS.2017.19)
  2. [On the Size of Lempel-Ziv and Lyndon Factorizations.](https://doi.org/10.4230/LIPIcs.STACS.2017.45)

#### [STOC 2017](https://dblp.org/db/conf/stoc/stoc2017.html)
  1. [Synchronization strings: codes for insertions and deletions approaching the Singleton bound.](https://doi.org/10.1145/3055399.3055498)

#### [Stringology 2017](https://dblp.org/db/conf/stringology/stringology2017.html)
  1. [A Family of Exact Pattern Matching Algorithms with Multiple Adjacent Search Windows.](http://www.stringology.org/event/2017/p14.html)
  2. [A Lempel-Ziv-style Compression Method for Repetitive Texts.](http://www.stringology.org/event/2017/p10.html)
  3. [Counting Mismatches with SIMD.](http://www.stringology.org/event/2017/p06.html)
  4. [Dismantling DivSufSort.](http://www.stringology.org/event/2017/p07.html)
  5. [Dynamic Succinct Data Structures and Compressed Random Access Memory.](http://www.stringology.org/event/2017/p01.html)
  6. [Faster Batched Range Minimum Queries.](http://www.stringology.org/event/2017/p09.html)
  7. [Many-MADFAct: Concurrently Constructing MADFAs.](http://www.stringology.org/event/2017/p13.html)
  8. [On Reverse Engineering the Lyndon Tree.](http://www.stringology.org/event/2017/p11.html)
  9. [Online Recognition of Dictionary with One Gap.](http://www.stringology.org/event/2017/p02.html)
  10. [Range Queries Using Huffman Wavelet Trees.](http://www.stringology.org/event/2017/p03.html)
  11. [Regular Expressions with Backreferences Re-examined.](http://www.stringology.org/event/2017/p04.html)
  12. [Speeding Up String Matching by Weak Factor Recognition.](http://www.stringology.org/event/2017/p05.html)
  13. [The Linear Equivalence of the Suffix Array and the Partially Sorted Lyndon Array.](http://www.stringology.org/event/2017/p08.html)
  14. [Trade-offs in Query and Target Indexing for the Selection of Candidates in Protein Homology Searches.](http://www.stringology.org/event/2017/p12.html)

#### [WABI 2017](https://dblp.org/db/conf/wabi/wabi2017.html)
  1. [Optimal Computation of Overabundant Words.](https://doi.org/10.4230/LIPIcs.WABI.2017.4)
  2. [Rainbowfish: A Succinct Colored de Bruijn Graph Representation.](https://doi.org/10.4230/LIPIcs.WABI.2017.18)

#### [WADS 2017](https://dblp.org/db/conf/wads/wads2017.html)
  1. [Optimal Query Time for Encoding Range Majority.](https://doi.org/10.1007/978-3-319-62127-2_35)

#### [WALCOM 2017](https://dblp.org/db/conf/walcom/walcom2017.html)
  1. [A Space-Efficient Algorithm for the Dynamic DFS Problem in Undirected Graphs.](https://doi.org/10.1007/978-3-319-53925-6_23)

#### [WORDS 2017](https://dblp.org/db/conf/cwords/words2017.html)
  1. [A de Bruijn Sequence Construction by Concatenating Cycles of the Complemented Cycling Register.](https://doi.org/10.1007/978-3-319-66396-8_6)
  2. [Burrows-Wheeler Transform and Run-Length Enconding.](https://doi.org/10.1007/978-3-319-66396-8_21)
  3. [Minimal Forbidden Factors of Circular Words.](https://doi.org/10.1007/978-3-319-66396-8_5)

#### ACM J. Exp. Algorithmics 2017
  1. [Practical Compact Indexes for Top-k Document Retrieval.](https://doi.org/10.1145/3043958)

#### Algorithmica 2017
  1. [A Framework for Space-Efficient String Kernels.](https://doi.org/10.1007/s00453-017-0286-4)
  2. [Compressed Subsequence Matching and Packed Tree Coloring.](https://doi.org/10.1007/s00453-015-0068-9)
  3. [Efficient Computation of Substring Equivalence Classes with Suffix Arrays.](https://doi.org/10.1007/s00453-016-0178-z)
  4. [Efficient Indexes for Jumbled Pattern Matching with Constant-Sized Alphabet.](https://doi.org/10.1007/s00453-016-0140-0)
  5. [On the Succinct Representation of Equivalence Classes.](https://doi.org/10.1007/s00453-016-0192-1)
  6. [String Powers in Trees.](https://doi.org/10.1007/s00453-016-0271-3)
  7. [Top-k Term-Proximity in Succinct Space.](https://doi.org/10.1007/s00453-016-0167-2)

#### Algorithms Mol. Biol. 2017
  1. [On avoided words, absent words, and their application to biological sequence analysis.](https://doi.org/10.1186/s13015-017-0094-z)

#### BMC Bioinform. 2017
  1. [A framework for space-efficient read clustering in metagenomic samples.](https://doi.org/10.1186/s12859-017-1466-6)

#### Bioinform. 2017
  1. [Succinct colored de Bruijn graphs.](https://doi.org/10.1093/bioinformatics/btx067)
  2. [emMAW: computing minimal absent words in external memory.](https://doi.org/10.1093/bioinformatics/btx209)

#### CoRR 2017
  1. [A Faster Implementation of Online Run-Length Burrows-Wheeler Transform.](https://arxiv.org/abs/1704.05233)
  2. [A Framework of Dynamic Data Structures for String Processing.](https://arxiv.org/abs/1701.07238)
  3. [A Grammar Compression Algorithm based on Induced Suffix Sorting.](https://arxiv.org/abs/1711.03205)
  4. [A Separation Between Run-Length SLPs and LZ77.](https://arxiv.org/abs/1711.07270)
  5. [A compressed dynamic self-index for highly repetitive text collections.](https://arxiv.org/abs/1711.02855)
  6. [A family of approximation algorithms for the maximum duo-preservation string mapping problem.](https://arxiv.org/abs/1702.02405)
  7. [A local search 2.917-approximation algorithm for duo-preservation string mapping.](https://arxiv.org/abs/1702.01877)
  8. [A succinct data structure for self-indexing ternary relations.](https://arxiv.org/abs/1707.02759)
  9. [Alphabet-dependent Parallel Algorithm for Suffix Tree Construction for Pattern Searching.](https://arxiv.org/abs/1704.05660)
  10. [Approximation ratio of RePair.](https://arxiv.org/abs/1703.06061)
  11. [Assembling sequences of DNA using an on-line algorithm based on DeBruijn graphs.](https://arxiv.org/abs/1705.05105)
  12. [At the Roots of Dictionary Compression: String Attractors.](https://arxiv.org/abs/1710.10964)
  13. [B-slack trees: Highly Space Efficient B-trees.](https://arxiv.org/abs/1712.05020)
  14. [Better Labeling Schemes for Nearest Common Ancestors through Minor-Universal Trees.](https://arxiv.org/abs/1707.06011)
  15. [Biased Predecessor Search.](https://arxiv.org/abs/1707.01182)
  16. [Bloom Filters, Adaptivity, and the Dictionary Problem.](https://arxiv.org/abs/1711.01616)
  17. [Bubble-Flip - A New Generation Algorithm for Prefix Normal Words.](https://arxiv.org/abs/1712.05876)
  18. [Cartesian trees and Lyndon trees.](https://arxiv.org/abs/1712.08749)
  19. [Closing in on Time and Space Optimal Construction of Compressed Indexes.](https://arxiv.org/abs/1712.04886)
  20. [Compaction of Church Numerals for Higher-Order Compression.](https://arxiv.org/abs/1706.10061)
  21. [Comparison of LZ77-type Parsings.](https://arxiv.org/abs/1708.03558)
  22. [Compressed Indexing with Signature Grammars.](https://arxiv.org/abs/1711.08217)
  23. [Compressed Representation of Dynamic Binary Relations with Applications.](https://arxiv.org/abs/1707.02769)
  24. [Compression with the tudocomp Framework.](https://arxiv.org/abs/1702.07577)
  25. [Computing Abelian regularities on RLE strings.](https://arxiv.org/abs/1701.02836)
  26. [Dismantling DivSufSort.](https://arxiv.org/abs/1710.01896)
  27. [Distinct Squares in Circular Words.](https://arxiv.org/abs/1708.00639)
  28. [Document Listing on Repetitive Collections with Guaranteed Performance.](https://arxiv.org/abs/1707.06374)
  29. [Duel and sweep algorithm for order-preserving pattern matching.](https://arxiv.org/abs/1705.09438)
  30. [Efficient Compression and Indexing of Trajectories.](https://arxiv.org/abs/1710.01952)
  31. [Efficient Dynamic Dictionary Matching with DAWGs and AC-automata.](https://arxiv.org/abs/1710.03395)
  32. [Efficient pattern matching in degenerate strings with the Burrows-Wheeler transform.](https://arxiv.org/abs/1708.01130)
  33. [Even faster sorting of (not only) integers.](https://arxiv.org/abs/1703.00687)
  34. [Exact Mean Computation in Dynamic Time Warping Spaces.](https://arxiv.org/abs/1710.08937)
  35. [Exploiting Computation-Friendly Graph Compression Methods.](https://arxiv.org/abs/1708.07271)
  36. [FAMOUS: Fast Approximate string Matching using OptimUm search Schemes.](https://arxiv.org/abs/1711.02035)
  37. [FMtree: A fast locating algorithm of FM-indexes for genomic data.](https://arxiv.org/abs/1704.04615)
  38. [Fast Compressed Self-Indexes with Deterministic Linear-Time Construction.](https://arxiv.org/abs/1707.01743)
  39. [Fast Computation of Graph Edit Distance.](https://arxiv.org/abs/1709.10305)
  40. [Fast Dynamic Arrays.](https://arxiv.org/abs/1711.00275)
  41. [Fast Label Extraction in the CDAWG.](https://arxiv.org/abs/1707.08197)
  42. [Fast Locating with the RLBWT.](https://arxiv.org/abs/1705.10382)
  43. [Fast and Simple Jumbled Indexing for Binary RLE Strings.](https://arxiv.org/abs/1702.01280)
  44. [Fast and Simple Parallel Wavelet Tree and Matrix Construction.](https://arxiv.org/abs/1702.07578)
  45. [Faster STR-IC-LCS computation via RLE.](https://arxiv.org/abs/1703.04954)
  46. [Faster algorithms for 1-mappability of a sequence.](https://arxiv.org/abs/1705.04022)
  47. [Faster batched range minimum queries.](https://arxiv.org/abs/1706.06940)
  48. [Faster range minimum queries.](https://arxiv.org/abs/1711.10385)
  49. [Faster truncated integer multiplication.](https://arxiv.org/abs/1703.00640)
  50. [From LZ77 to the Run-Length Encoded Burrows-Wheeler Transform, and Back.](https://arxiv.org/abs/1702.01340)
  51. [Grammar-Based Graph Compression.](https://arxiv.org/abs/1704.05254)
  52. [Greedy Shortest Common Superstring Approximation in Compact Space.](https://arxiv.org/abs/1707.07727)
  53. [How to answer a small batch of RMQs or LCA queries in practice.](https://arxiv.org/abs/1705.04589)
  54. [Hybridizing Non-dominated Sorting Algorithms: Divide-and-Conquer Meets Best Order Sort.](https://arxiv.org/abs/1704.04205)
  55. [HyperMinHash: Jaccard index sketching in LogLog space.](https://arxiv.org/abs/1710.08436)
  56. [Improved Average Complexity for Comparison-Based Sorting.](https://arxiv.org/abs/1705.00849)
  57. [Improved Bounds for Testing Forbidden Order Patterns.](https://arxiv.org/abs/1710.10660)
  58. [Improved bounds for testing Dyck languages.](https://arxiv.org/abs/1707.06606)
  59. [In-Place Initializable Arrays.](https://arxiv.org/abs/1709.08900)
  60. [Indexing Weighted Sequences: Neat and Efficient.](https://arxiv.org/abs/1704.07625)
  61. [Inverse Lyndon words and Inverse Lyndon factorizations of words.](https://arxiv.org/abs/1705.10277)
  62. [Lempel-Ziv: a "one-bit catastrophe" but not a tragedy.](https://arxiv.org/abs/1707.04312)
  63. [Linear-size CDAWG: new repetition-aware indexing and grammar compression.](https://arxiv.org/abs/1705.09779)
  64. [Longest common substring with approximately k mismatches.](https://arxiv.org/abs/1712.08573)
  65. [Lyndon Array Construction during Burrows-Wheeler Inversion.](https://arxiv.org/abs/1710.10105)
  66. [Maximal Unbordered Factors of Random Strings.](https://arxiv.org/abs/1704.04472)
  67. [Multiresolution Priority Queues.](https://arxiv.org/abs/1705.09642)
  68. [Near-Optimal Compression for the Planar Graph Metric.](https://arxiv.org/abs/1703.04814)
  69. [Near-optimal linear decision trees for k-SUM and related problems.](https://arxiv.org/abs/1705.01720)
  70. [New Cardinality Estimation Methods for HyperLogLog Sketches.](https://arxiv.org/abs/1706.07290)
  71. [New Variants of Pattern Matching with Constants and Variables.](https://arxiv.org/abs/1705.09504)
  72. [New cardinality estimation algorithms for HyperLogLog sketches.](https://arxiv.org/abs/1702.01284)
  73. [On Two LZ78-style Grammars: Compression Bounds and Compressed-Space Computation.](https://arxiv.org/abs/1705.09538)
  74. [On the Decision Tree Complexity of String Matching.](https://arxiv.org/abs/1712.09738)
  75. [On-line Assembling Mitochondrial DNA from de novo transcriptome.](https://arxiv.org/abs/1706.02828)
  76. [On-the-Fly Array Initialization in Less Space.](https://arxiv.org/abs/1709.10477)
  77. [Optimal Computation of Overabundant Words.](https://arxiv.org/abs/1705.03385)
  78. [Optimal Time and Space Construction of Suffix Arrays and LCP Arrays for Integer Alphabets.](https://arxiv.org/abs/1703.01009)
  79. [Optimal top dag compression.](https://arxiv.org/abs/1712.05822)
  80. [Optimal trade-offs for pattern matching with k mismatches.](https://arxiv.org/abs/1704.01311)
  81. [Order preserving pattern matching on trees and DAGs.](https://arxiv.org/abs/1706.00148)
  82. [Ordered Dags: HypercubeSort.](https://arxiv.org/abs/1710.00944)
  83. [Orthogonal Vectors Indexing.](https://arxiv.org/abs/1710.00586)
  84. [Palindromic Decompositions with Gaps and Errors.](https://arxiv.org/abs/1703.08931)
  85. [Persistent Cache-oblivious Streaming Indexes.](https://arxiv.org/abs/1707.08186)
  86. [Position Heaps for Parameterized Strings.](https://arxiv.org/abs/1702.02321)
  87. [Practical Evaluation of Lempel-Ziv-78 and Lempel-Ziv-Welch Tries.](https://arxiv.org/abs/1706.03035)
  88. [Practical and Effective Re-Pair Compression.](https://arxiv.org/abs/1704.08558)
  89. [Probabilistic Analysis of the Dual-Pivot Quicksort "Count".](https://arxiv.org/abs/1710.07505)
  90. [Relations Between Greedy and Bit-Optimal LZ77 Encodings.](https://arxiv.org/abs/1707.09789)
  91. [Reoptimization of the Closest Substring Problem under Pattern Length Modification.](https://arxiv.org/abs/1703.06644)
  92. [Representing the suffix tree with the CDAWG.](https://arxiv.org/abs/1705.08640)
  93. [Run Compressed Rank/Select for Large Alphabets.](https://arxiv.org/abs/1711.02910)
  94. [Small-space encoding LCE data structure with constant-time queries.](https://arxiv.org/abs/1702.07458)
  95. [Space-Efficient Algorithms for Longest Increasing Subsequence.](https://arxiv.org/abs/1712.09230)
  96. [Space-efficient K-MER algorithm for generalized suffix tree.](https://arxiv.org/abs/1703.02224)
  97. [Streaming Pattern Matching with d Wildcards.](https://arxiv.org/abs/1704.01646)
  98. [Streaming Periodicity with Mismatches.](https://arxiv.org/abs/1708.04381)
  99. [String Attractors.](https://arxiv.org/abs/1709.05314)
  100. [Succinct Approximate Rank Queries.](https://arxiv.org/abs/1704.07710)
  101. [Succinct Partial Sums and Fenwick Trees.](https://arxiv.org/abs/1705.10987)
  102. [Text Indexing and Searching in Sublinear Time.](https://arxiv.org/abs/1712.07431)
  103. [The Case for Learned Index Structures.](https://arxiv.org/abs/1712.01208)
  104. [The Compressed Overlap Index.](https://arxiv.org/abs/1707.05613)
  105. [The Hidden Binary Search Tree: A Balanced Rotation-Free Search Tree in the AVL RAM Model.](https://arxiv.org/abs/1711.07746)
  106. [The complexity of the Multiple Pattern Matching Problem for random strings.](https://arxiv.org/abs/1706.04928)
  107. [The streaming k-mismatch problem.](https://arxiv.org/abs/1708.05223)
  108. [Time-Space Trade-Offs for Lempel-Ziv Compressed Indexing.](https://arxiv.org/abs/1706.10094)
  109. [Tree Edit Distance Cannot be Computed in Strongly Subcubic Time (unless APSP can).](https://arxiv.org/abs/1703.08940)
  110. [Trie Compression for GPU Accelerated Multi-Pattern Matching.](https://arxiv.org/abs/1702.03657)
  111. [Truly Sub-cubic Algorithms for Language Edit Distance and RNA Folding via Fast Bounded-Difference Min-Plus Product.](https://arxiv.org/abs/1707.05095)
  112. [Twin Sort Technique.](https://arxiv.org/abs/1710.07992)
  113. [Whole Genome Phylogenetic Tree Reconstruction Using Colored de Bruijn Graphs.](https://arxiv.org/abs/1709.00164)
  114. [m-Bonsai: a Practical Compact Dynamic Trie.](https://arxiv.org/abs/1704.05682)

#### IEEE ACM Trans. Comput. Biol. Bioinform. 2017
  1. [Benchmark Dataset for Whole Genome Sequence Compression.](https://doi.org/10.1109/TCBB.2016.2568186)

#### IEICE Trans. Fundam. Electron. Commun. Comput. Sci. 2017
  1. [Packed Compact Tries: A Fast and Efficient Data Structure for Online String Processing.](https://doi.org/10.1587/transfun.E100.A.1785)

#### Inf. Process. Lett. 2017
  1. [Optimal suffix sorting and LCP array construction for constant alphabets.](https://doi.org/10.1016/j.ipl.2016.09.010)
  2. [Tight lower bounds for the longest common extension problem.](https://doi.org/10.1016/j.ipl.2017.05.003)
  3. [Two strings at Hamming distance 1 cannot be both quasiperiodic.](https://doi.org/10.1016/j.ipl.2017.08.005)

#### Inf. Retr. J. 2017
  1. [Document retrieval on repetitive string collections.](https://doi.org/10.1007/s10791-017-9297-7)

#### Inf. Syst. 2017
  1. [Compressed representation of dynamic binary relations with applications.](https://doi.org/10.1016/j.is.2017.05.003)

#### J. Comput. Syst. Sci. 2017
  1. [Fast algorithms for Abelian periods in words and greatest common divisor queries.](https://doi.org/10.1016/j.jcss.2016.09.003)
  2. [Fingerprints in compressed strings.](https://doi.org/10.1016/j.jcss.2017.01.002)

#### J. Discrete Algorithms 2017
  1. [A space efficient direct access data structure.](https://doi.org/10.1016/j.jda.2016.12.001)
  2. [A succinct data structure for self-indexing ternary relations.](https://doi.org/10.1016/j.jda.2016.10.002)
  3. [Burrows-Wheeler transform and LCP array construction in constant space.](https://doi.org/10.1016/j.jda.2016.11.003)
  4. [Grammar compressed sequences with rank/select support.](https://doi.org/10.1016/j.jda.2016.10.001)
  5. [Preface - Compact Data Structures.](https://doi.org/10.1016/j.jda.2017.04.002)
  6. [Subsequence automata with default transitions.](https://doi.org/10.1016/j.jda.2017.05.001)

#### Math. Comput. Sci. 2017
  1. [Block Graphs in Practice.](https://doi.org/10.1007/s11786-016-0286-9)
  2. [Compressed Spaced Suffix Arrays.](https://doi.org/10.1007/s11786-016-0283-z)
  3. [Engineering a Lightweight External Memory Suffix Array Construction Algorithm.](https://doi.org/10.1007/s11786-016-0281-1)

#### Math. Struct. Comput. Sci. 2017
  1. [Fast circular dictionary-matching algorithm.](https://doi.org/10.1017/S0960129515000134)

#### Pattern Recognit. Lett. 2017
  1. [A faster and more accurate heuristic for cyclic edit distance computation.](https://doi.org/10.1016/j.patrec.2017.01.018)

#### SIAM J. Comput. 2017
  1. [The "Runs" Theorem.](https://doi.org/10.1137/15M1011032)
  2. [Time-Optimal Top-k Document Retrieval.](https://doi.org/10.1137/140998949)

#### Theor. Comput. Sci. 2017
  1. [Covering problems for partial words and for indeterminate strings.](https://doi.org/10.1016/j.tcs.2017.05.026)
  2. [Inducing enhanced suffix arrays for string collections.](https://doi.org/10.1016/j.tcs.2017.03.039)
  3. [Inferring strings from Lyndon factorization.](https://doi.org/10.1016/j.tcs.2017.05.038)
  4. [String cadences.](https://doi.org/10.1016/j.tcs.2017.04.019)
  5. [Wheeler graphs: A framework for BWT-based data structures.](https://doi.org/10.1016/j.tcs.2017.06.016)

repository: stringologytimes/StringologyTimes

gha_language: TypeScript

repo_name: stringologytimes/StringologyTimes


text:
# 0.1.0

Initial release.

# 0.1.1

- Gitutils crashed when trying to save the data extracted from git. Fixed.
- Project path is now optional for `foliant build`. If you omit the `--path`
  option, the current directory will be used.

# 0.1.2

- Commandline interface for `build` changed: target format is now a mandatory
  argument, ``--target`` option is deprecated.

# 0.1.4

- Pandoc filters can be specified in config.json.

# 0.1.7

- In-place seqdiag diagram rendering added.

# 0.1.8

- Diagram rendering has become much faster.
- Swagger to Markdown file converter added.

# 0.1.9

- Swagger to Markdown converter now uses
  [swagger2markdown](https://github.com/moigagoo/swagger2markdown) instead of
  [swagger2markup](https://github.com/Swagger2Markup/swagger2markup).

# 0.2.0

- Output file is now named according to the following format:
  `<title>_<version>-<date>`.
- You can now specify the output file name explicitly with `file_name` value
  in `config.json`. If it's not set, the slugified `title` value is used.

# 0.2.1

- Fix false warning about "file_name" config value.

# 0.2.2

- Add Apidoc to Markdown converter
  (via [apidoc2markdown](https://github.com/moigagoo/apidoc2markdown)).
- Markdown file converted from Swagger or Apidoc is now called `api.md`
  (instead of `swagger.md`).

repository: foliant-docs/foliant-nim

repo_name: foliant-docs/foliant-nim


text:
# md564

Base64 MD5 of strings and JSON, where JSON attributes are sorted before comparison for consistent results

## Install :hammer:

```sh
npm install md564
```

## Usage :bulb:

```js
md564 = new require('md564')
console.log(MD564('hello'))        // 1TGK0fk4hfTgamcP4G5xMI
obj1  = {bar: 'grill', foo: 'man'}
obj2  = {foo: 'man', bar: 'grill'}
console.log(MD564(obj1, true))    // 3MEEOkTwl6NBNuzrYVpHeX
console.log(MD564(obj2))          // 0_8LeiEkKgp~Kwo5qV15E1
console.log(MD564(obj2, true))    // 3MEEOkTwl6NBNuzrYVpHeX
```

If param2 is ```true``` then the object attributes will be sorted prior to being made into a string and obtaining the MD5 value.

## Contributions :muscle:

:smile: Feedback, problem reports, enhancement requests are welcome.

:up: Example code are better.

:cool: Pull requests are best.

## License

### MIT


text:
# Reddit API

API fetch using axios and hooks on /r/reactjs for practice and knowledge. Inpired in @dceddia project on his ebook. See online on: https://csb-n5q0w-erb4ss1c7.vercel.app/

This is a Python library for interacting with the Reddit API. With it, you can easily access information about Reddit /r/reactjs/ posts.

### Usage
To use the Reddit API, you will first need to create a Reddit account and obtain an API key. Once you have your API key, you can use it to access the Reddit API by instantiating a Reddit object and passing it your API key.

### License
The Reddit API library is released under the MIT license. See the LICENSE file for more information.


text:
# Genie3 Parallel Processing for Human Genomes

*By Jiahui Tang, Wenhan Zhang, Xiaoman(Mandy) Zi*

For more details please proceed to the documentation [website](https://cs205-genie3-parallel.github.io/genie3-parallel/)

The following guide is a sub section of overall documentation that gives a short introduction into technical details, dataset, and how to use our code.

- [Links to Repository with Source Code, Evaluation Data Sets and Test Cases](#links-to-repository-with-source-code--evaluation-data-sets-and-test-cases)
- [Technical Description of the Software Design](#technical-description-of-the-software-design)
  * [SageMaker](#sagemaker)
  * [Spark](#spark)
  * [PySpark Graph](#pyspark-graph)
- [Code Baseline](#code-baseline)
- [Dependencies](#dependencies)
- [How to Use the Code](#how-to-use-the-code)
  * [SageMaker](#sagemaker-1)
  * [PySpark](#pyspark)
    + [*Spark on Single Node*](#-spark-on-single-node-)
    + [*Spark on EMR Hadoop Cluster*](#-spark-on-emr-hadoop-cluster-)
    + [*Scale Up to Full Dataset from SageMaker Output*](#-scale-up-to-full-dataset-from-sagemaker-output-)
  * [Graph](#graph)
- [System and Environment needed to Reproduce our Tests](#system-and-environment-needed-to-reproduce-our-tests)

## Links to Repository with Source Code, Evaluation Data Sets and Test Cases
[View Repo on GitHub](https://github.com/cs205-genie3-parallel/genie3-parallel)

**Input Datasets (Model Building, Evaluation and Test etc.):**

[Human RNAseq Raw Dataset](https://cs205-final.s3.amazonaws.com/healthy.tsv)

[PySpark Performance Evaluation Dataset - 17 millions gene records](https://genie3-proj.s3.amazonaws.com/ranking_idx.txt)

**Output Datasets: (Intermediate Result, Full Output etc.)**

[SageMaker Output Full Dataset](https://cs205-final.s3.amazonaws.com/output/healthy_0_3000_alljobs)

[PySpark Performance Evaluation Output Dataset - Vertices](https://genie3-proj.s3.amazonaws.com/vertices.csv/part-00000)

[PySpark Performance Evaluation Output Dataset - Edges](https://genie3-proj.s3.amazonaws.com/graph_edges.csv/part-00000)

[PySpark Full Output Dataset - Vertices](https://genie3-proj.s3.amazonaws.com/vertices.csv/part-00000)

[PySpark Full Output Dataset - Edges](https://genie3-proj.s3.amazonaws.com/full_graph_edges.csv/part-00000)

## Technical Description of the Software Design
### SageMaker

For a machine learning job with prepared data for training and testing, SageMaker offers a notebook interface for directly training, testing and deploying estimators on selected AWS EC2 instances. However, since we are using the GENIE3 programming model, we would need to run our custom training script. As novices to SageMaker, we used a SKLearn estimator just for the function of a wrapper. In the custom estimator, we can specify the entry point script, the type of EC2 instance and “hyperparameters” (understandably misused) which will be passed into the entry point script as command line arguments. This is probably a less standard solution of running a custom script on SageMaker and there might be other solutions out there. However, after extensive research and troubleshooting, we find that this is able to fulfil our requirements of orchestrating parallelized computation for now.

```python
customer_estimator = SKLearn(entry_point = 'GENIE3-sagemaker.py',
                            role=get_execution_role(),
                            instance_type='ml.m4.xlarge',
                            framework_version='0.20.0',
                            hyperparameters = {'start_idx': 0, 'stop_idx': 6000, 'n_jobs': -1, 'fname': 'healthy_0_6000_alljobs'})
healthy_uri = f"s3://{bucket_name}/healthy.tsv"
custom_estimator.fit({'train':healthy_uri})
```

In the entry point script, it contains functions from GENIE3 with changes intended to only limit the analysis for genes between the start index and the stop index, instead of running the analysis for all the genes. We also added in a n_jobs argument to parallelize each SKLearn random forest computation.
The main function contains a parser for the arguments passed from the custom estimator, the main GENIE3 function calls, as well as utility functions to read, process data from AWS S3 and to upload output to S3.

```python
if __name__ =='__main__':

    parser = argparse.ArgumentParser()

    # hyperparameters sent by the client are passed as command-line arguments to the script.
    parser.add_argument('--start_idx', type=int, default=0)
    parser.add_argument('--stop_idx', type=int, default=10)
    parser.add_argument('--nthreads', type=int, default=1)
    parser.add_argument('--n_jobs', type=int, default=1)
    parser.add_argument('--fname', type=str, default='output_ranking.txt')

    # Data, model, and output directories
    parser.add_argument('--output-data-dir', type=str, default=os.environ.get('SM_OUTPUT_DATA_DIR'))
    parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN'))

    args, _ = parser.parse_known_args()

    bucket_name = 'cs205-final'
    s3 = boto3.resource('s3')
    output_path = f"s3://{bucket_name}/output/"

    data, gene_names = preprocess_data(os.path.join(args.train, "healthy.tsv"))

    VIM = GENIE3(data, gene_names=gene_names, start_idx=args.start_idx, stop_idx=args.stop_idx,nthreads=args.nthreads,n_jobs=args.n_jobs)
    output_fname = os.path.join('/opt/ml/output/data', args.fname)
    get_link_list(VIM, gene_names=gene_names, file_name=output_fname)

    response = upload_file_to_s3(output_path, output_fname, args.fname)
    print(response)
```

### Spark
Due to the limit on vCPU numbers, we couldn’t start a cluster of GPUs instance **g3.4xlarge**, so we tested performance and speedup of Spark job locally on `Ubuntu 18.04` with **g3.4xlarge** and **m4.xlarge** instances, following guide from [Lab 9](https://harvard-iacs.github.io/2021-CS205/labs/I9/I9.pdf). And then we used m4.xlarge instances to run it in a hadoop cluster with more number of instances and threads per executor, following guide from [Lab 10](https://harvard-iacs.github.io/2021-CS205/labs/I10/I10.pdf).

When running Spark on a single node, there is a single executor to run the application that can be multi-threaded to use the multi-core parallelism. SparkConf allows you to configure some of the common properties, like for example, the number of threads of the application. Running with `local[2]` means two threads - which represents “minimal” parallelism, which can help detect bugs that only exist when we run in a distributed context or reduces execution time on multi-core systems.

we could set the number of threads used in Spark by changing numbers in setMaster to be passed to Spark.

```python

from pyspark import SparkConf, SparkContext
from pyspark.sql.functions import date_format
from pyspark.sql import SparkSession

conf = SparkConf().setMaster('local[*]').setAppName('genie3')
sc = SparkContext(conf = conf)
spark = SparkSession(sc)

text_file = sc.textFile("ranking_idx.txt")

def toCSVLine(data):
  return ','.join(str(d) for d in data.split('\t'))

# consider gene link > 0.02 as significant and print them out as edge file
text_file.filter(lambda line: float(line.split("\t")[2])> 0.005) \
	.repartition(1).map(toCSVLine).saveAsTextFile("graph_edges.csv")

# print all distinct value in the first two columns as vertices file
text_file.filter(lambda line: float(line.split("\t")[2])> 0.005) \
	.flatMap(lambda x: x.split("\t")[:-1]).distinct().repartition(1) \
	.map(toCSVLine).saveAsTextFile("vertices.csv")

```

Next, we could run the following line to create our spark job and reduce pairwise gene correlation we get from previous steps into vertices and edges.

```bash
$ spark-submit spark_output_to_edge_vertices.py
```

We run spark jobs both locally on a single node and in an EMR Hadoop cluster. Execution time is noted down for both Job 0 and Job 1 individually, and we calculated a total execution time by adding both jobs.

* Job 0 - Search for all lines in output from Random Forest Model Prediction with significant correlations between pairs (e.g. > 0.02), output as Edges of Graph
* Job 1 - Use flatmap and deduplication to get an unique list of genes from column 1 and 2 in output file, save it as Vertices of Graph
We could view Spark Job status via Spark UI during the time that Spark job is running:

Spark UI Showing Job Running![image](https://user-images.githubusercontent.com/6150979/117563107-48743f80-b0d6-11eb-83dd-8d70c9d63dc1.png)

Spark UI Showing Job Running![image](https://user-images.githubusercontent.com/6150979/117563125-6477e100-b0d6-11eb-852b-d13fa0e379bb.png)

We changed the number of threads used in Spark single node by editing numbers in setMaster to be passed to Spark, and noted down performance result and speedup for further evaluation in the next section.

Similarly, we could spin up EMR Spark Cluster with instance m4.xlarge to run code in distributed memory with different number of executors and threads per executors.

```bash
$ spark-submit --num-executors 2 --executor-cores 2 spark_output_to_edge_vertices.py
```

Everytime time we try to run it on a different number of executors and cores per executors, we remove previous output in the hadoop file system.

```bash
$ hadoop fs -rm -R -f graph_edges.csv/
$ hadoop fs -rm -R -f vertices.csv/
```

After successfully complete a spark job, we could download the result from hadoop file system and take a look:

![image](https://user-images.githubusercontent.com/6150979/117563197-cb959580-b0d6-11eb-9b88-a5a0754a85e5.png)

Eventually, we could copy our output to S3 bucket for future use and graph building in Spark.

![image](https://user-images.githubusercontent.com/6150979/117563216-eb2cbe00-b0d6-11eb-8ac5-5e45008b6b9d.png)

After submitting to S3, we could take a look in S3 bucket, our results are both included in graph_edges.csv/ and vertices.csv/ folders.

![image](https://user-images.githubusercontent.com/6150979/117563222-f7188000-b0d6-11eb-977a-6a35fb41263e.png)

### PySpark Graph

In our project, the final output produced by Spark could be further read into PySpark as the input of Graph and build a network. The genes names in `vertices.csv` could be used to build gene vertices, and the gene connection with significant pairwise correlation in `graph_edges.csv` could be used as gene edges and weights to build gene relationships.

More technical details could be found on [Advanced Feature - Part II Graph](https://cs205-genie3-parallel.github.io/genie3-parallel/7_advance_graph.html) page.

Graph Details with Vertices and Edges.
![image](https://user-images.githubusercontent.com/6150979/117568939-ba0fb600-b0f5-11eb-9b2e-80bc2240524f.png)

## Code Baseline
Code baseline is based on [GENIE3 package](https://github.com/vahuynh/GENIE3/blob/master/GENIE3_python/GENIE3.py).

The code used `multiprocessing` python package to map all gene data into different threads by the following lines.
It independently calls function `wr_GENIE3_single()` to build random forest model individually by setting `nthreads`.

```python
pool = Pool(nthreads)
alloutput = pool.map(wr_GENIE3_single, input_data)
```

However, the baseline has a high time and space complexity and it requires several days to run the dataset.

The baseline complexity for random forest is calculated as following when we set `n=30k`:
```
O(One tree) = # gene * sqrt(# gene) * log(sqrt(# gene))
O(RF per gene) = # tree * O(One tree)
O(all genes) = # gene * O(RF per gene)
	     = n^2 * sqrt(n) * log(sqrt(n))
	     = 30k * 1000 * 30k * sqrt(30k) * log(30k)
```

Thus, we intended to use SageMaker and parallelize for the scikit learn estimator and Random Forest model building as we described above, to improve computational performance.

## Dependencies
* sklearn==0.24.2
* sagemaker==2.39.0
* boto3
* graphframes 0.6.0

## How to Use the Code

### SageMaker
**System and Environment:**

- SageMaker Notebook Instance: ml.t2.medium (default)
- Kernel: conda_python3 (out of the options provided by SageMaker)

**Steps for Running the Code:**
- Start an AWS SageMaker notebook instance following [this guide](https://docs.aws.amazon.com/sagemaker/latest/dg/onboard-quick-start.html), setting Github repo to [our repo](https://github.com/cs205-genie3-parallel/genie3-parallel.git)

- Install requirements with
```bash
pip install -r requirements.txt
```
- Navigate to sagemaker/GENIE3-sagemaker.ipynb. Select conda_python3 as the kernel.
- Run all the cells, edit the `instance_type`, `hyperparameters` as needed and indicate `start_idx` and `stop_idx` to choose the target genes to compute on.

### PySpark

**System and Environment:**
- OS: Ubuntu 18.04
- Instance for Local Node: tried **m4.xlarge** and **g3.4xlarge**
- Instance for EMR Cluster: **m4.xlarge** with 2 worker nodes, then resize to 4 and 8 worker nodes
- Kernal: x86_64 GNU/Linux
- Java 1.8.0
- Scala 2.11.12
- PySpark 2.4.4
- graphframes 0.6.0

**Steps for Running the Code:**

#### *Spark on Single Node*

Spin up instance by following guide on [Lab 9](https://harvard-iacs.github.io/2021-CS205/labs/I9/I9.pdf)

Open another command line window to upload PySpark script to instance.

```bash
$ scp -i .ssh/cs205-gpu-keypair.pem ranking_idx.txt ubuntu@<IP_ADDRESS>:/home/ubuntu/
$ scp -i .ssh/cs205-gpu-keypair.pem spark_output_to_edge_vertices.py ubuntu@<IP_ADDRESS>:/home/ubuntu/
```

Install relevant package and libraries by following Lab 9.
```bash
$ sudo apt-add-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt install openjdk-8-jdk
$ java -version

$ sudo apt-get install scala
$ scala -version

$ sudo apt-get install python
$ python -h

$ sudo curl -O http://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
$ sudo tar xvf ./spark-2.2.0-bin-hadoop2.7.tgz
$ sudo mkdir /usr/local/spark
$ sudo cp -r spark-2.2.0-bin-hadoop2.7/* /usr/local/spark
```

To config environment, add the following line to `~/.profile`. And then execute ```source ~/.profile``` to update PATH in your current session

```bash
export PATH="$PATH:/usr/local/spark/bin"
```
Next, include the internal hostname and IP to `/etc/hosts` with a text editor, with `sudo vim /etc/hosts`

Then we could execute the spark job by running:

```bash
$ spark-submit spark_output_to_edge_vertices.py
```
We could use `sudo vim spark_output_to_edge_vertices.py` to change the number of thread/core used in that instance by setting different `local[k]` in the line
`conf = SparkConf().setMaster('local[2]').setAppName('genie3')`.

Then, we could rerun the spark job. Before that, remember to remove the previous output files by:

```bash
$ rm -R -f vertices.csv
$ rm -R -f graph_edges.csv
$ spark-submit spark_output_to_edge_vertices.py
```

#### *Spark on EMR Hadoop Cluster*
Spin up instance by following guide from [Lab 10](https://harvard-iacs.github.io/2021-CS205/labs/I10/I10.pdf)

Everytime when we set different number of executor and thread per executor, and resubmit spark job to evaluate the execution time and speedup performance,
we need to remove previously produced output file in Hadoop file system by using `hadoop fs -rm -R -f`.
Then we could resubmit the spark job in EMR Hadoop cluster.

```bash
$ hadoop fs -rm -R -f graph_edges.csv/
$ hadoop fs -rm -R -f vertices.csv/
$ spark-submit --num-executors 2 --executor-cores 1 spark_output_to_edge_vertices.py
```
Resize the number of workers in hardware option from 2 worker node to 4 nodes, and 8 nodes. Then repeat all the steps above to note down execution times for both job.

#### *Scale Up to Full Dataset from SageMaker Output*

Next, as we found out g3.4xlarg instance on a SINGLE node with 16 cores perform the best with 6.16 times speedup. (More details in Performance section)

We will then scale up our project to run on [full dataset](https://cs205-final.s3.amazonaws.com/output/healthy_0_3000_alljobs) (`~3.5G`) with the maximum speedup settings.

We set the `local[16]` and change the input file name to full dataset filename to use all threads in this GPU instance and rerun the following.

```bash
$ hadoop fs -rm -R -f graph_edges.csv/
$ hadoop fs -rm -R -f vertices.csv/
$ spark-submit --num-executors 2 --executor-cores 1 spark_output_to_edge_vertices.py
```

![image](https://user-images.githubusercontent.com/6150979/117566191-cbea5c80-b0e7-11eb-983b-4cc707c708f3.png)

Snippets of Final Output of Significant Human Gene Pairs with more than `0.005` Pairwise Correlations:

(Which could be used as input for graph network edge and weight in Spark Graph)

![image](https://user-images.githubusercontent.com/6150979/117566548-c9890200-b0e9-11eb-8eea-cbee2f75acd3.png)

Snippets of Final Output of Significant Human Distinct Value List:

(Which could be used as input for graph network vertices in Spark Graph)

![image](https://user-images.githubusercontent.com/6150979/117566556-ddccff00-b0e9-11eb-823b-fb6fa673c8ef.png)

### Graph

In the EC2 instance mentioned above in PySpark, start PySpark with Graphframe by

```bash
$ pyspark --packages graphframes:graphframes:0.6.0-spark2.3-s_2.11 --repositories https://repos.spark-packages.org
```

And then we could import output from the Spark Grep tool into PySpark as dataframe, then build graph.

```python
import graphframes as GF
from pyspark.sql import SQLContext
sql_context = SQLContext(sc)

vertice_df = sql_context.read.csv(
    "vertices.csv/part-00000",
    header=False
).toDF("id")

edge_df = sql_context.read.csv(
    "graph_edges.csv/part-00000",
    header=False,
    sep = ','
).toDF("src", "dst", "relationship")

g = GF.GraphFrame(vertice_df, edge_df)
```

Afterwards, more interesting details of graph could be explored.
```python
g.vertices.show()
g.edges.show()

vertexInDegrees = g.inDegrees
vertexInDegrees.show()
vertexOutDegrees = g.outDegrees

## explore network by group by or filtering
g.edges.filter("relationship > 0.008").count()
g.edges.filter("relationship > 0.008").show()
```

## System and Environment needed to Reproduce our Tests

**System and Environment for SageMaker:**
- SageMaker Notebook Instance: ml.t2.medium (default)
- Kernel: conda_python3 (out of the options provided by SageMaker)

**System and Environment for Spark/Graph:**
- OS: Ubuntu 18.04
- Instance for Local Node: tried **m4.xlarge** and **g3.4xlarge**
- Instance for EMR Cluster: **m4.xlarge** with 2 worker nodes, then resize to 4 and 8 worker nodes
- Kernal: x86_64 GNU/Linux
- Java 1.8.0
- Scala 2.11.12
- PySpark 2.4.4
- graphframes 0.6.0

gha_language: Python


text:
---
title: DataVec Records
short_title: Records
description: How to use data records in DataVec.
category: DataVec
weight: 1
---

# Records

## What is a record?

In the DataVec world a Record represents a single entry in a dataset. DataVec differentiates types of records to make data manipulation easier with built-in APIs. Sequences and 2D records are distinguishable.

## Using records

Most of the time you do not need to interact with the record classes directly, unless you are manually iterating records for the purpose of forwarding through a neural network.

## Types of records

### Record

[\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/datavec/datavec-api/src/main/java/org/datavec/api/records/impl/Record.java)

A standard implementation of the Record interface

### SequenceRecord

[\[source\]](https://github.com/eclipse/deeplearning4j/tree/master/datavec/datavec-api/src/main/java/org/datavec/api/records/impl/SequenceRecord.java)

A standard implementation of the SequenceRecord interface.


text:
Commonlounge
Big Data
About
Go Pro
How to Contribute
Sign up · Log in
TutorialBig DataLast updated 5 months ago
Part of course: The Hands-On Guide to Hadoop and Big Data
HBase Tutorial — Hadoop and NoSQL Part 1[ Edit ]
What is NoSQL?
A NoSQL database is a type of database that stores and retrieves data differently than a traditional relational database. NoSQL databases are also known as non relational databases, or as Not-only-SQL databases because they can have a SQL-like language that is used to query data.
NoSQL databases grew in popularity with the emergence of companies such as Google and Amazon because of the requirements of real time web applications combined with massive amounts of data (Big Data). It's simplistic design allows horizontally scalability across multiple nodes easily, resulting in high availability. They also offer a lot of options when it comes to how the data is stored and retrieved — data can be stored in key value pairs, wide column, document, and graphs, providing a great deal of flexibility.
Why does Hadoop need NoSQL?
Hadoop has some limitations when it comes to processing and retrieval of data. HDFS is really good at sequential querying of data. One thing HDFS cannot do is query data in a random fashion. For example, if Hadoop was to look up a word in the dictionary, it would start from the beginning and go through every page to find the right word. If you had a NoSQL database, you could “randomly” go to a page and start looking or go to a certain letter and start looking there. This is why HBase was created for the Hadoop ecosystem.
What is a columnar, or column-based database?
A columnar database stores tables by column instead of by row. Imagine, taking a relational database table and turning it on its side.
Imagine a user table such as the one below:

A user table
A row oriented database might store this table row by row keyed by the rowId
1:512; Williams; Jason<EMAIL_ADDRESS>2:344; Miller; Kelly<EMAIL_ADDRESS>...
However, a column oriented database will clump the columns together:
512:1, 344:2, 231:3, 983:4
Williams:1, Miller:2, Wilson:3, Brown:4
Jason:1, Kelly:2, John:3, Nancy:4
...
Note that in the row-oriented example, rowId is the primary key, whereas in the column-oriented example, the data itself is the primary key.
Choosing which type to use depends on whether it is more likely that an entire row needs to be fetched when doing queries (which row-based databases are better at) or just a subset of the data or a few columns.
What is HBase?
HBase is a columnar database that is built on top of HDFS (thereby inheriting the distributed nature of HDFS). It is very similar to Google’s Big Table and is designed to give the end user random access across huge sets of data quickly.
The schema of the table defines only column families, which are a collection of columns. A column is a collection of key value pairs, does not need to defined in the schema, and can be created on the fly.
HBase Example Schema
Let’s look at an example of what some data inside of an HBase table looks like:

Row Key, Customer Information, and Customer Property are examples of Column Families. customerId, name, age, car type, and car color are examples of columns.
HBase Features
Since it is based on HDFS, HBase has the ability to scale horizontally and to interact with Hadoop’s processing power as a source and destination.
Also, since HBase is a Hadoop application:
It inherits the native fault tolerance that comes with Hadoop.
It has a native java api that can be used to interact with it.
It is provided with data replication so that data is never lost.
HBase does provide consistent reads and writes and should be looked at when developing an application that is heavy on writes and needs a lot of random reads. Let’s dive in and get some hands-on experience.
HBase Example
Setting up Docker
First thing as always is to get your cloudera quickstart image up and running. Whether that is via a local instance or the Digital Ocean instance. First thing we need to do is get our data where it needs to be. You should still have the movie data from the Hive tutorial. If you don’t, follow these quick steps to get the data.
For this tutorial, we will be working with some movie data. You can download the data that we are going to be using from this website here: http://www.grouplens.org/system/files/ml-1m.zip. If you want to read about the data and how it is setup, feel free to check out this file: http://files.grouplens.org/papers/ml-1m-README.txt. In the ml-1m.zip file, you should see three files inside: movies.dat, ratings.dat, users.dat.
Let’s get that into our Cloudera Quickstart Docker image and load them into HBase.
If you aren’t using the Digital Ocean version of the tutorial, you can volume mount the directory that you saved your data into your quickstart docker image by running this command:
docker run --hostname=quickstart.cloudera --privileged=true -t -i -p8888:8888 -v /place/you/saved:/inside/your/container cloudera/quickstart:latest /usr/bin/docker-quickstart bash
Once you have the Digital Ocean instance spun up and resized, let’s run some commands to all get on the same page.
docker-machine scp -r ./Downloads/ml-1m/ root@docker-sandbox:/tmp
This command will get the data into your /tmp folder on the Digital Ocean server.
docker-machine ssh docker-sandbox
docker run --hostname=quickstart.cloudera --privileged=true -t -i -p8888:8888 -v /tmp:/dataset cloudera/quickstart:latest /usr/bin/docker-quickstart bash
Let’s put the data into HDFS quickly with some data prep.
[root@quickstart ml-1m]# hdfs dfs -mkdir -p /moviedata/ratings
[root@quickstart ml-1m]# hdfs dfs -mkdir -p /moviedata/movies
[root@quickstart ml-1m]# hdfs dfs -mkdir -p /moviedata/users
[root@quickstart /]# sed 's/::/#/g' /dataset/ml-1m/movies.dat >movies.t
[root@quickstart /]# sed 's/::/#/g' /dataset/ml-1m/ratings.dat > ratings.t
[root@quickstart /]# sed 's/::/#/g' /dataset/ml-1m/users.dat > users.t
[root@quickstart /]# hdfs dfs -put movies.t /moviedata/movies
[root@quickstart /]# hdfs dfs -put ratings.t /moviedata/ratings
[root@quickstart /]# hdfs dfs -put users.t /moviedata/users
Awesome! Let's put aside this data for now, and just get some practice interacting with HBase. We will come back to this data that we just put into HDFS shortly.
HBase Shell Examples
First thing's first. Let’s jump into the HBase shell by typing:
hbase shell
Let’s create a table now.
create 'tabletest', 'columnfamily1'
0 row(s) in 1.5990 seconds

=> Hbase::Table - tabletest
In the last command, we specify the create command with tabletest as the name of our table with columnfamily1 as a column family name — the command is create '<<table name>>', '<<column family>>'.
Use the list command to print out some information about your table:
list 'tabletest'
hbase(main):002:0> list 'tabletest'
TABLE
tabletest
1 row(s) in 0.0270 seconds
=> ["tabletest"]
Naturally we should put some data into this table to get a real feel for it. Let’s just throw some test data into it.
hbase(main):004:0> put 'tabletest', 'firstrow', 'columnfamily1','testValue1'
0 row(s) in 0.0130 seconds
hbase(main):005:0> put 'tabletest', 'secondrow', 'columnfamily1','testValue2'
0 row(s) in 0.0150 seconds
hbase(main):006:0> put 'tabletest', 'thirdrow', 'columnfamily1','testValue3'
0 row(s) in 0.0060 seconds
This format looks a little different.
The first part put tabletest is just telling HBase which table we want to put the data into.
firstrow denotes where in the table we want to put the data. Remember random read and writes, so we have to specify exactly where to store the data.
columnfamily1 refers to which column family we are going to be putting data into
Finally we put the value of testvalue.
It's important to note here that HBase doesn't support multiple columns in a single statement. So if you had a table with a column family with two columns under it, the put statement to add to both columns would look like the following:
PUT 'tableName', 'columnFamily:column1', 'data'
PUT 'tableName', 'columnFamily:column2', 'data'
In other words, the : is used to specify the column names.
Let’s query this data and see if it what we expected.
hbase(main):007:0> scan 'tabletest'
ROW            COLUMN+CELL
firstrow       column=columnfamily1:, timestamp=1518293802613, value=testValue1
secondrow      column=columnfamily1:, timestamp=1518293817887, value=testValue2
thirdrow       column=columnfamily1:, timestamp=1518293828203, value=testValue3
3 row(s) in 0.0610 seconds
Perfect. Notice that when we do a scan it brings up all values in the table. It shows us the ROW and COLUMN and CELL. Notice we just put one value into the column family but there are now two. The timestamp just magically appeared. That is because every row has a timestamp.
Let’s try and get just one row of the table.
get 'tabletest', 'thirdrow'
COLUMN               CELL
columnfamily1:       timestamp=1518293828203, value=testValue3
1 row(s) in 0.0260 seconds
This shows just the column and the cell.
A cell is the value that is at the intersection of a row and a column. To find a cell you need to know the column, row, and version (timestamp) to get the correct value. Pretty cool.
It shows us the column family and the cell with the value and the timestamp. The timestamp should match above because that timestamp is created when data is put into the row so it shouldn’t change unless something happens to that row.
Something a little different with HBase is the disable and enable commands. If you want to update the settings of a table or drop a table, you need to disable the table. Then when you are ready to use it again, enable the table. Let’s try it.
First let’s do a describe on the table.
describe 'tabletest'
Table tabletest is ENABLED

tabletest

COLUMN FAMILIES DESCRIPTION

{NAME => 'columnfamily1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1'
, COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_
MEMORY => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.0490 seconds
Notice the table is ENABLED.
disable 'tabletest'
describe 'tabletest'
Table tabletest is DISABLED
tabletest
COLUMN FAMILIES
{NAME => 'columnfamily1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1'
, COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_
MEMORY => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.0580 seconds
It’s now disabled. Let’s try and put some data into it and see what happens.
put 'tabletest', 'fourthrow', 'columnfamily1', 'testValue4'
ERROR: Failed 1 action: NotServingRegionException: 1 time,
Got an error. Let’s enable that table and try and put data into it.
hbase(main):013:0> enable 'tabletest'
0 row(s) in 1.2850 seconds

hbase(main):014:0> put 'tabletest', 'fourthrow', 'columnfamily1','testValue4'
0 row(s) in 0.0160 seconds

hbase(main):015:0> get 'tabletest', 'fourthrow'
COLUMN             CELL
columnfamily1:     timestamp=1518295072544, value=testValue4
1 row(s) in 0.0240 seconds
It worked. Awesome job!! Let’s try and drop the table.
drop 'tabletest'
ERROR: Table tabletest is enabled. Disable it first.
The table is enabled so it won’t let us drop it. Perfect. Go ahead and drop that table on your own. Once you get done with that, go ahead and type quit and press enter and it’ll get you back to the main command line.
Loading Data From HDFS into HBase using Pig
Let’s get back to that data that we put into HDFS and let’s put it into HBase using Pig. First thing that we need to do is go back into the HBase shell and make three tables.
create 'users', 'userdata'
0 row(s) in 1.4860 seconds
=> Hbase::Table - users
create 'ratings', 'ratingsdata'
0 row(s) in 1.5700 seconds
=> Hbase::Table - ratings
create 'movies', 'moviedata'
0 row(s) in 1.5380 seconds
=> Hbase::Table - movies
Great! Let’s create a script to load the data using pig. Copy the following code into a file called loadHbase.pig.
movies = LOAD '/moviedata/movies/movies.t' USING PigStorage(',') AS (movieid:int, title:chararray, genres:chararray);

STORE movies INTO 'hbase://movies' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('moviedata:title moviedata:genres');

ratings = LOAD '/moviedata/ratings/ratings.t' USING PigStorage(',') AS (userid:int, movieid:int, rating:int, tstamp:chararray);

STORE ratings INTO 'hbase://ratings' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('ratingsdata:movieid ratingsdata:rating ratingsdata:tstamp');

users = LOAD '/moviedata/users/users.t' USING PigStorage(',') AS (userid:int, gender:chararray, age:int, occupation:int, zipcode:chararray);

STORE users INTO 'hbase://users' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('userdata:gender userdata:age userdata:occupation userdata:zipcode');
We are loading some data from HDFS and using a special class (hbase://) to store data into HBase for each of the files. The command to run this script is a little different because we need to add some jars (Java ARchives) to the classpath of Pig that contain the special class that we used to store the data into HBase. Run the following command and watch the magic happen.

Job Stats (time in seconds):
JobId        Maps        Reduces        MaxMapTime        MinMapTIme        AvgMapTime        MedianMapTime        MaxReduceTime        MinReduceTime        AvgReduceTime        MedianReducetime        Alias        Feature        Outputs
job_1518379779375_0001        1        0        80        80        80        80        n/a        n/a        n/a        n/a        ratings        MAP_ONLY        hbase://ratings,
job_1518379779375_0002        1        0        21        21        21        21        n/a        n/a        n/a        n/a        movies        MAP_ONLY        hbase://movies,
job_1518379779375_0003        1        0        20        20        20        20        n/a        n/a        n/a        n/a        users        MAP_ONLY        hbase://users,

Input(s):
Successfully read 1000209 records (21593882 bytes) from: "/moviedata/ratings/ratings.t"
Successfully read 3883 records (163918 bytes) from: "/moviedata/movies/movies.t"
Successfully read 6040 records (110582 bytes) from: "/moviedata/users/users.t"

Output(s):
Successfully stored 1000209 records in: "hbase://ratings"
Successfully stored 3883 records in: "hbase://movies"
Successfully stored 6040 records in: "hbase://users"

Counters:
Total records written : 1010132
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1518379779375_0001
job_1518379779375_0002
job_1518379779375_0003
Awesome. The data should now be loaded into HBase. Let’s jump into the HBase shell and do some querying of the data.
scan 'movie
 988       column=moviedata:title, timestamp=1518381804300, value=Grace of My Heart (1996)
 989       column=moviedata:genres, timestamp=1518381804300, value=Drama
 989       column=moviedata:title, timestamp=1518381804300, value=Schlafes Bruder (Brother of Sleep) (1995)
 99        column=moviedata:genres, timestamp=1518381803685, value=Documentary
 99        column=moviedata:title, timestamp=1518381803685, value=Heidi Fleiss: Hollywood Madam (1995)
 990       column=moviedata:genres, timestamp=1518381804300, value=Action|Adventure|Thriller
 990       column=moviedata:title, timestamp=1518381804300, value=Maximum Risk (1996)
 991       column=moviedata:genres, timestamp=1518381804300, value=Drama|War
 991       column=moviedata:title, timestamp=1518381804300, value=Michael Collins (1996)
 992       column=moviedata:genres, timestamp=1518381804301, value= The (1996)
 992       column=moviedata:title, timestamp=1518381804301, value=Rich Man's Wife
 993       column=moviedata:genres, timestamp=1518381804301, value=Drama
 993       column=moviedata:title, timestamp=1518381804301, value=Infinity (1996)
 994       column=moviedata:genres, timestamp=1518381804301, value=Drama
 994       column=moviedata:title, timestamp=1518381804301, value=Big Night (1996)
 996       column=moviedata:genres, timestamp=1518381804301, value=Action|Drama|Western
 996       column=moviedata:title, timestamp=1518381804301, value=Last Man Standing (1996)
 997       column=moviedata:genres, timestamp=1518381804301, value=Drama|Thriller
 997       column=moviedata:title, timestamp=1518381804301, value=Caught (1996)
 998       column=moviedata:genres, timestamp=1518381804302, value=Action|Crime
 998       column=moviedata:title, timestamp=1518381804302, value=Set It Off (1996)
 999       column=moviedata:genres, timestamp=1518381804302, value=Crime
 999       column=moviedata:title, timestamp=1518381804302, value=2 Days in the Valley (1996)
3883 row(s) in 14.4740 seconds
There are a lot of rows there and scan will show you them all. I just put the last couple. Great job! You’ve successfully loaded some data into HBase using Pig.
Let’s do a quick query to see how the data is setup in HBase.
get 'movies', 77
Are
COLUMN                 CELL
moviedata:genres       timestamp=1518381803660, value=Documentary
moviedata:title        timestamp=1518381803660, value=Nico Icon (1995)
2 row(s) in 0.1390 seconds


text:
# Avoid expect(stub)… (no-expect-for-stub)

Disallow usage of `expect()` with a [Sinon-Chai
chainer](https://docs.cypress.io/guides/references/assertions#Sinon-Chai)
(e.g., `expect(stub).to.have.been.calledOnce`) preferring the usage of
`should()`.

Some problems reported by this rule are automatically fixable by the
`--fix` command line option. Please be aware that fixing the code with
this rule requires that the stub is aliased (using the [`.as()`
Cypress command](https://docs.cypress.io/api/commands/as)): if that's
not the case, you will have to add an alias manually for the test to
pass.

## Rule Details

This rule makes code easier to read when `expect()` is used on a stub
with [Sinon-Chai chainer
chainer](https://docs.cypress.io/guides/references/assertions#Sinon-Chai).

Example of **incorrect** code for this rule:

```js
let onBudgetChanged = cy.stub().as("onBudgetChanged");
// … do something with onBudgetChanged…

cy.get("foo").click();

cy.then(() => {
	expect(onBudgetChanged).to.be.called;
});
```

Note that the `expect()` call is wrapped in a `cy.then()`. This is
mandatory if the `expect()` is to be called after the Cypress commands
above it which is usually the case.

Example of **correct** code for this rule:

```js
// do something with cy.stub().as("onBudgetChanged")
cy.get("foo").click();
cy.get("@onBudgetChanged").should("be.called");
```

## When Not To Use It

- When you don't mind having code that is harder to read than necessary.
- When you don't want to add aliases to the stubs/spies.

repository: codacy/codacy-eslint

repo_name: codacy/codacy-eslint


text:
# Uniswap Arbitrage Flash Swap Contract

You want to play with "millions" on the blockchain then try https://uniswap.org/docs/v2/smart-contract-integration/using-flash-swaps/. This contract is able to arbitrage between DEX on the blockchain without providing any own capital.
Basically is usable on the **Binance Smart Chain** and provides callbacks for common DEX, feel free to extend.

**This is not a full project, build your or own infrastructure around it!**

## Links

 * https://github.com/yuyasugano/pancake-bakery-arbitrage
 * https://github.com/Uniswap/uniswap-v2-periphery/blob/master/contracts/examples/ExampleFlashSwap.sol

## Transactions

![transactions](examples-min.png "Transactions failed and success")

## Improvements

All examples are more or less overengineered for a quick start, so here we have a KISS solution.

 * Reduce overall contract size for low gas fee on deployment
 * Provide a on chain validation contract method to let nodes check arbitrage opportunities
 * Stop arbitrage opportunity execution as soon as possible if its already gone to reduce gas fee on failure
 * Interface callbacks for common DEX on "Binance Smart Chain" are included

## Infrastructure

Basically arbitrage opportunity dont last long, your transaction must make it into the next block. So you have <3 seconds watching for opportunities, decide and execute transaction. Sometimes there are also a chance to 2-3 have block, see example below.

```
[7920960] [6/1/2021, 5:50:37 PM]: alive (bsc-ws-node.nariox.org) - took 308.42 ms
[7920991] [6/1/2021, 5:52:09 PM]: [bsc-ws-node.nariox.org] [BAKE/BNB ape>bakery] Arbitrage opportunity found! Expected profit: 0.007 $2.43 - 0.10%
[7920991] [6/1/2021, 5:52:09 PM] [bsc-ws-node.nariox.org]: [BAKE/BNB ape>bakery] and go:  {"profit":"$1.79","profitWithoutGasCost":"$2.43","gasCost":"$0.64","duration":"539.35 ms","provider":"bsc-ws-node.nariox.org"}
[7920992] [6/1/2021, 5:52:13 PM]: [bsc-ws-node.nariox.org] [BAKE/BNB ape>bakery] Arbitrage opportunity found! Expected profit: 0.007 $2.43 - 0.10%
[7920992] [6/1/2021, 5:52:13 PM] [bsc-ws-node.nariox.org]: [BAKE/BNB ape>bakery] and go:  {"profit":"$1.76","profitWithoutGasCost":"$2.43","gasCost":"$0.67","duration":"556.28 ms","provider":"bsc-ws-node.nariox.org"}
[7921000] [6/1/2021, 5:52:37 PM]: alive (bsc-ws-node.nariox.org) - took 280.54 ms
```

### Requirements / Hints

 * You have a time window of 1000ms every 3 seconds (blocktime on BSC) and you should make it into the next transaction
 * Websocket connection is needed to listen directly for new incoming blocks
 * Public provided Websocket are useless `bsc-ws-node.nariox.org` simply they are way behind notify new blocks
 * Use a non public provider; or build your own node (light node helps) and better have multiple owns; let the fastest win
 * Spread your transaction execution around all possible providers, first one wins (in any case transactions are only execute once based on `nonce`)
 * Find suitable pairs with liquidity but not with much transaction
 * You can play with full pair liquidity, but dont be too greedy think of a price impact you would have
 * Common opportunities are just between 0,5 - 1%
 * Do not estimate transaction fee, just calculate it once and provide a static gas limit. Simply its takes to long
 * There is block parameter until the transaction is valid, so you can abort execution eg after +3 blocks
 * Payback is directly calculated by calling the foreign contracts so its project independent (no hardcoded fee calculation)
 * The profit is transferred to the owner / creator of the contract :)

### Function

The contract is plain and simple [contracts/Flashswap.sol] some basic hints:

Check arbitrage opportunity between DEX. Read only method the one blockchain

```
    function check(
        address _tokenBorrow, // example: BUSD
        uint256 _amountTokenPay, // example: BNB => 10 * 1e18
        address _tokenPay, // example: BNB
        address _sourceRouter,
        address _targetRouter
    ) public view returns(int256, uint256) {
```

Starts the execution. You are able to estimate the gas usage of the function, its also directly validating the opportunity. Its slow depending on connected nodes.

```
    function start(
        uint _maxBlockNumber,
        address _tokenBorrow, // example BUSD
        uint256 _amountTokenPay,
        address _tokenPay, // our profit and what we will get; example BNB
        address _sourceRouter,
        address _targetRouter,
        address _sourceFactory
    ) external {
```

As all developers are lazy and just forking projects around without any rename a common implementation is possible. Basically the pair contract call the foreign method of the contract. You can find them the naming in any pair contract inside the `swap()` method.

Example: https://bscscan.com/address/0x0eD7e52944161450477ee417DE9Cd3a859b14fD0#code: `if (data.length > 0) IPancakeCallee(to).pancakeCall(msg.sender, amount0Out, amount1Out, data);`

Extend method if needed:

```
# internal callback
function execute(address _sender, uint256 _amount0, uint256 _amount1, bytes calldata _data) internal

# foreign methods that get called
function pancakeCall(address _sender, uint256 _amount0, uint256 _amount1, bytes calldata _data) external
function uniswapV2Call(address _sender, uint256 _amount0, uint256 _amount1, bytes calldata _data) external
```

### Run it

Install node.js packages and compile a smart contract code
```sh
npm install
truffle compile
```

```sh
truffle migrate --network mainnet
```

Its not a full infrastructure, but a working workflow, if you deploy the contract.

``` bash
cp env.template .env # replace values inside ".env"
node watcher.js
```

```
started: wallet 0xXXXX - gasPrice<PHONE_NUMBER> - contract owner: 0xXXXX
[bsc-ws-node.nariox.org] You are connected on 0xXXXX
[8124531] [6/8/2021, 7:53:20 PM]: [bsc-ws-node.nariox.org] [BUSD/BNB pancake>panther] Arbitrage checked! Expected profit: -0.015 $-4.99 - -0.15%
[8124532] [6/8/2021, 7:53:21 PM]: [bsc-ws-node.nariox.org] [BUSD/BNB pancake>panther] Arbitrage checked! Expected profit: -0.015 $-4.99 - -0.15%
[8124533] [6/8/2021, 7:53:24 PM]: [bsc-ws-node.nariox.org] [BUSD/BNB pancake>panther] Arbitrage checked! Expected profit: -0.014 $-4.71 - -0.14%
[8124534] [6/8/2021, 7:53:27 PM]: [bsc-ws-node.nariox.org] [BUSD/BNB pancake>panther] Arbitrage checked! Expected profit: -0.014 $-4.61 - -0.14%
```

#### Hints

 * Designed to have multiple chain connectivities, play with some non public providers to be faster then the public once. Its all designed as "first win"

repository: mindfulme/uniswap-arbitrage-flash-swap

repo_name: mindfulme/uniswap-arbitrage-flash-swap


text:
# Data_Analysis_RMS_Titanic

Data_Analysis_RMS_Titanic is a Jupyter Notebook data analysis of the passengers on the RMS Titanic. The notebooks analysis passengers survivability rate depending on age, sex and cabin class. The data was not 100% clean so I had to do some data wrangling and cleaning to properly prepare and ETA on the data.

## Installation

No installation required, just open with any IDE that supports Jupyter notebooks, or click the link above the notebook to open with Google Colabs ( Preferred)

## Usage
If you want to run the file, on the second block of code, please comment out.

```titanic_df = pd.read_csv('/content/drive/MyDrive/general/Assignment 3/titanic_data.csv')```

and uncomment out.

```titanic_df = pd.read_csv('https://slack-files.com/TTQ6X9H24-F01M0DZAMEW-1afb6f2ac4')```

## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

## License
[MIT](https://choosealicense.com/licenses/mit/)