text:
# Indexer App

The purpose of the Indexer app is to:
- consume the Main Container's ChangeFeed (container name is configured via `EQUINOX_COSMOS_CONTAINER`)
- ensure all derived information maintained in the Views Container is kept in sync (`EQUINOX_COSMOS_VIEWS`)

It also implements some additional functions (all of which are based on the consumption of Equinox events from the Main Container via the ChangeFeed).

# Command: `index`

:warning: **Writes to the Views Container; Does not write to the Main Store Container**.

Reacts to each event written to the main store by updating all derived information held in the Views Container.

From the base of the repo, it's invoked as follows:

    dotnet run --project Indexer -- -g defaultIndexer index

- `defaultIndexer` is the Processor Name (equivalent to the notion of a Consumer Group in many messaging technologies)

## Common commandline overrides

The following overrides can be useful:

- `-w` overrides the streams parallelism limit (default: `8`). In general, Cosmos rate limiting policy and request latency means increasing beyond that is not normally useful. A potential exception is when retraversing due to the RU consumption from writes being avoided (i.e. correct efficient idempotent processing).
- `-r` overrides the maximum read-ahead limit (default: `2`).
  - In rare cases (where the I/O involved in processing events is efficient and/or the items are being traversed at a high throughput) increasing the read ahead limit can help to ensure that there's always something for the processor 'threads' to work on
  - reading ahead can help throughput in some cases: multiple events from the same stream are handled as a single span of events to the maximum degree possible, including events from batches beyond the one that's currently in progress.

There are other settings that relate to the source ChangeFeed (i.e. you specify them after the `index` subcommand name):

-  `-b`: adjusts the maximum batch size (default: `100` items).

   - Lower values induce more frequent checkpointing, and reduce the RU cost of each individual read.
   - Higher values increase the RU cost of each batch read and the number of events that need to be re-processed if a partition is re-assigned, or the Index App is restarted.

----

# Indexing logic fundamentals

Over time, the number of events in the system will keep growing. But, you still want to be able to reconstitute the content and/or add a new variant at any time.

This has the following implications:
- processing should work on a batched basis
- no permanent state should be held in the Views Container; it's literally just a structure that's built up to support the current needs of the system as a whole in terms of providing for efficient traversal of data to support the use cases and performance needs of both read and write paths.

## Key principle: The views Container is ephemeral

In a deployed instance, indexing typically runs as live loop working from the tail of the ChangeFeed.

However, it's important to note that in a well written event-sourced system that it should be considered absolutely normal to, at the drop of a hat be able to regenerate the views and/or provision a slightly tweaked version of the existing format.

It is easy, and should remain easy to validate that a given indexing function works correctly and efficiently: blow away the views container and retraverse on the desktop or in a sandbox environment.

:bulb: NOTE this is one reason why using Kafka or some form of Service Bus to consume events indirectly is problematic; you need to re-emit all the events to the 'bus'

## Example: Completely regenerating the Views Container

The only real 'state' in the system as a whole is the ChangeFeed processor checkpoint (identified by the Processor Name in the `-g` commandline argument)

If there's ever doubt about the integrity of the state in the Views Container, it should be possible to reconstitute its content at any time by:

1. delete the Views Container
2. re-initialize it with `eqx init`
3. run `index` over all the events from the start of time

## Example: capturing fields from an event into the View Container state that were not previously captured

In general, Indexes should not speculatively hold information that's not being used - writing more than is necessary consumes space and increases RU consumption.

Instead, where a new feature and/or a tweak to an existing one requires previously un-indexed information, one can simply use an Expand and Contract strategy:
1. Expand to add the extra information
   - change the `Category` for the view (i.e. if it was `$TodoIndex`, and a new field is needed, change the `Category` to `$TodoIndex2`)
   - add the logic to populate the relevant state
   - run the `index` subcommand (with a fresh _Processor Name_)
2. Write consumer logic
3. Deploy. This can be done in two common ways:
   - (if the consumption logic is not behind a feature flag) as part of the deploy, run the `index` before the system goes live
   - (if the consumption logic is behind a feature flag):
     - after the deploy, run an `index` operation
     - when the index has caught up, enable the feature flag that uses the information it provides

----

# Command: `snapshot`

:warning: **Writes to the Main Store Container; Does not write to the Views Container**.

This ancillary command is hosted within the Indexer App as it involves the same elements as the normal indexing behavior (reading from the ChangeFeed, using the domain fold logic to establish the state per stream, writing to the store as configured).

The key difference compared to the `index` subcommand is that it writes to the Main Store Container (as opposed to the Views Container).

From the base of the repo, it's invoked as follows:

    dotnet run --project Indexer -- -g defaultSnapshotter snapshot

- `defaultSnapshotter` is the _Processor Name_ (aka Consumer Group Name).
  - It should be different to the one used for the Indexing behavior (`defaultIndexer` in the example above).
  - To trigger full retraversal (as opposed to only visiting streams that have been updated since the last `snapshot` traversal, you'll want to use a fresh checkpoint name per run, see [resetting checkpoints](https://github.com/jet/propulsion/blob/doc/DOCUMENTATION.md#resetting-checkpoints))

## Implementation notes

Replaces the normal/default Handler with one that:
1. **loads the State of a stream** with the normal application-level `fold` logic appropriate for its category, noting whether the most recent Snapshot Event Type was present. Having completed the load, it does one of the following:
    - (if the state was derived from the latest Snapshot Event Type): _no further action is necessary_
    - (if no snapshot is present) Equinox builds the state in the normal manner by `fold`ing all the events
2. **(if the state was loaded based on the snapshot)**:
    - no writes take place
    - the Equinox Cache will retain the version and state (and via step `4` below, become Propulsion's updated 'write position' for that stream)
    - subsequent reads of events from the ChangeFeed will typically be discarded (as the `Index` will be less than the `Version` that became that stream's write position)

   :bulb: **NOTE:** if a fresh event enters the store after the stream has been visited in a given traversal, but the writer did not write a 'latest' snapshot alongside, the snapshotter would react by immediately adding the snapshot. (For avoidance of doubt, its extremely unlikely that this will work out better in terms of RU consumption and/or latency, as the RU impact of a write is a function of the original size plus the updated size)

   :bulb: **ASIDE:** If a [purge](https://github.com/jet/propulsion/blob/doc/DOCUMENTATION.md#purging) causes the write position to be jettisoned, the cache entry comes into play: the etag in the cached state will likely result in a 304 not modified read roundtrip
3. **(if the state was not derived from the Snapshot Event Type)**
    - yields a `Snapshotted` event from the `decide` function (to trigger a store Sync operation rather than the normal short-circuit if there are no events to write)
    - use `AccessStrategy.Custom` to `transmute` this tentative write:
        - _from_: the proposed snapshot event that the `decide` function yielded in the preceding step to trigger the store roundtrip (which would ordinarily we written as part of the streams permanent events, which would be bad)
        - _to_: not appending any events, but instead replacing the 0 or many current unfolds with the single desired snapshot

    :bulb: **NOTE:** the `CosmosStoreContext` used to do the processing is configured differently to how one normally would for this to work well; `tipMaxEvents` is set to a number that's effectively infinite:

      - e.g. if you set it to `20` and tried to update the snapshot of a stream with 30 events, a 'calve' would be triggered, resulting in:

        1. writing a calf Item containing the `30` ejected events
        2. updating the Tip to not have any `e`vents
        3. updating the Tip to have the updated `Snapshotted` event as the single entry in the `u`nfolds field
4. yields a `StreamResult.OverrideNextIndex` based on the observed version of the stream.

  :bulb: Normally, the first visit to the stream is the only one. The exception is if a new write takes place before the processing reaches the end of the ChangeFeed.

In other aspects (from the perspective of how it traverses the events, checkpoints progress, manages concurrency and/or reading ahead), this subcommand is equivalent to the  `index` subcommand; the same overrides apply.

## Versioning events

All types used in the body of an event need to be able to be safely round-tripped to/from the Event Store. While events are written exactly once, they will live in the store forever, so need to be designed carefully.

As a result, you want to follow the normal principles of doing changes to message contracts safely:
- the most obvious thing not to do is rename fields
- removing fields (as long as you know no reader anywhere is going to have an exception in their absence)
- adding new fields as `option`s (assuming you can do something workable in the case where the value is `None)

That said, the most important thing in defining the contract for a normal event sourced event is the semantic intent; if the intent/meaning behind an event is changing, it's normally best to add a new case to the `Event` union to reflect that.

:bulb: Whenever an Event is no longer relevant to the system as a whole, it's safe to remove it from the `type Event` union - Event Types that don't have cases in the union are simply ignored.

There's a wide variety of techniques and established strategies, most of which are covered in https://github.com/jet/fscodec#upconversion

## Versioning snapshots

The `Snapshotted` event in the `type Event =` union is explicitly tagged with a `Name`:

```fsharp
| [<DataMember(Name = "Snapshotted")>] Snapshotted of Snapshotted
```

As with any Event, all types used in the body (i.e. the entire tree of types used in the `type Snapshotted` needs to be able to be roundtripped safely to/from the Store).

The same rules above re Versioning Events apply equally.

The critical differences between 'real Events' and `Snapshotted` events are:
- Snapshotted event are stored in the `u`nfolds field of the tip, and get replaced with a fresh version every time an event is updated
- It's always theoretically safe to remove the `Snapshotted` case from the Union; no data would be lost. However it does mean that the state will need to be produced from the events, which will likely result in higher latency and RU consumption on the initial load (the Cache means you generally only pay this price once).

The following general rules apply:

- CONSIDER per aggregate whether a `Snapshotted` event is appropriate. It may make sense to avoid having a snapshotted stream if streams are short, and you are using Events In Tip mode
  - the cache will hold the state, and reads and writes don't pay the cost of conveying a snapshot
  - the additional cost of deserializing and folding the events is extremely unlikely to be relevant when considering the overall I/O costs
- ALWAYS change the `Name` _string_ from `Snapshotted` to `Snapshotted/2`, `Snapshotted/3` etc whenever a breaking change is needed - that removes any risk of a failure to parse the snapshot event body due to changes in the representation.

Given the above arrangements, you can at any time ensure snapshots have been updated in a given environment by running the following command against the Store:

  ```bash
  # establish prod env vars for EQUINOX_COSMOS_CONNECTION, EQUINOX_COSMOS_DATABASE, EQUINOX_COSMOS_CONTAINER etc
  dotnet run --project Indexer -- -g deployYYMMDD snapshot
  ```
----

# Command: `sync`

:warning: **Writes to the nominated 'Export' Container; Does not write to the `source` (or the Views Container)**.

Uses the ChangeFeed to export all the events from the Main Store Container to a nominated target Equinox Container.

This ancillary command is hosted within the Indexer App despite involving less of the infrastructure required for the `index` or `snapshot` subcommands as it is commonly used in conjunction with those other subcommands.

In addition to being useful on a one-off basis, it can also be run to maintain a constantly in sync read replica of a given Container.

From the base of the repo, it's invoked as follows:

    eqx init -A cosmos -c Export # once; make an empty container to Sync into
    dotnet run --project Indexer -g syncExport sync -c Export source
    eqx stats -A cosmos # get stats for Main store
    eqx stats -A cosmos -c Export # compare stats for exported version

- `syncExport` is a _Processor Name_ (aka Consumer Group Name) that is different to the one used for the Indexing behavior (`defaultIndexer` in the example above)
- `source` is a mandatory subcommand of the `sync` command; you can supply the typical arguments to control the ChangeFeed as one would for `index` or `snapshot`

:bulb: **NOTE** The Sync process only writes events. Each write will replace any snapshots stored in the `u`nfolds space in the Tip with an empty list. This means your next step after 'exporting' will frequently be to point the `EQUINOX_COSMOS_CONTAINER` at the exported version and then run the `snapshot` subcommand (while the system will work without doing so, each initial load will be less efficient).

:bulb: **ASIDE:** You could run the `sync` concurrently with the `snapshot`; the outcome would be correct, but it would typically be inefficient as they'd be duelling for the same RU capacity. Also, depending on whether or not the source events are stored compactly (using Events In Tip), the streams may be written to many times in the course of the export (and each of those appends would mark that stream 'dirty' and hence trigger another visit by the `snapshot` processing)

## Example: exporting to a local database from a production environment

```bash
EQUINOX_COSMOS_CONNECTION=<connection string>
EQUINOX_COSMOS_DATABASE=testdb
EQUINOX_COSMOS_CONTAINER=app # note we override this in most cases below with `-c test`
EQUINOX_COSMOS_VIEWS=app-views
SOURCE_COSMOS_KEY=<copy read only key from portal>
propulsion init -A cosmos -c app # creates app-aux
eqx init -A cosmos -c app-views
eqx init -A cosmos -c test
# copy from prod datastore into a temporary (`test`)
dotnet run --project Indexer -- -w 64 -g defaultSync `
    sync -s $EQUINOX_COSMOS_CONNECTION -c test -a app-aux `
    source -s $SOURCE_COSMOS_KEY -d productiondb -c app -b 1000
# apply snapshots to the exported database (optional; things will still work without them)
dotnet run --project Indexer -- -g defaultSnapshotter `
    snapshot -c test -a app-aux # note need to specify app-aux, or it would look for test-aux
# index the data into the app-views store based on the test export
dotnet run --project Indexer -- -g defaultIndexer `
    index -c test -a app-aux -b 9999
```

- `-w 64`: override normal concurrency of 8
- `-b 9999`: reduce ChangeFeed Reader messages

repository: jet/dotnet-templates

gha_language: F#

repo_name: jet/dotnet-templates


text:
## 0.5.1 / 2015-11-03

  * fix IE/Edge InvalidAccessError [#37](https://github.com/treojs/treo/pull/37) [@unkillbob](https://github.com/unkillbob)
  * deps<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>  * use<EMAIL_ADDRESS>to prevent size blow [#22](https://github.com/component/type/issues/22)

## 0.5.0 / 2015-04-01

  * add support for multi-field indexes [#24](https://github.com/treojs/treo/issues/24)
  * add support for web-worker environment
  * deps<EMAIL_ADDRESS>  * add LICENSE file

## 0.4.3 / 2015-03-09

  * add support for iOS 8.1.3 [#26](https://github.com/treojs/treo/pull/26)

Thanks: [@mariusk](https://github.com/mariusk)

## 0.4.2 / 2015-02-13

  * fix the iOS issue on IndexedDBShim [#21](https://github.com/treojs/treo/pull/21)

Thanks: [@capsula4](https://github.com/capsula4)

## 0.4.1 / 2015-01-17

  * use [idb-range](https://github.com/treojs/idb-range), and remove treo.range
  * handle `onversionchange` automatically and close db [#17](https://github.com/treojs/treo/issues/16)

## 0.4.0 / 2015-01-16 [PR](https://github.com/treojs/treo/pull/18)

  * pass treo as second argument to `db.use()`
  * add `schema.dropStore()` and `schema.dropIndex()`
  * fix iOS 8 & Safari 7.0.6, IE10 & IE11 support
  * `put` passes the key of the created/updated record to callback

Thanks: [@unkillbob](https://github.com/unkillbob) for fixing websql polyfill.

## 0.3.0 / 2014-09-24

  * add npm's files option for smaller tarbal
  * use dist folder for bower and component
  * add plugins/treo-websql pass same tests as real IndexedDB
  * add plugins/treo-promise for better promises support
  * add es6-generators example
  * update Readme #11

## 0.2.0 / 2014-09-08

  * use browserify for build instead of component(1)
  * fix npm support
  * key || keyPath, increment || autoIncretement, multi || multiEntry
  * (fix #12) schema.addStore({ increment: true }) to support autoIncrement

## 0.1.0 / 2014-07-23

  * initial release.
    It used to call indexed, and now this name exists only in git history
    as a prove of evolutionary thinking about this problem.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# SortingTimeComparisons

## About my code
My program asks to user input size and after that it writes running times in a .csv file. (It calculates mergesort and bubblesort in one  step. In mergesort, program creates an another sorted array. So bubblesort will sort unsorted array.)

## Tested in
`g++ version: 7.2.0 (ubuntu 16.04)`
`g++ version: 4.9.2 (windows 10)`
`IDE: Eclipse c++`

## Running Commands
`g++ -c -std=c++11 main.cpp -o main.o`
`g++ main.o -o main.exe`
`./main.exe`

## Asymptotic Upper Bound And Running Time

Mergesort is a divide and conquer algorithm which means the input size will be halved in each iteration. The entire input will be iterated thought, and each iteration must occur O(log(n)) times. n items iterated log(n) times and it gives us O(nlog(n)) as an upper bound.
Also, The worst case for Bubble Sort is in the case of the array is already sorted in descending order. In this case, in first iteration it have to make comparison n element, then n-1 and so on. Until the last iteration.
For BubbleSort;
n + (n-1) + (n-2) + (n-3)  ….  + 1 = [n * (n+1)] / 2 = n^2/2 + n/2 which means O(n^2)

## Which One is Better?
Bubblesort is fast when sorting sequential data which is already sorted. Also bubblesort should be more effective than merge sort in the case which you have limited memory usage. Because mergesort requires a second array with the same size as the array to be sorted, that’s why it doubles the memory requirements.
Extra
Also you can check my map implementation for this homework

https://github.com/Omerdogan3/Distances-to-Istanbul

### Result
https://omerdogan3.github.io/Distances-to-Istanbul/public/index.html


text:
# Token caching in MSAL Node

When MSAL Node acquires a token, it caches it in memory for future usage. MSAL Node manages the token lifetime and refreshing for you. APIs like `acquireTokenSilent()` retrieves the access tokens from the cache for a given account:

```javascript
const msal = require('@azure/msal-node');

// Create msal application object
const cca = new msal.ConfidentialClientApplication({
    auth: {
        clientId: "Enter_the_Application_Id_Here", // e.g. "b1b60dca-c49d-496e-9851-xxxxxxxxxxxx" (guid)
        authority: "https://login.microsoftonline.com/Enter_the_Tenant_Info_Here", // e.g. "common" or your tenantId (guid)
        clientSecret: "Enter_the_Client_Secret_Here" // obtained during app registration
    }
});

/**
* login* and acquireToken* APIs return an account object containing "homeAccountId"
* you should keep a record of this in your app and use it later on when calling acquireTokenSilent
* For more, see: https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-node/docs/accounts.md
*/
const someUserHomeAccountId = "Enter_User_Home_Account_Id";

const msalTokenCache = cca.getTokenCache();
const account = await msalTokenCache.getAccountByHomeId(someUserHomeAccountId);

const silentTokenRequest = {
    account: account,
    scopes: ["user.read"],
};

cca.acquireTokenSilent(silentTokenRequest).then((response) => {
    // do something with response
}).catch((error) => {
    // catch and handle errors
});
```

In production, you would most likely want to serialize and persist the token cache. Depending on the type of application, you have several alternatives:

* Desktop apps, headless apps (public client):
  * If you don't want persistence, you don't need to do anything else. MSAL's in-memory cache will suffice.
  * If you do want persistence, use [MSAL Node Extensions](../../../extensions/msal-node-extensions/README.md), which provide persistence solutions on Windows, Linux and Mac OS
* Web apps, web APIs, daemon apps (confidential client):
  * MSAL's in-memory token cache does not scale for production. Use the [distributed token caching](#performance-and-security) pattern to persist the cache in your choice of storage environment (Redis, MongoDB, SQL databases etc. -keep in mind that you can use these in tandem *e.g.* a Redis-like memory cache as a first layer of persistence, and a SQL database as a second, more stable persistence layer)

## Token cache schema

MSAL Node's cache schema is compatible with other MSALs. By default, MSAL's cache is not partitioned and not encrypted: all authentication artifacts by all users are located in a single cache blob, grouped by the type of the authentication artifact (see also: [cache.json](../cache.json)).

```json
{
    "Account": {},
    "IdToken": {},
    "AccessToken": {},
    "RefreshToken": {},
    "AppMetadata": {}
}
```

## Persistence

MSAL Node fires events when the cache is accessed, apps can choose whether to serialize or deserialize the cache. This often constitutes two actions:

1. Deserialize the cache from disk to MSAL's memory before accessing the cache
2. If the cache in memory has changed, serialize the cache back

For that, MSAL accepts a custom cache plugin in [configuration](./configuration.md). This should implement [ICachePlugin](https://azuread.github.io/microsoft-authentication-library-for-js/ref/interfaces/_azure_msal_common.icacheplugin.html):

```typescript
interface ICachePlugin {
    beforeCacheAccess: (tokenCacheContext: TokenCacheContext) => Promise<void>;
    afterCacheAccess: (tokenCacheContext: TokenCacheContext) => Promise<void>;
}
```

* If you are developing a public client app (such as desktop, headless etc.), [MSAL Node Extensions](../../../extensions/msal-node-extensions/README.md) handles this for you.
* If you are developing a confidential client app, you should persist the cache via a separate service, since a single, *per-server* cache instance isn't suitable for a cloud environment with many servers and app instances.

> :warning: We strongly recommend to encrypt the token cache when persisting it on disk. Again, for public client apps, this is offered out-of-box with [MSAL Node Extensions](../../../extensions/msal-node-extensions/README.md). For confidential clients however, you are responsible for devising an appropriate encryption solution.

## Performance and security

On public client apps, [MSAL Node Extensions](../../../extensions/msal-node-extensions/README.md) ensures performance and security for you.

On confidential client applications that handle users (web apps that sign in users and call web APIs, and web APIs calling downstream web APIs), there can be many users and the users are processed in parallel. Our recommendation is to serialize one cache blob per user. Use a key for partitioning the cache (*i.e.* **partition key**), such as:

* For web apps: **homeAccountId** (if on ADFS, use **localAccountId** instead)
* For multi-tenant daemon apps using client credentials grant: **tenantId** (or rather, `<clientId>.<tenantId>`)
* For web APIs calling other web APIs using OBO: hash of the incoming access token from the user (i.e. the token which will subsequently be exchanged for an OBO token)

For instance, when developing a web app that serve users, implement [ICachePlugin](https://azuread.github.io/microsoft-authentication-library-for-js/ref/interfaces/_azure_msal_common.icacheplugin.html) in a class where user's `homeAccountId` is used for partitioning:

```typescript
class DistributedCachePlugin implements ICachePlugin {

    // singleton
    private static instance: CachePlugin;

    private persistenceManager: any; // your implementation of a persistence client
    private partitionKey: string;

    private constructor(persistenceManager?: any, partitionKey?: string) {}

    static getInstance(persistenceManager?: any, partitionKey?: string): CachePlugin {}

    async beforeCacheAccess(cacheContext): Promise<void> {
        // given a partition key, get the relevant cache portion from the persistence store
        // deserialize and load the cache into msal's memory
    }
    async afterCacheAccess(cacheContext): Promise<void> {
        // if in-memory cache has changed
            // serialize and persist the cache using the same partition key
    }
}
```

It is also a good idea to obtain performance metrics for persistent cache operations, such as **cache hit ratios** from your persistence store. This would give you a reliable picture of how your app scales over time.

## More information

* [(Sample) Public client app using MSAL Node Extensions](../../../extensions/samples/msal-node-extensions/index.js)
* [(Sample) Confidential client Express MVC web app with distributed token cache](../../../samples/msal-node-samples/ExpressTestApp/README.md)
* [(Docs) Token cache serialization](https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/token-cache-serialization)
* [(Docs) App scenarios and authentication flows](https://docs.microsoft.com/azure/active-directory/develop/authentication-flows-app-scenarios)

branch_name: refs/heads/dev

gha_license_id: MIT


text:
# diskimgr

**diskimgr** provides a simple GUI-based workflow for making disk images of digital media, such as floppy disks, Flash drives and harddisks. It wraps around the [*dd*](https://linux.die.net/man/1/dd) and [*ddrescue*](https://linux.die.net/man/1/ddrescue) tools. After the imaging is done it also generates a checksum file with SHA-512 hashes of the extracted files.

## System requirements

*Diskimgr* is currently only available for Linux. So far it has been tested with Ubuntu 18.04 LTS (Bionic) and Linux Mint 18.3, which is based on Ubuntu 16.04 (Xenial). In addition it has the following dependencies:

- **Python 3.2 or more recent** (Python 2.x is not supported)

- **Tkinter**. If *tkinter* is not installed already, you need to use the OS's package manager to install (there is no PyInstaller package for *tkinter*). If you're using *apt* this should work:

        sudo apt-get install python3-tk

- **dd**, which is part of the [*GNU Core Utilities*](https://en.wikipedia.org/wiki/GNU_Core_Utilities).

- **ddrescue**, which can be installed using:

        sudo apt install gddrescue

## Installation

### Preparation: give non-root user permission to run dd

By default, running *dd* on a Linux system requires root access. To allow a non-root user to run *dd*, you must give them the permissions to do so by adding the user to the *disk* group. You can do this with the command below (replace $USER with the name of the user who will be using *diskimgr*):

        sudo adduser $USER disk

The user is now added to the 'disk' system group. Now log out, and then log in again for the changes to take effect.

### Global install

For a global (all-users) installation run the following command:

    sudo pip3 install diskimgr

Then run:

    sudo diskimgr-config

If all goes well this should result in the following output:

    INFO: writing configuration file /etc/diskimgr/diskimgr.json
    INFO: creating desktop file /usr/share/applications/diskimgr.desktop
    INFO: diskimgr configuration completed successfully!

### User install

Use the following command for a single-user installation:

    pip3 install --user diskimgr

Then run:

    ~/.local/bin/diskimgr-config

Result:

    INFO: writing configuration file /home/johan/.config/diskimgr/diskimgr.json
    INFO: creating desktop file /home/johan/.local/share/applications/diskimgr.desktop
    INFO: diskimgr configuration completed successfully!

*diskimgr* is now ready to roll!

## Basic operation

You can start *diskimgr* from the OS's main menu (in Ubuntu 18.04 the *diskimgr* item is located under *System Tools*). Depending on your distro, you might get an "Untrusted application launcher" warning the first time you activate the shortcut. You can get rid of this by clicking on "Mark as Trusted". On startup the main *diskimgr* window appears:

![](./img/diskimgr-1.png)

Use the *Select Output Directory* button to navigate to an (empty) directory where the output files are to be stored. The interface allows you to specify the following options:

|Option|Description|
|:-|:-|
|**Block Device**|Select the medium (device) you want to image from the drop-down list. Press the **Refresh** button to refresh the items in the drop-down list|
|**Block size**|This sets the size of the buffer (in bytes) that is used by *dd* / *ddrescue* default: `512`).|
|**Read method**|The method (application) that is used to read the medium (default: `dd`).|
|**Retries**|Maximum number of retries (setting only has effect with *ddrescue*) (default: `4`).|
|**Direct disc mode**|Check this option to read a medium in direct disc mode (setting only has effect with *ddrescue*) (disabled by default).|
|**Auto-retry with ddrescue on dd failure**|This checkbox controls the behaviour with media that result in read errors with *dd*. If checked, *diskimgr* will automatically re-try such a medium with *ddrescue*. Otherwise, *diskimgr* will first display a confirmation dialog.|
|**Load existing metadata**|Loads *Prefix*, *Extension*, *Identifier*, *Description* and *Notes* values (see below) from an existing metadata file in the output directory that was created by a previous *diskimgr* session. Useful for re-running media that were previously interrupted or unfinished. If no metadata file can be found, *diskimgr* will display an error, and the fields can be entered manually|
|**Prefix**|Output prefix (default: `disc`).|
|**Extension**|Output file extension (default: `img`).|
|**Identifier**|Unique identifier. You can either enter an existing identifier yourself, or press the *UUID* button to generate a [Universally unique identifier](https://en.wikipedia.org/wiki/Universally_unique_identifier).|
|**Description**|A text string that describes the medium (e.g. a title that is written on the label of a floppy disk).|
|**Notes**|Any additional info or notes you want to record with the medium.|

Press the *Start* button to start imaging a medium. You can monitor the progress of the extraction procedure in the progress window:

![](./img/diskimgr-2.png)

Note that the screen output is also written to a log file in the output directory. A prompt appears when the imaging is finished:

![](./img/diskimgr-success.png)

If the imaging finished without any errors, the output directory now contains the following files:

![](./img/diskimgr-files.png)

Here, **ks.img** is the created ISO image; **checksums.sha512** contains the SHA512 checksums of the image, **metadata.json** contains some basic metadata and **diskimgr.log** is the log file.

If *dd*'s attempt to read the disc resulted in any errors, *diskimgr* prompts the user to try again with *ddrescue*:

![](./img/error-dd.png)

After clicking *Yes*, *diskimgr* will move the disc image that was created by *dd* to a subdirectory *dd-failed*, and then start *ddrescue*. If *ddrescue* also exits with any errors, you will see this dialog:

![](./img/error-ddrescue.png)

From *ddrescue*'s output, you can see that 106 kB of data could not be recovered (value of *errsize* variable). After pressing *Yes*, you can do one or more additional passes with *ddrescue*. For instance, you can activate *Direct Disc* mode, or select another floppy drive. Press the *Start* button again to start reading the medium. Importantly, this won't overwrite the existing image, but it will update it with any additional data that can be rescued from the medium. As an example, below is the result of the second *ddrescue* pass, where *Direct Disc* mode was activated:

![](./img/ddrescue-pass2.png)

In this case *ddrescue* still reports a read error, but now *errsize* is reduced from 106 kB to just 512 bytes, which means that only one sector could not be recovered.

Note that *ddrescue* runs result in an additional [*mapfile*](https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Mapfile-structure) (**$prefix.map**). The map file contains information about the recovery status of data blocks, which allows *ddrescue* to resume previously interrupted recovery sessions.

## Suggested workflow

In general *dd* is the preferred tool to read a floppy disk, flash drive or harddisk. However, *dd* does not cope well with media that are degraded or otherwise damaged. Because of this, the suggested workflow is to first try reading the medium with *dd*. If this results in any errors, try *ddrescue*. If you check the **Auto-retry** box, *diskimgr* will automatically launch *ddrescue* if the initial attempt to read the medium with *dd* failed (i.e. it will not display the confirmation dialog).

It is possible to run multiple subsequent passes with *ddrescue*. If *ddrescue* fails with errors, it sometimes helps to re-run it in *Direct disc* mode (which can be selected from *diskimgr*'s interface). The results can sometimes be further improved by running multiple *ddrescue* passes with different reader devices (e.g. a few USB-connected floppy drives).

## Interrupting dd or ddrescue

Press the *Interrupt* button to interrupt any running *dd* or *ddrescue* instances. This is particularly useful for *ddrescue* runs, which may require many hours for media that are badly damaged. Note that interrupting *ddrescue* will not result in any data loss. Interrupting *dd* will generally result in an unreadable image file.

## Resuming an interrupted ddrescue run

Follow these steps to resume a *ddrescue* run that was previously interrupted:

1. After launching *diskimgr*, set the output directory to the directory of the interrupted run.

2. Set **Read method** to *ddrescue*.

3. Click on the **Load existing metadata** button; this loads the previously entered *Prefix*, *Extension*, *Identifier*, *Description* and *Notes* values.

4. Hit the **Start** button. Now *ddrescue* will simply pick up on where the interrupted run stopped.

## Metadata file

The file *metadata.json* contains metadata in JSON format. Below is an example:

    {
        "acquisitionEnd": "2019-04-04T17:53:11.489543+02:00",
        "acquisitionStart": "2019-04-04T17:52:29.692731+02:00",
        "autoRetry": false,
        "blockDevice": "/dev/sdb",
        "checksumType": "SHA-512",
        "checksums": {
            "ks.img": "79a17d3fa536b8fa750257b01d05124dadb888f1171e9ca5cc3398a2c16de81b1687b52c70135b966409a723ef5f3960536a6e994847c5ebe7d5eaffefa62dc7"
        },
        "description": "KS metingen origineel",
        "diskimgrVersion": "0.1.0b1",
        "extension": "img",
        "identifier": "5b159d32-56f1-11e9-9abb-2c4138b5272c",
        "interruptedFlag": false,
        "maxRetries": "4",
        "notes": "",
        "prefix": "ks",
        "readCommandLine": "dd if=/dev/sdb of=/home/bcadmin/test/1/ks.img bs=512 conv=notrunc",
        "readMethod": "dd",
        "readMethodVersion": "dd (coreutils) 8.28",
        "rescueDirectDiscMode": false,
        "successFlag": true
    }

Most of these fields are self-explanatory, but the following need some further explanation:

- **interruptedFlag** is a Boolean flag that is *true* if *dd* or *ddrescue* were interrupted, and *false* otherwise.
- **successFlag** is a Boolean flag that is *true* if the medium was imaged without any problems, and *false* otherwise.

## Configuration file

*Diskimgr*'s internal settings (default values for output file names, the optical device, etc.) are defined in a configuration file in Json format. For a global installation it is located at */etc/diskimgr/diskimgr.json*; for a user install it can be found at *~/.config/diskimgr/diskimgr.json*. The default configuration is show below:

    {
        "autoRetry": "False",
        "blockSize": "512",
        "checksumFileName": "checksums.sha512",
        "defaultDir": "",
        "extension": "img",
        "logFileName": "diskimgr.log",
        "metadataFileName": "metadata.json",
        "prefix": "disc",
        "rescueDirectDiscMode": "False",
        "retries": "4",
        "timeZone": "Europe/Amsterdam"
    }

You can change *diskimgr*'s default settings by editing this file. Most of the above settings are self-explanatory, with the exception of the following:

- **autoRetry**: this flag  sets the default value of the *Auto-retry* checkbox.

- **defaultDir**: this allows you to change the default file path that is opened after pressing *Select Output Directory*. By default *diskimgr* uses the current user's home directory. However, if *defaultDir* points to a valid directory path, that directory is used instead.

- **timeZone**: time zone string that is used to correctly format the *acquisitionStart* and *acquisitionEnd* date/time strings. You can adapt it to your own location by using the *TZ database name* from [this list of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones).

If you accidentally messed up the configuration file, you can always restore the original one by running the *diskimgr-config* tool again.

## Uninstalling diskimgr

To remove *diskimgr*, first run the *diskimgr-config* with the `--remove` flag to remove the configuration file and the start menu and desktop files. For a global install, run:

    sudo diskimgr-config --remove

For a user install, run:

    ~/.local/bin/diskimgr-config --remove

The resulting output (shown below for a user install):

    INFO: removing configuration file /home/johan/.config/diskimgr/diskimgr.json
    INFO: removing configuration directory /home/johan/.config/diskimgr
    INFO: removing desktop file /home/johan/.local/share/applications/diskimgr.desktop
    INFO: diskimgr configuration completed successfully!

Then remove the Python package with following command (global install):

    sudo pip3 uninstall diskimgr

For a user install use this:

    pip3 uninstall diskimgr

## Contributors

Written by Johan van der Knijff. Some parts of the code that processes ddrescue's terminal output were adapted from [DDRescue-GUI](https://launchpad.net/ddrescue-gui) by Hamish McIntyre-Bhatty.

## License

*Diskimgr* is released under the  Apache License 2.0.

repository: KBNLresearch/diskimgr

repo_name: KBNLresearch/diskimgr


text:
# README
Repo name: `random_hostname_nick3499`
Set system hostname with pseudo-random alpha-numeric string.

## Note: `random_hostname.py` changes system hostname; be sure user wants system hostname to change.

## Shebang/Interpreter Directive

The **shebang line** starts with `#!`, because those human-readable characters, encoded to ASCII, become **0x23** and **0x21**, which together become a _magic number_ or a magic byte string detected by functions in the [exec()](http://man7.org/linux/man-pages/man3/exec.3.html) family. Such `exec()` functions ascertain whether a file is an executable binary or a script.

**Note**: the space between the _shebang_ and the _interpreter directive_ is optional.

## Import Modules

```python
from random import randrange
from subprocess import run
```

- `random.randrange()` returns a random integer within a specific range.
- `subprocess.run()` executes Bash command described by args from Python.

## subprocess.run()

After the script is done juggling and playing slight of hand with characters, `rand_host` is passed as an arg to `hostnamectl` through the `subprocess.run()` method.

```python
run(['sudo', 'hostnamectl', 'set-hostname', rand_host], check=True)
```

- `subprocess.run()` runs `hostnamectl` in Bash, waits for `hostnamectl` to finish, then returns a `CompletedProcess` instance.
- `check=True` captures `stdout` or `stderr`.

## random.randrange()

`random.randrange()` returns a peudo-random number. _psuedo_-random, basically, because _random logic_ is an oxymoron. `random.randrange()` takes three integers: **random.randrange(start, stop, increment)**. The integer returned becomes a Unicode code point passed to the [chr()](https://docs.python.org/3/library/functions.html#chr) built-in method, where `chr(97)` returns `'a'`, so `chr(randrange(97, 123, 1))` will return a pseudo-random lowercase letter between `'a'` and `'z'`, just as the f-string `f'{rand_lwr()}'` returns a lowercase letter.

## `rand_host`

An [f-string](https://www.python.org/dev/peps/pep-0498/) is assigned to `rand_host`. Each set of braces contains an embedded expression. For example, the f-string `f'{rand_az()}'` could become `'q'`, since `rand_az()` returns a random lowercase letter.

```python
rand_host = f'{rand_az()}{rand_AZ()}{rand_09()}{rand_az()}{rand_AZ()}{rand_09()}'
```

`rand_host` would contain something akin to `mV4jA8`, as each embedded expression returns a character. See _Example Prompt_ section.

## Example Prompt

```shell
foo@sA1nJ5:~/Desktop $
```
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/R6R72LISM)


text:
---
layout: post
title: Sharing Presentations
description: In which I discuss a super easy way to share your presentations
permalink: /2013/08/sharing-presentations/index.html
categories:
- speaking
---

I've been using the supremely excellent
[reveal.js](http://lab.hakim.se/reveal-js/) library for all of my presentations
in the last year. It's super easy to use, it's very lightweight, and I can
build a sharp presentation without having to download or install some 400mb
software package on my computer *and* the computer I'm using to present, if,
God forbid, it's not the same as I used to write it.

Also, by virtue of being html, I can write my own theme files in CSS and
plugins in Javascript. Some plugins supported include secondary-monitor speaker
notes, remote control over mobile phone, markdown, and code syntax
highlighting. It can also print all the slides to PDF in individual pages. And,
perhaps best of all, you can use gifs as massive background images.

Oh, and if this wasn't already magical enough, I can store all of my
presentations on GitHub. That allows for sharing, pull requests, and quick
updates through either Vim or the web ui.

That's when it struck me today - not only can I put my slides on GitHub for
keeping track of edits and all that Git goodness, but I can create a `gh-pages`
branch, and it will *automatically host my slides under my domain name*.

But wait, there's more. You don't even have to hop into the terminal to do it.
In GitHub, click on the 'branches' button, type in `gh-pages`,
and a minute later you're up and running.

Check out these examples:

* [thejacklawson.com/how-to-hubot](/how-to-hubot)
* [thejacklawson.com/dismantling-the-monorail](/dismantling-the-monorail)
* [thejacklawson.com/javascript-testing](/javascript-testing)

extension: markdown


text:
---
tags: programming
layout: post
title: "Submissions, etc."
---

Finally got my OSCON presentations submitted; gnat's note had the intended effect :-) One is on OpenInteract, another on SPOPS. We'll see.

Transplanted the keyboard from my LCD-not-functioning NEC laptop to the LCD-functioning-but-keyboard-borked NEC laptop. And now I have a functioning laptop again. If this thing holds up and I'm able to get to San Diego I'll be one of those 1337 folks with wireless.

I get very nervous when taking apart laptops. With PCs, I've not only done it a million times but I know that if I screw something up it's easily replacable.  And with these (they're 5+ years old) I don't even know if they can be fixed.

From someone's note here (Dave?) I started poking around with kpresenter. It seems decent enough, but I'm always afeard of playing with software like this because it's got so many time-sucking bells and whistles. I might stick with HTML + Javascript  because I can tarball it up and run it on anything, even a Win32 machine if the aforementioned laptop breaks down. Another (X11 only) alternative might be Magic Point, which keeps everything in plaintext but has an option for generating postscript and it has a simple viewer for transitions, etc. I suppose the thing to do is keep it in plaintext until everything is settled, then move it to whatever platform I'll use.

I saw Jurassic Park III the other night. I figured it was probably dumb, but I remembered reading a review (from one of the Washington Post folks, I think) saying that it was a sparser but more fun dinosaur movie. I figured anything with lots of dinosaur action can't be all bad. I was wrong. Many of the action scenes were great, but they were far outweighed by the insulting "plot" and "motivations." I feel dumber for having rented it.

(Originally posted elsewhere)

gha_language: CSS


text:
---
title: 'Deploying Microservices with Traefik'
date: 2020-05-02
description: 'A detailed walkthrough of how to deploy multiple docker services using Traefik as reverse proxy'
tags:
  ['microservices', 'Traefik', 'DevOps', 'software', 'web', 'Docker', 'YAML']
---

I've been running this blog on a bare-metal server using [Nginx](https://www.nginx.com/) as a reverse proxy to basically serve the static files generated by Gatsby. But now I want to deploy multiple docker services on my [VPS](https://en.wikipedia.org/wiki/Virtual_private_server).

To help with that endeavour, I decided to try using Traefik, which is an edge router that can act as a reverse proxy and route multiple services on my personal domain.  So, I decided to document it.

## Goals

First, I'd like to plan out what the goals are so I can figure out how to get there:

- Run multiple containerised services on my domain under different subdomains, mainly [blog.imranc.io](https://blog.imranc.io) for now
- HTTPS everywhere

There, that seems pretty simple. So lets get on with it.

## Primer

Just a quick primer on some of the technologies being used for deployment:

- [**Traefik**](https://containo.us/traefik/) is an open-source edge router that will handle web requests sent to my domain
- [**Docker**](https://www.docker.com) is platform-as-a-service software that uses OS-level virtualisation to deliver software in _containers_
- [**Docker Compose**](https://docs.docker.com/compose/) is a tool for orchestrating multi-container applications using YAML config files

## Containerising My Blog

[_Containerisation_](https://hackernoon.com/what-is-containerization-83ae53a709a6) is the concept of bundling an application together with all of its related configuration files, libraries and dependencies required for it to run efficiently across a variety of different computing environments. I have chosen Docker to containerise my applications because I'm slightly familiar with it from work and it's been all the rage in DevOps for quite a while now.

Currently, the way I've been deploying changes to my blog is the following:

1. Make new markdown files in my blog [repo](https://github.com/imrancio/blog)
2. Commit blog changes and push to GitHub
3. SSH into my server and pull blog changes
4. Use `gatsby-cli` to build production files
5. Production files get symlinked to Nginx site folders
6. Nginx serves new production files

The problem with that is everything after 3 is done manually and a bit limiting. Nginx requires a lot of fiddling around with to configure everything correctly. I have to write server code for every application I want deployed and it's been years since I messed around with Nginx.

Docker simplifies the process of setting up a server to serve static files for my blog. Nginx itself is a containerised service in Docker Hub. That means using Docker, it's quite easy to set up my blog as a service that builds all the Gatsby static files and serves them on my domain using a single configuration file.

```Dockerfile
ARG GATSBY_ACTIVE_ENV=production
FROM node:12-alpine AS build

RUN yarn global add gatsby-cli
ARG GATSBY_ACTIVE_ENV
ENV GATSBY_ACTIVE_ENV=$GATSBY_ACTIVE_ENV

WORKDIR /app
ADD . ./
RUN yarn install
RUN gatsby build

FROM nginx
COPY --from=build /app/public /usr/share/nginx/html
COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 80
```

This single `Dockerfile` above is all I need in my blog repository to build the Docker image for my blog, along with a `.dockerignore` file to optimise what's included in my blog service.

In Docker, you can start from a variety of base _images_. These are basically minimal OSes with certain packages and applications pre-installed. Then, you keep adding _layers_ to this base image by running each command in the `Dockerfile` to build your service. Finally, you can use Docker Compose to spin up multiple _containers_ (instances) running from these built _images_.

Essentially, what my `Dockerfile` does is the following:

#### On a minimal Node.js Docker image

1. Installs `gatsby-cli`
2. Copies over all my blog src code, excluding any assets that will be built later
3. Installs all the packages required to build my blog
4. Runs `gatsby build` to build static files for my blog

#### On an Nginx server Docker image

5. Copies over static blog files from Node.js image
6. Copies over Nginx config to run the server
7. Blog server is exposed via HTTP (port 80)

## Traefik Routing

Now that my blog has been built into a Docker _image_, I can deploy my blog in a Docker _container_. The magic with all this is that Traefik, which is a service for routing to other services, can also run in a Docker _container_. This means all of my services will run in Docker containers and I can configure them all in a single `docker-compose.yaml` file.

The following is part of the `docker-compose.yaml` file used to orchestrate multiple services in my domain using Docker Compose:

```yaml
version: '3.3'

services:
  traefik:
    image: traefik:v2.2
    container_name: traefik
    command:
      # highlight-start
      # HTTP -> HTTPS entrypoint redirection
      - '--entrypoints.web.address=:80'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--entrypoints.websecure.address=:443'
      # Docker provider
      - '--providers.docker=true'
      # Traefik Console
      - '--api=true'
      # Let's Encrypt certificate resolver
      - '--certificatesresolvers.leresolver.acme.tlschallenge=true'
      - '--certificatesresolvers.leresolver.acme.email=ACME_EMAIL'
      - '--certificatesresolvers.leresolver.acme.storage=/letsencrypt/acme.json'
      # highlight-end
    ports:
      - '80:80'
      - '443:443'
    volumes:
      # Persist Let's Encrypt certificates
      - './letsencrypt:/letsencrypt'
      # For Docker service discovery
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    labels:
      # highlight-start
      - 'traefik.http.routers.api.rule=Host(`traefik.imranc.io`)'
      - 'traefik.http.routers.api.entrypoints=websecure'
      - 'traefik.http.routers.api.tls.certresolver=leresolver'
      - 'traefik.http.routers.api.service=api@internal'
      - 'traefik.http.routers.api.middlewares=auth'
      - 'traefik.http.middlewares.auth.basicauth.users=BASIC_AUTH_USERNAME:PASSWORD_STRING'
      # highlight-end

  blog:
    image: imrancio-blog
    container_name: blog
    labels:
      # highlight-start
      - 'traefik.enable=true'
      - 'traefik.http.routers.blog.rule=Host(`blog.imranc.io`)'
      - 'traefik.http.routers.blog.entrypoints=websecure'
      - 'traefik.http.routers.blog.tls.certresolver=leresolver'
      # Routes from root imranc.io
      - 'traefik.http.routers.root.rule=Host(`imranc.io`)'
      - 'traefik.http.routers.root.entrypoints=websecure'
      - 'traefik.http.routers.root.tls.certresolver=leresolver'
      - 'traefik.http.routers.root.middlewares=redirect-to-blog'
      # Routes from www.imranc.io
      - 'traefik.http.routers.www.rule=Host(`www.imranc.io`)'
      - 'traefik.http.routers.www.entrypoints=websecure'
      - 'traefik.http.routers.www.tls.certresolver=leresolver'
      - 'traefik.http.routers.www.middlewares=redirect-www-to-blog'
      # imranc.io -> blog.imranc.io middleware
      - "traefik.http.middlewares.redirect-to-blog.redirectregex.regex=^https?:\\/\\/(imranc\\.io)+(.*)$$"
      - 'traefik.http.middlewares.redirect-to-blog.redirectregex.replacement=https://blog.$${1}$${2}'
      # www.imranc.io -> blog.imranc.io middleware
      - "traefik.http.middlewares.redirect-www-to-blog.redirectregex.regex=^https?:\\/\\/www\\.(imranc\\.io)+(.*)$$"
      - 'traefik.http.middlewares.redirect-www-to-blog.redirectregex.replacement=https://blog.$${1}$${2}'
      # highlight-end
```

There's a lot going on there but the main thing to note is there are 2 services, **traefik** and **blog**:

- Traefik service is configured in the `commands` section and all the Traefik routes for services are configured via their respective `labels` sections
- All HTTP entrypoint routes are redirected to HTTPS
- There is a route setup for Traefik Dashboard at [traefik.imranc.io](https://traefik.imranc.io), which is hidden behind a basic auth wall.
- There is a route setup for my blog server image at [blog.imranc.io](https://blog.imranc.io)
- There's some redirect routes setup for [imranc.io](https://imranc.io) and [www.imranc.io](https://www.imranc.io), which both point to [blog.imranc.io](https://blog.imranc.io)

The idea is I can keep adding services here and routing them all via `labels` with very little configuration required.

## Deployments => Piece of :cake:

With the current setup of using containerised services and Traefik to route everything, I have simplified deployments to the following:

1. Make new markdown files in my blog [repo](https://github.com/imrancio/blog)
2. Commit blog changes and push to GitHub
3. SSH into my server and pull blog changes
4. Build new `imrancio-blog` image using Docker
5. Restart my `blog` container using Docker Compose

In fact, I've condensed steps 3-5 into a one-liner:

```bash
ssh imranc.io './update-blog.sh'
```

## Additional Services

As an example of additional services I can deploy, I went ahead and set up my own self-hosted instance of [Bitwaden](https://bitwarden.com/) at [vault.imranc.io](https://vault.imranc.io) using the [vaultwarden](https://github.com/dani-garcia/vaultwarden) Docker image. I am the sole user of the service and have disabled registration. As such, I can store all my credentials in my own server and have it sync across all my devices anywhere in the world from my private vault that no one else can access. I really :heart: that I can do that!

## Update

Since I wrote this post, I have hosted several different services on my domain. So I thought it would be a good idea to come and update a list of all services on my domain every so often.

Here is a list of all the services currently hosted on my domain:

| Service                                               | Description         |
| ----------------------------------------------------- | ------------------- |
| [vault.imranc.io](https://vault.imranc.io)            | password manager    |
| [blog.imranc.io](https://blog.imranc.io)              | my personal blog    |
| [cdn.imranc.io](https://cdn.imranc.io)                | static file hosting |
| [notes.imranc.io](https://notes.imranc.io)            | standard notes      |
| [proxy.imranc.io](https://proxy.imranc.io/dashboard/) | traefik dashboard   |

repository: imrancio/blog

branch_name: refs/heads/main

gha_language: TypeScript

repo_name: imrancio/blog


text:
# [Scriptable.app](https://scriptable.app) scripts

## Scripts
* [Import-Script.js](Import-Script.js) - a script to download and import other scripts into your Scriptable library. Supports github.com repos and gists, pastebin.com, hastebin.com and raw javascript code.
To use, share the URL via the share sheet or copy the url or actual code in the clipboard the run the script.

* [OAuth2](OAuth2) - see [Automators.fm topic](https://talk.automators.fm/t/building-a-general-purpose-oauth-redirect-proxy-for-shortcuts-and-scriptable/4420).

## Widgets
* [xkcd-widget](xkcd-widget) - a widget to show current/random xkcd comic
* [peanuts-widget](peanuts-widget) - a widget to show current/random Peanuts™ comic
* [ig-latest-post](instagram-widgets/ig-latest-post.js) - randomly show between the 12 of the most recent post from a user or users.
* [simple-weather-widget](openweathermap/simple-weather-widget.js) - example widget that uses the [openweathermap](openweathermap) module
* [transparent-backgrounds](no-background/examples) - example scripts that use the [no-background](no-background) module
* [us-elections.js](misc/us-elections.js) - show the latest electoral votes for all candidates
* [rh-profile-widget.js](routinehub-widgets/rh-profile-widget.js) - A widget to show the current shortcuts and download counts of a routinehub.co profile

## Utilites
---
* [no-background](no-background) - a module to simulate transparent background for widgets. Includes examples.
* [openweathermap](openweathermap) - A module to encapsulate OpenWeatherMap's [One Call API](https://openweathermap.org/api/one-call-api) and more
* [basic-ui](utilities/basic-ui.js) - a helper moduel for user interactions
* [json-utils](utilities/json-utils.js) - a helper module for reading, storing, and converint JSON.

repository: markogley/scriptable-scripts

repo_name: markogley/scriptable-scripts


text:
# Node Req

![](http://i1117.photobucket.com/albums/k594/thetutlage/poppins-1_zpsg867sqyl.png)

![](https://img.shields.io/travis/poppinss/node-req.svg)
[![Coverage Status](https://coveralls.io/repos/poppinss/node-req/badge.svg?branch=master&service=github)](https://coveralls.io/github/poppinss/node-req?branch=master)

`node-req` is an i/o module for parsing and returning values out of HTTP request object using helper methods.It is not express, neither connect. It is just an i/o module for parsing request object.

## See also

1. node-res
2. node-cookie

## Http

```javascript
var http = require('http')
var nodeReq = require('node-req')

http.createServer(function (req, res) {

  // get query string from req
  var query = nodeReq.get(req)

}).listen(3000)

```

Yes, that's all, `node-req` makes no assumption on how to add routes or handle HTTP requests. All it does it parse request object and return values out of it.

## What's different from express?

`express` is super cool, but it is a framework that has a lot more to offer than just parsing requests, where `node-req` is just a parser and will be an option based on your use case.

## Methods

#### get (req)
returns request query string parameters

```javascript
// req url is /user?name=doe

nodeReq.get(req)

// returns {name:'doe'}
```

#### method (req)
returns request method, or you can say HTTP verb

```javascript
nodeReq.method(req)

// returns GET,POST etc
```

#### headers (req)
returns request headers

```javascript
nodeReq.headers(req)
```

#### header (req, key)
returns request header for a given key

```javascript
nodeReq.header(req, 'Content-Type')
```

#### fresh (req, res)
test for request freshness based on E-tag and expires header

```javascript
nodeReq.fresh(req, res)
```

#### stale (req, res)
opposite of fresh

```javascript
nodeReq.stale(req, res)
```

#### ip (req, trust)
Returns request ip based on trusted proxy, check out [proxyaddr](https://www.npmjs.com/package/proxy-addr) for more info

```javascript
nodeReq.ip(req, ['<IP_ADDRESS>/8', '<IP_ADDRESS>/8'])
```

#### ips (req, trust)
Returns list of all IP address associated with a request, starting from closest to furthest based on trusted proxy. Also check out [proxyaddr](https://www.npmjs.com/package/proxy-addr) for more info

```javascript
nodeReq.ips(req, function (remoteAddress) {
  return remoteAddress === '<IP_ADDRESS>'
})
```

#### secure (req)
Is request from https ?

```javascript
nodeReq.secure(req)
```

#### subdomains (req, trust, offset=2)
List of subdomains on a given request. If trust is enabled it will refer to the `X-Forwarded-Host` header. Also check out [proxyaddr](https://www.npmjs.com/package/proxy-addr) for more info

```javascript
// request url gig.baz.com
nodeReq.subdomains(req, false)

// returns ['gig']
```

whereas

```javascript
// request url www.baz.com
nodeReq.subdomains(req)

// returns []
```

#### ajax (req)
determines whether a request is ajax or not based on `X-Requested-With` header.

```javascript
nodeReq.ajax(req)
```

#### pjax (req)
determines whether a request is pjax or not based on `X-Pjax` header

```javascript
nodeReq.pjax(req)
```

#### hostname (req, trust)
returns request hostname and if trust is enabled, it will refer to the `X-Forwarded-Host` header. Also check out [proxyaddr](https://www.npmjs.com/package/proxy-addr) for more info

```javascript
nodeReq.hostname(req, trust)
```

#### url (req)
returns request URL without the query string

```javascript
// request url /users?offset=0&limit=10

nodeReq.url(req)

// returns /users
```

#### originalUrl (req)
returns request originalUrl with query string

```javascript
// request url /users?offset=0&limit=10

nodeReq.originalUrl(req)

// returns /users?offset=0&limit=10
```

#### is (req, [types])
tells whether request is of certain type or not based on `Content-type` header.

```javascript
nodeReq.is(req, 'html')
// true
nodeReq.is(req, 'text/html')
// true
nodeReq.is(req, ['json','html'])
//true
```

#### accepts (req, [types])
tells whether request accepts data of certain type based on `Accepts` header.

```javascript
nodeReq.accepts(req, 'html')
// 'html'
nodeReq.accepts(req, 'text/html')
// 'text/html'
nodeReq.accepts(req, ['json','html'])
// 'html'
```

#### hasBody (req)
tells whether request has body to be read by any body parser.

```javascript
if (nodeReq.hasBody(req)) {
  req.on('data', function (chunk) {
    // ...
  })
}
```

## License
(The MIT License)

Copyright (c) 2015 Poppins

Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


text:
# Horns Lab App
Creators:
Ell, Beniam, Zerek

## Day One
### Task One
Feature #1: Display images

Estimate of time needed to complete: 1 hour

Start time: 930

Finish time: __12:04___

Actual time needed to complete: ___2.5 hrs__

## Day One
### Task One
Number and name of feature: _drop down form______

Estimate of time needed to complete: __1 hour___

Start time: __9am___

Finish time: ___9:45am__

Actual time needed to complete: __45 mins___

## Day One
### Task One
Number and name of feature: ___________sort by keyword_____________________

Estimate of time needed to complete: __1hr___

Start time: __9:45am___

Finish time: __10:42am___

Actual time needed to complete: __~1hr___

## Day Two
### Task Two
Number and name of feature: _________Pagination_______________________

Estimate of time needed to complete: ___2__

Start time: _____

Finish time: _____

Actual time needed to complete: _____
## Day Two
Number and name of feature: ________________________________

Estimate of time needed to complete: _____

Start time: _____

Finish time: _____

Actual time needed to complete: _____

gha_language: JavaScript


text:
couchdb-logging
=====================

Simple logging system for WordPress using CouchDB

## Installation

Download the plugin via GitHub and add this to your `wp-config.php` or something.

```php
<?php

define('CB_HOST', '<IP_ADDRESS>');
define('CB_PORT', '5984');
define('CB_NAME', 'my-awesome-database');

// you can also have login, optional.

define('CB_USER', 'user');
define('CB_PASSWORD', 'password');
?>
```

If no CouchDB options is defined it will use the host `<IP_ADDRESS>`, the default port `5984` and the database name `wordpress-logging` . Now you are ready to start logging!

```php
<?php

// Will return the document from the database or false if something goes wrong.
couchdb_log('info', array('status' => 'ok'));

?>
```

Get all logs from the database

```php
<?php

  $logs = get_couchdb_logs([$limit ,] [$endKey ,] [$descending]);

?>
```

* `$limit` Is the count of documents to return.
* `$endKey` Is the timestamp to stop at.
* `$descending` It's speaks for it self.

## Todo

* Add WordPress view where you can sort and view the logs.
* Fix so it don't break the site when Sag failes.

repository: frozzare/couchdb-logging

repo_name: frozzare/couchdb-logging


text:
# README Generator

  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

  ## Description

  This application is a README Generator.

  ## Table of Contents
  1. [Installation](#Installation)
  2. [Usage](#Usage)
  3. [License](#License)
  4. [Contributing](#Contributing)
  5. [Tests](#Tests)
  6. [Questions](#Questions)

  ## Installation

  This application utilizes Inquirer, so the installation requirements would be to install the Inquirer package. `npm i inquirer`

  ## Usage

  The user inputs the content they want to include in their README, and the application will write the README.md file for the user. The README has a general template, but the content will be unique to the user.

  ## License

  MIT

  ## Contributing

  Anyone can contribute. Please see [Questions](#Questions) below for the contact information of the dev team.

  ## Tests

  N/A

  ## Questions

  Please visit https://github.com/achueng to learn more about this project and the developer.

  If there are any question, issues, or concerns, please contact Agnes via email at [agneschueng@gmail.com](mailto:agneschueng@gmail.com).

repository: achueng/README-Generator

repo_name: achueng/README-Generator


text:
# Investec Programmable Banking

Investec's Programmable Banking make it possible to write Javascript which runs everytime you make a purchase using your Investec Programmable Banking enabled card.

This repository is intended to contain examples of how Investec's Programmable Banking can be used to accomplish common tasks. **You are welcome to use, modify,
and extend this code!** If you have an additional example you think would be valuable, please submit a pull request.


text:
# Monday night game delayed by black cat in second quarter

Published at: **2019-11-04T18:46:05+00:00**

Author: **The Associated Press**

Original: [The Seattle Times](https://www.seattletimes.com/sports/monday-night-game-delayed-by-black-cat-in-second-quarter/)

EAST RUTHERFORD, N.J. (AP) — A black cat got the best of the Dallas Cowboys and New York Giants for a couple minutes Monday night.
The cat scampered on the field during a Giants drive in the second quarter and forced referee Clay Martin to delay the game for few minutes while workers at MetLife Stadium and couple of New Jersey State Troopers herded the feline toward the end zone away from the players.
The cat did not depart right away. There was a point where the animal was directed into the corner of the end zone and then sprinted across the end line to a camera platform.
For a second, the cat jumped on the platform in and then sprang out. It finished its run along the end line before running up the tunnel to the cheers of the big crowd.
___
More AP NFL: https://apnews.com/NFL and https://twitter.com/AP_NFL
The Associated Press

license: Unlicense

repository: dsadlollioygjmzvcxv/tdc-exported

repo_name: dsadlollioygjmzvcxv/tdc-exported


text:
Bump version:

```
bumpversion patch
```

Generate Changelog [using this tool](https://pypi.python.org/pypi/gitchangelog):

TODO:

```

```


text:
# Elasticsearch Ansible Role

Installs and configures Elasticsearch 5.x, 6.x, or 7.x.

Requirements
------------
This role is tested on Ansible 2.6, 2.7, and 2.8 using the distributions below.
Other operating systems and versions may work, YMMV.

- Ubuntu 18.04
- Ubuntu 16.04
- Centos 7
- Amazon Linux 2

Also, this role uses the `json_query` Ansible filter, which requires the `jmespath`
library be installed on the Ansible workstation.

Role Variables
--------------
See [defaults/main.yml](defaults/main.yml) for a list and description of
variables used in this role.

Example Playbook
----------------
```yaml
- hosts: servers
  vars:
    es_version: "6.2.2"
    es_config:
      bootstrap.memory_lock: true
  roles:
     - role: elasticsearch
```

repository: bdclark/ansible-elasticsearch

repo_name: bdclark/ansible-elasticsearch


text:
# Pixel Art Pallete Generator

This is a project in React using Hooks, created as a tool to edit small pixel art images.
You can upload your image pixel art and it will get the pallet of colors, and you could change any color and or randomize them with some restrictions.

[Live App](https://jhonamath.github.io/pixel-art-pallete-generator/)

![Captura](https://user-images.githubusercontent.com/13243106/116637384-2a6e4880-a93a-11eb-8bef-5a26f9f9bf5b.PNG)

# Disclaimer

This was my first attempt using hooks and I want to improve the organizatiion of the code

Hope you enjoy this :)

# License
[MIT](https://choosealicense.com/licenses/mit/)

repository: JhonaMath/pixel-art-pallete-generator

repo_name: JhonaMath/pixel-art-pallete-generator


text:
## Avoid greedy builds

Hint:
See the [`timeout`](https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#timeout-enforce-time-limit) step.

repository: mramanathan/jenkins-pipeline-class

gha_language: Groovy

repo_name: mramanathan/jenkins-pipeline-class


text:
# c-multiaddr
multiaddr for IPFS in C.

### Multiaddr provides easy networking protocols nesting, easy encapsulation of extra protocols, easy tunneling, etc.
# Usage:
#### All you need to include is multiaddr.h
## Maddr struct:
* char string[]; // String that contains addresses such as /ip4/<IP_ADDRESS>/
* uint8_t bytes; //String that contains the enecoded address
* int bsize[]; //int[1] that contains the real bytes size (Use it whenever using the bytes so you don't input trash!)

## New Multi Address From String(new_maddr_fs)
  char addrstr[] = "/ip4/<IP_ADDRESS>/"
	struct maddr a;
	a=new_maddr_fs(addrstr);
## Obtaining the byte buffer(.bytes, .bsize[0]):
  printf("TEST BYTES: %s\n",Var_To_Hex(a.bsize[0], a.bytes));
  Var_To_Hex = Byte Buffer to Hex String
  Hex_To_Var = Hex String to Byte Buffer
## Encapsulation & Decapsulation(m_encapsulate, m_decapsulate)
#### Remember, Decapsulation happens from right to left, never in reverse, if you have /ip4/udp/ipfs/ if you decapsulate "udp" you will also take out ipfs!
* Now the string is: /ip4/<IP_ADDRESS>/
* m_encapsulate(&a,"/udp/3333/"); //Adds udp/3333/ to char addrstr
* Now the string is: /ip4/<IP_ADDRESS>/udp/3333/
* m_decapsulate(&a,"udp"); //Removes udp protocol and its address
* Now the string is: /ip4/<IP_ADDRESS>/
* m_encapsulate(&a,"/tcp/8080");
* Now the string is: /ip4/<IP_ADDRESS>/tcp/8080/

# Constructing a multiaddress from bytes:

* struct maddr beta;
* beta=new_maddr_fb(a.bytes,a.bsize[0]); //This will already construct back to the string too!
* printf("B STRING: %s\n",beta.string);  //So after encapsulation and decapsulation atm this string would
* contain: /ip4/<IP_ADDRESS>/tcp/8080/

repository: jmjatlanta/c-multiaddr

repo_name: jmjatlanta/c-multiaddr


text:
Reporting Issues
================

1.) Read the Documentation
--------------------------

Please [read the README](https://github.com/ordbok/core)
and [check the core API](https://ordbok.github.io/core)
before reporting new issues, even if you think you have found a bug.

Issues that ask questions answered in the documentation will be closed without
elaboration.

2.) Search for Duplicates
-------------------------

[Search existing issues](https://github.com/ordbok/core/search?type=Issues)
before reporting a new one.

Some search tips:

- *Don't* restrict your search to only open issues. An issue with a title
  similar to yours may have been closed as a duplicate of one with a
  less-findable title.

- Check for synonyms. For example, if your bug involves an interface, it likely
  also occurs with type aliases or classes.

- Search for the title of the issue you're about to report. This sounds obvious
  but 80% of the time this is sufficient to find a duplicate when one exists.

- Read more than the first page of results. Many issues here use the same words
  so relevancy sorting is not particularly strong.

- If you have a crash, search for the first few topmost function names shown in
  the call stack.

3.) Do You Have a Question?
---------------------------

The issue tracker is for **issues**, in other words, bugs and suggestions.
If you have a *question*, please use
[Stack Overflow](http://stackoverflow.com/questions/tagged/ordbok),
your favorite search engine, or other resources.

4.) Did You Found a Bug?
------------------------

When logging a bug, please be sure to include the following:

- The version of ORDBOK you're using (run `npx ordbok-assembler --version`).

- What version of TypeScript you're using (run `npx tsc --v`).

- If at all possible, an *isolated* way to reproduce the behavior.

- The behavior you expect to see, and the actual behavior.

5.) Do You Have a Suggestion?
-----------------------------

Suggestions are also accepted in the issue tracker. In general, things we find
useful when reviewing suggestions are:

- A description of the problem you're trying to solve

- An overview of the suggested solution

- Examples of how the suggestion would work in various scenarios

- If relevant, precedent in other languages can be useful for establishing
  context and expected behavior

Instructions for Contributing Code
==================================

Advices
-------

If you like to contribute code, an issue report is required, which has been
[labelled "contribute"](https://github.com/ordbok/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22contribute%22)
by a project maintainer.

Design changes will not be accepted at this time. If you have a design change
proposal, please log a suggestion issue.

Legal
-----

You will need to complete a Contributor License Agreement (CLA). Briefly, this
agreement testifies that you are granting us permission to use the submitted
change according to the terms of the project's license, and that the work being
submitted is under appropriate copyright.

Please submit a Contributor License Agreement (CLA) before submitting a pull
request. Download the agreement
([ORDBOK Contribution License Agreement.pdf](https://ordbok.github.io/cla/ORDBOK%20Contribution%20License%20Agreement.pdf)),
sign, scan, and email it back to<EMAIL_ADDRESS>Be sure to include your
github user name along with the agreement. Once we have received the signed CLA,
we'll review the request.

Pull Request
------------

Your pull request should:

- Include a description of what your change intends to do

- Contain child commits of a reasonably recent commit in the **master** branch

- Contain no merge commits

- Have clear commit messages

- Build and run without errors (`npm run build`)

- Avoid line ending issues, preferable set
  `autocrlf = input` and `whitespace = cr-at-eol` in your git configuration

repository: ordbok/core

repo_name: ordbok/core


text:
## text-decoration-color

### Summary

The `text-decoration-color` CSS property sets the color used when drawing underlines, overlines, or strike-throughs specified by [`text-decoration-line`][0]. This is the preferred way to color these text decorations, rather than using combinations of other HTML elements.

* _[Initial value][1]_ `currentColor`
* _Applies to_ all elements. It also applies to [`::first-letter`][2] and [`::first-line`][3].
* _[Inherited][4]_ no
* _Media_ [`visual`][5]
* _[Computed value][6]_ \*Not in database (color)\*
* _Animatable_ yes, as a [color][7]
* _Canonical order_ the unique non-ambiguous order defined by the formal grammar

### Syntax

    [Formal syntax][8]: [<color>][9]

    text-decoration-color: currentColor
    text-decoration-color: red
    text-decoration-color: #00ff00
    text-decoration-color: rgba(255, 128, 128, 0.5)
    text-decoration-color: transparent

    text-decoration-color: inherit

### Values

**`<color>`**

> The `color` property accepts various keywords and numeric notations. See [`<color>`][10] values for more details.

### Examples

    .example {
        text-decoration: underline;
        text-decoration-color: red;
    }

The example above has the same effect as the following code, which also adds a hover style.

    <!DOCTYPE html>

      black text with red underline and blue hover

[0]: https://developer.mozilla.org/en/docs/Web/CSS/text-decoration-line "The text-decoration-line CSS property sets what kind of line decorations are added to an element."
[1]: https://developer.mozilla.org/en/docs/CSS/initial_value
[2]: https://developer.mozilla.org/en/docs/Web/CSS/::first-letter "The ::first-letter CSS pseudo-element selects the first letter of the first line of a block, if it is not preceded by any other content (such as images or inline tables) on its line."
[3]: https://developer.mozilla.org/en/docs/Web/CSS/::first-line "The ::first-line CSS pseudo-element applies styles only to the first line of an element. The amount of the text on the first line depends of numerous factors, like the width of the elements or of the document, but also of the font size of the text. As all pseudo-elements, the selectors containing ::first-line does not match any real HTML element."
[4]: https://developer.mozilla.org/en/docs/CSS/inheritance
[5]: https://developer.mozilla.org/en/docs/CSS/@media#Media_groups
[6]: https://developer.mozilla.org/en/docs/CSS/computed_value
[7]: https://developer.mozilla.org/en/docs/CSS/color_value#Interpolation "Values of the <color> CSS data type are interpolated on each of their red, green, blue components, each handled as a real, floating-point number. Note that interpolation of colors happens in the alpha-premultiplied sRGBA color space to prevent unexpected grey colors to appear."
[8]: https://developer.mozilla.org/en/docs/CSS/Value_definition_syntax "CSS/Value_definition_syntax"
[9]: https://developer.mozilla.org/en/docs/Web/CSS/color_value
[10]: https://developer.mozilla.org/en/docs/Web/CSS/color_value "The documentation about this has not yet been written; please consider contributing!"


text:
# 10.12.19 - Call with Luke

## What we did since last meeting
- meeting and unpacking with Jens and Patrick
    - qualitative research (what is encoding)
- access to data:
- looked at data on server
    - programmed endpoints for data
    - evaluation on server with BP-Workspace
- already ideated on some visulizations
- started reading literature

## Questions regarding data
- What defines the individuals dataset? What defines the messages dataset? What is the difference between the datasets?
- Every messages and individuals object has four "main" keys: Data, Metadata, NestedTracedData and SHA - What is saved in under each of these keys?
- Is one line in the dataset one "trace", i.e. one dataset with "complete" provenance information regarding one response?
- Does one line in a dataset correspond to one in the other? If so, how?
- "unclean" dataset? We found strings with Somali content with the key "age_raw" in individuals? Should they also be replaced?
- Can anything be in "\_raw" keys? (non fitting data...)
- What do the keys mean? (e.g. "rqas04e01...", "suggestions", "recovered_run", "have_voice" )
   - do you have maybe a code book / documentation, where all encodings are explained?
- What keys encode the "themes"?
- What are the different "id"-fields for, since there are quite a lot of them ("run_id - csap_s02_demog", "recently_displaced_raw_id", ...)
- What do the objects with "MERGED" content mean?

## Next steps
- Schedule call with researcher to talk about her process (at specific example)
- design thinking to approach problem space

## Data Explanations
- Individuals/messages
- One way: not worrying about individuals
- Messages: questions - answers, counting grouped together
- Individuals: more complex, joining answers together to one individual

- Messages grouped to answers, we probably only see answers
- (Radio question and answer 1), we will get the questions asked

- nestedTraceData: historical values
- Data: category, decision
- Metadata: origin information how the decision was made to categorise

- One line one array: performance reasons for pipeline
- We can probably flatten that list together

- Probably one object is one answer

- Look at python code for unpacking data

- Avf core data modules

- Provenance: nestedTraceData
- Do not need to take id’s for links
- Take leaf of data structure

- Theoretically correlating both data sets would work, but is too much unnecessary work
- Contain same data

- Unclean data on purpose

- Schema is public, content only sensitive

- “Research question and answer season 4 episode 1”
- Social accountability

- recovered_run (ssl - sms problems)

- Suggestions: do you have any suggestions -> encoding
- Actual suggestion in text itself is in suggestions_raw_id with sha for message object

- MERGED - no idea

- STOP - person has opted out, we cannot report anything like this

- Have method to export data into excel sheet, because people sometimes rather look at that

- code_id standard network operator codes

- Themes -> code schemes being used and then look up code scheme values
- Scheme-ID

- [x] Look at avf GitHub repo

- Can use python library or reimplement some of that in lively

- nestedTraceData: list of maps, each head level is one

- have_voice: related to question

- Confidence: 1 -> person, 0 wasn’t applied here

- _ws: data was somewhere else in the scheme

- Districts are pretty big
- Location might not be correct, answers messy because we ask people where they are
- Location id: are they giving us a state, region, district, zone, so on

- First work on concrete visualisations, then ask researcher next year probably (user study), take report, ask them how they came up with report

- Element with empty nestedtracedata and unique string like sha

- [ ] Divergent designs, 20 different for each case


text:
The files in this directory aren't part of the database schema definition.

This entire directory and all its contents may be purged any point.

And, of course, DO NOT COMMIT SENSITIVE DATA.

If you're working on production data, or with authentication keys or any
other kind of sensitive data - then you shouldn't be working with that stuff
in this directoy.

No, not even if you have an ignore file. If it's sensitive data - don't let
it anywhere near the source tree.

This is a "development-harness" area where you can put whatever SQL you want.

As with a development-harness, it's good to have a safe place where you can
commit "work-in-progress" that is clearly defined as being "safe" from being
included into production or being subject to code-review requirements, etc.

It's useful to have this in the repo because sometimes it's good to be able
go back and look at the SQL your were slinging while figuring out how to
do something / write the flyway migration scripts.

repository: jasonadkison/cabbage

repo_name: jasonadkison/cabbage


text:
user-scripts
============

This is a collection of semi-useful bash user scripts that will provide
various bits of information about the system. These contain a mix of scripts
some intended only for the Raspberry PI, however, most will work on any
modern linux.

Scripts
-------

chkpkg.sh	Will check whether the specified package is installed.

getCPUSerial.sh	Will return the serial# of the cpu, if available. Will likely
		only work for Raspberry PIs

getIPAddr.sh	Will return the IP address of the requested interface, or
		eth0, if not specified.

getMAC.sh	Will return the MAC address of the specified interface, or
		eth0, if not specified.

getMemTotal.sh	Will return the amount of total system memory.

getMemFree.sh	Will return the amount of free system memory.

whereis_inc.sh	Will return the path where GCC believes it will find
		the specifed include header file.

INSTALL
-------

See the INSTALL file for instructions on how build/install this tool suite.

LICENSE
-------

See the LICENSE file for license info.

repository: kodetroll/user-scripts

repo_name: kodetroll/user-scripts


text:
This project was bootstrapped with [Create React App](https://github.com/facebookincubator/create-react-app).

Below you will find some information on how to perform common tasks.<br>
You can find the most recent version of this guide [here](https://github.com/facebookincubator/create-react-app/blob/master/packages/react-scripts/template/README.md).

# Features:

- [x] Server side rendering - with data (redux)
- [x] Critical css with css modules
- [x] Eslint + prettier (code style)
- [x] Autoformat (js, css) with prettier before commit (pre-commit hook)
- [x] Meta data header tags
- [x] Flow type checking
- [x] React router 4
- [x] Redux thunk, promise, array
- [x] PWA with SWPrecacheWebpackPlugin (disable by default)
- [x] Jest - testing framework
- [ ] CI - testing + report + deploy

repository: vunguyentuan/react-pwa-boilerplate

repo_name: vunguyentuan/react-pwa-boilerplate


text:
# random

- [zshrc](https://github.com/lserpietri/random/blob/master/zshrc): based on the [Powerlevel9k Guide](https://github.com/bhilburn/powerlevel9k/blob/master/README.md)

- [testSNI.sh](https://github.com/lserpietri/random/blob/master/testSNI.sh): based on the guide found here https://www.exampleloadbalancer.net/. Checks certificates and shows that they're all behind the same ALB.

repository: lserp/random

repo_name: lserp/random


text:
# raspberrypi-2048
2048 game adapted for raspberry pi sense hat.
Because there is not enough space on sense hat to place the numbers, they are coded as below

![screenshot](docs/blocks.png)

Below is a photo of how it looks like on a real device

![screenshot](docs/pic1.jpg)

## Run the application
Execute /2048/main.py to run the application. Use the joystick to control the game.


text:
---
layout: default
---

# 1997-10-09 - Re: 6502 ML programming

## Header Data

From<EMAIL_ADDRESS>(Matthew Ghio)

To<EMAIL_ADDRESS>Message Hash: 301797438c4a41d70ace666752a41d7cecc985fb4a11972e841a5bbeb2c5b363

Message ID: \<199710090632.CAA06768@myriad\>

Reply To<EMAIL_ADDRESS>UTC Datetime: 1997-10-09 06:56:33 UTC

Raw Date: Thu, 9 Oct 1997 14:56:33 +0800

## Raw message

```
{% raw  %}From<EMAIL_ADDRESS>(Matthew Ghio)
Date: Thu, 9 Oct 1997 14:56:33 +0800
To<EMAIL_ADDRESS>Subject: Re: 6502 ML programming
In-Reply-To<EMAIL_ADDRESS>Message-ID: <199710090632.CAA06768@myriad>
MIME-Version: 1.0
Content-Type: text/plain

Jim Choate<EMAIL_ADDRESS>wrote:
>
> There in no 'DATA' construct in 6502 assembly.
>
> This is some sort of macro that either the assembler or programmer
> defined. All official Rockwell/Commodore [1] 6502 assembly mnemonics are
> three letter.

True, but most assemblers would let you insert static data into the object
file using various pseudo-ops like that.

> Idealy a jump table was created that had the various target addresses for
> your routines.

Oh, those were fun to debug too...

> The most efficient way to store data in 6502 is to put it in the first 256
> bytes of ram, then it could be called in no more than 2 clock cycles.

Actually that took three cycles.  One to fetch the opcode, one to fetch the
target address, and one to fetch the data byte.  If you wanted to do it in
two cycles, you had to write the data into the operand of the instruction
that loaded it (self-modifying code).

{% endraw %}
```

## Thread

+ Return to [October 1997](/archive/1997/10)

+ Return to<EMAIL_ADDRESS>(Matthew Ghio)](/authors/ghio_at_temp0133_myriad_ml_org_matthew_ghio_)"
+ Return to "[Jim Choate <ravage@ssz.com>](/authors/jim_choate_ravage_at_ssz_com_)"
+ Return to "[Jonathan Wienke <JonWienk@ix.netcom.com>](/authors/jonathan_wienke_jonwienk_at_ix_netcom_com_)"

+ 1997-10-09 (Thu, 9 Oct 1997 12:20:40 +0800) - [6502 ML programming](/archive/1997/10/e62e6b9a0e68cd9696dfc851786c3962f5b5f9f31508fc82fbf117fd3b4a4fb1) - _Jim Choate<EMAIL_ADDRESS>  + 1997-10-09 (Thu, 9 Oct 1997 14:56:33 +0800) - Re: 6502 ML programming -<EMAIL_ADDRESS>(Matthew Ghio)_
  + 1997-10-11 (Sat, 11 Oct 1997 12:29:14 +0800) - [Re: 6502 ML programming](/archive/1997/10/6752bb4091242599dbe2df06f8a17fae62de8cb6f4eff67e12f781ce0ca6f383) - _Jonathan Wienke<EMAIL_ADDRESS>


text:
# vue-head-slot

Manage content for your `head` element easily. Internally uses Vue3's `<teleport>`.

## Installation
Installing
```sh
yarn add vue-head-slot
```
or
```sh
npm i -S vue-head-slot
```

## Usage

```js
import VueHeadSlot from 'vue-head-slot'
...
app.use(VueHeadSlot)
```

And anywhere in your app.
```html
<head-slot>
    <meta name="robots" content="nofollow">
</head-slot>
```

Results in...
```html
<html>
<head>
    ...
    <meta name="robots" content="nofollow">
</head>
...
```

Just be sure to remove content from `index.html` that you intend to manage like this. This appends content, so it will duplicate tags that are already declared.

## Other examples

Using with `vue-router`.
```html
<!-- Some page in your route scheme -->
<head-slot>
    <title>{{ $route.name }}</title>
    <link rel="canonical" :href="dynamicallyBuiltUrl" />
    <link v-for="lang in langs" :key="lang" rel="alternate" :hreflang="lang" :href="`${dynamicallyBuiltUrl}/${lang}`" />
</head-slot>
```

## Disclaimer
Vue3's Teleport makes this so easy that you probably don't need this module.

Just use `<teleport to="head">...</teleport>`.

I used render functions to seem a bit more clever.

```js
// pretty much the whole source code
import { h, Teleport } from 'vue'

export default {
  install(app, name) {
    app.component(name || 'head-slot', {
      render() {
        return h(
          Teleport,
          { to: 'head' },
          this.$slots.default()
        )
      }
    })
  }
}
```

## Trivia

This was almost called `vue-head-shot`. I expect credit if someone creates a fork with that name.

## License
Copyright 2020 nick-somebody

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: nick-somebody/vue-head-slot

repo_name: nick-somebody/vue-head-slot


text:
---
title: "Plague Phylogeography Demo"
authors: "Katherine Eaton"
dataset: "http://localhost:4000/plague150Local?d=map"
abstract: "
This is a demo of the plague exhibit.

The content displayed here is for testing display purposes only. It is not intended for interpretation, as the results are known to contain substantial and technical artifacts at this point."
---

<!-- This is left-side text 1-->
# [Time-Scaled Phylogeny](http://localhost:4000/plague150Local?d=tree&m=time)
* [START HERE: How to read a phylogeny](https://nextstrain.org/narratives/trees-background/).
* This is a time-scaled phylogeny, where samples (terminal nodes) correspond to their collection date and internal nodes are estimated to be the most likely time of divergence.
* There are numerous problems with this tree.
* Ex. The root of the tree is dated to be 1069 CE. We know from previous modern work, and ancient DNA papers, that the coalescence of these samples should be MUCH later in time, at least 5000 years ago.

<!-- This is left-side text 2-->
# [Temporal Signal](http://localhost:4000/plague150Local?d=tree&l=clock&m=time)
* And it's no surprise the time estimates are so bad when we look at the temporal signal in the data.
* Basically no dots fall on the dark black regression line.

<!-- This is left-side text 1-->
# [Divergence-Scaled Phylogeny](http://localhost:4000/plague150Local?d=tree&m=div&s=GCA_001601675.1_ASM160167v1_genomic)
* The lack of temporal signal makes a lot of sense if we look at the divergence-scaled phylogeny.
* Here, the distance between nodes measures nucleotide substitutions per genomic site (instead of time).
## Time Discrepancies
* These samples were collected between 1947 and 2016.
* The sample highlighted is from 1999.
* Compare this to the Peruvian clade (Purple) which is mostly from 2010.
* There is WILD variation in branch lengths, that does not seem to correlate with time.
## Investigate Further
* Click EXPLORE THE DATA YOURSELF in the top right panel.
* Click RESET LAYOUT in the top right of the tree panel.

# [A Complex Ecology](http://localhost:4000/plague150Local?d=tree,map&c=host&legend=open)
* This complex pacing of evolution could be due to a complex multi-host ecology.
* Plague can be found in many different mammalian hosts across the globe.
* When colored by host genus, no strong clustering pattern appears on the tree.

# [Human Outbreaks](http://localhost:4000/plague150Local?c=host&d=tree,map&f_host=Homo&legend=closed)
* Similarly, human outbreaks occur throughout the tree and of course, throughout the world.

<!-- This is left-side text 3-->
# [The Origins of Plague](http://localhost:4000/plague150Local?d=map&animate=1400-01-01,1675-01-01,1,1,30000)
* If we ONLY study modern genomes, Y. pestis origins are in Asia although the question of Russia/Mongolia/China is outstanding.
* Sampling of ancient genomes has been key (TO BE ADDED IN!) in changing this.
* Note: This map is an example of the dangers of poorly curated geocoding. Each plague sample has been geocoded using the country of origin which results in latitude and longitude of the center of the country (Nominatim, GeoPy). Russia is a a very large country, demonstrating why this is an inappropriate method.

# [Geographic Spread](http://localhost:4000/plague150Local?d=map&animate=1400-01-01,2017-01-01,0,1,30000)
* With all these issues in mind, this dataset probably isn't yet trustworthy for geographic reconstructions.
* But here is what the highly biased version looks like so far.

repository: ktmeaton/plague-phylogeography

gha_language: Jupyter Notebook

repo_name: ktmeaton/plague-phylogeography


text:
# In this exercise we will look at some useful considerations for managing OpenShift projects
## Disable self provisioning of new projects
```
$ oadm policy remove-cluster-role-from-group self-provisioner system:authenticated system:authenticated:oauth
```
## Set project request message
```
# vi /etc/origin/master/master-config.yaml
projectConfig:
  projectRequestMessagt "to request a new project, fill out the project request form located at
https://internal.example.com/openshift-project-request.:"
```
## Set default node selector for project
```
# vi /etc/origin/master/master-config.yaml
projectConfig:
  defaultNodeSelector: "type=user-node,region=east"
```
## Create project and set node selector for project
```
$ oadm new-project myproject --node-selector='zone=prague,region=europe'
```
## Set default project limits and limits based on user selector
```
# vi /etc/origin/master/master-config.yaml
admissionConfig:
  pluginConfig:
    ProjectRequestLimit:
      configuration:
        apiVersion: v1
        kind: ProjectRequestLimitConfig
        limits:
        - selector:
            level: admin
        - selector:
            level: advanced
          maxProjects: 10
        - maxProjects: 2
```
## Update all masters and restart API and controller
```
# systemctl restart atomic-openshift-master-api
# systemctl restart atomic-openshift-master-controllers
```

repository: ktenzer/ose-exercises

repo_name: ktenzer/ose-exercises