text:
---
layout: post.html
title: "Part 3: Bot.py Module"
tags: [Network]
---

Writing our `bot.py` module.

## Bot Setup

First, within your `network_project/talkback` directory, create the following file:

```bash
(NetworkProj) $ touch bot.py
```

`bot.py` should be in the same directory as `quote_picker.py` from the previous section.  Go ahead and open up `bot.py` within your text editor.

With `bot.py`, we only need to leverage modules from the Twisted library.  There’s no expectation that you would know which modules from Twisted to import; this is just an introduction to the package’s vast capabilities in networking.  In this package, we are taking advantage of Twisted’s `log` module for logging rather than using Python’s `logging` module, `protocol` module to create our bot factory (to be explained), as well as leverage Twisted’s `irc` module so we don’t reinvent the wheel.

Note that the order of import statements are alphabetical per [PEP-8](http://www.python.org/dev/peps/pep-0008/), Python’s style guide.

```python
from twisted.internet import protocol
from twisted.python import log
from twisted.words.protocols import irc
```

### Scaffolding for bot.py module

We will write two classes: `TalkBackBot` and `TalkBackBotFactory`.  The factory class actually instantiates the bot, while the bot class defines the bot’s behavior.

Let’s first start off with the bot factory scaffolding with comments and docstrings:

```python
# <--snip-->
class TalkBackBotFactory(protocol.ClientFactory):
    # instantiate the TalkBackBot IRC protocol

    def __init__(self, settings):
        """Initialize the bot factory with our settings."""

```

 The factory is in charge of creating/instantiating a protocol (here, the `TalkBackBot`).  With the bot factory, we inherit from Twisted’s `protocol.ClientFactory`.  This is so we can make use of creating a connection between our client and the protocol (our IRC connection), and handle any connection errors.

 Now our `TalkBackBot` scaffolding:

```python
# <--snip-->

class TalkBackBot(irc.IRCClient):

    def connectionMade(self):
        """Called when a connection is made."""

    def connectionLost(self, reason):
        """Called when a connection is lost."""

    # callbacks for events

    def signedOn(self):
        """Called when bot has successfully signed on to server."""

    def joined(self, channel):
        """Called when the bot joins the channel."""

    def privmsg(self, user, channel, msg):
        """Called when the bot receives a message."""

# <--snip-->
```

The `TalkBackBot` class inherits from `irc.IRCClient` from the Twisted library.  This is so we can make use of functions like `connectionMade`, `signedOn`, etc, and define desired behavior.

First, we’ll code out the bot factory, then return to the bot itself.

### TalkBackBotFactory class

We first define the protocol that the Factory will make the bot with:

```python
# <--snip-->

protocol = TalkBackBot

# <--snip-->
```

This calls an internal method within the `twisted.internet.protocol` library, `buildProtocol()`.  This instantiates a `ClientFactory` to be able to handle input of an incoming server connection.

Notice that in our import statements, we didn’t import our `settings.ini` file.  When we run our program, the plugin that we write (detailed in [Part 4]({{ get_url('/networks/part-4')}})) will pick up the file.  With that, our `TalkBackBotFactory` initializes with the settings:

```python
# <--snip-->

def __init__(self, channel, nickname, realname, quotes, triggers):
    """Initialize the bot factory with our settings."""
    self.channel = channel
    self.nickname = nickname
    self.realname = realname
    self.quotes = quotes
    self.triggers = triggers

# <--snip-->

```

The initialization of our factory is pretty self explanatory – the factory is created with settings that are defined in `settings.ini`.  When we write our plugin in [part 4]({{ get_url('/networks/part-4')}}), we will code out the passing of those configuration settings into our factory.

### TalkBackBot class

Now for the `TalkBackBot` class.  Revisiting the scaffolding we did earlier, we define 5 functions for our class, which will setup the behavior for our bot:

```python
# <--snip-->

class TalkBackBot(irc.IRCClient):

    def connectionMade(self):
        """Called when a connection is made."""

    def connectionLost(self, reason):
        """Called when a connection is lost."""

    # callbacks for events

    def signedOn(self):
        """Called when bot has successfully signed on to server."""

    def joined(self, channel):
        """Called when the bot joins the channel."""

    def privmsg(self, user, channel, msg):
        """Called when the bot receives a message."""

# <--snip-->
```
First, the `connectionMade` function: this is considered the initialization of the protocol because it is called when the connection from our client to the IRC server is completed.

```python
# <--snip-->

def connectionMade(self):
    """Called when a connection is made."""
    self.nickname = self.factory.nickname
    self.realname = self.factory.realname
    irc.IRCClient.connectionMade(self)
    log.msg("connectionMade")

# <--snip-->
```

When we connect to the IRC service (which we will code out in Part 4), want to assign the `nickname` and `realname` of the bot.  If we wanted a greeting message upon connecting to IRC, we would also define it here.

We then call `irc.IRCClient.connectionMade`, which takes in the whole TalkBackBot object (`self`), which contains  the `nickname` and `realname` variables.

Lastly, we wish to log this action.  We are taking advantage of Twisted’s `log` module, which will take care of the time stamps for when each `log.msg()` function is called.  We pass in the string `"connectionMade"` so when we consult our logs, we can see this function was called and a connection was made.  This is very helpful for debugging purposes.  If we were having issues with connecting to the IRC server, we would not hit this log message, and therefore narrow down where the issue is.

Our next function, `connectionLost`, is very similiar:

```python
# <--snip-->

def connectionLost(self, reason):
    """Called when a connection is lost."""
    irc.IRCClient.connectionLost(self, reason)
    log.msg("connectionLost {!r}".format(reason))

# <--snip-->
```

`connectionLost` is called when the connection to the IRC Server is closed and “tears down” our protocol. Here, we log the action and the reason.

In our `log.msg()` line, we pass in a string, `"connection lost, reconnecting {!r}"` followed by the string method, `format`.  The curly braces, `{}`, indicate a replacement field.  This field will be populated by `reason`, an argument passed into our `connectionLost` function.

The `!r` tells `format` to call the function `repr()` (rather than the `str()` function) on `reason`.  If we were to do `!s` instead, `format` would *not* include quotes around `reason`, [like so](http://docs.python.org/2/library/string.html#formatexamples):

```python
>>> "repr() shows quotes: {!r}; str() doesn't: {!s}".format('test1', 'test2')
"repr() shows quotes: 'test1'; str() doesn't: test2"
```

To understand `repr()` versus `str()` better, [StackOverflow](http://stackoverflow.com/questions/1436703/difference-between-str-and-repr-in-python) has a great explanation.

Next, we define our `signedOn` function:

```python
# <--snip-->

def signedOn(self):
    """Called when bot has successfully signed on to server."""
    log.msg("Signed on")
    if self.nickname != self.factory.nickname:
        log.msg('Your nickname was already occupied, actual nickname is '
                '"{}".'.format(self.nickname))
    self.join(self.factory.channel)

# <--snip-->
```

This function will be called when our bot has successfully signed on to our IRC server.  This is different than simply connecting to the IRC server; `connectionMade` is to be treated as the initialization/setup of our protocol (IRC), and `signedOn` is called when we have successfully signed on with our nickname to the server.

The log message is pretty self-explanatory; we are simply logging the action of signing on.

We also put logic in case there is another user or bot signed on with the same nickname.  When connecting to an IRC server and your nickname is already in use, the server will often give you a modified nickname, like `thatswhatshesaid_` with the trailing `_`.

Last, we call the `self.join()` function, defined in `irc.IRCClient`, to actually join our desired channel.

Our next function, `joined`, is called after the event of when the bot joins our desired channel.  We are simply logging our actions here:

```python
# <--snip-->

def joined(self, channel):
    """Called when the bot joins the channel."""
    log.msg("[{nick} has joined {channel}]"
            .format(nick=self.nickname, channel=self.factory.channel,))

# <--snip-->
```

Our last function for our `TalkBackBot` class is the fun part: defining what happens when someone says “that’s what she said” in our channel:

```python
# <--snip-->

def privmsg(self, user, channel, msg):
    """Called when the bot receives a message."""
    sendTo = None
    prefix = ''
    senderNick = user.split('!', 1)[0]
    if channel == self.nickname:
        # /MSG back
        sendTo = senderNick
    elif msg.startswith(self.nickname):
        # Reply back on the channel
        sendTo = channel
        prefix = senderNick + ': '
    else:
        msg = msg.lower()
        for trigger in self.factory.triggers:
            if msg in trigger:
                sendTo = channel
                prefix = senderNick + ': '
                break

    if sendTo:
        quote = self.factory.quotes.pick()
        self.msg(sendTo, prefix + quote)
        log.msg(
            "sent message to {receiver}, triggered by {sender}:\n\t{quote}"
            .format(receiver=sendTo, sender=senderNick, quote=quote)
        )

# <--snip-->
```

The `privmsg` is called whenever the bot receives a message.  We first initialize who we are replying to, `sendTo = None`, and the prefix for our eventual message, `prefix = ''`, as well as `senderNick` who is the user who prompts the bot with the trigger.

The `if channel == self.nickname` is for the condition when the bot receives a message directly, like with `/msg`. The second condition, `elif msg.startswith(self.nickname)` is for when a user starts a message with the bot’s nickname within the channel.  The last is if someone in the channel says the trigger, “That‘s what she said.”

Basically, if the bot receives a private message, gets mentioned in the beginning of a message, or if someone says a trigger, we set `sendTo` from `None` to the appropriate reply.

Then, if `sendTo` isn’t anything but `None`, we construct a quote by picking our quote at random, then using `self.msg` (which we can use because the `msg` method is defined in `irc.IRCClient`) to execute the sending of the message.  Lastly, we log our action.

### __init__.py

Within `network/talkback/` directory, you’ll notice that there is an empty `__init__.py` file.  It is used to mark directories that are a part of our Python package/application.  According to [Python’s documentation for packages](http://docs.python.org/2/tutorial/modules.html#packages):

> The `__init__.py` files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as `string`, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, `__init__.py` can just be an empty file, but it can also execute initialization code for the package or set the `__all__` variable, described later.

← Part 2: Quote Picker

Part 4: Twisted Plugin →

license: Zlib

repository: willingc/new-coder

gha_license_id: Zlib

repo_name: willingc/new-coder


text:
---
id: 97
title: FT232R to FTDI-cable Adapter
date: 2011-05-10T02:46:00+09:30
author: James Young
layout: post
guid: http://wordpress/wordpress/?p=97
permalink: /2011/05/ft232r-to-ftdi-cable-adapter/
blogger_blog:
  - coding.zencoffee.org
blogger_author:
  - DW
blogger_permalink:
  - /2011/05/ft232r-to-ftdi-cable-adapter.html
categories:
  - Electronics
tags:
  - arduino
---
I got myself one of Sparkfun's [FT232R Breakout](http://www.sparkfun.com/products/718) Boards, but I didn't get the basic version (whoops).  Now, the advanced version breaks out every pin of the FT232R, but it doesn't have a direct pinout to an FTDI cable such as what the [FT232R Basic](http://www.sparkfun.com/products/9716) supplies.  Oh, and it's set by default to 3.3v, but that's adjustable through a solder jumper.

I needed an FTDI adapter and a way to switch it between 5v and 3.3v sensibly.  No problem, as it turned out.

The very first thing to do is to desolder the solder jumper on the top of the board.  That jumper fixes the VccIO line to 3.3v, which is not what we want.  We want it to be 5v in this case.

Second thing is to come up with a mapping of pins on the FT232R to pins on a normal FTDI cable.  Consulting the [schematic](http://www.sparkfun.com/datasheets/DevTools/Arduino/FTDI%20Basic-v21-5V.pdf) for the basic version makes this pretty simple.  Then we simply map that back to what we have on the breakout board, add in a line to link up VccIO to 5v, and we're sorted;

      FTDI Adapter - 5 volt version

You can find the PDF and Eagle schematic files on my repository at [this link](http://code.google.com/p/zencoding-blog/source/browse/#svn%2Ftrunk%2Farduino%2Fftdi_5v_adapter).  Take careful note, the FTDI connector on that view is from the **top**, whereas the view of the breakout board is from the **bottom**.  Be careful not to wind up with one of the connectors upside down.  I've labelled every pin to reduce confusion in that regard.

Making a 3.3v version of this should be trivial, simply change the link from VccIO to 5v by moving it to 3.3v.  You could even do it with a switch, but I wouldn't advise that because fumble fingering it runs the risk of blowing up whatever you plug it into.  Just make two boards, and clearly label them.

Right now, I've only got it on a breadboard, but once my female pin headers come in, I'll build it onto a piece of stripboard to make a permanent adapter.

license: MIT

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
# Vecty Router

A declarative client-side router for [Vecty](https://www.github.com/hexops/vecty) applications.
Similar to [react-router v4](https://github.com/ReactTraining/react-router)

### Installation

`go get marwan.io/vecty-router`

### Usage

You don't need to declare your routes at the top level. You can declare them inside any component
and if they match they will render, otherwise, router will render an empty div instead.

```go
package components

import (
	"github.com/hexops/vecty"
	"github.com/hexops/vecty/elem"
	"marwan.io/vecty-router"
)

// Body renders the <body> tag
type Body struct {
	vecty.Core
}

// Render renders the <body> tag with the App as its children
func (b *Body) Render() vecty.ComponentOrHTML {
	return elem.Body(
		router.NewRoute("/", &MainView{}, router.NewRouteOpts{ExactMatch: true}),
		router.NewRoute("/blog", &Blog{}, router.NewRouteOpts{}),
		router.NewRoute("/blog/{id}", &PostView{}, router.NewRouteOpts{ExactMatch: true}),
	)
}
```

To retrieve a named variable like {id} in the example above you can do

```go
// Render returns every title
func (pv *PostView) Render() vecty.ComponentOrHTML {
	id := router.GetNamedVar(pv)["id"]
	return elem.Div(
		vecty.Text(id),
	)
}
```

### Other features

##### Navigation through links

```go
func (c *component) Render() vecty.ComponentOrHTML {
	return elem.Span(
		router.Link("/my/route", "click here", router.LinkOptions{}),
	)
}
```

##### Programatically navigate to a route

```go
router.Redirect("/my/route")
```

### Status

Currently vecty-router does not fallback to hash routing if the History API is not on your browser.
It also calls vecty.Rerender on all routes whenever a route changes. It should/will do its own deducing of whether to call rerender on a route or not based on route matches and whether it's already mounted or not.

### Alternatives

- https://github.com/go-humble/router

repository: marwan-at-work/vecty-router

gha_language: Go

repo_name: marwan-at-work/vecty-router


text:
---
layout: page
title: Technical proficiency
---

#### What is the overall quality of your code like?

# ★ ★ ★

I think that my coding is improving. After selecting a template, I felt it would be nice to restructure
the template to have separate files, instead of one.

The initial template had HTML, CSS and Javascript codes all in one file!.

So, basically, I cleaned the whole file up, which I think is a good coding style.

--------------------------------------------------------------------------------------

#### How well did you follow best practices in development?

# ★ ★ ★ ★

An example of this is found in the **[timeline](http://localhost:4000/refugeehub/2017/09/19/meeting-with-client.html)** of this portfolio.

As part of Agile process, after reaching a certain stage in the development we met with our client, for a feedback on the project.

So, I feel I have been able to follow best practices in development.

--------------------------------------------------------------------------------------

#### How well did you use appropriate version control?

# ★ ★ ★ ★ ★

I have made sure that this was paramount after making any changes to the source code files.

To avoid any loss of files and documents, other than saving the files securely locally, I always
push to Git.

It also helps to be able to work easily from wherever I have access to the internet, as I will only need to `clone`
to a new local machine and `pull` to an existing one.

Here is snapshot of my commits on Git;

![Commits](/assets/img/commits.png)

--------------------------------------------------------------------------------------

#### To what extent do you think you contributed an equal portion of the overall project?

# ★ ★ ★ ★

I believe I have contributed a good deal to the project. I ensure that I am on task, do some extra work outside of class
and attend meetings.

In addition, at meetings, I have been able to lead scrums and manage the team.

repository: olusinae/refugeehub

repo_name: olusinae/refugeehub


text:
# RMV Timetable Viewer

This is a small piece of software allowing bus and tram departures for some station to be shown on a small e-ink display.

It is specific for the RMV, the *Rhein-Main-Verkehrsverbund*, a regional Hessian public transport provider.
It interfaces with their [open data portal](https://opendata.rmv.de/), but should work with most HAFAS APIs with only minor modifications.

![](example.jpg)

## Hardware

It uses an (overkill) Raspeberry Pi 3, as the case was already designed for that.
The case can be 3D printed from the files given in `3d-print-template/`.
The display we used was the *2.9inch E-Ink Module 296x128*/*2.9inch e-Paper Module* by *Waveshare*. Links to: [Amazon](https://www.amazon.de/gp/product/B071JFRV2S/), [Waveshare Wiki](https://www.waveshare.com/wiki/2.9inch_e-Paper_Module). The required "drivers" are in `epd2in9/`.

## Installation

This assumes that you have an API token obtained from [RMV Open Data](https://opendata.rmv.de/site/anmeldeseite.html).

1. Download and install [*Raspbian Lite*](https://downloads.raspberrypi.org/raspbian_lite_latest). This was tested with version *Buster*, late 2019. Also configure it such that internet access is possible. See [headless setup](https://www.raspberrypi.org/documentation/configuration/wireless/headless.md).

2. (Optional.) Update the system initially and reboot.
```bash
sudo apt update
sudo apt full-upgrade
sudo reboot now
```

3. (Optional.) Enable automatic updates.
```bash
# install:
sudo apt install unattended-upgrades -y
# configure:
sudo dpkg-reconfigure -plow unattended-upgrades
```

4. Install some dependencies
```bash
sudo apt install git python3-pip python3-pil python3-spidev python3-rpi.gpio python3-dateutil fonts-lato -y
```

5. [Enable SPI](https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md).

6. Adjust the timezone to (for example) Germany/Berlin. This can be done from within `sudo raspi-config` and checked with the `date` command. Restart after this just to be sure. ;)

7. Clone the git repo onto the Pi. This assumes that the code was cloned into `/home/pi`, making the entry point sit at `/home/pi/rmv-rpi-display/app.py`.
```bash
git clone https://github.com/felixdivo/rmv-rpi-display.git
```

8. Adjust `API_TOKEN`, `ORIGIN_ID` and `DESTINATION_ID` in [`rmv_api.py`](rmv_api.py). The IDs are taken from the first row ("HAFAS_ID") of the official list of stations found on the [RMV Open Data platform](https://opendata.rmv.de/site/files/rmv01/RMV_Haltestellen.zip). Test execute it with:
```bash
cd the/directory/where/you/placed/it/
python3 app.py
```

9. Make it start automatically. Add the following line on the line before the final `exit 0` of `/etc/rc.local` (if you cloned it into `/home/pi`):
```bash
su pi -c "python3 /home/pi/rmv-rpi-display/app.py &"
```

10. Pure happiness and never being late for the bus or tram.

## Copyright

```
MIT License

Copyright (c) 2019 Felix Divo & Jonas Weidmann

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```

repository: felixdivo/rmv-rpi-display

repo_name: felixdivo/rmv-rpi-display


text:
---
templateKey: blog-post
title: How to use System.Web in a Console Application
path: blog-post
date: 2012-03-16T03:36:00.000Z
description: "I’ve been bitten by this and have seen others run into it enough
  times that I thought I’d blog about it. Let’s say you’re creating a new
  Console, WPF, or Windows Forms application in .NET 4. You’re using Visual
  Studio 2010, and everything is going great until you get to the part where you
  wanted to make an HTTP request. "
featuredpost: false
featuredimage: /img/snaghtml59c784_thumb.png
tags:
  - .net
  - visual studio
category:
  - Software Development
comments: true
share: true
---
I’ve been bitten by this and have seen others run into it enough times that I thought I’d blog about it. Let’s say you’re creating a new Console, WPF, or Windows Forms application in .NET 4. You’re using Visual Studio 2010, and everything is going great until you get to the part where you wanted to make an HTTP request. You know you can do this, you’ve done it before, heck, you might even be copying code straight out of MSDN that does what you need. But it doesn’t compile. If you try and add a reference to System.Web, you’ll find it’s not there:

![](/img/snaghtml53c449_1.png)

If you look closely, though, you’ll see at the top of the dialog it says “Filtered to: .NET Framework 4 Client Profile”. That’s the issue. Now, if you really are doing HTTP requests in a console application in .NET 4, then I recommend you use the System.Net.HttpWebRequest type that \*is\* available in the Client profile. But for argument’s sake, if you really do need something in one of the filtered out assemblies, like System.Web, then you simply need to change your project type to use the full .NET 4 Framework. You can do that by going into project properties and changing the Target framework from .NET Framework 4 Client Profile to simply .NET Framework 4.

![](/img/image_3_console_app.png)

Having done that, you’ll then be able to reference System.Web again in your project. Hope that helps!

![](/img/snaghtml59c784_thumb.png)

repository: ShadyNagy/ardalis-com-gatsby

branch_name: refs/heads/main

gha_license_id: MIT

repo_name: ShadyNagy/ardalis-com-gatsby


text:
---
title: "Security"
date: 2017-11-05T10:24:06-05:00
description: ""
categories: [concerns]
keywords: []
menu:
  docs:
    parent: "concern"
    weight: 93
weight: 93	#rem
aliases: []
toc: false
draft: false
---

API security is very important and today most APIs don't have security built-in at all but
relying on third party API gateway to handle the security at network level. This assumes
that within the organization, it is safe or some sort of firewall setup to ensure that
only Gateway host can access the service host.

Once we move to the cloud, everything is dynamic and firewall won't work anymore as services
can be down on one VM but started in another VM immediately with container orchestration tools.

The traditional gateway also add another layer of network calls and if there are so many
service to service calls, the added up latency is unacceptable.

Here is an [gateway article][] in the architecture section that talks about the drawbacks of the API
gateway and why it is not suitable for microservices architecture.

In order to make sure that security works for microservies, we have extended the OAuth 2.0
specification and come up with a brand new approach for authorization with JWT. Here is an
[security article][] talks about the API security in light*4j frameworks.

Basically, we have [light-oauth2][] OAuth 2.0 provider and built-in JWT security verification
in the frameworks. The security policy is managed by the OAuth2 provider with service registration
and client registration; however, the policy enforcement is done distributed at each service level.
With PIK signed JWT token issued from the light-oauth2, all services can verify the token with the
JWT signature public key certificate.

The following is a list of key components in light-4j security.

## light-oauth2 server

Light-4j and related frameworks support OAuth 2.0 and related specifications to authorize service
access. By default, the framework contains two pairs of public key certificates issued by our own
testing oauth2 server which can be installed from Docker. For more info, please refer to
https://github.com/networknt/light-oauth2

Light-4j also provides a client module that can communicate with light-oauth2 in the background to
get access token and renew the access token before it is expired.

## Kid

Since services are deployed in the cloud without static IP, the traditional push certificates to each
service is not working anymore. In this framework each service will pull certificate from OAuth2 server
key service by a kid from JWT token header if the key doesn't exist locally. This approach that uses
kid to identify public key certificate to verify the JWT token also helps if you have more than one key
used on your OAuth 2.0 provider. One use case is that you must support at least two keys during key
rotation on a yearly basis due to maximize the security.

## JwtMockHandler

This is a testing OAuth2 endpoints provider and it can be injected into the handler chain for unit
testing so that it won't depend on an instance of light-oauth2 for unit tests. It can be used by other
frameworks and also services developed on top of these frameworks.

## Long lived token

To make integration test easier, a long lived token is provided by the oauth2
server and it can be found at https://github.com/networknt/light-oauth2 README.md

## Configuration

Here is the default security.yml

```yaml
# Security configuration in light framework.
---
# Enable JWT verification flag.
enableVerifyJwt: true

# Enable JWT scope verification. Only valid when enableVerifyJwt is true.
enableVerifyScope: true

# User for test only. should be always be false on official environment.
enableMockJwt: false

# JWT signature public certificates. kid and certificate path mappings.
jwt:
  certificate:
    '100': oauth/primary.crt
    '101': oauth/secondary.crt
  clockSkewInSeconds: 60

# Enable or disable JWT token logging for audit. This is to log the entire token
# or choose the next option that only logs client_id, user_id and scope.
logJwtToken: true

# Enable or disable client_id, user_id and scope logging if you don't want to log
# the entire token. Choose this option or the option above.
logClientUserScope: false

# If OAuth2 provider support http2 protocol. If using light-oauth2, set this to true.
oauthHttp2Support: true

# Enable JWT token cache to speed up verification. This will only verify expired time
# and skip the signature verification as it takes more CPU power and long time.
enableJwtCache: true
```

### Enable security

To enable security, just update enableVerifyJwt to true. If you want to verify the scopes defined in
swagger.json against the scope registered on OAuth 2.0 provider, then set enableVerifyScope to true.
It is highly recommended to have both enabled on production.

### Enable mock JWT generation

If you want to test with different payloads in JWT token or other parameters with JWT token generation,
then you can set enableMockJwt to true; however, you'd better to create a new security.yml and put it
into src/test/resources/config folder so that it works only with your test cases.

### Public key certificates

The light-4j framework supports multiple public key certificates and they are defined in the jwt section
with kid that mapped to the certificate file in oauth sub folder from config folder. The light-codegen
generates two public key certificates to bootstrap the service startup and you should replace them with
your own certificate for production usage.

### Clock Skew in seconds

This is part of the jwt configuration and default is 60 seconds. It means if the JWT token receives are expired
already but still within 60 seconds from expires time, then it still consider valid as computer clocks might
not synced perfectly and there are network and prcessing latency as well.

### logJwtToken and logClientUserScope

The JWT token is considered sensitive so the customer needs make decision on how the information will be
logged. The logJwtToken will just log the raw token itself and logClientUserScope will log the three most
important piece of info only. Normally, you choose one of them or disable both. It doesn't make sense to
log both as information is duplicated.

### oauthHttp2Support

To indicate if the OAuth 2.0 provide support HTTP 2.0 protocol. The default is true as we are assuming
light-oauth2 is used.

### enableJwtCache

Cache the JWT token within the period of expiry so that we can only check if the token is not expired
without verify the signature as it is very time consuming and CPU intensive. Default is true and you
should only turn it off if you have a very good reason. For example, token chaining is used and every
request has a unique token so cache doesn't make sense in the case.

[gateway article]: /architecture/gateway/
[security article]: /architecture/security/
[light-oauth2]: /service/oauth/

repository: JesterCheung/light-doc

repo_name: JesterCheung/light-doc


text:
# miles.ink

  > The writing homepage for Miles Rausch.

[![Build Status](https://travis-ci.org/awayken/miles.ink.svg?branch=master)](https://travis-ci.org/awayken/miles.ink)

[Miles.ink](https://miles.ink) is the homepage to learn more about Miles Rausch as a writer. It's powered by [Eleventy](https://www.11ty.dev/) and written by me, Miles Rausch.

**WHAT??** Yes, the writer, Miles Rausch, is also [the developer, Miles Rausch](https://github.com/awayken/).

What's Miles.ink made out of?

  * [git](http://git-scm.com/)
  * [GitHub](https://github.com/)
  * [Eleventy](https://www.11ty.dev/)
  * [Travis CI](https://travis-ci.org/)

## Generating Miles.ink

Miles.ink is built on Eleventy. To see the site, we have to generate it. Eleventy builds static sites from all sorts of magic parts. There's data, posts, layouts and CSS to do styling.

To generate the site:

    npm run build

## Serving Miles.ink

Eleventy has a built-in server that can watch your files for updates and refresh the page for you. Cool? Def.

To serve the site:

    npm start

## Deploying Miles.ink

Miles.ink uses a Node script to deploy itself. To take advantage of `deploy.js`, you need to define some environment variables. The project includes the `dotenv` npm package to help facilitate this process. Make a copy of `sample.env`, rename it to `.env` and then fill out the values appropriately for your target FTP server. DO NOT commit your new `.env` file. It is ignored for a reason: to keep your secrets safe.

To deploy the site:

    npm run deploy

gha_language: HTML


text:
# PsRemoteImage

UPsRemoteImage is a UMG wrapper for "HTTP/REST image" with caching.

## Usage:

Just add an instance of `UPsRemoteImage` on palette and specify URL.
Should you wish to clear local image cache, please use the appropriate `UPsRemoteImageLibrary` function.

repository: californiawz/PsRemoteImage

repo_name: californiawz/PsRemoteImage


text:
# Unit Voice-overs

**Current Version:** 1.6 "Suppression Update"

See our Wiki for more documentation: https://github.com/the0utsider/unit-voiceovers/wiki

**Authors**

654wak654,
Gokmen,
Sceptre

**Description**

Dynamic, randomized 3D positional sound fx for units in Arma.

Using NWI's 'Insurgency' sound samples, this mod was created to improve the overall experience and bring dynamism to the in-game voices in Arma 3.

**Features**
- A whole new set of voices for various actions in Arma 3
- Opfor, independent, and blufor sides use different sets of voices
- Randomized samples for each action
- Spot enemy / callout targets direction (Default key 'T')
- ACE3 Medical compatibility
- Custom Nationality system
- Almost 1000 different sound samples total
- Supports transferring unit locality
- Works independent from 'enableSentences false' setting

**Customization**
- CBA Settings support
- 'enableSentences' CBA setting
- Toggle voice-overs for players
- Toggle ambient sounds
- Ambient radio sound settings
- Ratio customization for certain sounds
- Set volume level and travel distance to your liking
- Sound sample pitch adjustment
- Custom faction/nationality sound declaration system
- Rebind/change 'Callout' key via CBA add-on controls
- Unit disable/init function

**Dynamic voice-over events:**
- Magazine goes empty/reloading
- Enemy visibly killed
- Getting wounded
- Friendly fire
- Player spots enemy (via 'Callout' key)
- Using throw-able or plant-able items
- Low on ammo
- A hostile frag grenade is thrown nearby
- A nearby friendly is killed
- Ambient sounds at random intervals
- Units will be quiet when in stealth mode

# Installation

**Requirements**

CBA_A3: https://steamcommunity.com/sharedfiles/filedetails/?id=450814997

**Download**

Subscribe via Steam: http://steamcommunity.com/sharedfiles/filedetails/?id=1248372833

---

The MIT License (MIT)

Copyright (c) 2018 Gokmen 'the0utsider'

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repository: Reidond/unit-voiceovers

repo_name: Reidond/unit-voiceovers


text:
---
layout: layout.pug
navigationTitle:
excerpt: Overview of job scheduling options
title: Job Scheduling
menuWeight: 110
featureMaturity:
model: /services/spark/data.yml
render: mustache
---

This document is a simple overview of material described in greater detail in the Apache Spark documentation [here][1]
and [here][2].

# Modes

Spark on Mesos supports two modes of operation: coarse-grained mode and fine-grained mode. Coarse-grained mode
provides lower latency, whereas fine-grained mode provides higher utilization. More info [here][2].

## Coarse-grained mode

"Coarse-grained" mode is so-called because each Spark executor is represented by a single Mesos task. As a result,
executors have a constant size throughout their lifetime.

*   **Executor memory**: `spark.executor.memory`
*   **Executor CPUs**: `spark.executor.cores`, or all the cores in the offer.
*   **Number of Executors**: `spark.cores.max` / `spark.executor.cores`. Executors are brought up until
    `spark.cores.max` is reached. Executors survive for duration of the job.
*   **Executors per agent**: Multiple

**Note:** We highly recommend that you [set `spark.cores.max`](#set-spark-cores-max). If you do not, your Spark job may consume all available resources in your cluster, resulting in unhappy peers.

# Quotas for Drivers and Executors

Setting [Mesos Quota](http://mesos.apache.org/documentation/latest/quota/) for the Drivers prevents the Dispatcher from consuming too many resources, and assists queueing behavior. To control the concurrent number of Drivers, the
Spark service will run concurrently. We strongly recommend setting a Quota for the Drivers. The Quota will both
guarantee that the Spark Dispatcher has resources available to launch Drivers, and limit the total impact on the
cluster due to Drivers.  Optionally, set a Quota for the Drivers to consume, to ensure that Drivers will not be starved
of resources by other frameworks as well as make sure they do not consume too much of the cluster (see coarse-grained
mode above).

## Setting Quotas for the Drivers

Setting a Quota for the Drivers allows the cluster administrator to ensure that only a given number of Drivers are concurrently running. As additional Drivers are submitted, they will be enqueued by the Spark Dispatcher. Below are the recommended steps for setting Quotas for the Drivers:

1.  Set the Quota conservatively, keeping in mind that it will affect the number of jobs that can run concurrently.
1.  Decide how much of your cluster's resources to allocate to running Drivers. These resources will only be used for
    the Spark Drivers, meaning that here we can decide roughly how many concurrent jobs we would like to have running at a time. As additional jobs are submitted, they will be enqueued and run with first-in-first-out semantics.
1.  For the most predictable behavior, enforce uniform Driver resource requirements and a particular Quota size for the
    Dispatcher.  If each Driver consumes 1.0 CPU and it is desirable to run up to five Spark jobs concurrently, a Quota with 5 CPUs should be created:

Example:

SSH to the Mesos master and set the Quota for a role (`dispatcher` in this example):

```bash
$ cat dispatcher-quota.json
{
 "role": "dispatcher",
 "guarantee": [
   {
     "name": "cpus",
     "type": "SCALAR",
     "scalar": { "value": 5.0 }
   },
   {
     "name": "mem",
     "type": "SCALAR",
     "scalar": { "value": 5120.0 }
   }
 ]
}
$ curl -d @dispatcher-quota.json -X POST http://<master>:5050/quota
```

4. Install the Spark service with the following options (at a minimum):

```bash
$ cat options.json
{
    "service": {
        "role": "dispatcher"
    }
}
$ dcos package install spark --options=options.json
```

## Setting Quotas for the Executors

It is recommended to allocate Quotas for Spark job executors.  Allocating Quotas for the Spark executors provides:
-  A guarantee that Spark jobs will receive the requested amount of resources.
-  Additional assurance that even misconfigured Spark jobs (for example, a Driver with unset `spark.cores.max`)  do not consume too many resources, impacting other tenants on the cluster.

The drawback to allocating Quotas to the Executors is that Quota resources cannot be used by other frameworks in the
cluster.

Quotas can be allocated for Spark executors in the same way they are allocated for Spark dispatchers.  If we assume we want to be able to run 100 executors concurrently, each with 1.0 CPU and 4096 MB of memory, we should do the following:

```bash
$ cat executor-quota.json
{
  "role": "executor",
  "guarantee": [
    {
      "name": "cpus",
      "type": "SCALAR",
      "scalar": { "value": 100.0 }
    },
    {
      "name": "mem",
      "type": "SCALAR",
      "scalar": { "value": 409600.0 }
    }
  ]
}
$ curl -d @executor-quota.json -X POST http://<master>:5050/quota

```

When Spark jobs are submitted, they must indicate the role for which the Quota has been set in order to consume resources from this quota.

Example:

```bash
$ dcos spark run --verbose --name=spark --submit-args="\
--driver-cores=1 \
--driver-memory=1024M \
--conf spark.cores.max=8 \
--conf spark.mesos.role=executor \
--class org.apache.spark.examples.SparkPi \
http://downloads.mesosphere.com/spark/assets/spark-examples_2.11-2.0.1.jar 3000"

```

### Special considerations for streaming and long-running Spark jobs

To prevent a single long-running or streaming Spark job from consuming the entire Quota, the max CPUs for that Spark job should be set to roughly one “job’s worth” of the Quota’s resources. This ensures that the Spark job will get sufficient resources to make progress, and setting the max CPUs ensures that it will not starve other Spark jobs of resources; it will also predictably offer suppression semantics.

## Permissions when using Quotas with Strict mode

Strict mode clusters (see [security modes](https://docs.mesosphere.com/1.10/security/ent/#security-modes)) require extra permissions to be set in order to use Quota. Follow the instructions in [installing](https://github.com/mesosphere/spark-build/blob/master/docs/install.md) and add the additional permissions
for the roles you intend to use, detailed below. Following the example above they would be set as follows:

1.    First set Quota for the Dispatcher's role (`dispatcher`)

    ```bash
    $ cat dispatcher-quota.json
    {
     "role": "dispatcher",
     "guarantee": [
       {
         "name": "cpus",
         "type": "SCALAR",
         "scalar": { "value": 5.0 }
       },
       {
         "name": "mem",
         "type": "SCALAR",
         "scalar": { "value": 5120.0 }
       }
     ]
    }
    ```

     Download the CA certificate,`dcos-ca.crt` to your local machine via the `https://<dcos_url>/ca/dcos-ca.crt` endpoint.

    ```bash
    curl -X POST --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/mesos/quota -d @dispatcher-quota.json -H 'Content-Type: application/json'
    ```

     Then set the Quota **from your local machine**.

2.    Optionally set Quota for the executors also, this is the same as above:

    ```bash
    $ cat executor-quota.json
    {
      "role": "executor",
      "guarantee": [
        {
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 100.0 }
        },
        {
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 409600.0 }
        }
      ]
    }
    ```

    Then set the Quota from your local machine, again assuming you have `dcos-ca.crt` locally:

    ```bash
    curl -X POST --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/mesos/quota -d @executor-quota.json -H 'Content-Type: application/json'
    ```

3.    Install Spark with these minimal configurations:

    ```bash
    {
        "service": {
                "service_account": "spark-principal",
                "role": "dispatcher",
                "user": "root",
                "service_account_secret": "spark/spark-secret"
        }
    }
    ```

4.    Now you are ready to run a Spark job using the principal you set and the roles:

    ```bash
    dcos spark run --verbose --submit-args=" \
    --conf spark.mesos.principal=spark-principal \
    --conf spark.mesos.role=executor \
    --conf spark.mesos.containerizer=mesos \
    --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com/spark/assets/spark-examples_2.11-2.0.1.jar 100"
    ```

## Setting `spark.cores.max`

To improve Spark job execution reliability, set the maximum number of cores consumed by any given job.  This prevents
any particular Spark job from consuming too many resources in a cluster.  It is highly recommended that each Spark job be submitted with a limit on the maximum number of cores (CPUs) it can consume. This is especially important for
long-running and streaming Spark jobs.

```bash
$ dcos spark run --verbose --name=spark --submit-args="\
--driver-cores=1 \
--driver-memory=1024M \
--conf spark.cores.max=8 \ #<< Very important!
--class org.apache.spark.examples.SparkPi \
http://downloads.mesosphere.com/spark/assets/spark-examples_2.11-2.0.1.jar 3000"
```

When running multiple concurrent Spark jobs, consider setting `spark.cores.max` between
`<total_executor_quota>/<max_concurrent_jobs>` and `<total_executor_quota>`, depending on your workload characteristics
and goals.

# Fine-grained mode

**Note** Fine-grained mode has been deprecated and does not have all of the features of coarse-grained mode.

In "fine-grained" mode, each Spark task is represented by a single Mesos task. When a Spark task finishes, the
resources represented by its Mesos task are relinquished. Fine-grained mode enables finer-grained resource allocation at the cost of task startup latency.

*   **Executor memory**: `spark.executor.memory`
*   **Executor CPUs**: Increases and decreases as tasks start and terminate
*   **Number of Executors**: Increases and decreases as tasks start and terminate
*   **Executors per agent**: One at most

# Properties

The following is a description of the most common Spark on Mesos scheduling properties. For a full list, see the [Spark
configuration page][1] and the [Spark on Mesos configuration page][2].

Property
Default
Description

`spark.mesos.coarse`
True
Described above.

`spark.executor.memory`
1 Gb
Executor memory allocation.

`spark.executor.cores`
All available cores in the offer
Coarse-grained mode only. DC/OS Apache Spark >= 1.6.1. Executor CPU allocation.

`spark.cores.max`
Unlimited
Maximum number of cores to allocate.

[1]: http://spark.apache.org/docs/latest/configuration.html
[2]: http://spark.apache.org/docs/latest/running-on-mesos.html

repository: ShaunDeng/dcos-docs-site

repo_name: ShaunDeng/dcos-docs-site


text:
---
layout: post
title: "Dying for Sex Podcast"
excerpt: "When host Nikki Boyer’s best friend Molly is diagnosed with Stage IV breast cancer, she decides to do something bold..."
date: 2020-03-11 22:25:02 -0500
categories:
---

The [Dying For Sex Podcast](https://wondery.com/shows/dying-for-sex/):

>When host Nikki Boyer’s best friend Molly is diagnosed with Stage IV breast cancer, she decides to do something bold.

You subscribe because sex! Then you get to the last episode and if you've ever lost anyone you sob while drying dishes, and feel connected to humanity at a deeper level for having listened.

repository: dealingwith/poole

extension: markdown

gha_language: CSS

repo_name: dealingwith/poole


text:
---
id: installation
title: Installation
sidebar_label: Installation
---

## Installing uStart

Ensure you have the latest [Node](https://nodejs.org/en/download/) installed

> You have to be on Node >= 8.x and Git

Create a new project using the CLI. The `init` command will ask you four questions: answer **no** to all of them. This will create the project structure and install all NPM dependencies for you

```shell
npx ustart-cli init awesome-project
```

We use `npx` to avoid a global installation of the CLI. Once npx finish the `ustart-cli` will be available as a local package by typing `npx ustart <command>`.

## Run the project

uStart supports direct execution and docker containers. If you are under a Windows system use the Docker way.

### UNIX and friends (Linux, OSX, etc)

Run your project directly with NPM

```shell
cd awesome-project
npm run start
```

### Docker (UNIX, Linux, OSX, Windows)

First, create a volume using the terminal

```shell
cd awesome-project
docker volume create nodemodules
```

Second, run the docker compose builder file with `install` flag

```shell
docker-compose -f docker-compose.builder.yml run --rm install
```

Third, run it

```shell
docker-compose up
```

## Verifying Installation

If everything has gone well your console will log:
```
NODE_ENV: development
Server is running on http://localhost:4000
```

Good, that means you have running the playground on `localhost` port `4000`.

Execute your first query. Open your browser and open [localhost:4000](http://localhost:4000), go to the left panel and type:
```graphql
query {
  hello
}
```

Press the play icon in the middle of both panels, then the right panel should show:
```graphql
"data": {
  "hello": "Hello there, everything is right. You can keep reading the docs!"
}
```

If the response is the same you are ready to play!.

repository: ustart-dev/ustart

gha_language: JavaScript

repo_name: ustart-dev/ustart


text:
# CFG implemented in Lex and YACC

Lex and yacc help you write programs that transform structured input. This includes
an enormous range of applications—anything from a simple text search program that
looks for patterns in its input file to a C compiler that transforms a source program into
optimized object code.

One of those applications are the context-free grammars (CFG). CFG are a more
powerful method of describing languages. Such grammars can describe certain
features that have a recursive structure, which makes them useful in a variety
of applications.

## Activities

Please create a compiler for the Advance Calculator programing language:

Types In ac, there are only two data types:
    Integer: a sequence of decimal numerals
    Float : five fractional digits after the decimal point

Keywords In ac, there are three reserved keywords, each limited for simplicity
to a single letter:
    f (declares a float variable)
    i (declares an integer variable)
    p (prints the value of a variable).

Variables The ac language offers only 23 possible variable names, drawn from
the lowercase Roman alphabet and excluding the three reserved keywords f, i,
and p. Variables must be declared prior to using them.

repository: VictorRodriguez/compilers-lecture

gha_language: C

repo_name: VictorRodriguez/compilers-lecture


text:
<!--title={isValid()}-->

<!--badges={Algorithmns:18}-->

<!--concepts{Indexing 2D Lists, For Loops}-->

# isValid() Function

Starting off with the function definition accepting the `grid` as an argument, we will have:

```python
def isValid(grid)
```

Now since we know that we need to check if there are no repeats in any rows or columns within `grid`, we will need two nested for-loops: one to check that all rows contain no repeats and another to check that all columns contain no repeats. If both nested for-loops iterate through the grid successfully, then the grid is deemed valid so return True.

```python
def isValid(grid)
  for row in grid:
    for i in row:
      # implement logic here to check for repeats in all rows

  for c in range(0,9):
    for r in range (0,9):
      # implement logic here to check for repeats in all columns

  return True
```

For the first nested for-loop, the outer-loop iterates through each `row` in `grid` :

```python
for row in grid:
```

and then the inner-loop iterates through each element `i`  in  `row`:

```python
for i in row:
```

This is because each of the 9 lists inside of `grid` represents each `row`, and thus you can simply iterate through the elements in a `row` to make comparisons for checking repeated values.

Now notice that for the second nested for-loop, we are iterating through  `range(0,9)`, which is looping from 0 to 9, in both inner and outer loops. This is because we must iterate through each `row` at the same index in order to compare the elements within the same column since our `grid` is a list of rows.

```python
for c in range(0,9):
   for r in range (0,9):
```

For example, if we check all the values for the first column (`c == 0`), we will iterate through the rows (`r`) from `0-9`.

Here's a sample representation of how the rows and columns are within a 3x3 2-D list similar to `grid`:

![](https://4.bp.blogspot.com/-6EPvK1jRrqE/VemgzdJu4ZI/AAAAAAAADsk/QnkWDlg6zpI/s1600/how%2Bto%2Bloop%2Bover%2B2D%2Barray%2Bin%2BJava.jpg)

repository: vvennela/curriculum

repo_name: vvennela/curriculum


text:
# ATLA-Line-Visualization

Easy instructions for viewing the diagram:
1) Navigate to https://raw.githubusercontent.com/Joey5729/ATLA-Line-Visualization/master/out.html
2) Download the file as a .html (right click the page, save as, name it something.html)
3) Open the file with a web browser

Inspired by this Reddit post:
https://www.reddit.com/r/dataisbeautiful/comments/g2f3ip/oc_most_popular_movie_genre_combinations

Avatar: The Last Airbender was released on US Netflix on May 15, 2020. This just so happens to align with the end of finals for me, so I've decided to rewatch it and see what combinations of characters had the most lines spoken to each other and model them in a fun, interactive way.

I identified the following characters as important enough to be counted:

Aang, Katara, Sokka, Toph, Zuko, Iroh, Suki, Azula, Ozai, Mai, and Ty Lee (Sorry to all the Boomy, Gran-Gran, and Sozin fans out there)
All disguises/aliases (Painted Lady, Blind Bandit, Blue Spirit, Lee, Mushi, Wang Fire, Melon Lord, etc) are counted as the person they really are

For the purposes of this project, I defined a line of dialouge as a block of speech spoken by one character to another, unbroken by: the interuption of the block by another character speaking (including the person the speaker is talking to or other characters that aren't a part of this project), the speaker switching the addresse of their speaking, a scene change, or the end of an episode.

Lines spoken from a character not listed to one that is (and vice versa) are not counted.

Lines spoken during dream sequences/flashbacks/visions/episode recaps are not counted.

Notable one-off cases (major spoilers):

107; Winter Solstice- Part 1- The Spirit World: Aang tries the talk to Katara when he's in the spirit world, but she can't hear him. The first line was counted, since he thought he was in conversation; subsequent lines were not.

112; The Storm: Iroh's story to the Lieutenant portrays dialouge amongst himself, Zuko, and Ozai. I consider this a flashback, so the lines are not counted.

120; The Siege of the North- Part 2: Zuko directly addresses Aang's body while he is in the spirit world. I don't consider this dialouge, so I didn't count these lines.

207; Zuko Alone: This episode famously features no recurring characters but Zuko, apart from flashbacks featuring himself, Azula, Mai, Ty Lee, and Ozai. I didn't want to put an empty document through the program so I just didn't make a file for this episode.

212; The Serpent's Pass: Toph addresses Suki as if she was Sokka (you know exactly what I'm talking about); I counted this as Toph speaking to Suki.

216; Appa's Lost Days: This entire episode might be a flashback. I decided to count the lines in this episode anyway. Sue me.

219; The Guru: Katara addresses Azula as if she was Suki; I counted this as Katara speaking to Azula (and vice versa).

307; The Runaway: I consider the two lines that open the episode are a flash-forward of sorts. I count them the second time they are said in the episode, not the first.

309; Nightmares and Daydreams: Which of Aang's hallucinations are a dream and which are real isn't well defined. These are the scenes I decided were dreams, and therefor aren't counted: the pants nightmare, the math test nightmare, Ozai telling Aang he slept in, "Baby, you're my forever girl," and the UNO reverse card pants nightmare. I counted the lines spoken by and to Sokka, Katara, and Toph during the duel sequence because I think they were actually spoken.

317; The Ember Island Players: Lines spoken by/to the players don't count as lines spoken by/to the characters themselves.


text:
# Biscogniauxia maritima Lar.N. Vassiljeva, 1988 SPECIES

#### Status
SYNONYM

#### According to
The Catalogue of Life, 3rd January 2011

#### Published in
Mikol. Fitopatol. 22(5): 393 (1988)

#### Original name
Biscogniauxia maritima Lar.N. Vassiljeva, 1988

### Remarks
null

repository: mdoering/backbone

path: /life/Fungi/Ascomycota/Sordariomycetes/Xylariales/Xylariaceae/Biscogniauxia/Biscogniauxia atropunctata/ Syn. Biscogniauxia maritima/README.md

repo_name: mdoering/backbone


text:
Forked from [https://github.com/helm/charts/tree/master/incubator/druid](https://github.com/helm/charts/tree/master/incubator/druid)

# Apache Druid Helm Chart

[Apache Druid](https://druid.apache.org/) is a high performance analytics data store for event-driven data.

#### Why not just use incubator/druid helm chart?

While the official chart is a great start (thanks to maver1ck, the maintainer)
and we shamelessly copied many parts of it,
the official chart is not up to date with the newer version of druid,
and it does not support the apache-druid official docker image.

So we made some changes to match apache-druid 0.16+

### Install the Chart

Add the featurefm helm repo:

```
helm repo add featurefm 'https://raw.githubusercontent.com/listnplay/charts/master/'
```

Create a `my-values.yaml` and make your modifications. Use [my-values.yaml](my-values.yaml) as template.

**Note** The chart will not work properly without *deep storage* defined (e.g. in s3 bucket).
See [my-values.yaml](my-values.yaml) file for s3 example.
See the [Docs](https://druid.apache.org/docs/latest/configuration/index.html#deep-storage)
for details on the deep storage config options for GCP, Azure, etc.

To install the Druid Chart into your `bi` namespace of your Kubernetes cluster:

```bash
helm install --namespace "bi" --name "druid" featurefm/druid -f my-values.yaml
```

### Uninstall
If you want to delete your Chart, use this command:

```bash
helm delete  --purge "druid"
```

### Druid Services

This chart exposes the following Druid services as kubernetes services.
Most likely, you will only need the *router* for UI management, *turnilo* for data exploration,
and the *broker* for client queries.

- druid-router - the new druid UI management console
- druid-broker - entry point for queries
- druid-coordinator (+Overlord)
- druid-historical
- druid-middle-manager (+Peons)

- druid-turnilo - [Allegro](https://github.com/allegro/turnilo)'s
cool (and free) data exploration and visualization UI for druid.

### External Services

The chart install a Druid along with its supporting external services:

- *zookeeper* for leader election and instance discovery
- *postgresql* as the metadata store

these dependencies can be customized in the values.yaml,
or can be disabled altogether if you choose to provide your own installations.

### Druid configuration

The intention of this chart is to make druid easy to install,
yet keep the option to apply more advanced configuration where needed.

for example, the druid extension list can be stated as follow:

    extensions:
      loadList:
      - druid-histogram
      - druid-datasketches
      - druid-lookups-cached-global
      - postgresql-metadata-storage
      - druid-s3-extensions
      - druid-kafka-indexing-service

Any druid configuration can be supplied in the `env` section,
 substitution of dot (.) for an underscore (_);
for example, here's the configuration for s3 bucket as deep storage:

    env:
      # Deep storage
      # specify your bucket name for s3 deep storage & indexer logs
      druid_storage_type: s3
      druid_storage_bucket: <my-druid-deep-storage-bucket>
      druid_storage_baseKey: druid/segments
      druid_indexer_logs_type: s3
      druid_indexer_logs_s3Bucket: <my-druid-deep-storage-bucket>
      druid_indexer_logs_s3Prefix: druid/indexing-logs

Also note that each Druid service can have its own `env` section
to configure that service and possibly override the common `env` configuration.

### Monitoring

Prometheus monitoring can be enabled as follow:

    monitoring:
      enabled: true
      createServiceMonitor: true

and Druid will be configured to emmit metrics to a sidecaer that collects and exposes them to prometheus automatically.
You can use the [grafana dashboard](druid-grafana-dashboard.json) to view these metrics.

![Grafana Dashboard](druid-grafana.png)

### Middle Manager and Historical Persistence Notes

_Middle Managers_ and _Historicals_ use a StatefulSet.
Persistence is enabled by default.

**Note** that while the stateful services can retrieve their copy of the data from the deep storge
if restarted without persistence,
such recovery will take considerable time to complete.

### Other Chart Configuration

Full and up-to-date documentation can be found in the comments of the [values.yaml](values.yaml) file.

repository: ListnPlay/charts

repo_name: ListnPlay/charts


text:
+++
# Accomplishments widget.
widget = "accomplishments"  # See https://sourcethemes.com/academic/docs/page-builder/
headless = true  # This file represents a page section.
active = true  # Activate this widget? true/false
weight = 20  # Order that this section will appear.

title = "Teaching"
subtitle = "Workshops and Portfolio"

# Date format
#   Refer to https://sourcethemes.com/academic/docs/customization/#date-format
date_format = "Jan 2006"

# Accomplishments.
#   Add/remove as many `[[item]]` blocks below as you like.
#   `title`, `organization` and `date_start` are the required parameters.
#   Leave other parameters empty if not required.
#   Begin/end multi-line descriptions with 3 quotes `"""`.

[[item]]
  organization = "RStudio"
  organization_url = "https://education.rstudio.com/trainers/"
  title = "RStudio Instructor Certification"
  url = "https://education.rstudio.com/trainers/"
  certificate_url = "https://github.com/davidjohnbaker1/rstudio_certification_training"
  date_start = "2020-04-02"
  date_end = ""
  description = "Completed both eight hour training course on modern, evidence based teaching methods as well passed two exams on both teaching methods and programming with the tidyverse."
+++

Teaching and research are two sides of the same coin.
Just as it's important to create new ideas, it's equally important to devote time and resources to education in order to give others the tools to create their own
ideas and build stronger communities in the process.

On this page you will find:

* [How to Hire Me for Workshops](#workshops)
* [Examples of Pedagogical Material I've Created](#teachingportfolio)

## Workshops {#workshops}

I offer tranings and workshops on various topics on topics related to data science and am open to designing bespoke materials in order to suit your needs.
Recently I become a certified [RStudio Instructor](https://education.rstudio.com/trainers/), have worked as a Lead Instructor of Data Science at [Flatiron School](www.flatironschool.com), and have also designed and delivered workshops for [Minerva Statistical Consulting](https://minervastatisticalconsulting.co.uk/).
I specialize in teaching beginners with little to no experience in data science.

## Teaching Portfolio {#teachingportfolio}

Below I archive examples of materials I have created for pedagogical purposes.
Here you can find:

* [Examples of data science lesson plans](#datasciencelessonplans)
* Videos of me [delivering materials](#teachingvideos)
* Listings of [university level courses I've taught](#universitymaterials)

### Data Science Teaching Materials {#datasciencelessonplans}

#### Lesson Plans

* [Complete Lesson Plan used for RStudio Teaching Certification](https://github.com/davidjohnbaker1/rstudio_certification_training)

The following are examples of lesson material I wrote for Flatiron School's curriula, examples of documents I've produced to help students, and examples of lesson plans:

#### Curricula Writing

* [R to Python: An Overview](https://github.com/learn-co-curriculum/dsc-r-to-python-an-overview)
* [Installing R and RStudio](https://github.com/learn-co-curriculum/dsc-installing-r)
* [R Base-ics I](https://github.com/learn-co-curriculum/dsc-r-base-ics-1)
* [R Base-ics II](https://github.com/learn-co-curriculum/dsc-r-base-ics-2)
* [Introduction to the Tidyverse](https://github.com/learn-co-curriculum/dsc-intro-to-tidyverse)
* [Introduction to ggplot2](https://github.com/learn-co-curriculum/dsc-intro-to-ggplot2)
* [Introduction to ggplot2 Lab](https://github.com/learn-co-curriculum/dsc-intro-to-ggplot2-lab)

#### Project Templates

* [Project Template for Data Science Capstone](https://github.com/learn-co-students/capstone-template-london-ds)

#### Lesson Materials and Plans

* [Intro to Inferential Statistics](https://github.com/learn-co-students/probability-london-ds)
* [Central Limit Theorem](https://github.com/learn-co-students/distributions-sampling-london-ds)
* [Hypothesis Testing](https://github.com/learn-co-students/hypothesis-testing-london-ds)
* [A/B Testing](https://github.com/learn-co-students/ab-testing-london-ds)
* [Bayes for Days](https://github.com/learn-co-students/bayes-london-ds)
* [Linear Regression](https://github.com/learn-co-students/linear-regression-london-ds )
* [Multiple Regression](https://github.com/learn-co-students/multiple-linear-regression-london-ds)
* [Reg Coefs, EDA, and Train, Test, Split Review](https://github.com/learn-co-students/linear-regression-coef-lab-london-ds)
* [PCA](https://github.com/learn-co-students/pca-london-ds)
* [Clustering](https://github.com/learn-co-students/clustering-london-ds)
* [Recommendation Systems](https://github.com/learn-co-students/recsys-london-ds)
* [Big Data](https://github.com/learn-co-students/big-data-london-ds)
* [NLP](https://github.com/learn-co-students/intro-to-nlp-london-ds)
* [Deep NLP](https://github.com/learn-co-students/deep-nlp-london-ds)

### Videos of Me Teaching {#teachingvideos}

The Science of Netflix

{{<youtube xkMqgWHn36w>}}

R, RStudio, and Tidyverse for Pythonistas

{{<youtube OLbBwr6yono>}}

Clustering Algorithms Crash Course

{{<youtube EKTVTfAXVIo>}}

### University Course Listings {#universitymaterials}

* **Foundations of Music Study**
  - Taught remedial music theory and aural training course three times a week to undergraduates as Instructor of Record
  - Responsible for all course material ranging from lectures, to homework assignments, bi-weekly recitations, to play-and-sing piano preparations, and teaching of aural-skills, held office hours.
  - Homework assigned via Oxford Music Theory Skill Builder
* **Aural Skills IV (Chromatic and Atonal)**
  - Taught Aural Skills Labs to second-year undergraduate students, classes focused on providing individual attention to learning how to sight-sing chromatic and atonal melodies as well as complex rhythms.
  - Responsible for grading bi-weekly individual recitations with materials ranging from prepared melodies, to play-and-sing piano preparations, to sight-singing.
  - Aided Instructor of Record for large Music Theory lecture
* **Aural Skills III (Chromaticism)**
  - Taught Aural Skills Labs to second-year undergraduate students, classes focused on providing individual attention to learning how to sight-sing melodies and rhythms.
  - Responsible for grading bi-weekly individual recitations with materials ranging from prepared melodies,
to play-and-sing piano preparations, to sight-singing
  - Aided Instructor of Record for large Music Theory lecture


text:
# ggside 0.1.2

* Fixed bug where additional messages were printed to the console when using `ggplot2::facet_wrap()` or `ggplot2::facet_grid()`. Addresses issue #20.
* Fixed bug where `geom_xsidetext()` was using the incorrect `Geom` parameter. Addresses issue #19.

# ggside 0.1.1

* fixed bug in which `colour/color`/`fill` appear in guides when used as `params` instead of an `aes()` mapping. Now these aesthetics, and their `x/y` variants should behave like `ggplot2` API. Addresses issue #12.
* `ggside` should now work better with the `patchwork` package. Addresses issue #13.
* Fixed 'bug' in which computed aesthetics did not mix well with main panels of class `ContinuousScaleDate`. Extended better control to `XLayer` and `YLayer` to pull scales from the proper panels instead of using the prototype scales on `plot$scales`. Addresses issue #11.

# ggside 0.1.0

* Various `geom_*side*` have better default calling behavior. See `vignettes("ggside_aes_mapping")` for more details.
* Added various functions to help customize side panels. `xsidey` will affect the y-axis of the xside panel, and `ysidex` will affect the x-axis of the yside panel.
  * `scale_xsidey_continuous`
  * `scale_xsidey_discrete`
  * `scale_ysidex_continuous`
  * `scale_ysidex_discrete`
* Using proper semantic versioning
* Added documentation examples for the following side geometries:
  * `geom_*sidebar`
  * `geom_*sideboxplot`
  * `geom_*sidecol`
  * `geom_*sidedensity`
  * `geom_*sidefreqpoly`
  * `geom_*sideline`
  * `geom_*sidepoint`
  * `geom_*sidesegement`
  * `geom_*sidetile`
  * `geom_*sideviolin`
* Corrected bug where `geom_*sideviolin` was not returning `ggside_layer` class
* Updated `vignette(ggside_basic_usage)` 'How its done' section headers

# ggside <IP_ADDRESS>

* Initial CRAN release version

repository: seifudd/ggside

repo_name: seifudd/ggside


text:
# wksctl Documentation

`wksctl` allows simple creation of a Kubernetes cluster given a **set of IP addresses** and an **SSH key**. It can be run in a standalone environment but is best used via a [GitOps approach](https://www.weave.works/technologies/gitops/) in which cluster and machine descriptions are stored in Git and the state of the cluster tracks changes to the descriptions.

Its features include:

- simple creation of Kubernetes clusters
- manage cluster and machine descriptions using Git
- manage addons like Weave Net or Flux
- Sealed Secret integration

## Install wksctl binary

1. Download the OS specific `wksctl` release package from the [release page](https://github.com/weaveworks/wksctl/releases)
1. Unpack and add the `wksctl` binary to your path

   For example:

   ```console
   cd <download dir>
   tar xfz wksctl-0.7.0-linux-x86_64.tar.gz
   chmod +x wksctl
   sudo mv wksctl /usr/local/bin/
   ```

## Table of contents

- [Get started](get-started.md)
- [WKS and Firekube](wks-and-firekube.md)
- [WKS and Footloose](wks-and-footloose.md)
- [WKS and Vagrant](wks-and-vagrant.md)
- [WKS on GCE](wks-on-gce.md)
- [FAQ](faq.md)
- [Development](development.md)
- [Releasing](releasing.md)

## Getting Help

If you have any questions about, feedback for or problems with `wksctl`:

- Invite yourself to the Weave Users Slack.
- Ask a question on the [#wksctl](https://weave-community.slack.com/messages/wksctl/) slack channel.
- [File an issue](https://github.com/weaveworks/wksctl/issues/new).

Your feedback is always welcome!

repository: luxas/wksctl

repo_name: luxas/wksctl


text:
# AppliAR

[![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rbontekoe.github.io/AppliAR.jl/stable)
[![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://rbontekoe.github.io/AppliAR.jl/dev)
[![Build Status](https://travis-ci.com/rbontekoe/AppliAR.jl.svg?branch=master)](https://travis-ci.com/rbontekoe/AppliAR.jl)
[![Codecov](https://codecov.io/gh/rbontekoe/AppliAR.jl/branch/master/graph/badge.svg)](https://codecov.io/gh/rbontekoe/AppliAR.jl)
[![Coveralls](https://coveralls.io/repos/github/rbontekoe/AppliAR.jl/badge.svg?branch=master)](https://coveralls.io/github/rbontekoe/AppliAR.jl?branch=master)

AppliAR is a package we discuss in the course [BAWJ](https://www.appligate.nl/BAWJ/) (Building Business Applications with Julia).

The package uses the support packages AppliSales and AppliGeneralLedger.

See [https://github.com/rbontekoe/ar](https://github.com/rbontekoe/ar) for the client.ipynb notebook with a Microk8s (ar-storage.yaml, gl-storage.yaml, and cnt-storage.yaml) example.


text:
---
# Documentation: https://sourcethemes.com/academic/docs/managing-content/

title: Oxytocin pathway gene networks in the human brain
subtitle: ''
summary: 'Using voxel-by-voxel gene expression brain maps generated via human post-mortem tissue, this analysis revealed expression of critical oxytocin pathway genes (OXTR, OXT, CD38) are enriched in subcortical and olfactory regions. fMRI meta-analysis revealed that these oxytocin pathway gene maps correspond with the processing of anticipatory, appetitive, and aversive cognitive states.'
authors:
- D. S. Quintana
- Jaroslav Rokicki
- Dennis van der Meer
- Dag Alnæs
- Tobias Kaufmann
- Aldo Córdova-Palomera
- Ingrid Dieset
- Ole A. Andreassen
- Lars T. Westlye
tags: ["oxytocin"]
categories: []
date: '2019-02-01'
lastmod: 2020-08-28T12:57:59+02:00
featured: true
draft: false

# Featured image
# To use, add an image named `featured.jpg/png` to your page's folder.
# Focal points: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight.
image:
  caption: ''
  focal_point: ''
  preview_only: false

# Projects (optional).
#   Associate this post with one or more of your projects.
#   Simply enter your project's folder or file name without extension.
#   E.g. `projects = ["internal-project"]` references `content/project/deep-learning/index.md`.
#   Otherwise, set `projects = []`.
projects: ["oxytocin"]
publishDate: '2020-08-28T10:57:59.041916Z'
publication_types:
- 2
abstract: 'Oxytocin is a neuropeptide involved in animal and human reproductive and
  social behavior. Three oxytocin signaling genes have been frequently implicated
  in human social behavior: OXT (structural gene for oxytocin), OXTR (oxytocin receptor),
  and CD38 (oxytocin secretion). Here, we characterized the distribution of OXT, OXTR,
  and CD38 mRNA across the human brain by creating voxel-by-voxel volumetric expression
  maps, and identified putative gene pathway interactions by comparing gene expression
  patterns across 20,737 genes. Expression of the three selected oxytocin pathway
  genes was enriched in subcortical and olfactory regions and there was high co-expression
  with several dopaminergic and muscarinic acetylcholine genes, reflecting an anatomical
  basis for critical gene pathway interactions. fMRI meta-analysis revealed that the
  oxytocin pathway gene maps correspond with the processing of anticipatory, appetitive,
  and aversive cognitive states. The oxytocin signaling system may interact with dopaminergic
  and muscarinic acetylcholine signaling to modulate cognitive state processes involved
  in complex human behaviors.'
publication: '*Nature Communications*'
doi: 10.1038/s41467-019-08503-8

links:
  - name: Open Access PDF
    url: 'https://doi.org/10.1038/s41467-019-08503-8'
  - name: Data and analysis scripts
    url: 'https://osf.io/jp6zs/'
---


text:
#### Integrations
##### ServiceNow v2
- Fixed an issue in the ***servicenow-upload-file*** command where the file to upload was mishandled.
- Updated the Docker image to: *demisto/python3:<IP_ADDRESS>59*.

repository: demisto/content

gha_language: Python

repo_name: demisto/content


text:
## Dependencies

This playbook uses the following sub-playbooks, integrations, and scripts.

### Sub-playbooks

* Identity - Create Incident (subplaybook)

### Integrations

* Recorded Future Identity

### Scripts

This playbook does not use any scripts.

### Commands

* closeInvestigation
* recordedfuture-identity-search

## Playbook Inputs

---
There are no inputs for this playbook.

## Playbook Outputs

---
There are no outputs for this playbook.

## Playbook Image

---

![Identity - Lookup Identities (parent)](../doc_files/Recorded_Future_Identity_-_Lookup_Identities_parent.png)


text:
---
layout: post
title: Number letter counts
date: 2020-06-29
tags: [easy]
probid: 17
---

### Problem Statement

If the numbers 1 to 5 are written out in words: one, two, three, four, five, then there are 3 + 3 + 5 + 4 + 4 = 19 letters used in total.

If all the numbers from 1 to 1000 (one thousand) inclusive were written out in words, how many letters would be used?

*NOTE: Do not count spaces or hyphens. For example, 342 (three hundred and forty-two) contains 23 letters and 115 (one hundred and fifteen) contains 20 letters. The use of "and" when writing out numbers is in compliance with British usage.*

### Solution

```
units = ['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten',
'eleven', 'twelve' , 'thirteen', 'fourteen', 'fifteen', 'sixteen', 'seventeen', 'eighteen',
'nineteen']

tens = ['twenty', 'thirty', 'forty', 'fifty', 'sixty', 'seventy', 'eighty', 'ninety']

other = ['hundred', 'thousand', 'and']

result = 0

for i in range (1, 1001):

    unit = i% 10
    ten = (i // 10)% 10
    hundred = (i // 100)% 10
    thousand = (i // 1000)% 10

    if thousand != 0:
        result += len (units [0]) + len (other [1])
    if i% 1000 != 0:
        if hundred != 0:
            result += len (units [hundred-1]) + len (other [0])
            if i% 100 != 0:
                result += len (other [2])
        if i% 100 != 0:
            if ten <2:
                result += len (units [i% 100-1])
            else:
                result += len (tens [ten-2])
                if unit != 0:
                    result += len (units [unit-1])
print (result)
```

**Output**

```
21124
```

repository: ayanbag/Euler-Decode

repo_name: ayanbag/Euler-Decode


text:
# DDPAE

This is the code for the paper

**Learning to Decompose and Disentangle Representations for Video Prediction**

[Jun-Ting Hsieh](https://jthsieh.github.io/), [Bingbin Liu](https://clarabing.github.io/), [De-An Huang](http://ai.stanford.edu/~dahuang/), [Li Fei-Fei](http://vision.stanford.edu/feifeili/), [Juan Carlos Niebles](http://www.niebles.net/)

Will be presented at [NeurIPS 2018](https://nips.cc/).

Our goal is to predict future video frames given a sequence of input frames.
Our key insight into this high-dimensional prediction problem is to decompose it into sub-problems that can be more easily predicted.
To this end, we proposed the **Decompositional Disentangled Predictive Auto-Encoder (DDPAE)**, a framework that combines structured probabilistic models and deep networks to automatically
1. decompose the high-dimensional video into components, and
2. disentangle each component to have low-dimensional temporal dynamics that are easier to predict.

Our DDPAE is able to learn both the latent decomposition and disentanglement without explicit supervision.
We show strong results on the Moving MNIST and Bouncing Balls datasets.

**Note:** Our code is built with [Pyro](http://pyro.ai/). We encourage readers to read the [Pyro tutorial](http://pyro.ai/examples/intro_part_i.html) and [AIR tutorial](http://pyro.ai/examples/air.html) to better understand our code.

If you find this code useful in your research then please cite:
```
@inproceedings{hsieh2018learning,
  title={Learning to Decompose and Disentangle Representations for Video Prediction},
  author={Hsieh, Jun-Ting and Liu, Bingbin and Huang, De-An and Fei-Fei, Li and Niebles, Juan Carlos},
  booktitle={NeurIPS},
  year={2018}
}
```

## Setup
All code was developed and tested on Python 3.6, PyTorch 0.3.1, and Pyro 0.2.

To view results, you need to install `tensorflow` and [tensorbaordX](https://github.com/lanpa/tensorboardX).

## Datasets

### Moving MNIST
Download the original [MNIST](http://yann.lecun.com/exdb/mnist/) dataset and the [Moving MNIST test set](http://www.cs.toronto.edu/~nitish/unsupervised_video/) by running
```
./datasets/moving_mnist/download.sh
```

### Bouncing Balls
We generate our Bouncing Balls dataset with the [Neural Physics Engine](https://github.com/mbchang/dynamics), used by Chang et al, 2017.

First, clone the [bouncing_balls](https://github.com/jthsieh/dynamics) submodule,
```
git submodule update --init --recursive
```
Note that this is slightly modified from the original [dynamics](https://github.com/mbchang/dynamics) repository.

`npm` must be installed. With Anaconda, install by running: `conda install -c conda-forge nodejs`.

Move these folders to the desired location (dataset path).
Next, modify the `root` variable in `datasets/bouncing_balls/process.py` and run `python datasets/bouncing_balls/process.py` for both files.

## Training

Remember to set the arguments, such as `ckpt_dir`, `ckpt_name`, etc.
A `tb_train` directory will be created in `ckpt_dir`. You can see the training progress by running `tensorboard --logdir ckpt_dir/tb_train`.

### Moving MNIST
Run
```
./scripts/train_mnist.sh
```
In our paper, we let the model train for 200k iterations. However, 50k iterations should be enough to see the model separating the two digits.

### Moving MNIST
Run
```
./scripts/train_balls.sh
```
We also train for 200k iterations.

repository: eugenelet/DDPAE-video-prediction

repo_name: eugenelet/DDPAE-video-prediction


text:
# DPTP (**D**ata-**P**lane **T**ime synchronization **P**rotocol)

This source code synchronizes the network consisting of programmable (Barefoot Tofino) switches and IEEE1588 PTP-compatible NICs to support the global timing (64-bit) in the network data-plane (Switch and NICs).
Once DPTP is running, packets can have a global timing context, which is useful for several applications like latency tracking, monitoring, updates, ordering of events, distributed applications, etc.

Paper Link :point_right:	 https://dl.acm.org/doi/10.1145/3314148.3314353?cid=82658740957

## Source Code Pointers

### Switch Code
The switch data-plane code is written in [P4](https://github.com/p4lang) programming language. Once this code is up and running, two virtual switches are synchronized to a global time.

p4_14 : [v_14/](https://github.com/praveingk/DPTP/tree/master/v_14) (Tested on SDE 8.x.x)

p4_16 : [v_16/](https://github.com/praveingk/DPTP/tree/master/v_16) (Tested on SDE 9.x.x)

### End-Host Code

[dptpclient](https://github.com/praveingk/DPTP/tree/master/dptpclient) : Once Switches are synchronized, end-hosts connected to the switch could run the dpdk-based dptpclient to synchronize end-hosts/nic with the network time.

# Topology
The Topology used in as below :

![DPTP Topology](Tofino-minibed-timesync.png)

A single tofino switch named "tofino1" is virtualized into two switches Master(M) and Switch1. To do this virtualization, you will need to add a loopback link between port3 (160-163) and port5 (176-179). Once done, it will be configured as 10G ports, and we will be using only one link (160-176) as the connection between Switch1 and Master. Additionally, you will need atleast one host connected to port 1(128-131) to send DPTP requests.

### Steps to run p4_14 based DPTP in Tofino:

1) Navigate to the SDE PATH :
```shell
     cd ~/bf-sde-8.x.x
     export DPTP_PATH=<PATH TO DPTP FOLDER>
```
2) Set the env variables :
```shell
     . ./set_sde.bash
```
3) Build the p4 program using the command :
```shell
     ./p4_build.sh $DPTP_PATH/v_14/dptp_topo.p4
```
4) Load the p4 program, and run the control plane API code using :
```shell
     cd $DPTP_PATH/v_14/CP
     ./run.sh
```
5) This should automatically start the synchronization between Switch1 and master through packets from control-plane.

### Steps to run p4_16 based DPTP in Tofino:

1) Navigate to the SDE PATH :
```shell
     cd ~/bf-sde-9.x.x
     export DPTP_PATH=<PATH TO DPTP FOLDER>
```
2) Set the env variables :
```shell
     . ./set_sde.bash
```
3) DPTP program in p4_16 has multiple profiles :

     a) DPTP with logical switches to get measurements and to do a sanity check.
     ```shell
          ./p4_build.sh -DLOGICAL_SWITCHES $DPTP_PATH/v_16/p4_src/dptp.p4
     ```

     b) DPTP without logical switches. This is basically the program that can be included in any p4 programs
     ```shell
          ./p4_build.sh $DPTP_PATH/v_16/p4_src/dptp.p4
     ```

4) Load the p4 program, and run the control plane API code using :
```shell
     cd $DPTP_PATH/v_16/CP
     ./run.sh
```
   If you want to enable debugs the command is :

```shell
     ./run.sh debug
```

5) This should automatically start the synchronization between Switch1 and master through packets from control-plane.

### Steps to run DPTPClient for host synchronization:
Refer [Readme](https://github.com/praveingk/DPTP/tree/master/dptpclient)

### Steps to run MoonGen for host synchronization:
Moongen script sends synchronization requests packets between switches
Pull from https://github.com/praveingk/moongen/, Make sure the submodule libmoon is also pulled.
Follow the readme instructions in moongen to build it.
1) Enable the NIC to work with DPDK:
```shell
sudo ./libmoon/deps/dpdk/usertools/dpdk-devbind.py --b igb_uio <NIC Ports>
```

2) To start DPTP in the network between switch 1 and master:
```shell
 sudo ./build/MoonGen examples/dptp_host.lua <DPDK PORT id1> <DPDK Port id2>
```
Note that "DPDK Port id2" is redundant and is used only for switch-to-host DPTP accuracy measurement.

### Steps to Create CrossTraffic in the Link(160-176) during DPTP

1) To create cross-traffic in the link being used for synchronization towards the SW1 (i.e. 160 --> 176),
we simply craft a packet to destination address of SW1 (0x100000000001) and send it from any host link.
```shell
 sudo ./build/MoonGen examples/dptp_host.lua -d 1 <DPDK PORT id1> <DPDK Port id2>
```
Note that "DPDK Port id2" is redundant and is used only for switch-to-host DPTP accuracy measurement.

2) To create oversubscribed traffic, run the above example from two 10G host-links.


text:
# Stakebird Albatross Testnet Instructions

## TLDR

Block explorer: [https://explorer.albatross-1.publicawesome.dev/](https://explorer.albatross-1.publicawesome.dev/)

Binaries: [v0.3.2](https://github.com/public-awesome/stakebird/releases/tag/v0.3.2)

Genesis file: [genesis.json](https://github.com/public-awesome/stakebird-testnets/blob/master/albatross-1/genesis.json)

Seeds: `840fd59f5040abbd9b7f82e71ccf0e90cc1a0295@seed.albatross-1.publicawesome.dev:36656`

## Minimum hardware requirements

- 1GB RAM
- 25GB of disk space
- 1.4 GHz CPU

## Software requirements

Stakebird has releases for Linux and MacOS [here](https://github.com/public-awesome/stakebird/releases/tag/v0.3.2).

- [Ubuntu Setup Guide](./ubuntu.md)
- Latest version : [v0.3.2](https://github.com/public-awesome/stakebird/releases/tag/v0.3.2)

### Install Stakebird

You can install Stakebird by downloading the binary (easiest), or compiling from source.

#### Option 1: Download binary

1. Download the binary for your platform: [releases](https://github.com/public-awesome/stakebird/releases/tag/v0.3.2).
2. Copy it to a location in your PATH, i.e: `/usr/local/bin` or `$HOME/bin`.

i.e:
```sh
> wget https://github.com/public-awesome/stakebird/releases/download/v0.3.2/stakebird_0.3.2_linux_arm64.tar.gz
> sudo tar -C /usr/local/bin -zxvf stakebird_0.3.2_linux_arm64.tar.gz
```

#### Option 2: Build from source

Requires [Go version v1.14+](https://golang.org/doc/install).

```sh
> mkdir -p $GOPATH/src/github.com/public-awesome
> cd $GOPATH/src/github.com/public-awesome
> git clone https://github.com/public-awesome/stakebird && cd stakebird
> git checkout v0.3.2
> FAUCET_ENABLED=true make install
```

#### Verify installation

To verify if the installation was successful, execute the following command:

```sh
> staked version --long
```

It will display the version of staked currently installed:

```sh
name: stakebird
server_name: staked
version: 0.3.2
commit: a152fc6aab2f3259d24e071f58855cf445236b55
build_tags: netgo,faucet
go: go version go1.15.3 linux/amd64
```

NOTE: Make sure `build_tags` includes "faucet", which is required for testnet.

## Setup validator node

If you are looking to join the testnet post genesis time (_Nov 03 2020 1600 UTC_), skip to [Create Testnet Validator](#create-testnet-validator)

Below are the instructions to generate & submit your genesis transaction

### Generate genesis transaction (gentx)

1. Initialize the Stakebird directories and create the local genesis file with the correct
   chain-id

   ```sh
   > staked init <moniker-name> --chain-id=albatross-1
   ```

2. Create a local key pair

   ```sh
   > staked keys add <key-name>
   ```

3. Add your account to your local genesis file with a given amount and the key you
   just created. Use only `1000000000uegg`, other amounts will be ignored. EGG is testnet STB.

   ```sh
   > staked add-genesis-account $(staked keys show <key-name> -a)<PHONE_NUMBER>uegg
   ```

4. Create the gentx

   ```sh
   > staked gentx <key-name> --amount=900000000uegg --chain-id=albatross-1
   ```

   If all goes well, you will see a message similar to the following:

    ```sh
    Genesis transaction written to "/home/user/.staked/config/gentx/gentx-******.json"
    ```

### Submit genesis transaction

> NOTE: To prevent malicious validators, and to ensure a fair and decentralized launch, the following rules will be enforced:
> 1. Github accounts must be at least a year old
> 2. Only one gentx per Github account is allowed

Submit your gentx in a PR [here](https://github.com/public-awesome/stakebird-testnets)

- Fork [the testnets repo](https://github.com/public-awesome/stakebird-testnets) into your Github account

- Clone your repo using

    ```sh
    > git clone https://github.com/<your-github-username>/stakebird-testnets
    ```

- Copy the generated gentx json file to `<repo_path>/albatross-1/gentx/`

    ```sh
    > cd stakebird-testnets
    > cp ~/.staked/config/gentx/gentx*.json ./albatross-1/gentx/
    ```

- Commit and push to your repo
- Create a PR onto https://github.com/public-awesome/stakebird-testnets

### Start your validator node

Once after the genesis is released (_Nov 01 2020 1600 UTC_), follow the instructions below to start your validator node.

#### Genesis & seeds

Fetch `genesis.json` into `staked`'s `config` directory.

```sh
> curl https://raw.githubusercontent.com/public-awesome/stakebird-testnets/master/albatross-1/genesis.json > $HOME/.staked/config/genesis.json
```

Verify you have the correct genesis file:

```sh
> shasum -a 256 ~/.staked/config/genesis.json
fb13172f39d0e888601b828aea104e830aa64c3893ff478194e4d41b2e61f793  genesis.json
```

Add seed nodes in `config.toml`.

```sh
> vi $HOME/.staked/config/config.toml
```

Find the following section and add the seed nodes.

```sh
# Comma separated list of seed nodes to connect to
seeds = "840fd59f5040abbd9b7f82e71ccf0e90cc1a0295@seed.albatross-1.publicawesome.dev:36656"
```

```sh
# Comma separated list of persistent peers to connect to
persistent_peers = "a81c314a4619f85cccbb9bb69eeabd9d385bc82b@<IP_ADDRESS>:26656"
```

#### Set validator gas fees

You can set the minimum gas prices for transactions to be accepted into your node's mempool. This sets a lower bound on gas prices, preventing spam. Stakebird can accept gas in *any* currency. To accept both ATOM and EGG for example, set `minimum-gas-prices` in `app.toml`.

```sh
> vi $HOME/.staked/config/app.toml
```

```sh
minimum-gas-prices = "0.025uatom,0.025uegg"
```

#### Start node automatically (Linux only)

Create a `systemd` service

```sh
> sudo vi /etc/systemd/system/staked.service
```

Copy and paste the following and update `<your_username>` and `<go_workspace>`:

```sh
[Unit]
Description=staked
After=network-online.target

[Service]
User=<your_username>
ExecStart=/home/<your_username>/<go_workspace>/bin/staked start
Restart=always
RestartSec=3
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target
```

**This assumes `$HOME/go_workspace` to be your Go workspace. Your actual workspace directory may vary.**

```sh
> sudo systemctl enable staked
> sudo systemctl start staked
```

Check node status

```sh
> staked status
```

Check logs

```sh
> journalctl -u staked -f
```

## Create testnet validator

This section applies to those who are looking to join the testnet post genesis.

1. Init Chain and start your node

   ```sh
   > staked init <moniker-name> --chain-id=albatross-1 --stake-denom=uegg
   ```

   After that, please follow all the instructions from [Start your validator node](#start-your-validator-node)

2. Create a local key pair

   ```sh
   > staked keys add <key-name>
   > staked keys show <key-name> -a
   ```

3.  Request tokens from the [Stakebird Discord #validator channel](https://discord.gg/QeJWCrE) if you need more.

gha_language: Shell


text:
---
date: 2022-03-16T12:57:35-04:00
description: "Centralized repositories of raw data from a wide range of sources"
tags: [ "data-products" ]
title: "Data Lakes"
---

# Data lakes

A **data lake** is a type of [data product](data-products.md) that serves as a centralized repository of raw data sourced from a wide range of sources and stored for convenient retrieval for [analysis](data-analysis.md).

Data lakes generally store data "as is" - whether [structured](structured-data.md), [semi-structured](semi-structured-data.md), or [unstructured](unstructured-data.md) - with little or no transformation during ingestion. This makes data lakes relatively easy and inexpensive to create, especially compared with the highly formalized structures of [data warehouses](data-warehouses.md).

Data lakes can also be especially useful in [data science](data-science.md) because they can store data not traditionally kept in [databases](databases.md), such as audio and video files that may contain important data to train machine learning models alongside more "traditional", tabular data.

## Data swamps: Challenges with data lakes

Poorly maintained data lakes are often referred to as **data swamps**. When data in a data lake is not subject to effective [governance](data-governance.md) or gains a reputation for being unreliable, the usefulness of a data lake can quickly decline.

Additionally, access controls can be difficult to establish in data lakes due to their centralized nature. Personally identifiable information, intellectual property, and other secure data can be easily leaked if not carefully controlled. This need to control access on a granular level can negate some of the apparent simplicity of a data lake. This means that most data lakes are not suitable for [business intelligence](business-intelligence.md) workloads.

gha_language: TypeScript