text:
# Getting Started
To create a business process/native workflow and run it on the application server, we have several options:
- We can create an application-server with WPF frontend
- We can create an application-server with ASP.NET Blazor (server and client-wasm) frontend
- We can create an application-server with ASP.NET Core MVC Web Application frontend.

## ASP.NET Core MVC Web Application frontend
### Create app
1) Create ASP.NET Core Web Application (Model-View-Controller) (ASP.NET Core 3.1) with name 'ExampleApplicationServer'
2) Install NuGet package '**Cvl.ApplicationServer.Server**' and '**Cvl.VirtualMachine**'
3) In Startup.cs add -> using Cvl.ApplicationServer.Server.Extensions;
4) In Startup.cs->ConfigureServices add -> services.UseApplicationServerServices();
5) At this moment, ju can run your server (F5) and enter /process-types (https://localhost:44390/process-types) to get list of avaliable process types
There by 2 example processes Cvl.ApplicationServer.Server.Node.Processes.TestProcess.BankLoanProcess and Cvl.ApplicationServer.Server.Node.Processes.TestProcess.SimpleTestProcess.
```CSharp
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Cvl.ApplicationServer.Server.Extensions;

namespace ExampleApplicationServer
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllersWithViews();
       -->> services.UseApplicationServerServices(); <<---
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseExceptionHandler("/Home/Error");
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }
            app.UseHttpsRedirection();
            app.UseStaticFiles();
            app.UseRouting();
            app.UseAuthorization();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllerRoute(
                    name: "default",
                    pattern: "{controller=Home}/{action=Index}/{id?}");
            });
        }
    }
}
```

### Create process
6) Create your simple process, by creating folder Processes and add class MyTestProcess base on Cvl.ApplicationServer.Server.Node.Processes.Model.BaseProcess
7) Creat simple MyModel class with property string MyMessage
8) Override object Start(object inputParameter) and show form -ShowForm("MyTestForm", new MyModel() { MyMessage = "My message from process" });

```CSharp
public class MyModel
{
    public string MessageFromProcess { get; set; }
    public string UserMessage { get; set; }
}
public class MyTestProcess : Cvl.ApplicationServer.Server.Node.Processes.Model.BaseProcess
{
    protected override object Start(object inputParameter)
    {
        var response = ShowForm("MyTestForm", new MyModel() { MessageFromProcess = "My message from process" });

        return null;
    }
}
```

9) Run application (F5) and enter /process-list (https://localhost:44390/process-types) - you get your process (ExampleApplicationServer.Processes.MyTestProcess)
### Create view for your process
10) In SolutionExpolore in folder Views add floder 'Process'
11) Add Waiting view - in folder Process add razor view 'WaitingView.cshtml'
```html
<h1>The process is running</h1>
<p>Please refresh the page in a moment</p>
```
12) Add MyTestForm view - in folder Process add razor view 'MyTestForm.cshtml'
```html
@model Cvl.ApplicationServer.Server.Node.Processes.Model.FormModel<ExampleApplicationServer.Processes.MyModel>

<div class="row">
    <div class="col-lg-8 mx-auto">
        <h1>My test view</h1>
        <p>Message from process: @Model.Model.MessageFromProcess</p>
        <form action="/process-setdata" method="post">
            @Html.HiddenFor(x => x.ProcessId)
            <div class="form-group mt-5">
                <h3>User message</h3>
            </div>
            <div class="form-group">
                <input asp-for="Model.UserMessage" placeholder="Enter some message" class="form-control" />
            </div>
            <button class="btn btn-primary btn-block mt-2" type="submit"> Confirm </button>
        </form>
    </div>
</div>
```
![](https://github.com/cv-lang/application-server/blob/master/wiki/gettingStarted-mvc-vs.png?raw=true)

### Start application
13) Run your app - F5 - and in browers open link /process-start/ExampleApplicationServer.Processes.MyTestProcess (https://localhost:44390/process-start/ExampleApplicationServer.Processes.MyTestProcess).
This start your process and redirect You to your waiting page (https://localhost:44390/process-view/4) refresh page, you see your form (MyTestForm) with data from process.
14) Enter user message and confirm
15) in this moment in your process - response variable have your input data
16) Let's modyfy process, and add another ShowForm
```CSharp
public class MyTestProcess : Cvl.ApplicationServer.Server.Node.Processes.Model.BaseProcess
{
    protected override object Start(object inputParameter)
    {
        var response = ShowForm("MyTestForm", new MyModel() { MessageFromProcess = "My message from process" });

        ShowForm("MyTestForm", new MyModel() {
            MessageFromProcess = $"User message '{response.UserMessage}'" });

        return null;
    }
}
```
Process view:

![](https://github.com/cv-lang/application-server/blob/master/wiki/mytestview.png?raw=true)

repository: cv-lang/application-server

repo_name: cv-lang/application-server


text:
# Tests

This directory contains some unit tests for Sequentia.

## Running the tests

To run the tests, you'll need to make sure you have the required dependencies installed.

Once you've cloned the repository, you can do this by running the following command in the repository root.

```console
pip install .[test]
```

Once the dependencies are installed, you can run the tests with the following command (from the repository root).

```console
pytest lib/test
```

license: MIT

repository: bakhtiaris/sequentia

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: bakhtiaris/sequentia


text:
[![Build status](https://github.com/SpareBank1/Troxy/actions/workflows/build-and-deploy-troxy.yml/badge.svg)](https://github.com/SpareBank1/Troxy/actions/workflows/build-and-deploy-troxy.yml)
[![Maven Central](https://img.shields.io/maven-central/v/no.sparebank1.troxy/troxy-server.svg)](https://search.maven.org/artifact/no.sparebank1.troxy/troxy-server)
[![Docker Pulls](https://img.shields.io/docker/pulls/sparebank1/troxy)](https://hub.docker.com/r/sparebank1/troxy)
# Troxy

Troxy is an highly configurable and extendable application that simulates responses using the HTTP(S) protocol. Troxy can handle many simultaneous requests making it an excellent choice to use when running performance tests against your application.

## Build
Build Troxy
`mvn clean install`

###### Build docker image
`docker build -t troxy .`

###### Start docker container on port 8080
`docker run -p 8080:8080 troxy`

###### Running Troxy from code
Troxy can be started locally by running Troxy.main(). The working directory must be set to troxy/local

## Request Matching

Troxy can match requests based on different attributes in the HTTP protocol.
* Protocol
* HTTP Method
* Port
* Host
* Path
* Query parameters
* Header
* Content

All of these attributes use regular expressions.

## Modes

Troxy can be run in different modes.

##### Forwarding

In this mode, troxy will forward requests to the specified backend system and responses are then forwarded back to the client.

##### Recording

In this mode, troxy will forward requests to the backend system and make a mock-recording of the responses before forwarding them back to the client. The recordings are then saved on a file and can be viewed under the "Opptak" tab.

##### Playback

In this mode, troxy matches incoming requests with existing mocks. If a match is found, the matching response is sent back. If no match is found, Troxy will return HTTP status 418.

##### Record or Playback

In this mode, Troxy will send back a matching response if found. If a response is not found the request will be forwarded to the backend system and a new recording will be created.

##### Forwarding or Playback

In this mode, Troxy will send back a matching response if found. If a response is not found the request will be forwarded to the backend system but Troxy will not record the response. This mode is designed to be used for manual testing when you want to mock a set of requests, while sending other requests as usual to the backend system.

## API
Troxy provides an API where you can change the mode or upload mocks from disk

##### Upload mocks
~~~~
curl -v -F directory="troxyfolder/" -F<EMAIL_ADDRESS>http://localhost:8080/api/upload
curl -v -F directory="troxyfolder/" -F<EMAIL_ADDRESS>http://localhost:8080/api/upload
~~~~

##### Change mode
~~~~
Troxy modes
PLAYBACK
RECORD
PLAYBACK_OR_RECORD
PASSTHROUGH
PLAYBACK_OR_PASSTHROUGH

curl 'http://localhost:8080/api/status/mode' -X PUT --data 'RECORD'
~~~~

##### Statistics interval
~~~~
Statistics interval (minutes)
0
1
5
10
15
30
60
1440

curl 'http://localhost:8080/api/status/statisticsInterval' -X PUT --data '5'
~~~~

## Editing recordings

Recordings are saved as files with a .troxy file extension. All recordings are available for editing in the GUI under the "Opptak" tab. Individual recordings can also be activated or disabled under this tab.

### Time delay
Mocks can be configured using different time delay strategies.

**None** - The response is returned directly with no delay

**Permanent** - The response is returned after a configured value is reached

**Random** - The response is returned after a random time, between the limits that are configured, is reached

**Normally distributed** - The response time is calculated by a normally distributed curved where min and max values are configured.

**Exponential distribution** - The response time is calculated using an exponential distribution where min, average and max values are configured.

### Regex-matching

All HTTP attributes use regular expressions.

Below is an example illustrating how this works. The header must contain a specific SOAPAction but anything else in the same header field uses regular expressions.

![alt text](doc/images/request-example.png "Regex example")

**Tip:** you can press Ctrl + Space to get a small list of commonly used regular expressions.

#### Group capturing
You can use regular expression group capturing to extract values from the request and insert those in the response.

##### Example
In this example a social security number is extracted from a request body by using the following group capturing syntax: `(?<nnin>[0-9]*)`. The result (the social security number) is extracted to a group named nnin.

![alt text](doc/images/group-capturing-request.png "Group capturing example")

The extracted value can then be used in the response by the following syntax: `$content:nnin$`
Note that you need to specify where the value was extracted from (in this case content). If the value was extracted from the header the syntax would be `$header:nnin$`. To get a list of defined regular expression groups in the Response tab, you can again use Ctrl + Space to get a list.

![alt text](doc/images/group-capturing-response.png "Group capturing example")

#### Environment variables in mock response
You can fetch environment variables and insert those in the response to provide a more dynamic mock. Syntax is `$env:MYENVIRONMENTVARIABLE$`

### Sequence
Troxy can use the same request for several different responses. There are two strategies used when this mode is enabled for a recording.

**Random** - When an expected incoming request is made, one of the available responses are chosen at random. It is possible to weigh responses. For example, one response can have a 25% chance to be returned, and another can have a 75% chance.

**Sequential** - When an expected incoming request is made, the responses are returned sequentially. If a recording has several responses; the first response will be returned for the first request, the second response for the second request and so on. When the last response is reached, response 1 will be returned next. In this mode we also have the option of weighing responses so that one response can be returned several times before continuing to the next one.

## Logging

The logging tab shows the content from the log file `simulator.log`.

## Application configuration

In order to get your application to send its traffic through Troxy you have to add the Troxy URL before the backend request url.

~~~~
# Example
# Original setting
some_rest_api_url=https://some-api.sparebank1.no

# Through Troxy
some_rest_api_url=https://troxy.test.sparebank1.no/https://some-api.sparebank1.no
~~~~

If you use docker you can link to it instead in the compose file

~~~~
yourapplication:
  image: __IMAGE_REPO_AND_TAG__
  links:
  - troxy:some-api.sparebank1.no
~~~~

## Filters

Filters are a way to extend the functionality of Troxy. Filters have access to modify requests and their respective responses during both recording and playback. Compiled filters (.jar) must be moved to the `/opt/troxy/data/filters` folder in order to be picked up by Troxy.

#### Filter phases
A filter can manipulate a request and response in the following five phases in this specific order:

1. Upon receiving a client request, before Troxy searches for matching recordings.
2. Before a request is forwarded to the backend system
3. When receiving a response from the backend system
4. When creating a new recording (does not apply to recordings created through the user interface).
5. Before responses are sent to the client.

**Note:** If you do not communicate with a backend system (i.e mode Playback), phases 2, 3 and 4 will not be activated.

#### Filter configuration
Filters can be configured under the "Oppsett" tab.

A filter will not run unless it is first activated

`filter.<filtername>.enabled=true`

Implemented filters might have their own configuration. This configuration can be accessed as shown below

`filter.<filtername>.config.<key>=<value>`

#### Filters included by default
There are some filters that are included in Troxy by default. Examples of these can be found in the Troxy configuration file `troxy.properties`

**AutoRegexNewRecording**
A filter that searches for text in new recordings that matches the specified configured regular expression,
It then replaces the text with the configured regular expression or specified value.

**DelayResponse**
A filter that handles time delays for recordings

**KeyBasedReplace**
A filter that searches for a key in the request. If the key is found it will use the key to pick a value from a data file and use this value in the response.

**SetRecordingDelay**
A filter that will set time delays on new recording based on the real time it took during the record.

**FunctionReplace**
A filter that replaces keys with a support function. Keys are replaced in both the response body and header in the recorded Troxy response. More supported functions may be added at a later time.
One of the functions supported are `now`. now replaces a specified key with a timestamp on a specified format.

## Statistics
Under the "Statistikk" tab you can see statistics of the number of requests for different recordings. The interval is default set to 60 minutes but can be configured in the file troxy.properties or under the Status page.

gha_language: JavaScript

gha_license_id: MIT


text:
# Extending High-Level Synthesis for **Ta**sk-**Pa**rallel Programs

## Feature Synopsis

### Programmability Improvement

+ [Convenient kernel communication APIs](#kernel-communication-interface)
+ [Simple host-kernel interface](#host-kernel-interface)
+ [Universal software simulation w/ coroutines](#software-simulation)
+ [Hierarchical code generator w/ Xilinx HLS backend](#rtl-code-generation)

### Beyond Programmability

+ [Asynchronous/concurrent memory accesses](#memory-system)
+ [Flexible detached (autorun/free-running) tasks](#detached-tasks)
+ [Resource-efficient FIFO templates](#hardware-fifo)
+ [Improved timing closure](#timing-closure) (via [AutoBridge][autobridge])

## TAPA vs State-of-the-Art

In this project, we target PCIe-based FPGA accelerators that are commonly available in data centers (e.g., [Amazon EC2 F1][aws-f1], [Intel PAC D5005][d5005], [Xilinx Alveo U250][u250]).
In additional to the traditional register-transfer level (RTL) programming approach, these accelerators can also be programmed natively using vendor high-level synthesis (HLS) tools, e.g., [Intel FPGA SDK for OpenCL][iocl] and [Xilinx Vitis][xhls].
Both tools support task-parallel programs in HLS.

[Intel FPGA SDK for OpenCL][iocl] supports multiple OpenCL kernels running in parallel.
The kernels communicate with each other via `channel`.
Blocking/non-blocking read/write are supported.
Both the kernel instances and the communication channels are global variables, which has several implications:

+ Hierarchical designs must be flattened;
+ Sharing code with channel accesses among kernels is difficult;
+ Accessed channels are not clearly visible for each kernel;
+ Functionally identical kernels must be high-level–synthesized separately.

The [number of kernels is limited to 256](https://www.intel.com/content/www/us/en/programmable/documentation/mwh1391807965224.html#ewa1400860540321) due to limitations in software simulation.

[Xilinx Vitis][xhls] supports two different task-parallel granularities.
Fine-grained task-parallelism is supported via `#pragma HLS dataflow`.
In a dataflow region, C++ function calls are interpreted as if they run in parallel.
The communication interface for each C++ function is `hls::stream<T>&`, which generates `ap_fifo` RTL interface with simple ready-valid handshake.
Blocking/non-blocking read/write are supported.
The communication channels are instantiated using the same `hls::stream<T>` C++ class in the dataflow region.
Hierarchical dataflow regions are possible.
However, functionally identical kernels are still high-level–synthesized separately.

Coarse-grained task-parallelism is supported by linking multiple OpenCL kernels together (“[k2k streaming](https://www.xilinx.com/html_docs/xilinx2020_2/vitis_doc/streamingconnections.html)”).
Each OpenCL kernel itself may leverage `#pragma HLS dataflow`, thus enabling task-parallelism at a higher-level of granularity.
Inter–OpenCL kernel communication uses a more-complicated and less-general `hls::stream<ap_axiu<width, 0, 0, 0>>&` construct, which generates AXI4-Stream (`axis`) RTL interface.
In addition to blocking/non-blocking read/write operations, `axis` also provides a `last` bit that can implement transactions.
However, leveraging such task-parallelism requires additional information beyond the source code at link time, which complicates the overall workflow.

Compared with these state of the art, TAPA provides a more flexible and productive development experience.
In many cases, TAPA can even improve the quality of result (QoR).
Details are as follows.

### Kernel Communication Interface

In addition to blocking/non-blocking read/write operations, TAPA also supports *peeking* and *transactions*.
Peeking is defined as reading a token from a channel without removing the token from the channel.
This makes it possible to conditionally read tokens based on the content of tokens.
Transactions are supported by providing a special end-of-transaction (`eot`) token for each channel.
Such a token can be used to indicate the completion of a transaction without additional coding constructs.

TAPA uses separate `tapa::istream<T>&` and `tapa::ostream<T>&` classes for communication interfaces.
This makes the communication directions of each interface visually obvious in the source code.
IDEs will also only suggest available functions for code completion (e.g., `read` won't show up for `tapa::ostream<T>&`).

### Host-Kernel Interface

Both Intel FPGA SDK for OpenCL and Xilinx Vitis provide high-level host-kernel interface as specified in the OpenCL standard.
While programmers are relived from writing their own operating system drivers, they still have to write lengthy OpenCL host API calls.
For example, the [hello world example](https://github.com/Xilinx/Vitis_Accel_Examples/blob/21bb0cf788ace593c6075accff7f7783588ae8b4/hello_world/src/host.cpp#L58-L115) in Vitis Accel Examples spent 57 lines of code just to call the accelerator from the host.
Worse, running software simulation, hardware simulation, and on-board execution requires different [environment variable setup](https://www.xilinx.com/html_docs/xilinx2020_2/vitis_doc/runemulation1.html#ariaid-title5) in addition to different bitstream specifications.

TAPA cuts this down to a single function call to `tapa::invoke` and takes care of environmental variable setup.
Programmers only need to change the bitstream specification for different targets.

```cpp
tapa::invoke(
  vadd,                                     // Top-level kernel function.
  bitstream,                                // Path to bitstream. If empty, run software simulation.
  tapa::read_only_mmap<unsigned int>(in1),  // Kernel argument 0, read-only array.
  tapa::read_only_mmap<unsigned int>(in2),  // Kernel argument 1, read-only array.
  tapa::write_only_mmap<int>(out_r),        // Kernel argument 2, write-only array.
  size);                                    // Kernel argument 3, scalar.
```

### Software Simulation

C++ functions in a `#pragma HLS dataflow` region is called sequentially for software simulation.
This works correctly only for programs that do not have cyclic communication patterns (e.g., an image-processing pipeline).

A common approach that fixes the correctness issue is to launch a thread for each task (e.g., OpenCL kernel).
This approach however, has limited scalability due to contention caused by preemptive threads managed by the operating system.

TAPA takes a coroutine-based approach to solve the scalability problem.
Coroutines are managed by user space programs and do not have the contention issue.
Moreover, [context switch between coroutines are much faster than threads](https://www.boost.org/doc/libs/1_76_0/libs/coroutine/doc/html/coroutine/performance.html).

### RTL Code Generation

Although Vitis HLS can reuse HLS results for different instances of the same C++ function, it is not supported in a `#pragma HLS dataflow` region.
Both Intel FPGA SDK for OpenCL and Xilinx Vitis HLS run HLS for each instance of tasks as if they are different for task-parallel programs.
This can lead to long compilation time for programs with many identical task instances, e.g., as in [SODA][soda] and [AutoSA][autosa].
To work around this problem, [SODA][soda] implements an RTL backend that calls Vivado HLS for each task and then create the top-level RTL code by itself.
TAPA generalizes this approach and makes it available to all task-parallel programs.

### Memory System

Vitis HLS provides DRAM access via AXI4 memory-mapped (`m_axi`) interfaces.
However, in a `#pragma HLS dataflow` regions, task instances may not access the same `m_axi` interface.

TAPA provides `tapa::mmap` that has the same functionality as Vitis HLS's `m_axi` interfaces.
TAPA additionally provides asynchronous accesses to memory-mapped arguments, via separate address/data channels exposed as `tapa::i/ostream`.
This makes it possible to achieve a high random access throughput.

TAPA also makes it possible to access the same memory-mapped interface from multiple task instances.
This is implemented by instantiating AXI interconnect where an AXI interface is shared.
Note that *with great power comes great responsibility*.
The AXI interconnect do not provide built-in locking mechanisms.
Programmers must take care of correctness with concurrent reads and writes.

### Detached Tasks

Controlling tasks as OpenCL kernels can be expensive, yet often times each task instance is small and do not need runtime management.
Intel FPGA SDK for OpenCL supports “autorun” kernels, which are started as soon as the accelerator starts and is restarted when it stops.
However, [they cannot have any arguments other than the global communication channels](https://www.intel.com/content/www/us/en/programmable/documentation/mwh1391807965224.html#ewa1456413600674).
Similarly, Xilinx Vitis supports [“free-running” OpenCL kernels](https://www.xilinx.com/html_docs/xilinx2020_2/vitis_doc/streamingconnections.html#ariaid-title5).
While the state of these kernels do not have to be maintained at runtime and thus resources are saved, the degree of freedom is also greatly limited.
`#pragma HLS dataflow` do not support “autorun” or “free-running” instances, therefore all task instances must be properly terminated.
This can lead to unnecessary complexity to propagate termination signals.

TAPA makes it possible to “detach” a task instance once instantiated.
By default, a TAPA task is not considered finished until all its children instances finish.
Detach task instances resembles the concept of `std::thread::detach()`, and the parent task will not wait for it to finish.
This makes it possible to do necessary initialization before entering an infinite loop in detached tasks, thus having both resource efficiency and programming flexibility.

### Hardware FIFO

Vitis HLS can generate two different implementations for the communication channels, using shift-registers (SLR) and block RAMs (BRAM).
However, it seems to unnecessarily prefer BRAM in many cases (e.g., when the FIFO is wide but shallow).
TAPA uses a larger threshold that reduces BRAM usage.
Moreover, when URAM is more efficient (width ≥ 36 and depth ≥ 4096), it is used in place of BRAM.

### Timing Closure

TAPA natively supports [AutoBridge][autobridge].
By specifying a filename for the output TCL constraints, TAPA can generate coarse-grained floorplans that were often necessary to achieve good timing closure.

## Related Publications

+ Yuze Chi, Licheng Guo, Jason Lau, Young-kyu Choi, Jie Wang, Jason Cong.
  Extending High-Level Synthesis for Task-Parallel Programs.
  In FCCM, 2021.
  [[PDF]](https://about.blaok.me/pub/fccm21-tapa.pdf)
  [[Code]][tapa]
+ Licheng Guo, Yuze Chi, Jie Wang, Jason Lau, Weikang Qiao, Ecenur Ustun, Zhiru Zhang, Jason Cong.
  [AutoBridge: Coupling Coarse-Grained Floorplanning and Pipelining for High-Frequency HLS Design on Multi-Die FPGAs](https://doi.org/10.1145/3431920.3439289).
  In FPGA, 2021. (Best Paper Award)
  [[PDF]](https://about.blaok.me/pub/fpga21-autobridge.pdf)
  [[Code]][autobridge]

## Getting Started

### Prerequisites

+ Ubuntu 16.04+
  + Coroutine-based simulator only works on Ubuntu 18.04+
+ Vitis HLS 2020.2+
+ Vivado

### Install from Binary

```bash
./install.sh
```

### Install from Source

#### Build Prerequisites

+ CMake 3.13+
+ A C++ 11 compiler (e.g. `g++-9`)
+ Python 3
  + [`haoda`](https://github.com/Blaok/haoda), `pyverilog`
+ Google glog library (`libgoogle-glog-dev`)
+ Clang 8 and its headers (`clang-8`, `libclang-8-dev`)
+ Boost coroutine library (`libboost-coroutine-dev`)
+ Icarus Verilog (`iverilog`)
+ [FPGA Runtime](https://github.com/UCLA-VAST/fpga-runtime)

#### Build `tapacc`

```bash
mkdir build
cd build
cmake ..
make
make test
cd ..
sudo ln -s backend/python/tapac /usr/local/bin/
sudo ln -s build/backend/tapacc /usr/local/bin/
```

## Known Issues

+ Template functions cannot be tasks
+ Vitis HLS include paths (e.g., `/opt/Xilinx/Vitis_HLS/2020.2/include`) must not be specified in `tapac --cflags`;
  + Workaround is to use `CPATH` environment variable (e.g., `export CPATH=/opt/Xilinx/Vitis_HLS/2020.2/include`)

[tapa]: https://github.com/UCLA-VAST/tapa/
[autobridge]: https://github.com/Licheng-Guo/AutoBridge/
[aws-f1]: https://aws.amazon.com/ec2/instance-types/f1/
[d5005]: https://www.intel.com/content/www/us/en/programmable/products/boards_and_kits/dev-kits/altera/intel-fpga-pac-d5005/overview.html
[u250]: https://www.xilinx.com/products/boards-and-kits/alveo/u250.html
[iocl]: https://www.intel.com/content/www/us/en/software/programmable/sdk-for-opencl/overview.html
[xhls]: https://www.xilinx.com/products/design-tools/vitis.html
[soda]: https://github.com/UCLA-VAST/soda/
[autosa]: https://github.com/UCLA-VAST/AutoSA/

repository: tanvirarafin/tapa

repo_name: tanvirarafin/tapa


text:
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''

---

Please use the "Preview" tab above to view this message if you are seeing this in the new issue text box.

Please note that this is a **WoA Deployer** issue tracker, use this issue tracker only for the WoA Deployer related enhancements.

Please open ticket in other repositories for anything else, such as:

- Reporting bugs of other tools, drivers, etc.
- Asking for help
- Asking about project progress.

# Please remove the text above before opening an issue.

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

repository: WOA-Project/WOA-Deployer-Lumia

gha_language: C#

repo_name: WOA-Project/WOA-Deployer-Lumia


text:
# [Apache Curator](http://curator.apache.org/index.html)

## [Welcome to Apache Curator](http://curator.apache.org/index.html#)

Apache Curator is a Java/JVM client library for [Apache ZooKeeper](https://zookeeper.apache.org/), a distributed coordination service. It includes a highlevel API framework and utilities to make using Apache ZooKeeper much easier and more reliable. It also includes recipes for common use cases and extensions such as service discovery and a Java 8 **asynchronous DSL**.

> NOTE: 关于asynchronous DSL，参见`Programming-model\Asynchronous-programming`。

repository: dengking/Parallel-computing

repo_name: dengking/Parallel-computing


text:
---
title: Roulette Wheel
date: 2012-05-01T11:08:00+00:00
layout: post
permalink: /2012/05/01/roulette-wheel/
categories:
  - Daily Life
  - Faith
---

  I was working late yesterday, sitting at my desk, when my Grandmother called. She told me that my aunt had passed away over the weekend. I hung up the phone and sat there a while, wondering how I felt. I didn’t really know my aunt too well – the last time I’d seen her was at my cousin’s bridal shower. But I used to go to her house every Christmas Eve back in the old days. You know, Before Missions. Back before God spun the roulette wheel of my life and I became a bouncing black ball moving from mission house to airport to dorm room to new school…

  Maybe that’s why I get a sense of peace and stability when I think of her. Because I remember those days as being more peaceful and stable. Although I know that it really wasn’t. It’s just easier to look back and think that.

  But then, when I do sit quietly and look back, I can see that I was never really bouncing randomly from number to number. It was a very carefully controlled bounce. Landing in a classroom of English speakers in a French public school for a year. Stopping in a dorm room with a best friend for three months.  God has had it all planned out from the beginning of time. And I’m grateful for that.

  In the sad, scary, or simply confusing moments, I’m grateful for that.

  My life events are not random. They are not out of control. They do not depend on how hard someone spun the wheel. They rest in the Almighty’s hands. I rest in the Almighty’s hands. I cling to that. And I think, that at one point, my Aunt clung to that, too.

  Please pray for my cousins…


text:
# Offline Budget Tracker [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
  Budget tracker with offline functionality
  # Table of Contents
  1. [Detailed Description](#deatiled-description)
  2. [Installation](#installation)
  3. [Usage](#usage)
  4. [License](#license)
  5. [Contribution Guidelines](#contribution-guidelines)
  6. [Questions](#questions)
## Detailed Description
App deployed at [Heroku](https://mysterious-cove-07109.herokuapp.com/).

Apps look like ![screenshot](./shot.png)

Main idea of this project was adding offline functionality to budget tracker. It was realised with help of indexedDB, Service-workers and Cache.

File `db.js` contains code to work with indexedDB, that allows to store transaction when server is offline.

File `service-worker.js` contains code of service worker that works with cache and allows to "refresh" page even if there are no network connection.

## Installation
To install locally  please use:

`git clone`

`npm i`
## Usage
To use locally do next:
`npm start`
## License
[License: MIT](https://opensource.org/licenses/MIT)
## Contribution Guidelines
No contribution is needed at this moment. If you want to contribute you can contact me.
## Questions
For questionts please contact [me](https://github.com/Myau5x) at<EMAIL_ADDRESS>

branch_name: refs/heads/main


text:
## Configuration File

Configuration file must contains at least the _DEFAULT_ section with the OCI config and _OCI_TOOLS_ section with the script configuration.
OCI configuration file are also supported with the command line parameter __--profile__ ___<profile_name>___

Below the list of supported value in the _OCI_TOOLS_ section

#### region_filter
Comma separated region to work on.
Currently must not be empty.

```
region_filter=eu-frankfurt-1,uk-london-1
```

#### operation

available operation:
 - ___list___: scan the tenancy and list all the resources
 - ___cleanup___: delete all but toplevel compartments
 - ___delete___: delete all (currently = cleanup

```
operation=cleanup
```

#### preserve_compartments
Comma separated list of compartments to keep safe. The cleanup process ignores all the listed compartments and the resources that belong to them
```
preserve_compartments=comp_1,comp_2
```

#### preserve_tags
comma separated list of tag to keep safe.
The cleanup process ignores all the resources tagged with at least one of the listed tags
 - freeform tags must be inserted it the following format:
    - _key=value_
    - _key_
 - defined tags must be inserted in the following format:
    - _namespace:key=value_

```
preserve_tags=training.foundation=true,safe=true,not_delete
```

#### compartment_flter

Comma separated compartment OCID to limit the cleanup to.
The cleanup process will consider only the listed compartments
If empty all the compartment are terminated
```
compartment_flter=my_compartment
```

#### preserve_top_level_compartment
If true the cleanup script ignores the top level compartments
In case compartment_filter is used then the compartments specified is considered as top level compartments
___Default value___: _false_
```
preserve_top_level_compartment=true
```

#### preserve_compartments
Preserve the compartment structure. The cleanup script deletes all the resources but the compartments
___Default value___: _false_
```
preserve_compartments=true
```

#### simulate_deletion
simulate the cleanup without affect any resource
___Default value___: _false_
```
simulate_deletion=true
```

#### skip_scan_preserved_resources
if True don't inspect preserved resources (via _preserve_compartments_ or _preserve_tags_)
___Default value___: _true_
```
skip_scan_preserved_resources=false
```

repository: AnykeyNL/oci-tools

repo_name: AnykeyNL/oci-tools


text:
# sharedb-monaco

Fork of https://github.com/codecollab-io/sharedb-monaco

[![](https://img.shields.io/npm/v/sharedb-monaco)](https://github.com/Portatolova/sharedb-monaco/blob/master/LICENSE)
[![](https://img.shields.io/github/license/codecollab-io/sharedb-monaco)](https://github.com/Portatolova/sharedb-monaco/blob/master/LICENSE)

Two-way bindings between [ShareDB](https://github.com/share/sharedb) and the [Monaco Editor](https://github.com/microsoft/monaco-editor)

Developed for the [CodeCollab](https://codecollab.io) project.

Tested and works well with [monaco-react](https://github.com/suren-atoyan/monaco-react).

## Install
Using NPM:
```
$ npm install sharedb-monaco
```

## Usage

Sample usage of this library in action
```Javascript
import ShareDBMonaco from "sharedb-monaco";
import { editor } from "monaco-editor";

// Get a monaco editor (ICodeEditor) instance
let editor = editor.create(document.getElementById("editor"), {
    value: "print('Hello World!')",
    language: "python"
});

let binding = new ShareDBMonaco(options);

// Attach editor when document is initialised
binding.on("ready", () => {
    binding.add(editor, path);
});

```

### ShareDBMonaco Options
  * `opts.id` ID of the ShareDB document
  * `options.namespace` namespace of document within ShareDB, to be equal to that on the server
  * `options.wsurl` (Optional) Websocket URL for ShareDB Server API
  * `options.connection` (Optional) ShareDB Connection object. Pass this to reuse an existing Connection.

NOTE: Either `options.wsurl` or `options.connection` has to be set.

### ShareDBMonaco Instance
#### Events
  * `ready` Emitted when ShareDB document has initialised and ShareDBMonaco is ready for an editor to be attached.
  * `close` Emitted when the `close()` method is called.

#### Methods
`add(editor, path, viewOnly?)`:
Attaches an editor to the ShareDBMonaco instance

Parameters:
  * `editor` An instance of the monaco editor (iCodeEditor).
  * `path` Path on ShareDB JSON object to apply operations to. For example, if your ShareDB document is structured as ```{ foo: "", bar: "" }```, set ```path = "foo"``` for ShareDBMonaco to apply operations to ```foo```
  * `viewOnly (OPTIONAL)` Is the editor in viewOnly mode. Operations are treated differently for viewOnly editors.

`close()`: Closes the ShareDB and WebSocket connections between the server and this ShareDBMonaco instance. Also detaches any listeners on the editor.

## License
[MIT](https://github.com/codecollab-io/sharedb-monaco/blob/master/LICENSE)

repository: FlechaMaker/sharedb-monaco

repo_name: FlechaMaker/sharedb-monaco


text:
---
title: "Elasticsearch"
category: "hook"
type: "persistenceProvider"
state: "released"
usecase: "Publishes all Scan Findings to Elasticsearch."
---

## What is "Persistence ElasticSearch" Hook about?
The ElasticSearch persistenceProvider hook saves all findings and reports into the configured ElasticSearch index. This allows for some easy searching and visualization of the findings. To learn more about Elasticsearch visit [elastic.io].

Installing the Elasticsearch persistenceProvider hook will add a _ReadOnly Hook_ to your namespace.

## Deployment
The persistence-elastic chart can be deployed via helm:

```bash
# Install HelmChart (use -n to configure another namespace)
helm upgrade --install persistence-elastic secureCodeBox/persistence-elastic
```

## Requirements

Kubernetes: `>=v1.11.0-0`

| Repository | Name | Version |
|------------|------|---------|
| https://helm.elastic.co | elasticsearch | 7.17.3 |
| https://helm.elastic.co | kibana | 7.17.3 |

## Additional Chart Configurations

### Elasticsearch Indexing

For the elasticsearch `indexSuffix` you can provide a date format pattern. We use [Luxon](https://moment.github.io/luxon/) to format the date. So checkout
the [Luxon documentation](https://moment.github.io/luxon/docs/manual/formatting.html#table-of-tokens) to see what kind of format patterns you can use for the
`indexSuffix`. Default pattern is `yyyy-MM-dd`

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` |  |
| authentication | object | `{"apiKeySecret":null,"userSecret":null}` | Configure authentication schema and credentials the persistence provider should use to connect to elasticsearch user and apikey are mutually exclusive, only set one! |
| authentication.apiKeySecret | string | `nil` | Link a pre-existing generic secret with `id` and `key` key / value pairs |
| authentication.userSecret | string | `nil` | Link a pre-existing generic secret with `username` and `password` key / value pairs |
| dashboardImporter.enabled | bool | `true` | Enable if you want to import some example kibana dashboards for secureCodeBox findings analytics. |
| dashboardImporter.image.repository | string | `"securecodebox/persistence-elastic-dashboard-importer"` |  |
| dashboardImporter.image.tag | string | `nil` |  |
| elasticsearch | object | `{"enabled":true,"minimumMasterNodes":1,"replicas":1}` | Configures the included elasticsearch subchart (see: https://github.com/elastic/helm-charts/tree/elasticsearch) |
| elasticsearch.enabled | bool | `true` | Enable if you want to deploy an elasticsearch service. |
| elasticsearch.minimumMasterNodes | int | `1` | The value for discovery.zen.minimum_master_nodes. Should be set to (master_eligible_nodes / 2) + 1. Ignored in Elasticsearch versions >= 7 |
| elasticsearch.replicas | int | `1` | Kubernetes replica count for the StatefulSet (i.e. how many pods) |
| externalElasticStack.elasticsearchAddress | string | `"https://elasticsearch.example.com"` | The URL of the elasticsearch service to persists all findings to. |
| externalElasticStack.enabled | bool | `false` | Enable this when you already have an Elastic Stack running to which you want to send your results |
| externalElasticStack.kibanaAddress | string | `"https://kibana.example.com"` | The URL of the kibana service used to visualize all findings. |
| fullnameOverride | string | `""` |  |
| hook.affinity | object | `{}` | Optional affinity settings that control how the hook job is scheduled (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/) |
| hook.image.repository | string | `"docker.io/securecodebox/hook-persistence-elastic"` | Hook image repository |
| hook.image.tag | string | defaults to the charts version | The image Tag defaults to the charts version if not defined. |
| hook.labels | object | `{}` | Add Kubernetes Labels to the hook definition |
| hook.priority | int | `0` | Hook priority. Higher priority Hooks are guaranteed to execute before low priority Hooks. |
| hook.resources | object | { requests: { cpu: "200m", memory: "100Mi" }, limits: { cpu: "400m", memory: "200Mi" } } | Optional resources lets you control resource limits and requests for the hook container. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
| hook.tolerations | list | `[]` | Optional tolerations settings that control how the hook job is scheduled (see: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) |
| hook.ttlSecondsAfterFinished | string | `nil` | Seconds after which the kubernetes job for the hook will be deleted. Requires the Kubernetes TTLAfterFinished controller: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/ |
| imagePullSecrets | list | `[]` | Define imagePullSecrets when a private registry is used (see: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) |
| indexAppendNamespace | bool | `true` | Define if the name of the namespace where this hook is deployed to must be added to the index name. The namespace can be used to separate index by tenants (namespaces). |
| indexPrefix | string | `"scbv2"` | Define a specific index prefix used for all elasticsearch indices. |
| indexSuffix | string | `"“yyyy-MM-dd”"` | Define a specific index suffix based on date pattern (YEAR (yyyy), MONTH (yyyy-MM), WEEK (yyyy-'W'W), DATE (yyyy-MM-dd)). We use Luxon for date formatting (https://moment.github.io/luxon/docs/manual/formatting.html#table-of-tokens) |
| kibana | object | `{"enabled":true}` | Configures included Elasticsearch subchart |
| kibana.enabled | bool | `true` | Enable if you want to deploy an kibana service (see: https://github.com/elastic/helm-charts/tree/master/kibana) |
| nameOverride | string | `""` |  |
| nodeSelector | object | `{}` |  |
| podSecurityContext | object | `{}` |  |
| resources | object | `{}` |  |
| securityContext | object | `{}` |  |
| tolerations | list | `[]` |  |

## License
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

Code of secureCodeBox is licensed under the [Apache License 2.0][scb-license].

[scb-owasp]: https://www.owasp.org/index.php/OWASP_secureCodeBox
[scb-docs]: https://www.securecodebox.io/
[scb-site]: https://www.securecodebox.io/
[scb-github]: https://github.com/secureCodeBox/
[scb-twitter]: https://twitter.com/secureCodeBox
[scb-slack]: https://join.slack.com/t/securecodebox/shared_invite/enQtNDU3MTUyOTM0NTMwLTBjOWRjNjVkNGEyMjQ0ZGMyNDdlYTQxYWQ4MzNiNGY3MDMxNThkZjJmMzY2NDRhMTk3ZWM3OWFkYmY1YzUxNTU
[scb-license]: https://github.com/secureCodeBox/secureCodeBox/blob/master/LICENSE
[elastic.io]: https://www.elastic.co/products/elasticsearch

repository: secureCodeBox/secureCodeBox

gha_license_id: NOASSERTION

repo_name: secureCodeBox/secureCodeBox


text:
[![Build status](https://img.shields.io/travis/SURFnet/sp-dashboard.svg)](https://travis-ci.org/SURFnet/sp-dashboard)
[![License](https://img.shields.io/github/license/SURFnet/sp-dashboard.svg)](https://github.com/SURFnet/sp-dashboard/blob/master/LICENSE.txt)

# Service Provider Dashboard

The Service Provider Dashboard is a dashboard application where
[SURFconext](https://www.surf.nl/diensten-en-producten/surfconext/index.html) Service Providers can register and manage
their services. This can be both SAML 2.0 and OpenID Connect entities.

## Prerequisites

- [PHP](https://secure.php.net/manual/en/install.php) (5.6 or higher)
- [Composer](https://getcomposer.org/doc/00-intro.md)
- [Apache Ant](https://ant.apache.org/manual/install.html)
- [Ansible](https://docs.ansible.com/ansible/intro_installation.html)
- [Vagrant](https://www.vagrantup.com/docs/installation/)
  - Optional, but recommended: [Hostsupdater plugin](https://github.com/cogitatio/vagrant-hostsupdater)

The Ansible playbook for SP Dashboard depends on some roles from
[OpenConext-deploy](https://github.com/OpenConext/OpenConext-deploy), so in order to provision the Vagrant box you need
to have that repository checked out in a directory called `OpenConext-deploy` in the parent directory of where this
project lives.

## Getting started

First, run `composer install`. This will install all PHP dependencies, including the development dependencies.

In order to start the development environment, run `vagrant up`. This will build and start the virtual machine that is
used in development to run the application. When started for the first time, the Vagrant box will be provisioned using
Ansible.

Note: if you don't use the Vagrant Hostsupdater plugin, you have to manually add
`<IP_ADDRESS>  dev.support.surfconext.nl` to your hosts file so that requests will be routed to the virtual machine.

Install database migrations
```
bin/console doctrine:migrations:migrate
```

Now follow the instructions in [SAML configuration for development](docs/saml-dev-setup.md) to setup authentication.

The application is now up and running and can be accessed at
[https://spdashboard.dev.support.surfconext.nl/](https://spdashboard.dev.support.surfconext.nl/). Note that in development the `app_dev.php`
front controller is used automatically, so you don't have to include `/app_dev.php/` in the URLs.
* To view mails caught by Mailcatcher, visit [spdashboard.dev.support.surfconext.nl:1080](https://spdashboard.dev.support.surfconext.nl:1080/)

If you run into the `shibsp::ConfigurationException`, please reload your box, the issue should be resolved after a
reboot. The Shiboleth deamon might not come out 100% correctly out of the initial provisioning run.

### Running the tests

`ant test` will run the full suite of tests and static analysis.

### Xdebug
Xdebug is configured when provisioning your development Vagrant box. The Vagrantfile sets the `develop_spd`
environment variable in Ansible. This will prevent Xdebug from being enabled in test and production releases.

## Other resources

 - [SAML configuration for development](docs/saml-dev-setup.md)
 - [Developer documentation](docs/index.md)
 - [Issue tracker](https://www.pivotaltracker.com/n/projects/1400064)
 - [License](LICENSE.txt)

repository: BackEndTea/sp-dashboard

repo_name: BackEndTea/sp-dashboard


text:
# [Slides](../Notebooks/Pandas_Titanic.ipynb)

## Learning Objectives
- Learn all of the methods in pandas for data-frame manipulation
- Apply visualization to data-frame

### Titanic Dataset Description

VARIABLE DESCRIPTIONS:
survival        Survival
                (0 = No; 1 = Yes)
pclass          Passenger Class
                (1 = 1st; 2 = 2nd; 3 = 3rd)
name            Name
sex             Sex
age             Age
sibsp           Number of Siblings/Spouses Aboard
parch           Number of Parents/Children Aboard
ticket          Ticket Number
fare            Passenger Fare
cabin           Cabin
embarked        Port of Embarkation
                (C = Cherbourg; Q = Queenstown; S = Southampton)

```Python
import numpy as np
import pandas as pd

df = pd.read_csv('titanic.csv')
df.head()
```
### Plot how many of the passengers were children, youth, middle age and old?

```Python
import matplotlib.pyplot as plt

df['Age'].hist(bins=16)
plt.show()
```
### How many of Age values are empty (or null)?

```Python
# how many of Age values are null
df['Age'].isna().sum()
```
### Create a new column as gender, when Sex is female it is zero when sex is male it is one
```Python
# create a new column as gender, when Sex is female it is zero when sex is male it is one
df['Gender'] = df['Sex'].map( {'female': 0, 'male': 1} ).astype(int)
df.head()
```
### We have one more column (check it)

`df.shape`

### Show the majority of Age range

```Python
df['Age'].plot.box()
```

### List all of the Ages that are not null
```python
df['Age'].dropna().values
```

### Slice the dataframe for those whose Embarked section was 'C'

```Python
df[df['Embarked'] == 'C'].head()
```
### Plot the Age range for those whose Embarked were 'C'

```Python
# the age range of passenger whose their Embarked were 'C'
df[df['Embarked'] == 'C']['Age'].hist(bins=16)
```
### Apply couple of Normal Distributions to Histogram obtained above
```python
df[df['Embarked'] == 'C']['Age'].plot(kind='kde')
```
### Describe a specific column
```Python
df['Embarked'].describe()
```
### How many unique values does the 'Embarked' have?
```Python
df['Embarked'].nunique()
```
### Count the different 'Embarked' values the data-frame has
```Python
df['Embarked'].value_counts().plot(kind='bar')
```
### Count the different 'Embarked' values the data-frame has and plot horizontally
```Python
df['Embarked'].value_counts().plot('barh').invert_yaxis()

# Check df['Embarked'].value_counts().plot('barh')
```
### Another way to do the count and plot it

```Python
import seaborn as sns

# Bar Chart Example #1 (Simple): Categorical Variables Showing Counts
sns.countplot(x="Embarked", palette="spring", data=df)

df['Embarked'].value_counts()

df['Sex'].value_counts().plot(kind='bar')

df['Sex'].value_counts()
```

### Plot how many of the passengers were children, youth, middle age and old based on there Sex for those who 'Embarked' in section 'C'?

```Python
df[df['Embarked'] == 'C'].groupby('Sex')['Age'].hist(bins=16)

for i in df[df['Embarked'] == 'C'].groupby('Sex')['Age']:
    print(i)

df[df['Embarked'] == 'C'].groupby('Sex')['Age'].value_counts()
```

### What is the average Age for female and male (based on sex) for those who have 'Embarked' on section 'C'?

```Python
df[df['Embarked'] == 'C'].groupby('Sex')['Age'].agg(np.mean)
```
### Which Age is the oldest for female and male (based on sex) for those who have 'Embarked' on section 'C'?
```Python
df[df['Embarked'] == 'C'].groupby('Sex')['Age'].agg(np.max)
```
### For different Ages, plot the Fare they have paid?
```Python
sns.regplot(x="Age", y="Fare", fit_reg=False, data=df)
df.plot.scatter(x="Age", y="Fare")
```
### Plot how percentage Survived for two Sex group based on the passengers class
```python
sns.barplot(x="Sex", y="Survived", hue="Pclass", data=df)
```
### Plot how many male or female were in different Passenger classes
```Python
sns.countplot(x="Sex", hue="Pclass", data=df)
```
### Verify values obtained for pertentage
```Python
df[(df['Sex'] == 'female') & (df['Pclass'] == 1)]['Survived'].value_counts()
```
### Stack plot of count based on Sex for different Passenger Class

```Python
df.groupby(['Sex'])['Pclass'].value_counts().unstack().plot(kind='bar',stacked=True)
```

### Stack plot of count based on Sex and Survival for different Passenger Class
```python
df.groupby(['Sex', 'Survived'])['Pclass'].value_counts().unstack().plot(kind='bar',stacked=True)
```
### Sometimes it is hard to read values from plot, what are the number of female and male at each Passenger Class
```Python
# df.groupby(['Sex'])['Pclass'].value_counts().unstack()
# the above and crosstab are the same
pd.crosstab(df['Sex'], df['Pclass'])

pd.crosstab(df['Sex'], df['Survived'])

pd.crosstab(df['Sex'], df['Embarked'])

```
### How to represent the above cross tab in percentage and graphically present
```Python
sns.heatmap(pd.crosstab(df['Sex'], df['Embarked'], normalize='index'), cmap="YlGnBu", annot=True)
```

repository: KitsuneNoctus/makeschool

repo_name: KitsuneNoctus/makeschool


text:
# mruby-time-strftime   [![Build Status](https://travis-ci.org/monochromegane/mruby-time-strftime.svg?branch=master)](https://travis-ci.org/monochromegane/mruby-time-strftime)

Time#strftime for mruby.

## example

```ruby
p Time.now.strftime('%c')
#=> "Sat Jan  2 03:04:05 2016"
```

## Format directives

| format | description                                                       |
| ------ | ----------------------------------------------------------------- |
| Y      | Year with century if provided, will pad result at least 4 digits. |
| C      | year / 100 (rounded down such as 20 in 2009)                      |
| y      | year % 100 (00..99)                                               |
| m      | Month of the year, zero-padded (01..12)                           |
| B      | The full month name (January)                                     |
| b      | The abbreviated month name (Jan)                                  |
| h      | Equivalent to %b                                                  |
| d      | Day of the month, zero-padded (01..31)                            |
| e      | Day of the month, blank-padded ( 1..31)                           |
| j      | Day of the year (001..366)                                        |
| H      | Hour of the day, 24-hour clock, zero-padded (00..23)              |
| k      | Hour of the day, 24-hour clock, blank-padded ( 0..23)             |
| I      | Hour of the day, 12-hour clock, zero-padded (01..12)              |
| l      | Hour of the day, 12-hour clock, blank-padded ( 1..12)             |
| P      | Meridian indicator, lowercase (am or pm)                          |
| p      | Meridian indicator, uppercase (AM or PM)                          |
| M      | Minute of the hour (00..59)                                       |
| S      | Second of the minute (00..60)                                     |
| L      | Millisecond of the second (000..999)                              |
| Z      | Abbreviated time zone name or similar information. (OS dependent) |
| A      | The full weekday name  (Sunday)                                   |
| a      | The abbreviated name (Sun)                                        |
| u      | Day of the week (Monday is 1, 1..7)                               |
| w      | Day of the week (Sunday is 0, 0..6)                               |
| s      | Number of seconds since 1970-01-01 00:00:00 UTC.                  |
| %      | Literal `%` character                                             |
| c      | date and time (%a %b %e %T %Y)                                    |
| D      | Date (%m/%d/%y)                                                   |
| F      | The ISO 8601 date format (%Y-%m-%d)                               |
| v      | VMS date (%e-%^b-%4Y)                                             |
| x      | Same as %D                                                        |
| X      | Same as %T                                                        |
| r      | 12-hour time (%I:%M:%S %p)                                        |
| R      | 24-hour time (%H:%M)                                              |
| T      | 24-hour time (%H:%M:%S)                                           |

## install by mrbgems
- add conf.gem line to `build_config.rb`

```ruby
MRuby::Build.new do |conf|

    # ... (snip) ...

    conf.gem :github => 'monochromegane/mruby-time-strftime'
end
```

## TODO

- Format for week number of the year (`%U`, `%W`)
- Flags (`^`, `#`, `-`, `_`, `0`, `number`)

## License

[MIT](https://github.com/monochromegane/mruby-time-strftime/blob/master/LICENSE)

## Author

[monochromegane](https://github.com/monochromegane)

repository: monochromegane/mruby-time-strftime

gha_language: Ruby

repo_name: monochromegane/mruby-time-strftime


text:
# My MEAN stack under Docker

I've decided to Dockerize the MEAN stack, in order to ease my futur developments.

This stack is really basic, i really welcome improvments.

There is two ways to set up this stack. The first one is by getting the images from my DockerHub repository ; the second one is by building the images on your own, then runing it.

## Getting from the DockerHub

First of all, this is a quick way to set up the environnement. If you want to know more about the different images, please go to the "Building & Running on your own" section down bellow.

 As Docker creates a volume for each container on the host filesystem, I created this container to share its volume with other containers (like a hub).

You can build it by running a simple build command inside the directory.

(Assuming you're building with the tag "datastore") :

```shell
$> cd datastore
$> docker build -t datastore .
```

Then, you need to run it at least one time for the volume to be created on the host filesystem (I gave it the name of "my_datastore") :

```shell
$> docker run -d --name my_datastore datastore
```
You can check its existence with the following docker command :

```shell
$> docker ps -a
```

The volume has been created on the host filesystem, you can check it by listing the files in the /var/lib/docker/volumes directory.

### The Mongo

The MongoDB container can be built by running the build command in the appropriate directory (I gave him the tag "mongo"):

```shell
$> cd mongo
$> docker build -t mongo .
```

The "tricky" part is that I don't want the Mongo container to store data on his own. I would prefer that it stores data in the appropriate "datastore" container we built earlier.

In order to do so, we can use the --volumes-from parameter of the Docker "run" command. I also want to forward the default mongo port (27017) to the host port 27017. Finally, i run it in detached mode.

```shell
$> docker run -d --volumes-from my_datastore --name my_mongodb -p 27017:27017 mongo
```

Now, you can try to connect the host on the 27017 port, that will be forwarded to the container's 27017 port.

### The Node.js

The final part of the BackOffice of the MEAN Stack is the Node.js server, along with Express framework.

Building it :

```shell
$> cd node-express
$> docker build -t node .
```

In my stack, I assumed the app directory would be the /data/app directory on the host filesystem. I defined it in the package.json file. The base image I used in this container is the official node image. This image assumes that you have a package.json file in your directory, with a start script defined in it.

Then, i run the container in detached mode, and sharing the host's directory /data/app and port-forwarding the 8888 port.

```shell
$> docker run --name my_node -d -i -t  -v /data/app:/data/app -p 8888:8888 node
```

repository: quentinfayet/mean

repo_name: quentinfayet/mean


text:
---
layout: post
title: year++
tags:

---

It’s this time of the year again, which is traditionally an occasion to look back at what went down during the year, and set the sights for the year to come. 2009 has been a good year to me professionally. In spite of a gloomy economic context, Clear Lines is still alive and kicking, and revenue has remained stable, without requiring any particular marketing effort. I completed a few consulting projects without any noticeable glitch, managed to find time to release the first official version of Akin – and I am extremely motivated to begin 2010 on the right foot. So many things to do!

<!--more-->

## The blog

Readership has been going up through the year, slowly but regularly. I don’t think I’ll make the top blog list anytime soon, but I am still having fun doing it, so I’ll keep at it!

![website]({{ site.url }}/assets/2009-12-31-website_thumb.png)

The top visited posts this year were:

* [S-shaped market adoption curve]({{ site.url }}/2008/06/08/S-shaped-market-adoption-curve/)

* [Excel VSTO add-in menu manager]({{ site.url }}/2008/07/02/Excel-VSTO-add-in-menu-manager/)

* [Converting Excel date format to System.DateTime]({{ site.url }}/2008/09/05/Converting-Excel-date-format-to-SystemDateTime/)

* [VSTO-Add-In installation woes]({{ site.url }}/2008/08/29/VSTO-Add-In-installation-woes/)

* [Excel named ranges and INDIRECT()]({{ site.url }}/2008/04/22/Excel2c-named-ranges-and-INDIRECT()/)

* [Testing for exceptions with NUnit 2.5]({{ site.url }}/2009/05/11/Testing-for-exceptions-with-NUnit-25/)

This is interesting to me. Three of these are about using .Net and Excel together, two are “pure” Excel, and the first exclusively .Net post comes in 6th position only. Which tells me that, while .Net is well covered, there is a gap in VSTO-related information out there.

> 2010: like probably every other blogger, I could post more regularly. One per week it will be in 2010. I will work on publishing more resources on using Excel and .Net together.    If there are specific topics you want to hear about, I am all ears!

## Akin

I released [Akin](http://www.clear-lines.com/akin.aspx), my free application to compare Excel workbooks, with a few goals: learn WPF, get my feet wet with the process of releasing a piece of software, and simply produce something hopefully useful for the Excel power-users.

Lessons learnt: 1) WPF is great and I am not using WinForms again unless I have to, 2) getting actual input from users is not as easy as it seems. I got some useful feedback (thanks to [Debrah](http://blog.contextures.com/)>, [Jon](http://peltiertech.com/WordPress/) and [Richard](http://www.mrexcel.com/forum/showthread.php?t=406909) in particular), but while there were quite a few downloads, I haven’t heard back, which makes it difficult to improve the application!

![AkinDownloads]({{ site.url }}/assets/2009-12-31-AkinDownloads_thumb.png)

> 2010: I have a few features in mind for Akin, most notably, comparing the VBA code between workbooks, and producing a workbook summarizing the differences for reconciliations. If you have specific requests or criticisms, please let me know!

## Keeping up with technology

On the software end of things, I finally caught up with WPF, and I am glad I did. After a few projects using VSTO and Excel, I am now fairly comfortable with that technology. I also began learning F#, and intend to keep at it in 2010 – because I love math, and because it seems relevant to harness multi-cores. That being said, with the barrage of technology coming from Microsoft, it’s hard to stay on top of things, and select the right thing to spend your time on, and when you run a one-man show, you have to be solid in multiple areas. I have my work cut out for 2010.

> 2010: it’s time to improve my overall system architecture skills. I will likely experiment with Azure, very likely work on data-persistence related questions (an area I am still rather shallow in), and re-focus my attention on quantitative modeling methods, an area where I have not learnt much lately. I will also deepen my understanding of VSTO, specifically for Office 2007 and 2010.

## Balancing priorities

One of my goals when I started Clear Lines was to have more control over my time, and more freedom over what I spent my time on. I am definitely happy about the trade-off so far – I have had enough consulting projects to dedicate my time to various side-projects which keep me mentally fresh, and not worry too much about finances.  I also noticed that old habits die hard: I used to complain that work didn’t leave me enough time for fun, but now that the opportunity is here, I don’t always use it.

One very positive note this year has been my increased involvement with the .Net community, through the [Bay Area .Net user group](http://www.baynetug.org). I have now the honor of being co-chair of the San Francisco chapter, which gave me a chance to meet lots of great people!

> 2010: in parallel to a sustained consulting activity, I will work towards the goal of releasing product(s) to begin establishing a stable flow of revenue. I will also make time for fun and meeting people. Finally, I will work to make the San Francisco .Net user group an even better chapter than what it is now – a fun place for the .Net community to gather together!

That’s it.

gha_language: CSS


text:
# SWC Python Club Meetings

Repository to store prepared content & code for club meetings. Also contains an archive of content from past meetings.

## Archive

-   [2021-04-01](./meetings/2021-04-01)

## License

Unless explicitly stated otherwise, all code and content is licensed under the [MIT License](./LICENSE.md).

A notable exception are the CCC problem statements, which are not created by us, but are rather taken from the contest archive on the University of Waterloo website.

repository: vincor-qc/meetings

repo_name: vincor-qc/meetings


text:
# DeJongs' Palace

#### An application that helps user select their pizza preferances and have it delivered at their door step, 2.0
#### By **[Brian Ochieng]**
## Description
This is a web application that allows a user to place an order depending on the pizza size they'd like, the crust and also the kind of toppings they want placed on their pizza. It also allows useer to select the number of pizzas they'd like. Finally, the user gets to choose whether they'd want their pizza delivered or not, if they want it delivered, they are required to enter their location and add a little delivery fee.
Go ahead and try! Have your Pizza delivered in less than 40 minutes!
## screenshot
/Dejongs/images/Screenshot from 2019-10-12 16-44-47.png
## Setup/Installation Requirements
There are no set up or installation reqiurements
## Known Bugs
Apart from being able to take a maximum of 10 pizzas in quantity, there is no other bug.
## Technologies Used
* HTML5
* CSS3
* JAVASCRIPT
* JQuery
* Bootstrap CSS
* Markdown.
## BDD
| Behaviour      | Input        | Output       |
| :------------- | :----------: | -----------: |
| Choose your pizza size,crust type and topping   |  Medium,Mushroom topping & Cheesy crust.   |  summary of order made   ||
|Delivery option| yes|your order will be delivered at Komarock mall
| Press Submit|     |
## Support and contact details
Contribution and improvements to the application are welcome.
 Feel free to reach out to
 * Tel:+254 713 786 896
 *<EMAIL_ADDRESS>

repository: dejongbrian/Dejongs-Palace

gha_language: HTML

repo_name: dejongbrian/Dejongs-Palace


text:
# Changelog

## [2.0.1] - 07.03.2023

-   Fix: Fixed peer dependency of react-leaflet

## [2.0.0] - 06.03.2023

-   Refactor: Upgraded dependencies including React to 18 and Leaflet to 4

## [1.2.0] - 15.04.2020

-   Feat: Undo and redo actions

## [1.1.0] - 12.03.2020

-   Feat: Rectangle selection
-   Feat: Export polygons as GeoJSON, JTS or LatLng
-   Feat: Import polygons from GeoJSON
-   Fix: Prevent default behaviour for keyevents

## [0.0.1] - 08.03.2019

-   Fix: fix leaflet-path-drag dependency
-   Fix: change action bar position to the right bottom of the map

## [0.0.0] - 08.03.2019

-   Add tests
-   Add storybook
-   Add PoylgonDraw component
-   Create base project

repository: freenowtech/react-polygon-editor

gha_language: TypeScript

repo_name: freenowtech/react-polygon-editor


text:
#react-native-smartlocation
A Smart Location library for React Native. Uses combination Wi-Fi and GPS to get the best possible location.

Installation via rnpm
---
```
rnpm install react-native-smartlocation
```

repository: kenchangh/react-native-smartlocation

repo_name: kenchangh/react-native-smartlocation


text:
# Bolt-OAuth
A library to assist you with OAuth when using Slack's [Bolt](https://slack.dev/bolt) framework.

# Install

`npm install bolt-oauth`

# Usage

The library might seem complicated at first, but it's really not!

### Step 1: Setup Callback Functions
First, you need to create some callback functions for the OAuth events.

##### Success Callback
This is for when the OAuth was successful and nothing went wrong!

```
const oauthSuccess = ({ res, oAuthResult }) => {
  // save oAuthResult in MongoDB or somewhere else permanent since it has all the tokens
    res.redirect('<successfully_installed_thankyou_page_url>');
};
```
- `res` is an ExpressJS response object
- `oAuthResult` is the result from the Slack OAuth call with all the good stuff you need

A sample `oAuthResult` is
```
{
    ok: true,
    access_token: 'xoxp-12345678901-234567890123-abcdefghijklmnopqrstuvwxyzabcdef',
    scope: 'identify,bot,commands',
    user_id: 'U12345678',
    team_id: 'T12345678',
    enterprise_id: null,
    team_name: 'Sopinka Inc.',
    bot: {
        bot_user_id: 'U90123456',
        bot_access_token: 'xoxb-12345678901-456789012345-ghijklmnopqrstuvwxyzabcdefghijkl'
    },
    response_metadata: {}
}
```

##### Error Callback
This is for when the OAuth throws an error at any point.

```
const oauthError = (error) => {
    // do something about that error to let the user know
};
```

##### State Check
You'll want to implement this function if you're doing a state check on OAuth to prevent XSRF attacks. If you're not already, you'll want to do this as a best practice.

```
const oauthStateCheck = (oAuthState) => {
    // check the parameter state against your saved state to ensure everything is ok
    return true;
};
```

##### Authorize
Since you've got distribution on, and multiple teams using your app, you'll need to implement a custom authorize function, as outlined on the [official documentation](https://slack.dev/bolt/concepts#authorization).

```
const authorizeFn = ({ teamId, enterpriseId, userId, conversationId }) => {
    // go to your MongoDB or wherever you've stored the tokens and get the values based on teamId and/or userId
    return <your_tokens>;
};
```
`<your_tokens>` will need to be an object formatted something like this:
```
return {
    userToken: team.userToken,
    botToken: team.botToken,
    botId: team.botId,
    botUserId: team.botUserId
 }
```

### Step 2: Implement into your Bolt app
Now the fun stuff! Require the library and modify your Bolt `App` constructor to look something like this:

```
const { App } = require('@slack/bolt');
const Auth = require('bolt-oauth');

const app = new App({
    authorize: authorizeFn,
    receiver: Auth({
        clientId: process.env.SLACK_CLIENT_ID,
        clientSecret: process.env.SLACK_CLIENT_SECRET,
        signingSecret: process.env.SLACK_SIGNING_SECRET,
        redirectUrl: process.env.SLACK_REDIRECT_URL,
        stateCheck: oauthStateCheck,
        onSuccess: oauthSuccess,
        onError: oauthError,
        useSlackOauthV2: <true or false>
    })
});
```
- The environment variables above should be self-explanatory.
- All fields are **required** except
-- `useSlackOauthV2` which is true/false depending on whether you want to use the old Slack OAuth or the new V2 OAuth with granular scopes. Defaults to `false` if omitted.

### Step 3: Celebrate 🎉

That's it! You're all set! Now submit your app to the Slack App Directory and share your creation with the world!

# LICENSE

The MIT License (MIT)

Copyright (c) 2019 Alex Sopinka.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

 IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


text:
# Decoding Mirai Malware Traffic with Chopshop
## General Information
This is a presentation that covers using the open source tool Chopshop developed by Mitre to decode the Mirai DDoS Botnet command and control traffic.

### Why should we care?
Malware must run, and most malware wants to communicate. This communciation can do various things for malware actors including but not limited to:
* Exfiltrating stolen data and credentials
* Sending DDoS attacks
* Sending and retieving malicious instructions

Identifying and understanding these communications could give us insight behind the tools, tactics and procedures used by malicious actors and their motives. Once you have a clear understanding of the malware traffic you can create network signatures to detect this traffic and use the communication details to anticipate future behavior. For instance, if you have a malware command and control server that sends a DDoS attack details to an infected computer. You can expect that their will soon be DDoS attack traffic coming from the infected host. Knowing the target, attack type and duration allows defenders to prevent this traffic by blocking the target and/or packet types. However not all malware communication is straight forward and easy to interpret. Often times the malware traffic is encoded or encrypted to prevent inspection by malware analysts and network defenders.

To solve this analysis on the malware can to be done to identify how to decode its traffic, then tools such as Chopshop can be used to automatically decode the traffic. Chopshop developed by Mitre corp. This tool takes pcap files and provides the ability to run modules against them to conduct analysis. In additional to providing modules for general use, users are able to create their own modules for particular malware they are interested in. In this presentation I have analyzed Mirai command and control traffic to develop a module that would decode this traffic and extract information.

### Three Main Ideas
1. Most malware communicates as an essential part of its progression towards its objectives.
2. Sometimes malware traffic is encoded or decrypted making it difficult to understand
3. Tools like chopshop can be used to create modules that will automatically decode this traffic into a meaningful format.

## Conclusion
With Mitre's Chopshop, I was able to create a module that would decipher the Mirai DDoS Botnet command and control traffic. This process can be adapted to decode any malware or network traffic.

## Future Direction
In the future this research can be expanded upon by incorporating additonal modules and external libraries to provide coverage for more malware families.

## Additional Resources
* [Presentation Video (https://youtu.be/4bNoO8A-JYw)](https://youtu.be/4bNoO8A-JYw)
* [Mirai Module Github](https://github.com/tjnel/Decoding-Mirai-Malware-Traffic-with-Chopshop)
* [Chopshop Github](https://github.com/MITRECND/chopshop)
* [Mirai Traffic Research by Ixia](https://www.ixiacom.com/company/blog/mirai-botnet-things)


text:
# Trajectory of Malaysia COVID-19 cases
Inspired by aatishb and minutephysics, this project aims to monitor the Covid19 trends in Malaysia. These chart would tell us if our country's policy in combating Covid19 is effective or not. Additionally, such insight could help us in finding lights of hope during this difficult time.

Different from traditional cases versus time/date graph, these graphs gives the weekly new cases versus overall cases in log scales. The  trend of the outbreak would be in straight line instead of exponential curves. By monitoring these charts, we would be able to tell if the situation of the outbreak had been combat effectively (details to state). Based on the trend, we could pin point which state are coping well with the pandemic and which are not to further analysis better and efficient way in resolving the pandemic.

## Covid 19 trends of Malaysia

  Global Covid19 Trend in Malaysia (updated 21 September 2021, 12:00 AM),
  red markers indicate the date from 13 March to 01 April 2021

  Global Covid19 Trend in different state of Malaysia (updated 21 September 2021, 12:00 AM),
  red markers indicate the date from 13 March to 01 April 2021

## Original idea
>The graph plots the number of new confirmed cases in the past week, versus the total number of confirmed cases. When plotted in this way, exponential growth is displayed as a straight line.
*https://github.com/aatishb/covidtrends*

Screenshot from https://www.youtube.com/watch?v=54XLXg4fYsc indicating the trend of covid for log scale graph. This graph can be useful to identify if the pandemic is efficiently controlled by monitoring the downwards trend.

  Global Covid19 Trend by minutephysics

# Generate graphs and data
## Requirements
 - Python 3
   - numpy
   - requests
   - BeautifulSoup
   - scipy
   - matplotlib
   - pillow

Run generate_outbreak_json.py to create the json file as well as the graphs
>python generate_outbreak_json.py

# Credits
Special thanks to https://www.outbreak.my/ for data source.
Special thanks to https://aatishb.com/covidtrends/, and https://github.com/aatishb/covidtrends for the algorithm
Special thanks to minutephysics's comprehensive interpretation of the curve shown in https://www.youtube.com/watch?v=54XLXg4fYsc

# Supported by
Neuon AI Sdn Bhd.
www.neuon.ai


text:
# [Best Picture Oscars The Last 30 Years](http://alexa.amazon.com/#skills/amzn1.echo-sdk-ams.app.7141ab61-002b-42c0-b278-0efdaf46c2ff)
![4.6 stars](../../images/ic_star_black_18dp_1x.png)![4.6 stars](../../images/ic_star_black_18dp_1x.png)![4.6 stars](../../images/ic_star_black_18dp_1x.png)![4.6 stars](../../images/ic_star_black_18dp_1x.png)![4.6 stars](../../images/ic_star_half_black_18dp_1x.png) 5

To use the Best Picture Oscars The Last 30 Years skill, try saying...

* *Alexa, start Best Picture Oscars.*

* *My answer is four.*

* *Is it two?*

This is the perfect party game while waiting for the Oscars. Show off your movie trivia knowledge. Name the winner of the best picture between 1984 and 2015. Alexa will provide you with a year and four movie titles. You have to name the title that won best picture that year.  Prove it!

To start game, say "Alexa, play Best Picture Oscars"
Alexa will give you a year and four choices for what movie won best picture that year.
To answer, provide the number for your choice. Example: "Alexa, two."

***

### Skill Details

* **Invocation Name:** best picture oscars
* **Category:** null
* **ID:** amzn1.echo-sdk-ams.app.7141ab61-002b-42c0-b278-0efdaf46c2ff
* **ASIN:** B01C9E2H2I
* **Author:** QECE Publishing
* **Release Date:** February 26, 2016 @ 16:00:43
* **In-App Purchasing:** No


text:
---
title: "AWS S3 File Handling and Data Manipulation using Boto3 on Python."
date: 2020-06-17
categories: [Python]
tags: [AWS]
header:
  image: "/images/aws_s3_boto3/header.png"
excerpt: "Boto3 has widespread of methods and functionalities that are simple yet incredibly powerful. The objective of this notebook was to successfully make S3 Buckets, upload files to
it, made data modifications and discover ways to access private objects in the S3 buckets all this using python script with the help on Boto3"
mathjax: "true"
---

## INTRODUCTION TO AWS S3 USING BOTO3 IN PYTHON

**What is Boto3?**

Boto is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services.

Boto3 is built on the top of a library called Botocore, which is shared by the AWS CLI. Botocore provides the low level clients, session and credentials and configuration data. Boto3 built on the top of Botocore by providing its own session, resources, and collections.

**We will cover AWS S3 services in this notebook and how to access data and manipulate them using Boto3**

### What we will learn with this notbook?
- Make and delete Buckets
- Upload, Copy, Delete Objects within buckets
- Make security changes and ways to access the object from a Private Buckets

```python
# Import Library
import boto3

# Build a client to access the service methods
s3 = boto3.client('s3',region_name='us-east-1',
                  aws_access_key_id='****************',
                  aws_secret_access_key='****************')

```

```python
#List down the existing buckets
response = s3.list_buckets()
```

**The above code has empty output because we have no buckets created yet**
### Create S3 Buckets:

```python
#Create Bucket
s3.create_bucket(Bucket='example1811')
```

    {'ResponseMetadata': {'RequestId': '26151D4CBA49A11F',
      'HostId': 'GVFVb+AOuMcG3eFuhcrgoqCIMhjz/LK4G9jE1F2UxNiky3Yl33FBWklwOt6J4wal8vsa38fgMLs=',
      'HTTPStatusCode': 200,
      'HTTPHeaders': {'x-amz-id-2': 'GVFVb+AOuMcG3eFuhcrgoqCIMhjz/LK4G9jE1F2UxNiky3Yl33FBWklwOt6J4wal8vsa38fgMLs=',
       'x-amz-request-id': '26151D4CBA49A11F',
       'date': 'Tue, 16 Jun 2020 23:35:19 GMT',
       'location': '/example1811',
       'content-length': '0',
       'server': 'AmazonS3'},
      'RetryAttempts': 0},
     'Location': '/example1811'}

```python
s3.list_buckets()['Buckets']
```

    [{'Name': 'example1811',
      'CreationDate': datetime.datetime(2020, 6, 16, 23, 35, 19, tzinfo=tzutc())}]

```python
response = s3.delete_bucket(Bucket = 'example1811')
```

```python
s3.list_buckets()['Buckets']
```

    []

### Uploading Files:

We will use `upload_file` method.
It takes in 3 primary parameters:
- **Filename:** location of the file
- **Bucket:** Name of the Bucket you want to upload the file to
- **Key:** Name of the object file you are uploading

```python
# Creating the Bucket
s3.create_bucket(Bucket='example1811')
```

    {'ResponseMetadata': {'RequestId': '70C242129784AA04',
      'HostId': 'INZeeM11oVim4YDUOwg/A5gd3t4MhsVeAwvqhy43OdDDRWVbtDdGhNGjauRABApLJmumHfvkW5w=',
      'HTTPStatusCode': 200,
      'HTTPHeaders': {'x-amz-id-2': 'INZeeM11oVim4YDUOwg/A5gd3t4MhsVeAwvqhy43OdDDRWVbtDdGhNGjauRABApLJmumHfvkW5w=',
       'x-amz-request-id': '70C242129784AA04',
       'date': 'Tue, 16 Jun 2020 23:51:30 GMT',
       'location': '/example1811',
       'content-length': '0',
       'server': 'AmazonS3'},
      'RetryAttempts': 0},
     'Location': '/example1811'}

```python
# Uploadin the file
s3.upload_file(Filename= r'C:\Users\shrey\OneDrive\Desktop\s3\sjk1.txt',
               Bucket='example1811',
               Key= 'sjk1.txt')
```

We can provide `ExtraArgs` parameter to set and assigns the canned ACL (access control list) value 'public-read' to the S3 object. This will make your file accessable publicly. All public accesses are blocked by default.

```python
s3.list_objects(Bucket='example1811',MaxKeys=2,Prefix='sjk')['Contents']
```

    [{'Key': 'sjk1.txt',
      'LastModified': datetime.datetime(2020, 6, 16, 23, 51, 31, tzinfo=tzutc()),
      'ETag': '"bff45de157e360bd40661a2d264f74a8"',
      'Size': 12,
      'StorageClass': 'STANDARD',
      'Owner': {'DisplayName': 'Gandhi.ka',
       'ID': 'e9255dc22989ff9f56bb221d1755f1397e0f4d25c37f682aa7b9136a3b8a5417'}}]

`Prefix` parameter is to select multiple files with similarity in names

### Getting object metadata:
This plays important role in accessing data from private buckets that we will see ahead in this Notebook

```python
response = s3.head_object(Bucket='example1811',Key='sjk1.txt')
print(response)
```

    {'ResponseMetadata': {'RequestId': 'FEFFCC49AB85098F', 'HostId': '0RwM3oELZ6FYwaCtsbpmKg8t2pfXxrg/RPjSEC6coeFmQdyTVseCD8jEjpiOagNTS5tThvdGz90=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '0RwM3oELZ6FYwaCtsbpmKg8t2pfXxrg/RPjSEC6coeFmQdyTVseCD8jEjpiOagNTS5tThvdGz90=', 'x-amz-request-id': 'FEFFCC49AB85098F', 'date': 'Wed, 17 Jun 2020 00:04:30 GMT', 'last-modified': 'Tue, 16 Jun 2020 23:51:31 GMT', 'etag': '"bff45de157e360bd40661a2d264f74a8"', 'accept-ranges': 'bytes', 'content-type': 'binary/octet-stream', 'content-length': '12', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2020, 6, 16, 23, 51, 31, tzinfo=tzutc()), 'ContentLength': 12, 'ETag': '"bff45de157e360bd40661a2d264f74a8"', 'ContentType': 'binary/octet-stream', 'Metadata': {}}

### Downloading Objects from S3 Bucket:
This function is very similar to `upload_file` function as they have similar parameters.

```python
s3.download_file(Filename= r'C:\Users\shrey\OneDrive\Desktop\s3\sjk1.txt',
               Bucket='example1811',
               Key= 'sjk1.txt')
```

### Copying objects:
We can copy objects from one S3 bucket to another using Boto3

```python
s3.list_buckets()['Buckets']
```

    [{'Name': 'example1811',
      'CreationDate': datetime.datetime(2020, 6, 16, 23, 51, 30, tzinfo=tzutc())},
     {'Name': 'example1811b',
      'CreationDate': datetime.datetime(2020, 6, 17, 1, 16, 45, tzinfo=tzutc())}]

```python
CopySource = {'Bucket' : 'example1811',
             'Key':'sjk1.txt'}
```

```python
s3.copy_object(Bucket="example1811b", CopySource = CopySource, Key="sjk1.txt")
```

    {'ResponseMetadata': {'RequestId': '19B971F8D5491465',
      'HostId': 'GLZb7veZFmFaKTIPPuC+V+IpIWJ1gU/H4zEz17Wo+U9yJl0Wr6FO52x83IkX8xJ7aFs9lM45y5I=',
      'HTTPStatusCode': 200,
      'HTTPHeaders': {'x-amz-id-2': 'GLZb7veZFmFaKTIPPuC+V+IpIWJ1gU/H4zEz17Wo+U9yJl0Wr6FO52x83IkX8xJ7aFs9lM45y5I=',
       'x-amz-request-id': '19B971F8D5491465',
       'date': 'Wed, 17 Jun 2020 01:27:05 GMT',
       'content-type': 'application/xml',
       'content-length': '234',
       'server': 'AmazonS3'},
      'RetryAttempts': 0},
     'CopyObjectResult': {'ETag': '"bff45de157e360bd40661a2d264f74a8"',
      'LastModified': datetime.datetime(2020, 6, 17, 1, 27, 5, tzinfo=tzutc())}}

## Keeping objects secure:
This topic is very important. Companies have started using AWS to store their confidential data on cloud and it would not have been possible without aws's exceptional security.
It is a best practice to take all the steps in keeping your bucket private giving access permissions to only required users and IP addresses.

If you keep your bucket publically available it is as simple as putting this link "https://example1811.s3.amazonaws.com/sjk1811.txt" in your browser to access all the confidential data from anywhere in globe.

There are 4 main Access settings for these S3 Buckets:
- IAM Roles
- ACL
- Bucket Policy
- Presigned URL

![png](\images\aws_s3_boto3\security.png)
**Lets try making Object sjk.txt public for an example**

```python
s3.put_object_acl(Bucket='example1811', Key='sjk1.txt', ACL='public-read')
```

    {'ResponseMetadata': {'RequestId': 'DBD31162FEEB8CB5',
      'HostId': 'G+8ZhQ8wJfDKmXe1cdXFvOb60MohBQ/b4QLAEwnxbhAKkLP/8STudBP7W59zKqUPnLuo9aW9kSY=',
      'HTTPStatusCode': 200,
      'HTTPHeaders': {'x-amz-id-2': 'G+8ZhQ8wJfDKmXe1cdXFvOb60MohBQ/b4QLAEwnxbhAKkLP/8STudBP7W59zKqUPnLuo9aW9kSY=',
       'x-amz-request-id': 'DBD31162FEEB8CB5',
       'date': 'Wed, 17 Jun 2020 00:23:48 GMT',
       'content-length': '0',
       'server': 'AmazonS3'},
      'RetryAttempts': 0}}

**My nightmare has come true!!!** Just putting this link in my browser https://example1811.s3.amazonaws.com/sjk1811.txt downloads the file on my besktop.
You definetly do not want to be this guy.

```python
s3.put_object_acl(Bucket='example1811', Key='sjk1.txt', ACL='private')
```

    {'ResponseMetadata': {'RequestId': 'D4A15FA4470BE116',
      'HostId': 'YBHRXq9XB0rF/DJSxpua4OaYY0ybkFOfhy9ol6u4+wm24zIqrFTQ26JIhzt1UYn9saFPe3U1u6E=',
      'HTTPStatusCode': 200,
      'HTTPHeaders': {'x-amz-id-2': 'YBHRXq9XB0rF/DJSxpua4OaYY0ybkFOfhy9ol6u4+wm24zIqrFTQ26JIhzt1UYn9saFPe3U1u6E=',
       'x-amz-request-id': 'D4A15FA4470BE116',
       'date': 'Wed, 17 Jun 2020 00:26:12 GMT',
       'content-length': '0',
       'server': 'AmazonS3'},
      'RetryAttempts': 0}}

### Setting ACLs on uploading the files:
You can use `ExtraArgs` to set ACLs right when you upload the file.

```python
s3.upload_file(Bucket='example1811',Filename='sjk1.csv',
               Key='sjk1.csv',
               ExtraArgs={'ACL':'public-read'})
```

### How to access these private files?

This is not some sort of trick, you need body information of the object which is not possible without having the keys.

```python
import pandas as pd
```

```python
# getting obj information to retrieve `body`
obj = s3.get_object(Bucket='example1811', Key='sjk1.txt')
```

```python
obj['Body']
```

    <botocore.response.StreamingBody at 0x280195ef550>

```python
df = pd.DataFrame('<botocore.response.StreamingBody at 0x280195ef550>')
```

**There you have a Access to the file without any hussle**

### Pre-signed URLs
The presigned URLs are valid only for the specified duration. Anyone who receives the presigned URL can then access the object.
- The expiring time is in seconds as per ExpiresIn parameter provided.
- We use `Params` to provide information about what object from what bucket we want to give url access to for the given time.

```python
share_url = s3.generate_presigned_url(ClientMethod='get_object',ExpiresIn=3600,
                                      Params={'Bucket': 'example1811',
                                              'Key': 'sjk1.txt'})

```

**We can use this link to access the file. The link expires in 3600 seconds.**

```python
share_url
```

    'https://example1811.s3.amazonaws.com/sjk1.txt?AWSAccessKeyId=<REDACTED_AWS_ACCESS_KEY>&Signature=pZea4i%2BSakX68G43u1Tl3JnnuQc%3D&Expires=1592359332'

### Deleting objects:
It is very important to delete all the resources you built on aws if you are not using them anymore to avoid any form of charges.


text:
json-apis
========================

## Grails plugin for managing multiple JSON apis using domain class annotations

The goal of this plugin is to help convert Grails domain classes into various
JSON representations needed in different parts of your web application or to
support various API versions.

Features:

 - Allows you to declare multiple named JSON configurations for different use cases
 - Configuration is very straightforward: all that is required is to mark domain class
   properties with a single annotation naming the configurations under which that
   property should be included in the serialized JSON object
 - Works for collections as well as `belongsTo` properties
 - Enables developers to avoid the circular object reference problem elegantly by
   defining appopriate namespaces - this way it is possible to start the serialization
   either in a parent or child entity, depending on the use case
 - Works for objects serialized inside a [JSON builder](http://grails.org/doc/latest/guide/theWebLayer.html#moreOnJSONBuilder)
 - Uses the Grails' `ObjectMarshaller` mechanism under the hood
 - Detects API changes during development time live reloads

## Example of use

Several API variants can be easily defined in domain classes by annotating properties with
`JsonApi` and providing a list of API profile names under which that property should appear in the
resulting JSON. Marking a property with the `JsonApi` annotation but providing no API names will
include that property in all APIs. The database identity property will always be included
automatically. One could for instance define the following domain class:

```groovy
import grails.plugins.jsonapis.JsonApi

class User {
	@JsonApi
	String screenName

	@JsonApi('userSettings')
	String email

	@JsonApi(['userSettings', 'detailedInformation'])
	String twitterUsername
}
```

Then in the controller one would call the desired named JsonApi configuration to get only
the fields defined for that API. The following code:

```groovy
JSON.use("detailedInformation") {
	render person as JSON
}
```

...would convert the `person` object into JSON containing the `id`, `screenName` and `twitterUsername`
properties but not the `email`. It works for collections as well, converting each collection
member using the same API profile that was used to convert the parent:

```groovy
static hasMany = [
	pets: Pet
]
@JsonApi('detailedInformation')
Set pets
```

To include a domain object's parent in a JSON API, declare a `belongsTo` property explicitly
and annotate it with `JsonApi` (but be careful not to create circular paths by including both
ends of a `belongsTo`/`hasMany` pair):

```groovy
static belongsTo = [
	user:User
]

@JsonApi('petDetails')
User user
```

JSONBuilder is supported, too:

```groovy
JSON.use("userSettings") {
	render(contentType: "text/json") {
		user = User.first()
		pet = Pet.first()
	}
}
```

## Testing support

To register named marshallers in unit tests, you can use a static method that accepts the marshaller name
the classes you need registered:

```groovy
JsonApiRegistry.registerMarshaller("detailedInformation", ViciousPet)
```

## Future plans

 - Detect circular APIs and display a warning on startup, perhaps disable them entirely
 - Add a script/controller that would document the registered APIs in one or more formats
 - Read the domain class annotations and produce configurations for those 3rd party JSON
   renderers which currently seem to perform better than the native Grails implementation

repository: gregopet/grails-json-apis

gha_language: Groovy

repo_name: gregopet/grails-json-apis


text:
# Installation

If you have the [dependencies](#dependencies) installed, simply:

    pip install pywebview

To automatically fetch and install Python dependencies for your platform, install with the appropriate ["install extras"](http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies).

(Note that the same code is installed regardless; if more than one of the libraries below are available they will be tried in the order listed in this readme.)

On Windows (using WinForms with `pythonnet`):

    pip install pywebview[winforms]

On Mac (using the Cocoa WebKit widget via `pyobjc`):

    pip install pywebview[cocoa]

On Linux (using GTK3 via `PyGObject`):

    pip install pywebview[gtk3]

On Linux or Mac with either Qt 5:

    pip install pywebview[qt5]  # Qt5 with PyQt5

To use pywebview with PyQt4, you have to install it separately, as it is not available in pypi.

A second implementation for Windows using `pywin32` and `comtypes` is also available:

    pip install pywebview[win32]

# Dependencies

## Windows

On Windows you can choose between Win32 and Windows Forms implementation. The Win32 implementation is not actively developed and misses some features.

For Windows Forms
`pythonnet`

For Win32 you need
`pywin32`, `comtypes`. ActiveState distribution of Python 2 comes with pywin32 preinstalled

## OS X

`pyobjc`. PyObjC comes presintalled with the Python bundled in OS X. For a stand-alone Python installation you have to install it separately.

## Linux

For GTK3 based systems:`PyGObject`
On Debian based systems run
`sudo apt-get install python-gi gir1.2-webkit-3.0`

repository: hooloong/hooGnoolTools

repo_name: hooloong/hooGnoolTools


text:
---
layout: page
title: CSE
subtitle: "Exercises from 18.085: Computational Science and Engineering by Professor Gilbert Strang"
---

# Problem Set 1

$$
T_4^{-1} =
\begin{bmatrix}
4 & 3 & 2 & 1 \\
3 & 3 & 2 & 1 \\
2 & 2 & 2 & 1 \\
1 & 1 & 1 & 1 \\
\end{bmatrix}
$$

1. For every $$n$$, $$S_n = U_n^{-1}$$ is upper triangular with ones on and
above the diagonal. For every $$n=4$$ check that $$SS^T$$ produces the matrix
$$T_4^{-1}$$ predicted in Problem 1. Why is $$SS^T$$ certain to be a symmetric
matrix?

<br/>

# Problem Set 2

1. For a fixed-free line of 3 springs and masses (spring constants $$c_1$$,
$$c_2$$, $$c_3$$) write down $$A$$ and $$C$$ and $$K = A^T CA$$.

2. In this “statically determinate” problem with square matrices $$m = n$$ you
can do this: Invert $$K$$ by inverting its 3 factors $$A^T, C, A$$.

    Show that this $$K^{−1}$$ has all positive entries/all displacements are
    downward.

3. For one free-free spring sitting between 2 masses, show that the stiffness
matrix is

    $$
    “element \space matrix \space K” =
    \begin{bmatrix}
    c & −c  \\
    −c & c   \\
    \end{bmatrix}
    $$

    when $$m = 1$$.

    Now assemble the three element matrices for Problem 1 into the 3 by 3 matrix
    $$K$$ for Problem 1.

    The first element matrix will only be 1 by 1 because spring 1 is fixed at
    the top.

4. Suppose spring 2 gets weak and $$c_2$$ goes to zero. What is the limiting
$$K$$ when $$c_2 \approx 0$$? What are the properties of this limiting $$K$$? With
$$c_2 = 0$$ and $$c_1 = 1$$ and $$c_3 = 3$$, find the eigenvalues of $$K$$.
(OK to use `eig(K)` if you want.)

5. Let’s introduce random spring constants and run a number of simulations.
Choose $$c_1$$ and $$c_2$$ and $$c_3$$ each as `2 + rand(1)` in 100 simulations.

    Then try $$c_1$$, $$c_2$$, $$c_3$$ as `2 + rand(1)/10` 100 times. Find the
    averages of the matrices $$K^{−1}$$.

    Compare with $$c_1 = c_2 = c_3 = 2$$. Any interesting conclusions?
.