text:
---
date: 2021-09-23
title: "Performance Optimization on Kitex"
linkTitle: "Performance Optimization on Kitex"
description: >
author: ByteDance Architecture Team
---

## Preface
Kitex is the next generation high-performance and extensible Go RPC framework developed by Bytedance Service Framework Team. Compared with other RPC frameworks, in addition to its rich features for service governance,  it has the following characteristics: Integrated with the in-house developed network library - Netpoll; Supports multiple Message Protocols (Thrift, Protobuf) and Interactive Models (Ping-Pong, Oneway, Streaming); Provides a more flexible and extensible code generator.

Currently, Kitex has been widely used by the major lines of business in ByteDance, and statistics shows that the number of service access is up to 8K. We've been continuously optimizing Kitex's performance since its launch. This article will share our work on optimizing Netpoll and serialization.

## Optimization of the Network Library - Netpoll
Netpoll, the self-developed network library based on Epoll. Compared with the previous version and the go net library, its performance has been significantly improved. Test results indicated that compared with the last version (2020.05), the latest version (2020.12) has  ↑30% throughput capacity, ↓ 25% AVG  latency, and ↓67% TP99 . Its performance is far better than the Go Net library. Below, we'll share two solutions that can significantly improve its performance.

### Optimizing Scheduling Delays When Calling "epoll_wait"
When Netpoll was newly released, it encountered the problem of low AVG latency but high TP99. Through our research and analysis on "epoll_wait", we found that such a problem could be mitigated by integrating "polling" and "event trigger". With such improvements in scheduling strategy, the latency can be reduced considerably.

Let's have a look at the "syscall.EpollWait" method provided by Go first:
```go
func EpollWait(epfd int, events []EpollEvent, msec int) (n int, err error)
```
Three parameters are provided here, they represent "epoll fd", “callback events”, and "milliseconds to wait" respectively. Only "msec" is dynamic.

Normally, we would set "msec = -1" when we are actively calling "EpollWait",  as we want to wait for the event infinitely. In fact, many open source net libraries were also using it in this way. But our research showed that setting "msec =-1" was not the optimal solution.

The kernel source (below) of "epoll_wait" shows that setting "msec = -1" arises extra "fetch_events" checks than setting "msec = 0", and therefore consumes more time.
```go
static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
                   int maxevents, long timeout)
{
    ...
    if (timeout > 0) {
       ...
    } else if (timeout == 0) {
        ...
        goto send_events;
    }

fetch_events:
    ...
    if (eavail)
        goto send_events;

send_events:
    ...
```
The Benchmark shows that when an event is triggered, setting "msec = 0" is 18% faster than setting "msec = -1". Thus, when triggering complex events, setting "msec = 0" is obviously a better choice.
|Benchmark|time/op|bytes/op|
|:----|:----|:----|
|BenchmarkEpollWait, msec=0|270 ns/op|0 B/op|
|BenchmarkEpollWait, msec=-1|328 ns/op|0 B/op|
|EpollWait Delta|-17.68%|~|

However, setting "msec = 0" would lead to infinite polling when no event is triggered, consumes lots of resources.

Taking the previously mentioned factors into account, it's preferred to set "msec = 0" when an event is triggered and "msec = -1" when no event is triggered to reduce polling. The pseudocode is demonstrated as follow:
```go
var msec = -1
for {
   n, err = syscall.EpollWait(epfd, events, msec)
   if n <= 0 {
      msec = -1
      continue
   }
   msec = 0
   ...
}
```
Neverthless, our experiments have proved that the improvement is insignificant. Setting "msec = 0" merely reduces the delay of a single call by 50ns, which is not a considerable improvement. If we want to further reduce latency, adjustment must be made in Go runtime scheduling.
Thus, let's further explore this issue:
In the pseudocode above, setting "msec= -1" with no triggered event, and "continue" directly will immediately execute "EpollWait" again. Since there is no triggered event while "msec = -1",  the current "goroutine" will block and be switched by "P" passively. However, it is less efficient, and we can save time if we actively switch "goroutine" for "P" before "continue". So we modified the above pseudocode as follow:
```go
var msec = -1
for {
   n, err = syscall.EpollWait(epfd, events, msec)
   if n <= 0 {
      msec = -1
      runtime.Gosched()
      continue
   }
   msec = 0
   ...
}
```
The test results of the modified code showed that throughput ↑12% and TP99 ↓64%. The latency was significantly reduced.

### Utilizing "unsafe.Pointer"
Through further study of "epoll_wait", we find that the "syscall.EpollWait" published by Go and the "epollwait" used by "runtime" are two different versions, as they use different "EpollEvent". They are demonstrated as follow:
```go
// @syscall
type EpollEvent struct {
   Events uint32
   Fd     int32
   Pad    int32
}
// @runtime
type epollevent struct {
   events uint32
   data   [8]byte // unaligned uintptr
}
```
As we can see, the "epollevent" used by "runtime" is in its original structure defined by "epoll" at system layer. The published version warps it and splits "epoll_data(epollevent.data)" into two fixed fields: "Fd" and "Pad".  For "runtime", in its source code we found the following logic:
```go
*(**pollDesc)(unsafe.Pointer(&ev.data)) = pd

pd := *(**pollDesc)(unsafe.Pointer(&ev.data))
```
Obviously, "runtime" uses "epoll_data(&ev.data)" to store a pointer of the "fd" corresponding structure - "(pollDesc)" directly. Thus, when an event is triggered, the "struct" object can be found directly with the corresponding logic being executed. However, the external version can only obtain the warped "fd" parameters. So it needs to introduce additional "Map" to manipulate the "struct" object, and the performance will be diminished.

Therefore, we abandoned "syscall.EpollWait" and designed our own "EpollWait" call by referring to "runtime". We also use "unsafe.Pointer" to access "struct" objects. The test results showed that our "EpollWait" call contributed to ↑10% throughput and ↓10%  TP99 , which has significantly improved efficiency.

## Serialization/Deserialization Optimization of Thrift
Serialization refers to the process of converting a data structure or object into a binary or textual form. Deserialization is the opposite process. A serialization protocol needs to be agreed during RPC communication. The serialization process is executed before the client sends requests. The bytes are transmitted to the server over the network, and the server will logic-process the bytes to complete an RPC request. Thrift supports "Binary", "Compact", and "JSON" serialization protocols. Since "Binary" is the most common protocol used in Bytedance, we will only discuss about "Binary" protocol.

"Binary" protocol is "TLV" ("Type", "Length", "Value") encoded, that is, each field is described using "TLV" structure. It emphasizes that the "Value" can also be a "TLV" structure, where the "Type" and "Length" are fixed in length, and the length of "Value" is determined by the input value of "Length". TLV coding structure is simple, clear, and scalable. However, since it requires the input of "Type" and "Length", there is extra memory overhead incurred. It wastes considerable memory especially when most fields are in base types.

The performance of serilization and deserialization can be optimized from two dimensions - "time" & "space". To be compatible with the existing "binary" protocols, optimization in "space" seems to be infeasible. Improvement can only be made in "time", it includes:
- Reduce the frequency of operations on memory, notably memory allocation and replication. Try to allocate memory proactively to reduce unnecessary time consumption.
- Reduce the frequency of function calls by adjusting code structure or using "inline" etc.

### Research on Serilization Strategy
Based on "go_serialization_benchmarks", we investigated a number of serialization schemes that performed well to guide the optimization of our serialization strategy.

Analysis of "protobuf", "gogoprotobuf", and "Cap 'n Proto" has provided us the following results:
- Considering I/O, the transmitted data is usually compressed in size during online transactions. "protobuf" uses "Varint" encoding and has good data compression capabilities in most scenarios.
- "gogoprotobuf" uses precomputation to reduce memory allocations and replications during serialization. Thus, it eliminates the cost of system calls, locks and GC arise from memory allocations.
- "Cap 'n Proto" directly operates buffer, which also reduces memory allocations and replications. In addition, it also designs "struct pointer" in a way that processes fixed-length data and non-fixed-length data separately. Which enables fast processing for fixed-length data.
For compatibility reasons, it is impossible to change the existing "TLV" encoding format, so data compression is not feasible. But finding 2 and 3 are inspiring to our optimization work, and in fact we have taken a similar approach.

### Approaches
#### Reducing Operations on Memory
**Buffer management**
Both serialization and deserialization involve copying data from one piece of memory to another. It involves memory allocation and replications. Avoiding memory operations can reduce unnecessary overhead such as system calls, locks, and GC.

In fact, Kitex has provided "LinkBuffer" for buffer management purposes. "LinkBuffer" is designed with a linked structure and consists of multiple blocks, among which blocks are memory chunks with a fixed size. Object pool is constructed to maintain unoccupied block and support block multiplexing, thus, reduce memory usage and GC.

Initially we simply used "sync.Pool" to multiplex the "LinkBufferNode" of netpoll, but it didn't significantly contribute to multiplexing in large data package scenarios (large nodes can't be reclaimed or it would cause memory leaking). At present, we have changed our strategy to maintain a group of "sync.Pool", and the buffer sizes in each chunk are different. When new blocks are created, it is obtained from the pool with the closest size to the required size, so that the memory can be multiplexed as much as possible. And the test results also proved that it contributed to significant improvement in terms of memory allocation and GC.

**Copy-free String / Binary**
For some operations, such as video-related services, during its request or return processes, large-size "Binary" data will be arisen, representing the processed video or image data. Meanwhile, some operations will return large-size "String" data (such as full-text information, etc.). In this scenario, all the hot spots we see through the flame graph are on the copy of the data, so we thought, can we reduce the frequency of such copies?

The answer is positive. Since our underlying buffer is a linked list, it is easy to insert a node in the middle of the list.

![!image](/img/blog/buffer-linkerd-list.png)

Thus, we have taken a similar approach, when a "string" or "binary" data exists during serialization processes. First, split the node's buffer into two segments and then insert the buffer of the "string" / "binary" objects in the middle correspondingly. This avoids the copy of large "string" / "binary" .

Furthermore,  a copy will occur if we convert a string to "[]byte" using "[]byte(string)". Because "string" is immutable and "[]byte" is mutable in Golang language. "unsafe" is needed if you don't want to copy during the conversion:

```go
func StringToSliceByte(s string) []byte {
   l := len(s)
   return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
      Data: (*(*reflect.StringHeader)(unsafe.Pointer(&s))).Data,
      Len:  l,
      Cap:  l,
   }))
}
```
The demonstrated code first takes the address of the string, then gives it a slice byte header, and finally converts the "string" to "[]byte" without copying the data. Note that the resulting "[]byte" is not writable, or the behavior is undefined.

**Pre-Calculation**
Some services support transmissions of large data package, which incurs considerable serialization / deserialization overhead. Generally, large packets are associated with the large size of the container type. If the buffer can be pre-calculated, some O(n) operations can be reduced to O(1), and further reduce the frequency of function calls. In the case of large data packages, the number of memory allocation can also be greatly reduced, bringing considerable benefits.

- Base types
  - If the container element is defined as base type (bool, byte, i16, i32, i64, double), the total size can be pre-calculated during serialization as its size is fixed, and enough buffer can be allocated at once. The number of "malloc" operations of O(n) can be reduced to O(1), thus greatly reducing the frequency of "malloc" operations. Similarly, the number of "next" operations can be reduced during deserialization.

- Rearrangement of  "Struct" Fields
  - The above optimizations are valid only for container elements that are defined as base types. Can they be optimized for "struct" elements? The answer is yes.
  - If there are fields of base type in "struct", we can pre-calculate the size of these fields, then allocate buffer for these fields in advance during serialization and write these fields in the first order. We can also reduce the frequency of "malloc".

- Size calculation
  - The optimization mentioned above is for base types. If you first iterate over all the fields of the request during serialization, you can calculate the size of the entire request, allocate buffer in advance, and directly manipulate buffer during serialization and unsequence, so that the optimization effect can be achieved for non-base types.
  - Define a new "codec" interface:

```go
type thriftMsgFastCodec interface {
   BLength() int // count length of whole req/resp
   FastWrite(buf []byte) int
   FastRead(buf []byte) (int, error)
}
```

  - Change the "Marshal" and "Unmarshal" interfaces accordingly:

```go
func (c thriftCodec) Marshal(ctx context.Context, message remote.Message, out remote.ByteBuffer) error {
    ...
    if msg, ok := data.(thriftMsgFastCodec); ok {
       msgBeginLen := bthrift.Binary.MessageBeginLength(methodName, thrift.TMessageType(msgType), int32(seqID))
       msgEndLen := bthrift.Binary.MessageEndLength()
       buf, err := out.Malloc(msgBeginLen + msg.BLength() + msgEndLen)// malloc once
       if err != nil {
          return perrors.NewProtocolErrorWithMsg(fmt.Sprintf("thrift marshal, Malloc failed: %s", err.Error()))
       }
       offset := bthrift.Binary.WriteMessageBegin(buf, methodName, thrift.TMessageType(msgType), int32(seqID))
       offset += msg.FastWrite(buf[offset:])
       bthrift.Binary.WriteMessageEnd(buf[offset:])
       return nil
    }
    ...
}

func (c thriftCodec) Unmarshal(ctx context.Context, message remote.Message, in remote.ByteBuffer) error {
    ...
    data := message.Data()
if msg, ok := data.(thriftMsgFastCodec); ok && message.PayloadLen() != 0 {
   msgBeginLen := bthrift.Binary.MessageBeginLength(methodName, msgType, seqID)
   buf, err := tProt.next(message.PayloadLen() - msgBeginLen - bthrift.Binary.MessageEndLength()) // next once
   if err != nil {
      return remote.NewTransError(remote.PROTOCOL_ERROR, err.Error())
   }
   _, err = msg.FastRead(buf)
   if err != nil {
      return remote.NewTransError(remote.PROTOCOL_ERROR, err.Error())
   }
   err = tProt.ReadMessageEnd()
   if err != nil {
      return remote.NewTransError(remote.PROTOCOL_ERROR, err.Error())
   }
   tProt.Recycle()
   return err
   }
   ...
}
```
  - Modify the generated code accordingly:
```go
func (p *Demo) BLength() int {
        l := 0
        l += bthrift.Binary.StructBeginLength("Demo")
        if p != nil {
                l += p.field1Length()
                l += p.field2Length()
                l += p.field3Length()
    ...
        }
        l += bthrift.Binary.FieldStopLength()
        l += bthrift.Binary.StructEndLength()
        return l
}

func (p *Demo) FastWrite(buf []byte) int {
        offset := 0
        offset += bthrift.Binary.WriteStructBegin(buf[offset:], "Demo")
        if p != nil {
                offset += p.fastWriteField2(buf[offset:])
                offset += p.fastWriteField4(buf[offset:])
                offset += p.fastWriteField1(buf[offset:])
                offset += p.fastWriteField3(buf[offset:])
        }
        offset += bthrift.Binary.WriteFieldStop(buf[offset:])
        offset += bthrift.Binary.WriteStructEnd(buf[offset:])
        return offset
}
```
#### Optimize Thrift Encoding Using SIMD
"list<i64/i32>" is widely used in the company to carry the ID list, and the encoding method of "list<i64/i32>" is highly consistent with the rule of vectorization. Thus, we use SIMD to optimize the encoding process of list<i64/i32>.

We implement "avx2" to optimize the encoding process, and the optimized results are significant. When dealing with large amounts of data, the performance can be improved by 6 times for i64 and 12 times for i32. In the case of small data volume, the improvement is more obvious, which achieves 10 times for i64 and 20 times for I32.

#### Reducing Function Calls
**inline**
The purpose of "inline" is to expand a function call during its compliance and replace it with the implementation of the function. It improves program performance by reducing the overhead of the function call.

"inline" can't be implemented on all functions in Go. Run the process with the argument - (gflags="-m") to display the functions that are inline. The following conditions cannot be inlined:
- A function containing a loop;
- Functions that include: closure calls, select, for, defer, coroutines created by the go keyword;
- Functions over a certain length, by default when parsing the AST, Go applies 80 nodes. Each node consumes one unit of inline budget. For example, a = a + 1 contains five nodes: AS, NAME, ADD, NAME, LITERAL. When the overhead of a function exceeds this budget, it cannot be inlined.

You can specify the intensity (go 1.9+) of the compiler's inlined code by specifying -l at compile time. But it is not recommended, as in our test scenario, it is buggy and does not work:
```go
// The debug['l'] flag controls the aggressiveness. Note that main() swaps level 0 and 1, making 1 the default and -l disable. Additional levels (beyond -l) may be buggy and are not supported.
//      0: disabled
//      1: 80-nodes leaf functions, oneliners, panic, lazy typechecking (default)
//      2: (unassigned)
//      3: (unassigned)
//      4: allow non-leaf functions
```
Although using "inline" can reduce the overhead of function calls, it may also lead to lower CPU cache hit rate due to code redundancy. Therefore, excessive usage of "inline" should not be blindly pursued, and specific analysis should be carried out based on "profile" results.
```bash
go test -gcflags='-m=2' -v -test.run TestNewCodec 2>&1 | grep "function too complex" | wc -l
48

go test -gcflags='-m=2 -l=4' -v -test.run TestNewCodec 2>&1 | grep "function too complex" | wc -l
25
```
As you can see from the output above, increasing the inline intensity does reduce the "function too complex". Following are the benchmark results:

|Benchmark|time/op|bytes/op|allocs/op|
|:----|:----|:----|:----|
|BenchmarkOldMarshal-4|309 µs ± 2%|218KB|11|
|BenchmarkNewMarshal-4|310 µs ± 3%|218KB|11|

It reveals that turning on the highest level of inlining intensity does eliminate many functions that cannot be inlined due to "function too complex", but the test results show that the improvement is insignificant.

### Testing Results
We built benchmarks to compare performance before and after optimization, and here are the results.
Testing Enviornment:
Go 1.13.5 darwin/amd64 on a 2.5 GHz Intel Core i7 16GB

**Small Data Size**

Data size: 20KB

|Benchmark|time/op|bytes/op|allocs/op|
|:----|:----|:----|:----|
|BenchmarkOldMarshal-4|138 µs ± 3%|25.4KB|19|
|BenchmarkNewMarshal-4|29 µs ± 3%|26.4KB|11|
|Marshal Delta|-78.97%|3.87%|-42.11%|
|BenchmarkOldUnmarshal-4|199 µs ± 3%|4720|1360|
|BenchmarkNewUnmarshal-4|94µs ± 5%|4700|1280|
|Unmarshal Delta|-52.93%|-0.24%|-5.38%|

**Large Data Size**

Data size: 6MB

|Benchmark|time/op|bytes/op|allocs/op|
|:----|:----|:----|:----|
|BenchmarkOldMarshal-4|58.7ms ± 5%|6.96MB|3350|
|BenchmarkNewMarshal-4|13.3ms ± 3%|6.84MB|10|
|Marshal Delta|-77.30%|-1.71%|-99.64%|
|BenchmarkOldUnmarshal-4|56.6ms ± 3%|17.4MB|391000|
|BenchmarkNewUnmarshal-4|26.8ms ± 5%|17.5MB|390000|
|Unmarshal Delta|-52.54%|0.09%|-0.37%|

## Copy-free Serialization
In some services with large request and response data, the cost of serialization and deserialization is high. There are two ways for optimization:
- Implement the optimization strategy on serialization and deserialization as described earlier.
- Scheduling by copy-free serialization.

### Research on Copy-free Serilization
RPC through copy-free serialization, which originated from the "Cap 'n Proto" project of Kenton Varda. "Cap 'n Proto" provides a set of data exchange formats and corresponding codec libraries.

In essence, "Cap 'n Proto" creates a bytes slice as buffer, and all read & write operations on data structures are directly operated on buffer. After reading & writing, information contained by the buffer is added to the head and can be sent directly. And the peer end can read it after receiving it. Since there is no Go structure as an intermediate storage, serialization and deserialization are not required.

To briefly summarize the characteristics of "Cap 'n Proto":
- All data is read and written to a contiguous memory.
- The serialization operation is preceded. "Get/Set" data and encoding process in parallel.
- In the data exchange format, pointer ("offset" at the data memory) mechanism is used to store data at any location in the contiguous memory, so that data in the structure can be read and written in any order.
  - Fixed-size fields of a structure are rearranged so that they are stored in contiguous memory.
  - Fields with indeterminate size of a structure (e.g. list), are represented by a fixed-size pointer that stores information including the location of the data.
First of all, "Cap 'n Proto" has no Go language structure as an intermediate carrier, which can reduce a copy. Then, "Cap 'n Proto" operates on a contiguous memory, and the read and write of coded data can be completed at once. Because of these two reasons, Cap 'n Proto has excellent performance.

Here are the benchmarks of "Thrift" and "Cap 'n Proto" for the same data structure. Considering that "Cap 'n Proto" presets the codec operation, we compare the complete process including data initialization. That is, structure data initialization + (serialization) + write buffer + read from buffer + (deserialization) + read from structure.

```Thrift
struct MyTest {
    1: i64 Num,
    2: Ano Ano,
    3: list<i64> Nums, // 长度131072 大小1MB
}

struct Ano {
    1: i64 Num,
}
```

|Benchmark|Iter|time/op|bytes/op|alloc/op|
|:----|:----|:----|:----|:----|
|BenchmarkThriftReadWrite|172|6855840 ns/op|3154209 B/op|545 allocs/op|
|BenchmarkCapnpReadWrite|1500|844924 ns/op|2085713 B/op|9 allocs/op|
|ReadWrite Delta|/|-87.68%|-33.88%|-98.35%|

(deserialization) + read data, depending on the size of data package,"Cap 'n Proto" performance is about 8-9 times better than "Thrift". Write data + (serialization), depending on the size of data package, "Cap 'n Proto" performance is approximately 2-8 times better than "Thrift". Overall performance of "Cap 'n Proto" is approximately 4-8 times better than "Thrift".

Previously, we discussed the advantages of "Cap 'n Proto". We will then summarize some problems existing in "Cap 'n Proto":
- One problem with the contiguous memory of Cap 'n Proto is that when the data of variable size is resized and the required space is larger than the original space, the space of the original data can only be reallocated later. As a result, the original space becomes a hole that cannot be removed. This problem gets worse as the call link is resized, and can only be solved with strict constraints throughout the link: avoid resizing fields with indefined size, and when resize is necessary, rebuild a structure and make a deep copy of the data.
- "Cap 'n Proto" has no Go language structure as an intermediate carrier, so all fields can only be read and written through the interface, resulting in poor user experience.

### Thrift Protocol‘s Compatible Copy-free Serialization
In order to support copy-free serialization better and more efficiently, "Cap 'n Proto" uses a self-developed codec format, but it is difficult to be implemented in the current environment where "Thrift" and "ProtoBuf" are dominant. In order to achieve the performance of copy-free serialization with protocol compatibility, we started the exploration of copy-free serialization that is compatible with "Thrift" protocol.

"Cap 'n Proto" is a benchmark for copy-free serialization, so let's see if the optimizations on "Cap 'n Proto" can be applied to Thrift:
- Nature is the core of copy-free serialization, which does not use Go structure as intermediate carriers to reduce one copy. This optimization is not about a particular protocol and can be applied to any existing protocol (So it's naturally compatible with the Thrift protocol), but the user experience of "Cap 'n Proto" reflects that it needs to be carefully polished.
- "Cap 'n Proto" is operated on a contiguous memory. The read & write of the encoded data can be completed at once. "Cap 'n Proto" can operate on contiguous memory because there is a pointer mechanism that allows data to be stored anywhere, allowing fields to be written in any order without affecting decoding. However, on the one hand, it is very likely to leave a hole in resize due to misoperation on contiguous memory. Besides, "Thrift" has no pointer alike mechanism, so it has stricter requirements on data layout. Here are two ways to approach such problems:
  - Insist on contiguous memory operation, while imposing strict regulations on users' usage: 1. Resize operation must rebuild the data structure 2. When a structure is nested, there are strict requirements on the order in which the fields are written (we can think of it as unfolding a nested structure from the outside in, and being written in the same order) . In addition, due to TLV encoding such as Binary, when writing begins for each nesting, it requires declaration (such as "StartWriteFieldX").
  - Operating not entirely in contiguous memory, alterable fields are allocated a separate piece of memory. Since memory is not completely contiguous, the write operation can't complete the output at once. In order to get closer to the performance of writing data at once, we adopted a linked buffer scheme. On the one hand, when the variable field resize occurs, only one node of the linked buffer is replaced, and there is no need to reconstruct the structure like "Cap 'n Proto”. On the other hand, there is no need to clarify the actual structure like "Thrift" when the output is needed, just write the buffer on the link.
To summarize what we have determined previously: 1. Do not use Go structure as the intermediate carrier, directly operate the underlying memory through the interface, and complete the codec at the same time of "Get/Set". 2. Data is stored through a linked buffer.

Then let's take a look at the remaining issues:
- Degradation of the user experience caused by not using Go structures.
  - Solution: Improve the user experience of "Get/Set" interface and make it as easy to use as the Go structure.
- Binary Format of "Cap 'n Proto" is designed specifically for copy-free serialization scenarios, and although decoding is performed once for every Get, the decoding costs are minimal. The "Thrift" protocol (taking "Binary" as an example) has no mechanism that is similar to pointer. When there are multiple fields of variable size or nesting, they must be resolved sequentially instead of directly calculating the offset to get the field data location. Moreover, the cost of sequential resolution for each Get is too high.
  - Solution: In addition to recording the structure's buffer nodes, we also add an index that records the pointer to the buffer node at the beginning of each field with unfixed size. The following is the ultimate performance comparison test between the current copy-free serialization scheme and "FastRead/Write" under the condition of 4 cores:

|Package Size|Type|QPS|TP90|TP99|TP999|CPU|
|:----|:----|:----|:----|:----|:----|:----|
|1KB|Non-serialization|70,700|1 ms|3 ms|6 ms|/|
| |FastWrite/FastRead|82,490|1 ms|2 ms|4 ms|/|
|2KB|Non-serialization|65,000|1 ms|4 ms|9 ms|/|
| |FastWrite/FastRead|72,000|1 ms|2 ms|8 ms|/|
|4KB|Non-serialization|56,400|2 ms|5 ms|10 ms|380%|
| |FastWrite/FastRead|52,700|2 ms|4 ms|10 ms|380%|
|32KB|Non-serialization|27,400|/|/|/|/|
| |FastWrite/FastRead|19,500|/|/|/|/|
|1MB|Non-serialization|986|53 ms|56 ms|59 ms|260%|
| |FastWrite/FastRead|942|55 ms|59 ms|62 ms|290%|
|10MB|Non-serialization|82|630 ms|640 ms|645 ms|240%|
| |FastWrite/FastRead|82|630 ms|640 ms|640 ms|270

Summary of the test results:
- In small data package scenario, performance of non-serialization is poorer - about 85% of the "FastWrite/FastRead 's" performance .
- In large data package scenario, the performance of non-serialization is better. When processing packages larger than 4K, the performance of non-serialization is 7%-40% better compared with "FastWrite/FastRead".

## Postscript
Hope the above sharing can be helpful to the community. At the same time, we are also trying to share memory-based IPC, io_uring, TCP zero copy, RDMA, etc., to better improve Kitex performance; We will also focus on optimizing the communication under the contexts of same device and container. Welcome to join us and contribute to Go ecology together!

## Reference
- https://github.com/alecthomas/go_serialization_benchmarks
- https://capnproto.org/
- [Intel C++ Compiler Classic Developer Guide and Reference](https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/compiler-reference/intrinsics/intrinsics-for-intel-advanced-vector-extensions-2/intrinsics-for-shuffle-operations-1/mm256-shuffle-epi8.html)


text:
# Vanilla webtemplates

![wallpaper](./wallpaper.jpg)

A couple of WebTemplates that I use in various labs or testing situations
I used these to test webdeployment with different cloud providers, or to bring up
quickly a couple of websites in my lab.

These templates are not built by me, and the credits for the rights owners
can be found in links and text paragraphs where the owners intended for them
to appear.

## Content

* aitone
* breadfactory
* coffeewebsite
* cookingrecipe
* snack
* vertical-epilogue
* vertical-introspect
* zerotype

## Instructions

The templates are splitted by branches, and all of them are merged on master branch.
The ideea being that if you don't want to waste bandwidth, you can fetch only
one branch and do your thing.

repository: whadafunk/webplate

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: whadafunk/webplate


text:
# sarathi-consul-strategy
Implementation of Sarathi's discovery API for Consul.io

[![Build Status](https://travis-ci.org/nikhilw/sarathi-consul-strategy.svg?branch=master)](https://travis-ci.org/nikhilw/sarathi-consul-strategy) [![Coverage Status](https://coveralls.io/repos/github/nikhilw/sarathi-consul-strategy/badge.svg?branch=master)](https://coveralls.io/github/nikhilw/sarathi-consul-strategy?branch=master)

## Installation
```npm
npm install sarathi-consul-strategy --save
```

## Features
Lets you use [Sarathi](https://www.npmjs.com/package/sarathi) as declarative rest client with client-side load balancing when using [consul.io](https://www.consul.io/) discovery server.

## Usage
```javascript
var ConsulDiscoveryStrategy = require("sarathi-consul-strategy").DiscoveryStrategy;
var ds = new ConsulDiscoveryStrategy(...);
sarathiClientBuilder.setDiscoveryStrategy(ds);
```
## Options
* serviceId: ```String``` name of the service to discover, Ex: "testservice"
* client: ```Object``` instance of discovery service client, if you already have one; you might have used it for registering your service.
* clientConfig: ```Object``` Configuration to be passed to discovery service client, if you want sarathi to instantiate it. Client instance, if passed, takes precedence over this.
* refreshRate: ```Number``` in ms. timeout, to refresh discovered services. Ex: 30000 to mean every 30s.
* zone: ```String``` **not** yet supported, but for data-center awareness.
* discoveryTimeout: ```Number``` in ms. Timeout, time to wait for before the call to the discovery server times out.

## Examples
### When consul server is running locally
```javascript
var ds = new ConsulDiscoveryStrategy({serviceId: "express-service"});
```

### When consul client already instantiated
```javascript
var ds = new ConsulDiscoveryStrategy({serviceId: "express-service", client: consulClientInstance});
```

## API
A fluent API for setting configuration

### DiscoveryBuilder()
Object returned by ```require("sarathi-consul-strategy").StrategyBuilder```

#### #setClient(discoveryClient)
Set instance of the discovery server client. This shall come handy when you have already instantiated the client instance for registering with the server.

#### #setClientConfig(clientConfig)
Pass the config for sarathi to instantiate the client. If setClient and setClientConfig both are used, client instance passed will be used and the configuration provided to this method will be discarded. You can un-set the client instance by setting it to undefined.

#### #setRefreshRate(refreshRate)
Set service catalg refresh timeout

#### #setServiceId(serviceId)
Set the service name to look for on the discovery server

#### #setZone(zone)
**NOT** Implemented; but this is where you can set the data center preference

#### #setDiscoveryTimeout(timeout)
Sets the timeout period for throwing exception for when discovery server is taking too long to respond.

#### #build()
Builds the discovery handler instance and returns the instance of DiscoveryStrategy.

### ConsulDiscoveryStrategy
Sarathi consul discovery strategy

#### #getConsulInstance()
Returns the instance of consul client, if you needed to use it.

## Configuration Defaults
```javascript
{
	serviceId: undefined,
	client: undefined,
	clientConfig: {},
	refreshRate: 30000,
	zone: undefined,
	discoveryTimeout: 30000
}
```

license: MIT

repository: nikhilw/sarathi-consul-strategy

gha_language: JavaScript

gha_license_id: MIT

repo_name: nikhilw/sarathi-consul-strategy


text:
---
## [UT Austin Villa RoboCup 3D simulation team base code](https://github.com/LARG/utaustinvilla3d)

This product includes software developed by UT Austin Villa (http://www.cs.utexas.edu/~AustinVilla/).

repository: mitrecx/DynamicPlanningUncompleted

repo_name: mitrecx/DynamicPlanningUncompleted


text:
1000-digit Fibonacci number

> The Fibonacci sequence is defined by the recurrence relation:
>     Fn = Fn−1 + Fn−2, where F1 = 1 and F2 = 1.
> Hence the first 12 terms will be:
>     F1 = 1
>     F2 = 1
>     F3 = 2
>     F4 = 3
>     F5 = 5
>     F6 = 8
>     F7 = 13
>     F8 = 21
>     F9 = 34
>     F10 = 55
>     F11 = 89
>     F12 = 144
> The 12th term, F12, is the first term to contain three digits.
> What is the index of the first term in the Fibonacci sequence to contain 1000 digits?

Intrinsic Functions

```python
def main()
```

Solves this problem

Keeping a list of memoized values I increment (starting arbitrarily from 1000) and continue
to calculate the fibonacci values until the length of the str number is > 999.

This required utilization of a looped fibonacci method as the
indices exceed python's recursion limits.

Returns *Integer*: Solution to this problem

license: Unlicense


text:
[![Build Status](https://travis-ci.org/blemasle/gpstracker-firmware.svg?branch=master)](https://travis-ci.org/blemasle/gpstracker-firmware)
[![License](https://img.shields.io/badge/license-MIT%20License-blue.svg)](http://doge.mit-license.org)

# High autonomy GPS tracker.

This project aims to provides a GPS tracker based on an Arduino Pro mini as the central unit. Battery saving has been incorporated into every step of the design. A custom designed [PCB](https://github.com/blemasle/gpstracker-pcb) is available to avoid using poorly power efficient boards.

# Features
 * High autonomy (several weeks with a 1700 mAh battery)
 * Automatic acquisition of GPS position
 * Dynamically adjusted positions acquisition frequency based on speed and time of the day
 * Local buffer of 4095 positions with a 512K EEPROM
 * Automatic positions backup to any HTTP server using a GPRS powered network

# Requirements

## Hardware
This project has been build on a [ATMega328p](https://www.microchip.com/wwwproducts/en/ATMEGA328P) for its ease of use, low power consumption and wide support.

On top of that, the several ICs are needed :
* [SIM808](https://simcom.ee/documents/?dir=SIM808) : GPRS & GPS.
* [DS3231](https://www.maximintegrated.com/en/products/digital/real-time-clocks/DS3231.html) : Real Time Clock, used for time keeping and battery saving.
* [24LCxxx](https://www.microchip.com/wwwproducts/en/en010828) : EEPROM used to buffer raw data before sending.

## Using standard Arduino boards
This project can be build using standard Arduino boards and shields. For instance, you can start from an Arduino UNO with an Adafruit FONA shield.
This should only be done for testing and debugging purpose though, as both are not designed with energy saving in mind, and will rapidly consumes all your battery.

## Using a dedicated board
For optimum energy management and maximum autonomy, a double-sided, surface mounted [PCB](https://github.com/blemasle/gpstracker-pcb) has been created. Taking that path requires either using soldering paste or some fine soldering skills : the SIM808 IC can be quite hard to solder by hand.

## Software
The project has been build using Arduino 1.8.5. Ditching Arduino IDE in favor of an advanced IDE (VS Code, Visual Micro) is strongly recommended.

### Arduino libraries
Install the required libraries through the Arduino's Library Manager UI, or run the following script :
```sh
#!/bin/sh

arduino --install-library "E24"
arduino --install-library "Low-Power"
arduino --install-library "ArduinoLog"
arduino --install-library "MD_DS3231"
arduino --install-library "SIM808"
```

Disable unneeded features of MD_DS3231 for the HEX to fit in the ATMega328p by setting `ENABLE_12H`, `ENABLE_DOW` and `ENABLE_DYNAMIC_CENTURY` to `0` in [MD_DS3231.h](https://github.com/MajicDesigns/MD_DS3231/blob/master/src/MD_DS3231.h) or running `sed -E "s/^#define ENABLE_(12H|DOW|DYNAMIC_CENTURY) 1/#define ENABLE_\1 0/" -i MD_DS3231.h`

### Configuration
To fully use the project, you will need to adjust some values based on your needs. Each file under [src/config](src/config) can be modified, but the one you'll be most likely interested in are :
 * [User.h](/src/config/User.h) : Dumb, versionned file containing an example of the values in [Sensitive.h](/src/config/Sensitive.h)
 * Sensitive.h : Sensitive data that should *never* be versionned (phone number, web server address)
 * [Alerts.h](/src/config/Alerts.h) : Alerts levels
 * [BackupNetwork.h](/src/config/BackupNetwork.h) : Network backup and configuration
 * [Sleeps.h](/src/config/Pins.h) : Sleeps duration configuration

# Compiling
If you're using the [custom board](https://github.com/blemasle/gpstracker-pcb), set the board to `Arduino Pro or Pro Mini` and the processor to `ATMega328P (3.3V, 8Mhz)` before hitting `Verify` or `upload`.
Or simply run `arduino --verify --board arduino:avr:pro:cpu=8MHzatmega328 src/GpsTracker.ino`

# How does it work ?
Each `loop` execution procedes (roughly) through the following phases :
 * Acquisition of current GPS position
 * Triggering alerts if needed
 * Backuping positions if needed
 * Deep sleep until the next `loop`

Each one of these phases requires long waits (acquiring GPS signal, registering to the networks, etc). To preserve battery during those waits, all non-essentials devices are turned off, including the main ATMega328p. An interrupt coming from the RTC DS3231 chip (powered by the 3V coin battery) is used to woke up the main unit.

## Backuping positions
As positions backup can take quite some time, they're only backuped in two cases :
 * When stopping for a long time
 * When stopped and many positions are not yet backuped

This behaviour prevent the backup from distrupting the rate at which positions will be acquired, and avoid the device to look for the GPRS network while moving, which in some case can be difficult. As a downside, it is obvious that the positions are not backuped in realtime.

For reference, each position is sent in its own request HTTP POST request with a `Content-Type` set to `text/gpstracker` and a csv encoded body.

Field 				| Unit / Expression 	| Notes
--------------------|----------------------:|-------
freeRam 			| bytes 				| Read just before sending the position
signalAttenuation 	| dBm 					| Read once before the backup session
batteryLevel		| %						|
batteryVoltage		| mV					|
temperature			| °c 					| Expressed as `temp*100` as `printf` cannot deal with floats on Arduino
fixStatus			| `SIM808_GPS_STATUS`	|
timeToFix			| s						|

The rest of the string is the raw string returned by the [SIM808 `AT+CGNSINF` command](https://simcom.ee/documents/SIM800x/SIM800%20Series_GNSS_Application%20Note%20V1.00.pdf). The `UTC date & time` field can be used as a key for a given tracker.

You can then do whatever you want with the position, like learning Python and building a Google Timeline like website using OpenStreetMap (one day to be published) to track all your comings and goings during your nine months travel accross New-Zealand :sunglasses:.

# Limitations

## Adding features
The downside of using an ATMega328p is a relativly limited space. That space gets filled almost completly by the project in its current state, leaving very few possibilities for further improvments and features. Even adding Serial debugging statements can sometimes be impossible.

## Using an SD card
An SD card was first intended to be used for logging, data saving and configuration purposes. Due to the limited space available on an ATMega328p (32k), it has been removed early in the development process. Parts of the intended code (using [SdFat](https://github.com/greiman/SdFat)) is still in the repository, buts its use has been disabled.

## HTTP
Sadly, the SIM808 HTTP stack is unable to achieve reliable HTTPS. Currently, the backup is done with HTTP requests and there is no encryption, signature or authentication header (because without HTTPS, authentication header serves almost no purpose).
Any information on how to achieve proper and reliable HTTPS support or at least request signature (with the limited resource of an ATMega328p of course) is welcomed.

## Several firmwares
One possibility to include additional features would be to split the current firmware into several ones. For instance :
 - one "main" firmware
 - one "configuration" firmware to read and write EEPROM configuration rather than hardcoded values into the header files
 - one "debugging" firmware with additional logging and the ability to run specific parts of the main `loop` indepently

As I was using and needed the device *while* developing it, I couldn't take such a big risk that would potentionally break things and miss or lost some positions records.

repository: onethings/gpstracker-firmware

repo_name: onethings/gpstracker-firmware


text:
# Layout Reference

Qooxdoo comes with some of the most common layout managers. The following layout
managers are supported by Qooxdoo:

The Basic layout is used to position the children at absolute top/left
coordinates.

The Canvas layout is an extended Basic layout. It is possible to position a
widget relative to the right or bottom edge of the available space. The Canvas
layout furthermore supports dimension and location measures in percent.

The Box layouts arranges their children back-to-back. The horizontal box layout
arranges widgets in a horizontal row, from left to right, while the vertical box
layout arranges widgets in a vertical column, from top to bottom.

The Flow layout places widget next to each other from left to right. If the
available width is not sufficient an automatic line break is inserted.

A Dock layout attaches the children to the edges of the available space.

The Grid layout arranges items in a two-dimensional grid. Widgets can be placed
into the grid's cells and may span multiple rows and columns.

The Grow layout stretches all children to the full available size but still
respects limits configured by min/max values.

There are a few more layouts bundled with the default Qooxdoo distribution but
those are mostly intended for use by a specific component. For example the
widget/atom uses the [Atom Layout](apps://apiviewer/#qx.ui.layout.Atom) , the
widget/splitpane uses the two split layouts
[HLayout](apps://apiviewer/#qx.ui.splitpane.HLayout) and
[VLayout](apps://apiviewer/#qx.ui.splitpane.VLayout).

Through the simple API it should be quite easy to write custom layouts if the
included ones do not meet demands. Simply derive from the
[Abstract](apps://apiviewer/#qx.ui.layout.Abstract) layout and start with a
refined version of the method
[renderLayout()](apps://apiviewer/#qx.ui.layout.Abstract~renderLayout) .

repository: qooxdoo/qooxdoo

gha_license_id: NOASSERTION

repo_name: qooxdoo/qooxdoo


text:
![repo_log](/docs/repo_logo.png)

> `SAME AIRFLOW DATA PIPELINES | WHEREVER YOU RUN THEM`

# airflow-toolkit :rocket:

![Terragrunt Deployment-Validate Syntax and Plan](https://github.com/sungchun12/airflow-toolkit/workflows/Terragrunt%20Deployment-Validate%20Syntax%20and%20Plan/badge.svg) ![Checkov-Terraform Security Checks](https://github.com/sungchun12/airflow-toolkit/workflows/Checkov-Terraform%20Security%20Checks/badge.svg)

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND a Google Cloud Composer Airflow environment with working example DAGs across both :sparkles:

## Motivations

It is a painful exercise to setup secure airflow environments with parity(local desktop, dev, qa, prod). Too often, I've done all this work in my local desktop airflow environment only to find out the DAGs don't work in a Kubernetes deployment or vice versa. As I got more hands-on with infrastructure/networking, I was performing two jobs: Data and DevOps engineer. Responsibilities overlap and both roles are traditionally ill-equipped to come to consensus.  There's also the issue of security and DevOps saying that spinning up an airflow-dev-cloud-environment is too risky without several development cycles to setup bastion hosts, subnets, private IPs, etc. These conversations alone can lead to several-weeks delays before you can even START DRAFTING airflow pipelines! It doesn't have to be this way.

**This toolkit is for BOTH Data and DevOps engineers to solve the problems above** :astonished:

**High-Level Success Criteria:**

- Deploy airflow in 4 environments(local desktop, dev, qa, prod) in ONE day with this repo(save 4-5 weeks of development time)
- Confidence that base DAG integration components work based on successful example DAGs(save 1-2 weeks of development time)
- Simple setup and teardown for all toolkits and environments
- It FEELS less painful to iteratively develop airflow DAG code AND infrastructure as code
- Secure and private environments by default
- You are inspired to automate other painful parts of setting up airflow environments for others

## Use Cases

**In Scope**

- Airflow works on local computer with docker desktop installed
- Run meaningful example DAGs with passing unit and integration tests
- Easily setup and teardown any environment
- Sync DAGs real-time with local git repo directory(local desktop)
- Reusable configs in another kubernetes cluster
- Same DAGs work in both local desktop computer and Google Cloud Composer
- Secure cloud infrastructure and network(least privliges access and minimal public internet touchpoints)
- Low cost/Free(assumes you have Google Cloud free-trial credits)

**Out of Scope**

- Written end to end DAG tests as the dry runs are sufficient for this repo's scope
- `terratest` for terraform unit testing
- DAGs relying on external data sources outside this repo
- Hyper-specific IAM permission mechanics as it depends on your specific situation

## Table of Contents

- Pre-requisites
- Toolkit #1: Local Desktop Kubernetes Airflow Deployment
- Toolkit #2: Terragrunt-Driven Terraform Deployment to Google Cloud
- Toolkit #3: Simple Terraform Deployment to Google Cloud
- Post-Deployment Instructions for Toolkits #2 & #3
- Git Repo Folder Structure
- Frequently Asked Questions(FAQ)
- Resources

---

---

## Prerequisites

> Time to Complete: 5-10 minutes

1. [Sign up for a free trial](https://cloud.google.com/free/?hl=ar) _OR_ use an existing GCP account
2. Manually FORK the repo through the github interface _OR_ CLONE this repo: `git clone https://github.com/sungchun12/airflow-toolkit.git`

   ![fork git repo](/docs/fork-git-repo.png)

3. Create a new Google Cloud project

   ![create gcp project](/docs/create-gcp-project.gif)

4. Get into starting position for deployment: `cd airflow-toolkit/`

### One Time Setup for All Toolkits

> Time to Complete: 10-15 minutes

- [Download docker desktop](https://www.docker.com/products/docker-desktop) and start docker desktop

- Customize Docker Desktop for the below settings
- Click `Apply & Restart` where appropriate
  ![custom_resources](/docs/custom_resources.png)
  ![enable_kubernetes](/docs/enable_kubernetes.png)

- Run the below commands in your terminal

```bash
# install homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

# install helm, terragrunt, terraform, kubectl to local desktop
brew install helm terragrunt terraform kubectl

# Install Google Cloud SDK and follow the prompts
# https://cloud.google.com/sdk/install
curl https://sdk.cloud.google.com | bash
```

- Close the current terminal and start a new one for the above changes to take effect

- [Create a Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating)

- Add the `Editor`, `Secret Manager Admin`, `Role Administrator`, and `Security Admin` roles

  > Note: this provides wide permissions for the purposes of this demo, this will need to be updated based on your specific situation

- [Enable the Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts#enabling)

- [Create a Service Account Key JSON File-should automatically download](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys)

- Move private `JSON` key into the root directory of this git repo you just cloned and rename it `account.json`(don't worry it will be officially `gitignored`)

- Run the below commands in your local desktop terminal

```bash
# Authenticate gcloud commands with service account key file
gcloud auth activate-service-account --key-file account.json

# Enable in scope Google Cloud APIs
gcloud services enable \
    compute.googleapis.com \
    iam.googleapis.com \
    cloudresourcemanager.googleapis.com \
    bigquery.googleapis.com \
    storage-component.googleapis.com \
    storage.googleapis.com \
    container.googleapis.com \
    containerregistry.googleapis.com \
    composer.googleapis.com \
    secretmanager.googleapis.com

# Store contents of private service account key in Secrets Manager to be used by airflow later within the `add_gcp_connections.py` DAG
# Create a secrets manager secret from the key
gcloud secrets create airflow-conn-secret \
    --replication-policy="automatic" \
    --data-file=account.json

# List the secret
gcloud secrets list

# verify secret contents ad hoc
gcloud secrets versions access latest --secret="airflow-conn-secret"

# if you run the below toolkits multiple times, there may be times where you'll have to delete and recreate the secret
gcloud secrets delete airflow-conn-secret

# Optional: install specific Google Cloud version of kubectl
# The homebrew installation earlier above will suffice
# ┌──────────────────────────────────────────────────────────────────┐
# │               These components will be installed.                │
# ├─────────────────────┬─────────────────────┬──────────────────────┤
# │         Name        │       Version       │         Size         │
# ├─────────────────────┼─────────────────────┼──────────────────────┤
# │ kubectl             │             1.15.11 │              < 1 MiB │
# │ kubectl             │             1.15.11 │             87.1 MiB │
# └─────────────────────┴─────────────────────┴──────────────────────┘
gcloud components install kubectl

# Configure Docker
gcloud auth configure-docker

# Create SSH key pair for secure git clones
ssh-keygen
```

- Copy and paste contents of command below to your git repo SSH keys section

```bash
cat ~/.ssh/id_rsa.pub
```

- [Paste public ssh key contents location](https://github.com/settings/ssh/new)

- Manually create a `cloud source mirror repo` based on the GitHub repo

  > Note: documented to not be possible through the current state API-[further reading](https://issuetracker.google.com/issues/73122477)

  - [Full Instructions to Mirror a GitHub Repository](https://cloud.google.com/source-repositories/docs/mirroring-a-github-repository#create_a_mirrored_repository)

- Replace all relevant variables within `dags/` folder

```python
# file location
# /airflow-toolkit/dags/add_gcp_connections.py
CONN_PARAMS_DICT = {
    "gcp_project": "wam-bam-258119", # replace with your specific project
    "gcp_conn_id": "my_gcp_connection",
    "gcr_conn_id": "gcr_docker_connection",
    "secret_name": "airflow-conn-secret",
}

# file location
# /airflow-toolkit/dags/bigquery_connection_check.py
TASK_PARAMS_DICT = {
    "dataset_id": "dbt_bq_example",
    "project_id": "wam-bam-258119", # replace with your specific project
    "gcp_conn_id": "my_gcp_connection",
}

# file location
# /airflow-toolkit/dags/airflow_utils.py
GIT_REPO = "github_sungchun12_airflow-toolkit" # replace with the cloud source mirror repo name
PROJECT_ID = "wam-bam-258119" # replace with your specific project
```

> After doing the above ONCE, you can run the below toolkits multiple times with the same results(idempotent)

---

---

## Toolkit #1: Local Desktop Kubernetes Airflow Deployment

> Time to Complete: 5-8 minutes

> Note: This was ONLY tested on a Mac desktop environment

### System Design

![local_desktop_airflow.png](/docs/local_desktop_airflow.png)

### Specific Use Cases

- Free local dev environment
- Customize local cluster resources as needed
- Rapid DAG development without waiting for an equivalent cloud environment to sync DAG changes
- Experiment with a wider array of customization and permissions
- Minimal knowledge of kubernetes and helm required

### How to Deploy

**General Mechanics**

> shell script logs will generate similar mechanics

- Set environment variables
- Build and push a custom dbt docker image
- Copy Docker for Desktop Kube Config into git repo
- Download stable helm repo
- Setup Kubernetes airflow namespace
- Create Kubernetes Secrets for the local cluster to download docker images from Google Container Registry based on Service Account
- Create Kubernetes Secrets for dbt operations based on Service Account and ssh-keygen, to be used later in KubernetesPodOperator
- Setup and Install Airflow Kubernetes Cluster with Helm
- Wait for the Kubernetes Cluster to settle
- Open airflow UI webserver from terminal

---

> Note: I plan to automate this yaml setup in a future feature

- Manually update the extraVolumes section within `custom-setup.yaml`, starting at `line 174`
- Run `pwd` in your terminal from the root `airflow-toolkit/` directory and replace all the `<YOUR GIT REPO DIRECTORY HERE>` placeholders

```yaml
extraVolumes: # this will create the volume from the directory
  - name: dags
    hostPath:
      path: <YOUR GIT REPO DIRECTORY HERE>/dags/
  - name: dag-environment-configs
    hostPath:
      path: <YOUR GIT REPO DIRECTORY HERE>/dag_environment_configs/
  - name: kube-config
    hostPath:
      path: <YOUR GIT REPO DIRECTORY HERE>/.kube/
  - name: service-account
    hostPath:
      path: <YOUR GIT REPO DIRECTORY HERE>/account.json
  - name: tests
    hostPath:
      path: <YOUR GIT REPO DIRECTORY HERE>/tests/

# example below
extraVolumes: # this will create the volume from the directory
  - name: dags
    hostPath:
      path: /Users/sung/Desktop/airflow-toolkit/dags/
  - name: dag-environment-configs
    hostPath:
      path: /Users/sung/Desktop/airflow-toolkit/dag_environment_configs/
  - name: kube-config
    hostPath:
      path: /Users/sung/Desktop/airflow-toolkit/.kube/
  - name: service-account
    hostPath:
      path: /Users/sung/Desktop/airflow-toolkit/account.json
  - name: tests
    hostPath:
      path: /Users/sung/Desktop/airflow-toolkit/tests/
```

- Run the below commands in your terminal

```bash
#!/bin/bash
# follow terminal prompt after entering below command
# leave this terminal open to sustain airflow webserver
# Set of environment variables
export ENV="dev"
export PROJECT_ID="airflow-demo-build"
export DOCKER_DBT_IMG="gcr.io/$PROJECT_ID/dbt_docker:$ENV-latest"

source deploy_local_desktop_airflow.sh
```

- Turn on all the DAGs using the on/off button on the left side of the UI
- After waiting a couple minutes, all the DAGs should succeed
  > Note: `bigquery_connection_check` will fail unless `add_gcp_connections` succeeds first

![local_desktop_airflow.png](/docs/local_desktop_airflow_success.png)

> Note: the airflow webserver may freeze given resource limitations

> press `ctrl + c` within the terminal where you ran the deploy script

> Run the below commands in your terminal(these already exist within the deploy script)

```bash
# view airflow UI
export POD_NAME=$(kubectl get pods --namespace airflow -l "component=web,app=airflow" -o jsonpath="{.items[0].metadata.name}")

echo "airflow UI webserver --> http://<IP_ADDRESS>:8080"

kubectl port-forward --namespace airflow $POD_NAME 8080:8080
```

- Open a SEPARATE terminal and run the below commands

```bash
# start a remote shell in the airflow worker for ad hoc operations or to run pytests
kubectl exec -it airflow-worker-0 -- /bin/bash
```

- Airflow worker remote shell examples

```bash
➜  airflow-toolkit git:(feature-docs) ✗ kubectl exec -it airflow-worker-0 -- /bin/bash

# list files in current working directory
airflow@airflow-worker-0:/opt/airflow$ ls
airflow.cfg  dag_environment_configs  dags  logs  tests  unittests.cfg

# run all test scripts
airflow@airflow-worker-0:/opt/airflow$ pytest -vv --disable-pytest-warnings
======================================== test session starts ========================================
platform linux -- Python 3.6.10, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
plugins: celery-4.4.2
collected 19 items

tests/test_add_gcp_connections.py::test_import_dags PASSED                                    [  5%]
tests/test_add_gcp_connections.py::test_contains_tasks PASSED                                 [ 10%]
tests/test_add_gcp_connections.py::test_task_dependencies PASSED                              [ 15%]
tests/test_add_gcp_connections.py::test_schedule PASSED                                       [ 21%]
tests/test_add_gcp_connections.py::test_task_count_test_dag PASSED                            [ 26%]
tests/test_add_gcp_connections.py::test_tasks[t1] PASSED                                      [ 31%]
tests/test_add_gcp_connections.py::test_tasks[t2] PASSED                                      [ 36%]
tests/test_add_gcp_connections.py::test_end_to_end_pipeline SKIPPED                           [ 42%]
tests/test_dbt_example.py::test_import_dags PASSED                                            [ 47%]
tests/test_dbt_example.py::test_contains_tasks PASSED                                         [ 52%]
tests/test_dbt_example.py::test_task_dependencies PASSED                                      [ 57%]
tests/test_dbt_example.py::test_schedule PASSED                                               [ 63%]
tests/test_dbt_example.py::test_task_count_test_dag PASSED                                    [ 68%]
tests/test_dbt_example.py::test_dbt_tasks[dbt_debug] PASSED                                   [ 73%]
tests/test_dbt_example.py::test_dbt_tasks[dbt_run] PASSED                                     [ 78%]
tests/test_dbt_example.py::test_dbt_tasks[dbt_test] PASSED                                    [ 84%]
tests/test_dbt_example.py::test_end_to_end_pipeline SKIPPED                                   [ 89%]
tests/test_sample.py::test_answer PASSED                                                      [ 94%]
tests/test_sample.py::test_f PASSED                                                           [100%]

============================= 17 passed, 2 skipped, 1 warning in 39.77s =============================

# list DAGs
airflow@airflow-worker-0:/opt/airflow$ airflow list_dags
[2020-08-18 19:17:09,579] {__init__.py:51} INFO - Using executor CeleryExecutor
[2020-08-18 19:17:09,580] {dagbag.py:396} INFO - Filling up the DagBag from /opt/airflow/dags
Set custom environment variable GOOGLE_APPLICATION_CREDENTIALS for deployment setup: local_desktop
Set custom environment variable GOOGLE_APPLICATION_CREDENTIALS for deployment setup: local_desktop

-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
add_gcp_connections
bigquery_connection_check
dbt_example
kubernetes_sample
tutorial

# import, get, set airflow variables
airflow@airflow-worker-0:/opt/airflow$ airflow variables --import /opt/airflow/dag_environment_configs/test_airflow_variables.json
1 of 1 variables successfully updated.
airflow@airflow-worker-0:/opt/airflow$ airflow variables --get test_airflow_variable

do you see this?
airflow@airflow-worker-0:/opt/airflow$ airflow variables --set test_airflow_variable "lovely"
airflow@airflow-worker-0:/opt/airflow$ airflow variables --get test_airflow_variable
lovely
```

### How to Destroy

> Time to Complete: 1-2 minutes

```bash
#!/bin/bash
source teardown_local_desktop_airflow.sh
```

- Example terminal output

```bash
➜  airflow-toolkit git:(feature-docs) ✗ source teardown_local_desktop_airflow.sh
***********************
Delete Kuberenetes Cluster Helm Deployment and Secrets
***********************
release "airflow" uninstalled
kill: illegal process id: f19
secret "dbt-secret" deleted
secret "gcr-key" deleted
secret "ssh-key-secret" deleted
namespace "airflow" deleted
```

### How to give local desktop airflow more horsepower :horse:

- [Scaling airflow resources](https://www.astronomer.io/guides/airflow-scaling-workers/)

### Tradeoffs

#### Pros

- Simple setup and no need to worry about manually destroying airflow cloud infrastructure
- Free(minimal to no charges to your Google Cloud account)
- Same example DAGs work as is in cloud infrastructure setup
- Minimal kubernetes/helm skills required to start
- Ability to scale infrastructure locally(ex: add more worker pods)
- Run `pytest` directly within this setup

#### Cons

- Limited to how much processing power and storage you have on your local desktop machine
- Airflow webserver can freeze occasionally and requires quick restart

### General Kubernetes Concepts

- The local desktop airflow kubernetes cluster will use the service account within the `airflow` namespace to pull the image from Google Container Registry based on the manually created secret: `gcr-key`

```bash
kubectl get serviceaccounts

NAME      SECRETS   AGE
airflow   1         43m
default   1         43m
```

- The `KubernetesPodOperator` will pull the docker image based on the permissions above BUT will run `dbt` operations based on the manually created secret: `dbt-secret`

```bash
kubectl get secrets

NAME                            TYPE                                  DATA   AGE
airflow-postgresql              Opaque                                1      50m
airflow-redis                   Opaque                                1      50m
airflow-token-zfpz8             kubernetes.io/service-account-token   3      50m
dbt-secret                      Opaque                                1      50m
default-token-pz55g             kubernetes.io/service-account-token   3      50m
gcr-key                         kubernetes.io/dockerconfigjson        1      50m
sh.helm.release.v1.airflow.v1   helm.sh/release.v1                    1      50m
```

### View Local Kubernetes Dashboard

> Optional: Detailed resource management view for local desktop

```bash
# install kubernetes dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml

# start the web server
kubectl proxy

# view the dashboard
open http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy#/login

# copy and paste the token output into dashboard UI
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | awk '/^deployment-controller-token-/{print $1}') | awk '$1=="token:"{print $2}'
```

![kube_resource_dashboard](/docs/kube_resource_dashboard.png)

> Enter `ctrl + c` within the terminal where you ran the kubernetes dashboard script to close it

---

---

## Toolkit #2: Terragrunt-Driven Terraform Deployment to Google Cloud

**Follow `Post-Deployment Instructions for Toolkits #2 & #3` instructions AFTER deployment**

> Time to Complete: 50-60 minutes(majority of time waiting for cloud composer to finish deploying)

> Note: This follows the example directory structure provided by terragrunt with modules housed in the same git repo-[further reading](https://github.com/gruntwork-io/terragrunt-infrastructure-live-example)

> Do NOT run this in parallel with toolkit #3 as default variables will cause conflicts

### System Design

![terragrunt_deployment](/docs/terragrunt_deployment.png)

> terraform state will be split into multiple files per module

#### Terragrunt Resources

- [Keep your Terraform code DRY](https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/)
- [Relative Paths](https://community.gruntwork.io/t/relative-paths-in-terragrunt-modules/144/6)
- [Handling Dependencies](https://community.gruntwork.io/t/handling-dependencies/315/2)
- [Terraform force unlock](https://www.terraform.io/docs/commands/force-unlock.html)
- [Third Party Reasons to use terragrunt](https://transcend.io/blog/why-we-use-terragrunt)
- [Managing Terraform Secrets](https://blog.gruntwork.io/a-comprehensive-guide-to-managing-secrets-in-your-terraform-code-1d586955ace1)
- [Google Provider Documentation](https://www.terraform.io/docs/providers/google/guides/provider_reference.html#full-reference)

### Specific Use Cases

- Low cost Google Cloud dev airflow dev environment($10-$20/day)
- Test local desktop DAGs against cloud infrastructure that will have more parity with qa and prod environments
- Add more horsepower to your data pipelines
- Infrastructure as code that is DevOps friendly with terraform modules that do NOT change or duplicate, only terragrunt configs change

### How to Deploy

> Read Post-Deployment Instructions for Toolkits #2 & #3 after this deployment

- Create a service account secret to authorize the terragrunt/terraform deployment

```bash
#!/bin/bash
# create a secrets manager secret from the key
gcloud secrets create terraform-secret \
    --replication-policy="automatic" \
    --data-file=account.json

# List the secret
# example terminal output
# NAME                 CREATED              REPLICATION_POLICY  LOCATIONS
# airflow-conn-secret  2020-08-18T19:45:50  automatic           -
# terraform-secret     2020-08-12T14:34:50  automatic           -
gcloud secrets list

# verify secret contents ad hoc
gcloud secrets versions access latest --secret="terraform-secret"
```

- replace important terragrunt configs for your specific setup

```hcl
# file location
# /airflow-toolkit/terragrunt_infrastructure_live/non-prod/account.hcl

locals {
  project               = "wam-bam-258119" # replace this with your specific project
  service_account_email =<EMAIL_ADDRESS># replace this with your specific service account email
}

# file location
# /airflow-toolkit/terragrunt_infrastructure_live/terragrunt.hcl

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    project     = "${local.project}"
    location    = "${local.region}"
    credentials = "${local.credentials_file}"
    bucket      = "secure-bucket-tfstate-airflow-infra-${local.region}" # replace with something unique BEFORE `-${local.region}`
    prefix      = "${path_relative_to_include()}"
  }
}
```

```bash
#!/bin/bash
# assumes you are already in the the repo root directory
cd terragrunt_infrastructure_live/non-prod/us-central1/dev/

# export the Google Cloud project ID where the secrets live-to be used by terragrunt
# example: export PROJECT_ID="wam-bam-258119"
export PROJECT_ID=<your project id>

gcloud config set project $PROJECT_ID #TODO: add this step to the CICD pipeline rather than the get secret shell script

# this has mock outputs to emulate module dependencies with a prefix "mock-"
# OR you can run a more specific plan
# terragrunt run-all plan -out=terragrunt_plan
# --terragrunt-non-interactive flag if this is run for the first time to create the state gcs bucket without a prompt
# https://github.com/gruntwork-io/terragrunt/issues/486
terragrunt run-all plan --terragrunt-non-interactive

# this has mock outputs to emulate module dependencies
terragrunt run-all validate

# follow terminal prompt after entering below command
# do NOT interrupt this process until finished or it will corrupt terraform state
# OR you can apply a more specific plan
# terragrunt run-all apply terragrunt_plan
terragrunt run-all apply

```

### How to Destroy

> Time to Complete: 5-10 minutes

```bash
#!/bin/bash
# follow terminal prompt after entering below command
terragrunt destroy-all

# you may occasionally see terragrunt errors related to duplicate files
# run the below often to avoid those errors
cd terragrunt_infrastructure_live/

bash terragrunt_cleanup.sh
```

### Tradeoffs

#### Pros

- Explicit terraform module dependencies(through terragrunt functionality)
- Keeps your terraform code DRY
- terraform state is divided by module
- Separate config and terraform module management
- Ability to spin up multiple environments(dev, qa) with a single `terragrunt plan-all` command within the dir: `./terragrunt_infrastructure_live/non-prod/`
- Same example DAGs work as is in local desktop setup
- Minimal kubernetes skills required to start
- Access to more virtual horsepower(ex: add more worker pods through cloud composer configs)
- Dynamically authorizes infrastructure operations through secrets manager
- A DevOps engineer should only need to copy and paste the dev terragrunt configs and update inputs for other environments(qa, prod)
- VPC-native, private-IP, bastion host, ssh via identity aware proxy, and other security-based, reasonable defaults(minimal touchpoints with the public internet)

#### Cons

- You must destroy your respective dev environments every day or risk accruing costs for idle resources
- Time to complete is long
- Destroying all the infrastructure through terragrunt/terraform does NOT automatically destroy the cloud composer gcs bucket used to sync DAGs-[further reading](https://www.terraform.io/docs/providers/google/r/composer_environment.html)
- If you customize the VPC to prevent any kind of public internet access, cloud composer will not deploy properly(as it needs to reach out to pypi for python dependencies)
- Need to learn another tool: terragrunt(definitely worth it)

---

---

## Toolkit #3: Simple Terraform Deployment to Google Cloud

**Follow `Post-Deployment Instructions for Toolkits #2 & #3` instructions AFTER deployment**

> Time to Complete: 50-60 minutes(majority of time waiting for cloud composer to finish deploying)

> Note: This uses terragrunt as a thin wrapper within a single subdirectory

> Do NOT run this in parallel with toolkit #2 as default variables will cause conflicts

### System Design

![terragrunt_deployment](/docs/terragrunt_deployment.png)

> terraform state will be stored in one file

### Specific Use Cases

- Best used for quick and easy setup for a data engineer, NOT intended for hand off to a DevOps engineer
- Low cost Google Cloud dev airflow dev environment($10-$20/day)
- Test local desktop DAGs against cloud infrastructure that will have more parity with qa and prod environments
- Add more horsepower to your data pipelines

### How to Deploy

> Read Post-Deployment Instructions for Toolkits #2 & #3 after this deployment

- Create a service account secret to authorize the terraform deployment

```bash
#!/bin/bash
# create a secrets manager secret from the key
gcloud secrets create terraform-secret \
    --replication-policy="automatic" \
    --data-file=account.json

# List the secret
# example terminal output
# NAME                 CREATED              REPLICATION_POLICY  LOCATIONS
# airflow-conn-secret  2020-08-18T19:45:50  automatic           -
# terraform-secret     2020-08-12T14:34:50  automatic           -
gcloud secrets list

# verify secret contents ad hoc
gcloud secrets versions access latest --secret="terraform-secret"
```

- replace important terragrunt/terraform configs for your specific setup

```hcl
# file location
# /airflow-toolkit/terraform_simple_setup/terragrunt.hcl

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    project     = "wam-bam-258119" # replace with your GCP project id
    location    = "US"
    credentials = "service_account.json"
    bucket      = "secure-bucket-tfstate-composer" # replace with something unique
    prefix      = "dev"
  }
}
```

```hcl
# file location
# /airflow-toolkit/terraform_simple_setup/variables.tf

variable "project" {
  description = "name of your GCP project"
  type        = string
  default     = "big-dreams-please" # replace with your GCP project id
}

variable "service_account_email" {
  description = "Service account used for VMs"
  type        = string
  default     =<EMAIL_ADDRESS># replace with your service account email
}
```

- Copy and paste the `account.json` into the directory below and rename it `service_account.json`
  > Avoids the hassle of calling the terraform-secret for this simple terraform setup

```bash
#!/bin/bash

cd terraform_simple_setup/

# utilizes terragrunt as a thin wrapper utility to automatically create the gcs backend remote state bucket
terragrunt init

# preview the cloud resources you will create
# OR you can run a more specific plan
# terraform plan -out=terraform_plan
terraform plan

# validate terraform syntax and configuration
terraform validate

# follow terminal prompt after entering below command
# OR you can apply a more specific plan
# terraform apply terraform_plan
terraform apply
```

### How to Destroy

> Time to Complete: 5-10 minutes

```bash
#!/bin/bash
# follow terminal prompt after entering below command
terraform destroy
```

### Tradeoffs

#### Pros

- Explicit terraform module dependencies(through built-in terraform functionality)
- Separate config and terraform module management
- Same example DAGs work as is in local desktop setup
- Minimal kubernetes skills required to start
- Access to more virtual horsepower(ex: add more worker pods through cloud composer configs)
- Dynamically authorizes infrastructure operations through secrets manager
- A DevOps engineer should only need to copy and paste the dev terragrunt configs and update inputs for other environments(qa, prod)
- VPC-native, private-IP, bastion host, ssh via identity aware proxy, and other security-based, reasonable defaults(minimal touchpoints with the public internet)

#### Cons

- You must destroy your respective dev environments every day or risk accruing costs for idle resources
- Time to complete is long
- Destroying all the infrastructure through terragrunt/terraform does NOT automatically destroy the cloud composer gcs bucket used to sync DAGs-[further reading](https://www.terraform.io/docs/providers/google/r/composer_environment.html)
- If you customize the VPC to prevent any kind of public internet access, cloud composer will not deploy properly(as it needs to reach out to pypi for python dependencies)
- To create a similar environment(qa, prod), you would have to copy and paste the terraform modules into a separate directory and run the above steps. Terraform code is NOT DRY
- The DevOps engineer has to take on a lot more work to maintain this deployment across several environments
- terraform state is all contained within one file

---

---

## Post-Deployment Instructions for Toolkits #2 & #3

> Time to Complete: 5-10 minutes
> Only compute instances on the same VPC as Cloud Composer can access the environment programmatically
> `gcloud composer` commands will NOT work on your local desktop

- After the terragrunt/terraform deployment is successful, run the below commands in your local desktop terminal
- Add in `Compute Instance Admin(v1) and Service Account User` roles to the iap ssh service account(adjust the terraform code less) OR create a custom role with `compute.instances.setMetadata`(adjust the terraform code more)

> If you are the owner of the project, you can skip the identity aware proxy ssh step and simply ssh through the console itself

```bash
#!/bin/bash

# ssh via identity aware proxy into the bastion host(which will then run commands against cloud composer)
# update the env vars before running ssh tunnel
ACCESS_KEY_FILE="account.json"
PROJECT_ID="airflow-demo-build" # your GCP project ID
ZONE="us-central1-b" # your GCP compute engine ZONE defined in terraform/terragrunt variables, likely us-central1-a or us-central1-b
#<EMAIL_ADDRESS># Toolkit 3 Default
<EMAIL_ADDRESS># Toolkit 2 Default
KEY_FILE="iap-ssh-access-sa.json"
source utils/cloud_composer/iap_ssh_tunnel.sh

# install basic software in the bastion host
sudo apt-get install kubectl git

# Set Composer project, location, and zone
# The hard-code values are based on defaults set by terraform module variables
# Minimizes redundant flags in downstream commands
gcloud config set project airflow-demo-build # your GCP project ID
gcloud config set composer/location us-central1
gcloud config set compute/zone us-central1-b # your GCP compute engine ZONE defined in terraform/terragrunt variables, likely us-central1-a or us-central1-b

# list cloud composer DAGs
gcloud composer environments run dev-composer \
    list_dags

# capture cloud composer environment config
COMPOSER_ENVIRONMENT="dev-composer"
COMPOSER_CONFIG=$(gcloud composer environments describe ${COMPOSER_ENVIRONMENT} --format='value(config.gkeCluster)')
# COMPOSER_CONFIG ex: projects/wam-bam-258119/zones/us-central1-b/clusters/us-central1-dev-composer-de094856-gke

# capture kubernetes credentials and have kubectl commands point to this cluster
gcloud container clusters get-credentials $COMPOSER_CONFIG

# copy and paste contents of service account json file from local machine into the bastion host
cat <<EOF > account.json
<paste service account file contents>
EOF

# be very careful with naming convention for this secret or else the KubernetesPodOperator will timeout
kubectl create secret generic dbt-secret --from-file=account.json

# Create SSH key pair for secure git clones
ssh-keygen

# copy and paste contents to your git repo SSH keys section
# https://github.com/settings/keys
cat ~/.ssh/id_rsa.pub

# create the ssh key secret
kubectl create secret generic ssh-key-secret \
  --from-file=id_rsa=$HOME/.ssh/id_rsa \
  --from-file=id_rsa.pub=$HOME/.ssh/id_rsa.pub

kubectl get secrets
```

- Open a separate terminal to run the below

```bash
#!/bin/bash
# these commands work from the `airflow-toolkit/` root directory

# reauthorize the main service account to gcloud
gcloud auth activate-service-account --key-file account.json

# add secrets manager IAM policy binding to composer service account
# The hard-code values are based on defaults set by terraform module variables
PROJECT_ID="airflow-demo-build"
<EMAIL_ADDRESS># Toolkit 2 Default
#<EMAIL_ADDRESS># Toolkit 3 Default
SECRET_ID="airflow-conn-secret"

gcloud secrets add-iam-policy-binding $SECRET_ID \
    --member=$MEMBER_SERVICE_ACCOUNT_EMAIL \
    --role="roles/secretmanager.secretAccessor"

# Configure variables to interact with cloud composer
export PROJECT_DIR=$PWD

# Set Composer location
gcloud config set composer/location us-central1

COMPOSER_ENVIRONMENT="dev-composer"
COMPOSER_BUCKET=$(gcloud composer environments describe ${COMPOSER_ENVIRONMENT} --format='value(config.dagGcsPrefix)' | sed 's/\/dags//g')

# sync files in dags folder to the gcs bucket linked to cloud composer
# this may not work if you have python 3.8.5 installed on macOS
# see: https://github.com/GoogleCloudPlatform/gsutil/issues/961
gsutil -m rsync -r $PROJECT_DIR/dags $COMPOSER_BUCKET/dags
```

> Note: The airflow webserver will take 30 seconds to update the view with the updated DAGs. However, you can run DAGs as soon as you upload the new files to the gcs bucket.

- [Instructions to access the airflow webserver UI](https://cloud.google.com/composer/docs/how-to/accessing/airflow-web-interface?hl=fi#accessing_the_web_interface_via_the)
- Rerun all DAGs within cloud composer for success validation
  > Note: `bigquery_connection_check` will fail unless `add_gcp_connections` succeeds first

---

---

## Git Repo Folder Structure

| Folder                            | Purpose                                                                    |
| --------------------------------- | -------------------------------------------------------------------------- |
| .github/workflows                 | Quick terragrunt/terraform validations                                     |
| dags                              | airflow pipeline code                                                      |
| dags_archive                      | draft DAG code                                                             |
| dbt_bigquery_example              | Working and locally tested dbt code which performs BigQuery SQL transforms |
| Dockerfiles                       | Docker images to be used by Cloud Composer                                 |
| docs                              | Images and other relevant documentation                                    |
| terraform_simple_setup            | Terraform modules for a terraform-only setup                               |
| terragrunt_infrastructure_live    | Terragrunt orchestrator to run terraform operations                        |
| terragrunt_infrastructure_modules | Base terraform modules for terragrunt to consume in the `live` directory   |
| tests                             | Example DAG test cases                                                     |
| utils                             | Various utilities to automate more specific ad hoc tasks                   |

## Frequently Asked Questions(FAQ)

- Why do you use identity aware proxy for remote ssh access into the bastion host?
  - Creates a readable audit trail for ssh access patterns
- Do you have an equivalent deployment repo for AWS/Azure?
  - No, more then open to a pull request that includes this
- Why not use terratest?
  - I don't know the go programming language well enough, but plan to learn in the future ;)
- Why pytest?
  - Less verbose testing framework compared to python's built-in unittest
  - Already had battle-tested boilerplate testing code

## Resources

- [Helm Quickstart](https://helm.sh/docs/intro/quickstart/)
- [Helm Chart Official Release](https://artifacthub.io/packages/helm/airflow-helm/airflow)
- [Helm Chart Source Code](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
- [SQLite issue](https://github.com/helm/charts/issues/22477)
- [kubectl commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
- [What is a pod?](https://kubernetes.io/docs/concepts/workloads/pods/pod/)
- [Kubernetes Dashboard for Docker Desktop](https://medium.com/backbase/kubernetes-in-local-the-easy-way-f8ef2b98be68)
- [Cost effective way to scale the airflow scheduler](https://medium.com/@royzipuff/the-smarter-way-of-scaling-with-composers-airflow-scheduler-on-gke-88619238c77b)
- [Kubernetes on Docker Desktop Limitations](https://docs.docker.com/docker-for-mac/kubernetes/)
- [Installing Homebrew in GitHub Actions](https://github.community/t/installing-homebrew-on-linux/17994)

repository: buribae/airflow-toolkit

repo_name: buribae/airflow-toolkit


text:
# Download all recent tweets
 Download up to 3.200 (API limit) most recent tweets from any given username through Twitter API. Supports 240 character tweets.

## Install
Run `pipenv install`.

Then generate API tokens (instruction: https://developer.twitter.com/en/docs/basics/getting-started - this might require applying for a Twitter developer account and creating a developer app).

Create the following environment variables with you API tokens:
`CONSUMER_KEY = '[CONSUMER_KEY goes here]'`
`CONSUMER_SECRET = '[CONSUMER_SECRET goes here]'`
`ACCESS_TOKEN = '[ACCESS_TOKEN goes here]'`
`ACCESS_SECRET = '[ACCESS_SECRET goes here]'`

## Run
The username is supplied via the command line argument --screen_name. For example:
`main.py --screen_name yourscreenname`

repository: tcmetzger/Download-all-recent-tweets

gha_language: Python

repo_name: tcmetzger/Download-all-recent-tweets


text:
# Creating a new full-stack request

So you want the frontend to do something with the database?
Here are the steps you’ll need to follow.
You don’t need to do things in this order, of course - this is loosely modeled on the data flow:
1. Create the UI (if relevant) to initiate the request from the server/db.
2. Create an action that should be fired when the user uses the relevant UI
   (e.g., when they click a certain button).
   Make sure to import that action and use mapDispatchToProps to get the action onto the component’s props.
   1. This action should not actually do any ajax calls itself; that’s the job of the saga (see below).
3. Create a reducer that listens for the action.
   You’ll probably want to record somewhere in the Redux state that there’s a pending action.
   That will allow your UI to reflect the pending request - that might cause a spinner to appear,
   for example, to notify the user that their request has been initiated, but that it hasn’t fully saved yet.
   1. You might need to update the top-level structure of our Redux state. See this file: app/reducers.js
4. Set up a saga.
   1. If you’re not familiar with sagas, talk to someone on the project who does,
      and/or read some of the basic documentation.
      Then check out some of our existing saga files to see it implemented.
      Most of those are almost identical in structure to the structure recommended in the docs.
   2. Create a saga in the component’s saga file.
      It should listen for the request action you created in previous steps,
      and should initiate your AJAX request (or other asynchronous process).
      When that async function resolves, the saga will generate its own action.
      Usually you’ll have it either send a request-succeeded type of action
      (with the data from the async function as the payload), or a request-failed type of action.
      1. Add reducers to listen for the request-succeeded or request-failed actions, and update state accordingly.
   3. Now you need to make sure the saga you put together is being run by our saga middleware.
      Check out the `// Run sagas` comment in `app/store.js` to see how to do that.
5. If the AJAX request hits our server, you’ll need our server to handle the request correctly.
   First, create an express route here: `server/routes.js`.
   For now, you can leave the controller blank, because you need to create that.
   Come back after creating the controller and make sure the controller is listed in the route you create.
6. Create the appropriate controller to handle the request to the route you specified.
   That controller should probably go in our server/controllers directory.
7. If that controller needs to access the db, it should make use of one of our db models (found here: db directory).
   You’ll probably need to update one of the models there, or even create a whole new model,
   and maybe even a whole new table in our db. Have fun!

Here’s the actual flow of data:
1. The user does something in the UI.
   The react component responds by dispatching an action, indicating that there will be an async request.
2. The reducer hears that action and updates state in some way,
   probably indicating that there’s a request in progress.
3. The UI may update in some way in response to that change in redux state.
4. The saga (which is active because it was run by our saga middleware) also hears the action
   and kicks off the async function. It listens for that function to resolve (i.e., come back with a response).
5. If the async function is a call to our db, the request is heard by our express server
   Our server matches the request to the correct route, and invokes the corresponding controller.
6. The controller may make its own request to the db, or to some other async service, such as an external API.
   When the controller is ready (e.g., when it hears back from that async service), it responds to the frontend.
7. The saga (on the frontend) hears the response,
   and sends off its own action - hopefully an action indicating that the async function resolved correctly.
   The action may also include the data sent from the server.
8. The corresponding reducer listens for that action, and updates state accordingly.
9. The UI may update in response to the redux state changing.

repository: Get-In-Touch/get-in-touch

repo_name: Get-In-Touch/get-in-touch