text:
Full version with a nicer layout can be found on my blog: https://saza.re/posts/handle_exploration/

Once upon a corona quarantine day, I wanted to create a fully privileged process by becoming system. My desire was to get unlimited access to any user-mode process on the system. After stealing system's token I was granted unlimited power, or so I thought..

In this post I'll show you the journey I went on trying to get full access to a privileged process

# Humble beginnings

The goal of this prject being getting complete access to protected processes, we will start of by making a small c program which attempts to open a process with [PROCESS_ALL_ACCESS](https://docs.microsoft.com/en-us/windows/win32/procthread/process-security-and-access-rights) rights:

```c
	DWORD pid = atoi(argv[1]);

	while(1)
	{
		if ((h = OpenProcess(PROCESS_ALL_ACCESS, 0, pid)))
			break;

		printf("ACCESS DENIED:  upgrade token to system\n");
		Sleep(3000);
	}
```

This will keep looping until it manages to open a process with the specified access rights. We attempt to open handles to a variety of processes, using the following commands to give us system:

![313929cc6fea00ab97dbb65694918fc2.png](./_resources/ab65cda00dfb4dc0832b53877f02b7b0.png)
```
0: kd> !process 0 0 handle_open.exe
PROCESS ffffbf05cbc0a540
    SessionId: 1  Cid: 16d4    Peb: 6999d6c000  ParentCid: 228c
    DirBase: 1dd834002  ObjectTable: ffff948b96e97b40  HandleCount:  39.
    Image: handle_open.exe
```
```
0: kd> !process 0 0 system
PROCESS ffffbf05c6660300
    SessionId: none  Cid: 0004    Peb: 00000000  ParentCid: 0000
    DirBase: 001ad002  ObjectTable: ffff948b8ec09d00  HandleCount: 3013.
    Image: System
```
```
0: kd> ?? &((_eprocess *)0xffffbf05c6660300)->Token
struct _EX_FAST_REF * 0xffffbf05`c6660658
   +0x000 Object           : 0xffff948b`8ec0304e Void
   +0x000 RefCnt           : 0y1110
   +0x000 Value            : 0xffff948b`8ec0304e
```
```
0: kd> ?? &((_eprocess *)0xffffbf05cbc0a540)->Token
struct _EX_FAST_REF * 0xffffbf05`cbc0a898
   +0x000 Object           : 0xffff948b`96b8d064 Void
   +0x000 RefCnt           : 0y0100
   +0x000 Value            : 0xffff948b`96b8d064
```

```c
// Set the 8 bytes at the address of the token
//  field of handle_open.exe's eprocess equal to
//  the contents of the Token field of system's
//  eprocess
0: kd> eq 0xffffbf05`cbc0a898 0xffff948b`8ec0304e
```

After doing this and continuing the vm, we break out of the loop indicating that we've successfully opened a handle to lsass.exe with `PROCESS_ALL_ACCESS` rights. To confirm this we will check the GrantedAccessBits in the handle table of handle_open.exe:

First take the ObjectTable pointer of handle_open.exe's eprocess and cast it to a _handle_table

![54aa8307252846582300e622187bd694.png](./_resources/6d25edb50ee04fdf997e27738f03e57f.png)

Now dump the TableCode memory to see all the handle_table_entrys we have:
```c
0: kd> dq 0xffff948b`960b7000 l60
ffff948b`960b7000  00000000`00000000 00000000`00000000
ffff948b`960b7010  bf05cf2b`0dd0ffff 00000000`0012019f
... snip ...
ffff948b`960b7270  948b9670`5e40ffff 00000000`00020019
ffff948b`960b7280  bf05cc77`70900001 00000000`001fffff
ffff948b`960b7290  00000000`00000000 ffff948b`960b72a0
ffff948b`960b72a0  00000000`00000000 ffff948b`960b72b0
ffff948b`960b72b0  00000000`00000000 ffff948b`960b72c0
```
We can see the last added entry at `ffff948b960b7280` which, if casted to a _HANDLE_TABLE_ENTRY, we can see has the access rights(`0x1FFFFF: PROCESS_ALL_ACCESS`) we requested:
```c
0: kd> dt _handle_table_entry ffff948b`960b7280
ntdll!_HANDLE_TABLE_ENTRY
    ... snip ...
   +0x000 ObjectPointerBits : 0y10111111000001011100110001110111011100001001 (0xbf05cc77709)
   +0x008 GrantedAccessBits : 0y0000111111111111111111111 (0x1fffff)
    ... snip ...
```

Done! We have full access from our created process to the privileged process. Or do we...

### Odd one out
... This works, on my version of W10, for every process but 1. The process that doesn't play nice is `MsMpEng.exe`. If we take the same steps as before as well as removing the protection in it's eprocess we will see that, while OpenProcess succeeds, we don't get the complete access we were implicitly promised when OpenProcess succeeded. In fact we can use ReadProcessMemory on it but when we use WriteProcessMemory we get an AccessDenied error. How could it be that our handle returned from a successful call to OpenProcess including PROCESS_VM_WRITE rights, doesn't allow us to use WriteProcessMemory? Let's see what happened to our handle this time:
```
0: kd> dt _handle_table_entry ffff948b`960b7210
ntdll!_HANDLE_TABLE_ENTRY
    ... snip ...
   +0x000 ObjectPointerBits : 0y10111111000001011100110101011110011100101101 (0xbf05cd5e72d)
   +0x008 GrantedAccessBits : 0y0000111111111111111010100 (0x1fffd4)
    ... snip ...
```
Our handle isn't equal to `PROCESS_ALL_ACCESS(0x1FFFFF)` despite the OpenProcess call succeeding, this might explain why we can't use WriteProcessMemory here.

If we bitwise OR `PROCESS_ALL` with these access rights we will see what was stripped from our 0x1FFFFF to reach the 0x1FFFD4:
```c
PROCESS_CREATE_THREAD (0x0002)
PROCESS_TERMINATE (0x0001)
PROCESS_VM_WRITE (0x0020)
PROCESS_VM_OPERATION (0x0008)
```

But how could this happen with OpenProcess not giving an error? Let's find out.

# ObpCreateHandle
For this one we're gonna have to start at the source. The function responsible for creating handles is `ObpCreateHandle`. This function is incredibly complex, so I'll only go into the areas relevant for this research. If you want to read more about handle creation and other things relevant to handles I highly recommend [this blog post by Sinaei](https://rayanfam.com/topics/reversing-windows-internals-part1/).

### General overview
At the start of the function it decides between 2 major paths:
![d95767ddf8e6012ba1b93d684f9d0302.png](./_resources/9aed0c1d899e4f7bb2a465d66ead8910.png)

After picking the `ObCreateHandle/ObOpenHandle` path, we encounter the first interesting piece of code in deciding what will happen to our `OpenProcess` call, [SeAccessCheck](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-seaccesscheck):

![72cb7c6da566e33f33a868e3d8a07847.png](./_resources/70604d81a29742b3a333a0ae98e4b4ca.png)

This does a lot of privilege checks using our handle_open.exe's stolen system token. Because it's a lot to reverse, we can instead just follow along in the debugger by breaking on SeAccessCheck:

```c
//RemainingDesiredAccess before the call
kd> dd r14+0x10 l1
fffff90e`e9def860  001fffff
```
```c
//GrantedAccess after the call
kd> dd ebp-1c l1
fffff90e`e9def3a4  001fffff
```
```c
//Return value of SeAccessCheck after the call
kd> r al
al=1
```

From msdn:
>If access is allowed, SeAccessCheck returns TRUE.

Now we know that our system token is privileged enough to request the `PROCESS_ALL_ACCESS` access rights.

After some more minor tests and preperations we encounter the function `ObpCallPreOperationCallbacks` and then we write our data to the HANDLE_TABLE_ENTRY in the handle_open.exe process:

![3a61d920045732bf908b778320bc95ad.png](./_resources/52980dfa8e2846b8af877731dd1bd19d.png)

At this point the GrantedAccessBits are already 0x1FFFD4 instead of 0x1FFFFF, so something before this code and after the `SeAccessCheck` is responsible for changing our access rights despite having enough privileges.

# Callbacks
The only other area that really impacts the GrantedAccessBits variable written to the handle_table_entry is the part after `ObpCallPreOperationCallbacks`, which takes a returned argument from that call and edits our GrantedAccessBits with it:
![4402a7d0cbd8e67137ad41e475877ef5.png](./_resources/4f9c37db75e546618935a9475e136957.png)

So it seems that `ObpCallPreOperationCallbacks` is responsible for editing our DesiredAccess before they get to the handle entry.
Callbacks can be registered using [ObRegisterCallback](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-obregistercallbacks) and they trigger under predefined circumstances. In this case Windows Defender registered a callback named `WdFilter!ProcessObjectPreOperationCallback`for opening handles to processes or desktop objects:

![bec07f4dd1a0125677bd38f68f4a12b1.png](./_resources/b1d6ef940a504ca6bdb5b7c2405b2ed2.png)

### ProcessPreOperationCallback
>Before I get into the WdFilter callback I want to give credit to n4r1b. I used some of his [reversed structure definitions](https://github.com/n4r1b/WdFilter-Research/) to supplement the disassembly ahead, together with some old WdFilter I had symbols for. (I did discover some new flags and functionality in the callback which I roughly named as I saw them) Check out his WdFilter reverse engineering series [over here](https://n4r1b.netlify.app/posts/2020/01/dissecting-the-windows-defender-driver-wdfilter-part-1/).

After we check that our [OB_PRE_OPERATION_INFORMATION->ObjectType](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_ob_pre_operation_information) is PsProcessType we jump to the meat of our callback.

First we check a global flag set inside of a global WdFilter structure named MpData. This bitflag is a flag that seems to turn off, and on, the entire PreOperationCallback functionality.

![45a7f1cfb8f98b74de859bd119f0e4be.png](./_resources/0aba1d1e9ebb406c9022f539be655e08.png)

We then check if the handle that's being created isn't a Kernel handle ([OB_PRE_OPERATION_INFORMATION->KernelHandle](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_ob_pre_operation_information)) and whether the calling process is MsMpEng.exe(The user-land component of defender)

![7c0058f654769195f061589b29f6fca9.png](./_resources/39c833612ef040fa8b41fbc9f5d3582a.png)

And at last we have a seemingly useless check:

![c4eb0ae882ce71e34cd228c067445d85.png](./_resources/f3199483c4934d6cbc334ed5cc728580.png)

That's it for the checks that would allow WdFilter to ignore this function on handle-creation.

If there is no reason to skip the callback the driver will start by initialising 2 structs, one for the calling process and one for the target process.

![bf4b3e13faa822bfefe4749ea270e341.png](./_resources/783b01c06352446c88cf4be0ff44445d.png)

Somewhere inside of there it also loads our [OB_PRE_CREATE_HANDLE_INFORMATION ->DesiredAccess](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_ob_pre_create_handle_information), the AccessMask with which we called OpenProcess ([0x1ffff] PROCESS_ALL_ACCESS)

After that we make sure that our Operation == OB_OPERATION_HANDLE_CREATE
and save both processes' pids for (internal) event logging use.

![a6857f1788a8eb0289dbe5db41373fdd.png](./_resources/67d9ea40dd264704b509884258005504.png)

Because of this relatively new check not triggering in our case, we skip a bit of new functionality that looks like this (might be interesting to dive into at a later date):

![228c46d1f53b3f71548d4e9665e07959.png](./_resources/44b5776259d84404b37bf5556ed998e9.png)

Then after checking some global flags set inside the MpData structure:

![82588fcdf1c2116d8844ffbc16c74553.png](./_resources/f801d837ee5f4a4a906880e33d974a1d.png)

We get to a point where it checks 2 different sets of access rights:

![6cae502d30d13218b19d1f8839046140.png](./_resources/5e6cb55936de414c87f64add367d7d16.png)

and

![23a81eddca557d5d878ae90ebf70a193.png](./_resources/03b32681294546a18ba341309e92b16d.png)

### Process Injection path
Let's first follow the path that's responsible for handling handles with potential process injection access rights.

The main check deciding whether we should even call the functions responsible for checking if injection is allowed is this:

![c7339020af9544a03365c2da2e6a123a.png](./_resources/6e7679cc17cf4311b6107428d1c9dac4.png)

I'm not sure if I grasp this check, but if I understand correctly then the entire "code injection allowed" section is skipped IF the process hasn't had its handle opened with those flags more than twice. My best guess would be that this has to do with crss/lsass or other essential services already being expected to have opened a handle to the target process before it's possible for a normal process to create a handle to it. If we were able to trigger this check and avoid the injection allowed check, we would be able to bypass the first edit of our DesiredAccess entirely:

![a16a5bfb86db2d2f79281dfee8de2bd2.png](./_resources/70e1bc2837914c77b0965a3215407699.png)

But we would still have to deal with more checks later on (I'll get to those later).

In case the target process has 2 or more handles with these access rights open already we will be taken into a couple of functions that check more flags:

![4a965d206d1c87f45da7a2800dccb4ae.png](./_resources/c824a3389ffd463697f692d13988e755.png)

The first function, which according to the symbols of my old WdFilter is called `MpAllowCodeInjection`, will do exactly that. Check if we are allowed to inject code and keep our AccessRights.

#### MpAllowCodeInjection

We are first faced with some checks that would let us off the hook immediately:
* Is our calling process exempt because it's on the exclusion list?
* Is our calling process exempt because it's part of the friendly processes list?
* Do we have a service sid? Spawned by services manager? (not entirely sure on this)
* Are we Svchost (pretty sure)

![9772ec413d50fe817ff83ca265417bf5.png](./_resources/17f39abfcabc4138b55b57adad51c697.png)

In our case we are merely a normal process with a system token so no luck here, let's see the next checks:

![6edcfca9b942c346cc3b1d01ec70f568.png](./_resources/079760646d254fe28825dcde674014c2.png)

For the main portion of our code injection check the `CodeInjectionTargetMask` member of the target process' context and the `CodeInjectionRequestMask` member of our calling process' context are used.

The only way for this function to return FALSE and thus immediately disallow our process injection is if the following 3 conditions are met:
* CodeInjectionTargetMask of our target process != 0
* CodeInjectionRequestMask of our calling process != 0xFFFFFFFFFFFFFFFF
* CodeInjectionRequestMask & CodeInjectionTargetMask == False

In our case both Masks are 0 and so we pass this test.

#### MpAllowAccessBasedOnHipsRule
The hips check is a bit simpler, checking only 3 flags of which 2 are interesting to us:

![a1e8042c46557cde43c956338cd8f5ff.png](./_resources/20905a6e6a00462d8d59c95cdef12d9c.png)

We can infer from this that the ProcessContext+38h field contains some sort of a bitflag rules field which has at least these 2 rules:
* [0x8000]  Disallow code injection by process if set
* [0x80000] Allow code injection into process if set

In our case the calling process has a flag of 0 and our target has a flag of 0x7ff meaning that we aren't disallowed to inject code. The flag to allow code injection into the target process isn't set in this case,  but that doesn't matter because we skip that check by passing the first.

#### failed  route
If we fail to pass either of those check we immediately get hit by a DesiredAccess downgrade of the following access rights as shown before:

![a16a5bfb86db2d2f79281dfee8de2bd2.png](./_resources/70e1bc2837914c77b0965a3215407699.png)

### Second set of access rights
The second set of access rights that are checked are a little less intrusive, but might still be too invasive for normal processes to use on protected processes or other privileged processes.

These next set of checks also depend, as before with part of the code-injection checks, on hips functionality being enabled using a flag in MpData:

![f0d17208e89f5fa8b8c481234aa6e460.png](./_resources/7b08a2d0960448d7811192747f3d27d3.png)

If enabled it checks if we aren't a registered anti-malware process, or higher, by checking the _EPROCESS->Protection flag of our process. (for a more detailed explanation on Protected Processes, [this 3-part blog series](https://www.crowdstrike.com/blog/evolution-protected-processes-part-1-pass-hash-mitigations-windows-81/) by Alex Ionescu is a good place to start)

![385c7bc38b936624b21c63e13c86f5a3.png](./_resources/9a3310c7e1b349cbbebf80e670628305.png)

If we aren't a sufficiently privileged protected process it will proceed to do another HIPS rules check using the same function as before, but with 3 new flags. Again, only 2 are really relevant for this research
:
* [0x800000]  We would assume that this flag, if set, would mean something like "disallow synchronize / suspend / terminate / query-limited rights", but this isn't the case as I'll show you in a bit.
* [0x2000000] We would assume for this flag too that, if set, we would be allowed to use those AccessRights in our handle. (not the case)

If the first flag isnt set, or is set in combination with the second flag, then we will skip some altering of our handle. If we, however, do not pass the HIPS checks, our edit rights will be edited, but not in a way that makes much sense to me:

![1d842638f5ff4bf610e61d49371ccc5d.png](./_resources/996f474050d34dbe8e27b64614231a96.png)

So what I would expect when failing the check is:
```c
//Selectively remove the checked access rights
PROCESS_ALL_ACCESS &
(SYNCHRONIZE (0x00100000L)                 |
PROCESS_QUERY_LIMITED_INFORMATION (0x1000) |
PROCESS_SUSPEND_RESUME (0x0800)            |
PROCESS_TERMINATE (0x0001))
```
But what happens instead is:
```c
// Grant exactly the access rights we checked for
DesiredAccess = (SYNCHRONIZE (0x00100000L)                 |
                PROCESS_QUERY_LIMITED_INFORMATION (0x1000) |
                PROCESS_SUSPEND_RESUME (0x0800)            |
                PROCESS_TERMINATE (0x0001))
```

So if we have any of those 4 access rights defined in our DesiredAccess then, no matter what, after this check we will still have them. At worst we will lose some of the requested access rights that aren't one of those 4. If anyone knows the reason for this, let me know!

### Last check
After processing our DesiredAccess it seems to send a notification about the fact that a process was opened using `MpObSendOpenProcessBMNotification`
and then gets to the last check. Here we first check if the target process'(MsMpEng.exe) sid isnt that of the service manager

![a0f1d922b0fab73640f4d4ca8932f975.png](./_resources/f8e28278de384e51a5480998f59e415e.png)

If it is we do 2 more checks (which used to be done inside a function called `MpIsHardeningExemptByContext`):

![807d4c64bd201b2e191cdbf75ebb17c8.png](./_resources/f6018db0dc8b422395ff40f7f6a6ccdd.png)

The first is whether we are a whitelisted "friendly" process and the second is whether we have a service SID ourselves or not. If we aren't either of those then our access rights get stripped of terminate and code-injection rights, irregardless of any previous checks:

* PROCESS_CREATE_THREAD (0x0002)
* PROCESS_TERMINATE (0x0001)
* PROCESS_VM_WRITE (0x0020)
* PROCESS_VM_OPERATION (0x0008)

This is also what has happened, in the end, to our own handle which was requested with a system token.

# Conclusion
After analyzing the full callback we can take away the following points:

If any of these conditions apply then none of the other checks matter:
* Is the flag, that determines if the callback is used, set inside of the global struct? If it is then our handle's GrantedAccessBits stay as is.
* Are we MsMpEng.exe? Our handle stays the same
* Are we opening a handle from kernel-mode? Our handle stays the same
* Are we doing an Operation that isn't one of the 2 documented ones on msdn? Our handle stays intact

To keep ProcessInjection rights we definitely need to do the following:
* Not targeting a process with a service SID
    * But if we do, we better make sure we're either a whitelisted "friendly process"
    * Or have the service SID ourselves

As well as be one of these:
* Get our process on the excluded list (ProcessContext->Flags1 & 0x1)
* Be a whitelisted "friendly process"
* Have a service manager sid
* Be Svchost

OR (this can be done by simply being system as we've seen)
* Pass the CodeInjectionMask tests (how to do this might be part of future research)
* And pass the HIPS flag checks (Also not sure how these flags are calculated yet)

And don't forget:
* If you request any of the access rights in the other check and you fail that check, all access rights except for those 4 will be stripped.

So as system we can keep doing our process injection as long as we aren't targeting something(belonging to?) the service SID. If we do want to target something with such a SID we will need to become real friendly with microsoft or just use the Service SID for ourselves.

PS. You could just overwrite the GrantedPointerBits of the HANDLE_TABLE_ENTRY, but that would be no fun :)


text:
# TodoList for VSCode

VSCode extension to manage the todolist in a better way and bring todo one step closer to you directly in your dev area.
You can login with github, create boards, todolists and todos.

![Todolist Showcase](https://media.giphy.com/media/jaiqwN3Ny33U302gt6/giphy.gif)

## Features

Manage your tasks without leaving VSCode.
Create Boards, todolists, todos and drag and drop todos as you complete the tasks.

## Running Locally

1. Clone the repo.
2. Navigate to `api` folder, run `yarn` to install the dependancies.
3. Run `yarn watch` to put server in watch mode and in another terminal window run `yarn dev` to start the server.
4. Navigate to `extension` folder and run `yarn` to install the dependancies.
5. Run `yarn watch` to watch the changes in `extension` files.
6. Press `F5` key and select `VSCode extension development` from the dropdown, if that does not appear in your dropdown,
   open the `extension.ts` file and then `F5` again.

This will open up another VSCode window with our extension installed in it.

---

## Following extension guidelines

Ensure that you've read through the extensions guidelines and follow the best practices for creating your extension.

- [Extension Guidelines](https://code.visualstudio.com/api/references/extension-guidelines)

**Enjoy!**

branch_name: refs/heads/main

extension: md

license_type: permissive


text:
## Overview
The regression repository contains packages of blueprints that can be imported into .DO and used for regression tests.
A dummy router driver was created to simulate the behavior of a network device.
The infra package contains infrastructure resources used by the orchestration e.g FTP resource,vCenter...
Other packages contain different use cases for the orchestration and there is a description in blueprint's description

### Dummy router
This drivers is used for simulating a network device. On top of the standard functionality offered by a networking device driver, the following methods were added to influence on the expected behavior of the driver:
-	Set health status
    Sets the returned health status
    -	Status - The health status passed/ failed
-   Set firmware data
    Sets firmware version and success status
    -	Version – Version the driver will return in get version
    -	Will load fail? - Will load firmware fail true/false
-	Set saved file name
    Sets the name of the saved file
    -	Filename - The name of the saved file.
-	Get version
    Return firmware version

### FTP resource
In the topologies offered for the regression tests, we are using Quali’s FTP server. A resource named ‘FTP’ was created and it’s holding the ip address, username and password to Quali’s FTP.
The configuration and images files are located under folder Cloudshell\configs

repository: QualiSystemsLab/cloudshell-orch-sandbox

branch_name: refs/heads/develop

gha_language: Python

repo_name: QualiSystemsLab/cloudshell-orch-sandbox


text:
# Gotchas!

Mocking objects in PHP has its limitations and gotchas. Some functionality can't
be mocked or can't be mocked YET! If you locate such a circumstance, please please
(pretty please with sugar on top) create a new issue on GitHub so it can be
documented and resolved where possible. Here is a list to note:

1. Classes containing public `__wakeup()` methods can be mocked but the mocked
   `__wakeup()` method will perform no actions and cannot have expectations
   set for it. This is necessary since Mockery must serialize and unserialize
   objects to avoid some `__construct()` insanity and attempting to mock a
   `__wakeup()` method as normal leads to a BadMethodCallException been thrown.

2. Classes using non-real methods, i.e. where a method call triggers a `__call()`
   method, will throw an exception that the non-real method does not exist
   unless you first define at least one expectation (a simple `shouldReceive()`
   call would suffice). This is necessary since there is no other way for
   Mockery to be aware of the method name.

3. Mockery has two scenarios where real classes are replaced: Instance mocks
   and alias mocks. Both will generate PHP fatal errors if the real class is
   loaded, usually via a require or include statement. Only use these two mock
   types where autoloading is in place and where classes are not explicitly
   loaded on a per-file basis using `require()`, `require_once()`, etc.

4. Internal PHP classes are not entirely capable of being fully analysed using
   `Reflection`. For example, `Reflection` cannot reveal details of expected
   parameters to the methods of such internal classes. As a result, there will
   be problems where a method parameter is defined to accept a value by
   reference (Mockery cannot detect this condition and will assume a pass by
   value on scalars and arrays). If references as internal class method
   parameters are needed, you should use the
   `\Mockery\Configuration::setInternalClassMethodParamMap()` method.

The gotchas noted above are largely down to PHP's architecture and are assumed
to be unavoidable. But - if you figure out a solution (or a better one than what
may exist), let us know!

**[← Previous](21-PHP-MAGIC-METHODS.md) | [Contents](../README.md#documentation) | [Next →](23-QUICK-EXAMPLES.md)**

branch_name: refs/heads/master

gha_language: PHP


text:
# Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning

Original Repository Located Here: https://github.com/LittleYUYU/Interactive-Semantic-Parsing.git

**Update 10/31/2019:** As a follow-up work, we developed a more general interactive semantic parsing framework called *"Model-based Interactive Semantic Parser (MISP)"*. The work has been accepted to EMNLP'19. Please don't hesitate to check out our [paper](https://arxiv.org/pdf/1910.05389.pdf) and [code](https://github.com/sunlab-osu/MISP)!

## 1. Introduction
This repository contains source code and dataset for paper "[Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning](https://arxiv.org/pdf/1808.06740.pdf)" (AAAI'19). Please refer to Table 1 in the paper for an example.

## 2. Dataset
The processed dataset can be found:
- [Full data](data/lam/data.tar.gz) (compressed) for reconstructing the model.
- [Toy data](data/lam/toy_data_with_noisy_user_ans.pkl) that contains a subset of the full training data, for quick model test.

Data format: [Python Pickle](https://docs.python.org/2/library/pickle.html) files. Please open with `pickle.load(open(filename))`.

Data source: [Training set from (Ur et al., CHI'16)](https://www.blaseur.com/papers/chi16-ifttt.pdf) and [Test set from (Quirk et al., ACL'15)](https://www.microsoft.com/en-us/research/project/language-to-code/).

## 3. Code
All source code is in `cd code/Hierarchical-SP`.

Requirements:
- Python 2.7
- Tensorflow >= 1.4.0

### 3.1 Agent training/testing
To train the HRL agent:
```
python run.py --train --training_stage=0
```

To train the HRL_fixedOrder agent:
```
python run.py --train --training_state=1
```

- For testing them on the test set, replace `--train` with `--test`.
- To quick test on the toy dataset, append `--toy_data`.

To interactively test the four agents {HRL, HRL_fixedOrder, LAM_sup, LAM_rule}:
```
python interactive_test.py --level='VI-3' --user-name=yourname
```

### 3.2 User simulator
Please refer to the paper appendix for more details.
The scripts for [PPDB paraphrasing](code/Hierarchical-SP/ppdb.py) and [collecting from user data or official function descriptions](code/Hierarchical-SP/user_simulator_gen.py) can be found.

## 4. Citation
Please kindly cite the following paper if you use the code or the dataset in this repo:
```
@inproceedings{yao2019interactive,
  title={Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning},
  author={Yao, Ziyu and Li, Xiujun and Gao, Jianfeng and Sadler, Brian and Sun, Huan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={33},
  pages={2547--2554},
  year={2019}
}
```

repository: sunlab-osu/Interactive-Semantic-Parsing

repo_name: sunlab-osu/Interactive-Semantic-Parsing


text:
---
id: logging
title: Logging
---

## Installation

Logging are available as a separate project
`"tf.tofu" %% "tofu-logging" % tofu-version`

## Motivation

### Preface

Almost every application nowadays does logging, whether it is logging of business events, history of some state of
application or even debug record sequences. We log to files, distributed systems, standard outputs of OSes, and we got
used to it so much it seems simple to us. Many existing libraries provide many ways of logging from your application's
code, either automatic or manual.

### What's wrong (and how to fix that)?

First, logging is, in fact, a side effect. Whether we log anything to standard output or a file — it's an interaction
with the outside world ('outside' relative to our program). Thus, logging as a process can be represented as an effect
and operations - as an algebra.
Second, logging itself seems easy, but stuff around it - not so much. MDC, contexts, structured logging - all these
things are very helpful on a day-to-day basis, allowing us to read and write our logs, making them clean, comprehensible
and useful for many purposes. But there are few simple ways to provide all this functionality without bleeding and
screaming from seeing over-engineered solutions.
Tofu offers an easy and understandable abstraction over logging (it's even named like this!) but keeping in mind that
logging is an effect that can be composed.
It looks very similar to interfaces we all are used to (from SLF4J and other logging libraries and facades), introducing
some new concepts:

```scala:reset
import tofu.logging.LoggedValue

trait Logging[F[_]] {
  def trace(message: String, values: LoggedValue*): F[Unit]

  def debug(message: String, values: LoggedValue*): F[Unit]

  def info(message: String, values: LoggedValue*): F[Unit]

  def warn(message: String, values: LoggedValue*): F[Unit]

  def error(message: String, values: LoggedValue*): F[Unit]
}

```

Basically, that's the main user API you can interact with throughout your code. It resembles well-known solutions,
keeping in mind that logging is an effect, though introducing the concept of `LoggedValue`.

## Representation

To represent a value in logs we use a concept of `Loggable` (it's a typeclass). It describes how a value of some type
can be logged, both as a string representation in log message and as a component of structured logging. Given an
instance of `Loggable` for a type, a value of the type can be converted into the final internal representation
called `LoggedValue` and thus logged in a way that you provided.
There are multiple predefined ways to create an instance of `Loggable`, many of them can be found
in `tofu.logging.Loggable` object:

* `Loggable.empty` for no-op logging of value
* `Loggable.show` for using `cats.Show` instance as string representaion
* `Loggable.either` for logging either of `A` and `B`
* provided instances for all primitive types, as well as stdlib's collections and collections from Cats
* java.time.* instances

Of course, you can describe your `Loggable` instance yourself by extending existing traits that pre-implement some
functionality:

* `DictLoggable` for multi-field objects
* `ToStringLoggable` for using `.toString` for logging
* `HideLoggable` to exclude value from logging

### Loggable example

```scala:reset
import tofu.logging._
import cats.syntax.semigroup._

case class User(name: String, surname: String)

implicit val userLoggable = new DictLoggable[User] {
  override def fields[I, V, R, S](a: User, i: I)(implicit r: LogRenderer[I, V, R, S]): R = {
    r.addString("name", a.name, i) |+| r.addString("surname", a.surname, i)
  }

  override def logShow(a: User): String = s"name = ${a.name}, surname = ${a.surname}"
}
```

Let's take a look at this example.
First, we define a loggable for our `User` class as a `DictLoggable` which means that we want to log it as a multi-field
object with structure.
Second, we define two methods that describe how `User` should be logged:

* `fields`, that represents `User` as a structure, containing two fields with their respective names and values
* `logShow`, that represents `User` as a string in a log message

## Loggable derivation
a
Tofu has integration with [derevo](https://github.com/tofu-tf/derevo) library. It allows you to easily generate
instances of `Loggable[YourClass]` for case classes or ADTs:

```scala:reset
import tofu.logging.derivation.loggable
import derevo.derive

@derive(loggable)
case class Query(value: String, size: Int, isTagged: Tagged, tag: Tag)

@derive(loggable)
case class Tagged(b: Option[Boolean])

@derive(loggable)
sealed trait Tag

@derive(loggable)
case class DatabaseTag(db: String) extends Tag

@derive(loggable)
case object InMemoryTag extends Tag
```

so when logged

### Configuring Loggable generation

Tofu has several annotations to configure generation of `Loggable`

* `@hidden`
  ```scala:reset
  import tofu.logging.derivation.{loggable, hidden}
  import derevo.derive

  @derive(loggable)
  case class User(name: String, @hidden() password: String)

  ```
  `User("tofu", "pass")` will be logged as `User{name=tofu}`
* `@masked`
    ```scala:reset
  import tofu.logging.derivation.{loggable, masked, MaskMode}
  import derevo.derive

  @derive(loggable)
  case class Card(@masked(MaskMode.Erase) expirationDate: String, @masked(MaskMode.ForLength(4, 12)) cardNumber: String, @masked() owner: String)

  ```
  `Card(737, "1244345632322311", "TOFU PETROV")` will be logged as `Card{expirationDate=...,cardNumber=1244########2311,owner=**** ******}`
* `@unembed`
   ```scala:reset
  import tofu.logging.derivation.{loggable, unembed}
  import derevo.derive

  @derive(loggable)
  case class Scientist(@unembed owner: Person, age: Option[Int])

  @derive(loggable)
  case class Person(name: String)
  ```
  `Room(Person("Karl Sagan"), None)` will be logged as `Room{name=Karl Sagan, age=None}`

## Creation

Creating an instance of `Logging` in Tofu is an effect itself. You can use helper class `Logs` that can produce values
of type `I[Logging[F]]` (where `I` and `F` are intended to be two different effects, former is a `I`nitialization
effect (`Task` or `IO`)  and latter is some ReaderT-ish effect or just main program effect type, but they can be the
same though) and define a behaviour of your `Logging` instances. An instance of `Logs` is supposed to be shared by
different parts of user code. It's not a restriction though.

Tofu logs are built on top of existing Java logging infrastructure hence each logger for some class needs to have
distinctive name, as well as whole logging system has to be configured with a `logback.xml` (default configuration does
exist though).

`Logs` methods close the first requirement in different ways:

### Plain `Logging`

* by asking that name from user:
  For example here we create named instance and pass it explicitly.

```scala:reset
import tofu.logging._
import tofu.syntax.logging._
import tofu.syntax.monadic._
import cats._

class MyService[F[_] : Monad](logger: Logging[F]) {
  def makeFoo(arg: String): F[Int] =
    logger.info("Doing simplest thing").as(arg.length) <* logger.debug("Yep, counted length")
}

object MyService {
  def makeNamed[I[_] : Monad, F[_] : Monad](logs: Logs[I, F]): I[MyService[F]] =
    logs.byName("my-service-log").map(new MyService[F](_))
}

```

* by generating that name from type-parameter, which is a class intended to use logging with:

```scala:reset
def makeNamed[I[_] : Monad, F[_] : Monad](logs: Logs[I, F]): I[MyService[F]] =
  logs.byName("my-service-log").map(new MyService[F](_))
```

Logging instance created by this method for class `MyService[IO]` in package `com.bar.foo.pckg` will be
named `com.bar.foo.pckg.MyService`

### Typesafe `ServiceLogging`

Creating plain `Logging[F]` and using it around tends to be error-prune: one can easily pass logger created for
service `FooService` instead of `BarService`. Tofu has a solution — newtype `ServiceLogging[F, Service]`
and `LoggingCompanion` which contains simple type alias `Log[F[_]] = ServiceLogging[F, Service]`.

```scala:reset
import tofu.logging._
import tofu.syntax.logging._
import tofu.syntax.monadic._
import cats._

class FooService[F[_] : FooService.Log : Monad] {
  def makeFoo(arg: Int): F[String] =
    info"Doing yet another simplest thing".as(arg.toString) <* debug"Yep, called toString"
}

object FooService extends LoggingCompanion[FooService] {
  def make[I[_] : Functor, F[_] : Monad](logs: Logs[I, F]): I[FooService[F]] =
    logs.of[FooService].map(implicit l => new FooService[F])
}

```

Note that using it like `logs.named["foo"]` or `logs.byName("foo")` is going to fail compilation.

This is also the intended way to create `Logging[F]` or `ServiceLogging[F, Service]` instance for your classes.

### Ways to create Logs

There are multiple ways to create `Logs` instance, among them:

* `Logs.sync` that requires a `cats.effect.Sync` instance for delaying side-effects
* `Logs.withContext` that requires a `cats.effect.Sync` instance for delaying side-effects and a proof that `F` has a
  context to log
* `Logs.const` that wraps single `Logging` thus always returning the same instance
* `Logs.empty` that will produce a no-op `Logging` instances
* `Logs.combine` that combines two instances of `Logs`.
  `Logging` instance that is created this way will be a combination of two underlying instances, produced by both `Logs`
  . This means that both `Logging` implementations will be called in sequence

```scala:reset
import tofu.logging._

// for simplification, you can use whatever F you like
type F[A] = cats.effect.IO[A]

class MyService
class MyTaglessService[F[_]]
// defines simple logging behaviour without any notion of contexts
val syncLogs = Logs.sync[F, F]

// describing creation of logging instances and logging itself. When being run, this will log a message
val effect: F[Unit] = for {
  loggingBN <- syncLogs.byName("my-logger") // you can create instances by name
  loggingFS <- syncLogs.forService[MyService] // you can create instances by type of your non-tagless-final services
  loggingFS <- syncLogs.of[MyTaglessService] // you can create instances by type of your tagless-final services
  _ <- loggingBN.info("This is an INFO message")
  _ <- loggingFS.warn("This is a WARN message")
} yield ()
```

## Context logging

It is possible to log the context of your computations along with actual log message. Often it comes along with
structured logging. To illustrate it consider this pseudocode:

```
structuredLogger.info("I am a message", context = Map("transaction_id" -> 2342234, "request_name" -> "req"))
//it logs
{ "level": "INFO", "message": "I am a message", "transaction_id": 2342234, "request_name": "req" }
```

That's neat, right? But it is verbose and clumsy to use, so tofu provides an abstraction for it.

If you have an instance of `tofu.Context` for your effect type and a `Loggable` for your context, you can have it logged
automagically. That is natural — describe how we can get a context from a computation `F` and a way to represent that
context in log, and we will ensure it will be logged anytime you use `Logging` operations. You can log that context to
string as well as use it as a part of structured logging.

### Layouts

Tofu is built upon [Logback](http://logback.qos.ch/) so it needs a custom `logback.xml` file with contexual logging
support. Tofu uses mechanism called markers to store context in logs, so it won't work with existing Layouts e.g.
with [Logstash-encoder](https://github.com/logstash/logstash-logback-encoder).

Luckily for us, tofu has two special Layouts:

* [ELKLayout](https://github.com/tofu-tf/tofu/blob/master/logging/layout/src/main/scala/tofu/logging/ELKLayout.scala)
  that outputs structured logs in JSON format. Example appender looks like that:

```xml

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="tofu.logging.ELKLayout"/>
    </encoder>
</appender>
  ```

* [ConsoleContextLayout](https://github.com/tofu-tf/tofu/blob/master/logging/layout/src/main/scala/tofu/logging/ConsoleContextLayout.scala)
  that outputs simple text logs. Example appender looks like that:

```xml

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="tofu.logging.logback.ConsoleContextLayout">
            <pattern>%d{HH:mm:ss} %-5level %logger{36} - %msg%n [%mdc]%n</pattern>
        </layout>
    </encoder>
</appender>
```

### Example

```scala:reset
import tofu.WithRun
import tofu.logging.derivation.loggable._
import tofu.logging.{LoggableContext, Logging, LoggingCompanion, Logs}
import cats.{Functor, Monad}
import tofu.syntax.console._
import tofu.syntax.context._
import tofu.syntax.logging._
import tofu.syntax.monadic._
import tofu.zioInstances.implicits._
import zio._
import zio.interop.catz._

object Example extends zio.App {

  class BusinessService[F[_] : BusinessService.Log : Monad] {
    private def logic(arg: String): String = arg.repeat(3).replace("a", "b")

    def runLogic(arg: String): F[String] =
      info"Doing some business logic" as logic(arg)
  }

  object BusinessService extends LoggingCompanion[BusinessService] {

    def make[I[_] : Functor, F[_] : Monad](logs: Logs[I, F]): I[BusinessService[F]] =
      logs
        .of[BusinessService]
        .map(implicit l => new BusinessService[F])
  }

  case class RequestContext(id: Long, method: String)

  class Server[I[_] : Monad : Logging, F[_] : WithRun[*[_], I, RequestContext]](
                                                                                business: BusinessService[F]
                                                                               ) {

    def handleRequest(request: Server.Request): I[Server.Response] = for {
      _ <- info"Got request $request"
      context = RequestContext(request.id, request.method)
      logicComputation = business.runLogic(request.body)

      result <- runContext[F](logicComputation)(context)

      response = Server.Response(result)
      _ <- info"Got response $response"
    } yield response

  }

  object Server {

    case class Request(id: Long, body: String, method: String)

    case class Response(answer: String)

  }

  type Contextual[A] = RIO[RequestContext, A]

  def run(args: List[String]): URIO[ZEnv, ExitCode] = {
    implicit val loggableContext: LoggableContext[Contextual] =
      LoggableContext.of[Contextual].instance[RequestContext]
    val contextLogs: Logs[Task, Contextual] = Logs.withContext[Task, Contextual]
    val mainLogs: Logs[Task, Task] = Logs.sync[Task, Task]

    for {
      implicit0(mainLogging: Logging[Task]) <- mainLogs.byName("main")
      bs <- BusinessService.make(contextLogs)
      server = new Server[Task, Contextual](bs)
      userRequest <- readRequest
      response <- server.handleRequest(userRequest)
      _ <- putStrLn[Task](s"Got response $response")
    } yield ()

  }.catchAll(_ => URIO.unit).as(zio.ExitCode.success)

  def readRequest: Task[Server.Request] =
    (
      readStrLn[Task]
        .map(_.toLongOption)
        .someOrFail(new Exception("Please provide numeric id")),
      readStrLn[Task],
      readStrLn[Task]
      ).mapN(Server.Request.apply)

}

```

#### Business Logic

We got `BusinessService` as our business logic. It does some stuff but most importantly it does not know it has any kind
of special logging. It just logs some stuff and all it needs for it is `Logging` instance.

As you can see it is created with the same factory-pattern as discussed before.

#### Server

Next we have a `Server` of some kind that should get some request, run some business logic on it and then return
response. Also, we want to have all the logs that business logic produces to have special RequestContext in it. To
achieve this we run the logic in special contextual effect type `F` which then can be evaluated into our main effect `I`
.
(More information on this can be found in scaladoc for `tofu.WithRun` etc.)
Moreover `Server` itself logs some messages and requires its own instance of `Logging`.

#### Loggable derivation

As one can see in [Representation part](#representation) for any of those logged values (like `$request`)  `Loggable`
instance should be in scope. In the example they are generated by macro, we just need to import it:

```scala
 import tofu.logging.derivation.loggable._
 ```

#### Running

It is a simple ZIO-app that reads user input from console and passes it to server. So when running possible output will
look like this, if ran with `ConsoleContextLayout` in `logback.xml`:

```bash
sbt>run
10102
foo bar body
GET
23:20:14 INFO  main - Got request Request{id=10102,body=foo bar body,method=GET}
23:20:14 INFO  c.t.c.n.c.Example$BusinessService - Doing some business logic
 [method=GET, id=10102]
23:20:14 INFO  main - Got response Response{answer=foo bbr bodyfoo bbr bodyfoo bbr body}
Got response Response(foo bbr bodyfoo bbr bodyfoo bbr body)
```

## Syntax extensions

It's much more convenient to use pre-defined syntax extensions for logging operations since they do all heavy lifting
for you:

```scala:reset
import cats.effect._
import cats.syntax.functor._
import cats.{Monad, Show, Functor}
import tofu.logging._
import tofu.syntax.logging._

case class User(name: String, surname: String)
object User {
  implicit val userShow: Show[User]         = user => s"name=${user.name}, surname=${user.surname}"
  implicit val userInstance: Loggable[User] = Loggable.show
}

val user: User = User("name", "surname")

class MyService[F[_]: Logging: Monad] {
  def doLogging: F[Unit] =
    for {
      _ <- info"This is a syntax demo: $user"
    } yield ()
}

object MyService {
  def apply[I[_]: Functor, F[_]: Monad](implicit logs: Logs[I, F]): I[MyService[F]] = {
    logs.forService[MyService[F]].map(implicit l => new MyService[F])
  }
}

// An instance of `Logs` is supposed to be shared between mupltiple classes (or a single one for a whole application)
// Note that initializing effect of your application may differ from logging effect.
implicit val logs: Logs[IO, IO] = Logs.sync[IO, IO]

val io: IO[Unit] = for {
  service <- MyService[IO, IO]
  _       <- service.doLogging
} yield ()

io.unsafeRunSync()
// 13:56:26.918 [main] INFO tofu.logging.Test$MyService - This is a syntax demo: name=name, surname=surname
```

Using a syntax we achieve a few important goals.
First, it will log a message containing a string representation of your value of type `User` as you defined it
in `Loggable` instance.
Second, it will create a `LoggedValue` from your `Loggable` description and pass it to underlying logger and
your `LogRendered`, so your structured logging will work as expected.

## Integration with logs4cats

There is a library for effectful logging named [log4cats](https://github.com/typelevel/log4cats) which shares
the goal of representing logging as an effect and providing the ability to log context values. It is used by some of the
open-source libraries which may require you to pass an instance of `org.typelevel.log4cats.Logger` to it in order to
log something. Module `tofu-logging-log4cats` contains a helper that can create an instance of log4cats `Logger` from
our existing
`tofu.logging.Logging` one. If our `Logging` instance is contextual, it will continue leveraging that despite being
wrapped in log4cats `Logger`. Moveover, it will log context values that are passed to log4cats `Logger` API
as `Map[String, String]`.
This means that not only you will not lose your context, but additional values that are passed to `Logger` will also be
logged.
Let's take a look at the example:

```scala:reset
import org.typelevel.log4cats.{Logger, StructuredLogger}
import tofu.env.Env
import tofu.logging._

// Suppose we have some external APIs that leverage log4cats's Logger and StructuredLogger
def logWithLog4Cats[F[_]](implicit logger: Logger[F]): F[Unit] =
    logger.info("Hello Tofu!")

def logWithLog4CatsContextual[F[_]](implicit logger: StructuredLogger[F]): F[Unit] =
    logger.info(Map("field" -> "value"))("Hello contextual Tofu!")

// And we have our own context that can be logged as a structure
case class Context(start: Long)
object Context {
  implicit val loggable: Loggable[Context] = new DictLoggable[Context] {
    override def fields[I, V, R, S](a: Context, i: I)(implicit r: LogRenderer[I, V, R, S]): R =
      r.addInt("start", a.start.toInt, i)
    override def logShow(a: Context): String                                                  =
      a.toString
  }
}

// This context will be embedded in our Env monad and be logged at each logging call
type AppEnv[A] = Env[Context, A]
implicit val loggableContext: LoggableContext[AppEnv] = LoggableContext.of[AppEnv].instance
val logs: Logs[AppEnv, AppEnv]                        = Logs.withContext[AppEnv, AppEnv]

// first we need to import converter function
import tofu.logging.log4cats._

// then we can create an instance of Logger, passing an instance of tofu.logging.Logging implicitly
val env1: AppEnv[Unit] =
  logs.byName("Main").flatMap { implicit logging =>
    implicit val log4catsLogger: StructuredLogger[AppEnv] = toLog4CatsLogger
    logWithLog4Cats *> logWithLog4CatsContextual
  }

// or passing it explicitly in case you have more than one Logging instance in scope
val env2: AppEnv[Unit] =
  logs.byName("Main").flatMap { logging =>
    implicit val log4catsLogger: StructuredLogger[AppEnv] = toLog4CatsLogger(logging)
    logWithLog4Cats *> logWithLog4CatsContextual
  }

// you can also create an instance of Logger for each call, but this is not advised since it will make unnecessary allocations
val env3: AppEnv[Unit] =
  logs.byName("Main").flatMap(implicit logging => logWithLog4Cats *> logWithLog4CatsContextual)

// we can then import Scheduler and run our Env with some context
import monix.execution.Scheduler.Implicits.global

(for {
  _ <- env1.run(Context(123))
  _ <- env2.run(Context(456))
  _ <- env3.run(Context(789))
} yield ()).runSyncUnsafe(Duration.Inf)

// If we have a configured layout for structured logging, supporting context values (i.e. tofu.logging.ElkLayout)
// we will see JSON logs that contain not only values from our Context, but also values that were passed to log4cats Logger by the external API
// {"@timestamp":"2020-10-31T14:11:31.059Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello Tofu!","start":123}
// {"@timestamp":"2020-10-31T14:11:31.167Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello contextual Tofu!","start":123,"field":"value"}
// {"@timestamp":"2020-10-31T14:11:31.168Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello Tofu!","start":456}
// {"@timestamp":"2020-10-31T14:11:31.168Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello contextual Tofu!","start":456,"field":"value"}
// {"@timestamp":"2020-10-31T14:11:31.169Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello Tofu!","start":789}
// {"@timestamp":"2020-10-31T14:11:31.169Z","loggerName":"Main","threadName":"main","level":"INFO","message":"Hello contextual Tofu!","start":789,"field":"value"}
```

repository: catostrophe/tofu

gha_language: Scala

repo_name: catostrophe/tofu


text:
# Lambo_Unity2D
Game Platformer using unity2D

![main](https://user-images.githubusercontent.com/43342329/70964596-069cc080-20bf-11ea-8c0c-4018570ef116.png)

# Screenshoot
![ss1](https://user-images.githubusercontent.com/43342329/71822611-da75cd80-30c7-11ea-8ef4-deba915c6833.PNG)

![ss2](https://user-images.githubusercontent.com/43342329/71822613-db0e6400-30c7-11ea-9d8d-099f2d267ee3.PNG)

![ss3](https://user-images.githubusercontent.com/43342329/71822614-db0e6400-30c7-11ea-83c9-bf2337ce2085.PNG)

# How To Play
To play it use the right, left, top arrow
and Space to shoot

# Our Mission
In this game you have to fight the Buto and his commander who has kidnapped Jack's friend Lambo

# Credits

- Graphics Sprite [Free Platform Game Assets](https://craftpix.net/freebies/)
- Tutorial this game visit [this](https://www.beelancah.com) and [this](https://www.youtube.com/user/Brackeys)
- Game Engine [Unity](https://unity3d.com/)
- Thanks to all of the game development community for their awesome help.

# Software Requirement
- Any Unity 2018.4 or higher version
- Visual Studio Code (Recommended) or any text editor

# Contributor
- Ardian Ferdy Firmansyah [A11.2017.10363](http://mahasiswa.dinus.ac.id/images/foto/A/A11/2017/A11.2017.10363.jpg)
- Alif Ananda Wijapena P  [A11.2017.10351](http://mahasiswa.dinus.ac.id/images/foto/A/A11/2017/A11.2017.10351.jpg)
- Aditya Rizki Saputra    [A11.2017.10355](http://mahasiswa.dinus.ac.id/images/foto/A/A11/2017/A11.2017.10355.jpg)
- Andin Fairuz Khosyi     [A11.2017.10370](http://mahasiswa.dinus.ac.id/images/foto/A/A11/2017/A11.2017.10370.jpg)
- Lestari Khoirunnisa     [A11.2017.10415](http://mahasiswa.dinus.ac.id/images/foto/A/A11/2017/A11.2017.10415.jpg)

license: MIT

gha_language: ShaderLab


text:
---
date: 2019-11-27T10:44:48+0100
layout: rafaga
rafagas:
- desc: OCHA PDF report about the growing and worrying food security situation in
    Southern Africa, especially with the ramping droughts
  keyw: food security
  link: https://reliefweb.int/sites/reliefweb.int/files/resources/ROSEA_20191113_SA_Humanitarian_Snapshot_Nov2019.pdf
  microlink:
    logo: https://logo.clearbit.com/reliefweb.int
    title: ROSEA_20191113_SA_Humanitarian_Snapshot_Nov2019.pdf
- desc: The obsessive maps of Paula Scher, hand-drawn, and including topics like demographics,
    weather, housing prices, etc
  keyw: scher
  link: https://www.curbed.com/2016/2/22/11094204/paula-scher-maps-us-new
  microlink:
    desc: The artist’s latest work scrutinizes median home prices, extreme weather,
      interstate highways, and more
    image: https://cdn.vox-cdn.com/thumbor/2qqfS2e9uPTZwBQxfzcf_aPM5f4=/0x315:4000x2409/fit-in/1200x630/cdn.vox-cdn.com/uploads/chorus_asset/file/13082697/PS_Maps_2015_U.S._Counties_and_Zip_Codes_<IP_ADDRESS>4480519.jpg
    logo: https://cdn.vox-cdn.com/uploads/chorus_asset/file/13668559/google_amp.0.png
    title: Gorgeous Maps of the U.S., Painted in Obsessive Detail
- desc: Groundsman is an Android application allowing you to capture GPS traces and
    hand-drawn geometries directly from the field
  keyw: mobile
  link: https://play.google.com/store/apps/details?id=com.Groundsman.GeoApp&hl=en_US
  microlink:
    desc: 'Features:

      - Data Point, Line and Polygon Creation

      - Location Logging

      - Data Import/Export

      - Map Drawing

      - Offline Functionality'
    image: https://lh3.googleusercontent.com/4Zfr2kLyqk88SVouF-MzoyVndUt8MqADGcU4RLUWrid8tYkAQvYZqF7KU5mBruw2RuY
    logo: https://logo.clearbit.com/google.com
    title: Groundsman - Apps on Google Play
- desc: '@Matarocat city hall reuses @ICGCGeostarters WMS comparator, Open Source,
    and available at Github'
  keyw: compare
  link: https://github.com/gencat/ICGC-comparador-gificador
  microlink:
    desc: VISOR DE L’EVOLUCIÓ DEL TERRITORI. Contribute to gencat/ICGC-comparador-gificador
      development by creating an account on GitHub.
    image: https://avatars1.githubusercontent.com/u/10073721?s=400&v=4
    logo: https://logo.clearbit.com/github.com
    title: gencat/ICGC-comparador-gificador
rid: 1217
---

repository: geoinquiets/rafagas

branch_name: refs/heads/gh-pages

gha_language: SCSS

gha_license_id: NOASSERTION

repo_name: geoinquiets/rafagas


text:
# Setup Facebook authentication #

## Prerequisites ##

Install [Secret Manager tool](https://docs.asp.net/en/latest/security/app-secrets.html#secret-manager) to be able to store Facebook credentials.

## Register Facebook app ##

- Navigate to [https://developers.facebook.com/apps](https://developers.facebook.com/apps) and add new Facebook app
- Open the `Settings` tab and click on `Add new platform`
- Select `Website` as the platform
- In `Site URL` box put `https://localhost:44301/`

## Add Facebook app credentials ##

- Open shell
- Navigate to `./src/GovITHub.Auth.Identity/`
- Execute the following commands:
```
dotnet user-secrets set Authentication:Facebook:AppId <app-Id>
dotnet user-secrets set Authentication:Facebook:AppSecret <app-secret>
```

repository: namtzigla/auth-sso

repo_name: namtzigla/auth-sso


text:
# Advices on Modifying the Project

Generally, if you make architectural modifications to training software, please remember to change the HLS parameters accordingly.

## For Each New HLS Run

For reasons unknown, in each HLS run, the HLS tool will randomise the order of CONV/FC layers, meaning that LUTARRAY_1.v is almost never going to be the first layer.
Therefore, after the HLS run, we need to always look at the synthesised varilog and keep track of the order of layers.
This is done by looking at the layer id inside the generated source codes `tiled-lutnet/lutnet/src/network/LUTNET_c6/sol/syn/verilog/LUTARRAY*.v`.
For each file, look for a line that looks something like this.
```
parameter    ap_const_lv36_1 = 36'b1;
```
In this example, it means this source file LUTARRAY*.v corresponds to the first LUTNet layer.
After you record down the order of the layers, update them at (around) line 624 in `tiled-lutnet/lutnet/h5py-2-hls/${dataset}/h52header_51lut_tm_mnist_spase.py` before proceeding to generate the LUT array verilog files.

## Change Tiling Factors

The following source files should be changed accordingly.
```
tiled-lutnet/training-software/model_architectures.py
tiled-lutnet/training-software/MNIST-CIFAR-SVHN/Binary.py
tiled-lutnet/training-software/MNIST-CIFAR-SVHN/models/${dataset}/scripts/bnn_pruning.py
tiled-lutnet/lutnet/h5py-2-hls/${dataset}/h52header_51lut_tm_mnist_spase.py
tiled-lutnet/lutnet/src/network/MNIST/hw/config.h
```

IMPORTANT: Tiling factor of "1" (i.e. full unrolling) doesn't work with this project.
HLS will generate a completely sequential implementation instead (a complete opposite to full unrolling).
This is due to the coding style of HLS C source codes in this project.
For fully unrolled LUTNet/ReBNet layers, please go to `LUTNet/unrolled-lutnet`.
Mix and match of tiled and fully unrolled LUTNet layers is possible.

## Reproduce ReBNet Results

This repository also includes the training and implementation source codes of ReBNet.
Implementing ReBNet takes similar steps as LUTNet, except LUT array replacement is not needed -- all BNN weights are stored in the generated C header file `weight.h`.

## Change Microarchitecture

The default microarchitecture is (5,1)-LUTNet.
The source codes for other microarchitectures are included in this repository.

## LUTNet-ReBNet Hybrids

Mixing and matching LUTNet and ReBNet layers are supported.
The following source codes should be modified.
```
tiled-lutnet/training-software/model_architectures.py (LUT=LUT for LUTNet and LUT=False for ReBNet)
tiled-lutnet/lutnet/h5py-2-hls/MNIST/h52header_51lut_tm_mnist_spase.py
tiled-lutnet/lutnet/src/network/MNIST/hw/top.cpp
```

repository: avl-bsuir/LUTNet

repo_name: avl-bsuir/LUTNet


text:
# Calculator-program
This is a basic calculator program.
It contains a header and 2 cpp files.
Build it using following command:-

g++ main.cpp calculator.cpp -I(capital 'i')(Location of the folder in which the calculator.h is located) -std=c++2a -pedantic-errors -Wall -o calculator.exe

repository: sneh-chauhan/Calculator-program

repo_name: sneh-chauhan/Calculator-program


text:
# Matches Fashion Tech Test

Add below commands to terminal in order to download the repo, install related packages and run the server

	$ git clone https://github.com/ercekal/mf_TT
	$ cd gousto_tt
	$ yarn install
	$ yarn start

After the server run and compile is done run
	$ open http://localhost:3000

Redux fetches the JSON data via axios and it is passed as props to items component.

Items category gets the items props and once it is loaded it maps the data as itemInfo component for each item.


text:
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) > [sip.js](./sip.js.md) > [IncomingRequestDelegate](./sip.js.incomingrequestdelegate.md)

## IncomingRequestDelegate interface

Delegate providing custom handling of incoming requests.

<b>Signature:</b>

```typescript
export interface IncomingRequestDelegate
```

## Methods

|  Method | Description |
|  --- | --- |
|  [onCancel(message)](./sip.js.incomingrequestdelegate.oncancel.md) | Receive CANCEL request. https://tools.ietf.org/html/rfc3261\#section-9.2 Note: Currently CANCEL is being handled as a special case. No UAS is created to handle the CANCEL and the response to it CANCEL is being handled statelessly by the user agent core. As such, there is currently no way to externally impact the response to the a CANCEL request and thus the method here is receiving a "message" (as apposed to a "uas"). |
|  [onTransportError(error)](./sip.js.incomingrequestdelegate.ontransporterror.md) | A transport error occurred attempted to send a response. |

gha_license_id: MIT


text:
<snippet>
<content>
# 1:Gesture Controlled \& Autonomous Turtlebot

In this project, the Turtlebot is transformed into a robot bodyguard which can attack the dangerous target given human's command. A piece of paper printed with ar tags will be treated as the imaginary bad guy and the goal of the turtlebot is to pierce through the tag with a sharp spear when the master places the order. As in real battlefield, soldiers communicate with each other via professional gestures instead of giving vocal commands, a safe and easy approach to operate the turtlebot is to use gesture control via an armband named MYO.

## Installation

You need a MYO armband and a Turtlebot with Kinect to play with the code.

## Usage

TODO: Write usage instructions

## Hardware

MYO Arm Band, Ar Tag Tower, Turtlebot with Kinect, Toy Spear

## Video

## Credits

Qianli Ma

## License
BSD
</content>
<tabTrigger>readme</tabTrigger>
</snippet>

repository: yucnet/MYO-TurtleBot

repo_name: yucnet/MYO-TurtleBot


text:
---
layout: post
title: Bikeshed
---

![Task graph](../images/bikeshed-graph.jpg)

# Why?
Why would anyone spend time writing a job manager that is lock-free, fixed memory, header only, C99 compliant, has few dependencies, supports many to many dependency trees, state cloning, configurable platform bindings, custom assert handling in less than 500 lines of MIT licensed code? I mean there is already plenty of such stuff already, right? Truth is, I don't know if there are any and that is not really the point.

I have been looking at job manager implementations for quite some time and I have my own ideas on how it should be built - as all developers do. The truth is probably somewhere between that I find it fun and that I want it to work my way.

Also, I always try to limit the scope of stuff I do, there are tons of features that *could* be added and in truth, I've made multiple implementations of job managers that always tend to grow and add stuff that I thought would be useful. It always ends up being too complicated and hindering the effectiveness of the implementation and I throw away the code and take the learnings with me for the next time.

So this time I set out to just do what I would need for my hobby projects so this might not be useful for you. However, since it is small and MIT license you are free to take pieces and change it to something you want. Or you can just look at it and see a bit how I think when writing code.

# Basic principles
## Don't implement stuff that the user of the library is better suited to do
When writing libraries or code that others should build on, don't try to solve more than needed, you don't know much about the hows and whys of the code that uses your library so leave as much of the decision making to the user.

There are a couple of obvious things in this library that are left out just for that reason.

### No threading code
Although the library is built to be driven by multiple threads there is no code for creating or managing threads at all. There is just one function that your worker threads call and an (optional) callback for you to implement when new tasks are ready to execute where you can wake up threads if you have put them to sleep.

There are no rules about how many threads or how they synchronize or sleep, that is up to the library user. You may use the library with just a single thread where the library is used to execute tasks in the correct order to resolve dependencies, or you might use fibers instead of threads.

The library is thread safe but there are some limitations on what you can do with tasks once they have been marked ready for execution.

### No memory management
The user of the library is completely responsible for managing the memory both that the library uses and what the tasks use.

A job manager instance requires a certain amount of memory based on how many active tasks and active dependencies should be supported at a time, but it does not allocate the memory itself.

There are no "allocators" or calls for malloc/new/free/delete or smart pointers. The memory the job manager needs is the users' responsibility to provide, the library can answer as how much it needs to create a manager instance which allows you to specify where it should be allocated - on the heap, on the stack, a single memory allocation or allocated as part of a bigger allocation.

Each task provides a callback and a context (a void pointer) which the library user dictates what it is for and how its lifetime is managed.

Since the job manager requires all the needed memory to be allocated up front for creating and executing tasks it is possible to make a very low overhead implementation. No memory management is needed and creating a task is just one interlocked add and one (unless there is heavy contention, where it needs to try again) interlocked compare and swap. Creating dependencies is equally cheap, as is freeing tasks and dependencies.

The library user does not have pointers to data inside the job manager but instead works with IDs which is safer and uses less memory.

As an example, if you were to create a job manager that supports up to 1024 active tasks and 256 dependencies it would require around 43 kb of memory. Note though this is for 1024 *active* tasks, as soon as a task executes the task memory can be reused, the same for resolved dependencies. You can let a job manager execute an arbitrary amount of tasks, the limit is how many pending and executing tasks there is at a single point in time.

So the responsibility to create a job manager of the correct size is up to the user. The reward is a job manager that is low overhead and without dynamic memory behavior.

## Don't add "useful" features
There are a couple of features that would probably seem useful for a job manager that I have chosen to not include that has been part of other earlier iterations of my job managers.

The biggest reason for not including them is that I don't need it for my use right now - what I want/need is something that can execute a dependency graph using multiple threads efficiently.

### Priorities
The most obvious one would be priorities. I have decided to *not* add that in the library mostly because I don't need it for my use right now.

There are a couple of pretty easy ways to add it though, the most basic one would be to create one job manager for each priority level. That would work pretty good but would likely waste memory.

The other option is to use multiple "heads" of ready task queue - one for each priority level. This will add minimal memory overhead but would require some changes to the library, I have some experimental code but nothing ready for consumption yet.

If you were to add priorities you would also need to create a scheduler that has some kind of fairness to it so tasks of different priorities get executed in an orderly fashion. This can be a simple or a very complex thing to create and really not a priority for me right now.

### Cancelling of tasks
There is no way to cancel a task once it has been readied. Also, there is currently no way that you can free a task that you created without readying and executing it.
If you need to cancel a task you need to indicate for the task function callback to do nothing, it will need to be executed anyway.

I don't have a need for canceling tasks so I just avoided the complexity of it.

However, I might want to add the possibility to free a task that has not been readied to handle the case where you build a graph and for some reason do not want to actually execute it.

### Yielding of tasks
If a task wants to give some time to other tasks but execute as soon as the currently queued tasks are executed but not go into a wait state you can't do that with the library as is now.

The states of tasks are very limited which makes them easier to reason about.

A task can be created, at that time you can add dependencies to it. Once a task is ready, either through an explicit call to `ReadyTasks` or by it having all its dependencies resolved you can no longer add dependencies to it. You are not allowed to add dependencies to a task once one of the existing tasks in its dependency chain is made ready either.

The task will then be executed and the while executed you can not add new dependencies to it but you can add new tasks which in turn have dependencies, just not dependencies to the executing task or any task that is ready or have ready dependencies.

Once executed the task can return that is either complete which will free up the task or it can say that it is blocked. If it says blocked it goes back to the "created" state and will not execute until it is made ready again.

### Predictable order of non-dependant tasks
Tasks are not executed in any particular order, just because you readied task A before task B does not guarantee that task A will be executed before task B. Since the library is built to support multithreaded executing there is no guarantee of the order anyway so I have completely ignored adding any code that enforces that.

The way to achieve order is by adding dependencies (which are really cheap) or by creating a new task from inside the execution of a task and ready the new task.

### Cyclic dependency detection
There is no explicit support for detecting cyclic dependencies in the graph but there are safeguards when API usage validation is enabled.

A task will not get into ready for execution state as long as it has dependencies and if you try to ready a task that still has non-resolved dependencies you will get an error result back.

What this means is that if you build a graph that has cyclic dependencies there is no way to start the cyclic part of the graph without getting an error in return.

Of course, there is still possible to create a graph that never executes and there is no built-in code to prevent that.

### Thread local queues and job stealing
One common feature of job managers is to have thread local queues to reduce contention of the ready queue. Even though it might be useful I opted out of that for now to reduce complexity.

## A sample job graph
Let us go through how you would construct the job graph in the picture at the top of this blog post.

It consists of eight jobs with seven dependencies so we will make a shed of just that size.

### Creation and destruction of a bikeshed
There is only a function to create a job manager, no function to destructing it. This is because there is nothing to do at the destruction. If the library user has allocated the memory dynamically the memory needs, of course, to be freed but that is not done by the library.
Creating a job manager is actually very cheap so it is fully viable to create job managers on demand on the stack.

Creating a job manager on the stack:
```C
static const uint32_t MAX_TASK_COUNT = 8;
static const uint32_t MAX_DEPENDENCY_COUNT = 7;

// Reserve memory for the bikeshed on the stack
char mem[BIKESHED_SIZE(MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT)];

// Create a bikeshed without any ready-state notification
// The create function can only fail if the max task count or max dependency count
// exceeds 8 388 607 in which case it returns 0.
Bikeshed shed = Bikeshed_Create(mem, MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT, 0);

// Now the shed is ready to use and since we allocated the memory on the stack
// there is no need to destruct the shed.
```

Creating a job manager on the heap just a matter of allocating the memory from the heap and making sure to free it when done.
```C
static const uint32_t MAX_TASK_COUNT = 8;
static const uint32_t MAX_DEPENDENCY_COUNT = 7;

// Allocate memory for the bikeshed from the heap
void* mem = malloc(BIKESHED_SIZE(MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT));
// Make sure the malloc call succeeded before calling Bikeshed_Create()

// Create the bikeshed
Bikeshed shed = Bikeshed_Create(mem, MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT, 0);

// Now the shed is ready to use, the memory was allocated on the heap so it
// needs to be freed. The shed is also a pointer to the same memory address
// so we can just free that.
free(shed);
```

As we can see the library does not dictate how the memory was allocated and has no explicit destruction code.
Since it does not allocate any memory by itself there is no need for any cleanup function call.

### Creating tasks
You create one or more tasks in one go, the tasks are just created but will not execute until made ready. Each task needs a callback function and an (optional) context.
```C
struct MyJobData
{
  // data the task operates on
}

static Bikeshed_TaskResult MyJob(Bikeshed shed, Bikeshed_TaskID task_id, void* context)
{
  struct MyJobData* _this = (struct MyJobData*)context;

  // Do work with the job data here

  // We know how the data is allocated, in this sample it is on the stack so no need to free it.

  // The task is complete and the task will be freed
  return BIKESHED_TASK_RESULT_COMPLETE;
}

void DoJobs(Bikeshed shed)
{
  // The data each job uses
  MyJobData job_data[8];

  // The function for each task
  BikeShed_TaskFunc task_functions[8] =
    {
      MyJob, MyJob, MyJob, MyJob,
      MyJob, MyJob, MyJob, MyJob
    };

  // The context for each task
  void* task_contexts[8] =
    {
      &job_data[0], &job_data[1], &job_data[2], &job_data[3],
      &job_data[4], &job_data[5], &job_data[6], &job_data[7]
    }

  // The resulting task ids
  Bikeshed_TaskID task_ids[8];

  int ok = Bikeshed_CreateTasks(shed, 8, task_functions, task_contexts, task_ids);
  assert(ok);

  // The tasks have now been created and can be marked as ready for execution
  // Note that we put all our data on the stack so they all need to be
  // executed *before* we leave this function scope.
}
```
### Creating dependencies
We now have all our eight tasks and can now set up the seven dependencies. You set up dependencies by adding "children" to a task.
There is no concept of a parent type task, any task can have zero or more children which allows you to express complex graphs with one simple function.

```C
static const uint32_t TASK_A_INDEX = 0;
static const uint32_t TASK_B_INDEX = 1;
static const uint32_t TASK_C_INDEX = 2;
static const uint32_t TASK_D_INDEX = 3;
static const uint32_t TASK_E_INDEX = 4;
static const uint32_t TASK_F_INDEX = 5;
static const uint32_t TASK_G_INDEX = 6;
static const uint32_t TASK_H_INDEX = 7;

// Task 'A' is dependend on three other tasks - 'C', 'D' and 'E'
Bikeshed_TaskID task_A_dependencies[3] =
  {
    TASK_C_INDEX,
    TASK_D_INDEX,
    TASK_E_INDEX
};

ok = Bikeshed_AddDependencies(shed, task_ids[TASK_A_INDEX], 3, task_A_dependencies);
assert(ok):

// Task 'B' is dependend on two tasks - 'E' and 'H'
Bikeshed_TaskID task_B_dependencies[2] =
  {
    TASK_E_INDEX,
    TASK_H_INDEX
};

// A task can have multiple "parents" as well as multiple "children"
ok = Bikeshed_AddDependencies(shed, task_ids[TASK_B_INDEX], 2, task_B_dependencies);
assert(ok):

// If a task has just one dependency you don't need to create an array for the dependencies
ok = Bikeshed_AddDependencies(shed, task_ids[TASK_D_INDEX], 1, &task_ids[TASK_F_INDEX]);
assert(ok):

ok = Bikeshed_AddDependencies(shed, task_ids[TASK_E_INDEX], 1, &task_ids[TASK_G_INDEX]);
assert(ok):

ok = Bikeshed_AddDependencies(shed, task_ids[TASK_F_INDEX], 1, &task_ids[TASK_G_INDEX]);
assert(ok):

ok = Bikeshed_AddDependencies(shed, task_ids[TASK_G_INDEX], 1, &task_ids[TASK_H_INDEX]);
assert(ok):

```
### Readying tasks
Now we have built the entire graph that we want to execute, all we need to do is ready up the tasks that has no dependencies.

As the user of the library is the one creating the tasks and setting up the dependencies the user knows which tasks are ready to execute. There is no need for the library to try and figure that out.

```C
// Task 'C' and task 'H' has no dependencies so they can be readied.
// As tasks are executed and dependencies resolved more tasks will be readied.

Bikeshed_TaskID ready_tasks[2] =
  {
    TASK_C_INDEX,
    TASK_H_INDEX
  };

Bikeshed_ReadyTasks(shed, 2, ready_tasks);
// C and H are now ready to execute
```
### Executing tasks
There is one function that you call to execute tasks, it takes the first ready task if available and executes it. If the tasks completes it will also resolve its dependencies and ready tasks whose dependency count reaches zero as a result of the resolving.

It returns zero if there are no more tasks ready to execute. In our example we will just use the main thread to execute the tasks and once it returns zero all tasks have completed execution.

```C
while(Bikeshed_ExecuteOne(shed));
```

### Threaded execution
If you are using threaded executing each worker thread will call `Bikeshed_ExecuteOne` and you would probably want to go into an efficient wait state if it returns zero.

The library provides a callback interface when creating the shed that notifies you as the library readies tasks and from there you can wake up any sleeping workers as needed.

Example code (incomplete) to show the basic principles of threaded execution:
```C
// Define your callback and instance
struct MyWorker
{
  Bikeshed_ReadyCallback cb = { ready };
  static void ready(Bikeshed_ReadyCallback* primitive, uint32_t ready_count)
  {
    MyWorker* cb = (MyWorker*)primitive;
    wakeup_worker(cb, ready_count);
  }
  static void worker(Bikeshed shed, MyWorker* cb)
  {
    while (continue_worker(cb))
    {
      if (Bikeshed_ExecuteOne(shed))
      {
        continue;
      }
      wait_for_wakeup(cb);
    }
  }
  static int continue_worker(MyWorker* cb);
  static void wakeup_worker(MyWorker* cb, uint32_t worker_count);
  static void wait_for_wakeup(MyWorker* cb);
} myworker;

// Create the shed with the ready callback, you are responsible for keeping the callback instance alive for as long as the shed is alive.
Bikeshed shed = Bikeshed_Create(mem, MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT, &mycb.cb);

// Launch your threads
thread1 = StartThread(MyWorker::worker, shed, &myworker);
thread1 = StartThread(MyWorker::worker, shed, &myworker);
thread1 = StartThread(MyWorker::worker, shed, &myworker);
thread1 = StartThread(MyWorker::worker, shed, &myworker);

// Add tasks to your shed and they will be executed by the workers

// Stop the workers
```

### Cleaning up
Well, in the non-threaded example we allocated everything on the stack so there is nothing to clean up, just return from the function and you are done.

## Cloning state
The library has one other interesting feature and that is the ability to clone a task managers state.

Imagine you have a task dependency graph that you want to run multiple times - it has the same input to the job manager and you run it for example once a frame in a game loop.

Building up the graph each time is a bit redundant and even though it is efficient to build it again it is still a bit of logic and atomic operations involved.

Since the job manager state has very few pointers internally - five - regardless of how many tasks or dependencies it has, it is very easy to clone a prepared state, execute on the clone and leaving the original intact. Next time you want to execute the same thing, just clone the state again and go.
The clone operation is basically just the cost of a memcpy of the shed memory and seven pointer offset calculations.

Of course, this only preserves the state of the job manager, any data the tasks touch when executed will not be preserved.

If we were to take our example and add state cloning to it we only need to add a few lines of code around the execution part:
```C
// Reserve memory for the cloned state - make sure to use the same task and dependency count as the original otherwise you will get memory corruption
char clone_mem[BIKESHED_SIZE(MAX_TASK_COUNT, MAX_DEPENDENCY_COUNT)];

Bikeshed cloned_shed = Bikeshed_CloneState(clone_mem, shed, sizeof(clone_mem));

// We execute the cloned_shed and the original shed is untouched
while(Bikeshed_ExecuteOne(cloned_shed));

// If we want to run the graph again we just clone and execute again:
cloned_shed = Bikeshed_CloneState(clone_mem, shed, sizeof(clone_mem));
while(Bikeshed_ExecuteOne(cloned_shed));
```

## Optional and customizable debug validation
The library has a built-in macro to validate that the API is used correctly. You have the option enable or disable it with the preprocessor directive `BIKESHED_ASSERTS`.
It is not enabled by default to provide you with the highest performing library out of the box.

The assert functionality does not really have an implementation and if you do not set the callback with `Bikeshed_SetAssert` the assert function does nothing.

If the preprocessor flag `BIKESHED_ASSERTS` is not set the entire validation logic is removed.

The validation macros are focused on validate correct API usage, not to monitor the internal state of the job manager - the intention is that the tests validate the library code and the validation macros the users' code.

## Configurable platform specific functions
The default implementation for atomic add and atomic compare and swap relies on `Windows.h` for the MSCV compiler and on clang and gcc it uses built-in compiler intrinsics.

You can override one or both of the functions by defining your own macros for `BIKESHED_ATOMICADD` and `BIKESHED_ATOMICCAS`. If you set both the dependency to `Windows.h` will be excluded.

# Thank you
Thanks for reading this far, I hope it was an interesting read.

Feel free to contact me with feedback, good or bad at my twitter account: [Dan Engelbrecht](https://twitter.com/DanEngelbrecht).

You can find the repo with all the code here: [github.com/DanEngelbrecht/bikeshed](https://github.com/DanEngelbrecht/bikeshed).

2019-04-23

gha_language: CSS


text:
# Robot-Simulation-PyQt5

![Robot simulation](/robot_modelling.png)

Python script using PyQt5 as window frame and OpenGL as 3D environment.
Each robot is an instance of the DOF3_robot class.

Required modules:
  - PyQt5
  - pyqtgraph/opengl
  - Numpy


text:
# Global variables
## $model

In several instances, most prominently in `app.xqm`, a parameter called `$model` is passed to functions.
This is part of eXist's templating system and has been used in several locations to allow for uniform distribution of common parameters.

`$model` always is a `map(*)`. The most important parameters MUST be the same in all instances. This means:

| parameter | contents |
|--|--|
| `ed` | the ID of the Collection (= `@xml:id` of `meta:projectMD` or `mets:mets`) |
| `fileLoc` | full path to the current Resource |
| `id` | a file's ID (e.g. file to be displayed) |
| `infoFileLoc` | full path to `wdbmeta.xml` or `mets.xml` |
| `pathToEd` | the full DB-Path to the Collection (= `substring-before($infoFileLoc, '/wdbmeta.xml')`) |
| `projectFile` | path to Project’s `project.xqm`|
| `projectResources` | path to Project’s resource folder (= `substring-before($projectFile, 'project.xqm') || 'resources'`)|
| `title` | the title of the file or project |
| `view` (app.xqm only) | the `view` query parameter as pass in the call to `view.html` |
| `xslt` (app.xqm only) | full path to the XSLT to be used for transformation |
| `p` (function.xml only) | all values from `p` query parameter parsed into a `map(*)` |

## parameters in `wdb` namespace
The [[wdb namespace|list-of-namespaces]] contains several global parameters that can be accessed from every script that imports `app.xql`:

|name|contents|
|--|--|
|`$wdb:edocBaseDB`| same as `$config:root` – the path to where the application is installed within `/db/`.|
|`$wdb:configFile`| the (parsed) config file (`{$wdb:edocBaseDB}/config.xml`).|
|`wdb:data`| the path to the data collection. It is assumed that this is the top collection with a `wdbmeta.xml` file.|
|`wdb:server`| the server's address including protocol and port – if automatic resolution does not work, set it manually in [[config.xml\|global-configuration]].|
|`$wdb:edocBaseURL`| the full URL to the app's root or the current subcollection.|
|`$wdb:role`| the [[role\|server-roles]] of this instance.|
|`$wdb:peer`| this instance's peer, if it is a workbench.|

repository: dariok/wdbplus

branch_name: refs/heads/development

gha_language: XQuery

repo_name: dariok/wdbplus


text:
Wrapper scripts to run GitHub actions in a jail on FreeBSD
==========================================================

*DISCLAIMER*: This uses [a third-party implementation of the GitHub Actions protocol](https://github.com/ChristopherHX/github-act-runner) and is not officially supported by GitHub.

This repository provides some scripts to run CI driven from GitHub in a FreeBSD jail.
It uses `pot` to manage the jails.
The pot is cloned from an immutable ZFS filesystem, runs one action,  stops, the underlying ZFS filesystem is destroyed, and then the process repeats.
This allows the actions to do whatever they need to as root in the jail (install software, whatever).
*This is a work in progress, please file issues as you find them.*

Installing and configuring a runner
-----------------------------------

Install with `./install.sh`.
This will install a `pot` flavour and the RC scripts required to run the jails.
Register a GitHub action by [following the directions from GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners).
When you are asked to select an operating system, select Linux.
Ignore all of the commands that you are instructed to run except for the one that starts `./config`.
Paste that command into a root terminal in the directory where you have cloned this repository.

Three environment variables affect this command:

 - `FREEBSD_VERSION` specifies the version of FreeBSD to use (e.g. 13.1).
   If not specified this is the version of your host system.
 - `RUNNER_NAME` specifies the name to give to this runner.
   If not specified then this is the {your hostname}-freebsd-{version number}.
 - `RUNNER_FLAVOURS` specifies a space-separated list of flavours that will be applied to the pot.
   This allows you to provide a set of additional tools that will be made available to runner actions.

This script will create a pot (container) that has a configured GitHub Actions runner inside.
The name of the pot is the runner name with any dots replaced with underscores (pot names are not allowed to contain dots), the script will output this on the last line of the output.
The runner will have `freebsd` and `freebsd-{version}` labels, these can be used to select runners with the `runs-on` property in the YAML.

The runner's configuration is exported from into `runners/{pot name}`, allowing the runner to be re-created without having to re-register it with GitHub.
This exported configuration includes all of the environment variables that the script used when setting them up.

This pot can be exported and imported on another system using the [existing pot commands](https://pot.pizzamig.dev/Container/).
This pot will not be run directly by the scripts, it is treated as an immutable prototype and cloned for each invocation.

Installing dependencies
-----------------------

You can modify the pot to install dependencies such as compilers or other tools.
You can update the base-system image in the same way.

To make orchestration easier, you should provide your dependencies as one or more [`pot` flavours](https://pot.pizzamig.dev/Images/#images-creation-automated-with-flavours).
These can be injected into the pot by setting the `RUNNER_FLAVOURS` environment variable as outlined above.

Re-creating a runner
-------------------

The `recreate-runner.sh` script re-creates a runner.
This expects to find the `runners/{pot name}` directory containing the configuration.
This will read the environment variables that were provided to `config.sh` from `runners/{pot name}/act-config.sh`.
You can modify this to specify a newer FreeBSD base version (for example, moving from FreeBSD 12.2 to 12.3) or to change the set of flavours that are installed.
Note that recreating the runner will *not* change the labels and so you may need to manually modify labels that refer to the version if you replace a runner with a newer version.
If you wish to change the name of the pot when re-creating the runner (for example, to include a version or date) then you must copy or rename this directory.

Note that only one pot with any given name can exist on the system at a time and so you must either rename the pot or destroy the pot before recreating it.
Note that you can create the pot on one system, export it, and then import it on your deployment system.

If you have created your runners by providing flavours with all of the dependencies then this script allows you to generate a new version with all dependencies.

### Recreating all runners

The `recreate-all-runners.sh` script will recreate all runners that you have created.
This iterates over all of the configs in the `runners/` directory and so must be run from the same location as `config.sh`.
This can be run from a cron job to make sure that all runners have picked up the security updates.

Running the runners
-------------------

The install script will install a `gh_actions` RC script.
This is controlled by two variables in `/etc/rc.conf` (or any of the equivalent places):

 - `gh_actions_enable`, set this to `"YES"` to enable starting runners automatically.
 - `gh_actions_pots` is a space-separated list of pot names for the action runners.

Once these are set, `service gh_actions start` will start all configured runners.
`service gh_actions stop` will stop all runners.
*Note*: There is a bug in the current version of `pot` that prevents starting a pot immediately after it was forcibly stopped.

Using the runners
-----------------

The runners appear like any other GitHub Actions runners.

```yaml
name: Example

# Controls when the workflow will run
on:
  # Triggers the workflow on push
  push:

  # Triggers the workflow on pull-request
  pull_request:

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  # This is just a name
  freebsd:
    # Select any runner with the freebsd-12.2 tag.  This is added automatically
    # to runners that these scripts create from a 12.2 base.
    runs-on: freebsd-12.2
    name: Test in FreeBSD
    steps:
    # Use the existing (NodeJS) checkout action to clone the repo:
    - uses: actions/checkout@v2
    # Add any other GitHub actions here
    - name: Build
      run: {your build commands here}
```

repository: krionbsd/act-pot

repo_name: krionbsd/act-pot


text:
---
layout: cv
title:
---

# Charlotte Francoeur
Microbiology Ph.D. Candidate

<a href="francoeur<EMAIL_ADDRESS>| cfrancoeur.github.io

-----
## Education

**University of Wisconsin - Madison**  `August 2016 - Present`
Microbiology Doctoral Training Program
Laboratory of Dr. Cameron Currie, Department of Bacteriology
GPA: 4.0/4.0

**University of Maryland - College Park**    `August 2012 - May 2016`
Bachelor of Science in Microbiology with Black Women's Studies Minor
GPA: 3.905/4.0
Cum Laude Latin Honors
Integrated Life Sciences Honors
Cell Biology and Molecular Genetics Departmental Honors

## Research Experience
### Current
**[Currie Lab](https://currielab.wisc.edu/), University of Wisconsin-Madison** `January 2017-Present`
>Graduate research assistant in the laboratory of Dr. Cameron Currie investigating bacterial and viral associations in fungus-farming ants (tribe: Attini). You can find more details about my research at cfrancoeur.github.io/research/
### Past
**Wu Lab, University of Maryland - College Park** `September 2014 - July 2016`
>Institute for Bioscience and Biotechnology Research Laboratory
>Volunteer undergraduate research assistant in the laboratory of Dr. Louisa Wu
>Used the [Drosophila Genetics Research Panel](http://dgrp2.gnets.ncsu.edu/) to perform a genome-wide association study (GWAS) to find genes associated with the phagocytosis of fungi in *Drosophila melanogaster*

**Nou Lab, USDA-ARS** `August 2012 - May 2014`
>Biological Science Aid in the Environmental Microbial and Food Safety laboratory of Dr. Xiangwu Nou
>Investigated biofilm formation between *Ralstonia insidiosa* and foodborne pathogens, *Escherichia coli*, *Salmonella spp.*, and *Listeria monocytogenes*
>Resulted in a publication (see below)

**Martin Lab, USDA-ARS**  `August 2011 - May 2012`
>High School Research Intern in the Invasive Insect Biocontrol and Behavior laboratory of Dr. Phyllis Martin
>Investigated bacterial strains pathogenic to the brown marmorated stink bug using 16S rRNA sequencing, Biolog, and phenotypic tests (e.g. optimal growth conditions, hemolytic activity, urease production)

## Publications

Liu, N. T., Bauchan, G. R., **Francoeur, C. B.**, Shelton, D. R., Lo, Y. M., & Nou, X. (2016). Ralstonia insidiosa serves as bridges in biofilm formation by foodborne pathogens Listeria monocytogenes, Salmonella enterica, and Enterohemorrhagic Escherichia coli. Food Control, 65, 14–20.

## Mentoring ##
### Currie Lab ###
>`2018-Present`**Olivia Panthofer** Undergraduate Student. Isolation of phage from fungus gardens.
>`Summer 2018`**Jennifer Koehler:** REU student. Lipid Production of *Streptomyces* on Conversion Residue.
>`January 2018` **Donny Hoang:** MDTP rotation student. Inhibition of *Escovopsis* by *Burkholderia*.
>`2017-2018` **Josh Daniels:** Undergraduate student. Investigation of Bee-Associated *Streptomyces* species and their ability to produce lipids.
> `2017-2018`**Laura Williams:** Undergraduate student. Characterization of *Burkholderia sp.* isolated from the fungus gardens of fungus farming ants.

repository: cfrancoeur/markdown-cv

repo_name: cfrancoeur/markdown-cv


text:
### [CVE-2019-10771](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10771)
![](https://img.shields.io/static/v1?label=Product&message=iobroker.web&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=Cross-site%20Scripting%20(XSS)&color=brighgreen)

### Description

Characters in the GET url path are not properly escaped and can be reflected in the server response.

### POC

#### Reference
No PoCs from references.

#### Github
- https://github.com/ossf-cve-benchmark/CVE-2019-10771

repository: trickest/cve

gha_language: HTML

repo_name: trickest/cve