text:
# Changelog

## Unreleased

## [1.10.0] (2020.02.05)

### Fixed
- Fixed `max_sale_qty` getting wrong value when set per-product. [@rain2o](https://github.com/rain2o) ([#201](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/201))
- Fixed exporting products for Magento Commerce: Bundle products and Linked products([#202](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/202))

## [1.9.0] (2020.01.28)

### Fixed
- Emulate store when rebuilding cms block and pages. ([#179](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/179))
- Fix modules sequence declaration
- Fixes for clean magento install with sample data modules.
- Collecting attributes values when there is no default value ([#160](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/160))

### Added
- Product indexer - add dependencies, add dynamic catalogrule_product_price dependencies for Magento <= 2.2.4 ([#176](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/176))
- Reindexing products which have beed modified by mass action on product grid when indexers are 'On Save ([#157](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/157))

## [1.8.3] (2019.12.19)

### Fixes
- support table prefix with review_entity
- fix exporting final_price and price for configurable products when `final_price` equals `0`

## [1.8.2] (2019.11.29)

### Fixes
- fix notice undefined final_price

## [1.8.1] (2019.11.28)

### Fixes
- export catalog rule prices for Magento <= 2.2.4
- export update_at field for products

## [1.8.0] (2019.11.20)

### Added
- Add "final_price" to "configurable_children" ([#151](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/151))

## [1.7.0] (2019.11.15)

### Changed/Improved
- Speed up exporting for product/category data. Class `ConvertDataTypes` was removed, converting value to int/float is executed when loading product/category attributes.

## [1.6.0] (2019.11.07)

### Added
- Add support for Visual Swatches. ([#15](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/15))
Attribute options
```json
"options": [
    {
      "swatch": {
        "type": 1,
        "value": "#ffd500"
      },
      "value": "60",
      "label": "Yellow",
      "sort_order": 12
    },
    {
      "swatch": {
        "type": 3,
        "value": null
      },
      "value": "105",
      "label": "purple",
      "sort_order": 13
    },
    {
      "swatch": {
        "type": 2,
        "value": "/4/2/4245592.png"
      },
      "value": "106",
      "label": "File",
      "sort_order": 14
    }
]
```

Product configurable option values
```json
"values": [
    {
      "value_index": "50",
      "label": "blue defaul",
      "swatch": {
        "type": 1,
        "value": "#1857f7"
      }
    },
    {
      "value_index": "49",
      "label": "Black",
      "swatch": {
        "type": 1,
        "value": "#000000"
      }
    }
  ]
```
- Add new command `vsbridge:index` to run indexer for one product, cms block, etc.
`bin/magento vsbridge:index vsbridge_product_indexer default 1`
vsbridge_product_indexer -> indexer code
default -> store code
1 -> entity id (in this particular example - product id)
Useful for testing.

## [1.5.2] (2019.10.24)

### Fixes
 - Run `vsbridge:reindex` command only if all indices has valid status.

## [1.5.1] (2019.10.23)

### Fixes
-  Fix Notice: Undefined index: value in ConfigurableData.php on line 254 ([#139](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/139))
-  Switch indices when all types will be reindex ([#138](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/138))
-  Change method name `AbstractEavAttributes:canReindex` to '`AbstractEavAttributes:canIndexAttribute`

## [1.5.0] (2019.10.18)

### Changed/Improved
-  Change method `AbstractEavAttributes:canReindex` to public ([#136](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/136))

### Added
- Add support for aliases ([#3](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/3)).

## [1.4.0] (2019.10.15)

### Fixes
- Fix `vsbridge:reindex` command, fix initialize right areacode, loading di.xml files for adminhtml areacode. ([#127](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/127))
- Fix `vsbridge:reindex --all` command, data will be reindex store by store. ([#130](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/130))

### Changed/Improved
- Rework abstract mapper to allow configure static type map in di.xml ([#125](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/125))
- Minor refactoring for product resource model ([#126](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/126/)).
- Vsbridge Category Indexer - Save Mode: reindex subcategories if url_key has change in category ([#122](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/122)).
- Sort configurable options by option sort order. ([#129](https://github.com/DivanteLtd/magento2-vsbridge-indexer/pull/129))

## [1.3.0] (2019.09.18)

### Fixes
- Fixed Backward compatibility with 2.2.x (< 2.2.6)  ([#53](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/53))

### Changed/Improved
- Skip exporting configurable products without "configurable_options".

### Added
- Export ratings for reviews.

## [1.2.0] (2019.08.05)

### Fixed
- Uncaught TypeError in CMS indexer  - [@indiebytes](https://github.com/indiebytes) ([#85](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/85))
- Multiselect attribute values check if they are numeric before casting to `int` type to avoid casting strings to `int` and therefore causing the value to be `0` always. - [@rain2o](https://github.com/rain2o)
- Fix running `bin/magento vsbridge:reindex --store` per store. Only data for given store will be updated in ES. ([#91](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/91))

### Changed/Improved
- Provide more verbose feedback in console when running `bin/magento vsbridge:reindex`.  Removed `int` type cast from `--store` value to allow for store code or store ID since StoreManagerInterface allows both types. - [@rain2o](https://github.com/rain2o)

### Added
- New optional parameter for `bin/magento vsbridge:reindex` command `--all` to allow reindexing all stores. - [@rain2o](https://github.com/rain2o)

## [1.1.0] (2019.07.10)

### Fixed
- Fix bug causing empty product URL paths  - [@indiebytes](https://github.com/indiebytes) ([#63](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/63))
- Changed ElasticSearch password to be obscured, encrypted, and considered sensitive (will dump to `env.php` instead of `config.php`) - [@rain2o](https://github.com/rain2o) ([#69](https://github.com/DivanteLtd/magento2-vsbridge-indexer/issues/69))
- Fix exporting values for multiselect option arrays as integers (instead of strings)
- Fix getting stock_status value for products
- Magento Commerce - fix getting configurable_children

### Changed/Improved
- Change mapping for "category.name" field in product type
- Remove dependency from catalog_product_price and cataloginventory_stock indexer. On product save multiple request has been send to ES
- Change "Use Short Catalog Urls" setting option to "Use Catalog Url Keys"
- Add option to generate product/category slug base on Magento Url Key and ID. By default slug (and url_key) field is generated base on NAME and ID.

### Added
- Adding support for video data. Small change will be needed in VSF #19
- Add support for "product_count" in category. When products are reassign to category, category data is not updated automatically in ES.
There is not need really, VSF only has to know if `product_count > 0`, so if you already had products assign to categories (before running ` bin/magento indexer:reindex vsbridge_category_indexer`)
category will be visible in menu sidebar.
- Add support for reviews. Reviews are exported without ratings (VSF does not support ratings for now)
- Add support for custom options.
- Add option to enable/disable exporting data to ES.
- Add ProductCategory indexer to partially update product data in ES (category, category_ids fields). Trigger after changing products positions in category.
- Add the ability to choose between Store ID and Store Code to be used at the end of index names.
- Add label for configurable option value.  **Note:** When You modify any configurable attribute label or option label in Magento You should reindex all products manually.
```json
{
  "attribute_id": 93,
  "attribute_code": "color",
  "label": "Color",
  "values": [
    {
      "value_index": 61,
      "label": "Gray"
    },
    {
      "value_index": 66,
      "label": "Purple"
    },
    {
      "value_index": 69,
      "label": "Yellow"
    }
  ]
}
```
- Add new command ```php bin/magento vsbridge:reindex``` which will run all Magento indices which names start with "vsbridge_" (experimental).
```php
Description:
  Rebuild indexer in ES.

Usage:
  vsbridge:reindex [options]

Options:
      --store=STORE     Store ID
      --delete-index    Delete previous index and create new one (with new mapping)
```

## [1.0.0] (2019.04.03)
First version

## March 2019

### Added
- Adding 'url_path' for products.
You have to delete existing index from Elasticsearch and run full reindexation.
You need to get correct mapping for new product field.
- Add support for "available_sort_by" and "default_sort_by" default values.

### Fixed
- fix creating mapping for category EAV attributes.
 You have to delete existing index from Elasticsearch and run full reindexation.
- fix exporting final_price and price for configurable products created with option "Apply unique prices by attribute to each SKU"


text:
# bruschetta

quick, garlicky, refreshing.

makes 12 bruschette (that feeds 2 bruschetta fans or up to 6 people if it's for a snack)

## ingredients

- loaf of bread (something ciabatta-like, or a big sourdough)
- 2 heritage tomatoes
- 4 garlic cloves
- 6 sprigs fresh oregano
- 2 sprigs fresh basil
- olive oil
- salt
- pepper
- flaky salt

## preparation

preheat oven at 160C top grill + fan assisted. slice bread ~1.5cm thick. dice tomatoes in ~.5cm pieces.

grill bread slices in almost-top rack of the oven for 4m, flip, repeat for 3m. rub garlic on bread (should be crispy and consume the garlic as you rub it as if it was chalk on a rock). drizzle generously with oil. top with tomato, oregano, salt, pepper, more more quick oil drizzle on top of tomato.

grill in oven for 3m (tomato should start drying on the edges and not be too moist anymore, leave 1 more minute if not). top with basil leaves and flaky salt and serve.

## notes

![alt text](https://github.com/caligin/actual-cookbook/blob/master/img/bruschetta.jpg "made 6 at first, then 6 more because it was too good not to!")

license: Unlicense

branch_name: refs/heads/master

extension: md

license_type: permissive


text:
---
title: "Weekly 7/2018"
date: "2018-02-12"
---#### This week: world’s worst CEO, Airbnb for dinner, and nice corners you got there.

_Our weekly collection design and development resources for hackathon hackers is once again filled with some must reads and good-to-knows._

### 💻 Treats for developers

#### [Fallbacks for Videos-as-Images](https://css-tricks.com/fallbacks-videos-images/)

Css-tricks’ [Chris Coyier](https://medium.com/u/a789923b5cad) has a great little tip about using <picture> element to do a fallback when using video as a source of <img>.

#### [Web-Powered Augmented Reality: a Hands-On Tutorial](https://medium.com/@urish/web-powered-augmented-reality-a-hands-on-tutorial-9e6a882e323e)

Interested in learning web-powered augmented reality, but don’t know where to start? [Uri Shaked](https://medium.com/u/355b1dfe86ae) has written one of the best articles out there about this very subject.

#### [Udash](https://udash.io/)

Love Scala? There’s a new framework, Udash, that helps you build “beautiful and maintainable” web applications with Scala. Take look!

#### [Sandboxed Mac apps can record your screen at any time without you knowing](https://krausefx.com/blog/mac-privacy-sandboxed-mac-apps-can-take-screenshots)

There is a huge security risk in sandboxed Mac apps. Read more on Felix Krause’s blog.

---

### 🍬 Candy for designers

#### [Airbnb for dinner](https://medium.com/@johnyvino/airbnb-for-dinner-ae692bcc1790)

Case study by [Johny vino](https://medium.com/u/d31e7ef6a63c) on designing a Airbnb for dinner that provides _“An authentic home food for tourist which provides a cultural touch for tourist”._

#### [Design For The iPhone X Like A Boss](https://uxdesign.cc/designing-for-the-iphone-x-part-1-6cbc7f30734d)

Crash course on how to design for iPhone X by [Azhar Anwar](https://medium.com/u/cc76ebbb7841). Great article on the specific design aspects of iPhone X.

#### [Supporting Human Cognition with Better Design](https://medium.com/swlh/supporting-human-cognition-with-better-design-49b607ad9c2e)

[Monsoonfish](https://medium.com/u/d8667d12afd8) has written great article on taking human cognition (memory, language, etc.) into account when designing.

#### [iPhone X corners](https://medium.com/tall-west/no-cutting-corners-on-the-iphone-x-97a9413b94e)

How cool are the iPhone X corners! 😍 I mean really!

---

### ❤ Something Else

#### [Story of the world’s worst CEO](https://medium.com/@harrikiljander/operation-elop-6f2b043f52c5)

[Harri Kiljander](https://medium.com/u/ca6ec52131b2) has written a massive (reading time 380 according to Medium) analysis on Stephen Elop’s shortcomings when running Nokia. A definite read!

#### [Junction is taking steps to become a global community](https://blog.hackjunction.com/junction-2018-taking-steps-towards-a-global-community-8a2f5ee0f94d)

[Junction](https://medium.com/u/fe460b6a6ec7), the hackathon organizer, is taking steps to become a global community. What does this mean in practice?

#### [Slack is the opposite of organizational memory](https://abe-winter.github.io/plea%27s/help/2018/02/11/slack.html)

Is Slack bad for your organization? Abe Winter argues that it actually destroys teams’ ability think, plan and get complex out of the door.

license: MIT

gha_language: TypeScript

gha_license_id: MIT


text:
### [CVE-2011-2483](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2483)
![](https://img.shields.io/static/v1?label=Product&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Version&message=n%2Fa&color=blue)
![](https://img.shields.io/static/v1?label=Vulnerability&message=n%2Fa&color=brighgreen)

### Description

crypt_blowfish before 1.1, as used in PHP before 5.3.7 on certain platforms, PostgreSQL before 8.4.9, and other products, does not properly handle 8-bit characters, which makes it easier for context-dependent attackers to determine a cleartext password by leveraging knowledge of a password hash.

### POC

#### Reference
No PoCs from references.

#### Github
- https://github.com/ARPSyndicate/cvemon
- https://github.com/SaltwaterC/PasswordHash2

repository: trickest/cve

branch_name: refs/heads/main

gha_language: HTML

repo_name: trickest/cve


text:
# RailGame an Alexa skill

This is a simple Alexa skill example.

## Getting Stared

Here, `lambda_function.py` is AWS lambda function which will process all the requests and gives back the answer

repository: smitshilu/RailGame

repo_name: smitshilu/RailGame


text:
Final Exam review
===

The final exam is comprehensive, but will focus more on the latest things we covered (especially since the latest material uses lots of the earliest one :). I will probably also ask a recursion question from the earliest material. The assignments also provide good study material.

Materials
+ [Review](FinalExamReview.md) with [Solutions](FinalExamReviewWithSolutions.md)
+ [Midterm review](MidtermReview.md) - with links etc
+ Videos
    + [Final review Lecture](https://youtu.be/7MSgLYNeixg)
    + [Midterm review Lecture](https://youtu.be/UtyX2cd0yuo)

+ Some codingbat problems (I can't find any for classes :(
    + [Simple recursion](http://codingbat.com/java/Recursion-1)
    + [Arrays](http://codingbat.com/java/Array-2)
    + [Harder Arrays](http://codingbat.com/java/Array-3)

repository: okaram/IntroJava

repo_name: okaram/IntroJava


text:
# Questions
- How can a recipe system work with complex layouts?
- How can axes be reused in recipes to plot a second layer on top of a previous?
- How are legends constructed in single recipes and how are they correctly updated in case of a second layer plotted on an existing one?
- What should be sensible defaults for plotting with base commands, what should they return?

With MakieLayout, high-level elements are treated as mostly autonomous containers
that take care of how their content is presented. They adjust their visual size and position
by receiving a suggested bounding box. This should usually (but doesn't need to)
be generated by a GridLayout that the element is assigned to. Therefore, to be placed
in a scene, an element does not need to be in a layout, but being in a layout moves
and resizes the element so it sensibly fits in with the other content. Usually, most
elements will exist in a layout, so incorporating layouts into the Makie pipeline
should be the default mode of thinking.

GridLayouts are flexibly resizable, columns and rows can be added and removed,
and they can be placed inside other GridLayouts. Therefore, a whole group of
elements can be placed inside an existing layout as a cohesive layout element using
a common GridLayout. This makes it possible to easily assemble larger layouts from
smaller parts. A subplot group can be generated in a separate function (or recipe)
and the result be attached to an existing Scene. This way, complex plots can be assembled
easily.

There are multiple important objects that the user should keep track of in order
to correctly build their visualization and update parameters if they wish. These are:

- The top-level Scene in which all other content is placed
- The layout tree in form of the top GridLayout, or even every single used GridLayout
- Larger content elements of the top-level Scene such as other Scenes or SceneLikes
(Axis is a SceneLike) or elements such as Label, Colorbar, Legend, etc.
- Plots that have been added to Scenes or Axis instances

## Plotting without a scene, layout or axis

The user will not always want to specify all parameters needed for a single correctly
plotted element. For example, a Scene should be generated when creating a Scatter
plot, if there is no Scene given as a mutated argument. In the same way, an Axis
could be needed if none exists yet, which would be placed in the Scene. And for
the Axis to be placed correctly, a layout will be needed, too. Of course, the Scatter
object itself should also be accessible as a result of the function call.
Therefore, a simple Scatter plot can result in four (or even more?) objects being created.

```julia
scene, layout, ax, scat = scatter(rand(10, 2))
```

It is probably not desirable to force the user to assign all of these return objects to variables
every time they create a simple plot. Single values could easily be retrieved using
dot access, if they are needed.

```julia
scene = scatter(rand(10, 2)).scene
layout = scatter(rand(10, 2)).layout
```

It should still be possible to display the scene after a single command like this
is run, even though the return type is not Scene anymore. This could be achieved with
multiple dispatch on any NamedTuple that partially matches our return type signature.

```julia
function scatter(x, y)
    # create scene, layout, ax and scat
    # then return a NamedTuple
    NamedTuple(scene = scene, layout = layout, axis = ax, plot = scat)
end

function Base.display(nt::NamedTuple{T, Tuple{Scene,GridLayout,U, V}}) where {T, U, V}
    display(nt.scene)
end
```

## Plotting into an existing scene and layout

In this case an existing scene is used and an object added to it. There could already
be a top-level layout associated with the scene, in this case it should be used
to add the new element to it. A new axis could be created, in this case it should
be returned as well. In general, it should be clear from the inputs what outputs
are expected, not so much because of the performance hit of type instabilities, but
so that the user knows a priori how to deconstruct the returned NamedTuple

```julia
# layout[1, 1] should construct an object like `GridPosition(layout, 1, 1)` so it
# can be dispatched on to put the resulting object into the specified layout position.
axis, plotobj = plot!(scene, layout[1, 1], xyz)
```
> ⚠ This could be a problem if the user doesn't want to plot into an axis. Maybe for this purpose the `raw` keyword could still be used.

## Plotting into an existing axis

In this case the scene and layout are already decided through the existing axis,
so only the plot object needs to be returned.

```julia
plotobj = plot!(ax, xyz)
```

# Recipes

## Recipes that create layouts

It should be straightforward to allow recipes to place their created content
in an existing layout. For this, again a new type `GridPosition` could be used to
dispatch on, that specifies layout and position at the same time.

```julia
somerecipe!(scene, layout[1, 1], xyz)
```

If the GridLayout type was amended so it can have an optional parent Scene then
this could simplify plotting syntax, as any layout in a layout tree whose parent
has a Scene attached could serve as a surrogate for that Scene.

```julia
# if this holds
parent(layout) == scene

# then these two things should be equivalent
axis, plotobj = plot!(scene, layout[1, 1], xyz)
axis, plotobj = plot!(layout[1, 1], xyz)
```

The `axis` in the previous two function calls would be created because there was
no specific axis given to the function. This could potentially be confusing behavior
as users might expect to be able to just plot into a layout position and thereby
target the axis already located there.

Plotting like in the above example could also return new layout object if we're dealing
with a recipe that creates its own sublayout. To have a clear expectation whether a
recipe call creates a sublayout or not, a trait could be used (for example `CreatesSublayout`).

# Function signatures:

## Non-layout creating

```julia
scene, layout, axis, plotobj = plot(args...)
       layout, axis, plotobj = plot!(scene, args...)
               axis, plotobj = plot!(scene, layout[1, 1], args...)
               axis, plotobj = plot!(layout[1, 1], args...) # if layout has scene connected
                     plotobj = plot!(axis, args...)
```

## Layout creating

```julia
scene, layout, plotlayout, plotaxes, plotobjects = plot(args...)
       layout, plotlayout, plotaxes, plotobjects = plot!(scene, args...)
               plotlayout, plotaxes, plotobjects = plot!(scene, layout[1, 1], args...)
               plotlayout, plotaxes, plotobjects = plot!(layout[1, 1], args...) # if layout has scene connected
                                     plotobjects = plot!(plotaxes, args...)
```

Maybe the first version should also create a sublayout so the two signature groups collapse into one:

```julia
scene, layout, axislayout, axis, plotobj = plot(args...)
       layout, axislayout, axis, plotobj = plot!(scene, args...)
               axislayout, axis, plotobj = plot!(scene, layout[1, 1], args...)
               axislayout, axis, plotobj = plot!(layout[1, 1], args...) # if layout has scene connected
                                 plotobj = plot!(axis, args...)
```

Bundle `axislayout, axis` into one object? It seems to always go together in the signatures.
This is kind of the "infrastructure" that a plot brings with it.

```julia
scene, layout, infrastructure, plotobj = plot(args...)
       layout, infrastructure, plotobj = plot!(scene, args...)
               infrastructure, plotobj = plot!(scene, layout[1, 1], args...)
               infrastructure, plotobj = plot!(layout[1, 1], args...) # if layout has scene connected
                               plotobj = plot!(infrastructure, args...)
```

What type should `infrastructure` have? Flexible enough for any plot type, also reuseable between different plot types, like adding a `Scatter` to infrastructure returned from `Lines`.
Maybe just NamedTuple with a convention for fields.
This convention could be tied to Traits like `SingleAxis`, and `FacetedAxes` or `CustomAxes`

Fields for `SingleAxis`
- layout
- axis
- support

Fields for `FacetedAxes`
- layout
- axes # 2D array
- support

Fields for `CustomAxes`
- layout
- axes # named tuple
- support

# Keyword arguments

There needs to be a robust mechanism to get keyword arguments to where they should go from a very high-level call
like `plot(args...; kwargs...)`.
Maybe this can be achieved by defining a few high-level keywords that other recipes can't use like `scene` or `axis` / `axes` that route their named-tuple arguments separately from the rest.
Similar to Makie's nested attributes?

gha_language: Julia


text:
Changes Github's font to Helvetica.

Author: [Anthony Nguyen](http://www.antnguyen.com)

Inspiration: [John Michel's Optimize Legibility Addon](https://github.com/johnmichel/Optimize-Legibility-for-Chrome)


text:
# Security Package Delivery Test Case: Access OTA Blob as Executable

When a Fuchsia system downloads an OTA ("Over The Air" system update), the
contents of the update is not covered by the system's chain of verification
until the device reboots. This test simulates an OTA and attempts to use package
delivery APIs to map a blob from the OTA as executable. This procedure should be
prevented by the package delivery system to ensure
[verified execution](/docs/concepts/security/verified_execution.md).

## Test Composition

Detailed component composition is articulated in `.cml` files in the `meta/`
subdirectory. In general, the test brings up the `hello_world_v0` system
assembly in an isolated `fshost`, initiates an update to `hello_world_v1` via
an isolated `netstack` and `pkg_server`, then attempts to map the new version of
the "Hello, World!" executable as executable.

repository: gnoliyil/fuchsia

repo_name: gnoliyil/fuchsia


text:
# Ruby Capstone Project: Random-Movie-Twitter-Bot
A Twitter bot that recommends a movie based on the genre that the user tweets

> This project consists of building a twitter bot that replies to the user with a movie recommendation based on the genre of the movie that the user tweets.

![screenshot](./assets/TweetBot.gif)

# This project delivers on the following basic specifications
- When locally executed, the program checks if the user twitter handler has been mentioned and checks the user's reply_id to see if this tweet has been already answered.
- Reads the tweet text and run the helper and genres method to check if there is mention to any valid movie genres, contained in The Movie Database.
- If the tweet text contained a valid genre, the bot searches the Movie Database for a random page of the requested genre and selects a random movie from the page.
- The bot then gets the title and the ID of the selected movie.
- Finally the title movie and the link with the correspondent ID is replied to the user.

## Built With

- Ruby
- Stickler
- Rubocop
- Rspec
- VS Code
- Chatterbox

## Getting Started

To get a local copy up and running follow these steps.

### Prerequisites
To deploy a fully functional local copy, you must install the following dependencies/libraries:
- [Chatterbox](https://github.com/muffinista/chatterbot) Wrapper used to interact with the twitter API
- [The Movie DB](https://github.com/ahmetabdi/themoviedb/) Wrapper used to interact with the TMDB API
- [Dotenv](https://github.com/bkeepers/dotenv) Used for handling environmental variables

### Setup
- Clone this repository in your local environment
- Located on the root of repository execute

 ```bundle install```

 This will install all the required dependencies.

 See the details below to modify and include all your personal  keys and values in the .yml and .env files

## Adding Keys

### TMDB Key

- Inside the ```.env``` file on the root of your project add your own TMDB API key. You can request one here [TMDB API](https://www.themoviedb.org/documentation/api)

```ruby

export TMBD_KEY = WRITE YOUR TMBD API KEY HERE

```

### Twitter keys

- At the ```.RandomMovieBot1.yml``` file, located in the bin folder you have to add your own twitter development consumer/access key/secret/tokens from your twitter developer account.

You can request one here:

[Link for Twitter development account creation](https://developer.twitter.com/en/apply-for-access)

```ruby
:consumer_key: --WRITE YOUR TWITTER CONSUMER_KEY HERE--
:consumer_secret: --WRITE YOUR TWITTER CONSUMER_SECRET HERE--
:access_token: --WRITE YOUR TWITTER ACCESS_TOKEN HERE--
:access_token_secret: --WRITE YOUR TWITTER ACCESS_TOKEN_SECRET HERE--
```

## Usage
- To start using the program, go to any twitter account and tweet ```@RandomMovieBot1 -- text asking for a movie of X genre --```
for example you can tweet ```@RandomMovieBot1 I want to watch an action movie```
- Then using your terminal navigate to your project folder and execute the bot with the ```./bin/RandomMovieBot1.rb``` command
- You should see in the terminal output the twitter handler of the user the bot is replying, the title of the movie and the link to the movie database film page

![screenshot](./assets/screenshot1.png)

- Now go to the twitter account who twitted and you should see the original tweet has been replied with a movie suggestion of the requested genre

![screenshot](./assets/screenshot2.png)

## RSpec Testing

The code can be tested using RSpec. To run the tests first make sure you have RSpec installed on your local machine. To verify it open your terminal and type:

```rspec --version```

If it is not installed, you can do it running the following command:

```gem install rspec```

To run the tests, navigate to the root file of the project and run:

```rspec```

 The tests output should be like this:

![screenshot](./assets/screenshot3.png)

If you prefer to run your own tests, feel free to modify these files from the ```./spec``` folder, according to your criteria:

```genres_spec.rb
   helper_methods_spec.rb
   movies_spec.rb
```

## Deployment

For further testing purposes, this bot could be deployed in a server using Heroku.

## Author

👤 **Fernando Bahamondes**

- Github: [@ferbaco86](https://github.com/ferbaco86)
- Twitter: [@ferbac0](https://twitter.com/ferbac0)
- Linkedin: [Fernando Bahamondes](https://www.linkedin.com/in/fernando-bahamondes-correa)

## 🤝 Contributing

Contributions, issues and feature requests are welcome!

Feel free to check the [issues page](https://github.com/ferbaco86/Random-Movie-Twitter-Bot/issues).

## Show your support

Give a ⭐ if you like this project!

## 📝 License

This project is [MIT](https://github.com/ferbaco86/Random-Movie-Twitter-Bot/blob/twitter-bot/LICENSE) licensed.

branch_name: refs/heads/development


text:
+++
title = "Research Talk at IBCAS Youth Forum"
subtitle = "Build and use large-scale phylogenetic trees"

date = 2019-09-25T00:00:00
draft = false

# Authors. Comma separated list, e.g. `["Bob Smith", "David Jones"]`.
authors = ["Miao Sun"]

tags = ["Dimensions", "Tree of Life", "Conservation", "Biodiversity", "Phylogeny", "rosids"]
summary = "Research talk to introduce my 5-year post-doc research to students, and researchers at Institute of Botany, Chinese Academy of Sciences"

# Projects (optional).
#   Associate this post with one or more of your projects.
#   Simply enter your project's folder or file name without extension.
#   E.g. `projects = ["deep-learning"]` references
#   `content/project/deep-learning/index.md`.
#   Otherwise, set `projects = []`.
 projects = ["tree_of_life", "dimensions", "rosids"]

# Featured image
# To use, add an image named `featured.jpg/png` to your project's folder.
[image]
  # Caption (optional)
  caption = "Talk Cover"

  # Focal point (optional)
  # Options: Smart, Center, TopLeft, Top, TopRight, Left, Right, BottomLeft, Bottom, BottomRight
  focal_point = ""

  # Show image only in page previews?
  preview_only = true

+++

{{< figure src="intro.png" title="" numbered="true" >}}

I had the great honor and was invited by Dr. [Zhiduan Chen](http://sourcedb.ib.cas.cn/cn/expert/200904/t20090403_45016.html) to visit his lab. We had great conversations and we also discussed to collaborate a few great project in comming futher.

Later on I was invited by Dr. [Limin Lu](http://www.lseb.cn/lulimin), who hosted a Youth Forum organized by [the State Key Laboratory of Systematic and Evolutionary Botany (LSEB)](http://www.lseb.cn/Enindex.aspx) at IBCAS.

I briedly introduced my research works conducted in these 5-year post-doc life in [Soltis Lab](https://www.floridamuseum.ufl.edu/museum-voices/soltis-lab/people/principal-investigators/). I used [rosids]({{< ref "/project/rosids/index.md" >}}) as an example to illustrate how to build and use large-scale phylogentic trees. Key points are:

  - Data mining public database (e.g, GenBank and [GBIF](https://www.gbif.org/)), and data cleaning for downstream analyses. E.g, recent manuscript in [bioRxiv](https://doi.org/10.1101/694950) and paper in [AJB]({{< ref "/publication/folk2018challenges/index.md" >}}).

  - Knowing the potential issues when applying large-scale data for macro-ecology and -evolution. E.g., papers in [AJB]({{< ref "/publication/folk2018challenges/index.md" >}}), [Ecology]({{< ref "/publication/Lietal2019/index.md" >}}), and another recent manuscript in [bioRxiv](https://doi.org/10.1101/749325V1).

  - Large-scale trees can be used for [Diversification]. E.g., recent manuscript in [bioRxiv](https://doi.org/10.1101/749325V1) and an accepted paper collaborated with [Dr. Xue](https://www.biorxiv.org/content/10.1101/652065v1).

  - Large-scale trees can be used for [Phylogentic Diversity (PD)]({{< ref "/publication/lu2018evolutionary/index.md" >}})

  **Relationships matter!!**

  Please let me know if you have any questions and want to know more details.

Overall, it turned out full house!! More than expected!! :v: :v:

Especially I have met Prof. Jianfeng Mao from Beijing Forestry University. I had great cinversation with him and his students.
{{< figure src="Wechat1.jpg" title="" numbered="true" >}}

repository: Cactusolo/CactusOasis

repo_name: Cactusolo/CactusOasis


text:
---
id: publications
name: Research
heading: Selected Publications
---

[Asymmetric Network Connectedness of Fears](https://ideas.repec.org/p/arx/papers/1810.12022.html){:target="blank"} (with M.Bevilacqua, and R.Tunaru)

**The Review of Economics and Statistics** (2022), 104(6): 1-13

replication codes and data

[Quantile Spectral Beta: A Tale of Tail Risks, Investment Horizons, and Asset Prices](https://ideas.repec.org/p/arx/papers/1806.06148.html){:target="blank"} (with M.Nevrla)

**Journal of Financial Econometrics** (2022), forthcoming

replication codes

[Measurement of Common Risks in Tails: A Panel Quantile Regression Model for Financial Returns](https://ideas.repec.org/p/arx/papers/1708.08622.html){:target="blank"} (with F.Cech)

**Journal of Financial Markets** (2021),  52, 100562,

replication codes and data

[Forecasting dynamic return distributions based on ordered binary choice](https://ideas.repec.org/p/arx/papers/1711.05681.html){:target="blank"} (with S.Anatolyev)

**International Journal of Forecasting** (2019), 35(3), pp.823-835,

code and package

 [Quantile Coherency: A General Measure for Dependence between Cyclical Economic Variables](https://ideas.repec.org/p/arx/papers/1510.06946.html){:target="blank"} (with T.Kley)

**The Econometrics Journal** (2019), 22(2), pp. 131-152,

code and package

Supplementary material [available](assets/files/20181130_quantile_cross_EctJ_supp.pdf){:target="blank"}

[Total, asymmetric and frequency connectedness between oil and forex markets](https://ideas.repec.org/p/arx/papers/1805.03980.html){:target="blank"} (with E. Kocenda)

**The Energy Journal** (2019),vol 40, pp. 157 - 174,

codes for introduced measures

Note the package *frequencyConnectedness* available here can be used to replicate the paper

Supplementary material [available](assets/files/2019_EJ_Kocendasupp.pdf){:target="blank"}

[Measuring the Frequency Dynamics of Financial Connectedness and Systemic Risk](https://ideas.repec.org/p/arx/papers/1507.01729.html){:target="blank"} (with T.Krehlik)

**Journal of Financial Econometrics** (2018), 16 (2), pp. 271–296,

code and package

Supplementary material [available](assets/files/2018_JFEC_Krehlik_suppl.pdf){:target="blank"}

[Do co-jumps impact correlations in currency markets?](https://ideas.repec.org/p/arx/papers/1602.05489.html){:target="blank"} (with L.Vacha)

**Journal of Financial Markets** (2018), 37, pp.97-119,

codes for introduced measures

[Modeling and Forecasting Persistent Financial Durations](http://ideas.repec.org/p/arx/papers/1208.3087.html){:target="blank"} (with F.Zikes and N.Shenai)

**Econometric Reviews** (2017), 36:10, 1081-1110,

Supplementary material [available](assets/files/msmd_appendix_er_2.pdf){:target="blank"}

[Estimation of financial agent-based models with simulated maximum likelihood](https://ideas.repec.org/p/fau/wpaper/wp2016_07.html){:target="blank"} (with J.Kukacka)

**Journal of Economic Dynamics and Control** (2017), 85, pp. 21-45,

[Asymmetric volatility connectedness on forex markets](https://ideas.repec.org/p/arx/papers/1607.08214.html){:target="blank"} (with E. Kocenda and L.Vacha)

**Journal of International Money and Finance** (2017), 77C, pp. 39-56,

code for replication

[Cyclical properties of supply-side and demand-side shocks in oil-based commodity markets](https://ideas.repec.org/p/arx/papers/1603.07020.html){:target="blank"} (with T.Krehlik)

**Energy Economics** (2017), 65, pp.208-218,

code and package
codes for bootstrap

[Good volatility, bad volatility: Which drives the asymmetric connectedness of Australian electricity markets?](https://ideas.repec.org/a/eee/eneeco/v66y2017icp108-115.html){:target="blank"} (with N. Apergis and M.Lau)

**Energy Economics** (2017), 66, pp.108- 115,

codes for introduced measures

Note the package *frequencyConnectedness* available here can be used to replicate the paper

[Asymmetric connectedness on the U.S. stock market: Bad and good volatility spillovers](https://ideas.repec.org/p/arx/papers/1308.1221.html){:target="blank"} (with E. Kocenda and L.Vacha)

**Journal of Financial Markets** (2016), 27, 55–78,

codes for introduced measures

Note the package *frequencyConnectedness* available here can be used to replicate the paper

[Semiparametric Conditional Quantile Models for Financial Returns and Realized Volatility](https://ideas.repec.org/p/zbw/fmpwps/20.html){:target="blank"} (with F.Zikes)

**Journal of Financial Econometrics** (2016), 14 (1), 185–226,

Supplementary material [available](assets/files/2016_JFEC_Zikes_ONLINE_APPENDIX.pdf){:target="blank"}

[Forecasting the term structure of crude oil futures prices with neural networks](https://ideas.repec.org/p/arx/papers/1504.04819.html){:target="blank"} (with B.Malinska)

**Applied Energy** (2016), 164, pp.366–379,

[Modeling and forecasting exchange rate volatility in time-frequency domain](http://ideas.repec.org/p/arx/papers/1204.1452.html){:target="blank"} (with T.Krehlik and L.Vacha)

**European Journal of Operational Research** (2016), 251 (1), pp. 329–340,

codes to paper

[Volatility spillovers across petroleum markets](https://ideas.repec.org/p/wdi/papers/2015-1093.html){:target="blank"} (with E. Kocenda and L.Vacha)

**The Energy Journal** (2015), 36(3), 309-329

codes for introduced measures

Note the package *frequencyConnectedness* available here can be used to replicate the paper

[Are benefits from oil - stocks diversification gone? A new evidence from a dynamic copulas and high frequency data](http://www.sciencedirect.com/science/article/pii/S014098831500170X){:target="blank"} (with K.Avdulaj)

**Energy Economics** (2015), 51, pp. 31-44,

[Realizing stock market crashes: stochastic cusp catastrophe model of returns under time-varying volatility](https://ideas.repec.org/p/zbw/fmpwps/15.html){:target="blank"} (with J.Kukacka)

**Quantitative Finance** (2015), 15 (8), pp. 1347-1364,

[Realized wavelet-based estimation of integrated variance and jumps in the presence of noise](https://ideas.repec.org/p/arx/papers/1202.1854.html){:target="blank"} (with L.Vacha)

**Quantitative Finance** (2015), 15 (6), pp. 959-973,

codes for introduced measures

[Are Bayesian Fan Charts Useful? The Eﬀect of Zero Lower Bound and Evaluation of Financial Stability Stress Tests](https://ideas.repec.org/p/cnb/wpaper/2011-10.html){:target="blank"} (with M.Franta, R.Horvath,K.Smidkova)

**International Journal of Central Banking** (2014), 10(1), 159-187

[Comovement of energy commodities revisited: Evidence from wavelet coherence analysis](https://ideas.repec.org/p/arx/papers/1201.4776.html){:target="blank"} (with L.Vacha)

**Energy Economics** (2012), 34(1), pp. 241–247,

[Can a stochastic cusp catastrophe model explain stock market crashes?](https://ideas.repec.org/a/eee/dyncon/v33y2009i10p1824-1836.html){:target="blank"} (with M.Vosvrda)

**Journal of Economic Dynamics and Control** (2012), 33(10), 1824-1836

Full list of publications [available here](http://ies.fsv.cuni.cz/sci/publication/user/id/215/lang/en){:target="blank"}

### Working Papers

[The Dynamic Persistence of Economic Shocks](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4467110){:target="blank"} (with L.Vacha)

preprint draft (June 2023)

**Abstract**: This paper presents a model for smoothly varying heterogeneous persistence of economic data. We argue that such dynamics arise naturally from the dynamic nature of economic shocks with various degree of persistence. The identification of such dynamics from data is done using localised regressions. Empirically, we identify rich persistence structures that change smoothly over time in two important data sets: inflation, which plays a key role in policy formulation, and stock volatility, which is crucial for risk and market analysis.

code and package in Julia coming soon

[Common Idiosyncratic Quantile Risk](https://arxiv.org/abs/2208.14267){:target="blank"} (with M.Nevrla)

preprint draft (May 2023)

Revise and resubmit, **Review of Finance**

**Abstract**: We identify a new type of risk that is characterised by commonalities in the quantiles of the cross-sectional distribution of asset returns. Our newly proposed quantile risk factor is associated with a quantile-specific risk premium and provides new insights into how upside and downside risks are priced by investors. In contrast to the previous literature, we recover the common structure in cross-sectional quantiles without making confounding assumptions or aggregating potentially non-linear information. We dis- cuss how the new quantile-based risk factor differs from popular volatility and downside risk factors, and we identify where the quantile-dependent risks deserve greater compensation. Quantile factors also have predictive power for aggregate market returns.

[Currency Network Risk](https://ideas.repec.org/p/arx/papers/2101.09738.html){:target="blank"} (with M.Babiak)

preprint draft (May 2023)

**Presented at** at 2023 WFA (San Francisco), 2023 SGF Conference (Zurich), 2023 EEA-ESEM (Barcelona)

**Abstract**: This paper identifies a new currency risk stemming from linkages between option-implied currency volatilities. A volatility network strategy that buys net recipients and sells net transmitters of transitory shocks to ex-ante currency volatilities generates significant excess returns. Net recipients are more exposed to volatility spillovers and compensate investors with higher average returns. In contrast, net transmitters are more resilient to volatility transmissions and offer a lower risk premium because they hedge against volatility interdependencies. When volatility linkages are controlled for contemporaneous correlations, the strategy is uncorrelated with popular benchmarks. The volatility network factor is also priced in a currency cross-section.

The video below illustrates the short-term currency network during the Global Financial Crisis. If the video does not play properly, feel free to watch it [here](https://www.google.com/url?q=https%3A%2F%2Fwww.dropbox.com%2Fs%2Fbg6uk6prpodwnuj%2FCurrency%2520Network.mp4%3Fdl%3D0&sa=D&sntz=1&usg=AFQjCNEc2H1o7Yen4A_u-iPJZ7qFZOy1PQ).

  <video controls="true" poster="assets/files/currency_net_2.pdf">
    <source src="https://dl.dropboxusercontent.com/s/bg6uk6prpodwnuj/Currency%20Network.mp4?dl=0" type="video/mp4">
  </video>

[Learning Probability Distributions in Macroeconomics and Finance](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4083719){:target="blank"} (with L.Hanus)

preprint draft (Apr 2022)

**Abstract**: We propose a deep learning approach to probabilistic forecasting of macroeconomic and financial time series. Being able to learn complex patterns from a data rich environment, our approach is useful for a decision making that depends on uncertainty of large number of economic outcomes. Specifically, it is informative to agents facing asymmetric dependence of their loss on outcomes from possibly non-Gaussian and non-linear variables. We show the usefulness of the proposed approach on the two distinct datasets where a machine learns the pattern from data. First, we construct macroeconomic fan charts that reflect information from high-dimensional data set. Second, we illustrate gains in prediction of stock return distributions which are heavy tailed, asymmetric and suffer from low signal-to-noise ratio.

code and package in Julia coming soon

[Persistence in Economic Networks](https://ideas.repec.org/p/arx/papers/2007.07842.html){:target="blank"} (with M.Ellington)

preprint draft (Oct 2021)

Revise and resubmit, **European Journal of Operational Research**

**Abstract**: This paper studies heterogeneous network structures driven by different degrees of persistence in economic connections. Using frequency domain techniques, we introduce measures that characterize network connections stemming from transitory and persistent components of shocks. Our approach permits testing for statistical differences in such connections that evolve over time. We estimate uncertainty networks from the main US sectors and argue that they track transitory and persistent sources of systemic risk. Hence they may serve as a useful tool for macro-prudential supervisors and investors alike.

code and package in Julia  and Matlab

[Dynamic industry uncertainty networks and the business cycle](https://ideas.repec.org/p/arx/papers/2101.06957.html){:target="blank"} (with M.Bevilacqua and R.Faff)

preprint draft (March 2021)

Revise and resubmit, **Journal of Economic Dynamics and Control**

**Abstract**: We argue that uncertainty network structures extracted from option prices contain valuable information for business cycles. Classifying U.S. industries according to their contribution to system-related uncertainty across business cycles, we uncover an uncertainty hub role for the communications, industrials and information technology sectors, while shocks to materials, real estate and utilities do not create strong linkages in the network. Moreover, we find that this ex-ante network of uncertainty is a useful predictor of business cycles, especially when it is based on uncertainty hubs. The industry uncertainty network behaves counter-cyclically in that a tighter network tends to associate with future business cycle contractions.

[Deep Learning, Predictability, and Optimal Portfolio Returns](https://ideas.repec.org/p/arx/papers/2009.03394.html){:target="blank"} (with M.Babiak)

preprint draft (July 2021) available

**Abstract**: We study dynamic portfolio choice of a long-horizon investor who uses deep learning methods to predict equity returns when forming optimal portfolios. Our results show statistically and economically significant benefits from using deep learning to form optimal portfolios through certainty equivalent returns and Sharpe ratios. We demonstrate that a long-short-term-memory recurrent neural network, which excels in learning complex time-series dependencies, generates a superior performance among a variety of networks considered. Return predictability via deep learning generates substantially improved portfolio performance across different subsamples, particularly during recessionary periods. These gains are robust to including transaction costs, short-selling and borrowing constraints.

[Frequency-Dependent Higher Moment Risks](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3823002){:target="blank"} (with J.Kurka)

preprint draft (April 2021)

**Abstract**: Based on intraday data for a large cross-section of individual stocks and exchange traded funds, we show that short-term as well as long-term fluctuations of realized market and average idiosyncratic higher moments risks are priced in the cross-section of asset returns. Specifically, we find that market and average idiosyncratic volatility and kurtosis are significantly priced by investors mainly in the long-run even if controlled by market moments and other factors, while skewness is mostly short-run phenomenon. A conditional pricing model capturing the time-variation of moments confirms downward-sloping term structure of skewness risk and upward-sloping term structure of kurtosis risk, moreover the term structures connected to market skewness risk and average idiosyncratic skewness risk exhibit different dymanics.

[Co-jumping of Treasury Yield Curve Rates](https://ideas.repec.org/p/arx/papers/1905.01541.html){:target="blank"} (with P.Fiser)

Submitted (2019)

**Abstract**: We study the role of co-jumps in the interest rate futures markets. To disentangle continuous part of quadratic covariation from co-jumps, we localize the co-jumps precisely through wavelet coefficients and identify statistically significant ones. Using high frequency data about U.S. and European yield curves we quantify the effect of co-jumps on their correlation structure. Empirical findings reveal much stronger co-jumping behavior of the U.S. yield curves in comparison to the European one. Further, we connect co-jumping behavior to the monetary policy announcements, and study effect of 103 FOMC and 119 ECB announcements on the identified co-jumps during the period from January 2007 to December 2017.

Codes for introduced measures

### Unfinished

[Dynamic Network Risk](https://ideas.repec.org/p/arx/papers/2006.04639.html){:target="blank"} (with M.Ellington)

submitted (2020)

**Abstract**: This paper examines the pricing of short-term and long-term dynamic network risk in the cross-section of stock returns. Stocks with high sensitivities to dynamic network risk earn lower returns. We rationalize our finding with economic theory that allows the stochastic discount factor to load on network risk through the precautionary savings channel. A one-standard deviation increase in long-term (short-term) network risk loadings associate with a 7.66% (6.71%) drop in annualized expected returns.

code and package in Julia  and Matlab

[Sentiment-Driven Stochastic Volatility Model: A High-Frequency Textual Tool for Economists](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3397314){:target="blank"} (with Cathy Yi-Hsuan Chen, and J.Vecer)

Submitted (2019)

**Abstract**: We propose how to quantify high-frequency market sentiment using high-frequency news from NASDAQ news platform and support vector machine classifiers. News arrive at markets randomly and the resulting news sentiment behaves like a stochastic process. To characterize the joint evolution of sentiment, price, and volatility, we introduce a unified continuous-time sentiment-driven stochastic volatility model. We provide closed-form formulas for moments of the volatility and news sentiment processes and study the news impact. Further, we implement a simulation-based method to calibrate the parameters. Empirically, we document that news sentiment raises the threshold of volatility reversion, sustaining high market volatility.

codes to paper

Tales of sentiment driven tails (with W.Hardle and C.Chen) [slides available](https://www.wiwi.hu-berlin.de/de/forschung/irtg/lvb/members/personalpages/wh/talks/20181124-bar-che-hae-tale-of-tail-events.pdf)

[Investment Disputes and Abnormal Volatility of Stocks](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3630279){:target="blank"} (with Z.Drabek and M.Nevrla)

Submitted (2020)

**Abstract**: Dramatic growth of investment disputes between foreign investors and host states rises serious questions about the impact of those disputes on investors. This paper is the first to explain increased uncertainty of investors about the outcome of arbitration, which may or may not lead to compensation for damages claimed by the investor. We find robust evidence that investment disputes lead to abnormal share fluctuations of companies involved in disputes with host countries. Importantly, while a positive outcome for an investor decreases uncertainty back to original levels, we document strong increase in the volatility of companies with negative outcome for the investor. We find that several variables including size of the award, political instability, location of arbitration, country of origin of investor or public policy considerations in host country explain large portion of the investor's uncertainty.

gha_language: CSS


text:
# Component ID index

## Terminology

[Moniker](/docs/glossary?style=box#moniker)
[Component instance identifier](/docs/glossary?style=box#component-instance-identifier)

## Scope

This document describes how to define an index that maps instance ids to
monikers. This mapping matters for components that use isolated storage
capabilities.

## Overview

The goal of an index of component instance IDs is to assign stable identifiers
to component instances. This is done by mapping an instance ID to a moniker.
When a component instance is assigned an ID, its persistent resources are
identified on disk using this instance ID. This allows the component's URL or
realm to be changed while its resources still remain attributed to it, so long
as this index is also updated.

When the component runtime discovers an instance ID -> moniker mapping, it
automatically moves the component instance's existing storage directory to be
keyed under its instance ID.

Only components that use storage capabilities must to be included in the
index. Test components should not be included in the index.

## Define a new index

An index file is a JSON5 formatted file, mapping a component's instance ID to
its moniker. There may be multiple index files in a build tree, but they will
be merged together into a single index file, and this merged file will be made
available to the component runtime. This merged index file is immutable, and
can only be updated through another system update.

The schema for an index file is described in the following example:

```json5
// Index files are written in JSON5, so you may use comments!
{
  // A list of entries, where each entry maps an instance ID to a moniker.
  instances: [
    // An entry, mapping an instance ID to a moniker.
    {
      // Instance IDs are randomly generated, 256-bits of base-16 encoded
      // strings (in lower case). To generate a new instance ID, omit this
      // field and run the build; the build will fail and suggest a new
      // instance ID which you can copy-paste here.
      instance_id: "2bd6cc2bd10243354b873a4ddb8a188b1d29171e26eebac06567bcdc36614af6",
      // The `instance_id` above is associated to the following moniker:
      moniker: "/core/account/credential_manager",
    },

    // More than one entry can be included. However, all entries must be distinct:
    // * Two entries cannot reference the same `instance_id`
    // * Two entries cannot reference the same `moniker`
    {
      instance_id: "7db7e88479772e241229682b47f1794e12ac5d692f8d67421acd9d7ff318a975",
      moniker: "/core/account/password_authenticator",
    }
  ]
}
```

To supply an index file to the build, use the
[component_id_index()](/build/component/component_id_index.gni) GN template:

```gn
component_id_index("my_component_id_index") {
  source = "my_component_id_index.json5"
}
```

## Add a component to the index {#add-to-index}

### Locate the appropriate index file

In order to add a component to the index, you must insert an entry into the
appropriate index file. Currently, `fuchsia.git`'s components are listed in the
[core_component_id_index.json5](/src/sys/core/core_component_id_index.json5)
index file.

{% dynamic if user.is_googler %}

Note: Outside of fuchsia.git, you can usually find a *component_id_index.json5
file in the `bundles/config` directory of a specific `//vendor` repository hosting
a product's configuration.

{% dynamic endif %}

### Add an entry to the index

The first step is to determine the component instance's moniker. You can find the
moniker of a component on a particular product's eng build by running `ffx component show`.

Then, append an entry to the `instances` list with the component's moniker.
Omit the `instance_id` field to have the build fail and suggest a new one you
can use.

#### Example

In this example, the component
`fuchsia-pkg://example.com/my_other_package#meta/my_other_component.cm` is added to the index.

To determine the component instance's moniker, you can run
`ffx component show fuchsia-pkg://example.com/my_other_package#meta/my_other_component.cm`:

```shell
$ ffx component show fuchsia-pkg://example.com/my_other_package#meta/my_other_component.cm
               Moniker:  /core/my_other_component
                   URL:  fuchsia-pkg://example.com/my_other_package#meta/my_other_component.cm
                      ...
```

The above output shows us that the moniker of this instance is `/core/my_other_component`

Add `/core/my_other_component` to the index by appending this entry to
[core_component_id_index.json5](/src/sys/core/core_component_id_index.json5)'s
`instances` list:

```json5
  {
    moniker: "/core/my_other_component"
  }
```

Now run the build.  The build should fail, suggesting a new instance ID:

```bash
$ fx build
.
.
Error: Could not merge index file ../../src/sys/core/core_component_id_index.json5

Caused by:
    Some entries are missing `instance_id` fields. Here are some generated IDs for you:
[
  {
    instance_id: "47c3bf08f3e560c4dee659c28fa8d863dbdc0b1dbb74065e6cb1f38441ac759c",
    moniker: "/core/my_other_component"
  }
]
```

Update the entry you've added by copying the suggested `instance_id` field. The
build should now pass.

## Include a Component ID Index in a system assembly {#system-assembly}

_The target audience for this section are product owners who are setting up a
system assembly_

This section describes how to include the component ID index in a system
assembly.

A system assembly should include a component ID index if it contains components
which use isolated storage. Any product that builds on top of the `core`
product already includes a component ID index in its assembly, so the following
instructions may not be necessary.

All `component_id_index()`s in a system build are merged together using the
`component_id_index_config()` template. This template is currently used in
`assembled_system.gni`, and assembly will fail if you define your own alongside
the one from `assembled_system.gni`.

### Steps

1. Define any `component_id_index()`s you want included in the system.
1. Add these targets as dependencies of `base_packages` in your `assembled_system()`
target.


text:
---
title: Synchronization Library
layout: docs
sidenav: side-nav-cpp.html
type: markdown
---

The Synchronization library includes abstractions and primitives for managing
tasks across different threads. This library encompasses the following
header files:

* `mutex.h`

  Provides primitives for managing locks on resources. A mutex is the most
  important primitive in this library and the building block for most all
  concurrency utilities.
* `notification.h`

  Provides a simple mechanism for notifying threads of events.
* `barrier.h` and `blocking_counter.h`

  Provides synchronization abstractions for cumulative events.

The Abseil `base` library also includes a number of concurrency-related header
files:

* `base/thread_annotations.h`

  Provides macros for documenting the locking policy of multi-threaded code, and
  providing warnings and errors for misuse of such locks.
* `base/call_once.h`

  Provides an Abseil version of `std::call_once()` for invoking a callable
  object exactly once across all threads.

This document will cover all of the above.

## Synchronization Overview

In sequential (i.e. single-threaded) systems, we usually think of events as
happening in a specific total order: for any operations A and B, either A
happens before B, or B happens before A. In concurrent systems, this is no
longer the case: for some pairs of operations it may not be possible to say
which one happens earlier (i.e. events are only partially ordered), in which
case we say that they happen *concurrently*. Notice that this definition has
nothing to do with whether they "actually" happen simultaneously, but only with
whether we can *guarantee* that they won't.

Concurrent operations may conflict if they are not used (or designed) properly
within a multi-threaded environment, resulting in the following problems:

* Operations may require write access to shared resources. We call these issues
  *memory access* issues.
* Operations may need to occur in a specific order. We sometimes call these
  issues *synchronization* issues (although memory access issues are also
  synchronization issues).

In either case, lack of control on the shared resources or lack of control on
the operation order can lead to *race conditions*. The purpose of the
concurrency abstractions within this library is to address these issues and
avoid such race conditions.

### Memory Access Issues

Memory access issues are often addressed through a variety of means, including:

  * Making the shared resource private or read-only (for data where this is
    appropriate)
  * Converting the data access into a "message passing" scheme, to provide
    copies of the shared information for temporary use rather than direct
    access to the memory.
  * Locking access to the shared resource, typically for write operations, to
    prevent more than one user from reading or writing concurrently.
  * Using atomic operations to access to the shared resource, such as those
    provided by `std::atomic`. Note that the rules for properly applying atomic
    operations are quite complicated, which is one of many reasons you should
    avoid atomics.

Locking access to shared resources is usually addressed through
mutually-exclusive locks known as *mutexes*. Abseil provides its own `Mutex`
class for this purpose; similarly, the C++ standard library provides a
`std::mutex` class for the same purpose. (Reasons why we implement our own
`Mutex` class are discussed in [Mutex Design Notes](/about/design/mutex).)

Types that behave correctly regardless of the order, scheduling,
or interleaving of their operations are known as *thread-safe*. In most cases,
such types use mutexes and atomic operations underneath the hood
to guard access to the object's internal state.

See [Mutexes](#mutexes) below for more information.

### Synchronization Operations

Synchronization issues other than simple memory access issues are often more
complex and require abstractions specifically built to address the underlying
problem, (again, often through mutexes and atomic operations, which are quite
tricky to implement properly). Synchronization operations are designed to
control the order of events in different threads.

Keep in mind that operations on "thread-safe" types aren't necessarily
synchronization operations. When you read a value that another thread wrote, you
can't assume that the write happens before the read; they may happen
concurrently.

For example:

```cpp
foo::counter first, second;

void thread1() {
  first.Add(1);    // (a)
  second.Add(1);   // (b)
}

void thread2() {
  while (second.value() == 0) {
    sleep(10);
  }
  CHECK(first.value() == 1);   // ERROR
}
```

Even if foo::counter were thread-safe (and you wouldn't need to worry about data
races), you might think that the `CHECK()` will succeed, because line (a)
happens before line (b), and the `CHECK()` can't be reached until line (b) has
executed. However, unless `Add()` and `value()` are also synchronization
operations, none of the operations in thread1 need necessarily happen before any
of the operations in thread2, and that `CHECK()` may fail.

Abseil provides several synchronization abstractions. See
[Synchronization Operations](#other-synchronization-operations) for more
information.

## Mutexes

The major primitive for use within concurrency tasks is a mutex, which is a
*mut*ually *ex*clusive lock that can be used to prevent multiple threads from
accessing and/or writing to a shared resource.

### `absl::Mutex` and `std::mutex`

Abseil provides its own `Mutex` class, and within Google, we use this class
instead of the similar `std::mutex`. `Mutex` provides most of the functionality
of `std::mutex` but adds the following additional features:

* `absl::Mutex` adds *conditional critical sections*, an alternative to
  condition variables. `Mutex::Await()` and `Mutex::LockWhen()` allow the client
  to wait for a condition without needing a condition variable; the client need
  not write the while-loop, nor need they use `Signal()`. (See
  [Condition](#conditional-critical-sections) below.)
* `Mutex`  intrinsically supports deadlock detection (when locks are not
  acquired in a consistent order). The deadlock detector is enabled by default
  in most non-opt build modes, and it can detect deadlock risks that even Clang's
  [Thread Sanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html) misses.
  (See [Deadlock Detection](#deadlock-detection) below.)

Additionally, `absl::Mutex` can act as a reader-writer lock (like
`std::shared_mutex`) with special `ReaderLock()` and `ReaderUnlock()` functions.
(See [Reader-Writer Locks](#reader-writer-locks) below.)

We've found these features to be critically important for maintaining a large
and complex code base. We are not necessarily intending to compete with
`std::mutex` itself; if you find the features usable within your code base,
consider `absl::Mutex` and its utilities.

Like `std::mutex`, Abseil's Mutex is not re-entrant (also known as
non-recursive). As well, it does not provide strict FIFO behaviour or fairness
in the short term; to do so would require significant overhead. However, it
tends to be approximately fair in the long term.

### Mutex Basics

The `absl::Mutex` class implements a mutually exclusive lock on some
resource, allowing threads which also use the mutex to avoid concurrent access
to that resource, which is typically some variable or data structure with
associated invariants. For example, a financial transaction system may wish only
one writer to access certain data elements at one time. Mutexes are so common
that many words have been coined to describe their operation.

Each `Mutex` has two basic operations: `Mutex::Lock()` and `Mutex::Unlock()`.
Conceptually, it has just a single bit of abstract state: a boolean indicating
it is `true` (locked) or `false` (unlocked). When a `Mutex` is created, this
lock is initially `false` and the mutex is said to be *free* or *unlocked*.
`Lock()` blocks the caller until some moment when the mutex is free, and then
atomically changes this mutex state from `false` to `true`; the mutex is then
said to be *held* or *locked* by the caller. `Unlock()` sets this mutex state to
`false` once more.

Calling `Lock()` is often referred to as *locking* or *acquiring* a mutex, while
calling `Unlock()` is referred to as *unlocking* or *releasing* a mutex. An
action performed by a thread while holding a mutex is said to be performed
*under* the mutex. Data structures manipulated under the mutex, and their
invariants, are said to be *protected by* the mutex.

Clients of `Mutex` must obey these rules:

1.  Each time a thread acquires a `Mutex` it must later release it.
2.  A thread may not attempt to release a `Mutex` unless it holds it.
3.  A thread may not attempt to acquire an exclusive lock on a `Mutex` it
    already holds.

Because `Lock()` acts atomically to change the state of the state of the mutex,
we are guaranteed that, if these rules are followed, only one thread may hold a
mutex at any given time.

These rules must be followed both to prevent concurrent access to the protected
resource and avoid *deadlock*, in which a thread blocks waiting for a lock to be
released. The last rule prevents *self-deadlock*, when a holder of a mutex
attempts to acquire a mutex it already holds. Such mutexes are known as
non-recursive (or non-reentrant) mutexes.

These rules are best followed by bracketing regions of code with
matching calls to `Lock()` and `Unlock()` a mutex in the same procedure.
Such sections of code are called *critical regions*. Much Google C++ code uses
the helper class `MutexLock`, which through RAII automatically acquires a mutex
on construction and releases it when the lock goes out of scope.

### Mutexes and Invariants

Most mutexes are used to ensure some invariant state can only be changed
atomically while the mutex is held. The programmer is required to re-establish
the invariant before releasing the mutex; code can then assume the invariant
holds whenever acquiring the mutex, even in the face of updates that
*temporarily* invalidate the invariant during the critical section. However, one
cannot guarantee the invariant is true in a thread that does not hold the mutex,
because the mutex holder may be changing the monitored state at that moment.

For example, suppose `Mutex mu` protects the invariant that `a + b == 0`. This
code is then legal:

```cpp
mu.Lock();
assert(a + b == 0); // invariant assumed to hold
a++;                // invariant temporarily invalidated
b--;                // invariant restored before mu is released
mu.Unlock();
```

while this code is erroneous:

```cpp
mu.Lock();
assert(a + b == 0); // invariant assumed to hold
a++;                // invariant invalidated
mu.Unlock();        // BUG: mutex released while invariant invalid
mu.Lock();
b--;                // attempt to restore the invariant,
                    // but the damage is already done
mu.Unlock();
```

The following does not invalidate the invariant, but incorrectly assumes
it is true when the lock is not held:

```cpp
mu.Lock();
assert(a + b == 0); // correct: invariant assumed to hold
mu.Unlock();
assert(a + b == 0); // BUG: can't assume invariant without lock
```

The invariant holds only when evaluated on state observed within a single
critical section:

```cpp
mu.Lock();
assert(a + b == 0);    // correct: invariant assumed to hold
temp = a;              // takes a temporary copy of "a"
mu.Unlock();
mu.Lock();
assert(temp + b == 0); // BUG: can't assume invariant on state
                       // from two distinct critical sections
mu.Unlock();
```

### The `MutexLock` Wrapper

Forgetting to acquire and release locks on a `Mutex` often leads to errors in
your code. The Abseil concurrency library includes a `MutexLock` wrapper class
to make acquiring and releasing a `Mutex` easier. The class uses RAII to
acquire the mutex and automatically releases it when the class goes out of
scope.

Example:

```cpp
Class Foo {

  Foo::Bar* Baz() {
    MutexLock l(&lock_);
    ...
    return bar;
  }

  private:
    Mutex lock_;
  };
}
```

### Conditional Mutex Behavior

An Abseil `Mutex` can be configured to block threads until a certain condition
occurs. Such conditional behavior is accomplished in two ways: a traditional
*conditional variable* `CondVar` (similar to the `std::condition_variable`
available to `std::mutex`) or through a mechanism unique to Abseil's mutex:
*conditional critical sections*, using a `Mutex::Condition`.

Conditional Critical Sections (using the `Condition` construction) are generally
preferred as use of separate condition variables has proven to be error prone.
The `Mutex` contains a number of member functions (e.g. `Mutex::Await()`) that
are hard to get wrong. Generally, prefer use of `Condition`; in rare cases, when
there are multiple threads waiting on distinctly different conditions, however,
a battery of `CondVar`s may be more efficient.

#### Conditional Critical Sections

Abseil's `Mutex` has been extended through the addition of *conditional critical
sections*, an alternative to condition variables. Member functions such as
`Mutex::Await()` and `Mutex::LockWhen()` use intrinsic `Condition` predicates
to allow a client to wait for a condition without needing a condition variable;
the client need not write the while-loop, nor need they use `Signal()`.

Clients can imagine that a mutex contains an imaginary condition variable
`mu.cv`; with that assumption, these corresponding code fragments on the left
and right are equivalent:

      Conditional critical sections
      Condition variables

mu.Lock();
... // arbitrary code A
mu.Await(Condition(f, arg));

... // arbitrary code B
mu.Unlock();

mu.LockWhen(Condition(f, arg));

... // arbitrary code C
mu.Unlock();

mu.Lock();
... // arbitrary code A
while (!f(arg)) {
  mu.cv.Wait(&mu);
}
... // arbitrary code B
mu.Unlock();

mu.Lock();
while (!f(arg)) {
  mu.cv.Wait(&mu);
}
... // arbitrary code C
 mu.Unlock();

When `LockWhen()` and `Await()` are used, the condition must be encapsulated in
a function (`f` in the examples) with a `void *` argument (or by a lambda). As
with condition variables, the condition must be a function of state protected by
the mutex. The minor inconvenience of requiring a function is rewarded by
eliminating the need for the condition variable and the while-loop, which are
now hidden inside the `Mutex` implementation.

Even better, `Mutex::Unlock()` will take care of calling `Signal` or
`SignalAll()` to wake waiters whose conditions are now true; its performance is
usually as good as or better than that achieved with a condition variable. Thus,
the example of the previous subsection could be written as:

```cpp
bool f(bool *arg) { return *arg; }

// Waiter
mu.LockWhen(Condition(f, &cond_expr));
// cond_expr now holds
...
mu.Unlock();

// Waker
mu.Lock();
Make_cond_expr_True();
// cond_expr now holds
mu.Unlock();
```

In rare cases, when many threads may be waiting with many different and
usually-false conditions, it may be better to use multiple condition variables.
In general, we recommend conditional critical sections for simplicity.

The call `mu.LockWhen(Condition(f, arg))` is equivalent to
`mu.Lock(); mu.Await(Condition(f, arg))`.  Similarly, the call
`mu.Await(Condition(f, arg))` is equivalent to
`mu.Unlock(); mu.LockWhen(Condition(f, arg))`.

The variants `LockWhenWithTimeout()` and `AwaitWithTimeout()` allow a thread to
wait either for a condition to become true or for some time to elapse. They each
return `true` iff the condition is true:

```cpp
if (mu.LockWhenWithTimeout(Condition(f, &cond_expr), 1000 /*ms*/)) {
  // mu held; cond_expr true
} else {
  // mu held; cond_expr false; 1000ms timeout expired
}
mu.Unlock();
```

These calls are analogous to `cv.WaitWithTimeout()`.

#### `CondVar` Condition Variables

Condition variables serve the same purpose as conditional critical sections;
they are a means for blocking a thread until some condition has been satisfied.
Usually, conditional critical sections are easier to use, but condition
variables may be more familiar to programmers because they are included in the
POSIX standard and Java language.

Viewed in isolation, a condition variable allows threads to block and to be
woken by other threads. However, condition variables are designed to be used in
a specific way; a condition variable interacts with a mutex to make it easy to
wait for an arbitrary condition on state protected by the mutex.

Suppose that a thread is to wait for some boolean expression `cond_expr` to
become `true`, where the state associated with `cond_expr` is protected by a
mutex `mu`. The programmer would write:

```cpp
// Waiter
mu.Lock();
while (!cond_expr) {
  cv.Wait(&mu);
}
// cond_expr now holds
...
mu.Unlock();
```

The `Wait()` call atomically unlocks `mu` (which the thread must hold), and
blocks on the condition variable `cv`. When another thread signals the condition
variable, the thread will reacquire the `mu`, and go around the mandatory
while-loop to recheck `cond_expr`.

Another thread that makes cond_expr true might execute:

```cpp
// Waker
mu.Lock();
Make_cond_expr_True();
// cond_expr now holds
cv.Signal();
mu.Unlock();
```

The call to `Signal()` wakes at least one of the threads waiting on `cv`. Many
threads may be blocked on a condition variable at any given time; if it makes
sense to wake more than one such thread `SignalAll()` can be used. (The
`SignalAll()` functionality is often referred to as broadcast in other
implementations.)

A single condition variable can be used by threads waiting for different
conditions. However, in this case, `SignalAll()` must be used when any of the
conditions becomes `true`, because the `CondVar` implementation cannot otherwise
guarantee to wake the correct thread. It can be more efficient to use one
condition variable for each different condition; any number of condition
variables can be used with a single mutex.

Both `Signal()` and `SignalAll()` are efficient if there are no threads to wake.
Clients should call `Signal()` or `SignalAll()` inside the critical section that
makes the condition true.

The call `WaitWithTimeout()` allows a thread to wait until either a condition is
`true`, or until some time has elapsed. Like `Wait()`, `WaitWithTimeout()`
always reacquires the mutex before returning.

Example:

```cpp
static const int64 kMSToWait = 1000;  // we'll wait at most 1000ms
int64 ms_left_to_wait = kMSToWait;    // ms to wait at any given time
int64 deadline_ms = GetCurrentTimeMillis() + ms_left_to_wait;
mu.Lock();
while (!cond_expr && ms_left_to_wait > 0) {
  cv.WaitWithTimeout(&mu, ms_left_to_wait);
  ms_left_to_wait = deadline_ms - GetCurrentTimeMillis();
}
if (cond_expr) {
  // cond_expr true
} else {
  // cond_expr false; 1000ms timeout expired
}
mu.Unlock();
```

### Reader-Writer Locks

A reader-writer (shared-exclusive) lock has two locking modes. If the lock is
not free, it may be held either in write (a.k.a. exclusive) mode by a single
thread, or in read (a.k.a. shared) mode by one or more threads.

It is natural to use a reader-writer lock to protect a resource or data
structure that is read often, but modified infrequently. Critical sections that
modify the protected state must acquire the lock in write mode, while those that
merely read the state may acquire the lock in read mode.

Note: reader-writer locks can reduce lock contention in some situations, but
most locks are not contended enough for this to make a difference. When you use
shared locks, the onus is on you to ensure that the code in reader critical
sections really doesn't mutate the data (logically or physically), and any
mistakes here can lead to subtle race conditions.

Any `Mutex` can be used as a reader-writer lock.  The `Lock()` call acquires the
lock in write mode and must be paired with a call to `Unlock()`. The
`ReaderLock()` call acquires the lock in read mode and must be paired with a
call to `ReaderUnlock()`. `absl::Mutex` does not provide operations to convert
from a read lock to a write lock or vice versa without first releasing the lock.

Both condition variables and `mu.Await()` may be used with `Mutex` read-mode
critical sections as well as with write-mode critical sections.

`Mutex` does not allow readers to starve writers or vice versa. This leads to
the slightly surprising effect that a request for a read lock may block even if
the lock is already held by a reader. If this could not happen, a waiting writer
could be prevented ever from making progress by two or more readers that kept
the lock permanently in read mode, without any one holding the lock
indefinitely.

Beware that future maintainers may add mutations to "reader" critical sections
accidentally, thus introducing errors.  For example, self-optimizing data
structures such as splay trees or LRU caches may modify the data structure on
every read.  Even simple data structures may keep track of usage statistics.
Therefore, reader locks should be used with care, and their use should be easy
for developers to recognize. It can help to forbid modifications using a pointer
to a `const` data structure, though care must be still be exercised because of
C++'s `mutable` keyword.

We recommend using `Mutex` as a simple mutex initially; introduce reader locks
only when you know you have lock contention and you know that writes are
infrequent.

### Thread Annotations

The major drawbacks of `absl::Mutex` are drawbacks of any mutex type: you have
to remember to lock it before entering a critical section, you have to remember
to unlock it when you leave, and you have to avoid deadlock.

To help solve these problems, Abseil provides thread-safety annotations (in
`base/thread_annotations.h`) to specify which variables are guarded by which
mutexes, which mutexes should be held when calling which functions, what order
mutexes should be acquired in, and so forth. These constraints are then checked
at compile time, and although this mechanism is not foolproof, it does catch
many common `Mutex` usage errors. Besides being part of the code documentation,
annotations can also be used by the compiler or analysis tools to identify and
warn about potential thread safety issues.

#### Annotation Guidelines

Every data object (whether a global variable in namespace scope or a data member
in class scope) which requires protection from a mutex should have an annotation
`GUARDED_BY` indicating which mutex protects it:

```cpp
int accesses_ GUARDED_BY(mu_); // count of accesses
```

Every mutex should have a complementary comment indicating which variables and
also any non-obvious invariants it protects:

```cpp
Mutex mu_;       // protects accesses_, list_, count_
                 // invariant: count_ == number of elements in linked-list list_
```

Whenever a thread can hold two mutexes concurrently, either one (or both) of the
mutexes should be annotated with `ACQUIRED_BEFORE` or `ACQUIRED_AFTER` to
indicate which mutex must be acquired first:

```cpp
Mutex mu0_ ACQUIRED_BEFORE(mu1_); // protects foo_
```

If the mutex acquisition order is not consistent, deadlock may result. See
[Deadlock Detection](#deadlock-detection) for utilities within the Concurrency
library to detect deadlock.

Each routine should be annotated or have a comment indicating which mutexes must
and must not be held on entry. These annotations allow implementors to edit
routines without examining their call sites, and allows clients to use routines
without reading their bodies.

The annotations `EXCLUSIVE_LOCKS_REQUIRED`, `SHARED_LOCKS_REQUIRED`,
and `LOCKS_EXCLUDED` are used to document such information. Since we currently
use GCC's "attributes" to implement annotations, they can only be applied to the
function declarations, not the definitions (unless they are inside a class
definition).

If a routine acquires `mu`, we must annotate its declaration with
`LOCKS_EXCLUDED(mu)`:

```cpp
// Function declaration with an annotation
void CountAccesses() LOCKS_EXCLUDED(mu_);

// Function definition
void CountAccesses() {
  this->mu_.Lock();
  this->accesses_++;
  this->mu_.Unlock();
}
```

If a routine expects to be called with `mu` held, we must annotate it with
`EXCLUSIVE_LOCKS_REQUIRED(mu)` or `SHARED_LOCKS_REQUIRED(mu)` depending on
whether the routine needs a writer or reader lock:

```cpp
// Function declaration with an annotation
void CountAccessesUnlocked() EXCLUSIVE_LOCKS_REQUIRED(mu_);

// Function definition
void CountAccessesUnlocked() {
  this->accesses_++;
}
```

Without such annotations/comments, working with mutexes is significantly
harder.  We **strongly recommend** their use.

### Deadlock

A deadlock (sometimes called a *deadly embrace*) occurs when an *activity*
attempts to acquire a limited *resource* that has been exhausted and cannot be
replenished unless that activity makes progress.

When considering deadlocks involving only mutexes, each activity is
typically represented by a thread, and the resources are mutexes that
are exhausted when held, and replenished when released.

The simplest mutex-related deadlock is the *self-deadlock*:

```cpp
mu.Lock();
mu.Lock();      // BUG: deadlock: thread already holds mu
```

Deadlocks involving two resources, such as a mutex and a bounded-sized
thread pool, are easily generated too, but deadlocks involving three or
more resources are less common.  A two-mutex deadlock results when thread
T0 attempts to acquire M1 while holding M0 at the same time that thread
T1 attempts to acquire M0 while holding M1; each thread will wait
indefinitely for the other.

#### Deadlock Detection

Fortunately, deadlocks are among the easiest bugs to debug and avoid.
Debugging is typically easy because the address space stops exactly
where the bug occurs.  A stack trace of the threads is usually all that
is required to see what the threads are waiting for and what resources they
hold.

Additionally, the `absl::Mutex` API provides additional deadlock detection. Such
detection is enabled only when the application is compiled in debug mode and the
flag `-synch_deadlock_detection` is non-zero. When enabled, the API detects two
additional types of deadlock cases:

* Mutexes acquired in an inconsistent order. The deadlock detector maintains an
  *acquired-before* graph for mutexes in a process. An error is generated if
  potential deadlock (a cycle) is detected in that graph.
* Mutexes which are released by threads which do no hold them.

If `-synch_deadlock_detection=1`, a message is printed for each lock-order
error. If `-synch_deadlock_detection=2`, the first lock-order error causes the
process to abort.

The following calls are not recommended for production code, but are useful when
deadlock detection is enabled:

* `Mutex::AssertHeld()`: abort with high probability if `mu` is not held
  exclusively by the calling thread.
* `Mutex::AssertReaderHeld()`: abort with high probability if `mu` is not
  held in some mode by the calling thread.
* `Mutex::ForgetDeadlockInfo()`: forget ordering information gathered for
  this mutex. This routine should be used if the ordering of mutexes changes
  during execution (this is rare).

Note that deadlock detection introduces significant overhead; it should not be
enabled in production binaries.

## Other Synchronization Operations

Most concurrency issues that are not restricted to memory access issues fall
under the broad category of "synchronization" operations. Within a concurrent
system, a synchronization operation generally encompasses operations for
which a strict ordering must be ensured.

Abseil contains a number of synchronization abstractions:

* `call_once()`, which allows a function to be executed exactly once across all
  threads.
* A `Notification`, which allows threads to receive notification of a single
  occurrence of a single event.
* A `Barrier`, which blocks threads until a prespecified threshold of threads
  utilizes the barrier, at which point the barrier is unblocked.
* A `BlockingCounter`, which allows a thread to block for a pre-specified number
  of actions.

### `call_once()`

`absl::call_once()` is a fast implementation of the C++ standard library
`std::call_once()`, for invoking a given function at most once, across all
threads. You pass three arguments to `call_once()`: a `once_flag` to coordinate
and identify unique callers, a function to invoke, and the arguments to invoke
with the function.

The first call to `call_once()` with a particular `once_flag` argument (that
does not throw an exception) will run the passed function with the provided
arguments; other calls with the same `once_flag` argument will not run the
function, but will wait for the provided function to finish running (if it is
still running).

This mechanism provides a safe, simple, and fast mechanism for one-time
initialization in a multi-threaded process:

```cpp
class MyInitClass {
  public:
  ...
  absl::once_flag once_;

  MyInitClass* Init() {
    absl::call_once(once_, &MyInitClass::Init, this);
    return ptr_;
  }
}
```

### `Notification`

A `Notification` allows threads to receive notification of a single occurrence
of a single event. Threads sign up for notification using one of the
notification's `WaitForNotification*()` member functions.
`Notification::Notify()` is used to notify those waiting threads that the event
has occurred, and may be only called once for any given notification.

Example:

```cpp
// Create the notification
Notification notification_;

// Client waits for notification
void Foo() {
  notification_.WaitForNotification();
  // Do something based on that notification
}

//
void Bar() {
  // Do a bunch of stuff that needs to be done before notification
  notification_.Notify();
}

```

### Barrier

An `absl::Barrier` blocks threads until a prespecified threshold of threads
utilizes the barrier. A thread utilizes the `Barrier` by calling `Block()` on
the barrier, which will block that thread; no call to `Block()` will return
until the specified number of threads have called it.

Example:

```cpp
Barrier *barrier_ = new Barrier(num_active_threads);

void Foo() {
  if (barrier_->Block()) {
    delete barrier_;  // This thread is responsible for destroying barrier_;
  }
  // Do something now that the Barrier has been reached.
}
```

### BlockingCounter

An `absl::BlockingCounter` blocks all threads for a pre-specified number of
actions. Threads call `Wait()` on a blocking counter to block until the
specified number of events occur; worker threads then call `DecrementCount()` on
the counter upon completion of their work. Once the counter's internal "count"
reaches zero, the blocked thread unblocks.

Example:

```cpp
void Foo() {
  BlockingCounter things_to_process(things.size());
  Process(&things_to_process)
  things_to_process.Wait();
}

void Process(BlockingCounter* things) {
  // Do stuff
  things->DecrementCount();
  return;
}
```


text:
# gpool

A lightweight Goroutine worker pool, modified from [grpool](https://github.com/ivpusic/grpool)

## Example

```go
    runtime.GOMAXPROCS(runtime.NumCPU())

    // new a Goroutine pool with 100 workers and 50 job queue
    pool := gpool.NewPool(100, 50)
    defer pool.Release()

    // set 10 jobs that the pool should wait
    pool.WaitCount(10)

    // submit one or more jobs to pool
    for i := 0; i < 10; i++ {
        count := i

        pool.JobQueue <- func() {
            defer pool.JobDone()
            fmt.Printf("Pool job: %d\n", count)
        }
    }

    // wait until we call JobDone for all jobs
    pool.WaitAll()
```

## License
[MIT License](https://github.com/TimothyYe/gpool/blob/master/LICENSE)

repository: TimothyYe/gpool

repo_name: TimothyYe/gpool


text:
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.7.2] - 2018-11-
### Added
- #32 update to django version 2.3.1
- Add CHANGELOG.md
- travis python version 3.5 to 3.7
- travis django version 2.x

### Changed
- travis python version 3.4
- travis django version 1.x

### Removed
- travis python version
- support of django 1.x version

### Security
- update request package to 2.20.0

repository: ioO/django-billjobs

gha_language: Python

gha_license_id: NOASSERTION

repo_name: ioO/django-billjobs


text:
# A React Native wrapper for the [SaleMove](https://www.salemove.com) client library

[![npm version](https://badge.fury.io/js/react-native-salemove.svg)](https://badge.fury.io/js/react-native-salemove)

## Installation iOS

1. Run `npm install react-native-salemove --save` in your project directory
2. Open your `ios` folder and make sure the `cocoapods` are installed by running `pod --version`
3. Open `Podfile` and add the following `pod 'react-native-salemove', :path => '../node_modules/react-native-salemove'` and `pod 'SalemoveSDK', :git => 'https://github.com/salemove/ios-bundle'`
4. Run `pod install`

## Demo

Please consider taking a look at the [demo app][0] where the integration is done

[0]: https://github.com/salemove/react-native-demo

repository: salemove/react-native-salemove

gha_language: Swift

repo_name: salemove/react-native-salemove


text:
MAGCODE: models for calculating magnetic fields and interactions
================================================================

Public-facing documentation for this repository is located at: <http://wspr.io/magcode/>

## Citation and acknowledgement

If you use this code for your research, please cite the works from which the code is derived
where possible (ask if you are unsure).

In addition, please cite this code repository as:

* Title: ‘MAGCODE: framework of models for calculating magnetic fields and interactions’
* Author: William S. P. Robertson
* DOI: 10.5281/zenodo.1491206

DOI for the latest release: [![DOI](https://zenodo.org/badge/344445.svg)](https://zenodo.org/badge/latestdoi/344445)

Licence for modification and distribution
-----------------------------------------

This work is freely modifiable and distributable under the terms and conditions of the
[Apache License v2](http://www.apache.org/licenses/LICENSE-2.0).
In effect, you are free to do with this code as you wish for the development of free or proprietary software.
Distributions of modified works must retain the original copyright notices and contain a list of modifications made, but this is only an expository notice; please see the licence text for complete details.

repository: Sean-McGowan/magcode

extension: markdown

repo_name: Sean-McGowan/magcode


text:
---
layout: post
title: "Home Directory File Structure"
date: 2015-10-22 10:32:44 +0100
comments: true
sharing: true
footer: true
categories:
- Tech
tags:
- Filesystem
- Organised
---

Apples default directory structure is :

    /Users/username
    ├── Desktop
    ├── Documents
    ├── Downloads
    ├── Library   # Hidden by default
    ├── Movies
    ├── Music
    ├── Pictures
    ├── Public

My current setup After adding Dropbox, dotfiles stored on github, Code, and user level applications folder:

    /Users/username
    ├── Applications
    ├── Code
    ├── Desktop
    ├── Documents
    ├── Downloads
    ├── Dropbox
    ├── Library
    ├── Movies
    ├── Music
    ├── Pictures
    ├── Public
    ├── dotfiles

At present though my Dropbox is a mess of some documents, some code and lots of ebooks. Free Dropbox does not have enough space for me to the entire home directory.  BtSync will allow me to set the root as my home directory and select which folders from that to sync.

I have started using USB flash drives again, and have found [unison](http://www.cis.upenn.edu/~bcpierce/unison) to be a useful tool for syncing local drives to each other.

Me example unison sync (USB drive is called 'Dropbox'):

    unison ~/ /Volumes/Dropbox/Unison/ -fat -auto -links true \
    -path Code      \
    -path Documents \
    -path Dropbox

    #-fat Work With FAT filesystem (ThumbDrive)
    #-auto Accept default action for non-merges
    #-links true follow symlinks

Now that the 'Dropbox' root is my home directory I can reorganise. Books should become a top level folder, and share a place along side Music and Movies.

Dropbox is kept but mainly used for sharing temporary files with others.

    /Users/username
    ├── Applications
    ├── Books
    ├── Code
    ├── Desktop
    ├── Documents
    ├── Downloads
    ├── Dropbox
    ├── Library
    ├── Movies
    ├── Music
    ├── Pictures
    ├── Public
    ├── dotfiles

Dropbox: sending large files
BT Sync: Synchronising files (eBook collection)
Unison: Synchronising local drives (Thumbdrive to local documents/ebooks)
ReadyNAS: Remote access to larger volume drives (Movies and TV Collections)
 Above could be a BT Sync non populated folder


text:
# Bebbe-OS

Bebbe-OS is an Protected mode 32 bit operating system for X86 that is named after one of my cats. Bebbe is not the favorite cat, but the other one is named "The sausage" so it was an easy pick.

Bebbe is written in C with some elements of inline asm and standalone Assembly files. Assembly used is AT&T. It follow mutliboot specification and is currenlty loaded with GRUB. It does not probe the BIOS for memory map, instead it kindly asks GRUB.

## Features
Interrupts

## Planned features
Virtuell memory (Paging)
User processes
Basic filesystem FAT32
Build script that builds the cross-compiler for you

## Prerequisites

```
i686 cross-compiler, both asm and gcc
qemu
grub-mkrescue
```

The sad news is that you need to build a cross-compiler in order to compile this operatingsystem yourself.
https://wiki.osdev.org/GCC_Cross-Compiler
Is a good resource on getting started on how to setup the cross-compiler.
This project is currently compiled with GCC 9.2.0.

## Usage

```
make run
```

## Authors
Carl Söyseth

## License
[MIT](https://choosealicense.com/licenses/mit/)

repository: sojunator/Bebbe-OS

repo_name: sojunator/Bebbe-OS