text:
# Creating Blocks

This guide describes the basic concepts behind blocks, data types, and vectors,
and demonstrates how to create blocks and data types.

If you're new to Lua, don't worry. Lua is a simple, consistent language that
can be picked up easily.  Check out [Learn Lua in Y
Minutes](https://learnxinyminutes.com/docs/lua/) for a quick introduction.

## Table of Contents

* [Hierarchical Blocks](#hierarchical-blocks)
    * [Using `CompositeBlock` directly](#using-compositeblock-directly)
    * [Deriving from `CompositeBlock`](#deriving-from-compositeblock)
* [Basic Types](#basic-types)
* [Vectors](#vectors)
* [Blocks](#blocks)
    * [Methods](#methods)
    * [Template](#template)
    * [Type Signatures](#type-signatures)
    * [Example](#example)
    * [Testing](#testing)
    * [Benchmarking](#benchmarking)
    * [FFI](#ffi)
        * [Faster `MultiplyBlock`](#faster-multiplyblock)
        * [BFSK Modulator](#bfsk-modulator)
* [Custom Types](#custom-types)
    * [`CStruct` types](#cstruct-types)
    * [`Object` types](#object-types)
* [Manipulating Bits](#manipulating-bits)
* [Development Tips](#development-tips)

## Hierarchical Blocks

The simplest kind of custom block that can be created in LuaRadio is a
hierarchical block.

A hierarchical block is a composition of several blocks into a single block,
which abstracts the underlying flow graph and provides a simpler input/output
port interface. Creating a hierarchical block is a matter of instantiating
blocks, connecting them, and defining the boundary input and output ports.

Hierarchical blocks have no runtime overhead in LuaRadio; they're simply a
convenience. When running a flow graph containing a hierarchical block,
LuaRadio runs its internal blocks as if they were directly connected into the
top-level flow graph.

There are two ways to create a hierarchical block. The first is to use
[`CompositeBlock`](0.reference-manual.md#compositeblock) directly, and the
second is to derive a block class from `CompositeBlock`.

In the example below, we'll create a hierarchical block both ways for a mono
Wideband FM Demodulator for FM broadcast radio, described by the following flow
graph:

### Using `CompositeBlock` directly

``` lua
local radio = require('radio')

function wbfm_mono_demodulator_factory(tau)
    -- Blocks
    local fm_demod = radio.FrequencyDiscriminatorBlock(1.25)
    local af_filter = radio.LowpassFilterBlock(128, 15e3)
    local af_deemphasis = radio.FMDeemphasisFilterBlock(tau)
    local demodulator = radio.CompositeBlock()

    -- Connections
    demodulator:connect(fm_demod, af_filter, af_deemphasis)

    -- Type signature
    demodulator:add_type_signature({block.Input("in", radio.types.ComplexFloat32)},
                                   {block.Output("out", radio.types.Float32)})

    -- Aliases
    demodulator:connect(demodulator, "in", fm_demod, "in")
    demodulator:connect(demodulator, "out", af_deemphasis, "out")

    return demodulator
end
```

In this example, we create a function `wbfm_mono_demodulator_factory()` which
returns a `CompositeBlock` that can be used as an Wideband FM Mono Demodulator
in a flow graph. The argument to the function specifies the time constant of
the FM De-emphasis filter (typically 75 μS in the Americas and South Korea, and
50 μS everywhere else).

``` lua
    -- Blocks
    local fm_demod = radio.FrequencyDiscriminatorBlock(1.25)
    local af_filter = radio.LowpassFilterBlock(128, 15e3)
    local af_deemphasis = radio.FMDeemphasisFilterBlock(tau)
    local demodulator = radio.CompositeBlock()

    -- Connections
    demodulator:connect(fm_demod, af_filter, af_deemphasis)
```

The blocks section instantiates the internal blocks of the demodulator, as well
as a `CompositeBlock` that will hold their composition. The connections section
connects the internal blocks. This procedure is identical to building a
top-level flow graph, as seen in the [Getting
Started](2.getting-started.md#example) guide.

``` lua
    -- Type signature
    demodulator:add_type_signature({block.Input("in", radio.types.ComplexFloat32)},
                                   {block.Output("out", radio.types.Float32)})
```

The type signature section adds a type signature that defines the names and
data types of the boundary input/output ports of the hierarchical block. This
step creates the external interface to the hierarchical block, so we may
connect other blocks to it.

The first argument of
[`add_type_signature()`](0.reference-manual.md#blockadd_type_signatureinputs-outputs-process_funcnil-initialize_funcnil)
is an array of [`radio.block.Input()`](0.reference-manual.md#input)
descriptors, which specifies the name and data type of each input port. The
second argument is the analogous for output ports, with
[`radio.block.Output()`](0.reference-manual.md#output) descriptors.  In this
case, there is one input named `in` of type `ComplexFloat32`, and one output
named `out` of type `Float32`. We may call `add_type_signature()` as many times
as we need to register multiple type signatures of different data types as
needed, but they must all share the same input/output port count and names.

``` lua
    -- Aliases
    demodulator:connect(demodulator, "in", fm_demod, "in")
    demodulator:connect(demodulator, "out", af_deemphasis, "out")
```

The aliases section aliases the hierarchical block input/output ports we
defined in `add_type_signature()` to concrete input/output ports of its
internal blocks.  The first one aliases the demodulator's `in` port to the
frequency discriminator's `in` port. The second aliases the demodulator's `out`
port to the de-emphasis filter's `out` port.  Note that this invocation of
`connect()` differs from previous invocations, in that the `CompositeBlock`
passes itself in as a block for connection.

We can then instantiate and use our Wideband FM Mono Demodulator in a flow
graph:

``` lua
radio.CompositeBlock():connect(
    radio.RtlSdrSource(88.5e6 - 250e3, 1102500), -- RTL-SDR source, offset-tuned to 88.5MHz-250kHz
    radio.TunerBlock(-250e3, 200e3, 5),          -- Translate -250 kHz, filter 200 kHz, decimate by 5
    wbfm_mono_demodulator_factory(75e-6),        -- Wideband FM Mono Demodulator
    radio.DownsamplerBlock(5),                   -- Downsample by 5
    radio.PulseAudioSink(1)                      -- Play to system audio with PulseAudio
):run()
```

### Deriving from `CompositeBlock`

Instead of building our Wideband FM Mono Demodulator composition in a function,
we can build it in a new block class derived from `CompositeBlock`. The
resulting block class will look and feel like any other LuaRadio block.

``` lua
local radio = require('radio')

local WBFMMonoDemodulator = radio.block.factory('WBFMMonoDemodulator', radio.CompositeBlock)

function WBFMMonoDemodulator:instantiate(tau)
    radio.CompositeBlock.instantiate(self)

    -- Blocks
    local fm_demod = radio.FrequencyDiscriminatorBlock(1.25)
    local af_filter = radio.LowpassFilterBlock(128, 15e3)
    local af_deemphasis = radio.FMDeemphasisFilterBlock(tau)

    -- Connections
    self:connect(fm_demod, af_filter, af_deemphasis)

    -- Type signature
    self:add_type_signature({block.Input("in", radio.types.ComplexFloat32)},
                            {block.Output("out", radio.types.Float32)})

    -- Aliases
    self:connect(self, "in", fm_demod, "in")
    self:connect(self, "out", af_deemphasis, "out")
end
```

The third line manufactures a new block class with name "WBFMMonoDemodulator"
and parent class `CompositeBlock`. The name string is used for debugging
purposes.

The remaining lines populate the `instantiate()` constructor of the
manufactured block class. This method is called when the block is instantiated,
e.g. `WBFMMonoDemodulator(...)`.  The inner workings are basically the same as
the function factory above, except that any reference to `demodulator` is
replaced with `self`, since the class itself is a `CompositeBlock`.

We can use this Wideband FM Mono Demodulator in the same example as before,
after replacing `wbfm_mono_demodulator_factory(...)` with
`WBFMMonoDemodulator(...)`.  This Wideband FM Mono Demodulator will behave
identically, but will have the added benefits of a descriptive block name and
being an instance of a well-defined block class.

## Basic Types

Before we dive in to implementing blocks, we'll first need to take a look at
LuaRadio's basic types and how to create and manipulate vectors of them.

LuaRadio has four basic types:
[`ComplexFloat32`](0.reference-manual.md#complexfloat32),
[`Float32`](0.reference-manual.md#float32), [`Bit`](0.reference-manual.md#bit),
[`Byte`](0.reference-manual.md#byte). Each instance of these types represents
one element, so a `radio.types.ComplexFloat32(3, 2)` is a complex number with
real value of 3 and imaginary value of 2, and `radio.types.Float32(123.0)` is a
real number with a value of 123.0. These types are LuaJIT ctypes, distinct from
any of the Lua primitive types.

Each basic type is backed by a C structure type. This is primarily for
implementation reasons, so that LuaRadio can associate a metatable of
operations with it, and so that these instances are distinct from other
occurrences of the underlying C type (e.g. `float`, `uint8_t`).

The C types for the four basic types are:

``` c
/* radio.types.ComplexFloat32 */
typedef struct {
    float real;
    float imag;
} complex_float32_t;

/* radio.types.Float32 */
typedef struct {
    float value;
} float32_t;

/* radio.types.Bit */
typedef struct {
    uint8_t value;
} bit_t;

/* radio.types.Byte */
typedef struct {
    uint8_t value;
} byte_t;
```

Despite the structure wrapper, these types are compatible with the C types
`float complex`, `float`, `uint8_t`, `uint8_t`, respectively.

These basic types have metatables associated with them, that define basic
operations on the type:

``` lua
local a = radio.types.ComplexFloat32(3, 2)

print(a) --> ComplexFloat32<real=3, imag=2>
print(a + a) --> ComplexFloat32<real=6, imag=4>
print(a * a) --> ComplexFloat32<real=5, imag=12>
print(a:scalar_mul(2)) --> ComplexFloat32<real=6, imag=4>
print(a:abs()) --> 3.6055512428284
print(a:arg()) --> 0.58800262212753
```

See the [LuaRadio Reference Manual](0.reference-manual.md#basic-types) for all
operations defined on the basic types.

## Vectors

It would be inefficient for blocks to process one sample at a time, as the
overhead of serializing the sample and calling the block to process it would
exceed the cost of processing it. Instead, blocks operate on a vector of
samples at a time, to amortize the overhead of serialization.

A vector is a simple container class for a dynamically-allocated contiguous C
array, with only a few auxiliary methods for string representation, resizing,
and appending.  Each basic type provides two static methods for creating a
vector of itself: `.vector(num)` for a zero-initialized vector, or
`.vector_from_array(arr)` for a vector initialized from a Lua array.

The example below illustrates the important uses of the Vector:

``` lua
-- Create a length 4, zero-initialized Float32 vector
local vec = radio.types.Float32.vector(4)

-- Vector properties
print(vec.length) --> 4, length of vector
print(vec.size) --> 16, size of vector in bytes
print(vec.data_type) --> ctype<struct float32>, C type of vector element
print(vec.data_type == radio.types.Float32) --> true
print(ffi.typeof(vec.data)) --> ctype<struct float32 *>, C type of vector data

-- Vector string representation
print(vec) --> [Float32<value=0>, Float32<value=0>, Float32<value=0>, Float32<value=0>]

-- Accessing and modifying vector elements
print(vec.data[0]) --> Float32<value=0>
vec.data[0].value = 5
print(vec.data[0]) --> Float32<value=5>
print(vec) --> [Float32<value=5>, Float32<value=0>, Float32<value=0>, Float32<value=0>]

-- Iterating over a vector's elements
-- (Note: Lua for-loop ranges are inclusive)
for i = 0, vec.length-1 do
    print(vec.data[i])
end
--> Float<value=5>
--> Float<value=0>
--> Float<value=0>
--> Float<value=0>

-- Resizing a vector
vec:resize(6)
print(vec.length) --> 6
print(vec.size) --> 24

-- Appending an element to a vector
vec:append(radio.types.Float32(42))
print(vec.length) --> 7
vec:append(radio.types.Float32(26))
print(vec.length) --> 8
print(vec.data[6]) --> Float32<value=42>
print(vec.data[7]) --> Float32<value=26>
```

Much of block processing consists of either iterating over a vector and
modifying its elements, or passing the vector to an external library.  Since
`vec.data` is a `float *` (in the example above) and `vec.length` is its
length, we have all of the information we need to pass the vector memory to an
external library for processing.

Resizing a vector only causes a re-allocation of the underlying memory when it
is grown to a larger size. The underlying memory is retained on resizing to a
smaller size; just the bookkeeping is updated.

All vectors are allocated with page alignment. This is to enable their
processing with certain library routines that require, or perform better with,
aligned buffers. This is often because SIMD operations are involved.

##### Caveats

Note that a Vector's `data` is a C pointer with zero-based indexing, where as
Lua uses one-based indexing for its arrays. This inconsistency seems terribly
confusing, but in practice doesn't pose much ambiguity: just think C or
low-level when working with sample data, and Lua or high-level when working
with block arguments and curly bracket arrays.

## Blocks

LuaRadio blocks are written in pure Lua, but may interface with external
libraries via the [LuaJIT foreign function
interface](http://luajit.org/ext_ffi_api.html) (FFI) to offload processing,
when needed.

Each block is run in an independent process, under its own Lua state. Blocks
typically have one hot loop, so they can benefit greatly from LuaJIT's
just-in-time trace compiler, and many pure Lua blocks do achieve good
performance without needing acceleration from an external library.

In other cases — particularly when the block can be accelerated by SIMD
extensions — being able to prototype in pure Lua first, before wrapping an
external library, is a nice convenience in block development and can serve as a
fallback implementation for non-real-time applications.

### Methods

Blocks are classes manufactured by the
[`radio.block.factory(name)`](0.reference-manual.md#radioblockfactoryname-parent_classnil)
factory function. Blocks implement all of their functionality with four
methods:

* `instantiate(...)` — constructor
* `initialize()` — initialization (optional)
* `process(...)` — main work method
* `cleanup()` — clean up (optional)

The `instantiate(...)` method is the block constructor, called when the block
is instantiated. This method should establish basic state and register the
block's type signatures.

The `initialize()` method is called by the framework for additional block
initialization before the block is run. In this method, the block's
differentiated type signature and sample rate are available to the block, and
can be used to perform additional initialization.

The `process()` method is the main work method of the block. It receives
immutable input vectors of samples as arguments, and returns output vectors of
samples. This method is called repeatedly by the framework to process inputs
into outputs.

The `cleanup()` method is called by the framework when the flow graph has
collapsed, right before the block exits. This method can be optionally
implemented to perform additional clean up, if necessary.

Finally, source blocks and blocks that modify the sample rate must implement
the `get_rate()` method, which returns the block's sample rate as a number, in
samples per second.

### Template

The template below provides an outline of the block methods and what can be
done in each one:

``` lua
local radio = require('radio')

local MyBlock = radio.block.factory('MyBlock')

function MyBlock:instantiate(...)
    -- Perform instantiation here

    -- Call self:add_type_signature(<inputs>, <outputs>) for each type signature
end

function MyBlock:initialize()
    -- Perform initialization here

    -- Can use self:get_input_type(), self:get_output_type(), self:get_rate()
    -- to get differentiated type signature and sample rate
end

function MyBlock:process(...)
    -- Process input samples and return output samples here
end

function MyBlock:cleanup()
    -- Perform cleanup here
end

return MyBlock
```

### Type Signatures

A type signature defines the names and data types of input and output ports.
Type signatures must be added in the `instantiate(...)` method of a block, so
that the block can be connected into a flow graph after it has been
instantiated.  When a flow graph is run, every block — starting from sources,
downstream to sinks — is differentiated into the type signature that is
compatible with its input types.

The syntax for adding a type signature is:

```
self:add_type_signature(<inputs>, <outputs>)
```

The first argument of
[`add_type_signature()`](0.reference-manual.md#blockadd_type_signatureinputs-outputs-process_funcnil-initialize_funcnil)
is an array of [`radio.block.Input()`](0.reference-manual.md#input)
descriptors, which specify the name and data type of each input port. The
second argument is analogous, with
[`radio.block.Output()`](0.reference-manual.md#output) descriptors, for output
ports.  We may call `add_type_signature()` as many times as needed to register
type signatures of different data types, but they must all share the same
input/output port count and names.

Source and sink type signatures can be described with an empty array for inputs
or outputs, respectively.

Below are some examples of type signatures:

``` lua
-- One ComplexFloat32 input named `in` and one Float32 output named `out`
self:add_type_signature({radio.block.Input('in', radio.types.ComplexFloat32)},
                        {radio.block.Output('out', radio.types.Float32)})

-- Two Float32 inputs named `in1` and `in2`, and one Float32 output named `out`
self:add_type_signature({radio.block.Input('in1', radio.types.Float32),
                         radio.block.Input('in2', radio.types.Float32)},
                        {radio.block.Output('out', radio.types.Float32)})

-- Byte source, one Byte output named `out`
self:add_type_signature({}, {radio.block.Output('out', radio.types.Float32)})

-- ComplexFloat32 sink, one ComplexFloat32 input named `in`
self:add_type_signature({radio.block.Input('in', radio.types.ComplexFloat32)}, {})
```

A type signature may also specify custom `process()` and `initialize()`
methods.  When the block is differentiated, its `process()` and `initialize()`
methods are bound to these methods.

``` lua
self:add_type_signature(<inputs>, <outputs>[, <process method>, <initialize method>])
```

For example:

``` lua
-- Complex-valued type signature
self:add_type_signature({radio.block.Input('in', radio.types.ComplexFloat32)},
                        {radio.block.Output('out', radio.types.ComplexFloat32)},
                        self.process_complex, self.initialize_complex)

-- Real-valued type signature
self:add_type_signature({radio.block.Input('in', radio.types.Float32)},
                        {radio.block.Output('out', radio.types.Float32)},
                        self.process_real, self.initialize_real)
```

If a block with the type signatures above is differentiated to the
`ComplexFloat32` type, then `initialize()` will be mapped to
`initialize_complex()`, and `process()` will be mapped to `process_complex()`.
Analogous mapping occurs in the `Float32` type differentiation.

This feature removes redundant differentiation from the critical path of
`process()` and makes it easier to separate type-dependent initialization and
processing code in a block.

A block can look up its differentiated types in `initialize()` or `process()`
with the [`get_input_type()`](0.reference-manual.md#blockget_input_typeindex1)
and [`get_output_type()`](0.reference-manual.md#blockget_output_typeindex1)
methods.  See the [LuaRadio Reference Manual](0.reference-manual.md#block-1)
for more information.

### Example

In the example below, we create a multiply block capable of multiplying
complex-valued or real-valued inputs.

``` lua
local radio = require('radio')

local MultiplyBlock = radio.block.factory('MultiplyBlock')

function MultiplyBlock:instantiate()
    self:add_type_signature({radio.block.Input("in1", radio.types.ComplexFloat32),
                             radio.block.Input("in2", radio.types.ComplexFloat32)},
                            {radio.block.Output("out", radio.types.ComplexFloat32)})
    self:add_type_signature({radio.block.Input("in1", radio.types.Float32),
                             radio.block.Input("in2", radio.types.Float32)},
                            {radio.block.Output("out", radio.types.Float32)})
end

function MultiplyBlock:initialize()
    self.out = self:get_output_type().vector()
end

function MultiplyBlock:process(x, y)
    local out = self.out:resize(x.length)

    for i = 0, x.length - 1 do
        out.data[i] = x.data[i] * y.data[i]
    end

    return out
end
```

The block constructor `instantiate()` adds two type signatures with two inputs
and one output. The first is for complex-valued inputs, and the second is for
real-valued inputs.

The `initialize()` method creates a persistent vector of its differentiated
output type (either `ComplexFloat32` or `Float32`) in `self.out`, which the
`process()` method can use to resize and store output samples in.

The `process()` method resizes its output vector for the input length, performs
the multiplication on the two input vectors `x` and `y`, storing the results in
the output vector, and returns the output vector.  Note that this `process()`
implementation is the same for both data types.

Inputs are provided to the block's `process()` method in the same order that
they are defined in the type signature. In this case, the first argument `x`
corresponds to the input `in1`, and the second argument `y` corresponds to the
input `in2`. The framework guarantees that blocks with multiple inputs will
receive equal length vectors for all inputs, so `x.length` and `y.length` are
the same in `process()`. The framework also guarantees that all inputs to a
block have equal sample rates.

Although we could have created a new output vector in `process()`, instead of
reusing `self.out`, it would lead to unnecessary memory allocations and
deallocations as the vector is created on each call to `process()` and later
garbage collected. If this block were in the path of high sample throughput,
then this can also cause more active memory usage than necessary, as the
garbage collector plays catch-up.

Reusing `self.out` allows the output vector to approach a steady state size, as
it is resized to accommodate the inputs. In turn, this allows the block to
reach constant memory usage and spend most of its time on computation and
serialization.

### Testing

There are two ways to go about testing a block. The first is manually, by
calling the block directly in a test script. This requires differentiating and
initializing the block first, so that the correct type signature is selected
and the block is initialized. For example:

``` lua
-- Instantiate
local multiply = MultiplyBlock()

-- Differentiate to Float32 type
multiply:differentiate({radio.types.Float32, radio.types.Float32})

-- Initialize
multiply:initialize()

-- Now, we can use process()
local a = radio.types.Float32.vector_from_array({1, 3, 5})
local b = radio.types.Float32.vector_from_array({2, 4, 6})
local c = multiply:process(a, b)

print(c) --> [Float32<value=2>, Float32<value=12>, Float32<value=30>]
```

A block is differentiated with
[`differentiate()`](0.reference-manual.md#blockdifferentiateinput_data_types)
by providing an array of its input data types. In this case, an array of two
`Float32` data types was provided, so that the block would be differentiated
into the type signature with two `Float32` inputs and one `Float32` output.

The second way of testing a block is with LuaRadio's unit testing suite. This
suite provides a block testing jig, where only the block arguments, input
vectors, and expected output vectors need to be specified. For example:

``` lua
local radio = require('radio')
local jigs = require('tests.jigs')

jigs.TestBlock(MultiplyBlock, {
    {
        desc = "simple test",
        args = {},
        inputs = {radio.types.Float32.vector_from_array({1, 3, 5}),
                  radio.types.Float32.vector_from_array({2, 4, 6})},
        outputs = {radio.types.Float32.vector_from_array({2, 12, 30})},
    },
    -- More unit tests follow
}, {epsilon = 1e-6})
```

This test can then be run with `busted` from the LuaRadio repository:

```
$ busted multiply_test.lua
●
1 success / 0 failures / 0 errors / 0 pending : 0.013749 seconds
$
```

Many of testing jig specifications in the LuaRadio test suite are code
generated with Python 3. See the LuaRadio [`tests/`](../tests) folder for more
details.

### Benchmarking

LuaRadio includes a [`BenchmarkSink`](0.reference-manual.md#benchmarksink) for
benchmarking blocks. This sink reports the average throughput of samples
delivered to the sink in samples per second. This throughput can be used as a
rough performance benchmark of a block.

For example, to benchmark the `MultiplyBlock` we built above:

``` lua
local radio = require('radio')

local source = radio.ZeroSource(radio.types.ComplexFloat32, 0)
local multiply = MultiplyBlock()
local sink = radio.BenchmarkSink()
local top = radio.CompositeBlock()

top:connect(source, 'out', multiply, 'in1')
top:connect(source, 'out', multiply, 'in2')
top:connect(multiply, sink)
top:run()
```

In this flow graph, the multiply block inputs are fed by a
[`ZeroSource`](0.reference-manual.md#zerosource), which sources zero-valued
samples, and the multiply block output is connected to a `BenchmarkSink`, which
counts received samples over time. The `BenchmarkSink` periodically reports the
average rate of samples consumed.

The `BenchmarkSink` is technically benchmarking the entire flow graph,
including the `ZeroSource` and the framework's overhead of serializing samples
between blocks. However, in this case, the `MultiplyBlock` is the computational
limiting factor by several orders of magnitude, so the overhead of `ZeroSource`
and the framework is relatively insignificant, and the resulting rate
corresponds roughly to that of the `MultiplyBlock`. Other situations may
require a different strategy for benchmarking (see caveats below).

Running this flow graph on my platform:

```
$ luaradio multiply_benchmark.lua
[BenchmarkSink] 204.70 MS/s (1.64 GB/s)
[BenchmarkSink] 204.57 MS/s (1.64 GB/s)
[BenchmarkSink] 205.03 MS/s (1.64 GB/s)
[BenchmarkSink] 203.60 MS/s (1.63 GB/s)
[BenchmarkSink] 202.17 MS/s (1.62 GB/s)
[BenchmarkSink] 205.98 MS/s (1.65 GB/s)
...
```

##### Caveats

`BenchmarkSink` results are specific to the platform, affected by CPU load
caused by external processes, and do not take into account the performance
degradation that occurs after all available processor cores are utilized in a
larger flow graph.  Nonetheless, the `BenchmarkSink` gives a good idea of
whether or not a block will be a real-time bottleneck to an upstream block of a
particular rate, e.g. an SDR source at 2 MS/s.

It's also important to point out that while the `ZeroSource` is applicable to
many signal processing blocks, which will perform their computation regardless
of if the samples are all zero-valued, it might not be the appropriate stimulus
for blocks that have value-dependent code paths. In those cases, a random
source or a deliberately constructed source may be more appropriate.

### FFI

#### Faster `MultiplyBlock`

The `MultiplyBlock` we created above is pretty fast, but we can interface with
a external library to achieve better performance. In this incarnation of
`MultiplyBlock`, we will wrap the [VOLK library](http://libvolk.org/), a
library of signal processing routines with SIMD acceleration. VOLK provides
kernels for multiplying both complex-valued vectors and real-valued vectors,
with
[`volk_32fc_x2_multiply_32fc()`](http://libvolk.org/doxygen/volk_32fc_x2_multiply_32fc.html)
and
[`volk_32f_x2_multiply_32f()`](http://libvolk.org/doxygen/volk_32f_x2_multiply_32f.html),
respectively.

The [LuaJIT foreign function interface](http://luajit.org/ext_ffi_api.html)
(FFI) allows LuaJIT scripts to call C functions and access C symbols in
dynamically loaded external libraries. This requires defining the function
prototypes and relevant constants first, like a C header file would, so that
LuaJIT has the necessary information to translate types and operate the C ABI.

In the example below, we create `FasterMultiplyBlock`. This block is similar to
the `MultiplyBlock` above, except that it uses the VOLK library with the FFI in
its implementation of `process()`:

``` lua
local radio = require('radio')
local ffi = require('ffi')

local FasterMultiplyBlock = radio.block.factory('FasterMultiplyBlock')

function FasterMultiplyBlock:instantiate()
    self:add_type_signature({radio.block.Input("in1", radio.types.ComplexFloat32),
                             radio.block.Input("in2", radio.types.ComplexFloat32)},
                            {radio.block.Output("out", radio.types.ComplexFloat32)},
                            self.process_complex)
    self:add_type_signature({radio.block.Input("in1", radio.types.Float32),
                             radio.block.Input("in2", radio.types.Float32)},
                            {radio.block.Output("out", radio.types.Float32)},
                            self.process_real)
end

function FasterMultiplyBlock:initialize()
    self.out = self:get_output_type().vector()
end

ffi.cdef[[
void (*volk_32fc_x2_multiply_32fc)(
    complex_float32_t* cVector,
    const complex_float32_t* aVector,
    const complex_float32_t* bVector,
    unsigned int num_points
);
void (*volk_32f_x2_multiply_32f)(
    float32_t* cVector,
    const float32_t* aVector,
    const float32_t* bVector,
    unsigned int num_points
);
]]
local libvolk = radio.platform.libs.volk

function FasterMultiplyBlock:process_complex(x, y)
    local out = self.out:resize(x.length)

    libvolk.volk_32fc_x2_multiply_32fc(out.data, x.data, y.data, x.length)

    return out
end

function FasterMultiplyBlock:process_real(x, y)
    local out = self.out:resize(x.length)

    libvolk.volk_32f_x2_multiply_32f(out.data, x.data, y.data, x.length)

    return out
end
```

The example defines the functions for the two VOLK kernels in
[`ffi.cdef()`](http://luajit.org/ext_ffi_api.html#ffi_cdef). The first is for
multiplying complex floats, and the second is for multiplying real floats. The
definitions are function pointers, because VOLK kernels are exposed as function
pointers (despite their prototypes in the VOLK documentation).

These definitions are almost identical to their library prototypes —
[`volk_32fc_x2_multiply_32fc()`](http://libvolk.org/doxygen/volk_32fc_x2_multiply_32fc.html)
and
[`volk_32f_x2_multiply_32f()`](http://libvolk.org/doxygen/volk_32f_x2_multiply_32f.html)
— with the exception of substituting the VOLK `lv_32fc_t` type for
`complex_float32_t`, and the C `float` type for `float32_t`.

The substituted types are binary compatible. The substitution is necessary for
the FFI to accept the use of the LuaRadio `ComplexFloat32` and `Float32` data
types in those argument positions and not raise a type mismatch.  Other
alternatives are to use a `typedef` alias or
[`ffi.cast()`](http://luajit.org/ext_ffi_api.html#ffi_cast).

Next, the example looks up the VOLK library from the `radio.platform.libs`
table.  This pre-loaded table has a few computational libraries
([VOLK](http://libvolk.org/),
[liquid-dsp](https://github.com/jgaeddert/liquid-dsp),
[FFTW](http://www.fftw.org/)) that are used by several blocks in the LuaRadio
codebase for acceleration. To load a different library, the
[`ffi.load()`](http://luajit.org/ext_ffi_api.html#ffi_load) call can be used
directly.

Rather than sharing one `process()` method, as the previous `MultiplyBlock`
implementation did, this version of `MultiplyBlock` binds a custom process
method with `add_type_signature()` to each type signature, since the library
call depends on the differentiated type signature.

Each `process()` method resizes the output vector, calls the library routine
with the input vector data and output vector data, and returns the output
vector.

If we re-run the benchmark from above with this block, we see a roughly 1.5x
performance improvement:

```
$ luaradio multiply_bench.lua
[BenchmarkSink] 297.38 MS/s (2.38 GB/s)
[BenchmarkSink] 304.44 MS/s (2.44 GB/s)
[BenchmarkSink] 301.10 MS/s (2.41 GB/s)
[BenchmarkSink] 306.13 MS/s (2.45 GB/s)
[BenchmarkSink] 305.72 MS/s (2.45 GB/s)
[BenchmarkSink] 299.13 MS/s (2.39 GB/s)
...
```

This performance improvement is not substantial, given that the block is
already in the relatively high rate territory, but in other cases —
particularly filtering and DFT — the performance improvement from wrapping an
optimized external library can be an order of magnitude or more. Those cases
are worth accelerating for real-time applications.

#### BFSK Modulator

In this FFI example, we create a binary FSK (BFSK/2FSK) modulator block by
wrapping the FSK modulator in the digital signal processing library
[liquid-dsp](https://github.com/jgaeddert/liquid-dsp). This example
demonstrates adding new functionality to LuaRadio by wrapping an external
library with the LuaJIT FFI.

The `fskmod` module is a recent addition to liquid-dsp, so it has not yet made
it into the [online documentation](http://liquidsdr.org/doc/).  Nevertheless,
the docstrings are available in the implementation
[here](https://github.com/jgaeddert/liquid-dsp/blob/master/src/modem/src/fskmod.c),
and the module is straightforward to use. Below is the liquid-dsp style
documentation reproduced for `fskmod`:

* `fskmod_create(m, k, bandwidth)` creates and returns a `fskmod` object with
  bits per symbol `m`, samples per symbol `k`, and total normalized signal
  bandwidth `bandwidth`.
* `fskmod_destroy(q)` destroys an `fskmod` object, freeing all
  internally-allocated memory.
* `fskmod_modulate(q, s, y)` modulates the input symbol `s`, storing `k`
  samples per symbol number of output samples to `y`.

The `fskmod` module is a M-ary FSK modulator, but since we're only implementing
binary FSK for our block, we use 1 for the bits per symbol parameter `m`. The
other parameters (samples per symbol `m` and normalized signal bandwidth `k`)
are functions of the block's input bit rate, output sample rate, and specified
FSK deviation.

In the example below, we create a block that modulates an input bit stream to a
complex-valued baseband BFSK signal. The constructor accepts the deviation in
Hz (the frequency shift between mark and space frequencies), and the output
sample rate of the modulated samples in samples per second:

``` lua
local radio = require('radio')
local ffi = require('ffi')

local BFSKModulator = radio.block.factory('BFSKModulator')

function BFSKModulator:instantiate(deviation, sample_rate)
    self.deviation = deviation
    self.sample_rate = sample_rate

    self:add_type_signature({radio.block.Input("in", radio.types.Bit)},
                            {radio.block.Output("out", radio.types.ComplexFloat32)})
end

ffi.cdef[[
typedef struct fskmod_s * fskmod;
fskmod fskmod_create(unsigned int _m, unsigned int _k, float _bandwidth);
void fskmod_destroy(fskmod _q);
void fskmod_modulate(fskmod _q, unsigned int _s, complex_float32_t *_y);
]]
local libliquid = radio.platform.libs.liquid

function BFSKModulator:initialize()
    self.samples_per_bit = math.floor(self.sample_rate / radio.block.Block.get_rate(self))

    self.modulator = ffi.gc(
        libliquid.fskmod_create(1, self.samples_per_bit, (self.deviation / self.sample_rate)/2),
        libliquid.fskmod_destroy
    )
    if self.modulator == nil then
        error("Creating liquid fskmod object.")
    end

    self.out = radio.types.ComplexFloat32.vector()
end

function BFSKModulator:get_rate()
    return self.sample_rate
end

function BFSKModulator:process(x)
    local out = self.out:resize(x.length * self.samples_per_bit)

    for i = 0, x.length-1 do
        libliquid.fskmod_modulate(self.modulator, x.data[i].value, out.data[i*self.samples_per_bit])
    end

    return out
end
```

The `ffi.cdef()` call defines the liquid C API for the `fskmod` module. This
consists of an opaque structure pointer aliased to `fskmod`, create and destroy
functions for the `fskmod` object, and the main `fskmod_modulate()` method,
which takes an input symbol (in our case, a bit) and writes the samples of the
modulated signal to a complex-valued array. These definitions are identical to
those in the `liquid.h` header file, except for substituting the `float complex`
type for the compatible LuaRadio data type, `complex_float32_t`, in
`fskmod_modulate()`.

The block's `initialize()` method computes the relevant parameters for
`fskmod_create()`: the number of samples per bit and the normalized frequency
deviation. The samples per bit parameter is computed by dividing the output
sample rate — specified in the constructor — by the input bit rate. The input
bit rate is looked up by using the Block superclass's
[`get_rate()`](0.reference-manual.md#blockget_rate) implementation, which
returns the rate of the upstream block, since this block redefines `get_rate()`
for the samples it produces. It creates the `fskmod` object with these
parameters, and a `ComplexFloat32` persistent output vector in `self.out`.

The `fskmod` object  is wrapped with
[`ffi.gc()`](http://luajit.org/ext_ffi_api.html#ffi_gc), which attaches the
`fskmod_destroy()` finalizer to the object. In effect, `fskmod_destroy()` will
be called on the `fskmod` object when it is garbage collected. This allows us
to delegate the resource management of the `fskmod` to LuaJIT's reference
counting and garbage collection, so we don't have to worry about managing it.

The `BFSKModulator` implements a custom `get_rate()` because its output
samples, the complex-valued BFSK signal, do not share the sample rate of its
input samples, the bit stream.

The `process()` method resizes its output vector length to the input bit length
times the number of samples per bit. It calls `fskmod_modulate()` for each bit
value with the proper offset into the output vector, to modulate each bit into
samples in the output vector, and it returns the output vector.

We can test our BFSK modulator in a simple flow graph, by sourcing it with
random bits from
[`UniformRandomSource`](0.reference-manual.md#uniformrandomsource), and sinking
the modulated signal to a
[`GnuplotSpectrumSink`](0.reference-manual.md#gnuplotspectrumsink) spectrum
plotting sink:

``` lua
local top = radio.CompositeBlock():connect(
  radio.UniformRandomSource(radio.types.Bit, 9600), -- Random bit source, 9600 sample rate (bit rate)
  BFSKModulator(100e3, 1e6),                  -- BFSK Modulator, 100 kHz deviation, 1 MHz sample rate
  radio.ThrottleBlock(),                      -- Throttle block to pace plotting
  radio.GnuplotSpectrumSink()                 -- Spectrum plotting sink
):run()
```

The resulting plot shows the expected two lobes of the BFSK modulated signal,
100 kHz apart:

The liquid `fskmod` module actually supports modulating M-ary FSK, by choice of
M in `fskmod_create()`. This example can be extended to a generic M-ary FSK
modulator by propagating this parameter to `fskmod_create()` in `initialize()`,
and by batching input bits into a base-M number that is passed to the input
symbol argument of `fskmod_modulate()` in `process()`.

## Custom Types

LuaRadio's four basic types serve well for most signal processing needs, but
are insufficient for higher-level blocks that need to produce or consume more
complex aggregate types. This need arises quickly in digital systems, where a
bit stream may be framed into a data frame, a data frame decoded into a packet,
and so on.

LuaRadio offers two solutions for custom types. The first, called a [`CStruct`
type](0.reference-manual.md#cstructtype), is a type backed by a constant-size C
structure. This is appropriate for fixed size data types and is efficiently
serialized between blocks. All four basic types are themselves, in fact,
`CStruct` types.

The second, called an [`Object` type](0.reference-manual.md#objecttype), is a
type backed by any Lua object.  Instances of this type can be variable size and
contain any Lua primitive data types (tables, numbers, strings, booleans,
etc.). `Object` types are not as efficient as `CStruct` types, as they are
marshalled and unmarshalled with [MessagePack](http://msgpack.org/) when
serialized between blocks. However, for relatively low rate uses, they can be
very effective.

### `CStruct` types

`CStruct` types are manufactured with the
[`radio.types.CStructType.factory(ctype, methods)`](0.reference-manual.md#cstructtypefactoryctype-methods)
factory function. The first parameter is the name of the C structure type, and
the second parameter is an optional table of methods and metamethods to be
associated with the type.

The type will automatically inherit `.vector()` and `.vector_from_array()`
static methods, so that vectors of the type can be created by blocks, as well
as internal serialization methods, so that the framework can serialize the type
between blocks. See the [LuaRadio Reference
Manual](0.reference-manual.md#cstructtype) for more inherited methods of
`CStruct` types.

In this example, we construct a fictional type `FooFrame` with a 16-bit address
field, 8-bit type field, and 4-byte data field. We associate a `__tostring()`
string representation method on the type it, as well as `isbeacon()` and
`isdata()` methods to test the fictional frame type.

``` lua
local radio = require('radio')
local ffi = require('ffi')

ffi.cdef[[
typedef struct {
    uint16_t address;
    uint8_t type;
    uint8_t data[4];
} fooframe_t;
]]

local fooframe_methods = {}

function fooframe_methods:__tostring()
    return string.format(
        "FooFrame<address=0x%04x, type=0x%02x, data=[0x%02x,0x%02x,0x%02x,0x%02x]>",
        self.address, self.type, self.data[0], self.data[1], self.data[2], self.data[3]
    )
end

function fooframe_methods:isbeacon()
    return self.type == 0x1
end

function fooframe_methods:isdata()
    return self.type == 0x2
end

local FooFrame = radio.types.CStructType.factory("fooframe_t", fooframe_methods)
```

Now, we can use this type like any other `CStruct` data type:

``` lua
-- A single FooFrame
local a = FooFrame(0xbeef, 0x1, {0xaa, 0xbb, 0xcc, 0xdd})
print(a) --> FooFrame<address=0xbeef, type=0x01, data=[0xaa,0xbb,0xcc,0xdd]>
print(a:isbeacon(), a:isdata()) --> true    false

-- A vector of 10 FooFrame
local vec = FooFrame.vector(10)
print(vec.length) --> 10
print(vec.size) --> 80
print(vec.type) --> ctype<struct 1059>

-- Assign first FooFrame in vector
vec.data[0].address = 0x1234
vec.data[0].type = 0x02
vec.data[0].data = {0xaa, 0xbb, 0xcc, 0xdd}
print(vec.data[0]) --> FooFrame<address=0x1234, type=0x02, data=[0xaa,0xbb,0xcc,0xdd]>
```

Blocks can specify the data type in type signatures to produce or consume it:

``` lua
...
function FooFramerBlock:instantiate()
    ...
    self:add_type_signature({radio.block.Input("in", radio.types.Bit)},
                            {radio.block.Output("out", FooFrame))
end

function FooFramerBlock:process(x)
    local out = FooFrame.vector()

    for i = 0, x.length-1 do
        ...
        out:append(FooFrame(...))
    end

    return out
end
...
```

### `Object` types

`Object` types are manufactured with the
[`radio.types.ObjectType.factory()`](0.reference-manual.md#objecttypefactorymethods)
factory function.

The type will automatically inherit `.vector()` and `.vector_from_array()`
static methods, so that vectors of the type can be created by blocks, as well
as internal serialization methods, so that the framework can serialize the type
between blocks.  In addition, the type will inherit the `.to_json()` method to
form a JSON representation, enabling the type to be used with the
[`JSONSource`](0.reference-manual.md#jsonsource) and
[`JSONSink`](0.reference-manual.md#jsonsink). See the
[LuaRadio Reference Manual](0.reference-manual.md#objecttype) for more
inherited methods of `Object` types.

After an `Object` data type is created, its `.new()` constructor must be
implemented. Additional methods can be defined directly on the data type class.

In this example, we construct a fictional type `BarPacket` with string address
and data fields. We define its constructor and a `__tostring()` method.

``` lua
local radio = require('radio')

local BarPacket = radio.types.ObjectType.factory()

function BarPacket.new(address, data)
    -- Create a new object with its metatable set to BarPacket
    local self = setmetatable({}, BarPacket)

    self.address = address
    self.data = data

    return self
end

function BarPacket:__tostring()
    return string.format(
        'BarPacket<address="%s", data="%s">',
        tostring(self.address), tostring(self.data)
    )
end
```

Now, we can use this type like other data types:

``` lua
local a = BarPacket("foo", "abcd")
print(a) --> BarPacket<address="foo", data="abcd">

local b = BarPacket("waldo", "deadbeef")
print(b) --> BarPacket<address="waldo", data="deadbeef">

-- A vector of BarPacket
local vec = BarPacket.vector()
print(vec.length) --> 0

vec:append(a)
vec:append(b)
print(vec.length) --> 2
print(vec.data[0]) --> BarPacket<address="foo", data="abcd">
print(vec.data[1]) --> BarPacket<address="waldo", data="deadbeef">

print(vec)
--> [BarPacket<address="foo", data="abcd">, BarPacket<address="waldo", data="deadbeef">]
```

As the example shows, `BarPacket` can contain variably sized fields.

Vectors of `Object` types provide a similar interface to `CStruct` type
vectors. Note that their data array is also zero-indexed, for consistency with
`CStruct` type vectors.

Just like `CStruct` based types, blocks can use the `BarPacket` type in type
signatures to produce or consume vectors of the type:

``` lua
...
function BarDecoderBlock:instantiate()
    ...
    self:add_type_signature({radio.block.Input("in", BarFrame)},
                            {radio.block.Output("out", BarPacket))
end

function BarDecoderBlock:process(x)
    local out = BarPacket.vector()

    for i = 0, x.length-1 do
        ...
        out:append(BarPacket(...))
    end

    return out
end
...
```

## Manipulating Bits

Digital blocks that manipulate [`Bit`](0.reference-manual.md#bit) typed vectors
often need to convert bits into numbers and perform bitwise operations on them.

The `Bit` data type provides the static method
[`.tonumber()`](0.reference-manual.md#bittonumbervec-offset0-length0-ordermsb)
that operates on `Bit` vectors to extract numbers from bits. This method takes
a `Bit` vector as its first argument, and optional arguments offset, length,
and bit order to specify most kinds of bit extractions.

Below are a few examples of its use:

``` lua
local vec = radio.types.Bit.vector_from_array({0, 1, 0, 1})
print(vec.length) --> 4
print(radio.types.Bit.tonumber(vec)) --> 5
print(radio.types.Bit.tonumber(vec, 0, 4, 'lsb')) --> 10
print(radio.types.Bit.tonumber(vec, 2, 2, 'msb')) --> 1
print(radio.types.Bit.tonumber(vec, 2, 2, 'lsb')) --> 2
```

The Lua 5.1 language, which LuaJIT implements, does not support bitwise
operations natively, but LuaJIT includes a built-in [`bit`
library](http://bitop.luajit.org/api.html#operations) that can be used to
perform bitwise operations on numbers:

``` lua
bit.band(0x1234abcd, 0xff) --> 0x000000cd
bit.bor(0x1234ab00, 0xcd) --> 0x1234abcd
bit.rshift(0x1234abcd, 1) --> 0x091a55e6
bit.lshift(0x1234abcd, 1) --> 0x2469579a
bit.bnot(0xfffffff0) --> 0x0000000f
```

##### Caveats

There are two important caveats with the `bit` library. First, all operations
with `bit` are limited to 32-bits. For many use cases, this is not a major
issue, and easy to work around. Second, all results of bitwise operations are
returned as **signed 32-bit** numbers. This is generally not an issue for
bitwise operations, which are agnostic to signedness interpretation (with the
exception of arithmetic shifts), but it is an issue when comparing bitwise
operation results to literals.  For example:

``` lua
print(bit.band(0xdeadbeef, 0x80000000) == 0x80000000) --> false (?)
print(bit.bor(0x0fffffff, 0xf0000000) == 0xffffffff) --> false (?)
print(bit.lshift(0x7fffffff, 1) == 0xfffffffe) --> false (?)
```

The discrepancy here is that the Lua number type is a double, and the bit
operation results are returned as signed 32-bit numbers in a double. 0x80000000
and -2147483648, 0xffffffff and -1, 0xfffffffe and -2 are all two different
double numbers.

To compare the result of a bitwise operation to a literal that has its 31st bit
set (the sign bit), first normalize the literal to a signed 32-bit number with
`bit.tobit()`:

``` lua
print(bit.band(0xdeadbeef, 0x80000000) == bit.tobit(0x80000000)) --> true
print(bit.bor(0x0fffffff, 0xf0000000) == bit.tobit(0xffffffff)) --> true
print(bit.lshift(0x7fffffff, 1) == bit.tobit(0xfffffffe)) --> true
```

These caveats will disappear and bitwise operations will be a lot more natural
if a future version of LuaJIT implements the Lua 5.3 language, which has a
64-bit integer type and native bitwise operations.

## Development Tips

#### FFI

When manipulating samples with the LuaJIT FFI, use `ffi.copy()` instead of C
`memcpy()` and `ffi.fill()` instead of C `memset()` for better performance.
Avoid calling `ffi.typeof()` or `ffi.new()` repeatedly in the Block's
`process()` critical path, and instead call them once in `initialize()` (e.g.
when initializing C types, allocating memory, or allocating sample vectors).

#### REPL

For prototyping with Lua, the LuaJIT REPL is a bit too minimalist to be
productive. As in the original Lua 5.1 REPL, it requires preceding expressions
with "=" to print their value, and it doesn't support dumping tables. I suggest
using Torch's [trepl](https://github.com/torch/trepl) for prototyping,
available in [LuaRocks](https://luarocks.org/modules/vsergeev/trepl-torchless).
It supports table dumping, pretty printing, tab completion, and line history.

Another alternative is [lua-repl](https://github.com/hoelzro/lua-repl).

##### trepl

Install trepl with:

```
sudo luarocks --lua-version=5.1 --server=https://luarocks.org/dev install trepl-torchless
```

Add a Lua path modification to allow local loading of the `radio` package, and
an alias to run trepl under LuaJIT:

```
export LUA_PATH="./?/init.lua;;"
alias th='luajit /usr/lib/luarocks/rocks-5.1/trepl-torchless/scm-1/bin/th'
```

Start the REPL with:

```
th
```

##### lua-repl

Install linenoise and lua-repl with:

```
sudo luarocks --lua-version=5.1 install linenoise
sudo luarocks --lua-version=5.1 install luarepl
```

Add a Lua path modification to allow local loading of the `radio` package, and
an alias to run lua-repl under LuaJIT:

```
export LUA_PATH="./?/init.lua;;"
alias lua-repl='luajit /usr/lib/luarocks/rocks-5.1/luarepl/0.9-1/bin/rep.lua'
```

Start the REPL with:

```
lua-repl
```

#### Prototyping

For some blocks, it may be more productive to prototype signal processing
routines in Python, with libraries like [numpy](http://www.numpy.org/),
[scipy](http://docs.scipy.org/doc/scipy/reference/index.html), and
[matplotlib](http://matplotlib.org/), which allow for more traditional sample
by sample development and plotting than a flow graph paradigm like LuaRadio.
When the routine is ready, it can be ported to a LuaRadio block for performance
and reusability.

Samples from LuaRadio can be dumped to a file with the
[`RawFileSink`](0.reference-manual.md#rawfilesink),
[`IQFileSink`](0.reference-manual.md#iqfilesink), or
[`RealFileSink`](0.reference-manual.md#realfilesink), and imported into Python
for inspection with the
[`numpy.fromfile()`](https://docs.scipy.org/doc/numpy-1.17.0/reference/generated/numpy.fromfile.html)
function. The `numpy.float32` type is compatible with LuaRadio's `Float32`
type, and the `numpy.complex64` type is compatible with LuaRadio's
`ComplexFloat32` type. The reverse is also possible with numpy ndarray's
[`.tofile()`](https://docs.scipy.org/doc/numpy-1.17.0/reference/generated/numpy.ndarray.tofile.html#numpy.ndarray.tofile)
method, and the [`RawFileSource`](0.reference-manual.md#rawfilesink),
[`IQFileSource`](0.reference-manual.md#iqfilesource), or
[`RealFileSource`](0.reference-manual.md#realfilesource) blocks.

repository: vsergeev/luaradio

gha_language: Lua

repo_name: vsergeev/luaradio


text:
# N-body matrix elements

```@meta
CurrentModule = EnergyExpressions
DocTestSetup = quote
    using EnergyExpressions
    using AtomicLevels
end
```

The matrix element of an $N$-body operator between two Slater
determinants may be expanded according to the Löwdin rules (which
reduce to the Slater–Condon rules if all single-particle orbitals are
orthogonal):

$$\begin{equation}
\label{eqn:matrix-element-expansion}
\matrixel{\Phi_A}{\Omega_n}{\Phi_B} =
\frac{1}{n!}\sum_p (-)^p
\matrixel{k_1k_2...k_n}{\Omega_n}{l_1l_2...l_n}
D^{AB}({k_1k_2...k_n}|{l_1l_2...l_n})
\end{equation}$$

where $D^{AB}({k_1k_2...k_n}|{l_1l_2...l_n})$ is the determinant minor
of the orbital overlap determinant $D^{AB}$ with the rows
${k_1k_2...k_n}$ and columns ${l_1l_2...l_n}$ stricken out, and $p$
runs over all permutations.

In general, a term in the expansion is thus of the form

$$\begin{equation}
\alpha\matrixel{k_1k_2...k_n}{\Omega_n}{l_1l_2...l_n}\braket{a}{b}\braket{c}{d}\dots\braket{y}{z},
\end{equation}$$

where $\alpha$ is a scalar. This is represented by [`NBodyTerm`](@ref) type.

```@docs
NBodyTermFactor
OrbitalOverlap
OrbitalMatrixElement
numbodies
NBodyTerm
```

```@autodocs
Modules = [EnergyExpressions]
Filter = t -> (t === EnergyExpressions.NBodyMatrixElement)
```

```@docs
isdependent
transform
overlap_matrix
EnergyExpressions.compare
Base.:(==)(::EnergyExpressions.NBodyMatrixElement, ::EnergyExpressions.NBodyMatrixElement)
Base.isapprox(::EnergyExpressions.NBodyMatrixElement, ::EnergyExpressions.NBodyMatrixElement)
EnergyExpression
Matrix
```

## Calculation of determinants

Actually computing the matrix element expansion
$\eqref{eqn:matrix-element-expansion}$ is a combinatorial problem,
that grows factorially with the amount of non-orthogonal orbital
pairs. Furthermore, of the $(n!)^2$ terms generated from the
expansion, only $n!$ are distinct, due to the integrals being
symmetric with respect to interchange of the coordinates [hence the
normalization factor $(n!)^{-1}$]. Thankfully, there are few
symmetries that can be employed, to generate only the distinct
permutations, as well as the fact that the overlap matrix is very
sparse.

### Finding non-zero minors of the overlap determinant

The algorithm to find which minor determinants
$\Gamma^{(N)}(k_1k_2...k_N|l_1l_2...l_N)$ do not vanish, and hence
which $N$ orbitals $k_1k_2...k_N,l_1l_2...l_N$ the $N$-body operator
should be contracted over, is described briefly below. It is devised
to be optimal for orthogonal orbitals (i.e. linear complexity
$\mathcal{O}(Nn)$ where $n$ is the number of orbitals), and
near-optimal for a small amount of non-orthogonal orbitals.

Given:
- An $N$-body operator (implying $N$ rows and $N$ need to
  stricken out), and
- an $n\times n$ matrix, with coordinates of non-zero matrix
  elements: $I,J$ (from these vectors, vanishing rows/columns can
  easily be deduced $\implies$ $N_r$ row/$N_c$ column "rank",
  i.e. yet to be stricken out),
do
- find all $N_r$-combinations of the remaining rows,
- for each such combination, find all columns which would be
  affected if striking out that particular combination of rows,
  - if the "support" (i.e. the only non-zero elements) of any of those
    columns vanishes when striking out the rows, that column must be
    stricken out, too. Total number of these columns is named
    $N_{cm}$,
  - if more than $N_c$ columns must be stricken out ($N_{cm}>N_c$), that
    row combination is unviable,
  - find all $N_c - N_{cm}$-combinations of the remaining columns,
  - for each such combination of columns, find all rows which would be
    affected,
    - if the support of any of those rows vanishes when striking out the
      candidate columns, and the row is not in the candidate set of rows
      to be stricken out, the column combination is unviable.

```@docs
nonzero_minors
```

### Contracting over the stricken out orbitals

We use Julia's built-in `Base.Cartesian.@nloops` iterators to span the
space of all possible choices of orbitals for the contraction of
orbitals. If two or more orbitals are the same, the matrix element is
trivially zero (the “Fermi hole”). To avoid double-counting, we also
only consider those indices that are above the hyper-diagonal.

```@docs
detaxis
detminor
# indexsum
cofactor
EnergyExpressions.distinct_permutations
det
permutation_sign
powneg1
EnergyExpressions.@above_diagonal_loop
EnergyExpressions.@anti_diagonal_loop
```

## Occupation number representation

As an alternative to [`SlaterDeterminant`](@ref), we also provide an
implementation of the occupation number representation (i.e. second
quantization). Every configuration is thus represented as a bit
vector, indicating the occupation of a specific orbital by a `true`
value. Excitations between two configurations are easily computed
using bitwise operations. The present implementation is inspired by

- Scemama, A., & Giner, E. (2013). An efficient implementation of
  Slater–Condon rules. CoRR,
  [arXiv:1311.6244](https://arxiv.org/abs/1311.6244),

but extends upon it by also supporting non-orthogonal orbitals.

The benefit of this approach is much more efficient identification of
which cofactors in $\eqref{eqn:matrix-element-expansion}$ are
non-zero, than the approach taken in [`nonzero_minors`](@ref).

```@docs
BitConfigurations
EnergyExpressions.Orbitals
EnergyExpressions.non_zero_cofactors
```

```@meta
 DocTestSetup = nothing
```

license: MIT

branch_name: refs/heads/master

extension: md

gha_language: Julia

gha_license_id: NOASSERTION

license_type: permissive


text:
# Tic Tac Toe game using python

Tic Tac Toe game -> Use O or X to draw one line and you win.

## Modules used
import os

### How to Use

```bash
python TicTacToe.py
```


text:
# An introduction to Substra

To enable machine learning on distributed data, four types of objects are defined within a Substra network.

## Objective

It clearly defines the scientific question to be answered with a ML model. It specifies:

- the data format that the **Dataset**, **Algo** and **Model** have to follow,
- the identity of the test data points used to compare and evaluate the models,
- the metrics script which is used to quantify the accuracy of a model.

## Dataset

It aggregates numerous data points under a single format. It includes a single *Opener* script which imports and opens the file using libraries specific to the data type.

## Algorithm

It corresponds to a script which specifies the method to train a **Model** on a **Dataset**. In particular it specifies the model type and architecture, the loss function, the optimizer, hyperparameters and identifies the parameters that are tuned during training. Algo dependencies are also specified.

## Model

It corresponds to a file containing the parameters of a trained model. In the case of neural networks, it gathers the weights of the connections. It is associated with:

### Training tasks specification (*Traintuple*)

A *Traintuple* specifies which **Algo** must be trained on which **Dataset** for which **Objective** and starting from which **Model(s)**. Once the training is done, it specifies the resulting **Model**.

### Evaluation tasks specification (*Testtuple*)

A **Testtuple** specifies the output of Traintuple to be evaluated. This corresponds to evaluating a **Model** against a (test) **Dataset** associated with an **Objective**.

The core idea of Substra is to keep **Dataset** in the different nodes of a network and make **Algo** and **Models** travel from one node to another to train them on **Dataset**.

repository: Fabien-GELUS/substra-mnist

gha_language: Python

repo_name: Fabien-GELUS/substra-mnist


text:
# regex-tdd-parameterized-clojure.test-example
Regular Expressions TDD using clojure.test with a little embedded DSL and leiningen as test runner

## Getting Started

* [Download and install Leiningen](https://leiningen.org/#install) (we used version 2.8.1 from the Ubuntu 18.04 package repos in our test)
* Run `lein test`


text:
[![Build Status](https://magnum.travis-ci.com/Katlean/ladder.png?token=FttFscsYxRPLrsy56PFR)](http://magnum.travis-ci.com/Katlean/ladder)

### In ZestFinance, we do two things*:

1. Play Ping Pong
2. Give new engineers a starter project

What's a starter project? We don't hire based on list of buzzwords in a resume but on ability to learn.
As a result, developers with no experience in Ruby and Rails are hired.

These developers need to ramp up quickly.  So for a few weeks, a starter project
is assigned to new developer for the chance to ramp up quickly to the new technologies.

This was [my](https://github.com/dvillarama) starter project.

*And we do more than these two things.  Please see [Zestfinance](http://www.zestfinance.com).

---

## Zestping features
 * A ladder system thats ranks players/teams
 * Supports single and double play
 * Easy UI for adding players and creating doubles teams
 * Easy UI for adding match results

---

## Helpful resource
 * Updated to use ranking to use [TrueSkill](git://github.com/saulabs/trueskill.git)
 * [elovation](https://github.com/drewolson/elovation)

---
## Special Thanks
 * The [mentor](https://github.com/fsproru)

---
## Update 11/18/2013
 * This fork was to experiment with Heroku
  * Switch to Heroku
  * Upgraded to Rails 4.0.1
  * Switch to PostgreSQL from MySQL
  * (http://ping-pong-ladder.herokuapp.com/) if it's up

repository: dvillarama/ladder

repo_name: dvillarama/ladder


text:
---
title: Did Another Task :D
layout: post
author: oliver.rylance
permalink: /4/
source-id: 1Dp7jUyKGGbW7bPLClvAeu5tZcdFE0gfg5zH7NGJDXv0
published: true
---

    Title:
    Did Another Task :D
    Date:
    13/10/16

    Starting point:
    Task 3

    Target for this lesson?
    Complete another task like last time we did the Keyboard Device

    Did I reach my target?
(add details to "Lesson Review")
    Yes and we started the next one but didn't got much progress done

    Lesson Review

    How did I learn? What strategies worked?

    Copy correctly, last lesson we did on the Keyboard Device

    What limited my learning? Which habits do I need to work on?

    Nothing really we did well :D

    What will I change for next time? How will I improve my learning?

    By keeping up the good work :D


text:
### Fundamental Arithmetic Operators (Table 2-9 Page 64)
Operator | Meaning | Type | Example
--- | --- | --- | ---
+ | Addition | Binary | total = cost + tax;
−	 | Subtraction | Binary | cost = total - tax;
*	  | Multiplication | Binary | tax = cost * rate;
/	 | Division | Binary | salePrice = original / 2;
%	 | Modulus | Binary | remainder = value % 3;

repository: HNSS-US/DelTech

repo_name: HNSS-US/DelTech


text:
---
author: "Lujeni"
date: 2017-04-19
linktitle: contribute to ansible w12
title: Contribute to Ansible W12
---

I missed one week ! Was to busy to prepare my next holiday.

To make up for it, I thought I'd give you 2 contributions ;p

- [Ez to test&close](https://github.com/ansible/ansible/issues/17555)

- More and more review of current issues. [This one](https://github.com/ansible/ansible/issues/20762) seems fixed but i am waiting an `Member` approval.

After two months of contribution, i realise the importance of a simple review.

Many opened issues are already fixed in the devel branch.

Mostly "contributions" consist to reproduce and confirm a bug.

repository: Lujeni/settings

repo_name: Lujeni/settings


text:
# Traversable

The `#!hs Traversable` typeclass defined in `#!hs Data.Traversable` is given as

```haskell
class (Functor t, Foldable t) => Traversable t where
    {-# MINIMAL traverse | sequenceA #-}
    traverse :: Applicative f => (a -> f b) -> t a -> f (t b)
    traverse f = sequenceA . fmap f

    sequenceA :: Applicative f => t (f a) -> f (t a)
    sequenceA = traverse id
```

The `#!hs traverse` operation:

1. Maps elements of some structure to actions;
2. Evaluates such actions from left to right;
3. Collects the result.

## SequenceA

The `#!hs sequenceA` operation flips the two layers of structures/contexts and
does not allow any additional functions to be applied to values contained
inside.

??? example "`sequenceA`"
    ```haskell
    fmap Just [1, 2 ,3]
    -- ==> [Just 1, Just 2, Just 3]

    sequenceA $ fmap Just [1, 2, 3]
    -- ==> Just [1, 2, 3]

    sequenceA [Just 1, Just 2, Just 3]
    -- ==> Just [1, 2, 3]
    ```

## Traverse

```haskell
traverse :: (Applicative f, Traversable t) => (a -> f b) -> t a -> f (t b)
```

Which is similar to `#!hs fmap` and `#!hs (=<<)` (flipped bind):

```haskell
fmap     :: Functor f                      => (a ->   b) -> f a -> f    b
(=<<)    :: Monad m                        => (a -> m b) -> m a -> m    b
traverse :: (Applicative f, Traversable t) => (a -> f b) -> t a -> f (t b)
```

The same beneath-structure value-mapping still occurs like `#!hs fmap`, like
flipped bind, `#!hs traverse` also generates more structure. Notice that
different to `#!hs (=<<)`, `#!hs traverse` can have different structure type
in the result compared to the structure that was lifted for the function to
apply to, and eventually will flip the two structures `#!hs f` and `#!hs t`
around.

## MapM and Traverse, Sequence and SequenceA

There are legacy operations `#!hs mapM` and `#!hs sequence` which are defined
in GHC 7.10 and before which are less general versions of `#!hs traverse` and
`#!hs sequenceA` respectively.

```haskell
mapM     :: Monad m                        => (a -> m b) -> [] a -> m ([] b)
traverse :: (Applicative f, Traversable t) => (a -> f b) -> t  a -> f (t  b)
```

Notice that `#!hs traverse` is a generalization of `#!hs mapM` to other
`#!hs Traversable` structures not just limited to lists `#!hs []`, and also
loosening the typeclass constraint from `#!hs Monad` to only requring an
`#!hs Applicative`.

```haskell
sequence  :: Monad m                        => [] (m a) -> m ([] a)
sequenceA :: (Applicative f, Traversable t) => t  (f a) -> f (t  a)
```

Notice that `#!hs sequenceA` is a generalization of `#!hs sequence` in both
generalizing to `#!hs Traversable` structures not only restricted to lists
`#!hs []`, but also loosening the typeclass constraint from `#!hs Monad` to
`#!hs Applicative`.

## Use Case

`#!hs Traversable` is useful for flipping two type constructors (optionally
also mapping in the process).

??? example "`#!hs Traversable`"
    It may be case that some function returns a list of `#!hs Maybe`s, i.e.
    `#!hs [Maybe a]`, but we may want `#!hs Maybe [a]` instead. Semantically,
    it may convey different meaning: a list of possible values is different from
    a possible list of values.

## Traversable Laws

### traverse

1. **Naturality**.

    ```haskell
    t . traverse f = traverse (t . f)
    ```

2. **Identity**.

    ```haskell
    traverse Identity = Identity
    ```

3. **Composition**.

    ```haskell
    traverse (Compose . fmap g . f) = Compose . fmap (traverse g) . traverse f
    ```

### sequenceA

The `#!hs sequenceA` must satisfy:

1. **Naturality**.

    ```haskell
    t . sequenceA = sequenceA . fmap t
    ```

2. **Identity**.

    ```haskell
    sequenceA . fmap Identity = Identity
    ```

3. **Composition**.

    ```haskell
    sequenceA . fmap Compose = Compose . fmap sequenceA . sequenceA
    ```

repository: jieyouxu/Haskell-Notes

repo_name: jieyouxu/Haskell-Notes


text:
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode<EMAIL_ADDRESS>with any additional questions or comments

# OPC Publisher for Azure IoT Edge
This reference implementation demonstrates how Azure IoT Edge can be used to connect to existing OPC UA servers and publishes JSON encoded telemetry data from these servers in OPC UA "Pub/Sub" format (using a JSON payload) to Azure IoT Hub. All transport protocols supported by Azure IoT Edge can be used, i.e. HTTPS, AMQP and MQTT (the default).

This application, apart from including an OPC UA *client* for connecting to existing OPC UA servers you have on your network, also includes an OPC UA *server* on port 62222 that can be used to manage what gets published.

The application is implemented using .NET Core technology and is able to run on the platforms supported by .NET Core.

Publisher implements a retry logic to establish connections to endpoints which have not responded to a certain number of keep alive requests, for example if the OPC UA server on this endpoint had a power outage.

For each distinct publishing interval to an OPC UA server it creates a separate subscription over which all nodes with this publishing interval are updated.

Publisher supports batching of the data sent to IoTHub, to reduce network load. This batching is sending a packet to IoTHub only if the configured package size is reached.

This application uses the OPC Foundations's OPC UA reference stack and therefore licensing restrictions apply. Visit http://opcfoundation.github.io/UA-.NETStandardLibrary/ for OPC UA documentation and licensing terms.

|Branch|Status|
|------|-------------|
|master|[![Build status](https://ci.appveyor.com/api/projects/status/6t7ru6ow7t9uv74r/branch/master?svg=true)](https://ci.appveyor.com/project/marcschier/iot-gateway-opc-ua-r4ba5/branch/master) [![Build Status](https://travis-ci.org/Azure/iot-gateway-opc-ua.svg?branch=master)](https://travis-ci.org/Azure/iot-gateway-opc-ua)|

# Building the Application
The application requires the .NET Core SDK 2.0.

## As native Windows application
Open the OpcPublisher.sln project with Visual Studio 2017 and build the solution by hitting F7.

## As Docker container

Depending if you use Docker Linux or Docker Windows containers, there are different configuration files (Dockerfile or Dockerfile.Windows) to use for building the container.
From the root of the repository, in a console, type:

    docker build -f <docker-configfile-to-use> -t <your-container-name> .

The `-f` option for `docker build` is optional and the default is to use Dockerfile. Docker also support building directly from a git repository, which means you also can build a Linux container by:

    docker build -t <your-container-name> https://github.com/Azure/iot-edge-opc-publisher

# Configuring the OPC UA nodes to publish
The OPC UA nodes whose values should be published to Azure IoT Hub can be configured by creating a JSON formatted configuration file (defaultname: "publishednodes.json"). This file is updated and persisted by the application, when using it's OPC UA server methods "PublishNode" or "UnpublishNode".

The syntax of the configuration file is as follows:

    [
        {
            // example for an EnpointUrl is: opc.tcp://win10iot:51210/UA/SampleServer
            "EndpointUrl": "opc.tcp://<your_opcua_server>:<your_opcua_server_port>/<your_opcua_server_path>",
            "OpcNodes": [
                // Publisher will request the server at EndpointUrl to sample the node with the OPC sampling interval specified on command line (or the default value: OPC publishing interval)
                // and the subscription will publish the node value with the OPC publishing interval specified on command line (or the default value: server revised publishing interval).
                {
                    // The identifier specifies the NamespaceUri and the node identifier in XML notation as specified in Part 6 of the OPC UA specification in the XML Mapping section.
                    "ExpandedNodeId": "nsu=http://opcfoundation.org/UA/;i=2258"
                },
                // Publisher will request the server at EndpointUrl to sample the node with the OPC sampling interval specified on command line (or the default value: OPC publishing interval)
                // and the subscription will publish the node value with an OPC publishing interval of 4 seconds.
                // Publisher will use for each dinstinct publishing interval (of nodes on the same EndpointUrl) a separate subscription. All nodes without a publishing interval setting,
                // will be on the same subscription and the OPC UA stack will publish with the lowest sampling interval of a node.
                {
                    "ExpandedNodeId": "nsu=http://opcfoundation.org/UA/;i=2258",
                    "OpcPublishingInterval": 4000
                },
                // Publisher will request the server at EndpointUrl to sample the node with the given sampling interval of 1 second
                // and the subscription will publish the node value with the OPC publishing interval specified on command line (or the default value: server revised interval).
                // If the OPC publishing interval is set to a lower value, Publisher will adjust the OPC publishing interval of the subscription to the OPC sampling interval value.
                {
                    "ExpandedNodeId": "nsu=http://opcfoundation.org/UA/;i=2258",
                    // the OPC sampling interval to use for this node.
                    "OpcSamplingInterval": 1000
                }
            ]
        },

        // the format below (NodeId format) is only supported for backward compatibility. you need to ensure that the
        // OPC UA server on the configured EndpointUrl has the namespaceindex you expect with your configuration.
        // please use the ExpandedNodeId format as in the examples above instead.
        {
            "EndpointUrl": "opc.tcp://<your_opcua_server>:<your_opcua_server_port>/<your_opcua_server_path>",
            "NodeId": {
                "Identifier": "ns=0;i=2258"
            }
        }
        // please consult the OPC UA specification for details on how OPC monitored node sampling interval and OPC subscription publishing interval settings are handled by the OPC UA stack.
        // the publishing interval of the data to Azure IoTHub is controlled by the command line settings (or the default: publish data to IoTHub at least each 1 second).
    ]
# Running the Application

## Command line options
The complete usage of the application can be shown using the `--help` command line option and is as follows:

    OpcPublisher.exe <applicationname> [<iothubconnectionstring>] [<options>]

with:

    applicationname: the OPC UA application name to use, required
                     The application name is also used to register the publisher under this name in the
                     IoTHub device registry.

    iothubconnectionstring: the IoTHub owner connectionstring, optional

The following options are supported:

      --pf, --publishfile=VALUE
                             the filename to configure the nodes to publish.
                               Default: './publishednodes.json'
      --sd, --shopfloordomain=VALUE
                             the domain of the shopfloor. if specified this
                               domain is appended (delimited by a ':' to the '
                               ApplicationURI' property when telemetry is
                               sent to IoTHub.
                               The value must follow the syntactical rules of a
                               DNS hostname.
                               Default: not set
      --sw, --sessionconnectwait=VALUE
                             specify the wait time in seconds publisher is
                               trying to connect to disconnected endpoints and
                               starts monitoring unmonitored items
                               Min: 10
                               Default: 10
      --vc, --verboseconsole=VALUE
                             the output of publisher is shown on the console.
                               Default: False
      --ih, --iothubprotocol=VALUE
                             the protocol to use for communication with Azure
                               IoTHub (allowed values: Amqp, Http1, Amqp_
                               WebSocket_Only, Amqp_Tcp_Only, Mqtt, Mqtt_
                               WebSocket_Only, Mqtt_Tcp_Only).
                               Default: Mqtt
      --ms, --iothubmessagesize=VALUE
                             the max size of a message which can be send to
                               IoTHub. when telemetry of this size is available
                               it will be sent.
                               0 will enforce immediate send when telemetry is
                               available
                               Min: 0
                               Max: 256 * 1024
                               Default: 4096
      --si, --iothubsendinterval=VALUE
                             the interval in seconds when telemetry should be
                               send to IoTHub. If 0, then only the
                               iothubmessagesize parameter controls when
                               telemetry is sent.
                               Default: '1'
      --lf, --logfile=VALUE  the filename of the logfile to use.
                               Default: './Logs/<applicationname>.log.txt'
      --pn, --portnum=VALUE  the server port of the publisher OPC server
                               endpoint.
                               Default: 62222
      --pa, --path=VALUE     the enpoint URL path part of the publisher OPC
                               server endpoint.
                               Default: '/UA/Publisher'
      --lr, --ldsreginterval=VALUE
                             the LDS(-ME) registration interval in ms. If 0,
                               then the registration is disabled.
                               Default: 0
      --ot, --operationtimeout=VALUE
                             the operation timeout of the publisher OPC UA
                               client in ms.
                               Default: 120000
      --oi, --opcsamplinginterval=VALUE
                             the publisher is using this as default value in
                               milliseconds to request the servers to sample
                               the nodes with this interval
                               this value might be revised by the OPC UA
                               servers to a supported sampling interval.
                               please check the OPC UA specification for
                               details how this is handled by the OPC UA stack.
                               a negative value will set the sampling interval
                               to the publishing interval of the subscription
                               this node is on.
                               0 will configure the OPC UA server to sample in
                               the highest possible resolution and should be
                               taken with care.
                               Default: 1000
      --op, --opcpublishinginterval=VALUE
                             the publisher is using this as default value in
                               milliseconds for the publishing interval setting
                               of the subscriptions established to the OPC UA
                               servers.
                               please check the OPC UA specification for
                               details how this is handled by the OPC UA stack.
                               a value less than or equal zero will let the
                               server revise the publishing interval.
                               Default: 0
      --ct, --createsessiontimeout=VALUE
                             specify the timeout in seconds used when creating
                               a session to an endpoint. On unsuccessful
                               connection attemps a backoff up to 5 times the
                               specified timeout value is used.
                               Min: 1
                               Default: 10
      --ki, --keepaliveinterval=VALUE
                             specify the interval in seconds the publisher is
                               sending keep alive messages to the OPC servers
                               on the endpoints it is connected to.
                               Min: 2
                               Default: 2
      --kt, --keepalivethreshold=VALUE
                             specify the number of keep alive packets a server
                               can miss, before the session is disconneced
                               Min: 1
                               Default: 5
      --st, --opcstacktracemask=VALUE
                             the trace mask for the OPC stack. See github OPC .
                               NET stack for definitions.
                               To enable IoTHub telemetry tracing set it to 711.

                               Default: 285  (645)
      --as, --autotrustservercerts=VALUE
                             the publisher trusts all servers it is
                               establishing a connection to.
                               Default: False
      --tm, --trustmyself=VALUE
                             the publisher certificate is put into the trusted
                               certificate store automatically.
                               Default: True
      --fd, --fetchdisplayname=VALUE
                             enable to read the display name of a published
                               node from the server. this will increase the
                               runtime.
                               Default: False
      --at, --appcertstoretype=VALUE
                             the own application cert store type.
                               (allowed values: Directory, X509Store)
                               Default: 'X509Store'
      --ap, --appcertstorepath=VALUE
                             the path where the own application cert should be
                               stored
                               Default (depends on store type):
                               X509Store: 'CurrentUser\UA_MachineDefault'
                               Directory: 'CertificateStores/own'
      --tt, --trustedcertstoretype=VALUE
                             the trusted cert store type.
                               (allowed values: Directory, X509Store)
                               Default: Directory
      --tp, --trustedcertstorepath=VALUE
                             the path of the trusted cert store
                               Default (depends on store type):
                               X509Store: 'CurrentUser\UA_MachineDefault'
                               Directory: 'CertificateStores/trusted'
      --rt, --rejectedcertstoretype=VALUE
                             the rejected cert store type.
                               (allowed values: Directory, X509Store)
                               Default: Directory
      --rp, --rejectedcertstorepath=VALUE
                             the path of the rejected cert store
                               Default (depends on store type):
                               X509Store: 'CurrentUser\UA_MachineDefault'
                               Directory: 'CertificateStores/rejected'
      --it, --issuercertstoretype=VALUE
                             the trusted issuer cert store type.
                               (allowed values: Directory, X509Store)
                               Default: Directory
      --ip, --issuercertstorepath=VALUE
                             the path of the trusted issuer cert store
                               Default (depends on store type):
                               X509Store: 'CurrentUser\UA_MachineDefault'
                               Directory: 'CertificateStores/issuers'
      --dt, --devicecertstoretype=VALUE
                             the iothub device cert store type.
                               (allowed values: Directory, X509Store)
                               Default: X509Store
      --dp, --devicecertstorepath=VALUE
                             the path of the iot device cert store
                               Default Default (depends on store type):
                               X509Store: 'My'
                               Directory: 'CertificateStores/IoTHub'
      -h, --help                 show this message and exit

There are a couple of environment variables which can be used to control the application:
* _HUB_CS: sets the IoTHub owner connectionstring
* _GW_LOGP: sets the filename of the log file to use
* _TPC_SP: sets the path to store certificates of trusted stations
* _GW_PNFP: sets the filename of the publishing configuration file

Command line arguments overrule environment variable settings.

Typically you specify the IoTHub owner connectionstring only on the first start of the application. The connectionstring will be encrypted and stored in the platforms certificiate store.
On subsequent calls it will be read from there and reused. If you specify the connectionstring on each start, the device which is created for the application in the IoTHub device registry will be removed and recreated each time.

## Native on Windows
Open the OpcPublisher.sln project with Visual Studio 2017, build the solution and publish it. You can start the application in the 'Target directory' you have published to with:

    dotnet OpcPublisher.dll <applicationname> [<iothubconnectionstring>] [options]

## Using a self-built container
Build your own container and then start the container:

    docker run <your-container-name> <applicationname> [<iothubconnectionstring>] [options]

## Using a container from hub.docker.com
There is a prebuilt container available on DockerHub. To start it, just do:

    docker run microsoft/iot-edge-opc-publisher <applicationname> [<iothubconnectionstring>] [options]

## Important when using a container

### Access to the Publisher OPC UA server
The Publisher OPC UA server listens by default on port 62222. To expose this inbound port in a container, you need to use `docker run` option `-p`:

    docker run -p 62222:62222 microsoft/iot-edge-opc-publisher <applicationname> [<iothubconnectionstring>] [options]

### Enable intercontainer nameresolution
To enable name resolution from within the container to other containers, you need to create a user define docker bridge network and connect the container to this network using the `--network` option.
Additionally you need to assign the container a name using the `--name` option as in this example:

    docker network create -d bridge iot_edge
    docker run --network iot_edge --name publisher microsoft/iot-edge-opc-publisher <applicationname> [<iothubconnectionstring>] [options]

The container can now be reached by other containers via the name `publisher`over the network.

### Access other systems from within the container
Other containers, can be reached using the parameters described in the "Enable intercontainer nameresolution" paragraph.
If operating system on which docker is hosted is DNS enabled, then accessing all systems which are known by DNS will work..
A problems occurs in a network which does use NetBIOS name resolution. To enable access to other systems (including the one on which docker is hosted) you need to start your container using the `--add-host` option,
which effectevly is adding an entry to the containers host file.

    docker run --add-host mydevbox:<IP_ADDRESS>  microsoft/iot-edge-opc-publisher <applicationname> [<iothubconnectionstring>] [options]

### Assigning a hostname
Publisher uses the hostname of the machine is running on for certificate and endpoint generation. docker chooses a random hostname if there is none set by the `-h` option. Here an example to set the internal hostname of the container to `publisher`:

    docker run -h publisher microsoft/iot-edge-opc-publisher <applicationname> [<iothubconnectionstring>] [options]

### Using bind mounts (shared filesystem)
In certain use cases it may make sense to read configuration information from or write log files to locations on the host and not keep them in the container file system only. To achieve this you need to use the `-v` option of `docker run` in the bind mount mode.

### Store for X509 certificates
Storing X509 certificates does not work with bind mounts, since the permissions of the path to the store need to be `rw` for the owner. Instead you need to use the `-v` option of `docker run` in the volume mode.

# Debugging the Application

## Native on Winodws

Open the OpcPublisher.sln project with Visual Studio 2017 and start debugging the app by hitting F5.

If you need to access the OPC UA server in the publisher, you should ensure that the firewall setting allow access to the port the server is listening on (default: 62222).

## In a docker container

Visual Studio 2017 supports debugging of application in docker container. This is done by using docker-compose. Since this does not allow to pass command line parameters it is not convenient.
Another debugging option VS2017 supports is to debug via ssh. In the root of the repository the docker build configuration file `Dockerfile.ssh` can be used to create a SSH enabled container by:

    docker build -f .\Dockerfile.ssh -t publisherssh .

The container can now be started for publisher debugging purposes with:

    docker run -it publisherssh

In the container you need to manually start the ssh daemon with:

    service ssh start

At this point you should be able to create an ssh session as user `root` with the password `Passw0rd`.

To prepare debugging of the application in the container you need to do the following additional steps:

On the host side create a launch.json:

    {
      "version": "0.2.0",
      "adapter": "<path>\\plink.exe",
      "adapterArgs": "root@localhost -pw Passw0rd -batch -T ~/vsdbg/vsdbg --interpreter=vscode",
      "languageMappings": {
        "C#": {
          "languageId": "3F5162F8-07C6-11D3-9053-00C04FA302A1",
          "extensions": [ "*" ]
        }
      },
      "exceptionCategoryMappings": {
        "CLR": "449EC4CC-30D2-4032-9256-EE18EB41B62B",
        "MDA": "6ECE07A9-0EDE-45C4-8296-818D8FC401D4"
      },
      "configurations": [
        {
          "name": ".NET Core Launch",
          "type": "coreclr",
          "cwd": "~/publisher",
          "program": "Opc.Ua.Publisher.dll",
          "args": "<put-the-publisher-command-line-options-here>",

          "request": "launch"
        }
      ]
    }

Build your project and publish it to a directory of your choice.

Use a tool like WinSCP to copy over the published files to the container into the directory `/root/publisher` (this can be also a different directory, but needs to be in sync with the `cdw` property of launch.json.

Now you could start debugging with the following command in Visual Studio's Command Window (View->Other Windows->Command Window):
DebugAdapterHost.Launch /LaunchJson:"<path-to-the-launch.json-file-you-saved>"

repository: lcmstill/iot-edge-opc-publisher

repo_name: lcmstill/iot-edge-opc-publisher


text:
---
modified: 2020-04-16
---

In this guide we walk-through how-to setup the autorole service by using the April
dashboard. below an example of how the setting screen looks like.

![Dashboard autorole](../assets/images/dashboard-autorole-preview.png){.center}

Below are the steeps to setup autorole using dashboard:

1\. Go into autorole service screen in the dashboard, then in case you still
don't have any autorole click on Add button on the right, a modal will pop up and
you'll be asked to select a role, after that click on SAVE CHANGES button.

![Autorole](../assets/images/dashboard-autorole-add.png){.center}

In case the role that you're trying to target is higher than April role then
a message will pop up and April will refuse to add that role as Autorole, in this
case check out your Hierarchy and
place April role higher than the role that you're trying to add by going on
Server Setting and on Role tab drag and drop April Role at the top of the role
you're trying to add.

![Autorole](../assets/images/dashboard-autorole-add-alert.png){.center}

Below an example of how it should like, note that April role is at the top so it
can add any role, but if you prefer you can placing only at the top of the role
that you're trying to add to autorole.

![Autorole](../assets/images/dashboard-autorole-hierarchy-example.png){.center}

After changing your Hierarchy reload the page and try to add it again and if all
went right you should be able to see a screen like below.

![Autorole](../assets/images/dashboard-autorole-added.png){.center}

2\. By default the added autorole get enabled itself, but you can temporally disabled and
re-enable at any time by using the switch of the autorole.

![Autorole](../assets/images/dashboard-autorole-switch.png){.center}

3\. Finally, to remove an Autorole use the X icon on the right of your target autorole
after that a modal will pop up that will ask you if you are sure you want to delete
that autorole, if yes then press SAVE CHANGES button.

![Autorole](../assets/images/dashboard-autorole-delete.png){.center}

branch_name: refs/heads/main

gha_license_id: MIT


text:
---
content_title: Install Prebuilt Binaries
---

[[info | Previous Builds]]
| If you have previously installed lcscs from source using shell scripts, you must first run the [Uninstall Script](01_build-from-source/01_shell-scripts/05_uninstall-lcscs.md) before installing any prebuilt binaries on the same OS.

## Prebuilt Binaries

Prebuilt lcscs software packages are available for the operating systems below. Find and follow the instructions for your OS:

### Mac OS X:

#### Mac OS X Brew Install
```sh
brew tap lcscs/lcscs
brew install lcscs
```
#### Mac OS X Brew Uninstall
```sh
brew remove lcscs
```

### Ubuntu Linux:

#### Ubuntu 18.04 Package Install
```sh
wget https://github.com/lcscs/scs/releases/download/v2.0.7/lcscs_2.0.7-1-ubuntu-18.04_amd64.deb
sudo apt install ./lcscs_2.0.7-1-ubuntu-18.04_amd64.deb
```
#### Ubuntu 16.04 Package Install
```sh
wget https://github.com/lcscs/scs/releases/download/v2.0.7/lcscs_2.0.7-1-ubuntu-16.04_amd64.deb
sudo apt install ./lcscs_2.0.7-1-ubuntu-16.04_amd64.deb
```
#### Ubuntu Package Uninstall
```sh
sudo apt remove lcscs
```

### RPM-based (CentOS, Amazon Linux, etc.):

#### RPM Package Install
```sh
wget https://github.com/lcscs/scs/releases/download/v2.0.7/lcscs-2.0.7-1.el7.x86_64.rpm
sudo yum install ./lcscs-2.0.7-1.el7.x86_64.rpm
```
#### RPM Package Uninstall
```sh
sudo yum remove lcscs
```

## Location of lcscs binaries

After installing the prebuilt packages, the actual lcscs binaries will be located under:
* `/usr/opt/lcscs/<version-string>/bin` (Linux-based); or
* `/usr/local/Cellar/lcscs/<version-string>/bin` (MacOS)

where `version-string` is the lcscs version that was installed.

Also, soft links for each lcscs program (`nodscs`, `clscs`, `kscsd`, etc.) will be created under `usr/bin` or `usr/local/bin` to allow them to be executed from any directory.

## Previous Versions

To install previous versions of the lcscs prebuilt binaries:

1. Browse to https://github.com/lcscs/scs/tags and select the actual version of the lcscs software you need to install.

2. Scroll down past the `Release Notes` and download the package or archive that you need for your OS.

3. Follow the instructions on the first paragraph above to install the selected prebuilt binaries on the given OS.

repository: lcscs/scs

repo_name: lcscs/scs