text:
![kolla_intro_image][kolla_intro_image]

# kolla-ansible investigation

**kolla-ansible** is a tool to manage deployment of containerized OpenStack
with the aid of ansible. In other words, using that to deploy only monasca
next to cloud that it will monitor is not exactly the use case of Kolla.

    That assumption was confirmed by one of the kolla members.

## Setup

Setup is really tricky and constrained by kolla-prechecks
```kolla-ansible prechecks```. Biggest problems encountered were related
network setup:

* having one or two interfaces (or more) to control the network traffic in/out
deployed nodes
* having full visibility in the network between nodes
* having *kolla_internal* and *kolla_external* addresses to be available
on different networks if the *kolla_internal_interface* and
*kolla_external_interface* differs

Another set of problems were presented by the proxy.
If proxy is part of an environment, a setup fo kolla has to be carefully
reviewed and all IPs that will be used to communicate has to be put into
```NO_PROXY``` environmental variable.

Example from [Vagrantfile](./Vagrantfile):

```ruby
PUBLIC_IP="<IP_ADDRESS>"

KOLLA_VIP_INTERNAL='<IP_ADDRESS>'
KOLLA_VIP_EXTERNAL=PUBLIC_IP.gsub('25','33')

NO_PROXY = [KOLLA_VIP_INTERNAL, '<IP_ADDRESS>', PUBLIC_IP, KOLLA_VIP_EXTERNAL]

if Vagrant.has_plugin?("vagrant-proxyconf")
    if ENV["no_proxy"]
      config.proxy.no_proxy = ENV["no_proxy"] + ',' + NO_PROXY.join(',')
    end
end
```

    Note that, best results in kolla (and in general containers) are
    obviously achievable while using no proxy internet access.

## Environments

*kolla* covers pretty much all the operating systems where it is possible
to run docker containers. After all, **kolla** aims at being deployment manager.

However, it is worth mentioning that it is possible to pick the *Linux Distro*
that will be used as a foundation for
single container: ```centos, rhel, ubuntu, oraclelinux, debian```. Not to
mention that it flawlessly supports building

More details about image building can be found here [kolla_image_building]

## In CI

Since *kolla* deploys OpenStack using containers, it is possible to use that
inside any CI setup as it will be lightweight and fast (once all the images
are downloaded).

Development of X application would require rebuilding selected images prior
to deployment. It is achievable through *docker* feature: **own docker registry**
that keeps all working/stable images and only upon marges new versions
(even using the same tag) will enter the registry replacing the old ones.

    Note that this description can be applied to each solution that is
    somehow based on docker, as long as it is an option to point at
    internal docker registry.

## Upgrading the components

For now, everything that could be found about upgrading components with kolla
drilled down to following blueprint(s) and wiki(s):

* [graceful-upgrade]
* [kolla-upgrading]
* [kolla-operating]

Each service and each container have own tag that allows to set the desired
version of what exactly we wish to deploy.

It is worth mentioning now that images can be **source** or **binary** type.
That basically means that we can control what tag (version) of correspnding
upstream repository will be used to create image.
In other words, let's take nova image and upstream [repo](https://github.com/openstack/nova).
There will be certain tags that we could take to build **source** image.
For **binary** everything depends on the packagers/owners of the repositories
that were used in build process.

## Scaling

Scalling with **kolla-ansible** is done via ansible. If there is a need
to scale up/down, one must provide modified inventory file and redeploy.

## As development environment

**kolla* introduces development mode, where it will behave similar to
[devstack]. Here's the link to relevant [blueprint](kolla_mount_sources_bp).
This will be the best option (IMHO, looking at already implemented changes).

Other than that, it seems that it is possible to follow that path in more
manual way. This [build-openstack-from-source] part of kolla's documentation
shows that we can mount git repository and appropriate change from gerrit
for given container.

An example:

```ini
[monasca-api]
type=git
location=https://git.openstack.org/openstack/monasca-api
reference=refs/changes/20/473920/2
```

WARNING: This feature is somehow not working right now.
Guess someone (me <sick>) needs to send patch.

However, still we can mount all repositories into vagrant machine
or use kolla directly and build from sources using

```ini
[monasca-api]
type=local
location=/dev/monasca-api
```

NOTE: That does not require setting any references.
It purely reads the directory and install it.
Somehow similar to ```pip install /dev/monasca-api```.

## Community

### Status

Kolla seems to be quite active. There are new commits coming in and a lot of
blueprints.

* [all](https://blueprints.launchpad.net/kolla)
* [monasca](https://blueprints.launchpad.net/kolla?searchtext=monasca)

    Interestingly enough, containers for *kafka*, *zookeeper* were added
    as part of *monasca blueprint*

### Impact

Community has already shown the interest in having **monasca** inside
kolla. The oppurtunity lies there, just need to pick it up.

## Development effort

Most of the job is already done by kolla, therefore writing containers and
"glue" layer in **kolla-ansible** seems to be the only effort needed.
Best to mention: once the "glue" part is done, all that needs to be done is
to maintain containers.

However, when it comes to binary containers, one would have to pay attention
to packages (RPMs or DEBs, depends on the distro containers is built on top of)

## Maintenance

See above in [Development effort](#dev_effort)

## Risks

### Non-python components

Putting in storm and it's topologies will require more effort as we need
to provide already clustered deployment that includes all storm components
(ui, nimbus, supervisor and logviewer). Not to mention of having
a one-shot container that will upload the required topologies and exit.

WARNING: Still the problem lies with topology itself. Where we should
compile that? Should we compile that at all inside container environment?
Most likely kolla's community will also want to have storm to support
more than just one topology. However that question lies unanswered.

### Development mode

Prior to implementing **development mode**, all the containers and ansible
layer must be developed first. It is hard to tell, if that requires providing
only **source** containers or also the other types (**rdo**, **binary**, etc.).

### Lack of certain containers/roles

For deploying **monasca**, first containers/roles for following components must
be implemented:

* kafka (container available [kafka_container])
* zookeeper (container available [zookeeper_container])

Other dependencies are available.

### Deploying next to existing cloud

*kolla* mission is to provide unified deployment mechanism of
containerized OpenStack as **entire cloud**. According to members of kolla,
it might not work well for cases, where something needs to be deployed next to
existing cloud.

The statement sound correct enough. That is based on the fact that **ansible**
is used as a coordinator. All the necessary information
(IPs, versions, networking etc) are part of *ansible variables* hence available
for all the *ansible roles*. A role is final entity that is responsible for
all sorts of operation on a container (regardless if that is starting or
stopping it).

Theoretically, and just in that way, it should (as ansible supports that) provide
another set of group and/or host variables where details how to connect
to exisitng cloud are set.

[kolla_mount_sources_bp]: https://blueprints.launchpad.net/kolla/+spec/mount-sources
[devstack]: https://github.com/openstack-dev/devstack
[kafka_container]: https://github.com/openstack/kolla/containers/kafka
[zookeeper_container]: https://github.com/openstack/kolla/containers/zookeeper
[kolla_intro_image]: https://image.slidesharecdn.com/containerclustering-pub-160530043919/95/managing-container-clusters-in-openstack-native-way-15-638.jpg?cb=1464583250
[kolla_image_building]: https://docs.openstack.org/developer/kolla/image-building.html
[graceful-upgrade]: https://blueprints.launchpad.net/kolla/+spec/graceful-upgrade
[kolla-upgrading]: https://docs.openstack.org/developer/kolla-ansible/operating-kolla.html#upgrading
[kolla-operating]: https://github.com/openstack/kolla-ansible/blob/master/doc/operating-kolla.rst

repository: kornicameister/journeys

repo_name: kornicameister/journeys


text:
# Rocketman
![rocketman](./rocketman.jpg)
<sub>*yes, I know it says Starman on the image*</sub>
> *🎶 And I think it's gonna be a long long time 'Till touch down brings me round again to find 🎶*

Rocketman is a gem that introduces Pub-Sub mechanism within your Ruby code.

The main goal of Rocketman is not to replace proper message buses like Redis PubSub/Kafka, but rather be a stepping stone. You can read more about the [rationale behind the project](https://github.com/edisonywh/rocketman#why-use-rocketman-rather-than-a-proper-message-bus-eg-redis-pubsubkafka) down below.

As with all Pub-Sub mechanism, this greatly decouples your upstream producer and downstream consumer, allowing for scalability, and easier refactor when you decide to move Pub-Sub to a separate service.

Rocketman also works without Rails.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'rocketman'
```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install rocketman

## Usage

Rocketman exposes two module, `Rocketman::Producer` and `Rocketman::Consumer`. They do exactly as what their name implies. All you need to do is `include Rocketman::Producer` and `extend Rocketman::Consumer` into your code.

### Producer
Producer exposes one **instance** method to you: `:emit`. `:emit` takes in the event name and an optional payload and publishes it to the consumers. There's nothing more you need to do. The producer do not have to know who its consumers are.

```ruby
class Producer
  include Rocketman::Producer

  def hello_world
    emit :hello, payload: {"one" => 1, "two" => 2}
  end
end
```

Note that Producer emit events with threads that run in a thread pool. The default number of worker is 5, and the workers default to checking the job with a 3 seconds interval. You can tweak these to your liking, refer to the [`Configuration` section](https://github.com/edisonywh/rocketman#configuration) below for more informations.

### Consumer
Consumer exposes a **class** method, `:on_event`. `:on_event` takes in the event name, and also an additional block, which gets executed whenever a message is received. If an additional `payload` is emitted along with the event, you can get access to it in the form of block argument.

```ruby
class Consumer
  extend Rocketman::Consumer

  on_event :hello do |payload|
    puts "I've received #{payload} here!"
    # => I've received {:payload=>{"one"=>1, "two"=>2}} here!
  end
end
```

Simple isn't it?

#### Consume events from external services (Relays)

If you want to also consume events from external services, you can do that too.

Rocketman has the concept of a `Relay`. The cool thing is, `Relay`s are just `Producer`s that understand how to relay messages from external services (like `Redis`) into Rocketman events.

Rocketman ships with a `Redis` relay, which you can use it like so (assuming you have Redis installed):

```ruby
require 'rocketman/relay/redis' # This is not required by default, so you need to explicitly require it.
Rocketman::Relay::Redis.new.start(Redis.new)
```

> **NOTE**: You should always pass in a **new, dedicated** connection to `Redis` to the Redis relay. This is because `redis.psubscribe` will hog the whole Redis connection (not just Ruby process), so `Relay` expects a dedicated connection for itself.

That's it, the `Redis` relay service will now listen for events from external services on behalf of you, and then it'll push those events onto the internal `Registry`.

It'll translate the following:

```
redis-cli> PUBLISH hello payload
```

to something understandable by your consumer, so a consumer only has to do:

```ruby
on_event :hello do |payload|
  puts payload
end
```

Notice how it behaves exactly the same as if the events did not come from Redis :)

**This pattern is powerful because this means your consumers do not have to know where the events are coming from, as long as they're registed onto `Registry`.**

Right now, only `Redis` is supported, but it is extremely easy to add a `Relay` yourself since it's just a `Producer`. Checkout the implementation of `rocketman/relay/redis` for reference, upstream contributions for services are very welcomed too.

## Persisting emitted events

By default, the events emitted from your app will be stored in an in-memory `Queue`, which will get processed by Rocketman threaded workers.

However this also means that if your app dies with events still in your job queue, your emitted events which are stored in-memory will be lost.

That is obviously not desirable, so that's why **Rocketman ships with an option to use `Redis` as your backing storage mechanism.**

All you need to do is pass in a `Redis` connection to Rocketman. Refer to the [`Configuration` section below](https://github.com/edisonywh/rocketman#configuration) for more information.

## Configuration

Here are the available options to tweak for Rocketman.

```ruby
# config/initializers/rocketman.rb

Rocketman.configure do |config|
  config.worker_count = 10 # defaults to 5
  config.latency      = 1  # defaults to 3, unit is :seconds
  config.storage      = Redis.new # defaults to `nil`
  config.debug        = true # defaults to `false`
end
```

Currently `storage` only supports `Redis`, suggestions for alternative backing mechanisms are welcomed.

`debug` mode enables some debugging `puts` statements, and also tweak the `Thread` workers to `abort_on_exception = true`. So if you have failing jobs, this is how you can figure out what's happening inside your workers.

## Why use `Rocketman`, rather than a proper message bus (e.g Redis PubSub/Kafka)?

It is worth noting that `Rocketman` is not meant to be a replacement for the aforementioned projects -- both Redis PubSub and Kafka are battle-tested and I highly encourage to use them if you can.

**But**, `Rocketman` recognizes that it's not an easy task to spin up an external message bus to support event-driven architecture, and that's what it's trying to do - to be a stepping stone for eventual greatness.

Moving onto a event-driven architecture is not an easy task - your team has to agree on a message bus, the DevOps team needs the capacity to manage the message bus, and then what about clustering? failovers?

So what Rocketman offers you is that you can start writing your dream-state event-driven code **today**, and when the time comes and your team has the capacity to move to a different message bus, then it should be a minimal change.

## Architecture

Here's a very crude drawing of the architecture of Rocketman

![erd](./erd.png)

## Roadmap

Right now events are using a `fire-and-forget` mechanism, which is designed to not cause issue to producers. However, this also means that if a consumer fail to consume an event, it'll be lost forever. **Next thing on the roadmap is look into a retry strategy + persistence mechanism.**

~~Emitted events are also stored in memory in `Rocketman::Pool`, which means that there's a chance that you'll lose all emitted jobs. Something to think about is to perhaps move the emitted events/job queue onto a persistent storage, like Redis for example.~~ **Redis support is now available!**

The interface could also probably be better defined, as one of the goal of Rocketman is to be the stepping stone before migrating off to a real, proper message queue/pub-sub mechanism like Kafka. **I want to revisit and think about how can we make that transition more seamless.**

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are **very welcomed** on GitHub at https://github.com/edisonywh/rocketman, but before a pull request is submitted, **please first open up an issue** for discussion.

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

## License

The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

## Code of Conduct

Everyone interacting in the Rocketman project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/rocketman/blob/master/CODE_OF_CONDUCT.md).

## Why is it called Rocketman?

Uh.. well it's named after the song by Elton John, but really, it has nothing to do with an actual Rocketman.

license: MIT

repository: benjis/rocketman

branch_name: refs/heads/master

extension: md

license_type: permissive

repo_name: benjis/rocketman


text:
## Endpoint Request

> This template is provided to allow for requests to be made to add additional endpoints to the Transportation Systems 2018 API. Submission of this form is not a guarentee this will be added. Please note we are a volunteer based organization and changes can only be made as time and resources allow. Any changes will need to align with the mission of Hack Oregon and the Civic Software Foundation.

1. Is this a request for a new endpoint or update/change to existing (Y/N) (If N open a basic issue):

2. In your words, describe the data you would like to retrieve:

3. What is your intent behind the data request? For example a research project, back a website, for a class?
If you are planning on integrating into a website/live project which would incur ongoing use, please provide additional details on estimated use.

4.  If this request is directly related to the Civic Software Foundation or Hack Oregon, please note.

5. Do you have a timeline in which to fulfill this request?

6. Is there anything else important to know? Do you have certain units of measure/metrics you are attempting to address?

branch_name: refs/heads/staging

gha_language: Python

gha_license_id: MIT


text:
---
title: Linux内嵌链表(<sys/queue.h>)
date: 2021-01-25
categories:
- Technical-records
tags:
- Linux
---

C is not same as C++. Any structure must be realized on its own. There is a "queue" relies on "<sys/queue.h>".
Any detail can read the "man 3 queue".

These macros define and operate on four types of data structures:
1. singly-linked lists
2. singly-linked tail queues
3. lists
4. tail queues

All four structures support the following functionality:

1.   Insertion of a new entry at the head of the list.
2.   Insertion of a new entry after any element in the list.
3.   O(1) removal of an entry from the head of the list.
4.   Forward traversal through the list.
5.   Swapping the contents of two lists.

Every data structures have its own characteristics.

## Singly-linked lists

Singly-linked lists are the simplest of the four data structures and support only the above functionality.  Singly-
linked lists are ideal for applications with large datasets and few or no removals, or for implementing a LIFO queue.
Singly-linked lists add the following functionality:

1.   O(n) removal of any entry in the list.

SLIST_EMPTY(SLIST_HEAD *head);      //Test the list is empty.

SLIST_ENTRY(TYPE);      //      Help you to create the "next" point.

SLIST_FIRST(SLIST_HEAD *head);      // Help you to create the head of list.

SLIST_FOREACH(TYPE *var, SLIST_HEAD *head, SLIST_ENTRY NAME);      // Help you to forward the list

SLIST_HEAD(HEADNAME, TYPE);

SLIST_HEAD_INITIALIZER(SLIST_HEAD head);

SLIST_INIT(SLIST_HEAD *head);

SLIST_INSERT_AFTER(TYPE *listelm, TYPE *elm, SLIST_ENTRY NAME);

SLIST_INSERT_HEAD(SLIST_HEAD *head, TYPE *elm, SLIST_ENTRY NAME);

SLIST_NEXT(TYPE *elm, SLIST_ENTRY NAME);

SLIST_REMOVE_HEAD(SLIST_HEAD *head, SLIST_ENTRY NAME);

SLIST_REMOVE(SLIST_HEAD *head, TYPE *elm, TYPE, SLIST_ENTRY NAME);

### Use it

The macros is designed to be easy to use.

**Test code**

```c
#include <sys/queue.h>
#include <stdio.h>

struct my_list
{
    int data_1;
    int data_2;
    SLIST_ENTRY(my_list) next;
};

SLIST_HEAD(my_slist_head, my_list) my_slist_head;

int main()
{
    SLIST_INIT(&my_slist_head);

    struct my_list *var = NULL;

    struct my_list list_1 = {
        .data_1 = 1,
        .data_2 = 2,
    };
    struct my_list list_2 = {
        .data_1 = 3,
        .data_2 = 4,
    };

    SLIST_INSERT_HEAD(&my_slist_head, &list_1, next);

    SLIST_INSERT_AFTER(&list_1, &list_2, next);

    SLIST_REMOVE_HEAD(&my_slist_head, next);

    if(! SLIST_EMPTY(&my_slist_head)){
        SLIST_FOREACH(var, &my_slist_head, next){
            printf("data_1 = %d \t data_2 = %d\n", var->data_1, var->data_2);
        }
    }
    return 0;
}

/* OUTPUT */
data_1 = 3 	 data_2 = 4

```

## Singly-linked tail queues

Singly-linked tail queues add the following functionality:

1.   Entries can be added at the end of a list.
2.   O(n) removal of any entry in the list.
3.   They may be concatenated.

However:

1.   All list insertions must specify the head of the list.
2.   Each head entry requires two pointers rather than one.
3.   Code size is about 15% greater and operations run about 20% slower than singly-
    linked lists.

Singly-linked tail queues are ideal for applications with large datasets and few or no
removals, or for implementing a FIFO queue.

STAILQ_CONCAT(STAILQ_HEAD *head1, STAILQ_HEAD *head2);

STAILQ_EMPTY(STAILQ_HEAD *head);

STAILQ_ENTRY(TYPE);

STAILQ_FIRST(STAILQ_HEAD *head);

STAILQ_FOREACH(TYPE *var, STAILQ_HEAD *head, STAILQ_ENTRY NAME);

STAILQ_HEAD(HEADNAME, TYPE);

STAILQ_HEAD_INITIALIZER(STAILQ_HEAD head);

STAILQ_INIT(STAILQ_HEAD *head);

STAILQ_INSERT_AFTER(STAILQ_HEAD *head, TYPE *listelm, TYPE *elm, STAILQ_ENTRY NAME);

STAILQ_INSERT_HEAD(STAILQ_HEAD *head, TYPE *elm, STAILQ_ENTRY NAME);

STAILQ_INSERT_TAIL(STAILQ_HEAD *head, TYPE *elm, STAILQ_ENTRY NAME);

STAILQ_NEXT(TYPE *elm, STAILQ_ENTRY NAME);

STAILQ_REMOVE_HEAD(STAILQ_HEAD *head, STAILQ_ENTRY NAME);

STAILQ_REMOVE(STAILQ_HEAD *head, TYPE *elm, TYPE, STAILQ_ENTRY NAME);


text:
abstract-service-loader
=======================

Abstract Service Loader class to wrap java.util.ServiceLoader

[![Build Status](https://secure.travis-ci.org/ansell/abstract-service-loader.svg?branch=master)](https://travis-ci.org/ansell/abstract-service-loader)

[![Coveralls Test Coverage Status](https://coveralls.io/repos/ansell/abstract-service-loader/badge.svg)](https://coveralls.io/r/ansell/abstract-service-loader)

[![Coverity Security Scan Build Status](https://scan.coverity.com/projects/4817/badge.svg)](https://scan.coverity.com/projects/4817)

repository: ansell/abstract-service-loader

repo_name: ansell/abstract-service-loader


text:
## componentWillReceiveProps Not Triggered After Mounting [Back](./../react.md)

`componentWillReceiveProps` isn't triggered after the node is put on scene. This is by design. Check out other lifecycle methods for the one that suits your needs.

The reason for that is because `componentWillReceiveProps` often handles the logic of comparing with the old props and acting upon changes; not triggering it at mounting (where there are no old props) helps in defining what the method does.

repository: YYY-TEAM/PersonalWiki

repo_name: YYY-TEAM/PersonalWiki


text:
# Laravel Serve LiveReload

>**This package is daed, please use the new [Vite integrated](https://laravel.com/docs/9.x/vite#introduction) instead**

This package integrates into default `artisan serve` command an WebSockets server for live reloading the application when any file changed. How it works, checkout my [blog post](https://daudau.cc/posts/laravel-live-reload-without-javascript) for explaination if you don't like digging into source code.

![Laravel serve livereload](demo.gif)

## Installation

For laravel 8, please use version ^1.x, and below use version 0.x

`composer require bangnokia/laravel-serve-livereload --dev`

## Usage

Open terminal and run `php artisan serve`

This package works even when you use custom vhost such as `valet` or `laragon`

## Configuration

By default, this package looking for files changes in these directories:

```
/app
/public
/config
/routes
/resources
```

If you want to customize the watched forlders, you can publish the configuration file by this commmand:

```bash
php artisan vendor:publish --provider="BangNokia\ServeLiveReload\CommandServiceProvider"
```

and then you can config what you want in the `config/serve_livereload.php`.

repository: bangnokia/laravel-serve-livereload

gha_language: PHP

repo_name: bangnokia/laravel-serve-livereload


text:
2.36 - John Henry Thompson - Invent Your Future

[Yoga](../../../yoga.md) > [Patanjali's Yoga Sutras - A Comparison](../../patanjani.md) > [Book 2](../book-2.md) >

### 2.36

**सत्यप्रतिष्थायं क्रियाफलाश्रयत्वम् ॥३६॥**
satya-pratiṣthāyaṁ kriyā-phala-āśrayatvam ||36||

[\[RS\] 2.36 Once a state of truth (satya) has been permanently established, each statement will form the basis for a truthful result.](http://www.ashtangayoga.info/philosophy/yoga-sutra-patanjali/chapter-2/item/satya-pratishthayam-kriya-phala-ashrayatvam/)

[\[JW\] 2.36 As soon as he is grounded in abstinence from falsehood, actions and consequences depend upon him.](http://books.google.com/books?id=YzFImjtOxUwC&pg=PA186&ci=155%2C568%2C759%2C73&source=bookclip)

[\[SS\]](http://www.amazon.com/Yoga-Sutras-Patanjali-Commentary-Satchidananda/dp/0932040381) 2.36 To one established in truthfulness, actions and their results become subservient. \[p131\]

[\[EB\]](http://www.amazon.com/Yoga-Sutras-Patanjali-Translation-Commentary/dp/0865477361/ref=sr_1_1?ie=UTF8&s=books&qid=1250508322&sr=1-1) 2.36 When one is established in truthfulness, one ensures the fruition of actions. \[p262\]

[<Page 2.35](235.md)  [Page 2.37>](237.md)

(**सत्य**, satya) = truthfulness
(**प्रतिष्ठम्**, pratiṣṭham) = fixed; permanent; stable
(**क्रिया**, kriyā) = action; statement
(**फल**, phala) = outcome; result
(**आश्रयत्वम्**, āśrayatvam) = basis; foundation; support

gha_language: HTML


text:
# Electronics
Electronics App for logic circuts simulation

- All code located in Source folder

- Required packages located in packages folder (also can be downloaded via Nuget)

- Downloadable version of program located in release folder (.dll files included)

- Scheme examples can be found in example folder

![preview](https://user-images.githubusercontent.com/40578274/57950465-fdacfc00-78ef-11e9-88b2-1f75bbd62f83.png)

![preview](https://user-images.githubusercontent.com/40578274/57950337-a575fa00-78ef-11e9-8cf8-cd81f4ddeef9.png)

repository: jellyr/Electronics

repo_name: jellyr/Electronics


text:
Future Converter
================

Converts between various future types, [RxJava](https://github.com/Netflix/RxJava) Observables,
Spring 4 [ListenableFuture](http://docs.spring.io/spring/docs/4.0.0.BUILD-SNAPSHOT/javadoc-api/org/springframework/util/concurrent/ListenableFuture.html),
Java 8 [CompletableFuture](http://download.java.net/lambda/b88/docs/api/java/util/concurrent/CompletableFuture.html) and
Guava [ListenableFuture](http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html).

Please note that the conversion is not always straightforward. Especially RxObservables are completely different concept than
Futures. Nevertheless, the conversion is more or less possible.

I am aware of the following quirks:

* Observable can produce multiple values. When converting to Future, we take the first value
* It is not [possible to cancel CompletableFuture](http://stackoverflow.com/questions/23320407/how-to-cancel-java-8-completable-future) if it's blocked.

The project has pretty good test coverage, but testing asynchronous stuff is tricky. If you find any bug, please let me know.

# spring-java8
Converts between Spring 4 [ListenableFuture](http://docs.spring.io/spring/docs/4.0.0.BUILD-SNAPSHOT/javadoc-api/org/springframework/util/concurrent/ListenableFuture.html) and Java 8 [CompletableFuture](http://download.java.net/lambda/b88/docs/api/java/util/concurrent/CompletableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-spring-java8</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.springjava.FutureConverter.*;

    ...
    CompletableFuture<String> completable = toCompletableFuture(listenable);
    ...
    ListenableFuture<String> listenable = toListenableFuture(completable);

# spring-guava
Converts between Spring 4 [ListenableFuture](http://docs.spring.io/spring/docs/4.0.0.BUILD-SNAPSHOT/javadoc-api/org/springframework/util/concurrent/ListenableFuture.html)
and Guava [ListenableFuture](http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-spring-guava</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.springguava.FutureConverter.*;

    ...
    com.google.common.util.concurrent.ListenableFuture<String> guavaListenableFuture
            = toGuavaListenableFuture(springListenableFuture);
    ...
    org.springframework.util.concurrent.ListenableFuture<String> springListenableFuture
            = toSpringListenableFuture(guavaListenableFuture);

# java8-guava
Converts between Java 8 [CompletableFuture](http://download.java.net/lambda/b88/docs/api/java/util/concurrent/CompletableFuture.html)
and Guava [ListenableFuture](http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-java8-guava</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.java8guava.FutureConverter.*;

    ...
    ListenableFuture<String> guavaListenableFuture = toListenableFuture(completable);
    ...
    CompletableFuture<String> completable = toCompletableFuture(listenable);;

#RxJava
Please note that conversion from/to RxJava Observables is not straightforward.

* When converting Observable to a Future, only one element can be produced by the Observable. If your observable produces
multiple values, please limit it using `observable.take(1)`.
* When converting a Future to an Observable, it's not clear what should happen upon unsubscribe. Since version 0.2.2 RxJava support does
not cancel the Future, since there is no good place to keep track of the subscriptions (there may be multiple subscriptions for any given Future).
* Converting Observable to a Future registers exactly one subscription which is unsubscribed upon Future cancellation.

# rxjava-java8
Converts between [RxJava](https://github.com/Netflix/RxJava) Observables and Java 8 [CompletableFuture](http://download.java.net/lambda/b88/docs/api/java/util/concurrent/CompletableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-rxjava-java8</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.java8rx.FutureConverter.*;

    ...
    CompletableFuture<String> completable = toCompletableFuture(observable);
    ...
    Observable<String> observable = toObservable(completable);

# spring-rxjava
Converts between [RxJava](https://github.com/Netflix/RxJava) Observables and Spring 4 [ListenableFuture](http://docs.spring.io/spring/docs/4.0.0.BUILD-SNAPSHOT/javadoc-api/org/springframework/util/concurrent/ListenableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-spring-rxjava</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.springrx.FutureConverter.*;

    ...
    ListenableFuture<String> listenable = toListenableFuture(observable);
    ...
    Observable<String> observable = toObservable(listenable);

# guava-rxjava
Converts between [RxJava](https://github.com/Netflix/RxJava) Observables and Guava [ListenableFuture](http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html)

Import the dependency

    <dependency>
        <groupId>net.javacrumbs.future-converter</groupId>
        <artifactId>future-converter-guava-rxjava</artifactId>
        <version>0.3.0</version>
    </dependency>

And then use

    import static net.javacrumbs.futureconverter.guavarx.FutureConverter.*;

    ...
    ListenableFuture<String> listenable = toListenableFuture(observable);
    ...
    Observable<String> observable = toObservable(listenable);

repository: tigerMoon/future-converter

repo_name: tigerMoon/future-converter


text:
Reverse the order of words in a given string  `sentence`. You can assume that  `sentence`  does not have any leading, trailing or repeating spaces.

Example

For  `sentence = "Man bites dog"`, the output should be
`reverseSentence(sentence) = "dog bites Man"`.

Input/Output

-   **[execution time limit] 3 seconds (cs)**

-   **[input] string sentence**

    A string consisting of letters and spaces.

    _Guaranteed constraints:_
    `1 ≤ sentence.length ≤ 2 · 104`.

-   **[output] string**

repository: M-Ekrim/codeSignal

repo_name: M-Ekrim/codeSignal


text:
# GrowersBrains
GrowersBrains is an app designed to solve the problems that growers globally are Facing and to bring expert growing techniques to everyone. Grow Green and Grow Clean.

As a community project that will enable collaborators to:
- Sharpen their skill sets
- Learn new languages
- Work in a team enviorment
- Get real world experience
- Make something that helps the world
- learn all about the process of taking an app from idea to product
- and most imporantly, have fun along the way.

## Motivation
The way that I came up with the idea, is that I have 15 years of experience of growing plants. Starting in middle school horticulture class at 11 to now at 26, I've learned alot from books and the internet, but the place that I have always learned the most from was other growers teaching me the techniques that took them years to develop and perfect.

That lead me to the idea of creating an app that brings a collective consciousness of Growers together to change the way the world grows. Growing plants can be very expensive and can lead to great losses if not done properly. When done properly it can be a fun, exciting and relaxing way to control the quality of the food or medicine that you are growing and ensure that it is as clean and pure as possible.

## Tech Stack
- React Native
- Node
- MongoDB
- Redux

## Features & Designs
The current design that I have for the app is to have eight different sections.

Sections:
1) Welcome page that users would have to login to.
2) Growers Rankings- create rankings of the best growers in the world that could help people with one on one consulting.
3) Daily Grow- A daily picture of the plants a user is growing allowing to keep an eye on the plants and make sure their healthy, allows to diagnose any problems that arise.
4) Greenhouse-"What's in your Greenhouse" A locker of all the plants a person is growing and the data of what happens daily, once the plant is done growing a timelapse gets generated.
5) How to- Articles and videos from the top growers to teach every user the A to Z of growing.
6) Consulting- Allows top growers an opportunity to connect with new users to help them create a plan to make sure they optimize their grow and everything runs smoothly.
7) Competition - A daily, weekly, and monthly competition to see who the best grower is in many different categories. 8)Best products- Links and articles reviewing the best products for growers to use that are peer reviewed and approved. With affiliate codes included.

All of these features would allow for great data to be collected and machine learning to be created to diagnose plant problems, and make elite growing accessible to everyone no matter their experience level.

## Getting Started / Installtion
If you want to join send @nah a message in discord.

https://expo.io/learn follow the steps they list.

Trello board: https://trello.com/invite/b/1VgFXdo7/0135f2a7507e6062eb03f23bcf76ae3c/growers-brains

Project Lead: @nah

Project Manager: @Kael0527

Database Lead:@Vali

React Native Lead: @Gajendra

## To Contribute here(Version control)
Step 1: Click on Fork at the top right corner

Step 2: Clone your forked repository

Step 3: cd into the cloned folder | GrowersBrains

Step 4: git remote add upstream https://github.com/zero-to-mastery/GrowersBrains

Step 5: git pull upstream "branch"

Step 6: Check out to the task branch | git checkout -b <NAME_OF_THE_TASK>

Creating a pull request(when done with your code/changes)
Step 1: Run: git add .

Step 2: Run: git commit -m ""

Step 3: git push origin <BRANCH_NAME>

Go to the repository https://github.com/zero-to-mastery/GrowersBrains

As soon as you get there, you are going to see a green ‘compare and create a pull request’

Click on it, and type your message, click on create pull request

Thanks to: @chisombiri

## Handy Resources

https://reactnative.dev/docs/getting-started

https://www.google.com/amp/s/www.nbcnews.com/better/amp/ncna781806

https://www.google.com/amp/s/www.vox.com/platform/amp/the-highlight/2019/12/30/21031913/how-to-keep-houseplants-alive-masterclass-plants-swiss-cheese-millennials-plantfluencer

repository: mansurissa/GrowersBrains

repo_name: mansurissa/GrowersBrains


text:
---
layout: post
title:  "My first blogpost - Beijing Land Transaction"
date:   2015-03-23 22:34:25
categories: jekyll update
tags: featured
image: /assets/article_images/2015-03-23-Beijing-land-transaction/beijing_watercolor.jpeg
---

##Beijing Land Transaction 2003-2013
===================

###Source:

I was curious about visualization and geocoding work people have done in China, and I stumbled upon this [Beijing City Lab](http://www.beijingcitylab.com/). The Lab aims to quantify urban dynamics and generates new insights for urban planning and governance, with a focus on the capital Beijing.

While the projects are pretty impressive, I found the cover map of [Data 2](http://www.beijingcitylab.com/data-released-1/data1-20/) could be more informative.

It looks like this:
![](http://casey-huang.github.io/assets/article_images/2015-03-23-Beijing-land-transaction/original_image.jpg)

###What's wrong:

Dot plot on a map is not a good choice here:
> - The details of transactions besides geo-location are all left out, merely any information
> - When the dots are overlapped with each other, even density is hard to tell
> - Impossible to observe trends

###First look at the dataset:

After downloading the [shapefiles](https://www.dropbox.com/s/gc93o1iz7mcu6j9/DT2.rar), I want to use github's support with .geojson files to get a close look of data. Here's how I convert it:

```
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install gdal
ogr2ogr -f GeoJSON Downloads/beijing1.geojson Downloads/DT2geo/BJ_Land_Transactions.shp
```

<script src="https://embed.github.com/view/geojson/casey-huang/casey-huang.github.io/master/beijing1.geojson"></script>

=======================

*Questions the audience may be interested in and how I'd present the graph:*

####1. For the most recent transactions in year 2013, how do land unit prices of different land usage compare to each other?

First, some necessary data munging:

```
# read the shapefile and convert to data frame
library(rgdal)
bj <- readOGR(dsn="Downloads/DT2geo",layer="BJ_Land_Transactions")
bj <- as(bj, "data.frame")

# eliminate irrelevant columns
bj = subset(bj, select = -c(城市,宗地四至,签约时间,合同约定开,合同约定竣,浏览次数,PageUrl,coords.x1,coords.x2))

#change the names of the rest to English
colnames(bj) <- c('obj_id', 'land_id','transferee','location','district','plot_area','archi_area',
                  'use','price','floor_area_ratio','announce_date','y','x')

# remove records where 0 m^2 land was transfered
bj = subset(bj, plot_area!=0)

# adding column "unit price" (RMB/m^2)
bj$unit_price = 100000 * bj$price / bj$plot_area
```

The description in "use" is pretty messy now, I want to classify them into top categories and other.

```
library(sqldf)
industrial=sqldf("SELECT * FROM bj WHERE use LIKE '工业%'")
residential=sqldf("SELECT * FROM bj WHERE use LIKE '住宅%' OR use LIKE '城镇住宅%' OR use LIKE '居住%' OR use LIKE '公寓%'")
commercial=sqldf("SELECT * FROM bj WHERE use LIKE '商业%' OR use LIKE '商务%' OR use LIKE '商服%'")
office=sqldf("SELECT * FROM bj WHERE use LIKE '办公%'")
multiple=sqldf("SELECT * FROM bj WHERE use LIKE '综合%'")

others =sqldf("SELECT * FROM bj R WHERE R.obj_id not in (Select obj_id from 'industrial')")
others =sqldf("SELECT * FROM others R WHERE R.obj_id not in (Select obj_id from 'residential')")
others =sqldf("SELECT * FROM others R WHERE R.obj_id not in (Select obj_id from 'commercial')")
others =sqldf("SELECT * FROM others R WHERE R.obj_id not in (Select obj_id from 'office')")
others =sqldf("SELECT * FROM others R WHERE R.obj_id not in (Select obj_id from 'multiple')")

# add clean tags to each class and re-combine the dataset
land_use=c('residential','industrial','commercial','office','multiple','others')
residential$class=land_use[1]
industrial$class=land_use[2]
commercial$class=land_use[3]
office$class=land_use[4]
multiple$class=land_use[5]
others$class=land_use[6]
# join all the classes to get a new data frame
bj_all = rbind(residential, industrial, commercial, office, multiple, others)
```

Make descending bar chart of median(unit_price) ~ class within 2013:

```
all_2013 = subset(bj_all, as.Date(announce_date)>'2013-01-01')
library(ggplot2)
plotdf=aggregate(all_2013$unit_price, by=list(all_2013$class), FUN='median')
colnames(plotdf)=c('land_use_2013','median_unit_price')
plotdf$land_use <- with(plotdf, reorder(land_use_2013, -median_unit_price))
p1 = ggplot(plotdf, aes(y=median_unit_price, x=land_use_2013)) + geom_bar(stat = "identity")
p1
```

![](http://casey-huang.github.io/assets/article_images/2015-03-23-Beijing-land-transaction/unit_class.png)

Observe that the class "others" has the largest median price in 2013, after checking the original usage and unit prices, I think it is because the variety of function of land here, including airports, hotels&restaurants, public utility, education, tech R&D, etc.

*2. The movement of unit price and #transactions of different lands throughout 10 years?*

For unit price line, I didn't take each data point, because the extremity in price (a lot of 0 cost land and several super high cost) doesn't reflect the general trend, so I take the median of each class each year.

```
# number of land transactions each year by land
library(plyr)
count=count(bj_all, c("year", "class"))
colnames(count)=c('year','class','number_of_transactions')
p2 = ggplot(count, aes(x=year, y=number_of_transactions, group = class, colour = class)) + geom_line()

# unit price of different land each year
library(reshape2)
bj_all$year = format(as.Date(bj_all$announce_date), "%Y")
plotdf2=aggregate(bj_all$unit_price, by=list(bj_all$class, bj_all$year), FUN='median')
colnames(plotdf2)=c('class','year','median_unit_price')
p3 = ggplot(plotdf2, aes(x=year, y=median_unit_price, group = class, colour = class)) + geom_line()

library(gridExtra)
grid.arrange(arrangeGrob(p2, p3))
```

![](http://casey-huang.github.io/assets/article_images/2015-03-23-Beijing-land-transaction/by_year.png)

Observe that there's sharp drop in number of transactions of residential land in between 2004 and 2005. This should be due to the large amendment of land management law that took place in 2004.

####3. Popular location to build a new store/shopping mall?

Plot a static map using ggmap:

```
commercial_2011 = subset(commercial, as.Date(announce_date)>'2011-01-01', select=c('Y','X'))
library(ggmap)
bjbase = qmap(location= "Beijing", zoom = 11, maptype = "road", source = "google")
p4 = bjbase +
  stat_density2d(data = commercial_2011,
                 aes(x = X, y = Y, fill = ..level.., alpha = ..level..),
                 size = 15, bins = 8, geom = "polygon") +
  scale_alpha(range = c(0.3, 0.7), guide = FALSE) +
  scale_fill_gradient(low = "beige", high = "red")
p5 = bjbase +
  stat_bin2d(data = commercial_2011,
             aes(x=X, y=Y, fill=cut(..count.., c(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)), bins = 20),
             alpha = 0.4)+
  scale_fill_hue("count")
p4
p5
```

![](http://casey-huang.github.io/assets/article_images/2015-03-23-Beijing-land-transaction/p4.png)
![](http://casey-huang.github.io/assets/article_images/2015-03-23-Beijing-land-transaction/p5.png)
The two maps tells the same story, and the populat spots for industrial or residential development can be drawn similarly too.

####4 Interactive map?

I intended to make a choropleth that change with time, and struggled using rMaps, since the feature of sliding time bar to view the district choropleth over time is desired. Since there's some issue I can't fix, I use CartoDB as an alternative:

<iframe width='100%' height='520' frameborder='0' src='http://caseyhuang.cartodb.com/viz/35ace442-d234-11e4-a7aa-0e853d047bba/embed_map' allowfullscreen webkitallowfullscreen mozallowfullscreen oallowfullscreen msallowfullscreen></iframe>

=================

###Difficulties and more to-be-worked-on:

> - Missing data and messy preprocessing. Missing data is annoying , but the inconsistent preprocessing make things worse. By looking at the dataset, I would tell that not all missing data are marked as "NA", some of them are recorded as 0 (as in floor_area_ratio, and plot_area etc.).
> - In the bin map, since the "fill" is decided by bins, I didn't find a way to change coloring to the color look more intense for higher count.
> - rMaps or other tools to make choropleth with sliding time.

###Some other thoughts:

Compare to the original graph which is over-simplified, some complicated interactive map seems not necessary to me. For example, the ones in [Gallup Analytics](http://www.gallup.com/poll/125066/State-States.aspx) only have two columns of data.

It also has a major problem of coloring every map in green. For example, in "Feel active and productive", the greener the state is, the well-being is higher. This agrees with my instinct when seeing greener color. But the site also uses more green to show more people are worried about money too, which is somehow misleading.

extension: markdown


text:
# UltraSlackBot

# UltraSlackBot ![Language Badge](https://img.shields.io/badge/Language-Go-blue.svg) ![Go Report](https://goreportcard.com/badge/github.com/DiSiqueira/UltraSlackBot) ![License Badge](https://img.shields.io/badge/License-MIT-blue.svg) ![Status Badge](https://img.shields.io/badge/Status-Development-brightgreen.svg)

The best Slack bot ever.

## Project Status

UltraSlackBot is on development. Pull Requests [are welcome](https://github.com/DiSiqueira/UltraSlackBot#social-coding)

## Features

## Installation

## Usage

### Basic usage

### Show help

## Program Help

## Contributing

### Bug Reports & Feature Requests

Please use the [issue tracker](https://github.com/DiSiqueira/UltraSlackBot/issues) to report any bugs or file feature requests.

### Developing

PRs are welcome. To begin developing, do this:

1. [Fork it] (https://github.com/DiSiqueira/UltraSlackBot/fork)
2. `$ mkdir -p $GOPATH/src/github.com/disiqueira/ultraslackbot`
3. `$ git clone https://github.com/#YOURGITHUBUSERNAMEHERE#/UltraSlackBot.git $GOPATH/src/github.com/disiqueira/ultraslackbot
4. `$ cd $GOPATH/src/github.com/disiqueira/ultraslackbot`
5. `$ cp .env.example .env`
6. Edit your new .env file and put your Slack token on it
7. `$ docker-compose up -d`
8. Profit! :white_check_mark:

Other info:

1. All the plugins should have it's own folder inside the `/plugins` folder
2. All the dependencies should me managed using [Glide](https://github.com/Masterminds/glide). You can use the [cnpj](https://github.com/DiSiqueira/UltraSlackBot/tree/master/plugins/cnpj) plugin as an example
3. You can see all the events that can be handled by plugins [here](https://godoc.org/github.com/nlopes/slack)
4. The [choose](https://github.com/DiSiqueira/UltraSlackBot/tree/master/plugins/choose) plugin is a good example of a complete plugin
5. The [catfact](https://github.com/DiSiqueira/UltraSlackBot/tree/master/plugins/catfact) plugin is a good example of a "Command" plugin

### Debugging

To see the logs you can use:

`docker-compose logs -f ultraslackbot`

## Social Coding

1. Create an issue to discuss about your idea
2. [Fork it] (https://github.com/DiSiqueira/UltraSlackBot/fork)
3. Create your feature branch (`git checkout -b my-new-feature`)
4.  Push to the branch (`git push origin my-new-feature`)
6. Create a new Pull Request
7. Profit! :white_check_mark:

## License

The MIT License (MIT)

Copyright (c) 2013-2018 Diego Siqueira

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


text:
# Twilio SMS Automator Action
### MacOS Automator Action to send SMS text messages through the Twilio API.

This is the first release of the *Twilio SMS Automator Action*. Let me know what you think.

[Download](https://github.com/pereljon/Twilio-SMS-Automator-Action/releases/latest) the latest release.

Cheers,
JP

repository: pereljon/Twilio-SMS-Automator-Action

repo_name: pereljon/Twilio-SMS-Automator-Action


text:
# Fibonacci

Fibonacci sequence: The third number of the sequence and beyond is the sum of the two preceding numbers.

## Challenge

Write a function called Fibonacci which takes in 1 parameters: the nth sequence of the fibonacci sequence. The function returns the value.

## Approach and Efficiency

Time Complexity: O(log n)
> Time complexity is O(n) as the times the loop will run is dependent on the input.

Space Complexity: O(1)
> Space complexity is constant as we're only keeping track of constant (additional) variables : prevNum, currNum, temp, and so on.

## Solution

![fib](../../assets/fib.PNG)

repository: Inkh/Data-Structures-and-Algorithms

gha_language: C#

repo_name: Inkh/Data-Structures-and-Algorithms


text:
---
layout: default
title: Privacy
permalink: /privacy/
---

This web site contains general information about our firm, Developer Club GmbH, for clients, potential clients, and individuals considering possible employment with our Firm. This web site is not intended to be a source of tech consultancy advice. Therefore, you should not consider this information to be an invitation for an consultant-client relationship, should not rely on the information provided herein as consulting advice for any purpose.

Developer Club GmbH does not necessarily endorse, and is not responsible for, any third-party content that may be accessed through or reprinted on this web site. Also, please be aware that we are not and cannot be responsible for the privacy practices of other web sites.

The Firm maintains offices in Copenhagen, Denmark and does not intend or purport to practice in any other jurisdictions.

Developer Club GmbH is a limited liability company registered under the laws of the country of Bermuda.

The reproduction or retransmission of the contents of this web site is prohibited without the prior written consent of Developer Club GmbH. The host server for the web site is located in Frankfurt, Hessen, Germany.

© 2017-2018 Developer Club GmbH. All rights reserved. Last revised February 2018.


text:
---
layout: post
title: "West LA DevOps #4: Industry Updates & HashiCorp Talks About Service Meshes"
---

Last night [Brian Tai](https://github.com/btai24){:target="_blank"} and I had the pleasure of hosting HashiCorp at our 4th installment of West LA DevOps hosted at GumGum HQ. We had two speakers from HashiCorp, including our first female speaker [Jasmine Dahilig](https://github.com/jazzyfresh){:target="_blank"} (Software Engineer)! Jasmine's talk was entitled "Dancing with Data and Distributed Systems" and [Jake Lundberg](https://github.com/grove-mountain){:target="_blank"} (Staff Solutions Engineer) also spoke about creating service meshes using Consul.

We also had our 3rd "Industry Updates" segment where we provided a concise overview of some of the major happenings in DevOps and software in general. Check out our slides below for our most recent updates!

### Industry Updates slides

You can view our WLAD #4 Meetup slides, starting with our Industry Updates segment [here](https://slides.com/coreygale/wlad-2-5#/4){:target="_blank"}.

### Meetup recording

To watch our Industry Updates, as well as Jasmine & Jakes' talks, checkout [our recording on Vimeo](https://vimeo.com/369159877){:target="_blank"}.

### Meetup event link

Check out the official event on [Meetup](https://www.meetup.com/West-LA-DevOps/events/265221538/){:target="_blank"}


text:
# Neural Network from scratch

[![Packagist](https://img.shields.io/packagist/l/doctrine/orm.svg)](LICENSE.md)
---

### Author
Arpit Aggarwal

### Introduction to the Project
A two-layer neural network for classifying cats vs non-cats. The neural network is built using numpy library where the image is flattened to form the input layer and the hidden layer has 14 neurons and the output layer has one neuron predicting whether the input image is a cat or a non-cat.

### Results
The optimal performance was acheived using hyperparameters, learning rate = 0.003 and epochs = 7000. The results obtained using these hyperparameters is given below:

Training Loss = 0.047

Testing Accuracy = 76%

### Software Required
To run the .py files, use Python 3. Standard Python 3 libraries like numpy is used.

### Credits
The following links were helpful for this project:
1. https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6
2. https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/


text:
# o-audio

The FT audio player.

- [Usage](#usage)
	- [Markup](#markup)
	- [JavaScript](#javascript)
	- [Sass](#sass)
- [Migration](#migration)
- [Contact](#contact)
- [Licence](#licence)

## Usage

This is a beta version of oAudio. It will accept an `<audio>` element + attach o-tracking events to it (for occurences of `play`, `pause` etc). For more details on these events please see [Analytics](analytics.md).

The expectation is that oAudio will eventually render it's own player interface, potentially with different skins, to be used on ft.com and the app.

Check out [how to include Origami components in your project](https://origami.ft.com/documentation/components/#including-origami-components-in-your-project) to get started with `o-audio`.

### Markup

```html
<audio controls data-o-component="o-audio">
	<source
		src="https://media.acast.com/ftnewsbriefing/wednesday-november14/media.mp3"
		type="audio/mpeg"
	/>
</audio>
```

### JavaScript

No code will run automatically unless you are using the Build Service.
You must either construct an `o-audio` object or fire the `o.DOMContentLoaded` event, which oAudio listens for.

#### Constructing an o-audio

```js
import oAudio from "@financial-times/o-audio"
oAudio.init()
```

#### Firing an oDomContentLoaded event

```js
import "@financial-times/o-audio"

document.addEventListener("DOMContentLoaded", function () {
	document.dispatchEvent(new CustomEvent("o.DOMContentLoaded"))
})
```

### Sass

Since oAudio renders as a native HTML `<audio>` element only, there are no styles _as yet_.

## Migration

State | Major Version | Last Minor Release | Migration guide |
:---: | :---: | :---: | :---:
✨ active | 2 | N/A | [migrate to v2](MIGRATION.md#migrating-from-v1-to-v2) |
⚠ maintained | 1 | 1.4 | N/A |

## Contact

If you have any questions or comments about this component, or need help using it, please either [raise an issue](https://github.com/Financial-Times/o-audio/issues) or visit [#ft-next-support](https://financialtimes.slack.com/messages/ft-next-support).

***

## Licence

This software is published by the Financial Times under the [MIT licence](http://opensource.org/licenses/MIT).

repository: Financial-Times/origami

branch_name: refs/heads/main

gha_language: JavaScript

repo_name: Financial-Times/origami


text:
# Space Shooter: The Secret Lift of Sprites

This game was made as part of the TAG Lesson (Session 4 for Javascript After School Lesson and Best Kids After School Program).

Learning Objectives:

-  Students will be able to identify what a sprite is

-  Students will learn how a Sprite is added to a game

This game also contributes to the goal of building a collection of working demo games that will encourage students to start hacking/breaking/ and making these games their own.

### Lesson Objectives
The game starts with one bad guy, 'Bison'.

Instructors will work with students to complete the steps of adding new baddies and thus up the difficulty of the game.

Tasks:

1.  Add an image in Preload (Just an Uncomment, //--- Preload Task--//)

2.  Add Rules Code (Just an Uncomment, //--- Create Task--//)

3.  Add Collision detection Task Code in Update (Just an Uncomment, //--- Update Task--//)

### Controls
When the instructor first boots the game the controls are:

Bot 1 controls are
Space = Shoot
Up Arrow Key = Up
Down Arrow Key = Down

## Playing / Modifying the Game
The code is designed to be manipulated in the Fiddler.js editor.  This web based editor is available on DCPS computers.

If not in the classroom or on DCPS computers, It is suggested to Brackets IDE for development or any local server for testing and development.

To play the base game, you can find it here: https://rectheworld.github.io/botsvsaliens/

## Built With
* [Phaser.js 2.6.2 ]([https://phaser.io/](https://phaser.io/))

## Contributing
@notserk
@colwus
@Christina

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details


text:
## Field Conditions

**Click #2 (field condition button) or #3 to open the field condition dialog.**

![](Field%20Conditions_fieldcondition5copy.png)

**Field condition dialog.**

# Displays the field label
# A drop down similar to Advanced find allowing you to set various conditions
# Text box to enter a value for the condition
# Sets/updates the condition
# Cancels the field condition dialog
![](Field%20Conditions_fieldcondition4copy.png)

**Remove a condition.**

![](Field%20Conditions_fieldcondition2copy.png)

**Remove all conditions.**

![](Field%20Conditions_fieldcondition1copy.png)

repository: phoenixinobi/dcrmeg

path: /docs/Field Conditions.md

filename: Field Conditions.md

repo_name: phoenixinobi/dcrmeg


text:
---
layout: post
title:  "changed status of \"Timing Control for Script-Based Animations\" from CR to NOTE; updated \"Frame Timing\" abstract"
date:   2015-09-23
tags:   [ animation-timing , frame-timing ]
---

changed status of "[Timing Control for Script-Based Animations](/spec/animation-timing)" from CR to NOTE; updated "[Frame Timing](/spec/frame-timing)" abstract

license: Unlicense

branch_name: refs/heads/gh-pages

gha_license_id: Unlicense


text:
## Groovy Style guide
Apache Groovy is a dynamic language built on the JVM platform. Groovy is an optionally typed language that can compile
native Java, Groovy syntax, or a mix of both. Groovy was created as a way to quickly write Java code without much of the
excessive verbosity of Java, and to layer on dynamic language features on top of the statically types Java language.

Read more about Groovy at [http://groovy-lang.org](http://groovy-lang.org)

This Groovy style guide is a modification of the [default Groovy style guide](http://groovy-lang.org/style-guide.html). Unlike the default style guide, we require strict typing for more self-documenting code and better IDE auto-complete. We've also stripped out basic language information, as that is not related to code style, but just features of the language.

## Table of Contents
* [Semicolons](#semicolons)
* [Function, Class, and Variable declarations](#function-class-and-variable-declarations)
  * [If your function returns a value, use the return keyword](#if-your-function-returns-a-value-use-the-return-keyword)
  * [Explicitly type function parameters and return value](#explicitly-type-function-parameters-and-return-value)
  * [Explicitly type variables](#explicitly-type-variables)
  * [Avoid redundant `def` when declaring variables](#avoid-redundant-def-when-declaring-variables)
  * [Only use visibility modifiers for non-public methods](#only-use-visibility-modifiers-for-non-public-methods)
  * [Use parentheses when calling functions](#use-parentheses-when-calling-functions)
  * [Don't use `.class`](#dont-use--class)
  * [No visibility modifiers for class properties](#no-visibility-modifiers-for-class-properties)
  * [Initializing beans with named parameters using the default constructor](#initializing-beans-with-named-parameters-using-the-default-constructor)
  * [Use `with()` when setting multiple properties on the same bean](#use-with-when-setting-multiple-properties-on-the-same-bean)
* [Strings](#strings)
  * [Use GString for string interpolation](#use-gstring-for-string-interpolation)
  * [Omit curly braces if you're just accessing a variable](#omit-curly-braces-if-youre-just-accessing-a-variable)
  * [Use triple quotes for multi-line strings](#use-triple-quotes-for-multi-line-strings)
  * [Use "slashy" string notation for regular expressions](#use-slashy-string-notation-for-regular-expressions)
  * [Use single quotes for string constants](#use-single-quotes-for-string-constants)
* [Data structures](#data-structures)
  * [Use Groovy's native syntax over Java's](#use-groovys-native-syntax-over-Java's)
* [Truthiness and null checking](#truthiness-and-null-checking)
  * [Use Groovy truthiness to evaluate if something is null, zero, or empty](#use-groovy-truthiness-to-evaluate-if-something-is-null-zero-or-empty)
  * [Use `==` not `.equals()`](#use-not-equals)
  * [Use safe graph navigation](#use-safe-graph-navigation)
  * [Use the Elvis operator for default values](#use-the-elvis-operator-for-default-values)

## Semicolons

Omit semicolons in Groovy
> Why? Semicolons are not required in Groovy, and it's more idiomatic to omit them.

## Function, Class, and Variable declarations

#### If your function returns a value, use the return keyword

> Why? While Groovy does not require the use of the `return`, instead implicitly returning the final line in a function, it can easily lead to confusion as to whether a function modifies an object or returns a new value.
>
> Requiring it makes it easy to see what the function returns, if anything.

```
// Avoid
String toString() {
  "a server"
}

// Prefer
String toString() {
  return "a server"
}

```

```
// Avoid
def props(m1) {
    m2 = m1.findAll { k, v -> v % 2 == 0 }
    m2.c = 3
    m2
}

// Prefer
def props(m1) {
    m2 = m1.findAll { k, v -> v % 2 == 0 }
    m2.c = 3
    return m2
}
```

```
// Avoid
def foo(n) {
    if(n == 1) {
        "Roshan"
    } else {
        "Dawrani"
    }
}

// Prefer
String foo(int n) {
    if(n == 1) {
        return "Roshan"
    } else {
        return "Dawrani"
    }
}
```

#### Explicitly type function parameters and return value

> Why? Omitting types can lead to confusing bugs, misuse of functions, and prevents your IDE from being able to use auto-complete.

```
// Avoid
def foo(n) {
    if(n == 1) {
        "Roshan"
    } else {
        "Dawrani"
    }
}

// Prefer
String foo(int n) {
    if(n == 1) {
        return "Roshan"
    } else {
        return "Dawrani"
    }
}
```

#### Explicitly type variables

> Why? Using `def` everywhere is not self documenting, it leads to confusion about what the variable is, and breaks IDE auto-completion.

```
// Avoid
def name = "Guillaume"

// Prefer
String name = "Guillaume"
```

#### Avoid redundant `def` when declaring variables

> Why? The def keyword is for generic typing, nothing else. It is not used to declare a variable of a known type

```
// Avoid
def String name = "Guillaume"

// Prefer
String name = "Guillaume"
```

```
// Avoid
class MyClass {
  def MyClass() {}
}

// Prefer
class MyClass {
  MyClass() {}
}
```

#### Only use visibility modifiers for non-public methods

> Why? Groovy considers classes and methods `public` by default.

```
// Avoid
public class Server {
    public String toString() { return "a server" }
}

// Prefer
class Server {
    String toString() { "a server" }
}
```

Note: For package scoped visibility use the `@PackageScope` annotation
```
class Server {
    @PackageScope Cluster cluster
}
```

#### Use parentheses when calling functions

> Why? Using parentheses makes it obvious you are calling a function and assists the IDE in showing you the parameters you can pass. It also keeps code consistent, as even in Groovy there are times when parentheses are required.

```
// Avoid
println "Hello"
method a, b

// Prefer
println("Hello")
method(a, b)
```

```
def foo(n) { return n }
int bar() { return 1 }

// These won't work
println foo 1
def m = bar

// These will
println(foo(1))
int m = bar()
```

#### Don't use `.class`

> Why? The `.class` suffix is not needed in Groovy.

```
// Avoid
connection.doPost(BASE_URI + "/modify.hqu", params, ResourcesResponse.class)

// Prefer
connection.doPost("${BASE_URI}/modify.hqu", params, ResourcesResponse)
```

#### No visibility modifiers for class properties

> Why? Unlike methods which are public by default, properties are private by default, but getters and setters are automatically generated by Groovy.

Note: Only declare getters and setters if you wish to perform additional logic when getting or setting.

```
// Avoid
class Person {
    private String name
    String getName() { return name }
    void setName(String name) { this.name = name }
}

// Prefer
class Person {
    String name
}
```

#### Initializing beans with named parameters using the default constructor

Note: If you pass more than 2 or 3 parameters, put each parameter on a new line

```
// Avoid
Server server = new Server(name: "Obelix", cluster: aCluster, size: "Large", host: "Amazon", ipAddress: "<IP_ADDRESS>", isEnabled: true)

// Better, but more code and still hard to read
Server server = new Server()
server.name = "Obelix"
server.cluster = aCluster
server.size = "Large"
server.host = "Amazon"
server.ipAddress = "<IP_ADDRESS>"
server.isEnabled = true

// Best, less code, easy to read
Server server = new Server(
    name: "Obelix",
    cluster: aCluster,
    size: "Large",
    host: "Amazon",
    ipAddress: "<IP_ADDRESS>",
    isEnabled: true
)
```

#### Use with() when setting multiple properties on the same bean

> Why? Using `with` provides the same easy to read syntax as using the default constructor, but can be used to set properties on an existing bean.

```
// Avoid
server.name = application.name
server.status = status
server.sessionCount = 3

// Prefer
server.with {
    name = application.name
    status = status
    sessionCount = 3
}
```

## Strings

#### Use GString for string interpolation

> Why? String interpolation is easier to read, especially when incorporating multiple variables into a string

```
// Avoid
throw new Exception("Unable to convert resource: " + resource)

// Prefer
throw new Exception("Unable to convert resource: $resource")
```

#### Omit curly braces if you're just accessing a variable

> Why? They aren't needed for simple variable accessing, and not having them is easier to read

```
// Avoid
throw new Exception("Unable to convert resource: ${resource}")

// Prefer
throw new Exception("Unable to convert resource: $resource")

int i = 3

// Curly braces required because we're performing an operation
def s1 = "i's value is: ${i + 10}"
```

#### Use triple quotes for multi-line strings

> Why? Using triple strings and placing actual new lines is easier to ready than \n

```
throw new PluginException("""Failed to execute command list-applications:
    The group with name ${parameterMap.groupname[0]}
    is not compatible group of type ${SERVER_TYPE_NAME)}""")
```

Note: You can also strip the indentation appearing on the left side of the multiline strings by calling `.stripIndent()`on that string.

#### Use "slashy" string notation for regular expressions

```
assert "foooo/baaaaar" ==~ /fo+\/ba+r/
```

Note: The advantage of the "slashy" notation is that you don't need to double escape backslashes, making working with regex a bit simpler.

#### Use single quotes for string constants

> Why? Simply a styling preference, keeps code consistent.

## Data structures

#### Use Groovy's native syntax over Java's

> Why? Groovy provides native syntax constructs for data structures like lists, maps, regex, or ranges of values that are simpler, more concise, and easier to read than Java's

```
// Examples below:

def list = [1, 4, 6, 9]

// by default, keys are Strings, no need to quote them
// you can wrap keys with () like [(variableStateAcronym): stateName] to insert a variable or object as a key.
def map = [CA: 'California', MI: 'Michigan']

def range = 10..20
def pattern = ~/fo*/

// equivalent to add()
list << 5

// call contains()
assert 4 in list
assert 5 in list
assert 15 in range

// subscript notation
assert list[1] == 4

// add a new key value pair
map << [WA: 'Washington']
// subscript notation
assert map['CA'] == 'California'
// property notation
assert map.WA == 'Washington'

// matches() strings against patterns
assert 'foo' ==~ pattern
```

## Truthiness and null checking

#### Use Groovy truthiness to evaluate if something is null, zero, or empty

> Why? All objects can be 'coerced' to a boolean value: everything that's `null`, `void`, equal to zero, or empty evaluates to `false`, and if not, evaluates to `true`.

```
// Avoid
if (name != null && name.length > 0) {}

// Prefer
if (name) {}
```

#### Use `==` not `.equals()`

> Why? Groovy's `==` also takes care of avoiding `NullPointerException`, independently of whether the left or right is `null` or not.

```
// Avoid
status != null && status.equals(ControlConstants.STATUS_COMPLETED)

// Prefer
status == ControlConstants.STATUS_COMPLETED
```

#### Use safe graph navigation

> Why? Writing individual null check is harder to read and takes more effort to write

```
// Avoid
if (order != null) {
    if (order.getCustomer() != null) {
        if (order.getCustomer().getAddress() != null) {
            println(order.getCustomer().getAddress());
        }
    }
}

// Prefer
println order?.customer?.address
```

#### Use the Elvis operator for default values

> Why? The Elvis operator is a special ternary operator shortcut, equivalent to overloading JavaScript's `||`, that lets you apply a default value with less code.

```
// Avoid
String result = name != null ? name : 'Unknown'

// Prefer
String result = name ?: 'Unknown'
```

repository: christianlawler/voyage-api-java

repo_name: christianlawler/voyage-api-java


text:
# Slides_to_video

This script transforms the slides of a [Mediasite](http://www.sonicfoundry.com/mediasite/) export to a continuous mp4 of the same length as the original lecture recording.
The resulting video is easier to use for later processing in editing software.

## Getting started

To run this script, you will need to install ffmpeg on the local machine.
A good how to for different operating systems can be found here: https://github.com/adaptlearning/adapt_authoring/wiki/Installing-FFmpeg

## Author/s

The initial work on this project has been done by Adam Sasiadek<EMAIL_ADDRESS>
## License

The project is licensed under the MIT License


text:
---
layout: post
date: 'Wed 2016-03-03 23:54:06 +0800'
slug: "Unicode-in-RTF"
title: Unicode in RTF documents
description: How to encode unicode codepoints in RTF documents using PyRTF.
category:
tags: ZT, Python, RTF, Unicode, encoding
---

额，为了找把`cfv`输出的形如`\uhhhh`的编码还原为utf8串，搜到[这个](http://www.zopatista.com/python/2012/06/06/rtf-and-unicode/)。
特转载。
途中发现是*Jekyll*风，就搜了下源码，遂[得](https://github.com/mjpieters/mjpieters.github.com)[之](https://github.com/mjpieters/mjpieters.github.com/blob/master/_posts/2012-06-06-rtf-and-unicode.md)。

话说，他那个`{ {page.description} }`用得挺别致的。

咱自己当然还是关键词get后用*Perl*解决的。

````Perl
#!/usr/bin/env perl
use strict;
use warnings;
use File::Basename;
use File::Path 2.08;
use Unicode::Escape;

#my $Prefix='t/';

while (<>) {
	next unless s/ : file size does not match \((\d+)\!=(\d+)\)$//;
	chomp;
	my ($torrSize,$fileSize) = ($1,$2);
	my $tName = Unicode::Escape::unescape($_);
	my $dirname  = dirname($tName);
	#File::Path::make_path("$dirname",{verbose => 0,mode => 0755});
	#system('mv',$Prefix.$tName,$tName)
	my $diskSize = -s $tName;
	print STDERR "$tName\n\t$torrSize <= $fileSize,$diskSize\t";
	if ($diskSize != $fileSize) {
		warn "Skip for Reported != Real\n";
	} elsif ($torrSize == $fileSize) {
		warn "Skip for Already done.\n";
	} else {
		warn "truncate -s $torrSize\n";
		system('truncate','-s',$torrSize,$tName);
	}
}

__END__
[philosophy-raws][Sword Art Online]/CDs/\u6625\u5948\u308b\u306a OVERSKY BD\u4ed8\u521d\u56de\u751f\u7522\u9650\u5b9a\u76e4/BK/01.png : file size does not match (19327223!=19337438)

./torrtruncate.pl SwordArtOnLineA.err
````

---

*{{ page.description }}*

Some time ago I had to output some nicely formatted reports from a web application, to be usable offline by Windows users. Naturally, I used the aging but still reliable [`PyRTF` module](https://pypi.python.org/pypi/PyRTF) to generate RTF documents with headers, tables, and a consistent style.

As my application users are mostly Norwegians, however, I quickly discovered that the `PyRTF` module does not handle international characters (i.e. anything outside the ASCII codepoints), at all. There is no unicode support at all (it has been on the TODO list since forever), let alone converting unicode codepoints to whatever RTF uses to represent international characters.

Recently, a [Stack Overflow question](http://stackoverflow.com/q/10852810/100297) reminded me of how I solved this problem at the time, and clearly this question has [come](http://stackoverflow.com/q/9908647/100297) [up](https://groups.google.com/forum/?fromgroups#!topic/django-users/gZH1mnBfgoI) [before](http://osdir.com/ml/web2py/2010-03/msg01045.html). Because some approaches I've seen can actually produce incorrect or overly verbose output (including [pyrtf-ng](https://code.google.com/p/pyrtf-ng/)), I wanted to explain and expand on my solution to provide a definitive answer to the problem, and also see how my original method faired in terms of speed.

## So *does* RTF handle unicode?

Since PyRTF doesn't filter the text you add to a document at all we can just encode unicode strings ourselves. Lucky for me, the [Wikipidia entry on RTF](https://en.wikipedia.org/wiki/Rich_Text_Format) has a fairly detailed section on how [RTF handles characters outside the ASCII range](https://en.wikipedia.org/wiki/Rich_Text_Format#Character_encoding). Together with the [published RTF 1.9.1 specification](http://www.boumphrey.com/rtf/rtfspec.pdf) (PDF) there is plenty of information on how to encode unicode codepoints to RTF control sequences.

There basically are two choices:

* The `\'hh` control sequence; a backslash and single quote, followed by an 8-bit hexadecimal value. The value is interpreted as a code-point in a Windows codepage, limiting it's use. You *can* assign different codepages to different fonts, but you still cannot use the full range of unicode in a paragraph.

* The `\uN?` control sequence; backslash 'u' followed by a signed 16-bit integer value in decimal and a placeholder character (represented here by a question mark). The signed 16-bit integer number here is consistent with the RTF standard for control characters, a value between -32768 and 32767.

    This control sequence *can* properly represent unicode, at least for the U+0000 through to U+FFFF codepoints. This sequence was introduced in the 1.5 revision of the RTF spec, in 1997, so it should be widely supported. The placeholder character is meant to be used by readers that do not yet support this escape sequence and should be an ASCII character closest to the unicode codepoint.

The `\uN?` format is the easiest to produce, especially if you ignore the replacement character (just set it to '?' at all times, surely most RTF readers support the 1.5 RTF standard by now, it's been out there for 15 years).

## Encoding the slow (and incorrect) way

A quick search with Google showed me how [pyrtf-ng encodes unicode points](https://code.google.com/p/pyrtf-ng/source/browse/trunk/rtfng/Renderer.py?r=81#506):

{% highlight python %}
    def writeUnicodeElement(self, element):
        text = ''.join(['\u%s?' % str(ord(e)) for e in element])
        self._write(text or '')
{% endhighlight %}

Unfortunately, the above snippet does a few things wrong: it uses a control code for *every* character in the unicode string, producing output that is at least 5 times as long as the input, and it doesn't produce negative numbers for codepoints over `\u7fff`:

{% highlight pycon %}
>>> example = u'CJK Ideograph: \u8123'
>>> ''.join(['\u%s?' % str(ord(e)) for e in example])
u'\\u67?\\u74?\\u75?\\u32?\\u73?\\u100?\\u101?\\u111?\\u103?\\u114?\\u97?\\u112?\\u104?\\u58?\\u32?\\u33059?'
{% endhighlight %}

A recent [Stack Overflow answer](http://stackoverflow.com/a/9912561/100297) improved on this by only encoding characters over `\u007f` (decimal 127) but it still iterates over every character in the string to do so:

{% highlight pycon %}
>>> ''.join(['\u%s?' % str(ord(e)) if ord(e) > 127 else e for e in example])
u'CJK Ideograph: \\u33059?'
{% endhighlight %}

This outputs unicode because codepoints < 128 are left untouched; numbers are not properly converted to signed shorts either. Here is my variation that remedies these things, and dispenses with the `str()` call:

{% highlight pycon %}
>>> ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in example])
'CJK Ideograph: \\u-32477?'
{% endhighlight %}

This feels like rather a waste to me, and must be slow as well. I wanted to see how my own solution stacks up against the character-by-character, naive implementation.

## Encoding the lazy way

While casting around for my own solution, I also looked into the Python [`codecs` module](http://docs.python.org/library/codecs.html) to come up with ideas on how to do this more efficiently. Of course, the codecs provided by that module are all implemented in C, but the `unicode_escape` codec did produce output quite close to what I needed for RTF; codepoints between `\u0020` and `\u007f` are left alone, the rest are encoded to one of the `\xhh`, `\uhhhh` or `\Uhhhhhhhh` 8, 16 or 32-bit escapes (with the exception of `\t`, `\n` and `\r`). Would there be any way to reuse this output?

Well, if you combine this with a bit of [`re.sub`](http://docs.python.org/library/re.html#re.sub) magic, you can in fact produce convincing RTF command sequences:

{% highlight pycon %}
>>> import re
>>> import struct
>>> _charescape = re.compile(r'(?<!\\)\\(?:x([0-9a-fA-F]{2})|u([0-9a-fA-F]{4}))')
>>> def _replace_struct(match):
...     match = match.groups()
...     # Convert XX or XXXX hex string into 2 bytes
...     codepoint = (match[0] and '00' + match[0] or match[1]).decode('hex')
...     # Convert 2 bytes into a signed integer, insert into escape sequence
...     return '\\u%i?' % struct.unpack('!h', codepoint)
...
>>> escaped = example.encode('unicode_escape')
>>> escaped
'CJK Ideograph: \\u8123'
>>> _charescape.sub(_replace_struct, escaped)
'CJK Ideograph: \\u-32477?'
{% endhighlight %}

Using the [`struct` module](http://docs.python.org/library/struct.html) gave me a quick means to re-interpret the hexadecimal notation as produced by the `unicode_escape` format as a signed short, but I did have to make sure there were 2 bytes at all times.

Of course, the above trick does not handle newlines, returns or tabs (`\n`, `\r` and `\t` respectively) correctly, nor does it escape existing backslashes yet, but I hoped back when that this proof of concept should operate several orders of a magnitude faster than the naive character-by-character method when dealing with mostly-ASCII input; most of the work is done in C by the `codecs` and `re` modules, after all.

So this time around I decided to time these:

{% highlight pycon %}
>>> import timeit
>>> def test1(): ''.join(['\u%i?' % (ord(e) if e < u'\u8000' else ord(e) - 65536) if (e > '\x7f' or e < '\x20' or e in u'\\{}') else str(e) for e in testdocument])
...
>>> def test2(): _charescape.sub(_replace_struct, testdocument.encode('unicode_escape'))
...
>>> declaration = u'\
... Alle mennesker er f\xf8dt frie og med samme menneskeverd og menneskerettigheter. \
... De er utstyrt med fornuft og samvittighet og b\xf8r handle mot hverandre i brorskapets \xe5nd.'
>>> testdocument = declaration * 100
>>> timeit.timeit('test1()', 'from __main__ import test1', number=500)
5.982733964920044
>>> timeit.timeit('test2()', 'from __main__ import test2', number=500)
1.4459600448608398
{% endhighlight %}

Cool, so my hybrid encode plus regular-expression based solution looks to be around 4 times as fast, at least when it comes to simple Norwegian text with a handful of latin-1 characters, my most common case. Note however that I am not handling the RTF escape characters properly, nor are the `\n`, `\r` and `\t` characters handled correctly.

## Can I do better?

But I am actually being too clever by half (read: pretty dumb really); why did I encode to `unicode_escape` in the first place? I was still in the process of fully understanding the issues and saw a shortcut. My regular expression isn't particularly clever, I dabbled with the struct module to get my signed short values, and with all this hocus-pocus I lost sight of the goal: to escape certain classes of characters to RTF command codes.

But aren't regular expressions quite good at finding those classes all by themselves? I may as well use a decent expression that selects what needs to be encoded directly:

{% highlight pycon %}
>>> _charescape_direct = re.compile(u'([\x00-\x1f\\\\{}\x80-\uffff])')
>>> def _replace_direct(match):
...     codepoint = ord(match.group(1))
...     return '\\u%s?' % (codepoint if codepoint < 32768 else codepoint - 65536)
>>> _charescape_direct.sub(_replace_direct, example).encode('ascii')
'CJK Ideograph: \\u-32477?'
>>> def test3(): _charescape_direct.sub(_replace_direct, testdocument).encode('ascii')
...
>>> timeit.timeit('test3()', 'from __main__ import test3', number=500)
0.5356400012969971
{% endhighlight %}

Suddenly we have an 10 times speed increase! Not only that, I am now also properly escaping the three whitespace characers `\n`, `\r` and `\t`, and as an added bonus, the RTF special characters `\`, `{` and `}` are now also being escaped!

## Perhaps we can translate instead

We could also use a [translation table](http://docs.python.org/library/stdtypes.html#str.translate) to do my escaping for me. This is simply a dict that maps unicode codepoints to a replacement value. To create a static dict for all unicode values could be somewhat tricky, requiring either a custom `__missing__` method or loading a generated structure on import.

Before digging into clever solutions to that, I should perhaps first test the speed of a simple translation table, one that only covers codepoints up to '\u00ff', or latin-1:

{% highlight pycon %}
>>> _table = {i: u"\\'{0:02x}".format(i) for i in xrange(0, 32)}
>>> _table.update({ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\\{}'})
>>> _table.update({i: u"\\u{0}".format(i) for i in xrange(128, 256)})
>>> len(_table)
163
>>> timeit.timeit('testdocument.translate(_table).encode("ascii")', 'from __main__ import testdocument, _table', number=500)
2.66812801361084
{% endhighlight %}

Unfortunately, using `.translate` turns out to be slowing us down considerably. Reducing the table to just a few codepoints doesn't help either:

{% highlight pycon %}
>>> _basictable = {ord(c): u"\\'{0:02x}".format(ord(c)) for c in '\n\r\t\\{}'}
>>> len(_basictable)
6
>>> timeit.timeit('testdocument.translate(_basictable)', 'from __main__ import testdocument, _basictable', number=500)
2.0113179683685303
{% endhighlight %}

So it looks like I might want to avoid using `.translate` if at all possible.

## Worst-case scenario

So far, I've compared methods by testing them against some Norwegian text, typical of many European languages with a generous helping of ASCII characters.

To get a more complete picture, I need to test these methods against a worst-case scenario, a UTF-8 encoded test set from a great set of [UTF-8 test documents](https://github.com/bits/UTF-8-Unicode-Test-Documents):

{% highlight pycon %}
>>> import urllib
>>> utf8_sequence = urllib.urlopen('https://raw.github.com/bits/UTF-8-Unicode-Test-Documents/master/UTF-8_sequence_unseparated/utf8_sequence_0-0xffff_assigned_printable_unseparated.txt').read().decode('utf-8')
>>> len(utf8_sequence)
58081
>>> testdocument = utf8_sequence
>>> timeit.timeit('test1()', 'from __main__ import test1', number=10)
0.7785000801086426
>>> timeit.timeit('test3()', 'from __main__ import test3', number=10)
0.8913929462432861
{% endhighlight %}

Interesting! So in the worse-case scenario, where the vast majority (99.8%) of the text requires encoding, the character-by-character method is actually a little faster again! But this also means that for most cases, where you insert shorter text snippets into an RTF document, and where a far larger percentage of characters do not need escaping, the regular expression method will beat the character-by-character method hands down.

## So what about non-BMP Unicode?

So far I've focused only on characters within the [BMP](https://en.wikipedia.org/wiki/Unicode_plane#Basic_Multilingual_Plane). You can apparently use a [UTF-16 surrogate pair](https://en.wikipedia.org/wiki/UTF-16#Code_points_U.2B10000_to_U.2B10FFFF), at least according to Wikipedia for codepoints byond the BMP. However, the RTF specification itself is silent on this, and no endian-nes is documented anywhere that I can find. The Microsoft platform uses UTF-16-LE throughout, so perhaps RTF readers support little-endian surrogate pairs too.

However, I cannot at this time be bothered to extend my encoder to support such codepoints. On a UCS-2-compiled python there is a happy coincidence that codepoints beyond the BMP are treated mostly like UTF-16 surrogate pairs anyway, so they are sort-of supported by this method:

{% highlight pycon %}
>>> beyond = u'\U00010196'
>>> _charescape_direct.sub(_replace_direct, beyond).encode('ascii')
'\\u-10240?\\u-8810?'
{% endhighlight %}

Note, however, that the first byte is -10240, or `0xd800` in unsigned hexadecimal, making this a big-endian encoded surrogate pair. Presumably on Windows that'll encode the other way around.

On a UCS-4 platform the codepoint will be ignored by the regular expression and the `.encode('ascii')` call will raise a UnicodeEncodeError instead.

I am calling this 'unsupported' and a day. Suggestions for implementing this in a neat and performant way are welcome!

So I packaged up my regular expression method as a handy [module on PyPI](https://pypi.python.org/pypi/rtfunicode), complete with Python 2 and 3 support and a miniscule test suite; the [source code is available on GitHub](https://github.com/mjpieters/rtfunicode).

The module in fact registers a new codec, called `rtfunicode`, so after you import the package all you need do is use the new codec in the `.encode()` method:

{% highlight pycon %}
>>> import rtfunicode
>>> declaration.encode('rtfunicode')
'Alle mennesker er f\\u248?dt frie og med samme menneskeverd og menneskerettigheter. De er utstyrt med fornuft og samvittighet og b\\u248?r handle mot hverandre i brorskapets \\u229?nd.'
{% endhighlight %}

Hopefully it comes in handy for others. Feedback is most welcome, as are patches!

*[RTF]: Rich Text Format
*[ASCII]: American Standard Code for Information Interchange
*[BMP]: Basic Multilingual Plane


text:
---
layout: default
---

# 1996-01-27 - Feds on Internet Banking

## Header Data

From: "E. ALLEN SMITH"<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Message Hash: efaa7e295f0c29e95296a1e5863d4ac110ea29bddc3278366b4e5f38567d70d1

Message ID<EMAIL_ADDRESS>Reply To: _N/A_

UTC Datetime: 1996-01-27 21:07:34 UTC

Raw Date: Sun, 28 Jan 1996 05:07:34 +0800

## Raw message

```
{% raw  %}From: "E. ALLEN SMITH"<EMAIL_ADDRESS>Date: Sun, 28 Jan 1996 05:07:34 +0800
To<EMAIL_ADDRESS>Subject: Feds on Internet Banking
Message-ID<EMAIL_ADDRESS>MIME-Version: 1.0
Content-Type: text/plain

	Evidence that the Fed has noticed Internet banking. It looks like
they're more concentrating on electronic checks than on digital cash, though;
I may be mistaken in that, however. Any of the lawyers on here know much about
banking law?
	-Allen

   Reuters New Media

   _ Friday January 26 2:01 PM EST _

Boston Fed Minehan Warns Of "Virtual" Bank Danger

   NEW YORK - Federal Reserve Bank of Boston President Cathy Minehan is
   warning against potential risks of "virtual" banking -- or
   electronic account systems handling electronic money.

   Minehan stressed that the regulatory community has just begun to
   consider these issues and that much work needs to be done before it
   even knows all the questions to ask, let alone what answers to give.

   "Legislation and regulation of new payments system alternatives
   could be unwise right now, but that does not mean that participants in
   such systems should not oversee them or that central banks should not
   be concerned" Minehan said in an address to the Goldman, Sachs
   Conference on Risk Reduction in Payments, Clearance and Settlement
   Systems.

   "'Smart' card technology is now being used to store electronic
   'notes' authorized by the card-holder's bank that can be used to
   transfer value between banks, consumers and merchants," she said.

   "These electronic 'notes' flow over the Internet and give
   authorization for funds to be withdrawn from a bank account and paid
   to another party electronically," Minehan said.

   "This can come close to being a new form of currency in that the
   potential exists for the value on the card to remain in circulation,
   transferring from card to card, and one endpoint to another, without
   necessarily being converted to a more traditional form of money,"
   she said.

[...]

   "'In this regard, it is especially intriguing to consider how the
   'virtual' bank might be regulated," said Minehan. "Some
   maintain that virtual banking is just a new form of bank which all the
   usual rules and regulations apply," she said.

   Copyright, Reuters Ltd. All rights reserved

{% endraw %}
```

## Thread

+ Return to [January 1996](/archive/1996/01)

+ Return to "["E. ALLEN SMITH" <EALLENSMITH@ocelot.Rutgers.EDU>](/authors/e_allen_smith_eallensmith_at_ocelot_rutgers_edu_)"

+ 1996-01-27 (Sun, 28 Jan 1996 05:07:34 +0800) - Feds on Internet Banking - _"E. ALLEN SMITH"<EMAIL_ADDRESS>